Review of Screaming Frog SEO Spider Tool by ScreamingFrog

Before optimizing any website, it helps to have a big picture view of the website.  How many pages does it have?  How are the titles and meta descriptions? Are there broken links?  Site audit software is required for this job.  Read on for a Review of Screaming Frog SEO Spider Tool.

Screaming-Frog-LogoWhat does Screaming Frog SEO Spider do?  It “crawls” through your whole website and creates a list of all your internal pages. It tracks:

  • Crawl level
  • Internal and Outbound links per-page
  • Keyword elements like Title and Description and their lengths.
  • HTTP status codes

A review on how I use the Screaming Frog SEO Spider:

First, I use the software to index the whole website and save the results as the “before” crawl.  This is helpful to demonstrate to a client what changes I have made to a website in the future, by having a snapshot of the website before I started.  Site Audit software is essential for any SEO job.

Screaming Frog Screenshot

Second, I get the big picture of a website’s SEO elements and find what needs updated or fixed.  Screaming Frog SEO Spider collects the vital SEO elements for each page during the crawl: title, description, keywords, H1, h2 etc.  This makes it easy to examine the whole website and see what needs to be updated.  I like to make sure that every page has unique title and descriptions using this tool and also see if the H1 tags are used and the meta keyword is not being abused.  Then, I review the lengths of the titles and descriptions and make sure there aren’t ones that are too long or too short.  Keep in mind, I try to get all this info out of Google Webmaster tools first, before I use the Screamingfrog website audit tool.

Third, I use it to find and fix broken links within a website.  Websites, especially older and larger websites, often have many interior links that are broken.  Screaming Frog makes it easy to find pages that are generating 404 errors and then it shows you every page on the website that links to that 404 page so you can fix all the links.  Additionally, it also shows you 301 redirects, so you can update internal links to point to the new and actual location of the pages instead of being redirected.  This is called the “canonical URL” and any link that doesn’t point to it should be corrected.  My goal is to fix every single 404 and 301 page!

Fourth, I use it to find and fix broken links pointing to external websites.  One of Screaming Frog’s website audit reports shows external links and their http status.  This allows me to fix an 301 redirects and especially in links resulting in a 404 page.  I believe that Google may dislike websites with lots of bad out-going links and also web pages with bad external links.  Plus, broken links frustrate visitors and reduce your credibility.

Screaming frog is free for indexing website with less than 500 pages.  It works on PC, Mac or Linux.  The license which unlocks all of its features is £99 per year. Here is a video from the creators of the software with a short demo to help you review the software:

Screaming Frog SEO Spider has many site audit functions and SEO features I didn’t cover in the short article above. Most notably, it also:

  • Exports your data to a spreadsheet
  • Finds duplicate pages with duplicate content
  • Tracks the Meta Robots tag so you can find the noindex, nofollow problems
  • Measures file size
  • Tracks Page Depth level (how many links from the homepage)
  • Organizes Inlinks and Outlinks to any given URL
  • Tracks the Follow and NoFollow status of all links
  • Custom Source Code Search – find anything you want in the source code of a webstie
  • XML site

For years I have used Xenu’s Link Sleuth to index pages and ferret out broken links within a website.  It is super-fast, powerful and free.  But it is very geeky and hard to recommend to clients.    This year I switched to a MacBook and had to dump Xenu from my repertoire: it only works on a PC

Other software to investigate for an SEO site audit (but not reviewed here): SEOmoz Pro App, IIS SEO Toolkit, AnalogX Link Examiner, WinWebCrawler, Integrity (Mac) and A1 Site Analyzer.

Read More:




Related Posts

9 thoughts on “Review of Screaming Frog SEO Spider Tool by ScreamingFrog

  1. Sara

    “Screaming Frog makes it easy to find pages that are generating 404 errors and then it shows you every page on the website that links to that 404 page so you can fix all the links.” I am sure I am missing something obvious, but I can see the 404 errors but don’t know how to find the pages on the website that contain the broken links …

    1. Trent Blizzard

      Sara, In the top pane, click on the page that is having the 404 problem. then, go to the bottom of the screen, and click on the “In Links” button and it will show you all the internal pages in your website that are pointing to that broken page.

  2. Screamingfrog

    Screamingfrog ist one of the best Onpage-Tools at the Online Tool Markets for SEOs. I especially appreciate the way that you can analyze and optimize several hundred thousand sub-pages easily . This is unique!

  3. Graham O'Shea

    I’m the founder of Our toll does everything Screaming Frog does and much more. Our SEO Crawler is cloud based so it is extremely fast taking up no local resources. We were selected for the 2015 Web Summit Alpha Program. Check us out and we welcome any feedback.

  4. Rosie Williams

    Screaming Frog is such a useful tool for a reasonably technically minded marketer. I do SEO for a living and although I can’t code, screaming frog gives me the information and tools to improve a websites performance quickly by identifying errors, duplicates etc. You can also export everything to excel – we all love using excel right?
    At first I was hesitant on the price, but actually its very cheap compared to many of the other online tools and definitely value for money!

Leave a Reply

Your email address will not be published. Required fields are marked *

Skip to content