Review of Screaming Frog SEO Spider Tool by ScreamingFrog

Before optimizing any website, it helps to have a big picture view of the website.  How many pages does it have?  How are the titles and meta descriptions? Are there broken links?  Site audit software is required for this job.  Read on for a Review of Screaming Frog SEO Spider Tool.

What does Screaming Frog SEO Spider do?  It “crawls” through your whole website and creates a list of all your internal pages. It tracks:

• Crawl level
• Internal and Outbound links per-page
• Keyword elements like Title and Description and their lengths.
• HTTP status codes

A review on how I use the Screaming Frog SEO Spider:

First, I use the software to index the whole website and save the results as the “before” crawl.  This is helpful to demonstrate to a client what changes I have made to a website in the future, by having a snapshot of the website before I started.  Site Audit software is essential for any SEO job.

Second, I get the big picture of a website’s SEO elements and find what needs updated or fixed.  Screaming Frog SEO Spider collects the vital SEO elements for each page during the crawl: title, description, keywords, H1, h2 etc.  This makes it easy to examine the whole website and see what needs to be updated.  I like to make sure that every page has unique title and descriptions using this tool and also see if the H1 tags are used and the meta keyword is not being abused.  Then, I review the lengths of the titles and descriptions and make sure there aren’t ones that are too long or too short.  Keep in mind, I try to get all this info out of Google Webmaster tools first, before I use the Screamingfrog website audit tool.

Third, I use it to find and fix broken links within a website.  Websites, especially older and larger websites, often have many interior links that are broken.  Screaming Frog makes it easy to find pages that are generating 404 errors and then it shows you every page on the website that links to that 404 page so you can fix all the links.  Additionally, it also shows you 301 redirects, so you can update internal links to point to the new and actual location of the pages instead of being redirected.  This is called the “canonical URL” and any link that doesn’t point to it should be corrected.  My goal is to fix every single 404 and 301 page!

Fourth, I use it to find and fix broken links pointing to external websites.  One of Screaming Frog’s website audit reports shows external links and their http status.  This allows me to fix an 301 redirects and especially in links resulting in a 404 page.  I believe that Google may dislike websites with lots of bad out-going links and also web pages with bad external links.  Plus, broken links frustrate visitors and reduce your credibility.

Screaming frog is free for indexing website with less than 500 pages.  It works on PC, Mac or Linux.  The license which unlocks all of its features is £99 per year. Here is a video from the creators of the software with a short demo to help you review the software:

Screaming Frog SEO Spider has many site audit functions and SEO features I didn’t cover in the short article above. Most notably, it also:

• Exports your data to a spreadsheet
• Finds duplicate pages with duplicate content
• Tracks the Meta Robots tag so you can find the noindex, nofollow problems
• Measures file size
• Tracks Page Depth level (how many links from the homepage)
• Organizes Inlinks and Outlinks to any given URL
• Tracks the Follow and NoFollow status of all links
• Custom Source Code Search – find anything you want in the source code of a webstie
• XML site

For years I have used Xenu’s Link Sleuth to index pages and ferret out broken links within a website.  It is super-fast, powerful and free.  But it is very geeky and hard to recommend to clients.    This year I switched to a MacBook and had to dump Xenu from my repertoire: it only works on a PC

Other software to investigate for an SEO site audit (but not reviewed here): SEOmoz Pro App, IIS SEO Toolkit, AnalogX Link Examiner, WinWebCrawler, Integrity (Mac) and A1 Site Analyzer.

Read More:

• Crawler Face-off: Xenu vs. Screaming Frog
• Screaming Frog Review
• A Review of Screaming Frog’s SEO Spider Tool
• Move Over Xenu: Make Way for Screaming Frog’s SEO Spider [Review]