New Year Resolution: Crawl Quarterly
Search Engine Optimisation

New Year Resolution: Crawl Quarterly

16th December 2014

In the new year, if you are in charge of clients, are in-house, or just in charge of a site of some kind, I implore you to set a new year’s resolution to crawl your site regularly. Most of the issues that I see with clients from a technical perspective can be solved with a regular crawl and time to fix the small issues that arise.

Call this a new mission, but I am determined to get everyone to crawl their sites regularly. I recently spoke on this topic at the State of Search conference in Dallas, TX. Many of the people attending my session knew everything in my presentation, but there are more people out there that find crawling confusing or daunting and it’s those people I want to reach. Crawling is no longer just for “technical” people – there are so many tools that have been developed to help making crawling automated, fun, and actionable. It’s just a matter of finding the right tool, making it a regular action item, and focusing on the right things.

This is the shortened version of my talk … the action items if you will …

Finding the Right Crawler

There are a number of crawlers on the market and there is one for you or your client. I created a graphic that shows the right crawling tool for your specific situation. This graphic was created in my professional opinion, but if you want more information, check out my fill deck below. I suggest you test all of the tools to find the right one for you.

Crawler Scale by Kate Morris and Outspoken Media

I know I am missing some of the crawlers on the market. The nice people at Site Condor came by after my presentation to say hi and gave me a look at their crawler a few weeks later. This space is moving constantly so don’t get married to one tool. Keep your mind and eyes open – and share any thoughts you have for new features with your favorite tool. I know each company welcomes ideas, so share away!

Making a Regular Schedule

You might be thinking that it’s as simple as a recurring calendar invite, but to reap the full rewards of a regular crawl schedule, there are a few things to keep in mind.


If nothing changes about your site, a crawl should be performed quarterly. Start it in January and then do one at the top of every quarter. But things do change, and it’s recommended that you crawl the site before and after each major site change and define action items in the post crawl.

Withstanding Staff Changes

If you just set a calendar item for your own calendar, or have someone else do it, it will get lost over time with staff changes. Talk to company (internal or client) development teams to add this into their standing processes. Also ensure that the marketing team adds it to their process for any site revisions and marketing reviews.


The final thing to consider aligns well with changing processes to protect staff changes, and that is ensuring you have the resources to fix any issues in the crawl. This means development or marketing time each quarter and after every major site change. Your action items can be anything from changes to robots.txt to getting a copywriter to fill in missing meta descriptions.

Focusing on the Right Metrics

Doing a site crawl can be daunting with the amount of information you can get with a crawl. We use a crawler every time we do an SEO Audit at Outspoken, but there are a few facets that are the keys to any audit/crawl.

  • Errors – Naturally, the server (5XX) and Not Found errors (4XX) are going to be of the highest priority. They are issues for users and search crawlers.
  • Redirects – Redirects should never happen within your site. They are going to over time, but should be fixed as soon as possible. It’s link equity you can control, so try to keep any of it from passing through a redirect.
  • Duplicated Titles – These are the first warning sign of duplicate content and canonical issues. Check out titles for any that are missing, too short or long, or duplicated.
  • XML Sitemaps – Finally, use a crawler to check your XML sitemap to ensure that you are only giving search crawlers the best URLs to crawl.

Below is my presentation in full for those that are interested in all the tools reviewed, the metrics to watch, the recommended timeline, and other fun uses of crawlers.


Written By
Kate Morris is the Director of Client Strategies with Outspoken Media. She is a well seasoned online marketer with a passion for teaching others. For the last 10+ years she has covered the paid, natural, and social sides of search.
  • This field is for validation purposes and should be left unchanged.