How to Clean Up After Botched Site Migration
Search Engine Optimisation

How to Clean Up After Botched Site Migration

3rd October 2016

sad-robots

Much has been said recently about the importance of a site migration done well, at the most recent BrightonSEO, for example, you may have caught one of the two talks which covered this topic in some detail (disclosure, I delivered one of them, the other was by Jon Earnshaw – watch the recording here, with some more nuggets from Barry Adams, myself, David Bain & co here).

For those who didn’t see any of that you could summarise the key points easily:

  1. A migration is make or break for any website
  2. You need to plan a migration from the start
  3. Do your research first – be clear of the size of the task
  4. Get buy-in from all parties
  5. Make friends with those people who will make the project easier
  6. You don’t have to accept that a migration will always mean a loss in traffic

Looking back at a failed migration, all the time for research and extra diligence seems like a luxury, something that’s really easy to talk about in retrospect – but doesn’t help the immediate need. What these talks did was to paint a vivid picture of what happens if they go wrong, but what neither discussed was what you need to do to recover from it.

The guide that follows are some of the most important lessons in reversing this process, they have come from working with in-house marketers, third-party agencies and colleagues.

Stop Waiting – Do Something About it!

I’ll start with the painfully obvious one – actually addressing the problem. One of the most perplexing responses to a migration-gone-wrong is simply doing nothing. Graphs start to slide, arrows turn from green to red and a kind of paralysis sits in.

How long do you have to wait to see if the migration is a success/failure?

Getting ahead is getting started

Jon Earnshaw’s 14 day migration window, gives an idea of how quickly it can be on some projects (if you’ve got a high-authority website in particular), but in reality this two week window could stretch out to as much as 12 weeks with some.

This doesn’t mean, however, that you can keep putting off doing something about it. If traffic is plummeting soon after a migration there is a school of thought which suggests it’s unlikely to recover.

I have seen poorly implemented migrations sat on for more than six months before seeking assistance – the cruel twist here is that if you leave it this long to fix, the likelihood of a significant recovered is far, far less.

Whether it’s asking for expert help, or putting additional resources into the problem – it can’t always be “tomorrow” that you deal with it. Sooner is better than later.

Find out what has gone wrong

I’ve already bounced around the phrase “failed migration” in this article, but to really make that judgement, you need to understand what “failing” is, and the cause of it.

There are many different types of migration and many things which need to be considered as part of how you measure/judge the success. In this piece I’m only focusing on “SEO metrics” as this is the most common cause for issue.

For me a “failure” is losses beyond any acceptable or expected as part of the process. To re-enforce a point that Jon Earnshaw made, site migrations don’t have to equal loses in traffic/rankings.

However, this depends greatly on how the site changes as part of the migration process. If you lose content/pages in any great amount, no matter how well you 301 those pages, you will likely see drops. This doesn’t mean failure, this is inevitable and flagged early on in the process. What could be considered failure is when you’re not prepared for it to happen.

Let’s imagine that post-launch you’re seeing a 30-40% loss in organic traffic, there are some straightforward questions you need to ask yourself:

  1. Can Google still crawl my website?
  2. Do I have more or less content than before?
  3. Have I maintained any of the keyword optimisation from the previous site?
  4. Are the redirects in place, are they working correctly?

How easily these questions can be answered will vary significantly, but you’ll be able to establish why it isn’t worked with one of the following – I’m confident of that.

Google Can’t Crawl My Site Properly – How do I fix it?

How to fix crawl issues is easily a blog post in it’s own right, but the vast majority of crawl issues post-launch will be related to the following items:

  • Robots.txt – The “classic error”,  “Disavow: /“ has been left in, or some important scripts are still being blocked. Google and other search engines will simply stop crawling the website.

Robots txt

  • Meta Robots –  A nofollow/noindex rule is still in place, which leads to similar results as above.

Meta Robots Block

  • X-Robots – As above, but at an HTTP header. This one can be harder to spot if you’re not looking for it, but again equally as bad if left in place after migration.

X-Robots

The easiest way to check if Google can crawl your website is to ask it – or in other words, fetch as Googlebot.

Fetch as Googlebot

This will confirm whether Google can crawl your site and whether any scripts are getting blocked. Remember, this won’t detect whether any noindex tags are in place – you’ll need to check the code/ HTTP response for this.

Tip: If you’ve discovered and removed noindex tags which have been blocking key pages, fetch as Googlebot and then “Submit to index”. Your pages could be back in search results again in as little as 15mins.

You can also have crawl issues without blocking search engines specifically, i.e. where Google can crawl it, but it’s not able to do so properly. This list, again could be really long, but to give you somewhere to start off:

  • Navigation isn’t crawlable
  • Site structure is too deep
  • There’s not enough inter-linking between pages
  • Content is loaded through AJAX or similar
  • Incorrect canonical tags are being used

To identify and address these issues is based around more good ‘on-page’ SEO practice – which you can find more on here.

Is my Content Better or Worse?

A question of content quality is a subjective one; a Copywriter, a Graphic Designer and an SEO could all have very, very different ideas of that is “better” in regards to content.

A Copywriter, a graphic designer and an SEO...

To get you on my wavelength we’ll assume that “good” content is as follows:

  1. Written to a high standard
  2. Descriptive & gives the information which is needed
  3. Not too complicated for the audience
  4. Uses keywords in a way which makes sense (i.e. reads well)
  5. Is not copied from somewhere else
  6. Helps the user get to what they’re looking for
  7. There’s enough of it
  8. It’s not hidden

To work this out, you’ll need to run some comparisons from old to new, if you’re fortunate enough to have taken a copy of the old site database saved a crawl of the site, this is easy. If not, Wayback Machine can be really useful here.

Another tip: you can use Screaming Frog’s extraction to quickly audit your new site content – this guide will get you pointed in the right direction.

Run a comparison within the key pages (important ones, or ones which are performing significantly better/worse), ask yourself the following:

  • Are the important keywords missing?
  • Are entire pages missing?
  • For those pages which remain, has the content quantity dropped significantly?
  • Has the content been hidden or pushed too far out of the way?

If the answer of one or more of the above is “Yes”, this could quite likely be the problem right there. You’ll need to run back through more of the content methodically and establish what changes need to be made with the content in order to bring it more in line with what you had previously.

NB: Run the rest of the checks listed above & below before jumping on this one – if the content was over-optimised before, having less could be better. Rule out the other most-likely issues first.

Have the Redirects Worked/Been done Properly/Been done at all?

This is my favourite one.

When I mean “favourite” it’s most often the one which will shed light onto the problem. Redirect mapping can be hard, time-consuming and easy to get wrong.

If you’re arriving at the scene after the event has happened establishing if they’re working correctly can be really tough. You first need to establish what pages were needed to be redirected and then test them.

  • Check Google Search Console – As Google crawls through the new site, it’ll pick-up the broken links if the redirects haven’t worked. This can be slow to wait for and will show you what is missing, not what is being redirected incorrectly. Search Console must always be part of a regular post-launch checklist, for weeks/months following.

Search Console 404s

  • Check Historic Ranking Data – If you are tracking rankings, check which pages ranked for your keywords – Do they 404 or do they 301 to the correct destination?

Historic Ranking Data

  • Check Historic Google Analytics Landing Pages – Pull the last year’s worth of landing pages from Google and then check them – you’ll quickly see which pages have/haven’t been redirected properly.

Google Analytics Landing Pages

  • Check Historic Log File Data – As above, but instead of relying on traffic you’re looking at log files. There may be a lot of noise in here and for the purposes of this exercise this may be too-technical for some.

Log File Data

  • Check Wayback Machine – My favourite “hack” around this task, visit the old sitemap.xml on Wayback Machine – save a copy and then crawl it, that’s the quickest way of seeing if your key pages were redirected.

Way back machine sitemap

With any of the above steps you’ll get a picture of how much of this has been done properly. A pertinent question here may be, what does “properly” look like?

  • Do all your old key pages 301 redirect correctly to the new versions?
  • Do you have a many-to-few redirect scenario, i.e. old products being redirected to category pages rather than to the new product pages?
  • Is the correct redirect being used? I.e. 302 or 307 are temporary redirects which could be causing problems
  • Are you missing a large proportion of the old pages? This is a tough one to attach a number to, but 5%-25% pages missing will likely have substantial implications (depending on how large the site is).

Either way, follow your redirects, make yourself familiar where they’re redirecting to and whether they’re using the right response code. Any content which redirects to the wrong place with the wrong response needs addressing as a top priority.

In Summary

For an easily-digestible list (one more won’t hurt!), here’s what you need to do if you suspect your migration has not gone as planned:

  1. Do something about it – move quickly if things look like they’re going in the wrong direction. Migrations don’t have to mean losing traffic/rankings!
  2. Establish what about the migration hasn’t worked
    • Is the site crawlable or suffering from crawl issues?
    • Is the content worse than it was before?
    • Have the redirects worked?
  3. Ensure you’re not blocking Googlebot or other search engine spiders
  4. Add in internal linking and make changes to top-level navigation if needed
  5. Compare old and new content, plan a new content strategy if you’re missing anything important
  6. Check your main data sources for any missing/incorrect redirects – rebuild them as needed.

This can be a difficult and stressful time for all involved, especially if the business starts to fail because of it, but make sure and use the time and resources needed to get to the bottom of the migration issue and establish a way out of it.

It will always be cheaper to get it right the first time, but using time a budget to rectify the problem in the short term will always pay dividends – especially if the option is to wait for it to get better.

Written By
Chris Green is Head of Marketing Innovation at Footprint Digital, a digital marketing agency based in the East of England.
  • This field is for validation purposes and should be left unchanged.