Clicky

X

Subscribe to our newsletter

Get the State of Digital Newsletter
Join an elite group of marketers receiving the best content in their mailbox
* = required field
Daily Updates

How to Use Xenu for the basis of a SEO Campaign

16 June 2011 BY

5 Flares Twitter 0 Facebook 0 Google+ 5 LinkedIn 0 Buffer 0 Email -- StumbleUpon 0 Pin It Share 0 Filament.io 5 Flares ×

Depending on who you ask Xenu is either the dictator of the “Galactic Confederacy” who, 75 million years ago, brought billions of his people to Earth in a DC-8-like spacecraft or a tool that finds broken links. The second option doesn’t sound as interesting but to you and me it is far more valuable.

Earlier this week I was speaking to a few SEOs I know who work in house and I was surprised to hear that neither of them had heard about Xenu. I was even more surprised since they both work for large brands whose sites have over 50k pages indexed and if anyone can benefit fromXenu it is large sites.  Xenu was originally built as a broken link checker but as I will illustrate you can use it as a basis for technical and on poge analysis but to also as a basis for client questions and discovery. A lot of our regular readers will most probably be familiar with Xenu although there are hopefully some takeaways for everybody.

Getting started…


Load up Xenu and enter in the website you want it to crawl.  Before you click run, go into the more options section in the bottom left and make sure you click ‘treat redirections as errors’ as this will now identify the http status codes for all redirects (important if you want to benefit from amending 302s to 301s)


Once Xenu has finished, click file and export the report to a TAB separated file as displayed below.

Once you have saved this file, open up an excel spreadsheet, click file => open and find your text document and then click open. Follow the following instructions to transform your text document into an editable and manageable excel spreadsheet.


Make sure you have clicked ‘delimited’ as the file type that describes your data


Then click tab as the delimeter…


Now you have your spreadsheet, depending on what version of excel you are running, copy all the cells and content and filter it so you get the drop down arrows on the top level cells.


Finding quick wins via status codes

For anyone wanting to skip the main course and dive headfirst into the desert simply go to the status codes column. This drop down will give you a list of status codes but what you are looking for are the following for quick actionable issues:

-2            This is an external link the tool has found
302         A temporary redirect
301         A permanent redirect
404         Page not found

-2

-2 informs you of an external link on your site.

TAKEAWAY:

If you have just taken a new client on this can be valuable in three main ways. Firstly, you get a nicely organised list of all the external links, secondly, you can gain an insight into the existing relationships the site has and lastly it allows you to follow up and add these as potential link prospects for the later stages of your campaign.

302

This will give you an insight into the historical changes of the site and will highlight all 302 links. Most often than not these links are not temporary and are not passing on the link equity to the new pages.

TAKEAWAY:

Implementing a 301 redirect will pass the value to the new pages and depending on how many there are could give you domain/page a slight bump.

404

This identifies pages that could not be found by the server.

TAKEAWAY:
Apart from fixing these links, one tip is to find the URL within the Xenu tool, right click and use the wayback machine to see what content (if any) used to be published there. You might find out it is seasonal content in which case you should read the article surrounding what to do with old seasonal content and then try implementing “living URLs” as explained by Michael Grey.


Rev your Engine, Just how fast is your site?

SEOs have been saying this for years now but improving site speed is important. Not just for SEO purposes but for the end user. As explained in this article the faster a site is the more pages a user will be likely to view so go away and make sure you action the tips on optimising page site speed. I use Xenu and firebug to test site speed in the early stages of an audit. Start by sorting the spreadsheet by the ‘size’ column, in descending order.

TAKEAWAY:
Xenu automatically displays the file size in bytes which can be a bit overwhelming but to get a more laymen idea of the size you can use simple formula and divide by 1024 to convert the bytes to kilobytes and make things a bit easier to read. Dividing this new column by 1000 will give you the megabytes size which most people are familiar with and will give you a greater idea of whether the size is too big.

Make a note of files that are too large and use firebug to test how long it takes to load individual elements of a page. Think about compressing files or hosting video content on its own unique URL to reduce page load times of important pages.

Duplicate (duplicate) Content (content)

The recent Panda update means it is important to remove or amend duplicate content pages and filter out the poor performing pages of your site. The process of reassessing “low quality” pages, especially for large sites, should be a part of every SEOs strategy. Methods of removing low quality content have been discussed through this methodology and you can also use analytics to help.

TAKEAWAY:

Slightly off point but valuable nevertheless, use Xenu to check internal duplicate content as these pages can be fixed with redirects, canonical or no index tags. Some pages can also be improved, for example if it was a new product page which automatically implemented the default title tag setting that is common on many CMS. Manually checking your site would obviously take a lifetime so Xenu is great for speeding this process up by 5 years or so. In your spreadsheet click on the dropdown of the title column and deselect all. Now you can go through and click on the titles and see if they produce one or say several URLs. If they use the same title tag then they most probably have the exact same content on the page too. Look at this content and see how it can be dealt with or made more unique.

Analyse the “Money” Pages

What do we mean by money pages? Key pages which contain your core set of keyphrases and are more likely to result in conversions. These are the most important pages of a site so it is crucial they are not only reached within as few clicks as possible but also linked to effectively throughout the site. This is particularly important for larger sites which have a range of silos within their architecture.

Level

This equates to how many clicks away from the homepage the content is. Sorting your data by the level allows you to identify “money” pages which may be more hidden than ideal. Preferably all key pages should be located within 1-2 levels away from the root. If you find you have key pages hidden 3,4 or 5 levels down make a note and come back to address the information architecture.

Always remember that information architecture shouldn’t remain static, it should change in line what customers’ needs and tools such as analytics, Google trends & insights should be utilised to sculpt the architecture.

TAKEAWAY
One process that should be continuously monitored is the internal site search feature within your analytics software. If you are looking to insert additional internal linking or create new navigational areas this can be a goldmine. This process enables you to identify what users are actively searching for that they can’t find easily enough within your site. Out of this process could come cross linking opportunities, new categorisation, or new internal links such as ‘top selling products’ to drive visitors to pages they want quicker.  This article by Avinash Kaushik on internal site search is old but still has some very relevant points.

Links In

Organising your spreadsheet now by Links in and by descending order will highlight pages that have been either ignored or forgotten about.

If you imagine a pyramid built of glasses, if you pour champagne into the top glass you expect it to flow down slowly filling up the cup beneath a little less than the top layer. This is how your data should look and if it doesn’t then something is wrong and it should be addressed.

Since the Panda update, the importance of individual page link metrics has increased and pages located deep within the site without any links or treated negatively, or at least in comparison to how they used to be treated when pages would rank purely from being hosted on domains with strong metrics.

TAKEAWAY:

One way of increasing the amount of internal links to a page would be using a variety of tactical linking methods such as:

  • Standard navigation
  • Ancillary navigation (footers)
  • Breadcrumb trails
  • Cross linking from similar product pages
  • Listings of previous on-site search results
  • Implementing widgets to pull through most popular products
  • Using blog content to deep link

These are very simple yet very effective methods. It is utilised mainly by B2C organisations as it is very customer centric although if you are clever and creative enough you can find many ways of cross linking.

Here are some great examples:

 

 

There is a lot more you can do with Xenu so feel free to tweet me but in the meantime that should be enough to set you on your way.

AUTHORED BY:
h

Sam Murray graduated from University with a BA (Hons) in Marketing in 2007 and wrote his 10,000 word dissertation on Search Marketing. Sam is a freelance search manager.
  • http://www.searchmarketingstandard.com Jaan Kanellis

    Love Xenu, just wish it would crash on larger sites.

  • g1smd

    I don’t export to a spreadsheet. Instead I let Xenu generate its HTML report as it does that in sections by error type.

    I use Xenu to ensure I link out to http://www.example.com/ and not to http://www.example.com/index.html

    I use it to correct broken internal links.

    I use it to find dead external links.

    I use it to find 302 and 301 redirects.

    When linking out to other sites it is also informative to see which of those links lead to redirects. Those that redirect to add a cookie and those (such as used by MediaWiki) that 302 redirect from root to /Home I don’t change. Most of the others are fixed by changing the linking to instead point to the canonical URL for the other site, especially where the former linking hit a www/non-www canonicalisation redirect.

    I also use Xenu to find sites that have infinite URL space. This happens on many sites with calendar functions, or on those which use the current date/time as a search parameter within the site (big problem on MediaWiki sites for example).

    There’s several other things it can do. It’s a valuable tool.

  • Exploter

    Can i use xenu to get list of external links from mine site, like getting list of domains

  • http://www.shoppingcartstrategies.com/thanks-for-visiting/ Dave Beck

    Nice article Sam!

    Xenu has been a long time favourite tool of mine. I have used professional tools that are nowhere near as powerful especially when you consider that xenu is free.

  • http://www.greatwebsitesblog.com Barry Adams

    I <3 Xenu, I really do…. but ever since I started using Screaming Frog's SEO spider, there's just no going back. Xenu is a nice, comfy Toyota – reliable and gets the job done. But Screaming Frog is, well…. the Lamborghini of site crawlers.

    • http://www.redmudmedia.com/ Ralph du Plessis

      Funny you should say that, Barry. I’ve been using Xenu mainly, but we got a copy of Screaming Frog about a year ago which my colleague uses so we tend to run both tools and then cross reference.

  • http://blog.webpro.in Bharati Ahuja

    Nice Article.

    XENU is an all time favorite and I had used it for many years.

    But recently use other tools also. But yes XENU is very helpful.

  • Pingback: Xenu’s Link Sleuth – поиск сломанных ссылок | Мнение Чернова

  • Bill Slawski

    Nice post.

    Xenu has been one of my favorite SEO tools for a long time. I like using the HTML report, and sending the information to a spreadsheet as well.

    In addition to being able to identify and fix internal redirects and broken links, it is very useful for identifying and isolating spider traps. Disallowing URLs listed in robots.txt is recommended, and using the “disallow” for pages/folders that do have infinite URL spaces is recommended so that you can crawl the rest of the URLs on a site.

    In spreadsheet form, I find it helpful to also separate the images from the rest of the content of a site, and sort by size to more easily find images that might be too large. Identifying the locations of those images can sometimes be difficult unless you generate a Pagemap from Xenu as well.

    As g1smd noted, there are several other things that Xenu can do, too. Great tool.

    I also like Screaming Frog’s SEO Spider, and that’s very useful for creating a content inventory for a site. There are times when it misses some of the stuff that Xenu captures, so I find it useful to use both.

  • https://deakwebmicrosoftonlinecom-1.sharepoint.microsoftonline.com/ Doug Smith

    It also works on MSO SharePoint sites that don’t allow anonomyous access, just provide your credentials when prompted. Even a SharePoint site can have broken links.

  • http://www.gudipudi.com Gudipudi

    Xenu has been ( and still ) my favourite tool

  • http://www.bradfry.com Brad Fry

    I wish there was a 64bit version so it wouldn’t crash when it gets to 2 gb of memory.

    Has anyone found a tool that can handle sites larger than Xenu can?

  • http://www.highergroundcreative.co.uk Phil

    I’ve known about Xenu for some time but I haven’t used it that much, and even though I really don’t like its interface, I’ve found it very useful when organizing pages/strucure of a website, in the past. There are however some other tools out there that can do what Xenu does and much more, albeit not free!

  • http://www.vervesearch.com Sam Murray

    Thanks for everyones comments. I knew most people were familiar with Xenu but just wanted to get an article out there for those that wasnt and to also maybe look at how you can use it in a different way. As Bill and g1smd stated it can do a lot more too.

    @g1smd One of the reasons I export it to a spreadsheet is that I use it as part of a technical audit and when I present or explain it to the client it is easily organised and filtered via excel.

    @Exploter Yes you can, as I mentioned you can, where ou see -2 in the status that usually means it has found an external link. Just filter the columns and sort by status code and et voila.

    @Barry, does screaming frog crash when using it for larger sites in your experience?

  • http://pro4.es Rafael (presupuesto web)

    I love Xenu, it is a big secret of SEO users. With Xenu I have started with my progress

  • http://twitter.com/studiumcirclus James Allen

    Xenu, IIS SEO Toolkit and Screaming Frog are the backbone of timely SEO-auditing IMO. Considering that it’s free, Xenu does a great job.

5 Flares Twitter 0 Facebook 0 Google+ 5 LinkedIn 0 Buffer 0 Email -- StumbleUpon 0 Pin It Share 0 Filament.io 5 Flares ×

Nice job, you found it!

Now, go try out the 12th one:

Use Google Translate to bypass a paywall...

Ran into a page you can't read because it is blocked or paywalled? Here's a quick trick (doesn't always work, but often does!):

Type the page into Google translate (replace the example with the page you want):

http://translate.google.com/translate?sl=ja&tl=en&u=http://example.com/

How about that!?

Like this 12th trick? Tell others they need to look for this trick on our page: http://www.stateofdigital.com/search-hacks-marketers/

Or Tweet: Found the secret 12th one!