The value of a website’s internal search engine is often underestimated. This little box, often hidden away between all kinds of call to action elements, can create a lot of insight. Next to the insight given by the data it provides, you can use it as a marketing instrument by optimizing the recommendations given in the results.
Basically, there are two approaches to build a recommendation system within your internal search: the content-based approach and the collaborative filtering method. Content-based means you will make use of the data about the website’s content or products in the case of an e-commerce platform. Users buying a pot of paint probably also need a set of brushes. Collaborative filtering is a method that uses user behaviour data. User ratings, previous searches or buying patterns are a few example datasets that can be used.
Internal Search insights
Most websites have an internal search engine available for their users and nowadays it is very easy to setup your tracking. Have a look at Google guide Set up Site Search and you’ll see it takes a few second to collect internal site search data. After installing tracking code for the internal search engine, most marketers just use the data to optimize the content of their website. In the analytics dashboard, look for the % Search Exits. This is a valuable metric, since is measures the quality of the search results. This gives you a starting point for which queries you have start optimising the internal result pages. Google Analytics also shares E-commerce data within the Site Search dashboard, which shows the most valuable searches.
If you see a lot of people use a specific informational query, you can add that the answer to their question to the main menu. If a there is a product name very popular, make sure to add that product to the homepage. It is a powerful tool to research what your website visitors are looking for. If people don’t understand how your website works, they will make use of your search engine to find what they want. Visitors using your search engine are already engaged with your website, so make use of that.
How important is this data? Google understands how valuable this data is, that is why they have introduced the Sitelinks Search Box September last year: An improved search box within the search results By doing this, Google gets free insights into the queries people are using to navigate trhough your website. Based on that, they can adjust their organic results or optimize their advertising platform. Make sure you implement the JSON code on your website, otherwise user will stay in Google Search instead of being send to your internal search engine result pages. Within Google, competitors can advertise so you can actually loose traffic.
Making use of internal search data
There are four main reasons why you should invest time in analysing internal search data:
- Website optimization: based on the informational need of visitor, adjust your homepage and menu content.
- Insight in filtering: if visitors can’t find a specific product by using the available filters, they will start using long tail queries to see if they have missed something with the hope still finding the product they were trying to find by navigating via the menus
- Content ideas: analysis will reveal insights about naming conventions, information that is not available yet and gives insight into long tail keywords.
- Market research: often people can’t find a product, since the website simply doesn’t sell it. They will try to find it since they expect your website should be selling that specific product. Based on these queries you can develop new products or enrich your current product range.
Optimizing your internal search engine
There are several ways you can optimize your internal search ranging from easy to implement auto complete systems, to machine learning algorithms to determine the most suitable results based on user behavior.
Basic optimization
The most easy optimizationis integrating corrections for typographical errors and interpretation of queries. There are many paid solutions available to start with on a small scale, but I am really impressed by the possibilities of Algolia, which also has a WordPress plugin. They offer standard packages that include basic optimization for typo’s.
The next step is to order the results. Most logical would be the most read articles in the above case, but in theory you can order based on any value you have available for all your results available. For e-commerce you can filter on most sold products, the cheapest option or specific products that have temporary discounts. Basically any factor that can influence user behavior in a positive way. Below an example of results ordered by the cheapest price in one of the engines I’m currently developing:
Since speed is crucial, make sure you enable caching for the most use queries. For example, make a daily cache of all the responses (262) for the first one and two letter combinations “a”, “aa”, “ab” etcetera and refresh those once a day, or more often if search dynamics change during the day. This makes it possible to give the user a likely relevant autocomplete suggestions within milliseconds.
A more advanced technique would be to discover synonyms and find associated results for that. To do that, you need topic modelling, a process for discovering “topics” within a collection of products or documents:
Intuitively, given that a document is about a particular topic, one would expect particular words to appear in the document more or less frequently: “dog” and “bone” will appear more often in documents about dogs, “cat” and “meow” will appear in documents about cats, and “the” and “is” will appear equally in both. A document typically concerns multiple topics in different proportions; thus, in a document that is 10% about cats and 90% about dogs, there would probably be about 9 times more dog words than cat words. A topic model captures this intuition in a mathematical framework, which allows examining a set of documents and discovering, based on the statistics of the words in each, what the topics might be and what each document’s balance of topics is.
Source: http://en.wikipedia.org/wiki/Topic_model
User based possibilities
Once you have customers and they are logged in, track their onsite behaviour and based on that optimize the suggestions and result pages. Based on previous purchases and the use of machine learning systems like the Google Prediction API, it’s easy to give personalized suggestions in the search results. With machine learning becoming more accessible for non-scientists (currently Angel.co lists almost 600 machine learning start-ups), this will probably be the most interesting development with regards to e-commerce optimisation in the upcoming years. But you can start by giving suggestions based on previous purchases: if someone always buys black jeans, show them black jeans before blue jeans.
These systems can not only be used for internal search but also for all the other recommendations you can make on your website, think about “related” products, people also bought listings and dynamically ordering products in category pages to name a few possibilities. The biggest problem for most websites is the absence or sparsity of data so to begin with, base your systems on logic and test a lot.
To built a database used for predicting queries, built in a data collection system that saves every character typed in. By doing that, it is easy to predict the final query based on the sum of the historical queries made, what users are going to search for when their first input consist of specific character combinations.
Based on IP addresses, you could add or remove specific search results based on geographical data. Webshops that don’t deliver specific products in the neighbourhood of the user, could filter those products out. Or the other way around, make special offers within the search results based on the user’s location:
DIY solutions for creating a better internal search experience
Jake Rocheleau wrote an extensive article about the possibilities of optimizing your WordPress internal search with simple plugins: WordPress Search: Useful plugins and snippets. One of the most popular plugins is Relevanssi, this to start with. You could also have a look at using Google Custom Search plugin. More advanced options are using SolrHQ Search for WordPress which requires a server with Solr installed or use a hosted version via their website. I’m a happy customer of Algolia.com which is capable of dealing with advanced setups and can also incorporate user behaviour data combined with their own pre defined options like typographical errors and proximity factors. With an easy drag-and-drop interface you can easily change the ranking algorithms. This is a good first step for optimising your internal search and because of their distributed network of servers, their interface is striking fast.
By suggesting, correcting typographical errors and predicting what a user is looking for, you can guide a visitor and create a better experience. On the end, it is all about suggestions that make sense, the speed in which the results are presented combined with a user friendly design that improves the conversion of the website.
