What’s the biggest threat for Google? Bing? Wolfram|Alpha? Facebook? Twitter? Or is it something completely different? It could be the accessibility of information on the internet. While Google is trying to index more existing (offline) information with programs where they digitize information from libraries all over the world, more new information that is created isn’t created within the ‘open’ world wide web. A new threat is forming for Google.
The internet consists of several resources of information, the world wide web is just one of them. Search engines basically try to crawl and index the world wide web, which mainly consists of a collection of HTML documents. And that’s exactly where Google reigns. But since 2000 the web is in decline, as the chart below shows. Other parts of the internet are becoming increasingly important sources for information. At the same time within the web there are always old and new obstacles that obstruct search engines form either crawling or indexing the published information.
One of the most important upcoming sources of information are mobile apps. Mobile apps have one major problem for web search engines: the information they contain are not (yet) crawlable. Basically the most mobile apps at this moment are fed by information that’s also available on the web. But once the choice is made for app-only web search engines lose the possibility to access the information.
Luckily there is a solution for this problem: the mobile web. The mobile web has however some major limitations towards mobile apps. The integration of device capabilities like GPS, camera etc. is more limited than for mobile apps. In performance it heavily depends on the possibilities of HTML and CSS. And here lies the solution for Google: with HTML5 and CSS3 these possibilities can compete with mobile apps. That’s why Google is a major contributor to the development of HTML5. In a Dutch magazine the recently stated “Developing an app? Rather build a good mobile site”. Research shows that Google could be lucky and the mobile web is likely to win the battle.
Flash and Silverlight applications
The development of HTML5 and CSS3 is a solution to another indexation problem for web search engines. Flash and Silverlight are web applications to make heavy graphic and interactive elements. They can be used within HTML documents and therefore be integrated in websites. The indexation of Flash and recently Silverlight has always been a problem for Google. With HTML5 and CSS3 there is more support on the way for graphic possibilities within your browser. The major benefit for Google is that it easily crawls and indexes this information.
Recently some publishers started working with pay walls for their online newspapers. The major contributor to this development is Rupert Murdoch. He already realized a pay wall for the sites of the Times and the Sunday Times. A Pay Wall basically would also mean that search engines wouldn’t be able to crawl this information. The ideas to introduce pay walls were known for a much longer time for parties like Google. That’s why they introduced the First Click Free program. This program allows users from search engines to find information behind pay walls and access this information for free the first time. For Google it guarantees access to information behind pay walls.
Communities are becoming increasingly important on the internet nowadays. With the major growth of social media communities are easily formed. Some communities are open others are closed, some are web based others are application based. Facebook even was a closed platform for search engines till 2006. Communities within social networks have privacy settings to be able to close them from the wide world. When closed they can’t be crawled.
Other communities are based on a specific platform or application. Take for example Ping, Apple’s social network focused on music. This platform is only accessible through the iTunes apps for iPhone, iPod etc., which aren’t crawlable for web search engines. It’s unlikely Apple will open up this platform for Google’s search engine. Rumors are Google is developing their own Music platform called Google Music which of course will be web based and therefore crawlable.
Other factors that could obstruct the crawling and indexation of the information on the internet include:
- AJAX (possible solution: make AJAX crawlable)
- (protected) status updates (possible solution: status update feeds)
- Non-web based platforms like Xbox Live, iTunes, etc.
More web or less web?
As you can see there are many factors threatening the information accessibility on the internet. Google seems to be anticipating greatly on this change and has converted it into a challenge more than a threat. While the web seems in decline the major question for the coming years will be: will we be using more web or less web? The battle between mobile web and mobile apps has just started. Mobile is the future, Google needs it to be open to retain their power on the world wide web. Alternatives for uncrawlable applications in Flash and Silverlight are in development. HTML5 is an important factor in these battles. Furthermore Google is trying to steer the web browser into the direction of an operating system with Chrome OS. Google is working hard to keep the web the center of the internet and to keep the web open and get entrances to the current and coming walls it contains. But will they succeed?