The Real Time Web
The Internet has seen a lot of slow evolution over the last 10 years that has changed the way people live and work. In 2009 the Internet is poised to undergo its biggest transformation yet as the Real Time Web emerges. There are a lot of indicators that make me feel that we are on the verge of seeing the next big thing.
Most people will acknowledge that their activity on the Internet has changed significantly since they first went online. The amount of data that the average user has to sift through nowadays would perplex the average user ten years ago. For every one email that you send out, three more seem to come in. Compare this to the early days of AOL when most people remember hearing, “You’ve Got Mail!” whenever a new item was added to their inbox. Imagine how annoying that would be nowadays with the constant stream of information being fed into our computer.
For simplicity I am going to break down the Internet History into only two chapters. The first chapter is the race to build the perfect search engine and the second chapter will be the race to build a tool to organize the real time web.
Although it wasn’t first to market, Google perfected search and is by far the largest Internet company. Google uses a system of spiders to crawl the Internet in an effort to create an index of every website and all of its content. The only problem is this process can take several days/weeks/months before Google even finds a new website. This is OK because first generation websites can afford this delay. However, during the Obama Inauguration, Twitter received almost five thousand tweets per minute. This infusion of data onto the Internet is very powerful because it allows people to collaborate in ways never before possible. Developers can now leverage data streams of real time content into their websites/applications.
The movement which is afoot on the Internet today involves social websites such as Twitter, Tumblr, Posterous and Facebook which can all be traced back to blogging. The introduction of blogs and content management systems allowed for users to very easily create and edit their own content. This freedom resulted in the rapid growth of content on the web. Websites were being built by the thousands daily, which all led to Google’s current dominance. Google became so powerful because it was able to organize the content in a way that created value for the end user. Blogs unfortunately cannot keep up at the pace to which people wanted to create, so a new platform had to be introduced. Microblogging emerged as a powerful platform due to the fact that you didn’t need to be a great writer in order to generate content that people were interested in. It also allows for content to be created within seconds.
Google currently does not have a system or method of indexing such rapid creation of content. Don’t be tricked into thinking this realtime social data is not valuable based on the fact that Google is not indexing it fast enough. I would argue that it is perhaps more valuable because of the new search parameter of “Time”. Googles current algorithm to process PageRank is outdated in the sense that it cannot process pagerank fast enough. In order for a website to gain popularity and relevance in Google search results, other sources are required to write about and link to the original website. This is how PageRank is established and could also be Googles Achilles heal in their effort to index the Real Time Web. PageRank is not efficient when it comes to indexing breaking news on the scale that a social site would be. During the Mumbai attacks in India, I did a test and compared a Google Search for the keyword, “Mumbai Attacks” against a Twitter search for the keyword, “Mumbai Attacks”. I was able to get relevant real time results on Twitter that were few and far between on Google. I also went to see if The Drudge Report had any news on its homepage about the breaking news. The news broke on Twitter a good 30-40 minutes before I saw any posts on Drudge. This is ironic because Drudge came to fame in the late 90’s when it broke the Monica Lewinsky/Clinton story faster than any conventional news outlet. This gave the Internet respect when it came to breaking news. Less than 10 years later, static news sites are having a hard time keeping up with the Real Time Web. Twitter has a more collaborative infrastructure that doesn’t rely on one author which allows it to grow uncover stories quickly. There were thousands of people breaking the story compared to one or two writers covering a story at a newspaper. Crowd-sourcing the conversation and empowering civilian journalists is the what allows the news break so quickly. There is no way the old establishment can compete.
The web is flooded with new content every second and there is currently no way of it being indexed by the most powerful search company (Google). This is a huge problem as well as a huge opportunity for the modern entrepreneur. Companies like Twitter are more similar to infrastructure of previous generations that helped carry content (ie: AM/FM Radio) They are the infrastructure to which companies will be built upon. Using the data that people are feeding into the web via these tools will foster all new types of websites that could not be theorized even 5 years ago. It will be exciting to see the creative ways that the Real Time Web will influence the next generation of Internet Startup.