Content providers and webmasters started the optimization of websites for search engines during the middle of the1990s, when the initial search engines were making a catalog of the early Web. To begin with, all that was required of a webmaster was a page submission, or URL, to be sent to the different engines that then sent a spider to "crawl" the page, find and extract any links to it from other pages, then return the link information discovered there to be indexed. Basically, search engine spiders download a page and store it to the search engine's own server; there a second program, an indexer, extracts specific information regarding the page and its content, words within it, their location and any weight given to specific words, as well as all the links the page may contain that then get put into a scheduler for further crawls at some later date.
It wasn’t long before the value of highly ranked sites was recognized. Seeing a site returned in SERPs spurred site owners into exploiting the results with white hat and black hat SEO practices. According to Danny Sullivan, an industry analyst, search engine optimization was a spam message posted on Usenet on July 26, 1997, its earliest known usage.
In the beginning, search algorithms had to rely solely on information provided by the webmaster- like the keyword meta tag, or index files in engines such as ALIWEB. Meta tags then, were thought to provide the guide to content contained on each page, as they still do now but with more refinement. However, the early use of meta data for indexing pages proved unreliable as the webmaster's account of keywords contained within the meta tag didn’t always show true relevance to the actual keywords on the site. Inaccuracies, inconsistencies and incomplete data in meta tags meant pages got rankings for searches that were irrelevant. Many web content providers were quick to manipulate several attributes of the HTML source of a page in their efforts to improve their ranking in search engines.
Such reliance on factors controlled exclusively by a webmaster meant that the early search engines were abused and rankings were often contrived, manufactured or manipulated. So that users of search engines could have more pertinent and reliable results on the results pages, adaptations had to be made to close or at least tighten the existing loopholes that had permitted pages crammed with keywords, but unrelated to the perceived content, submitted by unscrupulous webmasters. Search engines are businesses too, meant to provide true, honest and relevant results to maintain integrity, and thereby popularity and success. It was feared their position would be otherwise jeopardized by the false returns on the ranking and the users would find other means of conducting the searches. The search engines developed more complex ranking algorithms, in response to the threat, thus making life more difficult for manipulating webmasters by adding further factors of determination to the process.
While at Stanford University, computer science graduate students Larry Page and Sergey Brin, founders of Google (initially in 1996), worked on the development of "backrub", based on the back links of one web page’s relationship to another. This search engine relied on a mathematical algorithm to rate the associated relevance of web pages through PageRank, which was able to quantify the strength of inbound links. PageRank approximated the probability of a given page being reached by a web user randomly surfing the web, and following links from one page to another, demonstrating that some links will be stronger than others, with higher PageRank pages more likely being reached by the random surfer.
Page and Brin founded Google and in 1998 it was incorporated as a privately held company. Google drew a loyal following from the increasing numbers of Internet users, appreciative of the simple design. Off-page factors (like PageRank and analysis of hyperlinks) were taken into consideration, as were on-page factors (keyword frequency, headings, site structure, links and meta tags) thus enabling Google to evade the sort of manipulation appearing in search engines only considering on-page factors for their rankings. PageRank was trickier to game, but webmasters were already in possession of schemes and link building tools that were capable of influencing the Inktomi search engine, devices that proved similar in application for gaming PageRank. So, many sites exchanged, bought, and sold links, frequently and with such magnitude, numerous schemes, or link farms, involving creating an abundance of sites solely for link spamming purposes. Recently though, major search engines give greater reliance to off-web factors like age, gender, location, and search history of users conducting searches for further refinement of results.
By 2007, search engines now included many more undisclosed factors within the ranking algorithms for minimizing link manipulation. Google claims that it ranks sites by employing over two hundred different signals. It’s no surprise that the three most popular search engines, Google, Yahoo and Microsoft's Live Search, will not reveal the algorithms they employ for ranking pages. Notable SEOs, such as Rand Fishkin, Barry Schwartz, Aaron Wall and Jill Whalen, have given in-depth study to various approaches involving search engine optimization, with their opinions published in online forums and blogs. SEO practitioners wishing to gain further insight into how algorithms function and what they do might investigate the patents that are held by various search engines.
