Prior to web search engines there was only a list of all the webservers, its entirety edited by Tim Berners-Lee and hosted on the CERN webserver. One historical snapshot from 1992 remains As the number of webservers coming online continued to grow, the central list was unable to keep up with the influx. On NCSA’s site where new servers had been announced under "What's New!" there no longer existed a definitive complete listing.
The original Internet (pre-web) search tool was Archie, which stands for "archive" with the "v" omitted. A Montreal student at McGill University, Alan Emtage, created Archie in 1990. ‘Archie’ downloaded all the directory listings of files to be found on public anonymous FTP (File Transfer Protocol) sites, thus creating a file names database capable of being searched; however, their content (whether the program was incapable of such or whether it was not a consideration at the time) was never indexed by Archie.
Gopher was created in 1991 by Mark McCahill at the University of Minnesota and it led to the development of two new search programs called Veronica and Jughead and like Archie, they searched file names and titles stored in Gopher’s index systems. Veronica, the acronym: Very Easy Rodent-Oriented Net-wide Index to Computerized Archives provided a keyword search of most Gopher menu titles in the entire Gopher listings. Jughead, the acronym: Jonzy's Universal Gopher Hierarchy Excavation And Display was a tool for getting menu information from specific Gopher servers. Though the search engine’s name, "Archie" was not a reference to the Archie comic book series, "Veronica" and "Jughead" directly are, thus referencing their predecessor - and ‘gopher’ though an actual member of the rodent family is commonly used for its homonym (sound-alike) qualities – ‘go-fer/go-for’, thus relating it directly to search functions as referenced in the acronyms Veronica and Jughead.
Wanderer was a web crawler developed by Matthew Gray at MIT for measuring the size of the World Wide Web. The first Web search engine was an index collected in 1993 by the World Wide Web Wanderer called 'Wandex'. In the same year another early search engine, Aliweb, appeared. JumpStation (released in early 1994) employed a crawler for searching web pages, but its search function was limited to web page titles only. 1994 saw WebCrawler, one of the original "full text" crawler-based search engines, come into use and unlike its predecessors users could search for any word in any webpage, the standard for all major search engines since. WebCrawler, a metasearch engine created by Brian Pinkerton, was the first to become widely recognized by the public, possibly due in part to its iconic spider that reinforced the name. 1994 also witnessed the launch of Lycos, which began at Carnegie Mellon University, and went on to become a major successful commercial enterprise.
With the technical and technological templates now available from the research that was beginning to come out of the universities, the field of commercial search engines expanded with many appearing to vie for public popularity; Magellan, Excite, Infoseek, Inktomi, Northern Light, and AltaVista making their debuts on the scene. Yahoo! was a very popular way for users to locate web pages that were of interest to them, however, its search function was limited to its web directory, which didn’t offer full-text copies of web pages though information seekers were also able to browse the directory as an alternative to doing searches based on a keyword.
In 1996, at the height of its market domination, Netscape was offering a deal for a single search engine to become their exclusively featured search engine. Such was the overwhelming interest generated that an alternative deal was struck instead with Netscape where five of the major search engines, Yahoo!, Magellan, Lycos, Infoseek and Excite, for an annual fee of $5Million, would each be in rotation on Netscape’s search engine page.
Search engines were perceived to be among the brightest stars of the Internet investment speculation frenzy of the late 1990s. Many companies had entered the market dramatically, with record gains achieved during their initial public offerings. Some have withdrawn their public search engine, like Northern Light and are marketing enterprise-only editions. During the dot-com bubble, a market boom driven by speculators that reached its peak during 1999 and ended spectacularly in 2001, many of the search engine companies lost their luster and appeal and suffered badly.
The Google search engine had risen to distinction by 2000[citation needed] and the company was achieving better results on its search engine for many searches with the innovative technology it had developed at Stanford University for placing importance on pages and web sites with PageRank. Originally, PageRank (PR) was a major ranking factor but is now one of hundreds in the algorithm determining page rankings. This iterative algorithm gives rank to web pages based primarily on the PageRank and number of other pages and web sites that link there, the premise being pages that are good or desirable (content relevant) have more links to pages than others. Success for Google may also have been through its maintaining a minimalist interface to its search engine in contrast to its many competitors who embedded a search engine in a web portal.
Yahoo!, by 2002, was also providing search services that were based on the search engine at Inktomi. Yahoo!’s acquisition of Inktomi in 2002 followed by Overture (which owned AlltheWeb and AltaVista) in 2003 provided the necessary technologies for it to launch its own search engine in 2004 after having ‘camped out’ temporarily at Google's search engine in the intervening two years.
MSN Search (since re-branded Live Search) was first launched by Microsoft in late 1998 when it used search results from Inktomi. The site then began to display listings from Looksmart blended with results from Inktomi in early 1999, except for a short interim period when AltaVista results were used instead. By 2004, Microsoft had begun a transition to its own search technology, powered by its own web crawler (called msnbot).
As of late 2007, Google was by far the most popular Web search engine worldwide.
A number of country-specific search engine companies have become prominent; for example Baidu is the most popular search engine in the People's Republic of China and guruji.com in India.