Search engine

The early internet was a labyrinth without a map. FTP sites, Gopher servers, and eventually web pages proliferated, but finding anything required knowing exactly where to look. The problem was fundamental: as networked information grew exponentially, human-curated directories couldn't keep pace.

Archie, created by Alan Emtage at McGill University in 1990, was the first search engine—a tool that indexed FTP archives and let users search filenames. Veronica and Jughead followed for Gopher content. But the web's explosion after 1993 created a problem of unprecedented scale. Web crawlers—programs that automatically followed links and indexed content—became essential.

The adjacent possible for web search required several streams to converge. The internet itself had to reach sufficient scale to contain findable information. Web protocols (HTTP, HTML) provided the addressable, linkable structure that made crawling possible. Storage and processing costs had to fall enough to index billions of pages. Information retrieval research, dating back decades, provided the algorithms for ranking relevance.

Early commercial search engines emerged rapidly: WebCrawler (1994), Lycos (1994), AltaVista (1995), Excite (1995), Yahoo (originally a directory, adding search). Each approached the ranking problem differently, primarily through keyword matching and metadata analysis. The results were often poor—easily manipulated by site owners stuffing pages with keywords.

Google's PageRank algorithm, developed by Larry Page and Sergey Brin at Stanford (1996-1998), transformed search by using link structure as a quality signal. Pages linked by many other pages ranked higher; pages linked by important pages ranked even higher. This recursive insight—treating the web's link graph as a voting system—produced dramatically better results than keyword-based approaches.

Geographic concentration was remarkable. Search development clustered around Stanford and the Bay Area: Google, Yahoo, and Excite all emerged from the university or nearby. Carnegie Mellon contributed Lycos. AltaVista came from Digital Equipment Corporation's Palo Alto research lab. The academic-startup pipeline that characterized Silicon Valley was particularly visible in search.

The cascade effects reshaped the internet economy. Advertising followed attention; Google's AdWords (2000) created the economic engine that made free search sustainable. Websites optimized for search visibility. Information discovery shifted from curation to algorithmic retrieval. The search box became the default entry point to the internet.

By 2025, Google dominated global search with 90%+ market share in most countries. Bing persisted as a distant second. Chinese search operated separately behind the Great Firewall. The integration of large language models into search—starting with AI-generated answers rather than lists of links—promised the next transformation, potentially as significant as PageRank had been.