Archie to Perplexity: The evolution of search engines and the rise of reasoning engines

By Rohan Chincholi

Search has always been integral to satisfying human curiosity, serving as a cornerstone of our evolutionary journey. It is a trait deeply embedded in us — the drive to seek, question, and uncover information. Throughout history, our search for knowledge has been the foundation for discovery and advancement.

In the digital age, search engines have become the modern-day tool that bridges curiosity with information. When we think of search today, we immediately think of Google. It has become synonymous with the act of searching itself. The technology that powers search engines allows us to instantly access vast repositories of information, continuing to fuel human curiosity and drive progress.

When we talk about the history of search engines, it's essential to remember "Archie," the world's first internet search engine, created in 1990 by Alan Emtage, a visionary student at McGill University in Montreal. Originally named after the word "archives," it was shortened to Archie, laying the groundwork for digital information retrieval.

By 1994, Yahoo, Webcrawler, and Lycos had taken the stage, each contributing to the evolving landscape of search. Following them, Altavista and Ask Jeeves emerged in 1997, offering new ways to navigate the growing internet. However, it was in 1998 that Google was launched, transforming search with its advanced algorithms and clean user interface, quickly becoming synonymous with the act of searching itself.

The early 2000s saw the introduction of MSN Search (now Bing), further diversifying the search ecosystem. Over time, Google remained dominant, continuously innovating and becoming the gateway to the world's information.

In the early days of search engines, platforms like Ask Jeeves and AltaVista relied heavily on algorithms that prioritized quantity over quality. As a result, consumers often found the accuracy of search results to be abysmally low, with relevant results hovering around just 1% or less. These search engines lacked the sophistication needed to deliver up-to-date, pertinent information to users.

Google, however, revolutionized this landscape. Unlike its predecessors, Google employed "spiders"—automated programs designed to crawl the web, discovering new pages and identifying changes on existing ones. This approach allowed Google to serve users with timely and relevant search results, a feat unmatched by any other search engine at the time.

Rather than relying on editors or researchers to manually curate links, Google developed algorithms that evaluated web content based on specific criteria. This included innovative concepts such as inbound links from trusted sources (an early precursor to what we now consider authority in SEO) in addition to more traditional metrics like keyword frequency and page titles. These components were factored into a sophisticated scoring system known as PageRank, which determined where a website would appear for any given search query. It allowed Google to quickly outshine its competitors, setting the standard for search engines and cementing its place as the go-to platform for information retrieval.

Today, performance has become synonymous with Google Search, a platform that continues to set the benchmark for delivering results at scale. This is clearly evident from the company's consistent growth in financial performance.

Alphabet Inc., Google's parent company, recently released its second-quarter 2024 financial results, showcasing the remarkable strength of its core search business.

•Google Search saw a 14% year-over-year increase in revenue, reinforcing its dominant role in the digital economy.

•Google Search revenue for Q2 2024: $48.5 billion (up from $42.6 billion in Q2 2023)

(reference : Alphabet Inc., Google’s parent company, released its second quarter 2024 )

This growth underscores how Google has managed to maintain its leadership in search by continually refining its algorithms and expanding its advertising ecosystem.

Today, Google holds a near-monopoly in the search market, commanding over 90% of global market share as of February 2024, according to Statcounter GlobalStats. Its closest competitor, Bing, trails far behind at around 3.5%, with DuckDuckGo accounting for less than 1%. Yet, the presence of DuckDuckGo is significant.

Launched in 2008, DuckDuckGo quickly distinguished itself by taking a unique, privacy-first approach to search. In a world where Google’s business model revolves around user profiling and targeted ads, DuckDuckGo's algorithm focused on not collecting personal data, offering users the same search results regardless of who they are, and refusing to engage in social engineering with user data. This no-ads, no-tracking model resonated with privacy-conscious users, helping DuckDuckGo secure a place among the top five search engines, despite its relatively small market share. Notably, while DuckDuckGo has its own crawler, it is also supported by Bing’s infrastructure.

Google, meanwhile, has made strategic investments to maintain its dominance. In 2022, it is said to have paid Apple around $20 billion to remain the default search engine on Apple devices—this amounted to 17.5% of Apple’s operating income and a 36% revenue share for Apple. In addition, Google spent another $10 billion on deals with other companies, further solidifying its position. (reference : Google Google during its ongoing antitrust dispute with the United States Department of Justice (DoJ), according to Bloomberg.

Despite Google's massive influence, it continues to face regulatory challenges. Most recently, on September 9, 2024, Google lost its appeal against a $2.7 billion antitrust fine imposed by the European Union, a ruling that underscores the growing global scrutiny of its business practices and market dominance.

Today, consumers remain eager for information and are willing to overlook ads or SEO implications as long as they get the answers they seek. Despite this, the process of sifting through pages to find the right words, images, products, videos, and services persists. Throughout generations and technological advancements, the essence of search has always been to provide answers. The main evolution has been in reducing the time it takes to access relevant information.

The transition over time:

•Search Engines’ Early Focus: Prioritized volume of results over accuracy.

•Google’s Innovation: Customized algorithms for personalized search results.

•DuckDuckGo’s Approach: Emphasized privacy, ad-free experience, and uniform search results.

We are now entering an era dominated not by traditional search engines, but by answer engines. The nature of search has fundamentally evolved—from finding a book in the library to having a personal analyst at work. This new wave of technology can be seen as the rise of reasoning engines.

While search engines excelled at discovering information, they relied heavily on constantly crawling the web to update their datasets. This approach was effective but struggled to comprehend complex queries. Answer engines, on the other hand, are designed to understand intricate strings of words and, more importantly, learn and adapt over time. This allows them to not only provide more accurate responses but also to improve as they accumulate data, making them particularly useful for delivering balanced, comprehensive insights.

A prime example of this shift is Perplexity AI, founded less than two years ago. Despite its youth, the company is already valued at $1 billion. Perplexity AI’s rapid rise illustrates the growing demand for AI-driven technologies capable of intelligent reasoning, capable of understanding nuanced questions and generating tailored, well-collated answers.

This transformation from search to reasoning engines signals a major leap forward in how we interact with information—no longer are we just searching; now, we are conversing with systems that can analyse, reason, and respond with human-like precision.

• As per industry reports the MAU for Perplexity AI stands at 10 MN as of January 2024

• Perplexity AI also reported over 500 million requests served in 2023 with minimal marketing expenses (by Perplexity AI & Industry Sources)

Perplexity AI doesn't just scrape the web for isolated bits of information. Instead, it interprets complex queries, considers their broader context, and then formulates comprehensive, well-rounded responses. This allows users to ask more intricate, multi-layered questions and receive answers that feel personalized and insightful—making the experience less like searching and more like conversing with an expert.

Perplexity AI operates with three distinct models to cater to various user needs:

1. Standard Plan:

Offers unlimited searches, 5 Pro searches, access to the Standard AI model, and the personalization of answers.

2. Professional Plan ($20/month):

Builds on the Standard Plan by providing additional features such as unlimited Pro searches, the ability to choose between advanced AI models, options to upload and analyze files, and tools to visualize data.

3. Enterprise Plan ($40/month):

Recently announced in a Bloomberg report, the Enterprise version is aimed at large-scale business users, with a focus on enhanced security capabilities and data protection measures. This shift underscores Perplexity’s goal to boost revenue through its enterprise offering, as organizations increasingly demand privacy safeguards for their data.

Perplexity AI’s business model primarily revolves around subscriptions, which form the foundation of its monetization strategy. However, the company sees significant potential in the enterprise segment, expecting that B2B revenue may surpass B2C, as enterprises look to leverage its AI for both data security and operational efficiency.

From a consumer standpoint, Perplexity AI is also exploring new revenue streams, including display promotions within its search engine results. These promotions would appear below the answers generated by Perplexity's AI, offering advertisers a new way to reach audiences. To incentivize content creators, Perplexity plans to implement a revenue-sharing model, where participating publishers would receive a flat fee when their content generates ad revenue. Additionally, participating news publications may receive a 1-year paid subscription to Perplexity’s premium version as part of the partnership.

Looking forward, technology integration will be paramount to Perplexity’s growth. The platform will help facilitate integrations that allow businesses and individuals to seamlessly incorporate Perplexity AI’s capabilities into their workflows, thereby expanding its use cases across industries.

As consumer adoption of answer engines rises, we are witnessing the dawn of a new era for SEO. The focus is shifting from traditional search optimization to curating content specifically for AI-powered answer engines. Brands will increasingly invest in what can be called AI Engine Optimization (AEO), where content is tailored not just to appear in search results but to be understood and utilized by advanced AI models.

Brands will continue to prioritize strategies aimed at increasing visibility and authority.

Here are some key areas that will see growth:

1. High Domain Authority Pages:

Brands will spend heavily on building and maintaining high-authority content. Since AI-driven engines rely on authoritative sources to generate answers, establishing a strong presence with well-regarded, credible content will be crucial for brands to maintain visibility.

2. Online PR:

The rise of answer engines will give online PR a new boost. Brands will need to secure media coverage on reputable platforms to gain external validation and enhance their domain authority.

3. Content Tailored to Brand Narrative:

AI models like Perplexity, ChatGPT, or Claude generate responses based on contextual understanding. Therefore, brands will focus on creating narrative-driven content that aligns with their identity, ensuring that the answers produced by AI reflect their messaging and values.

4. On-Page SEO:

Traditional on-page SEO efforts like technical SEO, link-building, and optimizing for site speed and mobile-friendliness will remain essential. These elements will still influence the AI’s ability to assess content quality and relevance.

5. Off-Page SEO:

Backlink strategies, guest posts, and influencer collaborations will continue to grow, as off-page factors remain critical in boosting a site’s domain authority and ensuring that its content gets surfaced by answer engines.

From its beginnings with Archie to its dominance with Google, the evolution of search engines mirrors humanity's relentless pursuit of knowledge and technological progress. Google's success highlights its role as a benchmark in the search landscape, while the rise of privacy-focused DuckDuckGo underscores shifting user preferences.

The transition from traditional search engines to advanced answer engines, like Perplexity AI, represents a significant leap forward, offering more nuanced and personalized responses. As we move into this new era, the focus will shift towards AI Engine Optimization (AEO), where brands must tailor content to engage effectively with sophisticated AI models and ensure visibility in an increasingly complex digital environment.

Rohan Chincholi is the chief digital officer of Havas Media India.