A search engine works in three stages: it crawls the web to find pages, indexes what each page says, then ranks the indexed pages it judges most relevant and useful for every search; AI answer engines add a fourth step that retrieves those pages and cites them inside a written answer. This guide follows a page through each stage at Google and Bing, shows what stops a page at each one, explains how AI Overviews, AI Mode, ChatGPT, Claude, Gemini and Perplexity choose the pages they cite, and corrects the search engine advice that no longer holds.
See how Aemorph gets businesses cited in AI answers
Originally published 20 February 2024 in the SEO section of aemorph.com. Updated October 2026.
What Is a Search Engine?
A search engine is a service that discovers pages across the web, stores what they say in an index and answers each query with the indexed pages it ranks highest for that query. A browser such as Chrome or Safari displays web pages; a search engine is the service the browser sends a query to. Google held 89.94% of search engine market share worldwide in September 2026, so most of this guide follows Google's documentation, with Bing's where it differs. For what search engine optimisation covers as a discipline, read what is SEO.
| Rank | Search engine | Worldwide share, September 2026 | What the figure measures | Source |
|---|---|---|---|---|
| 1 | 89.94% | Share of search engine use, all devices | StatCounter Global Stats | |
| 2 | Bing | 5.29% | Share of search engine use, all devices | StatCounter Global Stats |
| 3 | Yahoo! | 1.36% | Share of search engine use, all devices | StatCounter Global Stats |
| 4 | Yandex | 1.3% | Share of search engine use, all devices | StatCounter Global Stats |
| 5 | DuckDuckGo | 0.74% | Share of search engine use, all devices | StatCounter Global Stats |
| 6 | Baidu | 0.44% | Share of search engine use, all devices | StatCounter Global Stats |
Search engine market share worldwide, September 2026. The measure is share of use, not a count of searches, and it excludes AI chats.
Source: StatCounter, September 2026
How Do Search Engines Work, Stage by Stage?
Google describes Search in three stages, crawling, indexing and serving results, and states that not every page makes it through each stage. Each stage has its own failure point, so a page missing from the results has stopped at one of them. AI answers sit on top of the same index: Google's AI features link only to pages that passed all three stages.
| Stage | What happens | What Google states | What stops a page | Where to check |
|---|---|---|---|---|
| 1. Crawling | Googlebot downloads text, images and video from pages it has found | Pages are found through links from known pages and through submitted sitemaps | Blocked in robots.txt, server errors, no links pointing to the page | URL Inspection in Search Console |
| 2. Rendering | Google runs the page's JavaScript in a recent version of Chrome | Rendering happens during the crawl | Content that appears only after scripts fail | URL Inspection live test |
| 3. Indexing | Google analyses the content, groups duplicates and picks a canonical page | Indexing is not guaranteed for every page processed | A noindex rule, thin or duplicate content, a different page chosen as canonical | URL Inspection indexed result |
| 4. Serving | Google returns the indexed pages it judges most relevant and useful | Relevance depends on hundreds of factors, including location, language and device | Stronger pages for the same query | Search Console performance data |
| 5. AI answer | AI Overviews and AI Mode cite indexed pages as supporting links | A supporting link has to be indexed and eligible for a snippet | Any failure in stages 1 to 4, or snippet controls that block it | Search results for the query |
How a page moves through Google Search, from discovery to an AI answer.
Source: Google Search Central, December 2025
How Does Crawling Work?
Crawling is the stage where an automated program, Googlebot at Google and Bingbot at Bing, fetches pages it has found through links and sitemaps and downloads their content. Googlebot decides which sites to crawl, how often and how many pages to fetch, and it slows down when a server struggles. Google states that it does not accept payment to crawl a site more often or to rank it higher. During the crawl Google renders each page with a recent version of Chrome, so content that depends on JavaScript is seen only if those scripts run.
How do you get a page crawled and indexed faster?
Link to the page from pages Google already crawls, list it in an XML sitemap with an accurate last-modified date, and use Search Console's URL Inspection tool to request indexing for the pages that matter most. Google's Search Console Help states that a request does not guarantee the page enters the index, and that each property has a daily limit on requests. Bing reads the lastmod field in a sitemap as a key signal for which URLs to recrawl, and its IndexNow protocol notifies Bing and other participating engines the moment a page changes. Google does not use IndexNow.
| Method | Engine | What it does | Limit | Source |
|---|---|---|---|---|
| Internal links | Google and Bing | Lets the crawler follow a path from known pages to the new page | Links have to be standard HTML links | Google Search Central |
| XML sitemap | Google and Bing | Lists the URLs you want crawled, with last-modified dates | Each date has to match the real last change to the page | Google Search Central; Bing Webmaster Blog |
| URL Inspection, request indexing | Queues one URL for crawling after a quick check | No guarantee of indexing; a daily request limit | Search Console Help | |
| IndexNow | Bing and other participating engines | Notifies engines of a new, changed or deleted URL straight away | Google does not take part | Bing Webmaster Blog |
| robots.txt | All major crawlers | Tells crawlers which paths not to fetch | Controls crawling, not indexing | Google Search Central |
Ways to get pages discovered and recrawled, and the limit on each.
Sources: Google Search Central, October 2026 and others
How Does Indexing Work?
Indexing is the stage where the search engine analyses a crawled page, decides whether it duplicates another page, chooses one canonical version to show and stores what the page says. Google groups similar pages into a cluster and picks the most representative one as the canonical; the others in the cluster stay out of most results. Google states that indexing is not guaranteed for every page it processes: low-quality content, a robots meta rule that disallows indexing, and a duplicate that loses canonical selection are common reasons a crawled page is not indexed or not shown.
Does robots.txt keep a page out of Google?
No: Google states that robots.txt manages crawler traffic and is not a mechanism for keeping a page out of Google, and a blocked URL still appears in results without a description when other pages link to it. To keep a page out of the index, add a noindex rule or put the page behind a password. A noindex rule only works when robots.txt does not block the page, because Googlebot has to fetch the page to read the rule.
How Do Search Engines Rank Pages?
Google ranks with a set of automated ranking systems that weigh many signals across hundreds of billions of pages, and its published guide names the systems that do most of the work, from language understanding to link analysis, freshness and spam detection. No single system decides a ranking. Google's page experience guidance adds that Core Web Vitals are used by its ranking systems, while good scores alone do not guarantee a top position.
| Ranking system | What Google says it does | Type | What it means for your page | Status |
|---|---|---|---|---|
| BERT, neural matching and RankBrain | Understand how words relate to concepts and to the intent behind a query | AI language systems | Pages rank for meaning, not only for exact keywords | Current |
| Link analysis systems and PageRank | Use how pages link to each other to judge what they are about and how helpful they are | Link signals | Links from relevant, trusted pages count | Current |
| Freshness systems | Show newer content where a query expects it | Query type | Recency matters for news and fast-moving topics, not for every query | Current |
| Reviews system | Rewards high-quality reviews with original analysis | Content quality | Product and service reviews need first-hand detail | Current |
| Site diversity system | Generally limits the top results to two listings from one site | Results layout | Several weak pages from one site do not fill the first page | Current |
| Spam detection systems, including SpamBrain | Filter content and behaviour that breaks Google's spam policies | Spam | Link schemes and scaled low-value content lose visibility | Current |
| Helpful content system | Merged into Google's core ranking systems in March 2024 | Content quality | Helpfulness is now judged inside core ranking | Retired as a separate system |
Named Google ranking systems from Google's guide, last updated 10 December 2025.
Sources: Google Search Central, December 2025 and others
Do search engines personalise results?
Google tailors results to context it reads from each search, and its documentation names the user's location, language and device among the factors that decide what it serves. A search for a service near you returns local businesses; the same words searched from another country return different ones. Track rankings by market and device for that reason, as the keyword rankings guide explains.
How Do Google's AI Overviews and AI Mode Choose Their Sources?
AI Overviews and AI Mode build an answer by running several related searches across subtopics, a technique Google calls query fan-out, and link to pages from Google's index as supporting sources. Google states that a supporting link has to be indexed and eligible to appear with a snippet, and that there are no additional technical requirements. The same snippet controls, nosnippet, data-nosnippet, max-snippet and noindex, limit what Google shows from a page in these features. Google reported in July 2026 that AI Mode had passed 1 billion monthly active users. Aemorph's guide to featured snippets explains how Google selects them and what the snippet controls limit.
Google-Extended, a separate robots.txt token, controls whether Google uses a site's content to train future Gemini models and to ground answers in the Gemini apps and Vertex AI. Google states that Google-Extended does not affect a site's inclusion or ranking in Google Search. Earning a place in these Google answer surfaces is the work of AEO, set out in the AEO guide.
Sources: Google Search Central, December 2025 and others
Case study · Orthodontics, Australia
Smile Team is cited in Google AI Overviews for 223 Australian search terms.
Ahrefs organic keyword data for Australia, June 2026, records Google AI Overviews citing Smile Team for 223 search terms with 22,650 combined monthly searches, led by "retainer" at 5,500. Read the Smile Team case study
Google AI Overview citations, not citations inside ChatGPT, Claude, Gemini or Perplexity.
How Do ChatGPT, Claude, Gemini and Perplexity Find and Cite Pages?
AI chats with live search send their own crawlers to fetch pages: OpenAI, Anthropic and Perplexity each document a search crawler and a separate fetcher for pages a user asks about, and OpenAI and Anthropic also run a training crawler with its own robots.txt name. OpenAI states that sites opted out of OAI-SearchBot are not shown in ChatGPT search answers, though they still appear as navigational links. Anthropic states that blocking Claude-SearchBot reduces a site's visibility in Claude's search results, and Perplexity states that PerplexityBot surfaces and links sites in its results and is not used to train AI models. Blocking a training crawler and blocking a search crawler are separate decisions.
| Company and chat | Search crawler | Training crawler | User-triggered fetcher | What blocking the search crawler does |
|---|---|---|---|---|
| OpenAI, ChatGPT | OAI-SearchBot | GPTBot | ChatGPT-User, where robots.txt rules do not always apply | The site is left out of ChatGPT search answers, apart from navigational links |
| Anthropic, Claude | Claude-SearchBot | ClaudeBot | Claude-User; all three follow robots.txt | Lower visibility in Claude's search results |
| Perplexity | PerplexityBot | None; PerplexityBot does not crawl for model training | Perplexity-User, which generally ignores robots.txt | The site drops out of Perplexity's linked results |
| Google, Gemini | Googlebot, for Google Search | Google-Extended token for Gemini training | Not documented separately | Blocking Googlebot stops Google crawling the content; a known URL still appears without a description, so removal needs a crawlable noindex |
AI crawlers and fetchers as documented by each company, checked 29 October 2026.
Sources: OpenAI and others, 2026
ChatGPT passed 900 million weekly active users in February 2026, according to OpenAI. A chat answer usually names a short list of businesses with no ranked links beside it, so the businesses missing from that list lose the enquiry before a search results page loads. Earning those citations is GEO, Aemorph's main discipline; the GEO guide explains the method and the AI SEO guide shows how SEO, AEO and GEO run together. In a logged-out ChatGPT session on 4 August 2026, ChatGPT cited Aemorph 3 to 4 times and named it as a source for agency queries; that is one dated observation of one engine.
Sources: Google and OpenAI, July 2026 and others
Which Search Engine Advice Is Out of Date?
Several search engine rules still repeated online, including parts of the 2024 versions of this guide, describe retired tools or claims Google's own documentation contradicts. The table checks each one against Google, Bing and StatCounter.
| Earlier advice | What the documentation says | Source | What to do instead | Status |
|---|---|---|---|---|
| Submit URLs with "Fetch as Google" | Google replaced Fetch as Google with the URL Inspection tool in January 2019 | Google Search Central blog, 25 January 2019 | Use URL Inspection and request indexing for priority pages | Corrected |
| Use robots.txt to choose which parts of a site get indexed | robots.txt manages crawling and is not a mechanism for keeping a page out of Google | Google Search Central, robots.txt introduction | Use noindex or password protection to keep a page out | Corrected |
| Submit the site to each search engine | Google finds pages through links and sitemaps and does not accept payment to crawl or rank | Google Search Central, How Search works | Link to new pages, keep the sitemap current, use IndexNow for Bing | Corrected |
| Search engines organise pages by keywords | Google's ranking systems include AI language systems that relate words to concepts | Google ranking systems guide | Answer the question in plain language rather than repeating keywords | Corrected |
| Google handles more than 5 billion searches a day and over 90% of share (August 2019); a search returns 1.43 billion results in 0.69 seconds | No current primary source supports these figures | StatCounter, September 2026 | Use dated market share: Google 89.94% worldwide | Replaced |
| Bing trails only Google and Baidu | Bing ranked second worldwide in September 2026 at 5.29%, with Baidu at 0.44% | StatCounter, September 2026 | Treat Bing as the second engine for crawl and index checks | Corrected |
| Internet Archive and Ask.com listed among the main search engines | Neither appears in StatCounter's list of engines with measurable share | StatCounter, September 2026 | Focus on Google, then Bing | Removed |
Advice from the 2024 versions of this guide, checked against current documentation in October 2026.
Sources: Google Search Central, January 2019 and others
Make Every Page Easy to Crawl, Index, Rank and Cite
A page reaches a Google result only when it is crawlable, indexed and ranked for a query, and a supporting link in AI Overviews or AI Mode only when it is indexed and eligible to show with a snippet, while ChatGPT, Claude and Perplexity reach pages through their own search crawlers and user fetches, so checking each stage in order finds the stage where a page stops. Link every important page from pages already indexed, keep the sitemap accurate, use noindex rather than robots.txt to keep pages out, write in the plain language Google's ranking systems read, and allow OAI-SearchBot, Claude-SearchBot and PerplexityBot where AI citation matters. Aemorph's technical SEO work checks each stage for your site, and its GEO work earns the citations inside the AI chats your buyers use.