>
Home / Guides / How Each AI Engine Picks Its Sources

How Each AI Engine Picks Its Sources

How do AI search engines choose sources, and what determines whether a website appears in an AI-generated answer?

ChatGPT, Perplexity, Gemini and Google AI Overviews all retrieve information and display links to supporting sources. However, their documentation reveals important differences in how they access website content, manage crawlers and present citations.

None of these four experiences publishes a complete explanation of how individual sources are selected. OpenAI says ChatGPT search results are ranked using multiple factors that it does not disclose. Perplexity describes its sources as “top-tier” without publishing selection criteria. Google documents eligibility and retrieval mechanisms for Gemini and AI Overviews, but not their complete source-selection processes.

What these companies do publish is still useful. Their documentation explains crawler access, website-owner controls, retrieval mechanisms and how citations appear.

This guide compares what is publicly documented across the four platforms, identifies what remains unknown and explains how website owners can use that information to improve their AI search visibility.

How AI search engines choose sources: crawler controls by engine

This comparison is part of our wider work on AI search optimization. Our GEO services cover visibility across multiple AI platforms. We have a commercial interest in this subject, so we encourage readers to check the original vendor documentation linked throughout this article.

Key Takeaways

  • Each platform documents ways to access or restrict website content, but none publishes a complete list of citation-ranking factors.
  • ChatGPT, Perplexity and Google use different crawlers, tokens and controls. One robots.txt rule does not cover every platform.
  • Perplexity-User generally ignores robots.txt because it fetches content in response to a user request.
  • Google AI Overviews citations do not depend exclusively on appearing in the traditional top 10. An Ahrefs study found that approximately 37.9% of cited URLs also ranked in the top 10.
  • Website owners should check crawler access separately for each platform and test the same relevant prompts across engines.

In this guide: The comparison | Where the engines match | Where they differ | What no engine publishes | Robots.txt checklist | Research findings | Practical steps | FAQs

The Comparison: How Four AI Search Experiences Access and Cite Sources

The table below brings together information published by OpenAI, Perplexity and Google. Each row describes documented behavior. Where the documentation does not provide an answer, the table says so rather than assuming how the platform works.

Comparison areaChatGPT SearchPerplexityGemini appGoogle AI Overviews
Crawler or controlOAI-SearchBot for search, GPTBot for training and ChatGPT-User for user-initiated actionsPerplexityBot for search results and Perplexity-User for user requestsGoogle-Extended robots.txt token for training and grounding; crawling uses existing Google user agentsStandard Google Search controls, including indexing and snippet settings, plus the Search generative AI setting in Search Console
Documented eligibility or accessAllow OAI-SearchBot and ensure the hosting provider permits OpenAI’s published IP addressesPerplexity recommends allowing PerplexityBotContent from the Google Search index can be supplied to Gemini at prompt time for groundingPage must be indexed and eligible to appear with a snippet, and the site must be included in Search generative AI features
How it finds contentDecides per prompt whether to search the web and may rewrite a question into multiple searchesSearches the internet when asked and compiles findings into an answerNot published for the consumer app. Developer API documentation describes search decisions and query executionUses query fan-out: multiple related searches. Google describes supporting pages being identified while the response is generated
How sources appearCitations and a Sources option when availableNumbered citations linking to original sourcesSources button or inline links; some responses have no sourcesLinks to supporting webpages
Source-selection criteriaRanked using multiple factors that OpenAI does not listNot publishedNot publishedNot published
Documented opt-out or access effectsA site that opts out of OAI-SearchBot is not shown in ChatGPT search answers, although it may appear as a navigational linkPerplexity-User generally ignores robots.txt rulesGoogle-Extended does not affect Google Search inclusion or rankingExcluding a site through the Search generative AI setting stops traffic and impressions from the covered features
Official documentationOpenAI Help Center and OpenAI crawlersPerplexity crawlers and Perplexity HelpGemini Apps Help and Google crawlersGoogle AI features and Search generative AI control

Important scope note: Microsoft Copilot is not included in this comparison because it has a separate guide in our content series.

A distinction also matters throughout this article:

  • A citation is a link to a webpage or source within an AI-generated answer.
  • A mention is a brand or website name appearing in the answer, whether or not it includes a link.

This guide uses the word cited to mean linked, rather than simply mentioned.

Where the AI Engines Match

Despite their differences, the four platforms share a broad pattern.

They retrieve information from external sources as part of generating answers, and they provide links or citations in at least some responses. They also document ways for website owners to control crawler access or the use of their content.

This means website accessibility is a sensible starting point when investigating AI search visibility.

For example, a website owner should understand whether:

  • The relevant crawler can access the website.
  • Important pages can be discovered and indexed where the platform relies on a search index.
  • Robots.txt rules unintentionally block a crawler the business wants to allow.
  • The website contains accurate and understandable information.
  • The platform displays a link to the original page or another supporting source.

OpenAI and Google also acknowledge that AI-generated answers can contain errors. OpenAI’s Help Center says ChatGPT search citations can be incomplete, outdated or incorrect. Google’s Gemini documentation warns that Gemini can hallucinate and present inaccurate information as factual.

These warnings are important. A citation does not automatically mean that the information is complete or accurate.

Our guide to how generative engine optimization works explains the broader retrieval process and why website accessibility matters.

However, these shared principles should not be confused with confirmed citation-ranking factors. None of the vendors says that crawlability, indexing or a particular content format determines which individual page will be selected.

Where the Engines Differ

1. Crawler and Access Controls

The four platforms do not use one universal crawler or website-owner control.

OpenAI and Perplexity publish separate crawler documentation covering different purposes. Google’s approach is different: Gemini and Google Search generative AI features have distinct documented controls.

For ChatGPT, website owners should understand the difference between:

  • OAI-SearchBot: Used to surface websites in ChatGPT search.
  • GPTBot: Used for collecting content that may be used for model training.
  • ChatGPT-User: Used for user-initiated actions.

OpenAI explains that ChatGPT-User is not used to determine whether content appears in ChatGPT search.

Perplexity also separates its crawlers:

  • PerplexityBot: Used to surface and link websites in search results.
  • Perplexity-User: Used when a user asks Perplexity to retrieve a page.

Perplexity states that PerplexityBot is not used to crawl content for AI foundation models.

Google’s controls have a different structure:

  • Google-Extended: Controls certain uses of crawled content for Gemini model training and grounding.
  • Search generative AI setting: Controls whether a site is included in Google Search generative AI features covered by the setting.

These are separate controls. Changing Google-Extended does not change the Search generative AI setting.

2. Robots.txt and User-Triggered Fetches

One of the most important differences concerns Perplexity-User.

OpenAI says ChatGPT-User handles user-initiated actions and is not used to determine whether a website appears in ChatGPT search.

Perplexity explains that Perplexity-User generally ignores robots.txt rules because the fetch is triggered by a user’s request.

This distinction matters if your organisation relies on robots.txt as its primary method of controlling access to website content.

A robots.txt rule is not a universal access-control mechanism. Website owners should understand how each platform documents its user-triggered fetching behavior and review their server logs and other access controls where appropriate.

3. What Each Platform Says About Accuracy

OpenAI and Google explicitly warn that their AI-generated responses can be inaccurate.

OpenAI’s ChatGPT search documentation says citations may be incomplete, outdated or incorrect. Google’s Gemini Help documentation also warns that the system can present inaccurate information as factual.

The Perplexity Help page reviewed for this comparison describes numbered citations as a way for users to check information. It does not include an equivalent warning in the section examined.

That is a difference in published documentation, not evidence that one platform is more accurate than another.

4. Where Each Platform Finds Information

The available documentation also differs in how much it explains about information retrieval.

ChatGPT Search

OpenAI explains that ChatGPT can decide whether to search the web for a particular prompt. It may also rewrite a user’s question into multiple searches. OpenAI further states that ChatGPT search can use partner content.

This means the publisher’s own webpage may not be the only source involved in an answer.

Perplexity

Perplexity says that it searches the internet when asked and compiles findings into an answer with numbered citations. The documentation reviewed for this comparison does not specify which complete index or external providers it uses.

Gemini app

Google’s crawler documentation says content from the Google Search index can be supplied to Gemini at prompt time for grounding. However, Google does not publish the complete source-selection process for the consumer Gemini app.

Google AI Overviews

Google describes query fan-out, in which a search can involve multiple related queries to identify supporting pages. Its documentation also explains that AI Overviews may use different models and techniques from other Search experiences.

The same question can therefore produce different links across different engines. A citation in one platform does not establish that another platform will select the same webpage.

5. Developer Grounding Is Not the Same as a Consumer App

Another distinction is necessary when comparing Google’s documentation.

Google’s Grounding with Google Search developer documentation explains how the Gemini API can decide whether to search, run queries and return citations.

This is documentation for a developer-enabled API feature. It is not a complete description of the consumer Gemini app.

Using developer API documentation to explain every aspect of the consumer app would mix two different products.

The same care should be applied when comparing other AI platforms: always check whether a source describes a consumer product, a developer API, a crawler or a training process.

What No AI Engine Publishes About Source Selection

The biggest gap across these platforms is the absence of a complete, publicly documented source-selection formula.

Based on the vendor documentation reviewed for this article, none publishes:

  • A complete list of factors determining which websites are cited.
  • The relative weight of individual citation-selection factors.
  • Confirmation that traditional organic search position directly determines citation selection.
  • A universal schema type, page length or content format that increases citation probability.
  • A complete explanation of how frequently each platform searches or which queries it generates for every prompt.
  • A consistent, measurable definition of freshness or authority used to select sources.

OpenAI comes closest to acknowledging an internal ranking process by saying ChatGPT search results are ranked using multiple factors. However, it does not identify those factors.

Google documents some retrieval and eligibility mechanisms, but these are not the same as publishing a complete source-selection formula.

Perplexity also describes how it searches and presents citations without publishing its selection criteria.

This is why claims about guaranteed AI citation factors should be treated carefully.

Our GEO best practices guide discusses practical approaches while distinguishing established practices from assumptions about individual AI systems.

How to Evaluate Claims About AI Citation Factors

Before implementing a recommendation that claims to improve AI citations, ask:

  • How many prompts were tested? A handful of examples does not establish a general rule.
  • When were the tests conducted? AI answers and source links can change over time.
  • Which platform and location were used? Results from one engine or country should not automatically be applied to another.
  • Was there a comparison group? Without a comparison, it is difficult to establish whether a change caused a particular result.
  • Does the publisher have a commercial interest? A vendor selling a visibility score or optimisation service may benefit from presenting its methodology as essential.

A result from a few prompts describes those prompts. It does not establish a universal ranking factor.

The same standard should apply to our own recommendations. The practical steps below are based on published access documentation and a testing process, not on an undisclosed claim that we know how each engine ranks sources.

A Robots.txt Checklist by Engine

Robots.txt is an important part of understanding crawler access, but the correct configuration depends on the platform and your business decisions.

Use the following table as an audit checklist rather than a recommendation to allow every crawler.

Platform or purposeCrawler or controlWhat to check
ChatGPT searchOAI-SearchBotConfirm it is not unintentionally blocked and that your hosting provider or CDN allows OpenAI’s published IP addresses.
ChatGPT trainingGPTBotMake a deliberate decision about training-related access separately from search access.
Perplexity searchPerplexityBotCheck whether it is allowed, as recommended by Perplexity.
Perplexity user-triggered fetchingPerplexity-UserUnderstand that robots.txt rules are generally ignored for these user-requested fetches.
Gemini appGoogle-ExtendedCheck that the setting reflects your intentions for Gemini training and grounding. It does not affect Google Search inclusion or ranking.
Google AI OverviewsSearch generative AI settingCheck whether Include or Exclude is selected and whether the page remains indexed and eligible to appear with a snippet.

Example: Allowing ChatGPT Search and Perplexity Search Crawlers

If you want to explicitly allow OAI-SearchBot and PerplexityBot, the following is a simple example:

User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /

This example should be adapted to your existing robots.txt file.

Before making changes, review your current crawler rules, check for conflicting directives and consider whether your business wants to allow search access, training access or both.

Do not assume that allowing a search crawler also allows every training crawler from the same company.

What the Available Studies Measure

There is limited comparable public research measuring citation overlap across all four platforms.

One useful study concerns Google AI Overviews.

In March 2026, Ahrefs analysed 863,000 keyword search engine results pages and approximately four million AI Overview URLs.

Its findings were:

The findings show that not every URL cited by AI Overviews also appeared in Google’s traditional top 10 results.

However, the study has important limitations.

Ahrefs reported a higher top-10 overlap figure in 2025. It explained that its parsing method, the model behind AI Overviews and the sample size changed between studies. The two figures may therefore not be directly comparable.

More importantly, this research concerns Google AI Overviews. It does not establish how ChatGPT, Perplexity or Gemini select sources.

We have not verified a comparable citation-overlap figure for those three platforms, so their results should not be inferred from Google’s data.

The main takeaway is that traditional ranking position alone should not be treated as a complete explanation of AI citation selection.

What Website Owners Should Do

The comparison suggests a practical order of work. These steps combine documented crawler controls with our recommendations for measuring website visibility.

1. Check Access Separately for Each Platform

Review your website’s crawler settings and access rules.

  • Check OAI-SearchBot and your hosting provider’s handling of OpenAI’s published IP addresses.
  • Review PerplexityBot access.
  • Understand Perplexity-User’s documented robots.txt behavior.
  • Confirm that Google can index your important pages.
  • Review Google-Extended according to your Gemini training and grounding preferences.
  • Check the Search generative AI setting for Google Search features.

Do not apply one universal robots.txt rule to every platform.

2. Make Important Pages Clear and Useful

Create pages that address specific questions with clear explanations.

For example:

  • Start relevant sections with a direct answer.
  • Use descriptive headings.
  • Explain products, services and technical terms accurately.
  • Support factual claims with reliable sources.
  • Keep important business information consistent.
  • Review outdated or unsupported statements.

These are general content-quality practices, not confirmed citation-ranking rules for the four platforms.

3. Test the Same Prompts Across All Four Platforms

Choose 10 to 20 questions that reflect the information your potential customers are likely to seek.

Include:

  • Category questions about your industry.
  • Comparison questions naming your business and a competitor.
  • Brand questions about your products, services or organisation.

Run the same prompts across ChatGPT, Perplexity, Gemini and Google Search AI Overviews where available.

Record:

FieldWhat to record
DateWhen the test was performed
PlatformChatGPT, Perplexity, Gemini or Google AI Overviews
PromptExact question submitted
Sources displayedWhether citations or source links appeared
Cited websitesWhich websites were linked
Your websiteWhether your domain or page appeared
AccuracyWhether the answer described your business correctly

Repeat the tests over several days before deciding that a pattern exists.

4. Use Platform-Specific Guides

Each platform has different documentation and access controls. A detailed guide for one engine should not automatically be applied to another.

We cover ChatGPT and Perplexity in separate guides. Our Gemini and Google AI Overviews guides address their respective products.

5. Use Software to Support Larger Tests

Manual testing can become time-consuming when you are checking many prompts, pages and competitors.

Software may help organise prompt tracking, source monitoring and reporting. However, a vendor visibility score should not be treated as an official measurement from an AI platform.

Our comparison of GEO tools explains what to examine when evaluating these products.

Common Mistakes When Comparing AI Search Engines

Avoid these common assumptions when building an AI search strategy:

  • Applying one platform’s advice to every engine. Each vendor documents different crawlers, controls and retrieval behavior.
  • Assuming robots.txt controls every AI fetch. Perplexity-User generally ignores robots.txt because the fetch is user-triggered.
  • Treating Google-Extended as a Google Search setting. Google says it does not affect Search inclusion or ranking.
  • Trusting undocumented ranking-factor lists. Ask for the testing method, sample size and original evidence.
  • Judging visibility from one prompt. Repeat tests and record the results across multiple dates.
  • Counting every citation as a success. A link is not necessarily useful if the answer contains inaccurate information about your business.
  • Assuming a result from Google applies to ChatGPT or Perplexity. Research findings should remain within the platform and sample they actually measured.

Where GEO.solutions Can Help

Understanding where competitors appear in AI-generated answers can help identify gaps in your current visibility.

Our GEO audit is a starting point for reviewing competitor citations and areas where your website may be missing.

The objective is to understand how your website is represented across relevant AI search experiences and identify practical areas for further investigation.

No external provider can guarantee a citation. The platforms do not publish their complete source-selection processes, and website owners cannot directly control which pages are selected for an answer.

Frequently Asked Questions

How do AI search engines choose sources?

No major platform publishes a complete explanation. OpenAI says ChatGPT search results are ranked using multiple factors that it does not disclose. Perplexity does not publish its selection criteria. Google documents eligibility and retrieval mechanisms, including indexing and query fan-out, but not its complete source-selection process.

How does ChatGPT find sources?

ChatGPT decides whether web search could improve an answer and may rewrite a question into multiple searches. OpenAI identifies OAI-SearchBot as its search crawler and says websites should allow it to be eligible for ChatGPT search. Search results are ranked using multiple factors that OpenAI has not publicly listed.

Does Perplexity follow robots.txt?

Perplexity recommends allowing PerplexityBot so websites can be surfaced in its search results. Perplexity-User behaves differently: its documentation says it generally ignores robots.txt rules because it fetches pages in response to user requests. Website owners should understand this distinction when managing crawler access.

Do Gemini and Google AI Overviews use the same sources?

Google does not say they use identical sources. Its documentation explains that AI Mode and AI Overviews may use different models and techniques. For Gemini Apps, Google-Extended governs certain uses of Search index content for grounding. Google does not publish a rule saying both experiences select the same webpages.

Does ranking in Google determine which websites get cited?

Google has not confirmed that traditional ranking position determines AI Overview citations. An Ahrefs study found that 37.9% of sampled cited URLs also appeared in the top 10. The study covers Google AI Overviews only, and its findings should not be applied to ChatGPT, Perplexity or Gemini.

Can one robots.txt rule cover all four AI search engines?

No. OpenAI, Perplexity and Google use different crawlers, tokens and controls. These include OAI-SearchBot, PerplexityBot and Google-Extended, alongside Google Search controls. Perplexity-User generally ignores robots.txt. Review each platform separately and avoid broad rules that unintentionally block services you want to access your website.

Should I block AI crawlers?

That depends on your business priorities. OpenAI says websites that opt out of OAI-SearchBot are not shown in ChatGPT search answers. Perplexity recommends allowing PerplexityBot for search visibility. Training-related crawlers, such as GPTBot and Google-Extended, involve separate decisions from search access. Review each control before changing it.

💬 WhatsApp Order Now →