How AI engines actually choose which sources to cite
Three stages sit between your page and an AI answer. Most content fails at the second one, and almost nobody instruments it.
When an AI engine answers a question with citations, three things happened in sequence. Understanding which one you are failing tells you what to fix — and stops you from rewriting pages that were never the problem.
Stage one: retrieval
The engine issues one or more searches, often rewriting the user's prompt into several sharper queries first. It pulls a candidate set — typically tens of documents, sometimes hundreds — using a mix of keyword and vector similarity.
If you fail here, nothing else matters. Causes are almost always mechanical: the page is blocked to AI crawlers, requires JavaScript to render its main content, sits behind an interstitial, or has no meaningful text within the first render. In our audits, blocked documentation subdomains and JS-only content account for the large majority of retrieval failures.
Diagnosing this is easy and most teams skip it. Check your server logs for GPTBot, PerplexityBot, ClaudeBot, Google-Extended and Bingbot. If a section of your site shows no hits, it does not exist as far as those engines are concerned.
Stage two: reranking
The candidate set is scored and cut down to the handful of passages that will actually inform the answer. This is where the great majority of good content dies.
Rerankers evaluate the chunk, not the page. Four properties dominate:
- Self-containment. Does this passage answer the question without the surrounding text? A paragraph starting with "This means that…" scores badly no matter how good the reasoning is.
- Answer density. What proportion of the passage is directly responsive versus context, throat-clearing and transition?
- Specificity. Concrete numbers, named entities and defined conditions outscore general claims consistently.
- Structural signals. A heading that matches the question, a definition list, a labelled comparison table — all make a chunk easier to select confidently.
The practical implication is that your best content is often your least citable. Sophisticated writing builds an argument across paragraphs. Retrieval rewards paragraphs that stand alone. You do not have to choose — put a self-contained answer under each heading, then build the argument beneath it.
Stage three: synthesis and attribution
The model writes an answer from the selected passages and attaches citations. Two things decide whether your passage survives into the final text.
Corroboration. Where sources conflict, models favour claims repeated across independent sources. A number that appears only on your site is more likely to be dropped than one echoed by a trade publication and a forum thread.
Attributability. A passage that states a checkable fact gets a citation. A passage that states an opinion tends to get absorbed into the answer without one. This is why thought-leadership content that performs beautifully on social media generates almost no citations.
Instrumenting each stage
Most teams have no visibility into any of this. You can get it cheaply:
| Stage | Signal to check | Fix if failing |
|---|---|---|
| Retrieval | AI crawler hits in server logs | Unblock, server-render, fix status codes |
| Reranking | Passage self-containment audit | Answer-first restructuring, entity repetition |
| Synthesis | Citation share per prompt over time | Third-party corroboration, checkable claims |
Work them in order. There is no point restructuring passages on a subdomain no crawler can reach.
What does not appear to matter much
Two things get more attention than the evidence supports.
Word count. We see no meaningful relationship between page length and citation rate once passage quality is controlled for. A 600-word page with four clean answers outperforms a 4,000-word page with none.
Keyword density. Retrieval is largely semantic. Repeating a phrase does not help, and it damages readability, which does.
One more, more contentiously: llms.txt. It is cheap to add and harmless, but as of this writing there is no public evidence that any major engine reads it. Treat it as a low-cost bet, not a strategy.
The summary
Retrieval is a plumbing problem. Reranking is a structure problem. Synthesis is a trust problem. They fail differently, they are fixed differently, and diagnosing which one you have is the entire job.