How to measure AI share of voice without fooling yourself
Ask a model the same question three times and you can get three different answers. A single-sample dashboard is a random number generator with a logo on it.
The first question any finance team asks about AI visibility spend is how you know it worked. The honest answer requires a measurement method that survives being poked at, and most of what is currently shipping does not.
The core problem: non-determinism
Language models sample from a probability distribution. The same prompt, sent twice, can return different vendor lists in a different order. Temperature settings, model version rollouts, index refreshes and personalisation all add variance on top.
A single query per prompt per period produces a number that moves for reasons unrelated to your work. If your dashboard shows visibility jumping from 30% to 45% and back to 32% across three weeks, you are almost certainly looking at sampling noise, and you will make bad decisions from it.
The method that works
Sample repeatedly, report the distribution
Run each prompt at least five times per measurement period. Report the proportion of runs in which your brand is named, with a confidence interval. Five samples on a binary outcome is a coarse instrument but it is dramatically better than one, and the marginal value of samples beyond about ten is small for most prompt sets.
Fix your prompt set, and version it
A prompt set that changes between periods makes trend comparison meaningless. Lock a core set, version it, and record the version alongside every data point. Add new prompts to a separate expansion set so the core trend stays comparable.
Record the raw answer, not just the verdict
Store full response text. When a number moves you need to be able to read what actually changed. Teams that store only a boolean cannot diagnose anything.
Log the collection method on every data point
API versus consumer surface, model version, date, geography, whether the session was grounded in live search. These change constantly, and a jump in your metric is often a model release rather than a marketing result. Without this metadata you cannot tell the difference.
Metrics worth reporting
| Metric | Definition | Use it for |
|---|---|---|
| Presence rate | Share of runs naming your brand | Headline trend |
| Share of voice | Your mentions as a share of all brand mentions | Competitive position |
| Mention position | Where in the answer you appear | Quality of presence |
| Sentiment | Favourable, neutral or caveated | Narrative health |
| Citation rate | Share of runs linking your domain | Content effectiveness |
| Source overlap | Which third-party domains recur | Authority targeting |
Presence rate and share of voice are what a board wants. Citation rate and source overlap are what your team acts on.
Four traps
Vanity prompts. Tracking "what is [your brand]" guarantees a flattering number and measures nothing. Track the prompts where a buyer has not decided yet.
Weekly reporting. The signal-to-noise ratio at weekly cadence is poor for most prompt sets. Monthly is honest. Report weekly only if you are running enough samples to justify it.
Ignoring model releases. Annotate your timeline with major model updates. A step change on the day of a release is not your content programme.
Attributing revenue directly. There is usually no referrer and no clean attribution path. Treat AI visibility as an upper-funnel brand metric, correlate it with branded search volume and direct traffic, and resist the temptation to invent a last-click story.
What good looks like
A defensible programme reports: a fixed, versioned prompt set of 100 to 300 buyer-intent questions; five or more samples per prompt per period; monthly cadence; full answer archives; collection metadata on every point; and a timeline annotated with model releases. That is enough to say "citation share rose from 18% to 41% over two quarters, here are the answers, here is what changed" — and to have it hold up when someone pushes back.