The answer won’t hold still
The structure and markup were refreshed for current answer engines; the original analysis is preserved.
This month Bard started tailoring its answers to your own Gmail and Docs, and added a button to check itself against Search. The lesson for anyone trying to measure their visibility is bigger than one product: a generative answer is not one fixed thing you can look up. The honest response is to stop snapshotting a single response and start sampling the distribution — to count how often you appear, not whether you appeared once.
the short answer
A generative answer is not a stable object the way a search ranking is. It is produced fresh each time and varies with the model’s randomness, the wording, the moment, and now — with Bard pulling in your Gmail and Docs — the person asking. So a single check is one roll of the dice, not a measurement. The move is to sample the distribution: ask each important question many times, record how often you are cited, merely mentioned, or absent, and track that frequency over time. Personalization rules out a single universal number — not measurement itself.
key takeaways
- This month Bard began tailoring answers to your own Gmail and Docs, and added a button to double-check itself — two signs that a generative answer is not one fixed thing you can simply look up.
- Ask the same question twice and you can get two different answers, with different sources and different brands named. A single check is one roll of the dice, not a measurement.
- The shift for measurement: stop snapshotting a single response and start sampling the distribution — ask many times, across phrasings and moments, and record how often you appear.
- That frequency — how often you show up, in what form, against whom — is stable enough to track over time and across the changes you make, because it is built from many samples instead of one.
- Personalization rules out a single universal number, not measurement itself. You measure a distribution, you sample it honestly, and you stop pretending one lucky look was the truth.
one look vs a sample
Same brand, same question. The single query says "yes" and feels conclusive; it is one draw. The twenty sampled asks say "twelve of twenty", a frequency you can compare next month and after every change you make. Count how often, not whether once.
Why one number quietly lies
The pull toward a single check is understandable: it is fast, it is concrete, and it produces a screenshot you can paste into a deck. The problem is that the screenshot is true and useless at the same time — true, because that is genuinely what the model said that once; useless, because the next asking might say something else entirely. A metric earns its keep by being reproducible: if someone else runs it, or you run it again next week, you should land near the same place, and a divergence should mean something changed. A single generative answer fails that on day one, because variation is built into how it is produced. Reporting one look as your AI visibility is not a small rounding error; it is reporting the outcome of one coin flip as the probability of heads.
This month’s Bard update sharpens the point rather than creating it. Once an answer can be shaped by the asker’s own Gmail and Docs, there is not even a shared answer to screenshot — the response is partly about the person, not just the question. And the new self-check button is Google conceding that any given response is provisional, something to be verified rather than trusted on sight. None of that means the engines are unmeasurable. It means the unit of measurement was never a single answer; it was always going to have to be a frequency drawn from many. The engines did not become unmeasurable; the thing worth measuring simply stopped being a point and became a shape.
The shift, in three parts
The answer stopped being a single thing; one look is an anecdote, not a measurement; so sample the distribution and track the frequency. Open each part for where it changes the work.
01 The answer stopped being a single thing
A search ranking has always been a fairly stable object: at a given moment, the list is roughly the same for everyone, so you can look once and know where you stand. A generative answer is a different kind of object. It is produced fresh each time, and it moves — with the model’s built-in randomness, with the exact wording of the question, with the moment you ask. This month Google added another source of movement: Bard can now tailor its responses to your own Gmail, Docs, and Drive, which means the answer two different people get can diverge not just by chance but by who they are. Google also shipped a button to double-check Bard against Search, an implicit admission that any single response is provisional rather than settled. Put together, the message is hard to miss: the thing you were trying to "check your position in" is not one thing. It is a cloud of possible answers, and looking at one of them tells you about that one, not about the cloud. The instinct to look once is borrowed from a world of stable rankings, and it quietly misleads you.
02 One look is an anecdote, not a measurement
This is where a lot of early AI-visibility reporting goes wrong, and it is worth being blunt about it. Someone asks a chatbot the money question, sees the brand cited, screenshots it, and declares victory — or asks once, sees nothing, and declares a crisis. Both are reading a single draw as if it were the whole distribution. If the brand shows up in, say, half of all askings, then the one-time winner and the one-time loser saw exactly the same underlying reality and drew opposite conclusions from it, because each looked once. A measurement has to be reproducible enough that someone repeating it gets close to the same result; a single generative answer fails that test by construction, because the next person who asks may get something different. Treating one screenshot as a metric is not a small imprecision. It is mistaking an anecdote for evidence, and it will send you chasing noise.
03 So sample the distribution and track the frequency
The fix is the oldest idea in measurement: if one observation is noisy, take many and look at the shape. Pick the questions that actually matter to your business. Ask each of them many times — vary the phrasing the way real people do, spread the asks across moments — and record what happens every time: cited, merely mentioned, or absent, and alongside which competitors. Out of twenty asks you might find you are named in twelve; that twelve-in-twenty is a real, trackable number in a way a single yes-or-no never is. Establish that frequency as a baseline before you change anything, then watch how it moves as you do the work, and add more samples where the stakes justify the cost. You will never get a perfect census, and you should not claim one. But a frequency built from many samples, tracked consistently, is a measurement — and it is the kind the AC Group has insisted on for 27 years: count what actually happens, enough times to trust the count. We have measured noisy channels before — survey panels, multivariate tests, anything where a single observation lies — and the habit transfers cleanly: never let the vividness of one result substitute for the weight of many.
Two teams, one question
Two teams want to know how they show up when buyers ask a chatbot for the best option in their category. The first asks once, sees its brand named, and reports to leadership that the company is "winning in AI search". A week later a competitor asks the same question, does not see that brand at all, and reports the opposite. Both screenshots are real; both conclusions are built on a single draw; and the two teams now hold contradictory beliefs about the same reality, neither of them measured.
The second team treats it as a sampling problem. It writes the question a dozen ways a real buyer might phrase it, asks each across several sittings, and logs every outcome — cited, mentioned, or absent, and next to whom. The picture that comes back is not a triumph or a disaster but a number: present in about sixty per cent of askings, usually in the company of two particular rivals, rarely first. That number is unglamorous and enormously more useful, because next quarter the team can ask again the same way and see whether sixty has become seventy or fifty. They are no longer reacting to the mood of a single answer; they are tracking a frequency that responds to the work they do. Same engines, same question — but only the second team is measuring, and the gap between an anecdote and a baseline is the whole difference.
What to do with this
Build a small, repeatable sample instead of collecting screenshots. Choose the handful of questions whose answers actually move your business. For each, write several natural phrasings a real buyer would use, and ask them enough times — across different moments — that the frequency settles down rather than lurching with every new ask. Record each outcome the same way every time: were you cited, mentioned, or absent, and which competitors appeared with you. The point is consistency of method, so that the numbers you get this month and next month are comparable.
Then treat the frequency as your baseline and protect it. Run the sample before you change anything, so you know where you started; re-run it on a steady cadence, so movement is visible; and resist the urge to celebrate or panic over any single answer in between, because the single answer is exactly the thing you have decided not to trust. Be honest about what the sample represents — it is an estimate over the contexts you tested, not a census of every possible asking, and personalization means no number is universal. Held to that standard, the work gives you something rare in this space: a measurement you can stand behind. It is the same discipline the AC Group has applied to every channel for ' + years + ' years — count what happens enough times to trust the count, and never let one vivid anecdote stand in for the distribution it came from.
What sampling can’t fix
Sampling is the right tool, but it is worth being clear about what it does not buy you, because overclaiming here is its own kind of error. A sample is an estimate over the contexts you tested, not a census of every possible asking — if your real buyers phrase the question in ways you never tried, or ask from contexts you cannot reproduce, your frequency describes your test set, not their world. Personalization makes this sharper: once an answer leans on someone’s private Gmail and Docs, there are askings you simply cannot sample from the outside, because you are not that person and do not have their data. So a sampled frequency is honest only if it comes with its own footnotes — these phrasings, these moments, this much personalization accounted for and no more.
That is a limit, not a defeat. Every measurement worth trusting carries the same kind of caveat; a brand-lift study estimates over a panel, not the whole population, and is useful anyway because it states its bounds. The failure mode is not "the sample is imperfect" — every sample is — it is pretending the sample is the whole truth, or quietly swapping it back out for a single screenshot the moment the screenshot is flattering. Held honestly, with its limits on the label, a sampled frequency is the most truthful number available on this surface. Held dishonestly, it is just a fancier anecdote.
Make the sample worth trusting
The value of a sample lives entirely in the discipline of how you take it, so a few rules earn their keep. Fix the method before you look at any results — the phrasings, the number of asks, how you score an outcome — so you cannot quietly tune the method until it flatters you. Score every answer the same way, including the ones you wish had gone differently, because a sample you have cherry-picked is not a sample. Write down what you tested, so next quarter’s run is comparable to this one rather than a fresh improvisation. And keep the questions and phrasings stable over time, changing them deliberately and noting when you do, so a move in your frequency means the world moved and not your ruler.
None of this is exotic; it is just the ordinary hygiene of measurement applied to a new and slippery surface. The reason it matters more here than almost anywhere else is that the underlying thing is so noisy that a sloppy method will hand you a number indistinguishable from chance, dressed up to look rigorous. A clean method is what separates "we appear about sixty per cent of the time, here is exactly how we know" from "we asked a few times and it felt good". Only the first survives a skeptical question, and on a surface this volatile, the skeptical question is the one worth being able to answer.
Measuring a moving answer: quick answers
Why can’t I just check where I appear in a generative answer?
Because there is no single, stable answer to check. Unlike a search ranking, which is roughly the same for everyone at a given moment, a generative response is produced fresh each time and varies with the model’s randomness, the exact wording of the prompt, the moment you ask, and — as of this month, with Bard pulling in your own Gmail and Docs — the personal context of the person asking. Ask the same question twice and you can get two different answers, with different sources cited and different brands named. So a single check tells you what happened in one roll of the dice, not what generally happens. If you looked once and saw yourself cited, that is one data point, and it might be the lucky one; if you looked once and did not, that might be the unlucky one. Either way, a lone look is an anecdote, and an anecdote is not a measurement.
What does it mean to sample the distribution?
It means treating each answer as one draw from a range of possible answers, and taking enough draws to see the shape of the range rather than a single point. In practice you take a question that matters to your business, ask it many times — varying the phrasing the way real people would, across different moments — and record what happens each time: were you cited, merely mentioned, or absent, and alongside whom. Out of, say, twenty asks, you might find you are named in twelve, which is a real number you can track, rather than the yes-or-no of a single check. That frequency — how often you show up, in what form, against which competitors — is the measurement. It is stable enough to compare over time and across changes you make, precisely because it is built from many samples instead of one.
How many samples do I need?
Enough that the frequency stops swinging wildly when you add more. There is no magic number, but the logic is the same as any sampling: a handful of asks gives you a noisy estimate, and more asks tighten it. For most brands, asking a given question on the order of ten to twenty times, across varied phrasings and moments, is enough to tell a real sixty-per-cent presence from a real twenty-per-cent one — which is the distinction that actually matters for a decision. You do not need scientific precision; you need enough samples that a change in your frequency reflects a change in the world rather than the luck of a single draw. Start modest, watch how much the number moves as you add samples, and add more for the questions where the stakes justify it.
Does personalization make measurement impossible?
It makes a single universal number impossible, but it does not make measurement impossible — it just means you measure a distribution rather than a position, and you are honest about what your samples represent. When an answer can be shaped by the asker’s own data, there is no one answer that is true for everybody, so chasing a single "where do I rank" figure is the wrong goal from the start. What you can measure is your presence across a representative spread of askings: many phrasings, many moments, the contexts your actual audience would bring. That will never be a perfect census, and you should not pretend it is. But a well-sampled estimate of how often you show up, tracked consistently over time, tells you far more than a single personalized snapshot that was only ever true for one person on one afternoon.
A note on sources and timing
This is written in September 2023, just after Google gave Bard the ability to draw on a person’s own Gmail, Docs, and Drive, and improved its "Google it" button to check Bard’s answers against Search — by Google’s own account, a feature still being improved. We have described only what was public as of this writing and nothing announced later. The durable point does not depend on any one product: a generative answer varies by chance, by wording, by moment, and now by person, so a single look is an anecdote and the honest unit of measurement is a frequency sampled from many asks. That is the same standard the AC Group has held for 27 years — count what actually happens, enough times to trust the count — now applied to a surface where the answer will not hold still long enough to be photographed.