How agencies should choose 20–50 buyer questions per client
An agency should track 20 to 50 buyer questions per client: enough to pool into stable rates, few enough to review every week. A buyer question is something a real customer would ask an assistant before buying, such as "best accounting firm for a restaurant group in Denver" or "payroll software vs a PEO for 30 staff".
- Split branded from non-branded. Branded questions ("is Acme good for…") tell you how the client is described; non-branded questions ("best tool for…") tell you whether the client is found at all. Report them separately, because branded rates are always higher and would flatter a pooled number.
- Group paraphrases. Write each intent three to five ways and report the group, not the single wording. A May 2026 paper on OpenAI and Anthropic APIs (arXiv 2605.27440; a direct test, from authors who sell tracking) found cosmetic rewrites changed the recommended brands more than simply re-asking: brand overlap was 0.29 across rewrites against 0.50 to 0.61 across reruns.
- Fix the market and language. A 2026 study of logged-out ChatGPT (arXiv 2608.30052; strong but small, 234 runs) found the query language largely decides which country's brands appear, and the exit IP shifts it further. Ask in the client's buyers' language and location, and label every number with its market.
- Cover the funnel. Include comparison, pricing and "how do I" questions, not only "best X". Those are often where fan-out searches land on the client's own pages.
- Freeze the set. Changing questions every month breaks the trend line. Add new questions in a batch and start their baseline separately.
How to build a four-week AI visibility baseline
A four-week AI visibility baseline is the minimum before an agency reports any number to a client or ships a fix. AI answers vary so much from run to run that one week of data mostly measures noise.
"Don't Measure Once" (arXiv 2604.07585, Apr 2026; 45 days of repeated runs on four engines, graded confirmed) found that same-day randomness was about as large as day-to-day change. The error on a brand's detection rate fell below 0.05 only after about 24 days of data. The authors recommend two- to four-week rolling windows and a separate baseline per engine. Spreading samples across the week (for example Monday, Wednesday and Friday) buys more independence than running them all at once.
Record how the data was collected: engine, API or consumer app, market, logged-out. API answers with web search are an unpersonalized baseline, not a copy of what each user sees. Our resource on building an AI visibility baseline for a client walks through the setup step by step.
The weekly AEO review: pooled rates with ranges, not single answers
The weekly AEO review looks at pooled mention, recommendation and citation rates, each with a sample size and a 95% range, never at individual answers. A single answer is close to meaningless: in a SparkToro and Gumshoe test (Jan 2026; 2,961 runs from about 600 volunteers, graded strong), the same brand list came back less than 1% of the time. Mention frequency across many runs was stable.
| When | Task | What you look at | Output |
|---|---|---|---|
| Mon, Wed, Fri | Collect answers | Every tracked question on each engine, samples spread across the week | New answers added to the rolling window |
| Monday | Weekly review | Rolling four-week rates per client, engine and question group, with n and range | A short list of real changes (passed a difference test) |
| Tuesday | Triage | Questions where competitors lead: consulted sources, fan-out searches, search-off answers | Each gap labeled not found, found but not cited, or not known |
| Wed–Thu | Ship fixes | The top-ranked fix for each labeled gap | Fixes shipped, with a start date and a control set |
| Friday | Check fixes in flight | Leading indicators: page changed, bot fetched it, page consulted, page cited | Notes for the monthly report |
| First week of the month | Client report | Four-week rates against last month and competitors, fix results | One report per client |
| Each quarter | Model-release check | Model changes, API vs app agreement, crawler access, question set | Annotated trend lines, refreshed questions |
Two rules keep the review honest. First, a cell with fewer than about 20 answers shows as "collecting", not as a percent: at 10 answers a 50% rate has a 95% range of roughly 24% to 76%. Second, decide whether a rate really moved with a test on the difference, not by checking whether two ranges overlap, which hides real changes. Our methodology on difference tests explains the test we use. Optimz AI visibility tracking applies both rules by default, on rolling four-week windows.
Triage: not found vs found but not cited vs not known
Triage sorts every visibility gap into one of three causes, because each cause needs a different fix. Getting the label right is most of the work.
- Not found: the client's pages never enter the engine's retrieved set. Check the fan-out searches the engine ran and the sources it consulted. If the client's page isn't there, the problem is access, indexing or topical coverage.
- Found but not cited: the page was consulted, but the answer cited someone else. Ahrefs' 20-day study (Oct 2026, graded strong) found that even when ChatGPT ran identical searches, the cited sites changed 81% of the time, so much of the loss happens after retrieval. This is a content problem: the page doesn't answer the question early or specifically enough.
- Not known: with web search off, the model doesn't name the client at all. That is a gap in what the model learned in training, and it only moves when a new model ships. The fix is off-site footprint, not on-page work.
Our guide to why a brand is not in ChatGPT: known vs found covers how to run the search-off test and read the result.
Best AEO fixes ranked by evidence
The best AEO fixes, ranked by how strong the evidence is, run from access gates through answer-first pages and off-site mentions to freshness. Work down the list for each client, and don't skip to the bottom because it feels more creative.
- Access gates (confirmed). Search and AI retrieval bots allowed in robots.txt, no firewall challenge, main content in the server HTML, Google indexing and snippet eligibility, no Bing noarchive on pages you want in Copilot (nocache limits Copilot to the URL, title and snippet). A blocked page can't be read live, so nothing else matters until this passes.
- Answer-first pages for fan-out queries (strong for topical coverage; plausible for placement). Build or rewrite pages that answer the exact sub-questions engines search for, with the answer in the first sentences. Surfer's study of 173,902 URLs (correlational) found a 0.77 rank correlation between how many fan-out queries a page ranked for and its AI Overview citations. A Growth Memo dataset of 18,012 ChatGPT citations found 44.2% of cited text came from the first 30% of a page (one vendor dataset, no base rate).
- Off-site mentions (strong correlation, causation unproven). Ahrefs found web mentions correlated 0.66 to 0.71 with brand visibility in AI answers across 75,000 brands. Brand size confounds this, so present it as the strongest correlate, not a guaranteed lever.
- Freshness (strong, but engine-dependent). AI assistants cite pages newer than Google's organic results, but AI Overviews is the exception. Update the substance and the visible date together; never change only the date.
Where off-site mentions pay off depends on the client's vertical. Our working model, built from licensing deals, training-corpus papers and citation studies, puts these first:
- B2B SaaS: "best X" listicles, G2 and Capterra, niche subreddits, YouTube reviews.
- Developer tools: GitHub usage, Stack Overflow answers, documentation quality, package downloads.
- Consumer and e-commerce: YouTube reviews, Reddit, editorial review publishers.
- Local and hospitality: Yelp, TripAdvisor, Google Business Profile, city subreddits.
- Finance and health: authoritative news, institutions, Wikipedia.
Some popular fixes have been tested and failed. Ahrefs' controlled schema study (May 2026, 1,885 treated pages against about 4,000 controls) found no significant gain in ChatGPT or AI Mode. Ahrefs also found that 97% of the llms.txt files on 137,210 domains got no requests at all in May 2026. Adding quotes or statistics, word-count targets and FAQ schema don't hold up either. Leave them out of the plan.
How to measure an AEO fix with a control group
To measure an AEO fix, compare the change on the questions the fix targeted with the change, over the same weeks, on similar questions it didn't touch. Engines drift every week, and that drift is often larger than the effect of one page.
The Ahrefs schema study shows why. Treated pages gained about 43% on AI Mode in raw terms, but once control pages were subtracted the net gain was about 2%, and not significant. Without the control, that test would have read as a big win. Before shipping, also work out the detectable change: with a handful of questions and three samples a week, only large swings will show within a month. If the expected effect is smaller, add questions or samples, or tell the client the test can't see it.
- Pick the control before the fix ships: the client's untreated questions on the same engines.
- Report net change: treated minus control, with its range.
- Track leading indicators, which arrive earlier and are less noisy: the page changed, an AI bot fetched it, it entered the consulted set, an owned URL was cited, then the mention rate moved.
- Mark two fixes that touch the same questions in the same window as shared attribution, so neither claims the whole change.
- Note any model release during the test as a confounder.
Optimz Actions and agent prompts attaches a control group and a detectable-change estimate to each fix, and shows the leading-indicator funnel as results come in.
Monthly AEO client report: what to show and what never to claim
A monthly AEO client report should show rolling four-week rates with their ranges, the client next to its competitors on the same answers, and what happened to each fix. It should never claim more certainty than the data has.
- Show: mention, recommendation and citation rates per engine, each with n and a 95% range.
- Show: the same rates for named competitors, measured on the same answers.
- Show: fixes shipped, their net change against the control, and where each sits in the indicator funnel.
- Show: a method label on every number (engine, API or app, market, logged-out).
- Never claim: a "rank" or position in AI answers. Order isn't reproducible from run to run.
- Never claim: a win from one screenshot, or a loss from one bad week.
- Never claim: that a fix caused a change unless it beat its control.
- Never claim: that schema, llms.txt or added statistics moved AI visibility.
Optimz agency reports build this monthly report under the agency's own brand, with the ranges and method labels already on each number.
Quarterly model-release check for AEO programs
A quarterly model-release check catches the changes that weekly reviews can't explain. Engines switch models, sometimes silently: a ChatGPT study from NUS and Xi'an Jiaotong (arXiv 2610.15382, Oct 2026) saw two sessions switch model partway through a run.
- Annotate model and engine releases on every trend chart, so a step change isn't credited to a fix.
- Re-check how closely the API matches the app. The same paper found the API's sources covered only about 23% of the pages the ChatGPT app cited.
- Re-run the search-off test on key questions, since a new model can change what is known.
- Re-test crawler access on every client site after CDN or firewall changes.
- Review off-site source shares. Ahrefs saw ChatGPT's Reddit share fall from 19.1% to 2.3% between March and September 2026, so no fixed "be on Reddit" rule lasts.
- Retire stale questions and add new ones as a fresh batch with its own baseline.
Weekly AEO checklist for agencies
The weekly AEO checklist below is the short version of this workflow. Print it for each account lead.
- 20 to 50 buyer questions per client, branded and non-branded reported separately.
- Three to five paraphrases per intent, in the buyers' language and market.
- Four weeks of baseline before the first report or fix.
- Samples spread across the week.
- Pooled rates with n and a 95% range; "collecting" under about 20 answers.
- A difference test before calling any change real.
- Every gap labeled not found, found but not cited, or not known.
- Fixes in evidence order: access, answer-first pages, off-site mentions, freshness.
- A control group and a detectable-change estimate for every fix.
- Model releases annotated and re-checked each quarter.
Sources
- Schulte, Bleeker, Kaufmann: Don't Measure Once (arXiv 2604.07585, Apr 2026)
- Paraphrase sensitivity in AI brand recommendations (arXiv 2605.27440, May 2026)
- Żatuchin: language and location in ChatGPT brand answers (arXiv 2608.30052, Aug 2026)
- ChatGPT app vs API citations, NUS and Xi'an Jiaotong (arXiv 2610.15382, Oct 2026)
- SparkToro: AIs are highly inconsistent when recommending brands (Jan 2026)
- Ahrefs: How stable are AI citations? (Oct 2026)
- Ahrefs: AI brand visibility correlations
- Search Engine Journal: Schema markup didn't move AI citations in Ahrefs test (May 2026)
- Ahrefs: 97% of llms.txt files never get read (Jun 2026)
- Surfer: Query fan-out and AI Overview citations
- Ahrefs: Do AI assistants prefer to cite fresh content?
- Search Engine Land: ChatGPT citations content study (Growth Memo)