6 · Documentation
Noise band and runs
Why a single measurement is a sample, how we calculate the uncertainty and when a movement is a movement.
Whoever builds a score from a single measurement is partly measuring chance. augenmerk states the uncertainty for every project and reports changes only when they exceed it.
Same question, two answers
AI systems do not answer the same way every time. The same question yields two different answers on two days, with other brands in another order. Measured in the rice market across three runs, with nothing having changed about the brands:
| Brand | Run 1 | Run 2 | Run 3 | Span |
|---|---|---|---|---|
| Tilda | 67 % | 83 % | 78 % | 17 pts |
| Reishunger | 50 % | 69 % | 67 % | 19 pts |
| dennree | 56 % | 33 % | 52 % | 22 pts |
The market leader's number is stable, everyone else's is not. That is not a weakness of the measurement but statistics: the spread of a share is largest at 50 % and small at the edges. The brands that need a measurement sit in the middle.
How the noise band is calculated
The uncertainty depends above all on how many counting questions carry a number. From a project's existing questions we draw repeatedly and measure the width of the 90 % range. We draw questions, not measurements: one question produces an answer in every channel, and those are related.
From this follows a rule of thumb the application uses: with eight counting questions the noise band is ±10 percentage points. With more questions it shrinks with the square root of the number of questions; with 32 counting questions it is around ±5 points. The overview and the competition view state the value for the project at hand.
| Counting questions | Noise band |
|---|---|
| 8 | ±10 pts |
| 13 | ±8 pts |
| 20 | ±6 pts |
| 32 | ±5 pts |
This is an extrapolation and rather optimistic, because real questions are not independent of each other. For brands in the middle of the field the actual spread can be higher. The application therefore adds two rules: segment values are shown only from three runs on, and below about twenty questions no score is produced, only a number with its stated uncertainty.
Repetition hours apart
How much of the variation is chance inside the system, and how much is real change? A data set from 21 September 2026 separates the two: the same rice project with 14 counting questions on ten channels (140 answers per run) was measured four times in one night, two hours apart. Nothing in the world could have changed between the runs.
The overall value is stable. Oryza came in at 33.6, 35.0, 35.7 and 32.9 percent across the four runs (standard deviation 1.1 points), Tilda between 66 and 74, Reishunger between 54 and 61, Davert between 51 and 54. On 140 answers, a brand's overall value moves by one to three points within a night. The rule of thumb above (around ±8 points for 14 questions) is therefore cautious, not too narrow.
The single answer is not. Of 140 question-and-channel pairs, 50 came out differently across the four repetitions: the same question on the same channel no longer mentioned the brand two hours later, or the other way round. More than every third answer flips. Claude was the most restless (9 of 14 pairs), ChatGPT with search, Gemini with and without search (7 each); Google AI Overviews (1) and Perplexity (0) the calmest. Anyone who asks a question once and turns it into "mentioned" or "not mentioned" rolls dice in a third of the cases; the average over many answers does not.
Channel values carry more uncertainty than the overall value. A channel value rests on as many answers as there are counting questions, here 14. In the same night Oryza ranged between 29 and 57 percent on Claude with search, likewise on Gemini with search, between 57 and 79 on OpenAI; on Perplexity it was 7 four times. The channels with live search scatter most, because every search pulls different sources. A channel value is therefore only reliable with more questions or across several runs; the application shows it with its own uncertainty.
The weekly movement was real. Oryza stood at 43 percent on 7 September, at 34 on 13 September and at 33 to 36 in the night. The distance between two consecutive runs averaged 1.7 points in the night and 9.7 between the weekly runs. The move from 43 to 34 lay above the noise band and was a change, not the day's mood; the night confirms the new level three times over.
When a movement is a movement
augenmerk reports a change in presence since the last run only when it exceeds the noise band. Everything below is chance between two runs and does not appear as a finding. The same applies to gaps in the ranking:
- A place counts as secure when the lead over the next name exceeds the noise band. Below that, the overview reports a wobbling place: in the next run the order can flip without anything really having changed.
- A gain counts as real when the next run holds it. If the value falls back, it was an outlier.
- Stable means: no movement above the noise band for several runs. What you see then is reliable; changes from here on are real changes.
Runs
A run measures all active questions of a project on all channels. Every answer stays stored, with timestamp, model and source list. Presence across all runs is the reliable value; the latest run shows the direction. A new customer in an already measured market gets the history retroactively, because every answer stays stored.
Two things break a history, and the application says so:
- A reworded question is a new question. The evaluation recognises a question by its wording. The old history stays stored; the new one begins with the next run.
- A rebuild of the question set shifts the overall value when a large part of the counting questions is new. Such a rebuild should be dated and noted under "How we measure".
New and resolved
After every run the application compares the findings with the state of the previous run. New findings appear as "new", vanished ones as "resolved" and stay visible for one run. Resolved is determined by the measurement, not by the user: as soon as the number behind a finding moves, it goes to "Resolved". See Findings catalogue.
Updated: 2026-10-10