Working with AI
Troubleshooting AI analytics: when a number looks off
Scope first, freshness second, significance third, the diagnosis order that finds it.
The screenshot arrives with a red circle drawn on it and no other message. A number has fallen off a cliff. The channel is already three replies deep into why, and one of those replies contains the word strategy.
Before you escalate, file a ticket or rewrite a plan, there is an order that finds the explanation faster than instinct does. Scope first. Freshness second. Significance third.
It works because it checks the cheap explanations before the expensive ones, and the cheap ones are usually right. Most surprising numbers die at the first question.
Start with scope, because scope is cheap
Ask whether the number measures what you assumed it measured.
Every result card carries its scope in a row of chips: the date range, the segment, organic or paid, branded or not, the device. That row is provenance made visible: every answer carries where the number came from, what scope it covers and how fresh it is.
A whole-site number sitting beside a segment number will disagree. Neither of them is wrong.
Units live there too, and they cause the loudest false alarms in the building. Your CTR-modeled search market share of tracked demand, meaning your estimated share of the clicks available across the keywords you track, measured against the competitors you track, reads on a zero-to-one-hundred percentage scale.
So a value of 0.03 means three hundredths of a percent. Read it as three percent and you have invented a catastrophe.
Comparison scope hides the same trap twice. A number held up against the wrong prior period, or a filtered view against an unfiltered one, gives you a delta that is pure bookkeeping.
When two people in the room are looking at different numbers for the same week, compare chips before comparing grievances.
Then check the number's birthday
Ask when the data was last complete.
Sources lag differently: hours for server logs, a day or two for Search Console, biweekly for page-speed vitals.
A flat line that ends yesterday may simply not include yesterday. A drop on the most recent day of a lagging source is usually the lag rather than the world moving.
The as-of date on the card answers this in one glance, and the freshness article covers the full table. It is the date the data was last complete, printed on the answer so a lag never gets mistaken for a loss.
Two sources on one card can carry two different as-of dates. That is normal, and the card says so.
You can also just ask which sources are connected and how fresh each one is. One question, not an audit.
Ask it yourself
What data sources are connected, how fresh is each, and did anything change scope on this card?
See also: the full table of source lags →
Third: is the surprise even real?
Now, and only now, the question turns statistical.
If the scope is right and the data is current, the last cheap explanation is ordinary variance. A significance test answers it. That is a statistical check on whether the change is bigger than that metric's normal week-to-week wobble, reported with a p-value. It comes back real, noise, or cannot tell.
A noise result ends the investigation as legitimately as a found cause, and it deserves the same respect in the room. Closing a surprise as ordinary variance is a successful diagnosis that gives you back the afternoon the alarm would have eaten.
Direction changes nothing about the order. A pleasant surprise gets the same three questions as an alarming one, because a scope filter that flatters you is only funny until it reaches a deck.
Only now suspect the pipeline
Real, current, correctly scoped anomalies deserve a real investigation. Now you get to run one.
Sometimes the answer genuinely is a data problem: a feed hiccup, a tracking change, an incident on the source's side. The quattr-troubleshooter skill, a packaged analysis you run by name in your assistant, walks the whole order as one conversation and says plainly when the only explanation left is worth a support ticket.
The order is worth most for what it prevents. Strategy meetings called over a scope filter. Tickets filed against numbers that were never wrong.
It also shortens the ticket you do file. Arrive with the scope read, the as-of date noted and the significance result attached, and you get a faster answer, because you already killed the boring causes. Keep three lines of notes as you go and any escalation is a two-minute read for whoever picks it up.
The workflow that does this: Impressions without clicks →
What the order buys you
Trust in your own dashboards, mostly.
Teams that run it stop flinching at surprises, because every surprise now ends in one of three boring explanations or earns a real investigation. Scope, freshness, significance. Three questions, usually one answer, rarely a crisis.
The order also travels. Those same three questions diagnose a strange number in anybody's stack, because scope, staleness and variance are where numbers go wrong everywhere. The full reference lives in the docs and is worth one team read.