Real, noise or can't-tell
also called VerdictSignificance verdict
Definition
A three-way ruling on a period-over-period change: real, noise, or can't-tell.
How it's calculated
Derived, never eyeballed. A p-value below the threshold gives real. Otherwise the sample is checked for adequate power: a null result with adequate power gives noise, and a null result without it gives can't-tell. Power is a floor, not a computed power figure, at least eight time buckets in each period, the same floor for every metric including conversion rate. One further route to can't-tell bypasses the p-value entirely: when the change in the displayed period aggregate and the change in the per-bucket means point in opposite directions, the test result is not transferred to the displayed number at all.
Scope, grain and dimensions
- Grain
- One verdict per test.
- Required filters
- date_range · comparison_range
- Aggregation
- One verdict per test.
- Metric type
- statistical verdict · one of real, noise, can't-tell
Data sources
Where you'll see this
Named reports that normally include this metric.
AI Overview and SERP feature reportAI referral traffic reportAI visibility reportCannibalization reportCompetitor search reportContent decay reportContent performance reportCrawl log reportCTR opportunity reportKeyword ranking and movement reportMonthly executive search reportPaid and organic overlap reportPaid search performance reportSearch market share reportSearch to revenue reportSERP volatility and algorithm update reportWeekly search report
Skills and analyses that use it
Skills carry the judgment; the analysis verbs do the reading.
Analysis verbs
pop_significance
Method rungs and levers
A rung tells you what a movement here can and cannot explain, read the rungs below it first.
Ask Quattr
- "Is this drop real or just noise?" Simulate this →
- "Can we call this month a real improvement?" Simulate this →
How to read it
The three-word vocabulary exists because two words are not enough: an underpowered null is not stability. Can't-tell means the test lacked the sample to find a change of the size that would matter, and it is the correct answer more often than teams expect on low-traffic segments. The vocabulary is fixed, probably real, trending significant and basically stable are not options.
Caveats, freshness and failure modes
Power here is a fixed floor on the number of time buckets, not a computed statistical power figure. A change smaller than the floor could detect will still be called noise once the floor is cleared.
The floor is eight buckets per period, and it is the same for every metric, there is no separate sample-size floor for rates, because rates are tested the same way as counts.
Because the floor counts buckets, the bucket size chosen for the test decides whether a period is deemed powered at all. A month tested weekly cannot clear eight buckets; the same month tested daily clears it easily on the same data.
Only the web-analytics period-over-period tool emits this verdict. The cross-domain tool returns a two-state flag, so a can't-tell there is reported as not significant.
A real verdict is still not a cause; ruling on existence is the whole of the opinion.
Too few buckets, or no variation within either period, produces a can't-tell with an explanatory note rather than a test result, and the change itself is still reported beside it.
- Freshness
- loads nightly, a same-day question gets yesterday's number, labelled as yesterday's.
Common failure modes
- Reporting a can't-tell as stable or unchanged.
- Reading the power floor as a computed power figure, or as a floor on traffic rather than on buckets.
- Comparing verdicts produced at different bucket sizes as though they were equally powered.
- Expecting a verdict from the cross-domain tool, which does not produce one.
Not the same as
The confusions that cause the most wrong decisions.
Real, noise or can't-tell Significant Compare definitions →
The flag has two states and collapses an underpowered null into the same answer as a well-powered one. The verdict keeps them apart, which is the entire reason it exists.
Real, noise or can't-tell Practical magnitude Compare definitions →
Real means distinguishable from noise, not large enough to act on. No effect size is computed anywhere, so the verdict says nothing about magnitude.
Real, noise or can't-tell Period-over-period change Compare definitions →
A delta states how far the number moved; a significance verdict states whether the move is distinguishable from noise at an alpha. The verdict is a p-value threshold only, no effect size is computed anywhere, so a significant change can be trivially small and a large delta can fail the test.
Real, noise or can't-tell Period-over-period change, relative Compare definitions →
A relative change is a magnitude; a significance verdict is a test result at an alpha. Neither implies the other, and no effect size is computed on either side.
Real, noise or can't-tell Ranking movement (gainers and losers) Compare definitions →
Movement classification is a fixed ±10% cut on the relative change; a significance verdict is a two-sample test at an alpha. Being a gainer says nothing about whether the change is distinguishable from noise, and no effect size is computed on either side.
Watch this metric read in a real run
All runs →
2 min 5 secTraffic more than halved in six months. Nobody could say why.Search Console · Anomaly investigator◐Recreated from an anonymized session
3 min 4 secThe fix shipped. Traffic rose. Now two teams want the credit.Lighthouse + Search Console + Rank tracking · Significance referee · Cross domain correlator◐Recreated from an anonymized session
2 min 26 secConversion rate fell 7.5%. The number is real. The panic is not.Web analytics · Significance referee◐Recreated from an anonymized session
2 min 2 secSomething went right this quarter. Now prove it before you say it out loud.Rank tracking + Search Console · SERP feature watch · Significance referee◐Recreated from an anonymized session
2 min 18 secThe deck is due tomorrow. The numbers live in five places.Search Console + Rank tracking + AI visibility + Web analytics + Google Ads · Monthly exec review◐Recreated from an anonymized session
2 min 20 secWhere do we stand? Five questions answer it, in order.Search Console + Rank tracking + AI visibility · Monthly exec review◐Recreated from an anonymized session
Related metrics and workflows
Verification
A definition is the smallest part of this.
The measurement matters because something acts on it. Here is the rest of the showcase, in the order most people find useful.