Significant
also called Statistically significantSignificance flag
Definition
A true or false flag set when the p-value falls below the chosen threshold.
How it's calculated
The p-value compared against the threshold, which defaults to 0.05 and is a caller-settable parameter. That comparison is the whole of the flag, nothing else feeds it. The one exception is a refusal rather than a computation: where the displayed period change and the per-bucket change point in opposite directions, the period-over-period tool emits no flag at all rather than attaching the bucket-level result to a number the test did not measure.
Scope, grain and dimensions
- Grain
- One flag per test.
- Required filters
- date_range · comparison_range
- Aggregation
- One flag per test.
- Metric type
- statistical verdict · true or false, or absent
Data sources
Where you'll see this
Named reports that normally include this metric.
Keyword ranking and movement report
Skills and analyses that use it
Skills carry the judgment; the analysis verbs do the reading.
Analysis verbs
pop_significancesignificance_check
Method rungs and levers
A rung tells you what a movement here can and cannot explain, read the rungs below it first.
Ask Quattr
- "Is that significant?" Simulate this →
- "Did it really change?" Simulate this →
How to read it
A threshold crossing, not a judgment about importance. The threshold is a parameter, so a flag is only comparable against another flag computed at the same threshold. Where a decision is at stake, the verdict field is the better read, because it separates a well-powered null result from an underpowered one.
Caveats, freshness and failure modes
The flag is the p-value below the threshold and nothing else. It carries no effect size, no minimum detectable effect and no confidence interval, none of those is computed anywhere in the tool surface.
No multiple-comparison correction is applied, so a batch of tests will produce false flags at roughly the threshold rate.
A false flag does not mean no change. The cross-domain tool reports only this flag, so an underpowered null there is indistinguishable from a well-powered one, the rate-aware tool's verdict field exists to make that distinction.
The threshold is caller-settable, so two flags are only comparable when both used the same one.
An absent flag is a third state, not a false one. It means the tool declined to attach a test result to the displayed change, and reading it as not-significant inverts the refusal into a finding.
Significance is not causation.
- Freshness
- varies by site, priority URLs can run nightly; most pages far less often. The collection date rides on the card rather than being assumed.
Common failure modes
- Reading a false flag as evidence of stability.
- Treating an absent flag as a false one.
- Comparing flags computed at different thresholds.
- Reporting the significant rows from a large batch without noting that nothing corrected for the batch size.
Not the same as
The confusions that cause the most wrong decisions.
Significant Practical magnitude Compare definitions →
This flag is a threshold crossing on a p-value only. No effect size is ever computed by any tool, so the flag cannot speak to whether a change is large enough to act on.
Significant Real, noise or can't-tell Compare definitions →
The flag has two states; the verdict has three, because it separates a null result with adequate power from a null result without it. A false flag can mean either.
Watch this metric read in a real run
All runs →
2 min 5 secTraffic more than halved in six months. Nobody could say why.Search Console · Anomaly investigator◐Recreated from an anonymized session
3 min 4 secThe fix shipped. Traffic rose. Now two teams want the credit.Lighthouse + Search Console + Rank tracking · Significance referee · Cross domain correlator◐Recreated from an anonymized session
2 min 26 secConversion rate fell 7.5%. The number is real. The panic is not.Web analytics · Significance referee◐Recreated from an anonymized session
2 min 2 secSomething went right this quarter. Now prove it before you say it out loud.Rank tracking + Search Console · SERP feature watch · Significance referee◐Recreated from an anonymized session
2 min 18 secThe deck is due tomorrow. The numbers live in five places.Search Console + Rank tracking + AI visibility + Web analytics + Google Ads · Monthly exec review◐Recreated from an anonymized session
2 min 20 secWhere do we stand? Five questions answer it, in order.Search Console + Rank tracking + AI visibility · Monthly exec review◐Recreated from an anonymized session
Related metrics and workflows
Verification
A definition is the smallest part of this.
The measurement matters because something acts on it. Here is the rest of the showcase, in the order most people find useful.