Practical magnitude
also called Effect sizeMinimum detectable effectConfidence interval
Definition
How large a measured change is in terms that matter to a decision, an effect size, a minimum detectable effect, or an interval around the estimate.
Scope, grain and dimensions
- Grain
- Would be one value per test if it were produced.
- Metric type
- statistical verdict · not produced
Data sources
Where you'll see this
Named reports that normally include this metric.
AI referral traffic reportCannibalization reportContent decay reportContent performance reportInternal linking reportKeyword ranking and movement reportMonthly executive search reportPaid and organic overlap reportPaid search performance reportSearch market share reportSearch to revenue reportSERP volatility and algorithm update reportWeekly search report
Skills and analyses that use it
Skills carry the judgment; the analysis verbs do the reading.
Method rungs and levers
A rung tells you what a movement here can and cannot explain, read the rungs below it first.
Ask Quattr
- "Is this change big enough to act on?" Simulate this →
- "What is the confidence interval?" Simulate this →
- "What change could we even have detected?" Simulate this →
Caveats, freshness and failure modes
No tool produces an effect size, a minimum detectable effect or a confidence interval.
The absolute change and the relative change are available and are the closest thing to a magnitude read, but neither carries uncertainty around the estimate.
No multiple-comparison correction is applied anywhere either, so a batch of significant results has no adjusted view to fall back on.
When a magnitude question is asked, the answer is what exists, the change, the sample size and the verdict, plus a plain statement that the sizing statistics are not available.
- Freshness
- varies by site, priority URLs can run nightly; most pages far less often. The collection date rides on the card rather than being assumed.
Common failure modes
- Presenting a small p-value as evidence that a change is large.
- Computing an interval or an effect size outside the tools and presenting it as a product output.
- Reading the significance threshold as a minimum effect the test was designed to detect.
Not the same as
The confusions that cause the most wrong decisions.
Practical magnitude Significant Compare definitions →
The flag is the p-value below a threshold and nothing else. It carries no effect size, and none is computed anywhere, so significance can never stand in for magnitude.
Practical magnitude p-value Compare definitions →
A p-value answers whether a difference is distinguishable from noise. Magnitude answers whether it is worth acting on. The tools produce the first and not the second.
Practical magnitude Real, noise or can't-tell Compare definitions →
Real means distinguishable from noise, not large enough to act on. No effect size is computed anywhere, so the verdict says nothing about magnitude.
Practical magnitude Pearson correlation Compare definitions →
A coefficient describes the tightness of a relationship, not its business size, and no interval is computed around it.
Watch this metric read in a real run
All runs →
2 min 5 secTraffic more than halved in six months. Nobody could say why.Search Console · Anomaly investigator◐Recreated from an anonymized session
3 min 4 secThe fix shipped. Traffic rose. Now two teams want the credit.Lighthouse + Search Console + Rank tracking · Significance referee · Cross domain correlator◐Recreated from an anonymized session
2 min 26 secConversion rate fell 7.5%. The number is real. The panic is not.Web analytics · Significance referee◐Recreated from an anonymized session
2 min 2 secSomething went right this quarter. Now prove it before you say it out loud.Rank tracking + Search Console · SERP feature watch · Significance referee◐Recreated from an anonymized session
Related metrics and workflows
Verification
A definition is the smallest part of this.
The measurement matters because something acts on it. Here is the rest of the showcase, in the order most people find useful.