Quattr leads AEO, SEO, and content rankings on G2 Spring 2026. View our G2 badges →
Request demo
Request demo

Stats

How to tell if a metric change is real or noise

significance-referee a saved expert workflow your assistant follows, in Claude, ChatGPT or Cursor

The "is this real?" verdict: a statistical real / noise / can't-tell ruling on any metric change, clicks, share, sessions, conversions, paid, page experience, with p-value and power, no cause-hunting.

Try it

is this change real

Private beta Part of the quattr-ai-search plugin, rolling out org by org. Request access ↗

Say any of these

  • is this drop real
  • is that significant
  • real or noise
  • is this just seasonality noise
  • can we trust this lift
  • did it really change
  • statistically significant?

You never name the skill or a tool, the assistant routes on what you asked. These are the phrasings it recognises for this one.

And when it stays out of the way

Not when you ask WHY something moved (anomaly-investigator owns the cause hunt), or for routine reporting.

Use it instead of anomaly-investigator when you want only the verdict; use anomaly-investigator instead the moment causes are wanted.

What it runs

  • pop_significance
  • significance_check

2 governed analyses, composed and scoped for you. Named here for transparency, not because you have to ask for them.

The numbers behind it

Metrics it reads, each defined in the reference

AI referral session changeConversion rateConversion-rate changep-valuePeriod-over-period change, relativePractical magnitudeReal, noise or can't-tellRelative changeSearch market share changeSERP volatility indexSignificant

Reports these numbers feed

AI Overview and SERP feature reportAI referral traffic reportAI visibility reportCannibalization reportCompetitor search reportContent decay reportContent performance reportCrawl log reportCTR opportunity reportInternal linking reportKeyword ranking and movement reportMonthly executive search reportPaid and organic overlap reportPaid search performance reportSearch market share reportSearch to revenue reportSERP volatility and algorithm update reportWeekly search report

What it refuses to do

  • Never soften the vocabulary: "probably real", "trending significant" and "basically stable" are banned; the three verdict words plus their numbers are the whole opinion.
  • A significant change is still not a cause: this skill rules on existence, never on attribution; intervention claims need a designed test.

What the answer looks like

Skill: Real vs noise, refereed at α = 0.05. Simulated · illustrative

Is this drop real?

Analysis plan, every step, resolved for you

  • Routed to the governed workflowsignificance-referee
  • Dates bound explicitlyJul 26 to Aug 1, 2026
  • 2 analyses queued in parallelsignificance_check · pop_significance
  • Honesty gates armedsignificance α=0.05 · scope echo · freshness

A −4.2% week looks scary; the math says wait.

Organic clicks
41.2K ▼ −4.2%
Statistical power
0.41 seven daily buckets, underpowered for small effects

Jul 26 to Aug 1, 2026 vs Jul 19 to Jul 25, 2026organic

⚠ Observationalas of Aug 2, 2026 (GSC lags 1 to 2 days)IllustrativeOpen in Quattr ↗

The interval crosses zero, this week is indistinguishable from normal wobble.

Verdict
Noise, not signal
p-value
0.18
95% CI
−9.8% to +1.6%

Verdict: noise, not signal. Re-check after two more weeks of data before acting.

Jul 26 to Aug 1, 2026 vs Jul 19 to Jul 25, 2026organic

⚠ Observationalas of Aug 2, 2026 (GSC lags 1 to 2 days)IllustrativeOpen in Quattr ↗

No action, no cause hunt, the referee says play on.

A simulated conversation on fictional data. Scope, freshness and the ⚠ Observational marker are attached by the same director that renders every demo here, not written by hand.

See this Skill inside a real analysis

All runs →

A recorded run where this Skill does the work, the prompts as typed, and the decision that came back.

Want this Skill running on your own accounts? Every Quattr customer works with an AI Search Strategist who sets it up with you.

Consult an expert