{
  "slug": "significance-p-value",
  "name": "p-value",
  "aliases": [
    "Significance value",
    "p"
  ],
  "definition": "The probability of seeing a difference at least this large if there were no real difference between the two periods or samples.",
  "category": "Statistics",
  "type": "statistical verdict",
  "unit": "probability between 0 and 1, rounded to six decimal places",
  "direction": "neutral",
  "verification": "verified",
  "calculation": "One of two tests, chosen by sample size, and both run on per-bucket series rather than on period totals. Welch's two-sample t-test with Welch to Satterthwaite degrees of freedom is the default; where either period has fewer than eight buckets the cross-domain tool switches to the Mann to Whitney U rank test, using a normal approximation with a continuity correction. Conversion rate is no exception, it is tested with Welch's t-test over its per-bucket rate values, like every other metric. Both tests are computed in a pure module, never by the model.",
  "aggregation": "One p-value per test. It is never averaged, combined across tests, or carried between metrics.",
  "grain": "One value per test, a metric, a pair of date ranges, and a bucket size.",
  "dimensions": [],
  "requiredFilters": [
    "date_range",
    "comparison_range"
  ],
  "sources": [
    "search-console",
    "web-analytics",
    "google-ads",
    "rank-tracking",
    "lighthouse"
  ],
  "reports": [
    "weekly-search-report"
  ],
  "skills": [
    "significance-referee",
    "anomaly-investigator",
    "rank-movement-review",
    "monthly-exec-review",
    "cross-domain-correlator"
  ],
  "verbs": [
    "pop_significance",
    "significance_check",
    "correlate_domains"
  ],
  "rungs": [
    "R5",
    "R2"
  ],
  "levers": [],
  "questions": [
    "Is this drop real?",
    "Is the lift statistically significant?",
    "Can we trust this change?"
  ],
  "interpretation": "Evidence that a difference exists, and nothing more. It is not the probability the change is real, not a measure of how big the change is, and not a cause. Read it with the sample size beside it, the same p-value means something different at eight buckets and at eighty.",
  "caveats": [
    "No exact t-distribution is ever evaluated. The t-statistic is real, but its probability comes from a normal approximation with a Cornish to Fisher correction, documented in code as anti-conservative at small degrees of freedom: at five degrees of freedom the probability can be understated by roughly a quarter. Treat samples under ten as indicative rather than definitive.",
    "The Mann to Whitney probability also uses a normal approximation; the exact distribution would be preferable below about twenty observations.",
    "A pooled two-proportion test exists in the statistics module but nothing calls it. There is no count-based proportion test anywhere in the tool surface, a rate is tested as a series of bucket values, not as successes over trials, and the cross-domain tool records the proportion case as deferred.",
    "Every test compares per-bucket MEANS, not aggregate totals, a change in total driven by more buckets rather than bigger buckets will not show here, and where a displayed period aggregate moves opposite to the bucket means the rate-aware tool withholds the result rather than transferring it.",
    "The cross-domain tool needs at least two buckets per period and returns an explanatory note instead of a test below that.",
    "No multiple-comparison correction is applied anywhere. Testing many metrics or many rows will produce significant results by chance, and nothing in the tool surface adjusts for it.",
    "A significant change is still not a cause. The tools rule on existence, never on attribution."
  ],
  "freshness": "varies by site, priority URLs can run nightly; most pages far less often. The collection date rides on the card rather than being assumed.",
  "failureModes": [
    "Reading the p-value as the probability that the change is real.",
    "Running many tests and reporting the significant ones without noting that nothing corrected for the number of tests.",
    "Treating a small p-value as evidence of a large or important change."
  ],
  "notSameAs": [
    {
      "slug": "practical-effect-magnitude",
      "why": "The p-value speaks to whether a difference is distinguishable from noise; magnitude speaks to whether it matters. No effect size, minimum detectable effect or confidence interval is computed by any tool, so magnitude cannot be read off this number."
    },
    {
      "slug": "rate-change-percent",
      "why": "Evidence versus size. Neither implies the other, and reporting one in place of the other is the most common misreading of a significance result."
    }
  ],
  "related": [
    "significant-flag",
    "real-noise-verdict",
    "practical-effect-magnitude",
    "pearson-correlation"
  ],
  "workflows": [],
  "lastVerified": "2026-08-04"
}