Correlation ≠ causation in search analytics

Somebody wrote the sentence into the planning doc months ago, and by the time it reached you nobody could name the author. The redesign drove the ranking gains. It reads like a finding. What it records is that two things happened in the same six weeks.

Search analytics is where the old correlation lecture goes to be ignored. Rankings moved after the redesign. Assistants started naming your pages while the PR campaign ran. Sessions fell as page speed got worse. Each of those feels like a finding. None of them is one yet.

The fix is not to stop asking whether two things move together. Ask that constantly. The care goes into what you write down afterwards.

What co-movement can actually tell you

That two metrics moved together over the same window, and how strongly.

Two tests do the arithmetic. Pearson reads straight-line relationships. Spearman reads rank agreement, which holds up better when two freak weeks would drag the picture around. Both arrive with a significance value, a statistical check on whether the pattern is bigger than the normal week-to-week wobble, because a short enough sample hands you a beautiful correlation for nothing.

The useful version crosses domains that never meet otherwise: page speed against clicks, ad spend against organic traffic, sessions against how often assistants name your pages. The cross-domain-correlator skill, a packaged analysis you run by name, pulls both series and picks the right test.

Lining the series up is half the work and the dull half. Compare weekly page-speed numbers against daily clicks, or let two date ranges differ at one end, and you have manufactured a correlation out of bookkeeping. Duller than the math, and it matters more.

See also: the analyses your assistant runs by name →

What it can never tell you

Which way the arrow points, or whether there is an arrow at all.

Search analytics is a confounder factory, and not by accident. Updates, campaigns, releases and seasons ship in the same weeks, because everyone plans around one calendar. So the caveat here is not a formality. It is the shape of the finding.

The seasonal confounder is the classic. Holiday demand lifts your sessions while a holiday code freeze quietly improves your page speed, and the two lines agree for reasons that never touched each other.

Three traps do most of the damage in search data:

• The confounder: seasonality, an algorithm update or a migration moves both metrics at once, and the lines agree without touching each other.
• The direction problem: a real link says nothing about which one moved which.
• The short window: over a handful of weeks, unrelated lines slope the same way.

Why every answer comes back a candidate

Because a candidate is what the data supports. Every correlation result comes back labeled the same way, as a candidate factor, and it keeps that label.

Candidates are worth a great deal. They rank the hypotheses worth testing and kill the ones your data will not flirt with. What a candidate never does is turn up in a deck as a cause.

That line about correlation not implying causation prints on the card every time. Not because you forgot. Because the moment you write the slide is the moment everybody is tempted to forget.

The label is also a ranking, which teams miss. Strong, significant co-movement earns a test sooner than a weak flirtation. Candidate status is a queue position, and the queue is the output.

The only road from candidate to cause

It runs through a holdout, and the designs that clear that bar have their own article. You change part of the site, leave a comparable part alone, read the difference.

It works because the pages you left alone lived through the same weather as the treated ones, the same updates and season and everything else you did not control, so whatever separates the two groups afterwards belongs to the work you did.

Until that test reports, your deck says moved together, on purpose. Hold the line a quarter and your claimed causes start surviving scrutiny, because the weak ones never reached the slide.

The promotion path is narrow because the demotion path is easy. A candidate that fails its test dies quietly, no retraction to write and no meeting to book.

See also: the test designs that clear the causal bar →

Ask freely, conclude slowly

It fits in two moves. Ask the co-movement question freely. Read the answer like somebody who expects to be fooled.

Ask whether page speed tracks clicks, whether the pages assistants name track your sessions, whether paid spend tracks your organic decline. Then treat every yes as the opening of a test rather than the close of an argument, which is a smaller adjustment than it sounds and changes what your team believes.

After a few cycles that ordering stops being a rule you enforce and becomes reflex. Somebody puts a pretty scatter plot on the screen, and the first question in the room is what a holdout would say. That reflex outlives any finding it produces.