Attribution Describes a Path. An Incrementality Test Asks Whether the Ad Changed It

“Did marketing work?” is too broad for an experiment. A usable question names the intervention, eligible population, exposure period, outcome window, and decision that follows. For example: should paid search in these regions keep its present budget next quarter? That wording forces the team to decide what “without the ads” means.

Attribution distributes credit across observed interactions. Incrementality estimates the difference between an outcome under treatment and the outcome that would have occurred without it. The second quantity is a counterfactual; it cannot be recovered merely by choosing a more elaborate attribution model.

Randomization is valuable because it makes the treatment assignment independent of potential outcomes in expectation. It does not repair a broken launch. Budget leakage, overlapping campaigns, sales territories that cross geo boundaries, and customer movement can contaminate the contrast.

Keep an implementation log: intended spend, delivered spend, audience exclusions, outages, creative changes, promotions, and material competitor events. A technically sophisticated estimate built on an undocumented treatment is hard to interpret and harder to repeat.

Power analysis is not a ceremonial calculation performed after the regions have been chosen. Estimate baseline volume, variation, feasible holdout size, expected treatment strength, and the smallest effect that would change the decision. When the business cannot supply enough units or time, narrow the question or accept that the test may remain inconclusive.

A non-significant result does not prove zero effect. A statistically detectable lift can still be commercially trivial. Put the estimate, interval, absolute outcome, media cost, and operational caveats on the same page so neither statistical nor commercial importance disappears.

State where the finding applies: the tested markets, spend range, audience, offer, creative, and period. Extrapolating far beyond that support is a new assumption, not part of the result. Record follow-up conditions that would trigger another test.

Incrementality testing is most useful as a repeated calibration practice. It can challenge platform reporting, inform model priors, and show where another dollar has evidence behind it. It cannot issue a permanent certificate that a channel always works.