It’s possible — maybe even likely — that your “winner” in a point-in-time fundraising A/B test didn’t win because of your headline, photo, or copy change. It probably won because of random noise.

Before you yell at your screen that I’m an idiot (maybe true, but not for this reason), walk with me through the logic and the math.

Very little giving happens without an ask. Stop soliciting your donors and giving decays fast toward zero. That means the baseline effect of simply showing up,  asking, is massive.  But once that baseline is in place, the incremental effect of a single campaign change is usually tiny. Most gifts are a maintenance effect: you keep asking, they keep giving. Swapping a subject line or image rarely moves the needle in a way that breaks through the noise.

The noise in your data

When I hear the word noise I tend to think it’s random but that’s not accurate.  It isn’t “random in the universe,” it’s random to me and you.  It’s things you don’t control but that swamp out the impact of a lot of A/B testing:

• Giving cadence habits: Some donors give once a year, no matter what.
• Income and cash flow: Bonuses, bills, unexpected expenses.
• Life events: Moves, births, deaths, illnesses.
• Mental bandwidth: Calm Saturday mornings vs. hectic weekday evenings.
• Conflicting priorities: Another charity beat you to their wallet this week.

These forces are endogenous to your appeal schedule, they interact with when and how you ask, but they aren’t influenced by your photo swap or envelope color. They’re large, structural forces, and they overwhelm incremental testing ideas.

Let’s ground this in a simple acquisition response-rate example:

• Two test cells of 20,000 names each
• Control response rate: 0.5% → about 100 responders
• Standard two-sample proportion test, 95% confidence, 80% power

What lift do you actually need to detect a real effect?

• Treatment needed: about 0.72% response, ≈ 144 responders
• Absolute lift: +0.22 percentage points (0.50% → 0.72%)
• Relative lift: +43%
• Incremental responders: about +44 in treatment vs. control

Plain English: With 20k names per cell and a 0.5% baseline, you need roughly 44 more gifts in treatment (100 → 144) to be confident it wasn’t luck. A 5–10% “bump” from a copy tweak or image swap won’t clear that bar.

Now, here’s the part the textbook leaves out and why real life is harsher:  In a tidy model, every donor is like an independent coin flip with the same probability of responding. Real life isn’t tidy. Responses cluster because circumstances cluster:

• One drop hits early in the week; another bunch lands Friday.
• Some list sources are “hotter” than others.
• Some weeks, the physical mailbox is crowded with competing appeals; other weeks it’s quiet.

This clustering means results bounce around more than the clean model predicts—statisticians call that overdispersion. Practically, it shrinks your effective sample size, so the lift you need to detect a real effect gets even larger than the +43% above.

And zooming out one level further: overdispersion is just the statistical footprint of bigger, structural noise—income timing, life events, competing priorities, attention—that you don’t observe but that swamps most incremental testing.

Even if your p-value says “significant,” in this environment small apparent wins are often just the world’s messiness masquerading as your genius.

Why “statistical significance” isn’t enough

Most fundraisers lean hard on p-values. But a significant result at 95% confidence doesn’t mean your creative tweak truly caused the lift. Here’s why:

• Fragility at low base rates: A handful of donors shifting groups can swing the outcome from “win” to “loss.”
• Variance in amounts: One $5,000 donor randomly in treatment makes it “win” on revenue. That’s noise, not signal.
• Multiple testing: Run enough A/Bs and by definition some will show p < .05 just by chance.

In other words, stat sig doesn’t rescue you from noise when the noise is this big and the base rate this small.

So what should you do?

• Separate maintenance from incremental effects.
  Run suppression tests occasionally (intentionally don’t solicit a random slice or turn a channel spend to $0) to measure how much giving disappears without an ask. That’s your baseline, everything else is incremental.
• Aggregate results.
  Don’t hang your hat on a single test. Pool results across campaigns or over time. Replication smooths out noise and lets small true effects emerge.
• Reduce variance before testing.
  This feels counterintuitive — marketers usually like variance because it gives them “something to explain.” But in testing, variance is the enemy. Mixing $25 donors and $2,500 donors in one cell doesn’t give you more insight; it just adds noise that makes modest lifts undetectable. Segmenting into more homogeneous groups reduces that noise, which makes any real lift easier to see. Then you can compare results across groups to understand where effects are universal versus group-specific.

Because randomness is always in the room with you, the smart move is to design for it, not be fooled by it.

Kevin