The winning version of a fundraising campaign can still leave you poorer than running no campaign at all. It’s a possibility the budget and campaign calendars are remarkably good at leaving out.

We test the headline, photograph, envelope and ask string. Each new idea must prove it can beat the control, while the decision to keep funding the activity gets waved through because it was in the budget last year.   The paradox is sitting there unseen, we demand statistical significance from a copy tweak while accepting an entire spending category on faith.

Losing tests can teach us something but a calendar full of minor variations, most of which lose, can produce remarkably little learning about how to grow. Repeatedly discovering that similar appeals perform similarly ought to eventually raise a question or two about the testing agenda.

These “preference” tests help us choose between executions but even a decisive winner leaves another question unanswered: should we be spending money on this activity in the first place?

Every category/channel of spend must be held to this basic but seemingly elusive standard – create additional behavior that wouldn’t happen or get deleted.  Incrementality is the $0.50 cent word for this.

We recently tested Google Performance Max for a client whose platform reporting showed $6.44 in revenue for every dollar spent. At that rate, “we’ve used up the budget” is a peculiar complaint. If each additional dollar really caused another $6.44 in giving, we should be cashing in our 401ks to throw it into the Google machine.  It’s magic.

Our incrementality test required picking two like geographies and running pmax in one but not the other.  We look in the CRM and bank account to see if pmax created any new donors that wouldn’t have happened otherwise.  It did but at a cost of $577 per one-time donor, which yields no viable payback period.

The $6.44 was fake news, attributed but incorrectly.

The $577 per additional donor was a rather more consequential finding than which ad generated the better response. Giving the ad another headline would have been an expensive way to change the subject.

This applies just as readily to mail, email, SMS, etc.  A contact can claim a donation that would otherwise have arrived through another channel, or bring forward a gift without increasing what the donor gives over time. Better campaign response can coexist with very little improvement in total donor value.

Before you optimize within a channel, establish whether the activity earns its budget. Put a group that doesn’t receive it into the plan, ideally through random assignment. Compare giving across channels over a suitable period, allowing for future donor value as well as costs. The comparison needs to capture how much better off the charity is because it spent the money.

Before spending another year testing envelopes, test sending fewer mailings. If just as much money comes in, the winning packages were the ones you didn’t mail. That should liven up the agency meeting.

Kevin