The Idiocy Of ‘Testing’
One of the great barriers to growth is the idiocy of ‘testing’. Not because testing is bad, but because most fundraisers and their consultants don’t have the slightest idea what real testing is all about.
When in doubt, “Let’s test it”. Consequently countless thousands/millions are spent and the result is vapid stargazing, at best. And months and months of time wasted at worst.
‘Analysis Paralysis’, a horribly expensive disease that infects our sector, especially the direct response folks and their consultants, is epidemic. Problem is, everyone talks about it; few know what ‘testing’ really means.
Most of it is worthless. Leads to no conclusions. Costs a lot of money.
But yields zero, zilch when it comes to change.
So let’s get real.
Start by reading Stop The Direct Mail Testing by Kevin Schulman, CEO of our sister company DonorVoice, and then prepare to answer questions from your clients on why you’re recommending that testing strategy.
Truth is: few tests are meaningful. One more ‘check the box’ in your annual plan.
Truth is: you can almost never beat the control.
Based on real data — unlike the tribal wisdom in our profession — there really are some empirically-driven rules to guide your testing. Read them here in detail. But if you’re too impatient or time constrained, they are:
- Allocate 25% of your acquisition and house file budget to testing.
- Of the 25%, put 10% into incremental testing and 15% into big ideas.
- Set specific guidelines/goals for expected improvement.
You’d do yourself a favor to read this whole post by Kevin. It’ll save you a lot of time. Save you a lot of money. Most of all, it’ll demonstrate why most of us don’t know much about testing, not matter what we claim.
Roger
P.S. Most nonprofits rely on the Random Nth sample for their analysis. Do you know how foolish and false this is? Probably not. Check out Kevin’s post to find out why.
3 responses to “The Idiocy Of ‘Testing’”
Behavioral Science Q & A
Integrating an individual giving appeal with other communications from a charity can have both positive and negative effects, and the outcome largely depends on how it’s executed. Advantages of Integration Brand Consistency: Maintaining a consistent appearance and messaging across all communications can reinforce the org’s brand identity and strengthen brand recognition and trust among your […]
Read Full Answer
I’m not aware of any in-market tests specifically comparing recurring vs. gift frequency language. I suspect the answer might not be the same with all gift frequencies, nor with all people. It sounds like a great opportunity for you to test and find out what works for your audience. Based on the literature, here’s a couple […]
Read Full Answer
Based on what we know from existing data, those renewal notices can actually be pretty effective in getting people to donate. They tap into our psychology – creating a sense of urgency, reminding us of past support, and using personalization to make the message hit home. They’re playing on our natural tendencies to feel obligated […]
Read Full Answer
Interesting question. I had a quick look at the testing done on this topic. On the positive side, in all cases, over half of donors decide to cover the fee. In some cases, it goes as high as 65%. Not a negligible percentage at all. Here’s another test from iRaiser showing consistent results (see point […]
Read Full Answer
There’s just one thing to consider when designing a supporter journey: the supporter. More specifically, you need to take into account: Who the supporter is i.e. their identity, which is the reason they support this cause, and their personality, which describes the way they “see” and process the world. These will determine the kind of […]
Read Full Answer
I’m not an expert in this but a quick search surfaced this article on the effect of tax reforms on 2019’s charitable giving. The researchers didn’t find a reduction. Actually, they observed an “increase in charitable contributions in 2019, even with the lower tax rates and the dramatically smaller number of taxpayers who itemize their […]
Read Full Answer
well done Roger…I could not agree more.
There are only so many 25% in a FRs budget but 20% should be allocated to “innovation” …and this also means testing. How few spend that sadly.
Beautiful! I loved Kevin’s post!
hi, sorry Roger, I absolutely disagree… You can only improve on things by testing new ideas.
Now of course, you can’t go crazy or test things that really do not make much difference.. (the difference between a white and cream envelope… nah..) but things like the following I have tested many times over the years and they make an impact!!
ask amounts..
adding a give away or taking one out
size of the package from # 10 to 6 x 9 to 9 x 12…
all I have tested and retested and they do make a major difference
if it ain’t broke, no need to fix it, I agree, but if you’re looking to increase response to offset the extra cost (especially postage) for example…
by all means test it and test it well.
In other words, make sure you have enough for those magic 100 responses you need (typically at least 5,000 names)
If you are small and don’t have the bandwidth to test that quantity, go with what other organizations are doing… or trust your consultants.. 😉
Cheers, Erica