This is a long, but very, very important post. So freshen up your coffee before proceeding.

The other day I received an email from an Agitator reader asking: “Why do you think most fundraisers are so resistant to innovation and change?”

A good question. An important question.

I batted out a kneejerk and facile response — ‘risk adverse’ … ‘insufficient resources’ … ’lack of imagination’, etc. etc.. But I couldn’t hit the ‘Send’ button. Deep down I sensed there’s a far more basic reason.

After thinking about the times I’ve resisted, then walked away from some innovative approach and comparing my own experience with similar reactions I’ve seen from others, I believe I’ve discovered a better, more fundamental answer.

We resist experimenting with innovative tools and processes because we insist that ‘It’ – the ‘It’ being a predictive model, a new online tool, a new multi-channel process, you name it — be 100% correct, 100% of the time. Otherwise, forget it, we’ll just stick with the same-old-same-old, thank you.

Of course we arrive at such a silly expectation without the foggiest idea of what the ‘failure rates’ are on the techniques and technologies we’ve been using — without question — for years.

We reject the use of predictive models because they only work 80% of the time, not realizing of course that our old RFM segmentation processes may work only 50% or 60% of the time — if we only knew.   We reject the use of telemarketing for securing monthly donors because ‘it costs more’ and ‘upsets donors at dinner time’, not realizing or bothering to calculate that our ‘failure rate’ for acquiring sustainers by mail is 5 times higher than over the phone. And on and on.

When it comes to the ‘new’ or ‘innovative’ our expectation is that it must work better than what we’re doing 100% of the time, or we simply can’t be bothered trying it.

Why Most Direct Mail Acquisition Testing Sucks

There’s no better example of not understanding the importance of ‘failure rates’ than in the area of direct mail acquisition testing.

The continuing reliance on A/B testing of the most primitive and incremental variables — like color and size of envelopes, changes in fonts, types of response forms (nothing seems too trivial) — that most consultants and nonprofits practice is, of course,  horrendously expensive in terms of lost time, money and opportunity.

This conventional mindset blocks true message, offer and packaging breakthroughs in acquisition at the very time they’re desperately needed.

Even more worrisome is fact that the almost universal (and usually worthless) practice is to focus on what the late Ed Mayer called “testing whispers”, or what I call “testing incrementalism to nowhere” … a practice that by and large goes unchallenged by folks who should know better, or at least have the motivation to learn there’s a better way.

No doubt in my mind that the principal factor in the static state of direct mail testing is the absence of any real understanding of how to define and measure ‘failure’.

In the 6 months since I wrote Part One and Part Two on Direct Mail Acquisition Testing I’ve kept my eyes pealed, and made notes on the state of testing conducted by supposedly some of the most sophisticated consultants and in-house marketers in the business.

I’ve done this for two reasons. The first is out of self-interest, because The Agitator Toolbox features an innovative direct mail testing tool from our sister company DonorVoice, and I was curious to see how it’s been working and what issues it’s encountered. [In short: It’s a winner. Predicting winners and saving clients years of time and thousands and thousands of $$.]

The second reason is that in talking to knowledgeable consultants, fundraisers, technology and analytic vendors, and others who carefully watch trends, there is a constant refrain of how difficult it is to persuade direct mail folks to change their old ways. Just attend any DMA conference for proof of this.

It’s hard to persuade these experts, despite ample empirical evidence and math-based proof that the new ways do in fact work better. Why? Two reasons: 1)  they have no idea of the failure rate of the practices they’re now using; and 2) if the new application isn’t guaranteed to work 100% of the time they’re not about to try it.

Tradition and aversion to risk usually determine what’s defined as ‘Best Practices’. So, even though the failure rate on new acquisition packages is extraordinarily high (fewer than 1 in 6 tests ever beat the control) and the process of getting to failure is expensive and time consuming (ink has to dry, the postal service has to deliver, returns come back over a period of weeks and months, and then have to be analyzed) – there’s real aversion better, newer, less expensive ways of testing.

Too many consultants and fundraisers inside organizations seem to be inoculated against the new — even worse the math-based and empirically proven ‘new’.

Organizations and consultants like these are really not to blame. Somewhat like physicians in the 1950s they’ve profited just fine with primitive tools like the stethoscope and thermometer. No need for CAT-scans and MRI’s and all this genetic nonsense. Content in the consistency of the past they annually or semi-annually migrate to a host of conferences proudly and perpetually proclaiming their cures.

Only problem is that at a time when better and more innovative methods are needed, the smug practices of the past fail dismally when compared to the new approaches that mathematics and statistics make possible.

‘Multi-variate testing’, ‘conjoint analysis’, ‘predictive analytics’ are too often dismissed as some sort of witchcraft — all snake oil, lizard’s tails and newt’s eyes  — simply because they don’t succeed 100% of the time. Even though the ‘good old ways’ succeed only 16% of the time when it comes to beating acquisition controls.

The reality is that math-based models and the empirically-based processes are far more accurate and generally less expensive in both time and money than the ‘good old’ techniques still widely used among direct response fundraisers.

The Paradox of Success and A Case in Point

Now consider the paradox that’s at the heart of my answer to the question of innovation posed by that Agitator reader. On the one hand, we settle for our own testing that delivers only 10%-20% success.  Yet on the other hand, we reject new processes, new ideas and new analytics just because they deliver 50%-60% instead of 100% accuracy. Are we nuts?

Let me give you a real life example that occurred recently and serves as a great example for the point I’m making. A HUGE direct marketer (600 million acquisition pieces a year), attempting to ‘beat the  control’, proudly displayed the 8 variations of the response form they were about to test using the high tech DonorVoice pre-test tool.

The client was using the pre-test tool because to achieve statistical accuracy a conventional ‘wet test’ would necessarily involve mailing to 320,000 prospects, cost $92,000 and take at least 12 weeks before results could be evaluated.

Instead, by exposing the components to a multi-variate online test, the cost would be $25,000, a savings of $67,000. Most importantly, this type of test would create a statistically accurate indication of not only which one of 8 variations would perform best, but would also expose an additional 504 variations of the response form.

In short, identify ‘failure’ and ‘success’ from among the 512 variations instead of 8.

Explaining that the chances of getting a winner out of the 512 variations was vastly superior to what would be identified out of the 8 response forms submitted by various creative shops, the analyst was met with the usual reaction of the statistically naive and ‘test only whispers’ direct response trade: non-comprehension and doubt.

Fortunately, the client suspended disbelief. The test was run, and in less than 10 days the ‘winner’ from among the 512 variations was clearly identified. None of the 8 prospective ‘winners’ originally chosen by the client and its consultants was even close.

And this was a multi-million dollar best-practices direct response organization. No idea of what and how to test. No idea of the cost or frequency of ‘failure’. Safety in ‘whispers’.

My point is not to unnecessarily knock the testing skill of the direct mail trade, but to warn that:

• If you’re going to understand testing and/or the value of any innovation and measure it against current practices, you also have to understand your current ‘failure rate’ and the cost of testing those failures.

• If you’re going to use the new math-based approaches like predictive modeling and multi-variate analysis to dramatically improve your results — and they most likely will — you must evaluate their effectiveness against your own ‘failure rates’.

Remember: A superior result (meaning one that beats the control) of 50% with a 50% failure rate beats the hell out of the ‘superior’ result you now get — 10% or 20% success rate and a 90-80% failure rate — every time.

You don’t have to fall in love with ‘failure rates’. But you sure do have to understand them if you’re ever going to break out of the same-old/same-old pattern that will most certainly guarantee ultimate failure.

Roger