Direct Mail Testing To Nowhere

May 30, 2014      Roger Craver

I suspect a good part of the reason why fundraising and especially acquisition is so flat or down lies in the business-as-usual, risk adverse nature prevalent in the contemporary nonprofit mentality.

A mindset focused on protecting the institutional status quo … of defending one organization’s turf against another organization’s ambition … of making sweeping and noble pronouncements about societal good while quietly and narrowly advancing self-interest.

Unfortunately, this same mindset flows into today’s fundraising process — the slow and painstaking nudging forward of not-much-risk, not-much-gain — a phenomenon that a colleague of ours calls “massive incrementalism”.

Nowhere is this affliction among both agencies and nonprofits alike more pronounced than when it comes to direct mail testing.

The bread and butter of conventional nonprofit testing methodology has long been the A/B split test. And while the logic is sound, it is incredibly inefficient and unproductive.

In fact, even with a ridiculously over-simplified example of a direct mail package with 3 components — outer envelope, letter and reply form — and 6 choices for each component, there are 216 possible combinations. If a nonprofit does 15 tests a year it will take 14 years to test all the possibilities!

It’s like believing the telegraph is a good as the internet. But in reality there’s not enough time or money to do the amount of concept testing that’s essential for significant breakthroughs.

It shouldn’t – and doesn’t need to — be this way. Not when technology makes it possible to test 1,500 or even more concepts, price points, graphics, key messages all at once.

Meet the power of conjoint analysis. Apple, Microsoft, Marriott, AT&T, Proctor & Gamble, you name them … the big consumer companies have been using it successfully for years. Why not nonprofits?

Take a few minutes and watch this video on how the commercial world has revolutionized its marketing.

Well the same technology, adapted specially for nonprofit direct response, is also available.

You’ll find a description and video of this testing tool in The Agitator’s Toolkit on our homepage.  Just click here to go there.

Unless nonprofits become far bolder and far more willing to employ the proven technologies available, they will continue to spend endless years, and enormous amounts of money on A/B testing that amounts to little.

Perhaps even worse, the timid, take-little-risk that infects both agencies and nonprofits mostly ends up with testing the most marginal and incremental. Orange vs. blue envelopes … this letter vs. that letter signer … $15 vs. $25 … and sizes of envelopes. Incrementalism to nowhere.

Denny Hatch, one of the best copywriters in the biz and Editor of the must-read Business Common Sense reminds us of the late Ed Mayer’s admonition, “Don’t test whispers.”

Ed’s advice should be tattooed somewhere on every direct response fundraiser. Why? Because, small, incremental changes (“whispers”) produce, well, incremental results usually not even worth whispering let alone shouting about. Whether up or down, these tiny changes hardly matter.

While it’s certainly true that small changes in the response can yield meaningful changes on the top or bottom revenue line of large-volume mailers, it’s equally true that the vast, vast majority of these tests do not beat the control.

Sadly, most testing becomes more habitual than strategic or purposeful.

Kevin Schulman over at DonorVoice has prepared a brief presentation outlining the problems with current testing methods and how the conjoint method works. So instead of testing two variables with an A/B split, waiting for the ink to dry and the postal service to deliver, you can test 1,500 or more concepts in less than two weeks. Then you can choose from the best and go into the mail.

You can download Kevin’s testing overview here.

Whether you’re a consultant or the person in a nonprofit responsible for direct mail you owe it to your client and your organization to get up to speed.

By taking this scientific, disciplined route, nonprofits can greatly reduce cost by NOT mailing test packages likely to perform poorly and increase net revenue by increasing volume on likely winners.

Roger

P.S. As a copywriter of 45+ years, with my share of winning controls, there’s no way my ‘educated guesses’ can come close to what’s possible with today’s conjoint analysis. Fortunately, with this analysis I can safely claim to beat almost any control. No more shots in the dark.

P.P.S. Some of our most devoted readers noted there was no Agitator post yesterday. It was all Tom’s fault, and he will not be getting a Christmas bonus this year.

One response to “Direct Mail Testing To Nowhere”

  1. Larry May says:

    A pet peeve of mine is how long it takes most organizations to roll out a winning idea to other segments of the program, or insist on limiting a package to one function such as acquisition and never test it to the current file, etc. The idea “We’re testing this in acquisition and if it does well we’ll test it to the lapsed file” makes little sense to me. I always suggest testing a promising idea and testing it in acquisition, the active file and the lapsed program. Most often if the idea works it will work in all three and you don’t have to wait three years to find out you have an across-the-board winner. LM