See if this sounds at like your organization:

The direct mail test ideas that actually make it into the mail stream are almost all incremental and rarely (mostly never) beat the control.  You spend enormous amounts of time and money coming up with test ideas, producing them and managing the logistics, only to have the same poor results year over year.

The more creative or innovative ideas tend to get discarded out of fear and the need to stick with the mostly known, if lousy, results of the control and the incremental changes to it.  Even if you do manage to test big ideas, the results are muddled at best since you changed EVERYTHING, instead of 1 thing, and while there may be many good ideas in the creative package they are forever lost amid the bad ones.

There are three, interconnected big problems with how direct mail testing is done.

•  Incrementalism to nowhere – Incremental changes produce, well…incremental results, up or down, it hardly matters.  While it is certainly true that small changes on the response can yield meaningful changes on the top or bottom revenue line it is equally true that the vast, vast majority of these tests do not beat the control.  Testing becomes more habitual than strategic or purposeful.
• The A/B road to infinity.  The bread and butter of testing methodology is A/B tests.  And while the logic is sound, it is incredibly inefficient.  In fact, even with a ridiculously over-simplified example direct mail package with 3 components – outer envelope, letter and reply form – and 6 choices for each component, there are 729 possible combinations.  If a non-profit does 15 tests a year it will take 48 years to test all the possibilities.  When you consider a more realistic example that also includes a front or back end premium or both, the possible combinations quickly go to, for all intents and purposes, infinity.

What does this mean?  Does anybody believe that with a nearly infinite number of  choices to make on a direct mail package, that your control, which is really hard to beat, is the proverbial needle in the haystack – the winning combination among countless possibilities?

• Lack of wisdom in Conventional wisdom.  Most organizations will acknowledge that the process to determine what gets tested is not empirical, rigorous or efficient.  It is more typically, bordering on haphazardness, an abundance of caution and conventional wisdom.

Fortunately, our commercial brethren, not in direct mail, but rather, in product development can point to the solution.  Using a multivariate, survey based methodology non-profits can pre-identify the best test ideas, those most likely to compete with and beat the control.  Non-profits greatly reduce cost by NOT mailing test packages likely to perform poorly and increase net revenue by increasing volume on likely winners.

The pre-identification of likely winners and losers is done in two parts

1) First, surveying donors who are representative of those who will receive the actual mailing, showing them visuals of the direct mail package and measuring preference using a very specific and battle tested methodology.

2) Using the survey data to build a statistical model to assign a score to every single element that was evaluated.

This methodology is well established in the commercial sector and used by large, consumer companies (e.g. Coca Cola, General Mills, Proctor & Gamble) to guide product development for many of the sodas, cereals and detergents on grocery store shelves.

The time has come to drastically change the way direct mail testing is done, the solution is as close as that tube of toothpaste in your bathroom.

P.S. You can find two recent, non-profit case studies using this methodology here.