Is your direct mail testing on auto-pilot?  Are you testing out of habit?  We hear a lot of very smart, sophisticated direct marketers working for big non-profit brands tell us this.

If you are one of them it is time to get off the merry-go-round and stop testing (with the current approach).

These same marketers we hear from are looking at the time, effort and cost of their habitual, auto-pilot testing and the associated return and making the smart decision to cut way back on the number of tests.  It is hard after all to beat the control.

To address the “now what?” question that remains if you find yourself in this boat we put together this 10 point framework for testing.   The guiding principle here is that non-profits should think about all that time, effort and money going into habitual, rote testing as their pot of money for innovation.

We believe this testing protocol will lead to far fewer, more meaningful tests (a big plus) and more definitive decision making on outcomes (another big plus).

1)      Allocate 25% of your acquisition and house file budget to testing.

2)      Of the 25%, put 10% into incremental and 15% into big ideas.

• An important corollary here, some of this money should go into researching ideas or paying others to do it.  You can even use the online environment to pre-vet ideas with small, quick tests of the ideas to gather data.

3)      Set guidelines for expected improvement. 

• Any idea for incremental must deliver a 5% (or better) improvement in house and 10% in acquisition (will see why difference in minute)
• Any idea for breakthrough must deliver a 20% (or better) improvement.

4)      Any idea – incremental or breakthrough – must have a ‘reason to believe’ case made that relies on theory of how people make decisions, publicly available experimental test results or past client test results. 

• The reason to believe must include whether the idea is designed to improve response or average gift or both – this will be the metric(s) on which performance is evaluated.
• A major part of this protocol is guided by the view that far more time should be spent on generation of test ideas and therefore, creating the necessary “rules” and incentives to create this outcome.
• This may very well result in 3 to 5 tests per year.  If they are well conceived and vetted that is a great outcome.

5)      Determine test volume with math, not arbitrary, ”best practice” test panels of 25,000 (or whatever)

• Use one of many web based calculators (and underlying, simple statistical formulas).  Here is one we like but there are plenty – all free.
•  Inputting past control performance and desired improvement (i.e. the 5% of 20%).  Do not use arbitrary 25k and 50k test sizes.
• An acquisition example: if our control response rate is 1% and we want to be able to flag a 5% improvement – i.e. response rate greater than 1.05% – to say it is real  –the test size would need to be 626,231 (at 80% power and 95% confidence and 2-tail test).  That is not a typo.  How many acquisition test panels have been used in the history of non-profit DM that are  producing meaningless results because of all the statistical noise?  A sizeable majority, at least….  If we want to be able to flag a 10% improvement – i.e. better than 1.1% as meaningful – we need a test panel of 157,697.  For most large charities this size is very doable but only if the math is understood on why.

 6)      Do not create a “random nth” control panel that matches the test cell size for comparison.

• We are unsure how many charities employ this approach but it can lead to drawing the exact wrong answer on whether the test lost or won.
• The problem with the “random nth” control test panel of equal size to the test – e.g. two panels drawn with random nth at 25,000 each – is that creates a point of comparison that has its own statistical noise and far more than the main control with all the volume on it.  There are a few retorts that have surfaced in defense of this practice but they are simply off-base.

7)      Determine winners and losers with math, not eyeballing it.

• Use one of many web based calculators to input test and control performance and statistically declare a winner or loser.

8)      Declare a test a winner or loser

• Add results to the “reason to believe” document maintain a searchable archive.

 9)      All winners go full volume, rollout. 

 10)   Losers can be resurfaced and changed with a revised “reason to believe” case.