We’ve written about the massive inefficiencies with the way direct mail testing is done today.  The high points are covered here.  One particularly odious problem is throwing out the baby with the bath water when the organization mails a test package with many (or more than 1 for that matter) test element – i.e. a whole bunch of stuff is changed.

The mail results for the package are a very crude measuring stick for performance, only giving thumbs up or thumbs down for the entire package with zero guidance as to whether individual components were well received (i.e. the “baby”) even if the bath water needs to be changed.

This happens all the time and the only alternative, which as a general rule, NEVER happens, is to deconstruct the totally new package into a series of A/B tests with each test panel only including a single change.  Even if this were done, it would take forever and a day to execute. And certainly some groups may try to read the tea leaves and infer or guess based on years of experience and past testing about why a package did poorly and what might be salvageable but that is a process fraught with layers of personal bias.

There is a better, empirical way.  Our commercial brethren in product development have used a survey based methodology for the last 30 plus years to identify and ferret out the baby from the bath water.  This process can be done in weeks versus months, costs a fraction of what traditional testing costs and like a recent client told us, is “like doing 18 months of testing in a day”.  (To learn more about the methodology, click here.)

Here is a recent example of saving the baby.    Client X mails a totally new package – different OE, letter format, letter copy, inserts – against the control.  New package performs poorly.  Money is lost.  Time is lost.  New package is thrown out.

Back up….New package is never mailed because the client pre-tested it (and hundreds of other package combinations) and determined it would not perform well, as constructed, against the control.  Thousands of dollars are saved, many insights are gained that would have taken years to accumulate and the cost of the pre-test is more than covered.  And, a “baby” is potentially discovered with two components of the new package testing quite well (as determined by an actual score assigned to every single element).  New elements are now live tested in the control package and replace elements of the control that are identified as weak.

New bath water (i.e. poor performing test elements) is unfortunately easy to come by; baby elements (i.e. winning package elements) are much more difficult.

Isn’t it time to start identifying and saving more “babies”?