We did some very cool direct mail testing with charity: water.  They’ve agreed to let us publish the results.  Some things worked, others didn’t.  We’ll share it all.     This will be a two-part post but worth the ride and we’ll also combine both posts into a downloadable, single pdf for ease of future reference.

charity: water is a brand that screams innovation.  They aren’t alone in that category but they fit squarely in it.  For starters, they signup a ton of monthly donors from a long-form video posted on social and digital ad networks.

And they don’t use direct mail. Ah, the horror!  No, it’s a rational choice.   Equally rational is their interest in testing the value of direct mail, but not simply as a stand-alone, fulfillment route.  They wanted a brand-building, tactile touchpoint adding to engagement and overall value.

And they didn’t want to treat everyone the same.  They wanted personalized matching with behavioral science.

Let’s parse that mouthful out a bit.

Brand-building.  If you think your direct mail isn’t building your brand and that isn’t its primary job then you should not be in the direct mail business.  How so?  The vast majority of people getting your direct mail don’t respond to it.  Does that mean it had no value for the 95% +?   It had an impact.  It either made those people more or less likely to give in the future and since that’s the massive majority, the impact on that group matters more than the tiny responder group.  It builds or destroys donor level equity, which is their future value.  That is the sane definition of brand, building future value.

Not in vacuum:  What if a mailing has hardly anyone sending back a paper check?  Did it bomb?  In most cases the analysis would say “yes”, bombed.  But what if you had an extended view that looked at the possible influence of that mail touchpoint on other fulfillment channels, namely digital?

Not everyone is the same.  It’s the norm for 99% of sector testing and fundraising to assume everyone is the same.  charity: water is not the norm and needed an approach to personalized matching with rigor and evidence behind it.

Here is the experimental design.

It’s a 2×3 design.  The Personalized Matching is based on Personality.  How did we know who was Open and who was Agreeable?

We have a proprietary, validated set of 3^rd party data for each of the Big Five Personality traits.  We match each donor to the profiles and assign them based on best fit.  Why Personality?  It’s mostly innate, fixed and predictive of what we’ll find interesting and relevant.  If your copy matches the person’s Personality, they’re more likely to see it, process it, agree with it and act on it.

The QR code test is self-explanatory, there was a QR code on the outer envelope.

The other version had the code on the reply form.  Here is that letter and reply.

The post card test was premised on the idea that these are existing CW supporters who, at a bare minimum, are willing to transact online (and may prefer it) and have no need for additional messaging or narrative beyond what a landing page can provide.

The Personalized Matching by Personality had differing letter copy. Everything else was the same – main character, images in letter, signer, etc.

What’d We Find?

There was a holdout group who got the business-as-usual digital marketing but not the mail.  And the test groups got what the holdout group got + mail.

First off, we got hardly any paper checks.  Shockingly few. In a vacuum, the test bombed, big time.  Fortunately, charity: water was as focused on the indirect lift as the direct.  So, all results are over a 45 day time period looking at online + offline giving for the respective groups.

Format findings

• QR Code on Reply & OE is directionally better than on the reply alone. I say directional because rigid significance testing puts the result at 80% confidence and the p-value police will insist on the arbitrary 95% or 90% to flag as ‘significant’.   This result is in line with what we’d expect, make it as easy and convenient as possible and don’t require someone to open the OE.

• Letter beats postcard. Did the postcard get lost in the shuffle of daily mailbox clutter?  Did readers need or want more than what the postcard provided?

Sub-story:  We did see the postcard do better with Agreeable folks than the Openness group.  This is in keeping with what we’d expect. People high in Openness are not only capable of processing more (interesting) info but they like it.  The postcard probably doesn’t ever match their preference and need.  Agreeable folks want and require and respond to less.

Kevin

P.S. Part two on Wednesday will show the Personalized Match Findings (i.e. content matched to Personality) and the digital testing that was done.