This week, we’ve seen how normal reporting on Facebook and Google advertising is flawed because many of the donations that are directly attributed to advertising would have come in anyway.

But there are two important additional implications we haven’t discussed:

1. Conversely, the direct attribution model ignores revenue lifts outside of the direct click, or even the channel.
2. This ain’t just online.

On the first point, take DRTV, one of the best examples of a channel that acts as the rising tide lifting all boats.  After all, no donations are actually made through your television (yet!).  A CDR/Wounded Warrior Project analysis found that of the sustainers that came in because of DRTV, 86% came through in through means that could be tracked to the ads – either the call center or the featured microsite.

That means that 14% went into other channels: 12% through the main website (a spike in traffic and donations within ten minutes of the ad) and two percent through direct mail.  Yes, one out of every 50 sustainers from DRTV saw the ad, then used the reply device they had in their mailbox.

This isn’t an isolated incident.  The poor organizations who ditched mail because it was too expensive (this is “poor” in the sense of deserving of sympathy and subsequently in the sense of lacking resources) found that a decent portion of their “online revenues” went missing.  Likewise, emails that tell people an important mail piece is on its way to a mailbox near you and online cotargeting ads that support the message may not make their money from last-touch attribution, but could end up helping the program overall.

So it’s tough to test something against nothing because of both false positives and false negatives.

This isn’t just an online problem.  As we’ve discussed at some length, when you add another communication to the mix, you can not say with intellectual honesty that it summoned all the revenues that are “directly attributed” to it out of whole cloth.  Rather, we know that about 63% of a new mail piece’s revenues aren’t new – they would have been earned from the surrounding mail pieces.  This cannibalization rate is even higher among organizations that mail frequently, but has been documented even when an organization moves from two to four mail pieces.

In other words, when offline people point fingers at online strategies not paying their way because they aren’t an improvement over organic listing, someone’s a pot, someone’s a kettle, and they are both black.

So, with all the difficulties in testing something versus nothing, what’s a savvy direct marketer to do?

I wish I had an easy answer.  But the real answer is don’t not do it – do do it.  (This is why I was kicked off the Nike marketing team.)

Seriously, though, if you want to figure out if more or fewer mail pieces will work better, you need to do what Catholic Relief Services did and test it for a period of time with a test and control set of donors.  If you want to figure out how much your search ads are cannibalizing your organic results, you need to do what eBay did and turn them off for some donors.  And if you have a multi-channel, multi-communication campaign, you are going to have to add some channels/communications/ads/etc. for some people and subtract them for others.

All testing is an imperfect science.  But in order to be anywhere close to “science”, we need to be continually refining and discarding what does not work.

Nick