Can a statistic be bad?

Maybe “bad” is  a bit of an an overstatement but the “Average”, a universally used statistic can sure hide a lot.  And without assigning malice or intent to the “hiding”, reliance on the average as an input to decision making can result in lots of bad outcomes.

There are countless examples,  but consider two for illustration.  Education reformers – regardless of political orientation – can agree that reporting average standardized test scores for a given school (or even classroom) can hide a whole lot.

In this case, many would argue there is intent.  One of the requirements of No Child Left Behind was to break out test results by race/ethnicity.  The result?  A bright light shone under the bushel with whites (and Asians) doing much, much better than African Americans and Hispanics.  Just using the school or classroom average (shamefully) hides this.

Another example, donor satisfaction averages (or other survey measures).  Imagine two nonprofits with the exact same average donor satisfaction score of 50 on a 0 to 100 point scale.  Further imagine Nonprofit #1 has all its responses at a 50 and Nonprofit #2 has half at 0 and half at 100; radically different situations requiring radically different treatment.

What is Average?

The average exists based on combining a bunch of “non-average” results.  In other words, for a given outcome, average is our best guess but it is also wrong the majority of the time.  Sometimes it is only a little bit wrong, sometimes a lot.  If the distribution is really screwy – consider the earlier example of half the donors at 0 satisfaction and half at 100 – then it will be really wrong ALL of the time.

It is a fact that plans (e.g. direct mail plan, strategic plans) based on average assumptions are, on average, wrong.  This is not just a clever turn of phrase to discard. It doesn’t take a crazy, extreme distribution like the earlier examples to lead to bad (enough) outcomes a lot of the time when relying on averages.

How to Correct It?

Report other stuff too…

Using averages even for reporting (versus planning), which we admit to doing all the time, is flawed.  At the very least, one ought to report on the ‘spread’ of those scores/results – the standard deviation.   This is typically not done either because it is not understood by those doing the reporting or feared that it won’t be understood by those receiving the report.   This or any other reason is not good enough to warrant reliance on the use of averages

What about getting away from averages all together? Radical?  Maybe not when you consider that direct mail does not work on averages.  The average result from a direct mail campaign is failure – i.e. non-response to the offer.  It is the “extreme” outlier (responders) that matters.  This is not just conceptual think aloud criticism.

If we accept that direct mail responders represent the “extreme” outcome when putting them in the universe of all those mailed, shouldn’t we also be willing to accept that the responder universe (forget the non-responders now) also has extremes?  Of course it does.  And yet, we report on averages to describe the responder universe (average response rate, average gift).  The average in those cases is potentially no less misleading than when thinking about the average outcome for the universe of the “mailed”, which is failure (and yet we keep mailing for obvious reasons).

So what about these responders?  What else can we do?  We can certainly look at the distribution of giving.  One can compute a range (i.e. highest and lowest gift amount received, a mode, etc.  What about the distribution of response?  This is usually done by looking at the list source and calculating response/performance against a smaller universe of the mailed.

All of this additional analysis results in segments of responders.  And if we think about segments as nothing more than taking a single average and creating several – one for each segment this gets us to several profiles or patterns of behavior, not just one.  All positive steps…

1 to 1 marketing

But what about no statistics?  No segments.  The notion of 1 to 1 marketing, often just words on a page, is really about – when done right -, no statistics, no averages.  This is actually done all the time.  It is the reason you see ads for particular brands/products on your Facebook page if you  mentioned that particular brand in your post.  You receive this ad because of what you specifically did or indicated.  Compare this to you seeing a banner ad on a magazine website that is there because you and everyone else who visits the site fits a certain profile or segment description that certain advertisers find attractive.   You, as an individual, may actually represent zero potential to the advertiser because you may deviate dramatically from the ‘average’.

This is not just the domain of big, commercial marketers and brands.  Nonprofits can play in this game too.  And even if you elect not to, every organization can start to examine and understand the flaws that come from over-focusing on averages.

Let’s return to our classroom example.  Breaking it out by segment is great.  But, what about the distribution within the segment?  Do all Hispanics have poor test results?  Of course not; some do very well, some do really poorly and on average, they do less well than whites.  A treatment or remedial plan at the segment level would under-serve those at the extreme ends of the segment – really poor results and good ones.

Direct marketing to donors is no different.

Kevin