It’s tempting to say that every improvement is a win.  You got an extra three percent on response rate or average gift nudged up.  And it is to a significant extent.

But the question you must ask is what is the goal toward which you are optimizing.  To re-use an analogy, let’s say you wanted to climb the highest mountain by continually climbing the highest hill you could see and repeating.  If you were to do this in the village where I grew up (Greendale, WI) with a good set of binoculars, you would end up in Hales Corners – elevation 868 feet.

Progress, yes, just like that average gift or response rate boost.  But there are higher mountains to climb.

Analytics guru Avinash Kaushik (whose first book Web Analytics: An Hour a Day I highly recommend and who now has Web Analytics 2.0 out in case you were wondering what to get me for Christmas) just talked about optimizing for the global maximum, rather than the local one, in an excellent blog post here.

In it, he talks about how if you watch NFL games, you will see the Microsoft Surface in advertisements and in the hands of coaches and players.  They are doing great on reach and brand lift; they are at their local maxima.

(Source: AP. Associated Press, not Adrian Peterson)

But Surface has a .29% market share.  They haven’t solved for a global maximum goal.

This has two implications: one for how we test to get our goal and the other for what our goals are.

In terms of testing, a traditional A/B test for what teaser we should use or what color the envelope should be is locally optimizing for a locally optimized goal.  You may get improvement, but it’s specific to that communication at that time.

A way to break out of this is to go back to your seventh-grade science project: start with a hypothesis.  More specifically, start with a hypothesis that, if proven one way or the other, will change how you do business.

Take the Nudge-Award-winning test that showed UNHCR that it got a 42% lift when it presented its symbolic gifts symmetrically (e.g., five blankets, seven blankets, nine blankets versus blankets, radiator, and stove).  This is something that, while small, changes every donation form and every response device they use.

It’s even more powerful if you can test thousands of versions at once to find concepts that work for you globally.

But Kaushik’s look at global optimization also begs the question of whether we are looking at the right goals.

There is nothing wrong, and most things right, about goals like response rate, average gift, and net revenue per communication.  Or larger goals like hitting your net revenue budget and file size.

But all these could use a refinement like the one Kaushik recommends for the Microsoft Surface: instead of measuring brand recall, test whether people are more likely to choose the Surface (since buying is the behavior you are shooting for).

So, for example, do you want to measure file size?  Or do you want to measure file size of the donor segment with a connection to your cause, which you’ve determined is way more profitable (nonprofitable?) than those donors without a cause connection.

In the end, as the Blackbaud Vital Signs report concludes “American donors are more valuable to American nonprofit organizations than the organizations are to the donors.”  A global maximum metric needs to be, or approximate, whether you are leaving your donor potential greater than you started.  A “successful” mailing that nets $100,000 isn’t if it turns off donors with lifetime values of $200,000.

We measure this in commitment; you may have your own measure.  But it’s worth keeping as a North Star in your analytics constellation.