You may have heard the article of faith that donor surveys only give you misleading information. That statement is half right: bad donor surveys are bad. But, as you make have guessed, good surveys are not only good but essential.

But first, you must get past the jam trap. (This can involve complex technical names like biases in hedonic forecasting, impact biases, empathy gaps, Lombardi sweeps, etc. but it’s easier to think this way to me)

Researchers asked people what brands of strawberry jam they like. (The full study is here.)

When they asked people blind, their rankings were generally pretty close to expert opinion. However, when they asked people why the liked the various jams, not only did they have no idea, but their preferences changed away from this expert opinion. To quote the researchers:

“No evidence was found for the possibility that analyzing reasons moderated subjects’ judgments. Instead, it changed people’s minds about how they felt, presumably because certain aspects of the jams that were not central to their initial evaluations were weighted more heavily (e.g., their chunkiness or tartness).”

Yes, in a sort of market research Heisenberg effect, the act of observing affected the observed.

So, people are good at telling you what their preferences are. They can’t, however, tell you why their preferences are. And as a result, the subjects were not, in fact, ready for this jelly.

OK, you say, but I know it’s important to know why my donors do the things they do. How do I figure that out without stepping in the jam trap?

We do it by first asking a person’s commitment to an organization and then only asking people’s opinions about their experiences with that organization. From there, we can run the math on what experiences matter and which do not. Here’s a video about how Operation Smile increased retention through this type of analysis:

They didn’t ask broad questions like “why did you lapse?” or “are we mailing too much?”. These broad questions are too cognitively challenging and encompassing too many things.

Instead, they asked how committed a person is and how satisfied they were with how many times they were being asked. From those data points, they could see the relationship between the two numbers.

So, the key questions:

• Are you asking donors for feedback? (If not, know that everyone now does, including the Postal Service. You don’t want to have worse customer service than the USPS, no?)
• Are you asking what or why? Asking someone why they believe what they believe is a recipe for bad data.
• Are you asking your other questions effectively? Kevin’s post on donor survey myths and facts can be a great start.

Your thoughts? Please leave your feedback in the comments!