Most surveys are garbage. Not because people don’t care about the results but because the design is garbage. It’s not an art form where you wave your hands, throw in some clever wording, and hope for the best. It’s science. Methodology matters. Survey design matters. Think of it this way: if you’re hiring someone to build a bridge, you’d prefer an engineer over the guy who once stacked some rocks across a creek.

One (of many) Big Problem(s): Nondifferentiation
It’s common and understandable to want to know what people find important. The non-expert approach? Just ask them directly. Ironically, that’s the worst way to get meaningful answers. Because respondents aren’t “forced” to think critically and create mental trade-offs, they don’t. Ask someone how important freedom, equality, and democracy are, and guess what? They’ll say all are “extremely important.” Congratulations, you’ve learned nothing new.

The Fix: Scaled Paired Comparisons
Instead, ask respondents to choose between pairs. Which is more important: individual rights or political equality? And by how much? This method makes people think. No coasting through with “extremely important” on autopilot.

Compared to traditional rating scales, scaled paired comparisons:

• Increase differentiation: People actually reveal preferences. Imagine that.
• Reduce satisficing: Less lazy responding because the task requires real thought.
• Improve reliability: More consistent results.
• Improved validity: the survey better reflects what people really think
• Takes less time to answer.  Win-win, better data, less burden on respondent.

Here’s a study compared traditional ratings with scaled paired comparisons on a set of values (e.g., individual rights, civil rights, free press, checks & balances). The findings? Rating results asking “how important is____?” are clumped together on the left, everything is important.  Paired comparisons spread out nicely, showing clear hierarchies of what people actually value.

And how do we know that the paired comparsion answers are more valid?  The ratings for Republicans and Democrats aren’t very different – that makes no sense. Breaking out paired results by party shows what you’d expect, Republicans assign much more import to individual rights than Democrats and less import to civil rights.

The Bottom Line
If your survey results are garbage, maybe it’s because your survey is garbage. Want to fix it? Treat survey design like the science it is. Hire experts. Sure, they cost more than the intern who took Psych 101, but so does hiring a structural engineer over a guy with a stack of rocks. The difference? One gives you a bridge that works. The other gets you wet.

Kevin