The headline from a study of headlines?  Headlines don’t matter.

This was a study of 141,000 headline A/B tests across 293 newspaper websites with big data, reputtable academics and a research lab whose name suggests it should be making Iron Man suits: the Northwestern Computational Journalism Lab.

Their goal? Figure out what makes for a “winning” headline. What’d they find?  Absolutely nothing. No repeatable patterns. No silver bullets. No “always use power words” rule that worked consistently. The headline world, it turns out, is random as hell.

Does that mean headline testing is useless?   Yes and no, that’s the twist.

They also found testing headlines in the moment for a specific story on a specific day often produces a winner. So Tuesday’s investigative piece might get 18% more clicks with version A over version B.

But none of those results are transferable. The “winning” headline doesn’t win because it’s inherently better. It wins because of contextual noise: the weather, the lead photo, the competition that day, the story placement on the page, your mood, your breakfast, the Twitter outrage cycle, etc.

In other words: the headline didn’t drive success, the headline happened to show up when success was already going to happen. We mistake coincidence for cause because we’re measuring at the wrong level.

What This Means for Fundraising (a.k.a., Real Life):

• Testing headlines can work — in the same way gambling can: sometimes it pays off. So do it. Flip the coin. Run the A/B test. But don’t confuse that with insight.
• Stop pretending the “winning” headline teaches you anything about headlines. It doesn’t. There’s no cumulative insight. Today’s winner tells you nothing about tomorrow.
• Headline testing is a tactical nibble, not a strategic meal. Useful for short-term lift, meaningless for long-term learning.
• If your entire testing program is headline-focused, congratulations: you’re optimizing the least scalable part of your marketing. You’re in the volume game — tweaking crumbs while ignoring the loaf.

The real lesson?  Stop obsessing over copy tweaks that only move the needle in isolation. Start asking better questions:

• Who are we talking to?
• Why do they care?
• How do we build something that matters beyond the inbox, or beyond Tuesday?

Testing can be valuable. But pretending it gives us universal truths, especially about something as context-dependent as a headline, is delusional.

Headline testing? Do it. But don’t draw big conclusions. And for the love of scale, stop mistaking tactics for strategy.

Kevin