There are No Best Practices
That is our headline from an analysis of (newspaper) headlines that found no discernible pattern in determining what makes for winning headlines. I know, dizzying.
The analysis was performed on a a big data set: 141,000 A/B headline tests run by 293 newspaper websites. The project was done by academics at Northwestern’s Computational Journalism Lab. I have no idea what those three words mean in combination but it sounds impressive.
Their aim was simple; identifying what makes for a winning headline. For example, do attention grabbing headlines or negative ones or positive ones work best at garnering the most traffic and eyeballs?
What did they find? A nothing burger. Zippie. There are no universal best practices or winning, ingredients that consistently produce a winning headline.
Does that mean you should ignore headline testing? No. What they also found is that doing A/B headline testing for a given day of the week and a given story matters. One can increase traffic on a Tuesday in the middle of October with an investigative article if doing an A/B headline test.
How to square the two? There’s no set of headline best practices that reliably and consistently predict winners. But, one can and should do headline testing for given stories (or in our world, campaigns) to increase attention.
These two findings in combination tell us there’s an awful lot going on external to the headline copy that drives eyeballs for a given news item and a given day that is causing us to declare a “winning” A/B headline. We can further learn that the headline that won was mostly random and had very little to do with the success of the day since that success can’t be piled up to increase the chance of successes of the week, much less month or year.
When unmeasured, unknown factors still get you to a winner in the moment you should switch to that winner for the rest of the very short newsday (or email campaign in our world).
To sum it up:
-
- Headlines have very little to do with garnering extra traffic, attention and clicking.
- An A/B headline test will often produce a “winner” that we mistakenly attribute to the headline but only because we don’t have the unknown, more explanatory factors measured.
- Point #1 & 2 should not stop us from making more money from headline testing and quickly switching to the winner for the rest of that campaign
- Point #1 & #2 should stop us from drawing many, if any, sweeping, declarative “top 5 winning ingredients for a successful headline” claims.
Kevin
7 responses to “There are No Best Practices”
Behavioral Science Q & A
Integrating an individual giving appeal with other communications from a charity can have both positive and negative effects, and the outcome largely depends on how it’s executed. Advantages of Integration Brand Consistency: Maintaining a consistent appearance and messaging across all communications can reinforce the org’s brand identity and strengthen brand recognition and trust among your […]
Read Full Answer
I’m not aware of any in-market tests specifically comparing recurring vs. gift frequency language. I suspect the answer might not be the same with all gift frequencies, nor with all people. It sounds like a great opportunity for you to test and find out what works for your audience. Based on the literature, here’s a couple […]
Read Full Answer
Based on what we know from existing data, those renewal notices can actually be pretty effective in getting people to donate. They tap into our psychology – creating a sense of urgency, reminding us of past support, and using personalization to make the message hit home. They’re playing on our natural tendencies to feel obligated […]
Read Full Answer
Interesting question. I had a quick look at the testing done on this topic. On the positive side, in all cases, over half of donors decide to cover the fee. In some cases, it goes as high as 65%. Not a negligible percentage at all. Here’s another test from iRaiser showing consistent results (see point […]
Read Full Answer
There’s just one thing to consider when designing a supporter journey: the supporter. More specifically, you need to take into account: Who the supporter is i.e. their identity, which is the reason they support this cause, and their personality, which describes the way they “see” and process the world. These will determine the kind of […]
Read Full Answer
I’m not an expert in this but a quick search surfaced this article on the effect of tax reforms on 2019’s charitable giving. The researchers didn’t find a reduction. Actually, they observed an “increase in charitable contributions in 2019, even with the lower tax rates and the dramatically smaller number of taxpayers who itemize their […]
Read Full Answer
I’ve been on a crusade to stamp out the use of “best practice” when what people are really talking about is common practice, not best practice with a definition of evidence-based. It is rampant in the world of nonprofit boards advice … term limits as best practice, board members have to fundraise (or 100% board giving) as best practice, committees as best practice, etc. etc. There are very few practices when it comes to board governance that have any research behind them. Yes, there is a growing body of research in general on boards, which is a good thing, but the cases and tests include terribly few npos and are very limited in types of public charities they apply to. (the term limit research we have is on legislatures) https://www.pegasuskentucky.org/post/2019/07/31/term-limits-are-popular-there-is-little-evidence-they-are-effective So many we can just stop cavalierly using best practice in this sector and share common practices, with a strong caveat that is all they are. Thankfully, fundraising is much more studied, by academics and practitioners alike.
Hi Gayle, I like it, common practice replacing best. Best is so declarative of victory, there is no incentive to push further, expand our horizon and go from better to better still. Common has the added advantage of prompting the competitive juices to avoid doing that which is just common.
DonorVoice has lots of ‘best’ practice though our best practice of today is very different and improved from that of last year. If we are any good at continuous improvement, we’ll be able to say the same thing next year. Way too many “best” practices have enjoyed inordinate amount of time atop the hill but only because way too many are satisfied leaving those ideas unchallenged.
Love your framing Kevin.
Kevin,
So maybe it was–gasp–content, what the story was about, that mattered? But it sounds like that was accounted for. I’ve heard so much about “headlines (and subject lines) should be no longer than x and no shorter than x,” should have powerful/action words, etc etc. And also, do you mean clickbait headlines “10 ways to lose weight” garner no more views than “Joe Biden Does X”? I guess I’ll head over to the analysis from Northwestern. Fascinating though, thanks for your efforts, as usual.
Hi Yvonne,
The story was accounted for only in so far as there was coding to see how well headline matched story – e.g. was negative headline linked to negative story. The analysis did not tag and code all parts of the story as predictor variables so yes, that could be one of the many unaccounted for, unmeasured factors.
There was some modest evidence for negative working better than positive and click-bait headlines doing better but only at the margins and not in a predictable way. Many of the winning A/B tests became losers when they were re-run, for example.
I think a heavy, heavy dose of modesty in our “winning” anything claims is in order and not just for headlines. There is always an enormous amount of noise in the signal and our signal measures are given too much credit as the causal lever.
Kevin, let me know if you want me to send you one of our old Chuck Norris “Punch best practices in the face” t-shirts from the NIO Summit….if I can dig one up 🙂
Jeff, that’s brilliant, love it. If loving Chuck Norris memes is silly and immature than color me twice with those brushes.