Beware of Junk Science

March 17, 2021      Kevin Schulman, Founder, DonorVoice and DVCanvass

The Myers-Briggs Type Indicator is the most popular personality test in the world.  Fortune 100 love it.  Government agencies love it.  More than 1.5 million people take it every year.

Our only issue with it as social scientists is this:  It’s absolute garbage.  Otherwise, we too love it.

It fails on two fundamental requirements.

  • Not Valid – the measures don’t predict outcomes.
  • Not reliable – the same person can take the test two separate times and get very different answers.

As the kids would say, this is big, freaking problem. (I have no idea if kids say this but if they did it would come in the form of  incomprehensible shorthand on my kids’ tok-tik, instachat, snapgram accounts)

This got us thinking about all the other popular, equally junky, junk-science out there.

Here’s our list… our critique… and our suggested replacement for each.  We’ve written extensively about all of these and therefore include a link to the ball of thread for each.  And frankly, it’s enough thread to knit two ugly Christmas sweaters for every man, woman and child that wants one.

Junk Science

#1.  Myers -Briggs.  Not valid, not reliable, otherwise perfect.

#2.  Net Promoter Score.  NPS is a single, “willingness to recommend” question that must have the same kick-ass PR agent as Myers-Briggs because it’s super popular despite also being garbage.  Ball of thread. 

#3.  Generational Marketing.  This is our tilting-at-windmills passion.  The more data and evidence we surface that generational marketing is akin to reading tea leaves or your horoscope, the more the true believers, believe.   Ball of thread.

#4.  Personas.  Not a single person in Persona Group A matches the profile.  Not a single person.  Most people vary wildly from the theoretical average, especially when the Personas come from the Cluster F*$(% of cluster analysis.   Ball of thread.

#5.  Satisfaction as the measurement.  Having satisfied donors is better than dissatisfied.  But, the concept shouldn’t be seen as the measure.  Overall satisfaction or even satisfaction with a specific interaction is too crude a measure.  Ball of thread.

Speaking of thread, here is the red thread that cuts through all these pseudo-science concepts.

Common critique for all these:

  • No theoretical basis. There are also junk theories behind junk science, so claiming some theoretical basis is not a cure-all. But, the absence of  it is a death-knell.
  • Lack reliability and validity.  Everyone remembers these terms from stat 101.  They matter.
  • The grouping of people is arbitrary.  All the Myers-Briggs  types are arbitrary as are the Promoter/Detractor groupings for NPS.
  • There is often more difference within than between the arbitrary groups

Fortunately, better science exists.

The future is here, it just isn’t widely “lived” in.  Here are the existing, validated, already-in-use-by-charities  replacements.  (The numbering here matches the Junk Science numbering above)

#1.  Personality Big Five.  Personality is largely genetic and very predictive of what a person will pay attention to and find interesting –and most importantly, the choices one makes.   Ball of thread.

#2.  Commitment.  This is an attitudinal measure of loyalty.  It is based on Relationship Theory and it’s easily the best loyalty predictor on the market because in addition to predicting, it also provides cause and effect answers.   Ball of thread.

#3Demographics are useful as proxy data and for targeting, which is all about efficiency.  Demographics are almost always garbage for understanding the “why” of behavior.  We use demographic data as a proxy to score, for example,  acquisition lists or house files with Personality segment data. ).  Ball of thread.

#4.  Identity.  Each of our –and our donors– multifaceted personal narratives.  Much of charitable giving is tied to reinforcing one’s sense of self – e.g. conservationist, Globalist, disease caregiver, Dog lover.  Ball of thread.

#5.   Need Satisfaction.  There are three key, psychological needs that humans subconsciously want met when interacting with a charity: autonomy, competence and relatedness.  Your copy, scripts and design should be audited through this lens.

You also want to measure supporter experience after each interaction.  Make it part of your business process– giving it equal weight to collecting payment details.

But, don’t just measure. Act on the data in the moment and in aggregate to fix broken experiences.   Ball of thread.

What else would you add?  What do you disagree with?

Kevin

4 responses to “Beware of Junk Science”

  1. John J Glier says:

    Well said, Kevin. It’s time we got more rigorous about the frameworks we use as tools in our work.

    • Kevin says:

      John,

      Thanks for commenting, the rigorous work is out there and being used, just occasionally drowned out by the work that seemingly hired the better PR agent…

  2. Susan L Sweitzer says:

    The Myers Briggs was originally a marriage/compatibility type instrument developed by a mother-daughter team, long ago. The primary purpose was to realize and appreciate differences/perspectives within a relationship. It always amazed me when marketing caused the corporate world to adopt it so completely. Some even had you take it so they could have ‘more balanced’ teams.
    I’ve taken it several times and scores DO change. Mine did after I gave birth (I was high on the ‘thinking’ scale beforehand and that score wasn’t as pronounced after becoming a mother).
    It is a tool in the psychology trade and a good one, and was never meant as anything else. Shame on companies or nonprofits that use it to construct their teams or assess staff/donors.

  3. mary anne says:

    Also, Isabel Briggs Meyer as a person is not, perhaps someone whose worldview aligns with the readership of this blog. There is some evidence that she was pretty racist….

    https://www.moviemaker.com/persona-the-dark-truth-behind-personality-tests-isabel-briggs-myers/