Are Demographics Garbage?

August 5, 2016      Roger Craver

I was struck by the headline on a piece in Fortune that reads:

Netflix says Geography, Age and Gender are “Garbage” for Predicting Taste

The point of the article: Netflix uses one predictive algorithm worldwide, and it treats demographic data as almost irrelevant.

“Geography, age, and gender? We put that in the garbage heap,” VP of product Todd Yellin said. Instead, viewers are grouped into ‘clusters’ almost exclusively by common taste, and their Netflix homepages highlight the relatively small slice of content that matches their taste profile. Those profiles could be the same for someone in New Orleans as someone in New Delhi (though they would likely have access to very different libraries.)

Says Fortune, “Netflix  seems to have discovered (or built on) a powerful insight from sociology and psychology: That in general, the variation within any population group is much wider than the collective difference between any two groups. So if you want to, say, get someone to stream more of your content, you’re better off leveraging what you know about similar individuals in completely different demographic groups, than trying to cater to broad generalizations.”

What about the use of demographics in fundraising?

The folks over at our sister company DonorVoice believe that the vast majority of demographic data appends are “worse than useless”.

In a post titled Demographics Are Garbage, Kevin Schulman, CEO of DonorVoice, makes the following points:

  • You know your donors are older and that older donors stick around much longer than the younger ones, so how can age, among other demographics, be garbage?

Netfix 2Kevin argues, “What Netflix knows from their mountain of data (movie selection behavior mostly) is that people are in fact different, but not in the ways you’d imagine. Your donors are also different but not in any of the ways they are typically segmented — e.g. by RFM buckets or acquisition source.”

“Here is the critical nugget — the Netflix mountain of data, just like your mountain of data, is 99% garbage, 1% gold.

In the garbage pile are age, gender and geography. Think about this, geography is what makes up the cluster codes (e.g. Prizm) with the rationale being that people who live near each other are similar.

“That this is partly true misses the much larger reality, those similarities to our neighbors are dwarfed by the differences with our neighbors.”

  •  Netflix’s reality, which is the same as a nonprofit’s reality, is that differences within general/stereotypical groups (e.g. females) are greater than the differences between two general/stereotype groups (e.g. females and males).
  • The alluring but very superficial, dead-end trap that charities fall into — a trap Netflix avoids — is seeing a difference and mistaking the result as either optimum or causal. Case in point: if Netflix served up romantic comedies (i.e. ‘chick flicks’) then more females than males would choose that genre.
  • BUT, notes Kevin, this fact (that more females than males watch romantic comedies) obscures a missed opportunity. Why? Because there are females who don’t like ‘chick-flicks’ and some males who do. Missed opportunity in both cases and sub-optimum financial results from sub-optimum thinking — i.e. females choose romantic comedies because they are female (wrong).

Applying the Netflix analogy to testing in the nonprofit sector, Kevin points out that “the mailing label test using generalized, stereotypical assumptions about male preference – e.g. anchors and boats – and female preference –  e.g. flowers” — stands as an example of the “very limited, skim-the-surface type of mentality that keeps the charity sector on a slow decline trajectory.”

To put it another way, “Using these weak, demographic profiles and thinking you have some unique insight is akin to renting names to mail and thinking you have a unique universe of donors. Everybody does this, it is a ‘best practice’ to nowhere.”

What’s the alternative?

Netflix has customer clusters (i.e. segments).and those clusters are based on likes (and by extension, dislikes). In short, preferences.

And this is where the importance of understanding donor identity and preferences comes in. For example, Kevin recommends “that organizations think about a group of donors on your file who:

  • Have disease X that you are in the business of curing.
  • Like big predator birds (not small ones) if you’re in the business of conservation and birds.
  • Know someone who was a beneficiary of your charity.
  • Have a strong religious motivation to give even though your charity is secular.”

“If you profile these groups by their demographics you will see that they skew this way or that way (e.g. older, female, whatever…).  That is as inevitable as it is irrelevant.”

Key Questions

Kevin concedes that some demographic information — specifically ‘age’ and ‘income’ — may be useful for efficiency sake and perhaps for identifying potential planned givers. But, beyond that he wonders, “Why pay extra just to muddy the waters?”

Then he goes to the heart of the matter and asks:

QUESTION: “Why in the world is it beyond the capacity of a charity or any business to group folks based on preferences like the examples raised above?

ANSWER:  “There’s not. Doing so involves a mindset shift, not one of resource or database functionality. It is one that requires not thinking of donors as widgets on an assembly line. Or maybe it is two assembly lines; males and females … or is it old people and not old people …”

How are you using demographics? And why?

Roger

7 responses to “Are Demographics Garbage?”

  1. Alan McKeon says:
    August 5, 2016 at 8:16 am

    Interesting post – and from a conceptual viewpoint correct. However, it misses two key points:

    1) Netflix collects data on behavior that is highly relevant to them (movie selection) passively, continuously and at negligible cost. It can then apply that data directly to influencing the next action a consumer takes (choosing a movie) which has a positive loyalty feedback loop for them. No non-profit I know of is gathering mission relevant, behavioral data continuously in a similar way.

    2) Once you have these donors clustered into audiences, how do you “land” your campaign by targeting new look-a-like donors, or reach your existing donors at the exact point they are ready to take action? Again for Netflix it’s easy, the consumer is looking for a movie they want to watch – now or later – suggest somethings they will like based on their past behavior and watch if it is viewed now or goes on a watch later list. For example, how would a non-profit identify and target people who “Like big predator birds (not small ones) if you’re in the business of conservation and birds.”

    The closest we have today would be to use our own digital platforms to mine the clickstream data, or pay one of the behavior aggregators such as Facebook to create our own audiences and then then run campaigns through that same aggregator. Is this better than direct mail based purely on demographics and geography? Yes, political parties have proven that. Does it replace the broader insights that help craft strategy and campaign messages? No, for that you need the broad brush of aggregating donors based on human-scale categories that the organization can understand and rally around, and for this demographics will continue to have a role.

    I love that Roger continues to challenge our thinking on these topics, but caution against throwing the baby out with the bath water.

  2. Gregory Warner says:
    August 5, 2016 at 9:31 am

    I’ve seen a lot of folks promote “life-stage” marketing for planned giving outreach. That’s BUNK too! It’s the same as demographics really.

    There are two categories of data nonprofits must capture when it comes to raising major and planned gifts…. verbatims and digital body language.

    Verbatims are what they say (usually in face-to-face meetings or calls, or surveys).

    Digital body language is what they do (captured online if you track each donor’s user experience and score important actions).

    Do that stuff before tinkering with demographics!

  3. Jason Sears says:
    August 5, 2016 at 9:58 am

    How would you put together an acquisition, Roger?

    Any professional who puts together acquisitions knows that donors are often predisposed to give to organizations geographically closer to them. I’d venture to say that ignoring this fact would be devastating, and particularly so for non profits without a national appeal (most).

    Simply put, there isn’t a non profit in the world with the accessibility to the mountains of historical data Netflix has (or to the quality of data engineers they employ that can make the assessments noted above with certainty).

    There’s a lot in this article that makes a lot of sense, diving deep to determine your donors behaviors and preferences in lieu of weak demographic bucket, but this article ends up being a thought piece without much substance. I’d be much more interested in reading a article that fleshed out the ‘What’s the Alternative?’ segment.

    And let’s be honest, we all know women aren’t all flowers and men aren’t all anchors. Diving deeper is necessary. How?

  4. Kevin Schulman says:
    August 5, 2016 at 10:26 am

    Alan, thanks for the thoughtful reply. A few considerations.

    1) Agreed, non-profits aren’t gathering mission relevant behavioral data but this is very different from saying this data isn’t already there and therefore, requiring incremental time and effort to store it. In fact, lots of coding and storing tied to behavior and “movie” behavior (read: fundraising appeal) already gets done, it is just done poorly. For example, coding that someone replied to the annual appeal is not nearly as useful as coding that someone replied to Message A (vs. B). All that information exists – in spades in digital space – and it is being coded to some degree, just not enough and when done, using a woefully inadequate (but still complex and sucking up time) schema. This is a mindset problem.

    2) There is a reason Netflix asks for movie ratings from subscribers. That charities, by and large, don’t continuously collect this same preference/rating/feedback data on their interactions is, again, a mindset problem.

    3) The answer to your question of how does a charity identify and target people who “Like big predator birds (not small ones)? You ask them for starters. This is quite simple to do and if the business assigns value to the answer – because they have evidence and data on what to then serve up to make the next something more relevant – they will devise process to get this data for as many people as possible.

    This isn’t idle, pie in the sky talk. We have clients doing this at point of acquisition – i.e. asking the one thing they need to know in order to greatly increase the probability that the next something is on the mark.

    The direct asking can of course be augmented by behavior click data with the caveat that one should never assume a behavior is a preference. Merely a choice and in many cases, a sub-optimum one from the donor’s perspective.

    Demographics will always describe and this can be useful for targeting and selection, which is all about efficiency. To be effective charities need to get into the why business of human behavior. No matter how much you torture behavior data it will never tell you why other than as weak proxy.

  5. Ben Miller says:
    August 5, 2016 at 10:44 am

    I think a larger point here is being missed. It is not that Demographic data is garbage, or that categorical preferences are gold. It is not there is no correlation or even causation with demographic data, or that the differences among 55 – 60 years old are any better/worse than those that like big predator birds. My guess is the variability in both those sets of data would be pretty high.

    Think about some of the most popular Demographic variables used today; Income, Real Estate Value, zip code, all pointing to wealth. There will be a statistical relationship between these variables and those donors giving at the $1,000,000 level. Why because it is very rare that someone is able to donate that amount of money and lives in an impoverished zip code.

    The difference is that not all people who are rich also want to donate to your organization, let alone donate that amount. That is why transaction based modeling like we do at DonorTrends and like what is done at Netflix always reigns supreme. If I have given 24 donations, over the last 6 years, I like your organization. If I have watched 8 action movies in the last 3 months, I like action movies.

    Ultimately it comes down to whether the cost of appending external data outweighs the benefit (lift) you receive from using it. In my experience it is rarely the case, as you are able to get a much higher benefit (lift) using transactions or big bird preferences both of which are available at no additional cost. (Here I am assuming you track gift transactions and which campaigns they have responded.)

    I will caveat this by saying there is some disagreement out there particularly how it relates to auto-correlation, but it is also undoubtedly true that past behavior is the best predictive of future behavior.

  6. Kevin Schulman says:
    August 5, 2016 at 12:03 pm

    Ben,

    First off, hope all is well. A couple counterpoints to consider,

    1) in the scenario (real but modified to provide some client anonymity) of bird predator we know this cohort has specific content/product preferences. There is variance to be sure and further segmenting within this cohort can and will be done (not on RFM though) but it is unquestionably the case that this group has far less variance in their receptivity and response to a Comm designed for them as compared to the normal charity bucketing of RFM group or model deciles 1-3.

    I would go so far as to say demos are never causal. If my goal is a million dollar donation then yes, I am only going to ask people with the capacity to give that kind of money. As you point out, the vast majority of people with capacity have no inclination or motivation. But, even within the group of rich people who did give a million to Charity A it ain’t because they have a million to give. And within this tiny, fairly homogeneous group of rich folks who have interest in Charity A there is still variance on root cause of why they give.

    Those reasons can be identified, used to segment and used to determine marketing approach, content etc… This is likely done by major gift officers and those operations. It can be done at mass as well and it has nothing to do with resource, just mindset and what data we elect to assign value too.

    Past behavior being the best predictor of future is a circular truism that nevertheless does result in greater efficiency with selection. Of course you can efficient your way right out of business as the circular logic takes hold and starts to swirl round and round (like water in a drain).

    What Netflix is focused on is preference and interest. Past giving (and what they gave too) can be considered a proxy for both but it will never answer why and it never gives any answer on the vast majority of donors; the non-responders.

  7. Ben Miller says:
    August 5, 2016 at 11:49 pm

    Kevin, doing well as I hope you are too.

    Full disclosure, I actually hadn’t read your comments at the time of my post, so my comments were not pointed toward yours.

    That being said, there are two points in your reply that I could not leave unanswered.

    1. …but it is unquestionably the case that this group has far less variance in their receptivity and response to a Comm designed for them as compared to the normal charity bucketing of RFM group or model deciles 1-3.

    The thing that has always drawn me to math, is that things like this have a clear answer. Variance within a group with one variable versus a predictive model that is mathematically designed to minimize such variance, well I hardly think it is unquestionable. In either case, my real point wasn’t whether any one tool was worthless, I think all three have their merits. Use the hammer to nail, the saw to cut, and the paint brush to paint.

    2. …you can efficient your way right out of business as the circular logic takes hold and starts to swirl round and round (like water in a drain)

    Regardless of whether hyperbole is the best method of debate, I hope you will concede that predictive modeling as a tool does not equal efficiency as a strategy. Nor does using predictive modeling preclude one from also tailoring messaging to their constituents’ preferences or vice versa.

Ask A Behavioral Scientist

    Behavioral Science Q & A

    The Agitator Tool Box

    Ideas, applications, tools, processes, and case studies of break-through solutions in fundraising, including: