Nearly two years ago, I set out on a journey to understand how nonprofits and the social good sector uses data. What I found along the way is revealed in the new book Data Driven Nonprofits.
An important topic that I wanted to tackle was the proliferation of bad statistics. They are the unintended consequence of nonprofits, consultants, technology providers, and the media paying more attention to data, but not necessarily the details.
The problem is that bad statistics take on a life of their own when released into the wild. They get repeated, distorted, and over time no longer resemble their original form. It’s like the telephone game we played as kids, but with bigger consequences.
Then statistics like “one research study found that one-third of all U.S. giving happens in December” get reported and people assume that it’s true — when it’s not. It turns out that less than 20% of all charitable giving happens in December and this trend has held up for a number of years.
For years now, I’ve worked to try and debunk bad stats when they pop up. In some cases that has also meant trying to answer important questions using large sets of representative data and sound statistical practices. Sometimes that has meant approaching people about a statistic that has flaws in it and asking them to clarify the situation.
While I was working on Data Driven Nonprofits, the author, strategist, and media expert Brian Reich tweeted, “I take issue with Blackbaud’s data (for example, their ridiculous claim that 95% of people give).” We exchanged tweets and I let him know that Blackbaud wasn’t the source of that stat. But I made a vow to actually dig into this wayward data point and explain its true origins.
My detective work began by tracing down when the statistic began being used and what source it was attributed to. A search of the Internet and social media finds all manner of variations on the statistic:
- “95.4% of households donate to charities”
- “The majority of American households, 95.4%, donate to charities”
- “Each year, approximately 95.4% of households give to charity”
- “95.4% of households give to charity”
- “95.4% of Americans participate at some level of charitable giving”
- “According to the latest statistics from the National Philanthropic Trust, a whopping 95.4% of all U.S. households give to charity”
There’s just one problem with this statistic — it’s not true. Yes, this statistic is repeated time and time again and, like the telephone game, it no longer resembles the original message. Even the source of the statistic appears to be a mystery.
Many media reports said that the statistic came from the National Philanthropic Trust. Further searching led me to find that the National Philanthropic Trust cited a study by Bank of America and Indiana University.
I found a copy of “The 2014 Bank of America Study of High Net Worth Philanthropy conducted by the Center on Philanthropy at Indiana University” online. But even this report points to an earlier study and notes, “Almost all (98.4%) high net worth households gave to charity in 2013, an increase from the 95.4% that gave in 2011.” The mystery of the disputed statistic continued.
More statistical sleuthing was required to find the 2012 version of the same report that was based on 2011 data. The 2012 report notes that “the vast majority (95.4%) of high net worth households continued to give to charity in 2011. Although the findings show a 3 percentage point decline in the rate of giving by these households from 2009.”
The mystery has finally been solved. We now know that the 95.4% statistic is from a 2012 report based on 2011 data. But that’s not the biggest problem with this statistic. Saying that “95.4% of Americans give to charity” is not what the report actually says.
In fact, the study actually reports that 95.4% of high net worth households gave to charity in 2011 based on a survey sent to “20,000 high net worth donors in America’s wealthiest neighborhoods.” The 12-page survey was only completed by 996 of the 20,000 high net worth donors it was sent to. After excluding those donors not considered to be high net worth, the final sample size for this survey was 701 households.
Oh, and “high net worth households” were defined as “only households with incomes greater than $200,000 and/or net worth more than $1,000,000 — excluding the monetary value of their home.”
After all this data detective work, the real statistic should be: 95.4% of high net worth households in the United States claim to have donated to at least one charity in 2011 based on a survey of 701 households with an average net worth of $10.7 million.
But perhaps that’s just too long for a headline or a tweet. Instead, the telephone game turns it into something very different. There is an entire chapter in Data Driven Nonprofits devoted to the topic of bad stats. I include other examples and offer practical advice to nonprofit professionals.
As a sector, we need to up our game when it comes to the use of statistics in published content. We need to stop counting when we should be measuring. We need to use standard practices in the analysis of data. We need to be more aware as information consumers about what the data is telling us and what it is not. This requires a higher level of data literacy from nonprofit professionals, the companies that serve them, and the media that covers the sector.