Statistics

Part 6 of the 2025 Critical Thinking Week Series

There may be no other human construct quite like statistics. In a stroke of a pen or click-clack of a keyboard, something can be created that looks and acts like a statistic. However, for most of us, our understanding of the study of statistics is only a tiny fraction of that of a statistician. This is true regardless of the depth of our knowledge, and regardless of whether the statistic is a complete fabrication or a highly accurate, reliable, and valid report. 

Validity: Does the experiment measure what it is supposed to measure?

Reliability: Are the results consistent and replicable?

Accuracy: Are the measurements true?

Even when data is highly reliable, accurate, and valid, statistics can be used to mislead the user. 

Correlation does not imply causation: Most people have heard the term “correlation does not imply causation” and yet continue to confound the two terms. For example, 100 percent of people who drink milk die. More seriously, however, is the erroneous belief that vaccines cause autism. Because autism is diagnosed at approximately the same time that childhood vaccines are given, people make the connection and assume causation. However, since until recently, almost every child was vaccinated. It stands to reason that almost every child who gets autism was vaccinated, but there is no increase in the rate of autism between the vaccinated and unvaccinated children.

For other wildly amusing examples of non-causal correlations, see Spurious Correlations.

Meaning of statistics: Before declaring statistics to be useful, it is important to also consider what they actually mean. Think of the following claims:

• “We’ll help you save 20 percent more than the competition.”
• “Drug X will reduce your risk of heart attack and stroke by 50 percent more than Drug Y.”
• “Drug A will increase your risk of an adverse event by 5 percent compared to Drug B.”

There is missing critical information in interpreting these claims. 

• Baseline information: What is the baseline for the decreases and increases? If you have a 1 in 1000 chance of suffering a heart attack (0.01 percent chance) and a drug will reduce the risk by 50 percent, you will now have a 0.005 percent chance. The 50 percent reduction is real, but is not significant to most people. In contrast, if you have a 1-in-2 chance of suffering a heart attack, this claim means that your risk is reduced from 50 percent to 25 percent.

In other words:

• Drug X may be reducing the risk of heart attack and stroke from 75 percent this year to 25 percent, if you’re looking at an absolute risk reduction of 50 percent. But more likely, someone’s risk of heart attack and stroke is being reduced from two percent to one percent, which is also a 50 percent decrease, but only relatively better. 
• Drug A may increase the risk of an adverse event from 10 percent to 15 percent, a 5 percent absolute increase. But more likely, someone’s risk is increasing relatively from 10 percent to 10.5 percent, a 5 percent relative increase.

This is how marketing works. It takes advantage of our reliance upon, and widespread inability to properly interpret, statistics. But it isn’t just those driven by the Friedman Doctrine who misuse statistics. We must hold ourselves and those we trust to the standard of properly representing statistics. Demand of everyone, everywhere, and every time that the baseline rates and probabilities, and whether relative or absolute differences, are being reported.

For more information: See CFIC’s 2010 presentation, “The Curious World of Probabilities with Professor Jeffrey Rosenthal.”

Image by Jakub Zerdzicki via Pexels, used under the Pexels License

Discussion:

What statistics have you relied on and later learned were faulty? What are other ways that statistics can be misleading?