There is a popular saying in the skeptical community: the plural of anecdote is not data. The clever putdown is meant to remind the person you are speaking to that no matter how many people swear by, let’s say, homeopathy, their testimonials do not prove homeopathy’s worth.
I happen to disagree with the saying. The plural of anecdote is data. If a physician begins prescribing a new drug and three patients come back complaining of a weird and sudden skin rash, this trio of anecdotes is worth exploring. It’s just that these anecdotes on their own are not conclusive and this is a crucial point to make.
Anecdotes and testimonials are incredibly convincing to us. We love a good story, and hearing the compelling narrative of someone who was ill and recovered makes us want to know what led to the improvement. What was this thing that made such a dramatic difference? The problem is that figuring out what this thing is is more difficult than meets the eye.
We may believe that homeopathy helped us overcome the flu, but the flu is a self-limiting illness. Our immune system kicks in and our symptoms do not last forever. Therefore, whatever extra step we take may look like it’s curing us of the flu, but the real slayer here is our immune system. (As for claims that products can boost our immune system, they tend not to survive scrutiny.)
We may hear that energy healing helped someone get cured of their cancer, but was it the energy healing that did it or the combination of chemotherapy, radiotherapy and surgery? Did energy healing contribute anything to this assortment of approaches? Based on this testimonial alone, it’s impossible to know.
We may think an expensive series of acupuncture sessions is really helping with our chronic pain, but if we don’t record our pain levels daily, our pain may remain just as bad but we end up noticing the days when it’s better and forgetting the days when it’s worse. Investing in something predisposes us to believe it’s going to work, because to spend money on something useless would be foolish.
That is not to say that testimonials about health products and interventions are always wrong; it’s that they are not reliable. They can be contaminated by phenomena that are well known to skeptics: regression to the mean, self-limiting illnesses, multiple treatments used at once, confirmation bias, and the placebo responses. We need to conduct robust scientific studies to eliminate all of these phenomena and figure out if something truly works on its own.
So the next time you are tempted to believe something works based on an anecdote, remind yourself that anecdotes are dirty data: they are adulterated with many factors that may have played a role in the outcome we observe. Ask for better evidence if you don’t want to risk wasting time, money, and even jeopardizing your own health.
