By Mario Kratz, PhD
Published June 10, 2022
The internet and nutrition books are full of anecdotes in which people share stories of how a change in their diet led to an amazing health transformation. And even though I guess most people have heard that anecdotes are not high-level evidence, these personal accounts seem to be convincing to many people.
And quite honestly, I understand. I do nutrition science as my full-time job, and even I have spent more hours than I dare to admit watching videos on YouTube of people sharing how they lost a lot of weight, how they reversed their type 2 diabetes, or how they got rid of their debilitating autoimmune disease. So, I get it, these types of anecdotes are inspiring, and it’s fun to see people feeling empowered to take charge of their own health and getting great results.
But, how much can we actually learn from anecdotes? And specifically, should we consider anecdotes as part of the totality of the evidence about what we should eat?
That’s what we are going to cover in this blog post.
I will cover five specific reasons why it is important to be skeptical and cautious when it comes to anecdotes, no matter how amazing and plausible they may sound. And we’ll close with a discussion of how we can think about these anecdotes in the larger context of scientific evidence.
Reason #1: Lack of Data on Generalizability
Imagine you watch a video of someone who declares that they used to have chronic migraines that went away entirely when they switched to a vegan diet. They used to have migraines two or three times a week for years, but since they’ve gone vegan, they have barely had one.
The first problem I’d like to address is that from an anecdote like this, we have no way of knowing whether this would apply to anyone else with migraines.
Imagine you were a researcher, and you wanted to find out if a vegan diet helps with migraines. What would you do? You – hopefully – would not try to find a single person who claims to have cured their migraine affliction with a vegan diet. Instead, you would recruit a large group of people with migraines, let’s say two hundred, and randomize them to, say, switch to a vegan diet, or continue on their normal diet. Then you would observe them for a few weeks or months, and see whether the type of diet they are eating makes any difference. Now, that would give you a clear picture of how many people with migraines actually benefit from the vegan diet. If only one or two of your 100 participants on the vegan diet got better, well, they could still start a YouTube channel and tell the world about it, but you would know better that most people’s migraines don’t get better by going on a vegan diet.
So, the first big problem with anecdotes is that an anecdote doesn’t provide any data on how generalizable the phenomenon is to other people with the condition. It’s fine to be open to the possibility that the anecdote may hold some truth for others or maybe even all people with that condition, but it’s important to be clear that it’s as well possible that it isn’t generalizable.
Reason #2: Positive Selection Bias
In our previous example of a randomized controlled trial, let’s assume that one person reported not having any more migraines on the vegan diet, while the other 99 that were randomized to that diet reported no or very little benefit. Well, who is most likely to start sharing passionate stories about the benefits of a vegan diet on social media?
Quite obviously, those folks who did not benefit from the dietary change would have little to talk about, right? One general problem with social media is that people click on positive stories that promise help or that are unusual or entertaining in some way. If dozens of people tweeted or published YouTube videos that they still have migraines even though they went on a vegan diet, they would barely be noticed. The one person who cured their migraine would get all of the clicks, and as a result, that’s the kind of content that tends to get published in the first place.
Reason #3: Placebo Effect
Someone may absolutely have gotten fewer migraines on a vegan diet, but that effect may have little to do with the specific type of diet the person ate and more with their expectation to get better. That effect is called the placebo effect.
A meta-analysis, basically a study of studies, on the placebo effect in clinical trials testing migraine medications reported that among patients with migraines who were randomized to a placebo pill, 27% reported an improvement. In these studies, researchers were testing a drug that they thought would improve migraines, and as a comparison group, they gave controls a pill with no active ingredient in it. And in 27% of these patients who got the sugar pill, it helped, even though it shouldn’t have.
Amazingly, if the placebo was not given as a pill, but injected with a needle, then it helped even more people. The commonly accepted explanation for these observations is that people get better simply because they think they may get better. And apparently, people think of an injection as a more effective thing than a pill, so placebo injections work even better than placebo pills. Even though both should be equally ineffective.
The same is true for weight loss, by the way. In randomized controlled trials, it is common for people in control groups who receive a placebo pill to lose some weight because they imagine that they may have gotten the active weight loss drug. And again, participants commonly lose a lot more weight if the placebo is injected rather than given as a pill.
So if we want to understand how a drug, a food, or a diet affects a health condition, the placebo effect is one of the reasons why we want to compare changes in a treatment group to those in a control group. If a single person experiments with a dietary change, it is next to impossible to be certain that any change is indeed related to that dietary change, and not to a large degree, or entirely, because of the placebo effect.
Reason #4: Inability to Identify Key Dietary Factor
In people eating complex diets, it is surprisingly difficult to study just one factor. If someone changes their diet, they usually change many things in terms of what they eat, but also what they no longer eat, or what they eat less of.
In our imaginary example, let’s assume that someone may try the vegan diet because maybe they read somewhere that animal products can cause migraines. And let’s say they actually get rid of their migraines. It still may be wrong to conclude that the improvement was caused by cutting out animal foods. And that cutting out animal foods will help all people with migraines. Even if we assume that the reduction in migraines was actually caused by the change in diet, we need to consider that the improvement may have been the result of cutting out one particular animal food, and maybe one that this particular person had an allergy to. Or maybe the improvement had nothing to do with eating less animal products and more with eating more fruit and vegetables, or more sources of plant protein such as legumes or nuts.
#5: Benefits May be Temporary, Risks May Manifest Later
When someone shares an anecdote of health improvements, they do that almost certainly after they have gotten better, after they lost all the weight, after their migraines are gone. And that creates the problem that the condition may only temporarily be in remission, the weight may be re-gained, and migraine headaches may return. And similarly, while some benefits may have manifested early, certain risks or adverse effects may take a while to develop.
To illustrate this point, let’s use another example. Let’s assume you watch one of the very popular carnivore channels on YouTube, and someone shares that they used to have a nasty autoimmune disease that has entirely disappeared since they’ve gone on the carnivore diet.
This would be a typical example of a benefit that may only be temporary. Many autoimmune diseases show a pattern of remission and relapse, even if you don’t do anything special. To be really certain that an autoimmune disease is truly in remission or – as some people claim – even cured, you would need to be relapse-free for a very extended period of time. Just because someone currently doesn’t have any symptoms doesn’t mean their disease is cured. If someone can demonstrate convincingly that they have been relapse-free for many years, that would certainly strengthen their argument that their disease is indeed permanently in remission.
The other aspect is that of health risks or adverse effects. The carnivore diet is a good example in this regard as well. On the carnivore diet, most people eat only meat, fish, and shellfish, and some people also eat eggs and/or certain dairy foods such as cheese. That means that total fat and saturated fat intake are very high, and the intake of certain micronutrients such as calcium may be very low, depending on whether cheese is eaten. Without going into too much detail, suffice it to say that most experts would probably be concerned about an increased long-term risk of cardiovascular disease and osteoporosis on a carnivore diet. As such, even if a carnivore claims that they have been doing very well on this diet for many years, that is not to say they may not have substantial health problems down the road. Cardiovascular disease and osteoporosis take a while to maifest. Or even if they don’t, you may if you follow their advice.
Something else I’d like to get off my chest that is related to this point. The one thing I am the most annoyed with is when people share an anecdote of how they got rid of their migraines, how they lost weight, or how they suddenly have more energy, or whatever it is, they don’t just report this, which I absolutely agree would be an interesting observation. Instead, they often start to sell this food or this diet as the best human diet, the cure for everything, and deny that anything could ever be wrong with it.
Now, let’s be real. For an extreme diet such as a vegan diet or a carnivore diet, in particular, it is very well possible that you can feel better, and some health conditions can get better, while at the same time, that same way of eating increases your long-term risk of other diseases. Try to be open to that, in the interest of your health. In nutrition, as with most things in life, we make trade-offs, and every food has health-promoting and potentially harmful characteristics. If someone makes it seem as if their way of eating is the only correct way, and there are no risks associated with it whatsoever, that’s a major red flag to me.
Conclusions
Let me emphasize that it’s not my intent to hate on any one particular diet. As may have become clear to you if you read any of the other posts on this site, I hold the position that humans are pretty flexible and that we can be healthy on a wide range of very different diets.
I am hating though on exaggerated, superficial, and non-scientific claims made in support of eating a certain way, because real harm can come from that to people with chronic illnesses who are desperately trying to improve their health.
My goal with this post was to help you develop a healthy dose of skepticism whenever you come across nutrition anecdotes, particularly if someone uses the anecdote to sell you something. I absolutely understand the appeal: anecdotes tend to offer fairly simple, convincing solutions to sometimes desperate chronic health problems. But just because it sounds good doesn’t mean it is sound advice. Basing your health on anecdotes from YouTubers that provide no other sources of evidence, or even ignore scientific evidence, is similar to buying a car because the used car salesman told you it’s a great deal. You would be a little bit skeptical there, right, so why not apply the same kind of skepticism when you are making decisions about your health?
Still, as skeptical as I am, let me reiterate that I enjoy hearing about case studies of people who turned their health around by making a change to their diet and lifestyle. I even think that these types of anecdotes do provide some very-low level scientific value. But that needs to be clearly emphasized: anecdotes are very low-level evidence, at best,
If we look at the hierarchy of evidence, it is obvious that a well-conducted randomized controlled trial provides much stronger evidence.
Randomized controlled trials provide us with information about the generalizability of our observation, and at least an approximation of how what percentage of people with a certain condition can be helped by an intervention. Also, we are not just looking at people in whom the intervention worked, but everyone who was exposed to it. In a randomized controlled trial, we also have a control group, which helps us take any placebo effect into account. And ideally, we are also able to isolate the exact dietary factor we are interested in, and conduct the study for a time period that allows us to collect data also on long-term risks and adverse events. Admittedly, the last two points are sometimes not even possible in dietary intervention trials.
The different types of observational studies, as much as people constantly criticize them, also provide substantially more robust evidence than anecdotes.
So, ideally, we should form an opinion about the health effects of a food or a diet on a specific outcome by looking at systematic reviews or meta-analyses that consider the totality of the evidence from several randomized controlled trials and several observational studies.
Unfortunately, for many questions we have about nutrition and health, this is not an option. We simply have very little robust data, for example, on whether there is a dietary approach to improve certain conditions such as neurodegenerative diseases, or many autoimmune diseases. And that’s where I think we should at least consider lower-level evidence. The lowest level of evidence in the graph above, that’s where I would categorize anecdotes, but even that only if they have been clinically documented as a case study, or even better, a case series of several related cases. Here is also where we would categorize things like expert opinions or editorials. Then we have animal studies that I would actually rank similarly as very low-level evidence. To clarify, animal studies can be amazing to figure out the basics of how biology works, but – in my opinion – are not very useful to figure out what we as humans should eat, simply because it is too common that responses to specific dietary interventions are substantially different in humans when compared even to non-human primates. So I suppose most researchers would agree that these types of evidence at the bottom of the hierarchy of evidence graph are not strong pieces of evidence in and of themselves. At best, they can be useful to help inform hypotheses that can then be tested using these more rigorous methods.
So don’t totally discount anecdotal evidence. Be curious, let yourself be inspired, and let us use anecdotes to form hypotheses, but also, let’s remain appropriately skeptical, and let’s require much higher-quality evidence before we conduct an experiment on our own bodies.
Take care.
References
- De Craen et al.; Placebo effect in the acute treatment of migraine: subcutaneous placebos are better than oral placebos. J. Neurol. 2000; 247: 183-8