By Mario Kratz, PhD
Published: March 15, 2022
I like to read – or listen to – diet and nutrition books, and I guess that’s probably the case for many of you as well. If so, let me ask you this: how do you know that the information provided in a nutrition book is based on scientific evidence? Do you go by reviews on Amazon? Or by the academic credentials of the author? Or do you just trust that the book must be based on evidence if the author cites scientific publications?
Sadly, none of these methods are suitable to find out if a nutrition book is any good.
Doing your own research is probably also not a feasible option, so I suggest you check out Red Pen Reviews, a website that provides comprehensive, unbiased reviews of nutrition books. Red Pen Reviews is a non-profit organization that was founded a few years ago by my friend and colleague Dr. Stephan Guyenet. Its objective is to elevate the quality of health information in nutrition books, and specifically to make public to which degree a given nutrition book is based on scientific evidence. In the long run, we hope to also put some pressure on the publishing industry to develop stricter standards for nutrition books. Currently, there are seemingly no such standards, and as a result, most diet and nutrition books on the market do not provide high-quality information.
The Problem with Most Diet and Nutrition Books
What we are hoping for when we read a diet or nutrition book is that the author has reviewed the entire scientific literature on a given topic, and then provides us with a condensed but balanced version representing this totality of the evidence in the book. Sadly, this is rarely the case.
Quite often, references cited in support of specific claims have little or nothing to do with the claim made. Or the strength of the evidence is substantially overstated. For example, it is not uncommon that an author links a specific food or food component to a disease, with a strong statement such as “food x causes disease y”. You know, as in “red meat causes colon cancer”, “vitamin D deficiency causes multiple sclerosis”, or “honey cures COVID”, but when you look at the cited reference, it’s an observational study showing an association, or even just an animal or cell culture study.
This is a highly problematic way to use citations, because to you – the reader – the citation of a scientific publication gives the impression that a statement is based on actual and conclusive scientific evidence. If there are important limitations that cast doubt on whether a finding is reflective of a cause-and-effect relationship in humans, then this should be communicated as such.
Another misuse of science that I often see is that authors just talk about a certain aspect that fits their narrative, but not share other pertinent pieces of information around a given topic. An example would be vegan diet books talking endlessly about the amazing nutrient density of vegetables or the fiber and protein content of legumes without ever mentioning that animal foods are also rich sources of micronutrients and protein, or that some plant foods have components with potential negative health effects, often referred to as antinutrients. Or authors of keto diet books emphasizing heavily the negative health consequences of carbohydrates while downplaying potential negative health effects of fat, or at least certain fats.
That’s not science. It’s similar to that metaphor of blind men, or here in this picture (below), blindfolded people, describing an elephant. Depending on what part of the elephant they are exposed to, they describe it as a rope, a wall, a snake, or a tube.
The problem is that most complex things cannot be described adequately by just considering one portion or one aspect of it. Even if you define that portion accurately, overall you may still be wrong if you are not considering the entirety of what is known. This very thing happens a lot in communication about nutrition, in diet and nutrition books, but also here on the internet.
What you will also find a lot in diet and nutrition books is that authors try to give the appearance of being based on scientific evidence by citing the literature, but only in areas where it suits their argument. In other areas where their claims are not consistent with the scientific consensus, they ignore that inconvenient evidence or call the evidence false by pointing out all of the limitations of the science. Don’t get me wrong, critically evaluating the strengths and limitations of scientific evidence is important and in line with rigorous scientific practice. However, we need to use the same critical mindset when evaluating all of the evidence: the studies that support our hypotheses AND those that don’t. Being critical just with the evidence an author doesn’t like shows clearly that this is someone who is just trying to win an argument, rather than being seriously interested in an advancement of knowledge.
Introducing Red Pen Reviews
So a few years ago, I had lunch with Stephan. We talked about these exact issues that plague diet and nutrition books. He shared that he had been impressed by reference checks of nutrition books conducted by another colleague, Seth Yoder. Seth had simply taken a few nutrition books and checked the accuracy of all references. That’s a boatload of work, and I give Seth a lot of credit for engaging in this activity because it revealed how poor the adherence to scientific standards is in many popular nutrition books. We decided to contact Seth and ask him to work with us to expand on his idea, and Red Pen Reviews was born.
We decided that it would be critical to make sure that book reviews are conducted in a standardized, non-biased, and objective way. We also felt that it would be important to make sure that reviewers are sufficiently qualified to critically read and interpret the scientific literature. We developed a standardized review process and developed clear rules to make sure that reviewers who have a potential bias cannot review books in that area. We also decided that reviewers need to have at least a Master’s or PhD degree in nutrition or related field, and that they would need to demonstrate their ability to critically analyze the scientific literature before joining the team.
As a general rule, we assign a team of two reviewers to each book, not just one. One functions as the primary reviewer, while the other is the so-called peer reviewer. Peer reviewers also read the book as well as the completed review to check for errors or potential issues, and make sure that the primary reviewer followed the standardized review process.
The Standardized Red Pen Reviews Process
The standardized review process on Red Pen Reviews scores books across three main categories:
The first category is scientific accuracy, which assesses whether current scientific evidence supports a book’s key claims.
The second category is reference accuracy. Here, we assess how accurately the references in a book support specific statements made.
And the third category is healthfulness, which assesses whether following a book’s dietary advice is likely to improve or harm the reader’s health.
At the top of the review page, you can always see the summary scores for the book overall as well as for each of these three categories. Here is an example from the book The Plant Paradox I completed a few years ago to show you what this typically looks like:
For scientific accuracy, the reviewer identifies three key claims that are central to the book’s thesis, and evaluates them scientifically. This evaluation is done separately for each of the three claims by assessing how well a claim is supported by current evidence, whether references cited in the book in support of the claim are convincing, and how well the strength of the claim in the book lines up with the strength of the evidence.
After a thorough review of the scientific literature, not just those references cited in the book, the reviewer scores each claim for each of these three criteria. Scores range from zero to four, with higher scores indicating better scientific evidence in support of a claim.
For the second category, reference accuracy, we randomly pick ten references from the book and score each independently based on how well the reference supports a specific claim made in the book.
For healthfulness, we assess whether the nutrition information provided in the book is likely to improve the condition the book targets, whether it is likely to improve general health, and whether it promotes adequate nutrient intakes.
The average scores in each category are averaged and converted to a percentage for each, and then these percentages are listed at the top of the review page. The overall score is calculated as the average of these category scores.
What we’ve tried to do is make the review as comprehensive, unbiased, and objective as possible, but we still wanted to provide easy-to-understand summary scores and summary information for those readers who are not interested in all of the technical details of how a book was scored. You can basically get a really good idea to which degree a book is based on scientific evidence by looking at these summary scores and maybe spending 5 minutes reading through the summary at the top of the review. For those of you who want to follow along in our review process, you can easily spend half an hour or so with each review by reading through the detailed explanations of how we arrived at each score. Still, that may be time well spent, considering that you would otherwise spend a lot more time reading a book that may be of questionable value.
Published Book Reviews
Red Pen Reviews is still a very young organization, and so far we have managed to publish reviews of the following nutrition books:
Currently, we have reviews underway for nine more books, most of which we hope to publish this year.
Red Pen Reviews: A Voice in the Fight Against Misinformation
I left academic research and started Nourished by Science because I see the huge potential for people to prevent or even reverse chronic disease by using scientific evidence about the impact of different foods on our health. I also see the potential harm that can be done by the huge avalanche of misinformation or half-truths that we are all exposed to in this ever more complex world.
I am hoping I can do my part to help people sort fact from fiction partly with this new website here as well as my YouTube channel, and partly through my work with Red Pen Reviews. I strongly believe that providing sources of information that are based on verifiable facts is critical in all areas of our lives, and the field of nutrition and health is no exception.
I hope you will have a look at Red Pen Reviews in the next few days. Ideally, sign up for our email list on the Red Pen Reviews home page, so that you will be notified whenever a new book review is published. Or follow Red Pen Reviews on Twitter.
If you are a colleague with a graduate degree in nutrition or a closely related field, and you read a lot of nutrition books, consider joining our team of reviewers! Our goal is to publish book reviews for all of the major bestselling nutrition books. To be able to do this, we need to expand our team of reviewers, and we are currently actively seeking new team members.
And lastly, if you value our efforts to bring you detailed reviews of nutrition books, consider making a one-time or recurring donation. Red Pen Reviews is a 501(c)(3) non-profit organization, which means we don’t pay any state or federal taxes, and any contributions made to us are tax-deductible for taxpayers in the United States. Now, to make sure you are not getting the wrong impression: I am not asking this for myself. I am not being paid by Red Pen Reviews, and all of my time and work so far has been volunteered. Actually, all work so far has been done by volunteers: all book reviews, the incorporation of the non-profit organization in Washington State, the application to the United States Internal Revenue Service (IRS) for 501(c)(3) status, and the design and maintenance of the website. We are planning to start paying reviewers for completing a book review, but initially at least, that’s going to be a small amount and really more a token of appreciation than adequate compensation. We definitely have minimal overhead, and any donations would almost entirely go directly towards more book reviews.
I think it’s fair to say that for all of us currently on the Red Pen Reviews board and team, this is a labor of love and not something we engage in for personal financial gain. In this regard, let me finish this blog post by expressing my appreciation to the members of the Red Pen Reviews Team for their many contributions: our founder and director, Dr. Stephan Guyenet; Seth Yoder, who is also a member of the board as our treasurer; Dr. Hilary Bethancourt; Dr. Travis Masterson; Morgan Pfiffner; Shaun Ward; and Dan Dell’uomo; and for past contributions: Dr. Kevin C. Klatt; Dr. Maggie S. Burhans; and Katherine Pett.
If you are interested in this type of content, feel encouraged to sign up for our newsletter below. Also, please check out our social media pages on YouTube, Twitter, and Facebook.
Take care!