The Power of a Headline

Remember when we were told drinking red wine was as good for us as an hour in the gym? This headline surfaced after researchers at the University of Alberta published a study demonstrated the benefits of a heart-healthy antioxidant found in red wine. However, “cover stories” like this one largely misrepresents the data found from the research.

These days, our desire for a healthy lifestyle has made us gullible for any research that touts miracle health benefits. And it seems that companies, scientists, marketers, and non-profit organizations use scientific findings to sway public opinion so they can sell products or convince people to their point of view.

Furthermore, the media has fed this interest by featuring medical and scientific experts supporting their products and services on social media, TV, and radio, and print publications— making it easier for consumers to believe the information. We as educated consumers must go beyond the ad and do research of our own.

As readers and listeners from non-scientific or research backgrounds, how do we evaluate a study presented to us to determine its quality and accuracy?

Source: http://robertariail.com/

The problem, it seems, is that media outlets and publications are not always transparent about:

• How they decide what to report, and
• The methods they use to determine the scientific discoveries are fact-based and supportable.

There is a call for the media to be more accurate in their reporting of science.

Some non-profit groups are trying to improve the quality of science being reported. Organizations like the Science Media Centre help scientists engage more effectively with the media. The Centre will connect scientists with journalists so that there can be a conversation— particularly when it comes to controversial science-related issues.

Additionally, following some high profile retractions, the Center for Open Science, several major scientific journals and individual science journalists are calling for news outlets to do a better job of reporting science to the public by creating reporting guidelines for the industry.

Let’s be better-informed consumers…

When you hear about or read a second-hand review of a particular scientific study or a so-called science-based claim, be sure to read the original study or related studies.

Relying on reports written by someone other than the study author(s) increases the possibility of getting a flawed, biased interpretation of the study’s findings. Reading the primary source will get you closer to understanding the research findings.

If you’re not a scientist and have never cracked the cover of a scientific journal, this may seem daunting and that’s understandable! But if you wish to read original publications, we’d like to equip you with some tools to help you better understand what you are reading. If you decide reading scientific studies is not for you, this article provides some critical issues to look for when findings are being interpreted by others:

Another reliable method of fact-checking is to see if other scientists in the same field have critiqued the report. Scientists have opinions and, sometimes, their opinions cloud their reasoning,  just like everybody else. However, if multiple scientists point out the same flaws in a study, then there’s a good chance the criticism has merit.

For example, in our previously published post, Dear New York Times, D2D reported on the response of various scientists to a New York Times article on genetically modified crops. The scientists’ critiques were detailed regarding the choice of data and the analytical methods used.

The Power of the Scientist: Good studies require “Good Scientific Practice”

According to the National Academies of Science, the leading U.S. science body, good scientific practices include:

• Precision when defining terms, processes, context, results, and limitations;
• Openness to criticism and refutation; and
• Addressing bias and avoid overstatement.

Let’s explore each of these good scientific practices a bit further:

Scientific Precision = Addressing Uncertainty

All scientific data and processes have limitations and therefore include a measure of uncertainty to account for the unknown. For example, if you run 200 meters twice daily for two weeks you will post different times. This is why numerical data in a scientific or technical paper should never be only one value but should include a range of plausible values.

When designing or conducting a scientific study, one of the key tasks is to identify and control for errors or variations as much as possible and to estimate the magnitude of the remaining errors. Going back to our running example, to eliminate as much variability in your data as possible, you would run on the same indoor track at the same time each day, one hour after you eat a bowl of oatmeal and a banana. Your results might not otherwise be the same.

Size of the study matters

Another factor that affects the researcher’s ability to detect an effect, such as differences between treatment and controls, is the size of a study. This is referred to as the statistical “power” of a study and determines the confidence with which conclusions can be drawn. When it comes to the sample size, bigger is usually better. You can think of this in terms of the average: the average of a large number of samples is more informative than the average of a smaller sample set.

Scientists must be open to criticism and refutation…

Science is all about discovery and exploration – the pursuit of knowledge at the expense of opinions. When researchers discuss their work, they should compare their findings to what is already known and address how it fits as one piece into the larger puzzle. If their results conflict with others’ work, they should discuss what they believe is the reason for this. If their results were unanticipated or introduced unanswered questions, these should be discussed along with suggestions for further research that may provide the missing information.

What does “peer-reviewed” mean?

You may have seen the term “peer-reviewed” used to describe scientific and technical studies. What does this mean and why is it important? When a paper is “peer-reviewed” it means it was submitted to other experts in the particular field of research to judge the quality of the work.

The practice of peer-review offers a valuable way of evaluating and improving the quality of scientific studies. Peer-reviewed journals are publications that follow a process of subjecting an author’s scholarly research to the inspection of other experts in the same field before publishing a study. Journals that do not go through the peer-review process are missing an important quality control mechanism.  And by publishing a paper in a non-peer-reviewed journal, scientists run a greater risk of having to correct or retract flawed work after it was published versus making corrections prior to publication during the peer-review process.

Scientists must address/deal with bias…

Just as no measurement is free from error, human interpretation is not free from bias. However, when conducting research, scientists must design experiments to provide unbiased, useful data that, when analyzed, either do or do not support the hypothesis.

source

Of course, this is easier said than done since bias constructs are innate and difficult to recognize in ourselves. But the scientific process takes this into account and scientists must give significant effort in addressing it.

In designing a study, scientists incorporate methods (e.g., randomized assignment to groups, investigator “blinding” so they do not know which subjects are being treated, etc.) to eliminate or control bias as much as possible. Whatever bias is not eliminated or controlled by study design, must be considered and discussed when researchers interpret their results

Other sources of bias such as conflicts of interest are more overt. Some peer-reviewed journals mandate researchers declare potential conflicts of interest. Even if a conflict of interest statement does not appear in the article, a reader can do their own research to determine if the author and/or a funding source benefits in any way from reporting the results as they were reported.

How do you evaluate a study?

Now that we’ve reviewed the basic architecture of a scientific article and the National Academies’ good scientific practice, let’s consider how to critically evaluate the actual research findings and conclusions.

In addition to the National Academies’ publication, three prominent scientists published concepts for interpreting scientific claims in the acclaimed peer-reviewed journal, Nature. The authors created the list with politicians in mind to provide them with some basic understanding so they could ask their advisors informed questions. However, the authors also stated that if everyone in society understood these concepts it “would be a marked step forward,” and we here at D2D couldn’t agree more!