2 Evaluating Web Sites, Articles, and Product Claims

An understanding of research studies and methods will help you evaluate health information. This section builds on this knowledge, providing additional ways to consider information you find on Web sites, in newspapers or magazines, or in product advertisements.

Evaluating health information on the Web or in print

Even if you're confident that a Web site or print publication is of reasonable quality, you should still read the information closely and critically. Often, we read on a Web site or in the consumer press that researchers have "concluded thatā€?" But remember that "conclusions" are not the same as "results." The results are what happened during the study-the outcomes of the research experiment. The conclusions are what the results meant to the researcher, ie, the researchers' interpretation of the results.

To further complicate matters, Web site developers and print reporters translate some version of the research results and conclusions into simpler language to make the information accessible to as many people as possible. Sometimes, by the time the information actually makes it to the news or to a Web site, it may be very different from the original research article.

Here are some things to consider when reading health information on the Web or in print:

  • Is this the first study to get the result in question? Sound health advice is generally based on a body of research, not a single study. Be cautious of results claiming a "quick fix" that depart from previous research and scientific beliefs. Keep in mind that science does not generally advance by dramatic breakthroughs, but through a series of small steps, slowly building toward a consensusClick here for a definition of this term. You can use the skills in the FIND section to see if there is other research published on the same topic.
  • Check who sponsored the research. Just as you check the sponsor of a Web site, read an article closely to find out who sponsored the research. Was it the government, a university, a non-profit organization, or a for-profit company? In all research, there is a danger that conclusions may be worded in such a way as to benefit the sponsoring group. For example, for-profit companies may have a vested financial interest in publishing overly hopeful information about a new product or intervention.
  • Check who did the research. Do you recognize the university or research institution under which the study was conducted? While good research comes out of lesser-known places, generally you can be more confident in research conducted at well-known schools or research institutions, particularly those in the United States and other countries where medical research is closely regulated.
  • How many people were in the study? In general, the bigger the study group(s) the better. However, as noted previously, a small study of a good design (randomizedClick here for a definition of this term, controlledClick here for a definition of this term, double-blindClick here for a definition of this term) can also yield valuable results. However, it is again important to evaluate the overall result. Researchers will sometimes use an approach called meta-analysis to combine the results from several small studies into a combined (overall) result based on larger numbers. However, it is important that the individual studies are reasonably similar in design, quality, and outcome before combining them. In other words, the more differences there are among the studies being combined, the less meaningful the results will be.
  • There is a big difference between "associated with" and "caused by." Many print and electronic stories (and especially their headlines) play fast and loose with the notion of "causality"Click here for a definition of this term. A story may suggest that eating broccoli "prevents" cancer. However, prevention (or causation) can be "proven" only from the results of high-quality, controlled experiments. There may be weaknesses in a study that limit the conclusions that can be drawn. For example, the above study may have found that among a select population of nurses, those who recalled eating broccoli were less likely to develop cancer than those who did not. Protection against cancer may actually be related to factors other than eating broccoli, such as the fact that nurses may be more health conscious (eg, exercise more, smoke less, eat better, etc) than other people. It may also be that those who remembered eating broccoli are more health conscious that those who did not. To address these problems, some scientists may try to experimentally identify substances in broccoli that reduce the risk of cancer. Others may do a clinical experiment that "assigns" different diets to two groups of people with similar characteristics. One group is told to eat large amounts of broccoli, and the other group is told to eat none.
  • Is the study prospective or retrospective? Building off the last point, retrospective studiesClick here for a definition of this term that "look back" at subjects and their behavior may be less reliable than carefully designed prospective observations or a double-blindClick here for a definition of this term, controlledClick here for a definition of this term, intervention studyClick here for a definition of this term. Retrospective studies often rely on the participants' memory of their behavior, which may or may not be accurate or, in some cases, honest (eg, in studies asking people to recall certain dietary habits or alcohol consumption).

The Yale-New Haven "Health Information Library" (http://www.ynhh.org/online/health_lib_frset.html), a Web site associated with Yale New Haven Hospital, suggests that one should beware of the following:

  • Overly simple conclusions drawn from a complicated study.
  • Recommendations based on a single study.
  • Dramatic statements that are refuted by reputable scientific organizations.
  • Recommendations made to help sell a product.
  • Recommendations based on studies published without peer reviewClick here for a definition of this term.
  • Recommendations from studies that ignore differences among individuals or groups.

back to top

Health Compass Tip:

Do you want more information on understanding new research findings that may have an impact on your health care decisions and lifestyle choices? The American Federation for Aging Research (one of the sponsors of Health Compass) has a Web site for the public, http://www.infoaging.org/, that offers reports and summaries of current research on aging and health for the lay reader. This is one reliable source of research information that can help you interpret what you may read in the media.

You can also find critiques of health care-related news stories on Health News Review, a web site sponsored by the Foundation for Informed Medical Decision-Making. Health News Review is dedicated to improving the accuracy of news stories about medical treatments, tests, and procedures and helping consumers evaluate the evidence for and against new ideas in health care. A multi-disciplinary team of reviewers from journalism, medicine, health services research, and public health assesses the quality of the stories using a standardized rating system. Stories are graded and critiques are published on the web site.


Evaluating news stories and headlines

Headline editors for stories about medical research have a tough job. They need to distill complex information into a few words that will lead readers into the body of the story. As a result, headlines cannot include the subtleties of a study, and instead focus on the potential impact of the research. A headline may read: "Scientists uncover Alzheimer's gene." But when you read the story, you may find that the research was performed on mice or that the gene is linked to only 5% of cases in people. This may be important research, but it is not as the headline suggests, the gene that causes Alzheimer's disease in people. You need to dig deeper than the headline and get the whole story. Science writers may also "jump the gun" in reporting on new research findings that may be important to other researchers, but that are not yet meaningful to the public. Results on which treatment decisions or lifestyle changes are based, generally need to be retested in large study groups.

Too often, premature, sensationalized headlines are the by-products of reporters who read one article about a research study and arrive at their own "conclusions." Or, usually some type of percentages or statistics associated with a study are reported. In addition to these numbers, you should read or listen carefully for the actual numbers to consider the meaning of the results. Why is it important to know this? For example, a news story may say that results of a large study on people with high blood pressure showed that a certain drug decreased blood pressure in twice as many people who took the drug versus in those who took a placebo. That sounds good, but you need to know the actual numbers to understand what those results really mean. Let's say the study included 1,000 people, and the number of people whose blood pressure decreased with the drug was 6, and the number of people whose blood pressure decreased on placebo was 3. Although it's true that blood pressure decreased in twice as many people who took the drug versus in those who took the placebo (6 vs. 3), the results are not convincing because they involve such a small number of people compared to the total size of the study group.

The way that scientific journals promote research to the news media can also cause problems. A June 5, 2002 article in the Journal of the American Medical Association (JAMA) reported that journals' press releases often exaggerate the importance of study results. Moreover, journals, in their eagerness to attract publicity, can make three fundamental mistakes that have far-reaching implications:

  1. The data are often presented using formats that may exaggerate the importance of the findings.
  2. Press releases do not routinely highlight study limitations.
  3. Press releases do not routinely highlight the role of industry funding in a research study.

These concerns further emphasize the need to maintain a healthy skepticism when reading about medical research.

Try the following exercise to "practice" your evaluation skills.

An exercise in evaluating the headlines

Try comparing the information in an abstract of a research paper to a press release about the study:

  • Read the abstract of the article entitled "Nut Consumption and Decreased Risk of Sudden Cardiac Death in the Physicians' Health Study" (in the June 24, 2002 issue of the Archives of Internal Medicine, published by the American Medical Association). Click on this link to get the abstract: http://archinte.ama-assn.org/cgi/content/abstract/162/12/1382
  • The headline on this study on the MedEm Web site (which is a Web site maintained by professional medical societies) reads as follows: "News from the AMA: Frequent Nut Consumption Associated With Decreased Risk of Sudden Cardiac Death"
  • Now read the entire press release click here.
  • Note one of the conclusions in the AMA press release: "The authors conclude that - 'If the observed associations between dietary habits such as nut and fish consumption are causal, then these dietary interventions could be applied with little risk.' "
  • Now, think about the following questions:
    • What do you think about the MedEm Web site news release vs. the actual information in the abstract of the research study?
    • Is the headline misleading? Does the headline lead you to believe the study has wider implications for lifestyle choices than the abstract shows it actually has? Do you know from the headline that the study included only men and not women?

These are the types of questions you should ask yourself when reading about research findings in the media. We must all learn to read health information with a critical eye, taking into consideration the quality of the research and the way in which it is reported.

back to top

Evaluating a Web site's reliability - ask yourself...

  • Does the Web site's home page include the Health On the Net "seal?"
    The Health On the Net Foundation has created a "Code of Conduct" to help standardize the reliability of medical and health information available on the Internet.

    The HONcode is not an award system. It does not intend to rate the quality of the information provided by a Web site. It does define a set of rules to hold Web site developers to basic ethical standards in the presentation of information, and to help make sure readers always know the source and the purpose of the information they are reading.

    Web site developers submit an application to the Health on the Net Foundation to be approved for the HONcode. The site is reviewed by the Health on the Net Foundation. If it meets a set of criteria it is approved by the Foundation, and the HONcode symbol can be displayed on the siteā€™s home page. Look for this symbol:
  • When was the information last updated? The more recent the better. If a Web site hasn't been updated recently, results of the newest research findings won't be included.
  • Is there a clear indication of the name and contact information of the Web site developer or publisher? This can give you insight into possible biasesClick here for a definition of this term of the research or information presented. Also, it provides a way for you to ask additional questions or to follow-up on information that is unclear.
  • Is it clear which organization(s) contribute funding, advertising, services, or other support to the Web site? Again, bias is possible. There may be a connection between the sponsors and the content of the site that either dramatically or subtly alters the way in which the information or research is presented.
  • Who is evaluating the quality of the information posted? Advisory boards and reviewers guarantee that the information on the site has been reviewed by a medical or research professional to be sure that it is accurate and responsibly presented.
  • Is this a site for consumers, health professionals, or some other audience? This will give you a clue as to what kind of information will be presented and how technical the language will be. You may want to read the exact information your doctor is reading or that same information rewritten or "translated" for a layperson.

back to top

Warning signs when reading about medical products

In addition to articles about research, there is also a great deal of information on the Web and in print on medical products. Again, you should read this information critically. Beware of the following warning signs that indicate misleading or untruthful information on medical products:

  1. Information that promises a quick fix, or that sounds too good to be true. It probably is.
  2. Advertisements or information that use phrases such as "scientific breakthrough," "miraculous cure," "exclusive product," "secret formula," "ancient ingredient," "without risk," "anti-aging," "improve sexual performance," and "all natural."
  3. Case histories from "cured" customers claiming amazing results. When you see a testimonial, ask for proof of how common or typical the results are.
  4. Claims that one product can cure or treat a whole list of diseases, such as HIV/AIDS, cancer, arthritis, Alzheimer's disease, wrinkles, weight problems, memory loss, and others.
  5. Advertisements that promote the latest "trendy" ingredient in the news headlines.
  6. Claims that the product is available from only one source, for a limited time.
  7. Testimonials from "famous" medical experts or celebrities. These people may be paid to endorse a product, which should raise a "red flag" in your mind.
  8. Claims of "no risk" or lack of any information about risk. Remember: no product or treatment is completely risk-free.
  9. Claims that a product is "scientifically proven" and "absolutely safe."
  10. Products with the same name in different countries. Products with the same name may contain different ingredients in different countries. To avoid this problem when searching for information, you should look at the International Non-proprietary Name (INN) of the active ingredients and not just the product name (brand name, trade name).

Health Compass Tip:

EurekAlert! (http://www.eurekalert.com/) is an online, global news service operated by the American Association for the Advancement of Science. EurekAlert! provides a central place through which universities, medical centers, journals, government agencies, corporations and other organizations engaged in research can bring their news to the media and to the public. EurekAlert! features news and resources focused on all areas of science, medicine, and technology.

back to top