JAOA • Vol 107 • No 8 • August 2007 • 310-314
Evidence-Based Medicine, Part 4. An Introduction to Critical Appraisal of Articles on Harm
Roberto Cardarelli, DO, MPH;
Margaret M. Seater, OMS IV
From the Department of Family Medicine at the University of North Texas
Health Science Center—Texas College of Osteopathic Medicine in Fort
Worth.
Address correspondence to Roberto Cardarelli, DO, MPH, Department of Family
Medicine, Texas College of Osteopathic Medicine, University of North Texas
Health Science Center, 855 Montgomery St, Patient Care Center, 2nd Fl, Fort
Worth, TX 76107-2553.E-mail:
rcardare{at}hsc.unt.edu
This article provides an introductory step-by-step process to appraise an
article on harm. The authors introduce these principles using a systematic
approach and case-based format. The process of assessing the validity of an
article on harm, determining its importance, and applying it to an individual
patient is reviewed. The concepts of study population homogeneity, equal
treatment, sufficient follow-up periods, and completeness are discussed to
help physicians determine an article's validity. Instruction on calculating
odds ratios, relative risk, absolute risk increase, and the number needed to
harm is provided and applied to the clinical scenario. Finally, information
that is learned from the previous two steps is applied to patient care. Study
generalizability and the role of patient values, patient expectations, and
patient concerns are also addressed. The skills learned from appraising an
article on harm in the manner outlined provides a solid basis for lifelong
learning and improved patient care.
Evidence-based medicine (EBM) is the practice of using the most current
research information to help guide clinical decisions while also fully taking
into account patient values and
circumstances.1-3
This method of inquiry demands skill-development in appraising the validity,
importance, and applicability of new clinical evidence to individual
patients— and it begins when a physician formulates a clinical question
during daily practice.
As noted, in this article, we introduce a strategy for busy physicians,
physician residents, and medical students to critically assess the medical
literature on harm. In-depth details of research methods are beyond the scope
of this introductory series on EBM. Readers are encouraged to seek further
training on these topics with supplemental learning opportunities and
continuing medical education. Finally, the clinical scenario described has
been simplified to provide readers with an illustrative example for the
general concepts introduced.
 |
Searching the Evidence
|
|---|
As with other types of clinical questions, the most authoritative evidence
for treatment decisions comes from systematic
reviews.1 This
standard is especially true for clinical questions that include harm because
individual randomized controlled trials are seldom sufficiently powered to
allow researchers to conduct thorough assessments for potential harm to
patients. Unfortunately, well-designed systematic reviews are uncommon and
individualized studies frequently must be used in EBM. For this reason, the
present article will focus on how to use such studies in lieu of preferred
systematic reviews.
|
Validity of Articles on Harm
|
|---|
A valid study aims to discover objective truth while also attempting to
exclude competing
explanations.1,2
Only a study that is deemed valid should be further reviewed. The following
set of questions will help a clinician elucidate the validity of a study on
harm:
- Are the defined groups of patients similar—other than through
their exposure status to the treatment under study? When assessing the
effectiveness or harm of an exposure or intervention, factors that may
influence the outcome of interest must be balanced or accounted for in all
study groups to avoid confounding results and conclusions. Treatment and
comparison groups should be clearly defined, and crossover between allocated
groups should be minimized.
Both groups should share similar baseline characteristics, especially
characteristics that may confound the outcome of interest. In a study
examining a potentially harmful treatment, healthier subjects should not be
overrepresented in the treatment group because such a group allocation may
confound the study's outcome (ie, healthier people may be able to tolerate the
treatment better). Information on baseline characteristics can usually be
ascertained quickly from a study population characteristics table. In most
research articles, subject characteristics are placed in the first table.
In addition, patients assigned to receive active treatment should not also
receive the control drug.
- Were the measures of interest ascertained in the same way for both
groups?
To properly compare study results among study groups, outcome assessments
should be objective. Such clarity can be achieved only through the use of
clearly stated case definitions.
For example, in a study investigating cardiovascular outcome, the
investigators decide that a myocardial infarction (MI) must meet specific
criteria such as EKG changes or positive troponins. The definition of an MI
cannot be left open for interpretation.
- Were participants and researchers blinded to the measures of
interest?
Assessment should also be blinded to prevent observer bias. Observer bias
occurs when the researcher unconsciously (or consciously) looks harder for
outcomes in the treatment group than in the control group. Blinding
means that the researcher is unaware of the participants' group assignment.
Double-blinding means that study participants are also masked to
group allocation.
- Was the follow-up period sufficiently long and complete? Follow-up
in a valid study should be sufficiently long and complete. Short follow-up
periods may allow too little time for the disease under investigation to
manifest. The appropriate length of the study is dependent on the study
question, the intervention used, the outcomes of interest, and special
circumstances (eg, funding).
A clinical investigation is considered complete when all study participants
are accounted for at the study's completion. Patients who dropped out of the
study early because of death or adverse effects should be included and
analyzed with their original group assignment. This is called an
intention-to-treat analysis.
A failure to perform an intention-to-treat analysis may direct researchers
to providing readers with misleading results. The "5-and-20 rule"
can be used by a critical reader to evaluate a study's completeness. If less
than 5% of the study population is lost to follow-up, one can be assured that
the loss minimally impacted the results. If, however, more than 20% of the
study population is lost to follow-up, caution is advised when making clinical
decisions based on study findings. An attrition rate of 5% to 20%—and
its impact on the researchers' results—must be determined by the reader
based on other specifics of the study.
- Do the results of the study satisfy some of the criteria for
causation?
While most studies provide some insight as to whether an exposure is
associated with an outcome, that contact may not determine causality (ie, the
exposure causes the outcome). The following five questions will help a
cautious reviewer determine if the criteria are met to presume causality:
- Does the exposure precede the outcome?
To establish causation, the exposure must take place before the
outcome.
- Is a dose-response phenomenon present?
The more one is exposed (eg, dosage), the greater the effect.
- Is there evidence in the change of the outcome when the exposure is
removed and reintroduced?
Causation is suggested when the health status or clinical outcome improves
(or deteriorates) after exposure is discontinued. In addition, the outcome
recurs if the exposure is reintroduced.
- Do other studies find consistent results?
The assumption of causation is strengthened when more than one group of
researchers reports similar findings.
- Does the association make biological sense?
It is important for the relationship between the exposure and the study
outcome to be a logical one from the perspective of basic biology.
|
Study Results
|
|---|
Having ascertained a study's validity, it is necessary to determine if the
study's findings are
important.1,2
In the medical literature, importance refers to the study's results
and is measured in terms of the magnitude and precision of the
association.
- What is the magnitude of the association between the exposure and the
outcome?
The magnitude of association quantifies the benefit or risk of an
intervention when the results seen in the treatment group are compared with
that of the control group. In retrospective studies, this data is reported as
an odds ratio (hazards ratio). An odds ratio of less than 1 indicates that the
intervention confers a protective effect. An odds ratio greater than 1
indicates that the intervention confers a detrimental effect. An odds ratio
equal to 1 indicates that there is no difference in the outcome between the
treatment and control groups. In the everyday practice of EBM, correctly
interpreting the odds ratio is more important than knowing how to calculate it
(Figure 1).
Relative risk, which is used in prospective studies (ie, clinical trials
and cohort studies), is another way to report the magnitude of an association.
Relative risk is defined as "the ratio of the risk in the
treated group (experimental event rate) to the risk in the control group
(control event
rate)."1,4
Interpretation of the relative risk is the same as the odds ratio
(Figure 1).
For patients and physicians alike, the odds ratio and relative risk are
difficult to conceptualize. These measures are more digestible when reported
as the number needed to harm (NNH) when investigating adverse outcomes
(Figure 1). The NNH is
defined as "the number of patients who need to be exposed to the
causative agent to produce one additional harmful
event."1,2
The following methods can only be used for clinical trials and cohort
studies.
The NNH can also be calculated from case-control studies. Calculations are
not shown because they are complicated and unrealistic to perform in real
practice. However, free EBM calculators are widely available online
(http://www.cebm.utoronto.ca/).
Such calculators can easily convert an odds ratio into a NNH.
- What is the precision of the relationship between the exposure and the
outcome?
Measures of the magnitude of this association (eg, odds ratio, relative
risk, and number needed to harm) are estimates of some unknown
"true" value. If one were to repeat the experiment on different
samples, they would yield similar, but not identical,
results.1 The
results of repeat experiments are dispersed above and below the
"true" value. This sampling variation is referred to as precision.
The principle measure of precision is the 95% confidence interval (CI). The
95% CI quantifies the uncertainty of a measurement by reporting a range of
values within which there is a 95% certainty that the true value lies for the
entire
population.1,2,4
With respect to an odds ratio or relative risk, results are said to be
significant if the 95% CI does not include 1.0. If the 95% CI includes 1.0,
there is a 95% chance that there is no difference in the outcome between the
comparison groups.
View larger version (32K):
[in this window]
[in a new window]
|
Figure 1. Simple calculations used to analyze the results of clinical trials and
cohort studies for risk to patients (ie, patient harm). *The absolute
risk increase is the experimental event rate minus the control event
rate. *The number needed to harm is 1 divided by the absolute risk
increase.
|
|
|
Practical Use
|
|---|
Having verified the validity and importance of a study on harm, a
physician's ultimate consideration of the research would then become how its
results and the authors' conclusions can be applied in
practice.1,2
Making an evidence-based decision requires skills that integrate the patient's
unique values and circumstances with current and valid evidence. One of the
goals of EBM is to assist physicians in developing a system for backing up
clinical decisions with sound evidence. Those who criticize EBM presumably
focus on the first goal of treating patients according to the evidence.
However, the goal of EBM is twofold. The second, and most important, goal of
EBM is to shift physicians' focus from the disease to the patient, restoring
the patient to the center of the clinical decision-making process—and
better aligning the healthcare professions with long-standing osteopathic
principles and practice.
- Are the study subjects different in any significant way from your
specific patient?
The findings of a valid, important study are considered applicable and
relevant if the patient to be treated is similar to those described in the
study. As mentioned earlier, such information can be accessed by reviewing the
study population characteristics.
- Based on the study's results, what is your patient's individualized
potential for risk and benefit from treatment?
Physicians can estimate the potential for risk from treatment (ie, adverse
effect) for an individual patient versus a study population
(Figure 2). For
example, if a patient has a medical history that includes one previous MI, the
physician may determine that this particular individual is five times more
likely than study participants to have a cardiovascular event if prescribed
the nonsteroidal anti-inflammatory drug celecoxib— especially once it is
known that a previous MI was in the study's exclusion criteria. Or the
opposite may be true when the patient's general physical health is much better
than study participants (eg, comorbidities), reducing his or her overall risk
of cardiovascular events by half
(Figure 3). This
personalized rate of risk is called the F statistic, and it allows
clinicians to personalize a study's NNH for individual patients easily, by
dividing the NNH by the F statistic: NNH/f.
If a study's NNH is 42 (Figure
4), for example, a personalized NNH for the patient could be
determined by dividing 42 by 5 for an answer of 8.4. Therefore, according to
this theoretical study (N=42), 8 people would need to take celecoxib for 2.8
years to cause one adverse event or complication.
- What are the patient's concerns and expectations from the
treatment?
When a study is declared a landmark by journalists, its results are
irrelevant if incorporating them in practice would violate a patient's
preferences, concerns, or expectations. For example, any "wonder
drug" would probably be unacceptable to a patient who is completely
opposed to the idea of taking medication.
- Are other alternatives available?
The physician and the patient, together, should explore the relative safety
of alternative treatment options
(Figure 4).
Alternative choices may include lifestyle modifications and other
nonpharmacologic treatment modalities.
|
Conclusion
|
|---|
Although most clinicians are already incorporating EBM principles in their
practices, often instinctively, some physicians may require a more organized
approach to integrating this relatively new model of self-education. Improved
comfort levels and true expertise in the practice of EBM are the result of
additional education, repetition, and self-assessment. The principles of EBM
allow physicians to stay informed while also improving the quality of the
information communicated to patients during patient encounters. The systematic
approach that is used to appraise an article on harm is but one step in
practicing EBM. Remember, the goal is always to provide the best care possible
to patients—using one's clinical expertise to address patient values and
expectations for treatment.
|
Footnotes
|
|---|
[Editor's note: This article is part 2 of a six-article series intended
to introduce the principles of evidence-based medicine (EBM) to busy
clinicians, physician residents, and medical students. Because the application
of EBM is a career-long process, further training is needed beyond the
information provided within this article and series. A foundation of knowledge
about research methods is critical in understanding EBM; however, such
details, though introduced, are beyond the scope of this series.]
|
References
|
|---|
1. Straus SE, Richardson WS, Glasziou P, Haynes RB.
Evidence-Based Medicine: How to Practice and Teach
EBM. 3rd ed. London: BMJ Books; 2005.2. Levine M, Walter S, Lee H, Haines T, Holbrook A, Moyer V;
Evidence-Based Medicine Working Group. Users' guides to the medical
literature. IV. How to use an article about harm.
JAMA. 1994;271:1615
-1619.[Medline]
3. Greenhalgh T. How to Read a Paper: The Basics of
Evidence-Based Medicine. 3rd ed. Malden, Mass: Blackwell
Publishing Limited; 2006.
4. Rosner B. Fundamentals of Biostatistics.
6th ed. Belmont, Calif: Duxbury Press; 2005.
Top
Searching the Evidence
Validity of Articles on...
