If you prefer to view this article in PDF form, click here.

Is meta-analysis the silver bullet of research?

Kelley E. Swatzell, MPH; Patricia R. Jennings, DrPH, PA-C; Stephen P. Glasser, MD

Kelley Swatzell is an assistant professor and Patricia Jennings is a professor in the University of Alabama at Birmingham’s Surgical Physician Assistant Program. Stephen Glasser is a professor of medicine at the University of Alabama at Birmingham’s Division of Preventive Medicine.

Meta-analysis is a methodologic literature review technique that statistically combines the results of several studies (or the individual patient data from each of the studies) in order to provide more information about the topic of interest than any individual study can provide.¹ How is meta-analysis different from a traditional narrative literature review or from systematic reviews? How are meta-analyses performed, and what is is the best approach to evaluate the evidence presented in a meta-analysis? What are the strengths and limitations of this approach? We attempt to answer these questions in this installment of JAAPA’s Research Corner Online.

BACKGROUND

Meta-analyses are becoming increasingly popular. Before 1970, fewer than five had been published. In the 1970s, 13 meta-analyses were published; in the 1980s, fewer than 100 were done; since 1980, however, more than 5,000 have been conducted. Despite the widespread use of meta-analysis, PAs should remember that this technique has limitations.²

WHAT IS A META-ANALYSIS?

The answer to this question is not entirely clear. To illustrate this point, Egger, Smith, and Schneider conducted a MEDLINE search using the keyword meta-analysis. Their search identified 755 articles that were published in 1999. They randomly selected 100 to examine, and only 59 turned out to be meta-analyses. The other 41 were methodologic papers, traditional reviews, or other types of study summary reports.³

As previously stated, a meta-analysis statistically combines the results of several studies in order to provide more information about the subject than any individual study can provide.¹ Researchers do meta-analyses in hopes that considering the whole will provide information that is greater than the sum of the parts. Before we discuss meta-analysis further, it may be helpful to review other major types of literature reviews.

TYPES OF LITERATURE REVIEWS

A traditional literature review is a summary of published information about a topic. In addition to summarizing the literature, traditional literature reviews often try to draw conclusions by simply comparing the number of studies in favor of a particular viewpoint with the number of studies opposed.² Conclusions based on this approach are prone to bias and error because study methods, study design, sample size, and effect size are not adequately considered or controlled.²

A systematic review goes beyond a traditional narrative literature review. According to Rothstein and colleagues, systematic reviews aim to “provide a more objective appraisal of the evidence than traditional narrative reviews” and “are now generally accepted as the preferred methodology of summarizing a literature.”⁴ Davies and colleagues define systematic reviews as a “complete, unbiased collection of original, high-quality studies that examine the same therapeutic question.” A “complete collection” in a systematic review should include “all relevant studies published and unpublished.”⁵ Systematic reviews should have explicitly defined inclusion and exclusion criteria, and the review’s methods should be clearly described so that the review can be easily replicated.

Not every systematic review is a meta-analysis, but all meta-analyses should include a systematic review. Davies and colleagues advise that “the first requirement for a meta-analysis is a well-executed systematic review.”⁵ Egger and colleagues explain that while “it is always appropriate to systematically review a body of data, it may sometimes be inappropriate, or even misleading, to statistically pool results.”⁶

A meta-analysis is a statistical technique for combining the data from studies included in a systematic review and can only be done if certain criteria are met. In general, studies included in a meta-analysis should have similar study designs, treatments, exposures, controls for confounding, outcome measurements, and patient populations.

HOW ARE META-ANALYSES DONE?

Meta-analysis should be as carefully planned as any other research investigation. Egger and colleagues recommend writing a detailed research protocol in advance of the study.⁷ After the research plan is formulated, the next step is to conduct a comprehensive search for all relevant studies and to specify study eligibility criteria. These steps are essential in conducting a high quality meta-analysis.⁷

After the appropriate studies are selected, the next step in a meta-analysis is the calculation of an appropriate summary statistic for each study.⁸ Summary statistics are usually presented as risk ratios, odds ratios, or risk differences. Sometimes the data are presented as differences in means for continuous data or as hazard ratios for survival time data.

Following the calculation of each study’s summary statistic, the next step is the combination of these statistics into a weighted average. The overall treatment effect is a calculated weighted average of the summary statistics.⁸ Combining the data from all of the trials as if they were from a single large trial is not appropriate, as this would give misleading results.⁸ Egger and colleagues explain: “Results from small studies are more subject to the play of chance and should therefore be given less weight.” The statistical methods used for meta-analysis include a weighted average of the results, in which larger trials have more influence (weight) than smaller ones.⁷

After the data are abstracted and analyzed, the next step is to report the results. The results of a meta-analysis are usually displayed graphically. This graph lists each study on the one axis along with a corresponding circle, square, or triangle to represent the study’s measured effect; and, a horizontal line depicting the confidence intervals around the measured effect. Davies and colleagues explain that “the size of the circle, square, or triangle may vary to reflect the amount of information in that individual study, and that the length of the horizontal line represents the uncertainty of the estimate of the treatment effect for that study.”⁵ The overall measure of effect, calculated after weighing all the studies, is also displayed on this graph (see Figure 1).

STRENGTHS AND LIMITATIONS OF META-ANALYSIS

Meta-analyses offer the opportunity for a greater statistical and objective review of the literature. Meta-analyses can help solve type 2 error problems associated with small sample sizes by increasing statistical power and providing a better estimate of an effect.⁹ However, meta-analyses are still subject to many of the biases found in other research methods.⁷

Because including relevant studies is an essential ingredient in conducting a good meta-analysis, publication bias is of particular concern.⁶ Most readers know of the tendency for authors to submit, and for journals to publish, only studies with positive findings.¹ Therefore, studies that may be well done but that have null or negative results may never reach publication and thus remain unknown to the investigator performing a meta-analysis. This has great implications for a meta-analysis. Rothstein and colleagues describe publication bias as “what occurs whenever the research that appears in the published literature is systematically unrepresentative of the population of completed studies.”⁴ It can also occur whenever the research that is readily available differs in its results from the results of all the research (published and unpublished) that has been done in an area. Literature-based meta-analyses are then subject to publication bias and are therefore in “great danger of drawing the wrong conclusion about what the body of research shows.”⁴ When conducting or evaluating a meta-analysis, the researcher must attempt to evaluate whether the published studies are a true representation of all the work that has been conducted.¹

CONCLUSION

Whenever the results of multiple studies on a given topic give an ambiguous or inconsistent picture and individual studies have small sample size, a meta-analysis may provide a more precise understanding of the true effect. A meta-analysis is a statistical approach for combining the results of several studies in order to produce the single best estimate possible of the effect size. A meta-analysis must be done with care, however, and collaboration with a statistician is advised.

REFERENCES

© 2007 Haymarket Media, Inc. and the American Academy of Physician Assistants. All rights reserved.
Use of jaapa.com subject to License agreement. Please read our Disclaimer and Privacy policy.