Fragility of clinical trials across research fields: A synthesis of methodological reviews

Abstract

Background: Randomized controlled trials (RCTs) are often used to inform clinical practice and it is desirable that their results be robust. A fragility index (FI), defined as the smallest number of participants in whom an outcome change from non-event to event would turn a statistically significant result to a non-significant result, can be computed to measure robustness. We sought to determine the distribution of fragility indices across various research areas and summarized the factors associated with fragility.

Methods: We searched PubMed between February 2014 and May 2019 and included reviews that reported on fragility indices and the associated factors. Two investigators independently screened articles for eligibility and extracted all relevant data from each review. Fragility indices were pooled using random effects meta-analysis.

Results: Twenty-four (24) reviews met the inclusion criteria. They contained a median of 41 trials (first quartile [Q1]-third quartile [Q3]: 17-120). The overall mean FI across different fields of research was 4 (95% confidence interval [CI] 3-5), indicating a high level of fragility. Higher journal impact factor, larger sample size, bigger effect size, more outcome events, a lower p-value, and adequate allocation concealment were reported to be associated with the higher FI. The ecological correlation between median FI and median sample size (22 studies) was 0.95 (95% CI 0.58-0.99).

Conclusion: Trials across various fields of research are frequently fragile. We also identified some factors associated with fragility. Researchers should consider strategies to enhance the robustness of studies and minimize fragility.

Keywords: Events; Fragility index; Randomized trials; Robustness; Stability.