Importance: The size of estimated treatment effects on the basis of which the US Food and Drug Administration (FDA) has approved drugs and devices with data from nonrandomized clinical trials (non-RCTs) remains unknown.
Objectives: To determine how often the FDA has authorized novel interventions based on non-RCTs and to assess whether there is an association of the magnitude of treatment effects with FDA requirements for additional testing in randomized clinical trials (RCTs).
Data sources: Overall, 606 drug applications for the Breakthrough Therapy designation from its inception in January 2012 were downloaded from the FDA website in January 2017 and August 2018, and 71 medical device applications for the Humanitarian Device Exemption from its inception in June 1996 were downloaded in August 2017.
Study selection: Approved applications based on non-RCTs were included; RCTs, studies with insufficient information, duplicates, and safety data were excluded.
Data extraction and synthesis: Data were extracted by 2 independent investigators. A statistical association of the magnitude of estimated effect (expressed as an odds ratio) with FDA requests for RCTs was assessed. The data were also meta-analyzed to evaluate the differences in odds ratios between applications that required further testing and those that did not. The results are reported according to Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) guidelines.
Main outcomes and measures: Disease, laboratory, and patient-related outcomes, including disease response or patient survival, were considered.
Results: Among 677 drug and medical device applications, 68 (10.0%) were approved by the FDA based on non-RCTs. Estimates of effects were larger when no further RCTs were required (mean natural logarithm of the odds ratios, 2.18 vs 1.12; odds ratios, 8.85 vs 3.06; P = .03). The meta-analysis results confirmed these findings: estimated effects were approximately 2.5-fold higher for treatments or devices that were approved based on non-RCTs than for treatments or devices for which further testing in RCTs was required (6.30 [95% CI, 4.38-9.06] vs 2.46 [95% CI, 1.70-3.56]; P < .001). Overall, 9 of 677 total applications (1.3%) that were approved on the basis of non-RCTs had relative risks of 10 or greater and 12 (1.7%) had relative risks of 5 or greater. No clear threshold above which the FDA approved interventions based on the magnitude of estimated effect alone was detected.
Conclusions and relevance: In this study, estimated magnitudes of effect were larger among studies for which the FDA did not require RCTs compared with studies for which it did. There was no clear threshold of treatment effect above which no RCTs were requested.