Preventing false discovery of heterogeneous treatment effect subgroups in randomized trials
Abstract Background Heterogeneous treatment effects (HTEs), or systematic differences in treatment effectiveness among participants with different observable features, may be important when applying trial results to clinical practice. Current methods suffer from a potential for false detection of HT...
Main Authors: | , , |
---|---|
Format: | Article |
Language: | English |
Published: |
BMC
2018-07-01
|
Series: | Trials |
Subjects: | |
Online Access: | http://link.springer.com/article/10.1186/s13063-018-2774-5 |
id |
doaj-bb257b64785245808d6186ce9f9adcce |
---|---|
record_format |
Article |
spelling |
doaj-bb257b64785245808d6186ce9f9adcce2020-11-24T21:04:43ZengBMCTrials1745-62152018-07-0119111510.1186/s13063-018-2774-5Preventing false discovery of heterogeneous treatment effect subgroups in randomized trialsJoseph Rigdon0Michael Baiocchi1Sanjay Basu2Quantitative Sciences Unit, Stanford University School of MedicineStanford Prevention Research Center, Stanford University School of MedicineDepartments of Medicine and of Health Research and Policy, Center for Primary Care and Outcomes Research and Center for Population Health Sciences, Stanford University School of MedicineAbstract Background Heterogeneous treatment effects (HTEs), or systematic differences in treatment effectiveness among participants with different observable features, may be important when applying trial results to clinical practice. Current methods suffer from a potential for false detection of HTEs due to imbalances in covariates between candidate subgroups. Methods We introduce a new method, matching plus classification and regression trees (mCART), that yields balance in covariates in identified HTE subgroups. We compared mCART to a classical method (logistic regression [LR] with backwards covariate selection using the Akaike information criterion ) and two machine-learning approaches increasingly applied to HTE detection (random forest [RF] and gradient RF) in simulations with a binary outcome with known HTE subgroups. We considered an N = 200 phase II oncology trial where there were either no HTEs (1A) or two HTE subgroups (1B) and an N = 6000 phase III cardiovascular disease trial where there were either no HTEs (2A) or four HTE subgroups (2B). Additionally, we considered an N = 6000 phase III cardiovascular disease trial where there was no average treatment effect but there were four HTE subgroups (2C). Results In simulations 1A and 2A (no HTEs), mCART did not identify any HTE subgroups, whereas LR found 2 and 448, RF 5 and 2, and gradient RF 5 and 24, respectively (all false positives). In simulation 1B, mCART failed to identify the two true HTE subgroups whereas LR found 4, RF 6, and gradient RF 10 (half or more of which were false positives). In simulations 2B and 2C, mCART captured the four true HTE subgroups, whereas the other methods found only false positives. All HTE subgroups identified by mCART had acceptable treated vs. control covariate balance with absolute standardized differences less than 0.2, whereas the absolute standardized differences for the other methods typically exceeded 0.2. The imbalance in covariates in identified subgroups for LR, RF, and gradient RF indicates the false HTE detection may have been due to confounding. Conclusions Covariate imbalances may be producing false positives in subgroup analyses. mCART could be a useful tool to help prevent the false discovery of HTE subgroups in secondary analyses of randomized trial data.http://link.springer.com/article/10.1186/s13063-018-2774-5Classification and regression treesDecision support toolHeterogeneous treatment effectsMatching |
collection |
DOAJ |
language |
English |
format |
Article |
sources |
DOAJ |
author |
Joseph Rigdon Michael Baiocchi Sanjay Basu |
spellingShingle |
Joseph Rigdon Michael Baiocchi Sanjay Basu Preventing false discovery of heterogeneous treatment effect subgroups in randomized trials Trials Classification and regression trees Decision support tool Heterogeneous treatment effects Matching |
author_facet |
Joseph Rigdon Michael Baiocchi Sanjay Basu |
author_sort |
Joseph Rigdon |
title |
Preventing false discovery of heterogeneous treatment effect subgroups in randomized trials |
title_short |
Preventing false discovery of heterogeneous treatment effect subgroups in randomized trials |
title_full |
Preventing false discovery of heterogeneous treatment effect subgroups in randomized trials |
title_fullStr |
Preventing false discovery of heterogeneous treatment effect subgroups in randomized trials |
title_full_unstemmed |
Preventing false discovery of heterogeneous treatment effect subgroups in randomized trials |
title_sort |
preventing false discovery of heterogeneous treatment effect subgroups in randomized trials |
publisher |
BMC |
series |
Trials |
issn |
1745-6215 |
publishDate |
2018-07-01 |
description |
Abstract Background Heterogeneous treatment effects (HTEs), or systematic differences in treatment effectiveness among participants with different observable features, may be important when applying trial results to clinical practice. Current methods suffer from a potential for false detection of HTEs due to imbalances in covariates between candidate subgroups. Methods We introduce a new method, matching plus classification and regression trees (mCART), that yields balance in covariates in identified HTE subgroups. We compared mCART to a classical method (logistic regression [LR] with backwards covariate selection using the Akaike information criterion ) and two machine-learning approaches increasingly applied to HTE detection (random forest [RF] and gradient RF) in simulations with a binary outcome with known HTE subgroups. We considered an N = 200 phase II oncology trial where there were either no HTEs (1A) or two HTE subgroups (1B) and an N = 6000 phase III cardiovascular disease trial where there were either no HTEs (2A) or four HTE subgroups (2B). Additionally, we considered an N = 6000 phase III cardiovascular disease trial where there was no average treatment effect but there were four HTE subgroups (2C). Results In simulations 1A and 2A (no HTEs), mCART did not identify any HTE subgroups, whereas LR found 2 and 448, RF 5 and 2, and gradient RF 5 and 24, respectively (all false positives). In simulation 1B, mCART failed to identify the two true HTE subgroups whereas LR found 4, RF 6, and gradient RF 10 (half or more of which were false positives). In simulations 2B and 2C, mCART captured the four true HTE subgroups, whereas the other methods found only false positives. All HTE subgroups identified by mCART had acceptable treated vs. control covariate balance with absolute standardized differences less than 0.2, whereas the absolute standardized differences for the other methods typically exceeded 0.2. The imbalance in covariates in identified subgroups for LR, RF, and gradient RF indicates the false HTE detection may have been due to confounding. Conclusions Covariate imbalances may be producing false positives in subgroup analyses. mCART could be a useful tool to help prevent the false discovery of HTE subgroups in secondary analyses of randomized trial data. |
topic |
Classification and regression trees Decision support tool Heterogeneous treatment effects Matching |
url |
http://link.springer.com/article/10.1186/s13063-018-2774-5 |
work_keys_str_mv |
AT josephrigdon preventingfalsediscoveryofheterogeneoustreatmenteffectsubgroupsinrandomizedtrials AT michaelbaiocchi preventingfalsediscoveryofheterogeneoustreatmenteffectsubgroupsinrandomizedtrials AT sanjaybasu preventingfalsediscoveryofheterogeneoustreatmenteffectsubgroupsinrandomizedtrials |
_version_ |
1716770030355480576 |