How well can we assess the validity of non-randomised studies of medications? A systematic review of assessment tools
Objective To determine whether assessment tools for non-randomised studies (NRS) address critical elements that influence the validity of NRS findings for comparative safety and effectiveness of medications.Design Systematic review and Delphi survey.Data sources We searched PubMed, Embase, Google, b...
Main Authors: | , , , , , , , , , , |
---|---|
Format: | Article |
Language: | English |
Published: |
BMJ Publishing Group
2021-03-01
|
Series: | BMJ Open |
Online Access: | https://bmjopen.bmj.com/content/11/3/e043961.full |
id |
doaj-c40e150b0656466d9e86abd0a867f449 |
---|---|
record_format |
Article |
collection |
DOAJ |
language |
English |
format |
Article |
sources |
DOAJ |
author |
Dimitri Bennett Grammati Sarri Hongbo Yuan Xuerong Wen Elvira D'Andrea Lydia Vinals Jessica M. Franklin Joan A. Largent Daniela C. Moga Andrew R. Zullo Thomas P. A. Debray |
spellingShingle |
Dimitri Bennett Grammati Sarri Hongbo Yuan Xuerong Wen Elvira D'Andrea Lydia Vinals Jessica M. Franklin Joan A. Largent Daniela C. Moga Andrew R. Zullo Thomas P. A. Debray How well can we assess the validity of non-randomised studies of medications? A systematic review of assessment tools BMJ Open |
author_facet |
Dimitri Bennett Grammati Sarri Hongbo Yuan Xuerong Wen Elvira D'Andrea Lydia Vinals Jessica M. Franklin Joan A. Largent Daniela C. Moga Andrew R. Zullo Thomas P. A. Debray |
author_sort |
Dimitri Bennett |
title |
How well can we assess the validity of non-randomised studies of medications? A systematic review of assessment tools |
title_short |
How well can we assess the validity of non-randomised studies of medications? A systematic review of assessment tools |
title_full |
How well can we assess the validity of non-randomised studies of medications? A systematic review of assessment tools |
title_fullStr |
How well can we assess the validity of non-randomised studies of medications? A systematic review of assessment tools |
title_full_unstemmed |
How well can we assess the validity of non-randomised studies of medications? A systematic review of assessment tools |
title_sort |
how well can we assess the validity of non-randomised studies of medications? a systematic review of assessment tools |
publisher |
BMJ Publishing Group |
series |
BMJ Open |
issn |
2044-6055 |
publishDate |
2021-03-01 |
description |
Objective To determine whether assessment tools for non-randomised studies (NRS) address critical elements that influence the validity of NRS findings for comparative safety and effectiveness of medications.Design Systematic review and Delphi survey.Data sources We searched PubMed, Embase, Google, bibliographies of reviews and websites of influential organisations from inception to November 2019. In parallel, we conducted a Delphi survey among the International Society for Pharmacoepidemiology Comparative Effectiveness Research Special Interest Group to identify key methodological challenges for NRS of medications. We created a framework consisting of the reported methodological challenges to evaluate the selected NRS tools.Study selection Checklists or scales assessing NRS.Data extraction Two reviewers extracted general information and content data related to the prespecified framework.Results Of 44 tools reviewed, 48% (n=21) assess multiple NRS designs, while other tools specifically addressed case–control (n=12, 27%) or cohort studies (n=11, 25%) only. Response rate to the Delphi survey was 73% (35 out of 48 content experts), and a consensus was reached in only two rounds. Most tools evaluated methods for selecting study participants (n=43, 98%), although only one addressed selection bias due to depletion of susceptibles (2%). Many tools addressed the measurement of exposure and outcome (n=40, 91%), and measurement and control for confounders (n=40, 91%). Most tools have at least one item/question on design-specific sources of bias (n=40, 91%), but only a few investigate reverse causation (n=8, 18%), detection bias (n=4, 9%), time-related bias (n=3, 7%), lack of new-user design (n=2, 5%) or active comparator design (n=0). Few tools address the appropriateness of statistical analyses (n=15, 34%), methods for assessing internal (n=15, 34%) or external validity (n=11, 25%) and statistical uncertainty in the findings (n=21, 48%). None of the reviewed tools investigated all the methodological domains and subdomains.Conclusions The acknowledgement of major design-specific sources of bias (eg, lack of new-user design, lack of active comparator design, time-related bias, depletion of susceptibles, reverse causation) and statistical assessment of internal and external validity is currently not sufficiently addressed in most of the existing tools. These critical elements should be integrated to systematically investigate the validity of NRS on comparative safety and effectiveness of medications.Systematic review protocol and registration https://osf.io/es65q. |
url |
https://bmjopen.bmj.com/content/11/3/e043961.full |
work_keys_str_mv |
AT dimitribennett howwellcanweassessthevalidityofnonrandomisedstudiesofmedicationsasystematicreviewofassessmenttools AT grammatisarri howwellcanweassessthevalidityofnonrandomisedstudiesofmedicationsasystematicreviewofassessmenttools AT hongboyuan howwellcanweassessthevalidityofnonrandomisedstudiesofmedicationsasystematicreviewofassessmenttools AT xuerongwen howwellcanweassessthevalidityofnonrandomisedstudiesofmedicationsasystematicreviewofassessmenttools AT elviradandrea howwellcanweassessthevalidityofnonrandomisedstudiesofmedicationsasystematicreviewofassessmenttools AT lydiavinals howwellcanweassessthevalidityofnonrandomisedstudiesofmedicationsasystematicreviewofassessmenttools AT jessicamfranklin howwellcanweassessthevalidityofnonrandomisedstudiesofmedicationsasystematicreviewofassessmenttools AT joanalargent howwellcanweassessthevalidityofnonrandomisedstudiesofmedicationsasystematicreviewofassessmenttools AT danielacmoga howwellcanweassessthevalidityofnonrandomisedstudiesofmedicationsasystematicreviewofassessmenttools AT andrewrzullo howwellcanweassessthevalidityofnonrandomisedstudiesofmedicationsasystematicreviewofassessmenttools AT thomaspadebray howwellcanweassessthevalidityofnonrandomisedstudiesofmedicationsasystematicreviewofassessmenttools |
_version_ |
1721329262175191040 |
spelling |
doaj-c40e150b0656466d9e86abd0a867f4492021-07-02T13:09:37ZengBMJ Publishing GroupBMJ Open2044-60552021-03-0111310.1136/bmjopen-2020-043961How well can we assess the validity of non-randomised studies of medications? A systematic review of assessment toolsDimitri Bennett0Grammati Sarri1Hongbo Yuan2Xuerong Wen3Elvira D'Andrea4Lydia Vinals5Jessica M. Franklin6Joan A. Largent7Daniela C. Moga8Andrew R. Zullo9Thomas P. A. Debray10Pharmacoepidemiology, Takeda Pharmaceutical, Cambridge, Massachusetts, USAReal World Evidence Sciences, Visible Analytics Ltd, Oxford, UKCanadian Agency for Drugs and Technologies in Health (CADTH), Ottawa, Ontario, CanadaDepartment of Pharmacy Practice, University of Rhode Island, Kingston, RI, USADivision of Pharmacoepidemiology and Pharmacoeconomics, Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, Massachusetts, USAHEOR Department, Cytel Inc, Toronto, Quebec, CanadaDivision of Pharmacoepidemiology and Pharmacoeconomics, Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, Massachusetts, USAReal-World Solutions, IQVIA, California, Los Angeles, USADepartment of Pharmacy Practice and Science, College of Pharmacy, University of Kentucky, Lexington, KY, USADepartment of Health Services, Policy, and Practice, Brown University, Providence, Rhode Island, USADepartment of Epidemiology, Julius Center for Health Sciences and Primary Care, Utrecht, The NetherlandsObjective To determine whether assessment tools for non-randomised studies (NRS) address critical elements that influence the validity of NRS findings for comparative safety and effectiveness of medications.Design Systematic review and Delphi survey.Data sources We searched PubMed, Embase, Google, bibliographies of reviews and websites of influential organisations from inception to November 2019. In parallel, we conducted a Delphi survey among the International Society for Pharmacoepidemiology Comparative Effectiveness Research Special Interest Group to identify key methodological challenges for NRS of medications. We created a framework consisting of the reported methodological challenges to evaluate the selected NRS tools.Study selection Checklists or scales assessing NRS.Data extraction Two reviewers extracted general information and content data related to the prespecified framework.Results Of 44 tools reviewed, 48% (n=21) assess multiple NRS designs, while other tools specifically addressed case–control (n=12, 27%) or cohort studies (n=11, 25%) only. Response rate to the Delphi survey was 73% (35 out of 48 content experts), and a consensus was reached in only two rounds. Most tools evaluated methods for selecting study participants (n=43, 98%), although only one addressed selection bias due to depletion of susceptibles (2%). Many tools addressed the measurement of exposure and outcome (n=40, 91%), and measurement and control for confounders (n=40, 91%). Most tools have at least one item/question on design-specific sources of bias (n=40, 91%), but only a few investigate reverse causation (n=8, 18%), detection bias (n=4, 9%), time-related bias (n=3, 7%), lack of new-user design (n=2, 5%) or active comparator design (n=0). Few tools address the appropriateness of statistical analyses (n=15, 34%), methods for assessing internal (n=15, 34%) or external validity (n=11, 25%) and statistical uncertainty in the findings (n=21, 48%). None of the reviewed tools investigated all the methodological domains and subdomains.Conclusions The acknowledgement of major design-specific sources of bias (eg, lack of new-user design, lack of active comparator design, time-related bias, depletion of susceptibles, reverse causation) and statistical assessment of internal and external validity is currently not sufficiently addressed in most of the existing tools. These critical elements should be integrated to systematically investigate the validity of NRS on comparative safety and effectiveness of medications.Systematic review protocol and registration https://osf.io/es65q.https://bmjopen.bmj.com/content/11/3/e043961.full |