Predicting clinical outcomes among hospitalized COVID-19 patients using both local and published models
Abstract Background Many models are published which predict outcomes in hospitalized COVID-19 patients. The generalizability of many is unknown. We evaluated the performance of selected models from the literature and our own models to predict outcomes in patients at our institution. Methods We searc...
Main Authors: | , , , , , , , , |
---|---|
Format: | Article |
Language: | English |
Published: |
BMC
2021-07-01
|
Series: | BMC Medical Informatics and Decision Making |
Subjects: | |
Online Access: | https://doi.org/10.1186/s12911-021-01576-w |
id |
doaj-33d441a4fa2145288aa5b9b553d578d1 |
---|---|
record_format |
Article |
spelling |
doaj-33d441a4fa2145288aa5b9b553d578d12021-07-25T11:32:59ZengBMCBMC Medical Informatics and Decision Making1472-69472021-07-0121111810.1186/s12911-021-01576-wPredicting clinical outcomes among hospitalized COVID-19 patients using both local and published modelsWilliam Galanter0Jorge Mario Rodríguez-Fernández1Kevin Chow2Samuel Harford3Karl M. Kochendorfer4Maryam Pishgar5Julian Theis6John Zulueta7Houshang Darabi8Departments of Medicine and Pharmacy Systems, Outcomes and Policy, University of Illinois At Chicago (UIC)Department of Neurology, Clinical Informatics Fellowship, UICCollege of Medicine, UICDepartment of Mechanical and Industrial Engineering, UICDepartment of Family and Community Medicine, UICDepartment of Mechanical and Industrial Engineering, UICDepartment of Mechanical and Industrial Engineering, UICDepartment of Psychiatry, UICDepartment of Mechanical and Industrial Engineering, UICAbstract Background Many models are published which predict outcomes in hospitalized COVID-19 patients. The generalizability of many is unknown. We evaluated the performance of selected models from the literature and our own models to predict outcomes in patients at our institution. Methods We searched the literature for models predicting outcomes in inpatients with COVID-19. We produced models of mortality or criticality (mortality or ICU admission) in a development cohort. We tested external models which provided sufficient information and our models using a test cohort of our most recent patients. The performance of models was compared using the area under the receiver operator curve (AUC). Results Our literature review yielded 41 papers. Of those, 8 were found to have sufficient documentation and concordance with features available in our cohort to implement in our test cohort. All models were from Chinese patients. One model predicted criticality and seven mortality. Tested against the test cohort, internal models had an AUC of 0.84 (0.74–0.94) for mortality and 0.83 (0.76–0.90) for criticality. The best external model had an AUC of 0.89 (0.82–0.96) using three variables, another an AUC of 0.84 (0.78–0.91) using ten variables. AUC’s ranged from 0.68 to 0.89. On average, models tested were unable to produce predictions in 27% of patients due to missing lab data. Conclusion Despite differences in pandemic timeline, race, and socio-cultural healthcare context some models derived in China performed well. For healthcare organizations considering implementation of an external model, concordance between the features used in the model and features available in their own patients may be important. Analysis of both local and external models should be done to help decide on what prediction method is used to provide clinical decision support to clinicians treating COVID-19 patients as well as what lab tests should be included in order sets.https://doi.org/10.1186/s12911-021-01576-wMortalityHospitalizationCOVID-19Statistical modelPredictionModel generalizability |
collection |
DOAJ |
language |
English |
format |
Article |
sources |
DOAJ |
author |
William Galanter Jorge Mario Rodríguez-Fernández Kevin Chow Samuel Harford Karl M. Kochendorfer Maryam Pishgar Julian Theis John Zulueta Houshang Darabi |
spellingShingle |
William Galanter Jorge Mario Rodríguez-Fernández Kevin Chow Samuel Harford Karl M. Kochendorfer Maryam Pishgar Julian Theis John Zulueta Houshang Darabi Predicting clinical outcomes among hospitalized COVID-19 patients using both local and published models BMC Medical Informatics and Decision Making Mortality Hospitalization COVID-19 Statistical model Prediction Model generalizability |
author_facet |
William Galanter Jorge Mario Rodríguez-Fernández Kevin Chow Samuel Harford Karl M. Kochendorfer Maryam Pishgar Julian Theis John Zulueta Houshang Darabi |
author_sort |
William Galanter |
title |
Predicting clinical outcomes among hospitalized COVID-19 patients using both local and published models |
title_short |
Predicting clinical outcomes among hospitalized COVID-19 patients using both local and published models |
title_full |
Predicting clinical outcomes among hospitalized COVID-19 patients using both local and published models |
title_fullStr |
Predicting clinical outcomes among hospitalized COVID-19 patients using both local and published models |
title_full_unstemmed |
Predicting clinical outcomes among hospitalized COVID-19 patients using both local and published models |
title_sort |
predicting clinical outcomes among hospitalized covid-19 patients using both local and published models |
publisher |
BMC |
series |
BMC Medical Informatics and Decision Making |
issn |
1472-6947 |
publishDate |
2021-07-01 |
description |
Abstract Background Many models are published which predict outcomes in hospitalized COVID-19 patients. The generalizability of many is unknown. We evaluated the performance of selected models from the literature and our own models to predict outcomes in patients at our institution. Methods We searched the literature for models predicting outcomes in inpatients with COVID-19. We produced models of mortality or criticality (mortality or ICU admission) in a development cohort. We tested external models which provided sufficient information and our models using a test cohort of our most recent patients. The performance of models was compared using the area under the receiver operator curve (AUC). Results Our literature review yielded 41 papers. Of those, 8 were found to have sufficient documentation and concordance with features available in our cohort to implement in our test cohort. All models were from Chinese patients. One model predicted criticality and seven mortality. Tested against the test cohort, internal models had an AUC of 0.84 (0.74–0.94) for mortality and 0.83 (0.76–0.90) for criticality. The best external model had an AUC of 0.89 (0.82–0.96) using three variables, another an AUC of 0.84 (0.78–0.91) using ten variables. AUC’s ranged from 0.68 to 0.89. On average, models tested were unable to produce predictions in 27% of patients due to missing lab data. Conclusion Despite differences in pandemic timeline, race, and socio-cultural healthcare context some models derived in China performed well. For healthcare organizations considering implementation of an external model, concordance between the features used in the model and features available in their own patients may be important. Analysis of both local and external models should be done to help decide on what prediction method is used to provide clinical decision support to clinicians treating COVID-19 patients as well as what lab tests should be included in order sets. |
topic |
Mortality Hospitalization COVID-19 Statistical model Prediction Model generalizability |
url |
https://doi.org/10.1186/s12911-021-01576-w |
work_keys_str_mv |
AT williamgalanter predictingclinicaloutcomesamonghospitalizedcovid19patientsusingbothlocalandpublishedmodels AT jorgemariorodriguezfernandez predictingclinicaloutcomesamonghospitalizedcovid19patientsusingbothlocalandpublishedmodels AT kevinchow predictingclinicaloutcomesamonghospitalizedcovid19patientsusingbothlocalandpublishedmodels AT samuelharford predictingclinicaloutcomesamonghospitalizedcovid19patientsusingbothlocalandpublishedmodels AT karlmkochendorfer predictingclinicaloutcomesamonghospitalizedcovid19patientsusingbothlocalandpublishedmodels AT maryampishgar predictingclinicaloutcomesamonghospitalizedcovid19patientsusingbothlocalandpublishedmodels AT juliantheis predictingclinicaloutcomesamonghospitalizedcovid19patientsusingbothlocalandpublishedmodels AT johnzulueta predictingclinicaloutcomesamonghospitalizedcovid19patientsusingbothlocalandpublishedmodels AT houshangdarabi predictingclinicaloutcomesamonghospitalizedcovid19patientsusingbothlocalandpublishedmodels |
_version_ |
1721283059066601472 |