Improving risk adjustment in the PRAiS (Partial Risk Adjustment in Surgery) model for mortality after paediatric cardiac surgery and improving public understanding of its use in monitoring outcomes
Background: In 2011, we developed a risk model for 30-day mortality after children’s heart surgery. The PRAiS (Partial Risk Adjustment in Surgery) model uses data on the procedure performed, diagnosis, age, weight and comorbidity. Our treatment of comorbidity was simplistic because of data quality....
Main Authors: | , , , , , , , , , , , , , , , , , , , , , |
---|---|
Format: | Article |
Language: | English |
Published: |
NIHR Journals Library
2017-07-01
|
Series: | Health Services and Delivery Research |
Online Access: | https://doi.org/10.3310/hsdr05230 |
id |
doaj-feef09d1577143d9a675e41d0519ebfd |
---|---|
record_format |
Article |
collection |
DOAJ |
language |
English |
format |
Article |
sources |
DOAJ |
author |
Christina Pagel Libby Rogers Katherine Brown Gareth Ambler David Anderson David Barron Emily Blackshaw Sonya Crowe Kate English Rodney Franklin Emily Jesper Laura Meagher Mike Pearson Tim Rakow Marta Salamonowicz David Spiegelhalter John Stickley Joanne Thomas Shane Tibby Victor Tsang Martin Utley Thomas Witter |
spellingShingle |
Christina Pagel Libby Rogers Katherine Brown Gareth Ambler David Anderson David Barron Emily Blackshaw Sonya Crowe Kate English Rodney Franklin Emily Jesper Laura Meagher Mike Pearson Tim Rakow Marta Salamonowicz David Spiegelhalter John Stickley Joanne Thomas Shane Tibby Victor Tsang Martin Utley Thomas Witter Improving risk adjustment in the PRAiS (Partial Risk Adjustment in Surgery) model for mortality after paediatric cardiac surgery and improving public understanding of its use in monitoring outcomes Health Services and Delivery Research |
author_facet |
Christina Pagel Libby Rogers Katherine Brown Gareth Ambler David Anderson David Barron Emily Blackshaw Sonya Crowe Kate English Rodney Franklin Emily Jesper Laura Meagher Mike Pearson Tim Rakow Marta Salamonowicz David Spiegelhalter John Stickley Joanne Thomas Shane Tibby Victor Tsang Martin Utley Thomas Witter |
author_sort |
Christina Pagel |
title |
Improving risk adjustment in the PRAiS (Partial Risk Adjustment in Surgery) model for mortality after paediatric cardiac surgery and improving public understanding of its use in monitoring outcomes |
title_short |
Improving risk adjustment in the PRAiS (Partial Risk Adjustment in Surgery) model for mortality after paediatric cardiac surgery and improving public understanding of its use in monitoring outcomes |
title_full |
Improving risk adjustment in the PRAiS (Partial Risk Adjustment in Surgery) model for mortality after paediatric cardiac surgery and improving public understanding of its use in monitoring outcomes |
title_fullStr |
Improving risk adjustment in the PRAiS (Partial Risk Adjustment in Surgery) model for mortality after paediatric cardiac surgery and improving public understanding of its use in monitoring outcomes |
title_full_unstemmed |
Improving risk adjustment in the PRAiS (Partial Risk Adjustment in Surgery) model for mortality after paediatric cardiac surgery and improving public understanding of its use in monitoring outcomes |
title_sort |
improving risk adjustment in the prais (partial risk adjustment in surgery) model for mortality after paediatric cardiac surgery and improving public understanding of its use in monitoring outcomes |
publisher |
NIHR Journals Library |
series |
Health Services and Delivery Research |
issn |
2050-4349 2050-4357 |
publishDate |
2017-07-01 |
description |
Background: In 2011, we developed a risk model for 30-day mortality after children’s heart surgery. The PRAiS (Partial Risk Adjustment in Surgery) model uses data on the procedure performed, diagnosis, age, weight and comorbidity. Our treatment of comorbidity was simplistic because of data quality. Software that implements PRAiS is used by the National Congenital Heart Disease Audit (NCHDA) in its audit work. The use of PRAiS triggered the temporary suspension of surgery at one unit in 2013. The public anger that surrounded this illustrated the need for public resources around outcomes monitoring. Objectives: (1) To improve the PRAiS risk model by incorporating more information about comorbidities. (2) To develop online resources for the public to help them to understand published mortality data. Design: Objective 1 The outcome measure was death within 30 days of the start of each surgical episode of care. The analysts worked with an expert panel of clinical and data management representatives. Model development followed an iterative process of clinical discussion of risk factors, development of regression models and assessment of model performance under cross-validation. Performance was measured using the area under the receiving operator characteristic (AUROC) curve and calibration in the cross-validation test sets. The final model was further assessed in a 2014–15 validation data set. Objective 2 We developed draft website material that we iteratively tested through four sets of two workshops (one workshop for parents of children who had undergone heart surgery and one workshop for other interested users). Each workshop recruited new participants. The academic psychologists ran two sets of three experiments to explore further understanding of the web content. Data: We used pseudonymised NCHDA data from April 2009 to April 2014. We later unexpectedly received a further year of data (2014–15), which became a prospective validation set. Results: Objective 1 The cleaned 2009–14 data comprised 21,838 30-day surgical episodes, with 539 deaths. The 2014–15 data contained 4207 episodes, with 97 deaths. The final regression model included four new comorbidity groupings. Under cross-validation, the model had a median AUROC curve of 0.83 (total range 0.82 to 0.83), a median calibration slope of 0.92 (total range 0.64 to 1.25) and a median intercept of –0.23 (range –1.08 to 0.85). In the validation set, the AUROC curve was 0.86 [95% confidence interval (CI) 0.83 to 0.89], and its calibration slope and intercept were 1.01 (95% CI 0.83 to 1.18) and 0.11 (95% CI –0.45 to 0.67), respectively. We recalibrated the final model on 2009–15 data and updated the PRAiS software. Objective 2 We coproduced a website (http://childrensheartsurgery.info/) that provides interactive exploration of the data, two animations and background information. It was launched in June 2016 and was very well received. Limitations: We needed to use discharge status as a proxy for 30-day life status for the 14% of overseas patients without a NHS number. We did not have sufficient time or resources to extensively test the usability and take-up of the website following its launch. Conclusions: The project successfully achieved its stated aims. A key theme throughout has been the importance of collaboration and coproduction. In particular for aim 2, we generated a great deal of generalisable learning about how to communicate complex clinical and mathematical information. Further work: Extending our codevelopment approach to cover many other aspects of quality measurement across congenital heart disease and other specialised NHS services. Funding: The National Institute for Health Research Health Services and Delivery Research programme. |
url |
https://doi.org/10.3310/hsdr05230 |
work_keys_str_mv |
AT christinapagel improvingriskadjustmentinthepraispartialriskadjustmentinsurgerymodelformortalityafterpaediatriccardiacsurgeryandimprovingpublicunderstandingofitsuseinmonitoringoutcomes AT libbyrogers improvingriskadjustmentinthepraispartialriskadjustmentinsurgerymodelformortalityafterpaediatriccardiacsurgeryandimprovingpublicunderstandingofitsuseinmonitoringoutcomes AT katherinebrown improvingriskadjustmentinthepraispartialriskadjustmentinsurgerymodelformortalityafterpaediatriccardiacsurgeryandimprovingpublicunderstandingofitsuseinmonitoringoutcomes AT garethambler improvingriskadjustmentinthepraispartialriskadjustmentinsurgerymodelformortalityafterpaediatriccardiacsurgeryandimprovingpublicunderstandingofitsuseinmonitoringoutcomes AT davidanderson improvingriskadjustmentinthepraispartialriskadjustmentinsurgerymodelformortalityafterpaediatriccardiacsurgeryandimprovingpublicunderstandingofitsuseinmonitoringoutcomes AT davidbarron improvingriskadjustmentinthepraispartialriskadjustmentinsurgerymodelformortalityafterpaediatriccardiacsurgeryandimprovingpublicunderstandingofitsuseinmonitoringoutcomes AT emilyblackshaw improvingriskadjustmentinthepraispartialriskadjustmentinsurgerymodelformortalityafterpaediatriccardiacsurgeryandimprovingpublicunderstandingofitsuseinmonitoringoutcomes AT sonyacrowe improvingriskadjustmentinthepraispartialriskadjustmentinsurgerymodelformortalityafterpaediatriccardiacsurgeryandimprovingpublicunderstandingofitsuseinmonitoringoutcomes AT kateenglish improvingriskadjustmentinthepraispartialriskadjustmentinsurgerymodelformortalityafterpaediatriccardiacsurgeryandimprovingpublicunderstandingofitsuseinmonitoringoutcomes AT rodneyfranklin improvingriskadjustmentinthepraispartialriskadjustmentinsurgerymodelformortalityafterpaediatriccardiacsurgeryandimprovingpublicunderstandingofitsuseinmonitoringoutcomes AT emilyjesper improvingriskadjustmentinthepraispartialriskadjustmentinsurgerymodelformortalityafterpaediatriccardiacsurgeryandimprovingpublicunderstandingofitsuseinmonitoringoutcomes AT laurameagher improvingriskadjustmentinthepraispartialriskadjustmentinsurgerymodelformortalityafterpaediatriccardiacsurgeryandimprovingpublicunderstandingofitsuseinmonitoringoutcomes AT mikepearson improvingriskadjustmentinthepraispartialriskadjustmentinsurgerymodelformortalityafterpaediatriccardiacsurgeryandimprovingpublicunderstandingofitsuseinmonitoringoutcomes AT timrakow improvingriskadjustmentinthepraispartialriskadjustmentinsurgerymodelformortalityafterpaediatriccardiacsurgeryandimprovingpublicunderstandingofitsuseinmonitoringoutcomes AT martasalamonowicz improvingriskadjustmentinthepraispartialriskadjustmentinsurgerymodelformortalityafterpaediatriccardiacsurgeryandimprovingpublicunderstandingofitsuseinmonitoringoutcomes AT davidspiegelhalter improvingriskadjustmentinthepraispartialriskadjustmentinsurgerymodelformortalityafterpaediatriccardiacsurgeryandimprovingpublicunderstandingofitsuseinmonitoringoutcomes AT johnstickley improvingriskadjustmentinthepraispartialriskadjustmentinsurgerymodelformortalityafterpaediatriccardiacsurgeryandimprovingpublicunderstandingofitsuseinmonitoringoutcomes AT joannethomas improvingriskadjustmentinthepraispartialriskadjustmentinsurgerymodelformortalityafterpaediatriccardiacsurgeryandimprovingpublicunderstandingofitsuseinmonitoringoutcomes AT shanetibby improvingriskadjustmentinthepraispartialriskadjustmentinsurgerymodelformortalityafterpaediatriccardiacsurgeryandimprovingpublicunderstandingofitsuseinmonitoringoutcomes AT victortsang improvingriskadjustmentinthepraispartialriskadjustmentinsurgerymodelformortalityafterpaediatriccardiacsurgeryandimprovingpublicunderstandingofitsuseinmonitoringoutcomes AT martinutley improvingriskadjustmentinthepraispartialriskadjustmentinsurgerymodelformortalityafterpaediatriccardiacsurgeryandimprovingpublicunderstandingofitsuseinmonitoringoutcomes AT thomaswitter improvingriskadjustmentinthepraispartialriskadjustmentinsurgerymodelformortalityafterpaediatriccardiacsurgeryandimprovingpublicunderstandingofitsuseinmonitoringoutcomes |
_version_ |
1716809110128689152 |
spelling |
doaj-feef09d1577143d9a675e41d0519ebfd2020-11-24T20:48:03ZengNIHR Journals LibraryHealth Services and Delivery Research2050-43492050-43572017-07-0152310.3310/hsdr0523014/19/13Improving risk adjustment in the PRAiS (Partial Risk Adjustment in Surgery) model for mortality after paediatric cardiac surgery and improving public understanding of its use in monitoring outcomesChristina Pagel0Libby Rogers1Katherine Brown2Gareth Ambler3David Anderson4David Barron5Emily Blackshaw6Sonya Crowe7Kate English8Rodney Franklin9Emily Jesper10Laura Meagher11Mike Pearson12Tim Rakow13Marta Salamonowicz14David Spiegelhalter15John Stickley16Joanne Thomas17Shane Tibby18Victor Tsang19Martin Utley20Thomas Witter21Clinical Operational Research Unit, University College London, London, UKClinical Operational Research Unit, University College London, London, UKCardiac, Critical Care and Respiratory Division, Great Ormond Street Hospital for Children NHS Foundation Trust, London, UKDepartment of Statistical Science, University College London, London, UKCardiology and Critical Care, Evelina London Children’s Hospital, Guy’s and St Thomas’ NHS Foundation Trust, London, UKCardiothoracic Surgery, Birmingham Children’s Hospital, Birmingham, UKDepartment of Psychology, King’s College London, London, UKClinical Operational Research Unit, University College London, London, UKCardiology, Leeds Teaching Hospitals NHS Trust, Leeds, UKPaediatric Cardiology, Royal Brompton & Harefield NHS Foundation Trust, London, UKSense about Science, London, UKTechnology Development Group, Dairsie, UKStatistical Laboratory, Centre for Mathematical Sciences, University of Cambridge, Cambridge, UKDepartment of Psychology, King’s College London, London, UKChildren’s Heart Federation, Witham, UKStatistical Laboratory, Centre for Mathematical Sciences, University of Cambridge, Cambridge, UKCardiothoracic Surgery, Birmingham Children’s Hospital, Birmingham, UKSense about Science, London, UKCardiology and Critical Care, Evelina London Children’s Hospital, Guy’s and St Thomas’ NHS Foundation Trust, London, UKCardiac, Critical Care and Respiratory Division, Great Ormond Street Hospital for Children NHS Foundation Trust, London, UKClinical Operational Research Unit, University College London, London, UKCardiology and Critical Care, Evelina London Children’s Hospital, Guy’s and St Thomas’ NHS Foundation Trust, London, UKBackground: In 2011, we developed a risk model for 30-day mortality after children’s heart surgery. The PRAiS (Partial Risk Adjustment in Surgery) model uses data on the procedure performed, diagnosis, age, weight and comorbidity. Our treatment of comorbidity was simplistic because of data quality. Software that implements PRAiS is used by the National Congenital Heart Disease Audit (NCHDA) in its audit work. The use of PRAiS triggered the temporary suspension of surgery at one unit in 2013. The public anger that surrounded this illustrated the need for public resources around outcomes monitoring. Objectives: (1) To improve the PRAiS risk model by incorporating more information about comorbidities. (2) To develop online resources for the public to help them to understand published mortality data. Design: Objective 1 The outcome measure was death within 30 days of the start of each surgical episode of care. The analysts worked with an expert panel of clinical and data management representatives. Model development followed an iterative process of clinical discussion of risk factors, development of regression models and assessment of model performance under cross-validation. Performance was measured using the area under the receiving operator characteristic (AUROC) curve and calibration in the cross-validation test sets. The final model was further assessed in a 2014–15 validation data set. Objective 2 We developed draft website material that we iteratively tested through four sets of two workshops (one workshop for parents of children who had undergone heart surgery and one workshop for other interested users). Each workshop recruited new participants. The academic psychologists ran two sets of three experiments to explore further understanding of the web content. Data: We used pseudonymised NCHDA data from April 2009 to April 2014. We later unexpectedly received a further year of data (2014–15), which became a prospective validation set. Results: Objective 1 The cleaned 2009–14 data comprised 21,838 30-day surgical episodes, with 539 deaths. The 2014–15 data contained 4207 episodes, with 97 deaths. The final regression model included four new comorbidity groupings. Under cross-validation, the model had a median AUROC curve of 0.83 (total range 0.82 to 0.83), a median calibration slope of 0.92 (total range 0.64 to 1.25) and a median intercept of –0.23 (range –1.08 to 0.85). In the validation set, the AUROC curve was 0.86 [95% confidence interval (CI) 0.83 to 0.89], and its calibration slope and intercept were 1.01 (95% CI 0.83 to 1.18) and 0.11 (95% CI –0.45 to 0.67), respectively. We recalibrated the final model on 2009–15 data and updated the PRAiS software. Objective 2 We coproduced a website (http://childrensheartsurgery.info/) that provides interactive exploration of the data, two animations and background information. It was launched in June 2016 and was very well received. Limitations: We needed to use discharge status as a proxy for 30-day life status for the 14% of overseas patients without a NHS number. We did not have sufficient time or resources to extensively test the usability and take-up of the website following its launch. Conclusions: The project successfully achieved its stated aims. A key theme throughout has been the importance of collaboration and coproduction. In particular for aim 2, we generated a great deal of generalisable learning about how to communicate complex clinical and mathematical information. Further work: Extending our codevelopment approach to cover many other aspects of quality measurement across congenital heart disease and other specialised NHS services. Funding: The National Institute for Health Research Health Services and Delivery Research programme.https://doi.org/10.3310/hsdr05230 |