A german-language competency-based multisource feedback instrument for residents: development and validity evidence
Abstract Background In medical settings, multisource feedback (MSF) is a recognised method of formative assessment. It collects feedback on a doctor’s performance from several perspectives in the form of questionnaires. Yet, no validated MSF questionnaire has been publicly available in German. Thus,...
Main Authors: | , , , , , |
---|---|
Format: | Article |
Language: | English |
Published: |
BMC
2020-10-01
|
Series: | BMC Medical Education |
Subjects: | |
Online Access: | http://link.springer.com/article/10.1186/s12909-020-02259-2 |
id |
doaj-c64315cdf48d48dc9f552950cec43985 |
---|---|
record_format |
Article |
spelling |
doaj-c64315cdf48d48dc9f552950cec439852020-11-25T03:42:20ZengBMCBMC Medical Education1472-69202020-10-0120111310.1186/s12909-020-02259-2A german-language competency-based multisource feedback instrument for residents: development and validity evidenceEva K. Hennel0Ulrike Subotic1Christoph Berendonk2Daniel Stricker3Sigrid Harendza4Sören Huwendiek5Department for Assessment and Evaluation (AAE), Institute for Medical Education, University of BernUniversity Children’s Hospital BaselDepartment for Assessment and Evaluation (AAE), Institute for Medical Education, University of BernDepartment for Assessment and Evaluation (AAE), Institute for Medical Education, University of BernDepartment of Internal Medicine, University Medical Centre Hamburg-EppendorfDepartment for Assessment and Evaluation (AAE), Institute for Medical Education, University of BernAbstract Background In medical settings, multisource feedback (MSF) is a recognised method of formative assessment. It collects feedback on a doctor’s performance from several perspectives in the form of questionnaires. Yet, no validated MSF questionnaire has been publicly available in German. Thus, we aimed to develop a German MSF questionnaire based on the CanMEDS roles and to investigate the evidence of its validity. Methods We developed a competency-based MSF questionnaire in German, informed by the literature and expert input. Four sources of validity evidence were investigated: (i) Content was examined based on MSF literature, blueprints of competency, and expert-team discussions. (ii) The response process was supported by analysis of a think-aloud study, narrative comments, “unable to comment” ratings and evaluation data. (iii) The internal structure was assessed by exploratory factor analysis, and inter-rater reliability by generalisability analysis. Data were collected during two runs of MSF, in which 47 residents were evaluated once (first run) or several times (second and third run) on 81 occasions of MSF. (iv) To investigate consequences, we analysed the residents’ learning goals and the progress as reported via MSF. Results Our resulting MSF questionnaire (MSF-RG) consists of 15 items and one global rating, which are each rated on a scale and accompanied by a field for narrative comments and cover a construct of a physician’s competence. Additionally, there are five open questions for further suggestions. Investigation of validity evidence revealed that: (i) The expert group agreed that the content comprehensively addresses clinical competence; (ii) The response processes indicated that the questions are understood as intended and supported the acceptance and usability; (iii) For the second run, factor analysis showed a one-factor solution, a Cronbach’s alpha of 0.951 and an inter-rater reliability of 0.797 with 12 raters; (iv) There are indications that residents benefitted, considering their individual learning goals and based on their ratings reported via MSF itself. Conclusions To support residency training with multisource feedback, we developed a German MSF questionnaire (MSF-RG), which is supported by four sources of validity evidence. This MSF questionnaire may be useful to implement MSF in residency training in German-speaking regions.http://link.springer.com/article/10.1186/s12909-020-02259-2Multisource feedback360-degreeWorkplace-based assessmentAssessmentPostgraduate trainingContinuous professional development |
collection |
DOAJ |
language |
English |
format |
Article |
sources |
DOAJ |
author |
Eva K. Hennel Ulrike Subotic Christoph Berendonk Daniel Stricker Sigrid Harendza Sören Huwendiek |
spellingShingle |
Eva K. Hennel Ulrike Subotic Christoph Berendonk Daniel Stricker Sigrid Harendza Sören Huwendiek A german-language competency-based multisource feedback instrument for residents: development and validity evidence BMC Medical Education Multisource feedback 360-degree Workplace-based assessment Assessment Postgraduate training Continuous professional development |
author_facet |
Eva K. Hennel Ulrike Subotic Christoph Berendonk Daniel Stricker Sigrid Harendza Sören Huwendiek |
author_sort |
Eva K. Hennel |
title |
A german-language competency-based multisource feedback instrument for residents: development and validity evidence |
title_short |
A german-language competency-based multisource feedback instrument for residents: development and validity evidence |
title_full |
A german-language competency-based multisource feedback instrument for residents: development and validity evidence |
title_fullStr |
A german-language competency-based multisource feedback instrument for residents: development and validity evidence |
title_full_unstemmed |
A german-language competency-based multisource feedback instrument for residents: development and validity evidence |
title_sort |
german-language competency-based multisource feedback instrument for residents: development and validity evidence |
publisher |
BMC |
series |
BMC Medical Education |
issn |
1472-6920 |
publishDate |
2020-10-01 |
description |
Abstract Background In medical settings, multisource feedback (MSF) is a recognised method of formative assessment. It collects feedback on a doctor’s performance from several perspectives in the form of questionnaires. Yet, no validated MSF questionnaire has been publicly available in German. Thus, we aimed to develop a German MSF questionnaire based on the CanMEDS roles and to investigate the evidence of its validity. Methods We developed a competency-based MSF questionnaire in German, informed by the literature and expert input. Four sources of validity evidence were investigated: (i) Content was examined based on MSF literature, blueprints of competency, and expert-team discussions. (ii) The response process was supported by analysis of a think-aloud study, narrative comments, “unable to comment” ratings and evaluation data. (iii) The internal structure was assessed by exploratory factor analysis, and inter-rater reliability by generalisability analysis. Data were collected during two runs of MSF, in which 47 residents were evaluated once (first run) or several times (second and third run) on 81 occasions of MSF. (iv) To investigate consequences, we analysed the residents’ learning goals and the progress as reported via MSF. Results Our resulting MSF questionnaire (MSF-RG) consists of 15 items and one global rating, which are each rated on a scale and accompanied by a field for narrative comments and cover a construct of a physician’s competence. Additionally, there are five open questions for further suggestions. Investigation of validity evidence revealed that: (i) The expert group agreed that the content comprehensively addresses clinical competence; (ii) The response processes indicated that the questions are understood as intended and supported the acceptance and usability; (iii) For the second run, factor analysis showed a one-factor solution, a Cronbach’s alpha of 0.951 and an inter-rater reliability of 0.797 with 12 raters; (iv) There are indications that residents benefitted, considering their individual learning goals and based on their ratings reported via MSF itself. Conclusions To support residency training with multisource feedback, we developed a German MSF questionnaire (MSF-RG), which is supported by four sources of validity evidence. This MSF questionnaire may be useful to implement MSF in residency training in German-speaking regions. |
topic |
Multisource feedback 360-degree Workplace-based assessment Assessment Postgraduate training Continuous professional development |
url |
http://link.springer.com/article/10.1186/s12909-020-02259-2 |
work_keys_str_mv |
AT evakhennel agermanlanguagecompetencybasedmultisourcefeedbackinstrumentforresidentsdevelopmentandvalidityevidence AT ulrikesubotic agermanlanguagecompetencybasedmultisourcefeedbackinstrumentforresidentsdevelopmentandvalidityevidence AT christophberendonk agermanlanguagecompetencybasedmultisourcefeedbackinstrumentforresidentsdevelopmentandvalidityevidence AT danielstricker agermanlanguagecompetencybasedmultisourcefeedbackinstrumentforresidentsdevelopmentandvalidityevidence AT sigridharendza agermanlanguagecompetencybasedmultisourcefeedbackinstrumentforresidentsdevelopmentandvalidityevidence AT sorenhuwendiek agermanlanguagecompetencybasedmultisourcefeedbackinstrumentforresidentsdevelopmentandvalidityevidence AT evakhennel germanlanguagecompetencybasedmultisourcefeedbackinstrumentforresidentsdevelopmentandvalidityevidence AT ulrikesubotic germanlanguagecompetencybasedmultisourcefeedbackinstrumentforresidentsdevelopmentandvalidityevidence AT christophberendonk germanlanguagecompetencybasedmultisourcefeedbackinstrumentforresidentsdevelopmentandvalidityevidence AT danielstricker germanlanguagecompetencybasedmultisourcefeedbackinstrumentforresidentsdevelopmentandvalidityevidence AT sigridharendza germanlanguagecompetencybasedmultisourcefeedbackinstrumentforresidentsdevelopmentandvalidityevidence AT sorenhuwendiek germanlanguagecompetencybasedmultisourcefeedbackinstrumentforresidentsdevelopmentandvalidityevidence |
_version_ |
1724525774621376512 |