Reliability analysis of the objective structured clinical examination using generalizability theory
Background: The objective structured clinical examination (OSCE) is a widely used method for assessing clinical competence in health sciences education. Studies using this method have shown evidence of validity and reliability. There are no published studies of OSCE reliability measurement with gene...
Main Authors: | , , , |
---|---|
Format: | Article |
Language: | English |
Published: |
Taylor & Francis Group
2016-08-01
|
Series: | Medical Education Online |
Subjects: | |
Online Access: | http://med-ed-online.net/index.php/meo/article/view/31650/48304 |
id |
doaj-b3dbbf123563406a82cb487bb01405bc |
---|---|
record_format |
Article |
spelling |
doaj-b3dbbf123563406a82cb487bb01405bc2020-11-25T01:49:56ZengTaylor & Francis GroupMedical Education Online1087-29812016-08-012101710.3402/meo.v21.3165031650Reliability analysis of the objective structured clinical examination using generalizability theoryJuan Andrés Trejo-Mejía0Melchor Sánchez-Mendiola1Ignacio Méndez-Ramírez2Adrián Martínez-González3 Secretariat of Medical Education, UNAM Faculty of Medicine, México City, México Secretariat of Medical Education, UNAM Faculty of Medicine, México City, México Institute of Applied Mathematics and Systems, UNAM, México City, México Secretariat of Medical Education, UNAM Faculty of Medicine, México City, MéxicoBackground: The objective structured clinical examination (OSCE) is a widely used method for assessing clinical competence in health sciences education. Studies using this method have shown evidence of validity and reliability. There are no published studies of OSCE reliability measurement with generalizability theory (G-theory) in Latin America. The aims of this study were to assess the reliability of an OSCE in medical students using G-theory and explore its usefulness for quality improvement. Methods: An observational cross-sectional study was conducted at National Autonomous University of Mexico (UNAM) Faculty of Medicine in Mexico City. A total of 278 fifth-year medical students were assessed with an 18-station OSCE in a summative end-of-career final examination. There were four exam versions. G-theory with a crossover random effects design was used to identify the main sources of variance. Examiners, standardized patients, and cases were considered as a single facet of analysis. Results: The exam was applied to 278 medical students. The OSCE had a generalizability coefficient of 0.93. The major components of variance were stations, students, and residual error. The sites and the versions of the tests had minimum variance. Conclusions: Our study achieved a G coefficient similar to that found in other reports, which is acceptable for summative tests. G-theory allows the estimation of the magnitude of multiple sources of error and helps decision makers to determine the number of stations, test versions, and examiners needed to obtain reliable measurements.http://med-ed-online.net/index.php/meo/article/view/31650/48304OSCEreliabilitygeneralizability theoryclinical competenceMexico |
collection |
DOAJ |
language |
English |
format |
Article |
sources |
DOAJ |
author |
Juan Andrés Trejo-Mejía Melchor Sánchez-Mendiola Ignacio Méndez-Ramírez Adrián Martínez-González |
spellingShingle |
Juan Andrés Trejo-Mejía Melchor Sánchez-Mendiola Ignacio Méndez-Ramírez Adrián Martínez-González Reliability analysis of the objective structured clinical examination using generalizability theory Medical Education Online OSCE reliability generalizability theory clinical competence Mexico |
author_facet |
Juan Andrés Trejo-Mejía Melchor Sánchez-Mendiola Ignacio Méndez-Ramírez Adrián Martínez-González |
author_sort |
Juan Andrés Trejo-Mejía |
title |
Reliability analysis of the objective structured clinical examination using generalizability theory |
title_short |
Reliability analysis of the objective structured clinical examination using generalizability theory |
title_full |
Reliability analysis of the objective structured clinical examination using generalizability theory |
title_fullStr |
Reliability analysis of the objective structured clinical examination using generalizability theory |
title_full_unstemmed |
Reliability analysis of the objective structured clinical examination using generalizability theory |
title_sort |
reliability analysis of the objective structured clinical examination using generalizability theory |
publisher |
Taylor & Francis Group |
series |
Medical Education Online |
issn |
1087-2981 |
publishDate |
2016-08-01 |
description |
Background: The objective structured clinical examination (OSCE) is a widely used method for assessing clinical competence in health sciences education. Studies using this method have shown evidence of validity and reliability. There are no published studies of OSCE reliability measurement with generalizability theory (G-theory) in Latin America. The aims of this study were to assess the reliability of an OSCE in medical students using G-theory and explore its usefulness for quality improvement. Methods: An observational cross-sectional study was conducted at National Autonomous University of Mexico (UNAM) Faculty of Medicine in Mexico City. A total of 278 fifth-year medical students were assessed with an 18-station OSCE in a summative end-of-career final examination. There were four exam versions. G-theory with a crossover random effects design was used to identify the main sources of variance. Examiners, standardized patients, and cases were considered as a single facet of analysis. Results: The exam was applied to 278 medical students. The OSCE had a generalizability coefficient of 0.93. The major components of variance were stations, students, and residual error. The sites and the versions of the tests had minimum variance. Conclusions: Our study achieved a G coefficient similar to that found in other reports, which is acceptable for summative tests. G-theory allows the estimation of the magnitude of multiple sources of error and helps decision makers to determine the number of stations, test versions, and examiners needed to obtain reliable measurements. |
topic |
OSCE reliability generalizability theory clinical competence Mexico |
url |
http://med-ed-online.net/index.php/meo/article/view/31650/48304 |
work_keys_str_mv |
AT juanandrestrejomejia reliabilityanalysisoftheobjectivestructuredclinicalexaminationusinggeneralizabilitytheory AT melchorsanchezmendiola reliabilityanalysisoftheobjectivestructuredclinicalexaminationusinggeneralizabilitytheory AT ignaciomendezramirez reliabilityanalysisoftheobjectivestructuredclinicalexaminationusinggeneralizabilitytheory AT adrianmartinezgonzalez reliabilityanalysisoftheobjectivestructuredclinicalexaminationusinggeneralizabilitytheory |
_version_ |
1725003978904698880 |