Cognitive Diagnostic Models for Rater Effects
In recent decades, cognitive diagnostic models (CDMs) have been intensively researched and applied to various educational and psychological tests. However, because existing CDMs fail to consider rater effects, the application of CDMs to constructed-response (CR) items that involve human raters is se...
Main Authors: | , , |
---|---|
Format: | Article |
Language: | English |
Published: |
Frontiers Media S.A.
2020-03-01
|
Series: | Frontiers in Psychology |
Subjects: | |
Online Access: | https://www.frontiersin.org/article/10.3389/fpsyg.2020.00525/full |
id |
doaj-a8c62f3c8cea4435892cbd01e84466ca |
---|---|
record_format |
Article |
spelling |
doaj-a8c62f3c8cea4435892cbd01e84466ca2020-11-25T02:17:49ZengFrontiers Media S.A.Frontiers in Psychology1664-10782020-03-011110.3389/fpsyg.2020.00525503702Cognitive Diagnostic Models for Rater EffectsXiaomin Li0Wen-Chung Wang1Qin Xie2Centre for Child and Family Science, The Education University of Hong Kong, Tai Po, Hong KongAssessment Research Centre, The Education University of Hong Kong, Tai Po, Hong KongDepartment of Linguistics and Modern Language Studies, The Education University of Hong Kong, Tai Po, Hong KongIn recent decades, cognitive diagnostic models (CDMs) have been intensively researched and applied to various educational and psychological tests. However, because existing CDMs fail to consider rater effects, the application of CDMs to constructed-response (CR) items that involve human raters is seriously limited. Given the popularity of CR items, it is desirable to develop new CDMs that are capable of describing and estimating rater effects on CR items. In this study, we developed such new CDMs within the frameworks of facets models and hierarchical rater models, using the log-linear cognitive diagnosis model as a template. The parameters of the new models were estimated with the Markov chain Monte Carlo methods implemented in the freeware JAGS. Simulations were conducted to evaluate the parameter recovery of the new models. Results showed that the parameters were recovered fairly well and the more data there were, the better the recovery. Implications and applications of the new models were illustrated with an empirical study that adopted a fine-grained checklist to assess English academic essays.https://www.frontiersin.org/article/10.3389/fpsyg.2020.00525/fullcognitive diagnostic modelsfacets modelshierarchical rater modelsrater effectitem response theory |
collection |
DOAJ |
language |
English |
format |
Article |
sources |
DOAJ |
author |
Xiaomin Li Wen-Chung Wang Qin Xie |
spellingShingle |
Xiaomin Li Wen-Chung Wang Qin Xie Cognitive Diagnostic Models for Rater Effects Frontiers in Psychology cognitive diagnostic models facets models hierarchical rater models rater effect item response theory |
author_facet |
Xiaomin Li Wen-Chung Wang Qin Xie |
author_sort |
Xiaomin Li |
title |
Cognitive Diagnostic Models for Rater Effects |
title_short |
Cognitive Diagnostic Models for Rater Effects |
title_full |
Cognitive Diagnostic Models for Rater Effects |
title_fullStr |
Cognitive Diagnostic Models for Rater Effects |
title_full_unstemmed |
Cognitive Diagnostic Models for Rater Effects |
title_sort |
cognitive diagnostic models for rater effects |
publisher |
Frontiers Media S.A. |
series |
Frontiers in Psychology |
issn |
1664-1078 |
publishDate |
2020-03-01 |
description |
In recent decades, cognitive diagnostic models (CDMs) have been intensively researched and applied to various educational and psychological tests. However, because existing CDMs fail to consider rater effects, the application of CDMs to constructed-response (CR) items that involve human raters is seriously limited. Given the popularity of CR items, it is desirable to develop new CDMs that are capable of describing and estimating rater effects on CR items. In this study, we developed such new CDMs within the frameworks of facets models and hierarchical rater models, using the log-linear cognitive diagnosis model as a template. The parameters of the new models were estimated with the Markov chain Monte Carlo methods implemented in the freeware JAGS. Simulations were conducted to evaluate the parameter recovery of the new models. Results showed that the parameters were recovered fairly well and the more data there were, the better the recovery. Implications and applications of the new models were illustrated with an empirical study that adopted a fine-grained checklist to assess English academic essays. |
topic |
cognitive diagnostic models facets models hierarchical rater models rater effect item response theory |
url |
https://www.frontiersin.org/article/10.3389/fpsyg.2020.00525/full |
work_keys_str_mv |
AT xiaominli cognitivediagnosticmodelsforratereffects AT wenchungwang cognitivediagnosticmodelsforratereffects AT qinxie cognitivediagnosticmodelsforratereffects |
_version_ |
1724884904306540544 |