A Bayesian hierarchical latent trait model for estimating rater bias and reliability in large-scale performance assessment.

A Bayesian hierarchical latent trait model for estimating rater bias and reliability in large-scale performance assessment.

We propose a novel approach to modelling rater effects in scoring-based assessment. The approach is based on a Bayesian hierarchical model and simulations from the posterior distribution. We apply it to large-scale essay assessment data over a period of 5 years. Empirical results suggest that the mo...

Full description

Bibliographic Details
Main Authors: Kaja Zupanc, Erik Štrumbelj
Format: Article
Language:English
Published: Public Library of Science (PLoS) 2018-01-01
Series:PLoS ONE
Online Access:http://europepmc.org/articles/PMC5882162?pdf=render
id doaj-63a3882e2dea4652aa6a11aeb5281760
record_format Article
spelling doaj-63a3882e2dea4652aa6a11aeb52817602020-11-25T02:47:06ZengPublic Library of Science (PLoS)PLoS ONE1932-62032018-01-01134e019529710.1371/journal.pone.0195297A Bayesian hierarchical latent trait model for estimating rater bias and reliability in large-scale performance assessment.Kaja ZupancErik ŠtrumbeljWe propose a novel approach to modelling rater effects in scoring-based assessment. The approach is based on a Bayesian hierarchical model and simulations from the posterior distribution. We apply it to large-scale essay assessment data over a period of 5 years. Empirical results suggest that the model provides a good fit for both the total scores and when applied to individual rubrics. We estimate the median impact of rater effects on the final grade to be ± 2 points on a 50 point scale, while 10% of essays would receive a score at least ± 5 different from their actual quality. Most of the impact is due to rater unreliability, not rater bias.http://europepmc.org/articles/PMC5882162?pdf=render
collection DOAJ
language English
format Article
sources DOAJ
author Kaja Zupanc
Erik Štrumbelj
spellingShingle Kaja Zupanc
Erik Štrumbelj
A Bayesian hierarchical latent trait model for estimating rater bias and reliability in large-scale performance assessment.
PLoS ONE
author_facet Kaja Zupanc
Erik Štrumbelj
author_sort Kaja Zupanc
title A Bayesian hierarchical latent trait model for estimating rater bias and reliability in large-scale performance assessment.
title_short A Bayesian hierarchical latent trait model for estimating rater bias and reliability in large-scale performance assessment.
title_full A Bayesian hierarchical latent trait model for estimating rater bias and reliability in large-scale performance assessment.
title_fullStr A Bayesian hierarchical latent trait model for estimating rater bias and reliability in large-scale performance assessment.
title_full_unstemmed A Bayesian hierarchical latent trait model for estimating rater bias and reliability in large-scale performance assessment.
title_sort bayesian hierarchical latent trait model for estimating rater bias and reliability in large-scale performance assessment.
publisher Public Library of Science (PLoS)
series PLoS ONE
issn 1932-6203
publishDate 2018-01-01
description We propose a novel approach to modelling rater effects in scoring-based assessment. The approach is based on a Bayesian hierarchical model and simulations from the posterior distribution. We apply it to large-scale essay assessment data over a period of 5 years. Empirical results suggest that the model provides a good fit for both the total scores and when applied to individual rubrics. We estimate the median impact of rater effects on the final grade to be ± 2 points on a 50 point scale, while 10% of essays would receive a score at least ± 5 different from their actual quality. Most of the impact is due to rater unreliability, not rater bias.
url http://europepmc.org/articles/PMC5882162?pdf=render
work_keys_str_mv AT kajazupanc abayesianhierarchicallatenttraitmodelforestimatingraterbiasandreliabilityinlargescaleperformanceassessment
AT erikstrumbelj abayesianhierarchicallatenttraitmodelforestimatingraterbiasandreliabilityinlargescaleperformanceassessment
AT kajazupanc bayesianhierarchicallatenttraitmodelforestimatingraterbiasandreliabilityinlargescaleperformanceassessment
AT erikstrumbelj bayesianhierarchicallatenttraitmodelforestimatingraterbiasandreliabilityinlargescaleperformanceassessment
_version_ 1724754499254353920

Similar Items