| Summary: | 碩士 === 國立臺灣大學 === 數學研究所 === 97 === We investigate the inter-rater reliability when the ability of large number of examinees is classified to ordinal grade by raters through derangement. The polychoric correlation coefficient is used as inter-rater reliability when the latent trait model (LTM) is assumed.
To ensure at least hundreds examinees is graded by two raters when the number of raters is around a few hundred and the number of examinees is around three hundred thousand, we consider assigning examinees to raters through derangement. Under this setting, it is found that all raters are grouped into several cycles. Through analytic argument and simulation, it is found that the number of group is often not more than ten, the probability of getting at least one cycle of size 2 or 3 is close to 0.59, and the size of largest cycle is often exceeding one hundred. It also finds that the distributions of latent trait of examinees by different raters are close to each other up to a location shift.
Under the assumption of the LTM, the discriminate parameter in models can be regard as the accuracy of rating.The correlation between the grades given by raters and the latent trait of examinees was affected by the interaction of the thresholds and discriminate parameter. The correlation coefficient of perspective latent trait variables of two raters is the product of their discriminate parameter, and polychoric correlation coefficient can be estimated by two stages method. The parameter of the thresholds of raters were estimated by the proportion of rating, while as discriminate parameter can be estimates through appropriate derangement.
Finally according to the result of research, we propose the summary and some suggestions.
|
|---|
