Factors influencing the generalizability on Mathematics Performance Assessment

• Main Author: 林敬修
• Other Authors: 張麗麗
• Format: Others
• Language: zh-TW
• Published: 2003
• Online Access: http://ndltd.ncl.edu.tw/handle/kvs6pv

碩士 === 國立屏東師範學院 === 教育心理與輔導學系碩士班 === 91 === Factors influencing the generalizability on Mathematics Performance Assessment Abstract The main purpose of this research was to study the impact of rater’s mathematics background, rater training, task structure on reliability of mathematics performance assessment. A quasi-experimental study was implemented. Twelve raters, in eight elementary schools, participated in the study. According to rater’s mathematics background, twelve raters were separated into two groups (those with and those without mathematics background) with six raters in each group. And the raters in each group were then assigned to three different types of rater training (analytic scoring rubrics only, analytic scoring rubrics plus anchors, and a thorough training session with in-depth discussion and practice). After receiving the rater training, each rater scored students’ problem-solving performance on four mathematics items. Spearman rank correlation and generalizability theory were used in studying the reliability of the data. The major findings of the study were as follows: 1.In general, the largest variation of mathematics performance assessment scores was due to person and task interaction, the second was due to person, and the third was due to task. 2.Although rater variation accounted for only a small proportion of total variance components, rater consistency varied across raters with different mathematics backgrounds and rater training. 3.Task structure had an impact on variance components. Variance components were relatively smaller across items with similar task structure than those across items with different task structures. In addition, task structure interacted with rater mathematics background and rater training on rater consistency. For raters with no mathematics background and received scoring rubrics only/scoring rubrics plus anchors, rater consistency was low on less structured items, but for raters with mathematics background (no matter which rater training was received) and raters with no mathematics background but received an in-depth training session, the rater consistency was high regardless of the task structure. 4.Rater consistency on “communication” dimension was the lowest among the three scoring dimensions (i.e., understanding, strategies/procedural, communication). Based on the above findings, suggestions regarding rater training, task structure, model selection, and the research issues for further studies were also provided. Key words: performance assessment, mathematics performance assessment, reliability, generalizability theory.