The use of test scores from large-scale assessment surveys: psychometric and statistical considerations
Abstract Background Economists are making increasing use of measures of student achievement obtained through large-scale survey assessments such as NAEP, TIMSS, and PISA. The construction of these measures, employing plausible value (PV) methodology, is quite different from that of the more familiar...
Main Authors: | , |
---|---|
Format: | Article |
Language: | English |
Published: |
SpringerOpen
2017-11-01
|
Series: | Large-scale Assessments in Education |
Subjects: | |
Online Access: | http://link.springer.com/article/10.1186/s40536-017-0050-x |
id |
doaj-99e1e9ce379f4cf9aa27b4b04b0f2b39 |
---|---|
record_format |
Article |
spelling |
doaj-99e1e9ce379f4cf9aa27b4b04b0f2b392020-11-24T21:15:36ZengSpringerOpenLarge-scale Assessments in Education2196-07392017-11-015111610.1186/s40536-017-0050-xThe use of test scores from large-scale assessment surveys: psychometric and statistical considerationsHenry Braun0Matthias von Davier1Lynch School of Education, Campion Hall, Boston CollegeNational Board of Medical ExaminersAbstract Background Economists are making increasing use of measures of student achievement obtained through large-scale survey assessments such as NAEP, TIMSS, and PISA. The construction of these measures, employing plausible value (PV) methodology, is quite different from that of the more familiar test scores associated with assessments such as the SAT or ACT. These differences have important implications both for utilization and interpretation. Although much has been written about PVs, it appears that there are still misconceptions about whether and how to employ them in secondary analyses. Methods We address a range of technical issues, including those raised in a recent article that was written to inform economists using these databases. First, an extensive review of the relevant literature was conducted, with particular attention to key publications that describe the derivation and psychometric characteristics of such achievement measures. Second, a simulation study was carried out to compare the statistical properties of estimates based on the use of PVs with those based on other, commonly used methods. Results It is shown, through both theoretical analysis and simulation, that under fairly general conditions appropriate use of PV yields approximately unbiased estimates of model parameters in regression analyses of large scale survey data. The superiority of the PV methodology is particularly evident when measures of student achievement are employed as explanatory variables. Conclusions The PV methodology used to report student test performance in large scale surveys remains the state-of-the-art for secondary analyses of these databases.http://link.springer.com/article/10.1186/s40536-017-0050-xLarge-scale assessmentImputationPlausible valuesConditioning modelIRTUnbiasedness |
collection |
DOAJ |
language |
English |
format |
Article |
sources |
DOAJ |
author |
Henry Braun Matthias von Davier |
spellingShingle |
Henry Braun Matthias von Davier The use of test scores from large-scale assessment surveys: psychometric and statistical considerations Large-scale Assessments in Education Large-scale assessment Imputation Plausible values Conditioning model IRT Unbiasedness |
author_facet |
Henry Braun Matthias von Davier |
author_sort |
Henry Braun |
title |
The use of test scores from large-scale assessment surveys: psychometric and statistical considerations |
title_short |
The use of test scores from large-scale assessment surveys: psychometric and statistical considerations |
title_full |
The use of test scores from large-scale assessment surveys: psychometric and statistical considerations |
title_fullStr |
The use of test scores from large-scale assessment surveys: psychometric and statistical considerations |
title_full_unstemmed |
The use of test scores from large-scale assessment surveys: psychometric and statistical considerations |
title_sort |
use of test scores from large-scale assessment surveys: psychometric and statistical considerations |
publisher |
SpringerOpen |
series |
Large-scale Assessments in Education |
issn |
2196-0739 |
publishDate |
2017-11-01 |
description |
Abstract Background Economists are making increasing use of measures of student achievement obtained through large-scale survey assessments such as NAEP, TIMSS, and PISA. The construction of these measures, employing plausible value (PV) methodology, is quite different from that of the more familiar test scores associated with assessments such as the SAT or ACT. These differences have important implications both for utilization and interpretation. Although much has been written about PVs, it appears that there are still misconceptions about whether and how to employ them in secondary analyses. Methods We address a range of technical issues, including those raised in a recent article that was written to inform economists using these databases. First, an extensive review of the relevant literature was conducted, with particular attention to key publications that describe the derivation and psychometric characteristics of such achievement measures. Second, a simulation study was carried out to compare the statistical properties of estimates based on the use of PVs with those based on other, commonly used methods. Results It is shown, through both theoretical analysis and simulation, that under fairly general conditions appropriate use of PV yields approximately unbiased estimates of model parameters in regression analyses of large scale survey data. The superiority of the PV methodology is particularly evident when measures of student achievement are employed as explanatory variables. Conclusions The PV methodology used to report student test performance in large scale surveys remains the state-of-the-art for secondary analyses of these databases. |
topic |
Large-scale assessment Imputation Plausible values Conditioning model IRT Unbiasedness |
url |
http://link.springer.com/article/10.1186/s40536-017-0050-x |
work_keys_str_mv |
AT henrybraun theuseoftestscoresfromlargescaleassessmentsurveyspsychometricandstatisticalconsiderations AT matthiasvondavier theuseoftestscoresfromlargescaleassessmentsurveyspsychometricandstatisticalconsiderations AT henrybraun useoftestscoresfromlargescaleassessmentsurveyspsychometricandstatisticalconsiderations AT matthiasvondavier useoftestscoresfromlargescaleassessmentsurveyspsychometricandstatisticalconsiderations |
_version_ |
1716744656989978624 |