| Summary: | We demonstrate the
usefulness of cognitive models for combining human estimates of probabilities
in two experiments. The first experiment involves people's estimates of
probabilities for general knowledge questions such as ``What percentage of the
world's population speaks English as a first language?'' The second experiment
involves people's estimates of probabilities in football (soccer) games, such
as ``What is the probability a team leading 1--0 at half time will win the
game?'', with ground truths based on analysis of large corpus of games played
in the past decade. In both experiments, we collect people's probability
estimates, and develop a cognitive model of the estimation process, including
assumptions about the calibration of probabilities and individual differences.
We show that the cognitive model approach outperforms standard statistical
aggregation methods like the mean and the median for both experiments and,
unlike most previous related work, is able to make good predictions in a fully
unsupervised setting. We also show that the parameters inferred as part of the
cognitive modeling, involving calibration and expertise, provide useful
measures of the cognitive characteristics of individuals. We argue that the
cognitive approach has the advantage of aggregating over latent human knowledge
rather than observed estimates, and emphasize that it can be applied in
predictive settings where answers are not yet available.
|
|---|
