Using simulation to assess prediction performance change with simulated annealing on probability arrays
Thompson, James R.
Master of Arts
Experimental results suggest that significant improvements in forecast performance can be obtained by applying the simulated annealing on probability arrays (SAPA) algorithm to grouped event probability forecasts. Such forecasts are frequently probabilistically incoherent, even when elicited expert subjects. The algorithm corrects any incoherence within the set of responses from each subject, while at the same time minimizing the sum of the absolute adjustments made to the original probability estimates. These adjusted coherent probability estimates appear to yield improved overall forecast performance, as measured by several different metrics. However, with the only published results consisting of several small experiments, definitive conclusions regarding potential forecast improvements in wider applications are difficult to justify. To address this lack of experimental data, a method for extending the existing published results using simulation is described, and the SAPA algorithm and its effects on forecast performance are examined.
Statistics; Computer science