Accuracy and reliability of job evaluation: the roles of amount of information and ability to handle complex information

Abstract

The amount of information a rater is given must in some way influence the quality of their ratings. With respect to job evaluation, quality may be defined in terms of how well a rater discriminates among a set of jobs with provided dimensions and, to some extent, the amount of agreement with other raters. Previous research has manipulated amount of information as a job title only versus a job description (Hahn & Dipboye, 1988; Harvey & Lozada-Larsen, 1988). Thus, the nature of the relation between amount of information and rating quality is unclear. The present experiment has two goals. The first is to evaluate the nature of the relation between the amount of information and the accuracy and reliability of job evaluation ratings. The second is to examine whether individual variations in raters' abilities to handle complex information explain the effect of amount of information on the accuracy of job evaluation ratings. Both the reliability and the accuracy of job evaluation ratings were affected by the amount of information given to raters. In particular, the relation was non-linear in that the most accurate and reliable raters were not the ones who received the most information. Moreover, the relation between the amount of information and rating accuracy could largely be accounted for by raters' abilities to handle complex information. The results of the experiment have implications for the manner in which job evaluations are conducted in organizations. Contrary to popular belief, raters may not need large amounts of information in order to make accurate and reliable evaluations of jobs. In fact, asking raters to base their ratings on large amounts of information can cause a decrease in rating quality.