Monday, 7 December 2015

Questioning Optimism

I'm Adam Harris and I'm an experimental psychologist from University College London.

I am perhaps an unusual contributor to the Imperfect Cognitions blog as I have argued that cognitions might seem imperfect because of imperfections in prevalent methodologies, predominantly arising from a failure to appreciate the importance of understanding the appropriate normative basis of a task. Specifically, my work has suggested that the assumed ubiquity of optimism across our species is based on questionable evidence.

A prominent example of this work is presented in a paper I wrote with Ulrike Hahn (published in the Psychological Review) in which we demonstrated, through simulation, that rational agents could be labelled as optimistic on the prevalent, comparative method of testing unrealistic optimism. On this method, participants respond to the question "Compared with the average student of your age and sex, how likely are you to..." where future life events are inserted for the ellipsis.

Responses are typically provided on a -3 (much less likely than the average) to +3 (much more likely) scale, where a response of zero represents 'about the same as average'. The logic of the test rests upon the recognition that if participants are accurately reporting their chances, then the average of their responses should be the average. Consequently, any deviation from zero is taken as indicative of a systematic bias. The oft-observed result that average responses on this scale to negative events are significantly negative is taken as evidence that, on average, the members of the group underestimate their relative chances. Because we do not wish to experience negative events, such a result is taken as evidence of optimism.

In Harris and Hahn (2011), we demonstrated that 3 statistical artifacts could generate the oft-observed pattern of results from rational Bayesian agents who are, by definition, unbiased. Such a result raises questions over the interpretation of the same pattern of results observed in human participants. There is no longer any evidence for bias on these tests if the pattern of results are consistent with those of rational agents. Essentially, these tests fail the major pre-requisite for a satisfactory test of bias: Unbiased agents appear biased!  In ongoing (as yet unpublished) research, we have failed to identify any evidence for optimism after controlling for these confounding artifacts (my website will be updated when these results are published).

In addition to specifically raising concerns over the understanding of comparative unrealistic optimism, this work highlights, generally, the importance of understanding what participants’ responses represent and the appropriate normative standard for those responses. In unrealistic optimism research, participants’ responses represent their understanding of their own risk and the average person’s risk. Normatively, their own risk includes their estimate of the base rate as well as any individuating information they possess. This insight is a critical consideration when evaluating conclusions from any measure designed to assess bias in risk estimates about real-world events (see also, Harris et al., 2013)

Fortunately, there is also an initial, easy, check for an optimism bias that cannot be accounted for on statistical grounds. The implications of any statistical account in terms of researching optimism are opposite for events of opposite valence. If, for example, a statistical account predicts lower responses for negative events, it will also predict lower responses for comparable positive events. Because of the reversed desirability of positive and negative events, however, the same direction of effect that constitutes optimism for one valence would constitute pessimism for the opposing valence.

Thus, the inclusion of both positive and negative events can serve as a first-stage litmus test to identify a possible confounding influence of statistical artifacts. I therefore recommend that researchers routinely include both positive and negative events in their tests of optimism. In my own work, all such tests to date have observed the same direction of effect in each valence. This constitutes seeming optimism in one valence and pessimism in the other, thus failing to provide the conclusive evidence required for optimism.

No comments:

Post a Comment

Comments are moderated.