Aspen Global Change Institute Elements of Change 1996

AGCI Session II: Characterizing and Communicating Scientific Uncertainty

Session Chairs: Dr. Richard H. Moss and Dr. Stephen H. Schneider

July 31 to August 8, 1996

Quantitative Expert Subjective Judgment: Does It Have a Role in Future IPCC Assessments?

Granger Morgan

Carnegie Mellon University

Pittsburgh, Pennsylvania

There are two traditional approaches to uncertainty: ignore it or describe it qualitatively using words such as "unlikely" or "doubtful." In the case of a problem such as climate change, ignoring uncertainty is obviously not acceptable. Thus, in the past, many treatments have used qualitative descriptions. However, Morgan noted that the experimental social science literature indicates that qualitative descriptions of uncertainty can mean very different things to different people. For example, when a 1986 study by Wallstein, Budescu and Rapoport asked subjects for a subjective judgment of the chance that an event termed "unlikely" would in fact occur, responses ranged from 0 to almost 40 percent!

To avoid this problem, one can adopt a Bayesian perspective, using quantitative statements of probability to convey one's degree of belief. In at least limited ways, scientists have long engaged in such practices (e. g., "Our best estimate of the speed of light including all sources of error is 299,775 km/sec ± 5 km/sec.") In order to illustrate how expert judgments about uncertainty can be formalized, Morgan asked one participant to consider how long it would take him to drive from his office to the airport at a given time on a given morning. They began by refining and clarifying the question. It was quickly shown that developing a well-posed question requires a lot of clarification in order to be sure of getting the answer to the question one thinks was asked. Once that is accomplished, standard methods can be used to elicit the subjective probability distribution, typically starting with questions designed to identify the extremes of the distribution.

Qualitative descriptions of uncertainty can mean very different things to different people.

This basic idea of expert subjective judgment has been a main-stay in the field of decision analysis for many years. By way of background, before turning to the issue of applying such judgments in the IPCC process, Morgan discussed:

· the role of cognitive heuristics which people use in making judgments under uncertainty and the biases to which they can lead; and

· previous experience with the use of expert subjective judgments in related fields.

He went on to argue that in designing a procedure for use in the IPCC, it was important to focus on eliciting the reasons for the results (the whys are important, not just the numbers) and to pay careful attention to how to display the results and under what circumstances it does and does not make sense to combine judgments from different experts.

Morgan explained that people use a variety of heuristic procedures, or subconscious rules of thumb, when making judgments that involve uncertainty. One is availability, which results in probability judgments being driven by the ease with which people can think of previous occurrences of the event or can imagine such occurrences. Another is anchoring and adjustment, which results in probability judgments being overly influenced by the starting point which becomes an "anchor." A third is representativeness, which results in people judging the likelihood that an object belongs to a particular class in terms of how much it resembles that class. Often these heuristics serve us well, but in some situations they can lead to significant bias in judgments.

One consequence of the operation of these heuristics, particularly that of anchoring and adjustment, is a strong tendency to overconfidence. Experimental studies reveal that over a wide range of circumstances respondents are systematically overconfident. There is a very large literature on calibration which shows consistent overconfidence. Most of this literature is related to judgments by laypeople. Scientists may be different from laypeople in some respects, for example, in the greater amount of substantive knowledge they have to fall back on. But even among expert physical scientists, there is strong experimental evidence of frequent overconfidence, Morgan reported. Calibration does depend upon how hard the questions are. Interestingly, the harder the question, the greater the overconfidence. But calibration can get quite good with feedback and practice (as in the case of precipitation forecasts by weather forecasters). Calibration also improves when respondents are asked to give reasons for their answers and address counter-factual examples.

Over a wide range of circumstances respondents are systematically overconfident. Interestingly, the harder the question, the greater the overconfidence.

Morgan briefly described a number of examples in which subjective expert judgments about uncertainty have been used in addressing a variety of environmental problems. These included the Rasmussen, et al., 1975 study of nuclear reactor safety, the 1979 National Academy of Sciences (NAS) study of ozone depletion and CFCs, his own 1984 study of sulfur long-range transport and resulting health impacts, and the NAS 1986 study of radon from uranium mill tailing piles. He then focused on work he and David Keith performed in 1995, which involved detailed interviews with 16 U. S. climate scientists.

In the Morgan and Keith research, the interviews were far more technically detailed, and used a much more complex interview protocol than any previous expert elicitation on any topic, Morgan says. Morgan and Keith's interview protocol involved six parts:

1. introductions and an explanation of what the researchers were trying to accomplish;

2. general discussion in which the experts were asked to critique the researcher's background paper and discuss how they think about the problem;

3. judgments on a small number of policy-relevant global variables elicited in the form of full subjective probability distributions; judgments elicited about how well knowledge of global-scale climate change allows predictions on smaller scales;

4. disaggregated sources of uncertainty in global average temperature change dealt with by asking experts to systematically identify key contributors to overall uncertainty, and the extent to which the three global-scale variables were separable;

5. experts asked to discuss factors that should be considered in designing a national R&D program and then make a series of judgments about how resources should be allocated in such a program;

6. experts asked to talk about how research programs can lead to surprising results and then asked to make judgments about the likely state of their knowledge after 20 years of the research program they designed in part 5.

In the Morgan and Keith research, the interviews were far more technically detailed, and used a much more complex interview protocol than any previous expert elicitation on any topic, Morgan says.

Morgan displayed and discussed some illustrative results of the study, including the projected climate sensitivity, pole to equator temperature gradient with doubled CO2 forcing, zonally averaged precipitation, and radiative forcing due to anthropogenic aerosols. He also discussed a task in which the experts were asked to sort a set of cards on which possible causes of uncertainty were identified. Each expert thus ranked which uncertainty terms contributed the most uncertainty to his answers. The top five uncertainty terms identified were, in order, cloud optical properties, convection/water vapor feedback, CO2 exchange with terrestrial biota, CO2 exchange with the oceans (including ocean biota), and ocean convection.

Morgan then discussed a similar elicitation procedure he is developing to be administered to a group of terrestrial ecologists. They will be asked a series of questions designed to assess:

1. What factors and processes are most important in determining the responses of forests and grasslands to climate change?

2. What will be the likely change in standing biomass in tropical and northern forests and the extent of grasslands, under fairly gradual and modest climate change?

3. How rapidly can trees and grasses in the Northern Hemisphere migrate in the face of climate change?

4. How might the southern boundary for the region of Arctic tundra move under fairly gradual and modest climate change, given that the change in regions of permafrost is known?

5. questions related to species loss and the introduction of non -indigenous species;

6. questions about how the mix in biota in two specific northern forests might change over time in the face of fairly gradual and modest climate change;

7. questions similar to those above, but for fairly abrupt and substantial climate change;

8. questions about research needs and priorities for impacts of climate change and interaction between climate and natural ecosystems.

Finally, Morgan addressed how an elicitation procedure of this kind might work for the IPCC. He argued that someone with previous experience conducting elicitation studies should work with the authors to define the questions to be asked. Care should be taken to explicitly elaborate all the conditioning assumptions. The procedure should place emphasis on collecting information about the reasoning that underlies the probabilistic judgments obtained, not just probability distributions. The group of respondents should be chosen by the IPCC review experts.

The procedure should place emphasis on collecting information about the reasoning that underlies the probabilistic judgments obtained, not just probability distributions.

The objective should be to include all major points of view, not to produce a "statistically representative" sample. The initial judgments should be collected individually. Once the results are all in, they should be summarized, and then shared among the respondents. The results and associated reasons should be discussed at a workshop and respondents given an opportunity to revise their views.

For any particular question, there are a number of possible outcomes. For some questions, it may be suitable to combine the assessments into one summary distribution. For others, it might be more appropriate to combine them into several distributions reflecting different "schools of thought" on an issue, always conveying the reasons for the differences. In other cases, it might be best to summarize but not combine the assessments, again, conveying reasons. Morgan illustrated what some of the resulting displays might look like and stressed that understanding why the judgments are what they are is critical before combining them in any way.

Some discussion took place around the issue of how the group of experts should be chosen and how to ensure their independence. Ideally, Morgan thinks, the set of experts should be chosen to describe the range of seriously held views and the current state of knowledge including the diversity of opinions. There is a need, he says, to display the range of respectable opinion. It was also pointed out that it may be optimal to go to different experts for different questions.

Morgan concluded by pointing out that the point of this exercise is to summarize the state of knowledge and belief, not to achieve a false consensus. Masking disagreement or different perspectives will help neither scientists nor policymakers in the long run, he cautions. He suggests that the first IPCC report conveys a greater degree of consensus than does the results of the Morgan and Keith research. Finally, he stresses again that it is of utmost importance to convey the reasons which underlie any differences in views whenever expert opinions are combined.

Reference

Morgan, M. Granger and David W. Keith, 1995, Subjective Judgments by Climate Experts, Environmental Science & Technology, 29:468-476.

The point of this exercise is to summarize the state of knowledge and belief, not to achieve a false consensus. Masking disagreement or different perspectives will help neither scientists nor policymakers in the long run.

Forward to Next Section//Back to Table of Contents//AGCI Homepage//Comments: agcimail@agci.org