AGCI Session II: Characterizing and Communicating Scientific Uncertainty
Session Chairs: Dr. Richard H. Moss and Dr. Stephen H. Schneider
July 31 to August 8, 1996
Different Levels of Treatment of Uncertainty in Risk Analysis and Aggregation of Expert Opinions
Elisabeth Paté-Cornell
Stanford University
Stanford, California
Paté-Cornell comes from the engineering systems analysis and risk analysis tradition which uses probability more than the environmental risk analysis tradition. The level of sophistication with which uncertainty is treated should depend on the use that will be made of the information. Risk analysis is necessarily a snapshot of the information available at the time a particular decision is made. The level of treatment depends on both the importance of the risk and the costs of the mitigation measures. Sometimes, a "level zero" treatment of uncertainty is all that is needed, simply asking: is there a risk or not? At a slightly higher level, we can ask, what is the maximum loss that can occur? And if we believe that we can afford to incur this loss, the analysis can end there. The trouble is that often what we assume to be the worst case can be made still worse by adding other details to catastrophic scenarios, so this level is generally not sufficient for significant risks.
The most often used level of sophistication in uncertainty analysis in environmental health risk assessments is what Paté-Cornell calls "quasi-worst case" or "plausible upper bounds." This is the level of sophistication, for example, in carcinogen risk assessment by the Environmental Protection Agency. The problem with this approach, she says, is that it does not enable one to rank the risks because the probable level of overestimation is likely to vary from one risk to another. If one does not have unlimited funds to spend, and has to set priorities, it is important to be to able determine which risk is likely to result in the greatest losses. Using this method, one cannot be sure that the plausible upper bound of risk one, which may be greater than the plausible upper bound of risk two, will actually result in a greater value of losses in terms of expected values.
Risk analysis is
necessarily a snapshot of the information available at the time a
particular decision is made. The level of treatment depends on both
the importance of the risk and the costs of the mitigation measures.
Therefore, there is some need for a "central value" of the potential outcome distribution. The third level of uncertainty analysis in risk assessment generally involves such "best estimates" or "central values." In many of these risk analyses there are several possible models and mechanisms, and for each of these, a spectrum of possible parameter values. The approach to a "best estimate" analysis is to pick the most likely among these possible mechanisms, but often, the most likely thing is that nothing happens. The problem with this approach is that the most likely mechanism may not be the one that would result in the kinds of losses that one might be most concerned about in designing public policies or private risk management decisions. Because of all these shortcomings, the engineering field has developed the methods of probabilistic risk analysis.
In probabilistic risk analysis, one begins by dividing a complicated system into subsystems, analyzing the functions that have to be performed in order for the system to work, evaluating external events and loads, and computing the probability of failure of the whole system based on the probability of failure of its components as well as external factors. An example of this is the seismic risk analysis that civil engineers perform in which there are two basic problems to resolve. First, what is the seismicity and the probability that the ground will move with different levels of severity in a given time period? Second, given the seismicity, what will happen to structures. In such a problem, there are several kinds of experts involved in the assessment, and probabilistic methods are needed to combine these pieces together.
Very often fundamental uncertainties are encountered about the mechanisms themselves (for example, seismic mechanisms at a given site). Some of these are epistemological (fundamental) uncertainties and others are aleatory (arising from randomness or variability) uncertainties. When it comes to treating epistemic uncertainties in the face of limited knowledge, the heart of the problem is often how to treat the disagreements among experts. Paté-Cornell discussed several methods for aggregating expert opinions into single probability distributions for the spectrum of possible mechanisms, and for the parameter values for each model.
Very often
fundamental uncertainties are encountered. Some of these are
epistemological (fundamental) uncertainties and others are aleatory
(arising from randomness or variability) uncertainties.
The Delphi method is an iterative method which begins by interviewing experts separately. The results are collected and fed back to the experts in order to give them the opportunity to modify their opinions based on their colleagues' answers. This method generally leads to rapid convergence and consensus, but the answer may not bear much resemblance to what one would obtain by combining all of the evidence. This simple, purely interactive method is a social mechanism whereby experts are brought together to share their evidence and mental models. This allows them to constitute the larger body of evidence and to enrich each other's knowledge with new information. They are essentially left together until they can agree upon an answer.
The purely analytical method involves gathering the opinions of experts separately and then weighting their models, for example by Bayesian probabilistic analysis methods, involving a "super-expert" who determines the probability of the phenomenon given what each of the experts says. One problem that can arise is that the experts are generally not truly independent and it is difficult in practice to evaluate their influences on one another. Another problem is that they may not share the same evidence base and have no opportunity to exchange information in this method.
Another (probably better) method is thus a variation on this purely analytical one in which models are weighted according to some more subjective means than the "super-expert." This process is both social and analytical. It is analytical in that it is based on Bayesian logic, and it is also a social process in that the experts come to a common conclusion in an interactive manner. The result (and the success of this exercise) depends upon the kind of people in the group, the way they are allowed to intervene, and in what manner they are allowed to support their views, evaluate other models, and integrate that information to come up with the group's aggregation of material.
One problem that
can arise is that the experts are generally not truly independent and
it is difficult in practice to evaluate their influences on one
another.