AGCI Session II: Characterizing and Communicating Scientific Uncertainty
Session Chairs: Dr. Richard H. Moss and Dr. Stephen H. Schneider
July 31 to August 8, 1996
Seismic Risk Analysis as an Example of Aggregating Expert Opinion
C. Allin Cornell
Stanford University
Stanford, California
As an example of aggregating expert opinion on a scientific issue, Cornell presented a process known as Probabilistic Seismic Hazard Analysis (PSHA) designed by the Senior Seismic Hazard Analysis Committee (SSHAC). SSHAC's PSHA is a methodology for estimating the likelihood that various levels of earthquake-caused ground motions will be exceeded at a given location in a given future time period. Due to large uncertainties in the geosciences data and in their modeling, multiple model interpretations are often possible, leading to disagreements among the experts. Cornell reported on a project, co -sponsored by the U. S. Nuclear Regulatory Commission, the U. S. Department of Energy, and the Electric Power Research Institute that was undertaken to review the state-of-the-art and improve on the overall stability of the PSHA process by providing methodological guidance on how to best perform the analysis. Cornell says that the resulting process represents an effective way of dealing with the problem of aggregating expert opinion in a particular scientific area for purposes of policy making.
The objective of aggregation is to represent the scientific community's composite state of knowledge on a particular issue. The process should seek to capture the diversity of interpretations, as opposed to the judgment of any particular expert. What should be sought in a properly executed PSHA project are: a) a representation of the legitimate range of technically supportable interpretations among the entire informed technical community, and b) the relative importance or credibility (weight) that should be assigned to the various hypotheses across that range. The type of consensus being sought, therefore, is that all experts agree that a particular composite probability distribution represents, first, them as a panel, and secondly, perhaps modified, the informed community as a whole.
The objective of
aggregation is to represent the scientific community's composite
state of knowledge on a particular issue. The process should seek to
capture the diversity of interpretations.
The first stage of the process may involve winnowing, in which a large group of experts reviews, discusses and gives weights to different approaches to models, dropping some approaches as demonstrably wrong or as not worth pursuing further in depth because, for example, they have been replaced by later variations or extensions by the same authors. The second stage involves the careful selection of experts for the remainder of the process. These experts are asked to wear different hats; some are proponents of particular models or approaches while others serve as evaluators of different models or data sets. The major role is held by these latter experts who are charged with evaluating and passing judgment in the form of uncertainty distributions on parameter values and/or alternative models.
The ultimate responsibility for the success of the process lies with the Technical Facilitator/Integrator (TFI). The TFI is a single entity, sometimes an individual, but preferably a small team that is responsible for identifying the key issues and components of the analysis, structuring and directing the interaction and debates among the proponents and evaluators, conducting any necessary numerical analyses, and documenting the process followed and the results obtained. The evaluator-experts should be made up of people who have been through these kinds of exercises before and have enough experience to have seen the state of the science change over time; they must have maturity and good judgment and must agree to work long and hard. They should be paid experts (which means resources are needed) who will work as "science court"-type evaluators, conducting a collegial effort to capture consensus.
At this point, participants pointed out that the IPCC process has thus far been a strictly voluntary one, and that raising the amounts of money that would be needed to follow the procedure described above could be difficult.
Like the IPCC, peer review is an important part of the process Cornell describes. He points out that it should be a "participatory" peer review, in which reviewers have full and frequent access throughout and can provide mid-course advice on both technical and procedural aspects of the process (as opposed to a late-stage peer review that occurs only after the project is almost complete).
The
evaluator-experts should be made up of people who have been through
these kinds of exercises before and have enough experience to have
seen the state of the science change over time; they must have
maturity and good judgment and must agree to work long and hard.
Discussing how uncertainties are dealt with in the PSHA process, Cornell defined two classes of uncertainties. Those called epistemic are uncertainties resulting from lack-of-knowledge and which are in principle reducible through further research and gathering of more and better data. Those called aleatory or random in character are uncertainties that for all practical purposes cannot be known in detail or cannot be reduced. He mentioned using "uncertainty expression training" in which experts are taught how to express their uncertainties with more precision.
Interaction is key to the process of achieving the outcome of better science. There can be no anonymity, each evaluator must have the same information, and understand all of each other's assumptions. An important part of eliminating unintended disagreement is focused, directed interaction, which involves all participants and downplays socially dominant individuals.
In sum, the process described provides for clarification of objectives, the roles of experts, and the meaning of consensus. It also involves numerical or mechanical aggregation schemes, and social integration procedures. The objective of the process is to be in a position to simply give equal weight to each evaluator (this does not mean equal weight to each model). (Peer weighting does not seem to work unless the individuals are going to stop working in the field.) The composite uncertainty band which results is the TFI's representation of the group's uncertainty. Although he may seek the group's affirmation that his is a consensus position, the TFI holds the responsibility for defending this result as an accurate characterization of the group. The process involves very strong interaction, and should be facilitated by the TFI to exploit while not becoming the victim of scientific culture. The cost of doing this process well for any given issue may be up to $1 million. Some participants saw cost as a potential obstacle to the IPCC adopting such a process.
The process
involves very strong interaction, and should be facilitated by the
TFI to exploit while not becoming the victim of scientific culture.