Perspectives and Principles from Other Assessments

Chris Bernabo

Many assessments of complex environmental issues have failed to meet the needs of the decision makers they were intended to serve. This is due to:

• lack of focus on decision makers' specific information needs
• inadequate stakeholder involvement and communication at all phases
• insufficient integration of quantitative and qualitative factors relevant to decision makers
• processes driven by scientists who are often amateurs at outreach and facilitation and may not understand stakeholders needs

When researchers apply assessment models as mechanistic predictive tools, they risk producing results that are not sufficiently relevant and are prone to misuse in decision making (Bernabo, 1998). Policy-relevant assessment is a more organic process involving researchers and stakeholders in an interactive exercise that builds understanding rather than trying to define "truth." The emphasis must be on providing practical information in an easily usable form.

The Problem of Contrasting Cultures

The underlying difficulty in conducting scientific assessments to assist decision making is rooted in the divergent purposes and different professional cultures of researchers and managers (Bernabo, 1995). Scientists ultimately seek to understand an issue while the decision makers' job is to decide. Decisions usually have deadlines and must be based on whatever level of scientific understanding exists, taking account of all the other societal factors that apply to the decision. The technical aspects are generally given less weight than social and political factors, much to the dismay of researchers pursuing scientific "truth." In short, science is a culture based on objective facts, proof, rationality, measurements, and incremental progress. Policy, on the other hand, is a culture based on subjective human values, beliefs, emotions, perceptions, deadlines, and crises.

Policy-relevant assessment is a more organic process involving researchers and stakeholders in an interactive exercise that builds understanding rather than trying to define "truth."

Effective and useful assessments must bridge these two cultures, and so must not only be technically sound but also policy credible. Researchers are well-versed on what is required to ensure scientific credibility, such as using well-documented and peer-reviewed information. But they often find that the needs of decision makers require stretching assumptions beyond what they believe "good science" allows, forcing them to make expert judgments and informed guesses or else forfeit contributing effectively to the process. Making the assessment policy-credible depends on how transparent, inclusive and unbiased the whole process is perceived to be by stakeholders. By applying multi-stakeholder approaches, assessment processes inform decision makers in a manner that facilitates sustainable outcomes that are technically and politically viable for society.

A key problem observed in previous assessments is how relevance is defined. Assessments are often designed based on researchers understanding of the scientific issues and their educated guesses of what decision makers would like to know rather than eliciting the decision makers' needs at the outset. With an approach that fails to fully engage the assessment's target audience in the initial design, it is unlikely that even the assessment's questions will be focused on those matters most relevant to decision making. What usually results is a state-of-the-science assessment with minimal usefulness to decision making. In practice, such results unintentionally provide fodder for opposing advocates on the issue to use selectively for supporting their predetermined positions in the policy debate.

The Role of Uncertainty

Another recurring theme is the issue of scientific uncertainty in the policy context. It has been observed that science is more effective at converting ignorance into uncertainty than it is at converting uncertainty into certainty. The level of certainty required for action is a function of the societal perception of the issue. The greater the societal consensus on an issue, the less scientific certainty required for action. The higher the societal costs of a policy, the greater the degree of certainty required for action. The inverse of these principles is also true and imply that enough certainty in the science is always defined relative to the political certainty in the issue. Therefore, enough scientific certainty in the policy process is a dynamic factor, not a static end point from research. These are political issues, not scientific ones. (For a treatment of some of the scientific issues in "Characterizing and Communicating Scientific Uncertainty," see AGCI's Elements of Change 1996 , Session Two.)

Science alone cannot provide answers to policymakers' ultimate questions because science necessarily is silent on the human values that underlie the decisions societies make. The scientific method itself is designed to screen out the value preferences and biases of the subjective humans beings who conduct research. Technical information is useful in identifying issues, developing options, providing understanding, and evaluating consequences for policy actions. But in the end, human values must be applied to determine what is "good" policy for a given society on a specific issue.

Science alone cannot provide answers to policymakers' ultimate questions because science necessarily is silent on the human values that underlie the decisions societies make.

Lessons Learned from Past Assessments

Based on dozens of past environmental assessments on a range of issues in several countries, some significant lessons emerge that provide insights for improving the usefulness of assessments for decision makers. The Social Learning Project at Harvard University (Clark and Dickson, 1998) has documented how decision makers have responded to various assessments. Five general lessons about assessments that have been judged most effective by decision makers are as follows:

(1) The assessment process is inclusive and well-designed. Communication can be more important than model integration in determining success. Complex model wiring diagrams suggesting all the pathways to be studied are of limited value unless there is a carefully planned process of interactive flow of information among researchers, synthesizers, stakeholders, reviewers and end users.

(2) The assessment results in expanding options for decision makers. The assessment process can provide a vehicle for finding win-win solutions. Merely providing detailed analysis of specific policies rarely advances the decisions as usefully as when new options are generated in an interactive process between producers and users of the information.

(3) The assessment focuses on real uses at the regional level. Broad generalizations on the national or global scale have limited value at the more local scales where most impacts of a decision are experienced. Averages integrating over large areas can obscure the more localized texture that political representation is based on in national and global fora.

(4) Multiple partial assessments are performed on key components. The end results of large integrations of many factors tend to suffer from the limitations of their weakest links. These limitations and their complexity make them harder to use and interpret by decision makers than sets of comparable, separate analyses.

(5) Assessments are conducted repeatedly by a core group. Single-time grand integration efforts are less likely to benefit from the learning required to be most useful for decision makers. Assessments ultimately need iteration to yield successively better results with a stable core group of assessors who incorporate the learning as they proceed.

These lessons indicate the primary importance of the process, not just the report or other tangible outputs of an assessment effort. In fact, the value of a report is often outweighed by the interactive learning process involving researchers, assessors, decision makers, stakeholders, and the public.

As policy-making moves through this cycle (Figure 2.3), there is a shift in what decision makers want to know.

The assessment process can expand options for decision makers and provide a vehicle for finding win-win solutions.

Figure 2.3

The Policy-Making Life Cycle

What is a Policy-Relevant Assessment?

A policy-relevant assessment is an iterative multi-stakeholder process for systematically analyzing data and synthesizing information into a form that facilitates use in decision making (see Figure 2.4).

Figure 2.4

A policy-relevant assessment is an iterative process linking science and decision making.

Based on the lessons of past efforts, there are a number of factors that enhance the effectiveness of policy-relevant assessments. These attributes include:

• responding to clearly defined needs relevant to decision making
• involving decision makers in framing the questions and designing the process
• maintaining interaction with end users during planning and implementation
• developing and implementing a multi-stakeholder communication strategy
• establishing credibility with periodic open reviews of the science by research peers and the assessment's process and assumptions by stakeholders
• using the full spectrum of research approaches and tools to address the issues of importance to decision makers that can be quantitatively modeled and those that can only be qualitatively described, explicitly estimating uncertainties
• managing the process for successful assessment outputs rather than as research
• using assessment specialists to integrate the outputs of expert teams

Evolution of Assessments

When taken together, the attributes outlined above suggest a new model for conducting assessments that is less mechanistic and more organic than previous efforts. Assessment can be practiced as a learning-centered dialogue, rather than the traditional exercise in defining "truth" or predicting probable future outcomes. The previous style was mechanistic, Newtonian, and reductionist, based on "truth," predictions , science, outcomes and answers. The evolving style requires a more organic and holistic approach based on learning, options, the context of decision making, establishing processes and framing questions.

Lessons Learned from the Joint Climate Project

In 1990, a project was designed and conducted by Science & Policy Associates, Inc., called the "Joint Climate Project to Address Decision Makers' Uncertainties." This private-federal partnership, sponsored by the Electric Power Research Institute, the U. S. Environmental Protection Agency, the U. S. Forest Service, and the U. S. Departments of Energy, Agriculture and Interior, established a multi-stakeholder dialogue to help identify some major questions U. S. decision makers had about climate change and then had scientists determine what research and time frames would be required to address those questions. The project yielded several key findings with implications for enhancing communication and increasing the value of research results.

Assessment can be practiced as a learning-centered dialogue, rather than the traditional exercise in defining "truth" or predicting probable future outcomes.

From the decision makers' perspective

• International perspectives drive policy.
• Climate change impacts and human responses are key to decision making.
• Implications of uncertainties need clarification.
• Certainty is not a prerequisite for action.

From the researchers' perspective

• Timely and useful results are available before reliable predictions.
• Greater emphasis is needed on impacts and human response strategies.
• Integrated assessments and case studies are useful.
• Expect the unexpected.
• An international perspective for research is essential.

Lessons in Communication

Discussion during the Joint Climate Project provided ample evidence that both researchers and decision makers are uncomfortable with the current situation and are anxious to develop and sustain a productive dialogue. Doing this successfully will require:

• balancing research and response strategies (it's not a choice of whether we act OR do research; both are needed)
• approaching climate change in a "relative risk" context
• focusing on educating decision makers on the facts and uncertainties of the issue
• recognizing that research does not always provide the answer and may even increase uncertainty in some areas
• developing an ongoing facilitated process between decision makers and researchers to direct research

Negotiations Versus Dialogue

There is a distinction between negotiations, which are necessary but not sufficient, and dialogue, which facilitates more productive negotiations.

In negotiations:

• all societal perspectives are reduced to one official position
• motivations and options are not explicit
• each party seeks to maximize its advantage
• there is convergence toward a single least-objectionable outcome

Dialogue:

• expands the diversity of positions addressed
• examines motivations to enhance level of understanding
• reaches consensus on a range of options
• provides the basis for win/win solutions

Conclusions

Better solutions require:

In facilitated dialogues, decision makers and researchers:

• establish consensus on significance of science
• increase mutual understanding
• develop new options and analyses for better solutions

Targeted assessments are science for decision making which:

• focus on stakeholder needs, not global common denominators
• examine specific risks and benefits for stakeholders
• develop research to fill policy-relevant gaps

Improving assessments to meet the needs of decision makers requires new approaches that involve stakeholders in interactive learning processes. Facilitated dialogues between the producers and users of assessments build understanding, enhance relevancy, and increase the credibility of the results. The value of moving beyond traditional approaches is demonstrated by recent examples of successful assessments applying structured dialogues, interactive design phases, and multi-stakeholder implementation. Policy-relevant assessments are interactive learning processes rather than truth-defining end-points. To be of greatest value, they must include both the quantitative and qualitative factors for analyzing decision making options.

References

Bernabo, J. Christopher, 1995. Communication Among Scientists, Decision Makers and Society: Developing Policy Relevant Global Climate Change Research. Pages 103 -116 in Climate Change Research: Evaluation and Policy Implications , S. Zwerver (Ed.), Elsevier Science B. V., Amsterdam, Netherlands.

Bernabo, J. Christopher, 1998. Improving Integrated Assessments for Application to Decision Making. In Risk Assessment: Linking Science to Policy , S. Lee and T. Schneider (Eds.), Elsevier Science B. V., Amsterdam, Netherlands.

Clark, W. C., and N. Dickson, 1998. Learning to Manage Global Environmental Risks: A Comparative History of Social Responses to Climate Change, Ozone Depletion and Acid Rain. MIT Press, Cambridge, Massachusetts.

Facilitated dialogues between the producers and users of assessments build understanding, enhance relevancy, and increase the credibility of the results.

---

Forward to Next Section//Back to Table of Contents//AGCI Homepage//Comments: agcimail@agci.org