Appendix B: Consultation with Experts and Document Review
Monterey Bay National Marine Sanctuary

photo of humpback whale breaching

The process for preparing condition reports (and similarly, this update) involves a combination of accepted techniques for collecting and interpreting information gathered from subject matter experts. The approach varies somewhat from sanctuary to sanctuary, in order to accommodate different styles for work with partners. The Monterey Bay National Marine Sanctuary approach was closely related to the Delphi Method, a technique designed to organize group communication among a panel of geographically dispersed experts by using questionnaires, ultimately facilitating the formation of a group judgment. This method can be applied when it is necessary for decisionmakers to combine the testimony of a group of experts, whether in the form of facts or informed opinion, or both, into a single useful statement.

The Delphi Method relies on repeated interactions with experts who respond to questions with a limited number of choices to arrive at the best supported answers. Feedback to the experts allows them to refine their views, gradually moving the group toward the most agreeable judgment. For condition reports, the Office of National Marine Sanctuaries uses standardized questions related to the status and trends of sanctuary resources, with accompanying descriptions and five possible choices that describe resource condition (Appendix A).

In order to address the standardized questions, sanctuary staff selected and consulted outside experts familiar with water quality, living resources, habitat and maritime archaeological resources in the estuarine, nearshore, offshore and seamount environments. A few different approaches (e.g., small group meetings, conference calls, email and individual meetings) were used to get expert input on the questions, depending on the availability of experts (a list of experts who provided input is available in the Acknowledgement section of this report).

In these meetings and calls, experts were introduced to the questions and then asked to provide recommendations and supporting arguments. In small group settings and conference calls, the group converged in their opinion of the rating that most accurately described the current resource condition. In individual meetings and email correspondence, the sanctuary staff considered all input and decided on status and trend ratings. In all cases, draft status and trend ratings along with supporting narratives were made available to experts for individual comment.

Experts were also consulted to assign a level of confidence in status and trend ratings by: (1.) characterizing the sources of information they used to make judgments and (2.) their agreement that the available evidence supports the selected status and trend ratings. The evidence and agreement ratings were then combined to determine the overall confidence ratings, as described in the table here.

 

Step 1: Rate Evidence
Consider three categories of evidence typically used to make status or trend ratings: (1.) data,(2.) published information and(3.) personal experience.

Evidence Scores
Limited
Medium
Robust
Limited data or published information, and little or no substantive personal experience.Data available, some peer reviewed published information, or direct personal experience. Considerable data, extensive record of publication, or extensive personal experience.

Step 2: Rate Agreement
Rate agreement among those participating in determining the status and trend rating, or if possible, within the broader scientific community. Levels of agreement can be characterized as "low," "medium" or "high."

Step 3: Rate Confidence
Using the matrix below, combine ratings for both evidence and agreement to identify a level of confidence. Levels of confidence can be characterized as "very low," "low," "medium," "high" or "very high."
chart of rate confidence

 

An initial draft of the update, which was written by sanctuary staff, summarized the new information, expert opinions and level of confidence expressed by the experts (who based their input on knowledge and perceptions of local conditions). Comments, data and citations received from the experts were included, as appropriate, in text supporting the ratings.  This initial draft of the update was made available to contributing experts and data providers which allowed them to review the content and determine if the report accurately reflected their input, identify information gaps, provide comments or suggest revisions to the ratings and text. Upon receiving those comments, the writing team revised the text and ratings as they deemed appropriate. In some cases, additional review of certain sections, by those with specific expertise, was requested after revision. Sometimes, additional input on confidence scores was requested if the status and trend changed after those ratings had first been established in a small group setting.

In July 2015, a draft final report was sent to regional scientists for final review (listed in the Acknowledgemen’ section of this report). In December 2004, External Peer Review became a requirement when the White House Office of Management and Budget (OMB) issued a Final Information Quality Bulletin for Peer Review (OMB Bulletin) established peer review standards that would enhance the quality and credibility of the federal government’s scientific information. Along with other information, these standards apply to Influential Scientific Information, which is information that can reasonably be determined to have a "clear and substantial impact on important public policies or private sector decisions." The condition reports are considered Influential Scientific Information. For this reason, these reports are subject to the review requirements of both the Information Quality Act and the OMB Bulletin guidelines. Therefore, following the completion of every condition report, they are reviewed by a minimum of three individuals who are considered to be experts in their field, were not involved in the development of the report and are not ONMS employees. Comments from these peer reviews were incorporated into the final text of this report. Furthermore, OMB Bulletin guidelines require that reviewer comments, names and affiliations be posted on the agency website, http://www.cio.noaa.gov/. Reviewer comments, however, are not attributed to specific individuals. Comments by the External Peer Reviewers are posted at the same time as the formatted final document.

The reviewers were asked to review the technical merits of resource ratings and accompanying text, as well as to point out any omissions or factual errors. Following the External Peer Review, the comments and recommendations of the reviewers were considered by sanctuary staff and incorporated, as appropriate, into a final draft document. The final interpretation, ratings and text in the draft condition report were the responsibility of sanctuary staff, with final approval by the sanctuary superintendent. To emphasize this important point, authorship of the report is attributed to the sanctuary alone. Subject experts were not authors, though their efforts and affiliations are acknowledged in the report.

Estuarine Environment - Confidence Scoring Table

Question

2015 Rating

Evidence
(Limited, Medium or Robust)

Agreement
(Low, Medium or High)

Confidence
(Very Low, Low, Medium, High or Very High)

Water Quality

Question 1: Multiple Stressors

Status: Fair/Poor

Not updated

Not updated

Not updated

Trend: Declining

Not updated

Not updated

Not updated

Question 2: Eutrophic Condition

Status: Fair/Poor

Robust

High

Very High

Trend: Declining

Robust

High

Very High

Question 3: Risks to Human Health

Status: Fair/Poor

Not updated

Not updated

Not updated

Trend: Undetermined

Not updated

Not updated

Not updated

Question 4: Human activities and Water Quality

Status: Fair

Medium

High

High

Trend: Improving

Medium

High

High

Habitat

Question 5: Major Habitat

Status: Fair/Poor

Robust

High

Very High

Trend: Not changing

Medium

Low

Low

Question 6: Biologically-Structured

Status: Poor

Robust

High

Very High

Trend: Improving

Robust

Medium

High

Question 7: Contaminants

Status: Fair/Poor

Low

Low

Very Low

Trend: Declining

Low

Low

Very Low

Question 8: Human Activities and Habitat

Status: Poor

Medium

Medium

Medium

Trend: Improving

Medium

Low

Low

Living Resources

Question 9: Biodiversity

Status: Fair

Medium

Medium

Medium

Trend: Not changing

Medium

Low

Low

Question 11: Non-Indigenous Species

Status: Poor

Medium

Medium

Medium

Trend: Not changing

Medium

Medium

Medium

Question 12: Status Key Species

Status: Fair/Poor

Robust

High

Very High

Trend: Improving

Robust

High

Very High

Question 13: Condition Key Species

Status: Good/Fair

Limited

Medium

Low

Trend: Undetermined

Limited

Medium

Low

Question 14: Human Activities and Living Resources

Status: Fair/Poor

Medium

Medium

Medium

Trend: Undetermined

Limited

Medium

Low

Maritime Archaeological Resources

Question 15: Integrity

Status: Undetermined

Not updated

Not updated

Not updated

Trend: Undetermined

Not updated

Not updated

Not updated

Question 16: Threat to Environment

Status: Good

Not updated

Not updated

Not updated

Trend: Not changing

Not updated

Not updated

Not updated

Question 17: Human Activities

Status: Good

Not updated

Not updated

Not updated

Trend: Not changing

Not updated

Not updated

Not updated

 

Nearshore Environment - Confidence Scoring Table

Question

2015 Rating

Evidence
(Limited, Medium or Robust)

Agreement
(Low, Medium or High)

Confidence
(Very Low, Low, Medium, High or Very High)

Water Quality

Question 1: Multiple Stressors

Status: Fair

Not updated

Not updated

Not updated

Trend: Declining

Not updated

Not updated

Not updated

Question 2: Eutrophic Condition

Status: Fair

Robust

Medium

High

Trend: Declining

Robust

Medium

High

Question 3: Risks to Human Health

Status: Fair

Robust

High

Very High

Trend: Undetermined

Robust

High

Very High

Question 4: Human Activities and Water Quality

Status: Fair

Limited

High

Medium

Trend: Improving

Limited

High

Medium

Habitat

Question 5: Major Habitat

Status: Fair

Robust

High

Very High

Trend: Declining

Robust

High

Very High

Question 6: Biologically-Structured

Status: Good

Robust

High

Very High

Trend: Not changing

Robust

High

Very High

Question 7: Contaminants

Status: Fair/Poor

Medium

High

High

Trend: Declining

Medium

High

High

Question 8: Human Activities and Habitat

Status: Fair

Robust

Low

Medium

Trend: Undetermined

Robust

Low

Medium

Living Resources

Question 9: Biodiversity

Status: Fair

Robust

High

Very High

Trend: Not changing

Robust

High

Very High

Question 11: Non-Indigenous Species

Status: Good

Robust

High

Very High

Trend: Declining

Robust

High

Very High

Question 12: Status Key Species

Status: Fair

Robust

High

Very High

Trend: Declining

Robust

High

Very High

Question 13: Condition Key Species

Status: Fair

Robust

High

Very High

Trend: Declining

Robust

High

Very High

Question 14: Human Activities and Living Resources

Status: Fair

Robust

High

Very High

Trend: Declining

Robust

High

Very High

Maritime Archaeological Resources

Question 15: Integrity

Status: Fair

Not updated

Not updated

Not updated

Trend: Undetermined

Not updated

Not updated

Not updated

Question 16: Threat to Environment

Status: Fair

Medium

Medium

Medium

Trend: Declining

Medium

Medium

Medium

Question 17: Human Activities

Status: Good/Fair

Not updated

Not updated

Not updated

Trend: Undetermined

Not updated

Not updated

Not updated



Offshore Environment - Confidence Scoring Table

 
Question

2015 Rating

Evidence
(Limited, Medium or Robust)

Agreement
(Low, Medium or High)

Confidence
(Very Low, Low, Medium, High or Very High)

Water Quality

Question 1: Multiple Stressors

Status: Fair

Medium

High

High

Trend: Declining

Robust

High

Very High

Question 2: Eutrophic Condition

Status: Good/Fair

Robust

High

Very High

Trend: Declining

Medium

Medium

Medium

Question 3: Risks to Human Health

Status: Good/Fair

Not updated

Not updated

Not updated

Trend: Undetermined

Not updated

Not updated

Not updated

Question 4: Human Activities and Water Quality

Status: Fair

Not updated

Not updated

Not updated

Trend: Improving

Not updated

Not updated

Not updated

Habitat

Question 5: Major Habitat

Status: Fair

Medium

High

High

Trend: Improving

Low

High

Medium

Question 6: Biologically-Structured

Status: Fair/Poor

Medium

High

High

Trend: Undetermined

Low

High

Medium

Question 7: Contaminants

Status: Fair

Medium

High

High

Trend: Declining

Medium

High

High

Question 8: Human Activities and Habitat

Status: Fair

Medium

High

High

Trend: Improving

Medium

High

High

Living Resources

Question 9: Biodiversity

Status: Fair

 Medium

Medium

 Medium

Trend: Not changing

Low

Medium

Low

Question 11: Non-Indigenous Species

Status: Good

Not updated

Not updated

Not updated

Trend: Not changing

Not updated

Not updated

Not updated

Question 12: Status Key Species

Status: Good/Fair

 Medium

Medium

 Medium

Trend: Not changing

 Medium

Low

 Low

Question 13: Condition Key Species

Status: Good/Fair

 Medium

Medium

 Medium

Trend: Declining

 Medium

Low

 Low

Question 14: Human Activities  and Living Resources

Status: Fair

 Medium

Medium

 Medium

Trend: Not changing

 Medium

Medium

 Medium

Maritime Archaeological Resources

Question 15: Integrity

Status: Undetermined

Not updated

Not updated

Not updated

Trend: Undetermined

Not updated

Not updated

Not updated

Question 16: Threat to Environment

Status: Fair

Medium

Medium

Medium

Trend: Declining

Medium

Medium

Medium

Question 17: Human Activities

Status: Good/Fair

Not updated

Not updated

Not updated

Trend: Undetermined

Not updated

Not updated

Not updated

 


Seamount Environment - Confidence Scoring Table

 
Question

2015 Rating

Evidence
(Limited, Medium or Robust)

Agreement
(Low, Medium or High)

Confidence
(Very Low, Low, Medium, High or Very High)

Water Quality

Question 1: Multiple Stressors

Status: Undetermined

N/A

N/A

N/A

Trend: Undetermined

N/A

N/A

N/A

Question 2: Eutrophic Condition

Status: Undetermined

N/A

N/A

N/A

Trend: Undetermined

N/A

N/A

N/A

Question 3: Risks to Human Health

Status: Undetermined

N/A

N/A

N/A

Trend: Undetermined

N/A

N/A

N/A

Question 4: Human Activities and Water Quality

Status: Good/Fair

Limited

High

Medium

Trend: Undetermined

Limited

High

Medium

Habitat

Question 5: Major Habitat

Status: Good

Robust

High

Very High

Trend: Stable

Medium

High

High

Question 6: Biologically-Structured

Status: Good

Robust

High

Very High

Trend: Undetermined

Limited

High

Medium

Question 7: Contaminants

Status: Undetermined

N/A

N/A

N/A

Trend: Undetermined

N/A

N/A

N/A

Question 8: Human Activities and Habitat

Status: Good/Fair

Medium

High

High

Trend: Undetermined

Limited

High

Medium

Living Resources

Question 9: Biodiversity

Status: Good

Robust

High

Very High

Trend: Undetermined

Medium

High

High

Question 11: Non-Indigenous Species

Status: Good

Limited

High

Medium

Trend: Not changing

Limited

High

Medium

Question 12: Status Key Species

Status: Good/Fair

Robust

Medium

High

Trend: Increasing

Medium

High

High

Question 13: Condition Key Species

Status: Good

Medium

High

High

Trend: Not changing

Limited

High

Medium

Question 14: Human Activities and Living Resources

Status: Good/Fair

Medium

High

High

Trend: Undetermined

Limited

High

Medium

Maritime Archaeological Resources

Question 15: Integrity

Status: N/A

N/A

N/A

N/A

Trend: N/A

N/A

N/A

N/A

Question 16: Threat to Environment

Status: N/A

N/A

N/A

N/A

Trend: N/A

N/A

N/A

N/A

Question 17: Human Activities

Status: N/A

N/A

N/A

N/A

Trend: N/A

N/A

N/A

N/A