Skip Navigation

Appendix C: Performance Measures Definitions

 

STRATEGIC OBJECTIVE 3.2

Enhance the conservation and management of coastal and marine resources to meet America’s economic, social, and environmental needs

PERFORMANCE OBJECTIVE: Protect, restore, and manage the use of coastal and ocean resources through an ecosystem approach to management (NOAA)

Performance Measure:
  • Number of protected species designated as threatened, endangered, or depleted with stable or increasing population levels

This measure tracks progress at achieving partial recovery of endangered, threatened, or depleted protected species under the jurisdiction of the National Marine Fisheries Service (NMFS) from a baseline of 65 species established as of January 1, 2004. Protected species are defined as all marine mammal stocks (except walruses, polar bears, and manatees) and those domestic non-marine mammal species listed as threatened or endangered under the Endangered Species Act that are under the jurisdiction of NMFS. Marine mammal species included in this measure are those listed as “depleted” under the Marine Mammal Protection Act (MMPA), which includes any listed under Endangered Species Act. Recovery of threatened, endangered, or depleted protected species is very slow and can take decades. While it may not be possible to recover or delist a species in the near term, progress can be made to stabilize or increase the species. For some, it is trying to stop a steep decline (right whales, stellar sea lions); for others it is trying to increase their numbers/abundance (Ridley turtles). NOAA’s protected species management efforts are focused on halting declines and conserving species while still allowing human activities to continue.

Data Verification and Validation Summary
Data source MMPA stock assessment reports and Endangered Species Act status reviews
Frequency Annual
Data storage Excel spreadsheet maintained by NMFS’s Office of Protected Resources.
Internal Controls Results are reported quarterly in a signed memo from the Protected Species Program Manager to the NMFS CFO and are housed and made available in a database managed by NMFS; quarterly reporting on performance to NOAA Deputy Under Secretary.
Data Limitations MMPA stock assessment reports are updated only once a year and Endangered Species Act status reviews are updated only every one to five years depending on priority and fund availability.
Actions to be Taken Discussions are ongoing to include protected species in the NMFS Stock Information System (SIS).


Performance Measure:
  • Number of habitat acres restored (annual/cumulative)

Habitat restoration and long-term protection are critically needed to help maintain the function of important coastal and marine ecosystems. Under NOAA’s legislative mandates, NOAA protects and restores key habitats that provide critical ecosystem functions that support the health of endangered or threatened species, essential fish habitat, as well as provide a number of other societal or economic benefits. NOAA restores habitat areas lost or degraded as a result of development and other human activities, as well as specific pollution incidents and sources. Activities are geared toward NOAA trust resources found across the marine environment and supportive of anadromous fish species. This measure summarizes or projects the geographic area over which ecosystem function has been or will be improved as the direct result of habitat restoration efforts.

Data Verification and Validation Summary
Data source Interim and final progress reports from each project
Frequency Quarterly
Data storage Restoration Center Database (RCDB)
Internal Controls Results are reported quarterly in a signed memo from the Habitat Program Manager to the NMFS CFO and are housed and made available in a database managed by NMFS; quarterly reporting on performance to NOAA Deputy Under Secretary.
Data Limitations Data are primarily provided by grantees.
Actions to be Taken None


Performance Measures:
  • Cumulative number of coastal, marine, and Great Lakes issue-based forecasting capabilities developed and used for management
  • Annual number of coastal, marine, and Great Lakes ecological characterizations that meet management needs

Sound management of coastal and ocean ecosystems requires scientifically based information on their condition. NOAA is developing methods to scale up from the site characterizations it currently produces to ecosystem characterizations. Characterization includes identification of the physical location, spatial extent, and biological, chemical, and physical characteristics. Site characterizations improve understanding of the history, current state, and future condition of ecosystems, and ecosystem characterizations will be the cornerstones to ecosystem-based management and the basis for many coastal, marine, and Great Lakes management tools, including forecasts, assessments, and management plans. NOAA decides what to characterize based on: user community priorities, adequacy of indicators, significance of issue, and consequences of management action/inaction. Characterization of an ecosystem site (and in the future, a defined subecosystem) is measured as uncharacterized, substantially characterized, or adequately characterized. NOAA has initiated a process to divide each of the large marine ecosystems into subecosystems; ecosystem sites are being used as a proxy unit of measurement until that effort is completed. (At that point, NOAA will measure the cumulative number of ecosystems adequately characterized for management. Currently the measure tracks the progress of 13 National Marine Sanctuaries (NMS) and 26 National Estuarine Research Reserve Systems (NERRS) in completing monitoring and assessment to characterize the sites for ongoing management and long-term protection.

Data Verification and Validation Summary
Data source Characterizations focus on ecosystem sites: NMS, NERRS, coral reef ecosystems, the coastal zone, Great Lakes, essential fish habitat, ecological species units, and unexplored areas.
Frequency Annual
Data storage Metadata from all contributing sources to the measure are maintained by managers for the coastal and marine resources and ecosystem research programs and stored in an Excel database with limited access. The final performance data reported in quarterly and annual performance reports are managed in a secure NOS database for annual milestones and annual and long-term performance measures. Changes to reporting data require approval by the NOS administrator (managed by an email workflow approval system).
Internal Controls Results are reported monthly to the Ecosystems Research Program (ERP) program manager and NOAA CFOs; quarterly reports on performance data are submitted to the NOAA Deputy Under Secretary.
Data Limitations NOAA focuses on protected areas or areas where NOAA has a clear management mandate. NOAA works to identify key parameters for characterizing their conditions and develop assessments of their present health. Characterizations from all contributors are being tracked in this new measure in addition to criteria defining the indicator of what meets management needs for each ecosystem site because characterizations vary temporally and geographically.
Actions to be Taken None


Performance Measure:
  • Annual number of coastal, marine, and Great Lakes habitat acres acquired or designated for long-term protection

NOAA maintains the health of coastal, marine, and Great Lakes habitats by designating and managing important areas for long-term conservation and by providing support to state and local governments to protect additional key habitats by purchasing land from willing sellers. This measure tracks the number of acres acquired with NOAA funds by state or local government agencies from willing sellers for long-term protection of important coastal habitats, or the number of acres designated for long-term protection by NOAA or by state partners. The protected acres are the actual number of acres newly protected in a fiscal year. The cumulative total represents acres acquired or designated to date for NERRS, National Marine Sanctuary Program (NMSP), and Coastal and Estuarine Land Conservation Program. The goal for the long-term protection indicator is variable, as the yearly target can vary from hundreds to thousands of acres each year. For example, the initial designation or acquisition for a new reserve or sanctuary may add hundreds of thousands of acres in one year, while in other years acquisition may result in several hundred or thousand acres protected.

Data Verification and Validation Summary
Data source The cumulative total represents data on acres from the NERRS program, NMSP, and the Coastal and Estuarine Land Conservation Program.
Frequency Annually by each program manager.
Data storage Metadata from all contributing sources to the measure are managed by the Coastal and Marine Resources Program (CMRP) Manager and stored in an Excel spreadsheet with limited access. The final performance data reported annually in performance reports are managed in a secure NOS database for annual milestones and annual and long-term performance measures. Changes to reporting data require approval by the NOS administrator (managed by an email workflow approval system).
Internal Controls Results are reported annually to the contributing NOAA CMRP and NOAA CFOs for approval; monthly reports on performance data are submitted to the NOAA Deputy Under Secretary.
Data Limitations The goal for the long-term protection indicator is variable, as the yearly target can vary from hundreds to thousands of acres each year. For example, the initial designation or acquisition for a new reserve or sanctuary may add hundreds of thousands of acres in one year, while in other years acquisition may result in several hundred or thousand acres protected. Other limitations are the timeliness of reporting by grant recipients, accuracy of conversion from hectares to acres for some data, and the time delay between funding and completion.
Actions to be Taken Since this measure does not capture all NOAA’s activities to protect habitat, NOAA seeks to expand the measure in the future. NOAA is looking at the feasibility of further harmonizing methodologies used among contributing program components.


Performance Measure:
  • Fish stock sustainability index (FSSI)

The FSSI replaces the measure “Number of Overfished Major Stocks of Fish.” The index tracks the outcome of building and maintaining fish stocks at productive levels while also capturing the critical components of NOAA’s efforts to get to that outcome, i.e., managing fish harvest rates and increasing knowledge about the status of fish stocks. The measure provides a much more complete picture than the old measure of NOAA’s success at fisheries management. The FSSI is calculated by assigning a total score between 0 and 4 to each of 230 priority fish stocks (see below). Each stock receives one point if:

  • NOAA has determined whether or not (1) the stock is overfished (one-half point) and (2) the stock is subject to overfishing (one-half point); i.e., scientific knowledge is available about the stock;
  • NOAA’s management measures are succeeding at ensuring that fishing does not remove too many fish (i.e., level of fishing mortality does not exceed the threshold for overfishing);
  • The stock is managed at an acceptable level (i.e., biomass is above the level defined as overfished for the stock);
  • The stock is rebuilt or is at its “optimal” level, the ultimate long term end state for a stock (i.e., biomass is within 80 percent of that required to achieve maximum sustainable yield).

The FSSI is the sum of the scores of the individual stocks. The highest possible score for a stock is four and for the index is 920, since the FSSI is based on a set of 230 priority fish stocks selected for their importance to commercial and recreational fisheries. Criteria for selection of stocks include whether they are major stocks (landings greater than 200,000 pounds), whether they are overfished or subject to overfishing, whether they have assessments scheduled, whether they have previously been identified as important, or other factors as appropriate. These stocks represent about 90 percent of all commercial landings in the United States. NOAA plans for this set of stocks to be tracked over a five-year period.

Data Verification and Validation Summary
Data source Stock assessments and status determinations
Frequency Quarterly
Data storage NMFS SIS
Internal Controls Results will be reported quarterly in a signed memo from the Fishery Management Program Manager to the NMFS CFO and are housed and made available in a database managed by NMFS; monthly reporting on performance to NOAA Deputy Under Secretary.
Data Limitations Results can only be reported when the SIS is updated with new information from the field.
Actions to be Taken None


Performance Measure:
  • Percentage of living marine resources (LMR) with adequate population assessments and forecasts

This measure tracks the percent of priority fish stocks and protected species stocks that have adequate population assessments and forecasts available and useful to resource managers. The priority fish stocks consist of 230 stocks selected for their importance to commercial and recreational fisheries. They are the same stocks tracked under the FSSI. Protected species stocks tracked for this measure are those listed under the MMPA and/or Endangered Species Act, which happen also by coincidence to total 230. There are thus 460 stocks tracked under this measure. This measure combines the number of stock assessments for priority fish stocks and the number of stock assessments and forecasts for protected species to produce a percentage of LMR that tracks the scientific basis for supporting and for evaluating the impact of LMR management actions. The standard of “adequate” is in reference to improving the level of scientific information on a LMR stock to Tier II as described in the Fisheries and Protected Species Stock Assessment Improvement Plans (SAIP) developed by NMFS. To reach this standard, assessments would have to be based on recent quantitative information sufficient to determine current stock status (abundance and mortality) relative to established reference levels and to forecast stock status under different management scenarios.

Data Verification and Validation Summary
Data source Stock assessments reports and Endangered Species Act status reviews
Frequency Quarterly
Data storage NMFS SIS and Excel spreadsheet maintained by NMFS’s Office of Protected Resources
Internal Controls Results will be approved by the NMFS Chief Science Advisor and reported quarterly in a signed memo from the Ecosystem Observations Program Manager to the NMFS CFO and are housed and made available in a database managed by NMFS; quarterly reporting on performance to NOAA Deputy Under Secretary.
Data Limitations Results can only be reported when the SIS is updated with new information from the field.
Actions to be Taken Discussions are ongoing to include protected species in the NMFS SIS.


Performance Measure:
  • Percentage of tools, technologies, and information services that are used by NOAA partners/customers to improve ecosystem-based management

This measure tracks NOAA’s success in translating or synthesizing research findings into information, tools, and technology that improve the use and management of coastal, ocean, and Great Lakes ecosystems. Tracking the accessibility and use of information by target audiences will allow NOAA to identify and expand its most effective programs and products. NOAA partners and customers include federal, state, local, and tribal authorities who make decisions that affect the state of resources in the U.S. coastal zone, and other users whose actions impact the condition of coastal ecosystems (e.g., private industry, school children). Examples of tools include coastal population change data, land cover data, benthic habitat maps, and environmental sensitivity index maps. Technologies refer to the transfer of new or underused approaches for addressing coastal management (e.g., remote sensing, biosensors, Autonomous Underwater Vehicles (AUV), genetic markers for fishery stocks) and resource development (e.g., culture systems for aquaculture, marine pharmaceuticals). This includes the application of technology to coastal resource management through synthesis, integration, training, and the development of new management tools. Information services would include technical assistance, education materials and curricula, extension, and training. Tools or techniques used for modeling or forecasting are measured elsewhere and excluded here.

Data Verification and Validation Summary
Data source NOAA’s line offices (the Office of Atmospheric Research [OAR] and NOS) executing the NOAA programs through the Strategic Plan goal/program structure.
Frequency Annual
Data storage Each line office has an internal secure system for tracking the data contributions.
Internal Controls Use values will be reported by program offices as the number of tools, technologies, and information services (TTIS) used out of the number of TTIS provided. Each line office will report total annual values to a central repository where a single percentage value will be determined and archived in a secure repository. Data are managed in a decentralized system by contributing line offices with validation and verification on any partner for TTIS to ensure no double counting of data.
Data Limitations NOAA needs to ensure tracking systems are secure and data are validated and verified.
Actions to be Taken A secure central NOAA repository for matrixed measures is under development for improved management and tracking purposes.

 

PERFORMANCE OBJECTIVE: Support the Nation’s commerce with information for safe, efficient, and environmentally sound transportation (NOAA)

Performance Measure:
  • Reduce the hydrographic survey backlog within navigationally significant areas (square nautical miles surveyed per year)

NOAA conducts hydrographic surveys to determine the depths and configurations of the bottoms of water bodies, primarily for U.S. waters significant for navigation. This activity includes the detection, location, and identification of wrecks and obstructions with side scan and multi-beam sonar technology and GPS. NOAA uses the data to produce traditional paper, raster, and electronic navigational charts for safe and efficient navigation. In addition to the commercial shipping industry, other user communities that benefit include recreational boaters, the commercial fishing industry, port authorities, coastal zone managers, and emergency response planners. Ships traversing U.S. coastal waters rely on charts based on sounding data that are more than 50-years-old in many places. NOAA has identified approximately 510,000 square nautical miles of the U.S. Exclusive Economic Zone as navigationally significant and in need of resurvey. Since 1994, NOAA has focused primarily on surveying and reporting its accomplishments in the highest priority areas, many of which carry heavy commercial traffic, are less than 30 meters deep, and change constantly. However, this critical area constitutes only a small portion (eight percent) of the entire navigationally significant area used by large commercial vessels and recreational boaters. The square nautical miles reflect data collected within all areas designated as navigationally significant. NOAA’s surveying activities balance in-house resources with private sector contracts and use the latest full-bottom coverage sounding technologies to survey the Nation’s coastal areas for navigation. Weather, mechanical failure, and level of surveying difficulty are variables for both NOAA and its contractors, and therefore variances from the targets of +/- 50 square nautical miles per vessel are to be expected in a normal field season.

Data Verification and Validation Summary
Data source Progress reports on data collected from hydrographic survey platforms.
Frequency Monthly
Data storage NOS maintains hydrographic survey performance data at NOAA’s Hydrographic Surveys Division.
Internal Controls NOS applies its established verification and validation methods. The measure has a +/- 50 square nautical mile variance. Targets are set annually based on resources available; monthly reports on performance to the NOAA Deputy Under Secretary.
Data Limitations NOAA-owned ships and contractor survey assets can be affected by changes in vessel availability or condition. Weather can also affect scheduled surveys.
Actions to be Taken NOS maintains hydrographic survey performance data at NOAA’s Hydrographic Surveys Division.


Performance Measure:
  • Percentage of U.S. counties rated as fully enabled or substantially enabled with accurate positioning capacity

This tracks the progress of NOAA’s Geodesy Program in facilitating the capacity of state and local governments and the private sector to utilize accurate positioning information. NOAA will track county level use of its Online Positioning User Service (OPUS) to determine how well state and local governments are enabled with accurate positioning capacity. Assessing state and local government and private sector usage at the county level is the most appropriate geographic unit. County level assessments offer entire U.S. coverage and an existing infrastructure for addressing spatial issues. Utilizing OPUS is the right indicator for how well a county is enabled with accurate positioning capacity, because its usage requires a high level of positioning sophistication. Further, OPUS is a necessary step in obtaining accurate positions. The level of capacity varies across the Nation. This variation is measured as deficient, sufficiently enabled, and enabled. Deficient capacity to conduct accurate positioning indicates that the county has not demonstrated it has the NOAA-enabled infrastructure, tools, and local capacity needed for accurate positioning. Substantially enabled capacity to conduct accurate positioning indicates the county has demonstrated it has the NOAA-enabled infrastructure, tools, and local capacity needed for accurate positioning. Enabled capacity indicates the county has validated NOAA-enabled infrastructure, tools, and local capacity needed for accurate positioning. This is indicated by having local interaction through, for example, a submitted and accepted OPUS project for inclusion in the NOAA’s geodetic database.

Data Verification and Validation Summary
Data source OPUS
Frequency Quarterly
Data storage Automated database at NOS
Internal Controls NOAA will validate a county’s capacity for local positioning through direct coordination with localities, such as OPUS project acceptance by NOAA. By assessing the user needs of county surveyors, counties, and their associations through successive limited distributions of a county scorecard, NOAA will validate that the geodesy program is meeting local positioning needs; quarterly reporting on performance to NOAA Deputy Under Secretary.
Data Limitations OPUS customer data are limited and will be expanded through Paperwork Reduction Act-approved surveys of customers.
Actions to be Taken None


Performance Measures:
  • Aviation forecast accuracy of ceiling/visibility (1 mi/500 feet to less than 3 mi/1,000 feet) (%)
  • Aviation forecast FAR for ceiling/visibility (1 mi/500 feet to less than 3 mi/1,000 feet) (%)

This measure is based on a 1,000-foot ceiling and three miles of visibility for both accuracy and FAR, and is related to Instrument Flight Rule (IFR) conditions. Visibility and cloud ceiling forecasts are critical for the safety of aircraft operations. Accurately forecasting the transition between Visual Flight Rule and IFR conditions significantly improve general and commercial aviation flight planning capabilities, improving both flight safety and efficiencies.

Data Verification and Validation Summary
Data source NWS field offices
Frequency Monthly
Data storage NWS headquarters and OCWWS
Internal Controls Forecasts and observations are collected from each airport for which NWS issues a forecast. OCWWS stores and quality controls all data, compares forecasts to observations, and computes verification statistics. Forecasters within each WFO are able to stratify verification statistics to his/her personal scores on specific days to learn from recent experience. WFO managers regularly monitor forecast performance. The regional headquarters and OCWWS monitor performance monthly for their respective management areas. All data are reported on to NWS and NOAA leadership on a monthly basis.
Data Limitations Due to the largest volume of data gathered and computed, documentation for this measure cannot be finalized until well into the following fiscal year. Outyear measures depend on a stable funding profile and take into account improved use of the Weather Surveillance Radar 88 Doppler (WSR-88D), new satellites, improved forecast models, new and continued research activities of the USWRP, and investments in critical observing systems and implementation of Advanced Weather Interactive Processing System (AWIPS). Interannual scores tend to fluctuate due to varying weather patterns. Some patterns are more difficult to forecast than others. Year to year variability is +/- 3 percent for both accuracy and FAR. Typically, third and fourth quarter scores during the convective season have lower accuracy scores and increased FARs than the first and second Quarter cool season months.
Actions to be Taken Forecasters within each WFO will continue to monitor their recent past forecast performance to learn from experience. The regional headquarters and OCWWS will continue to monitor performance monthly for their respective management areas. The original measure, Aviation Forecast Accuracy of Ceiling/Visibility (1 mi/500 ft to less than 3 mi/1,000ft); will be changed to Aviation Forecast Accuracy of Ceiling/Visibility Forecasts (3 mi/1,000 ft or less). Similarly, the original measure, “Aviation Forecast False Alarm Rate for Ceiling/Visibility (1 mi/500 ft to less than 3 mi/1,000ft),” will be changed to “Aviation Forecast False Alarm Rate for Ceiling/Visibility (3 mi/1,000 ft or less).”


Performance Measures:
  • Marine wind – percentage of accurate forecasts (%)
  • Wave heights – percentage of accurate forecasts (%)

These measures were originally a “combined accuracy forecast for marine wind and wave.” The measure was revised to reflect the individual wind speed and wave height components. These measures track the accuracy of wind and wave forecasts, which are important for marine commerce.

Data Verification and Validation Summary
Data source NWS field offices
Frequency Monthly
Data storage NWS and NCEP’s Ocean Modeling Branch
Internal Controls Verification is the process of comparing the predicted weather with the actual event. Forecasts and observations are collected from each marine zone for which NWS issues a forecast. OCWWS stores and quality controls all data, compares forecasts to observations, and computes verification statistics. WFO managers regularly monitor forecast performance. The regional headquarters and OCWWS monitor performance monthly for their respective management areas. All data are reported to NWS and NOAA leadership on a monthly basis.
Data Limitations Due to the large volume of data gathered and computed, documentation for the accuracy of forecast for wind and waves cannot be finalized until well into the following fiscal year. Outyear measures depend on a stable funding profile and take into account improved use of the WSR-88D, new satellites, improved forecast models, new and continued research activities of the USWRP, and investments in critical observing systems and implementation of AWIPS. Interannual scores tend to fluctuate due to varying weather patterns. Some patterns are more difficult to forecast than others. Marine wind speed and wave height forecasts scores naturally vary (accuracy +/- 4 percent per year) due to fluctuations in the number of extreme events measured over NWS marine areas per year.
Actions to be Taken NOAA will deploy enhanced versions of AWIPS, upgrade new forecast models, implement new wave forecast models, and improve communication and dissemination techniques to marine users. In FY 2008, the Marine Wind Speed Forecast Accuracy metric (FY 2008 target of 58 percent) will be replaced by a new Marine Wind Forecast Accuracy metric, Percentage of Accurate Forecasts, with a target of 68 percent. In FY 2008, the Marine Wave Height Forecast Accuracy metric (FY 2008 target of 68 percent) will be replaced by a new Marine Wind Forecast Accuracy metric, Percentage of Accurate Forecasts, with a target of 73 percent.

 


Previous Page | Next Page