Appendix C: Performance Measures Definitions
STRATEGIC OBJECTIVE 3.2
Enhance the conservation and management of coastal and marine
resources to meet America’s economic, social, and environmental
needs
PERFORMANCE OBJECTIVE: Protect, restore, and manage the use of coastal
and ocean resources through an ecosystem approach to management (NOAA)
Performance Measure:
- Number of protected species designated
as threatened, endangered, or depleted with stable or increasing population
levels
This measure tracks progress at achieving partial recovery of endangered,
threatened, or depleted protected species under the jurisdiction of the
National Marine Fisheries Service (NMFS) from a baseline of 65 species
established as of January 1, 2004. Protected species are defined as all
marine mammal stocks (except walruses, polar bears, and manatees) and
those domestic non-marine mammal species listed as threatened or endangered
under the Endangered Species Act that are under the jurisdiction of NMFS.
Marine mammal species included in this measure are those listed as “depleted”
under the Marine Mammal Protection Act (MMPA), which includes any listed
under Endangered Species Act. Recovery of threatened, endangered, or depleted
protected species is very slow and can take decades. While it may not
be possible to recover or delist a species in the near term, progress
can be made to stabilize or increase the species. For some, it is trying
to stop a steep decline (right whales, stellar sea lions); for others
it is trying to increase their numbers/abundance (Ridley turtles). NOAA’s
protected species management efforts are focused on halting declines and
conserving species while still allowing human activities to continue.
Data Verification and Validation Summary
Data source |
MMPA stock
assessment reports and Endangered Species Act status reviews |
Frequency |
Annual |
Data storage |
Excel spreadsheet
maintained by NMFS’s Office of Protected Resources. |
Internal Controls |
Results are reported
quarterly in a signed memo from the Protected Species Program Manager
to the NMFS CFO and are housed and made available in a database managed
by NMFS; quarterly reporting on performance to NOAA Deputy Under Secretary. |
Data Limitations |
MMPA stock
assessment reports are updated only once a year and Endangered Species
Act status reviews are updated only every one to five years depending
on priority and fund availability. |
Actions to be Taken |
Discussions
are ongoing to include protected species in the NMFS Stock Information
System (SIS). |
Performance Measure:
- Number of habitat acres restored (annual/cumulative)
Habitat restoration and long-term protection are critically needed to
help maintain the function of important coastal and marine ecosystems.
Under NOAA’s legislative mandates, NOAA protects and restores key
habitats that provide critical ecosystem functions that support the health
of endangered or threatened species, essential fish habitat, as well as
provide a number of other societal or economic benefits. NOAA restores
habitat areas lost or degraded as a result of development and other human
activities, as well as specific pollution incidents and sources. Activities
are geared toward NOAA trust resources found across the marine environment
and supportive of anadromous fish species. This measure summarizes or
projects the geographic area over which ecosystem function has been or
will be improved as the direct result of habitat restoration efforts.
Data Verification and Validation Summary
Data source |
Interim
and final progress reports from each project |
Frequency |
Quarterly |
Data storage |
Restoration Center Database (RCDB) |
Internal Controls |
Results are reported
quarterly in a signed memo from the Habitat Program Manager to the
NMFS CFO and are housed and made available in a database managed by
NMFS; quarterly reporting on performance to NOAA Deputy Under Secretary. |
Data Limitations |
Data are
primarily provided by grantees. |
Actions to be Taken |
None |
Performance Measures:
- Cumulative number of coastal, marine,
and Great Lakes issue-based forecasting capabilities developed and used
for management
- Annual number of coastal, marine,
and Great Lakes ecological characterizations that meet management needs
Sound management of coastal and ocean ecosystems requires scientifically
based information on their condition. NOAA is developing methods to scale
up from the site characterizations it currently produces to ecosystem
characterizations. Characterization includes identification of the physical
location, spatial extent, and biological, chemical, and physical characteristics.
Site characterizations improve understanding of the history, current state,
and future condition of ecosystems, and ecosystem characterizations will
be the cornerstones to ecosystem-based management and the basis for many
coastal, marine, and Great Lakes management tools, including forecasts,
assessments, and management plans. NOAA decides what to characterize based
on: user community priorities, adequacy of indicators, significance of
issue, and consequences of management action/inaction. Characterization
of an ecosystem site (and in the future, a defined subecosystem) is measured
as uncharacterized, substantially characterized, or adequately characterized.
NOAA has initiated a process to divide each of the large marine ecosystems
into subecosystems; ecosystem sites are being used as a proxy unit of
measurement until that effort is completed. (At that point, NOAA will
measure the cumulative number of ecosystems adequately characterized for
management. Currently the measure tracks the progress of 13 National Marine
Sanctuaries (NMS) and 26 National Estuarine Research Reserve Systems (NERRS)
in completing monitoring and assessment to characterize the sites for
ongoing management and long-term protection.
Data Verification and Validation Summary
Data source |
Characterizations
focus on ecosystem sites: NMS, NERRS, coral reef ecosystems, the coastal
zone, Great Lakes, essential fish habitat, ecological species units,
and unexplored areas. |
Frequency |
Annual |
Data storage |
Metadata
from all contributing sources to the measure are maintained by managers
for the coastal and marine resources and ecosystem research programs
and stored in an Excel database with limited access. The final performance
data reported in quarterly and annual performance reports are managed
in a secure NOS database for annual milestones and annual and long-term
performance measures. Changes to reporting data require approval by
the NOS administrator (managed by an email workflow approval system).
|
Internal Controls |
Results are reported
monthly to the Ecosystems Research Program (ERP) program manager and
NOAA CFOs; quarterly reports on performance data are submitted to
the NOAA Deputy Under Secretary. |
Data Limitations |
NOAA focuses
on protected areas or areas where NOAA has a clear management mandate.
NOAA works to identify key parameters for characterizing their conditions
and develop assessments of their present health. Characterizations
from all contributors are being tracked in this new measure in addition
to criteria defining the indicator of what meets management needs
for each ecosystem site because characterizations vary temporally
and geographically. |
Actions to be Taken |
None |
Performance Measure:
- Annual number of coastal, marine,
and Great Lakes habitat acres acquired or designated for long-term protection
NOAA maintains the health of coastal, marine, and Great Lakes habitats
by designating and managing important areas for long-term conservation
and by providing support to state and local governments to protect additional
key habitats by purchasing land from willing sellers. This measure tracks
the number of acres acquired with NOAA funds by state or local government
agencies from willing sellers for long-term protection of important coastal
habitats, or the number of acres designated for long-term protection by
NOAA or by state partners. The protected acres are the actual number of
acres newly protected in a fiscal year. The cumulative total represents
acres acquired or designated to date for NERRS, National Marine Sanctuary
Program (NMSP), and Coastal and Estuarine Land Conservation Program. The
goal for the long-term protection indicator is variable, as the yearly
target can vary from hundreds to thousands of acres each year. For example,
the initial designation or acquisition for a new reserve or sanctuary
may add hundreds of thousands of acres in one year, while in other years
acquisition may result in several hundred or thousand acres protected.
Data Verification and Validation Summary
Data source |
The cumulative
total represents data on acres from the NERRS program, NMSP, and the
Coastal and Estuarine Land Conservation Program. |
Frequency |
Annually by each program manager. |
Data storage |
Metadata
from all contributing sources to the measure are managed by the Coastal
and Marine Resources Program (CMRP) Manager and stored in an Excel
spreadsheet with limited access. The final performance data reported
annually in performance reports are managed in a secure NOS database
for annual milestones and annual and long-term performance measures.
Changes to reporting data require approval by the NOS administrator
(managed by an email workflow approval system). |
Internal Controls |
Results are reported
annually to the contributing NOAA CMRP and NOAA CFOs for approval;
monthly reports on performance data are submitted to the NOAA Deputy
Under Secretary. |
Data Limitations |
The goal
for the long-term protection indicator is variable, as the yearly
target can vary from hundreds to thousands of acres each year. For
example, the initial designation or acquisition for a new reserve
or sanctuary may add hundreds of thousands of acres in one year, while
in other years acquisition may result in several hundred or thousand
acres protected. Other limitations are the timeliness of reporting
by grant recipients, accuracy of conversion from hectares to acres
for some data, and the time delay between funding and completion. |
Actions to be Taken |
Since this
measure does not capture all NOAA’s activities to protect habitat,
NOAA seeks to expand the measure in the future. NOAA is looking at
the feasibility of further harmonizing methodologies used among contributing
program components. |
Performance Measure:
- Fish stock sustainability index (FSSI)
The FSSI replaces the measure “Number of Overfished Major Stocks
of Fish.” The index tracks the outcome of building and maintaining
fish stocks at productive levels while also capturing the critical components
of NOAA’s efforts to get to that outcome, i.e., managing fish harvest
rates and increasing knowledge about the status of fish stocks. The measure
provides a much more complete picture than the old measure of NOAA’s
success at fisheries management. The FSSI is calculated by assigning a
total score between 0 and 4 to each of 230 priority fish stocks (see below).
Each stock receives one point if:
- NOAA has determined whether or not (1) the stock is overfished (one-half
point) and (2) the stock is subject to overfishing (one-half point);
i.e., scientific knowledge is available about the stock;
- NOAA’s management measures are succeeding at ensuring that fishing
does not remove too many fish (i.e., level of fishing mortality does
not exceed the threshold for overfishing);
- The stock is managed at an acceptable level (i.e., biomass is above
the level defined as overfished for the stock);
- The stock is rebuilt or is at its “optimal” level, the
ultimate long term end state for a stock (i.e., biomass is within 80
percent of that required to achieve maximum sustainable yield).
The FSSI is the sum of the scores of the individual stocks. The highest
possible score for a stock is four and for the index is 920, since the
FSSI is based on a set of 230 priority fish stocks selected for their
importance to commercial and recreational fisheries. Criteria for selection
of stocks include whether they are major stocks (landings greater than
200,000 pounds), whether they are overfished or subject to overfishing,
whether they have assessments scheduled, whether they have previously
been identified as important, or other factors as appropriate. These stocks
represent about 90 percent of all commercial landings in the United States.
NOAA plans for this set of stocks to be tracked over a five-year period.
Data Verification and Validation Summary
Data source |
Stock assessments
and status determinations |
Frequency |
Quarterly |
Data storage |
NMFS SIS |
Internal Controls |
Results will be reported
quarterly in a signed memo from the Fishery Management Program Manager
to the NMFS CFO and are housed and made available in a database managed
by NMFS; monthly reporting on performance to NOAA Deputy Under Secretary. |
Data Limitations |
Results
can only be reported when the SIS is updated with new information
from the field. |
Actions to be Taken |
None |
Performance Measure:
- Percentage of living marine resources
(LMR) with adequate population assessments and forecasts
This measure tracks the percent of priority fish stocks and protected
species stocks that have adequate population assessments and forecasts
available and useful to resource managers. The priority fish stocks consist
of 230 stocks selected for their importance to commercial and recreational
fisheries. They are the same stocks tracked under the FSSI. Protected
species stocks tracked for this measure are those listed under the MMPA
and/or Endangered Species Act, which happen also by coincidence to total
230. There are thus 460 stocks tracked under this measure. This measure
combines the number of stock assessments for priority fish stocks and
the number of stock assessments and forecasts for protected species to
produce a percentage of LMR that tracks the scientific basis for supporting
and for evaluating the impact of LMR management actions. The standard
of “adequate” is in reference to improving the level of scientific
information on a LMR stock to Tier II as described in the Fisheries and
Protected Species Stock Assessment Improvement Plans (SAIP) developed
by NMFS. To reach this standard, assessments would have to be based on
recent quantitative information sufficient to determine current stock
status (abundance and mortality) relative to established reference levels
and to forecast stock status under different management scenarios.
Data Verification and Validation Summary
Data source |
Stock assessments
reports and Endangered Species Act status reviews |
Frequency |
Quarterly |
Data storage |
NMFS SIS
and Excel spreadsheet maintained by NMFS’s Office of Protected
Resources |
Internal Controls |
Results will be approved
by the NMFS Chief Science Advisor and reported quarterly in a signed
memo from the Ecosystem Observations Program Manager to the NMFS CFO
and are housed and made available in a database managed by NMFS; quarterly
reporting on performance to NOAA Deputy Under Secretary. |
Data Limitations |
Results can only
be reported when the SIS is updated with new information from the field. |
Actions to be Taken |
Discussions
are ongoing to include protected species in the NMFS SIS. |
Performance Measure:
- Percentage of tools, technologies,
and information services that are used by NOAA partners/customers to
improve ecosystem-based management
This measure tracks NOAA’s success in translating or synthesizing
research findings into information, tools, and technology that improve
the use and management of coastal, ocean, and Great Lakes ecosystems.
Tracking the accessibility and use of information by target audiences
will allow NOAA to identify and expand its most effective programs and
products. NOAA partners and customers include federal, state, local, and
tribal authorities who make decisions that affect the state of resources
in the U.S. coastal zone, and other users whose actions impact the condition
of coastal ecosystems (e.g., private industry, school children). Examples
of tools include coastal population change data, land cover data, benthic
habitat maps, and environmental sensitivity index maps. Technologies refer
to the transfer of new or underused approaches for addressing coastal
management (e.g., remote sensing, biosensors, Autonomous Underwater Vehicles
(AUV), genetic markers for fishery stocks) and resource development (e.g.,
culture systems for aquaculture, marine pharmaceuticals). This includes
the application of technology to coastal resource management through synthesis,
integration, training, and the development of new management tools. Information
services would include technical assistance, education materials and curricula,
extension, and training. Tools or techniques used for modeling or forecasting
are measured elsewhere and excluded here.
Data Verification and Validation Summary
Data source |
NOAA’s
line offices (the Office of Atmospheric Research [OAR] and NOS) executing
the NOAA programs through the Strategic Plan goal/program structure. |
Frequency |
Annual |
Data storage |
Each line
office has an internal secure system for tracking the data contributions. |
Internal Controls |
Use values will be
reported by program offices as the number of tools, technologies,
and information services (TTIS) used out of the number of TTIS provided.
Each line office will report total annual values to a central repository
where a single percentage value will be determined and archived in
a secure repository. Data are managed in a decentralized system by
contributing line offices with validation and verification on any
partner for TTIS to ensure no double counting of data. |
Data Limitations |
NOAA needs
to ensure tracking systems are secure and data are validated and verified. |
Actions to be Taken |
A secure
central NOAA repository for matrixed measures is under development
for improved management and tracking purposes. |
PERFORMANCE OBJECTIVE: Support the Nation’s commerce with information
for safe, efficient, and environmentally sound transportation (NOAA)
Performance Measure:
- Reduce the hydrographic survey backlog
within navigationally significant areas (square nautical miles surveyed
per year)
NOAA conducts hydrographic surveys to determine the depths and configurations
of the bottoms of water bodies, primarily for U.S. waters significant
for navigation. This activity includes the detection, location, and identification
of wrecks and obstructions with side scan and multi-beam sonar technology
and GPS. NOAA uses the data to produce traditional paper, raster, and
electronic navigational charts for safe and efficient navigation. In addition
to the commercial shipping industry, other user communities that benefit
include recreational boaters, the commercial fishing industry, port authorities,
coastal zone managers, and emergency response planners. Ships traversing
U.S. coastal waters rely on charts based on sounding data that are more
than 50-years-old in many places. NOAA has identified approximately 510,000
square nautical miles of the U.S. Exclusive Economic Zone as navigationally
significant and in need of resurvey. Since 1994, NOAA has focused primarily
on surveying and reporting its accomplishments in the highest priority
areas, many of which carry heavy commercial traffic, are less than 30
meters deep, and change constantly. However, this critical area constitutes
only a small portion (eight percent) of the entire navigationally significant
area used by large commercial vessels and recreational boaters. The square
nautical miles reflect data collected within all areas designated as navigationally
significant. NOAA’s surveying activities balance in-house resources
with private sector contracts and use the latest full-bottom coverage
sounding technologies to survey the Nation’s coastal areas for navigation.
Weather, mechanical failure, and level of surveying difficulty are variables
for both NOAA and its contractors, and therefore variances from the targets
of +/- 50 square nautical miles per vessel are to be expected in a normal
field season.
Data Verification and Validation Summary
Data source |
Progress
reports on data collected from hydrographic survey platforms. |
Frequency |
Monthly |
Data storage |
NOS maintains
hydrographic survey performance data at NOAA’s Hydrographic
Surveys Division. |
Internal Controls |
NOS applies its established
verification and validation methods. The measure has a +/- 50 square
nautical mile variance. Targets are set annually based on resources
available; monthly reports on performance to the NOAA Deputy Under
Secretary. |
Data Limitations |
NOAA-owned
ships and contractor survey assets can be affected by changes in vessel
availability or condition. Weather can also affect scheduled surveys. |
Actions to be Taken |
NOS maintains
hydrographic survey performance data at NOAA’s Hydrographic
Surveys Division. |
Performance Measure:
- Percentage of U.S. counties rated
as fully enabled or substantially enabled with accurate positioning
capacity
This tracks the progress of NOAA’s Geodesy Program in facilitating
the capacity of state and local governments and the private sector to
utilize accurate positioning information. NOAA will track county level
use of its Online Positioning User Service (OPUS) to determine how well
state and local governments are enabled with accurate positioning capacity.
Assessing state and local government and private sector usage at the county
level is the most appropriate geographic unit. County level assessments
offer entire U.S. coverage and an existing infrastructure for addressing
spatial issues. Utilizing OPUS is the right indicator for how well a county
is enabled with accurate positioning capacity, because its usage requires
a high level of positioning sophistication. Further, OPUS is a necessary
step in obtaining accurate positions. The level of capacity varies across
the Nation. This variation is measured as deficient, sufficiently enabled,
and enabled. Deficient capacity to conduct accurate positioning indicates
that the county has not demonstrated it has the NOAA-enabled infrastructure,
tools, and local capacity needed for accurate positioning. Substantially
enabled capacity to conduct accurate positioning indicates the county
has demonstrated it has the NOAA-enabled infrastructure, tools, and local
capacity needed for accurate positioning. Enabled capacity indicates the
county has validated NOAA-enabled infrastructure, tools, and local capacity
needed for accurate positioning. This is indicated by having local interaction
through, for example, a submitted and accepted OPUS project for inclusion
in the NOAA’s geodetic database.
Data Verification and Validation Summary
Data source |
OPUS |
Frequency |
Quarterly |
Data storage |
Automated database at NOS |
Internal Controls |
NOAA will validate
a county’s capacity for local positioning through direct coordination
with localities, such as OPUS project acceptance by NOAA. By assessing
the user needs of county surveyors, counties, and their associations
through successive limited distributions of a county scorecard, NOAA
will validate that the geodesy program is meeting local positioning
needs; quarterly reporting on performance to NOAA Deputy Under Secretary. |
Data Limitations |
OPUS customer
data are limited and will be expanded through Paperwork Reduction
Act-approved surveys of customers. |
Actions to be Taken |
None |
Performance Measures:
- Aviation forecast accuracy of ceiling/visibility
(1 mi/500 feet to less than 3 mi/1,000 feet) (%)
- Aviation forecast FAR for ceiling/visibility
(1 mi/500 feet to less than 3 mi/1,000 feet) (%)
This measure is based on a 1,000-foot ceiling and three miles of visibility
for both accuracy and FAR, and is related to Instrument Flight Rule (IFR)
conditions. Visibility and cloud ceiling forecasts are critical for the
safety of aircraft operations. Accurately forecasting the transition between
Visual Flight Rule and IFR conditions significantly improve general and
commercial aviation flight planning capabilities, improving both flight
safety and efficiencies.
Data Verification and Validation Summary
Data source |
NWS field offices |
Frequency |
Monthly |
Data storage |
NWS headquarters and OCWWS |
Internal Controls |
Forecasts and observations
are collected from each airport for which NWS issues a forecast. OCWWS
stores and quality controls all data, compares forecasts to observations,
and computes verification statistics. Forecasters within each WFO
are able to stratify verification statistics to his/her personal scores
on specific days to learn from recent experience. WFO managers regularly
monitor forecast performance. The regional headquarters and OCWWS
monitor performance monthly for their respective management areas.
All data are reported on to NWS and NOAA leadership on a monthly basis. |
Data Limitations |
Due to
the largest volume of data gathered and computed, documentation for
this measure cannot be finalized until well into the following fiscal
year. Outyear measures depend on a stable funding profile and take
into account improved use of the Weather Surveillance Radar 88 Doppler
(WSR-88D), new satellites, improved forecast models, new and continued
research activities of the USWRP, and investments in critical observing
systems and implementation of Advanced Weather Interactive Processing
System (AWIPS). Interannual scores tend to fluctuate due to varying
weather patterns. Some patterns are more difficult to forecast than
others. Year to year variability is +/- 3 percent for both accuracy
and FAR. Typically, third and fourth quarter scores during the convective
season have lower accuracy scores and increased FARs than the first
and second Quarter cool season months. |
Actions to be Taken |
Forecasters
within each WFO will continue to monitor their recent past forecast
performance to learn from experience. The regional headquarters and
OCWWS will continue to monitor performance monthly for their respective
management areas. The original measure, Aviation Forecast Accuracy
of Ceiling/Visibility (1 mi/500 ft to less than 3 mi/1,000ft); will
be changed to Aviation Forecast Accuracy of Ceiling/Visibility Forecasts
(3 mi/1,000 ft or less). Similarly, the original measure, “Aviation
Forecast False Alarm Rate for Ceiling/Visibility (1 mi/500 ft to less
than 3 mi/1,000ft),” will be changed to “Aviation Forecast
False Alarm Rate for Ceiling/Visibility (3 mi/1,000 ft or less).” |
Performance Measures:
- Marine wind – percentage of
accurate forecasts (%)
- Wave heights – percentage of
accurate forecasts (%)
These measures were originally a “combined accuracy forecast for
marine wind and wave.” The measure was revised to reflect the individual
wind speed and wave height components. These measures track the accuracy
of wind and wave forecasts, which are important for marine commerce.
Data Verification and Validation Summary
Data source |
NWS field offices |
Frequency |
Monthly |
Data storage |
NWS and
NCEP’s Ocean Modeling Branch |
Internal Controls |
Verification is the
process of comparing the predicted weather with the actual event.
Forecasts and observations are collected from each marine zone for
which NWS issues a forecast. OCWWS stores and quality controls all
data, compares forecasts to observations, and computes verification
statistics. WFO managers regularly monitor forecast performance. The
regional headquarters and OCWWS monitor performance monthly for their
respective management areas. All data are reported to NWS and NOAA
leadership on a monthly basis. |
Data Limitations |
Due to
the large volume of data gathered and computed, documentation for
the accuracy of forecast for wind and waves cannot be finalized until
well into the following fiscal year. Outyear measures depend on a
stable funding profile and take into account improved use of the WSR-88D,
new satellites, improved forecast models, new and continued research
activities of the USWRP, and investments in critical observing systems
and implementation of AWIPS. Interannual scores tend to fluctuate
due to varying weather patterns. Some patterns are more difficult
to forecast than others. Marine wind speed and wave height forecasts
scores naturally vary (accuracy +/- 4 percent per year) due to fluctuations
in the number of extreme events measured over NWS marine areas per
year. |
Actions to be Taken |
NOAA will
deploy enhanced versions of AWIPS, upgrade new forecast models, implement
new wave forecast models, and improve communication and dissemination
techniques to marine users. In FY 2008, the Marine Wind Speed Forecast
Accuracy metric (FY 2008 target of 58 percent) will be replaced by
a new Marine Wind Forecast Accuracy metric, Percentage of Accurate
Forecasts, with a target of 68 percent. In FY 2008, the Marine Wave
Height Forecast Accuracy metric (FY 2008 target of 68 percent) will
be replaced by a new Marine Wind Forecast Accuracy metric, Percentage
of Accurate Forecasts, with a target of 73 percent. |
|