Appendix C: Performance Measures Definitions

STRATEGIC OBJECTIVE 3.1

Advance understanding and predict changes in the Earth’s environment to meet America’s economic, social, and environmental needs

PERFORMANCE OBJECTIVE: Serve society’s needs for weather and water information (NOAA)

Performance Measures:

Severe weather warnings for tornadoes (county based) – Lead time (minutes)
Severe weather warnings for tornadoes (county based) – Accuracy (%)
Severe weather warnings for tornadoes (county based) – False alarm rate (%)

The lead time for a tornado warning is the difference between the time the warning was issued and the time the tornado affected the area for which the warning was issued. The lead times for all tornado occurrences within the continental United States are averaged to get this statistic for a given fiscal year. This average includes all warned events with zero lead times and all unwarned events. Accuracy is the percentage of time a tornado actually occurred in an area that was covered by a warning. The difference between the accuracy percentage and 100 percent represents the percentage of events without a warning. The false alarm rate (FAR) is the percentage of times a tornado warning was issued but no tornado occurrence was verified.

Data Verification and Validation Summary
Data source	National Weather Service (NWS) field offices
Frequency	Monthly
Data storage	NWS headquarters and the Office of Climate, Water, and Weather Services (OCWWS)
Internal Controls	Verification is the process of comparing the predicted weather to reported event. Warnings are collected from every NWS office, quality controlled, and matched to confirmed tornado reports. Reports are validated by Weather Forecast Offices (WFO) using concise and stringent guidelines outlined in NWS Instruction 10-1605. From these data, verification statistics are computed. OCWWS monitors monthly performance throughout NWS, and the regional headquarters monitor performance within their respective regions. All data are reported on to NWS and National Oceanic and Atmospheric Administration (NOAA) leadership on a monthly basis.
Data Limitations	Only confirmed tornado reports are used to verify tornado warnings. Radar reports are not used. If a tornado occurs but is not reported, it doesn’t go into the database for verification. Therefore, it is possible for tornadoes to be underreported, especially in sparsely populated areas. While long-term performance has shown a steady increase in forecast accuracy, interannual scores tend to fluctuate due to varying weather patterns from year to year. Some weather patterns are more difficult to forecast than others. Forecasters perform better during large outbreaks due a high level of situational awareness, well defined tornadic radar images, and increased confidence based on tornado reports which verify warnings during these large scale events. These three factors lead to longer lead times, higher accuracy, and lower FARs. The peak level of tornadic activity occurs April through June each year. A secondary peak activity time period is October and November in the southeastern United States.
Actions to be Taken	Review all warnings and storm data after each event to learn from past experiences. Use the information learned to improve forecast skill and product quality in the future.

Performance Measures:

Severe weather warnings for flash floods – Lead time (minutes)
Severe weather warnings for flash floods – Accuracy (%)

The lead time for a flash flood warning is the difference between the time the warning was issued and the time the flash flood affected the area for which the warning was issued. The lead times for all flash flood occurrences within the continental United States are averaged to get this statistic for a given fiscal year. This average includes all warned events with zero lead times and all unwarned events. Accuracy is measured by the percentage of times a flash flood actually occurred in an area that was covered by a warning. The difference between the accuracy percentage figure and 100 percent represents the percentage of events without a warning.

Data Verification and Validation Summary
Data source	NWS field offices
Frequency	Monthly
Data storage	NWS headquarters and OCWWS
Internal Controls	Verification is the process of comparing the predicted weather to reported event. Warnings are collected from each NWS office, quality controlled, and matched to confirmed flash flood reports. Reports are validated by WFOs using concise and stringent guidelines outlined in NWS Instruction 10-1605. OCWWS monitors monthly performance throughout NWS, and the regional headquarters monitor performance within their respective regions. All data are reported on to NWS and NOAA leadership on a monthly basis.
Data Limitations	While long-term performance has shown a steady increase in forecast accuracy, interannual scores tend to fluctuate due to varying weather patterns from year to year. Some weather patterns are more difficult to forecast than others. Typically, first and second quarters have higher lead times, while the third and fourth quarters, during the convective season, bring the annual average down. Spring/summer mesoscale events (e.g., thunderstorms) are more difficult to predict than larger synoptic scale systems; hence lower scores are expected in the third and fourth quarters.
Actions to be Taken	Review all warnings and storm data after each event to learn from past experiences. Use the information learned to improve forecast skill and product quality in the future.

Performance Measure:

Hurricane forecast track error (48 hours) (nautical miles)

The public, emergency managers, government institutions at all levels in the United States and abroad, and the private sector use NOAA hurricane and tropical storm track forecasts to make decisions on life and property. This measure calculates the difference between the projected location of the center of these storms and the actual location in nautical miles for the Atlantic basin. The actual is computed by averaging the differences (errors) for all the 48-hour forecasts occurring during the calendar year. This measure can show significant annual volatility. Projecting the long-term trend, and basing outyear targets on that trend, is preferred over making large upward or downward changes to the targets each year.

Data Verification and Validation Summary
Data source	NWS/Tropical Prediction Center (TPC)
Frequency	Annually
Data storage	TPC
Internal Controls	Hurricane storm verification is performed for hurricanes, tropical storms, and tropical depressions regardless of whether these systems are over land or water. TPC issues track and intensity forecasts throughout the life of a hurricane. The actual track and intensity are verified through surface and aircraft measurements. NOAA calculates the average accuracy of the TPC track and intensity forecasts for the Atlantic basin at the end of each hurricane season. Reported errors are for hurricane and tropical storm stages only because of a more limited historical verification record for tropical depressions. All data are reported to NWS and NOAA leadership on an annual basis.
Data Limitations	Verification of actual track and intensity versus forecast is very accurate. However, actual annual scores vary up to 20 percent in some years due to the type and location of the hurricane events. Some types of systems can be more accurately forecasted than others. For example, hurricanes that begin in the northern sections of the hurricane formation zone tend to be much harder to accurately forecast. Outyear measures depend on a stable funding profile and take into account new satellites, improved forecast models, new and continued research activities of the U.S. Weather Research Program (USWRP), and investments in critical observing systems.
Actions to be Taken	NOAA will report on the tracking of forecasts at 24, 48. and 72-hour intervals.

Performance Measure:

Accuracy (%) (threat score) of day 1 precipitation forecasts

This performance measure tracks the ability of the weather forecasters of NOAA’s Hydrometeorological Prediction Center (HPC) to predict accurately the occurrence of one inch or more of precipitation (rain or the water equivalent of melted snow or ice pellets) 24 hours in advance across the contiguous United States. Through this measure, HPC focuses on relatively heavy amounts of precipitation, usually a half inch or more in a 24-hour period (short-term flood and flash flood warnings), because of the major safety and economic impacts such heavy precipitation can have in producing flooding, alleviating drought, and affecting river navigation. These forecasts indicate how much precipitation is expected across the United States, not just whether it will rain or snow. HPC tracks the accuracy of these forecasts using a metric with the statistical name of “threat score” or equivalently “critical success indicator.” This accuracy metric ranges from zero percent, indicating no skill, to 100 percent for a perfect forecast. For example, in verifying the accuracy of a forecast of one inch or more of precipitation for day 1 (the next 24 hours), HPC first determines everywhere in the United States where an inch or more actually fell and was observed by rain gauges. On a given day this occurs only over a very small percentage of the country (although a one inch or more precipitation event is significant for the inhabitants of that particular area). HPC then compares these observed areas of at least one inch of precipitation with the forecasted areas of at least one inch, counting only those points in the United States where HPC forecasted and observed at least an inch as being an accurate forecast. (These points are called “hits.”) Thus, if HPC forecasts one inch to fall at the point representing Washington, DC, and it observed only three-quarters of an inch actually had fallen in that specific area, the forecast is then rated as a miss, even if an inch of rain was observed to have fallen at the points nearby. The overall accuracy score for the country for that particular day 1 forecast is then determined by dividing the total number of correctly forecast points (hits) by the total number of points where HPC had either forecast at least one inch of liquid precipitation or one inch of liquid precipitation had actually occurred. Thus this measure takes into consideration those areas where one inch or more of precipitation was correctly forecast, where it was forecasted but did not occur, and where it occurred but had not been forecasted. To earn a high accuracy score, HPC has to forecast the time, place, and amount of precipitation very well.

Data Verification and Validation Summary
Data source	HPC and state agencies
Frequency	Monthly
Data storage	World Weather Building
Internal Controls	HPC has produced Quantitative Precipitation Forecasts since the early 1960s and has kept verification statistics related to the Quantitative Precipitation Forecast program since that time. HPC forecasters work under the supervisory control of the Senior Branch Forecaster (SBF), who is responsible for the quality and content of all products issued during the shift. The SBF having the additional duty of 24-hour precipitation forecast verification verifies the precipitation forecasts. All data are examined for accuracy and quality control procedures are applied, as described in the Description of Measure section. Verification is the process of comparing the predicted precipitation amounts to the observed amounts over the conterminous United States. All data are reported on to NWS and NOAA leadership on a monthly basis.
Data Limitations	The 40-year record of performance indicates there can be considerable variation in the performance measure from year to year. This variation is heavily dependent on the variation of weather regimes over the course of a year and from year to year. Scores are usually lower, for example, in years with considerable summertime precipitation not associated with tropical cyclones.
Actions to be Taken	NOAA will implement planned weather observation and numerical modeling improvements along with ongoing research projects. The Hydrometeorological Test Bed will be expanded to accelerate the transition of research advancements into the operational prediction of precipitation.

Performance Measures:

Winter storm warnings – Lead time (hours)
Winter storm warnings – Accuracy (%)

A winter storm warning provides NOAA customers and partners advanced notice of a hazardous winter weather event that endangers life or property, or provides an impediment to commerce. Winter storm warnings are issued for winter weather phenomena like blizzards, ice storms, heavy sleet, and heavy snow. These measures reflect advance warning lead time and the accuracy of winter storm events. Improving the accuracy and advance warnings of winter storms enables the public to take the necessary steps to prepare for disruptive winter weather conditions.

Data Verification and Validation Summary
Data source	NWS field offices
Frequency	Monthly
Data storage	The regional headquarters, NWS headquarters, and OCWWS
Internal Controls	Verification is the process of comparing predicted weather to a reported event. Warnings are collected from each NWS office, quality controlled, and matched to confirmed winter storm reports. Reports are validated by WFOs using concise and stringent guidelines outlined in NWS Instruction 10-1605. OCWWS monitors monthly performance throughout NWS, and the regional headquarters monitor performance within their respective regions. All data are reported on to NWS and NOAA leadership on a quarterly basis.
Data Limitations	While long-term performance has shown steady increase in forecast accuracy, interannual scores tend to fluctuate due to varying weather patterns from year to year. Some weather patterns are more difficult to forecast than others.
Actions to be Taken	Review all warnings and storm data after each event to learn from past experiences. Use the information learned to improve forecast skill and product quality in the future.

Performance Measure:

Cumulative percentage of U.S. shoreline and inland areas that have improved ability to reduce coastal hazard impacts

This measure tracks improvements in NOAA’s ability to assist coastal areas by estimating the risks of natural hazards. Activities are underway to develop a coastal risk atlas that will enable communities to evaluate the risk, extent, and severity of natural hazards in coastal areas. The risk atlas will help coastal communities make more effective hazard mitigation decisions to reduce impacts to life and property. Through the coastal risk atlas, National Ocean Service (NOS) provides a mechanism for coastal communities to evaluate their risks and vulnerabilities to natural hazards and improve their hazard mitigation planning capabilities.

Data Verification and Validation Summary
Data source	NOS Coastal Services Center; National Satellite, Data, and Information Service (NESDIS); National Coastal Data Development Center; and other federal and state agencies
Frequency	Annually
Data storage	NOS and NESDIS will collect information, conduct assessments, and store data.
Internal Controls	This measure tracks the cumulative percent of shoreline and inland areas with improved ability to reduce the impact of coastal hazards. In the past, the types of projects included in the reported results differed from one year to the next; therefore, the potential for counting a portion of the shoreline more than once existed. For example, one year a project may improve an area’s ability to reduce the impacts of hurricanes, and then another year a separate project may improve the same area’s ability to reduce the impacts of another coastal hazard, such as inland flooding. To avoid confusion, this measure currently only tracks the development and implementation of the Coastal Risk Atlas. All data used in the Coastal Risk Atlas are quality controlled and the risk assessment methodologies have been peer reviewed with quarterly reporting on performance to NOAA Deputy Under Secretary.
Data Limitations	This measure tracks the development and implementation of the Coastal Risk Atlas as an indicator of improved ability to identify the extent and severity of coastal hazards. Reaching these targets will depend on the activities of other federal and state agencies with management responsibilities in this area.
Actions to be Taken	None

PERFORMANCE OBJECTIVE: Understand climate variability and change to enhance society’s ability to plan and respond (NOAA)

Performance Measure:

U.S. temperature forecasts (cumulative skill score computed over the regions where predictions are made)

Accurate temperature forecasts are critical to many sectors of the national economy, including agriculture and energy utilities. This measure compares actual observed temperatures with forecasted temperatures from areas around the country. For those areas of the United States where a temperature forecast (warmer than normal, cooler than normal, near-normal) is made, this score (Heidke Skill Score) measures how much better the forecast is than the random chance of being correct. Areas where no forecast for surface temperature is made (i.e., areas designated as “equal chance” on the Climate Prediction Center (CPC) seasonal forecast maps) are not included in the computation of this score. The Heidke Skill Score is the metric used for this measure to compare actual and observed temperatures and is one of several accepted standards of forecasting in the scientific community.

The Heidke Skill Score is based on a scale of -50 to +100. If forecasters match a random prediction, the skill score is zero. Anything above zero shows positive skill in forecasting. Given the difficulty of making seasonal temperature and precipitation forecasts for specific locations, a skill score of 20 is considered quite good and means the forecast was correct in almost 50 percent of the locations forecasted.

Data Verification and Validation Summary
Data source	Forecast data, observations from WFOs, and from a cooperative network maintained by volunteers across the Nation.
Frequency	Monthly
Data storage	NWS National Centers for Environmental Prediction (NCEP)
Internal Controls	NOAA performs quality control on the observed data (for example, error checking, elimination of duplicates, and inter-station comparison) both at the CPC and WFO level. In June 2005, NOAA also implemented an objective verification procedure to minimize the impact of human errors in the computation of skill score; monthly reporting on performance to NOAA Deputy Under Secretary.
Data Limitations	Because of natural (and unpredictable) variability of climate regimes, the skill score can fluctuate considerably from one season to another. For example, for the periods influenced by a strong El Niño/Southern Oscillation (ENSO) forcing, Government Performance Results Act (GPRA) measure tends to be high. Lower scores occur during the periods when ENSO is in its neutral phase.
Actions to be Taken	None

Performance Measure:

Reduced the uncertainty in the magnitude of the North American carbon uptake

This measure tracks the uncertainty of atmospheric estimates of the North American carbon uptake by half, assuming a full network of 36 stations has been established and monitored. The uncertainty is estimated on an annual basis, to track progress toward a goal of +/- 0.3 total carbon dioxide emissions (GtC) per year by FY 2009. The baseline uncertainty is +/- 0.6 GtC per year (as determined in 2000). Reducing the uncertainty by 50 percent will allow resolution of the interannual variability in the North American carbon flux and U.S. regional GtCs and uptake.

Carbon dioxide is the most important of the greenhouse gases that are undergoing changes in abundance in the atmosphere due to human activity. On average, about one-half of all the carbon dioxide emitted by human activity is taken up by the oceans and the terrestrial biosphere (trees, plants, and soils), also known as carbon sinks. The variation in the uptake from year to year is very large and poorly understood. A large portion of the variability is thought to be related to the terrestrial biosphere in the Northern Hemisphere, and quite likely North America itself. NOAA needs to assess and quantify the source of this variability if it is to provide scientific guidance to policymakers who are concerned with managing emissions and sequestration of carbon dioxide.

Data Verification and Validation Summary
Data source	NOAA’s Global Carbon Cycle Research Program
Frequency	Annual
Data storage	NOAA’s Earth System Research Laboratory
Internal Controls	Quality assurance and calibration against known standards performed by NOAA.
Data Limitations	Number of tall tower/aircraft sites and NOAA’s ability to incorporate these data into advanced carbon models.
Actions to be Taken	None

Performance Measure:

Reduced the uncertainty in model simulations of the influence of aerosols on climate

Aerosols are liquid or solid particles suspended in the atmosphere. They force changes in the climate system by (1) directly absorbing and scattering of radiation from the sun, and (2) by changing the way clouds reflect back solar radiation. While greenhouse gases warm the atmosphere, aerosols and clouds can both counteract greenhouse gases by reflecting incoming solar radiation and cooling the atmosphere, or, under different conditions, some aerosols can absorb solar radiation and some clouds can trap heat, thus heating the atmosphere. The role of aerosols, clouds, and climate is deemed to be the largest single uncertainty in the prediction of how human activities influence climate change (Intergovernmental Panel on Climate Change [IPCC] 2001). This GPRA measure now addresses the first of the two factors. In later years the second factor will also be included.

Annual targets quantitatively score the success of each of the individual research tasks in preceding years. Success in each of these preceding steps is necessary for success in meeting the 10 percent improvement of uncertainty associated with the 2007 goal and the 15 percent improvement in uncertainty for the 2008 goal.

The desired outcome is an improved science-vetted set of options for changing the impact of North American aerosols on climate, which can be considered by governments, the private sector, e.g., transportation and energy production, and the public. Reductions in the uncertainties surrounding aerosols relate directly to the confidence with which model simulations can support policy decisions on the climate issue. Furthermore, since aerosols are also a human-health, air quality issue, there is the opportunity to quantify “win-win” opportunities of how decisions made to improve air quality may also contribute to reduce the forcing of climate change.

Data Verification and Validation Summary
Data source	NOAA’s Atmospheric Composition and Climate Program
Frequency	Annual
Data storage	NOAA’s Earth System Research Laboratory
Internal Controls	Quality assurance and comparisons against 2001 international assessments by leading experts in the aerosol climate community.
Data Limitations	Number of monitoring sites for vertical distribution of aerosols, process studies that include intensive field campaigns and laboratory-based data, and NOAA’s ability to include these in global models.
Actions to be Taken	None

Performance Measure:

Determine the national explained variance (%) for temperature and precipitation for the contiguous United States using USCRN stations

This measure addresses the significant shortcomings in past and present observing systems by capturing 98 percent of the long-term changes in the national annual average surface air temperature and 95 percent of the long-term changes in the national annual average precipitation throughout the contiguous United States using the U.S. Climate Reference Network (USCRN). Inadequacies in the present observing system increase the level of uncertainty when government and business decisionmakers consider long-range strategic policies and plans. The USCRN, a benchmark climate-observing network, provides the Nation with long-term (50 to 100 years) high quality climate observations and records with minimal time-dependent biases affecting the interpretation of decadal to centennial climate variability and change.

Data Verification and Validation Summary
Data source	NOAA’s National Climatic Data Center
Frequency	Monthly
Data storage	NOAA’s National Climatic Data Center
Internal Controls	Monte Carlo simulations based on operation stations, monthly reporting on performance to NOAA Deputy Under Secretary
Data Limitations	Number of stations commissioned in the USCRN.
Actions to be Taken	None

Performance Measure:

Reduced the error in global measurement of sea surface temperature

This measure documents progress in accurately measuring the global sea surface temperature. The unit of measure is potential satellite bias error (in degrees Celsius) of global sea surface temperature. Bias error is due to a systematic difference between multiple types of observing instrumentation (e.g., satellites and in situ buoys, ships, etc.). The current satellite bias error is 0.53°C (2006). The sea surface, covering over 70 percent of the Earth surface, has a tremendous influence on global climate. It is where the atmosphere responds to the ocean, via the transfer of heat either to or from the atmosphere. Warmer than normal sea surface temperatures in the tropical Pacific is a dominant characteristic of the El Niño phenomenon, and predictive climate models for El Niño must be initialized using the most precise observed surface temperature possible to produce accurate forecasts. Since sea-surface temperature is measured by buoys, ships, and satellites, this performance measure is well-suited as an indicator of the effectiveness of NOAA’s Integrated Ocean Observing System (IOOS). This performance measure also reflects how improvements in ocean observations will decrease the uncertainty in global sea surface temperature measurements, which will ultimately play a role in calculations of the ocean-atmosphere exchange of heat and the heat storage in the global ocean. More accurate estimates of sea surface temperature and ocean heat content will improve ability to respond to changes in the climate system.

Data Verification and Validation Summary
Data source	NOAA’s Office of Climate Observations
Frequency	Quarterly
Data storage	Pacific Marine Environmental Laboratory
Internal Controls	Quarterly reporting mechanism on uncertainty in sea surface temperature measurements, quarterly reporting on performance to NOAA Deputy Under Secretary.
Data Limitations	Number of deployed observing platforms in the global ocean.
Actions to be Taken	None

Performance Measure:

Improve society’s ability to plan and respond to climate variability and change using NOAA climate products and information

This measure documents the success in working with stakeholders to develop and enhance a suite of climate data, monitoring, and prediction products that are valuable to customers and stakeholders. The unit of measure is: regionally-focused climate impacts and adaptation studies communicated to decisionmakers. NOAA currently provides state of the art science and discovery information products to a range of decisionmakers, from water resource managers and regional forecast offices, to national and international assessments. These information summaries highlight important deliverables such as reducing uncertainty in climate forcing models, and in seasonal, interannual, and decadal climate forecasts. These deliverables form the basis of NOAA’s emerging climate products and services. NOAA requires stakeholder input and feedback for product development and improvement. These interactions are facilitated by both interdisciplinary research and NOAA operations, bridging the gap between research and production, and decisionmakers. By increasing the interactions between NOAA and the users of climate information, NOAA ensures that climate products and services reach the key decision-making sectors.

Data Verification and Validation Summary
Data source	NOAA’s Office of Global Programs
Frequency	Annual
Data storage	NOAA’s Climate Program Office
Internal Controls	Annual examination of grants awarded and research activities undertaken that result in various outputs (e.g. peer review publications, workshops) showing evidence of research-based interactions with decisionmakers.
Data Limitations	Challenge of systematically collecting research-based outputs showing evidence of interactions with stakeholders to communicate risks of climate variability and change and to develop means of coping with impacts.
Actions to be Taken	None

Previous Page | Next Page