Appendix C: Performance Measures Definitions

STRATEGIC OBJECTIVE 2.1

Develop tools and capabilities that improve the productivity, quality, dissemination, and efficiency of research

PERFORMANCE OUTCOME: Promote innovation, facilitate trade, and ensure public safety and security by strengthening the Nation’s measurements and standards infrastructure (NIST)

Performance Measure:

Qualitative assessment and review of technical quality and merit using peer review

Since 1959, the National Research Council (NRC) has annually reviewed the National Institute for Standards and Technology (NIST) Laboratories. The annual NRC Board on Assessment (BOA) of NIST programs review is independent, technically sophisticated, and extensive. The assessment process focuses on the quality, relevance, and technical merit of the NIST Laboratories program to ensure it is developing and promoting the infrastructure tools and measurement standards needed by industry, academia, and other government agencies. The review board consists of approximately 150 scientists and engineers organized into seven panels (one for each of the seven NIST Laboratories), plus two sub-panels for specialized programs. Each year the laboratory-specific panels conduct a two to three-day on-site review of each laboratory’s technical quality. This measure simply reflects whether NRC conducted the review.

Data Verification and Validation Summary
Data source	On-site interviews and discussions with NIST management and research staff by independent external scientific and technical experts, managed by the NRC.
Frequency	Annual reviews, biennial reports
Data storage	NRC
Internal Controls	Oversight of laboratory-specific expert review panels provided by the NRC BOA of NIST programs.
Data Limitations	Data are qualitative in nature.
Actions to be Taken	None

Performance Measure:

Peer-reviewed technical publications produced

Technical publications represent a way NIST transfers the results of its research and provides measurements and standards to those in industry, academia, and other government agencies. Each year, NIST produces between 2,000 and 2,200 manuscripts and publications with approximately 50 to 60 percent appearing in prestigious scientific peer-reviewed journals. This measure is a direct count of NIST technical manuscripts that have been reviewed and approved for publication in peer-reviewed journals by the NIST Editorial Review Boards at both the Gaithersburg and Boulder sites, and the number of approved manuscripts published in peer-reviewed journals in that fiscal year. A peer-reviewed journal is a publication in which articles are formally reviewed by the journal’s editors and/or a panel of experts and respected researchers in a specific field of study before being accepted for publication. This measure reflects in part the quality and demand for NIST publications. In addition to peer-reviewed journals, NIST publishes its measurement methods and standards through conference proceedings, NIST interagency reports, and special publications.

Data Verification and Validation Summary
Data source	NIST Office of Information Services
Frequency	Ongoing
Data storage	Publications data are gathered and maintained by NIST Office of Information Services.
Internal Controls	Data represent direct and verifiable counts of NIST technical manuscripts to be published in peer-reviewed journals and have been cleared for publication by the internal Washington and Boulder Editorial Review Boards. Internal controls include verification using random checks of review paper work with approved publications by the Intelligent Systems Division (ISD) staff and data review by the NIST Director’s Office.
Data Limitations	Output only
Actions to be Taken	None

Performance Measures:

Standard Reference Materials (SRM) sold
NIST-maintained datasets downloaded
Number of calibration tests performed
Citation impact of NIST-authored publications

These four measures share the same methods of validation and verification. SRMs are the definitive source of measurement traceability in the United States; all measurements using SRMs can be traced to a common and recognized set of basic standards that provides the basis for compatibility of measurements among different laboratories. SRMs are certified in the NIST Laboratories for their specific chemical and material properties. Customers use SRMs to achieve measurement quality and conformance to process requirements that address both national and international needs for commerce and trade and public safety and health.

NIST provides online access to over 80 scientific and technical databases. These databases cover a broad range of substances and properties from a variety of scientific disciplines. Some datasets, such as the NIST Chemistry WebBook, NIST Physical Reference Data Systems, and the NIST Ceramics WebBook, are comprehensive and contain a large number of databases, while others serve very specific applications. Industry, academia, other government agencies, and the general public use NIST’s online data systems with this measure representing another method NIST uses to deliver its measurements and standards tools, data, and information. This measure is a direct count of the average annual number of downloads of NIST-maintained data.

NIST offers more than 500 different types of physical calibrations in areas as diverse as radiance temperature, surface finish characterization, and impedance. NIST calibration services provide the customer with direct traceability to national and international primary standards. This measure illustrates the quantity of physical measurement services provided by NIST for its customers, including calibration services, special tests, and Measurement Assurance programs. The output data represent a direct count of calibration tests performed.

The citation impact measure demonstrates that NIST consistently produces relevant scientific and technical publications. Citation impact reflects the utility and relevance of NIST research and is outcome-oriented. Citation impact has remained consistently above average for the past 26 years (1981-2006).

Data Verification and Validation Summary
Data source	NIST Technology Services
Frequency	Ongoing
Data storage	NIST Technology Services
Internal Controls	Data represent direct and verifiable counts of: (1) the number of SRMs sold to customers, (2) the number of times a NIST-maintained dataset has been downloaded, and (3) items of calibration tests performed by the NIST Laboratories. Internal controls include verification and review by NIST Technology Services and the NIST Director’s Office and Budget Division.
Data Limitations	Data provide information on output levels only. NIST measure “NIST-maintained datasets downloaded” reflects the number of users accessing these datasets; it does not reflect unique users or capture how the data were used.
Actions to be Taken	None

PERFORMANCE OUTCOME: Accelerate private investment in and development of high-risk, broad-impact technologies (NIST)

Performance Measures:

Cumulative number of publications
Cumulative number of patents
Cumulative number of projects with technologies under commercialization

These three measures reflect the outcomes of the Advanced Technology Program (ATP). Publications indicate the diffusion of technical knowledge that results from ATP investment in the development of new technologies, and participants in more than half of ATP-funded projects have published and presented papers in technical professional journals. The cumulative count of publications generated by all ATP-funded research through the close of a given fiscal year represents a major channel for the diffusion of technical knowledge that results from ATP funding.

Patents focuses on the creation of new knowledge resulting from ATP-funded projects, tracking the cumulative direct count of the number of patents filed by all ATP-funded research project participants through the close of a given fiscal year.

Projects under commercialization tabulates the cumulative number of projects with new technologies under commercialization that are traceable to all ATP-funded projects through the close of a given fiscal year. The measure indicates the extent to which ATP-funded research and development has either leveraged or catalyzed new products or services, which in turn improve the prospects for technology-led economic growth.

Data Verification and Validation Summary
Data source	Data are gathered from the portfolio of ATP project participants (funded since 1993) through company filings of patent information to the NIST Grants Office (a legal requirement) and an electronic survey instrument under ATP’s Business Reporting System (BRS). Separate portfolio-based telephone surveys are conducted of project participants funded prior to 1993 and for post-project data collection.
Frequency	Annual over the course of ATP funding for projects funded since 1993; intermittent for projects funded prior to 1993; every two years (up to six years) after ATP funding ends.
Data storage	ATP’s Office of Economic Assessment maintains BRS data in an integrated set of databases covering both descriptive information about the funded organizations and survey responses for all participants in ATP-funded research projects.
Internal Controls	All ATP reports using BRS data and patent reports filed through the NIST Grants Office are monitored closely by ATP for research quality and are subject to extensive NIST-wide review and critique prior to being issued.
Data Limitations	The BRS electronic survey and other telephone survey instruments represent a standardized reporting system. Standard sources of uncertainty include variation in interpretation of specific questions, variation in the estimation techniques used in response to specific questions, variation in the quality of industry data, and missing values.
Actions to be Taken	None

PERFORMANCE OUTCOME: Raise the productivity and competitiveness of small manufacturers (NIST)

Performance Measures:

Number of clients served by Hollings MEP centers receiving federal funding
Increased sales attributed to Hollings MEP centers receiving federal funding
Capital investment attributed to Hollings MEP centers receiving federal funding
Cost savings attributed to Hollings MEP centers receiving federal funding

The Manufacturing Extension Partnership (MEP) works with the Nation’s small manufacturing firms to provide assistance to overcome barriers to productivity growth and competitiveness. These measures provide quantitative indicators of the bottom-line impacts MEP services provide. The number of clients represents the annual number of new and repeat clients served by MEP centers who received training, technical, and business assistance ranging from informational seminars and training classes to in-depth technical assistance in areas such as lean implementation, ISO 9000, and quality improvement practices. Increased sales, capital investment and cost savings indicate changes that are positively associated with productivity growth and competitiveness—two factors that are crucial for U.S. manufacturers to manage and succeed in the rapidly changing manufacturing environment. Data are collected through an annual survey of clients receiving services from MEP centers.

Data Verification and Validation Summary
Data source	The client impact survey is administered by a private firm, Synovate, located in Arlington Heights, IL.
Frequency	The survey is conducted four times per year, and clients are selected based on when they completed the first project with an MEP center in the previous year. For example, a client that completed a project with an MEP center in February 2004 was surveyed in January/February 2005. This process is used to reduce respondent burden, raise overall response rates, and improve data quality. Clients are asked to estimate how the group of MEP-provided services over the previous two years has affected their business performance in the 12-month period prior to the survey date.
Data storage	Survey data are sent directly to MEP for analysis. MEP reviews and stores survey data received from Synovate.
Internal Controls	Internal controls include review of the Synovate data by MEP staff. Criteria are in place for identifying outliers in the data. Centers verify the outlier and if necessary, the data are revised based on the center review.
Data Limitations	Sources of uncertainty include variation in interpretation of specific questions, variation in the estimation techniques used in response to specific questions, variation in the quality of industry data, missing values, and other common survey problems. Synovate uses standard survey techniques to clean the data, ensure accuracy and reliability, and improve the response rate. Reported data reflect the impact of MEP services primarily on small manufacturing establishments; on some occasions, centers may elect to serve establishments with over 500 employees.
Actions to be Taken	None

PERFORMANCE OUTCOME: Enhance public access to worldwide scientific and technical information through improved acquisition and dissemination activities (NTIS)

Performance Measure:

Number of updated items available (annual)

The number of items available for sale to the public from the National Technical Information Service (NTIS) includes scientific, technical, and engineering information products added to the permanent collection, as well as items made available through online electronic subscriptions. Each publication added to the permanent collection is abstracted, catalogued, and indexed so that it can be identified and merged into the permanent bibliographic database for future generations of researchers and the public who may benefit from this valuable research. Other information products are available as full text documents in electronic format through numerous NTIS online information services. This material is acquired primarily from U.S. government agencies, their contractors and grantees, and also from international sources. The number of new information products available each year from NTIS is approximately 665,000, but the number largely depends on input from other government agencies.

Data Verification and Validation Summary
Data source	NTIS operates and maintains internal systems for collecting acquisition statistics.
Frequency	Data are available daily. Reports are produced monthly.
Data storage	All data are stored within NTIS systems.
Internal Controls	NTIS accounting and budget offices analyze and report performance data to management. Data verification is provided through regular internal independent auditor reporting.
Data Limitations	Output only
Actions to be Taken	None

Performance Measure:

Number of information products disseminated (annual)

This measure represents information disseminated and includes compact discs, diskettes, tapes, online subscriptions, Web site pages, as well as traditional paper and microfiche products. The shift in information dissemination practices from traditional paper copy to electronic-based dissemination has improved NTIS’s ability to provide quality products, increase the number of products distributed, and increase the number of customers that have access to valuable scientific and technical information.

Data Verification and Validation Summary
Data source	NTIS records every transaction using a commercial order processing system modified to meet its specific needs together with a standard Web analysis software package used by industry.
Frequency	Internal management activity reports are produced daily, summaries are produced monthly.
Data storage	All data are stored within NTIS systems.
Internal Controls	NTIS accounting and budget offices analyze and report performance data to management. Data verification is provided through regular internal independent auditor reporting.
Data Limitations	Output only
Actions to be Taken	None

Performance Measure:

Customer satisfaction

This measure represents the percentage of NTIS customers who are satisfied with the quality of their order, the ease of order placement, and the timely processing of that order. Orders are received by phone, fax, mail, and online, and are filled in a variety of formats. The percentage of satisfied customers is derived from the number of customer complaints compared to the total number of orders taken. It does not take into account inquiries about the status of an order or other general questions.

Data Verification and Validation Summary
Data source	NTIS operates and maintains internal systems for processing collected information. NTIS records every transaction using a commercial order processing system modified to meet its specific needs.
Frequency	Internal management activity reports are produced daily, summaries are produced monthly.
Data storage	All information is stored within NTIS systems.
Internal Controls	NTIS accounting and budget offices analyze and report performance data to management. Data verification is provided through regular internal and independent auditor reporting.
Data Limitations	None
Actions to be Taken	None

Previous Page | Next Page