Open Government Plan - Transparency

Open Government Plan - Transparency (Why Commerce needs to do this)

III.    Transparency

“Transparency promotes accountability and provides information for citizens about what their government is doing. Information maintained by the Federal Government is a national asset.” 

President Barack Obama

Unlocking public access to government data is a critical component of the President’s Open Government Initiative because it can help drive innovation by tapping into the ingenuity of the American people, increase agency accountability, and create a standard for government that is open, transparent and participatory.  Transparency is imperative to a government that is accountable to its people.  

This principle particularly applies to Commerce, given its partnerships with state, local and tribal governments, educational and scientific institutions, nonprofit organizations, and for-profit businesses this principle is particularly relevant to Commerce.  From the amount of rainfall measured by NOAA’s National Weather Service (NWS) to the complex portrait of the nation developed every 10 years by the Census Bureau, data collection and dissemination is a vital aspect of what the Department does.  As a result, Commerce – as a whole – must be fully committed to making its data sets more broadly accessible through the government-wide initiative.  It is also prepared to serve as a leader in the Federal Government’s effort to make more raw data and tool sets of packaged data available through the website and other data repositories.

Open Government Plan - Transparency (What Commerce Has Done)

III. Transparency (con.)

What Commerce Has Done

Even before the issuance of the Open Government Directive, Commerce had published over 104,000 data sets and/or data tools on and, in the last 120 days, it has released over 60,000 more.  While the vast majority of the data sets involve Census geodata that will allow the public to map other bits of information, Commerce’s 166,512 data sets also include 40 tools that should make other sets easier to use and 18 different sets of raw, machine readable collections of information. 

The following table provides additional detail about the data the Department is making available on, the operating units that have provided or have responsibility for them, and how often they were downloaded during the week beginning March 28, 2010.

Commerce Data Sets Released to the Public

Agency / Operating Unit

Raw Datasets




Number of Times Downloaded
(3/28/10 – 4/3/10)


18 (4)

40 (2)





























11 (1)






1 (1)






1 (1)












1 (1)

28 (2)




In many cases, the data Commerce has made available on over the last year was already available elsewhere online, but in a larger number of cases data was not available in the raw format that developers and other members of the public find most useful.

Open Government Plan - Transparency (What Commerce Will Do )

III. Transparency (Con.)

What Commerce Will Do

Throughout its history, the Department of Commerce has published high value data as part of its scientific, technological and economic programs.  This has allowed Commerce to establish best practices in distribution and publication processes that meet the ever-evolving needs of the public.  It has established a robust network of content owners and technology specialists to assist it in further converting existing data to machine-readable, high-value data sets for use by the public, and in retaining its leadership role across the government in information dissemination.  The Department will also expand its use of developing technologies to make information about its operations and decisions readily accessible online, and to solicit input from the public to help identify information that will be of greatest use or interest prospectively.   

Commerce has launched a series of projects to increase high-value information that is readily accessible by the public. Information concerning 17 of the projects currently underway across Commerce to improve transparency is provided below.

Projects to Increase
Data Availability and Accessibility




Improving Dissemination of Basic Research Results via Web and Social Media

Improving Access to the Digital Data Repository of NIST Collections, including Publications, Artifacts, and Photographs Relating to Measurement Science


Modernizing the NOAA Climate Database

Severe Weather Data Inventory

Ocean Surface Current Simulator

San Francisco Exploratorium

U.S. Drought Portal – Addition of Soil Moisture Observation Data

Historical Climate Reanalysis Project

NOAA Climate Services Portal


Broadband Mapping

Broadband Survey


Making 5 years of Bibliographic Data Searchable


Enhancing Patent Maintenance Fee Events Data

Expansion of Patent Bibliographic Data

Enhancing Existing USPTO Data Capabilities Available to the Public

Office of the Secretary

Public Schedule Data for Secretary

Streaming Video for More Meetings

National Institute of Standards and Technology (NIST)

  • Project – Improving Dissemination of Basic Research Results via Web and Social Media

What’s New – Tagging content to ease search, simplifying public feedback process

As part of an effort to disseminate more broadly its research results, NIST will implement a new website design based on a content management system. This system includes access to an improved database of research papers authored or co-authored by NIST researchers. Content posted on the new website will be “tagged” by topic, enabling the public to subscribe to receive new information posted on the website on specific topics of interest such as nanotechnology or energy-related research.  The new website will also allow the public to comment or ask questions about posted research articles and to share easily content from the NIST site with their own websites.  NIST has recently created YouTube, Facebook, and Twitter sites as well.  To ensure that as many people as possible benefit from NIST’s work, news of major research results posted on the new NIST website will be routinely announced through these additional social media sites.

  • Project – Improving Access to the Digital Data Repository of NIST Collections, including Publications, Artifacts, and Photographs Relating to Measurement Science

What’s New – Using Open Archives Protocol to allow automatic harvesting by major search engines and research repositories

Currently, information regarding NIST’s publications is electronically available through its Research Library’s online catalog, which includes links to the full text of many publications. Information about some of the objects in NIST’s museum is also available through the NIST Virtual Museum (NVM).  The online catalog and the NVM are available to the public.  In Fiscal Year (FY) 2010, NIST will implement a digital library repository.  This repository will conform to the latest library and publishing metadata standards to enhance the ability of other scholarly and research repositories to discover and harvest information.  The repository will contain the full text of NIST’s technical publications, including the Journal of Research, as well as images of and information about historical scientific objects that it maintains. The metadata will conform with the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH), which is the accepted standard within scholarly and scientific communities for making the contents of information collections available to researchers.  File formats will adhere to Government Printing Office, Library of Congress, and National Archives preservation requirements.  The repository will permit the digital forms of NIST’s technical publications and other content to be easily searchable by the public through major Internet search engines, such as Google, Google Books, Google Scholar, WorldCat, and Yahoo.  This will significantly enhance publication and distribution of NIST’s research results.

National Oceanic and Atmospheric Administration (NOAA)

  • Project – Modernizing the NOAA Climate Database

    What’s New – Digitizing data from weather stations collected in the 18th and 19th centuries

    The Climate Data Modernization Program (CDMP) supports NOAA’s responsibility to collect, integrate, assimilate and effectively manage Earth observations on a global scale, ranging from atmospheric, weather, and climate observations to oceanic, coastal, and marine life observations.  Many of these data were originally recorded on paper, film, and other fragile media.  Prior to CDMP, not only were these valuable data sources mostly unavailable to the scientific community, but storage technology for the archive had become obsolete.  Today, CDMP has greatly improved the preservation of and access to NOAA’s holdings by migrating many of these resources to new digital media.  CDMP has placed online over      53 million weather and environmental images that are now available to researchers around the world via the Internet.  The amount of data online has grown from 1.75 terabytes in 2001 to over 11 terabytes in 2009.  Hourly weather records digitized through CDMP continue to be integrated into NOAA’s digital database holdings, extending the period of record for many stations back into the 1890’s.  Additional daily data records digitized through the CDMP will extend this data period back to the 18th Century for several weather stations.
  • Project – Severe Weather Data Inventory (SWDI)

What’s New – Simplified access to current and past information about severe weather incidents

The SWDI at NOAA’s National Climatic Data Center (NCDC) provides users access to archives of several data sets critical to the detection and evaluation of severe weather.  These data sets include:

- Next Generation Radar – or NEXRAD – Level-III point features describing general storm structure, hail, mesocyclone and tornado signatures,

- NWS local storm reports collected from storm spotters,

- NWS warnings, and

- Lightning strikes from Vaisala’s National Lightning Detection Network.

SWDI archives these data sets in a spatial database that permits convenient searching.  These data are accessible via the NCDC website, FTP or automated Web services.  The results of interactive Google Maps-based Web page queries may be saved in a variety of formats, including plain text, XML, Google Earth’s KMZ and Shapefile.  Summary statistics, such as daily counts, allow efficient discovery of severe weather events.  More information may be obtained at

  • Project – Ocean Surface Current Simulator

    What’s New – Upgrading the ability to visualize changes in ocean surface currents

    The Ocean Surface Current Simulator (OSCURS) numerical model is a research tool that allows oceanographers and fisheries scientists to perform retrospective analyses of daily ocean surface currents anywhere in an ocean-wide grid of 90 km cells that stretches from Baja California to China and from 10 degrees north of the equator to the Bering Strait.  The model is used to measure the movement of surface currents over time, as well as the movement of what is in, or on, the water.  Ocean surface currents affect organisms suspended in the water column—such as fish eggs, small larvae, and plankton–and may affect their survival by determining their location after a few months of drift.  Even swimming or migrating fish or mammals may have their destinations significantly offset by currents or the annual variability of currents.  OSCURS has gained visibility as an accidental debris tracker to analyze accidental but fortuitous at-sea events beyond the scale of normal oceanographic science.  Investigations of events such as spills of cargo containers loaded with plastic bathtub toys have been used to fine-tune the OSCURS model.

    The model has been served for many years by a Live Access Server (LAS) at NOAA and has been used heavily. However, LAS requires the outdated Netscape browser and only allows the user to visualize and download one OSCURS run at a time.  Data serving technology has greatly improved, and NOAA is developing a new interface to serve the OSCURS model ( that uses Google Maps as the visualization tool and the latest in AJAX technology to improve users’ experience.  Users will be    able to visualize many runs at a time and possibly view other relevant environmental data using the same interface.  This project should be ready for public use by the end of calendar year 2010.
  • Project – San Francisco Exploratorium

    What’s New – Near real-time ability to visualize weather and water conditions in               San Francisco Bay

    NOAA’s National Marine Fisheries Service is developing a new way to visualize regional data in the San Francisco Bay (  Data from shore stations, buoys, high-frequency radar, and satellites are available, but scattered among many Web pages and stored in many formats.  It is difficult for regional and public interests in the      San Francisco Bay area to visualize and use this system to assess real-time conditions.  As a demonstration tool to support NOAA’s new partnership with the renowned science museum, the Exploratorium, and in collaboration with the Central and Northern California Ocean Observing System and other regional data providers, NOAA is developing a Web page to make it easy to visualize near-real time data in San Francisco Bay.  The interface will use Google Maps and the latest AJAX technology to combine and compare data from diverse sources.  Users will be able to visualize water temperature, salinity, and other station-based measurements along with overlays of satellite measurements of Sea Surface Temperature (SST) and radar measurements of currents.  Users will also be able to compare time series of measurements from various stations and sources.  Model data and animations will be added as they become available. 

    A public version of the project will be ready by the end of 2010, and will be continually developed with new data included as they become available.
  • Project – U.S. Drought Portal – Addition of Soil Moisture Observation Data

What’s New – Making public for the first time soil moisture observation data

Timely recognition of drought risks depends on the ability to monitor and forecast the diverse physical indicators of climatological drought, as well as relevant economic, social, and environmental impacts.  A 2004 report from the Western Governors’ Association makes it clear that recent and ongoing droughts underscore the critical need for a coordinated, integrated drought monitoring, forecasting, and early warning information system.  To fill this need, Congress passed the National Integrated Drought Information System Act of 2006 (Public Law 109-430) (NIDIS).  The first component of NIDIS is the Drought Portal (, which is part of an interactive system to:

- Provide early warning about emerging and anticipated droughts,

- Assimilate and control the quality of data about droughts and models,

- Provide information about the risk and impacts of droughts to different agencies and stakeholders,

- Provide information about past droughts in order to compare them with and to better understand current conditions,

- Explain how to plan for and manage the impacts of droughts, and  

- Provide a forum for different stakeholders to discuss drought-related issues.

During the first quarter, FY2010, the Drought Portal was expanded to include soil moisture observation data from the U.S. Climate Reference Network, which had not been previously available to the public.

  • Project – Historical Climate Reanalysis Project

What’s New – Re-launching and expanding access to data sets describing past weather

The Historical Climate Reanalysis Project uses a three-dimensional globally-complete climate model and available weather observations to produce output fields of weather variables measured four times daily from 1871, to the present.  Using what are often, especially in earlier years, sparse data sets of observations, the Project is able to reconstruct past weather and fill in missing data values around the rest of the globe.  These data will be available through a number of different types of Web-based, interactive plotting pages as well as file download.  In addition to generating plots, users will be able to conduct basic analyses of data, download subsets of data, and obtain data in Google Earth format, which will permit easy visualization by the general public using the Google Earth application.  Currently, the data are available at

the Physical Sciences Division of NOAA’s Earth Systems Research Laboratory only in “grib” format, which is difficult to read and not available for online plotting and analysis.  The complete dataset itself is well over four terabytes, so examining even portions of it requires enormous storage space and computing resources.  

By enabling the public to work with the data and data products online, NOAA will allow users to examine past weather and climate events in a detailed way never before possible. Version 1 of the Project is available today at  It spans, however, only the years 1908 through 1958 and does not include the interactive plotting tools described above.  NOAA expects Version 2 to include data for 1891, to the present, provide online plotting and analysis tools, and be available online during the second quarter of calendar year 2010.

  • Project – NOAA Climate Services Portal

What’s New – Enhanced Access to NOAA Climate Information

NOAA is enhancing its climate information Web presence in response to customer requirements, emerging needs for improved decision-making capabilities across all sectors of society facing impacts from climate variability and change, and the value of leveraging climate data and services to support research and public education.  NOAA has initiated development of a NOAA Climate Services Portal (NCS Portal) with the goal to become the “go-to” website for NOAA’s climate data, products, and services for all users.  The NCS portal is currently available as a prototype, which only scratches the surface of the many climate datasets, products, and services available across NOAA.  The prototype highlights some of most popular datasets and products based on customer usage of the data, focused on several datasets and products from NOAA’s National Climatic Data Center, Coastal Services Center, and Climate Prediction Center, among others.  NOAA plans to gather user feedback through focus groups, usability studies, and informal communications.  Over the next several years, NOAA will expand the NCS Portal’s scope and functionality in a user-driven manner to enhance the access to, and extensibility of, climate data and services, timely articles and information, education resources, and tools for engagement and decision-making.

National Telecommunications and Information Administration (NTIA) 

NTIA is embarking on a series of data collection and dissemination initiatives to provide a more detailed, quantitative understanding of broadband Internet access and use in the United States. This information will inform efforts to increase broadband access and adoption, thereby supporting economic growth.  Initiatives will include data collected through NTIA’s broadband mapping program and a new broadband-related survey.

  • Project – Broadband Mapping

What’s New:  National, interactive map showing broadband availability and speeds

Through its State Broadband Data and Development Grant Program, funded through the American Recovery and Reinvestment Act, NTIA awards financial assistance grants for broadband data collection and planning.  Data will be displayed in NTIA’s national broadband map, which will be made publicly available by March 2011.  The map will display the geographic areas where broadband service is available and its speed, as well as the technology by which it is provided, and its availability at public schools, libraries, hospitals, colleges, universities, and public buildings.  The national map will be interactive and searchable by address, and will identify broadband service providers by census block or street segment.  Data collection began in 2009, and will continue to take place semiannually between now and 2011. Data will be presented in a clear, accessible, and open format to the public, government, and research community.  This new initiative will provide broadband information at an unprecedented level of comprehensiveness and granularity.

  • Project – Broadband Survey

What’s New – Resuming use of the Census Bureau’s periodic Current Population Survey to study Internet usage

Working with the Census Bureau, NTIA launched a 75,000-household Internet-use survey via the October 2009 Current Population Survey.  Through this effort, NTIA will examine why people do not use high-speed Internet service and explore differences in Internet usage patterns around the country and across socio-economic groups.  NTIA intends to release data in open, Web-based formats and to make the survey instruments and associated reports as  widely available as possible.

National Technical Information Service (NTIS)

  • Project – Making Five Years of Bibliographic Data Searchable

What’s New – Making 180,000 records describing federal reports available in XML format

NTIS is making the latest five years of the NTIS Bibliographic File searchable via The file contains over 180,000 bibliographic records that link to a Web-store of federally funded technical reports from a broad spectrum of federal agencies.  This bibliographic file is being made available through in a compiled XML format, which will – for the first time – fully open access to NTIS’ technical reports collection to Web exposure and extraction.  NTIS will measure the effect of increased exposure via by comparing future ordering information to existing baseline data.  The increased exposure of scientific and technical content will be a significant step forward in opening public access to a valuable collection that has heretofore had limited library and commercial vendor availability.

United States Patent and Trademark Office (USPTO)

  • Project Name – Enhancement to Patent Maintenance Fee Events Data

    What’s New – Making fee data available in machine-readable form for the first time

    During the first quarter of FY2010, USPTO met an expressed public need for data by making available a new machine-readable online product: Patent Maintenance Fee Events, a record of patent maintenance payments, expired patents, and related transactions.  Patent maintenance fee information had been previously available only through interactive patent application retrieval from USPTO’s Public Patent Application Information Retrieval (PAIR) System.  This data had been frequently requested by USPTO’s data customers and is the first machine-readable, raw data made available from the Public PAIR system.
  • Project – Expansion of Patent Bibliographic Data

What’s New – Expanding online availability of information on past patent grants and applications

Also during the first quarter of FY2010, USPTO made available more bibliographic information for patent grants issued from September 1996 through December 2008, and patent applications submitted from mid-March 2001 through December 2008.  These data expanded the current USPTO dataset offerings on Data.Gov.

In addition, USPTO plans to make all patents, published patent applications, and related materials more easily searchable by the public online.  With intellectual property-based businesses estimated to contribute nearly 40 percent of growth achieved by all U.S. private industry, the impact on jobs and the economy of more patent transparency should be significant.  As a step toward improved access to patent information, USPTO will begin posting this data online for free download through a third-party provider during calendar  year 2010.

  • Project – Enhancement of Existing USPTO Data Capabilities Available to the Public

What’s New – Upgrading of existing mechanisms for training users on intellectual property rights

USPTO is developing an outsourcing model for public e-learning opportunities to educate and train the public on intellectual property, patents and trademarks.  It is identifying better search tools and is re-designing application management systems to improve applicants’ electronic business experience with around-the-clock capability.  In addition to data sets already available on, USPTO is working with the public to identify mechanisms to quickly expand public access to more USPTO data.

Office of the Secretary

  • Project – Public Schedule Data for Secretary

What’s new – Regularly updated, searchable feed of the Secretary’s public schedule

The Office of Public Affairs, in conjunction with the Offices of Scheduling and Business Liaison, is planning to release the Commerce Secretary’s daily public schedule on either a daily or weekly basis.  Commerce intends for this data to be as readable and as complete as possible.  It is currently examining options for dealing with technical and managerial hurdles involved in regularly releasing this data, but plans to begin posting it on and by the end of 2010.

  • Project – Streaming Video for More Meetings

What’s new – Expanded availability of Commerce meetings via the Web

One of the more widely requested methods of being more transparent involves streaming video of appropriate Commerce meetings for public viewing.  While it would be prohibitively expensive to provide video access to all meetings that occur at Commerce on any given day, it is important to increase the extent to which streaming is currently employed.  To do so, Commerce plans to meet core new media objectives relative to streaming meetings or events in 2011.

Open Government Plan - Transparency (Identifying & Release More Data Sets)

III. Transparency (con.)

How Commerce will Identify and Publicly Release More Data Sets

To assist Commerce in adhering to the guidance provided in OMB’s “Open Government Directive” to make existing and new data sets and tools more easily accessible by the public, the Department has established an internal community of points of contact (POCs) from each of its operating units.  This communication network supports the enhanced exchange of information throughout the Department.

The responsibilities of the POCs are built on OMB’s guidelines for increasing transparency, participation, and collaboration while providing value to the public and other agencies.  In doing so, they focus on certain central themes:  expanding access; utilizing open platforms; disaggregating data; adopting rapid integration; emphasizing program responsibility; growing and improving through user feedback; and embracing and driving best practices.

Commerce will continue to be a key participant in developing requirements, collecting and analyzing data, and disseminating information to the public for the Federal Government.  As it moves forward in implementing the Open Government Initiative, Commerce is using the POCs to build integrated, replicable processes that allow interaction between data owners, technical staff, knowledge management staff and the public.

As part of those processes, Commerce has encouraged data owners to develop timelines to publish new information and enhance previously published information.  It is improving existing data user tools to allow greater access across all of its operating units and by other federal agencies and departments. 

To carry out its various responsibilities, Commerce must maintain a diverse workforce with a broad spectrum of skills and access to a wide range of information and knowledge.  Effective data sharing at all levels ensures that stakeholders receive services based upon information that meets the highest standards for accuracy, integrity and quality.  To provide consistency across government, communities of practice (COPs) and communities of interest (COIs) are reviewing existing standards and establishing inter-agency best practices to encourage a well-defined, active information dissemination environment. 

Through its involvement, Commerce will emphasize the following:

  • Information QualityA primary concern within Commerce is making data more usable through better tools, user-friendly metadata, and sharing how the data was collected and its intended purpose. 

  • Federal Spending Information – The Chief Financial Officer and Assistant Secretary for Administration (CFO/ASA) serves as the senior official responsible for the integrity of Commerce-related data posted to the website.  The Department’s grants are timely and comprehensively reported through as reflected in the “green” ratings that Commerce has consistently earned from 2007 to the present.  A formal process has been established to ensure that all grant offices are reporting in a timely manner and implementing a formalized data content oversight program.  For contracts, the Department regularly conducts an independent validation and verification of data recorded in the Federal Procurement Data System (FPDS) each quarter.  Feedback on the results of those reviews is provided to the operating units. Quality control for non-Recovery Act transactions has been further strengthened by processes developed for reporting under the American Recovery Act.  Commerce continues to meet all requirements for publishing spending information on this website and is an active participant in planning for its expansion. 

    The American Recovery and Reinvestment Act requires recipients of Recovery funds to report quarterly on how money is being spent.  All data is posted on  so the public can monitor Recovery Act efforts.  Other reporting requirements include posting program plans and performance metrics, weekly updates on financial status and major activities, notices of all funding announcements, and notifications of press releases and other public information about Recovery Act-related activities.

    To develop processes, procedures and policies for complying with the Recovery Act and all related guidance, Commerce formed several interlocking intra-agency working groups in March 2009.  Taking direction from OMB guidance, it created a team to implement the Recovery Act and report in as transparent a manner as possible.  All reporting requirements are being met.  Commerce has complied with all OMB guidance, timelines and milestones and, in many cases, has played a leading role in inter-agency working groups to improve timeliness, efficiency and effectiveness in reporting on Recovery Act activities. 
  • Citation – Commerce will explore best practices for citing published data to make reference data easier to locate for end-users.  This will assist in establishing the context needed to understand fully the meaning of published data.

  • Intergovernmental Working Groups – The government-wide Open Government Working Group has created cross-agency groups to examine data sets that would support health and green energy COPs.  At Commerce’s suggestion, an additional group has been established to identify information and data sets to support job creation.  This group is chaired by Commerce and focuses on data and resources across government.  As a model, it uses National  Weather Service data, which has created an entire industry based on its machine readable formats.
  • SecurityCommerce is using its POCs to verify that information being published poses no security risk once it is released to the public.

  • Privacy Commerce is also relying on its POCs to verify that information being published does not pose a risk of personally identifiable information being released.