Making Public Web Pages Easy to Find

Best Practice:

All Department of Commerce (DOC) organizations should use HTML title tags and
appropriate meta tags to make their Web pages easier to find by Web search engines
and should consider submission to the major search engines.

Purpose:

To make Department Web sites easy to find.

Discussion:

To understand how Web pages can be made easier to find by search engines, one must know how search engines work. They index Web site content and maintain the results in databases. Although the indexing schemes used by search engines vary, they use programs called robots to build the index. The robot reads a Web site, follows any hyperlinks, and enters the HTML tag information and Web page text into its database. The results of indexing, and hence the relative ranking of a Web page or site, will vary with each search engine.

In general, a Web page is most likely to be found through a search when the following elements are included in the <HEAD> section of the page:

  1. <TITLE> tag: words describing the content of the Web page (since search results are often displayed based on contents of the <TITLE> tag).

  2. Description <META> tag: a brief description of the page or site (since search results may display this as a page summary).

  3. Keyword <META> tag: words people would associate with the content of the Web page and might use in a Web search.

  4. Robot <META> tag: allows HTML authors to indicate to visiting robots whether a document should be indexed or used to harvest more links.

Web developers and administrators should consider submitting their Web sites to the major search engines. Submission lets the search engine know the Web site exists and is ready to be indexed. Site administrators should review each search engine's Web site for information about effective submission and be aware that, due to demand on search engines, indexing may not occur until months after submission. When a DOC organization Web address changes, the Web site should be re-submitted to the major search engines.

In those instances where a page should not be indexed, use of the Robots Exclusion Protocol should be considered.

Resource List

Disclaimer: The Department of Commerce does not endorse any of the Websites links noted below. They are merely examples of some of the resources which may be helpful to Web developers attempting to make their Web sites easier to find. The links are outside of the WAC webresources website, you will be responsible for reviewing the Privacy Notice of each site you visit, as its information collection practices may differ from ours. The list is not intended to be exhaustive.

Helping Search Engines Index Your Site
http://www.w3.org/TR/html401/appendix/notes.html#h-B.4

Web Robots
http://www.robotstxt.org/wc/robots.html

<Head> and <Title> tags
http://www.w3.org/TR/html401/struct/global.html#h-7.4

Robots <META> tag
http://www.w3.org/TR/html401/appendix/notes.html#h-B.4.1.2
http://www.robotstxt.org/wc/exclusion.html#robotstxt

Open Directory Project (database for some major search engines)
http://dmoz.org/about.html

Major Search Engines
http://searchenginewatch.com/links/Major_Search_Engines/The_Major_Search_Engines

Department of Commerce Web Advisory Council (WAC)
U.S. Department of Commerce

Send questions and comments about this page to WAC@doc.gov
Page last updated October 12, 2010