The robots meta tag is used by search engine crawlers to know if they should crawl and index the web page they are on. Most major search engines use the robots meta tag but using the robots.txt standard is more efficient. Since all search engines will try to crawl every web page it’s only necessary to use the meta tag when trying to prevent a web page from being indexed. Below is an example of the robots meta tag syntax.
The above code must be inserted between thetag in your web page’s source code. The section in red is where you can insert one of the below elements to tell a search engine crawler what to do this this page, you can insert more than one element just seperate them with commas.
* INDEX (Crawl and add the web page to the crawlers index)
* NOINDEX (Do not crawl and do not add to the crawlers index)
* NOFOLLOW (Do not crawl or index outbound links to other sites)
* NOARCHIVE (Do not cache and archive a copy of this web page)
* NOSNIPPET (Do not display a description in search results)
* NOODP (Do not use the description Open Directory Project)
Remember, you only need to use this tag to prevent crawling, caching, etc of a web page. You can usually just leave it out of your web page’s source code or use the robots.txt standard to direct crawlers on indexing and crawling of your site.