X-robots-tag is Like meta robots tags, this tag provides crawlers with instructions on how to crawl or index web page content. The X-Robots-Tag can be used as an element of the HTTP header response for a given URL. Any directive that can be used in a robots meta tag can also be specified as an X-Robots-Tag.

Comments