What is Robots.Txt?

Robots.Txt is Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website. The robots.txt file is part of the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users. The REP also includes directives like meta robots, as well as page-, subdirectory-, or site-wide instructions for how search engines should treat links (such as “follow” or “nofollow”). In practice, robots.txt files indicate whether certain user agents (web-crawling software) can or cannot crawl parts of a website. These crawl instructions are specified by “disallowing” or “allowing” the behavior of certain (or all) user agents. robots.txt is a standard used by websites to communicate with web crawlers and other web robots.

From Simba’s SEO Dictionary

Leave your vote

More

Comments

Leave a Comment

Ad Blocker Detected!

For Simbacademy to fulfil its mission of teaching the world's entrepreneurs and marketers digital marketing for free, we rely on money earned from showing advertisements and affiliate products/services on our website.

Refresh

Log In

Or with username:

Forgot password?

Don't have an account? Register

Forgot password?

Enter your account data and we will send you a link to reset your password.

Your password reset link appears to be invalid or expired.

Log in

Privacy Policy

Add to Collection

No Collections

Here you'll find all collections you've created before.

Home
Mvideos
Mgag!
Feed
Learn