QuestionQ58

VerifiedWeb Communication Security

Which file tells programs such as web spiders not to search particular areas of a site?

Explanation

A robots.txt file implements the Robots Exclusion Protocol, which specifies paths that compliant web crawlers should not access.

Community Discussion

No comments yet. Be the first to start the discussion!