QuestionQ58Verified/Web Communication SecurityCopy linkReport a problemAsk AstroTutorSave questionWhich file tells programs such as web spiders not to search particular areas of a site?ARobots.txtBRestricted.txtCSpider.txtDSearch.txtShow AnswerAnswerExplanationA robots.txt file implements the Robots Exclusion Protocol, which specifies paths that compliant web crawlers should not access.Show comments0Show commentsCommunity DiscussionNo comments yet. Be the first to start the discussion!
Community Discussion