How does robots.txt handle multiple user-agent lines?
In this example:
Does the Disallow: / apply to only slurp or does it apply to both slurp and msnbot?
- omLv 69 years agoFavorite Answer
That Disallow applies to both slurp and msnbot.
If a User-Agent string matches the robot, the next collection of Disallow and Allow directives applies to that robot. That happens regardless of whether additional User-Agent lines appear before the Disallow and Allow lines.Source(s): The original robots.txt rules are at http://www.robotstxt.org/orig.html A more recent slightly-expanded set of rules (including the 'Allow' command) is at http://www.robotstxt.org/norobots-rfc.txt