promotion image of download ymail app
Promoted

How does robots.txt handle multiple user-agent lines?

In this example:

User-agent: msnbot

User-agent: slurp

Disallow: /

Does the Disallow: / apply to only slurp or does it apply to both slurp and msnbot?

1 Answer

Relevance
  • om
    Lv 6
    9 years ago
    Favorite Answer

    That Disallow applies to both slurp and msnbot.

    If a User-Agent string matches the robot, the next collection of Disallow and Allow directives applies to that robot. That happens regardless of whether additional User-Agent lines appear before the Disallow and Allow lines.

    Source(s): The original robots.txt rules are at http://www.robotstxt.org/orig.html A more recent slightly-expanded set of rules (including the 'Allow' command) is at http://www.robotstxt.org/norobots-rfc.txt
    • Commenter avatarLogin to reply the answers
Still have questions? Get your answers by asking now.