// problem description
Protego constructs regular expressions to match URLs against robots.txt Allow: and Disallow: directives, see protego.urlpattern.URLPattern.preparepatternforregex(). Every in the directive value is translated into a lazy .? regex piece, thus a specially crafted directive value with many asterisks may produce a regex that freezes the parser due to exponential backtracking.
// impactParsing a specially crafted robots.txt with protego.Protego.parse() and then trying to match an URL with protego.Protego.canfetch() results in the latter call not returning for a period dependent on the length of the URL.
// proof of conceptfrom protego import Protego
robotstxt = f"""
User-agent: *
Disallow: /{"*1" * 12}*Z
"""
rp = Protego.parse(robotstxt)
url = "/" + "1" * 60
rp.can_fetch(url, "mybot") # freezesCVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:P/VC:N/VI:N/VA:H/SC:N/SI:N/SA:N
- Attack vector
- Network
- Attack complexity
- Low
- Attack requirements
- None
- Privileges required
- None
- User interaction
- Passive
- Confidentiality (vulnerable system)
- None
- Integrity (vulnerable system)
- None
- Availability (vulnerable system)
- High
- Confidentiality (subsequent systems)
- None
- Integrity (subsequent systems)
- None
- Availability (subsequent systems)
- None