Rémi Lapeyre 8047e0e1c6 bpo-35922: Fix RobotFileParser when robots.txt has no relevant crawl delay or request rate (GH-11791) ...

Co-Authored-By: Tal Einat <taleinat+github@gmail.com>

2019-06-16 09:48:57 +03:00

9.2 KiB

Raw Blame History

View Raw