What’s New

Scrapling v0.4.4

v0.4.4

Release v0.4.4

Added
  • Add robots.txt compliance to the Spider framework with a new robots_txt_obey option that automatically fetches and respects robots.txt rules including Disallow, Crawl-delay, and Request-rate directives
  • Add robots.txt cache pre-warming to fetch and parse robots.txt for all start_urls domains before the crawl loop begins
  • Add a new robots_disallowed_count stat to CrawlStats to track how many requests were blocked by robots.txt rules
  • Add protego as a new dependency under the fetchers optional group for robots.txt parsing
Changed
  • Update dependencies for latest fingerprints and related improvements
Fixed
  • Fix a critical MRO issue with ProxyRotator where the _build_context_with_proxy stub was shadowing the real implementation from child classes, causing proxy rotation to always raise NotImplementedError
  • Fix a page pool leak when using per-request proxy rotation with browser sessions where pages created inside temporary contexts were not removed from the pool on cleanup
  • Fix a missing type assertion in the static fetcher where curl_cffi could return None from session.request(), causing downstream errors

A new update with important spider improvements and bug fixes 🎉

🚀 New Stuff and quality of life changes
  • Added robots.txt compliance to the Spider framework with a new robots_txt_obey option. When enabled, the spider will automatically fetch and respect robots.txt rules before crawling, including Disallow, Crawl-delay, and Request-rate directives. Robots.txt files are fetched concurrently and cached per domain for the entire crawl. By @AbdullahY36 in #226
  • Added robots.txt cache pre-warming so all start_urls domains have their robots.txt fetched and parsed before the crawl loop begins, avoiding delays on the first request to each domain.
  • Added a new robots_disallowed_count stat to CrawlStats to track how many requests were blocked by robots.txt rules during a crawl.

Check it out on the website from here

🐛 Bug Fixes
  • Fixed a critical MRO issue with ProxyRotator where the _build_context_with_proxy stub was shadowing the real implementation from child classes, causing proxy rotation to always raise NotImplementedError (Fixes #215). Thanks @yetval
  • Fixed a page pool leak when using per-request proxy rotation with browser sessions. Pages created inside temporary contexts were not removed from the pool on cleanup, leading to stale references accumulating over time. By @yetval in #223
  • Fixed a missing type assertion in the static fetcher where curl_cffi could return None from session.request(), causing downstream errors.
Other
  • Updated dependencies, so expect the latest fingerprints and other stuff.
  • Added protego as a new dependency under the fetchers optional group for robots.txt parsing.

🙏 Special thanks to the community for all the continuous testing and feedback

Big shoutout to our Platinum Sponsors
View original