Crawlee v4.0.0-rc.1

v4.0.0-rc.1Pre-release
Added 5
  • DomCrawler with a pluggable DOM parser
  • OpenTelemetry instrumentation via @crawlee/otel
  • skipRequest and afterCommit context helpers
  • initialConcurrency option
  • extractMicrodata helper
Changed 4
  • Crawler-only code moved from @crawlee/core to @crawlee/basic, and more internals made private
  • Input handling moved to the Apify SDK, requiring apify >= 4.0.0-beta.40
  • The request queue is no longer emptied between run() calls
  • Statistics renamed and IRequestManager / IBrowserPool contracts updated
Fixed 3
  • HTTP redirect handling for credentials, cookies and request bodies
  • Cloudflare challenge detection
  • A robots.txt 404 response now means allow-all
Removed 1
  • @crawlee/jsdom and @crawlee/linkedom moved out of the monorepo

From Crawlee

Breaking since rc.0
  • @crawlee/jsdom and @crawlee/linkedom moved out of the monorepo (#4119)
  • Crawler-only code moved from @crawlee/core to @crawlee/basic, more internals made private (#4117, #4036, #4060)
  • Input handling moved to the Apify SDK, requires apify >= 4.0.0-beta.40 (#4155)
  • The request queue is no longer emptied between run() calls (#4056)
  • Statistics renames, IRequestManager / IBrowserPool contract updates (#4109, #4061, #4161)
New
  • DomCrawler with a pluggable DOM parser (#4104)
  • OpenTelemetry instrumentation via @crawlee/otel (#3361)
  • skipRequest and afterCommit context helpers (#4158, #4110), initialConcurrency option (#4067), extractMicrodata helper (#3233)
Fixes

HTTP redirect handling for credentials, cookies and request bodies (#4137, #4154, #4136), Cloudflare challenge detection (#4170), a robots.txt 404 now means allow-all (#4171), and more.

Upgrading guide: https://crawlee.dev/js/docs/next/upgrading/upgrading-to-v4

Full Changelog: https://github.com/apify/crawlee/compare/v4.0.0-rc.0...v4.0.0-rc.1

View original

Upgraded? How did it go?

Discussion