v4.0.0-rc.1Pre-release
Added 5
- DomCrawler with a pluggable DOM parser
- OpenTelemetry instrumentation via @crawlee/otel
- skipRequest and afterCommit context helpers
- initialConcurrency option
- extractMicrodata helper
Changed 4
- Crawler-only code moved from @crawlee/core to @crawlee/basic, and more internals made private
- Input handling moved to the Apify SDK, requiring apify >= 4.0.0-beta.40
- The request queue is no longer emptied between run() calls
- Statistics renamed and IRequestManager / IBrowserPool contracts updated
Fixed 3
- HTTP redirect handling for credentials, cookies and request bodies
- Cloudflare challenge detection
- A robots.txt 404 response now means allow-all
Removed 1
- @crawlee/jsdom and @crawlee/linkedom moved out of the monorepo
From Crawlee
Breaking since rc.0
@crawlee/jsdomand@crawlee/linkedommoved out of the monorepo (#4119)- Crawler-only code moved from
@crawlee/coreto@crawlee/basic, more internals made private (#4117, #4036, #4060) - Input handling moved to the Apify SDK, requires
apify>= 4.0.0-beta.40 (#4155) - The request queue is no longer emptied between
run()calls (#4056) - Statistics renames,
IRequestManager/IBrowserPoolcontract updates (#4109, #4061, #4161)
New
DomCrawlerwith a pluggable DOM parser (#4104)- OpenTelemetry instrumentation via
@crawlee/otel(#3361) skipRequestandafterCommitcontext helpers (#4158, #4110),initialConcurrencyoption (#4067),extractMicrodatahelper (#3233)
Fixes
HTTP redirect handling for credentials, cookies and request bodies (#4137, #4154, #4136), Cloudflare challenge detection (#4170), a robots.txt 404 now means allow-all (#4171), and more.
Upgrading guide: https://crawlee.dev/js/docs/next/upgrading/upgrading-to-v4
Full Changelog: https://github.com/apify/crawlee/compare/v4.0.0-rc.0...v4.0.0-rc.1