# Scrapling changelog > Scrapling release notes. - Vendor: Scrapling - Category: AI - Official site: https://scrapling.readthedocs.io/en/latest/ - Tracked by: What's New (https://whatsnew.fyi/product/scrapling) - Harvested from: GitHub (D4Vinci/Scrapling) - Entries below: 10 (newest first) What's New is an index, not a publisher: every entry below links to the vendor's own release notes, which are the authoritative source. Entries are labelled where they are hand-curated sample data, pre-releases, or drawn from a secondary source such as a developer blog. ## Releases ### v0.4.12 — Release v0.4.12 - Date: 2026-07-26 - Version: v0.4.12 - Original notes: https://github.com/D4Vinci/Scrapling/releases/tag/v0.4.12 - Permalink: https://whatsnew.fyi/product/scrapling/releases/v0.4.12 - **added** — Spiders can now tune their own speed with AutoThrottle, which measures how fast each website answers and adjusts the delay of every domain on its own - **added** — Export your results to CSV and XML formats, with support for items with different keys and nested values written as JSON - **added** — The MCP server can now require authentication with a token and restrict which hostnames it answers to - **added** — Browsers now accept CDP URLs over HTTP, not just WebSocket ones - **added** — Docker images are now tagged with their release version instead of only latest - **fixed** — Fixed cached responses losing all their cookies when the response came from a browser engine - **fixed** — Fixed StealthyFetcher forcing the en-US locale on every browser instead of following the system's locale - **fixed** — Fixed a misleading error message in the storage system and removed a dead call left after inserts - **changed** — get_all_text() is now O(nodes) instead of walking up the ancestors of every single text node, making it around 5-6x faster on deeply nested pages **A release focused on making your spiders smarter about the websites they crawl** > [!NOTE] > **[Follow us on X for daily tips and tricks](https://x.com/Scrapling_dev)** ##### 🚀 New Stuff and quality of life changes - **Spiders can now tune their own speed with AutoThrottle.** Instead of guessing a `download_delay` that's either too slow or gets you banned, the spider measures how fast each website answers and adjusts the delay of every domain on its own. When a website starts blocking or rate-limiting you, it doubles the delay (or waits exactly what the `Retry-After` header asks for) until that stops, then speeds back up. Your `download_delay` and any robots.txt `Crawl-delay` are still respected as the minimum. (Check the [docs](https://scrapling.readthedocs.io/en/latest/spiders/advanced.html#autothrottle)) ```python class MySpider(Spider): name = "adaptive" start_urls = ["https://example.com"] autothrottle_enabled = True ``` - **Export your results to CSV and XML**, next to the JSON/JSONL exporters you already had. Items that don't all share the same keys are still exported without losing anything, and nested values are written as JSON. (Check the [docs](https://scrapling.readthedocs.io/en/latest/spiders/getting-started.html)) ```python result = MySpider().start() result.items.to_csv("products.csv") result.items.to_xml("products.xml") ``` - **The MCP server can now require authentication**, so you can safely expose it instead of keeping it on your own machine. Any request without the token is rejected, and you can also restrict which hostnames the server answers to. (Check the [docs](https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html)) ```bash scrapling mcp --http --auth-token "$(openssl rand -hex 32)" ``` - **Browsers now accept CDP URLs over HTTP**, not just WebSocket ones. So next to the `wss://` endpoints managed browser providers hand out, you can now point any browser fetcher or MCP session at a Chrome you started yourself with `--remote-debugging-port=9222`. - **Published Docker images are now tagged with their release version** instead of only `latest`, so you can pin the exact version you want, by @JanRK in [#384](https://github.com/D4Vinci/Scrapling/pull/384). ##### 🐛 Bug Fixes - **Fixed cached responses losing all their cookies** when the response came from a browser engine, which silently broke any session or auth logic relying on them while using the spiders' development mode, by @amitvijapur in [#379](https://github.com/D4Vinci/Scrapling/pull/379). (Fixes [#376](https://github.com/D4Vinci/Scrapling/issues/376)) - **Fixed `StealthyFetcher` forcing the `en-US` locale** on every browser instead of following your system's, which made websites see a mismatch between your locale and your IP address and treat you as suspicious, like Google answering with 429s. (Fixes [#381](https://github.com/D4Vinci/Scrapling/issues/381)) - **Fixed a misleading error message in the storage system** and removed a dead call left after inserts, by @fix2015 in [#377](https://github.com/D4Vinci/Scrapling/pull/377). ##### Performance - **`get_all_text()` is now O(nodes)** instead of walking up the ancestors of every single text node, which makes it around 5-6x faster on deeply nested pages, by @yetval in [#378](https://github.com/D4Vinci/Scrapling/pull/378). _🙏 Special thanks to the community for all the continuous testing and feedback_ --- Big shoutout to our Platinum Sponsors
< _[Truncated at 4000 characters — full notes: https://github.com/D4Vinci/Scrapling/releases/tag/v0.4.8]_ ### v0.4.7 — Release v0.4.7 - Date: 2026-04-17 - Version: v0.4.7 - Original notes: https://github.com/D4Vinci/Scrapling/releases/tag/v0.4.7 - Permalink: https://whatsnew.fyi/product/scrapling/releases/v0.4.7 - **added** — Add a screenshot MCP tool that captures a page and returns it as an ImageContent block, supporting PNG and JPEG, full-page captures, JPEG quality, and readiness controls - **added** — Add a custom session_id parameter to open_session to allow naming sessions meaningfully instead of random hex identifiers - **fixed** — Fix FetcherSession state corruption and lazy session close crash - **fixed** — Fix TypeError when using the CLI's --ai-targeted flag with HTTP commands due to unexpected block_ads keyword argument **A focused update bringing eyes to your AI agents 📸** > [!NOTE] > **[Follow us on X for daily tips and tricks](https://x.com/Scrapling_dev)** ##### 🚀 New Stuff and quality of life changes - **Added a `screenshot` MCP tool** that captures a page and returns it as a real MCP `ImageContent` block so the model can actually see it. The tool requires an open browser session, so you call `open_session` first (either `dynamic` or `stealthy`) and pass the `session_id` here. Supports PNG and JPEG, full-page captures, JPEG quality, and the usual readiness controls (`wait`, `wait_selector`, `network_idle`, `timeout`). (implements [#244](https://github.com/D4Vinci/Scrapling/issues/244)) - **Added a custom `session_id` parameter to `open_session`** so you can name sessions meaningfully (`"search"`, `"checkout"`) instead of the random 12-character hex default. By @hauntedhost in [#243](https://github.com/D4Vinci/Scrapling/pull/243) ##### 🐛 Bug Fixes - **Fixed `FetcherSession` state corruption and a lazy session close crash**. By @yetval in [#245](https://github.com/D4Vinci/Scrapling/pull/245) - **Fixed `TypeError: Session.request() got an unexpected keyword argument 'block_ads'`** when using the CLI's `--ai-targeted` flag with HTTP commands. By @voidborne-d in [#249](https://github.com/D4Vinci/Scrapling/pull/249) (Fixes [#247](https://github.com/D4Vinci/Scrapling/issues/247)) ##### Translations - **Added a Brazilian Portuguese README translation** By @rgomids in [#250](https://github.com/D4Vinci/Scrapling/pull/250) _🙏 Special thanks to the community for all the continuous testing and feedback_ --- ###### Big shoutout to our Platinum Sponsors ### v0.4.6 — Release v0.4.6 - Date: 2026-04-13 - Version: v0.4.6 - Original notes: https://github.com/D4Vinci/Scrapling/releases/tag/v0.4.6 - Permalink: https://whatsnew.fyi/product/scrapling/releases/v0.4.6 - **added** — Added built-in ad blocking for browser fetchers with `block_ads=True` parameter to block requests to approximately 3,500 known ad and tracker domains at the route interception level - **added** — Added DNS-over-HTTPS support with `dns_over_https=True` parameter to prevent DNS leaks when using proxies by routing DNS queries through Cloudflare - **added** — Added `page_setup` callback for browser fetchers that runs before page navigation to register event listeners, routes, or scripts - **added** — Added `--block-ads` and `--dns-over-https` CLI options to both `fetch` and `stealthy-fetch` commands - **fixed** — Fixed `Seconds` type alias rejecting float values in browser fetchers - **fixed** — Fixed duplicate ID segments in full-path selector generation where elements with `id` attributes had their selectors appended twice - **fixed** — Fixed full-path XPath generation emitting bare `[@id='x']` predicates instead of valid `*[@id='x']` syntax - **fixed** — Fixed missing shell signature parameters including `blocked_domains`, `block_ads`, `retries`, `retry_delay`, `capture_xhr`, `executable_path`, and `dns_over_https` **A focused update on browser stealth, privacy, and developer experience 🔒** > [!NOTE] > **[Follow us on X for daily tips and tricks](https://x.com/Scrapling_dev)** ##### 🚀 New Stuff and quality of life changes - **Added built-in ad blocking** for browser fetchers. Pass `block_ads=True` to block requests to ~3,500 known ad and tracker domains at the route interception level -- no DNS, no TCP, instant abort. Can be combined with `blocked_domains` for custom lists. The MCP server and CLI `--ai-targeted` mode enable this automatically to save tokens and speed up page loads. ```python page = StealthyFetcher.fetch('https://example.com', block_ads=True) ``` - **Added DNS-over-HTTPS support** to prevent DNS leaks when using proxies. Pass `dns_over_https=True` to route DNS queries through Cloudflare's DoH, so your real location isn't exposed through DNS resolution even when your HTTP traffic goes through a proxy. ```python page = StealthyFetcher.fetch('https://example.com', proxy='http://proxy:8080', dns_over_https=True) ``` - **Added `page_setup` callback** for browser fetchers. A function that runs before `page.goto()`, letting you register event listeners, routes, or scripts that must be set up before the page navigates. Pairs with `page_action` (which runs after navigation). (Solves [#237](https://github.com/D4Vinci/Scrapling/issues/237)) ```python def capture_websockets(page): page.on("websocket", lambda ws: print(f"WS: {ws.url}")) page = DynamicFetcher.fetch('https://example.com', page_setup=capture_websockets) ``` - **Added `--block-ads` and `--dns-over-https` CLI options** to both `fetch` and `stealthy-fetch` commands. ##### 🐛 Bug Fixes - **Fixed `Seconds` type alias** rejecting float values. Passing `wait=1.5` or `timeout=500.0` to browser fetchers would fail with a type error because the type alias incorrectly treated `float` as metadata instead of a type. by @kuishou68 in [#240](https://github.com/D4Vinci/Scrapling/pull/240) - **Fixed duplicate ID segments in full-path selector generation**. Elements with `id` attributes had their selector appended twice when generating full CSS/XPath paths, producing selectors like `body > #main > #main > #target > #target`. Also fixed full-path XPath emitting bare `[@id='x']` predicates (invalid XPath) instead of `*[@id='x']`. by @sjhddh in [#241](https://github.com/D4Vinci/Scrapling/pull/241) - **Fixed missing shell signature parameters**. The interactive shell was missing `blocked_domains`, `block_ads`, `retries`, `retry_delay`, `capture_xhr`, `executable_path`, and `dns_over_https` from its function signatures. _🙏 Special thanks to the community for all the continuous testing and feedback_ --- ###### Big shoutout to our Platinum Sponsors [!NOTE] > **[Follow us on X for daily tips and tricks](https://x.com/Scrapling_dev)** ##### 🚀 New Stuff and quality of life changes - **Spider Development Mode**: Iterating on a spider's `parse()` logic used to mean re-hitting the target servers on every run, which is slow, noisy, and a great way to get rate-limited while you're still figuring out your selectors. The new development mode caches every response to disk on the first run and replays them from disk on every subsequent run, so you can tweak your callbacks and re-run as many times as you want without making a single network request. Enable it with one class attribute: ```python class MySpider(Spider): name = "my_spider" start_urls = ["https://example.com"] development_mode = True async def parse(self, response): yield {"title": response.css("title::text").get("")} ``` The cache lives in `.scrapling_cache/{spider.name}/` by default and can be redirected anywhere with `development_cache_dir`. Two new stat counters, `cache_hits` and `cache_misses`, let you see how the cache performed. Cache replay bypasses `download_delay`, rate limiting, and the blocked-request retry path so iteration is as fast as the disk allows. Don't ship a spider with `development_mode = True` -- it's a development tool, not a production cache. See the [docs](https://scrapling.readthedocs.io/en/latest/spiders/advanced.html#development-mode) for the full story. - **Safer redirects by default**: `follow_redirects` now defaults to `"safe"` across all HTTP fetchers, the MCP server, and the shell. Redirects are still followed, but ones targeting internal/private IPs (loopback, private networks, link-local) are rejected. This protects you from SSRF when scraping user-supplied URLs. Pass `follow_redirects="all"` to get the old behavior, or `False` to disable redirects entirely. ##### 🐛 Bug Fixes - **Force-stop no longer loses your checkpoint**: Pressing Ctrl+C twice (force-stop) on a spider with `crawldir` enabled used to race against the checkpoint write -- the cancel scope would tear down the task before the pickle finished, leaving `paused=False` and triggering the cleanup path that *deletes* the previous checkpoint. The result was that force-stopping a long crawl could lose all the progress you were trying to save. The engine now writes the checkpoint **before** calling `cancel_scope.cancel()`, so a force-stop always preserves the latest pending state. By @voidborne-d in [#230](https://github.com/D4Vinci/Scrapling/pull/230). _🙏 Special thanks to the community for all the continuous testing and feedback_ ---