# Datasets 5.0.0 - Product: Datasets (https://whatsnew.fyi/product/hugging-face-datasets) - Vendor: Hugging Face - Date: 2026-06-05 - Version: 5.0.0 - Original notes: https://github.com/huggingface/datasets/releases/tag/5.0.0 - Permalink: https://whatsnew.fyi/product/hugging-face-datasets/releases/5.0.0 What's New is an index, not a publisher: every entry below links to the vendor's own release notes, which are the authoritative source. Entries are labelled where they are hand-curated sample data, pre-releases, or drawn from a secondary source such as a developer blog. Reuse: the summaries, labels and curation here are © What's New. Quote freely with attribution and a link back; wholesale republication of the corpus is not permitted — terms: https://whatsnew.fyi/terms. The vendors' own release notes remain their publishers'. --- - **added** — Parse Agent traces messages for SFT using the teich library to enable training on traces from claude_code, pi, codex and other sources - **added** — Use multiple input shards for shuffle buffer in streaming mode with configurable max_buffer_input_shards parameter - **added** — Add batch(by_column=...) method to batch examples by a specified column, useful for robotics datasets - **added** — Add Apache Iceberg format support - **added** — Add TsFile (Apache IoTDB) packaged builder with per-device wide format - **added** — Add 3D mesh support and MeshFolder builder - **added** — Add .conll / .conllu dataset format loader for CoNLL-2003, 2000, and U formats - **added** — Add num_proc argument to Dataset.to_sql - **changed** — Default shuffling mechanism now uses multiple input shards instead of single shard - **changed** — Support fsspec 2026.4.0 - **changed** — Pass library_name and version to HfApi in dataset push and delete paths - **fixed** — Fix storage_options lookup for streaming Lance datasets - **fixed** — Fix Parquet streaming hangs at the end of script - **fixed** — Fix parquet reshard - **fixed** — Fix parquet columns argument - **fixed** — Fix progress bar exceeding total when load_from_cache_file=False in map operation - **fixed** — Fix single lance file from pylance 7.0 ##### Datasets Features ###### Agent traces * Parse Agent traces messages for SFT using `teich` by @lhoestq in https://github.com/huggingface/datasets/pull/8232 * Agent traces from claude_code/pi/codex and others can now be loaded with load_dataset * Using the `teich` library (new optional dependency), traces are parsed to `messages` to enable training on traces using e.g. `trl` * Load the data: ```python >>> from datasets import load_dataset >>> ds = load_dataset("lhoestq/agent-traces-example", split="train") >>> ds[0]["messages"] [{'role': 'user', 'content': 'Download a random dataset from Hugging Face, use DuckDB to inspect it, and come back with a short report about it. Be concise and include: dataset name, what files/format you found, row count or rough size if you can determine it,...' ...] ``` * Train on agent traces: ```bash trl sft --dataset-name lhoestq/agent-traces-example ... ``` * find all the Agent traces datasets on HF here: https://huggingface.co/datasets?format=format:agent-traces&sort=trending ###### Next-level shuffling in streaming mode * Use multiple input shards for shuffle buffer by @lhoestq in https://github.com/huggingface/datasets/pull/8194 ```python ds = load_dataset(..., streaming=True) ds = ds.shuffle(seed=42) # or configure local buffer shuffling manually, default is: ds = ds.shuffle(seed=42, buffer_size=1000, max_buffer_input_shards=10) ``` before👎: image after✨: image toy example comparison ```python from datasets import IterableDataset ds = IterableDataset.from_dict({"i": range(123_456_789)}, num_shards=1024) ds = ds.shuffle(seed=42) print("Cold start ids:") print(list(ds.take(10)["i"])) print("Nominal regime ids:") print(list(ds.skip(10_000).take(10)["i"])) ``` before👎: ``` Cold start ids: [6148853, 6149537, 6149418, 6149202, 6149197, 6149622, 6148849, 6149461, 6148965, 6148858] Nominal regime ids: [6149537, 6149418, 6149202, 6149197, 6149622, 6148849, 6149461, 6148965, 6148858, 6149290] ``` after✨: ``` Cold start ids: [7836668, 9283505, 95847927, 482299, 9283471, 482341, 112003312, 59920157, 43764666, 95847871] Nominal regime ids: [9283505, 95847927, 482299, 9283471, 482341, 112003312, 59920157, 43764666, 95847871, 16758448] ``` Note: `ds.state_dict()` and `ds.load_state_dict()` are still supported for this improved shuffling :) enabling dataset checkpointing Note 2: it uses threads to fetch the first examples in parallel from the input shards Note 3: This is a BREAKING CHANGE: the default shuffling mechanism now uses multiple input shards. You can get the old mechanism by passing `max_buffer_input_shards=1` to `IterableDataset.shuffle()` ###### New batching features for robotics datasets * Add batch(by_column=...) by @lhoestq in https://github.com/huggingface/datasets/pull/8172 ```python from datasets import Dataset ds = Dataset.from_dict({"episode": [0] * 10 + [1] * 10, "frame": list(range(10)) * 2}) # ds = ds.to_iterable_dataset() ds = ds.batch(by_column="episode") for x in ds: print(x) # {'episode': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'frame': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]} # {'episode': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'frame': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]} ``` ###### New supported formats * Add Apache Iceberg format support by @frankliee in https://github.com/huggingface/datasets/pull/8148 * feat: add TsFile (Apache IoTDB) packaged builder with per-device wide format by @JackieTien97 in https://githu _[Truncated at 4000 characters — full notes: https://github.com/huggingface/datasets/releases/tag/5.0.0]_