What changed in Datasets from 4 to 5

2 releases numbered after 4.8.5 up to and including 5.0.1, stable releases only. 4.8.5 and 5.0.1 are the newest stable releases of 4 and 5 we track; this page follows them as new ones ship.

38 changes across 2 releases

Added 11

5.0.1

  • Support hermes traces
  • Support droid agent traces
  • Support batched=True in Dataset.to_dict

5.0.0

  • Parse Agent traces messages for SFT using the teich library to enable training on traces from claude_code, pi, codex and other sources
  • Use multiple input shards for shuffle buffer in streaming mode with configurable max_buffer_input_shards parameter
  • Add batch(by_column=...) method to batch examples by a specified column, useful for robotics datasets
  • Add Apache Iceberg format support
  • Add TsFile (Apache IoTDB) packaged builder with per-device wide format
  • Add 3D mesh support and MeshFolder builder
  • Add .conll / .conllu dataset format loader for CoNLL-2003, 2000, and U formats
  • Add num_proc argument to Dataset.to_sql
Changed 3

5.0.0

  • Default shuffling mechanism now uses multiple input shards instead of single shard
  • Support fsspec 2026.4.0
  • Pass library_name and version to HfApi in dataset push and delete paths
Fixed 22

5.0.1

  • Fix version string in __init__.py
  • Fix conda build
  • Fix JSON loader schema inference for files starting with a UTF-8 BOM
  • Fix batch(by_column=...) crashing after shard/shuffle/split
  • Fix traces streaming
  • Fix lance auth
  • Fix resuming dataloader twice resetting the dataloader
  • Validate Arrow IPC record batches
  • Make the dataset fingerprint independent of Arrow chunking
  • Fix casting a nullable LargeList to a different inner type
  • Fix column drop in Arrow path of axis=1 concatenation
  • Raise on length mismatch in batched IterableDataset.map
  • Fix require_storage_embed recursing into require_storage_cast
  • Fix hdf5 external files
  • Keep flat numeric columns with nulls numeric in numpy format
  • Fix CSV loader dropping on_bad_lines/encoding_errors on pandas 2.0-2.2

5.0.0

  • Fix storage_options lookup for streaming Lance datasets
  • Fix Parquet streaming hangs at the end of script
  • Fix parquet reshard
  • Fix parquet columns argument
  • Fix progress bar exceeding total when load_from_cache_file=False in map operation
  • Fix single lance file from pylance 7.0
Security 2

5.0.1

  • Fix symlink-following arbitrary file write in archive extraction
  • Fix path traversal via metadata file_name in folder-based builders

Original release notes, newest first

The list above is our reading of these notes; the originals from Hugging Face are here, one fold per release.

5.0.1
Bug fixes
Docs
New Contributors

Full Changelog: https://github.com/huggingface/datasets/compare/5.0.0...5.0.1

View originalPermalink

5.0.0
Datasets Features
Agent traces

  • Parse Agent traces messages for SFT using teich by @lhoestq in https://github.com/huggingface/datasets/pull/8232

    • Agent traces from claude_code/pi/codex and others can now be loaded with load_dataset
    • Using the teich library (new optional dependency), traces are parsed to messages to enable training on traces using e.g. trl
    • Load the data:
    >>> from datasets import load_dataset
>>> ds = load_dataset("lhoestq/agent-traces-example", split="train")
>>> ds[0]["messages"]
[{'role': 'user', 'content': 'Download a random dataset from Hugging Face, use DuckDB to inspect it, and come back with a short report about it. Be concise and include: dataset name, what files/format you found, row count or rough size if you can determine it,...'
 ...]
    • Train on agent traces:
    trl sft --dataset-name lhoestq/agent-traces-example ...
Next-level shuffling in streaming mode

  • Use multiple input shards for shuffle buffer by @lhoestq in https://github.com/huggingface/datasets/pull/8194

    ds = load_dataset(..., streaming=True)
ds = ds.shuffle(seed=42)
# or configure local buffer shuffling manually, default is:
ds = ds.shuffle(seed=42, buffer_size=1000, max_buffer_input_shards=10)

    before👎:

    after✨:

    toy example comparison

    from datasets import IterableDataset

ds = IterableDataset.from_dict({"i": range(123_456_789)}, num_shards=1024)
ds = ds.shuffle(seed=42)

print("Cold start ids:")
print(list(ds.take(10)["i"]))
print("Nominal regime ids:")
print(list(ds.skip(10_000).take(10)["i"]))

    before👎:

    Cold start ids:
[6148853, 6149537, 6149418, 6149202, 6149197, 6149622, 6148849, 6149461, 6148965, 6148858]
Nominal regime ids:
[6149537, 6149418, 6149202, 6149197, 6149622, 6148849, 6149461, 6148965, 6148858, 6149290]

    after✨:

    Cold start ids:
[7836668, 9283505, 95847927, 482299, 9283471, 482341, 112003312, 59920157, 43764666, 95847871]
Nominal regime ids:
[9283505, 95847927, 482299, 9283471, 482341, 112003312, 59920157, 43764666, 95847871, 16758448]

    Note: ds.state_dict() and ds.load_state_dict() are still supported for this improved shuffling :) enabling dataset checkpointing

    Note 2: it uses threads to fetch the first examples in parallel from the input shards

    Note 3: This is a BREAKING CHANGE: the default shuffling mechanism now uses multiple input shards. You can get the old mechanism by passing max_buffer_input_shards=1 to IterableDataset.shuffle()

New batching features for robotics datasets

  • Add batch(by_column=...) by @lhoestq in https://github.com/huggingface/datasets/pull/8172

    from datasets import Dataset

ds = Dataset.from_dict({"episode": [0] * 10 + [1] * 10, "frame": list(range(10)) * 2})
# ds = ds.to_iterable_dataset()
ds = ds.batch(by_column="episode")
for x in ds:
    print(x)
# {'episode': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'frame': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]}
# {'episode': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'frame': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]}
New supported formats
Other improvements and bug fixes
New Contributors

Full Changelog: https://github.com/huggingface/datasets/compare/4.8.5...5.0.0

View originalPermalink