Datasets

AI

Loads and processes machine-learning datasets of any size from one API.

Latest 5.0.1 · by Hugging FaceWebsitehuggingface/datasets

Release activity

Release activity — 10 releases across 9 days since Feb 27, 2026. Each cell is one day; darker means more releases that day. Nothing is recorded before Feb 27, 2026. Older weeks are hidden at this screen width.
MayJunJulAug
SundayNo releases on Apr 26, 2026No releases on May 3, 2026No releases on May 10, 2026No releases on May 17, 2026No releases on May 24, 2026No releases on May 31, 2026No releases on Jun 7, 2026No releases on Jun 14, 2026No releases on Jun 21, 2026No releases on Jun 28, 2026No releases on Jul 5, 2026No releases on Jul 12, 2026No releases on Jul 19, 2026No releases on Jul 26, 2026No releases on Aug 2, 2026No releases on Aug 9, 2026
Monday1 release on Apr 27, 2026No releases on May 4, 2026No releases on May 11, 2026No releases on May 18, 2026No releases on May 25, 2026No releases on Jun 1, 2026No releases on Jun 8, 2026No releases on Jun 15, 2026No releases on Jun 22, 2026No releases on Jun 29, 2026No releases on Jul 6, 2026No releases on Jul 13, 2026No releases on Jul 20, 2026No releases on Jul 27, 2026No releases on Aug 3, 2026No releases on Aug 10, 2026
TuesdayNo releases on Apr 28, 2026No releases on May 5, 2026No releases on May 12, 2026No releases on May 19, 2026No releases on May 26, 2026No releases on Jun 2, 2026No releases on Jun 9, 2026No releases on Jun 16, 2026No releases on Jun 23, 2026No releases on Jun 30, 2026No releases on Jul 7, 2026No releases on Jul 14, 2026No releases on Jul 21, 20261 release on Jul 28, 2026No releases on Aug 4, 2026
WednesdayNo releases on Apr 29, 2026No releases on May 6, 2026No releases on May 13, 2026No releases on May 20, 2026No releases on May 27, 2026No releases on Jun 3, 2026No releases on Jun 10, 2026No releases on Jun 17, 2026No releases on Jun 24, 2026No releases on Jul 1, 2026No releases on Jul 8, 2026No releases on Jul 15, 2026No releases on Jul 22, 2026No releases on Jul 29, 2026No releases on Aug 5, 2026
ThursdayNo releases on Apr 30, 2026No releases on May 7, 2026No releases on May 14, 2026No releases on May 21, 2026No releases on May 28, 2026No releases on Jun 4, 2026No releases on Jun 11, 2026No releases on Jun 18, 2026No releases on Jun 25, 2026No releases on Jul 2, 2026No releases on Jul 9, 2026No releases on Jul 16, 2026No releases on Jul 23, 2026No releases on Jul 30, 2026No releases on Aug 6, 2026
FridayNo releases on May 1, 2026No releases on May 8, 2026No releases on May 15, 2026No releases on May 22, 2026No releases on May 29, 20261 release on Jun 5, 2026No releases on Jun 12, 2026No releases on Jun 19, 2026No releases on Jun 26, 2026No releases on Jul 3, 2026No releases on Jul 10, 2026No releases on Jul 17, 2026No releases on Jul 24, 2026No releases on Jul 31, 2026No releases on Aug 7, 2026
SaturdayNo releases on May 2, 2026No releases on May 9, 2026No releases on May 16, 2026No releases on May 23, 2026No releases on May 30, 2026No releases on Jun 6, 2026No releases on Jun 13, 2026No releases on Jun 20, 2026No releases on Jun 27, 2026No releases on Jul 4, 2026No releases on Jul 11, 2026No releases on Jul 18, 2026No releases on Jul 25, 2026No releases on Aug 1, 2026No releases on Aug 8, 2026

10 releases since Feb 27, 2026, busiest day 2

Changelog

5.0.1

Added 3
  • Support hermes traces
  • Support droid agent traces
  • Support batched=True in Dataset.to_dict
Fixed 16
  • Fix version string in __init__.py
  • Fix conda build
  • Fix JSON loader schema inference for files starting with a UTF-8 BOM
  • Fix batch(by_column=...) crashing after shard/shuffle/split
  • Fix traces streaming
  • Fix lance auth
Security 2
  • Fix symlink-following arbitrary file write in archive extraction
  • Fix path traversal via metadata file_name in folder-based builders
Bug fixes
Docs
New Contributors

Full Changelog: https://github.com/huggingface/datasets/compare/5.0.0...5.0.1

View originalPermalink
How 5.0.1 went

5.0.0

Added 8
  • Parse Agent traces messages for SFT using the teich library to enable training on traces from claude_code, pi, codex and other sources
  • Use multiple input shards for shuffle buffer in streaming mode with configurable max_buffer_input_shards parameter
  • Add batch(by_column=...) method to batch examples by a specified column, useful for robotics datasets
  • Add Apache Iceberg format support
  • Add TsFile (Apache IoTDB) packaged builder with per-device wide format
  • Add 3D mesh support and MeshFolder builder
  • Add .conll / .conllu dataset format loader for CoNLL-2003, 2000, and U formats
  • Add num_proc argument to Dataset.to_sql
Changed 3
  • Default shuffling mechanism now uses multiple input shards instead of single shard
  • Support fsspec 2026.4.0
  • Pass library_name and version to HfApi in dataset push and delete paths
Fixed 6
  • Fix storage_options lookup for streaming Lance datasets
  • Fix Parquet streaming hangs at the end of script
  • Fix parquet reshard
  • Fix parquet columns argument
  • Fix progress bar exceeding total when load_from_cache_file=False in map operation
  • Fix single lance file from pylance 7.0
Datasets Features
Agent traces

  • Parse Agent traces messages for SFT using teich by @lhoestq in https://github.com/huggingface/datasets/pull/8232

    • Agent traces from claude_code/pi/codex and others can now be loaded with load_dataset
    • Using the teich library (new optional dependency), traces are parsed to messages to enable training on traces using e.g. trl
    • Load the data:
    >>> from datasets import load_dataset
>>> ds = load_dataset("lhoestq/agent-traces-example", split="train")
>>> ds[0]["messages"]
[{'role': 'user', 'content': 'Download a random dataset from Hugging Face, use DuckDB to inspect it, and come back with a short report about it. Be concise and include: dataset name, what files/format you found, row count or rough size if you can determine it,...'
 ...]
    • Train on agent traces:
    trl sft --dataset-name lhoestq/agent-traces-example ...
Next-level shuffling in streaming mode

  • Use multiple input shards for shuffle buffer by @lhoestq in https://github.com/huggingface/datasets/pull/8194

    ds = load_dataset(..., streaming=True)
ds = ds.shuffle(seed=42)
# or configure local buffer shuffling manually, default is:
ds = ds.shuffle(seed=42, buffer_size=1000, max_buffer_input_shards=10)

    before👎:

    after✨:

    toy example comparison

    from datasets import IterableDataset

ds = IterableDataset.from_dict({"i": range(123_456_789)}, num_shards=1024)
ds = ds.shuffle(seed=42)

print("Cold start ids:")
print(list(ds.take(10)["i"]))
print("Nominal regime ids:")
print(list(ds.skip(10_000).take(10)["i"]))

    before👎:

    Cold start ids:
[6148853, 6149537, 6149418, 6149202, 6149197, 6149622, 6148849, 6149461, 6148965, 6148858]
Nominal regime ids:
[6149537, 6149418, 6149202, 6149197, 6149622, 6148849, 6149461, 6148965, 6148858, 6149290]

    after✨:

    Cold start ids:
[7836668, 9283505, 95847927, 482299, 9283471, 482341, 112003312, 59920157, 43764666, 95847871]
Nominal regime ids:
[9283505, 95847927, 482299, 9283471, 482341, 112003312, 59920157, 43764666, 95847871, 16758448]

    Note: ds.state_dict() and ds.load_state_dict() are still supported for this improved shuffling :) enabling dataset checkpointing

    Note 2: it uses threads to fetch the first examples in parallel from the input shards

    Note 3: This is a BREAKING CHANGE: the default shuffling mechanism now uses multiple input shards. You can get the old mechanism by passing max_buffer_input_shards=1 to IterableDataset.shuffle()

New batching features for robotics datasets

  • Add batch(by_column=...) by @lhoestq in https://github.com/huggingface/datasets/pull/8172

    from datasets import Dataset

ds = Dataset.from_dict({"episode": [0] * 10 + [1] * 10, "frame": list(range(10)) * 2})
# ds = ds.to_iterable_dataset()
ds = ds.batch(by_column="episode")
for x in ds:
    print(x)
# {'episode': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'frame': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]}
# {'episode': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'frame': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]}
New supported formats
Other improvements and bug fixes
New Contributors

Full Changelog: https://github.com/huggingface/datasets/compare/4.8.5...5.0.0

View originalPermalink
How 5.0.0 went

4.8.5

Added 1
  • Parse agent traces
Changed 1
  • Don't include files list in DatasetInfo
Fixed 9
  • Decode Json() values before calling DataFrame.to_json()
  • Decode JSON type before to_list or to_dict is called
  • Fix batching for table-formatted datasets
  • Fix iterable map resume state
  • Don't embed remote files in download_and_prepare to parquet
  • Fix mask in embed_storage for remote files
Removed 1
  • Remove print statement in JSON processing
Main bug fixes
Other improvements and bug fixes
New Contributors

Full Changelog: https://github.com/huggingface/datasets/compare/4.8.4...4.8.5

View originalPermalink
How 4.8.5 went

4.8.4

Changed 1
  • Support latest torchvision
Fixed 1
  • Fix regression when loading JSON with one file = one object
What's Changed

Full Changelog: https://github.com/huggingface/datasets/compare/4.8.3...4.8.4

View originalPermalink
How 4.8.4 went

4.8.0

Added 4
  • Read and write from HF Storage Buckets to load raw data, process, and save to Dataset Repos
  • Support hf:// paths for loading data files from Storage Buckets
  • Add max_shard_size parameter to IterableDataset.push_to_hub
  • Add more arrow-native iterable operations for IterableDataset
Changed 3
  • Improve support of glob patterns in archives, e.g. zip://*.jsonl::hf://datasets/username/dataset-name/data.zip
  • Bump dill and multiprocess versions to support Python 3.14
  • Improve error message for invalid data_files pattern format
Fixed 4
  • Fix multiprocessed push_to_hub on macOS that was causing segfault by using spawn instead of fork
  • Fix reshard_data_sources
  • Fix null filling in missing jsonl columns
  • Fix to_pandas, videofolder, and load_dataset_builder kwargs in iterable datasets
Dataset Features

  • Read (and write) from HF Storage Buckets: load raw data, process and save to Dataset Repos by @lhoestq in https://github.com/huggingface/datasets/pull/8064

    from datasets import load_dataset
# load raw data from a Storage Bucket on HF
ds = load_dataset("buckets/username/data-bucket", data_files=["*.jsonl"])
# or manually, using hf:// paths
ds = load_dataset("json", data_files=["hf://buckets/username/data-bucket/*.jsonl"])
# process, filter
ds = ds.map(...).filter(...)
# publish the AI-ready dataset
ds.push_to_hub("username/my-dataset-ready-for-training")

    This also fixes multiprocessed push_to_hub on macos that was causing segfault (now it uses spawn instead of fork). And it bumps dill and multiprocess versions to support python 3.14

  • Datasets streaming iterable packaged improvements and fixes by @Michael-RDev in https://github.com/huggingface/datasets/pull/8068

    • added max_shard_size to IterableDataset.push_to_hub (but requires iterating twice to know the full dataset twice - improvements are welcome)
    • more arrow-native iterable operations for IterableDataset
    • better support of glob patterns in archives, e.g. zip://*.jsonl::hf://datasets/username/dataset-name/data.zip
    • fixes for to_pandas, videofolder, load_dataset_builder kwargs
What's Changed
New Contributors

Full Changelog: https://github.com/huggingface/datasets/compare/4.7.0...4.8.0

View originalPermalink
How 4.8.0 went

4.7.0

Added 2
  • Add Json() type to support JSON Lines files with arbitrary JSON objects and mixed types in features, usable with load_dataset(), .map(), .cast(), .from_dict(), and .from_list()
  • Add on_mixed_types="use_json" parameter to .from_dict(), .from_list(), and .map() to automatically use Json() type for mixed types
Changed 2
  • Use num_examples instead of len(self) for iterable_dataset's SplitInfo
  • Limit dataset listing to first 20 entries in readme
Fixed 6
  • Fix silent data loss in push_to_hub when num_proc > num_shards
  • Fix non-deterministic behavior by sorting metadata extensions
  • Preserve features when chaining filter() on typed IterableDataset
  • Handle nested null types in feature alignment for multi-proc map
  • Fix unstable tokenizer fingerprinting to enable map cache reuse
  • Don't extract bad files
Datasets Features
  • Add Json() type by @lhoestq in https://github.com/huggingface/datasets/pull/8027
    • JSON Lines files that contain arbitrary JSON objects like tool calling datasets are now supported. When there is a field or subfield containing mixed types (e.g. mix of str/int/float/dict/list or dictionaries with arbitrary keys), the Json()type is used to store such data that would normally not be supported in Arrow/Parquet
    • Use the Json() type in Features() for any dataset, it is supported in any functions that accepts features=like load_dataset(), .map(), .cast(), .from_dict(), .from_list()
    • Use on_mixed_types="use_json" to automatically set the Json() type on mixed types in .from_dict(), .from_list() and .map()

Examples:

You can use on_mixed_types="use_json" or specify features= with a [Json] type:

>>> ds = Dataset.from_dict({"a": [0, "foo", {"subfield": "bar"}]})
Traceback (most recent call last):
  ...
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Could not convert 'foo' with type str: tried to convert to int64

>>> features = Features({"a": Json()})
>>> ds = Dataset.from_dict({"a": [0, "foo", {"subfield": "bar"}]}, features=features)
>>> ds.features
{'a': Json()}
>>> list(ds["a"])
[0, "foo", {"subfield": "bar"}]

This is also useful for lists of dictionaries with arbitrary keys and values, to avoid filling missing fields with None:

>>> ds = Dataset.from_dict({"a": [[{"b": 0}, {"c": 0}]]})
>>> ds.features
{'a': List({'b': Value('int64'), 'c': Value('int64')})}
>>> list(ds["a"])
[[{'b': 0, 'c': None}, {'b': None, 'c': 0}]]  # missing fields are filled with None

>>> features = Features({"a": List(Json())})
>>> ds = Dataset.from_dict({"a": [[{"b": 0}, {"c": 0}]]}, features=features)
>>> ds.features
{'a': List(Json())}
>>> list(ds["a"])
[[{'b': 0}, {'c': 0}]]  # OK

Another example with tool calling data and the on_mixed_types="use_json" argument (useful to not have to specify features= manually):

>>> messages = [
...     {"role": "user", "content": "Turn on the living room lights and play my electronic music playlist."},
...     {"role": "assistant", "tool_calls": [
...         {"type": "function", "function": {
...             "name": "control_light",
...             "arguments": {"room": "living room", "state": "on"}
...         }},
...         {"type": "function", "function": {
...             "name": "play_music",
...             "arguments": {"playlist": "electronic"}  # mixed-type here since keys ["playlist"] and ["room", "state"] are different
...         }}]
...     },
...     {"role": "tool", "name": "control_light", "content": "The lights in the living room are now on."},
...     {"role": "tool", "name": "play_music", "content": "The music is now playing."},
...     {"role": "assistant", "content": "Done!"}
... ]
>>> ds = Dataset.from_dict({"messages": [messages]}, on_mixed_types="use_json")
>>> ds.features
{'messages': List({'role': Value('string'), 'content': Value('string'), 'tool_calls': List(Json()), 'name': Value('string')})}
>>> ds[0][1]["tool_calls"][0]["function"]["arguments"]
{"room": "living room", "state": "on"}
What's Changed
New Contributors

Full Changelog: https://github.com/huggingface/datasets/compare/4.6.1...4.7.0

View originalPermalink
How 4.7.0 went

More in AI

View all
codex

codex release notes.

codex
Ollama

Get up and running with large language models locally.

Ollama
openclaw

Your own personal AI assistant. Any OS. Any Platform. The lobster way. 🦞

openclaw
superpowers

An agentic skills framework & software development methodology that works.

obra
ECC

The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.

affaan-m
hermes-agent

The agent that grows with you

NousResearch

Discussion