Ray 2.58.0

Highlights

• Ray Serve LLM: In this release we've completed KV cache and token aware request routing, which was previewed in 2.57. Tokenization now happens in-process on the LLMRouter ingress replica, the routing decision is made there, tokens are transmitted out-of-band so the engine does not re-tokenize, KV lifecycle events are broadcast to every ingress replica (#64642, #64920, #64949, #65010, #65095). KV cache and token aware routing is also aware of CPU KV caches, so offloaded KV cache blocks count toward a replica's cache hit (#65063).
• Ray Core: We enabled the capability to offload task events from. With RAY_enable_task_events_to_dashboard_head on, the task event buffer is replaced by the ray event recorder, events are exported from the aggregator agent to a task events head that keeps an in-memory store, and the state APIs and ray.timeline read from it (#64835, #65028, #65123, #65160, #65218). Enabling the feature removes task event ingestion and serving from the GCS hot path.
• Ray Data: We’ve added Databricks integrations for writing to DeltaLake, with Catalog support. We’ve also shipped a new shuffle v2 backend, featuring improved performance for joins and aggregations.
• Sandboxing: We've also added experimental Ray Sandbox, which runs task and actor code under gVisor and can run Docker-built images directly (#64964, #65570).
• TPU Support: Ray Train adds support for TorchTPU backend (#64796), and Ray Core adds SubslicePlacementGroup for gang scheduling on TPU subslices, single-host TPU support in SlicePlacementGroup, and resource accounting for tpu7x and multi-core chips (#64578, #64079, #64058). This lets TPU slices and subslices be reserved and trained on without external gang-scheduling glue.

Ray Data

🎉 New Features

• Add Dataset.with_columns for multi-column expression projection (#63858)
• Add write_delta for Delta Lake, with catalog support (#64923, #65079)
• Add Torch inference API (#65157)
• Promote hash shuffle v2 to a selectable shuffle strategy, with aggregation support, vectorized aggregation, and block splitting during aggregation (#64953, #64652, #64956, #65329, #64897)
• Add ignore_missing_paths and skip_paths to read_parquet on DatasourceV2 (#65118)
• Add delta_timestamps (temporal windows) to read_lerobot (#64877)
• Collect cluster usage metrics by sampling in background threads during execution (#64686)
• Tolerate actor deaths during init via DataContext.max_consecutive_actor_init_deaths (#64846)
• Expose RAY_DATA_HASH_SHUFFLE_MAP_TASK_TARGET_INPUT_BYTES in DataContext (#65103)

💫 Enhancements

• Generate sortable, collision-resistant dataset IDs (#65075)
• Push Limit into ReadFiles when it sits directly on it, and extract FileIndexer.list_file_infos (#65167, #65168)
• Use BlockRefCounter for object store memory estimation and remove BlockRefCounter.clear() (#64456, #64521)
• Fail execution if no operator makes progress within a timeout, instead of hanging (#65349)
• Improve BatchIterator and iter_torch_batches, and allow a custom collate_fn with a custom device (#64994, #64967)
• Skip downstream-capacity backpressure for eligible materializers (#64844)
• Shard exact download partitioning (#64066)
• Bump the sort_reduce memory multiplier to 3x and stop using estimated_input_blocks as the shuffle partition count (#65176, #65296, #65335)
• Disable cluster autoscaling when PlacementGroupSchedulingStrategy is in use (#64417)
• Propagate the operator name to shuffle tasks (#64802)
• Rename reports_custom_op_stats to should_report_custom_op_stats across all MapTransformFn variants (#64461, #64515)
• Rename allocated to reserved and add type aliases in the autoscaling coordinator (#64997, #65096)
• Deprecate ray_remote_args_fn and Dataset.zip (#64963, #65111)

🔨 Fixes

• Fix an RCE where read_lance or nested pickle objects could execute arbitrary code (#64881)
• Stop converting Arrow null columns to null[pyarrow] in to_pandas (#65187)
• Fix iter_torch_batches device resolution and typing (#65059, #64947)
• Propagate isolate_read_workers to DatasourceV2 (#65191)
• Avoid signalling epoch end for failed attempts (#65082)
• Add tf-keras to the text_embedding pip packages (#64889)

📖 Documentation

• Fix the stale use_datasource_v2 docstring default and an incorrect default_map_logical_memory_enabled reference (#65155, #65091)

Ray Serve

🎉 New Features

• Configurable status code and Retry-After header for backpressure rejections (#65193, #65319)
• Add dependency-ordered shutdown for deployments (#64922)
• Add an optional tie-break key to best-fit node scheduling (#64914)
• Scale ingress request router replicas per proxy node (#64724)

💫 Enhancements

• Reconcile health checks from a dirty set, sweeping RUNNING replicas round-robin instead of every tick (#64690)
• Gate the rank-consistency check on replica membership changes (#64911)
• Read the multiplex marker statically so probing cannot initialize handles (#65064)
• Type CreatePlacementGroupRequest.runtime_env as a dict (#64892)
• Add a deployment-state accessor for testing (#64790)

🔨 Fixes

• Fix the Serve replica ASGIService bypassing token authentication (#65189)
• Fix the proxy update loop getting stuck when a proxy's node is removed (#64403)

Ray Train

🎉 New Features

• Ray Train Integrates with TorchTPU backend (#64796)
• Add a public preemption API and a controller PreemptingState (#64360)
• Add data ingest metrics to the Train dashboard (#64523)

💫 Enhancements

• Make Train V1 and V2 use the autoscaling coordinator (#64824)
• Expand the contains_tensor check and add a serialization check for the results return value (#64930)
• Pin PlacementGroupCleaner to the head node (#64705)

🔨 Fixes

• Fix Torch environment setup for V1 worker groups (#65005)

Ray Tune

🔨 Fixes

• Fix HyperOptSearch dropping tune.choice categories that are constant dicts (#64537)
• Deflake test_multi_trial_reuse_with_failing and decide test_experiment_restore completion from measured progress (#64526, #65212)

Ray LLM

🎉 New Features

• KV-cache-aware routing: move tokenization into the LLMRouter ingress replica, decide KV/token routing there, broadcast KV lifecycle events to all ingress replicas, and make selection and reservation atomic (#64642, #64920, #64949, #65010)
• Enable KV cache offloading, make KV routing aware of CPU KV caches, and transmit tokens out-of-band so the engine skips tokenization (#65063, #65095)
• Add a KV cache offload/reload dashboard and a Ray Serve LLM SGLang metrics dashboard (#65122, #64797)

💫 Enhancements

• Upgrade to vLLM 0.26.0 (#65045)
• Route direct-streaming ingress to the co-located router (#64489)
• Reuse vLLM's resolved HF config in apply_checkpoint_info (#62962)
• Preserve mapping-valued vLLM frontend arguments (#65146)

📖 Documentation

• Document loading models from Azure storage, including az:// Blob streaming with RunAI Streamer (#64819, #64825)

Ray RLlib

🔨 Fixes

• Fix TQC critic divergence by stopping actor-loss gradients from leaking into the critics (#65125)
• Fix squashed-Gaussian log-prob corruption for saturated policies (#65036)
• Use the target_qf_twin head in IQL target prediction (#64932)
• Properly enforce use_kl_loss in the PPO Torch and TF policies (#61562)
• Fix a KeyError in the multi-agent module-to-env connector (#64803)
• Fix the API-doc consistency check for the new reverse/dedup policies, and deflake the test_env_runner callback-count tests (#64807, #64989)

Ray Core

🎉 New Features

• Ray Sandbox (experimental): run task and actor code under gVisor, and run Docker-built images out of the box (#64964, #65397, #65570, #65622)
• Move task events out of the GCS: the ray event recorder replaces the task event buffer, the aggregator agent exports to a task events head with an in-memory store, and the state APIs and ray.timeline are rerouted to it, with reconciliation on worker death and job completion (#64835, #65028, #65057, #65123, #65141, #65160, #65218, #65247, #65288)
• Add SubslicePlacementGroup for gang scheduling on TPU subslices, support single-host TPUs in SlicePlacementGroup, and add a per_slice_pgs parameter (#64578, #64079, #64072)
• Introduce a native, lightweight C++ leader election client for active-passive GCS (#63773)
• Add Apple silicon GPU (mps) support and an Intel GPU ZE_AFFINITY_MASK mapping (#38464, #64440)
• Add worker lifecycle events to the events export pipeline (#64887)
• Add GPU UUID to the labels of GPU metrics (#65113)
• Enable resource accounting for tpu7x and multi-core chips, add gb200/gb300 accelerator constants, and add TTNPU custom accelerator resources (#64058, #65009, #61554)
• Support cross-device transfers in RDT NIXL (#64815)

💫 Enhancements

• Publish node death before persisting it and drop RocksDB soft durability (REP-64) (#64702)
• Make RedisContext::Connect non-fatal on connection failure (#64299)
• Move the pending resource load pull off the GCS main io_context (#65024)
• Subscribe only to the specific owner worker's death for generator backpressure, and only when actor-level backpressure is enabled (#65195, #65136)
• Evict dead actors from ActorPool instead of recycling them (#64646)
• Rename label_domain to topology strategy in the scheduling policy (#64384)
• Warn on use of the deprecated dynamic generator (#64749)
• Add object resolution debug logging for lineage reconstruction (#64853)
• Block only on the CUDA stream used to create tensors in RDT (#64823)
• Preserve StateSchema column order in filter_fields (#65052)
• Prepare StreamResponse on an empty log stream (#62296)
• Refine ObjectRefStreamEndOfStreamError from _get_next_ref_n (#64602)
• Drop a redundant FunctionDescriptor rebuild in CallSiteString (#64874)
• Upgrade bundled dependencies: log4j 2.25.4, jackson-databind 2.18.8 (CVE-2026-54512, CVE-2026-54513), gson 2.11.0, aiohttp, idna, and azure (#64269, #64575, #64273, #65131, #64056, #65046)

🔨 Fixes

• Give canceled_tasks_ its own mutex to break a lock-order cycle (#65393, #65620)
• Fix a GIL/mutex deadlock in actor-level backpressure with sync and async streaming generators (#64896)
• Fix a deadlock between metric registration and collect() in OpenTelemetryMetricRecorder (#64946)
• Report shutdown from check_signals instead of exiting the process (#65184, #65400)
• Fix a Python 3.14 async-actor memory leak by re-anchoring stack protection to fiber stacks (#64772)
• Keep only the latest object-location pubsub snapshot to fix an owner memory leak (#65133)
• Fix a spurious OwnerDiedError during graceful raylet shutdown (#64899)
• Fail ray.get on refs from a non-restartable streaming generator when those objects are lost (#64756)
• Serve the spilled copy when Push hits a stale local_objects_ mirror (#64916)
• Fix spill_manager_objects_bytes reporting the restored object count instead of restored bytes (#65013)
• Forward node-pinned actors to the pinned node in GCS actor scheduling (#64951)
• Fix a leaked named actor and name conflict when registration times out (#64948)
• Fix a set-before-register race in RDTManager that could SIGSEGV (#64558)
• Fix task log info fields in lifecycle events (#65190)
• Guard Status::operator<< against an OK status, and fix UB in StatusOr swap and assignment on error-state operands (#64983, #64799)
• Compare all ResourceRequest fields in operator== (#64838)
• Verify VFIO groups are backed by Google TPU PCI devices, and use POSIX paths for VFIO sysfs vendor checks on Windows (#65105, #65182)

📖 Documentation

• Document the embedded RocksDB GCS backend (#64731)
• Clarify actor class state isolation and the Ray Core walkthrough benchmark setup (#64597, #64529)
• Document how to set up placement group topology strategy on Kubernetes (#64117)

Dashboard

🎉 New Features

• Capture Kubernetes Pod events in the Ray Dashboard head (#63937)

💫 Enhancements

• Hide the GPU and GRAM columns when no GPUs are present (#64567)
• Update axios to ^1.18.0 (#65130)

🔨 Fixes

• Fix the profiling status check so it works behind a reverse proxy (#65126)
• Fix a dashboard startup crash from an unguarded kubernetes import (#64962)

📖 Documentation

• Add a user guide for Kubernetes events in the Ray Dashboard (#64734)

Ray Wheels and Images

🎉 New Features

• Enable Windows py3.13 and py3.14 wheel builds (#64970)
• Publish arm64 Ray LLM images (#65002)
• Add a ray-torch release test image (py3.14, cu12.8) and a hello_world_py314 smoke release test (#65114, #64857)

💫 Enhancements

• Bump the Anyscale CLI to 0.26.105 and bake pybase64 into the ML release-test image (#64980, #64791)
• Recompile dependency lock files on dependabot PRs, and regenerate the ray-torch py3.14 lock for aiohttp 3.14.3 (#65056, #65343, #65378)
• Pin grpcio to 1.75.0 in test deps to avoid the grpc.aio performance regression (#65112)
• Give ray-wheel-minimal-build a distinct wanda image name (#65318, #65383)
• Move cu130 job tests off g4dn.4xlarge to g6.4xlarge (#65213)
• Bump the version to 2.58.0 and publish 2.56.0 perf metrics (#65252, #64196)

🔨 Fixes

• Fix org_lzma_lzma download failures by using the SourceForge redirector (#64906)
• Install the data CI depset after conda ffmpeg so removed packages are restored (#65334, #65341)
• Drop -Wl,-pie from the vendored RocksDB WITH_TSAN link flags (#64917)
• Floor peft>=0.18 for transformers 5.x in the huggingface_accelerate release test (#65062)

Documentation

• Add initial documentation for Ray sandboxing (#65503, #65573)
• Update the Ray History Server docs for RAY_ROOT_DIR -> STORAGE_ROOT_DIR and use a RayJob sample YAML (#65139, #65441, #65510, #65505, #65531)
• Add a contributor guide for editing and managing Python dependencies (#63547)
• Clarify the API deprecation policy (#65093)
• Document the safe-to-evict annotation for the Ray head Pod, and autoscaler v1 restartPolicy behavior (#64907, #64900)
• Update the Gaudi tutorials and examples to the latest versions (#58861)
• Correct the TLS verification comments and the metric cardinality default comments (#61977, #64478)
• Route API reference pages and autodoc machinery to the API-surface checks, ignore inherited API annotations, walk ray.data.llm as its own head module, and reserve the doc tag for doc validation (#64812, #65196, #65040, #64775, #65208)
• Scope API signature bold weight to the object name, and reclassify sphinx unknown-document and docutils-inline-markup as judgment (#64933, #64839)
• Fix minor typos, bump the docs template build-id pins, and add Douglas Strodtman to the committer list (#64864, #65041, #65092)

Thanks

Many thanks to all those who contributed to this release!

@bveeramani, @yjaw, @sampan-s-nayak, @spencer-p, @iaalm, @risjai, @kahlun, @liulehui, @eicherseiji, @jhasm, @ronny-anyscale, @KuongB, @Ranoobaba, @Sparks0219, @alimaazamat, @yuhuan130, @jeffreywang88, @AarryaSaraf, @johntomcat7408-cmyk, @tqKhanh1712, @praneethkaturi, @elliot-barn, @tanmayrauth, @skpark-rh, @YashwanthRanjanSingaravel, @Yicheng-Lu-llll, @pseudo-rnd-thoughts, @nadongjun, @vineethsaivs, @owenowenisme, @saitejabandaru-in, @RocMarshal, @karticam, @win5923, @Hyunoh-Yeo, @dragongu, @YoyinZyc, @chiayi, @HrushiYadav, @martinlhw, @sai-miduthuri, @dstrodtman, @JasonLi1909, @2uchan, @mukktinaadh, @coqian, @ayushk7102, @LuciferYang, @NripeshN, @MortalHappiness, @verma-divyanshu-git, @ans9868, @johntaylor-cell, @prasad-anyscale, @edoakes, @ShockYoungCHN, @fscnick, @JiangJiaWei1103, @subpath, @shivamsingh-007, @RinZ27, @richabanker, @iamjustinhsu, @malsbat, @andrewsykim, @ryanaoleary, @vinay7373, @Kunchd, @petern48, @kyuds, @rueian, @rayhhome, @xyuzh, @hao-aaron, @CaiZhanqi, @kalyanamdewri, @abhishekverma-ray, @khluu, @nh-atuan, @odncode, @robertnishihara, @machichima, @Arkit003, @goutamvenkat-anyscale, @ArturNiederfahrenhorst, @Myasuka