Ray

Ray Libraries<a id="ray-libraries"></a>

Ray Data<a id="ray-data"></a>

🎉 New Features:
- Add `Dataset.rename_columns` (47906)
- Basic structured logging (47210)

💫 Enhancements:
- Add `partitioning` parameter to `read_parquet` (47553)
- Add `SERVICE_UNAVAILABLE` to list of retried transient errors (47673)
- Re-phrase the streaming executor current usage string (47515)
- Remove ray.kill in ActorPoolMapOperator (47752)
- Simplify and consolidate progress bar outputs (47692)
- Refactor `OpRuntimeMetrics` to support properties (47800)
- Refactor `plan_write_op` and `Datasink`s (47942)
- Link `PhysicalOperator` to its `LogicalOperator` (47986)
- Allow specifying both `num_cpus` and `num_gpus` for map APIs (47995)
- Allow specifying insertion index when registering custom plan optimization `Rule`s (48039)
- Adding in better framework for substituting logging handlers (48056)

🔨 Fixes:
- Fix bug where Ray Data incorrectly emits progress bar warning (47680)
- Yield remaining results from async `map_batches` (47696)
- Fix event loop mismatch with async map (47907)
- Make sure `num_gpus` provide to Ray Data is appropriately passed to `ray.remote` call (47768)
- Fix unequal partitions when grouping by multiple keys (47924)
- Fix reading multiple parquet files with ragged ndarrays (47961)
- Removing unneeded test case (48031)
- Adding in better json checking in test logging (48036)
- Fix bug with inserting custom optimization rule at index 0 (48051)
- Fix logging output from `write_xxx` APIs (48096)

📖 Documentation:
- Add docs section for Ray Data progress bars (47804)
- Add reference to parquet predicate pushdown (47881)
- Add tip about how to understand map_batches format (47394)

Ray Train<a id="ray-train"></a>

🏗 Architecture refactoring:
- Remove deprecated mosaic and sklearn trainer code (47901)

Ray Tune<a id="ray-tune"></a>

🔨 Fixes:
- Fix WandbLoggerCallback to reuse actors upon restore (47985)

Ray Serve<a id="ray-serve"></a>

🔨 Fixes:
- Stop scheduling task early when requests have been canceled (47847)

RLlib<a id="rllib"></a>

🎉 New Features:
- Enable cloud checkpointing. (47682)

💫 Enhancements:
- PPO on new API stack now shuffles batches properly before each epoch. (47458)
- Other enhancements: 47705, 47501, 47731, 47451, 47830, 47970, 47157

🔨 Fixes:
- Fix spot node preemption problem (RLlib now run stably with EnvRunner workers on spot nodes) (47940)
- Fix action masking example. (47817)
- Various other fixes: 47973, 46721, 47914, 47880, 47304, 47686

🏗 Architecture refactoring:
- Switch on new API stack by default for SAC and DQN. (47217)
- Remove Tf support on new API stack for PPO/IMPALA/APPO (only DreamerV3 on new API stack remains with tf now). (47892)
- Discontinue support for "hybrid" API stack (using RLModule + Learner, but still on RolloutWorker and Policy) (46085)
- RLModule (new API stack) refinements: 47884, 47885, 47889, 47908, 47915, 47965, 47775

📖 Documentation:
- Add new API stack migration guide. (47779)
- New API stack example script: BC pre training, then PPO finetuning using same RLModule class. (47838)
- New API stack: Autoregressive actions example. (47829)
- Remove old API stack connector docs entirely. (47778)

Ray Core and Ray Clusters
Ray Core <a id="ray-core"></a>

🎉 New Features:
- CompiledGraphs: support multi readers in multi node when DAG is created from an actor (47601)

💫 Enhancements:
- Add a flag to raise exception for out of band serialization of `ObjectRef` (47544)
- Store each GCS table in its own Redis Hash (46861)
- Decouple create worker vs pop worker request. (47694)
- Add metrics for GCS jobs (47793)

🔨 Fixes:
- Fix broken dashboard cluster page when there are dead nodes (47701)
- Fix the `ray_tasks{State="PENDING_ARGS_FETCH"}` metric counting (47770)
- Separate the attempt_number with the task_status in memory summary and object list (47818)
- Fix object reconstruction hang on arguments pending creation (47645)
- Fix check failure: `sync_reactors_.find(reactor->GetRemoteNodeID()) == sync_reactors_.end()` (47861)
- Fix check failure `RAY_CHECK(it != current_tasks_.end())`; (47659)

📖 Documentation:
- KubeRay docs: Add docs for YuniKorn Gang scheduling 47850

Dashboard<a id="dashboard"></a>

💫 Enhancements:
- Performance improvements for large scale clusters (47617)

🔨 Fixes:
- Placement group and required resources not showing correctly in dashboard (47754)

Thanks

Many thanks to all those who contributed to this release!
GeneDer, rkooo567, dayshah, saihaj, nikitavemuri, bill-oconnor-anyscale, WeichenXu123, can-anyscale, jjyao, edoakes, kekulai-fredchang, bveeramani, alexeykudinkin, raulchen, khluu, sven1977, ruisearch42, dentiny, MengjinYan, Mark2000, simonsays1980, rynewang, PatricYan, zcin, sofianhnaide, matthewdeng, dlwh, scottjlee, MortalHappiness, kevin85421, win5923, aslonnie, prithvi081099, richardsliu, milesvant, omatthew98, Superskyyy, pcmoritz

Ray Libraries<a id="ray-libraries"></a>

Ray Data<a id="ray-data"></a>
💫 Enhancements:
- Simplify custom metadata provider API (47575)
- Change counts of metrics to rates of metrics (47236)
- Throw exception for non-streaming HF datasets with "override_num_blocks" argument (47559)
- Refactor custom optimizer rules (47605)

🔨 Fixes:
- Remove ineffective retry code in `plan_read_op` (47456)
- Fix incorrect pending task size if outputs are empty (47604)

Ray Train<a id="ray-train"></a>
💫 Enhancements:
- Update run status and add stack trace to `TrainRunInfo` (46875)

Ray Serve<a id="ray-serve"></a>
💫 Enhancements:
- Allow control of some serve configuration via env vars ([47533](https://github.com/ray-project/ray/pull/47533))
- [serve] Faster detection of dead replicas ([47237](https://github.com/ray-project/ray/pull/47237))

🔨 Fixes:
- [Serve] fix component id logging field ([47609](https://github.com/ray-project/ray/pull/47609))

RLlib<a id="rllib"></a>
💫 Enhancements:
- New API stack:
- Add restart-failed-env option to EnvRunners. ([47608](https://github.com/ray-project/ray/pull/47608)[)](https://github.com/ray-project/ray/commit/e75f5e7aa950e30097a0323f4baf14d90b1b6b9b)
- Offline RL: Store episodes in state form. ([47294](https://github.com/ray-project/ray/pull/47294)[)](https://github.com/ray-project/ray/commit/aa7179a6fa24a0d95a1c9b85014bfb322d3447e6)
- Offline RL: Replace GAE in MARWILOfflinePreLearner with `GeneralAdvantageEstimation` connector in learner pipeline. ([47532](https://github.com/ray-project/ray/pull/47532))
- Off-policy algos: Add episode sampling to EpisodeReplayBuffer. ([47500](https://github.com/ray-project/ray/pull/47500))
- RLModule APIs: Add `SelfSupervisedLossAPI` for RLModules[ that bri](https://github.com/ray-project/ray/commit/f422376cda3ae0dc52fc7686df3b1cb03342be7f)ng their own loss and `InferenceOnlyAPI`. ([#47581](https://github.com/ray-project/ray/pull/47581), [#47572](https://github.com/ray-project/ray/pull/47572))

Ray Core<a id="ray-core"></a>
💫 Enhancements:
- [aDAG] Allow custom NCCL group for aDAG (47141)
- [aDAG] support buffered input (47272)
- [aDAG] Support multi node multi reader (47480)
- [Core] Make is_gpu, is_actor, root_detached_id fields late bind to workers. (47212)
- [Core] Reconstruct actor to run lineage reconstruction triggered actor task (47396)
- [Core] Optimize GetAllJobInfo API for performance (47530)

🔨 Fixes:
- [aDAG] Fix ranks ordering for custom NCCL group (47594)

Ray Clusters<a id="ray-clusters"></a>
📖 Documentation:
- [KubeRay] add a guide for deploying vLLM with RayService (47038)

Thanks

Many thanks to all those who contributed to this release!
ruisearch42, andrewsykim, timkpaine, rkooo567, WeichenXu123, GeneDer, sword865, simonsays1980, angelinalg, sven1977, jjyao, woshiyyya, aslonnie, zcin, omatthew98, rueian, khluu, justinvyu, bveeramani, nikitavemuri, chris-ray-zhang, liuxsh9, xingyu-long, peytondmurray, rynewang

Ray Core<a id="ray-core"></a>
🔨 Fixes:
- Fix broken dashboard cluster page when there are dead nodes (47701)
- Fix broken dashboard worker page (47714)

Ray Libraries<a id="ray-libraries"></a>

Ray Data<a id="ray-data"></a>
💫 Enhancements:
- Remove limit on number of tasks launched per scheduling step (47393)
- Allow user-defined Exception to be caught. (47339)

🔨 Fixes:
- Display pending actors separately in the progress bar and not count them towards running resources (46384)
- Fix bug where `arrow_parquet_args` aren't used (47161)
- Skip empty JSON files in `read_json()` (47378)
- Remove remote call for initializing `Datasource` in `read_datasource()` (47467)
- Remove dead `from_*_operator` modules (47457)
- Release test fixes
- Add `AWS ACCESS_DENIED` as retryable exception for multi-node Data+Train benchmarks (47232)
- Get AWS credentials with boto (47352)
- Use worker node instead of head node for `read_images_comparison_microbenchmark_single_node` release test (47228)

📖 Documentation:
- Add docstring to explain `Dataset.deserialize_lineage` (47203)
- Add a comment explaining the bundling behavior for `map_batches` with default batch_size (47433)

Ray Train<a id="ray-train"></a>

💫 Enhancements:
- Decouple device-related modules and add Huawei NPU support to Ray Train (44086)

🔨 Fixes:
- Update TORCH_NCCL_ASYNC_ERROR_HANDLING env var (47292)

📖 Documentation:
- Add missing Train public API reference (47134)


Ray Tune<a id="ray-tune"></a>
📖 Documentation:
- Add missing Tune public API references (47138)


Ray Serve<a id="ray-serve"></a>
💫 Enhancements:
- Mark proxy as unready when its routers are aware of zero replicas (47002)
- Setup default serve logger (47229)

🔨 Fixes:
- Allow get_serve_logs_dir to run outside of Ray's context (47224)
- Use serve logger name for logs in serve (47205)

📖 Documentation:
- [HPU] [Serve] [experimental] Add vllm HPU support in vllm example (45893)

🏗 Architecture refactoring:
- Remove support for nested DeploymentResponses (47209)

RLlib<a id="rllib"></a>
🎉 New Features:
- New API stack: Add CQL algorithm. ([47000](https://github.com/ray-project/ray/pull/47000), [#47402](https://github.com/ray-project/ray/pull/47402))
- New API stack: Enable GPU and multi-GPU support for DQN/SAC/CQL. ([47179](https://github.com/ray-project/ray/pull/47179))

💫 Enhancements:
- New API stack: Offline RL enhancements: [47195](https://github.com/ray-project/ray/pull/47195), [#47359](https://github.com/ray-project/ray/pull/47359)
- Enhance new API stack stability: [46324](https://github.com/ray-project/ray/pull/46324), [#47196](https://github.com/ray-project/ray/pull/47196), [#47245](https://github.com/ray-project/ray/pull/47245), [#47279](https://github.com/ray-project/ray/pull/47279)
- Fix large batch size for synchronous algos (e.g. PPO) after EnvRunner failures. ([47356](https://github.com/ray-project/ray/pull/47356))
- Add torch.compile config options to old API stack. ([47340](https://github.com/ray-project/ray/pull/47340)[)](https://github.com/ray-project/ray/commit/78402bc1fde669cb4015f9393106acb144ac45bf)
- Add kwargs to torch.nn.parallel.DistributedDataParallel ([47276](https://github.com/ray-project/ray/pull/47276))
- Enhanced CI stability: [47197](https://github.com/ray-project/ray/pull/47197), [#47249](https://github.com/ray-project/ray/pull/47249)

📖 Documentation:
- New API stack example scripts:
- Float16 training example script. ([47362](https://github.com/ray-project/ray/pull/47362))
- Mixed precision training example script ([47116](https://github.com/ray-project/ray/pull/47116))
- ModelV2 -> RLModule wrapper for migrating to new API stack. ([47425](https://github.com/ray-project/ray/pull/47425))
- Remove "new API stack experimental" hint from docs. ([47301](https://github.com/ray-project/ray/pull/47301))

🏗 Architecture refactoring:
- Remove 2nd Learner ConnectorV2 pass from PPO ([47401](https://github.com/ray-project/ray/pull/47401))
- Add separate learning rates for policy and alpha to SAC. ([47078](https://github.com/ray-project/ray/pull/47078))

🔨 Fixes:
- Various bug fixes: [47401](https://github.com/ray-project/ray/pull/47401), [#47194](https://github.com/ray-project/ray/pull/47194), [#47259](https://github.com/ray-project/ray/pull/47259), [#47271](https://github.com/ray-project/ray/pull/47271), [#47277](https://github.com/ray-project/ray/pull/47277), [#47382](https://github.com/ray-project/ray/pull/47382)

Ray Core<a id="ray-core"></a>
💫 Enhancements:
- [ADAG] Raise proper error message for nccl within the same actor (47250)
- [[ADAG] Support multi-read of the same shm channel (](https://github.com/ray-project/ray/commit/c9c150a1f0460589e76415710afa5e940d834311)[#47311](https://github.com/ray-project/ray/pull/47311)[)](https://github.com/ray-project/ray/commit/c9c150a1f0460589e76415710afa5e940d834311)
- [Log why core worker is not idle during HandleExit (](https://github.com/ray-project/ray/commit/43250f4481849ae0233b96e68637b294dc43748d)[#47300](https://github.com/ray-project/ray/pull/47300)[)](https://github.com/ray-project/ray/commit/43250f4481849ae0233b96e68637b294dc43748d)
- Add PREPARED state for placement groups in GCS for better fault tolerance. ([46858](https://github.com/ray-project/ray/pull/46858))

🔨 Fixes:
- Fix ray_unintentional_worker_failures_total to only count unintentional worker failures (47368)
- Fix runtime env race condition when uploading the same package concurrently (47482)

Dashboard<a id="dashboard"></a>
🔨 Fixes:
- Performance optimizations for dashboard backend logic ([47392](https://github.com/ray-project/ray/pull/47392)) ([#47367](https://github.com/ray-project/ray/pull/47367)) ([#47160](https://github.com/ray-project/ray/pull/47160)) (#47213)
- Refactor to simplify dashboard backend logic ([47324](https://github.com/ray-project/ray/pull/47324))

Docs<a id="dashboard"></a>

💫 Enhancements:
- Add sphinx-autobuild and documentation for make local (47275): Speed up of local docs builds with `make local`.
- Add Algolia search to docs ([46477](https://github.com/ray-project/ray/pull/46477))
- Update PyTorch Mnist Training doc for KubeRay 1.2.0 ([47321](https://github.com/ray-project/ray/pull/47321))
- Life-cycle of documentation [policy](https://docs.ray.io/en/latest/ray-contribute/api-policy.html) of Ray APIs

Thanks

Many thanks to all those who contributed to this release!
GeneDer, Bye-legumes, nikitavemuri, kevin85421, MortalHappiness, LeoLiao123, saihaj, rmcsqrd, bveeramani, zcin, matthewdeng, raulchen, mattip, jjyao, ruisearch42, scottjlee, can-anyscale, khluu, aslonnie, rynewang, edoakes, zhanluxianshen, venkatram-dev, c21, allenyin55, alexeykudinkin, snehakottapalli, BitPhinix, hongchaodeng, dengwxn, liuxsh9, simonsays1980, peytondmurray, KepingYan, bryant1410, woshiyyya, sven1977

**Notice**: Starting from this release, `pip install ray[all]` will not include `ray[cpp]`, and will not install the respective `ray-cpp` package. To install everything that includes `ray-cpp`, one can use `pip install ray[cpp-all]` instead.

Ray Libraries<a id="ray-libraries"></a>

Ray Data<a id="ray-data"></a>
🎉 New Features:
- Upgrade supported Arrow version from 16 to 17 (47034)
- Add support for reading from Iceberg (46889)

💫 Enhancements:
- Various Progress Bar UX improvements (46816, 46801, 46826, 46692, 46699, 46974, 46928, 47029, 46924, 47120, 47095, 47106)
- Try get `size_bytes` from metadata and consolidate metadata methods (46862)
- Improve warning message when read task is large (46942)
- Extend API to enable passing sample weights via ray.dataset.to_tf (45701)
- Add a parameter to allow overriding LanceDB scanner options (46975)
- Add failure retry logic for read_lance (46976)
- Clarify warning for reading old Parquet data (47049)
- Move datasource implementations to `_internal` subpackage (46825)
- Handle logs from tensor extensions (46943)

🔨 Fixes:
- Change type of `DataContext.retried_io_errors` from tuple to list (46884)
- Make Parquet tests more robust and expose Parquet logic (46944)
- Change pickling log level from warning to debug (47032)
- Add validation for shuffle arg (47055)
- Fix validation bug when size=0 in ActorPoolStrategy (47072)
- Fix exception in async map (47110)
- Fix wrong metrics group for `Object Store Memory` metrics on Ray Data Dashboard (47170)
- Handle errors in SplitCoordinator when generating a new epoch (47176)

📖 Documentation:
- Auto-gen GroupedData api (46925)
- Fix signature of `Rule.plan` (47094)

Ray Train<a id="ray-train"></a>
💫 Enhancements:
- [train] Updates to support xgboost==2.1.0 (46667)
- [train] Add hardware stats (46719)

Ray Tune<a id="ray-tune"></a>
🔨 Fixes:
- [RLlib; Tune] Fix WandB metric overlap after restore from checkpoint. (46897)

Ray Serve<a id="ray-serve"></a>
💫 Enhancements:
- Improved handling of replica death and replica unavailability in deployment handle routers before controller restarts replica (47008)
- Eagerly create routers in proxy for better GCS fault tolerance (47031)
- Immediately send ping in router when receiving new replica set (47053)

🏗 Architecture refactoring:
- Deprecate passing arguments that contain `DeploymentResponses` in nested objects to downstream deployment handle calls (46806)

RLlib<a id="rllib"></a>

🎉 New Features:
- Offline RL on the new API stack:
- Record offline data (46818, 47046, 47133, 47155) and support to directly read from episodes. (46865)
- RLUnplugged example. (46792)
- Progress on BC/MARWIL migration: 44970, 47154, 46799
- Progress on CQL migration: 46969, 47105

💫 Enhancements:
- Add ObservationPreprocessor (ConnectorV2). (47077)

🔨 Fixes:
- New API stack: Fix IMPALA/APPO + LSTM for single- and multi-GPU. (47132, 47158)
- Various bug fixes: 46898, 47047, 46963, 47021, 46897
- Add more control to Algorithm.add_module/policy methods. (46932, 46836)

📖 Documentation:
- Example scripts for new API stack:
- Curiosity (inverse dynamics model-based) RLModule example. (46841)
- Add example script for Env with protobuf observation space. (47071)
- New API stack documentation:
- Cleanup old API stack docs (rllib-dev.rst). (47172)
- Episodes (SingleAgentEpisode). (46985)
- Redo rllib-algorithms.rst page. (46916)

🏗 Architecture refactoring:
- Rename MultiAgent...RLModule... into MultiRL...Module for more generality. (46840)
- Add learner_only flag to RLModuleConfig/Spec and simplify creation of RLModule specs from algo-config. (46900)

Ray Core<a id="ray-core"></a>
💫 Enhancements:
- Emit total lineage bytes metrics (46725)
- Adding accelerator type H100 (46823)
- More structured logging in core worker (46906)
- Change all callbacks to move to save copies. (46971)
- Add ray[adag] option to pip install (47009)

🔨 Fixes:
- Fix dashboard process reporting on windows (45578)
- Fix Ray-on-Spark cluster crashing bug when user cancels cell execution (46899)
- Fix PinExistingReturnObject segfault by passing owner_address (46973)
- Fix raylet CHECK failure from runtime env creation failure. (46991)
- Fix typo in memray command (47006)
- [ADAG] Fix for asyncio outputs (46845)

📖 Documentation:
- Clarify behavior of placement_group_capture_child_tasks in docs (46885)
- Update ray.available_resources() docstring (47018)

🏗 Architecture refactoring:
- Async APIs for the New GcsClient. (46788)
- Replace GCS stubs in the dashboard to use NewGcsAioClient. (46846)

Dashboard<a id="dashboard"></a>

💫 Enhancements:
- Polish and minor improvements to the Serve page (46811)

🔨 Fixes:
- Fix CPU/GPU/RAM not being reported correctly on Windows (44578)

Docs<a id="dashboard"></a>

💫 Enhancements:
- Add more information about developer tooling for docs contributions (46636), including `esbonio` section

🔨 Fixes:
- Use PyData Sphinx theme version switcher (46936)

Thanks

Many thanks to all those who contributed to this release!
simonsays1980, bveeramani, tungh2, zcin, xingyu-long, WeichenXu123, aslonnie, MaxVanDijck, can-anyscale, galenhwang, omatthew98, matthewdeng, raulchen, sven1977, shrekris-anyscale, deepyaman, alexeykudinkin, stephanie-wang, kevin85421, ruisearch42, hongchaodeng, khluu, alanwguo, hongpeng-guo, saihaj, Superskyyy, tespent, slfan1989, justinvyu, rynewang, nikitavemuri, amogkam, mattip, dev-goyal, ryanaoleary, peytondmurray, edoakes, venkatajagannath, jjyao, cristianjd, scottjlee, Bye-legumes

Ray Libraries<a id="ray-libraries"></a>

Ray Data<a id="ray-data"></a>
💫 Enhancements:
- Add better support for UDF returns from list of datetime objects (46762)

🔨 Fixes:
- Remove read task warning if size bytes not set in metadata (46765)

📖 Documentation:
- Fix read_tfrecords() docstring to display tfx-bsl tip (46717)
- Update Dataset.zip() docs (46757)


Ray Train<a id="ray-train"></a>
🔨 Fixes:
- Sort workers by node ID rather than by node IP (46163)

🏗 Architecture refactoring:
- Remove dead RayDatasetSpec (46764)

RLlib<a id="rllib"></a>

🎉 New Features:
- Offline RL support on new API stack:
- Initial design for Ray-Data based offline RL Algos (on new API stack). (44969)
- Add user-defined schemas for data loading. (46738)
- Make data pipeline better configurable and tuneable for users. (46777)

💫 Enhancements:
- Move DQN into the TargetNetworkAPI (and deprecate `RLModuleWithTargetNetworksInterface`). (46752)

🔨 Fixes:
- Numpy version fix: Rename all np.product usage to np.prod (46317)

📖 Documentation:
- Examples for new API stack: Add 2 (count-based) curiosity examples. (46737)
- Remove RLlib CLI from docs (soon to be deprecated and replaced by python API). (46724)

🏗 Architecture refactoring:
- Cleanup, rename, clarify: Algorithm.workers/evaluation_workers, local_worker(), etc.. (46726)

Ray Core<a id="ray-core"></a>

🏗 Architecture refactoring:
- New python GcsClient binding (46186)




Many thanks to all those who contributed to this release! KyleKoon, ruisearch42, rynewang, sven1977, saihaj, aslonnie, bveeramani, akshay-anyscale, kevin85421, omatthew98, anyscalesam, MaxVanDijck, justinvyu, simonsays1980, can-anyscale, peytondmurray, scottjlee