Ray

Ray Libraries<a id="ray-libraries"></a>

Ray Core<a id="ray-core"></a>

💫 Enhancements:
- Add "last exception" to error message when GCS connection fails in ray.init() (46516)

🔨 Fixes:
- Add object back to memory store when object recovery is skipped (46460)
- Task status should start with PENDING_ARGS_AVAIL when retry (46494)
- Fix ObjectFetchTimedOutError ([46562](https://github.com/ray-project/ray/pull/46562))
- Make working_dir support files created before 1980 ([46634](https://github.com/ray-project/ray/pull/46634))
- Allow full path in conda runtime env. ([45550](https://github.com/ray-project/ray/pull/45550))
- Fix worker launch time formatting in state api ([43516](https://github.com/ray-project/ray/pull/43516))
-
Ray Data<a id="ray-data"></a>
🎉 New Features:
- Deprecate Dataset.get_internal_block_refs() (46455)
- Add read API for reading Databricks table with Delta Sharing (46072)
- Add support for objects to Arrow blocks (45272)

💫 Enhancements:
- Change offsets to int64 and change to LargeList for ArrowTensorArray (45352)
- Prevent from_pandas from combining input blocks (46363)
- Update Dataset.count() to avoid unnecessarily keeping `BlockRef`s in-memory (46369)
- Use Set to fix inefficient iteration over Arrow table columns (46541)
- Add AWS Error UNKNOWN to list of retried write errors (46646)
- Always print traceback for internal exceptions (46647)
- Allow unknown estimate of operator output bundles and `ProgressBar` totals (46601)
- Improve filesystem retry coverage (46685)

🔨 Fixes:
- Replace lambda mutable default arguments (46493)

📖 Documentation:
- Auto-generate Dataset API documentation (46557)
- Update outdated ExecutionPlan docstring (46638)


Ray Train<a id="ray-train"></a>
💫 Enhancements:
- Update run status and actor status for train runs. (46395)

🔨 Fixes:
- Replace lambda default arguments (46576)

📖 Documentation:
- Add MNIST training using KubeRay doc page (46123)
- Add example of pre-training Llama model on Intel Gaudi (45459)
- Fix tensorflow example by using ScalingConfig (46565)

Ray Tune<a id="ray-tune"></a>
🔨 Fixes:
- Replace lambda default arguments (46596)

Ray Serve<a id="ray-serve"></a>

🎉 New Features:
- Fully deprecate `target_num_ongoing_requests_per_replica` and `max_concurrent_queries`, respectively replaced by `max_ongoing_requests` and `target_ongoing_requests` (46392 and 46427)
- Configure the task launched by the controller to build an application with Serve’s logging config (46347)

RLlib<a id="rllib"></a>

💫 Enhancements:
- Moving sampling coordination for `batch_mode=complete_episodes` to `synchronous_parallel_sample`. (46321)
- Enable complex action spaces with stateful modules. (46468)

🏗 Architecture refactoring:
- Enable multi-learner setup for hybrid stack BC. (46436)
- Introduce Checkpointable API for RLlib components and subcomponents. (46376)

🔨 Fixes:
- Replace Mapping typehint with Dict: 46474

📖 Documentation:
- More example scripts for new API stack: Two separate optimizers (w/ different learning rates). (46540) and custom loss function. (46445)

Dashboard<a id="dashboard"></a>

🔨 Fixes:
- Task end time showing the incorrect time (46439)
- Events Table rows having really bad spacing (46701)
- UI bugs in the serve dashboard page (46599)

Thanks

Many thanks to all those who contributed to this release!

alanwguo, hongchaodeng, anyscalesam, brucebismarck, bt2513, woshiyyya, terraflops1048576, lorenzoritter, omrishiv, davidxia, cchen777, nono-Sang, jackhumphries, aslonnie, JoshKarpel, zjregee, bveeramani, khluu, Superskyyy, liuxsh9, jjyao, ruisearch42, sven1977, harborn, saihaj, zcin, can-anyscale, veekaybee, chungen04, WeichenXu123, GeneDer, sergey-serebryakov, Bye-legumes, scottjlee, rynewang, kevin85421, cristianjd, peytondmurray, MortalHappiness, MaxVanDijck, simonsays1980, mjovanovic9999

Highlight: aDAG Developer Preview <a id="highlilght"></a>

This is a new Ray Core specific feature called Ray accelerated DAGs (aDAGs).

- aDAGs give you a Ray Core-like API but with extensibility to pre-compile execution paths across pre-allocated resources on a Ray Cluster to possible benefits for optimization on throughput and latency. Some practical examples include:
- Up to 10x lower task execution time on single-node.
- Native support for GPU-GPU communication, via NCCL.
- This is still very early, but please reach out on ray-core on Ray Slack to learn more!

Ray Libraries<a id="ray-libraries"></a>

Ray Data<a id="ray-data"></a>

💫 Enhancements:
- Support async callable classes in `map_batches()` (46129)

🔨 Fixes:
- Ensure `InputDataBuffer` doesn't free block references (46191)
- `MapOperator.num_active_tasks` should exclude pending actors (46364)
- Fix progress bars being displayed as partially completed in Jupyter notebooks (46289)

📖 Documentation:
- Fix docs: `read_api.py` docstring (45690)
- Correct API annotation for `tfrecords_datasource` (46171)
- Fix broken links in `README` and in `ray.data.Dataset` (45345)

Ray Train<a id="ray-train"></a>

📖 Documentation:
- Update PyTorch Data Ingestion User Guide (45421)

Ray Serve<a id="ray-serve"></a>

💫 Enhancements:
- Optimize `ServeController.get_app_config()` (45878)
- Change default for max and target ongoing requests (45943)
- Integrate with Ray structured logging (46215)
- Allow configuring handle cache size and controller max concurrency (46278)
- Optimize `DeploymentDetails.deployment_route_prefix_not_set()` (46305)

RLlib<a id="rllib"></a>

🎉 New Features:
- APPO on new API stack (w/ `EnvRunners`). (46216)

💫 Enhancements:
- Stability: APPO, SAC, and DQN activate multi-agent learning tests (45542, 46299)
- Make Tune trial ID available in `EnvRunners` (and callbacks). (46294)
- Add `env-` and `agent_steps` to custom evaluation function. (45652)
- Remove default-metrics from Algorithm (tune does NOT error anymore if any stop-metric is missing). (46200)

🔨 Fixes:
- Various bug fixes: 45542

📖 Documentation:
- Example for new API stack: Offline RL (BC) training on single-agent, while evaluating w/ multi-agent setup. (46251)
- Example for new API stack: Custom RLModule with an LSTM. (46276)

Ray Core<a id="ray-core"></a>

🎉 New Features:
- aDAG Developer Preview.

💫 Enhancements:
- Allow env setup logger encoding (46242)
- ray list tasks filter state and name on GCS side (46270)
- Log ray version and ray commit during GCS start (46341)

🔨 Fixes:
- Decrement lineage ref count of an actor when the actor task return object reference is deleted (46230)
- Fix negative ALIVE actors metric and introduce IDLE state (45718)
- `psutil` process attr `num_fds` is not available on Windows (46329)

Dashboard<a id="dashboard"></a>

🎉 New Features:
- Added customizable refresh frequency for metrics on Ray Dashboard (44037)

💫 Enhancements:
- Upgraded to MUIv5 and React 18 (45789)

🔨 Fixes:
- Fix for multi-line log items breaking log viewer rendering (46391)
- Fix for UI inconsistency when a job submission creates more than one Ray job. (46267)
- Fix filtering by job id for tasks API not filtering correctly. (45017)

Docs<a id="docs"></a>

🔨 Fixes:
- Re-enabled automatic cross-reference link checking for Ray documentation, with Sphinx nitpicky mode (46279)
- Enforced naming conventions for public and private APIs to maintain accuracy, starting with Ray Data API documentation (46261)

📖 Documentation:
- Upgrade Python 3.12 support to alpha, marking the release of the Ray wheel to PyPI and conducting a sanity check of the most critical tests.

Thanks

Many thanks to all those who contributed to this release!

stephanie-wang, MortalHappiness, aslonnie, ryanaoleary, jjyao, jackhumphries, nikitavemuri, woshiyyya, JoshKarpel, ruisearch42, sven1977, alanwguo, GeneDer, saihaj, raulchen, liuxsh9, khluu, cristianjd, scottjlee, bveeramani, zcin, simonsays1980, SumanthRH, davidxia, can-anyscale, peytondmurray, kevin85421

Ray Libraries<a id="ray-libraries"></a>

Ray Data<a id="ray-data"></a>

🔨 Fixes:
- Fixed bug where `preserve_order` doesn’t work with file reads ([46135](https://github.com/ray-project/ray/pull/46135))

📖 Documentation:
- Added documentation for `dataset.Schema` ([46170](https://github.com/ray-project/ray/pull/46170))

Ray Train<a id="ray-train"></a>

💫 Enhancements:
- Add API for Ray Train run stats (45711)

Ray Tune<a id="ray-tune"></a>

💫 Enhancements:
- Missing stopping criterion should not error (just warn). (45613)

📖 Documentation:
- Fix broken references in Ray Tune documentation (45233)

Ray Serve<a id="ray-serve"></a>

**WARNING**: the following default values will change in Ray 2.32:
- Default for `max_ongoing_requests` will change from 100 to 5.
- Default for `target_ongoing_requests` will change from 1 to 2.

💫 Enhancements:
- Optimize DeploymentStateManager.get_deployment_statuses ([45872](https://github.com/ray-project/ray/pull/45872))

🔨 Fixes:
- Fix logging error on passing traceback object into exc_info ([46105](https://github.com/ray-project/ray/pull/46105))
- Run __del__ even if constructor is still in-progress ([45882](https://github.com/ray-project/ray/pull/45882))
- Spread replicas with custom resources in torch tune serve release test ([46093](https://github.com/ray-project/ray/pull/46093))
- [1k release test] don't run replicas on head node ([46130](https://github.com/ray-project/ray/pull/46130))

📖 Documentation:
- Remove todo since issue is fixed ([45941](https://github.com/ray-project/ray/pull/45941))

RLlib<a id="rllib"></a>

🎉 New Features:
- IMPALA runs on the new API stack (with EnvRunners and ConnectorV2s). ([42085](https://github.com/ray-project/ray/pull/42085))
- SAC/DQN: Prioritized multi-agent episode replay buffer. ([45576](https://github.com/ray-project/ray/pull/45576)[)](https://github.com/ray-project/ray/commit/a7aa5e4c7bc4654b5ce710bf3551e3775470b57b)

💫 Enhancements:
- New API stack stability: Add systematic CI learning tests for all possible combinations of: [PPO|IMPALA] + [1CPU|2CPU|1GPU|2GPU] + [single-agent|multi-agent]. ([46162](https://github.com/ray-project/ray/pull/46162), [#46161](https://github.com/ray-project/ray/pull/46161))

📖 Documentation:
- New API stack: Example script for action masking [(](https://github.com/ray-project/ray/commit/9d661677f942ccd7dfbbf2c2838e5dad9f50d12e)[#46146](https://github.com/ray-project/ray/pull/46146))
- New API stack: PyFlight example script cleanup ([45956](https://github.com/ray-project/ray/pull/45956)[)](https://github.com/ray-project/ray/commit/af45a8999b5d9ca09054de29c2bdd0b5b02184b5)
- Old API stack: Enhanced ONNX example (+LSTM). ([43592](https://github.com/ray-project/ray/pull/43592)[)](https://github.com/ray-project/ray/commit/d15204f5a0d5f2fa70935efa4935a7a61d8aafd1)

Ray Core and Ray Clusters<a id="ray-core-and-ray-clusters"></a>

Ray Core<a id="ray-core"></a>

💫 Enhancements:
- [runtime-env] automatically infer worker path when starting worker in container ([42304](https://github.com/ray-project/ray/pull/42304))

🔨 Fixes:
- On GCS restart, destroy not forget the unused workers. Fixing PG leaks. (45854)
- Cancel lease requests before returning a PG bundle ([45919](https://github.com/ray-project/ray/pull/45919))
- Fix boost fiber stack overflow (46133)


Thanks

Many thanks to all those who contributed to this release!

jjyao, kevin85421, vincent-pli, khluu, simonsays1980, sven1977, rynewang, can-anyscale, richardsliu, jackhumphries, alexeykudinkin, bveeramani, ruisearch42, shrekris-anyscale, stephanie-wang, matthewdeng, zcin, hongchaodeng, ryanaoleary, liuxsh9, GeneDer, aslonnie, peytondmurray, Bye-legumes, woshiyyya, scottjlee, JoshKarpel

Ray Libraries<a id="ray-libraries"></a>

Ray Data<a id="ray-data"></a>

💫 Enhancements:
- Improve fractional CPU/GPU formatting (45673)
- Use sampled fragments to estimate Parquet reader batch size (45749)
- Refactoring ParquetDatasource and metadata fetching logic (45728, 45727, 45733, 45734, 45767)
- Refactor planner.py (45706)


Ray Tune<a id="ray-tune"></a>
💫 Enhancements:
- Change the behavior of a missing stopping criterion metric to warn instead of raising an error. This enables the use case of reporting different sets of metrics on different iterations (ex: a separate set of training and validation metrics). ([45613](https://github.com/ray-project/ray/pull/45613))

Ray Serve<a id="ray-serve"></a>

💫 Enhancements:
- Create internal request id to track request objects ([45761](https://github.com/ray-project/ray/pull/45761))

RLLib<a id="rllib"></a>

💫 Enhancements:
- Stability: [DreamerV3 weekly release test](https://github.com/ray-project/ray/commit/4adb78b2bf3c968f88f72ae9064189b846833230) ([#45654](https://github.com/ray-project/ray/pull/45654)); [Add "official" benchmark script for Atari PPO benchmarks](https://github.com/ray-project/ray/commit/d49f15b1112e67d15a80d696249f587ea7b95b57). ([#45697](https://github.com/ray-project/ray/pull/45697))
- Enhance env-rendering callback (45682)

🔨 Fixes:
- Bug fix in new MetricsLogger API: [EMA stats w/o window would lead to infinite l](https://github.com/ray-project/ray/commit/cbb1634a23ff4b59090f43dd853cf437e19fc0c8)ist mem-leak. ([#45752](https://github.com/ray-project/ray/pull/45752))
- Various other bug fixes: ([45819](https://github.com/ray-project/ray/pull/45819), [#45820](https://github.com/ray-project/ray/pull/45820), #[45683](https://github.com/ray-project/ray/pull/45683), [#45651](https://github.com/ray-project/ray/pull/45651), [#45753](https://github.com/ray-project/ray/pull/45753))

📖 Documentation:
- Re-do `examples` overview page (new API stack): [45382](https://github.com/ray-project/ray/pull/45382)
- PyFlyt QuadX WayPoints example [44758](https://github.com/ray-project/ray/pull/44758), [#45956](https://github.com/ray-project/ray/pull/45956)
- RLModule inference on new API stack ([45831](https://github.com/ray-project/ray/pull/45831), [#45845](https://github.com/ray-project/ray/pull/45845))
- How to resume a tune.Tuner.fit() experiment from checkpoint. ([45681](https://github.com/ray-project/ray/pull/45681))
- Custom RLModule (tiny CNN): [45774](https://github.com/ray-project/ray/pull/45774)
- Connector examples docstrings ([45864](https://github.com/ray-project/ray/pull/45864))
- Old API stack examples: [43592](https://github.com/ray-project/ray/pull/43592), [#45829](https://github.com/ray-project/ray/pull/45829)

Ray Core<a id="ray-core"></a>
🎉 New Features:
- Alpha release of job level [logging configuration](https://docs.ray.io/en/master/ray-core/api/doc/ray.LoggingConfig.html#ray.LoggingConfig): users can now config the user logging to be logfmt format with logging context attached. (45344)

💫 Enhancements:
- Integrate amdsmi in AMDAcceleratorManager (44572)

🔨 Fixes:
- Fix the C++ GcsClient Del not respecting del_by_prefix (45604)
- Fix exit handling of FiberState threads (45834)

Dashboard<a id="ray-dashboard"></a>
💫 Enhancements:
- Parse out json logs (45853)

Many thanks to all those who contributed to this release: liuxsh9, peytondmurray, pcmoritz, GeneDer, saihaj, khluu, aslonnie, yucai, vickytsang, can-anyscale, bthananjeyan, raulchen, hongchaodeng, x13n, simonsays1980, peterghaddad, kevin85421, rynewang, angelinalg, jjyao, BenWilson2, jackhumphries, zcin, chris-ray-zhang, c21, shrekris-anyscale, alanwguo, stephanie-wang, Bye-legumes, sven1977, WeichenXu123, bveeramani, nikitavemuri

Ray Libraries<a id="ray-libraries"></a>

Ray Data<a id="ray-data"></a>

🎉 New Features:
- Allow user to configure timeout for actor pool (45508)
- Add override_num_blocks to from_pandas and perform auto-partition (44937)
- Upgrade Arrow version to 16 in CI (45565)

💫 Enhancements:
- Clarify that num_rows_per_file isn't strict (45529)
- Record more telemetry for newly added datasources (45647)
- Avoid pickling LanceFragment when creating read tasks for Lance (45392)

Ray Train<a id="ray-train"></a>

📖 Documentation:
- [HPU] Add example of Stable Diffusion fine-tuning and serving on Intel Gaudi (45217)
- [HPU] Add example of Llama-2 fine-tuning on Intel Gaudi (44667)


Ray Tune<a id="ray-tune"></a>
🏗 Architecture refactoring:
- Improve excessive syncing warning and deprecate TUNE_RESULT_DIR, RAY_AIR_LOCAL_CACHE_DIR, local_dir (45210)

Ray Serve<a id="ray-serve"></a>

💫 Enhancements:
- Clean up Serve proxy files ([45486](https://github.com/ray-project/ray/pull/45486))

📖 Documentation:
- vllm example to serve llm models ([45430](https://github.com/ray-project/ray/pull/45430))

RLLib<a id="rllib"></a>

💫 Enhancements:
- DreamerV3 on tf: Bug fix, so it can run again with tf==2.11.1 (2.11.0 is not available anymore) (45419); Added weekly release test for DreamerV3.
- Added support for multi-agent off-policy algorithms (DQN and SAC) in the new (45182)
- Config option for APPO/IMPALA to change number of GPU-loader threads (45467)

🔨 Fixes:
- Various MetricsLogger bug fixes (45543, 45585, 45575)
- Other fixes: 45588, 45617, 45517, 45465

📖 Documentation:
- Example script for new API stack: How-to restore 1 of n agents from a checkpoint. (45462)
- Example script for new API stack: Autoregressive action module. 45525

Ray Core<a id="ray-core"></a>

💫 Enhancements:
- Improve node death observability (45320, 45357, 45533, 45644, 45497)
- Ray c++ backend structured logging (44468)

🔨 Fixes:
- Fix worker crash when getting actor name from runtime context (45194)
- log dedup should not dedup number only lines (45385)

📖 Documentation:
- Improve doc for `--object-store-memory` to describe how the default value is set ([45301](https://github.com/ray-project/ray/pull/45301))

Dashboard<a id="ray-dashboard"></a>
🔨 Fixes:
- Move Job package uploading to another thread to unblock the event loop. ([45282](https://github.com/ray-project/ray/pull/45282))

Many thanks to all those who contributed to this release: maxliuofficial, simonsays1980, GeneDer, dudeperf3ct, khluu, justinvyu, andrewsykim, Catch-Bull, zcin, bveeramani, rynewang, angelinalg, matthewdeng, jjyao, kira-lin, harborn, hongchaodeng, peytondmurray, aslonnie, timkpaine, 982945902, maxpumperla, stephanie-wang, ruisearch42, alanwguo, can-anyscale, c21, Atry, KamenShah, sven1977, raulchen

Ray Libraries<a id="ray-libraries"></a>

Ray Data<a id="ray-data"></a>

🎉 New Features:
- Add support for using GPUs with map_groups (45305)
- Add support for using actors with map_groups (45310)

💫 Enhancements:
- Refine exception handling from arrow data conversion (45294)

🔨 Fixes:
- Fix Ray databricks UC reader with dynamic Databricks notebook scope token (45153)
- Fix bug where you can't return objects and array from UDF (45287 )
- Fix bug where map_groups triggers execution during input validation (45314)

Ray Tune<a id="ray-tune"></a>
🔨 Fixes:
- [tune] Fix PB2 scheduler error resulting from trying to sort by Trial objects (45161)

Ray Serve<a id="ray-serve"></a>
🔨 Fixes:
- Log application unhealthy errors at error level instead of warning level ([45211](https://github.com/ray-project/ray/pull/45211))

RLLib<a id="rllib"></a>
💫 Enhancements:
- Examples and `tuned_examples` learning test for new API stack are now “self-executable” (don’t require a third-party script anymore to run them). + WandB support. ([45023](https://github.com/ray-project/ray/pull/45023))

🔨 Fixes:
- Fix result dict “spam” (duplicate, deprecated keys, e.g. “sampler_results” dumped into top level). ([45330](https://github.com/ray-project/ray/pull/45330))

📖 Documentation:
- Add example for training with fractional GPUs on new API stack. ([45379](https://github.com/ray-project/ray/pull/45379))
- Cleanup examples folder and remove deprecated sub directories. ([45327](https://github.com/ray-project/ray/pull/45327))

Ray Core<a id="ray-core"></a>
💫 Enhancements:
- [Logs] Add runtime env started logs to job driver ([45255](https://github.com/ray-project/ray/pull/45255))
- `ray.util.collective` support `torch.bfloat16` ([39845](https://github.com/ray-project/ray/pull/39845))
- [Core] Better propagate node death information ([45128](https://github.com/ray-project/ray/pull/45128))

🔨 Fixes:
- [Core] Fix worker process leaks after job finishes ([44214](https://github.com/ray-project/ray/pull/44214))


Many thanks to all those who contributed to this release: hongchaodeng, khluu, antoni-jamiolkowski, ameroyer, bveeramani, can-anyscale, WeichenXu123, peytondmurray, jackhumphries, kevin85421, jjyao, robcaulk, rynewang, scottsun94, swang, GeneDer, zcin, ruisearch42, aslonnie, angelinalg, raulchen, ArthurBook, sven1977, wuxibin89