Ray

The Ray 2.5.1 patch release adds wheels for MacOS for Python 3.11.
It also contains fixes for multiple components, along with fixes for our documentation.

Ray Train
🔨 Fixes:
* Don't error on eventual success when running with auto-recovery (36266)

Ray Core

🎉 New Features:
* Build Python wheels on Mac OS for Python 3.11 (36373)

🔨 Fixes:
* [Autoscaler] Fix a bug that can cause undefined behavior when clusters attempt to scale up aggressively. (36241)
* Fix mypy error: Module "ray" does not explicitly export attribute "remote" (36356)

The Ray 2.5 release features focus on a number of enhancements and improvements across the Ray ecosystem, including:

* Training LLMs with Ray Train: New support for checkpointing distributed models, and Pytorch Lightning FSDP to enable training large models on Ray Train’s LightningTrainer
* LLM applications with Ray Serve & Core: New support for streaming responses and model multiplexing
* Improvements to Ray Data: In 2.5, [strict mode ](https://docs.ray.io/en/master/data/faq.html#migrating-to-strict-mode)is enabled by default. This means that schemas are required for all Datasets, and standalone Python objects are no longer supported. Also, the default batch format is fixed to NumPy, giving better performance for [batch inference](https://docs.ray.io/en/master/data/batch_inference.html).
* RLlib enhancements: New support for multi-gpu training, along with [ray-project/rllib-contrib](https://github.com/ray-project/rllib-contrib) to contain the community contributed algorithms
* Core enhancements: Enable new feature of lightweight resource broadcasting to improve reliability and scalability. Add many enhancements for Core reliability, logging, scheduler, and worker process.

Ray Libraries
Ray AIR

💫Enhancements:
* Experiment restore stress tests (33706)
* Context-aware output engine
* Add parameter columns to status table (35388)
* Context-aware output engine: Add docs, experimental feature docs, prepare default on (35129)
* Fix trial status at end (more info + cut off) (35128)
* Improve leaked mentions of Tune concepts (35003)
* Improve passed time display (34951)
* Use flat metrics in results report, use Trainable._progress_metrics (35035)
* Print experiment information at experiment start (34952)
* Print single trial config + results as table (34788)
* Print out worker ip for distributed train workers. (33807)
* Minor fix to print configuration on start. (34575)
* Check `air_verbosity` against None. (33871)
* Better wording for empty config. (33811)
* Flatten config and metrics before passing to mlflow (35074)
* Remote_storage: Prefer fsspec filesystems over native pyarrow (34663)
* Use filesystem wrapper to exclude files from upload (34102)
* GCE test variants for air_benchmark and air_examples (34466)
* New storage path configuration
* Add `RunConfig.storage_path` to replace `SyncConfig.upload_dir` and `RunConfig.local_dir`. (33463)
* Use Ray storage URI as default storage path, if configured [no_early_kickoff] (34470)
* Move to new storage_path API in tests and examples (34263)

🔨 Fixes:
* Store unflattened metrics in _TrackedCheckpoint (35658) (35706)
* Fix `test_tune_torch_get_device_gpu` race condition (35004)
* Deflake test_e2e_train_flow.py (34308)
* Pin deepspeed version for now to unblock ci. (34406)
* Fix AIR benchmark configuration link failure. (34597)
* Fix unused config building function in lightning MNIST example.

📖Documentation:
* Change doc occurrences of ray.data.Dataset to ray.data.Datastream (34520)
* DreamBooth example: Fix code for batch size > 1 (34398)
* Synced tabs in AIR getting started (35170)
* New Ray AIR link for try it out (34924)
* Correctly Render the Enumerate Numbers in `convert_torch_code_to_ray_air` (35224)


Ray Data Processing

🎉 New Features:
* Implement Strict Mode and enable it by default.
* Add column API to Dataset (35241)
* Configure progress bars via DataContext (34638)
* Support using concurrent actors for ActorPool (34253)
* Add take_batch API for collecting data in the same format as iter_batches and map_batches (34217)

💫Enhancements:
* Improve map batches error message for strict mode migration (35368)
* Improve docstring and warning message for from_huggingface (35206)
* Improve notebook widget display (34359)
* Implement some operator fusion logic for the new backend (35178 34847)
* Use wait based prefetcher by default (34871)
* Implement limit physical operator (34705 34844)
* Require compute spec to be explicitly spelled out 34610
* Log a warning if the batch size is misconfigured in a way that would grossly reduce parallelism for actor pool. (34594)
* Add alias parameters to the aggregate function, and add quantile fn (34358)
* Improve repr for Arrow Table and pandas types (34286 34502)
* Defer first block computation when reading a Datasource with schema information in metadata (34251)
* Improve handling of KeyboardInterrupt (34441)
* Validate aggregation key in Aggregate LogicalOperator (34292)
* Add usage tag for which block formats are used (34384)
* Validate sort key in Sort LogicalOperator (34282)
* Combine_chunks before chunking pyarrow.Table block into batches (34352)
* Use read stage name for naming Data-read tasks on Ray Dashboard (34341)
* Update path expansion warning (34221)
* Improve state initialization for ActorPoolMapOperator (34037)

🔨 Fixes:
* Fix ipython representation (35414)
* Fix bugs in handling of nested ndarrays (and other complex object types) (35359)
* Capture the context when the dataset is first created (35239)
* Cooperatively exit producer threads for iter_batches (34819)
* Autoshutdown executor threads when deleted (34811)
* Fix backpressure when reading directly from input datasource (34809)
* Fix backpressure handling of queued actor pool tasks (34254)
* Fix row count after applying filter (34372)
* Remove unnecessary setting of global logging level to INFO when using Ray Data (34347)
* Make sure the tf and tensor iteration work in dataset pipeline (34248)
* Fix '_unwrap_protocol' for Windows systems (31296)

📖Documentation:
* Add batch inference object detection example (35143)
* Refine batch inference doc (35041)


Ray Train
🎉 New Features:
* Experimental support for distributed checkpointing (34709)

💫Enhancements:
* LightningTrainer: Enable prog bar (35350)
* LightningTrainer enable checkpoint full dict with FSDP strategy (34967)
* Support FSDP Strategy for LightningTrainer (34148)

🔨 Fixes:
* Fix HuggingFace -> Transformers wrapping logic (35276, 35284)
* LightningTrainer always resumes from the latest AIR checkpoint during restoration. (35617) (35791)
* Fix lightning trainer devices setting (34419)
* TorchCheckpoint: Specifying pickle_protocol in `torch.save()` (35615) (35790)

📖Documentation:
* Improve visibility of Trainer restore and stateful callback restoration (34350)
* Fix rendering of diff code-blocks (34355)
* LightningTrainer Dolly V2 FSDP Fine-tuning Example (34990)
* Update LightningTrainer MNIST example. (34867)
* LightningTrainer Advanced Example (34082, 34429)

🏗 Architecture refactoring:
* Restructure `ray.train` HuggingFace modules (35270) (35488)
* rename _base_dataset to _base_datastream (34423)


Ray Tune
🎉 New Features:
* Ray Tune's new execution path is now enabled per default (34840, 34833)

💫Enhancements:
* Make `Tuner.restore(trainable=...) a required argument (34982)
* Enable `tune.ExperimentAnalysis` to pull experiment checkpoint files from the cloud if needed (34461)
* Add support for nested hyperparams in PB2 (31502)
* Release test for durable multifile checkpoints (34860)
* GCE variants for remaining Tune tests (34572)
* Add tune frequent pausing release test. (34501)
* Add PyArrow to ray[tune] dependencies (34397)
* Fix new execution backend for BOHB (34828)
* Add tune frequent pausing release test. (34501)

🔨 Fixes:
* Set config on trial restore (35000)
* Fix `test_tune_torch_get_device_gpu` race condition (35004)
* Fix a typo in `tune/execution/checkpoint_manager` state serialization. (34368)
* Fix tune_scalability_network_overhead by adding `--smoke-test`. (34167)
* Fix lightning_gpu_tune_.* release test (35193)

📖Documentation:
* Fix Tune tutorial (34660)
* Fix typo in Tune restore guide (34247)

🏗 Architecture refactoring:
* Use Ray-provided `tabulate` package (34789)


Ray Serve
🎉 New Features:
* Add support for json logging format.(35118)
* Add experimental support for model multiplexing.(35399, 35326)
* Added experimental support for HTTP StreamingResponses. (35720)
* Add support for application builders & arguments (34584)

💫Enhancements:
* Add more bucket size for histogram metrics. (35242).
* Add route information into the custom metrics. (35246)
* Add HTTPProxy details to Serve Dashboard UI (35159)
* Add status_code to http qps & latency (35134)
* Stream Serve logs across different drivers (35070)
* Add health checking for http proxy actors (34944)
* Better surfacing of errors in serve status (34773)
* Enable TLS on gRPCIngress if RAY_USE_TLS is on (34403)
* Wait until replicas have finished recovering (with timeout) to broadcast `LongPoll` updates (34675)
* Replace `ClassNode` and `FunctionNode` with `Application` in top-level Serve APIs (34627)

🔨 Fixes:
* Set `app_msg` to empty string by default (35646)
* Fix dead replica counts in the stats. (34761)
* Add default app name (34260)
* gRPC Deployment schema check & minor improvements (34210)

📖Documentation:
* Clean up API reference and various docstrings (34711)
* Clean up `RayServeHandle` and `RayServeSyncHandle` docstrings & typing (34714)


RLlib
🎉 New Features:
* Migrating approximately ~25 of the 30 algorithms from RLlib into [rllib_contrib](https://github.com/ray-project/ray/tree/master/rllib_contrib). You can review the REP [here](https://github.com/ray-project/enhancements/blob/main/reps/2023-04-28-remove-algorithms-from-rllib.md). This release we have covered A3C and MAML.
* The APPO / IMPALA and PPO are all moved to the new Learner and RLModule stack.
* The RLModule now supports Checkpointing.([34717](https://github.com/ray-project/ray/pull/34717) [#34760](https://github.com/ray-project/ray/pull/34760))

💫Enhancements:
* Introduce experimental larger than GPU train batch size feature for torch (34189)
* Change occurences of "_observation_space_in_preferred_format" to "_obs_space_in_preferred_format" (33907)
* Add a flag to allow disabling initialize_loss_from_dummy_batch logit. (34208)
* Remove check specs from default Model forward code path to improve performance (34877)
* Remove some specs from encoders to smoothen dev experience (34911)

🔨 Fixes:
* Fix MultiCallbacks class: To be used only with utility function that returns a class to use in the config. (33863)
* Fix test backward compatibility test for RL Modules (33857)
* Don't serialize config in Policy states (unless needed for msgpack-type checkpoints). (33865)
* DM control suite wrapper fix: dtype of obs needs to be pinned to float32. (33876)
* In the Json_writer convert all non string keys to keys (33896)
* Fixed a bug with kl divergence calculation of torch.Dirichlet distribution within RLlib (34209)
* Change broken link in parameter_noise.py (34231)
* Fixed bug in restoring a gpu trained algorithm (35024)
* Fix IMPALA/APPO when using multi GPU setup and Multi-Agent Env (35120)

📖Documentation:
* Add examples and docs for Catalog (33898)

Ray Core and Ray Clusters
Ray Core
🎉 New Features:
* Support both sync and async actor generator interface. (35584 35708 35324 35656 35803 35794 35707)

💫Enhancements:
* [Scheduler] Introduce spill_on_unavailable option for soft NodeAffinitySchedulingStrategy (34224)
* [Data] Use wait based prefetcher by default (34871)
* [Reliability] During GCS restarts, grpc based resource broadcaster should only add ALIVE nodes during initialization (35349)
* [Reliability] Guarantee the ordering of put ActorTaskSpecTable and ActorTable (35683) (35718)
* [Reliability] Graceful handling of returning bundles when node is removed (34726) (35542)
* [Reliability] Task backend - marking tasks failed on worker death (33818)
* [Reliability] Task backend - Add worker dead info to failed tasks when job exits. (34166)
* [Logging] Make ray.get(timeout=0) to throw timeout error (35126)
* [Logging] Provide good error message if the factional resource precision is beyond 0.0001 (34590)
* [Logging] Add debug logs to show UpdateResourceUsage rpc source (35062)
* [Logging] Add actor_id as an attribute of RayActorError when the actor constructor fails (34958)
* [Logging] Worker startup hook (34738)
* [Worker] Partially addresses ray child process leaks by killing all child processes in the CoreWorker shutdown sequence. (33976)
* [Worker] Change worker niceness in job submission environment (34727)
* Shorten the membership checking time to 5 seconds. (34769)
* [Syncer] Remove spammy logs. (34654)
* [Syncer] Delete disconnected node view in ray syncer when connection is broken. (35312)
* [Syncer] Turn on ray syncer again. (35116)
* [Syncer] Start ray syncer reconnection after a delay (35115)
* Serialize requests in the redis store client. (35123)
* Reduce self alive check from 60s to 5s. (34992)
* Add object owner and copy metrics to node stats (35119)
* Start the synchronization connection after receiving all nodes info. (34645)
* Improve the workflow finding Redis leader. (34108)
* Make execute_after accept chrono (35099)
* Lazy import autoscaler + don't import opentelemetry unless setup hook (33964)

🔨 Fixes:
* [pubsub] Handle failures when publish fails. (33115)
* Convert gcs port read from env variable from str to int (34482)
* Fix download_wheels.sh wheel urls (34616)
* Fix ray start command output (34081)
* Fetch_local once for each object ref (34884)
* Combine_chunks before chunking pyarrow.Table block into batches (34352)
* Replace deprecated usage of get_runtime_context().node_id (34874)
* Fix std::move without std namespace (34149)
* Fix the recursion error when an async actor has lots of deserialization. (35494) (35532)
* Fix async actor shutdown issue when exit_actor is used (32407)
* [Event] Fix incorrect event timestamp (34402)
* [Metrics] Fix shared memory is not displayed properly (34301)
* Fix GCS FD usage increase regression. (35624) (35738)
* Fix raylet memory leak in the wrong setup. (35647) (35673)
* Retry failed redis request (35249) (35481)
* Add more messages when accessing a dead actor. (34697)
* Fix the placement group stress test regression. (34192)
* Mark Raylet unhealthy if GCS can't recognize it. (34087)
* Remove multiple core workers in one process 2/n (34942)
* Remove python 3.6 support (34373 34416)

📖Documentation:
* Make doc code snippet testable (35274 35057)
* Revamp ray core api reference [1/n] (34428)
* Add Ray core fault tolerance guide for GCS and node (33446)
* Ray Debugging Doc Part 1 (OOM) (34309)
* Rewrite the placement group documentation (33518)


Ray Clusters
💫Enhancements:
* [Docker] [runtime env] Bump boto3 version from 1.4.8 to 1.26.82, add pyOpenSSL and cryptography (33273)
* [Jobs] Fix race condition in supervisor actor creation and add timeout for pending jobs (34223)
* [Release test] [Cluster launcher] Add gcp minimal and full cluster launcher release test (34878)
* [Release test] [Cluster launcher] Add release test for aws `example-full.yaml` (34487)

📖Documentation:
* [runtime env] Clarify conditions for local `pip` and `conda` requirements files (34071)
* [KubeRay] Provide GKE instructions in KubeRay example (33339)
* [KubeRay] Update KubeRay doc for release v0.5.0 (34178)


Dashboard
💫Enhancements:
* Feature flag task logs recording (34056)
* Fix log proxy not loading non test/plain files. (33870)
* [no_early_kickoff] Make dashboard address connectable from remote nodes when not set to 127.0.0.1 (localhost) (35027)
* [state][job] Supporting job listing(getting) and logs from state API (35124)
* [state][ci] Fix stress_test_state_api_scale (35332)
* [state][dashboard][log] Fix subdirectory log getting (35283)
* [state] Push down filtering to GCS for listing/getting task from state api (35109)(34433)
* [state] Task log - Improve log tailing from log_client and support tailing from offsets [2/4] (28188)
* [state] Use `--err` flag to query stderr logs from worker/actors instead of `--suffix=err` (34300)
* [state][no_early_kickoff] Make state api return results that are strongly typed (34297)
* [state] Efficient get/list actors with filters on some high-cardinality fields (34348)
* [state] Fix list nodes test in test_state_api.py (34349)
* [state] Add head node flag `is_head_node` to state API and GcsNodeInfo (34299)
* Make actor tasks' name default to <actor_repr>.<task_name> (35371)
* Task backend GC policy - worker update [1/3] (34896)
* [state] Support task logs from state API (35101)

Known Issues
* A bug in the Autoscaler can cause undefined behaviour when clusters attempt to scale up aggressively. This is fixed in following releases, as well as post-release on [the 2.5.0 branch](https://github.com/ray-project/ray/tree/releases/2.5.0) (#36482).


Many thanks to all those who contributed to this release!

vitsai, XiaodongLv, justinvyu, Dan-Yeh, dependabot[bot], alanwguo, grimreaper, yiwei00000, pomcho555, ArturNiederfahrenhorst, maxpumperla, jjyao, ijrsvt, sven1977, Yard1, pcmoritz, c21, architkulkarni, jbedorf, amogkam, ericl, jiafuzha, clarng, shrekris-anyscale, matthewdeng, gjoliver, jcoffi, edoakes, ethanabrooks, iycheng, Rohan138, angelinalg, Linniem, aslonnie, zcin, wuisawesome, Catch-Bull, woshiyyya, avnishn, jjyyxx, jianoaix, bveeramani, sihanwang41, scottjlee, YQ-Wang, mattip, can-anyscale, xwjiang2010, fedassembly, joncarter1, robin-anyscale, rkooo567, DACUS1995, simran-2797, ProjectsByJackHe, zen-xu, ashahab, larrylian, kouroshHakha, raulchen, sofianhnaide, scv119, nathan-az, kevin85421, rickyyx, Sahar-E, krfricke, chaowanggg, peytondmurray, cadedaniel

The Ray 2.3.1 patch release contains fixes for multiple components:

Ray Data Processing
* Support different number of blocks/rows per block in `zip()` (https://github.com/ray-project/ray/pull/32795)

Ray Serve
* Revert `serve run` to use Ray Client instead of Ray Jobs (https://github.com/ray-project/ray/pull/32976)
* Fix issue with `max_concurrent_queries` being ignored when autoscaling (https://github.com/ray-project/ray/pull/32772 and https://github.com/ray-project/ray/pull/33022)
Ray Core
* Write Ray address even if Ray node is started with `--block` (https://github.com/ray-project/ray/pull/32961)
* Fix Ray on Spark running on layered virtualenv python environment (https://github.com/ray-project/ray/pull/32996)
Dashboard
* Fix disk metric showing double the actual value (https://github.com/ray-project/ray/pull/32674)

Release Highlights



* The streaming backend for Ray Datasets is in Developer Preview. It is designed to enable terabyte-scale ML inference and training workloads. Please contact us if you'd like to try it out on your workload, or you can find the preview guide here: https://docs.google.com/document/d/1BXd1cGexDnqHAIVoxTnV3BV0sklO9UXqPwSdHukExhY/edit
* New Information Architecture (**Beta**): We’ve restructured the [Ray dashboard](https://docs.ray.io/en/master/ray-core/ray-dashboard.html) to be organized around user personas and workflows instead of entities.
* Ray-on-Spark is now available (Preview)!: You can launch Ray clusters on Databricks and Spark clusters and run Ray applications. Check out the [documentation](https://docs.ray.io/en/releases-2.3.0/cluster/vms/user-guides/community/spark.html) to learn more.

Ray Libraries


Ray AIR

💫Enhancements:



* Add `set_preprocessor` method to `Checkpoint` (31721)
* Rename Keras callback and its parameters to be more descriptive (31627)
* Deprecate MlflowTrainableMixin in favor of setup_mlflow() function (31295)
* W&B
* Have train_loop_config logged as a config (31901)
* Allow users to exclude config values with WandbLoggerCallback (31624)
* Rename WandB `save_checkpoints` to `upload_checkpoints` (31582)
* Add hook to get project/group for W&B integration (31035, 31643)
* Use Ray actors instead of multiprocessing for WandbLoggerCallback (30847)
* Update `WandbLoggerCallback` example (31625)
* Predictor
* Place predictor kwargs in object store (30932)
* Delegate BatchPredictor stage fusion to Datasets (31585)
* Rename `DLPredictor.call_model` `tensor` parameter to `inputs` (30574)
* Add `use_gpu` to `HuggingFacePredictor` (30945)
* Checkpoints
* Various `Checkpoint` improvements (30948)
* Implement lazy checkpointing for same-node case (29824)
* Automatically strip "module." from state dict (30705)
* Allow user to pass model to `TensorflowCheckpoint.get_model` (31203)

🔨 Fixes:



* Fix and improve support for HDFS remote storage. (31940)
* Use specified Preprocessor configs when using stream API. (31725)
* Support nested Chain in BatchPredictor (31407)

📖Documentation:



* Restructure API References (32535)
* API Deprecations (31777, 31867)
* Various fixes to docstrings, documentation, and examples (30782, 30791)

🏗 Architecture refactoring:



* Use NodeAffinitySchedulingPolicy for scheduling (32016)
* Internal resource management refactor (30777, 30016)


Ray Data Processing

🎉 New Features:



* Lazy execution by default (31286)
* Introduce streaming execution backend (31579)
* Introduce DatasetIterator (31470)
* Add per-epoch preprocessor (31739)
* Add TorchVisionPreprocessor (30578)
* Persist Dataset statistics automatically to log file (30557)

💫Enhancements:



* Async batch fetching for map_batches (31576)
* Add informative progress bar names to map_batches (31526)
* Provide an size bytes estimate for mongodb block (31930)
* Add support for dynamic block splitting to actor pool (31715)
* Improve str/repr of Dataset to include execution plan (31604)
* Deal with nested Chain in BatchPredictor (31407)
* Allow MultiHotEncoder to encode arrays (31365)
* Allow specify batch_size when reading Parquet file (31165)
* Add zero-copy batch API for `ds.map_batches()` (30000)
* Text dataset should save texts in ArrowTable format (30963)
* Return ndarray dicts for single-column tabular datasets (30448)
* Execute randomize_block_order eagerly if it's the last stage for ds.schema() (30804)

🔨 Fixes:



* Don't drop first dataset when peeking DatasetPipeline (31513)
* Handle np.array(dtype=object) constructor for ragged ndarrays (31670)
* Emit warning when starting Dataset execution with no CPU resources available (31574)
* Fix the bug of eagerly clearing up input blocks (31459)
* Fix Imputer failing with categorical dtype (31435)
* Fix schema unification for Datasets with ragged Arrow arrays (31076)
* Fix Discretizers transforming ignored cols (31404)
* Fix to_tf when the input feature_columns is a list. (31228)
* Raise error message if user calls Dataset.__iter__ (30575)

📖Documentation:



* Refactor Ray Data API documentation (31204)
* Add seealso to map-related methods (30579)


Ray Train

🎉 New Features:



* Add option for per-epoch preprocessor (31739)

💫Enhancements:



* Change default `NCCL_SOCKET_IFNAME` to blacklist `veth` (31824)
* Introduce DatasetIterator for bulk and streaming ingest (31470)
* Clarify which `RunConfig` is used when there are multiple places to specify it (31959)
* Change `ScalingConfig` to be optional for `DataParallelTrainer`s if already in Tuner `param_space` (30920)

🔨 Fixes:



* Use specified `Preprocessor` configs when using stream API. (31725)
* Fix off-by-one AIR Trainer checkpoint ID indexing on restore (31423)
* Force GBDTTrainer to use distributed loading for Ray Datasets (31079)
* Fix bad case in ScalingConfig->RayParams (30977)
* Don't raise TuneError on `fail_fast="raise"` (30817)
* Report only once in `SklearnTrainer` (30593)
* Ensure GBDT PGFs match passed ScalingConfig (30470)

📖Documentation:



* Restructure API References (32535)
* Remove Ray Client references from Train docs/examples (32321)
* Various fixes to docstrings, documentation, and examples (29463, 30492, 30543, 30571, 30782, 31692, 31735)

🏗 Architecture refactoring:



* API Deprecations (31763)


Ray Tune

💫Enhancements:



* Improve trainable serialization error (31070)
* Add support for Nevergrad optimizer with extra parameters (31015)
* Add timeout for experiment checkpoint syncing to cloud (30855)
* Move `validate_upload_dir` to Syncer (30869)
* Enable experiment restore from moved cloud uri (31669)
* Save and restore stateful callbacks as part of experiment checkpoint (31957)

🔨 Fixes:



* Do not default to reuse_actors=True when mixins are used (31999)
* Only keep cached actors if search has not ended (31974)
* Fix best trial in ProgressReporter with nan (31276)
* Make ResultGrid return cloud checkpoints (31437)
* Wait for final experiment checkpoint sync to finish (31131)
* Fix CheckpointConfig validation for function trainables (31255)
* Fix checkpoint directory assignment for new checkpoints created after restoring a function trainable (31231)
* Fix `AxSearch` save and nan/inf result handling (31147)
* Fix `AxSearch` search space conversion for fixed list hyperparameters (31088)
* Restore searcher and scheduler properly on `Tuner.restore` (30893)
* Fix progress reporter `sort_by_metric` with nested metrics (30906)
* Don't raise TuneError on `fail_fast="raise"` (30817)
* Fix duplicate printing when trial is done (30597)

📖Documentation:



* Restructure API references (32449)
* Remove Ray Client references from Tune docs/examples (32321)
* Various fixes to docstrings, documentation, and examples (29581, 30782, 30571, 31045, 31793, 32505)

🏗 Architecture refactoring:



* Deprecate passing a custom trial executor (31792)
* Move signal handling into separate method (31004)
* Update staged resources in a fixed counter for faster lookup (32087)
* Rename `overwrite_trainable` argument in Tuner restore to `trainable` (32059)


Ray Serve

🎉 New Features:



* Serve python API to support multi application (31589)

💫Enhancements:



* Add exponential backoff when retrying replicas (31436)
* Enable Log Rotation on Serve (31844)
* Use tasks/futures for asyncio.wait (31608)
* Change target_num_ongoing_requests_per_replica to positive float (31378)

🔨 Fixes:



* Upgrade deprecated calls (31839)
* Change Gradio integration to take a builder function to avoid serialization issues (31619)
* Add initial health check before marking a replica as RUNNING (31189)

📖Documentation:



* Document end-to-end timeout in Serve (31769)
* Document Gradio visualization (28310)


RLlib

🎉 New Features:



* Gymnasium is now supported. ([Notes](https://docs.google.com/document/u/0/d/1lxYK1dI5s0Wo_jmB6V6XiP-_aEBsXDykXkD1AXRase4/edit))
* Connectors are now activated by default (31693, 30388, 31618, 31444, 31092)
* Contribution of LeelaChessZero algorithm for playing chess in a MultiAgent env. (31480)

💫Enhancements:



* [RLlib] Error out if action_dict is empty in MultiAgentEnv. (32129)
* [RLlib] Upgrade tf eager code to no longer use `experimental_relax_shapes` (but `reduce_retracing` instead). (29214)
* [RLlib] Reduce SampleBatch counting complexity (30936)
* [RLlib] Use PyTorch vectorized max() and sum() in SampleBatch.__init__ when possible (28388)
* [RLlib] Support multi-gpu CQL for torch (tf already supported). (31466)
* [RLlib] Introduce IMPALA off_policyness test with GPU (31485)
* [RLlib] Properly serialize and restore StateBufferConnector states for policy stashing (31372)
* [RLlib] Clean up deprecated concat_samples calls (31391)
* [RLlib] Better support MultiBinary spaces by treating Tuples as superset of them in ComplexInputNet. (28900)
* [RLlib] Add backward compatibility to MeanStdFilter to restore from older checkpoints. (30439)
* [RLlib] Clean up some signatures for compute_actions. (31241)
* [RLlib] Simplify logging configuration. (30863)
* [RLlib] Remove native Keras Models. (30986)
* [RLlib] Convert PolicySpec to a readable format when converting to_dict(). (31146)
* [RLlib] Issue 30394: Add proper `__str__()` method to PolicyMap. (31098)
* [RLlib] Issue 30840: Option to only checkpoint policies that are trainable. (31133)
* [RLlib] Deprecate (delete) `contrib` folder. (30992)
* [RLlib] Better behavior if user does not specify stopping condition in RLLib CLI. (31078)
* [RLlib] PolicyMap LRU cache enhancements: Swap out policies (instead of GC'ing and recreating) + use Ray object store (instead of file system). (29513)
* [RLlib] `AlgorithmConfig.overrides()` to replace `multiagent->policies->config` and `evaluation_config` dicts. (30879)
* [RLlib] `deprecation_warning(.., error=True)` should raise `ValueError`, not `DeprecationWarning`. (30255)
* [RLlib] Add `gym.spaces.Text` serialization. (30794)
* [RLlib] Convert `MultiAgentBatch` to `SampleBatch` in offline_rl.py. (30668)
* [RLlib; Tune] Make `Algorithm.train()` return Tune-style config dict (instead of AlgorithmConfig object). (30591)

🔨 Fixes:



* [RLlib] Fix waterworld example and test (32117)
* [RLlib] Change Waterworld v3 to v4 and reinstate indep. MARL test case w/ pettingzoo. (31820)
* [RLlib] Fix OPE checkpointing. Save method name in configuration dict. (31778)
* [RLlib] Fix worker state restoration. (31644)
* [RLlib] Replace ordinary pygame imports by `try_import_..()`. (31332)
* [RLlib] Remove crude VR checks in agent collector. (31564)
* [RLlib] Fixed the 'RestoreWeightsCallback' example script. (31601)
* [RLlib] Issue 28428: QMix not working w/ GPUs. (31299)
* [RLlib] Fix using yaml files with empty stopping conditions. (31363)
* [RLlib] Issue 31174: Move all checks into AlgorithmConfig.validate() (even simple ones) to avoid errors when using tune hyperopt objects. (31396)
* [RLlib] Fix `tensorflow_probability` imports. (31331)
* [RLlib] Issue 31323: BC/MARWIL/CQL do work with multi-GPU (but config validation prevents them from running in this mode). (31393)
* [RLlib] Issue 28849: DT fails with num_gpus=1. (31297)
* [RLlib] Fix `PolicyMap.__del__()` to also remove a deleted policy ID from the internal deque. (31388)
* [RLlib] Use `get_model_v2()` instead of `get_model()` with MADDPG. (30905)
* [RLlib] Policy mapping fn can not be called with keyword arguments. (31141)
* [RLlib] Issue 30213: Appending RolloutMetrics to sampler outputs should happen after(!) all callbacks (such that custom metrics for last obs are still included). (31102)
* [RLlib] Make convert_to_torch tensor adhere to docstring. (31095)
* [RLlib] Fix convert to torch tensor (31023)
* [RLlib] Issue 30221: random policy does not handle nested spaces. (31025)
* [RLlib] Fix crashing remote envs example (30562)
* [RLlib] Recursively look up the original space from obs_space (30602)

📖Documentation:



* [RLlib; docs] Change links and references in code and docs to "Farama foundation's gymnasium" (from "OpenAI gym"). (32061)


Ray Core and Ray Clusters


Ray Core

🎉 New Features:



* Task Events Backend: Ray aggregates all submitted task information to provide better observability (31840, 31761, 31278, 31247, 31316, 30934, 30979, 31207, 30867, 30829, 31524, 32157). This will back up features like task state API, advanced progress bar, and Ray timeline.

💫Enhancements:



* Remote generator now works for ray actors and ray clients (31700, 31710).
* Revamp default scheduling strategy, improve worker startup performance up to 8x for embarrassingly parallel workloads (31934, 31868).
* Worker code clean up and allow workers lazy bind to jobs (31836, 31846, 30349, 31375).
* A single Ray cluster can scale up to 2000 nodes and 20k actors(32131, 30131, 31939, 30166, 30460, 30563).
* [Out-of-memory prevention enhancement](https://docs.ray.io/en/master/ray-core/scheduling/ray-oom-prevention.html) is now GA with more robust worker killing policies and better user experiences (#32217, 32361, 32219, 31768, 32107, 31976, 31272, 31509, 31230).

🔨 Fixes:



* Improve garbage collection upon job termination (32127, 31155)
* Fix opencensus protobuf bug (31632)
* Support python 3.10 for runtime_env conda (30970)
* Fix crashes and memory leaks (31640, 30476, 31488, 31917, 30761, 31018)

📖Documentation:



* Deprecation (31845, 31140, 31528)


Ray Clusters

🎉 New Features:
* Ray-on-Spark is now available as Preview! (28771, 31397, 31962)

💫Enhancements:



* [observability] Better memory formatting for `ray status` and autoscaler (32337)
* [autoscaler] Add flag to disable periodic cluster status log. (31869)

🔨 Fixes:



* [observability][autoscaler] Ensure pending nodes is reset to 0 after scaling (32085)
* Make ~/.bashrc optional in cluster launcher commands (32393)

📖Documentation:



* Improvements to job submission
* Remove references to Ray Client


Dashboard

🎉 New Features:



* New Information Architecture (beta): We’ve restructured the Ray dashboard to be organized around user personas and workflows instead of entities. For developers, the jobs and actors tab will be most useful. For infrastructure engineers, the cluster tab may be more valuable.
* Advanced progress bar: Tasks visualization that allow you to see the progress of all your ray tasks
* Timeline view: We’ve added a button to download detailed timeline data about your ray job. Then, one can click a link and use the perfetto open-source visualization tool to visualize the timeline data.
* More metadata tables. You can now see placement groups, tasks, actors, and other information related to your jobs.

📖Documentation:



* We’ve restructured the documentation to make the dashboard documentation more prominent
* We’ve improved the documentation around setting up Prometheus and Grafana for metrics.

Many thanks to all those who contributed to this release!

minerharry, scottsun94, iycheng, DmitriGekhtman, jbedorf, krfricke, simonsays1980, eltociear, xwjiang2010, ArturNiederfahrenhorst, richardliaw, avnishn, WeichenXu123, Capiru, davidxia, andreapiso, amogkam, sven1977, scottjlee, kylehh, yhna940, rickyyx, sihanwang41, n30111, Yard1, sriram-anyscale, Emiyalzn, simran-2797, cadedaniel, harelwa, ijrsvt, clarng, pabloem, bveeramani, lukehsiao, angelinalg, dmatrix, sijieamoy, simon-mo, jbesomi, YQ-Wang, larrylian, c21, AndreKuu, maxpumperla, architkulkarni, wuisawesome, justinvyu, zhe-thoughts, matthewdeng, peytondmurray, kevin85421, tianyicui-tsy, cassidylaidlaw, gvspraveen, scv119, kyuyeonpooh, Siraj-Qazi, jovany-wang, ericl, shrekris-anyscale, Catch-Bull, jianoaix, christy, MisterLin1995, kouroshHakha, pcmoritz, csko, gjoliver, clarkzinzow, SongGuyang, ckw017, ddelange, alanwguo, Dhul-Husni, Rohan138, rkooo567, fzyzcjy, chaokunyang, 0x2b3bfa0, zoltan-fedor, Chong-Li, crypdick, jjyao, emmyscode, stephanie-wang, starpit, smorad, nikitavemuri, zcin, tbukic, ayushthe1, mattip

* [Ray Jobs API](https://docs.ray.io/en/releases-2.2.0/cluster/running-applications/job-submission/index.html#ray-jobs-api) is now GA. The Ray Jobs API allows you to submit locally developed applications to a remote Ray Cluster for execution. It simplifies the experience of packaging, deploying, and managing a Ray application.
* [Ray Dashboard](https://docs.ray.io/en/releases-2.2.0/ray-core/ray-dashboard.html#ray-dashboard) has received a number of improvements, such as the ability to see cpu flame graphs of your Ray workers and new metrics for memory usage.
* The [Out-Of-Memory (OOM) Monitor](https://docs.ray.io/en/releases-2.2.0/ray-core/scheduling/ray-oom-prevention.html) is now enabled by default. This will increase the stability of memory-intensive applications on top of Ray.
* [Ray Data] we’ve heard numerous users report that when files are too large, Ray Data can have out-of-memory or performance issues. In this release, we’re enabling [dynamic block splitting](https://docs.ray.io/en/releases-2.2.0/data/dataset-internals.html#execution-memory) by default, which will address the above issues by avoiding holding too much data in memory.

Ray Libraries

Ray AIR

🎉 New Features:
* Add a NumPy first path for Torch and TensorFlow Predictors (28917)

💫Enhancements:
* Suppress "NumPy array is not writable" error in torch conversion (29808)
* Add node rank and local world size info to session (29919)

🔨 Fixes:
* Fix MLflow database integrity error (29794)
* Fix ResourceChangingScheduler dropping PlacementGroupFactory args (30304)
* Fix bug passing 'raise' to FailureConfig (30814)
* Fix reserved CPU warning if no CPUs are used (30598)

📖Documentation:
* Fix examples and docs to specify batch_format in BatchMapper (30438)

🏗 Architecture refactoring:
* Deprecate Wandb mixin (29828)
* Deprecate Checkpoint.to_object_ref and Checkpoint.from_object_ref (30365)

Ray Data Processing

🎉 New Features:
* Support all PyArrow versions released by Apache Arrow (29993, 29999)
* Add `select_columns()` to select a subset of columns (29081)
* Add `write_tfrecords()` to write TFRecord files (29448)
* Support MongoDB data source (28550)
* Enable dynamic block splitting by default (30284)
* Add `from_torch()` to create dataset from Torch dataset (29588)
* Add `from_tf()` to create dataset from TensorFlow dataset (29591)
* Allow to set `batch_size` in `BatchMapper` (29193)
* Support read/write from/to local node file system (29565)

💫Enhancements:
* Add `include_paths` in `read_images()` to return image file path (30007)
* Print out Dataset statistics automatically after execution (29876)
* Cast tensor extension type to opaque object dtype in `to_pandas()` and `to_dask()` (29417)
* Encode number of dimensions in variable-shaped tensor extension type (29281)
* Fuse AllToAllStage and OneToOneStage with compatible remote args (29561)
* Change `read_tfrecords()` output from Pandas to Arrow format (30390)
* Handle all Ray errors in task compute strategy (30696)
* Allow nested Chain preprocessors (29706)
* Warn user if missing columns and support `str` exclude in `Concatenator` (29443)
* Raise ValueError if preprocessor column doesn't exist (29643)

🔨 Fixes:
* Support custom resource with remote args for `random_shuffle()` (29276)
* Support custom resource with remote args for `random_shuffle_each_window()` (29482)
* Add PublicAPI annotation to preprocessors (29434)
* Tensor extension column concatenation fixes (29479)
* Fix `iter_batches()` to not return empty batch (29638)
* Change `map_batches()` to fetch input blocks on-demand (29289)
* Change `take_all()` to not accept limit argument (29746)
* Convert between block and batch correctly for `map_groups()` (30172)
* Fix `stats()` call causing Dataset schema to be unset (29635)
* Raise error when `batch_format` is not specified for `BatchMapper` (30366)
* Fix ndarray representation of single-element ragged tensor slices (30514)

📖Documentation:
* Improve `map_batches()` documentation about execution model and UDF pickle-ability requirement (29233)
* Improve `to_tf()` docstring (29464)

Ray Train

🎉 New Features:
* Added MosaicTrainer (29237, 29620, 29919)

💫Enhancements:
* Fast fail upon single worker failure (29927)
* Optimize checkpoint conversion logic (29785)

🔨 Fixes:
* Propagate DatasetContext to training workers (29192)
* Show correct error message on training failure (29908)
* Fix prepare_data_loader with enable_reproducibility (30266)
* Fix usage of NCCL_BLOCKING_WAIT (29562)

📖Documentation:
* Deduplicate Train examples (29667)

🏗 Architecture refactoring:
* Hard deprecate train.report (29613)
* Remove deprecated Train modules (29960)
* Deprecate old prepare_model DDP args 30364

Ray Tune

🎉 New Features:
* Make `Tuner.restore` work with relative experiment paths (30363)
* `Tuner.restore` from a local directory that has moved (29920)

💫Enhancements:
* `with_resources` takes in a `ScalingConfig` (30259)
* Keep resource specifications when nesting `with_resources` in `with_parameters` (29740)
* Add `trial_name_creator` and `trial_dirname_creator` to `TuneConfig` (30123)
* Add option to not override the working directory (29258)
* Only convert a `BaseTrainer` to `Trainable` once in the Tuner (30355)
* Dynamically identify PyTorch Lightning Callback hooks (30045)
* Make `remote_checkpoint_dir` work with query strings (30125)
* Make cloud checkpointing retry configurable (30111)
* Sync experiment-checkpoints more often (30187)
* Update generate_id algorithm (29900)

🔨 Fixes:
* Catch SyncerCallback failure with dead node (29438)
* Do not warn in BayesOpt w/ Uniform sampler (30350)
* Fix `ResourceChangingScheduler` dropping PGF args (30304)
* Fix Jupyter output with Ray Client and `Tuner` (29956)
* Fix tests related to `TUNE_ORIG_WORKING_DIR` env variable (30134)

📖Documentation:
* Add user guide for analyzing results (using `ResultGrid` and `Result`) (29072)
* Tune checkpointing and Tuner restore docfix (29411)
* Fix and clean up PBT examples (29060)
* Fix TrialTerminationReporter in docs (29254)

🏗 Architecture refactoring:
* Remove hard deprecated SyncClient/Syncer (30253)
* Deprecate Wandb mixin, move to `setup_wandb()` function (29828)

Ray Serve

🎉 New Features:
* Guard for high latency requests (29534)
* Java API Support ([blog](http://anyscale-staging.herokuapp.com/blog/flexible-cross-language-distributed-model-inference-framework-ray-serve-with))

💫Enhancements:
* Serve K8s HA benchmarking (30278)
* Add method info for http metrics (29918)

🔨 Fixes:
* Fix log format error (28760)
* Inherit previous deployment num_replicas (29686)
* Polish serve run deploy message (29897)
* Remove calling of get_event_loop from python 3.10

RLlib

🎉 New Features:
* Fault tolerant, elastic WorkerSets: An asynchronous Ray Actor manager class is now used inside all of RLlib’s Algorithms, adding fully flexible fault tolerance to rollout workers and workers used for evaluation. If one or more workers (which are Ray actors) fails - e.g. due to a SPOT instance going down - the RLlib Algorithm will now flexibly wait it out and periodically try to recreate the failed workers. In the meantime, only the remaining healthy workers are used for sampling and evaluation. (29938, 30118, 30334, 30252, 29703, 30183, 30327, 29953)

💫Enhancements:
* RLlib CLI: A new and enhanced RLlib command line interface (CLI) has been added, allowing for automatically downloading example configuration files, python-based config files (defining an AlgorithmConfig object to use), better interoperability between training and evaluation runs, and many more. For a detailed overview of what has changed, check out the [new CLI documentation](https://docs.ray.io/en/releases-2.2.0/rllib/rllib-cli.html). (#29204, 29459, 30526, 29661, 29972)
* Checkpoint overhaul: Algorithm checkpoints and Policy checkpoints are now more cohesive and transparent. All checkpoints are now characterized by a directory (with files and maybe sub-directories), rather than a single pickle file; Both Algorithm and Policy classes now have a utility static method (`from_checkpoint()`) for directly instantiating instances from a checkpoint directory w/o knowing the original configuration used or any other information (having the checkpoint is sufficient). [For a detailed overview, see here](https://docs.ray.io/en/releases-2.2.0/rllib/rllib-saving-and-loading-algos-and-policies.html). (#28812, 29772, 29370, 29520, 29328)
* A new metric for APPO/IMPALA/PPO has been added that measures off-policy’ness: The difference in number of grad-updates the sampler policy has received thus far vs the trained policy’s number of grad-updates thus far. (29983)

🏗 Architecture refactoring:
* AlgorithmConfig classes: All of RLlib’s Algorithms, RolloutWorkers, and other important classes now use AlgorithmConfig objects under the hood, instead of python config dicts. It is no longer recommended (however, still supported) to create a new algorithm (or a Tune+RLlib experiment) using a python dict as configuration. For more details on how to convert your scripts to the [new AlgorithmConfig design, see here](https://docs.ray.io/en/releases-2.2.0/rllib/rllib-training.html#configuring-rllib-algorithms). (29796, 30020, 29700, 29799, 30096, 29395, 29755, 30053, 29974, 29854, 29546, 30042, 29544, 30079, 30486, 30361)
* Major progress was made on the new Connector API and making sure it can be used (tentatively) with the “config.rollouts(enable_connectors=True)” flag. Will be fully supported, across all of RLlib’s algorithms, in Ray 2.3. (30307, 30434, 30459, 30308, 30332, 30320, 30383, 30457, 30446, 30024, 29064, 30398, 29385, 30481, 30241, 30285, 30423, 30288, 30313, 30220, 30159)
* Progress was made on the upcoming RLModule/RLTrainer/RLOptimizer APIs. (30135, 29600, 29599, 29449, 29642)

🔨 Fixes:
* Various bug fixes: 25925, 30279, 30478, 30461, 29867, 30099, 30185, 29222, 29227, 29494, 30257, 29798, 30176, 29648, 30331

📖Documentation:
* [RLlib CLI](https://docs.ray.io/en/releases-2.2.0/rllib/rllib-cli.html), [Checkpoint overhaul](https://docs.ray.io/en/releases-2.2.0/rllib/rllib-saving-and-loading-algos-and-policies.html), [AlgorithmConfigs](https://docs.ray.io/en/releases-2.2.0/rllib/rllib-training.html#configuring-rllib-algorithms)
* Minor fixes: 29261, 29752

Ray Core and Ray Clusters

Ray Core

🎉 New Features:
* [Out-of-memory monitor](https://docs.ray.io/en/releases-2.2.0/ray-core/scheduling/ray-oom-prevention.html) is now Beta and is enabled by default.

💫Enhancements:
* The [Ray Jobs API](https://docs.ray.io/en/releases-2.2.0/cluster/running-applications/job-submission/index.html#ray-jobs-api) has graduated from Beta to GA. This means Ray Jobs will maintain API backward compatibility.
* Run Ray job entrypoint commands (“driver scripts”) on worker nodes by specifying `entrypoint_num_cpus`, `entrypoint_num_gpus`, or `entrypoint_resources`. (28564, 28203)
* (Beta) OpenAPI spec for Ray Jobs REST API (30417)
* Improved Ray health checking mechanism. The fix will reduce the frequency of GCS marking raylets fail mistakenly when it is overloaded. (29346, 29442, 29389, 29924)

🔨 Fixes:
* Various fixes for hanging / deadlocking (29491, 29763, 30371, 30425)
* Set OMP_NUM_THREADS to `num_cpus` required by task/actors by default (30496)
* set worker non recyclable if gpu is envolved by default (30061)

📖Documentation:
* General improvements of Ray Core docs, including design patterns and tasks.

Ray Clusters

💫Enhancements:
* Stability improvements for Ray Autoscaler / KubeRay Operator integration. (29933 , 30281, 30502)

Dashboard

🎉 New Features:
* Additional improvements from the default metrics dashboard. We now have actor, placement group, and per component memory usage breakdown. [You can see details from the doc](https://docs.ray.io/en/releases-2.2.0/ray-observability/ray-metrics.html#system-metrics).
* New profiling feature using py-spy under the hood. You can click buttons to see stack trace or cpu flame graphs of your workers.
* Autoscaler and job events are available from the dashboard. You can also access the same data using `ray list cluster-events`.

🔨 Fixes:
* Stability improvements from the dashboard
* Dashboard now works at large scale cluster! It is tested with 250 nodes and 10K+ actors (which matches the [Ray scalability envelope](https://github.com/ray-project/ray/tree/master/release/benchmarks)).
* Smarter api fetching logic. We now wait for the previous API to finish before sending a new API request when polling for new data.
* Fix [agent memory leak](https://github.com/ray-project/ray/issues/29199) and high CPU usage.

💫Enhancements:
* General improvements to the progress bar. You can now see progress bars for each task name if you drill into the job details.
* More metadata is available in the jobs and actors tables.
* There is now a feedback button embedded into the dashboard. Please submit any bug reports or suggestions!

Many thanks to all those who contributed to this release!

shrekris-anyscale, rickyyx, scottjlee, shogohida, liuyang-my, matthewdeng, wjrforcyber, linusbiostat, clarkzinzow, justinvyu, zygi, christy, amogkam, cool-RR, jiaodong, EvgeniiTitov, jjyao, ilee300a, jianoaix, rkooo567, mattip, maxpumperla, ericl, cadedaniel, bveeramani, rueian, stephanie-wang, lcipolina, bparaj, JoonHong-Kim, avnishn, tomsunelite, larrylian, alanwguo, VishDev12, c21, dmatrix, xwjiang2010, thomasdesr, tiangolo, sokratisvas, heyitsmui, scv119, pcmoritz, bhavika, yzs981130, andraxin, Chong-Li, clarng, acxz, ckw017, krfricke, kouroshHakha, sijieamoy, iycheng, gjoliver, peytondmurray, xcharleslin, DmitriGekhtman, andreichalapco, vitrioil, architkulkarni, simon-mo, ArturNiederfahrenhorst, sihanwang41, pabloem, sven1977, avivhaber, wuisawesome, jovany-wang, Yard1

Release Highlights