Torchrl

What's Changed

We introduce wrappers for [**ML-Agents**](https://github.com/Unity-Technologies/ml-agents) and [**OpenSpiel**](https://github.com/google-deepmind/open_spiel). See the doc [here for OpenSpiel](https://pytorch.org/rl/main/reference/generated/torchrl.envs.OpenSpielEnv.html?highlight=openspiel#torchrl.envs.OpenSpielEnv) and [here for MLAgents](https://pytorch.org/rl/main/reference/generated/torchrl.envs.UnityMLAgentsEnv.html#torchrl.envs.UnityMLAgentsEnv).

We introduce support for [partial steps](2377, 2381), allowing you to run rollouts that ends only when all envs are done without resetting those who have reached a termination point.

We add the capability of passing replay buffers directly to data collectors, to avoid inter-process synced communications - thereby drastically speeding up data collection. See the [doc of the collectors](https://pytorch.org/rl/main/reference/collectors.html) for more info.

The GAIL algorithm has also been integrated in the library (2273).

We ensure that all loss modules are compatible with torch.compile without graph breaks (for a typical built). Execution of compiled losses is usually in the range of 2x faster than its eager counterpart.

Finally, we have sadly decided not to support Gymnasium v1.0 and future releases as the new autoreset API is fundamentally incompatible with TorchRL. Furthermore, it does not guarantee the same level of reproducibility as previous releases. See [this discussion](https://github.com/pytorch/rl/discussions/2483) for more information.

We provide wheels for aarch64 machines, but not being able to upload them to PyPI we provide them attached to these release notes.

Deprecations
* [Deprecation] Deprecate default num_cells in MLP (2395) by vmoens
* [Deprecations] Deprecate in view of v0.6 release 2446 by vmoens

New environments
* [Feature] Add `OpenSpielWrapper` and `OpenSpielEnv` (2345) by kurtamohler
* [Feature] Add env wrapper for Unity MLAgents (2469) by kurtamohler

New features
* [Feature] Add `group_map` support to MLAgents wrappers (2491) by kurtamohler
* [Feature] Add scheduler for alpha/beta parameters of PrioritizedSampler (2452) Co-authored-by: Vincent Moens by LTluttmann
* [Feature] Check number of kwargs matches num_workers (2465) Co-authored-by: Vincent Moens by antoine.broyelle
* [Feature] Compiled and cudagraph for policies 2478 by vmoens
* [Feature] Consistent Dropout (2399) Co-authored-by: Vincent Moens by depictiger
* [Feature] Deterministic sample for Masked one-hot 2440 by vmoens
* [Feature] Dict specs in vmas (2415) Co-authored-by: Vincent Moens by 55539777+matteobettini
* [Feature] Ensure transformation keys have the same number of elements (2466) by f.broyelle
* [Feature] Make benchmarked losses compatible with torch.compile 2405 by vmoens
* [Feature] Partial steps in batched envs 2377 by vmoens
* [Feature] Pass replay buffers to MultiaSyncDataCollector 2387 by vmoens
* [Feature] Pass replay buffers to SyncDataCollector 2384 by vmoens
* [Feature] Prevent loading existing mmap files in storages if they already exist 2438 by vmoens
* [Feature] RNG for RBs (2379) by vmoens
* [Feature] Randint on device for buffers 2470 by vmoens
* [Feature] SAC compatibility with composite distributions. (2447) by albertbou92
* [Feature] Store MARL parameters in module (2351) by vmoens
* [Feature] Support wrapping IsaacLab environments with GymEnv (2380) by yu-fz
* [Feature] TensorDictMap 2306 by vmoens
* [Feature] TensorDictMap Query module 2305 by vmoens
* [Feature] TensorDictMap hashing functions 2304 by vmoens
* [Feature] break_when_all_done in rollout 2381 by vmoens
* [Feature] inline `hold_out_net` 2499 by vmoens
* [Feature] replay_buffer_chunk 2388 by vmoens

New Algorithms
* [Algorithm] GAIL (2273) Co-authored-by: Vincent Moens by Sebastian.dittert

Fixes
* [BugFix, CI] Set `TD_GET_DEFAULTS_TO_NONE=1` in all CIs (2363) by vmoens
* [BugFix] Add `MultiCategorical` support in PettingZoo action masks (2485) Co-authored-by: Vincent Moens by matteobettini
* [BugFix] Allow for composite action distributions in PPO/A2C losses (2391) by albertbou92
* [BugFix] Avoid `reshape(-1)` for inputs to `DreamerActorLoss` (2496) by kurtamohler
* [BugFix] Avoid `reshape(-1)` for inputs to `objectives` modules (2494) Co-authored-by: Vincent Moens by kurtamohler
* [BugFix] Better dumps/loads (2343) by vmoens
* [BugFix] Extend RB with lazy stack 2453 by vmoens
* [BugFix] Extend RB with lazy stack (revamp) 2454 by vmoens
* [BugFix] Fix Compose input spec transform (2463) Co-authored-by: Louis Faury louisfaury
* [BugFix] Fix DeviceCastTransform 2471 by vmoens
* [BugFix] Fix LSTM in GAE with vmap (2376) by vmoens
* [BugFix] Fix MARL-DDPG tutorial and other MODE usages (2373) by vmoens
* [BugFix] Fix displaying of tensor sizes in buffers 2456 by vmoens
* [BugFix] Fix dumps for SamplerWithoutReplacement (2506) by vmoens
* [BugFix] Fix get-related errors (2361) by vmoens
* [BugFix] Fix invalid CUDA ID error when loading Bounded variables across devices (2421) by cbhua
* [BugFix] Fix listing of updated keys in collectors (2460) by vmoens
* [BugFix] Fix old deps tests 2500 by vmoens
* [BugFix] Fix support for MiniGrid envs (2416) by kurtamohler
* [BugFix] Fix tictactoeenv.py 2417 by vmoens
* [BugFix] Fixes to RenameTransform (2442) Co-authored-by: Vincent Moens by thomasbbrunner
* [BugFix] Make sure keys are exclusive in envs (1912) by vmoens
* [BugFix] TensorDictPrimer updates spec instead of overwriting (2332) Co-authored-by: Vincent Moens by matteobettini
* [BugFix] Use a RL-specific NO_DEFAULT instead of TD's one (2367) by vmoens
* [BugFix] compatibility to new Composite dist log_prob/entropy APIs 2435 by vmoens
* [BugFix] torch 2.0 compatibility fix 2475 by vmoens

Performance
* [Performance] Faster `CatFrames.unfolding` with `padding="same"` (2407) by kurtamohler
* [Performance] Faster `PrioritizedSliceSampler._padded_indices` (2433) by kurtamohler
* [Performance] Faster `SliceSampler._tensor_slices_from_startend` (2423) by kurtamohler
* [Performance] Faster target update using foreach (2046) by vmoens

Documentation
* [Doc] Better doc for inverse transform semantic 2459 by vmoens
* [Doc] Correct minor erratum in `knowledge_base` entry (2383) by depictiger
* [Doc] Document losses in README.md 2408 by vmoens
* [Doc] Fix README example (2398) by vmoens
* [Doc] Fix links to tutos (2409) by vmoens
* [Doc] Fix pip3install typos in Readme (2342) by 43245438+TheRisenPhoenix
* [Doc] Fix policy in getting started (2429) by vmoens
* [Doc] Fix tutorials for release 2476 by vmoens
* [Doc] Fix wrong default value for flatten_tensordicts in ReplayBufferTrainer (2502) by vmoens
* [Doc] Minor fixes to comments and docstrings (2443) by thomasbbrunner
* [Doc] Refactor README (2352) by vmoens
* [Docs] Use more appropriate ActorValueOperator in PPOLoss documentation (2350) by GaetanLepage
* [Documentation] README rewrite and broken links (2023) by vmoens

Not user facing
* [CI, BugFix] Fix CI (2489) by vmoens
* [CI] Add aarch64-linux wheels (2434) by vmoens
* [CI] Disable compile tests on windows 2510 by vmoens
* [CI] Fix 3.12 gymnasium installation 2474 by vmoens
* [CI] Fix CI errors (2394) by vmoens
* [CI] Fix GPU benchmark upload (2508) by vmoens
* [CI] Fix Minari tests (2419) Co-authored-by: Vincent Moens by 42100908+younik
* [CI] Fix benchmark workflows (2488) by vmoens
* [CI] Fix broken workflows (2418) by vmoens
* [CI] Fix broken workflows (2428) by vmoens
* [CI] Fix gymnasium version in minari 2512 by vmoens
* [CI] Fix h5py dependency in olddeps 2513 by vmoens
* [CI] Fix windows build legacy 2450 by vmoens
* [CI] Fix winndows compile tests 2511 by vmoens
* [CI] Remove 3.8 jobs 2412 by vmoens
* [CI] Resolve DMC and mujoco pinned versions (2396) by vmoens
* [CI] Run docs on all PRs (2413) by vmoens
* [CI] pin DMC and mujoco (2374) by vmoens
* [Minor] Fix test_compose_action_spec (2493) Co-authored-by: Louis Faury by louisfaury
* [Minor] Fix typos in `advantages.py` (2492) Co-authored-by: Louis Faury by louisfaury
* [Quality] Split utils.h and utils.cpp (2348) by vmoens
* [Refactor] Limit the deepcopies in collectors 2451 by vmoens
* [Refactor] Refactor calls to get without default that raise KeyError (2353) by vmoens
* [Refactor] Rename specs to simpler names (2368) by vmoens
* [Refactor] Use empty_like in storage construction 2455 by vmoens
* [Versioning] Fix torch deps (2340) by vmoens
* [Versioning] Gymnasium 1.0 incompatibility errors 2484 by vmoens
* [Versioning] Versions for 0.6 (2509) by vmoens

New Contributors
As always, we want to show how appreciative we are of the vibrant open-source community that keeps TorchRL alive.

* yu-fz made their first contribution in https://github.com/pytorch/rl/pull/2380
* cbhua made their first contribution in https://github.com/pytorch/rl/pull/2421
* younik made their first contribution in https://github.com/pytorch/rl/pull/2419
* thomasbbrunner made their first contribution in https://github.com/pytorch/rl/pull/2442
* LTluttmann made their first contribution in https://github.com/pytorch/rl/pull/2452
* louisfaury made their first contribution in https://github.com/pytorch/rl/pull/2463
* antoinebrl made their first contribution in https://github.com/pytorch/rl/pull/2466

**Full Changelog**: https://github.com/pytorch/rl/compare/v0.5.0...v0.6.0

What's Changed

This new release makes it possible to run environments that output non-tensor data. 1944

We also introduce dynamic specs, allowing environments to change the size of the observations / actions during the
course of a rollout. This feature is compatible with parallel environment and collectors! 2143

Additionally, it is now possible to update a Replay Buffer in-place by assigning values at a given index. 2224

Finally, TorchRL is now compatible with Python 3.12 (2282, 2281).

As always, a huge thanks to the vibrant OSS community that helps us developt this library!

New algorithms
* [Algorithm] CrossQ by BY571 in https://github.com/pytorch/rl/pull/2033
* [Algorithm] TD3+BC by BY571 in https://github.com/pytorch/rl/pull/2249

Features
* [Feature] ActionDiscretizer by vmoens in https://github.com/pytorch/rl/pull/2247
* [Feature] Add KL approximation in PPO loss metadata by albertbou92 in https://github.com/pytorch/rl/pull/2166
* [Feature] Add `modules.AdditiveGaussianModule` by kurtamohler in https://github.com/pytorch/rl/pull/2296
* [Feature] Add `modules.OrnsteinUhlenbeckProcessModule` by kurtamohler in https://github.com/pytorch/rl/pull/2297
* [Feature] Autocomplete for losses by vmoens in https://github.com/pytorch/rl/pull/2148
* [Feature] Crop Transform by albertbou92 in https://github.com/pytorch/rl/pull/2336
* [Feature] Dynamic specs by vmoens in https://github.com/pytorch/rl/pull/2143
* [Feature] Extract primers from modules that contain RNNs by albertbou92 in https://github.com/pytorch/rl/pull/2127
* [Feature] Jumanji from_pixels=True by vmoens in https://github.com/pytorch/rl/pull/2129
* [Feature] Make ProbabilisticActor compatible with Composite distributions by vmoens in https://github.com/pytorch/rl/pull/2220
* [Feature] Replay buffer checkpointers by vmoens in https://github.com/pytorch/rl/pull/2137
* [Feature] Some improvements to VecNorm by vmoens in https://github.com/pytorch/rl/pull/2251
* [Feature] Split-trajectories and represent as nested tensor by vmoens in https://github.com/pytorch/rl/pull/2043
* [Feature] _make_ordinal_device by vmoens in https://github.com/pytorch/rl/pull/2237
* [Feature] assigning values to RB storage by vmoens in https://github.com/pytorch/rl/pull/2224

Bug fixes
* [BugFix,Feature] Allow non-tensor data in envs by vmoens in https://github.com/pytorch/rl/pull/1944
* [BugFix] Allow zero alpha value for PrioritizedSampler by albertbou92 in https://github.com/pytorch/rl/pull/2164
* [BugFix] Expose MARL modules by vmoens in https://github.com/pytorch/rl/pull/2321
* [BugFix] Fit vecnorm out_keys by vmoens in https://github.com/pytorch/rl/pull/2157
* [BugFix] Fix Brax by vmoens in https://github.com/pytorch/rl/pull/2233
* [BugFix] Fix OOB sampling in PrioritizedSliceSampler by vmoens in https://github.com/pytorch/rl/pull/2239
* [BugFix] Fix VecNorm test in test_collectors.py by vmoens in https://github.com/pytorch/rl/pull/2162
* [BugFix] Fix `to` in MultiDiscreteTensorSpec by Quinticx in https://github.com/pytorch/rl/pull/2204
* [BugFix] Fix and test PRB priority update across dims and rb types by vmoens in https://github.com/pytorch/rl/pull/2244
* [BugFix] Fix another ctx test by vmoens in https://github.com/pytorch/rl/pull/2284
* [BugFix] Fix async gym env with non-sync resets by vmoens in https://github.com/pytorch/rl/pull/2170
* [BugFix] Fix async gym when all reset by vmoens in https://github.com/pytorch/rl/pull/2144
* [BugFix] Fix brax wrapping by vmoens in https://github.com/pytorch/rl/pull/2190
* [BugFix] Fix collector tests where device ordinal is needed by vmoens in https://github.com/pytorch/rl/pull/2240
* [BugFix] Fix collectors with non tensors by vmoens in https://github.com/pytorch/rl/pull/2232
* [BugFix] Fix done/terminated computation in slice samplers by vmoens in https://github.com/pytorch/rl/pull/2213
* [BugFix] Fix info reading with async gym by vmoens in https://github.com/pytorch/rl/pull/2150
* [BugFix] Fix isaac - bis by vmoens in https://github.com/pytorch/rl/pull/2119
* [BugFix] Fix lib tests by vmoens in https://github.com/pytorch/rl/pull/2218
* [BugFix] Fix max value within buffer during update priority by vmoens in https://github.com/pytorch/rl/pull/2242
* [BugFix] Fix max-priority update by vmoens in https://github.com/pytorch/rl/pull/2215
* [BugFix] Fix non-tensor passage in _StepMDP by vmoens in https://github.com/pytorch/rl/pull/2260
* [BugFix] Fix non-tensor passage in _StepMDP by vmoens in https://github.com/pytorch/rl/pull/2262
* [BugFix] Fix prefetch in samples without replacement - .sample() compatibility issues by vmoens in https://github.com/pytorch/rl/pull/2226
* [BugFix] Fix sampling in NonTensorSpec by vmoens in https://github.com/pytorch/rl/pull/2172
* [BugFix] Fix sampling of values from NonTensorSpec by vmoens in https://github.com/pytorch/rl/pull/2169
* [BugFix] Fix slice sampler end computation at the cursor place by vmoens in https://github.com/pytorch/rl/pull/2225
* [BugFix] Fix sliced PRB when only traj is provided by vmoens in https://github.com/pytorch/rl/pull/2228
* [BugFix] Fix strict length in PRB+SliceSampler by vmoens in https://github.com/pytorch/rl/pull/2202
* [BugFix] Fix strict_length in prioritized slice sampler by vmoens in https://github.com/pytorch/rl/pull/2194
* [BugFix] Fix tanh normal mode by vmoens in https://github.com/pytorch/rl/pull/2198
* [BugFix] Fix tensordict private imports by vmoens in https://github.com/pytorch/rl/pull/2275
* [BugFix] Fix test_specs.py by vmoens in https://github.com/pytorch/rl/pull/2214
* [BugFix] Fix torch 2.3 compatibility of padding indices by vmoens in https://github.com/pytorch/rl/pull/2216
* [BugFix] Fix truncated normal by vmoens in https://github.com/pytorch/rl/pull/2147
* [BugFix] Fix typo in weight assignment in PRB by vmoens in https://github.com/pytorch/rl/pull/2241
* [BugFix] Fix update_priority generic signature for Samplers by vmoens in https://github.com/pytorch/rl/pull/2252
* [BugFix] Fix vecnorm state-dicts by vmoens in https://github.com/pytorch/rl/pull/2158
* [BugFix] Global import of optional library by matteobettini in https://github.com/pytorch/rl/pull/2217
* [BugFix] Gym async with _reset full of `True` by vmoens in https://github.com/pytorch/rl/pull/2145
* [BugFix] MLFlow logger by GJBoth in https://github.com/pytorch/rl/pull/2152
* [BugFix] Make DMControlEnv aware of truncated signals by vmoens in https://github.com/pytorch/rl/pull/2196
* [BugFix] Make `_reset` follow `done` shape by matteobettini in https://github.com/pytorch/rl/pull/2189
* [BugFix] `EnvBase._complete_done` to complete "terminated" key properly by kurtamohler in https://github.com/pytorch/rl/pull/2294
* [BugFix] `LazyTensorStorage` only allocates data on the given device by matteobettini in https://github.com/pytorch/rl/pull/2188
* [BugFix] `done = done | truncated` in collector by vmoens in https://github.com/pytorch/rl/pull/2333
* [BugFix] buffer __iter__ for samplers without replacement + prefetch by JulianKu in https://github.com/pytorch/rl/pull/2185
* [BugFix] buffer `__iter__` for samplers without replacement + prefetch by JulianKu in https://github.com/pytorch/rl/pull/2178
* [BugFix] missing deprecated kwargs by fedebotu in https://github.com/pytorch/rl/pull/2125

Docs
* [Doc] Add Custom Options for VideoRecorder by N00bcak in https://github.com/pytorch/rl/pull/2259
* [Doc] Add documentation for masks in tensor specs by kurtamohler in https://github.com/pytorch/rl/pull/2289
* [Doc] Better doc for make_tensordict_primer by vmoens in https://github.com/pytorch/rl/pull/2324
* [Doc] Dynamic envs by vmoens in https://github.com/pytorch/rl/pull/2191
* [Doc] Edit README for local installs by vmoens in https://github.com/pytorch/rl/pull/2255
* [Doc] Fix algorithms references in tutos by vmoens in https://github.com/pytorch/rl/pull/2320
* [Doc] Fix documentation mismatch for default argument by TheRisenPhoenix in https://github.com/pytorch/rl/pull/2149
* [Doc] Fix links in doc by vmoens in https://github.com/pytorch/rl/pull/2151
* [Doc] Fix mistakes in docs for Trainer checkpointing backends by kurtamohler in https://github.com/pytorch/rl/pull/2285
* [Doc] Indicate necessary context to run multiprocessed collectors in doc by GJBoth in https://github.com/pytorch/rl/pull/2126
* [Doc] Restore colab links by vmoens in https://github.com/pytorch/rl/pull/2197
* [Doc] Update README.md by KPCOFGS in https://github.com/pytorch/rl/pull/2155
* [Doc] default_interaction_type doc by vmoens in https://github.com/pytorch/rl/pull/2177
* [Docs] InitTracker cleanup by matteobettini in https://github.com/pytorch/rl/pull/2330
* [Docs] Reintroduce BenchMARL pointers in MARL tutos by matteobettini in https://github.com/pytorch/rl/pull/2159


Performance
* [Performance, Refactor, BugFix] Faster loading of uninitialized storages by vmoens in https://github.com/pytorch/rl/pull/2221
* [Performance] consolidate TDs in ParallelEnv without buffers by vmoens in https://github.com/pytorch/rl/pull/2231

Others
* Fix "Run in Colab" and "Download Notebook" links in tutorials by kurtamohler in https://github.com/pytorch/rl/pull/2268
* Fix brax examples by Jendker in https://github.com/pytorch/rl/pull/2318
* Fixed several broken links in readme.md by drMJ in https://github.com/pytorch/rl/pull/2156
* Revert "[BugFix] Fix non-tensor passage in _StepMDP" by vmoens in https://github.com/pytorch/rl/pull/2261
* Revert "[BugFix] Fix tensordict private imports" by vmoens in https://github.com/pytorch/rl/pull/2276
* Revert "[BugFix] buffer `__iter__` for samplers without replacement + prefetch" by vmoens in https://github.com/pytorch/rl/pull/2182
* [CI, Tests] Fix windows tests by vmoens in https://github.com/pytorch/rl/pull/2337
* [CI] Bump jinja2 from 3.1.3 to 3.1.4 in /docs by dependabot in https://github.com/pytorch/rl/pull/2250
* [CI] Fix CI by vmoens in https://github.com/pytorch/rl/pull/2245
* [CI] Fix nightly by vmoens in https://github.com/pytorch/rl/pull/2279
* [CI] Fix wheels by vmoens in https://github.com/pytorch/rl/pull/2274
* [CI] Pin transformers version to < 4.42.0 to make vmap happy by vmoens in https://github.com/pytorch/rl/pull/2278
* [CI] Upgrade SDL to install pygame 2.6 by vmoens in https://github.com/pytorch/rl/pull/2248
* [CI] Windows build fix by vmoens in https://github.com/pytorch/rl/pull/2335
* [CI] python 3.12 nightlies by vmoens in https://github.com/pytorch/rl/pull/2281
* [Example,BugFix] Add a Async gym env example by vmoens in https://github.com/pytorch/rl/pull/2139
* [MINOR] Fix unclear language by software-samurai in https://github.com/pytorch/rl/pull/2165
* [Minor] Code quality improvements by vmoens in https://github.com/pytorch/rl/pull/2140
* [Quality] Fix low/high in SOTA implementations by vmoens in https://github.com/pytorch/rl/pull/2266
* [Quality] Fix repr of MARL modules by vmoens in https://github.com/pytorch/rl/pull/2192
* [Quality] Remove global seeding in set_seed by vmoens in https://github.com/pytorch/rl/pull/2195
* [Quality] Warn if the sampler is not prioritized but update_priority is called by vmoens in https://github.com/pytorch/rl/pull/2253
* [Quality] better error message for CompositeSpec shape mismatch by vmoens in https://github.com/pytorch/rl/pull/2223
* [Refactor] Deprecate NormalParamWrapper by vmoens in https://github.com/pytorch/rl/pull/2308
* [Refactor] Remove `_run_checks` from `TensorDict.__init__` by vmoens in https://github.com/pytorch/rl/pull/2256
* [Refactor] Update all instances of exploration `*Wrapper` to `*Module` by kurtamohler in https://github.com/pytorch/rl/pull/2298
* [Refactor] Use td.transpose in multi-step transform by vmoens in https://github.com/pytorch/rl/pull/2288
* [Refactor] tensordict._tensordict -> tensordict._C by vmoens in https://github.com/pytorch/rl/pull/2286
* [Tests] Fix VMAS tests by matteobettini in https://github.com/pytorch/rl/pull/2287
* [Tests] Fix windows tests by vmoens in https://github.com/pytorch/rl/pull/2219
* [Versioning] Add python 3.12 to setup.py by vmoens in https://github.com/pytorch/rl/pull/2282
* [Versioning] Allow any torch version for local builds by vmoens in https://github.com/pytorch/rl/pull/2130
* [Versioning] Bump torch 2.0 as minimal version by vmoens in https://github.com/pytorch/rl/pull/2200
* [Versioning] v0.5 bump by vmoens in https://github.com/pytorch/rl/pull/2267
* [Versioning] windows build - add legacy back and .bat env-script by vmoens in https://github.com/pytorch/rl/pull/2339
* init by vmoens in https://github.com/pytorch/rl/pull/2322

New Contributors
* GJBoth made their first contribution in https://github.com/pytorch/rl/pull/2126
* TheRisenPhoenix made their first contribution in https://github.com/pytorch/rl/pull/2149
* drMJ made their first contribution in https://github.com/pytorch/rl/pull/2156
* KPCOFGS made their first contribution in https://github.com/pytorch/rl/pull/2155
* software-samurai made their first contribution in https://github.com/pytorch/rl/pull/2165
* JulianKu made their first contribution in https://github.com/pytorch/rl/pull/2178
* Quinticx made their first contribution in https://github.com/pytorch/rl/pull/2204
* kurtamohler made their first contribution in https://github.com/pytorch/rl/pull/2268
* N00bcak made their first contribution in https://github.com/pytorch/rl/pull/2259
* Jendker made their first contribution in https://github.com/pytorch/rl/pull/2318

**Full Changelog**: https://github.com/pytorch/rl/compare/v0.4.0...v0.5.0

New Features:

- Better video rendering
* [Feature] A PixelRenderTransform by vmoens in https://github.com/pytorch/rl/pull/2099
* [Feature] Video recording in SOTA examples by vmoens in https://github.com/pytorch/rl/pull/2070
* [Feature] VideoRecorder for datasets and replay buffers by vmoens in https://github.com/pytorch/rl/pull/2069
- Replay buffer: sampling trajectories is now much easier, cleaner and faster
* [Benchmark] Benchmark slice sampler by vmoens in https://github.com/pytorch/rl/pull/1992
* [Feature] Add PrioritizedSliceSampler by Cadene in https://github.com/pytorch/rl/pull/1875
* [Feature] Span slice indices on the left and on the right by vmoens in https://github.com/pytorch/rl/pull/2107
* [Feature] batched trajectories - SliceSampler compatibility by vmoens in https://github.com/pytorch/rl/pull/1775
* [Performance] Faster slice sampler by vmoens in https://github.com/pytorch/rl/pull/2031
- Datasets: allow preprocessing datasets after download
* [Feature] Preproc for datasets by vmoens in https://github.com/pytorch/rl/pull/1989
- Losses: reduction parameters and non-functional execution
* [Feature] Add reduction parameter to On-Policy losses. by albertbou92 in https://github.com/pytorch/rl/pull/1890
* [Feature] Adds value clipping in ClipPPOLoss loss by albertbou92 in https://github.com/pytorch/rl/pull/2005
* [Feature] Offline objectives reduction parameter by albertbou92 in https://github.com/pytorch/rl/pull/1984
- Environment API: support "fork" start method in ParallelEnv, better handling of auto-resetting envs.
* [Feature] Use non-default mp start method in ParallelEnv by vmoens in https://github.com/pytorch/rl/pull/1966
* [Feature] Auto-resetting envs by vmoens in https://github.com/pytorch/rl/pull/2073
- Transforms
* [Feature] Allow any callable to be used as transform by vmoens in https://github.com/pytorch/rl/pull/2027
* [Feature] invert transforms appended to a RB by vmoens in https://github.com/pytorch/rl/pull/2111
* [Feature] Extend TensorDictPrimer default_value options by albertbou92 in https://github.com/pytorch/rl/pull/2071
* [Feature] Fine grained DeviceCastTransform by vmoens in https://github.com/pytorch/rl/pull/2041
* [Feature] BatchSizeTransform by vmoens in https://github.com/pytorch/rl/pull/2030
* [Feature] Allow non-sorted keys in CatFrames by vmoens in https://github.com/pytorch/rl/pull/1913
* [Feature] env.append_transform by vmoens in https://github.com/pytorch/rl/pull/2040
- New environment and improvements:
* [Environment] Meltingpot by matteobettini in https://github.com/pytorch/rl/pull/2054
* [Feature] Return depth from RoboHiveEnv by sriramsk1999 in https://github.com/pytorch/rl/pull/2058
* [Feature] PettingZoo possibility to choose reset strategy by matteobettini in https://github.com/pytorch/rl/pull/2048

Other features
* [Feature] Add time_dim arg in value modules by vmoens in https://github.com/pytorch/rl/pull/1946
* [Feature] Batched actions wrapper by vmoens in https://github.com/pytorch/rl/pull/2018
* [Feature] Better repr of RBs by vmoens in https://github.com/pytorch/rl/pull/1991
* [Feature] Execute rollouts with regular nn.Module instances by vmoens in https://github.com/pytorch/rl/pull/1947
* [Feature] Logger by vmoens in https://github.com/pytorch/rl/pull/1858
* [Feature] Passing lists of keyword arguments in ``reset`` for batched envs by vmoens in https://github.com/pytorch/rl/pull/2076
* [Feature] RB MultiStep transform by vmoens in https://github.com/pytorch/rl/pull/2008
* [Feature] Replace RewardClipping with SignTransform in Atari examples by albertbou92 in https://github.com/pytorch/rl/pull/1870
* [Feature] `reset_parameters` for multiagent nets by matteobettini in https://github.com/pytorch/rl/pull/1970
* [Feature] optionally set truncated = True at the end of rollouts by vmoens in https://github.com/pytorch/rl/pull/2042

Miscellaneous

* Fix onw typo by kit1980 in https://github.com/pytorch/rl/pull/1917
* Rename SOTA-IMPLEMENTATIONS.md to README.md by matteobettini in https://github.com/pytorch/rl/pull/2093
* Revert "[BugFix] Fix Isaac" by vmoens in https://github.com/pytorch/rl/pull/2118
* Update getting-started-5.py by vmoens in https://github.com/pytorch/rl/pull/1894
* [BugFix, Performance] Fewer imports at root by vmoens in https://github.com/pytorch/rl/pull/1930
* [BugFix,CI] Fix Windows CI by vmoens in https://github.com/pytorch/rl/pull/1983
* [BugFix,CI] Fix sporadically failing tests in CI by vmoens in https://github.com/pytorch/rl/pull/2098
* [BugFix,Refactor] Dreamer refactor by BY571 in https://github.com/pytorch/rl/pull/1918
* [BugFix] Adaptable non-blocking for mps and non cuda device in batched-envs by vmoens in https://github.com/pytorch/rl/pull/1900
* [BugFix] Call contiguous on rollout results in TestMultiStepTransform by vmoens in https://github.com/pytorch/rl/pull/2025
* [BugFix] Dedicated tests for on policy losses reduction parameter by albertbou92 in https://github.com/pytorch/rl/pull/1974
* [BugFix] Extend with a list of tensordicts by vmoens in https://github.com/pytorch/rl/pull/2032
* [BugFix] Fix Atari DQN ensembling by vmoens in https://github.com/pytorch/rl/pull/1981
* [BugFix] Fix CQL/IQL pbar update by vmoens in https://github.com/pytorch/rl/pull/2020
* [BugFix] Fix Exclude / Double2Float transforms by vmoens in https://github.com/pytorch/rl/pull/2101
* [BugFix] Fix Isaac by vmoens in https://github.com/pytorch/rl/pull/2072
* [BugFix] Fix KLPENPPOLoss KL computation by vmoens in https://github.com/pytorch/rl/pull/1922
* [BugFix] Fix MPS sync in device transform by vmoens in https://github.com/pytorch/rl/pull/2061
* [BugFix] Fix OOB TruncatedNormal LP by vmoens in https://github.com/pytorch/rl/pull/1924
* [BugFix] Fix R2Go once more by vmoens in https://github.com/pytorch/rl/pull/2089
* [BugFix] Fix Ray collector example error by albertbou92 in https://github.com/pytorch/rl/pull/1908
* [BugFix] Fix Ray collector on Python > 3.8 by albertbou92 in https://github.com/pytorch/rl/pull/2015
* [BugFix] Fix RoboHiveEnv tests by sriramsk1999 in https://github.com/pytorch/rl/pull/2062
* [BugFix] Fix _reset data passing in parallel env by vmoens in https://github.com/pytorch/rl/pull/1880
* [BugFix] Fix a bug in SliceSampler, indexes outside sampler lengths were produced by vladisai in https://github.com/pytorch/rl/pull/1874
* [BugFix] Fix args/kwargs passing in advantages by vmoens in https://github.com/pytorch/rl/pull/2001
* [BugFix] Fix batch-size expansion in functionalization by vmoens in https://github.com/pytorch/rl/pull/1959
* [BugFix] Fix broken gym tests by vmoens in https://github.com/pytorch/rl/pull/1980
* [BugFix] Fix clip_fraction in PO losses by vmoens in https://github.com/pytorch/rl/pull/2021
* [BugFix] Fix colab in tutos by vmoens in https://github.com/pytorch/rl/pull/2113
* [BugFix] Fix env.shape regex matches by vmoens in https://github.com/pytorch/rl/pull/1940
* [BugFix] Fix examples by vmoens in https://github.com/pytorch/rl/pull/1945
* [BugFix] Fix exploration in losses by vmoens in https://github.com/pytorch/rl/pull/1898
* [BugFix] Fix flaky rb tests by vmoens in https://github.com/pytorch/rl/pull/1901
* [BugFix] Fix habitat by vmoens in https://github.com/pytorch/rl/pull/1941
* [BugFix] Fix jumanji by vmoens in https://github.com/pytorch/rl/pull/2064
* [BugFix] Fix load_state_dict and is_empty td bugfix impact by vmoens in https://github.com/pytorch/rl/pull/1869
* [BugFix] Fix mp_start_method for ParallelEnv with single_for_serial by vmoens in https://github.com/pytorch/rl/pull/2007
* [BugFix] Fix multiple context syntax in multiagent examples by matteobettini in https://github.com/pytorch/rl/pull/1943
* [BugFix] Fix offline CatFrames by vmoens in https://github.com/pytorch/rl/pull/1953
* [BugFix] Fix offline CatFrames for pixels by vmoens in https://github.com/pytorch/rl/pull/1964
* [BugFix] Fix prints of size error when no file is associated with memmap by vmoens in https://github.com/pytorch/rl/pull/2090
* [BugFix] Fix replay buffer extension with lists by vmoens in https://github.com/pytorch/rl/pull/1937
* [BugFix] Fix reward2go for nd tensors by vmoens in https://github.com/pytorch/rl/pull/2087
* [BugFix] Fix robohive by vmoens in https://github.com/pytorch/rl/pull/2080
* [BugFix] Fix sampling without replacement with ndim storages by vmoens in https://github.com/pytorch/rl/pull/1999
* [BugFix] Fix slice sampler compatibility with split_trajs and MultiStep by vmoens in https://github.com/pytorch/rl/pull/1961
* [BugFix] Fix slicesampler terminated/truncated signaling by vmoens in https://github.com/pytorch/rl/pull/2044
* [BugFix] Fix strict-length for spanning trajectories by vmoens in https://github.com/pytorch/rl/pull/1982
* [BugFix] Fix strict_length=True in SliceSampler by vmoens in https://github.com/pytorch/rl/pull/2037
* [BugFix] Fix unwanted lazy stacks by vmoens in https://github.com/pytorch/rl/pull/2102
* [BugFix] Fix update in serial / parallel env by vmoens in https://github.com/pytorch/rl/pull/1866
* [BugFix] Fix vmas stacks by vmoens in https://github.com/pytorch/rl/pull/2105
* [BugFix] Fixed import for importlib by DanilBaibak in https://github.com/pytorch/rl/pull/1914
* [BugFix] Make KL-controllers independent of the model by vmoens in https://github.com/pytorch/rl/pull/1903
* [BugFix] Make sure ParallelEnv does not overflow mem when policy requires grad by vmoens in https://github.com/pytorch/rl/pull/1909
* [BugFix] More robust _StepMDP and multi-purpose envs by vmoens in https://github.com/pytorch/rl/pull/2038
* [BugFix] No grad on collector reset by matteobettini in https://github.com/pytorch/rl/pull/1927
* [BugFix] Non exclusive terminated and truncated by vmoens in https://github.com/pytorch/rl/pull/1911
* [BugFix] Refactor reductions by vmoens in https://github.com/pytorch/rl/pull/1968
* [BugFix] Remove `split_trajectories`'s reference to `("next", "done")`. by initmaks in https://github.com/pytorch/rl/pull/2094
* [BugFix] Remove reset on last step of a rollout by matteobettini in https://github.com/pytorch/rl/pull/1936
* [BugFix] Robust sync for non_blocking=True by vmoens in https://github.com/pytorch/rl/pull/2034
* [BugFix] Set default value for `normalize_advantage` to `False`. by DobromirM in https://github.com/pytorch/rl/pull/2050
* [BugFix] Set strict=False in tensordict.select() calls for objective classes by albertbou92 in https://github.com/pytorch/rl/pull/2004
* [BugFix] SliceSampler device and index mesh by vmoens in https://github.com/pytorch/rl/pull/1996
* [BugFix] Solve recursion issue in losses hook by vmoens in https://github.com/pytorch/rl/pull/1897
* [BugFix] Update cql docstring example by BY571 in https://github.com/pytorch/rl/pull/1951
* [BugFix] Update iql docstring example by BY571 in https://github.com/pytorch/rl/pull/1950
* [BugFix] Use same signature for append_transform in all cases by vmoens in https://github.com/pytorch/rl/pull/2091
* [BugFix] Use setdefault in _cache_values by vmoens in https://github.com/pytorch/rl/pull/1910
* [BugFix] Use traj_terminated in SliceSampler by Cadene in https://github.com/pytorch/rl/pull/1884
* [BugFix] Vmap randomness for value estimator by BY571 in https://github.com/pytorch/rl/pull/1942
* [BugFix] better device consistency in EGreedy by vmoens in https://github.com/pytorch/rl/pull/1867
* [BugFix] check_env_specs seeding logic by vmoens in https://github.com/pytorch/rl/pull/1872
* [BugFix] fix formatting for VideoRecorder docstring by sriramsk1999 in https://github.com/pytorch/rl/pull/1985
* [BugFix] fix trunc normal device by vmoens in https://github.com/pytorch/rl/pull/1931
* [BugFix] missing annotations import by vmoens in https://github.com/pytorch/rl/pull/2074
* [BugFix] state typo in RNG control module by vmoens in https://github.com/pytorch/rl/pull/1878
* [BugFix] to_observation_norm now works with keys which are not strings by maxweissenbacher in https://github.com/pytorch/rl/pull/2045
* [BugFix] union -> intersection in _StepMDP check by vmoens in https://github.com/pytorch/rl/pull/2039
* [CI,Doc] Sanitize version by vmoens in https://github.com/pytorch/rl/pull/2120
* [CI] Doc on release tag by vmoens in https://github.com/pytorch/rl/pull/2116
* [CI] Fix CI issues by vmoens in https://github.com/pytorch/rl/pull/2084
* [CI] Fix Doc CI by matteobettini in https://github.com/pytorch/rl/pull/2106
* [CI] Fixes sympy error by fixing mpmath version by vmoens in https://github.com/pytorch/rl/pull/1988
* [CI] Install ffmpeg in Robohive tests by vmoens in https://github.com/pytorch/rl/pull/2063
* [CI] Install stable torch and tensordict for release tests by vmoens in https://github.com/pytorch/rl/pull/1978
* [CI] Remove all macos x86 jobs by vmoens in https://github.com/pytorch/rl/pull/2117
* [CI] Remove x86 OSX jobs by vmoens in https://github.com/pytorch/rl/pull/2112
* [CI] Schedule workflows for releases by vmoens in https://github.com/pytorch/rl/pull/2114
* [CI] Temporarily remove snapshot from CI by vmoens in https://github.com/pytorch/rl/pull/2000
* [CI] Unpin mpmath by vmoens in https://github.com/pytorch/rl/pull/1997
* [CI] Upgrade 3.8 to 3.10 GPU jobs by vmoens in https://github.com/pytorch/rl/pull/2013
* [Deprecation] Deprecate in prep for release by vmoens in https://github.com/pytorch/rl/pull/1820
* [Doc,Feature] Better doc for modules and list of kwargs when possible by vmoens in https://github.com/pytorch/rl/pull/1990
* [Doc] Fix tutos by vmoens in https://github.com/pytorch/rl/pull/1863
* [Doc] Getting started tutos by vmoens in https://github.com/pytorch/rl/pull/1886
* [Doc] Improve PrioritizedSampler doc and get rid of np dependency as much as possible by vmoens in https://github.com/pytorch/rl/pull/1881
* [Doc] Installation instructions in API ref by vmoens in https://github.com/pytorch/rl/pull/1871
* [Doc] Per-release doc by vmoens in https://github.com/pytorch/rl/pull/2108
* [Documentation] Correct MaskedEnv Example in ActionMask Transform Documentation by Jonathanace in https://github.com/pytorch/rl/pull/2060
* [Examples] Move examples to sota-implementations by vmoens in https://github.com/pytorch/rl/pull/2016
* [Minor] Add env.shape attribute by vmoens in https://github.com/pytorch/rl/pull/1938
* [Minor] Lint by vmoens in https://github.com/pytorch/rl/pull/2096
* [Minor] Move distributed examples to examples by vmoens in https://github.com/pytorch/rl/pull/2097
* [Minor] Remove duplicate if statement in storages by vmoens in https://github.com/pytorch/rl/pull/2066
* [Minor] Remove warnings in test_cost by vmoens in https://github.com/pytorch/rl/pull/1902
* [Minor] Support init lazy storages with add by vmoens in https://github.com/pytorch/rl/pull/2028
* [Minor] Use the main branch for the M1 build wheels by DanilBaibak in https://github.com/pytorch/rl/pull/1965
* [Performance] Faster DMC by vmoens in https://github.com/pytorch/rl/pull/2002
* [Quality] Capture errors in specs transforms by vmoens in https://github.com/pytorch/rl/pull/2092
* [Quality] Make sure deprec warnings are displayed by vmoens in https://github.com/pytorch/rl/pull/2088
* [Refactor,Feature] Refactor collector shapes and stack_result in sync collector by vmoens in https://github.com/pytorch/rl/pull/1994
* [Refactor] Clearer separation between single_task and share_individual_td by vmoens in https://github.com/pytorch/rl/pull/2026
* [Refactor] Faster and more generic multi-agent nets by vmoens in https://github.com/pytorch/rl/pull/1921
* [Refactor] Refactor split_trajectories by vmoens in https://github.com/pytorch/rl/pull/1955
* [Refactor] Remove remnant legacy functional calls by vmoens in https://github.com/pytorch/rl/pull/1973
* [Refactor] Use filter_empty=False in apply for params by vmoens in https://github.com/pytorch/rl/pull/1882
* [Refactor] Use filter_empty=True in apply by vmoens in https://github.com/pytorch/rl/pull/1879
* [Tutorial] PettingZoo Parallel competitive tutorial by matteobettini in https://github.com/pytorch/rl/pull/2047
* [Versioning] Deprecations for 0.4 by vmoens in https://github.com/pytorch/rl/pull/2109
* [Versioning] New torch version by vmoens in https://github.com/pytorch/rl/pull/2110
* [Versioning] v0.4.0 by vmoens in https://github.com/pytorch/rl/pull/1860

New Contributors
* vladisai made their first contribution in https://github.com/pytorch/rl/pull/1874
* Cadene made their first contribution in https://github.com/pytorch/rl/pull/1884
* sriramsk1999 made their first contribution in https://github.com/pytorch/rl/pull/1985
* DobromirM made their first contribution in https://github.com/pytorch/rl/pull/2050
* Jonathanace made their first contribution in https://github.com/pytorch/rl/pull/2060
* maxweissenbacher made their first contribution in https://github.com/pytorch/rl/pull/2045
* initmaks made their first contribution in https://github.com/pytorch/rl/pull/2094

A big thanks to our dear contributors as well as the entire user base for helping with this lib!

**Full Changelog**: https://github.com/pytorch/rl/compare/v0.3.0...v0.4.0

This release provides a bunch of bug fixes and speedups.

What's Changed

[BugFix] Fix broken gym tests (1980)
[BugFix,CI] Fix Windows CI (1983)
[Minor] Cleanup
[CI] Install stable torch and tensordict for release tests (1978)
[Refactor] Remove remnant legacy functional calls (1973)
[Minor] Use the main branch for the M1 build wheels (1965)
[BugFix] Fixed import for importlib (1914)
[BugFix] Fix offline CatFrames for pixels (1964)
[BugFix] Fix offline CatFrames (1953)
[BugFix] Fix batch-size expansion in functionalization (1959)
[BugFix] Update iql docstring example (1950)
[BugFix] Update cql docstring example (1951)
[BugFix] Fix examples (1945)
[BugFix] Remove reset on last step of a rollout (1936)
[BugFix] Vmap randomness for value estimator (1942)
[BugFix] Fix multiple context syntax in multiagent examples (1943)
[BugFix] Fix habitat (1941)
[BugFix] Fix env.shape regex matches (1940)
[Minor] Add env.shape attribute (1938)
[BugFix] Fix replay buffer extension with lists (1937)
[BugFix] No grad on collector reset (1927)
[BugFix] fix trunc normal device (1931)
[BugFix, Performance] Fewer imports at root (1930)
[BugFix] Fix OOB TruncatedNormal LP (1924)
[BugFix] Fix KLPENPPOLoss KL computation (1922)
[Doc] Fix onw typo (1917)
[BugFix] Make sure ParallelEnv does not overflow mem when policy requires grad (1909)
[BugFix] Non exclusive terminated and truncated (1911)
[BugFix] Use setdefault in _cache_values (1910)
[BugFix] Fix Ray collector example error (1908)
[BugFix] Make KL-controllers independent of the model (1903)
[Minor] Remove warnings in test_cost (1902)
[BugFix] Adaptable non-blocking for mps and non cuda device in batched-envs (1900)
[BugFix] Fix flaky rb tests (1901)
[BugFix] Fix exploration in losses (1898)
[BugFix] Solve recursion issue in losses hook (1897)
[Doc] Update getting-started-5.py (1894)
[Doc] Getting started tutos (1886)
[BugFix] Use traj_terminated in SliceSampler (1884)
[Doc] Improve PrioritizedSampler doc and get rid of np dependency as much as possible (1881)
[BugFix] Fix _reset data passing in parallel env (1880)
[BugFix] state typo in RNG control module (1878)
[BugFix] Fix a bug in SliceSampler, indexes outside sampler lengths were produced (1874)
[BugFix] check_env_specs seeding logic (1872)
[BugFix] Fix update in serial / parallel env (1866)
[Doc] Installation instructions in API ref (1871)
[BugFix] better device consistency in EGreedy (1867)
[BugFix] Fix load_state_dict and is_empty td bugfix impact (1869)
[Doc] Fix tutos (1863)

**Full Changelog**: https://github.com/pytorch/rl/compare/v0.3.0...v0.3.1

In this release, we focused on building a [**Data Hub for offline RL**](https://pytorch.org/rl/reference/data.html#datasets), providing a universal <lib>2gym conversion tool (1795) and improving the doc.

TorchRL Data Hub

TorchRL now offers many offline datasets in robotics and control or gaming, all under a single data format ([TED for TorchRL Episode Data Format](https://pytorch.org/rl/reference/data.html#torchrl-episode-data-format-ted)). All datasets are one step away of being downloaded: `dataset = <Name>ExperienceReplay(dataset_id, root="/path/to/storage", download=True)` is all you need to get started.
This means that you can now download OpenX 1751 or Roboset 1743 datasets and combine them in a single replay buffer 1768 or swap one another in no time and with no extra code.
We allow many new sampling techniques, like sampling slices of trajectories with or without repetition etc.
As always you can append your favourite transform to these transforms.

TorchRL2Gym universal converter

1795 introduces a new universal converter for simulation libraries to gym.
As RL practitioner, it's sometimes difficult to accommodate for the many different environment APIs that exist. TorchRL now provides a way of registering any env in gym(nasium). This allows users to build their dataset in torchrl and integrate them in their code base with no effort if they are already using gym as a backend. It also allows to transform DMControl or Brax envs (among others) to gym without the need for an extra library.

PPO and A2C compatibility with distributed models

Functional calls can now be turned off for PPO and A2C loss modules, allowing users to run RLHF training loops at scale! 1804

 TensorDict-free replay buffers

You can now use TorchRL's replay buffer with ANY tensor-based structure, whether it involves dict, tuples or lists. In principle, storing data **contiguously** on **disk** given any gym environment is as simple as

python
rb = ReplayBuffer(storage=LazyMemmapStorage(capacity))
obs_, reward, terminal, truncated, info = env.step(action)
rb.add((obs, obs_, reward, terminal, truncated, info, action))

sampling a tuple obs, reward, terminal, truncated, info
obs, obs_, reward, terminal, truncated, info = rb.sample()


This is independent of TensorDict and it supports many components of our replay buffers as well as transforms. Check the doc [here](https://pytorch.org/rl/reference/data.html#composable-replay-buffers).

 Multiprocessed replay buffers

TorchRL's replay buffers can now be shared across processes. Multiprocessed RBs can not only be read from but also extended on different workers. 1724

SOTA checks

We introduce a list of scripts to check that our training scripts work ok before each release: 1822

Throughput of Gym and DMControl

We removed loads of checks in GymLikeEnv if some basic conditions are met, which improves the throughput significantly for simple envs. 1803

 Algorithms

We introduce discrete CQL 1666 , discrete IQL 1793 and Impala 1506.

What's Changed: PR description
* [BugFix] Fix incorrect deprecation warning by mikemykhaylov in https://github.com/pytorch/rl/pull/1655
* [Bug] TensorDictMaxValueWriter raises error when no sample in a batch is accepted by albertbou92 in https://github.com/pytorch/rl/pull/1664
* [BugFix] Fix "done" instead of "terminated" mistakes by MarCnu in https://github.com/pytorch/rl/pull/1661
* [Feature] CatFrames constant padding by albertbou92 in https://github.com/pytorch/rl/pull/1663
* doc(README): remove typo by Deep145757 in https://github.com/pytorch/rl/pull/1665
* [Docs] Update README.md by vaibhav-009 in https://github.com/pytorch/rl/pull/1667
* [Minor] Update dreamer example tests by vmoens in https://github.com/pytorch/rl/pull/1668
* [Feature] Introduce grouping in VMAS by matteobettini in https://github.com/pytorch/rl/pull/1658
* [BugFix] assertion error message, envs/util.py by laszloKopits in https://github.com/pytorch/rl/pull/1669
* [Doc] Set `action_spec` instead of `input_spec` by FrankTianTT in https://github.com/pytorch/rl/pull/1657
* [BugFix] Fix submitit IP address/node name retrieval by vmoens in https://github.com/pytorch/rl/pull/1672
* [Doc] Document (and test) compound actor by vmoens in https://github.com/pytorch/rl/pull/1673
* [Doc] Update rollout_recurrent.png to account for terminal by vmoens in https://github.com/pytorch/rl/pull/1677
* [Doc] Add EGreedyWrapper back in the doc by vmoens in https://github.com/pytorch/rl/pull/1680
* [Doc] Fix `TanhDelta` docstring by matteobettini in https://github.com/pytorch/rl/pull/1683
* [Doc] Add discord badge on README by vmoens in https://github.com/pytorch/rl/pull/1686
* [CI] Downgrade RAY to fix CI by vmoens in https://github.com/pytorch/rl/pull/1687
* [BugFix] MaxValueWriter cuda compatibility by albertbou92 in https://github.com/pytorch/rl/pull/1689
* Upload docs for preview on HUD by DanilBaibak in https://github.com/pytorch/rl/pull/1682
* [Doc] Update pendulum and rnn tutos by vmoens in https://github.com/pytorch/rl/pull/1691
* [Algorithm] Discrete CQL by BY571 in https://github.com/pytorch/rl/pull/1666
* [BugFix] Minor fix in the logging of PPO and A2C examples by albertbou92 in https://github.com/pytorch/rl/pull/1693
* [CI] Enable retry mechanism by DanilBaibak in https://github.com/pytorch/rl/pull/1681
* [Refactor] Minor changes in prep of https://github.com/pytorch/tensordict/pull/541 by vmoens in https://github.com/pytorch/rl/pull/1696
* [BugFix] fix dreamer actor by FrankTianTT in https://github.com/pytorch/rl/pull/1697
* [Refactor] Deprecate direct usage of memmap tensors by vmoens in https://github.com/pytorch/rl/pull/1684
* Revert "[Refactor] Deprecate direct usage of memmap tensors" by vmoens in https://github.com/pytorch/rl/pull/1698
* [Refactor] Deprecate direct usage of memmap tensors by vmoens in https://github.com/pytorch/rl/pull/1699
* [Doc] Fix discord link by vmoens in https://github.com/pytorch/rl/pull/1701
* [BugFix] make sure the params of exploration-wrapper is float by FrankTianTT in https://github.com/pytorch/rl/pull/1700
* [Fix] EndOfLifeTransform fix in end of life detection by albertbou92 in https://github.com/pytorch/rl/pull/1705
* [CI] Fix benchmark on gpu by vmoens in https://github.com/pytorch/rl/pull/1706
* [Algorithm] IMPALA and VTrace module by albertbou92 in https://github.com/pytorch/rl/pull/1506
* [Doc] Fix discord link by vmoens in https://github.com/pytorch/rl/pull/1712
* [Refactor] Refactor functional calls in losses by vmoens in https://github.com/pytorch/rl/pull/1707
* [CI] Fix CI by vmoens in https://github.com/pytorch/rl/pull/1711
* [BugFix] Make casting to 'meta' device uniform across cost modules by vmoens in https://github.com/pytorch/rl/pull/1715
* [BugFix] Change ppo mujoco example to match paper results by albertbou92 in https://github.com/pytorch/rl/pull/1714
* [Minor] Hide params in ddpg actor-critic by vmoens in https://github.com/pytorch/rl/pull/1716
* [BugFix] Fix hold_out_net by vmoens in https://github.com/pytorch/rl/pull/1719
* [BugFix] `RewardSum` key check by matteobettini in https://github.com/pytorch/rl/pull/1718
* [Feature] Allow usage of a different device on main and sub-envs in ParallelEnv and SerialEnv by vmoens in https://github.com/pytorch/rl/pull/1626
* [Refactor] Better weight update in collectors by vmoens in https://github.com/pytorch/rl/pull/1723
* [Feature] Shared replay buffers by vmoens in https://github.com/pytorch/rl/pull/1724
* [CI] FIx nightly builds on osx by vmoens in https://github.com/pytorch/rl/pull/1726
* [BugFix] _call_actor_net does not handle multiple inputs by albertbou92 in https://github.com/pytorch/rl/pull/1728
* [Feature] Python-based RNN Modules by albertbou92 in https://github.com/pytorch/rl/pull/1720
* [BugFix, Test] Fix flaky gym vecenvs tests by vmoens in https://github.com/pytorch/rl/pull/1727
* [BugFix] Fix non-full TensorStorage indexing by vmoens in https://github.com/pytorch/rl/pull/1730
* [Feature] Minari datasets by vmoens in https://github.com/pytorch/rl/pull/1721
* [Feature] All VMAS scenarios available by matteobettini in https://github.com/pytorch/rl/pull/1731
* [Feature] pickle-free RB checkpointing by vmoens in https://github.com/pytorch/rl/pull/1733
* [CI] Fix doc upload by vmoens in https://github.com/pytorch/rl/pull/1738
* [BugFix] Fix RNNs trajectory split in VMAP calls by vmoens in https://github.com/pytorch/rl/pull/1736
* [CI] Fix doc upload by vmoens in https://github.com/pytorch/rl/pull/1739
* [BugFix, Feature] Fix DDQN implementation by vmoens in https://github.com/pytorch/rl/pull/1737
* [Algorithm] Update DQN example by albertbou92 in https://github.com/pytorch/rl/pull/1512
* [BugFix] Use rsync in doc workflow by vmoens in https://github.com/pytorch/rl/pull/1741
* [BugFix] Fix compat with new memmap API by vmoens in https://github.com/pytorch/rl/pull/1744
* [Feature] Roboset datasets by vmoens in https://github.com/pytorch/rl/pull/1743
* [Algorithm] Simpler IQL example by BY571 in https://github.com/pytorch/rl/pull/998
* [Performance] Faster RNNs by vmoens in https://github.com/pytorch/rl/pull/1732
* [BugFix, Test] Fix torch.vmap call in RNN tests by vmoens in https://github.com/pytorch/rl/pull/1749
* [BugFix] Fix discrete SAC log-prob by vmoens in https://github.com/pytorch/rl/pull/1750
* [Minor] Remove dead code in RolloutFromModel by ianbarber in https://github.com/pytorch/rl/pull/1752
* [Minor] Fix runnability of RLHF example in examples/rlhf by ianbarber in https://github.com/pytorch/rl/pull/1753
* [Feature] SliceSampler by vmoens in https://github.com/pytorch/rl/pull/1748
* [CI] Fix windows CI by vmoens in https://github.com/pytorch/rl/pull/1746
* [CI] Fix CI for optional dependencies by vmoens in https://github.com/pytorch/rl/pull/1754
* [Feature] V-D4RL by vmoens in https://github.com/pytorch/rl/pull/1756
* [Benchmark] Fix RB benchmarks by vmoens in https://github.com/pytorch/rl/pull/1760
* [BugFix] Fix RLHF by vmoens in https://github.com/pytorch/rl/pull/1757
* [BugFix] Fix slice sampler by vmoens in https://github.com/pytorch/rl/pull/1762
* [Feature] BurnInTransform by albertbou92 in https://github.com/pytorch/rl/pull/1765
* [Bug] Minor change burnin transform by albertbou92 in https://github.com/pytorch/rl/pull/1770
* [BugFix] Fix sampling of last item in SliceSampler by vmoens in https://github.com/pytorch/rl/pull/1774
* [Feature] Open-X Embodiement datasets by vmoens in https://github.com/pytorch/rl/pull/1751
* [BugFix] Fix documentation of threads for batched envs. by skandermoalla in https://github.com/pytorch/rl/pull/1776
* [BugFix, CI] Fix OpenML datasets runs by vmoens in https://github.com/pytorch/rl/pull/1779
* [Versioning] Bump v0.3.0 and fix m1-wheels by vmoens in https://github.com/pytorch/rl/pull/1780
* [Feature] Composite replay buffers by vmoens in https://github.com/pytorch/rl/pull/1768
* [BugFix, Feature] Vmap randomness in losses by BY571 in https://github.com/pytorch/rl/pull/1740
* [Algorithm] Update discrete SAC example by BY571 in https://github.com/pytorch/rl/pull/1745
* [Docs] Pointers to BenchMARL by matteobettini in https://github.com/pytorch/rl/pull/1710
* [Feature] Immutable writer for datasets by vmoens in https://github.com/pytorch/rl/pull/1781
* [Feature] Remove and check for prints in codebase using flake8-print by vmoens in https://github.com/pytorch/rl/pull/1758
* [BUG] Missing import for some Samplers in Data module by albertbou92 in https://github.com/pytorch/rl/pull/1784
* [BugFix] Ensure that infos and samples have the same batch-size in SamplerEnsemble by vmoens in https://github.com/pytorch/rl/pull/1786
* [BugFix] Writers extend() method should always return indices in data.device by albertbou92 in https://github.com/pytorch/rl/pull/1785
* [Doc] Revamp envs doc by vmoens in https://github.com/pytorch/rl/pull/1787
* [BugFix] Less flaky gym vecenv test by vmoens in https://github.com/pytorch/rl/pull/1790
* [CI] Regroup tests by vmoens in https://github.com/pytorch/rl/pull/1791
* [CI] Remove stable GPU tests from CI by vmoens in https://github.com/pytorch/rl/pull/1792
* Update README.md to fix CI banner by vmoens in https://github.com/pytorch/rl/pull/1794
* [Feature] `SamplerWithoutReplacement` state dictionary by matteobettini in https://github.com/pytorch/rl/pull/1788
* [BugFix] Higher time threshold for PEnv by vmoens in https://github.com/pytorch/rl/pull/1799
* [Feature] SignTransform by albertbou92 in https://github.com/pytorch/rl/pull/1798
* [Feature] Extend MaxValueWriter with reduce parameter for the rank_key by albertbou92 in https://github.com/pytorch/rl/pull/1796
* [BugFix] Fixes bug in MaxValueWriter tests by albertbou92 in https://github.com/pytorch/rl/pull/1801
* [Performance] faster gym-like class by vmoens in https://github.com/pytorch/rl/pull/1803
* [Feature] GenDGRL by vmoens in https://github.com/pytorch/rl/pull/1773
* [Performance] Minor improvements to step_and_maybe_reset in batched envs by vmoens in https://github.com/pytorch/rl/pull/1807
* [Algorithm] Discrete IQL by BY571 in https://github.com/pytorch/rl/pull/1793
* [Doc] More depth in VMAS docs by matteobettini in https://github.com/pytorch/rl/pull/1802
* [BugFix] Remove select() in favor of empty() by vmoens in https://github.com/pytorch/rl/pull/1811
* Bump jinja2 from 3.1.2 to 3.1.3 in /docs by dependabot in https://github.com/pytorch/rl/pull/1812
* [BugFix] Make `TransformedEnv` mirror `allow_done_after_reset` property of base env by matteobettini in https://github.com/pytorch/rl/pull/1810
* [Doc] Update StepCounter doc by skandermoalla in https://github.com/pytorch/rl/pull/1813
* [Feature] Improve info_dict reader by vmoens in https://github.com/pytorch/rl/pull/1809
* [CI, Minor] Regroup Gen-DGRL CI with other libs by vmoens in https://github.com/pytorch/rl/pull/1814
* [Versioning] Housekeeping in setup.py by vmoens in https://github.com/pytorch/rl/pull/1816
* [Feature] TorchRL2Gym conversion by vmoens in https://github.com/pytorch/rl/pull/1795
* [BugFix, CI] Fix snapshop imports in stable CI by vmoens in https://github.com/pytorch/rl/pull/1821
* [Feature] More flexibility in loading PettingZoo by matteobettini in https://github.com/pytorch/rl/pull/1817
* [Docs] Fix doc of ToTensorImage transforms.py by skandermoalla in https://github.com/pytorch/rl/pull/1824
* [BugFix] Fix device of container generated values in transforms by vmoens in https://github.com/pytorch/rl/pull/1827
* [Feature] Atari DQN dataset by vmoens in https://github.com/pytorch/rl/pull/1815
* [Feature] Non-functional objectives (PPO, A2C, Reinforce) by vmoens in https://github.com/pytorch/rl/pull/1804
* [Refactor] change default CKPT_BACKEND to torch by vmoens in https://github.com/pytorch/rl/pull/1830
* pyproject.toml: remove unknown properties by GaetanLepage in https://github.com/pytorch/rl/pull/1828
* [Doc, Feature] Doc improvements for video recording and CSV video formats by vmoens in https://github.com/pytorch/rl/pull/1829
* [Feature] PyTrees in replay buffers by vmoens in https://github.com/pytorch/rl/pull/1831
* [BugFix] Fix sequential step counts by vmoens in https://github.com/pytorch/rl/pull/1838
* [Doc] TED format by vmoens in https://github.com/pytorch/rl/pull/1836
* [Doc] References to TED by vmoens in https://github.com/pytorch/rl/pull/1839
* [BugFix] Temporarily set lazy legacy to True by vmoens in https://github.com/pytorch/rl/pull/1840
* [BugFix] Fix gym info scalar infos by vmoens in https://github.com/pytorch/rl/pull/1842
* [Refactor] LAZY_LEGACY_OP=False by vmoens in https://github.com/pytorch/rl/pull/1832
* [Feature] `serial_for_single` arg in batched envs by vmoens in https://github.com/pytorch/rl/pull/1846
* [BugFix] Fix VD4RL by vmoens in https://github.com/pytorch/rl/pull/1834
* [Doc] Make tutos runnable without colab by vmoens in https://github.com/pytorch/rl/pull/1826
* [Feature] Fine control over devices in collectors by vmoens in https://github.com/pytorch/rl/pull/1835
* [Feature, BugFix] Better thread control in penv and collectors by vmoens in https://github.com/pytorch/rl/pull/1848
* [CI] Update macos image by vmoens in https://github.com/pytorch/rl/pull/1849
* [BugFix] thread setting bug by vmoens in https://github.com/pytorch/rl/pull/1852
* Remove unused completed_keys property from StepCounter. by skandermoalla in https://github.com/pytorch/rl/pull/1854
* [Feature] Submitit run script by albertbou92 in https://github.com/pytorch/rl/pull/1822
* [BugFix] Fix flaky gym penv test by vmoens in https://github.com/pytorch/rl/pull/1853
* [CI] Fix macos build by vmoens in https://github.com/pytorch/rl/pull/1856

New Contributors
* mikemykhaylov made their first contribution in https://github.com/pytorch/rl/pull/1655
* MarCnu made their first contribution in https://github.com/pytorch/rl/pull/1661
* Deep145757 made their first contribution in https://github.com/pytorch/rl/pull/1665
* vaibhav-009 made their first contribution in https://github.com/pytorch/rl/pull/1667
* laszloKopits made their first contribution in https://github.com/pytorch/rl/pull/1669
* ianbarber made their first contribution in https://github.com/pytorch/rl/pull/1752
* dependabot made their first contribution in https://github.com/pytorch/rl/pull/1812
* GaetanLepage made their first contribution in https://github.com/pytorch/rl/pull/1828

**Full Changelog**: https://github.com/pytorch/rl/compare/v0.2.1...v0.3.0

New Contributors
* duburcqa made their first contribution in https://github.com/pytorch/rl/pull/1615

**Full Changelog**: https://github.com/pytorch/rl/compare/v0.2.0...v0.2.1