Torchrl

This release provides many new features and bug fixes.

TorchRL now publishes Apple Silicon compatible wheels.
We drop coverage of python 3.7 in favour of 3.11.

New and updated algorithms

Most algorithms have been cleaned and designed to reach (at least) SOTA results.


Compatibility with MARL settings has been drastically improved, and we provide a good amount of MARL examples within the library:


A prototype RLHF training script is also proposed (1597)

A whole new category of offline RL algorithms have been integrated: Decision transformers.

* [Algorithm] Update offpolicy examples by BY571 in https://github.com/pytorch/rl/pull/1206
* [Algorithm] Online Decision transformer by BY571 in https://github.com/pytorch/rl/pull/1149
* [Algorithm] QMixer loss and multiagent models by matteobettini in https://github.com/pytorch/rl/pull/1378
* [Algorithm] RLHF end-to-end, clean by vmoens in https://github.com/pytorch/rl/pull/1597
* [Algorithm] Update A2C examples by albertbou92 in https://github.com/pytorch/rl/pull/1521
* [Algorithm] Update DDPG Example by BY571 in https://github.com/pytorch/rl/pull/1525
* [Algorithm] Update DT by BY571 in https://github.com/pytorch/rl/pull/1560
* [Algorithm] Update PPO examples by albertbou92 in https://github.com/pytorch/rl/pull/1495
* [Algorithm] Update SAC Example by BY571 in https://github.com/pytorch/rl/pull/1524
* [Algorithm] Update TD3 Example by BY571 in https://github.com/pytorch/rl/pull/1523

New features

One of the major new features of the library is the introduction of the terminated / truncated / done distinction at __no cost__ within the library. All third-party and primary environments are now compatible with this, as well as losses and data collection primitives (collector etc). This feature is also compatible with complex data structures, such as those found in MARL training pipelines.

All losses are now compatible with tensordict-free inputs, for a more generic deployment.

New transforms

Atari games can now benefit from a EndOfLifeTransform that allows to use the end-of-life as a done state in the loss (1605)

We provide a KL transform to add a KL factor to the reward in RLHF settings.

Action masking is made possible through the ActionMask transform (1421)

VC1 is also integrated for better image embedding.

* [Feature] Allow sequential transforms to work offline by vmoens in https://github.com/pytorch/rl/pull/1136
* [Feature] ClipTransform + rename `min/maximum` -> `low/high` by vmoens in https://github.com/pytorch/rl/pull/1500
* [Feature] End-of-life transform by vmoens in https://github.com/pytorch/rl/pull/1605
* [Feature] KL Transform for RLHF by vmoens in https://github.com/pytorch/rl/pull/1196
* [Features] Conv3dNet and PermuteTransform by xmaples in https://github.com/pytorch/rl/pull/1398
* [Feature, Refactor] Scale in ToTensorImage based on the dtype and new from_int parameter by hyerra in https://github.com/pytorch/rl/pull/1208
* [Feature] CatFrames used as inverse by BY571 in https://github.com/pytorch/rl/pull/1321
* [Feature] Masking actions by vmoens in https://github.com/pytorch/rl/pull/1421
* [Feature] VC1 integration by vmoens in https://github.com/pytorch/rl/pull/1211

New models

We provide GRU alongside LSTM for POMDP training.

MARL model coverage is now richer of a MultiAgentMLP and MultiAgentCNN! Other improvments for MARL include coverage for nested keys in most places of the library (losses, data collection, environments...)/

* [Feature] Support for GRU by vmoens in https://github.com/pytorch/rl/pull/1586
* [Feature] TanhModule by vmoens in https://github.com/pytorch/rl/pull/1213
* [Features] Conv3dNet and PermuteTransform by xmaples in https://github.com/pytorch/rl/pull/1398
* [Feature] CNN version of MultiAgentMLP by MarkHaoxiang in https://github.com/pytorch/rl/pull/1479

Other features (misc)

* [Feature] RLHF Rollouts (reopened) by vmoens in https://github.com/pytorch/rl/pull/1329
* [Feature] Add CQL by BY571 in https://github.com/pytorch/rl/pull/1239
* [Feature] Allow multiple (nested) action, reward, done keys in `env`,`vec_env` and `collectors` by matteobettini in https://github.com/pytorch/rl/pull/1462
* [Feature] Auto-DoubleToFloat by vmoens in https://github.com/pytorch/rl/pull/1442
* [Feature] CompositeSpec.lock by vmoens in https://github.com/pytorch/rl/pull/1143
* [Feature] Device transform by vmoens in https://github.com/pytorch/rl/pull/1472
* [Feature] Dispatch DiscreteSAC loss module by Blonck in https://github.com/pytorch/rl/pull/1248
* [Feature] Dispatch PPO loss module by Blonck in https://github.com/pytorch/rl/pull/1249
* [Feature] Dispatch REDQ loss module by Blonck in https://github.com/pytorch/rl/pull/1251
* [Feature] Dispatch SAC loss module by Blonck in https://github.com/pytorch/rl/pull/1244
* [Feature] Dispatch TD3 loss module by Blonck in https://github.com/pytorch/rl/pull/1254
* [Feature] Dispatch for DDPG loss module by Blonck in https://github.com/pytorch/rl/pull/1215
* [Feature] Dispatch for SAC loss module by Blonck in https://github.com/pytorch/rl/pull/1223
* [Feature] Dispatch reinforce loss module by Blonck in https://github.com/pytorch/rl/pull/1252
* [Feature] Distpatch IQL loss module by Blonck in https://github.com/pytorch/rl/pull/1230
* [Feature] Fix DType casting lazy init by vmoens in https://github.com/pytorch/rl/pull/1589
* [Feature] Heterogeneous Environments compatibility by matteobettini in https://github.com/pytorch/rl/pull/1411
* [Feature] Log hparams from python dict by matteobettini in https://github.com/pytorch/rl/pull/1517
* [Feature] MARL exploration e-greedy compatibility by matteobettini in https://github.com/pytorch/rl/pull/1277
* [Feature] Make advantages compatible with Terminated, Truncated, Done by vmoens in https://github.com/pytorch/rl/pull/1581
* [Feature] Make losses inherit from TDMBase by vmoens in https://github.com/pytorch/rl/pull/1246
* [Feature] Making action masks compatible with q value modules and e-greedy by matteobettini in https://github.com/pytorch/rl/pull/1499
* [Feature] Nested keys in `OrnsteinUhlenbeckProcess` by matteobettini in https://github.com/pytorch/rl/pull/1305
* [Feature] Optional mapping of "state" in gym specs by matteobettini in https://github.com/pytorch/rl/pull/1431
* [Feature] Parallel environments lazy heterogenous data compatibility by matteobettini in https://github.com/pytorch/rl/pull/1436
* [Feature] Pettingzoo: add multiagent dimension to single agent groups by matteobettini in https://github.com/pytorch/rl/pull/1550
* [Feature] RLHF Reward Model (reopened) by vmoens in https://github.com/pytorch/rl/pull/1328
* [Feature] RLHF dataloading by vmoens in https://github.com/pytorch/rl/pull/1309
* [Feature] RLHF networks by apbard in https://github.com/pytorch/rl/pull/1319
* [Feature] Refactor categorical dists: Masked one-hot and pass-through gradients by vmoens in https://github.com/pytorch/rl/pull/1488
* [Feature] ReplayBuffer.empty by vmoens in https://github.com/pytorch/rl/pull/1238
* [Feature] Separate losses by MateuszGuzek in https://github.com/pytorch/rl/pull/1240
* [Feature] Single call to value network in advantages [bis] by vmoens in https://github.com/pytorch/rl/pull/1263
* [Feature] Single call to value network in advantages by vmoens in https://github.com/pytorch/rl/pull/1256
* [Feature] TensorStorage by vmoens in https://github.com/pytorch/rl/pull/1310
* [Feature] Threaded collection and parallel envs by vmoens in https://github.com/pytorch/rl/pull/1559
* [Feature] Unbind specs by vmoens in https://github.com/pytorch/rl/pull/1555
* [Feature] VMAS obs dict by matteobettini in https://github.com/pytorch/rl/pull/1419
* [Feature] VMAS: choose between categorical or one-hot actions by matteobettini in https://github.com/pytorch/rl/pull/1484
* [Feature] dispatch for DQNLoss by vmoens in https://github.com/pytorch/rl/pull/1194
* [Feature] log histograms by vmoens in https://github.com/pytorch/rl/pull/1306
* [Feature] make csv logger `exist_ok` on logging folder by matteobettini in https://github.com/pytorch/rl/pull/1561
* [Feature] shifted for all adv by vmoens in https://github.com/pytorch/rl/pull/1276

New environments and third-party improvements

We now cover SMAC-v2, PettingZoo, IsaacGymEnvs (prototype) and RoboHive. The D4RL dataset can now be used without the eponym library, which permit training with more recent or older versions of gym.

* [Environment, Docs] SMACv2 and docs on action masking by matteobettini in https://github.com/pytorch/rl/pull/1466
* [Environment] Petting zoo by matteobettini in https://github.com/pytorch/rl/pull/1471
* [Feature] D4rl direct download by MateuszGuzek in https://github.com/pytorch/rl/pull/1430
* [Feature] Gym 'vectorized' envs compatibility by vmoens in https://github.com/pytorch/rl/pull/1519
* [Feature] Gym compatibility: Terminal and truncated by vmoens in https://github.com/pytorch/rl/pull/1539
* [Feature] IsaacGymEnvs integration by vmoens in https://github.com/pytorch/rl/pull/1443
* [Feature] RoboHive integration by vmoens in https://github.com/pytorch/rl/pull/1119

Performance improvements

We provide several speed improvements, in particular for data collection.


* [Performance] Accelerate GAE by Blonck in https://github.com/pytorch/rl/pull/1142
* [Performance] Accelerate TD lambda return estimate by Blonck in https://github.com/pytorch/rl/pull/1158
* [Performance] Accelerate `_split_and_pad_sequence` by Blonck in https://github.com/pytorch/rl/pull/1147
* [Performance] Faster GAE by vmoens in https://github.com/pytorch/rl/pull/1153
* [Performance] Faster losses by vmoens in https://github.com/pytorch/rl/pull/1272
* [Performance] Improve performance and streamline the generating of the gammalambda tensor by Blonck in https://github.com/pytorch/rl/pull/1171
* [Performance] Miscellaneous efficiency improvements by vmoens in https://github.com/pytorch/rl/pull/1513
* [Performance] Reduce key accessing in transforms by matteobettini in https://github.com/pytorch/rl/pull/1590
* [Performance] Some efficiency improvements by vmoens in https://github.com/pytorch/rl/pull/1250
* [Performance] Vmas vectorized reset by matteobettini in https://github.com/pytorch/rl/pull/1146

Bug fixes

* [BugFIx] Fix entropy signature in truncated normal by vmoens in https://github.com/pytorch/rl/pull/1536
* [BugFix,CI] Fix virtualenv not found by vmoens in https://github.com/pytorch/rl/pull/1280
* [BugFix] Add `torch.no_grad()` for rendering in multiagent PPO tutorial by matteobettini in https://github.com/pytorch/rl/pull/1511
* [BugFix] Batched envs compatibility with custom keys by matteobettini in https://github.com/pytorch/rl/pull/1348
* [BugFix] C++17 by vmoens in https://github.com/pytorch/rl/pull/1169
* [BugFix] Check env specs for nested envs by matteobettini in https://github.com/pytorch/rl/pull/1332
* [BugFix] CompositeSpec.unsqueeze by btx0424 in https://github.com/pytorch/rl/pull/1464
* [BugFix] DDPG select also critic input for actor loss by matteobettini in https://github.com/pytorch/rl/pull/1563
* [BugFix] DQN loss dispatch respect configured tensordict keys by Blonck in https://github.com/pytorch/rl/pull/1285
* [BugFix] Discrete SAC rewrite by matteobettini in https://github.com/pytorch/rl/pull/1461
* [BugFix] Empty-spec tolerance by vmoens in https://github.com/pytorch/rl/pull/1501
* [BugFix] Fix Brax reset by vmoens in https://github.com/pytorch/rl/pull/1195
* [BugFix] Fix CatFrames by vmoens in https://github.com/pytorch/rl/pull/1336
* [BugFix] Fix ClipTransform device by vmoens in https://github.com/pytorch/rl/pull/1508
* [BugFix] Fix Cython for D4RL by vmoens in https://github.com/pytorch/rl/pull/1429
* [BugFix] Fix DDPG by vmoens in https://github.com/pytorch/rl/pull/1183
* [BugFix] Fix DDPG squeezing by matteobettini in https://github.com/pytorch/rl/pull/1487
* [BugFix] Fix Dreamer test error by vmoens in https://github.com/pytorch/rl/pull/1558
* [BugFix] Fix Gym Categorical/One-hot issues by vmoens in https://github.com/pytorch/rl/pull/1482
* [BugFix] Fix KL import errors by vmoens in https://github.com/pytorch/rl/pull/1207
* [BugFix] Fix KLTransform execution with LSTM by vmoens in https://github.com/pytorch/rl/pull/1426
* [BugFix] Fix KeyError in inverse transform replay buffer by BY571 in https://github.com/pytorch/rl/pull/1165
* [BugFix] Fix LSTM - VecEnv compatibility by vmoens in https://github.com/pytorch/rl/pull/1427
* [BugFix] Fix LSTM use with padded/masked segments by smorad in https://github.com/pytorch/rl/pull/1399
* [BugFix] Fix NoopResetEnv behavior when trials exceeded. by skandermoalla in https://github.com/pytorch/rl/pull/1477
* [BugFix] Fix QValueModule multi_one_hot by smorad in https://github.com/pytorch/rl/pull/1439
* [BugFix] Fix RLHF tests - transformers v4.34 by vmoens in https://github.com/pytorch/rl/pull/1601
* [BugFix] Fix RewardSum spec transform to mimic reward spec by matteobettini in https://github.com/pytorch/rl/pull/1478
* [BugFix] Fix SAC alpha optim by vmoens in https://github.com/pytorch/rl/pull/1192
* [BugFix] Fix SAC by vmoens in https://github.com/pytorch/rl/pull/1189
* [BugFix] Fix SAC by vmoens in https://github.com/pytorch/rl/pull/1190
* [BugFix] Fix SACv2 by vmoens in https://github.com/pytorch/rl/pull/1191
* [BugFix] Fix SMAC-v2 by vmoens in https://github.com/pytorch/rl/pull/1538
* [BugFix] Fix TD3 and compat with https://github.com/pytorch-labs/tensordict/pull/482 by vmoens in https://github.com/pytorch/rl/pull/1375
* [BugFix] Fix TD3 inplace updates by vmoens in https://github.com/pytorch/rl/pull/1219
* [BugFix] Fix TD3 target net by vmoens in https://github.com/pytorch/rl/pull/1186
* [BugFix] Fix `LazyStackedCompositeSpec` and introducing `consolidate_spec` by matteobettini in https://github.com/pytorch/rl/pull/1392
* [BugFix] Fix `step_mdp()` by matteobettini in https://github.com/pytorch/rl/pull/1334
* [BugFix] Fix action mask test by vmoens in https://github.com/pytorch/rl/pull/1492
* [BugFix] Fix brax by vmoens in https://github.com/pytorch/rl/pull/1346
* [BugFix] Fix bug in ppo example config by degensean in https://github.com/pytorch/rl/pull/1396
* [BugFix] Fix envpool by vmoens in https://github.com/pytorch/rl/pull/1530
* [BugFix] Fix error message of .set_keys() in advantage modules by Blonck in https://github.com/pytorch/rl/pull/1218
* [BugFix] Fix examples by vmoens in https://github.com/pytorch/rl/pull/1173
* [BugFix] Fix locked params modif by vmoens in https://github.com/pytorch/rl/pull/1307
* [BugFix] Fix max length by vmoens in https://github.com/pytorch/rl/pull/1233
* [BugFix] Fix missing ("next", "observation") key in dispatch of losses by Blonck in https://github.com/pytorch/rl/pull/1235
* [BugFix] Fix nested CompositeSpec creation by vmoens in https://github.com/pytorch/rl/pull/1261
* [BugFix] Fix nightly tensordict dependency by skandermoalla in https://github.com/pytorch/rl/pull/1302
* [BugFix] Fix ppo example by vmoens in https://github.com/pytorch/rl/pull/1225
* [BugFix] Fix ppo training NaN occurences by vmoens in https://github.com/pytorch/rl/pull/1403
* [BugFix] Fix reward sum within parallel envs by vmoens in https://github.com/pytorch/rl/pull/1454
* [BugFix] Fix run_type_checks by vmoens in https://github.com/pytorch/rl/pull/1570
* [BugFix] Fix safe tanh for older torch versions by vmoens in https://github.com/pytorch/rl/pull/1220
* [BugFix] Fix serialization of parallel envs by vmoens in https://github.com/pytorch/rl/pull/1197
* [BugFix] Fix split_trajs by vmoens in https://github.com/pytorch/rl/pull/1444
* [BugFix] Fix tanh/atanh vmap compatibility by vmoens in https://github.com/pytorch/rl/pull/1217
* [BugFix] Fix the bug of `RoundRobinWriter.extend(data)` by xmaples in https://github.com/pytorch/rl/pull/1295
* [BugFix] Fix tutorials by vmoens in https://github.com/pytorch/rl/pull/1382
* [BugFix] Fix typo in CatFrames Transform error message. by skandermoalla in https://github.com/pytorch/rl/pull/1491
* [BugFix] Fix vmap in VmapModule (torch 1.13 compat) by vmoens in https://github.com/pytorch/rl/pull/1350
* [BugFix] Improve collector buffer initialisation when policy spec is unavailable by matteobettini in https://github.com/pytorch/rl/pull/1547
* [BugFix] Instantiate 2 losses with different keys by matteobettini in https://github.com/pytorch/rl/pull/1553
* [BugFix] KL module integration by vmoens in https://github.com/pytorch/rl/pull/1212
* [BugFix] Key selection in batched envs by vmoens in https://github.com/pytorch/rl/pull/1253
* [BugFix] Load collector frames and iter by matteobettini in https://github.com/pytorch/rl/pull/1557
* [BugFix] Make VecNorm Transform pickable by albertbou92 in https://github.com/pytorch/rl/pull/1596
* [BugFix] Minor fixes PPO / A2C examples by albertbou92 in https://github.com/pytorch/rl/pull/1591
* [BugFix] Multiagent "auto" entropy fix in SAC by matteobettini in https://github.com/pytorch/rl/pull/1494
* [BugFix] Nested envs compatibility by matteobettini in https://github.com/pytorch/rl/pull/1347
* [BugFix] Nested key in replay buffer by matteobettini in https://github.com/pytorch/rl/pull/1485
* [BugFix] Nested keys in transforms by matteobettini in https://github.com/pytorch/rl/pull/1355
* [BugFix] Nested keys to probabilistic modules by matteobettini in https://github.com/pytorch/rl/pull/1363
* [BugFix] Parametric `rand_action()` in `BaseEnv` by matteobettini in https://github.com/pytorch/rl/pull/1267
* [BugFix] Parametric collectors by matteobettini in https://github.com/pytorch/rl/pull/1303
* [BugFix] Patch SAC to allow state_dict manipulation before exec by vmoens in https://github.com/pytorch/rl/pull/1607
* [BugFix] PettingZoo seeding by matteobettini in https://github.com/pytorch/rl/pull/1554
* [BugFix] Pickable buffer by albertbou92 in https://github.com/pytorch/rl/pull/1410
* [BugFix] QValue modules and nested action by matteobettini in https://github.com/pytorch/rl/pull/1351
* [BugFix] Reward sum custom key by matteobettini in https://github.com/pytorch/rl/pull/1413
* [BugFix] SafeModule not safely handling specs by matteobettini in https://github.com/pytorch/rl/pull/1352
* [BugFix] Small patches to SMAC by matteobettini in https://github.com/pytorch/rl/pull/1533
* [BugFix] Sparse info in SMACv2 by matteobettini in https://github.com/pytorch/rl/pull/1546
* [BugFix] ToTensorImage unsqueeze would not update the observation spec by hyerra in https://github.com/pytorch/rl/pull/1161
* [BugFix] Torch 1.13 compat by vmoens in https://github.com/pytorch/rl/pull/1294
* [BugFix] Unbreak tensordict import by vmoens in https://github.com/pytorch/rl/pull/1231
* [BugFix] Vectorized priority update in replay buffers by matteobettini in https://github.com/pytorch/rl/pull/1598
* [BugFix] _transpose_time with single dim by vmoens in https://github.com/pytorch/rl/pull/1155
* [BugFix] `RewardSum` transform for multiple reward keys by matteobettini in https://github.com/pytorch/rl/pull/1544
* [BugFix] `step_mdp` nested keys by matteobettini in https://github.com/pytorch/rl/pull/1339
* [BugFix] include buffers in policy_weights by vmoens in https://github.com/pytorch/rl/pull/1185
* [BugFix] load_state_dict in param updates for collectors by vmoens in https://github.com/pytorch/rl/pull/1145
* [BugFix] make value estimator with value_key from the PPOLoss init arg by xmaples in https://github.com/pytorch/rl/pull/1144
* [BugFix] unlock in tensordictmodules tests by vmoens in https://github.com/pytorch/rl/pull/1417
* [BugFix] valid_size not saved as attribute by tcbegley in https://github.com/pytorch/rl/pull/1337

Miscellaneous

* Envpool Tests to Nova by osalpekar in https://github.com/pytorch/rl/pull/1283
* Fix CI by matteobettini in https://github.com/pytorch/rl/pull/1368
* Fix MacOS Mujoco Failure by osalpekar in https://github.com/pytorch/rl/pull/1450
* Linux GPU Brax Unittests by osalpekar in https://github.com/pytorch/rl/pull/1133
* Linux Gym Unittests to GHA by osalpekar in https://github.com/pytorch/rl/pull/1139
* Linux Olddeps tests to Nova by osalpekar in https://github.com/pytorch/rl/pull/1289
* Move to More Efficient Windows Runner by osalpekar in https://github.com/pytorch/rl/pull/1476
* OptDeps Tests to Nova by osalpekar in https://github.com/pytorch/rl/pull/1290
* Remove Distributed CCI job by osalpekar in https://github.com/pytorch/rl/pull/1374
* Remove Envpool from CCI by osalpekar in https://github.com/pytorch/rl/pull/1390
* Remove old CircleCI Lint by osalpekar in https://github.com/pytorch/rl/pull/1134
* Removing Migrated and Unused CCI jobs by osalpekar in https://github.com/pytorch/rl/pull/1288
* Revert "[Feature] Single call to value network in advantages" by vmoens in https://github.com/pytorch/rl/pull/1262
* Revert "[Refactor,Performance] Faster collectors" by vmoens in https://github.com/pytorch/rl/pull/1330
* Sklearn test to Nova by osalpekar in https://github.com/pytorch/rl/pull/1291
* Windows Unittests on GHA by osalpekar in https://github.com/pytorch/rl/pull/1086
* [Benchmark,CI] Benchmarks in PR (pre) by vmoens in https://github.com/pytorch/rl/pull/1342
* [Benchmark,CI] Benchmarks in PR by vmoens in https://github.com/pytorch/rl/pull/1341
* [Benchmark] Benchmark Gym vs TorchRL by vmoens in https://github.com/pytorch/rl/pull/1602
* [Benchmark] Benchmark losses by vmoens in https://github.com/pytorch/rl/pull/1287
* [Benchmark] Benchmark number GPU vectorised environments in VMAS (TorchRL vs RLlib) by matteobettini in https://github.com/pytorch/rl/pull/1446
* [Benchmark] Improve benchmark precision + step_mdp + fix GPU by vmoens in https://github.com/pytorch/rl/pull/1340
* [CI] Add macOS M1 binaries Wheels by DanilBaibak in https://github.com/pytorch/rl/pull/1504
* [CI] Add ninja for MacOS builts by vmoens in https://github.com/pytorch/rl/pull/1564
* [CI] Concurrency on gha by vmoens in https://github.com/pytorch/rl/pull/1152
* [CI] Deprecate Windows GPU CCI by osalpekar in https://github.com/pytorch/rl/pull/1387
* [CI] Doc CI fix by matteobettini in https://github.com/pytorch/rl/pull/1384
* [CI] Fix CI PettingZoo by matteobettini in https://github.com/pytorch/rl/pull/1528
* [CI] Fix CI by vmoens in https://github.com/pytorch/rl/pull/1529
* [CI] Fix GHA gpu tests by vmoens in https://github.com/pytorch/rl/pull/1356
* [CI] Fix Jax version in Jumanji by vmoens in https://github.com/pytorch/rl/pull/1242
* [CI] Fix Mujoco version by vmoens in https://github.com/pytorch/rl/pull/1475
* [CI] Fix RoboHive CI by vmoens in https://github.com/pytorch/rl/pull/1541
* [CI] Fix brax and habitat by vmoens in https://github.com/pytorch/rl/pull/1353
* [CI] Fix examples CI by matteobettini in https://github.com/pytorch/rl/pull/1489
* [CI] Fix failing jobs by vmoens in https://github.com/pytorch/rl/pull/1318
* [CI] Fix failing jobs by vmoens in https://github.com/pytorch/rl/pull/1335
* [CI] Fix habitat CI by vmoens in https://github.com/pytorch/rl/pull/1537
* [CI] Fix jumanji by vmoens in https://github.com/pytorch/rl/pull/1566
* [CI] Fix nightly build dependency on tensordict by vmoens in https://github.com/pytorch/rl/pull/1300
* [CI] Fix opt deps machine and docker by vmoens in https://github.com/pytorch/rl/pull/1362
* [CI] Fix tuto deps by matteobettini in https://github.com/pytorch/rl/pull/1416
* [CI] Fix wheels by vmoens in https://github.com/pytorch/rl/pull/1301
* [CI] Less old deps by vmoens in https://github.com/pytorch/rl/pull/1255
* [CI] Less warnings in CI (costs) by vmoens in https://github.com/pytorch/rl/pull/1349
* [CI] Merge Distributed and Linux GPU job by osalpekar in https://github.com/pytorch/rl/pull/1182
* [CI] Migrate examples by vmoens in https://github.com/pytorch/rl/pull/1364
* [CI] Move linux stable to GHA by vmoens in https://github.com/pytorch/rl/pull/1503
* [CI] Reduce CI time by vmoens in https://github.com/pytorch/rl/pull/1226
* [CI] Remove CCI Config by osalpekar in https://github.com/pytorch/rl/pull/1456
* [CI] Remove examples from CCI by vmoens in https://github.com/pytorch/rl/pull/1367
* [CI] Update cuda version by vmoens in https://github.com/pytorch/rl/pull/1380
* [CI] Windows GPU Tests by osalpekar in https://github.com/pytorch/rl/pull/1386
* [Doc] Add link to paper in readme by giadefa in https://github.com/pytorch/rl/pull/1298
* [Doc] Add paper refs in doc and KB by vmoens in https://github.com/pytorch/rl/pull/1241
* [Doc] CITATION.cff by vmoens in https://github.com/pytorch/rl/pull/1229
* [Doc] Do not clean gh-pages by vmoens in https://github.com/pytorch/rl/pull/1150
* [Doc] Fix GPU benchmark by vmoens in https://github.com/pytorch/rl/pull/1151
* [Doc] Fix advantage examples by vmoens in https://github.com/pytorch/rl/pull/1600
* [Doc] Fix default value of `tanh_loc` in the documentation of `TruncatedNormal`. by skandermoalla in https://github.com/pytorch/rl/pull/1205
* [Doc] Fix doctest examples by degensean in https://github.com/pytorch/rl/pull/1393
* [Doc] Fix exploration modules docstrings by vmoens in https://github.com/pytorch/rl/pull/1326
* [Doc] Fix tanh_loc in docstrings by vmoens in https://github.com/pytorch/rl/pull/1203
* [Doc] TorchRL Logo by vmoens in https://github.com/pytorch/rl/pull/1234
* [Doc] Update citation by vmoens in https://github.com/pytorch/rl/pull/1228
* [Doc] Update coding_ppo.py by kushaangupta in https://github.com/pytorch/rl/pull/1483
* [Doc] correct typos in pendulum tutorial by kushaangupta in https://github.com/pytorch/rl/pull/1502
* [Doc] fixed typos in ppo tutorial by MatteoGaetzner in https://github.com/pytorch/rl/pull/1314
* [Docs] Fix multi-agent tutorial by matteobettini in https://github.com/pytorch/rl/pull/1599
* [Docs] Multi-agent environments by matteobettini in https://github.com/pytorch/rl/pull/1383
* [Example] Multiagent examples: MAPPO-IPPO-MADDPG-IDDPG-IQL-QMIX-VDN by matteobettini in https://github.com/pytorch/rl/pull/1027
* [Fix] Remove loss device by matteobettini in https://github.com/pytorch/rl/pull/1395
* [Lint] Add TorchFix linter by kit1980 in https://github.com/pytorch/rl/pull/1580
* [Minor] Capture error in CatFrame edit by vmoens in https://github.com/pytorch/rl/pull/1498
* [Minor] Fix prints by vmoens in https://github.com/pytorch/rl/pull/1257
* [Minor] Fix typo by vmoens in https://github.com/pytorch/rl/pull/1193
* [Minor] Missing commit from 1488 by vmoens in https://github.com/pytorch/rl/pull/1490
* [Minor] Missing lint by vmoens in https://github.com/pytorch/rl/pull/1556
* [Minor] More efficient SAC v1 by vmoens in https://github.com/pytorch/rl/pull/1507
* [Minor] Remove ya gymnasium deprecation warning in vectorized envs by vmoens in https://github.com/pytorch/rl/pull/1573
* [Minor] small fixes by vmoens in https://github.com/pytorch/rl/pull/1237
* [Nova] Jumanji Tests to GHA by osalpekar in https://github.com/pytorch/rl/pull/1282
* [Nova] Remove windows Unittests from CCI by osalpekar in https://github.com/pytorch/rl/pull/1159
* [Nova] Removing CircleCI Gym Unittests by osalpekar in https://github.com/pytorch/rl/pull/1179
* [Nova] Vmas Tests to GHA by osalpekar in https://github.com/pytorch/rl/pull/1284
* [Quality] Filter out warnings in subprocs by vmoens in https://github.com/pytorch/rl/pull/1552
* [Refacto] Migration due to tensordict 473 and 474 by vmoens in https://github.com/pytorch/rl/pull/1354
* [Refactor,Performance] Faster collectors (bis) by vmoens in https://github.com/pytorch/rl/pull/1331
* [Refactor,Performance] Faster collectors by vmoens in https://github.com/pytorch/rl/pull/1327
* [Refactor] Better GymLikeEnv by vmoens in https://github.com/pytorch/rl/pull/1168
* [Refactor] Better batch-size handling by RBs by vmoens in https://github.com/pytorch/rl/pull/1311
* [Refactor] Better updaters by vmoens in https://github.com/pytorch/rl/pull/1184
* [Refactor] Change objectives parameter/buffer/target logic by vmoens in https://github.com/pytorch/rl/pull/1424
* [Refactor] Edit ppo params by vmoens in https://github.com/pytorch/rl/pull/1322
* [Refactor] Expose all wrappers in torchrl.envs by vmoens in https://github.com/pytorch/rl/pull/1532
* [Refactor] Faster envs (2) by vmoens in https://github.com/pytorch/rl/pull/1457
* [Refactor] Fix imports by vmoens in https://github.com/pytorch/rl/pull/1551
* [Refactor] Follow-up on tensordict PR 473 by vmoens in https://github.com/pytorch/rl/pull/1361
* [Refactor] More unravel fixes by vmoens in https://github.com/pytorch/rl/pull/1357
* [Refactor] Nested reward and done specs by vmoens in https://github.com/pytorch/rl/pull/1115
* [Refactor] Refactor DDPG loss in standalone methods by vmoens in https://github.com/pytorch/rl/pull/1603
* [Refactor] Refactor _reset in ParallelEnv by vmoens in https://github.com/pytorch/rl/pull/1172
* [Refactor] Refactor losses for generalization by vmoens in https://github.com/pytorch/rl/pull/1286
* [Refactor] Remove pkg_resources import by vmoens in https://github.com/pytorch/rl/pull/1379
* [Refactor] Remove private calls to _set by vmoens in https://github.com/pytorch/rl/pull/1370
* [Refactor] Shape ops in LSTM based on tensor shape, not tensordict by vmoens in https://github.com/pytorch/rl/pull/1170
* [Refactor] Use _set_tuple for faster set by vmoens in https://github.com/pytorch/rl/pull/1372
* [Refactor] Use `wait` instead of `is_set` to get results in ParallelEnv by vmoens in https://github.com/pytorch/rl/pull/1562
* [Refactor] Use masking in collectors by vmoens in https://github.com/pytorch/rl/pull/1412
* [Refactor] Vmas nested by matteobettini in https://github.com/pytorch/rl/pull/1366
* [Refactor] the usage of tensordict keys in loss modules by Blonck in https://github.com/pytorch/rl/pull/1175
* [Setup] Update setup.py python versions by vmoens in https://github.com/pytorch/rl/pull/1496
* [Test,BugFix] Fix Jax backend tests by vmoens in https://github.com/pytorch/rl/pull/1162
* [Test,CI,Feature] Total time per test by vmoens in https://github.com/pytorch/rl/pull/1232
* [Test] Remove import of test class by matteobettini in https://github.com/pytorch/rl/pull/1549
* [Test] Skip tests in python 3.11 by vmoens in https://github.com/pytorch/rl/pull/1535
* [Test] Skip threading tests in OSX by vmoens in https://github.com/pytorch/rl/pull/1571
* [Test] Test split trajs by vmoens in https://github.com/pytorch/rl/pull/1445
* [Test] Test state_dict and loss modules by vmoens in https://github.com/pytorch/rl/pull/1527
* [Tests] Collector compatibility for heterogeneous environments by matteobettini in https://github.com/pytorch/rl/pull/1414
* [Tests] DDPG extra critic input tests by matteobettini in https://github.com/pytorch/rl/pull/1568
* [Tutorial] Multiagent PPO tutorial by matteobettini in https://github.com/pytorch/rl/pull/1385
* [Versioning] Python 3.11 by vmoens in https://github.com/pytorch/rl/pull/1433
* [Versioning] Use python 3.8 for GPU tests by vmoens in https://github.com/pytorch/rl/pull/1577
* [Versioning] Write version all cases in setup.py by vmoens in https://github.com/pytorch/rl/pull/1579
* d4rl Test to Nova by osalpekar in https://github.com/pytorch/rl/pull/1293
* python 3.11 in README by vmoens in https://github.com/pytorch/rl/pull/1434

New Contributors
* Blonck made their first contribution in https://github.com/pytorch/rl/pull/1142
* hyerra made their first contribution in https://github.com/pytorch/rl/pull/1161
* skandermoalla made their first contribution in https://github.com/pytorch/rl/pull/1205
* giadefa made their first contribution in https://github.com/pytorch/rl/pull/1298
* MatteoGaetzner made their first contribution in https://github.com/pytorch/rl/pull/1314
* MateuszGuzek made their first contribution in https://github.com/pytorch/rl/pull/1240
* degensean made their first contribution in https://github.com/pytorch/rl/pull/1393
* smorad made their first contribution in https://github.com/pytorch/rl/pull/1399
* kushaangupta made their first contribution in https://github.com/pytorch/rl/pull/1483
* kit1980 made their first contribution in https://github.com/pytorch/rl/pull/1580
* MarkHaoxiang made their first contribution in https://github.com/pytorch/rl/pull/1479
* DanilBaibak made their first contribution in https://github.com/pytorch/rl/pull/1504

A great THANKS to our contributors, in particular (but not in any particular order) skandermoalla, matteobettini, BY571 and albertbou92 for their tremendous dedication.

**Full Changelog**: https://github.com/pytorch/rl/compare/v0.1.1...v0.2.0

What's Changed
* [Feature] Stacking specs by vmoens in https://github.com/pytorch/rl/pull/892
* [Feature] Multicollector interruptor by albertbou92 in https://github.com/pytorch/rl/pull/963
* [BugFix] VMAS api fix by matteobettini in https://github.com/pytorch/rl/pull/978
* [CI] Fix D4RL tests in CI by vmoens in https://github.com/pytorch/rl/pull/976
* [CI] Fix CI by vmoens in https://github.com/pytorch/rl/pull/982
* [Refactor] Binary spec inherits from discrete spec by matteobettini in https://github.com/pytorch/rl/pull/984
* [Feature] `_DataCollector` -> `DataCollectorBase` by vmoens in https://github.com/pytorch/rl/pull/985
* [Feature] Discrete SAC by BY571 in https://github.com/pytorch/rl/pull/882
* [Refactor, Doc] Refactor refs to SafeModule to TensorDictModule unless necessary by vmoens in https://github.com/pytorch/rl/pull/986
* [BugFix] Quickfix by vmoens in https://github.com/pytorch/rl/pull/991
* [Feature] Add Dropout to MLP module by BY571 in https://github.com/pytorch/rl/pull/988
* [Feature] Warn when collectors collect more frames than requested by matteobettini in https://github.com/pytorch/rl/pull/989
* [BugFix] make "_reset", "step_count", and other done_based keys follow done_spec by matteobettini in https://github.com/pytorch/rl/pull/981
* [Feature] Bandit datasets by vmoens in https://github.com/pytorch/rl/pull/912
* [BugFix] Fix sampling in PPO tutorial by vmoens in https://github.com/pytorch/rl/pull/996
* [Refactor] Refactor losses (value function, doc, input batch size) by vmoens in https://github.com/pytorch/rl/pull/987
* [BugFix,Feature,Doc] Fix replay buffers sampling info, docstrings and iteration by vmoens in https://github.com/pytorch/rl/pull/1003
* [Feature] Replace ValueError by warning in collectors when total_frames is not an exact multiple of frames_per_batch by albertbou92 in https://github.com/pytorch/rl/pull/999
* [BugFix] Only call replay buffer transforms when there are by vmoens in https://github.com/pytorch/rl/pull/1008
* [BugFix] Patch tests in 1008 by vmoens in https://github.com/pytorch/rl/pull/1009
* [Feature] Multidim value functions by vmoens in https://github.com/pytorch/rl/pull/1007
* [BugFix] Fix exploration (OU and Gaussian) by vmoens in https://github.com/pytorch/rl/pull/1006
* [CI] Fix python version in habitat by vmoens in https://github.com/pytorch/rl/pull/1010
* Advantages pass `time_dim`and docfix by matteobettini in https://github.com/pytorch/rl/pull/1014
* [Refactor] Faster transformed distributions by vmoens in https://github.com/pytorch/rl/pull/1017
* [WIP, CI] Upgrade cuda channel by vmoens in https://github.com/pytorch/rl/pull/1019
* [BugFix] Fix collector reset with truncation by vmoens in https://github.com/pytorch/rl/pull/1021
* [Refactor] Improve collector performance by matteobettini in https://github.com/pytorch/rl/pull/1020
* [BugFix] Fix params and buffer casting for policies by vmoens in https://github.com/pytorch/rl/pull/1022
* [Feature] PPO allow entropy logging when entropy_coeff is 0 by matteobettini in https://github.com/pytorch/rl/pull/1025
* [Feature] Distributed data collector (ray) by albertbou92 in https://github.com/pytorch/rl/pull/930
* [Refactor] Minor changes in tensordict construction by vmoens in https://github.com/pytorch/rl/pull/1029
* [CI] Fix Brax 0.9.0 by vmoens in https://github.com/pytorch/rl/pull/1011
* [Feature] Multiagent API in vmas by matteobettini in https://github.com/pytorch/rl/pull/983
* [Feature] Benchmarking worflow by vmoens in https://github.com/pytorch/rl/pull/1028
* [Benchmark] Fix adv benchmark by vmoens in https://github.com/pytorch/rl/pull/1030
* [Doc] Refactor DDPG and DQN tutos to narrow the scope by vmoens in https://github.com/pytorch/rl/pull/979
* Revert "[Doc] Refactor DDPG and DQN tutos to narrow the scope" by vmoens in https://github.com/pytorch/rl/pull/1032
* [BugFix] Advantage normalisation in ClipPPOLoss is done after computing gain1 by albertbou92 in https://github.com/pytorch/rl/pull/1033
* [BugFix] Codecov SHA error by vmoens in https://github.com/pytorch/rl/pull/1035
* [Doc] DDPG and DQN refactoring -- Doc cleaning by vmoens in https://github.com/pytorch/rl/pull/1036
* [BugFix,CI] Fix macos codecov install by vmoens in https://github.com/pytorch/rl/pull/1039
* [BugFix] kwargs update in distributed collectors by vmoens in https://github.com/pytorch/rl/pull/1040
* [Feature] `make_composite_from_td` by vmoens in https://github.com/pytorch/rl/pull/1042
* [Refactor] Import envpool locally to avoid importing gym at root level by vmoens in https://github.com/pytorch/rl/pull/1041
* [Minor] Fix a typo by FrankTianTT in https://github.com/pytorch/rl/pull/1046
* [BugFix] Fix param tying in loss modules by vmoens in https://github.com/pytorch/rl/pull/1037
* [Refactor] less ad-hoc disable_env_checker check by vmoens in https://github.com/pytorch/rl/pull/1047
* [Refactor] Improve distributed collectors by vmoens in https://github.com/pytorch/rl/pull/1044
* [Doc] Document tensordict modules by vmoens in https://github.com/pytorch/rl/pull/1053
* [Doc] Minor changes to contributing.md by vmoens in https://github.com/pytorch/rl/pull/1054
* [Doc] A bit more doc on modules by vmoens in https://github.com/pytorch/rl/pull/1056
* [Refactor] Import enum and interaction_type utils by Goldspear in https://github.com/pytorch/rl/pull/1055
* [Feature] Deduplicate calls to common layers in PPO by vmoens in https://github.com/pytorch/rl/pull/1057
* [BugFix] CompositeSpec nested key deletion by btx0424 in https://github.com/pytorch/rl/pull/1059
* [Feature] Add MaskedCategorical distribution by xiaomengy in https://github.com/pytorch/rl/pull/1012
* [Refactor] resetting envs in collectors always passes the _reset entry by vmoens in https://github.com/pytorch/rl/pull/1061
* [Refactor] Better integration of QValue tools by vmoens in https://github.com/pytorch/rl/pull/1063
* MUJOCO_INSTALLATION.md: Fix typo by traversaro in https://github.com/pytorch/rl/pull/1064
* [Refactor] Removes "reward" from root tensordicts by vmoens in https://github.com/pytorch/rl/pull/1065
* [Test] Fix tests for older pytorch versions by vmoens in https://github.com/pytorch/rl/pull/1066
* [Feature] Reward2go Transform by BY571 in https://github.com/pytorch/rl/pull/1038
* [CI] Reduce tests by vmoens in https://github.com/pytorch/rl/pull/1071
* [Feature] Skip existing for advantage modules by vmoens in https://github.com/pytorch/rl/pull/1070
* [BugFix] Fix parallel env data passing on cuda by vmoens in https://github.com/pytorch/rl/pull/1024
* [Refactor] Deprecate interaction_mode by vmoens in https://github.com/pytorch/rl/pull/1067
* [Doc] Update KB: cannot find -lGL by vmoens in https://github.com/pytorch/rl/pull/1073
* [Doc] fix figures display issues in documentation of actors.py by DamienAllonsius in https://github.com/pytorch/rl/pull/1074
* [Example] PPO simplified example by albertbou92 in https://github.com/pytorch/rl/pull/1004
* [Feature] Update td in step (not overwrite) by vmoens in https://github.com/pytorch/rl/pull/1075
* [CI] Remove migrated CircleCI macOS jobs by seemethere in https://github.com/pytorch/rl/pull/1069
* [Feature] Target Return Transform by BY571 in https://github.com/pytorch/rl/pull/1045
* [Test] Fix tensorboard tests with ImageIO 2.26 by vmoens in https://github.com/pytorch/rl/pull/1083
* [Feature] LSTMModule by vmoens in https://github.com/pytorch/rl/pull/1084
* [BugFix] Change default of skip_existing to None by tcbegley in https://github.com/pytorch/rl/pull/1082
* [Example] A2C simplified example by albertbou92 in https://github.com/pytorch/rl/pull/1076
* [BugFix] Fix output_spec transform calls by vmoens in https://github.com/pytorch/rl/pull/1091
* [Feature] Indexing Discrete and OneHot specs by remidomingues in https://github.com/pytorch/rl/pull/1081
* [Refactor] Refactor DQN by vmoens in https://github.com/pytorch/rl/pull/1085
* [Feature] Auto-init updaters and raise a warning if not present by vmoens in https://github.com/pytorch/rl/pull/1092
* [BugFix] Remove false warnings in losses by vmoens in https://github.com/pytorch/rl/pull/1096
* [CI, BugFix] Fix CI warnings and errors by vmoens in https://github.com/pytorch/rl/pull/1100
* [Refactor] Update vmap imports to torch by vmoens in https://github.com/pytorch/rl/pull/1102
* [Refactor] Make advantages non-differentiable by default (except in losses) by vmoens in https://github.com/pytorch/rl/pull/1104
* [Feature] Indexing specs by remidomingues in https://github.com/pytorch/rl/pull/1105
* [BugFix] Fix EnvPoool by vmoens in https://github.com/pytorch/rl/pull/1106
* [Feature,Doc] QValue refactoring and QNet + RNN tuto by vmoens in https://github.com/pytorch/rl/pull/1060
* [BugFix] Fix Gym imports by vmoens in https://github.com/pytorch/rl/pull/1023
* [CI] pytest should not skip tests for dependencies by rohitnig in https://github.com/pytorch/rl/pull/1048
* [BugFix, Doc] Fix tutos by vmoens in https://github.com/pytorch/rl/pull/1107
* [CI] Fix tutos (2) by vmoens in https://github.com/pytorch/rl/pull/1109
* [Doc] Fix doc rendering by vmoens in https://github.com/pytorch/rl/pull/1112
* Added the entry for skip-tests in the environment.yml by rohitnig in https://github.com/pytorch/rl/pull/1113
* [CI] Upgrade ubuntu version in GHA by vmoens in https://github.com/pytorch/rl/pull/1116
* Fix in windows unit test by mischab in https://github.com/pytorch/rl/pull/1099
* Revert "Fix in windows unit test" by mischab in https://github.com/pytorch/rl/pull/1117
* [Nova] Lint job on GHA by osalpekar in https://github.com/pytorch/rl/pull/1114
* [Nova] Remove CircleCI Wheels Builds by osalpekar in https://github.com/pytorch/rl/pull/1121
* [BugFix] Set exploration mode to MODE in all losses by default by vmoens in https://github.com/pytorch/rl/pull/1123
* [BugFix] Instruct the value key to PPOLoss by vmoens in https://github.com/pytorch/rl/pull/1124
* [Feature] CatFrames for offline data by vmoens in https://github.com/pytorch/rl/pull/1122
* [CI] Fix windows CI by vmoens in https://github.com/pytorch/rl/pull/1128
* [Refactor] Buffers tensorclass compat and tutorial by vmoens in https://github.com/pytorch/rl/pull/1101
* [Feature] Marking the time dimension by vmoens in https://github.com/pytorch/rl/pull/1095
* [Doc] Add tuto and time dim info in docs by vmoens in https://github.com/pytorch/rl/pull/1130
* [Doc] Fix locked samples from RBs and ccl of tuto by vmoens in https://github.com/pytorch/rl/pull/1132
* [BugFix] Fix unlock in RB by vmoens in https://github.com/pytorch/rl/pull/1135
* [BugFix] extract the info dict from a list by xmaples in https://github.com/pytorch/rl/pull/1131
* [Feature] Added support for vector-based rewards from environments in MO-Gymnasium by dennismalmgren in https://github.com/pytorch/rl/pull/992
* [Versioning] v0.1.1 by vmoens in https://github.com/pytorch/rl/pull/1137

New Contributors
* FrankTianTT made their first contribution in https://github.com/pytorch/rl/pull/1046
* Goldspear made their first contribution in https://github.com/pytorch/rl/pull/1055
* btx0424 made their first contribution in https://github.com/pytorch/rl/pull/1059
* traversaro made their first contribution in https://github.com/pytorch/rl/pull/1064
* DamienAllonsius made their first contribution in https://github.com/pytorch/rl/pull/1074
* seemethere made their first contribution in https://github.com/pytorch/rl/pull/1069
* remidomingues made their first contribution in https://github.com/pytorch/rl/pull/1081
* rohitnig made their first contribution in https://github.com/pytorch/rl/pull/1048
* mischab made their first contribution in https://github.com/pytorch/rl/pull/1099
* osalpekar made their first contribution in https://github.com/pytorch/rl/pull/1114
* xmaples made their first contribution in https://github.com/pytorch/rl/pull/1131
* dennismalmgren made their first contribution in https://github.com/pytorch/rl/pull/992

**Full Changelog**: https://github.com/pytorch/rl/compare/v0.1.0...v0.1.1

First official beta release of the library!

What's Changed
* QuickFix Versioning by fedebotu in https://github.com/pytorch/rl/pull/958
* Version 0.0.5 by vmoens in https://github.com/pytorch/rl/pull/957
* [Minor] Warning when loading memmap storage on uninitialized td by vmoens in https://github.com/pytorch/rl/pull/961
* [Refactor] Defaults split_trajs to False by vmoens in https://github.com/pytorch/rl/pull/947
* [Feature] InitTracker transform by vmoens in https://github.com/pytorch/rl/pull/962
* [Feature] RenameTransform by vmoens in https://github.com/pytorch/rl/pull/964
* [Feature] Implicit Q-Learning (IQL) by BY571 in https://github.com/pytorch/rl/pull/933
* [Refactor] Refactor data collectors constructors by vmoens in https://github.com/pytorch/rl/pull/970
* [Feature, Refactor] Iterable replay buffers by vmoens in https://github.com/pytorch/rl/pull/968
* [Doc] README rewrite by vmoens in https://github.com/pytorch/rl/pull/971
* [Refactor] A less verbose torchrl by vmoens in https://github.com/pytorch/rl/pull/973
* [Feature] `torch.distributed` collectors by vmoens in https://github.com/pytorch/rl/pull/934
* [Feature] Offline datasets: D4RL by vmoens in https://github.com/pytorch/rl/pull/928


**Full Changelog**: https://github.com/pytorch/rl/compare/v0.0.5...v0.1.0

We change the env.step API, see https://github.com/pytorch/rl/pull/941 for more info.

What's Changed
* [BugFix] Fix dreamer training loop by vmoens in https://github.com/pytorch/rl/pull/915
* [Doc] PPO Tutorial by vmoens in https://github.com/pytorch/rl/pull/913
* [Doc] Create your pendulum tutorial by vmoens in https://github.com/pytorch/rl/pull/911
* [BugFix] Deploy doc by vmoens in https://github.com/pytorch/rl/pull/920
* [BugFix] Nvidia not found fix by vmoens in https://github.com/pytorch/rl/pull/922
* [Feature] Rework `to_one_hot` and `to_categorical` to take a tensor as parameter by riiswa in https://github.com/pytorch/rl/pull/816
* [Doc] Tutorial revamp by vmoens in https://github.com/pytorch/rl/pull/926
* [BugFix] Fix EnvPool spec shapes by vmoens in https://github.com/pytorch/rl/pull/932
* [BugFix] Fix CompositeSpec.to_numpy method by riiswa in https://github.com/pytorch/rl/pull/931
* [CI] Do not run nightly workflows on forked repos by XuehaiPan in https://github.com/pytorch/rl/pull/936
* [Refactor] set_default -> setdefault by tcbegley in https://github.com/pytorch/rl/pull/935
* [BugFix] Step and maybe reset by vmoens in https://github.com/pytorch/rl/pull/938
* [Doc] Minor doc improvements by vmoens in https://github.com/pytorch/rl/pull/907
* [Doc] Add debug doc by acohen13 in https://github.com/pytorch/rl/pull/940
* [BugFix] Propagate args to TransformedEnv's `state_dict` by fedebotu in https://github.com/pytorch/rl/pull/944
* [BugFix] Vmas expanded specs by matteobettini in https://github.com/pytorch/rl/pull/942
* [Quality] RB constuctors cleanup by vmoens in https://github.com/pytorch/rl/pull/945
* [Doc] Refactor KB by vmoens in https://github.com/pytorch/rl/pull/946
* [BugFix] Upgrade vision's functional import by vmoens in https://github.com/pytorch/rl/pull/948
* [BugFix] Deprecate tensordict.set check skips in transforms by vmoens in https://github.com/pytorch/rl/pull/951
* [BugFix] Upgrade tensordict deps by vmoens in https://github.com/pytorch/rl/pull/953
* [CI] Fix windows CI by vmoens in https://github.com/pytorch/rl/pull/954
* [Refactor] Refactor composite spec keys to match tensordict by vmoens in https://github.com/pytorch/rl/pull/956
* [Refactor] Refactor the step to include reward and done in the 'next' tensordict by vmoens in https://github.com/pytorch/rl/pull/941

New Contributors
* XuehaiPan made their first contribution in https://github.com/pytorch/rl/pull/936
* acohen13 made their first contribution in https://github.com/pytorch/rl/pull/940
* fedebotu made their first contribution in https://github.com/pytorch/rl/pull/944

**Full Changelog**: https://github.com/pytorch/rl/compare/v0.0.4...v0.0.5

What's Changed
* [CI, Doc] Update functorch source installation command by zou3519 in https://github.com/pytorch/rl/pull/446
* [BugFix] TransformedEnv attributes inheritance by vmoens in https://github.com/pytorch/rl/pull/467
* [Feature] Cleanup mocking envs init and new by vmoens in https://github.com/pytorch/rl/pull/469
* [Tests] Adding tensordict `__repr__` tests by sladebot in https://github.com/pytorch/rl/pull/435
* [Logging]: implement MLFlow logging integration by rayanht in https://github.com/pytorch/rl/pull/432
* [BugFix] MLFlow import fix by vmoens in https://github.com/pytorch/rl/pull/473
* [BugFix] Fixed pip install by brandonsj in https://github.com/pytorch/rl/pull/475
* [Features]: Changed `_inplace_update ` cls parameter passing in `__new__` by nicolas-dufour in https://github.com/pytorch/rl/pull/464
* [Feature]: ModelBased Envs by nicolas-dufour in https://github.com/pytorch/rl/pull/333
* [Feature] make ReplayBufferTrainer compatible with storing trajectories by vmoens in https://github.com/pytorch/rl/pull/476
* [Tutorial] DQN tutorial by vmoens in https://github.com/pytorch/rl/pull/474
* [Feature] reader hooks for GymLike by vmoens in https://github.com/pytorch/rl/pull/478
* [BugFix] TensorSpec.zero(None) failure fix by vmoens in https://github.com/pytorch/rl/pull/483
* [Feature]: Support for planners and CEM by nicolas-dufour in https://github.com/pytorch/rl/pull/384
* [Feature] Replaced `device_safe()` with `device` by ordinskiy in https://github.com/pytorch/rl/pull/485
* [Feature]: `TensorDictPrimer` transform by nicolas-dufour in https://github.com/pytorch/rl/pull/456
* [Feature]: `erase()` method for `torchrl.timeit` by nicolas-dufour in https://github.com/pytorch/rl/pull/480
* [Feature] Added support for single collector in sync_async_collector by nicolas-dufour in https://github.com/pytorch/rl/pull/482
* [BugFix] removing unwanted device_safe() by vmoens in https://github.com/pytorch/rl/pull/486
* [Refactoring] Refactored get_stats_random_rollout by nicolas-dufour in https://github.com/pytorch/rl/pull/481
* [Feature] VIP Integration by JasonMa2016 in https://github.com/pytorch/rl/pull/487
* [Refactoring] Minor tweaks to recorder and logger by nicolas-dufour in https://github.com/pytorch/rl/pull/489
* [Feature]: Deactivate typechecks in envs by nicolas-dufour in https://github.com/pytorch/rl/pull/490
* [BugFix] Vectorized td_lambda with gamma tensor does not match the serial version by vmoens in https://github.com/pytorch/rl/pull/400
* [BugFix] Fix TensorDictPrimer init by vmoens in https://github.com/pytorch/rl/pull/491
* [Feature] Optional auto-reset when done for collectors and batched envs by vmoens in https://github.com/pytorch/rl/pull/492
* [BugFix] Defaulting `passing_devices` to `None` by himjohntang in https://github.com/pytorch/rl/pull/477
* Revert "[BugFix] Defaulting `passing_devices` to `None`" by vmoens in https://github.com/pytorch/rl/pull/494
* [BugFix] Multi-agent fixes by vmoens in https://github.com/pytorch/rl/pull/488
* [BugFix] Defaulting `passing_devices` to `None` by vmoens in https://github.com/pytorch/rl/pull/495
* [Feature] Lazy initialization of CatTensors by vmoens in https://github.com/pytorch/rl/pull/497
* [Cleanup] Removing cuda 10.2 references by vmoens in https://github.com/pytorch/rl/pull/498
* [BugFix] Migration to pytorch org by vmoens in https://github.com/pytorch/rl/pull/499
* [Refactoring] Import at root to enable vmap monkey-patching by vmoens in https://github.com/pytorch/rl/pull/500
* [BugFix] python version for linting checks by vmoens in https://github.com/pytorch/rl/pull/502
* [Feature] Replay Buffers refactor by bamaxw in https://github.com/pytorch/rl/pull/330
* [Feature] Rename `step_tensordict` in `step_mdp` by romainjln in https://github.com/pytorch/rl/pull/512
* [Lint] re-instantiate F821 by vmoens in https://github.com/pytorch/rl/pull/516
* [BugFix] run_type_checks for TransformedEnvs by vmoens in https://github.com/pytorch/rl/pull/513
* [BugFix] making first_dim and last_dim negative in FlattenObservation when a parent is set by vmoens in https://github.com/pytorch/rl/pull/511
* [Feature] Add info dict key-spec pairs to observation_spec by tcbegley in https://github.com/pytorch/rl/pull/504
* [BugFix] Changing the dm_control import to fail if not installed by zeenolife in https://github.com/pytorch/rl/pull/515
* [CI] Add coverage with codecov by silvestrebahi in https://github.com/pytorch/rl/pull/523
* Revert "[CI] Add coverage with codecov" by vmoens in https://github.com/pytorch/rl/pull/525
* [Quality] Use relative imports for local c++ deps by apbard in https://github.com/pytorch/rl/pull/526
* [Feature] Nightly release by vmoens in https://github.com/pytorch/rl/pull/519
* [Feature] Add make_tensordict() function by sicong-huang in https://github.com/pytorch/rl/pull/522
* [Doc] Misc readme fixes by GavinPHR in https://github.com/pytorch/rl/pull/532
* [BugFix] Replacing inference_mode decorator with no_grad to fix state_dict loading error by GavinPHR in https://github.com/pytorch/rl/pull/530
* [BugFix] Transformed ParallelEnv meta data are broken when passing to device by vmoens in https://github.com/pytorch/rl/pull/531
* [Doc] Add coverage banner by vmoens in https://github.com/pytorch/rl/pull/533
* [BugFix] Fix colab link of coding_dqn.ipynb by Benjamin-eecs in https://github.com/pytorch/rl/pull/543
* [BugFix] Fix optional imports by vmoens in https://github.com/pytorch/rl/pull/535
* [BugFix] Restore missing keys in data collector output by tcbegley in https://github.com/pytorch/rl/pull/521
* [Lint] reorganize imports by apbard in https://github.com/pytorch/rl/pull/545
* [BugFix] Single-cpu compatibility by vmoens in https://github.com/pytorch/rl/pull/548
* [BugFix] vision install and other deps in optdeps by vmoens in https://github.com/pytorch/rl/pull/552
* [Feature] Implemented `device` argument for `modules.models` by yushiyangk in https://github.com/pytorch/rl/pull/524
* [BugFix] Fix ellipsis indexing of 2d TensorDicts by vmoens in https://github.com/pytorch/rl/pull/559
* [BugFix] Additive gaussian exploration spec fix by vmoens in https://github.com/pytorch/rl/pull/560
* [BugFix] Disabling video step for wandb by vmoens in https://github.com/pytorch/rl/pull/561
* [BugFix] Various device fix by vmoens in https://github.com/pytorch/rl/pull/558
* [Feature] Allow collectors to accept regular modules as policies by tcbegley in https://github.com/pytorch/rl/pull/546
* [BugFix] Fix push binary nightly action by psolikov in https://github.com/pytorch/rl/pull/566
* [BugFix] TensorDict comparison by vmoens in https://github.com/pytorch/rl/pull/567
* [BugFix] Fix SyncDataCollector reset by jrobine in https://github.com/pytorch/rl/pull/571
* [Doc] Banners on README.md by vmoens in https://github.com/pytorch/rl/pull/572
* [Feature] Log printing in alphabetical order when creating a replay buffer by nikhlrao in https://github.com/pytorch/rl/pull/573
* [BugFix] Add eps to reward normalization by vmoens in https://github.com/pytorch/rl/pull/574
* [BugFix] Fix argument for PPOLoss.get_entropy_bonus() by vmoens in https://github.com/pytorch/rl/pull/578
* [Feature] Restructure torchrl/objectives by sgrigory in https://github.com/pytorch/rl/pull/580
* [Docs] Documentation revamp by vmoens in https://github.com/pytorch/rl/pull/581
* [Doc] Publishing on pytorch.org by vmoens in https://github.com/pytorch/rl/pull/582
* Revert "[Doc] Publishing on pytorch.org" by vmoens in https://github.com/pytorch/rl/pull/584
* [Doc] Publishing on pytorch.org by vmoens in https://github.com/pytorch/rl/pull/585
* Revert "[Doc] Publishing on pytorch.org" by vmoens in https://github.com/pytorch/rl/pull/586
* [Doc] Publishing on pytorch.org by vmoens in https://github.com/pytorch/rl/pull/587
* [Feature] More restrictive tests on docstrings by vmoens in https://github.com/pytorch/rl/pull/457
* [BugFix] Wrong stack import in tests by vmoens in https://github.com/pytorch/rl/pull/590
* [Feature] Exclude `"_"` out_keys in tensordictmodel by jlesuffleur in https://github.com/pytorch/rl/pull/589
* [Feature]: Dreamer support by nicolas-dufour in https://github.com/pytorch/rl/pull/341
* [Doc] Missing doc for prototype RB by vmoens in https://github.com/pytorch/rl/pull/595
* [Feature] Update list of supported libraries by vmoens in https://github.com/pytorch/rl/pull/594
* [BugFix] Fix timeit count registration by vmoens in https://github.com/pytorch/rl/pull/598
* [Naming] Renaming `ProbabilisticTensorDictModule` keys by vmoens in https://github.com/pytorch/rl/pull/603
* [Feature] Categorical encoding for action space by artkorenev in https://github.com/pytorch/rl/pull/593
* [BugFix] ReplayBuffer's storage now signal back when changes happen by paulomarciano in https://github.com/pytorch/rl/pull/614
* [Doc] Typos in tensordict tutorial by PaLeroy in https://github.com/pytorch/rl/pull/621
* [Doc] Integrate knowledge base in docs by hatala91 in https://github.com/pytorch/rl/pull/622
* [Doc] Updating docs requirements by vmoens in https://github.com/pytorch/rl/pull/624
* [Feature] Make torchrl runnable without functorch and with gym==0.13 by vmoens in https://github.com/pytorch/rl/pull/386
* [Feature] Habitat integration by vmoens in https://github.com/pytorch/rl/pull/514
* [Feature] Checkpointing by vmoens in https://github.com/pytorch/rl/pull/549
* Add support for null `dim` argument in `TensorDict.squeeze` by jgonik in https://github.com/pytorch/rl/pull/608
* [Version] Updating to torch 1.13 by vmoens in https://github.com/pytorch/rl/pull/627
* [Feature] Sub-memmap tensors by vmoens in https://github.com/pytorch/rl/pull/626
* [BugFix] `copy_` changes the index if the dest and source memmap tensors share the same file location by vmoens in https://github.com/pytorch/rl/pull/631
* [Feature] Unfold transforms for folded `TransformedEnv` by alexanderlobov in https://github.com/pytorch/rl/pull/630
* [BugFix] make TensorDictReplayBuffer.extend call super().extend with stacked_td by vmoens in https://github.com/pytorch/rl/pull/634
* [BugFix] correct the use of `step_mdp` method in data collector by adityagandhamal in https://github.com/pytorch/rl/pull/637
* [Feature] Added `implement_for` decorator by ordinskiy in https://github.com/pytorch/rl/pull/618
* [Feature] Make DQN compatible with nn.Module by svarolgunes in https://github.com/pytorch/rl/pull/632
* [Example] Distributed Replay Buffer Prototype Example Implementation by adityagoel4512 in https://github.com/pytorch/rl/pull/615
* [Feature] Benchmark storage types by adityagoel4512 in https://github.com/pytorch/rl/pull/633
* [Feature] Remove wild imports in the library by sosmond in https://github.com/pytorch/rl/pull/642
* [BugFix] Prevent transform parent from being reassigned by jasonfkut in https://github.com/pytorch/rl/pull/641
* [Feature] Too many `deepcopy` in `transforms.py` by romainjln in https://github.com/pytorch/rl/pull/625
* [Naming] Rename keys_in to in_keys in transforms.py and related modules by sardaankita in https://github.com/pytorch/rl/pull/656
* [Refactoring] Refactor dreamer helper in smaller pieces by vmoens in https://github.com/pytorch/rl/pull/662
* [Feature] VIPRewardTransform by vmoens in https://github.com/pytorch/rl/pull/658
* [BugFix] make_trainer possible bug for on-policy cases by albertbou92 in https://github.com/pytorch/rl/pull/655
* [Naming] Fixing key names by vmoens in https://github.com/pytorch/rl/pull/668
* [Test] Check dtypes of envs by vmoens in https://github.com/pytorch/rl/pull/666
* [Refactor] Relying on the standalone tensordict -- phase 1 by vmoens in https://github.com/pytorch/rl/pull/650
* [Doc] More doc on trainers by vmoens in https://github.com/pytorch/rl/pull/663
* [BugFix] PPO example GAE import by albertbou92 in https://github.com/pytorch/rl/pull/671
* [BugFix] Use GitHub for flake8 pre-commit hook by vmoens in https://github.com/pytorch/rl/pull/679
* [BugFix] Update to strict select by vmoens in https://github.com/pytorch/rl/pull/675
* [Feature] Auto-compute stats for ObservationNorm by romainjln in https://github.com/pytorch/rl/pull/669
* [Doc] _make_collector helper function by albertbou92 in https://github.com/pytorch/rl/pull/678
* [Doc] BatchSubSampler class docstrings example by albertbou92 in https://github.com/pytorch/rl/pull/677
* [BugFix] PPO objective crashes if advantage_module is None by albertbou92 in https://github.com/pytorch/rl/pull/676
* [Refactor] Refactor 'next_' into nested tensordicts by vmoens in https://github.com/pytorch/rl/pull/649
* [Doc] More doc about environments by vmoens in https://github.com/pytorch/rl/pull/683
* [Doc] Fix missing tensordict install for doc by vmoens in https://github.com/pytorch/rl/pull/685
* [CI] Added CircleCI pipeline to test compatibility across supported gym versions by ordinskiy in https://github.com/pytorch/rl/pull/645
* [BugFix] ConvNet forward method with tensors of more than 4 dimensions by albertbou92 in https://github.com/pytorch/rl/pull/686
* [Feature] add `standard_normal` for RewardScaling by adityagandhamal in https://github.com/pytorch/rl/pull/682
* [Feature] Jumanji envs by yingchenlin in https://github.com/pytorch/rl/pull/674
* [Feature] Default collate_fn by vmoens in https://github.com/pytorch/rl/pull/688
* [BugFix] Fix Examples by vmoens in https://github.com/pytorch/rl/pull/687
* [Refactoring] Replace direct gym version checks with decorated functions () by ordinskiy in https://github.com/pytorch/rl/pull/691
* Version 0.0.3 by vmoens in https://github.com/pytorch/rl/pull/696
* [Docs] Host TensorDict docs inside TorchRL docs by tcbegley in https://github.com/pytorch/rl/pull/693
* [BugFix] Fix docs build by tcbegley in https://github.com/pytorch/rl/pull/698
* [BugFix] Proper error messages for orphan transform creation by vmoens in https://github.com/pytorch/rl/pull/697
* [Feature] Append, init and insert transforms in ReplayBuffer by altre in https://github.com/pytorch/rl/pull/695
* [Feature] A2C objective class and train example by albertbou92 in https://github.com/pytorch/rl/pull/680
* [Doc, Test] Add A2C script test and doc by vmoens in https://github.com/pytorch/rl/pull/702
* [BugFix] Initialising the classes LazyTensorStorage with a nested TensorDict raises error by albertbou92 in https://github.com/pytorch/rl/pull/703
* [BugFix] Fix init_random_frames in A2C example test by vmoens in https://github.com/pytorch/rl/pull/706
* [Formatting] Upgrade formatting libs by vmoens in https://github.com/pytorch/rl/pull/705
* [Doc] Document undefined symbol error with torch version < 1.13 by nickspell in https://github.com/pytorch/rl/pull/707
* [Doc] Tuto integration by vmoens in https://github.com/pytorch/rl/pull/681
* [Quality] Deprecate .ipynb tutos by vmoens in https://github.com/pytorch/rl/pull/710
* [Test] Fix wrong skip message when functorch is installed by vmoens in https://github.com/pytorch/rl/pull/711
* [BugFix, Doc] Clone TensorDict docs into _local_build by tcbegley in https://github.com/pytorch/rl/pull/712
* [Feature] Migrate to `tensordict.nn.TensorDictModule` by tcbegley in https://github.com/pytorch/rl/pull/700
* [Doc] Fix Tutos TODOs by vmoens in https://github.com/pytorch/rl/pull/713
* [BugFix] RoundRobinWriter, possible duplicated code in the extend method by albertbou92 in https://github.com/pytorch/rl/pull/709
* [Feature] Add OptimizerHook by aakhundov in https://github.com/pytorch/rl/pull/716
* [Feature] Support for in-place functionalization by tcbegley in https://github.com/pytorch/rl/pull/714
* [BugFix] Fix TorchRL demo tutorial by vmoens in https://github.com/pytorch/rl/pull/721
* [Docs] Update tutorial links in readme by tcbegley in https://github.com/pytorch/rl/pull/724
* [Feature] Extend PPO loss helper to allow for more customisation by albertbou92 in https://github.com/pytorch/rl/pull/718
* [BugFix] Model maker functions for A2C and PPO fail for discrete action space envs by albertbou92 in https://github.com/pytorch/rl/pull/717
* [Minor] docstrings and setup fixes by vmoens in https://github.com/pytorch/rl/pull/726
* [BugFix] Avoid wrongfully erasing observation keys from specs in CatTensors by vmoens in https://github.com/pytorch/rl/pull/727
* [BugFix] Avoid wrongfully erasing observation keys from tensordict in CatTensors by vmoens in https://github.com/pytorch/rl/pull/729
* [Doc] More doc for data collectors by vmoens in https://github.com/pytorch/rl/pull/732
* [Feature] Port test_fake_tensordict to torchrl by vmoens in https://github.com/pytorch/rl/pull/731
* [Feature] Use ObservationNorm.init_stats for stats computation in example scripts by romainjln in https://github.com/pytorch/rl/pull/715
* [BugFix] init_stats over multiple dimensions by vmoens in https://github.com/pytorch/rl/pull/735
* [Refactor] logger creation in examples by acforvs in https://github.com/pytorch/rl/pull/733
* [Feature] Brax envs by yingchenlin in https://github.com/pytorch/rl/pull/722
* [Refactor] Adopt prototype ProbabilisticTensorDictModule and ProbabilisticTensorDictSequential by tcbegley in https://github.com/pytorch/rl/pull/728
* [Doc] Link to doc in README by vmoens in https://github.com/pytorch/rl/pull/740
* [Feature] Make GAE return a 'value_target' entry by vmoens in https://github.com/pytorch/rl/pull/741
* [Feature] SamplerWithoutReplacement by vmoens in https://github.com/pytorch/rl/pull/742
* [Doc, CI] Update doc workflow to run on PR and only publishes doc on main. by EmGarr in https://github.com/pytorch/rl/pull/745
* [Feature] Better advantage API for higher order derivatives by vmoens in https://github.com/pytorch/rl/pull/744
* [Refactor] Cosmetic improvements to advantage modules by vmoens in https://github.com/pytorch/rl/pull/746
* [BugFix] Fix NoopReset in parallel settings by vmoens in https://github.com/pytorch/rl/pull/747
* [Refactor] Remove env.is_done attribute by vmoens in https://github.com/pytorch/rl/pull/748
* [Refactor] Drop prototype imports by tcbegley in https://github.com/pytorch/rl/pull/738
* [BugFix] Fixes for `speed` branch merge on tensordict by vmoens in https://github.com/pytorch/rl/pull/755
* [BugFix] Fix size-match unsqueeze deprecation by vmoens in https://github.com/pytorch/rl/pull/750
* [Feature] FrameSkipTransform by vmoens in https://github.com/pytorch/rl/pull/749
* [BugFix] Better memory management for collectors by vmoens in https://github.com/pytorch/rl/pull/763
* Minor cleaning in BaseEnv classes by matteobettini in https://github.com/pytorch/rl/pull/767
* Revert "Minor cleaning in BaseEnv classes" by vmoens in https://github.com/pytorch/rl/pull/768
* Cleaning in envs `common.py` by matteobettini in https://github.com/pytorch/rl/pull/769
* Making `_set_seed` abstract by matteobettini in https://github.com/pytorch/rl/pull/770
* [Feature] Remove the Nd*TensorSpec classes by riiswa in https://github.com/pytorch/rl/pull/772
* [BugFix] Reinstantiate custom value key for multioutput value networks by vmoens in https://github.com/pytorch/rl/pull/754
* [Feature] Add Step Counter transform by riiswa in https://github.com/pytorch/rl/pull/756
* [BugFix] Batched environments with non empty batch size by matteobettini in https://github.com/pytorch/rl/pull/774
* Allow undounded boxes creation from gym spaces by matteobettini in https://github.com/pytorch/rl/pull/778
* [BugFix] Doc built cmake error by vmoens in https://github.com/pytorch/rl/pull/780
* [Feature] Lazy TensorClass storage by tcbegley in https://github.com/pytorch/rl/pull/752
* [BugFix] SyncDataCollector init when device and env_device are different by albertbou92 in https://github.com/pytorch/rl/pull/765
* [Feature] RewardSum transform by albertbou92 in https://github.com/pytorch/rl/pull/751
* [BugFix] Fix PPO clip by vmoens in https://github.com/pytorch/rl/pull/786
* [Feature] MultiDiscreteTensorSpec by riiswa in https://github.com/pytorch/rl/pull/783
* [Doc] Doc revamp by vmoens in https://github.com/pytorch/rl/pull/782
* [BugFix] `ParallelEnv` handling of done flag by matteobettini in https://github.com/pytorch/rl/pull/788
* [BugFix] Sorting nested keys by matteobettini in https://github.com/pytorch/rl/pull/787
* [Doc] README index by vmoens in https://github.com/pytorch/rl/pull/791
* Add windows wheel build to CircleCI by yohann-benchetrit in https://github.com/pytorch/rl/pull/759
* [Algorithm] MPPI planner by vmoens in https://github.com/pytorch/rl/pull/701
* [Doc] Better doc links by vmoens in https://github.com/pytorch/rl/pull/795
* [Doc] Missing headers by vmoens in https://github.com/pytorch/rl/pull/796
* [Doc] Knowledge base section by vmoens in https://github.com/pytorch/rl/pull/797
* [Feature] Vmas library wrapper by matteobettini in https://github.com/pytorch/rl/pull/785
* [Doc] Duplicate `HabitatEnv` entry in docs by matteobettini in https://github.com/pytorch/rl/pull/798
* [Feature] MultiDiscreteTensorSpec nvec with several axes by riiswa in https://github.com/pytorch/rl/pull/789
* [Refactor] Graduate Replay Buffer prototype by KamilPiechowiak in https://github.com/pytorch/rl/pull/794
* [BugFix] Solve R3MTransform init problem by vmoens in https://github.com/pytorch/rl/pull/803
* [Refactor] Simplify FlattenObservation default kwargs by vmoens in https://github.com/pytorch/rl/pull/805
* [Format] Fix lint by vmoens in https://github.com/pytorch/rl/pull/811
* [Doc, BugFix] Fix tutos errors by vmoens in https://github.com/pytorch/rl/pull/817
* [Doc] Pretrained models tutorial by vmoens in https://github.com/pytorch/rl/pull/814
* [Doc, BugFix] Fix tensordictmodule tutorial by vmoens in https://github.com/pytorch/rl/pull/819
* [BugFix] Fix MultOneHotDiscreteTensorSpec.is_in by riiswa in https://github.com/pytorch/rl/pull/818
* [Doc] Using R3M with a replay buffer by vmoens in https://github.com/pytorch/rl/pull/820
* [CodeQuality] call all() without making a list by riiswa in https://github.com/pytorch/rl/pull/821
* [BugFix] [Feature] "_reset" flag for env reset by matteobettini in https://github.com/pytorch/rl/pull/800
* [CI] Add unit test workflows for Windows by yohann-benchetrit in https://github.com/pytorch/rl/pull/804
* [BugFix] Fix habitat integration and doc by vmoens in https://github.com/pytorch/rl/pull/812
* [Minor] Better error reporting by vmoens in https://github.com/pytorch/rl/pull/822
* [Minor] Add ninja to deps in toml file by vmoens in https://github.com/pytorch/rl/pull/823
* [BugFix] Device of info specs by vmoens in https://github.com/pytorch/rl/pull/824
* [BugFix] Fix envs specs and info reading by vmoens in https://github.com/pytorch/rl/pull/825
* [Feature] Dtype in vmas tests by matteobettini in https://github.com/pytorch/rl/pull/827
* [BugFix] Fix R3M observation spec transform by vmoens in https://github.com/pytorch/rl/pull/830
* small change to make robandpdx a contributor by robandpdx in https://github.com/pytorch/rl/pull/831
* [Feature] Exclude and select transforms by vmoens in https://github.com/pytorch/rl/pull/832
* [BugFix] Updating Recorder to accomodate "solved" key by ShahRutav in https://github.com/pytorch/rl/pull/833
* [BugFIx] Changed "set_count" set in collectors by matteobettini in https://github.com/pytorch/rl/pull/835
* [Algorithm] Td3 by BY571 in https://github.com/pytorch/rl/pull/684
* [Doc] A Succinct Summary of Reinforcement Learning by vmoens in https://github.com/pytorch/rl/pull/840
* [Feature, BugFix] ObservationNorm keep_dims and RewardSum init by vmoens in https://github.com/pytorch/rl/pull/839
* [BugFix] Improve done checking of collectors by matteobettini in https://github.com/pytorch/rl/pull/838
* [BugFix] Sync with tensordict (meta-tensor deprecation) by vmoens in https://github.com/pytorch/rl/pull/842
* [Feature] Refactor CatFrames using a proper preallocated buffer by vmoens in https://github.com/pytorch/rl/pull/847
* [CI] Add Github-Actions workflows for Windows wheels & nightly-build by yohann-benchetrit in https://github.com/pytorch/rl/pull/837
* [Doc] Fix broken link Dreamer by atonkamanda in https://github.com/pytorch/rl/pull/853
* [BugFix] Loading state_dict on uninitialized CatFrames by vmoens in https://github.com/pytorch/rl/pull/855
* [Refactor] Move loggers to `torchrl.record` by vmoens in https://github.com/pytorch/rl/pull/854
* [Refactor] specs batch size refactoring by vmoens in https://github.com/pytorch/rl/pull/829
* [Feature] Max pool Transform by albertbou92 in https://github.com/pytorch/rl/pull/841
* [Feature] Refactor advantages for continuous batches by vmoens in https://github.com/pytorch/rl/pull/848
* [BugFix, Doc] Minor fix in doc by vmoens in https://github.com/pytorch/rl/pull/858
* [Versioning] Version 0.0.4a by vmoens in https://github.com/pytorch/rl/pull/859
* [Feature] Vmas to device by matteobettini in https://github.com/pytorch/rl/pull/850
* [BugFix] Fix zero-ing from specs in RewardSum by vmoens in https://github.com/pytorch/rl/pull/860
* [Feature] Loading R3M and VIP from ResNet by vmoens in https://github.com/pytorch/rl/pull/863
* [Feature] SAC V2 by vmoens in https://github.com/pytorch/rl/pull/864
* [BugFix] Avoid collision of "step_count" key from transform and collector by vmoens in https://github.com/pytorch/rl/pull/868
* [Refactor] Better init for CatFrames buffers + removing default init values by vmoens in https://github.com/pytorch/rl/pull/874
* [Refactor] Minor refactorings to envs by vmoens in https://github.com/pytorch/rl/pull/872
* [Refactor] Removing inplace transform attribute by vmoens in https://github.com/pytorch/rl/pull/871
* [BugFix] Run checks when creating fake_td by vmoens in https://github.com/pytorch/rl/pull/877
* [Refactor] Box device by vmoens in https://github.com/pytorch/rl/pull/881
* [Feature] Multithreaded env by sgrigory in https://github.com/pytorch/rl/pull/734
* [Refactor] Turn off default advantage normalization in PPO by vmoens in https://github.com/pytorch/rl/pull/887
* [CI] Fix habitat-gym imports by vmoens in https://github.com/pytorch/rl/pull/890
* [CI] Fix cuda versions by vmoens in https://github.com/pytorch/rl/pull/889
* [CI] Fix windows install by vmoens in https://github.com/pytorch/rl/pull/888
* MacOS CPU unit test workflow using GitHub Actions by robandpdx in https://github.com/pytorch/rl/pull/886
* Linux CPU unit test workflow using GitHub Actions by robandpdx in https://github.com/pytorch/rl/pull/826
* [Major, BugFix, Test] Refactor Transforms tests by vmoens in https://github.com/pytorch/rl/pull/878
* [Bugfix] Codecov does not cover multiprocessed tests 879 by kadeng in https://github.com/pytorch/rl/pull/893
* [CI, BugFix] Fix gym related errors by vmoens in https://github.com/pytorch/rl/pull/895
* [WIP] Linux GPU unit test workflow using GitHub Actions by robandpdx in https://github.com/pytorch/rl/pull/885
* [BugFix] Compose cloning fix by vmoens in https://github.com/pytorch/rl/pull/899
* [Feature] Simplifying collector envs by vmoens in https://github.com/pytorch/rl/pull/870
* [CI,Feature] Upgrade to gymnasium by vmoens in https://github.com/pytorch/rl/pull/898
* [Doc] Add record utils to doc by vmoens in https://github.com/pytorch/rl/pull/904
* [Test] Improve exception message match by apbard in https://github.com/pytorch/rl/pull/906
* [BugFix] Dreamer helpers are broken with batched envs by vmoens in https://github.com/pytorch/rl/pull/903
* [Feature] RandomCropTensorDict transform by vmoens in https://github.com/pytorch/rl/pull/908
* [Versioning] Version 0.0.4b by vmoens in https://github.com/pytorch/rl/pull/909

New Contributors
* sladebot made their first contribution in https://github.com/pytorch/rl/pull/435
* rayanht made their first contribution in https://github.com/pytorch/rl/pull/432
* brandonsj made their first contribution in https://github.com/pytorch/rl/pull/475
* ordinskiy made their first contribution in https://github.com/pytorch/rl/pull/485
* JasonMa2016 made their first contribution in https://github.com/pytorch/rl/pull/487
* himjohntang made their first contribution in https://github.com/pytorch/rl/pull/477
* romainjln made their first contribution in https://github.com/pytorch/rl/pull/512
* apbard made their first contribution in https://github.com/pytorch/rl/pull/526
* sicong-huang made their first contribution in https://github.com/pytorch/rl/pull/522
* psolikov made their first contribution in https://github.com/pytorch/rl/pull/566
* jrobine made their first contribution in https://github.com/pytorch/rl/pull/571
* nikhlrao made their first contribution in https://github.com/pytorch/rl/pull/573
* sgrigory made their first contribution in https://github.com/pytorch/rl/pull/580
* jlesuffleur made their first contribution in https://github.com/pytorch/rl/pull/589
* artkorenev made their first contribution in https://github.com/pytorch/rl/pull/593
* paulomarciano made their first contribution in https://github.com/pytorch/rl/pull/614
* hatala91 made their first contribution in https://github.com/pytorch/rl/pull/622
* jgonik made their first contribution in https://github.com/pytorch/rl/pull/608
* adityagandhamal made their first contribution in https://github.com/pytorch/rl/pull/637
* svarolgunes made their first contribution in https://github.com/pytorch/rl/pull/632
* adityagoel4512 made their first contribution in https://github.com/pytorch/rl/pull/615
* jasonfkut made their first contribution in https://github.com/pytorch/rl/pull/641
* sardaankita made their first contribution in https://github.com/pytorch/rl/pull/656
* albertbou92 made their first contribution in https://github.com/pytorch/rl/pull/655
* yingchenlin made their first contribution in https://github.com/pytorch/rl/pull/674
* altre made their first contribution in https://github.com/pytorch/rl/pull/695
* nickspell made their first contribution in https://github.com/pytorch/rl/pull/707
* aakhundov made their first contribution in https://github.com/pytorch/rl/pull/716
* acforvs made their first contribution in https://github.com/pytorch/rl/pull/733
* EmGarr made their first contribution in https://github.com/pytorch/rl/pull/745
* matteobettini made their first contribution in https://github.com/pytorch/rl/pull/767
* riiswa made their first contribution in https://github.com/pytorch/rl/pull/772
* yohann-benchetrit made their first contribution in https://github.com/pytorch/rl/pull/759
* KamilPiechowiak made their first contribution in https://github.com/pytorch/rl/pull/794
* robandpdx made their first contribution in https://github.com/pytorch/rl/pull/831
* ShahRutav made their first contribution in https://github.com/pytorch/rl/pull/833
* BY571 made their first contribution in https://github.com/pytorch/rl/pull/684
* atonkamanda made their first contribution in https://github.com/pytorch/rl/pull/853
* kadeng made their first contribution in https://github.com/pytorch/rl/pull/893

**Full Changelog**: https://github.com/pytorch/rl/compare/v0.0.2a...v0.0.4b

What's Changed
* [CI, Doc] Update functorch source installation command by zou3519 in https://github.com/pytorch/rl/pull/446
* [BugFix] TransformedEnv attributes inheritance by vmoens in https://github.com/pytorch/rl/pull/467
* [Feature] Cleanup mocking envs init and new by vmoens in https://github.com/pytorch/rl/pull/469
* [Tests] Adding tensordict `__repr__` tests by sladebot in https://github.com/pytorch/rl/pull/435
* [Logging]: implement MLFlow logging integration by rayanht in https://github.com/pytorch/rl/pull/432
* [BugFix] MLFlow import fix by vmoens in https://github.com/pytorch/rl/pull/473
* [BugFix] Fixed pip install by brandonsj in https://github.com/pytorch/rl/pull/475
* [Features]: Changed `_inplace_update ` cls parameter passing in `__new__` by nicolas-dufour in https://github.com/pytorch/rl/pull/464
* [Feature]: ModelBased Envs by nicolas-dufour in https://github.com/pytorch/rl/pull/333
* [Feature] make ReplayBufferTrainer compatible with storing trajectories by vmoens in https://github.com/pytorch/rl/pull/476
* [Tutorial] DQN tutorial by vmoens in https://github.com/pytorch/rl/pull/474
* [Feature] reader hooks for GymLike by vmoens in https://github.com/pytorch/rl/pull/478
* [BugFix] TensorSpec.zero(None) failure fix by vmoens in https://github.com/pytorch/rl/pull/483
* [Feature]: Support for planners and CEM by nicolas-dufour in https://github.com/pytorch/rl/pull/384
* [Feature] Replaced `device_safe()` with `device` by ordinskiy in https://github.com/pytorch/rl/pull/485
* [Feature]: `TensorDictPrimer` transform by nicolas-dufour in https://github.com/pytorch/rl/pull/456
* [Feature]: `erase()` method for `torchrl.timeit` by nicolas-dufour in https://github.com/pytorch/rl/pull/480
* [Feature] Added support for single collector in sync_async_collector by nicolas-dufour in https://github.com/pytorch/rl/pull/482
* [BugFix] removing unwanted device_safe() by vmoens in https://github.com/pytorch/rl/pull/486
* [Refactoring] Refactored get_stats_random_rollout by nicolas-dufour in https://github.com/pytorch/rl/pull/481
* [Feature] VIP Integration by JasonMa2016 in https://github.com/pytorch/rl/pull/487
* [Refactoring] Minor tweaks to recorder and logger by nicolas-dufour in https://github.com/pytorch/rl/pull/489
* [Feature]: Deactivate typechecks in envs by nicolas-dufour in https://github.com/pytorch/rl/pull/490
* [BugFix] Vectorized td_lambda with gamma tensor does not match the serial version by vmoens in https://github.com/pytorch/rl/pull/400
* [BugFix] Fix TensorDictPrimer init by vmoens in https://github.com/pytorch/rl/pull/491
* [Feature] Optional auto-reset when done for collectors and batched envs by vmoens in https://github.com/pytorch/rl/pull/492
* [BugFix] Defaulting `passing_devices` to `None` by himjohntang in https://github.com/pytorch/rl/pull/477
* Revert "[BugFix] Defaulting `passing_devices` to `None`" by vmoens in https://github.com/pytorch/rl/pull/494
* [BugFix] Multi-agent fixes by vmoens in https://github.com/pytorch/rl/pull/488
* [BugFix] Defaulting `passing_devices` to `None` by vmoens in https://github.com/pytorch/rl/pull/495
* [Feature] Lazy initialization of CatTensors by vmoens in https://github.com/pytorch/rl/pull/497
* [Cleanup] Removing cuda 10.2 references by vmoens in https://github.com/pytorch/rl/pull/498
* [BugFix] Migration to pytorch org by vmoens in https://github.com/pytorch/rl/pull/499
* [Refactoring] Import at root to enable vmap monkey-patching by vmoens in https://github.com/pytorch/rl/pull/500
* [BugFix] python version for linting checks by vmoens in https://github.com/pytorch/rl/pull/502
* [Feature] Replay Buffers refactor by bamaxw in https://github.com/pytorch/rl/pull/330
* [Feature] Rename `step_tensordict` in `step_mdp` by romainjln in https://github.com/pytorch/rl/pull/512
* [Lint] re-instantiate F821 by vmoens in https://github.com/pytorch/rl/pull/516
* [BugFix] run_type_checks for TransformedEnvs by vmoens in https://github.com/pytorch/rl/pull/513
* [BugFix] making first_dim and last_dim negative in FlattenObservation when a parent is set by vmoens in https://github.com/pytorch/rl/pull/511
* [Feature] Add info dict key-spec pairs to observation_spec by tcbegley in https://github.com/pytorch/rl/pull/504
* [BugFix] Changing the dm_control import to fail if not installed by zeenolife in https://github.com/pytorch/rl/pull/515
* [CI] Add coverage with codecov by silvestrebahi in https://github.com/pytorch/rl/pull/523
* Revert "[CI] Add coverage with codecov" by vmoens in https://github.com/pytorch/rl/pull/525
* [Quality] Use relative imports for local c++ deps by apbard in https://github.com/pytorch/rl/pull/526
* [Feature] Nightly release by vmoens in https://github.com/pytorch/rl/pull/519
* [Feature] Add make_tensordict() function by sicong-huang in https://github.com/pytorch/rl/pull/522
* [Doc] Misc readme fixes by GavinPHR in https://github.com/pytorch/rl/pull/532
* [BugFix] Replacing inference_mode decorator with no_grad to fix state_dict loading error by GavinPHR in https://github.com/pytorch/rl/pull/530
* [BugFix] Transformed ParallelEnv meta data are broken when passing to device by vmoens in https://github.com/pytorch/rl/pull/531
* [Doc] Add coverage banner by vmoens in https://github.com/pytorch/rl/pull/533
* [BugFix] Fix colab link of coding_dqn.ipynb by Benjamin-eecs in https://github.com/pytorch/rl/pull/543
* [BugFix] Fix optional imports by vmoens in https://github.com/pytorch/rl/pull/535
* [BugFix] Restore missing keys in data collector output by tcbegley in https://github.com/pytorch/rl/pull/521
* [Lint] reorganize imports by apbard in https://github.com/pytorch/rl/pull/545
* [BugFix] Single-cpu compatibility by vmoens in https://github.com/pytorch/rl/pull/548
* [BugFix] vision install and other deps in optdeps by vmoens in https://github.com/pytorch/rl/pull/552
* [Feature] Implemented `device` argument for `modules.models` by yushiyangk in https://github.com/pytorch/rl/pull/524
* [BugFix] Fix ellipsis indexing of 2d TensorDicts by vmoens in https://github.com/pytorch/rl/pull/559
* [BugFix] Additive gaussian exploration spec fix by vmoens in https://github.com/pytorch/rl/pull/560
* [BugFix] Disabling video step for wandb by vmoens in https://github.com/pytorch/rl/pull/561
* [BugFix] Various device fix by vmoens in https://github.com/pytorch/rl/pull/558
* [Feature] Allow collectors to accept regular modules as policies by tcbegley in https://github.com/pytorch/rl/pull/546
* [BugFix] Fix push binary nightly action by psolikov in https://github.com/pytorch/rl/pull/566
* [BugFix] TensorDict comparison by vmoens in https://github.com/pytorch/rl/pull/567
* [BugFix] Fix SyncDataCollector reset by jrobine in https://github.com/pytorch/rl/pull/571
* [Doc] Banners on README.md by vmoens in https://github.com/pytorch/rl/pull/572
* [Feature] Log printing in alphabetical order when creating a replay buffer by nikhlrao in https://github.com/pytorch/rl/pull/573
* [BugFix] Add eps to reward normalization by vmoens in https://github.com/pytorch/rl/pull/574
* [BugFix] Fix argument for PPOLoss.get_entropy_bonus() by vmoens in https://github.com/pytorch/rl/pull/578
* [Feature] Restructure torchrl/objectives by sgrigory in https://github.com/pytorch/rl/pull/580
* [Docs] Documentation revamp by vmoens in https://github.com/pytorch/rl/pull/581
* [Doc] Publishing on pytorch.org by vmoens in https://github.com/pytorch/rl/pull/582
* Revert "[Doc] Publishing on pytorch.org" by vmoens in https://github.com/pytorch/rl/pull/584
* [Doc] Publishing on pytorch.org by vmoens in https://github.com/pytorch/rl/pull/585
* Revert "[Doc] Publishing on pytorch.org" by vmoens in https://github.com/pytorch/rl/pull/586
* [Doc] Publishing on pytorch.org by vmoens in https://github.com/pytorch/rl/pull/587
* [Feature] More restrictive tests on docstrings by vmoens in https://github.com/pytorch/rl/pull/457
* [BugFix] Wrong stack import in tests by vmoens in https://github.com/pytorch/rl/pull/590
* [Feature] Exclude `"_"` out_keys in tensordictmodel by jlesuffleur in https://github.com/pytorch/rl/pull/589
* [Feature]: Dreamer support by nicolas-dufour in https://github.com/pytorch/rl/pull/341
* [Doc] Missing doc for prototype RB by vmoens in https://github.com/pytorch/rl/pull/595
* [Feature] Update list of supported libraries by vmoens in https://github.com/pytorch/rl/pull/594
* [BugFix] Fix timeit count registration by vmoens in https://github.com/pytorch/rl/pull/598
* [Naming] Renaming `ProbabilisticTensorDictModule` keys by vmoens in https://github.com/pytorch/rl/pull/603
* [Feature] Categorical encoding for action space by artkorenev in https://github.com/pytorch/rl/pull/593
* [BugFix] ReplayBuffer's storage now signal back when changes happen by paulomarciano in https://github.com/pytorch/rl/pull/614
* [Doc] Typos in tensordict tutorial by PaLeroy in https://github.com/pytorch/rl/pull/621
* [Doc] Integrate knowledge base in docs by hatala91 in https://github.com/pytorch/rl/pull/622
* [Doc] Updating docs requirements by vmoens in https://github.com/pytorch/rl/pull/624
* [Feature] Make torchrl runnable without functorch and with gym==0.13 by vmoens in https://github.com/pytorch/rl/pull/386
* [Feature] Habitat integration by vmoens in https://github.com/pytorch/rl/pull/514
* [Feature] Checkpointing by vmoens in https://github.com/pytorch/rl/pull/549
* Add support for null `dim` argument in `TensorDict.squeeze` by jgonik in https://github.com/pytorch/rl/pull/608
* [Version] Updating to torch 1.13 by vmoens in https://github.com/pytorch/rl/pull/627
* [Feature] Sub-memmap tensors by vmoens in https://github.com/pytorch/rl/pull/626
* [BugFix] `copy_` changes the index if the dest and source memmap tensors share the same file location by vmoens in https://github.com/pytorch/rl/pull/631
* [Feature] Unfold transforms for folded `TransformedEnv` by alexanderlobov in https://github.com/pytorch/rl/pull/630
* [BugFix] make TensorDictReplayBuffer.extend call super().extend with stacked_td by vmoens in https://github.com/pytorch/rl/pull/634
* [BugFix] correct the use of `step_mdp` method in data collector by adityagandhamal in https://github.com/pytorch/rl/pull/637
* [Feature] Added `implement_for` decorator by ordinskiy in https://github.com/pytorch/rl/pull/618
* [Feature] Make DQN compatible with nn.Module by svarolgunes in https://github.com/pytorch/rl/pull/632
* [Example] Distributed Replay Buffer Prototype Example Implementation by adityagoel4512 in https://github.com/pytorch/rl/pull/615
* [Feature] Benchmark storage types by adityagoel4512 in https://github.com/pytorch/rl/pull/633
* [Feature] Remove wild imports in the library by sosmond in https://github.com/pytorch/rl/pull/642
* [BugFix] Prevent transform parent from being reassigned by jasonfkut in https://github.com/pytorch/rl/pull/641
* [Feature] Too many `deepcopy` in `transforms.py` by romainjln in https://github.com/pytorch/rl/pull/625
* [Naming] Rename keys_in to in_keys in transforms.py and related modules by sardaankita in https://github.com/pytorch/rl/pull/656
* [Refactoring] Refactor dreamer helper in smaller pieces by vmoens in https://github.com/pytorch/rl/pull/662
* [Feature] VIPRewardTransform by vmoens in https://github.com/pytorch/rl/pull/658
* [BugFix] make_trainer possible bug for on-policy cases by albertbou92 in https://github.com/pytorch/rl/pull/655
* [Naming] Fixing key names by vmoens in https://github.com/pytorch/rl/pull/668
* [Test] Check dtypes of envs by vmoens in https://github.com/pytorch/rl/pull/666
* [Refactor] Relying on the standalone tensordict -- phase 1 by vmoens in https://github.com/pytorch/rl/pull/650
* [Doc] More doc on trainers by vmoens in https://github.com/pytorch/rl/pull/663
* [BugFix] PPO example GAE import by albertbou92 in https://github.com/pytorch/rl/pull/671
* [BugFix] Use GitHub for flake8 pre-commit hook by vmoens in https://github.com/pytorch/rl/pull/679
* [BugFix] Update to strict select by vmoens in https://github.com/pytorch/rl/pull/675
* [Feature] Auto-compute stats for ObservationNorm by romainjln in https://github.com/pytorch/rl/pull/669
* [Doc] _make_collector helper function by albertbou92 in https://github.com/pytorch/rl/pull/678
* [Doc] BatchSubSampler class docstrings example by albertbou92 in https://github.com/pytorch/rl/pull/677
* [BugFix] PPO objective crashes if advantage_module is None by albertbou92 in https://github.com/pytorch/rl/pull/676
* [Refactor] Refactor 'next_' into nested tensordicts by vmoens in https://github.com/pytorch/rl/pull/649
* [Doc] More doc about environments by vmoens in https://github.com/pytorch/rl/pull/683
* [Doc] Fix missing tensordict install for doc by vmoens in https://github.com/pytorch/rl/pull/685
* [CI] Added CircleCI pipeline to test compatibility across supported gym versions by ordinskiy in https://github.com/pytorch/rl/pull/645
* [BugFix] ConvNet forward method with tensors of more than 4 dimensions by albertbou92 in https://github.com/pytorch/rl/pull/686
* [Feature] add `standard_normal` for RewardScaling by adityagandhamal in https://github.com/pytorch/rl/pull/682
* [Feature] Jumanji envs by yingchenlin in https://github.com/pytorch/rl/pull/674
* [Feature] Default collate_fn by vmoens in https://github.com/pytorch/rl/pull/688
* [BugFix] Fix Examples by vmoens in https://github.com/pytorch/rl/pull/687
* [Refactoring] Replace direct gym version checks with decorated functions () by ordinskiy in https://github.com/pytorch/rl/pull/691
* Version 0.0.3 by vmoens in https://github.com/pytorch/rl/pull/696
* [Docs] Host TensorDict docs inside TorchRL docs by tcbegley in https://github.com/pytorch/rl/pull/693
* [BugFix] Fix docs build by tcbegley in https://github.com/pytorch/rl/pull/698
* [BugFix] Proper error messages for orphan transform creation by vmoens in https://github.com/pytorch/rl/pull/697
* [Feature] Append, init and insert transforms in ReplayBuffer by altre in https://github.com/pytorch/rl/pull/695
* [Feature] A2C objective class and train example by albertbou92 in https://github.com/pytorch/rl/pull/680
* [Doc, Test] Add A2C script test and doc by vmoens in https://github.com/pytorch/rl/pull/702
* [BugFix] Initialising the classes LazyTensorStorage with a nested TensorDict raises error by albertbou92 in https://github.com/pytorch/rl/pull/703
* [BugFix] Fix init_random_frames in A2C example test by vmoens in https://github.com/pytorch/rl/pull/706
* [Formatting] Upgrade formatting libs by vmoens in https://github.com/pytorch/rl/pull/705
* [Doc] Document undefined symbol error with torch version < 1.13 by nickspell in https://github.com/pytorch/rl/pull/707
* [Doc] Tuto integration by vmoens in https://github.com/pytorch/rl/pull/681
* [Quality] Deprecate .ipynb tutos by vmoens in https://github.com/pytorch/rl/pull/710
* [Test] Fix wrong skip message when functorch is installed by vmoens in https://github.com/pytorch/rl/pull/711
* [BugFix, Doc] Clone TensorDict docs into _local_build by tcbegley in https://github.com/pytorch/rl/pull/712
* [Feature] Migrate to `tensordict.nn.TensorDictModule` by tcbegley in https://github.com/pytorch/rl/pull/700
* [Doc] Fix Tutos TODOs by vmoens in https://github.com/pytorch/rl/pull/713
* [BugFix] RoundRobinWriter, possible duplicated code in the extend method by albertbou92 in https://github.com/pytorch/rl/pull/709
* [Feature] Add OptimizerHook by aakhundov in https://github.com/pytorch/rl/pull/716
* [Feature] Support for in-place functionalization by tcbegley in https://github.com/pytorch/rl/pull/714
* [BugFix] Fix TorchRL demo tutorial by vmoens in https://github.com/pytorch/rl/pull/721
* [Docs] Update tutorial links in readme by tcbegley in https://github.com/pytorch/rl/pull/724
* [Feature] Extend PPO loss helper to allow for more customisation by albertbou92 in https://github.com/pytorch/rl/pull/718
* [BugFix] Model maker functions for A2C and PPO fail for discrete action space envs by albertbou92 in https://github.com/pytorch/rl/pull/717
* [Minor] docstrings and setup fixes by vmoens in https://github.com/pytorch/rl/pull/726
* [BugFix] Avoid wrongfully erasing observation keys from specs in CatTensors by vmoens in https://github.com/pytorch/rl/pull/727
* [BugFix] Avoid wrongfully erasing observation keys from tensordict in CatTensors by vmoens in https://github.com/pytorch/rl/pull/729
* [Doc] More doc for data collectors by vmoens in https://github.com/pytorch/rl/pull/732
* [Feature] Port test_fake_tensordict to torchrl by vmoens in https://github.com/pytorch/rl/pull/731
* [Feature] Use ObservationNorm.init_stats for stats computation in example scripts by romainjln in https://github.com/pytorch/rl/pull/715
* [BugFix] init_stats over multiple dimensions by vmoens in https://github.com/pytorch/rl/pull/735
* [Refactor] logger creation in examples by acforvs in https://github.com/pytorch/rl/pull/733
* [Feature] Brax envs by yingchenlin in https://github.com/pytorch/rl/pull/722
* [Refactor] Adopt prototype ProbabilisticTensorDictModule and ProbabilisticTensorDictSequential by tcbegley in https://github.com/pytorch/rl/pull/728
* [Doc] Link to doc in README by vmoens in https://github.com/pytorch/rl/pull/740
* [Feature] Make GAE return a 'value_target' entry by vmoens in https://github.com/pytorch/rl/pull/741
* [Feature] SamplerWithoutReplacement by vmoens in https://github.com/pytorch/rl/pull/742
* [Doc, CI] Update doc workflow to run on PR and only publishes doc on main. by EmGarr in https://github.com/pytorch/rl/pull/745
* [Feature] Better advantage API for higher order derivatives by vmoens in https://github.com/pytorch/rl/pull/744
* [Refactor] Cosmetic improvements to advantage modules by vmoens in https://github.com/pytorch/rl/pull/746
* [BugFix] Fix NoopReset in parallel settings by vmoens in https://github.com/pytorch/rl/pull/747
* [Refactor] Remove env.is_done attribute by vmoens in https://github.com/pytorch/rl/pull/748
* [Refactor] Drop prototype imports by tcbegley in https://github.com/pytorch/rl/pull/738
* [BugFix] Fixes for `speed` branch merge on tensordict by vmoens in https://github.com/pytorch/rl/pull/755
* [BugFix] Fix size-match unsqueeze deprecation by vmoens in https://github.com/pytorch/rl/pull/750
* [Feature] FrameSkipTransform by vmoens in https://github.com/pytorch/rl/pull/749
* [BugFix] Better memory management for collectors by vmoens in https://github.com/pytorch/rl/pull/763
* Minor cleaning in BaseEnv classes by matteobettini in https://github.com/pytorch/rl/pull/767
* Revert "Minor cleaning in BaseEnv classes" by vmoens in https://github.com/pytorch/rl/pull/768
* Cleaning in envs `common.py` by matteobettini in https://github.com/pytorch/rl/pull/769
* Making `_set_seed` abstract by matteobettini in https://github.com/pytorch/rl/pull/770
* [Feature] Remove the Nd*TensorSpec classes by riiswa in https://github.com/pytorch/rl/pull/772
* [BugFix] Reinstantiate custom value key for multioutput value networks by vmoens in https://github.com/pytorch/rl/pull/754
* [Feature] Add Step Counter transform by riiswa in https://github.com/pytorch/rl/pull/756
* [BugFix] Batched environments with non empty batch size by matteobettini in https://github.com/pytorch/rl/pull/774
* Allow undounded boxes creation from gym spaces by matteobettini in https://github.com/pytorch/rl/pull/778
* [BugFix] Doc built cmake error by vmoens in https://github.com/pytorch/rl/pull/780
* [Feature] Lazy TensorClass storage by tcbegley in https://github.com/pytorch/rl/pull/752
* [BugFix] SyncDataCollector init when device and env_device are different by albertbou92 in https://github.com/pytorch/rl/pull/765
* [Feature] RewardSum transform by albertbou92 in https://github.com/pytorch/rl/pull/751
* [BugFix] Fix PPO clip by vmoens in https://github.com/pytorch/rl/pull/786
* [Feature] MultiDiscreteTensorSpec by riiswa in https://github.com/pytorch/rl/pull/783
* [Doc] Doc revamp by vmoens in https://github.com/pytorch/rl/pull/782
* [BugFix] `ParallelEnv` handling of done flag by matteobettini in https://github.com/pytorch/rl/pull/788
* [BugFix] Sorting nested keys by matteobettini in https://github.com/pytorch/rl/pull/787
* [Doc] README index by vmoens in https://github.com/pytorch/rl/pull/791
* Add windows wheel build to CircleCI by yohann-benchetrit in https://github.com/pytorch/rl/pull/759
* [Algorithm] MPPI planner by vmoens in https://github.com/pytorch/rl/pull/701
* [Doc] Better doc links by vmoens in https://github.com/pytorch/rl/pull/795
* [Doc] Missing headers by vmoens in https://github.com/pytorch/rl/pull/796
* [Doc] Knowledge base section by vmoens in https://github.com/pytorch/rl/pull/797
* [Feature] Vmas library wrapper by matteobettini in https://github.com/pytorch/rl/pull/785
* [Doc] Duplicate `HabitatEnv` entry in docs by matteobettini in https://github.com/pytorch/rl/pull/798
* [Feature] MultiDiscreteTensorSpec nvec with several axes by riiswa in https://github.com/pytorch/rl/pull/789
* [Refactor] Graduate Replay Buffer prototype by KamilPiechowiak in https://github.com/pytorch/rl/pull/794
* [BugFix] Solve R3MTransform init problem by vmoens in https://github.com/pytorch/rl/pull/803
* [Refactor] Simplify FlattenObservation default kwargs by vmoens in https://github.com/pytorch/rl/pull/805
* [Format] Fix lint by vmoens in https://github.com/pytorch/rl/pull/811
* [Doc, BugFix] Fix tutos errors by vmoens in https://github.com/pytorch/rl/pull/817
* [Doc] Pretrained models tutorial by vmoens in https://github.com/pytorch/rl/pull/814
* [Doc, BugFix] Fix tensordictmodule tutorial by vmoens in https://github.com/pytorch/rl/pull/819
* [BugFix] Fix MultOneHotDiscreteTensorSpec.is_in by riiswa in https://github.com/pytorch/rl/pull/818
* [Doc] Using R3M with a replay buffer by vmoens in https://github.com/pytorch/rl/pull/820
* [CodeQuality] call all() without making a list by riiswa in https://github.com/pytorch/rl/pull/821
* [BugFix] [Feature] "_reset" flag for env reset by matteobettini in https://github.com/pytorch/rl/pull/800
* [CI] Add unit test workflows for Windows by yohann-benchetrit in https://github.com/pytorch/rl/pull/804
* [BugFix] Fix habitat integration and doc by vmoens in https://github.com/pytorch/rl/pull/812
* [Minor] Better error reporting by vmoens in https://github.com/pytorch/rl/pull/822
* [Minor] Add ninja to deps in toml file by vmoens in https://github.com/pytorch/rl/pull/823
* [BugFix] Device of info specs by vmoens in https://github.com/pytorch/rl/pull/824
* [BugFix] Fix envs specs and info reading by vmoens in https://github.com/pytorch/rl/pull/825
* [Feature] Dtype in vmas tests by matteobettini in https://github.com/pytorch/rl/pull/827
* [BugFix] Fix R3M observation spec transform by vmoens in https://github.com/pytorch/rl/pull/830
* small change to make robandpdx a contributor by robandpdx in https://github.com/pytorch/rl/pull/831
* [Feature] Exclude and select transforms by vmoens in https://github.com/pytorch/rl/pull/832
* [BugFix] Updating Recorder to accomodate "solved" key by ShahRutav in https://github.com/pytorch/rl/pull/833
* [BugFIx] Changed "set_count" set in collectors by matteobettini in https://github.com/pytorch/rl/pull/835
* [Algorithm] Td3 by BY571 in https://github.com/pytorch/rl/pull/684
* [Doc] A Succinct Summary of Reinforcement Learning by vmoens in https://github.com/pytorch/rl/pull/840
* [Feature, BugFix] ObservationNorm keep_dims and RewardSum init by vmoens in https://github.com/pytorch/rl/pull/839
* [BugFix] Improve done checking of collectors by matteobettini in https://github.com/pytorch/rl/pull/838
* [BugFix] Sync with tensordict (meta-tensor deprecation) by vmoens in https://github.com/pytorch/rl/pull/842
* [Feature] Refactor CatFrames using a proper preallocated buffer by vmoens in https://github.com/pytorch/rl/pull/847
* [CI] Add Github-Actions workflows for Windows wheels & nightly-build by yohann-benchetrit in https://github.com/pytorch/rl/pull/837
* [Doc] Fix broken link Dreamer by atonkamanda in https://github.com/pytorch/rl/pull/853
* [BugFix] Loading state_dict on uninitialized CatFrames by vmoens in https://github.com/pytorch/rl/pull/855
* [Refactor] Move loggers to `torchrl.record` by vmoens in https://github.com/pytorch/rl/pull/854
* [Refactor] specs batch size refactoring by vmoens in https://github.com/pytorch/rl/pull/829
* [Feature] Max pool Transform by albertbou92 in https://github.com/pytorch/rl/pull/841
* [Feature] Refactor advantages for continuous batches by vmoens in https://github.com/pytorch/rl/pull/848
* [BugFix, Doc] Minor fix in doc by vmoens in https://github.com/pytorch/rl/pull/858
* [Versioning] Version 0.0.4a by vmoens in https://github.com/pytorch/rl/pull/859
* [Feature] Vmas to device by matteobettini in https://github.com/pytorch/rl/pull/850
* [BugFix] Fix zero-ing from specs in RewardSum by vmoens in https://github.com/pytorch/rl/pull/860
* [Feature] Loading R3M and VIP from ResNet by vmoens in https://github.com/pytorch/rl/pull/863
* [Feature] SAC V2 by vmoens in https://github.com/pytorch/rl/pull/864
* [BugFix] Avoid collision of "step_count" key from transform and collector by vmoens in https://github.com/pytorch/rl/pull/868
* [Refactor] Better init for CatFrames buffers + removing default init values by vmoens in https://github.com/pytorch/rl/pull/874
* [Refactor] Minor refactorings to envs by vmoens in https://github.com/pytorch/rl/pull/872
* [Refactor] Removing inplace transform attribute by vmoens in https://github.com/pytorch/rl/pull/871
* [BugFix] Run checks when creating fake_td by vmoens in https://github.com/pytorch/rl/pull/877
* [Refactor] Box device by vmoens in https://github.com/pytorch/rl/pull/881
* [Feature] Multithreaded env by sgrigory in https://github.com/pytorch/rl/pull/734
* [Refactor] Turn off default advantage normalization in PPO by vmoens in https://github.com/pytorch/rl/pull/887
* [CI] Fix habitat-gym imports by vmoens in https://github.com/pytorch/rl/pull/890
* [CI] Fix cuda versions by vmoens in https://github.com/pytorch/rl/pull/889
* [CI] Fix windows install by vmoens in https://github.com/pytorch/rl/pull/888
* MacOS CPU unit test workflow using GitHub Actions by robandpdx in https://github.com/pytorch/rl/pull/886
* Linux CPU unit test workflow using GitHub Actions by robandpdx in https://github.com/pytorch/rl/pull/826
* [Major, BugFix, Test] Refactor Transforms tests by vmoens in https://github.com/pytorch/rl/pull/878
* [Bugfix] Codecov does not cover multiprocessed tests 879 by kadeng in https://github.com/pytorch/rl/pull/893
* [CI, BugFix] Fix gym related errors by vmoens in https://github.com/pytorch/rl/pull/895
* [WIP] Linux GPU unit test workflow using GitHub Actions by robandpdx in https://github.com/pytorch/rl/pull/885
* [BugFix] Compose cloning fix by vmoens in https://github.com/pytorch/rl/pull/899
* [Feature] Simplifying collector envs by vmoens in https://github.com/pytorch/rl/pull/870
* [CI,Feature] Upgrade to gymnasium by vmoens in https://github.com/pytorch/rl/pull/898
* [Doc] Add record utils to doc by vmoens in https://github.com/pytorch/rl/pull/904
* [Test] Improve exception message match by apbard in https://github.com/pytorch/rl/pull/906
* [BugFix] Dreamer helpers are broken with batched envs by vmoens in https://github.com/pytorch/rl/pull/903
* [Feature] RandomCropTensorDict transform by vmoens in https://github.com/pytorch/rl/pull/908
* [Versioning] Version 0.0.4b by vmoens in https://github.com/pytorch/rl/pull/909

New Contributors
* sladebot made their first contribution in https://github.com/pytorch/rl/pull/435
* rayanht made their first contribution in https://github.com/pytorch/rl/pull/432
* brandonsj made their first contribution in https://github.com/pytorch/rl/pull/475
* ordinskiy made their first contribution in https://github.com/pytorch/rl/pull/485
* JasonMa2016 made their first contribution in https://github.com/pytorch/rl/pull/487
* himjohntang made their first contribution in https://github.com/pytorch/rl/pull/477
* romainjln made their first contribution in https://github.com/pytorch/rl/pull/512
* apbard made their first contribution in https://github.com/pytorch/rl/pull/526
* sicong-huang made their first contribution in https://github.com/pytorch/rl/pull/522
* psolikov made their first contribution in https://github.com/pytorch/rl/pull/566
* jrobine made their first contribution in https://github.com/pytorch/rl/pull/571
* nikhlrao made their first contribution in https://github.com/pytorch/rl/pull/573
* sgrigory made their first contribution in https://github.com/pytorch/rl/pull/580
* jlesuffleur made their first contribution in https://github.com/pytorch/rl/pull/589
* artkorenev made their first contribution in https://github.com/pytorch/rl/pull/593
* paulomarciano made their first contribution in https://github.com/pytorch/rl/pull/614
* hatala91 made their first contribution in https://github.com/pytorch/rl/pull/622
* jgonik made their first contribution in https://github.com/pytorch/rl/pull/608
* adityagandhamal made their first contribution in https://github.com/pytorch/rl/pull/637
* svarolgunes made their first contribution in https://github.com/pytorch/rl/pull/632
* adityagoel4512 made their first contribution in https://github.com/pytorch/rl/pull/615
* jasonfkut made their first contribution in https://github.com/pytorch/rl/pull/641
* sardaankita made their first contribution in https://github.com/pytorch/rl/pull/656
* albertbou92 made their first contribution in https://github.com/pytorch/rl/pull/655
* yingchenlin made their first contribution in https://github.com/pytorch/rl/pull/674
* altre made their first contribution in https://github.com/pytorch/rl/pull/695
* nickspell made their first contribution in https://github.com/pytorch/rl/pull/707
* aakhundov made their first contribution in https://github.com/pytorch/rl/pull/716
* acforvs made their first contribution in https://github.com/pytorch/rl/pull/733
* EmGarr made their first contribution in https://github.com/pytorch/rl/pull/745
* matteobettini made their first contribution in https://github.com/pytorch/rl/pull/767
* riiswa made their first contribution in https://github.com/pytorch/rl/pull/772
* yohann-benchetrit made their first contribution in https://github.com/pytorch/rl/pull/759
* KamilPiechowiak made their first contribution in https://github.com/pytorch/rl/pull/794
* robandpdx made their first contribution in https://github.com/pytorch/rl/pull/831
* ShahRutav made their first contribution in https://github.com/pytorch/rl/pull/833
* BY571 made their first contribution in https://github.com/pytorch/rl/pull/684
* atonkamanda made their first contribution in https://github.com/pytorch/rl/pull/853
* kadeng made their first contribution in https://github.com/pytorch/rl/pull/893

**Full Changelog**: https://github.com/pytorch/rl/compare/v0.0.2a...v0.0.4b