Garage

Added
- Integration tests which cover all example scripts (
[1078](https://github.com/rlworkgroup/garage/pull/1078),
[1090](https://github.com/rlworkgroup/garage/pull/1090))
- Deterministic mode support for PyTorch ([1068](https://github.com/rlworkgroup/garage/pull/1068))
- Install script support for macOS 10.15.1 ([1051](https://github.com/rlworkgroup/garage/pull/1051))
- PyTorch modules now support either functions or modules for specifying their non-linearities ([1038](https://github.com/rlworkgroup/garage/pull/1038))

Fixed
- Errors in the documentation on implementing new algorithms ([1074](https://github.com/rlworkgroup/garage/pull/1074))
- Broken example for DDPG+HER in TensorFlow ([1070](https://github.com/rlworkgroup/garage/pull/1070))
- Error in the documentation for using garage with conda ([1066](https://github.com/rlworkgroup/garage/pull/1066))
- Broken pickling of environment wrappers ([1061](https://github.com/rlworkgroup/garage/pull/1061))
- `garage.torch` was not included in the PyPI distribution ([1037](https://github.com/rlworkgroup/garage/pull/1037))
- A few broken examples for `garage.tf` ([1032](https://github.com/rlworkgroup/garage/pull/1032))

Added
- Algorithms
* (D)DQN in TensorFlow ([582](https://github.com/rlworkgroup/garage/pull/582))
* Maximum-entropy and entropy regularization for policy gradient algorithms in
TensorFlow ([632](https://github.com/rlworkgroup/garage/pull/632))
* DDPG in PyTorch ([815](https://github.com/rlworkgroup/garage/pull/815))
* VPG (i.e. policy gradients) in PyTorch ([883](https://github.com/rlworkgroup/garage/pull/883))
* TD3 in TensorFlow ([458](https://github.com/rlworkgroup/garage/pull/458))
- APIs
* Runner API for executing experiments and `LocalRunner` implementation for
executing them on the local machine (
[541](https://github.com/rlworkgroup/garage/pull/541),
[593](https://github.com/rlworkgroup/garage/pull/593),
[602](https://github.com/rlworkgroup/garage/pull/602),
[816](https://github.com/rlworkgroup/garage/pull/816),
)
* New Logger API, provided by a sister project [dowel](https://github.com/rlworkgroup/dowel) ([#464](https://github.com/rlworkgroup/garage/pull/464), [#660](https://github.com/rlworkgroup/garage/pull/660))
- Environment wrappers for pixel-based algorithms, especially DQN ([556](https://github.com/rlworkgroup/garage/pull/556))
- Example for how to use garage with Google Colab ([476](https://github.com/rlworkgroup/garage/pull/476))
- Advantage normalization for recurrent policies in TF ([626](https://github.com/rlworkgroup/garage/pull/626))
- PyTorch support ([725](https://github.com/rlworkgroup/garage/pull/725), [#764](https://github.com/rlworkgroup/garage/pull/764))
- Autogenerated API docs on [garage.readthedocs.io](https://garage.readthedocs.io/en/latest/py-modindex.html) ([#802](https://github.com/rlworkgroup/garage/pull/802))
- GPU version of the pip package ([834](https://github.com/rlworkgroup/garage/pull/834))
- PathBuffer, a trajectory-oriented replay buffer ([838](https://github.com/rlworkgroup/garage/pull/838))
- RaySampler, a remote and/or multiprocess sampler based on ray ([793](https://github.com/rlworkgroup/garage/pull/793))
- Garage is now distributed on PyPI ([870](https://github.com/rlworkgroup/garage/pull/870))
- `rollout` option to only sample policies deterministically ([896](https://github.com/rlworkgroup/garage/pull/896))
- MultiEnvWrapper, which wraps multiple `gym.Env` environments into a discrete
multi-task environment ([946](https://github.com/rlworkgroup/garage/pull/946))

Changed
- Optimized Dockerfiles for fast rebuilds ([557](https://github.com/rlworkgroup/garage/pull/557))
- Random seed APIs moved to `garage.experiment.deterministic` ([578](https://github.com/rlworkgroup/garage/pull/578))
- Experiment wrapper script is now an ordinary module ([586](https://github.com/rlworkgroup/garage/pull/586))
- numpy-based modules and algorithms moved to `garage.np` ([604](https://github.com/rlworkgroup/garage/pull/604))
- Algorithm constructors now use `EnvSpec` rather than `gym.Env` ([575](https://github.com/rlworkgroup/garage/pull/575))
- Snapshotter API moved from `garage.logger` to `garage.experiment` ([658](https://github.com/rlworkgroup/garage/pull/658))
- Moved `process_samples` API from the Sampler to algorithms ([652](https://github.com/rlworkgroup/garage/pull/652))
- Updated Snapshotter API ([699](https://github.com/rlworkgroup/garage/pull/699))
- Updated Resume API ([777](https://github.com/rlworkgroup/garage/pull/777))
- All algorithms now have a default sampler ([832](https://github.com/rlworkgroup/garage/pull/832))
- Experiment lauchers now require an explicit `snapshot_config` to their
`run_task` function ([860](https://github.com/rlworkgroup/garage/pull/860))
- Various samplers moved from `garage.tf.sampler` to `garage.sampler` ([836](https://github.com/rlworkgroup/garage/pull/836),
[840](https://github.com/rlworkgroup/garage/pull/840))
- Dockerfiles are now based on Ubuntu 18.04 LTS by default ([763](https://github.com/rlworkgroup/garage/pull/763))
- `dm_control` is now an optional dependency, installed using the extra
`garage[dm_control]` ([828](https://github.com/rlworkgroup/garage/pull/828))
- MuJoCo is now an optional dependency, installed using the extra
`garage[mujoco]` ([848](https://github.com/rlworkgroup/garage/pull/828))
- Samplers no longer flatten observations and actions ([930](https://github.com/rlworkgroup/garage/pull/930),
[938](https://github.com/rlworkgroup/garage/pull/938),
[967](https://github.com/rlworkgroup/garage/pull/967))
- Implementations, tests, and benchmarks for all TensorFlow primitives, which
are now based on `garage.tf.Model` ([574](https://github.com/rlworkgroup/garage/pull/574),
[606](https://github.com/rlworkgroup/garage/pull/606),
[615](https://github.com/rlworkgroup/garage/pull/615),
[616](https://github.com/rlworkgroup/garage/pull/616),
[618](https://github.com/rlworkgroup/garage/pull/618),
[641](https://github.com/rlworkgroup/garage/pull/641),
[642](https://github.com/rlworkgroup/garage/pull/642),
[656](https://github.com/rlworkgroup/garage/pull/656),
[662](https://github.com/rlworkgroup/garage/pull/662),
[668](https://github.com/rlworkgroup/garage/pull/668),
[672](https://github.com/rlworkgroup/garage/pull/672),
[677](https://github.com/rlworkgroup/garage/pull/677),
[730](https://github.com/rlworkgroup/garage/pull/730),
[722](https://github.com/rlworkgroup/garage/pull/722),
[765](https://github.com/rlworkgroup/garage/pull/765),
[855](https://github.com/rlworkgroup/garage/pull/855),
[878](https://github.com/rlworkgroup/garage/pull/878),
[888](https://github.com/rlworkgroup/garage/pull/888),
[898](https://github.com/rlworkgroup/garage/pull/898),
[892](https://github.com/rlworkgroup/garage/pull/892),
[897](https://github.com/rlworkgroup/garage/pull/897),
[893](https://github.com/rlworkgroup/garage/pull/893),
[890](https://github.com/rlworkgroup/garage/pull/890),
[903](https://github.com/rlworkgroup/garage/pull/903),
[916](https://github.com/rlworkgroup/garage/pull/916),
[891](https://github.com/rlworkgroup/garage/pull/891),
[922](https://github.com/rlworkgroup/garage/pull/922),
[931](https://github.com/rlworkgroup/garage/pull/931),
[933](https://github.com/rlworkgroup/garage/pull/933),
[906](https://github.com/rlworkgroup/garage/pull/906),
[945](https://github.com/rlworkgroup/garage/pull/945),
[944](https://github.com/rlworkgroup/garage/pull/944),
[943](https://github.com/rlworkgroup/garage/pull/943),
[972](https://github.com/rlworkgroup/garage/pull/972))
- Dependency upgrades:
* mujoco-py to 2.0 ([661](https://github.com/rlworkgroup/garage/pull/661))
* gym to 0.12.4 ([661](https://github.com/rlworkgroup/garage/pull/661))
* dm_control to 7a36377879c57777e5d5b4da5aae2cd2a29b607a ([661](https://github.com/rlworkgroup/garage/pull/661))
* akro to 0.0.6 ([796](https://github.com/rlworkgroup/garage/pull/796))
* pycma to 2.7.0 ([861](https://github.com/rlworkgroup/garage/pull/861))
* tensorflow to 1.15 ([953](https://github.com/rlworkgroup/garage/pull/953))
* pytorch to 1.3.0 ([952](https://github.com/rlworkgroup/garage/pull/952))

Removed
- `garage.misc.autoargs`, a tool for decorating classes with autogenerated
command-line arguments ([573](https://github.com/rlworkgroup/garage/pull/573))
- `garage.misc.ext`, a module with several unrelated utilities ([578](https://github.com/rlworkgroup/garage/pull/578))
- `config_personal.py` module, replaced by environment variables where relevant ([578](https://github.com/rlworkgroup/garage/pull/578), [#747](https://github.com/rlworkgroup/garage/pull/747))
- `contrib.rllab_hyperopt`, an experimental module for using `hyperopt` to tune
hyperparameters ([684](https://github.com/rlworkgroup/garage/pull/684))
- `contrib.bichenchao`, a module of example launchers ([683](https://github.com/rlworkgroup/garage/pull/683))
- `contrib.alexbeloi`, a module with an importance-sampling sampler and examples
(there were merged into garage) ([717](https://github.com/rlworkgroup/garage/pull/717))
- EC2 cluster documentation and examples ([835](https://github.com/rlworkgroup/garage/pull/835))
- `DeterministicMLPPolicy`, because it duplicated `ContinuousMLPPolicy` ([929](https://github.com/rlworkgroup/garage/pull/929))
- `garage.tf.layers`, a custom high-level neural network definition API, was replaced by `garage.tf.models` ([939](https://github.com/rlworkgroup/garage/pull/939))
- `Parameterized`, which was replaced by `garage.tf.Model` ([942](https://github.com/rlworkgroup/garage/pull/942))
- `garage.misc.overrides`, whose features are no longer needed due proper ABC
support in Python 3 and sphinx-autodoc ([974](https://github.com/rlworkgroup/garage/pull/942))
- `Serializable`, which became a maintainability burden and has now been
replaced by regular pickle protocol (`__getstate__`/`__setstate__`)
implementations, where necessary ([982](https://github.com/rlworkgroup/garage/pull/982))
- `garage.misc.special`, a library of mostly-unused math subroutines ([986](https://github.com/rlworkgroup/garage/pull/986))
- `garage.envs.util`, superceded by features in [akro](https://github.com/rlworkgroup/akro) ([#986](https://github.com/rlworkgroup/garage/pull/986))
- `garage.misc.console`, a library of mostly-unused helper functions for writing
shell scripts ([988](https://github.com/rlworkgroup/garage/pull/988))

Fixed
- Bug in `ReplayBuffer` [554](https://github.com/rlworkgroup/garage/pull/554)
- Bug in `setup_linux.sh` [560](https://github.com/rlworkgroup/garage/pull/560)
- Bug in `examples/sim_policy.py` ([691](https://github.com/rlworkgroup/garage/pull/691))
- Bug in `FiniteDifferenceHvp` ([745](https://github.com/rlworkgroup/garage/pull/745))
- Determinism bug for some samplers ([880](https://github.com/rlworkgroup/garage/pull/880))
- `use_gpu` in the experiment runner ([918](https://github.com/rlworkgroup/garage/pull/918))

Fixed
- Bug in entropy regularization in TensorFlow PPO/TRPO ([579](https://github.com/rlworkgroup/garage/pull/579))
- Bug in which advantage normalization was broken for recurrent policies ([626](https://github.com/rlworkgroup/garage/pull/626))
- Bug in `examples/sim_policy.py` ([691](https://github.com/rlworkgroup/garage/pull/691))
- Bug in `FiniteDifferenceHvp` ([745](https://github.com/rlworkgroup/garage/pull/745))

Fixed
- Fix overhead in GaussianMLPRegressor by optionally creating assign operations ([622](https://github.com/rlworkgroup/garage/pull/622))

Added
- Epsilon-greedy exploration strategy, DiscreteMLPModel, and
QFunctionDerivedPolicy (all needed by DQN)
- Base Model class for TensorFlow-based primitives
- Dump plots generated with matplotlib to TensorBoard
- Relative Entropy Policy Search (REPS) algorithm
- GaussianConvBaseline and GaussianConvRegressor primitives
- New Dockerfiles, docker-compose files, and Makefiles for running garage using
Docker
- Vanilla policy gradient loss to NPO
- Truncated Natural Policy Gradient (TNPG) algorithm for TensorFlow
- Episodic Reward Weighted Regression (ERWR) algorithm for TensorFlow
- gym.Env wrappers used for pixel environments
- Convolutional Neural Network primitive

Changed
- Move dependencies from environment.yml to setup.py
- Update dependencies:
- tensorflow-probability to 0.5.x
- dm_control to commit 92f9913
- TensorFlow to 1.12
- MuJoCo to 2.0
- gym to 0.10.11
- Move dm_control tests into the unit test tree
- Use GitHub standard .gitignore
- Improve the implementation of RandomizedEnv (Dynamics Randomization)
- Decouple TensorBoard from the logger
- Move files from garage/misc/instrument to garage/experiment
- setup.py to be canonical in format and use automatic versioning

Removed
- Move some garage subpackages into their own repositories:
- garage.viskit to [rlworkgroup/viskit](https://github.com/rlworkgroup/viskit)
- garage.spaces to [rlworkgroup/akro](https://github.com/rlworkgroup/akro)
- Remove Theano backend, algorithms, and dependencies
- Custom environments which duplicated [openai/gym](https://github.com/openai/gym)
- Some dead files from garage/misc (meta.py and viewer2d.py)
- Remove all code coverage tracking providers except CodeCov

Fixed
- Clean up warnings in the test suite
- Pickling bug in GaussianMLPolicyWithModel
- Namescope in LbfgsOptimizer
- Correctly sample paths in OffPolicyVectorizedSampler
- Implementation bugs in tf/VPG
- Bug when importing Box
- Bug in test_benchmark_her

Added
- PPO and DDPG for the TensorFlow branch
- HER for DDPG
- Recurrent Neural Network policy support for NPO, PPO and TRPO
- Base class for ReplayBuffer, and two implementations: SimpleReplayBuffer
and HerReplayBuffer
- Sampler classes OffPolicyVectorizedSampler and OnPolicyVectorizedSampler
- Base class for offline policies OffPolicyRLAlgorithm
- Benchmark tests for TRPO, PPO and DDPG to compare their performance with
those produced by OpenAI Baselines
- Dynamics randomization for MuJoCo environments
- Support for dm_control environments
- DictSpace support for garage environments
- PEP8 checks enforced in the codebase
- Support for Python imports: maintain correct ordering and remove unused
imports or import errors
- Test on TravisCI using Docker images for managing dependencies
- Testing code reorganized
- Code Coverage measurement with codecov
- Pre-commit hooks to enforce PEP8 and to verify imports and commit messages,
which are also applied in the Travis CI verification
- Docstring verification for added files that are not in the test branch or
moved files
- TensorBoard support for all key-value/log_tabular calls, plus support for
logging distributions
- Variable and name scope for symbolic operations in TensorFlow
- Top-level base Space class for garage
- Asynchronous plotting for Theano and Tensorflow
- GPU support for Theano

Changed
- Rename rllab to garage, including all the rllab references in the packages
and modules inside the project
- Rename run_experiment_lite to run_experiment
- The file cma_es_lib.py was replaced by the pycma library available on PyPI
- Move the contrib package to garage.contrib
- Move Theano-dependent code to garage.theano
- Move all code from sandbox.rocky.tf to garage.tf
- Update several dependencies, mainly:
- Python to 3.6.6
- TensorFlow to 1.9
- Theano to 1.0.2
- mujoco-py to 1.50.1
- gym to 0.10.8
- Transfer various dependencies from conda to pip
- Separate example script files in the Theano and TensorFlow branch
- Update LICENSE, CONTRIBUTING.md and .gitignore
- Use convenience imports, that is, import classes and functions that share the
same or similar name to its module in the corresponding `__init__.py` file of
their package
- Replace ProxyEnv with gym.Wrapper
- Update installation scripts for Linux and macOS

Removed
- All unused imports in the Python files
- Unused packages from environment.yml
- The files under rllab.mujoco_py were removed to use the pip release instead
- Empty `__init__.py` files
- The environment class defined by rllab.envs.Env was not imported to garage
and the environment defined by gym.Env is used now

Fixed
- Sleeping processes produced by the parallel sampler. NOTE: although the
frequency of this issue has been reduced, our tests in TravisCI occasionally
detect the issue and currently it seems to be an issue with re-entrant locks
and multiprocessing in Python.