Pyro-ppl

Misc changes

* Updated to PyTorch 1.4.0 and torchvision 0.5.0.
* Changed license from MIT to Apache 2.0 and removed Uber CLA as part of Pyro's move to the [Linux foundation](https://lfai.foundation).

Reparameterization

This release adds a new effect handler and a collection of strategies that reparameterize models to improve geometry. These tools are largely orthogonal to other inference tools in Pyro, and can be used with SVI, MCMC, and other inference algorithms.

* [poutine.reparam()](http://docs.pyro.ai/en/latest/poutine.html#pyro.poutine.handlers.reparam) is a new effect handler that transforms models into other models for which inference may be easier [(Gorinova et al. 2019)](https://arxiv.org/abs/1906.03028).
* [pyro.infer.reparam](http://docs.pyro.ai/en/latest/infer.reparam.html#module-pyro.infer.reparam) is a collection of reparameterization strategies following a standard [Reparam](http://docs.pyro.ai/en/latest/infer.reparam.html#pyro.infer.reparam.reparam.Reparam) interface:
* [Decentering transforms](http://docs.pyro.ai/en/latest/infer.reparam.html#module-pyro.infer.reparam.loc_scale) for location-scale families [(Gorinova et al. 2019)](https://arxiv.org/abs/1906.03028).
* [Transform unwrapping](http://docs.pyro.ai/en/latest/infer.reparam.html#module-pyro.infer.reparam.transform) to deconstruct `TransformedDistribution`s.
* [Discrete Cosine transforms](http://docs.pyro.ai/en/latest/infer.reparam.html#module-pyro.infer.reparam.discrete_cosine) for frequency-domain parameterizations (useful for inference in time series).
* Auxiliary variable methods for [Levy Stable](http://docs.pyro.ai/en/latest/infer.reparam.html#module-pyro.infer.reparam.stable) and [StudentT](http://docs.pyro.ai/en/latest/infer.reparam.html#module-pyro.infer.reparam.studentt) distributions.
* [Linear Hidden Markov Model](http://docs.pyro.ai/en/latest/infer.reparam.html#module-pyro.infer.reparam.hmm) reparameterization, allowing a range of non-Gaussian HMMs to be treated as conditional Gaussian processes.
* [Neural Transport](http://docs.pyro.ai/en/latest/infer.reparam.html#module-pyro.infer.reparam.neutra) uses SVI to learn the geometry of a model before drawing samples using HMC [(Hoffman et al. 2019)](https://arxiv.org/abs/1903.03704).
* A [tutorial](http://pyro.ai/examples/stable.html) on inference with [Levy Stable](http://docs.pyro.ai/en/latest/distributions.html#stable) distrubutions, demonstrating [StableReparam](http://docs.pyro.ai/en/latest/infer.reparam.html#pyro.infer.reparam.stable.StableReparam), [DiscreteCosineReparam](http://docs.pyro.ai/en/latest/infer.reparam.html#pyro.infer.reparam.discrete_cosine.DiscreteCosineReparam), and [EnergyDistance](http://docs.pyro.ai/en/latest/inference_algos.html#pyro.infer.energy_distance.EnergyDistance).


Other new features
* A [tutorial](http://pyro.ai/examples/dirichlet_process_mixture.html) on Dirichlet process mixture modeling, contributed by m-k-S
* Added a [LinearHMM](http://docs.pyro.ai/en/latest/distributions.html#linearhmm) distribution with an `.rsample()` method. This supports non-Gaussian noise such as [Levy Stable](http://docs.pyro.ai/en/latest/distributions.html#stable) and [StudentT](http://docs.pyro.ai/en/latest/distributions.html#studentt), but requires [reparameterization](http://docs.pyro.ai/en/latest/infer.reparam.html) for inference.
* Implemented a [GaussianHMM.rsample()](http://docs.pyro.ai/en/latest/distributions.html#pyro.distributions.GaussianHMM.rsample) method for drawing joint samples from a linear-Gaussian HMM.
* Added a [LowerCholeskyAffine](http://docs.pyro.ai/en/latest/distributions.html#lowercholeskyaffine) transform.
* [2264](https://github.com/pyro-ppl/pyro/pull/2264) improves speed and numerical stability of `MultivariateNormal` conversion from `scale_tril` to `precision`.

Bug fixes

- [2263](https://github.com/pyro-ppl/pyro/pull/2263) fixes MCMC api to allow implementations other than HMC and NUTS.
- [2244](https://github.com/pyro-ppl/pyro/pull/2244) fixes an `event_dim` issue in `ConditionedPlanar` flow.
- [2243](https://github.com/pyro-ppl/pyro/pull/2243) fixes a bug in `AffineCoupling`.
- [2227](https://github.com/pyro-ppl/pyro/issues/2227) fixes device placement of the `MultivariateStudentT.df` param.
- [2226](https://github.com/pyro-ppl/pyro/pull/2226) fixes an edge case bug in discrete enumeration.

New Features
- [pyro.infer.ReweightedWakeSleep](http://docs.pyro.ai/en/latest/inference_algos.html#pyro.infer.rws.ReweightedWakeSleep) implements the Reweighted Wake Sleep algorithm [(Le et al. 2019)](https://arxiv.org/abs/1805.10469). Contributed by [Siddharth Narayanaswamy](https://github.com/iffsid) and [Tuan Anh Le](https://github.com/tuananhle7).
- [pyro.infer.TraceTMC_ELBO](http://docs.pyro.ai/en/latest/inference_algos.html#pyro.infer.tracetmc_elbo.TraceTMC_ELBO) implements the Tensor Monte Carlo marginal likelihood estimator [(Aitchinson 2019)](https://papers.nips.cc/paper/8936-tensor-monte-carlo-particle-methods-for-the-gpu-era.pdf), a generalization of the importance-weighted autoencoder objective.
- [pyro.infer.EnergyDistance](http://docs.pyro.ai/en/latest/inference_algos.html#pyro.infer.energy_distance.EnergyDistance) implements a likelihood-free inference algorithm based on Szekely's energy statistics, a multidimensional generalization of CRPS [(Gneiting & Raftery 2007)](https://www.stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf).
- [pyro.contrib.cevae]() implements the Causal Inference VAE of [(Louizos et al. 2017)](http://papers.nips.cc/paper/7223-causal-effect-inference-with-deep-latent-variable-models.pdf). See [examples/contrib/cevae/synthetic.py](https://github.com/pyro-ppl/pyro/blob/dev/examples/contrib/cevae/synthetic.py) for an end-to-end usage example.
- [pyro.deterministic](http://docs.pyro.ai/en/latest/primitives.html#pyro.deterministic) primitive to record deterministic values in the trace.
- [pyro.nn.to_pyro_module_()](http://docs.pyro.ai/en/latest/nn.html#pyro.nn.module.to_pyro_module_) recursively converts an regular `nn.Module` to a [PyroModule](http://docs.pyro.ai/en/latest/nn.html#pyro.nn.module.PyroModule) in-place.

- A default implementation for [Distribution.expand()](http://docs.pyro.ai/en/latest/distributions.html#pyro.distributions.torch_distribution.TorchDistributionMixin.expand) that is available to all Pyro distributions that subclass from `TorchDistribution`, making it easier to create custom distributions.

New distributions and transforms
- [MultivariateStudentT](http://docs.pyro.ai/en/latest/distributions.html#multivariatestudentt) is a heavy-tailed multivariate distribution.
- [Stable](http://docs.pyro.ai/en/latest/distributions.html#stable) implements a Lévy α-stable distribution with reparametrized `.rsample()` method but no `.log_prob()`. This can be fit using [EnergyDistance](http://docs.pyro.ai/en/latest/inference_algos.html#pyro.infer.energy_distance.EnergyDistance) inference.
- [ZeroInflatedNegativeBinomial](http://docs.pyro.ai/en/latest/distributions.html#zeroinflatednegativebinomial) is a distribution for count data.
- [LowerCholeskyAffine](http://docs.pyro.ai/en/latest/distributions.html#lowercholeskyaffine) is a multivariate affine transform.


Other Changes / Bug Fixes
- `pyro.util.save_visualization` has been deprecated, and dependency on `graphviz` is removed.
- [2197](https://github.com/pyro-ppl/pyro/pull/2197) fixed a naming bug in [PyroModule](http://docs.pyro.ai/en/latest/nn.html#pyro.nn.module.PyroModule) that affected mutliple sub-PyroModules with conflicting names.
- [2192](https://github.com/pyro-ppl/pyro/pull/2192) Bug fix in Planar normalizing flow implementation
- [2188](https://github.com/pyro-ppl/pyro/issues/2188) Make error messages for incorrect arguments to effect handlers more informative

The objective of this release is to stabilize Pyro's interface and thereby make it safer to build high level components on top of Pyro.

Stability statement
- Behavior of documented APIs will remain stable across minor releases, except for bug fixes and features marked EXPERIMENTAL or DEPRECATED.
- Serialization formats will remain stable across patch releases, but may change across minor releases (e.g. if you save a model in 1.0.0, it will be safe to load it in 1.0.1, but not in 1.1.0).
- Undocumented APIs, features marked EXPERIMENTAL or DEPRECATED, and anything in`pyro.contrib` may change at any time (though we aim for stability).
- All deprecated features throw a `FutureWarning` and specify possible work-arounds. Features marked as deprecated will not be maintained, and are likely to be removed in a future release.
- If you want more stability for a particular feature, [contribute](https://github.com/pyro-ppl/pyro/blob/dev/CONTRIBUTING.md) a unit test.

New features
- [pyro.infer.Predictive](http://docs.pyro.ai/en/1.0.0/inference_algos.html#pyro.infer.predictive.Predictive) is a new utility for serving models, supporting jit tracing and serialization.
- [pyro.distributions.transforms](http://docs.pyro.ai/en/1.0.0/distributions.html#transforms) has many new transforms, and includes helper functions to easily create a variety of normalizing flows. The transforms library has also been reorganized.
- [pyro.contrib.timeseries](http://docs.pyro.ai/en/1.0.0/contrib.timeseries.html) is an experimental new module with fast Gaussian Process inference for univariate and multivariate time series and state space models.
- [pyro.nn.PyroModule](http://docs.pyro.ai/en/1.0.0/nn.html#module-pyro.nn.module) is an experimental new interface that adds Pyro effects to an `nn.Module`. `PyroModule` is already used internally by `AutoGuide`, `EasyGuide` `pyro.contrib.gp`, `pyro.contrib.timeseries`, and elsewhere.
- [FoldedDistribution]() is a new distribution factory, essentially equivalent to `TransformedDistribution(-, AbsTransform())` but providing a `.log_prob()` method.
- A new tutorial illustrates the usage of [pyro.contrib.oed](http://docs.pyro.ai/en/1.0.0/contrib.oed.html) in the context of adaptive election polling.

Breaking changes

- Autoguides have slightly changed interfaces:
- `AutoGuide` and `EasyGuide` are now `nn.Module`s and can be serialized separately from the param store. This enables serving via [torch.jit.trace_module](https://pytorch.org/docs/1.0.0/jit.html#torch.jit.trace_module).
- The `Auto*Normal` family of autoguides now have `init_scale` arguments, and `init_loc_fn` has better support. Autoguides no longer support initialization by writing directly to the param store.
- Many transforms have been renamed to enforce a consistent interface, such as the renaming of `InverseAutoregressiveFlow` to `AffineAutoregressive`.
- `pyro.generic` has been moved to a separate project [pyroapi](https://github.com/pyro-ppl/pyro-api).
- [poutine.do](http://docs.pyro.ai/en/1.0.0/poutine.html#pyro.poutine.do) has slightly changed semantics to follow Single World Intervention Graph semantics.
- `pyro.contrib.glmm` has been moved to `pyro.contrib.oed.glmm` and will eventually be replaced by [BRMP](https://github.com/pyro-ppl/brmp).
- Existing `DeprecationWarning`s have been promoted to `FutureWarning`s.

Deprecated features
- `pyro.random_module`: The `pyro.random_module` primitive has been deprecated in favor of [PyroModule](http://docs.pyro.ai/en/1.0.0/nn.html#pyro.nn.module.PyroModule) which can be used to create Bayesian modules from `torch.nn.Module` instances.
- `SVI.run`: The `SVI.run` method is deprecated and users are encouraged to use the [.step](http://docs.pyro.ai/en/1.0.0/inference_algos.html#pyro.infer.svi.SVI.step) method directly to run inference. For drawing samples from the posterior distribution, we recommend using the [Predictive](http://docs.pyro.ai/en/1.0.0/inference_algos.html#module-pyro.infer.predictive.Predictive) utility class, or directly by using the `trace` and `replay` effect handlers.
- `TracePredictive`: The `TracePredictive` class is deprecated in favor of [Predictive](http://docs.pyro.ai/en/1.0.0/inference_algos.html#module-pyro.infer.predictive.Predictive), that can be used to gather samples from the posterior and predictive distributions in SVI and MCMC.
- `mcmc.predictive`: This utility function has been absorbed into the more general [Predictive](http://docs.pyro.ai/en/1.0.0/inference_algos.html#module-pyro.infer.predictive.Predictive) class.

Patches `0.5.0` with the following bug fixes:
- Removes f-string which is only supported in Python 3.6+, so that Python 3.5 is supported.
- Fix incompatibility with recent tqdm releases which make multiple bars not work in the notebook environment (for MCMC with multiple chains).

New features

- [pyro.factor](http://docs.pyro.ai/en/stable/primitives.html#pyro.factor) to add arbitrary log probability factor to a probabilistic model.
- Conditional MADE Autoregressive Network available in [pyro.nn](http://docs.pyro.ai/en/stable/nn.html#pyro.nn.auto_reg_nn.ConditionalAutoRegressiveNN).
- Tutorial on [adaptive experiment design](http://pyro.ai/examples/working_memory.html) for studying working memory.
- KL divergence for `Delta` and `Independent` distributions.
- A fast `n log(n)` implementation of the [Continuous Ranked Probability Score (CRPS)](https://www.stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf) for sample sets: [pyro.ops.stats.crps_empirical](http://docs.pyro.ai/en/dev/ops.html#pyro.ops.stats.crps_empirical)



Code changes and bug fixes

- Moved `pyro.generic` to a separate [pyro-api](https://github.com/pyro-ppl/pyro-api) package.
- Minor changes to ensure compatibility with [pyro-api](https://github.com/pyro-ppl/pyro-api), a generic modeling and inference API for dispatch to different Pyro backends.
- Improve numerical stability of MixtureOfDiagonals distribution using `logsumexp` operation.
- Improved U-Turn check condition in NUTS for better sampling efficiency.
- Reorganized `constraints` and `transforms` module to match `torch.distributions`.
- Fixed AutoGuide intitialization stragtegies, resolving a bug in `init_to_median`.

**New Features:**

- [*HMM.filter()](http://docs.pyro.ai/en/dev/distributions.html#pyro.distributions.DiscreteHMM.filter) methods for forecasting.
- Support for Independent(Normal) observations in GaussianHMM.

**Fixes:**
- Fix for HMC / NUTS to handle errors arising from numerical issues when computing Cholesky decomposition.