Gpytorch

Starting with this release, the `LazyTensor` functionality of GPyTorch has been pulled out into its own separate Python package, called [linear_operator](https://github.com/cornellius-gp/linear_operator). Most users won't notice the difference (at the moment), but power users will notice a few changes.

If you have your own custom LazyTensor code, don't worry: this release is backwards compatible! However, you'll see a lot of annoying deprecation warnings 😄

LazyTensor -> LinearOperator
- All `gpytorch.lazy.*LazyTensor` classes now live in the `linear_operator` repo, and are now called `linear_operator.operator.*LinearOperator`.
- For example, `gpytorch.lazy.DiagLazyTensor` is now `linear_operator.operators.DiagLinearOperator`
- The only major naming change: `NonLazyTensor` is now `DenseLinearOperator`
- `gpytorch.lazify` and `gpytorch.delazify` are now `linear_operator.to_linear_operator` and `linear_operator.to_dense`, respectively.
- The `_quad_form_derivative` method has been renamed to `_bilinear_derivative` (a more accurate name!)
- `LinearOperator` method names now reflect their corresponding PyTorch names. This includes:
- `add_diag` -> `add_diagonal`
- `diag` -> `diagonal`
- `inv_matmul` -> `solve`
- `symeig` -> `eigh` and `eigvalsh`
- `LinearOperator` now has the `mT` property

__torch_function__ functionality

LinearOperators are now compatible with the torch api! For example, the following code works:

python
diag_linear_op = linear_operator.operators.DiagLinearOperator(torch.randn(10))
torch.matmul(diag_linear_op, torch.randn(10, 2)) returns a torch.Tensor!


Other files that have moved:

- `gpytorch.functions` - all of the core functions used by LazyTensors now live in the LinearOperator repo. This includes: diagonalization, dsmm, inv_quad, inv_quad_logdet, matmul, pivoted_cholesky, root_decomposition, solve (formally inv_matmul), and sqrt_inv_matmul
- `gpytorch.utils` - a few have moved to the LinearOperator repo. This includes: broadcasting, cholesky, contour_intergral_quad, getitem, interpolation, lanczos, linear_cg, minres, permutation, stable_pinverse, qr, sparse, SothcasticLQ, and toeplitz.

**Full Changelog**: https://github.com/cornellius-gp/gpytorch/compare/v1.8.1...v1.9.0

Bug fixes

* MultitaskMultivariateNormal: fix tensor reshape issue by adamjstewart in https://github.com/cornellius-gp/gpytorch/pull/2081
* Fix handling of prior terms in ExactMarginalLogLikelihood by saitcakmak in https://github.com/cornellius-gp/gpytorch/pull/2039
* Fix bug in preconditioned KISS-GP / Hadamard Multitask GPs by gpleiss in https://github.com/cornellius-gp/gpytorch/pull/2090
* Add constant_constraint to ConstantMean by gpleiss in https://github.com/cornellius-gp/gpytorch/pull/2082

New Contributors
* mone27 made their first contribution in https://github.com/cornellius-gp/gpytorch/pull/2076

**Full Changelog**: https://github.com/cornellius-gp/gpytorch/compare/v1.8.0...v1.8.1

Major Features
* add variational nearest neighbor GP by LuhuanWu in https://github.com/cornellius-gp/gpytorch/pull/2026

New Contributors
* adamjstewart made their first contribution in https://github.com/cornellius-gp/gpytorch/pull/2061
* m-julian made their first contribution in https://github.com/cornellius-gp/gpytorch/pull/2054
* ngam made their first contribution in https://github.com/cornellius-gp/gpytorch/pull/2059
* LuhuanWu made their first contribution in https://github.com/cornellius-gp/gpytorch/pull/2026

**Full Changelog**: https://github.com/cornellius-gp/gpytorch/compare/v1.7.0...v1.8.0

**Important**: This release requires Python 3.7 (up from 3.6) and PyTorch 1.10 (up from 1.9)

New Features
- gpytorch.metrics module offers easy-to-use metrics for GP performance.(1870) This includes:
- gpytorch.metrics.mean_absolute_error
- gpytorch.metrics.mean_squared_error
- gpytorch.metrics.mean_standardized_log_loss
- gpytorch.metrics.negative_log_predictive_density
- gpytorch.metrics.quantile_coverage_error
- Large scale inference (using matrix-multiplication techniques) now implements the variance reduction scheme described in [Wenger et al., ICML 2022](https://arxiv.org/abs/2107.00243). (#1836)
- This makes it possible to use LBFGS, or other line search based optimization techniques, with large scale (exact) GP hyperparameter optimization.
- Variational GP models support online updates (i.e. “fantasizing new models). (1874)
- This utilizes the method described in [Maddox et al., NeurIPS 2021](https://papers.nips.cc/paper/2021/hash/325eaeac5bef34937cfdc1bd73034d17-Abstract.html)
- Improvements to gpytorch.priors
- New HalfCauchyPrior (1961)
- LKJPrior now supports sampling (1737)

Minor Features
- Add LeaveOneOutPseudoLikelihood for hyperparameter optimization (1989)
- The PeriodicKernel now supports ARD lengthscales/periods (1919)
- LazyTensors (A) can now be matrix multiplied with tensors (B) from the left hand side (i.e. B x A) (1932)
- Maximum Cholesky retries can be controlled through a setting (1861)
- Kernels, means, and likelihoods can be pickled (1876)
- Minimum variance for FixedNoiseGaussianLikelihood can be set with a context manager (2009)

Bug Fixes
- Fix backpropagation issues with KeOps kernels (1904)
- Fix broadcasting issues with lazily evaluated kernels (1971)
- Fix batching issues with PolynomialKernel (1977)
- Fix issues with PeriodicKernel.diag() (1919)
- Add more informative error message when train targets and the train prior distribution mismatch (1905)
- Fix issues with priors on ConstantMean (2042)

This release contains several bug fixes and performance improvements.

New Features
- Variational multitask models can output a single task per input (rather than all tasks per input) (1769)

Small fixes
- LazyTensorto method more closely matches the torch Tensor API (1746)
- Add type hints and exceptions to kernels to improve usability (1802)

Performance
- Improve the speed of fantasy models (1752)
- Improve the speed of solves and log determinants with KroneckerProductLazyTensor (1786)
- Prevent explicit kernel evaluation when expanding a LazyTensor kernel (1813)

Fixes
- Fix indexing bugs with kernels (1802, 1819, 1828)
- Fix cholesky bugs on CUDA (1848)
- Remove lines of code that generate warnings in PyTorch 1.9 (1835)

New features

- Add `gpytorch.kernels.PiecewisePolynomialKernel` (1738)
- Include ability to turn off diagonal correction for SGPR models (1717)
- Include ability to cast LazyTensor to half and float types (1726)


Performance improvements

- Specialty MVN log_prob method for Gaussians with sum-of-Kronecker covariances (1674)
- Ability to specify devices when concatenating rows of LazyTensors (1712)
- Improvements to LazyTensor symeig method (1725)


Bug fixes

- Fix to computing batch sizes of kernels (1685)
- Fix SGPR prediction when `fast_computations` flags are turned off (1709)
- Improve stability of `stable_qr` function (1714)
- Fix bugs with pyro integration for full Bayesian inference (1721)
- `num_classes` in `gpytorch.likelihoods.DirichletLikelihood` should be an integer (1728)