Torch-xla

Cloud TPUs now support the [PyTorch 1.8.1 release](https://github.com/pytorch/pytorch/releases/tag/v1.8.1), via PyTorch/XLA integration. The release has daily automated testing for the supported models: [Torchvision ResNet](https://cloud.google.com/tpu/docs/tutorials/resnet-pytorch), [FairSeq Transformer](https://cloud.google.com/tpu/docs/tutorials/transformer-pytorch) and [RoBERTa](https://cloud.google.com/tpu/docs/tutorials/roberta-pytorch), [HuggingFace GLUE and LM](https://github.com/huggingface/transformers), and [Facebook Research DLRM](https://cloud.google.com/tpu/docs/tutorials/pytorch-dlrm).

On top of the underlying bug fixes in PyTorch's 1.8.1 release, this release adds a few bug fixes on the PyTorch XLA side around the XRT server and TPU Pods training.

Summary

Cloud TPUs now support the [PyTorch 1.8 release](https://github.com/pytorch/pytorch/releases/tag/v1.8.0), via PyTorch/XLA integration. The release has daily automated testing for the supported models: [Torchvision ResNet](https://cloud.google.com/tpu/docs/tutorials/resnet-pytorch), FairSeq [Transformer](https://cloud.google.com/tpu/docs/tutorials/transformer-pytorch) and [RoBERTa](https://cloud.google.com/tpu/docs/tutorials/roberta-pytorch), [HuggingFace GLUE](https://github.com/huggingface/transformers) and LM, and [Facebook Research DLRM](https://cloud.google.com/tpu/docs/tutorials/pytorch-dlrm).

This release focused on making PyTorch XLA easier to use and debug. See below for a list of new features.

New Features
* Enhanced usability:
* Profiler tools to help you pinpoint the areas where you can improve the memory usage or speed of your TPU models. The tools are ready to use; check out our [main README](https://github.com/pytorch/xla/blob/master/README.md) for some upcoming tutorials.
* Simpler error messages (2771)
* Less log spam using TPU Pods (2662)
* Able to view images in Tensorboard (2679)
* `TriangularSolve` (2498) (example)
* New ops supported by PyTorch/XLA:
* `random_` (2617)
* `adaptive_avg_pool3d` (2616)
* `UpsampleNearest2D` (2597)

Bug Fixes
* Crashing while using dynamic shapes (2602)
* `all_to_all` crashing on TPU pods (2601)
* `SiLU` fix (2721)

Summary
Cloud TPUs now support the [PyTorch 1.7 release](https://github.com/pytorch/pytorch/releases/tag/v1.7.0), via PyTorch/XLA integration. The release has daily automated testing for the supported models: [Torchvision ResNet](https://cloud.google.com/tpu/docs/tutorials/resnet-pytorch), FairSeq [Transformer](https://cloud.google.com/tpu/docs/tutorials/transformer-pytorch) and [RoBERTa](https://cloud.google.com/tpu/docs/tutorials/roberta-pytorch), [HuggingFace GLUE](https://github.com/huggingface/transformers) and LM, and [Facebook Research DLRM](https://cloud.google.com/tpu/docs/tutorials/pytorch-dlrm).

New Features
* `TriangularSolve` (2498) (example)
* New ops supported by PyTorch/XLA:
* `round` (2457)
* `baddbmm_out` (2451)
* `var_out` (2495)
* `asinh`, `acosh`, `atanh` (2501)
* Documentation on adding more supported ops: (2458)

Bug Fixes
* `exponential_()` returning 0 (2562)
* `cross_entropy` on `inf` input (2553)

Highlights
Cloud TPUs now support the PyTorch 1.6 release, via PyTorch/XLA integration. With this release we mark our general availability (GA) with the models such as ResNet, FairSeq Transformer and RoBERTa, and HuggingFace GLUE task models that have been rigorously tested and optimized.

In addition, with our PyTorch/XLA 1.6 release, you no longer need to run the env-setup.py script on Colab/Kaggle as those are now compatible with native `torch` wheels. See [here](https://github.com/pytorch/xla/blob/master/contrib/colab/getting-started.ipynb) for an example of the new Colab/Kaggle install step. You can still continue to use that script if you would like to run with our latest unstable releases.

New Features
* XLA RNG state checkpointing/loading (https://github.com/pytorch/xla/pull/2096)
* Device Memory XRT API (https://github.com/pytorch/xla/pull/2295)
* [Kaggle/Colab] Small host VM memory environment utility (https://github.com/pytorch/xla/pull/2025)
* [Advanced User] XLA Builder Support (https://github.com/pytorch/xla/pull/2125)
* New ops supported on PyTorch/XLA
* Hardsigmoid (https://github.com/pytorch/xla/pull/1940)
* true_divide (https://github.com/pytorch/xla/pull/1782)
* max_unpool2d (https://github.com/pytorch/xla/pull/2188)
* max_unpool3d (https://github.com/pytorch/xla/pull/2188)
* Replication_pad1d (https://github.com/pytorch/xla/pull/2188)
* Replication_pad2d (https://github.com/pytorch/xla/pull/2188)
* Dynamic shape support on XLA:CPU and XLA:GPU (experimental)

Bug Fixes
* RNG Fix (proper randomness with bernoulli and dropout) (https://github.com/pytorch/xla/pull/1932)
* Manual all-reduce in backward pass (https://github.com/pytorch/xla/pull/2325)

Cloud TPUs and Cloud TPU Pods now support [PyTorch 1.5](https://github.com/pytorch/pytorch/releases/tag/v1.5.0) via the PyTorch/XLA integration. This integration aims to make it possible for PyTorch users to do everything they can do on GPUs on Cloud TPUs as well while minimizing changes to the user experience. You can try out PyTorch on an 8-core Cloud TPU device for free via Google Colab, and you can use PyTorch on Cloud TPUs at a much larger scale on Google Cloud (all the way up to full Cloud TPU Pods).

Three PyTorch models have been added to our list of supported models, which are rigorously and continuously tested:
- ResNet-50
- Fairseq Transformer
- Fairseq RoBERTa

Additional notes:
- New Operators added
- masked_scatter (1525)
- logdet (1407)
- reflection_pad2d (1409)
- gelu(1278)
- take(1389)
- upsample_bilinear2d (1239)
- upsample_nearest2d (1233)
- mse_loss(1206)
- Exposed APIs to enable different types of cross-replica reduce operations using the TPU interconnect link (1709)
- Exposed API to perform rendezvous operations among the different replica processes (1669)
- Added support for reading/writing GCS files (1230)
- Added support to read TFRecords (1220)
- Miscellaneous bug fixes