Chainer

This is the release note of v6.0.0b2. See [here](https://github.com/chainer/chainer/milestone/83?closed=1) for the complete list of solved issues and merged PRs.


New Features

- Asynchronous snapshot writers (4472, thanks tyohei!)
- Add `D.Cauchy` (5337)
- Add `D.Geometric` (5343)
- Add `cached_property` decorator (5416)
- Make `build_computational_graph` accept single output (5445)
- Add trigger to be fired only once (5565, thanks hitsgub!)
- Use default dtype in `L.NegativeSampling` (5664)
- Add optional property `finished` to trigger object (5681, thanks hitsgub!)
- Support all float dtypes in `F.spatial_transformer_sampler` (5751)
- Add a naive `TimerHook` link hook. (5842, thanks crcrpar!)
- Add `F.as_strided` (5902, thanks fiarabbit!)
- Add 'mean' value as an option for VAE loss reduce (5966, thanks 23pointsNorth!)

Enhancements

- Support inputs with `ndim!=2` for `F.huber_loss` (5534)
- Show forward stacktrace in backward (5603)
- Add type check for `r` arg of `F.rrelu` (5619)
- Support unretained `Variable`s in `_check_grad_type` (5640)
- `FunctionNode` automatic fallback of array attributes in forward (5745)
- Switch device during `gradient_check` (5777)
- Raise CuPy not available error early in `cuda.GpuDevice` initialization (5780)
- Add `hasattr` check to user-specified `flush` call to file-like objects. (5794, thanks grafi-tt!)
- Support custom initializer in `links.CRF1d` (5807, thanks himkt!)
- Remove `F.clip` type restriction (5813)
- Batched pack/unpack params before/after allreduce (5829, thanks anaruse!)
- Remove unnecessary cast in `F.huber_loss` (5835)
- Reimplement `F.LocalResponseNormalization` as `FunctionNode` (5851)
- Stop managing memory in max pooling specific manner (5861)
- Do not retain input on iDeep `F.relu` (5871, thanks grafi-tt!)
- Set grad of `F.clip` 1 at `x_min` and `x_max` (5876, thanks grafi-tt!)
- Warn if `reset` method is not implemented in an iterator (5882)
- Cache attributes of distributions (5892)
- Use `FunctionNode` on `ROIPooling2D` (5957)
- Use more precise timer in `function_hooks/timer.py` (5971, thanks crcrpar!)
- Improve `F.elu` memory consumption by retaining output (5972, thanks grafi-tt!)

Bug Fixes

- Fix `dump_graph` not to memory leak (5538, thanks hitsgub!)
- Fix `F.batch_normalization` + `F.forget` combination (5557)
- Bugfix of `MultiNodeOptimizer` with loss scaling (5659)
- Fix usage of `downsample_fb` in resnet (5737, thanks milhidaka!)
- Fix `device` argument passed to `MultiprocessParallelUpdater` being modified (5739, thanks Guriido!)
- Fix bug when CuPy not installed and `cuda.fuse` decorator used without parentheses (5809, thanks grafi-tt!)
- Fix `F.cast` gradient for casts between the same dtypes (5811)
- Accept splitting at the tail of dataset in `split_dataset` (5895)
- Fix broken `F.leaky_relu` grad when `slope = 0` (5898, thanks grafi-tt!)
- Add copyparams method to Sequential (5914)
- Override `_to_device` for consistency (5948)
- Allow `import chainer.testing` without pytest (5973)
- Raise an appropriate error on cuDNN RNN backward in testing mode (5981)
- Fix stochastic failure in `WalkerAlias` (6057)

Documentation

- Remove deprecation notices for v1 and v2 in documentation (5081)
- Add description for initializer dtype (5246)
- Add Code of Conduct (5629)
- Improve installation guide of ChainerMN (5656)
- Add explanations for LeNet5 (5686)
- Make docs of activation functions refer `ndarray` (5718)
- Add `robots`.txt to hide older versions from search results (5768)
- Fix typo in v2 Upgrade Guide (5771)
- Fix a couple of broken links from markdown files (5789)
- Model Parallel Documentation (5791, thanks levelfour!)
- Fix wording in documentation (5795)
- Write "Wx + b" in the document of `Linear`. (5852)
- Make docs of array functions refer `ndarray` (5863)
- Some small fixes to grammar and spelling (5869)
- Make docs of connection functions refer `ndarray` (5875)
- Fix `static_graph` module path in documentation (5883)
- Correct the stable version in master branch (5891, thanks jinjiren!)
- Change `.data` to `.array` in Guides and Examples docs (5907, thanks jinjiren!)
- Fix typo (5915, thanks MannyKayy!)
- Transform dataset documentation fix (5938, thanks 23pointsNorth!)
- Fix typo (5942)
- Update the note in DCGAN example to be compatible with the code. (5951, thanks jinjiren!)
- Fix doc of `F.softmax_cross_entropy` on output shape with `reduce=no` (5965)
- Make some docs of functions refer `ndarray` (5975)
- Fix document in NStepLSTM/NStepRNN (5979)
- Make docs of math functions refer `ndarray` (6032)
- Fix wrong MNIST MLP anchor (6046)

Installation

- Check integrity of CuPy wheel for CUDA 10 (5955)

Examples

- Add inference code to MNIST example (4741)
- Use `iter.reset()` in PTB example (5834)
- Some small improvements to the Mushrooms example (5982)

Tests

- `FunctionTestCase` for function tests (3499)
- Test statistics of initializers (5511)
- Add test mode to text classification example (5666)
- Fix test of `F.connectionist_temporal_classification` (5727)
- Refactor tests of `F.split_axis` and `F.concat` (5733)
- Return exitcode of `make html` to Travis (5769)
- Fix `testing.BackendConfig` context for repeated use (5779)
- Encode parameters in parameterized class name (5782)
- Add test for conveter device argument in `Evaluator` (5806)
- Fix error message of `testing.assert_allclose` (5814)
- Refactor CI scripts (5858)
- Refactor Travis script (5859)
- Remove some CI requirements (5865)
- Allow multiple application of `testing.parameterize` (5893)
- Allow mixing `testing.inject_backend_tests` and `testing.parameterize` (5904)
- Adjust testing tolerance of numerical gradient (5923)
- Adjust testing tolerance of `F.connectionist_temporal_classification` (5928)
- Do not ignore `FutureWarning` other than experimental features (5949)
- Move mypy to static checks (5987)
- Skip test on Theano<=1.0.3 and NumPy>=1.16.0 (6001)
- Fix travis script to continue on failure in each step (6002)
- Fix `inject_backend_tests` `multi_gpu` test mark (6028)
- Allow doctest to run in single-GPU environment (6029)
- Test if the default CUDA device keeps being 0 after each test (6044)

ChainerX

- Add ChainerX native float16 (5761)
- CuPy/ChainerX memory pool sharing (5821)
- Automatic ChainerX fallback of array attributes in `Function` (5828)
- ChainerX backward w.r.t. inputs (C++ `chainerx.grad` ) (5747)
- Improve gradient mismatch error (5748)
- Forbid fallback get/setitem for arrays with backprop required (5754)
- Implement BFC algorithm in ChainerX CUDA memory pool (5760)
- Resolve `_as_noncontiguous_array` workaround for ChainerX (5781)
- `L.NegativeSampling` ChainerX support (5816)
- Stop using Unified Memory by default (5912)
- Avoid `cudaMemcpyAsync` for pinned memory for faster host-to-device transfer (5940)
- Remove `chainerx.asscalar` (6007)
- Fix scalar handling of `indices_and_sections` in `chainerx.split` (5788)
- Fix ChainerX Python docstring allocation issue (5815)
- Fix `chainerx.maximum` to restore CUDA device (6043)
- Build ChainerX on ReadTheDocs (5766)
- Add `chainerx.ndarray` to the `ndarray` doc (5864)
- Document CuPy memory pool sharing (6017)
- Do not overwrite `CMAKE_CXX_FLAGS` a user specified (5770)
- Patch files for macOS (5776, thanks ktnyt!)
- Update `pybind` dependency to v2.2.4 (5798)
- Update `gsl-lite` to v0.32.0 (5849)
- Enable ChainerX in docker image (5879)
- Update third-party.cmake to follow the recent way (5911)
- Made ChainerX setup and compile on Windows (5932, thanks durswd!)
- Fix visibility for `pybind` exception registration for macOS (5936)
- Fix manifest typos (6065)
- ChainerX MNIST C++ example (5746)
- Remove some TODOs of the chainerx resnet example (5775)
- Fix jenkins script to allow explicit repo root (5774)
- Fix to test against new `chainerx.GradientError` (5787)
- Add Travis matrix for macOS ChainerX tests (5846)
- Remove `.circleci` (5860)
- Add C++ linter checks in Travis CI (5867)
- Fix `FixedCapacityDummyAllocator` in CUDA memory pool test (5993)
- Fix CUDA specific Python binding (6037)
- Add chainerx-generated reference docs to `.gitignore` (5805, thanks knorth55!)
- Disable clang-tidy `modernize-use-auto` (5839)

Code Fixes

- Simplify batch normalization with cuDNN (5568)
- Add type hints for `Link`, `LinkHook`, `Initializer` and ChainerX (5675)
- Refactor gradient setter in `gradient_check` (5699)
- Use new RNN implementation (5726)
- Backprop from multiple variables (5741)
- Fixes for clang (5744)
- Improve coding style (5763)
- Fix style of `setup.py` (5764)
- Code enhancements: avoid array copies (5800)
- Random code enhancements (5801)
- Add comment to `MultiprocessIterator.__copy__` (5833)
- Move workaround `utils._getitem`/`_setitem` to `chainerx` (5840)
- Fix clang-tidy error (5870)
- Fix typo on internal attribute (5894)
- Fix clang-tidy warnings on clang-tidy 6 (5901)
- Fix for clang-tidy 7 (5933)
- Fix code formatting (5941)
- Remove `overload` annotations outside the stub files (5960)
- Avoid deprecated `numpy.asscalar` (5994)
- Post macro comment for consistency (6014)
- Remove `chainerx.asscalar` from mypy stub file (6024)

Others

- Fix `.gitignore` to avoid ignoring some necessary files (5836)
- Allow skipping linkcode in docs with environment variable (5868)

This is the release note of v6.0.0b1. See [here](https://github.com/chainer/chainer/milestone/81?closed=1) for the complete list of solved issues and merged PRs.

Highlights

ChainerX

ChainerX is an ndarray implementation with Define-by-Run automatic differentiation capability. It roughly corresponds to "NumPy/CuPy + Chainer Variable", while some additional features follow:

- **Speed**: The whole ndarray and autograd implementation is written in C++, with a thin Python binding. It lowers the overhead existing in the pure Python implementation of Chainer.
- **Extensibility**: The backend is pluggable so that it is much easier to add support of new devices.

The speed is best achieved by directly using ChainerX APIs,
while it also provides a compatibility layer through the conventional `Variable` interface for easier adoption of ChainerX in existing projects.
See the [ChainerX Tutorial](https://github.com/chainer/chainer/blob/master/docs/source/chainerx/tutorial/index.rst) for more details and concrete examples.

New Features

- Implement double backward of SLSTM function (4824, thanks tohmae!)
- Add `F.roi_max_align_2d` (5198, thanks knorth55!)
- Add `F.roi_average_pooling_2d` (5285, thanks knorth55!)
- Add `F.roi_max_pooling_2d` (5304, thanks knorth55!)
- Support all float dtypes in `F.negative_sampling` (5336)
- Add `D.Chisquare` (5338)
- Add `D.Gumbel` (5352)
- Add `D.Poisson` (5364)
- Add `D.OneHotCategorical` (5372)
- Serialize `BestValueTrigger` (5402, thanks ktns!)
- Add `return_samples` argument to `F.negative_sampling` and `L.NegativeSampling` (5597)
- Support all float dtypes in `F.embed_id` (5624)
- Support default dtype in `L.BlackOut` (5638)
- Support default dtype in `L.BinaryHierarchicalSoftmax` (5648)
- Support all float dtypes in `F.connectionist_temporal_classification` (5680)
- ChainerX (5725)

Enhancements

- Add type compatibility check in npz deserializer (5483)
- Use `cupy.linalg.det` in `F.det` (5525)
- Avoid unnecessary copy in `ndarray.astype` (5547)
- Avoid cuDNN handle around `DropoutStates` (5563)
- Simplify softmax with cuDNN (5566)
- Simplify pooling with cuDNN (5567)
- Add KL divergence test for `D.OneHotCategorical` (5587)
- Add `compute_stream` argument in `ConcatWithAsyncTransfer` to allow more overlap between computation transfer in CUDA (5606, thanks anaruse!)
- Use `chainer.utils.size_of_shape` in ChainerMN (5610)
- Import `testing/backend.py` definitions in `testing/__init__.py` (5633)
- Avoid using dype char codes (5646)
- More consistent use of `Variable.array` in codes under links (5657, thanks crcrpar!)
- Use automatic broadcasting instead of `F.repeat` (5662)
- Refactor the statemachine of iterators that iterates indices (5669, thanks grafi-tt!)
- Refactor `train_mnist_dual_parallel.py` (5678)
- Change `Link.add_hook` to return `self` (5736, thanks crcrpar!)

Bug Fixes

- Fix `reporter.Summary` float value deserialization (5482)
- Fix `text_classification` example fails on Python 3 (5591, thanks koreyou!)
- Improve iDeep version checking (5600)
- Fix `D.OneHotCategorical` (5604)
- Fix Python 3.7 test failures in `F.roi_average_pooling_2d` (5611)
- Fix `F.negative_sampling` output dtype in CPU mode (5613)
- Fix args check in `F.roi_average_align_2d` and `F.roi_average_pooling_2d` (5627, thanks knorth55!)
- Fix `L.BatchNormalization` with lazy initialization fail on GPU (5683, thanks koreyou!)

Documentation

- Simplify array type information fields in function documentation (4887)
- Update installation guide of numpy with openblas on macOS (5021)
- Add links to ChainerCV documentation (5434)
- Add ChainerMN paper to references (5570)
- Fix docstring of `F.forget` (5586, thanks fiarabbit!)
- Fix typo in updaters (5589, thanks okayu9!)
- Fix extensions guide error regarding method to implement (5602, thanks lehy!)
- Update `F.roi_average_align_2d` doc to refer wrapper function (5609, thanks knorth55!)
- Fix a typo in `Chain` example code (5653)
- Fix typo in `F.max_pooling_nd` docstring (5654)
- Fix a typo in `chainer.distributions` documentation (5658)
- Add documentation of `ndarray` (5660)
- Fix typo in `L.ResNetLayers` (5665, thanks takaaki82!)
- Minor typo correction (in docs/variables). (5670, thanks grigorisg9gr!)
- Fix typo in docstrings (5676)
- Fix docs for `backprop_step` (5692)
- Make docs in `chainer.distributions` refer `ndarray` (5717)
- Fix image URL in README (5720, thanks levelfour!)
- Add warning in ChainerX documentation (5752)

Installation

- Require setuptools and add docs for (5532)

Examples

- Add WaveNet example (4922, thanks dhgrs!)
- Rewrite the example of VAE using Chainer distributions (5356, thanks ganow!)

Tests

- Fix test warnings in NumPy 1.15 (5596)
- Fix test of `F.rrelu` (5618)
- Fix regex of protobuf modules warned by Python 3.7 (5642)
- Ignore h5py warning in Python 3.7 (5691)
- Add gradient consistency checks in `numerical_grad` (5698)

Other

- Update style check tools to the versions compatible with pycodestyle 2.4 (5643)

This is the release note of v5.4.0. This is the final release of v5.x series. See [here](https://github.com/chainer/chainer/milestone/87?closed=1) for the complete list of solved issues and merged PRs.

Enhancements
- Fix error messages in `get_fans` (6413)
- Change backward of `F.log_ndtr` to avoid NaN (6431)
- Avoid using pytest attributes during import (6470)
- Support cupy-cuda101 package (6701)

Bug Fixes
- Fix `text_classification` example fails on Python 3 (5651, thanks koreyou!)
- Fix lazy init of `BatchNormalization` link (6480)
- Fix `chainer.links.NStepRNN` and its variants (6517, thanks crcrpar!)
- Fix NCCL version check error in ChainerMN (6504)

Code Fixes
- Avoid ` NOQA` in docstrings (6549)
- Change the deprecated `collections` (6676)
- Fix F632 (use ==/!= to compare str) (6714)

Documentation
- Remove duplicated entry in reference (6578)
- Fix `F.rrelu` documentation (6586, thanks fiarabbit!)
- Add `gradient_check.check_double_backward` in reference (6587)
- Override `forward` in `LinkHook` documentation (6594, thanks crcrpar!)
- Fix `:meth:` link (6614, thanks 23pointsNorth!)
- Improve docs and exception message in `F.erfcx`, `F.erfcinv` and `F.erfinv` (6632)
- Fix missing module declaration (6671)
- Fix `chainer.backend.get_array_module` documentation (6685)
- Fix typo: 'Notatition' -> 'Notation' (6686, thanks nai62!)
- Fixes typo in docstring for `classification_summary` (6697, thanks yewang!)
- Write guides to implement new-style functions (6730)

Examples
- Fix `dali_util` in imagenet example for fp16 (6377, thanks anaruse!)
- Use `args.out` in `train_cifar_custom_loop.py` (6411, thanks crcrpar!)
- Remove FP16 specific models from imagenet example (6564)
- Fix iterator syntax in MNIST custom loop example (6565)
- Use float division instead of `__future__.division` for Python2 (6567)
- Fix DCGAN example to work with default dtype (6591)
- Use `F.matmul` instead of `F.batch_matmul` in memnn example (6631)

Tests
- Do not ignore `FutureWarning` other than experimental features (6052)
- Suppress warning in caffe test (6409)
- Test all codes in guides/functions (6428)
- Remove `SPHINXOPTS` env from Makefile (6491)
- Fix Python 3.4 NumPy Accelerate polyfit error (6495)
- Change pytest version requirement (6513)
- Adjust tolerances of `F.einsum` tests (6672)
- Fix test failure in cuDNN v7.5 (6716)

This is the release note of v5.3.0. See [here](https://github.com/chainer/chainer/milestone/85?closed=1) for the complete list of solved issues and merged PRs.

Enhancements
- Reduce memory usage in `MultiprocessParallelUpdater` (6113)
- Check input size consistency in RNN and LSTM when using cuDNN (6186)
- Add `group` option value of `Convolution2D` to Caffe exporter (6293, thanks ohnabe!)
- Add support for importing and exporting Caffe `Sigmoid` layer (6294, thanks notogawa!)

Performance Improvements
- Improve `F.relu` performance with CuPy (6270)
- Reduce `chainer.is_debug()` overhead (6297)

Bug Fixes
- Bugfix of `MultiNodeOptimizer` with loss scaling (5783)
- Fix BN+`F.forget` (6076)
- Fix cupy import failure detection (6112)
- Fix memory leak during backprop in Python 2 (6125)
- Use ideep in optimizers properly (6143)
- Fix `dump_graph` not to memory leak (6147, thanks hitsgub!)
- Fix warning message for backward on a scalar array (6319)

Documentation
- Fix wrong MNIST MLP anchor (6055)
- Fix document in `NStepLSTM`/`NStepRNN` (6074)
- Fix typo in `classifier.py` (6102, thanks hiden-cubist!)
- Document NumPy 1.16 support (6141)
- Reorganize documentation for easier access to tutorials and examples (6152)
- Fix old and broken PTB url (6180)
- Add upgrade guide for use of forward method (6193)
- Add imports of initializers and math, required in "Define your own function" examples (6220, thanks Qwinpin!)
- Add references for n-dimensional arrays (6221)
- Imagenet README.md typo (6224)
- Update URL of PTB dataset (6239)
- Make docs of functions refer `ndarray` (6288)

Examples
- Refactor `train_mnist_dual_parallel.py` (5716)
- Fix seq2seq example (6093)
- Minor fix on README of seq2seq example (6208)
- Fix seq2seq example encoding problem on Python3 (6209)
- Remove `PrintReport` entries in seq2seq example (6321)

Tests
- Fix tolerance and numeric grad precision in `F.triplet` test (6144)
- Fix chainermn tests (6086)

This is the release note of v5.2.0. See [here](https://github.com/chainer/chainer/milestone/84?closed=1) for the complete list of solved issues and merged PRs.

New Features

- Support default dtype in `L.BinaryHierarchicalSoftmax` (5714)
- Support all float dtypes in `F.embed_id` (5926)
- Support all float dtypes in `F.spatial_transformer_sampler` (6003)
- Support all float dtypes in `F.connectionist_temporal_classification` (6011)
- Support all float dtypes in `F.det` and `F.inv` (6012)
- Use default dtype in `L.NegativeSampling` (6013)
- Introduce `utils.mixed_presision` decorator (6022)
- Add a naive `TimerHook` link hook (6038, thanks crcrpar!)

Enhancements

- Change `Link.add_hook` to return `self` (5750, thanks crcrpar!)
- Add `hasattr` check to user-specified `flush` call to file-like objects (5803, thanks grafi-tt!)
- Support unretained `Variable`s in `_check_grad_type` (5826)
- Use new RNN implementation (5827)
- Simplify batch normalization with cuDNN (5853)
- Reimplement `F.LocalResponseNormalization` as `FunctionNode` (5900)
- Support custom initializer in links.CRF1d (5905, thanks himkt!)
- Use FunctionNode on ROIPooling2D (5967)
- Fix error message of `testing.assert_allclose` (5984)
- Use more precise timer in `function_hooks/timer.py` (6021, thanks crcrpar!)

Bug Fixes

- Fix BatchNormalization with lazy initialization fail on GPU (5713, thanks koreyou!)
- Fix `device` argument passed to `MultiprocessParallelUpdater` being modified (5790, thanks Guriido!)
- Fix `F.cast` gradient for casts between the same dtypes (5818)
- Fix bug when CuPy not installed and `cuda.fuse` decorator used without parentheses (5825, thanks grafi-tt!)
- Fix usage of `downsample_fb` in resnet (5850, thanks milhidaka!)
- Accept splitting at the tail of dataset in `split_dataset` (5899)
- Fix broken `F.leaky_relu` grad when slope = 0 (5922, thanks grafi-tt!)
- Raise an appropriate error on cuDNN RNN backward in testing mode (5983)
- Add copyparams method to Sequential (5990)
- Allow `import chainer.testing` without pytest (5998)
- Fix `.gitignore` to avoid ignoring some necessary files (5838)

Documentation

- Fix image URL in README (5755, thanks levelfour!)
- Fix typo in v2 Upgrade Guide (5772)
- Fix a couple of broken links from markdown files (5792)
- Fix wording in documentation (5820)
- Make docs of activation functions refer `ndarray` (5831)
- Model Parallel Documentation (5843, thanks levelfour!)
- Add explanations for lenet5 (5855)
- Add description for initializer dtype (5872)
- Add Code of Conduct (5873)
- Make docs of array functions refer `ndarray` (5881)
- [v5] Document optional arguments as `None` (5886)
- Make docs of connection functions refer `ndarray` (5889)
- Fix `static_graph` module path in documentation (5906)
- Change `.data` to `.array` in Guides and Examples docs (5913, thanks jinjiren!)
- Fix typo (5917, thanks MannyKayy!)
- Write "Wx + b" in the document of `Linear`. (5919)
- Improve installation guide of ChainerMN (5937)
- Transform dataset documentation fix (5947, thanks 23pointsNorth!)
- Update the note in DCGAN example to be compatible with the code. (5962, thanks jinjiren!)
- Fix doc of `F.softmax_cross_entropy` on output shape with `reduce=no` (5969)
- Make some docs of functions refer `ndarray` (5976)
- Make docs of math functions refer `ndarray` (6034)

Installation

- Check integrity of CuPy wheel for CUDA 10 (5956)

Examples

- Use `iter.reset()` in PTB example (5857)

Tests

- Add test mode to text classification example (5784)
- Adjust testing tolerance of numerical gradient (5946)
- Test statistics of initializers (5961)
- Fix pytest plugin version (5968)
- Adjust testing tolerance of F.connectionist_temporal_classification (6035)
- Test if the default CUDA device keeps being 0 after each test (6047)