Devito

pip
* Removed direct dependencies to enable PyPI

Air-Speed Velocity (ASV)
* Added performance regression testing via ASV

CI/CD
* Automatically start and stop Azure VM's for testing.

Benchmarking
* Additional performance metrics

Documentation
* New examples for TTI adjoint and viscoacoustic.
* Updated project logo and documentation version.
* Various improvements to examples/notebooks.

Bugfixes
* Loop fusion.
* MPI/SubDomain.

Many thanks to all the contributors to this release (last surname alphabetical order):
George Bisbas (Imperial College London)
Gerard Gorman (Imperial College London)
Mathias Louboutin (Georgia Institute of Technology)
Fabio Luporini (Devito Codes)
Oscar Mojica (SENAI CIMATEC)
Rhodri Nelson (Imperial College London)
Peterson Nogueira (SENAI CIMATEC)
João Henrique Speglich (SENAI CIMATEC)
Lauê Rami Souza Costa de Jesus (SENAI CIMATEC)
John Washbourne (Chevron)

GPU support

* Added OpenACC 2.6 code generation support.
* Added domain decomposition support for multi-node-multi-GPU via MPI and OpenACC.

Testing, CI/CD

* Added testing with PGI compiler suite v19.10 on NVidia K80 GPU's.
* Enhanced CI testing for OpenMP 5 offloading onto GPU's.
* Enhanced CI testing for domain decomposition with MPI.

Performance optimizations

* Wide range of performance optimizations to the Devito compiler to speed-up compile time (~2-5x speedup in compilation time).
* Various low-level performance and memory optimizations (e.g. identifying and removing redundant computations).

DSL extensions

* tools.Evaluable -> finite_differences.Evaluable
* Dynamic FD shortcuts (i.e. `u.dx(fd_order=2)`). Supports FD order, `x0` and `side`.

YASK and OPS backends

* After a great deal of consideration, we have decided to drop support for YASK (https://01.org/yask) and OPS (https://github.com/OP-DSL/OPS) as Devito backends. While YASK and OPS have many desirable features, the software engineering overhead (person-effort) to maintain feature-complete support proved unsustainable. We would like to take this opportunity to thank Chuck Yount (YASK) and Istvan Reguly (OPS) for their collaboration on both software development and algorithm development, which has been of great benefit to Devito.

Tutorials and documentation

* Revised FAQ - https://github.com/devitocodes/devito/wiki/FAQ
* New Jupyter notebooks:
* GPU tutorial.
* PML boundary condition tutorial.
* Normal Moveout (NMO) tutorial.
* Revised viscoelastic example.

General updates

* Added Developer Certificate of Origin to Devito contribution guidelines.
* General improvements to documentation, examples and tutorials.
* Several minor bug fixes.
* Increased test coverage.
* Improvements to test code coverage reporting.
* Added PGI compiler support.
* Added MPI support for the SubDomainSet feature.
* Additional runtime warnings for inadequate problem specifications, e.g. stencil radius greater than rank-domain width.
* Deprecate DEVITO_DLE, replaced by DEVITO_OPT.
* Deprecate DEVITO_OPENMP, replaced by DEVITO_LANGUAGE.

Many thanks to all the contributors to this release (last surname alphabetical order):

* George Bisbas (Imperial College London)
* Tim Burgess (DownUnder Geosolutions)
* Edward Caunt (Imperial College London)
* Gerard Gorman (Imperial College London)
* Mathias Louboutin (Georgia Institute of Technology)
* Fabio Luporini (Devito Codes)
* Rhodri Nelson (Imperial College London)
* Peterson Nogueira (SENAI CIMATEC and UFBA)
* Joao Speglich (SENAI CIMATEC)
* John Washbourne (Chevron)
* Philipp Witte (Georgia Institute of Technology)

Special thanks and congratulations to Ken Hester (NVidia), winner of the Devito hackathon at Rice OGHPC 2020, who used the Devito JIT backdoor to develop the prototype OpenACC code which informed the implementation of OpenACC code generation support within this release.

GPUs
* Support for more Operators -- isotropic acoustic, TTI, elastic, viscoelastic, ...
* Support for reductions
* Fixed several bugs concerning code generation

DSL
* Fixed minor bugs related to derivatives and finite-differences

MPI
* Fixed several bugs arising in presence of SubDomains and/or SubDimensions
* Fixed code generation in full mode (compute-communication overlap), when the same symbol (e.g. `comm`) was appearing twice in the signature of elemental functions

Documentation
* Updated user API
* Additional tutorials in the form of Jupyter notebooks (SymPy, DSL, SubDomains)

Benchmarking
* A new mode exploiting the JIT_BACKDOOR mechanism was added
* DevitoBench package merged into Devito

CI/CD:
* Dropped Azure Pipelines and Travis; testing infrastructure moved to GitHub Actions
* Added GPU testing
* Added testing for osx
* Auto-deployment on Pypi upon release

Compiler:
* Improved architecture auto-detection
* Refactored DLE as a sequence of "standard" compilation passes
* Moved data dependence analysis to the Cluster level
* Fixed data dependence analysis in some corner cases arising with SubDomains and/or SubDimensions
* Moved interpolation to separate sub-package
* Integrated profiling for the compilation pipeline

Many thanks to all the contributors to this release (last surname alphabetical order):
* Tarek Allam (The Alan Turing Institute)
* George Bisbas (Imperial College London)
* Maelso Bruno (Federal University of Paraíba)
* Tim Burgess (DownUnder Geosolutions)
* Gerard Gorman (Imperial College London)
* Navjot Kukreja (Imperial College London)
* Mathias Louboutin (Georgia Institute of Technology)
* Fabio Luporini (Devito Codes)
* Rhodri Nelson (Imperial College London)
* BlockSprintZIf (Independent)

Tensor algebra support (873):
- `VectorFunction` and `VectorTimeFunction`
- (2nd order) `TensorFunction` and `TensorTimeFunction`
- Full support for FD and related operations (derivatives, shortcuts, solve, ...)
- Differential operators such as `div`, `grad` and `curl`

FD extensions:
- Custom FD with user-supplied coefficients as `Function` (964)

Extended and more rigorous support for staggered grids (873):
- True half-grid staggering (`u(x + h_x/2)`)
- Automatic evaluation at half-nodes (averaging only)
- Automatic staggered FD of any order

MPI:
- MPI data slicing fixes(1008)
- SparseFunction fixes (gathering with correct coordinates) (922)
- MPI parameters correctly passed to the `Operator` (1016, 1019)
- Gathering bug fix (916)
- Domain decomposition improvements (915, 904)
- Effective computation/communication overlap
- Fixed argument processing to acquire the correct MPI communicator

GPU:
- More progress on the OPS backend
- OpenMP-5.x offloading experimental support (1004, 1012)
(note: compiler installation instructions available on the Devito wiki)

Performance:
- Better OpenMP scheduling and sparse loop handling (995)
- Faster compilation (932)

Misc:
- More robust inheritance from `sympy` and support for latest version (1005)
- `Model` bug fixes and cleanup (1000, 1015)
- `Model` now initializes/pads all object via Operators (911)
- `CustomCompiler` now uses the system/user provided parameters correctly (925)
- Devito based Gaussian smoother (931)
- ARM support (940)
- Better caching infrastructure (956) removing the majority of memory leaks
- Improved benchmarking infrastructure

Many thanks to all the contributors to this release (last surname alphabetical order):

* George Bisbas (Imperial College London)
* Maelso Bruno (Federal University of Paraíba)
* Lucas Cavalcante (Federal University of Rio Grande do Norte)
* Gabriel Sebastian von Conta (Universidade Federal do Rio Grande do Norte)
* Arthur Giannotta (University of São Paulo)
* Gerard Gorman (Imperial College London)
* Navjot Kukreja (Imperial College London)
* Mathias Louboutin (Georgia Institute of Technology)
* Fabio Luporini (Imperial College London)
* Vitor Mickus (Federal University of Rio Grande do Norte)
* Rhodri Nelson (Imperial College London)
* Vincenzo Pandolfo (G-Research)
* Elias Gabriel Amaral da Silva (Federal University of Rio Grande do Norte)
* Jaime Freire de Souza (University of São Paulo)

Release notes
--------------

* MPI support:
- Python-level: MPI-distributed NumPy arrays.
- C-level: code generation for sub-domains, staggered grids, operators with coupled PDEs.
- C-level: performance optimizations (e.g., computation-communication overlap).
* Lazy evaluation of derivatives.
* Revisited staggered grids API (now Dimension-based, previously mask-based).
* Re-engineered clustering (which means smarter loop fusion/fission).
* DSE: Improved aliases detection.
* DLE: OpenMP nested parallelism; hierarchical loop blocking.
* Auto-padding for Functions/TimeFunctions.
* Improved data dependency analysis.
* Smarter Operator auto-tuning.
* New tutorials: Operator application, MPI, new propagators, custom stencils, and more.
* Revisited benchmarking scripts.
* Revisited examples, new models and propagators (e.g., visco-elastic).
* Smarter continuous integration: now Travis sided by Azure Pipelines; dropped Jenkins.
* Misc bug fixes.
* Hundreds of tests added.
* More sophisticated platform auto-detection.
* Update in checkpointing (via PyRevolve) to significantly reduce memory consumption and overall runtimes. Not backward compatible.
* Experimental support for lossy compression of checkpoints through PyRevolve + ZFP.

Release notes
--------------

* Preliminary support for MPI (no changes to user code requested)
* Support for staggered grids
* Improved compilation technology
* Improved Operator autotuning
* More powerful DSL (e.g., take derivatives of entire expressions such as `(u+v).dx`)
* More efficient pickling
* Misc bug fixes
* New modeling examples based on the elastic wave equation
* New examples describing aspects of the compilation technology