Devito

Changes

🐛 Bug Fixes

- compiler: Patch TempFunction pickling FabioLuporini (1677)
- bench: Patch ASV's generation of new plots FabioLuporini (1672)
- gpu: Fix leaks due to excessive fetching/prefetching FabioLuporini (1658)
- types: Drop unsafe/unnecessary memoization in the arguments processing engine FabioLuporini (1647)
- compiler: Fix processing of grid spacing FabioLuporini (1628)
- misc: Avoid crashing on missing _memfree_args FabioLuporini (1612)
- compiler: Fix ScheduleTree construction in presence of guards and/or syncs FabioLuporini (1611)
- misc: Patch ThreadID pickling FabioLuporini (1606)
- compiler: Patch issue 1592 (HaloScheme with time subdimensions) Leitevmd (1597)
- dsl: Patch symbolic coefficients with staggered grids EdCaunt (1595)
- Fix find_library issue to MacOS Big Sur speglich (1584)
- Fix hierarchical blocking + parallelism FabioLuporini (1580)
- BoundSymbol constructor to be cached mloubout (1576)

Compiler

- compiler: Tweak nested-par candidate condition georgebisbas (1669)
- compiler: Singletonize special symbols (e.g. nthreads) FabioLuporini (1650)
- misc: Drop unused backend infrastructure FabioLuporini (1632)
- compiler: Improve aliases detection, processing, and optimization FabioLuporini (1631)
- mpi: Add diag2 mode FabioLuporini (1630)
- compiler: Add skewing pass towards Temporal Blocking georgebisbas (1620)
- misc: Use pickled soname rather than generating a new one FabioLuporini (1605)
- compiler: Add option to use Functions, in place of Arrays, for compiler-generated temporary FabioLuporini (1591)
- compiler: Improve the cost model used by CIRE FabioLuporini (1585)
- Refactor Array sharing FabioLuporini (1583)
- Refactor Operator hierarchy FabioLuporini (1573)
- Introduce devito/arch FabioLuporini (1563)
- operator: Use FD-Gpts/s instead of Gpts/s georgebisbas (1544)

API

- sympy: Support v1.8 mloubout (1549)
- compiler: Check consistency between shape and grid for TimeFunction, too tjb900 (1667)
- dsl: Add MatrixSparseTimeFunction to support multi-point sources FabioLuporini (1603)
- dsl: Enable overriding over SubDimension thickness FabioLuporini (1608)
- Shift argument as a tuple Leitevmd (1561)
- BoundSymbol constructor to be cached mloubout (1576)
- gpu: Add `devicerm` API for conditional deletions FabioLuporini (1571)
- Introduce `deviceid` API to offload Operators on specific GPUs FabioLuporini (1569)

Examples

- advisor: merge roofline and json georgebisbas (1649)
- examples: TTI 1st order operators ofmla (1602)
- examples: Add viscoacoustic Born operator to 2nd sls equation nogueirapeterson (1617)
- examples: Born approximation for TTI media ofmla (1555)
- examples: add first order adjoint viscoacoustic equations nogueirapeterson (1567)
- Tutorials: add shift parameter to Ren visco-acoustic equation nogueirapeterson (1562)

Documentation

- Bump static release number FabioLuporini (1564)

MPI

- compiler: Check consistency between shape and grid for TimeFunction, too tjb900 (1667)
- mpi: Add diag2 mode FabioLuporini (1630)
- mpi: Add ability to specify MPI topology used for grid division FabioLuporini (1604)
- mpi: Prevent double finalization with devito used as a lib FabioLuporini (1609)
- mpi: Pass MPI_COMM_SELF to Distributor upon unpickling FabioLuporini (1607)

GPU

- gpu: Fixup prefetch jitting when using extra symbols FabioLuporini (1678)
- gpu: Update Dockerfile.nvidia to HPC SDK 21.3 kenhester (1659)
- gpu: Fix leaks due to excessive fetching/prefetching FabioLuporini (1658)
- gpu: Add gpu-fit value for all functions (fix 1642) Leitevmd (1645)
- compiler: Work around clang[10,11,?] omp-offloading bug FabioLuporini (1634)
- arch: Review get_gpu_info Leitevmd (1626)
- misc: Updating to NVIDIA HPC SDK 21.2 kenhester (1594)
- compiler: Target gpu for PGI openacc Leitevmd (1587)
- arch: Add nvidia-smi parser to GPU checking Leitevmd (1615)
- gpu: Updated Dockerfile.nvidia to HPC SDK 21.1 kenhester (1593)
- gpu: Add `devicerm` API for conditional deletions FabioLuporini (1571)
- Data streaming support with OpenMP offloading FabioLuporini (1556)

Testing

- ci: Modify mpi-example workflow and update docker actions rhodrin (1664)
- ci: Switch to default gcc(9.3) for conda build mloubout (1651)
- ci: Fix conda build mloubout (1648)
- ci: Update Ci-gpu for nvidia openmp Leitevmd (1635)
- ci: Work around archives missing error on apt install FabioLuporini (1623)
- ci: Transfer gpu workflow to self-hosted runners rhodrin (1618)
- ci: Separate adjoint based tests rhodrin (1572)

🐛 Bug Fixes

- Patch aliases' min-storage option FabioLuporini (1535)
- Fix loop collapsing FabioLuporini (1534)
- operator: Patch grid-carried argument overrides FabioLuporini (1523)
- Fix profiling IET for Intel Advisor benchmarking georgebisbas (1516)
- Fix ARM issues georgebisbas (1515)
- make test tools less random ggorman (1507)
- ir: Fixup _lower_stepping_dims Leitevmd (1504)
- Fix custom coefficients on staggered grids EdCaunt (1497)
- Tensor grid fixup mloubout (1489)
- Patch issue 1477 FabioLuporini (1488)
- Patch compilation time FabioLuporini (1487)
- Conditonals: Added diff2sympy to conditionals EdCaunt (1475)
- Minor tweaks FabioLuporini (1469)
- Fix SubDomainSet bug rhodrin (1457)
- Function globabl size method rhodrin (1499)
- Fix parallel increments FabioLuporini (1446)
- fix misspelling of openmp-targets for amdgcn-amd-amdhsa paklui (1429)
- examples: Fix `opt` arg-passing for elastic and TTI georgebisbas (1425)

API

- FD: add missing shift option Leitevmd (1551)
- Refactor and enhance section profiling FabioLuporini (1547)
- GPU data streaming FabioLuporini (1520)
- builtins: initialize FD halo mloubout (1511)
- Fix custom coefficients on staggered grids EdCaunt (1497)
- Tensor grid fixup mloubout (1489)
- Added origin_map attribute to Grid EdCaunt (1461)
- Tensor API: improve behavior and interleaving with sympy. mloubout (1390)

Examples

- Example: add second-order viscoacoustic equations nogueirapeterson (1406)
- Updated cavity flow notebook for cfd examples kmn319 (1502)
- Notebooks with Damping, PML and HABC absorbing boudary conditions. felipeaugustogudes (1323)
- Examples: Add tests and damp safety checks for backward compatibility mloubout (1434)
- Move time block notebook to seismic/tutorials jkwashbourne (1441)
- examples: Fix `opt` arg-passing for elastic and TTI georgebisbas (1425)
- Time Blocking Notebook POC w/serialization and compression jkwashbourne (1424)
- examples: Patch plot_velocity FabioLuporini (1428)
- Self-adjoint seismic: Update correctness linearization plot jkwashbourne (1423)

Documentation

- Correct typo in examples/cfd/03_diffusion.ipynb jakubbober (1451)
- Replaced mini-web-app for slack invites to slacks native solution. ggorman (1439)

MPI

- builtins: initialize FD halo mloubout (1511)
- Fix halo update hoisting Leitevmd (1494)
- Fixup _hoist_halospots FabioLuporini (1496)
- Fixup _drop_halospots FabioLuporini (1495)
- Outhalo size warning rhodrin (1493)
- Fixing halo spot merge rule (issue 1459) Leitevmd (1482)
- MPI + OpenMP 5.0 italoaug (1363)
- mpi data gatherer rhodrin (1376)

GPU

- Skip viscoacoustic tests with openacc rhodrin (1559)
- GPU data streaming FabioLuporini (1520)
- Use omp_set_default_device for OpenMP GPU offloading italoaug (1517)
- Tidy gpu CI workflow rhodrin (1470)
- Fix MPI in Dockerfile.nvidia kenhester (1492)

Testing

- Skip viscoacoustic tests with openacc rhodrin (1559)
- Tweak ASV and GPU workflows rhodrin (1558)
- docker publish tweaks rhodrin (1548)
- Docker workflow updates rhodrin (1546)
- CI: Update gpu workflow rhodrin (1542)
- Migrate asv workflow to new runner rhodrin (1530)
- Move mpi examples to new runner rhodrin (1543)
- Migrate MPI CI rhodrin (1531)
- Add test that compares gradient equivalence navjotk (1519)
- ci: Disable dask tutos on OSX build FabioLuporini (1505)
- CI: skip dask on osc builds mloubout (1458)

Misc

- Update distributed requirement to <2021.2 dependabot (1557)
- Improve bitwise reproducibility ggorman (1438)
- Update to NVidia HPC 20.9; MPI Fixes kenhester (1526)
- perf: Add global gflopss to operator log georgebisbas (1462)
- Advisor miscellaneous improvements georgebisbas (1503)
- Updated dependencies in Dockerfile.nvidia: HPC SDK 20.9, cupy110 kenhester (1484)
- Improve detection and scheduling of CIREs FabioLuporini (1465)
- Intel Advisor jupyter notebook benchmarks/user/advisor scripts update jack-lascala (1456)
- Added sniffing Intel MPI as MPI version in compiler.py jack-lascala (1455)
- Moving and packaging the Intel Advisor 2020 scripts into benchmarks/user/advisor jack-lascala (1452)
- Moved from PGI to HPC SDK. New NVIDIA Based-Container. kenhester (1450)
- Intel Advisor 2020 automatic roofline tools jack-lascala (1440)
- Potentially smarter CIRE via cire-rotate option FabioLuporini (1430)
- Update pytest requirement from to >=3.6,<7.0 dependabot (1414)

Synopsis
- Performance optimizations in the symbolic layer and generated code for x86, GPU and MPI.
- Various minor correctness and performance bug fixes.
- Improvements to application developer API.
- Added new tutorial notebooks.
- Increased test coverage - particularly for MPI and GPU's.

Backwards compatibility breaks and deprecations
None
Changes
- Restrict pytest to < 6.0 rhodrin (1411)
- pip prod(deps): update distributed requirement from <2.20 to <2.22 dependabot (1402)
- MPI minor bug fixes rhodrin (1408)
- mac notebook fix rhodrin (1407)
- Run docker test mloubout (1398)
- docker: fix typos mloubout (1397)
- compiler: version checks avoid unreliable exception dbowman-ion (1386)
- Fix cross-loop blocking with imperfect nests FabioLuporini (1381)
- Cancel previous CI at new commit mloubout (1378)
- Heapification of temporaries FabioLuporini (1349)
- Fix missing halo exchange over invariants fields FabioLuporini (1359)
- tutorials: 00_index fix broken link georgebisbas (1356)
- CI: add autolog of PRs mloubout (1338)

🐛 Bug Fixes
- Fix issue 1298 FabioLuporini (1421)
- ir: Fix lowering of ConditionalDimension FabioLuporini (1419)
- mpi: Patch compute_loc_indices + test FabioLuporini (1416)
- Fix estimate_cost FabioLuporini (1405)
- Fix issue 1332 (fablup version) FabioLuporini (1389)
- checkpointing: set right size_ckp ofmla (1361)
- `Data` tweaks rhodrin (1369)
- Fix min storage FabioLuporini (1372)
- Avoid dererencing dangling pointers on GC'd objects tjb900 (1316)
- Replace subdimension in expressions mloubout (1345)
- Indexify with non-symbolic numeric indices mloubout (1343)
- Fix numeric indices mloubout (1341)

API
- Buffer: remove unused from args to allow multi buffers mloubout (1420)
- operator: Expose more opt knobs to the user level FabioLuporini (1387)
- Function/Expr shift utility mloubout (1377)
- Prevent sympy cache blowup mloubout (1366)
- Symbolic speedup mloubout (1362)

Examples
- examples: fix issue 1393 ofmla (1400)
- tutorials: add missing ignore tag in dask tutorial mloubout (1403)
- Dask tutorial with operator created only once ofmla (945)
- tests: Fix args and test viscoacoustic georgebisbas (1365)
- Add norm asserts for ssa iso notebooks jkwashbourne (1373)
- Make dt a solve property to use latest model parameters mloubout (1371)
- examples: Drop unnecessary NBVALs FabioLuporini (1367)
- Refine CFL condition in seismic mloubout (1348)
- Freesurface with subdomain (iso-acoustic) mloubout (1344)
- Improve solve speed for complicated equations mloubout (1342)
- elastic-fix mloubout (1339)

Documentation
- tutos: Inception of performance modes notebook georgebisbas (1358)
- Fix issue 1185 dabiged (1360)
- `ConditionalDimension`s notebook georgebisbas (1269)
- Add binder directory with requirements.txt including matplotlib jkwashbourne (1351)

MPI

- ci: Test all critical seismic examples with MPI FabioLuporini (1418)
- Single precision interpolation mloubout (1413)
- Update halo warning rhodrin (1370)
- Fix MPI benchmark json results jaimesouza (1295)
- Improve detection of redundant halo exchanges FabioLuporini (1352)

GPU
- Add MPI+GPU runs to CI georgebisbas (1251)
- Tweak CIRE for GPUs FabioLuporini (1409)
- Add docker publish mloubout (1388)
- Nvidia dockerfile FabioLuporini (1392)
- Tool get_gpu_info (using lshw, lspci) with test jack-lascala (1383)

Contributors
Many thanks to all the contributors to this release (last surname alphabetical order):

* George Bisbas (Imperial College London)
* David Bowman (ION)
* Tim Burgess (DUG)
* Jaime Freire de Souza
* Chris Dinneen
* Ken Hester (NVidia)
* Navjot Kukreja (Imperial College London)
* Giacomo La Scala
* Mathias Louboutin (Georgia Institute of Technology)
* Fabio Luporini (Devito Codes)
* Oscar Mojica (SENAI CIMATEC)
* Rhodri Nelson (Imperial College London)
* John Washbourne (Chevron)

Compiler:
- Improve CIRE
- Conditionals improvement
- Improve aliases detection

Misc:
- Updated minimum SymPy version requirement
- Refreshed docs
- Various bug fixes.
- Added more tests for CI and performance regression.

Benchmark:
- Improved asv
- Added support for adjoint/jacobian/jacobian_adjoint
- Improved JIT support for AMD, added AOMP compiler

Example:
- New Skew Self Adjoint operator
- Homogenize seismic examples and Model.


Many thanks to all the contributors to this release (last surname alphabetical order):
George Bisbas (Imperial College London)
Gerard Gorman (Imperial College London)
Mathias Louboutin (Georgia Institute of Technology)
Fabio Luporini (Devito Codes)
Oscar Mojica (SENAI CIMATEC)
Rhodri Nelson (Imperial College London)
Peterson Nogueira (SENAI CIMATEC)
João Henrique Speglich (SENAI CIMATEC)
Lauê Rami Souza Costa de Jesus (SENAI CIMATEC)
John Washbourne (Chevron)

pip
* Removed direct dependencies to enable PyPI

Air-Speed Velocity (ASV)
* Added performance regression testing via ASV

CI/CD
* Automatically start and stop Azure VM's for testing.

Benchmarking
* Additional performance metrics

Documentation
* New examples for TTI adjoint and viscoacoustic.
* Updated project logo and documentation version.
* Various improvements to examples/notebooks.

Bugfixes
* Loop fusion.
* MPI/SubDomain.

Many thanks to all the contributors to this release (last surname alphabetical order):
George Bisbas (Imperial College London)
Gerard Gorman (Imperial College London)
Mathias Louboutin (Georgia Institute of Technology)
Fabio Luporini (Devito Codes)
Oscar Mojica (SENAI CIMATEC)
Rhodri Nelson (Imperial College London)
Peterson Nogueira (SENAI CIMATEC)
João Henrique Speglich (SENAI CIMATEC)
Lauê Rami Souza Costa de Jesus (SENAI CIMATEC)
John Washbourne (Chevron)

GPU support

* Added OpenACC 2.6 code generation support.
* Added domain decomposition support for multi-node-multi-GPU via MPI and OpenACC.

Testing, CI/CD

* Added testing with PGI compiler suite v19.10 on NVidia K80 GPU's.
* Enhanced CI testing for OpenMP 5 offloading onto GPU's.
* Enhanced CI testing for domain decomposition with MPI.

Performance optimizations

* Wide range of performance optimizations to the Devito compiler to speed-up compile time (~2-5x speedup in compilation time).
* Various low-level performance and memory optimizations (e.g. identifying and removing redundant computations).

DSL extensions

* tools.Evaluable -> finite_differences.Evaluable
* Dynamic FD shortcuts (i.e. `u.dx(fd_order=2)`). Supports FD order, `x0` and `side`.

YASK and OPS backends

* After a great deal of consideration, we have decided to drop support for YASK (https://01.org/yask) and OPS (https://github.com/OP-DSL/OPS) as Devito backends. While YASK and OPS have many desirable features, the software engineering overhead (person-effort) to maintain feature-complete support proved unsustainable. We would like to take this opportunity to thank Chuck Yount (YASK) and Istvan Reguly (OPS) for their collaboration on both software development and algorithm development, which has been of great benefit to Devito.

Tutorials and documentation

* Revised FAQ - https://github.com/devitocodes/devito/wiki/FAQ
* New Jupyter notebooks:
* GPU tutorial.
* PML boundary condition tutorial.
* Normal Moveout (NMO) tutorial.
* Revised viscoelastic example.

General updates

* Added Developer Certificate of Origin to Devito contribution guidelines.
* General improvements to documentation, examples and tutorials.
* Several minor bug fixes.
* Increased test coverage.
* Improvements to test code coverage reporting.
* Added PGI compiler support.
* Added MPI support for the SubDomainSet feature.
* Additional runtime warnings for inadequate problem specifications, e.g. stencil radius greater than rank-domain width.
* Deprecate DEVITO_DLE, replaced by DEVITO_OPT.
* Deprecate DEVITO_OPENMP, replaced by DEVITO_LANGUAGE.

Many thanks to all the contributors to this release (last surname alphabetical order):

* George Bisbas (Imperial College London)
* Tim Burgess (DownUnder Geosolutions)
* Edward Caunt (Imperial College London)
* Gerard Gorman (Imperial College London)
* Mathias Louboutin (Georgia Institute of Technology)
* Fabio Luporini (Devito Codes)
* Rhodri Nelson (Imperial College London)
* Peterson Nogueira (SENAI CIMATEC and UFBA)
* Joao Speglich (SENAI CIMATEC)
* John Washbourne (Chevron)
* Philipp Witte (Georgia Institute of Technology)

Special thanks and congratulations to Ken Hester (NVidia), winner of the Devito hackathon at Rice OGHPC 2020, who used the Devito JIT backdoor to develop the prototype OpenACC code which informed the implementation of OpenACC code generation support within this release.