Reframe-hpc

ReFrame 2.17 Release Notes

New features and enhancements

- Add support for selecting tests using regular expressions (664)
- Add logging handler for sending records to syslog (675)
- Remove support for passing options to the `RegressionTest.compile()` method (697)
- Support command substitution in variable expansion (683)
- Drop support for setting flags in environment (678)
- Extend `Job` API for retrieving the list of nodes assigned to the regression test (660)
- Add support for directories in `keep_files` attribute (673)

Bug fixes

- Define the set of run environments correctly when `-p` option is used with regular expressions (716)
- Fix Slurm backend failures when trying node lists with inexistent nodes (713)
- Fix unit test failure with EBUSY when running on NFS filesystem (715)
- Fix error message in case of performance errors (704)
- Fix deprecation warnings with Python 3.7 (711)
- Correct the units of the kernel latency test (703)
- Fix flexible task allocation bug ignoring Slurm partition defined in `access` (693)
- Use the default Slurm partition with `--flex-alloc-tasks` if none is specified (676)
- Fix crash in failure report when test is killed due to timeout (680)
- Fix bug in 'keep_files' unit tests (672)

Regression tests

- Add OSU Allreduce test (686)
- Add Intel Vtune test (714)
- Add description to the collective halo tests (690)
- Add Intel Inspector test (710)
- Add Fulen as a supported system in CSCS config (684)
- Add MPI halo cell exchange benchmark (666)
- Enlarge buffer sizes for H2D and D2D multi GPU tests (689)
- Extend DGEMM test to build with OpenBLAS (691)
- Add test for COSMO post-processing with Fieldextra (685)
- Change resources dir for GROMACS test (695)
- Fix tags of flexible tests + other improvements (681)
- Add test for gperftools (670)
- Add CPU affinity test (657)
- Add test for NERSC IPM (631)
- Update maintenance tests (679)
- Extend DGEMM test for using MKL (662)

Other

- Reenable Dom in the Jenkinsfile (709)
- Disable Dom from running in the ci (696)
- Add documentation for regular expression support in test selection options (717)
- Move documentation to readthedocs.org (705)
- Remove link to prior to 2.6.1 documentation (687)

ReFrame 2.16 Release Notes

New features and enhancements

- Add support for setting a minimum number of required tasks in tests with flexible task allocation (652)
- Add command line option for increasing the verbosity level of output (650)
- Add command line options for showing the system and environments configuration (633)
- Print error information when a modules system operation fails (628)
- Add support for units in performance references (576)
- Add a new `count_uniq()` sanity function (584)

Bug fixes

- Improve handling of bad tests to avoid frontend crashing (623)
- Fix bug in variable names in Slurm backend (639)
- Do not crash if Slurm node does not contain partition information (621)
- Check if 'UnavailableNodes' are down to cancel job (599)
- Fix deadlock in the flexible task allocation when the node list output is too large (614)

Regression tests

- Port remaining tests to the new syntax (656)
- Add Likwid benchmarks for cache, memory and cross-socket bandwidths (647)
- Add new `MPI_Init_thread()` check (620)
- Add benchmark measuring CUDA kernel launch latency (646)
- Add HPX 1D stencil tests (617)
- Add tests for SPEC-ACCEL benchmarks (618)
- Add test for distributed training with TensorFlow and Horovod (637)
- Add flexible HPCG test (604)
- Add HPX hello world test (616)
- Add allocation speed check (597)
- Fail if test is not compiled with OpenACC (615)
- Convert `GpuBandwidthCheck` to a flexible diagnostic check (586)
- Testing the environment of the login nodes when accessed remotely over SSH (583)
- Promote DGEMM to a multi-node check (530)
- Enable NCO and CDO for the C2SM-RCM environments (624)
- Make Alltoallv tests non-strict (622)
- Add specific tag for MCH tests (594)
- Remove `ifdef USE_MPI` from the MCH OpenACC/CUDA/MPI/C++ test (585)
- Set `exclusive_access` for Kesch tests (591)
- Adapt MCH regression tests to C2SM-RCM environments (513)
- Adapt CP2K test (596)

Other

- Make `.pep8speaks.yml` consistent with coding style (649)
- Add option to disable CSCS CI tests (648)
- Enable pep8speaks for code style checking in the repository (626)
- Use integer comparison for ReFrame versions in deployment script (600)
- Replace python 3.7-dev with 3.7 in `.travis.yml` (588)
- Document verbosity option (665)
- Add link for the automatic invites to Slack (661)
- Update README with contact info (642)
- Move SC18 presentation & demo to docs (606)

ReFrame 2.15 Release Notes

New features and enhancements

- Update retries stage/output directory naming convention (580)
- Full support for flexible task allocation (522)
- Add new command-line option for more verbose test listing (506)
- List only the checks that are supported on the current system (487)
- Use different stage and output directories for different retries of the same test (464)

Bug fixes

- Directly assign to `num_tasks` if `--flex-alloc-tasks` has an integer value (558)
- Show also hidden partitions in Slurm, when trying to flexibly allocate tasks (553)
- Fix redundant export statements (448)
- Fix retry report when test fails before the setup phase (535)
- Do not silently abort tasks in hold when all currently running tasks fail (533)
- Remove the validation of the name of the test's base classes (480)
- Remove time limit from build jobs (474)

Regression tests

- Update NAMD test (574)
- Run libsci_acc and libsci_resolve tests on compute nodes as well (581)
- Update GROMACS test for Dom (579)
- Ruby 2.2.2 NArray test (575)
- Add test checking if `CRAY_CPU_TARGET` is set (572)
- Update MAGMA tests (571)
- Update CP2K test (559)
- Adapt the GPU burn test for multi-gpu nodes (564)
- Disable Trilinos tests which fail due to the corresponding module (551)
- Adapt usertool tests to new syntax (552)
- Update GROMACS performance numbers (548)
- Remove Monch from supported systems (537)
- Remove read-only files attribute from the CP2K check (529)
- Remove wildcards from programming environment specification in tests (534)
- Fix NetCDF test on Kesch (523)
- Sync OpenMP versions for Dom/Daint in notool test (519)
- Add MCH CUDA stress test (514)
- Adapt boost tests to latest pe18.08 (518)
- Adapt the TensorFlow module name for PE 18.08 (516)
- Adapt Score-P modules to PE 18.08 (515)
- Add MCH halo exchange benchmark (509)
- Adapt IOR checks to build systems (505)
- Add MCH `alltoallv` check (347)
- Adjust LAMMPS performance values (504)
- Adapt the LAMMPS test to PE18.08 (502)
- Adapt QE test to PE18.08 (501)
- Remove `PrgEnv-gnu` support from the Scalapack test on Kesch (500)
- Adapt NetCDF test to Kesch (482)
- New OpenACC/CUDA/C++ test from MCH (342)
- Adapt Kesch tests to final CSCS prog. environments (479)
- Publish MAGMA tests and adapt to PE 18.08 (463)
- Remove the matrix file from version control (484)
- Set required version for tests using build systems (476)
- Add matrix file add20.rua in the src dir (481)
- Adapt trilinos regression test to PE18.08 and build systems (466)
- Adapt gpu direct test to build systems (462)
- Adapt CUDA NVML test to build systems (461)
- Increase GROMACS sanity check tolerance to 0.1% (459)
- Replace FFLAGS with FCFLAGS in Makefiles (457)

Other

- Add C2SM environments on Kesch (512)
- Re-enable running modified tests on Daint (517)
- Adapt Kesch settings (477)
- Automate deployment and generation of release notes (460)
- Document the `--flex-alloc-tasks` feature (582)
- Add User Lab Day slides to documentation (520)
- Update documentation on the topic of filtering tests by system (492)

ReFrame 2.14 Release Notes

New features and enhancements

- Add support for Autotools-based projects (452)
- Add support for CMake (447)
- Add a new infrastructure for build systems (340)
- Setting compilation flags directly in the programming environment is now deprecated.
- Passing arguments to the `compile()` method is now deprecated.
- For more information on how to use the new build system feature, please refer to the [tutorial](https://eth-cscs.github.io/reframe/tutorial.html#customizing-the-compilation-phase) and the [reference guide](https://eth-cscs.github.io/reframe/reference.html#build-systems).
- Evaluate and log all performance values before asserting them (359)
- Allow purging modules before running regression tests (399)

Bug fixes

- Fix reading of `logdir` setting from system config (445)
- Convert message for inexistent test files to warning (440)
- Do not load environments twice (441)
- Add a persistent version of rmtree() (437)
- Filter out empty strings before writing preamble (410)
- Fix bug in unloading of conflicting modules (405)
- Enable sphinx versioning only for releases (361)
- Change the resources dir path (358)
- Fix expected OpenMP versions for Dom (350)
- Comment out Graylog handler from CSCS config (352)
- Fix `site_configuration` in docs (344)

Regression tests

- Adapt stream microbenchmark test to build systems (420)
- Adapt prgenv test to build systems (449)
- Fix sourcesdir for the scorep test (455)
- Update OSU microbenchmark test to new syntax and build system (442)
- Adapt GpuBandwidthTest to build_systems (438)
- Adapt perftools to use build systems (439)
- Adapt helloworld tests to build systems (416)
- Adapt netcdf tests to build systems (434)
- Adapt cdo test to PE 18.07 (431)
- Adapt ddt regression test to PE 18.07 (425)
- Adapt cuda_gdb to build systems (428)
- Adapt notools check to build systems (427)
- Adapt hdf5 tests to build systems (422)
- Adapt libsci_resolve test to PE 18.07 (424)
- Adapt scorep test to PE 18.07 (426)
- Adapt cuda checks to build systems (421)
- Adapt dgemm test to use build systems (418)
- Adapt the scalapack tests to use build systems (417)
- Adapt haswell_fma to buildsystems style (415)
- Adapt nvprof test to PE 18.07 (414)
- Adapt the nco test for PE 18.07 (412)
- Adapt Tensorflow tests for PE 18.07 (409)
- Adapt the Boost Python test for PE 18.07 (408)
- Adapt Paraview check for PE 18.07 (401)
- Adapt the petsc test for PE 18.07 (404)
- Update PrgEnv-pgi definitions on Kesch (348)
- New flexible CUDA memtest check (346)

Other

- Add testing for python3.7 in travis (356)
- Remove monch from the machinesList (436)
- New programming environments for Kesch (432)
- Temporarily disable running of checks on Daint (406)
- Check added/modified regression tests w.r.t master (353)
- Update documentation for the build systems feature (433)

ReFrame 2.13 Release Notes



API Changes

- Enhanced syntax of regression tests using decorators that eliminate the need of boilerplate code (264, 294)
- Support for specifying compatible ReFrame versions for tests (287)
- Enhance behavior of `pre_run` and `post_run` attributes, allowing to specify multiple job steps (333)

Framework

- Redesign of performance logging (213)
- Performance logging can now be configured from the configuration file
- Two different backends: file logging, Graylog
- More logging attributes added for formatting log messages (213, 337)
- Support for PBS job scheduler (316)
- Support for TMod4 (282)
- Include name of the regression test file to the list of checks (288)
- Better filtering of tests (319)
- Add a knob for controlling the 'sacct'/'squeue' ratio in Slurm 'sacct' backend (289)
- Migrate to `pytest` (231)
- Bug fixes: 293, 305, 307, 308, 312, 318, 323, 325


Regression tests

- New regression tests and enhancements: 281, 284, 286, 310, 311
- Bug fixes: 309, 330, 331, 332

ReFrame 2.12 Release Notes

API Changes

- `post_run` and `pre_run` attributes removed from the `Job` class
- Support for wildcards in `valid_prog_environs`


Framework

- Support for automatic retry of failing tests (new `--max-retries` option)
- Support for custom configuration files (new `-C` option)
- Poll rate of the asynchronous execution policy is now lowered
- Unique test names are now enforced
- Environment variables are expanded in the `-c` option
- Documentation fixes and updates
- Internal abstraction layer for treating ReFrame versions
- Bug fixes: 192, 207, 221, 266, 274, 279


Regression tests

- New regression test for TensorFlow
- New flexible MPI Hello World test
- Add numerical sanity checks for Amber, CP2K, CPMD, GROMACS, LAMMPS, NAMD
- Corrected use of scoped dict key updates in the multi-GPU CUDA test
- Bug fixes: 242