Libensemble

--------------

:Date: May 26, 2023

New capabilities:

* Enhance portability and simplify the assignment of procs/GPUs to worker resources 928 / 983
* Auto-detect GPUs across systems (inc. Nvidia, AMD, and Intel GPUs).
* Auto-determination of GPU assignment method by MPI runner or provided platform.
* Portable `auto_assign_gpus` / `match_procs_to_gpus` and `num_gpus` arguments added to the MPI executor submit.
* Add `set_to_gpus` function (similar to `set_to_slots`).
* Allow users to specify known systems via option or environment variable.
* Allow users to specify their own system configurations.
* These changes remove a number of tweaks that were needed for particular platforms.

* Resource management supports GPU and non-GPU simulations in the same ensemble. 993
* User's can specify `num_procs` and `num_gpus` in the generator for each evaluation.

* Pydantic models are used for validating major libE input (input can be provided as classes or dictionaries). 878
* Added option to store output and ensemble directories in a workflow directory. 982
* Simplify user function interface. Valid user functions can accept <4 parameters and return <3 values. 971
* New option to parse settings from **TOML**. 745
* New `dry_run` option to `libE()` that checks scripts are valid and returns. 987
* Added an option to the executor submit function to pre-execute a script in the task environment. 996

Breaking changes:

* Removed old Balsam Executor. 921
* Ensemble class moved from `libensemble.api` to `libensemble.ensemble`. 1003
* Default to one resource set per simulation in dynamic scheduling mode. 996

Documentation:

* Added type hints/annotations for major modules/functions. 823
* Added Polaris Guide. 930
* Added Frontier Guide. 909
* Added PBS example scripts. 956 930
* Streamlined and improved the readability of docs. 1004

Tests and Examples:

* Updated forces_gpu tutorial example. 956
* Source code edit is not required for the GPU version.
* Reports whether running on device or host.
* Increases problem size.
* Added versions with persistent generator and multi-task (GPU v non-GPU).
* Moved multiple tests, generators, and simulators to the community repo.
* Added ytopt example. And updated heFFTe example. 943
* Support Python 3.11 922

:Note:

* Tested platforms include Linux, MacOS, Windows and major systems: Frontier (OLCF), Polaris (ALCF), and Perlmutter (NERSC). The major system tests ran heterogeneous workflows.
* Recent testing was also carried out on Summit (IBM Power9/LSF), but this was not possible at time of release.
* Tested Python versions: (Cpython) 3.7, 3.8, 3.9, 3.10, 3.11.

:Known issues:

* See known issues section in the documentation.

-------------

:Date: October 13, 2022

New capabilities:

* New pair of utilities, `liberegister` and `libesubmit` (based on *PSI/J*), for easily preparing and launching libEnsemble workflows with local comms onto most machines and schedulers. 807
* New persistent support function to cancel sim_ids (`request_cancel_sim_ids`). 880
* `keep_state` option for persistent workers: this lets the manager know that the information being sent is intermediate. 880

Other enhancements:

* The Executor `manager_poll()` interface now sets consistent flags instead of literal strings. 877
* Some internal modules and the test suite now work on Windows. 869 888
* Specifying the `num_resource_sets` *libE_specs* option instead of `zero_resource_workers` is now recommended except when using a fixed worker/resource mapping. Use ``persis_info["gen_resources"]`` to assign persistent generator resources (default is zero). 905
* An extraneous warning removed. 903

:Note:

* Tested platforms include Linux, MacOS, Windows, Theta (Cray XC40/Cobalt), Summit (IBM Power9/LSF), Bebop (Cray CS400/Slurm), Swing (A100 GPU system), Perlmutter (HPE Cray EX with A100 NVIDIA GPUs). For Perlmutter, see "Known issues" below.
* Tested Python versions: (Cpython) 3.7, 3.8, 3.9, 3.10.

:Known issues:

* At time of testing on Perlmutter there was an issue running concurrent applications on a node, following a recent system update. This also affects previous versions of libEnsemble, and is being investigated.
* See known issues section in the documentation.

-------------

:Date: July 06, 2022

New capabilities:

* Support auto-detection of PBS node lists. 602
* Added configuration options for `libE_stats.txt` file. 743
* Support for `spawn` and `forkserver` multiprocessing start methods. 797

* Note that macOS no longer switches to using `fork`. macOS (since Python 3.8) and Windows default to
using `spawn`. When using `spawn`, we recommend placing calling script code in an ``if __name__ == "__main__":`` block.
The multiprocessing interface can be used to switch methods (https://docs.python.org/3/library/multiprocessing.html#multiprocessing.set_start_method).

Updates to example functions:

Added simple dynamic sampling example. 833
Added heFFTe example. 844
Regression tests separated into problem examples and functionality tests. 839

:Note:

* Tested platforms include Linux, MacOS, Theta (Cray XC40/Cobalt), Summit (IBM Power9/LSF), Bebop (Cray CS400/Slurm), Swing (A100 GPU system), Perlmutter (HPE Cray EX with A100 NVIDIA GPUs).
* Tested Python versions: (Cpython) 3.7, 3.8, 3.9, 3.10.

:Known issues:

* The APOSMM generator function has been noted to operate slower than expected with the `spawn` multiprocessing start method. For this reason we recommend using `fork` with APOSMM, when using `local` comms (`fork` is the default method on Linux systems).
* See known issues section in the documentation.

-------------

:Date: May 11, 2022

This release has minimal changes, but a large number of touched lines.

* Reformatted code for **black** compliance, including string normalization. 811, 814, 821
* Added Spock and Crusher guides. 802
* User can now set ``calc_status`` to any string (for output in libE_stats). 808
* Added a workflows community initiative file. 817

:Note:

* Tested platforms include Linux, MacOS, Theta (Cray XC40/Cobalt), Summit (IBM Power9/LSF), Bebop (Cray CS400/Slurm), Swing (A100 GPU system), Perlmutter (HPE Cray EX with A100 NVIDIA GPUs).
* Tested Python versions: (Cpython) 3.7, 3.8, 3.9, 3.10.

:Known issues:

* See known issues section in the documentation.

-------------

:Date: Apr 29, 2022

Featured new capabilities:

* New `Balsam` Executor with multi-site capability (run user applications on remote systems). 631, 729
* Support for `funcX` (place user functions on remote systems). 712 / 713
* Added partial support for concurrent/futures interface. (cancel(), cancelled(), done(), running(), result(), exception() and context manager) 719

Breaking API / helper function changes:

See "Updating for libEnsemble v0.9.0" wiki for details:
https://github.com/Libensemble/libensemble/wiki/Updating-for-libEnsemble-v0.9.0

* Scheduler options moved from `alloc_specs['user']` to `libE_specs`. 790
* `BalsamMPIExecutor` is now `LegacyBalsamMPIExecutor`. 729
* The exit_criteria `elapsed_wallclock_time` has been renamed `wallclock_max`. 750 (with a deprecation warning)
* Clearer and consistent naming of libE-protected fields in history array. 760

Updates to example functions:

* Moved some examples to new repository - [libe-community-examples](https://github.com/Libensemble/libe-community-examples) (VTMOP, DEAP, DeepDriveMD). #716, 721, 726
* Updates to Tasmanian examples to include asynchronous generator example. 727 / 732
* Added multi-task, multi-fidelity optimization regression tests using `ax`. 717 / 720

Other functionality enhancements:

* Non-blocking option added for persistent user function receives. 752
* Added `match_slots` option to resource scheduler. 746

Documentation:

* Added tutorial on assigning tasks to GPUs. 768
* Refactored Executor tutorial for simplicity. 749
* Added Perlmutter guide. 728
* Added Slurm guide. 728
* Refactored examples and tutorials - added exercises. 736 / 737
* Updated history array documentation with visual workflow example. 723

:Note:

* Tested platforms include Linux, MacOS, Theta (Cray XC40/Cobalt), Summit (IBM Power9/LSF), Bebop (Cray CS400/Slurm), Swing (A100 GPU system), Perlmutter (HPE Cray EX with A100 NVIDIA GPUs).
* Tested Python versions: (Cpython) 3.7, 3.8, 3.9, 3.10.

:Known issues:

* Open-MPI does not work with direct MPI job launches in ``mpi4py`` comms mode,
since it does not support nested MPI launches.
(Either use local mode or the Balsam Executor.)
* See known issues section in the documentation for more issues.

-------------

:Date: Oct 20, 2021

Featured new capabilities:

* Variable resource workers (dynamic reassignment of resources to workers). 643
* Alternative libE interface. An Ensemble object is created and can be parameterized by a YAML file. 645
* Improved support classes/functions for alloc/gen/sims and executors.
* Many new example generator/simulators and workflows.

Breaking API / helper function changes:

See "Updating for libEnsemble v0.8.0" wiki for details:
https://github.com/Libensemble/libensemble/wiki/Updating-for-libEnsemble-v0.8.0

* Resources management is now independent of the executor. 345
* The ``'persis_in'`` field has been added to gen_specs/sim_specs (instead of being hard-coded in alloc funcs). 626 / 670
* ``alloc support`` module is now a class. 643 / 656
* ``gen_support`` module is replaced by Persistent Worker support module (now a class). 609 / 671
* Remove ``libE_`` prefix from the logger. 608
* ``avail_worker_ids`` function should specify ``EVAL_GEN_TAG`` or ``EVAL_SIM_TAG`` instead of ``True``. 615 643
* Pass ``libE_info`` to allocation functions (allows more flexibility for user and efficiency improvements). 672
* ``'given_back'`` is now a protected libEnsemble field in the manager's history array. 651
* Several name changes to functions and parameters (See the wiki above for details). 529 / 659

Updates to example functions:

* Suite of distributed optimization methods for minimizing sums of convex functions. 647 / 649. Methods include:

* primal-dual sliding (https://arxiv.org/pdf/2101.00143).
* N-agent, or distributed gradient descent w/ gradient tracking (https://arxiv.org/abs/1908.11444).
* proximal sliding (https://arxiv.org/abs/1406.0919).

* Added batched construction for Tasmanian example. 644
* Added Tasmanian dependency to Spack package. spack/spack25762
* Added VTMOP source code and example usage. 676
* Added a multi-fidelity persistent_gp regression test. 683 / 684
* Added a DeepDriveMD inspired workflow. 630
* Created a persistent sim example. 614 / 615
* Added an example where cancellations are given from the alloc func. 677

Other functionality changes:

* A helper function for generic task polling loop has been added. 572 / 612
* Break main loop now happens when sim_max is returned rather than given out. 624
* Enable a final communication with gen. 620 / 628
* Logging updates - includes timestamps, enhanced debug logging, and libEnsemble version. 629 / 674

:Note:

* Tested platforms include Linux, MacOS, Theta (Cray XC40/Cobalt), Summit (IBM Power9/LSF), Bebop (Cray CS400/Slurm), Swing (A100 GPU system).
* Tested Python versions: (Cpython) 3.6, 3.7, 3.8, 3.9, 3.10 []_.

.. [] A reduced set of tests were run for python 3.10 due to some unavailable test dependencies at time of release.

:Known issues:

* Open-MPI does not work with direct MPI job launches in ``mpi4py`` comms mode,
since it does not support nested MPI launches.
(Either use local mode or the Balsam Executor.)
* See known issues section in the documentation for more issues.