This feature release comes with a number of new features as well as quite a few fixes of bugs and stability issues.
Further down you will find a complete list of changes, after a short description of some of the most important changes:
- [Automatic input serialization in calculation and work functions](automatic-input-serialization-in-calculation-and-work-functions)
- [Improved interface for creating codes](improved-interface-for-creating-codes)
- [Support for running code in containers](support-for-running-code-in-containers)
- [Control daemon and processes from the API](control-daemon-and-processes-from-the-api)
- [REST API can serve multiple profiles](rest-api-can-serve-multiple-profiles)
- [Pluginable data storage backends](pluginable-data-storage-backends)
- [Full list of changes](full-list-of-changes)
Automatic input serialization in calculation and work functions
The inputs to `calcfunction`s and `workfunction`s are now automatically converted to AiiDA data types if they are one of the basic Python types (`bool`, `dict`, `Enum`, `float`, `int`, `list` or `str`).
This means that code that looked like:
python
from aiida.engine import calcfunction
from aiida.orm import Bool, Float, Int, Str
calcfunction
def function(switch, threshold, count, label):
...
function(Bool(True), Float(0.25), Int(10), Str('some-label'))
can now be simplified to:
python
from aiida.engine import calcfunction
from aiida.orm import Bool, Float, Int, Str
calcfunction
def function(switch, threshold, count, label):
...
function(True, 0.25, 10, 'some-label')
Improved interface for creating codes
The `Code` data plugin was a single class that served two different types of codes: "remote" codes and "local" codes.
These names "remote" and "local" have historically caused a lot of confusion.
Likewise, using a single class `Code` for both implementations also has led to confusing interfaces.
To address this issue, the functionality has been split into two new classes [`InstalledCode`](https://aiida.readthedocs.io/projects/aiida-core/en/v2.1.0/topics/data_types.html#installedcode) and [`PortableCode`](https://aiida.readthedocs.io/projects/aiida-core/en/v2.1.0/topics/data_types.html#portablecode), that replace the "remote" and "local" code, respectively.
The installed code represents an executable binary that is already pre-installed on some compute resource.
The portable code represents a code (executable plus any additional required files) that are stored in AiiDA's storage and can be automatically transfered to any computer before being executed.
Creating a new instance of these new code types is easy:
python
from pathlib import Path
from aiida.orm import InstalledCode, PortableCode
installed_code = InstalledCode(
label='installed-code',
computer=load_computer('localhost'),
filepath_executable='/usr/bin/bash'
)
portable_code = PortableCode(
label='portable-code',
filepath_files=Path('/some/path/code'),
filepath_executable='executable.exe'
)
Codes can also be created through the new `verdi` command `verdi code create`.
To specify the type of code to create, pass the corresponding entry point name as an argument.
For example, to create a new installed code, invoke:
bash
verdi code create core.code.installed
`
The options for each subcommand are automatically generated based on the code type, and so only options that are relevant to that code type will be prompted for.
The new code classes both subclass the `aiida.orm.nodes.data.code.abstract.AbstractCode` base class.
This means that both `InstalledCode`s and `PortableCode`s can be used as the `code` input for `CalcJob`s without problems.
The old `Code` class remains supported for the time being as well, however, it is deprecated and will be remove at some point.
The same goes for the `verdi code setup` command; please use `verdi code create` instead.
Existing codes will be automatically migrated to either an `InstalledCode` or a `PortableCode`.
It is strongly advised that you update any code that creates new codes to use these new plugin types.
Support for running code in containers
Support is added to run calculation jobs inside a container.
A containerized code can be setup through the CLI:
bash
verdi code create core.code.containerized \
--label containerized \
--image-name docker://alpine:3 \
--filepath-executable /bin/sh \
--engine-command "singularity exec --bind $PWD:$PWD {image_name}"
as well as through the API:
python
from aiida.orm import ContainerizedCode, load_computer
code = ContainerizedCode(
computer=load_computer('some-computer')
filepath_executable='/bin/sh'
image_name='docker://alpine:3',
engine_command='singularity exec --bind $PWD:$PWD {image_name}'
).store()
In the example above we use the [Singularity](https://singularity-docs.readthedocs.io/en/latest/) containerization technology.
For more information on what containerization programs are supported and how to configure them, please refer to the [documentation](https://aiida.readthedocs.io/projects/aiida-core/en/v2.1.0/topics/data_types.html#containerizedcode).
Control daemon and processes from the API
Up till now, the daemon and live processes could only easily be controlled through `verdi daemon` and `verdi process`, respectively.
In this release, modules are added to provide the same functionality through the Python API.
Daemon API
The daemon can now be started and stopped through the `DaemonClient` which can be obtained through the `get_daemon_client` utility function:
python
from aiida.engine.daemon.client import get_daemon_client
client = get_daemon_client()
By default, this will give the daemon client for the current default profile.
It is also possible to explicitly specify a profile:
python
client = get_daemon_client(profile='some-profile')
The daemon can be started and stopped through the client:
python
client.start_daemon()
assert client.is_daemon_running
client.stop_daemon(wait=True)
Process API
The functionality of `verdi process` to `play`, `pause` and `kill` is now made available through the `aiida.engine.process.control` module.
Processes can be played, paused or killed through the `play_processes`, `pause_processes`, and `kill_processes`, respectively.
The processes to act upon are defined through their `ProcessNode` which can be loaded using `load_node`.
python
from aiida.engine.process import control
processes = [load_node(<PK1>), load_node(<PK2>)]
pause_processes(processes) Pause the processes
play_processes(processes) Play them again
kill_processes(processes) Kill the processes
Instead of specifying an explicit list of processes, the functions also take the `all_entries` keyword argument:
python
pause_processes(all_entries=True) Pause all running processes
REST API can serve multiple profiles
Before, a single REST API could only serve data of a single profile at a time.
This limitation has been removed and a single REST API instance can now serve data from all profiles of an AiiDA instance.
To maintain backwards compatibility, the new functionality needs to be explicitly enabled through the configuration:
bash
verdi config set rest_api.profile_switching True
After the REST API is restarted, it will now accept the `profile` query parameter, for example:
bash
http://127.0.0.1:5000/api/v4/computers?profile=some-profile-name
`
If the specified is already loaded, the REST API functions exactly as without profile switching enabled.
If another profile is specified, the REST API will first switch profiles before executing the request.
If the profile parameter is specified in a request and the REST API does not have profile switching enabled, a 400 response is returned.
Pluginable data storage backends
Warning: this is beta functionality.
It is now possible to implement custom storage backends to control where all data of an AiiDA profile is stored.
To provide a data storage plugin, one should implement the `aiida.orm.implementation.storage_backend.StorageBackend` interface.
The default implementation provided by `aiida-core` is the `aiida.storage.psql_dos.backend.PsqlDosBackend` which uses a PostgreSQL database for the provenance graph and a [`disk-objectstore`](https://pypi.org/project/disk-objectstore/) container for repository files.
Storage backend plugins should be registered in the new entry point group `aiida.storage`.
The default storage backend `PsqlDosBackend` has the `core.psql_dos` entry point name.
The storage backend to be used for a profile can be specified using the `--db-backend` option in `verdi setup` and `verdi quicksetup`.
The entry point of the selected backend is stored in the `storage.backend` key of a profile configuration:
json
{
"profiles": {
"profile-name": {
"PROFILE_UUID": "",
"storage": {
"backend": "core.psql_dos",
"config": {}
},
"process_control": {},
"default_user_email": "aiidalocalhost",
"test_profile": false
},
}
At the moment, it is not quite clear if the abstract interface `StorageBackend` properly abstracts everything that is needed to implement any storage backend.
For the time being then, it is advised to subclass the `PsqlDosBackend` and replace parts required for the use-case, such as just replacing the file repository implementation.
Full list of changes
Features
- `Process`: Add hook to customize the `process_label` attribute [[5713]](https://github.com/aiidateam/aiida-core/pull/5713)
- Add the `ContainerizedCode` data plugin [[5667]](https://github.com/aiidateam/aiida-core/pull/5667)
- API: Add the `aiida.engine.processes.control` module [[5630]](https://github.com/aiidateam/aiida-core/pull/5630)
- `PluginVersionProvider`: Add support for entry point strings [[5662]](https://github.com/aiidateam/aiida-core/pull/5662)
- `verdi setup`: Add the `--profile-uuid` option [[5673]](https://github.com/aiidateam/aiida-core/pull/5673)
- Process control: Add the `revive_processes` method [[5677]](https://github.com/aiidateam/aiida-core/pull/5677)
- Process functions: Add the `get_source_code_function` method [[4554]](https://github.com/aiidateam/aiida-core/pull/4554)
- CLI: Improve the quality of `verdi code list` output [[5750]](https://github.com/aiidateam/aiida-core/pull/5750)
- CLI: Add the `verdi devel revive` command [[5677]](https://github.com/aiidateam/aiida-core/pull/5677)
- CLI: `verdi process status --max-depth` [[5727]](https://github.com/aiidateam/aiida-core/pull/5727)
- CLI: `verdi setup/quicksetup` store autofill user info early [[5729]](https://github.com/aiidateam/aiida-core/pull/5729)
- CLI: Add the `devel launch-add` command [[5733]](https://github.com/aiidateam/aiida-core/pull/5733)
- CLI: Make filename in `verdi node repo cat` optional for `SinglefileData` [[5747]](https://github.com/aiidateam/aiida-core/pull/5747)
- CLI: Add the `verdi devel rabbitmq` command group [[5718]](https://github.com/aiidateam/aiida-core/pull/5718)
- API: Add function to start the daemon [[5625]](https://github.com/aiidateam/aiida-core/pull/5625)
- `BaseRestartWorkChain`: add the `get_outputs` hook [[5618]](https://github.com/aiidateam/aiida-core/pull/5618)
- `CalcJob`: extend `retrieve_list` syntax with `depth=None` [[5651]](https://github.com/aiidateam/aiida-core/pull/5651)
- `CalcJob`: allow wildcards in `stash.source_list` paths [[5601]](https://github.com/aiidateam/aiida-core/pull/5601)
- Add global config option `rest_api.profile_switching` [[5054]](https://github.com/aiidateam/aiida-core/pull/5054)
- REST API: make the profile configurable as request parameter [[5054]](https://github.com/aiidateam/aiida-core/pull/5054)
- `ProcessFunction`: Automatically serialize Python base type inputs [[5688]](https://github.com/aiidateam/aiida-core/pull/5688)
- `BaseRestartWorkChain`: allow to override priority in `handler_overrides` [[5546]](https://github.com/aiidateam/aiida-core/pull/5546)
- ORM: add `entry_point` classproperty to `Node` and `Group` [[5437]](https://github.com/aiidateam/aiida-core/pull/5437)
- Add the `aiida.storage` entry point group [[5501]](https://github.com/aiidateam/aiida-core/pull/5501)
- Add the config option `storage.sandbox` [[5501]](https://github.com/aiidateam/aiida-core/pull/5501)
- Add the `InstalledCode` and `PortableCode` data plugins [[5510]](https://github.com/aiidateam/aiida-core/pull/5510)
- CLI: Add the `verdi code create` command group [[5510]](https://github.com/aiidateam/aiida-core/pull/5510)
- CLI: Add the `DynamicEntryPointCommandGroup` command group [[5510]](https://github.com/aiidateam/aiida-core/pull/5510)
- Add a client to connect to RabbitMQ Manamegement HTTP API [[5718]](https://github.com/aiidateam/aiida-core/pull/5718)
- `LsfScheduler`: add support for `num_machines` [[5153]](https://github.com/aiidateam/aiida-core/pull/5153)
- `JobResource`: add the `accepts_default_memory_per_machine` [[5642]](https://github.com/aiidateam/aiida-core/pull/5642)
- `AbstractCode`: add abstraction methods for command line parameters [[5664]](https://github.com/aiidateam/aiida-core/pull/5664)
- `ArithmeticAddCalculation`: Add the `metadata.options.sleep` input [[5663]](https://github.com/aiidateam/aiida-core/pull/5663)
- `DaemonClient`: add the `get_env` method [[5631]](https://github.com/aiidateam/aiida-core/pull/5631)
- Tests: Make daemon fixtures available to plugin packages [[5701]](https://github.com/aiidateam/aiida-core/pull/5701)
- `verdi plugin list`: Show which exit codes invalidate cache [[5710]](https://github.com/aiidateam/aiida-core/pull/5710)
- `verdi plugin list`: Show full help for input and output ports [[5711]](https://github.com/aiidateam/aiida-core/pull/5711)
Fixes
- `ArrayData`: replace `nan` and `inf` with `None` when dumping to JSON [[5613]](https://github.com/aiidateam/aiida-core/pull/5613)
- Archive: add missing migration of transport entry points [[5604]](https://github.com/aiidateam/aiida-core/pull/5604)
- `BaseRestartWorkChain`: fix `handler_overrides` ignoring `enabled=False` [[5598]](https://github.com/aiidateam/aiida-core/pull/5598)
- CLI: allow setting options for config without profiles [[5544]](https://github.com/aiidateam/aiida-core/pull/5544)
- CLI: normalize use of colors [[5547]](https://github.com/aiidateam/aiida-core/pull/5547)
- `Config`: fix bug in downgrade past version 6 [[5528]](https://github.com/aiidateam/aiida-core/pull/5528)
- `DaemonClient`: close `CircusClient` after call [[5631]](https://github.com/aiidateam/aiida-core/pull/5631)
- Engine: Do not call serializer for `None` values [[5694]](https://github.com/aiidateam/aiida-core/pull/5694)
- Engine: Do not let `DuplicateSubcriberError` except a `Process` [[5715]](https://github.com/aiidateam/aiida-core/pull/5715)
- ORM: raise when trying to pickle instance of `Entity` [[5549]](https://github.com/aiidateam/aiida-core/pull/5549)
- ORM: Return `None` in `get_function_source_code` instead of excepting [[5730]](https://github.com/aiidateam/aiida-core/pull/5730)
- Fix `get_entry_point` not raising even for duplicate entry points [[5531]](https://github.com/aiidateam/aiida-core/pull/5531)
- Fix: reference to command in message for `verdi storage maintain` [[5558]](https://github.com/aiidateam/aiida-core/pull/5558)
- Fix: `is_valid_cache` setter for `ProcessNode`s [[5583]](https://github.com/aiidateam/aiida-core/pull/5583)
- Fix exception when importing an archive into a profile with many nodes [[5740]](https://github.com/aiidateam/aiida-core/pull/5740)
- `Profile`: make definition of daemon filepaths dynamic [[5631]](https://github.com/aiidateam/aiida-core/pull/5631)
- Fixtures: Fix bug in reset of `empty_config` fixture [[5717]](https://github.com/aiidateam/aiida-core/pull/5717)
- `PsqlDosBackend`: ensure sqla sessions are garbage-collected on `close` [[5728]](https://github.com/aiidateam/aiida-core/pull/5728)
- `TrajectoryData`: Fix bug in `get_step_data` [[5734]](https://github.com/aiidateam/aiida-core/pull/5734)
- `ProfileManager`: restart daemon in `clear_profile` [[5751]](https://github.com/aiidateam/aiida-core/pull/5751)
Changes
- Mark relevant `Process` exit codes as `invalidates_cache=True`[[5709]](https://github.com/aiidateam/aiida-core/pull/5709)
- `TemplatereplacerCalculation`: Change exit codes to be in 300 range [[5709]](https://github.com/aiidateam/aiida-core/pull/5709)
- Add the prefix `core.` to all storage entry points [[5501]](https://github.com/aiidateam/aiida-core/pull/5501)
- `CalcJob`: Fully abstract interaction with `AbstractCode` in presubmit [[5666]](https://github.com/aiidateam/aiida-core/pull/5666)
- CLI: make label the default group list order in `verdi group list` [[5523]](https://github.com/aiidateam/aiida-core/pull/5523)
- Config: add migration to properly prefix storage backend [[5501]](https://github.com/aiidateam/aiida-core/pull/5501)
- Move query utils from `aiida.cmdline` to `aiida.tools` [[5630]](https://github.com/aiidateam/aiida-core/pull/5630)
- `SandboxFolder`: decouple the location from the profile [[5496]](https://github.com/aiidateam/aiida-core/pull/5496)
- `TemplatereplacerDoublerParser`: rename and generalize implementation [[5669]](https://github.com/aiidateam/aiida-core/pull/5669)
- `Process`: Allow `None` for input ports that are not required [[5722]](https://github.com/aiidateam/aiida-core/pull/5722)
Dependencies
- RabbitMQ: Remove support for v3.5 and older [[5718]](https://github.com/aiidateam/aiida-core/pull/5718)
- Relax `wrapt` requirement [[5607]](https://github.com/aiidateam/aiida-core/pull/5607)
- Set upper limit `werkzeug<2.2` [[5606]](https://github.com/aiidateam/aiida-core/pull/5606)
- Update requirement `click~=8.1` [[5504]](https://github.com/aiidateam/aiida-core/pull/5504)
Deprecations
- Deprecate `Profile.repository_path` [[5516]](https://github.com/aiidateam/aiida-core/pull/5516)
- Deprecate: `verdi code setup` and `CodeBuilder` [[5510]](https://github.com/aiidateam/aiida-core/pull/5510)
- Deprecate the method `aiida.get_strict_version` [[5512]](https://github.com/aiidateam/aiida-core/pull/5512)
- Remove use of legacy `Code` [[5510]](https://github.com/aiidateam/aiida-core/pull/5510)
Documentation
- Add section on basic performance benchmark with automated benchmark script [[5724]](https://github.com/aiidateam/aiida-core/pull/5724)
- Add `-U` flag to PostgreSQL database backup command [[5550]](https://github.com/aiidateam/aiida-core/pull/5550)
- Clarify excepted and killed calculations are not cached [[5525]](https://github.com/aiidateam/aiida-core/pull/5525)
- Correct snippet for workchain context nested keys [[5551]](https://github.com/aiidateam/aiida-core/pull/5551)
- Plugin package setup add PEP 621 example [[5626]](https://github.com/aiidateam/aiida-core/pull/5626)
- Remove note on disk space for caching [[5534]](https://github.com/aiidateam/aiida-core/pull/5534)
- Remove explicit release tag in Docker image name [[5671]](https://github.com/aiidateam/aiida-core/pull/5671)
- Remove example REST API extension with POST requests [[5737]](https://github.com/aiidateam/aiida-core/pull/5737)
- Resubmit a `Process` from a `ProcessNode` [[5579]](https://github.com/aiidateam/aiida-core/pull/5579)
Devops
- Add a notification for nightly workflow on fail [[5605]](https://github.com/aiidateam/aiida-core/pull/5605)
- CI: Remove `--use-feature` flag in `pip install` of CI [[5703]](https://github.com/aiidateam/aiida-core/pull/5703)
- Fixtures: Add `started_daemon_client` and `stopped_daemon_client` [[5631]](https://github.com/aiidateam/aiida-core/pull/5631)
- Fixtures: Add the `entry_points` fixture to dynamically add and remove entry points [[5745]](https://github.com/aiidateam/aiida-core/pull/5745)
- Refactor: `Process` extract `CalcJob` specific input handling from `Process` [[5539]](https://github.com/aiidateam/aiida-core/pull/5539)
- Refactor: remove unnecessary use of `tempfile.mkdtemp` [[5639]](https://github.com/aiidateam/aiida-core/pull/5639)
- Refactor: Remove internal use of various deprecated resources [[5716]](https://github.com/aiidateam/aiida-core/pull/5716)
- Refactor: Turn `aiida.manage.external.rmq` into a package [[5718]](https://github.com/aiidateam/aiida-core/pull/5718)
- Tests: remove legacy `tests/utils/configuration.py` [[5500]](https://github.com/aiidateam/aiida-core/pull/5500)
- Tests: fix the RPN work chains for the nightly build [[5529]](https://github.com/aiidateam/aiida-core/pull/5529)
- Tests: Manually stop daemon after `verdi devel revive` test [[5689]](https://github.com/aiidateam/aiida-core/pull/5689)
- Tests: Add verbose info if `submit_and_wait` times out [[5689]](https://github.com/aiidateam/aiida-core/pull/5689)
- Tests: Do not set default memory for `localhost` fixture [[5689]](https://github.com/aiidateam/aiida-core/pull/5689)
- Tests: Suppress RabbitMQ and developer version warnings [[5689]](https://github.com/aiidateam/aiida-core/pull/5689)
- Tests: Add the `EntryPointManager` exposed as `entry_points` fixture [[5656]](https://github.com/aiidateam/aiida-core/pull/5656)
- Tests: Only reset database connection at end of suite [[5641]](https://github.com/aiidateam/aiida-core/pull/5641)
- Tests: Suppress logging and warnings from temporary profile fixture [[5702]](https://github.com/aiidateam/aiida-core/pull/5702)