This release comes with a number of improvements, some of the more useful and important of which are quickly highlighted.
A full list of changes can be found below.
- [Process function improvements](process-function-improvements)
- [Scheduler plugins: including `environment_variables`](scheduler-plugins-including-environment_variables)
- [`WorkChain`: conditional predicates should return boolean-like](workchain-conditional-predicates-should-return-boolean-like)
- [Controlling usage of MPI](controlling-usage-of-mpi)
- [Add support for Docker containers](add-support-for-docker-containers)
- [Exporting code configurations](exporting-code-configurations)
- [Full list of changes](full-list-of-changes)
- [Features](features)
- [Fixes](fixes)
- [Deprecations](deprecations)
- [Changes](changes)
- [Documentation](documentation)
- [DevOps](devops)
- [Dependencies](dependencies)
- [New contributors](new-contributors)
Process function improvements
A number of improvements in the usage of process functions, i.e., `calcfunction` and `workfunction`, have been added.
Each subsection title is a link to the documentation for more details.
[Variadic arguments](https://aiida.readthedocs.io/projects/aiida-core/en/latest/topics/processes/functions.html#variadic-and-keyword-arguments)
Variadic arguments can be used in case the function should accept a list of inputs of unknown length.
Consider the example of a calculation function that computes the average of a number of `Int` nodes:
python
calcfunction
def average(*args):
return sum(args) / len(args)
result = average(*(1, 2, 3))
[Automatic type validation](https://aiida.readthedocs.io/projects/aiida-core/en/latest/topics/processes/functions.html#type-validation)
Type hint annotations can now be used to add automatic type validation to process functions.
python
calcfunction
def add(x: Int, y: Int):
return x + y
add(1, 1.0) Passes
add(1, '1.0') Raises an exception
Since the Python base types (`int`, `str`, `bool`, etc.) are automatically serialized, these can also be used in type hints.
The following example is therefore identical to the previous:
python
calcfunction
def add(x: int, y: int):
return x + y
[Docstring parsing](https://aiida.readthedocs.io/projects/aiida-core/en/latest/topics/processes/functions.html#docstring-parsing)
The `calcfunction` and `workfunction` generate a `Process` of the decorated function on-the-fly.
In doing so, it automatically defines the `ProcessSpec` that is normally done manually, such as for a `CalcJob` or a `WorkChain`.
Before, this would just define the ports that the function process accepts, but the `help` attribute of the port would be left empty.
This is now parsed from the docstring, if it can be correctly parsed:
python
calcfunction
def add(x: int, y: int):
"""Add two integers.
:param x: Left hand operand.
:param y: Right hand operand.
"""
return x + y
assert add.spec().inputs['a'].help == 'Left hand operand.'
assert add.spec().inputs['b'].help == 'Right hand operand.'
This functionality is particularly useful when exposing process functions in work chains.
Since the process specification of the exposed function will be automatically inherited, the user can inspect the `help` string through the builder.
The automatic documentation produced by the Sphinx plugin will now also display the help string parsed from the docstring.
[Nested labels for output nodes](https://aiida.readthedocs.io/projects/aiida-core/en/latest/topics/processes/functions.html#return-values)
The keys in the output dictionary can now contain nested namespaces:
python
calcfunction
def add(alpha, beta):
return {'nested.sum': alpha + beta}
result = add(Int(1), Int(2))
assert result['nested']['sum'] == 3
[As class member methods](https://aiida.readthedocs.io/projects/aiida-core/en/latest/topics/processes/functions.html#as-class-member-methods)
Process functions can now be defined as class member methods of work chains:
python
class CalcFunctionWorkChain(WorkChain):
classmethod
def define(cls, spec):
super().define(spec)
spec.input('x')
spec.input('y')
spec.output('sum')
spec.outline(
cls.run_compute_sum,
)
staticmethod
calcfunction
def compute_sum(x, y):
return x + y
def run_compute_sum(self):
self.out('sum', self.compute_sum(self.inputs.x, self.inputs.y))
The function should be declared as a `staticmethod` and it should not include the `self` argument in its function signature.
It can then be called from within the work chain as `self.function_name(*args, **kwargs)`.
Scheduler plugins: including `environment_variables`
The `Scheduler` base class implements the concrete method `_get_submit_script_environment_variables` which formats the lines for the submission script that set the environment variables that were defined in the `metadata.options.environment_variables` input.
Before it was left up to the plugins to actually call this method in the `_get_submit_script_header`, but this is now done by the base class in the `get_submit_script`.
You can now remove the call to `_get_submit_script_environment_variables` from your scheduler plugins, as the base class will take care of it.
A deprecation warning is emitted if the base class detects that the plugin is still calling it manually.
See the [pull request](https://github.com/aiidateam/aiida-core/pull/5948) for more details.
`WorkChain`: conditional predicates should return boolean-like
Up till now, work chain methods that are used as the predicate in a conditional, e.g., `if_` or `while_` could return any type.
For example:
python
class SomeWorkChain(WorkChain):
classmethod
def define(cls, spec):
super().define(spec)
spec.outline(if_(cls.some_conditional)())
def some_conditional(self):
if self.ctx.something == 'something':
return True
The `some_conditional` method is used as the "predicate" of the `if_` conditional.
It returns `True` or `None`.
Since the `None` value in Python is "falsey", it would be considered as returning `False`.
However, this duck-typing could accidentally lead to unexpected situations, so we decided to be more strict on the return type.
As of now, a deprecation warning is emitted if the method returns anything that is not "boolean-like", i.e., does not implement the `__bool__` method.
If you see this warning, please make sure to return a boolean, like the built-ins `True` or `False`, or a `numpy.bool` or `aiida.orm.Bool`.
See the [pull request](https://github.com/aiidateam/aiida-core/pull/5924) for more details.
Controlling usage of MPI
It is now possible to define on a code object whether it should be run with or without MPI through the `with_mpi` attribute.
It can be set from the Python API as `AbstractCode(with_mpi=with_mpi)` or through the `--with-mpi / --no-with-mpi` option of the `verdi code create` CLI command.
This option adds a manner to control the use of MPI in calculation jobs, in addition to the existing ones defined by the `CalcJob` plugin and the `metadata.options.withmpi` input.
For more details on how these are controlled and how conflicts are handled, please refer to [the documentation](https://aiida.readthedocs.io/projects/aiida-core/en/latest/topics/calculations/usage.html#controlling-mpi).
Add support for Docker containers
Support is added for running calculation within Docker containers.
For example, to run Quantum ESPRESSO pw.x in a Docker container, write the following file to `config.yml`:
yaml
label: qe-pw-on-docker
computer: localhost
engine_command: docker run -i -v $PWD:/workdir:rw -w /workdir {image_name} sh -c
image_name: haya4kun/quantum_espresso
filepath_executable: pw.x
default_calc_job_plugin: quantumespresso.pw
use_double_quotes: false
wrap_cmdline_params: true
and run the CLI command:
verdi code create core.code.containerized --config config.yml --non-interactive
This should create a `ContainerizedCode` that you can now use to launch a `PwCalculation`.
For more details, please refer to [the documentation](https://aiida.readthedocs.io/projects/aiida-core/en/latest/topics/data_types.html#supported-container-technologies).
Exporting code configurations
It is now possible to export the configuration of an existing code through the `verdi code export` command.
The produced YAML file can be used to recreate the code through the `verdi code create` command.
Note that you should use the correct subcommand based on the type of the original code.
For example, if it was an `InstalledCode` you should use `verdi code create core.code.installed`.
For the legacy `Code` instances, you should use `verdi code setup`.
See the [pull request](https://github.com/aiidateam/aiida-core/pull/5860) for more details.
Full list of changes
Features
- `AbstractCode`: Add the `with_mpi` attribute [[5922]](https://github.com/aiidateam/aiida-core/pull/5922)
- `ContainerizedCode`: Add support for Docker images to use as `Code` for `CalcJob`s [[5841]](https://github.com/aiidateam/aiida-core/pull/5841)
- `InstalledCode`: Allow relative path for `filepath_executable` [[5879]](https://github.com/aiidateam/aiida-core/pull/5879)
- CLI: Allow specifying output filename in `verdi node graph generate` [[5897]](https://github.com/aiidateam/aiida-core/pull/5897)
- CLI: Add `--timeout` option to all `verdi daemon` commands [[5966]](https://github.com/aiidateam/aiida-core/pull/5966)
- CLI: Add the `verdi calcjob remotecat` command [[4861]](https://github.com/aiidateam/aiida-core/pull/4861)
- CLI: Add the `verdi code export` command [[5860]](https://github.com/aiidateam/aiida-core/pull/5860)
- CLI: Improved customizability and scriptability of `verdi storage maintain` [[5936]](https://github.com/aiidateam/aiida-core/pull/5936)
- CLI: `verdi quicksetup`: Further reduce required user interaction [[5768]](https://github.com/aiidateam/aiida-core/pull/5768)
- CLI: `verdi computer test`: Add test for login shell being slow [[5845]](https://github.com/aiidateam/aiida-core/pull/5845)
- CLI: `verdi process list`: Add `exit_message` as projectable attribute [[5853]](https://github.com/aiidateam/aiida-core/pull/5853)
- CLI: `verdi node delete`: Add verbose list of pks to be deleted [[5878]](https://github.com/aiidateam/aiida-core/pull/5878)
- CLI: Fail command if `--config` file contains unknown key [[5939]](https://github.com/aiidateam/aiida-core/pull/5939)
- CLI: `verdi daemon status`: Do not except when no profiles are defined [[5874]](https://github.com/aiidateam/aiida-core/pull/5874)
- ORM: Add unary operations `+`, `-` and `abs` to `NumericType` [[5946]](https://github.com/aiidateam/aiida-core/pull/5946)
- Process functions: Support class member functions as process functions [[4963]](https://github.com/aiidateam/aiida-core/pull/4963)
- Process functions: Infer argument `valid_type` from type hints [[5900]](https://github.com/aiidateam/aiida-core/pull/5900)
- Process functions: Parse docstring to set input port help attribute [[5919]](https://github.com/aiidateam/aiida-core/pull/5919)
- Process functions: Add support for variadic arguments [[5691]](https://github.com/aiidateam/aiida-core/pull/5691)
- Process functions: Allow nested output namespaces [[5954]](https://github.com/aiidateam/aiida-core/pull/5954)
- `Process`: Store JSON-serializable metadata inputs on the node [[5801]](https://github.com/aiidateam/aiida-core/pull/5801)
- `Port`: Add the `is_metadata` keyword [[5801]](https://github.com/aiidateam/aiida-core/pull/5801)
- `ProcessBuilder`: Include metadata inputs in `get_builder_restart` [[5801]](https://github.com/aiidateam/aiida-core/pull/5801)
- `StructureData`: Add `mode` argument to `get_composition` [[5926]](https://github.com/aiidateam/aiida-core/pull/5926)
- `Scheduler`: Allow terminating job if submission script is invalid [[5849]](https://github.com/aiidateam/aiida-core/pull/5849)
- `SlurmScheduler`: Detect broken submission scripts for invalid account [[5850]](https://github.com/aiidateam/aiida-core/pull/5850)
- `SlurmScheduler`: Parse the `NODE_FAIL` state [[5866]](https://github.com/aiidateam/aiida-core/pull/5866)
- `WorkChain`: Add dataclass serialisation to context [[5833]](https://github.com/aiidateam/aiida-core/pull/5833)
- `IcsdDbImporter`: Add `is_theoretical` tag to queried entries [[5868]](https://github.com/aiidateam/aiida-core/pull/5868)
Fixes
- CLI: Prefix the `verdi data` subcommands with `core.` [[5846]](https://github.com/aiidateam/aiida-core/pull/5846)
- CLI: Respect config log levels if `--verbosity` not explicitly passed [[5925]](https://github.com/aiidateam/aiida-core/pull/5925)
- CLI: `verdi config list`: Do not except if no profiles are defined [[5921]](https://github.com/aiidateam/aiida-core/pull/5921)
- CLI: `verdi code show`: Add missing code attributes [[5916]](https://github.com/aiidateam/aiida-core/pull/5916)
- CLI: `verdi quicksetup`: Fix error incorrect role when creating database [[5828]](https://github.com/aiidateam/aiida-core/pull/5828)
- CLI: Fix error in `aiida.cmdline.utils.log.CliFormatter` [[5957]](https://github.com/aiidateam/aiida-core/pull/5957)
- Daemon: Fix false-positive of stopped daemon in `verdi daemon status` [[5862]](https://github.com/aiidateam/aiida-core/pull/5862)
- `DaemonClient`: Fix and homogenize use of `timeout` in client calls [[5960]](https://github.com/aiidateam/aiida-core/pull/5960)
- `ProcessBuilder`: Fix bug in `_recursive_merge` [[5801]](https://github.com/aiidateam/aiida-core/pull/5801)
- `QueryBuilder`: Catch new exception raised by `sqlalchemy>=1.4.45` [[5875]](https://github.com/aiidateam/aiida-core/pull/5875)
- Fix the `%verdi` IPython magics utility [[5961]](https://github.com/aiidateam/aiida-core/pull/5961)
- Fix bug in `aiida.engine.utils.instantiate_process` [[5952]](https://github.com/aiidateam/aiida-core/pull/5952)
- Fix incorrect import of exception from `kiwipy.communications` [[5947]](https://github.com/aiidateam/aiida-core/pull/5947)
Deprecations
- `Scheduler`: Move setting of environment variables into base class [[5948]](https://github.com/aiidateam/aiida-core/pull/5948)
- `WorkChains`: Emit deprecation warning if predicate `if_/while_` does not return boolean-like [[5924]](https://github.com/aiidateam/aiida-core/pull/5924)
Changes
- `DaemonClient`: Refactor to include parsing of client response [[5850]](https://github.com/aiidateam/aiida-core/pull/5850)
- ORM: Remove `Entity.from_backend_entity` from the public API [[5447]](https://github.com/aiidateam/aiida-core/pull/5447)
- `PbsproScheduler`: Replace deprecated `ppn` tag with `ncpus` [[5910]](https://github.com/aiidateam/aiida-core/pull/5910)
- `ProcessBuilder`: Move `_prune` method to standalone utility [[5801]](https://github.com/aiidateam/aiida-core/pull/5801)
- `verdi process list`: Simplify the daemon load implementation [[5850]](https://github.com/aiidateam/aiida-core/pull/5850)
Documentation
- Add FAQ on MFA-enabled computers [[5887]](https://github.com/aiidateam/aiida-core/pull/5887)
- Add link to all `metadata.options` inputs in `CalcJob` submission example [[5912]](https://github.com/aiidateam/aiida-core/pull/5912)
- Add warning that `Data` constructor is not called on loading [[5898]](https://github.com/aiidateam/aiida-core/pull/5898)
- Add note on how to create a code that uses Conda environment [[5905]](https://github.com/aiidateam/aiida-core/pull/5905)
- Add `--without-daemon` flag to benchmark script [[5839]](https://github.com/aiidateam/aiida-core/pull/5839)
- Add alternative for conda env activation in submission script [[5950]](https://github.com/aiidateam/aiida-core/pull/5950)
- Clarify that process functions can be exposed in work chains [[5919]](https://github.com/aiidateam/aiida-core/pull/5919)
- Fix the `intro/tutorial.md` notebook [[5961]](https://github.com/aiidateam/aiida-core/pull/5961)
- Fix the overindentation of lists [[5915]](https://github.com/aiidateam/aiida-core/pull/5915)
- Hide the "Edit this page" button on the API reference pages [[5956]](https://github.com/aiidateam/aiida-core/pull/5956)
- Note that an entry point is required for using a data plugin [[5907]](https://github.com/aiidateam/aiida-core/pull/5907)
- Set `use_login_shell=False` for `localhost` in performance benchmark [[5847]](https://github.com/aiidateam/aiida-core/pull/5847)
- Small improvements to the benchmark script [[5854]](https://github.com/aiidateam/aiida-core/pull/5854)
- Use mamba instead of conda [[5891]](https://github.com/aiidateam/aiida-core/pull/5891)
DevOps
- Add devcontainer for easy integration with VSCode [[5913]](https://github.com/aiidateam/aiida-core/pull/5913)
- CI: Update `sphinx-intl` and install transifex CLI [[5908]](https://github.com/aiidateam/aiida-core/pull/5908)
- Fix the `test-install` workflow [[5873]](https://github.com/aiidateam/aiida-core/pull/5873)
- Pre-commit: Improve typing of `aiida.schedulers.scheduler` [[5849]](https://github.com/aiidateam/aiida-core/pull/5849)
- Pre-commit: Set `yapf` option `allow_split_before_dict_value = false`[[5931]](https://github.com/aiidateam/aiida-core/pull/5931)
- Process functions: Replace `getfullargspec` with `signature` [[5900]](https://github.com/aiidateam/aiida-core/pull/5900)
- Fixtures: Add argument `use_subprocess` to `run_cli_command` [[5846]](https://github.com/aiidateam/aiida-core/pull/5846)
- Fixtures: Change default `use_subprocess=False` for `run_cli_command` [[5846]](https://github.com/aiidateam/aiida-core/pull/5846)
- Tests: Use `use_subprocess=False` and `suppress_warnings=True` [[5846]](https://github.com/aiidateam/aiida-core/pull/5846)
- Tests: Fix bugs revealed by running with `use_subprocess=True` [[5846]](https://github.com/aiidateam/aiida-core/pull/5846)
- Typing: Annotate `aiida/orm/utils/serialize.py` [[5832]](https://github.com/aiidateam/aiida-core/pull/5832)
- Typing: Annotate `aiida/tools/visualization/graph.py` [[5821]](https://github.com/aiidateam/aiida-core/pull/5821)
- Typing: Use modern syntax for `aiida.engine.processes.functions` [[5900]](https://github.com/aiidateam/aiida-core/pull/5900)
Dependencies
- Add compatibility for `ipython~=8.0` [[5888]](https://github.com/aiidateam/aiida-core/pull/5888)
- Bump cryptography from 36.0.0 to 39.0.1 [[5885]](https://github.com/aiidateam/aiida-core/pull/5885)
- Remove upper limit on `werkzeug` [[5904]](https://github.com/aiidateam/aiida-core/pull/5904)
- Update pre-commit requirement `isort==5.12.0` [[5877]](https://github.com/aiidateam/aiida-core/pull/5877)
- Update requirement `importlib-metadata~=4.13` [[5963]](https://github.com/aiidateam/aiida-core/pull/5963)
- Bump `graphviz` version to `0.19` [[5965]](https://github.com/aiidateam/aiida-core/pull/5965)
New contributors
Thanks a lot to the following new contributors:
- [Ahmed Basem](https://github.com/AhmedBasem20)
- [Mahhheshh](https://github.com/Mahhheshh)
- [Kyle Wang](https://github.com/TurboKyle)
- [Kartikey Saran](https://github.com/kartikeysaran)
- [zahid47](https://github.com/zahid47)