Duqtools

What's Changed

* Add a function to get all variables from a handle by v1kko in https://github.com/duqtools/duqtools/pull/568

Bug fixes

* Update notebook for running in docker by v1kko in https://github.com/duqtools/duqtools/pull/556
* Revert to absolute paths for duqduq create by v1kko in https://github.com/duqtools/duqtools/pull/563
* Fix scipy to minimal version 1.09 by v1kko in https://github.com/duqtools/duqtools/pull/561
* Exclude pydantic version 1.10.{3,4,5} by v1kko in https://github.com/duqtools/duqtools/pull/567
* Raise error if data.csv does not have a header by stefsmeets in https://github.com/duqtools/duqtools/pull/570
* Fix dependencies and add jinja2 requirement by stefsmeets in https://github.com/duqtools/duqtools/pull/574
* Add a bit more info on get_all_variables by v1kko in https://github.com/duqtools/duqtools/pull/575


**Full Changelog**: https://github.com/duqtools/duqtools/compare/1.6.0...1.6.1

This is mostly a documentation update that adds several notebooks demoing how to run duqtools in different situations.

What's Changed

* Add message about number of ops that will not be submitted by stefsmeets in https://github.com/duqtools/duqtools/pull/543

Under the hood

* Use `imasdef.EMPTY_*` for empty check by stefsmeets in https://github.com/duqtools/duqtools/pull/544

Documentation

* Add notebook for single run by stefsmeets in https://github.com/duqtools/duqtools/pull/551
* Add notebook for large scale validation by stefsmeets in https://github.com/duqtools/duqtools/pull/553
* Add notebook for running with docker by v1kko in https://github.com/duqtools/duqtools/pull/548
* Add notebook for running on prominence by v1kko in https://github.com/duqtools/duqtools/pull/555

Maintanance

* Re-enable mkdocs-jupyter in docs by stefsmeets in https://github.com/duqtools/duqtools/pull/552
* Run and update notebooks by stefsmeets in https://github.com/duqtools/duqtools/pull/546
* Fix pypi action version by stefsmeets in https://github.com/duqtools/duqtools/pull/537


**Full Changelog**: https://github.com/duqtools/duqtools/compare/1.5.0...1.6.0

This release of duqtools adds `duqtools setup` to generate single runs from a template, and `duqduq merge`, which automates merging large scale validation data. It is now also easier to submit jobs in a slurm array, and to restart failed jobs.

What's Changed

* Array mode supports `max_jobs` (default=10) by v1kko in https://github.com/duqtools/duqtools/pull/512
* Support for relative path in runs (data.csv, runs.yaml) by v1kko in https://github.com/duqtools/duqtools/pull/516
* Submit status filter by stefsmeets in https://github.com/duqtools/duqtools/pull/518
* Implement `duqtools setup` by stefsmeets in https://github.com/duqtools/duqtools/pull/521
* Implement `duqduq merge` by stefsmeets in https://github.com/duqtools/duqtools/pull/524
* Errorbars for each data set in dashboard by stefsmeets in https://github.com/duqtools/duqtools/pull/535

Under the hood

* Pass config to queued operations where necessary by v1kko in https://github.com/duqtools/duqtools/pull/520

Bug fixes

* Do not store relative location if `run_dir` not relative to current dir by v1kko in https://github.com/duqtools/duqtools/pull/533
* Fix scientific notation in variables yaml by stefsmeets in https://github.com/duqtools/duqtools/pull/513
* Do not filter if filter is empty by stefsmeets in https://github.com/duqtools/duqtools/pull/522
* Force output of imas data by stefsmeets in https://github.com/duqtools/duqtools/pull/527
* Fix duqtools submit to array by stefsmeets in https://github.com/duqtools/duqtools/pull/529

Maintanance

* Create `requirements.txt` and add them to the github releases by v1kko in https://github.com/duqtools/duqtools/pull/523
* Revert tests to use `containerized_runs/main` by stefsmeets in https://github.com/duqtools/duqtools/pull/534


**Full Changelog**: https://github.com/duqtools/duqtools/compare/1.4.0...1.5.0

This release of duqtools contains updates to the [large scale validation tooling](https://duqtools.readthedocs.io/en/latest/large_scale_validation/) with `duqduq` and support for running jetto inside a docker container / on prominence. This version of duqtools also has better support for [Jintrac v220922 and v210921 systems](https://duqtools.readthedocs.io/en/latest/config/#defining-the-system).

Note that this version of duqtools drops supporrt for python 3.7 and 3.8. We now target 3.9+, in sync with the IMAS/Jintrac docker container which we use for testing.

What's Changed

* Dropped support for Python 3.7 and 3.8 by stefsmeets in https://github.com/duqtools/duqtools/pull/479
* Small enhancements for large scale validation (`duqduq`) by stefsmeets in https://github.com/duqtools/duqtools/pull/477
* Implementation for jintrac v220922 by stefsmeets in https://github.com/duqtools/duqtools/pull/487
* Added docker submit_system by v1kko in https://github.com/duqtools/duqtools/pull/480
* Grab value from IDS for some values by stefsmeets in https://github.com/duqtools/duqtools/pull/494
* Improve large scale validation tools by stefsmeets in https://github.com/duqtools/duqtools/pull/483
* Implement lookup table to define IDS-to-jetto conversion by stefsmeets in https://github.com/duqtools/duqtools/pull/502
* Add `yes` option to duqduq cli by stefsmeets in https://github.com/duqtools/duqtools/pull/508
* Add tag to config schema to name jobs/runs by stefsmeets in https://github.com/duqtools/duqtools/pull/509
* Add fine-grained status reports for duqduq and improve efficiency by stefsmeets in https://github.com/duqtools/duqtools/pull/510

Documentation

* Update documentation of jetto system by v1kko in https://github.com/duqtools/duqtools/pull/504
* Update documentation for large scale validation by stefsmeets in https://github.com/duqtools/duqtools/pull/507

Under the hood

* Support adding an index at the end of a key in `IDSMapping.__getitem__` by stefsmeets in https://github.com/duqtools/duqtools/pull/500
* Write container name in duqtools lockfile for docker submit by v1kko in https://github.com/duqtools/duqtools/pull/505

Bug fixes

* Small fixes by v1kko in https://github.com/duqtools/duqtools/pull/475
* Fixed submit to recreate status, and scheduled submit by v1kko in https://github.com/duqtools/duqtools/pull/495
* Raise if jetto restart files exist in template by stefsmeets in https://github.com/duqtools/duqtools/pull/499

Maintanance

* Update .readthedocs.yaml by stefsmeets in https://github.com/duqtools/duqtools/pull/474
* Support running tests in docker container by stefsmeets in https://github.com/duqtools/duqtools/pull/481
* Add ruff as linter and import sorter by stefsmeets in https://github.com/duqtools/duqtools/pull/489
* Build docs on pull request by v1kko in https://github.com/duqtools/duqtools/pull/493

**Full Changelog**: https://github.com/duqtools/duqtools/compare/1.3.0...1.4.0

What's Changed

* Synchronize merge/plot options and add group common options by stefsmeets in https://github.com/duqtools/duqtools/pull/457
* Name plot output file after grid/data variable by stefsmeets in https://github.com/duqtools/duqtools/pull/460
* Drop support for python 3.7 and 3.8 by stefsmeets in https://github.com/duqtools/duqtools/pull/472

Bug fixes

* fix errorbars while plotting multiple imas data by v1kko in https://github.com/duqtools/duqtools/pull/455
* Check and abort if merge target already exists by stefsmeets in https://github.com/duqtools/duqtools/pull/461
* Jintrac/Imas container fixes by stefsmeets in https://github.com/duqtools/duqtools/pull/465

Maintanance

* Ci imas container by v1kko in https://github.com/duqtools/duqtools/pull/466
* Try to cache venv by stefsmeets in https://github.com/duqtools/duqtools/pull/470
* Update jetto tools to 1.8.8 by stefsmeets in https://github.com/duqtools/duqtools/pull/471
* Work-around streamlit version for docker image by stefsmeets in https://github.com/duqtools/duqtools/pull/473

**Full Changelog**: https://github.com/duqtools/duqtools/compare/1.2.0...1.3.0