Duqtools

This version of duqtools is all about future proofing:

- It now depends on [pydantic v2](https://docs.pydantic.dev/latest/blog/pydantic-v2-final/) and [altair v5](https://altair-viz.github.io/releases/changes.html#version-5-0-0-released-may-9-2023), both major dependencies requiring substantial changes to duqtools.
- The configuration and system modules were also refactored, making it easier to modify or add new systems.
- Test coverage improved to 72%, making the duqtools more robust

What's Changed

* Add option to create array submissions script and limit total number of jobs to CLI by stefsmeets in https://github.com/duqtools/duqtools/pull/644
* Adding a `sync-prominence` command by v1kko in https://github.com/duqtools/duqtools/pull/648
* Add option to run duqtools without a system by stefsmeets in https://github.com/duqtools/duqtools/pull/659
* Update pattern to allow parsing localdb strings by stefsmeets in https://github.com/duqtools/duqtools/pull/666

Under the hood

* Update code to support pydantic V2 by stefsmeets in https://github.com/duqtools/duqtools/pull/653
* Update code to be compatible with Pydantic V2 by stefsmeets in https://github.com/duqtools/duqtools/pull/654
* Update code for API change in altair v5 by stefsmeets in https://github.com/duqtools/duqtools/pull/656
* Refactor systems by stefsmeets in https://github.com/duqtools/duqtools/pull/660
* Refactor systems and schemas by stefsmeets in https://github.com/duqtools/duqtools/pull/661

Bug fixes

* Fix bug and add tests for reading `data.csv` by stefsmeets in https://github.com/duqtools/duqtools/pull/647
* Fix compatibility with `pydantic_yaml` >= 1.0.0 by stefsmeets in https://github.com/duqtools/duqtools/pull/652
* Skip loading variable on `EmptyVarError` in dashboard by stefsmeets in https://github.com/duqtools/duqtools/pull/658

Maintanance

* Add Pypi trusted publishing by v1kko in https://github.com/duqtools/duqtools/pull/649
* Update workflow to generate `requirements.txt` by stefsmeets in https://github.com/duqtools/duqtools/pull/657
* Improve test coverage by stefsmeets in https://github.com/duqtools/duqtools/pull/664
* Improve test coverage by stefsmeets in https://github.com/duqtools/duqtools/pull/662
* Add test for duqtools merge by stefsmeets in https://github.com/duqtools/duqtools/pull/667
* Add tests for status, setup and submit by stefsmeets in https://github.com/duqtools/duqtools/pull/668


**Full Changelog**: https://github.com/duqtools/duqtools/compare/1.9.0...1.10.0

The highlight for this release is preliminary support for the ETS6 system.
The operators are now more flexible, with support for upper and lower bounds, gradients, and custom functions if that is still not enough.

Note that if you are updating to this version, systems are now defined in their own block in the config:

yaml
system:
name: jetto
...


See [the documentation](https://duqtools.readthedocs.io/en/latest/config/#defining-the-system) for more info.

What's Changed

* Adding ets6 System for use with duqtools by v1kko in https://github.com/duqtools/duqtools/pull/557
* Add ceiling and floor option to operator by stefsmeets in https://github.com/duqtools/duqtools/pull/630
* Add slope to operator by stefsmeets in https://github.com/duqtools/duqtools/pull/632
* Apply custom operator to data by stefsmeets in https://github.com/duqtools/duqtools/pull/638

Under the hood

* Refactoring the Systems, also fixing ETS6 by v1kko in https://github.com/duqtools/duqtools/pull/623
* Add naive copy instead of complex copy of ids data by v1kko in https://github.com/duqtools/duqtools/pull/613
* Revise testing strategy and update python api by stefsmeets in https://github.com/duqtools/duqtools/pull/610
* Move `jruns` keyword to `system` config by stefsmeets in https://github.com/duqtools/duqtools/pull/634


Documentation

* Add banner to readme by stefsmeets in https://github.com/duqtools/duqtools/pull/615
* Update logo and add it to dashboard by stefsmeets in https://github.com/duqtools/duqtools/pull/618
* Add base / uq run example to docs by stefsmeets in https://github.com/duqtools/duqtools/pull/629

Bug fixes

* Update default for `max_jobs` in submit cli by stefsmeets in https://github.com/duqtools/duqtools/pull/642
* Fix missing config error for submit by stefsmeets in https://github.com/duqtools/duqtools/pull/641
* Get prominence working again by v1kko in https://github.com/duqtools/duqtools/pull/635
* Fix crashes in submit / status by stefsmeets in https://github.com/duqtools/duqtools/pull/627

Maintanance

* Move setup config to pyproject.toml by stefsmeets in https://github.com/duqtools/duqtools/pull/608
* Avoid pip install in workflow on cache hit by stefsmeets in https://github.com/duqtools/duqtools/pull/611
* Avoid installing typing-extensions==4.6.0 which crashes duqtools on Python 3.9 by stefsmeets in https://github.com/duqtools/duqtools/pull/617
* Update IMAS docker workflow to only run once a week by stefsmeets in https://github.com/duqtools/duqtools/pull/622




**Full Changelog**: https://github.com/duqtools/duqtools/compare/1.8.0...1.9.0

This release focuses on expanding the use-cases for `duqtools`. We have added a way to directly specify which operations to always apply to a dataset (`operations` keyword in `duqtools.yaml`). This will make it easier to separate 'fixes' to a dataset from sampling variables or for generating base runs.

This release also introduces a Python API, so that runs can be created and submitted from a Python script or Jupyter Notebook. For more information, see the [documentation](https://duqtools.readthedocs.io/en/latest/examples/create_api/).

The `duqduq` CLI is also more flexible to specify which runs to create and submit, which will give more control over your data for large scale validation.

What's Changed

* Add path and imas filters for duqduq create and submit by stefsmeets in https://github.com/duqtools/duqtools/pull/593
* Update duqtools init template by stefsmeets in https://github.com/duqtools/duqtools/pull/594
* Add operations keyword to create config by stefsmeets in https://github.com/duqtools/duqtools/pull/598
* Add workflow for generating base and uq runs from a single template config by stefsmeets in https://github.com/duqtools/duqtools/pull/603
* Add python api for duqtools create and submit by stefsmeets in https://github.com/duqtools/duqtools/pull/606

Under the hood

* Revise config loading by stefsmeets in https://github.com/duqtools/duqtools/pull/604

Documentation

* Update contributing guidelines by stefsmeets in https://github.com/duqtools/duqtools/pull/592
* Add documentation for setup by stefsmeets in https://github.com/duqtools/duqtools/pull/595
* Add quickstart example to cli by stefsmeets in https://github.com/duqtools/duqtools/pull/596
* Add option to create base run only by stefsmeets in https://github.com/duqtools/duqtools/pull/600
* Add docs for top level keywords in config by stefsmeets in https://github.com/duqtools/duqtools/pull/602

Bug fixes

* Remove print statement in operations.py by v1kko in https://github.com/duqtools/duqtools/pull/589
* Do not copy restart files if restart toggle is off by stefsmeets in https://github.com/duqtools/duqtools/pull/599
* Fixes missing keyword argument in `duqduq submit -array` by stefsmeets in https://github.com/duqtools/duqtools/pull/601
* Make dashboard merge compatible with local imas database by stefsmeets in https://github.com/duqtools/duqtools/pull/607

Maintanance

* Add linting and autoformatting for notebooks by stefsmeets in https://github.com/duqtools/duqtools/pull/591


**Full Changelog**: https://github.com/duqtools/duqtools/compare/1.7.0...1.8.0

What's Changed

* Add duqmap function to map functions over runs/handles by v1kko in https://github.com/duqtools/duqtools/pull/585
* Implement `--max_array_size` (default 100) option in `duqtools submit` by v1kko in https://github.com/duqtools/duqtools/pull/584

Bug fixes

* Fixed publish pipeline by v1kko in https://github.com/duqtools/duqtools/pull/582
* Fix bug in submit.py by v1kko in https://github.com/duqtools/duqtools/pull/579

Maintanance

* Add action for building imas on the CI by stefsmeets in https://github.com/duqtools/duqtools/pull/586

**Full Changelog**: https://github.com/duqtools/duqtools/compare/1.6.4...1.7.0

Bug fixes

* Docker fix and mostly a new publishing pipeline by v1kko in https://github.com/duqtools/duqtools/pull/578

**Full Changelog**: https://github.com/duqtools/duqtools/compare/1.6.3...1.6.4

Fix paths in create.

**Full Changelog**: https://github.com/duqtools/duqtools/compare/1.6.2...1.6.3