------
Highlights
~~~~~~~~~~
- ESMValCore now supports wildcards in recipes and offers improved support for
ancillary variables and dataset versioning thanks to contributions by
:user:`bouweandela`. For details, see
:ref:`Automatically populating a recipe with all available datasets <dataset_wildcards>`
and :ref:`Defining supplementary variables <supplementary_variables>`.
- Support for CORDEX datasets in a rotated pole coordinate system has been
added by :user:`sloosvel`.
- Native :ref:`ICON <read_icon>` output is now made UGRID-compliant
on-the-fly to unlock the use of more sophisticated regridding algorithms,
thanks to :user:`schlunma`.
- The Python API has been extended with the addition of three
modules: :mod:`esmvalcore.config`, :mod:`esmvalcore.dataset`, and
:mod:`esmvalcore.local`, all these features courtesy of
:user:`bouweandela`. For details, see our new
example :doc:`example-notebooks`.
- The preprocessor :func:`~esmvalcore.preprocessor.multi_model_statistics`
has been extended to support more use-cases thanks to contributions by
:user:`schlunma`. For details, see
:ref:`Multi-model statistics <multi-model statistics>`.
This release includes:
Backwards incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Please read the descriptions of the linked pull requests for detailed upgrade instructions.
- The algorithm for automatically defining the ancillary variables and cell
measures has been improved (:pull:`1609`) by :user:`bouweandela`.
If this does not work as expected, more examples of how to adapt your recipes
are given
`here <https://github.com/ESMValGroup/ESMValCore/pull/1609#Backward-incompatible-changes>`__
and in the corresponding sections of the
:ref:`recipe documentation <supplementary_variables>` and the
:ref:`preprocessor documentation <preprocessors_using_supplementary_variables>`.
- Remove deprecated features scheduled for removal in v2.8.0 or earlier
(:pull:`1826`) by :user:`schlunma`.
Removed ``esmvalcore.iris_helpers.var_name_constraint`` (has been deprecated
in v2.6.0; please use :class:`iris.NameConstraint` with the keyword argument
``var_name`` instead) and the option ``always_use_ne_mask`` for
:func:`esmvalcore.preprocessor.mask_landsea` (has been deprecated in v2.5.0;
the same behavior can now be achieved by specifying ``supplementary_variables``.
- No files will be found if a non-existent version of a dataset is specified
(:pull:`1835`) by :user:`bouweandela`. If a ``version`` of a
dataset is specified in the recipe, the tool will now search for exactly that
version, instead of simply using the latest version. Therefore, it is
necessary to make sure that the version number in the directory tree matches
with the version number in the recipe to find the files.
- The default filename template for obs4MIPs has been updated to better match
filenames used in this project in (:pull:`1866`) by :user:`bouweandela`. This
may cause issues if you are storing all the files for obs4MIPs in a
directory with no subdirectories per dataset.
Deprecations
~~~~~~~~~~~~
Please read the descriptions of the linked pull requests for detailed upgrade instructions.
- Various configuration related options that are now available through
:mod:`esmvalcore.config` have been deprecated (:pull:`1769`) by :user:`bouweandela`.
- The ``fx_variables`` preprocessor argument and related features have been
deprecated (:pull:`1609`) by :user:`bouweandela`.
See :pull:`1609Deprecations` for more information.
- Combined ``offline`` and ``always_search_esgf`` into a single option ``search_esgf``
(:pull:`1935`)
:user:`schlunma`. The configuration
option/command line argument ``offline`` has been deprecated in favor of
``search_esgf``. The previous ``offline: true`` is now ``search_esgf: never``
(the default); the previous ``offline: false`` is now
``search_esgf: when_missing``. More details on how to adapt your workflow
regarding these new options are given in :pull:`1935` and the
`documentation <https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html?highlight=search_esgf#user-configuration-file>`__.
- :func:`esmvalcore.preprocessor.cleanup` has been deprecated (:pull:`1949`)
:user:`schlunma`. Please do not use this
anymore in the recipe (it is not necessary).
Python API
~~~~~~~~~~
- Support searching ESGF for a specific version of a file and add :obj:`esmvalcore.esgf.ESGFFile.facets` (:pull:`1822`) by :user:`bouweandela`
- Fix issues with searching for files on ESGF (:pull:`1863`) by :user:`bouweandela`
- Move the :mod:`esmvalcore.experimental.config` module to :mod:`esmvalcore.config` (:pull:`1769`) by :user:`bouweandela`
- Add :mod:`esmvalcore.local`, a module to search data on the local filesystem (:pull:`1835`) by :user:`bouweandela`
- Add :mod:`esmvalcore.dataset` module (:pull:`1877`) by :user:`bouweandela`
Bug fixes
~~~~~~~~~
- Import from :mod:`esmvalcore.config` in the :mod:`esmvalcore.experimental` module (:pull:`1816`) by :user:`bouweandela`
- Added scalar coords of input cubes to output of esmpy_regrid (:pull:`1811`) by :user:`schlunma`
- Fix severe bug in :func:`esmvalcore.preprocessor.mask_fillvalues` (:pull:`1823`) by :user:`schlunma`
- Fix LWP of ICON on-the-fly CMORizer (:pull:`1839`) by :user:`schlunma`
- Fixed issue in irregular regridding regarding scalar coordinates (:pull:`1845`) by :user:`schlunma`
- Update product attributes and `metadata.yml` with cube metadata before saving files (:pull:`1837`) by :user:`schlunma`
- Remove an extra space character from a filename (:pull:`1883`) by :user:`bouweandela`
- Improve resilience of ESGF search (:pull:`1869`) by :user:`bouweandela`
- Fix issue with no files found if timerange start/end differs in length (:pull:`1880`) by :user:`bouweandela`
- Add `driver` and `sub_experiment` tags to generate dataset aliases (:pull:`1886`) by :user:`sloosvel`
- Fixed time points of native CESM2 output (:pull:`1772`) by :user:`schlunma`
- Fix type hints for Python versions < 3.10 (:pull:`1897`) by :user:`bouweandela`
- Fixed `set_range_in_0_360` for dask arrays (:pull:`1919`) by :user:`schlunma`
- Made equalized attributes in concatenated cubes consistent across runs (:pull:`1783`) by :user:`schlunma`
- Fix issue with reading dates from files (:pull:`1936`) by :user:`bouweandela`
- Add institute name used on ESGF for CMIP5 CanAM4, CanCM4, and CanESM2 (:pull:`1937`) by :user:`bouweandela`
- Fix issue where data was not loaded and saved (:pull:`1962`) by :user:`bouweandela`
- Fix type hints for Python 3.8 (:pull:`1795`) by :user:`bouweandela`
- Update the institute facet of the CSIRO-Mk3L-1-2 model (:pull:`1966`) by :user:`remi-kazeroni`
- Fixed race condition that may result in errors in :func:`esmvalcore.preprocessor.cleanup` (:pull:`1949`) by :user:`schlunma`
- Update notebook so it uses supplementaries instead of ancillaries (:pull:`1945`) by :user:`bouweandela`
Documentation
~~~~~~~~~~~~~
- Fix anaconda badge in README (:pull:`1759`) by :user:`valeriupredoi`
- Fix mistake in the documentation of :obj:`esmvalcore.esgf.find_files` (:pull:`1784`) by :user:`bouweandela`
- Support linking to "stable" ESMValTool version on readthedocs (:pull:`1608`) by :user:`bouweandela`
- Updated ICON doc with information on usage of extract_levels preprocessor (:pull:`1903`) by :user:`schlunma`
- Add changelog for latest released version v2.7.1 (:pull:`1905`) by :user:`valeriupredoi`
- Update `preprocessor.rst` due to renaming of NCEP dataset to NCEP-NCAR-R1 (:pull:`1908`) by :user:`hb326`
- Replace timerange nested lists in docs with overview table (:pull:`1940`) by :user:`zklaus`
- Updated section "backward compatibility" in `contributing.rst` (:pull:`1918`) by :user:`axel-lauer`
- Add link to ESMValTool release procedure steps (:pull:`1957`) by :user:`remi-kazeroni`
- Synchronize documentation table of contents with ESMValTool (:pull:`1958`) by :user:`bouweandela`
Improvements
~~~~~~~~~~~~
- Support wildcards in the recipe and improve support for ancillary variables and dataset versioning (:pull:`1609`) by :user:`bouweandela`. More details on how to adapt your recipes are given in the corresponding pull request description and in the corresponding sections of the `recipe documentation <https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/overview.html#defining-supplementary-variables-ancillary-variables-and-cell-measures>`__ and the `preprocessor documentation <https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/preprocessor.html#preprocessors-using-supplementary-variables>`__.
- Create a session directory with suffix "-1", "-2", etc if it already exists (:pull:`1818`) by :user:`bouweandela`
- Message for users when they use esmvaltool executable from esmvalcore only (:pull:`1831`) by :user:`valeriupredoi`
- Order recipe output in index.html (:pull:`1899`) by :user:`bouweandela`
- Improve reading facets from ESGF search results (:pull:`1920`) by :user:`bouweandela`
Fixes for datasets
~~~~~~~~~~~~~~~~~~
- Fix rotated coordinate grids and `tas` and `pr` for CORDEX datasets (:pull:`1765`) by :user:`sloosvel`
- Made ICON output UGRID-compliant (on-the-fly) (:pull:`1664`) by :user:`schlunma`
- Fix automatic download of ICON grid file and make ICON UGRIDization optional (`default: true`) (:pull:`1922`) by :user:`schlunma`
- Add siconc fixes for EC-Earth3-Veg and EC-Earth3-Veg-LR models (:pull:`1771`) by :user:`egalytska`
- Fix siconc in KIOST-ESM (:pull:`1829`) by :user:`LisaBock`
- Extension of ERA5 CMORizer (variable cl) (:pull:`1850`) by :user:`axel-lauer`
- Add standard variable names for EMAC (:pull:`1853`) by :user:`FranziskaWinterstein`
- Fix for FGOALS-f3-L clt (:pull:`1928`) by :user:`LisaBock`
Installation
~~~~~~~~~~~~
- Add all deps to the conda-forge environment and suppress installing and reinstalling deps with pip at readthedocs builds (:pull:`1786`) by :user:`valeriupredoi`
- Pin netCDF4<1.6.1 (:pull:`1805`) by :user:`bouweandela`
- Unpin NetCF4 (:pull:`1814`) by :user:`valeriupredoi`
- Unpin flake8 (:pull:`1820`) by :user:`valeriupredoi`
- Add iris-esmf-regrid as a dependency (:pull:`1809`) by :user:`sloosvel`
- Pin esmpy<8.4 (:pull:`1871`) by :user:`zklaus`
- Update esmpy import for ESMF v8.4.0 (:pull:`1876`) by :user:`bouweandela`
Preprocessor
~~~~~~~~~~~~
- Allow :func:`esmvalcore.preprocessor.multi_model_statistics` on cubes with arbitrary dimensions (:pull:`1808`) by :user:`schlunma`
- Smarter removal of coordinate metadata in :func:`esmvalcore.preprocessor.multi_model_statistics` preprocessor (:pull:`1813`) by :user:`schlunma`
- Allowed usage of :func:`esmvalcore.preprocessor.multi_model_statistics` on single cubes/products (:pull:`1849`) by :user:`schlunma`
- Allowed usage of :func:`esmvalcore.preprocessor.multi_model_statistics` on cubes with identical ``name()`` and ``units`` (but e.g. different long_name) (:pull:`1921`) by :user:`schlunma`
- Allowed ignoring scalar coordinates in :func:`esmvalcore.preprocessor.multi_model_statistics` (:pull:`1934`) by :user:`schlunma`
- Refactored :func:`esmvalcore.preprocessor.regrid` and removed unnecessary code not needed anymore due to new iris version (:pull:`1898`) by :user:`schlunma`
- Do not realise coordinates during CMOR check (:pull:`1912`) by :user:`sloosvel`
- Make :func:`esmvalcore.preprocessor.extract_volume` work with closed and mixed intervals and allow nearest value selection (:pull:`1930`) by :user:`sloosvel`
Release
~~~~~~~
- Changelog for `v2.8.0rc1` (:pull:`1952`) by :user:`remi-kazeroni`
- Increase version number for ESMValCore `v2.8.0rc1` (:pull:`1955`) by :user:`remi-kazeroni`
- Changelog for `v2.8.0rc2` (:pull:`1959`) by :user:`remi-kazeroni`
- Increase version number for ESMValCore `v2.8.0rc2` (:pull:`1973`) by :user:`remi-kazeroni`
- Changelog for `v2.8.0` (:pull:`1978`) by :user:`remi-kazeroni`
- Increase version number for ESMValCore `v2.8.0` (:pull:`1983`) by :user:`remi-kazeroni`
Automatic testing
~~~~~~~~~~~~~~~~~
- Set implicit optional to true in `mypy` config to avert side effects and test fails from new mypy version (:pull:`1790`) by :user:`valeriupredoi`
- Remove duplicate `implicit_optional = True` line in ``setup.cfg`` (:pull:`1791`) by :user:`valeriupredoi`
- Fix failing test due to missing sample data (:pull:`1797`) by :user:`bouweandela`
- Remove outdated cmor_table facet from data finder tests (:pull:`1798`) by :user:`bouweandela`
- Modernize tests for :func:`esmvalcore.preprocessor.save` (:pull:`1799`) by :user:`bouweandela`
- No more sequential tests since SegFaults were not noticed anymore (:pull:`1819`) by :user:`valeriupredoi`
- Update pre-commit configuration (:pull:`1821`) by :user:`bouweandela`
- Updated URL of ICON grid file used for testing (:pull:`1914`) by :user:`schlunma`
Variable Derivation
~~~~~~~~~~~~~~~~~~~
- Add derivation of sea ice extent (:pull:`1695`) by :user:`sloosvel`
.. _changelog-v2-7-1: