Esmvalcore

------

Highlights
~~~~~~~~~~

- ESMValCore now supports wildcards in recipes and offers improved support for
ancillary variables and dataset versioning thanks to contributions by
:user:`bouweandela`. For details, see
:ref:`Automatically populating a recipe with all available datasets <dataset_wildcards>`
and :ref:`Defining supplementary variables <supplementary_variables>`.
- Support for CORDEX datasets in a rotated pole coordinate system has been
added by :user:`sloosvel`.
- Native :ref:`ICON <read_icon>` output is now made UGRID-compliant
on-the-fly to unlock the use of more sophisticated regridding algorithms,
thanks to :user:`schlunma`.
- The Python API has been extended with the addition of three
modules: :mod:`esmvalcore.config`, :mod:`esmvalcore.dataset`, and
:mod:`esmvalcore.local`, all these features courtesy of
:user:`bouweandela`. For details, see our new
example :doc:`example-notebooks`.
- The preprocessor :func:`~esmvalcore.preprocessor.multi_model_statistics`
has been extended to support more use-cases thanks to contributions by
:user:`schlunma`. For details, see
:ref:`Multi-model statistics <multi-model statistics>`.

This release includes:

Backwards incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Please read the descriptions of the linked pull requests for detailed upgrade instructions.

- The algorithm for automatically defining the ancillary variables and cell
measures has been improved (:pull:`1609`) by :user:`bouweandela`.
If this does not work as expected, more examples of how to adapt your recipes
are given
`here <https://github.com/ESMValGroup/ESMValCore/pull/1609#Backward-incompatible-changes>`__
and in the corresponding sections of the
:ref:`recipe documentation <supplementary_variables>` and the
:ref:`preprocessor documentation <preprocessors_using_supplementary_variables>`.
- Remove deprecated features scheduled for removal in v2.8.0 or earlier
(:pull:`1826`) by :user:`schlunma`.
Removed ``esmvalcore.iris_helpers.var_name_constraint`` (has been deprecated
in v2.6.0; please use :class:`iris.NameConstraint` with the keyword argument
``var_name`` instead) and the option ``always_use_ne_mask`` for
:func:`esmvalcore.preprocessor.mask_landsea` (has been deprecated in v2.5.0;
the same behavior can now be achieved by specifying ``supplementary_variables``.
- No files will be found if a non-existent version of a dataset is specified
(:pull:`1835`) by :user:`bouweandela`. If a ``version`` of a
dataset is specified in the recipe, the tool will now search for exactly that
version, instead of simply using the latest version. Therefore, it is
necessary to make sure that the version number in the directory tree matches
with the version number in the recipe to find the files.
- The default filename template for obs4MIPs has been updated to better match
filenames used in this project in (:pull:`1866`) by :user:`bouweandela`. This
may cause issues if you are storing all the files for obs4MIPs in a
directory with no subdirectories per dataset.

Deprecations
~~~~~~~~~~~~
Please read the descriptions of the linked pull requests for detailed upgrade instructions.

- Various configuration related options that are now available through
:mod:`esmvalcore.config` have been deprecated (:pull:`1769`) by :user:`bouweandela`.
- The ``fx_variables`` preprocessor argument and related features have been
deprecated (:pull:`1609`) by :user:`bouweandela`.
See :pull:`1609Deprecations` for more information.
- Combined ``offline`` and ``always_search_esgf`` into a single option ``search_esgf``
(:pull:`1935`)
:user:`schlunma`. The configuration
option/command line argument ``offline`` has been deprecated in favor of
``search_esgf``. The previous ``offline: true`` is now ``search_esgf: never``
(the default); the previous ``offline: false`` is now
``search_esgf: when_missing``. More details on how to adapt your workflow
regarding these new options are given in :pull:`1935` and the
`documentation <https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html?highlight=search_esgf#user-configuration-file>`__.
- :func:`esmvalcore.preprocessor.cleanup` has been deprecated (:pull:`1949`)
:user:`schlunma`. Please do not use this
anymore in the recipe (it is not necessary).

Python API
~~~~~~~~~~

- Support searching ESGF for a specific version of a file and add :obj:`esmvalcore.esgf.ESGFFile.facets` (:pull:`1822`) by :user:`bouweandela`
- Fix issues with searching for files on ESGF (:pull:`1863`) by :user:`bouweandela`
- Move the :mod:`esmvalcore.experimental.config` module to :mod:`esmvalcore.config` (:pull:`1769`) by :user:`bouweandela`
- Add :mod:`esmvalcore.local`, a module to search data on the local filesystem (:pull:`1835`) by :user:`bouweandela`
- Add :mod:`esmvalcore.dataset` module (:pull:`1877`) by :user:`bouweandela`

Bug fixes
~~~~~~~~~

- Import from :mod:`esmvalcore.config` in the :mod:`esmvalcore.experimental` module (:pull:`1816`) by :user:`bouweandela`
- Added scalar coords of input cubes to output of esmpy_regrid (:pull:`1811`) by :user:`schlunma`
- Fix severe bug in :func:`esmvalcore.preprocessor.mask_fillvalues` (:pull:`1823`) by :user:`schlunma`
- Fix LWP of ICON on-the-fly CMORizer (:pull:`1839`) by :user:`schlunma`
- Fixed issue in irregular regridding regarding scalar coordinates (:pull:`1845`) by :user:`schlunma`
- Update product attributes and `metadata.yml` with cube metadata before saving files (:pull:`1837`) by :user:`schlunma`
- Remove an extra space character from a filename (:pull:`1883`) by :user:`bouweandela`
- Improve resilience of ESGF search (:pull:`1869`) by :user:`bouweandela`
- Fix issue with no files found if timerange start/end differs in length (:pull:`1880`) by :user:`bouweandela`
- Add `driver` and `sub_experiment` tags to generate dataset aliases (:pull:`1886`) by :user:`sloosvel`
- Fixed time points of native CESM2 output (:pull:`1772`) by :user:`schlunma`
- Fix type hints for Python versions < 3.10 (:pull:`1897`) by :user:`bouweandela`
- Fixed `set_range_in_0_360` for dask arrays (:pull:`1919`) by :user:`schlunma`
- Made equalized attributes in concatenated cubes consistent across runs (:pull:`1783`) by :user:`schlunma`
- Fix issue with reading dates from files (:pull:`1936`) by :user:`bouweandela`
- Add institute name used on ESGF for CMIP5 CanAM4, CanCM4, and CanESM2 (:pull:`1937`) by :user:`bouweandela`
- Fix issue where data was not loaded and saved (:pull:`1962`) by :user:`bouweandela`
- Fix type hints for Python 3.8 (:pull:`1795`) by :user:`bouweandela`
- Update the institute facet of the CSIRO-Mk3L-1-2 model (:pull:`1966`) by :user:`remi-kazeroni`
- Fixed race condition that may result in errors in :func:`esmvalcore.preprocessor.cleanup` (:pull:`1949`) by :user:`schlunma`
- Update notebook so it uses supplementaries instead of ancillaries (:pull:`1945`) by :user:`bouweandela`

Documentation
~~~~~~~~~~~~~

- Fix anaconda badge in README (:pull:`1759`) by :user:`valeriupredoi`
- Fix mistake in the documentation of :obj:`esmvalcore.esgf.find_files` (:pull:`1784`) by :user:`bouweandela`
- Support linking to "stable" ESMValTool version on readthedocs (:pull:`1608`) by :user:`bouweandela`
- Updated ICON doc with information on usage of extract_levels preprocessor (:pull:`1903`) by :user:`schlunma`
- Add changelog for latest released version v2.7.1 (:pull:`1905`) by :user:`valeriupredoi`
- Update `preprocessor.rst` due to renaming of NCEP dataset to NCEP-NCAR-R1 (:pull:`1908`) by :user:`hb326`
- Replace timerange nested lists in docs with overview table (:pull:`1940`) by :user:`zklaus`
- Updated section "backward compatibility" in `contributing.rst` (:pull:`1918`) by :user:`axel-lauer`
- Add link to ESMValTool release procedure steps (:pull:`1957`) by :user:`remi-kazeroni`
- Synchronize documentation table of contents with ESMValTool (:pull:`1958`) by :user:`bouweandela`

Improvements
~~~~~~~~~~~~

- Support wildcards in the recipe and improve support for ancillary variables and dataset versioning (:pull:`1609`) by :user:`bouweandela`. More details on how to adapt your recipes are given in the corresponding pull request description and in the corresponding sections of the `recipe documentation <https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/overview.html#defining-supplementary-variables-ancillary-variables-and-cell-measures>`__ and the `preprocessor documentation <https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/preprocessor.html#preprocessors-using-supplementary-variables>`__.
- Create a session directory with suffix "-1", "-2", etc if it already exists (:pull:`1818`) by :user:`bouweandela`
- Message for users when they use esmvaltool executable from esmvalcore only (:pull:`1831`) by :user:`valeriupredoi`
- Order recipe output in index.html (:pull:`1899`) by :user:`bouweandela`
- Improve reading facets from ESGF search results (:pull:`1920`) by :user:`bouweandela`

Fixes for datasets
~~~~~~~~~~~~~~~~~~

- Fix rotated coordinate grids and `tas` and `pr` for CORDEX datasets (:pull:`1765`) by :user:`sloosvel`
- Made ICON output UGRID-compliant (on-the-fly) (:pull:`1664`) by :user:`schlunma`
- Fix automatic download of ICON grid file and make ICON UGRIDization optional (`default: true`) (:pull:`1922`) by :user:`schlunma`
- Add siconc fixes for EC-Earth3-Veg and EC-Earth3-Veg-LR models (:pull:`1771`) by :user:`egalytska`
- Fix siconc in KIOST-ESM (:pull:`1829`) by :user:`LisaBock`
- Extension of ERA5 CMORizer (variable cl) (:pull:`1850`) by :user:`axel-lauer`
- Add standard variable names for EMAC (:pull:`1853`) by :user:`FranziskaWinterstein`
- Fix for FGOALS-f3-L clt (:pull:`1928`) by :user:`LisaBock`

Installation
~~~~~~~~~~~~

- Add all deps to the conda-forge environment and suppress installing and reinstalling deps with pip at readthedocs builds (:pull:`1786`) by :user:`valeriupredoi`
- Pin netCDF4<1.6.1 (:pull:`1805`) by :user:`bouweandela`
- Unpin NetCF4 (:pull:`1814`) by :user:`valeriupredoi`
- Unpin flake8 (:pull:`1820`) by :user:`valeriupredoi`
- Add iris-esmf-regrid as a dependency (:pull:`1809`) by :user:`sloosvel`
- Pin esmpy<8.4 (:pull:`1871`) by :user:`zklaus`
- Update esmpy import for ESMF v8.4.0 (:pull:`1876`) by :user:`bouweandela`

Preprocessor
~~~~~~~~~~~~
- Allow :func:`esmvalcore.preprocessor.multi_model_statistics` on cubes with arbitrary dimensions (:pull:`1808`) by :user:`schlunma`
- Smarter removal of coordinate metadata in :func:`esmvalcore.preprocessor.multi_model_statistics` preprocessor (:pull:`1813`) by :user:`schlunma`
- Allowed usage of :func:`esmvalcore.preprocessor.multi_model_statistics` on single cubes/products (:pull:`1849`) by :user:`schlunma`
- Allowed usage of :func:`esmvalcore.preprocessor.multi_model_statistics` on cubes with identical ``name()`` and ``units`` (but e.g. different long_name) (:pull:`1921`) by :user:`schlunma`
- Allowed ignoring scalar coordinates in :func:`esmvalcore.preprocessor.multi_model_statistics` (:pull:`1934`) by :user:`schlunma`
- Refactored :func:`esmvalcore.preprocessor.regrid` and removed unnecessary code not needed anymore due to new iris version (:pull:`1898`) by :user:`schlunma`
- Do not realise coordinates during CMOR check (:pull:`1912`) by :user:`sloosvel`
- Make :func:`esmvalcore.preprocessor.extract_volume` work with closed and mixed intervals and allow nearest value selection (:pull:`1930`) by :user:`sloosvel`

Release
~~~~~~~
- Changelog for `v2.8.0rc1` (:pull:`1952`) by :user:`remi-kazeroni`
- Increase version number for ESMValCore `v2.8.0rc1` (:pull:`1955`) by :user:`remi-kazeroni`
- Changelog for `v2.8.0rc2` (:pull:`1959`) by :user:`remi-kazeroni`
- Increase version number for ESMValCore `v2.8.0rc2` (:pull:`1973`) by :user:`remi-kazeroni`
- Changelog for `v2.8.0` (:pull:`1978`) by :user:`remi-kazeroni`
- Increase version number for ESMValCore `v2.8.0` (:pull:`1983`) by :user:`remi-kazeroni`

Automatic testing
~~~~~~~~~~~~~~~~~

- Set implicit optional to true in `mypy` config to avert side effects and test fails from new mypy version (:pull:`1790`) by :user:`valeriupredoi`
- Remove duplicate `implicit_optional = True` line in ``setup.cfg`` (:pull:`1791`) by :user:`valeriupredoi`
- Fix failing test due to missing sample data (:pull:`1797`) by :user:`bouweandela`
- Remove outdated cmor_table facet from data finder tests (:pull:`1798`) by :user:`bouweandela`
- Modernize tests for :func:`esmvalcore.preprocessor.save` (:pull:`1799`) by :user:`bouweandela`
- No more sequential tests since SegFaults were not noticed anymore (:pull:`1819`) by :user:`valeriupredoi`
- Update pre-commit configuration (:pull:`1821`) by :user:`bouweandela`
- Updated URL of ICON grid file used for testing (:pull:`1914`) by :user:`schlunma`

Variable Derivation
~~~~~~~~~~~~~~~~~~~

- Add derivation of sea ice extent (:pull:`1695`) by :user:`sloosvel`


.. _changelog-v2-7-1:

------

Highlights
~~~~~~~~~~

This is a bugfix release where we unpin `cf-units` to allow the latest `iris=3.4.0` to be installed. It also includes an update to the default configuration used when searching the ESGF for files, to account for a recent change of the CEDA ESGF index node hostname. The changelog contains only changes that were made to the ``main`` branch.

Installation
~~~~~~~~~~~~

- Set the version number on the development branches to one minor version more than the previous release (:pull:`1854`) by :user:`bouweandela`
- Unpin cf-units (:pull:`1770`) by :user:`bouweandela`

Bug fixes
~~~~~~~~~

- Improve error handling if an esgf index node is offline (:pull:`1834`) by :user:`bouweandela`

Automatic testing
~~~~~~~~~~~~~~~~~

- Removed unnecessary test that fails with iris 3.4.0 (:pull:`1846`) by :user:`schlunma`
- Update CEDA ESGF index node hostname (:pull:`1838`) by :user:`valeriupredoi`


.. _changelog-v2-7-0:

------

Highlights
~~~~~~~~~~

- We have a new preprocessor function called `'rolling_window_statistics' <https://docs.esmvaltool.org/projects/ESMValCore/en/latest/recipe/preprocessor.html#rolling-window-statistics>`__ implemented by :user:`malininae`
- We have improved the support for native models, refactored native model fixes by adding common base class `NativeDatasetFix`, changed default DRS for reading native ICON output, and added tests for input/output filenames for `ICON <https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/find_data.html#icon>`__ and `EMAC <https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/find_data.html#emac>`__ on-the-fly CMORizer, all these features courtesy of :user:`schlunma`
- Performance of preprocessor functions that use time dimensions has been sped up by **two orders of magnitude** thanks to contributions by :user:`bouweandela`

This release includes:

Backwards incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- Change default DRS for reading native ICON output (:pull:`1705`) by :user:`schlunma`

Bug fixes
~~~~~~~~~

- Add support for regions stored as MultiPolygon to extract_shape preprocessor (:pull:`1670`) by :user:`bouweandela`
- Fixed type annotations for Python 3.8 (:pull:`1700`) by :user:`schlunma`
- Core `_io.concatenate()` may fail due to case when one of the cubes is scalar - this fixes that (:pull:`1715`) by :user:`valeriupredoi`
- Pick up esmvalcore badge instead of esmvaltool one in README (:pull:`1749`) by :user:`valeriupredoi`
- Restore support for scalar cubes to time selection preprocessor functions (:pull:`1750`) by :user:`bouweandela`
- Fix calculation of precipitation flux in EMAC on-the-fly CMORizer (:pull:`1755`) by :user:`schlunma`

Deprecations
~~~~~~~~~~~~

- Remove deprecation warning for regrid schemes already deprecated for v2.7.0 (:pull:`1753`) by :user:`valeriupredoi`

Documentation
~~~~~~~~~~~~~

- Add Met Office Installation Method (:pull:`1692`) by :user:`mo-tgeddes`
- Add MO-paths to config file (:pull:`1709`) by :user:`mo-tgeddes`
- Update MO obs4MIPs paths in the user configuration file (:pull:`1734`) by :user:`mo-tgeddes`
- Update `Making a release` section of the documentation (:pull:`1689`) by :user:`sloosvel`
- Added changelog for v2.7.0 (:pull:`1746`) by :user:`valeriupredoi`
- update CITATION.cff file with 2.7.0 release info (:pull:`1757`) by :user:`valeriupredoi`

Improvements
~~~~~~~~~~~~

- New preprocessor function 'rolling_window_statistics' (:pull:`1702`) by :user:`malininae`
- Remove `pytest_flake8` plugin and use `flake8` instead (:pull:`1722`) by :user:`valeriupredoi`
- Added CESM2 CMORizer (:pull:`1678`) by :user:`schlunma`
- Speed up functions that use time dimension (:pull:`1713`) by :user:`bouweandela`
- Modernize and minimize pylint configuration (:pull:`1726`) by :user:`bouweandela`

Fixes for datasets
~~~~~~~~~~~~~~~~~~

- Refactored native model fixes by adding common base class `NativeDatasetFix` (:pull:`1694`) by :user:`schlunma`

Installation
~~~~~~~~~~~~

- Pin `netCDF4 != 1.6.1` since that seems to throw a flurry of Segmentation Faults (:pull:`1724`) by :user:`valeriupredoi`

Automatic testing
~~~~~~~~~~~~~~~~~

- Pin `flake8<5.0.0` since Circle CI tests are failing copiously (:pull:`1698`) by :user:`valeriupredoi`
- Added tests for input/output filenames for ICON and EMAC on-the-fly CMORizer (:pull:`1718`) by :user:`schlunma`
- Fix failed tests for Python<3.10 resulting from typing (:pull:`1748`) by :user:`schlunma`

.. _changelog-v2-6-0:

------

Highlights
~~~~~~~~~~

- A new set of CMOR fixes is now available in order to load native EMAC model output and CMORize it on the fly. For details, see :ref:`Supported native models: EMAC <read_emac>`.
- The version number of ESMValCore is now automatically generated using `setuptools_scm <https://github.com/pypa/setuptools_scm/#default-versioning-scheme>`__, which extracts Python package versions from git metadata.

This release includes

Deprecations
~~~~~~~~~~~~

- Deprecate the function `esmvalcore.var_name_constraint` (:pull:`1592`) by :user:`schlunma`. This function is scheduled for removal in v2.8.0. Please use :class:`iris.NameConstraint` with the keyword argument `var_name` instead: this is an exact replacement.

Bug fixes
~~~~~~~~~

- Added `start_year` and `end_year` attributes to derived variables (:pull:`1547`) by :user:`schlunma`
- Show all results on recipe results webpage (:pull:`1560`) by :user:`bouweandela`
- Regridding regular grids with similar coordinates (:pull:`1567`) by :user:`tomaslovato`
- Fix timerange wildcard search when deriving variables or downloading files (:pull:`1562`) by :user:`sloosvel`
- Fix `force_derivation` bug (:pull:`1627`) by :user:`sloosvel`
- Correct `build-and-deploy-on-pypi` action (:pull:`1634`) by :user:`sloosvel`
- Apply `clip_timerange` to time dependent fx variables (:pull:`1603`) by :user:`sloosvel`
- Correctly handle requests.exceptions.ConnectTimeout when an ESGF index node is offline (:pull:`1638`) by :user:`bouweandela`

CMOR standard
~~~~~~~~~~~~~

- Added custom CMOR tables used for EMAC CMORizer (:pull:`1599`) by :user:`schlunma`
- Extended ICON CMORizer (:pull:`1549`) by :user:`schlunma`
- Add CMOR check exception for a basin coord named sector (:pull:`1612`) by :user:`dhohn`
- Custom user-defined location for custom CMOR tables (:pull:`1625`) by :user:`schlunma`

Containerization
~~~~~~~~~~~~~~~~

- Remove update command in Dockerfile (:pull:`1630`) by :user:`sloosvel`

Community
~~~~~~~~~

- Add David Hohn to contributors' list (:pull:`1586`) by :user:`valeriupredoi`

Documentation
~~~~~~~~~~~~~

- [Github Actions Docs] Full explanation on how to use the GA test triggered by PR comment and added docs link for GA hosted runners (:pull:`1553`) by :user:`valeriupredoi`
- Update the command for building the documentation (:pull:`1556`) by :user:`bouweandela`
- Update documentation on running the tool (:pull:`1400`) by :user:`bouweandela`
- Add support for DKRZ-Levante (:pull:`1558`) by :user:`remi-kazeroni`
- Improved documentation on native dataset support (:pull:`1559`) by :user:`schlunma`
- Tweak `extract_point` preprocessor: explain what it returns if one point coord outside cube and add explicit test (:pull:`1584`) by :user:`valeriupredoi`
- Update CircleCI, readthedocs, and Docker configuration (:pull:`1588`) by :user:`bouweandela`
- Remove support for Mistral in `config-user.yml` (:pull:`1620`) by :user:`remi-kazeroni`
- Add changelog for v2.6.0rc1 (:pull:`1633`) by :user:`sloosvel`
- Add a note on transferring permissions to the release manager (:pull:`1645`) by :user:`bouweandela`
- Add documentation on building and uploading Docker images (:pull:`1644`) by :user:`bouweandela`
- Update documentation on ESMValTool module at DKRZ (:pull:`1647`) by :user:`remi-kazeroni`
- Expanded information on deprecations in changelog (:pull:`1658`) by :user:`schlunma`

Improvements
~~~~~~~~~~~~

- Removed trailing whitespace in custom CMOR tables (:pull:`1564`) by :user:`schlunma`
- Try searching multiple ESGF index nodes (:pull:`1561`) by :user:`bouweandela`
- Add CMIP6 `amoc` derivation case and add a test (:pull:`1577`) by :user:`valeriupredoi`
- Added EMAC CMORizer (:pull:`1554`) by :user:`schlunma`
- Improve performance of `volume_statistics` (:pull:`1545`) by :user:`sloosvel`

Fixes for datasets
~~~~~~~~~~~~~~~~~~

- Fixes of ocean variables in multiple CMIP6 datasets (:pull:`1566`) by :user:`tomaslovato`
- Ensure lat/lon bounds in FGOALS-l3 atmos variables are contiguous (:pull:`1571`) by :user:`sloosvel`
- Added `AllVars` fix for CMIP6's ICON-ESM-LR (:pull:`1582`) by :user:`schlunma`

Installation
~~~~~~~~~~~~

- Removed `package/meta.yml` (:pull:`1540`) by :user:`schlunma`
- Pinned iris>=3.2.1 (:pull:`1552`) by :user:`schlunma`
- Use setuptools-scm to automatically generate the version number (:pull:`1578`) by :user:`bouweandela`
- Pin cf-units to lower than 3.1.0 to temporarily avoid changes within new version related to calendars (:pull:`1659`) by :user:`valeriupredoi`

Preprocessor
~~~~~~~~~~~~

- Allowed special case for unit conversion of precipitation (`kg m-2 s-1` <--> `mm day-1`) (:pull:`1574`) by :user:`schlunma`
- Add general `extract_coordinate_points` preprocessor (:pull:`1581`) by :user:`sloosvel`
- Add preprocessor `accumulate_coordinate` (:pull:`1281`) by :user:`jvegreg`
- Add `axis_statistics` and improve `depth_integration` (:pull:`1589`) by :user:`sloosvel`

Release
~~~~~~~

- Increase version number for ESMValCore v2.6.0rc1 (:pull:`1632`) by :user:`sloosvel`
- Update changelog and version for 2.6rc3 (:pull:`1646`) by :user:`sloosvel`
- Add changelog for rc4 (:pull:`1662`) by :user:`sloosvel`


Automatic testing
~~~~~~~~~~~~~~~~~

- Refresh CircleCI cache weekly (:pull:`1597`) by :user:`bouweandela`
- Use correct cache restore key on CircleCI (:pull:`1598`) by :user:`bouweandela`
- Install git and ssh before checking out code on CircleCI (:pull:`1601`) by :user:`bouweandela`
- Fetch all history in Github Action tests (:pull:`1622`) by :user:`sloosvel`
- Test Github Actions dashboard badge from meercode.io (:pull:`1640`) by :user:`valeriupredoi`
- Improve esmvalcore.esgf unit test (:pull:`1650`) by :user:`bouweandela`

Variable Derivation
~~~~~~~~~~~~~~~~~~~

- Added derivation of `hfns` (:pull:`1594`) by :user:`schlunma`

.. _changelog-v2-5-0:

------

Highlights
~~~~~~~~~~

- The new preprocessor :func:`~esmvalcore.preprocessor.extract_location` can extract arbitrary locations on the Earth using the `geopy <https://pypi.org/project/geopy/>`__ package that connects to OpenStreetMap. For details, see :ref:`Extract location <extract_location>`.
- Time ranges can now be extracted using the `ISO 8601 format <https://en.wikipedia.org/wiki/ISO_8601>`_. In addition, wildcards are allowed, which makes the time selection much more flexible. For details, see :ref:`Recipe section: Datasets <Datasets>`.
- The new preprocessor :func:`~esmvalcore.preprocessor.ensemble_statistics` can calculate arbitrary statistics over all ensemble members of a simulation. In addition, the preprocessor :func:`~esmvalcore.preprocessor.multi_model_statistics` now accepts the keyword ``groupy``, which allows the calculation of multi-model statistics over arbitrary multi-model ensembles. For details, see :ref:`Ensemble statistics <ensemble statistics>` and :ref:`Multi-model statistics <multi-model statistics>`.

This release includes

Backwards incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- Update Cordex section in `config-developer.yml` (:pull:`1303`) by :user:`francesco-cmcc`. This changes the naming convention of ESMValCore's output files from CORDEX dataset. This only affects recipes that use CORDEX data. Most likely, no changes in diagnostics are necessary; however, if code relies on the specific naming convention of files, it might need to be adapted.
- Dropped Python 3.7 (:pull:`1530`) by :user:`schlunma`. ESMValCore v2.5.0 dropped support for Python 3.7. From now on Python >=3.8 is required to install ESMValCore. The main reason for this is that conda-forge dropped support for Python 3.7 for OSX and arm64 (more details are given `here <https://github.com/ESMValGroup/ESMValTool/issues/2584#issuecomment-1063853630>`__).

Bug fixes
~~~~~~~~~

- Fix `extract_shape` when fx vars are present (:pull:`1403`) by :user:`sloosvel`
- Added support of `extra_facets` to fx variables added by the preprocessor (:pull:`1399`) by :user:`schlunma`
- Augmented input for derived variables with extra_facets (:pull:`1412`) by :user:`schlunma`
- Correctly use masked arrays after `unstructured_nearest` regridding (:pull:`1414`) by :user:`schlunma`
- Fixing the broken derivation script for XCH4 (and XCO2) (:pull:`1428`) by :user:`hb326`
- Ignore `.pymon-journal` file in test discovery (:pull:`1436`) by :user:`valeriupredoi`
- Fixed bug that caused automatic download to fail in rare cases (:pull:`1442`) by :user:`schlunma`
- Add new `JULIA_LOAD_PATH` to diagnostic task test (:pull:`1444`) by :user:`valeriupredoi`
- Fix provenance file permissions (:pull:`1468`) by :user:`bouweandela`
- Fixed usage of `statistics=std_dev` option in multi-model statistics preprocessors (:pull:`1478`) by :user:`schlunma`
- Removed scalar coordinates `p0` and `ptop` prior to merge in `multi_model_statistics` (:pull:`1471`) by :user:`axel-lauer`
- Added `dataset` and `alias` attributes to `multi_model_statistics` output (:pull:`1483`) by :user:`schlunma`
- Fixed issues with multi-model-statistics timeranges (:pull:`1486`) by :user:`schlunma`
- Fixed output messages for CMOR logging (:pull:`1494`) by :user:`schlunma`
- Fixed `clip_timerange` if only a single time point is extracted (:pull:`1497`) by :user:`schlunma`
- Fixed chunking in `multi_model_statistics` (:pull:`1500`) by :user:`schlunma`
- Fixed renaming of auxiliary coordinates in `multi_model_statistics` if coordinates are equal (:pull:`1502`) by :user:`schlunma`
- Fixed timerange selection for automatic downloads (:pull:`1517`) by :user:`schlunma`
- Fixed chunking in `multi_model_statistics` (:pull:`1524`) by :user:`schlunma`

Deprecations
~~~~~~~~~~~~

- Renamed vertical regridding schemes (:pull:`1429`) by :user:`schlunma`. Old regridding schemes are supported until v2.7.0. For details, see :ref:`Vertical interpolation schemes <Vertical interpolation schemes>`.

Documentation
~~~~~~~~~~~~~

- Remove duplicate entries in changelog (:pull:`1391`) by :user:`zklaus`
- Documentation on how to use HPC central installations (:pull:`1409`) by :user:`valeriupredoi`
- Correct brackets in preprocessor documentation for list of seasons (:pull:`1420`) by :user:`bouweandela`
- Add Python=3.10 to package info, update Circle CI auto install and documentation for Python=3.10 (:pull:`1432`) by :user:`valeriupredoi`
- Reverted unintentional change in `.zenodo.json` (:pull:`1452`) by :user:`schlunma`
- Synchronized config-user.yml with version from ESMValTool (:pull:`1453`) by :user:`schlunma`
- Solved issues in configuration files (:pull:`1457`) by :user:`schlunma`
- Add direct link to download conda lock file in the install documentation (:pull:`1462`) by :user:`valeriupredoi`
- CITATION.cff fix and automatic validation of citation metadata (:pull:`1467`) by :user:`valeriupredoi`
- Updated documentation on how to deprecate features (:pull:`1426`) by :user:`schlunma`
- Added reference hook to conda lock in documentation install section (:pull:`1473`) by :user:`valeriupredoi`
- Increased ESMValCore version to 2.5.0rc1 (:pull:`1477`) by :user:`schlunma`
- Added changelog for v2.5.0 release (:pull:`1476`) by :user:`schlunma`
- Increased ESMValCore version to 2.5.0rc2 (:pull:`1487`) by :user:`schlunma`
- Added some authors to citation and zenodo files (:pull:`1488`) by :user:`SarahAlidoost`
- Restored `scipy` intersphinx mapping (:pull:`1491`) by :user:`schlunma`
- Increased ESMValCore version to 2.5.0rc3 (:pull:`1504`) by :user:`schlunma`
- Fix download instructions for the MSWEP dataset (:pull:`1506`) by :user:`remi-kazeroni`
- Documentation updated for the new cmorizer framework (:pull:`1417`) by :user:`remi-kazeroni`
- Added tests for duplicates in changelog and removed duplicates (:pull:`1508`) by :user:`schlunma`
- Increased ESMValCore version to 2.5.0rc4 (:pull:`1519`) by :user:`schlunma`
- Add Github Actions Test badge in README (:pull:`1526`) by :user:`valeriupredoi`
- Increased ESMValCore version to 2.5.0rc5 (:pull:`1529`) by :user:`schlunma`
- Increased ESMValCore version to 2.5.0rc6 (:pull:`1532`) by :user:`schlunma`

Fixes for datasets
~~~~~~~~~~~~~~~~~~

- Added fix for AIRS v2.1 (obs4mips) (:pull:`1472`) by :user:`axel-lauer`

Preprocessor
~~~~~~~~~~~~

- Added bias preprocessor (:pull:`1406`) by :user:`schlunma`
- Improve error messages when a preprocessor is failing (:pull:`1408`) by :user:`schlunma`
- Added option to explicitly not use fx variables in preprocessors (:pull:`1416`) by :user:`schlunma`
- Add `extract_location` preprocessor to extract town, city, mountains etc - anything specifiable by a location (:pull:`1251`) by :user:`jvegreg`
- Add ensemble statistics preprocessor and 'groupby' option for multimodel (:pull:`673`) by :user:`sloosvel`
- Generic regridding preprocessor (:pull:`1448`) by :user:`zklaus`

Automatic testing
~~~~~~~~~~~~~~~~~

- Add `pandas` as dependency :panda_face: (:pull:`1402`) by :user:`valeriupredoi`
- Fixed tests for python 3.7 (:pull:`1410`) by :user:`schlunma`
- Remove accessing `.xml()` cube method from test (:pull:`1419`) by :user:`valeriupredoi`
- Remove flag to use pip 2020 solver from Github Action pip install command on OSX (:pull:`1357`) by :user:`valeriupredoi`
- Add Python=3.10 to Github Actions and switch to Python=3.10 for the Github Action that builds the PyPi package (:pull:`1430`) by :user:`valeriupredoi`
- Pin `flake8<4` to keep getting relevant error traces when tests fail with FLAKE8 issues (:pull:`1434`) by :user:`valeriupredoi`
- Implementing conda lock (:pull:`1164`) by :user:`valeriupredoi`
- Relocate `pytest-monitor` outputted database `.pymon` so `.pymon-journal` file should not be looked for by `pytest` (:pull:`1441`) by :user:`valeriupredoi`
- Switch to Mambaforge in Github Actions tests (:pull:`1438`) by :user:`valeriupredoi`
- Turn off conda lock file creation on any push on `main` branch from Github Action test (:pull:`1489`) by :user:`valeriupredoi`
- Add DRS path test for IPSLCM files (:pull:`1490`) by :user:`senesis`
- Add a test module that runs tests of `iris` I/O every time we notice serious bugs there (:pull:`1510`) by :user:`valeriupredoi`
- [Github Actions] Trigger Github Actions tests (`run-tests.yml` workflow) from a comment in a PR (:pull:`1520`) by :user:`valeriupredoi`
- Update Linux condalock file (various pull requests) github-actions[bot]

Installation
~~~~~~~~~~~~

- Move `nested-lookup` dependency to `environment.yml` to be installed from conda-forge instead of PyPi (:pull:`1481`) by :user:`valeriupredoi`
- Pinned `iris` (:pull:`1511`) by :user:`schlunma`
- Updated dependencies (:pull:`1521`) by :user:`schlunma`
- Pinned iris<3.2.0 (:pull:`1525`) by :user:`schlunma`

Improvements
~~~~~~~~~~~~

- Allow to load all files, first X years or last X years in an experiment (:pull:`1133`) by :user:`sloosvel`
- Filter tasks earlier (:pull:`1264`) by :user:`jvegreg`
- Added earlier validation for command line arguments (:pull:`1435`) by :user:`schlunma`
- Remove `profile_diagnostic` from diagnostic settings and increase test coverage of `_task.py` (:pull:`1404`) by :user:`valeriupredoi`
- Add `output2` to the `product` extra facet of CMIP5 data (:pull:`1514`) by :user:`remi-kazeroni`
- Speed up ESGF search (:pull:`1512`) by :user:`bouweandela`


.. _changelog-v2-4-0:

------

Highlights
~~~~~~~~~~

- ESMValCore now has the ability to automatically download missing data from ESGF. For details, see :ref:`Data Retrieval<data-retrieval>`.
- ESMValCore now also can resume an earlier run. This is useful to reuse expensive preprocessor results. For details, see :ref:`Running<running>`.

This release includes

Bug fixes
~~~~~~~~~

- Crop on the ID-selected region(s) and not on the whole shapefile (:pull:`1151`) by :user:`stefsmeets`
- Add 'comment' to list of removed attributes (:pull:`1244`) by :user:`Peter9192`
- Speed up multimodel statistics and fix bug in peak computation (:pull:`1301`) by :user:`bouweandela`
- No longer make plots of provenance (:pull:`1307`) by :user:`bouweandela`
- No longer embed provenance in output files (:pull:`1306`) by :user:`bouweandela`
- Removed automatic addition of areacello to obs4mips datasets (:pull:`1316`) by :user:`schlunma`
- Pin docutils <0.17 to fix bullet lists on readthedocs (:pull:`1320`) by :user:`zklaus`
- Fix obs4MIPs capitalization (:pull:`1328`) by :user:`bouweandela`
- Fix Python 3.7 tests (:pull:`1330`) by :user:`bouweandela`
- Handle fx variables in `extract_levels` and some time operations (:pull:`1269`) by :user:`sloosvel`
- Refactored mask regridding for irregular grids (fixes 772) (:pull:`865`) by :user:`zklaus`
- Fix `da.broadcast_to` call when the fx cube has different shape than target data cube (:pull:`1350`) by :user:`valeriupredoi`
- Add tests for _aggregate_time_fx (:pull:`1354`) by :user:`sloosvel`
- Fix extra facets (:pull:`1360`) by :user:`bouweandela`
- Pin pip!=21.3 to avoid pypa/pip10573 with editable installs (:pull:`1359`) by :user:`zklaus`
- Add a custom `date2num` function to deal with changes in cftime (:pull:`1373`) by :user:`zklaus`
- Removed custom version of `AtmosphereSigmaFactory` (:pull:`1382`) by :user:`schlunma`

Deprecations
~~~~~~~~~~~~

- Remove write_netcdf and write_plots from config-user.yml (:pull:`1300`) by :user:`bouweandela`

Documentation
~~~~~~~~~~~~~

- Add link to plot directory in index.html (:pull:`1256`) by :user:`stefsmeets`
- Work around issue with yapf not following PEP8 (:pull:`1277`) by :user:`bouweandela`
- Update the core development team (:pull:`1278`) by :user:`bouweandela`
- Update the documentation of the provenance interface (:pull:`1305`) by :user:`bouweandela`
- Update version number to first release candidate 2.4.0rc1 (:pull:`1363`) by :user:`zklaus`
- Update to new ESMValTool logo (:pull:`1374`) by :user:`zklaus`
- Update version number for third release candidate 2.4.0rc3 (:pull:`1384`) by :user:`zklaus`
- Update changelog for 2.4.0rc3 (:pull:`1385`) by :user:`zklaus`
- Update version number to final 2.4.0 release (:pull:`1389`) by :user:`zklaus`
- Update changelog for 2.4.0 (:pull:`1366`) by :user:`zklaus`

Fixes for datasets
~~~~~~~~~~~~~~~~~~

- Add fix for differing latitude coordinate between historical and ssp585 in MPI-ESM1-2-HR r2i1p1f1 (:pull:`1292`) by :user:`bouweandela`
- Add fixes for time and latitude coordinate of EC-Earth3 r3i1p1f1 (:pull:`1290`) by :user:`bouweandela`
- Apply latitude fix to all CCSM4 variables (:pull:`1295`) by :user:`bouweandela`
- Fix lat and lon bounds for FGOALS-g3 mrsos (:pull:`1289`) by :user:`thomascrocker`
- Add grid fix for tos in fgoals-f3-l (:pull:`1326`) by :user:`sloosvel`
- Add fix for CIESM pr (:pull:`1344`) by :user:`bouweandela`
- Fix DRS for IPSLCM : split attribute 'freq' into : 'out' and 'freq' (:pull:`1304`) by :user:`senesis`

CMOR standard
~~~~~~~~~~~~~

- Remove history attribute from coords (:pull:`1276`) by :user:`jvegreg`
- Increased flexibility of CMOR checks for datasets with generic alevel coordinates (:pull:`1032`) by :user:`schlunma`
- Automatically fix small deviations in vertical levels (:pull:`1177`) by :user:`bouweandela`
- Adding standard names to the custom tables of the `rlns` and `rsns` variables (:pull:`1386`) by :user:`remi-kazeroni`

Preprocessor
~~~~~~~~~~~~

- Implemented fully lazy climate_statistics (:pull:`1194`) by :user:`schlunma`
- Run the multimodel statistics preprocessor last (:pull:`1299`) by :user:`bouweandela`

Automatic testing
~~~~~~~~~~~~~~~~~

- Improving test coverage for _task.py (:pull:`514`) by :user:`valeriupredoi`
- Upload coverage to codecov (:pull:`1190`) by :user:`bouweandela`
- Improve codecov status checks (:pull:`1195`) by :user:`bouweandela`
- Fix curl install in CircleCI (:pull:`1228`) by :user:`jvegreg`
- Drop support for Python 3.6 (:pull:`1200`) by :user:`valeriupredoi`
- Allow more recent version of `scipy` (:pull:`1182`) by :user:`schlunma`
- Speed up conda build `conda_build` Circle test by using `mamba` solver via `boa` (and use it for Github Actions test too) (:pull:`1243`) by :user:`valeriupredoi`
- Fix numpy deprecation warnings (:pull:`1274`) by :user:`bouweandela`
- Unpin upper bound for iris (previously was at <3.0.4) (:pull:`1275`) by :user:`valeriupredoi`
- Modernize `conda_install` test on Circle CI by installing from conda-forge with Python 3.9 and change install instructions in documentation (:pull:`1280`) by :user:`valeriupredoi`
- Run a nightly Github Actions workflow to monitor tests memory per test (configurable for other metrics too) (:pull:`1284`) by :user:`valeriupredoi`
- Speed up tests of tasks (:pull:`1302`) by :user:`bouweandela`
- Fix upper case to lower case variables and functions for flake compliance in `tests/unit/preprocessor/_regrid/test_extract_levels.py` (:pull:`1347`) by :user:`valeriupredoi`
- Cleaned up a bit Github Actions workflows (:pull:`1345`) by :user:`valeriupredoi`
- Update circleci jobs: renaming tests to more descriptive names and removing conda build test (:pull:`1351`) by :user:`zklaus`
- Pin iris to latest `>=3.1.0` (:pull:`1341`) by :user:`valeriupredoi`

Installation
~~~~~~~~~~~~

- Pin esmpy to anything but 8.1.0 since that particular one changes the CPU affinity (:pull:`1310`) by :user:`valeriupredoi`

Improvements
~~~~~~~~~~~~

- Add a more friendly and useful message when using default config file (:pull:`1233`) by :user:`valeriupredoi`
- Replace os.walk by glob.glob in data finder (only look for data in the specified locations) (:pull:`1261`) by :user:`bouweandela`
- Machine-specific directories for auxiliary data in the `config-user.yml` file (:pull:`1268`) by :user:`remi-kazeroni`
- Add an option to download missing data from ESGF (:pull:`1217`) by :user:`bouweandela`
- Speed up provenance recording (:pull:`1327`) by :user:`bouweandela`
- Improve results web page (:pull:`1332`) by :user:`bouweandela`
- Move institutes from config-developer.yml to default extra facets config and add wildcard support for extra facets (:pull:`1259`) by :user:`bouweandela`
- Add support for reusing preprocessor output from previous runs (:pull:`1321`) by :user:`bouweandela`
- Log fewer messages to screen and hide stack trace for known recipe errors (:pull:`1296`) by :user:`bouweandela`
- Log ESMValCore and ESMValTool versions when running (:pull:`1263`) by :user:`jvegreg`
- Add "grid" as a tag to the output file template for CMIP6 (:pull:`1356`) by :user:`zklaus`
- Implemented ICON project to read native ICON model output (:pull:`1079`) by :user:`bsolino`


.. _changelog-v2-3-1: