Esmvalcore

-------

Highlights
~~~~~~~~~~

- Preprocessor :func:`esmvalcore.preprocessor.extract_time` now allows to
extract time blocks in each year by making parameters ``start_year`` and
``end_year`` optional.
- A new way of :ref:`configuring the tool <config>` has been developed.
- Performance improvements:

- An iris-esmf-regrid scheme has been added to
preprocessor :func:`esmvalcore.preprocessor.regrid`, which improves
the regridding of 2D grids and adds the capability to regrid UGRID meshes
out of the box.
- Data is now saved from one preprocessing task at the time when using the
distributed scheduler, in order to avoid running out of memory.
- A better default ``num_workers`` has been set when using more than one
``max_parallel_tasks`` with an unconfigured threaded scheduler.

- An on-the-fly cmorizer for ACCESS native data is now available.

This release includes

Backwards incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- Make derivation of total column ozone (`toz`) more flexible and add derivation of stratospheric and tropospheric column ozone (:pull:`2509`) by :user:`schlunma`

- The units of `toz` have been changed from ``DU`` to ``m`` to be consistent with the CMIP6 CMOR table.
To restore the old behaviour, preprocessor :func:`esmvalcore.preprocessor.convert_units` can be used
to set the units back to ``DU``.

- Remove deprecated CMOR fix/check code (:pull:`2552`) by :user:`schlunma`

- CMOR fixes and checks have been clearly separated in v2.10.0, and the old code has now been removed.
Use functions :func:`esmvalcore.preprocessors.fix_metadata`, :func:`esmvalcore.preprocessors.fix_data`,
or :func:`esmvalcore.dataset.Dataset.load` to fix data.
Use functions :func:`esmvalcore.preprocessor.cmor_check_metadata`, :func:`esmvalcore.preprocessor.cmor_check_data`,
or :func:`esmvalcore.preprocessor.cmor_check` to check data.

- Remove deprecated statistical operators (:pull:`2553`) by :user:`schlunma`

- Old statistical operators that have been deprecated in v2.10.0 have now been removed.
Please refer to :ref:`stat_preprocs` for a detailed description on how to use the operators.

- Save all files in a task at the same time to avoid recomputing intermediate results (:pull:`2522`) by :user:`bouweandela`

- The signature of the preprocessor function :func:`~esmvalcore.preprocessor.save`
has changed. The function now accepts a ``compute`` argument that can be :obj:`True`,
in which case the return value will be :obj:`None` or :obj:`False`, in which case
the return value will be a :class:`~dask.delayed.Delayed` object that can be used
to compute and save the data of the cube.

Deprecations
~~~~~~~~~~~~

- Merge configuration object from multiple files (instead of one single file) (:pull:`2448`) by :user:`schlunma`

- The single configuration file ``config-user.yml`` has been deprecated in favour of configuration directories.
By default, the directory ``~/.config/esmvaltool`` will be considered.
To switch to the new format run:

.. code-block:: bash

mkdir -p ~/.config/esmvaltool && mv ~/.esmvaltool/config-user.yml ~/.config/esmvaltool

You can also specify the location of the configuration directory with the ``--config_dir`` flag.
Please refer to :ref:`config` for a detailed description on how to configure the tool.

- Make Dask configurable in our configuration (:pull:`2616`) by :user:`schlunma`

- The old Dask configuration file that needed to be located at ``~/.esmvaltool/dask.yml`` is now deprecated.
Please refer to :ref:`config-dask` for a detailed description on how to configure Dask.

Bug fixes
~~~~~~~~~

- Concatenate by experiment before concatenating all input files (:pull:`2343`) by :user:`dhohn`
- Update `CFG` with configuration options given via command line (:pull:`2595`) by :user:`schlunma`
- Avoid a crash when there is a timeout when shutting down the Dask cluster (:pull:`2580`) by :user:`bouweandela`
- More reliable datasets to recipe conversion (:pull:`2472`) by :user:`bouweandela`
- Avoid mutating the input cubes when building the combined cube in preprocessor function ``multi_model_statistics`` (:pull:`2564`) by :user:`bouweandela`
- Do not change function argument names when decorator `preserve_float_dtype` is used (:pull:`2645`) by :user:`schlunma`
- Always write target coordinates to source cube after regridding (:pull:`2673`) by :user:`schlunma`

CMOR standard
~~~~~~~~~~~~~

- New custom variable for tos uncertainty (:pull:`2470`) by :user:`LisaBock`
- Add `prc` fix for native6 ERA5 CMORization (:pull:`2550`) by :user:`malininae`
- Added more variables to EMAC extra facets (:pull:`2617`) by :user:`schlunma`
- Adding custom tables for ETCCDI indices (:pull:`2442`) by :user:`malininae`

Configuration
~~~~~~~~~~~~~

- Add public `Config.update_from_dirs()` method (:pull:`2538`) by :user:`schlunma`
- Do not use `Path` objects as configuration dictionary keys to avoid errors in `dask.config.merge` (:pull:`2578`) by :user:`schlunma`
- Revise the `user-config.yml` to support updated MO user config requirements (:pull:`2658`) by :user:`ehogan`

Computational performance improvements
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- Add an iris-esmf-regrid based regridding scheme (:pull:`2457`) by :user:`bouweandela`
- Miscellaneous lazy preprocessor improvements (:pull:`2520`) by :user:`bouweandela`
- Only save data from one preprocessing task at a time with the Distributed scheduler (:pull:`2610`) by :user:`bouweandela`
- Use better defaults when using ``max_parallel_tasks`` with an unconfigured threaded scheduler (:pull:`2626`) by :user:`bouweandela`
- Fix OSX compatibility (:pull:`2636`) by :user:`bouweandela`
- Make sure that supplementary variables and weights have same chunks as parent cube (:pull:`2637`) by :user:`schlunma`

Documentation
~~~~~~~~~~~~~

- Ignore autosummary warning in documentation build (:pull:`2480`) by :user:`bouweandela`
- Fix documentation build and broken link (:pull:`2519`) by :user:`bouweandela`
- Dark mode compatible logo (:pull:`2532`) by :user:`lukruh`
- Add a pre-commit badge to README (:pull:`2534`) by :user:`valeriupredoi`
- Retire Mambaforge (:pull:`2556`) by :user:`valeriupredoi`
- Readthedocs configuration: temporary revert to miniconda before miniforge3 becomes available (remove Mambaforge) (:pull:`2562`) by :user:`valeriupredoi`
- Remove Docker build badge in README (:pull:`2565`) by :user:`valeriupredoi`
- Optimize documentation about Earth mover distance in distance_metric preprocessor (:pull:`2423`) by :user:`schlunma`
- Update the `esmvaltool` command welcome message (:pull:`2635`) by :user:`bouweandela`
- Pin sphinx to < 8.2 (:pull:`2671`) by :user:`bouweandela`

Fixes for datasets
~~~~~~~~~~~~~~~~~~

- Extra facets added for EMAC to map o3, tro3 and aps (:pull:`2501`) by :user:`FranziskaWinterstein`
- Fix for CMIP6 AWI-ESM-1-1-LR parent time units (:pull:`2507`) by :user:`brittaGrusdt`
- Use our own unit conversion function in our fixes (:pull:`2560`) by :user:`schlunma`
- Missing 2m height coordinate and monotonicity for tasmin in CESM2 and CESM2-WACCM (:pull:`2574`) by :user:`Karen-A-Garcia`
- Monotonicity fixes for Fgoals (:pull:`2603`) by :user:`Karen-A-Garcia`
- Expand Amon fix of FIO-ESM-2-0 (CMIP6) (:pull:`2619`) by :user:`schlunma`
- Added fixes for some 3D atmospheric variables of E3SM-1-1 (CMIP6) (:pull:`2620`) by :user:`schlunma`
- Correct incorrect time bounds in EMAC data (:pull:`2621`) by :user:`FranziskaWinterstein`
- Do not copy ICON horizontal grid every time it is used (:pull:`2633`) by :user:`schlunma`
- Fix `oh` for model: EC-Earth3-AerChem mip: AERMonZ (:pull:`2634`) by :user:`valeriupredoi`
- Update CMIP5 EC-EARTH pr fix (:pull:`2666`) by :user:`bouweandela`
- Add a fix for differing index coord long names in NorESM2-MM and EC-Earth3-Veg-LR (:pull:`2667`) by :user:`bouweandela`

Installation
~~~~~~~~~~~~

- Free esmpy of ` >=8.6.0` pin and pin `iris-grib >=0.20.0` (:pull:`2542`) by :user:`valeriupredoi`
- Use pyproject.toml instead of setup.py/setup.cfg (:pull:`2540`) by :user:`bouweandela`
- Use `miniforge3` for our docker builds instead of `mambaforge` (:pull:`2558`) by :user:`valeriupredoi`
- Support Python 3.13 (:pull:`2566`) by :user:`valeriupredoi`
- Pin dask (:pull:`2654`) by :user:`sloosvel`

Iris
~~~~

- Set iris.FUTURE flags in one place (:pull:`2622`) by :user:`bouweandela`
- Silence Iris warnings in ``area_statistics`` preprocessor function (:pull:`2625`) by :user:`bouweandela`

Preprocessor
~~~~~~~~~~~~

- Merge input cubes only once when computing lazy multimodel statistics (:pull:`2518`) by :user:`bouweandela`
- Make `start_year`, `end_year` in `extract_time` optional to obtain time blocks in each year (:pull:`2490`) by :user:`malininae`
- Adding hurs (relative humidity) derivation script (:pull:`2397`) by :user:`malininae`
- Added cumulative sum preprocessor (:pull:`2642`) by :user:`schlunma`
- Unified ignoring of `iris.warnings.IrisVagueMetadataWarning` in preprocessors (:pull:`2646`) by :user:`schlunma`
- Raise error if weights are used with unweighted aggregator (:pull:`2640`) by :user:`schlunma`
- Allow using multi model statistics preprocessor on datasets without `timerange` (:pull:`2644`) by :user:`schlunma`
- Allow using output from `multi_model_statistics` or `ensemble_statistics` as reference for `bias` or `distance_metric` (:pull:`2652`) by :user:`schlunma`
- Add option to ignore horizontal coordinates if there are multiple when regridding (:pull:`2672`) by :user:`bouweandela`

Observational and re-analysis dataset support
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- Add support for native ERA5 data in GRIB format (:pull:`2178`) by :user:`schlunma`

Automatic testing
~~~~~~~~~~~~~~~~~

- Run a nightly test with the development version of dependencies (:pull:`2478`) by :user:`bouweandela`
- Use ruff formatter and pre-commit (:pull:`2524`) by :user:`bouweandela`
- Fix tests if deprecated `~/.esmvaltool/config-user.yml` file is available (:pull:`2543`) by :user:`schlunma`
- Disable upstream tests on commits (:pull:`2548`) by :user:`bouweandela`
- Disable collecting test coverage by default (:pull:`2456`) by :user:`bouweandela`
- Enable ruff flake8-bugbear rule (:pull:`2536`) by :user:`bouweandela`
- Pin mamba in conda lock creation github action (:pull:`2561`) by :user:`valeriupredoi`
- [Numpy2] Support for `numpy==2.0.0` (and pin `iris >=3.11`) (:pull:`2395`) by :user:`valeriupredoi`
- Switch back to Python 3.12 for conda lock file creation due to mamba<2 pin (:pull:`2606`) by :user:`valeriupredoi`
- Always ignore user's configuration when running Dask tests (:pull:`2624`) by :user:`schlunma`
- Restrict runs of cron Github Actions on forks (:pull:`2649`) by :user:`valeriupredoi`
- Fix test that loads realistic GRIB file (:pull:`2665`) by :user:`schlunma`
- Fix failing test with Dask 2025.2: ours issue not theirs (:pull:`2663`) by :user:`valeriupredoi`

Variable Derivation
~~~~~~~~~~~~~~~~~~~

- Introduction of the variable prodlnox for EMAC (:pull:`2499`) by :user:`FranziskaWinterstein`

Improvements
~~~~~~~~~~~~

- On-the-fly cmoriser for ACCESS native data (:pull:`2430`) by :user:`rhaegar325`
- Fix CFF file (:pull:`2476`) by :user:`rbeucher`
- Write settings.yml parameters in original order (:pull:`2352`) by :user:`enekomartinmartinez`
- Fix `access-mapping.yml` extra_facets title (:pull:`2485`) by :user:`rhaegar325`
- Remove ability to log on to ESGF (:pull:`2508`) by :user:`bouweandela`
- Disable automatic fixes by pre-commit.ci (:pull:`2527`) by :user:`bouweandela`
- Ignore reformatting when viewing git blame (:pull:`2539`) by :user:`bouweandela`
- Enable ruff pydocstyle linter rule (:pull:`2547`) by :user:`bouweandela`
- Allows relative paths for diagnostic scripts. (:pull:`2329`) by :user:`rbeucher`
- Fix 2593 Change log INFO to DEBUG (:pull:`2600`) by :user:`rbeucher`


.. _changelog-v2-11-1:

-------

Highlights
~~~~~~~~~~

This is a bugfix release which enables lazy computations in more preprocessors
and allows installing the latests version of various dependencies, including
Iris (`v3.11.0 <https://github.com/SciTools/iris/releases/tag/v3.11.0>`__).

This release includes

Computational performance improvements
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- Optimize functions ``mask_landsea()``, ``mask_landseaice()`` and ``calculate_volume()`` for lazy input (:pull:`2515`) by :user:`schlunma`

Installation
~~~~~~~~~~~~

- Remove support for Python 3.9 (:pull:`2447`) by :user:`valeriupredoi`
- Switch to new iris >= 3.10.0 API (:pull:`2500`) by :user:`schlunma`
- Pin dask to avoid 2024.8.0 - problems with masked fill/missing values (:pull:`2504`) by :user:`valeriupredoi`
- Fix rounding of Pandas datetimes in ICON CMORizer to allow installing latest Pandas version (:pull:`2529`) by :user:`valeriupredoi`

Automatic testing
~~~~~~~~~~~~~~~~~

- Fix type hint for new mypy version (:pull:`2497`) by :user:`schlunma`
- Reformat datetime strings be in line with new ``isodate==0.7.0`` and actual ISO8601 and pin ``isodate>=0.7.0`` (:pull:`2546`) by :user:`valeriupredoi`

.. _changelog-v2-11-0:

-------

Highlights
~~~~~~~~~~

- Performance improvements have been made to many preprocessors:

- Preprocessors :func:`esmvalcore.preprocessor.mask_landsea`,
:func:`esmvalcore.preprocessor.mask_landseaice`,
:func:`esmvalcore.preprocessor.mask_glaciated`,
:func:`esmvalcore.preprocessor.extract_levels` are now lazy

- Several new preprocessors have been added:

- :func:`esmvalcore.preprocessor.local_solar_time`
- :func:`esmvalcore.preprocessor.distance_metrics`
- :func:`esmvalcore.preprocessor.histogram`

- NEW TREND: First time release manager shout-outs!

- This is the first ESMValTool release managed by the Met Office! We want to
shout this out - and for all future first time release managers to
shout-out - to celebrate the growing, thriving ESMValTool community.

This release includes

Backwards incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- Allow contiguous representation of extracted regions (:pull:`2230`) by :user:`rebeccaherman1`

- The preprocessor function :func:`esmvalcore.preprocessor.extract_region`
no longer automatically maps the extracted :class:`iris.cube.Cube` to the
0-360 degrees longitude domain. If you need this behaviour, use
``cube.intersection(longitude=(0., 360.))`` in your Python code after
extracting the region. There is no possibility to restore the previous
behaviour from a recipe.

- Use ``iris.FUTURE.save_split_attrs = True`` to remove iris warning (:pull:`2398`) by :user:`schlunma`

- Since `v3.8.0`_, Iris explicitly distinguishes between local and global
netCDF attributes. ESMValCore adopted this behavior with v2.11.0. With
this change, attributes are written as local attributes by default, unless
they already existed as global attributes or belong to a special list of
global attributes (in which case attributes are written as global
attributes). See :class:`iris.cube.CubeAttrsDict` for details.

.. _v3.8.0: https://scitools-iris.readthedocs.io/en/stable/whatsnew/3.8.html#v3-8-29-feb-2024

Deprecations
~~~~~~~~~~~~

- Refactor regridding (:pull:`2231`) by :user:`schlunma`

- This PR deprecated two regridding schemes, which will be removed with
ESMValCore v2.13.0:

- ``unstructured_nearest``: Please use the scheme ``nearest`` instead.
This is an exact replacement for data on unstructured grids. ESMValCore
is now able to determine the most suitable regridding scheme based on
the input data.
- ``linear_extrapolate``: Please use a generic scheme with
``reference: iris.analysis:Linear`` and
``extrapolation_mode: extrapolate`` instead.

- Allow deprecated regridding scheme ``linear_extrapolate`` in recipe checks (:pull:`2324`) by :user:`schlunma`
- Allow deprecated regridding scheme ``unstructured_nearest`` in recipe checks (:pull:`2336`) by :user:`schlunma`

Bug fixes
~~~~~~~~~

- Do not overwrite facets from recipe with CMOR table facets for derived variables (:pull:`2255`) by :user:`bouweandela`
- Fix error message in variable definition check (:pull:`2313`) by :user:`enekomartinmartinez`
- Unify dtype handling of preprocessors (:pull:`2393`) by :user:`schlunma`
- Fix bug in ``_rechunk_aux_factory_dependencies`` (:pull:`2428`) by :user:`ehogan`
- Avoid loading entire files into memory when downloading from ESGF (:pull:`2434`) by :user:`bouweandela`
- Preserve cube attribute global vs local when concatenating (:pull:`2449`) by :user:`bouweandela`

CMOR standard
~~~~~~~~~~~~~

- Also read default custom CMOR tables if custom location is specified (:pull:`2279`) by :user:`schlunma`
- Add custom CMOR table for total cloud water (tcw) (:pull:`2277`) by :user:`axel-lauer`
- Add height for sfcWindmax in MPI HighRes models (:pull:`2292`) by :user:`malininae`
- Fixed ``positive`` attribute in custom rtnt table (:pull:`2367`) by :user:`schlunma`
- Fix ``positive`` attributes in custom CMOR variables (:pull:`2380`) by :user:`schlunma`
- Log CMOR check and generic fix output to separate file (:pull:`2361`) by :user:`schlunma`

Computational performance improvements
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- More lazy fixes and preprocessing functions (:pull:`2325`) by :user:`bouweandela`
- Made preprocessors :func:`esmvalcore.preprocessor.mask_landsea`, :func:`esmvalcore.preprocessor.mask_landseaice` and :func:`esmvalcore.preprocessor.mask_glaciated` lazy (:pull:`2268`) by :user:`joergbenke`
- More lazy :func:`esmvalcore.preprocessor.extract_levels` preprocessor function (:pull:`2120`) by :user:`bouweandela`
- Use lazy weights for :func:`esmvalcore.preprocessor.climate_statistics` and :func:`esmvalcore.preprocessor.axis_statistics` (:pull:`2346`) by :user:`schlunma`
- Fixed potential memory leak in :func:`esmvalcore.preprocessor.local_solar_time` (:pull:`2356`) by :user:`schlunma`
- Cache regridding weights if possible (:pull:`2344`) by :user:`schlunma`
- Implement lazy area weights (:pull:`2354`) by :user:`schlunma`
- Avoid large chunks in :func:`esmvalcore.preprocessor.climate_statistics` preprocessor function with `period='full'` (:pull:`2404`) by :user:`bouweandela`
- Load data only once for ESMPy regridders (:pull:`2418`) by :user:`bouweandela`

Documentation
~~~~~~~~~~~~~

- Use short links in changelog (:pull:`2287`) by :user:`bouweandela`
- National Computing Infrastructure (NCI), Site specific configuration (:pull:`2281`) by :user:`rbeucher`
- Update :func:`esmvalcore.preprocessor.multi_model_statistics` doc with latest changes (new operators, etc.) (:pull:`2321`) by :user:`schlunma`
- Fix Codacy badge (:pull:`2382`) by :user:`bouweandela`
- Change 'mean' to 'percentile' in doc strings of preprocessor statistics (:pull:`2327`) by :user:`lukruh`
- Fixed typo in doc about weighted statistics (:pull:`2387`) by :user:`schlunma`

Fixes for datasets
~~~~~~~~~~~~~~~~~~

- Fixing missing height 2m coordinates in GFDL-CM4 and KIOST-ESM (:pull:`2294`) by :user:`Karen-A-Garcia`
- Added fix for wrong units of ``clt`` for CIESM and FIO-ESM-2-0 (:pull:`2330`) by :user:`schlunma`
- Cmip6 gfdl_cm4: fix tas height fix to work for concatenated scenarios (:pull:`2332`) by :user:`mwjury`
- Cordex GERICS REMO2015 lon differences above 10e-4 (:pull:`2334`) by :user:`mwjury`
- Download ICON grid without locking (:pull:`2359`) by :user:`bouweandela`
- Added ICON fixes for hfls and hfss (:pull:`2360`) by :user:`diegokam`
- Added ICON fix for rtnt (:pull:`2366`) by :user:`diegokam`
- Expanded ICON extra facets (:pull:`2379`) by :user:`schlunma`
- Add 10m height coordinate to SfcWind GFDL-CM4 instead of 2m height (:pull:`2385`) by :user:`Karen-A-Garcia`
- Cordex wrf381p: fix tas,tasmax,tasmin height (:pull:`2333`) by :user:`mwjury`
- Several minor fixes needed for marine BGC data. (:pull:`2110`) by :user:`ledm`

Installation
~~~~~~~~~~~~

- Pin pandas yet again avoid new ``2.2.1`` as well (:pull:`2353`) by :user:`valeriupredoi`
- Update Iris pin to avoid using versions with memory issues (:pull:`2408`) by :user:`chrisbillowsMO`
- Pin esmpy <8.6.0 (:pull:`2402`) by :user:`valeriupredoi`
- Pin numpy<2.0.0 to avoid pulling 2.0.0rcX (:pull:`2415`) by :user:`valeriupredoi`
- Add support for Python=3.12 (:pull:`2228`) by :user:`valeriupredoi`

Preprocessor
~~~~~~~~~~~~

- New preprocessor: :func:`esmvalcore.preprocessor.local_solar_time` (:pull:`2258`) by :user:`schlunma`
- Read derived variables from other MIP tables (:pull:`2256`) by :user:`bouweandela`
- Added special unit conversion m -> DU for total column ozone (toz) (:pull:`2270`) by :user:`schlunma`
- Allow cubes as input for :func:`esmvalcore.preprocessor.bias` preprocessor (:pull:`2183`) by :user:`schlunma`
- Add normalization with statistics to many statistics preprocessors (:pull:`2189`) by :user:`schlunma`
- Adding sfcWind derivation from uas and vas (:pull:`2242`) by :user:`malininae`
- Update interval check in resample_hours (:pull:`2362`) by :user:`axel-lauer`
- Broadcast properly ``cell_measures`` when using :func:`esmvalcore.preprocessor.extract_shape` with ``decomposed: True`` (:pull:`2348`) by :user:`sloosvel`
- Compute volume from ``cell_area`` if available (:pull:`2318`) by :user:`enekomartinmartinez`
- Do not expand wildcards for datasets of derived variables where not all input variables are available (:pull:`2374`) by :user:`schlunma`
- Modernize :func:`esmvalcore.preprocessor.regrid_time` and allow setting a common calendar for decadal, yearly, and monthly data (:pull:`2311`) by :user:`schlunma`
- Added unstructured linear regridding (:pull:`2350`) by :user:`schlunma`
- Add preprocessors :func:`esmvalcore.preprocessor.distance_metrics` and :func:`esmvalcore.preprocessor.histogram` (:pull:`2299`) by :user:`schlunma`

Automatic testing
~~~~~~~~~~~~~~~~~

- Increase resources for testing installation from conda-forge (:pull:`2297`) by :user:`bouweandela`
- Pin pandas to avoid broken ``round`` function (:pull:`2305`) by :user:`schlunma`
- Remove team reviewers from conda lock generation workflow in Github Actions (:pull:`2307`) by :user:`valeriupredoi`
- Remove mocking from tests in ``tests/unit/preprocessor/_regrid/test_extract_point.py`` (:pull:`2193`) by :user:`ehogan`
- Pin ``pytest-mypy`` plugin to ``>=0.10.3`` comply with new ``pytest==8`` (:pull:`2315`) by :user:`valeriupredoi`
- Fix regridding test for unstructured nearest regridding on OSX (:pull:`2319`) by :user:`schlunma`
- Fix flaky regrid test by clearing LRU cache after each test (:pull:`2322`) by :user:`valeriupredoi`
- Xfail ``tests/integration/cmor/_fixes/test_common.py::test_cl_hybrid_height_coord_fix_metadata`` while Iris folk fix behaviour (:pull:`2363`) by :user:`valeriupredoi`
- Update codacy reporter orb to latest version (:pull:`2388`) by :user:`valeriupredoi`
- Add calls to ``conda list`` in Github Action test workflows to inspect environment (:pull:`2391`) by :user:`valeriupredoi`
- Pin pandas yet again :panda_face: ``test_icon`` fails again with pandas=2.2.2 (:pull:`2394`) by :user:`valeriupredoi`
- Fixed units of cl test data (necessary since iris>=3.8.0) (:pull:`2403`) by :user:`schlunma`

Improvements
~~~~~~~~~~~~

- Show files of supplementary variables explicitly in log (:pull:`2303`) by :user:`schlunma`
- Remove warning about logging in to ESGF (:pull:`2326`) by :user:`bouweandela`
- Do not read ``~/.esmvaltool/config-user.yml`` if ``--config-file`` is used (:pull:`2309`) by :user:`schlunma`
- Support loading ICON grid from ICON rootpath (:pull:`2337`) by :user:`schlunma`
- Handle warnings about invalid units for iris>=3.8 (:pull:`2378`) by :user:`schlunma`
- Added note on how to access ``index.html`` on remote server (:pull:`2276`) by :user:`schlunma`
- Remove custom fix for concatenation of aux factories now that bug in iris is solved (:pull:`2392`) by :user:`schlunma`
- Ignored iris warnings about global attributes (:pull:`2400`) by :user:`schlunma`
- Add native6, OBS6 and RAWOBS rootpaths to metoffice config-user.yml template, and remove temporary dir (:pull:`2432`) by :user:`alistairsellar`

.. _changelog-v2-10-0:

-------

Highlights
~~~~~~~~~~

- All statistics preprocessors support the same operators and have a common
:ref:`documentation <stat_preprocs>`. In addition, arbitrary keyword arguments
for the statistical operation can be directly given to the preprocessor.

- The output webpage generated by the tool now looks better and provides
methods to select and filter the output.

- Improved computational efficiency:

- Automatic rechunking between preprocessor steps to keep the
`graph size smaller <https://docs.dask.org/en/latest/best-practices.html#avoid-very-large-graphs>`_
and the `chunk size optimal <https://blog.dask.org/2021/11/02/choosing-dask-chunk-sizes>`__.
- Reduce the size of the dask graph created by :func:`esmvalcore.preprocessor.anomalies`.
- Preprocessors :func:`esmvalcore.preprocessor.mask_above_threshold`,
:func:`esmvalcore.preprocessor.mask_below_threshold`,
:func:`esmvalcore.preprocessor.mask_inside_range`,
:func:`esmvalcore.preprocessor.mask_outside_range` are now lazy.
- Lazy coordinates bounds are no longer loaded into memory by the CMOR checks and fixes.

This release includes

Backwards incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- Remove the deprecated option ``use_legacy_supplementaries`` (:pull:`2202`) by :user:`bouweandela`

- The recommended upgrade procedure is to remove ``use_legacy_supplementaries`` from config-user.yml
(if it was there) and remove any mention of ``fx_variables`` from the recipe. If automatically defining
the required supplementary variables does not work, define them in the variable or
(``additional_``) ``datasets`` section as described in :ref:`supplementary_variables`.

- Use smarter (units-aware) weights (:pull:`2139`) by :user:`schlunma`

- Some preprocessors handle units better. For details, see the pull request.

- Removed deprecated configuration option ``offline`` (:pull:`2213`) by :user:`schlunma`

- In :ref:`changelog-v2-8-0`, we replaced the old ``offline`` configuration option. From this version on, it stops working.
Please refer to :ref:`changelog-v2-8-0` for upgrade instructions.

- Fix issue with CORDEX datasets requiring different dataset tags for downloads and fixes (:pull:`2066`) by :user:`ljoakim`

- Due to the different facets for CORDEX datasets, there was an inconsistency in the fixing mechanism.
This change requires changes to existing recipes that use CORDEX datasets. Please refer to the pull request for detailed update instructions.

- For the following changes, no user change is necessary

- Remove deprecated way of calling :func:`~esmvalcore.cmor.table.read_cmor_tables` (:pull:`2201`) by :user:`bouweandela`

- Remove deprecated callback argument from preprocessor ``load`` function (:pull:`2207`) by :user:`bouweandela`

- Remove deprecated preprocessor function `cleanup` (:pull:`2215`) by :user:`bouweandela`

Deprecations
~~~~~~~~~~~~

- Clearly separate fixes and CMOR checks (:pull:`2157`) by :user:`schlunma`
- Added new operators for statistics preprocessor (e.g., ``'percentile'``) and allowed passing additional arguments (:pull:`2191`) by :user:`schlunma`

- This harmonizes the operators for all statistics preprocessors. From this version, the new names can be used; the old arguments will stop working from
version 2.12.0. Please refer to :ref:`stat_preprocs` for a detailed description.

Bug fixes
~~~~~~~~~

- Re-add correctly region-extracted cell measures and ancillary variables after :ref:`extract_region` (:pull:`2166`) by :user:`valeriupredoi`, :user:`schlunma`
- Fix sorting of datasets

- Fix sorting of ensemble members in :func:`~esmvalcore.dataset.datasets_to_recipe` (:pull:`2095`) by :user:`bouweandela`
- Fix a problem with sorting datasets that have a mix of facet types (:pull:`2238`) by :user:`bouweandela`
- Avoid a crash if dataset has supplementary variables (:pull:`2198`) by :user:`bouweandela`

CMOR standard
~~~~~~~~~~~~~

- ERA5 on-the-fly CMORizer: changed sign of variables ``evspsbl`` and ``evspsblpot`` (:pull:`2115`) by :user:`katjaweigel`
- Add ``ch4`` surface custom cmor table entry (:pull:`2168`) by :user:`hb326`
- Add CMIP3 institutes names used at NCI (:pull:`2152`) by :user:`rbeucher`
- Added :func:`~esmvalcore.cmor.fixes.get_time_bounds` and :func:`~esmvalcore.cmor.fixes.get_next_month` to public API (:pull:`2214`) by :user:`schlunma`
- Improve concatenation checks

- Relax concatenation checks for ``--check_level=relax`` and ``--check_level=ignore`` (:pull:`2144`) by :user:`sloosvel`
- Fix ``concatenate`` preprocessor function (:pull:`2240`) by :user:`bouweandela`
- Fix time overlap handling in concatenation (:pull:`2247`) by :user:`zklaus`

Computational performance improvements
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- Make :ref:`threshold_masking` preprocessors lazy (:pull:`2169`) by :user:`joergbenke`

- Restored usage of numpy in `_mask_with_shp` (:pull:`2209`) by :user:`joergbenke`
- Do not realize lazy coordinate bounds in CMOR check (:pull:`2146`) by :user:`sloosvel`
- Rechunk between preprocessor steps (:pull:`2205`) by :user:`bouweandela`
- Reduce the size of the dask graph created by the ``anomalies`` preprocessor function (:pull:`2200`) by :user:`bouweandela`

Documentation
~~~~~~~~~~~~~

- Add reference to release v2.9.0 in the changelog (:pull:`2130`) by :user:`remi-kazeroni`
- Add merge instructions to release instructions (:pull:`2131`) by :user:`zklaus`
- Update `mamba` before building environment during Readthedocs build (:pull:`2149`) by :user:`valeriupredoi`
- Ensure compatible zstandard and zstd versions for .conda support (:pull:`2204`) by :user:`zklaus`
- Remove outdated documentation (:pull:`2210`) by :user:`bouweandela`
- Remove meercode badge from README because their API is broken (:pull:`2224`) by :user:`valeriupredoi`
- Correct usage help text of version command (:pull:`2232`) by :user:`jfrost-mo`
- Add ``navigation_with_keys: False`` to ``html_theme_options`` in Readthedocs ``conf.py`` (:pull:`2245`) by :user:`valeriupredoi`
- Replace squarey badge with roundy shield for Anaconda sticker in README (:pull:`2233`, :pull:`2260`) by :user:`valeriupredoi`

Fixes for datasets
~~~~~~~~~~~~~~~~~~

- Updated doc about fixes and added type hints to fix functions (:pull:`2160`) by :user:`schlunma`

Installation
~~~~~~~~~~~~

- Clean-up how pins are written in conda environment file (:pull:`2125`) by :user:`valeriupredoi`
- Use importlib.metadata instead of deprecated pkg_resources (:pull:`2096`) by :user:`bouweandela`
- Pin shapely to >=2.0 (:pull:`2075`) by :user:`valeriupredoi`
- Pin Python to <3.12 in conda environment (:pull:`2272`) by :user:`bouweandela`

Preprocessor
~~~~~~~~~~~~

- Improve preprocessor output sorting code (:pull:`2111`) by :user:`bouweandela`
- Preprocess datasets in the same order as they are listed in the recipe (:pull:`2103`) by :user:`bouweandela`

Automatic testing
~~~~~~~~~~~~~~~~~

- [Github Actions] Compress all bash shell setters into one default option per workflow (:pull:`2126`) by :user:`valeriupredoi`
- [Github Actions] Fix Monitor Tests Github Action (:pull:`2135`) by :user:`valeriupredoi`
- [condalock] update conda lock file (:pull:`2141`) by :user:`valeriupredoi`
- [Condalock] make sure mamba/conda are at latest version by forcing a pinned mamba install (:pull:`2136`) by :user:`valeriupredoi`
- Update code coverage orbs (:pull:`2206`) by :user:`bouweandela`
- Revisit the comment-triggered Github Actions test (:pull:`2243`) by :user:`valeriupredoi`
- Remove workflow that runs Github Actions tests from PR comment (:pull:`2244`) by :user:`valeriupredoi`

Improvements
~~~~~~~~~~~~

- Merge v2.9.x into main (:pull:`2128`) by :user:`schlunma`
- Fix typo in citation file (:pull:`2182`) by :user:`bouweandela`
- Cleaned and extended function that extracts datetimes from paths (:pull:`2181`) by :user:`schlunma`
- Add file encoding (and some read modes) at open file step (:pull:`2219`) by :user:`valeriupredoi`
- Check type of argument passed to :func:`~esmvalcore.cmor.table.read_cmor_tables` (:pull:`2217`) by :user:`valeriupredoi`
- Dynamic HTML output for monitoring (:pull:`2062`) by :user:`bsolino`
- Use PyPI's trusted publishers authentication (:pull:`2269`) by :user:`valeriupredoi`

.. _changelog-v2-9-0:

------

Highlights
~~~~~~~~~~

It is now possible to use the
`Dask distributed scheduler <https://docs.dask.org/en/latest/deploying.html>`__,
which can
`significantly reduce the run-time of recipes <https://github.com/ESMValGroup/ESMValCore/pull/2049#pullrequestreview-1446279391>`__.
Configuration examples and advice are available in
:ref:`our documentation <config-dask>`.
More work on improving the computational performance is planned, so please share
your experiences, good and bad, with this new feature in :discussion:`1763`.

This release includes

Backwards incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- Remove deprecated configuration options (:pull:`2056`) by :user:`bouweandela`

- The module ``esmvalcore.experimental.config`` has been removed.
To upgrade, import the module from :mod:`esmvalcore.config` instead.

- The module ``esmvalcore._config`` has been removed.
To upgrade, use :mod:`esmvalcore.config` instead.

- The methods ``esmvalcore.config.Session.to_config_user`` and ``esmvalcore.config.Session.from_config_user`` have been removed.
To upgrade, use :obj:`esmvalcore.config.Session` to access the configuration values directly.

Bug fixes
~~~~~~~~~

- Respect ``ignore_warnings`` settings from the :ref:`project configuration <filterwarnings_config-developer>` in :func:`esmvalcore.dataset.Dataset.load` (:pull:`2046`) by :user:`schlunma`
- Fixed usage of custom location for :ref:`custom CMOR tables <custom_cmor_tables>` (:pull:`2052`) by :user:`schlunma`
- Fix issue with writing index.html when :ref:`running a recipe <running>` with ``--resume-from`` (:pull:`2055`) by :user:`bouweandela`
- Fixed bug in ICON CMORizer that lead to shifted time coordinates (:pull:`2038`) by :user:`schlunma`
- Include ``-`` in allowed characters for bibtex references (:pull:`2097`) by :user:`alistairsellar`
- Do not raise an exception if the requested version of a file is not available for all matching files on ESGF (:pull:`2105`) by :user:`bouweandela`

Computational performance improvements
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- Add support for :ref:`configuring Dask distributed <config-dask>` (:pull:`2049`, :pull:`2122`) by :user:`bouweandela`
- Make :func:`esmvalcore.preprocessor.extract_levels` lazy (:pull:`1761`) by :user:`bouweandela`
- Lazy implementation of :func:`esmvalcore.preprocessor.multi_model_statistics` and :func:`esmvalcore.preprocessor.ensemble_statistics` (:pull:`968` and :pull:`2087`) by :user:`Peter9192`
- Avoid realizing data in preprocessor function :func:`esmvalcore.preprocessor.concatenate` when cubes overlap (:pull:`2109`) by :user:`bouweandela`

Documentation
~~~~~~~~~~~~~

- Remove unneeded sphinxcontrib extension (:pull:`2047`) by :user:`valeriupredoi`
- Show ESMValTool logo on `PyPI webpage <https://pypi.org/project/ESMValCore/>`__ (:pull:`2065`) by :user:`valeriupredoi`
- Fix gitter badge in README (:pull:`2118`) by :user:`remi-kazeroni`
- Add changelog for v2.9.0 (:pull:`2088` and :pull:`2123`) by :user:`bouweandela`

Fixes for datasets
~~~~~~~~~~~~~~~~~~

- Pass the :obj:`esmvalcore.config.Session` to fixes (:pull:`1988`) by :user:`schlunma`
- ICON: Allowed specifying vertical grid information in recipe (:pull:`2067`) by :user:`schlunma`
- Allow specifying ``raw_units`` for CESM2, EMAC, and ICON CMORizers (:pull:`2043`) by :user:`schlunma`
- ICON: allow specifying horizontal grid file in recipe/extra facets (:pull:`2078`) by :user:`schlunma`
- Fix tas/tos CMIP6: FIO, KACE, MIROC, IITM (:pull:`2061`) by :user:`pepcos`
- Add fix for EC-Earth3-Veg tos calendar (:pull:`2100`) by :user:`bouweandela`
- Correct GISS-E2-1-G ``tos`` units (:pull:`2099`) by :user:`bouweandela`

Installation
~~~~~~~~~~~~

- Drop support for Python 3.8 (:pull:`2053`) by :user:`bouweandela`
- Add python 3.11 to Github Actions package (conda and PyPI) installation tests (:pull:`2083`) by :user:`valeriupredoi`
- Remove ``with_mypy`` or ``with-mypy`` optional tool for prospector (:pull:`2108`) by :user:`valeriupredoi`

Preprocessor
~~~~~~~~~~~~

- Added ``period='hourly'`` for :func:`esmvalcore.preprocessor.climate_statistics` and :func:`esmvalcore.preprocessor.anomalies` (:pull:`2068`) by :user:`schlunma`
- Support IPCC AR6 regions in :func:`esmvalcore.preprocessor.extract_shape` (:pull:`2008`) by :user:`schlunma`


.. _changelog-v2-8-1:

------

Highlights
~~~~~~~~~~

This release adds support for Python 3.11 and includes several bugfixes.

This release includes:

Bug fixes
~~~~~~~~~

- Pin numpy !=1.24.3 (:pull:`2011`) by :user:`valeriupredoi`
- Fix a bug in recording provenance for the ``mask_multimodel`` preprocessor (:pull:`1984`) by :user:`schlunma`
- Fix ICON hourly data rounding issues (:pull:`2022`) by :user:`BauerJul`
- Use the default SSL context when using the ``extract_location`` preprocessor (:pull:`2023`) by :user:`ehogan`
- Make time-related CMOR fixes work with time dimensions `time1`, `time2`, `time3` (:pull:`1971`) by :user:`schlunma`
- Always create a cache directory for storing ICON grid files (:pull:`2030`) by :user:`schlunma`
- Fixed altitude <--> pressure level conversion for masked arrays in the ``extract_levels`` preprocessor (:pull:`1999`) by :user:`schlunma`
- Allowed ignoring of scalar time coordinates in the ``multi_model_statistics`` preprocessor (:pull:`1961`) by :user:`schlunma`

Fixes for datasets
~~~~~~~~~~~~~~~~~~

- Add support for hourly ICON data (:pull:`1990`) by :user:`BauerJul`
- Fix areacello in BCC-CSM2-MR (:pull:`1993`) by :user:`remi-kazeroni`

Installation
~~~~~~~~~~~~

- Add support for Python=3.11 (:pull:`1832`) by :user:`valeriupredoi`
- Modernize conda lock file creation workflow with mamba, Mambaforge etc (:pull:`2027`) by :user:`valeriupredoi`
- Pin `libnetcdf!=4.9.1` (:pull:`2072`) by :user:`remi-kazeroni`

Documentation
~~~~~~~~~~~~~
- Add changelog for v2.8.1 (:pull:`2079`) by :user:`bouweandela`

Automatic testing
~~~~~~~~~~~~~~~~~

- Use mocked `geopy.geocoders.Nominatim` to avoid `ReadTimeoutError` (:pull:`2005`) by :user:`schlunma`
- Update pre-commit hooks (:pull:`2020`) by :user:`bouweandela`


.. _changelog-v2-8-0: