Esmvalcore

------

Highlights
~~~~~~~~~~

ESMValCore is now using the recently released `Iris 3 <https://scitools-iris.readthedocs.io/en/latest/whatsnew/3.0.html>`__.
We acknowledge that this change may impact your work, as Iris 3 introduces
several changes that are not backward-compatible, but we think that moving forward is the best
decision for the tool in the long term.

This release is also the first one including support for downloading CMIP6 data
using Synda and we have also started supporting Python 3.9. Give it a try!


This release includes

Bug fixes
~~~~~~~~~

- Fix path settings for DKRZ/Mistral (:pull:`852`) by :user:`bouweandela`
- Change logic for calling the diagnostic script to avoid problems with scripts where the executable bit is accidentally set (:pull:`877`) by :user:`bouweandela`
- Fix overwriting in generic level check (:pull:`886`) by :user:`sloosvel`
- Add double quotes to script args in rerun screen message when using vprof profiling (:pull:`897`) by :user:`valeriupredoi`
- Simplify time handling in multi-model statistics preprocessor (:pull:`685`) by :user:`Peter9192`
- Fix links to Iris documentation (:pull:`966`) by :user:`jvegreg`
- Bugfix: Fix units for MSWEP data (:pull:`986`) by :user:`stefsmeets`

Deprecations
~~~~~~~~~~~~

- Deprecate defining write_plots and write_netcdf in config-user file (:pull:`808`) by :user:`bouweandela`

Documentation
~~~~~~~~~~~~~

- Fix numbering of steps in release instructions (:pull:`838`) by :user:`bouweandela`
- Add labels to changelogs of individual versions for easy reference (:pull:`899`) by :user:`zklaus`
- Make CircleCI badge specific to main branch (:pull:`902`) by :user:`bouweandela`
- Fix docker build badge url (:pull:`906`) by :user:`stefsmeets`
- Update github PR template (:pull:`909`) by :user:`stefsmeets`
- Refer to ESMValTool GitHub discussions page in the error message (:pull:`900`) by :user:`bouweandela`
- Support automatically closing issues (:pull:`922`) by :user:`bouweandela`
- Fix checkboxes in PR template (:pull:`931`) by :user:`stefsmeets`
- Change in config-user defaults and documentation with new location for esmeval OBS data on JASMIN (:pull:`958`) by :user:`valeriupredoi`
- Update Core Team info (:pull:`942`) by :user:`axel-lauer`
- Update iris documentation URL for sphinx (:pull:`964`) by :user:`bouweandela`
- Set version to 2.2.0 (:pull:`977`) by :user:`jvegreg`
- Add first draft of v2.2.0 changelog (:pull:`983`) by :user:`jvegreg`
- Add checkbox in PR template to assign labels (:pull:`985`) by :user:`jvegreg`
- Update install.rst (:pull:`848`) by :user:`bascrezee`
- Change the order of the publication steps (:pull:`984`) by :user:`jvegreg`
- Add instructions how to use esmvaltool from HPC central installations (:pull:`841`) by :user:`valeriupredoi`

Fixes for datasets
~~~~~~~~~~~~~~~~~~

- Fixing unit for derived variable rsnstcsnorm to prevent overcorrection2 (:pull:`846`) by :user:`katjaweigel`
- Cmip6 fix awi cm 1 1 mr (:pull:`822`) by :user:`mwjury`
- Cmip6 fix ec earth3 veg (:pull:`836`) by :user:`mwjury`
- Changed latitude longitude fix from Tas to AllVars. (:pull:`916`) by :user:`katjaweigel`
- Fix for precipitation (pr) to use ERA5-Land cmorizer (:pull:`879`) by :user:`katjaweigel`
- Cmip6 fix ec earth3 (:pull:`837`) by :user:`mwjury`
- Cmip6_fix_fgoals_f3_l_Amon_time_bnds (:pull:`831`) by :user:`mwjury`
- Fix for FGOALS-f3-L sftlf (:pull:`667`) by :user:`mwjury`
- Improve ACCESS-CM2 and ACCESS-ESM1-5 fixes and add CIESM and CESM2-WACCM-FV2 fixes for cl, clw and cli (:pull:`635`) by :user:`axel-lauer`
- Add fixes for cl, cli, clw and tas for several CMIP6 models (:pull:`955`) by :user:`schlunma`
- Dataset fixes for MSWEP (:pull:`969`) by :user:`stefsmeets`
- Dataset fixes for: ACCESS-ESM1-5, CanESM5, CanESM5 for carbon cycle (:pull:`947`) by :user:`bettina-gier`
- Fixes for KIOST-ESM (CMIP6) (:pull:`904`) by :user:`remi-kazeroni`
- Fixes for AWI-ESM-1-1-LR (CMIP6, piControl) (:pull:`911`) by :user:`remi-kazeroni`

CMOR standard
~~~~~~~~~~~~~

- CMOR check generic level coordinates in CMIP6 (:pull:`598`) by :user:`sloosvel`
- Update CMIP6 tables to 6.9.33 (:pull:`919`) by :user:`jvegreg`
- Adding custom variables for tas uncertainty (:pull:`924`) by :user:`LisaBock`
- Remove monotonicity coordinate check for unstructured grids (:pull:`965`) by :user:`jvegreg`

Preprocessor
~~~~~~~~~~~~

- Added clip_start_end_year preprocessor (:pull:`796`) by :user:`schlunma`
- Add support for downloading CMIP6 data with Synda (:pull:`699`) by :user:`bouweandela`
- Add multimodel tests using real data (:pull:`856`) by :user:`stefsmeets`
- Add plev/altitude conversion to extract_levels (:pull:`892`) by :user:`axel-lauer`
- Add possibility of custom season extraction. (:pull:`247`) by :user:`mwjury`
- Adding the ability to derive xch4 (:pull:`783`) by :user:`hb326`
- Add preprocessor function to resample time and compute x-hourly statistics (:pull:`696`) by :user:`jvegreg`
- Fix duplication in preprocessors DEFAULT_ORDER introduced in 696 (:pull:`973`) by :user:`jvegreg`
- Use consistent precision in multi-model statistics calculation and update reference data for tests (:pull:`941`) by :user:`Peter9192`
- Refactor multi-model statistics code to facilitate ensemble stats and lazy evaluation (:pull:`949`) by :user:`Peter9192`
- Add option to exclude input cubes in output of multimodel statistics to solve an issue introduced by 949 (:pull:`978`) by :user:`Peter9192`


Automatic testing
~~~~~~~~~~~~~~~~~

- Pin cftime>=1.3.0 to have newer string formatting in and fix two tests (:pull:`878`) by :user:`valeriupredoi`
- Switched miniconda conda setup hooks for Github Actions workflows (:pull:`873`) by :user:`valeriupredoi`
- Add test for latest version resolver (:pull:`874`) by :user:`stefsmeets`
- Update codacy coverage reporter to fix coverage (:pull:`905`) by :user:`nielsdrost`
- Avoid hardcoded year in tests and add improvement to plev test case (:pull:`921`) by :user:`bouweandela`
- Pin scipy to less than 1.6.0 until :issue:`927` gets resolved (:pull:`928`) by :user:`valeriupredoi`
- Github Actions: change time when conda install test runs (:pull:`930`) by :user:`valeriupredoi`
- Remove redundant test line from test_utils.py (:pull:`935`) by :user:`valeriupredoi`
- Removed netCDF4 package from integration tests of fixes (:pull:`938`) by :user:`schlunma`
- Use new conda environment for installing ESMValCore in Docker containers (:pull:`951`) by :user:`bouweandela`

Notebook API (experimental)
~~~~~~~~~~~~~~~~~~~~~~~~~~~

- Implement importable config object in experimental API submodule (:pull:`868`) by :user:`stefsmeets`
- Add loading and running recipes to the notebook API (:pull:`907`) by :user:`stefsmeets`
- Add displaying and loading of recipe output to the notebook API (:pull:`957`) by :user:`stefsmeets`
- Add functionality to run single diagnostic task to notebook API (:pull:`962`) by :user:`stefsmeets`

Improvements
~~~~~~~~~~~~

- Create CODEOWNERS file (:pull:`809`) by :user:`jvegreg`
- Remove code needed for Python <3.6 (:pull:`844`) by :user:`bouweandela`
- Add requests as a dependency (:pull:`850`) by :user:`bouweandela`
- Pin Python to less than 3.9 (:pull:`870`) by :user:`valeriupredoi`
- Remove numba dependency (:pull:`880`) by :user:`schlunma`
- Add Listing and finding recipes to the experimental notebook API (:pull:`901`) by :user:`stefsmeets`
- Skip variables that don't have dataset or additional_dataset keys (:pull:`860`) by :user:`valeriupredoi`
- Refactor logging configuration (:pull:`933`) by :user:`stefsmeets`
- Xco2 derivation (:pull:`913`) by :user:`bettina-gier`
- Working environment for Python 3.9 (pin to !=3.9.0) (:pull:`885`) by :user:`valeriupredoi`
- Print source file when using config get_config_user command (:pull:`960`) by :user:`valeriupredoi`
- Switch to Iris 3 (:pull:`819`) by :user:`stefsmeets`
- Refactor tasks (:pull:`959`) by :user:`stefsmeets`
- Restore task summary in debug log after 959 (:pull:`981`) by :user:`bouweandela`
- Pin pre-commit hooks (:pull:`974`) by :user:`stefsmeets`
- Improve error messages when data is missing (:pull:`917`) by :user:`jvegreg`
- Set remove_preproc_dir to false in default config-user (:pull:`979`) by :user:`valeriupredoi`
- Move fiona to be installed from conda forge (:pull:`987`) by :user:`valeriupredoi`
- Re-added fiona in setup.py (:pull:`990`) by :user:`valeriupredoi`

.. _changelog-v2-1-0:

------

This release includes

Bug fixes
~~~~~~~~~

- Set unit=1 if anomalies are standardized (:pull:`727`) by :user:`bascrezee`
- Fix crash for FGOALS-g2 variables without longitude coordinate (:pull:`729`) by :user:`bouweandela`
- Improve variable alias management (:pull:`595`) by :user:`jvegreg`
- Fix area_statistics fx files loading (:pull:`798`) by :user:`jvegreg`
- Fix units after derivation (:pull:`754`) by :user:`schlunma`

Documentation
~~~~~~~~~~~~~

- Update v2.0.0 release notes with final additions (:pull:`722`) by :user:`bouweandela`
- Update package description in setup.py (:pull:`725`) by :user:`mattiarighi`
- Add installation instructions for pip installation (:pull:`735`) by :user:`bouweandela`
- Improve config-user documentation (:pull:`740`) by :user:`bouweandela`
- Update the zenodo file with contributors (:pull:`807`) by :user:`valeriupredoi`
- Improve command line run documentation (:pull:`721`) by :user:`jvegreg`
- Update the zenodo file with contributors (continued) (:pull:`810`) by :user:`valeriupredoi`

Improvements
~~~~~~~~~~~~

- Reduce size of docker image (:pull:`723`) by :user:`jvegreg`
- Add 'test' extra to installation, used by docker development tag (:pull:`733`) by :user:`bouweandela`
- Correct dockerhub link (:pull:`736`) by :user:`bouweandela`
- Create action-install-from-pypi.yml (:pull:`734`) by :user:`valeriupredoi`
- Add pre-commit for linting/formatting (:pull:`766`) by :user:`stefsmeets`
- Run tests in parallel and when building conda package (:pull:`745`) by :user:`bouweandela`
- Readable exclude pattern for pre-commit (:pull:`770`) by :user:`stefsmeets`
- Github Actions Tests (:pull:`732`) by :user:`valeriupredoi`
- Remove isort setup to fix formatting conflict with yapf (:pull:`778`) by :user:`stefsmeets`
- Fix yapf-isort import formatting conflict (Fixes 777) (:pull:`784`) by :user:`stefsmeets`
- Sorted output for `esmvaltool recipes list` (:pull:`790`) by :user:`stefsmeets`
- Replace vmprof with vprof (:pull:`780`) by :user:`valeriupredoi`
- Update CMIP6 tables to 6.9.32 (:pull:`706`) by :user:`jvegreg`
- Default config-user path now set in config-user read function (:pull:`791`) by :user:`jvegreg`
- Add custom variable lweGrace (:pull:`692`) by :user:`bascrezee`
- Create Github Actions workflow to build and deploy on Test PyPi and PyPi (:pull:`820`) by :user:`valeriupredoi`
- Build and publish the esmvalcore package to conda via Github Actions workflow (:pull:`825`) by :user:`valeriupredoi`

Fixes for datasets
~~~~~~~~~~~~~~~~~~

- Fix cmip6 models (:pull:`629`) by :user:`npgillett`
- Fix siconca variable in EC-Earth3 and EC-Earth3-Veg models in amip simulation (:pull:`702`) by :user:`egalytska`

Preprocessor
~~~~~~~~~~~~

- Move cmor_check_data to early in preprocessing chain (:pull:`743`) by :user:`bouweandela`
- Add RMS iris analysis operator to statistics preprocessor functions (:pull:`747`) by :user:`pcosbsc`
- Add surface chlorophyll concentration as a derived variable (:pull:`720`) by :user:`sloosvel`
- Use dask to reduce memory consumption of extract_levels for masked data (:pull:`776`) by :user:`valeriupredoi`

.. _changelog-v2-0-0:

------

This release includes

Bug fixes
~~~~~~~~~

- Fixed derivation of co2s (:pull:`594`) by :user:`schlunma`
- Padding while cropping needs to stay within sane bounds for shapefiles that span the whole Earth (:pull:`626`) by :user:`valeriupredoi`
- Fix concatenation of a single cube (:pull:`655`) by :user:`bouweandela`
- Fix mask fx dict handling not to fail if empty list in values (:pull:`661`) by :user:`valeriupredoi`
- Preserve metadata during anomalies computation when using iris cubes difference (:pull:`652`) by :user:`valeriupredoi`
- Avoid crashing when there is directory 'esmvaltool' in the current working directory (:pull:`672`) by :user:`valeriupredoi`
- Solve bug in ACCESS1 dataset fix for calendar. (:pull:`671`) by :user:`Peter9192`
- Fix the syntax for adding multiple ensemble members from the same dataset (:pull:`678`) by :user:`SarahAlidoost`
- Fix bug that made preprocessor with fx files fail in rare cases (:pull:`670`) by :user:`schlunma`
- Add support for string coordinates (:pull:`657`) by :user:`jvegreg`
- Fixed the shape extraction to account for wraparound shapefile coords (:pull:`319`) by :user:`valeriupredoi`
- Fixed bug in time weights calculation (:pull:`695`) by :user:`schlunma`
- Fix diagnostic filter (:pull:`713`) by :user:`jvegreg`

Documentation
~~~~~~~~~~~~~

- Add pandas as a requirement for building the documentation (:pull:`607`) by :user:`bouweandela`
- Document default order in which preprocessor functions are applied (:pull:`633`) by :user:`bouweandela`
- Add pointers about data loading and CF standards to documentation (:pull:`571`) by :user:`valeriupredoi`
- Config file populated with site-specific data paths examples (:pull:`619`) by :user:`valeriupredoi`
- Update Codacy badges (:pull:`643`) by :user:`bouweandela`
- Update copyright info on readthedocs (:pull:`668`) by :user:`bouweandela`
- Updated references to documentation (now docs.esmvaltool.org) (:pull:`675`) by :user:`axel-lauer`
- Add all European grants to Zenodo (:pull:`680`) by :user:`bouweandela`
- Update Sphinx to v3 or later (:pull:`683`) by :user:`bouweandela`
- Increase version to 2.0.0 and add release notes (:pull:`691`) by :user:`bouweandela`
- Update setup.py and README.md for use on PyPI (:pull:`693`) by :user:`bouweandela`
- Suggested Documentation changes (:pull:`690`) by :user:`ssmithClimate`

Improvements
~~~~~~~~~~~~

- Reduce the size of conda package (:pull:`606`) by :user:`bouweandela`
- Add a few unit tests for DiagnosticTask (:pull:`613`) by :user:`bouweandela`
- Make ncl or R tests not fail if package not installed (:pull:`610`) by :user:`valeriupredoi`
- Pin flake8<3.8.0 (:pull:`623`) by :user:`valeriupredoi`
- Log warnings for likely errors in provenance record (:pull:`592`) by :user:`bouweandela`
- Unpin flake8 (:pull:`646`) by :user:`bouweandela`
- More flexible native6 default DRS (:pull:`645`) by :user:`bouweandela`
- Try to use the same python for running diagnostics as for esmvaltool (:pull:`656`) by :user:`bouweandela`
- Fix test for lower python version and add note on lxml (:pull:`659`) by :user:`valeriupredoi`
- Added 1m deep average soil moisture variable (:pull:`664`) by :user:`bascrezee`
- Update docker recipe (:pull:`603`) by :user:`jvegreg`
- Improve command line interface (:pull:`605`) by :user:`jvegreg`
- Remove utils directory (:pull:`697`) by :user:`bouweandela`
- Avoid pytest version that crashes (:pull:`707`) by :user:`bouweandela`
- Options arg in read_config_user_file now optional (:pull:`716`) by :user:`jvegreg`
- Produce a readable warning if ancestors are a string instead of a list. (:pull:`711`) by :user:`katjaweigel`
- Pin Yamale to v2 (:pull:`718`) by :user:`bouweandela`
- Expanded cmor public API (:pull:`714`) by :user:`schlunma`

Fixes for datasets
~~~~~~~~~~~~~~~~~~

- Added various fixes for hybrid height coordinates (:pull:`562`) by :user:`schlunma`
- Extended fix for cl-like variables of CESM2 models (:pull:`604`) by :user:`schlunma`
- Added fix to convert "geopotential" to "geopotential height" for ERA5 (:pull:`640`) by :user:`egalytska`
- Do not fix longitude values if they are too far from valid range (:pull:`636`) by :user:`jvegreg`

Preprocessor
~~~~~~~~~~~~

- Implemented concatenation of cubes with derived coordinates (:pull:`546`) by :user:`schlunma`
- Fix derived variable ctotal calculation depending on project and standard name (:pull:`620`) by :user:`valeriupredoi`
- State of the art FX variables handling without preprocessing (:pull:`557`) by :user:`valeriupredoi`
- Add max, min and std operators to multimodel (:pull:`602`) by :user:`jvegreg`
- Added preprocessor to extract amplitude of cycles (:pull:`597`) by :user:`schlunma`
- Overhaul concatenation and allow for correct concatenation of multiple overlapping datasets (:pull:`615`) by :user:`valeriupredoi`
- Change volume stats to handle and output masked array result (:pull:`618`) by :user:`valeriupredoi`
- Area_weights for cordex in area_statistics (:pull:`631`) by :user:`mwjury`
- Accept cubes as input in multimodel (:pull:`637`) by :user:`sloosvel`
- Make multimodel work correctly with yearly data (:pull:`677`) by :user:`valeriupredoi`
- Optimize time weights in time preprocessor for climate statistics (:pull:`684`) by :user:`valeriupredoi`
- Add percentiles to multi-model stats (:pull:`679`) by :user:`Peter9192`

.. _changelog-v2-0-0b9:

--------

This release includes

Bug fixes
~~~~~~~~~

- Cast dtype float32 to output from zonal and meridional area preprocessors (:pull:`581`) by :user:`valeriupredoi`

Improvements
~~~~~~~~~~~~

- Unpin on Python<3.8 for conda package (run) (:pull:`570`) by :user:`valeriupredoi`
- Update pytest installation marker (:pull:`572`) by :user:`bouweandela`
- Remove vmrh2o (:pull:`573`) by :user:`mattiarighi`
- Restructure documentation (:pull:`575`) by :user:`bouweandela`
- Fix mask in land variables for CCSM4 (:pull:`579`) by :user:`zklaus`
- Fix derive scripts wrt required method (:pull:`585`) by :user:`zklaus`
- Check coordinates do not have repeated standard names (:pull:`558`) by :user:`jvegreg`
- Added derivation script for co2s (:pull:`587`) by :user:`schlunma`
- Adapted custom co2s table to match CMIP6 version (:pull:`588`) by :user:`schlunma`
- Increase version to v2.0.0b9 (:pull:`593`) by :user:`bouweandela`
- Add a method to save citation information (:pull:`402`) by :user:`SarahAlidoost`

For older releases, see the release notes on https://github.com/ESMValGroup/ESMValCore/releases.