Xscen

-------------------
Contributors to this version: Trevor James Smith (:user:`Zeitsperre`), Pascal Bourgault (:user:`aulemahal`), Gabriel Rondeau-Genesse (:user:`RondeauG`), Juliette Lavoie (:user:`juliettelavoie`), Marco Braun (:user:`vindelico`).

New features and enhancements
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* ``xs.reduce_ensemble`` will now call ``xclim.ensembles.create_ensemble`` and ``xclim.ensembles.make_critera`` if required. (:pull:`386`).

Breaking changes
^^^^^^^^^^^^^^^^
* Removed support for the old instances of the `region` argument in ``spatial_mean``, ``extract_dataset``, and ``subset``. (:pull:`367`).
* Removed ``xscen.extract.clisops_subset``. (:pull:`367`).
* ``dtr`` (the function) was renamed to ``dtr_from_minmax`` to avoid confusion with the `dtr` variable. (:pull:`372`).
* The ``xscen.reduce`` module has been abandoned. (:pull:`386`).
* ``build_reduction_data`` has been made redundant by ``xclim.ensembles.make_critera`` and will be removed in a future release.
* ``xscen.reduce.reduce_ensemble`` has been moved to ``xscen.ensembles.reduce_ensemble``, as a module was no longer necessary.

Internal changes
^^^^^^^^^^^^^^^^
* Modified ``xscen.utils.change_unit`` to always adopt the name from the `variables_and_units dictionary` if the physical units are equal but their names are not (ex. degC <-> ˚C) (:pull:`373`).
* Updated the `cookiecutter` template to the latest version. (:pull:`358`):
* Addresses a handful of misconfigurations in the GitHub Workflows.
* Added a few free `grep`-based hooks for finding unwanted artifacts in the code base.
* Updated `ruff` to v0.2.0 and `black` to v24.2.0.
* Added more tests. (:pull:`366`, :pull:`367`, :pull:`372`).
* Refactored ``xs.spatial.subset`` into smaller functions. (:pull:`367`).
* An `encoding` argument was added to ``xs.config.load_config``. (:pull:`370`).
* Various small fixes to the code to address FutureWarnings. (:pull:`380`).
* ``xs.spatial.subset`` will try to guess CF coordinate if it can't find "latitude" or "longitude" in ``ds.cf``. (:pull:`384`).
* ``xs.extract_dataset`` and ``xs.DataCatalog.to_dataset`` will now default to opening datasets with option ``chunks={}``, which tries to respect chunking on disk. (:pull:`398`, :issue:`368`).

Bug fixes
^^^^^^^^^
* Fix ``unstack_dates`` for the new frequency syntax introduced by pandas v2.2. (:pull:`359`).
* ``subset_warming_level`` will not return partial subsets if the warming level is reached at the end of the timeseries. (:issue:`360`, :pull:`359`).
* Loading of training in `adjust` is now done outside of the periods loop. (:pull:`366`).
* Fixed bug for adding the preprocessing attributes inside the `adjust` function. (:pull:`366`).
* Fixed a bug to accept `group = False` in `adjust` function. (:pull:`366`).
* `creep_weights` now correctly handles the case where the grid is small, `n` is large, and `mode=wrap`. (:issue:`367`).
* Fixed a bug in ``tasmin_from_dtr`` and ``tasmax_from_dtr``, when `dtr` units differed from tasmin/max. (:pull:`372`).
* Fixed a bug where the requested chunking would be ignored when saving a dataset (:pull:`379`).
* The missing value check in ``health_checks`` will no longer crasg if a variable has no time dimension. (:pull:`382`).

-------------------
Contributors to this version: Juliette Lavoie (:user:`juliettelavoie`), Trevor James Smith (:user:`Zeitsperre`), Gabriel Rondeau-Genesse (:user:`RondeauG`), Pascal Bourgault (:user:`aulemahal`).

Announcements
^^^^^^^^^^^^^
* `xscen` now has a `security disclosure policy <https://github.com/Ouranosinc/xscen/tree/main?tab=security-ov-file#security-ov-file>`_. (:pull:`353`).
* Various frequency-related changes to match the new `pandas` naming conventions. (:pull:`351`).

Internal changes
^^^^^^^^^^^^^^^^
* Added tests for diagnostics. (:pull:`352`).
* Added a `SECURITY.md` file to the repository and the documentation. (:pull:`353`).
* Added `tox` modifier for testing builds against the `main` development branch of `xclim`. (:pull:`351`, :pull:`355`).
* Added a `requirements_upstream.txt` file to the repository to track the development branches of relevant dependencies. (:pull:`355`).
* Added a dedicated GitHub Workflow to evaluate compatibility with upstream dependencies. (:pull:`355`).

Breaking changes
^^^^^^^^^^^^^^^^
* `xscen` now requires `pandas` >= 2.2 and `xclim` >= 0.48.2. (:pull:`351`).
* Functions that output a dict with keys as xrfreq (such as ``extract_dataset``, ``compute_indicators``) will now return the new nomenclature (e.g. ``"YS-JAN"`` instead of ``"AS-JAN"``). (:pull:`351`).
* Going from `xrfreq` to frequencies or timedeltas will still work, but the opposite (frequency --> xrfreq/timedelta) will now only result in the new `pandas` nomenclature. (:pull:`351`).

-------------------
Contributors to this version: Trevor James Smith (:user:`Zeitsperre`), Pascal Bourgault (:user:`aulemahal`)

New features and enhancements
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* Added a new argument ``indicators_kw`` to ``xs.ensembles.build_partition_data``. (:pull:`315`).
* `xscen` is `Semantic Versioning 2.0.0 <https://semver.org/spec/v2.0.0.html>`_ compliant. (:pull:`319`).
* `xesmf` made an optional dependency, making `xscen` easier to install with `pip`. (:pull:`337`).

Internal changes
^^^^^^^^^^^^^^^^
* Granular permissions and dependency scanning actions have been added to all GitHub CI Workflows. (:pull:`313`).
* Updated the list of dependencies to add missing requirements. (:pull:`314`).
* The `cookiecutter` template has been updated to the latest commit via `cruft`. (:pull:`319`):
* `actions-versions-updater.yml` has been replaced with `Dependabot <https://docs.github.com/en/code-security/dependabot/working-with-dependabot>`_ (it's just better).
* The OpenSSF `scorecard.yml` workflow has been added to the GitHub workflows to evaluate package security.
* Code formatting tools (`black`, `blackdoc`, `isort`) are now hard-pinned. These need to be kept in sync with changes from `pre-commit`. (Dependabot should perform this task automatically.)
* The versioning system has been updated to follow the Semantic Versioning 2.0.0 standard.
* Fixed an issue with `pytest -m "not requires_netcdf"` not working as expected. (:pull:`345`).

-------------------
Contributors to this version: Gabriel Rondeau-Genesse (:user:`RondeauG`), Pascal Bourgault (:user:`aulemahal`), Juliette Lavoie (:user:`juliettelavoie`), Sarah-Claude Bourdeau-Goulet (:user:`sarahclaude`), Trevor James Smith (:user:`Zeitsperre`), Marco Braun (:user:`vindelico`).

Announcements
^^^^^^^^^^^^^
* `xscen` now adheres to PEPs 517/518/621 using the `setuptools` and `setuptools-scm` backend for building and packaging. (:pull:`292`).

New features and enhancements
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* New function ``xscen.indicators.select_inds_for_avail_vars`` to filter the indicators that can be calculated with the variables available in a ``xarray.Dataset``. (:pull:`291`).
* Replaced aggregation function ``climatological_mean()`` with ``climatological_op()`` offering more types of operations to aggregate over climatological periods. (:pull:`290`)
* Added the ability to search for simulations that reach a given warming level. (:pull:`251`).
* ``xs.spatial_mean`` now accepts the ``region="global"`` keyword to perform a global average (:issue:`94`, :pull:`260`).
* ``xs.spatial_mean`` with ``method='xESMF'`` will also automatically segmentize polygons (down to a 1° resolution) to ensure a correct average (:pull:`260`).
* Added documentation for `require_all_on` in `search_data_catalogs`. (:pull:`263`).
* ``xs.save_to_table`` and ``xs.io.to_table`` to transform datasets and arrays to DataFrames, but with support for multi-columns, multi-sheets and localized table of content generation.
* Better ``xs.extract.resample`` : support for weighted resampling operations when starting with frequencies coarser than daily and missing timesteps/values handling. (:issue:`80`, :issue:`93`, :pull:`265`).
* New argument ``attribute_weights`` to ``generate_weights`` to allow for custom weights. (:pull:`252`).
* ``xs.io.round_bits`` to round floating point variable up to a number of bits, allowing for a better compression. This can be combined with the saving step through argument ``"bitround"`` of ``save_to_netcdf`` and ``save_to_zarr``. (:pull:`266`).
* Added annual global tas timeseries for CMIP6's models CMCC-ESM2 (ssp245, ssp370, ssp585), EC-Earth3-CC (ssp245, ssp585), KACE-1-0-G (ssp245, ssp370, ssp585) and TaiESM1 (ssp245, ssp370). Moved global tas database to a netCDF file. (:issue:`268`, :pull:`270`).
* Implemented support for multiple levels and models in ``xs.subset_warming_level``. Better support for `DataArray` and `DataFrame` in ``xs.get_warming_level``. (:pull:`270`).
* Added the ability to directly provide an ensemble dataset to ``xs.ensemble_stats``. (:pull:`299`).
* Added support in ``xs.ensemble_stats`` for the new robustness-related functions available in `xclim`. (:pull:`299`).
* New function ``xs.ensembles.get_partition_input`` (:pull:`289`).

Breaking changes
^^^^^^^^^^^^^^^^
* ``climatological_mean()`` has been replaced with ``climatological_op()`` and will be abandoned in a future version. (:pull:`290`)
* ``experiment_weights`` argument in ``generate_weights`` was renamed to ``balance_experiments``. (:pull:`252`).
* New argument ``attribute_weights`` to ``generate_weights`` to allow for custom weights. (:pull:`252`).
* For a sequence of models, the output of ``xs.get_warming_level`` is now a list. Revert to a dictionary with ``output='selected'`` (:pull:`270`).
* The global average temperature database is now a netCDF, custom databases must follow the same format (:pull:`270`).

Bug fixes
^^^^^^^^^
* Fixed a bug in ``xs.search_data_catalogs`` when searching for fixed fields and specific experiments/members. (:pull:`251`).
* Fixed a bug in the documentation build configuration that prevented stable/latest and tagged documentation builds from resolving on ReadTheDocs. (:pull:`256`).
* Fixed ``get_warming_level`` to avoid incomplete matches. (:pull:`269`).
* `search_data_catalogs` now eliminates anything that matches any entry in `exclusions`. (:issue:`275`, :pull:`280`).
* Fixed a bug in ``xs.scripting.save_and_update`` where ``build_path_kwargs`` was ignored when trying to guess the file format. (:pull:`282`).
* Add a warning to ``xs.extract._dispatch_historical_to_future``. (:issue:`286`, :pull:`287`).
* Modify use_cftime for the calendar conversion in ``to_dataset``. (:issue:`303`, :pull:`289`).

Internal changes
^^^^^^^^^^^^^^^^
* Continued work on adding tests. (:pull:`251`).
* Fixed `pre-commit`'s `pretty-format-json` hook so that it ignores notebooks. (:pull:`254`).
* Fixed the labeler so docs/CI isn't automatically added for contributions by new collaborators. (:pull:`254`).
* Made it so that `tests` are no longer treated as an installable package. (:pull:`248`).
* Renamed the pytest marker from ``requires_docs`` to ``requires_netcdf``. (:pull:`248`).
* Included the documentation in the source distribution, while excluding the NetCDF files. (:pull:`248`).
* Reduced the size of the files in ``/docs/notebooks/samples`` and changed the notebooks and tests accordingly. (:issue:`247`, :pull:`248`).
* Added a new `xscen.testing` module with the `datablock_3d` function previously located in ``/tests/conftest.py``. (:pull:`248`).
* New function `xscen.testing.fake_data` to generate fake data for testing. (:pull:`248`).
* xESMF 0.8 Regridder and SpatialAverager argument ``out_chunks`` is now accepted by ``xs.regrid_dataset`` and ``xs.spatial_mean``. (:pull:`260`).
* Testing, Packaging, and CI adjustments. (:pull:`274`):
* `xscen` builds now install in a `tox` environment with `conda`-provided `ESMF` in GitHub Workflows.
* `tox` now offers a method for installing esmpy from a tag/branch (via ESMF_VERSION environment variable).
* `$ make translate` is now called on ReadTheDocs and within `tox`.
* Linters are now called by order of most common failures first, to speed up the CI.
* `Manifest.in` is much more specific about what is installed.
* Re-adds a dev recipe to the `setup.py`.
* Multiple improvements to the docstrings and type annotations. (:pull:`282`).
* `pip check` in conda builds in GitHub workflows have been temporarily set to always pass. (:pull:`288`).
* The `cookiecutter` template has been updated to the latest commit via `cruft`. (:pull:`292`):
* `setup.py` has been mostly hollowed-out, save for the `babel`-related translation function.
* `pyproject.toml` has been added, with most package configurations migrated into it.
* `HISTORY.rst` has been renamed to `CHANGES.rst`.
* `actions-version-updater.yml` has been added to automate the versioning of the package.
* `pre-commit` hooks have been updated to the latest versions; `check-toml` and `toml-sort` have been added to cleanup the `pyproject.toml` file, and `check-json-schema` has been added to ensure GitHub and ReadTheDocs workflow files are valid.
* `ruff` has been added to the linting tools to replace most `flake8` and `pydocstyle` verifications.
* `tox` builds are more pure Python environment/PyPI-friendly.
* `xscen` now uses `Trusted Publishing` for TestPyPI and PyPI uploads.
* Linting checks now examine the testing folder, function complexity, and alphabetical order of `__all__` lists. (:pull:`292`).
* ``publish_release_notes`` now uses better logic for finding and reformatting the `CHANGES.rst` file. (:pull:`292`).
* ``bump2version`` version-bumping utility was replaced by ``bump-my-version``. (:pull:`292`).
* Documentation build checks no longer fail due to broken external links; Notebooks are now nested and numbered. (:pull:`304`).

-------------------
* Update dependencies by removing ``pygeos``, pinning ``shapely>=2`` and ``intake-esm>=2023.07.07`` as well as other small fixes to the environment files. (:pull:`243`).
* Fix ``xs.aggregate.spatial_mean`` with method ``cos-lat`` when the data is on a rectilinear grid. (:pull:`243`).

Internal changes
^^^^^^^^^^^^^^^^
* Added a workflow that removes obsolete GitHub Workflow caches from merged pull requests. (:pull:`250`).
* Added a workflow to perform automated labeling of pull requests, dependent on the files changed. (:pull:`250`).

-------------------
Contributors to this version: Gabriel Rondeau-Genesse (:user:`RondeauG`), Pascal Bourgault (:user:`aulemahal`), Trevor James Smith (:user:`Zeitsperre`), Juliette Lavoie (:user:`juliettelavoie`), Marco Braun (:user:`vindelico`).

Announcements
^^^^^^^^^^^^^
* Dropped support for Python 3.8, added support for 3.11. (:pull:`199`, :pull:`222`).
* `xscen` is now available on `conda-forge <https://anaconda.org/conda-forge/xscen>`_, and can be installed with ``conda install -c conda-forge xscen``. (:pull:`241`)

New features and enhancements
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* `xscen` now tracks code coverage using `coveralls <https://coveralls.io/>`_. (:pull:`187`).
* New function `get_warming_level` to search within the IPCC CMIP global temperatures CSV without requiring data. (:issue:`208`, :pull:`210`).
* File re-structuration from catalogs with ``xscen.catutils.build_path``. (:pull:`205`, :pull:`237`).
* New scripting functions `save_and_update` and `move_and_delete`. (:pull:`214`).
* Spatial dimensions can be generalized as X/Y when rechunking and will be mapped to rlon/rlat or lon/lat accordingly. (:pull:`221`).
* New argument `var_as_string` for `get_cat_attrs` to return variable names as strings. (:pull:`233`).
* New argument `copy` for `move_and_delete`. (:pull:`233`).
* New argument `restrict_year` for `compute_indicators`. (:pull:`233`).
* Add more comments in the template. (:pull:`233`, :issue:`232`).
* ``generate_weights`` now allows to split weights between experiments, and make them vary along the time/horizon axis. (:issue:`108`, :pull:`231`).
* New independence_level, `institution`, added to ``generate_weights``. (:pull:`231`).
* Updated ``produce_horizon`` so it can accept multiple periods or warming levels. (:pull:`231`, :pull:`240`).
* Add more comments in the template. (:pull:`233`, :pull:`235`, :issue:`232`).
* New function ``diagnostics.health_checks`` that can perform multiple checkups on a dataset. (:pull:`238`).

Breaking changes
^^^^^^^^^^^^^^^^
* Columns ``date_start`` and ``date_end`` now use a ``datetime64[ms]`` dtype. (:pull:`222`).
* The default output of ``date_parser`` is now ``pd.Timestamp`` (``output_dtype='datetime'``). (:pull:`222`).
* ``date_parser(date, end_of_period=True)`` has time "23:59:59", instead of "23:00". (:pull:`222`, :pull:`237`).
* ``driving_institution`` was removed from the "default" xscen columns. (:pull:`222`).
* Folder parsing utilities (``parse_directory``) moved to ``xscen.catutils``. Signature changed : ``globpattern`` removed, ``dirglob`` added, new ``patterns`` specifications. See doc for all changes. (:pull:`205`).
* ``compute_indicators`` now returns all outputs produced by indicators with multiple outputs (such as `rain_season`). (:pull:`228`).
* In ``generate_weights``, independence_level `all` was renamed `model`. (:pull:`231`).
* In response to a bugfix, results for ``generate_weights(independence_level='GCM')`` are significantly altered. (:issue:`230`, :pull:`231`).
* Legacy support for `stats_kwargs` in ``ensemble_stats`` was dropped. (:pull:`231`).
* `period` in ``produce_horizon`` has been deprecated and replaced with `periods`. (:pull:`231`).
* Some automated `to_level` were updated to reflect more recent changes. (:pull:`231`).
* Removed ``diagnostics.fix_unphysical_values``. (:pull:`238`).

Bug fixes
^^^^^^^^^
* Fix bug in ``unstack_dates`` with seasonal climatological mean. (:issue:`202`, :pull:`202`).
* Added NotImplemented errors when trying to call `climatological_mean` and `compute_deltas` with daily data. (:pull:`187`).
* Minor documentation fixes. (:issue:`223`, :pull:`225`).
* Fixed a bug in ``unstack_dates`` where it failed for anything other than seasons. (:pull:`228`).
* ``cleanup`` with `common_attrs_only` now works even when no `cat` attribute is present in the datasets. (:pull:`231`).

Internal changes
^^^^^^^^^^^^^^^^
* Removed the pin on xarray's version. (:issue:`175`, :pull:`199`).
* Folder parsing utilities now in pure python, platform independent. New dependency ``parse``. (:pull:`205`).
* Updated ReadTheDocs configuration to prevent ``--eager`` installation of xscen (:pull:`209`).
* Implemented a template to be used for unit tests. (:pull:`187`).
* Updated GitHub Actions to remove deprecation warnings. (:pull:`187`).
* Updated the cookiecutter used to generate boilerplate documentation and code via `cruft`. (:pull:`212`).
* A few changes to `subset_warming_level` so it doesn't need `driving_institution`. (:pull:`215`).
* Added more tests. (:pull:`228`).
* In ``compute_indicators``, the logic to manage indicators returning multiple outputs was simplified. (:pull:`228`).