Diive

New features

Ridgeline plot

`diive` can now create ridgeline plots.


The ridgeline plot visualizes the distribution of a quantitative variable by stacking overlapping density plots,
creating a "ridged" landscape. I think this is quite pleasing to look at. With the implementation in `diive`, it
facilitates the comparison of distributional shapes and changes of time series data across weeks, months and years.
Ridgeline plots are quite space-efficient and hopefully visually intuitive for revealing patterns and trends in data.

This is also the first function that uses a simplified API. After importing `diive`, the plot can simply be accessed via
`.ridgeline()`. This is a shortcut to access the class `RidgeLinePlot` that is otherwise deeply buried in the code
here: `diive.core.plotting.ridgeline.RidgeLinePlot`. In the future, other classes and functions will also be
accessible via similar shortforms.

Basic example:


import diive as dv
rp = dv.ridgeline(series=series) Initialize instance, series is a pandas Series
rp.plot() Generate basic plot


See the notebook here for more examples:
`notebooks/Plotting/ridgeline.ipynb`

Additions

- Additions to the flux processing chain:
- Added two methods to get details about training and testing when using machine-learning models in the flux
processing chain: `.report_traintest_model_scores()` and `.report_traintest_details()`
- Added parameter `setflag_timeperiod` to set the flag for the SSITC to another value during certain time periods,
for example when a time period needs stricter filtering (e.g. due to issues with the sonic anemometer). In this
case the parameter can be used to set all values where flag=1 (medium quality data) to flag=2 (bad data).
- Example from docstring:
Set flag 1 to value 2 between '2022-05-01' and '2023-09-30', and between
'2024-04-02' and '2024-04-19' (dates inclusive):
setflag_timeperiod={2: [ [1, '2022-05-01', '2023-09-30'], [1, '2024-04-02', '2024-04-19'] ]}
(`diive.pkgs.qaqc.eddyproflags.flag_ssitc_eddypro_test`)
- Added params to export some gap-filling results (e.g. model scores) to csv files (e.g.,
`.report_gapfilling_model_scores(outpath=...)`)
- (`diive.pkgs.fluxprocessingchain.fluxprocessingchain.FluxProcessingChain`)
- Added check if time series has a name when plotting heatmaps. If time series does not have a name, it is automatically
assigned the name `data`. Implemented in class `HeatmapBase` that is used by all heatmap plotters. (
`diive.core.plotting.heatmap_base.HeatmapBase`)
- Added new filetype for 60MIN EddyPro output (`diive/configs/filetypes/EDDYPRO-FLUXNET-CSV-60MIN.yml`)

Notebooks

- Added notebook for ridgeline plot (`notebooks/Plotting/ridgeline.ipynb`)

Bugfixes

- Fixed bug where the flux processing chain would crash when a variable with the same name as one of the automatically
generated variables was already present in the input data. For example, the potential radiation `SW_IN_POT` is
generated when the flux processing chain starts and then it is added also to the input data. If the input data already
has a variable with the same name, the processing chain would crash. Now, the automatically generated `SW_IN_POT` is
given priority, which means the variable in the input data is overwritten. (
`diive.pkgs.fluxprocessingchain.fluxprocessingchain.FluxProcessingChain`)

Environment

- Updated packages

Unittests

- 53/53 unittests ran successfully

New features

- Added class for formatting meteo data for upload to FLUXNET (`diive.pkgs.formats.meteo.FormatMeteoForFluxnetUpload`)

Notebooks

- Added new notebook `notebooks/Formats/FormatMeteoForFluxnetUpload.ipynb`

New features

- Added class to format meteo data as input file for EddyPro flux calcs (
`diive.pkgs.formats.meteo.FormatMeteoForEddyProFluxProcessing`)

Changes

- Updated formatting for FLUXNET upload (`diive.pkgs.formats.fluxnet.FormatEddyProFluxnetFileForUpload`)
- `HeatmapYearMonth` plot now shows every year on y-axis (`diive.core.plotting.heatmap_datetime.HeatmapYearMonth`)
- Improved check for excluded columns when creating lagged variants (
`diive.pkgs.createvar.laggedvariants.lagged_variants`)
- More text output when reducting features (`diive.core.ml.common.MlRegressorGapFillingBase.reduce_features`)
- Fixed colorwheel running out of colors when plotting feature ranks (
`diive.pkgs.gapfilling.longterm.LongTermGapFillingBase.showplot_feature_ranks_per_year`)
- Less text output when filling storage term (
`diive.pkgs.fluxprocessingchain.level31_storagecorrection.FluxStorageCorrectionSinglePointEddyPro._gapfill_storage_term`)
- Smaller fixes

Notebooks

- Added new notebook `notebooks/Formats/FormatMeteoForEddyProFluxProcessing.ipynb`
- Updated notebook `notebooks/Formats/notebooks/Formats/FormatEddyProFluxnetFileForUpload.ipynb`

Updates to MDS gap-filling

The community-standard MDS gap-filling method for eddy covariance ecosystem fluxes (e.g., CO2 flux) is now integrated
into the `FluxProcessingChain`. MDS is used during gap-filling in flux Level-4.1.

- **Example notebook** using MDS as part of the flux processing chain where it is used together with random
forest: [Flux Processing Chain](/notebooks/FluxProcessingChain/FluxProcessingChain.ipynb)
- **Example notebook** using MDS as stand alone class
`FluxMDS`: [MDS gap-filling of ecosystem fluxes](/notebooks/GapFilling/FluxMDSGapFilling.ipynb)

The `diive` implementation of the MDS gap-filling method adheres to the descriptions in Reichstein et al. (2005) and
Vekuri et al. (2023), similar to the standard gap-filling procedures used by FLUXNET, ICOS, ReddyProc, and other similar
platforms. This method fills gaps by substituting missing flux values with average flux values observed under comparable
meteorological conditions.


Background: different flux levels

- The class `FluxProcessingChain` in `diive` follows the flux processing steps as shown in
the [Flux Processing Chain](https://www.swissfluxnet.ethz.ch/index.php/data/ecosystem-fluxes/flux-processing-chain/)
outlined by [Swiss FluxNet](https://www.swissfluxnet.ethz.ch/).
-
- The flux processing chain uses different levels for different steps in the chain:
- Level-0: preliminary flux calculations, e.g. during the year,
using [EddyPro](https://www.licor.com/products/eddy-covariance/eddypro)
- Level-1: final flux calculations, e.g. for complete year,
using [EddyPro](https://www.licor.com/products/eddy-covariance/eddypro)
- Level-2: quality flag expansion (flagging)
- Level-3.1: storage correction (using one point measurement only, from profile not included by default)
- Level-3.2: outlier removal (flagging)
- Level-3.3: USTAR filtering (constant threshold, must be known, detection process not included by default) (
flagging)
- Following Level 3.3, a comprehensive quality flag (`QCF`) is generated by combining individual quality flags.
Prior to subsequent processing steps, low-quality data (flag=2) is removed. Medium-quality data (flag=1) can be
retained if necessary, while the highest quality data (flag=0) is always kept.
- Level-4.1: gap-filling (MDS, long-term random forest)

Changes

- Changes in `FluxMDS`:
- Added parameter `avg_min_n_vals` in MDS gap-filling
- Renamed tolerance parameters for MDS gap-filling to `*_tol`
- (`diive.pkgs.gapfilling.mds.FluxMDS`)
- When reading a parquet file, sanitizing the timestamp is now optional (`diive.core.io.files.load_parquet`)
- The function for creating lagged variants is now found in `diive.pkgs.createvar.laggedvariants.lagged_variants`

Additions

- Added more text output for fill quality during gap-filling with MDS (`diive.pkgs.gapfilling.mds.FluxMDS`)
- Added MDS gap-filling to flux processing chain (
`diive.pkgs.fluxprocessingchain.fluxprocessingchain.FluxProcessingChain`)
- Allow fitting to unbinned data (`diive.pkgs.fits.fitter.BinFitterCP`)
- Added parameter to edit y-label (`diive.core.plotting.dielcycle.DielCycle`)
- Added preliminary USTAR filtering for NEE to quick flux processing chain (
`diive.pkgs.fluxprocessingchain.fluxprocessingchain.QuickFluxProcessingChain`)
- `FileSplitter`:
- Added parameter to directly output splits as `parquet` files in `FileSplitter` and `FileSplitterMulti`. These two
classes split longer time series files (e.g., 6 hours) into several smaller splits (e.g., 12 half-hourly files).
Usage of parquet speeds up not only the splitting part, but also the process when later re-reading the files for
other processing steps.
- After splitting, missing values in the split files are numpy NAN (`diive.core.io.filesplitter.FileSplitter`)
- Added parameter to hide default plot when called. The method `defaultplot` is used e.g. by outlier detection methods
to plot the data after outlier removal, to show flagged vs. unflagged values. (
`diive.core.base.flagbase.FlagBase.defaultplot`)
- Added new filetype `ETH-SONICREAD-BICO-MOD-CSV-20HZ`
- Added `fig` property that contains the default plot for outlier removal methods. This is useful when the default plot
is needed elsewhere, e.g. saved to a file. At the moment, the parameter `showplot` must be `True` for the property to
be accessible. (`diive.core.base.flagbase.FlagBase`)
- Example for class `zScoreRolling`:

zsr = zScoreRolling(..., showplot=True, ...)
zsr.calc(repeat=True)
fig = zsr.fig Contains the figure instance
fig.savefig(...) Figure can then be saved to a file etc.


Notebooks

- Added notebook example for creating lagged variants of variables (
`notebooks/CalculateVariable/Create_lagged_variants.ipynb`)
- Updated flux processing chain notebook to `v9.0`: added option for MDS gap-filling, more descriptions
- Bugfix: import for loading from `Path` was missing in flux processing chain notebook
- Updated MDS gap-filling notebook to `v1.1`, added more descriptions and example for `min_n_vals_nt` parameter
- Updated quick flux processing chain notebook

Unittests

- Added test case `tests.test_createvar.TestCreateVar.test_lagged_variants`
- Updated test case `tests.test_gapfilling.TestGapFilling.test_fluxmds`
- Updated test case `tests.test_fluxprocessingchain.TestFluxProcessingChain.test_fluxprocessingchain`
- 53/53 unittests ran successfully

Bugfixes

- The setting for features that should not be lagged was not properly implemented (
`diive.pkgs.fluxprocessingchain.fluxprocessingchain.FluxProcessingChain._get_ml_feature_settings`)
- Fixed bug when plotting (`diive.pkgs.outlierdetection.localsd.LocalSD`)

Changes

- Adjust version number to avoid publishing conflict

Bugfixes

- Removed invalid imports

Tests

- Added test case for `diive` imports (`tests.test_imports.TestImports.test_imports`)
- 52/52 unittests ran successfully