Diive

Expanding Flux Processing Capabilities

This update brings advancements for post-processing eddy covariance data in the context of the `FluxProcessingChain`.
The goal is to offer a complete chain for post-processing ecosystem flux data, specifically designed to work seamlessly
with the standardized `_fluxnet` output file from the
widely-used [EddyPro](https://www.licor.com/env/products/eddy-covariance/eddypro) software.

Now, diive offers the option for USTAR filtering based on *known* constant thresholds across the entire dataset (similar
to the `CUT` scenarios in FLUXNET data). While seasonal (DJF, MAM, JJA, SON) thresholds are calculated internally,
applying them on a seasonal basis or using variable thresholds per year (like FLUXNET's `VUT` scenarios) isn't yet
implemented.

With this update, the `FluxProcessingChain` class can handle various data processing steps:

- Level-2: Quality flag expansion
- Level-3.1: Storage correction
- Level-3.2: Outlier removal
- Level-3.3: (new) USTAR filtering (with constant thresholds for now)
- (upcoming) Level-4.1: long-term gap-filling using random forest and XGBoost
- For info about the different flux levels
see [Swiss FluxNet flux processing chain](https://www.swissfluxnet.ethz.ch/index.php/data/ecosystem-fluxes/flux-processing-chain/)

New features

- Added class to apply multiple known constant USTAR (friction velocity) thresholds, creating flags that indicate time
periods characterized by low turbulence for multiple USTAR scenarios. The constant thresholds must be known
beforehand, e.g., from an earlier USTAR detection run, or from results from FLUXNET (
`diive.pkgs.flux.ustarthreshold.FlagMultipleConstantUstarThresholds`)
- Added class to apply one single known constant USTAR thresholds (
`diive.pkgs.flux.ustarthreshold.FlagSingleConstantUstarThreshold`)
- Added `FlagMultipleConstantUstarThresholds` to the flux processing chain (
`diive.pkgs.fluxprocessingchain.fluxprocessingchain.FluxProcessingChain.level33_constant_ustar`)
- Added USTAR detection algorithm based on Papale et al., 2006 (`diive.pkgs.flux.ustarthreshold.UstarDetectionMPT`)
- Added function to analyze high-quality ecosystem fluxes that helps in understanding the range of highest-quality data(
`diive.pkgs.flux.hqflux.analyze_highest_quality_flux`)

Additions

- `LocalSD` outlier detection can now use a constant SD:
- Added parameter to use standard deviation across all data (constant) instead of the rolling SD to calculate the
upper and lower limits that define outliers in the median rolling window (
`diive.pkgs.outlierdetection.localsd.LocalSD`)
- Added to step-wise outlier detection (
`diive.pkgs.outlierdetection.stepwiseoutlierdetection.StepwiseOutlierDetection.flag_outliers_localsd_test`)
- Added to meteoscreening from database (
`diive.pkgs.qaqc.meteoscreening.StepwiseMeteoScreeningDb.flag_outliers_localsd_test`)
- Added to flux processing chain (
`diive.pkgs.fluxprocessingchain.fluxprocessingchain.FluxProcessingChain.level32_flag_outliers_localsd_test`)

Changes

- Replaced `.plot_date()` from the Matplotlib library with `.plot()` due to deprecation

Notebooks

- Added notebook for plotting cumulative sums per year (`notebooks/Plotting/CumulativesPerYear.ipynb`)
- Added notebook for removing outliers based on the z-score in rolling time window (
`notebooks/OutlierDetection/zScoreRolling.ipynb`)

Bugfixes

- Fixed bug when saving a pandas Series to parquet (`diive.core.io.files.save_parquet`)
- Fixed bug when plotting `doy_mean_cumulative`: no longer crashes when years defined in parameter
`excl_years_from_reference` are not in dataset (`diive.core.times.times.doy_mean_cumulative`)
- Fixed deprecation warning when plotting in `bokeh` (interactive plots)

Tests

- Added unittest for `LocalSD` using constant SD (
`tests.test_outlierdetection.TestOutlierDetection.test_localsd_with_constantsd`)
- Added unittest for rolling z-score outlier removal (
`tests.test_outlierdetection.TestOutlierDetection.test_zscore_rolling`)
- Improved check if figure and axis were created in (`tests.test_plots.TestPlots.test_histogram`)
- 39/39 unittests ran successfully

Environment

- Added new package `scikit-optimize`
- Added new package `category_encoders`

Additions

- Added outlier tests to step-wise meteoscreening from database: `Hampel`, `HampelDaytimeNighttime` and `TrimLow` (
`diive.pkgs.qaqc.meteoscreening.StepwiseMeteoScreeningDb`)
- Added parameter to control whether or not to output the middle timestamp when loading parquet files with
`load_parquet()`. By default, `output_middle_timestamp=True`. (`diive.core.io.files.load_parquet`)

Environment

- Re-created environment and created new `lock` file
- Currently using Python 3.9.19

Notebooks

- Added new notebook for creating a flag that indicates missing values (
`notebooks/OutlierDetection/MissingValues.ipynb`)
- Updated notebook for meteoscreening from database (
`notebooks/MeteoScreening/StepwiseMeteoScreeningFromDatabase.ipynb`)
- Updated notebook for loading and saving parquet files (`notebooks/Formats/LoadSaveParquetFile.ipynb`)

Tests

- Added unittest for flagging missing values (`tests.test_outlierdetection.TestOutlierDetection.test_missing_values`)
- 37/37 unittests ran successfully

Bugfixes

- Fixed links in README, needed absolute links to notebooks
- Fixed issue with return list in (`diive.pkgs.analyses.histogram.Histogram.peakbins`)

Additions

- Added new function to apply quality flags to certain time periods only (`diive.pkgs.qaqc.flags.restrict_application`)
- Added to option to restrict the application of the angle-of-attack flag to certain time periods (
`diive.pkgs.fluxprocessingchain.level2_qualityflags.FluxQualityFlagsEddyPro.angle_of_attack_test`)

Changes

- Test options in `FluxProcessingChain` are now always passed as dict. This has the advantage that in addition to run
the test by setting the dict key `apply` to `True`, various other test settings can be passed, for example the new
parameter `application dates` for the angle-of-attack flag. (
`diive.pkgs.fluxprocessingchain.fluxprocessingchain.FluxProcessingChain`)

Tests

- Added unittest for Flux Processing Chain up to Level-2 (
`tests.test_fluxprocessingchain.TestFluxProcessingChain.test_fluxprocessingchain_level2`)
- 36/36 unittests ran successfully

This version introduces a histogram plot that has the option to display z-score as vertical lines superimposed on the
distribution, which helps in assessing z-score settings used by some outlier removal functions.


*Histogram plot of half-hourly air temperature measurements at the ICOS Class 1 ecosystem
station [Davos](https://www.swissfluxnet.ethz.ch/index.php/sites/site-info-ch-dav/) between 2013 and 2022, displayed in
20 equally-spaced bins. The dashed vertical lines show the z-score and the corresponding value calculated based on the
time series. The bin with most counts is highlighted orange.*

New features

- Added new class `HistogramPlot`for plotting histograms, based on the Matplotlib
implementation (`diive.core.plotting.histogram.HistogramPlot`)
- Added function to calculate the value for a specific z-score, e.g., based on a time series it calculates the value
where z-score = `3` etc. (`diive.core.funcs.funcs.val_from_zscore`)

Additions

- Added histogram plots to `FlagBase`, histograms are now shown for all outlier methods (
`diive.core.base.flagbase.FlagBase.defaultplot`)
- Added daytime/nighttime histogram plots to (`diive.pkgs.outlierdetection.hampel.HampelDaytimeNighttime`)
- Added daytime/nighttime histogram plots to (`diive.pkgs.outlierdetection.zscore.zScoreDaytimeNighttime`)
- Added daytime/nighttime histogram plots to (`diive.pkgs.outlierdetection.lof.LocalOutlierFactorDaytimeNighttime`)
- Added daytime/nighttime histogram plots to (
`diive.pkgs.outlierdetection.absolutelimits.AbsoluteLimitsDaytimeNighttime`)
- Added option to calculate the z-score with sign instead of absolute (`diive.core.funcs.funcs.zscore`)

Changes

- Improved daytime/nighttime outlier plot used by various outlier removal classes (
`diive.core.base.flagbase.FlagBase.plot_outlier_daytime_nighttime`)

Notebooks

- Added notebook for plotting histograms (`notebooks/Plotting/Histogram.ipynb`)
- Added notebook for manual removal of data points (`notebooks/OutlierDetection/ManualRemoval.ipynb`)
- Added notebook for outlier detection using local outlier factor, separately during daytime and nighttime (
`notebooks/OutlierDetection/LocalOutlierFactorDaytimeNighttime.ipynb`)
- Updated notebook (`notebooks/OutlierDetection/HampelDaytimeNighttime.ipynb`)
- Updated notebook (`notebooks/OutlierDetection/AbsoluteLimitsDaytimeNighttime.ipynb`)
- Updated notebook (`notebooks/OutlierDetection/zScoreDaytimeNighttime.ipynb`)
- Updated notebook (`notebooks/OutlierDetection/LocalOutlierFactorAllData.ipynb`)

Tests

- Added unittest for plotting histograms (`tests.test_plots.TestPlots.test_histogram`)
- Added unittest for calculating histograms (without plotting) (`tests.test_analyses.TestCreateVar.test_histogram`)

Additions

- Added CITATIONS file

Changes

- Added option to set different `n_sigma` for daytime and nightime data
in `HampelDaytimeNighttime` (`diive.pkgs.outlierdetection.hampel.HampelDaytimeNighttime`)
- Updated `flag_outliers_hampel_dtnt_test` in step-wise outlier detection
- Updated `level32_flag_outliers_hampel_dtnt_test` in flux processing chain

Notebooks

- Updated notebook `HampelDaytimeNighttime`
- Updated notebook `FluxProcessingChain`

Tests

- Updated unittest `test_hampel_filter_daytime_nighttime`
- 35/35 unittests ran successfully