Diive

New features

- Added new class `ScatterXY`: a simple scatter plot that supports bins (`core.plotting.scatter.ScatterXY`)


Notebooks

- Added notebook `notebooks/Plotting/ScatterXY.ipynb`

New features

- Added new class `DaytimeNighttimeFlag` to calculate daytime flag (1=daytime, 0=nighttime),
nighttime flag (1=nighttime, 0=daytime) and potential radiation from latitude and
longitude (`diive.pkgs.createvar.daynightflag.DaytimeNighttimeFlag`)

Additions

- Added support for N2O and CH4 fluxes during the calculation of the `QCF` quality flag in class `FlagQCF`
- Added first code for USTAR threshold detection for NEE

Notebooks

- Added new notebook `notebooks/CalculateVariable/Daytime_and_nighttime_flag.ipynb`

Changes

- `diive` repository is now hosted on GitHub.

Additions

- Added first code for XGBoost gap-filling, not production-ready yet
- Added check if enough columns for lagging features in class `RandomForestTS`
- Added more details in report for class `FluxStorageCorrectionSinglePointEddyPro`

Bugfixes

- Fixed check in `RandomForestTS` for bug in `QuickFillRFTS`: number of available columns was checked too early
- Fixed `QuickFillRFTS` implementation in `OutlierSTLRZ`
- Fixed `QuickFillRFTS` implementation in `ThymeBoostOutlier`

Environment

- Added new package [xgboost](https://xgboost.readthedocs.io/en/stable/#)
- Updated all packages

New features

- Implemented feature reduction (permutation importance) as separate method in `RandomForestTS`
- Added new function to set values within specified time ranges to a constant
value(`pkgs.corrections.setto_value.setto_value`)
- The function is now also implemented as method
in `StepwiseMeteoScreeningDb` (`pkgs.qaqc.meteoscreening.StepwiseMeteoScreeningDb.correction_setto_value`)

Notebooks

- Updated notebook `notebooks/GapFilling/RandomForestGapFilling.ipynb`
- Updated notebook `notebooks/GapFilling/QuickRandomForestGapFilling.ipynb`
- Updated notebook `notebooks/MeteoScreening/StepwiseMeteoScreeningFromDatabase.ipynb`

Environment

- Added new package [SHAP](https://shap.readthedocs.io/en/latest/)
- Added new package [eli5](https://pypi.org/project/eli5/)

Tests

- Updated testcase for gap-filling with random
forest (`test_gapfilling.TestGapFilling.test_gapfilling_randomforest`)

New features

- Re-implemented gap-filling of long-term time series spanning multiple years, where the model
to gap-fill a specific year is built from data from the respective year and its two closest
neighboring years. (`pkgs.gapfilling.randomforest_ts.LongTermRandomForestTS`)

Bugfixes

- Fixed bug in `StepwiseMeteoScreeningDb` where position of `return` during setup was incorrect

New features

- Added function to calculate the daily correlation between two time
series (`pkgs.analyses.correlation.daily_correlation`)
- Added function to calculate potential radiation (`pkgs.createvar.potentialradiation.potrad`)

Bugfixes

- Fixed bug in `StepwiseMeteoScreeningDb` where the subclass `StepwiseOutlierDetection`
did not use the already sanitized timestamp from the parent class, but sanitized the timestamp
a second time, leading to potentially erroneous and irregular timestamps.

Changes

- `RandomForestTS` now has the following functions included as methods:
- `steplagged_variants`: includes lagged variants of features
- `include_timestamp_as_cols`: includes timestamp info as data columns
- `add_continuous_record_number`: adds continuous record number as new column
- `sanitize`: validates and prepares timestamps for further processing
- `RandomForestTS` now outputs an additional predictions column where predictions from
the full model and predictions from the fallback model are collected
- Renamed function `steplagged_variants` to `lagged_variants` (`core.dfun.frames.lagged_variants`)
- Updated function `lagged_variants`: now accepts a list of lag times. This makes it possible
to lag variables in both directions, i.e., the observed value can be paired with values before
and after the actual time. For example, the variable `TA` is the observed value at the current
timestamp, `TA-1` is the value from the preceding record, and `TA+1` is the value from the next
record. Using values from the next record can be useful when modeling observations using data
from a neighboring measurement location that has similar records but lagged in time due to
distance.
- Updated README

Tests

- Updated testcase for gap-filling with random
forest (`test_gapfilling.TestGapFilling.test_gapfilling_randomforest`)

Notebooks

- Updated `notebooks/MeteoScreening/StepwiseMeteoScreeningFromDatabase.ipynb`

Additions

- Added more args for better control of `TimestampSanitizer` (`core.times.times.TimestampSanitizer`)
- Refined various docstrings