Diive

New features

- Implemented feature reduction (permutation importance) as separate method in `RandomForestTS`
- Added new function to set values within specified time ranges to a constant
value(`pkgs.corrections.setto_value.setto_value`)
- The function is now also implemented as method
in `StepwiseMeteoScreeningDb` (`pkgs.qaqc.meteoscreening.StepwiseMeteoScreeningDb.correction_setto_value`)

Notebooks

- Updated notebook `notebooks/GapFilling/RandomForestGapFilling.ipynb`
- Updated notebook `notebooks/GapFilling/QuickRandomForestGapFilling.ipynb`
- Updated notebook `notebooks/MeteoScreening/StepwiseMeteoScreeningFromDatabase.ipynb`

Environment

- Added new package [SHAP](https://shap.readthedocs.io/en/latest/)
- Added new package [eli5](https://pypi.org/project/eli5/)

Tests

- Updated testcase for gap-filling with random
forest (`test_gapfilling.TestGapFilling.test_gapfilling_randomforest`)

New features

- Re-implemented gap-filling of long-term time series spanning multiple years, where the model
to gap-fill a specific year is built from data from the respective year and its two closest
neighboring years. (`pkgs.gapfilling.randomforest_ts.LongTermRandomForestTS`)

Bugfixes

- Fixed bug in `StepwiseMeteoScreeningDb` where position of `return` during setup was incorrect

New features

- Added function to calculate the daily correlation between two time
series (`pkgs.analyses.correlation.daily_correlation`)
- Added function to calculate potential radiation (`pkgs.createvar.potentialradiation.potrad`)

Bugfixes

- Fixed bug in `StepwiseMeteoScreeningDb` where the subclass `StepwiseOutlierDetection`
did not use the already sanitized timestamp from the parent class, but sanitized the timestamp
a second time, leading to potentially erroneous and irregular timestamps.

Changes

- `RandomForestTS` now has the following functions included as methods:
- `steplagged_variants`: includes lagged variants of features
- `include_timestamp_as_cols`: includes timestamp info as data columns
- `add_continuous_record_number`: adds continuous record number as new column
- `sanitize`: validates and prepares timestamps for further processing
- `RandomForestTS` now outputs an additional predictions column where predictions from
the full model and predictions from the fallback model are collected
- Renamed function `steplagged_variants` to `lagged_variants` (`core.dfun.frames.lagged_variants`)
- Updated function `lagged_variants`: now accepts a list of lag times. This makes it possible
to lag variables in both directions, i.e., the observed value can be paired with values before
and after the actual time. For example, the variable `TA` is the observed value at the current
timestamp, `TA-1` is the value from the preceding record, and `TA+1` is the value from the next
record. Using values from the next record can be useful when modeling observations using data
from a neighboring measurement location that has similar records but lagged in time due to
distance.
- Updated README

Tests

- Updated testcase for gap-filling with random
forest (`test_gapfilling.TestGapFilling.test_gapfilling_randomforest`)

Notebooks

- Updated `notebooks/MeteoScreening/StepwiseMeteoScreeningFromDatabase.ipynb`

Additions

- Added more args for better control of `TimestampSanitizer` (`core.times.times.TimestampSanitizer`)
- Refined various docstrings

New features

- Added new class for optimizing random forest parameters (`pkgs.gapfilling.randomforest_ts.OptimizeParamsRFTS`)
- Added new plots for prediction error and residuals (`core.ml.common.plot_prediction_residuals_error_regr`)
- Added function that adds a continuous record number as new column in a dataframe. This
could be useful to include as feature in gap-filling models for long-term datasets spanning multiple years.
(`core.dfun.frames.add_continuous_record_number`)

Changes

- When reading CSV files with pandas `.read_csv()`, the arg `mangle_dupe_cols=True`
was removed because it is deprecated since pandas 2.0 ...
- ... therefore the check for duplicate column names in class `ColumnNamesSanitizer`
has been refactored. In case of duplicate columns names, an integer suffix is added to
the column name. For example: `VAR` is renamed to `VAR.1` if it already exists in the
dataframe. In case `VAR.1` also already exists, it is renamed to `VAR.2`, and so on.
The integer suffix is increased until the variable name is unique. (`core.io.filereader.ColumnNamesSanitizer`)
- Similarly, when reading CSV files with pandas `.read_csv()`, the arg `date_parser` was
removed because it is deprecated since pandas 2.0. When reading a CSV, the arg `date_format`
is now used instead. The input format remains unchanged, it is still a string giving the datetime
format, such as `"%Y%m%d%H%M"`.
- The random feature variable is now generated using the same random state as the
model. (`pkgs.gapfilling.randomforest_ts.RandomForestTS`)
- Similarly, `train_test_split` is now also using the same random state as the
model. (`pkgs.gapfilling.randomforest_ts.RandomForestTS`)

Notebooks

- Added new notebook `notebooks/GapFilling/RandomForestParamOptimization.ipynb`

Tests

- Added testcase for loading dataframe from parquet file (`test_loaddata.TestLoadFiletypes.test_exampledata_parquet`)
- Added testcase for gap-filling with random forest (`test_gapfilling.TestGapFilling.test_gapfilling_randomforest`)

Environment

- Updated `poetry` to latest version `1.6.1`
- Updated all packages to their latest versions
- Added new package [yellowbrick](https://www.scikit-yb.org/en/latest/)

MeteoScreening from database - update

The class `StepwiseMeteoScreeningDb`, which is used for quality-screening of meteo data
stored in the ETH Grassland Sciences database, has been refactored. It is now using the
previously introduced class `StepwiseOutlierDetection` for outlier
tests. (`pkgs.qaqc.meteoscreening.StepwiseMeteoScreeningDb`)

Removed

The following classes are no longer used and were removed from step-wise outlier detection:

- Removed z-score IQR test, too unreliable (`pkgs.outlierdetection.zscore.zScoreIQR`)
- Similarly, removed seasonal trend decomposition that used z-score IQR test, too
unreliable (`pkgs.outlierdetection.seasonaltrend.OutlierSTLRIQRZ`)

Notebooks

- Updated notebook `notebooks/MeteoScreening/StepwiseMeteoScreeningFromDatabase.ipynb`

Notebooks

- Added new notebook `notebooks/GapFilling/RandomForestGapFilling.ipynb`
- Added new notebook `notebooks/GapFilling/QuickRandomForestGapFilling.ipynb`
- Added new notebook `notebooks/Workbench/Remove_unneeded_cols.ipynb`