Diive

Random Forest Update

- Refactored code for `pkgs/gapfilling/randomforest_ts.py`
- Implemented lagged variants of variables
- Implemented long-term gap-filling, where the model to gap-fill a specific year is built from the
respective year and its neighboring years
- Implemented feature reduction using sklearn's RFECV
- Implemented TimeSeriesSplit used as the cross-validation splitting strategy during feature reduction
- Implemented `TimestampSanitizer` also when reading from file with `core.io.filereader.DataFileReader`
- Removed old code in `.core.dfun.files` and moved files logistics to `.core.io.files` instead
- Implemented saving and loading Python `pickles` in `.core.io.files`

Meteoscreening PA, RH

- Added function `pkgs.corrections.offsetcorrection.remove_relativehumidity_offset` to correct
humidity measurements for values > 100%

Other

- Added first code for outlier detection via seasonal trends in `pkgs/outlierdetection/seasonaltrend.py`
- Prepared `pkgs/analyses/optimumrange.py` for future updates

MeteoScreening Radiation

MeteoScreening

- Implemented corrections and quality screening for radiation data in `pkgs.qaqc.meteoscreening`

Corrections

Additions to `pkgs.corrections`:

- Added function `.offsetcorrection.remove_radiation_zero_offset` to correct radiation
data for nighttime offsets
- Added function `.setto_threshold.setto_threshold` to set values above or below a
specfied threshold value to the threshold.

Plotting

- Added function `core.plotting.plotfuncs.quickplot` for quickly plotting pandas
Series and DataFrame data

Resampling

- Implemented `TimeSanitizer` in `core.times.resampling.resample_series_to_30MIN`

Other

- Added decorator class `core.utils.prints.ConsoleOutputDecorator`, a wrapper to
execute functions with additional info that is output to the console.

MeteoScreening Preparations

- Added new class `core.times.times.TimestampSanitizer`
- Class that handles timestamp checks and fixes, such as the creation of a continuous
timestamp without date gaps.
- Added `pkgs.createvar.nighttime_latlon.nighttime_flag_from_latlon`
- Function for the calculation of a nighttime flag (1=nighttime) from latitude and
longitude coordinates of a specific location.
- Added `core.plotting.heatmap_datetime.HeatmapDateTime`
- Class to generate a heatmap plot from timeseries data.

MeteoScreening Air Temperature

MeteoScreening uses a general settings file `pipes_meteo.yaml` that contains info how
specific `measurements` should be screened. Such `measurements` group similar variables
together, e.g. different air temperatures are measurement `TA`.
Additions to module `pkgs.qaqc.meteoscreening`:

- Added class `ScreenVar`
- Performs quality screening of air temperature `TA`.
- As first check, I implemented outlier detection via the newly added package `ThymeBoost`,
along with checks for absolute limits.
- Screening applies the checks defined in the file `pipes_meteo.yaml` for the respective
`measurement`, e.g. `TA` for air temperature.
- The screening outputs a separate dataframe that contains `QCF` flags for each check.
- The checks do not change the original time series. Instead, only the flags are generated.
- Screening routines for more variables will be added over the next updates.
- Added class `MeteoScreeningFromDatabaseSingleVar`
- Performs quality screening *and* resampling to 30MIN of variables downloaded from the database.
- It uses the `detailed` data when downloading data from the database using `dbc-influxdb`.
- The `detailed` data contains the measurement of the variable, along with multiple tags that
describe the data. The tags are needed for storage in the database.
- After quality screening of the original high-resolution data, flagged values are removed and
then data are resampled.
- It also handles the issue that data downloaded for a specific variable can have different time
resolution over the years, although I still need to test this.
- After screening and resampling, data are in a format that can be directly uploaded to the
database using `dbc-influxdb`.
- Added class `MeteoScreeningFromDatabaseMultipleVars`
- Wrapper where multiple variables can be screened in one run.
- This should also work in combination of different `measurements`. For example, screening
radiation and temperature data in one run.

Outlier Detection

Additions to `pkgs.outlierdetection`:

- Added module `thymeboost`
- Added module `absolute_limits`

[//]: (- optimum range)

[//]: (- `diive.core.times` `DetectFrequency` )

[//]: (- `diive.core.times`: `resampling` module )

[//]: (- New package in env: `ThymeBoost` [GitHub]&40;https://github.com/tblume1992/ThymeBoost/tree/main/ThymeBoost) )

Carbon cost

**GENERAL**

- This version introduces the code for calculating carbon cost and critical heat days.

**NEW PACKAGES**

- Added new package for flux-specific calculations: `diive.pkgs.flux`

**NEW MODULES**

- Added new module for calculating carbon cost: `diive.pkgs.flux.carboncost`
- Added new module for calculating critical heat days: `diive.pkgs.flux.criticalheatdays`

**CHANGES & ADDITIONS**

- None

**BUGFIXES**

- None