Diive

Additions

- Added plotting library `bokeh` to dependencies

Changes

- When combining data of different time resolutions, the data are now combined using
`.combine_first()` instead of `.concat()` to avoid duplicates during merging. This
should work reliably because data of the highest resolution are available first, and then
lower resolution upsampled (backfilled) data are added, filling gaps in the high
resolution data. Because gaps are filled, overlaps between the two resolutions are avoided.
With `.concat()`, gaps were not filled, but timestamps were simply added as new records,
and thus duplicates in the timestamp occurred.
(`pkgs.qaqc.meteoscreening.StepwiseMeteoScreeningDb._harmonize_timeresolution`)
- Updated dependencies to newest possible versions

Changes

- Removed the packages `jupyterlab` and `jupyter-bokeh` from dependencies, because
the latter caused issues when trying to install `diive` in a `conda` environment
on a shared machine. Both dependencies are still listed in the `pyproject.toml`
file as `dev` dependencies. It makes sense to keep both packages separate from
`diive` because they are specifically for `jupyter` notebooks and not strictly
related to `diive` functionality.

Additions

- In `StepwiseMeteoScreeningDb` the current cleaned timeseries can now be
plotted with `showplot_current_cleaned`.
- Timeseries can now be plotted using the `bokeh` library. This plot are interactive
and can be directly used in jupyter notebooks. (`core.plotting.timeseries.TimeSeries`)
- Added new plotting package `jupyter_bokeh` for interactive plotting in Jupyter lab.
- Added new plotting package `seaborn`.

Bugfixes

- `StepwiseMeteoScreeningDb` now works on a copy of the input data to avoid
unintended data overwrite of input.

New Features

- **Data formats**: Added new package `diive/pkgs/formats` that assists in converting
data outputs to formats required e.g. for data sharing with FLUXNET.
- Convert the EddyPro `_fluxnet_` output file to the FLUXNET data format for
data upload (data sharing). (`pkgs.formats.fluxnet.ConvertEddyProFluxnetFileForUpload`)
- **Insert timestamp column**: Insert timestamp column that shows the START, END
or MIDDLE time of the averaging interval (`core.times.times.insert_timestamp`)
- **Manual removal of data points**: Flag manually defined data points as outliers.
(`pkgs.outlierdetection.manualremoval.ManualRemoval`)

Additions

Added additional outlier detection algorithms
to `StepwiseMeteoScreeningDb` (`pkgs.qaqc.meteoscreening.StepwiseMeteoScreeningDb`):

- Added local outlier factor test, across all data
(`pkgs.qaqc.meteoscreening.StepwiseMeteoScreeningDb.flag_outliers_lof_test`)
- Added local outlier factor test, separately for daytime and nighttime
(`pkgs.qaqc.meteoscreening.StepwiseMeteoScreeningDb.flag_outliers_lof_dtnt_test`)

Random uncertainty

- Implemented random flux uncertainty calculation, based on Holliner and Richardson (2005)
and Pastorello et al. (2020). Calculations also include a first estimate of the error
propagation when summing up flux values to annual sums. See end of CHANGELOG for links to references.
(`pkgs.flux.uncertainty.RandomUncertaintyPAS20`)

Additions

- Added example data in `diive/configs/exampledata`, including functions to load the data.

Changes

- In `core.io.filereader`, the following classes now also accept `output_middle_timestamp`
(boolean with default `True`) as parameter: `MultiDataFileReader`, `ReadFileType`,`DataFileReader`.
This allows to keep the original timestamp of the data.
- Some minor plotting adjustments

StepwiseMeteoScreeningDb

**Stepwise quality-screening of meteorological data, directly from the database**

In this update, the stepwise meteoscreening directly from the database introduced in the
previous update was further refined and extended, with additional outlier tests and corrections
implemented. The stepwise meteoscreening allows to perform step-by-step quality tests on
meteorological. A preview plot after running a test is shown and the user can decide if
results are satisfactory or if the same test with different parameters should be re-run.
Once results are satisfactory, the respective test flag is added to the data. After running
the desired tests, an overall flag `QCF` is calculated from all individual tests.

In addition to the creation of quality flags, the stepwise screening allows to correct
data for common issues. For example, short-wave radiation sensors often measure negative
values during the night. These negative values are useful because they give info about
the accuracy and precision of the sensor. In this case, values during the night should
be zero. Instead of cutting off negative values, `diive` detects the nighttime offset
for each day and then calculates a correction slope between individual days. This way,
the daytime values are also corrected.

After quality-screening and corrections, data are resampled to 30MIN time resolution.

At the moment, the stepwise meteoscreening works for data downloaded from the `InfluxDB`
database. The screening respects the database format (including tags) and prepares
the screened, corrected and resampled data for direct database upload.

Due to its modular approach, the stepwise screening can be easily adjusted
to work with any type of data files. This adjustment will be done in one of the next
updates.

Changes

- Renamed class `MetScrDbMeasurementVars`
to `StepwiseMeteoScreeningDb` (`pkgs.qaqc.meteoscreening.StepwiseMeteoScreeningDb`)

Additions

- **Stepwise MeteoScreening**:
Added access to multiple methods for easy stepwise execution:
- Added local SD outlier test (`pkgs.qaqc.meteoscreening.StepwiseMeteoScreeningDb.flag_outliers_localsd_test`)
- Added absolute limits outlier test (`pkgs.qaqc.meteoscreening.StepwiseMeteoScreeningDb.flag_outliers_abslim_test`)
- Added correction to remove radiation zero
offset (`pkgs.qaqc.meteoscreening.StepwiseMeteoScreeningDb.correction_remove_radiation_zero_offset`)
- Added correction to remove relative humidity
offset (`pkgs.qaqc.meteoscreening.StepwiseMeteoScreeningDb.correction_remove_relativehumidity_offset`)
- Added correction to set values above a threshold to
threshold (`pkgs.qaqc.meteoscreening.StepwiseMeteoScreeningDb.correction_setto_max_threshold`)
- Added correction to set values below a threshold to
threshold (`pkgs.qaqc.meteoscreening.StepwiseMeteoScreeningDb.correction_setto_min_threshold`)
- Added comparison plot before/after QC and
corrections (`pkgs.qaqc.meteoscreening.StepwiseMeteoScreeningDb.showplot_resampled`)