Diive

Additions

- In `StepwiseMeteoScreeningDb` the current cleaned timeseries can now be
plotted with `showplot_current_cleaned`.
- Timeseries can now be plotted using the `bokeh` library. This plot are interactive
and can be directly used in jupyter notebooks. (`core.plotting.timeseries.TimeSeries`)
- Added new plotting package `jupyter_bokeh` for interactive plotting in Jupyter lab.
- Added new plotting package `seaborn`.

Bugfixes

- `StepwiseMeteoScreeningDb` now works on a copy of the input data to avoid
unintended data overwrite of input.

New Features

- **Data formats**: Added new package `diive/pkgs/formats` that assists in converting
data outputs to formats required e.g. for data sharing with FLUXNET.
- Convert the EddyPro `_fluxnet_` output file to the FLUXNET data format for
data upload (data sharing). (`pkgs.formats.fluxnet.ConvertEddyProFluxnetFileForUpload`)
- **Insert timestamp column**: Insert timestamp column that shows the START, END
or MIDDLE time of the averaging interval (`core.times.times.insert_timestamp`)
- **Manual removal of data points**: Flag manually defined data points as outliers.
(`pkgs.outlierdetection.manualremoval.ManualRemoval`)

Additions

Added additional outlier detection algorithms
to `StepwiseMeteoScreeningDb` (`pkgs.qaqc.meteoscreening.StepwiseMeteoScreeningDb`):

- Added local outlier factor test, across all data
(`pkgs.qaqc.meteoscreening.StepwiseMeteoScreeningDb.flag_outliers_lof_test`)
- Added local outlier factor test, separately for daytime and nighttime
(`pkgs.qaqc.meteoscreening.StepwiseMeteoScreeningDb.flag_outliers_lof_dtnt_test`)

Random uncertainty

- Implemented random flux uncertainty calculation, based on Holliner and Richardson (2005)
and Pastorello et al. (2020). Calculations also include a first estimate of the error
propagation when summing up flux values to annual sums. See end of CHANGELOG for links to references.
(`pkgs.flux.uncertainty.RandomUncertaintyPAS20`)

Additions

- Added example data in `diive/configs/exampledata`, including functions to load the data.

Changes

- In `core.io.filereader`, the following classes now also accept `output_middle_timestamp`
(boolean with default `True`) as parameter: `MultiDataFileReader`, `ReadFileType`,`DataFileReader`.
This allows to keep the original timestamp of the data.
- Some minor plotting adjustments

StepwiseMeteoScreeningDb

**Stepwise quality-screening of meteorological data, directly from the database**

In this update, the stepwise meteoscreening directly from the database introduced in the
previous update was further refined and extended, with additional outlier tests and corrections
implemented. The stepwise meteoscreening allows to perform step-by-step quality tests on
meteorological. A preview plot after running a test is shown and the user can decide if
results are satisfactory or if the same test with different parameters should be re-run.
Once results are satisfactory, the respective test flag is added to the data. After running
the desired tests, an overall flag `QCF` is calculated from all individual tests.

In addition to the creation of quality flags, the stepwise screening allows to correct
data for common issues. For example, short-wave radiation sensors often measure negative
values during the night. These negative values are useful because they give info about
the accuracy and precision of the sensor. In this case, values during the night should
be zero. Instead of cutting off negative values, `diive` detects the nighttime offset
for each day and then calculates a correction slope between individual days. This way,
the daytime values are also corrected.

After quality-screening and corrections, data are resampled to 30MIN time resolution.

At the moment, the stepwise meteoscreening works for data downloaded from the `InfluxDB`
database. The screening respects the database format (including tags) and prepares
the screened, corrected and resampled data for direct database upload.

Due to its modular approach, the stepwise screening can be easily adjusted
to work with any type of data files. This adjustment will be done in one of the next
updates.

Changes

- Renamed class `MetScrDbMeasurementVars`
to `StepwiseMeteoScreeningDb` (`pkgs.qaqc.meteoscreening.StepwiseMeteoScreeningDb`)

Additions

- **Stepwise MeteoScreening**:
Added access to multiple methods for easy stepwise execution:
- Added local SD outlier test (`pkgs.qaqc.meteoscreening.StepwiseMeteoScreeningDb.flag_outliers_localsd_test`)
- Added absolute limits outlier test (`pkgs.qaqc.meteoscreening.StepwiseMeteoScreeningDb.flag_outliers_abslim_test`)
- Added correction to remove radiation zero
offset (`pkgs.qaqc.meteoscreening.StepwiseMeteoScreeningDb.correction_remove_radiation_zero_offset`)
- Added correction to remove relative humidity
offset (`pkgs.qaqc.meteoscreening.StepwiseMeteoScreeningDb.correction_remove_relativehumidity_offset`)
- Added correction to set values above a threshold to
threshold (`pkgs.qaqc.meteoscreening.StepwiseMeteoScreeningDb.correction_setto_max_threshold`)
- Added correction to set values below a threshold to
threshold (`pkgs.qaqc.meteoscreening.StepwiseMeteoScreeningDb.correction_setto_min_threshold`)
- Added comparison plot before/after QC and
corrections (`pkgs.qaqc.meteoscreening.StepwiseMeteoScreeningDb.showplot_resampled`)

New Features

- **Stepwise MeteoScreening**: (`pkgs.qaqc.meteoscreening.MetScrDbMeasurementVars`)
- **Helper class to screen time series of meteo variables directly from the
database**. The class is optimized to work in Jupyter notebooks. Various outlier
detection methods can be called on-demand. Outlier results are displayed and
the user can accept the results and proceed, or repeat the step with adjusted
method parameters. An unlimited amount of tests can be chained together. At
the end of the screening, an overall flag is calculated from ALL single flags.
The overall flag is then used to filter the time series.
- **Variables**: The class allows the simultaneous quality-screening of multiple
variables from one single measurement, e.g., multiple air temperature variables.
- **Resampling**:Filtered time series are resampled to 30MIN time resolution.
- **Database tags**: Is optimized to work with the InfluxDB format of the ETH
Grassland Sciences Group. The class can handle database tags and updates tags
after data screening and resampling.
- **Handling different time resolutions**: One challenging aspect of the screening
were the different time resolutions of the raw data. In some cases, the time
resolution changed from e.g. 10MIN for older data to 1MIN for newer date. In
cases of different time resolution, the lower resolution is upsampled to the
higher resolution, the emerging gaps are back-filled with available data.
Back-filling is used because the timestamp in the database always is TIMESTAMP_END,
i.e., it gives the *end* of the averaging interval. The advantage of upsampling
is that all outlier detection routines can be applied to the whole dataset.
Since data are resampled to 30MIN after screening and since the TIMESTAMP_END
is respected, the upsampling itself has no impact on resulting aggregates.

Changes

- Generating the plot NEP penalty vs hours above threshold now requires a
minimum of 2 bootstrap runs to calculate prediction intervals
(`pkgs.flux.nep_penalty.NEPpenalty.plot_critical_hours`)

Bugfixes

- Fixed bug in `BinFitter`, the parameter to set the number of predictions is now correctly
named `n_predictions`. Similar `n_bins_x`.
- Fixed typos in functions `insert_aggregated_in_hires`, `SortingBinsMethod`, `FindOptimumRange`
and `pkgs.analyses.optimumrange.FindOptimumRange._values_in_optimum_range` and others.
- Other typos

New Features

- **USTAR threshold**: (`pkgs.flux.ustarthreshold.UstarThresholdConstantScenarios`)
- Calculates how many records of e.g. a flux variable are still available after the application
of different USTAR thresholds. In essence, it gives an overview of the sensitivity of the
variable to different thresholds.
- **Outlier detection, LOF across all data**: (`pkgs.outlierdetection.lof.LocalOutlierFactorAllData`)
- Calculation of the local outlier factor across all data, i.e., no differentiation between
daytime and nighttime data.
- **Outlier detection, increments**: (`pkgs.outlierdetection.incremental.zScoreIncremental`)
- Based on the absolute change of on record in comparison to the previous record. These
differences are stored as timeseries, the z-score is calculated and outliers are removed
based on the observed differences. Works well with data that do not have a diel cycle,
e.g. soil water content.