Diive

New Features

- **Stepwise MeteoScreening**: (`pkgs.qaqc.meteoscreening.MetScrDbMeasurementVars`)
- **Helper class to screen time series of meteo variables directly from the
database**. The class is optimized to work in Jupyter notebooks. Various outlier
detection methods can be called on-demand. Outlier results are displayed and
the user can accept the results and proceed, or repeat the step with adjusted
method parameters. An unlimited amount of tests can be chained together. At
the end of the screening, an overall flag is calculated from ALL single flags.
The overall flag is then used to filter the time series.
- **Variables**: The class allows the simultaneous quality-screening of multiple
variables from one single measurement, e.g., multiple air temperature variables.
- **Resampling**:Filtered time series are resampled to 30MIN time resolution.
- **Database tags**: Is optimized to work with the InfluxDB format of the ETH
Grassland Sciences Group. The class can handle database tags and updates tags
after data screening and resampling.
- **Handling different time resolutions**: One challenging aspect of the screening
were the different time resolutions of the raw data. In some cases, the time
resolution changed from e.g. 10MIN for older data to 1MIN for newer date. In
cases of different time resolution, the lower resolution is upsampled to the
higher resolution, the emerging gaps are back-filled with available data.
Back-filling is used because the timestamp in the database always is TIMESTAMP_END,
i.e., it gives the *end* of the averaging interval. The advantage of upsampling
is that all outlier detection routines can be applied to the whole dataset.
Since data are resampled to 30MIN after screening and since the TIMESTAMP_END
is respected, the upsampling itself has no impact on resulting aggregates.

Changes

- Generating the plot NEP penalty vs hours above threshold now requires a
minimum of 2 bootstrap runs to calculate prediction intervals
(`pkgs.flux.nep_penalty.NEPpenalty.plot_critical_hours`)

Bugfixes

- Fixed bug in `BinFitter`, the parameter to set the number of predictions is now correctly
named `n_predictions`. Similar `n_bins_x`.
- Fixed typos in functions `insert_aggregated_in_hires`, `SortingBinsMethod`, `FindOptimumRange`
and `pkgs.analyses.optimumrange.FindOptimumRange._values_in_optimum_range` and others.
- Other typos

New Features

- **USTAR threshold**: (`pkgs.flux.ustarthreshold.UstarThresholdConstantScenarios`)
- Calculates how many records of e.g. a flux variable are still available after the application
of different USTAR thresholds. In essence, it gives an overview of the sensitivity of the
variable to different thresholds.
- **Outlier detection, LOF across all data**: (`pkgs.outlierdetection.lof.LocalOutlierFactorAllData`)
- Calculation of the local outlier factor across all data, i.e., no differentiation between
daytime and nighttime data.
- **Outlier detection, increments**: (`pkgs.outlierdetection.incremental.zScoreIncremental`)
- Based on the absolute change of on record in comparison to the previous record. These
differences are stored as timeseries, the z-score is calculated and outliers are removed
based on the observed differences. Works well with data that do not have a diel cycle,
e.g. soil water content.


New Features

- **Outlier detection**: LOF, local outlier factor**: (`pkgs.outlierdetection.lof.LocalOutlierFactorDaytimeNighttime`)
- Identify outliers based on the local outlier factor, done separately for
daytime and nighttime data
- **Multiple z-score outlier detections**:
- Simple outlier detection based on the z-score of observations, calculated from
mean and std from the complete timeseries. (`pkgs.outlierdetection.zscore.zScore`)
- z-score outlier detection separately for daytime and nighttime
data (`pkgs.outlierdetection.zscore.zScoreDaytimeNighttime`)
- Identify outliers based on the z-score of the interquartile range data (`pkgs.outlierdetection.zscore.zScoreIQR`)
- **Outlier detection**: (`pkgs.fluxprocessingchain.level32_outlierremoval.OutlierRemovalLevel32`):
- Class that allows to apply multiple methods for outlier detection during as part of the flux processing chain

Changes

- **Flux Processing Chain**:
- Worked on making the chain more accessible to users. The purpose of the modules in
`pkgs/fluxprocessingchain` is to expose functionality to the user, i.e., they make
functionality needed in the chain accessible to the user. This should be as easy as possible
and this update further simplified this access. At the moment there are three modules in
`pkgs/fluxprocessingchain/`: `level2_qualityflags.py`, `level31_storagecorrection.py` and
`level32_outlierremoval.py`. An example for the chain is given in `fluxprocessingchain.py`.
- **QCF flag**: (`pkgs.qaqc.qcf.FlagQCF`)
- Refactored code: the creation of overall quality flags `QCF` is now done using the same
code for flux and meteo data. The general logic of the `QCF` calculation is that results
from multiple quality checks that are stored as flags in the data are combined into
one single quality flag.
- **Outlier Removal using STL**:
- Module was renamed to `pkgs.outlierdetection.seasonaltrend.OutlierSTLRIQRZ`. It is not the
most convenient name, I know, but it stands for **S**easonal **T**rend decomposition using
**L**OESS, based on **R**esidual analysis of the **I**nter**Q**uartile **R**ange using **Z**-scores
- **Search files** can now search in subfolders of multiple base folders (`core.io.filereader.search_files`)

New Features

- **Outlier Removal using STL**: (`pkgs.outlierdetection.seasonaltrend.OutlierSTLIQR`)
- Implemented first code to remove outliers using seasonal-srend decomposition using LOESS.
This method divides a time series into seasonal, trend and residual components. `diive`
uses the residuals to detect outliers based on z-score calculations.
- **Overall quality flag for meteo data**: (`pkgs.qaqc.qcf.MeteoQCF`)
- Combines the results from multiple flags into one single flag
- Very similar to the calculation of the flux QCF flag

Changes

- **MeteoScreening**: (`diive/pkgs/qaqc/meteoscreening.py`)
- Refactored most of the code relating to the quality-screening of meteo data
- Implemented the calculation of the overall quality flag QCF
- Two overview figures are now created at the end on the screening
- Flags for tests used during screening are now created using a base class (`core.base.flagbase.FlagBase`)
- **Flux Processing Chain**: All modules relating to the Swiss FluxNet flux processing
chain are now collected in the dedicated package `fluxprocessingchain`. Relevant
modules were moved to this package, some renamed:
- `pkgs.fluxprocessingchain.level2_qualityflags.QualityFlagsLevel2`
- `pkgs.fluxprocessingchain.level31_storagecorrection.StorageCorrectionSinglePoint`
- `pkgs.fluxprocessingchain.qcf.QCF`
- **Reading YAML files**: (`core.io.filereader.ConfigFileReader`)
- Only filetype configuration files are validated, i.e. checked if they follow the
expected file structure. However, there can be other YAML files, such as the file
`pipes_meteo.yaml` that defines the QA/QC steps for each meteo variable. For the
moment, only the filetype files are validated and the validation is skipped for
the pipes file.
- Refactored calculation of nighttime flag from sun altitude: code is now vectorized
and runs - unsurprisingly - much faster (`pkgs.createvar.nighttime_latlon.nighttime_flag_from_latlon`)
- Some smaller changes relating to text output to the console

New Features

- **Flux storage correction**: (`pkgs.flux.storage.StorageCorrectionSinglePoint`)
- Calculate storage-corrected fluxes
- Creates Level-3.1 in the flux processing chain
- **Overall quality flag**: (`pkgs.qaqc.qcf.QCF`)
- Calculate overall quality flag from multiple individual flags

Changes

- **Flux quality-control**: (`pkgs.qaqc.fluxes.QualityFlagsLevel2`)
- Flags now have the string `_L2_` in their name to identify them as
flags created during Level-2 calculations in the Swiss FluxNet flux
processing chain.
- All flags can now be returned to the main data
- Renamed `pkgs.qaqc.fluxes.FluxQualityControlFlag` to `pkgs.qaqc.fluxes.QualityFlagsLevel2`

Changes

- **Flux quality-control**: (`pkgs.qaqc.fluxes.FluxQualityControlFlag`)
- Added heatmap plots for before/after QC comparison
- Improved code for calculation of overall flag `QCF`
- Improved console output