Diive

New Features

- **Outlier detection**: LOF, local outlier factor**: (`pkgs.outlierdetection.lof.LocalOutlierFactorDaytimeNighttime`)
- Identify outliers based on the local outlier factor, done separately for
daytime and nighttime data
- **Multiple z-score outlier detections**:
- Simple outlier detection based on the z-score of observations, calculated from
mean and std from the complete timeseries. (`pkgs.outlierdetection.zscore.zScore`)
- z-score outlier detection separately for daytime and nighttime
data (`pkgs.outlierdetection.zscore.zScoreDaytimeNighttime`)
- Identify outliers based on the z-score of the interquartile range data (`pkgs.outlierdetection.zscore.zScoreIQR`)
- **Outlier detection**: (`pkgs.fluxprocessingchain.level32_outlierremoval.OutlierRemovalLevel32`):
- Class that allows to apply multiple methods for outlier detection during as part of the flux processing chain

Changes

- **Flux Processing Chain**:
- Worked on making the chain more accessible to users. The purpose of the modules in
`pkgs/fluxprocessingchain` is to expose functionality to the user, i.e., they make
functionality needed in the chain accessible to the user. This should be as easy as possible
and this update further simplified this access. At the moment there are three modules in
`pkgs/fluxprocessingchain/`: `level2_qualityflags.py`, `level31_storagecorrection.py` and
`level32_outlierremoval.py`. An example for the chain is given in `fluxprocessingchain.py`.
- **QCF flag**: (`pkgs.qaqc.qcf.FlagQCF`)
- Refactored code: the creation of overall quality flags `QCF` is now done using the same
code for flux and meteo data. The general logic of the `QCF` calculation is that results
from multiple quality checks that are stored as flags in the data are combined into
one single quality flag.
- **Outlier Removal using STL**:
- Module was renamed to `pkgs.outlierdetection.seasonaltrend.OutlierSTLRIQRZ`. It is not the
most convenient name, I know, but it stands for **S**easonal **T**rend decomposition using
**L**OESS, based on **R**esidual analysis of the **I**nter**Q**uartile **R**ange using **Z**-scores
- **Search files** can now search in subfolders of multiple base folders (`core.io.filereader.search_files`)

New Features

- **Outlier Removal using STL**: (`pkgs.outlierdetection.seasonaltrend.OutlierSTLIQR`)
- Implemented first code to remove outliers using seasonal-srend decomposition using LOESS.
This method divides a time series into seasonal, trend and residual components. `diive`
uses the residuals to detect outliers based on z-score calculations.
- **Overall quality flag for meteo data**: (`pkgs.qaqc.qcf.MeteoQCF`)
- Combines the results from multiple flags into one single flag
- Very similar to the calculation of the flux QCF flag

Changes

- **MeteoScreening**: (`diive/pkgs/qaqc/meteoscreening.py`)
- Refactored most of the code relating to the quality-screening of meteo data
- Implemented the calculation of the overall quality flag QCF
- Two overview figures are now created at the end on the screening
- Flags for tests used during screening are now created using a base class (`core.base.flagbase.FlagBase`)
- **Flux Processing Chain**: All modules relating to the Swiss FluxNet flux processing
chain are now collected in the dedicated package `fluxprocessingchain`. Relevant
modules were moved to this package, some renamed:
- `pkgs.fluxprocessingchain.level2_qualityflags.QualityFlagsLevel2`
- `pkgs.fluxprocessingchain.level31_storagecorrection.StorageCorrectionSinglePoint`
- `pkgs.fluxprocessingchain.qcf.QCF`
- **Reading YAML files**: (`core.io.filereader.ConfigFileReader`)
- Only filetype configuration files are validated, i.e. checked if they follow the
expected file structure. However, there can be other YAML files, such as the file
`pipes_meteo.yaml` that defines the QA/QC steps for each meteo variable. For the
moment, only the filetype files are validated and the validation is skipped for
the pipes file.
- Refactored calculation of nighttime flag from sun altitude: code is now vectorized
and runs - unsurprisingly - much faster (`pkgs.createvar.nighttime_latlon.nighttime_flag_from_latlon`)
- Some smaller changes relating to text output to the console

New Features

- **Flux storage correction**: (`pkgs.flux.storage.StorageCorrectionSinglePoint`)
- Calculate storage-corrected fluxes
- Creates Level-3.1 in the flux processing chain
- **Overall quality flag**: (`pkgs.qaqc.qcf.QCF`)
- Calculate overall quality flag from multiple individual flags

Changes

- **Flux quality-control**: (`pkgs.qaqc.fluxes.QualityFlagsLevel2`)
- Flags now have the string `_L2_` in their name to identify them as
flags created during Level-2 calculations in the Swiss FluxNet flux
processing chain.
- All flags can now be returned to the main data
- Renamed `pkgs.qaqc.fluxes.FluxQualityControlFlag` to `pkgs.qaqc.fluxes.QualityFlagsLevel2`

Changes

- **Flux quality-control**: (`pkgs.qaqc.fluxes.FluxQualityControlFlag`)
- Added heatmap plots for before/after QC comparison
- Improved code for calculation of overall flag `QCF`
- Improved console output

New Features

- **Flux quality-control**: (`pkgs.qaqc.fluxes.FluxQualityControlFlag`)
- First implementation of quality control of ecosystem fluxes. Generates one
overall flag (`QCF`=quality control flag) from multiple quality test results
in EddyPro's `fluxnet` output file. The resulting `QCF` is Level-2 in the
Swiss FluxNet processing chain,
described [here](https://www.swissfluxnet.ethz.ch/index.php/data/ecosystem-fluxes/flux-processing-chain/).
`QCF` is mostly based on the ICOS methodology, described
by [Sabbatini et al. (2018)](https://doi.org/10.1515/intag-2017-0043).
- **Histogram**: (`pkgs.analyses.histogram.Histogram`)
- Calculates histogram from time series, identifies peak distribution
- **Percentiles**: (`pkgs.analyses.quantiles.percentiles`)
- Calculates percentiles (0-100) for a time series
- **Scatter**: Implemented first version of `core.plotting.scatter.Scatter`, which will
be used for scatter plots in the future

Changes

- **Critical days**: (`pkgs.flux.criticaldays.CriticalDays`)
- Renamed Variables, now using Dcrit (instead of CRD) and nDcrit (instead of nCRD)
- **NEP Penalty**: (`pkgs.flux.nep_penalty.NEPpenalty`)
- Code was refactored to work with NEP (net ecosystem productivity) instead of NEE
(net ecosystem exchange)
- CO2 penalty was renamed to the more descriptive NEP penalty
- **Sanitize column names**: implemented in `core.io.filereader.ColumnNamesSanitizer`
Column names are now checked for duplicates. Found duplicates are renamed by adding a
suffix to the column name. Example: `co2_mean` and `co2_mean` are renamed to
`co2_mean.1` and `co2_mean.2`. This check is now implemented during the reading of
the data file in `core.io.filereader.DataFileReader`.
- **Configuration files**: When reading filetype configuration files in `core.io.filereader.ConfigFileReader`,
the resulting dictionary that contains all configurations is now validated. The validation makes
sure the parameters for `.read_csv()` are in the proper format.
- Updated all dependencies to their newest (possible) version

Additions

- Added support for filetype `EDDYPRO_FLUXNET_30MIN` (`configs/filetypes/EDDYPRO_FLUXNET_30MIN.yml`)

New Features

- **Frequency groups detection**: Data in long-term datasets are often characterized by changing time
resolutions at which data were recorded. `core.times.times.detect_freq_groups` detects changing
time resolutions in datasets and adds a group identifier in a new column that gives info about the
detected time resolution in seconds, e.g., `600` for 10MIN data records. This info allows to
address and process the different time resolutions separately during later processing, which is
needed e.g. during data quality-screening and resampling.
- **Outlier removal using z-score**: First version of `pkgs.outlierdetection.zscore.zscoreiqr`
Removes outliers based on the z-score of interquartile range data. Data are divided
into 8 groups based on quantiles. The z-score is calculated for each data point
in the respective group and based on the mean and SD of the respective group.
The z-score threshold to identify outlier data is calculated as the max of
z-scores found in IQR data multiplied by *factor*. z-scores above the threshold
are marked as outliers.
- **Outlier removal using local standard deviation**: First version of `pkgs.outlierdetection.local3sd.localsd`
Calculates mean and SD in a rolling window and marks data points outside a specified range.

Additions

- **MeteoScreening**: Added the new parameter `resampling_aggregation` in the meteoscreening setting
`diive/pkgs/qaqc/pipes_meteo.yaml`. For example, `TA` needs `mean`, `PRECIP` needs `sum`.

Changes

- **MeteoScreening**: `pkgs.qaqc.meteoscreening.MeteoScreeningFromDatabaseSingleVar`
Refactored the merging of quality-controlled 30MIN data when more than one raw data time
resolution is involved.
- **Resampling**: `core.times.resampling.resample_series_to_30MIN`
The minimum required values for resampling is `1`. However, this is only relevant for
lower resolution data e.g. 10MIN and 30MIN, because for higher resolutions the calculated value
for minimum required values yields values > 1 anyway. In addition, if data are already in
30MIN resolution, they are still going through the resampling processing although it would not
be necessary, because the processing includes other steps relevant to all data resolutions, such
as the change of the timestamp from TIMESTAMP_MIDDLE to TIMESTAMP_END.

Bugs

- Removed display bug when showing data after high-res meteoscreening in heatmap. Plot showed
original instead of meteoscreened data