Diive

Bugfixes & changes

- Fixed: Replaced all references to old filetypes using the underscore to their respective new filetype names,
e.g. all occurrences of `EDDYPRO_FLUXNET_30MIN` were replaced with the new name `EDDYPRO-FLUXNET-CSV-30MIN`.
- Environment: Python 3.11 is now allowed in `pyproject.toml`: `python = ">=3.9,<3.12"`
- Environment: Removed `fitter` library from dependencies, was not used.
- Docs: Testing documentation generation using [Sphinx](https://www.sphinx-doc.org/en/master/), although it looks very
rough at the moment.

High-resolution update

This update focuses on the implementation of several classes that work with high-resolution (20 Hz) data.

The main motivation behind these implementations is the upcoming new version of another
script, [dyco](https://github.com/holukas/dyco), which will make direct use of these new classes. `dyco` allows
to detect and remove time lags from time series data and can also handle drifting lags, i.e., lags that
are not constant over time. This is especially useful for eddy covariance data, where the detection of
accurate time lags is of high importance for the calculation of ecosystem fluxes.

*Plot showing the covariance between the turbulent departures of vertical wind and CO2 measurements.
Maximum (absolute) covariance was found at record -26, which means that the CO2 signal has to be shifted
by 26 records in relation to the wind data to obtain the maximum covariance between the two variables.
Since the covariance was calculated on 20 Hz data, this corresponds to a time lag of 1.3 seconds
between CO2 and wind (20 Hz = measurement every 0.05 seconds, 26 * 0.05 = 1.3), or, to put it
another way, the CO2 signal arrived 1.3 seconds later at the sensor than the wind signal. Maximum
covariance was calculated using the `MaxCovariance` class.*

New features

- Added new class `MaxCovariance` to find the maximum covariance between two
variables (`diive.pkgs.echires.lag.MaxCovariance`)
- Added new class `FileDetector` to detect expected and unexpected files from a list of
files (`diive.core.io.filesdetector.FileDetector`)
- Added new class `FileSplitter` to split file into multiple smaller parts and export them as multiple CSV
files. (`diive.core.io.filesplitter.FileSplitter`)
- Added new class `FileSplitterMulti` to split multiple files into multiple smaller parts
and save them as CSV or compressed CSV files. (`diive.core.io.filesplitter.FileSplitterMulti`)
- Added new function `create_timestamp` that calculates the timestamp for each record in a dataframe,
based on number of records in the file and the file duration. (`diive.core.times.times.create_timestamp`)

Additions

- Added new filetype `ETH-SONICREAD-BICO-CSVGZ-20HZ`, these files contain data that were originally logged
by the `sonicread` script which is in use in the [ETH Grassland Sciences group](https://gl.ethz.ch/) since the early
2000s to record eddy covariance data within the [Swiss FluxNet](https://www.swissfluxnet.ethz.ch/). Data were
then converted to a regular format using the Python script [bico](https://github.com/holukas/bico), which
also compressed the resulting CSV files to `gz` files (`gzipped`).
- Added new filetype `GENERIC-CSV-HEADER-1ROW-TS-MIDDLE-FULL-NS-20HZ`, which corresponds to a CSV file with
one header row with variable names, a timestamp that describes the middle of the averaging period, whereby
the timestamp also includes nanoseconds. Time resolution of the file is 20 Hz.

Changes

- Renamed class `TurbFlux` to `WindRotation2D` and updated code a bit, e.g., now it is possible to get
rotated values for all three wind components (`u'`, `v'`, `w'`) in addition to the rotated
scalar `c'`. (`diive.pkgs.echires.windrotation.WindRotation2D`)
- Renamed filetypes: all filetypes now use the dash instead of an underscore
- Renamed filetype to `ETH-RECORD-DAT-20HZ`: this filetype originates from the new eddy covariance real-time
logging script `rECord` (currently not open source)
- Missing values are now defined for all files
as: `NA_VALUES: [ -9999, -6999, -999, "nan", "NaN", "NAN", "NA", "inf", "-inf", "-" ]`

- Updated (and cleaned) notebook `StepwiseMeteoScreeningFromDatabase.ipynb`

New features

- In `StepwiseOutlierDetection`, it is now possible to re-run an outlier detection method. The re-run(s)
would produce flag(s) with the same name(s) as for the first (original) run. Therefore, an integer is added
to the flag name. For example, if the test z-score daytime/nighttime is run the first time, it produces the
flag with the name `FLAG_TA_T1_2_1_OUTLIER_ZSCOREDTNT_TEST`. When the test is run again (e.g. with different
settings) then the name of the flag of this second run is `FLAG_TA_T1_2_1_OUTLIER_ZSCOREDTNT_2_TEST`,
etc ... The script now checks whether a flag of the same name was already created, in which case an
integer is added to the flag name. These re-runs are now available in addition to the `repeat=True` keyword.
(`diive.pkgs.outlierdetection.stepwiseoutlierdetection.StepwiseOutlierDetection.addflag`)
Example:
- `METHOD` with `SETTINGS` is applied with `repeat=True` and therefore repeated until no more outliers
were found with these settings. The name of the flag produced is `TEST_METHOD_FLAG`.
- Next, `METHOD` is applied again with `repeat=True`, but this time with different `SETTINGS`. Like before,
the test is repeated until no more outliers were found with the new settings. The name of the flag produced
is `TEST_METHOD_2_FLAG`.
- `METHOD` can be re-run any number of times, each time producing a new
flag: `TEST_METHOD_3_FLAG`, `TEST_METHOD_4_FLAG`, ...
- Added new function to format timestamps to FLUXNET ISO
format (`YYYYMMDDhhmm`) (`diive.core.times.times.format_timestamp_to_fluxnet_format`)

Bugfixes

- Refactored and fixed class to reformat data for FLUXNET
upload (`diive.pkgs.formats.fluxnet.FormatEddyProFluxnetFileForUpload`)
- Fixed `None` error when reading data files (`diive.core.io.filereader.DataFileReader._parse_file`)

Notebooks

- Updated notebook `FormatEddyProFluxnetFileForUpload.ipynb`

New features

- Added new functions to extract info from a binary that was stored as
integer. These functions convert a subrange of bits from an integer or an integer series to floats with an
optional gain applied. See docstring of the respective functions for more
info. (`diive.pkgs.binary.extract.get_encoded_value_from_int`) (`diive.pkgs.binary.extract.get_encoded_value_series`)
- Added new filetype `RECORD_DAT_20HZ` (`diive/configs/filetypes/RECORD_DAT_20HZ.yml`) for eddy covariance
high-resolution (20Hz) raw data files recorded by the ETH `rECord` logging script.

- Fixed bugs in `FluxProcessingChain`, flag creation for missing values did not work because of the missing `repeat`
keyword (`diive.pkgs.fluxprocessingchain.fluxprocessingchain.FluxProcessingChain`)