Tum-esm-utils

🦚  New Features

* **`opus` Module**
Read OPUS files. Credits to FriedrichKlappenbach for decoding the OPUS file structure.

🐜  Internal Changes

* Switch from YAPF to RUFF formatting (easier to configure and better maintained)
* Rename `polars` dependency group to `em27`. The old name will be kept as an alias until the next breaking release (`3.0.0`)

🦚  New Features

* **`timing.ExponentialBackoff`**
Add class to exponentially increase the waiting time after an error
[see in docs](https://tum-esm-utils.netlify.app/example-usage#exponential-backoff)

* **`StricterBaseModel`, `StrictIPv4Adress`**
More specialized Pydantic base classes
[see in docs](https://tum-esm-utils.netlify.app/api-reference#stricterbasemodel-objects)

* **`timing.date_span_intersection`, `timing.datetime_span_intersection`**
Functions to compute the overlap of two date/datetime periods
[see in docs](https://tum-esm-utils.netlify.app/api-reference#datetime_span_intersection)

🦀  Dependency Updates

* Explicitly add Python 3.13 support (tests, PyPI package)

🦚  New Features

* **`em27.PROFFAST_MULTIPLIERS`, `em27. PROFFAST_UNITS`, ...**
Add EM27/SUN constants for instrument colors, serial numbers, data units, etc.
[see in docs](https://tum-esm-utils.netlify.app/api-reference#tum_esm_utilsem27)

* **`plotting.add_colorpatch_legend`**
A function to add a "Colorpatch Legend" to a plot
[see in docs](https://tum-esm-utils.netlify.app/api-reference#add_colorpatch_legend)

* **`RandomLabelGenerator`**
Generate random names - like Docker container names `curious-einstein`, `wild-picard` and so on
[see in docs](https://tum-esm-utils.netlify.app/api-reference#randomlabelgenerator-objects)

* **`tum_esm_utils.timing.parse_iso_8601_datetime`**
Added because `datetime.datetime.fromisoformat` does not support all ISO 8601 timezone variants
[see in docs](https://tum-esm-utils.netlify.app/api-reference#parse_iso_8601_datetime)

* **`tum_esm_utils.files.list_directory`**
Adds more options the `os.listdir` method - only include directories/files, filter by regex, ignore by Unix shell patterns
[see in docs](https://tum-esm-utils.netlify.app/api-reference#list_directory)

* **`tum_esm_utils.files.render_directory_tree`**
Render a directory tree as a string
[see in docs](https://tum-esm-utils.netlify.app/api-reference#render_directory_tree)

* **`tum_esm_utils.text.simplify_string_characters`**
Simplify a string by replacing special characters with their ASCII counterparts and removing unwanted characters with. For example, simplify_string_characters("Héllo, wörld!") will return "hello-woerld".
[see in docs](https://tum-esm-utils.netlify.app/api-reference#simplify_string_characters)

* **`tum_esm_utils.text.replace_consecutive_characters`**
Remove consecutive whitespaces or other characters with `tum_esm_utils.text.replace_consecutive_characters`
[see in docs](https://tum-esm-utils.netlify.app/api-reference#replace_consecutive_characters)

🦀  Dependency Updates

* Upgrade minimum Python version to 3.10. This follows [SPEC 0 — Minimum Supported Dependencies](https://scientific-python.org/specs/spec-0000/) recommendations.

Upgrade Numpy (v1 to v2) and Polars (v0 to v1).

Add `Version` validator (19).

This release brings a significantly improved opus file validation based on Proffast 2.4. The old version could only handle a few reasons of file corruption. Furthermore the output did not list the files that the preprocessing could not even read.

**Old validator:**

python
detection_results = tum_esm_utils.em27.detect_corrupt_ifgs(
"/path/to/a/folder/with/interferograms"
)
assert detection_results == {
"md20220409s0e00a.0199": [
"charfilter 'GFW' is missing",
"charfilter 'GBW' is missing",
"charfilter 'HFL' is missing",
"charfilter 'LWN' is missing",
"charfilter 'TSC' is missing",
]
}


**New validator:**

python
detection_results = tum_esm_utils.em27.detect_corrupt_opus_files(
"/path/to/a/folder/with/interferograms"
)

detection_results == {
'md20220409s0e00a.0199': [
'Charfilter "DUR" is missing',
'Charfilter "GBW" is missing',
'Charfilter "GFW" is missing',
'Charfilter "HFL" is missing',
'Charfilter "LWN" is missing',
'Charfilter "TSC" is missing',
'Differing sizes of dual channel IFGs!',
'IFG size too small!',
'Inconsistent dualifg!',
'Inconsistent parameter kind in OPUS file!'
],
'comb_invparms_ma_SN061_210329-210329.csv': [
'File not even readible by the parser'
],
'comb_invparms_mc_SN115_220602-220602.csv': [
'File not even readible by the parser'
],
'md20220409s0e00a.0200': [
'File not even readible by the parser'
]
}


The function `tum_esm_utils.em27.detect_corrupt_ifgs` has been renamed to `tum_esm_utils.em27.detect_corrupt_opus_files` but is of course still available under the old name too (until the next breaking release).

This breaking release is necessary to make the `polars` library an optional dependency. Until now, every codebase that wanted to use the `tum-esm-utils` library would have to use the same major release of `polars`.

The library now has two optional modules: `em27` and `plotting`, for which the dependencies (`polars` and `matplotlib` are not automatically installed). Install them using:

python
pip install "tum-esm-utils[polars]"
pip install "tum-esm-utils[plotting]"
pip install "tum-esm-utils[polars,plotting]"


Due to this change, the base library is kept at ~12MB, including all (sub-)dependencies, and only if you want the plotting and EM27 extras, this grows to ~174MB.

🦞  Breaking changes

_I included many breaking changes here, so there will not be a need for another breaking release soon._

* Module `interferograms` has been renamed `em27` and made optional
* Removed the `context` and `testing` modules: functionality has been moved into `timing` and `files`
* Renamed module `github` to `code`
* Removed unused functionality and changed some return types (see 18 for complete list)
* Dropped Python3.8 support because Matplotlib also dropped it
* Changed from MIT to GPL license

🦚  New features

* Added new optional `plotting` module
* Refactored function to load Proffast2 output files
* Added function to `code` module to load files from GitLab