Tum-esm-utils

🦚  New Features

* **`opus.OpusHTTPInterface` class**
Communicate with OPUS using its HTTP interface (see [docs](https://tum-esm-utils.netlify.app/example-usage#opus-http-interface) and [API Reference](https://tum-esm-utils.netlify.app/api-reference#opushttpinterface-objects))

🐜  Internal Changes

* Fix bug when only reading the first channel from OPUS files
* Refine pytests: add test groups, make tests run when `.git` folder is not present
* Add repository to Zenodo

🦚  New Features

* **`opus` Module**
Read OPUS files. Credits to FriedrichKlappenbach for decoding the OPUS file structure.

🐜  Internal Changes

* Switch from YAPF to RUFF formatting (easier to configure and better maintained)
* Rename `polars` dependency group to `em27`. The old name will be kept as an alias until the next breaking release (`3.0.0`)

🦚  New Features

* **`timing.ExponentialBackoff`**
Add class to exponentially increase the waiting time after an error
[see in docs](https://tum-esm-utils.netlify.app/example-usage#exponential-backoff)

* **`StricterBaseModel`, `StrictIPv4Adress`**
More specialized Pydantic base classes
[see in docs](https://tum-esm-utils.netlify.app/api-reference#stricterbasemodel-objects)

* **`timing.date_span_intersection`, `timing.datetime_span_intersection`**
Functions to compute the overlap of two date/datetime periods
[see in docs](https://tum-esm-utils.netlify.app/api-reference#datetime_span_intersection)

🦀  Dependency Updates

* Explicitly add Python 3.13 support (tests, PyPI package)

🦚  New Features

* **`em27.PROFFAST_MULTIPLIERS`, `em27. PROFFAST_UNITS`, ...**
Add EM27/SUN constants for instrument colors, serial numbers, data units, etc.
[see in docs](https://tum-esm-utils.netlify.app/api-reference#tum_esm_utilsem27)

* **`plotting.add_colorpatch_legend`**
A function to add a "Colorpatch Legend" to a plot
[see in docs](https://tum-esm-utils.netlify.app/api-reference#add_colorpatch_legend)

* **`RandomLabelGenerator`**
Generate random names - like Docker container names `curious-einstein`, `wild-picard` and so on
[see in docs](https://tum-esm-utils.netlify.app/api-reference#randomlabelgenerator-objects)

* **`tum_esm_utils.timing.parse_iso_8601_datetime`**
Added because `datetime.datetime.fromisoformat` does not support all ISO 8601 timezone variants
[see in docs](https://tum-esm-utils.netlify.app/api-reference#parse_iso_8601_datetime)

* **`tum_esm_utils.files.list_directory`**
Adds more options the `os.listdir` method - only include directories/files, filter by regex, ignore by Unix shell patterns
[see in docs](https://tum-esm-utils.netlify.app/api-reference#list_directory)

* **`tum_esm_utils.files.render_directory_tree`**
Render a directory tree as a string
[see in docs](https://tum-esm-utils.netlify.app/api-reference#render_directory_tree)

* **`tum_esm_utils.text.simplify_string_characters`**
Simplify a string by replacing special characters with their ASCII counterparts and removing unwanted characters with. For example, simplify_string_characters("Héllo, wörld!") will return "hello-woerld".
[see in docs](https://tum-esm-utils.netlify.app/api-reference#simplify_string_characters)

* **`tum_esm_utils.text.replace_consecutive_characters`**
Remove consecutive whitespaces or other characters with `tum_esm_utils.text.replace_consecutive_characters`
[see in docs](https://tum-esm-utils.netlify.app/api-reference#replace_consecutive_characters)

🦀  Dependency Updates

* Upgrade minimum Python version to 3.10. This follows [SPEC 0 — Minimum Supported Dependencies](https://scientific-python.org/specs/spec-0000/) recommendations.

Upgrade Numpy (v1 to v2) and Polars (v0 to v1).

Add `Version` validator (19).

This release brings a significantly improved opus file validation based on Proffast 2.4. The old version could only handle a few reasons of file corruption. Furthermore the output did not list the files that the preprocessing could not even read.

**Old validator:**

python
detection_results = tum_esm_utils.em27.detect_corrupt_ifgs(
"/path/to/a/folder/with/interferograms"
)
assert detection_results == {
"md20220409s0e00a.0199": [
"charfilter 'GFW' is missing",
"charfilter 'GBW' is missing",
"charfilter 'HFL' is missing",
"charfilter 'LWN' is missing",
"charfilter 'TSC' is missing",
]
}


**New validator:**

python
detection_results = tum_esm_utils.em27.detect_corrupt_opus_files(
"/path/to/a/folder/with/interferograms"
)

detection_results == {
'md20220409s0e00a.0199': [
'Charfilter "DUR" is missing',
'Charfilter "GBW" is missing',
'Charfilter "GFW" is missing',
'Charfilter "HFL" is missing',
'Charfilter "LWN" is missing',
'Charfilter "TSC" is missing',
'Differing sizes of dual channel IFGs!',
'IFG size too small!',
'Inconsistent dualifg!',
'Inconsistent parameter kind in OPUS file!'
],
'comb_invparms_ma_SN061_210329-210329.csv': [
'File not even readible by the parser'
],
'comb_invparms_mc_SN115_220602-220602.csv': [
'File not even readible by the parser'
],
'md20220409s0e00a.0200': [
'File not even readible by the parser'
]
}


The function `tum_esm_utils.em27.detect_corrupt_ifgs` has been renamed to `tum_esm_utils.em27.detect_corrupt_opus_files` but is of course still available under the old name too (until the next breaking release).