Tum-esm-utils

This release brings a significantly improved opus file validation based on Proffast 2.4. The old version could only handle a few reasons of file corruption. Furthermore the output did not list the files that the preprocessing could not even read.

**Old validator:**

python
detection_results = tum_esm_utils.em27.detect_corrupt_ifgs(
"/path/to/a/folder/with/interferograms"
)
assert detection_results == {
"md20220409s0e00a.0199": [
"charfilter 'GFW' is missing",
"charfilter 'GBW' is missing",
"charfilter 'HFL' is missing",
"charfilter 'LWN' is missing",
"charfilter 'TSC' is missing",
]
}


**New validator:**

python
detection_results = tum_esm_utils.em27.detect_corrupt_opus_files(
"/path/to/a/folder/with/interferograms"
)

detection_results == {
'md20220409s0e00a.0199': [
'Charfilter "DUR" is missing',
'Charfilter "GBW" is missing',
'Charfilter "GFW" is missing',
'Charfilter "HFL" is missing',
'Charfilter "LWN" is missing',
'Charfilter "TSC" is missing',
'Differing sizes of dual channel IFGs!',
'IFG size too small!',
'Inconsistent dualifg!',
'Inconsistent parameter kind in OPUS file!'
],
'comb_invparms_ma_SN061_210329-210329.csv': [
'File not even readible by the parser'
],
'comb_invparms_mc_SN115_220602-220602.csv': [
'File not even readible by the parser'
],
'md20220409s0e00a.0200': [
'File not even readible by the parser'
]
}


The function `tum_esm_utils.em27.detect_corrupt_ifgs` has been renamed to `tum_esm_utils.em27.detect_corrupt_opus_files` but is of course still available under the old name too (until the next breaking release).

This breaking release is necessary to make the `polars` library an optional dependency. Until now, every codebase that wanted to use the `tum-esm-utils` library would have to use the same major release of `polars`.

The library now has two optional modules: `em27` and `plotting`, for which the dependencies (`polars` and `matplotlib` are not automatically installed). Install them using:

python
pip install "tum-esm-utils[polars]"
pip install "tum-esm-utils[plotting]"
pip install "tum-esm-utils[polars,plotting]"


Due to this change, the base library is kept at ~12MB, including all (sub-)dependencies, and only if you want the plotting and EM27 extras, this grows to ~174MB.

🦞  Breaking changes

_I included many breaking changes here, so there will not be a need for another breaking release soon._

* Module `interferograms` has been renamed `em27` and made optional
* Removed the `context` and `testing` modules: functionality has been moved into `timing` and `files`
* Renamed module `github` to `code`
* Removed unused functionality and changed some return types (see 18 for complete list)
* Dropped Python3.8 support because Matplotlib also dropped it
* Changed from MIT to GPL license

🦚  New features

* Added new optional `plotting` module
* Refactored function to load Proffast2 output files
* Added function to `code` module to load files from GitLab

🦚  New Features

* Add a new method `tum_esm_utils.files.read_last_n_lines` ([docs](https://tum-esm-utils.netlify.app/api-reference#read_last_n_lines))
* Add a new method `tum_esm_utils.timing.parse_timezone_string` ([docs](https://tum-esm-utils.netlify.app/api-reference#parse_timezone_string))

🦚  New Features

* Add a new method `tum_esm_utils.timing.date_range`: https://tum-esm-utils.netlify.app/api-reference#date_range-1

🦀  Dependency Updates

* Update `polars0.19` to `0.20`

🐜  Internal Changes

* Copy all the functions in the `context` module (all time related) to the new `timing` module
* Add deprecation warnings with the link to the new `timing` module to all migrated functions in the `context` module
* Use `timing.date_range` inside `text.date_range` (the latter is based on date strings like `YYYYMMDD`)

🦚  New Features

* Add a new method to strictly validate paths using Pydantic root models: https://tum-esm-utils.netlify.app/example-usage#strict-path-validation-with-pydantic
* Add a new very clean method to convert paths relative to the script into absolute files: https://tum-esm-utils.netlify.app/example-usage#get-absolute-paths
* Add a function to set alarms in order to catch infinite loops in your code: https://tum-esm-utils.netlify.app/example-usage#set-alarms-to-catch-infinite-loops

All of the above is tested, statically typed and documented of course 🐝

We moved from [Docsify](https://docsify.js.org/) to [Nextra](https://nextra.site/) to render the actual docs page. We moved from [`lazydocs`](https://github.com/ml-tooling/lazydocs) to [`pydoc-markdown`](https://github.com/NiklasRosenstein/pydoc-markdown) to automatically generate the markdown API reference from the codebase

Feel free to use this documentation setup in your own projects to improve the usability of the content you produce 🌈

**API reference - Old | New - now all on one scrollable page**

<img alt="api-reference-old" src="https://github.com/tum-esm/utils/assets/29046316/40e5e64d-22f4-42bb-9b7f-c02e218a92f8" width=500/> <img alt="api-reference-new" src="https://github.com/tum-esm/utils/assets/29046316/77718bf2-ca04-4f5a-96d1-efd841e70cd6" width=500/>

**Examples - Old | New - more and improved usage examples**

<img alt="examples-old" src="https://github.com/tum-esm/utils/assets/29046316/ef8d14c4-2bc6-411e-93f9-4637090313eb" width=500/> <img alt="examples-new" src="https://github.com/tum-esm/utils/assets/29046316/ab92db55-d0c0-484a-ad86-95d269d05b21" width=500>

**Search - Old | New - a search that is actually useful**

<img alt="search-old" src="https://github.com/tum-esm/utils/assets/29046316/d9e2da1c-5eba-4e25-8586-ca59fb57a7d4" width=500/> <img alt="search-new" src="https://github.com/tum-esm/utils/assets/29046316/67e834f3-be62-4f63-8afb-be688e81bb1c" width=500/>