Logprep

Breaking
Features

* add multiarch container builds for AMD64 and ARM64

Improvements
Bugfix

Breaking

* drop support for python 3.10 and add support for python 3.13
* `CriticalInputError` is raised when the input preprocessor values can't be set, this was so far only true
for the hmac preprocessor, but is now also applied for all other preprocessors.
* fix `delimiter` typo in `StringSplitterRule` configuration
* removed the configuration `tld_lists` in `domain_resolver`, `domain_label_extractor` and `pseudonymizer` as
the list is now fixed inside the packaged logprep
* remove SQL feature from `generic_adder`, fields can only be added from rule config or from file
* use a single rule tree instead of a generic and a specific rule tree
* replace the `extend_target_list` parameter with `merge_with_target` for improved naming clarity
and functionality across `FieldManager` based processors (e.g., `FieldManager`, `Clusterer`,
`GenericAdder`).

Features

* configuration of `initContainers` in logprep helm chart is now possible

Improvements

* fix `requester` documentation
* replace `BaseException` with `Exception` for custom errors
* refactor `generic_resolver` to validate rules on startup instead of application of each rule
* regex pattern lists for the `generic_resolver` are pre-compiled
* regex matching from lists in the `generic_resolver` is cached
* matching in the `generic_resolver` can be case-insensitive
* rewrite the helper method `add_field_to` such that it always raises an `FieldExistsWarning` instead of return a bool.
* add new helper method `add_fields_to` to directly add multiple fields to one event
* refactored some processors to make use of the new helper methods
* add `pre-commit` hooks to the repository, install new dev dependency and run `pre-commit install` in the root dir
* the default `securityContext`for the pod is now configurable
* allow `TimeParser` to get the current time with a specified timezone instead of always using local time and setting the timezone to UTC
* remove `tldextract` dependency
* remove `urlextract` dependency
* fix wrong documentation for `timestamp_differ`
* add container signatures to images build in ci pipeline
* add sbom to images build in ci pipeline
* `FieldManager` supports merging dictionaries

Bugfix

* fix `confluent_kafka.store_offsets` if `last_valid_record` is `None`, can happen if a rebalancing happens
before the first message was pulled.
* fix pseudonymizer cache metrics not updated
* fix incorrect timezones for log arrival time and delta time in input preprocessing
* fix `_get_value` in `FilterExpression` so that keys don't match on values
* fix `auto_rule_tester` to work with `LOGPREP_BYPASS_RULE_TREE` enabled
* fix `opensearch_output` not draining `message_backlog` on shutdown
* silence `FieldExists` warning in metrics when `LOGPREP_APPEND_MEASUREMENT_TO_EVENT` is active

Breaking

* remove AutoRuleCorpusTester
* removes the option to use synchronous `bulk` or `parallel_bulk` operation in favor of `parallel_bulk` in `opensearch_output`
* reimplement error handling by introducing the option to configure an error output
* if no error output is configured, failed event will be dropped

Features

* adds health check endpoint to metrics on path `/health`
* changes helm chart to use new readiness check
* adds `healthcheck_timeout` option to all components to tweak the timeout of healthchecks
* adds `desired_cluster_status` option to opensearch output to signal healthy cluster status
* initially run health checks on setup for every configured component
* make `imagePullPolicy` configurable for helm chart deployments
* it is now possible to use Lucene compliant Filter Expressions
* make `terminationGracePeriodSeconds` configurable in helm chart values
* adds ability to configure error output
* adds option `default_op_type` to `opensearch_output` connector to set the default operation for indexing documents (default: index)
* adds option `max_chunk_bytes` to `opensearch_output` connector to set the maximum size of the request in bytes (default: 100MB)
* adds option `error_backlog_size` to logprep configuration to configure the queue size of the error queue
* the opensearch default index is now only used for processed events, errors will be written to the error output, if configured

Improvements

* remove AutoRuleCorpusTester
* adds support for rust extension development
* adds prebuilt wheels for architectures `x86_64` on `manylinux` and `musllinux` based linux platforms to releases
* add manual how to use local images with minikube example setup to documentation
* move `Configuration` to top level of documentation
* add `CONTRIBUTING` file
* sets the default for `flush_timeout` and `send_timeout` in `kafka_output` connector to `0` seconds
* changed python base image for logprep to `bitnami/python` in cause of better CVE governance

Bugfix

* ensure `logprep.abc.Component.Config` is immutable and can be applied multiple times
* remove lost callback reassign behavior from `kafka_input` connector
* remove manual commit option from `kafka_input` connector
* pin `mysql-connector-python` to >=9.1.0 to accommodate for CVE-2024-21272 and update `MySQLConnector` to work with the new version

Bugfix

* fixes a bug not increasing but decreasing timeout throttle factor of ThrottlingQueue
* handle DecodeError and unexpected Exceptions on requests in `http_input` separately
* fixes unbound local error in http input connector

Improvements

* adds ability to bypass the processing of events if there is no pipeline. This is useful for pure connector deployments.
* adds experimental feature to bypass the rule tree by setting `LOGPREP_BYPASS_RULE_TREE` environment variable

Bugfix

* fixes a bug in the `http_output` used by the http generator, where the timeout parameter does only set the read_timeout not the write_timeout
* fixes a bug in the `http_input` not handling decode errors

Features

* `pre_detector` now normalizes timestamps with configurable parameters timestamp_field, source_format, source_timezone and target_timezone
* `pre_detector` now writes tags in failure cases
* `ProcessingWarnings` now can write `tags` to the event
* add `timeout` parameter to logprep http generator to set the timeout in seconds for requests
* add primitive rate limiting to `http_input` connector

Improvements

* switch to `uvloop` as default loop for the used threaded http uvicorn server
* switch to `httptools` as default http implementation for the used threaded http uvicorn server

Bugfix

* remove redundant chart features for mounting secrets