Logprep

Breaking

* remove AutoRuleCorpusTester
* removes the option to use synchronous `bulk` or `parallel_bulk` operation in favor of `parallel_bulk` in `opensearch_output`
* reimplement error handling by introducing the option to configure an error output
* if no error output is configured, failed event will be dropped

Features

* adds health check endpoint to metrics on path `/health`
* changes helm chart to use new readiness check
* adds `healthcheck_timeout` option to all components to tweak the timeout of healthchecks
* adds `desired_cluster_status` option to opensearch output to signal healthy cluster status
* initially run health checks on setup for every configured component
* make `imagePullPolicy` configurable for helm chart deployments
* it is now possible to use Lucene compliant Filter Expressions
* make `terminationGracePeriodSeconds` configurable in helm chart values
* adds ability to configure error output
* adds option `default_op_type` to `opensearch_output` connector to set the default operation for indexing documents (default: index)
* adds option `max_chunk_bytes` to `opensearch_output` connector to set the maximum size of the request in bytes (default: 100MB)
* adds option `error_backlog_size` to logprep configuration to configure the queue size of the error queue
* the opensearch default index is now only used for processed events, errors will be written to the error output, if configured

Improvements

* remove AutoRuleCorpusTester
* adds support for rust extension development
* adds prebuilt wheels for architectures `x86_64` on `manylinux` and `musllinux` based linux platforms to releases
* add manual how to use local images with minikube example setup to documentation
* move `Configuration` to top level of documentation
* add `CONTRIBUTING` file
* sets the default for `flush_timeout` and `send_timeout` in `kafka_output` connector to `0` seconds
* changed python base image for logprep to `bitnami/python` in cause of better CVE governance

Bugfix

* ensure `logprep.abc.Component.Config` is immutable and can be applied multiple times
* remove lost callback reassign behavior from `kafka_input` connector
* remove manual commit option from `kafka_input` connector
* pin `mysql-connector-python` to >=9.1.0 to accommodate for CVE-2024-21272 and update `MySQLConnector` to work with the new version

Bugfix

* fixes a bug not increasing but decreasing timeout throttle factor of ThrottlingQueue
* handle DecodeError and unexpected Exceptions on requests in `http_input` separately
* fixes unbound local error in http input connector

Improvements

* adds ability to bypass the processing of events if there is no pipeline. This is useful for pure connector deployments.
* adds experimental feature to bypass the rule tree by setting `LOGPREP_BYPASS_RULE_TREE` environment variable

Bugfix

* fixes a bug in the `http_output` used by the http generator, where the timeout parameter does only set the read_timeout not the write_timeout
* fixes a bug in the `http_input` not handling decode errors

Features

* `pre_detector` now normalizes timestamps with configurable parameters timestamp_field, source_format, source_timezone and target_timezone
* `pre_detector` now writes tags in failure cases
* `ProcessingWarnings` now can write `tags` to the event
* add `timeout` parameter to logprep http generator to set the timeout in seconds for requests
* add primitive rate limiting to `http_input` connector

Improvements

* switch to `uvloop` as default loop for the used threaded http uvicorn server
* switch to `httptools` as default http implementation for the used threaded http uvicorn server

Bugfix

* remove redundant chart features for mounting secrets

Improvements

* a result object was added to processors and pipelines
* each processor returns an object including the processor name, generated extra_data, warnings
and errors
* the pipeline returns an object with the list of all processor result objects
* add kubernetes opensiem deployment example
* move quickstart setup to compose example

Bugfix

* This release limits the mysql-connector-python dependency to have version less the 9

Breaking

* This release limits the maximum python version to `3.12.3` because of the issue
[612](https://github.com/fkie-cad/Logprep/issues/612).
* Remove `normalizer` processor, as it's functionality was replaced by the `grokker`, `timestamper` and `field_manager` processors
* Remove `elasticsearch_output` connector to reduce maintenance effort

Features

* add a helm chart to install logprep in kubernetes based environments

Improvements

* add documentation about behavior of the `timestamper` on `ISO8601` and `UNIX` time parsing
* add unit tests for helm chart templates
* add helm to github actions runner
* add helm chart release to release pipeline

Bugfix

* fixes a bug where it could happen that a config value could be overwritten by a default in a later configuration in a multi source config scenario
* fixes a bug in the `field_manager` where extending a non list target leads to a processing failure
* fixes a bug in `pseudonymizer` where a missing regex_mapping from an existing config_file causes logprep to crash continuously