Logprep

Features

* `pre_detector` now normalizes timestamps with configurable parameters timestamp_field, source_format, source_timezone and target_timezone
* `pre_detector` now writes tags in failure cases
* `ProcessingWarnings` now can write `tags` to the event
* add `timeout` parameter to logprep http generator to set the timeout in seconds for requests
* add primitive rate limiting to `http_input` connector

Improvements

* switch to `uvloop` as default loop for the used threaded http uvicorn server
* switch to `httptools` as default http implementation for the used threaded http uvicorn server

Bugfix

* remove redundant chart features for mounting secrets

Improvements

* a result object was added to processors and pipelines
* each processor returns an object including the processor name, generated extra_data, warnings
and errors
* the pipeline returns an object with the list of all processor result objects
* add kubernetes opensiem deployment example
* move quickstart setup to compose example

Bugfix

* This release limits the mysql-connector-python dependency to have version less the 9

Breaking

* This release limits the maximum python version to `3.12.3` because of the issue
[612](https://github.com/fkie-cad/Logprep/issues/612).
* Remove `normalizer` processor, as it's functionality was replaced by the `grokker`, `timestamper` and `field_manager` processors
* Remove `elasticsearch_output` connector to reduce maintenance effort

Features

* add a helm chart to install logprep in kubernetes based environments

Improvements

* add documentation about behavior of the `timestamper` on `ISO8601` and `UNIX` time parsing
* add unit tests for helm chart templates
* add helm to github actions runner
* add helm chart release to release pipeline

Bugfix

* fixes a bug where it could happen that a config value could be overwritten by a default in a later configuration in a multi source config scenario
* fixes a bug in the `field_manager` where extending a non list target leads to a processing failure
* fixes a bug in `pseudonymizer` where a missing regex_mapping from an existing config_file causes logprep to crash continuously

Breaking

* `pseudonymizer` change rule config field `pseudonyms` to `mapping`
* `clusterer` change rule config field `target` to `source_fields`
* `generic_resolver` change rule config field `append_to_list` to `extend_target_list`
* `hyperscan_resolver` change rule config field `append_to_list` to `extend_target_list`
* `calculator` now adds the error tag `_calculator_missing_field_warning` to the events tag field instead of `_calculator_failure` in case of missing field in events
* `domain_label_extractor` now writes `_domain_label_extractor_missing_field_warning` tag to event tags in case of missing fields
* `geoip_enricher` now writes `_geoip_enricher_missing_field_warning` tag to event tags in case of missing fields
* `grokker` now writes `_grokker_missing_field_warning` tag to event tags instead of `_grokker_failure` in case of missing fields
* `requester` now writes `_requester_missing_field_warning` tag to event tags instead of `_requester_failure` in case of missing fields
* `timestamp_differ` now writes `_timestamp_differ_missing_field_warning` tag to event tags instead of `_timestamp_differ_failure` in case of missing fields
* `timestamper` now writes `_timestamper_missing_field_warning` tag to event tags instead of `_timestamper_failure` in case of missing fields
* rename `--thread_count` parameter to `--thread-count` in http generator
* removed `--report` parameter and feature from http generator
* when using `extend_target_list` in the `field manager`the ordering of the given source fields is now preserved
* logprep now exits with a negative exit code if pipeline restart fails 5 times
* this was implemented because further restart behavior should be configured on level of a system init service or container orchestrating service like k8s
* the `restart_count` parameter is configurable. If you want the old behavior back, you can set this parameter to a negative number
* logprep now exits with a exit code of 2 on configuration errors

Features

* add UCL into the quickstart setup
* add logprep http output connector
* add pseudonymization tools to logprep -> see: `logprep pseudo --help`
* add `restart_count` parameter to configuration
* add option `mode` to `pseudonymizer` processor and to pseudonymization tools to chose the AES Mode for encryption and decryption
* add retry mechanism to opensearch parallel bulk, if opensearch returns 429 `rejected_execution_exception`

Improvements

* remove logger from Components and Factory signatures
* align processor architecture to use methods like `write_to_target`, `add_field_to` and `get_dotted_field_value` when reading and writing from and to events
* required substantial refactoring of the `hyperscan_resolver`, `generic_resolver` and `template_replacer`
* change `pseudonymizer`, `pre_detector`, `selective_extractor` processors and `pipeline` to handle `extra_data` the same way
* refactor `clusterer`, `pre_detector` and `pseudonymizer` processors and change `rule_tree` so that the processor do not require `process` override
* required substantial refactoring of the `clusterer`
* handle missing fields in processors via `_handle_missing_fields` from the field_manager
* add `LogprepMPQueueListener` to outsource logging to a separate process
* add a single `Queuehandler` to root logger to ensure all logs were handled by `LogprepMPQueueListener`
* refactor `http_generator` to use a logprep http output connector
* ensure all `cached_properties` are populated during setup time

Bugfix

* make `--username` and `--password` parameters optional in http generator
* fixes a bug where `FileNotFoundError` is raised during processing

Features

* add gzip handling to `http_input` connector
* adds advanced logging configuration
* add configurable log format
* add configurable datetime formate in logs
* makes `hostname` available in custom log formats
* add fine grained log level configuration for every logger instance

Improvements

* rename `logprep.event_generator` module to `logprep.generator`
* shorten logger instance names

Bugfix

* fixes exposing OpenSearch/ElasticSearch stacktraces in log when errors happen by making loglevel configurable for loggers `opensearch` and `elasticsearch`
* fixes the logprep quickstart profile

Bugfix

* fixes bug, that leads to spawning exporter http server always on localhost