Logprep

Improvements

* Bump `requests` to `>=2.31.0` to circumvent `CVE-2023-32681`
* Include a lucene representation of the rule filter into the predetector results. The
representation is not completely lucene compatible due to non-existing regex functionality.

Bugfix

* Fix error handling of FieldManager if no mapped source field exists in the event.
* Fix Grokker such that only the first grok pattern match is applied instead of all matching
pattern
* Fix Grokker such that nested parentheses in oniguruma pattern are working (3 levels are supported
now)
* Fix Grokker such that two or more oniguruma can point to the same target. This ensures
grok-pattern compatibility with the normalizer and other grok tools

Features

* Extend dissector such that it can trim characters around dissected field with `%{field-( )}`
notation.
* Extend timestamper such that it can take multiple source_formats. First format that matches
will be used, all following formats will be ignored

Improvements

* Extend the `FieldManager` such that it can move/copy multiple source fields into multiple targets
inside one rule.

Bugfix

* Fix error handling of missing source fields in grokker
* Fix using same output fields in list of grok pattern in grokker

Features
* add `timestamper` processor to extract timestamp functionality from normalizer

Improvements
* removed `arrow` dependency and depending features for performance reasons
* switched to `datetime.strftime` syntax in `timestamp_differ`, `s3_output`, `elasticsearch_output` and `opensearch_output`
* encapsulate time related functionality in `logprep.util.time.TimeParser`


Bugfix
* Fix missing default grok patterns in packaged logprep version

Features

* Add `amides` processor to extends conventional rule matching by applying machine learning components
* Add `grokker` processor to extract grok functionality from normalizer
* `Normalizer` writes failure tags if nomalization fails
* Add `flush_timeout` to `opensearch` and `elasticsearch` outputs to ensure message delivery within a configurable period
* add `kafka_config` option to `confluent_kafka_input` and `confluent_kafka_output` connectors to provide additional config options to `librdkafka`

Improvements

* Harmonize error messages and handling for processors and connectors
* Add ability to schedule periodic tasks to all components
* Improve performance of pipeline processing by switching form builtin `json` to `msgspec` in pipeline and kafka connectors
* Rewrite quickstart setup:
* Remove logstash, replace elasticsearch by opensearch and use logprep opensearch connector to stick to reference architecture
* Use kafka without zookeeper and switch to bitnami container images

Bugfix

* Fix resetting processor caches in the `auto_rule_corpus_tester` by initializing all processors
between test cases.
* Fix processing of generic rules after there was an error inside the specific rules.
* Remove coordinate fields from results of the geoip enricher if one of them has `None` values

Breaking

* Remove rules deprecations introduced in `v4.0.0`
* Changes rule language of `selective_extractor`, `pseudonymizer`, `pre_detector` to support multiple outputs

Features

* Add `string_splitter` processor to split strings of variable length into lists
* Add `ip_informer` processor to enrich events with ip information
* Allow running the `Pipeline` in python without input/output connectors
* Add `auto_rule_corpus_tester` to test a whole rule corpus against defined expected outputs.
* Add shorthand for converting datatypes to `dissector` dissect pattern language
* Add support for multiple output connectors
* Apply processors multiple times until no new rule matches anymore. This enables applying rules on
results of previous rules.

Improvements

* Bump `attrs` to `>=22.2.0` and delete redundant `min_len_validator`
* Specify the metric labels for connectors (add name, type and direction as labels)
* Rename metric names to clarify their meanings (`logprep_pipeline_number_of_warnings` to
`logprep_pipeline_sum_of_processor_warnings` and `logprep_pipeline_number_of_errors` to
`logprep_pipeline_sum_of_processor_errors`)

Bugfix

* Fixes a bug that breaks templating config and rule files with environment variables if one or more variables are not set in environment
* Fixes a bug for `opensearch_output` and `elasticsearch_output` not handling authentication issues
* Fix metric `logprep_pipeline_number_of_processed_events` to actually count the processed events per pipeline
* Fix a bug for enrichment with environment variables. Variables must have one of the following prefixes now: `LOGPREP_`, `CI_`, `GITHUB_` or `PYTEST_`

Improvements

* reimplements the `selective_extractor`

Breaking

* drop support for python `3.6`, `3.7`, `3.8`
* change default prefix behavior on appending to strings of `dissector`

Features

* Add an `http input connector` that spawns a uvicorn server which parses requests content to events.
* Add an `file input connector` that reads generic logfiles.
* Provide the possibility to consume lists, rules and configuration from files and http endpoints
* Add `requester` processor that enriches by making http requests with field values
* Add `calculator` processor to calculate with or without field values
* Make output subfields of the `geoip_enricher` configurable by introducing the rule config
`customize_target_subfields`
* Add a `timestamp_differ` processor that can parse two timestamps and calculate their respective time delta.
* Add `config_refresh_interval` configuration option to refresh the configuration on a given timedelta
* Add option to `dissector` to use a prefix pattern in dissect language for appending to strings and add the default behavior to append to strings without any prefixed separator

Improvements

* Add support for python `3.10` and `3.11`
* Add option to submit a template with `list_search_base_path` config parameter in `list_comparison` processor
* Add functionality to `geoip_enricher` to download the geoip-database
* Add ability to use environment variables in rules and config
* Add list access including slicing to dotted field notation for getting values
* Add processor boilerplate generator to help adding new processors

Bugfixes

* Fix count of `number_of_processed_events` metric in `input` connector. Will now only count actual
events.