Intelmq

------------------

Core
- `intelmq.lib.utils.drop_privileges`: When IntelMQ is called as `root` and dropping the privileges to user `intelmq`, also set the non-primary groups associated with the `intelmq` user. Makes the behaviour of running intelmqctl as `root` closer to the behaviour of `sudo -u intelmq ...` (PR2507 by Mikk Margus Möll).
- `intelmq.lib.utils.unzip`: Ignore directories themselves when extracting data to prevent the extraction of empty data for a directory entries (PR2512 by Kamil Mankowski).

Bots
Collectors
- `intelmq.bots.collectors.shadowserver.collector_reports_api.py`:
- Added support for the types parameter to be either a string or a list (PR2495 by elsif2).
- Refactored to utilize the type field returned by the API to match the requested types instead of a sub-string match on the filename.
- Fixed timezone issue for collecting reports (PR2506 by elsif2).
- Fixed behaviour if parameter `reports` value is empty string, behave the same way as not set, not like no report (PR2523 by Sebastian Wagner).
- `intelmq.bots.collectors.shodan.collector_stream` (PR2492 by Mikk Margus Möll):
- Add `alert` parameter to Shodan stream collector to allow fetching streams by configured alert ID
- `intelmq.bots.collectors.mail._lib`: Remove deprecated parameter `attach_unzip` from default parameters (PR2511 by Sebastian Wagner).

Parsers
- `intelmq.bots.parsers.shadowserver._config`:
- Fetch schema before first run (PR2482 by elsif2, fixes 2480).
- `intelmq.bots.parsers.dataplane.parser`: Use ` | ` as field delimiter, fix parsing of AS names including `|` (PR2488 by DigitalTrustCenter).
- all parsers: add `copy_collector_provided_fields` parameter allowing copying additional fields from the report, e.g. `extra.file_name`.
(PR2513 by Kamil Mankowski).

Experts
- `intelmq.bots.experts.sieve.expert`:
- For `:contains`, `=~` and `!~`, convert the value to string before matching avoiding an exception. If the value is a dict, convert the value to JSON (PR2500 by Sebastian Wagner).
- Add support for variables in Sieve scripts (PR2514 by Mikk Margus Möll, fixes 2486).
- `intelmq.bots.experts.filter.expert`:
- Treat value `false` for parameter `filter_regex` as false (PR2499 by Sebastian Wagner).

Outputs
- `intelmq.bots.outputs.misp.output_feed`: Handle failures if saved current event wasn't saved or is incorrect (PR by Kamil Mankowski).
- `intelmq.bots.outputs.smtp_batch.output`: Documentation on multiple recipients added (PR2501 by Edvard Rejthar).

Documentation
- Bots: Clarify some section of Mail collectors and the Generic CSV Parser (PR2510 by Sebastian Wagner).

Known Issues
This is short list of the most important known issues. The full list can be retrieved from [GitHub](https://github.com/certtools/intelmq/labels/bug?page=2&q=is%3Aopen+label%3Abug).
- `intelmq.parsers.html_table` may not process invalid URLs in patched Python version due to changes in `urllib` (2382).
- Breaking changes in 'rt' 3.0 library (2367).
- Type error with SQL output bot's `prepare_values` returning list instead of tuple (2255).
- `intelmq_psql_initdb` does not work for SQLite (2202).
- intelmqsetup: should install a default state file (2175).
- Misp Expert - Crash if misp event already exist (2170).
- Spamhaus CERT parser uses wrong field (2165).
- Custom headers ignored in HTTPCollectorBot (2150).
- intelmqctl log: parsing syslog does not work (2097).
- Bash completion scripts depend on old JSON-based configuration files (2094).
- Bots started with IntelMQ-API/Manager stop when the webserver is restarted (952).
- Corrupt dump files when interrupted during writing (870).

------------------

Configuration
- Add new optional configuration parameters for `intelmq.bots.collectors.stomp.collector`
and `intelmq.bots.outputs.stomp.output` (PR2408 by Jan Kaliszewski):
- `auth_by_ssl_client_certificate` (Boolean, default: *true*; if *false* then
`ssl_client_certificate` and `ssl_client_certificate_key` will be ignored);
- `username` (STOMP authentication login, default: "guest"; to be used only
if `auth_by_ssl_client_certificate` is *false*);
- `password` (STOMP authentication passcode, default: "guest"; to be used only
if `auth_by_ssl_client_certificate` is *false*).
- Add the possibility to set the `ssl_ca_certificate` configuration parameter for
`intelmq.bots.collectors.stomp.collector` and/or `intelmq.bots.outputs.stomp.output`
to an empty string - which means that the SSL machinery used for STOMP communication
will attempt to load the system’s default CA certificates (PR2414 by Jan Kaliszewski).

Core
- `intelmq.lib.message`: For invalid message keys, add a hint on the failure to the exception: not allowed by configuration or not matching regular expression (PR2398 by Sebastian Wagner).
- `intelmq.lib.exceptions.InvalidKey`: Add optional parameter `additional_text` (PR2398 by Sebastian Wagner).
- Change the way we discover bots to allow easy extending based on the entry point name. (PR2413 by Kamil Mankowski)
- `intelmq.lib.mixins`: Add a new class, `StompMixin` (defined in a new submodule: `stomp`),
which provides certain common STOMP-bot-specific operations, factored out from
`intelmq.bots.collectors.stomp.collector` and `intelmq.bots.outputs.stomp.output`
(PR2408 and PR2414 by Jan Kaliszewski).
- `intelmq.lib.upgrades`: Replace deprecated instances of `url2fqdn` experts by the new `url` expert in runtime configuration (PR2432 by Sebastian Wagner).
- `intelmq.lib.bot`: Ensure closing log files on reloading (PR2435 by Kamil Mankowski).
- AMQP Pipeline: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski).
- Only load the config once when starting intelmqctl (which makes IntelMQ API calls take less time) (PR2444 by DigitalTrustCenter).

Development
- Makefile: Add codespell and test commands (PR2425 by Sebastian Wagner).

Data Format

Bots
Collectors
- `intelmq.bots.collectors.stomp.collector` (PR2408 and PR2414 by Jan Kaliszewski):
- Drop support for versions of `stomp.py` older than `4.1.12`.
- Update the code to support new versions of `stomp.py`, including the latest (`8.1.0`);
fixes [2342](https://github.com/certtools/intelmq/issues/2342).
- Add support for authentication based on STOMP login and passcode, introducing three
new configuration parameters (see above: *Configuration*).
- Add support for loading the system’s default CA certificates, as an alternative to
specifying the CA certificate(s) file path explicitly (see above: *Configuration*).
- Fix (by carefully targeted monkey patching) certain security problems caused by
SSL-related weaknesses that some versions of `stomp.py` suffer from.
- Fix the reconnection behavior: do not attempt to reconnect after `shutdown`. Also,
never attempt to reconnect if the version of `stomp.py` is older than `4.1.21` (it
did not work properly anyway).
- Add coercion of the `port` config parameter to `int`.
- Add implementation of the `check` hook (verifying, in particular, accessibility
of necessary file(s)).
- Remove undocumented and unused attributes of `StompCollectorBot` instances:
`ssl_ca_cert`, `ssl_cl_cert`, `ssl_cl_cert_key`.
- Minor fixes/improvements and some refactoring (see also above: *Core*...).
- `intelmq.bots.collectors.amqp`: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski).
- `intelmq.bots.collectors.shadowserver.collector_reports_api`:
- The 'json' option is no longer supported as the 'csv' option provides better performance (PR2372 by elsif2).
- `intelmq.bots.collectors.alienvault_otx.collector` (PR2449 by qux-bbb):
- Fix modified_pulses_only is always False.


Parsers
- `intelmq.bots.parsers.netlab_360.parser`: Removed as the feed is discontinued. (2442 by Filip Pokorný)
- `intelmq.bots.parsers.webinspektor.parser`: Removed as the feed is discontinued. (2442 by Filip Pokorný)
- `intelmq.bots.parsers.sucuri.parser`: Removed as the feed is discontinued. (2442 by Filip Pokorný)
- `intelmq.bots.parsers.shadowserver._config`:
- Switch to dynamic configuration to decouple report schema changes from IntelMQ releases by regularly downloading them from the Shadowserver server (PR2372 by elsif2).
- `intelmq.bots.parsers.cymru`: Save current line. (PR by Kamil Mankowski)

Experts
- `intelmq.bots.experts.jinja` (PR2417 by Mikk Margus Möll):
- Add optional `socket_perms` and `socket_group` parameters to change
file permissions on socket file, if it is in use.
- `intelmq.bots.experts.ripe` (PR2461 by Mikk Margus Möll):
- Handle "No abuse contact found for" messages for non-ASN resources

Outputs
- `intelmq.bots.outputs.stomp.output` (PR2408 and PR2414 by Jan Kaliszewski):
- Drop support for versions of `stomp.py` older than `4.1.12`.
- Update the code to support new versions of `stomp.py`, including the latest (`8.1.0`).
- Add support for authentication based on STOMP login and passcode, introducing three
new configuration parameters (see above: *Configuration*).
- Add support for loading the system’s default CA certificates, as an alternative to
specifying the CA certificate(s) file path explicitly (see above: *Configuration*).
- Fix (by carefully targeted monkey patching) certain security problems caused by
SSL-related weaknesses that some versions of `stomp.py` suffer from.
- Fix `AttributeError` caused by attempts to get unset attributes of `StompOutputBot`
(`ssl_ca_cert` et consortes).
- Add coercion of the `port` config parameter to `int`.
- Add implementation of the `check` hook (verifying, in particular, accessibility
of necessary file(s)).
- Add `stomp.py` version check (raise `MissingDependencyError` if not `>=4.1.12`).
- Minor fixes/improvements and some refactoring (see also above: *Core*...).
- `intelmq.bots.outputs.stomp.output` (PR2423 by Kamil Mankowski):
- Try to reconnect on `NotConnectedException`.
- `intelmq.bots.outputs.smtp_batch.output` (PR 2439 by Edvard Rejthar):
- Fix ability to send with the default `bcc`
- `intelmq.bots.outputs.amqp`: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski).

Documentation
- Add a readthedocs configuration file to fix the build fail (PR2403 by Sebastian Wagner).
- Add a guide of developing extensions packages (PR2413 by Kamil Mankowski)
- Update/fix/improve the stuff related to the STOMP bots and integration with the *n6*'s
Stream API (PR2408 and PR2414 by Jan Kaliszewski).
- Complete documentation overhaul. Change to markdown format. Uses the mkdocs-material (PR2419 by Filip Pokorný).
- Adds warning banner if not browsing the latest version of the docs (PR2445 by Filip Pokorný).
- Fix logo path in index.md when building the docs (PR2445 by Filip Pokorný).

Packaging
- Add `pendulum` to suggested packages, as it is required for the sieve bot (PR2424 by Sebastian Wagner).
- `debian/control`: in `Suggests` field, replace ``python3-stomp.py (>= 4.1.9)`` with
``python3-stomp (>= 4.1.12)``, i.e., fix the package name by removing the `.py`
suffix and bump the minimum version to `4.1.12` (PR2414 by Jan Kaliszewski).

Tests

Tools
- `intelmq_psql_initdb`:
- got support for providing custom harmonization file, generating view for storing `raw` fields separately, and adding `IF NOT EXISTS`/`OR REPLACE` clauses ([PR2404](https://github.com/certtools/intelmq/pull/2404) by Kamil Mankowski).
- got support for generating JSONB fields for PostgreSQL schema (PR2436 by Kamil Mankowski).

------------------

Core
- Fixed issue preventing bots from stopping after reloading (PR by Kamil Mankowski).

Bots
Experts
- `intelmq.bots.experts.reverse_dns.expert`:
- Fix the cache key to not cache results for /24 (IPv4) and /128 (IPv6) networks but for single IP-Adresses (PR2395 by Sebastian Wagner, fixes 2394).

------------------

Core
- `intelmq.lib.utils`:
- `resolve_dns`: Deprecate dnspython versions pre-2.0.0 and disable search domains (PR2352)
- Fixed not resetting destination path statistics in the stats cache after restarting bot (Fixes [2331](https://github.com/certtools/intelmq/issues/2331))
- Force flushing statistics if bot will sleep longer than flushing delay (Fixes [2336](https://github.com/certtools/intelmq/issues/2336))
- `intelmq.lib.upgrages`: Fix a bug in the upgrade function for version 3.1.0 which caused an exception if a generic csv parser instance had no parameter `type` (PR2319 by Filip Pokorný).
- `intelmq.lib.datatypes`: Adds `TimeFormat` class to be used for the `time_format` bot parameter (PR2329 by Filip Pokorný).
- `intelmq.lib.exceptions`: Fixes a bug in `InvalidArgument` exception (PR2329 by Filip Pokorný).
- `intelmq.lib.harmonization`:
- Changes signature and names of `DateTime` conversion functions for consistency, backwards compatible (PR2329 by Filip Pokorný).
- Ensure rejecting URLs with leading whitespaces after changes in CPython (fixes [2377](https://github.com/certtools/intelmq/issues/2377))
- `intelmq.lib.bot.Bot`: Allow setting the parameters via parameter on bot initialization.

Development
- CI: pin the Codespell version to omit troubles caused by its new releases (PR 2379).
- CI: Updated the versions of the github actions in the CI workflows. (PR2392 by Sebastian Kufner)

Bots

Collectors
- `intelmq.bots.collector.rt`:
- restrict `python-rt` to be below version 3.0 due to introduced breaking changes,
- added support for `Subject NOT LIKE` queries,
- added support for multiple values in ticket subject queries.
- `intelmq.bots.collectors.rsync`: Support for optional private key, relative time parsing for the source path, extra rsync parameters and strict host key checking (PR2241 by Mateo Durante).

Parsers
- `intelmq.bots.parsers.shadowserver._config`:
- Reset detected `feedname` at shutdown to re-detect the feedname on reloads (PR2361 by elsif2, fixes 2360).
- Switch to dynamic configuration to decouple report schema changes from IntelMQ releases.
- Added 'IPv6-Vulnerable-Exchange' alias and 'Accessible-WS-Discovery-Service' report. (PR2338)
- Removed unused `p0f_genre` and `p0f_detail` from the 'DNS-Open-Resolvers' report. (PR2338)
- Added 'Accessible-SIP' report. (PR2348)
- Added 'IPv6-Open-HTTP-Proxy' and 'IPv6-Accessible-HTTP-Proxy' aliases. (PR2348)
- Removed duplicate mappings from the 'Spam-URL' report. (PR2348)
- `intelmq.bots.parsers.generic.parser_csv`: Changes `time_format` parameter to use new `TimeFormat` class (PR2329 by Filip Pokorný).
- `intelmq.bots.parsers.html_table.parser`: Changes `time_format` parameter to use new `TimeFormat` class (PR2329 by Filip Pokorný).
- `intelmq.bots.parsers.turris.parser.py` Updated to the latest data format (issue 2167). (PR2373 by Filip Pokorný).

Experts
- `intelmq.bots.experts.sieve`:
- Allow empty lists in sieve rule files (PR2341 by Mikk Margus Möll).
- `intelmq.bots.experts.cymru_whois`:
- Ignore AS names with unexpected unicode characters (PR2352, fixes 2132)
- Avoid extraneous search domain-based queries on NXDOMAIN result (PR2352)
- `intelmq.bots.experts.sieve`:
- Added :before and :after keywords (PR2374)

Outputs
- `intelmq.bots.outputs.cif3.output`: Added (PR2244 by Michael Davis).
- `intelmq.bots.outputs.sql.output`: New parameter `fail_on_errors` (PR2362 by Sebastian Wagner).
- `intelmq.bots.outputs.smtp_batch.output`: Added a bot to gathering the events and sending them by e-mails at a stroke as CSV files (PR2253 by Edvard Rejthar)

Documentation
- API: update API installation to be aligned with the rewritten API, and clarify some missing steps.

Tests
- New decorator `skip_installation` and environment variable `INTELMQ_TEST_INSTALLATION` to skip tests requiring an IntelMQ installation on the test host by default (PR2370 by Sebastian Wagner, fixes 2369)

Tools
- `intelmqsetup`:
- SECURITY: fixed a low-risk bug causing the tool to change owner of `/` if run with the `INTELMQ_PATHS_NO_OPT` environment variable set. This affects only the PIP package as the DEB/RPM packages don't contain this tool. (PR2355 by Kamil Mańkowski, fixes 2354)
- `contrib.eventdb.separate-raws-table.sql`: Added the missing commas to complete the sql syntax. (PR2386, fixes 2125 by Sebastian Kufner)
- `intelmq_psql_initdb`:
- Added parameter `-o` to set the output file destination. (by Sebastian Kufner)
- `intelmqctl`:
- Increased the performance through removing unnecessary reads. (by Sebastian Kufner)

Known Issues
This is short list of the most important known issues. The full list can be retrieved from [GitHub](https://github.com/certtools/intelmq/labels/bug?page=2&q=is%3Aopen+label%3Abug).
- `intelmq.parsers.html_table` may not process invalid URLs in patched Python version due to changes in `urllib` (2382).
- Breaking changes in 'rt' library (2367).
- Stomp collector failed (2342).
- Type error with SQL output bot's `prepare_values` returning list instead of tuple (2255).
- `intelmq_psql_initdb` does not work for SQLite (2202).
- intelmqsetup: should install a default state file (2175).
- Misp Expert - Crash if misp event already exist (2170).
- Turris greylist has been updated (2167).
- Spamhaus CERT parser uses wrong field (2165).
- Custom headers ignored in HTTPCollectorBot (2150).
- intelmqctl log: parsing syslog does not work (2097).
- Bash completion scripts depend on old JSON-based configuration files (2094).
- Bot configuration examples use JSON instead of YAML (2066).
- Bots started with IntelMQ-API/Manager stop when the webserver is restarted (952).
- Corrupt dump files when interrupted during writing (870).

------------------

- Upgraded syntax to Python 3.6 (mostly Format-Strings) using pyuprade (PR2136 by Sebastian Wagner).

Core
- `intelmq.lib.upgrades`:
- Refactor upgrade functions global configuration handling removing the old-style defaults configuration (PR2058 by Sebastian Wagner).
- Pass version history as parameter to upgrade functions (PR2058 by Sebastian Wagner).
- `intelmq.lib.message`:
- Fix and pre-compile the regular expression for harmonization key names and also check keys in the `extra.` namespace (PR2059 by Sebastian Wagner, fixes 1807).
- `intelmq.lib.bot.SQLBot` was replaced by an SQLMixin in `intelmq.lib.mixins.SQLMixin`. The Generic DB Lookup Expert bot and the SQLOutput bot were updated accordingly.
- Added support for MSSQL (PR2171 by Karl-Johan Karlsson).
- Added optional reconnect delay parameter (PR2171 by Karl-Johan Karlsson).
- Added an ExpertBot class - it should be used by all expert bots as a parent class
- Introduced a module for IntelMQ related datatypes `intelmq.lib.datatypes` which for now only contains an Enum listing the four bot types
- Added a `bottype` attribute to CollectorBot, ParserBot, ExpertBot, OutputBot
- Introduces a module for IntelMQ processmanagers. The processmanagers were up until now part of the intelmqct script.
They now reside in `intelmq.lib.processmanager` which also contains an interface definition the processmanager implementations must adhere to.
Both the processmanagers and the `intelmqctl` script were cleaned up a bit.
The `LogLevel` and `ReturnType` Enums were added to `intelmq.lib.datatypes`.
- `intelmq.lib.bot`:
- Enhance behaviour if an unconfigured bot is started (PR2054 by Sebastian Wagner).
- Fix line recovery and message dumping of the `ParserBot` (PR2192 by Sebastian Wagner).
- Previously the dumped message was always the last message of a report if the report contained multiple lines leading to data-loss.
- Fix crashing at start in multithreaded bots (PR2236 by DigitalTrustCenter).
- Added `default_fields` parameter to `ParserBot` (PR2293 by Filip Pokorný)
- `intelmq.lib.pipeline`:
- Changed `BRPOPLPUSH` to `BLMOVE`, because `BRPOPLPUSH` has been marked as deprecated by redis in favor of `BLMOVE` (PR2149 and PR2240 by Sebastian Waldbauer and Sebastian Wagner, fixes 1827, 2233).
- `intelmq.lib.utils`:
- Added wrapper `resolve_dns` for querying DNS, with the support for recommended methods from `dnspython` package in versions 1 and 2.
- Moved line filtering inside `RewindableFileHandle` for easier handling and limiting number of temporary objects.
- `intelmq.lib.harmonization`:
- Fixed DateTime handling of naive time strings (previously assumed local timezone, now assumes UTC) (PR2279 by Filip Pokorný, fixes 2278)
- Removes `tzone` argument from `DateTime.from_timestamp` and `DateTime.from_epoch_millis`
- `DateTime.from_timstamp` now also allows string argument
- Removes `pytz` global dependency
- Removed support for Python 3.6, including removing conditional dependencies and updating syntax to use features from newest versions. (fixes [2272](https://github.com/certtools/intelmq/issues/2272))

Development
- Removed Python 3.6 from CI.
- Enabled tests against Python 3.11.

Bots
- Set the parent class of all bots to the correct bot class

Collectors
- `intelmq.bots.collectors.mail._lib`:
- Add support for unverified SSL/STARTTLS connections (PR2055 by Sebastian Wagner).
- Fix exception handling for aborted IMAP connections (PR2187 by Sebastian Wagner).
- `intelmq.bots.collectors.blueliv`: Fix Blueliv collector requirements (PR2161 by Gethvi).
- `intelmq.bots.collectors.github_api._collector_github_api`: Added personal access token support (PR2145 by Sebastian Waldbauer, fixes 1549).
- `intelmq.bots.collectors.file.collector_file`: Added file lock support, no more race conditions (PR2147 by Sebastian Waldbauer, fixes 2128)
- `intelmq.bots.collectors.shadowserver.collector_reports_api.py`: Added file_format option to download reports in CSV format for better performance (PR2246 by elsif2)

Parsers
- `intelmq.bots.parsers.alienvault.parser_otx`: Save CVE data in `extra.cve` instead of `extra.CVE` due to the field name restriction on lower-case characters (PR2059 by Sebastian Wagner).
- `intelmq.bots.parsers.anubisnetworks.parser`: Changed field name format from `extra.communication.http.x_forwarded_for_1` to `extra.communication.http.x_forwarded_for_1` due to the field name restriction on alphanumeric characters (PR2059 by Sebastian Wagner).
- `intelmq.bots.parsers.dataplane.parser`:
- Add support for additional feeds (PR2102 by Mikk Margus Möll).
- DNS Recursion Desired
- DNS Recursion Desired ANY
- DNS Version
- Protocol 41
- SMTP Greet
- SMTP Data
- Telnet Login
- VNC/RFB Login
- Fix event object creation (PR2298 by DigitalTrustCenter).
- Removed `intelmq.bots.parsers.malc0de`: this bot was marked as deprecated and removed from feed due to offline status (PR2184 by Tamas Gutsohn, fixes 2178).
- `intelmq.bots.parsers.microsoft.parser_ctip`:
- New parameter `overwrite` (PR2112 by Sebastian Wagner, fixes 2022).
- Fix handling of field `Payload.domain` if it contains the same IP address as `Payload.serverIp` (PR2144 by Mikk Margus Möll and Sebastian Wagner).
- Handle Payload field with non-base64-encoded JSON content and numbered dictionaries (PR2193 by Sebastian Wagner)
- `intelmq.bots.parsers.shodan.parser` (PR2117 by Mikk Margus Möll):
- Instead of keeping track of `extra.ftp.<something>.parameters`, FTP parameters are collected together into `extra.ftp.features` as a list of said features, reducing field count.
- Shodan field `rsync.modules` is collected.
- Conversion functions can raise `NoValueException` with a string argument to signify that the conversion would not succeed, such as in the case of a single IP address being given in hostnames, which would then be passed into `source.reverse_dns and` fail to validate as a FQDN.
- Variable `_common_keys` is moved out of the class.
- `_dict_dict_to_obj_list` is introduced, for converting a string-to-dict mapping into a list of dicts with the previous key as an attribute of the dict; this can be useful for preventing issues where, when feeding the data into aggregating tools, you'd end up with many more fields than necessary, e.g `vulns.CVE-2010-0001.cvss`, `CVE-2010-0002.cvss` etc.
- `_get_first` to get the first item from a list, with `NoValueException` raised on empty lists.
- `_get_first_hostname` to handle the first valid FQDN from a list of hostnames for hostnames in the Shodan banner, if there is one, and gives `NoValueException` otherwise.
- `ssl.cert.serial` and `ssl.dhparams.generator`, which may return both integers and strings, are converted to strings.
- Changes to method `apply_mapping`, such as reducing needless loop iterations, removing a big try-except, and adding the `NoValueException` handling described above.
- Stops falsy values (False, 0) besides None from being filtered out.
- `intelmq.bots.parsers.shadowserver._config`:
- Added support for `Accessible AMQP`, `Device Identification Report` (IPv4 and IPv6) (PR2134 by Mateo Durante).
- Added file name mapping for `SSL-POODLE-Vulnerable-Servers IPv6` (file name `scan6_ssl_poodle`) (PR2134 by Mateo Durante).
- Added `Malware-URL`, `Sandbox-Connection`, `Sandbox-DNS`, `Accessible-AMQP`, `Open-AnonymouIs-MQTT`, `Accessible-QUIC`, `Accessible-SSH`, `SYNful-Knock`, and `Special` (PR2227 by elsif2)
- Removed legacy reports `Amplification-DDoS-Victim`, `CAIDA-IP-Spoofer`, `Darknet`, `Drone`, `Drone-Brute-Force`, `IPv6-Sinkhole-HTTP-Drone`, `Microsoft-Sinkhole`, and `Sinkhole-HTTP-Drone` (PR2227 by elsif2).
- Users storing events in a database should be aware that field names and types have been updated (PR2227 by elsif2).
- Corrected "Accessible-AMQP" message_length type (int) and added "STUN" support (PR2235 by elsif2).
- Added amplification factor to UDP scan reports (PR2238 by elsif2).
- Added version and build_date to "Vulnerable-HTTP" report (PR2238 by elsif2).
- The following field types have been standardized across all Shadowserver reports (PR2246 by elsif2):
destination.fqdn (validate_fqdn)
destination.url (convert_http_host_and_url)
extra.browser_trusted (convert_bool)
extra.duration (convert_int)
extra.end_time (convert_date_utc)
extra.freak_vulnerable (convert_bool)
extra.ok (convert_bool)
extra.password (validate_to_none)
extra.ssl_poodle (convert_bool)
extra.status (convert_int)
extra.uptime (convert_int)
extra.version (convert_to_none)
source.network (validate_network)
- The following report field names have changed to better represent their values:
scan_rsync:extra.password renamed to extra.has_password
scan_elasticsearch:status renamed to http_code
- Added `Accessible-HTTP-proxy` and `Open-HTTP-proxy` (PR2246 by elsif2).
- Added http_agent to the `Honeypot-DDoS` report and added the `DDoS-Participant` report (PR2303 by elsif2)
- Added `Accessible-SLP`, `IPv6 Accesssible-SLP`, `IPv6-DNS-Open-Resolvers`, and `IPv6-Open-LDAP-TCP` reports (PR2311 by elsif2)
- Standardized response_length to response_size in `Accessible-ICS` and `Open-MSSQL` (PR2311 by elsif2)

- `intelmq.bots.parsers.cymru.parser_cap_program`: The parser mapped the hostname into `source.fqdn` which is not allowed by the IntelMQ Data Format. Added a check (PR2215 by Sebastian Waldbauer, fixes 2169)
- `intelmq.bots.parsers.generic.parser_csv`:
- Use RewindableFileHandle to use the original current line for line recovery (PR2192 by Sebastian Wagner).
- Recovering CSV lines preserves the original line ending (PR2280 by Kamil Mankowski, fixes [1597](https://github.com/certtools/intelmq/issues/1597))
- `intelmq.bots.parsers.autoshun.parser`: Removed, as the feed is discontinued (PR2214 by Sebastian Waldbauer, fixes 2162).
- `intelmq.bots.parsers.openphish.parser_commercial`: Refactored complete code (PR2160 by Filip Pokorný).
- Fixes wrong mapping of `host` field to `source.fqdn` when the content was an IP address.
- Adds newly added fields in the feed.
- `intelmq.bots.parsers.phishtank.parser`: Refactored code (PR2270 by Filip Pokorný)
- Changes feed URL to JSON format (contains more information). The URL needs to by manually updated in the configuration!
- Adds fields from the JSON feed.
- `intelmq.bots.parsers.dshield.parser_domain`: Has been removed, due to the feed is discontinued. (PR2276 by Sebastian Waldbauer)
- `intelmq.bots.parsers.abusech.parser_ip`: Removed (PR2268 by Filip Pokorný).
- `intelmq.bots.parsers.abusech.parser_domain`: Removed (PR2268 by Filip Pokorný).
- `intelmq.bots.parsers.abusech.parser_feodotracker`: Added new parser bot (PR2268 by Filip Pokorný)
- Changes feed URL to JSON format (contains more information).
- Adds fields from the JSON feed.
- `intelmq.bots.parsers.generic.parser_csv`: Parameter `type` is deprecated, `default_fields` should be used. (PR2293 by Filip Pokorný)
- `intelmq.bots.parsers.generic.parser_csv`: Parameter `skip_header` now allows also integer as a fixed number of lines to skip. (PR2313 by Filip Pokorný)
- `intelmq.bots.parsers.taichung.parser`: Removed (PR2266 by Filip Pokorný)

Experts
- `intelmq.bots.experts.domain_valid`: New bot for checking domain's validity (PR1966 by Marius Karotkis).
- `intelmq.bots.experts.truncate_by_delimiter.expert`: Cut string if its length is higher than a maximum length (PR1967 by Marius Karotkis).
- `intelmq.bots.experts.remove_affix`: Remove prefix or postfix strings from a field (PR1965 by Marius Karotkis).
- `intelmq.bots.experts.asn_lookup.expert`: Fixes update-database script on the last few days of a month (PR2121 by Filip Pokorný, fixes 2088).
- `intelmq.bots.experts.threshold.expert`: Correctly use the standard parameter `redis_cache_ttl` instead of the previously used parameter `timeout` (PR2155 by Karl-Johan Karlsson).
- `intelmq.bots.experts.jinja2.expert`: Lift restriction on requirement jinja2 < 3 (PR2158 by Sebastian Wagner).
- `intelmq.bots.experts.asn_lookup.expert`, `intelmq.bots.experts.domain_suffix.expert`, `intelmq.bots.experts.maxmind_geoip.expert`, `intelmq.bots.experts.recordedfuture_iprisk.expert`, `intelmq.bots.experts.tor_nodes.expert`: New parameter `autoupdate_cached_database` to disable automatic updates (downloads) of cached databases (PR2180 by Sebastian Wagner).
- `intelmq.bots.experts.url.expert`: New bot for extracting additional information from `source.url` and/or `destination.url` (PR2315 by Filip Pokorný).

Outputs
- Removed `intelmq.bots.outputs.postgresql`: this bot was marked as deprecated in 2019 announced to be removed in version 3 of IntelMQ (PR2045 by Birger Schacht).
- Added `intelmq.bots.outputs.rpz_file.output` to create RPZ files (PR1962 by Marius Karotkis).
- Added `intelmq.bots.outputs.bro_file.output` to create Bro intel formatted files (PR1963 by Marius Karotkis).
- `intelmq.bots.outputs.templated_smtp.output`:
- Add new function `from_json()` (which just calls `json.loads()` in the standard Python environment), meaning the Templated SMTP output bot can take strings containing JSON documents and do the formatting itself (PR2120 by Karl-Johan Karlsson).
- Lift restriction on requirement jinja2 < 3 (PR2158 by Sebastian Wagner).
- `intelmq.bots.outputs.sql`:
- For PostgreSQL, escape Nullbytes in text to prevent "unsupported Unicode escape sequence" issues (PR2223 by Sebastian Wagner, fixes 2203).

Documentation
- Feeds: Add documentation for newly supported dataplane feeds, see above (PR2102 by Mikk Margus Möll).
- Installation: Restructured the whole document to make it clearer and straight-forward (PR2113 by Sebastian Wagner).
- Add workaround for https://github.com/sphinx-doc/sphinx/issues/10701 (PR#2225 by Sebastian Wagner, kudos yarikoptic, fixes 2224).
- Fix wrong operator for list-contains-value operation in sieve expert documentation (PR2256 by Filip Pokorný).
- Added documentation on `default_fields` parameter (PR2293 by Filip Pokorný).
- Updated documentation on `skip_header` parameter (PR2313 by Filip Pokorný).
- Viriback Unsafe Sites feed replaced with Viriback C2 Tracker. (PR2266 by Filip Pokorný)
- Netlab 360 Mirai Scanner feed removed as it is discontinued. (PR2266 by Filip Pokorný)
- Benkow Malware Panels Tracker feed changed parser configuration. (PR2266 by Filip Pokorný)
- Taichung feed removed as it is discontinued. (PR2266 by Filip Pokorný)
- Added new URL Expert bot. (PR2315 by Filip Pokorný)

Packaging
- Remove deleted `intelmq.bots.experts.sieve.validator` from executables in `setup.py` (PR2256 by Filip Pokorný).
- Run the geoip database cron-job twice a week (PR2285 by Filip Pokorný).

Tests
- Add GitHub Action to run regexploit on all Python, JSON and YAML files (PR2059 by Sebastian Wagner).
- `intelmq.lib.test`:
- Decorator `skip_ci` also detects `dpkg-buildpackage` environments by checking the environment variable `DEB_BUILD_ARCH` (PR2123 by Sebastian Wagner).
- Fixing regex to catchall after python version and process ID, add tests for it (PR2216 by Sebastian Waldbauer and Sebastian Wagner, fixes 2185)
- Also test on Python 3.10 (PR2140 by Sebastian Wagner).
- Switch from nosetests to pytest, as the former does not support Python 3.10 (PR2140 by Sebastian Wagner).
- CodeQL Github Actions `exponential backtracking on strings` fixed. (PR2148 by Sebastian Waldbauer, fixes 2138)
- Reverse DNS expert tests: remove outdated failing test `test_invalid_ptr` (PR2208 by Sebastian Wagner, fixes 2206).
- Add test dependency `requests_mock` to the `development` extra requirements in `setup.py` (PR2210 by Sebastian Wagner).
- Threshold Expert tests: Use environment variable `INTELMQ_PIPELINE_HOST` as redis host, analogous to other tests (PR2209 by Sebastian Wagner, fixes 2207).
- Remove codecov action as it failed regularly (PR2237 by Sebastian Wagner, fixes 2229).
- `intelmq.lib.test.BotTestCase`: Adds `skip_checks` variable to not fail on non-empty messages from calling `check` function (PR2315 by Filip Pokorný).

Tools
- `intelmqctl`:
- fix process manager initialization if run non-interactively, as intelmqdump does it (PR2189 by Sebastian Wagner, fixes 2188).
- `check`: handle `SyntaxError` in bot modules and report it without breaking execution (fixes 2177)
- Privilege drop before logfile creation (PR2277 by Sebastian Waldbauer, fixes 2176)
- `intelmqsetup`: Revised installation of manager by building the static files at setup, not build time, making it behave more meaningful. Requires intelmq-manager >= 3.1.0 (PR2198 by Sebastian Wagner, fixes 2197).
- `intelmqdump`: Respected global and per-bot custom settings of `logging_path` (fix 1605).

Contrib
- logrotate: Move compress and ownership rules to the IntelMQ-blocks to prevent that they apply to other files (PR2111 by Sebastian Wagner, fixes 2110).

Known issues
This is short list of the most important known issues. The full list can be retrieved from [GitHub](https://github.com/certtools/intelmq/labels/bug?page=2&q=is%3Aopen+label%3Abug).
- intelmq_psql_initdb does not work for SQLite (2202).
- intelmqsetup: should install a default state file (2175).
- Misp Expert - Crash if misp event already exist (2170).
- Turris greylist has been updated (2167).
- Spamhaus CERT parser uses wrong field (2165).
- Custom headers ignored in HTTPCollectorBot (2150).
- Missing commas in SQL query for separate Events table (2125).
- intelmqctl log: parsing syslog does not work (2097).
- Bash completion scripts depend on old JSON-based configuration files (2094).
- Bot configuration examples use JSON instead of YAML (2066).
- Bots started with IntelMQ-API/Manager stop when the webserver is restarted (952).
- Corrupt dump files when interrupted during writing (870).

------------------

Core
- `intelmq.lib.bot.CollectorBot`: Fixed an issue with within the `new_report` function, which re-loads the harmonization file after a new incoming dataset, which leads to CPU drain and decreased performance (PR2106 by Sebastian Waldbauer, fixes 2098).
- `intelmq.lib.bot.Bot`: Make private members `__is_multithreadable` and `__collector_empty_process` protected members `_is_multithreadable` and `_collector_empty_process` to make them easily modifiable by Bot classes (PR2109 by Sebastian Wagner, fixes 2108).
Also affected and adapted bots by this change are:
- `intelmq.bots.collectors.api.collector_api`
- `intelmq.bots.collectors.stomp.collector`
- `intelmq.bots.experts.splunk_saved_search.expert`
- `intelmq.bots.experts.threshold.expert`
- `intelmq.bots.outputs.file.output`
- `intelmq.bots.outputs.misp.output_api`
- `intelmq.bots.outputs.misp.output_feed`
- `intelmq.bots.outputs.tcp.output`
- `intelmq.bots.outputs.udp.output`
- `intelmq.lib.cache`: Do not create the Cache class if the host is null, allows deactivating the bot statistics (PR2104 by Sebastian Waldbauer, fixes 2103).

Bots
Experts
- `intelmq.bots.experts.domain_suffix.expert`: Only print skipped database update message if verbose mode is active (PR2107 by Sebastian Wagner, fixes 2016).

Documentation
- Add configuration upgrade steps for 3.0 to NEWS (PR2101 by Sebastian Wagner).

Known issues
See [open bug reports](https://github.com/certtools/intelmq/issues?q=is%3Aissue+is%3Aopen+label%3Abug) for a more detailed list.
- ParserBot: erroneous raw line recovery in error handling (1850).