Intelmq

Latest version: v3.3.1

Safety actively analyzes 714919 Python packages for vulnerabilities to keep your Python projects secure.

Scan your dependencies

Page 2 of 7

3.0.1

------------------

Configuration

Core
- `intelmq.lib.bot_debugger`: Fix accessing the bot's destination queues (PR2027 by Mikk Margus Möll).
- `intelmq.lib.pipeline`: Fix handling of `load_balance` parameter (PR2027 by Mikk Margus Möll).
- `intelmq.lib.bot`: Fix handling of parameter `destination_queues` if value is an empty dictionary (PR2051 by Sebastian Wagner, fixes 2034).

Bots
Collectors
- `intelmq.bots.collectors.shodan.collector_stream`: Fix access to parameters, the bot wrongly used `self.parameters` (PR2020 by Mikk Margus Möll).
- `intelmq.bots.collectors.mail.collector_mail_attach`: Add attachment file name as `extra.file_name` also if the attachment is not compressed (PR2021 by Alex Kaplan).
- `intelmq.bots.collectors.http.collector_http_stream`: Fix access to parameters, the bot wrongly used `self.parameters` (by Sebastian Wagner).

Parsers
- `intelmq.bots.parsers.microsoft.parser_ctip`: Map `Payload.domain` to `destination.fqdn` instead of `extra.payload.domain` as it matches to `destination.ip` from `DestinationIp` (PR2023 by Sebastian Wagner).
- Removed `intelmq.bots.parsers.malwaredomains` because the upstream data source (malwaredomains.com) does not exist anymore (PR2026 by Birger Schacht, fixes 2024).
- `intelmq.bots.parsers.shadowserver._config`:
- Add support for feed "Vulnerable SMTP Server" (PR2037 by Mikk Margus Möll).
- Fix differentiation between feeds "Accessible HTTP" and "Vulnerable HTTP" (PR2037 by Mikk Margus Möll, fixes 1984).
- Add support for the new feeds *Microsoft Sinkhole Events Report*, *Microsoft Sinkhole HTTP Events Report* (PR2036 by Birger Schacht).
- Complement feed mappings and documentation for feeds with IPv4 and IPv6 variants (PR2046 by Mikk Margus Möll and Sebastian Wagner).
- Feed names with and without the optional IPv4/IPv6 postfix can be used now consistently.
- Add support for feed "Honeypot HTTP Scan" (PR2047 by Mikk Margus Möll).
- Update filename mapping for changed filename of feed "Accessible-MSRDPUDP" (PR2060 by abr4xc).

Experts
- `intelmq.bots.experts.gethostbyname.expert`: Handle numeric values for the `gaierrors_to_ignore` parameter (PR2073 by Sebastian Wagner, fixes 2072).
- `intelmq.bots.experts.filter.expert`: Fix handling of empty-string parameters `not_after` and `not_before` (PR2075 by Sebastian Wagner, fixes 2074).

Outputs
- `intelmq.bots.outputs.mcafee.output_esm_ip`: Fix access to parameters, the bot wrongly used `self.parameters` (by Sebastian Wagner).
- `intelmq.bots.outputs.misp.output_api`: Fix access to parameters, the bot wrongly used `self.parameters` (by Sebastian Wagner).
- `intelmq.bots.outputs.smtp.output`: Add `Content-Disposition`-header to the attachment, fixing the display in Mail Clients as actual attachment (PR2052 by Sebastian Wagner, fixes 2018).

Documentation
- Various formatting fixes (by Sebastian Wagner).
- Removed the malwaredomains feed from the feeds list because the upstream data source (malwaredomains.com) does not exist anymore (PR2026 by Birger Schacht, fixes 2024).
- Update Docker installation instructions (PR2035 by Sebastian Waldbauer).

Packaging
- intelmq-update-database crontab: Add missing `recordedfuture_iprisk` update call (by Sebastian Wagner).

Tests
- Replace calls to deprecated/undocumented `logging.warn` with `logging.warning` (by Sebastian Wagner, fixes 2013).
- `intelmq.tests.bots.experts.rdap.test_expert`: Declare cache use, fixes build failures (by Sebastian Wagner, fixes 2014).
- `intelmq.tests.bots.collectors.mail.test_collector_attach`: Test text attachment (by Sebastian Wagner).

Tools
- `intelmqctl`:
- Also honour parameters from environment variables (PR2068 by Sebastian Wagner, fixes 2063).
- Fix management actions (start/stop/status/reload/restart) for groups (PR2086 by Sebastian Wagner, fixes 2085).
- Do not use hardcoded logging path in `/opt/intelmq`, use the internal default instead (PR2092 by Sebastian Wagner, fixes 2091).

Known issues
See [open bug reports](https://github.com/certtools/intelmq/issues?q=is%3Aissue+is%3Aopen+label%3Abug) for a more detailed list.
- ParserBot: erroneous raw line recovery in error handling (1850).

3.0.0

------------------

Configuration
- The `BOTS` file is no longer used and has been removed (by Sebastian Wagner).
- The `defaults.conf` file is no longer used and has been removed (PR1814 by Birger Schacht).
- The `pipeline.conf` file is no longer used and has been removed (PR1849 by Birger Schacht).
- The `runtime.conf` was renamed to `runtime.yaml` and is now in YAML format (PR1812 by Birger Schacht).

Core
- `intelmq.lib.harmonization`:
- New class `ClassificationTaxonomy` with fixed list of taxonomies and sanitiation (by Sebastian Wagner).
- `intelmq.lib.bot`:
- Handle `InvalidValue` exceptions upon message retrieval by dumping the message instead of repeating endlessly (1765, PR1766 by Filip Pokorný).
- Rewrite of the parameter loading and handling, getting rid of the `parameters` member (PR1729 by Birger Schacht).
- The pipeline is now initialized before the call of `init` to allow bots accessing data directly on startup/initialization for cleanup or maintenance tasks (PR1982 by Sebastian Wagner).
- `intelmq.lib.exceptions`:
- `InvalidValue`: Add optional parameter `object` (PR1766 by Filip Pokorný).
- `intelmq.lib.utils`:
- New function `list_all_bots` to list all available/installed bots as replacement for the BOTS file (368, 552, 644, 757, 1069, 1750, PR1751 by Sebastian Waldbauer).
- New function `get_bots_settings` to return the effective bot parameters, with global parameters applied (PR1928 by Sebastian Wagner, 1927).
- Removed deprecated function `create_request_session_from_bot` (PR1997 by Sebastian Wagner, 1404).
- `parse_relative`: Add support for parsing minutes and seconds (PR1857 by Sebastian Wagner).
- `intelmq.lib.bot_debugger`:
- Set bot's `logging_level` directly in `__init__` before the bot's initialization by changing the default value (by Sebastian Wagner).
- Rewrite `load_configuration_patch` by adapting it to the parameter and configuration rewrite (by Sebastian Wagner).
- Do not rely on the runtime configuration's `group` setting of bots to determine the required message type of messages given on the command line (PR1949 by Sebastian Wagner).

Development
- `rewrite_config_files.py`: Removed obsolete BOTS-file-related rewriting functionality (by Sebastian Wagner, 1543).
- A GitHub Action that checks for [reuse compliance](https://reuse.software) of all the license and copyright headers was added (PR#1976 by Birger Schacht).
- PyYAML is no longer a required dependency for development environments, all calls to it have been replaced by ruamel.yaml (by Sebastian Wagner).

Data Format
The IntelMQ Data Harmonization ("DHO") is renamed to IntelMQ Data Format ("IDF"). Internal files remain and work the same as before (PR1818 by Sebastian Waldbauer, fixes 1810).
Update allowed classification fields to version 1.3 (2021-05-18) (by Sebastian Wagner, fixes 1409, 1476).
- The taxonomy `abusive content` has been renamed to `abusive-content`.
- The taxonomy `information content security` has been renamed to `information-content-security`.
- The validation of type `unauthorised-information-access` has been fixed, a bug prevented the use of it.
- The validation of type `unauthorised-information-modification` has been fixed, a bug prevented the use of it.
- The type `leak` has been renamed to `data-leak`.
- The type `dropzone` has been removed. Taxonomy `other` with type `other` and identifier `dropzone` can be used instead. Ongoing discussion in the RSIT WG.
- The taxonomy `intrusion attempts` has been renamed to `intrusion-attempts`.
- For the taxonomy intrusions (PR1993 by Sebastian Wagner, addresses 1409):
- The type `compromised` has been renamed to `system-compromise`.
- The type `unauthorized-command` has been merged into `system-compromise`.
- The type `unauthorized-login` has been merged into `system-compromise`.
- The type `backdoor` has been merged into `system-compromise` (PR1995 by Sebastian Wagner, addresses 1409).
- The type `defacement` has been merged into taxonomy `information-content-security`, type `unauthorised-information-modification` (PR1994 by Sebastian Wagner, addresses 1409).
- The taxonomy `information gathering` has been rename to `information-gathering`.
- The taxonomy `malicious code` has been renamed to `malicious-code`.
- The type `c2server` has been renamed to `c2-server`.
- The type `malware` has been integrated into `infected-system` and `malware-distribution`, respectively (PR1917 by Sebastian Wagner addresses 1409).
- The type `ransomware` has been integrated into `infected-system`.
- The type `dga domain` has been moved to the taxonomy `other` renamed `dga-domain` (PR1992 by Sebastian Wagner fixes 1613).
- For the taxonomy 'availability', the type `misconfiguration` is new.
- For the taxonomy 'other', the type `unknown` has been renamed to `undetermined`.
- For the taxonomy 'vulnerable':
- The type `vulnerable client` has been renamed to `vulnerable-system`.
- The type `vulnerable service` has been renamed to `vulnerable-system`.

Bots
- The parameters handling of numerous bots has been refactored (PR1751, PR1729, by Birger Schacht, Sebastian Wagner, Sebastian Waldbauer).

Collectors
- Remove `intelmq.bots.collectors.xmpp`: one of the dependencies of the bot was deprecated and according to a short survey on the IntelMQ
users mailinglist, the bot is not used by anyone. (https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html, PR#1761 by Birger Schacht, closes 1614).
- `intelmq.bots.collectors.mail._lib`: Added parameter `mail_starttls` for STARTTLS in all mail collector bots (PR1831 by Marius Karotkis, fixes 1128).
- Added `intelmq.bots.collectors.fireeye`: A bot that collects indicators from Fireeye MAS appliances (PR1745 by Christopher Schappelwein).
- `intelmq.bots.collectors.api.collector_api` (PR1987 by Mikk Margus Möll, fixes 1986):
- Added UNIX socket capability.
- Correctly close the IOLoop in the shutdown method to fix reload.
- `intelmq.bots.collectors.rt.collector_rt` (PR1997 by Sebastian Wagner, 1404):
- compatibility with the deprecated parameter `unzip_attachment` (removed in 2.1.0) was removed.

Parsers
- Added `intelmq.bots.parsers.fireeye`: A bot that parses hashes and URLs from Fireeye MAS indicators (PR1745 by Christopher Schappelwein).
- `intelmq.bots.parsers.shadowserver._config`:
- Improved the feed-mapping and all conversion functions (PR1971 by Mikk Margus Möll).
- `intelmq.bots.parsers.generic.parser_csv`:
- Fix handling of empty string values for parameter `time_format` (by Sebastian Wagner).

Experts
- `intelmq.bots.experts.domain_suffix.expert`:
- Added `--update-database` option to update domain suffix database (by Sebastian Wagner).
- Fix `check` method: load database with UTF-8 encoding explicitly (by Sebastian Wagner).
- Added `intelmq.bots.experts.http.expert_status`: A bot that fetches the HTTP Status for a given URI and adds it to the message (PR1789 by Birger Schacht, fixes 1047 partly).
- Added `intelmq.bots.experts.http.expert_content`: A bot that fetches an HTTP resource and checks if it contains a specific string (PR1811 by Birger Schacht).
- Added `intelmq.bots.experts.lookyloo.expert`: A bot that sends requests to a lookyloo instance & adds `screenshot_url` to the event (PR1844 by Sebastian Waldbauer, fixes 1048).
- Added `intelmq.bots.experts.rdap.expert`: A bot that checks the rdap protocol for an abuse contact for a given domain (PR1881 by Sebastian Waldbauer and Sebastian Wagner).
- `intelmq.bots.experts.sieve.expert`:
- Add operators for comparing lists and sets (PR1895 by Mikk Margus Möll):
- `:equals`
- `:overlaps`
- `:supersetof`
- `:subsetof`
- `:equals`
- Add support for comparing boolean values (PR1895 by Mikk Margus Möll).
- Add support for rule negation with `!` (PR1895, PR1923 by Mikk Margus Möll).
- Add support for values types float, int, bool and string for all lists items (PR1895 by Mikk Margus Möll).
- Add actions for lists (PR1895 by Mikk Margus Möll).
- `append`
- `append!` (forced/overwriting)
- Rewrite the rule-processing and operator-handling code to make it more comprehensible and extensible (PR1895, PR1923 by Mikk Margus Möll).
- Nested if statements, plus mixed actions and actions in the same scope (PR 1923 by Mikk Margus Möll).
- The attribute manipulation actions add, add! and update support non-string (bool/int/float) values (PR 1923 by Mikk Margus Möll).
- Drop the `:notcontains` operator, as it made is redundant by generic negation: `! foo :contains 'x'` instead of `foo :notcontains 'x'` (PR1957 by Mikk Margus Möll).
- Split string and numeric matches into single- and multivalued variants, with the relevant new operators `:in`, `:containsany` and `:regexin` for string lists, and `:in` for numeric value lists (PR1957 by Mikk Margus Möll).
- Removed the `==` operator for lists, with the previous meaning of `:in`. Have a look at the NEWS.md for more information.
- Added `intelmq.bots.experts.uwhoisd`: A bot that fetches the whois entry from a uwhois-instance (PR1918 by Raphaël Vinot).
- Removed deprecated `intelmq.bots.experts.ripencc_abuse_contact.expert`. It was replaced by `intelmq.bots.experts.ripe.expert` and marked as deprecated in 2.0.0.beta1 (PR1997 by Sebastian Wagner, 1404).
- `intelmq.bots.experts.modify.expert`:
- Removed compatibility with deprecated configuration format before 1.0.0.dev7 (PR1997 by Sebastian Wagner, 1404).
- Added `intelmq.bots.experts.aggregate`: A bot that aggregate events based upon given fields & a timespan (PR1959 by Sebastian Waldbauer).
- Added `intelmq.bots.experts.tuency`: A bot that queries the IntelMQ API of a tuency instance (PR1857 by Sebastian Wagner, fixes 1856).

Outputs
- Remove `intelmq.bots.outputs.xmpp`: one of the dependencies of the bot was deprecated and according to a short survey on the IntelMQ
users mailinglist, the bot is not used by anyone. (https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html, PR#1761 by Birger Schacht, closes 1614)
- `intelmq.bots.outputs.smtp`: Add more debug logging (PR1949 by Sebastian Wagner).
- Added new bot `intelmq.bots.outputs.templated_smtp` (PR1901 by Karl-Johan Karlsson).

Documentation
- Updated user and developer documentation to reflect the removal of the BOTS file (PR1780 by Birger Schacht).
- Bots documentation:
- Added anchors to all bot sections derived from the module names for easier linking (PR1943 by Sebastian Wagner fixes part of certtools/intelmq-api4).
- License and copyright information was added to all the bots (PR1976 by Birger Schacht).
- Added documentation on the EventDB (PR1955 by Birger Schacht, PR1985 by Sebastian Wagner).
- Added TimescaleDB for time-series documentation (PR1990 by Sebastian Waldbauer).
- Improved n6 interoperability documentation by adding more graphs and illustrations (PR1991 by Sebastian Wagner).
- Feed documentation generation: fix and simplify formatting of parameters of types lists, non-string values have been ill-treated (by Sebastian Wagner).
- Added documentation on abuse-contact look-ups (PR2021 by Sebastian Waldbauer and Sebastian Wagner).

Packaging
- Docker images tagged with `certat/intelmq-full:develop` are built and published on every push to the develop branch (PR1753 by Sebastian Waldbauer).
- Adapt packaging to IntelMQ 3.0 changes: ruamel.yaml dependency, changed configuration, updated database-update scripts (by Birger Schacht and Sebastian Wagner).

Tests
- `intelmq.tests.lib.test_bot`:
- Add test case for a raised `InvalidValue` exception upon message retrieval (1765, PR1766 by Filip Pokorný and Sebastian Wagner).
- `intelmq.lib.test`:
- Compare content of the `output` field as dictionaries, not as string in `assertMessageEqual` (PR1975 by Karl-Johan Karlsson).
- Support multiple calls to `run_bot` from test cases (PR1989 by Sebastian Wagner).
- Split `prepare_source_queue` out of `prepare_bot`.
- Added new optional parameter `stop_bot` to `run_bot`.

Tools
- intelmqdump (PR1997 by Sebastian Wagner, 1404):
- The command `e` for deleting single entries by given IDs has been merged into the command `d` ("delete"), which can now delete either entries by ID or the whole file.
- The command `v` for editing entries has been renamed to `e` ("edit").

Contrib
- eventdb:
- Added `separate-raws-table.sql` (PR1985 by Sebastian Wagner).
- cron-jobs: Removed the deprecated update scripts (PR1997 by Sebastian Wagner, 1404):
- `update-asn-data`
- `update-geoip-data`
- `update-tor-nodes`
- `update-rfiprisk-data`
in favor of the built-in update-mechanisms (see the bots' documentation). A crontab file for calling all new update command can be found in `contrib/cron-jobs/intelmq-update-database`.

Known issues
- ParserBot: erroneous raw line recovery in error handling (1850).
- ruamel.yaml loader and dumper: human readability bug / support for comments (2003).

2.3.3

------------------

Core
- `intelmq.lib.upgrade`:
- Added `v233_feodotracker_browse` for Abuse.ch Feodotracker Browse parser configuration adaption (PR1941 by Sebastian Wagner).

Bots
Parsers
- `intelmq.bots.parsers.microsoft.parser_ctip`:
- Add support for new field `SourceIpInfo.SourceIpv4Int` (PR1940 by Sebastian Wagner).
- Fix mapping of "ConnectionType" fields, this is not `protocol.application`. Now mapped to `extra.*.connection_type` (PR1940 by Sebastian Wagner).
- `intelmq.bots.parsers.shadowserver._config`:
- Add support for the new feeds *Honeypot-Amplification-DDoS-Events*, *Honeypot-Brute-Force-Events*, *Honeypot-Darknet*, *IP-Spoofer-Events*, *Sinkhole-Events*, *Sinkhole-HTTP-Events*, *Vulnerable-Exchange-Server*, *Sinkhole-Events-HTTP-Referer* (PR1950, PR1952, PR1953, PR1954, PR1970 by Birger Schacht and Sebastian Wagner, PR1971 by Mikk Margus Möll).

Experts
- `intelmq.bots.experts.splunk_saved_search.expert`:
- fixed erroneous string formatting (PR1960 by Karl-Johan Karlsson).

Outputs
- `intelmq.bots.outputs.smtp.output`:
- Handle empty "fieldnames" parameter by sending no attachment (PR1932 by Sebastian Wagner).

Documentation
- `dev/data-harmonization` renamed to `dev/data-format` (by Sebastian Waldbauer)
- Feeds:
- Fixed Abuse.ch Feodotracker Browse parser configuration (PR1941 by Sebastian Wagner fixes 1938).

Tests
- `intelmq.bots.parsers.html_table`:
- Added testcase for Abuse.ch Feodotracker Browse (PR1941 by Sebastian Wagner).

Tools
- intelmqsetup:
- Set ownershop of state file path and its parent directory (PR1911 by Sebastian Wagner).

Known issues
- ParserBot: erroneous raw line recovery in error handling (1850).

2.3.2

------------------

Core
- `intelmq.lib.harmonization`:
- `TLP` type: accept value "yellow" for TLP level AMBER.

Bots
Collectors
- `intelmq.bots.collectors.shadowserver.collector_reports_api`:
- Handle timeouts by logging the error and continuing to next report (PR1852 by Marius Karotkis and Sebastian Wagner, fixes 1823).

Parsers
- `intelmq.bots.parsers.shadowserver.config`:
- Parse and harmonize field `end_time` as date in Feeds "Drone-Brute-Force" and "Amplification-DDoS-Victim" (PR1833 by Mikk Margus Möll).
- Add conversion function `convert_date_utc` which assumes UTC and sanitizes the data to datetime (by Sebastian Wagner, fixes 1848).
- `intelmq.bots.parsers.shadowserver.parser_json`:
- Use the overwrite parameter for optionally overwriting the "feed.name" field (by Sebastian Wagner).
- `intelmq.bots.parsers.microsoft.parser_ctip`:
- Handle fields `timestamp`, `timestamp_utc`, `source_ip`, `source_port`, `destination_ip`, `destination_port`, `computer_name`, `bot_id`, `asn`, `geo` in `Payload` of CTIP Azure format (PR1841, PR1851 and PR1879 by Sebastian Wagner).
- `intelmq.bots.parsers.shodan.parser`:
- Added support for unique keys and verified vulns (PR1835 by Mikk Margus Möll).
- `intelmq.bots.parsers.cymru.parser_cap_program`:
- Fix parsing in whitespace edge case in comments (PR1870 by Alex Kaplan, fixes 1862).

Experts
- `intelmq.bots.experts.modify`:
- Add a new rule to the example configuration to change the type of malicious-code events to `c2server` if the malware name indicates c2 (PR1854 by Sebastian Wagner).
- `intelmq.bots.experts.gethostbyname.expert`:
- Fix handling of parameter `gaierrors_to_ignore` with value `None` (PR1890 by Sebastian Wagner, fixes 1886).

Outputs
- `intelmq.bots.outputs.elasticsearch`: Fix log message on required elasticsearch library message (by Sebastian Wagner).

Documentation
- `dev/data-harmonization`: Fix taxonomy name "information gathering" should be "information-gathering" (by Sebastian Wagner).

Tests
- `intelmq.tests.bots.parsers.microsoft.test_parser_ctip_azure`:
- Add test case for TLP level "YELLOW".

Known issues
- ParserBot: erroneous raw line recovery in error handling (1850).

2.3.1

------------------

Configuration

Core
- `intelmq.lib.utils`:
- `log`: Handle null value for logging parameter `logging_max_size` (PR1786 by Sebastian Wagner, fixes 1778).
- `intelmq.lib.pipeline`:
- `Amqp._get_queues`: Check virtual host when retrieving queue sizes. Fixes output of `intelmqctl check` for orphaned queues if AMQP is used and the AMQP user has access to more virtual hosts (PR1830 by Sebastian Wagner, fixes 1746).

Bots
Collectors
- `intelmq.bots.collectors.shadowserver.collector_reports_api`: Added debug logging to show number of downloaded reports and download size (PR1826 by Sebastian Wagner, partly addresses 1688 and 1823).

Parsers
- `intelmq.bots.parsers.cymru.parser_cap_program`:
- Adapt parser to new upstream format for events of category "bruteforce" (PR1795 by Sebastian Wagner, fixes 1794).
- `intelmq.bots.parsers.shodan.parser`:
- Support nested conversions, improved protocol detection and extended Shodan parser mappings (PR1821 by Mikk Margus Möll).

Documentation
- Add missing newlines at end of `docs/_static/intelmq-manager/*.png.license` files (PR1785 by Sebastian Wagner, fixes 1777).
- Ecosystem: Revise sections on intelmq-cb-mailgen and fody (PR1792 by Bernhard Reiter).
- intelmq-api: Add documentation about necessary write permission for the session database file (PR1798 by Birger Schacht, fixes intelmq-api23).
- FAQ: Section on redis socket permissions: set only minimal necessary permissions (PR1809 by Sebastian Wagner).
- Add document on hardware requirements (PR1811 by Sebastian Wagner).
- Feeds: Added Shodan Country Stream (by Sebastian Wagner).

Tests
- Add missing newlines at end of various test input files (PR1785 by Sebastian Wagner, fixes 1777).
- `intelmq.tests.bots.parsers.shodan.test_parser`: Add test cases for new code (PR1821 by Mikk Margus Möll).
- `intelmq.tests.lib.test_harmonization.test_datetime_convert`: Only run this test in timezone UTC (PR1825 by Sebastian Wagner).

Tools
- `intelmqsetup`:
- Also cover required directory layout and file permissions for `intelmq-api` (PR1787 by Sebastian Wagner, fixes 1783).
- Also cover webserver and sudoers configuration for `intelmq-api` and `intelmq-manger` (PR1805 by Sebastian Wagner, fixes 1803).
- `intelmqctl`:
- Do not log an error message if logging to file is explicitly disabled, e.g. in calls from `intelmsetup`. The error message would not be useful for the user and is not necessary.

Known issues
- Bots started with IntelMQ-API/Manager stop when the webserver is restarted (952).
- Corrupt dump files when interrupted during writing (870).
- CSV line recovery forces Windows line endings (1597).
- intelmqdump: Honor logging_path variable (1605).
- Timeout error in mail URL fetcher (1621).
- Shadowserver Parser: Drone feed has (also?) application protocol in type field (mapped to transport protocol) (1763).

2.3.0

------------------

IntelMQ no longer supports Python 3.5 (and thus Debian 9 and Ubuntu 16.04), the minimum supported Python version is 3.6.

Configuration

Core
- `intelmq.lib.bot`:
- `ParserBot.recover_line_json_stream`: Make `line` parameter optional, as it is not needed for this method (by Sebastian Wagner).
- `Bot.argparser`: Added class method `_create_argparser` (returns `argparse.ArgumentParser`) for easy command line arguments parsing (PR1586 by Filip Pokorný).
- Runtime configuration does not necessarily need a parameter entry for each block. Previously at least an empty block was required (PR1604 by Filip Pokorný).
- Allow setting the pipeline host and the Redis cache host by environment variables for docker usage (PR1669 by Sebastian Waldbauer).
- Better logging message for SIGHUP handling if the handling of the signal is not delayed (by Sebastian Wagner).
- `intelmq.lib.upgrades`:
- Add upgrade function for removal of *HPHosts Hosts file* feed and `intelmq.bots.parsers.hphosts` parser (1559, by Sebastian Wagner).
- `intelmq.lib.exceptions`:
- `PipelineError`: Remove unused code to format exceptions (by Sebastian Wagner).
- `intelmq.lib.utils`:
- `create_request_session_from_bot`:
- Changed bot argument to optional, uses defaults.conf as fallback, renamed to `create_request_session`. Name `create_request_session_from_bot` will be removed in version 3.0.0 (PR1524 by Filip Pokorný).
- Fixed setting of `http_verify_cert` from defaults configuration (PR1758 by Birger Schacht).
- `log`: Use `RotatingFileHandler` for allow log file rotation without external tools (PR1637 by Vasek Bruzek).
- `intelmq.lib.harmonization`:
- The `IPAddress` type sanitation now accepts integer IP addresses and converts them to the string representation (by Sebastian Wagner).
- `DateTime.parse_utc_isoformat`: Add parameter `return_datetime` to return `datetime` object instead of string ISO format (by Sebastian Wagner).
- `DateTime.convert`: Fix `utc_isoformat` format, it pointed to a string and not a function, causing an exception when used (by Sebastian Wagner).
- `DateTime.from_timestamp`: Ensure that time zone information (`+00:00`) is always present (by Sebastian Wagner).
- `DateTime.__parse` now handles OverflowError exceptions from the dateutil library, happens for large numbers, e.g. telehpone numbers (by Sebastian Wagner).
- `intelmq.lib.upgrades`:
- Added upgrade function for CSV parser parameter misspelling (by Sebastian Wagner).
- Check for existence of collector and parser for the obsolete Malware Domain List feed and raise warning if found (1762, PR1771 by Birger Schacht).

Development
- `intelmq.bin.intelmq_gen_docs`:
- Add bot name to the resulting feed documentation (PR1617 by Birger Schacht).
- Merged into `docs/autogen.py` (PR1622 by Birger Schacht).

Bots
Collectors
- `intelmq.bots.collectors.eset.collector`: Added (PR1554 by Mikk Margus Möll).
- `intelmq.bots.collectors.http.collector_http`:
- Added PGP signature check functionality (PR1602 by sinus-x).
- If status code is not 2xx, the request's and response's headers and body are logged in debug logging level (1615, by Sebastian Wagner).
- `intelmq.bots.collectors.kafka.collector`: Added (PR1654 by Birger Schacht, closes 1634).
- `intelmq.bots.collectors.xmpp.collector`: Marked as deprecated, see https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html (#1614, PR1685 by Birger Schacht).
- `intelmq.bots.collectors.shadowserver.collector_api`:
- Added (1683, PR1700 by Birger Schacht).
- Change file names in the report to `.json` instead of the original and wrong `.csv` (PR1769 by Sebastian Wagner).
- `intelmq.bots.collectors.mail`: Add content of the email's `Date` header as `extra.email_date` to the report in all email collectors (PR1749 by aleksejsv and Sebastian Wagner).
- `intelmq.bots.collectors.http.collector_http_stream`: Retry on common connection issues without raising exceptions (1435, PR1747 by Sebastian Waldbauer and Sebastian Wagner).
- `intelmq.bots.collectors.shodan.collector_stream`: Retry on common connection issues without raising exceptions (1435, PR1747 by Sebastian Waldbauer and Sebastian Wagner).
- `intelmq.bots.collectors.twitter.collector_twitter`:
- Proper input validation in URLs using urllib. CWE-20, found by GitHub's CodeQL (PR1754 by Sebastian Wagner).
- Limit replacement ("pastebin.com", "pastebin.com/raw") to a maximum of one (PR1754 by Sebastian Wagner).

Parsers
- `intelmq.bots.parsers.eset.parser`: Added (PR1554 by Mikk Margus Möll).
- Ignore invalid "NXDOMAIN" IP addresses (PR1573 by Mikk Margus Möll).
- `intelmq.bots.parsers.hphosts`: Removed, feed is unavailable (1559, by Sebastian Wagner).
- `intelmq.bots.parsers.cznic.parser_haas`: Added (PR1560 by Filip Pokorný and Edvard Rejthar).
- `intelmq.bots.parsers.cznic.parser_proki`: Added (PR1599 by sinus-x).
- `intelmq.bots.parsers.key_value.parser`: Added (PR1607 by Karl-Johan Karlsson).
- `intelmq.bots.parsers.generic.parser_csv`: Added new parameter `compose_fields` (by Sebastian Wagner).
- `intelmq.bots.parsers.shadowserver.parser_json`: Added (PR1700 by Birger Schacht).
- `intelmq.bots.parsers.shadowserver.config`:
- Fixed mapping for Block list feed to accept network ranges in CIDR notation (1720, PR1728 by Sebastian Waldbauer).
- Added mapping for new feed MSRDPUDP, Vulnerable-HTTP, Sinkhole DNS (1716, 1726, 1733, PR1732, PR1735, PR1736 by Sebastian Waldbauer).
- Ignore value `0` for `source.asn` and `destination.asn` in all mappings to avoid parsing errors (PR1769 by Sebastian Wagner).
- `intelmq.bots.parsers.abusech.parser_ip`: Adapt to changes in the Feodo Tracker Botnet C2 IP Blocklist feed (PR1741 by Thomas Bellus).
- `intelmq.bots.parsers.malwaredomainlist`: Removed, as the feed is obsolete (1762, PR1771 by Birger Schacht).

Experts
- `intelmq.bots.experts.rfc1918.expert`:
- Add support for ASNs (PR1557 by Mladen Markovic).
- Speed improvements.
- More output in debug logging mode (by Sebastian Wagner).
- Checks parameter length on initialization and in check method (by Sebastian Wagner).
- `intelmq.bots.experts.gethostbyname.expert`:
- Added parameter `fallback_to_url` and set to True (PR1586 by Edvard Rejthar).
- Added parameter `gaierrors_to_ignore` to optionally ignore other `gethostbyname` errors (1553).
- Added parameter `overwrite` to optionally overwrite existing IP addresses (by Sebastian Wagner).
- `intelmq.bots.experts.asn_lookup.expert`:
- Added `--update-database` option (PR1524 by Filip Pokorný).
- The script `update-asn-data` is now deprecated and will be removed in version 3.0.
- `intelmq.bots.experts.maxmind_geoip.expert`:
- Added `--update-database` option (PR1524 by Filip Pokorný).
- Added `license_key` parameter (PR1524 by Filip Pokorný).
- The script `update-geoip-data` is now deprecated and will be removed in version 3.0.
- `intelmq.bots.experts.tor_nodes.expert`:
- Added `--update-database` option (PR1524 by Filip Pokorný).
- The script `update-tor-nodes` is now deprecated and will be removed in version 3.0.
- `intelmq.bots.experts.recordedfuture_iprisk.expert`:
- Added `--update-database` option (PR1524 by Filip Pokorný).
- Added `api_token` parameter (PR1524 by Filip Pokorný).
- The script `update-rfiprisk-data` is now deprecated and will be removed in version 3.0.
- Added `intelmq.bots.experts.threshold` (PR1608 by Karl-Johan Karlsson).
- Added `intelmq.bots.experts.splunk_saved_search.expert` (PR1666 by Karl-Johan Karlsson).
- `intelmq.bots.experts.sieve.expert`:
- Added possibility to give multiple queue names for the `path` directive (1462, by Sebastian Wagner).
- Added possibility to run actions without filtering expression (1706, PR1708 by Sebastian Waldbauer).
- Added datetime math operations (1680, PR1696 by Sebastian Waldbauer).
- `intelmq.bots.experts.maxmind_geoip.expert`:
- Fixed handing over of `overwrite` parameter to `event.add` (PR1743 by Birger Schacht).

Outputs
- `intelmq.bots.outputs.rt`: Added Request Tracker output bot (PR1589 by Marius Urkis).
- `intelmq.bots.outputs.xmpp.output`: Marked as deprecated, see https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html (#1614, PR1685 by Birger Schacht).
- `intelmq.bots.outputs.smtp.output`: Fix sending to multiple recipients when recipients are defined by event-data (1759, PR1760 by Sebastian Waldbauer and Sebastian Wagner).

Documentation
- Feeds:
- Add ESET URL and Domain feeds (by Sebastian Wagner).
- Remove unavailable *HPHosts Hosts file* feed (1559 by Sebastian Wagner).
- Added CZ.NIC HaaS feed (PR1560 by Filip Pokorný and Edvard Rejthar).
- Added CZ.NIC Proki feed (PR1599 by sinus-x).
- Updated Abuse.ch URLhaus feed (PT1572 by Filip Pokorný).
- Added CERT-BUND CB-Report Malware infections feed (PR1598 by sinus-x and Sebastian Wagner).
- Updated Turris Greylist feed with PGP verification information (by Sebastian Wagner).
- Fixed parsing of the `public` field in the generated feeds documentation (PR1641 by Birger Schacht).
- Change the `rate_limit` parameter of some feeds from 2 days (129600 seconds) to one day (86400 seconds).
- Update the cAPTure Ponmocup Domains feed documentation (PR1574 by Filip Pokorný and Sebastian Wagner).
- Added Shadowserver Reports API (by Sebastian Wagner).
- Change the `rate_limit` parameter for many feeds from 2 days to the default one day (by Sebastian Wagner).
- Removed Malware Domain List feed, as the feed is obsolete (1762, PR1771 by Birger Schacht).
- Bots:
- Enhanced documentation of RFC1918 Expert (PR1557 by Mladen Markovic and Sebastian Wagner).
- Enhanced documentation of SQL Output (PR1620 by Edvard Rejthar).
- Updated documentation for MaxMind GeoIP, ASN Lookup, TOR Nodes and Recorded Future experts to reflect new `--update-database` option (PR1524 by Filip Pokorný).
- Added documentation for Shadowserver API collector and parser (PR1700 by Birger Schacht and Sebastian Wagner).
- Add n6 integration documentation (by Sebastian Wagner).
- Moved 'Orphaned Queues' section from the FAQ to the intelmqctl documentation (by Sebastian Wagner).
- Generate documentation using Sphinx (PR1622 by Birger Schacht).
- The documentation is now available at https://intelmq.readthedocs.io/en/latest/
- Refactor documentation and fix broken syntax (1639, PRs 1638 1640 1642 by Birger Schacht).
- Integrate intelmq-manager and intelmq-api user documentation to provide unified documentation place (PR1714 & PR1714 by Birger Schacht).

Packaging
- Fix paths in the packaged logcheck rules (by Sebastian Wagner).
- Build the sphinx documentation on package build (PR1701 by Birger Schacht).
- Ignore non-zero exit-codes for the `intelmqctl check` call in postinst (1748, by Sebastian Wagner).

Tests
- Added tests for `intelmq.lib.exceptions.PipelineError` (by Sebastian Wagner).
- `intelmq.tests.bots.collectors.http_collector.test_collector`: Use `requests_mock` to mock all requests and do not require a local webserver (by Sebastian Wagner).
- `intelmq.tests.bots.outputs.restapi.test_output`:
- Use `requests_mock` to mock all requests and do not require a local webserver (by Sebastian Wagner).
- Add a test for checking the response status code (by Sebastian Wagner).
- `intelmq.tests.bots.collectors.mail.test_collector_url`: Use `requests_mock` to mock all requests and do not require a local webserver (by Sebastian Wagner).
- `intelmq.tests.bots.experts.ripe.test_expert`: Use `requests_mock` to mock all requests and do not require a local webserver (by Sebastian Wagner).
- The test flag (environment variable) `INTELMQ_TEST_LOCAL_WEB` is no longer used (by Sebastian Wagner).
- Added tests for `intelmq.harmonization.DateTime.parse_utc_isoformat` and `convert_fuzzy` (by Sebastian Wagner).
- Move from Travis to GitHub Actions (PR1707 by Birger Schacht).
- `intelmq.lib.test`:
- `test_static_bot_check_method` checks the bot's static `check(parameters)` method for any exceptions, and a valid formatted return value (1505, by Sebastian Wagner).
- `setUpClass`: Skip tests if cache was requests with `use_cache` member, but Redis is deactivated with the environment variable `INTELMQ_SKIP_REDIS` (by Sebastian Wagner).
- `intelmq.tests.bots.experts.cymru_whois.test_expert`:
- Switch from `example.com` to `ns2.univie.ac.at` for hopefully more stable responses (1730, PR1731 by Sebastian Waldbauer).
- Do not test for exact expected values in the 6to4 network test, as the values are changing regularly (by Sebastian Wagner).
- `intelmq.tests.bots.parsers.abusech`: Remove tests cases of discontinued feeds (PR1741 by Thomas Bellus).
- Activate GitHub's CodeQL Code Analyzing tool as GitHub Action (by Sebastian Wagner).

Tools
- `intelmqdump`:
- Check if given queue is configured upon recovery (1433, PR1587 by Mladen Markovic).
- `intelmqctl`:
- `intelmq list queues`: `--sum`, `--count`, `-s` flag for showing total count of messages (1408, PR1581 by Mladen Markovic).
- `intelmq check`: Added a possibility to ignore queues from the orphaned queues check (by Sebastian Wagner).
- Allow setting the pipeline host by environment variables for docker usage (PR1669 by Sebastian Waldbauer).

Contrib
- EventDB:
- Add SQL script for keeping track of the oldest inserted/update "time.source" information (by Sebastian Wagner).
- Cron Jobs: The script `intelmq-update-data` has been renamed to `intelmq-update-database` (by Filip Pokorný).
- Dropped utterly outdated contrib modules (by Sebastian Wagner):
- ansible
- vagrant
- vagrant-ansible
- logrotate:
- Do not use the deprecated "copytruncate" option as intelmq re-opens the log anyways (by Sebastian Wagner).
- Set file permissions to `0644` (by Sebastian Wagner).

Known issues
- Bots started with IntelMQ-API/Manager stop when the webserver is restarted (952).
- Corrupt dump files when interrupted during writing (870).
- CSV line recovery forces Windows line endings (1597).
- Timeout error in mail URL fetcher (1621).
- AMQP pipeline: get_queues needs to check vhost of response (1746).

Page 2 of 7

© 2025 Safety CLI Cybersecurity Inc. All Rights Reserved.