------------------
Configuration
- The `BOTS` file is no longer used and has been removed (by Sebastian Wagner).
- The `defaults.conf` file is no longer used and has been removed (PR1814 by Birger Schacht).
- The `pipeline.conf` file is no longer used and has been removed (PR1849 by Birger Schacht).
- The `runtime.conf` was renamed to `runtime.yaml` and is now in YAML format (PR1812 by Birger Schacht).
Core
- `intelmq.lib.harmonization`:
- New class `ClassificationTaxonomy` with fixed list of taxonomies and sanitiation (by Sebastian Wagner).
- `intelmq.lib.bot`:
- Handle `InvalidValue` exceptions upon message retrieval by dumping the message instead of repeating endlessly (1765, PR1766 by Filip Pokorný).
- Rewrite of the parameter loading and handling, getting rid of the `parameters` member (PR1729 by Birger Schacht).
- The pipeline is now initialized before the call of `init` to allow bots accessing data directly on startup/initialization for cleanup or maintenance tasks (PR1982 by Sebastian Wagner).
- `intelmq.lib.exceptions`:
- `InvalidValue`: Add optional parameter `object` (PR1766 by Filip Pokorný).
- `intelmq.lib.utils`:
- New function `list_all_bots` to list all available/installed bots as replacement for the BOTS file (368, 552, 644, 757, 1069, 1750, PR1751 by Sebastian Waldbauer).
- New function `get_bots_settings` to return the effective bot parameters, with global parameters applied (PR1928 by Sebastian Wagner, 1927).
- Removed deprecated function `create_request_session_from_bot` (PR1997 by Sebastian Wagner, 1404).
- `parse_relative`: Add support for parsing minutes and seconds (PR1857 by Sebastian Wagner).
- `intelmq.lib.bot_debugger`:
- Set bot's `logging_level` directly in `__init__` before the bot's initialization by changing the default value (by Sebastian Wagner).
- Rewrite `load_configuration_patch` by adapting it to the parameter and configuration rewrite (by Sebastian Wagner).
- Do not rely on the runtime configuration's `group` setting of bots to determine the required message type of messages given on the command line (PR1949 by Sebastian Wagner).
Development
- `rewrite_config_files.py`: Removed obsolete BOTS-file-related rewriting functionality (by Sebastian Wagner, 1543).
- A GitHub Action that checks for [reuse compliance](https://reuse.software) of all the license and copyright headers was added (PR#1976 by Birger Schacht).
- PyYAML is no longer a required dependency for development environments, all calls to it have been replaced by ruamel.yaml (by Sebastian Wagner).
Data Format
The IntelMQ Data Harmonization ("DHO") is renamed to IntelMQ Data Format ("IDF"). Internal files remain and work the same as before (PR1818 by Sebastian Waldbauer, fixes 1810).
Update allowed classification fields to version 1.3 (2021-05-18) (by Sebastian Wagner, fixes 1409, 1476).
- The taxonomy `abusive content` has been renamed to `abusive-content`.
- The taxonomy `information content security` has been renamed to `information-content-security`.
- The validation of type `unauthorised-information-access` has been fixed, a bug prevented the use of it.
- The validation of type `unauthorised-information-modification` has been fixed, a bug prevented the use of it.
- The type `leak` has been renamed to `data-leak`.
- The type `dropzone` has been removed. Taxonomy `other` with type `other` and identifier `dropzone` can be used instead. Ongoing discussion in the RSIT WG.
- The taxonomy `intrusion attempts` has been renamed to `intrusion-attempts`.
- For the taxonomy intrusions (PR1993 by Sebastian Wagner, addresses 1409):
- The type `compromised` has been renamed to `system-compromise`.
- The type `unauthorized-command` has been merged into `system-compromise`.
- The type `unauthorized-login` has been merged into `system-compromise`.
- The type `backdoor` has been merged into `system-compromise` (PR1995 by Sebastian Wagner, addresses 1409).
- The type `defacement` has been merged into taxonomy `information-content-security`, type `unauthorised-information-modification` (PR1994 by Sebastian Wagner, addresses 1409).
- The taxonomy `information gathering` has been rename to `information-gathering`.
- The taxonomy `malicious code` has been renamed to `malicious-code`.
- The type `c2server` has been renamed to `c2-server`.
- The type `malware` has been integrated into `infected-system` and `malware-distribution`, respectively (PR1917 by Sebastian Wagner addresses 1409).
- The type `ransomware` has been integrated into `infected-system`.
- The type `dga domain` has been moved to the taxonomy `other` renamed `dga-domain` (PR1992 by Sebastian Wagner fixes 1613).
- For the taxonomy 'availability', the type `misconfiguration` is new.
- For the taxonomy 'other', the type `unknown` has been renamed to `undetermined`.
- For the taxonomy 'vulnerable':
- The type `vulnerable client` has been renamed to `vulnerable-system`.
- The type `vulnerable service` has been renamed to `vulnerable-system`.
Bots
- The parameters handling of numerous bots has been refactored (PR1751, PR1729, by Birger Schacht, Sebastian Wagner, Sebastian Waldbauer).
Collectors
- Remove `intelmq.bots.collectors.xmpp`: one of the dependencies of the bot was deprecated and according to a short survey on the IntelMQ
users mailinglist, the bot is not used by anyone. (https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html, PR#1761 by Birger Schacht, closes 1614).
- `intelmq.bots.collectors.mail._lib`: Added parameter `mail_starttls` for STARTTLS in all mail collector bots (PR1831 by Marius Karotkis, fixes 1128).
- Added `intelmq.bots.collectors.fireeye`: A bot that collects indicators from Fireeye MAS appliances (PR1745 by Christopher Schappelwein).
- `intelmq.bots.collectors.api.collector_api` (PR1987 by Mikk Margus Möll, fixes 1986):
- Added UNIX socket capability.
- Correctly close the IOLoop in the shutdown method to fix reload.
- `intelmq.bots.collectors.rt.collector_rt` (PR1997 by Sebastian Wagner, 1404):
- compatibility with the deprecated parameter `unzip_attachment` (removed in 2.1.0) was removed.
Parsers
- Added `intelmq.bots.parsers.fireeye`: A bot that parses hashes and URLs from Fireeye MAS indicators (PR1745 by Christopher Schappelwein).
- `intelmq.bots.parsers.shadowserver._config`:
- Improved the feed-mapping and all conversion functions (PR1971 by Mikk Margus Möll).
- `intelmq.bots.parsers.generic.parser_csv`:
- Fix handling of empty string values for parameter `time_format` (by Sebastian Wagner).
Experts
- `intelmq.bots.experts.domain_suffix.expert`:
- Added `--update-database` option to update domain suffix database (by Sebastian Wagner).
- Fix `check` method: load database with UTF-8 encoding explicitly (by Sebastian Wagner).
- Added `intelmq.bots.experts.http.expert_status`: A bot that fetches the HTTP Status for a given URI and adds it to the message (PR1789 by Birger Schacht, fixes 1047 partly).
- Added `intelmq.bots.experts.http.expert_content`: A bot that fetches an HTTP resource and checks if it contains a specific string (PR1811 by Birger Schacht).
- Added `intelmq.bots.experts.lookyloo.expert`: A bot that sends requests to a lookyloo instance & adds `screenshot_url` to the event (PR1844 by Sebastian Waldbauer, fixes 1048).
- Added `intelmq.bots.experts.rdap.expert`: A bot that checks the rdap protocol for an abuse contact for a given domain (PR1881 by Sebastian Waldbauer and Sebastian Wagner).
- `intelmq.bots.experts.sieve.expert`:
- Add operators for comparing lists and sets (PR1895 by Mikk Margus Möll):
- `:equals`
- `:overlaps`
- `:supersetof`
- `:subsetof`
- `:equals`
- Add support for comparing boolean values (PR1895 by Mikk Margus Möll).
- Add support for rule negation with `!` (PR1895, PR1923 by Mikk Margus Möll).
- Add support for values types float, int, bool and string for all lists items (PR1895 by Mikk Margus Möll).
- Add actions for lists (PR1895 by Mikk Margus Möll).
- `append`
- `append!` (forced/overwriting)
- Rewrite the rule-processing and operator-handling code to make it more comprehensible and extensible (PR1895, PR1923 by Mikk Margus Möll).
- Nested if statements, plus mixed actions and actions in the same scope (PR 1923 by Mikk Margus Möll).
- The attribute manipulation actions add, add! and update support non-string (bool/int/float) values (PR 1923 by Mikk Margus Möll).
- Drop the `:notcontains` operator, as it made is redundant by generic negation: `! foo :contains 'x'` instead of `foo :notcontains 'x'` (PR1957 by Mikk Margus Möll).
- Split string and numeric matches into single- and multivalued variants, with the relevant new operators `:in`, `:containsany` and `:regexin` for string lists, and `:in` for numeric value lists (PR1957 by Mikk Margus Möll).
- Removed the `==` operator for lists, with the previous meaning of `:in`. Have a look at the NEWS.md for more information.
- Added `intelmq.bots.experts.uwhoisd`: A bot that fetches the whois entry from a uwhois-instance (PR1918 by Raphaël Vinot).
- Removed deprecated `intelmq.bots.experts.ripencc_abuse_contact.expert`. It was replaced by `intelmq.bots.experts.ripe.expert` and marked as deprecated in 2.0.0.beta1 (PR1997 by Sebastian Wagner, 1404).
- `intelmq.bots.experts.modify.expert`:
- Removed compatibility with deprecated configuration format before 1.0.0.dev7 (PR1997 by Sebastian Wagner, 1404).
- Added `intelmq.bots.experts.aggregate`: A bot that aggregate events based upon given fields & a timespan (PR1959 by Sebastian Waldbauer).
- Added `intelmq.bots.experts.tuency`: A bot that queries the IntelMQ API of a tuency instance (PR1857 by Sebastian Wagner, fixes 1856).
Outputs
- Remove `intelmq.bots.outputs.xmpp`: one of the dependencies of the bot was deprecated and according to a short survey on the IntelMQ
users mailinglist, the bot is not used by anyone. (https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html, PR#1761 by Birger Schacht, closes 1614)
- `intelmq.bots.outputs.smtp`: Add more debug logging (PR1949 by Sebastian Wagner).
- Added new bot `intelmq.bots.outputs.templated_smtp` (PR1901 by Karl-Johan Karlsson).
Documentation
- Updated user and developer documentation to reflect the removal of the BOTS file (PR1780 by Birger Schacht).
- Bots documentation:
- Added anchors to all bot sections derived from the module names for easier linking (PR1943 by Sebastian Wagner fixes part of certtools/intelmq-api4).
- License and copyright information was added to all the bots (PR1976 by Birger Schacht).
- Added documentation on the EventDB (PR1955 by Birger Schacht, PR1985 by Sebastian Wagner).
- Added TimescaleDB for time-series documentation (PR1990 by Sebastian Waldbauer).
- Improved n6 interoperability documentation by adding more graphs and illustrations (PR1991 by Sebastian Wagner).
- Feed documentation generation: fix and simplify formatting of parameters of types lists, non-string values have been ill-treated (by Sebastian Wagner).
- Added documentation on abuse-contact look-ups (PR2021 by Sebastian Waldbauer and Sebastian Wagner).
Packaging
- Docker images tagged with `certat/intelmq-full:develop` are built and published on every push to the develop branch (PR1753 by Sebastian Waldbauer).
- Adapt packaging to IntelMQ 3.0 changes: ruamel.yaml dependency, changed configuration, updated database-update scripts (by Birger Schacht and Sebastian Wagner).
Tests
- `intelmq.tests.lib.test_bot`:
- Add test case for a raised `InvalidValue` exception upon message retrieval (1765, PR1766 by Filip Pokorný and Sebastian Wagner).
- `intelmq.lib.test`:
- Compare content of the `output` field as dictionaries, not as string in `assertMessageEqual` (PR1975 by Karl-Johan Karlsson).
- Support multiple calls to `run_bot` from test cases (PR1989 by Sebastian Wagner).
- Split `prepare_source_queue` out of `prepare_bot`.
- Added new optional parameter `stop_bot` to `run_bot`.
Tools
- intelmqdump (PR1997 by Sebastian Wagner, 1404):
- The command `e` for deleting single entries by given IDs has been merged into the command `d` ("delete"), which can now delete either entries by ID or the whole file.
- The command `v` for editing entries has been renamed to `e` ("edit").
Contrib
- eventdb:
- Added `separate-raws-table.sql` (PR1985 by Sebastian Wagner).
- cron-jobs: Removed the deprecated update scripts (PR1997 by Sebastian Wagner, 1404):
- `update-asn-data`
- `update-geoip-data`
- `update-tor-nodes`
- `update-rfiprisk-data`
in favor of the built-in update-mechanisms (see the bots' documentation). A crontab file for calling all new update command can be found in `contrib/cron-jobs/intelmq-update-database`.
Known issues
- ParserBot: erroneous raw line recovery in error handling (1850).
- ruamel.yaml loader and dumper: human readability bug / support for comments (2003).