Intelmq

----------------------------------

Requirements
Python 3.8 or newer is required.

Bots
CIF 3 API Output deprecation
The CIF 3 API Output bot is not compatible with Python version greater or equal to 3.12 and will be removed in the future due to lack of maintenance.
See https://lists.cert.at/pipermail/intelmq-users/2024-December/000474.html for more information.

Twitter Collector removal
As the bot does not work anymore and uses an unmaintained library, it is removed from IntelMQ.
Please remove if from your setup.

`intelmqctl check` and `intelmqctl upgrade-config` command warns if you have the bot in use.

Twitter Parser renaming
The Twitter parser is renamed to *IoC Extractor Parser* (`intelmq.bots.parsers.ioc_extractor`).
`intelmqctl upgrade-config` will automatically adapt the configuration.

The previous module name is left as a stub to load the IoC Extractor parser for backwards-compatibility.

Packaging
Packages are now also available for Ubuntu 24.04.
To upgrade an Ubuntu 22.04 installation to 24.04 please refer to the Ubuntu documentation: https://documentation.ubuntu.com/server/how-to/software/upgrade-your-release/index.html

---------------------------------

No changes are required by administrators.

----------------------------------

Documentation
The documentation is now available at [docs.intelmq.org](https://docs.intelmq.org/). Documentation has been updated and restructured into User, Administrator and Developer Guide. It provides modern look with various quality of life improvements. Big thanks to to gethvi.
We now have a slick, modern mkdocs based documentation. Please do check it out!


Bots
Shadowserver dynamic parser / collector

**Note well**: if you use shadowserver feeds, **please read this section carefully**.

Thanks to shadowserver (elsif2), we have a new dynamic shadowserver reports API integration. What does it do?
It connects to the [Shadowserver API](https://www.shadowserver.org/what-we-do/network-reporting/api-documentation/),
requests a list of all the reports for a specific country and processes the ones that are new.

Motivation for this change:

Shadowserver adds new scans on a nearly weekly basis. IntelMQ's release cycle and the need for a stable release could not keep up with this high intensity of shadowserver parser changes.
We therefore (thanks to eslif2) move the shadowserver reports collector and parser to a new, dynamic system. It can:

- fetch the shadowserver schema from shadowserver (https://interchange.shadowserver.org/intelmq/v1/schema)
- dynamically collect new reports (see also https://docs.intelmq.org/latest/user/bots/?h=shadow#shadowserver-reports-api)
- parse the new reports

**Note well**: if your IntelMQ system runs in an airgapped environment or if it may only reach out to specific IPs/sites, you should read the notes here:
https://docs.intelmq.org/latest/user/bots/#shadowserver.
You will need to download shadowserver-schema.json periodically yourself in this case.

**Note well:**: since dynamic changes are a bit tricky, we defined that there is a schema contract:

> Schema contract
>
> Once set in the schema, the classification.identifier, classification.taxonomy, and classification.type fields will remain static for a specific report.

This makes things deterministic again.

Alienvault OTX

Fix of a bug where a certain condition would have always evaluated to False. (PR2449 by qux-bbb. Thanks)

AMQP
Quite a few changes (thanks to Kamil, gethvi) on AMQP

Obsoleted bots

- `intelmq.bots.parsers.netlab_360.parser`: Removed as the feed is discontinued. (2442 by Filip Pokorný)
- `intelmq.bots.parsers.webinspektor.parser`: Removed as the feed is discontinued. (2442 by Filip Pokorný)
- `intelmq.bots.parsers.sucuri.parser`: Removed as the feed is discontinued. (2442 by Filip Pokorný)

General changes and bug fixes

Digital Trust Center fixed a bug where the config was loaded twice in intelmqctl which created quite some speedups. Thanks!
This speeds up IntelMQ API calls.

Data Format

Shadowserver dynamic parser (see above).

General remarks

The full list of changes can be seen in the CHANGELOG.md file.

---------------------------------

All Bots
Fixes an issue which prevented bots from stopping gracefully after reloading.
As logrotate reloads all bots regularly, this bug affects most IntelMQ installations.

Reverse DNS Expert
Until IntelMQ version 3.2.0, the bot incorrectly cached and re-used results for /24 networks instead of single IP addresses.
If the bot retrieved the PTR for `192.0.43.7`, it was cached for `192.0.43.0/24` and used for all IP addresses in this range, for example for `192.0.43.8`.
IntelMQ version 3.2.1 fixes this issue.

The bugfix will correctly increase the cache sizes and decrease the performance, as less (incorrect) data is re-used.

----------------------------------

No changes are required by administrators.

[IEP007: Running IntelMQ bots as Python Library](https://github.com/certtools/ieps/tree/main/007) is implemented.

The accompanying 3.2.0 release of intelmq-api switches it's backend from the library hug to fastapi.
Deb-packages of intelmq-api 3.2.0 are delayed for some distributions because of necessary changes in packaging.

----------------------------------

Bots
Shadowserver Reports API collector
The misleading `country` parameter has been depreciated and a `reports` parameter has been added.
The backwards-compatibility will be removed in IntelMQ version 4.0.0.
See the [Shadowserver Reports API bot's documentation](https://docs.intelmq.org/latest/user/bots/#shadowserver-reports-api).

GitHub Collector
GitHub removed the basic `Username/Password` Authentication in favor of personal access tokens. So the GitHub Collector uses an Personal Access Token for authentication [GitHub Documentation: Generate a personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token)

url expert bot will deprecate url2fqdn
There is a new, more abstract bot (URL expert bot) which will gradually replace the old url2fqdn bot. The new one builds on top of the urlparse lib of python.
If you use the url2fqdn bot, please start migrating.

Feeds
Abuse.ch Feodo Tracker
IntelMQ previously supported two feeds: "Feodo Tracker IPs" (downloaded as CSV file) and "Feodo Tracker Browse" (downloaded as HTML table). These two feeds contain the same data but differ in the additional details. This IntelMQ release replaces both these feeds (and their parsers) with a feed called "Feodo Tracker" (downloaded as JSON file) which contains all the additional details from both feeds. The parser module for the new feed is `intelmq.bots.parsers.abusech.parser_feodotracker`.

Data Format
Field name checks
The field names for all data added to messages must match a pre-defined format.
The check which ensures this, was ineffective prior to this version and is effective again starting with version 3.1.0.
The [Data format documentation](https://docs.intelmq.org/latest/dev/data-format/#rules-for-keys) describes the required format.

Logrotate
The packaged configuration for *logrotate* falsely contained options applying to other programs' log files. This caused wrong ownership of log files.
This issues is corrected, but the ownership of affected log files may need to be changed manually.
To find affected files, you may use:
bash
sudo find /var/log/ -user intelmq ! -path \*intelmq\*


Configuration
Threshold Expert
The parameter `timeout` has been merged into `redis_cache_ttl`.

Postgres databases

The develop branch previously contained a set of unnecessary statements in a pre-release version to update the `classification.identifier`. If these changes were made please revert back to the previous state.