------------------
Requirements
- The python library `requests` is (again) listed as dependency of the core (1519).
Core
- `intelmq.lib.upgrades`:
- Harmonization upgrade: Also check and update regular expressions.
- Add function to migrate the deprecated parameter `attach_unzip` to `extract_files` for the mail attachment collector.
- Add function to migrate changed Taichung URL feed.
- Check for discontinued Abuse.CH Zeus Tracker feed.
- `intelmq.lib.bot`:
- `ParserBot.recover_line`: Parameter `line` needs to be optional, fix usage of fallback value `self.current_line`.
- `start`: Handle decoding errors in the pipeline different so that the bot is not stuck in an endless loop (1494).
- `start`: Only acknowledge a message in case of errors, if we actually had a message to dump, which is not the case for collectors.
- `_dump_message`: Dump messages with encoding errors base64 encoded, not in JSON format as it's not possible to decode them (1494).
- `intelmq.lib.test`:
- `BotTestCase.run_bot`: Add parameters `allowed_error_count` and `allowed_warning_count` to allow set the number per run, not per test class.
- Set `source_pipeline_broker` and `destination_pipeline_broker` to `pythonlist` instead of the old `broker`, fixes `intelmq.tests.lib.test_bot.TestBot.test_pipeline_raising`.
- Fix test for (allowed) errors and warnings.
- `intelmq.lib.exceptions`:
- `InvalidKey`: Add `KeyError` as parent class.
- `DecodingError`: Added, string representation has all relevant information on the decoding error, including encoding, reason and the affected string (1494).
- `intelmq.lib.pipeline`:
- Decode messages in `Pipeline.receive` not in the implementation's `_receive` so that the internal counter is correct in case of decoding errors (1494).
- `intelmq.lib.utils`:
- `decode`: Raise new `DecodingError` if decoding fails.
Harmonization
- `protocol.transport`: Adapt regular expression to allow the value `nvp-ii` (protocol 11).
Bots
Collectors
- `intelmq.bots.collectors.mail.collector_mail_attach`:
- Fix handling of deprecated parameter name `attach_unzip`.
- Fix handling of attachments without filenames (1538).
- `intelmq.bots.collectors.stomp.collector`: Fix compatibility with stomp.py versions `> 4.1.20` and catch errors on shutdown.
- `intelmq.bots.collectors.microsoft`:
- Update `REQUIREMENTS.txt` temporarily fixing deprecated Azure library (1530, PR1532).
- `intelmq.bots.collectors.microsoft.collector_interflow`: Add method for printing the file list.
Parsers
- `intelmq.bots.parsers.cymru.parser_cap_program`: Support for protocol 11 (`nvp-ii`) and `conficker` type.
- `intelmq.bots.parsers.taichung.parser`: Support more types/classifications:
- Application Compromise: Apache vulnerability & SQL injections
- Brute-force: MSSQL & SSH password guess attacks; Office 365, SSH & SIP attacks
- C2 Sever: Attack controller
- DDoS
- DoS: DNS, DoS, Excess connection
- IDS Alert / known vulnerability exploitation: backdoor
- Malware: Malware Proxy
- Warn on new unknown types.
- `intelmq.bots.parsers.bitcash.parser`: Removed as feed is discontinued.
- `intelmq.bots.parsers.fraunhofer.parser_ddosattack_cnc` and `intelmq.bots.parsers.fraunhofer.parser_ddosattack_target`: Removed as feed is discontinued.
- `intelmq.bots.parsers.malwaredomains.parser`: Correctly classify `C&C` and `phishing` events.
- `intelmq.bots.parsers.shadowserver.parser`: More verbose error message for missing report specification (1507).
- `intelmq.bots.parsers.n6.parser_n6stomp`: Always add n6 field `name` as `malware.name` independent of `category`.
- `intelmq.bots.parsers.anubisnetworks`: Update parser with new data format.
- `intelmq.bots.parsers.bambenek`: Add new feed URLs with Host `faf.bambenekconsulting.com` (1525, PR1526).
- `intelmq.bots.parsers.abusech.parser_ransomware`: Removed, as the feed is discontinued (1537).
- `intelmq.bots.parsers.nothink.parser`: Removed, as the feed is discontinued (1537).
- `intelmq.bots.parsers.n6.parser`: Remove not allowed characters in the name field for `malware.name` and write original value to `event_description.text` instead.
Experts
- `intelmq.bots.experts.cymru_whois.lib`: Fix parsing of AS names with Unicode characters.
Outputs
- `intelmq.bots.outputs.mongodb`:
- Set default port 27017.
- Use different authentication mechanisms per MongoDB server version to fix compatibility with server version >= 3.4 (1439).
Documentation
- Feeds:
- Remove unavailable feed Abuse.CH Zeus Tracker.
- Remove the field `status`, offline feeds should be removed.
- Add a new field `public` to differentiate between private and public feeds.
- Adding documentation URLs to nearly all feeds.
- Remove unavailable Bitcash.cz feed.
- Remove unavailable Fraunhofer DDos Attack feeds.
- Remove unavailable feed Abuse.CH Ransomware Tracker (1537).
- Update information on Bambenek Feeds, many require a license now (1525).
- Remove discontinued Nothink Honeypot Feeds (1537).
- Developers Guide: Fix the instructions for `/opt/intelmq` file permissions.
Packaging
- Patches: `fix-logrotate-path.patch`: also include path to rotated file in patch.
- Fix paths from `/opt` to LSB for `setup.py` and `contrib/logrotate/intelmq` in build process (1500).
- Add runtime dependency `debianutils` for the program `which`, which is required for `intelmqctl`.
Tests
- Dropping Travis tests for 3.4 as required libraries dropped 3.4 support.
- `intelmq.tests.bots.experts.cymru_whois`:
- Drop missing ASN test, does not work anymore.
- IPv6 to IPv4 test: Test for two possible results.
- `intelmq.lib.test`: Fix compatibility of logging capture with Python >= 3.7 by reworking the whole process (1342).
- `intelmq.bots.collectors.tcp.test_collector`: Removing custom mocking and bot starting, not necessary anymore.
- Added tests for `intelmq.bin.intelmqctl.IntelMQProcessManager._interpret_commandline`.
- Fix and split `tests.bots.experts.ripe.test_expert.test_ripe_stat_error_json`.
- Added tests for invalid encodings in input messages in `intelmq.tests.lib.test_bot` and `intelmq.tests.lib.test_pipeline` (1494).
- Travis: Explicitly enable RabbitMQ management plugin.
- `intelmq.tests.lib.test_message`: Fix usage of the parameter `blacklist` for Message hash tests (1539).
Tools
- `intelmqsetup`: Copy missing BOTS file to IntelMQ's root directory (1498).
- `intelmq_gen_docs`: Feed documentation generation: Handle missing/empty parameters.
- `intelmqctl`:
- `IntelMQProcessManager`: For the status of running bots also check the bot ID of the commandline and ignore the path of the executable (1492).
- `IntelMQController`: Fix exit codes of `check` command for JSON output (now 0 on success and 1 on error, was swapped, 1520).
- `intelmqdump`:
- Handle base64-type messages for show, editor and recovery actions.
Contrib
- `intelmq/bots/experts/asn_lookup/update-asn-data`: Use `pyasn_util_download.py` to download the data instead from RIPE, which cannot be parsed currently (1517, PR1518, https://github.com/hadiasghari/pyasn/issues/62).
Known issues
- HTTP stream collector: retry on regular connection problems? (1435).
- Bots started with IntelMQ-Manager stop when the webserver is restarted. (952).
- Reverse DNS: Only first record is used (877).
- Corrupt dump files when interrupted during writing (870).