Intelmq

------------------

Core
- `intelmq.lib.bot`:
- `Bot.__handle_sighup`: Handle exceptions in `shutdown` method of bots.

Harmonization
- FQDN: Disallow `:` in FQDN values to prevent values like '10.0.0.1:8080' (1235).

Bots
Collectors
- `intelmq.bots.collectors.stomp.collector`
- Fix name of shutdown method, was ineffective in the past.
- Ignore `NotConnectedException` errors on disconnect during shutdown.
- `intelmq.bots.collectors.mail.collector_mail_url`: Decode body if it is bytes (1367).
- `intelmq.bots.collectors.tcp.collector`: Timeout added. More stable version.

Parsers
- `intelmq.bots.parsers.shadowserver`:
- Add support for the `Amplification-DDoS-Victim`, `HTTP-Scanners`, `ICS-Scanners` and `Accessible-Ubiquiti-Discovery-Service` feeds (1368, 1383)
- `intelmq.bots.parsers.microsoft.parser_ctip`:
- Workaround for mis-formatted data in `networkdestinationipv4` field (since 2019-03-14).
- Ignore "hostname" ("destination.fqdn") if it contains invalid data.
- `intelmq.bots.parsers.shodan.parser`:
- In `minimal_mode`:
- Fix the parsing, previously only `source.geolocation.cc` and `extra.shodan` was correctly filled with information.
- Add a `classification.type` = 'other' to all events.
- Added tests for this mode.
- Normal mode:
- Fix the parsing of `timestamp` to `time.source in the normal mode, previously no timezone information has been added and thus every event raised an exception.
- ISAKMP: Ignore `isakmp.aggressive`, as the content is same as `isakmp` or less.
- `intelmq.bots.parsers.abusech.parser_ip`: Re-structure the bot and support new format of the changed "Feodo Tracker Domains" feed.
- `intelmq.bots.parsers.n6.parser`:
- Add parsing for fields "confidence", "expires" and "source".
- Add support for type "bl-other" (category "other").

Experts
- `intelmq.bots.experts.sieve.expert`: Fix key definition to allow field names with numbers (`malware.hash.md5`/`sha1`, 1371).

Outputs
- `intelmq.bots.outputs.tcp.output`: Timeout added. When no separator used, awaits that every message is acknowledged by a simple "Ok" string to ensure more stability.

Documentation
- Install: Update operating system versions
- Sieve Expert: Fix `elsif` -> `elif`.
- Rephrase the description of `time.*` fields.
- Feeds: New URL and format of the "Feodo Tracker IPs" feed. "Feodo Tracker Domains" has been discontinued.

Packaging

Tests
- Add missing `__init__.py` files in 4 bot's test directories. Previously these tests have never been executed.
- `intelmq.lib.test`: Allow bot test class names with an arbitrary postfix separated by an underscore. E.g. `TestShodanParserBot_minimal`.

Tools
- intelmqctl:
- status: Show commandline differences if a program with the expected PID could be found, but they do not match (previous output was `None`).
- Use logging level from defaults configuration if possible, otherwise intelmq's internal default. Previously, DEBUG was used unconditionally.

Known issues
- Bots started with IntelMQ-Manager stop when the webserver is restarted (952).
- stomp collector bot constantly uses 100% of CPU (1364).

------------------

Core
- `lib/harmonization.py`: Change `parse_utc_isoformat` of `DateTime` class from private to public (related to 1322).
- `lib/utils.py`: Add new function `object_pair_hook_bots`.
- `lib.bot.py`:
- `ParserBot`'s method `recover_line_csv` now also handles given `tempdata`.
- `Bot.acknowledge_message()` deletes `__current_message` to free the memory, saves memory in idling parsers with big reports.
- `start()`: Warn once per run if `error_dump_message` is set to false.
- `Bot.start()`, `ParserBot.process()`: If errors happen on bots without destination pipeline, the `on_error` path has been queried and lead to an exception being raised.
- `start()`: If `error_procedure` is pass and on pipeline errors, the bot retries forever (1333).
- `lib/message.py`:
- Fix add('extra', ..., overwrite=True): old extra fields have not been deleted previously (1335).
- Do not ignore empty or ignored (as defined in `_IGNORED_VALUES`) values of `extra.*` fields for backwards compatibility (1335).
- `lib/pipeline.py` (`Redis.receive`): Wait in 1s steps if redis is busy loading its snapshot from disk (1334).

Default configuration
- Set `error_dump_message` to true by default in `defaults.conf`.
- Fixed typo in `defaults.conf`: `proccess_manager` -> `process_manager`

Development
- `bin/rewrite_config_files.py`: Fix ordering of BOTS file (1327).

Harmonization
Update allowed classification fields to 2018-09-26 version (802, 1350, 1380). New values for `classification.type` are per taxonomy:
- Taxonomy 'intrusions':
- "application-compromise"
- "burglary"
- "privileged-account-compromise"
- "unprivileged-account-compromise"
- Taxonomy 'fraud':
- "copyright"
- "masquerade"
- "unauthorized-use-of-resources"
- Taxonomy 'information content security':
- "data-loss"
- Taxonomy 'vulnerable':
- "ddos-amplifier"
- "information-disclosure"
- "potentially-unwanted-accessible"
- "vulnerable-system"
- "weak-crypto"
- Taxonomy 'availability':
- "dos"
- "outage"
- "sabotage"
- Taxonomy 'abusive-content':
- "harmful-speech"
- "violence"
- Taxonomy 'malicious code':
- "malware-distribution"
- Taxonomy 'information-gathering':
- "social-engineering"
- "sniffing"
- Taxonomy 'information content security':
- "Unauthorised-information-access"
- "Unauthorised-information-modification"

Bots
Collectors
- `intelmq.bots.collectors.http.collector_http`:
- Fix parameter name `extract_files` in BOTS (1331).
- Fix handling of `extract_files` parameter if the value is an empty string.
- Handle not installed dependency library `requests` gracefully.
- Explain `extract_files` parameter in docs and use a sane default in BOTS file.
- `intelmq.bots.collectors.mail.collector_mail_url`:
- Handle HTTP status codes != 2xx the same as HTTP timeouts: No exception, but graceful handling.
- Handle HTTP errors (bad status code and timeouts) with `error_procedure` == 'pass' but marking the mail as read and logging the error.
- Handle not installed dependency library `requests` gracefully.
- `intelmq.bots.collectors.http.collector_http_stream`:
- Handle not installed dependency library `requests` gracefully.
- `intelmq.bots.collectors.microsoft.collector_interflow`:
- Handle not installed dependency library `requests` gracefully.
- `intelmq.bots.collectors.rt.collector_rt`:
- Handle not installed dependency library `requests` gracefully.
- added `intelmq.bots.collectors.shodan.collector_stream` for collecting shodan stream data (1096).
- Correctly check the version of the shodan library, it resulted in wrong comparisons with two digit numbers.
- `intelmq.bots.collectors.microsoft.collector_interflow`:
- Add check if Cache's TTL is big enough compared to `not_older_than` and throw an error otherwise.

Parsers
- `intelmq.bots.parsers.misp`: Fix Object attribute (1318).
- `intelmq.bots.parsers.cymru.parser_cap_program`:
- Add support for new format (extra data about botnet of 'bots').
- Handle AS number 0.
- `intelmq.bots.parsers.shadowserver`:
- Spam URL reports: remove `src_naics`, `src_sic` columns.
- fix parsing of 'spam' events in ShadowServer's 'Botnet Drone Hadoop' Report (1271).
- Add support in parser to ignore some columns in config file by using `False` as intelmq key.
- Add support for the `Outdated-DNSSEC-Key` and `Outdated-DNSSEC-Key-IPv6` feeds.
- Add support for the `Accessible-Rsync` feed.
- Document support for the `Open-LDAP-TCP` feed.
- Add support for `Accessible-HTTP` and `Open-DB2-Discovery-Service` (1349).
- Add support for `Accessible-AFP` (1351).
- Add support for `Darknet` (1353).
- `intelmq.bots.parsers.generic.parser_csv`: If the `skip_header` parameter was set to `True`, the header was not part of the `raw` field as returned by the `recover_line` method. The header is now saved and handled correctly by the fixed recovery method.
- `intelmq.bots.parsers.cleanmx.parser`: Use field `first` instead of `firsttime` for `time.source` (1329, 1348).
- `intelmq.bots.parsers.twitter.parser`: Support for `url-normalize` >= 1.4.1 and recommend it. Added new optional parameter `default_scheme`, passed to `url-normalize` (1356).

Experts
- `intelmq.bots.experts.national_cert_contact_certat.expert`:
- Handle not installed dependency library `requests` gracefully.
- `intelmq.bots.experts.ripencc_abuse_contact.expert`:
- Handle not installed dependency library `requests` gracefully.
- `intelmq.bots.experts.sieve.expert`:
- check method: Load missing harmonization, caused an error for every check.
- Add text and more context to error messages.
- README: Fix 'modify' to 'update' (1340).
- Handle empty rules file (1343).
- `intelmq.bots.experts.idea.expert`: Add mappings for new harmonization `classification.type` values, see above.

Outputs
- `intelmq.bots.outputs.redis`:
- Fix sending password to redis server.
- Fix for redis-py >= 3.0.0: Convert Event to string explicitly (1354).
- Use `Redis` class instead of deprecated `StrictRedis` for redis-py >= 3.0.0 (1355).
- `intelmq.bots.outputs.mongodb`:
- New parameter `replacement_char` (default: `'_'`) for non-hierarchical output as dots in key names are not allowed (1324, 1322).
- Save value of fields `time.observation` and `time.source` as native datetime object, not as string (1322).
- `intelmq.bots.outputs.restapi.output`:
- Handle not installed dependency library `requests` gracefully.

Documentation
- FAQ
- Explanation and solution on orphaned queues.
- Section on how and why to remove `raw` data.
- Add or fix the tables of contents for all documentation files.
- Feeds:
- Fix Autoshun Feed URL (1325).
- Add parameters `name` and `provider` to `intelmq/etc/feeds.yaml`, `docs/Feeds.md` and `intelmq/bots/BOTS` (1321).
- Add SECURITY.md file.

Packaging
- Change the maintainer from Sasche Wilde to Sebastian Wagner (1320).

Tests
- `intelmq.tests.lib.test_bot`: Skip `test_logging_level_other` on python 3.7 because of unclear behavior related to copies of loggers (1269).
- `intelmq.tests.bots.collectors.rt.test_collector`: Remove test because the REST interface of the instance has been closed (see also https://github.com/CZ-NIC/python-rt/issues/28).

Tools
- `intelmqctl check`: Shows more detailed information on orphaned queues.
- `intelmqctl`:
- Correctly determine the status of bots started with `intelmqctl run`.
- Fix output of errors during bot status determination, making it compatible to IntelMQ Manager.
- `check` subcommand: Show bot ID for messages also in JSON output.
- `run [bot-id] process -m [message]` works also with bots without a configured source pipeline (1307).

Contrib
- elasticsearch/elasticmapper: Add tlp field (1308).
- `feeds-config-generator/intelmq_gen_feeds_conf`:
- Add parameters to write resulting configuration directly to files (1321).
- Handle collector's `feed.name` and `feed.provider` (1314).

Known issues
- Bots started with IntelMQ-Manager stop when the webserver is restarted (952).
- Tests: capture logging with context manager (1342).
- stomp collector bot constantly uses 100% of CPU (1364).

------------------
- Support for Python 3.3 has been dropped in IntelMQ and some dependencies of it. Python 3.3 reached its end of life and Python 3.4 or newer is a hard requirement now.
- The list of feeds docs/Feeds.md has now a machine-readable equivalent YAML file in intelmq/etc/feeds.yaml
A tool to convert from yaml to md has been added.

Tools
- `intelmq_gen_feeds_docs` added to bin directory, allows generating the Feeds.md documentation file from feeds.yaml
- `intelmq_gen_docs` merges both `intelmq_gen_feeds_docs` and `intelmq_gen_harm_docs` in one file and automatically updates the documentation files.

intelmqctl
- `intelmqctl start` prints the bot's last error messages if the bot failed to start (1021).
- `intelmqctl start` message "is running" is printed every time. (Until now, it wasn't said when a bot was just starting.)
- `intelmqctl start/stop/restart/reload/status` now has a "--group" flag which allows you to specify the group of the bots that should be influenced by the command.
- `intelmqctl check` checks for defaults.conf completeness if the shipped file from the package can be found.
- `intelmqctl check` shows errors for non-importable bots.
- `intelmqctl list bots -q` only prints the IDs of enabled bots.
- `intelmqctl list queues-and-status` prints both queues and bots statuses (so that it can be used in eg. intelmq-manager).
- `intelmqctl run` parameter for showing a sent message.
- `intelmqctl run` if message is sent to a non-default path, it is printed out.
- `intelmqctl restart` bug fix; returned some half-nonsense, now returns return state of start and stop operation in a list (1226).
- `intelmqctl check`: New parameter `--no-connections` to prevent the command from making connections e.g. to the redis pipeline.s
- `intelmqctl list queues`: don't display named paths among standard queues.
- The process status test failed if the PATH did not include the bot executables and the `which` command failed. Then the proccess's command line could not be compared correctly. The fix warns of this and adds a new status 'unknown' (1297).


Contrib
- tool `feeds-config-generator` to automatically generate the collector and parser runtime and pipeline configurations.
- `malware_name_mapping`: Download and convert tool for malware family name mapping has been added.
- Added a systemd script which creates systemd units for bots (953).
- `contrib/cron-jobs/update-asn-data`, `contrib/cron-jobs/update-geoip-data`, `contrib/cron-jobs/update-tor-nodes`: Errors produce proper output.

Core
- lib/bot
- use SIGTERM instead of SIGINT to stop bots (981).
- Bots can specify a static method `check(parameters)` which can perform individual checks specific to the bot.
These functions will be called by `intelmqctl check` if the bot is configured with the given parameters
- top level bot parameters (description, group, module, name) are exposed as members of the class.
- The parameter `feed` for collectors is deprecated for 2.0 and has been replaced by the more consistent `name` (1144).
- bug: allow path parameter for CollectorBot class.
- Handle errors better when the logger could not be initialized.
- `ParserBot`:
- For the csv parsing methods, `ParserBot.csv_params` is now used for all these methods.
- `ParserBot.parse_csv_dict` now saves the field names in `ParserBot.csv_fieldnames`.
- `ParserBot.parse_csv_dict` now saves the raw current line in `ParserBot.current_line`.
- `ParserBot.recover_line_csv_dict` now uses the raw current line.
- lib/message:
- Subitems in fields of type `JSONDict` (see below) can be accessed directly. E.g. you can do:
event['extra.foo'] = 'bar'
event['extra.foo'] gives 'bar'
It is still possible to set and get the field as whole, however this may be removed or changed in the future:
event['extra'] = '{"foo": "bar"}'
event['extra'] gives '{"foo": "bar"}'
"Old" bots and configurations compatible with 1.0.x do still work.
Also, the extra field is now properly exploded when exporting events, analogous to all other fields.
The `in` operator works now for both - the old and the new - behavior.
- `Message.add`: The parameter `overwrite` accepts now three different values: `True`, `False` and `None` (new).
True: An existing value will be overwritten
False: An existing value will not be overwritten (previously an exception has been raised when the value was given).
None (default): If the value exists an `KeyExists` exception is thrown (previously the same as False).
This allows shorter code in the bots, as an 'overwrite' configuration parameter can be directly passed to the function.
- The message class has now the possibility to return a default value for non-existing fields, see `Message.set_default_value`.
- Message.get behaves the same like `Message.__getitem__` (1305).
- Add `RewindableFileHandle` to utils making handling of CSV files more easy (optionally)
- lib/pipeline:
- you may now define more than one destination queues path the bot should pass the message to, see [Pipelines](https://github.com/certtools/intelmq/blob/develop/docs/User-Guide.md#pipeline-configuration) (1088, 1190).
- the special path `"_on_error"` can be used to pass messages to different queues in case of processing errors (1133).
- `lib/harmonization`: Accept `AS` prefix for ASN values (automatically stripped).
- added `intelmq.VAR_STATE_PATH` for variable state data of bots.

Bots
- Removed print statements from various bots.
- Replaced various occurrences of `self.logger.error()` + `self.stop()` with `raise ValueError`.

Collectors
- `bots.collectors.mail`:
- New parameters; `sent_from`: filter messages by sender, `sent_to`: filter messages by recipient
- More debug logs
- `bots.collectors.n6.collector_stomp`: renamed to `bots.collectors.stomp.collector` (716)
- bots.collectors.rt:
- New parameter `search_requestor` to search for field Requestor.
- Empty strings and `null` as value for search parameters are ignored.
- Empty parameters `attachment_regex` and `url_regex` handled.
- `bots.collectors.http.collector_http`: Ability to optionally use the current time in parameter `http_url`, added parameter `http_url_formatting`.
- `bots.collectors.stomp.collector`: Heartbeat timeout is now logged with log level info instead of warning.
- added `intelmq.bots.collectors.twitter.collector_twitter`
- added `intelmq.bots.collectors.tcp.collector` that can be bound to another IntelMQ instance by a TCP output
- `bots.collectors.microsoft.collector_interflow`: added for MS interflow API
- Automatic ungzipping for .gz files.
- added `intelmq.bots.collectors.calidog.collector_certstream` for collecting certstream data (1120).
- added `intelmq.bots.collectors.shodan.collector_stream` for collecting shodan stream data (1096).
- Add proxy support.
- Fix handling of parameter `countries`.

Parsers
- `bots.parsers.shadowserver`:
- changed feednames. Please refer to it's README for the exact changes.
- If the conversion function fails for a line, an error is raised and the offending line will be handled according to the error handling configuration.
Previously errors like these were only logged and ignored otherwise.
- add support for the feeds
- `Accessible-Hadoop` (1231)
- `Accessible ADB` (1285)
- Remove deprecated parameter `override`, use `overwrite` instead (1071).
- The `raw` values now are exactly the input with quotes unchanged, the ParserBot methods are now used directly (1011).
- The Generic CSV Parser `bots.parsers.generic.parser_csv`:
- It is possible to filter the data before processing them using the new parameters `filter_type` and `filter_text`.
- It is possible to specify multiple columns using `|` character in parameter `columns`.
- The parameter `time_format` now supports `'epoch_millis'` for seconds since the Epoch, milliseconds are supported but not used.
- renamed `bots.parsers.cymru_full_bogons.parser` to `bots.parsers.cymru.parser_full_bogons`, compatibility shim will be removed in version 2.0
- added `bots.parsers.cymru.parser_cap_program`
- added `intelmq.bots.parsers.zoneh.parser` for ZoneH feeds
- added `intelmq.bots.parsers.sucuri.parser`
- added `intelmq.bots.parsers.malwareurl.parser`
- added `intelmq.bots.parsers.threatminer.parser`
- added `intelmq.bots.parsers.webinspektor.parser`
- added `intelmq.bots.parsers.twitter.parser`
- added `intelmq.bots.parsers.microsoft.parser_ctip`
- ignore the invalid IP '0.0.0.0' for the destination
- fix the raw/dumped messages, did not contain the paling list previously.
- use the new harmonization field `tlp` instead of `extra.tlp`.
- `bots.parsers.alienvault.parser_otx`: Save TLP data in the new harmonization field `tlp`.
- added `intelmq.bots.parsers.openphish.parser_commercial`
- added `intelmq.bots.parsers.microsoft.parser_bingmurls`
- added `intelmq.bots.parsers.calidog.parser_certstream` for parsing certstream data (1120).
- added `intelmq.bots.parsers.shodan.parser` for parsing shodan data (1096).
- change the classification type from 'botnet drone' to 'infected system' in various parses.
- `intelmq.bots.parsers.spamhaus.parser_cert`: Added support for all known bot types.

Experts
- Added sieve expert for filtering and modifying events (1083)
- capable of distributing the event to appropriate named queues
- `bots.experts.modify`
- default rulesets: all malware name mappings have been migrated to the [Malware Name Mapping repository](https://github.com/certtools/malware_name_mapping) ruleset. See the new added contrib tool for download and conversion.
- new parameter `case_sensitive` (default: True)
- Added wait expert for sleeping
- Added domain suffix expert to extract the TLD/Suffix from a domain name.
- `bots.experts.maxmind_geoip`: New (optional) parameter `overwrite`, by default false. The current default was to overwrite!
- `intelmq.bots.experts.ripencc_abuse_contact`:
- Extend deprecated parameter compatibility `query_ripe_stat` until 2.0 because of a logic bug in the compatibility code, use `query_ripe_stat_asn` and `query_ripe_stat_ip` instead (1071, 1291).
- Handle HTTP status code 404 for DB AS queries.
- Add caching capability.
- `intelmq/bots/experts/asn_lookup/update-asn-data`: Errors produce proper output on stdout/stderr.
- `intelmq/bots/experts/maxmind_geoip/update-geoip-data`: Errors produce proper output on stdout/stderr.
- `intelmq/bots/experts/tor_nodes/update-tor-nodes`: Errors produce proper output on stdout/stderr.

Outputs
- `bots.outputs.file`:
- String formatting can be used for file names with new parameter `format_filename`.
- New parameter `single_key` to only save one field.
- New parameter `encoding_errors_mode` with default value `'strict'` to handle encoding errors for the files written.

Harmonization
- Renamed `JSON` to `JSONDict` and added a new type `JSON`. `JSONDict` saves data internally as JSON, but acts like a dictionary. `JSON` accepts any valid JSON.
- fixed regex for `protocol.transport` it previously allowed more values than it should have.
- New ASN type. Like integer but checks the range.
- added `destination.urlpath` and `source.urlpath` to harmonization.
- New field `tlp` for tlp level specification.
- New TLP type. Allows all four tlp levels, removes 'TLP:' prefix and converts to upper case.
- Added new `classification.type` 'vulnerable client'
- Added `(destination|source).domain_suffix` to hold the TLD/domain suffix.
- New allowed value for `classification.type`: `infected system` for taxonomy `malicious code` (1197).

Requirements
- Requests is no longer listed as dependency of the core. For depending bots the requirement is noted in their `REQUIREMENTS.txt` file.

Documentation
- Use Markdown for README again, as pypi now supports it.
- Developers Guide: Add instructions for pre-release testing.

Packaging
- Add logcheck configuration to the packages.
- Fix packaging of bash completion script.

Tests
- Travis now correctly stops if a requirement could not be installed (1257).
- New tests for validating `etc/feeds.yaml` and `bots/BOTS` using cerberus and schemes are added (1166).
- New test for checking if `docs/Feeds.md` is up to date with `etc/feeds.yaml`.

Known bugs
- contrib: feeds-config-generator does not add feed name as parameter (1314).
- bot debugger requires configured source pipeline (1307).
- shadowserver parser: drone feed has spam events (1271).
- debug log level on python 3.7 not applied (1269).
- `bots.experts.sieve` does not support textX (1246).
- Bots started with IntelMQ-Manager stop when the webserver is restarted (952).

---------------------------------

Bots
Collectors
- `bots.collectors.rt.collector_rt`: Log ticket id for downloaded reports.

Parsers
- `bots.parsers.shadowserver`:
- if required fields do not exist in data, an exception is raised, so the line will be dumped and not further processed.
- fix a bug in the parsing of column `cipher_suite` in ssl poodle reports (1288).

Experts
- Reverse DNS Expert: ignore all invalid results and use first valid one (1264).
- `intelmq/bots/experts/tor_nodes/update-tor-nodes`: Use check.torproject.org as source as internet2.us is down (1289).

Outputs
- `bots.output.amqptopic`:
- The default exchange must not be declared (1295).
- Unencodable characters are prepended by backslashes by default. Otherwise Unicode characters can't be encoded and sent (1296).
- Gracefully close AMQP connection on shutdown of bot.

Documentation
- Bots: document redis cache parameters.
- Installation documentation: Ubuntu needs universe repositories.

Packaging
- Dropped support for Ubuntu 17.10, it reached its End of Life as of 2018-07-19.

Tests
- Drop tests for Python 3.3 for the mode with all requirements, as some optional dependencies do not support Python 3.3 anymore.
- `lib.test`: Add parameter `compare_raw` (default: `True`) to `assertMessageEqual`, to optionally skip the comparison of the raw field.
- Add tests for RT collector.
- Add tests for Shadowserver Parser:
- SSL Poodle Reports.
- Helper functions.

Tools
- `intelmqctl list` now sorts the output of bots and queues (1262).
- `intelmqctl`: Correctly handle the corner cases with collectors and outputs for getting/sending messages in the bot debugger (1263).
- `intelmqdump`: fix ordering of dumps in a file in runtime. All operations are applied to a sorted list (1280).

Contrib
- `cron-jobs/update-tor-nodes`: Use check.torproject.org as source as internet2.us is down (1289).

Known issues
- shadowserver parser: drone feed has spam events (1271).

---------------------------------

Core
- `lib/message`: `Report()` can now create a Report instance from Event instances (1225).
- `lib/bot`:
- The first word in the log line `Processed ... messages since last logging.` is now adaptable and set to `Forwarded` in the existing filtering bots (1237).
- Kills oneself again after proper shutdown if the bot is XMPP collector or output (970). Previously these two bots needed two stop commands to get actually stopped.
- `lib/utils`: log: set the name of the `py.warnings` logger to the bot name (1184).

Harmonization
- Added new types `unauthorized-command` and `unauthorized-login` to `intrusions` taxonomy.

Bots
Collectors
- `bots.collectors.mail.collector_mail_url`: handle empty downloaded reports (988).
- `bots.collectors.file.collector_file`: handle empty files (1244).

Parsers
- Shadowserver parser:
- SSL FREAK: Remove optional column `device_serial` and add several new ones.
- Fixed HTTP URL parsing for multiple feeds (1243).
- Spamhaus CERT parser:
- add support for `smtpauth`, `l_spamlink`, `pop`, `imap`, `rdp`, `smb`, `iotscan`, `proxyget`, `iotmicrosoftds`, `automatedtest`, `ioturl`, `iotmirai`, `iotcmd`, `iotlogin` and `iotuser` (1254).
- fix `extra.destination.local_port` -> `extra.source.local_port`.

Experts
- `bots.experts.filter`: Pre-compile regex at bot initialization.

Tests
- Ensure that the bots did process all messages (291).

Tools
- `intelmqctl`:
- `intelmqctl run` has a new parameter `-l` `--loglevel` to overwrite the log level for the run (1075).
- `intelmqctl run [bot-id] message send` can now send report messages (1077).
- `intelmqdump`:
- has now command completion for bot names, actions and queue names in interactive console.
- automatically converts messages from events to reports if the queue the message is being restored to is the source queue of a parser (1225).
- is now capable to read messages in dumps that are dictionaries as opposed to serialized dicts as strings and does not convert them in the show command (1256).
- truncated messages are no longer used/saved to the file after being shown (1255).
- now again denies recovery of dumps if the corresponding bot is running. The check was broken (1258).
- now sorts the dump by the time of the dump. Previously, the list was in random order (1020).

Known issues
no known issues

---------------------------------
- make code style compatible to pycodestyle 2.4.0
- fixed permissions of some files (they were executable but shouldn't be)

Core
- lib/harmonization:
- FQDN validation now handles None correctly (raised an Exception).
- Fixed several sanitize() methods, the generic sanitation method were called by is_valid, not the sanitize methods (1219).

Bots
* Use the new pypi website at https://pypi.org/ everywhere.

Parsers
- Shadowserver parser:
* The fields `url` and `http_url` now handle HTTP URL paths and HTTP requests for all feeds (1204).
* The conversion function `validate_fqdn` now handles empty strings correctly.
* Feed 'drone (hadoop)':
* Correct validation of field `cc_dns`, will now only be added as `destination.fqdn` if correct FQDN, otherwise ignored. Previously this field could be saved in extra containing an IP address.
* Adding more mappings for added columns.
* Added feeds:
* Drone-Brute-Force
* IPv6-Sinkhole-HTTP-Drone
* A lot of newly added fields and fixed conversions.
* Optional fields can now use one column multiple times.
* Add newly added columns of `Ssl-Scan` feed to parser
- Spamhaus CERT parser:
* fix parsing and classification for bot names 'openrelay', 'iotrdp', 'sshauth', 'telnetauth', 'iotcmd', 'iotuser', 'wpscanner', 'w_wplogin', 'iotscan'
see the NEWS file - Postgresql section - for all changes.
- CleanMX phishing parser: handle FQDNs in IP column (1162).

Experts
- `bots.experts.ripencc_abuse_contact`: Add existing parameter `mode` to BOTS file.

Tools
- intelmqctl check: Fixed and extended message for 'run_mode' check.
- `intelmqctl start` botnet. When using `--type json`, no non-JSON information about wrong bots are output because that would confuse eg. intelmq-manager

Tests
- lib/bot: No dumps will be written during tests (934).
- lib/test: Expand regular expression on python version to match pre-releases (debian testing).

Packaging
* Static data is now included in source tarballs, development files are excluded

Known issues
- `bots.collectors/outputs.xmpp` must be killed two times (970).
- When running bots with `intelmqctl run [bot-id]` the log level is always INFO (1075).
- `intelmqctl run [bot-id] message send [msg]` does only support Events, not Reports (1077).
- A warning issued by the python warnings module is logged without the bot-id (1184).