Intelmq

------------------

Configuration
- Default configuration:
- Remove discontinued feed "Feodo Tracker Domains" from default configuration.
- Add "Feodo Tracker Browse" feed to default configuration.

Core
- `intelmq.lib.pipeline`: AMQP: using port 15672 as default (like RabbitMQ's defaults) for the monitoring interface for getting statistical data (`intelmqctl_rabbitmq_monitoring_url`).
- `intelmq.lib.upgrades`: Added a generic upgrade function for harmonization, checking of all message types, it's fields and their types.
- `intelmq.lib.utils`:
- `TimeoutHTTPAdapter`: A subclass of `requests.adapters.HTTPAdapter` with the possibility to set the timeout per adapter.
- `create_request_session_from_bot`: Use the `TimeoutHTTPAdapter` with the user-defined timeout. Previously the timeout was not functional.

Bots
Parsers
- `intelmq.bots.parsers.shadowserver.parser`: Fix logging message if the parameter `feedname` is not present.
- `intelmq.bots.parsers.shodan.parser`: Also add field `classification.identifier` (`'network-scan'`) in minimal mode.
- `intelmq.bots.parsers.spamhaus.parser_cert`: Add support for category `'misc'`.
- `intelmq.bots.parsers.cymru.parser_cap_program`:
- Add support for phishing events without URL.
- Add support for protocols >= 143 (unassigned, experiments, testing, reserved), saving the number to extra, as the data would be bogus.
- `intelmq.bots.parsers.microsoft.parser_bingmurls`:
- Save the `Tags` data as `source.geolocation.cc`.

Experts
- `intelmq.bots.experts.modify.expert`: Fix bug with setting non-string values (1460).

Outputs
- `intelmq.bots.outputs.smtp`:
- Allow non-existent field in text formatting by using a default value `None` instead of throwing errors.
- Fix Authentication (1464).
- Fix sending to multiple recipients (1464).

Documentation
- Feeds:
- Fix configuration of `Feodo Tracker Browse` feed.
- Bots:
- Sieve expert: Document behavior of `!=` with lists.

Tests
- Adaption and extension of the test cases to the changes.

Tools
- `intelmq.bin.intelmqctl`:
- check: Check if running the upgrade function for harmonization is necessary.
- upgrade-config: Run the upgrade function for harmonization.
- `intelmqctl restart` did throw an error as the message for restarting was not defined (1465).

Known issues
- MongoDB authentication: compatibility on different MongoDB and pymongo versions (1439)
- ctl: shell colorizations are logged (1436)
- http stream collector: retry on regular connection problems? (1435)
- tests: capture logging with context manager (1342)
- Bots started with IntelMQ-Manager stop when the webserver is restarted. (952)
- n6 parser: mapping is modified within each run (905)
- reverse DNS: Only first record is used (877)
- Corrupt dump files when interrupted during writing (870)

-----------------

Core
- `intelmq.lib.harmonization`:
- Use correct parent classes.
- Add `DateTime.convert` as interface for all existing conversion functions.
- add `DateTime.convert_from_format`.
- add `DateTime.convert_from_format_midnight`.
- add `DateTime.convert_fuzzy`.
- `intelmq.lib.pipeline`:
- Redis: Use single connection client if calling bot is not multithreaded. Gives a small speed advantage.
- Require the bot instance as parameter for all pipeline classes.
- New internal variable `_has_message` to keep the state of the pipeline.
- Split receive and acknowledge into public-facing and private methods.
- Add `reject_message` method to the Pipeline class for explicit re-queue of messages.
- AMQP:
- Make exchange configurable.
- If exchange is set, the queues are not declared, the queue name is for routing used by exchanges.
- `intelmq.lib.bot`:
- Log message after successful bot initialization, no log message anymore for ready pipeline.
- Use existing current message if receive is called and the current message still exists.
- Fix handling of received messaged after a SIGHUP that happened during a blocking receiving connection using explicit rejection (1438).
- New method `_parse_common_parameters` called before `init` to parse commonly used argument. Currently supported: `extract_files`.
- `intelmq.lib.test`:
- Fix the tests broker by providing the testing pipeline.
- `intelmq.lib.utils`:
- `unzip`:
- new parameter `return_names` to optionally return the file names.
- support for zip
- new parameters `try_zip`, `try_gzip` and `try_tar` to control which compressions are tried.
- rewritten to an iterative approach
- add `file_name_from_response` to extract a file name from a Response object for downloaded files.
- `intelmq.lib.upgrades`: Added `v210_deprecations` for deprecated parameters.

Harmonization
- Add extra to reports.

Bots
Collectors
- `intelmq.bots.collectors.http.collector_http`:
- More extensive usage of `intelmq.lib.utils.unzip`.
- Save the file names in the report if files have been extracted form an archive.
- `intelmq.bots.collectors.rt.collector_rt`:
- Save ticket information/metadata in the extra fields of the report.
- Support for RT 3.8 and RT 4.4.
- New parameters `extract_attachment` and `extract_download` for generic archive extraction and consistency. The parameter `unzip_attachment` is deprecated.
- `intelmq.bots.collectors.mail.*`: Save email information/metadata in the extra fields of the report. See the bots documentation for a complete list of provided data.
- `intelmq.bots.collectors.mail.collector_mail_attach`: Check for existence/validity of the `attach_regex` parameter.
- Use the lib's `unzip` function for uncompressing attachments and use the .
- `intelmq.bots.collectors.mail.collector_mail_url`: Save the file name of the downloaded file as `extra.file_name`.
- `intelmq.bots.collectors.amqp.collector_amqp`: New collector to collect data from (remote) AMQP servers, for bot IntelMQ as well as external data.
- use default SSL context for client purposes, fixes compatibility with python `<` 3.6 if TLS is used.

Parsers
- `intelmq.bots.parsers.html_table.parser`:
* New parameter "html_parser".
* Use time conversion functions directly from `intelmq.lib.harmonization.DateTime.convert`.
- Limit lxml dependency on 3.4 to `<` 4.4.0 (incompatibility).
- `intelmq.bots.parsers.netlab_360.parser`: Add support for hajime scanners.
- `intelmq.bots.parsers.hibp.parser_callback`: A new parser to parse data retrieved from a HIBP Enterprise Subscription.
- `intelmq.bots.parsers.shadowserver.parser`:
- Ability to detect the feed base on the reports's field `extra.file_name`, so the parameter `feedname` is no longer required and one configured parser can parse any feed (1442).

Experts
- Add geohash expert.
- `intelmq.bots.experts.generic_db_lookup.expert`
- new optional parameter `engine` with `postgresql` (default) and `sqlite` (new) as possible values.

Outputs
- Add `intelmq.bots.outputs.touch.output`.
- `intelmq.bots.outputs.postgresql.output`:
- deprecated in favor of `intelmq.bots.outputs.sql.output`
- Compatibility shim will be available in the 2.x series.
- `intelmq.bots.outputs.sql.output` added generic SQL output bot. Comparted to
- new optional parameter `engine` with `postgresql` (default) and `sqlite` (new) as possible values.
- `intelmq.bots.outputs.stomp.output`: New parameters `message_hierarchical`, `message_jsondict_as_string`, `message_with_type`, `single_key`.

Documentation
- Feeds:
- Add ViriBack feed.
- Add Have I Been Pwned Enterprise Callback.
- `intelmq.tests.bots.outputs.amqptopic.test_output`: Added.
- Move the documentation of most bots from separate README files to the central Bots.md and feeds.yaml files.

Tests
- Travis:
- Use UTC timezone.
- Tests for `utils.unzip`.
- Add a new asset: Zip archive with two files, same as with `.tar.gz` archive.
- Added tests for the Mail Attachment & Mail URL collectors.
- Ignore logging-tests on Python 3.7 temporarily (1342).

Tools
- intelmqctl:
- Use green and red text color for some interactive output to indicate obvious errors or the absence of them.
- intelmqdump:
- New edit action `v` to modify a message saved in the dump (1284).

Contrib
* malware name mapping:
* Add support for MISP treat actors data, see it's README for more information.
* And handle empty synonyms in misp's galxies data.
* Move apply-Script to the new EventDB directory
* EventDB: Scripts for applying malware name mapping and domain suffixes to an EventDB.

Known issues
- MongoDB authentication: compatibility on different MongoDB and pymongo versions (1439)
- ctl: shell colorizations are logged (1436)
- http stream collector: retry on regular connection problems? (1435)
- tests: capture logging with context manager (1342)
- Bots started with IntelMQ-Manager stop when the webserver is restarted. (952)
- n6 parser: mapping is modified within each run (905)
- reverse DNS: Only first record is used (877)
- Corrupt dump files when interrupted during writing (870)

-----------------

Core
- `intelmq.lib.bot.CollectorBot`: Support the deprecated parameter `feed` until version 2.2 as the documentation was not properly updated (1445).
- `intelmq.lib.bot.Bot`:
- `_dump_message`: Wait for up to 60 seconds instead of 50 if the dump file is locked (the log message was said 60, but the code was for only 50).
- `intelmq.lib.upgrades.v202_fixes`
- Migration of deprecated parameter `feed` for Collectors.
- Ripe expert parameter `query_ripe_stat_ip` was not correctly configured in `v110_deprecations`, now use `query_ripe_stat_asn` as default if it does not exist.
- `intelmq.lib.upgrades.v110_deprecations`: Fix upgrade of ripe expert configuration.
- `intelmq.lib.bot_debugger`:
- Fix handling of empty messages generated by parser when user wanted to show the result by "--show-sent" flag.
- Fix handling of sent messages for bots using the `path_permissive` parameter (1453).
- `intelmq.lib.pipeline.Amqp`:
- use default SSL context for client purposes, fixes compatibility with python `<` 3.6 if TLS is used.
- Reconnect once on sending messages if disconnect detected.

Bots
Collectors
- `intelmq.bots.collectors.api.collector_api`:
- Handle non-existing IO loop in shutdown.
- Close socket on shutdown, fixes reloading.
- Marked as non-threadable.
- `intelmq.bots.collectors.rt.collector_rt`: Check for matching URLs if no `attachment_regex` is given.
- `intelmq.bots.collectors.stomp.collector_stomp`: Handle disconnects by actively reconnecting.

Parsers
- `intelmq.bots.cymru.parser_cap_program`: Fix parsing of the new `$certname_$date.txt` report format (1443):
- Support protocol ICMP.
- Fix error message for unsupported protocols.
- Support fields `destination_port_numbers`, `port`.
- Support for all proxy types without ports.
- Use Country Code of AS as `source.geolocation.cc`.
- Support for 'scanner' and 'spam' categories.
- Handle bogus lines with missing separator.
- Fix bug preventing use of old format after using the new format.
- Handle postfix ` (total_count:..)` for destination port numbers.

Experts
- `intelmq.bots.experts.cymru_whois.expert`: Add optional parameter `overwrite`, current behavior was `True`, default if not given is `True` now, will change to `False` in 3.0.0 (1452, 1455).
- `intelmq.bots.experts.modify.expert`: Add optional parameter `overwrite`, current behavior was `True`, default if not given is `True` now, will change to `False` in 3.0.0 (1452, 1455).
- `intelmq.bots.experts.reverse_dns.expert`: Add optional parameter `overwrite`, current behavior was `True`, default if not given is `True` now, will change to `False` in 3.0.0 (1452, 1455).

Outputs
- `intelmq.bots.outputs.amqptopic.output`: use default SSL context for client purposes, fixes compatibility with python `<` 3.6 if TLS is used.

Packaging
- Rules:
- Exclude intelmqsetup tool in packages
- Include update-rfiprisk-data in packages

Tests
- Tests for `intelmq.lib.upgrades.v202_fixes`.
- Tests for `intelmq.lib.upgrades.v110_deprecations`.
- Extended tests for `intelmq.bots.parser.cymru.parser_cap_program`.

Tools
- intelmqctl:
- More and more precise logging messages for botnet starting and restarting, enable and disable.
- No error message for disabled bots on botnet reload.
- Fix `upgrade-conf` is state file is empty or not existing.
- Use arpgarse's `store_true` action for flags instead of `store_const`.
- If the loading of the defaults configuration failed, a variable definition was missing and causing an exception (1456).

Contrib
- Check MK Statistics Cronjob:
- Use `statistics_*` parameters.
- Make file executable
- Handle None values in `*.temporary.*` keys and treat them as 0.
- systemd:
- Add `PIDFile` parameter to service file.

Known issues
- MongoDB authentication: compatibility on different MongoDB and pymongo versions (1439)
- ctl: shell colorizations are logged (1436)
- http stream collector: retry on regular connection problems? (1435)
- tests: capture logging with context manager (1342)
- Bots started with IntelMQ-Manager stop when the webserver is restarted. (952)
- n6 parser: mapping is modified within each run (905)
- reverse DNS: Only first record is used (877)
- Corrupt dump files when interrupted during writing (870)

------------------

Core
- `intelmq.lib.harmonization`:
- `IPAddress`: Remove Scope/Zone IDs for IPv6 addresses in sanitation.
- All types: Handle `None` for validation and sanitation gracefully.
- `intelmq.lib.bot`:
- fix parameters of ParserBot and CollectorBot constructors, allowing `intelmqctl run` with these bots again (1414).
- Also run `rate_limit` after retry counter reset (1431).
- `__version_info__`:
- is now available in the top level module.
- uses integer values now instead of strings for numerical version parts
- Also provide (empty) `ROOT_DIR` for non-pip installations.
- `intelmq.lib.upgrades`: New library file `upgrades` with upgrade functions.
- `intelmq.lib.utils`:
- New function `setup_list_logging` for intelmqctl check an possibly others.
- Fix return values (1423).
- New function `version_smaller` for version comparisons.
- New function `lazy_int` for version conversions.
- `parse_logline`: Handle thread IDs.
- `log` takes a new argument `logging_level_stream` for the logging level of the console handler.
- New constant `LOG_FORMAT_SIMPLE`, used by intelmqctl.
- New function `write_configuration` to write dicts to files in the correct json formatting.
- New function `create_request_session_from_bot`.
- `intelmq.lib.pipeline`:
- AMQP:
- Actually use `source/destination_pipeline_amqp_virtual_host` parameter.
- Support for SSL with `source/destination_pipeline_ssl` parameter.
- pipeline base class: add missing dummy methods.
- Add missing return types.
- Redis: Evaluate return parameter of queue/key deletion.
- Variable `STATE_FILE_PATH` added.

Development
- `intelmq.bin.intelmq_gen_docs`: For yaml use `safe_load` instead of unsafe `load`.

Harmonization
- IPAddress type: Remove Scope/Zone IDs for IPv6 addresses in sanitation.
- TLP: Sanitation handles now more cases: case-insensitive prefixes and arbitrary whitespace between the prefix and the value (1420).

Bots
Collectors
- `intelmq.bots.collectors.http.collector_http`: Use `utils.create_request_session_from_bot`.
- `intelmq.bots.collectors.http.collector_http_stream`: Use `utils.create_request_session_from_bot` and thus fix some retries on connection timeouts.
- `intelmq.bots.collectors.mail.collector_mail_url`: Use `utils.create_request_session_from_bot`.
- `intelmq.bots.collectors.microsoft.collector_interflow`: Use `utils.create_request_session_from_bot` and thus fix retries on connection timeouts.
- `intelmq.bots.collectors.rt.collector_rt`: Use `utils.create_request_session_from_bot` and thus fix retries on connection timeouts.
- `intelmq.bots.collectors.twitter.collector_twitter`: Use `utils.create_request_session_from_bot` and thus fix retries on connection timeouts for non-twitter connections.

Parsers
- `intelmq.bots.parsers.n6.parser_n6stomp`: use `malware-generic` instead of `generic-n6-drone` for unknown infected system events.
- `intelmq.bots.parsers.abusech.parser_ip`: Support LastOnline column in feodo feed (1400) and use it for `time.source` if available.
- Use lower case malware names as default, should not make a difference in practice.
- Fix handling of CSV header for feodotracker (1417, 1418).
- `intelmq.bots.parsers.netlab_360.parser`: Detect feeds with `https://` too.

Experts
- `intelmq.bots.experts.generic_db_lookup`: Recommend psycopg2-binary package.
- `intelmq.bots.experts.modify.expert`:
- Compile regular expressions (all string rules) at initialization, improves the speed.
- Warn about old configuration style deprecation.
- `intelmq.bots.experts.do_portal.expert`:
- Use `utils.create_request_session_from_bot` and thus fix retries on connection timeouts (1432).
- Treat "502 Bad Gateway" as timeout which can be retried.
- `intelmq.bots.experts.ripe.expert`: Use `utils.create_request_session_from_bot` and thus fix retries on connection timeouts.
- `intelmq.bots.experts.url2fqdn.expert`: Support for IP addresses in hostnames (1416).
- `intelmq.bots.experts.national_cert_contact_certat.expert`: Use `utils.create_request_session_from_bot` and thus fix retries on connection timeouts.

Outputs
- `intelmq.bots.outputs.postgresql`: Recommend psycopg2-binary package.
- `intelmq.bots.outputs.amqptopic`:
- Shutdown: Close connection only if connection exists.
- Add support for pika > 1. Pika changed the way it indicates (Non-)Acknowledgments of sent messages.
- Gracefully handle unroutable messages and give advice.
- Support for connections without authentication.
- Replace deprecated parameter `type` with `exchange_type` for `exchange_declare`, supporting pika >= 0.11 (1425).
- New parameters `message_hierarchical_output`, `message_with_type`, `message_jsondict_as_string`.
- New parameter `use_ssl` for SSL connections.
- New parameter `single_key` for sending single fields instead of the full event.
- `intelmq.bots.outputs.mongodb.output`: Support for pymongo >= 3.0.0 (1063, PR1421).
- `intelmq.bots.outputs.file`: `time.*` field serialization: support for microseconds.
- `intelmq.bots.outputs.mongodb.output`: Support for authentication in pymongo >= 3.5 (1062).
- `intelmq.bots.outputs.restapi.output`: Use `utils.create_request_session_from_bot` and thus fix retries on connection timeouts.

Documentation
- Add certbund-contact to the ecosystem document.
- Rename the IDEA expert to "IDEA Converter".
- Add the new configuration upgrade function to the docs.
- User Guide:
- Clarify on Uninstallation

Packaging
- Do not execute the tcp collector tests during Debian and Ubuntu builds as they fail there.

Tests
- `intelmq.lib.test`: Disable statistics for test runs of bots.
- `contrib.malware_name_mapping`: Added tests.
- Travis: Also run tests of contrib.

Tools
- `intelmqsetup`: Only change directory ownerships if necessary.
- `intelmqctl`:/**---
- Provide new command `upgrade-conf` to upgrade configuration to a newer version.
- Makes backups of configurations files on its own.
- Also checks for previously skipped or new functions of older versions and catches up.
- Provides logging level on class layer.
- Fix `-q` flag for `intelmqctl list queues` by renaming its alternative name to `--non-zero` to avoid a name collision with the global `--quiet` parameter.
- For console output the string `intelmqctl: ` at the beginning of each line is no longer present.
- `check`: Support for the state file added. Checks if it exists and all upgrade functions have been executed successfully.
- Waits for up to 2 seconds when stopping a bot (1434).
- Exits early on restart when stopping a bot did not work (1434).
- `intelmqctl run process -m` debugging: Mock acknowledge method if incoming message is mocked too, otherwise a different message is acknowledged.
- Queue listing for AMQP: Support non-default monitoring URLs, see User-Guide.

Contrib
* logcheck rules: Adapt ignore rule to cover the instance IDs of bot names.
* malware name mapping:
- Ignore lines in mapping starting with ''.
- Optionally include malpedia data.
- Fix command line parsing for not arguments (1427).
- bash-completion: Support for `intelmqctl upgrade-config` added.

Known issues
- http stream collector: retry on regular connection problems? (1435)
- tests: capture logging with context manager (1342)
- Bots started with IntelMQ-Manager stop when the webserver is restarted. (952)
- n6 parser: mapping is modified within each run (905)
- reverse DNS: Only first record is used (877)
- Corrupt dump files when interrupted during writing (870)

------------------

See also the changelog for 2.0.0.beta1 below.

Configurations
- Defaults: New parameters `statistics_host`, `statistics_port`, `statistics_databasae`, `statistics_password` for statistics redis database (1402).

Core
- Add more and fix some existing type annotations.
- `intelmq.lib.bot`:
- Use `statistics_*` parameters for bot's statistics (1402).
- Introduce `collector_empty_process` for collectors with an empty `process()` method, hardcoded 1s minimum sleep time, preventing endless loops, causing high load (1364).
- Allow to disable multithreading by initialization parameter, used by intelmqctl / the bot debugger (1403).
- `intelmq.lib.pipeline`: redis: OOM can also be low memory, add this to log message (1405).
- `intelmq.lib.harmonization`: ClassificationType: Update RSIT mapping (1380):
- replace `botnet drone` with `infected-system`
- replace `infected system` with `infected-system`
- replace `ids alert` with `ids-alert`
- replace `c&c` with `c2server`
- replace `malware configuration` with `malware-configuration`
- sanitize replaces these values on the fly
- Allow using non-opt/ (LSB) paths with environment variable `INTELMQ_PATHS_NO_OPT`.
- Disable/disallow threading for all collectors and some other bots.

Development
- Applied isort to all core files and core-related test files, sorting the imports there (every thing except bots and bots' tests).

Harmonization
- See the Core section for the changes in the allowed values for `classification.type`.

Bots
- Use the new RSIT types in several bots, see above

Parsers
- `intelmq.bots.parsers.spamhaus.parser_cert`: Added support for `extortion` events.

Experts
- added `intelmq.bots.experts.do_portal.expert`.

Outputs
- `intelmq.bots.outputs.elasticsearch.output`: Support for TLS added (1406).
- `intelmq.bots.outputs.tcp.output`: Support non-intelmq counterparts again. New parameter `counterpart_is_intelmq`, see NEWS.md for more information (1385).

Packaging
- Update IntelMQ path fix patch after `INTELMQ_PATHS_NO_OPT` introduction, provide `INTELMQ_PATHS_OPT` environment variable for packaged instances.

Tests
- `test_conf`: For yaml use `safe_load` instead of unsafe `load`.
- Travis: Switch distribution from trusty to xenial, adapt scripts.
- Add Python 3.7 to tests.
- Don't use Cerberus 1.3 because of https://github.com/pyeve/cerberus/issues/489
- Add tests for `intelmqctl.lib.upgrades`.

Tools
- intelmqdump: Fix creation of pipeline object by providing a logger.
- intelmqctl: Disable multithreading for interactive runs / the bot debugger (1403).

Known issues
- tests: capture logging with context manager (1342)
- pymongo 3.0 deprecates used insert method (1063)
- pymongo >= 3.5: authentication changes (1062)
- Bots started with IntelMQ-Manager stop when the webserver is restarted. (952)
- n6 parser: mapping is modified within each run (905)
- reverse DNS: Only first record is used (877)
- Corrupt dump files when interrupted during writing (870)

------------------------
There are some features considered as beta and marked as such in the documentation, do not use them in production yet.

Removals of deprecated code:
- Removed compatibility shim `intelmq.bots.collectors.n6.collector_stomp`, use `intelmq.bots.collectors.stomp.collector` instead (see 1124).
- Removed compatibility shim `intelmq.bots.parsers.cymru_full_bogons.parser`, use `intelmq.bots.parsers.cymru.parser_full_bogons` instead.
- Removed compatibility shim handling deprecated parameter `feed` for collectors. Use `name` instead.
- Removed deprecated and unused method `intelmq.lib.pipeline.Pipeline.sleep`.
- Removed support for deprecated parameter `query_ripe_stat` in `intelmq.bots.experts.ripe.expert`, use `query_ripe_stat_asn` and `query_ripe_stat_ip` instead (1291).
- Removed deprecated and unused function `intelmq.lib.utils.extract_tar`.

Core
- `lib/pipeline`:
- Allow setting the broker of source and destination independently.
- Support for a new AMQP broker. See User Guide for configuration. (1179)
- `lib/bot`:
- Dump messages locks the dump file using Unix file locks (574).
- Print idle/rate limit time also in human readable format (1332).
- `set_request_parameters`: Use `{}` as default proxy value instead of `None`. Allows updating of existing proxy dictionaries.
- Bots drop privileges if they run as root.
- Save statistics on successfully and failed processed messages in the redis database 3.
- `lib/utils`
- Function `unzip` to extract files from gz-zipped and/or tar-archives.
- New class `ListHandler`: new handler for logging purpose which saves the messages in a list.
- Add function `seconds_to_human`.
- Add function `drop_privileges`.
- `parse_relative`: Strip string before parsing.
- `parse_logline`: Do not convert the timestamps to UTC, leave them as is.
- `lib/cache`:
- Allow ttl to be None explicitly.
- Overwrite existing cache keys in the database instead of discarding the new data.
- `lib/bot`:
- Basic, but easy-to-configure multi-threading using python's `threading` library. See the User-Guide for more information (111, 186).
- `bin/intelmqctl`:
- Support for Supervisor as process manager (693, 1360).

Development
- upgraded all files to python3-only syntax, e.g. use `super()` instead of `super(..., ...)` in all files. Migration from old to new string formatting has not been applied if the resulting code would be longer.

Bots
Collectors
- added `intelmq.bots.parsers.opendxl.collector` (1265).
- added `intelmq.bots.collectors.api`: collecting data using an HTTP API (123, 1187).
- added `intelmq.bots.collectors.rsync` (1286).
- `intelmq.bots.collectors.http.collector_http`:
- Add support for uncompressing of gz-zipped-files (1270).
- Add time-delta support for time formatted URLs (1366).
- `intelmq.collectors.blueliv.collector_crimeserver`: Allow setting the API URL by parameter (1336).
- `intelmq.collectors.mail`:
- Use internal lib for functionality.
- Add `intelmq.bots.collectors.mail.collector_mail_body`.
- Support for `ssl_ca_certificate` parameter (1362).

Parsers
- added `intelmq.bots.parsers.mcafee.parser_atd` (1265).
- `intelmq.bots.parsers.generic.parser_csv`:
- New parameter `columns_required` to optionally ignore parse errors for columns.
- added `intelmq.bots.parsers.cert_eu.parser_csv` (1287).
- Do not overwrite the local `time.observation` with the data from the feed. The feed's field 'observation time' is now saved in the field `extra.cert_eu_time_observation`.
- Fix parsing of `asn` (renamed to `source asn`, `source.asn` internally) and handle existing `feed.accuracy` for parsing `confidence`.
- Update columns and mapping to current (2019-04-02) data.
- added `intelmq.bots.parsers.surbl.surbl`
- added `intelmq.bots.parsers.html_table` (1381).
- `intelmq.bots.parsers.netlab_360.parser`: Handle empty lines containing blank characters (1393).
- `intelmq.bots.parsers.n6.parser_n6stomp`: Handle events without IP addresses.
- `intelmq.bots.parsers.cymru.parser_cap_program`: Handle new feed format.
- `intelmq.bots.parsers.shadowserver`:
- Add support for the `Accessible-FTP` feed (1391).
- `intelmq.bots.parsers.dataplane.parser`:
- Fix parse errors and log more context (1396).
- added `intelmq.bots.parsers.fraunhofer.parser_ddosattack_cnc.py` and `intelmq.bots.parsers.fraunhofer.parser_ddosattack_target.py` (1373).

Experts
- added `intelmq.bots.experts.recordedfuture_iprisk` (1267).
- added `intelmq.bots.experts.mcafee.expert_mar` (1265).
- renamed `intelmq.bots.experts.ripencc_abuse_contact.expert` to `intelmq.bots.experts.ripe.expert`, compatibility shim will be removed in version 3.0.
- Added support for geolocation information in ripe expert with a new parameter `query_ripe_stat_geolocation` (1317).
- Restructurize the expert and code de-duplicataion (1384).
- Handle '?' in geolocation country data (1384).
- `intelmq.bots.experts.ripe.expert`:
- Use a requests session (1363).
- Set the requests parameters once per session.
- `intelmq.bots.experts.maxmind_geoip.expert`: New parameter `use_registered` to use the registered country (1344).
- `intelmq.bots.experts.filter.expert`: Support for paths (1208).

Outputs
- added `intelmq.bots.experts.mcafee.output_esm` (1265).
- added `intelmq.bots.outputs.blackhole` (1279).
- `intelmq.bots.outputs.restapi.expert`:
- Set the requests parameters once per session.
- `intelmq.bots.outputs.redis`:
- New parameter `hierarchichal_output` (1388).
- New parameter `with_type`.
- `intelmq.bots.outputs.amqptopic.output`: Compatibility with pika 1.0.0 (1084, 1394).

Documentation
- added documentation for feeds
- CyberCrime Tracker
- Feodo Tracker Latest
- Feeds: Document abuse.ch URLhaus feed (1379).
- Install and Upgrading: Use `intelmqsetup` tool.
- Added an ecosystem overview document describing related software.

Tests
- Add tests of AMQP broker.
- Travis: Change the ownership of `/opt/intelmq` to the current user.

Tools
- `intelmqctl check`: Now uses the new `ListHandler` from utils to handle the logging in JSON output mode.
- `intelmqctl run`: The message that a running bot has been stopped, is not longer a warning, but an informational message. No need to inform sysadmins about this intended behavior.
- `intelmqdump`: Inspecting dumps locks the dump file using unix file locks (574).
- `intelmqctl`:
- After the check if the program runs as root, it tries to drop privileges. Only if this does not work, a warning is shown.
- `intelmqsetup`: New tool for initializing an IntelMQ environment.

Contrib
- `malware_name_mapping`:
- Added the script `apply_mapping_eventdb.py` to apply the mapping to an EventDB.
- Possibility to add local rules using the download tool.
- `check_mk`:
- Added scripts for monitoring queues and statistics.

Known issues
- Multi-threaded bots require multiple SIGTERMs (1403)
- Stats can't be saved with AMQP if redis is password-protected (1402)
- Update taxonomies to current RSIT and vice-versa (1380)
- stomp collector bot constantly uses 100% of CPU (1364)
- tests: capture logging with context manager (1342)
- Consistent message counter log messages for all kind of bots (1278)
- pymongo 3.0 deprecates used insert method (1063)
- pymongo >= 3.5: authentication changes (1062)
- Bots started with IntelMQ-Manager stop when the webserver is restarted. (952)
- n6 parser: mapping is modified within each run (905)
- reverse DNS: Only first record is used (877)
- Corrupt dump files when interrupted during writing (870)