Intelmq

---------------------------------
Contrib
* logrotate: use sudo for postrotate script
* cron-jobs: use the scripts in the bots' directories and link them (1056, 1142)

Core
- `lib.harmonization`: Handle idna encoding error in FQDN sanitation (1175, 1176).
- `lib.bot`:
- Bots stop when redis gives the error "OOM command not allowed when used memory > 'maxmemory'." (1138).
- warnings of bots are caught by the logger (1074, 1113).
- Fixed exitcodes 0 for graceful shutdowns .
- better handling of problems with pipeline and especially it's initialization (1178).
- All parsers using `ParserBot`'s methods now log the sum of successfully parsed and failed lines at the end of each run (1161).

Harmonization
- Rule for harmonization keys is enforced (1104, 1141).
- New allowed values for `classification.type`: `tor` & `leak` (see n6 parser below ).

Bots
Collectors
- `bots.collectors.mail.collector_mail_attach`: Support attachment file parsing for imbox versions newer than 0.9.5 (1134).

Parsers
- All CSV parsers ignore NULL-bytes now, because the csv-library cannot handle it (967, 1114).
- `bots.parsers.shadowserver.parser`: Add Accessible Cisco Smart Install (1122).
- `bots.parsers.cleanmx.parser`: Handle new columns `first` and `last`, rewritten for XML feed. See NEWS.md for upgrade instructions (1131, 1136, 1163).
- `bots.parsers.n6.parser`: Fix classification mappings. See NEWS file for changes values (738, 1127).

Experts
- `bots.experts.modify` default ruleset: changed conficker rule to catch more spellings.

Outputs
- `bots.outputs.smtp.output`: Fix STARTTLS, threw an exception (1152, 1153).

Documentation
- `Release.md` add release procedure documentation
- `Bots.md`: fix example configuration for modify expert

Tools
- intelmqctl now exits with exit codes > 0 when errors happened or the operation was not successful. Also, the status operation exits with 1, if bots are stopped, but enabled. (977, 1143)
- `intelmctl check` checks for valid `run_mode` in runtime configuration (1140).

Tests
- `tests.lib.test_pipeline`: Redis tests clear all queues before and after tests (1086).
- Repaired debian package build on travis (1169).
- Warnings are not allowed by default, an allowed count can be specified (1129).
- `tests.bots.experts.cymru_whois/abusix`: Skipped on travis because of ongoing problems.

Packaging
* cron jobs: fix paths of executables

Known issues
- `bots.collectors/outputs.xmpp` must be killed two times (970).
- When running bots with `intelmqctl run [bot-id]` the log level is always INFO (1075).
- `intelmqctl run [bot-id] message send [msg]` does only support Events, not Reports (1077).
- `python3 setup.py sdist` does not include static files in the resulting tarballs (1146).
- `bots.parsers.cleanmx.parser`: The cleanMX feed may have FQDNs as IPs in rare cases, such lines are dumped (1162).

---------------------------------

Core
- `lib.message.add`: parameter force has finally been removed, should have been gone in 1.0.0.rc1 already

Bots
- `collectors.mail.collector_mail_url`: Fix bug which prevented marking emails seen due to disconnects from server (852).
- `parsers.spamhaus.parser_cert`: Handle/ignore 'AS?' in feed (1111)

Packaging
- The following changes have been in effect for the built packages already since version 1.0.0
- Support building for more distributions, now supported: CentOS 7, Debian 8 and 9, Fedora 25 and 26, RHEL 7, openSUSE Leap 42.2 and 42.3 and Tumbleweed, Ubuntu 14.04 and 16.04
- Use LSB-paths for created packages (/etc/intelmq/, /var/lib/intelmq/, /run/intelmq/) (470). Does does not affect installations with setuptools/pip.
- Change the debian package format from native to quilt
- Fix problems in postint and postrm scripts
- Use systemd-tmpfile for creation of /run/intelmq/

Documentation
- Add disclaimer on maxmind database in bot documentation and code and the cron-job (1110)

---------------------------------
Documentation
- Feeds: use more https:// URLs
- minor fixes

Bots
- bots/experts/ripencc_abuse_contact/expert.py: Use HTTPS URLs for rest.db.ripe.net
- bots/outputs/file/output.py: properly close the file handle on shutdown
- bots/parser/shadowserver: If conversion of a value via conversion function fails, only log the function name, not the representation string (1157).

Core
- lib/bot: Bots will now log the used intelmq version at startup

Tools
- intelmqctl: To check the status of a bot, the command line of the running process is compared to the actual executable of the bot. Otherwise unrelated programs with the same PID are detected as running bot.
- intelmqctl: enable, disable, check, clear now support the JSON output

---------------------------------
Core
- Fixes a thrown `FileNotFound` exception when stopping bots started with `intelmqctl run ...`

Harmonization
- leading dots in FQDNs are rejected and removed in sanitation (1022, 1030)

Bots
- shadowserver parser Accessible-SMB: smb_implant is converted to bool

----------------------------------------
Core
- Changing the value of an existing field to `None` deletes the field.
- `Message.update` now behaves like `dict.update`. The old behavior is implemented in `Message.change`
- Deprecated `http_ssl_proxy` has been dropped, use `https_proxy` instead
- Deprecated `http_timeout` has been dropped, use `http_timeout_sec` instead
- Deprecated parameters force and ignore of `Message.add` have been removed
- Deprecated method `Message.contains` has been removed
- Drop support for deprecated configuration files `startup.conf` and `system.conf`

Development
- We are now testing with and without optional libraries/lowest recommended versions and most current versions of required libraries
- Tests shadowserver with more data and checks for warnings and errors
- Tests: if bots log warnings this counts as failure if not allowed explicitly
- Tests: Bot preparation can be skipped

Documentation
- The branching/releasing mechanism has been documented

Bots
Collectors
- HTTP collectors: If `http_username` and `http_password` are both given and empty or null, 'None:None' has been used to authenticate. It is now checked that the username evaluates to non-false/null before adding the authentication. (fixes 1017)
- Dropped unmaintained and undocumented FTP(S) collectors `bots.collectors.ftp`. Also, the FTPS collector had a license conflict (842).
- `bots.collectors.http.collector_http_stream`: drop deprecated parameter `url` in favor of `http_url`

Parsers
- Removed bots.parsers.openbl as the source is offline since end of may (1018, https://twitter.com/sshblorg/status/854669263671615489)
- Removed bots.parsers.proxyspy as the source is offline (1031)
- Shadowserver: Added Accessible SMB
- `bots.experts.ripencc_abuse_contact` now has the two additional parameters `query_ripe_stat_asn` and `query_ripe_stat_ip`.
Deprecated parameter `query_ripe_stat`. New parameter `mode`.
- `bots.experts.certat_contact` has been renamed to `bots.experts.national_cert_contact_certat` (995)
- `bots.experts.cymru_whois` ignores registry `other` (996)
- `bots.parsers.alienvault.parser_otx`: handle timestamps without floating point seconds

Experts
- bots.experts.deduplicator: New parameter `bypass` to deactivate deduplication, default: False

-------------------------------------

General changes
- It's now configurable how often the bots are logging how much events they have sent, based on both the amount and time. (fixes 743)
- switch from pycodestyle to pep8

Configuration
- Added `log_processed_messages_count` (500) and `log_processed_messages_seconds` (900) to defaults.conf.
- `http_timeout` has been renamed to `http_timeout_sec` and `http_timeout_max_tries` has been added.
This setting is honored by `bots.collectors.http.*` and `bots.collectors.mail.collector_mail_url`, `bots.collectors.rt` (only `http_timeout_sec`), `bots.outputs.restapi.output` and `bots.experts.ripencc_abuse_contact`.

Documentation
- Minor fixes
- Dropped install scripts, see INSTALL.md for more detailed instructions and explanations
- Better structure of INSTALL.md
- Better documentation of packages

Tools
- added a bot debugger (975)
- missing bot executable is detected and handled by intelmqctl (979)

Core
- fix bug which prevented dumps to be written if the file did not exist (986)
- Fix reload of bots regarding logging
- type annotations for all core libraries

Bots
- added `bots.experts.idea`, bots.outputs.files
- possibility to split large csv Reports into Chunks, currently possible for mail url and file collector
- elasticsearch output supports HTTP Basic Auth
- `bots.collectors.mail.collector_mail_url` and bots collectors.file.collector can split large reports (680)
- `bots.parsers.shadowserver` support the VNC feed
- handling of HTTP timeouts, see above 859
- `bots.parsers.bambenek` saves the malware name
- `bots.parsers.fraunhofer.parser_dga` saves the malware name
- `bots.parsers.shadowserver` handles NULL bytes
- `bots.parsers.abusech.parser_ransomware` handles the IP 0.0.0.0 specially

Harmonization
- New field named `output` to support export to foreign formats