* web service
* flags:
- CHANGED:
- flag `--file` does not have anymore shortcut '-f'.
- `custom_field_modules` renamed to `external_fields`
- both INI and CLI flags
- delimiter flag
- quote_char flag
- new header / no-header flags
- new output CLI flag
- web, whois, nmap
- new flag single_query_ignored_fields
- flag --verbose, --quiet and config `verbosity`
- flag compute_preview
- user-agent
- headless
- multiple-hostname-ip 32
- whois_ttl 34
- CLI only flags
- --yes for skipping dialogues
- --config to open configuration
- --show-uml to get UML overview
- --json
- --field, --field-excluded, --delete, --split, --sort
- --single-query, --csv-processing
- --version
- --disable-external, --whois-delete
- INI only flags
- github_crash_submit
* fixes:
- fix: new installation should now correctly place contacts files in .config
- fix: refresh abroad contact list e-mails when restart even if the file has been processed before
- PyPi installer requirements fix
* Whois module
- won't throw error if host can't be resolved
- huge refactoring (may lead to the behaviour when Country is taken from ARIN and Netname from AfriNIC if AfriNIC points us to ARIN that doesn't state Netname)
- thousands of unique prefixes tried, many uses cases handled
- naive database of country names, so that country may be guessed from non-standardised "address" field
- some well known erroneous LIR responses are re-requested by their respective RIR
- CIDR with host bits set ( = invalid network) translates to IP when asking whois (RIR would return 'invalid search key')
- CIDR translated to an IP before asking whois
- number of discovered prefixes should during processing displayed, as well as real whois server URLs
- socket.gethostbyname non-existing domain exception caught
- Whois cache common for all convey files
- every cache record has TTL
- LACNIC quota exceeded marks the rows to be re-queued
- sets internal process locale to en_US so that we can grep the same result at environments using different language
* processing:
- logs in format "time IP src port > IP dst port" can be automatically transformed to CSV before processing
- multiline base64 and quoted_printable strings (seen in e-mails) may be input → automatically decoded
- methods may return a list (the row will be duplicated)
- implicitly convert to plaintext if possible when adding new column (ex: base64 will be implicitly decoded before its value being passed to a new reg column)
- decorator PickMethod, PickInput to allow the user choose the way generating will work
* internal
- os replaced by pathlib
- Config.getboolean merged to Config.get
- Identifier (former CSVGuesses) is now object oriented, no more spaghetti
- huge refactoring
- fix: log files
- Processor handles Whois statistics no more, so Identifier Whois methods do not return tuple anymore (standardised)
- advanced header detection
* interface
- able to automatically add/remove config file flags at program upgrade if allowed by user
- when no known method is known to process a field, an example is given
- when more than 9 options, you can use letters as shortcuts
- main screen shows colorized result, preferable in the form of table if the terminal is wide enough
- new 'code' field type for writing arbitrary code
- when not in debug mode, a GitHub issue is filled out automatically at crash
- autoopen_editor opens when analysis starts but now only if splitting by a column
- Config file startup integrity check: missing items and sections may be inserted automatically after an upgrade
- column sorting/picking in the main menu
- much more options to handle STDIN saving
* types
- reg type and its reg_m and reg_s
- dns types (a, aaaa, mx, spf, ns, dmarc, txt)
- tld
- web: text, http_status, html, redirects, x_frame_options, csp
- port, ports
- urlencode, quoted_printable
- charset, bytes
- timestamp, isotimestamp, time, date, formatted_time
- phone, unit