Ntopng

Breakthroughs

* Plugins engine to tap into flows, hosts and other network elements
* Migration to Bootstrap 4 and Font Awesome 5 for a renewed ntopng look-and-feel with light and dark themes
* Processes and containers monitoring thanks to the eBPF integration via libebpfflow https://github.com/ntop/libebpfflow
* Active monitoring of hosts ICMP/ICMPv6/HTTP/HTTPS Round Trip Times (RTT)

New features

* X.509 client certificate authentication
* ERSPAN transparent ethernet bridging
* Webhook export module for exporting alarms
* Identifications of the hosts in broadcast domain
* Category Lists editor to manage ip/domain lists
* Handling of PEN fields from nProbe
* Add anomalous flows to the looking glass
* Visibility of ICMP port-unreachable flows IPv4
* TCP states filtering (est., connecting, closed and rst)
* Ability to serialize local hosts in the broadcast domain via MAC address
* Japanese, portugese/brazilian localization
* Add process memory, cpu load, InfluxDB, Redis status pages and charts
* Implement ntopng Plugins, self contained modules to extend the ntopng functionalities
* Implement ZMQ/Suricata companion interface
* SSL traffic analysis and alerts via JA3 fingerprint, unsafe ciphers detection
* SSH traffic analysis and alerts via HASSH fingerprint
* Host traffic profile generation via the (MUD) Manufacturer Usage Descriptor
* Experimental Prometheus timeseries export
* Introduce the System interface to manage system wide settings and status
* Read events from Suricata and generate alerts
* SNMP network topology visualization
* Automatic ntopng update check and upgrade
* Calculate host anomaly score and trigger alerts when it exceeds a threshold
* Add ability to extract timeseries data with a click
* Initial Marketplace droplet using Fabric
* Alerts on duplex status change on SNMP interface

Improvements

* View interfaces are now optimized for big networks and use less memory
* Systemd macros are now used to start/restart the ntopng services
* Handles n2disk traffic extractions from recording processes non managed by ntopng
* Interface in/out now available also for non PF_RING interfaces (read from /proc)
* Automatic InfluxDB rollup support
* MDNS discovery improvements
* Rework of the alerts engine and api for efficient engaged alerts triggering
* Faster ZMQ communication to nProbe thanks to the implementation of a binary TLV format
* Stats update for ZMQ interfaces is now based on the idle/active flows timeout
* Timeseries export improvements via queues, detect if InfluxDB is down and stop the export
* Implemented reusable Lua engine to reduce the overhead of periodic scripts
* Improve Lua error handling
* Exclude certain categories from Elephant/Long lived flows alerts

nEdge

* Ability to set up port forwarding
* Support for Ubuntu 18.04
* Fix users and other prefs deleted during nEdge data reset
* Japanese localization
* Block unsupported L3 protocols (currently only ARP and IPv4 are supported)
* DNS mapping port to avoid conflicts with system programs

Fixes

* Fix export to mysql on shutdown in case of Pcap file in community mode
* Fix failing SYN-scan detection
* Fix ZMQ decompression errors with large templates
* Fix possible XSS in login.lua referer param and `runtime.lua`
* Update geolocation due to changes in the library usage policy
* Fix to support browsers dark mode
* Option `--zmq-encryption-key <pub key>` can be used with `-I <endpoint>` to encrypt data hi hierarchical mode
* Fix nIndex missing data while performing some queries and throughput calculation

----------------------------------------------------------------

New features

* Remote assistance to temporarily grant encrypted ntopng access to remote
parties
* Works with a transparent overlay-network spawned on-demand just
for the time necessary for the assistance
* Passes through firewalls and NATs
* https://www.ntop.org/ntopng/use-remote-assistance-to-connect-to-ntopng-instances/
* Custom URLs and IP addresses mappings to traffic categories
* Ability to associate websites (HTTP and HTTPS) to certain traffic
categories using their names
* Ability to use IP addresses (IPv4 and IPv6) to associate hosts to
traffic categories
* https://www.ntop.org/guides/ntopng/web_gui/categories.html?highlight=categories#custom-category-hosts
* Continuous traffic recording
* Interfaces with n2disk for the recording and extraction of traffic
* https://www.ntop.org/guides/ntopng/traffic_recording.html
* Download live pcap captures of monitored hosts and interfaces
* Delivers packets in pcap format over the web
* Works with single hosts, interfaces
* Allows BPF filters
* https://www.ntop.org/guides/ntopng/advanced_features/live_pcap_download.html?highlight=pcap#live-pcap-download
* User activities logging
* Records an alerts ntopng web users activities, including changes
in the configurations, deletion/addition of new users, login
attempts, and password changes.
* http://www.ntop.org/guides/ntopng/basic_concepts/alerts.html
* Extended chart metrics
* Relative-Strength Index (RSI)
* Moving and Exponentially-Moving Averages
* https://www.ntop.org/guides/ntopng/web_gui/historical.html

Improvements

* Alerts
* Scan-detection for remote hosts
* Configurable alerts for long-lived and elephant flows
* InfluxDB export failed alerts
* Remote-to-remote host alerts
* Optional JSON alerts export to Syslog
* Improve InfluxDB support
* Handles slow and aborted queries
* Uses authentication
* Adds RADIUS and HTTP authenticators
* Options to allow users login via RADIUS and HTTP
* https://www.ntop.org/ntopng/remote-ntopng-authentication-with-radius-and-ldap/
* Lua 5.3 support
* Improve performance
* Better memory management
* Native support for 64-bit integers
* Native support for bitwise operations
* Adds the new libmaxminddb geolocation library
* Storage utilization indicators
* Global storage indicator to show the disk used by each interface
* Per-interface storage indicator to show the disk used to store timeseries and flows
* Support for Sonicwall PEN field names
* Option to disable LDAP referrals
* Requests and configures Keepalive support for ZMQ sockets
* Three-way-handshake detection
* Adds SNMP mac addresses to the search function

nEdge

* Implement nEdge policies test page
* Implement device presets
* DNS
* Add more DNS servers
* Remove deprecated DNS


Fixes

* Fix missing flows dump on shutdown
* HTTP dissection fixes
* SNMP
* Fix SNMP step when high resolution timeseries are enabled
* Fix SNMP devices permissions to prevent non-admins to delete or add devices
* Properly handles endianness over ZMQ
* Fix early expiration of some TCP flows
* Fix non-deterministic expiration of flows

----------------------------------------------------------------

New features

* New pro charts
* Ability to compare data with the past (time shift)
* Trend lines based on ASAP
* Average and percentile lines overlayed on the graph and animated
* New color scheme that uses pastel colors for better visualization
* https://www.ntop.org/ntopng/ntopng-and-time-series-from-rrd-to-influxdb-new-charts-with-time-shift/
* New timeseries API with support for RRD and InfluxDB
* Abstracts and handles multiple sources transparently
* https://www.ntop.org/guides/ntopng/api/lua/timeseries/index.html
* Streaming pcap captures with BPF support
* Download live packet captures right from the browser
* New SNMP devices caching
* Periodically cache information of all the SNMP device configured
* Calculate and visualize interfaces throughput


Improvements

* Security
* Access to the web user interface is controlled with ACLs
* Secure ntopng cookies with SameSite and HttpOnly
* HTTP cookie authentication
* Improve random session id generation
* Various SNMP improvemenets
* Caching
* Interfaces status change alerts
* Device interfaces page
* Devices and interfaces added to flows
* Fix several library memory leaks
* Improve device and interface charts
* Interfaces throughput calculation and visualization
* Ability to delete all SNMP devices at once
* Improve active devices discovery
* OS detection via HTTP User-Agent
* Alerts
* Crypto miners alerts toggle
* Detection and alerting of anomalous terminations
* Module for sending telegram.org alerts
* Slack
* Configurable Slack channel names
* Add Slack test button
* Charts
* Active flows vs local hosts chart
* Active flows vs interface traffic chart
* Ubuntu 18.04 support
* Support for ElasticSearch 6 export
* Add support for custom categories lists
* Add ability to use the non-JIT Lua interpreter
* Improve ntopng startup and shutdown time
* Support for capturing from interface pairs with PF_RING ZC
* Support for variable PPP header lenght
* Migrated geolocation to GeoLite2 and libmaxminddb
* Configuration backup and restore
* Improve IE browser support
* Using client SSL certificate for protocol detection
* Optimized host/flows purging


nEdge

* Netfilter queue fill level monitoring
* Bridging support with VLANs
* Add user members management page
* Add systemd service alias to ntopng
* Captive portal fixes
* Informative captive portal (no login)
* Improve captive portal support with WISPr XML
* Disabled global DNS forging by default
* Add netfilter stats RRDs
* Fix bad MAC traffic increment
* Fix slow shutdown/reboot
* Fix invalid banned site redirection
* Fix bad gateway status
* Fix gateway network unreacheable when gateway is down
* Fix SSL traffic not blocked when captive portal is active
* Fix invalid read during local DNS lookup
* Workaround for dhclient bug stuck while a lease already exists


Fixes

* SNMP
* Fix SNMP devices deletion
* Fix format for odd SNMP interfaces speed
* Fix SNMP community selection
* Fix MDNS decoding
* Fix login redirection
* Fix MAC manufacturers escaping
* Fix host validation errors
* Fix traffic throughput burst when loading a serialized host
* Allowing multiple consecutive dots in password fields
* Rework shutdown to allow graceful periodic activities termimation
* Fix validation error in profiles with spaces in names
* Fix old top talkers stats deletion
* Fix 32-bit integers pushed to Lua
* Fix service dependency from pfring
* Fix for enabling broken SSL certificate mismatch alerts
* Fix allowed interfaces users access
* Fix for crashes on Windows
* Fix lua platform dependent execution
* Fix subnet search in hist data explorer
* Fix flow devices and sflow mappings with SNMP
* Fix invalid login page encoding
* LDAP fixes (overflow, invalid LDAP fields length)
* Fix encoding for local/LDAP UTF-8 passwords
* Add POST timeout to prevent housekeeping from blocking indefinitely
* Windows resize fixes
* Fix invalid uPnP URL
* Fix wrong hosts retrv by pool id, OS, network, and country
* Fix JS errors with IE browser
* Fix custom categories matching

----------------------------------------------------------------

New features

* Improve alerts generation
* Send alerts via email
* SNMP alerts on port status change
* Alerts at ntopng startup/shutdown
* ARP/IP re-assignments alerts
* Beta support for InfluxDB and Prometheus
* Multi-language support
* English
* Italian
* German
* "hide-from-top" to selectively hide hosts from top stats


Improvements

* Discovery with SSH scan and MDNS dissection
* HTML documentation with ReadTheDocs
* ERSPAN Type 2 detunneling
* per-AS network latency stats
* TCP KeepAlive stats
* Redis connection via Unix domain socket


Security Fixes

* Disables CGI support in mongoose
* Hardened options parsing


Fixes

* Fix memory leaks with SNMP
* Fix possible out-of-bounds reads with SSDP dissection

----------------------------------------------------------------

New features

* Support for the official ntopng Grafana datasource plugin
* Plugin available at: https://grafana.com/plugins/ntop-ntopng-datasource
* Newtork devices discovery
* Discovery of smartphones, laptops, IoT devices, routers, smart TVs, etc
* Device type and operating system detection
* ARP scan, SSDP dissection, Multicast DNS (MDNS) resolution
* DHCP fingerprinting
* Adds an active flows page to the AS details
* Bridge mode
* Enforcement of global per-pool time and byte quotas
* Support of per-host traffic shapers
* Add support for banned sites detection with informative splash screen
* Implement per-host/mac/pool flow drop count
* nDPI traffic categories and RRDs
* Implements MySQL database interoperability between ntopng and nProbe


Improvements

* Flows sent by nProbe over ZMQ:
* Batched, compressed ZMQ flow format to optimize data exchange
* Use of post-nat src/dst addresses and ports
* Handles multiple balanced ZMQ endpoints
* Periodic tasks performed by a thread-pool to optimize cores utilization
* Hosts and devices are walked in batches to greatly reduce Lua VM memory
* Full systemd support for Debian, Ubuntu, Centos, and Raspbian
* Extended sFlow support to include sample packet drops and counter stats in interface views
* Stacked applications and categories charts for ASes, Networks, etc

Security Fixes

* More restrictive permissions for created files and directories
* Fix of a possible dissectHTTP reads beyond end of payload

----------------------------------------------------------------

New features (Community)

* Layer-2 Devices
* MAC devices page
* Implement MAC last seen tracking in redis
* Manufacturer filter and sort
* Host pools (logical groups of hosts)
* Logstash flow export extension
* Implemented data anonymization: hosts and top sites
* Implements CPU load average and memory usage
* Virtual Interfaces
* ZMQ: disaggregate based on probeIP or ingress interfaceId
* Packet: disaggregate on VLANId
* ElasticSearch and MySQL flow export statistics
* Tiny Flows
* Alerts
* Implements alerts on a per-interface per-vlan basis
* Global alert thresolds for all local hosts/interfaces/local networks
* LUA alerts generation
* Adds hosts stateful syn attacks alerts
* Visualization/Retrieval of Host Alerts
* Add the ability to generate alert when ntopng detects traffic produced by malware hosts
* Slack integration: send alerts to slack
* Alerts for anomalous flows
* Host blacklisted alerts
* Alerts delete by type, older than, by host
* SSL certificates mismatch alerts generation
* Implement SSL/TLS handshake detection
* Integrated MSDN support
* Implemented DHCP dissection for name resolution

New features

* Traffic bridging
* Per host pool, per host pool member policies
* Per L7 protocol category policies
* Flashstart categories to block
* Time and Traffic quotas
* Support to google Safe Search DNS
* Ability to set custom DNS
* Captive portal
* Limited lifetime users
* Support for pc, kindle, android, ipad devices
* SNMP
* Periodic SNMP device monitoring and polling
* Historical SNMP timeseries
* Host-to-SNMP devices mapping
* Daily/Weekly/Monthly Traffic Report: per host, interface, network
* Add ability to define host blacklists
* DNS flow characterization with FlashStart (www.flashstart.it)
* Flow LUA scripts: on flow creation, protocol detected, expire
* Periodic MySQL flows aggregation
* Batched MySQL flows insertions
* sFlow device/interface counters
* Implementation of flow devices stats

Improvements

* Allows web server binding to system ports for non-privileged users
* Improve VLAN support
* Improve IPv6 support
* Implements a script to add users from the command line
* View interfaces rework
* Reported number of Layer-2 devices in ntopng footer
* Preferences re-organization and search
* Adds RIPE integration for Autonomous Systems
* Search host by custom name
* Move to the UTF-8 encoding
* Make real-time statics refresh time configurable (footer, dashboard)
* Adds support for localization (i18n)
* Traffic bridging: improved stability
* Traffic profiles: improved stability and data persistence
* Charts
* Improve historical graphs
* Traffic report rework and optimizations
* Improves the responsiveness and interactivity of historical exploration (ajax)
* Stacked top hosts
* Add ZMQ flows/sec graph
* Profiles graphs
* Implement ICMP detailed stats for local hosts
* ASN graphs: traffic and protocols history
* ARP requests VS replies sent and received by hosts
* Implement host TCP flags distribution
* DNS packets ratio
* FlashStart category graphs
* Add ARP protocol in interface statistics
* SNMP port graphs

Voip (nProbe required)

* Changes and rework for SIP and RTP protocol
* Adds VoIP SIP to RTP flow search
* Improves VoIP visualization (RTP)

Security Fixes

* Disable TLS 1.0 (vulnerable) in mongoose
* Disabled insecure cyphers in SSL (when using ntopng over SSL)
* Hardens the code to prevent SQL injections
* Enforce POST form CSRF to prevent programmer mistakes
* Strict GET and POST parameters validation to prevent XSS
* Prevent HTTP splitting attacks
* Force default admin password change

----------------------------------------------------------------