Fixed
- Exclusion of empty size values, which skew average message size statistics. (See Processing Behavior for more details)
Added
- Added option --exclude-ips to exclude senders by IP address. (This option requires the sender's IP addresses to be included in the CSV.)
- Added filter statistic for exclusion by IP
Messages excluded by empty senders: 307020
Messages excluded by invalid size: 268934
Messages excluded by IP address: 176273
Messages excluded by domain: 0
Messages excluded by sender: 0
Messages excluded by constraint: 0
Processing Behavior
The primary purpose of this tool is to identify sender message volumes and calculate data transfer rates for legitimate
emails.
Input Requirements
- **Expected Fields**: The input CSV should include at least the envelope sender and message size fields.
- **Exclusions**: Messages will be excluded if:
- The envelope sender is empty (common for bounce replies or calendar actions).
- The message size is missing or not a valid number (typically rejects that can skew reporting).
Exclusion Rules
1. **Domain-Based Exclusions**:
- Messages from system domains such as `ppops.net`, `pphosted.com`, and `knowledgefront.com` are omitted by default
to filter out monitoring messages.
- To include these messages, use the `--no-default-exclude-domains` flag.
2. **IP-Based Exclusions**:
- For messages from `127.0.0.1` (e.g., system reports and digests on Proofpoint Protection Gateway), use the
`--exclude-ips` flag to exclude them.
- This option requires sender IP addresses to be included in the CSV.
Each exclusion step ensures the accuracy of volume and average message size reporting by filtering out unnecessary data.
Usage Options
usage: senderstats [-h] [--version] -i <file> [<file> ...] -o <xlsx> [--ip IP]
[--mfrom MFrom] [--hfrom HFrom] [--rcpts Rcpts]
[--rpath RPath] [--msgid MsgID] [--subject Subject]
[--size MsgSz] [--date Date] [--gen-hfrom] [--gen-rpath]
[--gen-alignment] [--gen-msgid] [--expand-recipients]
[--no-display-name] [--remove-prvs] [--decode-srs]
[--no-empty-hfrom] [--sample-subject]
[--exclude-ips <ip> [<ip> ...]]
[--exclude-domains <domain> [<domain> ...]]
[--restrict-domains <domain> [<domain> ...]]
[--exclude-senders <sender> [<sender> ...]]
[--date-format DateFmt] [--no-default-exclude-domains]
This tool helps identify the top senders based on smart search outbound
message exports.
Input / Output arguments (required):
-i <file> [<file> ...], --input <file> [<file> ...] Smart search files to
read.
-o <xlsx>, --output <xlsx> Output file
Field mapping arguments (optional):
--ip IP CSV field of the IP
address. (default=Sende
r_IP_Address)
--mfrom MFrom CSV field of the
envelope sender
address.
(default=Sender)
--hfrom HFrom CSV field of the header
From: address.
(default=Header_From)
--rcpts Rcpts CSV field of the header
recipient addresses.
(default=Recipients)
--rpath RPath CSV field of the
Return-Path: address.
(default=Header_Return-
Path)
--msgid MsgID CSV field of the
message ID.
(default=Message_ID)
--subject Subject CSV field of the
Subject, only used if
--sample-subject is
specified.
(default=Subject)
--size MsgSz CSV field of message
size.
(default=Message_Size)
--date Date CSV field of message
date/time.
(default=Date)
Reporting control arguments (optional):
--gen-hfrom Generate report showing
the header From: data
for messages being
sent.
--gen-rpath Generate report showing
return path for
messages being sent.
--gen-alignment Generate report showing
envelope sender and
header From: alignment
--gen-msgid Generate report showing
parsed Message ID.
Helps determine the
sending system
Parsing behavior arguments (optional):
--expand-recipients Expand recipients
counts messages by
destination. E.g. 1
message going to 3
people, is 3 messages
sent.
--no-display-name Remove display and use
address only. Converts
'Display Name
<userdomain.com>' to
'userdomain.com'
--remove-prvs Remove return path
verification strings
e.g. prvs=tag=senderdo
main.com
--decode-srs Convert sender rewrite
scheme, forwardmailbox+
srs=hash=tt=domain.com=
user to userdomain.com
--no-empty-hfrom If the header From: is
empty the envelope
sender address is used
--sample-subject Enable probabilistic
random sampling of
subject lines found
during processing
--exclude-ips <ip> [<ip> ...] Exclude ips from
processing.
--exclude-domains <domain> [<domain> ...] Exclude domains from
processing.
--restrict-domains <domain> [<domain> ...] Constrain domains for
processing.
--exclude-senders <sender> [<sender> ...] Exclude senders from
processing.
--date-format DateFmt Date format used to
parse the timestamps. (
default=%Y-%m-%dT%H:%M:
%S.%f%z)
Extended processing controls (optional):
--no-default-exclude-domains Will not include the
default Proofpoint
excluded domains.
Usage:
-h, --help Show this help message
and exit
--version Show the program's
version and exit
**Full Changelog**: https://github.com/pfptcommunity/senderstats/compare/v2.0.5...v2.0.6