Pcp

- https://github.com/performancecopilot/pcp/projects/1

- Client tools and utilities:
pmlogger: massive startup latency improvements over slow links
pmlogger: reduce "error sending connection ACK to client" msgs
pmlogger: abort if we see PM_ERR_IPC during config file parsing
pmlogger: rework early control port servicing
pmlogger: source zeroconf variables from an additional env file
pmseries: fix --load option failure on 32-bit platforms only
pcp-dstat: add modern mongodb dstat plugin configuration
pcp-atop: add support for Nvidia GPUs using pmdanvidia metrics
- PMDA additions, enhancements and bug fixes:
pmdamongodb: new PMDA to export PCP metrics from MongoDB
pmdamssql: share username/password with the Assessments API
pmdalinux: additions and updates to mem.zoneinfo metrics
pmdalinux: cull empty NUMA zones from zoneinfo instance domain
pmdalinux: increase /proc/net/netstat metric coverage
pmdalinux: increase /proc/net/snmp6 metric coverage
pmdalinux: increase /proc/net/snmp metric coverage
pmdalinux: improved robustness of procfs parsers
pmdakvm: add several new KVM kernel metrics
pmdanvidia: updates to latest NVML, add a batch of new metrics
pmdaopenmetrics: add vmware URL for use with vmware_exporter
pmdabpf: use bpf skeletons
- Server-side utilities and log management scripts:
pmproxy: provide transparent Redis reconnect after connection drop
pmproxy: do not connect to Redis if it's not enabled in config file
services: disable service advertising by default in pmcd and pmproxy
services: remove run-level check in pmlogger systemd service config
pmieutil: add pmie_farm service for non-primary pmie instances
logutil: add pmlogger_farm service, update associated admin tools
logutil: add pmlogctl -m flag, to enable service migration
pm{log,pmie}ctl: increase max delay waiting for a process to stop
pmlogconf: resolve issues in persisting changes to pmlogger config
pmlogconf: log metrics required by MS SQL Server Grafana dashboard
pmcpp: add optional outfile on command line
pmdaproc.sh: tweak tests for ping version in _setup_localhost
- libpcp, libpcp_pmda, libpcp_mmv, libpcp_web and language bindings:
libpcp: allow pmNameLookup(3) to use multiple PDU round-trips
libpcp, pmcd: support a new bulk pmLookupDescs(3) PMAPI routine
libpcp: add __pmHashFree interface to explicitly release nodes
libpcp: fix SASL authentication where hostname differs to FQDN
libpcp_web: support Redis authentication with password and ACL
python api: make PY_SSIZE_T_CLEAN, stop using deprecated CB interfaces
- Misc build, infrastructure and packaging updates:
packaging: enable pmlogger_farm if pmlogger is enabled pre-el9
packaging: cull old debian conffiles
packaging: rework OpenMandriva controls
packaging: obsolete the pcp-pmda-vmware rpm
build: transition to using vendored jsonsl code
build: update vendored htop code to 3.1.1 release
build: update vendored hiredis code to 1.0.2 release
build: convert ini header and source to vendored code
build: allow custom bpftool with --with-pmdabpf-bpftool
build: consolidate the multiple copies of sds strings code
- Documentation and QA infrastructure:
qa: resolve new pylint warnings in Fedora 35 python
ci: add Fedora 35 to CI and release workflow
ci: move daily QA 2h earlier, so it always finishes on the same day
man: add PMLOGGER_INTERVAL note in pmlogger -t option description

- Client tools and utilities:
pcp-atop: enable detailed processor frequency scaling reporting
pcp-atop: handle different event naming of ix86arch PMUs
pcp-atop: report user and group names using PMAPI calls
pcp-htop: fix crash on terminal resize during sampling
pcp-htop: fix mouse wheel collision with autogroups nice adjustment
pcp-htop: do not reset MainPanel on disabled mouse events
pmlogger: fix small error in EOF logic on pmcd channel
pmlogger: install a zeroconf file for setting interval
- PMDA additions, enhancements and bug fixes:
pmdalinux: add detailed processor frequency scaling metrics
pmdapmproxy: add logrewrite rules to transition metrics
pmdasockets: fix the parsing of reord_seen values
pmdabpf: resolve help text warning from pmdaInit
pmdads389: fix syntax error in alternate branches
pmdads389: allow installation when ds389 server is down
pmdahacluster: fix types for some corosync and drbd metrics
- Server-side utilities and log management scripts:
logutil: cleanly separate primary and farm logger services
logutil: tweak timeouts in pmlogger.service
logutil: add -P / --only-primary option to pmlogger_check
pmproxy, libpcp_web: timer cleanups, updates to metrics
- libpcp, libpcp_pmda, libpcp_mmv, libpcp_web and language bindings:
libpcp: use urlencoding for hostspec parameter values
libpcp: improve memory footprint of pmSetProcessIdentity
libpcp: __pmResult restructuring for highres timestamps
libpcp: rework fetchgroup max 64-bit value comparisons
libpcp: add new PM_ERR_BOTCH error macro
libpcp: large-PDU-related improvements and fixes
libpcp_web: smaller memory footprint in Redis cluster code
libpcp_web: optimize webgroup metric updating with mmv_set
libpcp_mmv: add new convenience interfaces for inc/add/set
libpcp_fault: fault injection 50,000km service
- Misc build, infrastructure and packaging updates:
misc: numerous minor changes to resolve LGTM.com issues
configure.ac: fix $PCP_PS_ALL_FLAGS for OpenBSD
build: require libbpf >= 0.4.0
build: fix tempfile handling in Makepkgs script
build: add OpenBSD "init" integration for postinstall and remove
packaging: additional RPM changes for pcp-zeroconf conf file overrides
packaging: fix ownership of /usr/lib/systemd/system/pmlogger.service.d
- Documentation and QA infrastructure:
docs: add a new pmdabpf(1) man page
qa: tweak valgrind options for speed
qa: admin/list-packages new `cmd` feature
ci: bump limit to retain 15 QA reports
ci: add libcmocka-devel to QA package lists

- Client tools and utilities:
pcp-htop: add new "dynamic columns" feature
pcp-htop: enable all mouse-driven functionality
pcp-htop: support offline and hotplug CPUs
pcp-htop: columns for process autogroup identifier and nice value
pcp-htop: add combined memory and swap meter
pcp-htop: limit meter LED mode by width
pcp-htop: versioned config files and config_reader_min_version
pcp-htop: add a new option to change the header layout
pcp-htop: move shipped column/meter configs below /usr/share
pmieconf: improve the CPU saturation rules
pmieconf: add several per-disk pmie rule templates
pmie: fix sigsegv when used with archives lacking needed metrics
- PMDA additions, enhancements and bug fixes:
pmdabpf: introduce the new BPF PMDA based on libbpf
pmdabcc: free BPF memory after module setup completes
pmdalinux: add -A option for overriding permissions checks
pmdaproc: add proc.autogroup scheduling metrics
pmdaproc: ensure ENODATA error code not exposed on fetch callbacks
pmdapodman: correct the metric count calculation for pmdaInit
- Server-side utilities and log management scripts:
pmproxy: avoid duplicate instname labels in /metrics response
pmlogger: add -I/--pmlc-ipc-version command line option
pmlc-pmlogger: rework PDU_LOG_STATUS
Fix PMDA upgrade issue relating to python vs python3 switch
services: switch logutil and pmieutil scripts from type oneshot to exec
services: replace daily poll services with persistent timer setting
- libpcp, libpcp_pmda, libpcp_mmv, libpcp_web and language bindings:
python api: additional converter functions for struct timeval
python api: refer to ctypes consistently
python api: fix pmiAddMetric parameter type
libpcp: __pmZoneinfo implementation suitable for library use
libpcp: internal restructuring and initial v3 PCP archive code
libpcp: explicit readdir64 use is removed, via compile options only
libpcp: add -Dpmlc diags for the pmlc and pmlogger IPC protocol
libpcp_mmv: add new convenience routines for value set/inc
- Misc build, infrastructure and packaging updates:
build: fixed errors and warnings reported by LGTM.com
build: fixed warnings reported by Coverity
build: initial fixes of gcc warnings from -Wshadow
build: updates to resolve warnings from latest version of pylint
build: fix pmdasockets compiler warning on 32bit platforms
packaging: remove any stale *-poll systemd service symlinks
- Security Enhanced Linux:
selinux: additional policy for podman socket access
selinux: make use of container_var_run policy conditional
selinux: make use of glusterd_log policy conditional
- Documentation and QA infrastructure:
pmdasample: add sample.proc.* metrics for dynamic indom testing
docs: new pcp-htop(1) manual page for dynamic columns and meters
qa: use containers for all PCP CI and release builds
qa: add common.redis routine _wait_for_redis() for tests to share
ci: add Fedora 35 and disable Python 2 there (missing packages)
ci: refactorings and misc improvements across the board
ci: disable debuginfod due to massive valgrind slowdown
ci: retain the 10 most recent QA reports for comparison -
https://performancecopilot.github.io/qa-reports/reports/

- Client tools and utilities:
python: minimum supported python2 version is now 2.7
pcp-ss: new socket statistics client tool
pcp-htop: add new "dynamic meters" feature
pcp-htop: use code vendoring and git subtree of htop.dev
pcp-htop: fix per-process user and system time conversions
pcp-htop: use the correct metric for shared memory calculations
pcp-htop: fix the color of PROC_COMM for PCP
pminfo: plug small memory leak on error paths for Coverity scan
pmdumplog: add -I option to report on-disk instance domains
pcp-{iostat,tapestat}.1: add --version description
pmrep/pmconfig.py: fix archive writing with scaled metrics
newhelp: reporting for empty or missing help text
pmlogger: add empty string help text
zsh completion: pmevent is an alias for pmval, fix command sorting
bash,zshc completions: update for pmdumplog -I
bash completions: add -P option for pmclient
- PMDA additions, enhancements and bug fixes:
pmdadenki: add new PMDA 'denki' for power metrics
pmdalinux: fix sign extension in buddyinfo total calculation
pmdalinux: use buddyinfo's order in calculating total size
pmdaproc: add proc.psinfo.cwd and proc.psinfo.exe metrics
pmdasockets: default to daemon and migrate from DSO to daemon
pmdasockets: add persistent filter, new metrics, parser fixes
pmdasockets: add missing help text for reord_seen and delivered
pmdasockets: fix ! for Install and Remove
pmdasockets: fix resource leak coverity CID371684
pmdasockets: fix regex in Upgrade script
pmdahacluster: Remove un-needed pmdaCacheLookupName() calls
pmdahacluster: add support for labels on key metrics
pmdammv: add check for item uniqueness
pmdapmcd: add pmcd.zoneinfo metric
pmdabcc: drop defunct usdt_jvm_alloc module
pmdads389: add two additional cn attributes
pmdads389: automated discovery of replication agreements
- Server-side utilities and log management scripts:
dbpmda: add -flag to debug command
pmproxy: honour the secure.enabled config option
pmlogger: block SIGALRM before re-exec
pmlogger: tighten up "lost pmcd connection" logic
pmlogger: don't try to catch SIGSTOP
pmlogger_daily.sh: extra -VV diagnostics
pmcd: avoid mem leak on error path, Coverity issues
pmcd: small tweak to getzoneinfo_plan_b()
pmproxy: add mutex for client req lists, fix https/tls support
- libpcp, libpcp_pmda, libpcp_mmv, libpcp_web and language bindings:
python api: guard against fetchgroup destructor race condition
libpcp_web: add generic timer API
libpcp_web: add generic server metrics
libpcp_web: improve discovery lock handling and scalability
libpcp_web: add mutex to struct webgroup protecting the context dict
libpcp_web: add new metric for metadata discovery partial reads
libpcp_web: fix many resource leaks, null checks from Coverity scan
libpcp_web: plug mem leak in redisMapInsert during daily log-rolling
libpcp_web: disable Redis connection if version check fails
libpcp_import: fix per-instance error handling in pmiWrite
libpcp: don't clobber help text buffer, fix "fallback" logic
libpcp: complete remaining libpcp.h internal struct renaming
libpcp: replace sighold() and sigrelse() with sigprocmask()
- Misc build, infrastructure and packaging updates:
build: optionally enable sanitizers in configure
build: conditionally enable gperftools (default is no)
build: mark python3-bpfcc dependency as architecture specific
build: bump Fedora version of the PCP container
build: do not install /var/tmp as part of PCP itself
build: add configure and makefile macros for libbpf, libelf
libpcp/pmapi.h/pmdbg: add -Ddev0,dev1,dev2
- Security Enhanced Linux:
selinux QA: rework QA for other SELinux implementations
selinux QA: fix semodule -l output format, has changed in Fedora 34
selinux QA: small fixup for semodule -l filter
selinux: small changes for CentOS Linux7.9.2009
- Documentation and QA infrastructure:
docs: add "SELinux Considerations" section to pmdaopenmetrics(1)
docs: add man page for pmWebTimerRegister(3) and related functions
docs: document pmdasockets(1) filter config file
docs: add missing long-options in many man(1) pages
docs: move Grafana troubleshooting out of scaling doc
docs: document hostspec parameter for openmetrics endpoint
docs: update scaling doc with federated setup and troubleshooting
docs: pmdastatsd, update man page and Install instructions
docs: pmseries(1) man page fix ":" operator description
docs: drop the unused --profile output from pcp-dstat(1)
qa/common.avahi: be more careful with stderr from avahi-browse
qa/common.check: revise _service "stop" logic for pmlogger
qa/common.check: improve recovery of failed systemctl services

- Client tools and utilities:
pcp-atop: fix threading options process-sort-order bug
pmieconf: add SQL Server stolen node memory rule
pmieconf: detect sustained CPU thermal throttling
pmrep: add support for config directory
pmrep: split default configuration into separate files
pmrep: sort collectl(1) metricsets
pmrep: tweak proc metricsets
pmrep: improve instance matching
pmrep: add option to omit instance info
pmrep: implement sar-v metricset in pmrep.conf
pmrep: implement sar-r-ALL metricset in default pmrep.conf
pmrep: add sar-R metricset in pmrep.conf
pmrep: add support for repeating header based on terminal height
pmrep: set repeat_header = auto for sar-like metricsets
pmrep: tweak :sar-h metricset output
pmrep: more pmrep.conf kb vs kB vs KB tweaking
pmrep: add more pidstat-like (-drsuvw) metricsets to pmrep.conf
pmrep: add vmstat -d like metricset to pmrep.conf
pmrep: update pmrep.conf sar disk metricsets to include discards
pmrep: add iostat-12.3 like iostat -dktx metricset to pmrep.conf
pmrep: add new proc-activity metricset
pmrep: improve proc metricsets in pmrep.conf
pmrep: remove unneeded sar-old-kernel.conf file
pmrep: harden config file parsing
pmrep: update pmproxy metricset
pmrep: add RHEL8 sar metricsets to default pmrep conf directory
pmrep et al: better process name matching for instance filtering
pmrep et al: better name matching for live filtering also
pmchart: save changes to users prefered archive location
pmchart: use libpcp_qmc diagnostic for archives from same host
pmstat: correctly handle the -V/--verbose option
pmval: add -X flag to report higher precision timestamps from archives
pmlogger: fix file descriptor leak on daily re-exec
pmdumplog: make -x control ALL timestamps
- PMDA additions, enhancements and bug fixes:
pmdalinux: fix persistence of the slabinfo metrics indom
pmdaproc: performance improvements for the Linux per-process PMDA
pmdaproc: provide a proc.psinfo.policy_s metric
pmdaproc: ensure log-finished message not logged prematurely
pmdapodman: switch from libvarlink to the podman REST API
pmdaperfevent: fix help text, allow DSO installation
- Server-side utilities and log management scripts:
pmlogger_daily_report: always disable pmrep header repeating
rc_pmcd: change .Need* protocols (slightly)
rc_pmcd: block concurrency for automated PMDA updates
rc_pmie: do not discard pmie stderr when starting pmie service
pmproxy: move metrics from pmproxy.redis.discover to pmproxy.discover
pmproxy: add long help texts for new pmproxy metrics
pmproxy: add pmproxy.discover.changed_callbacks metric
pmproxy: do not init default uv_loop twice
pmproxy: start pmproxy after Redis (if available)
pmcd: support for high resolution timestamp sampling
- libpcp, libpcp_pmda, libpcp_mmv, libpcp_web and language bindings:
libpcp: support for HIGHRES fetch and result PDUs, pmHighResFetch(3)
libpcp: fix fetchgroup instance profile mangling
libpcp: fix off-by-one error in pmfstring(3)
libpcp: simplify exec.c thread-safe locking
libpcp_web: integrate hiredis and hiredis-cluster
libpcp_web: throttle discovery callbacks to improve scalability
libpcp_web: add new metrics for Redis operations
libpcp_web: free dictEntry struct of dropped contexts
libpcp_web: ignore duplicate or early stream messages
libpcp_web: expose number of inflight redis requests to discovery
libpcp_web: register discovery callback once for initial directories
libpcp_web: add a limit on storable instances to cap mem alloc
derived metrics: add additional iostat metrics
pmconfig.py: more consistent error handling
- Misc build, infrastructure and packaging updates:
build: fix debian builds with missing openpyxl packages
build: fix rpm file permissions for the pcp-testsuite package
build: ensure zabbix.agent.d and sasl2 configs marked as such
build: retire the RPM PMDA, unused and not performance related
- Security Enhanced Linux:
selinux: fix detection of lockdown policy class
selinux: add capability2 { bpf } when needed
selinux: add rules for type kmod_exec_t
selinux: use logging_watch_all_log_dirs_path macro when needed
selinux: add policy changes for pmdaproc dma_device_t stat check
- Documentation and QA infrastructure:
docs: update donations link to https://opencollective.com/pcp
docs: add sizing recommendations table to the Scaling doc
docs: update Fedora/RHEL/CentOS installation instructions
docs: transition from bintray (retired) to artifactory
docs: pmrep man page improvements around instance selection
docs: fix a typo in pmregisterderived(3)
docs: pmrep and pcp2xxx utilities man page tweaks
docs: simplify systemctl commands in the quickstart guide
docs: pcp-dstat(1) man page path tweaks
docs: add missing long options to various man pages
docs: escape pmdiff(1) dashes in options properly
docs: update pmie_check(1) and pmie_daily(1) man page
docs: drop commented env vars from command line option configs
qa: valgrind and helgrind variants for multithread14
qa: add helgrind to group and associated tests
qa: specify PCP libraries only on libpcp_web link lines
ci: add script to open local Allure report
ci: add Fedora 34
ci: csv export