Barman

Bugfixes

- Add isoformat fields for backup start and end times in json output

This patch modifies the json output of the infofile object
adding two new fields: `begin_time_iso` and `end_time_iso`.
The new fields allow the use of a more standard and timezone aware
time format, preserving compatibility with previous versions.
It is worth noting that in the future the iso format for dates will be the
standard used by barman for storing dates and will be used everywhere
non human readable output is requested.

As part of the work, this patch reverts BAR-316, which was introduced on Barman
3.12.0.

References: BAR-494.

Minor changes

- Add FIPS support to Barman

The `md5` hash algorithm is not FIPS compliant, so it is going to be replaced by
`sha256`. `sha256` is FIPS compliant, vastly used, and is considered secure for most
practical purposes.
Up until this release, Barman's WAL archive client used `hashlib.md5` to generate
checksums for tar files before they were sent to the Barman server. Here, a tar file is
a file format used for bundling multiple files together with a `MD5SUMS` file that lists
the checksums and their corresponding paths.
In this release, the `md5` hashing algorithm is replaced by `sha256` as the default.
As a result, checksums for the tar files will be calculated using `sha256`, and the
`MD5SUMS` file will be named `SHA256SUMS`. Barman still has the ability to use the
nondefault `md5` algorithm and the `MD5SUMS` file from the client if there is a use
case for it. The user just needs to add the `--md5` flag to the `barman-wal-archive`
`archive_command`.

References: BAR-155, CP-34954, CP-34391.

- Removed el7, debian10, and ubuntu1804 support; updated Debian and SLES.

Support for el7, debian10, and ubuntu1804 has been removed. Additionally, version 12
and version name "bookworm" has been added for Debian, addressing a previously
missing entry. The SLES image version has also been updated from sp4 to sp5.

References: BAR-389.

- Add support for Postgres Extended 17 (PGE) and Postgres Advanced Server 17 (EPAS)

Tests were conducted on Postgres Extended 17 (PGE) and Postgres Advanced Server 17
(EPAS), confirming full compatibility with the latest features in Barman. This
validation ensures that users of the latest version of PGE and EPAS can leverage all the new
capabilities of Barman with confidence.

References: BAR-331.

- Improve WAL compression with `zstd`, `lz4` and `xz` algorithms

Introduced support for xz compression on WAL files. It can be enabled by specifying
`xz` in the `compression` server parameter. WALs will be compressed when entering
the Barman's WAL archive. For the cloud, it can be enabled by specifying `--xz`
when running `barman-cloud-wal-archive`.

Introduced support for zstandard compression on WAL files. It can be enabled by
specifying `zstd` in the `compression` server parameter. WALs will be compressed
when entering the Barman's WAL archive. For the cloud, it can be enabled by
specifying `--zstd` when running `barman-cloud-wal-archive`.

Introduced support for lz4 compression on WAL files. It can be enabled by
specifying `lz4` in the `compression` server parameter. WALs will be compressed
when entering the Barman's WAL archive. For the cloud, it can be enabled by
specifying `--lz4` when running `barman-cloud-wal-archive`.

References: BAR-265, BAR-423, BAR-264.

- Improve WAL upload performance on S3 buckets by avoiding multipart uploads

Previously, WAL files were being uploaded to S3 buckets using multipart uploads
provided by the boto3 library via the `upload_fileobj` method. It was noticed that
multipart upload is slower when used for small files, such as WAL segments,
compared to when uploading it in a single PUT request.
This has been improved by avoiding multipart uploads for files smaller than 100MB.
The average upload time of each WAL file is expected to be reduced by around 15%
with this change.

References: BAR-374.

- Modify behavior when enforcing retention policy for `KEEP:STANDALONE` full backups

When enforcing the retention policy on full backups created with
`backup_method = postgres`, Barman was previously marking all dependent (child)
incremental backups as `VALID`, regardless of the KEEP target used. However, this
approach is incorrect:

- For backups labeled `KEEP:STANDALONE`, Barman only retains the WAL files needed to
restore the server to the exact state of that backup. Because these backups are
self-contained, any dependent child backups are no longer needed once the root
backup is outside the retention policy.

- In contrast, backups marked `KEEP:FULL` are intended for point-in-time recovery.
To support this, Barman retains all WALs, as well as any child backups, to ensure
the backup's consistency and allow recovery to the latest possible point.

This distinction ensures that `KEEP:STANDALONE` backups serve as snapshots of a
specific moment, while `KEEP:FULL` backups retain everything needed for full
point-in-time recovery.

References: BAR-366.

- Update documentation and user-facing features for Barman's recovery process.

Barman docs and the tool itself used to use the terms "recover"/"recovery" both for
referencing:

- The Postgres recovery process;
- The process of restoring a backup and preparing it for recovery.

Both the code and documentation have been revised to accurately reflect the usage of
the terms "restore" and "recover"/"recovery".

Also, the `barman recover` command was renamed to `barman restore`. The old name is
still kept as an alias for backward compatibility.

References: BAR-337.

- Add --keep-compression flag to barman-wal-restore and get-wal

A new `--keep-compression` option has been added to both `barman-wal-restore` and
`get-wal`. This option controls whether compressed WAL files should be decompressed
on the Barman server before being fetched. When specified with `get-wal`, default
decompression is skipped, and the output is the WAL file content in its original
state. When specified with `barman-wal-restore`, the WAL file is fetched as-is and,
if compressed, decompressed on the client side.

References: BAR-435.

- Ransomware protection - Add AWS Snapshot Lock Support

Barman now supports AWS EBS Snapshot Lock, a new integrated feature to prevent
accidental or malicious deletions of Amazon EBS snapshots. When a snapshot is
locked, it can't be deleted by any user but remains fully accessible for use. This
feature enables you to store snapshots in WORM (Write-Once-Read-Many) format for a
specified duration, helping to meet regulatory requirements by keeping the data
secure and tamper-proof until the lock expires.

Special thanks to Rui Marinho, our community contributor who started this feature.

References: BAR-242.

- Prevent orphan files from being left from a crash while deleting a backup

This commit fixes an issue where backups could leave behind files if the system
crashed during the deletion of a backup.

Now, when a backup is deleted, it will get a "delete marker" at the start.
If a crash happens while the backup is being deleted, the marker will help
recognize incomplete backup removals when the server restarts.

The Barman cron job has been updated to look for these deleted markers. If it finds
a backup with a "delete marker", it will complete the process.

References: BAR-244.

- Add support for using tags with snapshots

Barman now supports tagging the snapshots when creating backups using the
barman-cloud-backup script command. A new argument called --tags was added.

Special thanks to Rui Marinho, our community contributor who started this feature.

References: BAR-417.

- Use ISO format instead of ctime when producing JSON output of Barman cloud commands

The ctime format has no information about the time zone associated with the timestamp.
Besides that, that format is better suited for human consumption. For machine
consumption the ISO format is better suited.

References: BAR-316.

Bugfixes

- Fix barman check which returns wrong results for Replication Slot

Previously, when using architectures which backup from a standby node and stream WALs
from the primary, Barman would incorrectly use `conninfo` (pointing to a standby server)
for replication checks, leading to errors such as:

`replication slot (WAL streaming): FAILED (replication slot 'barman' doesn't exist.
Please execute 'barman receive-wal --create-slot pg17')`

This fixes the following issue
[1024](https://github.com/EnterpriseDB/barman/issues/1024) by ensuring
`wal_conninfo` is used for WAL replication checks if it's set.

`wal_conninfo` takes precedence over `wal_streaming_conninfo`, when both are set.
With this change, if only `wal_conninfo` is set, it will be used and will not fall
back to `conninfo`.

Also, in the documentation, changes were made so it is explicit that when `conninfo`
points to a standby server, `wal_conninfo` must be set and used for accurate
replication status checks.

References: BAR-409.

- Fix missing options for `barman keep`

The error message that the Barman CLI emitted when running `barman keep`
without any options suggested there were shortcut aliases for status and
release. These aliases, -s and -r, do not exist, so the error message was
misleading.
This fixes the issue by including these short options in the Barman CLI,
aligning it with other tools like `barman-cloud-backup-keep`, where these
shortcuts already exist.

References: BAR-356.

- Lighten standby checks related to conninfo and primary_conninfo

When backing up a standby server, Barman performs some checks to assert
that `conninfo` is really pointing to a standby (in recovery mode) and
that `primary_conninfo` is pointing to a primary (not in recovery).

The problem, as reported in the issues 704 and 744, is that when a
failover occurs, the `conninfo` will now be pointing to a primary
instead and the checks will start failing, requiring the user to change
Barman configs manually whenever a failover occurs.

This fix solved the issue by making such checks non-critical, which
means they will still fail but Barman will keep operating regardless.
Essentially, Barman will ignore `primary_conninfo` if `conninfo` does
not point to a standby. Warnings about this misconfiguration will also
be emitted whenever running any Barman command so the user can be aware.

References: BAR-348.

- Check for USAGE instead of MEMBER when calling pg_has_role in Barman

To work correctly Barman database user needs to be included in some roles. Barman was
verifying the conditions was satisfied by calling `pg_has_role` in Postgres. However,
it was check for the `MEMBER` privilege instead of `USAGE`. This oversight was fixed.

This change is a contribution from RealGreenDragon.

References: BAR-489.

Bug fixes

- Fix failures in `barman-cloud-backup-delete`. This command was failing when
applying retention policies due to a bug introduced by the previous release.

Features

- Add support for Postgres 17+ incremental backups. This major feature is
composed of several small changes:

- Add `--incremental` command-line option to `barman backup` command. This is
used to specify the parent backup when taking an incremental backup. The
parent can be either a full backup or another incremental backup.

- Add `latest-full` shortcut backup ID. Along with `latest`, this can be used
as a shortcut to select the parent backup for an incremental backup. While
`latest` takes the latest backup independently if it is full or incremental,
`latest-full` takes the latest full backup.

- `barman keep` command can only be applied to full backups when
`backup_method = postgres`. If a full backup has incremental backups that
depend on it, all of the incrementals are also kept by Barman.

- When deleting a backup all the incremental backups depending on it, if any,
are also removed.

- Retention policies do not take incremental backups into consideration. As
incremental backups cannot be recovered without having the complete chain of
backups available up to the full backup, only full backups account for
retention policies.

- `barman recover` needs to combine the full backup with the chain of incremental
backups when recovering. The new CLI option `--local-staging-path`, and the
corresponding `local_staging_path` configuration option, are used to specify
the path in the Barman host where the backups will be combined when recovering
an incremental backup.

- Changes to `barman show-backup` output:

- Add the “Estimated cluster size” field. It's useful to have an estimation
of the data directory size of a cluster when restoring a backup. It’s
particularly useful when recovering compressed backups or incremental
backups, situations where the size of the backup doesn’t reflect the size of the
data directory in Postgres. In JSON format, this is stored as
`cluster_size`.

- Add the “WAL summarizer” field. This field shows if `summarize_wal` was
enabled in Postgres at the time the backup was taken. In JSON format, this
is stored as `server_information.summarize_wal`. This field is omitted for
Postgres 16 and older.

- Add “Data checksums” field. This shows if `data_checkums` was enabled in
Postgres at the time the backup was taken. In JSON format, this is stored as
`server_information.data_checksums`.

- Add the “Backup method” field. This shows the backup method used for this
backup. In JSON format, this is stored as
`base_backup_information.backup_method`.

- Rename the field “Disk Usage” as “Backup Size”. The latter provides a more
comprehensive name which represents the size of the backup in the Barman
host. The JSON field under `base_backup_information` was also renamed from
`disk_usage` to `backup_size`.

- Add the “WAL size” field. This shows the size of the WALs required by the
backup. In JSON format, this is stored as
`base_backup_information.wal_size`.

- Refactor the field “Incremental size”. It is now named “Resources saving”
and it now shows an estimation of resources saved when taking incremental
backups with `rsync` or `pg_basebackup`. It compares the backup size with
the estimated cluster size to estimate the amount of disk and network
resources that were saved by taking an incremental backup. In JSON format,
the field was renamed from `incremental_size` to `resource_savings` under `base_backup_information`.

- Add the `system_id` field to the JSON document. This field contains the
system identifier of Postgres. It was present in console format, but was
missing in JSON format.

- Add fields related with Postgres incremental backups:

- “Backup type”: indicates if the Postgres backup is full or incremental. In
JSON format, this is stored as `backup_type` under `base_backup_information`.

- “Root backup”: the ID of the full backup that is the root of a chain of
one or more incremental backups. In JSON format, this is stored as
`catalog_information.root_backup_id`.

- “Parent backup”: the ID of the full or incremental backup from which this
incremental backup was taken. In JSON format, this is stored as
`catalog_information.parent_backup_id`.

- “Children Backup(s)”: the IDs of the incremental backups that were taken
with this backup as the parent. In JSON format, this is stored as
`catalog_information.children_backup_ids`.

- “Backup chain size”: the number of backups in the chain from this
incremental backup up to the root backup. In JSON format, this is
stored as `catalog_information.chain_size`.

- Changes to `barman list-backup` output:

- It now includes the backup type in the JSON output, which can be either
`rsync` for backups taken with rsync, `full` or `incremental` for backups
taken with `pg_basebackup`, or `snapshot` for cloud snapshots. When printing
to the console the backup type is represented by the corresponding labels
`R`, `F`, `I` or `S`.

- Remove tablespaces information from the output. That was bloating the
output. Tablespaces information can still be found in the output of
`barman show-backup`.

- Always set a timestamp with a time zone when configuring
`recovery_target_time` through `barman recover`. Previously, if no time zone
was explicitly set through `--target-time`, Barman would configure
`recovery_target_time` without a time zone in Postgres. Without a time zone,
Postgres would assume whatever is configured through `timezone` GUC in
Postgres. From now on Barman will issue a warning and configure
`recovery_target_time` with the time zone of the Barman host if no time zone
is set by the user through `--target-time` option.

- When recovering a backup with the “no get wal” approach and `--target-lsn` is set,
copy only the WAL files required to reach the configured target. Previously
Barman would copy all the WAL files from its archive to Postgres.

- When recovering a backup with the “no get wal” approach and `--target-immediate`
is set, copy only the WAL files required to reach the consistent point.
Previously Barman would copy all the WAL files from its archive to Postgres.

- `barman-wal-restore` now moves WALs from the spool directory to `pg_wal`
instead of copying them. This can improve performance if the spool directory
and the `pg_wal` directory are in the same partition.

- `barman check-backup` now shows the reason why a backup was marked as `FAILED`
in the output and logs. Previously for a user to know why the backup was
marked as `FAILED`, they would need to run `barman show-backup` command.

- Add configuration option `aws_await_snapshots_timeout` and the corresponding
`--aws-await-snapshots-timeout` command-line option on `barman-cloud-backup`.
This specifies the timeout in seconds to wait for snapshot backups to reach
the completed state.

- Add a keep-alive mechanism to rsync-based backups. Previously the Postgres
session created by Barman to run `pg_backup_start()` and `pg_backup_stop()` would
stay idle for as long as the base backup copy would take. That could lead to a
firewall or router dropping the connection because it was idle for a long
time. The keep-alive mechanism sends heartbeat queries to Postgres
through that connection, thus reducing the likelihood of a connection
getting dropped. The interval between heartbeats can be controlled through the new
configuration option `keepalive_interval` and the corresponding CLI
option `--keepalive-interval` of the `barman backup` command.

Bug fixes

- When recovering a backup with the “no get wal” approach and `--target-time`
set, copy all WAL files. Previously Barman would attempt to “guess” the WAL
files required by Postgres to reach the configured target time. However,
the mechanism was not robust enough as it was based on the stats of the WAL
file in the Barman host (more specifically the creation time). For example:
if there were archiving or streaming lag between Postgres and Barman, that
could be enough for recovery to fail because Barman would miss to copy all
the required WAL files due to the weak check based on file stats.

- Pin `python-snappy` to `0.6.1` when running Barman through Python 3.6 or
older. Newer versions of `python-snappy` require `cramjam` version `2.7.0` or
newer, and these are only available for Python 3.7 or newer.

- `barman receive-wal` now exits with code `1` instead of `0` in the following
cases:

- Being unable to run with `--reset` flag because `pg_receivewal` is
running.

- Being unable to start `pg_receivewal` process because it is already
running.

- Fix and improve information about Python in `barman diagnose` output:

- The command now makes sure to use the same Python interpreter under which
Barman is installed when outputting the Python version through
`python_ver` JSON key. Previously, if an environment had multiple Python
installations and/or virtual environments, the output could eventually be
misleading, as it could be fetched from a different Python interpreter.

- Added a `python_executable` key to the JSON output. That contains the path
to the exact Python interpreter being used by Barman.

Bug fixes

- Make `argcomplete` optional to avoid installation issues on some
platforms.
- Load `barman.auto.conf` only when the file exists.
- Emit a warning when the `cfg_changes.queue` file is malformed.
- Correct in documentation the postgresql version where
`pg_checkpoint` is available.
- Add `--no-partial` option to `barman-cloud-wal-restore`.

Features
- Limit the average bandwidth used by `barman-cloud-backup` when backing
up to either AWS S3 or Azure Blob Storage according to the value set by
a new CLI option `--max-bandwidth`.
- Add the new configuration option `lock_directory_cleanup`
That enables cron to automatically clean up the barman_lock_directory
from unused lock files.
- Add support for a new type of configuration called `model`.
The model acts as a set of overrides for configuration options
for a given Barman server.
- Add a new barman command `barman config-update` that allows the creation
and the update of configurations using JSON

Bug fixes:
- Fix a bug that caused `--min-chunk-size` to be ignored when using
barman-cloud-backup as hook script in Barman.