Barman

Minor changes

- Fix errors when using an immutable storage

Added a new `worm_mode` configuration to enable WORM (Write Once Read Many)
handling in Barman, allowing it to support backups on immutable storage.

This fix also provides automatic relocation of the backup.info file in a new
directory `meta` inside `backup_directory`. This will let Barman update it
in future when needed.

Barman will also _not_ purge the wals directory for WAL files that are not
needed when running the first backup. This will add some extra space
which will be reclaimed when this first backup is obsolete and removed
(by that time, the backups and the WALs will be outside the retention
policy window).

Added additional notes to the documentation explaining limitations when
running with an immutable storage for backups. In particular the need
for a grace period in the immutability of files and the fact that
`barman keep` is not supported in these environments.

References: BAR-649, BAR-645, BAR-650, BAR-651, BAR-652.

Minor changes

- Improve behavior of the backup shortcuts `last-full` / `latest-full`

The shortcuts `last-full` / `latest-full` were retrieving not the last full backup of
the server, but the last full backup of the server which was eligible as the parent
for an incremental backup.

While this was the expected behavior, the feedback from the community has shown that
it was confusing for the users.

From now on, the shortcuts `last-full` / `latest-full` will retrieve the last full
backup of the Barman server, independently if that backup is eligible as the parent
for an incremental backup or not.

The eligibility of the full backup as the parent of an incremental backup will still
be validated by Barman in a later step, and a proper message will be displayed in
case it doesn't suit as a parent.

References: BAR-555.

Bugfixes

- Fix error message when parsing invalid `--target-time` in `barman restore`

When using the `barman restore` command, the error message when parsing invalid
`--target-time` string was:


EXCEPTION: local variable 'parsed_target' referenced before assignment


That exception was replaced with an understandable error message.

References: BAR-627.

- Fix mutual exclusive arguments in the cloud restore command

In the `barman-cloud-restore` command, we were checking that `target_tli` and
`target_lsn` were mutually exclusive arguments, where the correct pair to check
would be `target_time` and `target_lsn`.

References: BAR-624.

- Fix Barman not honoring `custom_decompression_filter`

Fixed an issue where Barman was not honoring the configured
`custom_decompression_filter` if the compression algorithm specified
was natively supported by Barman. Custom filters now take priority
over native handlers when decompressing WAL files.

References: BAR-584.

- Fix barman restore with --no-get-wal and --standby

Fixed an issue where Barman was removing the `pg_wal` directory during
recovery if `--no-get-wal` and `--standby-mode` were specified together.
The issue happened due to Barman incorrectly filling the recovery parameters
referencing `pg_wal`, including `recovery_end_command`, which led to this
issue. Barman will now ignore filling such parameters as they are not required
for this specific case.

References: BAR-630.

- Fix argument parsing issue in `barman restore` and `barman-cloud-restore`

In Barman 3.13.0, a regression was introduced causing errors when using
`barman restore` and `barman-cloud-restore` commands. Specifically, the
`backup_id` positional argument, which was made optional in that version,
conflicted with other arguments, causing unrecognized arguments and errors.

For example, running `barman-cloud-restore` like this:

barman-cloud-restore source_url server_name backup_id --cloud-provider aws-s3 recovery_dir


Would trigger an error like this:

barman-cloud-restore: error: unrecognized arguments: recovery_dir


This fix resolves the issue by making `backup_id` a required argument
again. Additionally, a new "auto" value is now accepted as a `backup_id`,
allowing Barman to automatically choose the best backup for restoration
without needing a specific `backup_id`. This update fixes argument handling
and still allows a smooth and flexible restoration process for the user.

References: BAR-596.

Notable changes

- Add new xlogdb_directory configuration

Introduces a new `xlogdb_directory` configuration option. This parameter can be
set either globally or per-server, and allows you to specify a custom directory
for the `xlog.db` file. This file stores metadata of archived WAL files and is used
internally by Barman in various scenarios. If unset, it defaults to the value of
`wals_directory`. Additionally, the file was also renamed to contain the server name
as a prefix.

References: BAR-483.

- Make "backup_id" optional when restoring a backup

Historically, Barman always required a "backup_id" to restore a backup, and would
use that backup as the source for the restore.

This feature removes the need for specifying which backup to use as a source for
restore, making it optional.

This change applies to both Barman and the barman-cloud scripts.

Now the user is able to restore a backup in the following ways:
1. Provide a "backup_id"
2. Do not provide a "backup_id". It will retrieve the most recent backup
3. Do not provide a "backup_id", but provide a recovery target, such as:
- "target_time" (mutually exclusive with target_lsn)
Will get the closest backup prior to the "target_time"
- "target_lsn" (mutually exclusive with "target_time")
Will get the closest backup prior to the "target_lsn"
- "target_tli" (can be used combined with "target_time" or "target_lsn")
Will get the most recent backup that matches the timeline. If combined with
other recovery targets, it will get the most recent backup prior to the
target_time or target_lsn that matches the timeline

The recovery targets `--target-xid`, `--target-name` and `--target-immediate`
are not supported, and will error out with a message if used.

This feature will provide flexibility and ease when restoring a postgres cluster.

References: BAR-541, BAR-473.

Minor changes

- Add current active model to `barman show-server` and `barman status`

Previously, after applying a configuration model, the only way to check
which model is currently active for a server was via the `barman diagnose`
command. With this update, the `barman status` and `barman show-server`
commands now also display the current active configuration model for a
server, if any.

References: BAR-524, BAR-400.

- Add `--staging-wal-directory` option to `barman restore` command to allow alternative WAL directory on PITR

A new command line option `--staging-wal-directory` was added to the `restore`/`recover`
command to allow an alternative destination directory for WAL files when performing
PITR. Previously, WAL files were copied to a `barman_wal` directory within
the restore destination directory. This enhancement provides greater flexibility, such as
storing WALs on separate partitions during recovery.

References: BAR-224.

- Pin boto3 version to any version <= 1.35.99

Boto3 version 1.36 has changed the way S3 integrity is checked making this version
incompatible with the current Barman code, generating the following error:

An error occurred (MissingContentLength) when calling the PutObject operation

As a temporary workaround, the version for boto3 is pinned to any version <= 1.35.99
until support for 1.36 is implemented in Barman.

References: BAR-535.

- Make barman-wal-archive smarter when dealing with duplicate WAL files

Under some corner cases, Postgres could attempt to archive the same WAL twice.
For example: if `barman-wal-archive` copies the WAL file over to the Barman host,
but the script is interrupted before reporting success to Postgres. New executions
of `barman-wal-archive` could fail when trying to archive the same file again
because the WAL was already copied from Postgres to Barman, but not yet processed by
the asynchronous Barman WAL archiver.

This minor change deals with this situation by verifying the checksum of the
existing and the incoming file. If the checksums match the incoming file is
ignored, otherwise an output info message is sent and the incoming file is moved to
the errors directory. The code will exit with 0 in both situations, avoiding WALs
piling up in the Postgres host due to a failing `archive_command`.

References: BAR-225.

- Document procedure to clear WAL archive failure check

While redesigning the Barman docs we missed adding a note advising
users to run a `switch-wal` command if the server is idle and
`barman check` returns a failure on "WAL archiving".

This addresses the gap left from the previous documentation.

References: BAR-521.

- Delete WALs by deleting the entire directory at once, when possible

Previously, when WAL files needed to be deleted (e.g., due to deletion of a backup),
Barman would iterate over every WAL file and delete them individually. This could
cause performance issues, mainly in systems which use ZFS filesystem. With this
change, the entire directory will be deleted whenever noticed that all files in
the directory are no longer needed by Barman.

References: BAR-511.

- Add support for `DefaultAzureCredential` option on Azure authentication

Users can now explicitly use Azure's `DefaultAzureCredential` for authentication
by using the `default` option for `azure_credential` in the server configuration
or the `--azure-credential default` option in the case of `barman-cloud-*`.
Previously, that could only be set as a fallback when no credential was provided
and no environment variables were set.

References: BAR-539.

- Improve diagnose output for retention policy info

Improves the output of the barman diagnose command to display a more user-friendly
string representations. Specifically, "REDUNDANCY 2" is shown instead of
"redundancy 2 b" for the 'retention_policy' attribute, and "MAIN" is shown instead
of "simple-wal 2 b" for the 'wal_retention_policy' attribute.

References: BAR-100.

Bugfixes

- Fix PITR when using `barman restore` with `--target-tli`

Barman was not creating the `recovery.signal` nor filling `recovery_target_timeline`
in `postgresql.auto.conf` in these cases:

* The only recovery target passed to `barman restore` was `--target-tli`; or
* `--target-tli` was specified with some other `--target-*` option, but the
specified target timeline was the same as the timeline of the chosen backup.

Now, if any `--target-*` option is passed to `barman restore`, that will be
correctly treated as PITR.

References: BAR-543.

- Fix bug when AWS 'profile' variable is referenced before assignment

An issue was introduced by BAR-242 as part of the Barman 3.12.0 release.
The issue was causing `barman-cloud-backup-delete` (and possibly other
commands) to fail with errors like this when `--aws-profile` argument or
`aws_profile` configuration were not set:

bash
ERROR: Barman cloud backup delete exception: local
variable 'profile' referenced before assignment`


References: BAR-518.

- Fix --zstd flag on barman-cloud-wal-archive

Fixed a bug with the `--zstd` flag on `barman-cloud-wal-archive` where it was
essentially being ignored and not really compressing the WAL file before upload.

References: BAR-567.

Bugfixes

- Add isoformat fields for backup start and end times in json output

This patch modifies the json output of the infofile object
adding two new fields: `begin_time_iso` and `end_time_iso`.
The new fields allow the use of a more standard and timezone aware
time format, preserving compatibility with previous versions.
It is worth noting that in the future the iso format for dates will be the
standard used by barman for storing dates and will be used everywhere
non human readable output is requested.

As part of the work, this patch reverts BAR-316, which was introduced on Barman
3.12.0.

References: BAR-494.

Minor changes

- Add FIPS support to Barman

The `md5` hash algorithm is not FIPS compliant, so it is going to be replaced by
`sha256`. `sha256` is FIPS compliant, vastly used, and is considered secure for most
practical purposes.
Up until this release, Barman's WAL archive client used `hashlib.md5` to generate
checksums for tar files before they were sent to the Barman server. Here, a tar file is
a file format used for bundling multiple files together with a `MD5SUMS` file that lists
the checksums and their corresponding paths.
In this release, the `md5` hashing algorithm is replaced by `sha256` as the default.
As a result, checksums for the tar files will be calculated using `sha256`, and the
`MD5SUMS` file will be named `SHA256SUMS`. Barman still has the ability to use the
nondefault `md5` algorithm and the `MD5SUMS` file from the client if there is a use
case for it. The user just needs to add the `--md5` flag to the `barman-wal-archive`
`archive_command`.

References: BAR-155, CP-34954, CP-34391.

- Removed el7, debian10, and ubuntu1804 support; updated Debian and SLES.

Support for el7, debian10, and ubuntu1804 has been removed. Additionally, version 12
and version name "bookworm" has been added for Debian, addressing a previously
missing entry. The SLES image version has also been updated from sp4 to sp5.

References: BAR-389.

- Add support for Postgres Extended 17 (PGE) and Postgres Advanced Server 17 (EPAS)

Tests were conducted on Postgres Extended 17 (PGE) and Postgres Advanced Server 17
(EPAS), confirming full compatibility with the latest features in Barman. This
validation ensures that users of the latest version of PGE and EPAS can leverage all the new
capabilities of Barman with confidence.

References: BAR-331.

- Improve WAL compression with `zstd`, `lz4` and `xz` algorithms

Introduced support for xz compression on WAL files. It can be enabled by specifying
`xz` in the `compression` server parameter. WALs will be compressed when entering
the Barman's WAL archive. For the cloud, it can be enabled by specifying `--xz`
when running `barman-cloud-wal-archive`.

Introduced support for zstandard compression on WAL files. It can be enabled by
specifying `zstd` in the `compression` server parameter. WALs will be compressed
when entering the Barman's WAL archive. For the cloud, it can be enabled by
specifying `--zstd` when running `barman-cloud-wal-archive`.

Introduced support for lz4 compression on WAL files. It can be enabled by
specifying `lz4` in the `compression` server parameter. WALs will be compressed
when entering the Barman's WAL archive. For the cloud, it can be enabled by
specifying `--lz4` when running `barman-cloud-wal-archive`.

References: BAR-265, BAR-423, BAR-264.

- Improve WAL upload performance on S3 buckets by avoiding multipart uploads

Previously, WAL files were being uploaded to S3 buckets using multipart uploads
provided by the boto3 library via the `upload_fileobj` method. It was noticed that
multipart upload is slower when used for small files, such as WAL segments,
compared to when uploading it in a single PUT request.
This has been improved by avoiding multipart uploads for files smaller than 100MB.
The average upload time of each WAL file is expected to be reduced by around 15%
with this change.

References: BAR-374.

- Modify behavior when enforcing retention policy for `KEEP:STANDALONE` full backups

When enforcing the retention policy on full backups created with
`backup_method = postgres`, Barman was previously marking all dependent (child)
incremental backups as `VALID`, regardless of the KEEP target used. However, this
approach is incorrect:

- For backups labeled `KEEP:STANDALONE`, Barman only retains the WAL files needed to
restore the server to the exact state of that backup. Because these backups are
self-contained, any dependent child backups are no longer needed once the root
backup is outside the retention policy.

- In contrast, backups marked `KEEP:FULL` are intended for point-in-time recovery.
To support this, Barman retains all WALs, as well as any child backups, to ensure
the backup's consistency and allow recovery to the latest possible point.

This distinction ensures that `KEEP:STANDALONE` backups serve as snapshots of a
specific moment, while `KEEP:FULL` backups retain everything needed for full
point-in-time recovery.

References: BAR-366.

- Update documentation and user-facing features for Barman's recovery process.

Barman docs and the tool itself used to use the terms "recover"/"recovery" both for
referencing:

- The Postgres recovery process;
- The process of restoring a backup and preparing it for recovery.

Both the code and documentation have been revised to accurately reflect the usage of
the terms "restore" and "recover"/"recovery".

Also, the `barman recover` command was renamed to `barman restore`. The old name is
still kept as an alias for backward compatibility.

References: BAR-337.

- Add --keep-compression flag to barman-wal-restore and get-wal

A new `--keep-compression` option has been added to both `barman-wal-restore` and
`get-wal`. This option controls whether compressed WAL files should be decompressed
on the Barman server before being fetched. When specified with `get-wal`, default
decompression is skipped, and the output is the WAL file content in its original
state. When specified with `barman-wal-restore`, the WAL file is fetched as-is and,
if compressed, decompressed on the client side.

References: BAR-435.

- Ransomware protection - Add AWS Snapshot Lock Support

Barman now supports AWS EBS Snapshot Lock, a new integrated feature to prevent
accidental or malicious deletions of Amazon EBS snapshots. When a snapshot is
locked, it can't be deleted by any user but remains fully accessible for use. This
feature enables you to store snapshots in WORM (Write-Once-Read-Many) format for a
specified duration, helping to meet regulatory requirements by keeping the data
secure and tamper-proof until the lock expires.

Special thanks to Rui Marinho, our community contributor who started this feature.

References: BAR-242.

- Prevent orphan files from being left from a crash while deleting a backup

This commit fixes an issue where backups could leave behind files if the system
crashed during the deletion of a backup.

Now, when a backup is deleted, it will get a "delete marker" at the start.
If a crash happens while the backup is being deleted, the marker will help
recognize incomplete backup removals when the server restarts.

The Barman cron job has been updated to look for these deleted markers. If it finds
a backup with a "delete marker", it will complete the process.

References: BAR-244.

- Add support for using tags with snapshots

Barman now supports tagging the snapshots when creating backups using the
barman-cloud-backup script command. A new argument called --tags was added.

Special thanks to Rui Marinho, our community contributor who started this feature.

References: BAR-417.

- Use ISO format instead of ctime when producing JSON output of Barman cloud commands

The ctime format has no information about the time zone associated with the timestamp.
Besides that, that format is better suited for human consumption. For machine
consumption the ISO format is better suited.

References: BAR-316.

Bugfixes

- Fix barman check which returns wrong results for Replication Slot

Previously, when using architectures which backup from a standby node and stream WALs
from the primary, Barman would incorrectly use `conninfo` (pointing to a standby server)
for replication checks, leading to errors such as:

`replication slot (WAL streaming): FAILED (replication slot 'barman' doesn't exist.
Please execute 'barman receive-wal --create-slot pg17')`

This fixes the following issue
[1024](https://github.com/EnterpriseDB/barman/issues/1024) by ensuring
`wal_conninfo` is used for WAL replication checks if it's set.

`wal_conninfo` takes precedence over `wal_streaming_conninfo`, when both are set.
With this change, if only `wal_conninfo` is set, it will be used and will not fall
back to `conninfo`.

Also, in the documentation, changes were made so it is explicit that when `conninfo`
points to a standby server, `wal_conninfo` must be set and used for accurate
replication status checks.

References: BAR-409.

- Fix missing options for `barman keep`

The error message that the Barman CLI emitted when running `barman keep`
without any options suggested there were shortcut aliases for status and
release. These aliases, -s and -r, do not exist, so the error message was
misleading.
This fixes the issue by including these short options in the Barman CLI,
aligning it with other tools like `barman-cloud-backup-keep`, where these
shortcuts already exist.

References: BAR-356.

- Lighten standby checks related to conninfo and primary_conninfo

When backing up a standby server, Barman performs some checks to assert
that `conninfo` is really pointing to a standby (in recovery mode) and
that `primary_conninfo` is pointing to a primary (not in recovery).

The problem, as reported in the issues 704 and 744, is that when a
failover occurs, the `conninfo` will now be pointing to a primary
instead and the checks will start failing, requiring the user to change
Barman configs manually whenever a failover occurs.

This fix solved the issue by making such checks non-critical, which
means they will still fail but Barman will keep operating regardless.
Essentially, Barman will ignore `primary_conninfo` if `conninfo` does
not point to a standby. Warnings about this misconfiguration will also
be emitted whenever running any Barman command so the user can be aware.

References: BAR-348.

- Check for USAGE instead of MEMBER when calling pg_has_role in Barman

To work correctly Barman database user needs to be included in some roles. Barman was
verifying the conditions was satisfied by calling `pg_has_role` in Postgres. However,
it was check for the `MEMBER` privilege instead of `USAGE`. This oversight was fixed.

This change is a contribution from RealGreenDragon.

References: BAR-489.

Bug fixes

- Fix failures in `barman-cloud-backup-delete`. This command was failing when
applying retention policies due to a bug introduced by the previous release.