Patroni

- Added check for empty subsets (Cody Coons)

Kubernetes 1.10.0+ started returning `Endpoints.subsets` set to `None` instead of `[]`.

**Bootstrap improvements**

- Make deleting recovery.conf optional (Brad Nicholson)

If `bootstrap.<custom_bootstrap_method_name>.keep_existing_recovery_conf` is defined and set to ``True``, Patroni will not remove the existing ``recovery.conf`` file. This is useful when bootstrapping from a backup with tools like pgBackRest that generate the appropriate `recovery.conf` for you.

- Allow options to the basebackup built-in method (Oleksii Kliukin)

It is now possible to supply options to the built-in basebackup method by defining the `basebackup` section in the configuration, similar to how those are defined for custom replica creation methods. The difference is in the format accepted by the `basebackup` section: since pg_basebackup accepts both `--key=value` and `--key` options, the contents of the section could be either a dictionary of key-value pairs, or a list of either one-element dictionaries or just keys (for the options that don't accept values). See :ref:`replica creation method <custom_replica_creation>` section for additional examples.

-------------

Released 2020-08-23

**New features**

- Master stop timeout (Krishna Sarabu)

The number of seconds Patroni is allowed to wait when stopping Postgres. Effective only when ``synchronous_mode`` is enabled. When set to value greater than 0 and the ``synchronous_mode`` is enabled, Patroni sends ``SIGKILL`` to the postmaster if the stop operation is running for more than the value set by ``master_stop_timeout``. Set the value according to your durability/availability tradeoff. If the parameter is not set or set to non-positive value, ``master_stop_timeout`` does not have an effect.

- Don't create permanent physical slot with name of the primary (Alexander Kukushkin)

It is a common problem that the primary recycles WAL segments while the replica is down. Now we have a good solution for static clusters, with a fixed number of nodes and names that never change. You just need to list the names of all nodes in the ``slots`` so the primary will not remove the slot when the node is down (not registered in DCS).

- First draft of Config Validator (Igor Yanchenko)

Use ``patroni --validate-config patroni.yaml`` in order to validate Patroni configuration.

- Possibility to configure max length of timelines history (Krishna Sarabu)

Patroni writes the history of failovers/switchovers into the ``/history`` key in DCS. Over time the size of this key becomes big, but in most cases only the last few lines are interesting. The ``max_timelines_history`` parameter allows to specify the maximum number of timeline history items to be kept in DCS.

- Kazoo 2.7.0 compatibility (Danyal Prout)

Some non-public methods in Kazoo changed their signatures, but Patroni was relying on them.


**Improvements in patronictl**

- Show member tags (Kostiantyn Nemchenko, Alexander Kukushkin)

Tags are configured individually for every node and there was no easy way to get an overview of them

- Improve members output (Alexander Kukushkin)

The redundant cluster name won't be shown anymore on every line, only in the table header.

.. code-block:: bash

$ patronictl list
+ Cluster: batman (6813309862653668387) +---------+----+-----------+---------------------+
| Member | Host | Role | State | TL | Lag in MB | Tags |
+-------------+----------------+--------+---------+----+-----------+---------------------+
| postgresql0 | 127.0.0.1:5432 | Leader | running | 3 | | clonefrom: true |
| | | | | | | noloadbalance: true |
| | | | | | | nosync: true |
+-------------+----------------+--------+---------+----+-----------+---------------------+
| postgresql1 | 127.0.0.1:5433 | | running | 3 | 0.0 | |
+-------------+----------------+--------+---------+----+-----------+---------------------+

- Fail if a config file is specified explicitly but not found (Kaarel Moppel)

Previously ``patronictl`` was only reporting a ``DEBUG`` message.

- Solved the problem of not initialized K8s pod breaking patronictl (Alexander Kukushkin)

Patroni is relying on certain pod annotations on K8s. When one of the Patroni pods is stopping or starting there is no valid annotation yet and ``patronictl`` was failing with an exception.


**Stability improvements**

- Apply 1 second backoff if LIST call to K8s API server failed (Alexander Kukushkin)

It is mostly necessary to avoid flooding logs, but also helps to prevent starvation of the main thread.

- Retry if the ``retry-after`` HTTP header is returned by K8s API (Alexander Kukushkin)

If the K8s API server is overwhelmed with requests it might ask to retry.

- Scrub ``KUBERNETES_`` environment from the postmaster (Feike Steenbergen)

The ``KUBERNETES_`` environment variables are not required for PostgreSQL, yet having them exposed to the postmaster will also expose them to backends and to regular database users (using pl/perl for example).

- Clean up tablespaces on reinitialize (Krishna Sarabu)

During reinit, Patroni was removing only ``PGDATA`` and leaving user-defined tablespace directories. This is causing Patroni to loop in reinit. The previous workarond for the problem was implementing the :ref:`custom bootstrap <custom_bootstrap>` script.

- Explicitly execute ``CHECKPOINT`` after promote happened (Alexander Kukushkin)

It helps to reduce the time before the new primary is usable for ``pg_rewind``.

- Smart refresh of Etcd members (Alexander Kukushkin)

In case Patroni failed to execute a request on all members of the Etcd cluster, Patroni will re-check ``A`` or ``SRV`` records for changes of IPs/hosts before retrying the next time.

- Skip missing values from ``pg_controldata`` (Feike Steenbergen)

Values are missing when trying to use binaries of a version that doesn't match PGDATA. Patroni will try to start Postgres anyway, and Postgres will complain that the major version doesn't match and abort with an error.


**Bugfixes**

- Disable SSL verification for Consul when required (Julien Riou)

Starting from a certain version of ``urllib3``, the ``cert_reqs`` must be explicitly set to ``ssl.CERT_NONE`` in order to effectively disable SSL verification.

- Avoid opening replication connection on every cycle of HA loop (Alexander Kukushkin)

Regression was introduced in 1.6.4.

- Call ``on_role_change`` callback on failed primary (Alexander Kukushkin)

In certain cases it could lead to the virtual IP remaining attached to the old primary. Regression was introduced in 1.4.5.

- Reset rewind state if postgres started after successful pg_rewind (Alexander Kukushkin)

As a result of this bug Patroni was starting up manually shut down postgres in the pause mode.

- Convert ``recovery_min_apply_delay`` to ``ms`` when checking ``recovery.conf``

Patroni was indefinitely restarting replica if ``recovery_min_apply_delay`` was configured on PostgreSQL older than 12.

- PyInstaller compatibility (Alexander Kukushkin)

PyInstaller freezes (packages) Python applications into stand-alone executables. The compatibility was broken when we switched to the ``spawn`` method instead of ``fork`` for ``multiprocessing``.

-------------

Released 2020-01-27

**New features**

- Implemented ``--wait`` option for ``patronictl reinit`` (Igor Yanchenko)

Patronictl will wait for ``reinit`` to finish is the ``--wait`` option is used.

- Further improvements of Windows support (Igor Yanchenko, Alexander Kukushkin)

1. All shell scripts which are used for integration testing are rewritten in python
2. The ``pg_ctl kill`` will be used to stop postgres on non posix systems
3. Don't try to use unix-domain sockets


**Stability improvements**

- Make sure ``unix_socket_directories`` and ``stats_temp_directory`` exist (Igor Yanchenko)

Upon the start of Patroni and Postgres make sure that ``unix_socket_directories`` and ``stats_temp_directory`` exist or try to create them. Patroni will exit if failed to create them.

- Make sure ``postgresql.pgpass`` is located in the place where Patroni has write access (Igor Yanchenko)

In case if it doesn't have a write access Patroni will exit with exception.

- Disable Consul ``serfHealth`` check by default (Kostiantyn Nemchenko)

Even in case of little network problems the failing ``serfHealth`` leads to invalidation of all sessions associated with the node. Therefore, the leader key is lost much earlier than ``ttl`` which causes unwanted restarts of replicas and maybe demotion of the primary.

- Configure tcp keepalives for connections to K8s API (Alexander Kukushkin)

In case if we get nothing from the socket after TTL seconds it can be considered dead.

- Avoid logging of passwords on user creation (Alexander Kukushkin)

If the password is rejected or logging is configured to verbose or not configured at all it might happen that the password is written into postgres logs. In order to avoid it Patroni will change ``log_statement``, ``log_min_duration_statement``, and ``log_min_error_statement`` to some safe values before doing the attempt to create/update user.


**Bugfixes**

- Use ``restore_command`` from the ``standby_cluster`` config on cascading replicas (Alexander Kukushkin)

The ``standby_leader`` was already doing it from the beginning the feature existed. Not doing the same on replicas might prevent them from catching up with standby leader.

- Update timeline reported by the standby cluster (Alexander Kukushkin)

In case of timeline switch the standby cluster was correctly replicating from the primary but ``patronictl`` was reporting the old timeline.

- Allow certain recovery parameters be defined in the custom_conf (Alexander Kukushkin)

When doing validation of recovery parameters on replica Patroni will skip ``archive_cleanup_command``, ``promote_trigger_file``, ``recovery_end_command``, ``recovery_min_apply_delay``, and ``restore_command`` if they are not defined in the patroni config but in files other than ``postgresql.auto.conf`` or ``postgresql.conf``.

- Improve handling of postgresql parameters with period in its name (Alexander Kukushkin)

Such parameters could be defined by extensions where the unit is not necessarily a string. Changing the value might require a restart (for example ``pg_stat_statements.max``).

- Improve exception handling during shutdown (Alexander Kukushkin)

During shutdown Patroni is trying to update its status in the DCS. If the DCS is inaccessible an exception might be raised. Lack of exception handling was preventing logger thread from stopping.

-------------

Released 2019-12-05

**Bugfixes**

- Don't expose password when running ``pg_rewind`` (Alexander Kukushkin)

Bug was introduced in the `1301 <https://github.com/patroni/patroni/pull/1301>`__

- Apply connection parameters specified in the ``postgresql.authentication`` to ``pg_basebackup`` and custom replica creation methods (Alexander Kukushkin)

They were relying on url-like connection string and therefore parameters never applied.

-------------

Released 2019-12-05

**New features**

- Implemented ``patroni --version`` (Igor Yanchenko)

It prints the current version of Patroni and exits.

- Set the ``user-agent`` http header for all http requests (Alexander Kukushkin)

Patroni is communicating with Consul, Etcd, and Kubernetes API via the http protocol. Having a specifically crafted ``user-agent`` (example: ``Patroni/1.6.2 Python/3.6.8 Linux``) might be useful for debugging and monitoring.

- Make it possible to configure log level for exception tracebacks (Igor Yanchenko)

If you set ``log.traceback_level=DEBUG`` the tracebacks will be visible only when ``log.level=DEBUG``. The default behavior remains the same.


**Stability improvements**

- Avoid importing all DCS modules when searching for the module required by the config file (Alexander Kukushkin)

There is no need to import modules for Etcd, Consul, and Kubernetes if we need only e.g. Zookeeper. It helps to reduce memory usage and solves the problem of having INFO messages ``Failed to import smth``.

- Removed python ``requests`` module from explicit requirements (Alexander Kukushkin)

It wasn't used for anything critical, but causing a lot of problems when the new version of ``urllib3`` is released.

- Improve handling of ``etcd.hosts`` written as a comma-separated string instead of YAML array (Igor Yanchenko)

Previously it was failing when written in format ``host1:port1, host2:port2`` (the space character after the comma).


**Usability improvements**

- Don't force users to choose members from an empty list in ``patronictl`` (Igor Yanchenko)

If the user provides a wrong cluster name, we will raise an exception rather than ask to choose a member from an empty list.

- Make the error message more helpful if the REST API cannot bind (Igor Yanchenko)

For an inexperienced user it might be hard to figure out what is wrong from the Python stacktrace.


**Bugfixes**

- Fix calculation of ``wal_buffers`` (Alexander Kukushkin)

The base unit has been changed from 8 kB blocks to bytes in PostgreSQL 11.

- Use ``passfile`` in ``primary_conninfo`` only on PostgreSQL 10+ (Alexander Kukushkin)

On older versions there is no guarantee that ``passfile`` will work, unless the latest version of ``libpq`` is installed.

-------------

Released 2019-11-15

**New features**

- Added ``PATRONICTL_CONFIG_FILE`` environment variable (msvechla)

It allows configuring the ``--config-file`` argument for ``patronictl`` from the environment.

- Implement ``patronictl history`` (Alexander Kukushkin)

It shows the history of failovers/switchovers.

- Pass ``-c statement_timeout=0`` in ``PGOPTIONS`` when doing ``pg_rewind`` (Alexander Kukushkin)

It protects from the case when ``statement_timeout`` on the server is set to some small value and one of the statements executed by pg_rewind is canceled.

- Allow lower values for PostgreSQL configuration (Soulou)

Patroni didn't allow some of the PostgreSQL configuration parameters be set smaller than some hardcoded values. Now the minimal allowed values are smaller, default values have not been changed.

- Allow for certificate-based authentication (Jonathan S. Katz)

This feature enables certificate-based authentication for superuser, replication, rewind accounts and allows the user to specify the ``sslmode`` they wish to connect with.

- Use the ``passfile`` in the ``primary_conninfo`` instead of password (Alexander Kukushkin)

It allows to avoid setting ``600`` permissions on postgresql.conf

- Perform ``pg_ctl reload`` regardless of config changes (Alexander Kukushkin)

It is possible that some config files are not controlled by Patroni. When somebody is doing a reload via the REST API or by sending SIGHUP to the Patroni process, the usual expectation is that Postgres will also be reloaded. Previously it didn't happen when there were no changes in the ``postgresql`` section of Patroni config.

- Compare all recovery parameters, not only ``primary_conninfo`` (Alexander Kukushkin)

Previously the ``check_recovery_conf()`` method was only checking whether ``primary_conninfo`` has changed, never taking into account all other recovery parameters.

- Make it possible to apply some recovery parameters without restart (Alexander Kukushkin)

Starting from PostgreSQL 12 the following recovery parameters could be changed without restart: ``archive_cleanup_command``, ``promote_trigger_file``, ``recovery_end_command``, and ``recovery_min_apply_delay``. In future Postgres releases this list will be extended and Patroni will support it automatically.

- Make it possible to change ``use_slots`` online (Alexander Kukushkin)

Previously it required restarting Patroni and removing slots manually.

- Remove only ``PATRONI_`` prefixed environment variables when starting up Postgres (Cody Coons)

It will solve a lot of problems with running different Foreign Data Wrappers.


**Stability improvements**

- Use LIST + WATCH when working with K8s API (Alexander Kukushkin)

It allows to efficiently receive object changes (pods, endpoints/configmaps) and makes less stress on K8s master nodes.

- Improve the workflow when PGDATA is not empty during bootstrap (Alexander Kukushkin)

According to the ``initdb`` source code it might consider a PGDATA empty when there are only ``lost+found`` and ``.dotfiles`` in it. Now Patroni does the same. If ``PGDATA`` happens to be non-empty, and at the same time not valid from the ``pg_controldata`` point of view, Patroni will complain and exit.

- Avoid calling expensive ``os.listdir()`` on every HA loop (Alexander Kukushkin)

When the system is under IO stress, ``os.listdir()`` could take a few seconds (or even minutes) to execute, badly affecting the HA loop of Patroni. This could even cause the leader key to disappear from DCS due to the lack of updates. There is a better and less expensive way to check that the PGDATA is not empty. Now we check the presence of the ``global/pg_control`` file in the PGDATA.

- Some improvements in logging infrastructure (Alexander Kukushkin)

Previously there was a possibility to loose the last few log lines on shutdown because the logging thread was a ``daemon`` thread.

- Use ``spawn`` multiprocessing start method on python 3.4+ (Maciej Kowalczyk)

It is a known `issue <https://bugs.python.org/issue6721>`__ in Python that threading and multiprocessing do not mix well. Switching from the default method ``fork`` to the ``spawn`` is a recommended workaround. Not doing so might result in the Postmaster starting process hanging and Patroni indefinitely reporting ``INFO: restarting after failure in progress``, while Postgres is actually up and running.

**Improvements in REST API**

- Make it possible to check client certificates in the REST API (Alexander Kukushkin)

If the ``verify_client`` is set to ``required``, Patroni will check client certificates for all REST API calls. When it is set to ``optional``, client certificates are checked for all unsafe REST API endpoints.

- Return the response code 503 for the ``GET /replica`` health check request if Postgres is not running (Alexander Anikin)

Postgres might spend significant time in recovery before it starts accepting client connections.

- Implement ``/history`` and ``/cluster`` endpoints (Alexander Kukushkin)

The ``/history`` endpoint shows the content of the ``history`` key in DCS. The ``/cluster`` endpoint shows all cluster members and some service info like pending and scheduled restarts or switchovers.


**Improvements in Etcd support**

- Retry on Etcd RAFT internal error (Alexander Kukushkin)

When the Etcd node is being shut down, it sends ``response code=300, data='etcdserver: server stopped'``, which was causing Patroni to demote the primary.

- Don't give up on Etcd request retry too early (Alexander Kukushkin)

When there were some network problems, Patroni was quickly exhausting the list of Etcd nodes and giving up without using the whole ``retry_timeout``, potentially resulting in demoting the primary.


**Bugfixes**

- Disable ``synchronous_commit`` when granting execute permissions to the ``pg_rewind`` user (kremius)

If the bootstrap is done with ``synchronous_mode_strict: true`` the `GRANT EXECUTE` statement was waiting indefinitely due to the non-synchronous nodes being available.

- Fix memory leak on python 3.7 (Alexander Kukushkin)

Patroni is using ``ThreadingMixIn`` to process REST API requests and python 3.7 made threads spawn for every request non-daemon by default.

- Fix race conditions in asynchronous actions (Alexander Kukushkin)

There was a chance that ``patronictl reinit --force`` could be overwritten by the attempt to recover stopped Postgres. This ended up in a situation when Patroni was trying to start Postgres while basebackup was running.

- Fix race condition in ``postmaster_start_time()`` method (Alexander Kukushkin)

If the method is executed from the REST API thread, it requires a separate cursor object to be created.

- Fix the problem of not promoting the sync standby that had a name containing upper case letters (Alexander Kukushkin)

We converted the name to the lower case because Postgres was doing the same while comparing the ``application_name`` with the value in ``synchronous_standby_names``.

- Kill all children along with the callback process before starting the new one (Alexander Kukushkin)

Not doing so makes it hard to implement callbacks in bash and eventually can lead to the situation when two callbacks are running at the same time.

- Fix 'start failed' issue (Alexander Kukushkin)

Under certain conditions the Postgres state might be set to 'start failed' despite Postgres being up and running.