Patroni

-------------

Released 2019-08-05

This version adds compatibility with PostgreSQL 12, makes is possible to run pg_rewind without superuser on PostgreSQL 11 and newer, and enables IPv6 support.


**New features**

- Psycopg2 was removed from requirements and must be installed independently (Alexander Kukushkin)

Starting from 2.8.0 ``psycopg2`` was split into two different packages, ``psycopg2``, and ``psycopg2-binary``, which could be installed at the same time into the same place on the filesystem. In order to decrease dependency hell problem, we let a user choose how to install it. There are a few options available, please consult the :ref:`documentation <psycopg2_install_options>`.

- Compatibility with PostgreSQL 12 (Alexander Kukushkin)

Starting from PostgreSQL 12 there is no ``recovery.conf`` anymore and all former recovery parameters are converted into `GUC <https://www.enterprisedb.com/blog/what-is-a-guc-variable>`_. In order to protect from ``ALTER SYSTEM SET primary_conninfo`` or similar, Patroni will parse ``postgresql.auto.conf`` and remove all standby and recovery parameters from there. Patroni config remains backward compatible. For example despite ``restore_command`` being a GUC, one can still specify it in the ``postgresql.recovery_conf.restore_command`` section and Patroni will write it into ``postgresql.conf`` for PostgreSQL 12.

- Make it possible to use ``pg_rewind`` without superuser on PostgreSQL 11 and newer (Alexander Kukushkin)

If you want to use this feature please define ``username`` and ``password`` in the ``postgresql.authentication.rewind`` section of Patroni configuration file. For an already existing cluster you will have to create the user manually and ``GRANT EXECUTE`` permission on a few functions. You can find more details in the PostgreSQL `documentation <https://www.postgresql.org/docs/11/app-pgrewind.html#id-1.9.5.8.8>`__.

- Do a smart comparison of actual and desired ``primary_conninfo`` values on replicas (Alexander Kukushkin)

It might help to avoid replica restart when you are converting an already existing primary-standby cluster to one managed by Patroni

- IPv6 support (Alexander Kukushkin)

There were two major issues. Patroni REST API service was listening only on ``0.0.0.0`` and IPv6 IP addresses used in the ``api_url`` and ``conn_url`` were not properly quoted.

- Kerberos support (Ajith Vilas, Alexander Kukushkin)

It makes possible using Kerberos authentication between Postgres nodes instead of defining passwords in Patroni configuration file

- Manage ``pg_ident.conf`` (Alexander Kukushkin)

This functionality works similarly to ``pg_hba.conf``: if the ``postgresql.pg_ident`` is defined in the config file or DCS, Patroni will write its value to ``pg_ident.conf``, however, if ``postgresql.parameters.ident_file`` is defined, Patroni will assume that ``pg_ident`` is managed from outside and not update the file.


**Improvements in REST API**

- Added ``/health`` endpoint (Wilfried Roset)

It will return an HTTP status code only if PostgreSQL is running

- Added ``/read-only`` and ``/read-write`` endpoints (Julien Riou)

The ``/read-only`` endpoint enables reads balanced across replicas and the primary. The ``/read-write`` endpoint is an alias for ``/primary``, ``/leader`` and ``/master``.

- Use ``SSLContext`` to wrap the REST API socket (Julien Riou)

Usage of ``ssl.wrap_socket()`` is deprecated and was still allowing soon-to-be-deprecated protocols like TLS 1.1.


**Logging improvements**

- Two-step logging (Alexander Kukushkin)

All log messages are first written into the in-memory queue and later they are asynchronously flushed into the stderr or file from a separate thread. The maximum queue size is limited (configurable). If the limit is reached, Patroni will start losing logs, which is still better than blocking the HA loop.

- Enable debug logging for GET/OPTIONS API calls together with latency (Jan Tomsa)

It will help with debugging of health-checks performed by HAProxy, Consul or other tooling that decides which node is the primary/replica.

- Log exceptions caught in Retry (Daniel Kucera)

Log the final exception when either the number of attempts or the timeout were reached. It will hopefully help to debug some issues when communication to DCS fails.


**Improvements in patronictl**

- Enhance dialogues for scheduled switchover and restart (Rafia Sabih)

Previously dialogues did not take into account scheduled actions and therefore were misleading.

- Check if config file exists (Wilfried Roset)

Be verbose about configuration file when the given filename does not exists, instead of ignoring silently (which can lead to misunderstanding).

- Add fallback value for ``EDITOR`` (Wilfried Roset)

When the ``EDITOR`` environment variable was not defined, ``patronictl edit-config`` was failing with `PatroniCtlException`. The new strategy is to try ``editor`` and than ``vi``, which should be available on most systems.


**Improvements in Consul support**

- Allow to specify Consul consistency mode (Jan Tomsa)

You can read more about consistency mode `here <https://www.consul.io/api/features/consistency.html>`__.

- Reload Consul config on SIGHUP (Cameron Daniel Kucera, Alexander Kukushkin)

It is especially useful when somebody is changing the value of ``token``.


**Bugfixes**

- Fix corner case in switchover/failover (Sharoon Thomas)

The variable ``scheduled_at`` may be undefined if REST API is not accessible and we are using DCS as a fallback.

- Open trust to localhost in ``pg_hba.conf`` during custom bootstrap (Alexander Kukushkin)

Previously it was open only to unix_socket, which was causing a lot of errors: ``FATAL: no pg_hba.conf entry for replication connection from host "127.0.0.1", user "replicator"``

- Consider synchronous node as healthy even when the former leader is ahead (Alexander Kukushkin)

If the primary loses access to the DCS, it restarts Postgres in read-only, but it might happen that other nodes can still access the old primary via the REST API. Such a situation was causing the synchronous standby not to promote because the old primary was reporting WAL position ahead of the synchronous standby.

- Standby cluster bugfixes (Alexander Kukushkin)

Make it possible to bootstrap a replica in a standby cluster when the standby_leader is not accessible and a few other minor fixes.

-------------

Released 2019-08-03

**New features**

- Support work with etcd cluster via set of proxies (Alexander Kukushkin)

It might happen that etcd cluster is not accessible directly but via set of proxies. In this case Patroni will not perform etcd topology discovery but just round-robin via proxy hosts. Behavior is controlled by `etcd.use_proxies`.

- Changed callbacks behavior when role on the node is changed (Alexander Kukushkin)

If the role was changed from `master` or `standby_leader` to `replica` or from `replica` to `standby_leader`, `on_restart` callback will not be called anymore in favor of `on_role_change` callback.

- Change the way how we start postgres (Alexander Kukushkin)

Use `multiprocessing.Process` instead of executing itself and `multiprocessing.Pipe` to transmit the postmaster pid to the Patroni process. Before that we were using pipes, what was leaving postmaster process with stdin closed.

**Bug fixes**

- Fix role returned by REST API for the standby leader (Alexander Kukushkin)

It was incorrectly returning `replica` instead of `standby_leader`

- Wait for callback end if it could not be killed (Julien Tachoires)

Patroni doesn't have enough privileges to terminate the callback script running under `sudo` what was cancelling the new callback. If the running script could not be killed, Patroni will wait until it finishes and then run the next callback.

- Reduce lock time taken by dcs.get_cluster method (Alexander Kukushkin)

Due to the lock being held DCS slowness was affecting the REST API health checks causing false positives.

- Improve cleaning of PGDATA when `pg_wal`/`pg_xlog` is a symlink (Julien Tachoires)

In this case Patroni will explicitly remove files from the target directory.

- Remove unnecessary usage of os.path.relpath (Ants Aasma)

It depends on being able to resolve the working directory, what will fail if Patroni is started in a directory that is later unlinked from the filesystem.

- Do not enforce ssl version when communicating with Etcd (Alexander Kukushkin)

For some unknown reason python3-etcd on debian and ubuntu are not based on the latest version of the package and therefore it enforces TLSv1 which is not supported by Etcd v3. We solved this problem on Patroni side.

-------------

Released 2019-02-15

This version introduces the possibility of automatic reinit of the former master, improves patronictl list output and fixes a number of bugs.

**New features**

- Add support of `PATRONI_ETCD_PROTOCOL`, `PATRONI_ETCD_USERNAME` and `PATRONI_ETCD_PASSWORD` environment variables (Étienne M)

Before it was possible to configure them only in the config file or as a part of `PATRONI_ETCD_URL`, which is not always convenient.

- Make it possible to automatically reinit the former master (Alexander Kukushkin)

If the pg_rewind is disabled or can't be used, the former master could fail to start as a new replica due to diverged timelines. In this case, the only way to fix it is wiping the data directory and reinitializing. This behavior could be changed by setting `postgresql.remove_data_directory_on_diverged_timelines`. When it is set, Patroni will wipe the data directory and reinitialize the former master automatically.

- Show information about timelines in patronictl list (Alexander Kukushkin)

It helps to detect stale replicas. In addition to that, `Host` will include ':{port}' if the port value isn't default or there is more than one member running on the same host.

- Create a headless service associated with the $SCOPE-config endpoint (Alexander Kukushkin)

The "config" endpoint keeps information about the cluster-wide Patroni and Postgres configuration, history file, and last but the most important, it holds the `initialize` key. When the Kubernetes master node is restarted or upgraded, it removes endpoints without services. The headless service will prevent it from being removed.

**Bug fixes**

- Adjust the read timeout for the leader watch blocking query (Alexander Kukushkin)

According to the Consul documentation, the actual response timeout is increased by a small random amount of additional wait time added to the supplied maximum wait time to spread out the wake up time of any concurrent requests. It adds up to `wait / 16` additional time to the maximum duration. In our case we are adding `wait / 15` or 1 second depending on what is bigger.

- Always use replication=1 when connecting via replication protocol to the postgres (Alexander Kukushkin)

Starting from Postgres 10 the line in the pg_hba.conf with database=replication doesn't accept connections with the parameter replication=database.

- Don't write primary_conninfo into recovery.conf for wal-only standby cluster (Alexander Kukushkin)

Despite not having neither `host` nor `port` defined in the `standby_cluster` config, Patroni was putting the `primary_conninfo` into the `recovery.conf`, which is useless and generating a lot of errors.

-------------

Released 2019-01-15

This version implements flexible logging and fixes a number of bugs.

**New features**

- Improvements in logging infrastructure (Alexander Kukushkin, Lucas Capistrant, Alexander Anikin)

Logging configuration could be configured not only from environment variables but also from Patroni config file. It makes it possible to change logging configuration in runtime by updating config and doing reload or sending SIGHUP to the Patroni process. By default Patroni writes logs to stderr, but now it becomes possible to write logs directly into the file and rotate when it reaches a certain size. In addition to that added support of custom dateformat and the possibility to fine-tune log level for each python module.

- Make it possible to take into account the current timeline during leader elections (Alexander Kukushkin)

It could happen that the node is considering itself as a healthiest one although it is currently not on the latest known timeline. In some cases we want to avoid promoting of such node, which could be achieved by setting `check_timeline` parameter to `true` (default behavior remains unchanged).

- Relaxed requirements on superuser credentials

Libpq allows opening connections without explicitly specifying neither username nor password. Depending on situation it relies either on pgpass file or trust authentication method in pg_hba.conf. Since pg_rewind is also using libpq, it will work the same way.

- Implemented possibility to configure Consul Service registration and check interval via environment variables (Alexander Kukushkin)

Registration of service in Consul was added in the 1.5.0, but so far it was only possible to turn it on via patroni.yaml.

**Stability Improvements**

- Set archive_mode to off during the custom bootstrap (Alexander Kukushkin)

We want to avoid archiving wals and history files until the cluster is fully functional. It really helps if the custom bootstrap involves pg_upgrade.

- Apply five seconds backoff when loading global config on start (Alexander Kukushkin)

It helps to avoid hammering DCS when Patroni just starting up.

- Reduce amount of error messages generated on shutdown (Alexander Kukushkin)

They were harmless but rather annoying and sometimes scary.

- Explicitly secure rw perms for recovery.conf at creation time (Lucas Capistrant)

We don't want anybody except patroni/postgres user reading this file, because it contains replication user and password.

- Redirect HTTPServer exceptions to logger (Julien Riou)

By default, such exceptions were logged on standard output messing with regular logs.

**Bug fixes**

- Removed stderr pipe to stdout on pg_ctl process (Cody Coons)

Inheriting stderr from the main Patroni process allows all Postgres logs to be seen along with all patroni logs. This is very useful in a container environment as Patroni and Postgres logs may be consumed using standard tools (docker logs, kubectl, etc). In addition to that, this change fixes a bug with Patroni not being able to catch postmaster pid when postgres writing some warnings into stderr.

- Set Consul service check deregister timeout in Go time format (Pavel Kirillov)

Without explicitly mentioned time unit registration was failing.

- Relax checks of standby_cluster cluster configuration (Dmitry Dolgov, Alexander Kukushkin)

It was accepting only strings as valid values and therefore it was not possible to specify the port as integer and create_replica_methods as a list.

-------------

Released 2018-12-03

Compatibility and bugfix release.

- Improve stability when running with python3 against zookeeper (Alexander Kukushkin)

Change of `loop_wait` was causing Patroni to disconnect from zookeeper and never reconnect back.

- Fix broken compatibility with postgres 9.3 (Alexander Kukushkin)

When opening a replication connection we should specify replication=1, because 9.3 does not understand replication='database'

- Make sure we refresh Consul session at least once per HA loop and improve handling of consul sessions exceptions (Alexander Kukushkin)

Restart of local consul agent invalidates all sessions related to the node. Not calling session refresh on time and not doing proper handling of session errors was causing demote of the primary.

-------------

Released 2018-11-26

Compatibility and bugfix release.

- Compatibility with kazoo-2.6.0 (Alexander Kukushkin)

In order to make sure that requests are performed with an appropriate timeout, Patroni redefines create_connection method from python-kazoo module. The last release of kazoo slightly changed the way how create_connection method is called.

- Fix Patroni crash when Consul cluster loses the leader (Alexander Kukushkin)

The crash was happening due to incorrect implementation of touch_member method, it should return boolean and not raise any exceptions.