Patroni

-------------

Released 2018-03-05

**Improvements in logging**

- Make log level configurable from environment variables (Andy Newton, Keyvan Hedayati)

`PATRONI_LOGLEVEL` - sets the general logging level
`PATRONI_REQUESTS_LOGLEVEL` - sets the logging level for all HTTP requests e.g. Kubernetes API calls
See `the docs for Python logging <https://docs.python.org/3.6/library/logging.html#levels>` to get the names of possible log levels

**Stability improvements and bug fixes**

- Don't rediscover etcd cluster topology when watch timed out (Alexander Kukushkin)

If we have only one host in etcd configuration and exactly this host is not accessible, Patroni was starting discovery of cluster topology and never succeeding. Instead it should just switch to the next available node.

- Write content of bootstrap.pg_hba into a pg_hba.conf after custom bootstrap (Alexander Kukushkin)

Now it behaves similarly to the usual bootstrap with `initdb`

- Single user mode was waiting for user input and never finish (Alexander Kukushkin)

Regression was introduced in https://github.com/patroni/patroni/pull/576

-------------

Released 2018-01-30

**Improvements in patronictl**

- Rename scheduled failover to scheduled switchover (Alexander Kukushkin)

Failover and switchover functions were separated in version 1.4, but `patronictl list` was still reporting `Scheduled failover` instead of `Scheduled switchover`.

- Show information about pending restarts (Alexander Kukushkin)

In order to apply some configuration changes sometimes it is necessary to restart postgres. Patroni was already giving a hint about that in the REST API and when writing node status into DCS, but there were no easy way to display it.

- Make show-config to work with cluster_name from config file (Alexander Kukushkin)

It works similar to the `patronictl edit-config`

**Stability improvements**

- Avoid calling pg_controldata during bootstrap (Alexander Kukushkin)

During initdb or custom bootstrap there is a time window when pgdata is not empty but pg_controldata has not been written yet. In such case pg_controldata call was failing with error messages.

- Handle exceptions raised from psutil (Alexander Kukushkin)

cmdline is read and parsed every time when `cmdline()` method is called. It could happen that the process being examined
has already disappeared, in that case `NoSuchProcess` is raised.

**Kubernetes support improvements**

- Don't swallow errors from k8s API (Alexander Kukushkin)

A call to Kubernetes API could fail for a different number of reasons. In some cases such call should be retried, in some other cases we should log the error message and the exception stack trace. The change here will help debug Kubernetes permission issues.

- Update Kubernetes example Dockerfile to install Patroni from the master branch (Maciej Szulik)

Before that it was using `feature/k8s`, which became outdated.

- Add proper RBAC to run patroni on k8s (Maciej Szulik)

Add the Service account that is assigned to the pods of the cluster, the role that holds only the necessary permissions, and the rolebinding that connects the Service account and the Role.

-------------

Released 2018-01-17

**Fixes in patronictl**

- Don't show current leader in suggested list of members to failover to. (Alexander Kukushkin)

patronictl failover could still work when there is leader in the cluster and it should be excluded from the list of member where it is possible to failover to.

- Make patronictl switchover compatible with the old Patroni api (Alexander Kukushkin)

In case if POST /switchover REST API call has failed with status code 501 it will do it once again, but to /failover endpoint.

-----------

Released 2018-01-10

This version adds support for using Kubernetes as a DCS, allowing to run Patroni as a cloud-native agent in Kubernetes without any additional deployments of Etcd, Zookeeper or Consul.

**Upgrade notice**

Installing Patroni via pip will no longer bring in dependencies for (such as libraries for Etcd, Zookeper, Consul or Kubernetes, or support for AWS). In order to enable them one need to list them in pip install command explicitly, for instance `pip install patroni[etcd,kubernetes]`.

**Kubernetes support**

Implement Kubernetes-based DCS. The endpoints meta-data is used in order to store the configuration and the leader key. The meta-data field inside the pods definition is used to store the member-related data.
In addition to using Endpoints, Patroni supports ConfigMaps. You can find more information about this feature in the :ref:`Kubernetes chapter of the documentation <kubernetes>`

**Stability improvements**

- Factor out postmaster process into a separate object (Ants Aasma)

This object identifies a running postmaster process via pid and start time and simplifies detection (and resolution) of situations when the postmaster was restarted behind our back or when postgres directory disappeared from the file system.

- Minimize the amount of SELECT's issued by Patroni on every loop of HA cycle (Alexander Kukushkin)

On every iteration of HA loop Patroni needs to know recovery status and absolute wal position. From now on Patroni will run only single SELECT to get this information instead of two on the replica and three on the master.

- Remove leader key on shutdown only when we have the lock (Ants Aasma)

Unconditional removal was generating unnecessary and misleading exceptions.

**Improvements in patronictl**

- Add version command to patronictl (Ants Aasma)

It will show the version of installed Patroni and versions of running Patroni instances (if the cluster name is specified).

- Make optional specifying cluster_name argument for some of patronictl commands (Alexander Kukushkin, Ants Aasma)

It will work if patronictl is using usual Patroni configuration file with the ``scope`` defined.

- Show information about scheduled switchover and maintenance mode (Alexander Kukushkin)

Before that it was possible to get this information only from Patroni logs or directly from DCS.

- Improve ``patronictl reinit`` (Alexander Kukushkin)

Sometimes ``patronictl reinit`` refused to proceed when Patroni was busy with other actions, namely trying to start postgres. `patronictl` didn't provide any commands to cancel such long running actions and the only (dangerous) workarond was removing a data directory manually. The new implementation of `reinit` forcefully cancels other long-running actions before proceeding with reinit.

- Implement ``--wait`` flag in ``patronictl pause`` and ``patronictl resume`` (Alexander Kukushkin)

It will make ``patronictl`` wait until the requested action is acknowledged by all nodes in the cluster.
Such behaviour is achieved by exposing the ``pause`` flag for every node in DCS and via the REST API.

- Rename ``patronictl failover`` into ``patronictl switchover`` (Alexander Kukushkin)

The previous ``failover`` was actually only capable of doing a switchover; it refused to proceed in a cluster without the leader.

- Alter the behavior of ``patronictl failover`` (Alexander Kukushkin)

It will work even if there is no leader, but in that case you will have to explicitly specify a node which should become the new leader.

**Expose information about timeline and history**

- Expose current timeline in DCS and via API (Alexander Kukushkin)

Store information about the current timeline for each member of the cluster. This information is accessible via the API and is stored in the DCS

- Store promotion history in the /history key in DCS (Alexander Kukushkin)

In addition, store the timeline history enriched with the timestamp of the corresponding promotion in the /history key in DCS and update it with each promote.

**Add endpoints for getting synchronous and asynchronous replicas**

- Add new /sync and /async endpoints (Alexander Kukushkin, Oleksii Kliukin)

Those endpoints (also accessible as /synchronous and /asynchronous) return 200 only for synchronous and asynchronous replicas correspondingly (excluding those marked as `noloadbalance`).

**Allow multiple hosts for Etcd**

- Add a new `hosts` parameter to Etcd configuration (Alexander Kukushkin)

This parameter should contain the initial list of hosts that will be used to discover and populate the list of the running etcd cluster members. If for some reason during work this list of discovered hosts is exhausted (no available hosts from that list), Patroni will return to the initial list from the `hosts` parameter.

-------------

Released 2017-11-10

**Stability improvements**

- Verify process start time when checking if postgres is running. (Ants Aasma)

After a crash that doesn't clean up postmaster.pid there could be a new process with the same pid, resulting in a false positive for is_running(), which will lead to all kinds of bad behavior.

- Shutdown postgresql before bootstrap when we lost data directory (ainlolcat)

When data directory on the master is forcefully removed, postgres process can still stay alive for some time and prevent the replica created in place of that former master from starting or replicating.
The fix makes Patroni cache the postmaster pid and its start time and let it terminate the old postmaster in case it is still running after the corresponding data directory has been removed.

- Perform crash recovery in a single user mode if postgres master dies (Alexander Kukushkin)

It is unsafe to start immediately as a standby and not possible to run ``pg_rewind`` if postgres hasn't been shut down cleanly.
The single user crash recovery only kicks in if ``pg_rewind`` is enabled or there is no master at the moment.

**Consul improvements**

- Make it possible to provide datacenter configuration for Consul (Vilius Okockis, Alexander Kukushkin)

Before that Patroni was always communicating with datacenter of the host it runs on.

- Always send a token in X-Consul-Token http header (Alexander Kukushkin)

If ``consul.token`` is defined in Patroni configuration, we will always send it in the 'X-Consul-Token' http header.
python-consul module tries to be "consistent" with Consul REST API, which doesn't accept token as a query parameter for `session API <https://www.consul.io/api/session.html>`__, but it still works with 'X-Consul-Token' header.

- Adjust session TTL if supplied value is smaller than the minimum possible (Stas Fomin, Alexander Kukushkin)

It could happen that the TTL provided in the Patroni configuration is smaller than the minimum one supported by Consul. In that case, Consul agent fails to create a new session.
Without a session Patroni cannot create member and leader keys in the Consul KV store, resulting in an unhealthy cluster.

**Other improvements**

- Define custom log format via environment variable ``PATRONI_LOGFORMAT`` (Stas Fomin)

Allow disabling timestamps and other similar fields in Patroni logs if they are already added by the system logger (usually when Patroni runs as a service).

-------------

Released 2017-10-12

**Bugfix**

- Set role to 'uninitialized' if data directory was removed (Alexander Kukushkin)

If the node was running as a master it was preventing from failover.

**Stability improvement**

- Try to run postmaster in a single-user mode if we tried and failed to start postgres (Alexander Kukushkin)

Usually such problem happens when node running as a master was terminated and timelines were diverged.
If ``recovery.conf`` has ``restore_command`` defined, there are really high chances that postgres will abort startup and leave controldata unchanged.
It makes impossible to use ``pg_rewind``, which requires a clean shutdown.

**Consul improvements**

- Make it possible to specify health checks when creating session (Alexander Kukushkin)

If not specified, Consul will use "serfHealth". From one side it allows fast detection of isolated master, but from another side it makes it impossible for Patroni to tolerate short network lags.

**Bugfix**

- Fix watchdog on Python 3 (Ants Aasma)

A misunderstanding of the ioctl() call interface. If mutable=False then fcntl.ioctl() actually returns the arg buffer back.
This accidentally worked on Python2 because int and str comparison did not return an error.
Error reporting is actually done by raising IOError on Python2 and OSError on Python3.