Relstorage

==================

- 3.6.0 final release is tested on CI servers.
- Substantial performance improvements for PostgreSQL, both on reading
and writing. Reading objects can be 20-40% faster. Writing objects
can be 15-25% faster (the most benefit will be seen by history-free
databases on PostgreSQL 9.5 and above). MySQL may have a (much)
smaller improvement too, especially for small transactions. This was
done through the use of prepared statements for the most important
queries and the new `'ON CONFLICT UPDATE'
<https://wiki.postgresql.org/wiki/What's_new_in_PostgreSQL_9.5#INSERT_..._ON_CONFLICT_DO_NOTHING.2FUPDATE_.28.22UPSERT.22.29>`_
syntax. See :pr:`157` and :issue:`156`.
- The umysqldb driver no longer attempts to automatically reconnect on
a closed cursor exception. That fails now that prepared statements
are in use. Instead, it translates the internal exception to one
that the higher layers of RelStorage recognize as requiring
reconnection at consistent times (transaction boundaries).
- Add initial support for the `MySQL Connector/Python
<https://dev.mysql.com/doc/connector-python/en/>`_ driver. See
:issue:`155`.
- Backport `ZODB 140
<https://github.com/zopefoundation/ZODB/pull/140>`_ to older
versions of ZODB. This improves write performance, especially in
multi-threaded scenarios, by up to 10%. See :pr:`160`.
- MySQL temporary tables now use the InnoDB engine instead of MyISAM.
See :pr:`162`.

==================

- MySQL and Postgres now use the same optimized methods to get the
latest TID at transaction commit time as they do at poll time. This
is similar to :issue:`89`.
- MySQL now releases the commit lock (if acquired) during pre-pack
with GC of a history-free storage at the same time as PostgreSQL and
Oracle did (much earlier). Reported and initial fix provided in
:pr:`9` by jplouis.

=====================

- Writing persistent cache files has been changed to reduce the risk
of stale temporary files remaining. Also, files are kept open for a
shorter period of time and removed in a way that should work better
on Windows.

- RelStorage is now tested on Windows for MySQL and PostgreSQL thanks
to AppVeyor.

- Add support for Python 3.6.

====================

- The MySQL adapter will now produce a more informative error if it
gets an unexpected result taking the commit lock. Reported by Josh
Zuech.

- Compatibility with transaction 2.0 on older versions of ZODB (prior
to the unreleased version that handles encoding meta data for us),
newer versions of ZODB (that do the encoding), while maintaining
compatibility with transaction 1.x. In particular, the ``history``
method consistently returns bytes for username and description.

- In very rare cases, persistent cache files could result in a corrupt
cache state in memory after loading them, resulting in
AttributeErrors until the cache files were removed and the instance
restarted. Reported in :issue:`140` by Carlos Sanchez.

====================

- List CFFI in `setup_requires` for buildout users.

====================

- Add the ability to limit the persistent cache files size. Thanks to
Josh Zuech for the suggestion, which led to the next change.

- Move the RelStorage shared cache to a `windowed-LFU with segmented
LRU
<http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html>`_
instead of a pure LRU model. This can be a nearly optimal caching
strategy for many workloads. The caching code itself is also faster
in all tested cases.

It's especially helpful when using persistent cache files together
with a file size limit, as we can now ensure we write out the most
frequently useful data to the file instead of just the newest.

For more information see :issue:`127` and :pr:`128`. Thanks to Ben
Manes for assistance talking through issues related to the cache
strategy.

For write-heavy workloads, you may want to increase
``cache_delta_size_limit``.

The internal implementation details of the cache have been
completely changed. Only the ``StorageCache`` class remains
unchanged (though that's also an implementation class). CFFI is now
required, and support for PyPy versions older than 2.6.1 has been dropped.

- On CPython, use LLBTrees for the cache delta maps. This allows using
a larger, more effective size while reducing memory usage. Fixes :issue:`130`.

- Persistent cache files use the latest TID in the cache as the file's
modification time. This allows a more accurate choice of which file
to read at startup. Fixes :issue:`126`.

- Fix packing of history-preserving Oracle databases. Reported in
:issue:`135` by Peter Jacobs.