Relstorage

==================

- SQLite: Require the database to be in dedicated directory.

.. caution::

This introduces a change to the <sqlite3> configuration.
Please review the documentation. It is possible to migrate a
database created earlier to the new structure, but no automated
tooling or documentation is provided for that.

- SQLite: Allow configuration of many of SQLite's PRAGMAs for advanced
tuning.

- SQLite: Fix resetting OIDs when zapping a storage. This could be a
problem for benchmarks.

- SQLite: Fix large prefetches resulting in ``OperationalError``

- SQLite: Improve the speed of copying transactions into a SQLite
storage (e.g., with zodbconvert).

- SQLite: Substantially improve general performance. See :pr:`368`.

- SQLite: Add the ``gevent sqlite3`` driver that periodically yields
to the gevent loop at configurable intervals.

- PostgreSQL: Improve the speed of writes when using the 'gevent
psycopg2' driver.

==================

- Make SQLite and Oracle both use UPSERT queries instead of multiple
database round trips.

- Fix an exception with large transactions on SQLite.

- Fix compiling the C extension on very new versions of Microsoft
Visual Studio.

===================

- Further speed improvements and memory efficiency gains of around 30%
for the cache.

- Restore support for Python 2.7 on Windows.

- No longer require Cython to build from a sdist (.tar.gz).

- Add support for using a SQLite file as a RelStorage backend, if all
processes accessing it will be on a single machine. The advantage
over FileStorage is that multiple processes can use the database
concurrently. To allow multiple processes to use a FileStorage one
must deploy ZEO, even if all processes are on a single machine. See
:pr:`362`.

- Fix and test Oracle. The minimum required cx_oracle is now 6.0.

- Add support for Python 3.8.

===================

- Add the ``gevent psycopg2`` driver to allow using the fast psycopg2
driver with gevent.

- Conflict resolution prefetches data for conflicted objects, reducing
the number of database queries and locks needed.

- Introduce a driver-agnostic method for elevating database connection
priority during critical times of two-phase commit, and implement it
for the ``gevent MySQLdb`` driver. This reduces the amount of gevent
switches that occur while database locks are held under a carefully
chosen set of circumstances that attempt to balance overall
throughput against latency. See :issue:`339`.

- Drop support for Python 2.7 on Windows. The required compiler is
very old. See :issue:`358`.

- Substantially reduce the overhead of the cache, making it mome
memory efficient. Also make it substantially faster. This was done
by rewriting it in C. See :issue:`358`.

===================

- Make ``poll_invalidations`` handle other retryable internal
exceptions besides just ``ReadConflictError`` so they don't
propagate out to ``transaction.begin()``.

- Make the zodburi resolver entry points not require a specific
RelStorage extra such as 'postgres', in case there is a desire to
use a different database driver than the default that's installed
with that extra. See :issue:`342`, reported by Éloi Rivard.

- Make the zodburi resolvers accept the 'driver' query paramater to
allow selecting a specific driver to use. This functions the same as
in a ZConfig configuration.

- Make the zodburi resolvers more strict on the distinction between
boolean arguments and arbitrary integer arguments. Previously, a
query like ``?read_only=12345&cache_local_mb=yes`` would have been
interpreted as ``True`` and ``1``, respectively. Now it produces errors.

- Fix the calculation of the persistent cache size, especially on
Python 2. This is used to determine when to shrink the disk cache.
See :issue:`317`.

- Fix several race conditions when packing history-free storages
through a combination of changes in ordering and more strongly
consistent (``READ ONLY REPEATABLE READ``) transactions.
Reported in :issue:`325` by krissik with initial PR by Andreas
Gabriel.

- Make ``zodbpack`` pass RelStorage specific options like
``--prepack`` and ``--use-prepack-state`` to the RelStorage, even
when it has been wrapped in a ``zc.zlibstorage``.

- Reduce the amount of memory required to pack a RelStorage through
more careful datastructure choices. On CPython 3, the peak
memory usage of the prepack phase can be up to 9 times less. On
CPython 2, pre-packing a 30MM row storage required 3GB memory; now
it requires about 200MB.

- Use server-side cursors during packing when available, further
reducing the amount of memory required. See :issue:`165`.

- Make history-free database iterators from the same storage use a
consistent view of the database (until a transaction is committed
using the storage or ``sync()`` is called). This prevents data loss
in some cases. See :issue:`344`.

- Make copying transactions *from* a history-free RelStorage (e.g., with
``zodbconvert``) require substantially less memory (75% less).

- Make copying transactions *to* a RelStorage clean up temporary blob
files.

- Make ``zodbconvert`` log progress at intervals instead of for every
transaction. Logging every transaction could add significant overhead
unless stdout was redirected to a file.

- Avoid attempting to lock objects being created. See :issue:`329`.

- Make cache vacuuming faster.

===================

- Fix a bug where the persistent cache might not properly detect
object invalidations if the MVCC index pulled too far ahead at save
time. Now it explicitly checks for invalidations at load time, as
earlier versions did. See :pr:`343`.

- Require perfmetrics 3.0.