Relstorage

====================

- Correct a packaging problem in 3.5.0a5 (which is not on PyPI).

====================

- Allow RelStorage to be used in a FIPS enabled environment. See
:issue:`480`
- Fix ``RelStorage.zap_all()`` and ``zodbconvert --clear`` against
existing PostgreSQL databases with very large numbers of Blobs and
relatively small amounts of shared memory (e.g., default values for
``max_locks_per_transaction`` and ``max_connections``). Previously,
this could raise an ``out of shared memory`` error. See
:issue:`468`.
- Use C++ hashmaps and sets to store maps and sets of transaction IDs
and object IDs instead of using BTrees. The memory footprint is about
the same, but the performance is better for common operations (e.g.,
``O(1)`` for lookups instead of logarithmic.) See :pr:`479`.
- Rewrite the cache vacuum algorithm and supporting data structures to
be substantially faster. See :issue:`474`.

====================

- Stop closing RDBMS connections when ``tpc_vote`` raises a
semi-expected ``TransientError`` such as a ``ConflictError``.
- PostgreSQL: Now uses advisory locks instead of row-level locks
during the commit process. This benchmarks substantially faster and
reduces the potential for table bloat.

For environments that process many large, concurrent transactions,
or deploy many RelStorage instances to the same database server, it
might be necessary to increase the PostgreSQL configuration value
``max_locks_per_transaction.`` The default value of 64 is multiplied
by the default value of ``max_connections`` (100) to allow for 6,400
total objects to be locked across the entire database server. See
`the PostgreSQL documentation
<https://www.postgresql.org/docs/13/runtime-config-locks.html>`_ for
more information.

.. caution:: Be careful deploying this version while older versions
are executing. There could be a small window of time
where the locking strategies are different, leading to
database corruption.

.. note:: Deploying multiple RelStorage instances to separate
schemas in the same PostgreSQL database (e.g., the default
of "public" plus another) has never been supported. It is
even less supported now.

See :pr:`476`.

====================

- PostgreSQL: Stop sorting rows unnecessarily during the
``lock_and_move`` part of ``tpc_finish`` (MySQL was already not
sorting). On larger transactions and/or busier servers, this shows a
slight performance increase in benchmarks.

- Include the transaction ID in log messages about long-running
transactions (once available).

====================

- Revert :issue:`469` and return to taking shared locks before
exclusive locks. Testing in a large, busy application indicated that
performance was overall slightly worse this way. See :pr:`471`.

- Use the cache to cheaply check if a ``readCurrent()`` violation
will take place during an early part of two-phase commit, instead of
waiting until ``tpc_vote`` when we've sent data to the database. If
the cache can prove that there is a newer version of an object
stored, the conflict error will be raised during ``commit``; if the
cache can't prove it, the error will still be raised during
``tpc_vote``.

This more closely matches what ``FileStorage`` does and can help
avoid some unnecessary work.

- Fix the speed of getting the approximate number of objects in a
storage by using ``len(storage)`` on PostgreSQL. This was a
regression after 3.0a13.

====================

- Increase the default value of the ``RS_CACHE_MVCC_MAX_DEPTH``
advanced tuning parameter from 100 to 1000 based on observations of
production workloads. (Connections that haven't polled
for the last ``RS_CACHE_MVCC_MAX_DEPTH`` committed transactions ---
and thus are expected to have a large number of invalidations ---
are "detached" and forced to invalidate their entire persistent
object cache if they get used again.)

- Add StatsD counter metric
"relstorage.cache.mvcc.invalidate_all_detached" that is incremented
when a previously-detached Connection is required to invalidate its
entire persistent object cache. In a well-tuned environment, this
counter should be very low and as such is not sampled but always sent.

- Fix the logging of some environment variables RelStorage uses.

- If there is a read conflict error, PostgreSQL no longer holds any
database locks while the error is raised and the transaction is
rolled back in Python. Previously, shared locks could be held during
this process, preventing other transactions from moving forward.

- Take exclusive locks first, and then shared locks in NOWAIT mode.
This reverses :pr:`317`, but it eliminates the requirement that the
database server finds and breaks deadlocks (by eliminating
deadlocks). Deadlocks could never be resolved without retrying the
entire transaction, and which transaction got killed was unknowable.
Provisions are made to keep fast detection of ``readCurrent``
conflicts. Benchmarks with zodbshootout find no substantial
differences. See :issue:`469`.