Sqlalchemy

:released: March 4, 2024

.. change::
:tags: engine, usecase
:tickets: 10974

Added new core execution option
:paramref:`_engine.Connection.execution_options.preserve_rowcount`. When
set, the ``cursor.rowcount`` attribute from the DBAPI cursor will be
unconditionally memoized at statement execution time, so that whatever
value the DBAPI offers for any kind of statement will be available using
the :attr:`_engine.CursorResult.rowcount` attribute from the
:class:`_engine.CursorResult`. This allows the rowcount to be accessed for
statements such as INSERT and SELECT, to the degree supported by the DBAPI
in use. The :ref:`engine_insertmanyvalues` also supports this option and
will ensure :attr:`_engine.CursorResult.rowcount` is correctly set for a
bulk INSERT of rows when set.

.. change::
:tags: bug, orm, regression
:tickets: 11010

Fixed regression caused by :ticket:`9779` where using the "secondary" table
in a relationship ``and_()`` expression would fail to be aliased to match
how the "secondary" table normally renders within a
:meth:`_sql.Select.join` expression, leading to an invalid query.

.. change::
:tags: bug, orm, performance, regression
:tickets: 11085

Adjusted the fix made in :ticket:`10570`, released in 2.0.23, where new
logic was added to reconcile possibly changing bound parameter values
across cache key generations used within the :func:`_orm.with_expression`
construct. The new logic changes the approach by which the new bound
parameter values are associated with the statement, avoiding the need to
deep-copy the statement which can result in a significant performance
penalty for very deep / complex SQL constructs. The new approach no longer
requires this deep-copy step.

.. change::
:tags: bug, asyncio
:tickets: 8771

An error is raised if a :class:`.QueuePool` or other non-asyncio pool class
is passed to :func:`_asyncio.create_async_engine`. This engine only
accepts asyncio-compatible pool classes including
:class:`.AsyncAdaptedQueuePool`. Other pool classes such as
:class:`.NullPool` are compatible with both synchronous and asynchronous
engines as they do not perform any locking.

.. seealso::

:ref:`pool_api`


.. change::
:tags: change, tests

pytest support in the tox.ini file has been updated to support pytest 8.1.

.. changelog::

:released: February 13, 2024

.. change::
:tags: bug, postgresql, regression
:tickets: 11005

Fixed regression caused by just-released fix for :ticket:`10863` where an
invalid exception class were added to the "except" block, which does not
get exercised unless such a catch actually happens. A mock-style test has
been added to ensure this catch is exercised in unit tests.


.. changelog::

:released: February 11, 2024

.. change::
:tags: usecase, postgresql, reflection
:tickets: 10777

Added support for reflection of PostgreSQL CHECK constraints marked with
"NO INHERIT", setting the key ``no_inherit=True`` in the reflected data.
Pull request courtesy Ellis Valentiner.

.. change::
:tags: bug, sql
:tickets: 10843

Fixed issues in :func:`_sql.case` where the logic for determining the
type of the expression could result in :class:`.NullType` if the last
element in the "whens" had no type, or in other cases where the type
could resolve to ``None``. The logic has been updated to scan all
given expressions so that the first non-null type is used, as well as
to always ensure a type is present. Pull request courtesy David Evans.

.. change::
:tags: bug, mysql
:tickets: 10850

Fixed issue where NULL/NOT NULL would not be properly reflected from a
MySQL column that also specified the VIRTUAL or STORED directives. Pull
request courtesy Georg Wicke-Arndt.

.. change::
:tags: bug, regression, postgresql
:tickets: 10863

Fixed regression in the asyncpg dialect caused by :ticket:`10717` in
release 2.0.24 where the change that now attempts to gracefully close the
asyncpg connection before terminating would not fall back to
``terminate()`` for other potential connection-related exceptions other
than a timeout error, not taking into account cases where the graceful
``.close()`` attempt fails for other reasons such as connection errors.


.. change::
:tags: oracle, bug, performance
:tickets: 10877

Changed the default arraysize of the Oracle dialects so that the value set
by the driver is used, that is 100 at the time of writing for both
cx_oracle and oracledb. Previously the value was set to 50 by default. The
setting of 50 could cause significant performance regressions compared to
when using cx_oracle/oracledb alone to fetch many hundreds of rows over
slower networks.

.. change::
:tags: bug, mysql
:tickets: 10893

Fixed issue in asyncio dialects asyncmy and aiomysql, where their
``.close()`` method is apparently not a graceful close. replace with
non-standard ``.ensure_closed()`` method that's awaitable and move
``.close()`` to the so-called "terminate" case.

.. change::
:tags: bug, orm
:tickets: 10896

Replaced the "loader depth is excessively deep" warning with a shorter
message added to the caching badge within SQL logging, for those statements
where the ORM disabled the cache due to a too-deep chain of loader options.
The condition which this warning highlights is difficult to resolve and is
generally just a limitation in the ORM's application of SQL caching. A
future feature may include the ability to tune the threshold where caching
is disabled, but for now the warning will no longer be a nuisance.

.. change::
:tags: bug, orm
:tickets: 10899

Fixed issue where it was not possible to use a type (such as an enum)
within a :class:`_orm.Mapped` container type if that type were declared
locally within the class body. The scope of locals used for the eval now
includes that of the class body itself. In addition, the expression within
:class:`_orm.Mapped` may also refer to the class name itself, if used as a
string or with future annotations mode.

.. change::
:tags: usecase, postgresql
:tickets: 10904

Support the ``USING <method>`` option for PostgreSQL ``CREATE TABLE`` to
specify the access method to use to store the contents for the new table.
Pull request courtesy Edgar Ramírez-Mondragón.

.. seealso::

:ref:`postgresql_table_options`

.. change::
:tags: bug, examples
:tickets: 10920

Fixed regression in history_meta example where the use of
:meth:`_schema.MetaData.to_metadata` to make a copy of the history table
would also copy indexes (which is a good thing), but causing naming
conflicts indexes regardless of naming scheme used for those indexes. A
"_history" suffix is now added to these indexes in the same way as is
achieved for the table name.


.. change::
:tags: bug, orm
:tickets: 10967

Fixed issue where using :meth:`_orm.Session.delete` along with the
:paramref:`_orm.Mapper.version_id_col` feature would fail to use the
correct version identifier in the case that an additional UPDATE were
emitted against the target object as a result of the use of
:paramref:`_orm.relationship.post_update` on the object. The issue is
similar to :ticket:`10800` just fixed in version 2.0.25 for the case of
updates alone.

.. change::
:tags: bug, orm
:tickets: 10990

Fixed issue where an assertion within the implementation for
:func:`_orm.with_expression` would raise if a SQL expression that was not
cacheable were used; this was a 2.0 regression since 1.4.

.. change::
:tags: postgresql, usecase
:tickets: 9736

Correctly type PostgreSQL RANGE and MULTIRANGE types as ``Range[T]``
and ``Sequence[Range[T]]``.
Introduced utility sequence :class:`_postgresql.MultiRange` to allow better
interoperability of MULTIRANGE types.

.. change::
:tags: postgresql, usecase

Differentiate between INT4 and INT8 ranges and multi-ranges types when
inferring the database type from a :class:`_postgresql.Range` or
:class:`_postgresql.MultiRange` instance, preferring INT4 if the values
fit into it.

.. change::
:tags: bug, typing

Fixed the type signature for the :meth:`.PoolEvents.checkin` event to
indicate that the given :class:`.DBAPIConnection` argument may be ``None``
in the case where the connection has been invalidated.

.. change::
:tags: bug, examples

Fixed the performance example scripts in examples/performance to mostly
work with the Oracle database, by adding the :class:`.Identity` construct
to all the tables and allowing primary generation to occur on this backend.
A few of the "raw DBAPI" cases still are not compatible with Oracle.


.. change::
:tags: bug, mssql

Fixed an issue regarding the use of the :class:`.Uuid` datatype with the
:paramref:`.Uuid.as_uuid` parameter set to False, when using the pymssql
dialect. ORM-optimized INSERT statements (e.g. the "insertmanyvalues"
feature) would not correctly align primary key UUID values for bulk INSERT
statements, resulting in errors. Similar issues were fixed for the
PostgreSQL drivers as well.


.. change::
:tags: bug, postgresql

Fixed an issue regarding the use of the :class:`.Uuid` datatype with the
:paramref:`.Uuid.as_uuid` parameter set to False, when using PostgreSQL
dialects. ORM-optimized INSERT statements (e.g. the "insertmanyvalues"
feature) would not correctly align primary key UUID values for bulk INSERT
statements, resulting in errors. Similar issues were fixed for the
pymssql driver as well.

.. changelog::

:released: January 2, 2024

.. change::
:tags: oracle, asyncio
:tickets: 10679

Added support for :ref:`oracledb` in asyncio mode, using the newly released
version of the ``oracledb`` DBAPI that includes asyncio support. For the
2.0 series, this is a preview release, where the current implementation
does not yet have include support for
:meth:`_asyncio.AsyncConnection.stream`. Improved support is planned for
the 2.1 release of SQLAlchemy.

.. change::
:tags: bug, orm
:tickets: 10800

Fixed issue where when making use of the
:paramref:`_orm.relationship.post_update` feature at the same time as using
a mapper version_id_col could lead to a situation where the second UPDATE
statement emitted by the post-update feature would fail to make use of the
correct version identifier, assuming an UPDATE was already emitted in that
flush which had already bumped the version counter.

.. change::
:tags: bug, typing
:tickets: 10801, 10818

Fixed regressions caused by typing added to the ``sqlalchemy.sql.functions``
module in version 2.0.24, as part of :ticket:`6810`:

* Further enhancements to pep-484 typing to allow SQL functions from
:attr:`_sql.func` derived elements to work more effectively with ORM-mapped
attributes (:ticket:`10801`)

* Fixed the argument types passed to functions so that literal expressions
like strings and ints are again interpreted correctly (:ticket:`10818`)


.. change::
:tags: usecase, orm
:tickets: 10807

Added preliminary support for Python 3.12 pep-695 type alias structures,
when resolving custom type maps for ORM Annotated Declarative mappings.


.. change::
:tags: bug, orm
:tickets: 10815

Fixed issue where ORM Annotated Declarative would mis-interpret the left
hand side of a relationship without any collection specified as
uselist=True if the left type were given as a class and not a string,
without using future-style annotations.

.. change::
:tags: bug, sql
:tickets: 10817

Improved compilation of :func:`_sql.any_` / :func:`_sql.all_` in the
context of a negation of boolean comparison, will now render ``NOT (expr)``
rather than reversing the equality operator to not equals, allowing
finer-grained control of negations for these non-typical operators.

.. changelog::

:released: December 28, 2023

.. change::
:tags: bug, orm
:tickets: 10597

Fixed issue where use of :func:`_orm.foreign` annotation on a
non-initialized :func:`_orm.mapped_column` construct would produce an
expression without a type, which was then not updated at initialization
time of the actual column, leading to issues such as relationships not
determining ``use_get`` appropriately.


.. change::
:tags: bug, schema
:tickets: 10654

Fixed issue where error reporting for unexpected schema item when creating
objects like :class:`_schema.Table` would incorrectly handle an argument
that was itself passed as a tuple, leading to a formatting error. The
error message has been modernized to use f-strings.

.. change::
:tags: bug, engine
:tickets: 10662

Fixed URL-encoding of the username and password components of
:class:`.engine.URL` objects when converting them to string using the
:meth:`_engine.URL.render_as_string` method, by using Python standard
library ``urllib.parse.quote`` while allowing for plus signs and spaces to
remain unchanged as supported by SQLAlchemy's non-standard URL parsing,
rather than the legacy home-grown routine from many years ago. Pull request
courtesy of Xavier NUNN.

.. change::
:tags: bug, orm
:tickets: 10668

Improved the error message produced when the unit of work process sets the
value of a primary key column to NULL due to a related object with a
dependency rule on that column being deleted, to include not just the
destination object and column name but also the source column from which
the NULL value is originating. Pull request courtesy Jan Vollmer.

.. change::
:tags: bug, postgresql
:tickets: 10717

Adjusted the asyncpg dialect such that when the ``terminate()`` method is
used to discard an invalidated connection, the dialect will first attempt
to gracefully close the connection using ``.close()`` with a timeout, if
the operation is proceeding within an async event loop context only. This
allows the asyncpg driver to attend to finalizing a ``TimeoutError``
including being able to close a long-running query server side, which
otherwise can keep running after the program has exited.

.. change::
:tags: bug, orm
:tickets: 10732

Modified the ``__init_subclass__()`` method used by
:class:`_orm.MappedAsDataclass`, :class:`_orm.DeclarativeBase` and
:class:`_orm.DeclarativeBaseNoMeta` to accept arbitrary ``**kw`` and to
propagate them to the ``super()`` call, allowing greater flexibility in
arranging custom superclasses and mixins which make use of
``__init_subclass__()`` keyword arguments. Pull request courtesy Michael
Oliver.


.. change::
:tags: bug, tests
:tickets: 10747

Improvements to the test suite to further harden its ability to run
when Python ``greenlet`` is not installed. There is now a tox
target that includes the token "nogreenlet" that will run the suite
with greenlet not installed (note that it still temporarily installs
greenlet as part of the tox config, however).

.. change::
:tags: bug, sql
:tickets: 10753

Fixed issue in stringify for SQL elements, where a specific dialect is not
passed, where a dialect-specific element such as the PostgreSQL "on
conflict do update" construct is encountered and then fails to provide for
a stringify dialect with the appropriate state to render the construct,
leading to internal errors.

.. change::
:tags: bug, sql

Fixed issue where stringifying or compiling a :class:`.CTE` that was
against a DML construct such as an :func:`_sql.insert` construct would fail
to stringify, due to a mis-detection that the statement overall is an
INSERT, leading to internal errors.

.. change::
:tags: bug, orm
:tickets: 10776

Ensured the use case of :class:`.Bundle` objects used in the
``returning()`` portion of ORM-enabled INSERT, UPDATE and DELETE statements
is tested and works fully. This was never explicitly implemented or
tested previously and did not work correctly in the 1.4 series; in the 2.0
series, ORM UPDATE/DELETE with WHERE criteria was missing an implementation
method preventing :class:`.Bundle` objects from working.

.. change::
:tags: bug, orm
:tickets: 10784

Fixed 2.0 regression in :class:`.MutableList` where a routine that detects
sequences would not correctly filter out string or bytes instances, making
it impossible to assign a string value to a specific index (while
non-sequence values would work fine).

.. change::
:tags: change, asyncio

The ``async_fallback`` dialect argument is now deprecated, and will be
removed in SQLAlchemy 2.1. This flag has not been used for SQLAlchemy's
test suite for some time. asyncio dialects can still run in a synchronous
style by running code within a greenlet using :func:`_util.greenlet_spawn`.

.. change::
:tags: bug, typing
:tickets: 6810

Completed pep-484 typing for the ``sqlalchemy.sql.functions`` module.
:func:`_sql.select` constructs made against ``func`` elements should now
have filled-in return types.

.. changelog::

:released: November 2, 2023

.. change::
:tags: bug, oracle
:tickets: 10509

Fixed issue in :class:`.Interval` datatype where the Oracle implementation
was not being used for DDL generation, leading to the ``day_precision`` and
``second_precision`` parameters to be ignored, despite being supported by
this dialect. Pull request courtesy Indivar.

.. change::
:tags: bug, orm
:tickets: 10516

Fixed issue where the ``__allow_unmapped__`` directive failed to allow for
legacy :class:`.Column` / :func:`.deferred` mappings that nonetheless had
annotations such as ``Any`` or a specific type without ``Mapped[]`` as
their type, without errors related to locating the attribute name.

.. change::
:tags: bug, mariadb
:tickets: 10056

Adjusted the MySQL / MariaDB dialects to default a generated column to NULL
when using MariaDB, if :paramref:`_schema.Column.nullable` was not
specified with an explicit ``True`` or ``False`` value, as MariaDB does not
support the "NOT NULL" phrase with a generated column. Pull request
courtesy Indivar.


.. change::
:tags: bug, mariadb, regression
:tickets: 10505

Established a workaround for what seems to be an intrinsic issue across
MySQL/MariaDB drivers where a RETURNING result for DELETE DML which returns
no rows using SQLAlchemy's "empty IN" criteria fails to provide a
cursor.description, which then yields result that returns no rows,
leading to regressions for the ORM that in the 2.0 series uses RETURNING
for bulk DELETE statements for the "synchronize session" feature. To
resolve, when the specific case of "no description when RETURNING was
given" is detected, an "empty result" with a correct cursor description is
generated and used in place of the non-working cursor.

.. change::
:tags: bug, orm
:tickets: 10570

Fixed caching bug where using the :func:`_orm.with_expression` construct in
conjunction with loader options :func:`_orm.selectinload`,
:func:`_orm.lazyload` would fail to substitute bound parameter values
correctly on subsequent caching runs.

.. change::
:tags: usecase, mssql
:tickets: 6521

Added support for the ``aioodbc`` driver implemented for SQL Server,
which builds on top of the pyodbc and general aio* dialect architecture.

.. seealso::

:ref:`mssql_aioodbc` - in the SQL Server dialect documentation.



.. change::
:tags: bug, sql
:tickets: 10535

Added compiler-level None/NULL handling for the "literal processors" of all
datatypes that include literal processing, that is, where a value is
rendered inline within a SQL statement rather than as a bound parameter,
for all those types that do not feature explicit "null value" handling.
Previously this behavior was undefined and inconsistent.

.. change::
:tags: usecase, orm
:tickets: 10575

Implemented the :paramref:`_orm.Session.bulk_insert_mappings.render_nulls`
parameter for new style bulk ORM inserts, allowing ``render_nulls=True`` as
an execution option. This allows for bulk ORM inserts with a mixture of
``None`` values in the parameter dictionaries to use a single batch of rows
for a given set of dicationary keys, rather than breaking up into batches
that omit the NULL columns from each INSERT.

.. seealso::

:ref:`orm_queryguide_insert_null_params`

.. change::
:tags: bug, postgresql
:tickets: 10479

Fixed 2.0 regression caused by :ticket:`7744` where chains of expressions
involving PostgreSQL JSON operators combined with other operators such as
string concatenation would lose correct parenthesization, due to an
implementation detail specific to the PostgreSQL dialect.

.. change::
:tags: bug, postgresql
:tickets: 10532

Fixed SQL handling for "insertmanyvalues" when using the
:class:`.postgresql.BIT` datatype with the asyncpg backend. The
:class:`.postgresql.BIT` on asyncpg apparently requires the use of an
asyncpg-specific ``BitString`` type which is currently exposed when using
this DBAPI, making it incompatible with other PostgreSQL DBAPIs that all
work with plain bitstrings here. A future fix in version 2.1 will
normalize this datatype across all PG backends. Pull request courtesy
Sören Oldag.


.. change::
:tags: usecase, sql
:tickets: 9737

Implemented "literal value processing" for the :class:`.Interval` datatype
for both the PostgreSQL and Oracle dialects, allowing literal rendering of
interval values. Pull request courtesy Indivar Mishra.

.. change::
:tags: bug, oracle
:tickets: 10470

Fixed issue where the cx_Oracle dialect claimed to support a lower
cx_Oracle version (7.x) than was actually supported in practice within the
2.0 series of SQLAlchemy. The dialect imports symbols that are only in
cx_Oracle 8 or higher, so runtime dialect checks as well as setup.cfg
requirements have been updated to reflect this compatibility.

.. change::
:tags: sql

Removed unused placeholder method :meth:`.TypeEngine.compare_against_backend`
This method was used by very old versions of Alembic.
See https://github.com/sqlalchemy/alembic/issues/1293 for details.

.. change::
:tags: bug, orm
:tickets: 10472

Fixed bug in ORM annotated declarative where using a ``ClassVar`` that
nonetheless referred in some way to an ORM mapped class name would fail to
be interpreted as a ``ClassVar`` that's not mapped.

.. change::
:tags: bug, asyncio
:tickets: 10421

Fixed bug with method :meth:`_asyncio.AsyncSession.close_all`
that was not working correctly.
Also added function :func:`_asyncio.close_all_sessions` that's
the equivalent of :func:`_orm.close_all_sessions`.
Pull request courtesy of Bryan不可思议.

.. changelog::