Sqlalchemy

:released: February 1, 2021

.. change::
:tags: bug, ext
:tickets: 5836

Fixed issue where the stringification that is sometimes called when
attempting to generate the "key" for the ``.c`` collection on a selectable
would fail if the column were an unlabeled custom SQL construct using the
``sqlalchemy.ext.compiler`` extension, and did not provide a default
compilation form; while this seems like an unusual case, it can get invoked
for some ORM scenarios such as when the expression is used in an "order by"
in combination with joined eager loading. The issue is that the lack of a
default compiler function was raising :class:`.CompileError` and not
:class:`.UnsupportedCompilationError`.

.. change::
:tags: bug, postgresql
:tickets: 5645

For SQLAlchemy 1.3 only, setup.py pins pg8000 to a version lower than
1.16.6. Version 1.16.6 and above is supported by SQLAlchemy 1.4. Pull
request courtesy Giuseppe Lumia.

.. change::
:tags: bug, postgresql
:tickets: 5850

Fixed issue where using :meth:`_schema.Table.to_metadata` (called
:meth:`_schema.Table.tometadata` in 1.3) in conjunction with a PostgreSQL
:class:`_postgresql.ExcludeConstraint` that made use of ad-hoc column
expressions would fail to copy correctly.

.. change::
:tags: bug, sql
:tickets: 5816

Fixed bug where making use of the :meth:`.TypeEngine.with_variant` method
on a :class:`.TypeDecorator` type would fail to take into account the
dialect-specific mappings in use, due to a rule in :class:`.TypeDecorator`
that was instead attempting to check for chains of :class:`.TypeDecorator`
instances.


.. change::
:tags: bug, mysql, reflection
:tickets: 5860

Fixed bug where MySQL server default reflection would fail for numeric
values with a negation symbol present.


.. change::
:tags: bug, oracle
:tickets: 5813

Fixed regression in Oracle dialect introduced by :ticket:`4894` in
SQLAlchemy 1.3.11 where use of a SQL expression in RETURNING for an UPDATE
would fail to compile, due to a check for "server_default" when an
arbitrary SQL expression is not a column.


.. change::
:tags: usecase, mysql
:tickets: 5808

Casting to ``FLOAT`` is now supported in MySQL >= (8, 0, 17) and
MariaDb >= (10, 4, 5).

.. change::
:tags: bug, mysql
:tickets: 5898

Fixed long-lived bug in MySQL dialect where the maximum identifier length
of 255 was too long for names of all types of constraints, not just
indexes, all of which have a size limit of 64. As metadata naming
conventions can create too-long names in this area, apply the limit to the
identifier generator within the DDL compiler.

.. change::
:tags: bug, oracle
:tickets: 5812

Fixed bug in Oracle dialect where retrieving a CLOB/BLOB column via
:meth:`_dml.Insert.returning` would fail as the LOB value would need to be
read when returned; additionally, repaired support for retrieval of Unicode
values via RETURNING under Python 2.

.. change::
:tags: bug, mysql
:tickets: 5821

Fixed deprecation warnings that arose as a result of the release of PyMySQL
1.0, including deprecation warnings for the "db" and "passwd" parameters
now replaced with "database" and "password".


.. change::
:tags: bug, mysql
:tickets: 5800

Fixed regression from SQLAlchemy 1.3.20 caused by the fix for
:ticket:`5462` which adds double-parenthesis for MySQL functional
expressions in indexes, as is required by the backend, this inadvertently
extended to include arbitrary :func:`_sql.text` expressions as well as
Alembic's internal textual component, which are required by Alembic for
arbitrary index expressions which don't imply double parenthesis. The
check has been narrowed to include only binary/ unary/functional
expressions directly.

.. changelog::

:released: December 18, 2020

.. change::
:tags: bug, oracle
:tickets: 5784
:versions: 1.4.0b2

Fixed regression which occurred due to :ticket:`5755` which implemented
isolation level support for Oracle. It has been reported that many Oracle
accounts don't actually have permission to query the ``v$transaction``
view so this feature has been altered to gracefully fallback when it fails
upon database connect, where the dialect will assume "READ COMMITTED" is
the default isolation level as was the case prior to SQLAlchemy 1.3.21.
However, explicit use of the :meth:`_engine.Connection.get_isolation_level`
method must now necessarily raise an exception, as Oracle databases with
this restriction explicitly disallow the user from reading the current
isolation level.

.. changelog::

:released: December 17, 2020

.. change::
:tags: bug, orm
:tickets: 5774
:versions: 1.4.0b2

Added a comprehensive check and an informative error message for the case
where a mapped class, or a string mapped class name, is passed to
:paramref:`_orm.relationship.secondary`. This is an extremely common error
which warrants a clear message.

Additionally, added a new rule to the class registry resolution such that
with regards to the :paramref:`_orm.relationship.secondary` parameter, if a
mapped class and its table are of the identical string name, the
:class:`.Table` will be favored when resolving this parameter. In all
other cases, the class continues to be favored if a class and table
share the identical name.

.. change::
:tags: sqlite, usecase
:tickets: 5685

Added ``sqlite_with_rowid=False`` dialect keyword to enable creating
tables as ``CREATE TABLE … WITHOUT ROWID``. Patch courtesy Sean Anderson.

.. change::
:tags: bug, sql
:tickets: 5691

A warning is emitted if a returning() method such as
:meth:`_sql.Insert.returning` is called multiple times, as this does not
yet support additive operation. Version 1.4 will support additive
operation for this. Additionally, any combination of the
:meth:`_sql.Insert.returning` and :meth:`_sql.ValuesBase.return_defaults`
methods now raises an error as these methods are mutually exclusive;
previously the operation would fail silently.


.. change::
:tags: bug, mssql
:tickets: 5751

Fixed bug where a CREATE INDEX statement was rendered incorrectly when
both ``mssql-include`` and ``mssql_where`` were specified. Pull request
courtesy Adiorz.

.. change::
:tags: bug, postgresql, mysql
:tickets: 5729
:versions: 1.4.0b2

Fixed regression introduced in 1.3.2 for the PostgreSQL dialect, also
copied out to the MySQL dialect's feature in 1.3.18, where usage of a non
:class:`_schema.Table` construct such as :func:`_sql.text` as the argument
to :paramref:`_sql.Select.with_for_update.of` would fail to be accommodated
correctly within the PostgreSQL or MySQL compilers.


.. change::
:tags: bug, mssql
:tickets: 5646

Added SQL Server code "01000" to the list of disconnect codes.


.. change::
:tags: usecase, postgresql
:tickets: 5604
:versions: 1.4.0b2

Added new parameter :paramref:`_postgresql.ExcludeConstraint.ops` to the
:class:`_postgresql.ExcludeConstraint` object, to support operator class
specification with this constraint. Pull request courtesy Alon Menczer.

.. change::
:tags: bug, mysql, reflection
:tickets: 5744
:versions: 1.4.0b2

Fixed issue where reflecting a server default on MariaDB only that
contained a decimal point in the value would fail to be reflected
correctly, leading towards a reflected table that lacked any server
default.


.. change::
:tags: bug, orm
:tickets: 5664

Fixed bug in :meth:`_query.Query.update` where objects in the
:class:`_ormsession.Session` that were already expired would be
unnecessarily SELECTed individually when they were refreshed by the
"evaluate"synchronize strategy.

.. change::
:tags: usecase, oracle
:tickets: 5755

Implemented support for the SERIALIZABLE isolation level for Oracle
databases, as well as a real implementation for
:meth:`_engine.Connection.get_isolation_level`.

.. seealso::

:ref:`oracle_isolation_level`

.. change::
:tags: mysql, sql
:tickets: 5696

Added missing keywords to the ``RESERVED_WORDS`` list for the MySQL
dialect: ``action``, ``level``, ``mode``, ``status``, ``text``, ``time``.
Pull request courtesy Oscar Batori.

.. change::
:tags: bug, orm
:tickets: 5737
:versions: 1.4.0b2

Fixed bug involving the ``restore_load_context`` option of ORM events such
as :meth:`_ormevent.InstanceEvents.load` such that the flag would not be
carried along to subclasses which were mapped after the event handler were
first established.



.. change::
:tags: bug, sql
:tickets: 5656

Fixed structural compiler issue where some constructs such as MySQL /
PostgreSQL "on conflict / on duplicate key" would rely upon the state of
the :class:`_sql.Compiler` object being fixed against their statement as
the top level statement, which would fail in cases where those statements
are branched from a different context, such as a DDL construct linked to a
SQL statement.


.. change::
:tags: mssql, sqlite, reflection
:tickets: 5661

Fixed issue with composite primary key columns not being reported
in the correct order. Patch courtesy fulpm.

.. changelog::

:released: October 12, 2020

.. change::
:tags: bug, orm
:tickets: 4428

An :class:`.ArgumentError` with more detail is now raised if the target
parameter for :meth:`_query.Query.join` is set to an unmapped object.
Prior to this change a less detailed ``AttributeError`` was raised.
Pull request courtesy Ramon Williams.

.. change::
:tags: bug, mysql
:tickets: 5568

The "skip_locked" keyword used with ``with_for_update()`` will emit a
warning when used on MariaDB backends, and will then be ignored. This is
a deprecated behavior that will raise in SQLAlchemy 1.4, as an application
that requests "skip locked" is looking for a non-blocking operation which
is not available on those backends.



.. change::
:tags: bug, engine
:tickets: 5599

Fixed issue where a non-string object sent to
:class:`_exc.SQLAlchemyError` or a subclass, as occurs with some third
party dialects, would fail to stringify correctly. Pull request
courtesy Andrzej Bartosiński.

.. change::
:tags: bug, sql
:tickets: 5644

Fixed issue where the ``pickle.dumps()`` operation against
:class:`_expression.Over` construct would produce a recursion overflow.

.. change::
:tags: postgresql, usecase
:tickets: 4392

The psycopg2 dialect now support PostgreSQL multiple host connections, by
passing host/port combinations to the query string. Pull request courtesy
Ramon Williams.

.. seealso::

:ref:`psycopg2_multi_host`

.. change::
:tags: bug, mysql
:tickets: 5617

Fixed bug where an UPDATE statement against a JOIN using MySQL multi-table
format would fail to include the table prefix for the target table if the
statement had no WHERE clause, as only the WHERE clause were scanned to
detect a "multi table update" at that particular point. The target
is now also scanned if it's a JOIN to get the leftmost table as the
primary table and the additional entries as additional FROM entries.


.. change::
:tags: bug, postgresql
:tickets: 5518

Adjusted the :meth:`_types.ARRAY.Comparator.any` and
:meth:`_types.ARRAY.Comparator.all` methods to implement a straight "NOT"
operation for negation, rather than negating the comparison operator.

.. change::
:tags: bug, pool
:tickets: 5582

Fixed issue where the following pool parameters were not being propagated
to the new pool created when :meth:`_engine.Engine.dispose` were called:
``pre_ping``, ``use_lifo``. Additionally the ``recycle`` and
``reset_on_return`` parameter is now propagated for the
:class:`_engine.AssertionPool` class.

.. change::
:tags: bug, ext, associationproxy
:tickets: 5541, 5542

An informative error is now raised when attempting to use an association
proxy element as a plain column expression to be SELECTed from or used in a
SQL function; this use case is not currently supported.


.. change::
:tags: bug, sql
:tickets: 5618

Fixed bug where an error was not raised in the case where a
:func:`_sql.column` were added to more than one :func:`_sql.table` at a
time. This raised correctly for the :class:`_schema.Column` and
:class:`_schema.Table` objects. An :class:`_exc.ArgumentError` is now
raised when this occurs.

.. change::
:tags: bug, orm
:tickets: 4589

Fixed issue where using a loader option against a string attribute name
that is not actually a mapped attribute, such as a plain Python descriptor,
would raise an uninformative AttributeError; a descriptive error is now
raised.



.. change::
:tags: mysql, usecase
:tickets: 5462

Adjusted the MySQL dialect to correctly parenthesize functional index
expressions as accepted by MySQL 8. Pull request courtesy Ramon Williams.

.. change::
:tags: bug, engine
:tickets: 5632

Repaired a function-level import that was not using SQLAlchemy's standard
late-import system within the sqlalchemy.exc module.


.. change::
:tags: change, mysql
:tickets: 5539

Add new MySQL reserved words: ``cube``, ``lateral`` added in MySQL 8.0.1
and 8.0.14, respectively; this indicates that these terms will be quoted if
used as table or column identifier names.

.. change::
:tags: bug, mssql
:tickets: 5592

Fixed issue where a SQLAlchemy connection URI for Azure DW with
``authentication=ActiveDirectoryIntegrated`` (and no username+password)
was not constructing the ODBC connection string in a way that was
acceptable to the Azure DW instance.

.. change::
:tags: bug, postgresql
:tickets: 5520

Fixed issue where the :class:`_postgresql.ENUM` type would not consult the
schema translate map when emitting a CREATE TYPE or DROP TYPE during the
test to see if the type exists or not. Additionally, repaired an issue
where if the same enum were encountered multiple times in a single DDL
sequence, the "check" query would run repeatedly rather than relying upon a
cached value.


.. change::
:tags: bug, tests
:tickets: 5635

Fixed incompatibilities in the test suite when running against Pytest 6.x.


.. changelog::

:released: August 17, 2020

.. change::
:tags: usecase, py3k
:tickets: 5357

Added a ``**kw`` argument to the :meth:`.DeclarativeMeta.__init__` method.
This allows a class to support the :pep:`487` metaclass hook
``__init_subclass__``. Pull request courtesy Ewen Gillies.


.. change::
:tags: bug, sql
:tickets: 5470

Repaired an issue where the "ORDER BY" clause rendering a label name rather
than a complete expression, which is particularly important for SQL Server,
would fail to occur if the expression were enclosed in a parenthesized
grouping in some cases. This case has been added to test support. The
change additionally adjusts the "automatically add ORDER BY columns when
DISTINCT is present" behavior of ORM query, deprecated in 1.4, to more
accurately detect column expressions that are already present.

.. change::
:tags: usecase, mysql
:tickets: 5481

The MySQL dialect will render FROM DUAL for a SELECT statement that has no
FROM clause but has a WHERE clause. This allows things like "SELECT 1 WHERE
EXISTS (subquery)" kinds of queries to be used as well as other use cases.


.. change::
:tags: bug, mssql, sql
:tickets: 5467

Fixed bug where the mssql dialect incorrectly escaped object names that
contained ']' character(s).

.. change::
:tags: bug, reflection, sqlite, mssql
:tickets: 5456

Applied a sweep through all included dialects to ensure names that contain
single or double quotes are properly escaped when querying system tables,
for all :class:`.Inspector` methods that accept object names as an argument
(e.g. table names, view names, etc). SQLite and MSSQL contained two
quoting issues that were repaired.

.. change::
:tags: bug, mysql
:tickets: 5411

Fixed an issue where CREATE TABLE statements were not specifying the
COLLATE keyword correctly.

.. change::
:tags: bug, datatypes, sql
:tickets: 4733

The ``LookupError`` message will now provide the user with up to four
possible values that a column is constrained to via the :class:`.Enum`.
Values longer than 11 characters will be truncated and replaced with
ellipses. Pull request courtesy Ramon Williams.

.. change::
:tags: bug, postgresql
:tickets: 5476

Fixed issue where the return type for the various RANGE comparison
operators would itself be the same RANGE type rather than BOOLEAN, which
would cause an undesirable result in the case that a
:class:`.TypeDecorator` that defined result-processing behavior were in
use. Pull request courtesy Jim Bosch.



.. change::
:tags: bug, mysql
:tickets: 5493

Added MariaDB code 1927 to the list of "disconnect" codes, as recent
MariaDB versions apparently use this code when the database server was
stopped.

.. change::
:tags: usecase, declarative, orm
:tickets: 5513

The name of the virtual column used when using the
:class:`_declarative.AbstractConcreteBase` and
:class:`_declarative.ConcreteBase` classes can now be customized, to allow
for models that have a column that is actually named ``type``. Pull
request courtesy Jesse-Bakker.

.. change::
:tags: usecase, orm
:tickets: 5494

Adjusted the workings of the :meth:`_orm.Mapper.all_orm_descriptors`
accessor to represent the attributes in the order that they are located in
a deterministic way, assuming the use of Python 3.6 or higher which
maintains the sorting order of class attributes based on how they were
declared. This sorting is not guaranteed to match the declared order of
attributes in all cases however; see the method documentation for the exact
scheme.



.. change::
:tags: bug, sql
:tickets: 5500

Fixed issue where the
:paramref:`_engine.Connection.execution_options.schema_translate_map`
feature would not take effect when the :meth:`_schema.Sequence.next_value`
function function for a :class:`_schema.Sequence` were used in the
:paramref:`_schema.Column.server_default` parameter and the create table
DDL were emitted.

.. changelog::

:released: June 25, 2020

.. change::
:tags: bug, sqlite
:tickets: 5395

Added "exists" to the list of reserved words for SQLite so that this word
will be quoted when used as a label or column name. Pull request courtesy
Thodoris Sotiropoulos.

.. change::
:tags: bug, mssql
:tickets: 5366, 5364

Refined the logic used by the SQL Server dialect to interpret multi-part
schema names that contain many dots, to not actually lose any dots if the
name does not have bracking or quoting used, and additionally to support a
"dbname" token that has many parts including that it may have multiple,
independently-bracketed sections.



.. change::
:tags: bug, mssql, pyodbc
:tickets: 5346

Fixed an issue in the pyodbc connector such that a warning about pyodbc
"drivername" would be emitted when using a totally empty URL. Empty URLs
are normal when producing a non-connected dialect object or when using the
"creator" argument to create_engine(). The warning now only emits if the
driver name is missing but other parameters are still present.

.. change::
:tags: bug, mssql
:tickets: 5373

Fixed issue with assembling the ODBC connection string for the pyodbc
DBAPI. Tokens containing semicolons and/or braces "{}" were not being
correctly escaped, causing the ODBC driver to misinterpret the
connection string attributes.

.. change::
:tags: usecase, orm
:tickets: 5326

Improve error message when using :meth:`_query.Query.filter_by` in
a query where the first entity is not a mapped class.

.. change::
:tags: sql, schema
:tickets: 5324

Introduce :class:`.IdentityOptions` to store common parameters for
sequences and identity columns.

.. change::
:tags: usecase, sql
:tickets: 5309

Added a ".schema" parameter to the :func:`_expression.table` construct,
allowing ad-hoc table expressions to also include a schema name.
Pull request courtesy Dylan Modesitt.

.. change::
:tags: bug, mssql
:tickets: 5339

Fixed issue where ``datetime.time`` parameters were being converted to
``datetime.datetime``, making them incompatible with comparisons like
``>=`` against an actual :class:`_mssql.TIME` column.

.. change::
:tags: bug, mssql
:tickets: 5359

Fixed an issue where the ``is_disconnect`` function in the SQL Server
pyodbc dialect was incorrectly reporting the disconnect state when the
exception message had a substring that matched a SQL Server ODBC error
code.

.. change::
:tags: bug, engine
:tickets: 5326

Further refinements to the fixes to the "reset" agent fixed in
:ticket:`5326`, which now emits a warning when it is not being correctly
invoked and corrects for the behavior. Additional scenarios have been
identified and fixed where this warning was being emitted.


.. change::
:tags: usecase, sqlite
:tickets: 5297

SQLite 3.31 added support for computed column. This change
enables their support in SQLAlchemy when targeting SQLite.

.. change::
:tags: bug, schema
:tickets: 5276

Fixed issue where ``dialect_options`` were omitted when a
database object (e.g., :class:`.Table`) was copied using
:func:`.tometadata`.

.. change::
:tags: bug, sql
:tickets: 5344

Correctly apply self_group in type_coerce element.

The type coerce element did not correctly apply grouping rules when using
in an expression

.. change::
:tags: bug, oracle, reflection
:tickets: 5421

Fixed bug in Oracle dialect where indexes that contain the full set of
primary key columns would be mistaken as the primary key index itself,
which is omitted, even if there were multiples. The check has been refined
to compare the name of the primary key constraint against the index name
itself, rather than trying to guess based on the columns present in the
index.

.. change::
:tags: change, sql, sybase
:tickets: 5294

Added ``.offset`` support to sybase dialect.
Pull request courtesy Alan D. Snow.

.. change::
:tags: bug, engine
:tickets: 5341

Fixed issue in :class:`.URL` object where stringifying the object
would not URL encode special characters, preventing the URL from being
re-consumable as a real URL. Pull request courtesy Miguel Grinberg.

.. change::
:tags: usecase, mysql
:tickets: 4860

Implemented row-level locking support for mysql. Pull request courtesy
Quentin Somerville.

.. change::
:tags: change, mssql
:tickets: 5321

Moved the ``supports_sane_rowcount_returning = False`` requirement from
the ``PyODBCConnector`` level to the ``MSDialect_pyodbc`` since pyodbc
does work properly in some circumstances.

.. change::
:tags: change, examples

Added new option ``--raw`` to the examples.performance suite
which will dump the raw profile test for consumption by any
number of profiling visualizer tools. Removed the "runsnake"
option as runsnake is very hard to build at this point;

.. change::
:tags: bug, sql
:tickets: 5353

Added :meth:`.Select.with_hint` output to the generic SQL string that is
produced when calling ``str()`` on a statement. Previously, this clause
would be omitted under the assumption that it was dialect specific.
The hint text is presented within brackets to indicate the rendering
of such hints varies among backends.


.. change::
:tags: usecase, orm
:tickets: 5198

Added a new parameter :paramref:`_orm.query_expression.default_expr` to the
:func:`_orm.query_expression` construct, which will be appled to queries
automatically if the :func:`_orm.with_expression` option is not used. Pull
request courtesy Haoyu Sun.

.. changelog::