Spinta

=======

Backwards incompatible:
- conversion of XSD schemas to DSA manifests in an improved way. (`842`_)
- support for language tag for properties. (`582`_)

.. _842: https://github.com/atviriduomenys/spinta/issues/842
.. _582: https://github.com/atviriduomenys/spinta/issues/582

===================

Bug fixes:

- Fixed `SqliteQueryBuilder` importing wrong `Sqlite` class (`1174`_).

.. _1174: https://github.com/atviriduomenys/spinta/issues/1174

===================

Backwards incompatible:

- `sql` backend no longer tries to automatically change it's query functions based on dsn dialect. Now in order to access
specific dialect's functionality, you need to specify it through type (`1127`_).

Currently supported `sql` backend types:
- `sql` - generic default sql type (tries to use dialect indifferent functions).
- `sql/postgresql` - PostgreSQL dialect.
- `sql/mssql` - Microsoft SQL server dialect.
- `sql/mysql` - MySQL dialect.
- `sql/mariadb` - MariaDB dialect.
- `sql/sqlite` - Sqlite dialect.
- `sql/oracle` - Oracle database dialect.

It is recommended to specify dialects in the manifest or config, this will ensure better performance and can unlock
more functionality (in case some dialects support unique functions). Because system no longer tries to automatically
detect the dialect there is a possibility of errors or invalid values if you do not set the correct dialect.

- `Backend` objects now store `result_builder_class` and `query_builder_class` properties, which can be used to initialize
their respective builders. This changes how `QueryBuilders` and `ResultBuilders` are now created. Each `Backend` now has
to specify their builder through `result_builder_type` and `query_builder_type`, which are strings, that map with
corresponding classes in `config.components` (`1127`_).

All `QueryBuilder` classes are stored in `config.components.querybuilders` path.

Currently there are these builders, that can be used:
- '' - Empty default query builder.
- `postgresql` - Internal postgresql query builder.
- `mongo` - Internal mongo query builder.
- `sql`- External default sql query builder.
- `sql/sqlite` - External sqlite dialect query builder.
- `sql/mssql` - External microsoft sql dialect query builder.
- `sql/postgresql` - External postgresql dialect query builder.
- `sql/oracle` - External oracle dialect query builder.
- `sql/mysql` - External mysql dialect query builder.
- `sql/mariadb` - External mariadb dialect query builder.
- `dask` - External Dask dataframe query builder.

All `ResultBuilder` classes are stored in `config.components.resultbuilders` path.

Currently there are these builders, that can be used:
- '' - Empty default result builder.
- `postgresql` - Internal postgresql result builder.
- `sql`- External sql result builder.

- In order to maintain cohesiveness in code and data structure, dask backends have gone through same treatment as `sql`.
Before they worked similar to the new system (users had to manually specify their type), but now to make sure that
naming convention is same with all components `csv`, `json` and `xml` types have been renamed to `dask/csv`, `dask/json`,
`dask/xml`. If you used these backends before, you will now need to add `dask/` prefix to their types (`1127`_).

Because so many datasets use `csv`, `json` and `xml` types, they will not be fully removed, but they will be deprecated
and eventually might be removed, so it's encouraged to change them to `dask` format.


New features:

- Added exposed intermediate table support for external `Sql` backend (`663`_).

.. _663: https://github.com/atviriduomenys/spinta/issues/663

Improvements:

- Added better error messages for scalar to ref migrations (when system cannot determine previous primary keys) (`1123`_).

.. _1123: https://github.com/atviriduomenys/spinta/issues/1123

- `export` command now supports `access` argument, that can filter models and properties
if they are the same or higher level than given `access` (default is `private`, meaning everything is exported) (`1130`_).

.. _1130: https://github.com/atviriduomenys/spinta/issues/1130

- Separated `sql` `backend` dialects to their own separate backends (`1127`_).

- Added `dask/` prefix to `csv`, `xml` and `json` backends (`1127`_).

.. _1127: https://github.com/atviriduomenys/spinta/issues/1127


Bug fix:

- Convertion from scalar to ref (and ref to scalar) now uses `alias` when there is self reference (`1105`_).

.. _1105: https://github.com/atviriduomenys/spinta/issues/1105

- `spyna` when reading string values and escaping characters now properly restores converted `unicode` characters back
to `utf-8` encoding, which will allow the use Lithuanian characters in query (`1139`_).

.. _1139: https://github.com/atviriduomenys/spinta/issues/1139

===================

Backwards incompatible:

- `postgresql` `backend` now no longer ignores `prepare` functions. Meaning if there are properties, which has functions
set in `prepare` column, it can cause errors (if those functions are not supported in `postgresql` `backend`) (`1048`_).

- `InternalSqlManifest` no longer is capable of knowing when to hide `Text` or `C` language (`940`_). That means if you have
`tabular` `manifest` with hidden `Text`, like so:

.. code-block:: text

d | r | b | m | property | type | ref | access | title
example | | | |
| | | |
| | | City | | id | |
| | | | id | integer | | open |
| | | | namelt | string | | open |
| | | | nameen | string | | open |

if you were to convert it to `InternalSqlManifest` and back, you would get this result:

.. code-block:: text

d | r | b | m | property | type | ref | access | title
example | | | |
| | | |
| | | City | | id | |
| | | | id | integer | | open |
| | | | name | text | | open |
| | | | namelt | string | | open |
| | | | nameen | string | | open |

New features:

- Added support for `Object` type with `external` `Sql` `backend` (`973`_).

.. _973: https://github.com/atviriduomenys/spinta/issues/973

- Added 'flip` function, which currently only supports `Geometry` type (flips coordinate axis). This features only works
when reading data, meaning, when writing, you still need to provide coordinates in the right order (`1048`_).

.. _1048: https://github.com/atviriduomenys/spinta/issues/1048

- Added `point` function support to `postgresql` `backend` (`1053`_).

.. _1053: https://github.com/atviriduomenys/spinta/issues/1053

Improvements:

- Client data and `keymap` is now cached. This will reduce amount of file reads with each request (`948`_).

.. _948: https://github.com/atviriduomenys/spinta/issues/948

- `Tabular` `manifest` now supports `Text` type nesting with other complex types (`Object`, `Ref`, etc.) (`940`_).

.. _940: https://github.com/atviriduomenys/spinta/issues/940

===================

Backwards incompatible:

- `SqlAlchemyKeyMap` synchronization no longer uses individual transactions for each synchronization action. Now it
batches the actions under multiple transactions. By default it batches `10000` rows. In order to change that value,
set `sync_transaction_size` in `config` under your `keymaps` configuration (`1011`_).

Like so:

.. code-block:: yaml

keymaps:
default:
type: sqlalchemy
dsn: ...
sync_transaction_size: 20000

- Changed `postgresql` naming convention. This will result in old tables having incorrect constraint and index names.
`spinta migrate` should be able to find most of them (`P153`).

- `AccessLog` no longer stores `scope` field on every request. Instead it will store `token` field (token `JTI` value).
In order to track what scopes token uses, now we log `auth` requests (`/auth/token`), which will store list of scopes.
This change should reduce the spam in logging and reduce log file size.

In order track unique token identifiers, `JTI` field has been added to all new tokens (meaning old tokens, that still
do not have the field, will not be properly logged) (`1003`_).

Improvements:

- `SqlAlchemyKeyMap` now uses batch transactions to synchronize data, which greatly improves performance (`1011`_).

.. _1011: https://github.com/atviriduomenys/spinta/issues/1011

- added enum level support, allowing to indicate a level for enum. (`982`_)

.. _982: https://github.com/atviriduomenys/spinta/issues/982

- Standardized `postgresql` naming convention, now all new constraints and indexes should follow same naming
scheme (`P153`).

- `spinta migrate` now tries to rename constraints and indexes (if the name only changed) instead of dropping them and
adding them with correct name (`P153`).

- `JWT` tokens now also store `JTI` claim (`1003`_).

- `AccessLog` now has `auth` logging (`1003`_).

.. _1003: https://github.com/atviriduomenys/spinta/issues/1003

Bug fix:

- `Postgresql` `summary` now properly handles tables with long names (`P160`).

- Fixed various cases where `migrate` command would not take into account truncated names (`P153`).

===================

Backwards incompatible:

- Keymap synchronization now uses `sync_page_size` config argument to limit amount of data being fetched with a single
request. This will result in more actions being called to remote server. If `keymap` synchronization takes too long
to start the process, reduce `sync_page_size` value. Keep in mind, that lower values reduce performance and increase
server load (`985`_).

- `push` command now has explicit timeouts set for requests.
Previously, there were no timeouts set for requests, which meant that execution time was unlimited.
After the changes the default values are `300` seconds (5min) for `read` and `5` seconds for `connect` timeouts.
The timeout values can be adjusted using `--read-timeout` and `--connect-timeout` push command options (`662`_).

New features:

- Add `-d --datasets` option to migrate command (`935`_).

.. _935: https://github.com/atviriduomenys/spinta/issues/935

- Add `export` cli command, that will export data to specified format (`960`_).

.. _960: https://github.com/atviriduomenys/spinta/issues/960

- Add `keymap sync` command (`666`_).

.. _666: https://github.com/atviriduomenys/spinta/issues/666

- Add `--read-timeout`, `--connect-timeout` options to `spinta push` command (`662`_).

.. _662: https://github.com/atviriduomenys/spinta/issues/662

Improvements:

- Keymap synchronization now uses pagination to fetch data (`985`_).

.. _985: https://github.com/atviriduomenys/spinta/issues/985