Ormar

✨ Features

* Add **Python style** to `filter` and `order_by` with field access instead of dunder separated strings. [51](https://github.com/collerek/ormar/issues/51)
* Accessing a field with attribute access (chain of dot notation) can be used to construct `FilterGroups` (`ormar.and_` and `ormar.or_`)
* Field access overloads set of python operators and provide a set of functions to allow same functionality as with dunder separated param names in `**kwargs`, that means that querying from sample model `Track` related to model `Album` now you have more options:
* exact - exact match to value, sql `column = <VALUE>`
* OLD: `album__name__exact='Malibu'`
* NEW: can be also written as `Track.album.name == 'Malibu`
* iexact - exact match sql `column = <VALUE>` (case insensitive)
* OLD: `album__name__iexact='malibu'`
* NEW: can be also written as `Track.album.name.iexact('malibu')`
* contains - sql `column LIKE '%<VALUE>%'`
* OLD: `album__name__contains='Mal'`
* NEW: can be also written as `Track.album.name % 'Mal')`
* NEW: can be also written as `Track.album.name.contains('Mal')`
* icontains - sql `column LIKE '%<VALUE>%'` (case insensitive)
* OLD: `album__name__icontains='mal'`
* NEW: can be also written as `Track.album.name.icontains('mal')`
* in - sql ` column IN (<VALUE1>, <VALUE2>, ...)`
* OLD: `album__name__in=['Malibu', 'Barclay']`
* NEW: can be also written as `Track.album.name << ['Malibu', 'Barclay']`
* NEW: can be also written as `Track.album.name.in_(['Malibu', 'Barclay'])`
* isnull - sql `column IS NULL` (and sql `column IS NOT NULL`)
* OLD: `album__name__isnull=True` (isnotnull `album__name__isnull=False`)
* NEW: can be also written as `Track.album.name >> None`
* NEW: can be also written as `Track.album.name.isnull(True)`
* NEW: not null can be also written as `Track.album.name.isnull(False)`
* NEW: not null can be also written as `~(Track.album.name >> None)`
* NEW: not null can be also written as `~(Track.album.name.isnull(True))`
* gt - sql `column > <VALUE>` (greater than)
* OLD: `position__gt=3`
* NEW: can be also written as `Track.album.name > 3`
* gte - sql `column >= <VALUE>` (greater or equal than)
* OLD: `position__gte=3`
* NEW: can be also written as `Track.album.name >= 3`
* lt - sql `column < <VALUE>` (lower than)
* OLD: `position__lt=3`
* NEW: can be also written as `Track.album.name < 3`
* lte - sql `column <= <VALUE>` (lower equal than)
* OLD: `position__lte=3`
* NEW: can be also written as `Track.album.name <= 3`
* startswith - sql `column LIKE '<VALUE>%'` (exact start match)
* OLD: `album__name__startswith='Mal'`
* NEW: can be also written as `Track.album.name.startswith('Mal')`
* istartswith - sql `column LIKE '<VALUE>%'` (case insensitive)
* OLD: `album__name__istartswith='mal'`
* NEW: can be also written as `Track.album.name.istartswith('mal')`
* endswith - sql `column LIKE '%<VALUE>'` (exact end match)
* OLD: `album__name__endswith='ibu'`
* NEW: can be also written as `Track.album.name.endswith('ibu')`
* iendswith - sql `column LIKE '%<VALUE>'` (case insensitive)
* OLD: `album__name__iendswith='IBU'`
* NEW: can be also written as `Track.album.name.iendswith('IBU')`
* You can provide `FilterGroups` not only in `filter()` and `exclude()` but also in:
* `get()`
* `get_or_none()`
* `get_or_create()`
* `first()`
* `all()`
* `delete()`
* With `FilterGroups` (`ormar.and_` and `ormar.or_`) you can now use:
* `&` - as `and_` instead of next level of nesting
* `|` - as `or_' instead of next level of nesting
* `~` - as negation of the filter group
* To combine groups of filters into one set of conditions use `&` (sql `AND`) and `|` (sql `OR`)
python
Following queries are equivalent:
sql: ( product.name = 'Test' AND product.rating >= 3.0 )

ormar OPTION 1 - OLD one
Product.objects.filter(name='Test', rating__gte=3.0).get()

ormar OPTION 2 - OLD one
Product.objects.filter(ormar.and_(name='Test', rating__gte=3.0)).get()

ormar OPTION 3 - NEW one (field access)
Product.objects.filter((Product.name == 'Test') & (Product.rating >=3.0)).get()

* Same applies to nested complicated filters
python
Following queries are equivalent:
sql: ( product.name = 'Test' AND product.rating >= 3.0 )
OR (categories.name IN ('Toys', 'Books'))

ormar OPTION 1 - OLD one
Product.objects.filter(ormar.or_(
ormar.and_(name='Test', rating__gte=3.0),
categories__name__in=['Toys', 'Books'])
).get()

ormar OPTION 2 - NEW one (instead of nested or use `|`)
Product.objects.filter(
ormar.and_(name='Test', rating__gte=3.0) |
ormar.and_(categories__name__in=['Toys', 'Books'])
).get()

ormar OPTION 3 - NEW one (field access)
Product.objects.filter(
((Product.name='Test') & (Product.rating >= 3.0)) |
(Product.categories.name << ['Toys', 'Books'])
).get()

* Now you can also use field access to provide OrderActions to `order_by()`
* Order ascending:
* OLD: `Product.objects.order_by("name").all()`
* NEW: `Product.objects.order_by(Product.name.asc()).all()`
* Order descending:
* OLD: `Product.objects.order_by("-name").all()`
* NEW: `Product.objects.order_by(Product.name.desc()).all()`
* You can of course also combine different models and many order_bys:
`Product.objects.order_by([Product.category.name.asc(), Product.name.desc()]).all()`

🐛 Fixes

* Not really a bug but rather inconsistency. Providing a filter with nested model i.e. `album__category__name = 'AA'`
is checking if album and category models are included in `select_related()` and if not it's auto-adding them there.
The same functionality was not working for `FilterGroups` (`and_` and `or_`), now it works (also for python style filters which return `FilterGroups`).

✨ Features

* `ForeignKey` and `ManyToMany` now support `skip_reverse: bool = False` flag [118](https://github.com/collerek/ormar/issues/118).
If you set `skip_reverse` flag internally the field is still registered on the other
side of the relationship so you can:
* `filter` by related models fields from reverse model
* `order_by` by related models fields from reverse model

But you cannot:
* access the related field from reverse model with `related_name`
* even if you `select_related` from reverse side of the model the returned models won't be populated in reversed instance (the join is not prevented so you still can `filter` and `order_by`)
* the relation won't be populated in `dict()` and `json()`
* you cannot pass the nested related objects when populating from `dict()` or `json()` (also through `fastapi`). It will be either ignored or raise error depending on `extra` setting in pydantic `Config`.
* `Model.save_related()` now can save whole data tree in once [148](https://github.com/collerek/ormar/discussions/148)
meaning:
* it knows if it should save main `Model` or related `Model` first to preserve the relation
* it saves main `Model` if
* it's not `saved`,
* has no `pk` value
* or `save_all=True` flag is set

in those cases you don't have to split save into two calls (`save()` and `save_related()`)
* it supports also `ManyToMany` relations
* it supports also optional `Through` model values for m2m relations
* Add possibility to customize `Through` model relation field names.
* By default `Through` model relation names default to related model name in lowercase.
So in example like this:
python
... course declaration omitted
class Student(ormar.Model):
class Meta:
database = database
metadata = metadata

id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
courses = ormar.ManyToMany(Course)

will produce default Through model like follows (example simplified)
class StudentCourse(ormar.Model):
class Meta:
database = database
metadata = metadata
tablename = "students_courses"

id: int = ormar.Integer(primary_key=True)
student = ormar.ForeignKey(Student) default name
course = ormar.ForeignKey(Course) default name

* To customize the names of fields/relation in Through model now you can use new parameters to `ManyToMany`:
* `through_relation_name` - name of the field leading to the model in which `ManyToMany` is declared
* `through_reverse_relation_name` - name of the field leading to the model to which `ManyToMany` leads to

Example:
python
... course declaration omitted
class Student(ormar.Model):
class Meta:
database = database
metadata = metadata

id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
courses = ormar.ManyToMany(Course,
through_relation_name="student_id",
through_reverse_relation_name="course_id")

will produce default Through model like follows (example simplified)
class StudentCourse(ormar.Model):
class Meta:
database = database
metadata = metadata
tablename = "students_courses"

id: int = ormar.Integer(primary_key=True)
student_id = ormar.ForeignKey(Student) set by through_relation_name
course_id = ormar.ForeignKey(Course) set by through_reverse_relation_name


🐛 Fixes

* Fix weakref `ReferenceError` error [118](https://github.com/collerek/ormar/issues/118)
* Fix error raised by Through fields when pydantic `Config.extra="forbid"` is set
* Fix bug with `pydantic.PrivateAttr` not being initialized at `__init__` [149](https://github.com/collerek/ormar/issues/149)
* Fix bug with pydantic-type `exclude` in `dict()` with `__all__` key not working

💬 Other
* Introduce link to `sqlalchemy-to-ormar` auto-translator for models
* Provide links to fastapi ecosystem libraries that support `ormar`
* Add transactions to docs (supported with `databases`)

✨ Features

* `Model.save_related(follow=False)` now accept also two additional arguments: `Model.save_related(follow=False, save_all=False, exclude=None)`.
* `save_all:bool` -> By default (so with `save_all=False`) `ormar` only upserts models that are not saved (so new or updated ones),
with `save_all=True` all related models are saved, regardless of `saved` status, which might be useful if updated
models comes from api call, so are not changed in the backend.
* `exclude: Union[Set, Dict, None]` -> set/dict of relations to exclude from save, those relation won't be saved even with `follow=True` and `save_all=True`.
To exclude nested relations pass a nested dictionary like: `exclude={"child":{"sub_child": {"exclude_sub_child_realtion"}}}`. The allowed values follow
the `fields/exclude_fields` (from `QuerySet`) methods schema so when in doubt you can refer to docs in queries -> selecting subset of fields -> fields.
* `Model.update()` method now accepts `_columns: List[str] = None` parameter, that accepts list of column names to update. If passed only those columns will be updated in database.
Note that `update()` does not refresh the instance of the Model, so if you change more columns than you pass in `_columns` list your Model instance will have different values than the database!
* `Model.model_dump()` method previously included only directly related models or nested models if they were not nullable and not virtual,
now all related models not previously visited without loops are included in `dict()`. This should be not breaking
as just more data will be dumped to dict, but it should not be missing.
* `QuerySet.delete(each=False, **kwargs)` previously required that you either pass a `filter` (by `**kwargs` or as a separate `filter()` call) or set `each=True` now also accepts
`exclude()` calls that generates NOT filter. So either `each=True` needs to be set to delete whole table or at least one of `filter/exclude` clauses.
* Same thing applies to `QuerySet.update(each=False, **kwargs)` which also previously required that you either pass a `filter` (by `**kwargs` or as a separate `filter()` call) or set `each=True` now also accepts
`exclude()` calls that generates NOT filter. So either `each=True` needs to be set to update whole table or at least one of `filter/exclude` clauses.
* Same thing applies to `QuerysetProxy.update(each=False, **kwargs)` which also previously required that you either pass a `filter` (by `**kwargs` or as a separate `filter()` call) or set `each=True` now also accepts
`exclude()` calls that generates NOT filter. So either `each=True` needs to be set to update whole table or at least one of `filter/exclude` clauses.

🐛 Fixes

* Fix improper relation field resolution in `QuerysetProxy` if fk column has different database alias.
* Fix hitting recursion error with very complicated models structure with loops when calling `dict()`.
* Fix bug when two non-relation fields were merged (appended) in query result when they were not relation fields (i.e. JSON)
* Fix bug when during translation to dict from list the same relation name is used in chain but leads to different models
* Fix bug when bulk_create would try to save also `property_field` decorated methods and `pydantic` fields
* Fix wrong merging of deeply nested chain of reversed relations

💬 Other

* Performance optimizations
* Split tests into packages based on tested area

Features

* add `get_or_none(**kwargs)` method to `QuerySet` and `QuerysetProxy`. It is exact equivalent of `get(**kwargs)` but instead of raising `ormar.NoMatch` exception if there is no db record matching the criteria, `get_or_none` simply returns `None`.

Fixes

* Fix dialect dependent quoting of column and table names in order_by clauses not working
properly in postgres.

Breaking

* Dropped supported for long deprecated notation of field definition in which you use ormar fields as type hints i.e. `test_field: ormar.Integger() = None`
* Improved type hints -> `mypy` can properly resolve related models fields (`ForeignKey` and `ManyToMany`) as well as return types of `QuerySet` methods.
Those mentioned are now returning proper model (i.e. `Book`) instead or `ormar.Model` type. There is still problem with reverse sides of relation and `QuerysetProxy` methods,
to ease type hints now those return `Any`. Partially fixes 112.

Features

* add `select_all(follow: bool = False)` method to `QuerySet` and `QuerysetProxy`.
It is kind of equivalent of the Model's `load_all()` method but can be used directly in a query.
By default `select_all()` adds only directly related models, with `follow=True` also related models
of related models are added without loops in relations. Note that it's not and end `async` model
so you still have to issue `get()`, `all()` etc. as `select_all()` returns a QuerySet (or proxy)
like `fields()` or `order_by()`.

Internals

* `ormar` fields are no longer stored as classes in `Meta.model_fields` dictionary
but instead they are stored as instances.

Features
* Add possibility to change default ordering of relations and models.
* To change model sorting pass `orders_by = [columns]` where `columns: List[str]` to model `Meta` class
* To change relation order_by pass `orders_by = [columns]` where `columns: List[str]`
* To change reverse relation order_by pass `related_orders_by = [columns]` where `columns: List[str]`
* Arguments can be column names or `-{col_name}` to sort descending
* In relations you can sort only by directly related model columns
or for `ManyToMany` columns also `Through` model columns `"{through_field_name}__{column_name}"`
* Order in which order_by clauses are applied is as follows:
* Explicitly passed `order_by()` calls in query
* Relation passed `orders_by` if exists
* Model `Meta` class `orders_by`
* Model primary key column asc (fallback, used if none of above provided)
* Add 4 new aggregated functions -> `min`, `max`, `sum` and `avg` that are their
corresponding sql equivalents.
* You can pass one or many column names including related columns.
* As of now each column passed is aggregated separately (so `sum(col1+col2)` is not possible,
you can have `sum(col1, col2)` and later add 2 returned sums in python)
* You cannot `sum` and `avg` non numeric columns
* If you aggregate on one column, the single value is directly returned as a result
* If you aggregate on multiple columns a dictionary with column: result pairs is returned
* Add 4 new signals -> `pre_relation_add`, `post_relation_add`, `pre_relation_remove` and `post_relation_remove`
* The newly added signals are emitted for `ManyToMany` relations (both sides)
and reverse side of `ForeignKey` relation (same as `QuerysetProxy` is exposed).
* Signals receive following args: `sender: Type[Model]` - sender class,
`instance: Model` - instance to which related model is added, `child: Model` - model being added,
`relation_name: str` - name of the relation to which child is added,
for add signals also `passed_kwargs: Dict` - dict of kwargs passed to `add()`

Changes
* `Through` models for ManyToMany relations are now instantiated on creation, deletion and update, so you can provide not only
autoincrement int as a primary key but any column type with default function provided.
* Since `Through` models are now instantiated you can also subscribe to `Through` model
pre/post save/update/delete signals
* `pre_update` signals receivers now get also passed_args argument which is a
dict of values passed to update function if any (else empty dict)

Fixes
* `pre_update` signal now is sent before the extraction of values so you can modify the passed
instance in place and modified fields values will be reflected in database
* `bulk_update` now works correctly also with `UUID` primary key column type