Scrapy-zyte-api

-------------------

* Added support for :ref:`custom attribute extraction <custom-attrs>`.
* Added the :class:`~scrapy_zyte_api.LocationSessionConfig` class.

-------------------

* Fixed an issue in the handling of excessive session initialization failures
during session refreshing, which would manifest as an asyncio messages about
unretrieved ``TooManyBadSessionInits`` task exceptions instead of stopping
the spider as intended.

-------------------

* ``scrapy-zyte-api[provider]`` now requires :doc:`zyte-common-items
<zyte-common-items:index>` 0.20.0+.

* Added the :setting:`ZYTE_API_AUTO_FIELD_STATS` setting.

* Added the :func:`~scrapy_zyte_api.is_session_init_request` function.

* Added the :data:`~scrapy_zyte_api.session_config_registry` variable.

-------------------

* **Backward-incompatible change:** The precedence of session param settings,
request metadata keys and session config override methods has changed.

Before, priority from higher to lower was:

. :meth:`~scrapy_zyte_api.SessionConfig.params`

. :meth:`~scrapy_zyte_api.SessionConfig.location`

. :reqmeta:`zyte_api_session_location`

. :setting:`ZYTE_API_SESSION_LOCATION`

. :reqmeta:`zyte_api_session_params`

. :setting:`ZYTE_API_SESSION_PARAMS`

Now, it is:

. :reqmeta:`zyte_api_session_params`

. :reqmeta:`zyte_api_session_location`

. :setting:`ZYTE_API_SESSION_PARAMS`

. :setting:`ZYTE_API_SESSION_LOCATION`

. :meth:`~scrapy_zyte_api.SessionConfig.location`

. :meth:`~scrapy_zyte_api.SessionConfig.params`

* When using the :reqmeta:`zyte_api_session_params` or
:reqmeta:`zyte_api_session_location` request metadata keys, a different pool
ID is now generated by default based on their value. See
:meth:`~scrapy_zyte_api.SessionConfig.pool` for details.

* The new :reqmeta:`zyte_api_session_pool` request metadata key allows
overriding the pool ID of a request.

* Added :ref:`pool management documentation <session-pools>`.

* Fixed some documentation examples where the parameters of the ``check``
method of :setting:`ZYTE_API_SESSION_CHECKER` were in reverse order.

-------------------

* If the :setting:`AUTOTHROTTLE_ENABLED <scrapy:AUTOTHROTTLE_ENABLED>` setting
is ``False``, the delay of download slots for Zyte API requests no longer
resets to zero, and instead scrapy-zyte-api respects the
:setting:`DOWNLOAD_DELAY <scrapy:DOWNLOAD_DELAY>` setting and
``zyte-api``-prefixed entries in the :setting:`DOWNLOAD_SLOTS
<scrapy:DOWNLOAD_SLOTS>` setting.

A new :setting:`ZYTE_API_PRESERVE_DELAY` setting allows overriding this
behavior, i.e. enabling delay resetting even if
:setting:`AUTOTHROTTLE_ENABLED <scrapy:AUTOTHROTTLE_ENABLED>` is ``False`` or
disabling delay resetting even if :setting:`AUTOTHROTTLE_ENABLED
<scrapy:AUTOTHROTTLE_ENABLED>` is ``True``.

* The :reqmeta:`zyte_api_session_location` and
:reqmeta:`zyte_api_session_params` request metadata keys, if present in a
request that triggers a session initialization request, will be copied into
the session initialization request, so that they are available when
:setting:`ZYTE_API_SESSION_CHECKER` or :meth:`SessionConfig.check
<scrapy_zyte_api.SessionConfig.check>` are called for a session
initialization request.

* The new :meth:`SessionConfig.enabled <scrapy_zyte_api.SessionConfig.enabled>`
method allows configuring whether session management should be enabled or
disabled for any given request.

* A new stat, ``scrapy-zyte-api/sessions/use/disabled``, indicates the number
of requests for which session management was disabled.

-------------------

* Implemented a :ref:`session management API <session>`.

* The recommended position for ``ScrapyZyteAPIDownloaderMiddleware`` changed
from 1000 to 633, to accommodate for the new
``ScrapyZyteAPISessionDownloaderMiddleware``, which needs to be after
``ScrapyZyteAPIDownloaderMiddleware`` and before the Scrapy cookie downloader
middleware (700).