Scrapy-zyte-api

-------------------

* **Backward-incompatible change:** The precedence of session param settings,
request metadata keys and session config override methods has changed.

Before, priority from higher to lower was:

. :meth:`~scrapy_zyte_api.SessionConfig.params`

. :meth:`~scrapy_zyte_api.SessionConfig.location`

. :reqmeta:`zyte_api_session_location`

. :setting:`ZYTE_API_SESSION_LOCATION`

. :reqmeta:`zyte_api_session_params`

. :setting:`ZYTE_API_SESSION_PARAMS`

Now, it is:

. :reqmeta:`zyte_api_session_params`

. :reqmeta:`zyte_api_session_location`

. :setting:`ZYTE_API_SESSION_PARAMS`

. :setting:`ZYTE_API_SESSION_LOCATION`

. :meth:`~scrapy_zyte_api.SessionConfig.location`

. :meth:`~scrapy_zyte_api.SessionConfig.params`

* When using the :reqmeta:`zyte_api_session_params` or
:reqmeta:`zyte_api_session_location` request metadata keys, a different pool
ID is now generated by default based on their value. See
:meth:`~scrapy_zyte_api.SessionConfig.pool` for details.

* The new :reqmeta:`zyte_api_session_pool` request metadata key allows
overriding the pool ID of a request.

* Added :ref:`pool management documentation <session-pools>`.

* Fixed some documentation examples where the parameters of the ``check``
method of :setting:`ZYTE_API_SESSION_CHECKER` were in reverse order.

-------------------

* If the :setting:`AUTOTHROTTLE_ENABLED <scrapy:AUTOTHROTTLE_ENABLED>` setting
is ``False``, the delay of download slots for Zyte API requests no longer
resets to zero, and instead scrapy-zyte-api respects the
:setting:`DOWNLOAD_DELAY <scrapy:DOWNLOAD_DELAY>` setting and
``zyte-api``-prefixed entries in the :setting:`DOWNLOAD_SLOTS
<scrapy:DOWNLOAD_SLOTS>` setting.

A new :setting:`ZYTE_API_PRESERVE_DELAY` setting allows overriding this
behavior, i.e. enabling delay resetting even if
:setting:`AUTOTHROTTLE_ENABLED <scrapy:AUTOTHROTTLE_ENABLED>` is ``False`` or
disabling delay resetting even if :setting:`AUTOTHROTTLE_ENABLED
<scrapy:AUTOTHROTTLE_ENABLED>` is ``True``.

* The :reqmeta:`zyte_api_session_location` and
:reqmeta:`zyte_api_session_params` request metadata keys, if present in a
request that triggers a session initialization request, will be copied into
the session initialization request, so that they are available when
:setting:`ZYTE_API_SESSION_CHECKER` or :meth:`SessionConfig.check
<scrapy_zyte_api.SessionConfig.check>` are called for a session
initialization request.

* The new :meth:`SessionConfig.enabled <scrapy_zyte_api.SessionConfig.enabled>`
method allows configuring whether session management should be enabled or
disabled for any given request.

* A new stat, ``scrapy-zyte-api/sessions/use/disabled``, indicates the number
of requests for which session management was disabled.

-------------------

* Implemented a :ref:`session management API <session>`.

* The recommended position for ``ScrapyZyteAPIDownloaderMiddleware`` changed
from 1000 to 633, to accommodate for the new
``ScrapyZyteAPISessionDownloaderMiddleware``, which needs to be after
``ScrapyZyteAPIDownloaderMiddleware`` and before the Scrapy cookie downloader
middleware (700).

-------------------

* Now the :setting:`ZYTE_API_PROVIDER_PARAMS` setting and the
:reqmeta:`zyte_api_provider` request metadata key can influence the
resolution of an :class:`~web_poet.page_inputs.response.AnyResponse`
dependency.

-------------------

* The log messages from the download handler that indicate the source request
URL of an exception have switched from ``ERROR`` log level to ``DEBUG``. The
exceptions themselves that follow those messages will still be logged as
errors unless you handle them.

-------------------

* The ``Accept``, ``Accept-Encoding``, ``Accept-Language``, and ``User-Agent``
headers are now dropped automatically during :ref:`header mapping
<header-mapping>` unless they have user-defined values. This fix can improve
success rates on some websites when using :ref:`HTTP requests <zapi-http>`.