Scrapy-zyte-api

-------------------

* Removed support for Python 3.7.
* Updated requirement versions:

* :doc:`scrapy-poet <scrapy-poet:index>` >= 0.21.0
* :doc:`web-poet <web-poet:index>` >= 0.16.0

* Added support for :class:`web_poet.AnyResponse
<web_poet.page_inputs.response.AnyResponse>` dependency.
* Added support to specify the country code via :class:`typing.Annotated` and
:class:`scrapy_zyte_api.Geolocation` dependency *(supported only on Python
3.9+)*.
* Improved tests.

-------------------

* Updated requirement versions:

* :doc:`scrapy-poet <scrapy-poet:index>` >= 0.20.1

* Dependency injection :ref:`through scrapy-poet <scrapy-poet>` is now taken
into account for request fingerprinting.

Now, when scrapy-poet is installed, the default value of the
:setting:`ZYTE_API_FALLBACK_REQUEST_FINGERPRINTER_CLASS` setting is
:class:`scrapy_poet.ScrapyPoetRequestFingerprinter`, and a warning will be
issued if a custom value is not a subclass of
:class:`~scrapy_poet.ScrapyPoetRequestFingerprinter`.

* :ref:`Zyte Smart Proxy Manager special headers <spm-request-headers>` will
now be dropped automatically when using :ref:`transparent mode <transparent>`
or :ref:`automatic request parameters <automap>`. Where possible, they will
be replaced with equivalent Zyte API parameters. In all cases, a warning will
be issued.

* Covered the configuration of
:class:`scrapy_zyte_api.ScrapyZyteAPISpiderMiddleware` in the :ref:`setup
documentation <setup>`.

:class:`~scrapy_zyte_api.ScrapyZyteAPISpiderMiddleware` was added in
scrapy-zyte-api 0.13.0, and is required to automatically close spiders when
all start requests fail because they are pointing to domains forbidden by
Zyte API.

-------------------

* The assignment of a custom download slot to requests that use Zyte API now
also happens in the spider middleware, not only in the downloader middleware.

This way requests get a download slot assigned before they reach the
scheduler, making Zyte API requests work as expected with
:class:`scrapy.pqueues.DownloaderAwarePriorityQueue`.

.. note:: New requests created from downloader middlewares do not get their
download slot assigned before they reach the scheduler. So, unless
they reuse the metadata from a requests that did get a download
slot assigned (e.g. retries, redirects), they will continue not to
work as expected with
:class:`~scrapy.pqueues.DownloaderAwarePriorityQueue`.

-------------------

* Updated requirement versions:

* andi >= 0.6.0
* scrapy-poet >= 0.19.0
* zyte-common-items >= 0.8.0

* Added support for ``zyte_common_items.JobPosting`` to the scrapy-poet provider.

-------------------

* Updated requirement versions:

* andi >= 0.5.0
* scrapy-poet >= 0.18.0
* web-poet >= 0.15.1
* zyte-api >= 0.4.8

* The spider is now closed and the finish reason is set to
``"zyte_api_bad_key"`` or ``"zyte_api_suspended_account"`` when receiving
"Authentication Key Not Found" or "Account Suspended" responses from Zyte
API.

* The spider is now closed and the finish reason is set to
``"failed_forbidden_domain"`` when all start requests fail because they are
pointing to domains forbidden by Zyte API.

* The spider is now closed and the finish reason is set to
``"plugin_conflict"`` if both scrapy-zyte-smartproxy and the transparent mode
of scrapy-zyte-api are enabled.

* The ``extractFrom`` extraction option can now be requested by annotating the
dependency with a ``scrapy_zyte_api.ExtractFrom`` member (e.g.
``product: typing.Annotated[Product, ExtractFrom.httpResponseBody]``).

* The ``Set-Cookie`` header is now removed from the response if the cookies
were returned by Zyte API (as ``"experimental.responseCookies"``).

* The request fingerprinting was improved by refining which parts of the
request affect the fingerprint.

* Zyte API Request IDs are now included in the error logs.

* Split README.rst into multiple documentation files and publish them on
ReadTheDocs.

* Improve the documentation for the ``ZYTE_API_MAX_REQUESTS`` setting.

* Test and CI improvements.

-------------------

* Unused ``<data type>Options`` (e.g. ``productOptions``) are now dropped
from ``ZYTE_API_PROVIDER_PARAMS`` when sending the Zyte API request
* When logging Zyte API requests, truncation now uses
"..." instead of Unicode ellipsis.