Scrapy-zyte-api

------------------

* It is now possible to `log the parameters of requests sent`_.

.. _log the parameters of requests sent: https://github.com/scrapy-plugins/scrapy-zyte-api#logging-request-parameters

* Stats for HTTP and HTTPS traffic used to be kept separate, and only one of
those sets of stats would be reported. This is fixed now.

* Fixed some code examples and references in the README.

------------------

When upgrading, you should set the following in your Scrapy settings:

.. code-block:: python

DOWNLOADER_MIDDLEWARES = {
"scrapy_zyte_api.ScrapyZyteAPIDownloaderMiddleware": 633,
}
only applicable for Scrapy 2.7+
REQUEST_FINGERPRINTER_CLASS = "scrapy_zyte_api.ScrapyZyteAPIRequestFingerprinter"

* Fixes the issue where scrapy-zyte-api is slow when Scrapy Cloud has Autothrottle
Addon enabled. The new ``ScrapyZyteAPIDownloaderMiddleware`` fixes this.

* It now supports Scrapy 2.7's new ``REQUEST_FINGERPRINTER_CLASS`` which ensures
that Zyte API requests are properly fingerprinted. This addresses the issue
where Scrapy marks POST requests as duplicate if they point to the same URL
despite having different request bodies. As a workaround, users were marking
their requests with ``dont_filter=True`` to prevent such dupe filtering.

For users having ``scrapy >= 2.7``, you can simply update your Scrapy settings
to have ``REQUEST_FINGERPRINTER_CLASS = "scrapy_zyte_api.ScrapyZyteAPIRequestFingerprinter"``.

If your Scrapy project performs other requests aside from Zyte API, you can set
``ZYTE_API_FALLBACK_REQUEST_FINGERPRINTER_CLASS = "custom.RequestFingerprinter"``
to allow custom fingerprinting. By default, the default Scrapy request
fingerprinter is used for non-Zyte API requests.

For users having ``scrapy < 2.7``, check the following link to see different
ways on handling the duplicate request issue:
https://github.com/scrapy-plugins/scrapy-zyte-api#request-fingerprinting-before-scrapy-27.

More information about the request fingerprinting topic can be found in
https://github.com/scrapy-plugins/scrapy-zyte-api#request-fingerprinting.

* Various improvements to docs and tests.

------------------

* Add a ``ZYTE_API_TRANSPARENT_MODE`` setting, ``False`` by default, which can
be set to ``True`` to make all requests use Zyte API by default, with request
parameters being automatically mapped to Zyte API parameters.
* Add a Request meta key, ``zyte_api_automap``, that can be used to enable
automatic request parameter mapping for specific requests, or to modify the
outcome of automatic request parameter mapping for specific requests.
* Add a ``ZYTE_API_AUTOMAP_PARAMS`` setting, which is a counterpart for
``ZYTE_API_DEFAULT_PARAMS`` that applies to requests where automatic request
parameter mapping is enabled.
* Add the ``ZYTE_API_SKIP_HEADERS`` and ``ZYTE_API_BROWSER_HEADERS`` settings
to control the automatic mapping of request headers.
* Add a ``ZYTE_API_ENABLED`` setting, ``True`` by default, which can be used to
disable this plugin.
* Document how Zyte API responses are mapped to Scrapy response subclasses.

------------------

* Raise the minimum dependency of Zyte API's Python API to ``zyte-api>=0.4.0``.
This changes all the requests to Zyte API to have have ``Accept-Encoding: br``
and automatically decompress brotli responses.
* Rename "Zyte Data API" to simply "Zyte API" in the README.
* Lower the minimum Scrapy version from ``2.6.0`` to ``2.0.1``.

------------------

* Zyte Data API error responses (after retries) are no longer ignored, and
instead raise a ``zyte_api.aio.errors.RequestError`` exception, which allows
user-side handling of errors and provides better feedback for debugging.
* Allowed retry policies to be specified as import path strings, which is
required for the ``ZYTE_API_RETRY_POLICY`` setting, and allows requests with
the ``zyte_api_retry_policy`` request.meta key to remain serializable.
* Fixed the naming of stats for some error types.
* Updated the output examples on the README.

------------------

* Cleaned up Scrapy stats names: fixed an issue with ``//``, renamed
``scrapy-zyte-api/api_error_types/..`` to ``scrapy-zyte-api/error_types/..``,
added ``scrapy-zyte-api/error_types/<empty>`` for cases error type is unknown;
* Added error type to the error log messages
* Testing improvements