Webchanges

==================
2025-03-29

Added
-----
* README links to a new Docker implementation which includes the Chrome browser. Generously offered by and maintained
by `Jeff Hedlund <https://github.com/jhedlund>`__ as per `#96 <https://github.com/mborsetti/webchanges/issues/96>`__.
* New filter ``jsontoyaml`` to convert JSON to YAML, which is generally more readable in a report for humans.
* New ``yaml`` data type for the ``deepdiff`` differ (in addition to ``json`` and ``xml``).

Changed
-------
* The ``deepdiff`` differ will now try to derive the ``data-type`` (when it is not specified) from the data's media
type (fka MIME type) before defaulting to ``json``.

Fixed
-----
* Fixed confusing warning when no default hooks.py file exists. Thanks to `Marcos Alano <https://github.com/mhalano>`__
for reporting in `97 <https://github.com/mborsetti/webchanges/issues/97>`__.
* The ``format-json`` filter now uses JSON text instead of plain text to report errors caused by it not receiving
valid JSON data, to be compatible with downstream filters or differs.
* Fixed the differ ``ai_google`` (BETA), which under certain circumstances was omitting the footnote with the version
of the GenAI model used.

==================
2025-03-23

⚠ Breaking Changes
-------------------
* The differ ``command`` now requires that the ``name: command`` subdirective of ``differ`` be specified.

Changed
-------
* The differ ``command`` now has a sub-directive ``is_html`` to indicate when output is in HTML format. Thanks to `Jeff
Hedlund <https://github.com/jhedlund>`__ for requesting this enhancement in
`95 <https://github.com/mborsetti/webchanges/issues/95>`__.
* Added a tip in the documentation on how to `add bullet points
<https://webchanges.readthedocs.io/en/stable/advanced.html#bullet-points>`__ to improve the legibility of HTML
reports.

Fixed
-----
* Fixed reporting of errors arising from filters or reporters.
* Fixed reporting of repeated errors (i.e. when the same error occurs multiple times).
* Fixed header and colorization of the differ ``command``.

==================
2025-03-11

Changed
-------
* The filter ``format-json`` will no longer raise an error when it is not fed JSON data, but, to facilitate
troubleshooting, it will report the JSONDecodeError details and the full string causing the error.
* Documentation for the ``css`` and ``xml`` filters has been split into two separate entries for ease of reference.
* Minor word editing of error messages to improve clarity.
* Various updates to the differ ``ai_google`` (BETA):

- ``top_p`` is set to 1.0 if ``temperature`` is 0.0 (its default value) to eliminate randomness in results.
- Wordsmitting of the default system prompt leading to small improvements.
- The footnote now shows the model actually used vs. the one specified in the ``model`` sub-directive, useful when
omitting the version when using an experimental version (e.g. specifying ``gemini-2.0-pro-exp`` instead of
``gemini-2.0-pro-exp-02-05``).

Internals
---------
* Tested the ``image`` differ's ``ai_google`` directive (ALPHA, undocumented), which uses GenAI to summarize
differences between two images, with the new ``gemini-2.0-pro-exp-02-05`` `experimental
<https://ai.google.dev/gemini-api/docs/models/experimental-models#available-models>`__ and improved default system
prompt. While the new model shows improvements by producing a plausible-sounding summary instead of gibberish, the
summary is highly inaccurate and therefore unusable. Development paused again until model accuracy improves.

==================
2025-02-11

Changed
-------
* Differ ``ai_google`` (BETA) now defaults to using the newer ``gemini-2.0-flash`` GenAI model, as it performs better.
Please note that this model "only" handles 1,048,576 input tokens: if you require the full 2M tokens, manually revert
to using the ``gemini-1.5-pro`` model or try the newer ``gemini-2.0-pro-exp-02-05`` `experimental
<https://ai.google.dev/gemini-api/docs/models/experimental-models#available-models>`__ one.

Fixed
-----
* Fixed bug introduced in 3.28.0 causing an execution error when loading a configuration file that does not contain
the new directive ``differ_defaults``. Thanks `yubiuser <https://github.com/yubiuser>`__ for
reporting this in `issue 93 <https://github.com/mborsetti/webchanges/issues/93>`__.

Internals
---------
* When running with ``-verbose``, no longer logs an INFO message for the internal exception raised when receiving a
an HTTP 304 status code "Not Modified".

==================
2025-02-11

Added
-----
* Added support for setting default differ directives in config.yaml. This is particularly useful for the ``ai_google``
differ to specify a default GenAI model.
* Added automatic installation of the `zstandard <https://github.com/indygreg/python-zstandard>`__ library to support
zstd (`RFC 8878 <https://datatracker.ietf.org/doc/html/rfc8878>`__) compression in ``url`` jobs using the default
HTTPX HTTP client.

Changed
-------
* Renamed job directives ``filter`` and ``diff_filter`` to ``filters`` and ``diff_filters`` (plural nouns) to better
reflect their list nature. The singular forms remain backward-compatible.
* Consolidated HTTP proxy configuration into a single ``proxy`` directive, replacing the separate ``http_proxy`` and
``https_proxy`` directives while maintaining backward compatibility.
* Improved maximum parallel executions of ``use_browser: true`` to ensuring each Chrome instance has at least 400 MB
of available memory (or the maximum available, if lower).

Fixed
-----
* Fixed handling of "Error Ended" reports to only send them with ``suppress_repeated_errors: true``.
* Fixed error message when using job directive ``http_client: requests`` without the `requests
<https://pypi.org/project/requests/>`__ library installed. Thanks `yubiuser <https://github.com/yubiuser>`__ for
reporting this in `issue 90 <https://github.com/mborsetti/webchanges/issues/90>`__.
* Improved and standardized the ogic and documentation for the use of environment variables ``HTTPS_PROXY`` and
``HTTP_PROXY`` in proxy settings.
* Modified ``--prepare-jobs`` command line argument to append never run jobs to command line jobs (``joblist``), if
present, rather than replacing them.

Internals
---------
* Replaced JobBase attributes ``http_proxy`` and ``https_proxy`` with a unified ``proxy`` attribute.
* Updated JobBase attributes from singular ``filter`` and ``diff_filter`` to plural ``filters`` and ``diff_filters``.
* Removed unused JobBase attribute ``chromium_revision`` (deprecated since Pypetteer removal on 2022-05-02).

==================
2025-02-03

Added
-----
* Python 3.13: **webchanges** is now fully tested on Python 3.13 before releasing. However, ``orderedset``, a dependency
of the `aioxmpp <https://pypi.org/project/aioxmpp/>`__ library required by the ``xmpp`` reporter will not install in
Python 3.13 (at least on Windows) and this reporter is therefore not included in the tests. It appears that the
development of this `library <https://codeberg.org/jssfr/aioxmpp>`__ has been halted.

- Python 3.13t (free-threaded, GIL-free) remains unsupported due to the lack of free-threaded wheels for dependencies
such as ``cryptography``, ``msgpack``, ``lxml``, and the optional ``jq``.
* New job directive ``suppress_repeated_errors`` to notify an error condition only the first time it is encountered. No
more notifications will be sent unless the error resolves or a different error occurs. This enhancement was
requested by `toxin-x <https://github.com/toxin-x>`__ in issue `#86
<https://github.com/mborsetti/webchanges/issues/86>`__.
* New command line argument ``--log-file`` to write the log to a file. Suggested by `yubiuser
<https://github.com/yubiuser>`__ in `issue #88 <https://github.com/mborsetti/webchanges/issues/88>`__.
* ``pypdf`` filter has a new ``extraction_mode`` optional sub-directive to enable experimental layout text extraction
mode functionality.
* New command-line option ``--prepare-jobs`` to run only newly added jobs (to capture and save their initial snapshot).

Fixed
-----
* Fixed command line argument ``--errors`` to use the same exact logic as the one used when running *webchanges*.
Reported by `yubiuser <https://github.com/yubiuser>`__ in `issue #88
<https://github.com/mborsetti/webchanges/issues/88>`__.
* Fixed incorrect reporting of job error when caused by an HTTP response status code that is not `IANA-registered
<https://docs.python.org/3/library/http.html#http-status-codes>`__.

Changed
-------
* Command line ``--test`` can now be combined with ``--test-reporter`` to have the output sent to a different reporter.
* Improved error reporting, including reporting error message in ``--test`` and adding proxy information if the error
is a network error and the job has a proxy and.
* Updated the default model instructions for the ``ai_google`` (BETA) differ to improve quality of summary.

Internals
---------
* Now storing error information in snapshot database.
* Added ``ai_google`` directive to the ``image`` differ to test Generative AI summarization of changes between two
images, but in testing the results are unusable. This feature is in ALPHA and undocumented, and will not be
developed further until the models improve to the point where the summary becomes useful.