Webchanges

===================
2024-10-13

Added
-----
* Python 3.13 Support: **webchanges** now supports Python 3.13, but complete testing is pending due to dependencies
such has ``lxml`` not having yet published installation packages ("wheels") for 3.13.
* Glob Pattern Support for Hooks Files: The ``--hooks`` command-line argument now accepts glob patterns for flexible
hook file selection.
* Multiple Hook Specifications: Specify multiple hook files or glob patterns by repeating the ``--hooks`` argument.
* Enhanced Version Information: ``--detailed-versions`` now displays the system's default value for
``--max-threads``.
* Optional ``zstd`` Compression: URL jobs without ``browser: true`` can now utilize ``zstd`` compression for
improved efficiency (requires ``pip install -U webchanges[zstd]``).
* ``ai_google`` Differ Enhancements (BETA):

* New ``additions_only`` Subdirective: When set to true, generates AI-powered summaries of only the added text. This
is particularly helpful for monitoring pages with regularly added content (e.g., press releases).
* New ``unified_diff_new`` Field: Added to the ``prompt`` directive.

Changed
-------
* Relaxed Security for Job and Hook Files: The ownership requirement for files containing ``command`` jobs,
``shellpipe`` filters, or hook files has been expanded to include root ownership, in addition to the current user.
* ``ai_google`` Differ Refinements (BETA):

* Renamed Prompt Fields (⚠ BETA breaking change): For clarity, ``old_data`` and ``new_data`` fields in the
``prompt`` directive have been renamed to ``old_text`` and ``new_text``, respectively.
* Improved Output Quality: Significantly enhanced output quality by revising the default values for
``system_instructions`` and ``prompt``.
* Updated Documentation.

Fixed
-----
* Markdown Handling: Improved handling of links with empty text in the Markdown to HTML converter.
* ``image`` Differ Formatting: Fixed HTML formatting issues within the ``image`` differ.

Removed
-------
* Python 3.9 Support: Support for Python 3.9 has been dropped. As a reminder, older Python versions are supported for 3
years after being superseded by a new major release (i.e. approximately 4 years after their initial release).

===================
2024-08-15

Added
-------------------
* Multiple job files or glob patterns can now be specified by repeating the ``--jobs`` argument.
* Job list filtering using `Python regular expression
<https://docs.python.org/3/library/re.html#regular-expression-syntax>`__. Example: ``webchanges --list blue`` lists
jobs with 'blue' in their name (case-sensitive, so not 'Blue'), while ``webchanges --list (?i)blue`` is
`case-insensitive <https://docs.python.org/3/library/re.html#re.I>`__.
* New URL job directive ``params`` for specifying URL parameters (query strings), e.g. as a dictionary.
* New ``gotify`` reporter (upstream contribution: `link <https://github.com/thp/urlwatch/pull/823/files>`__).
* Improved messaging at startup when a legacy database that requires conversion is found.

Changed
-------------------
* Updated ``ai_google`` differ to reflect Gemini 1.5 Pro's 2M token context window.

Fixed
-------------------
* Corrected the automated handling in differs and reporters of data with a 'text/markdown' MIME type.
* Multiple ``wdiff`` differ fixes and improvements:
- Fixed body font issues;
- Removed spurious ``^\n`` insertions;
- Corrected ``range_info`` lines;
- Added word break opportunities (``<wbr>``) in HTML output for better browser handling of long lines.
* ``deepdiff`` differ now breaks a list into its individual elements.
* Improved URL matching for jobs by normalizing %xx escapes and plus signs (e.g. ``https://www.example.org/El Niño``
will now match ``https://www.example.org/El+Ni%C3%B1o`` and vice versa).
* Improved the text-to-HTML URL parser to accurately extract URLs with multiple parameters.

Internals
-------------------
* Replaced ``requests.structures.CaseInsensitiveDict`` with ``httpx.Headers`` as the Class holding headers.
* The ``Job.headers`` attribute is now initialized with an empty ``httpx.Headers`` object instead of None.

===================
2024-06-14

Added
-------------------
* Command line argument ``--rollback-database`` now accepts dates in ISO-8601 format in addition to Unix timestamps.
If the library dateutil (not a dependency of **webchanges**) is found installed, then it will also accept any
string recognized by ``dateutil.parser`` such as date only, time only, date and time, etc. (suggested
by `Markus Weimar <https://github.com/Markus00000>`__ in issue `#78
<https://github.com/mborsetti/webchanges/issues/78>`__).
* ``ai-google`` differ (BETA) now supports calls to the Gemini 1.5 Pro with 2M tokens model (early access required).

===================
2024-06-06

Added
-------------------
* New ``wdiff`` differ to perform word-by-word comparisons. Replaces the dependency on an outside executable and
allows for much better formatting and integration.
* New ``system_instructions`` directive added to the ``ai-google`` differ (BETA).
* Added to the documentation examples on how to use the ``re.findall`` filter to extract only the first or last line
(suggested by `Marcos Alano <https://github.com/malano>`__ in issue `#81
<https://github.com/mborsetti/webchanges/issues/81>`__).

Changed
------------------
* Updated the documentation for the ``ai-google`` differ (BETA), mostly to reflect billing changes by Google, which is
still free for most.

Fixed
------------------
* Fixed a data type check in preventing ``URL`` jobs' ``data`` (for POSTs etc.) to be a list.

===================
2024-05-22

Changed
------------------
* Updated the ``ai-google`` differ (BETA)'s default model to ``gemini-1.5-flash-latest`` due to changes in the Google
API, and its default prompt to ``Identify and summarize the changes between the old and new
documents:\n\n<old>\n{old_data}\n</old>\n\n``, due to the old prompt not generating the expected output. Updated
the documentation.

===================
2024-05-15

Changed
------------------
* The ``ai-google`` (BETA) differ now defaults to using the new ``gemini-1.5-flash`` model (see documentation `here
<https://ai.google.dev/gemini-api/docs/models/gemini#gemini-1.5-flash-expandable>`__), as it still supports
1M tokens, "excels at summarization" (per `here <https://blog
.google/technology/ai/google-gemini-update-flash-ai-assistant-io-2024/gemini-model-updates:~:text=1
.5%20flash%20excels%20at%20summarization%2C>`__), allows for a higher number of requests per minute (in the
free version, 15 vs. 2 of ``gemini-1.5-pro``), is faster, and, if you're paying for it, cheaper. To continue to
use ``gemini-1.5-pro``, which may produce more "complex" results, specify it in the job's ``differ`` directive.

Fixed
-----
* Fixed header of ``deepdiff`` and ``image`` (BETA) differs to be more consistent with the default ``unified`` differ.
* Fixed the way images are handled in the email reporter so that they now display correctly in clients such as Gmail.

Internals
---------
* Command line argument ``--test-differs`` now processes the new ``mime_type`` attribute correctly (``mime_type`` is
an internal work in progress attribute to facilitate future automation of filtering, diffing, and reporting).