Reader

-----------

Released 2020-09-19

* Add new methods to support feed tags: :meth:`~Reader.add_feed_tag`,
:meth:`~Reader.remove_feed_tag`, and :meth:`~Reader.get_feed_tags`.
Allow filtering feeds and entries by their feed tags.
(:issue:`184`)
* Add the ``broken`` argument to :meth:`~Reader.get_feeds`,
which allows getting only feeds that failed / did not fail
during the last update.
(:issue:`189`)
* feedparser 5.x support is deprecated in favor of feedparser 6.x.
Using feedparser 5.x will raise a deprecation warning in version 1.7,
and support will be removed the following version.
(:issue:`190`)
* Tag-related web application features:
show tags in the feed list;
allow adding/removing tags;
allow filtering feeds and entries by their feed tag;
add a page that lists all tags.
(:issue:`184`)
* In the web application, allow showing only feeds that failed / did not fail.
(:issue:`189`)
* In the ``preview_feed_list`` plugin, add ``<meta>`` tags as
a feed detection heuristic.
* Add a few property-based tests. (:issue:`188`)

-----------

Released 2020-09-04

* Add the ``feed_root`` argument to :func:`make_reader`,
which allows limiting local feed parsing to a specific directory
or disabling it altogether.
Using it is recommended, since by default *reader* will access
any local feed path
(in 2.0, local file parsing will be disabled by default).
(:issue:`155`)
* Support loading CLI and web application settings from a
:doc:`configuration file <config>`. (:issue:`177`)
* Fail fast for feeds that return HTTP 4xx or 5xx status codes,
instead of (likely) failing later with an ambiguous XML parsing error.
The cause of the raised :exc:`ParseError` is now an instance of
:exc:`requests.HTTPError`. (:issue:`182`)
* Add ``cloudflare_ua_fix`` plugin (work around Cloudflare sometimes
blocking requests). (:issue:`181`)
* feedparser 6.0 (beta) compatibility fixes.
* Internal parser API changes to support alternative parsers, pre-request hooks,
and making arbitrary HTTP requests using the same logic :class:`Reader` uses.
(:issue:`155`)
* In the /preview page and the ``preview_feed_list`` plugin,
use the same plugins the main :class:`Reader` does.
(enabled by :issue:`155`)

-----------

Released 2020-07-30

* Use rowid when deleting from the search index, instead of the entry id.
Previously, each :meth:`~Reader.update_search` call would result in a full
scan, even if there was nothing to update/delete.
This should reduce the amount of reads significantly
(deleting 4 entries from a database with 10k entries
resulted in an 1000x decrease in bytes read).
(:issue:`178`)
* Require at least SQLite 3.18 (released 2017-03-30) for the current
:meth:`~Reader.update_search` implementation;
all other *reader* features continue to work with SQLite >= 3.15.
(:issue:`178`)
* Run ``PRAGMA optimize`` on :meth:`~Reader.close()`.
This should increase the performance of all methods.
As an example, in :issue:`178` it was found that :meth:`~Reader.update_search`
resulted in a full scan of the entries table,
even if there was nothing to update;
this change should prevent this from happening.
(:issue:`143`)

.. note::
``PRAGMA optimize`` is a no-op in SQLite versions earlier than 3.18.
In order to avoid the case described above, you should run `ANALYZE`_
regularly (e.g. every few days).

.. _ANALYZE: https://www.sqlite.org/lang_analyze.html

-----------

Released 2020-07-13

* Work to reduce the likelihood of "database is locked" errors during updates
(:issue:`175`):

* Prepare entries to be added to the search index
(:meth:`~Reader.update_search`) outside transactions.
* Fix bug causing duplicate rows in the search index
when an entry changes while updating the search index.
* Update the search index only when the indexed values change (details below).
* Use SQLite WAL (details below).

* Update the search index only when the indexed values change.
Previously, any change on a feed would result in all its entries being
re-indexed, even if the feed title or the entry content didn't change.
This should reduce the :meth:`~Reader.update_search` run time significantly.
* Use SQLite's `write-ahead logging`_ to increase concurrency.
At the moment there is no way to disable WAL.
This change may be reverted in the future.
(:issue:`169`)
* Require at least click 7.0 for the ``cli`` extra.
* Do not fail for feeds with incorrectly-declared media types,
if feedparser can parse the feed;
this is similar to the current behavior for incorrectly-declared encodings.
(:issue:`171`)
* Raise :exc:`ParseError` during update for feeds feedparser can't detect
the type of, instead of silently returning an empty feed. (:issue:`171`)
* Add ``sort`` argument to :meth:`~Reader.search_entries`.
Allow sorting search results by recency in addition to relevance
(the default). (:issue:`176`)
* In the web application, display a nice error message for invalid search
queries instead of returning an HTTP 500 Internal Server Error.
* Other minor web application improvements.
* Minor CLI logging improvements.

.. _write-ahead logging: https://www.sqlite.org/wal.html

-----------

Released 2020-06-23

* If a feed failed to update, provide details about the error
in :attr:`Feed.last_exception`. (:issue:`68`)
* Show details about feed update errors in the web application. (:issue:`68`)
* Expose the :attr:`~Feed.added` and :attr:`~Feed.last_updated` Feed attributes.
* Expose the :attr:`~Entry.last_updated` Entry attribute.
* Raise :exc:`ParseError` / log during update if an entry has no id,
instead of unconditionally raising :exc:`AttributeError`. (:issue:`170`)
* Fall back to <link> as entry id if an entry in an RSS feed has no <guid>;
previously, feeds like this would fail on update. (:issue:`170`)
* Minor web application improvements (show feed added/updated date).
* In the web application, handle previewing an invalid feed nicely
instead of returning an HTTP 500 Internal Server Error. (:issue:`172`)
* Internal API changes to support multiple storage implementations
in the future. (:issue:`168`)

-----------

Released 2020-05-18

* Minor web application improvements.
* Remove unneeded additional query in methods that use pagination
(for n = len(result) / page size, always do n queries instead n+1).
:meth:`~Reader.get_entries` and :meth:`~Reader.search_entries` are now
33–7% and 46–36% faster, respectively, for results of size 32–256.
(:issue:`166`)
* All queries are now chunked/paginated to avoid locking the SQLite storage
for too long, decreasing the chance of concurrent queries timing out;
the problem was most visible during :meth:`~Reader.update_search`.
This should cap memory usage for methods returning an iterable
that were not paginated before;
previously the whole result set would be read before returning it.
(:issue:`167`)