Wayback

In v0.4.4, we broke *archived mementos* of rate limit errors — they started raising exceptions instead of returning the actual memento. We now correctly return mementos of rate limit errors while still raising exceptions for actual live rate limit errors from the Wayback Machine itself. (158)

**Full Changelog**: https://github.com/edgi-govdata-archiving/wayback/compare/v0.4.4...v0.4.5

This release makes some small fixes to rate limits and retries in order to better match with the current behavior of Wayback Machine servers:

- Updates the `WaybackClient.search` rate limit to 1 call per second (it was previously 1.5 per second). (140)

- Delays retries for 60 seconds when receiving rate limit errors from the server. (142)

- Adds more logging around requests and rate limiting. This should make it easier to debug future rate limit issues. (139)

- Fixes calculation of the `time` attribute on `wayback.exceptions.WaybackRetryError`. It turns out it was only accounting for the time spent waiting between retries and skipping the time waiting for the server to respond! (142)

- Fixes some spots where we leaked HTTP connections during retries or during exception handling. (142)

The next minor release (v0.5) will almost certainly include some bigger changes to how rate limits and retries are handled.

This is mainly a compatibility release: it adds support for urllib3 v2.x and the next upcoming major release of Python, v3.12.0. It also adds support for multiple filters in searches. There are no breaking changes.


Features

You can now apply multiple filters to a search by using a list or tuple for the ``filter_field`` parameter of `WaybackClient.search`. Previously, you could only supply a string with a single filter. (119)

For example, to search for all captures at `nasa.gov` with a 404 status code and “feature” somewhere in the URL:

python
client.search('nasa.gov/',
match_type='prefix',
filter_field=['statuscode:404',
'urlkey:.*feature.*'])



Fixes & Maintenance

- Add support for Python 3.12.0. (123)
- Add support for urllib3 v2.x (urllib3 v1.20+ also still works). (116)

This is a test release for properly supporting the upcoming release of Python 3.12.0. Please file an issue if you encounter issues using on Python 3.12.0rc3 or later. (123)

Wayback is not compatible with urllib3 v2, and this release updates the package's requirements to make sure Pip and other package managers install compatible versions of Wayback and urllib3. There are no other fixes or new features.

Features

`wayback.Memento` now has a `links` property with information about other URLs that are related to the memento, such as the previous or next mementos in time. It’s a dict where the keys identify the relationship (e.g. `'prev memento'`) and the values are dicts with additional information about the link. (57)

For example::

py
{
'original': {
'url': 'https://www.fws.gov/birds/',
'rel': 'original'
},
'first memento': {
'url': 'https://web.archive.org/web/20050323155300id_/http://www.fws.gov:80/birds',
'rel': 'first memento',
'datetime': 'Wed, 23 Mar 2005 15:53:00 GMT'
},
'prev memento': {
'url': 'https://web.archive.org/web/20210125125216id_/https://www.fws.gov/birds/',
'rel': 'prev memento',
'datetime': 'Mon, 25 Jan 2021 12:52:16 GMT'
},
'next memento': {
'url': 'https://web.archive.org/web/20210321180831id_/https://www.fws.gov/birds',
'rel': 'next memento',
'datetime': 'Sun, 21 Mar 2021 18:08:31 GMT'
},
'last memento': {
'url': 'https://web.archive.org/web/20221006031005id_/https://fws.gov/birds',
'rel': 'last memento',
'datetime': 'Thu, 06 Oct 2022 03:10:05 GMT'
}
}


One use for these is to iterate through additional mementos. For example, to get the previous memento::

py
client.get_memento(memento.links['prev memento']['url'])



Fixes & Maintenance

- Fix an issue where the `Memento.url` attribute might be slightly off from the exact URL that was captured (it could have a different protocol, different upper/lower-casing, etc.). (99)

- Fix an error when getting a memento for a redirect in `view` mode. If you called `wayback.WaybackClient.get_memento` with a URL that turned out to be a redirect at the given time and set the `mode` option to `wayback.Mode.view`, you’d get an exception saying “Memento at {url} could not be played.” Now this works just fine. (109)