Granary

Breaking changes

* Drop Python 3.6 support. Python 3.7 is now the minimum required version.
* Twitter, Instagram, Mastodon:
* Drop `get_activities` `cache` kwarg's support for App Engine memcache interface. It's now only used as a plain `dict`. `get_activities` will now make many small modifications, so if you pass an object that implements those as API calls, you'll probably want to batch those separately.
* Twitter, Mastodon, Flickr, GitHub:
* `create`/`preview`: support the AS1 `favorite` verb as well as `like`. ([bridgy1345](https://github.com/snarfed/bridgy/issues/1345))
* Atom:
* Switch to converting AS1 `id` (instead of `url`) to Atom `id`.
* Reddit:
* Implement `get_actor`.
* Mastodon:
* `create`/`preview`: allow non-Mastodon replies, ie activities that include `inReplyTo` URLs even if none of them point to a toot. ([bridgy1321](https://github.com/snarfed/bridgy/issues/1321))
* Raise `requests.HTTPError` with `response.status_code` 502 instead of `JSONDecodeError` on non-JSON responses. This is synthetic, but more helpful for error handling.
* microformats2:
* `object_to_json` and related functions: handle all escaped HTML entities, not just `&` `<` `>`.
* Unify `microformats2.prefix_image_urls` and `prefix_video_urls` into a new `as1.prefix_urls` function.
* RSS:
* Remove `itunes:category`. It has to be [one of Apple's explicit categories](https://feedgen.kiesow.be/ext/api.ext.podcast.html#feedgen.ext.podcast.PodcastExtension.itunes_category), which we aren't prepared to validate, so don't try.
* ActivityStreams 2:
* Translate both `url` and `urls` from AS1 into multi-valued AS2 `url` field.
* Move a number of utility methods from the `Source` class to a new `as1` module: `object_type`, `merge_by_id`, `is_public`, `add_rsvps_to_event`, `get_rsvps_from_event`, `activity_changed`, `append_in_reply_to`, `actor_name`, `original_post_discovery`.
* `as1.original_post_discovery`: remove deprecated `cache` kwarg.

Non-breaking changes

* ActivityStreams 2:
* Fix spec compliance bug: [`icon` and `image` are singly valued, not multiply valued](https://www.w3.org/TR/activitystreams-vocabulary/#dfn-icon).
* Add new `is_public` method and `PUBLIC_AUDIENCE` constant.
* Prefer `"objectType": "featured"` first in the `image` field when converting from AS1, last in the `icon` field. This matches the ActivityPub (Mastodon) convention of using `icon` for profile pictures and `image` for header images.
* Propagate `url` values into new `PropertyValue` attachments on `Person` objects; these end up in Mastodon's "profile metadata" link fields.
* `to_as1`: if an attachment's `mediaType` is `image/...`, override `objectType` and set it to `image`.
* Twitter
* Trim alt text in line between post preview and creation
* Correctly trim Twitter alt text
* Facebook
* Scraping: extract post id and owner id from `data-ft` attribute and `_ft_` query param more often instead of `story_fbid`, which is now an opaque token that changes regularly. ([facebook-atom27](https://github.com/snarfed/facebook-atom/issues/27))
* Instagram
* Add new `Instagram.scraped_json_to_activities` method.
* GitHub
* `create` and `preview`: convert profile URLs to -mentions, eg `https://github.com/snarfed` to `snarfed` ([bridgy#1090](https://github.com/snarfed/bridgy/issues/1090)).
* `get_activities` with `activity_id` now supports `fetch_replies` and `fetch_likes`.
* Reddit
* Add `cache` support to `get_activities`.
* REST API
* Add new `/scraped` endpoint that accepts `POST` requests with silo HTML as input. Currently only supports Instagram. Requires `site=instagram`, `output=...` (any supported output format), and HTML as either raw request body or MIME multipart encoded file in the `input` parameter.
* microformats2
* Add new `extra` and `body_class` kwargs to `activities_to_html`.
* When converting `u-featured` images to AS1, add new non-standard `"objectType": "featured"` field to distinguish them from `u-photo`.
* Convert `p-note` to AS1 `summary`.
* Bug fixes for converting `image` attachments to `photo`.
* `Source.original_post_discovery`: add new `max_redirect_fetches` keyword arg.

Breaking changes

* Drop Python 3.5 support. Python 3.6 is now the minimum required version.

Non-breaking changes

* Add new `include_shares` kwarg to `get_activities`, implemented for Twitter and Mastodon. Defaults to `True`. If `False`, shares (retweets in Twitter, boosts in Mastodon) will be discarded and not returned. Also add a corresponding `shares` query param to the REST API.
* RSS:
* Add support for RSS input via new `rss.to_activities` function.
* Instagram (scraping):
* Handle media items with no `user` object, add new fetch for comments.
* Add `Instagram.merge_scraped_comments()`.
* ActivityStreams 2:
* Handle error when `type` isn't a string.
* Reddit:
* Implement `get_activities()` to fetch posts by the current user or a user specified with `user_id`.
* Facebook scraping:
* Skip "Suggested for you" posts.
* Add `log_html` kwarg to `get_activities`; defaults to False.
* Miscellaneous bug fixes.
* JSONFeed:
* Handle malformed `items.author` element.

Notable changes
* `Source.original_post_discovery`: add new `include_reserved_hosts` kwarg, defaults to `True`.
* Facebook:
* Scraping: handle pictures, videos, link attachments, and text links in timeline/news feed posts.
* Mastodon:
* Bug fix for `get_activities()` with `fetch_mentions=True`: handle notifications with `status: null`. Maybe happens when a status is deleted?
* `create`/`preview_create`: support bookmarks. (Nothing special happens with them; their `content` is posted as a normal toot.)
* microformats2:
* Stop rendering `image.displayName` as visible text in HTML, since it's already in the `<img>`'s `alt` attribute.
* Add [`bookmark-of`](https://indieweb.org/bookmark#How_to_markup) support.
* Add `prefix_image_urls()` function.
* Handle null `content` in AS1/2 objects.
* `json_to_object` bug fix for composite `bookmark-of` properties.
* Twitter:
* `create`/`preview`: [support large videos](https://twittercommunity.com/t/large-file-can-not-be-finalized-synchronously/82929/3) via async upload. We now pass `media_category=tweet_video` to the chunked upload `INIT` stage, and then make blocking `STATUS` calls until the video is finished processing. ([bridgy#1043](https://github.com/snarfed/bridgy/issues/1043))
* `create`/`preview`: allow bookmarks. ([bridgy1045](https://github.com/snarfed/bridgy/issues/1045))
* `create`/`preview`: allow non-Twitter replies, ie activities that include `inReplyTo` URLs even if none of them point to a tweet. ([bridgy1063](https://github.com/snarfed/bridgy/issues/1063))
* `get_activities`: support list ids as well as slugs.
* Bug fixes for removing t.co links to quoted tweets.
* Bug fix for multiple instances of the same link in tweet text.
* `get_activities()`: raise `ValueError` on invalid `user_id`.
* REST API: ported web framework from webapp2 to Flask. No user-visible behavior change expected.

Notable changes

* Add Python 3.8 support, drop 3.3 and 3.4. Python 3.5 is now the minimum required version.
* Add [Pixelfed](https://pixelfed.org/)! Heavily based on Mastodon.
* Standardize Instagram's and Facebook's scraping into new common `scraped_to_activities()`, `scraped_to_activity()`, and `merge_scraped_reactions()` methods.
* Atom:
* Add the `summary` element ([157](https://github.com/snarfed/granary/issues/157)).
* REST API:
* Bug fix: URL-encode Unicode characters in `Link` HTTP headers (eg `rel=self`, `rel=header`).
* Facebook:
* Scraping now uses [mbasic.facebook.com](https://mbasic.facebook.com/) instead of [m.facebook.com](https://m.facebook.com/).
* Flickr:
* Add support for adding tags to existing photos ([bridgy857](https://github.com/snarfed/bridgy/issues/857)).
* `get_comment()`: skip fetching comments from API if `activity` kwarg is provided and contains the requested comment.
* GitHub:
* Handle [HTTP 451 Unavailable for Legal Reasons](https://en.wikipedia.org/wiki/HTTP_451) responses ([eg for DMCA takedowns](https://developer.github.com/changes/2016-03-17-the-451-status-code-is-now-supported/)) gracefully.
* Add create/preview support for reactions on pull review request comments (ie URLs with `discussion_r...` fragments).
* HTML/microformats2:
* Add `aria-hidden="true"` to empty links ([bridgy947](https://github.com/snarfed/bridgy/issues/947)).
* Bug fix: escape `&`, `<`, and `>` characters in bare mf2 `content` properties ([aaronpk/XRay102](https://github.com/aaronpk/XRay/issues/102)).
* `json_to_object()`: convert `nickname` to `username`.
* JSON Feed:
* Gracefully handle when `content_html` and `content_text` are [incorrectly](https://jsonfeed.org/version/1#items) lists instead of strings.
* Instagram:
* Include threaded (ie nested) comments in scraping ([bridgy958](https://github.com/snarfed/bridgy/issues/958)).
* Mastodon:
* Bug fix for alt text with image attachments ([bridgy975](https://github.com/snarfed/bridgy/issues/975)).
* Omit empty `limit` param [for compatibility with Pleroma](https://git.pleroma.social/pleroma/pleroma/-/issues/2198) ([bridgy#977](https://github.com/snarfed/bridgy/issues/977)).
* Meetup:
* `create()`: handle API errors and return the error message in the `CreationResult` ([bridgy921](https://github.com/snarfed/bridgy/issues/921)).
* Twitter:
* Bug fix: URL-encode list names in API calls.
* Bug fix: propagate alt text into AS1 `photo.displayName` so that it gets all the way into microformats2 JSON and HTML ([183](https://github.com/snarfed/granary/issues/183)).
* Reddit:
* Implement `post_id()`.
* Cache user data fetched from the API for 5m to avoid repeating user profile API requests ([bridgy1021](https://github.com/snarfed/bridgy/issues/1021)).
when fetching multiple comments or posts from the same author
* Bug fix: use 'displayName' instead of 'name' in AS1 objects for submissions.
* Bug fix: use tag URIs for activity ids.
* ActivityStreams 2:
* `to_as1()`: for `Create` activities, include the activity actor's data in the object's author ([snarfed/bridgy-fed75](https://github.com/snarfed/bridgy-fed/issues/75)).
* `to_as1()`: convert `preferredUsername` to `username`.
* `from_as1()`: convert `username` to `preferredUsername`.
* `from_as1()`: bug fix, make `context` kwarg actually work.

_Breaking changes:_

* _Python 2 is no longer supported!_ Including the [App Engine Standard Python 2 runtime](https://cloud.google.com/appengine/docs/standard/python/). On the plus side, the [Python 3 runtime](https://cloud.google.com/appengine/docs/standard/python3/) is now supported! See this [list of differences](https://cloud.google.com/appengine/docs/standard/python3/python-differences) for more details.

Non-breaking changes:

* Migrate demo app and API to the App Engine Standard Python 3 runtime.
* Instagram:
* Scraping: fetch 50 likes instead of 24. ([snarfed/bridgy898](https://github.com/snarfed/bridgy/issues/898))
* Scraping bug fix for `get_actor()` with `user_id`.
* Twitter:
* Add [image alt text](https://blog.twitter.com/developer/en_us/a/2016/alt-text-support-for-twitter-cards-and-the-rest-api.html) support to `get_activites()` etc ([#183](https://github.com/snarfed/granary/issues/183)).
* RSS:
* Add `itunes:image`, `itunes:author`, and `itunes:category`.
* Strip HTML from `title` element ([177](https://github.com/snarfed/granary/issues/177)). [Background.](https://validator.w3.org/feed/docs/warning/ContainsHTML.html)
* Always include author in items ([177](https://github.com/snarfed/granary/issues/177)).
* Bug fix: extract feed image from `hfeed` correctly.
* Bug fix: don't crash on `article` or `mention` tags in items with enclosures.
* Atom:
* Bug fix: extract feed image from `hfeed` correctly.
* REST API:
* Add HTTP `HEAD` support.
* Add support for URL fragments with `input=html`. If a fragment is provided, only that specific element is extracted and converted. ([185](https://github.com/snarfed/granary/issues/185))
* GitHub:
* Publish: preserve `<code>` tags instead of converting them to \`s so that GitHub renders HTML entities like `&gt;` inside them instead of leaving them escaped. [Background.](https://chat.indieweb.org/dev/2019-12-24#t1577174464779200)
* JSON Feed:
* Handle malformed attachments better.
* microformats2:
* Don't crash on string `context` fields.
* `html_to_activities()`: limit to `h-entry`, `h-event`, and `h-cite` items ([192](https://github.com/snarfed/granary/issues/192)).
* The `cache` kwarg to `Source.original_post_discovery()` now has no effect. `webutil.util.follow_redirects()` has its own built in caching now.
* Added Meetup.com support for publishing RSVPs.

Notable changes
* Add Mastodon support!
* Add Python 3.7 support, and improve overall Python 3 compatibility.
* Update a number of dependencies.
* Switch from Python's built in `json` module to [`ujson`](https://github.com/esnme/ultrajson/) to speed up JSON parsing and encoding.
* Add `duration` and `size` support to ActivityStreams 1 and 2, RSS, and microformats2 HTML and JSON. [microformats2 support is still emerging for both](https://indieweb.org/podcast#Brainstorming). Both integer seconds and [ISO 8601 string durations](https://en.wikipedia.org/wiki/ISO_8601#Durations) are supported for `duration`. Integer bytes is used for `size` everywhere. microformats2 HTML also includes human-readable strings, eg `5.1 MB`. ([169](https://github.com/snarfed/granary/issues/169))
* Twitter:
* `[preview]_create()`: detect attempts to upload [images over 5MB](https://developer.twitter.com/en/docs/media/upload-media/uploading-media/media-best-practices#image-specs) and return an error.
* Facebook:
* Add `get_activities(scrape=True)` for scraping HTML from [m.facebook.com](https://m.facebook.com/). Requires `c_user` and `xs` cookies from a logged in session ([snarfed/bridgy#886](https://github.com/snarfed/bridgy/issues/886)).
* [Upgrade Graph API version from 2.10 to 4.0.](https://developers.facebook.com/docs/graph-api/changelog)
* Atom:
* Bug fix for de-duping images in attachments.
* RSS:
* Wrap all `<description>` element contents in `CDATA` sections.
* Render images in `<description>` with HTML `<img>` tags ([175](https://github.com/snarfed/granary/issues/175)).
* `from_activities()` bug fix: don't crash when converting multiple attachments to enclosures in a single item. (RSS only supports one enclosure per item, so we now only include the first, and log a warning if the activity has more.)