Granary

_Breaking changes:_

* `jsonfeed`:
* `jsonfeed_to_activities`: return AS1 objects, not activities.

_Non-breaking changes:_

* `as1`:
* `activity_changed`: add `displayName`, `summary` fields.
* `is_public`: return `False` if the object/activity contains `to` that's empty or has only unknown aliases.
* `as2`:
* Add support for the `Application`, `Block`, `Flag`, and `Link` types.
* Generalize actor logic in `to/from_as1` across all actor types, not just `Person`.
* Add new `link_tags` function.
* `atom`:
* `activities_to_atom`: handle image attachments without `url` field.
* `bluesky`:
* `to_as1`:
* Add support for:
* `app.bsky.embed.record`
* `app.bsky.embed.recordWithMedia`
* `app.bsky.feed.defsnotFoundPost`
* `app.bsky.feed.generator`
* `app.bsky.graph.block`
* `app.bsky.graph.list`
* `app.bsky.graph.listitem`
* `com.atproto.admin.defsrepoRef`
* `com.atproto.moderation.createReportinput`
* `com.atproto.repo.strongRef`
* Add hashtag facet support.
* Convert blobs in embeds to `getBlob` image URLs.
* `app.bsky.actor.profile`: add HTML links for URLs in `summary` ([snarfed/bridgy-fed1065](https://github.com/snarfed/bridgy-fed/issues/1065)).
* Escape HTML characters (`<`, `>`, `&`) in `app.bsky.actor.profile` `description` field.
* Bug fix for `create`/`update` activities with bare string `object`.
* `from_as1`:
* Add hashtag, mention, block, and flag support. Interpret `tags` with missing `objectType` as hashtags.
* Guess missing indices in facets based on content text. Otherwise, if we still don't know a facet's indices, discard it.
* Extract HTML links (<a> tags) from HTML content and convert to link facets ([snarfed/bridgy-fed976](https://github.com/snarfed/bridgy-fed/issues/976)).
* If an output string value is longer than its ``maxGraphemes`` or ``maxLength`` in its lexicon, truncate it with an ``…`` ellipsis character at the end in order to fit. If this happens to post text, include a link embed pointing to the original post.
* If the object has a video, include an external embed pointing to the original post and mark it as `[Video]` ([snarfed/bridgy-fed1078](https://github.com/snarfed/bridgy-fed/issues/1078)).
* If the object has images, add the original post link to the end of the text, since Bluesky doesn't support both image and external embeds in the same post ([bluesky-social/atproto2575](https://github.com/bluesky-social/atproto/discussions/2575), [snarfed/bridgy-fed#1106](https://github.com/snarfed/bridgy-fed/issues/1106)).
* If a `note` has `summary` - often used for content warnings in the fediverse - add it to `content` as a prefix instead of overriding `content` ([snarfed/bridgy-fed1001](https://github.com/snarfed/bridgy-fed/issues/1001)).
* Populate `reply.root` properly in reply posts ([snarfed/bridgy1696](https://github.com/snarfed/bridgy/issues/1696)).
* Add new `original_fields_prefix` kwarg to store original data in custom (off-Lexicon) `*OriginalDescription` and `*OriginalUrl` fields in `app.bsky.actor.profile` and `*OriginalText` and `*OriginalUrl` fields in `app.bsky.feed.post` ([snarfed/bridgy-fed1092](https://github.com/snarfed/bridgy-fed/issues/1092)).
* Support `lexrpc.Client` as well as `Bluesky` for `client` kwarg.
* `from_as1_to_strong_ref`:
* Add `value` boolean kwarg.
* Change `client` kwarg from `Bluesky` to `lexrpc.Client`.
* `microformats2`:
* Generalize actor logic across all actor types, not just `person`.
* `json_to_object`:
* Strip leading `` prefix (if present) from hashtag `u-category`s.
* Bug fix for when `name` property is an object, eg an `h-card`.
* `object_to_json`:
* Convert both `id` and `url` inside `inReplyTo` to `in-reply-to.`
* `nostr`:
* Handle connection closing while sending initial query.
* `source`:
* `Source.postprocess`: when extracting -mentions, defer to existing tag if it has the same `displayName` and has `url`.

Notable changes

* `as1`:
* `get_owner` bug fix for `post`, `update`, `delete` activities.
* `activity_changed`: add new `inReplyTo` kwarg.
* `is_public`: add new `unlisted` kwarg.
* `as2`:
* `to_as1`: bug fix, preserve `objectType: featured` for banner/header images even when `mediaType` is also set.
* `is_public`: add new `unlisted` kwarg.
* `from_as1`:
* For `icon` field, prefer image types that are [allowed by Mastodon](https://github.com/mastodon/mastodon/blob/b4c332104a8b3748f619de250f77c0acc8e80628/app/models/concerns/account/avatar.rb#L6).
* Bug fix, handle `stop-following` with string `object` id.
* `atom`:
* Add new `extract_entries` function.
* `activity_to_atom`: default actor/author name to username.
* `atom_to_activities`: support top-level `entry` element as well as `feed`.
* `atom_to_*`:
* add `object.author`
* default `objectType` to `article`/`note` and `verb` to `post`
* convert `link rel=self`/`alternate` to `url`
* use `displayName` in objects instead of `title`
* Interpret entry `link` without `rel` as self link.
* If `entry.author` doesn't have id or url, default them to feed author's.
* `bluesky`:
* Implement `create` and `preview`.
* Fully support both `record` and `object` types in `from_as1` and `to_as1`. Use `to_as1`'s `type` kwarg and `from_as1`'s `out_type` kwarg to disambiguate.
* Implement `Bluesky.post_id`.
* Add new `blob_to_url` function.
* Delete `as1_to_profile`, switch `from_as1` to return `$type: app.bsky.actor.profile`.
* Convert HTML `summary` and `content` to plain text.
* Implement `Bluesky.user_to_actor`, `Bluesky.get_actor`.
* Don't log in (fetch an access token) eagerly in the constructor; wait until the client makes a call.
* Prefer DID to handle in API calls that accept either.
* `at_uri_to_web_url`: support lists.
* `web_url_to_at_uri`: convert profile URLs like `https://bsky.app/profile/snarfed.org` to profile record URIs (`at://snarfed.org/app.bsky.actor.profile/self`) instead of repo URIs (`at://snarfed.org`).
* Add `from_as1_to_strong_ref`.
* Allow `:`s in record keys ([atproto2224](https://github.com/bluesky-social/atproto/discussions/2224)).
* `to_as1`:
* Convert blobs, [both new and old style](https://atproto.com/specs/data-model#blob-type), to PDS `getBlob` URLs.
* Add new `uri` kwarg.
* Translate `handle` to `username`, add new `repo_handle` kwarg.
* Add support for `app.bsky.feed.repost`, `app.bsky.graph.defslistView`, `app.bsky.feed.defsblockedPost`.
* Add `actor`/`author` based on `repo_did`.
* Improve `url` field: include custom handles, only use `repo_did/handle` for `app.bsky.actor.profile`.
* Handle bad facet indices that point inside Unicode code points ([example](https://bsky.app/profile/did:plc:2ythpj4pwwpka2ljkabouubm/post/3kkfszbaiic2g); [discussion](https://discord.com/channels/1097580399187738645/1097580399187738648/1203118842516082848)).
* Convert [`!no-unauthenticated` label](https://github.com/bluesky-social/atproto/blob/main/packages/api/docs/labels.md#label-behaviors) on profiles to [AS1 `unlisted` audience target](https://activitystrea.ms/specs/json/targeting/1.0/) ([bridgy-fed#828](https://github.com/snarfed/bridgy-fed/issues/828)).
* `from_as1`:
* Add `out_type` kwarg to specify desired output type, eg `app.bsky.actor.profile` vs `app.bsky.actor.defsprofileViewBasic` vs `app.bsky.actor.defsprofileView`.
* Add `blobs` kwarg to provide blob objects to use for image URLs.
* Add `client` kwarg to fetch and populate CIDs.
* Handle mention tags pointing to bare DIDs.
* Use `parent` as `root` in replies. (Technically wrong in cases where the parent isn't the root, but we don't actually know the root. 🤷)
* Bug fix: handle bare string URLs in `image` field.
* Bug fix: handle tags without `url` field.
* Strip trailing slash from home page URLs in order to remove visible `/` from rel-me verified links on Mastodon etc.
* Convert `attributedTo` to singular if it has only one element.
* If `name` isn't set, fall back to `preferredUsername` or infer Webfinger handle from `id` or `url`.
* Prioritize bsky.app profile URL before handle URL in `url` field ([bridgy1640](https://github.com/snarfed/bridgy/issues/1640)).
* Convert `bsky.app` `inReplyTo` URLs to `at://` URIs.
* Tighten up `datetime` conversion to match the [ATProto recommended format](https://atproto.com/specs/lexicon#datetime).
* `facebook`:
* Remove `Facebook.fql_stream_to_post`. [Facebook turned down FQL in 2016.](https://en.wikipedia.org/wiki/Facebook_Query_Language#History)
* `github`:
* When converting data to AS1, use `displayName` in objects instead of `title`.
* `mastodon`:
* `get_activities` bug fix: use query params for `/api/v1/notifications` API call, not JSON body.
* Convert HTTP 200 responses with `error` JSON field (eg from Sharkey) to 400/401 exceptions.
* Prefer `media_attachments.remote_url` when available since it may be more long-lived than `url` for remote statuses ([bridgy1675](https://github.com/snarfed/bridgy/issues/1675)).
* `microformats2`:
* `object_to_json` bug fix: handle singular `inReplyTo`.
* `json_to_object` bug fix: handle list-valued `location`.
* `nostr:`
* `get_*`: return partial results when the websocket connection is closed prematurely.
* `to_as1`: handle invalid NIP05 values (eg `{}`)
* `rss`:
* `to_activities`:
* Use `objectType: note` if `title` isn't set or is a prefix (possibly ellipsized) of `content`/`description`.
* Add support for images in `media:content` tags ([674](https://github.com/snarfed/granary/issues/674)).
* `Source`:
* `postprocess_activity/object`: add `mentions` kwarg to convert -mentions in HTML links to `mention` tags.

Notable changes

Nostr, Bluesky `get_activities`, lots of improvements in `as2` and `microformats2`, and more!

_REST API breaking changes:_

[Twitter is dead](https://snarfed.org/2023-04-03_so-long-twitter-api-and-thanks-for-all-the-fish), at least in the REST API.

_Non-breaking changes:_

* Add new `nostr` module!
* `as1`:
* Add `get_owner`, `targets`.
* Add `accept`, `reject`, `stop-following` to `VERBS_WITH_OBJECT` and remove `repost`, [it's not an AS1 verb](https://activitystrea.ms/specs/json/schema/activity-schema.html#verbs).
* Handle `url` field list values (even though it's invalid AS1).
* `as2`:
* `to_as1`:
* Improve `Video` handling: support `Link` objects in `url`, extract stream URLs and types from link `tag`s.
* Coerce non-float `latitude` and `longitude` to float, raise `ValueError` on failure.
* Put image attachments into `image` as well as `attachments` ([bridgy-fed429](https://github.com/snarfed/bridgy-fed/issues/429)).
* Handle Hubzilla's composite object attachment `value`s.
* Bug fix for null `mediaType` in `attachment` and `tags`.
* Add new `TYPES_WITH_OBJECT` constant.
* Add new `get_urls`, `address` functions.
* Improve `Content-Type` compatibility with `application/ld+json; profile="https://www.w3.org/ns/activitystreams"`.
* Bug fix for `Undo` activities with bare string id `object`s.
* Revise HTML in `PropertyValue` attachments on actors to include full URL in anchro text to be compatible with Mastodon's profile link verification.
* `atom`:
* `activities_to_atom` etc:
* Switch `content` from XHTML to HTML inside CDATA to support non-XHTML input content ([bridgy-fed624](https://github.com/snarfed/bridgy-fed/issues/624).
* Bug fix, handle bare string URL `image` values.
* Bug fix, emove incorrect `type="application/atom+xml"` from `rel="self"` `link` in `entry`.
* Render `objectType: comment` attachments.
* Remove invalid `<a>` element for tags.
* Bug fixes in `activity_to_atom`/`activities_to_atom` for dict-valued `url` fields.
* Render images in article/note attachments.
* Render `objectType: service` attachments, eg Bluesky custom feeds.
* `bluesky`:
* Implement `Bluesky` API class, including `get_activities`.
* Drop bundled `app.bsky`/`com.atproto` lexicons, use lexrpc's instead.
* Convert reposts, quotes, inline links, attached links, and mentions, both directions. Includes Bluesky facet (rich text) support.
* Handle quote posts with attached images, both directions.
* Handle likes, both directions.
* Add new `web_url_to_at_uri` function.
* `from_as1`: handle link tags without start/end indices.
* `to_as1`:
* Add new `type` kwarg.
* Generate staging.bsky.app profile and post URLs.
* Propagate profile `did` into actor `id`.
* Add unimplemented stub for custom feeds, eg `app.bsky.feed.defsgeneratorView`.
* Add `as1_to_profile`.
* Bug fix for converting follows, both directions: `subject` in `app.bsky.graph.follow` is followee, not follower. ([That field is badly named!](https://discord.com/channels/1097580399187738645/1097580399187738648/1151933384738746478))
* `jsonfeed`:
* `activities_to_jsonfeed`:
* Bug fix, handle bare string values for `image` and `stream`.
* Bug fix: handle non-object `author`.
* `mastodon`:
* `status_to_object`: add/fix alt text handling for images.
* `microformats2`:
* `json_to_html`:
* HTML-escape tag and quote attachment names. Fixes [GHSA-4w4f-g49g-3f7j](https://github.com/snarfed/bridgy/security/advisories/GHSA-4w4f-g49g-3f7j); thank you [janboddez](https://github.com/janboddez)!
* `json_to_object`:
* Improve handling of items with multiple types by using [post type discovery](https://indiewebcamp.com/post-type-discovery) more aggressively.
* Normalize ISO-8601 format of `published` and `updated` timestamps.
* `object_to_json`:
* Bug fix, handle bare string URL `image` values.
* Normalize ISO-8601 format of `published` and `updated` timestamps.
* Handle bare string ids for `replies ` and `shares` (usually from AS2.)
* Include `objectType: service` attachments, eg Bluesky custom feeds, in JSON and HTML output.
* `rss`:
* `from_activities`: handle bare string id `author`.

Breaking changes

* `as2`:
* Interpret bare string `object`, `inReplyTo`, etc values as ids, convert them to bare strings or `id` instead of `url`.
* `microformats2`:
* Convert simple string `in-reply-to`, `repost-of`, `like-of` etc values to AS1 bare strings or `id`s instead of `url`s.

Non-breaking changes

* Add new `bluesky` module for [Bluesky](https://blueskyweb.org/)/[AT Protocol](https://atproto.com/)!
* `as1`:
* Add the `organization` object type and `ACTOR_TYPES` constant ([based on AS2](https://www.w3.org/TR/activitystreams-core/#actors)).
* Add new `get_ids`, `get_object`, and `get_objects` functions.
* `activity_changed`: ignore `inReplyTo.author` ([snarfed/bridgy1338](https://github.com/snarfed/bridgy/issues/1338))
* `as2`:
* Support converting between AS1 `stop-following` and AS2 `Undo` `Follow`.
* Support AS2 `Accept` and `Reject` for follows as well as event RSVPs.
* Add support for the `Question` (ie poll), `Organization`, and `Delete` object types.
* Convert `to`/`cc` to/from AS1 `to` for public and unlisted.
* Handle `type: Document` video attachments like Mastodon emits.
* `from_as1`: bug fix for image objects with `url` and `value` fields (for alt text).
* `from_as1`: bug fix, handle bare string URL `image` values.
* `from_as1`: convert `urls.displayName` to `attachment.name` ([bridgy-fed331](https://github.com/snarfed/bridgy-fed/issues/331)).
* `from_as1`: preserve `inReplyTo` object values as objects, inline single-element lists down down to just single element.
* `to_as1`: use `objectType: featured` for first image in `image` field.
* `to_as1`: populate `actor` into `object.author` for `Update`s as well as `Create`s.
* `to_as1`: convert Mastodon profile metadata `PropertyValue` attachments to `url` composite objects with `displayName`.
* Preserve `to` and `cc` values when converting both directions.
* `atom`:
* Bug fix for rendering image attachments without `image` field to Atom.
* Bug fix for `published` and `updated` in entries with objects, eg likes, reposts, RSVPs, bookmarks. Thanks [gregorlove](https://gregorlove.com/)! ([#480](https://github.com/snarfed/granary/issues/480))
* Bug fix for content `activity/ies_to_atom` when `object` is present and empty.
* Bug fix for objects with elements without `objectType` in the `to` field.
* `flickr`:
* `get_activities`: add support for the `count` kwarg.
* `github`:
* `get_activities`: add support for the `count` kwarg.
* `jsonfeed`:
* Switch from `white-space: pre` CSS to converting newlines to `<br>`s because some feed readers follow it strictly and don't even line wrap ([456](https://github.com/snarfed/granary/issues/456)).
* `mastodon`:
* Add compatibility support for [Truth Social](https://truthsocial.com/).
* Handle truncated JSON API responses.
* `microformats2`:
* `json_to_object`: drop backward compatibility support for `like` and `repost` properties. [Background discussion.](https://chat.indieweb.org/dev/2022-12-23#t1671833687984200)
* `json_to_object`: add new `rel_urls` kwarg to allow attaching `displayName`s to `urls` based on HTML text or `title` attribute ([bridgy-fed331](https://github.com/snarfed/bridgy-fed/issues/331)).
* Add new `json_to_activities` function.
* `hcard_to_html`/`maybe_linked_name`: when `name` is missing, use pretty URL as visible text.
* Support the `h-card` `org` property.
* `json_to_object`: handle composite `rsvp` property value.
* `json_to_object`: bug fix when `fetch_mf2` is True, handle when we run the authorship algorithm and fetch an author URL that has a `u-photo` with `alt`.
* `rss`:
* `from_activities`: fix item ordering to match input activities.

Breaking changes

* Drop Python 3.6 support. Python 3.7 is now the minimum required version.
* Twitter, Instagram, Mastodon:
* Drop `get_activities` `cache` kwarg's support for App Engine memcache interface. It's now only used as a plain `dict`. `get_activities` will now make many small modifications, so if you pass an object that implements those as API calls, you'll probably want to batch those separately.
* Twitter, Mastodon, Flickr, GitHub:
* `create`/`preview`: support the AS1 `favorite` verb as well as `like`. ([bridgy1345](https://github.com/snarfed/bridgy/issues/1345))
* Atom:
* Switch to converting AS1 `id` (instead of `url`) to Atom `id`.
* Reddit:
* Implement `get_actor`.
* Mastodon:
* `create`/`preview`: allow non-Mastodon replies, ie activities that include `inReplyTo` URLs even if none of them point to a toot. ([bridgy1321](https://github.com/snarfed/bridgy/issues/1321))
* Raise `requests.HTTPError` with `response.status_code` 502 instead of `JSONDecodeError` on non-JSON responses. This is synthetic, but more helpful for error handling.
* microformats2:
* `object_to_json` and related functions: handle all escaped HTML entities, not just `&amp;` `&lt;` `&gt;`.
* Unify `microformats2.prefix_image_urls` and `prefix_video_urls` into a new `as1.prefix_urls` function.
* RSS:
* Remove `itunes:category`. It has to be [one of Apple's explicit categories](https://feedgen.kiesow.be/ext/api.ext.podcast.html#feedgen.ext.podcast.PodcastExtension.itunes_category), which we aren't prepared to validate, so don't try.
* ActivityStreams 2:
* Translate both `url` and `urls` from AS1 into multi-valued AS2 `url` field.
* Move a number of utility methods from the `Source` class to a new `as1` module: `object_type`, `merge_by_id`, `is_public`, `add_rsvps_to_event`, `get_rsvps_from_event`, `activity_changed`, `append_in_reply_to`, `actor_name`, `original_post_discovery`.
* `as1.original_post_discovery`: remove deprecated `cache` kwarg.

Non-breaking changes

* ActivityStreams 2:
* Fix spec compliance bug: [`icon` and `image` are singly valued, not multiply valued](https://www.w3.org/TR/activitystreams-vocabulary/#dfn-icon).
* Add new `is_public` method and `PUBLIC_AUDIENCE` constant.
* Prefer `"objectType": "featured"` first in the `image` field when converting from AS1, last in the `icon` field. This matches the ActivityPub (Mastodon) convention of using `icon` for profile pictures and `image` for header images.
* Propagate `url` values into new `PropertyValue` attachments on `Person` objects; these end up in Mastodon's "profile metadata" link fields.
* `to_as1`: if an attachment's `mediaType` is `image/...`, override `objectType` and set it to `image`.
* Twitter
* Trim alt text in line between post preview and creation
* Correctly trim Twitter alt text
* Facebook
* Scraping: extract post id and owner id from `data-ft` attribute and `_ft_` query param more often instead of `story_fbid`, which is now an opaque token that changes regularly. ([facebook-atom27](https://github.com/snarfed/facebook-atom/issues/27))
* Instagram
* Add new `Instagram.scraped_json_to_activities` method.
* GitHub
* `create` and `preview`: convert profile URLs to -mentions, eg `https://github.com/snarfed` to `snarfed` ([bridgy#1090](https://github.com/snarfed/bridgy/issues/1090)).
* `get_activities` with `activity_id` now supports `fetch_replies` and `fetch_likes`.
* Reddit
* Add `cache` support to `get_activities`.
* REST API
* Add new `/scraped` endpoint that accepts `POST` requests with silo HTML as input. Currently only supports Instagram. Requires `site=instagram`, `output=...` (any supported output format), and HTML as either raw request body or MIME multipart encoded file in the `input` parameter.
* microformats2
* Add new `extra` and `body_class` kwargs to `activities_to_html`.
* When converting `u-featured` images to AS1, add new non-standard `"objectType": "featured"` field to distinguish them from `u-photo`.
* Convert `p-note` to AS1 `summary`.
* Bug fixes for converting `image` attachments to `photo`.
* `Source.original_post_discovery`: add new `max_redirect_fetches` keyword arg.

Breaking changes

* Drop Python 3.5 support. Python 3.6 is now the minimum required version.

Non-breaking changes

* Add new `include_shares` kwarg to `get_activities`, implemented for Twitter and Mastodon. Defaults to `True`. If `False`, shares (retweets in Twitter, boosts in Mastodon) will be discarded and not returned. Also add a corresponding `shares` query param to the REST API.
* RSS:
* Add support for RSS input via new `rss.to_activities` function.
* Instagram (scraping):
* Handle media items with no `user` object, add new fetch for comments.
* Add `Instagram.merge_scraped_comments()`.
* ActivityStreams 2:
* Handle error when `type` isn't a string.
* Reddit:
* Implement `get_activities()` to fetch posts by the current user or a user specified with `user_id`.
* Facebook scraping:
* Skip "Suggested for you" posts.
* Add `log_html` kwarg to `get_activities`; defaults to False.
* Miscellaneous bug fixes.
* JSONFeed:
* Handle malformed `items.author` element.