Granary

Notable changes

* `as2`:
* Add new `set_content` function to help keep `content` and `contentMap` in sync.
* `to_as1`: support integer seconds `duration`, which is non-standard but sent by some AP implementations, [eg Funkwhale](https://dev.funkwhale.audio/funkwhale/funkwhale/-/issues/1566).
* `link_tags`: add `class="hashtag"` for hashtag (`Tag`, `Hashtag`) tags ([bridgy-fed/1634](https://github.com/snarfed/bridgy-fed/issues/1634#issuecomment-2577519871)).
* `bluesky`:
* Translate Bluesky `app.bsky.feed.posttags` to/from AS1 `tags` ([snarfed/bridgy-fed1394](https://github.com/snarfed/bridgy-fed/issues/1394)).
* Add `auth` kwarg to `Bluesky` constructor to pass through as custom auth object to `requests.get`/`post`.
* `from_as1`:
* Bug fix for generating external embeds: convert HTML `content`/`summary` to plain text `description` ([bridgy-fed1615](https://github.com/snarfed/bridgy-fed/issues/1615)).
* Don't add `app.bsky.feed.posttags` that are over `maxGraphemes` (64).
* Add new `raise_` kwarg to raise `ValueError` if a required object (eg the target of a like or repost) can't be fetched via ATProto.
* Bug fix: ignore `inReplyTo` for DMs.
* Don't convert HTML links in `content` with bad URLs to `link` facets.
* `to_as1`:
* Bug fix: HTML-escape `<` and `>` characters, while preserving facet indices, so that they don't disappear ([snarfed/bridgy-fed1144](https://github.com/snarfed/bridgy-fed/issues/1144)).
* `preview`/`create`:
* Add support for follows.
* `to_external_embed`: bug fix: handle composite `url` field.
* `mastodon`:
* `preview`/`create`:
* Add support for follows.
* `nostr`:
* Add new `Nostr.delete` method.
* Add new `sign` function.
* `source`:
* `Source.postprocess_object`: relax mention text matching with `mentions=True`, ignore server part of webfinger addresses.
* Add new `get_follows` and `get_followers` methods, implement in Mastodon and Bluesky.

_Breaking changes:_

* `as2`:
* `from_as1`: In `Link` objects (including `Tag`s and `Mention`s), convert `url` to `href`. Before this, we left it as `url`, which was incorrect AS2.

_Non-breaking changes:_

Standardize function and method names in all modules to `to_as1`, `from_as`, etc. Old method names are now deprecated but won't be removed until at least v9.0, if not later.

* `as1`:
* Add new `is_dm`, `recipient_if_dm`, `get_id`, and `is_audience` functions.
* `as2`:
* Add [`sensitive`](https://swicg.github.io/miscellany/#sensitive), [`indexable`](https://codeberg.org/fediverse/fep/src/branch/main/fep/5feb/fep-5feb.md#specifying-search-indexing-consent-at-the-actor-level), and [`discoverable`](https://docs.joinmastodon.org/spec/activitypub/#discoverable) support.
* Add new `is_server_actor` function ([FEP-d556](https://codeberg.org/fediverse/fep/src/branch/main/fep/d556/fep-d556.md), [discussion](https://socialhub.activitypub.rocks/t/fep-d556-server-level-actor-discovery-using-webfinger/3861)).
* `from_as1`:
* Always convert images to objects with `type: Image`, never to bare string URLs ([bridgy-fed/1000](https://github.com/snarfed/bridgy-fed/issues/1000)).
* Bug fixes for converting links to facets when the link text is the link URL.
* `to_as1`:
* Handle other types of tags better, eg non-standard `Hashtag` and inner `tag` field for name.
* Bug fix for videos, `mimeType` goes in outer object, not in `stream`.
* Bug fix for `to`/`cc` with mixed dict and string elements.
* `link_tags`: add `class="mention"` for `Mention` tags ([bridgy-fed/887](https://github.com/snarfed/bridgy-fed/issues/887#issuecomment-2452141758)).
* `atom`:
* `atom_to_activity/ies`: Get URL from `link` for activities as well as objects. ([Thanks imax9000!](https://github.com/snarfed/granary/issues/752))
* `bluesky`:
* Translate Bluesky `app.bsky.feed.postlangs` to/from AS1 `contentMap` (which isn't officially part of AS1; we steal it from AS2).
* Translate AS2 `sensitive` on posts to Bluesky `graphic-media` self label, and many Bluesky self labels back to `sensitive` with content warning(s) in `summary`.
* Translate AS1/2 DMs to/from Bluesky chats.
* Translate video embeds in posts.
* `create`/`previewCreate`:
* If `inReplyTo` isn't a Bluesky URL or AT URI, return `CreationResult` instead of raising `ValueError`.
* `from_as1`:
* Convert `article`s to external embeds with no post text.
* Add new `as_embed` boolean kwarg to do the same thing for any object.
* When truncating and adding a link to the original post, use `id` if `url` is not available ([snarfed/bridgy-fed1155](https://github.com/snarfed/bridgy-fed/issues/1155)).
* If the input object has `inReplyTo` or `object` or `target` with no recognizable ATProto or Bluesky object, raise `ValueError`.
* Omit images that aren't in `blobs`.
* Bug fix for quote posts with text content that's longer than Bluesky's limit ([snarfed/bridgy-fed1197](https://github.com/snarfed/bridgy-fed/issues/1197)).
* When a `flag` has multiple objects, use the first one that's an ATProto record.
* Handle URLs more carefully, don't add link facets with invalid `uri`s.
* Populate `blobs` into external embed `thumb`s.
* Parse image blobs and add `aspectRatio` to image record.
* Bug fix: handle HTML links with `title` in `content` correctly.
* Bug fix: handle attachments with no `id` or `url`.
* `to_as1`:
* Extract links from `app.bsky.actor.profiledescription` and `summary` into `url`/`urls` fields.
* Bug fix: first URL (singular) goes in `url`, list of URLs goes in `urls`.
* Bug fix: handle hashtags with regexp special characters.
* Support string and bytes CIDs in blob `ref`s as well as `CID` instances.
* Link hashtags to bsky.app hashtag search pages ([bridgy-fed1634](https://github.com/snarfed/bridgy-fed/issues/1634)).
* `Bluesky.get_activities`: skip unknown record types instead of raising `ValueError`.
* `microformats2`:
* `object_to_json`: Improve handling of items with multiple types by removing `inReplyTo` from likes, shares, etc ([snarfed/bridgy-fed941](https://github.com/snarfed/bridgy-fed/issues/941)).
* `to_as1`: don't crash on integer UNIX timestamps in `published` and `updated`.
* `rss`:
* Support image enclosures, both directions.
* `from_as1`:
* Bug fix: remove use of default `author` value `'-'` since RSS spec requires author values to include valid email addresses.
* `source`:
* `Source.postprocess_object`: add new `first_link_to_attachment` boolean kwarg to fetch and generate a preview `attachment` for the first link in the HTML `content`, if any.

_Breaking changes:_

* `jsonfeed`:
* `jsonfeed_to_activities`: return AS1 objects, not activities.

_Non-breaking changes:_

* `as1`:
* `activity_changed`: add `displayName`, `summary` fields.
* `is_public`: return `False` if the object/activity contains `to` that's empty or has only unknown aliases.
* `as2`:
* Add support for the `Application`, `Block`, `Flag`, and `Link` types.
* Generalize actor logic in `to/from_as1` across all actor types, not just `Person`.
* Add new `link_tags` function.
* `atom`:
* `activities_to_atom`: handle image attachments without `url` field.
* `bluesky`:
* `to_as1`:
* Add support for:
* `app.bsky.embed.record`
* `app.bsky.embed.recordWithMedia`
* `app.bsky.feed.defsnotFoundPost`
* `app.bsky.feed.generator`
* `app.bsky.graph.block`
* `app.bsky.graph.list`
* `app.bsky.graph.listitem`
* `com.atproto.admin.defsrepoRef`
* `com.atproto.moderation.createReportinput`
* `com.atproto.repo.strongRef`
* Add hashtag facet support.
* Convert blobs in embeds to `getBlob` image URLs.
* `app.bsky.actor.profile`: add HTML links for URLs in `summary` ([snarfed/bridgy-fed1065](https://github.com/snarfed/bridgy-fed/issues/1065)).
* Escape HTML characters (`<`, `>`, `&`) in `app.bsky.actor.profile` `description` field.
* Bug fix for `create`/`update` activities with bare string `object`.
* `from_as1`:
* Add hashtag, mention, block, and flag support. Interpret `tags` with missing `objectType` as hashtags.
* Guess missing indices in facets based on content text. Otherwise, if we still don't know a facet's indices, discard it.
* Extract HTML links (<a> tags) from HTML content and convert to link facets ([snarfed/bridgy-fed976](https://github.com/snarfed/bridgy-fed/issues/976)).
* If an output string value is longer than its ``maxGraphemes`` or ``maxLength`` in its lexicon, truncate it with an ``…`` ellipsis character at the end in order to fit. If this happens to post text, include a link embed pointing to the original post.
* If the object has a video, include an external embed pointing to the original post and mark it as `[Video]` ([snarfed/bridgy-fed1078](https://github.com/snarfed/bridgy-fed/issues/1078)).
* If the object has images, add the original post link to the end of the text, since Bluesky doesn't support both image and external embeds in the same post ([bluesky-social/atproto2575](https://github.com/bluesky-social/atproto/discussions/2575), [snarfed/bridgy-fed#1106](https://github.com/snarfed/bridgy-fed/issues/1106)).
* If a `note` has `summary` - often used for content warnings in the fediverse - add it to `content` as a prefix instead of overriding `content` ([snarfed/bridgy-fed1001](https://github.com/snarfed/bridgy-fed/issues/1001)).
* Populate `reply.root` properly in reply posts ([snarfed/bridgy1696](https://github.com/snarfed/bridgy/issues/1696)).
* Add new `original_fields_prefix` kwarg to store original data in custom (off-Lexicon) `*OriginalDescription` and `*OriginalUrl` fields in `app.bsky.actor.profile` and `*OriginalText` and `*OriginalUrl` fields in `app.bsky.feed.post` ([snarfed/bridgy-fed1092](https://github.com/snarfed/bridgy-fed/issues/1092)).
* Support `lexrpc.Client` as well as `Bluesky` for `client` kwarg.
* `from_as1_to_strong_ref`:
* Add `value` boolean kwarg.
* Change `client` kwarg from `Bluesky` to `lexrpc.Client`.
* `microformats2`:
* Generalize actor logic across all actor types, not just `person`.
* `json_to_object`:
* Strip leading `` prefix (if present) from hashtag `u-category`s.
* Bug fix for when `name` property is an object, eg an `h-card`.
* `object_to_json`:
* Convert both `id` and `url` inside `inReplyTo` to `in-reply-to.`
* `nostr`:
* Handle connection closing while sending initial query.
* `source`:
* `Source.postprocess`: when extracting -mentions, defer to existing tag if it has the same `displayName` and has `url`.

Notable changes

* `as1`:
* `get_owner` bug fix for `post`, `update`, `delete` activities.
* `activity_changed`: add new `inReplyTo` kwarg.
* `is_public`: add new `unlisted` kwarg.
* `as2`:
* `to_as1`: bug fix, preserve `objectType: featured` for banner/header images even when `mediaType` is also set.
* `is_public`: add new `unlisted` kwarg.
* `from_as1`:
* For `icon` field, prefer image types that are [allowed by Mastodon](https://github.com/mastodon/mastodon/blob/b4c332104a8b3748f619de250f77c0acc8e80628/app/models/concerns/account/avatar.rb#L6).
* Bug fix, handle `stop-following` with string `object` id.
* `atom`:
* Add new `extract_entries` function.
* `activity_to_atom`: default actor/author name to username.
* `atom_to_activities`: support top-level `entry` element as well as `feed`.
* `atom_to_*`:
* add `object.author`
* default `objectType` to `article`/`note` and `verb` to `post`
* convert `link rel=self`/`alternate` to `url`
* use `displayName` in objects instead of `title`
* Interpret entry `link` without `rel` as self link.
* If `entry.author` doesn't have id or url, default them to feed author's.
* `bluesky`:
* Implement `create` and `preview`.
* Fully support both `record` and `object` types in `from_as1` and `to_as1`. Use `to_as1`'s `type` kwarg and `from_as1`'s `out_type` kwarg to disambiguate.
* Implement `Bluesky.post_id`.
* Add new `blob_to_url` function.
* Delete `as1_to_profile`, switch `from_as1` to return `$type: app.bsky.actor.profile`.
* Convert HTML `summary` and `content` to plain text.
* Implement `Bluesky.user_to_actor`, `Bluesky.get_actor`.
* Don't log in (fetch an access token) eagerly in the constructor; wait until the client makes a call.
* Prefer DID to handle in API calls that accept either.
* `at_uri_to_web_url`: support lists.
* `web_url_to_at_uri`: convert profile URLs like `https://bsky.app/profile/snarfed.org` to profile record URIs (`at://snarfed.org/app.bsky.actor.profile/self`) instead of repo URIs (`at://snarfed.org`).
* Add `from_as1_to_strong_ref`.
* Allow `:`s in record keys ([atproto2224](https://github.com/bluesky-social/atproto/discussions/2224)).
* `to_as1`:
* Convert blobs, [both new and old style](https://atproto.com/specs/data-model#blob-type), to PDS `getBlob` URLs.
* Add new `uri` kwarg.
* Translate `handle` to `username`, add new `repo_handle` kwarg.
* Add support for `app.bsky.feed.repost`, `app.bsky.graph.defslistView`, `app.bsky.feed.defsblockedPost`.
* Add `actor`/`author` based on `repo_did`.
* Improve `url` field: include custom handles, only use `repo_did/handle` for `app.bsky.actor.profile`.
* Handle bad facet indices that point inside Unicode code points ([example](https://bsky.app/profile/did:plc:2ythpj4pwwpka2ljkabouubm/post/3kkfszbaiic2g); [discussion](https://discord.com/channels/1097580399187738645/1097580399187738648/1203118842516082848)).
* Convert [`!no-unauthenticated` label](https://github.com/bluesky-social/atproto/blob/main/packages/api/docs/labels.md#label-behaviors) on profiles to [AS1 `unlisted` audience target](https://activitystrea.ms/specs/json/targeting/1.0/) ([bridgy-fed#828](https://github.com/snarfed/bridgy-fed/issues/828)).
* `from_as1`:
* Add `out_type` kwarg to specify desired output type, eg `app.bsky.actor.profile` vs `app.bsky.actor.defsprofileViewBasic` vs `app.bsky.actor.defsprofileView`.
* Add `blobs` kwarg to provide blob objects to use for image URLs.
* Add `client` kwarg to fetch and populate CIDs.
* Handle mention tags pointing to bare DIDs.
* Use `parent` as `root` in replies. (Technically wrong in cases where the parent isn't the root, but we don't actually know the root. 🤷)
* Bug fix: handle bare string URLs in `image` field.
* Bug fix: handle tags without `url` field.
* Strip trailing slash from home page URLs in order to remove visible `/` from rel-me verified links on Mastodon etc.
* Convert `attributedTo` to singular if it has only one element.
* If `name` isn't set, fall back to `preferredUsername` or infer Webfinger handle from `id` or `url`.
* Prioritize bsky.app profile URL before handle URL in `url` field ([bridgy1640](https://github.com/snarfed/bridgy/issues/1640)).
* Convert `bsky.app` `inReplyTo` URLs to `at://` URIs.
* Tighten up `datetime` conversion to match the [ATProto recommended format](https://atproto.com/specs/lexicon#datetime).
* `facebook`:
* Remove `Facebook.fql_stream_to_post`. [Facebook turned down FQL in 2016.](https://en.wikipedia.org/wiki/Facebook_Query_Language#History)
* `github`:
* When converting data to AS1, use `displayName` in objects instead of `title`.
* `mastodon`:
* `get_activities` bug fix: use query params for `/api/v1/notifications` API call, not JSON body.
* Convert HTTP 200 responses with `error` JSON field (eg from Sharkey) to 400/401 exceptions.
* Prefer `media_attachments.remote_url` when available since it may be more long-lived than `url` for remote statuses ([bridgy1675](https://github.com/snarfed/bridgy/issues/1675)).
* `microformats2`:
* `object_to_json` bug fix: handle singular `inReplyTo`.
* `json_to_object` bug fix: handle list-valued `location`.
* `nostr:`
* `get_*`: return partial results when the websocket connection is closed prematurely.
* `to_as1`: handle invalid NIP05 values (eg `{}`)
* `rss`:
* `to_activities`:
* Use `objectType: note` if `title` isn't set or is a prefix (possibly ellipsized) of `content`/`description`.
* Add support for images in `media:content` tags ([674](https://github.com/snarfed/granary/issues/674)).
* `Source`:
* `postprocess_activity/object`: add `mentions` kwarg to convert -mentions in HTML links to `mention` tags.

Notable changes

Nostr, Bluesky `get_activities`, lots of improvements in `as2` and `microformats2`, and more!

_REST API breaking changes:_

[Twitter is dead](https://snarfed.org/2023-04-03_so-long-twitter-api-and-thanks-for-all-the-fish), at least in the REST API.

_Non-breaking changes:_

* Add new `nostr` module!
* `as1`:
* Add `get_owner`, `targets`.
* Add `accept`, `reject`, `stop-following` to `VERBS_WITH_OBJECT` and remove `repost`, [it's not an AS1 verb](https://activitystrea.ms/specs/json/schema/activity-schema.html#verbs).
* Handle `url` field list values (even though it's invalid AS1).
* `as2`:
* `to_as1`:
* Improve `Video` handling: support `Link` objects in `url`, extract stream URLs and types from link `tag`s.
* Coerce non-float `latitude` and `longitude` to float, raise `ValueError` on failure.
* Put image attachments into `image` as well as `attachments` ([bridgy-fed429](https://github.com/snarfed/bridgy-fed/issues/429)).
* Handle Hubzilla's composite object attachment `value`s.
* Bug fix for null `mediaType` in `attachment` and `tags`.
* Add new `TYPES_WITH_OBJECT` constant.
* Add new `get_urls`, `address` functions.
* Improve `Content-Type` compatibility with `application/ld+json; profile="https://www.w3.org/ns/activitystreams"`.
* Bug fix for `Undo` activities with bare string id `object`s.
* Revise HTML in `PropertyValue` attachments on actors to include full URL in anchro text to be compatible with Mastodon's profile link verification.
* `atom`:
* `activities_to_atom` etc:
* Switch `content` from XHTML to HTML inside CDATA to support non-XHTML input content ([bridgy-fed624](https://github.com/snarfed/bridgy-fed/issues/624).
* Bug fix, handle bare string URL `image` values.
* Bug fix, emove incorrect `type="application/atom+xml"` from `rel="self"` `link` in `entry`.
* Render `objectType: comment` attachments.
* Remove invalid `<a>` element for tags.
* Bug fixes in `activity_to_atom`/`activities_to_atom` for dict-valued `url` fields.
* Render images in article/note attachments.
* Render `objectType: service` attachments, eg Bluesky custom feeds.
* `bluesky`:
* Implement `Bluesky` API class, including `get_activities`.
* Drop bundled `app.bsky`/`com.atproto` lexicons, use lexrpc's instead.
* Convert reposts, quotes, inline links, attached links, and mentions, both directions. Includes Bluesky facet (rich text) support.
* Handle quote posts with attached images, both directions.
* Handle likes, both directions.
* Add new `web_url_to_at_uri` function.
* `from_as1`: handle link tags without start/end indices.
* `to_as1`:
* Add new `type` kwarg.
* Generate staging.bsky.app profile and post URLs.
* Propagate profile `did` into actor `id`.
* Add unimplemented stub for custom feeds, eg `app.bsky.feed.defsgeneratorView`.
* Add `as1_to_profile`.
* Bug fix for converting follows, both directions: `subject` in `app.bsky.graph.follow` is followee, not follower. ([That field is badly named!](https://discord.com/channels/1097580399187738645/1097580399187738648/1151933384738746478))
* `jsonfeed`:
* `activities_to_jsonfeed`:
* Bug fix, handle bare string values for `image` and `stream`.
* Bug fix: handle non-object `author`.
* `mastodon`:
* `status_to_object`: add/fix alt text handling for images.
* `microformats2`:
* `json_to_html`:
* HTML-escape tag and quote attachment names. Fixes [GHSA-4w4f-g49g-3f7j](https://github.com/snarfed/bridgy/security/advisories/GHSA-4w4f-g49g-3f7j); thank you [janboddez](https://github.com/janboddez)!
* `json_to_object`:
* Improve handling of items with multiple types by using [post type discovery](https://indiewebcamp.com/post-type-discovery) more aggressively.
* Normalize ISO-8601 format of `published` and `updated` timestamps.
* `object_to_json`:
* Bug fix, handle bare string URL `image` values.
* Normalize ISO-8601 format of `published` and `updated` timestamps.
* Handle bare string ids for `replies ` and `shares` (usually from AS2.)
* Include `objectType: service` attachments, eg Bluesky custom feeds, in JSON and HTML output.
* `rss`:
* `from_activities`: handle bare string id `author`.

Breaking changes

* `as2`:
* Interpret bare string `object`, `inReplyTo`, etc values as ids, convert them to bare strings or `id` instead of `url`.
* `microformats2`:
* Convert simple string `in-reply-to`, `repost-of`, `like-of` etc values to AS1 bare strings or `id`s instead of `url`s.

Non-breaking changes

* Add new `bluesky` module for [Bluesky](https://blueskyweb.org/)/[AT Protocol](https://atproto.com/)!
* `as1`:
* Add the `organization` object type and `ACTOR_TYPES` constant ([based on AS2](https://www.w3.org/TR/activitystreams-core/#actors)).
* Add new `get_ids`, `get_object`, and `get_objects` functions.
* `activity_changed`: ignore `inReplyTo.author` ([snarfed/bridgy1338](https://github.com/snarfed/bridgy/issues/1338))
* `as2`:
* Support converting between AS1 `stop-following` and AS2 `Undo` `Follow`.
* Support AS2 `Accept` and `Reject` for follows as well as event RSVPs.
* Add support for the `Question` (ie poll), `Organization`, and `Delete` object types.
* Convert `to`/`cc` to/from AS1 `to` for public and unlisted.
* Handle `type: Document` video attachments like Mastodon emits.
* `from_as1`: bug fix for image objects with `url` and `value` fields (for alt text).
* `from_as1`: bug fix, handle bare string URL `image` values.
* `from_as1`: convert `urls.displayName` to `attachment.name` ([bridgy-fed331](https://github.com/snarfed/bridgy-fed/issues/331)).
* `from_as1`: preserve `inReplyTo` object values as objects, inline single-element lists down down to just single element.
* `to_as1`: use `objectType: featured` for first image in `image` field.
* `to_as1`: populate `actor` into `object.author` for `Update`s as well as `Create`s.
* `to_as1`: convert Mastodon profile metadata `PropertyValue` attachments to `url` composite objects with `displayName`.
* Preserve `to` and `cc` values when converting both directions.
* `atom`:
* Bug fix for rendering image attachments without `image` field to Atom.
* Bug fix for `published` and `updated` in entries with objects, eg likes, reposts, RSVPs, bookmarks. Thanks [gregorlove](https://gregorlove.com/)! ([#480](https://github.com/snarfed/granary/issues/480))
* Bug fix for content `activity/ies_to_atom` when `object` is present and empty.
* Bug fix for objects with elements without `objectType` in the `to` field.
* `flickr`:
* `get_activities`: add support for the `count` kwarg.
* `github`:
* `get_activities`: add support for the `count` kwarg.
* `jsonfeed`:
* Switch from `white-space: pre` CSS to converting newlines to `<br>`s because some feed readers follow it strictly and don't even line wrap ([456](https://github.com/snarfed/granary/issues/456)).
* `mastodon`:
* Add compatibility support for [Truth Social](https://truthsocial.com/).
* Handle truncated JSON API responses.
* `microformats2`:
* `json_to_object`: drop backward compatibility support for `like` and `repost` properties. [Background discussion.](https://chat.indieweb.org/dev/2022-12-23#t1671833687984200)
* `json_to_object`: add new `rel_urls` kwarg to allow attaching `displayName`s to `urls` based on HTML text or `title` attribute ([bridgy-fed331](https://github.com/snarfed/bridgy-fed/issues/331)).
* Add new `json_to_activities` function.
* `hcard_to_html`/`maybe_linked_name`: when `name` is missing, use pretty URL as visible text.
* Support the `h-card` `org` property.
* `json_to_object`: handle composite `rsvp` property value.
* `json_to_object`: bug fix when `fetch_mf2` is True, handle when we run the authorship algorithm and fetch an author URL that has a `u-photo` with `alt`.
* `rss`:
* `from_activities`: fix item ordering to match input activities.