Arroba

_Breaking changes:_

* `repo`:
* `apply_commit`, `apply_writes`: raise an exception if the repo is inactive.
* `storage`:
* `create_repo`: remove `signing_key` and `rotation_key` kwargs, read them from input repo instead.
* `load_repo`: don't raise an exception if the repo is tombstoned.
* `datastore_storage`:
* Stop storing `AtpBlock.decoded` in the datastore, it's now just an in memory `property`.
* `util`:
* Rename `TombstonedRepo` to `InactiveRepo`.

_Non-breaking changes:_

* `datastore_storage`:
* `DatastoreStorage`:
* Add new `ndb_context_kwargs` constructor kwarg.
* `apply_commit`: handle deactivated repos.
* `create_repo`: propagate `Repo.status` into `AtpRepo`.
* `AtpRemoteBlob`:
* `get_or_create`: drop datastore transaction.
* Add `width` and `height` properties, populated for images and videos, to be used in image/video embed `aspectRatio` ([snarfed/bridgy-fed1571](https://github.com/snarfed/bridgy-fed/issues/1571)).
* Check video length, raise `ValidationError` on [videos over 3 minutes](https://bsky.app/profile/bsky.app/post/3lk26lxn6sk2u).
* `did`:
* Add new `get_signing_key`, `get_handle` functions.
* `create_plc`: remove trailing slash from `services.atproto_pds.endpoint`.
* `storage`:
* `Storage`: add new `write_blocks` method, implement in `MemoryStorage` and `DatastoreStorage`.
* `xrpc_repo`:
* `describe_server`: include all `app.bsky` collections and others like `chat.bsky.actor.declaration`; fetch and include DID doc.
* Implement `com.atproto.repo.importRepo`.
* `xrpc_sync`:
* `get_blob`:
* If we have more than one blob URL for the same CID, serve the latest one ([bridgy-fed1650](https://github.com/snarfed/bridgy-fed/issues/1650).
* Add HTTP `Cache-Control` to cache for 1h.
* `list_repos`:
* Bug fix: Use string TID for `rev`, not integer sequence number.
* Bug fix: don't set status to `null` if the account is active.

_Breaking changes:_

* Add much more lexicon schema validation for records and XRPC method input, output, and parameters.
* `storage`:
* Switch `Storage.write` to return `Block` instead of `CID`.

_Non-breaking changes:_

* `did`:
* Add new `update_plc` method.
* `create_plc`: add new `also_known_as` kwarg.
* `resolve_handle`: drop `Content-Type: text/plain` requirement for HTTPS method.
* `mst`:
* Add new optional `start` kwarg to `load_all`.
* `repo`:
* [Emit new identity and account events](https://github.com/snarfed/bridgy-fed/issues/1119) to `subscribeRepos` when creating new repos.
* `storage`:
* Add new `deactivate_repo`, `activate_repo`, and `write_event` methods.
* Add new optional `repo` kwarg to `read_blocks_by_seq` and `read_events_by_seq` to limit returned results to a single repo.
* `datastore_storage`:
* Add new `max_size` and `accept_types` kwarg to `AtpRemoteBlob.get_or_create` for the blob's `maxSize` and `accept` parameters in its lexicon. If the fetched file doesn't satisfy those constraints, raises `lexrpc.ValidationError.`
* `DatastoreStorage.read_blocks_by_seq`: use strong consistency for datastore query. May fix occasional `AssertionError` when serving `subscribeRepos`.
* `xrpc_sync`:
* Switch `getBlob` from returning HTTP 302 to 301.
* Implement `since` param in `getRepo`.
* `subscribeRepos`: wait up to 60s on a skipped sequence number before giving up and emitting it as a gap.
* `util`:
* `service_jwt`: add new `**claims` parameter for additional JWT claims, eg [`lxm`](https://github.com/bluesky-social/atproto/discussions/2687).

_Breaking changes:_

* `datastore_storage`:
* `DatastoreStorage`: add new required `ndb_client` kwarg to constructor, used to get new context in lexrpc websocket subscription handlers that run server methods like `subscribeRepos` in separate threads ([snarfed/lexrpc8](https://github.com/snarfed/lexrpc/issues/8)).
* `DatastoreStorage.read_blocks_by_seq`: if the ndb context gets closed while we're still running, log a warning and return. (This can happen in eg `flask_server` if the websocket client disconnects early.)
* `AtpRemoteBlob`: if the blob URL doesn't return the `Content-Type` header, infer type from the URL, or fall back to `application/octet-stream` ([bridgy-fed1073](https://github.com/snarfed/bridgy-fed/issues/1073)).
* `did`:
* Cache `resolve_plc`, `resolve_web`, and `resolve_handle` for 6h, up to 5000 total results per call.
* `storage`: rename `Storage.read_commits_by_seq` to `read_events_by_seq` for new account tombstone support.
* `xrpc_sync`: rename `send_new_commits` to `send_events`, ditto.
* `xrpc_repo`: stop requiring auth for read methods: `getRecord`, `listRecords`, `describeRepo`.

_Non-breaking changes:_

* `did`:
* Add `HANDLE_RE` regexp for handle validation.
* `storage`:
* Add new `Storage.tombstone_repo` method, implemented in `MemoryStorage` and `DatastoreStorage`. [Used to delete accounts.](https://github.com/bluesky-social/atproto/discussions/2503#discussioncomment-9502339) ([bridgy-fed783](https://github.com/snarfed/bridgy-fed/issues/783))
* Add new `Storage.load_repos` method, implemented in `MemoryStorage` and `DatastoreStorage`. Used for `com.atproto.sync.listRepos`.
* `util`:
* `service_jwt`: add optional `aud` kwarg.
* `xrpc_sync`:
* `subscribeRepos`:
* Add support for non-commit events, starting with account tombstones.
* Add `ROLLBACK_WINDOW` environment variable to limit size of [rollback window](https://atproto.com/specs/event-stream#sequence-numbers). Defaults to no limit.
* For commits with create or update operations, always include the record block, even if it already existed in the repo beforehand ([snarfed/bridgy-fed1016](https://github.com/snarfed/bridgy-fed/issues/1016)).
* Bug fix, populate the time each commit was created in `time` instead of the current time ([snarfed/bridgy-fed1015](https://github.com/snarfed/bridgy-fed/issues/1015)).
* Start serving `getRepo` queries with the `since` parameter. `since` still isn't actually implemented, but we now serve the entire repo instead of returning an error.
* Implement `getRepoStatus` method.
* Implement `listRepos` method.
* `getRepo` bug fix: include the repo head commit block.
* `xrpc_repo`:
* `getRecord`: encoded returned records correctly as [ATProto-flavored DAG-JSON](https://atproto.com/specs/data-model).
* `xrpc_*`: return `RepoNotFound` and `RepoDeactivated` errors when appropriate ([snarfed/bridgy-fed1083](https://github.com/snarfed/bridgy-fed/issues/1083)).

Notable changes

* Bug fix: base32-encode TIDs in record keys, `at://` URIs, commit `rev`s, etc. Before, we were using the integer UNIX timestamp directly, which happened to be the same 13 character length. Oops.
* Switch from `BGS_HOST` environment variable to `RELAY_HOST`. `BGS_HOST` is still supported for backward compatibility.
* `datastore_storage`:
* Bug fix for `DatastoreStorage.last_seq`, handle new NSID.
* Add new `AtpRemoteBlob` class for storing "remote" blobs, available at public HTTP URLs, that we don't store ourselves.
* `did`:
* `create_plc`: strip padding from genesis operation signature (for [did-method-plc54](https://github.com/did-method-plc/did-method-plc/pull/54), [atproto#1839](https://github.com/bluesky-social/atproto/pull/1839)).
* `resolve_handle`: return None on bad domain, eg `.foo.com`.
* `resolve_handle` bug fix: handle `charset` specifier in HTTPS method response `Content-Type`.
* `util`:
* `new_key`: add `seed` kwarg to allow deterministic key generation.
* `xrpc_repo`:
* `getRecord`: try to load record locally first; if not available, forward to AppView.
* `xrpc_sync`:
* Implement `getBlob`, right now only based on "remote" blobs stored in `AtpRemoteBlob`s in datastore storage.

Notable changes

* Migrate to [ATProto repo v3](https://atproto.com/blog/repo-sync-update). Specifically, the existing `subscribeRepos` sequence number is reused as the new `rev` field in commits. ([Discussion.](https://github.com/bluesky-social/atproto/discussions/1607)).
* Add new `did` module with utilities to create and resolve `did:plc`s and resolve `did:web`s.
* Add new `util.service_jwt` function that generates [ATProto inter-service JWTs](https://atproto.com/specs/xrpc#inter-service-authentication-temporary-specification).
* `Repo`:
* Add new `signing_key`/`rotation_key` attributes. Generate store, and load both in `datastore_storage`.
* Remove `format_init_commit`, migrate existing calls to `format_commit`.
* `Storage`:
* Rename `read_from_seq` => `read_blocks_by_seq` (and in `MemoryStorage` and `DatastoreStorage`), add new `read_commits_by_seq` method.
* Merge `load_repo` `did`/`handle` kwargs into `did_or_handle`.
* XRPCs:
* Make `subscribeRepos` check storage for all new commits every time it wakes up.
* As part of this, replace `xrpc_sync.enqueue_commit` with new `send_new_commits` function that takes no parameters.
* Drop bundled `app.bsky`/`com.atproto` lexicons, use [lexrpc](https://lexrpc.readthedocs.io/)'s instead.

Big milestone: arroba is successfully federating with the [ATProto sandbox](https://atproto.com/blog/federation-developer-sandbox)! See [app.py](https://github.com/snarfed/arroba/blob/main/app.py) for the minimal demo code needed to wrap arroba in a fully functional PDS.

* Add Google Cloud Datastore implementation of repo storage.
* Implement `com.atproto` XRPC methods needed to federate with sandbox, including most of `repo` and `sync`.
* Notably, includes `subscribeRepos` server side over websocket.
* ...and much more.