Arroba

_Breaking changes:_

* Add much more lexicon schema validation for records and XRPC method input, output, and parameters.
* `storage`:
* Switch `Storage.write` to return `Block` instead of `CID`.

_Non-breaking changes:_

* `did`:
* Add new `update_plc` method.
* `create_plc`: add new `also_known_as` kwarg.
* `resolve_handle`: drop `Content-Type: text/plain` requirement for HTTPS method.
* `mst`:
* Add new optional `start` kwarg to `load_all`.
* `repo`:
* [Emit new identity and account events](https://github.com/snarfed/bridgy-fed/issues/1119) to `subscribeRepos` when creating new repos.
* `storage`:
* Add new `deactivate_repo`, `activate_repo`, and `write_event` methods.
* Add new optional `repo` kwarg to `read_blocks_by_seq` and `read_events_by_seq` to limit returned results to a single repo.
* `datastore_storage`:
* Add new `max_size` and `accept_types` kwarg to `AtpRemoteBlob.get_or_create` for the blob's `maxSize` and `accept` parameters in its lexicon. If the fetched file doesn't satisfy those constraints, raises `lexrpc.ValidationError.`
* `DatastoreStorage.read_blocks_by_seq`: use strong consistency for datastore query. May fix occasional `AssertionError` when serving `subscribeRepos`.
* `xrpc_sync`:
* Switch `getBlob` from returning HTTP 302 to 301.
* Implement `since` param in `getRepo`.
* `subscribeRepos`: wait up to 60s on a skipped sequence number before giving up and emitting it as a gap.
* `util`:
* `service_jwt`: add new `**claims` parameter for additional JWT claims, eg [`lxm`](https://github.com/bluesky-social/atproto/discussions/2687).

_Breaking changes:_

* `datastore_storage`:
* `DatastoreStorage`: add new required `ndb_client` kwarg to constructor, used to get new context in lexrpc websocket subscription handlers that run server methods like `subscribeRepos` in separate threads ([snarfed/lexrpc8](https://github.com/snarfed/lexrpc/issues/8)).
* `DatastoreStorage.read_blocks_by_seq`: if the ndb context gets closed while we're still running, log a warning and return. (This can happen in eg `flask_server` if the websocket client disconnects early.)
* `AtpRemoteBlob`: if the blob URL doesn't return the `Content-Type` header, infer type from the URL, or fall back to `application/octet-stream` ([bridgy-fed1073](https://github.com/snarfed/bridgy-fed/issues/1073)).
* `did`:
* Cache `resolve_plc`, `resolve_web`, and `resolve_handle` for 6h, up to 5000 total results per call.
* `storage`: rename `Storage.read_commits_by_seq` to `read_events_by_seq` for new account tombstone support.
* `xrpc_sync`: rename `send_new_commits` to `send_events`, ditto.
* `xrpc_repo`: stop requiring auth for read methods: `getRecord`, `listRecords`, `describeRepo`.

_Non-breaking changes:_

* `did`:
* Add `HANDLE_RE` regexp for handle validation.
* `storage`:
* Add new `Storage.tombstone_repo` method, implemented in `MemoryStorage` and `DatastoreStorage`. [Used to delete accounts.](https://github.com/bluesky-social/atproto/discussions/2503#discussioncomment-9502339) ([bridgy-fed783](https://github.com/snarfed/bridgy-fed/issues/783))
* Add new `Storage.load_repos` method, implemented in `MemoryStorage` and `DatastoreStorage`. Used for `com.atproto.sync.listRepos`.
* `util`:
* `service_jwt`: add optional `aud` kwarg.
* `xrpc_sync`:
* `subscribeRepos`:
* Add support for non-commit events, starting with account tombstones.
* Add `ROLLBACK_WINDOW` environment variable to limit size of [rollback window](https://atproto.com/specs/event-stream#sequence-numbers). Defaults to no limit.
* For commits with create or update operations, always include the record block, even if it already existed in the repo beforehand ([snarfed/bridgy-fed1016](https://github.com/snarfed/bridgy-fed/issues/1016)).
* Bug fix, populate the time each commit was created in `time` instead of the current time ([snarfed/bridgy-fed1015](https://github.com/snarfed/bridgy-fed/issues/1015)).
* Start serving `getRepo` queries with the `since` parameter. `since` still isn't actually implemented, but we now serve the entire repo instead of returning an error.
* Implement `getRepoStatus` method.
* Implement `listRepos` method.
* `getRepo` bug fix: include the repo head commit block.
* `xrpc_repo`:
* `getRecord`: encoded returned records correctly as [ATProto-flavored DAG-JSON](https://atproto.com/specs/data-model).
* `xrpc_*`: return `RepoNotFound` and `RepoDeactivated` errors when appropriate ([snarfed/bridgy-fed1083](https://github.com/snarfed/bridgy-fed/issues/1083)).

Notable changes

* Bug fix: base32-encode TIDs in record keys, `at://` URIs, commit `rev`s, etc. Before, we were using the integer UNIX timestamp directly, which happened to be the same 13 character length. Oops.
* Switch from `BGS_HOST` environment variable to `RELAY_HOST`. `BGS_HOST` is still supported for backward compatibility.
* `datastore_storage`:
* Bug fix for `DatastoreStorage.last_seq`, handle new NSID.
* Add new `AtpRemoteBlob` class for storing "remote" blobs, available at public HTTP URLs, that we don't store ourselves.
* `did`:
* `create_plc`: strip padding from genesis operation signature (for [did-method-plc54](https://github.com/did-method-plc/did-method-plc/pull/54), [atproto#1839](https://github.com/bluesky-social/atproto/pull/1839)).
* `resolve_handle`: return None on bad domain, eg `.foo.com`.
* `resolve_handle` bug fix: handle `charset` specifier in HTTPS method response `Content-Type`.
* `util`:
* `new_key`: add `seed` kwarg to allow deterministic key generation.
* `xrpc_repo`:
* `getRecord`: try to load record locally first; if not available, forward to AppView.
* `xrpc_sync`:
* Implement `getBlob`, right now only based on "remote" blobs stored in `AtpRemoteBlob`s in datastore storage.

Notable changes

* Migrate to [ATProto repo v3](https://atproto.com/blog/repo-sync-update). Specifically, the existing `subscribeRepos` sequence number is reused as the new `rev` field in commits. ([Discussion.](https://github.com/bluesky-social/atproto/discussions/1607)).
* Add new `did` module with utilities to create and resolve `did:plc`s and resolve `did:web`s.
* Add new `util.service_jwt` function that generates [ATProto inter-service JWTs](https://atproto.com/specs/xrpc#inter-service-authentication-temporary-specification).
* `Repo`:
* Add new `signing_key`/`rotation_key` attributes. Generate store, and load both in `datastore_storage`.
* Remove `format_init_commit`, migrate existing calls to `format_commit`.
* `Storage`:
* Rename `read_from_seq` => `read_blocks_by_seq` (and in `MemoryStorage` and `DatastoreStorage`), add new `read_commits_by_seq` method.
* Merge `load_repo` `did`/`handle` kwargs into `did_or_handle`.
* XRPCs:
* Make `subscribeRepos` check storage for all new commits every time it wakes up.
* As part of this, replace `xrpc_sync.enqueue_commit` with new `send_new_commits` function that takes no parameters.
* Drop bundled `app.bsky`/`com.atproto` lexicons, use [lexrpc](https://lexrpc.readthedocs.io/)'s instead.

Big milestone: arroba is successfully federating with the [ATProto sandbox](https://atproto.com/blog/federation-developer-sandbox)! See [app.py](https://github.com/snarfed/arroba/blob/main/app.py) for the minimal demo code needed to wrap arroba in a fully functional PDS.

* Add Google Cloud Datastore implementation of repo storage.
* Implement `com.atproto` XRPC methods needed to federate with sandbox, including most of `repo` and `sync`.
* Notably, includes `subscribeRepos` server side over websocket.
* ...and much more.

Implement repo and commit chain in new Repo class, including pluggable storage. This completes the first pass at all PDS data structures. Next release will include initial implementations of the `com.atproto.sync.*` XRPC methods.