Tigerbeetle

Released: 2024-12-27

Safety And Performance

- [2608](https://github.com/tigerbeetle/tigerbeetle/pull/2608)

Improve replication reliability for tiny messages.

Features

- [2607](https://github.com/tigerbeetle/tigerbeetle/pull/2607)

Add info-level logging for basic progress events.

- [2603](https://github.com/tigerbeetle/tigerbeetle/pull/2603)

Add logging and runtime configuration parameters for state sync.

Internals

- [2604](https://github.com/tigerbeetle/tigerbeetle/pull/2604)

Fix `fuzz_lsm_scan` checkpoint schedule.

TigerTracks 🎧

- [Street Spirit](https://open.spotify.com/track/2QwObYJWyJTiozvs0RI7CF?si=d0c0c7d6f9e946a4)

Released: 2024-12-22

Safety And Performance

- [2592](https://github.com/tigerbeetle/tigerbeetle/pull/2592)

Coalesce LSM tables in memory before writing them to disk. This significantly improves workloads
with small batch sizes, that would otherwise churn in the top level of the LSM while incurring
heavy write amplification.

Features

- [2590](https://github.com/tigerbeetle/tigerbeetle/pull/2590)

TigerBeetle recently gained the ability to do runtime debug logging with `--log-debug`. Extend
that to other subcommands - not just `start`.

Additionally, the Python client now has logs integrated with Python's native `logging` module,
which means no more printing to stderr!

Internals

- [2591](https://github.com/tigerbeetle/tigerbeetle/pull/2591)

Fix broken links to TigerBeetle blog posts, thanks PThorpe92!

TigerTracks 🎧

- [Unwritten](https://www.youtube.com/watch?v=b7k0a5hYnSI)

Released: 2024-12-19

Safety And Performance

- [2584](https://github.com/tigerbeetle/tigerbeetle/pull/2584)

Our repair can create a feedback loop. Repair requests prepares and headers, but, upon receiving
back, we re-trigger repair, which could lead to duplicate repair work.

To avoid that, make sure that we are not sending more than two repair messages per replica per our
repair timeout.

- [2586](https://github.com/tigerbeetle/tigerbeetle/pull/2586)

Trace AOF write duration

- [2582](https://github.com/tigerbeetle/tigerbeetle/pull/2582)

Timeouts with exponential backoff should reset to their original delay when the timeout is
stopped.

- [2577](https://github.com/tigerbeetle/tigerbeetle/pull/2577)

Assert against ABA problem during commit.

- [2578](https://github.com/tigerbeetle/tigerbeetle/pull/2578)

Add retry for `flock`. `flock`s are cleaned up by the kernel when the file descriptor is closed,
but since that file descriptor is used by io_uring, it actually outlives the process itself.

- [2579](https://github.com/tigerbeetle/tigerbeetle/pull/2579)

Fix `mlock` flag value.

- [2571](https://github.com/tigerbeetle/tigerbeetle/pull/2571)

Don't crash if a round of view changes happened while repairing the pipeline.

Features

- [2423](https://github.com/tigerbeetle/tigerbeetle/pull/2423)

Tab completion in REPL

- [2566](https://github.com/tigerbeetle/tigerbeetle/pull/2566)

Fix `--account-count-hot` flag.

- [2576](https://github.com/tigerbeetle/tigerbeetle/pull/2576)

Fix multi-debit recipe.

TigerTracks 🎧

- [By The Way](https://www.youtube.com/watch?v=qxQnSH3x3sg)

TigerBeetle (unreleased)

Released: 2024-12-16

Safety And Performance

- [2537](https://github.com/tigerbeetle/tigerbeetle/pull/2537)

Exit cleanly if the database grows to the maximum size.

- [2511](https://github.com/tigerbeetle/tigerbeetle/pull/2511)

Simulate virtual machine migration in the VOPR.

- [2561](https://github.com/tigerbeetle/tigerbeetle/pull/2561)

Test that Java client behaves correctly when its thread is interrupted.

Features

- [2540](https://github.com/tigerbeetle/tigerbeetle/pull/2540)

`tigerbeetle format` now automatically generates a random cluster id if it isn't passed on the
command line. To avoid operational errors, it is important that each cluster gets a globally
unique id.

- [2558](https://github.com/tigerbeetle/tigerbeetle/pull/2558),
[2552](https://github.com/tigerbeetle/tigerbeetle/pull/2552)

Improve error reporting when a client gets evicted due to a mismatched version.

Internals

- [2542](https://github.com/tigerbeetle/tigerbeetle/pull/2542)

Switch CLI client to static allocation.

- [2562](https://github.com/tigerbeetle/tigerbeetle/pull/2562),
[2560](https://github.com/tigerbeetle/tigerbeetle/pull/2560)

Run benchmark under sudo to enable memory locking for accurate RSS stats.

- [2556](https://github.com/tigerbeetle/tigerbeetle/pull/2556),
[2555](https://github.com/tigerbeetle/tigerbeetle/pull/2555)

Remove many false negatives from unused imports tidy check.

TigerTracks 🎧

- [You Bring Out The Boogie In Me](https://youtu.be/0AsLHBAbByk)

Released: 2024-12-09

This release includes a correctness fix for the preview API `get_account_balances`,
`get_account_transfers`, `query_accounts`, and `query_transfers`. If your application uses these
features it might be affected.

Safety And Performance

- [2544](https://github.com/tigerbeetle/tigerbeetle/pull/2544)

Fix correctness bug which affected the preview `get_account_balances`, `get_account_transfers`,
`query_accounts`, and `query_transfers` queries. Specifically, if several filters are used (that
is, several fields in `QueryFilter` or `AccountFilter` are set), then some objects might be
missing from the result set.

The underlying data is safely stored in the database, so re-running these queries using the new
version of TigerBeetle will give correct results.

- [2550](https://github.com/tigerbeetle/tigerbeetle/pull/2550)

Fix panic in the node client which occurred on eviction.

- [2534](https://github.com/tigerbeetle/tigerbeetle/pull/2534)

Fix incorrect ABI used on aarch64 for the client library. To prevent such issues from cropping up
in the future, add aarch64 testing to CI.

- [2520](https://github.com/tigerbeetle/tigerbeetle/pull/2520)

Add extra assertions to verify object cache consistency.

- [2485](https://github.com/tigerbeetle/tigerbeetle/pull/2485)

Implement network fault injection in Vörtex, our non-deterministic whole system simulator.

Features

- [2539](https://github.com/tigerbeetle/tigerbeetle/pull/2539)

Log level can now be specified at runtime.

- [2538](https://github.com/tigerbeetle/tigerbeetle/pull/2538)

Log messages now include UTC timestamp (formatted as per [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339)).

Internals

- [2543](https://github.com/tigerbeetle/tigerbeetle/pull/2543)

Relax compiler requirements for building TigerBeetle. While we only support building using one
specific version of Zig, the one downloaded via `./zig/download.sh`, you can try using other
versions.

- [2533](https://github.com/tigerbeetle/tigerbeetle/pull/2533)

Document how to handle errors during release. TigerBeetle's release process is complicated, as we
need to release, in lockstep, both the `tigerbeetle` binary and all the related client libraries.
The release process is intentionally designed to be "highly-available", such that a failure of any
aspect of a release can be safely detected, isolated, and repaired. But, so far, this wasn't
clearly spelled out in the documentation!

TigerTracks 🎧

- [The Beetle's Buzz](https://soundcloud.com/kprotty/the-beetles-buzz)

Released: 2024-12-02

The highlight of today's release is the new official Python client, implemented in
[2527](https://github.com/tigerbeetle/tigerbeetle/pull/2527), and
[2487](https://github.com/tigerbeetle/tigerbeetle/pull/2487). Please kick the tires!

Safety And Performance

- [2517](https://github.com/tigerbeetle/tigerbeetle/pull/2517)

Require that all replicas in a cluster have the latest TigerBeetle binary as a precondition for an
upgrade.

- [2506](https://github.com/tigerbeetle/tigerbeetle/pull/2506)

Fix several bugs when handling misdirected writes. That is, situations when the disk reports
a write as successful, despite the write ending up in the wrong place on the disk!

- [2478](https://github.com/tigerbeetle/tigerbeetle/pull/2478)

Make sure that TigerBeetle memory is not swappable, otherwise a storage fault can occur in a
currently swapped-out page, circumventing TigerBeetle guarantees.

- [2522](https://github.com/tigerbeetle/tigerbeetle/pull/2522)

REPL correctly emits errors when several objects are used as an argument of an operation that
only works for a single object, like `get_account_transfers`.

- [2502](https://github.com/tigerbeetle/tigerbeetle/pull/2502)

Add randomized integration tests for the Java client.

Features

- [2527](https://github.com/tigerbeetle/tigerbeetle/pull/2527),
[2487](https://github.com/tigerbeetle/tigerbeetle/pull/2487)

The Python client!

Internals

- [2526](https://github.com/tigerbeetle/tigerbeetle/pull/2526)

Ignore OOM failures during fuzzing. Fuzzers normally don't use that much memory, but, depending on
random parameters selected by swarm testing, there are big outliers. If all concurrent fuzzers hit
a seed that requires a lot of memory, a fuzzing machine runs out of physical RAM. Handle such
errors and don't treat them as fuzzing failures.

- [2510](https://github.com/tigerbeetle/tigerbeetle/pull/2510)

Track the number of untriaged issues on [DevHub](https://tigerbeetle.github.io/tigerbeetle/).

TigerTracks 🎧

- [Always Look On The Bright Side of Life](https://www.youtube.com/watch?v=X_-q9xeOgG4)

Released: 2024-11-25

Safety And Performance

- [2501](https://github.com/tigerbeetle/tigerbeetle/pull/2501)

Call `DetachCurrentThread` when the Java client is closed. The underlying Zig TigerBeetle client
runs in a separate thread internally, and a handler to this thread was being leaked.

This is not noticeable in normal operation, but could impact long running processes that
create and close clients frequently.

- [2492](https://github.com/tigerbeetle/tigerbeetle/pull/2492)

Document that it's not possible to currently look up or query more than a full batch of accounts
atomically without using the history flag and querying balances.

- [2434](https://github.com/tigerbeetle/tigerbeetle/pull/2434)

Add the ability to check timestamp order - and verify they are monotonically increasing - for
accounts and transfers inside Vortex.

Internals

- [2455](https://github.com/tigerbeetle/tigerbeetle/pull/2455)

Recently the VOPR has gotten too good, and it's very tempting to switch to an empirical mode of
coding: write some code and let the VOPR figure out whether it is correct or not.

This is suboptimal - silence of the VOPR doesn't guarantee total absence of bugs and safety comes
in layers and cross checks. Just formal or informal reasoning is not enough, we need both.

Document this in TigerStyle.

- [2472](https://github.com/tigerbeetle/tigerbeetle/pull/2472)

Previously, the Zig part of languages clients logged directly to stderr using Zig's `std.log`, but
since directly outputting to stderr is considered rude for a library, logging was disabled.

This PR adds in scaffolding for sending these logs to the client language to be handled there,
tying in with native log libraries (e.g., Log4j). No languages use it yet, however.

Additionally, log `warn` and `err` directly to stderr, if there's no handler.

TigerTracks 🎧

- [Hello](https://www.youtube.com/watch?v=kK42LZqO0wA)