Added
- Add bounded `Rows.join_sorted()` for explicitly ordered records, with consumed-prefix
validation and first-right-record schema rules.
- Report explicit sorted group, merge, and join execution through existing report fields.
- Add optional atomic path output to Flow CSV/JSON/JSONL and Rows CSV/JSONL sinks,
including no-overwrite publication. Existing direct output remains the default.
- `Flow.to_jsonl()` streams arbitrary JSON values as one value per line.
`Rows.to_jsonl()` now accepts `default` for custom serialization.
- `Flow.merge_sorted()` stably merges two ascending inputs without sorting or
materializing them. Ties prefer the left input and outputs retain their identity.
- `Rows.group_by_sorted()` aggregates adjacent ascending keys in Python with
current group state and one lookahead row. Consumed keys are checked for exact
builtin types and order; this explicit mode does not sort or support spill.
- `Pairs.run_with_report()` executes a pair terminal once and returns its value
with an execution report. It supports `to_dict`, `group_values`,
`collect_values`, and `aggregate_values`.
Fixed
- Rust source builds support the declared 1.85 minimum. Native kernels no longer
require Rust 1.88 through let-chain syntax; CI checks the minimum compiler.
- SQLite sinks match existing table, view, and column names using SQLite's ASCII
case rules. Duplicate column aliases raise `DuplicateKeyError` before inserts.
Fail mode and view rejection check the destination before reading rows.
- Sync and async flows accept built-in ranges longer than `sys.maxsize`. Bounded
reads and exact counts no longer fail during source size detection.
- `spreadsheet_safe=True` also neutralizes formula-like CSV headers, including
inferred record field names and explicitly named empty output. Record lookup
still uses the original names; the raw-output default is unchanged.
- Relative Parquet output paths stay anchored to the initial working directory
if a source or conversion callback changes directories during the write.
- Scalar keys in explicit sorted operations avoid temporary shape tuples while
retaining per-row type and ordering checks.
- Atomic CSV and JSON output keeps its original destination when a source or
serializer changes the working directory during execution.
- CSV and SQLite record sinks propagate `StopIteration` from first-record
conversion instead of treating it as empty input. SQLite replacement leaves
the existing table intact when that conversion fails.
- Sync and async `chunk`, `window`, and `batch_by_size` validate integer bounds
before execution, including their aliases. Floats, NaN, and infinity now raise
`TypeError` instead of failing after source consumption or leaving async batches
unbounded. Objects implementing `__index__` are normalized once per bound.
- `unique()`, `unique_by()`, and `Pairs.unique_keys()` propagate equality errors
instead of treating them as unhashable keys. Hash callbacks keep their existing
lookup and insertion counts.
- Async uniqueness, `agg.count_distinct()`, and native pair-uniqueness continuation
also propagate equality errors without extra hash calls. Iterator cleanup retains
nested exception notes when several owned resources fail to close.
- Distinct operations avoid key-wrapper allocations for exact built-in strings.
String subclasses and custom keys retain guarded hash and equality handling.
- Grouped reductions and frequency counts propagate `KeyError` raised by a key's
hash or equality method instead of treating it as a missing group. This includes
async reductions, grouping collectors, Pairs, and spilled grouping.
- Parquet `if_exists="error"` publishes with an atomic no-overwrite hard link.
A concurrent creator or dangling symlink cannot be overwritten. Filesystems
without hard-link support report an error; `replace` still uses atomic rename.
- Owned Arrow resources report close failures on successful queries and attach
cleanup diagnostics to an existing query error. Cleanup attempts every owned
resource. This also changes Arrow `first()`, which previously ignored close errors.
- Engine benchmarks calibrate warmed timing blocks for short tasks and first-row
latency, retaining elapsed time and call counts in their reports. Rebuild
single-call baselines before comparing. Timing and resource regression limits
are unchanged; scheduled CI now uploads the three raw reference reports.
- Benchmark report schema 6 records Python and glibc allocator environment
settings. Baseline creation and comparison reject missing or mismatched
settings; regenerate older reports. The runners do not change allocator defaults.
- NumPy frequency benchmarks check key types, integer counts, first-key order,
and floating-point bit patterns. Separate NaN entries, their signs, and their
payloads remain distinct when comparing independently computed outputs.
- Generated Python `with_columns` loops call the live enrichment function.
Changes to captured accessors, expression evaluators, or the selector list
remain visible after plan caching and while reading the source. The transform
still copies the row before invoking selectors against the original input.
- Arrow file projections keep selector changes made by custom scan openers
visible. Parquet rechecks the projection after creating its dataset, including
with an explicit source filter. CSV keeps full fields when its public reader
hooks differ from the extension entrypoints. Normal CSV projection retains
the bounded schema probe and the default data reader.
- `Rows.select()` rechecks captured field accessors before using Rust, NumPy,
or Arrow projection metadata. The generated Python loop calls the live
projection, so changes made during input reads remain visible. Custom Arrow
batch sources keep fields available for fallback; changed projections can
infer their output dtype instead of forcing the former field's schema.
NumPy fallback finishes the already-opened source without reopening it.
- Pivot calls the live index, column, and value selectors on its Python path.
Direct dict lookups had ignored changes to selector code, closure bindings,
and globals. Native admission now checks those bindings before execution;
its fallback shares the general selector loop.
- Native pivot falls back when Rows/Flow iteration or source hooks change,
including changes to function code. Replacement results, exceptions, and
iterator cleanup remain observable.
- `frequencies(key=...)` uses the Python pipeline for sequential `auto` plans,
so key callbacks can affect subsequent input reads. Native materialization
could hide those changes. Reports identify this route as `python_frequency`.
- Benchmark speedup requirements for composite count/sum groups and NamedTuple
callable joins now apply from 1,000 rows. Smaller inputs still emit timing and
allocation data for cross-run checks. The original ratios and CI workloads
are unchanged; small runs no longer inherit speedup requirements validated
for larger inputs.
- The engine benchmark CLI lists scenarios without running their tasks, timing,
allocation tracking, or execution observation. Listing and measurement share
filtering and fixture cleanup, including when a later scenario builder fails.
- Single-collector grouping preserves the general collector program's state
release order, including when output is closed early. Unused input keys are
released before reading the current finisher, so changes made by their release
callbacks take effect.
- Single-collector grouping uses the current step after truth-testing a custom
completion result. It also releases replaced completion values before pulling
another row, matching the general collector program's callback behavior.
- Python grouping calls the live key selector for each row. A field shortcut
could keep using an old field after a source or callback changed the selector's
code or closure, merging distinct groups. Selector globals and error types
now follow the same function calls as general grouped aggregation.
- Scalar caches no longer confuse numerically equal constants of different
types. Directly constructed expressions retain their Python result types,
wide integer values, and custom constant identities. Expressions with
nonstandard operand types use Python in `auto` mode; forcing `native` raises
`NativeUnsupportedError` instead of changing their numeric representation.
- Float expressions retain the sign of `-0.0` in their instructions and evaluator
cache. Compiling a positive-zero expression first no longer changes a later
negative-zero result. Scalar and Pairs execution keep their existing routes.
- Python grouped sums call the live collector lifecycle throughout traversal
and output. Changes to function code or selector closure cells made by a source,
key callback, or output consumer are no longer hidden by an inlined sum loop.
- Two-key count/sum groups also use the live collector program. Their separate
loop skipped changes to initializer and step code or the sum selector closure
made while reading the source, even though the same collectors worked correctly
in other group layouts.
- Grouped aggregation uses the general collector program for custom lifecycle
getters. A dynamic `step` property or `__getattribute__` override is read for
each step instead of being cached as a fixed function.
- Single-collector grouping no longer caches a temporary lifecycle hook between
key selection and lookup. If hashing replaces that hook, its release callback
runs before the next hash call, matching the general collector program.
- Exact builtin checks no longer use custom metaclass equality to infer source
size or select grouping, spill, range lookup, and float-expression shortcuts.
Custom iterables are counted by traversal; map/filter fusion does not request
their length hints. Grouping retains custom key and serialization calls.
- Execution reports distinguish successful top-level Rust and Arrow record
joins from Python joins. Native pair aggregation also records its direct route.
- Benchmark comparisons reject missing provenance and mismatched workloads.
Both suites record dependencies, Git state, code and workload fingerprints,
and an untimed observation of the task. Median baselines retain each run's
provenance. Older reports need to be regenerated for report schema 6.
- Reports include CPU affinity, NumPy CPU dispatch, and an allowlist of runtime
settings. Comparisons reject mismatched configurations; both runners also
reject configuration changes during measurement.
- Join benchmarks honor the requested engine. Twelve Python controls separate
dict and Mapping records, field and callable keys, and inner, left, and repeated
matches. References preserve callable-key columns and snapshots taken before
key selection, including unmatched left rows.
- NumPy frequency benchmarks cover integer and float arrays, with and without
a callable key, at low and high cardinality. All references preserve first-key
order; Python array-to-list conversion is included in its timed task.
- Competitive samples warm each task for at least 1ms after GC and record the
number of warmup calls. One warmup call left inconsistent timings for short
NumPy tasks in local measurements.
- Competitive benchmarks now record peak Python allocation in a separate,
untimed call for every implementation, using the engine suite's shared helper.
Schema 6 rejects missing or invalid resource measurements, and baseline
creation rejects inconsistent resource sets instead of substituting zero.
Earlier competitive reports contain no allocation evidence.
- Regression checks accept zero allocated bytes when both runs report zero.
- `frequencies()` on retained lists and tuples falls back to Python when the
optional Rust extension is unavailable, including in the browser wheel.
- Frequency benchmarks use bulk conversion for NumPy dictionaries and pandas'
`to_dict()`, avoiding per-key normalization overhead in the reference tasks.
Changed
- Automatic NumPy `frequencies()` without a key can count a selected native
stream in Rust through its existing iterator. Source fallback and query cleanup
stay in the physical executor. Custom keys return to Python before hashing;
old wheels and free-threaded CPython retain the prior counting path.
- Python `Rows.select()` keeps its per-row selector snapshot as a list, avoiding
an intermediate tuple conversion. Selector edits still affect subsequent rows,
and replaced selectors stay alive until the current row finishes.
- Python scalar iteration over an exact NumPy ndarray checks its live size
without creating a shape tuple for every value. Dimension and length checks,
lazy reads, and the fallback for custom array objects remain in place.
- Python joins reduce layout-checking overhead for left records with more than
four fields. Repeated private dict snapshots need no temporary field-name tuple
or Python comparison generator. Field identity, short-circuiting, suffix keys,
snapshot lifetimes, and the bounded layout cache retain their existing behavior.
- Two-key count/sum groups can run in Rust with `engine="auto"` when a retained
list or tuple contains exact tuple rows and both keys and the selected values
are plain signed 64-bit integers. The path preserves first-key identities,
encounter order, and wider sums. Changed functions, one-shot sources, and
unsupported types use the Python collector program; older extensions can
decline the optional entry point.
- `frequencies()` can continue counting in Rust after its bounded integer
prefix. The continuation supports exact builtin integer, boolean, string,
bytes, float, and `None` keys in retained lists and tuples on GIL-enabled
CPython. It updates the output dictionary directly and hands custom keys
back to Python before calling their hash or equality methods.
- Frequency execution reports record a completed native count as `rust_direct`
and a count completed by Python after a native prefix as `python_frequency`.
Internal
- Browser wheels record checkout identity and working-tree state. Release labels
require a clean checkout matching the version tag; dirty or unknown builds
remain development builds. The playground displays this provenance.
- Release smoke checks exercise paid-order aggregation and bounded async mapping
in addition to Python/native integer sums.
- Single-collector groups write step results directly to their stored entries
and avoid reading stored state for completed groups. Group benchmarks now
include `agg.first()` with repeated and distinct keys.
- Group benchmarks now cover custom completion predicates with repeated and
distinct keys, alongside collectors that cannot finish early.
- Generated scalar expressions, row expressions, and fused loops share a smaller
AST location pass. It preserves traversal order and the existing synthetic
source positions used in tracebacks.
- Scalar program fingerprints use fewer intermediate records while retaining
the binary framing format, arbitrary-size integers, and float payload bits.
- Scalar fingerprint encoding reuses fixed instruction headers instead of
rebuilding their bytes for every query. Expression reads and type checks are
unchanged; the table retains no expressions or source data.
- Planning benchmarks separate compilation from execution for integer and float
expressions and a callable control, using the same pipeline for each pair.
- Python grouped output uses one fewer forwarding generator. The collector
iterator still starts lazily and retains its existing cleanup path.
- Collector lifecycle properties read their original slots directly, removing
an extra Python wrapper. Frozen assignment, explicit replacement, and deletion
errors are preserved. The API manifest now classifies those existing fields
as properties; their names and constructor signatures are unchanged.
- Add Python dict-group benchmarks for field and callable selectors with repeated
and distinct keys. Execution observations record the case's requested engine.
- Include nominal Mapping and mappingproxy field-group cases with repeated and
distinct keys when measuring Python selector changes.
- Extend the two-key count/sum benchmarks to repeated keys. Keep the original
direct-versus-callable timing threshold and report any regression against it.
- Move the narrow integer-key record-join ABI adapter into the existing join
executor module. Kernel order, shape guards, and fallback behavior are unchanged.
Documentation
- Repair split return descriptions and unresolved API references in generated
pages. The two `partition_results()` methods now describe their success and
exception lists separately.
- Correct the DB-API example's `batch_size` argument and the distinction between
`Rows.skip()` and `Rows.drop()` in the API index.
- Explain the difference between a compiled outer plan and a recorded direct
terminal route, including the limits of reports for compound queries.
- Revise the website's introductory and performance text and document the next
benchmark and refactoring steps in the roadmap.