Tantivy

================================
Tantivy 0.23 will be backwards compatible with indices created with v0.22 and v0.21. The new minimum rust version will be 1.75.

Bugfixes
- fix potential endless loop in merge [2457](https://github.com/quickwit-oss/tantivy/pull/2457)(PSeitz)
- fix bug that causes out-of-order sstable key. [2445](https://github.com/quickwit-oss/tantivy/pull/2445)(fulmicoton)
- fix ReferenceValue API flaw [2372](https://github.com/quickwit-oss/tantivy/pull/2372)(PSeitz)
- fix `OwnedBytes` debug panic [2512](https://github.com/quickwit-oss/tantivy/pull/2512)(b41sh)

Breaking API Changes
- remove index sorting [2434](https://github.com/quickwit-oss/tantivy/pull/2434)(PSeitz)

Features/Improvements
- **Aggregation**
- Support for cardinality aggregation [2337](https://github.com/quickwit-oss/tantivy/pull/2337) [#2446](https://github.com/quickwit-oss/tantivy/pull/2446) (raphaelcoeffic PSeitz)
- Support for extended stats aggregation [2247](https://github.com/quickwit-oss/tantivy/pull/2247)(giovannicuccu)
- Add Key::I64 and Key::U64 variants in aggregation to avoid f64 precision issues [2468](https://github.com/quickwit-oss/tantivy/pull/2468)(PSeitz)
- Faster term aggregation fetch terms [2447](https://github.com/quickwit-oss/tantivy/pull/2447)(PSeitz)
- Improve custom order deserialization [2451](https://github.com/quickwit-oss/tantivy/pull/2451)(PSeitz)
- Change AggregationLimits behavior [2495](https://github.com/quickwit-oss/tantivy/pull/2495)(PSeitz)
- lower contention on AggregationLimits [2394](https://github.com/quickwit-oss/tantivy/pull/2394)(PSeitz)
- fix postcard compatibility for top_hits, add postcard test [2346](https://github.com/quickwit-oss/tantivy/pull/2346)(PSeitz)
- reduce top hits memory consumption [2426](https://github.com/quickwit-oss/tantivy/pull/2426)(PSeitz)
- check unsupported parameters top_hits [2351](https://github.com/quickwit-oss/tantivy/pull/2351)(PSeitz)
- Change AggregationLimits to AggregationLimitsGuard [2495](https://github.com/quickwit-oss/tantivy/pull/2495)(PSeitz)
- **Range Queries**
- Support fast field range queries on json fields [2456](https://github.com/quickwit-oss/tantivy/pull/2456)(PSeitz)
- Add support for str fast field range query [2460](https://github.com/quickwit-oss/tantivy/pull/2460) [#2452](https://github.com/quickwit-oss/tantivy/pull/2452) [#2453](https://github.com/quickwit-oss/tantivy/pull/2453)(PSeitz)
- modify fastfield range query heuristic [2375](https://github.com/quickwit-oss/tantivy/pull/2375)(trinity-1686a)
- add FastFieldRangeQuery for explicit range queries on fast field (for `RangeQuery` it is autodetected) [2477](https://github.com/quickwit-oss/tantivy/pull/2477)(PSeitz)

- add format backwards-compatibility tests [2485](https://github.com/quickwit-oss/tantivy/pull/2485)(PSeitz)
- add columnar format compatibility tests [2433](https://github.com/quickwit-oss/tantivy/pull/2433)(PSeitz)
- Improved snippet ranges algorithm [2474](https://github.com/quickwit-oss/tantivy/pull/2474)(gezihuzi)
- make find_field_with_default return json fields without path [2476](https://github.com/quickwit-oss/tantivy/pull/2476)(trinity-1686a)
- feat(query): Make `BooleanQuery` support `minimum_number_should_match` [2405](https://github.com/quickwit-oss/tantivy/pull/2405)(LebranceBW)

- **RegexPhraseQuery**
`RegexPhraseQuery` supports phrase queries with regex. E.g. query "b.* b.* wolf" matches "big bad wolf". Slop is supported as well: "b.* wolf"~2 matches "big bad wolf" [2516](https://github.com/quickwit-oss/tantivy/pull/2516)(PSeitz)

- **Optional Index in Multivalue Columnar Index**
For mostly empty multivalued indices there was a large overhead during creation when iterating all docids (merge case).
This is alleviated by placing an optional index in the multivalued index to mark documents that have values.
This will slightly increase space and access time. [2439](https://github.com/quickwit-oss/tantivy/pull/2439)(PSeitz)

- **Store DateTime as nanoseconds in doc store** DateTime in the doc store was truncated to microseconds previously. This removes this truncation, while still keeping backwards compatibility. [2486](https://github.com/quickwit-oss/tantivy/pull/2486)(PSeitz)

- **Performace/Memory**
- lift clauses in LogicalAst for optimized ast during execution [2449](https://github.com/quickwit-oss/tantivy/pull/2449)(PSeitz)
- Use Vec instead of BTreeMap to back OwnedValue object [2364](https://github.com/quickwit-oss/tantivy/pull/2364)(fulmicoton)
- Replace TantivyDocument with CompactDoc. CompactDoc is much smaller and provides similar performance. [2402](https://github.com/quickwit-oss/tantivy/pull/2402)(PSeitz)
- Recycling buffer in PrefixPhraseScorer [2443](https://github.com/quickwit-oss/tantivy/pull/2443)(fulmicoton)

- **Json Type**
- JSON supports now all values on the root level. Previously an object was required. This enables support for flat mixed types. allow more JSON values, fix i64 special case [2383](https://github.com/quickwit-oss/tantivy/pull/2383)(PSeitz)
- add json path constructor to term [2367](https://github.com/quickwit-oss/tantivy/pull/2367)(PSeitz)

- **QueryParser**
- fix de-escaping too much in query parser [2427](https://github.com/quickwit-oss/tantivy/pull/2427)(trinity-1686a)
- improve query parser [2416](https://github.com/quickwit-oss/tantivy/pull/2416)(trinity-1686a)
- Support field grouping `title:(return AND "pink panther")` [2333](https://github.com/quickwit-oss/tantivy/pull/2333)(trinity-1686a)

- add access benchmark for columnar [2432](https://github.com/quickwit-oss/tantivy/pull/2432)(PSeitz)
- extend indexwriter proptests [2342](https://github.com/quickwit-oss/tantivy/pull/2342)(PSeitz)
- add bench & test for columnar merging [2428](https://github.com/quickwit-oss/tantivy/pull/2428)(PSeitz)
- Change in Executor API [2391](https://github.com/quickwit-oss/tantivy/pull/2391)(fulmicoton)
- Removed usage of num_cpus [2387](https://github.com/quickwit-oss/tantivy/pull/2387)(fulmicoton)
- use bingang for agg and stacker benchmark [2378](https://github.com/quickwit-oss/tantivy/pull/2378)[#2492](https://github.com/quickwit-oss/tantivy/pull/2492)(PSeitz)
- cleanup top level exports [2382](https://github.com/quickwit-oss/tantivy/pull/2382)(PSeitz)
- make convert_to_fast_value_and_append_to_json_term pub [2370](https://github.com/quickwit-oss/tantivy/pull/2370)(PSeitz)
- remove JsonTermWriter [2238](https://github.com/quickwit-oss/tantivy/pull/2238)(PSeitz)
- validate sort by field type [2336](https://github.com/quickwit-oss/tantivy/pull/2336)(PSeitz)
- Fix trait bound of StoreReader::iter [2360](https://github.com/quickwit-oss/tantivy/pull/2360)(adamreichold)
- remove read_postings_no_deletes [2526](https://github.com/quickwit-oss/tantivy/pull/2526)(PSeitz)

================================

Tantivy 0.22 will be able to read indices created with Tantivy 0.21.

Bugfixes
- Fix null byte handling in JSON paths (null bytes in json keys caused panic during indexing) [2345](https://github.com/quickwit-oss/tantivy/pull/2345)(PSeitz)
- Fix bug that can cause `get_docids_for_value_range` to panic. [2295](https://github.com/quickwit-oss/tantivy/pull/2295)(fulmicoton)
- Avoid 1 document indices by increase min memory to 15MB for indexing [2176](https://github.com/quickwit-oss/tantivy/pull/2176)(PSeitz)
- Fix merge panic for JSON fields [2284](https://github.com/quickwit-oss/tantivy/pull/2284)(PSeitz)
- Fix bug occurring when merging JSON object indexed with positions. [2253](https://github.com/quickwit-oss/tantivy/pull/2253)(fulmicoton)
- Fix empty DateHistogram gap bug [2183](https://github.com/quickwit-oss/tantivy/pull/2183)(PSeitz)
- Fix range query end check (fields with less than 1 value per doc are affected) [2226](https://github.com/quickwit-oss/tantivy/pull/2226)(PSeitz)
- Handle exclusive out of bounds ranges on fastfield range queries [2174](https://github.com/quickwit-oss/tantivy/pull/2174)(PSeitz)

Breaking API Changes
- rename ReloadPolicy onCommit to onCommitWithDelay [2235](https://github.com/quickwit-oss/tantivy/pull/2235)(giovannicuccu)
- Move exports from the root into modules [2220](https://github.com/quickwit-oss/tantivy/pull/2220)(PSeitz)
- Accept field name instead of `Field` in FilterCollector [2196](https://github.com/quickwit-oss/tantivy/pull/2196)(PSeitz)
- remove deprecated IntOptions and DateTime [2353](https://github.com/quickwit-oss/tantivy/pull/2353)(PSeitz)

Features/Improvements
- Tantivy documents as a trait: Index data directly without converting to tantivy types first [2071](https://github.com/quickwit-oss/tantivy/pull/2071)(ChillFish8)
- encode some part of posting list as -1 instead of direct values (smaller inverted indices) [2185](https://github.com/quickwit-oss/tantivy/pull/2185)(trinity-1686a)
- **Aggregation**
- Support to deserialize f64 from string [2311](https://github.com/quickwit-oss/tantivy/pull/2311)(PSeitz)
- Add a top_hits aggregator [2198](https://github.com/quickwit-oss/tantivy/pull/2198)(ditsuke)
- Support bool type in term aggregation [2318](https://github.com/quickwit-oss/tantivy/pull/2318)(PSeitz)
- Support ip addresses in term aggregation [2319](https://github.com/quickwit-oss/tantivy/pull/2319)(PSeitz)
- Support date type in term aggregation [2172](https://github.com/quickwit-oss/tantivy/pull/2172)(PSeitz)
- Support escaped dot when addressing field [2250](https://github.com/quickwit-oss/tantivy/pull/2250)(PSeitz)

- Add ExistsQuery to check documents that have a value [2160](https://github.com/quickwit-oss/tantivy/pull/2160)(imotov)
- Expose TopDocs::order_by_u64_field again [2282](https://github.com/quickwit-oss/tantivy/pull/2282)(ditsuke)

- **Memory/Performance**
- Faster TopN: replace BinaryHeap with TopNComputer [2186](https://github.com/quickwit-oss/tantivy/pull/2186)(PSeitz)
- reduce number of allocations during indexing [2257](https://github.com/quickwit-oss/tantivy/pull/2257)(PSeitz)
- Less Memory while indexing: docid deltas while indexing [2249](https://github.com/quickwit-oss/tantivy/pull/2249)(PSeitz)
- Faster indexing: use term hashmap in fastfield [2243](https://github.com/quickwit-oss/tantivy/pull/2243)(PSeitz)
- term hashmap remove copy in is_empty, unused unordered_id [2229](https://github.com/quickwit-oss/tantivy/pull/2229)(PSeitz)
- add method to fetch block of first values in columnar [2330](https://github.com/quickwit-oss/tantivy/pull/2330)(PSeitz)
- Faster aggregations: add fast path for full columns in fetch_block [2328](https://github.com/quickwit-oss/tantivy/pull/2328)(PSeitz)
- Faster sstable loading: use fst for sstable index [2268](https://github.com/quickwit-oss/tantivy/pull/2268)(trinity-1686a)

- **QueryParser**
- allow newline where we allow space in query parser [2302](https://github.com/quickwit-oss/tantivy/pull/2302)(trinity-1686a)
- allow some mixing of occur and bool in strict query parser [2323](https://github.com/quickwit-oss/tantivy/pull/2323)(trinity-1686a)
- handle * inside term in lenient query parser [2228](https://github.com/quickwit-oss/tantivy/pull/2228)(trinity-1686a)
- add support for exists query syntax in query parser [2170](https://github.com/quickwit-oss/tantivy/pull/2170)(trinity-1686a)
- Add shared search executor [2312](https://github.com/quickwit-oss/tantivy/pull/2312)(MochiXu)
- Truncate keys to u16::MAX in term hashmap [2299](https://github.com/quickwit-oss/tantivy/pull/2299)(PSeitz)
- report if a term matched when warming up posting list [2309](https://github.com/quickwit-oss/tantivy/pull/2309)(trinity-1686a)
- Support json fields in FuzzyTermQuery [2173](https://github.com/quickwit-oss/tantivy/pull/2173)(PingXia-at)
- Read list of fields encoded in term dictionary for JSON fields [2184](https://github.com/quickwit-oss/tantivy/pull/2184)(PSeitz)
- add collect_block to BoxableSegmentCollector [2331](https://github.com/quickwit-oss/tantivy/pull/2331)(PSeitz)
- expose collect_block buffer size [2326](https://github.com/quickwit-oss/tantivy/pull/2326)(PSeitz)
- Forward regex parser errors [2288](https://github.com/quickwit-oss/tantivy/pull/2288)(adamreichold)
- Make FacetCounts defaultable and cloneable. [2322](https://github.com/quickwit-oss/tantivy/pull/2322)(adamreichold)
- Derive Debug for SchemaBuilder [2254](https://github.com/quickwit-oss/tantivy/pull/2254)(GodTamIt)
- add missing inlines to tantivy options [2245](https://github.com/quickwit-oss/tantivy/pull/2245)(PSeitz)

================================
Bugfixes
- Range queries on fast fields with less values on that field than documents had an invalid end condition, leading to missing results. [2226](https://github.com/quickwit-oss/tantivy/issues/2226)(appaquet PSeitz)
- Increase the minimum memory budget from 3MB to 15MB to avoid single doc segments (API fix). [2176](https://github.com/quickwit-oss/tantivy/issues/2176)(PSeitz)

================================
Bugfixes
- Fix track fast field memory consumption, which led to higher memory consumption than the budget allowed during indexing [2148](https://github.com/quickwit-oss/tantivy/issues/2148)[#2147](https://github.com/quickwit-oss/tantivy/issues/2147)(PSeitz)
- Fix a regression from 0.20 where sort index by date wasn't working anymore [2124](https://github.com/quickwit-oss/tantivy/issues/2124)(PSeitz)
- Fix getting the root facet on the `FacetCollector`. [2086](https://github.com/quickwit-oss/tantivy/issues/2086)(adamreichold)
- Align numerical type priority order of columnar and query. [2088](https://github.com/quickwit-oss/tantivy/issues/2088)(fmassot)
Breaking Changes
- Remove support for Brotli and Snappy compression [2123](https://github.com/quickwit-oss/tantivy/issues/2123)(adamreichold)
Features/Improvements
- Implement lenient query parser [2129](https://github.com/quickwit-oss/tantivy/pull/2129)(trinity-1686a)
- order_by_u64_field and order_by_fast_field allow sorting in ascending and descending order [2111](https://github.com/quickwit-oss/tantivy/issues/2111)(naveenann)
- Allow dynamic filters in text analyzer builder [2110](https://github.com/quickwit-oss/tantivy/issues/2110)(fulmicoton fmassot)
- **Aggregation**
- Add missing parameter for term aggregation [2149](https://github.com/quickwit-oss/tantivy/issues/2149)[#2103](https://github.com/quickwit-oss/tantivy/issues/2103)(PSeitz)
- Add missing parameter for percentiles [2157](https://github.com/quickwit-oss/tantivy/issues/2157)(PSeitz)
- Add missing parameter for stats,min,max,count,sum,avg [2151](https://github.com/quickwit-oss/tantivy/issues/2151)(PSeitz)
- Improve aggregation deserialization error message [2150](https://github.com/quickwit-oss/tantivy/issues/2150)(PSeitz)
- Add validation for type Bytes to term_agg [2077](https://github.com/quickwit-oss/tantivy/issues/2077)(PSeitz)
- Alternative mixed field collection [2135](https://github.com/quickwit-oss/tantivy/issues/2135)(PSeitz)
- Add missing query_terms impl for TermSetQuery. [2120](https://github.com/quickwit-oss/tantivy/issues/2120)(adamreichold)
- Minor improvements to OwnedBytes [2134](https://github.com/quickwit-oss/tantivy/issues/2134)(adamreichold)
- Remove allocations in split compound words [2080](https://github.com/quickwit-oss/tantivy/issues/2080)(PSeitz)
- Ngram tokenizer now returns an error with invalid arguments [2102](https://github.com/quickwit-oss/tantivy/issues/2102)(fmassot)
- Make TextAnalyzerBuilder public [2097](https://github.com/quickwit-oss/tantivy/issues/2097)(adamreichold)
- Return an error when tokenizer is not found while indexing [2093](https://github.com/quickwit-oss/tantivy/issues/2093)(naveenann)
- Delayed column opening during merge [2132](https://github.com/quickwit-oss/tantivy/issues/2132)(PSeitz)

================================
- Align numerical type priority order on the search side. [2088](https://github.com/quickwit-oss/tantivy/issues/2088) (fmassot)
- Fix is_child_of function not considering the root facet. [2086](https://github.com/quickwit-oss/tantivy/issues/2086) (adamreichhold)

================================
- Fix building on windows with mmap [2070](https://github.com/quickwit-oss/tantivy/issues/2070) (ChillFish8)