Tantivy

================================
Bugfixes
- Fix phrase queries with slop (slop supports now transpositions, algorithm that carries slop so far for num terms > 2) [2031](https://github.com/quickwit-oss/tantivy/issues/2031)[#2020](https://github.com/quickwit-oss/tantivy/issues/2020)(PSeitz)
- Handle error for exists on MMapDirectory [1988](https://github.com/quickwit-oss/tantivy/issues/1988) (PSeitz)
- Aggregation
- Fix min doc_count empty merge bug [2057](https://github.com/quickwit-oss/tantivy/issues/2057) (PSeitz)
- Fix: Sort order for term aggregations (sort order on key was inverted) [1858](https://github.com/quickwit-oss/tantivy/issues/1858) (PSeitz)

Features/Improvements
- Add PhrasePrefixQuery [1842](https://github.com/quickwit-oss/tantivy/issues/1842) (trinity-1686a)
- Add `coerce` option for text and numbers types (convert the value instead of returning an error during indexing) [1904](https://github.com/quickwit-oss/tantivy/issues/1904) (PSeitz)
- Add regex tokenizer [1759](https://github.com/quickwit-oss/tantivy/issues/1759)(mkleen)
- Move tokenizer API to separate crate. Having a separate crate with a stable API will allow us to use tokenizers with different tantivy versions. [1767](https://github.com/quickwit-oss/tantivy/issues/1767) (PSeitz)
- **Columnar crate**: New fast field handling (fulmicoton PSeitz) [1806](https://github.com/quickwit-oss/tantivy/issues/1806)[#1809](https://github.com/quickwit-oss/tantivy/issues/1809)
- Support for fast fields with optional values. Previously tantivy supported only single-valued and multi-value fast fields. The encoding of optional fast fields is now very compact.
- Fast field Support for JSON (schemaless fast fields). Support multiple types on the same column. [1876](https://github.com/quickwit-oss/tantivy/issues/1876) (fulmicoton)
- Unified access for fast fields over different cardinalities.
- Unified storage for typed and untyped fields.
- Move fastfield codecs into columnar. [1782](https://github.com/quickwit-oss/tantivy/issues/1782) (fulmicoton)
- Sparse dense index for optional values [1716](https://github.com/quickwit-oss/tantivy/issues/1716) (PSeitz)
- Switch to nanosecond precision in DateTime fastfield [2016](https://github.com/quickwit-oss/tantivy/issues/2016) (PSeitz)
- **Aggregation**
- Add `date_histogram` aggregation (only `fixed_interval` for now) [1900](https://github.com/quickwit-oss/tantivy/issues/1900) (PSeitz)
- Add `percentiles` aggregations [1984](https://github.com/quickwit-oss/tantivy/issues/1984) (PSeitz)
- [**breaking**] Drop JSON support on intermediate agg result (we use postcard as format in `quickwit` to send intermediate results) [1992](https://github.com/quickwit-oss/tantivy/issues/1992) (PSeitz)
- Set memory limit in bytes for aggregations after which they abort (Previously there was only the bucket limit) [1942](https://github.com/quickwit-oss/tantivy/issues/1942)[#1957](https://github.com/quickwit-oss/tantivy/issues/1957)(PSeitz)
- Add support for u64,i64,f64 fields in term aggregation [1883](https://github.com/quickwit-oss/tantivy/issues/1883) (PSeitz)
- Allow histogram bounds to be passed as Rfc3339 [2076](https://github.com/quickwit-oss/tantivy/issues/2076) (PSeitz)
- Add count, min, max, and sum aggregations [1794](https://github.com/quickwit-oss/tantivy/issues/1794) (guilload)
- Switch to Aggregation without serde_untagged => better deserialization errors. [2003](https://github.com/quickwit-oss/tantivy/issues/2003) (PSeitz)
- Switch to ms in histogram for date type (ES compatibility) [2045](https://github.com/quickwit-oss/tantivy/issues/2045) (PSeitz)
- Reduce term aggregation memory consumption [2013](https://github.com/quickwit-oss/tantivy/issues/2013) (PSeitz)
- Reduce agg memory consumption: Replace generic aggregation collector (which has a high memory requirement per instance) in aggregation tree with optimized versions behind a trait.
- Split term collection count and sub_agg (Faster term agg with less memory consumption for cases without sub-aggs) [1921](https://github.com/quickwit-oss/tantivy/issues/1921) (PSeitz)
- Schemaless aggregations: In combination with stacker tantivy supports now schemaless aggregations via the JSON type.
- Add aggregation support for JSON type [1888](https://github.com/quickwit-oss/tantivy/issues/1888) (PSeitz)
- Mixed types support on JSON fields in aggs [1971](https://github.com/quickwit-oss/tantivy/issues/1971) (PSeitz)
- Perf: Fetch blocks of vals in aggregation for all cardinality [1950](https://github.com/quickwit-oss/tantivy/issues/1950) (PSeitz)
- Allow histogram bounds to be passed as Rfc3339 [2076](https://github.com/quickwit-oss/tantivy/issues/2076) (PSeitz)
- `Searcher` with disabled scoring via `EnableScoring::Disabled` [1780](https://github.com/quickwit-oss/tantivy/issues/1780) (shikhar)
- Enable tokenizer on json fields [2053](https://github.com/quickwit-oss/tantivy/issues/2053) (PSeitz)
- Enforcing "NOT" and "-" queries consistency in UserInputAst [1609](https://github.com/quickwit-oss/tantivy/issues/1609) (bazhenov)
- Faster indexing
- Refactor tokenization pipeline to use GATs [1924](https://github.com/quickwit-oss/tantivy/issues/1924) (trinity-1686a)
- Faster term hash map [2058](https://github.com/quickwit-oss/tantivy/issues/2058)[#1940](https://github.com/quickwit-oss/tantivy/issues/1940) (PSeitz)
- tokenizer-api: reduce Tokenizer allocation overhead [2062](https://github.com/quickwit-oss/tantivy/issues/2062) (PSeitz)
- Refactor vint [2010](https://github.com/quickwit-oss/tantivy/issues/2010) (PSeitz)
- Faster search
- Work in batches of docs on the SegmentCollector (Only for cases without score for now) [1937](https://github.com/quickwit-oss/tantivy/issues/1937) (PSeitz)
- Faster fast field range queries using SIMD [1954](https://github.com/quickwit-oss/tantivy/issues/1954) (fulmicoton)
- Improve fast field range query performance [1864](https://github.com/quickwit-oss/tantivy/issues/1864) (PSeitz)
- Make BM25 scoring more flexible [1855](https://github.com/quickwit-oss/tantivy/issues/1855) (alexcole)
- Switch fs2 to fs4 as it is now unmaintained and does not support illumos [1944](https://github.com/quickwit-oss/tantivy/issues/1944) (Toasterson)
- Made BooleanWeight and BoostWeight public [1991](https://github.com/quickwit-oss/tantivy/issues/1991) (fulmicoton)
- Make index compatible with virtual drives on Windows [1843](https://github.com/quickwit-oss/tantivy/issues/1843) (gyk)
- Add stop words for Hungarian language [2069](https://github.com/quickwit-oss/tantivy/issues/2069) (tnxbutno)
- Auto downgrade index record option, instead of vint error [1857](https://github.com/quickwit-oss/tantivy/issues/1857) (PSeitz)
- Enable range query on fast field for u64 compatible types [1762](https://github.com/quickwit-oss/tantivy/issues/1762) (PSeitz) [#1876]
- sstable
- Isolating sstable and stacker in independent crates. [1718](https://github.com/quickwit-oss/tantivy/issues/1718) (fulmicoton)
- New sstable format [1943](https://github.com/quickwit-oss/tantivy/issues/1943)[#1953](https://github.com/quickwit-oss/tantivy/issues/1953) (trinity-1686a)
- Use DeltaReader directly to implement Dictionary::ord_to_term [1928](https://github.com/quickwit-oss/tantivy/issues/1928) (trinity-1686a)
- Use DeltaReader directly to implement Dictionary::term_ord [1925](https://github.com/quickwit-oss/tantivy/issues/1925) (trinity-1686a)
- Add separate tokenizer manager for fast fields [2019](https://github.com/quickwit-oss/tantivy/issues/2019) (PSeitz)
- Make construction of LevenshteinAutomatonBuilder for FuzzyTermQuery instances lazy. [1756](https://github.com/quickwit-oss/tantivy/issues/1756) (adamreichold)
- Added support for madvise when opening an mmapped Index [2036](https://github.com/quickwit-oss/tantivy/issues/2036) (fulmicoton)
- Rename `DatePrecision` to `DateTimePrecision` [2051](https://github.com/quickwit-oss/tantivy/issues/2051) (guilload)
- Query Parser
- Quotation mark can now be used for phrase queries. [2050](https://github.com/quickwit-oss/tantivy/issues/2050) (fulmicoton)
- PhrasePrefixQuery is supported in the query parser via: `field:"phrase ter"*` [2044](https://github.com/quickwit-oss/tantivy/issues/2044) (adamreichold)
- Docs
- Update examples for literate docs [1880](https://github.com/quickwit-oss/tantivy/issues/1880) (PSeitz)
- Add ip field example [1775](https://github.com/quickwit-oss/tantivy/issues/1775) (PSeitz)
- Fix doc store cache documentation [1821](https://github.com/quickwit-oss/tantivy/issues/1821) (PSeitz)
- Fix BooleanQuery document [1999](https://github.com/quickwit-oss/tantivy/issues/1999) (RT_Enzyme)
- Update comments in the faceted search example [1737](https://github.com/quickwit-oss/tantivy/issues/1737) (DawChihLiou)

================================
Bugfixes
- Fix missing fieldnorms for u64, i64, f64, bool, bytes and date [1620](https://github.com/quickwit-oss/tantivy/pull/1620) (PSeitz)
- Fix interpolation overflow in linear interpolation fastfield codec [1480](https://github.com/quickwit-oss/tantivy/pull/1480) (PSeitz fulmicoton)

Features/Improvements
- Add support for `IN` in queryparser , e.g. `field: IN [val1 val2 val3]` [1683](https://github.com/quickwit-oss/tantivy/pull/1683) (trinity-1686a)
- Skip score calculation, when no scoring is required [1646](https://github.com/quickwit-oss/tantivy/pull/1646) (PSeitz)
- Limit fast fields to u32 (`get_val(u32)`) [1644](https://github.com/quickwit-oss/tantivy/pull/1644) (PSeitz)
- The `DateTime` type has been updated to hold timestamps with microseconds precision.
`DateOptions` and `DatePrecision` have been added to configure Date fields. The precision is used to hint on fast values compression. Otherwise, seconds precision is used everywhere else (i.e terms, indexing) [1396](https://github.com/quickwit-oss/tantivy/pull/1396) (evanxg852000)
- Add IP address field type [1553](https://github.com/quickwit-oss/tantivy/pull/1553) (PSeitz)
- Add boolean field type [1382](https://github.com/quickwit-oss/tantivy/pull/1382) (boraarslan)
- Remove Searcher pool and make `Searcher` cloneable. (PSeitz)
- Validate settings on create [1570](https://github.com/quickwit-oss/tantivy/pull/1570) (PSeitz)
- Detect and apply gcd on fastfield codecs [1418](https://github.com/quickwit-oss/tantivy/pull/1418) (PSeitz)
- Doc store
- use separate thread to compress block store [1389](https://github.com/quickwit-oss/tantivy/pull/1389) [#1510](https://github.com/quickwit-oss/tantivy/pull/1510) (PSeitz fulmicoton)
- Expose doc store cache size [1403](https://github.com/quickwit-oss/tantivy/pull/1403) (PSeitz)
- Enable compression levels for doc store [1378](https://github.com/quickwit-oss/tantivy/pull/1378) (PSeitz)
- Make block size configurable [1374](https://github.com/quickwit-oss/tantivy/pull/1374) (kryesh)
- Make `tantivy::TantivyError` cloneable [1402](https://github.com/quickwit-oss/tantivy/pull/1402) (PSeitz)
- Add support for phrase slop in query language [1393](https://github.com/quickwit-oss/tantivy/pull/1393) (saroh)
- Aggregation
- Add aggregation support for date type [1693](https://github.com/quickwit-oss/tantivy/pull/1693)(PSeitz)
- Add support for keyed parameter in range and histogram aggregations [1424](https://github.com/quickwit-oss/tantivy/pull/1424) (k-yomo)
- Add aggregation bucket limit [1363](https://github.com/quickwit-oss/tantivy/pull/1363) (PSeitz)
- Faster indexing
- [1610](https://github.com/quickwit-oss/tantivy/pull/1610) (PSeitz)
- [1594](https://github.com/quickwit-oss/tantivy/pull/1594) (PSeitz)
- [1582](https://github.com/quickwit-oss/tantivy/pull/1582) (PSeitz)
- [1611](https://github.com/quickwit-oss/tantivy/pull/1611) (PSeitz)
- Added a pre-configured stop word filter for various language [1666](https://github.com/quickwit-oss/tantivy/pull/1666) (adamreichold)

================================
- Hotfix: positions computation. 1629 (fmassot, fulmicoton, PSeitz)

================================

- For date values `chrono` has been replaced with `time` (uklotzde) 1304 :
- The `time` crate is re-exported as `tantivy::time` instead of `tantivy::chrono`.
- The type alias `tantivy::DateTime` has been removed.
- `Value::Date` wraps `time::PrimitiveDateTime` without time zone information.
- Internally date/time values are stored as seconds since UNIX epoch in UTC.
- Converting a `time::OffsetDateTime` to `Value::Date` implicitly converts the value into UTC.
If this is not desired do the time zone conversion yourself and use `time::PrimitiveDateTime`
directly instead.
- Add [histogram](https://github.com/quickwit-oss/tantivy/pull/1306) aggregation (PSeitz)
- Add support for fastfield on text fields (PSeitz)
- Add terms aggregation (PSeitz)
- Add support for zstd compression (kryesh)

================================

- LogMergePolicy now triggers merges if the ratio of deleted documents reaches a threshold (shikhar fulmicoton) [115](https://github.com/quickwit-oss/tantivy/issues/115)
- Adds a searcher Warmer API (shikhar fulmicoton)
- Change to non-strict schema. Ignore fields in data which are not defined in schema. Previously this returned an error. 1211
- Facets are necessarily indexed. Existing index with indexed facets should work out of the box. Index without facets that are marked with index: false should be broken (but they were already broken in a sense). (fulmicoton) 1195 .
- Bugfix that could in theory impact durability in theory on some filesystems [1224](https://github.com/quickwit-oss/tantivy/issues/1224)
- Schema now offers not indexing fieldnorms (lpouget) [922](https://github.com/quickwit-oss/tantivy/issues/922)
- Reduce the number of fsync calls [1225](https://github.com/quickwit-oss/tantivy/issues/1225)
- Fix opening bytes index with dynamic codec (PSeitz) [1278](https://github.com/quickwit-oss/tantivy/issues/1278)
- Added an aggregation collector for range, average and stats compatible with Elasticsearch. (PSeitz)
- Added a JSON schema type fulmicoton [1251](https://github.com/quickwit-oss/tantivy/issues/1251)
- Added support for slop in phrase queries halvorboe [1068](https://github.com/quickwit-oss/tantivy/issues/1068)

================================

- Bugfix in FuzzyTermQuery. (transposition_cost_one was not doing anything)