Milvus

Features

- [cherry-pick] changes to propagate traceid from client ([34640](https://github.com/milvus-io/milvus/pull/34640))

Bug fixes

- Fix a security vulnerability ([36155](https://github.com/milvus-io/milvus/pull/36155))
- Fix proxy panic after getimportstate failed ([35541](https://github.com/milvus-io/milvus/pull/35541))

Miluvs v2.3.21 addresses a critical bug in data deletion ([35390](https://github.com/milvus-io/milvus/pull/35390)) and introduces some improvements in metrics and performance.

Improvements

- Enhanced data deletion process by modifying the delegator's delete buffer handling ([35074](https://github.com/milvus-io/milvus/pull/35074))
- Added metrics to track disk quota usage ([35320](https://github.com/milvus-io/milvus/pull/35320))
- Reduced delegator memory usage by adjusting the overloaded factor to 0.1 ([35165](https://github.com/milvus-io/milvus/pull/35165))
- Improved health check by skipping manually stopped components ([35122](https://github.com/milvus-io/milvus/pull/35122))
- Added metric to record maxinsertrate and querynodememoryhighwaterlevel ([35193](https://github.com/milvus-io/milvus/pull/35193))
- Upgraded the Conan version ([35217](https://github.com/milvus-io/milvus/pull/35217))
- Changed the default metric type for autoindex ([34328](https://github.com/milvus-io/milvus/pull/34328))
- Sped up the datacoord gc quit process ([35057](https://github.com/milvus-io/milvus/pull/35057))
- Avoided panic due to nil schema ([35065](https://github.com/milvus-io/milvus/pull/35065))
- Reduced duplicate primary keys in the segcore component ([35291](https://github.com/milvus-io/milvus/pull/35291))
- Tracked the number of times Milvus enters a force-deny-writing state ([34990](https://github.com/milvus-io/milvus/pull/34990))

Bug fixes

- Resolved an issue with the BloomFilter returning false negatives when the statslog contained multiple K values, which could occur when a delete operation failed to apply ([35390](https://github.com/milvus-io/milvus/pull/35390))
- Enabled a limiter for RESTful server ([35354](https://github.com/milvus-io/milvus/pull/35354))
- Added a retry mechanism to avoid getting incomplete query results ([35061](https://github.com/milvus-io/milvus/pull/35061))
- Fixed issues with delete-by-expression operations failing during retry progress ([35421](https://github.com/milvus-io/milvus/pull/35421))
- Addressed a problem where the datanode binlog size metrics included dropped segments ([35420](https://github.com/milvus-io/milvus/pull/35420))
- Resolved comparison operations between incompatible operands ([35308](https://github.com/milvus-io/milvus/pull/35308))
- Fixed an issue where the datanode could not watch the channel ([35136](https://github.com/milvus-io/milvus/pull/35136))
- Addressed a deadlock issue on compaction when stopping the datanode ([35199](https://github.com/milvus-io/milvus/pull/35199))

Milvus v2.3.20 brings several important bug fixes focused on enhancing segment management. This release also includes an upgrade to Go version 1.21, which allows the pprof tool to capture C++ code profiles, significantly aiding in diagnosing Milvus. Additionally, the Knowhere version has been updated to v2.2.7, enabling Milvus to be compiled with glibc versions ≤ 2.30. A new metric has been introduced to record the duration that requests wait in the proxy queue, improving Milvus's observability.

Features

- Added a metric to record the duration of requests waiting in the proxy queue ([34791](https://github.com/milvus-io/milvus/pull/34791)).

Improvements

- Upgraded Knowhere to version 2.2.7 ([34825](https://github.com/milvus-io/milvus/pull/34825)).
- Upgraded Go version from 1.20 to 1.21 ([34759](https://github.com/milvus-io/milvus/pull/34759)).

Bug fixes

- Initialized the hook when creating the proxy object ([34936](https://github.com/milvus-io/milvus/pull/34936)).
- Fixed a parsing issue for the plan proto for search types ([34945](https://github.com/milvus-io/milvus/pull/34945)).
- Resolved an issue where segments could bounce between delegator and worker ([34889](https://github.com/milvus-io/milvus/pull/34889)).
- Prevented the generation of reduce channel tasks before channel balancing is complete ([34721](https://github.com/milvus-io/milvus/pull/34721)).
- Avoided segment shortages caused by deduplicated segment tasks ([34882](https://github.com/milvus-io/milvus/pull/34882)).
- Discarded compaction plans when closing DataSyncService ([34737](https://github.com/milvus-io/milvus/pull/34737)).
- Fixed a typo in the compatible ascending index ([34712](https://github.com/milvus-io/milvus/pull/34712)).

Milvus v2.3.19 introduces significant improvements and bug fixes to enhance memory management, error handling, and query performance. This release continues to bolster the robustness and functionality of Milvus, ensuring optimal performance and user experience.

Improvements

- Preserved fixed-size memory in delegator node for growing segments ([34602](https://github.com/milvus-io/milvus/pull/34602)).
- Refined Prometheus Buckets at different scales to improve monitoring precision ([34627](https://github.com/milvus-io/milvus/pull/34627)).
- Avoided assigning too many segments/channels to new QueryNode, ensuring balanced load distribution ([34461](https://github.com/milvus-io/milvus/pull/34461)).
- Provided clearer error messages when the string/varchar exceeded its max length ([34324](https://github.com/milvus-io/milvus/pull/34324), [#34034](https://github.com/milvus-io/milvus/pull/34034)).
- Prevented memory slice increase during queries, optimizing memory usage ([34256](https://github.com/milvus-io/milvus/pull/34256)).
- Sped up the loading process for small collections, reducing latency ([33863](https://github.com/milvus-io/milvus/pull/33863)).

Bug fixes

- Fixed error message when field name is invalid, improving user feedback ([33901](https://github.com/milvus-io/milvus/pull/33901)).
- Ensured Segment Manager filters flushed segments correctly during import ([34650](https://github.com/milvus-io/milvus/pull/34650)).
- Made knowhere-build-pool-size configurable on QueryNode for better resource management ([34647](https://github.com/milvus-io/milvus/pull/34647)).
- Eliminated unnecessary vchannels' merge when curTS is 0, optimizing performance ([34626](https://github.com/milvus-io/milvus/pull/34626)).
- Fixed issue where IndexNode couldn't be stopped due to a missing lifetime end ([34560](https://github.com/milvus-io/milvus/pull/34560)).
- Enhanced RESTful API query to be more user-friendly ([34447](https://github.com/milvus-io/milvus/pull/34447)).
- Checked load state of all partitions instead of just the first one, ensuring completeness ([34321](https://github.com/milvus-io/milvus/pull/34321)).
- Implemented singleflight for segcore chunkcache to avoid redundant operations ([34284](https://github.com/milvus-io/milvus/pull/34284)).
- Broadcasted collection's new properties to DataCoord, ensuring consistency ([34147](https://github.com/milvus-io/milvus/pull/34147)).
- DataCoord now returns checkpoint after successfully flushing segments ([34115](https://github.com/milvus-io/milvus/pull/34115)).
- Checked nodeid wildcard when removing pkoracle, preventing potential errors ([34022](https://github.com/milvus-io/milvus/pull/34022)).
- Ensured queries by primary key return the latest row rather than historical data, maintaining data accuracy ([34026](https://github.com/milvus-io/milvus/pull/34026)).

Milvus v2.3.18 includes several bug fixes and improvements to enhance query result quality and maintainability. This release also backports improvements and fixes from the 2.4 branch, which enhance control over delete data amounts (according to rate limit configuration) and increase Bloom filter processing speed. As with previous releases, this update reinforces Milvus's commitment to functionality and robustness.

Improvements

- Added RESTful APIs to trigger component stop ([33798](https://github.com/milvus-io/milvus/pull/33798)).
- Improved delete by expression to respect delete rate limit rules according to the matched data size instead of delete request size ([33794](https://github.com/milvus-io/milvus/pull/33794)).
- Accelerated Bloom filter processing by submitting in batch and working in parallel ([33870](https://github.com/milvus-io/milvus/pull/33870)).
- Enabled parallel processing for `applydelete` at the segment level ([33841](https://github.com/milvus-io/milvus/pull/33841)).
- Printed `UseDefaultConsistency` parameter in read requests ([33645](https://github.com/milvus-io/milvus/pull/33645)).

Bug fixes

- Fixed a bug where closure capture iteration variable caused delete data to fail to apply to the correct data ([33912](https://github.com/milvus-io/milvus/pull/33912)).
- Fixed a bug where filtered results could be less than topk by upgrading Knowhere to 2.2.10 ([33930](https://github.com/milvus-io/milvus/pull/33930)).
- Fixed a bug where insert rate was not limited when `collection.insertrate.max.mb` config was set to 0 ([33725](https://github.com/milvus-io/milvus/pull/33725)).
- Fixed a bug where old collections created before v2.2.8 could reappear and become zombies after being dropped and restarted ([33695](https://github.com/milvus-io/milvus/pull/33695)).
- Fixed a bug where upsert latency used the wrong unit, causing abnormal metrics ([33580](https://github.com/milvus-io/milvus/pull/33580)).
- Fixed a bug where Milvus exception info could not be passed outside of segcore ([33395](https://github.com/milvus-io/milvus/pull/33395)).

Milvus version 2.3.17 primarily introduces several bug fixes to improve query accuracy, cluster reliability, and privilege maintenance. The most notable fix ensures that query iterators in Milvus return the correct number of results, rather than missing some data. This release reinforces Milvus's commitment to functionality and robustness.

Bug fixes

- Fixed issue where query iterators missed some results when upserts and deletes were performed very frequently ([33468](https://github.com/milvus-io/milvus/pull/33468))
- Fixed issue where the cluster could block during startup due to improper handling of standby status ([33371](https://github.com/milvus-io/milvus/pull/33371))
- Fixed issue where specifying the database name when granting privileges failed ([33292](https://github.com/milvus-io/milvus/pull/33292))