Milvus

Milvus 2.4.15 was a critical bug-fix release focused on enhancing system stability, performance, and compatibility. This version addressed a major deadlock issue that could occur during QueryNode crashes and introduced compatibility updates for the backup tool with the database feature. Additionally, Milvus 2.4.15 improved delete performance and stability through significant optimizations in L0 handling. **Upgrading to v2.4.15 was strongly recommended** to benefit from these critical enhancements.

Critical bug fixes

- Resolved a deadlock issue if the QueryNode crashed during shard client initialization ([37354](https://github.com/milvus-io/milvus/pull/37354)).
- Reverted the enhancement to support databases for bulk insert ([37421](https://github.com/milvus-io/milvus/pull/37421)).

Bug fixes

- Fixed a bug where certain expressions did not correctly parse values ([37342](https://github.com/milvus-io/milvus/pull/37342)).
- Enhanced the Proxy to retry getting the shard leader on unloaded collections ([37326](https://github.com/milvus-io/milvus/pull/37326)).
- Corrected an issue where the L0 row count metrics value was always empty ([37307](https://github.com/milvus-io/milvus/pull/37307)).
- Skipped marking compaction timeout for mixed and L0 compaction scenarios ([37194](https://github.com/milvus-io/milvus/pull/37194)).
- Rectified the containment logic of OffsetOrderedArray ([37309](https://github.com/milvus-io/milvus/pull/37309)).
- Added a check for resources when loading delta logs ([37263](https://github.com/milvus-io/milvus/pull/37263)).

Improvements

- Moved L0 logic outside of the delta lock for better performance ([37340](https://github.com/milvus-io/milvus/pull/37340)).
- Released compacted growing segments if present in the dropped list ([37266](https://github.com/milvus-io/milvus/pull/37266)).
- Introduced middleware to monitor RESTful V2 input/output RPC stats ([37224](https://github.com/milvus-io/milvus/pull/37224), [#37440](https://github.com/milvus-io/milvus/pull/37440)).

Milvus 2.4.14 addresses a critical issue from version 2.4.13 that could cause collection information to be lost after `snapshotKV` garbage collection. It also fixed a couple of resource leaks. Additionally, this release includes numerous enhancements focused on improving stability in large-scale delete operations and compaction performance.

Features

- Supported memory mode chunk cache ([35836](https://github.com/milvus-io/milvus/pull/35836))
- Supported db for bulkinsert ([37017](https://github.com/milvus-io/milvus/pull/37017))

Improvements

- Delete/Compaction Optimization
- Enabled parallel execution of l0 compactions ([36985](https://github.com/milvus-io/milvus/pull/36985))
- Batched forward delete when using direct forward ([37107](https://github.com/milvus-io/milvus/pull/37107))
- Skipped loading delta data in delegater when using remoteload ([37112](https://github.com/milvus-io/milvus/pull/37112))
- Directly forwarded delta excluding l0 segments ([36914](https://github.com/milvus-io/milvus/pull/36914))
- Added prioritization of compaction tasks in DataCoord ([36979](https://github.com/milvus-io/milvus/pull/36979))
- Tracked complex delete rates ([36958](https://github.com/milvus-io/milvus/pull/36958))
- Refactored CreateCollection in RESTFul API ([36885](https://github.com/milvus-io/milvus/pull/36885))
- Fused multiple 'and' and 'or' operations into a single op ([36973](https://github.com/milvus-io/milvus/pull/36973))
- Made skip load work for all branches ([37161](https://github.com/milvus-io/milvus/pull/37161))
- Upgraded Minio dependency to support EKS Pod Identities ([37089](https://github.com/milvus-io/milvus/pull/37089))
- Tidied import options ([37078](https://github.com/milvus-io/milvus/pull/37078))
- Limited maximum number of import jobs ([36892](https://github.com/milvus-io/milvus/pull/36892))
- Preallocated data slice to avoid re-allocating memory ([37044](https://github.com/milvus-io/milvus/pull/37044))
- Prevented DataNode from loading the bf ([37027](https://github.com/milvus-io/milvus/pull/37027))
- Avoided limiting ddl operations repeatedly ([37011](https://github.com/milvus-io/milvus/pull/37011))
- Made the configuration item `datanode.import.maxconcurrenttasknum` dynamically adjustable ([37103](https://github.com/milvus-io/milvus/pull/37103))
- Used `queryNode.mmap.growingMmapEnabled` to control the behavior of interim index ([36391](https://github.com/milvus-io/milvus/pull/36391))
- Populated the `Level` and `StartPosition` fields in segmentLoadInfo of growing segment ([36911](https://github.com/milvus-io/milvus/pull/36911))
- Forced to stop buffer messages when receiving the drop collection message ([36917](https://github.com/milvus-io/milvus/pull/36917))
- Added metrics for querynode delete buffer info ([37097](https://github.com/milvus-io/milvus/pull/37097))
- Added collection name label for some metric ([37159](https://github.com/milvus-io/milvus/pull/37159))
- Used middleware to observe RESTful v2 in/out rpc stats ([37224](https://github.com/milvus-io/milvus/pull/37224))
- Changed GPU default memory pool size ([36969](https://github.com/milvus-io/milvus/pull/36969))
- Updated Knowhere version to 2.3.12 ([37132](https://github.com/milvus-io/milvus/pull/37132))
- Allowed deleting data when disk quota exhausted ([37139](https://github.com/milvus-io/milvus/pull/37139))

Bug fixes

- Fixed collection info that could not be recovered from metakv after restart if all snapshots were garbage collected ([36950](https://github.com/milvus-io/milvus/pull/36950))
- Corrected the rpc error code to avoid invalid retry in client ([37025](https://github.com/milvus-io/milvus/pull/37025))
- Ignored db not found error in quota center ([36850](https://github.com/milvus-io/milvus/pull/36850))
- Fixed goroutine leakage in QueryNode by using singleton delete pool ([37225](https://github.com/milvus-io/milvus/pull/37225))
- Fixed collection leak in querynode ([37079](https://github.com/milvus-io/milvus/pull/37079))
- Fixed leakage of clustering compaction task ([36803](https://github.com/milvus-io/milvus/pull/36803))
- Prohibited renaming a collection that had an alias ([37208](https://github.com/milvus-io/milvus/pull/37208))
- Made sure alias was cached ([36808](https://github.com/milvus-io/milvus/pull/36808))
- Search/query could have failed during updating delegator cache ([37174](https://github.com/milvus-io/milvus/pull/37174))
- Excluded l0 compaction when clustering was executing ([37142](https://github.com/milvus-io/milvus/pull/37142))
- Referenced collection meta when loading l0 segment meta only ([37179](https://github.com/milvus-io/milvus/pull/37179))
- Delegator might have become unserviceable after querycoord restart ([37100](https://github.com/milvus-io/milvus/pull/37100))
- Dynamic release partition might have failed search/query ([37099](https://github.com/milvus-io/milvus/pull/37099))
- Rectified delete buffer row count quota value ([37068](https://github.com/milvus-io/milvus/pull/37068))
- Passed full field list when partial load enabled ([37063](https://github.com/milvus-io/milvus/pull/37063))
- Query node panic occurred during sending rpc to worker ([36988](https://github.com/milvus-io/milvus/pull/36988))
- Datacoord got stuck at stopping progress ([36961](https://github.com/milvus-io/milvus/pull/36961))
- Fixed the out-of-bounds access in the growing segment when raw data was replaced by interim index ([36938](https://github.com/milvus-io/milvus/pull/36938))
- Rootcoord got stuck at graceful stop progress ([36881](https://github.com/milvus-io/milvus/pull/36881))



v2.4.13-hotfix
v2.4.13-hotfix

Release Date: October 17, 2024

| Milvus version | Python SDK version | Java SDK version | Node.js SDK version |
|----------------|--------------------|------------------|---------------------|

Milvus 2.4.13 introduces dynamic replica load, allowing users to adjust the number of collection replicas without needing to release and reload the collection. This version also addresses several critical bugs related to bulk importing, expression parsing, load balancing, and failure recovery. Additionally, significant improvements have been made to MMAP resource usage and import performance, enhancing overall system efficiency. We highly recommend upgrading to this release for better performance and stability.

Features

- Dynamic replica adjustment for loaded collections ([36417](https://github.com/milvus-io/milvus/pull/36417))
- Sparse vector MMAP in growing segment types ([36565](https://github.com/milvus-io/milvus/pull/36565))

Bug fixes

- Fixed a flush performance issue ([36741](https://github.com/milvus-io/milvus/pull/36741))
- Fixed a bug with JSON expressions in "[]" ([36722](https://github.com/milvus-io/milvus/pull/36722))
- Removed neighbors if compact target is unindexed ([36694](https://github.com/milvus-io/milvus/pull/36694))
- Improved performance for Rocksmq when channel is full ([36618](https://github.com/milvus-io/milvus/pull/36618))
- Fixed an issue where errors during unpinning were not deferred ([36665](https://github.com/milvus-io/milvus/pull/36665))
- Resolved a memory leak for imported segments in the segment manager ([36631](https://github.com/milvus-io/milvus/pull/36631))
- Skipped unnecessary health checks for query nodes in the proxy ([36553](https://github.com/milvus-io/milvus/pull/36553))
- Fixed an overflow issue with term expressions ([36534](https://github.com/milvus-io/milvus/pull/36534))
- Recorded node ID before assigning tasks to prevent task misallocation ([36493](https://github.com/milvus-io/milvus/pull/36493))
- Resolved data race issues in clustering compaction ([36499](https://github.com/milvus-io/milvus/pull/36499))
- Added a check for string array max length after type matching ([36497](https://github.com/milvus-io/milvus/pull/36497))
- Addressed race conditions in mix or standalone mode ([36459](https://github.com/milvus-io/milvus/pull/36459))
- Fixed segment imbalance after repeated load and release operations ([36543](https://github.com/milvus-io/milvus/pull/36543))
- Corrected a corner case where segments couldn't be moved from a stopping node ([36475](https://github.com/milvus-io/milvus/pull/36475))
- Updated segment info properly even if some segments were missing ([36729](https://github.com/milvus-io/milvus/pull/36729))
- Prevented etcd transactions from exceeding the max limit in snapshot KV ([36773](https://github.com/milvus-io/milvus/pull/36773))

Improvements

- Enhanced MMAP resource estimation:
- Improved MMAP-related code in column.h ([36521](https://github.com/milvus-io/milvus/pull/36521))
- Refined resource estimation when loading collections ([36728](https://github.com/milvus-io/milvus/pull/36728))
- Performance Enhancements:
- Improved expression parsing efficiency by converting Unicode to ASCII ([36676](https://github.com/milvus-io/milvus/pull/36676))
- Enabled parallel production of messages for multiple topics ([36462](https://github.com/milvus-io/milvus/pull/36462))
- Reduced CPU overhead when calculating index file size ([36580](https://github.com/milvus-io/milvus/pull/36580))
- Retrieved message type from header to minimize unmarshalling ([36454](https://github.com/milvus-io/milvus/pull/36454))
- Optimized workload-based replica selection policy ([36384](https://github.com/milvus-io/milvus/pull/36384))
- Split delete task messages to fit within max message size limits ([36574](https://github.com/milvus-io/milvus/pull/36574))
- Added new RESTful URL to describe import jobs ([36754](https://github.com/milvus-io/milvus/pull/36754))
- Optimized import scheduling and added a time cost metric ([36684](https://github.com/milvus-io/milvus/pull/36684))
- Added balance report log for query coordinator balancer ([36749](https://github.com/milvus-io/milvus/pull/36749))
- Switched to using common GC configuration ([36670](https://github.com/milvus-io/milvus/pull/36670))
- Added streaming forward policy switch for delegator ([36712](https://github.com/milvus-io/milvus/pull/36712))
- Enabled manual compaction for collections without indexes ([36581](https://github.com/milvus-io/milvus/pull/36581))
- Enabled load balancing on query nodes with varying memory capacities ([36625](https://github.com/milvus-io/milvus/pull/36625))
- Unified case for inbound labels using metrics.label ([36616](https://github.com/milvus-io/milvus/pull/36616))
- Made transfer channel/segment operations idempotent ([36552](https://github.com/milvus-io/milvus/pull/36552))
- Added metrics to monitor import throughput and imported row count ([36588](https://github.com/milvus-io/milvus/pull/36588))
- Prevented creation of multiple timer objects in targets ([36573](https://github.com/milvus-io/milvus/pull/36573))
- Updated expression version and formatted HTTP response for expressions ([36467](https://github.com/milvus-io/milvus/pull/36467))
- Enhanced garbage collection in snapshot KV ([36793](https://github.com/milvus-io/milvus/pull/36793))
- Added support to execute methods with context parameters ([36798](https://github.com/milvus-io/milvus/pull/36798))

Milvus 2.4.12 introduces significant enhancements and critical bug fixes. This release addresses data duplication issues and improves failure recovery speed, particularly when handling large numbers of deletions. However, a known issue persists where failure recovery can be slow when deleting massive amounts of data. We are actively working on resolving this issue.

Improvements

- Implemented graceful stop for flowgraph manager ([36358](https://github.com/milvus-io/milvus/pull/36358))
- Disabled index checks for non-loaded vector fields ([36280](https://github.com/milvus-io/milvus/pull/36280))
- Filtered out non-hit delete records during delta loading ([36272](https://github.com/milvus-io/milvus/pull/36272))
- Improved error handling for std::stoi exceptions ([36296](https://github.com/milvus-io/milvus/pull/36296))
- Disallowed keywords as field names or dynamic field names ([36108](https://github.com/milvus-io/milvus/pull/36108))
- Added metrics for delete entries in L0 segments ([36227](https://github.com/milvus-io/milvus/pull/36227))
- Implemented L0 forward policy to support remote loading ([36208](https://github.com/milvus-io/milvus/pull/36208))
- Added ANN field loading check in proxy ([36194](https://github.com/milvus-io/milvus/pull/36194))
- Enabled empty sparse row support ([36061](https://github.com/milvus-io/milvus/pull/36061))
- Fixed a security vulnerability ([36156](https://github.com/milvus-io/milvus/pull/36156))
- Implemented stats handler for request/response size metrics ([36118](https://github.com/milvus-io/milvus/pull/36118))
- Corrected size estimation for encoded array data ([36379](https://github.com/milvus-io/milvus/pull/36379))

Bug fixes

- Resolved metric type errors for collections with two vector fields ([36473](https://github.com/milvus-io/milvus/pull/36473))
- Fixed long buffering issues causing message queue reception failures ([36425](https://github.com/milvus-io/milvus/pull/36425))
- Implemented proper compact-to-segments return after split support ([36429](https://github.com/milvus-io/milvus/pull/36429))
- Resolved data race issues with node ID check goroutine ([36377](https://github.com/milvus-io/milvus/pull/36377))
- Removed element type check ([36324](https://github.com/milvus-io/milvus/pull/36324))
- Fixed concurrent access issues for growing and sealed segments ([36288](https://github.com/milvus-io/milvus/pull/36288))
- Implemented future stateful lock ([36333](https://github.com/milvus-io/milvus/pull/36333))
- Corrected offset usage in HybridSearch ([36287](https://github.com/milvus-io/milvus/pull/36287), [#36253](https://github.com/milvus-io/milvus/pull/36253))
- Resolved dirty segment/channel leaks on QueryNode ([36259](https://github.com/milvus-io/milvus/pull/36259))
- Fixed primary key duplication handling ([36274](https://github.com/milvus-io/milvus/pull/36274))
- Enforced metric type setting in search requests ([36279](https://github.com/milvus-io/milvus/pull/36279))
- Fixed stored_index_files_size metric clearing issue ([36161](https://github.com/milvus-io/milvus/pull/36161))
- Corrected readwrite privilege group behavior for global API access ([36145](https://github.com/milvus-io/milvus/pull/36145))

Milvus 2.4.11 is a bug-fix release that addresses multiple critical issues related to the MARISA trie index, compaction, and loading operations. This release introduces new features to view expressions and improve delete stability. We recommend all users of the 2.4.x series to upgrade to this version to benefit from these improvements and fixes.

Features

- Added static view for expressions in 2.4 ([35954](https://github.com/milvus-io/milvus/pull/35954))
- Implemented delete buffer related quota logic ([35997](https://github.com/milvus-io/milvus/pull/35997))

Bug fixes

- Resolved Trie index range operation issue for GreaterThan and GreaterThanEqual comparisons ([36126](https://github.com/milvus-io/milvus/pull/36126))
- Corrected `marisa_label_order` usage in Trie index construction ([36060](https://github.com/milvus-io/milvus/pull/36060))
- Enhanced value checking for `trie.predictive_search` ([35999](https://github.com/milvus-io/milvus/pull/35999))
- Enabled Binary arithmetic expression support on inverted index ([36097](https://github.com/milvus-io/milvus/pull/36097))
- Fixed segment fault caused by Skipindex ([35908](https://github.com/milvus-io/milvus/pull/35908))
- Resolved memory leak in proxy meta cache ([36076](https://github.com/milvus-io/milvus/pull/36076))
- Renamed mmap file path to prevent directory conflicts ([35975](https://github.com/milvus-io/milvus/pull/35975))
- Improved logging and cleanup for failed/timeout tasks in mix compaction ([35967](https://github.com/milvus-io/milvus/pull/35967))
- Addressed logic deadlock during high memory usage by delegator ([36066](https://github.com/milvus-io/milvus/pull/36066))
- Implemented empty segment creation when compaction deletes all inserts ([36045](https://github.com/milvus-io/milvus/pull/36045))
- Corrected load field list population from old version load info in 2.4 ([36018](https://github.com/milvus-io/milvus/pull/36018))
- Fixed tracing config update logic in 2.4 ([35998](https://github.com/milvus-io/milvus/pull/35998))
- Resolved search/query request failures during dynamic partition release ([36019](https://github.com/milvus-io/milvus/pull/36019))
- Prevented override of fallback parameters ([36006](https://github.com/milvus-io/milvus/pull/36006))
- Ensured proper registration of privilege groups for validation ([35938](https://github.com/milvus-io/milvus/pull/35938))
- Prevented mistaken cleanup of db limiter nodes ([35992](https://github.com/milvus-io/milvus/pull/35992))
- Addressed issue with replicas not participating in queries after failure recovery ([35925](https://github.com/milvus-io/milvus/pull/35925))
- Resolved data race in clustering compaction writer ([35958](https://github.com/milvus-io/milvus/pull/35958))
- Fixed variable reference after move operation ([35904](https://github.com/milvus-io/milvus/pull/35904))
- Implemented clustering key skip load behavior check ([35899](https://github.com/milvus-io/milvus/pull/35899))
- Ensured single startup of querycoord observers in 2.4 ([35817](https://github.com/milvus-io/milvus/pull/35817))

Improvements

- Upgraded Milvus & proto version to 2.4.11 ([36069](https://github.com/milvus-io/milvus/pull/36069))
- Addressed memory leak in unit tests and enable use_asan option for unittest builds ([35857](https://github.com/milvus-io/milvus/pull/35857))
- Adjusted l0segmentsrowcount limits to more appropriate values ([36015](https://github.com/milvus-io/milvus/pull/36015))
- Modified deltalog memory estimation factor to one ([36035](https://github.com/milvus-io/milvus/pull/36035))
- Implemented slicesetequal for load field list comparisons ([36062](https://github.com/milvus-io/milvus/pull/36062))
- Reduced log frequency for delete operations ([35981](https://github.com/milvus-io/milvus/pull/35981))
- Upgraded etcd version to 3.5.14 ([35977](https://github.com/milvus-io/milvus/pull/35977))
- Optimized mmap-rss reduction after warmup ([35965](https://github.com/milvus-io/milvus/pull/35965))
- Removed cooling off period in rate limiter for read requests ([35936](https://github.com/milvus-io/milvus/pull/35936))
- Enhanced load field checking for previously loaded collections ([35910](https://github.com/milvus-io/milvus/pull/35910))
- Added support for dropping roles related to privilege lists in 2.4 ([35863](https://github.com/milvus-io/milvus/pull/35863))
- Implemented depguard rules to prohibit deprecated proto library usage ([35818](https://github.com/milvus-io/milvus/pull/35818))

Others

- Updated Knowhere version ([36067](https://github.com/milvus-io/milvus/pull/36067))

Milvus 2.4.10 introduces significant improvements in functionality and stability. Key features include support for upsert operations on AutoID-enabled collections, partial collection loading capabilities, and various memory-mapped (MMAP) configurations to optimize memory usage. This release also addresses several bugs causing panics, core dumps, and resource leaks. We recommend upgrading to take full advantage of these improvements.

Features

- **Upsert with Auto ID**: Support for upsert operations with automatic ID generation ([34633](https://github.com/milvus-io/milvus/pull/34633))
- **Field Partial Load Collection** [Beta Preview]: Allows loading specific fields of a collection ([35696](https://github.com/milvus-io/milvus/pull/35696))
- **RBAC Enhancements**:
- Added RBAC message support for Change Data Capture (CDC) ([35562](https://github.com/milvus-io/milvus/pull/35562))
- Introduced readonly/readwrite/admin privilege groups to simplify RBAC grant process ([35543](https://github.com/milvus-io/milvus/pull/35543))
- New API for backing up and restoring RBAC configurations ([35513](https://github.com/milvus-io/milvus/pull/35513))
- Refresh proxy cache after restoring RBAC metadata ([35636](https://github.com/milvus-io/milvus/pull/35636))
- **Improved MMAP Configuration**: More general configuration options to control MMAP behavior ([35609](https://github.com/milvus-io/milvus/pull/35609))
- **Database Access Restrictions**: New properties to restrict read access to databases ([35754](https://github.com/milvus-io/milvus/pull/35754))

Bug fixes

- Fixed Arrow Go client don't return error issue ([35820](https://github.com/milvus-io/milvus/pull/35820))
- Corrected inaccurate rate limiting ([35700](https://github.com/milvus-io/milvus/pull/35700))
- Resolved proxy panic after import-related API failures ([35559](https://github.com/milvus-io/milvus/pull/35559))
- Fixed potential mistaken deletions during GC channel checkpoints ([35708](https://github.com/milvus-io/milvus/pull/35708))
- Addressed panic due to empty candidate import segments ([35674](https://github.com/milvus-io/milvus/pull/35674))
- Corrected mmap memory deallocation ([35726](https://github.com/milvus-io/milvus/pull/35726))
- Ensured proper channel watching for upgrades from 2.2 to 2.4 ([35695](https://github.com/milvus-io/milvus/pull/35695))
- Fixed DataNode unwatching channel release function ([35657](https://github.com/milvus-io/milvus/pull/35657))
- Corrected partition count in RootCoord metadata ([35601](https://github.com/milvus-io/milvus/pull/35601))
- Resolved issues with dynamic config updates for certain parameters ([35637](https://github.com/milvus-io/milvus/pull/35637))

Improvements

Performance

- Optimized retrieval on dynamic fields ([35602](https://github.com/milvus-io/milvus/pull/35602))
- Improved bitset performance for AVX512 ([35480](https://github.com/milvus-io/milvus/pull/35480))
- Re-read value after `once` initialization for better efficiency ([35643](https://github.com/milvus-io/milvus/pull/35643))

Rolling upgrade improvements

- Marked query node as read-only after suspended ([35586](https://github.com/milvus-io/milvus/pull/35586))
- Prevented coexistence of old coordinator with new node/proxy ([35760](https://github.com/milvus-io/milvus/pull/35760))

Others

- Optimized Milvus core building process ([35660](https://github.com/milvus-io/milvus/pull/35660))
- Updated to protobuf-go v2 ([35555](https://github.com/milvus-io/milvus/pull/35555))
- Enhanced tracing with hex string encoding for traceid and spanid ([35568](https://github.com/milvus-io/milvus/pull/35568))
- Added hit segment number metrics for query hook ([35619](https://github.com/milvus-io/milvus/pull/35619))
- Improved compatibility with old SDK for configure load param feature ([35573](https://github.com/milvus-io/milvus/pull/35573))
- Added support for HTTP v1/v2 throttling ([35504](https://github.com/milvus-io/milvus/pull/35504))
- Fixed index memory estimation ([35670](https://github.com/milvus-io/milvus/pull/35670))
- Ability to write multiple segments in mix compactor to avoid large segment generation ([35648](https://github.com/milvus-io/milvus/pull/35648))