Milvus

Milvus 2.1.2 is a minor bug-fix version of Milvus 2.1.0. It fixed issues when garbage collector parses binlog path, a few other issues causing search hang, and performance regression when the user authentication feature is enabled.

Bug Fixes

- [18733](https://github.com/milvus-io/milvus/pull/18733), [18783](https://github.com/milvus-io/milvus/pull/18783), [18844](https://github.com/milvus-io/milvus/pull/18844), [18886](https://github.com/milvus-io/milvus/pull/18886), [18906](https://github.com/milvus-io/milvus/pull/18906) Fixes load/cluster restart/scale hang.

- [18678](https://github.com/milvus-io/milvus/pull/18678) Fixes flush panic after compaction.

- [18690](https://github.com/milvus-io/milvus/pull/18690) Fixes DataType::bool cast to double.

- [18884](https://github.com/milvus-io/milvus/pull/18884), [18808](https://github.com/milvus-io/milvus/pull/18808) Fixes garbage collection failure when the storage root path starts with "/".

- [18790](https://github.com/milvus-io/milvus/pull/18790) Fixes watchDmChannel being out-of-date after compaction issue.

- [18872](https://github.com/milvus-io/milvus/issues/18872) Disables empty string insertion to avoid server crash.

- [18701](https://github.com/milvus-io/milvus/pull/18701) Fixes RHNSWPQ pqm divided by zero.

- [18714](https://github.com/milvus-io/milvus/pull/18714) Fixes flush hang when Pulsar error causes empty segments.

- [18732](https://github.com/milvus-io/milvus/pull/18732) Fixes performance degradation caused by missing password verification cache.

Improvements

- [18683](https://github.com/milvus-io/milvus/pull/18683) Deduplicates output fields for query.

- [18895](https://github.com/milvus-io/milvus/pull/18895) Adds manual compaction periodically to clean up deleted RocksMQ data.

- [18795](https://github.com/milvus-io/milvus/pull/18795), [18850](https://github.com/milvus-io/milvus/pull/18850) Refactors compaction concurrency logic.

Features

- [17899](https://github.com/milvus-io/milvus/pull/17899) Supports configurable SASL mechanism for Kafka.

Milvus 2.1.1 is a minor bug fix version of Milvus 2.1.0. It fixed Query Node crash under high concurrency, garbage collector failure to parse segment ID issue, and a few other stability issues.

<h3 id="v2.1.1">Bug fixes</h3>

- [18383](https://github.com/milvus-io/milvus/pull/18383), [#18432](https://github.com/milvus-io/milvus/pull/18432) Fixed Garbage collector parse segment ID panics with bad input.

- [18418](https://github.com/milvus-io/milvus/pull/18418) Fixed metatable related error when etcd compaction error happens.

- [18568](https://github.com/milvus-io/milvus/pull/18568) Closes Node/Segment detector when closing ShardCluster.

- [18542](https://github.com/milvus-io/milvus/pull/18542) Adds CGO worker pool for Query Node to avoid OpenMP creating too many threads.

- [18569](https://github.com/milvus-io/milvus/pull/18569) Releases collection resources when all partitions are released to avoid resource leakage.

<h3 id="v2.1.1">Improvements</h3>

- [18394](https://github.com/milvus-io/milvus/pull/18394) Removes watch delta channel task-related logic.

- [18513](https://github.com/milvus-io/milvus/pull/18513) Uses chunkManager rather than minio.Client in Data Node garbage collection.

- [18410](https://github.com/milvus-io/milvus/pull/18410) Limits the concurrency level for single load request

<h3 id="v2.1.1">Featuress</h3>

- [18570](https://github.com/milvus-io/milvus/pull/18570) Supports dynamic change log level through HTTP.

<h3 id="v2.1.0">Features</h3>

- Support for VARCHAR data type

Milvus now supports variable-length string as a scalar data type. Like previous scalar types, VARCHAR can be specified as an output field or be used for attribute filtering. A MARISA-trie-based inverted index is also supported to accelerate prefix query and exact match.

- In-memory replicas

In-memory replicas enable you to load data on multiple query nodes. Like read replicas in traditional databases, in-memory replicas can help increase throughput if you have a relatively small dataset but want to scale read throughput with more hardware resources. We will support hedged read in future releases to increase availability when applying in-memory replicas.

- Embedded Milvus

Embedded Milvus enables you to [pip install Milvus](https://milvus.io/docs/v2.1.x/install_embedded_milvus.md) in one command, try quick demos and run short scripts in Python on your Macbook, including on the ones with M1 processor.

- Kafka support (Beta)

Apache Kafka is the most widely used open-source distributed message store. In Milvus 2.1.0, you can simply use Kafka for message storage by modifying configurations.

- RESTful API (Beta)

Milvus 2.1.0 now provides RESTful API for applications written in PHP or Ruby. GIN, one of the most popular Golang web frameworks, is adopted as the web server.

<h3 id="v2.1.0">Performance</h3>

The Milvus core team conducted a full performance benchmarking and profiling, and fixed a few bottlenecks on load/search paths. Under some test cases, Milvus search performance is boosted about 3.2 times thanks to the search combination logic.
- [16014](https://github.com/milvus-io/milvus/pull/16014) Enables ZSTD compression for pulsar.
- [16514](https://github.com/milvus-io/milvus/pull/16514) [#17273](https://github.com/milvus-io/milvus/pull/17273) Improves load performance.
- [17005](https://github.com/milvus-io/milvus/pull/17005) Loads binlog for different fields in parallel.
- [17022](https://github.com/milvus-io/milvus/pull/17022) Adds logic for search merging and a simple task scheduler for read tasks.
- [17194](https://github.com/milvus-io/milvus/pull/17194) Simplifies the merge logic of searchTask.
- [17287](https://github.com/milvus-io/milvus/pull/17287) Reduces default seal proportion.

<h3 id="v2.1.0">Stability</h3>

To improve stability, especially during streaming data insertion, we fixed a few critical issues including:
- Fixed out of memory issues.
- Fixed message queue backlog full caused by message queue subscription leakge.
- Fixed the issue of deleted entities can still be readable.
- Fixed data being erroneously cleaned by compaction during load or index.


<h3 id="v2.1.0">Other improvements</h3>

- Security

Starting from Milvus 2.1.0, we support username, password, and TLS connection. We also enable safe connections to our dependencies such as S3, Kafka and etcd.

- ANTLR parser

Milvus now adopts Go ANTLR as the plan parser to make adding new grammar such as arithmetic operations on numerical fields more flexible. The adoption of ANTLR also prepares for Milvus query language support in future releases.

- Observability

We refined monitoring metrics by adding important [metrics](https://milvus.io/docs/v2.1.x/metrics_dashboard.md) including search QPS and latency to the new dashboard. Please notify us if any metrics critical to your production environment are not listed.

- Deployment

For users who don't have a K8s environment but still want to deploy a cluster, Milvus now supports Ansible deployment. See [Install Milvus Cluster](https://milvus.io/docs/v2.1.x/install_cluster-ansible.md) for more information.

<h3 id="v2.1.0">Known issues</h3>

1. Partition is not a fully released feature so we recommend user not to rely on it. [17648 When a partition is dropped, the data and index cannot be cleaned.](https://github.com/milvus-io/milvus/issues/17648)
2. When building index after load, the collection need to released and reloaded. [17809 When an index is created on a loaded collection, the segment already loaded will not be notified to load the index.](https://github.com/milvus-io/milvus/issues/17809)

Release date: 2022-04-02

<h3 id="v2.0.2">Compatibility</h3>

<table class="version">
<thead>
<tr>
<th>Milvus version</th>
<th>Python SDK version</th>
<th>Java SDK version</th>
<th>Go SDK version</th>
<th>Node.js SDK version</th>
</tr>
</thead>
<tbody>
<tr>
<td>2.0.2</td>
<td>2.0.2</td>
<td>2.0.4</td>
<td>2.0.0</td>
<td>2.0.2</td>
</tr>
</tbody>
</table>


Milvus 2.0.2 is a minor bug-fix version of Milvus 2.0. We fixed multiple critical issues of collection load failure and server crash. We've also greatly boosted the query by ID performance by utilizing primary key index. The Prometheus metrics is redesigned in this version and we highly recommend you to deploy the monitoring system in production environment.

<h3 id="v2.0.2">Bug fixes</h3>

- [16338](https://github.com/milvus-io/milvus/pull/16338) Data coord uses VChannel when when unsubscribing to data node.
- [16178](https://github.com/milvus-io/milvus/pull/16178) [#15725](https://github.com/milvus-io/milvus/pull/15725) Query node crashes.
- [16035](https://github.com/milvus-io/milvus/pull/16035) [#16063](https://github.com/milvus-io/milvus/pull/16063) [#16066](https://github.com/milvus-io/milvus/pull/16066) Collection load error.
- [15932](https://github.com/milvus-io/milvus/pull/15932) Compaction runtime error.
- [15823](https://github.com/milvus-io/milvus/pull/15823) `DescribeCollection` RPC fails in data node failover.
- [15783](https://github.com/milvus-io/milvus/pull/15783) Recall drops after compaction.
- [15790](https://github.com/milvus-io/milvus/pull/15790) Shallow copy of `typeutil.AppendFieldData`.
- [15728](https://github.com/milvus-io/milvus/pull/15728) Query coord sets wrong `watchDmchannelInfo` when one partition is empty.
- [15712](https://github.com/milvus-io/milvus/pull/15712) `DEPLOY_MODE` is got or used before set.
- [15702](https://github.com/milvus-io/milvus/pull/15702) Data coord panics if message queue service quits before it.
- [15707](https://github.com/milvus-io/milvus/pull/15707) Compaction generates empty segment.

<h3 id="v2.0.2">Performance</h3>

- [16327](https://github.com/milvus-io/milvus/pull/16327) Accelerates query speed in sealed segments.
- [15732](https://github.com/milvus-io/milvus/pull/15732) [#15738](https://github.com/milvus-io/milvus/pull/15738) [#15774](https://github.com/milvus-io/milvus/pull/15774) [#15749](https://github.com/milvus-io/milvus/pull/15749) [#15614](https://github.com/milvus-io/milvus/pull/15614) Avoids memory copy for `kv` interface && GGO.

<h3 id="v2.0.2">Improvements</h3>

- [16244](https://github.com/milvus-io/milvus/pull/16244) [#16243](https://github.com/milvus-io/milvus/pull/16243) [#16245](https://github.com/milvus-io/milvus/pull/16245) Extends `DataCluster` Watch information via etcd.
- [15760](https://github.com/milvus-io/milvus/pull/15760) [#15787](https://github.com/milvus-io/milvus/pull/15787) [#16252](https://github.com/milvus-io/milvus/pull/16252) Fixes CentOS build.
- [15759](https://github.com/milvus-io/milvus/pull/15759) Refines code of data node `binlogIO`.

<h3 id="v2.0.2">Features</h3>

- [15540](https://github.com/milvus-io/milvus/pull/15540) Adds and implements `chunkManager` interface.
- [16072](https://github.com/milvus-io/milvus/pull/16072) [#15839](https://github.com/milvus-io/milvus/pull/15839) [#15684](https://github.com/milvus-io/milvus/pull/15684) [#15640](https://github.com/milvus-io/milvus/pull/15640) [#15582](https://github.com/milvus-io/milvus/pull/15582) [#15649](https://github.com/milvus-io/milvus/pull/15649) [#15650](https://github.com/milvus-io/milvus/pull/15650) [#15606](https://github.com/milvus-io/milvus/pull/15606) Refines Prometheus metrics.

Release date: 2022-02-23

<h3 id="v2.0.0">Compatibility</h3>

<table class="version">
<thead>
<tr>
<th>Milvus version</th>
<th>Python SDK version</th>
<th>Java SDK version</th>
<th>Go SDK version</th>
<th>Node.js SDK version</th>
</tr>
</thead>
<tbody>
<tr>
<td>2.0.1</td>
<td>2.0.1</td>
<td>2.0.4</td>
<td>2.0.0</td>
<td>2.0.1</td>
</tr>
</tbody>
</table>


Milvus 2.0.1 is a minor bug-fix version of Milvus 2.0. The key progress of Milvus 2.0.1 includes that, first, the execution engine of Milvus `knowhere` was separated from the Milvus repository and moved to a new one - [milvus-io/knowhere](https://github.com/milvus-io/knowhere), and, second, supports were enabled for Milvus to be compiled across multiple platforms. We fixed a few critical issues that cause query node crash, index building failure, and server hang. The default dependency of Golang is upgraded to solve memory usage issues. We also upgrade the default dependency of Pulsar to solve the `log4j` security issue.

<h3 id="v2.0.1">Improvements</h3>

- [15491](https://github.com/milvus-io/milvus/pull/15491) Supports compiling and running Milvus on Mac.
- [15453](https://github.com/milvus-io/milvus/pull/15453) Adds log when removing keys in garbage collector.
- [15551](https://github.com/milvus-io/milvus/pull/15551) Avoids copying while converting C bytes to Go bytes.
- [15377](https://github.com/milvus-io/milvus/pull/15377) Adds `collectionID` to the return of `SearchResults` and `QueryResults`.

<h3 id="v2.0.1">Features</h3>

- [14418](https://github.com/milvus-io/milvus/pull/14418) Implements automatic item expiration on compaction.
- [15542](https://github.com/milvus-io/milvus/pull/15542) Implements mixed compaction logic.

<h3 id="v2.0.1">Bug fixes</h3>

- [15702](https://github.com/milvus-io/milvus/pull/15702) Data coord panics if message queue service quits before it closes.
- [15663](https://github.com/milvus-io/milvus/pull/15663) Query node crashes on concurrent search.
- [15580](https://github.com/milvus-io/milvus/pull/15580) Data node panics when compacting empty segment.
- [15626](https://github.com/milvus-io/milvus/pull/15626) Failed to create index when segment size is set to large than 2GB.
- [15497](https://github.com/milvus-io/milvus/pull/15497) `SessionWatcher` quits if not re-watch logic is provided when meeting `ErrCompacted`.
- [15530](https://github.com/milvus-io/milvus/pull/15530) Segments under Flushing status are not treated as Flushed segment.
- [15436](https://github.com/milvus-io/milvus/pull/15436) Watch DML channel failed because of no collection meta, causing load collection failure.
- [15455](https://github.com/milvus-io/milvus/pull/15455) `SegmentIDs` is not respected when `querynode.GetSegmentInfo` is called.
- [15482](https://github.com/milvus-io/milvus/pull/15482) `EntriesNum` of delta logs is not recorded correctly in segment meta.

<h3 id="v2.0.1">Dependency Upgrade</h3>

- [11393](https://github.com/milvus-io/milvus/pull/11393) Upgrades Golang from 1.15.2 to 1.16.9.
- [15603](https://github.com/milvus-io/milvus/pull/15603) Upgrades `Knowhere` to 1.0.1.
- [15580](https://github.com/milvus-io/milvus/pull/15580) Upgrades Pulsar from 2.7.3 to 2.8.2.

Specific breaking changes include:

- JAVA, Go, or C++ SDK is not yet supported.

- Delete or update is not yet supported.

- PyMilvus-ORM does not support force flush.

- Data format is incompatible with all prior versions.

- Mishards is deprecated because Milvus 2.0 is distributed and sharding middleware is no longer necessary.

- Local file system and distributed system storage are not yet supported.