Starrocks

Release date: June 29, 2022

Improvements
- UDFs can be used across databases. [6865](https://github.com/StarRocks/starrocks/pull/6865) [#7211](https://github.com/StarRocks/starrocks/pull/7211)
- Optimized concurrency control for internal processing such as schema change. This reduces pressure on FE metadata management. In addition, the possibility that load jobs may pile up or slow down is reduced in scenarios where huge volume of data needs to be loaded at high concurrency. [6838](https://github.com/StarRocks/starrocks/pull/6838)

BugFix
The following bugs are fixed:
- The number of replicas (replication_num) created by using CTAS is incorrect. [7036](https://github.com/StarRocks/starrocks/pull/7036)
- Metadata may be lost after ALTER ROUTINE LOAD is performed. [7068](https://github.com/StarRocks/starrocks/pull/7068)
- Runtime filters fail to be pushed down. [7206](https://github.com/StarRocks/starrocks/pull/7206) [#7258](https://github.com/StarRocks/starrocks/pull/7258)
- Pipeline issues that may cause memory leaks. [7295](https://github.com/StarRocks/starrocks/pull/7295)
- Deadlock may occur when a Routine Load job is aborted. [6849](https://github.com/StarRocks/starrocks/pull/6849)
- Some profile statistics information is inaccurate. [7074](https://github.com/StarRocks/starrocks/pull/7074) [#6789](https://github.com/StarRocks/starrocks/pull/6789)
- The get_json_string function incorrectly processes JSON arrays.

**Full Changelog**: https://github.com/StarRocks/starrocks/compare/2.2.1...2.2.2

Thanks to
Astralidea, DorianZheng, HangyuanLiu, Seaven, Youngwb, ZiheLiu, caneGuy, decster, dirtysalt, gengjun-git, huangfeng1993, imay, liuyehcf, meegoo, mergify, mofeiatwork, rickif, satanson, sduzh, sevev, stdpain, trueeyu, xiaoyong-z

Release date: June 2, 2022

Improvements

- Optimized the data loading performance and reduced long tail latency by reconstructing part of the hotspot code and reducing lock granularity. [6641](https://github.com/StarRocks/starrocks/pull/6641)
- Added the CPU and memory usage information of the machines on which BEs are deployed for each query to the FE audit log. [6208](https://github.com/StarRocks/starrocks/pull/6208) [#6209](https://github.com/StarRocks/starrocks/pull/6209)
- Supported JSON data types in the tables that use the Primary Key model and tables that use the Unique Key model. [6544](https://github.com/StarRocks/starrocks/pull/6544)
- Reduced FEs load by reducing lock granularity and deduplicating BE report requests. Optimized the report performance when a large number of BEs are deployed, and solved the issue of Routine Load tasks getting stuck in a large cluster. [6293](https://github.com/StarRocks/starrocks/pull/6293)

Bug Fixes
The following bugs are fixed:
- An error occurs when StarRocks parses the escape characters specified in the SHOW FULL TABLES FROM DatabaseName statement. [6559](https://github.com/StarRocks/starrocks/issues/6559)
- FE disk space usage rises sharply (Fix this bug by rolling back the BDBJE version). [6708](https://github.com/StarRocks/starrocks/pull/6708)
- BEs become faulty because relevant fields cannot be found in the data returned after columnar scanning is enabled (enable_docvalue_scan=true). [6600](https://github.com/StarRocks/starrocks/pull/6600)

**Full Changelog**: https://github.com/StarRocks/starrocks/compare/2.2.0...2.2.1

Thanks to
Astralidea, HangyuanLiu, Linkerist, RowenWoo, Seaven, Youngwb, ZiheLiu, dirtysalt, gengjun-git, meegoo, mergify, mofeiatwork, rickif, sevev, trueeyu, xiaoyong-z

New Features
* [Preview] Resource groups are supported. By using resource groups to control CPU and memory usage, StarRocks can achieve resource isolation and rational use of resources when different tenants perform complex and simple queries in the same cluster.
* [Preview] Java UDFs (user-defined functions) are supported. StarRocks supports writing UDFs in Java, extending StarRocks' functions.
* [Preview] Primary key model supports partial updates when data is loaded to the primary key model using Stream Load, Broker Load, and Routine Load. In real-time data update scenarios such as updating orders and joining multiple streams, partial updates allow users to update only a few columns.
* [Preview] JSON data types and JSON functions are supported.
* External tables based on Apache Hudi are supported, which further improves data lake analytics experience.
* The following functions are supported:
* ARRAY functions, including array_agg, array_sort, array_distinct, array_join, reverse, array_slice, array_concat, array_difference, array_overlap, and array_intersect
* BITMAP functions, including bitmap_max and bitmap_min
* Other functions, including retention and square
Improvement
* CBO's Parser and Analyzer are reconstructed, code structure is optimized and syntax such as Insert with CTE is supported. So the performance of complex queries is optimized, such as those queries reusing common table expression (CTE).
* The query performance of object storage-based (AWS S3, Alibaba Cloud OSS, Tencent COS) Apache Hive external table is optimized. After optimization, the performance of object storage-based queries is comparable to that of HDFS-based queries. Also, late materialization of ORC files is supported, improving query performance of small files.
* When external tables are used to query Apache Hive, StarRocks supports automatic and incremental updating of cached metastore data by consuming Hive Metastore events, such as data changes and partition changes. Moreover, it also supports querying DECIMAL and ARRAY data in Apache Hive.
* The performance of UNION ALL operator is optimized, delivering improvement of up to 2-25 times.
* The pipeline engine which can adaptively adjust query parallelism is released, and its profile is optimized. The pipeline engine can improve performance for small queries in high concurrent scenarios.
* StarRocks supports the loading of CSV files with multi-character row delimiters.
Bug Fixes
The following bugs are fixed:
* Deadlocks occur when data is loaded and changes are committed into tables based on Primary Key model. [4998](https://github.com/StarRocks/starrocks/pull/4998)
* Some FE (including BDBJE) stability issues. [4428](https://github.com/StarRocks/starrocks/pull/4428), [#4666](https://github.com/StarRocks/starrocks/pull/4666), [#2](https://github.com/StarRocks/bdb-je/pull/2)
* The return value overflows when the SUM function is used to calculate a large amount of data. [3944](https://github.com/StarRocks/starrocks/pull/3944)
* The return values of ROUND and TRUNCATE functions have precision issues. [4256](https://github.com/StarRocks/starrocks/pull/4256)
Some bugs detected by SQLancer. Please see [SQLancer related issues](https://github.com/StarRocks/starrocks/issues?q=is:issue++label:sqlancer++milestone:2.2).
Others
* The Flink connector flink-connector-starrocks supports Flink 1.14.

Release date: September 6, 2022

Improvements
- Added a BE configuration item enable_check_string_lengths to check the length of loaded data. This mechanism helps prevent compaction failures caused by VARCHAR data size out of range. [10380](https://github.com/StarRocks/starrocks/issues/10380)
- Optimized the query performance when a query contains more than 1000 OR operators. [9332](https://github.com/StarRocks/starrocks/pull/9332)

Bug Fixes
The following bugs are fixed:
- An error may occur and BEs may crash when you query ARRAY columns (calculated by using the REPLACE_IF_NOT_NULL function) from a table using the Aggregate Key Model. [10144](https://github.com/StarRocks/starrocks/issues/10144)
- The query result is incorrect if more than one IFNULL() function is nested in the query. [5028](https://github.com/StarRocks/starrocks/issues/5028) [#10486](https://github.com/StarRocks/starrocks/pull/10486)
- After a dynamic partition is truncated, the number of tablets in the partition changes from the value configured by dynamic partitioning to the default value. [10435](https://github.com/StarRocks/starrocks/issues/10435)
- If the Kafka cluster is stopped when you use Routine Load to load data into StarRocks, deadlocks may occur, affecting query performance. [8947](https://github.com/StarRocks/starrocks/issues/8947)
- An error occurs when a query contains both subqueries and ORDER BY clauses. [10066](https://github.com/StarRocks/starrocks/pull/10066)


**Full Changelog**: https://github.com/StarRocks/starrocks/compare/2.1.12...2.1.13

Thanks to
Astralidea, Linkerist, Seaven, ZiheLiu, amber-create, dulong41, meegoo, mergify, rickif, trueeyu, xiaoyong-z

Release date: August 9, 2022

Improvements
Added two parameters, bdbje_cleaner_threads and bdbje_replay_cost_percent, to speed up metadata cleanup in BDB JE.

Bug Fixes
The following bugs are fixed:
- Some queries are forwarded to the Leader FE, causing the /api/query_detail action to return incorrect execution information about SQL statements such as SHOW FRONTENDS. [9185](https://github.com/StarRocks/starrocks/issues/9185)
- After a BE is terminated, the current process is not completely terminated, resulting in a failed restart of the BE. [9175](https://github.com/StarRocks/starrocks/pull/9267)
- When multiple Broker Load jobs are created to load the same HDFS data file, if one job encounters exceptions, the other jobs may not be able to properly read data and consequently fail. [9506](https://github.com/StarRocks/starrocks/issues/9506)
- The related variables are not reset when the schema of a table changes, resulting in an error (no delete vector found tablet) when querying the table. [9192](https://github.com/StarRocks/starrocks/issues/9192)

**Full Changelog**: https://github.com/StarRocks/starrocks/compare/2.1.11...2.1.12

Thanks to
EsoragotoSpirit, HangyuanLiu, Pslydhh, Seaven, amber-create, chaoyli, decster, dulong41, femiiii, gengjun-git, hellolilyliuyi, jaogoy, liuyehcf, minchowang, padmejin, rickif, stdpain, trueeyu, xiaoyong-z, yongbingwang

Release date: July 9, 2022

Bug Fixes
The following bugs are fixed:
- Data loading into a table of the Primary Key model is suspended in the event of frequent data loads into that table.[7763](https://github.com/StarRocks/starrocks/issues/7763)
- Aggregate expressions are processed in an incorrect sequence during low-cardinality optimization, causing the count distinct function to return unexpected results. [7659](https://github.com/StarRocks/starrocks/issues/7659)
- No results are returned for the LIMIT clause, because the pruning rule in the clause cannot be properly processed. [7894](https://github.com/StarRocks/starrocks/pull/7894)
- If the global dictionary for low-cardinality optimization is applied on columns that are defined as join conditions for a query, the query returns unexpected results. [8302](https://github.com/StarRocks/starrocks/issues/8302)

**Full Changelog**: https://github.com/StarRocks/starrocks/compare/2.1.10...2.1.11

Thanks to
HangyuanLiu, Seaven, Youngwb, chaoyli, decster, gengjun-git, imay, kangkaisen, mergify, rickif, sduzh, stdpain, trueeyu