Starrocks

**Bug Fixes**

The following bugs are fixed:

- Query result may be inaccurate when you query an Hive external table stored as a text file. [11546](https://github.com/StarRocks/starrocks/pull/11546)

- Nested arrays are not supported when you query Parquet files. [10983](https://github.com/StarRocks/starrocks/pull/10983)

- Queries or a query may time out if concurrent queries that read data from StarRocks and external data sources are routed to the same resource group, or a query reads data from StarRocks and external data sources. [10983](https://github.com/StarRocks/starrocks/pull/10983)

- When the Pipeline execution engine is enabled by default, the parameter parallel_fragment_exec_instance_num is changed to 1. It will cause data loading by using INSERT INTO to be slow. [11462](https://github.com/StarRocks/starrocks/pull/11462)

- BE may crash if there are mistakes when a expression is initialized. [11396](https://github.com/StarRocks/starrocks/pull/11396)

- The error heap-buffer-overflow may occur if you execute ORDER BY LIMIT. [11185](https://github.com/StarRocks/starrocks/pull/11185)

- Schema change fails if you restart Leader FE in the meantime. [11561](https://github.com/StarRocks/starrocks/pull/11561)

**New Features**
- Late materialization is supported to accelerate range filter-based queries on external tables in Parquet format. [9738](https://github.com/StarRocks/starrocks/pull/9738)
- The SHOW AUTHENTICATION statement is added to display user authentication-related information. [9996](https://github.com/StarRocks/starrocks/pull/9996)

**Improvements**
- A configuration item is provided to control whether StarRocks recursively traverses all data files for the bucketed Hive table from which StarRocks queries data. [10239](https://github.com/StarRocks/starrocks/pull/10239)
- The resource group type realtime is renamed as short_query. [10247](https://github.com/StarRocks/starrocks/pull/10247)
- StarRocks no longer distinguishes between uppercase letters and lowercase letters in Hive external tables by default. [10187](https://github.com/StarRocks/starrocks/pull/10187)

**Bug Fixes**
The following bugs are fixed:
- Queries on an Elasticsearch external table may unexpectedly exit when the table is divided into multiple shards. [10369](https://github.com/StarRocks/starrocks/pull/10369)
- StarRocks throws errors when sub-queries are rewritten as common table expressions (CTEs). [10397](https://github.com/StarRocks/starrocks/pull/10397)
- StarRocks throws errors when a large amount of data is loaded. [10370](https://github.com/StarRocks/starrocks/issues/10370) [#10380](https://github.com/StarRocks/starrocks/issues/10380)
- When the same Thrift service IP address is configured for multiple catalogs, deleting one catalog invalidates the incremental metadata updates in the other catalogs. [10511](https://github.com/StarRocks/starrocks/pull/10511)
- The statistics of memory consumption from BEs are inaccurate. [9837](https://github.com/StarRocks/starrocks/pull/9837)
- StarRocks throws errors for queries on Primary Key tables. [10811](https://github.com/StarRocks/starrocks/pull/10811)
- Queries on logical views are not allowed even when you have SELECT permissions on these views. [10563](https://github.com/StarRocks/starrocks/pull/10563)
- StarRocks does not impose limits on the naming of logical views. Now logical views need to follow the same naming conventions as tables. [10558](https://github.com/StarRocks/starrocks/pull/10558)

Release date: August 22, 2022

**Improvements**

- Broker Load supports transforming the List type in Parquet files into non-nested ARRAY data type. [9150](https://github.com/StarRocks/starrocks/pull/9150)
- Optimizes the performance of JSON-related functions (json_query, get_json_string, and get_json_int). [9623](https://github.com/StarRocks/starrocks/pull/9623)
- Optimizes the exception message: during a query on Hive, Iceberg, or Hudi, if the data type of the column in the query is not supported by StarRocks, the system throws an exception on the corresponding column. [10139](https://github.com/StarRocks/starrocks/pull/10139)
- Reduces the schedule latency of resource group to optimize the resource isolation performance. [10122](https://github.com/StarRocks/starrocks/pull/10122)

**Bugfixes**

- Wrong result is returned from the query on Elasticsearch external table caused by the incorrect pushdown of the `limit` operator. [9952](https://github.com/StarRocks/starrocks/pull/9952)
- Query on Oracle external table fails when the `limit` operator is used. [9542](https://github.com/StarRocks/starrocks/pull/9542)
- BE is blocked when all Kafka Brokers are stopped during a Routine Load. [9935](https://github.com/StarRocks/starrocks/pull/9935)
- BE crashes during a query on a Parquet file whose data type mismatches that of the corresponding external table. [10107](https://github.com/StarRocks/starrocks/pull/10107)
- Query hangs because the external table scan range is empty. [10091](https://github.com/StarRocks/starrocks/pull/10091)
- The system throws an exception when ORDER BY clause is included in a sub-query. [10180](https://github.com/StarRocks/starrocks/pull/10180)
- Hive Metastore hangs when the Hive metadata is reloaded asynchronously. [10132](https://github.com/StarRocks/starrocks/pull/10132)

**New Features**
- The Primary Key model supports the full DELETE WHERE syntax.[4500](https://github.com/StarRocks/starrocks/pull/4500)
- The primary key model supports persistent primary key indexes to prevent excessive memory consumption.[3399](https://github.com/StarRocks/starrocks/pull/3399)
- When a Routine Load job is executed, and a global dictionary is built to optimize queries on low-cardinality columns, the global dictionary can be updated, thus improving query performance.
- The CREATE TABLE AS SELECT statement can be executed asynchronously and write results to a new table. [5543](https://github.com/StarRocks/starrocks/pull/5543)
- Support the following resource group-related features:
- Monitor resource groups: You can use the audit log to view a query in which resource group and call API operations to obtain the monitoring metrics about specific resource groups.
- Limit the consumption of large queries on CPU, memory, or I/O resources: You can route queries to resource groups by matching classifiers or configure session variables to directly specify resource groups for queries.
- StarRocks provides JDBC external tables to query Oracle, PostgreSQL, MySQL, SQLServer, Clickhouse, and other databases. StarRocks also supports predicate pushdown when performing queries. [3352](https://github.com/StarRocks/starrocks/pull/3352) [#3353](https://github.com/StarRocks/starrocks/pull/3353) [#3354](https://github.com/StarRocks/starrocks/pull/3354) [#3723](https://github.com/StarRocks/starrocks/pull/3723) [#3724](https://github.com/StarRocks/starrocks/pull/3724) [#3725](https://github.com/StarRocks/starrocks/pull/3725)
- [Preview] A new Data Source Connector framework is released to support user-defined external catalogs. You can use the external catalogs to directly access and analyze Hive without creating external tables. [5062](https://github.com/StarRocks/starrocks/issues/5062) [#7268](https://github.com/StarRocks/starrocks/issues/7268)
- Add the following functions:
- window_funnel
- ntile
- bitmap_union_count、array_to_bitmap [7404](https://github.com/StarRocks/starrocks/issues/7404)、base64_to_bitmap
- week、time_slice
- Add the EXECUTE AS statement. After you use the GRANT statement to impersonate a specific user identity to perform operations, you can use the EXECUTE AS statement to switch the execution context of the current session to this user.

**Improvements**
- The compaction mechanism can merge large metadata more quickly. This prevents metadata squeezing and excessive disk usage that can occur shortly after frequent data updates.
- The performance of loading Parquet files and compressed files is optimized. [3945](https://github.com/StarRocks/starrocks/pull/3945)
- The mechanism of creating materialized views is optimized. After the optimization, materialized views can be created at a speed up to 10 times higher than before.
- The performance of the following operators is optimized:
- TopN and sort operators [3953](https://github.com/StarRocks/starrocks/pull/3953) [#4106](https://github.com/StarRocks/starrocks/pull/4106) [#4227](https://github.com/StarRocks/starrocks/pull/4227) [#4680](https://github.com/StarRocks/starrocks/pull/4680)
- Equivalence comparison operators that contain functions can use Zone Map indexes when these operators are pushed down to scan operators.
- Optimize the Apache Hive™ external tables.
- When Apache Hive™ tables are stored as Parquet, ORC, or CSV formats, StarRocks can synchronize the schema changes like ADD COLUMN and REPLACE COLUMN from hive tables when you perform REFRESH statement on external tables.
- Hive resources' hive.metastore.uris can be modified. [3843](https://github.com/StarRocks/starrocks/pull/3843)
- Optimize the Apache Iceberg external tables. A custom catalog can be used to create an Iceberg resource. [5062](https://github.com/StarRocks/starrocks/issues/5062)
- Optimize the Elasticsearch external tables. Sniffing the addresses of the data nodes in an Elasticsearch cluster can be disabled. [6314](https://github.com/StarRocks/starrocks/issues/6314)
- When the sum() function calculates numeric strings, implicit conversion is performed.
- The year, month, and day functions support the DATE data type.
Bug Fixes
The following bugs are fixed:
- CPU utilization increases abnormally high due to an excessive number of tablets. [5875](https://starrocks.atlassian.net/browse/SR-5875)
- Problems cause the "fail to prepare tablet reader" error message to occur.[7248](https://starrocks.atlassian.net/browse/SR-7248)、
[7854](https://starrocks.atlassian.net/browse/SR-7854)、 [#8257](https://starrocks.atlassian.net/browse/SR-8257)
Problems cause the "fail to prepare tablet reader" error message to occur.
- The FEs fail to restart.[5642](https://github.com/StarRocks/starrocks/issues/5642 )、[#4969](https://github.com/StarRocks/starrocks/issues/4969 )、[#5580](https://github.com/StarRocks/starrocks/issues/5580)。
- The CTAS statement cannot be run successfully when the statement includes a JSON function.[6498](https://github.com/StarRocks/starrocks/issues/6498)


**Others**
- [Preview] StarGo, a cluster management tool, can deploy, start, upgrade, and roll back clusters and manage multiple clusters.
- StarRocks clusters can be quickly deployed on AWS by using CloudFormation.
- Pipeline Engine is turned on as default now.

Release date: April 6, 2023

Improvements
- Optimized the bitmap_contains() function to reduce its memory consumption and improve its performance in some scenarios. [20616](https://github.com/StarRocks/starrocks/issues/20616)
- Optimized the compaction framework to reduce its CPU resource consumption. [11746](https://github.com/StarRocks/starrocks/issues/11746)

Bug Fixes
The following bugs are fixed:
- If the requested URL in a Stream Load job is not correct, the responsible FE hangs and is unable to handle the HTTP request. [18468](https://github.com/StarRocks/starrocks/issues/18468)
- When the responsible FE collects statistics, it may consume abnormally large amount of memory, which causes OOM. [16331](https://github.com/StarRocks/starrocks/issues/16331)
- BEs crash if memory release is not properly handled in some queries. [11395](https://github.com/StarRocks/starrocks/issues/11395)
- After the command TRUNCATE TABLE is executed, a NullPointerException may occur and the responsible FE fails to restart . [16773](https://github.com/StarRocks/starrocks/issues/16773)


**Full Changelog**: https://github.com/StarRocks/starrocks/compare/2.2.12...2.2.13

Improvements
- Added the session variable hive_partition_stats_sample_size to control the number of Hive partitions from which to collect statistics. An excessive number of partitions will cause errors in obtaining Hive metadata. [12700](https://github.com/StarRocks/starrocks/pull/12700)
- Elasticsearch external tables support custom time zones. [12662](https://github.com/StarRocks/starrocks/pull/12662)

Bug Fixes
The following bugs are fixed:
- The DECOMMISSION operation is stuck if an error occurs during metadata synchronization for external tables. [12369](https://github.com/StarRocks/starrocks/pull/12368)
- Compaction crashes if a column that is newly added is deleted. [12907](https://github.com/StarRocks/starrocks/pull/12907)
- SHOW CREATE VIEW does not display the comments that were added when creating the view. [4163](https://github.com/StarRocks/starrocks/issues/4163)
- Memory leak in Java UDF may cause OOM. [12418](https://github.com/StarRocks/starrocks/pull/12418)

Behavior Changes
Extended the length of Hive STRING columns that can be queried by StarRocks from 64 KB to 1 MB. If a STRING column exceeds 1 MB, it will be processed as a null column during queries. [12986](https://github.com/StarRocks/starrocks/pull/12986)