Starrocks

Behavior Change

- Constrained the session variable `query_timeout` with an upper limit of `259200` and a lower limit of `1`.

Improvement

- Optimized the performance of Bucket Hint when a multitude of buckets exist. [13142](https://github.com/StarRocks/starrocks/pull/13142)

Bug Fixes

The following bugs are fixed:

- Flushing the Primary Key index may cause BE to crash. [14857](https://github.com/StarRocks/starrocks/pull/14857) [#14819](https://github.com/StarRocks/starrocks/pull/14819)
- Materialized view types cannot be correctly identified by `SHOW FULL TABLES`. [13954](https://github.com/StarRocks/starrocks/pull/13954)
- Upgrading StarRocks v2.2 to v2.4 may cause BE to crash. [13795](https://github.com/StarRocks/starrocks/pull/13795)
- Broker Load may cause BE to crash. [13973](https://github.com/StarRocks/starrocks/pull/13973)
- The session variable `statistic_collect_parallel` does not take effect. [14352](https://github.com/StarRocks/starrocks/pull/14352)
- INSERT INTO may cause BE to crash. [14818](https://github.com/StarRocks/starrocks/pull/14818)
- JAVA UDF may cause BE to crash. [13947](https://github.com/StarRocks/starrocks/pull/13947)
- Cloning replicas during partial updates may cause BE to crash and fail to restart. [13683](https://github.com/StarRocks/starrocks/pull/13683)
- Colocated Join may not take effect. [13561](https://github.com/StarRocks/starrocks/pull/13561)

New Feature

- Supports non-equi joins - LEFT SEMI JOIN and ANTI JOIN. Optimized the JOIN function. [13019](https://github.com/StarRocks/starrocks/pull/13019)

Improvements

- Supports property `aliveStatus` in `HeartbeatResponse`. `aliveStatus` indicates if a node is alive in the cluster. Mechanisms that judge the `aliveStatus` are further optimized. [12713](https://github.com/StarRocks/starrocks/pull/12713)

- Optimized the error message of Routine Load. [12155](https://github.com/StarRocks/starrocks/pull/12155)

Bug Fixes

- BE crashes after being upgraded from v2.4.0RC to v2.4.0. [13128](https://github.com/StarRocks/starrocks/pull/13128)

- Late materialization causes incorrect results to queries on data lakes. [13133](https://github.com/StarRocks/starrocks/pull/13133)

- The get_json_int function throws exceptions. [12997](https://github.com/StarRocks/starrocks/pull/12997)

- Data may be inconsistent after deletion from a PRIMARY KEY table with a persistent index.[12719](https://github.com/StarRocks/starrocks/pull/12719)

- BE may crash during compaction on a PRIMARY KEY table. [12914](https://github.com/StarRocks/starrocks/pull/12914)

- The json_object function returns incorrect results when its input contains an empty string. [13030](https://github.com/StarRocks/starrocks/issues/13030)

- BE crashes due to `RuntimeFilter`. [12807](https://github.com/StarRocks/starrocks/pull/12807)

- FE hangs due to excessive recursive computations in CBO. [12788](https://github.com/StarRocks/starrocks/pull/12788)

- BE may crash or report an error when exiting gracefully. [12852](https://github.com/StarRocks/starrocks/pull/12852)

- Compaction crashes after data is deleted from a table with new columns added to it. [12907](https://github.com/StarRocks/starrocks/pull/12907)

- Data may be inconsistent due to incorrect mechanisms in OLAP external table metadata synchronization. [12368](https://github.com/StarRocks/starrocks/pull/12368)

- When one BE crashes, the other BEs may execute relevant queries till timeout. [12954](https://github.com/StarRocks/starrocks/pull/12954)

Behavior Change

- When parsing Hive external table fails, StarRocks throws error messages instead of converting relevant columns into NULL columns. [12382](https://github.com/StarRocks/starrocks/pull/12382)

**New Features**

- Supports creating a materialized view based on multiple base tables to accelerate queries with JOIN operations.
- Supports overwriting data via INSERT OVERWRITE.
- [Preview] Provides stateless Compute Nodes (CN) that can be horizontally scaled. You can use StarRocks Operator to deploy CN into your Kubernetes (K8s) cluster to achieve automatic horizontal scaling.
- Outer Join supports non-equi joins in which join items are related by comparison operators including `<`, `<=`, `>`, `>=`, and `<>`.
- Supports creating Iceberg catalogs and Hudi catalogs, which allow direct queries on data from Apache Iceberg and Apache Hudi.
- Supports querying ARRAY-type columns from Apache Hive™ tables in CSV format.
- Supports viewing the schema of external data via DESC.
- Supports granting a specific role or IMPERSONATE permission to a user via GRANT and revoking them via REVOKE, and supports executing an SQL statement with IMPERSONATE permission via EXECUTE AS.
- Supports FDQN access: now you can use a domain name or the combination of hostname and port as the unique identification of a BE or an FE node. This prevents access failures caused by changing IP addresses.
- flink-connector-starrocks supports Primary Key model partial update.
- Provides the following new functions:
- array_contains_all: checks whether a specific array is a subset of another.
- percentile_cont: calculates the percentile value with linear interpolation.

**Improvements**

- The Primary Key model supports flushing VARCHAR-type primary key indexes to disks. From version 2.4.0, the Primary Key model supports the same data types for primary key indexes regardless of whether the persistent primary key index is turned on or not.
- Optimized the query performance on external tables.
- Supports late materialization during queries on external tables in Parquet format to optimize the query performance on data lakes with small-scale filtering involved.
- Small I/O operations can be merged to reduce the delay for querying data lakes, thereby improving the query performance on external tables.
- Optimized the performance of window functions.
- Optimized the performance of Cross Join by supporting predicate pushdown.
- Histograms are added to CBO statistics. Full statistics collection is further optimized.
- Adaptive multi-threading is enabled for tablet scanning to reduce the dependency of scanning performance on the tablet number. As a result, you can set the number of buckets more easily.
- Supports querying compressed TXT files in Apache Hive.
- Adjusted the mechanisms of default PageCache size calculation and memory consistency check to avoid OOM issues during multi-instance deployments.
- Improved the performance of large-size batch load on the PRIMARY KEY model up to two times by removing final_merge operations.
- Supports a Stream Load transaction interface to implement a two-phase commit (2PC) for transactions that are run to load data from external systems such as Apache Flink® and Apache Kafka®, improving the performance of highly concurrent stream loads.
- Functions:
- You can use COUNT DISTINCT over multiple columns to calculate the number of distinct column combinations.
- Window functions min() and max() support sliding windows.
- Optimized the performance of the window_funnel function.

**Bug Fixes**

The following bugs are fixed:
- DECIMAL data types returned by DESC are different from those specified in the CREATE TABLE statement. [7309](https://github.com/StarRocks/starrocks/pull/7309)
- FE metadata management issues that affect the stability of FE. [6685](https://github.com/StarRocks/starrocks/pull/6685) [#9445](https://github.com/StarRocks/starrocks/pull/9445) [#7974](https://github.com/StarRocks/starrocks/pull/7974) [#7455](https://github.com/StarRocks/starrocks/pull/7455)
- Data load-related issues:
- Broke Load fails when ARRAY-type column is set. [9158](https://github.com/StarRocks/starrocks/pull/9158)
- Replicas are inconsistent after data is loaded to a non-Duplicate Key table via Broker Load. [8714](https://github.com/StarRocks/starrocks/pull/8714)
- Executing ALTER ROUTINE LOAD raises NPE. [7804](https://github.com/StarRocks/starrocks/pull/7804)
- Data Lake analytic-related issues:
- Queries on Parquet-format in Hive external tables fail. [7413](https://github.com/StarRocks/starrocks/pull/7413) [#7482](https://github.com/StarRocks/starrocks/pull/7482) [#7624](https://github.com/StarRocks/starrocks/pull/7624)
- Incorrect results are returned to queries with `limit` clause on Elasticsearch external table.[9226](https://github.com/StarRocks/starrocks/pull/9226)
- An unknown error is raised during queries on an Apache Iceberg table with a complex data type. [11298](https://github.com/StarRocks/starrocks/pull/11298)
- Metadata can be inconsistent between the Leader FE and Follower FE nodes. [11215](https://github.com/StarRocks/starrocks/pull/11215)
- BE crashes when BITMAP type data size is larger than 2GB. [11178](https://github.com/StarRocks/starrocks/pull/11178)

**Behavior Change**

- Page Cache is enabled by default. The default cache size is 20% of the system memory.

**Others**

- Announcing the stable release of Resource Group.
- Announcing the stable release of the JSON data type and its related functions.

**Improvements**

- The Pipeline execution engine supports INSERT INTO statements. To enable it, set the FE configuration item enable_pipeline_load_for_insert to true. [14723](https://github.com/StarRocks/starrocks/pull/14723)
- The memory used by Compaction for the primary key table is reduced. [13861](https://github.com/StarRocks/starrocks/pull/13861) [#13862](https://github.com/StarRocks/starrocks/pull/13862)


**Bug Fixes**
The following bugs are fixed:

- For aggregation queries and multi-table JOIN queries, the statistics are not collected accurately and CROSS JOIN occurs in the execution plan, resulting in long query latency. [15497](https://github.com/StarRocks/starrocks/pull/15497)
- When you create a materialized view by using CREATE MATERIALIZED VIEW AS SELECT, if the SELECT clause does not use aggregate functions, and uses GROUP BY, for example CREATE MATERIALIZED VIEW test_view AS SELECT a,b from test group by b,a order by a;, then the BE nodes all crash. [13743](https://github.com/StarRocks/starrocks/pull/13743)
- If you restart the BE immediately after you use INSERT INTO to frequently load data into the primary key table to make data changes, the BE may restart very slowly. [15128](https://github.com/StarRocks/starrocks/pull/15128)
- If only JRE is installed on the environment and JDK is not installed, queries fail after FE restarts. After the bug is fixed, FE cannot restart in that environment and it returns error JAVA_HOME can not be jre. To successfully restart FE, you need to install JDK on the environment. [14332](https://github.com/StarRocks/starrocks/pull/14332)
- Queries cause BE crashes. [14221](https://github.com/StarRocks/starrocks/pull/14221)
- exec_mem_limit cannot be set to an expression. [13647](https://github.com/StarRocks/starrocks/pull/13647)
- You cannot create a sync refreshed materialized view based on subquery results. [13507](https://github.com/StarRocks/starrocks/pull/13507)
- The comments for columns are deleted after you refresh the Hive external table. [13742](https://github.com/StarRocks/starrocks/pull/13742)
- During a correlated JOIN, the right table is processed before the left table and the right table is very large. If compaction is performed on the left table while the right table is being processed, the BE node crashes. [14070](https://github.com/StarRocks/starrocks/pull/14070)
- If the Parquet file column names are case-sensitive, and the query condition uses upper-case column names from the Parquet file, the query returns no result. [13860](https://github.com/StarRocks/starrocks/pull/13860) [#14773](https://github.com/StarRocks/starrocks/pull/14773)
- During bulk loading, if the number of connections to Broker exceeds the default maximum number of connections, Broker is disconnected and the loading job fails with an error message list path error. [13911](https://github.com/StarRocks/starrocks/pull/13911)
- When BEs are highly loaded, the metric for resource groups starrocks_be_resource_group_running_queries may be incorrect. [14043](https://github.com/StarRocks/starrocks/pull/14043)
- If the query statement uses OUTER JOIN, it may cause the BE node to crash. [14840](https://github.com/StarRocks/starrocks/pull/14840)
- After you create an asynchronous materialized view by using StarRocks 2.4, and you roll back it to 2.3, you may find FE fails to start. [14400](https://github.com/StarRocks/starrocks/pull/14400)
- When the primary key table uses delete_range, and the performance is not good, it may slow down data reading from RocksDB and cause high CPU usage. [15130](https://github.com/StarRocks/starrocks/pull/15130)

**Improvements**
- Colocate Join supports Equi Join. [13546](https://github.com/StarRocks/starrocks/pull/13546)
- Fix the problem that primary key index files are too large due to continuously appending WAL records when data is frequently loaded. [12862](https://github.com/StarRocks/starrocks/pull/12862)
- FE scans all tablets in batches so that FE releases db.readLock at scanning intervals in case of holding db.readLock for too long. [13070](https://github.com/StarRocks/starrocks/pull/13070)

**Bug Fixes**
The following bugs are fixed:
- When a view is created based directly on the result of UNION ALL, and the UNION ALL operator's input columns include NULL values, the schema of the view is incorrect since the data type of columns is NULL_TYPE rather than UNION ALL's input columns. [13917](https://github.com/StarRocks/starrocks/pull/13917)
- The query result of SELECT * FROM ... and SELECT * FROM ... LIMIT ... is inconsistent. [13585](https://github.com/StarRocks/starrocks/pull/13585)
- External tablet metadata synchronized to FE may overwrite local tablet metadata, which causes data loading from Flink to fail. [12579](https://github.com/StarRocks/starrocks/pull/12579)
- BE nodes crash when null filter in Runtime Filter handles literal constants. [13526](https://github.com/StarRocks/starrocks/pull/13526)
- An error is returned when you execute CTAS. [12388](https://github.com/StarRocks/starrocks/pull/12388)
- The metrics ScanRows collected by pipeline engine in audit log may be wrong. [12185](https://github.com/StarRocks/starrocks/pull/12185)
- The query result is incorrect when you query compressed HIVE data. [11546](https://github.com/StarRocks/starrocks/pull/11546)
- Queries are timeout and StarRocks responds slowly after a BE node crashes. [12955](https://github.com/StarRocks/starrocks/pull/12955)
- The error of Kerberos authentication failure occurs when you use Broker Load to load data. [13355](https://github.com/StarRocks/starrocks/pull/13355)
- Too many OR predicates cause statistics estimation to take too long. [13086](https://github.com/StarRocks/starrocks/pull/13086)
- BE node crashes if Broker Load loads ORC files (Snappy compression) contain uppercase column names. [12724](https://github.com/StarRocks/starrocks/pull/12724)
- An error is returned when unloading or querying Primary Key table takes more than 30 minutes. [13403](https://github.com/StarRocks/starrocks/pull/13403)
- The backup task fails when you back up large data volumes to HDFS by using a broker. [12836](https://github.com/StarRocks/starrocks/pull/12836)
- The data StarRocks read from Iceberg may be incorrect, which is caused by the parquet_late_materialization_enable parameter. [13132](https://github.com/StarRocks/starrocks/pull/13132)
- An error failed to init view stmt is returned when a view is created. [13102](https://github.com/StarRocks/starrocks/pull/13102)
- An error is returned when you use JDBC to connect StarRock and execute SQL statements. [13526 ](https://github.com/StarRocks/starrocks/pull/13526)
- The query is timeout because the query involves too many buckets and uses tablet hint. [13272](https://github.com/StarRocks/starrocks/pull/13272)
- A BE node crashes and cannot be restarted, and in the meantime, the loading job into a newly built table reports an error. [13701](https://github.com/StarRocks/starrocks/pull/13701)
- All BE nodes crash when a materialized view is created. [13184](https://github.com/StarRocks/starrocks/pull/13184)
- When you execute ALTER ROUTINE LOAD to update the offset of consumed partitions, an error The specified partition 1 is not in the consumed partitionsmay be returned, and followers eventually crash. [12227](https://github.com/StarRocks/starrocks/pull/12227)

Release date: November 10, 2022

**Improvements**

- The error message provides a solution when StarRocks fails to create a Routine Load job because the number of running Routine Load job exceeds the limit. [12204](https://github.com/StarRocks/starrocks/pull/12204)
- The query fails when StarRocks queries data from Hive and fails to parse CSV files. [13013](https://github.com/StarRocks/starrocks/pull/13013)

**Bug Fixes**
The following bugs are fixed:

- The query may fail if HDFS files paths contain (). [12660](https://github.com/StarRocks/starrocks/pull/12660)
- The result of ORDER BY ... LIMIT ... OFFSET is incorrect when the subquery contains LIMIT. [9698](https://github.com/StarRocks/starrocks/issues/9698)
- StarRocks is case-insensitive when querying ORC files. [12724](https://github.com/StarRocks/starrocks/pull/12724)
- BE may crash when RuntimeFilter is closed without invoking the prepare method. [12906](https://github.com/StarRocks/starrocks/issues/12906)
- BE may crash because of memory leak. [12906](https://github.com/StarRocks/starrocks/issues/12906)
- The query result may be incorrect after you add a new column and immediately delete data. [12907](https://github.com/StarRocks/starrocks/pull/12907)
- BE may crash because of sorting data. [11185](https://github.com/StarRocks/starrocks/pull/11185)
- If StarRocks and MySQL client are not on the same LAN, the loading job created by using INSERT INTO SELECT can not be terminated successfully by executing KILL only once. [11879](https://github.com/StarRocks/starrocks/pull/11897)
- The metrics ScanRows collected by pipeline engine in audit log may be wrong. [12185](https://github.com/StarRocks/starrocks/pull/12185)