Starrocks

Release date: November 29, 2023

Improvements
- Added slow request logs to track slow requests. https://github.com/StarRocks/starrocks/pull/33908
- Optimized the performance of using Spark Load to read Parquet and ORC files when there are a large number of files. https://github.com/StarRocks/starrocks/pull/34787
- Optimized the performance of some Bitmap-related operations, including:
- Optimized nested loop joins. https://github.com/StarRocks/starrocks/pull/34804 https://github.com/StarRocks/starrocks/pull/35003
- Optimized the bitmap_xor function. https://github.com/StarRocks/starrocks/pull/34069
- Supports Copy on Write to optimize Bitmap performance and reduce memory consumption. https://github.com/StarRocks/starrocks/pull/34047

Compatibility Changes
Parameters
- The FE dynamic parameter enable_new_publish_mechanism is changed to a static parameter. You must restart the FE after you modify the parameter settings. https://github.com/StarRocks/starrocks/pull/35338

Bug Fixes
- If a filtering condition is specified in a Broker Load job, BEs may crash during the data loading in certain circumstances. https://github.com/StarRocks/starrocks/pull/29832
- Failures in replaying replica operations may cause FEs to crash. https://github.com/StarRocks/starrocks/pull/32295
- Setting the FE parameter recover_with_empty_tablet to true may cause FEs to crash. https://github.com/StarRocks/starrocks/pull/33071
- The error "get_applied_rowsets failed, tablet updates is in error state: tablet:18849 actual row size changed after compaction" is returned for queries. https://github.com/StarRocks/starrocks/pull/33246
- A query that contains a window function may cause BEs to crash. https://github.com/StarRocks/starrocks/pull/33671
- Running show proc '/statistic' may cause a deadlock. [34237](https://github.com/StarRocks/starrocks/pull/34237/files)
- Errors may be thrown if large amounts of data are loaded into a Primary Key table with persistent index enabled. https://github.com/StarRocks/starrocks/pull/34566
- After StarRocks is upgraded from v2.4 or earlier to a later version, compaction scores may rise unexpectedly. https://github.com/StarRocks/starrocks/pull/34618
- If INFORMATION_SCHEMA is queried by using the database driver MariaDB ODBC, the CATALOG_NAME column returned in the schemata view holds only null values. https://github.com/StarRocks/starrocks/pull/34627
- If schema changes are being executed while a Stream Load job is in the PREPARD state, a portion of the source data to be loaded by the job is lost. https://github.com/StarRocks/starrocks/pull/34381
- Including two or more slashes (/) at the end of the HDFS storage path causes the backup and restore of the data from HDFS to fail. https://github.com/StarRocks/starrocks/pull/34601
- Running a loading task or a query may cause the FEs to hang. https://github.com/StarRocks/starrocks/pull/34569

Release date: November 14, 2023
Improvements
- The COLUMNS table in the system database INFORMATION_SCHEMA can display ARRAY, MAP, and STRUCT columns. https://github.com/StarRocks/starrocks/pull/33431

Bug Fixes
Fixed the following issues:

- The error java.lang.IllegalStateException: null is reported if the ON condition is nested with a subquery. https://github.com/StarRocks/starrocks/pull/30876
- The result of COUNT() is inconsistent among replicas if COUNT() is run immediately after INSERT INTO SELECT ... LIMIT is successfully executed. https://github.com/StarRocks/starrocks/pull/24435
- BE may crash for specific data types if the target data type specified in CAST is the same as the original data type. https://github.com/StarRocks/starrocks/pull/31465
- An error is reported if specific path formats are used during data loading via Broker Load: msg:Fail to parse columnsFromPath, expected: [rec_dt]. https://github.com/StarRocks/starrocks/issues/32721
- During an upgrade to 3.x, if some column types are also upgraded (for example, Decimal is upgraded to Decimal v3), BEs crash when Compaction is performed on tables with specific characteristics. https://github.com/StarRocks/starrocks/pull/31626
- When data is loaded by using Flink Connector, the load job is suspended unexpectedly if there are highly concurrent load jobs and both the number of HTTP and Scan threads have reached their upper limits. https://github.com/StarRocks/starrocks/pull/32251
- BEs crash when libcurl is invoked. https://github.com/StarRocks/starrocks/pull/31667
- Adding BITMAP columns to a Primary Key table fails with the following error: Analyze columnDef error: No aggregate function specified for 'userid'. https://github.com/StarRocks/starrocks/pull/31763
- Long-time, frequent data loading into a Primary Key table with persistent index enabled may cause BEs to crash. https://github.com/StarRocks/starrocks/pull/33220
- The query result is incorrect when Query Cache is enabled. https://github.com/StarRocks/starrocks/pull/32778
- Specifying a nullable Sort Key when creating a Primary Key table causes compaction to fail. https://github.com/StarRocks/starrocks/pull/29225
- The error "StarRocks planner use long time 10000 ms in logical phase" occassionally occurs for complex Join queries. https://github.com/StarRocks/starrocks/pull/34177

Release date: September 28, 2023

Improvements
- Window functions COVAR_SAMP, COVAR_POP, CORR, VARIANCE, VAR_SAMP, STD, and STDDEV_SAMP now support the ORDER BY clause and Window clause. [30786](https://github.com/StarRocks/starrocks/pull/30786)
- An error instead of NULL is returned if a decimal overflow occurs during queries on the DECIMAL type data. [30419](https://github.com/StarRocks/starrocks/pull/30419)
- Executing SQL commands with invalid comments now returns results consistent with MySQL. [30210](https://github.com/StarRocks/starrocks/pull/30210)
- Rowsets corresponding to tablets that have been deleted are cleaned up, reducing the memory usage during BE startup. [30625](https://github.com/StarRocks/starrocks/pull/30625)

Bug Fixes
Fixed the following issues:

- An error "Set cancelled by MemoryScratchSinkOperator" occurs when users read data from StarRocks using the Spark Connector or Flink Connector. [30702](https://github.com/StarRocks/starrocks/pull/30702) [#30751](https://github.com/StarRocks/starrocks/pull/30751)
- An error "java.lang.IllegalStateException: null" occurs during queries with an ORDER BY clause that includes aggregate functions. [30108](https://github.com/StarRocks/starrocks/pull/30108)
- FEs fail to restart when there are inactive materialized views. [30015](https://github.com/StarRocks/starrocks/pull/30015)
- Performing INSERT OVERWRITE operations on duplicate partitions corrupts the metadata, leading to FE restart failures. [27545](https://github.com/StarRocks/starrocks/pull/27545)
- An error "java.lang.NullPointerException: null" occurs when users modify columns that do not exist in a Primary Key table. [30366](https://github.com/StarRocks/starrocks/pull/30366)
- An error "get TableMeta failed from TNetworkAddress" occurs when users load data into a partitioned StarRocks external table. [30124](https://github.com/StarRocks/starrocks/pull/30124)
- If users use CloudCanal to load data into table columns that are set to NOT NULL but have no default value specified, an error "Unsupported dataFormat value is : \N" is thrown. [30799](https://github.com/StarRocks/starrocks/pull/30799)
- An error "current running txns on db xxx is 200, larger than limit 200" occurs when users load data via the Flink Connector or perform DELETE and INSERT operations. [18393](https://github.com/StarRocks/starrocks/pull/18393)
- Asynchronous materialized views which use HAVING clauses that include aggregate functions cannot rewrite queries properly. [29976](https://github.com/StarRocks/starrocks/pull/29976)

Release date: September 4, 2023

Improvements
- Comments in an SQL are retained in the Audit Log. [29747](https://github.com/StarRocks/starrocks/pull/29747)
- Added CPU and memory statistics of INSERT INTO SELECT to the Audit Log. [29901](https://github.com/StarRocks/starrocks/pull/29901)

Bug Fixes
Fixed the following issues:

- When Broker Load is used to load data, the NOT NULL attribute of some fields may cause BEs to crash or cause the "msg:mismatched row count" error. [29832](https://github.com/StarRocks/starrocks/pull/29832)
- Queries against ORC-formatted files fail because the bugfix ORC-1304 ([apache/orc1299](https://github.com/apache/orc/pull/1299)) from Apache ORC is not merged. [#29804](https://github.com/StarRocks/starrocks/pull/29804)
- Restoring Primary Key tables causes metadata inconsistency after BEs are restarted. [30135](https://github.com/StarRocks/starrocks/pull/30135)

Release date: August 7, 2023

New features
- Supports aggregate functions [COVAR_SAMP](https://docs.starrocks.io/en-us/2.5/sql-reference/sql-functions/aggregate-functions/covar_samp), [COVAR_POP](https://docs.starrocks.io/en-us/2.5/sql-reference/sql-functions/aggregate-functions/covar_pop), and [CORR](https://docs.starrocks.io/en-us/2.5/sql-reference/sql-functions/aggregate-functions/corr).
- Supports the following [window functions](https://docs.starrocks.io/en-us/2.5/sql-reference/sql-functions/Window_function): COVAR_SAMP, COVAR_POP, CORR, VARIANCE, VAR_SAMP, STD, and STDDEV_SAMP.

Improvements
- Optimized the scheduling logic of TabletChecker to prevent the checker from repeatedly scheduling tablets that are not repaired. [27648](https://github.com/StarRocks/starrocks/pull/27648)
- When Schema Change and Routine Load occur simultaneously, Routine Load jobs may fail if Schema Change completes first. The error message reported in this situation is optimized. [28425](https://github.com/StarRocks/starrocks/pull/28425)
- Users are prohibited from defining NOT NULL columns when they create external tables (If NOT NULL columns are defined, errors will occur after an upgrade and the table must be created again). External catalogs are recommended starting from v2.3.0 to replace external tables. [25485](https://github.com/StarRocks/starrocks/pull/25441)
- Added an error message when Broker Load retries encounter an error. This facilitates troubleshooting and debugging during data loading. [21982](https://github.com/StarRocks/starrocks/pull/21982)
- Supports large-scale data writes when a load job involves both UPSERT and DELETE operations. [17264](https://github.com/StarRocks/starrocks/pull/17264)
- Optimized query rewrite using materialized views. [27934](https://github.com/StarRocks/starrocks/pull/27934) [#25542](https://github.com/StarRocks/starrocks/pull/25542) [#22300](https://github.com/StarRocks/starrocks/pull/22300) [#27557](https://github.com/StarRocks/starrocks/pull/27557) [#22300](https://github.com/StarRocks/starrocks/pull/22300) [#26957](https://github.com/StarRocks/starrocks/pull/26957) [#27728](https://github.com/StarRocks/starrocks/pull/27728) [#27900](https://github.com/StarRocks/starrocks/pull/27900)

Bug Fixes
Fixed the following issues:
- When CAST is used to convert a string into an array, the result may be incorrect if the input includes constants. [19793](https://github.com/StarRocks/starrocks/pull/19793)
- SHOW TABLET returns incorrect results if it contains ORDER BY and LIMIT. [23375](https://github.com/StarRocks/starrocks/pull/23375)
- Outer join and Anti join rewrite errors for materialized views. [28028](https://github.com/StarRocks/starrocks/pull/28028)
- Incorrect table-level scan statistics in FE cause inaccurate metrics for table queries and loading. [27779](https://github.com/StarRocks/starrocks/pull/27779)
- "An exception occurred when using the current long link to access metastore. msg: Failed to get next notification based on last event id: 707602" is reported in FE logs if event listener is configured on the HMS to incrementally update Hive metadata. [21056](https://github.com/StarRocks/starrocks/pull/21056)
- The query result is not stable if the sort key is modified for a partitioned table. [27850](https://github.com/StarRocks/starrocks/pull/27850)
- Data loaded using Spark Load may be distributed to the wrong buckets if the bucketing column is a DATE, DATETIME, or DECIMAL column. [27005](https://github.com/StarRocks/starrocks/pull/27005)
- The regex_replace function may cause BEs to crash in some scenarios. [27117](https://github.com/StarRocks/starrocks/pull/27117)
- BE crashes if the input of the sub_bitmap function is not a BITMAP value. [27982](https://github.com/StarRocks/starrocks/pull/27982)
- "Unknown error" is returned for a query when Join Reorder is enabled. [27472](https://github.com/StarRocks/starrocks/pull/27472)
- Inaccurate estimation of average row size causes Primary Key partial updates to occupy excessively large memory. [27485](https://github.com/StarRocks/starrocks/pull/27485)
- Some INSERT jobs return "[42000][1064] Dict Decode failed, Dict can't take cover all key :0" if low-cardinality optimization is enabled. [26463](https://github.com/StarRocks/starrocks/pull/26463)
- If users specify "hadoop.security.authentication" = "simple" in their Broker Load jobs created to load data from HDFS, the job fails. [27774](https://github.com/StarRocks/starrocks/pull/27774)
- Modifying the refresh mode of materialized views causes inconsistent metadata between the leader FE and follower FE. [28082](https://github.com/StarRocks/starrocks/pull/28082) [#28097](https://github.com/StarRocks/starrocks/pull/28097)
- Passwords are not hidden when SHOW CREATE CATALOG and SHOW RESOURCES are used to query specific information. [28059](https://github.com/StarRocks/starrocks/pull/28059)
- FE memory leak caused by blocked LabelCleaner threads. [28311](https://github.com/StarRocks/starrocks/pull/28311)

Release date: July 19, 2023

New features
- Queries that contain a different type of join than the materialized view can be rewritten. [25099](https://github.com/StarRocks/starrocks/pull/25099)

Improvements
- StarRocks external tables whose destination cluster is the current StarRocks cluster cannot be created. [25441](https://github.com/StarRocks/starrocks/pull/25441)
- If the queried fields are not included in the output columns of a materialized view but are included in the predicate of the materialized view, the query can still be rewritten. [23028](https://github.com/StarRocks/starrocks/issues/23028)
- Added a new field table_id to the table tables_config in the database Information_schema. You can join tables_config with be_tablets on the column table_id to query the names of the database and table to which a tablet belongs. [24061](https://github.com/StarRocks/starrocks/pull/24061)

Bug Fixes
Fixed the following issues:
- Count Distinct result is incorrect for Duplicate Key tables. [24222](https://github.com/StarRocks/starrocks/pull/24222)
- BEs may crash if the Join key is a large BINARY column. [25084](https://github.com/StarRocks/starrocks/pull/25084)
- The INSERT operation hangs if the length of CHAR data in a STRUCT to be inserted exceeds the maximum CHAR length defined in the STRUCT column. [25942](https://github.com/StarRocks/starrocks/pull/25942)
- The result of coalesce() is incorrect. [26250](https://github.com/StarRocks/starrocks/pull/26250)
- The version number for a tablet is inconsistent between the BE and FE after data is restored. [26518](https://github.com/StarRocks/starrocks/pull/26518/files)
- Partitions cannot be automatically created for recovered tables. [26813](https://github.com/StarRocks/starrocks/pull/26813)