Starrocks

Release date: May 15, 2024

Parameter changes

- Decreased the default value of BE parameter `update_compaction_size_threshold` from 256 MB to 64 MB to increase the trigger frequency for Primary Key table compaction.

Improvements

- Optimized the usage of database locks for materialized view refresh to prevent deadlock. [42801](https://github.com/StarRocks/starrocks/pull/42801)
- Both `s3a://` and `s3://` can be used to access data in AWS S3. [42460](https://github.com/StarRocks/starrocks/pull/42460)

Bug Fixes

Fixed the following issues:

- Schema change may cause issues in prefix index sorting, leading to incorrect results for queries based on prefix indexes. [44941](https://github.com/StarRocks/starrocks/pull/44941)
- After a Routine Load task is paused due to Kafka cluster abnormalities, the background still attempts to connect to this abnormal Kafka cluster, which prevents other Routine Load tasks in this StarRocks cluster from consuming normal Kafka messages. [45029](https://github.com/StarRocks/starrocks/pull/45029)
- When querying views in `information_schema`, the database lock is held for an unexpectedly long time, which prolongs the overall query time. [45392](https://github.com/StarRocks/starrocks/pull/45392)
- Enabling Query Cache may cause BEs to crash if the SQL query contains a HAVING clause. This issue can be resolved by disabling Query Cache using `set enable_query_cache=false`. [43823](https://github.com/StarRocks/starrocks/pull/43823)
- When Query Cache is enabled, some queries may return an error message `All slotIds should be remapped`. [42861](https://github.com/StarRocks/starrocks/pull/42861)

Release date: March 22, 2024

Improvements

- `replace_if_not_null` supports BITMAP columns in an Aggregate table. Users can specify `replace_if_not_null` as the aggregate function for BITMAP columns in an Aggregate table. [42104](https://github.com/StarRocks/starrocks/pull/42104)
- G1 Garbage Collector is used for JDK 9 and later by default. [41374](https://github.com/StarRocks/starrocks/pull/41374)

Parameter Changes

- The default value of the BE parameter `update_compaction_size_threshold` is changed from 256 MB to 64 MB to accelerate compaction. [42776](https://github.com/StarRocks/starrocks/pull/42776)

Bug Fixes

Fixed the following issues:

- Synchronizing data using StarRocks external tables encounters the error "commit and publish txn failed". The synchronization succeeds after a retry but the same copy of data is loaded twice. [25165](https://github.com/StarRocks/starrocks/pull/25165)
- RPC transmit resources are temporarily unavailable due to GC issues. [41636](https://github.com/StarRocks/starrocks/pull/41636)
- array_agg() in v2.5 processes NULLs in a different way than it does in v2.3. As a result, the query result is incorrect after an upgrade from v2.3 to v2.5. [42639](https://github.com/StarRocks/starrocks/pull/42639)
- The Sink Operator in a query unexpectedly exits, which causes BEs to crash. [38662](https://github.com/StarRocks/starrocks/pull/38662)
- Executing the DELETE command on an Aggregate table results in a race for accessing tablet metadata, which causes BEs to crash. [42174](https://github.com/StarRocks/starrocks/pull/42174)
- The MemTracker encounters the Use-After-Free issue during UDF calling, which causes BEs to crash. [41710](https://github.com/StarRocks/starrocks/pull/41710)
- The unnest() function does not support aliases. [42138](https://github.com/StarRocks/starrocks/pull/42138)

Release date: February 8, 2024

New features

- Added a pattern-matching function: [regexp_extract_all](https://docs.starrocks.io/docs/sql-reference/sql-functions/like-predicate-functions/regexp_extract_all/).
- Added Bitmap value processing functions: serialize, deserialize, and serializeToString. [40162](https://github.com/StarRocks/starrocks/pull/40162/files)

Improvements

- Supports automatic activation of inactive materialized views when refreshing these materialized views. [38521](https://github.com/StarRocks/starrocks/pull/38521)
- Optimized BE log printing to prevent too many irrelevant logs. [22820](https://github.com/StarRocks/starrocks/pull/22820) [#36187](https://github.com/StarRocks/starrocks/pull/36187)
- Supports using [Hive UDFs](https://docs.starrocks.io/docs/integrations/hive_bitmap_udf/) to process and load Bitmap data into StarRocks and export Bitmap data from StarRocks to Hive. [#40165](https://github.com/StarRocks/starrocks/pull/40165) [#40168](https://github.com/StarRocks/starrocks/pull/40168)
- Added date formats `yyyy-MM-ddTHH:mm` and `yyyy-MM-dd HH:mm` to support TIMESTAMP partition fields in Apache Iceberg tables. [39986](https://github.com/StarRocks/starrocks/pull/39986)

Bug Fixes

Fixed the following issues:

- Running a Spark Load job that has no PROPERTIES specified causes null pointer exceptions (NPEs). [38765](https://github.com/StarRocks/starrocks/pull/38765)
- INSERT INTO SELECT occasionally encounters the error "timeout by txn manager". [36688](https://github.com/StarRocks/starrocks/pull/36688)
- The memory consumption of PageCache exceeds the threshold specified by the BE dynamic parameter `storage_page_cache_limit` in certain circumstances. [37740](https://github.com/StarRocks/starrocks/pull/37740)
- After a table is dropped and then re-created with the same table name, refreshing asynchronous materialized views created on that table fails. [38008](https://github.com/StarRocks/starrocks/pull/38008) [#38982](https://github.com/StarRocks/starrocks/pull/38982)
- Writing data to S3 buckets using SELECT INTO occasionally encounters the error "The tablet write operation update metadata take a long time". [38443](https://github.com/StarRocks/starrocks/pull/38443)
- Some operations during data loading may encounter "reached timeout". [36746](https://github.com/StarRocks/starrocks/pull/36746)
- The DECIMAL type returned by SHOW CREATE TABLE is inconsistent with that specified in CREATE TABLE. [39297](https://github.com/StarRocks/starrocks/pull/39297)
- If partition columns in external tables contain null values, queries against those tables will cause BEs to crash. [38888](https://github.com/StarRocks/starrocks/pull/38888)
- When deleting data from a Duplicate Key table, if the condition in the WHERE clause of the DELETE statement has a leading space, the deleted data can still be queried using SELECT. [39797](https://github.com/StarRocks/starrocks/pull/39797)
- Loading `array<string>` data from ORC files into StarRocks (`array<json>`) may cause BEs to crash. [39233](https://github.com/StarRocks/starrocks/pull/39233)
- Querying Hive catalogs may be stuck and even expire. [39863](https://github.com/StarRocks/starrocks/pull/39863)
- Partitions cannot be dynamically created if hour-level partitions are specified in the PARTITION BY clause. [40256](https://github.com/StarRocks/starrocks/pull/40256)
- The error message "failed to call frontend service" is returned during loading from Apache Flink. [40710](https://github.com/StarRocks/starrocks/pull/40710)

RRelease date: Jan 10, 2024

New Features

- Users can set or modify session variables when they [CREATE](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-definition/CREATE_MATERIALIZED_VIEW/#parameters) or [ALTER](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-definition/ALTER_MATERIALIZED_VIEW/) asynchronous materialized views. [#37401](https://github.com/StarRocks/starrocks/pull/37401)

Improvements

- When using JDK, the default GC algorithm is G1. [37498](https://github.com/StarRocks/starrocks/pull/37498)
- The result returned by the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/SHOW_ROUTINE_LOAD/) statement now includes the timestamps of consumption messages from each partition. [#36222](https://github.com/StarRocks/starrocks/pull/36222)

Behavior Change

- Added the session variable `enable_materialized_view_for_insert`, which controls whether materialized views rewrite the queries in INSERT INTO SELECT statements. The default value is `false`. [37505](https://github.com/StarRocks/starrocks/pull/37505)
- Added the session variable `enable_strict_order_by`. When this variable is set to the default value `TRUE`, an error is reported for such a query pattern: Duplicate alias is used in different expressions of the query and this alias is also a sorting field in ORDER BY, for example, `select distinct t1.* from tbl1 t1 order by t1.k1;`. The logic is the same as that in v2.3 and earlier. When this variable is set to `FALSE`, a loose deduplication mechanism is used, which processes such queries as valid SQL queries. [37910](https://github.com/StarRocks/starrocks/pull/37910)

Parameter Change

- Added session variables `transaction_read_only` and `tx_read_only` to specify the transaction access mode, which are compatible with MySQL versions 5.7.20 and above. [37249](https://github.com/StarRocks/starrocks/pull/37249)
- Added the FE configuration item `routine_load_unstable_threshold_second`. [36222](https://github.com/StarRocks/starrocks/pull/36222)
- Added the FE configuration item `http_worker_threads_num`, which specifies the number of threads for HTTP server to deal with HTTP requests. The default value is `0`. If the value for this parameter is set to a negative value or 0, the actual thread number is twice the number of CPU cores. [37530](https://github.com/StarRocks/starrocks/pull/37530)
- Added the BE configuration item `pindex_major_compaction_limit_per_disk` to configure the maximum concurrency of compaction on a disk. This addresses the issue of uneven I/O across disks due to compaction. This issue can cause excessively high I/O for certain disks. The default value is `1`. [37695](https://github.com/StarRocks/starrocks/pull/37695)

Bug Fixes

Fixed the following issues:

- Using NaN (Not a Number) columns as ORDER BY columns may cause BEs to crash. [30759](https://github.com/StarRocks/starrocks/pull/30759)
- Failure to update primary key indexes may cause the error "get_applied_rowsets failed". [27488](https://github.com/StarRocks/starrocks/pull/27488)
- Hive metadata in [Hive catalogs](https://docs.starrocks.io/docs/2.5/data_source/catalog/hive_catalog/) is not automatically refreshed when new fields are added to Hive tables. [#37668](https://github.com/StarRocks/starrocks/pull/37668)
- When `SELECT ... FROM ... INTO OUTFILE` is executed to export data into CSV files, the error "Unmatched number of columns" is reported if the FROM clause contains multiple constants. [38045](https://github.com/StarRocks/starrocks/pull/38045)
- In some cases, `bitmap_to_string` may return incorrect result due to data type overflow. [37405](https://github.com/StarRocks/starrocks/pull/37405)

Release date: December 19, 2023

New Features

- Added a new metric `max_tablet_rowset_num` for setting the maximum allowed number of rowsets. This metric helps detect possible compaction issues and thus reduces the occurrences of the error "too many versions". [36539](https://github.com/StarRocks/starrocks/pull/36539)
- Added the [subdivide_bitmap](https://docs.starrocks.io/zh/docs/sql-reference/sql-functions/bitmap-functions/subdivide_bitmap/) function. [#35817](https://github.com/StarRocks/starrocks/pull/35817)

Improvements

- The result returned by the [SHOW ROUTINE LOAD](https://docs.starrocks.io/zh/docs/sql-reference/sql-statements/data-manipulation/SHOW_ROUTINE_LOAD/) statement provides a new field `OtherMsg`, which shows information about the last failed task. [#35806](https://github.com/StarRocks/starrocks/pull/35806)
- The default retention period of trash files is changed to 1 day from the original 3 days. [37113](https://github.com/StarRocks/starrocks/pull/37113)
- Optimized the performance of persistent index update when compaction is performed on all rowsets of a Primary Key table, which reduces disk read I/O. [36819](https://github.com/StarRocks/starrocks/pull/36819)
- Optimized the logic used to compute compaction scores for Primary Key tables, thereby aligning the compaction scores for Primary Key tables within a more consistent range with the other three table types. [36534](https://github.com/StarRocks/starrocks/pull/36534)
- Queries on MySQL external tables and the external tables within JDBC catalogs support including keywords in the WHERE clause. [35917](https://github.com/StarRocks/starrocks/pull/35917)
- Added the bitmap_from_binary function to Spark Load to support loading Binary data. [36050](https://github.com/StarRocks/starrocks/pull/36050)
- The bRPC expiration time is shortened from 1 hour to the duration specified by the session variable [`query_timeout`](https://docs.starrocks.io/zh/docs/3.2/reference/System_variable/#query_timeout). This prevents query failures caused by RPC request expiration. [36778](https://github.com/StarRocks/starrocks/pull/36778)

Compatibility Changes

Parameters

- A new BE configuration item `enable_stream_load_verbose_log` is added. The default value is `false`. With this parameter set to `true`, StarRocks can record the HTTP requests and responses for Stream Load jobs, making troubleshooting easier. [36113](https://github.com/StarRocks/starrocks/pull/36113)
- The BE static parameter `update_compaction_per_tablet_min_interval_seconds` becomes mutable. [36819](https://github.com/StarRocks/starrocks/pull/36819)

Bug Fixes

Fixed the following issues:

- Queries fail during hash joins, causing BEs to crash. [32219](https://github.com/StarRocks/starrocks/pull/32219)
- The FE performance plunges after the FE configuration item `enable_collect_query_detail_info` is set to `true`. [35945](https://github.com/StarRocks/starrocks/pull/35945)
- Errors may be thrown if large amounts of data are loaded into a Primary Key table with persistent index enabled. [34352](https://github.com/StarRocks/starrocks/pull/34352)
- The starrocks_be process may exit unexpectedly when `./agentctl.sh stop be` is used to stop a BE. [35108](https://github.com/StarRocks/starrocks/pull/35108)
- The [array_distinct](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_distinct/) function occasionally causes the BEs to crash. [#36377](https://github.com/StarRocks/starrocks/pull/36377)
- Deadlocks may occur when users refresh materialized views. [35736](https://github.com/StarRocks/starrocks/pull/35736)
- In some scenarios, dynamic partitioning may encounter an error, which causes FE start failures. [36846](https://github.com/StarRocks/starrocks/pull/36846)

Release date: December 1, 2023

Bug Fixes
Fixed the following issues:

- Global Runtime Filter may cause BEs to crash in certain scenarios. https://github.com/StarRocks/starrocks/pull/35776