Starrocks

Release date: May 30, 2024

New Features
- Flink connector supports reading complex data types ARRAY, MAP, and STRUCT from StarRocks. [42932](https://github.com/StarRocks/starrocks/pull/42932) [#347](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/347)


Improvements
- When using the Broker process, Broker Load supports loading data from COS posix buckets, and SELECT ... FROM ... INTO OUTFILE supports unloading data to COS posix buckets. The format of the path parameter is cosn://some_bucket/xxx. [46090](https://github.com/StarRocks/starrocks/pull/46090)
- Previously, when BE failed to communicate with FE via RPC, FE would return a generic error message: call frontend service failed reason=xxx, making it unclear what the specific issue was. The error messages are now optimized to include specific reasons, such as timeout or server busy. [44153](https://github.com/StarRocks/starrocks/pull/44153)
- Improved error messages to indicate specific issues during data loading, such as the number of error data rows exceeding limits, mismatched column numbers, invalid column names, and no data in any partition.

Security
- Upgraded Kafka client dependency to v3.4.0 to fix the CVE-2023-25194 security issue. [45382](https://github.com/StarRocks/starrocks/pull/45382)


Bug Fixes
Fixed the following issues:
- If a materialized view definition includes multiple self-joins of the same table and incremental refreshes by partitions based on that table, incorrect results would occur due to wrong partition selection. [45936](https://github.com/StarRocks/starrocks/pull/45936)
- FEs crash when a Bitmap index is created in a materialized view in shared-data clusters causes . [45665](https://github.com/StarRocks/starrocks/pull/45665)
- BEs crash due to null pointer issues when FE follower is connected via ODBC and CREATE TABLE is executed. [45043](https://github.com/StarRocks/starrocks/pull/45043)
- Querying information_schema.task_runs fails frequently when many asynchronous tasks exist. [45520](https://github.com/StarRocks/starrocks/pull/45520)
- When a SQL statement contains multiple COUNT DISTINCT and includes LIMIT, LIMIT is wrongly processed, resulting in inconsistent data returned each time the statement is executed. [44749](https://github.com/StarRocks/starrocks/pull/44749)
- Queries with ORDER BY LIMIT clauses on Duplicate Key tables and Aggregate tables produce incorrect results. [45037](https://github.com/StarRocks/starrocks/pull/45037)

Release date: April 28, 2024

Behavior Changes

- Users are not allowed to drop views in the system database `information_schema` using DROP TABLE. [43556](https://github.com/StarRocks/starrocks/pull/43556)
- Users are not allowed to specify duplicate keys in the ORDER BY clause when creating a Primary Key table. [43374](https://github.com/StarRocks/starrocks/pull/43374)

Improvements

- Queries on Parquet-formatted Iceberg v2 tables support equality deletes.

Bug Fixes

Fixed the following issues:

- When a user queries data from an external table in an external catalog, access to this table is denied even when the user has the SELECT privilege on this table. SHOW GRANTS also shows that the user has this privilege. [44061](https://github.com/StarRocks/starrocks/pull/44061)
- `str_to_map` may cause BEs to crash. [43930](https://github.com/StarRocks/starrocks/pull/43930)
- When a Routine Load job is going on, running `show proc '/routine_loads'` is stuck due to deadlock. [44249](https://github.com/StarRocks/starrocks/pull/44249)
- Persistent Index of Primary Key tables may cause BEs to crash due to issues in concurrency control. [43720](https://github.com/StarRocks/starrocks/pull/43720)
- The `pending_task_run_count` displayed on the page of `leaderFE_IP:8030` is incorrect. The displayed number is the sum of Pending and Running tasks, not Pending tasks. In addition, the information of the metric `refresh_pending` cannot be displayed using `followerFE_IP:8030`. [43052](https://github.com/StarRocks/starrocks/pull/43052)
- Some SQL queries that contain CTEs may encounter the `Invalid plan: PhysicalTopNOperator` error. [44185](https://github.com/StarRocks/starrocks/pull/44185)

Release date: March 29, 2024

> TIP
> This version has been taken offline due to privilege issues in querying external tables in external catalogs such as Hive and Iceberg.
>
> Problem: When a user queries data from an external table in an external catalog, access to this table is denied even when the user has the SELECT privilege on this table. SHOW GRANTS also shows that the user has this privilege.
>
> Impact scope: This problem only affects queries on external tables in external catalogs. Other queries are not affected.
>
> Temporary workaround: The query succeeds after the SELECT privilege on this table is granted to the user again. But SHOW GRANTS will return duplicate privilege entries. After an upgrade to v3.2.6, users can run REVOKE to remove one of the privilege entries.

New Features

- Primary Key tables support Size-tiered Compaction. [42474](https://github.com/StarRocks/starrocks/pull/42474)

Behavior Changes

- When null values in JSON data are evaluated based on the `IS NULL` operator, they are considered NULL values following SQL language. For example, `true` is returned for `SELECT parse_json('{"a": null}') -> 'a' IS NULL` (before this behavior change, `false` is returned). [42815](https://github.com/StarRocks/starrocks/pull/42815)

Improvements

- When Broker Load is used to load data from ORC files that contain TIMESTAMP-type data, StarRocks supports retaining microseconds in the timestamps when converting the timestamps to match its own DATETIME data type. [42348](https://github.com/StarRocks/starrocks/pull/42348)

Bug Fixes

Fixed the following issues:

- In shared-data mode, the garbage collection and thread eviction mechanisms for handling persistent indexes created on Primary Key tables cannot take effect on CN nodes. As a result, obsolete data cannot be deleted. [42241](https://github.com/StarRocks/starrocks/pull/42241)
- When users query ORC files by using Hive catalogs, the query results may be incorrect because StarRocks used to read ORC files from Hive based on mapping by position. To resolve this issue, users can set the session variable `orc_use_column_names` to `true`, which specifies to read ORC files from Hive based on mapping by column name. [42905](https://github.com/StarRocks/starrocks/pull/42905)
- When LDAP authentication for the AD system is adopted, logins without passwords are allowed. [42476](https://github.com/StarRocks/starrocks/pull/42476)
- When disk device names end with digits, the values of monitoring metrics remain 0s because the disk device names may be invalid after such digits are removed. [42741](https://github.com/StarRocks/starrocks/pull/42741)

Release date: March 8, 2024

New Features

- Cloud-native Primary Key tables in shared-data clusters support Size-tiered Compaction to reduce write I/O amplification for the loading of a large number of small-sized files. [41610](https://github.com/StarRocks/starrocks/pull/41610)
- Added the function `regexp_extract_all`. [42178](https://github.com/StarRocks/starrocks/pull/42178)
- Added the view `information_schema.partitions_meta`, which records detailed metadata of partitions. [41101](https://github.com/StarRocks/starrocks/pull/41101)
- Added the view `sys.fe_memory_usage`, which records the memory usage for StarRocks. [41083](https://github.com/StarRocks/starrocks/pull/41083)

Behavior Changes

- The logic of dynamic partitioning is changed. Now partition columns of the DATE type do not support hour-level data. Note that partition columns of the DATETIME type still support hour-level data. [40328](https://github.com/StarRocks/starrocks/pull/40328)
- The user who can refresh materialized views is changed from the `root` user to the user who creates the materialized views. This change does not affect existing materialized views. [40698](https://github.com/StarRocks/starrocks/pull/40698)
- By default, when comparing columns of constant and string types, StarRocks compares them as strings. Users can use the session variable `cbo_eq_base_type` to adjust the default rule used for the comparison. For example, users can set `cbo_eq_base_type` to `decimal`, and StarRocks then compares the columns as numeric values. [41712](https://github.com/StarRocks/starrocks/pull/41712)

Improvements

- StarRocks supports using the parameter `s3_compatible_fs_list` to specify which S3-compatible object storage can be accessed via AWS SDK, and supports using the parameter `fallback_to_hadoop_fs_list` to specify non-S3-compatible object storage that require access via HDFS Schema (this method necessitates the use of vendor-provided JAR packages). [41612](https://github.com/StarRocks/starrocks/pull/41612)
- The compatibility with Trino's SQL statement syntax is optimized to support converting the following functions of Trino: `current_catalog`, `current_schema`, `to_char`, `from_hex`, `to_date`, `to_timestamp`, and `index`. [41505](https://github.com/StarRocks/starrocks/pull/41505) [#41270](https://github.com/StarRocks/starrocks/pull/41270) [#40838](https://github.com/StarRocks/starrocks/pull/40838)
- A new session variable `cbo_materialized_view_rewrite_related_mvs_limit` is added to control the maximum number of candidate materialized views allowed during query planning. The default value of this session variable is `64`. This session variable helps mitigate the excessive resource consumption caused by a large number of candidate materialized views for a query during the query planning. [39829](https://github.com/StarRocks/starrocks/pull/39829)
- The `agg_type` of BITMAP-type columns in an Aggregate table can be set to `replace_if_not_null` to support updates only to a few columns of the table. [42102](https://github.com/StarRocks/starrocks/pull/42102)
- The session variable `cbo_eq_base_type` is optimized to support specifying the implicit conversion rule applied to the comparison of data that contains both string and numeric data types. By default, such data is compared as strings. [40619](https://github.com/StarRocks/starrocks/pull/41712)
- More DATE-type data (for example, "%Y-%m-%e %H:%i") can be recognized to better support partition expressions for Iceberg tables. [40474](https://github.com/StarRocks/starrocks/pull/40474)
- The JDBC connector supports the TIME data type. [31940](https://github.com/StarRocks/starrocks/pull/31940)
- The `path` parameter in the SQL statement for creating a file external table supports wildcards (`*`). However, like the `DATA INFILE` parameter in the SQL statement for creating a Broker Load job, the `path` parameter supports using wildcards (`*`) to match at most one level of directory or file. [40844](https://github.com/StarRocks/starrocks/pull/40844)
- A new internal SQL log file is added to record log data related to statistics and materialized views. [40682](https://github.com/StarRocks/starrocks/pull/40682)

Bug Fixes

Fixed the following issues:

- "Analyze Error" is thrown if inconsistent letter cases are assigned to the names or aliases of tables or views queried in the creation of a Hive view. [40921](https://github.com/StarRocks/starrocks/pull/40921)
- I/O usage reaches the upper limit if persistent indexes are created on Primary Key tables. [39959](https://github.com/StarRocks/starrocks/pull/39959)
- In shared-data clusters, the primary key index directory is deleted every 5 hours. [40745](https://github.com/StarRocks/starrocks/pull/40745)
- After a table for which list partitioning is enabled is truncated or its partitions are truncated, queries based on the partitioning keys of the table return no data. [40495](https://github.com/StarRocks/starrocks/pull/40495)
- After users execute ALTER TABLE COMPACT by hand, the memory usage statistics for compaction operations are abnormal. [41150](https://github.com/StarRocks/starrocks/pull/41150)
- During data migration between clusters, if only some columns are updated in column mode, the destination cluster may crash. [40692](https://github.com/StarRocks/starrocks/pull/40692)
- The SQL blacklist may not take effect if the submitted SQL statement contains multiple spaces or newline characters. [40457](https://github.com/StarRocks/starrocks/pull/40457)

Release date: February 6, 2024

New Features
- StarRocks Community provides the StarRocks Cross-cluster Data Migration Tool, which supports migrating data from a shared-nothing cluster to either another shared-nothing cluster or a shared-data cluster.
- Supports creating synchronous materialized views with the WHERE clause specified.
- Added metrics that show memory usage of the data cache to MemTracker. [39600](https://github.com/StarRocks/starrocks/pull/39600)
- Added an array function, array_unique_agg.

Parameter Change
- Added a BE configuration item, lake_pk_compaction_max_input_rowsets, which controls the maximum number of input rowsets allowed in a Primary Key table compaction task in a shared-data StarRocks cluster. This helps optimize resource consumption for compaction tasks. [39611](https://github.com/StarRocks/starrocks/pull/39611)

Improvements
- Supports ORDER BY and INDEX clauses in CTAS statements. [38886](https://github.com/StarRocks/starrocks/pull/38886)
- Supports equality deletes on ORC-formatted Iceberg v2 tables. [37419](https://github.com/StarRocks/starrocks/pull/37419)
- Supports setting the datacache.partition_duration property for cloud-native tables created with the list partitioning strategy. This property controls the validity period of the data cache and can be dynamically configured. [35681](https://github.com/StarRocks/starrocks/pull/35681) [#38509](https://github.com/StarRocks/starrocks/pull/38509)
- Optimized the BE configuration item update_compaction_per_tablet_min_interval_seconds. This parameter is originally used only to control the frequency of compaction tasks on Primary Key tables. After the optimization, it can also be used to control the frequency of major compaction tasks on Primary Key table indexes. [39640](https://github.com/StarRocks/starrocks/pull/39640)
- Parquet Reader supports converting INT32-type data in Parquet-formatted data to DATETIME-type data and storing the resulting data to StarRocks. [39808](https://github.com/StarRocks/starrocks/pull/39808)

Bug Fixes
Fixed the following issues:
- Using NaN (Not a Number) columns as ORDER BY columns may cause BEs to crash. [30759](https://github.com/StarRocks/starrocks/pull/30759)
- Failure to update primary key indexes may cause the error "get_applied_rowsets failed". [27488](https://github.com/StarRocks/starrocks/pull/27488)
- The resources occupied by compaction_state_cache are not recycled after compaction task failures. [38499](https://github.com/StarRocks/starrocks/pull/38499)
- If partition columns in external tables contain null values, queries against those tables will cause BEs to crash. [38888](https://github.com/StarRocks/starrocks/pull/38888)
- After a table is dropped and then re-created with the same table name, refreshing asynchronous materialized views created on that table fails. [38008](https://github.com/StarRocks/starrocks/pull/38008)
- Refreshing asynchronous materialized views created on empty Iceberg tables fail. [24068](https://starrocks.atlassian.net/browse/SR-24068)

Release date: January 12, 2024

New Features

- Added a new function, `unnest_bitmap`. [38136](https://github.com/StarRocks/starrocks/pull/38136)
- Supports conditional updates for [Broker Load](https://docs.starrocks.io/docs/3.1/sql-reference/sql-statements/data-manipulation/BROKER_LOAD/#opt_properties). [37400](https://github.com/StarRocks/starrocks/pull/37400)

Behavior Change

- Added the session variable `enable_materialized_view_for_insert`, which controls whether materialized views rewrite the queries in INSERT INTO SELECT statements. The default value is `false`. [37505](https://github.com/StarRocks/starrocks/pull/37505)
- The FE dynamic parameter `enable_new_publish_mechanism` is changed to a static parameter. You must restart the FE after you modify the parameter settings. [35338](https://github.com/StarRocks/starrocks/pull/35338)
- Added the session variable `enable_strict_order_by`. When this variable is set to the default value `TRUE`, an error is reported for such a query pattern: Duplicate alias is used in different expressions of the query and this alias is also a sorting field in ORDER BY, for example, `select distinct t1.* from tbl1 t1 order by t1.k1;`. The logic is the same as that in v2.3 and earlier. When this variable is set to `FALSE`, a loose deduplication mechanism is used, which processes such queries as valid SQL queries. [37910](https://github.com/StarRocks/starrocks/pull/37910)

Parameter Change

- Added the FE configuration item `routine_load_unstable_threshold_second`. [36222](https://github.com/StarRocks/starrocks/pull/36222)
- Added the FE configuration item `http_worker_threads_num`, which specifies the number of threads for HTTP server to deal with HTTP requests. The default value is `0`. If the value for this parameter is set to a negative value or `0`, the actual thread number is twice the number of CPU cores. [37530](https://github.com/StarRocks/starrocks/pull/37530)
- Added the BE configuration item `pindex_major_compaction_limit_per_disk` to configure the maximum concurrency of compaction on a disk. This addresses the issue of uneven I/O across disks due to compaction. This issue can cause excessively high I/O for certain disks. The default value is `1`. [36681](https://github.com/StarRocks/starrocks/pull/36681)
- Added session variables `transaction_read_only` and `tx_read_only` to specify the transaction access mode, which are compatible with MySQL versions 5.7.20 and above. [37249](https://github.com/StarRocks/starrocks/pull/37249)
- Added the FE configuration item `default_mv_refresh_immediate`, which specifies whether to immediately refresh the materialized view after the materialized view is created. The default value is `true`. [37093](https://github.com/StarRocks/starrocks/pull/37093)
- Added a new BE configuration item `lake_enable_vertical_compaction_fill_data_cache`, which specifies whether to allow compaction tasks to cache data on local disks in a shared-data cluster. The default value is `false`. [37296](https://github.com/StarRocks/starrocks/pull/37296)

Improvements

- INSERT INTO FILE() SELECT FROM supports reading BINARY-type data from tables and exporting the data to Parquet-formatted files in remote storage. [36797](https://github.com/StarRocks/starrocks/pull/36797)
- Asynchronous materialized views support dynamically setting the `datacache.partition_duration` property, which controls the validity period of the hot data in the data cache. [35681](https://github.com/StarRocks/starrocks/pull/35681)
- Wen using JDK, the default GC algorithm is G1. [37386](https://github.com/StarRocks/starrocks/pull/37386)
- The `date_trunc`, `adddate`, and `time_slice` functions support setting the `interval` parameter to values that are accurate to the millisecond and microsecond. [36386](https://github.com/StarRocks/starrocks/pull/36386)
- When the string on the right side of the LIKE operator within the WHERE clause does not include `%` or `_`, the LIKE operator is converted into the `=` operator. [37515](https://github.com/StarRocks/starrocks/pull/37515)
- A new field `LatestSourcePosition` is added to the return result of [SHOW ROUTINE LOAD](https://docs.starrocks.io/zh/docs/3.1/sql-reference/sql-statements/data-manipulation/SHOW_ROUTINE_LOAD/) to record the position of the latest message in each partition of the Kafka topic, helping check the latencies of data loading. [#38298](https://github.com/StarRocks/starrocks/pull/38298)
- Added a new resource group property, `spill_mem_limit_threshold`, to control the memory usage threshold (percentage) at which a resource group triggers the spilling of intermediate results when the system variable `spill_mode` is set to `auto`. The valid range is (0, 1). The default value is `1`, indicating the threshold does not take effect. [37707](https://github.com/StarRocks/starrocks/pull/37707)
- The result returned by the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/3.1/sql-reference/sql-statements/data-manipulation/SHOW_ROUTINE_LOAD/) statement now includes the timestamps of consumption messages from each partition. [#36222](https://github.com/StarRocks/starrocks/pull/36222)
- The scheduling policy for Routine Load is optimized, so that slow tasks do not block the execution of the other normal tasks. [37638](https://github.com/StarRocks/starrocks/pull/37638)

Bug Fixes

Fixed the following issues:

- The execution of [ANALYZE TABLE](https://docs.starrocks.io/docs/3.1/sql-reference/sql-statements/data-definition/ANALYZE_TABLE/) gets stuck occasionally. [#36836](https://github.com/StarRocks/starrocks/pull/36836)
- The memory consumption by PageCache exceeds the threshold specified by the BE dynamic parameter `storage_page_cache_limit` in certain circumstances. [37740](https://github.com/StarRocks/starrocks/pull/37740)
- Hive metadata in [Hive catalogs](https://docs.starrocks.io/docs/3.1/data_source/catalog/hive_catalog/) is not automatically refreshed when new fields are added to Hive tables. [#37668](https://github.com/StarRocks/starrocks/pull/37668)
- In some cases, `bitmap_to_string` may return incorrect results due to data type overflow. [37405](https://github.com/StarRocks/starrocks/pull/37405)
- Executing the DELETE statement on an empty table returns "ERROR 1064 (HY000): Index: 0, Size: 0". [37461](https://github.com/StarRocks/starrocks/pull/37461)
- When the FE dynamic parameter `enable_sync_publish` is set to `TRUE`, queries on data that is written after the BEs crash and then restart may fail. [37398](https://github.com/StarRocks/starrocks/pull/37398)
- The value of the `TABLE_CATALOG` field in `views` of the StarRocks Information Schema is `null`. [37570](https://github.com/StarRocks/starrocks/pull/37570)
- When `SELECT ... FROM ... INTO OUTFILE` is executed to export data into CSV files, the error "Unmatched number of columns" is reported if the FROM clause contains multiple constants. [38045](https://github.com/StarRocks/starrocks/pull/38045)