Starrocks

Release date: September 9, 2024

Improvements

- Supports masking authentication information for Files() and PIPE. [47629](https://github.com/StarRocks/starrocks/pull/47629)
- Support automatic inference for the STRUCT type when reading Parquet files through Files(). [50481](https://github.com/StarRocks/starrocks/pull/50481)

Bug Fixes

Fixed the following issues:

- An error is returned for equi-join queries because they failed to be rewritten by the global dictionary. [50690](https://github.com/StarRocks/starrocks/pull/50690)
- The error "version has been compacted" caused by an infinite loop on the FE side during Tablet Clone. [50561](https://github.com/StarRocks/starrocks/pull/50561)
- Incorrect scheduling for unhealthy replica repairs after distributing data based on labels. [50331](https://github.com/StarRocks/starrocks/pull/50331)
- An error in the statistics collection log: "Unknown column '%s' in '%s." [50785](https://github.com/StarRocks/starrocks/pull/50785)
- Incorrect timezone usage when reading complex types like TIMESTAMP from Parquet files via Files(). [50448](https://github.com/StarRocks/starrocks/pull/50448)

Behavior Changes

- When downgrading StarRocks from v3.3.x to v3.2.11, the system will ignore it if there is incompatible metadata. [49636](https://github.com/StarRocks/starrocks/pull/49636)

Release date: August 23, 2024
Improvements
- Files() will automatically convert `BYTE_ARRAY` data with a `logical_type` of `JSON` in Parquet files to the JSON type in StarRocks. [49385](https://github.com/StarRocks/starrocks/pull/49385)
- Optimized error messages for Files() when Access Key ID and Secret Access Key are missing. [49090](https://github.com/StarRocks/starrocks/pull/49090)
- `information_schema.columns` supports the `GENERATION_EXPRESSION` field. [49734](https://github.com/StarRocks/starrocks/pull/49734)

Bug Fixes

Fixed the following issues:

- Downgrading a v3.3 shared-data cluster to v3.2 after setting the Primary Key table property `"persistent_index_type" = "CLOUD_NATIVE"` causes a crash. [48149](https://github.com/StarRocks/starrocks/pull/48149)
- Exporting data to CSV files using SELECT INTO OUTFILE may cause data inconsistency. [48052](https://github.com/StarRocks/starrocks/pull/48052)
- Queries encounter failures during concurrent query execution. [48180](https://github.com/StarRocks/starrocks/pull/48180)
- Queries would hang due to a timeout in the Plan phase without exiting. [48405](https://github.com/StarRocks/starrocks/pull/48405)
- After disabling index compression for Primary Key tables in older versions and then upgrading to v3.2.9, accessing `page_off` information causes an array out-of-bounds crash. [48230](https://github.com/StarRocks/starrocks/pull/48230)
- BE crash caused by concurrent execution of ADD/DROP COLUMN operations. [49355](https://github.com/StarRocks/starrocks/pull/49355)
- Queries against negative `TINYINT` values in ORC format files return `None` on the aarch64 architecture. [49517](https://github.com/StarRocks/starrocks/pull/49517)
- If the disk write operation fails, failures of `l0` snapshots for Primary Key Persistent Index may cause data loss. [48045](https://github.com/StarRocks/starrocks/pull/48045)
- Partial Update in Column mode for Primary Key tables fails under scenarios with large-volume data updates. [49054](https://github.com/StarRocks/starrocks/pull/49054)
- BE crash caused by Fast Schema Evolution when downgrading a v3.3.0 shared-data cluster to v3.2.9. [42737](https://github.com/StarRocks/starrocks/pull/42737)
- `partition_linve_nubmer` does not take effect. [49213](https://github.com/StarRocks/starrocks/pull/49213)
- The conflict between index persistence and compaction in Primary Key tables could cause clone failures. [49341](https://github.com/StarRocks/starrocks/pull/49341)
- Modifications of `partition_line_number` using ALTER TABLE do not take effect. [49437](https://github.com/StarRocks/starrocks/pull/49437)
- Rewrite of CTE distinct grouping sets generates an invalid plan. [48765](https://github.com/StarRocks/starrocks/pull/48765)
- RPC failures polluted the thread pool. [49619](https://github.com/StarRocks/starrocks/pull/49619)
- authentication failure issues when loading files from AWS S3 via PIPE. [49837](https://github.com/StarRocks/starrocks/pull/49837)

Behavior Changes

- Added a check for the `meta` directory in the FE startup script. If the directory does not exist, it will be automatically created. [48940](https://github.com/StarRocks/starrocks/pull/48940)
- Added a memory limit parameter `load_process_max_memory_hard_limit_ratio` for data loading. If memory usage exceeds the limit, subsequent loading tasks will fail. [48495](https://github.com/StarRocks/starrocks/pull/48495)

New Features
- Paimon tables now support DELETE Vectors. [45866](https://github.com/StarRocks/starrocks/issues/45866)
- Supports Column-level access control through Apache Ranger. [47702](https://github.com/StarRocks/starrocks/pull/47702)
- Stream Load can automatically convert JSON strings into STRUCT/MAP/ARRAY types during loading. [45406](https://github.com/StarRocks/starrocks/pull/45406)
- JDBC Catalog now supports Oracle and SQL Server. [35691](https://github.com/StarRocks/starrocks/issues/35691)
Improvements
- Improved privilege management by restricting user_admin role users from resetting the password of the root user. [47801](https://github.com/StarRocks/starrocks/pull/47801)
- Stream Load now supports using \t and \n as row and column delimiters. Users do not need to convert them to their hexadecimal ASCII codes. [47302](https://github.com/StarRocks/starrocks/pull/47302)
- Optimized memory usage during data loading. [47047](https://github.com/StarRocks/starrocks/pull/47047)
- Supports masking authentication information for the Files() function in audit logs. [46893](https://github.com/StarRocks/starrocks/pull/46893)
- Hive tables now support the skip.header.line.count property. [47001](https://github.com/StarRocks/starrocks/pull/47001)
- JDBC Catalog supports more data types. [47618](https://github.com/StarRocks/starrocks/pull/47618)
Bug Fixes
Fixed the following issues:

- BE crash caused by ALTER TABLE ADD COLUMN after upgrading a shared-data cluster from v3.2.x to v3.3.0 and then rolling it back. [47826](https://github.com/StarRocks/starrocks/pull/47826)
- Tasks initiated through SUBMIT TASK showed a Running status indefinitely in the QueryDetail interface. [47619](https://github.com/StarRocks/starrocks/pull/47619)
- Forwarding queries to the FE Leader node caused a null pointer exception. [47559](https://github.com/StarRocks/starrocks/pull/47559)
- SHOW MATERIALIZED VIEWS with WHERE conditions caused a null pointer exception. [47811](https://github.com/StarRocks/starrocks/pull/47811)
- Vertical Compaction fails for Primary Key tables in shared-data clusters. [47192](https://github.com/StarRocks/starrocks/pull/47192)
- Improper handling of I/O Error when sinking data to Hive or Iceberg tables. [46979](https://github.com/StarRocks/starrocks/pull/46979)
- Table properties do not take effect when whitespaces are added to their values. [47119](https://github.com/StarRocks/starrocks/pull/47119)
- BE crash caused by concurrent migration and Index Compaction operations on Primary Key tables. [46675](https://github.com/StarRocks/starrocks/pull/46675)

Release date: June 7, 2024
New Features
- [Supports adding labels on BEs](https://docs.starrocks.io/docs/3.3/administration/management/resource_management/be_label/): Supports adding labels on BEs based on information such as the racks and data centers where BEs are located. It ensures even data distribution among racks and data centers, and facilitates disaster recovery in case of power failures in certain racks or faults in data centers. [#38833](https://github.com/StarRocks/starrocks/pull/38833)
Bug Fixes
Fixed the following issues:
- An error is returned when users DELETE data rows from tables that use the expression partitioning method with str2date. [45939](https://github.com/StarRocks/starrocks/pull/45939)
- BEs in the destination cluster crash when the StarRocks Cross-cluster Data Migration Tool fails to retrieve the Schema information from the source cluster. [46068](https://github.com/StarRocks/starrocks/pull/46068)
- The error Multiple entries with same key is returned to queries with non-deterministic functions. [46602](https://github.com/StarRocks/starrocks/pull/46602)

Release date: May 25, 2024
New Features
- Stream Load supports data compression during transmission, reducing network bandwidth overhead. Users can specify different compression algorithms using parameters compression and Content-Encoding. Supported compression algorithms including GZIP, BZIP2, LZ4_FRAME, DEFLATE, and ZSTD. [43732](https://github.com/StarRocks/starrocks/pull/43732)
- Optimized the garbage collection (GC) mechanism in shared-data clusters. Supports manual compaction for tables or partitions stored in object storage. [39532](https://github.com/StarRocks/starrocks/issues/39532)
- Flink connector supports reading complex data types ARRAY, MAP, and STRUCT from StarRocks. [42932](https://github.com/StarRocks/starrocks/pull/42932) [#347](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/347)
- Supports populating Data Cache asynchronously during queries, reducing the impact of populating cache on query performance. [40489](https://github.com/StarRocks/starrocks/pull/40489)
- ANALYZE TABLE supports collecting histograms for external tables, effectively addressing data skews. For more information, see [CBO statistics](https://docs.starrocks.io/docs/using_starrocks/Cost_based_optimizer/#collect-statistics-of-hiveiceberghudi-tables). [42693](https://github.com/StarRocks/starrocks/pull/42693)
- Lateral Join with [UNNEST](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/unnest/) supports LEFT JOIN. [#43973](https://github.com/StarRocks/starrocks/pull/43973)
- Query Pool supports configuring memory usage threshold that triggers spilling via BE static parameter `query_pool_spill_mem_limit_threshold`. Once the threshold is reached, intermediate results of queries will be spilled to disks to reduce memory usage, thus avoiding OOM.[44063](https://github.com/StarRocks/starrocks/pull/44063)
- Supports creating asynchronous materialized views based on Hive views.[45085](https://github.com/StarRocks/starrocks/pull/45085)
Improvements
- Optimized the error message returned for Broker Load tasks when there is no data under the specified HDFS paths. [43839](https://github.com/StarRocks/starrocks/pull/43839)
- Optimized the error message returned when the Files function is used to read data from AWS S3 without Access Key and Secret Key specified. [42450](https://github.com/StarRocks/starrocks/pull/42450)
- Optimized the error message returned for Broker Load tasks that load no data to any partitions. [44292](https://github.com/StarRocks/starrocks/pull/44292)
- Optimized the error message returned for INSERT INTO SELECT tasks when the column count of the destination table does not match that in the SELECT statement. [44331](https://github.com/StarRocks/starrocks/pull/44331)
Bug Fixes
Fixed the following issues:
- Concurrent read or write of the BITMAP-type data may cause BE to crash. [44167](https://github.com/StarRocks/starrocks/pull/44167)
- Primary key indexes may cause BE to crash. [43793](https://github.com/StarRocks/starrocks/pull/43793) [#43569](https://github.com/StarRocks/starrocks/pull/43569) [#44034](https://github.com/StarRocks/starrocks/pull/44034)
- Under high query concurrency scenarios, the str_to_map function may cause BE to crash. [43901](https://github.com/StarRocks/starrocks/pull/43901)
- When the Masking policy of Apache Ranger is used, an error is returned when table aliases are specified in queries. [44445](https://github.com/StarRocks/starrocks/pull/44445)
- In shared-data clusters, query execution cannot be routed to a backup node when the current node encounters exceptions. The corresponding error message is optimized for this issue. [43489](https://github.com/StarRocks/starrocks/pull/43489)
- Memory information is incorrect in the container environment. [43225](https://github.com/StarRocks/starrocks/issues/43225)
- An exception is thrown when INSERT tasks are canceled. [44239](https://github.com/StarRocks/starrocks/pull/44239)
- Expression-based dynamic partitions cannot be automatically created. [44163](https://github.com/StarRocks/starrocks/pull/44163)
- Creating partitions may cause FE deadlock. [44974](https://github.com/StarRocks/starrocks/pull/44974)

Release date: April 18, 2024

Bug Fixes
Fixed the following issue:

- The privileges of external tables cannot be found due to incompatibility issues. [44030](https://github.com/StarRocks/starrocks/pull/44030)