Starrocks

Release date: March 14, 2022

BugFix

The following bugs are fixed:

- Query fails when BE nodes are in suspended animation.
- Query fails when there is no appropriate execution plan for single-tablet table joins. [3854](https://github.com/StarRocks/starrocks/issues/3854)
- A deadlock problem may occur when an FE node collects information to build a global dictionary for low-cardinality optimization. [3839](https://github.com/StarRocks/starrocks/issues/3839)

**Improvement**
- Memory usage is optimized. Users can specify the label_keep_max_num parameter to control the maximum number of loading jobs to retain within a period of time. This prevents full GC caused by high memory usage of FE during frequent data loading. [2410](https://github.com/StarRocks/starrocks/issues/2410)

**BugFix**
The following bugs are fixed:
- BE nodes fail when the column decoder encounters an exception. [3510](https://github.com/StarRocks/starrocks/issues/3510)
- Auto __op mapping does not take effect when jsonpaths is specified in the command used for loading JSON data. [3405](https://github.com/StarRocks/starrocks/issues/3405)
- BE nodes fail because the source data changes during data loading using Broker Load. [3481](https://github.com/StarRocks/starrocks/issues/3481)
- Some SQL statements report errors after materialized views are created. [3053](https://github.com/StarRocks/starrocks/issues/3053)
- Query may fail if an SQL clause contains a predicate that supports global dictionary for low-cardinality optimization and a predicate that does not. [3421](https://github.com/StarRocks/starrocks/issues/3421)

Release date: Jan 21, 2022

Improvement
* Add implicit_cast in hive external table (2829)
* Use read/write lock avoid cpu cost too much when collect metrics(2901)
* optimize some statistics.

Bugfix
* Fix Global dictionary when replica is inconsistent. (2700) (2765)
* Add exec_mem_limit for stream load (2693)
* Fix OOM when loading into primary key model. (2743)(2777)
* Fix BE hang when query external mysql table (2881)

2.0.0-GA
Release date: Jan 4, 2022

New Feature
- External Table
- [Experimental Function]Support for Hive external table on S3
- DecimalV3 support for external table 425
- Implement complex expressions to be pushed down to the storage layer for computation, thus gaining performance gains
- Primary Key is officially released, which supports Stream Load, Broker Load, Routine Load, and also provides a second-level synchronization tool for MySQL data based on Flink-cdc

Improvement
- Arithmetic operators optimization
- Optimize the performance of dictionary with low cardinality 791
- Optimize the scan performance of int for single table [273](https://github.com/StarRocks/starrocks/issues/273)
- Optimize the performance of count(distinct int) with high cardinality [139](https://github.com/StarRocks/starrocks/pull/139) [#250](https://github.com/StarRocks/starrocks/pull/250) [#544](https://github.com/StarRocks/starrocks/pull/544)[#570](https://github.com/StarRocks/starrocks/pull/570)
- Execution level optimization and refinement Group by 2 int / limit / case when / not equal
- Optimize Group by 2 int / limit / case when / not equal in implementation-level
- Memory management optimization
- Refactor the memory statistics and control framework to accurately count memory usage and completely solve OOM
- Optimize metadata memory usage
- Solve the problem of large memory release stuck in execution threads for a long time
- Add process graceful exit mechanism and support memory leak check 1093

Bugfix
- Fix the problem that the Hive external table is timeout to get metadata in a large amount.
- Fix the problem of unclear error message of materialized view creation.
- Fix the implementation of like in vectorization engine 722
- Repair the error of parsing the predicate is in alter table 725
- Fix the problem that the curdate function can not format the date.

Improvement

* Improve shuffle bucket plan [(](https://github.com/StarRocks/starrocks/commit/ff7993c0cf73547ee405363e47e330a5fcb5c975)[#2184](https://github.com/StarRocks/starrocks/pull/2184)[)](https://github.com/StarRocks/starrocks/commit/ff7993c0cf73547ee405363e47e330a5fcb5c975)
* Add a num_segment threshold when loading multiple big files [(](https://github.com/StarRocks/starrocks/commit/3a2b3b66bd3bef7cd7494e3ff341fe054ae1e613)[#2067](https://github.com/StarRocks/starrocks/pull/2067)[)](https://github.com/StarRocks/starrocks/commit/3a2b3b66bd3bef7cd7494e3ff341fe054ae1e613)

Bugfix
* **Upgrade Log4j2 to 2.17.0** [(](https://github.com/StarRocks/starrocks/commit/a3567692801727189d649d38ccd42fc1bd1394ca)[#2284](https://github.com/StarRocks/starrocks/pull/2284)[) (](https://github.com/StarRocks/starrocks/commit/a3567692801727189d649d38ccd42fc1bd1394ca)[#2290](https://github.com/StarRocks/starrocks/pull/2290)[)](https://github.com/StarRocks/starrocks/commit/a3567692801727189d649d38ccd42fc1bd1394ca)
* Fix query hive empty partition Error [707](https://github.com/StarRocks/starrocks/pull/707) [(](https://github.com/StarRocks/starrocks/commit/55997a0a2357d45b020dc45e0a0799f4f6a55c66)[#2082](https://github.com/StarRocks/starrocks/pull/2082)[)](https://github.com/StarRocks/starrocks/commit/55997a0a2357d45b020dc45e0a0799f4f6a55c66)

Imporvement
* support cast(varchar as bitmap) (1941)
* Modify Hive external table scan scheduling strategy (1394) (1807)

Bugfix
* Fix cross join lose predicate in JoinAssociativityRule (1918)
* Fix cast to decimal(0,0) bug (1709) (1738)
* Fix replicate join in plan fragment builder (1727)
* Fix several planner cost calculation

Improvement
Upgrade jprotobuf to enhance security (1506)

Major Bugfix
Fix some CBO bad cases and correctness issues.
Fix grouping sets with same column bug(1395) (1119)
Fix some date function issues (1385) (1627)
Fix streaming aggregation issue（1584）