Apache-iotdb

Features & Improvements

- Storage Engine: Added new data types String, Blob, Date, and Timestamp.
- Storage Engine: Multi-level storage has added a rate-limiting mechanism.
- Storage Engine: New merge target file splitting feature, with additional configuration file parameters, and improved memory control performance of the merge module.
- Data Query: Filter performance optimization, enhancing the speed of aggregate queries and where condition queries.
- Data Query: New client query requests load balancing optimization.
- Data Query: New active metadata statistics query added.
- Data Query: Optimized memory control strategy during the query planning phase.
- Data Synchronization: The sender supports transferring files to a specified directory, and the receiver automatically loads them into IoTDB.
- Data Synchronization: The receiver has a new automatic conversion mechanism for data type requests.
- Data Synchronization: Enhanced observability on the receiver side, supporting ops/latency statistics for multiple internal interfaces, consolidated into a single pipeTransfer display.
- Data Loading: DataNode actively listens and loads TsFiles, with additional observability metrics.
- Stream Processing Module: New data subscription capability, supporting subscription to database data in the form of data points or tsfile files.
- Stream Processing Module: Alter Pipe supports the ability to alter the source.
- System Management: Optimized configuration files, with the original three configuration files merged into one, reducing user operational costs.
- System Management: Optimized restart recovery performance, reducing startup time.
- System Management: Internal addition of monitoring items such as device count, estimated remaining time for data synchronization, size of data to be synchronized, and synchronization speed.
- Scripts and Tools: The import-tsfile script is expanded to support running the script on a different server from the IoTDB server.
- Scripts and Tools: New metadata import and export scripts added.
- Scripts and Tools: New support for Kubernetes Helm added.
- AINode: AINode module added.
...

Bugs

- Fixed the issue of NullPointerException (NPE) when merging chunks with modifications and empty pages in the sequential space.
- Fixed the issue where the wrong parent file was used when reassigning the file position for skipped files during merge, leading to failure in creating hard links.
- Fixed the issue where the newly added four data types had null values written, and the TsFile handling of the STRING type was incorrect, causing a BufferUnderflowException: null.
- Fixed the issue in the high availability scenario where stopping the DataNode resulted in a PipeException: Failed to start consensus pipe.
- Fixed the issue in Stream mode where the first batch of written data points might require a flush to be synchronized.
- Fixed the compatibility issue with pipe plugin upgrades.
- Fixed the issue where the `ORDER BY` clause became ineffective when used in combination with `LIMIT` in the last query.
...

New Contributors
* alxkhi made their first contribution in https://github.com/apache/iotdb/pull/12296
* SihanLiu2024 made their first contribution in https://github.com/apache/iotdb/pull/12323
* supersshhhh made their first contribution in https://github.com/apache/iotdb/pull/12512
* bdemers made their first contribution in https://github.com/apache/iotdb/pull/12971

**Full Changelog**: https://github.com/apache/iotdb/compare/v1.3.2...v1.3.3

Features & Improvements
- Storage Module: Performance improvement in the insertRecords interface for writing
- Query Module: New Explain Analyze statement added (monitoring the time spent on each stage of a single SQL execution)
- Query Module: New UDAF (User-Defined Aggregate Function) framework added
- Query Module: New MaxBy/MinBy functions added, supporting the retrieval of maximum/minimum values along with the corresponding timestamps
- Query Module: Performance improvement in value filtering queries
- Data Synchronization: Path matching supports path pattern
- Data Synchronization: Supports metadata synchronization (including time series and related attributes, permissions, etc.)
- Stream Processing: Added Alter Pipe statement, supporting hot updates of plugins for Pipe tasks
- System Module: System data point count statistics now include statistics for data imported by loading TsFile
- Scripts and Tools: New local upgrade backup tool added (backing up original data through hard links)
- Scripts and Tools: New export-data/import-data scripts added, supporting data export in CSV, TsFile formats or SQL statements
- Scripts and Tools: Windows environment now supports distinguishing ConfigNode, DataNode, and Cli by window name
...
Bugs
- Optimize the error message when a NullPointerException (NPE) occurs due to a timeout when dropping a database.
- Add logs for notifyLeaderReady, notifyLeaderChanged, and procedure worker.
- Add compatibility handling for existing erroneous data during file merging.
- Fix the deadlock issue caused by flushing empty files during querying.
- Fix the issue where Ratis becomes unresponsive during read, write, and delete operations.
- Fix the concurrent bug in load and merge operations.
- Fix the issue where the system's compression ratio is recorded as a negative number in the file for certain scenarios.
- Fix the ConcurrentModificationException issue during memory estimation for merge tasks.
- Fix potential deadlocks that may occur when writing, automatically creating, and deleting databases concurrently.
...

What's Changed
* Delete write mem control parameters by l2280212 in https://github.com/apache/iotdb/pull/12007
* Pipe: Ensure at-least-once semantic by forcibly starting PipeHistoricalDataRegionTsFileExtractor after pipe restart by Caideyipi in https://github.com/apache/iotdb/pull/12010
* Separate the synchronized block of AbstractCompactionEstimator file info cache by shuwenwei in https://github.com/apache/iotdb/pull/12019
* Pipe: using clusterId to judge whether the target cluster is source cluster by xuanronaldo in https://github.com/apache/iotdb/pull/11994
* Storage Engine: fix wal delete thread exception because multi thread safety question by caozj1011 in https://github.com/apache/iotdb/pull/12020
* PBTree: Fix Dead Lock and Refactor write/update inteface by bigreybear in https://github.com/apache/iotdb/pull/11985
* Remove sql `merge` by shuwenwei in https://github.com/apache/iotdb/pull/12021
* Load: Trigger last flush time map & tsfile resource degrading after successful load by SteveYurongSu in https://github.com/apache/iotdb/pull/12024
* feat(jdbc):getDouble add type check by CritasWang in https://github.com/apache/iotdb/pull/12023
* Remove infinite retry logic for replica inconsistency to avoid potential problems by OneSizeFitsQuorum in https://github.com/apache/iotdb/pull/12028
* Pipe: Support "sink.host" and "connector.host" key in receiver IP specification by Caideyipi in https://github.com/apache/iotdb/pull/12017
* Fixed Ratis query not retrying when DataNode restarts by OneSizeFitsQuorum in https://github.com/apache/iotdb/pull/12029
* Pipe: support replace and modify mode for alter pipe sql by VGalaxies in https://github.com/apache/iotdb/pull/12018
* Storage Engine: put DeviceCache into the total memory of the storage engine by caozj1011 in https://github.com/apache/iotdb/pull/12016
* Bump jackson.version from 2.15.2 to 2.15.3 by dependabot in https://github.com/apache/iotdb/pull/12014
* IT framework supports restarting cluster by l2280212 in https://github.com/apache/iotdb/pull/12022
* Fix typo for parameter dn_multi_dir_strategy by OneSizeFitsQuorum in https://github.com/apache/iotdb/pull/12033
* Upgrade github action version to fix the warning by HTHou in https://github.com/apache/iotdb/pull/12032
* Introduce UDAF(User-Defined Aggregation Functions) by Sh-Zh-7 in https://github.com/apache/iotdb/pull/12005
* Pipe: Move parsing event logic into PipeProcessorSubtask to separate CPU intensive and IO intensive tasks by DanielWang2035 in https://github.com/apache/iotdb/pull/11965
* DataNodeDevicePathCache should use free memory of memtable by HTHou in https://github.com/apache/iotdb/pull/12034
* Pipe: Revert "Pipe: stop pipe using restarting strategy to unpin the wal's reference count to avoid WAL stacking (11971)" to avoid unnecessary pipe drop during subtask exception handling by Caideyipi in https://github.com/apache/iotdb/pull/12031
* Pipe: Removed mayLinkedTsFileCountReachDangerousThreshold judgement from stuck restart logic by Caideyipi in https://github.com/apache/iotdb/pull/12041
* Pipe: fixed the bug that processor dies when encountered memory shortage for tablets in a period of time by Caideyipi in https://github.com/apache/iotdb/pull/12042
* Pipe: Reduced pipe logs and controlled how often the log is printed below a certain frequency by Caideyipi in https://github.com/apache/iotdb/pull/11973
* [IOTDB-6296] Fix memory leak in MQTTService by JackieTien97 in https://github.com/apache/iotdb/pull/12046
* Pipe: Pattern parsing pruning: When pattern is at the level below database, the parsing logic can be skipped if tsfiles / tablets completely match with the pattern by Caideyipi in https://github.com/apache/iotdb/pull/12049
* Bump org.apache.commons:commons-compress from 1.21 to 1.26.0 by dependabot in https://github.com/apache/iotdb/pull/12058
* Bump ip from 2.0.0 to 2.0.1 in /iotdb-connector/grafana-plugin by dependabot in https://github.com/apache/iotdb/pull/12057
* Pipe: fixed potential lose point bug caused by cancelled flush of historical extractor by Caideyipi in https://github.com/apache/iotdb/pull/12056
* Fix it which use 'merge' command by shuwenwei in https://github.com/apache/iotdb/pull/12060
* Pipe: Fixed the bug that CN cannot sense drop pipe failure in meta sync and may lead to constantly skip of drop pipe by Caideyipi in https://github.com/apache/iotdb/pull/12059
* [IOTDB-6297] Optimize the distribute plan in the situation of `aggregation with align by device` by Beyyes in https://github.com/apache/iotdb/pull/12043
* [IOTDB-6298] Fix number overflow in group by time interval by JackieTien97 in https://github.com/apache/iotdb/pull/12065
* [IOTDB-6299] Fix bug in merging overlapped data process caused by filter & offset push down by liuminghui233 in https://github.com/apache/iotdb/pull/12068
* Introduce Aggregation Function MinBy by lancelly in https://github.com/apache/iotdb/pull/12071
* Add recover IT for Procedure, and delete ProcedureStore by liyuheng55555 in https://github.com/apache/iotdb/pull/12045
* PBTree: Implement dual-buffer container for MNode management by linxt20 in https://github.com/apache/iotdb/pull/12048
* Optimize flushing memtable check by HTHou in https://github.com/apache/iotdb/pull/12081
* [IOTDB-6302] Enhance the support of ISO_LOCAL_DATE_TIME timestamp format by HTHou in https://github.com/apache/iotdb/pull/12086
* IT: revert some changes in restart IT to make the error log clear when failed to restart by l2280212 in https://github.com/apache/iotdb/pull/12066
* change ci to github ci by CritasWang in https://github.com/apache/iotdb/pull/12073
* feat(bat): add title and set format by CritasWang in https://github.com/apache/iotdb/pull/12069
* Fix dead lock in PBTree.SchemaFile by bigreybear in https://github.com/apache/iotdb/pull/12079
* fix.root privileges lost and grant role. by ColinLeeo in https://github.com/apache/iotdb/pull/12091
* [IOTDB-6300] Support place time column at any column index in insert statement by JackieTien97 in https://github.com/apache/iotdb/pull/12074
* Make Drop database timeout return message more detailed by Cpaulyz in https://github.com/apache/iotdb/pull/12082
* Fix VolatileSubtreeIterator missing update buffer if all nodes in new buffer are not qualified by Cpaulyz in https://github.com/apache/iotdb/pull/12093
* Pbtree: MNode iterating with merge sort upon disk and buffer by linxt20 in https://github.com/apache/iotdb/pull/12077
* Add FileUtils.moveFileSafe function by liyuheng55555 in https://github.com/apache/iotdb/pull/12094
* Add logs for data partition allocation by CRZbulabula in https://github.com/apache/iotdb/pull/12090
* fix: IoTDBPreparedStatement implement addBatch function by CritasWang in https://github.com/apache/iotdb/pull/12095
* [IOTDB-6303] Correct the default path of user, role and procedure folder by CRZbulabula in https://github.com/apache/iotdb/pull/12089
* Cache clusterId in IoTDBConfig by liyuheng55555 in https://github.com/apache/iotdb/pull/12053
* Implement new plan optimizer: PredicatePushDown by liuminghui233 in https://github.com/apache/iotdb/pull/11859
* Optimize ConfigNode ConsensusManager init logic by OneSizeFitsQuorum in https://github.com/apache/iotdb/pull/12098
* Do not wait to retry when configLeader exists by Pengzna in https://github.com/apache/iotdb/pull/12075
* Faster repair data by shuwenwei in https://github.com/apache/iotdb/pull/12039
* Bump org.scala-lang:scala-library from 2.12.18 to 2.12.19 by dependabot in https://github.com/apache/iotdb/pull/12104
* Bump flink.version from 1.17.0 to 1.17.2 by dependabot in https://github.com/apache/iotdb/pull/12103
* [IOTDB-6305] Add show current_timestamp statement by yx-zhang in https://github.com/apache/iotdb/pull/12106
* [IOTDB-6306] Fix the issue that UDTF with boolean type is not supported in filter by lancelly in https://github.com/apache/iotdb/pull/12105
* [IOTDB-6301] Optimize insert first by CRZbulabula in https://github.com/apache/iotdb/pull/12080
* Add remove TsFile for DataRegion by HeimingZ in https://github.com/apache/iotdb/pull/12126
* Fix fail to read memTable ids from the wal file by HTHou in https://github.com/apache/iotdb/pull/12132
* Pipe Subscription: initialize the subscription client RPC payload by VGalaxies in https://github.com/apache/iotdb/pull/12121
* Fix some ignored ITs of udf lib by HTHou in https://github.com/apache/iotdb/pull/12135
* Fix compaction schedule task manager cannot stop and modify some output in log by shuwenwei in https://github.com/apache/iotdb/pull/12125
* [IOTDB-6304] Support EXPLAN ANALYZE in query. by ycycse in https://github.com/apache/iotdb/pull/12100
* Pipe: fixed the bug that released PipeRawTabletInsertionEvent's tablet is still referenced and causing OOM by Caideyipi in https://github.com/apache/iotdb/pull/12117
* flink-iotdb-connector: IoTDBSink throws NPE by xuanronaldo in https://github.com/apache/iotdb/pull/12133
* [IOTDB-6297] Optimize the distribute plan in aggregation align by device when some device cross data regions by Beyyes in https://github.com/apache/iotdb/pull/12102
* Set ConfigNode list early during DataNode registration by liyuheng55555 in https://github.com/apache/iotdb/pull/12137
* Revert "change ci to github ci" by HTHou in https://github.com/apache/iotdb/pull/12140
* Ignore log: memtableId -9223372036854775808 not found in MemTableId2Info by SteveYurongSu in https://github.com/apache/iotdb/pull/12141
* [IOTDB-6309] Lager interval for auto ratis leader balance by CRZbulabula in https://github.com/apache/iotdb/pull/12116
* Fixed ConfigNode startup error in SimpleConsensus by OneSizeFitsQuorum in https://github.com/apache/iotdb/pull/12143
* Pipe meta sync: support meta data sync between different clusters by SteveYurongSu in https://github.com/apache/iotdb/pull/11794
* Remove dependency on org.openjdk.jol.jol-core and io.airlift.slice by JackieTien97 in https://github.com/apache/iotdb/pull/12111
* fix bug : the change logic error of the CachedMNodeContainerIterator by linxt20 in https://github.com/apache/iotdb/pull/12147
* Pipe: fix `PipeRealtimeDataRegionExtractor` cannot fetch time partition bound by VGalaxies in https://github.com/apache/iotdb/pull/12118
* Fix bug: parent procedure should not execute before sub-procedure finished by liyuheng55555 in https://github.com/apache/iotdb/pull/12134
* Pipe: fix legacy receiver's unsafe execution race problem (fix IoTDBPipeDataSinkIT.testLegacyConnector) by SteveYurongSu in https://github.com/apache/iotdb/pull/12152
* Pipe IT: add parameter "connector.batch.enable" = "false" for IoTDBPipeNullValueIT to temp fix CI by SteveYurongSu in https://github.com/apache/iotdb/pull/12153
* Fix StorageEngine recover slowly by shuwenwei in https://github.com/apache/iotdb/pull/12151
* Pipe: Fixed potential lose point bug when an unclosed tsfile reaches `isEventTimeOverlappedWithTimeRange()` judgement (CI fails on testInsertNullValueTemplate) by Caideyipi in https://github.com/apache/iotdb/pull/12156
* [IOTDB-6301] Optimize insert first by BUAAserein in https://github.com/apache/iotdb/pull/12136
* Pipe: support IoTDB-style pattern by DanielWang2035 in https://github.com/apache/iotdb/pull/12085
* Pipe Meta: Completed the logical view related logics & some bug fixes in IT and dirs by Caideyipi in https://github.com/apache/iotdb/pull/12159
* UDF: add "timestampPrecision" on UDFParameters by caozj1011 in https://github.com/apache/iotdb/pull/12158
* Fix cpp client IT on windows by HTHou in https://github.com/apache/iotdb/pull/12162
* Pipe: Refactored IT to complement necessary flush to all the pipe ITs by Caideyipi in https://github.com/apache/iotdb/pull/12164
* [IOTDB-6310] Optimize for query resource init by JackieTien97 in https://github.com/apache/iotdb/pull/12160
* fix error log caused by ClosedByInterruptException in compaction module by shuwenwei in https://github.com/apache/iotdb/pull/12142
* Refactor IQueryExecution to support Table Model in the future by JackieTien97 in https://github.com/apache/iotdb/pull/12157
* fix(jdbc): when catalog is "Apache IoTDB", ignore it by CritasWang in https://github.com/apache/iotdb/pull/12166
* Print native request api in datanode_slow_sql.log by Beyyes in https://github.com/apache/iotdb/pull/12155
* Bump google.golang.org/protobuf from 1.31.0 to 1.33.0 in /iotdb-connector/grafana-plugin by dependabot in https://github.com/apache/iotdb/pull/12168
* Random schedule compaction of bound data regions in one compaction schedule worker by shuwenwei in https://github.com/apache/iotdb/pull/12167
* Fixed logical view creation with "into" pattern failure problem by Caideyipi in https://github.com/apache/iotdb/pull/12169
* Support schema region snapshot parser. by ColinLeeo in https://github.com/apache/iotdb/pull/12145
* Pipe: implemented sdt-sampling-processor & refactored down-sampling-processor by xuanronaldo in https://github.com/apache/iotdb/pull/12114
* UDF: add an udf function for envelope demodulation analysis by caozj1011 in https://github.com/apache/iotdb/pull/12146
* Pipe: Fixed create logical view idempotency problem & some IT check coverage missing by Caideyipi in https://github.com/apache/iotdb/pull/12170
* Integration test cluster stop faster by liyuheng55555 in https://github.com/apache/iotdb/pull/12110
* Fix Concurrent bug between NodeSupplier.close() and its run() by JackieTien97 in https://github.com/apache/iotdb/pull/12175
* Fix wal delete sort method by HeimingZ in https://github.com/apache/iotdb/pull/12176
* Fixed simple consensus recover problem by Caideyipi in https://github.com/apache/iotdb/pull/12173
* Update iotdb-doap.rdf to update the info in https://projects.apache.org/project.html?iotdb by HTHou in https://github.com/apache/iotdb/pull/12179
* Thrift zero-copy optimization for TElasticFramedTransport by MrQuansy in https://github.com/apache/iotdb/pull/12050
* Optimize Count/Delete Timeseries root.** or root.db.** by Cpaulyz in https://github.com/apache/iotdb/pull/12174
* Subscription: Subscription Management On ConfigNode by MiniSho in https://github.com/apache/iotdb/pull/12149
* Bump follow-redirects from 1.15.4 to 1.15.6 in /iotdb-connector/grafana-plugin by dependabot in https://github.com/apache/iotdb/pull/12182
* Bump org.apache.tomcat.embed:tomcat-embed-core from 9.0.83 to 9.0.86 by dependabot in https://github.com/apache/iotdb/pull/12181
* Bump com.nimbusds:nimbus-jose-jwt from 9.34 to 9.37.2 by dependabot in https://github.com/apache/iotdb/pull/12180
* Fix deadlock when flush an empty TsFile and query concurrently by HTHou in https://github.com/apache/iotdb/pull/12171
* Update master verison to 1.3.2-SNAPSHOT by HTHou in https://github.com/apache/iotdb/pull/12185
* Fixed the bug that TimeoutChangeableTFastFramedTransport is unused so that timeout changing is disabled in thrift transferring by Caideyipi in https://github.com/apache/iotdb/pull/12186
* [IOTDB-6312] Throw correct error msg while using wrong password by JackieTien97 in https://github.com/apache/iotdb/pull/12191
* Fix flush command failed when dataregion using ratis consensus by HTHou in https://github.com/apache/iotdb/pull/12194
* Add more query metrics for estimated memory and max overlapped page by Beyyes in https://github.com/apache/iotdb/pull/12172
* Change default threshold of slow query from 30s to 10s by Beyyes in https://github.com/apache/iotdb/pull/12187
* Fix NPE in SessionConnection while first node in nodeurls is unavailable by JackieTien97 in https://github.com/apache/iotdb/pull/12196
* Pipe: fix unable to handle timeseries already existed exception caused by CreateMultiTimeSeriesStatement & add throwing-exception-processor for auto recovery show cases by SteveYurongSu in https://github.com/apache/iotdb/pull/12198
* Append time column at the end of input columns in mappable UDTF by Sh-Zh-7 in https://github.com/apache/iotdb/pull/12199
* [IOTDB-6313] Fix uncorrect implementation in FILL(LINEAR) by JackieTien97 in https://github.com/apache/iotdb/pull/12200
* [IOTDB-6314] Treat ExplainAnalyze as a query by ycycse in https://github.com/apache/iotdb/pull/12204
* Add Query Metric for TsFileResourceManager by JackieTien97 in https://github.com/apache/iotdb/pull/12205
* Bump black from 22.3.0 to 24.3.0 in /iotdb-client/client-py by dependabot in https://github.com/apache/iotdb/pull/12207
* Bump org.springframework:spring-web from 5.3.30 to 5.3.33 by dependabot in https://github.com/apache/iotdb/pull/12084
* [IOTDB-6301] Optimize insert first for single replica by BUAAserein in https://github.com/apache/iotdb/pull/12209
* Replace string device id with interface IDeviceID by shuwenwei in https://github.com/apache/iotdb/pull/12177
* Pipe: avoid executing too many PipeMetaSyncProcedure after system reboot by SteveYurongSu in https://github.com/apache/iotdb/pull/12213
* Remove isClusterMode configuration by HTHou in https://github.com/apache/iotdb/pull/12193
* Bump org.apache.commons:commons-configuration2 from 2.9.0 to 2.10.1 by dependabot in https://github.com/apache/iotdb/pull/12214
* Refactor Procedure recover framework by liyuheng55555 in https://github.com/apache/iotdb/pull/12202
* Display 'UNKNOWN' in `SHOW FUNCTIONS` when CN accidentally drop DN's function. by Sh-Zh-7 in https://github.com/apache/iotdb/pull/12208
* Left rotate time column to the end of input columns in UDAF by Sh-Zh-7 in https://github.com/apache/iotdb/pull/12201
* Introducing new features to IoTDB JVM GC options by Pengzna in https://github.com/apache/iotdb/pull/12088
* Pipe: filtered empty tsFiles which should not be parsed or reported by Caideyipi in https://github.com/apache/iotdb/pull/12216
* support config snapshot parser by ColinLeeo in https://github.com/apache/iotdb/pull/12211
* Pipe Subscription: initialize the subscription agent on DN by VGalaxies in https://github.com/apache/iotdb/pull/12138
* Fix NPE problem in EXPLAIN ANALYZE by ycycse in https://github.com/apache/iotdb/pull/12218
* Allow series data type not consistent in compaction by shuwenwei in https://github.com/apache/iotdb/pull/12203
* Remove duplicate lookups in dictionary encoder by MrQuansy in https://github.com/apache/iotdb/pull/12224
* fix: consider more general type of nodes above TopK-Node as parent by ycycse in https://github.com/apache/iotdb/pull/12225
* Pipe: Introduced aggregate processor as the universal aggregation framework by Caideyipi in https://github.com/apache/iotdb/pull/12210
* Fix procedure concurrency problem caused by non-atomic procedure Id generation by liyuheng55555 in https://github.com/apache/iotdb/pull/12229
* fix cn parser and code style. by ColinLeeo in https://github.com/apache/iotdb/pull/12231
* UDF: Fix envelope function execution error without frequency parameter by caozj1011 in https://github.com/apache/iotdb/pull/12233
* Log: add logs for notifyLeaderReady, notifyLeaderChanged and procedure workers by SteveYurongSu in https://github.com/apache/iotdb/pull/12235
* Use heartbeat to update DataNode's ConfigNodeList by liyuheng55555 in https://github.com/apache/iotdb/pull/12232
* [RatisConsensus] Remove JavaUtils dependencies by SzyWilliam in https://github.com/apache/iotdb/pull/12236
* Warn user if they use GraalVM by liyuheng55555 in https://github.com/apache/iotdb/pull/12238
* Region migration improvement by liyuheng55555 in https://github.com/apache/iotdb/pull/12165
* Remove -XX:-UseBiasedLock in IoTDB JVM options by Pengzna in https://github.com/apache/iotdb/pull/12239
* feat: Refactored the visibility of some methods to make it possible to add BasicAuthorizer implementations in other packages. by chrisdutz in https://github.com/apache/iotdb/pull/12195
* Optimize SeriesScanUtil by memorizing the order time and satisfied information for each Seq and Unseq Resource by JackieTien97 in https://github.com/apache/iotdb/pull/12227
* Subscription: fix some issues on CN by DanielWang2035 in https://github.com/apache/iotdb/pull/12217
* change cli classpath by CritasWang in https://github.com/apache/iotdb/pull/12237
* Fix C++ client build problem by liyuheng55555 in https://github.com/apache/iotdb/pull/12241
* [RatisConsensus] Bump ratis snapshot version to support force snapshot by SzyWilliam in https://github.com/apache/iotdb/pull/12242
* Optimize String value type infer logic by HTHou in https://github.com/apache/iotdb/pull/12223
* IoTConsensus: Skip retry sending batch caused by TApplicationException by BUAAserein in https://github.com/apache/iotdb/pull/12243
* Bump jackson.version from 2.15.3 to 2.15.4 by dependabot in https://github.com/apache/iotdb/pull/12190
* Pipe: fix deadlock caused by PipeResourceManagerHolder.<clinit> and PipePeriodicalJobExecutor by SteveYurongSu in https://github.com/apache/iotdb/pull/12245
* [IOTDB-6316] Fix potential concurrent bug between compaction and load by JackieTien97 in https://github.com/apache/iotdb/pull/12244
* Subscription: stop meta sync properly when no topic/consumer group by DanielWang2035 in https://github.com/apache/iotdb/pull/12247
* Subscription: initialize the subscription session and consumer & Pipe: fix data points loss caused by unclosed tsfile listened by historical extractor by VGalaxies in https://github.com/apache/iotdb/pull/12228
* Fix uncompress byte buffer in RepairDataFileScanUtil by shuwenwei in https://github.com/apache/iotdb/pull/12250
* Pipe: Support mod transfer in pipe tsFiles & schema region snapshot transferring logic by Caideyipi in https://github.com/apache/iotdb/pull/11998
* fix: consider more general type of nodes in optimization above LimitNode by ycycse in https://github.com/apache/iotdb/pull/12240
* Forcibly take snapshot when upgrade procedure WAL by liyuheng55555 in https://github.com/apache/iotdb/pull/12252
* Modify compaction task priority comparator by shuwenwei in https://github.com/apache/iotdb/pull/12255
* Add the import and export function of SQl files on the basis of CSV import and export scripts by 2b3c511 in https://github.com/apache/iotdb/pull/12248
* invert insertRecords to insertTablets by l2280212 in https://github.com/apache/iotdb/pull/12113
* Fix the mistaken argument in LZ4Uncompressor by jt2594838 in https://github.com/apache/iotdb/pull/12249
* Subscription: fix some issues on DN and session & improve IT by VGalaxies in https://github.com/apache/iotdb/pull/12254
* Pipe: Added handshake logic when config receiver is reset due to receiver side DataNode redirection & Enhance IT & Refactor by Caideyipi in https://github.com/apache/iotdb/pull/12256
* Region migration related work by liyuheng55555 in https://github.com/apache/iotdb/pull/12246
* [IOTDB-6307] GCR algorithm for multi-database by CRZbulabula in https://github.com/apache/iotdb/pull/12183
* Region migration related work by liyuheng55555 in https://github.com/apache/iotdb/pull/12259
* [IOTDB-6308] CFD algorithm for multi-database by CRZbulabula in https://github.com/apache/iotdb/pull/12184

**Full Changelog**: https://github.com/apache/iotdb/compare/v1.3.1...v1.3.2

Features & Improvements
- Add cluster script for one-click start-stop (start-all/stop-all.sh & start-all/stop-all.bat)
- Add script for one-click instance information collection (collect-info.sh & collect-info.bat)
- Add new statistical aggregators include stddev and variance
- Add repair tsfile data command
- Support setting timeout threshold for Fill clause. When time beyond the threshold, do not fill value.
- Simplify the time range specification for data synchronization, directly set start and end times
- Improved system observability (adding dispersion monitoring of cluster nodes, observability of distributed task scheduling framework)
- Optimized default log output strategy
- Enhance memory control for Load TsFile, covering the entire process
- Rest API (Version 2) adds column type return.
- Improve the process of query execution
- Session automatically fetch all available DataNodes
...
Bugs
- Fix issue with abnormal behavior when time precision is not in milliseconds during grouping by month.
- Fix issue with abnormal behavior when duration contains multiple units during grouping by month.
- Fix bug where limit and offset cannot be pushed down when there is an order by clause.
- Fix abnormal behavior in combination scenario of grouping by month + align by device + limit.
- Fix deserialization errors during IoT protocol synchronization.
- Fix concurrent exceptions in deleting timeseries.
- Fix issue where group by level in view sequences does not execute correctly.
- Fix potential issue of metadata creation failure when increasing election timeout
New Contributors
* Jgj1006 made their first contribution in https://github.com/apache/iotdb/pull/11730

**Full Changelog**: https://github.com/apache/iotdb/compare/v1.3.0...v1.3.1

Features & Improvements

- Optimize the permission module and support timeseries permission control
- Optimize heap and off-heap memory configuration in startup script
- Computed-type view timeseries support LAST queries
- Add pipe-related monitoring metrics
- Pipe rename 'Extractor' to 'Source' and 'Connector' to 'Sink'
- [IOTDB-6138] Support Negative timestamp
- [IOTDB-6193] Reject Node startup when loading configuration file failed
- [IOTDB-6194] Rename target_config_node_list to seed_config_node
- [IOTDB-6200] Change schema template to device template
- [IOTDB-6207] Add Write Point Metrics for load function
- [IOTDB-6208] Node error detection through broken thrift pipe
- [IOTDB-6217] When the number of time series reaches the upper limit, the prompt message should be changed to prioritize using device templates
- [IOTDB-6218] Rename storage_query_schema_consensus_free_memory_proportion to datanode_memory_proportion
- [IOTDB-6219] Fix the display problem of explain that the print result is not aligned
- [IOTDB-6220] Pipe: Add check logic to avoid self-transmission
- [IOTDB-6222] Optimize the performance of Python client
- [IOTDB-6230] Add HEAPDUMP configuration in datanode-env.sh
- [IOTDB-6231]SchemaCache supports precise eviction
- [IOTDB-6232] Adding SSL function to dn_rpc_port

Bugs

- [IOTDB-6160] while using ` in target path, select into will throw error
- [IOTDB-6167] DataNode can't register to cluster when fetch system configuration throws NPE
- [IOTDB-6168] ConfigNode register retry logic does not worked
- [IOTDB-6171] NPE will be thrown while printing FI with debug on
- [IOTDB-6184] Merge Sort finishes one iterator too long
- [IOTDB-6191] Fix group by year not considering leap years
- [IOTDB-6226] Fix the problem of inaccurate GC monitor detection at the beginning and adjusting the alert threshold
- [IOTDB-6239] Show regions display error create time

New Contributors
* clayburn made their first contribution in https://github.com/apache/iotdb/pull/11243
* chunshao90 made their first contribution in https://github.com/apache/iotdb/pull/11385

**Full Changelog**: https://github.com/apache/iotdb/compare/v1.2.2...v1.3.0

Features & Improvements

- [IOTDB-6029] Implementing flink-sql-iotdb-connector
- [IOTDB-6084] Pipe: support node-urls in connector-v1
- [IOTDB-6103] Adding count_time aggregation feature
- [IOTDB-6112] Limit & Offset push down doesn't take effect while there exist time filter
- [IOTDB-6115] Limit & Offset push down doesn't take effect while there exist null value
- [IOTDB-6120] push down limit/offset in query with group by time
- [IOTDB-6129] ConfigNode restarts without relying on Seed-ConfigNode
- [IOTDB-6131] Iotdb rest service supports insertRecords function
- [IOTDB-6151] Move DataNode's system.properties to upper dir
- [IOTDB-6173] Change default encoder of INT32 and INT64 from RLE to TS_2DIFF
- Adjust the default thrift timeout parameter to 60s
- Accelerate the deletion execution

Bugs

- [IOTDB-6064] Pipe: Fix deadlock in rolling back procedures concurrently
- [IOTDB-6081] Pipe: use HybridExtractor instead of LogExtractor when realtime mode is set to log to avoid OOM under heavy insertion load
- [IOTDB-6145] Pipe: can not release TsFile or WAL resource after pipe is dropped
- [IOTDB-6146] Pipe: can not transfer data after 1000+ pipes' creating and dropping
- [IOTDB-6082] Improve disk space metrics
- [IOTDB-6104] tmp directory won't be cleaned after udf query end
- [IOTDB-6119] Add ConfigNode leader service check
- [IOTDB-6125] Fix DataPartition allocation bug when insert big batch data
- [IOTDB-6127] Pipe: buffered events in processor stage can not be consumed by connector
- [IOTDB-6132] CrossSpaceCompaction: The estimated memory size is too large for cross space compaction task
- [IOTDB-6133] NullPointerException occurs in unsequence InnerSpaceCompactionTask
- [IOTDB-6148] Pipe: Fixed the bug that some uncommited progresses may be reported
- [IOTDB-6156] Fixed TConfiguration invalidly in Thrift AsyncServer For IoTConsensus
- [IOTDB-6164] Can create illegal path through rest api
- Fix datanode status is ReadOnly because the disk is full
- Fix DataPartition allocation bug when insert big batch
- Fix flush point statistics
- Fix SchemaFileSketchTool is not found
- Refactoring DeleteOutdatedFileTask in WalNode
- Add compression and encoding type check for FastCompactionPerformer
- Add lazy page reader for aligned page reader to avoid huge memory cost when reading rows of aligned timeseries
- Pipe: use PipeTaskCoordinatorLock instead of ReentrantLock for multi thread sync
- Pipe: fix pipe procedure stuck because of data node async request forever waiting for response
- Pipe: fix NPE when HybridProgressIndex.updateToMinimumIsAfterProgressIndex after system reboot (DR: SimpleConsensus)
- Pipe: fix pipe coordinator deadlock causing CN election timeout
- Pipe: Improve performance for 10000+ pipes
- RATIS-1873. Remove RetryCache assertion that doesn't hold

**Full Changelog**: https://github.com/apache/iotdb/compare/v1.2.1...v1.2.2

Features & Improvements

- [IOTDB-5557] The metadata query results are inconsistent
- [IOTDB-5997] Improve efficiency of ConfigNode PartitionInfo loadSnapshot
- [IOTDB-6019] Fix concurrent update of last query
- [IOTDB-6036] The mods file is too large, causing Query very slow even OOM problem
- [IOTDB-6055] Enable auto restart of the pipes stopped by ConfigNode because of critical exception
- [IOTDB-6066] Add ConfigNode timeslot metric
- [IOTDB-6073] Add ClientManager metrics
- [IOTDB-6077] Add force stop
- [IOTDB-6079] Cluster computing resource balance
- [IOTDB-6082] Improve disk space metrics
- [IOTDB-6087] Implement stream interface of Mods read
- [IOTDB-6090] Add memory estimator on inner space compaction
- [IOTDB-6092] Factor mods files into memory estimates for cross-space compaction tasks
- [IOTDB-6093] Add multiple validation methods after compaction
- [IOTDB-6106] Fixed the timeout parameter not working in thrift asyncClient
- [IOTDB-6108] AlignedTVList memory calculation is imprecise
-
Bugs

- [IOTDB-5855] DataRegion leader Distribution is same as DataRegion Distribution
- [IOTDB-5860] Total Number of file is wrong
- [IOTDB-5996] Incorrect time display of show queries
- [IOTDB-6057] Resolve the compatibility from 1.1.x to 1.2.0
- [IOTDB-6065] Considering LastCacheContainer in the memory estimation of SchemaCacheEntry
- [IOTDB-6074] Ignore error message when TagManager createSnapshot
- [IOTDB-6075] Pipe: File resource races when different tsfile load operations concurrently modify the same tsfile at receiver
- [IOTDB-6076] Add duplicate checking when upsert alias
- [IOTDB-6078] fix timeChunk default compressType
- [IOTDB-6089] Improve the lock behaviour of the pipe heartbeat
- [IOTDB-6091] Add compression and encoding type check for FastCompactionPerformer
- [IOTDB-6094] Load：Fix construct tsFileResource bug
- [IOTDB-6095] Tsfiles in sequence space may be overlap with each other due to LastFlushTime bug
- [IOTDB-6096] M4 will output zero while meeting null
- [IOTDB-6097] ipe subscription running with the pattern option may cause OOM
- [IOTDB-6098] Flush error when writing aligned timeseries
- [IOTDB-6100] Pipe: Fix running in hybrid mode will cause wal cannot be deleted & some pipe data lost due to wrong ProducerType of Disruptor
- [IOTDB-6105] Load: NPE when analyzing tsfile

New Contributors
* ZhangChaoming made their first contribution in https://github.com/apache/iotdb/pull/10152
* qiuyluo made their first contribution in https://github.com/apache/iotdb/pull/10422

**Full Changelog**: https://github.com/apache/iotdb/compare/v1.2.0...v1.2.1