Apache-iotdb

Features & Improvements

- [IOTDB-5557] The metadata query results are inconsistent
- [IOTDB-5997] Improve efficiency of ConfigNode PartitionInfo loadSnapshot
- [IOTDB-6019] Fix concurrent update of last query
- [IOTDB-6036] The mods file is too large, causing Query very slow even OOM problem
- [IOTDB-6055] Enable auto restart of the pipes stopped by ConfigNode because of critical exception
- [IOTDB-6066] Add ConfigNode timeslot metric
- [IOTDB-6073] Add ClientManager metrics
- [IOTDB-6077] Add force stop
- [IOTDB-6079] Cluster computing resource balance
- [IOTDB-6082] Improve disk space metrics
- [IOTDB-6087] Implement stream interface of Mods read
- [IOTDB-6090] Add memory estimator on inner space compaction
- [IOTDB-6092] Factor mods files into memory estimates for cross-space compaction tasks
- [IOTDB-6093] Add multiple validation methods after compaction
- [IOTDB-6106] Fixed the timeout parameter not working in thrift asyncClient
- [IOTDB-6108] AlignedTVList memory calculation is imprecise
-
Bugs

- [IOTDB-5855] DataRegion leader Distribution is same as DataRegion Distribution
- [IOTDB-5860] Total Number of file is wrong
- [IOTDB-5996] Incorrect time display of show queries
- [IOTDB-6057] Resolve the compatibility from 1.1.x to 1.2.0
- [IOTDB-6065] Considering LastCacheContainer in the memory estimation of SchemaCacheEntry
- [IOTDB-6074] Ignore error message when TagManager createSnapshot
- [IOTDB-6075] Pipe: File resource races when different tsfile load operations concurrently modify the same tsfile at receiver
- [IOTDB-6076] Add duplicate checking when upsert alias
- [IOTDB-6078] fix timeChunk default compressType
- [IOTDB-6089] Improve the lock behaviour of the pipe heartbeat
- [IOTDB-6091] Add compression and encoding type check for FastCompactionPerformer
- [IOTDB-6094] Load：Fix construct tsFileResource bug
- [IOTDB-6095] Tsfiles in sequence space may be overlap with each other due to LastFlushTime bug
- [IOTDB-6096] M4 will output zero while meeting null
- [IOTDB-6097] ipe subscription running with the pattern option may cause OOM
- [IOTDB-6098] Flush error when writing aligned timeseries
- [IOTDB-6100] Pipe: Fix running in hybrid mode will cause wal cannot be deleted & some pipe data lost due to wrong ProducerType of Disruptor
- [IOTDB-6105] Load: NPE when analyzing tsfile

New Contributors
* ZhangChaoming made their first contribution in https://github.com/apache/iotdb/pull/10152
* qiuyluo made their first contribution in https://github.com/apache/iotdb/pull/10422

**Full Changelog**: https://github.com/apache/iotdb/compare/v1.2.0...v1.2.1

New Feature

* [IOTDB-5567] add SQL for querying seriesslotid and timeslotid
* [IOTDB-5631] Add a built-in aggregation functions named time_duration
* [IOTDB-5636] Add round as built-in scalar function
* [IOTDB-5637] Add substr as built-in scalar function
* [IOTDB-5638] Support case when syntax in IoTDB
* [IOTDB-5643] Add REPLACE as a built-in scalar function
* [IOTDB-5683] Support aggregation function Mode for query
* [IOTDB-5711] Python API should support connecting multiple nodes
* [IOTDB-5752] Python Client supports write redirection
* [IOTDB-5765] Support Order By Expression
* [IOTDB-5771] add SPRINTZ and RLBE encodor and LZMA2 compressor
* [IOTDB-5924] Add SessionPool deletion API
* [IOTDB-5950] Support Dynamic Schema Template
* [IOTDB-5951] Support show timeseries/device with specific string contained in path
* [IOTDB-5955] Support create timeseries using schema template in Session API

Improvements

* [IOTDB-5630] Make function cast a built-in function
* [IOTDB-5689] Close Isink when ISourceHandle is closed
* [IOTDB-5715] Improve the performance of query order by time desc
* [IOTDB-5763] Optimize the memory estimate for INTO operations
* [IOTDB-5887] Optimize the construction performance of PathPatternTree without wildcards
* [IOTDB-5888] TTL logs didn' t consider timestamp precision
* [IOTDB-5896] Failed to execute delete statement
* [IOTDB-5908] Add more query metrics
* [IOTDB-5911] print-iotdb-data-dir tool cannot work
* [IOTDB-5914] Remove redundant debug log in Session
* [IOTDB-5919] show variables add a variable timestamp_precision
* [IOTDB-5926] Optimize metric implementation
* [IOTDB-5929] Enable DataPartition inherit policy
* [IOTDB-5943] Avoid rpc invoking for SimpleQueryTerminator when endpoint is local address
* [IOTDB-5944] Follower doesn' t need to update last cache when using IoT_consensus
* [IOTDB-5945] Add a cache to avoid initialize duplicated device id object in write process
* [IOTDB-5946] Optimize the implement of tablet in Go client
* [IOTDB-5949] Support show timeseries with datatype filter
* [IOTDB-5952] Support FIFO strategy in DataNodeSchemaCache
* [IOTDB-6022] The WAL piles up when multi-replica iotconsensus is written at high concurrency


Bug Fixes

* [IOTDB-5604] NPE when execute Agg + align by device query without assigned DataRegion
* [IOTDB-5619] group by tags query NPE
* [IOTDB-5644] Unexpected result when there are no select expressions after analyzed in query
* [IOTDB-5657] Limit does not take effect in last query
* [IOTDB-5700] UDF query did not clean temp file after the query is finished
* [IOTDB-5716] Wrong dependency when pipeline consumeOneByOneOperator
* [IOTDB-5717] Incorrect result when querying with limit push-downing & order by time desc
* [IOTDB-5722] Wrong default execution branch in PlanVisitor
* [IOTDB-5735] The result of adding the distinct function to the align by device is incorrect
* [IOTDB-5755] Fix the problem that 123w can not be used in Identifier
* [IOTDB-5756] NPE when where predicate is NotEqualExpression and one of subExpression is not exist
* [IOTDB-5757] Not Supported Exception when use like ' s3 || false' in where even Type of s3 is Boolean
* [IOTDB-5760] Query is blocked because of no memory
* [IOTDB-5764] Cannot specify alias successfully when the FROM clause contains multiple path suffixes
* [IOTDB-5769] Offset doesn' t take effect in some special case
* [IOTDB-5774] The syntax that path nodes start or end with a wildcard to fuzzy match is not supported
* [IOTDB-5784] Incorrect result when querying with offset push-down and time filter
* [IOTDB-5815] NPE when using UDF to query
* [IOTDB-5837] Exceptions for select into using placeholders
* [IOTDB-5851] Using limit clause in show devices query will throw NPE
* [IOTDB-5858] Metric doesn' t display the schemaCache hit ratio
* [IOTDB-5861] Last quey is incomplete
* [IOTDB-5889] TTL Cannot delete expired tsfiles
* [IOTDB-5897] NullPointerException In compaction
* [IOTDB-5905] Some aligned timeseries data point lost after flush
* [IOTDB-5934] Optimize cluster partition policy
* [IOTDB-5953] LastCache memory control param does not take effect
* [IOTDB-5963] Sometimes we may get out-of-order query result
* [IOTDB-6016] Release file num cost after cross compaction task

New Contributors
* EliasMei made their first contribution in https://github.com/apache/iotdb/pull/9291
* nikam14 made their first contribution in https://github.com/apache/iotdb/pull/9368
* riverbuilding made their first contribution in https://github.com/apache/iotdb/pull/9379
* lichenyu20 made their first contribution in https://github.com/apache/iotdb/pull/9458

**Full Changelog**: https://github.com/apache/iotdb/compare/v1.1.2...v1.2.0

New Feature

* [IOTDB-5919]show variables add a variable timestamp_precision
* Add Python SessionPool

Improvement/Bugfix

* [IOTDB-5901] Load: load tsfile without data will throw NPE
* [IOTDB-5903] Fix cannot select any inner space compaction task when there is only unsequence data
* [IOTDB-5878] Allow ratis-client retry when gRPC IO Unavailable
* [IOTDB-5939] Correct Flusing Task Timeout Detect Thread
* [IOTDB-5905] Fix aligned timeseries data point lost after flushed in some scenario
* [IOTDB-5963] Make sure that TsBlock blocked on memory is added in queue before the next TsBlock returned by root operator
* [IOTDB-5819] Fix npe when booting net metrics
* [IOTDB-6023] Pipe: Fix load tsfile error while handling empty value chunk
* [IOTDB-5971] Fix potential QUOTE problem in iotdb reporter
* [IOTDB-5993] ConfigNode leader changing causes lacking some DataPartition allocation result in the response of getOrCreateDataPartition method
* [IOTDB-5910] Fix compaction scheduler thread pool is not shutdown when aborting compaction
* [IOTDB-6056] Pipe: Failed to load tsfile with empty pages (NPE occurs when loading)
* [IOTDB-5916]Fix exception when file is deleted during compaction selection
* [IOTDB-5896] Fix the NPE issue when taking snapshot in WAL combined with Aligned Binary
* [IOTDB-5929] Enable DataPartition inherit policy
* [IOTDB-5934] Optimize cluster partition policy
* [IOTDB-5926] Remove Useless Rater in Timer
* [IOTDB-6030] Improve efficiency of ConfigNode PartitionInfo takeSnapshot
* [IOTDB-5997] Improve efficiency of ConfigNode PartitionInfo loadSnapshot
* Fix potential deadlock when freeing memory in MemoryPool
* Release resource of FI after all drivers have been closed
* Set default degree of parallelism back to the num of CPU
* Make SequenceStrategy and MaxDiskUsableSpaceFirstStrategy are allowed in cluster mode
* Fix npe exception when invalid in metric module
* Fix CQ does not take effect in ns time_precision
* Fix storage engine memory config initialization
* Fix template related schema query
* add default charset setting in start-datanode.bat and print default charset when starting
* Fix TsfileResource error after delete device in sequence working memtable
* load TsFile bugs: Not checking whether the tsfile data loaded locally is in the same time partition during the loading process & LoadTsFilePieceNode error when loading tsfile with empty value chunks
* Fix alias query failure after restarting DataNode

New Feature

* [IOTDB-2569] ZSTD compression

Improvement/Bugfix

* [IOTDB-5781] Change the default strategy to SequenceStrategy
* [IOTDB-5780] Let users know a node was successfully removed and data is recovered
* [IOTDB-5735] The result of adding the distinct function to the align by device is incorrect
* [IOTDB-5777] When writing data using non-root users, the permission authentication module takes too long
* [IOTDB-5835] Fix wal accumulation caused by datanode restart
* [IOTDB-5828] Optimize the implementation of some metric items in the metric module to prevent Prometheus pull timeouts
* [IOTDB-5813] ConfigNode restart error due to installSnapshot failed
* [IOTDB-5657] Limit does not take effect in last query
* [IOTDB-5717] Incorrect result when querying with limit push-downing & order by time desc
* [IOTDB-5722] Wrong default execution branch in PlanVisitor
* [IOTDB-5784] Incorrect result when querying with offset push-down and time filter
* [IOTDB-5815] NPE when using UDF to query
* [IOTDB-5829] Query with limit clause will cause other concurrent query break down
* [IOTDB-5824] show devices with * cannot display satisfied devices
* [IOTDB-5831] Drop database won't delete totally files in disk
* [IOTDB-5818] Cross_space compaction of Aligned timeseries is stucked
* [IOTDB-5859] Compaction error when using Version as first sort dimension
* [IOTDB-5869] Fix load overlap sequence TsFile

New Features

* [IOTDB-4572] support order by time in align by device
* [IOTDB-4816] Support Show Queries command
* [IOTDB-4817] Support kill query command
* [IOTDB-5077] Support new command formats in SHOW REGIONS
* [IOTDB-5108] Support region migration
* [IOTDB-5202] Support show regions with specific database
* [IOTDB-5282] Add SQL show variables, which can display the current cluster variables
* [IOTDB-5341] Support GROUP BY VARIATION in aggregation query
* [IOTDB-5372] Support data type cast in SELECT INTO
* [IOTDB-5382] Support DIFF as built-in scalar function in IoTDB
* [IOTDB-5393] Show Region creation time when execute show regions
* [IOTDB-5456] Implement COUNT_IF built-in aggregation function
* [IOTDB-5515] Support GROUP BY CONDITION in aggregation query
* [IOTDB-5555] Enable modify dn_rpc_port and dn_rpc_address in datanode-engine.properties
* Support docker deployment
* Rename SeriesSlotId to SeriesSlotNum in show regions

Improvements

* [IOTDB-4497] Improve NodeStatus definition
* [IOTDB-5066] Upgraded the GetSlots SQLs
* [IOTDB-5161] Add output type check for WHERE & HAVING clause and refuse expressions whose return type are not boolean exist in WHERE & HAVING clause
* [IOTDB-5185] Fixed that old snapshot is not deleted in IoTConsensus
* [IOTDB-5287] Added status “Discouraged” to RegionGroup
* [IOTDB-5449] Wait for query resource while the query queue is full instead of throwing exception directly
* change STARTUP_RETRY_INTERVAL_IN_MS from 30s to 3s

Bug Fixes

* [IOTDB-4684] Fix devices with the same name but different alignment properties are compacted into the wrong alignment property
* [IOTDB-5061] Add initialize state check when create snapshot
* [IOTDB-5090] Fix the NPE when executing stop-datanode.sh
* [IOTDB-5112] Fixed IoTConsensus synchronization stuck under low load or during restart
* [IOTDB-5126] Fix that show-regions show IP rather than hostname even though the hostname is used when registering
* [IOTDB-5165] Fix fail to pass compaction validation when resources degrade to FileTimeIndex
* [IOTDB-5170] Fix the problem that datanode is closed when executing stop-confignode.bat on Windows platform
* [IOTDB-5199] Fix NPE in StorageExector inLoading process
* [IOTDB-5216] Fix order by timeseries doesn't take effect in aligned last query
* [IOTDB-5228] NPE if the file does not exist while creating TsFileSequenceReader instance
* [IOTDB-5240] Fix schema template timeseries read/write error after restart
* [IOTDB-5245] Fix data is incomplete in last query
* [IOTDB-5277] Fix DataNode restart failure when SchemaRegion loading snapshot
* [IOTDB-5278] Fix JDBC Driver cannot connect to the dbeaver
* [IOTDB-5285] Fix TimePartition error when restarting with different time partition configuration
* [IOTDB-5269] Fix wrong data of devices ought to be deleted after executing `delete from` sql
* [IOTDB-5324] Fix wal cann't be deleted in destDataNode after region migration when data_replication_factor is 1 in IoTConsensus
* [IOTDB-5389] Cause DataNode startup to fail when wal_mode is disabled in IoTConsensus
* [IOTDB-5414] Fix timeseries with alias deleted success but can still be queried by alias
* [IOTDB-5426] Cannot trigger flush for sequence file when timed flush enabled
* [IOTDB-5441] Fix NPE while fetch schema that is not in template used by related device
* [IOTDB-5469] Fix get schema info failure after creating template with backquote characters successfully
* [IOTDB-5474] Fix count nodes using level bug
* [IOTDB-5480] IoTConsensus sync lag may be negative under single copy
* [IOTDB-5488] Support set system to readonly on local
* [IOTDB-5492] Skip broken tsfile when recovering system
* [IOTDB-5512] Fixed IoTConsensus may repeatedly send some log when restarting
* [IOTDB-5516] Fix creating timeseries failure after dropping all databases
* [IOTDB-5526] Fix deleting timeseries failure when it belongs to a device activating template
* [IOTDB-5548] Fix dropping template failure when using pure number as template name
* [IOTDB-5592] Fix unexpected error while using full path(starting with root) in having/where clause
* [IOTDB-5620] Fix flush stuck when there is a lot of time partitions in each DataRegion
* [IOTDB-5639] Fix file not found exception in cross space compaction selector
* [IOTDB-5657] Fix LIMIT&OFFSET does not take effect in last query
* [IOTDB-5684] Fix log folder generation when using ConfigNode's Simple consensus protocol
* [IOTDB-5685] Fix error msg of failing to create a timeseries on an existing path when ReadOnly state
* [IOTDB-5686] Fix devices with the same name but different alignment properties meets error in inner seq compaction
* [IOTDB-5700] Clean temporary files created by UDF query after it finishes
* Add SHOW_CQ privilege

New Features

* [IOTDB-5102] Support explain the sqls using align by device clause
* [IOTDB-5282] Add SQL show variables, which can display the current cluster variables

Improvements

* [IOTDB-4595] Add real-time monitoring data for tsfile merge
* [IOTDB-5125] Stop the process when port is already in use when launching ConfigNode or DataNode
* [IOTDB-5185] Fixed that old snapshot is not deleted in IoTConsensus
* [IOTDB-5202] Support showing regions regarding database
* [IOTDB-5205] Construct Cluster Node startup protocol, reject start or restart request of ConfigNode or DataNode when using wrong data
* [IOTDB-5208] Fix increase of file handles when TsFileResource is degraded and compacted with fast performer
* [IOTDB-5223] Cause DataNode startup to fail when multiple disks are configured in the data directory and the dataRegion consensus protocol uses Ratis
* [IOTDB-5232] Add Schema Memory Usage metric monitor
* [IOTDB-5249] Add interface to set datatype when users customize the payload formatter of mqtt
* [IOTDB-5258] Add timeout parameter for ExportCSV/ExportTsFile Tool
* [IOTDB-5306] Change the default port of ConfigNode and DataNode
* [IOTDB-5326] Add cluster_name parameter to distinguish different cluster
* [IOTDB-5389] Cause DataNode starup to fail when wal_mode is disabled in IoTConsensus

Bug Fixes

* [IOTDB-4978] Fix possible deadLock while using select into
* [IOTDB-5090] Fix the NPE when executing stop-datanode.sh
* [IOTDB-5116] Fix wrong empty result set in aggregation query while there is no data in queried timeseries
* [IOTDB-5118] Fix that datanode opens too many file handles （TYPE DIR）
* [IOTDB-5174] Fixed restart failure when IoTConsensus hostname containing '-'
* [IOTDB-5126] Fix that show-regions show IP rather than hostname even though the hostname is used when registering
* [IOTDB-5135] Fix the Sync snapshot failure (also blocks ConfigNode snapshot) when sync file is empty
* [IOTDB-5141] Fix fast compaction scheduler bug
* [IOTDB-5144] Fix classCastException in IoTDBDatabaseMetadata of jdbc
* [IOTDB-5158] Fix failure of getting devices from tsfile
* [IOTDB-5170] Fix the problem that datanode is closed when executing stop-confignode.bat on Windows platform
* [IOTDB-5210] Fix concurrent bug caused by init and close of driver
* [IOTDB-5212] Upgrade an illegal CQ verification prompt from v0.14.0 to v1.0
* [IOTDB-5215] Fix the wrong usage of chunkPointNumLowerBound in IoTDBDescriptor
* [IOTDB-5216] Fix order by timeseries doesn't take effect in aligned last query
* [IOTDB-5222] Fix the load of some metric related configuration
* [IOTDB-5231] Fix that Datanode could not start when binding 9091 error
* [IOTDB-5233] Correct the message of unsupported thrift interface of schema template
* [IOTDB-5236] Fix DeleteData Python API cannot work
* [IOTDB-5240] Fix schema template timeseries read/write error after restart
* [IOTDB-5261] Fix the number of clients in client pool of AsyncIoTConsensusServiceClient can not be changed through dn_max_connection_for_internal_service
* [IOTDB-5266] Refine the code of cross selector and fix the bug when using FileTImeIndex
* [IOTDB-5269] Fix wrong data of devices ought to be deleted after executing `delete from` sql
* [IOTDB-5277] Fix DataNode restart failure when SchemaRegion loading snapshot
* [IOTDB-5278] Fix JDBC Driver cannot connect to the dbeaver
* [IOTDB-5285] Fix TimePartition error when restarting with different time partition configuration
* [IOTDB-5286] Fix target files are selected to participate in other compaction task
* [IOTDB-5313] Fix Python client doesn't handle REDIRECTION_RECOMMEND correctly
* [IOTDB-5452] Add leastDataRegionGroupNumSetByUser examination to ensure that leastDataRegionGroupNum will not be changed by system
* [IOTDB-5498] Fix SessionPool OOM problem when the number of devices and sessions is large
* [IOTDB-5501] Fix memory leak in MemoryPool
* [IOTDB-5525] Fix the partition error of device after partition fetcher