Apache-iotdb

Improvements

* [IOTDB-842] Better Export/Import-CSV Tool
* [IOTDB-1738] Cache paths list in batched insert plan
* [IOTDB-1792] remove tomcat-embed dependency and make all transitive dependencies versions consistent
* [ISSUE-4072] Parallel insert records in Session
* Print the file path while meeting error in case of reading chunk

Bug Fixes

* [IOTDB-1275] Fix backgroup exec for cli -e function causes an infinite loop
* [IOTDB-1287] Fix C++ class Session has 2 useless sort()
* [IOTDB-1289] fix CPP mem-leak in SessionExample.cpp insertRecords()
* [IOTDB-1484] fix auto create schema in cluster
* [IOTDB-1578] Set unsequnce when loading TsFile with the same establish time
* [IOTDB-1619] Fix an error msg when restart iotdb-cluster
* [IOTDB-1629] fix the NPE when using value fill in cluster mode
* [IOTDB-1632] Fix Value fill function fills even when the data exists
* [IOTDB-1651] add reconnect to solve out of sequence in sync module
* [IOTDB-1659] Fix Windows CLI cannot set maxPRC less than or equal to 0
* [IOTDB-1670] Fix cli -e mode didn't fetch timestamp_precision from server
* [IOTDB-1674] Fix command interpret error causing somaxconn warning failed
* [IOTDB-1677] Fix not generate file apache-iotdb-0.x.x-client-cpp-linux-x86_64-bin.zip.sha512
* [IOTDB-1678] Fix client-cpp session bug: can cause connection leak.
* [IOTDB-1679] client-cpp: Session descontruction need release server resource
* [IOTDB-1690] Fix align by device type cast error
* [IOTDB-1693] fix IoTDB restart does not truncate broken ChunkGroup bug
* [IOTDB-1703] Fix MManager slow recover with tag
* [IOTDB-1714] fix Could not find or load main class when start with jmx on win
* [IOTDB-1723] Fix concurrency issue in compaction selection
* [IOTDB-1726] Wrong hashCode() and equals() method in ChunkMetadata
* [IOTDB-1727] Fix Slow creation of timeseries with tag
* [IOTDB-1731] Fix sync error between different os
* [IOTDB-1733] Fix dropping built-in function
* [IOTDB-1741] Avoid double close in level compaction execution
* [IOTDB-1785] Fix Illegal String ending with . being parsed to PartialPath
* [IOTDB-1836] Fix Query Exception Bug after deleting all sgs
* [IOTDB-1837] Fix tagIndex rebuild failure after upgrade mlog from mlog.txt to mlog.bin
* [IOTDB-1838] The compacting status in SGP is always false
* [IOTDB-1846] Fix the error when count the total number of devices in cluster mode
* [IoTDB-1847] Not throw excpetion when pulling non--existent time series
* [IOTDB-1850] Fix deserialize page merge rate limiter
* [IoTDB-1865] Compaction is blocking when removing old files in Cluster
* [IOTDB-1868] Use RwLock to reduce the lock time for nodeRing
* [IOTDB-1872] Fix data increases abnormally after IoTDB restarts
* [IOTDB-1877] Fix Sync recovery and reconnection bugs in both sender and receiver
* [IOTDB-1879] Fix some Unsequence files never be merged to higher level or Sequence folder
* [IOTDB-1887] Fix importing csv data containing null throws exception
* [IOTDB-1893] Fix Can not release file lock in sync verify singleton
* [IOTDB-1895] Cache leader optimization for batch write interfaces on multiple devices
* [IOTDB-1903] Fix IndexOutOfRangeException when starting IoTDB
* [IoTDB-1913] Fix When exporting a amount of data from csv, it will report network error or OOM
* [IOTDB-1925] Fix the modification of max_select_unseq_file_num_in_each_compaction parameter does not take effect
* [IOTDB-1958] Add storage group not ready exception
* [IOTDB-1961] Cluster query memory leak
* [IOTDB-1975] OOM caused by that MaxQueryDeduplicatedPathNum doesn't take effect
* [IOTDB-1983] Fix DescReadWriteBatchData serializing bug
* [IOTDB-1990] Fix unchecked null result by calling IReaderByTimestamp.getValuesInTimestamps()
* [ISSUE-3945] Fix Fuzzy query not support multiDevices and alignByDevice Dataset
* [ISSUE-4288] Fix CI issue caused by the invalid pentaho download url
* [ISSUE-4293] SessionPool: InterruptedException is not properly handled in synchronized wait()
* [ISSUE-4308] READ_TIMESERIES privilege granted to users and roles can not take effect when quering by UDFs
* fix merge ClassCastException: MeasurementMNode
* change sync version check to major version
* init dummyIndex after restart cluster

New Features
* [IOTDB-959] Add create storage group Grammar
* [IOTDB-1399] Add a session interface to connect multiple nodes
* [IOTDB-1466] Support device template
* [IOTDB-1491] UDTF query supported in cluster
* [IOTDB-1496] Timed flush memtable
* [IOTDB-1536] Support fuzzy query REGEXP
* [IOTDB-1561] Support fill by specific value
* [IOTDB-1565] Add sql: set system to readonly/writable
* [IOTDB-1569] Timed close TsFileProcessor
* [IOTDB-1586] Support mysql-style Like clause
* [ISSUE-3811] Provide a data type column for the last query dataset
* TTL can be set to the prefix path of storage group
* add JMX monitor to all ThreadPools in the server module

Improvements
* [IOTDB-1566] Do not restrict concurrent write partitions
* [IOTDB-1585] ModificationFile‘s write interface blocking
* [IOTDB-1587] SessionPool optimization: a more aggressive Session creation strategy
* Use StringCachedPool in TsFileResource to reduce the memory size
* write performance optimization when replicaNum == 1
* Optimize Primitive Array Manager
* Function Improvement: add overlapped page rate in Tracing


Bug Fixes
* [IOTDB-1282] fix C++ class SessionDataSet mem-leak
* [IOTDB-1407] fix Filtering time series based on tags query fails Occasionally
* [IOTDB-1437] Fix the TsFileSketchTool NPE
* [IOTDB-1442] Time filter & TTL do not take effect in cluster
* [IOTDB-1452] remove compaction log/ change logger to daily
* [IOTDB-1447] ClientPool is blocking other nodes when one node fails
* [IOTDB-1456] Fix Error occurred while executing delete timeseries statement
* [IOTDB-1461] Fix compaction conflicts with ttl
* [IOTDB-1462] Fix cross space compaction recover null pointer bug
* [IOTDB-1464] fix take byte array null pointer
* [IOTDB-1469] fix cross space compaction lost data bug
* [IOTDB-1471] Fix path not right in "sg may not ready" log
* [IOTDB-1475] MeasurementId check while create timeseries or template/ disable time or timestamp in timeseries path
* [IOTDB-1488] Fix metaMember's forwarding clientPool timeout in cluster module
* [IOTDB-1494] fix compaction block flush bug
* [IoTDB-1499] Remove series registration using IoTDBSink
* [IoTDB-1501] Fix compaction recover delete tsfile bug
* [IOTDB-1529] Fix mlog recover idx bug and synchronize setStorageGroup
* [IOTDB-1537] fix insertTablet permission
* [IOTDB-1539] Fix delete operation with value filter is abnormal
* [IOTDB-1540] Bug Fix: 500 when using IN operator
* [IOTDB-1541] Fix query result not right due to non-precise time index of resource
* [IOTDB-1542] Cpp client segment fault: char[] buffer overflow caused by long exception message
* [IOTDB-1545] Query dataset memory leak on server caused by cpp client
* [IOTDB-1546] Optimize the Upgrade Tool rewrite logic to reduce the temp memory cost
* [IOTDB-1552] Only allow equivalent filter for TEXT data type
* [IOTDB-1556] Abort auto create device when meet exception in setStorageGroup
* [IOTDB-1574] Deleted file handler leak
* [IOTDB-1580] Error result of order by time desc when enable time partition
* [IOTDB-1584] Doesn't support order by time desc in cluster mode
* [IOTDB-1588] Bug fix: MAX_TIME is incorrect in cluster mode
* [IOTDB-1594] Fix show timeseries returns incorrect tag value
* [IOTDB-1600] Fix InsertRowsOfOneDevicePlan being not supported in cluster mode
* [IOTDB-1610] Fix TsFileRewriteTool writing incorrect data file
* [ISSUE-3116] Bug when using natural month unit in time interval in group by query
* [ISSUE-3316] Query result with the same time range is inconsistent in group by query
* [ISSUE-3436] Fix query result not right after deleting multiple time interval of one timeseries
* [ISSUE-3458] fix load configuration does not take effect
* [ISSUE-3545] Fix Time interval value is disorder in group by month
* [ISSUE-3653] fix Max_time and last return inconsistent result
* [ISSUE-3690] Memory leaks on the server when cpp client invokes checkTimeseriesExists
* [ISSUE-3805] OOM caused by Chunk cache
* [ISSUE-3865] Meaningless connection reset issues caused by low default value for SOMAXCONN
* Fix DataMigrationExample OOM if migrate too many timeseries
* Handle false positive cases which may cause NPE of tsfile bloom filter
* Fix Windows shell error on JDK11 & fix iotdb-env.bat not working
* Fix cluster auto create schema bug when retry locally
* Fix thrift out of sequence in cluster module
* Skip non exist measurement in where clause in align by device
* fix blocking query when selecting TsFile in compaction
* Fix redundant data in compaction recover
* Fix load tsfile with time partition enable

Incompatible changes
* [IOTDB-1485] Replace tsfile_size_threshold by unseq_tsfile_size/seq_tsfile_size

Miscellaneous changes
* [IOTDB-1499] Remove unused exception throwing notation in IoTDBSink
* [IOTDB-1500] Remove current dynamic query memory control
* [ISSUE-3674] Disable thrift code generation for Javascript
* enable cacheLeader by default
* add audit log when execute delete and set sg for tracing
* modify nodeTool user to root

Bug Fixes
* [GITHUB-3373] Remove the broken cached leader connection & optimize the insertRecords method in session
* [IOTDB-1433] Fix bug in getMetadataAndEndOffset when querying non-exist device
* [IOTDB-1432] fix level compaction loss data
* [IOTDB-1427] Fix compaction lock with query
* [IOTDB-1420] Fix compaction ttl bug
* [IOTDB-1419] Remove redundant clearCompactionStatus, fix continuous compaction doesn't take effect when enablePartition
* [IOTDB-1415] Fix OOM caused by ChunkCache
* [IOTDB-1414] NPE occurred when call getStorageGroupNodeByPath() method using not exist path
* [IOTDB-1412] Unclear exception message thrown when executing empty InsertTabletPlan
* [IOTDB-1411] Fix thriftMaxFrameSize and thriftDefaultBufferSize does not in effect
* [IOTDB-1398] Do not select unseq files when there are uncompacted old unseq files
* [IOTDB-1390] Fix unseq compaction loss data bug
* [IOTDB-1384] Fix group by bug
* [ISSUE-3378] Fix NPE when clear upgrade folder; Fix some upgraded pageHeader missing statistics
* [GITHUB-3339] Try to fix sg dead lock
* [GITHUB-3329] Fix upgrade NPE and DeadLock
* [GITHUB-3319] Fix upgrade tool cannot close file reader
* [IOTDB-1212] Fix The given error message is not right when executing select sin(non_existence) from root.sg1.d1
* [IOTDB-1219] Fix a potential NPE issue in UDF module
* [IOTDB-1286] Fix 4 C++ mem-leak points
* [IOTDB-1294] Fix delete operation become invalid after compaction
* [IOTDB-1313] Fix lossing time precision when import csv with unsupported timestamp format
* [IOTDB-1316] The importCsv tool should continue inserting if a part of insertion failed
* [IOTDB-1317] Fix log CatchUp always failed due to not check the follower's match index
* [IOTDB-1323] Fix return a success message when encounter RuntimeException during the insertion process
* [IOTDB-1325] Fix StackOverflow Exception in group by natural month query
* [IOTDB-1330] Fix the load tsfile bug when the cross multi partition's tsfile only have one page
* [IOTDB-1348] Fix Last plan not work in cluster mode
* [IOTDB-1376] Fix BatchProcessException was not correctly handled in BaseApplier
* [ISSUE-3277] Fix TotalSeriesNumber in MManager counted twice when recovering
* [ISSUE-3116] Fix bug when using natural month unit in time interval in group by query
* [ISSUE-3309] Fix InsertRecordsOfOneDevice runs too slow
* Fix the plan index is always zero when using insertRecords interface to run the cluster
* Add authority check for users create timeseries using executeBatch interface without the privilege
* Fix versionInfo NPE when query upgrading 0.11 tsfile
* Fix upgrade tool cannot load old tsfile if time partition enabled in 0.11
* Fix import csv throw ArrayOutOfIndexError when the last value in a line is null
* Fix upgrade tool cannot close file reader

Improvements
* [GITHUB-3399] Change the default primitive array size to 32
* [IOTDB-1387] Support Without Null ALL in align by device clause, Filter RowRecord automatically if any column in it is null or all columns are null
* [IOTDB-1385] Extract the super user to the configuration
* [IOTDB-1315] ExportCsvTool should support timestamp `yyyy-MM-dd'T'HH:mm:ss.SSSZ`
* [IOTDB-1339] optimize TimeoutChangeableTSnappyFramedTransport
* [IOTDB-1356] Separate unseq_file_num_in_each_level from selecting candidate file in unseq compaction
* [IOTDB-1357] Compaction use append chunk merge strategy when chunk is already large enough
* [IOTDB-1380] Automatically close the dataset while there is no more data
* Optimize sync leader for meta

New Features
* [GITHUB-3389] TTL can be set to any path
* [GITHUB-3387] Add parameter compaction_interval=10000ms
* [IOTDB-1190] Fully support HTTP URL char set in timeseries path
* [IOTDB-1321][IOTDB-1322] Filter RowRecord automatically if any column in it is null or all columns are null
* [IOTDB-1357] Compaction use append chunk merge strategy when chunk is already large
* [ISSUE-3089] Make it possible for storage groups to have name with hyphen

Miscellaneous changes
* [GITHUB-3346] upgrade netty and claim exclusion for enforcer check
* [IOTDB-1259] upgrade libthrift from 0.12.0/0.13.0 to 0.14.1
* Uncomment the less used configurations
* Enable the configration `concurrent_writing_time_partition`

New Features
* [IOTDB-68] New shared-nothing cluster
* [IOTDB-507] Add zeppelin-interpreter module
* [IOTDB-825] Aggregation by natural month
* [IOTDB-890] support SDT lossy compression
* [IOTDB-944] Support UDTF (User-defined Timeseries Generating Function)
* [IOTDB-965] Add timeout parameter for query
* [IOTDB-1077] Add insertOneDeviceRecords API in java session
* [IOTDB-1055] Support data compression type GZIP
* [IOTDB-1024] Support multiple aggregated measurements for group by level statement
* [IOTDB-1276] Add explain sql support and remove debug_state parameter
* [IOTDB-1197] Add iotdb-client-go as a git submodule of IoTDB repo
* [IOTDB-1230] Support spans multi time partitions when loading one TsFile
* [IOTDB-1273] Feature/restrucutre python module as well as supporting pandas dataframe
* [IOTDB-1277] support IoTDB as Flink's data source
* [PR-2605] Add level merge to "merge" command

Incompatible changes
* [IOTDB-1081] New TsFile Format
* [ISSUE-2730] Add the number of unseq merge times in TsFile name.


Miscellaneous changes
* [IOTDB-868] Change mlog from txt to bin
* [IOTDB-1069] Restrict the flushing memtable number to avoid OOM when mem_control is disabled
* [IOTDB-1104] Refactor the error handling process of query exceptions
* [IOTDB-1108] Add error log to print file name while error happened
* [IOTDB-1152] Optimize regular data size in traversing
* [IOTDB-1180] Reset the system log file names and maximal disk-space size
* [ISSUE-2515] Set fetchsize through JDBC and Session
* [ISSUE-2598] Throw explicit exception when time series is unknown in where clause
* [PR-2944] Throw exception when device to be queried is not in TsFileMetaData
* [PR-2967] Log memory usage information in SystemInfo for better diagnosis

Bug Fixes
* [IOTDB-1049] Fix NullpointerException and a delete bug in Last query
* [IOTDB-1050] Fix Count timeserise column name is wrong
* [IOTDB-1068] Fix Time series metadata cache bug
* [IOTDB-1084] Fix temporary memory of flushing may cause OOM
* [IOTDB-1106] Fix delete timeseries bug
* [IOTDB-1126] Fix the unseq tsfile delete due to merge
* [IOTDB-1135] Fix the count timeseries prefix path bug
* [IOTDB-1137] Fix MNode.getLeafCount error when existing sub-device
* [ISSUE-2484] Fix creating timeseries error by using "create" or "insert" statement
* [ISSUE-2545, 2549] Fix unseq merge end time bug
* [ISSUE-2611] An unsequence file that covers too many sequence file causes OOM query
* [ISSUE-2688] LRULinkedHashMap does not work as an LRU Cache
* [ISSUE-2709, 1178] Fix cache not cleared after unseq compaction bug, Fix windows 70,10 ci bug in unseq compaction ci
* [ISSUE-2741] getObject method in JDBC should return an Object
* [ISSUE-2746] Fix data overlapped bug after unseq compaction
* [ISSUE-2758] NullPointerException in QueryTimeManager.checkQueryAlive()
* [ISSUE-2905] Fix Files.deleteIfExists() doesn't work for HDFS file
* [ISSUE-2919] Fix C++ client memory leak bug
* [PR-2613] Fix importCSVTool import directory bug & encode bug
* [PR-2409] Fix import csv which can't import time format str
* [PR-2582] Fix sync bug for tsfiles's directory changed by vitural storage group
* [ISSUE-2911] Fix The write stream is not closed when executing the command 'tracing off'

Bug Fixes
* IOTDB-1303 Disable group by without aggregation function in select clause
* IOTDB-1306 Fix insertion blocked caused the deadlock in memory control module
* IOTDB-1308 Fix users with READ_TIMESERIES permission cannot execute group by fill queries
* IOTDB-1344 Fix cannot create timeseries caused by the timeseries count doesn't reset when deleting storage group
* IOTDB-1384 Some value will disappear while using group by query
* IOTDB-1398 Do not select unseq files when there are uncompacted old unseq files
* ISSUE-3316 Fix query result with the same time range is inconsistent in group by query
* Fix TotalSeriesNumber in MManager counted twice when recovering
* Fix unseq compaction throws a wrong exception if some paths are not in the file
* Fix overlapped data should be consumed first exception when query

Improvements
* [IOTDB-1356] Separate unseq_file_num_in_each_level from selecting candidate file in unseq compaction
* [IOTDB-1412] Unclear exception message thrown when executing empty InsertTabletPlan
* continuous compaction in level compaction strategy when no tsfile is to be closed

New Features
* support brackets with number in timeseries path

Bug Fixes
* ISSUE-2505 ignore PathNotExistException in recover and change recover error to warn
* IOTDB-1119 Fix C++ SessionDataSet bug when reading value buffer
* Fix SessionPool does not recycle session and can not offer new Session due to RunTimeException
* ISSUE-2588 Fix dead lock between deleting data and querying in parallel
* ISSUE-2546 Fix first chunkmetadata should be consumed first
* IOTDB-1126 Fix unseq tsfile is deleted due to compaction
* IOTDB-1137 MNode.getLeafCount error when existing sub-device
* ISSUE-2624 ISSUE-2625 Avoid OOM if user don't close Statement and Session manually
* ISSUE-2639 Fix possible NPE during end query process
* Alter IT for An error is reported and the system is suspended occasionally
* IOTDB-1149 print error for -e param when set maxPRC<=0
* IOTDB-1247 Fix the insert blocked caused the bugs in mem control module
* ISSUE-2648 Last query not right when having multiple devices
* Delete mods files after compaction
* ISSUE-2687 fix insert NaN bug
* ISSUE-2598 Throw explicit exception when time series is unknown in where clause
* Fix timeseriesMetadata cache is not cleared after the TsFile is deleted by a compaction
* ISSUE-2611 An unsequence file that covers too many sequence file causes OOM query
* IOTDB-1135 Fix count timeseries bug when the paths are nested
* ISSUE-2709 IOTDB-1178 Fix cache is not cleared after compaction
* ISSUE-2746 Fix data overlapped bug after the elimination unseq compaction process
* Fix getObject method in JDBC should return an Object
* IOTDB-1188 Fix IoTDB 0.11 unable to delete data bug
* Fix when covering a tsfile resource with HistoricalVersion = null, it’ll throw a NPE
* fix the elimination unseq compaction may loss data bug after a delete operation is executed
* Fix a bug of checking time partition in DeviceTimeIndex
* Throw exeception when device to be queried is not in tsFileMetaData
* Fix unseq compaction file selector conflicts with time partition bug
* Fix high CPU usage during the compaction process

Improvements
* IOTDB-1140 optimize regular data encoding
* Add more log for better tracing
* Add backgroup exec for cli -e function
* Add max direct memory size parameter to env.sh
* Change last cache log to debug level

New Features
* Add explain sql support