Thingsboard-gateway

Minor release of ThingsBoard IoT gateway. Contains usability improvements and bug fixes.

What's changed:

**Gateway service:**

- Added data duplicate detector
- Added statistics files examples for different OS
- Added `minEventsProcess` parameter
- Added Gateway Shell
- Added device filtering
- Fixed reconnecting
- Added device provisioning
- Added converter remote configuration update
- General Improvements
​

**Docker Image:**
​
- General Improvements
​

**MQTT Connector:**
​
- Added `sendDataOnlyOnChange` parameter
- Added shared subscriptions support
- General Improvements
​

**Modbus Connector:**
- Added TLS connection support
- General Improvements
​

**BLE Connector:**
- Added advertising mode support
- General Improvements
​

**OPC-UA AsyncIO Connector:**
- General Improvements
​

**REST Connector:**
- General Improvements
​

**FTP Connector:**
- General Improvements
​

**SNMP Connetor:**
- General Improvements
​

**Request Connector:**
- General Improvements
​

**OCPP Connector:**
- General Improvements


List of pull requests:
* Fixed getting converter type by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/970
* Fixed Dockerfile by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/971
* Fixed getting custom converter by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/972
* Added converter config error for MQTT Connector by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/974
* Updated GRPC Connectors by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/976
* Fixed grpcio version in requirements.txt by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/979
* Added sendDataOnlyOnChange parameter for MQTT Connector by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/991
* Fixed data-sending delay by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/994
* Fixed data misplaced for OPC-UA AsyncIO Connector by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/996
* Added shared subscriptions support for MQTT Connector by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/997
* Added statistics file examples for different OS by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1001
* Change ERROR to WARNING in statistics service log by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1002
* Updated SNMP Connector by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1003
* Fixed RPC for Request Connector by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1004
* Added parameter minEventsProcess by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1005
* Updated commands in linux statistics file by ahmed-0011 in https://github.com/thingsboard/thingsboard-gateway/pull/1008
* fix statistics service thread is blocking on rasperberry pi by xunyuw in https://github.com/thingsboard/thingsboard-gateway/pull/1009
* [Core] Data duplicate detector feature. by devaskim in https://github.com/thingsboard/thingsboard-gateway/pull/1000
* Deleted requirements.txt from docker folder by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1011
* Revert code for fix rasperberry PI, block Statistics function work by xunyuw in https://github.com/thingsboard/thingsboard-gateway/pull/1026
* [Core] Support of TTL in duplicate filtering. by devaskim in https://github.com/thingsboard/thingsboard-gateway/pull/1019
* Duplicate Detector: Configuration parameter mistype by devaskim in https://github.com/thingsboard/thingsboard-gateway/pull/1027
* Added responseValueExpression RPC param for Request Connector by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1031
* Fixed on_attributes_update for OPC-UA AsyncIO Connector by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1034
* Fixed writing multiple registers for Modbus Connector by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1035
* Fixed reading boolean for Modbus Connector by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1036
* Updated modbus.json file by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1038
* Added device filtering by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1039
* Fixed version resolving by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1040
* [WIP] Added Gateway Shell by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1017
* Added entrypoint for Gateway Shell by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1047
* Added BLE Advertising mode support by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1049
* Fixed modbus_serial.json config file by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1053
* Fixed imports in OCPP Connector by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1057
* Fixed starting Gateway with incorrectly closed manager by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1058
* Fixed modbus reconnecting by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1062
* Fixed Shell connection for Windows by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1067
* Added additional error log for Modbus Connector by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1069
* Request connector bug by Kevin-ox in https://github.com/thingsboard/thingsboard-gateway/pull/1072
* Fixed TB Gateway Service reconnect by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1076
* Updated submodule by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1078
* Updated requires by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1081
* Updated submodule by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1082
* Added TLS support for Modbus TCP Connection by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1083
* Added Gateway Provisioning by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1084
* Updated submodule by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1085
* Added converter update from shared attr for Connectors by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1087
* Updated generate_certificate method by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1088
* Added on demand caching of custom MQTT converters. by devaskim in https://github.com/thingsboard/thingsboard-gateway/pull/1090
* Load .pyc modules by CoderPuppy in https://github.com/thingsboard/thingsboard-gateway/pull/1086
* Changed queries using for REST Connector by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1095
* Fixed process_remote_converter_configuration_update method by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1101
* Get rid of warning while setting connector name. by devaskim in https://github.com/thingsboard/thingsboard-gateway/pull/1100
* [FEATURE] Add sharing id instead of global converter cache for MQTT connector. by devaskim in https://github.com/thingsboard/thingsboard-gateway/pull/1099
* Updated converter update request for MQTT Connector by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1104
* Fixed FTP Converter by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1107
* Fixed statistics sending period by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1109
* Fixed KeyError for some edge cases. by devaskim in https://github.com/thingsboard/thingsboard-gateway/pull/1111
* Changed the default converters to use local instead of class variables by mircopz in https://github.com/thingsboard/thingsboard-gateway/pull/1110
* Added synchronizing method for converters config by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1112
* Fix a race condition in the Sqlite storage which caused all data to stop by CoderPuppy in https://github.com/thingsboard/thingsboard-gateway/pull/1114
* Remove sleep per data point pushed by CoderPuppy in https://github.com/thingsboard/thingsboard-gateway/pull/1113
* bug tb_gateway_service.py -> __form_statistics by Kevin-ox in https://github.com/thingsboard/thingsboard-gateway/pull/1122
* Updated tests by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1132
* Fixed modbus tests by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1133
* Updated submodule by samson0v in https://github.com/thingsboard/thingsboard-gateway/pull/1134

New Contributors
* ahmed-0011 made their first contribution in https://github.com/thingsboard/thingsboard-gateway/pull/1008
* xunyuw made their first contribution in https://github.com/thingsboard/thingsboard-gateway/pull/1009
* Kevin-ox made their first contribution in https://github.com/thingsboard/thingsboard-gateway/pull/1072
* CoderPuppy made their first contribution in https://github.com/thingsboard/thingsboard-gateway/pull/1086

**Full Changelog**: https://github.com/thingsboard/thingsboard-gateway/compare/v3.2...3.3

**Gateway:**
​
- Added configurable statistics collecting
- Added ⚡️ HotReloader for dev mode
- Fixed and optimized data packing algorithm
- Added certificates end date checking and generation new one
- SDK as a dependency for Gateway
- Added ability to connect to ThingsBoard via the username and password
- General Improvements
​

**MQTT Connector:**
​
- Added default bytes data converter
- Added MQTT 5 support
- Added `get/set` RPC methods
- General Improvements
​

**Modbus Connector:**

- Updated version of the used Modbus library to version >=3.0.0.
(Installation required, after installation gateway restart required)
- Added `get/set` RPC methods
- General Improvements
​

**Socket Connector:**
​
- Added `set` RPC method
​

**OPC-UA AsyncIO Connector:**

- Bug fixes
- General Improvements
​

**REST Connector:**
​
- Added `get/set` RPC methods
​

**OPC-UA Connector:**
​
- General Improvements
​

**SNMP Connector:**
​
- General Improvements
​

**ODBC Connector:**
​
- General Improvements
​

**GRPC Service:**
​
- General Improvements
​

**Docker Image:**
​
- Gateway configuration via the env variables
- Fix for logs location folder
- Fixed and optimized
​

**[NEW] XMPP Connector**
​

**[NEW] OCPP Connector**
​

**[NEW] GRPC MQTT Connector**
​

**[NEW] GRPC Modbus Connector**
​

**[NEW] GRPC OPCUA Connector**
​

**Contributors:**
​
- devascim
- JavierNR
- KevinPoole
- zhufeng88
- anirudh-mehta
- micropz
- nagarjunredla
- wilterdinkrobert
- bcl-dev
- Maksym2022

**Gateway:**

- Added functionality for checking devices activity
- General Improvements
- Added new statistic service, it may configured like regular connector by configuration file

**Remote Configurator:**

- General Improvements

**All Type of Connectors:**

- From now users can add the `useGRPC` flag for on/off the default GRPC connector realization (using GRPC allows you to get real multiprocessing for the gateway and gives more hardware resources to the gateway)

**MQTT Connector:**

- General Improvements
- From now connector won't pass the None value from the converter

**BLE Connector:**

- Improved stability
- General Improvements

**Modbus Connector:**

- General Improvements
- Improved Modbus Compatibility Adapter
- Added tests

**OPC-UA Connector:**

- From now connector will use node name if key not found in config file
- Perfomance Improvements
- General Improvements

**REST Connector:**

- Added `Remote Response` and `Custom Response`
- Added opportunity to process query params
- Added opportunity to receive different data types
- Added `Attribute Request`
- General Improvements

**Request Connector:**

- General Improvements

**GRPC:**

- Perfomance Improvements
- General Improvements

**Docker Image:**

- Added support for ARM v.7

**[New] Socket Connector**

- Added `Attribute Request`

**[New] GRPC Socket Connector**

**[New] OPC-UA AsyncIO connector**
Designed to have better performance and stay updated with actual

**Contributors:**
[ysimonx](https://github.com/thingsboard/thingsboard-gateway/commits?author=ysimonx),
[jeffreylauck](https://github.com/thingsboard/thingsboard-gateway/commits?author=jeffreylauck),
[wilterdinkrobert](https://github.com/thingsboard/thingsboard-gateway/commits?author=wilterdinkrobert),
[VeryBitter](https://github.com/thingsboard/thingsboard-gateway/commits?author=VeryBitter),
[ferguscan](https://github.com/thingsboard/thingsboard-gateway/commits?author=ferguscan),
[Liangzx](https://github.com/thingsboard/thingsboard-gateway/commits?author=Liangzx),
[Kretiss](https://github.com/thingsboard/thingsboard-gateway/commits?author=Kretiss),
[DavideBorsatti](https://github.com/thingsboard/thingsboard-gateway/commits?author=DavideBorsatti).

**Full Changelog**: https://github.com/thingsboard/thingsboard-gateway/compare/3.0.1...3.1

Minor release for ThingsBoard IoT gateway.

Contains hotfix for docker image.

All changes are the same like in [release 3.0](https://github.com/thingsboard/thingsboard-gateway/releases/tag/3.0).

**Major release of ThingsBoard IoT gateway.**

In this release:

**Features**:

- [_Service_] GRPC API - ThingsBoard IoT gateway will start GRPC server and listen for client connections, clients can work as a connectors. Using this feature you can create your own connectors on any programming language.
- [_Modbus Connector_] Refactored Modbus Connector. Added ability to use it as a Modbus slave and Modbus master device. Improved downlink converter.
- [_Service_] Added ability to handle device removing/renaming on Thingsboard (Works with ThingsBoard 3.3.3 or above).
- [_OPC-UA Connector_] Added service RPC methods get and set for nodes on OPC-UA server.
- [_BLE Connector_] Remastered. Changed library for connecting and processing.
- [_SQLite Storage_] Improvements and added additional configuration parameters.
- [_Configuration wizard_] Added additional configuration parameters.

**Fixes**:
- [_Service_] Fix for reconnecting on sending big message.
- [_Service_] Fixed high CPU usage by connectors.
- [_Service_] Changed logic for persistent device saving, to keep device type.
- [_REST Connector_] Fixed aiohttp loop error.

**Gateway:**

- Fixed reconnection Gateway to ThingsBoard bug
- Added minPackSendDelayMS to config file
- Fixed certificate loading
- Fixed remote logging feature
- Fix for RPC calls to connectors. Thanks mjbernet3

**MQTT Connector:**

- Added JSONPath parsing to RPC `valueExpression` config section
- Added `attributeRequests` config section
- Added `retain` flag for attributeUpdates, attributeRequests
- Added opportunity to combine values in attributes, telemetry and serverSideRpc config sections

**Modbus Connector:**

Added configuration parameters:

- retries - Count of retries if polling was failed (default 3)
- retryOnError - Retry to get data if server returns an error (default true)
- retryOnInvalid - Retry to get data if server returns invalid data (default true)

**REST Connector:**

- Added SSL Support
- Added JSONPath parsing to RPC `valueExpression` config section
- Added opportunity to combine values in attributes, telemetry and serverSideRpc config sections

**BACNet Connector:**

- General improvements

**Request Connector:**

- General improvements
- Added JSONPath parsing to RPC `valueExpression` config section
- Added opportunity to combine values in attributes, telemetry and serverSideRpc config sections

**ODBC Connector:**

- Bugs fixed (adding timestamp to telemetry)

**FTP Connector:**

- Added opportunity to combine values in attributes, telemetry config section for JSON files