Ekuiper

Overview

eKuiper 1.5.0 enriches the functions and sources/sinks to extend the capability of computing and connecting. We also take a lot of efforts to improve the stability. Finally, we launch the community website [https://ekuiper.org](https://ekuiper.org) and host the brand-new doc there.

Breaking Changes

- Mqtt source `servers` property rename to `server` and allow only one server instead of an array. When starting docker image, using `docker run -p 9081:9081 -d --name kuiper -e MQTT_SOURCE__DEFAULT__SERVER="tcp://broker.emqx.io:1883" lfedge/ekuiper:$tag`

Features

- A series of functions to detect changes and deduplicate in a universal and flexible way. Checkout the [examples](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/sqls/built-in_functions.md#functions-to-detect-changes).
- [Build selective features](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/features.md) to customize your eKuiper binary and to save disk and memory.
- Provide neuron [source](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sources/builtin/neuron.md)/[sink](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sinks/builtin/neuron.md) to consume data from neuron and control neuron.
- Handle errors including panic in rule execution to avoid affecting other rules.
- Generic SQL pull source to pull data from various databases periodically.
- Generic SQL sink to write data to various databases.
- Data communication inside rules, shared sources or memory source/sink are now running in non-blocking mode to separate workloads between rules.
- Add function object_construct to grouping the select result.
- Separate context method to parse template and jsonpath so that sink plugins can leverage them for [dynamic property parsing](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/native/develop/sink.md#parse-dynamic-properties).
- [Tdengine sink](https://github.com/lf-edge/ekuiper/tree/master/docs/en_US/rules/sinks/plugin) plugin to support sTable.


Fixes

- CLI query result now can show in multiple rows.
- Rule which does not specify option will have options like buffer size.
- Avoid message dropping in memory sink/source pair.
- Fix portable source plugin hang problem if receive timeout.
- Fix python plugin error is not shown for json dump error.
- Fix state mixing problem when invoking the same function multiple times in the same SQL clause.

Build

- Bring back Armv7 binary build
- Upgrade to go 1.17
- Sync doc to eKuiper.org


Docs

- Add concepts chapter
- Updates navigation tree
- Update doc homepage
- Separate extension development and shipped plugin
- Fix some image links and typos
- Add Sql [source](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sources/plugin/sql.md) [sink](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sinks/plugin/sql.md) description

Overview

eKuiper 1.4.4 is a maintain version that fixes several issues.

Fixes

- Fix edgex source can not subscribe same topics for multiple rules when using share connection problem.
- Fix portable source plugin receive message hang problem .
- Fix tdengine plugin crash problem.
- Let eKuiper home path in same place for all docker images

Overview

eKuiper 1.4.3 is a maintenance version that fixes several issues.

Fixes

- Fix edgex source can not subscribe same topics for multiple rules when using share connection problem.
- Fix edgex source can not connect to mqtt broker problem.
- Fix mqtt source can not subscribe same topics for multiple rules when using share connection problem.
- Fix rule hang problem when python plugin has exited.
- Fix eKuiper can not run in armv7 problem.
- Fix tdengine plugin crash problem.
- Fix omitEmpty can not take effect when sendSingle is set problem.
- Return error when replace stream sql does not match name

Docs

- [fix typos in edgex meta tutorial](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/edgex/edgex_meta.md)
- [Add split_values in doc](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/sqls/built-in_functions.md)

Overview

eKuiper 1.4.2 is a minor version that fixes several issues.


Fixes

- Fix a portable plugin function communication disorder problem

- Cache isAggregate call result for portable plugin functions

- Fix `parseDynamicProp` syntax conflict with mqtt topic that start with `$` problem




Docs

- [Correct docs for sink retry count](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/overview.md)
- [Update dynamic properties syntax from json path to go data template](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/overview.md#dynamic-properties)
- [Update docs layout](https://github.com/lf-edge/ekuiper/tree/master/docs/en_US)

Overview

eKuiper 1.4.1 fixed several defects and add some properties to stabilize the product . Firstly we provided an official docker image to help portable python plugins' develop and deployment. We also did some clean up work for portable plugins runtime. Secondly we optimized SQL runtime to reduce CPU consumption. Thirdly, We support `Object` Type for EdgeX so users can define complex value type for devices. For example, user can define the device value in json format. Lastly, we cooperate closely with eKuiper-manager to make the eKuiper UI more easy use.



Features

- Support `Object` Type for EdgeX so that users can use complex value type.
- Support certification based authentication for built-in http source/sink
- Optimize SQL runtime to reduce CPU consumption

Fixes

- Avoid panic when call window function in count window
- Fix anonymous field name duplicate problem
- Return detailed error message for installation failure in portable plugin
- Backend support for UI which makes UI more user friendly


Docs

- [Update cross compile doc](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/cross-compile.md)
- [Add configuration options for stream data case sensitive optimization](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/operation/configuration_file.md)
- [Provide official docker image for portable plugin development](https://github.com/lf-edge/ekuiper/blob/master/deploy/docker/README.md#lfedgeekuipertag-slim-python)
- [Update table management doc](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sources/file.md)

Introduction

As a major release, eKuiper v1.4.0 brings many exciting features into the product. We introduce `portable` plugin system to ease the development. build and deployment of plugin in addition to the native go plugin. Users can now use both go and python to compose a plugin and the system will be extended to more languages support in the future. We enable the built-in support of flexible rule pipeline by in-memory source and sink. Additionally, we extract a standalone connection layer（mqtt and edgeX now) so that the source, sink or user's plugin can share a connection instance to avoid connection overhead, which is required to connect to secure edgeX message bus.


Features

- Add portable plugin support, user can develop plugins in multiple languages
- portable plugin [intrduction](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/portable/overview.md)
- go [sdk](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/portable/go_sdk.md)
- python [sdk](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/portable/python_sdk.md)
- plugin develop help [tool](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/portable/overview.md#debugging)
- Support source/sink to share connections for mqtt or edgeX, please check this for [more info](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sources/mqtt.md#connectionselector)
- Support secured edgeX sink by shared connection configuration since edgeX Jakarta, please check this for [more info](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/edgex/edgex_rule_engine_tutorial.md#connection-reuse)
- Add memory source and sink, please check this for [more info](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/rule_pipeline.md)
- KV storage implementation is now configurable. Provide redis as an alternative KV storage option in addition to default sqlite, please check the [doc](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/operation/configuration_file.md#store-configurations) for how to configure the storage.
- eKuiper configuration can be overridden via environment variables, please check the [doc](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/operation/configuration_file.md#basic-configurations) for more info.
- Support JWT RSA256 Token authentication for rest API
- Support dynamic properties for some sinks such as [mqtt sink dynamic topic](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sinks/mqtt.md#dynamic-topic), [rest sink dynamic properties](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sinks/rest.md#configure-dynamic-properties) etc. Also export the context function to support [dynamic properties](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/native/sink.md#parse-dynamic-properties) so that customized plugin can also use dynamic properties.
- Retry mechanism now works for all sinks


Fixes

- Mqtt source should stop executing after errors

- Update tdengine client version to 2.2.0.5

- Fix certification authentication bug in mqtt and http

- Clean up the json description files of shipped sources, sinks and functions

- Fix some SQL syntax validation issue

- Return more message for get status when error happen

- EdgeX sink support all type when meta is not specified



Docs

- [Rule pipeline](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/rule_pipeline.md)
- [Portable Plugin](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/portable/overview.md)
- [Fix the typo in plugin management rest api](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/restapi/plugins.md#drop-a-plugin)
- [Add connectionSelector property for mqtt/edgex source/sink](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sources/mqtt.md#connectionselector)
- [Correct data compatibility comparison table](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/sqls/data_types.md#compatibility-of-comparison-and-calculation)
- [Use JWT RSA256 Token authentication for rest api](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/operation/configuration_file.md#authentication)
- [Add instructions on how to use share connection feature for edgeX Jakarta release](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/edgex/edgex_rule_engine_tutorial.md#connection-reuse)

Thanks

- rwadowski provides redis kv storage
- rwadowski provides initial memory source/sink type
- rwadowski provides config eKuiper by env variables