Ekuiper

Overview

eKuiper 1.4.1 fixed several defects and add some properties to stabilize the product . Firstly we provided an official docker image to help portable python plugins' develop and deployment. We also did some clean up work for portable plugins runtime. Secondly we optimized SQL runtime to reduce CPU consumption. Thirdly, We support `Object` Type for EdgeX so users can define complex value type for devices. For example, user can define the device value in json format. Lastly, we cooperate closely with eKuiper-manager to make the eKuiper UI more easy use.



Features

- Support `Object` Type for EdgeX so that users can use complex value type.
- Support certification based authentication for built-in http source/sink
- Optimize SQL runtime to reduce CPU consumption

Fixes

- Avoid panic when call window function in count window
- Fix anonymous field name duplicate problem
- Return detailed error message for installation failure in portable plugin
- Backend support for UI which makes UI more user friendly


Docs

- [Update cross compile doc](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/cross-compile.md)
- [Add configuration options for stream data case sensitive optimization](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/operation/configuration_file.md)
- [Provide official docker image for portable plugin development](https://github.com/lf-edge/ekuiper/blob/master/deploy/docker/README.md#lfedgeekuipertag-slim-python)
- [Update table management doc](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sources/file.md)

Introduction

As a major release, eKuiper v1.4.0 brings many exciting features into the product. We introduce `portable` plugin system to ease the development. build and deployment of plugin in addition to the native go plugin. Users can now use both go and python to compose a plugin and the system will be extended to more languages support in the future. We enable the built-in support of flexible rule pipeline by in-memory source and sink. Additionally, we extract a standalone connection layer（mqtt and edgeX now) so that the source, sink or user's plugin can share a connection instance to avoid connection overhead, which is required to connect to secure edgeX message bus.


Features

- Add portable plugin support, user can develop plugins in multiple languages
- portable plugin [intrduction](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/portable/overview.md)
- go [sdk](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/portable/go_sdk.md)
- python [sdk](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/portable/python_sdk.md)
- plugin develop help [tool](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/portable/overview.md#debugging)
- Support source/sink to share connections for mqtt or edgeX, please check this for [more info](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sources/mqtt.md#connectionselector)
- Support secured edgeX sink by shared connection configuration since edgeX Jakarta, please check this for [more info](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/edgex/edgex_rule_engine_tutorial.md#connection-reuse)
- Add memory source and sink, please check this for [more info](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/rule_pipeline.md)
- KV storage implementation is now configurable. Provide redis as an alternative KV storage option in addition to default sqlite, please check the [doc](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/operation/configuration_file.md#store-configurations) for how to configure the storage.
- eKuiper configuration can be overridden via environment variables, please check the [doc](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/operation/configuration_file.md#basic-configurations) for more info.
- Support JWT RSA256 Token authentication for rest API
- Support dynamic properties for some sinks such as [mqtt sink dynamic topic](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sinks/mqtt.md#dynamic-topic), [rest sink dynamic properties](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sinks/rest.md#configure-dynamic-properties) etc. Also export the context function to support [dynamic properties](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/native/sink.md#parse-dynamic-properties) so that customized plugin can also use dynamic properties.
- Retry mechanism now works for all sinks


Fixes

- Mqtt source should stop executing after errors

- Update tdengine client version to 2.2.0.5

- Fix certification authentication bug in mqtt and http

- Clean up the json description files of shipped sources, sinks and functions

- Fix some SQL syntax validation issue

- Return more message for get status when error happen

- EdgeX sink support all type when meta is not specified



Docs

- [Rule pipeline](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/rule_pipeline.md)
- [Portable Plugin](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/portable/overview.md)
- [Fix the typo in plugin management rest api](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/restapi/plugins.md#drop-a-plugin)
- [Add connectionSelector property for mqtt/edgex source/sink](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sources/mqtt.md#connectionselector)
- [Correct data compatibility comparison table](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/sqls/data_types.md#compatibility-of-comparison-and-calculation)
- [Use JWT RSA256 Token authentication for rest api](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/operation/configuration_file.md#authentication)
- [Add instructions on how to use share connection feature for edgeX Jakarta release](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/edgex/edgex_rule_engine_tutorial.md#connection-reuse)

Thanks

- rwadowski provides redis kv storage
- rwadowski provides initial memory source/sink type
- rwadowski provides config eKuiper by env variables

Introduction

As a big milestone, eKuiper v1.4.0 will bring many exciting features into the product. User can start to try some of these features in eKuiper 1.4.0-beta.1. We introduce `portable` plugin system to ease the development. build and deployment of plugin in addition to the native go plugin. Users can now use go and python to compose a plugin and the system can be easily extended to more language support. We enable the built-in support of flexible rule pipeline by memory source and sink. Additionally, we extract a standalone connection layer（mqtt and edgex now) so that the source, sink or user's plugin can share a connection instance to avoid connection overhead.


Features

- KV storage implementation is now configurable. Provide redis as an alternative KV storage option in addition to default sqlite, please check the [doc](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/operation/configuration_file.md#store-configurations) for how to configure the storage.
- eKuiper configuration can be overwritten via environment variables, please check the [doc](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/operation/configuration_file.md#basic-configurations) for more info.
- Add portable plugin support, user can develop plugins in multiple languages
- portable plugin [intrduction](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/portable/overview.md)
- go [sdk](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/portable/go_sdk.md)
- python [sdk](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/portable/python_sdk.md)
- plugin develop help [tool](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/portable/overview.md#debugging)
- Support source/sink to share connections for mqtt or edgex, please check this for [more info](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sources/mqtt.md#connectionselector)
- Add memory source and sink, please check this for [more info](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/rule_pipeline.md)


Fixes

- Mqtt source should stop executing after errors
- Update tdengine client version to 2.2.0.5

Docs

- [Rule pipeline](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/rule_pipeline.md)
- [Portable Plugin](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/portable/overview.md)
- [Fix the typo in plugin management rest api](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/restapi/plugins.md#drop-a-plugin)
- [Add connectionSelector property for mqtt/edgex source/sink](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sources/mqtt.md#connectionselector)

Thanks

- rwadowski provides redis kv storage
- rwadowski provides memory source/sink type
- rwadowski provides config eKuiper by env variables

Introduction

As a big milestone, eKuiper v1.4.0 will bring many exciting features into the product. User can start to try some of these features in eKuiper 1.4.0-beta.0. We introduce `portable` plugin system to ease the development. build and deployment of plugin in addition to the native go plugin. Users can now use go and python to compose a plugin and the system can be easily extended to more language support. We enable the built-in support of flexible rule pipeline by memory source and sink. Additionally, we extract a standalone connection layer（mqtt and edgex now) so that the source, sink or user's plugin can share a connection instance to avoid connection overhead.


Features

- KV storage implementation is now configurable. Provide redis as an alternative KV storage option in addition to default sqlite, please check the [doc](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/operation/configuration_file.md#store-configurations) for how to configure the storage.
- eKuiper configuration can be overwritten via environment variables, please check the [doc](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/operation/configuration_file.md#basic-configurations) for more info.
- Add portable plugin support, user can develop plugins in multiple languages
- portable plugin [intrduction](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/portable/overview.md)
- go [sdk](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/portable/go_sdk.md)
- python [sdk](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/portable/python_sdk.md)
- plugin develop help [tool](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/portable/overview.md#debugging)
- Support source/sink to share connections for mqtt or edgex, please check this for [more info](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sources/mqtt.md#connectionselector)
- Add memory source and sink, please check this for [more info](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/rule_pipeline.md)


Fixes

- Mqtt source should stop executing after errors
- Update tdengine client version to 2.2.0.5

Docs

- [Rule pipeline](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/rule_pipeline.md)
- [Portable Plugin](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/portable/overview.md)
- [Fix the typo in plugin management rest api](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/restapi/plugins.md#drop-a-plugin)
- [Add connectionSelector property for mqtt/edgex source/sink](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sources/mqtt.md#connectionselector)

Thanks

- rwadowski provides redis kv storage
- rwadowski provides memory source/sink type
- rwadowski provides config eKuiper by env variables

Features

- Add `messageType` and `topicPrefix` property to edgex sink to support to publishing to EdgeX message bus like a device. We also update the [doc](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sinks/edgex.md) to elaborate all kinds of edgex sink usage scenario.

Fixes

- Avoid panic for some function call such as `json_path_query` when passing nil parameter
- Remove unnecessary console log prints
- [Set timestamp format property in processing time mode](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/sqls/streams.md)
- Fix redis plugin issue when using with edgex and release the right plugin
- Fix config partially missing problem in helm chart
- Graceful error message when server fails to start
- Enable strictValidation=false in preprocessor to remove requirement check for all fields
- Rest sink close body when response parsed to avoid memory leak

- SQL
- Fix inner join for multiple values issue
- Fix cross join issue when one table is null
- Fix some multiple join issues
- Add more UT for join operations
- Fix some alias problem when the alias name is the same as a json field
- Fix meta(*) does not select all problem when another meta(col) is using in other clauses like where
- Add nil check for string functions
- Add runtime check for substring function index to avoid index out of bound panic
- Fix index expression evaluation problem when the expression is a field ref and did not referred in other SQL clause
- Make window_start() and window_end() to be non-aggregate function so that they can be used in group by clause

Docs

- [Enrich edgex sink doc](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sinks/edgex.md)
- [Clear description for join/group by](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/sqls/query_language_elements.md)
- [Fix the typos in window description page](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/sqls/windows.md)
- [Update LF Edge slack workplace invitation address](https://github.com/lf-edge/ekuiper/blob/master/README.md)

Thanks

- wfnuser provides a fix

Overview

eKuiper 1.3.0 refactors the project to comply to LF edge standard including renaming the module name, project layout and adding copyright statements. We also refactor the build and CI scripts to migrate the docker images and rename the artifacts etc. Moreover, we provide a lot of new features for the SQL runtime and the EdgeX support.

Breaking Changes

- Project module, layout and package refactor. The plugins must be revised to import the new module of eKuiper and adapt to the package change.
- Move docker images lfedge organization: [lfedge/ekuiper](https://hub.docker.com/r/lfedge/ekuiper) and [lfedge/ekuiper-kubernetes-tool](https://hub.docker.com/r/lfedge/ekuiper-kubernetes-tool)
- Refactor ui meta API

Features

- Refactor alias mechanism and support more usage scenarios
- Add [window_start() and window_end()](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/sqls/built-in_functions.md#aggregate-functions) function
- Support message type property for [edgeX source](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/edgex/edgex_source_tutorial.md) to allow connecting to the message bus directly
- Support more sink [data template](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/data_template.md) function
- Support [shared instance of source](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/sqls/streams.md)
- Refactor SQL validation to refine aggregate related validation
- Support index expression for array
- Add options for eKuiper kubernetes tool to print log to different location
- Add [redis sink plugin](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/plugins/sinks/reids.md)

Fixes

- Refactor sqlite KV store to prevent errors when creating hundreds rules/streams simultaneously
- Fix influxdb plugin for multiple instances
- MQTT source: Fix resubscription failure when reconnecting
- Clean expired checkpoint memory and storage
- Clean checkpoint store when deleting rule
- Reduce the docker image size
- Close dynamic buffer in source node when source node deleted
- Fix memory leak in httppull source

Docs

- Updates to rename kuiper to eKuiper
- Update EdgeX v2 related doc
- Update EdgeX source tutorial
- Add tutorial for eKuiper and OpenYurt integration
- Update plugin devekopment tutorial
- Update plugin download path
- Update contributing instruction/issue templates/meeting info

Thanks

- tixff provides a fix for the mqtt source
- rwadowski provides a fix to refactor sqlite kv store
- feng-crazy provides redis sink plugin