Ekuiper

Introduction

As a big milestone, eKuiper v1.4.0 will bring many exciting features into the product. User can start to try some of these features in eKuiper 1.4.0-beta.1. We introduce `portable` plugin system to ease the development. build and deployment of plugin in addition to the native go plugin. Users can now use go and python to compose a plugin and the system can be easily extended to more language support. We enable the built-in support of flexible rule pipeline by memory source and sink. Additionally, we extract a standalone connection layer（mqtt and edgex now) so that the source, sink or user's plugin can share a connection instance to avoid connection overhead.


Features

- KV storage implementation is now configurable. Provide redis as an alternative KV storage option in addition to default sqlite, please check the [doc](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/operation/configuration_file.md#store-configurations) for how to configure the storage.
- eKuiper configuration can be overwritten via environment variables, please check the [doc](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/operation/configuration_file.md#basic-configurations) for more info.
- Add portable plugin support, user can develop plugins in multiple languages
- portable plugin [intrduction](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/portable/overview.md)
- go [sdk](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/portable/go_sdk.md)
- python [sdk](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/portable/python_sdk.md)
- plugin develop help [tool](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/portable/overview.md#debugging)
- Support source/sink to share connections for mqtt or edgex, please check this for [more info](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sources/mqtt.md#connectionselector)
- Add memory source and sink, please check this for [more info](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/rule_pipeline.md)


Fixes

- Mqtt source should stop executing after errors
- Update tdengine client version to 2.2.0.5

Docs

- [Rule pipeline](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/rule_pipeline.md)
- [Portable Plugin](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/portable/overview.md)
- [Fix the typo in plugin management rest api](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/restapi/plugins.md#drop-a-plugin)
- [Add connectionSelector property for mqtt/edgex source/sink](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sources/mqtt.md#connectionselector)

Thanks

- rwadowski provides redis kv storage
- rwadowski provides memory source/sink type
- rwadowski provides config eKuiper by env variables

Introduction

As a big milestone, eKuiper v1.4.0 will bring many exciting features into the product. User can start to try some of these features in eKuiper 1.4.0-beta.0. We introduce `portable` plugin system to ease the development. build and deployment of plugin in addition to the native go plugin. Users can now use go and python to compose a plugin and the system can be easily extended to more language support. We enable the built-in support of flexible rule pipeline by memory source and sink. Additionally, we extract a standalone connection layer（mqtt and edgex now) so that the source, sink or user's plugin can share a connection instance to avoid connection overhead.


Features

- KV storage implementation is now configurable. Provide redis as an alternative KV storage option in addition to default sqlite, please check the [doc](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/operation/configuration_file.md#store-configurations) for how to configure the storage.
- eKuiper configuration can be overwritten via environment variables, please check the [doc](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/operation/configuration_file.md#basic-configurations) for more info.
- Add portable plugin support, user can develop plugins in multiple languages
- portable plugin [intrduction](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/portable/overview.md)
- go [sdk](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/portable/go_sdk.md)
- python [sdk](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/portable/python_sdk.md)
- plugin develop help [tool](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/portable/overview.md#debugging)
- Support source/sink to share connections for mqtt or edgex, please check this for [more info](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sources/mqtt.md#connectionselector)
- Add memory source and sink, please check this for [more info](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/rule_pipeline.md)


Fixes

- Mqtt source should stop executing after errors
- Update tdengine client version to 2.2.0.5

Docs

- [Rule pipeline](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/rule_pipeline.md)
- [Portable Plugin](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/extension/portable/overview.md)
- [Fix the typo in plugin management rest api](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/restapi/plugins.md#drop-a-plugin)
- [Add connectionSelector property for mqtt/edgex source/sink](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sources/mqtt.md#connectionselector)

Thanks

- rwadowski provides redis kv storage
- rwadowski provides memory source/sink type
- rwadowski provides config eKuiper by env variables

Features

- Add `messageType` and `topicPrefix` property to edgex sink to support to publishing to EdgeX message bus like a device. We also update the [doc](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sinks/edgex.md) to elaborate all kinds of edgex sink usage scenario.

Fixes

- Avoid panic for some function call such as `json_path_query` when passing nil parameter
- Remove unnecessary console log prints
- [Set timestamp format property in processing time mode](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/sqls/streams.md)
- Fix redis plugin issue when using with edgex and release the right plugin
- Fix config partially missing problem in helm chart
- Graceful error message when server fails to start
- Enable strictValidation=false in preprocessor to remove requirement check for all fields
- Rest sink close body when response parsed to avoid memory leak

- SQL
- Fix inner join for multiple values issue
- Fix cross join issue when one table is null
- Fix some multiple join issues
- Add more UT for join operations
- Fix some alias problem when the alias name is the same as a json field
- Fix meta(*) does not select all problem when another meta(col) is using in other clauses like where
- Add nil check for string functions
- Add runtime check for substring function index to avoid index out of bound panic
- Fix index expression evaluation problem when the expression is a field ref and did not referred in other SQL clause
- Make window_start() and window_end() to be non-aggregate function so that they can be used in group by clause

Docs

- [Enrich edgex sink doc](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sinks/edgex.md)
- [Clear description for join/group by](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/sqls/query_language_elements.md)
- [Fix the typos in window description page](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/sqls/windows.md)
- [Update LF Edge slack workplace invitation address](https://github.com/lf-edge/ekuiper/blob/master/README.md)

Thanks

- wfnuser provides a fix

Overview

eKuiper 1.3.0 refactors the project to comply to LF edge standard including renaming the module name, project layout and adding copyright statements. We also refactor the build and CI scripts to migrate the docker images and rename the artifacts etc. Moreover, we provide a lot of new features for the SQL runtime and the EdgeX support.

Breaking Changes

- Project module, layout and package refactor. The plugins must be revised to import the new module of eKuiper and adapt to the package change.
- Move docker images lfedge organization: [lfedge/ekuiper](https://hub.docker.com/r/lfedge/ekuiper) and [lfedge/ekuiper-kubernetes-tool](https://hub.docker.com/r/lfedge/ekuiper-kubernetes-tool)
- Refactor ui meta API

Features

- Refactor alias mechanism and support more usage scenarios
- Add [window_start() and window_end()](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/sqls/built-in_functions.md#aggregate-functions) function
- Support message type property for [edgeX source](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/edgex/edgex_source_tutorial.md) to allow connecting to the message bus directly
- Support more sink [data template](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/data_template.md) function
- Support [shared instance of source](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/sqls/streams.md)
- Refactor SQL validation to refine aggregate related validation
- Support index expression for array
- Add options for eKuiper kubernetes tool to print log to different location
- Add [redis sink plugin](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/plugins/sinks/reids.md)

Fixes

- Refactor sqlite KV store to prevent errors when creating hundreds rules/streams simultaneously
- Fix influxdb plugin for multiple instances
- MQTT source: Fix resubscription failure when reconnecting
- Clean expired checkpoint memory and storage
- Clean checkpoint store when deleting rule
- Reduce the docker image size
- Close dynamic buffer in source node when source node deleted
- Fix memory leak in httppull source

Docs

- Updates to rename kuiper to eKuiper
- Update EdgeX v2 related doc
- Update EdgeX source tutorial
- Add tutorial for eKuiper and OpenYurt integration
- Update plugin devekopment tutorial
- Update plugin download path
- Update contributing instruction/issue templates/meeting info

Thanks

- tixff provides a fix for the mqtt source
- rwadowski provides a fix to refactor sqlite kv store
- feng-crazy provides redis sink plugin

Overview

eKuiper 1.3.0 refactors the project to comply to LF edge standard including renaming the module name, project layout and adding copyright statements. We also refactore the build and CI scripts to migrate the docker images and rename the artifacts etc. We also provide a lot of new features for the SQL runtime and the EdgeX support.

Breaking Changes

- Project module, layout and package refactor. The plugins must be revised to import the new module of eKuiper and adapt to the package change.
- Refactor ui meta API

Features

- Refactor alias mechanism and support more usage scenarios
- Add window_start() and window_end() function
- Support message type property for edgeX source to allow connect to message bus directly
- Support more sink data template function
- Support shared instance of source
- Refactor SQL validation to refine aggregate related validation
- Add copyright statement
- Rename docker image

Fixes

- Fix influxdb plugin for multiple instances
- MQTT source: Fix resubscription failure when reconnecting
- Clean expired checkpoint memory and storage
- Clean checkpoint store when deleting rule
- Reduce the docker image size

Docs

- Update to rename kuiper to eKuiper
- Update EdgeX v2 related doc
- Add tutorial for eKuiper and OpenYurt integration

Thanks

- wb.lixinlin provides a fix for the mqtt source

Overview

eKuiper 1.2.1 supports EdgeX Ireland and adapts to EdgeX V2 API. This brings some [breaking changes](https://github.com/lf-edge/ekuiper/blob/1.2.1/docs/en_US/edgex/edgex_rule_engine_tutorial.md#migrate-to-edgex-v2). Especially, the commonly used meta `Device` has now renamed to `DeviceName`, rules that refer to the meta must be updated with the new name. Additionally, EdgeX binary readings are now supported to read into eKuiper stream. And Kuiper byte data can also be sinked to EdgeX binary reading.

Since the project has joined LF Edge and renamed to eKuiper, we also update the documentations to adopt the new name.

Features

- EdgeXSource and EdgeXSink support EdgeX V2 API
- EdgeXSource and EdgeXSink support EdgeX binary readings

Fixes

- Fix parse error for subtraction operator in some condition
- Allow short-circuit for logical operation
- Fix [the join calculation problem](https://github.com/lf-edge/ekuiper/issues/817) when joined streams have multiple tuples
- Fix sink meta API problem when a rule has multiple sinks
- Build
- Refactor plugins and tools to submodules
- Error Message Fixes
- Refine error messages for get rules API
- Do not print nil error when closing rule normally
- Documentation fixes
- Rename Kuiper to eKuiper
- Fix incorrect port in CLI/REST examples
- Fix incorrect translation in the Chinese version windows doc

Thanks

- ccb1900 provides a fix for the documentation