Ekuiper

Overview

eKuiper 1.7.0 continue to improve the expressiveness of the rules by providing lookup table, updatable sink, more SQL syntax and functions. We also enrich the sources/sinks and adapt them to the new runtimes. Moreover, we improve the usability by refactoring the connection configure APIs to unify the configuration of source/sink and do connection testing when editing the configurations. We have made the etc folder readonly so that it is easier to deploy eKuiper by Kubernetes. At the same time, ruleset can import/export now and provision at startup to help migration of eKuiper instance. Please check the full change list below.

Features

SQL & Runtime
* [Lookup table](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/concepts/sources/table.md#lookup-table) runtime and API by ngjaying
* [Updatable sink](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sinks/overview.md#updatable-sink) runtime by ngjaying
* Support partition for [analytic functions](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/sqls/built-in_functions.md#analytic-functions) by ngjaying in https://github.com/lf-edge/ekuiper/pull/1432
* Add [latest function](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/sqls/built-in_functions.md#analytic-functions) by ngjaying in https://github.com/lf-edge/ekuiper/pull/1434
* SQL support wildcard with other selects by ngjaying in https://github.com/lf-edge/ekuiper/pull/1436

Source/Sink
* [Httppush source](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sources/builtin/http_push.md) by ngjaying in https://github.com/lf-edge/ekuiper/pull/1386
* [Influx V2 sink](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sinks/plugin/influx2.md) by elpsyr in https://github.com/lf-edge/ekuiper/pull/1396
* Add support for edgex source application/cbor format by superrxan in https://github.com/lf-edge/ekuiper/pull/1397
* Add [memory](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sources/builtin/memory.md), [redis](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sources/builtin/redis.md) and [SQL](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sources/plugin/sql.md) lookup source by ngjaying
* Make [memory](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sinks/builtin/memory.md), [redis](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sinks/builtin/redis.md) and [SQL](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sinks/plugin/sql.md) sink updatable by ngjaying
* Move [redis sink](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sinks/builtin/redis.md) to built-in by ngjaying

Operation
* API to import and export stream/rule set by ngjaying in https://github.com/lf-edge/ekuiper/pull/1438
* Provision ruleset at first startup by ngjaying in https://github.com/lf-edge/ekuiper/pull/1439
* Support sink config by resource id by superrxan in https://github.com/lf-edge/ekuiper/pull/1442

Improvements
* Move plugin meta data out of etc folder by superrxan in https://github.com/lf-edge/ekuiper/pull/1392
* Make etc files readonly for API by superrxan in https://github.com/lf-edge/ekuiper/pull/1427
* Graph API updates by ngjaying in https://github.com/lf-edge/ekuiper/pull/1393
* Refactor portable plugin test by ngjaying in https://github.com/lf-edge/ekuiper/pull/1408
* Refactor: move from io/ioutil to io and os packages by Juneezee in https://github.com/lf-edge/ekuiper/pull/1412
* Refactor planner to run analytic functions at the beginning by ngjaying in https://github.com/lf-edge/ekuiper/pull/1430
* Fix function/operator meta data by superrxan in https://github.com/lf-edge/ekuiper/pull/1437
* Allow to use the same port for REST and prometheus by ngjaying in https://github.com/lf-edge/ekuiper/pull/1448

Fixes
* Fix sink cache problems by ngjaying in https://github.com/lf-edge/ekuiper/pull/1384
* Influx sink to check whether dataTemplate is an empty field by elpsyr in https://github.com/lf-edge/ekuiper/pull/1394
* Fix sink image plugin config error by superrxan in https://github.com/lf-edge/ekuiper/pull/1398
* Fix message map to avoid concurrent read/write by ngjaying in https://github.com/lf-edge/ekuiper/pull/1407
* Fix show plugin fail in standard build tag by ngjaying in https://github.com/lf-edge/ekuiper/pull/1453
* Do not recover rule that has error by superrxan in https://github.com/lf-edge/ekuiper/pull/1454

Docs
* Edgex Source doc fix by ngjaying in https://github.com/lf-edge/ekuiper/pull/1383
* Monitor with prometheus by ngjaying in https://github.com/lf-edge/ekuiper/pull/1387
* Tutorial to use python plugin by ngjaying in https://github.com/lf-edge/ekuiper/pull/1400
* Fix docs search by Swilder-M in https://github.com/lf-edge/ekuiper/pull/1431
* Doc for lookup table and updatable sink by ngjaying in https://github.com/lf-edge/ekuiper/pull/1428
* Add quick start for k8s tool by ngjaying in https://github.com/lf-edge/ekuiper/pull/1444
* Fix and add example of environment variables by MonicaisHer in https://github.com/lf-edge/ekuiper/pull/1447
* Add assets cdn by Swilder-M in https://github.com/lf-edge/ekuiper/pull/1446

New Contributors
* elpsyr made their first contribution in https://github.com/lf-edge/ekuiper/pull/1394
* Juneezee made their first contribution in https://github.com/lf-edge/ekuiper/pull/1412

**Full Changelog**: https://github.com/lf-edge/ekuiper/compare/1.6.1...1.7.0

Overview

eKuiper 1.6.3 is a fixpack which contains one fix.

Fixes
* Fix possible concurrent map read/write crash problem for long running rule with alias

Overview
eKuiper 1.6.2 is a maintainance version that fixes several issues and add some properties to stabilize the product.
Most importantly, fix the concurrent read/write of message map to avoid crash. Secondly, avoid print warning log when native plugin not found in native manager. Lastly, update the docs for edgex/prometheus/python plugin use case

Fixes
* fix(doc): edgex source doc fix by ngjaying in https://github.com/lf-edge/ekuiper/pull/1383
* Fix sink cache problems by ngjaying in https://github.com/lf-edge/ekuiper/pull/1384
* doc(op): monitor with prometheus by ngjaying in https://github.com/lf-edge/ekuiper/pull/1387
* fix(plugin): fix sink image plugin config error by superrxan in https://github.com/lf-edge/ekuiper/pull/1398
* doc(plugin): tutorial to use python plugin by ngjaying in https://github.com/lf-edge/ekuiper/pull/1400
* fix(sql): should not change message map to avoid concurrent read/write by ngjaying in https://github.com/lf-edge/ekuiper/pull/1407
* test(portable): refactor portable plugin test by ngjaying in https://github.com/lf-edge/ekuiper/pull/1408
* opt(buffer): Optimize dynamic buffer and add unit test by ngjaying in https://github.com/lf-edge/ekuiper/pull/1416
* fix(portable): need to close process when deleting plugin by ngjaying in https://github.com/lf-edge/ekuiper/pull/1414
* [fix(bug): Remove warnings in native plugin manager by superrxan in https://github.com/lf-edge/ekuiper/pull/1418

**Full Changelog**: https://github.com/lf-edge/ekuiper/compare/1.6.1...1.6.2

Overview

eKuiper 1.6.1 is a maintainance version that fixes several issues and add some properties to stabilize the product.
Most importantly, fix the instability connection issue with neuron, now neuron do not need to start before eKuiper.
For eKuiper cannot run directly in old linux distribution(ubuntu-18.04 centos-7) because of glibc compatibility issue,
now build eKuiper using old glibc. For the prebuilt native plugins, now they can run in alpine docker images. For documents, provide
protobuf tutorial, upload management api and correct outdated message.

Fixes

- Fix connecting to NanoMQ issue and use MQTT 3.1.1(4) when connect to MQTT broker by default ngjaying
- Fix the invalid sink conf ngjaying
- Use async connection with neuron ngjaying
- Avoid chunked encoding for rest response superrxan
- Add lock to toMap ngjaying
- Remove strict check for plugin download superrxan
- Fix restart may fail because the previous detach has not done issue ngjaying
- Cache the Symbols when eKuiper starts up and plugins register superrxan
- Fix the prebuilt plugins can not run on official build eKuiper binary issue superrxan
- Fix the official build eKuiper binary can not run on lower linux distribution issue superrxan

Feature

- Add alpine base build image and build alpine prebuilt plugins with it
- Connection exception management

Docs

- Provide protobuf [tutorial](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/tutorials/usage/protobuf_tutorial.md)
- Provide upload management api [document](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/operation/restapi/uploads.md)
- Correct broken address reference
- Update go version to 1.18 in readme.md


New Contributors
* sixhj made their first contribution in https://github.com/lf-edge/ekuiper/pull/1350
* guowei-gong made their first contribution in https://github.com/lf-edge/ekuiper/pull/1363
* MonicaisHer made their first contribution in https://github.com/lf-edge/ekuiper/pull/1375

**Full Changelog**: https://github.com/lf-edge/ekuiper/compare/1.6.0...1.6.1

Overview

eKuiper 1.6.0 provides graph API as an alternative way to create rule in addition to SQL. It is friendly as the backend for a drag and drop UI and also more expressive then SQL. Beside this, we also highlight on enhancing SQL by adding more expressions and functions; on enhancing external connection by supporting protobuf format and enhancing various source/sink.

New Features

- [Schema registry for protobuf](https://github.com/lf-edge/ekuiper/blob/1.6.0/docs/en_US/rules/codecs.md#schema) by ngjaying (https://github.com/lf-edge/ekuiper/pull/1271)
- [Support protobuf as source/sink data format](https://github.com/lf-edge/ekuiper/blob/1.6.0/docs/en_US/rules/codecs.md#format) by ngjaying (https://github.com/lf-edge/ekuiper/pull/1274)
- [Config file upload REST api](https://github.com/lf-edge/ekuiper/blob/1.6.0/docs/en_US/operation/restapi/uploads.md) by ngjaying (https://github.com/lf-edge/ekuiper/pull/1302)
- Support SQL [in expression](https://github.com/lf-edge/ekuiper/blob/1.6.0/docs/en_US/sqls/query_language_elements.md#where) by ngjaying (https://github.com/lf-edge/ekuiper/pull/1303)
- [Rule graph API](https://github.com/lf-edge/ekuiper/blob/1.6.0/docs/en_US/rules/graph_rule.md) by ngjaying (https://github.com/lf-edge/ekuiper/pull/1317)
- [Lag function](https://github.com/lf-edge/ekuiper/blob/1.6.0/docs/en_US/sqls/built-in_functions.md#other-functions) by SuperRxan (https://github.com/lf-edge/ekuiper/pull/1321)
- SQL [between expression](https://github.com/lf-edge/ekuiper/blob/1.6.0/docs/en_US/sqls/query_language_elements.md#where) by ngjaying (https://github.com/lf-edge/ekuiper/pull/1327)
- SQL [like expression](https://github.com/lf-edge/ekuiper/blob/1.6.0/docs/en_US/sqls/query_language_elements.md#where) by ngjaying (https://github.com/lf-edge/ekuiper/pull/1327)
- [SQL sink](https://github.com/lf-edge/ekuiper/blob/1.6.0/docs/en_US/rules/sinks/plugin/sql.md) and [TDEngine sink](https://github.com/lf-edge/ekuiper/blob/1.6.0/docs/en_US/rules/sinks/plugin/tdengine.md) to support writing array data by SuperRxan (https://github.com/lf-edge/ekuiper/pull/1331)

Improvements

- [Helm chart](https://github.com/lf-edge/ekuiper/tree/1.6.0/deploy/chart/ekuiper): change ekuiper deployment type from stateful set to deployment by brianbinbin (https://github.com/lf-edge/ekuiper/pull/1318)
- Rework [sink cache and resend mechanism](https://github.com/lf-edge/ekuiper/blob/1.6.0/docs/en_US/rules/sinks/overview.md#caching) by ngjaying (https://github.com/lf-edge/ekuiper/pull/1304)
- [Time window](https://github.com/lf-edge/ekuiper/blob/1.6.0/docs/en_US/sqls/windows.md#time-units) align to nature time by ngjaying (https://github.com/lf-edge/ekuiper/pull/1323)
- Unify rule status response to alwasy return JSON by SuperRxan (https://github.com/lf-edge/ekuiper/pull/1335)
- Upgrade Go vesion to 1.18 by ngjaying (https://github.com/lf-edge/ekuiper/pull/1337)
- Fix order by alias bug by SuperRxan (https://github.com/lf-edge/ekuiper/pull/1299)
- Validate the native plugin in register time by SuperRxan (https://github.com/lf-edge/ekuiper/pull/1301)
- EdgeX and MQTT source: do not exit the rule when disconnected by SuperRxan (https://github.com/lf-edge/ekuiper/pull/1334)

Fixes

- Fix memory leak problem when starting rules multiple times by ngjaying (https://github.com/lf-edge/ekuiper/pull/1309)
- Fix default array properties is null in the meta APIs by SuperRxan (https://github.com/lf-edge/ekuiper/pull/1311)
- Filter must not apply on table before join align in planner by ngjaying (https://github.com/lf-edge/ekuiper/pull/1316)

Docs

- Add Rule grap API description
- Add protobuf scheme upload description
- Add sink cache mechanism description
- Add SQL lag function description
- Add SQL IN/BETWEEN/LIKE clause description

Incompatible changes

During rework of sink cache mechanism, we delete some properties from `etc/kuiper.yaml` and sink properties. Please update your rules which are dependant of these properties and try to migrate to the [new properties](https://ekuiper.org/docs/en/latest/rules/sinks/overview.html#configuration).

Remove from `etc/kuiper.yaml`: cacheThreshold, cacheTriggerCount, disableCache
Remove from sink properties: retryInterval, retryCount, cacheLength, cacheSaveInterval

New Contributors
* brianbinbin made their first contribution in https://github.com/lf-edge/ekuiper/pull/1318

**Full Changelog**: https://github.com/lf-edge/ekuiper/compare/1.5.1...1.6.0

Most importantly, fix the instability connection issue with neuron. In order to reformat the output, add datatemplate
functionality for edgex and memory sink. For the mqtt connection, clear the session flag on server side. For documents, provide
documents for neuron integration and correct outdated message.

Fixes

- Fix connection lost prblem for neuron source when restarting the rule
- Fix the rule can not be deleted issue if the rule syntax become invalid because of funtion updated by plugins
- Let eKuiper home path in same place for dev docker images
- Fix CORS issue for authorization header in restful api
- Fix window_start() and window_end() can not use as parameters issue

Feature

- Add datatemplate support for edgex and memory sink
- Clear the mqtt session flag in server side when ekuiper have mqtt source or sink

Docs

- Provide a tutorial for [neuron integration](https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/tutorials/neuron/neuron_integration_tutorial.md)