Ekuiper

Overview

[The Kuiper & EdgeX integration work](https://github.com/emqx/kuiper/projects/4) is almost completed, Kuiper will be integrated with EdgeX nightly Docker composer files soon. Kuiper 0.3.1will be released with EdgeX officially.

Features

- EdgeX integration

- Fixed some issues based on user's feedback, and updated the [tutorial doc](https://github.com/emqx/kuiper/blob/master/docs/en_US/edgex/edgex_rule_engine_tutorial.md) .
- New implementation [EdgeX sink support](https://github.com/emqx/kuiper/blob/master/docs/en_US/rules/sinks/edgex.md) based on the feedback from community, the analysis result can be published with the required format of EdgeX Message bus.
- Translated EdgeX document to [Chinese](https://github.com/emqx/kuiper/blob/master/docs/zh_CN/edgex/edgex_rule_engine_tutorial.md).
- Plugin management
- Kuiper now supports plugin management, but it was not provided in old versions. In this version, Kuiper provides REST API & command line tools. User can add, delete and view plugins through management tools.

Bug fixes

- [null handling in pre-processor](https://github.com/emqx/kuiper/issues/185)
- Rest sink [exceptions](https://github.com/emqx/kuiper/issues/173)
- Fixes some document issues.

Overview

[The initial version of Kuiper & EdgeX integration work](https://github.com/emqx/kuiper/projects/4) was just completed, users can start to try it. In the coming EdgeX Geneva release, Kuiper 0.3.0 or 0.4.0 will be released with EdgeX officially.

Features

- EdgeX integration

- EdgeX source support, now Kuiper can consume data from EdgeX Message Bus directly. You can refer to [tutorial doc](https://github.com/emqx/kuiper/blob/master/docs/en_US/edgex/edgex_rule_engine_tutorial.md) for learning how to use Kuiper to analyze data from EdgeX.
- EdgeX sink support, the analysis result can be published to EdgeX Message Bus directly.

- Schemaless stream definition

In previous Kuiper releases, user must create a stream with schema, but we found that in some user cases, the data schema could be very complex, and it will be difficult to create data schema for it. Now Kuiper supports to create a stream that does not have any fields, in this mode, Kuiper framework can't validate data types during data processing. So user need to very clear about the data types when writing rules, otherwise, it could probably have problems. [EgeX tutorial doc](https://github.com/emqx/kuiper/blob/master/docs/en_US/edgex/edgex_rule_engine_tutorial.md) is a good example of using schemaless stream.

- FVT test schenarios enhancements

- Added 4 EdgeX testcases
- Added testcases for Docker image

Bug fixes

- Fixed several problems of Github Actions

Overview

[Kuiper 2020 Roadmap](https://github.com/emqx/kuiper/projects) was updated to Github, and several other projects were also created, such as [Kuiper & EdgeX integration project](https://github.com/emqx/kuiper/projects/4). Please click related link if you're interested in it.

Features

- [Rest mangement API](https://github.com/emqx/kuiper/blob/master/docs/en_US/restapi/overview.md) now is provided. Besides CLI tools, user can manage streams & rules by Rest API.
- Streams management
- Rules management
- Max support rule number benchmark
- 8000 rules with 800 message/second with 2 core * 4GB memory in AWS
- Resource usage
- Memory: 89% ~ 72%
- CPU: 25%
- Setup FVT pipeline in [Github action](https://github.com/emqx/kuiper/actions). FVT test scenarios will be run automatically with any code commit or PR, so the product quality can be ensured.
- 8 Kuiper [FVT](https://github.com/emqx/kuiper/tree/master/fvt_scripts) (functional verification tests) were wrote, and covered following scenarios.
- Basic functions of HTTP REST-API
- Basic functions of CLI
- Complex end-2-end scenario for Kuiper source, processing and sink

Bug fixes

- Fixed [the sink result is not correct](https://github.com/emqx/kuiper/issues/101) issue
- Fixed several problems for running multiple rules.

This is a minor version.

- Fixed a regression bug, the password config cannot be loaded correctly.
- Fixed all of the unit test problems.
- Added continuous testing (Github actions) support.

Features

- Optimized performance
- Provides configuration for setting concurrency for Kuiper rule, so it can be optimized in different scenarios
- ``concurrency`` setting in [source](https://github.com/emqx/kuiper/blob/develop/docs/en_US/rules/sources/mqtt.md): How many instances will be started. By default, only an instance will be run. If more than one instance is specified, the topic must be a shared subscription topic.
- ``concurrency`` settings in [sink](https://github.com/emqx/kuiper/blob/develop/docs/en_US/rules/overview.md#actions): Specify how many instances of the sink will be run. If the value is bigger than 1, the order of the messages may not be retained.
- ``concurrency`` settings in [SQL plans](https://github.com/emqx/kuiper/blob/develop/docs/en_US/rules/overview.md#options): A rule is processed by several phases of plans according to the SQL statement. This option will specify how many instances will be run for each plan. If the value is bigger than 1, the order of the messages may not be retained.
- Performance test result
- Raspberry Pi 3B+: 12k messages/second; CPU utilization (sys+user): 70%; Memory: 20M
- AWS t2.micro( 1 Core * 1 GB, Ubuntu18.04): 10k messages/second; CPU utilization (sys+user): 25%; Memory: 20M
- Support [metrics collection for rules](https://github.com/emqx/kuiper/blob/develop/docs/en_US/cli/rules.md#get-the-status-of-a-rule), which can be used for message processing status tracking. The metrics includes,
- ``in, out, exception`` message count for all operators
- ``process_latency_ms`` for all operators
- ``buffer_length``, the used buffer length for all operators
- ``last_invocation``, the last invocation timestamp for all operators
- Tested in OpenWrt Linux (Chaos Calmer 15.05) with 1core CPU * 256M memory, and it works fine.
- Support re-connect to MQTT broker if MQTT source or sink is disconnected.

Bug fixes

- Print error message in ``cli`` command if any error is found during starting a rule.
- Fixed several issues in ``rest sink``.

Features

- Support [extension](https://github.com/emqx/kuiper/blob/master/docs/en_US/extension/overview.md).
- Supported the extension at compilation and architect, now support the extension
- Supported ZeroMQ source and sink
- Supported HTTP REST sink
- Refactored code to support aggregate functions
- Kuiper can be programmatically invoked by the 3rd party applications, so it can be easily integrated with the 3rd party frameworks (such as EdgeX Foundry rule engine)
- Optimized memory footprint (memory footprint is 10MB+ when startup)
- Build improvement
- Provides the Docker images, and images can be pulled at https://hub.docker.com/r/emqx/kuiper
- Provided Helm file, and Kuiper can be deployed with [K3s](https://github.com/emqx/kuiper/blob/master/deploy/chart/kuiper/README.md).

Bug fixes

- [16](https://github.com/emqx/kuiper/issues/16) Rule status issue