Ekuiper

Overview

Kuiper 1.0.2 optimized the SQL execution plan and runtime, it greatly improves performance for some scenarios. Also Kuiper support the binary type in this release, and it can be used for processing binary streams, such as image.

Features & fixes

- SQL plan optimization
- Optimized the SQL plan, add `PushDownPredicate` support and it can filter some records in advance according to the condition specified in WHERE clause. It can reduce memory usage for long time window processing.
- More optimization will be introduced in future releases.
- Binary data type support
- Now the binary data type is supported, and user can process binary streams, such as processing images.
- [An image sink plugin](https://github.com/emqx/kuiper/blob/master/docs/zh_CN/plugins/sinks/image.md) for saving images files into local disks.
- Change streams & rules storage from file to SQLite.
- Ping API support, which is used for Kuiper service alive detection.
- Fixed one memory leak problem - some of memory is not released when the rule is deleted.
- No way to limit client and REST APIs to localhost
- File logging error message when server fileLog=false
- Added a retry count for all sinks
- Logrotate support for log files - rotate time is 24 hrs, and max age is 3 days
- Added several SQL validation during compilation stage
- Enhanced document compilation.
- Fixed several document problem.
- Fixed several issues for httppull source
- Fixed issue when loading array types in management-ui

Overview

Kuiper 1.0.1 fixed several problems and enhanced Kuiper manager.

Features & fixes

- Kuiper manager
- Support for stream and rule update.
- Add rule status visualization support.
- Add support for writing to Linux syslog.
- TDengine ARM64 sink plugin support.
- Improved EdgeX using tutorial.
- Fixed issues

- [The Kuiper crash issue for ARM64 environment](https://github.com/emqx/kuiper/issues/570).
- Change to a smaller time unit for elapsed time (rule status)
- Kuiper manager UI
- [The list value cannot be displayed in sink editor](https://github.com/emqx/kuiper/issues/597)
- Incorrect UI display for the complex streaming definition

Overview

Kuiper 1.0.0 is the 1st major stable version. This version is fully integrated with other EMQ softwares for edge, including [Neuron](https://www.emqx.io/products/neuron), [Edge](https://www.emqx.io/products/edge) and `edge manager`. Readers can refer to [edge-stack project](https://github.com/emqx/edge-stack) for how to use EMQ softwares for edge.

Features & fixes

- A break change! If any program or scripts that references to Kuiper binary file name `bin/server` & `bin/cli`, you'll have to update correspondingly as in below.
- Renamed `bin/server` to `bin/kuiperd`
- Renamed `bin/cli` to `bin/kuiper`
- EdgeX enhancements & fixes
- Gives error when configured to use redisstreams for EdgeX MessageBus
- Updated EdgeX docker-compose file, and upgrade to Kuiper 0.9.1 version
- Go 15 upgrading
- Message bus SDK upgrading
- EdgeX Kuiper document & tutorial update
- Containerized Kuiper fails when started with read-only file system
- Updated the documents
- Kuiper docker images usage document.
- Wrote and translated the document for using Kuiper manager console.
- Enhanced the TDengine plugins
- Renamed the plugin name from taos to TDengine
- [Allow user to control the timestamp for data saving into TDengine](https://github.com/emqx/kuiper/issues/520)
- [Set default values for TDengine sink plugin](https://github.com/emqx/kuiper/issues/527)
- Allowed user to pass parameters for plugin setup scripts, which make the plugins setup more flexible.
- Returns i18N message with rest services.
- Fixed below issues
- [The line number is not correct in log file](https://github.com/emqx/kuiper/issues/518)

Thanks

- [TateDeng](https://github.com/TateDeng) provides fix for DynamicChannelBuffer setting

Overview

Kuiper 0.9.1 provides a management console, which can be used for Kuiper node management, stream, rule and plugin visualize edit. It greatly improves the using experience.

Features & fixes

- Visualize management: A separated new Docker image was released with this new version. The image is a web based management console, and it provides streams, rules and plugins management. Please refer to [doc](https://github.com/emqx/kuiper/tree/master/docs/en_US/manager-ui/overview.md) fore more detailed information.
- Binary plugins: From this version, all of the plugins that submit to Github main repository will be compiled and published automatically when a new version is released. User can install & use the plugins through management console. When developer create PR for plugin code, the [plugin metadata information](https://github.com/emqx/kuiper/blob/master/docs/zh_CN/plugins/overview.md) can be provided, and then the related information will be displayed in the management console. It can greatly reduce the effort for using the plugins.
- User can control to cache the data that failed to publish in the sink by set `disableCache` of `kuiper.yaml`.
- Kubernetes configuration file dispatch tool multiple CPU arch support.
- Add Collect function support, which can be used for return all of data in the window.
- Add Deduplicate function support, which can deduplicate the data of window.
- Fixed below issues
- [Edgex-ui cannot post cross-domain requests](https://github.com/emqx/kuiper/issues/405)
- [Kuiper process exits in log debug mode]( https://github.com/emqx/kuiper/issues/438)
- [Rule is created even with error reported during rule creation](https://github.com/emqx/kuiper/issues/426)
- [Set sendSingle as true but no data template is specified, the process is quit](https://github.com/emqx/kuiper/issues/416)
- [Use describe plugin command error](https://github.com/emqx/kuiper/issues/413)
- Nested struct definition reports syntax error.

Thanks

- soyoo provides several log print fixes

Overview

Kuiper 0.9.0 is a major version that includes the stream state management, KubeEdge device model adoption, EdgeX array type support and TDengine database sink support.

Features & fixes

- [State management function is supported](https://github.com/emqx/kuiper/blob/develop/docs/zh_CN/rules/state_and_fault_tolerance.md). This feature enables Kuiper to implement stream with state:
- Fault tolerant processing is supported. If an unexpected interruption occurs during stream processing, it can be resumed after the rule is restarted;
- Checkpointing is supported, which allows users to implement QoS settings during stream processing, including At-most-once(0), At-least-once(1) and Exactly-once(2);
- By consuming data from the specified offset, users can extend the relevant interface to realize the data source that can re-consume the data, so that it can be recovered offline or when there is an error in stream processing;
- Configurable state persistent storage. The system stores the state in the file system by default, and also supports storing state data in a third-party system, such as Redis;
- The user is supported to call the API provided by Kuiper to realize the storage of custom state data when extending the source, sink and function;
- Access support for [KubeEdge data model](https://github.com/emqx/kuiper/blob/develop/docs/en_US/rules/sources/mqtt.md#kubeedgeversion) is provided, with a tool for automatically issuing configuration files through Kubernetes configmap. Users can use Kuiper to directly support the analysis of device data from KubeEdge;
- [TDengine plug-in](https://github.com/emqx/kuiper/blob/master/docs/en_US/plugins/sinks/taos.md) is added, which can support saving analysis results to TDengine time series database;
- All Chinese documents are translated and synchronized;
- The execution process of FVT in Github Action is optimized, and some unnecessary test processes are deleted;
- `filter` in the window with [support for filtering data](https://github.com/emqx/kuiper/blob/cfbdf6503e7e63e0680d038cb06aece0415f91a0/docs/en_US/sqls/windows.md#filter-window-inputs) can realize the data filtering and window grouping; this function is more important for counting windows: the result of filtering through the WHERE statement and then window grouping will be different;

Thanks

- [chensheng0](https://github.com/emqx/kuiper/commits?author=chensheng0) provides a fix for Kubernetes configmap, which can be integrated with [Baidu Baetyl](https://github.com/baetyl/baetyl) framework.
- [GZJ](https://github.com/emqx/kuiper/commits?author=GZJ) provides a fix that cleans up the site when Kuiper exits.
- [smart33690](https://github.com/smart33690) provided a fix for the [Influxdb sink sample plug-in](https://github.com/emqx/kuiper/blob/master/docs/zh_CN/plugins/sinks/influxdb.md).

This release fixed several issues.

- Refactor the `kuiper.yaml` loading
- Fixed rest sink header settings
- Fixed rest sink cache problems - the cache entries cannot be deleted even they are sent successfully
- Add `insecureSkipVerify` in rest sink, which allows user to set in-secured verification flag