This version has done a lot of refactoring in the streaming runtine, especially in source. The source node has been splitted into multiple nodes including connector, decompressor and decoder according to the source configurations like format, compressing and shared etc. This will help users to monitor rules in more detail, knowing which process either decode and decompress takes more time and other metrics, thus, we can be able to optimize in a finer grain. It will make it easier for developers to maitain and add more features into source without a need to maintain an all-in-one but complex source node. In this version, the source split only applies to MQTT source. We will expand the support to all source progressively in the following versions. And feedbacks are welcomed.
Streaming Runtime
* Dynamic Schema
* feat: support merge schema by Yisaer in https://github.com/lf-edge/ekuiper/pull/2502
* feat: support merge schema for wildcard by Yisaer in https://github.com/lf-edge/ekuiper/pull/2516
* feat: support decode schemaless json by Yisaer in https://github.com/lf-edge/ekuiper/pull/2500
* feat: support runtime schema in rule explain by Yisaer in https://github.com/lf-edge/ekuiper/pull/2545
- Refactor source to connector, decompressor and decoder and allow ordered concurrency. This should improve throughput a lot if the payload is big and decoding latency is long
- Refactor ping for connectors, provide more accurate info
SQL Syntax
- New function object_pick and kv/kvarray conversion functions
- feat: implement math functions by carlclone in https://github.com/lf-edge/ekuiper/pull/2483
- feat: implement CONV function by carlclone in https://github.com/lf-edge/ekuiper/pull/2506
- feat(func): add locale argument support by carlclone in https://github.com/lf-edge/ekuiper/pull/2522
Extension
* JavaScript Function
* feat(script): add javascript function by ngjaying in https://github.com/lf-edge/ekuiper/pull/2549
* feat(cli): add script support by ngjaying in https://github.com/lf-edge/ekuiper/pull/2553
* feat(script): add script import/export by ngjaying in https://github.com/lf-edge/ekuiper/pull/2552
Configuration and Validation
* feat: support raw tls config by Yisaer in https://github.com/lf-edge/ekuiper/pull/2550
* Introduce error code
All Changes
* feat: implement math functions by carlclone in https://github.com/lf-edge/ekuiper/pull/2483
* fix: fix lookup plan column pruning by Yisaer in https://github.com/lf-edge/ekuiper/pull/2494
* refactor(trial): catch sink errors by ngjaying in https://github.com/lf-edge/ekuiper/pull/2499
* fix: fix sourceNode options by Yisaer in https://github.com/lf-edge/ekuiper/pull/2501
* chore: add sniffio dependency for python plugins by Yisaer in https://github.com/lf-edge/ekuiper/pull/2513
* feat: implement CONV function by carlclone in https://github.com/lf-edge/ekuiper/pull/2506
* chore(deps): bump github.com/dvsekhvalnov/jose2g by dependabot in https://github.com/lf-edge/ekuiper/pull/2514
* chore(deps): bump github.com/dvsekhvalnov/jose2go by dependabot in https://github.com/lf-edge/ekuiper/pull/2515
* feat: support merge schema by Yisaer in https://github.com/lf-edge/ekuiper/pull/2502
* feat: support merge schema for wildcard by Yisaer in https://github.com/lf-edge/ekuiper/pull/2516
* docs(example): merge multiple stream by ngjaying in https://github.com/lf-edge/ekuiper/pull/2518
* feat: support decode schemaless json by Yisaer in https://github.com/lf-edge/ekuiper/pull/2500
* fix(decode): perf update by ngjaying in https://github.com/lf-edge/ekuiper/pull/2519
* chore: update httppush.json file by oucb in https://github.com/lf-edge/ekuiper/pull/2520
* fix(rest): return 404 when rule not found by carlclone in https://github.com/lf-edge/ekuiper/pull/2505
* feat(func): add locale argument support by carlclone in https://github.com/lf-edge/ekuiper/pull/2522
* fix: fix confKey API can't replace url/passwd by Yisaer in https://github.com/lf-edge/ekuiper/pull/2538
* fix(trial): avoid rule id conflict by ngjaying in https://github.com/lf-edge/ekuiper/pull/2539
* fix: fix sql reconnect by Yisaer in https://github.com/lf-edge/ekuiper/pull/2541
* feat: support hide url password in log by Yisaer in https://github.com/lf-edge/ekuiper/pull/2542
* fix: fix install plugin failed after upgrade by Yisaer in https://github.com/lf-edge/ekuiper/pull/2543
* fix(doc): complete directory description by carlclone in https://github.com/lf-edge/ekuiper/pull/2546
* feat: support runtime schema in rule explain by Yisaer in https://github.com/lf-edge/ekuiper/pull/2545
* feat(script): add javascript function by ngjaying in https://github.com/lf-edge/ekuiper/pull/2549
* refactor(component): extract import/export manager by ngjaying in https://github.com/lf-edge/ekuiper/pull/2551
* feat(cli): add script support by ngjaying in https://github.com/lf-edge/ekuiper/pull/2553
* feat(script): add script import/export by ngjaying in https://github.com/lf-edge/ekuiper/pull/2552
* docs(script): add script support related doc by ngjaying in https://github.com/lf-edge/ekuiper/pull/2556
* fix: fix memory leak if window condition unmatch by Yisaer in https://github.com/lf-edge/ekuiper/pull/2559
* fix: fix schema explain by Yisaer in https://github.com/lf-edge/ekuiper/pull/2555
* feat: support raw tls config by Yisaer in https://github.com/lf-edge/ekuiper/pull/2550
* perf(sink): avoid reconstruct send param by ngjaying in https://github.com/lf-edge/ekuiper/pull/2557
* feat: support object_pick function by rextang in https://github.com/lf-edge/ekuiper/pull/2560
* fix: delete default username of redisPubSub by retoool in https://github.com/lf-edge/ekuiper/pull/2566
* feat: support message processed metrics by Yisaer in https://github.com/lf-edge/ekuiper/pull/2561
* feat: support connection native ping by Yisaer in https://github.com/lf-edge/ekuiper/pull/2562
* test: add more test for lines format by carlclone in https://github.com/lf-edge/ekuiper/pull/2570
* refactor: extract format reader from file source by carlclone in https://github.com/lf-edge/ekuiper/pull/2571
* feat: support more kv functions by rextang in https://github.com/lf-edge/ekuiper/pull/2575
* refactor: node runtime by ngjaying in https://github.com/lf-edge/ekuiper/pull/2582
* fix: fix object function args is nil by Yisaer in https://github.com/lf-edge/ekuiper/pull/2586
* fix: fix and test array function if has nil by Yisaer in https://github.com/lf-edge/ekuiper/pull/2581
* feat: support errorcode framework by Yisaer in https://github.com/lf-edge/ekuiper/pull/2577
* feat(node): split source connector and decoder by ngjaying in https://github.com/lf-edge/ekuiper/pull/2583
* fix(functions): args validation of `date_calc` by yai-dev in https://github.com/lf-edge/ekuiper/pull/2585
* fix: fix TestPing test by Yisaer in https://github.com/lf-edge/ekuiper/pull/2595
* feat: support parser/plan/executor error code by Yisaer in https://github.com/lf-edge/ekuiper/pull/2594
* feat: introduce undefined error code by Yisaer in https://github.com/lf-edge/ekuiper/pull/2596
* feat: revise restart rule by Yisaer in https://github.com/lf-edge/ekuiper/pull/2597
* feat: support neuron/kafka ping by Yisaer in https://github.com/lf-edge/ekuiper/pull/2591
* feat(node): ordered concurrency support by ngjaying in https://github.com/lf-edge/ekuiper/pull/2592
* fix(topo): avoid plan twice by ngjaying in https://github.com/lf-edge/ekuiper/pull/2600
* fix(stats): fix get metrics wrong pos by ngjaying in https://github.com/lf-edge/ekuiper/pull/2601
* fix: fix source kafka plugin ci build by Yisaer in https://github.com/lf-edge/ekuiper/pull/2604
* fix: fix schemaless decode error by Yisaer in https://github.com/lf-edge/ekuiper/pull/2605
* feat: add error code by Yisaer in https://github.com/lf-edge/ekuiper/pull/2598
* refactor(node): split decompressor op by ngjaying in https://github.com/lf-edge/ekuiper/pull/2602
* fix(topo): subtopo runtime fix by ngjaying in https://github.com/lf-edge/ekuiper/pull/2609
* feat: add stream/table/rule/confkey error code by Yisaer in https://github.com/lf-edge/ekuiper/pull/2607
* feat: support expr validate by Yisaer in https://github.com/lf-edge/ekuiper/pull/2608
* fix: fix protobuf oneof decode by Yisaer in https://github.com/lf-edge/ekuiper/pull/2610
* fix(docs): correct typo by xiaozexuan in https://github.com/lf-edge/ekuiper/pull/2612
* feat(node): split shared connection as a node by ngjaying in https://github.com/lf-edge/ekuiper/pull/2613
* fix: fix rest and sql error and log by Yisaer in https://github.com/lf-edge/ekuiper/pull/2620
* docs: add error code doc by Yisaer in https://github.com/lf-edge/ekuiper/pull/2621
* fix: skip check cert if not necessary by Yisaer in https://github.com/lf-edge/ekuiper/pull/2623
* fix(node): avoid excessive log when enable cp by ngjaying in https://github.com/lf-edge/ekuiper/pull/2622
* feat: revise mqtt tls error msg by Yisaer in https://github.com/lf-edge/ekuiper/pull/2626
* fix: fix gob encoding by Yisaer in https://github.com/lf-edge/ekuiper/pull/2639
* fix(func): fix ln function by ngjaying in https://github.com/lf-edge/ekuiper/pull/2630
* fix: fix status message by Yisaer in https://github.com/lf-edge/ekuiper/pull/2636
* feat: let stmt validate by checker by Yisaer in https://github.com/lf-edge/ekuiper/pull/2638
* feat: support validate conf before creating by Yisaer in https://github.com/lf-edge/ekuiper/pull/2635
* feat: introduce failpoint and fix resend error by Yisaer in https://github.com/lf-edge/ekuiper/pull/2642
* feat: check rule before creating it by Yisaer in https://github.com/lf-edge/ekuiper/pull/2624
* fix: fix sql error status by Yisaer in https://github.com/lf-edge/ekuiper/pull/2643
* fix: check redis connector db by Yisaer in https://github.com/lf-edge/ekuiper/pull/2644
* feat: support validate rule id by Yisaer in https://github.com/lf-edge/ekuiper/pull/2645
* fix: support validate rest sink method by Yisaer in https://github.com/lf-edge/ekuiper/pull/2647
* fix: add pynng version by Yisaer in https://github.com/lf-edge/ekuiper/pull/2648
* fix: lock pynng version 0.7.3 by Yisaer in https://github.com/lf-edge/ekuiper/pull/2649
* feat(service): desc func return addr by ngjaying in https://github.com/lf-edge/ekuiper/pull/2651
* fix: lock pynng into 0.7.2 by Yisaer in https://github.com/lf-edge/ekuiper/pull/2654
* ci(build): add workflow_dispatch to build packages by Rory-Z in https://github.com/lf-edge/ekuiper/pull/2655
* fix(docs): clean up en content by ngjaying in https://github.com/lf-edge/ekuiper/pull/2656
* fix: fix sink sql for oracle by Yisaer in https://github.com/lf-edge/ekuiper/pull/2657
* fix(analyzer): correct embedded analytic funcs by ngjaying in https://github.com/lf-edge/ekuiper/pull/2653
* fix: fix zh doc by Yisaer in https://github.com/lf-edge/ekuiper/pull/2658
* fix: fix buffer length metrics by Yisaer in https://github.com/lf-edge/ekuiper/pull/2659
* fix(service): delete schemaless func by ngjaying in https://github.com/lf-edge/ekuiper/pull/2660
* fix(build): remove script from default by ngjaying in https://github.com/lf-edge/ekuiper/pull/2662
* fix: fix converter schema display by Yisaer in https://github.com/lf-edge/ekuiper/pull/2663
* fix: fix conv validate by Yisaer in https://github.com/lf-edge/ekuiper/pull/2664
* fix: fix cert options by Yisaer in https://github.com/lf-edge/ekuiper/pull/2671
* fix(topo): avoid wrong topo open state by ngjaying in https://github.com/lf-edge/ekuiper/pull/2672
* fix(topo): catch recover rule error by ngjaying in https://github.com/lf-edge/ekuiper/pull/2673
* fix: fix influx sink configure by Yisaer in https://github.com/lf-edge/ekuiper/pull/2670
* fix(rest): sink should return unrecoverable error by ngjaying in https://github.com/lf-edge/ekuiper/pull/2676
* docs: fix Chinese doc errors for substring func by Dream95 in https://github.com/lf-edge/ekuiper/pull/2661
* fix: fix wildcard flag and default fastjson decode by Yisaer in https://github.com/lf-edge/ekuiper/pull/2677
* feat: union mqtt sink tls error behavior by Yisaer in https://github.com/lf-edge/ekuiper/pull/2675
* ci(docker): add full tag by ngjaying in https://github.com/lf-edge/ekuiper/pull/2682
* fix: tdengine fields setting problem by gooker in https://github.com/lf-edge/ekuiper/pull/2681
* refactor: split shared source schema support by ngjaying in https://github.com/lf-edge/ekuiper/pull/2678
* docs: revise sqlserver doc by Yisaer in https://github.com/lf-edge/ekuiper/pull/2683
* fix: fix neuron ping by Yisaer in https://github.com/lf-edge/ekuiper/pull/2686
* chore: move the plugin yaml by ngjaying in https://github.com/lf-edge/ekuiper/pull/2684
* fix: fix kafka sasl ping by Yisaer in https://github.com/lf-edge/ekuiper/pull/2687
* fix: fix explain unexist rule by Yisaer in https://github.com/lf-edge/ekuiper/pull/2685
* fix(func): last_value for nil array by ngjaying in https://github.com/lf-edge/ekuiper/pull/2690
* fix: fix ruleID check by Yisaer in https://github.com/lf-edge/ekuiper/pull/2693
* chore(deps): bump github.com/jackc/pgx/v4 from 4.18.1 to 4.18.2 by dependabot in https://github.com/lf-edge/ekuiper/pull/2699
* chore(deps): bump github.com/jackc/pgx/v4 from 4.18.1 to 4.18.2 in /extensions by dependabot in https://github.com/lf-edge/ekuiper/pull/2698
* fix: fix join exit check by Yisaer in https://github.com/lf-edge/ekuiper/pull/2695
* feat: add log when remove metrics by Yisaer in https://github.com/lf-edge/ekuiper/pull/2701
* fix: remove cpu consuming log by Yisaer in https://github.com/lf-edge/ekuiper/pull/2697
* chore(deps): bump google.golang.org/protobuf from 1.28.0 to 1.33.0 in /internal/converter/protobuf/test by dependabot in https://github.com/lf-edge/ekuiper/pull/2705
* chore: update sql.json file by oucb in https://github.com/lf-edge/ekuiper/pull/2703
* fix(config): should not export empty config by ngjaying in https://github.com/lf-edge/ekuiper/pull/2702
* docs(func): remind of mem usage for lpad/rpad by ngjaying in https://github.com/lf-edge/ekuiper/pull/2708
* fix: fix rule id check by Yisaer in https://github.com/lf-edge/ekuiper/pull/2709
* fix: revise ping message by Yisaer in https://github.com/lf-edge/ekuiper/pull/2710
* fix: remove neuron ping by Yisaer in https://github.com/lf-edge/ekuiper/pull/2711
* fix: revise mqtt source on msg behavior by Yisaer in https://github.com/lf-edge/ekuiper/pull/2712
New Contributors
* rextang made their first contribution in https://github.com/lf-edge/ekuiper/pull/2560
* xiaozexuan made their first contribution in https://github.com/lf-edge/ekuiper/pull/2612
* Dream95 made their first contribution in https://github.com/lf-edge/ekuiper/pull/2661
* gooker made their first contribution in https://github.com/lf-edge/ekuiper/pull/2681
**Full Changelog**: https://github.com/lf-edge/ekuiper/compare/1.12.1...1.13.0