Feldera

What's Changed

* Fix aggregating metrics when multiple connectors per relation by Karakatiza666 in https://github.com/feldera/feldera/pull/2089
* py: add SQLContext.input_json() to send json data by abhizer in https://github.com/feldera/feldera/pull/2088
* [SQL] validate table properties syntax by mihaibudiu in https://github.com/feldera/feldera/pull/2091
* demo: new faster data generator for Hopsworks Tiktok RecSys by abhizer in https://github.com/feldera/feldera/pull/2085
* [SQL] Fix bugs related to SQL UDFs by mihaibudiu in https://github.com/feldera/feldera/pull/2099
* [README] fix broken documentation links by kshuta in https://github.com/feldera/feldera/pull/2109
* Adds a new datagen input adapter for tables. by gz in https://github.com/feldera/feldera/pull/2107
* docs: fix getting started link in README by lalithsuresh in https://github.com/feldera/feldera/pull/2114
* [SQL] FOREIGN KEY validation and metadata construction by mihaibudiu in https://github.com/feldera/feldera/pull/2108
* Connector orchestration by ryzhyk in https://github.com/feldera/feldera/pull/2113
* Debezium Postgres demo. by ryzhyk in https://github.com/feldera/feldera/pull/2116
* [SQL] Document and test TIMESTAMPADD function; improve Calcite optimizer structure by mihaibudiu in https://github.com/feldera/feldera/pull/2118
* Test aggregates by rivudhk in https://github.com/feldera/feldera/pull/2103
* Optimize asynchronous merger by blp in https://github.com/feldera/feldera/pull/2115
* Revert "Test aggregates (2103)" by rivudhk in https://github.com/feldera/feldera/pull/2123
* dbsp: Don't reserve excess capacity for vector merge results. by blp in https://github.com/feldera/feldera/pull/2127
* Merger revert by blp in https://github.com/feldera/feldera/pull/2135
* release: bump project version to 0.22.0 by gz in https://github.com/feldera/feldera/pull/2138

New Contributors
* kshuta made their first contribution in https://github.com/feldera/feldera/pull/2109
* rivudhk made their first contribution in https://github.com/feldera/feldera/pull/2103

**Full Changelog**: https://github.com/feldera/feldera/compare/v0.21.0...v0.22.0

What's Changed

* pipeline-manager: remove service probing by snkas in https://github.com/feldera/feldera/pull/2002
* Add tracing for dbsp steps and general docs for monitoring feldera by gz in https://github.com/feldera/feldera/pull/1999
* [SQL] Improved monotonicity analysis for filter operators by mihaibudiu in https://github.com/feldera/feldera/pull/2006
* DBSP: Asof-join operator. by ryzhyk in https://github.com/feldera/feldera/pull/1998
* Merge new WebConsole layout prototype by Karakatiza666 in https://github.com/feldera/feldera/pull/1994
* Automate Flink testing in CI by aehmttw in https://github.com/feldera/feldera/pull/2005
* dbsp: test using `window` to implement SQL `now`. by ryzhyk in https://github.com/feldera/feldera/pull/2015
* demo: Hopsworks Tiktok RecSys pipeline by abhizer in https://github.com/feldera/feldera/pull/1992
* [SQL] Fix bugs in implementations of LIMIT by mihaibudiu in https://github.com/feldera/feldera/pull/2020
* Add docs for the new WebConsole project by Karakatiza666 in https://github.com/feldera/feldera/pull/2021
* Disable sveltekit build by Karakatiza666 in https://github.com/feldera/feldera/pull/2024
* Re-enable and fix sveltekit build by Karakatiza666 in https://github.com/feldera/feldera/pull/2025
* Disable coloring in maven builds by mihaibudiu in https://github.com/feldera/feldera/pull/2026
* [SQL] Fix bug for primary key fields that need cloning by mihaibudiu in https://github.com/feldera/feldera/pull/2029
* Update README.md by gz in https://github.com/feldera/feldera/pull/2035
* [SQL] Preliminary implementation of the `now()` function by mihaibudiu in https://github.com/feldera/feldera/pull/2019
* WebConsole: Fix pipeline relation metrics not aggregated … by Karakatiza666 in https://github.com/feldera/feldera/pull/2036
* [SQL] Fix bugs in window bound computations for INTERVAL types by mihaibudiu in https://github.com/feldera/feldera/pull/2034
* [SQL] Fix handling of monotonicity for tables with primary keys by mihaibudiu in https://github.com/feldera/feldera/pull/2041
* Make sure dbsp and adapters metrics verison match. by gz in https://github.com/feldera/feldera/pull/2044
* Update grafana_dashboard.json with some storage metrics by gz in https://github.com/feldera/feldera/pull/2046
* [SQL] Correctly handle nullable types in ARG_MAX by mihaibudiu in https://github.com/feldera/feldera/pull/2047
* [SQL] Use waterline and not waterline_monotonic by mihaibudiu in https://github.com/feldera/feldera/pull/2048
* Fix bun build in deploy/Dockerfile by Karakatiza666 in https://github.com/feldera/feldera/pull/2050
* Generalize SQL benchmarks to make it easier to add more by aehmttw in https://github.com/feldera/feldera/pull/2013
* Reset min-storage-bytes to default by aehmttw in https://github.com/feldera/feldera/pull/2053
* [SQL] Have the compiler synthesize the 'NOW' table by mihaibudiu in https://github.com/feldera/feldera/pull/2049
* [SQL] Improved monotonicity analysis for joins followed by filters by mihaibudiu in https://github.com/feldera/feldera/pull/2014
* Make Flink run all queries by aehmttw in https://github.com/feldera/feldera/pull/2054
* TikTok demo: Fixes and refactoring. by ryzhyk in https://github.com/feldera/feldera/pull/2055
* dbsp_adapters: Wait a bit before disconnecting on pause in `url` adapter. by blp in https://github.com/feldera/feldera/pull/2057
* Improve profiling output to make storage performance easier to understand by blp in https://github.com/feldera/feldera/pull/2059
* nexmark: Use same `metrics` version as the rest of the crates. by blp in https://github.com/feldera/feldera/pull/2061
* dbsp: Name background threads so system tools can distinguish them. by blp in https://github.com/feldera/feldera/pull/2062
* Update program.sql by gz in https://github.com/feldera/feldera/pull/2064
* Fix port in documentation. by gz in https://github.com/feldera/feldera/pull/2065
* [SQL] Two optimizations by mihaibudiu in https://github.com/feldera/feldera/pull/2068
* Storage optimizations by blp in https://github.com/feldera/feldera/pull/2066
* Revert "WebConsole: Fix pipeline relation metrics not accumulated when there are multiple connectors per relation" by lalithsuresh in https://github.com/feldera/feldera/pull/2069
* Benchmark with 100M events by aehmttw in https://github.com/feldera/feldera/pull/2077
* Add metric for records dropped due to lateness by mihaibudiu in https://github.com/feldera/feldera/pull/2074
* Read input connector names from SQL by aehmttw in https://github.com/feldera/feldera/pull/2076


**Full Changelog**: https://github.com/feldera/feldera/compare/v0.20.0...v0.21.0

What's Changed
* Post release v0.19.0 by gz in https://github.com/feldera/feldera/pull/1944
* Improve py docs by abhizer in https://github.com/feldera/feldera/pull/1942
* dbsp: Fix build of `galen` benchmark. by blp in https://github.com/feldera/feldera/pull/1928
* py: max_queue_records param to connect_source_kafka by ryzhyk in https://github.com/feldera/feldera/pull/1948
* Improved monotonicity analysis by mihaibudiu in https://github.com/feldera/feldera/pull/1949
* Switch binary download port from 9090 to 8085. by gz in https://github.com/feldera/feldera/pull/1951
* pipeline-manager: SQL MAP type by snkas in https://github.com/feldera/feldera/pull/1941
* benchmark: API key and Kafka option arguments by snkas in https://github.com/feldera/feldera/pull/1954
* [SQL] Do not sort collection after TOP K unless it is needed at the output by mihaibudiu in https://github.com/feldera/feldera/pull/1958
* Fix for numpy regression. by ryzhyk in https://github.com/feldera/feldera/pull/1961
* Update earthly version/install. by gz in https://github.com/feldera/feldera/pull/1960
* build(deps): bump scikit-learn from 1.2.1 to 1.5.0 in /demo/demo_notebooks by dependabot in https://github.com/feldera/feldera/pull/1888
* [SQL] Reorganize compilation stages and fix 3 bugs by mihaibudiu in https://github.com/feldera/feldera/pull/1966
* [SQL] support MATERIALIZED table and views by mihaibudiu in https://github.com/feldera/feldera/pull/1959
* [SQL] Improve monotonicity analysis by mihaibudiu in https://github.com/feldera/feldera/pull/1970
* Do not automatically distinct ouputs. by ryzhyk in https://github.com/feldera/feldera/pull/1972
* Switch from log to tracing. by gz in https://github.com/feldera/feldera/pull/1976
* [SQL] Improve monotonicity for aggregations by mihaibudiu in https://github.com/feldera/feldera/pull/1974
* ci: run python tests by abhizer in https://github.com/feldera/feldera/pull/1873
* Nexmark sql output by blp in https://github.com/feldera/feldera/pull/1979
* Minor profiling-related improvements by ryzhyk in https://github.com/feldera/feldera/pull/1971
* WebConsole: Drop all Data Services functionality by Karakatiza666 in https://github.com/feldera/feldera/pull/1978
* Post-fix for unsuccessful rebase in PR1971. by ryzhyk in https://github.com/feldera/feldera/pull/1981
* Expose dbsp metrics in pipeline metrics endpoint. by gz in https://github.com/feldera/feldera/pull/1969
* Fix docker command by aehmttw in https://github.com/feldera/feldera/pull/1984
* Use a single tokio runtime instead of `n`. by gz in https://github.com/feldera/feldera/pull/1983
* docs: feldera enterprise by snkas in https://github.com/feldera/feldera/pull/1991
* docs: remove deploy from sidebar and fix secret management links by snkas in https://github.com/feldera/feldera/pull/1993
* Improve Kafka input performance for small messages by blp in https://github.com/feldera/feldera/pull/1967
* docs: eks cluster storage class check by snkas in https://github.com/feldera/feldera/pull/1996
* dbsp: Change threshold for spilling to storage from rows to bytes. by blp in https://github.com/feldera/feldera/pull/1917
* Make profiles downloadable via API by blp in https://github.com/feldera/feldera/pull/1988
* benchmark: Make poller threads configurable for SQL benchmarks. by blp in https://github.com/feldera/feldera/pull/1997
* Enable playwright tests by Karakatiza666 in https://github.com/feldera/feldera/pull/1952
* Add script to run flink on CI by aehmttw in https://github.com/feldera/feldera/pull/1985
* [SQL] a few small optimizations: combine join/filter/map and map/mapindex by mihaibudiu in https://github.com/feldera/feldera/pull/1986
* Disable navigation from materialized to non-materialized view by Karakatiza666 in https://github.com/feldera/feldera/pull/2001
* release: bump project version to 0.20.0 by gz in https://github.com/feldera/feldera/pull/2007


**Full Changelog**: https://github.com/feldera/feldera/compare/v0.19.0...v0.20.0

New Contributors

* aehmttw made their first contribution in https://github.com/feldera/feldera/pull/1857

**Full Changelog**: https://github.com/feldera/feldera/compare/v0.18.0...v0.19.0

Added

- [SQL] CREATE LOCAL VIEW statement to declare views which are not outputs. ([1851](https://github.com/feldera/feldera/pull/1851))
- Documentation: added the [fraud detection use case](https://feldera.com/docs/use_cases/fraud_detection) to the docs section of the website.
- API: storage class can be configured via pipeline resources ([1829](https://github.com/feldera/feldera/pull/1829))
- [Python] Added support for Kafka connector via Python SDK ([1807](https://github.com/feldera/feldera/pull/1807))
- [Python] Added support for HTTP GET connector via Python SDK
- [Python] Add support for Avro format ([1836](https://github.com/feldera/feldera/pull/1836))
- [Python] allow specifying pipeline resource configuration and compilation profile via SQLContext ([1835](https://github.com/feldera/feldera/pull/1835))
- [Python] Add function to wait for the pipeline to become idle ([1841](https://github.com/feldera/feldera/pull/1841))
- WebConsole: DeltaLake input (1744) and output (1584) conenctors
- WebConsole: Add output buffer configuration UI (1591)

Fixed

- WebConsole: Fix crash when displaying anonymous output columns (1757)

Added

- Added backpressure mode to the `/egress` endpoint, which applies backpressure on the pipeline if the HTTP client cannot keep up
with the output instead of dropping data. ([1780](https://github.com/feldera/feldera/pull/1780))
- `/heap_profile` endpoint that generates the heap profile of a pipeline in the pprof format. ([1767](https://github.com/feldera/feldera/pull/1767))
- Python SDK for Feldera ([1745](https://github.com/feldera/feldera/pull/1745)).