Risingwave

For installation and running instructions, see [Get started](https://docs.risingwave.com/docs/current/get-started/).

Main changes

SQL features
SQL features

- Query syntax:
- Supports using subqueries in `UPDATE` and `DELETE` statements. (https://github.com/risingwavelabs/risingwave/pull/12995)
- SQL commands
- Supports `COMMENT ON` clause for tables and columns. (https://github.com/risingwavelabs/risingwave/pull/12849)
- Supports persistent background materialized views. (https://github.com/risingwavelabs/risingwave/pull/12167)
- Exposes hidden columns and distribution keys when using `SHOW COLUMNS FROM` command. (https://github.com/risingwavelabs/risingwave/pull/12839)
- Exposes hidden columns when using `DESCRIBE` command. (https://github.com/risingwavelabs/risingwave/pull/12839)
- SQL functions & operators
- Supports `substring` and `substr` functions for `bytea` data type. (https://github.com/risingwavelabs/risingwave/pull/13088)
- Supports functions `jsonb_pretty`,  `jsonb_object`, `jsonb_strip_nulls`, and `jsonb_extract_path`. (https://github.com/risingwavelabs/risingwave/pull/13050), (https://github.com/risingwavelabs/risingwave/pull/13036), (https://github.com/risingwavelabs/risingwave/pull/13169), (https://github.com/risingwavelabs/risingwave/pull/13143)
- Supports jsonb `>`, `<`, `?`, `?|, ?&`, `>`,  `>>`, `-` and `-` operators. (https://github.com/risingwavelabs/risingwave/pull/13056), (https://github.com/risingwavelabs/risingwave/pull/13110), (https://github.com/risingwavelabs/risingwave/pull/13118)
- Supports `greatest` and `least` functions. (https://github.com/risingwavelabs/risingwave/pull/12838)
- Supports `regexp_split_to_array` function. (https://github.com/risingwavelabs/risingwave/pull/12844)
- Supports `bit_and` and `bit_or` aggregate functions in materialized views. (https://github.com/risingwavelabs/risingwave/pull/12758)
- Supports `jsonb_agg` and `jsonb_object_agg` in streaming mode. (https://github.com/risingwavelabs/risingwave/pull/12836)
- Supports general `rank` and `dense_rank` window functions. (https://github.com/risingwavelabs/risingwave/pull/13183)
- System catalog
- Adds column `parallelism` in system table `rw_fragments`. (https://github.com/risingwavelabs/risingwave/pull/12901)
- Adds columns `is_hidden`, `is_primary_key` and `is_distribution_key` in `rw_columns` system table. (https://github.com/risingwavelabs/risingwave/pull/12839)

Sources & sinks
- Adds `google.protobuf.Any` support for Protobuf sources. (https://github.com/risingwavelabs/risingwave/pull/12291)
- Adds `schemas.enable` support for Kafka sinks with upsert json. (https://github.com/risingwavelabs/risingwave/pull/12113)
- Adds support for Kafka sinks with Upsert Avro using schema registry. (https://github.com/risingwavelabs/risingwave/pull/13007)
- `server.id` option is now optional for MySQL CDC source. (https://github.com/risingwavelabs/risingwave/pull/13031)
- Enables `timestamptz.handling.mode` option to control the timestamptz output format for certain sinks. (https://github.com/risingwavelabs/risingwave/pull/13109)
- Adds the `stream` field and support for multiple inputs for the `subject` field for NATS source connector. (https://github.com/risingwavelabs/risingwave/pull/12799)
- Adds new option `properties.allow.auto.create.topics` for Kafka sink. (https://github.com/risingwavelabs/risingwave/pull/12766)
- Adds support for `s3_v2` source connector, a more efficient version of the S3 source. (https://github.com/risingwavelabs/risingwave/pull/12595)
- Adds support for Google BigQuery sink. (https://github.com/risingwavelabs/risingwave/pull/12873)
- Adds support for Redis sink. (https://github.com/risingwavelabs/risingwave/pull/11999),(https://github.com/risingwavelabs/risingwave/pull/13003)

Deployment
- Release RisingWave all-in-one binary with connector libraries. (https://github.com/risingwavelabs/risingwave/pull/13133)

**Full Changelog**: https://github.com/risingwavelabs/risingwave/compare/v1.3.0...v1.4.0

For installation and running instructions, see [Get started](https://docs.risingwave.com/docs/current/get-started/).

Main changes

SQL features

- SQL commands
- Experimental: Variable `BACKGROUND_DDL` can be set to `true` when creating a materialized view. https://github.com/risingwavelabs/risingwave/pull/12355
- `ALTER COLUMN` command can be used for tables with non-schema-registry source. https://github.com/risingwavelabs/risingwave/pull/12164
- SQL functions & operators
- Supports `array_min`. https://github.com/risingwavelabs/risingwave/pull/12071
- Supports `array_max`. https://github.com/risingwavelabs/risingwave/pull/12100
- Supports `array_sort`. https://github.com/risingwavelabs/risingwave/pull/12189
- Supports `array_sum`. https://github.com/risingwavelabs/risingwave/pull/12162
- `format` function supports variable inputs. https://github.com/risingwavelabs/risingwave/pull/12178
- Regular expression functions support back reference, positive, negative lookahead, and positive, negative lookbehind. https://github.com/risingwavelabs/risingwave/pull/12329
- Supports `||` operator for concatenating JSONB data. https://github.com/risingwavelabs/risingwave/pull/12502
- Supports `bool_and` and `bool_or` in materialized views. https://github.com/risingwavelabs/risingwave/pull/11956
- Query syntax:
- Supports `WITH ORDINALITY` clause. https://github.com/risingwavelabs/risingwave/pull/12273
- System catalog
- Adds new system function `pg_sleep`. https://github.com/risingwavelabs/risingwave/pull/12294
- Adds new system function `_pg_expandarray`. https://github.com/risingwavelabs/risingwave/pull/12448
- Adds new storage related system tables:
- `rw_hummock_sstables` https://github.com/risingwavelabs/risingwave/pull/12532
- `rw_hommock_pinned_versions`, `rw_hommock_pinned_snapshots` https://github.com/risingwavelabs/risingwave/pull/12285
- `rw_hummock_branched_objects` , `rw_hummock_current_version` , `rw_hummock_checkpoint_version` , `rw_hummock_version_deltas` https://github.com/risingwavelabs/risingwave/pull/12309
- `rw_hummock_meta_configs`, `rw_hummock_compaction_group_configs` https://github.com/risingwavelabs/risingwave/pull/12337

Sources & sinks
- Generated columns defined with non-deterministic functions cannot be part of the primary key. https://github.com/risingwavelabs/risingwave/pull/12181
- Adds new `properties.enable.auto.commit` parameter for the Kafka consumer, which sets the `enable.auto.commit` parameter for the Kafka client. https://github.com/risingwavelabs/risingwave/pull/12223
- Adds `privatelink.endpoint` parameter to the WITH clause, to support private link for Kafka connector on GCP and AWS. https://github.com/risingwavelabs/risingwave/pull/12266
- Adds parameters `[message.timeout.ms](http://message.timeout.ms)` and `max.in.flight.requests.per.connection` for Kafka sources. https://github.com/risingwavelabs/risingwave/pull/12574
- Allows Kinesis source to start ingesting data from a specific timestamp. `sequence_number` is no longer supported as a startup mode option. https://github.com/risingwavelabs/risingwave/pull/12241
- Allow optional `FORMAT DEBEZIUM ENCODE JSON` after the connector definition of CDC tables. Allow optional `FORMAT NATIVE ENCODE NATIVE` after the connector definition of Nexmark sources or tables. https://github.com/risingwavelabs/risingwave/pull/12306
- Allows multiple URLs when defining schema registries. https://github.com/risingwavelabs/risingwave/pull/11982
- Adds support for sinking data to versions 7 and 8 of Elasticsearch. https://github.com/risingwavelabs/risingwave/pull/10357, https://github.com/risingwavelabs/risingwave/pull/10415https://github.com/risingwavelabs/risingwave-docs/issues/1303
- Adds support for sinking append-only data to the NATS messaging system. https://github.com/risingwavelabs/risingwave/pull/11924
- Adds support for sinking data to Doris. https://github.com/risingwavelabs/risingwave/pull/12336
- Adds support for sinking data to Apache Pulsar. https://github.com/risingwavelabs/risingwave/pull/12286
- Adds support for sinking data to Cassandra and ScyllaDB. https://github.com/risingwavelabs/risingwave/pull/11878
- Adds support for creating upsert Iceberg sinks. https://github.com/risingwavelabs/risingwave/pull/12576
- Supports specifying the `sink_decouple` session variable as `default`, `true` and `enable`, or `false` and `disable`. https://github.com/risingwavelabs/risingwave/pull/12544
- A `varchar` column in RisingWave can sink into a `uuid` column in Postgres. https://github.com/risingwavelabs/risingwave/pull/12704
- New syntaxes for specifying data format and data encode when creating a Kafka, Kinesis, and Pulsar sink. https://github.com/risingwavelabs/risingwave/pull/12556

Administration & observability
Adds `information_schema.views`, which contains information about views defined in the database.

**Full Changelog**: https://github.com/risingwavelabs/risingwave/compare/v1.2.0...v1.3.0

For installation and running instructions, see [Get started](https://www.risingwave.dev/docs/latest/get-started/).

Main changes

SQL features

- SQL commands:
- Breaking change: Syntax of emit-on-window-close has changed. If your application contains integration code, please update your code accordingly. https://github.com/risingwavelabs/risingwave/pull/11363
In v1.1:

sql
CREATE MATERIALIZED VIEW mv
EMIT ON WINDOW CLOSE
AS SELECT
...;


In v1.2 and onwards:

sql
CREATE MATERIALIZED VIEW mv
AS SELECT
...
EMIT ON WINDOW CLOSE;


- Privileges for tables can now be granted or revoked. https://github.com/risingwavelabs/risingwave/pull/11725
- The default DISTRIBUTED BY columns have been changed from the whole index columns into the first index column. https://github.com/risingwavelabs/risingwave/pull/11865
- Supports `ALTER SOURCE ADD COLUMN`. https://github.com/risingwavelabs/risingwave/pull/11350
- Supports `SHOW JOBS` and `CANCEL JOBS` , with which you can show the in-progress streaming jobs and cancel jobs by their IDs. https://github.com/risingwavelabs/risingwave/pull/11854
- Supports [I]LIKE in SHOW commands. https://github.com/risingwavelabs/risingwave/pull/11791
- SQL functions & operators
- Supports lambda functions via `array_transform`. https://github.com/risingwavelabs/risingwave/pull/11888 https://github.com/risingwavelabs/risingwave/pull/11937
- `to_date()` https://github.com/risingwavelabs/risingwave/pull/11241
- The `to_char()` function now supports `timestamptz` input. https://github.com/risingwavelabs/risingwave/pull/11778
- `scale`, `min_scale` , and `trim_scale` https://github.com/risingwavelabs/risingwave/pull/11663
- Supports `regexp_replace` . https://github.com/risingwavelabs/risingwave/pull/11819
- Supports `regexp_count` . https://github.com/risingwavelabs/risingwave/pull/11975
- Supports `[NOT] ILIKE` expressions. https://github.com/risingwavelabs/risingwave/pull/11743
- Adds support for `[!]~~[*]` operators. They’ll be parsed to `[NOT] [I]LIKE` expressions. https://github.com/risingwavelabs/risingwave/pull/11748
- Supports `IS JSON` predicate. https://github.com/risingwavelabs/risingwave/pull/11831

- Query syntax:
- Adds support for LIMIT clauses in streaming queries. https://github.com/risingwavelabs/risingwave/pull/11566
- Supports `LATERAL` subqueries. https://github.com/risingwavelabs/risingwave/pull/11780

- System catalog
- A new group of system catalogs (`rw_relations`, `rw_system_tables`, `rw_types`, `rw_user_secrets`, and `rw_columns`) are available for you to retrieve system data and metadata. https://github.com/risingwavelabs/risingwave/pull/11334
- Adds new system function `pg_relation_size()` . https://github.com/risingwavelabs/risingwave/pull/11687
- Adds support for transactions for single-table CDC data. https://github.com/risingwavelabs/risingwave/pull/11453

Sources & sinks
- Adds a new parameter `schema.registry.name.strategy` to the Kafka connector, with with you can specify naming strategies for schema registries. https://github.com/risingwavelabs/risingwave/pull/11384
- Breaking Change: Implements a Rust-native Iceberg sink connector to improve stability and performance. The connector introduces new parameters. Applications that rely on the previous version of the feature (specifically, the version included in RisingWave v1.0.0 and v1.1) may no longer function correctly. To restore functionality to your applications, please carefully review the syntax and parameters outlined on this page and make any necessary revisions to your code. Please refer to [Sink data to Iceberg](https://docs.risingwave.com/docs/dev/sink-to-iceberg/) for details. https://github.com/risingwavelabs/risingwave/pull/11326
- Adds support for sinking data to ClickHouse. https://github.com/risingwavelabs/risingwave/pull/11240
- Experimental: An enhancement has been made to the mysql-cdc connector to improve data ingestion performance. It achieves so by optimizing the data backfilling logic for CDC tables. This feature is not enabled by default. To enable it, run this command: `SET cdc_backfill="true";` https://github.com/risingwavelabs/risingwave/pull/11707
- Adds a parameter `client.id` for Kafka sources. https://github.com/risingwavelabs/risingwave/pull/11911

Deployment
- Supports HDFS as the storage backend for deployments via Docker Compose. https://github.com/risingwavelabs/risingwave/pull/11632

Administration & observability
- Adds a new system parameter `max_concurrent_creating_streaming_jobs`, with which users can specify the maximum number of streaming jobs that can be created concurrently. https://github.com/risingwavelabs/risingwave/pull/11601
- Improves the calculation logic of the *Mem Table Size (Max)* metric in RisingWave Dashboard. https://github.com/risingwavelabs/risingwave/pull/11442
- Adds new metrics to RisingWave Dashboard:
- *Materialized View Memory Usage* https://github.com/risingwavelabs/risingwave/pull/10958
- *Materialized View Read Size*, *Materialized View Write Size* https://github.com/risingwavelabs/risingwave/pull/11054
- *Active Sessions* https://github.com/risingwavelabs/risingwave/pull/11688

**Full Changelog**: https://github.com/risingwavelabs/risingwave/compare/v1.1.0...v1.2.0