Quixstreams

What's Changed
Fixes
* Fix `QuixEnvironmentSource` behavior when streaming data from one Quix environment to another by quentin-quix in https://github.com/quixio/quix-streams/pull/520
* Fix consumers not fetching the data when connecting to the Quix broker by temporarily downgrading confluent-kafka to 2.4.0 by daniil-quix in https://github.com/quixio/quix-streams/pull/522

Other changes
* Update custom-sources.md by mikerosam in https://github.com/quixio/quix-streams/pull/509
* Add an example of custom websocket source by daniil-quix in https://github.com/quixio/quix-streams/pull/505
* Update README.md by daniil-quix in 512, 514
* Script to test Conda build by gwaramadze in https://github.com/quixio/quix-streams/pull/493
* Document writing custom source in a Jupyter Notebook by quentin-quix in https://github.com/quixio/quix-streams/pull/518
* Update Documentation by github-actions in https://github.com/quixio/quix-streams/pull/521


**Full Changelog**: https://github.com/quixio/quix-streams/compare/v2.11.0...v2.11.1

What's Changed
[New] Source API and built-in Sources

With the new Sources API, you can stream data from any data source to a Kafka topic and process it with Streaming DataFrames in the same application.

You can either use one of the built-in sources (e.g. [KafkaReplicatorSource](https://quix.io/docs/quix-streams/connectors/sources/kafka-source.html), [CSVSource](https://quix.io/docs/quix-streams/connectors/sources/csv-source.html), [QuixEnvironmentSource](https://quix.io/docs/quix-streams/connectors/sources/quix-source.html)) or create a custom one.

To learn more about Sources, please see the [Sources documentation](https://quix.io/docs/quix-streams/connectors/sources/index.html)

PRs: 420, 448, 490 , 494 , 495, 498, 506

Dependencies updates
* Bump testcontainers from 4.8.0 to 4.8.1 by dependabot in https://github.com/quixio/quix-streams/pull/492
* Update pydantic requirement from <2.9,>=2.7 to >=2.7,<2.10 by dependabot in https://github.com/quixio/quix-streams/pull/491
* Update pydantic-settings requirement from <2.5,>=2.3 to >=2.3,<2.6 by dependabot in https://github.com/quixio/quix-streams/pull/500

Documentation updates
* Fix typo in FixedTimeWindowDefinition.reduce docstring by gwaramadze in https://github.com/quixio/quix-streams/pull/501
* Fix documentation link by gwaramadze in https://github.com/quixio/quix-streams/pull/478
* Update README.md by daniil-quix in https://github.com/quixio/quix-streams/pull/479

Other changes
* Add Conda configuration by gwaramadze in https://github.com/quixio/quix-streams/pull/482
* Optimize get_window_ranges function by gwaramadze in https://github.com/quixio/quix-streams/pull/489


**Full Changelog**: https://github.com/quixio/quix-streams/compare/v2.10.0...v2.11.0

What's Changed

Schema Registry Support
Introduced Schema Registry support for JSONSchema, Avro, and Protobuf formats.

To learn how to use Schema Registry, please follow the docs on the [Schema Registry page.](https://quix.io/docs/quix-streams/advanced/schema-registry.html).

PRs: 447, 449, 451, 454, 458, 472, 476).

Dependencies updates
* Support confluent-kafka versions 2.5.x by gwaramadze in https://github.com/quixio/quix-streams/pull/459
* Bump testcontainers from 4.5.1 to 4.8.0 by dependabot in https://github.com/quixio/quix-streams/pull/462
* Update pydantic requirement from <2.8,>=2.7 to >=2.7,<2.9 by dependabot in https://github.com/quixio/quix-streams/pull/463
* Update pydantic-settings requirement from <2.4,>=2.3 to >=2.3,<2.5 by dependabot in https://github.com/quixio/quix-streams/pull/464
* Update pre-commit requirement from <3.5,>=3.4 to >=3.4,<3.9 by dependabot in https://github.com/quixio/quix-streams/pull/465
* Update black requirement from <24.4,>=24.3.0 to >=24.3.0,<24.9 by dependabot in https://github.com/quixio/quix-streams/pull/466


Documentation updates

* fix(docs): minor correction in an example by shrutimantri in https://github.com/quixio/quix-streams/pull/444
* fix(docs): correcting the output showcased for word count with other minor corrections by shrutimantri in https://github.com/quixio/quix-streams/pull/445
* Update docs headers structure by daniil-quix in https://github.com/quixio/quix-streams/pull/456


Other changes
* Application config API by quentin-quix in https://github.com/quixio/quix-streams/pull/470


**Full Changelog**: https://github.com/quixio/quix-streams/compare/v2.9.0...v2.10.0

What's Changed
NEW: Optional installs (extras)

With this release, we have introduced optional requirements for various features. These requirements will be outlined alongside its given feature.

To install one, simply do `pip install quixstreams[{extra}]` (or a comma-separated list like `extra1,extra2`)

There is also an option to install all extras with extra=`all` (`pip install quixstreams[all]`)

Features
More Message Serialization Options

Additional serialization options have been added:

- `JSON Schema` (original plain `JSON` option still supported)
- `Avro` (requires installed extra=`avro`)
- `Protobuf` (requires installed extra=`protobuf`)

For more details on their usage, [see the Serialization docs](https://quix.io/docs/quix-streams/advanced/serialization.html).


Sinks (beta)

>NOTE: This feature is in beta; functionality may change at any time!

We have introduced a new `Sink` API/framework for sending data from Kafka to an external destination in a robust manner. It additionally includes a template/class for users to generate their own sink implementations!

We have also included two fully implemented sinks for use out of the box:

- `InfluxDB v3`
- `CSV`

Example usage with `InfluxDB v3`:

python
from quixstreams import Application
from quixstreams.sinks.influxdb3 import InfluxDB3Sink

app = Application(broker_address="localhost:9092")
topic = app.topic("numbers-topic")

Initialize InfluxDB3Sink
influx_sink = InfluxDB3Sink(...params...)

sdf = app.dataframe(topic)
Do some processing here ...
Sink data to InfluxDB
sdf.sink(influx_sink)


For more details on their usage, [see the Sinks docs](https://quix.io/docs/quix-streams/connectors/sinks/index.html)

`commit_every` option for `Applications`

Applications can now commit every `M` consumed messages in addition to every `N` seconds (whichever occurs first for that checkpoint).

By default, it is `0`, which means no limit (how it worked before introducing this setting).

For more details, [see the Checkpoint docs](https://quix.io/docs/quix-streams/advanced/checkpointing.html#configuring-the-checkpointing)

python
app = Application(commit_every=10000)


`errors` option for `StreamingDataFrame.drop()`

You can now ignore the default behavior of an exception being raised when the specified column(s) are missing with `errors="ignore"`.

python
app = Application()
sdf = app.dataframe()
sdf = sdf.drop(["col_a", "col_b"], errors="ignore")



Enhancements
* README updates
* Various Documentation improvements



Changelog
**Full Changelog**: https://github.com/quixio/quix-streams/compare/v2.8.1...v2.9.0

What's Changed
Bugfixes
* fix `Topic.deserialize` not using the correct value deserializer by tim-quix in https://github.com/quixio/quix-streams/pull/413

Other changes
* Update docs by daniil-quix in https://github.com/quixio/quix-streams/pull/406
* fix bad timestamp values in groupby tests by tim-quix in https://github.com/quixio/quix-streams/pull/412


**Full Changelog**: https://github.com/quixio/quix-streams/compare/v2.8.0...v2.8.1

What's Changed

`.update()` and `.to_topic()` now modify the `StreamingDataFrame` objects in-place.
In previous versions, methods like `StreamingDataFrame.update()` and `StreamingDataFrame.to_topic()` always returned a new SDF object.
We were getting feedback that this behavior is not always obvious, and it's easy to forget to re-assign the result of `.update()` or `.to_topic()` calls to the variable.
Now, both of these methods modify the SDF object in place and return itself, so the previous usage will still work:


sdf = app.dataframe(...)

The SDF is now modified in place and it's not necessary to re-assign it to the variable
sdf.update(...)
sdf.to_topic(...)

This code will keep working as before
sdf = sdf.update(...)
sdf = sdf.to_topic(...)
or
sdf = sdf.update(...).to_topic(...)


⚠️ **Note:** If there's an `sdf.update()`or `sdf.to_topic()` in the code not assigned back to the variable, it will now update the SDF instance.

----
New method `StreamingDataFrame.print()`

Users can now easily debug the value and metadata of the current record in the stream with `StreamingDataFrame.print()`.
By default, it will print the current record value wrapped into a dictionary.
If called with `metadata=True,` it will also print the record's key, timestamp, and headers.

It's aimed to be a shortcut for the previous workaround with `StreamingDataFrame.update(print)`.

Example:

sdf = app.dataframe(...)

Print only the value of the current record
sdf.print()

>>> {'value': {'number': 163937}}
{'value': {'number': 163938}}


Print value and metadata of the current record
sdf.print(metadata=True)

>>> { 'value': {'number': 12175},
'key': b'd22d884a-eb88-44de-b22f-abfdc0b215f6',
'timestamp': 1721129697926,
'headers': [('header', b'123'), ('header', b'1234')]}

-----
New method `StreamingDataFrame.drop()`
With `StreamingDataFrame.drop()`, users can easily drop the unnecessary keys from the dictionary-like values.

`StreamingDataFrame.drop()` mutates data in place by deleting the keys from the original dictionary.


sdf = app.dataframe(...)

sdf.drop('A') Drop a key "A" from the value assuming it's a dict

sdf.drop(['A', 'B']) Drop multiple keys from the value


----
Other changes

* Fix doc formatting for processing guarantees by stereosky in https://github.com/quixio/quix-streams/pull/396
* Update doc reference to exactly once delivery guarantees by stereosky in https://github.com/quixio/quix-streams/pull/397
* Optimize tree() method of Stream class to reduce time complexity by aparnadotk in https://github.com/quixio/quix-streams/pull/400
* Consumer docstring cleanup by tim-quix in https://github.com/quixio/quix-streams/pull/394

New Contributors
* aparnadotk made their first contribution in https://github.com/quixio/quix-streams/pull/400

**Full Changelog**: https://github.com/quixio/quix-streams/compare/v2.7.0...v2.8.0