This Release of the Kafka toolkit contains following enhancements:
KafkaProducer
1. Metric reporting
The Kafka producer in the client collects various performance metrics. A subset has been exposed as custom metrics to the operator (issue 112).
Custom Metric name | Description
-- | --
connection-count | The current number of active connections.
compression-rate-avg | The average compression rate of record batches (as percentage, 100 means no compression).
topic:compression-rate | The average compression rate of record batches for a topic (as percentage, 100 means no compression).
record-queue-time-avg | The average time in ms record batches spent in the send buffer.
record-queue-time-max | The maximum time in ms record batches spent in the send buffer.
record-send-rate | The average number of records sent per second.
record-retry-total | The total number of retried record sends
*topic*:record-send-total | The total number of records sent for a topic.
*topic*:record-retry-total | The total number of retried record sends for a topic
*topic*:record-error-total | The total number of record sends that resulted in errors for a topic
records-per-request-avg | The average number of records per request.
requests-in-flight | The current number of in-flight requests awaiting a response.
request-rate | The number of requests sent per second
request-size-avg | The average size of requests sent.
request-latency-avg | The average request latency in ms
request-latency-max | The maximum request latency in ms
batch-size-avg | The average number of bytes sent per partition per-request.
outgoing-byte-rate | The number of outgoing bytes sent to all servers per second
bufferpool-wait-time-total | The total time an appender waits for space allocation.
buffer-available-bytes | The total amount of buffer memory that is not being used (either unallocated or in the free list).
2. Default producer configs
Previous releases of the toolkit have used the Kafka default producer configs unless otherwise configured by the user. For optimum throughput these settings had to be tuned. Now, the important producer configs have default values, which result in higher throughput to the broker and reliability (issue 113):
Property name | Kafka default | New operator default
-- | -- | --
retries | `0` | `10`. When `0` is provided as **retries** and **consistentRegionPolicy** parameter is `Transactional` **retries** is adjusted to `1`.
compression.type | `none` | `lz4`
linger.ms | `0` | `100`
batch.size | `16384` | `32768`
max.in.flight.requests.per.connection | `5` | `1` when **guaranteeOrdering** parameter is `true`, limited to `5` when provided and **consistentRegionPolicy** parameter is `Transactional`, or `10` in all other cases.
3. New optional operator parameter **guaranteeOrdering**
If set to true, the operator guarantees that the order of records within a topic partition is the same as the order of processed tuples when it comes to retries. This implies that the operator sets the **max.in.flight.requests.per.connection** producer config automatically to `1` if retries are enabled, i.e. when the **retries** config is unequal 0, what is the operator default value.
If unset, the default value of this parameter is false, which means that the order can change due to retries.
4. Queue time control
In previous releases including 1.5.1, the producer operator was easily damageable when the producer did not come up transferring the data to the broker nodes. Then records stayed to long in the accumulator (basically a buffer), what caused timeouts with subsequent restarts (issue 128).
The producer now has an adaptive control that monitors several producer metrics and flushes the producer on occasion to ensure that the maximum queue time of records stays below typically 5 seconds. This enhancement addresses the robustness of the producer operator.
KafkaConsumer
1. Metric reporting
The Kafka consumer in the client collects various performance metrics. A subset has been exposed as custom metrics to the operator:
Custom Metric name | Description
-- | --
connection-count | The current number of active connections.
incoming-byte-rate | The number of bytes read off all sockets per second
*topic-partition*:records-lag | The latest lag of the partition
records-lag-max | The maximum lag in terms of number of records for any partition in this window
fetch-size-avg | The average number of bytes fetched per request
*topic*:fetch-size-avg | The average number of bytes fetched per request for a topic
commit-rate | The number of commit calls per second
commit-latency-avg | The average time taken for a commit request
One of the most interesting metric is the record lag for every consumed topic partition. The lag is the difference between the offset of the last inserted record and current reading position.