Kafka-python

Improvements
* Add optional compression libs to extras_require (2123, 2387)
* KafkaConsumer: Exit poll if consumer is closed (2152)
* Support configuration of custom kafka client for Admin/Consumer/Producer (2144)
* Core Protocol: Add support for flexible versions (2151)
* (Internal) Allow disabling thread wakeup in _send_request_to_node (2335)
* Change loglevel of cancelled errors to info (2467)
* Strip trailing dot off hostname for SSL validation. (2472)
* Log connection close(error) at ERROR level (2473)
* Support DescribeLogDirs admin api (2475)

Compatibility
* Support for python 3.12 (2379, 2382)
* Kafka 2.5 / 2.6 (2162)
* Try collections.abc imports in vendored selectors34 (2394)
* Catch OSError when checking for gssapi import for windows compatibility (2407)
* Update vendored six to 1.16.0 (2398)

Documentation
* Update usage.rst (2308, 2334)
* Fix typos (2319, 2207, 2178)
* Fix links to the compatibility page (2295, 2226)
* Cleanup install instructions for optional libs (2139)
* Update license_file to license_files (2462)
* Update some RST documentation syntax (2463)
* Add .readthedocs.yaml; update copyright date (2474)

Fixes
* Use isinstance in builtin crc32 (2329)
* Use six.viewitems instead of six.iteritems to avoid encoding problems in StickyPartitionAssignor (2154)
* Fix array encoding TypeError: object of type 'dict_itemiterator' has no len() (2167)
* Only try to update sensors fetch lag if the unpacked list contains elements (2158)
* Avoid logging errors during test fixture cleanup (2458)
* Release coordinator lock before calling maybe_leave_group (2460)
* Dont raise RuntimeError for dead process in SpawnedService.wait_for() (2461)
* Cast the size of a MemoryRecordsBuilder object (2438)
* Fix DescribeConfigsResponse_v1 config_source (2464)
* Fix base class of DescribeClientQuotasResponse_v0 (2465)
* Update socketpair w/ CVE-2024-3219 fix (2468)

Testing
* Transition CI/CD to GitHub Workflows (2378, 2392, 2381, 2406, 2419, 2418, 2417, 2456)
* Refactor Makefile (2457)
* Use assert_called_with in client_async tests (2375)
* Cover sticky assignor's metadata method with tests (2161)
* Update fixtures.py to check "127.0.0.1" for auto port assignment (2384)
* Use -Djava.security.manager=allow for Java 23 sasl tests (2469)
* Test with Java 23 (2470)
* Update kafka properties template; disable group rebalance delay (2471)

Consumer
* KIP-54: Implement sticky partition assignment strategy (aynroot / PR 2057)
* Fix consumer deadlock when heartbeat thread request timeout (huangcuiyang / PR 2064)

Compatibility
* Python 3.8 support (Photonios / PR 2088)

Cleanups
* Bump dev requirements (jeffwidman / PR 2129)
* Fix crc32c deprecation warning (crc32c==2.1) (jeffwidman / PR 2128)
* Lint cleanup (jeffwidman / PR 2126)
* Fix initialization order in KafkaClient (pecalleja / PR 2119)
* Allow installing crc32c via extras (mishas / PR 2069)
* Remove unused imports (jameslamb / PR 2046)

Admin Client
* Merge _find_coordinator_id methods (jeffwidman / PR 2127)
* Feature: delete consumergroups (swenzel / PR 2040)
* Allow configurable timeouts in admin client check version (sunnyakaxd / PR 2107)
* Enhancement for Kafka Admin Client's "Describe Consumer Group" (Apurva007 / PR 2035)

Protocol
* Add support for zstd compression (gabriel-tincu / PR 2021)
* Add protocol support for brokers 1.1.0 - 2.5.0 (gabriel-tincu / PR 2038)
* Add ProduceRequest/ProduceResponse v6/v7/v8 (gabriel-tincu / PR 2020)
* Fix parsing NULL header values (kvfi / PR 2024)

Tests
* Add 2.5.0 to automated CI tests (gabriel-tincu / PR 2038)
* Add 2.1.1 to build_integration (gabriel-tincu / PR 2019)

Documentation / Logging / Errors
* Disable logging during producer object gc (gioele / PR 2043)
* Update example.py; use threading instead of multiprocessing (Mostafa-Elmenbawy / PR 2081)
* Fix typo in exception message (haracejacob / PR 2096)
* Add kafka.structs docstrings (Mostafa-Elmenbawy / PR 2080)
* Fix broken compatibility page link (anuragrana / PR 2045)
* Rename README to README.md (qhzxc0015 / PR 2055)
* Fix docs by adding SASL mention (jeffwidman / 1990)

Admin Client
* KAFKA-8962: Use least_loaded_node() for AdminClient.describe_topics() (jeffwidman / PR 2000)
* Fix AdminClient topic error parsing in MetadataResponse (jtribble / PR 1997)

This release includes breaking changes for any application code that has not
migrated from older Simple-style classes to newer Kafka-style classes.

Deprecation
* Remove deprecated SimpleClient, Producer, Consumer, Unittest (jeffwidman / PR 1196)

Admin Client
* Use the controller for topic metadata requests (TylerLubeck / PR 1995)
* Implement list_topics, describe_topics, and describe_cluster (TylerLubeck / PR 1993)
* Implement __eq__ and __hash__ for ACL objects (TylerLubeck / PR 1955)
* Fixes KafkaAdminClient returning `IncompatibleBrokerVersion` when passing an `api_version` (ian28223 / PR 1953)
* Admin protocol updates (TylerLubeck / PR 1948)
* Fix describe config for multi-broker clusters (jlandersen / PR 1869)

Miscellaneous Bugfixes / Improvements
* Enable SCRAM-SHA-256 and SCRAM-SHA-512 for sasl (swenzel / PR 1918)
* Fix slots usage and use more slots (carsonip / PR 1987)
* Optionally return OffsetAndMetadata from consumer.committed(tp) (dpkp / PR 1979)
* Reset conn configs on exception in conn.check_version() (dpkp / PR 1977)
* Do not block on sender thread join after timeout in producer.close() (dpkp / PR 1974)
* Implement methods to convert a Struct object to a pythonic object (TylerLubeck / PR 1951)

Test Infrastructure / Documentation / Maintenance
* Update 2.4.0 resource files for sasl integration (dpkp)
* Add kafka 2.4.0 to CI testing (vvuibert / PR 1972)
* convert test_admin_integration to pytest (ulrikjohansson / PR 1923)
* xfail test_describe_configs_topic_resource_returns_configs (dpkp / Issue 1929)
* Add crc32c to README and docs (dpkp)
* Improve docs for reconnect_backoff_max_ms (dpkp / PR 1976)
* Fix simple typo: managementment -> management (timgates42 / PR 1966)
* Fix typos (carsonip / PR 1938)
* Fix doc import paths (jeffwidman / PR 1933)
* Update docstring to match conn.py's (dabcoder / PR 1921)
* Do not log topic-specific errors in full metadata fetch (dpkp / PR 1980)
* Raise AssertionError if consumer closed in poll() (dpkp / PR 1978)
* Log retriable coordinator NodeNotReady, TooManyInFlightRequests as debug not error (dpkp / PR 1975)
* Remove unused import (jeffwidman)
* Remove some dead code (jeffwidman)
* Fix a benchmark to Use print() function in both Python 2 and Python 3 (cclauss / PR 1983)
* Fix a test to use ==/!= to compare str, bytes, and int literals (cclauss / PR 1984)
* Fix benchmarks to use pyperf (carsonip / PR 1986)
* Remove unused/empty .gitsubmodules file (jeffwidman / PR 1928)
* Remove deprecated `ConnectionError` (jeffwidman / PR 1816)

This is a minor release focused on KafkaConsumer performance, Admin Client
improvements, and Client concurrency. The KafkaConsumer iterator implementation
has been greatly simplified so that it just wraps consumer.poll(). The prior
implementation will remain available for a few more releases using the optional
KafkaConsumer config: `legacy_iterator=True` . This is expected to improve
consumer throughput substantially and help reduce heartbeat failures / group
rebalancing.

Client
* Send socket data via non-blocking IO with send buffer (dpkp / PR 1912)
* Rely on socket selector to detect completed connection attempts (dpkp / PR 1909)
* Improve connection lock handling; always use context manager (melor,dpkp / PR 1895)
* Reduce client poll timeout when there are no in-flight requests (dpkp / PR 1823)

KafkaConsumer
* Do not use wakeup when sending fetch requests from consumer (dpkp / PR 1911)
* Wrap `consumer.poll()` for KafkaConsumer iteration (dpkp / PR 1902)
* Allow the coordinator to auto-commit on old brokers (justecorruptio / PR 1832)
* Reduce internal client poll timeout for (legacy) consumer iterator interface (dpkp / PR 1824)
* Use dedicated connection for group coordinator (dpkp / PR 1822)
* Change coordinator lock acquisition order (dpkp / PR 1821)
* Make `partitions_for_topic` a read-through cache (Baisang / PR 1781,1809)
* Fix consumer hanging indefinitely on topic deletion while rebalancing (commanderdishwasher / PR 1782)

Miscellaneous Bugfixes / Improvements
* Fix crc32c avilability on non-intel architectures (ossdev07 / PR 1904)
* Load system default SSL CAs if `ssl_cafile` is not provided (iAnomaly / PR 1883)
* Catch py3 TimeoutError in BrokerConnection send/recv (dpkp / PR 1820)
* Added a function to determine if bootstrap is successfully connected (Wayde2014 / PR 1876)

Admin Client
* Add ACL api support to KafkaAdminClient (ulrikjohansson / PR 1833)
* Add `sasl_kerberos_domain_name` config to KafkaAdminClient (jeffwidman / PR 1852)
* Update `security_protocol` config documentation for KafkaAdminClient (cardy31 / PR 1849)
* Break FindCoordinator into request/response methods in KafkaAdminClient (jeffwidman / PR 1871)
* Break consumer operations into request / response methods in KafkaAdminClient (jeffwidman / PR 1845)
* Parallelize calls to `_send_request_to_node()` in KafkaAdminClient (davidheitman / PR 1807)

Test Infrastructure / Documentation / Maintenance
* Add Kafka 2.3.0 to test matrix and compatibility docs (dpkp / PR 1915)
* Convert remaining `KafkaConsumer` tests to `pytest` (jeffwidman / PR 1886)
* Bump integration tests to 0.10.2.2 and 0.11.0.3 (jeffwidman / 1890)
* Cleanup handling of `KAFKA_VERSION` env var in tests (jeffwidman / PR 1887)
* Minor test cleanup (jeffwidman / PR 1885)
* Use `socket.SOCK_STREAM` in test assertions (iv-m / PR 1879)
* Sanity test for `consumer.topics()` and `consumer.partitions_for_topic()` (Baisang / PR 1829)
* Cleanup seconds conversion in client poll timeout calculation (jeffwidman / PR 1825)
* Remove unused imports (jeffwidman / PR 1808)
* Cleanup python nits in RangePartitionAssignor (jeffwidman / PR 1805)
* Update links to kafka consumer config docs (jeffwidman)
* Fix minor documentation typos (carsonip / PR 1865)
* Remove unused/weird comment line (jeffwidman / PR 1813)
* Update docs for `api_version_auto_timeout_ms` (jeffwidman / PR 1812)

This is a patch release primarily focused on bugs related to concurrency,
SSL connections and testing, and SASL authentication:

Client Concurrency Issues (Race Conditions / Deadlocks)
* Fix race condition in `protocol.send_bytes` (isamaru / PR 1752)
* Do not call `state_change_callback` with lock (dpkp / PR 1775)
* Additional BrokerConnection locks to synchronize protocol/IFR state (dpkp / PR 1768)
* Send pending requests before waiting for responses (dpkp / PR 1762)
* Avoid race condition on `client._conns` in send() (dpkp / PR 1772)
* Hold lock during `client.check_version` (dpkp / PR 1771)

Producer Wakeup / TimeoutError
* Dont wakeup during `maybe_refresh_metadata` -- it is only called by poll() (dpkp / PR 1769)
* Dont do client wakeup when sending from sender thread (dpkp / PR 1761)

SSL - Python3.7 Support / Bootstrap Hostname Verification / Testing
* Wrap SSL sockets after connecting for python3.7 compatibility (dpkp / PR 1754)
* Allow configuration of SSL Ciphers (dpkp / PR 1755)
* Maintain shadow cluster metadata for bootstrapping (dpkp / PR 1753)
* Generate SSL certificates for local testing (dpkp / PR 1756)
* Rename ssl.keystore.location and ssl.truststore.location config files (dpkp)
* Reset reconnect backoff on SSL connection (dpkp / PR 1777)

SASL - OAuthBearer support / api version bugfix
* Fix 0.8.2 protocol quick detection / fix SASL version check (dpkp / PR 1763)
* Update sasl configuration docstrings to include supported mechanisms (dpkp)
* Support SASL OAuthBearer Authentication (pt2pham / PR 1750)

Miscellaneous Bugfixes
* Dont force metadata refresh when closing unneeded bootstrap connections (dpkp / PR 1773)
* Fix possible AttributeError during conn._close_socket (dpkp / PR 1776)
* Return connection state explicitly after close in connect() (dpkp / PR 1778)
* Fix flaky conn tests that use time.time (dpkp / PR 1758)
* Add py to requirements-dev (dpkp)
* Fixups to benchmark scripts for py3 / new KafkaFixture interface (dpkp)