Kafka-python

Networking
* Remove unused client bootstrap backoff code
* 200ms timeout for client.poll in ensure_active_group and admin client

Fixes
* Admin client: check_version only if needed, use node_id kwarg for controller
* Check for -1 controller_id in admin client
* Only acquire coordinator lock in heartbeat thread close if not self thread

Testing
* Also sleep when waiting for consumers in test_describe_consumer_group_exists
* Refactor sasl_integration test_client - wait for node ready; use send future
* Add timeout to test_kafka_consumer
* Add error str to assert_message_count checks
* Retry on error in test fixture create_topic_via_metadata
* Fixup variable interpolation in test fixture error

Documentation
* Update compatibility docs
* Include client_id in BrokerConnection __str__ output

Project Maintenance
* Add make targets `servers/*/api_versions` and `servers/*/messages`

Networking
* Check for wakeup socket errors on read and close and reinit to reset (2482)
* Improve client networking backoff / retry (2480)
* Check for socket and unresolved futures before creating selector in conn.check_version (2477)
* Handle socket init errors, e.g., when IPv6 is disabled (2476)

Fixes
* Avoid self-join in heartbeat thread close (2488)

Error Handling
* Always log broker errors in producer.send (2478)
* Retain unrecognized broker response error codes with dynamic error class (2481)
* Update kafka.errors with latest types (2485)

Compatibility
* Do not validate snappy xerial header version and compat fields (for redpanda) (2483)

Documentation
* Added missing docstrings in admin/client.py (2487)

Testing
* Update kafka broker test matrix; test against 3.9.0 (2486)
* Add default resources for new kafka server fixtures (2484)
* Drop make test-local; add PYTESTS configuration var
* Fix pytest runs when KAFKA_VERSION is not set

Project Maintenance
* Migrate to pyproject.toml / PEP-621
* Remove old travis files; update compatibility tests link to gha

Improvements
* Add optional compression libs to extras_require (2123, 2387)
* KafkaConsumer: Exit poll if consumer is closed (2152)
* Support configuration of custom kafka client for Admin/Consumer/Producer (2144)
* Core Protocol: Add support for flexible versions (2151)
* (Internal) Allow disabling thread wakeup in _send_request_to_node (2335)
* Change loglevel of cancelled errors to info (2467)
* Strip trailing dot off hostname for SSL validation. (2472)
* Log connection close(error) at ERROR level (2473)
* Support DescribeLogDirs admin api (2475)

Compatibility
* Support for python 3.12 (2379, 2382)
* Kafka 2.5 / 2.6 (2162)
* Try collections.abc imports in vendored selectors34 (2394)
* Catch OSError when checking for gssapi import for windows compatibility (2407)
* Update vendored six to 1.16.0 (2398)

Documentation
* Update usage.rst (2308, 2334)
* Fix typos (2319, 2207, 2178)
* Fix links to the compatibility page (2295, 2226)
* Cleanup install instructions for optional libs (2139)
* Update license_file to license_files (2462)
* Update some RST documentation syntax (2463)
* Add .readthedocs.yaml; update copyright date (2474)

Fixes
* Use isinstance in builtin crc32 (2329)
* Use six.viewitems instead of six.iteritems to avoid encoding problems in StickyPartitionAssignor (2154)
* Fix array encoding TypeError: object of type 'dict_itemiterator' has no len() (2167)
* Only try to update sensors fetch lag if the unpacked list contains elements (2158)
* Avoid logging errors during test fixture cleanup (2458)
* Release coordinator lock before calling maybe_leave_group (2460)
* Dont raise RuntimeError for dead process in SpawnedService.wait_for() (2461)
* Cast the size of a MemoryRecordsBuilder object (2438)
* Fix DescribeConfigsResponse_v1 config_source (2464)
* Fix base class of DescribeClientQuotasResponse_v0 (2465)
* Update socketpair w/ CVE-2024-3219 fix (2468)

Testing
* Transition CI/CD to GitHub Workflows (2378, 2392, 2381, 2406, 2419, 2418, 2417, 2456)
* Refactor Makefile (2457)
* Use assert_called_with in client_async tests (2375)
* Cover sticky assignor's metadata method with tests (2161)
* Update fixtures.py to check "127.0.0.1" for auto port assignment (2384)
* Use -Djava.security.manager=allow for Java 23 sasl tests (2469)
* Test with Java 23 (2470)
* Update kafka properties template; disable group rebalance delay (2471)

Consumer
* KIP-54: Implement sticky partition assignment strategy (aynroot / PR 2057)
* Fix consumer deadlock when heartbeat thread request timeout (huangcuiyang / PR 2064)

Compatibility
* Python 3.8 support (Photonios / PR 2088)

Cleanups
* Bump dev requirements (jeffwidman / PR 2129)
* Fix crc32c deprecation warning (crc32c==2.1) (jeffwidman / PR 2128)
* Lint cleanup (jeffwidman / PR 2126)
* Fix initialization order in KafkaClient (pecalleja / PR 2119)
* Allow installing crc32c via extras (mishas / PR 2069)
* Remove unused imports (jameslamb / PR 2046)

Admin Client
* Merge _find_coordinator_id methods (jeffwidman / PR 2127)
* Feature: delete consumergroups (swenzel / PR 2040)
* Allow configurable timeouts in admin client check version (sunnyakaxd / PR 2107)
* Enhancement for Kafka Admin Client's "Describe Consumer Group" (Apurva007 / PR 2035)

Protocol
* Add support for zstd compression (gabriel-tincu / PR 2021)
* Add protocol support for brokers 1.1.0 - 2.5.0 (gabriel-tincu / PR 2038)
* Add ProduceRequest/ProduceResponse v6/v7/v8 (gabriel-tincu / PR 2020)
* Fix parsing NULL header values (kvfi / PR 2024)

Tests
* Add 2.5.0 to automated CI tests (gabriel-tincu / PR 2038)
* Add 2.1.1 to build_integration (gabriel-tincu / PR 2019)

Documentation / Logging / Errors
* Disable logging during producer object gc (gioele / PR 2043)
* Update example.py; use threading instead of multiprocessing (Mostafa-Elmenbawy / PR 2081)
* Fix typo in exception message (haracejacob / PR 2096)
* Add kafka.structs docstrings (Mostafa-Elmenbawy / PR 2080)
* Fix broken compatibility page link (anuragrana / PR 2045)
* Rename README to README.md (qhzxc0015 / PR 2055)
* Fix docs by adding SASL mention (jeffwidman / 1990)

Admin Client
* KAFKA-8962: Use least_loaded_node() for AdminClient.describe_topics() (jeffwidman / PR 2000)
* Fix AdminClient topic error parsing in MetadataResponse (jtribble / PR 1997)

This release includes breaking changes for any application code that has not
migrated from older Simple-style classes to newer Kafka-style classes.

Deprecation
* Remove deprecated SimpleClient, Producer, Consumer, Unittest (jeffwidman / PR 1196)

Admin Client
* Use the controller for topic metadata requests (TylerLubeck / PR 1995)
* Implement list_topics, describe_topics, and describe_cluster (TylerLubeck / PR 1993)
* Implement __eq__ and __hash__ for ACL objects (TylerLubeck / PR 1955)
* Fixes KafkaAdminClient returning `IncompatibleBrokerVersion` when passing an `api_version` (ian28223 / PR 1953)
* Admin protocol updates (TylerLubeck / PR 1948)
* Fix describe config for multi-broker clusters (jlandersen / PR 1869)

Miscellaneous Bugfixes / Improvements
* Enable SCRAM-SHA-256 and SCRAM-SHA-512 for sasl (swenzel / PR 1918)
* Fix slots usage and use more slots (carsonip / PR 1987)
* Optionally return OffsetAndMetadata from consumer.committed(tp) (dpkp / PR 1979)
* Reset conn configs on exception in conn.check_version() (dpkp / PR 1977)
* Do not block on sender thread join after timeout in producer.close() (dpkp / PR 1974)
* Implement methods to convert a Struct object to a pythonic object (TylerLubeck / PR 1951)

Test Infrastructure / Documentation / Maintenance
* Update 2.4.0 resource files for sasl integration (dpkp)
* Add kafka 2.4.0 to CI testing (vvuibert / PR 1972)
* convert test_admin_integration to pytest (ulrikjohansson / PR 1923)
* xfail test_describe_configs_topic_resource_returns_configs (dpkp / Issue 1929)
* Add crc32c to README and docs (dpkp)
* Improve docs for reconnect_backoff_max_ms (dpkp / PR 1976)
* Fix simple typo: managementment -> management (timgates42 / PR 1966)
* Fix typos (carsonip / PR 1938)
* Fix doc import paths (jeffwidman / PR 1933)
* Update docstring to match conn.py's (dabcoder / PR 1921)
* Do not log topic-specific errors in full metadata fetch (dpkp / PR 1980)
* Raise AssertionError if consumer closed in poll() (dpkp / PR 1978)
* Log retriable coordinator NodeNotReady, TooManyInFlightRequests as debug not error (dpkp / PR 1975)
* Remove unused import (jeffwidman)
* Remove some dead code (jeffwidman)
* Fix a benchmark to Use print() function in both Python 2 and Python 3 (cclauss / PR 1983)
* Fix a test to use ==/!= to compare str, bytes, and int literals (cclauss / PR 1984)
* Fix benchmarks to use pyperf (carsonip / PR 1986)
* Remove unused/empty .gitsubmodules file (jeffwidman / PR 1928)
* Remove deprecated `ConnectionError` (jeffwidman / PR 1816)