Kafka-python

Consumer
* KIP-54: Implement sticky partition assignment strategy (aynroot / PR 2057)
* Fix consumer deadlock when heartbeat thread request timeout (huangcuiyang / PR 2064)

Compatibility
* Python 3.8 support (Photonios / PR 2088)

Cleanups
* Bump dev requirements (jeffwidman / PR 2129)
* Fix crc32c deprecation warning (crc32c==2.1) (jeffwidman / PR 2128)
* Lint cleanup (jeffwidman / PR 2126)
* Fix initialization order in KafkaClient (pecalleja / PR 2119)
* Allow installing crc32c via extras (mishas / PR 2069)
* Remove unused imports (jameslamb / PR 2046)

Admin Client
* Merge _find_coordinator_id methods (jeffwidman / PR 2127)
* Feature: delete consumergroups (swenzel / PR 2040)
* Allow configurable timeouts in admin client check version (sunnyakaxd / PR 2107)
* Enhancement for Kafka Admin Client's "Describe Consumer Group" (Apurva007 / PR 2035)

Protocol
* Add support for zstd compression (gabriel-tincu / PR 2021)
* Add protocol support for brokers 1.1.0 - 2.5.0 (gabriel-tincu / PR 2038)
* Add ProduceRequest/ProduceResponse v6/v7/v8 (gabriel-tincu / PR 2020)
* Fix parsing NULL header values (kvfi / PR 2024)

Tests
* Add 2.5.0 to automated CI tests (gabriel-tincu / PR 2038)
* Add 2.1.1 to build_integration (gabriel-tincu / PR 2019)

Documentation / Logging / Errors
* Disable logging during producer object gc (gioele / PR 2043)
* Update example.py; use threading instead of multiprocessing (Mostafa-Elmenbawy / PR 2081)
* Fix typo in exception message (haracejacob / PR 2096)
* Add kafka.structs docstrings (Mostafa-Elmenbawy / PR 2080)
* Fix broken compatibility page link (anuragrana / PR 2045)
* Rename README to README.md (qhzxc0015 / PR 2055)
* Fix docs by adding SASL mention (jeffwidman / 1990)

Admin Client
* KAFKA-8962: Use least_loaded_node() for AdminClient.describe_topics() (jeffwidman / PR 2000)
* Fix AdminClient topic error parsing in MetadataResponse (jtribble / PR 1997)

This release includes breaking changes for any application code that has not
migrated from older Simple-style classes to newer Kafka-style classes.

Deprecation
* Remove deprecated SimpleClient, Producer, Consumer, Unittest (jeffwidman / PR 1196)

Admin Client
* Use the controller for topic metadata requests (TylerLubeck / PR 1995)
* Implement list_topics, describe_topics, and describe_cluster (TylerLubeck / PR 1993)
* Implement __eq__ and __hash__ for ACL objects (TylerLubeck / PR 1955)
* Fixes KafkaAdminClient returning `IncompatibleBrokerVersion` when passing an `api_version` (ian28223 / PR 1953)
* Admin protocol updates (TylerLubeck / PR 1948)
* Fix describe config for multi-broker clusters (jlandersen / PR 1869)

Miscellaneous Bugfixes / Improvements
* Enable SCRAM-SHA-256 and SCRAM-SHA-512 for sasl (swenzel / PR 1918)
* Fix slots usage and use more slots (carsonip / PR 1987)
* Optionally return OffsetAndMetadata from consumer.committed(tp) (dpkp / PR 1979)
* Reset conn configs on exception in conn.check_version() (dpkp / PR 1977)
* Do not block on sender thread join after timeout in producer.close() (dpkp / PR 1974)
* Implement methods to convert a Struct object to a pythonic object (TylerLubeck / PR 1951)

Test Infrastructure / Documentation / Maintenance
* Update 2.4.0 resource files for sasl integration (dpkp)
* Add kafka 2.4.0 to CI testing (vvuibert / PR 1972)
* convert test_admin_integration to pytest (ulrikjohansson / PR 1923)
* xfail test_describe_configs_topic_resource_returns_configs (dpkp / Issue 1929)
* Add crc32c to README and docs (dpkp)
* Improve docs for reconnect_backoff_max_ms (dpkp / PR 1976)
* Fix simple typo: managementment -> management (timgates42 / PR 1966)
* Fix typos (carsonip / PR 1938)
* Fix doc import paths (jeffwidman / PR 1933)
* Update docstring to match conn.py's (dabcoder / PR 1921)
* Do not log topic-specific errors in full metadata fetch (dpkp / PR 1980)
* Raise AssertionError if consumer closed in poll() (dpkp / PR 1978)
* Log retriable coordinator NodeNotReady, TooManyInFlightRequests as debug not error (dpkp / PR 1975)
* Remove unused import (jeffwidman)
* Remove some dead code (jeffwidman)
* Fix a benchmark to Use print() function in both Python 2 and Python 3 (cclauss / PR 1983)
* Fix a test to use ==/!= to compare str, bytes, and int literals (cclauss / PR 1984)
* Fix benchmarks to use pyperf (carsonip / PR 1986)
* Remove unused/empty .gitsubmodules file (jeffwidman / PR 1928)
* Remove deprecated `ConnectionError` (jeffwidman / PR 1816)

This is a minor release focused on KafkaConsumer performance, Admin Client
improvements, and Client concurrency. The KafkaConsumer iterator implementation
has been greatly simplified so that it just wraps consumer.poll(). The prior
implementation will remain available for a few more releases using the optional
KafkaConsumer config: `legacy_iterator=True` . This is expected to improve
consumer throughput substantially and help reduce heartbeat failures / group
rebalancing.

Client
* Send socket data via non-blocking IO with send buffer (dpkp / PR 1912)
* Rely on socket selector to detect completed connection attempts (dpkp / PR 1909)
* Improve connection lock handling; always use context manager (melor,dpkp / PR 1895)
* Reduce client poll timeout when there are no in-flight requests (dpkp / PR 1823)

KafkaConsumer
* Do not use wakeup when sending fetch requests from consumer (dpkp / PR 1911)
* Wrap `consumer.poll()` for KafkaConsumer iteration (dpkp / PR 1902)
* Allow the coordinator to auto-commit on old brokers (justecorruptio / PR 1832)
* Reduce internal client poll timeout for (legacy) consumer iterator interface (dpkp / PR 1824)
* Use dedicated connection for group coordinator (dpkp / PR 1822)
* Change coordinator lock acquisition order (dpkp / PR 1821)
* Make `partitions_for_topic` a read-through cache (Baisang / PR 1781,1809)
* Fix consumer hanging indefinitely on topic deletion while rebalancing (commanderdishwasher / PR 1782)

Miscellaneous Bugfixes / Improvements
* Fix crc32c avilability on non-intel architectures (ossdev07 / PR 1904)
* Load system default SSL CAs if `ssl_cafile` is not provided (iAnomaly / PR 1883)
* Catch py3 TimeoutError in BrokerConnection send/recv (dpkp / PR 1820)
* Added a function to determine if bootstrap is successfully connected (Wayde2014 / PR 1876)

Admin Client
* Add ACL api support to KafkaAdminClient (ulrikjohansson / PR 1833)
* Add `sasl_kerberos_domain_name` config to KafkaAdminClient (jeffwidman / PR 1852)
* Update `security_protocol` config documentation for KafkaAdminClient (cardy31 / PR 1849)
* Break FindCoordinator into request/response methods in KafkaAdminClient (jeffwidman / PR 1871)
* Break consumer operations into request / response methods in KafkaAdminClient (jeffwidman / PR 1845)
* Parallelize calls to `_send_request_to_node()` in KafkaAdminClient (davidheitman / PR 1807)

Test Infrastructure / Documentation / Maintenance
* Add Kafka 2.3.0 to test matrix and compatibility docs (dpkp / PR 1915)
* Convert remaining `KafkaConsumer` tests to `pytest` (jeffwidman / PR 1886)
* Bump integration tests to 0.10.2.2 and 0.11.0.3 (jeffwidman / 1890)
* Cleanup handling of `KAFKA_VERSION` env var in tests (jeffwidman / PR 1887)
* Minor test cleanup (jeffwidman / PR 1885)
* Use `socket.SOCK_STREAM` in test assertions (iv-m / PR 1879)
* Sanity test for `consumer.topics()` and `consumer.partitions_for_topic()` (Baisang / PR 1829)
* Cleanup seconds conversion in client poll timeout calculation (jeffwidman / PR 1825)
* Remove unused imports (jeffwidman / PR 1808)
* Cleanup python nits in RangePartitionAssignor (jeffwidman / PR 1805)
* Update links to kafka consumer config docs (jeffwidman)
* Fix minor documentation typos (carsonip / PR 1865)
* Remove unused/weird comment line (jeffwidman / PR 1813)
* Update docs for `api_version_auto_timeout_ms` (jeffwidman / PR 1812)

This is a patch release primarily focused on bugs related to concurrency,
SSL connections and testing, and SASL authentication:

Client Concurrency Issues (Race Conditions / Deadlocks)
* Fix race condition in `protocol.send_bytes` (isamaru / PR 1752)
* Do not call `state_change_callback` with lock (dpkp / PR 1775)
* Additional BrokerConnection locks to synchronize protocol/IFR state (dpkp / PR 1768)
* Send pending requests before waiting for responses (dpkp / PR 1762)
* Avoid race condition on `client._conns` in send() (dpkp / PR 1772)
* Hold lock during `client.check_version` (dpkp / PR 1771)

Producer Wakeup / TimeoutError
* Dont wakeup during `maybe_refresh_metadata` -- it is only called by poll() (dpkp / PR 1769)
* Dont do client wakeup when sending from sender thread (dpkp / PR 1761)

SSL - Python3.7 Support / Bootstrap Hostname Verification / Testing
* Wrap SSL sockets after connecting for python3.7 compatibility (dpkp / PR 1754)
* Allow configuration of SSL Ciphers (dpkp / PR 1755)
* Maintain shadow cluster metadata for bootstrapping (dpkp / PR 1753)
* Generate SSL certificates for local testing (dpkp / PR 1756)
* Rename ssl.keystore.location and ssl.truststore.location config files (dpkp)
* Reset reconnect backoff on SSL connection (dpkp / PR 1777)

SASL - OAuthBearer support / api version bugfix
* Fix 0.8.2 protocol quick detection / fix SASL version check (dpkp / PR 1763)
* Update sasl configuration docstrings to include supported mechanisms (dpkp)
* Support SASL OAuthBearer Authentication (pt2pham / PR 1750)

Miscellaneous Bugfixes
* Dont force metadata refresh when closing unneeded bootstrap connections (dpkp / PR 1773)
* Fix possible AttributeError during conn._close_socket (dpkp / PR 1776)
* Return connection state explicitly after close in connect() (dpkp / PR 1778)
* Fix flaky conn tests that use time.time (dpkp / PR 1758)
* Add py to requirements-dev (dpkp)
* Fixups to benchmark scripts for py3 / new KafkaFixture interface (dpkp)

This release is primarily focused on addressing lock contention
and other coordination issues between the KafkaConsumer and the
background heartbeat thread that was introduced in the 1.4 release.

Consumer
* connections_max_idle_ms must be larger than request_timeout_ms (jeffwidman / PR 1688)
* Avoid race condition during close() / join heartbeat thread (dpkp / PR 1735)
* Use last offset from fetch v4 if available to avoid getting stuck in compacted topic (keithks / PR 1724)
* Synchronize puts to KafkaConsumer protocol buffer during async sends (dpkp / PR 1733)
* Improve KafkaConsumer join group / only enable Heartbeat Thread during stable group (dpkp / PR 1695)
* Remove unused `skip_double_compressed_messages` (jeffwidman / PR 1677)
* Fix commit_offsets_async() callback (Faqa / PR 1712)

Client
* Retry bootstrapping after backoff when necessary (dpkp / PR 1736)
* Recheck connecting nodes sooner when refreshing metadata (dpkp / PR 1737)
* Avoid probing broker versions twice on newer brokers (dpkp / PR 1738)
* Move all network connections and writes to KafkaClient.poll() (dpkp / PR 1729)
* Do not require client lock for read-only operations (dpkp / PR 1730)
* Timeout all unconnected conns (incl SSL) after request_timeout_ms (dpkp / PR 1696)

Admin Client
* Fix AttributeError in response topic error codes checking (jeffwidman)
* Fix response error checking in KafkaAdminClient send_to_controller (jeffwidman)
* Fix NotControllerError check (jeffwidman)

Core/Protocol
* Fix default protocol parser version / 0.8.2 version probe (dpkp / PR 1740)
* Make NotEnoughReplicasError/NotEnoughReplicasAfterAppendError retriable (le-linh / PR 1722)

Bugfixes
* Use copy() in metrics() to avoid thread safety issues (emeric254 / PR 1682)

Test Infrastructure
* Mock dns lookups in test_conn (dpkp / PR 1739)
* Use test.fixtures.version not test.conftest.version to avoid warnings (dpkp / PR 1731)
* Fix test_legacy_correct_metadata_response on x86 arch (stanislavlevin / PR 1718)
* Travis CI: 'sudo' tag is now deprecated in Travis (cclauss / PR 1698)
* Use Popen.communicate() instead of Popen.wait() (Baisang / PR 1689)

Compatibility
* Catch thrown OSError by python 3.7 when creating a connection (danjo133 / PR 1694)
* Update travis test coverage: 2.7, 3.4, 3.7, pypy2.7 (jeffwidman, dpkp / PR 1614)
* Drop dependency on sphinxcontrib-napoleon (stanislavlevin / PR 1715)
* Remove unused import from kafka/producer/record_accumulator.py (jeffwidman / PR 1705)
* Fix SSL connection testing in Python 3.7 (seanthegeek, silentben / PR 1669)