Kafka-python-ng

Bugfixes
* Fix partition assignment race condition (jeffwidman 1240)
* Fix consumer bug when seeking / resetting to the middle of a compressed messageset (dpkp 1239)
* Fix traceback sent to stderr not logging (dbgasaway 1221)
* Stop using mutable types for default arg values (jeffwidman 1213)
* Remove a few unused imports (jameslamb 1188)

Client
* Refactor BrokerConnection to use asynchronous receive_bytes pipe (dpkp 1032)

Consumer
* Drop unused sleep kwarg to poll (dpkp 1177)
* Enable KafkaConsumer beginning_offsets() and end_offsets() with older broker versions (buptljy 1200)
* Validate consumer subscription topic strings (nikeee 1238)

Documentation
* Small fixes to SASL documentation and logging; validate security_protocol (dpkp 1231)
* Various typo and grammar fixes (jeffwidman)

Bugfixes
* Avoid multiple connection attempts when refreshing metadata (dpkp 1067)
* Catch socket.errors when sending / recving bytes on wake socketpair (dpkp 1069)
* Deal with brokers that reappear with different IP address (originsmike 1085)
* Fix join-time-max and sync-time-max metrics to use Max() measure function (billyevans 1146)
* Raise AssertionError when decompression unsupported (bts-webber 1159)
* Catch ssl.EOFErrors on Python3.3 so we close the failing conn (Ormod 1162)
* Select on sockets to avoid busy polling during bootstrap (dpkp 1175)
* Initialize metadata_snapshot in group coordinator to avoid unnecessary rebalance (dpkp 1174)

Client
* Timeout idle connections via connections_max_idle_ms (dpkp 1068)
* Warn, dont raise, on DNS lookup failures (dpkp 1091)
* Support exponential backoff for broker reconnections -- KIP-144 (dpkp 1124)
* Add gssapi support (Kerberos) for SASL (Harald-Berghoff 1152)
* Add private map of api key -> min/max versions to BrokerConnection (dpkp 1169)

Consumer
* Backoff on unavailable group coordinator retry (dpkp 1125)
* Only change_subscription on pattern subscription when topics change (Artimi 1132)
* Add offsets_for_times, beginning_offsets and end_offsets APIs (tvoinarovskyi 1161)

Producer
* Raise KafkaTimeoutError when flush times out (infecto)
* Set producer atexit timeout to 0 to match del (Ormod 1126)

Core / Protocol
* 0.11.0.0 protocol updates (only - no client support yet) (dpkp 1127)
* Make UnknownTopicOrPartitionError retriable error (tvoinarovskyi)

Test Infrastructure
* pylint 1.7.0+ supports python 3.6 and merge py36 into common testenv (jianbin-wei 1095)
* Add kafka 0.10.2.1 into integration testing version (jianbin-wei 1096)
* Disable automated tests for python 2.6 and kafka 0.8.0 and 0.8.1.1 (jianbin-wei 1096)
* Support manual py26 testing; dont advertise 3.3 support (dpkp)
* Add 0.11.0.0 server resources, fix tests for 0.11 brokers (dpkp)
* Use fixture hostname, dont assume localhost (dpkp)
* Add 0.11.0.0 to travis test matrix, remove 0.10.1.1; use scala 2.11 artifacts (dpkp 1176)

Logging / Error Messages
* Improve error message when expiring batches in KafkaProducer (dpkp 1077)
* Update producer.send docstring -- raises KafkaTimeoutError (infecto)
* Use logging's built-in string interpolation (jeffwidman)
* Fix produce timeout message (melor 1151)
* Fix producer batch expiry messages to use seconds (dnwe)

Documentation
* Fix typo in KafkaClient docstring (jeffwidman 1054)
* Update README: Prefer python-lz4 over lz4tools (kiri11 1057)
* Fix poll() hyperlink in KafkaClient (jeffwidman)
* Update RTD links with https / .io (jeffwidman 1074)
* Describe consumer thread-safety (ecksun)
* Fix typo in consumer integration test (jeffwidman)
* Note max_in_flight_requests_per_connection > 1 may change order of messages (tvoinarovskyi 1149)

Core / Protocol
* Derive all api classes from Request / Response base classes (dpkp 1030)
* Prefer python-lz4 if available (dpkp 1024)
* Fix kwarg handing in kafka.protocol.struct.Struct (dpkp 1025)
* Fixed couple of "leaks" when gc is disabled (Mephius 979)
* Added `max_bytes` option and FetchRequest_v3 usage. (Drizzt1991 962)
* CreateTopicsRequest / Response v1 (dpkp 1012)
* Add MetadataRequest_v2 and MetadataResponse_v2 structures for KIP-78 (Drizzt1991 974)
* KIP-88 / KAFKA-3853: OffsetFetch v2 structs (jeffwidman 971)
* DRY-up the MetadataRequest_v1 struct (jeffwidman 966)
* Add JoinGroup v1 structs (jeffwidman 965)
* DRY-up the OffsetCommitResponse Structs (jeffwidman 970)
* DRY-up the OffsetFetch structs (jeffwidman 964)
* time --> timestamp to match Java API (jeffwidman 969)
* Add support for offsetRequestV1 messages (jlafaye 951)
* Add FetchRequest/Response_v3 structs (jeffwidman 943)
* Add CreateTopics / DeleteTopics Structs (jeffwidman 944)

Test Infrastructure
* Add python3.6 to travis test suite, drop python3.3 (exponea 992)
* Update to 0.10.1.1 for integration testing (dpkp 953)
* Update vendored berkerpeksag/selectors34 to ff61b82 (Mephius 979)
* Remove dead code (jeffwidman 967)
* Update pytest fixtures to new yield syntax (jeffwidman 919)

Consumer
* Avoid re-encoding message for crc check (dpkp 1027)
* Optionally skip auto-commit during consumer.close (dpkp 1031)
* Return copy of consumer subscription set (dpkp 1029)
* Short-circuit group coordinator requests when NodeNotReady (dpkp 995)
* Avoid unknown coordinator after client poll (dpkp 1023)
* No longer configure a default consumer group (dpkp 1016)
* Dont refresh metadata on failed group coordinator request unless needed (dpkp 1006)
* Fail-fast on timeout constraint violations during KafkaConsumer creation (harelba 986)
* Default max_poll_records to Java default of 500 (jeffwidman 947)
* For 0.8.2, only attempt connection to coordinator if least_loaded_node succeeds (dpkp)

Producer
* change default timeout of KafkaProducer.close() to threading.TIMEOUT_MAX on py3 (mmyjona 991)

Client
* Add optional kwarg to ready/is_ready to disable metadata-priority logic (dpkp 1017)
* When closing a broker connection without error, fail in-flight-requests with Cancelled (dpkp 1010)
* Catch socket errors during ssl handshake (dpkp 1007)
* Drop old brokers when rebuilding broker metadata (dpkp 1005)
* Drop bad disconnect test -- just use the mocked-socket test (dpkp 982)
* Add support for Python built without ssl (minagawa-sho 954)
* Do not re-close a disconnected connection (dpkp)
* Drop unused last_failure time from BrokerConnection (dpkp)
* Use connection state functions where possible (dpkp)
* Pass error to BrokerConnection.close() (dpkp)

Bugfixes
* Free lz4 decompression context to avoid leak (dpkp 1024)
* Fix sasl reconnect bug: auth future must be reset on close (dpkp 1003)
* Fix raise exception from SubscriptionState.assign_from_subscribed (qntln 960)
* Fix blackout calculation: mark last_attempt time during connection close (dpkp 1008)
* Fix buffer pool reallocation after raising timeout (dpkp 999)

Logging / Error Messages
* Add client info logging re bootstrap; log connection attempts to balance with close (dpkp)
* Minor additional logging for consumer coordinator (dpkp)
* Add more debug-level connection logging (dpkp)
* Do not need str(self) when formatting to %s (dpkp)
* Add new broker response errors (dpkp)
* Small style fixes in kafka.errors (dpkp)
* Include the node id in BrokerConnection logging (dpkp 1009)
* Replace %s with %r in producer debug log message (chekunkov 973)

Documentation
* Sphinx documentation updates (jeffwidman 1019)
* Add sphinx formatting to hyperlink methods (jeffwidman 898)
* Fix BrokerConnection api_version docs default (jeffwidman 909)
* PEP-8: Spacing & removed unused imports (jeffwidman 899)
* Move BrokerConnection docstring to class (jeffwidman 968)
* Move docstring so it shows up in Sphinx/RTD (jeffwidman 952)
* Remove non-pip install instructions (jeffwidman 940)
* Spelling and grammar changes (melissacrawford396 923)
* Fix typo: coorelation --> correlation (jeffwidman 929)
* Make SSL warning list the correct Python versions (jeffwidman 924)
* Fixup comment reference to _maybe_connect (dpkp)
* Add ClusterMetadata sphinx documentation (dpkp)

Legacy Client
* Add send_list_offset_request for searching offset by timestamp (charsyam 1001)
* Use select to poll sockets for read to reduce CPU usage (jianbin-wei 958)
* Use select.select without instance bounding (adamwen829 949)

Core
* Add kafka.serializer interfaces (dpkp 912)
* from kafka import ConsumerRebalanceListener, OffsetAndMetadata
* Use 0.10.0.1 for integration tests (dpkp 803)

Consumer
* KAFKA-3007: KafkaConsumer max_poll_records (dpkp 831)
* Raise exception if given a non-str topic (ssaamm 824)
* Immediately update metadata for pattern subscription (laz2 915)

Producer
* Update Partitioners for use with KafkaProducer (barrotsteindev 827)
* Sort partitions before calling partitioner (ms7s 905)
* Added ssl_password config option to KafkaProducer class (kierkegaard13 830)

Client
* Always check for request timeouts (dpkp 887)
* When hostname lookup is necessary, do every connect (benauthor 812)

Bugfixes
* Fix errorcode check when socket.connect_ex raises an exception (guojh 907)
* Fix fetcher bug when processing offset out of range (sibiryakov 860)
* Fix possible request draining in ensure_active_group (dpkp 896)
* Fix metadata refresh handling with 0.10+ brokers when topic list is empty (sibiryakov 867)
* KafkaProducer should set timestamp in Message if provided (Drizzt1991 875)
* Fix murmur2 bug handling python2 bytes that do not ascii encode (dpkp 815)
* Monkeypatch max_in_flight_requests_per_connection when checking broker version (dpkp 834)
* Fix message timestamp_type (qix 828)

Logging / Error Messages
* Always include an error for logging when the coordinator is marked dead (dpkp 890)
* Only string-ify BrokerResponseError args if provided (dpkp 889)
* Update warning re advertised.listeners / advertised.host.name (jeffwidman 878)
* Fix unrecognized sasl_mechanism error message (sharego 883)

Documentation
* Add docstring for max_records (jeffwidman 897)
* Fixup doc references to max_in_flight_requests_per_connection
* Fix typo: passowrd --> password (jeffwidman 901)
* Fix documentation typo 'Defualt' -> 'Default'. (rolando 895)
* Added doc for `max_poll_records` option (Drizzt1991 881)
* Remove old design notes from Kafka 8 era (jeffwidman 876)
* Fix documentation typos (jeffwidman 874)
* Fix quota violation exception message (dpkp 809)
* Add comment for round robin partitioner with different subscriptions
* Improve KafkaProducer docstring for retries configuration

Bugfixes
* Fix AttributeError in BrokerConnectionMetrics after reconnecting

Incompatible Changes
* Delete KafkaConnection class (dpkp 769)
* Rename partition_assignment -> assignment in MemberMetadata for consistency
* Move selectors34 and socketpair to kafka.vendor (dpkp 785)
* Change api_version config to tuple; deprecate str with warning (dpkp 761)
* Rename _DEFAULT_CONFIG -> DEFAULT_CONFIG in KafkaProducer (dpkp 788)

Improvements
* Vendor six 1.10.0 to eliminate runtime dependency (dpkp 785)
* Add KafkaProducer and KafkaConsumer.metrics() with instrumentation similar to java client (dpkp 754 / 772 / 794)
* Support Sasl PLAIN authentication (larsjsol PR 779)
* Add checksum and size to RecordMetadata and ConsumerRecord (KAFKA-3196 / 770 / 594)
* Use MetadataRequest v1 for 0.10+ api_version (dpkp 762)
* Fix KafkaConsumer autocommit for 0.8 brokers (dpkp 756 / 706)
* Improve error logging (dpkp 760 / 759)
* Adapt benchmark scripts from https://github.com/mrafayaleem/kafka-jython (dpkp 754)
* Add api_version config to KafkaClient (dpkp 761)
* New Metadata method with_partitions() (dpkp 787)
* Use socket_options configuration to setsockopts(). Default TCP_NODELAY (dpkp 783)
* Expose selector type as config option (dpkp 764)
* Drain pending requests to the coordinator before initiating group rejoin (dpkp 798)
* Send combined size and payload bytes to socket to avoid potentially split packets with TCP_NODELAY (dpkp 797)

Bugfixes
* Ignore socket.error when checking for protocol out of sync prior to socket close (dpkp 792)
* Fix offset fetch when partitions are manually assigned (KAFKA-3960 / 786)
* Change pickle_method to use python3 special attributes (jpaulodit 777)
* Fix ProduceResponse v2 throttle_time_ms
* Always encode size with MessageSet (771)
* Avoid buffer overread when compressing messageset in KafkaProducer
* Explicit format string argument indices for python 2.6 compatibility
* Simplify RecordMetadata; short circuit callbacks (768)
* Fix autocommit when partitions assigned manually (KAFKA-3486 / 767 / 626)
* Handle metadata updates during consumer rebalance (KAFKA-3117 / 766 / 701)
* Add a consumer config option to exclude internal topics (KAFKA-2832 / 765)
* Protect writes to wakeup socket with threading lock (763 / 709)
* Fetcher spending unnecessary time during metrics recording (KAFKA-3785)
* Always use absolute_import (dpkp)

Test / Fixtures
* Catch select errors while capturing test fixture logs
* Fix consumer group test race condition (dpkp 795)
* Retry fixture failures on a different port (dpkp 796)
* Dump fixture logs on failure

Documentation
* Fix misspelling of password (ssaamm 793)
* Document the ssl_password config option (ssaamm 780)
* Fix typo in KafkaConsumer documentation (ssaamm 775)
* Expand consumer.fetcher inline comments
* Update kafka configuration links -> 0.10.0.0 docs
* Fixup metrics_sample_window_ms docstring in consumer