Wpull

==================

* Implements ``--no-warc-digests``, ``--warc-cdx``.
* Improvements on reducing CPU usage consumption.
* API: Engine and Processor interaction refactored to be asynchronous.

* The Engine and Processor classes were modified significantly.
* The Engine no longer is concerned with fetching requests.
* Requests are handled within Processors. This will benefit future Processors to allow them to make arbitrary requests during processing.
* The ``RedirectTracker`` was moved to a new ``web`` module.
* A ``RichClient`` is implemented. It handles robots.txt, cookies, and redirect concerns.
* ``WARCRecord`` was moved into a new ``warc`` module.

===================

* Fixes ca-bundle file missing during install.
* Fixes AttributeError on ``retry_dns_error``.

===================

* Another attempt to possibly fix 27.
* Implements cleaning inactive connections from the connection pool.

===================

* Another attempt to possibly fix 27.
* API: Refactored ``ConnectionPool``. It now calls ``put`` on ``HostConnectionPool`` to avoid sharing a queue.

=================

* Implements cookie support.
* Fixes non-recursive downloads where robots.txt was checked unnecessarily.
* Possibly fix issue 27 where HTTP workers get stuck.

===================

* Adds some documentation about stopping Wpull and a list of all options.
* API: ``Builder`` now exposes ``Factory``.
* API: ``WebProcessorSession`` was refactored to not pass arguments through the initializer. It also now uses ``DemuxDocumentScraper`` and ``DemuxURLFilter``.