Grab

Fixed

Changed
- Restrict pycurl version to <7.43.0.1 (see 354)

Fixed
- Fix 346: spider does not process `initial_urls`
- Fix 344: raise GrabInvalidUrl for pycurl error 3

Fixed
- Fix bug: task generator works incorrectly
- Fix bug: pypi package misses http api html file
- Fix bug: dictionary changed size during iteration in stat logging
- Fix bug: multiple errors in urllib3 transport and threaded network service
- Fix short names of errors in stat logging
- Improve error handling in urrllib3 transport
- Fix 299: multi-added errors
- Fix bug: pypi package misses http api html file
- Fix 285: pyquery extension parses html incorrectly
- Fix 267: normalize handling of too many redirect error
- Fix 268: fix processing of utf cookies
- Fix 241: form_fields() fails on some HTML forms
- Fix normalize_unicode issue in debug post method
- Fix 323: urllib3 transport fails with UnicodeError on some invalid URLs
- Fix 31: support for multivalue form inputs
- Fix 328, fix 67: remove hard link between document and grab
- Fix 284: option headers affects content of common_headers
- Fix 293: processing non-latin chars in Location header
- Fix 324: refactor response header processing

Changed
- Refactor Spider into set of async. services
- Add certifi dependency into grab[full] setup target
- Fix 315: use psycopg2-binary package for postgres cache
- Related to 206: do not use connection_reuse=False for proxy connections in spider

Removed
- Remove cache timeout option
- Remove structured extension

Fixed
- Fix "error:None" in spider rps logging
- Fix race condition bug in task generator

Added
- Add original_exc attribute to GrabNetworkError (and subclasses) that points to original exception

Changed
- Remove IOError from the ancestors of GrabNetworkError
- Add default values to --spider-transport and --grab-transport options of crawl script

Added
- Add --spider-transport and --grab-transport options to crawl script
- Add SOCKS5 proxy support in urllib3 transport

Fixed
- Fix 237: urllib3 transport fails without pycurl installed
- Fix bug: incorrect spider request logging when cache is enabled
- Fix bug: crawl script fails while trying to process a lock key
- Fix bug: urllib3 transport fails while trying to throw GrabConnectionError exception
- Fix bug: Spider add_task method fails while trying to log invalid URL error

Removed
- Remove obsoleted hammer_mode and hammer_timeout config options