Frontera

Here is the change log:
- latest SQLAlchemy unicode-related crashes are fixed,
- corporate website friendly canonical solver has been added.
- crawling strategy concept evolved: added ability to add to queue an arbitrary URL (with transparent state check), `FrontierManager` available on construction,
- strategy worker code was refactored,
- default state introduced for links generated during crawling strategy operation,
- got rid of Frontera logging in favor of Python native logging,
- logging system configuration by means of logging.config using file,
- partitions to instances can be assigned from command line now,
- improved test coverage from Preetwinder.

Enjoy!

This release prevents installing `kafka-python` package versions newer than 0.9.5. Newer version has significant architectural changes and requires Frontera code adaptation and testing. If you are using Kafka message bus, than you're encouraged to install this update.

- fixed API docs generation on RTD,
- added `body` field in Request objects, to support POST-type requests,
- guidance on how to set `MAX_NEXT_REQUESTS` and settings docs fixes,
- fixed colored logging.

A tremendous work was done:
- `distributed-frontera` and `frontera` were merged together into the single project: to make it easier to use and understand,
- Backend was completely redesigned. Now it's consisting of `Queue`, `Metadata` and `States` objects for low-level code and higher-level `Backend` implementations for crawling policies,
- Added definition of run modes: single process, distributed spiders, distributed spider and backend.
- Overall distributed concept is now integrated into Frontera, making difference between usage of components in single process and distributed spiders/backend run modes clearer.
- Significantly restructured and augmented documentation, addressing user needs in a more accessible way.
- Much less configuration footprint.

Enjoy this new year release and let us know what you think!

- tldextract is no longer minimum required dependency,
- SQLAlchemy backend now persists headers, cookies, and method, also `_create_page` method added to ease customization,
- Canonical solver code (needs documentation)
- Other fixes and improvements

Now, it's possible to configure Frontera from Scrapy settings. The order of precedence for configuration sources is following:
1. Settings defined in the module pointed by FRONTERA_SETTINGS (higher precedence)
2. settings defined in the Scrapy settings,
3. default frontier settings.