Frontera

Main issue solved in this version is that now, request callbacks and request.meta contents are successfully serializing and deserializing in SQL Alchemy-based backend. Therefore, majority of Scrapy extensions shouldn't suffer from loosing meta or callbacks passing over Frontera anymore. Second, there is hot fix for cold start problem, when seeds are added, and Scrapy is quickly finishing with no further activity. Well thought solution for this will be offered later.

- Frontera is the new name for Crawl Frontier.
- Signature of get_next_requests method is changed, now it accepts arbitrary key-value arguments.
- Overused buffer (subject to remove in the future in favor of downloader internal queue).
- Backend internals became more customizable.
- Scheduler now requests for new requests when there is free space in Scrapy downloader queue, instead of waiting for absolute emptiness.
- Several Frontera middlewares are disabled by default.

- Added documentation (Scrapy Seed Loaders+Tests+Examples)
- Refactored backend tests
- Added requests library example
- Added requests library manager and object converters
- Added FrontierManagerWrapper
- Added frontier object converters
- Fixed script examples for new changes
- Optional Color logging (only if available)
- Changed Scrapy frontier and recorder integration to scheduler+middlewares
- Changed default frontier backend
- Added comment support to seeds
- Added doc requirements for RTD build
- Removed optional dependencies for setup.py and requirements
- Changed tests to pytest
- Updated docstrings and documentation
- Changed frontier componets (Backend and Middleware) to abc
- Modified Scrapy frontier example to use seed loaders
- Refactored Scrapy Seed loaders
- Added new fields to Request and Response frontier objects
- Added ScrapyFrontierManager (Scrapy wrapper for Frontier Manager)
- Changed frontier core objects (Page/Link to Request/Response)