Crawl-frontier

Latest version: v0.2.0

Safety actively analyzes 682404 Python packages for vulnerabilities to keep your Python projects secure.

Scan your dependencies

Page 2 of 4

0.5.2.1

Before that release the default compression codec was Snappy. We found out Snappy support is broken in certain Kafka versions, and issued that release. The latest version has no compression codec enabled by default, and allows to choose the compression codec with `KAFKA_CODEC_LEGACY` option.

0.5.2

From now on, strategy worker continues to operate after internal exception. There were also minor improvements.

0.5.1.1

Graphs import was removed from frontera module, therefore SQLAlchemy isn't required anymore, when it's not used.

0.5.0

Here is the change log:
- latest SQLAlchemy unicode-related crashes are fixed,
- corporate website friendly canonical solver has been added.
- crawling strategy concept evolved: added ability to add to queue an arbitrary URL (with transparent state check), `FrontierManager` available on construction,
- strategy worker code was refactored,
- default state introduced for links generated during crawling strategy operation,
- got rid of Frontera logging in favor of Python native logging,
- logging system configuration by means of logging.config using file,
- partitions to instances can be assigned from command line now,
- improved test coverage from Preetwinder.

Enjoy!

0.4.2

This release prevents installing `kafka-python` package versions newer than 0.9.5. Newer version has significant architectural changes and requires Frontera code adaptation and testing. If you are using Kafka message bus, than you're encouraged to install this update.

0.4.1

- fixed API docs generation on RTD,
- added `body` field in Request objects, to support POST-type requests,
- guidance on how to set `MAX_NEXT_REQUESTS` and settings docs fixes,
- fixed colored logging.

Page 2 of 4

© 2024 Safety CLI Cybersecurity Inc. All Rights Reserved.