- removed redundant `argparse` dependency, - updated the user agents list, - version bump.
Web crawler and scraper written in Python and based on Scrapy and Playwright's headless browser.
1.6
- auto throttling bug fix and value changes for the `-at` option, - added new `-s` option for sleeping between two consecutive requests to the same domain, - updated the user agents list, - version bump.
Web crawler and scraper written in Python and based on Scrapy and Playwright's headless browser.
1.5
- install instructions update, - updated the user agents list, - version bump.
Web crawler and scraper written in Python and based on Scrapy and Playwright's headless browser.
1.4
- argument parsing code rebase and improved few other stuff, - boolean options / arguments no longer require `yes` or `no` value, - directory `-d` option has been changed to directory `-dir` option,
Web crawler and scraper written in Python and based on Scrapy and Playwright's headless browser.