Initial release of website-scraper
Features:
- Multiprocessing support for faster scraping
- Rate limiting and random delays for bot detection avoidance
- Rotating User-Agents and browser fingerprints
- Comprehensive logging system with debug and info levels
- Progress tracking with progress bar
- XML content detection and proper handling
- SSL verification options
- Command-line interface with short and long options
- Python package with easy-to-use API
Installation
bash
pip install website-scraper
Usage
bash
website-scraper https://example.com -o results.json