- Fixed common crawl downloading to be more repatable. Now it is no more like a DDoS
0.1.0b6
- Fixed Common crawl downloader. It doesn't download same URLs no more.
0.1.0b5
- Basic downloader now can save the downloaded data as a temp files in disk or download the data to the memory - CommonCrawl uses temp files downloading to prevent huge RAM usage