Spider-rs

- perf(links): filter dup links after async batch
- chore(delay): fix crawl delay thread groups
- perf(page): slim channel page sending required props

- feat(regex): add optional regex black listing

- chore(bin): fix bin executable [17](https://github.com/madeindjs/spider/pull/17/commits/b41e25fc507c6cd3ef251d2e25c97b936865e1a9)
- feat(cli): add cli separation binary [17](https://github.com/madeindjs/spider/pull/17/commits/b41e25fc507c6cd3ef251d2e25c97b936865e1a9)
- feat(robots): add robots crawl delay respect and ua assign [24](https://github.com/madeindjs/spider/pull/24)
- feat(async): add async page body gathering
- perf(latency): add connection re-use across request [25](https://github.com/madeindjs/spider/pull/25)

- feat(cli): add cli ability ([16](https://github.com/madeindjs/spider/pull/16) thanks to [j-mendez](https://github.com/j-mendez))
- feat(concurrency): dynamic concurrent cpu defaults ([15](https://github.com/madeindjs/spider/pull/15) thanks to [j-mendez](https://github.com/j-mendez))
- docs: add a changelog

- fix(crawl): fix field type ([14](https://github.com/madeindjs/spider/pull/14) thanks to [j-mendez](https://github.com/j-mendez))

- feat(crawl): callback to run when link is found ([13](https://github.com/madeindjs/spider/pull/13) thanks to [j-mendez](https://github.com/j-mendez))