**New Features**
- Scuwl now attempts to abide by a robots.txt found at the root of the input URL.
- Connections now have a default of 20 seconds. Additionally a new flag, -T (--timeout) has been added to change the default timeout value.
**Improvements**
- The default user-agent is now scuwl/version.
- Exceptions from the fetch method from Scraper have been moved from the general Exception to a series of more specific exceptions.
- The fetch method also filters out any requests that don't contain "text/html" in "content-type".
- Link extraction can now handle scheme-relative, origin-relative, and directory-relative URLs.
- JPG and SVG links, as well as link fragments are now ignored.
- Global variable __version__ is now pulled by get_distribution from pkg_resources.
- Added functionaility where a a new set (scraper.urls) keeps track of urls already visited. Scraper.urls stores the blake2b 32-byte digest instead of the actual urls. This cuts down on recursive calls of scraper.recursive_scrape.