Archivebox

This is a minor bugfix release with some Dockerfile improvements to qualify for the official docker image library.

We add a major new archive method in this release: SingleFile. On bare metal it requires installing Node and Chrome/Chromium, but it works out-of-the-box in the Docker version.

This finally allows ArchiveBox to pass all of the acid tests except one, and the archive for Github and many other sites are nicer than Wget was able to do on its own.

<img src="https://user-images.githubusercontent.com/511499/89605240-0f532a00-d83b-11ea-91dd-39e0624d3f98.png" width="33%" align="top"><img src="https://user-images.githubusercontent.com/511499/89605242-0f532a00-d83b-11ea-9ea7-f92fda70bbae.png" width="30%" align="top"><img src="https://user-images.githubusercontent.com/511499/89605243-0f532a00-d83b-11ea-99fb-9a70245c849c.png" width="30%" align="top">

Enjoy!

🎉 Big thanks to everyone who helped! Especially the Monadical team cdvv7788 apkallum afreydev and also drpfenderson who helped us track down the last few index importing bugs! 🎉

The docs still have some work left to finish updating, but the CLI help text is all up-to-date (when in doubt, just pass `--help`).
Let us know if you find any rough edges here: https://github.com/pirate/ArchiveBox/issues/new/choose

bash
pip install archivebox

cd path/to/your/archive/folder

archivebox init this doubles as the migrate command, it will safely upgrade existing index files automatically
archviebox add 'https://example.com'
archviebox add 'https://getpocket.com/users/USERNAME/feed/all' --depth=1
archivebox status
archivebox server
archivebox help


Or if you prefer docker, the CLI works exactly the same `archivebox [subcommand] [...args]`:
bash
docker run -v $PWD:/data nikisweeting/archivebox init
docker run -v $PWD:/data nikisweeting/archivebox add 'https://example.com'
docker run -v $PWD:/data -p 8000 nikisweeting/archivebox server


yaml

- better archive corruption guards (check structure invariants on every parse & save)
- remove title prefetching in favor of new FETCH_TITLE archive method
- slightly improved CLI output for parsing and remote url downloading
- re-save index after archiving completes to update titles and urls
- remove redundant derivable data from link json schema
- markdown link parsing support
- faster link parsing and better symbol handling using a new compiled URL_REGEX

- fixed issues with parsing titles including trailing tags
- fixed issues with titles defaulting to URLs instead of attempting to fetch
- fixed issue where bookmark timestamps from RSS would be ignored and current ts used instead
- fixed issue where ONLY_NEW would overwrite existing links in archive with only new ones
- fixed lots of issues with URL parsing by using `urllib.parse` instead of hand-written lambdas
- ignore robots.txt when using wget (ssshhh don't tell anyone 😁)
- fix RSS parser bailing out when there's whitespace around XML tags
- fix issue with browser history export trying to run ls on wrong directory