Shot-scraper

- Documentation is now hosted at **[shot-scraper.datasette.io](https://shot-scraper.datasette.io/)** and has been expanded to multiple pages. [#73](https://github.com/simonw/shot-scraper/issues/73)
- `shot-scraper $URL --wait-for EXPRESSION` can be used to take the screenshot only once the provided JavaScript expression returns `true`. See [Waiting until a specific condition](https://shot-scraper.datasette.io/en/stable/screenshots.html#waiting-until-a-specific-condition). [72](https://github.com/simonw/shot-scraper/issues/72)
- The `wait_for:` key in the YAML format used by `shot-scraper multi` provides equivalent functionality for scripted multiple screenshots.
- New documentation on [Using shot-scraper with GitHub Actions](https://shot-scraper.datasette.io/en/stable/github-actions.html). [#74](https://github.com/simonw/shot-scraper/issues/74)
- New example: [Extracting page content with Readability.js](https://shot-scraper.datasette.io/en/stable/javascript.html#example-extracting-page-content-with-readability-js)

- `shot-scraper auth --devtools` option for opening DevTools when authenticating with a site. [66](https://github.com/simonw/shot-scraper/issues/66)
- `shot-scraper multi -n/--no-clobber` option for skipping screenshots where the output image already exists on disk. [70](https://github.com/simonw/shot-scraper/issues/70)

- `--selector-all` option to take a screenshot that encompasses every element matching the specified CSS selector - complements `--selector` which takes a screenshot of just the first element matching that selector. See [Specifying elements using JavaScript filters](https://github.com/simonw/shot-scraper/blob/0.12/README.md#specifying-elements-using-javascript-filters). [64](https://github.com/simonw/shot-scraper/issues/64)
- `selector_all:` and `selectors_all:` keys in the `shot-scraper multi` YAML format.
- `--js-selector` and `--js-selector-all` options for specifying elements to screenshot using a JavaScript expression, for cases which cannot be handled using CSS selectors [43](https://github.com/simonw/shot-scraper/issues/43). The following example takes a screenshot of the first paragraph on the page that mentions `shot-scraper`:

shot-scraper https://github.com/simonw/shot-scraper \
--js-selector 'el.tagName == "P" && el.innerText.includes("shot-scraper")'

- `js_selector:`, `js_selectors:`, `js_selector_all:` and `js_selectors_all:` equivalent keys in the `shot-scraper multi` YAML format.
- `--user-agent` option for setting a custom user agent header. [63](https://github.com/simonw/shot-scraper/issues/63)
- `--browser webkit` option for running [WebKit](https://playwright.dev/docs/browsers#webkit) - thanks, [Ryan Murphy](https://github.com/rdmurphy). [#56](https://github.com/simonw/shot-scraper/pull/56)

- New `shot-scraper accessibility --timeout` option, thanks [Ben Welsh](https://github.com/palewire). [#59](https://github.com/simonw/shot-scraper/pull/59)
- `shot-scraper auth --browser` option for authentication using a browser other than Chromium. [61](https://github.com/simonw/shot-scraper/issues/61)
- Using `--quality` now results in a JPEG file with the correct `.jpg` extension. Thanks, [Ian Wootten](https://github.com/iwootten). [#58](https://github.com/simonw/shot-scraper/pull/58)
- New `--reduced-motion` flag for emulating the "prefers-reduced-motion" media feature. Thanks, [Ryan Murphy](https://github.com/rdmurphy). [#49](https://github.com/simonw/shot-scraper/issues/49)

- New `-b/--browser` option for the `shot-scraper install`, `shot`, `multi` and `javascript` commands. This can be used to install and run alternative browsers `firefox`, `chrome` or `chrome-beta`. Thanks, [Ben Welsh](https://github.com/palewire). [#53](https://github.com/simonw/shot-scraper/issues/53)
- New `--timeout` option for `shot-scraper shot` and `shot-scraper multi`. Thanks, [Ben Welsh](https://github.com/palewire). [#47](https://github.com/simonw/shot-scraper/pull/47)
- `shot-scraper multi` now continues to create other shots despite a timeout error, unless `--fail-on-error` is passed. Thanks, [Ryan Cheley](https://github.com/ryancheley). [#50](https://github.com/simonw/shot-scraper/issues/50)
- README now describes the `async () => { ... }` pattern for `shot-scraper javascript`, as discussed in [Extracting web page content using Readability.js and shot-scraper](https://til.simonwillison.net/shot-scraper/readability).
- New demo: [shot-scraper-template](https://github.com/simonw/shot-scraper-template), which lets you setup shot-scraper running in a GitHub repository in a single click. [See also Instantly create a GitHub repository to take screenshots of a web page](https://simonwillison.net/2022/Mar/14/shot-scraper-template/).
- New scraping demo: [scrape-hacker-news-by-domain](https://github.com/simonw/scrape-hacker-news-by-domain) uses `shot-scraper javascript` to scrape a web page. See [Scraping web pages from the command-line with shot-scraper](https://simonwillison.net/2022/Mar/14/scraping-web-pages-shot-scraper/).

- New [shot-scraper javascript](https://github.com/simonw/shot-scraper/blob/0.9/README.md#scraping-pages-using-javascript) command for executing JavaScript against a web page and returning the result to the console as JSON: [38](https://github.com/simonw/shot-scraper/issues/38)

% shot-scraper javascript datasette.io document.title
"Datasette: An open source multi-tool for exploring and publishing data"

This can be used for web scraping and data extraction. Any JavaScript errors will cause the command to return an exit code of 1, so this can also be used to run tests against a website from within a continuous integration environment such as GitHub Actions.

- The `shot-scraper pdf` and `shot-scraper accessibility` commands can both now be used with local files in addition to URLs. [37](https://github.com/simonw/shot-scraper/issues/37)
- The `output:` key is no longer required in YAML shot configuration: if omitted, an automatic filename will be used instead. [40](https://github.com/simonw/shot-scraper/issues/40)
- An empty YAML file no longer produces an error. [41](https://github.com/simonw/shot-scraper/issues/41)