Manubot

Manubot version 0.3.1 includes the `pandoc-manubot-cite` command, which is a Pandoc filter for citation by identifier.
`manubot process` has a `--skip-citations` option to leave citation processing for `pandoc-manubot-cite`.
This option may become required in the future.

See commits for additional enhancements in this release.

Commits
-------

- 95ec387 Do not warn about citation-tags.tsv when missing (194)
- 6a35918 Fix black formatting error
- 54ed780 get_continuous_integration_parameters: support GitHub actions (195)
- b807720 Version citaiton in testing input.md
- 5ac49d7 pandoc-manubot-cite filter (99)
- 92ad55b CI: do not use pytest --verbose
- 8b6e177 Remove python 3.8 from GH actions test
- 9ce0c62 Add GitHub workflow for tests (136)
- 87f6314 Configure & enforce flake8 linting (193)
- 7d6bb21 readme: add documentation badge
- 11d189d process.util: more modular get_citekeys_df & generate_csl_items
- 36bc1c4 CSL_Item: add date IO functionality (189)
- b00e49f CI: Update pandoc to version 2.8 (178)
- 9c58b4e process: YAML metadata use 'authors' not 'author_info' (188)
- 82c3d40 Expand url_to_citekey support for Sci-Hub links (185)
- f1772c1 CSL Item: Remove newlines in arXiv abstract (184)
- a59e73d manubot.cite.citekey.url_to_citekey URL parser (182)
- d223302 Fix minor linter errors
- b3f6a23 zotero search_or_web_query deduces whether URL or ID (183)

Manubot version 0.3.0 updates the schema of output variables & metadata for the `manubot process` command. Now, Pandoc's `header-includes` metadata field is set to provide manuscript-specific metadata that improves indexing by bibliographic databases and assists sharing on social media.

The terminology around citations has been updated. We now refer to identifiers for specific references as "citekeys" or "citation keys". The following external-facing functions have updated names: `manubot.cite.citekey_to_csl_item` and `manubot.cite.standardize_citekey`.

There is a new subcommand `manubot webpage` for managing creation of a webpage directory for manuscripts.

Commits
-------

- a9fc2ea Log user-provided variables at debug level (179)
- 2ffcec1 don't encapsulate header-includes in code block
- e892999 Set Pandoc's header-includes with HTML `<meta>` (138)
- 23e687a Revise metadata.yaml processing (175)
- 74f7a85 Update copyright assertion statement
- 98de2bd miscellaneous edits from gitter reviews
- ebac7ab process: combined pandoc & manubot metadata via load_variables (173)
- 49343c7 readme: improve development commands (171)
- 785b4aa move is_http_url to manubot.util (174)
- 9023b7f process: set / detect manuscript thumbnail image (169)
- 0b7c14a Apply black style to Python codebase (164)
- 43be195 webpage: fix `process` referenced before assignment (166)
- ddd0995 Function to get doi metadata via translation-server
- d5f9871 Travis CI: test on Python 3.8 (162)
- 2f494fe skip test_get_continuous_integration_parameters_* on forks (160)
- 94e3f4c Migrate standardize ids & notes to CSL_Item class (157)
- 04a3574 Generate docs and deploy to gh-pages (153)
- 065ae7f Migrate csl_item_passthrough to CSL_Item.clean (156)
- 40e5b7a test_cite_command_render_stdout: version expected files
- 1aa21dc CSL_Item class for bibliographic metadata of a single reference
- 75b259e TST: pytest option to cache requests (151)
- 0478ce2 Switch pytest to use verbose & colored output (154)
- cf9564d split cite.util into csl_item & citekey submods
- 307b339 TST: read files as utf-8-sig (143)
- 71d4d39 Travis: use API token for PyPI authorization
- df480ae Fix webpage warning during ots upgrade (134)
- 23a0841 Adopt "citekeys" terminology (129)
- efb0adb webpage redirect-template.html: remove blank line
- 00405aa manubot webpage subcommand (132)
- 1b885c6 process.ci: generalize CI variables & support AppVeyor

Contributors
-----------

Authors of commits included in this release:

- [Daniel Himmelstein](https://github.com/dhimmel)
- [Evgeny Pogrebnyak](https://github.com/epogrebnyak)
- [Olga Botvinnik](https://github.com/olgabot)

Manubot version 0.2.4 contains various enhancements and improvements to the citation processing workflow.

New features
------------

Create the new `manubot.pandoc` submodule with code for interacting with the system Pandoc installation (see [GH103](https://github.com/manubot/manubot/pull/103)). This module creates an organized location for Pandoc-related code, which will help with development of creating a Pandoc filter for citation-by-identifier (see [GH99](https://github.com/manubot/manubot/pull/99)).

Manual references can now be supplied in formats other than CSL JSON. Formats supported by `pandoc-citeproc --bib2json` can now be supplied to `manubot process`. See [GH100](https://github.com/manubot/manubot/issues/100) and [GH104](https://github.com/manubot/manubot/pull/104).

Changes
-------

Additional refactoring of the `manubot.cite` submodule has moved the package closer to a well-defining processing pipeline for citations ([GH113](https://github.com/manubot/manubot/pull/113) and [GH114](https://github.com/manubot/manubot/issues/114)). The column names in `citations.tsv` changed to [`manuscript_id`, `detagged_id`, `standard_id`, `short_id`].

Make any missing parent directories for the `--output-directory` and `--cache-directory` arguments of `manubot process`. See [GH102](https://github.com/manubot/manubot/issues/102) and [GH115](https://github.com/manubot/manubot/pull/115).

Read text files using the `utf-8-sig` encoding (to strip BOMs if present). Write text files using `utf-8` encoding. UTF-8 ensures compatibility with Pandoc, which uses it for I/O. Also keeps operation consistent across files / platforms. See [GH125](https://github.com/manubot/manubot/issues/125) and [GH127](https://github.com/manubot/manubot/pull/127).

The README has been updated with improved installation instructions and the Manubot software paper citation. See [GH118](https://github.com/manubot/manubot/pull/118) and [GH121](https://github.com/manubot/manubot/pull/121).

Manubot version 0.2.3 contains various enhancements. In addition, the source code location has moved from <https://github.com/greenelab/manubot> to <https://github.com/manubot/manubot> (see [GH94](https://github.com/manubot/manubot/issues/94)).

New features
------------

Citations of shortDOIs are now supported (see [GH92](https://github.com/manubot/manubot/issues/92) and [GH93](https://github.com/manubot/manubot/pull/93)). [shortDOIs](http://shortdoi.org/), which start with `10/` rather than `10.`, can now be cited just like a DOI. For example, `doi:10/gddkhn` is a supported citation. Manubot expands shortDOI citations to their regular DOIs, e.g. `doi:10.1098/rsif.2017.0387`, such that `manubot process` will treat both the short and regular form as the same citation.

Bug fixes
---------

Queries to Manubot's translation-server now specify `single=1` to enforce returning a single record per persistent identifier (see [GH90](https://github.com/manubot/manubot/pull/90)). Previously, multiple results were [sometimes](https://github.com/zotero/translation-server/issues/65) returned, causing Manubot's JSON CSL retrieval to fail. Furthermore, Zotero child notes are now ignored, fixing another [failure mode](https://github.com/zotero/translation-server/issues/67) for CSL export of Zotero metadata.

Null authors are now allowed in `metadata.yaml` and do not crash Manubot with a TypeError (see [GH91](https://github.com/manubot/manubot/pull/91)).

The codebase has been updated to avoid deprecation warnings in Pandas v0.24 (see [GH95](https://github.com/manubot/manubot/pull/95)).

Setting the header will help upstream resources contact the Manubot developers should our requests be problematic or should downtime be anticipated. Furthermore, it will allow Manubot's translation-server to monitor Manubot usage, including which operating system, Python version, and package version are being used.

Manubot's test suite has been reorganized such that testing modules correspond one-to-one with package modules (see [GH87](https://github.com/greenelab/manubot/pull/87)).

Manubot version 0.2.1 contains several improvements to the package's citation infrastructure.

New features
------------

Support has been added for `raw` citations for references without supported persistent identifiers (see [GH62](https://github.com/greenelab/manubot/issues/62) and [GH74](https://github.com/greenelab/manubot/pull/74)). Raw citations require the user to manually specify the corresponding CSL JSON.

Error messages for invalid citations have been improved (see [GH76](https://github.com/greenelab/manubot/pull/76) and [GH71](https://github.com/greenelab/manubot/issues/71)). More types of incorrect citations are now caught internally before any external APIs are queried to retrieve metadata.

The `manubot cite` command has been updated to generate metadata for all valid citations, while logging error messages for invalid citations (see [GH77](https://github.com/greenelab/manubot/pull/77)). Previously, a single invalid citation would cause the program to exit before outputting references for valid citations.

Bug fixes
---------

Previously, metadata for `pmcid` citations was retrieved from the NCBI Citation Exporter. This service was [taken offline](https://twitter.com/dhimmel/status/1061787168820092929) without notice causing citation retrieval to fail. NCBI replaced the previous service with the [Literature Citation Exporter](https://api.ncbi.nlm.nih.gov/lit/ctxp). The `manubot.cite.pubmed.get_pmc_citeproc` function has been changed to use the new service (see [GH80](https://github.com/greenelab/manubot/pull/80)).

Previously, CSL JSON Items were being generated with empty `date-parts` arrays, which would cause pandoc-citeproc to fail. Manubot's CSL JSON pruning infrastructure has been updated to delete empty `date-parts` arrays (see [GH66](https://github.com/greenelab/manubot/issues/66) and [GH65](https://github.com/greenelab/manubot/pull/65)).

Entrez E-Utils returned integer-encoded months for certain `pmid` citations causing `citeproc_from_pubmed_article` to fail. Both integer and character month encodings are now supported (see [GH72](https://github.com/greenelab/manubot/pull/72)).