Caper

Minor release to fix several issues.
- `caper cleanup` does not lock a file to be deleted.
- No automatic writing of `metadata.json` file for subworkflows.

Change of CLI parameters:
- `--hide-subworkflow` -> `--show-subworkflow`. Caper defaults to hide subworkflows.

Added features
- `--disable-auto-update-metadata`: Disables auto write/update of `metadata.json` file on workflow's output root directory.
- `NBSubprocThread`
- Separated streaming (with two deamonized threads instead of one) of STDOUT and STDERR of a subprocess.
- Better error handling. Correctly sends signal (`signal.SIGTERM` by default) to a subprocess thread when interrupted by `stop()`.
- `CromwellRestAPI`
- `find()` defaults to `exclude_subworkflow`.
- Does not retrieve all workflows unless wildcard (`*` or `?`) is in a query as `workflow_id` or `label` (as value in a key/value pair).

Bug fixes
- `--java-heap-server` not working
- `NBSubprocThread`
- Not correctly capturing Cromwell's STDOUT/returncode while gracefully dying.
- Recursively sending too many requests to retrieve information of all workflows.
- This recursively happened every update of workflow/subworkflow/task's status change, so lead to crash of Cromwell server running WDLs (e.g. ENCODE `dnase-seq-pipeline`) with nested subworkflows.

If pip-upgrading from a previous version, users should upgrade Python `requests` >= 2.20.
bash
$ pip install requests --upgrade


Bug fixes
- Job submission parameters for PBS backend.
- Python `requests` bug (`bool` type in REST API).

Use Cromwell's native subworkflow recursion for retrieving metadata.
- This will fix most subworkflow-related issues.

Upgrade Autouri to fix hanging issue due to permission on lock file on GCS (`gs://`).
bash
$ pip install autouri --upgrade


Bug fixes
- Service account key file uploading/sharing on Caper server instance.

Bug fixes
- To provision worker instances, use user's service account (`--gcp-service-account-key-json`) instead of default service account of Compute Engine.

gcp-memory-retry` currently doesn't work. It should be fixed on Cromwell's side. See [this](https://github.com/broadinstitute/cromwell/issues/5815) for details.

If upgrading from old Caper, then manually upgrade `autouri` to >=0.2.1 to fix the public bucket access issue.
bash
$ pip install autouri --upgrade
$ autouri -v check if version >= 0.2.1


Linear resource analysis for workflows run on Google Cloud.
- Can run this with any workflows run with Caper>=1.2.
- Usage: `caper gcp_res_analsysis WORKFLOW_ID_OR_METADATA_JSON ... MORE_WORKFLOW_ID --plot-pdf result.pdf`
- For each task, collect resource data from multiple workflows and do linear regression on
- `x`: Input file sizes.
- To simplify the problem to a single linear regression, all input file sizes are summed up by default. But users can use multiple linear regression without summation (scatter plot won't be available).
- 2D scatter plot with a fitting line.
- `y`: Resource taken
- Max memory
- Max disk usage
- Examples (ENCODE ATAC-Seq pipeline)
- Plot: https://storage.googleapis.com/caper-data/gcp_resource_analysis/example_plot/atac.pdf
- Input file variable definition JSON
- Can be defined with `--in-file-var-def-json`.
- If not defined, then analyze all tasks with all input variables.
- https://storage.googleapis.com/caper-data/gcp_resource_analysis/in_file_vars_json/atac.json.

Change of default parameters for Python API.
- `local_loc_dir`: defaults to `CWD/.caper_tmp`.
- `local_out_dir`: defaults to `CWD`.

Bug fixes
- Cannot access to Google public buckets.
- Updated localization module `autouri>=0.2.1` to use Google's anonymous client for public buckets.
- `caper init` fails due to `~/` not expanded correctly.
- Server instance creation shell script fails when parsing startup script.
- `scripts/gcp_caper_server/create_instance.sh`
- Misleading error messages
- `work_dir` -> `loc_dir`.