Octue

<!--- SKIP AUTOGENERATED NOTES --->
Contents

Enhancements
- [x] Give cloud `scandir` option to include subdirectories

Fixes
- [x] Ensure `Dataset` construction from a cloud directory with no `dataset.json` file does not include subdirectories

Dependencies
- [x] Upgrade GCP storage emulator version

Operations
- [x] Unify GitHub workflow skip commands

<!--- END AUTOGENERATED NOTES --->

Summary
Allow simpler instantiation of datasets from the cloud.

<!--- START AUTOGENERATED NOTES --->
Contents

Enhancements
- [x] Allow instantiation of datasets from any cloud directory rather than just those that have a dataset serialised as a `dataset.json` file in them

<!--- END AUTOGENERATED NOTES --->

Summary
Improve the `Datafile` interface and allow instantiation with local and cloud paths simultaneously.

<!--- SKIP AUTOGENERATED NOTES --->
Contents

Enhancements
- [x] Allow `Datafile` instantiation with local and cloud paths simultaneously
- [x] Add `cloud_path` property to Datafile

Fixes
- [x] Ensure `Datafile` serializes with its `cloud_path` and `project_name`

Refactoring
- [x] Convert `Datafile.cloud_protocol`, `Datafile.bucket_name`, and `Datafile.path_in_bucket` to properties
- [x] Remove redundant `Datafile.reset_local_path` method
- [x] Simplify cloud location setting and getting within `Datafile`

<!--- END AUTOGENERATED NOTES --->

Summary
Improve and simplify the `Datafile` interface and allow datafiles to simultaneously exist in the cloud and locally.

Contents

Enhancements
- [x] Allow datafiles to have a local and a cloud location simultaneously
- [x] If a local path is added to a cloud datafile, sync datafile changes to the new local file and the cloud object
- [x] Raise error if project name missing when attempting to instantiate a datafile from a cloud object
- [x] Make `Datafile` cloud location attributes public (`cloud_path`, `bucket_name`, `project_name`, `path_in_bucket`, `cloud_protocol`)
- [x] Add `Datafile.exists_locally` property

Fixes
- [x] Stop correct input and configuration manifest paths being overruled in `Runner.run`

Refactoring
- [x] Remove parameters from `Datafile` cloud methods except `Datafile.to_cloud` so that there is only one way to update the cloud location explicitly and none implicitly (this ties a single `Datafile` instance more strongly to a single cloud location)
- [x] Convert `Datafile.get_local_path` method to `Datafile.local_path` property
- [x] Rename `Datafile.is_in_cloud` to `Datafile.exists_in_cloud`

Testing
- [x] Fix incorrect dataset and datafile paths in manifest template app test

<!--- SKIP AUTOGENERATED NOTES --->

Summary
Allow explicit non-lazy downloads of `Datafile`s and single-command downloads of all `Datafile`s in a `Dataset`.

<!--- SKIP AUTOGENERATED NOTES --->
Contents

Enhancements
- [x] Allow all files in a dataset to be downloaded at once

Fixes
- [x] Create non-existent intermediate directories when downloading files using `GoogleCloudStorageClient`

Refactoring
- [x] Move download logic from `Datafile.get_local_path` into new `download` method to allow explicit non-lazy downloads

Testing
- [x] Skip entire `octue.utils.processes` test class if on Windows
- [x] Improve use of mocks in processes test
- [x] Use made-up project name for most cloud tests

Operations
- [x] Use latest `conventional-commits` pre-commit hook
<!--- END AUTOGENERATED NOTES --->


Quality Checklist
- [x] New features are fully tested (No matter how much Coverage Karma you have)

Summary
Add a function that makes it easy to run and capture any `stdout` and `stderr` output of subprocesses. This is useful for child services that forward their logs back to the parent as all output produced by non-python processes can be included.

<!--- START AUTOGENERATED NOTES --->
Contents

Enhancements
- [x] Add function for logging subprocess stdout and stderr

<!--- END AUTOGENERATED NOTES --->

Quality Checklist
- [x] New features are fully tested (No matter how much Coverage Karma you have)