Observatory-platform

What's Changed
* streamtelescope: add diff merge by lex order by tuanchien in https://github.com/The-Academic-Observatory/observatory-platform/pull/557
* Add functionality to use BigQuery snapshots by aroelo in https://github.com/The-Academic-Observatory/observatory-platform/pull/559
* Add api server isolation and db seeding by tuanchien in https://github.com/The-Academic-Observatory/observatory-platform/pull/562
* Add dag tags to Workflow class by tuanchien in https://github.com/The-Academic-Observatory/observatory-platform/pull/565
* Parametrise host api port by tuanchien in https://github.com/The-Academic-Observatory/observatory-platform/pull/564
* Create directories that are mounted as volumes with Docker by aroelo in https://github.com/The-Academic-Observatory/observatory-platform/pull/563
* Separate config loading from config use by jdddog in https://github.com/The-Academic-Observatory/observatory-platform/pull/560
* Use find_free_port in more unit tests by tuanchien in https://github.com/The-Academic-Observatory/observatory-platform/pull/569
* Raise exception if dataset missing when adding releases by tuanchien in https://github.com/The-Academic-Observatory/observatory-platform/pull/566
* Add api_port to config generation by tuanchien in https://github.com/The-Academic-Observatory/observatory-platform/pull/570
* Propagate dag tag to telescopes by tuanchien in https://github.com/The-Academic-Observatory/observatory-platform/pull/571
* Update to Elastic & Kibana v8 by jdddog in https://github.com/The-Academic-Observatory/observatory-platform/pull/568


**Full Changelog**: https://github.com/The-Academic-Observatory/observatory-platform/compare/0.4.0...0.5.0

What's Changed
* Add a DOI badge to README.md by aroelo in https://github.com/The-Academic-Observatory/observatory-platform/pull/549
* Create .zenodo.json by jdddog in https://github.com/The-Academic-Observatory/observatory-platform/pull/550
* Add functionality to use a schema when creating table from query by aroelo in https://github.com/The-Academic-Observatory/observatory-platform/pull/551
* MEL-798 added data trust zenodo community by kathrynnapier in https://github.com/The-Academic-Observatory/observatory-platform/pull/552
* api: add local api server by tuanchien in https://github.com/The-Academic-Observatory/observatory-platform/pull/553
* Add dataset release utils by tuanchien in https://github.com/The-Academic-Observatory/observatory-platform/pull/540
* Fix/api update by jdddog in https://github.com/The-Academic-Observatory/observatory-platform/pull/554
* Fix dag-delete error by jdddog in https://github.com/The-Academic-Observatory/observatory-platform/pull/556
* Build Observatory API Image & Push to Google Cloud Artifact Registry by jdddog in https://github.com/The-Academic-Observatory/observatory-platform/pull/558

New Contributors
* kathrynnapier made their first contribution in https://github.com/The-Academic-Observatory/observatory-platform/pull/552

**Full Changelog**: https://github.com/The-Academic-Observatory/observatory-platform/compare/0.3.0...0.4.0

What's Changed
* Fix potential duplicates in table after merge using stream telescope by aroelo in https://github.com/The-Academic-Observatory/observatory-platform/pull/496
* Fix double bucket delete race by tuanchien in https://github.com/The-Academic-Observatory/observatory-platform/pull/510
* Remove bq merge days functionality by aroelo in https://github.com/The-Academic-Observatory/observatory-platform/pull/512
* Inf 32/installer script after repo separation by tuanchien in https://github.com/The-Academic-Observatory/observatory-platform/pull/509
* installer script doc fixes by tuanchien in https://github.com/The-Academic-Observatory/observatory-platform/pull/514
* Add global prefix_dir override by tuanchien in https://github.com/The-Academic-Observatory/observatory-platform/pull/515
* disable download timeout by tuanchien in https://github.com/The-Academic-Observatory/observatory-platform/pull/517
* INF-166/airflow 2.2 by jdddog in https://github.com/The-Academic-Observatory/observatory-platform/pull/516
* add get_airflow_connection_login by tuanchien in https://github.com/The-Academic-Observatory/observatory-platform/pull/519
* Enable Airflow operators to be added directly as tasks by jdddog in https://github.com/The-Academic-Observatory/observatory-platform/pull/518
* remove unused wos/scopus code by tuanchien in https://github.com/The-Academic-Observatory/observatory-platform/pull/520
* installer: add https/ssh clone option by tuanchien in https://github.com/The-Academic-Observatory/observatory-platform/pull/521
* Fix ModuleNotFoundError: No module named 'wtforms.compat' error by jdddog in https://github.com/The-Academic-Observatory/observatory-platform/pull/522
* Upgrade apache-airflow to 2.2.1 by jdddog in https://github.com/The-Academic-Observatory/observatory-platform/pull/523
* Enable a bucket path to specified in azure_to_google_cloud_storage_transfer by jdddog in https://github.com/The-Academic-Observatory/observatory-platform/pull/524
* Inf 65/mag update by jdddog in https://github.com/The-Academic-Observatory/observatory-platform/pull/525
* Update requirements.txt by jdddog in https://github.com/The-Academic-Observatory/observatory-platform/pull/527
* Inf 278/workflow xcom cleanup by tuanchien in https://github.com/The-Academic-Observatory/observatory-platform/pull/528
* Inf 279/port vm create destroy template by tuanchien in https://github.com/The-Academic-Observatory/observatory-platform/pull/529
* Fix xcom topic by jdddog in https://github.com/The-Academic-Observatory/observatory-platform/pull/530
* Add Dataset, DatasetRelease, DatasetStorage API extension by tuanchien in https://github.com/The-Academic-Observatory/observatory-platform/pull/526
* Fix VM warning message on slack by aroelo in https://github.com/The-Academic-Observatory/observatory-platform/pull/532
* Add ignore_unknown_values by jdddog in https://github.com/The-Academic-Observatory/observatory-platform/pull/531
* on_failure_callback: handle exception value that is a string by jdddog in https://github.com/The-Academic-Observatory/observatory-platform/pull/535
* Add bigquery bytes processed tripwire by tuanchien in https://github.com/The-Academic-Observatory/observatory-platform/pull/533
* Remove description fields from 401 error by jdddog in https://github.com/The-Academic-Observatory/observatory-platform/pull/536
* BigQuery bytes processed by jdddog in https://github.com/The-Academic-Observatory/observatory-platform/pull/537
* Add functionality to use multiple instances for Elastic Import workflow by aroelo in https://github.com/The-Academic-Observatory/observatory-platform/pull/538
* Parameterise select table shard date limit by tuanchien in https://github.com/The-Academic-Observatory/observatory-platform/pull/539
* OpenAlex telescope changes by aroelo in https://github.com/The-Academic-Observatory/observatory-platform/pull/542
* Add check_blob_hash parameter by jdddog in https://github.com/The-Academic-Observatory/observatory-platform/pull/544
* Ensure that the blob name is unique across tables with the same name … by jdddog in https://github.com/The-Academic-Observatory/observatory-platform/pull/548


**Full Changelog**: https://github.com/The-Academic-Observatory/observatory-platform/compare/0.2.1...0.3.0

This release includes the following bugfix in the Dockerfile:
* Install apache-airflow-providers-google==5.1.0 with --no-deps so that pip doesn't spend forever trying to resolve dependencies for the package, which we only use for remote logging and secret manager backend in the cloud deployment. The google-cloud-secret-manager Python package is added as a dependency in requirements.txt.

This release includes the following changes / new features:
* Upgrade to Airflow 2.1.4.
* Stream Telescope: remove use of XComs so that it is easier to maintain.
* Updated documentation.
* download_files: uses DownloadInfo class and prefix_dir parameter to allow prefixing the filename paths.
* Remove third party get_file and _hash_file functions as they are replaced by get_file_hash and download_file.
* Command line interface: added generate workflow and project commands.
* Added OrganisationTelescope.

And the following bugfixes:
* Docker Compose file: rename deprecated Airflow config environment variables.
* Docker Compose file: change AIRFLOW__SECRETS__BACKEND to use class installed from apache-airflow-providers-google package and remove airflow subpackage as is no longer required.
* Fix typo in config.yaml.jinja2.
* Fix on_failure_callback function.