Oaebu-workflows

What's Changed
* Fix oapen metadata 'invalid date' issue by aroelo in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/66
* MEL-399 created new hierarchy by niamhq in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/65
* MEL-399 Added Fulcrum content by niamhq in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/67
* MEL-399 Added BISAC content by niamhq in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/69
* Add DOI badge by jdddog in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/71
* Feature/onix schema descriptions by jdddog in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/72
* MEL-399 Added overview content by niamhq in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/73
* MEL-399 Added Book ISBNs content by niamhq in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/74
* Use country.google_analytics_name for matching with Google Analytics by jdddog in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/68
* Mel 773/update oaebu docs by kathrynnapier in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/70
* Add fields for google analytics by aroelo in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/61
* Mel 399 remaining dashboard content by niamhq in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/78
* Mel 399 dashboard content by niamhq in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/75
* MEL-399 New FAQ, Known issues and Contact content by niamhq in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/79
* Mel 778/update overview and new schema by kathrynnapier in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/80
* Update CITATION.cff by bechandcock in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/77
* Create .zenodo.json by jdddog in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/76
* Author citation missing initial by bechandcock in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/81
* Update telescope summaries for readthedocs by aroelo in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/64
* Update onix_1900-01-01.json by jdddog in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/82
* Update CITATION.cff by bechandcock in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/84
* Mel 789/final documentation updates by kathrynnapier in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/85
* Update .zenodo.json by jdddog in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/83
* MEL-797 included data trust community and update direct link to embed… by kathrynnapier in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/88
* Update Google Books schedule interval by aroelo in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/86
* INF-384: Use 2nd gen cloud function by aroelo in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/89
* Mel 670/dataset release creation by tuanchien in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/57
* Fix/api update by jdddog in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/90
* Fix oapen metadata dc.date.issued field by aroelo in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/91
* Use DatasetTypeId class by aroelo in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/93
* Update Oapen IRUS UK by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/92
* Inf 406/add scripts to seed api db by tuanchien in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/96
* Use find_free_port in tests by tuanchien in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/97
* Parallelise some onix workflow tasks by tuanchien in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/99
* Add dag tags by tuanchien in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/98
* BAD-142: ANU google analytics update by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/95
* Fixed the google analytics schema by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/100
* Increased lookback time to October 2016 by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/101
* Bad 214 oapen onix unittest table updates by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/102
* Change start date of oapen_irus_uk_telescope by alexmassen-hane in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/104
* Feature/kibana spaces by jdddog in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/106
* Bad 241 - UCL Discovery data collection - Added ISBN10s by alexmassen-hane in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/109
* INF-532: Pre-commit update by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/110
* INF-540: Updated JSTOR telescope to ignore new fields in the schema by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/111
* Bad 210 wp2 standardised data ingest pipelines workflows by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/105
* INF-550: Updated JSTOR transform for new report format by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/114
* INF-562: JSTOR schedule update by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/115
* Removed the loading of the OAEBU service account credentials by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/117
* BAD-308: schema and BQ description updates by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/116
* Update github unit tests in github workflow/actions by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/120
* Inf 574 onix schema fix by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/121
* Bad 257 OAPEN ONIX Integration by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/118
* BAD-412: Added view creation to onix workflow by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/122
* OAPEN API seed fix by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/125
* BAD-372: Thoth Telescope by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/119
* OAPEN metadata telescope type update by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/126
* Onix keyword null ammendment by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/127
* Oapen docs by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/123
* BAD-314: CEU Press onboarding by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/124
* INF-591: Updated HTTP requests to use new retry function by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/129
* Onix workflow's integration with onix telescope alternatives by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/128
* BAD-490: Removed field selection from crossref metadata uri by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/133
* Remove helper functions from Thoth telescope by alexmassen-hane in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/134
* Added OBP ONIX workflow dataset to seed by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/130
* Updated filenames in dag to use telescope default by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/131
* BAD-489: Retry loop for crossref metadata by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/132
* Fixes incorrect parameter in select_table_shard_dates function by jdddog in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/138
* INF-604: Crossref pools for the ONIX workflow by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/137
* INF-598: ONIX telescope keyword parsing by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/136
* Oapen metadata structure change and personname field by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/135
* Handle case: OAPEN Metadata returns only the XML header by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/140
* Updated telescope documentation index by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/139
* BAD-276: Fulcrum Telescope Development by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/113
* Oapen metadata transform bucket consistency by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/141
* Feature/crossref academic observatory by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/144
* Changed crossref metadata table load by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/145
* Feature/workflow config file by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/142
* BAD-507: Add personname field to Thoth feed by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/143
* Remove ISBN filter on OA books by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/146
* Changed partition write disposition to append by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/149
* Update ONIX parsing and function consolidation by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/147
* Google books sftp update by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/148
* Add dags folder by jdddog in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/151
* Fixed file saving for empty contents by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/152
* Fix GA schema find by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/153
* Fixed the file matching regex by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/154
* Added the LCCN column to the ONIX schema by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/155
* fixed jstor write disposition by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/157
* Fixed the name used for crossref metadata dataset by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/156
* Fixed GA partition date and added tests by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/159
* Added fulcrum to data export SQL by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/158
* Removed additional table load in test by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/161
* IRUS schedule change by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/162
* oapen irus uk rename to irus oapen by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/160
* COK-531/update bad readthedocs by kathrynnapier in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/163
* Update of UCL discovery telescope by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/164
* Rename empty IRUS tables by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/165
* Fixed irus names by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/168
* Update for Airflow 2.6.3 by jdddog in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/167
* Fix/jstor heading update by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/169
* Updated oapen metadata to work with additional OAPEN data sources by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/166
* Remove Elastic mappings by jdddog in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/171
* Fixed empty table creation in SQL by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/170
* Add Github Actions merge_group trigger by jdddog in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/182
* Update cloud function version by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/172
* Moved crossref dataset creation to metadata table task by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/173
* Added worldreader and internet archive by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/183
* Updated parser fields for tests by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/184
* Include published date field by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/185
* Added InvertedPersonName field by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/186
* templated ga3 views type by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/188
* Fix published_date for YYYY entries by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/189
* Fix issue with OAPEN metadata filter by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/190
* Bad 638 update readthedocs by kathrynnapier in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/191
* Added readthedocs config file by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/193
* OaebuPartner Class Expansion by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/187
* Bad 645 rtd amendments by kathrynnapier in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/195
* python 3.10 update by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/192
* BAD-646 updates to formatting after review from Laura. Changes made t… by kathrynnapier in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/196
* added CODE_OF_CONDUCT.md and CONTRIBUTING.md by kathrynnapier in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/198
* Related Products Elevation & Normalisation by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/194
* BAD-657 inserted new text into FAQ as requested by Laura and amended … by kathrynnapier in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/200
* Add collections to JSTOR Telescope by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/197
* NOT NULL statements for crossref and GA by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/202
* include jstor country_name join for collections by kathrynnapier in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/204
* Update irus oapen cloud function version by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/201
* fix elevation to be optional by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/205
* File structure change by keegansmith21 in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/203

New Contributors
* niamhq made their first contribution in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/65
* kathrynnapier made their first contribution in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/70
* bechandcock made their first contribution in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/77
* alexmassen-hane made their first contribution in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/104

**Full Changelog**: https://github.com/The-Academic-Observatory/oaebu-workflows/compare/2022.03.0...2023.12.0

What's Changed
* Update StreamTelescope telescopes by aroelo in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/25
* Update Google Books to handle multiple accounts for 1 publisher by aroelo in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/22
* Remove 'expand=metadata' param to fix hanging request. by aroelo in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/27
* Add data partners and onix telescope fix by jdddog in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/26
* INF-166/airflow-2.2 by jdddog in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/28
* Fix JSTOR missing reports by aroelo in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/29
* Inf 166/add operator by jdddog in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/30
* Fix JSTOR failing DAG due to updated report header by aroelo in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/31
* changed month metrics filter by rhosking in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/32
* MEL-534 updating mapping to cover missing UCL Discovery field by rhosking in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/33
* MEL-535 Ensures unmatched isbns in ucl_discovery are exported by rhosking in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/35
* MEL-556 added usage flag to book product list by rhosking in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/34
* onix aggregation: filter out duplicate isbns by tuanchien in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/41
* Fix DOAB telescope by aroelo in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/39
* Fix OAPEN Metadata telescope by aroelo in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/40
* MEL-569 Changed mapping naming of google books by rhosking in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/38
* Mel 557 bisac and thema codes by rhosking in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/36
* Mel 553 autocomplete lists by rhosking in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/42
* Mel 559 individual author book metrics by rhosking in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/43
* changed jinja variable name by rhosking in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/44
* Add institution_list mapping by jdddog in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/45
* MEL quickfix add subject code by rhosking in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/46
* Updated country metrics export by rhosking in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/47
* Update OAPEN IRUS UK telescope by aroelo in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/48
* Fix JSTOR email filtering by aroelo in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/49
* Mel 581/onix workflow use api by tuanchien in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/51
* Mel-615 - ONIX workflow readthedocs by rhosking in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/50
* Ignore unknown fields by jdddog in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/52
* Test fixes after bq quota check feature by tuanchien in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/53
* Change OaebuPartner(name=telescope.organisation.name) to to OaebuPartner(name=dataset.name) by jdddog in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/54
* Update config instances for elastic import workflow by aroelo in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/55
* Change DAG id for public umich export by jdddog in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/56
* Fix error in create_oaebu_book_product_table task. by aroelo in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/58
* Inf 339/read the docs windows by jdddog in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/59
* Add ONIX telescope documentation by jdddog in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/62
* Use new countries table, rather than the two previous tables. by jdddog in https://github.com/The-Academic-Observatory/oaebu-workflows/pull/63


**Full Changelog**: https://github.com/The-Academic-Observatory/oaebu-workflows/compare/2021.09.1...2022.03.0

This release adds the following changes / features:
* Add black to precommit config.
* Refactor relevant workflows to use OrganisationTelescope.
* Use BITSTREAM_ISBN for Oapen Workflow.
* Refactor workflows to use XComless StreamTelescope.
* Removed get_file _hash_file from workflows.

This release adds the following bug fixes:
* Oapen Workflow: fix pendulum type hints, use correct Google Cloud project id, use new functions from observatory platform.
* Onix Workflow: fix pendulum type hints, use new functions from observatory platform.
* elastic_import_workflow.py: fix configuration.

This release adds the following telescope workflows:
* DOAB
* Google Analytics
* Google Books
* JSTOR
* OAPEN IRUS UK
* OAPEN Metadata
* ONIX
* UCL Discover

This release adds the following analytic workflows:
* OAPEN Workflow
* ONIX Workflow