Breaking Changes
- complete refactor of download system.
Feature
- add new interface hook that runs after the download finishes
- link doi directly to landing page
- include current stage in error message
- add doi to download status page
- **interfaces**: allow plugins to modify download args and manifest
- add button styling to download link
- improve layout of download status page
Fix
- **downloads**: refresh database models created in another session
- stop excluding the record with the information
- load transforms correctly
- add derivative_gen status update to beginning of section
- use FileNotFoundError when removing temp files
- handle downloads where resource details are not available
- remove cached package from context
- remove temp files when download finishes
- use update() method when updating status
- show custom error message if download cannot be found
- set rq timeout for downloads to 24h
- allow objects to be null in avro schema
- get a different schema for each resource + add more nesting
- get detail for existing resources first
- return a better error message
- only search for records containing the resource count
- return zero count if resource hasn't been counted yet
- commit changes instead of saving if instance already in session
- eager-load download relationships
- **download-status**: only show resources with records
- **migrations**: use .first() because there may be multiple records
- **downloads**: don't return none if the downloads folder doesn't exist
- return tuple from check_for_records
- initialise core and derivative records earlier
- **downloads**: exclude resources with 0 results
- add alt text for total if None
- **migration**: add defaults for file options
- **derivatives**: add setup method to refresh writers
- spell xlsx correctly
- use makedirs to make intermediate folders if necessary
- remove LESS_BIN for download-status-css
Refactor
- add type hints to datastore_queue_download
- move fixture into conftest.py
- move tests into new folder structure
Docs
- add a notice about v4 to the readme
- add more interface methods to the readme
- fix the migration filter
- make docs generation compatible with py <3.9
- add usage docs for downloads
- remove duplicate param in docstring
- exclude migration dir from API docs generation
- enable permalinks
- fix some comment wrapping
- add docstring for get_schema
Style
- fix comment line lengths
Tests
- revert changes to test_downloads_runmanager
- add new interface method for tests
- fix get_schemas test
- add tests for download plugin hooks
- add more reusable patch methods
- **helpers**: set scope for vds tables fixture to class
- patch schemas, don't overwrite them
- **downloads**: patch get_rounded_versions not SEARCH_HELPER
- add tests for flatten_dict
- add tests for filter_data_fields
- add another resource to the test data, test ignore_empty properly
- set emptyField to none
- **downloads**: add more checks for download file content
- **helpers**: move test data into separate file
- **downloads**: add additional checks to download tests
- **downloads**: add tests for dwc schema serialisation
- **downloads**: add tests for notifiers
- **downloads**: patch some parts of query_dois to test some dwc logic
- clear download dir before running tests
- **downloads**: patch url_for
- **helpers**: give an exception if wait time exceeded
- **helpers**: remove table dropping
- **downloads**: add test for integration with query-dois
- **downloads**: patch enqueue_job for the whole class
- **helpers**: drop vds tables at end of test
- **downloads**: add test for transformations
- **downloads**: add more scenarios for download integration tests
- **helpers**: yield created resource dict from vds resource fixture
- **helpers**: make es/mongo clear fixture a teardown method
- patch get_available_datastore_resources again
- fix query test
- **helpers**: add fixture to clear mongo and es
- wait for data to be added to datastore
- omit format arg when creating data, specify url
- **helpers**: add sync enqueue mock that uses a thread
- **downloads**: replace elasticsearch_scan fixture with patches
- **downloads**: remove old download tests
- **downloads**: remove unnecessary ES patches and duplicate tests
- **downloads**: add tests for creating queries
- add patch helpers for simple common patches
- **downloads**: add test for download with filter query
- **downloads**: add test for get_schema
- **downloads**: add test that runs a basic download
- **downloads**: add new tests for datastore_queue_download
Build System(s)
- add migration dir to package-data
- **migration**: add migration script for new download models
- **docker**: use latest ckantest image tag
- add query-dois and attribution as optional dependencies
- source the test script
- add fastavro dependency
CI System(s)
- use bash not source, bump docker image version
- source the test script in github actions too
Chores/Misc
- remove incorrect tag
- rename some tests
- merge in new changes from dev