Irods-capability-automated-ingest

This release adds the ability to delete data objects and collections from iRODS which DO NOT exist in the source being ingested. The feature exposes two new events and an event handler method for controlling the delete mode.

Changed

- Improve documentation (23, 137, 140, 150, 183, 214, 257, 289).
- Display more information about jobs in `list` subcommand output (91).
- Improve testing, Docker demo, and code formatting (100, 132, 180).
- Replace Celery application and tasks (211).
- Remove /tmp mount directory from Docker test harness (235).
- Bump PRC dependency version to <3.0.0 (263).
- Refactor and clean up code (180, 272, 274).
- Bump Celery dependency version to <6.0.0 (266).
- Improve handling of irods_session (269).
- Disable mingling for Celery works in tests (280).
- Replace use of "sync" with "tasks" in Celery tasks names (281).

Removed

- Remove --append_json option (60).
- Remove unnecessary directories and files (245, 246, 247, 248, 262).

Fixed

- Handle KeyboardInterrupt for `watch` subcommand (93).
- Use logical path for redis_lock key in create_dirs (124).
- Do not allow data transfers to redirect by default (276).
- Do not add unreadable files to sync chunks (277).
- Restore syncs from S3 bucket to iRODS (285).
- Fix periodic task name (293).

Added

- Add Delete mode (48, 261, 288).
- Track start time of sync jobs (92).
- Track jobs stopped by "irods_sync stop" (210).
- Add Docker Compose project for testing (244).

This release adds more functionality when scanning an S3 bucket
and updates the testing harness and a number of dependencies.

Note: The signatures for all pre/post event handler methods
(e.g. `pre_data_obj_create`) have been changed to include an
`*args` parameter. Any existing event handler files will need
to be updated by inserting the new parameter just before the
`**options` keyword argument:

diff
staticmethod
- def pre_data_obj_create(hdlr_mod, logger, session, meta, **options):
+ def pre_data_obj_create(hdlr_mod, logger, session, meta, *args, **options):


- [180] Add tags to gitignore
- [219] Add tests for pre/post event handler methods
- [219] Add *args to all example pre/post event handler methods
- [180] Rename 'syncer' to 'scanner_instance'
- Revert "[219] Add 'op' and 'scanner' to meta"
- Revert "[180] Rename 'syncer' to 'scanner_instance'"
- Bump certifi from 2023.7.22 to 2024.7.4
- [180] Rename 'syncer' to 'scanner_instance'
- [219] Add 'op' and 'scanner' to meta
- [222] Use %-formatting in log statement
- Bump urllib3 from 1.26.18 to 1.26.19
- [232] Update deployment instructions in README
- [216] Remove non-Compose test instructions
- [174] Update Redis instructions in README
- Bump werkzeug from 2.3.8 to 3.0.3
- Bump jinja2 from 3.1.3 to 3.1.4
- Bump certifi from 2022.12.7 to 2023.7.22
- Bump urllib3 from 1.26.5 to 1.26.18
- Bump jinja2 from 2.11.3 to 3.1.3
- Bump flask from 1.0.2 to 2.2.5
- Bump werkzeug from 2.2.3 to 2.3.8
- Bump redis from 2.10.6 to 4.4.4
- [215] Fix test failures
- [220] Update test environment
- [180] Update supported Python versions
- [180] black formatter - no functional changes
- [212] changed REGISTER to REGISTER_SYNC
- [207] multi read and write from S3 to iRODS for put, putsync
- [129] Added functionality for PUT, PUT_SYNC with S3 via Minio
- [129] put_sync functionality for data in S3

This release fixes the exclude and post_job behavior
and updates two dependencies.

- [200] Add --exclude_file_type test
- [201] Amend test for post_job
- [195] apply CELERY_BROKER_URL env var globally to tests
- [198] update to Python 3.11 in docker test suite
- [201] Fix job done condition
- [200] Fix exclude_file_name/exclude_file_type
- [200] Add test for --exclude_file_name
- Bump certifi from 2018.11.29 to 2022.12.7
- Bump werkzeug from 0.15.3 to 2.2.3

This release fixes an exit code bug and adds a
character_map event handler method.

- [188] eliminate exit call in check_event_handler
- [40][166] tests work for unicodeEncodeError and char_map put/register
- [166] implement object path character remapping (with AVU hints)
- [180] add .gitignore
- [177] Fix wrong exit code with --synchronous option

This release abstracts the scanners, eases deployment
by putting the event handler in redis, provides better
SSL support, and now requires Python 3.7+.

- [171] Un-skip tests with resolved issues
- [167] Bump versions in setup.py and test image
- [170] Fix tests to use event_handler files
- Bump celery from 4.2.1 to 5.2.2
- Bump urllib3 from 1.24.2 to 1.26.5
- Bump jinja2 from 2.10 to 2.11.3
- [102] event_handler goes into redis
- [159] add performance benchmark test harness
- [147][157] Allow running workers with env only
- [156] modified test to use resc_hier string
- [155] added helper for unicode errors and renamed variables
- [110] Add several interfaces for refactor
- [irods/python-irodsclient237] load certificate into ssl context
- fixed the parsing of the S3 region parameter
- Bump werkzeug from 0.14.1 to 0.15.3
- [125] Add non-SSL connection option for S3
- [86][117] Test suite cleanup + docker image
- Correct README.md for docker instructions
- [109] Update docker steps for Celery
- [114] Remove zone hint check
- [90] Honor CELERY_BROKER_URL when present