Etl-manager

Change

- Glue jobs now run using Python 3 and Spark 2.4 as default

Added

- ETL manager now allows use of STRUCT and ARRAY col types in your hive metadata tables.

Updated

- Method function in TableMeta `refresh_paritions` renamed to `refresh_partitions`.
- `refresh_partitions` function now wait for athena to complete the query. This should avoid errors where you hit limits of concurrent Athena queries (max 4) when using `refresh_all_table_partitions` (from DatabaseMeta class).

Added

- Two new input arguments to GlueJob method function `wait_for_completion`.
- Input `back_off_retries` now is the number of retries to boto API to avoid Throttling Error. Retries are done with exponential back off.
- `cleanup_if_successful` will delete the glue job if the `wait_for_completion` doesn't raise an error. i.e. Glue job completes successfully.

Added

- Fixed issue [91](https://github.com/moj-analytical-services/etl_manager/issues/91) and [92](https://github.com/moj-analytical-services/etl_manager/issues/92)
- Improved python format
- Refactored to Python 3.6
- Fixed unknown issue where arguments passed into function were not copied (same memory location)

Change

- Updated output from `GlueJob` class function `wait_for_job_completion()` (when verbose is set to True), now states how long Glue has been running the job.