Pydbtools

Added

- Function that creates a temporary table from a `pandas.DataFrame`, see `pydbtools.dataframe_to_temp_table`.

Added

- Function that saves the result of SQL query to a parquet file, see `pydbtools.save_query_as_parquet`.

Added

- Functions that read SQL from strings of multiple statements, see `pydbtools.read_sql_queries` and `pydbtools.read_sql_queries_gen`.
- Functions that delete database entities and the S3 data for those entities, see `pydbtools.delete_database_and_data`, `pydbtools.delete_table_and_data` and `pydbtools.delete_partitions_and_data`.

Changed

- Temporary database names are now compatible with EKS.
- `ctas_approach` now defaults to `True`, the default behaviour for `awswrangler`, using a temporary database.
- Queries automatically use values for `pyarrow_additional_kwargs` that solve [this awswrangler issue](https://github.com/awslabs/aws-data-wrangler/issues/592)

Changed

- Updated project dependencies to align with other packages we heavily use.

Added

- Jinja templating to SQL see `pydbtools.render_sql_template`.
- Read SQL function that takes an SQL file and returns it as a string (with Jinja arguments if provided) see `pydbtools.get_sql_from_file`.

Changed

- removed DEPRECATED functions: `pydbtools.get_athena_query_response`
`pydbtools.read_sql`
- Made modules not for access private.
- Added ability to change `bucket`, `temp_database_name_prefix` and `aws_default_region` in `pydbtools.utils`.

Changed

- Fixing bug where the `sqlparse` package (used to check if SQL is viable to be wrapped in a CTAS query) couldn't pass some `WITH` SQL queries that had comments after a comma.