Petaly

Added

New connectors were added Redshift Connectors:
- Redshift Cluster over IAM using iam credentials as well as over TCP using host and port
- Redshift Serverless over IAM using iam credentials as well as over TCP using host and port
- AWS S3-Bucket connections

Changed
The attribute in pipeline.yaml file: `destination_file_dir` was renamed to `destination_dir`

The attribute in pipeline.yaml file: `destination_blob_dir` was renamed to `bucket_pipeline_prefix`
[Optional] Define the path prefix to your objects in the bucket without the bucket name. Use a forward slash (/) to separate folders.
By default, the pattern {pipeline_name} will be added to the prefix and automatically replaced with the pipeline name during runtime. If not needed, you can remove it manually after the pipeline is created.

Fixed
bug fixed

Added
GCP connectors added:
- BigQuery
- Google Storage (GS-Bucket)

Changed
The attribute in pipeline.yaml file: data_attributes:object_default_settings:quote_char was renamed to columns_quote
The possible option for this attribute is changed to: ["double","single","none"]

pipeline:

object_default_settings:
header: true
columns_delimiter: ','
columns_quote: double


Fixed
several bug fixes

Added

Improved logging behaviour and added new section in petaly.ini config file:

[global_settings]
The logging mode has two settings: INFO and DEBUG.
By default, it is set to INFO, which generates minimal log output.
If an issue occurs, switch to DEBUG for more detailed output that can assist in troubleshooting.

logging_mode=DEBUG

Changed

renamed pipeline/data_objects_spec parameters:
renamed `recreate_target_object` to `recreate_destination_object`
renamed `files_source_dir` `object_source_dir`
renamed `object_attributes` to `object_spec`
relocate the parameter `object_name` into nested part under `object_spec:`

previous structure of `data_objects_spec`:


data_objects_spec:
- object_name: stocks
object_attributes:
object_name: stocks
destination_object_name:
recreate_target_object: true
cleanup_linebreak_in_fields: true
exclude_columns:
-
files_source_dir:
file_names:
-


new structure of `data_objects_spec`:


data_objects_spec:
- object_spec:
object_name: stocks
destination_object_name:
recreate_destination_object: true
cleanup_linebreak_in_fields: true
exclude_columns:
-
object_source_dir:
file_names:
-


The entire documentation is now consolidated in the README.md file

Fixed

Added

New documentation is added under [run_pipeline.md](.docs/tutorial/run_pipeline.md)
Added gzip csv files as test data to the `.tests/data/csv/` folder: `stocks.csv.gz` and `options.csv.gz`

Changed

The parameter `csv_parse_options` has been renamed to `object_default_settings` because the context of csv_parse_options was very limited.
The new object_default_settings allows for the inclusion of a broader range of options.

This is a major release that affects all files and changes the logic flow of pipelines, as well as renaming several parameters and changing their behaviour.
For pipelines created with the previous version, it is recommended to rebuild all pipelines with the wizard and set all parameters again.

Added

- New tutorial files: petaly_init_workspace.md, petaly_install.md
- Improved petaly_install.md
- Added main parsing definition for csv files for: header, delimiter, quote

csv_parse_options:
header: true
columns_delimiter: ','
quote_char: double-quote
-
Changed
The following pipeline parameters have been modified:
- `endpoint_type` to `connector_type`
- `use_data_objects_spec` to `data_objects_spec_mode`; The behaviour has also changed. New modes have been added: ***only***, ***ignore***, ***prefer***
- Removed all unused parameters
- Simplified the entire pipeline to make it easier to use

Fixed
- All md links have been changed to absolute paths. This should work on both github.com and pypi.com.
- Fixed bugs

minor change