Viadot

Added
- Added `SQLServerToParquet` flow.
- Added `SAPBW` source class.
- Added `SAPBWToDF` task class.
- Added `SAPBWToADLS` flow class.
- Added a new `end_point` parameter in `genesys_api_connection` to make it more generic.
- Added `VidClubToADLS` flow class.

Fixed
- Fixed a bug in `subject` (extra separator) and in `receivers` (long strings) parameters in `Outlook` connector.
- Fixed issue with credentials handling in `VidClub` source class.
- Fixed issue with missing arguments in `VidClubToDF` task class.

Changed
- Genesys API call method and the name changed from `genesys_generate_exports` to `genesys_api_connection`.
- Added `GET` connection inside the method `genesys_api_connection`.
- Added new parameters in the `GenesysToCSV` task to be able to extract `web message` files.
- Changed looping structure for API calls in `VidClub` source class to use time intervals.
- Changed `VidClubToDF` task class to use total_load function from source.

Removed
- Removed methods never used in production: `get_analitics_url_report`, `get_all_schedules_job`, `schedule_report`,
`to_df`, `delete_scheduled_report_job` and `generate_reporting_export`.

Fixed
- Fixed issue with `tzlocal` for O365 package

Added
- Added `VidClub` source class
- Added `VidClubToDF` task class
- Added `GetPendingSalesOrderData`, `GetSalesInvoiceData`, `GetSalesReturnDetailData`
`GetSalesOrderData` endpoints in `BusinessCore()` source.
- Added `url` parameter to `CustomerGauge` source, and `endpoint_url` parameter to `CustomerGaugeToDF` task
and `CustomerGaugeToADLS` flow. This parameter enables to pass the endpoint URL by user.
- Added new parameter `outbox_list` at all leves in `Outlook` connector to tag mailbox folders.

Fixed
- Fixed `to_parquet()` from `base.py` when there is no directory specified in path

Changed
- Changed loop when retrieving email in `Outlook` source file, to cover all possible folders and subfolders.

Added
- Added `BusinessCore` source class
- Added `BusinessCoreToParquet` task class
- Added `Eurostat` source, task and flow classes
- Added `verify` parameter to `handle_api_response()`.
- Added `to_parquet()` in `base.py`
- Added new source class `SAPRFCV2` in `sap_rfc.py` with new approximation.
- Added new parameter `rfc_replacement` to `sap_rfc_to_adls.py` to replace
an extra separator character within a string column to avoid conflicts.
- Added `rfc_unique_id` in `SAPRFCV2` to merge chunks on this column.
- Added `close_connection()` to `SAPRFC` and `SAPRFCV2`

Fixed
- Removed `try-except` sentence and added a new logic to remove extra separators in `sap_rfc.py`
source file, to vaoid a mismatch in columns lenght between iterative connections to SAP tables.
- When `SAP` tables are updated during `sap_rfc.py` scrip running, if there are chunks, the
columns in the next chunk are unrelated rows.
- Fixed `sap_rfc.py` source file to not breakdown by both,
and extra separator in a row and adding new rows in SAP table between iterations.

Added
- Added `anonymize_df` task function to `task_utils.py` to anonymize data in the dataframe in selected columns.
- Added `Hubspot` source class
- Added `HubspotToDF` task class
- Added `HubspotToADLS` flow class
- Added `CustomerGauge` source class
- Added `CustomerGaugeToDF` task class
- Added `CustomerGaugeToADLS` flow class

Added
- Added `validate_date_filter` parameter to `Epicor` source, `EpicorOrdersToDF` task and `EpicorOrdersToDuckDB` flow.
This parameter enables user to decide whether or not filter should be validated.
- Added `Mediatool` source class
- Added `MediatoolToDF` task class
- Added `MediatoolToADLS` flow class
- Added option to disable `check_dtypes_sort` in `ADLSToAzureSQL` flow.
- Added `query` parameter to `BigQueryToADLS` flow and `BigqueryToDF` task to be able to enter custom SQL query.
- Added new end point `conversations/details/query` connection to `Genesys` task.
- Added new task `filter_userid` in `GenesysToADLS` flow to filter out by user Ids list, previously passed by the user.

Changed
- Changed parameter name in `BigQueryToADLS` flow - from `credentials_secret` to `credentials_key`