Pydiverse-pipedag

Latest version: v0.9.8

Safety actively analyzes 687918 Python packages for vulnerabilities to keep your Python projects secure.

Scan your dependencies

Page 5 of 6

0.5.0

- add support for DuckDB
- add support for pyarrow backed pandas dataframes
- support execution of subflow
- store final state of task in flow result object
- tasks now have a `position_hash` associated with them to identify them purely based on their position (e.g. stage, name and input wiring) inside a flow.
- breaking change to metadata: added position_hash to `tasks` metadata table and change type of hash columns from String(32) to String(20).
- `Flow`, `Subflow`, and `Result` objects now provide additional options for visualizing them
- added `unlogged_tables` flag to SQLTableStore for creating UNLOGGED tables with Postgres.
- created [`pipedag-manage`](reference/cli) command line utility with [`clear-metadata`](reference/cli:clear-metadata) command to help with migrating between different pipedag metadata versions.

0.4.1

- implement [](DaskEngine): orchestration engine for running multiple tasks in parallel
- implement [](DatabaseLockManager): lock manager based on locking mechanism provided by database

0.4.0

- update public interface
- encrypt IPC communication
- remove preemptive `os.makedirs` from ParquetTableCache
- improve logging and provide structlog utilities

0.3.0

- breaking change to pipedag.yaml:
introduced `args` subsections for arguments that are passed to backend classes
- fix ibm_db_sa bug when copying dataframes from cache: uppercase table names by default
- nicer readable SQL queries: use automatic aliases for inputs of SQLAlchemy tasks
- implement option ignore_task_version: disable eager task caching for some instances to reduce overhead from task version bumping
- implement local table cache: store input/output of dataframe tasks in parquet files and allow using it as cache to avoid rereading from database

0.2.4

- fix errors by increasing output_json length in metadata table
- fix cache invalidation: query normalization before checking for changes
- add rudimentary support for ibis tasks (postgres + mssql)
- add rudimentary support for polars + tidypolars tasks
- implemented pandas type mapping to avoid row wise type checks of object columns
- support pandas 2.0 (no arrow features used that)
- support sqlalchemy 2.0 (except for with polars)

0.2.3

- fixed python 3.9 compatibility (`traceback.format_exception` syntax changed)
- fixed deferred table copy when task is invalid (introduced with 0.2.2)
- fixed mssql to not reflect full schema while renamings happen
- fixed clearing of metadata tables for lazy tables and raw sql tables
- fixed mssql synonym resolution when reading input table for pandas task
- initial implementation of issue 62: make query canonical before hashing
- retry some DB calls in case they are aborted as deadlock victim

Page 5 of 6

© 2024 Safety CLI Cybersecurity Inc. All Rights Reserved.