Adds a `create_dags` function, located in `gusty.experimental`, that allows users to create multiple DAGs from a single function.
By default, a simple `for` loop is used to create DAGs, but by setting `concurrent=True`, an asynchronous loop is used to create the DAGs, (hopefully) speeding up creation time.
Assuming multiple gusty DAG directories live behind `/usr/local/airflow/dags/gusty_dags` (e.g. `.../gusty_dags/dag_a`, `.../gusty_dags/dag_b`, so on), the concurrent implementation would look something like this:
python3
import airflow
from gusty.experimental import create_dags
create_dags(
"/usr/local/airflow/dags/gusty_dags",
globals(),
concurrent=True,
schedule_interval="0 12 * * *",
default_args={"email": "defaultgusty.com", "owner": "default"},
wait_for_defaults={"timeout": 679},
)
Additional parameters available when `concurrent=True`:
- `concurrent_timeout` - The maximum length (in seconds) that `create_dags` can run before a timeout error occurs. Defaults to 90% of [Airflow's `dagbag_import_timeout`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#dagbag-import-timeout).
- `max_concurrency` - The maximum number of concurrent `create_dag` operations. Defaults to `8`.