Gusty

gusty 0.5.0 introduces an intuitive way to use .py files as task files. The full release notes for gusty 0.5.0 can be found [here](https://github.com/chriscardillo/gusty/releases/tag/v0.5.0).

In short, by default, gusty treats .py files in DAG directory as a file to be executed by Airflow's `PythonOperator`, meaning your .py file can say as little as:

python
print("this will become an operator")


and gusty will take care of the rest.

Of course, .py files can be configured just like any other task file. For this, gusty reads raw markdown in the following format at the beginning of your file.

python
---
operator: airflow.operators.python.PythonVirtualenvOperator
requirements:
- "siuba==0.0.24"
python_callable: print_phrase
---

def print_phrase():
print("this will become an operator")


We designated a new `operator`, whose arguments include `requirements`, and we specified a `python_callable`, which tells gusty that there is a defined function named `print_phrase` that we want to use as the `python_callable` function for our operator.

Please note gusty will only search for/attach an actual callable function to `python_callable` when the operator inherits from Airflow's `PythonOperator` class.

In addition to .yml, .ipynb, and .Rmd file types, gusty now offers support for .py files!

gusty will turn any .py file located in a DAG directory into a `PythonOperator`. The only requirement for these .py files is that they define a function called `python_callable`. If gusty does not detect a function called `python_callable` from the .py task file, it will raise an error.

Here is a simple hello world .py task file:

python
def python_callable():
phrase = "hello world"
print(phrase)


gusty will also accept `dependencies` and `external_dependencies` variables from .py task files. Adding to the example above:

python
dependencies = [
"same_dag_task"
]
external_dependencies = [
{"another_dag": "another_task"},
{"a_whole_dag": "all"}
]
def python_callable():
phrase = "hello world"
print(phrase)


Lastly, gusty will also utilize any variables that relate to Airflow's [BaseOperator class](https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/models/baseoperator/index.html).

There are plans to make .py task files more feature complete in the future, but hopefully this simple design will allow users to create `PythonOperator` tasks without even thinking about them!

ignore_subfolders` now ignores **all** subfolders at all levels.

In previous gusty versions (e.g. 0.2.0), in the internals, `template_fields` on an operator were checked to determine which fields were to be passed by gusty to that operator. As of gusty 0.3.x, `template_fields` has been made redundant as gusty now checks the operator's `__init__` method (as well as Airflow's `BaseOperator`'s `__init__` method) for which fields from a YAML template should be passed to the operator. This should benefit users who use local custom operators, as they will no longer have to add a `template_fields` attribute to the operator to ensure gusty passes through the correct fields. Instead, anything in the `__init__` method will get detected and passed by gusty.

leaf_tasks` is a list of task ids that can be passed to `create_dag` or a DAG's `METADATA.yml` which explicitly sets those tasks at the end of the DAG. It is the opposite of `root_tasks`.

The motivation for adding `leaf_tasks` is that, together with `root_tasks`, users now have full control to set up and break down dags using designated tasks, e.g. a root task spins up a Dask cluster, subsequent tasks run against that Dask cluster, and finally the leaf task terminates the Dask cluster.

ignore_subfolders` boolean can now be passed as an argument to either `create_dag` or at the DAG-level `METADATA.yml` to ignore all subfolders in a DAG folder.

This may be useful if you had previously been storing rendered reports in a subfolder within your DAG folder, but don't wish for this subfolder to accidentally be turned into a task group. As a reminder, all subfolders that start with either `.` or `_` will be ignored by gusty.

- **DAG-level External Dependencies** - External dependencies can now be listed at the DAG level, either in `METADATA.yml` or in `create_dag`.
- **Root Tasks** - Users can designate root tasks for DAGs by name using the `root_tasks` argument. gusty will place these tasks at the root of the DAG, and raise a helpful error if the designated root task either lists dependencies or is listed as a dependency by other tasks.
- **Better Sensor Support** - Some sensors keyword arguments were previously not being detected by the task builder, causes sensor YAML specs to error. This has been fixed.