Gusty

Adding the option for users to specify a `GUSTY_HOME` environment variable, which will be used to locate their local operators

.py task files are now more intuitive to use.

At the bare minimum, all users have to do is write Python code in a .py file and gusty will pass `PythonOperator` a small function to execute the .py file. A `PythonOperator` task can be generated with as little code as:

python
print("hello world")


For parameters and configuration, a raw markdown chunk like below will work behave like any other task file:

python
---
dependencies:
- other_task
---

print("hello world")


.py task files can accept an `operator` parameter in markdown, which means that users can run whatever operator they wish against their Python code.

Lastly, if a user does specify a function name under `python_callable`, and the operator for the .py file inherits Airflow's `PythonOperator`, then gusty will search the .py task file for a function named the same as what is specified in the markdown. For example this markdown will tell gusty to look for a function named `say_hello` in the .py task file contents:

python
---
dependencies:
- other_task
python_callable: say_hello
---

def say_hello():
print("hello world")


These improvements make starting with .py task files as easy as just writing any Python code, and allow for a greater level of configuration compared to previously.

In addition to .yml, .ipynb, and .Rmd file types, gusty now offers support for .py files!

gusty will turn any .py file located in a DAG directory into a `PythonOperator`. The only requirement for these .py files is that they define a function called `python_callable`. If gusty does not detect a function called `python_callable` from the .py task file, it will raise an error.

Here is a simple hello world .py task file:

python
def python_callable():
phrase = "hello world"
print(phrase)


gusty will also accept `dependencies` and `external_dependencies` variables from .py task files. Adding to the example above:

python
dependencies = [
"same_dag_task"
]
external_dependencies = [
{"another_dag": "another_task"},
{"a_whole_dag": "all"}
]
def python_callable():
phrase = "hello world"
print(phrase)


Lastly, gusty will also utilize any variables that relate to Airflow's [BaseOperator class](https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/models/baseoperator/index.html).

There are plans to make .py task files more feature complete in the future, but hopefully this simple design will allow users to create `PythonOperator` tasks without even thinking about them!

ignore_subfolders` now ignores **all** subfolders at all levels.

In previous gusty versions (e.g. 0.2.0), in the internals, `template_fields` on an operator were checked to determine which fields were to be passed by gusty to that operator. As of gusty 0.3.x, `template_fields` has been made redundant as gusty now checks the operator's `__init__` method (as well as Airflow's `BaseOperator`'s `__init__` method) for which fields from a YAML template should be passed to the operator. This should benefit users who use local custom operators, as they will no longer have to add a `template_fields` attribute to the operator to ensure gusty passes through the correct fields. Instead, anything in the `__init__` method will get detected and passed by gusty.

leaf_tasks` is a list of task ids that can be passed to `create_dag` or a DAG's `METADATA.yml` which explicitly sets those tasks at the end of the DAG. It is the opposite of `root_tasks`.

The motivation for adding `leaf_tasks` is that, together with `root_tasks`, users now have full control to set up and break down dags using designated tasks, e.g. a root task spins up a Dask cluster, subsequent tasks run against that Dask cluster, and finally the leaf task terminates the Dask cluster.

ignore_subfolders` boolean can now be passed as an argument to either `create_dag` or at the DAG-level `METADATA.yml` to ignore all subfolders in a DAG folder.

This may be useful if you had previously been storing rendered reports in a subfolder within your DAG folder, but don't wish for this subfolder to accidentally be turned into a task group. As a reminder, all subfolders that start with either `.` or `_` will be ignored by gusty.