Airflow

New features:

[AIRFLOW-2524] Airflow integration with AWS Sagemaker
[AIRFLOW-2657] Add ability to delete DAG from web ui
[AIRFLOW-2780] Adds IMAP Hook to interact with a mail server
[AIRFLOW-2794] Add delete support for Azure blob
[AIRFLOW-2912] Add operators for Google Cloud Functions
[AIRFLOW-2974] Add Start/Restart/Terminate methods Databricks Hook
[AIRFLOW-2989] No Parameter to change bootDiskType for DataprocClusterCreateOperator
[AIRFLOW-3078] Basic operators for Google Compute Engine
[AIRFLOW-3147] Update Flask-AppBuilder version
[AIRFLOW-3231] Basic operators for Google Cloud SQL (deploy / patch / delete)
[AIRFLOW-3276] Google Cloud SQL database create / patch / delete operators

Improvements:

[AIRFLOW-393] Add progress callbacks for FTP downloads
[AIRFLOW-520] Show Airflow version on web page
[AIRFLOW-843] Exceptions now available in context during on_failure_callback

--------------------------
[AIRFLOW-2870] Use abstract TaskInstance for migration
[AIRFLOW-2859] Implement own UtcDateTime (3708)
[AIRFLOW-2140] Don't require kubernetes for the SparkSubmit hook
[AIRFLOW-2869] Remove smart quote from default config
[AIRFLOW-2857] Fix Read the Docs env
[AIRFLOW-2817] Force explicit choice on GPL dependency
[AIRFLOW-2716] Replace async and await py3.7 keywords
[AIRFLOW-2810] Fix typo in Xcom model timestamp
[AIRFLOW-2710] Clarify fernet key value in documentation
[AIRFLOW-2606] Fix DB schema and SQLAlchemy model
[AIRFLOW-2646] Fix setup.py not to install snakebite on Python3
[AIRFLOW-2604] Add index to task_fail
[AIRFLOW-2650] Mark SchedulerJob as succeed when hitting Ctrl-c
[AIRFLOW-2678] Fix db schema unit test to remove checking fab models
[AIRFLOW-2624] Fix webserver login as anonymous
[AIRFLOW-2654] Fix incorret URL on refresh in Graph View of FAB UI
[AIRFLOW-2668] Handle missing optional cryptography dependency
[AIRFLOW-2681] Include last dag run of externally triggered DAGs in UI.
[AIRFLOW-1840] Support back-compat on old celery config
[AIRFLOW-2612][AIRFLOW-2534] Clean up Hive-related tests
[AIRFLOW-2608] Implements/Standardize custom exceptions for experimental APIs
[AIRFLOW-2607] Fix failing TestLocalClient
[AIRFLOW-2638] dbapi_hook: support REPLACE INTO
[AIRFLOW-2542][AIRFLOW-1790] Rename AWS Batch Operator queue to job_queue
[AIRFLOW-2567] Extract result from the kubernetes pod as Xcom
[AIRFLOW-XXX] Adding REA Group to readme
[AIRFLOW-2601] Allow user to specify k8s config
[AIRFLOW-2559] Azure Fileshare hook
[AIRFLOW-1786] Enforce correct behavior for soft-fail sensors
[AIRFLOW-2355] Airflow trigger tag parameters in subdag
[AIRFLOW-2613] Fix Airflow searching .zip bug
[AIRFLOW-2627] Add a sensor for Cassandra
[AIRFLOW-2634][AIRFLOW-2534] Remove dependency for impyla
[AIRFLOW-2611] Fix wrong dag volume mount path for kubernetes executor
[AIRFLOW-2562] Add Google Kubernetes Engine Operators
[AIRFLOW-2630] Fix classname in test_sql_sensor.py
[AIRFLOW-2534] Fix bug in HiveServer2Hook
[AIRFLOW-2586] Stop getting AIRFLOW_HOME value from config file in bash operator
[AIRFLOW-2605] Fix autocommit for MySqlHook
[AIRFLOW-2539][AIRFLOW-2359] Move remaing log config to configuration file
[AIRFLOW-1656] Tree view dags query changed
[AIRFLOW-2617] add imagePullPolicy config for kubernetes executor
[AIRFLOW-2429] Fix security/task/sensors/ti_deps folders flake8 error
[AIRFLOW-2550] Implements API endpoint to list DAG runs
[AIRFLOW-2512][AIRFLOW-2522] Use google-auth instead of oauth2client
[AIRFLOW-2429] Fix operators folder flake8 error
[AIRFLOW-2585] Fix several bugs in CassandraHook and CassandraToGCSOperator
[AIRFLOW-2597] Restore original dbapi.run() behavior
[AIRFLOW-2590] Fix commit in DbApiHook.run() for no-autocommit DB
[AIRFLOW-1115] fix github oauth api URL
[AIRFLOW-2587] Add TIMESTAMP type mapping to MySqlToHiveTransfer
[AIRFLOW-2591][AIRFLOW-2581] Set default value of autocommit to False in DbApiHook.run()
[AIRFLOW-59] Implement bulk_dump and bulk_load for the Postgres hook
[AIRFLOW-2533] Fix path to DAG's on kubernetes executor workers
[AIRFLOW-2581] RFLOW-2581] Fix DbApiHook autocommit
[AIRFLOW-2578] Add option to use proxies in JiraHook
[AIRFLOW-2575] Make gcs to gcs operator work with large files
[AIRFLOW-437] Send TI context in kill zombies
[AIRFLOW-2566] Change backfill to rerun failed tasks
[AIRFLOW-1021] Fix double login for new users with LDAP
[AIRFLOW-XXX] Typo fix
[AIRFLOW-2561] Fix typo in EmailOperator
[AIRFLOW-2573] Cast BigQuery TIMESTAMP field to float
[AIRFLOW-2560] Adding support for internalIpOnly to DataprocClusterCreateOperator
[AIRFLOW-2565] templatize cluster_label
[AIRFLOW-83] add mongo hook and operator
[AIRFLOW-2558] Clear task/dag is clearing all executions
[AIRFLOW-XXX] Fix doc typos
[AIRFLOW-2513] Change `bql` to `sql` for BigQuery Hooks & Ops
[AIRFLOW-2557] Fix pagination for s3
[AIRFLOW-2545] Eliminate DeprecationWarning
[AIRFLOW-2500] Fix MySqlToHiveTransfer to transfer unsigned type properly
[AIRFLOW-2462] Change PasswordUser setter to correct syntax
[AIRFLOW-2525] Fix a bug introduced by commit dabf1b9
[AIRFLOW-2553] Add webserver.pid to .gitignore
[AIRFLOW-1863][AIRFLOW-2529] Add dag run selection widgets to gantt view
[AIRFLOW-2504] Log username correctly and add extra to search columns
[AIRFLOW-2551] Encode binary data with base64 standard rather than base64 url
[AIRFLOW-2537] Add reset-dagrun option to backfill command
[AIRFLOW-2526] dag_run.conf can override params
[AIRFLOW-2544][AIRFLOW-1967] Guard against next major release of Celery, Flower
[AIRFLOW-XXX] Add Yieldr to who is using airflow
[AIRFLOW-2547] Describe how to run tests using Docker
[AIRFLOW-2538] Update faq doc on how to reduce airflow scheduler latency
[AIRFLOW-2529] Improve graph view performance and usability
[AIRFLOW-2517] backfill support passing key values through CLI
[AIRFLOW-2532] Support logs_volume_subpath for KubernetesExecutor
[AIRFLOW-2466] consider task_id in _change_state_for_tis_without_dagrun
[AIRFLOW-2519] Fix CeleryExecutor with SQLAlchemy
[AIRFLOW-2402] Fix RBAC task log
[AIRFLOW-XXX] Add M4U to user list
[AIRFLOW-2536] docs about how to deal with airflow initdb failure
[AIRFLOW-2530] KubernetesOperator supports multiple clusters
[AIRFLOW-1499] Eliminate duplicate and unneeded code
[AIRFLOW-2521] backfill - make variable name and logging messages more acurate
[AIRFLOW-2429] Fix hook, macros folder flake8 error
[Airflow-XXX] add Prime to company list
[AIRFLOW-2525] Fix PostgresHook.copy_expert to work with "COPY FROM"
[AIRFLOW-2515] Add dependency on thrift_sasl to hive extra
[AIRFLOW-2523] Add how-to for managing GCP connections
[AIRFLOW-2510] Introduce new macros: prev_ds and next_ds
[AIRFLOW-1730] Unpickle value of XCom queried from DB
[AIRFLOW-2518] Fix broken ToC links in integration.rst
[AIRFLOW-1472] Fix SLA misses triggering on skipped tasks.
[AIRFLOW-2520] CLI - make backfill less verbose
[AIRFLOW-2107] add time_partitioning to run_query on BigQueryBaseCursor
[AIRFLOW-1057][AIRFLOW-1380][AIRFLOW-2362][2362] AIRFLOW Update DockerOperator to new API
[AIRFLOW-2415] Make airflow DAG templating render numbers
[AIRFLOW-2473] Fix wrong skip condition for TransferTests
[AIRFLOW-2472] Implement MySqlHook.bulk_dump
[AIRFLOW-2419] Use default view for subdag operator
[AIRFLOW-2498] Fix Unexpected argument in SFTP Sensor
[AIRFLOW-2509] Separate config docs into how-to guides
[AIRFLOW-2429] Add BaseExecutor back
[AIRFLOW-2429] Fix dag, example_dags, executors flake8 error
[AIRFLOW-2502] Change Single triple quotes to double for docstrings
[AIRFLOW-2503] Fix broken links in CONTRIBUTING.md
[AIRFLOW-2501] Refer to devel instructions in docs contrib guide
[AIRFLOW-2429] Fix contrib folder's flake8 errors
[AIRFLOW-2471] Fix HiveCliHook.load_df to use unused parameters

[AIRFLOW-1897][AIRFLOW-1873] Task Logs for running instance not visible in WebUI
[AIRFLOW-XXX] Make sure session is committed
[AIRFLOW-1896] FIX bleach <> html5lib incompatibility
[AIRFLOW-XXX] Fix log handler test
[AIRFLOW-1873] Set TI.try_number to right value depending TI state
[AIRFLOW-1554] Fix wrong DagFileProcessor termination method call
[AIRFLOW-1872] Set context for all handlers including parents
[AIRFLOW-XXX] Add dask lock files to excludes
[AIRFLOW-1839] Fix more bugs in S3Hook boto -> boto3 migration
[AIRFLOW-1795] Correctly call S3Hook after migration to boto3
[AIRFLOW-1813] Bug SSH Operator empty buffer
[AIRFLOW-1794] Remove uses of Exception.message for Python 3
[AIRFLOW-1799] Fix logging line which raises errors

-------------------------

9a53e66 [AIRFLOW-809][AIRFLOW-1] Use __eq__ ColumnOperator When Testing Booleans
333e0b3 [AIRFLOW-1296] Propagate SKIPPED to all downstream tasks
93825d5 Re-enable caching for hadoop components
33a9dcb Pin Hive and Hadoop to a specific version and create writable warehouse dir
7cff6cd [AIRFLOW-1308] Disable nanny usage for Dask

69bd269 [AIRFLOW-1160] Update Spark parameters for Mesos
9692510 [AIRFLOW 1149][AIRFLOW-1149] Allow for custom filters in Jinja2 templates
6de5330 [AIRFLOW-1119] Fix unload query so headers are on first row[]
b4e9eb8 [AIRFLOW-1089] Add Spark application arguments
a4083f3 [AIRFLOW-1078] Fix latest_runs endpoint for old flask versions
7a02841 [AIRFLOW-1074] Don't count queued tasks for concurrency limits
a2c18a5 [AIRFLOW-1064] Change default sort to job_id for TaskInstanceModelView
d1c64ab [AIRFLOW-1038] Specify celery serialization options explicitly
b4ee88a [AIRFLOW-1036] Randomize exponential backoff
9fca409 [AIRFLOW-993] Update date inference logic
272c2f5 [AIRFLOW-1167] Support microseconds in FTPHook modification time
c7c0b72 [AIRFLOW-1179] Fix Pandas 0.2x breaking Google BigQuery change
acd0166 [AIRFLOW-1263] Dynamic height for charts
7f33f6e [AIRFLOW-1266] Increase width of gantt y axis
fc33c04 [AIRFLOW-1290] set docs author to 'Apache Airflow'
2e9eee3 [AIRFLOW-1282] Fix known event column sorting
2389a8a [AIRFLOW-1166] Speed up _change_state_for_tis_without_dagrun
bf966e6 [AIRFLOW-1192] Some enhancements to qubole_operator
57d5bcd [AIRFLOW-1281] Sort variables by key field by default
802fc15 [AIRFLOW-1244] Forbid creation of a pool with empty name
1232b6a [AIRFLOW-1243] DAGs table has no default entries to show
b0ba3c9 [AIRFLOW-1227] Remove empty column on the Logs view
c406652 [AIRFLOW-1226] Remove empty column on the Jobs view
51a83cc [AIRFLOW-1199] Fix create modal
cac7d4c [AIRFLOW-1200] Forbid creation of a variable with an empty key
5f3ee52 [AIRFLOW-1186] Sort dag.get_task_instances by execution_date
f446c08 [AIRFLOW-1145] Fix closest_date_partition function with before set to True If we're looking for the closest date before, we should take the latest date in the list of date before.
93b8e96 [AIRFLOW-1180] Fix flask-wtf version for test_csrf_rejection
bb56805 [AIRFLOW-1170] DbApiHook insert_rows inserts parameters separately
093b2f0 [AIRFLOW-1150] Fix scripts execution in sparksql hook[]
777f181 [AIRFLOW-1168] Add closing() to all connections and cursors
bc8e912 [AIRFLOW-XXX] Updating CHANGELOG, README, and UPDATING after 1.8.1 release

-------------------------

[AIRFLOW-1142] SubDAG Tasks Not Executed Even Though All Dependencies Met
[AIRFLOW-1138] Add licenses to files in scripts directory
[AIRFLOW-1127] Move license notices to LICENSE instead of NOTICE
[AIRFLOW-1124] Do not set all task instances to scheduled on backfill
[AIRFLOW-1120] Update version view to include Apache prefix
[AIRFLOW-1062] DagRunfind returns wrong result if external_trigger=False is specified
[AIRFLOW-1054] Fix broken import on test_dag
[AIRFLOW-1050] Retries ignored - regression
[AIRFLOW-1033] TypeError: can't compare datetime.datetime to None
[AIRFLOW-1017] get_task_instance should return None instead of throw an exception for non-existent TIs
[AIRFLOW-1011] Fix bug in BackfillJob._execute() for SubDAGs
[AIRFLOW-1004] `airflow webserver -D` runs in foreground
[AIRFLOW-1001] Landing Time shows "unsupported operand type(s) for -: 'datetime.datetime' and 'NoneType'" on example_subdag_operator
[AIRFLOW-1000] Rebrand to Apache Airflow instead of Airflow
[AIRFLOW-989] Clear Task Regression
[AIRFLOW-974] airflow.util.file mkdir has a race condition
[AIRFLOW-906] Update Code icon from lightning bolt to file
[AIRFLOW-858] Configurable database name for DB operators
[AIRFLOW-853] ssh_execute_operator.py stdout decode default to ASCII
[AIRFLOW-832] Fix debug server
[AIRFLOW-817] Trigger dag fails when using CLI + API
[AIRFLOW-816] Make sure to pull nvd3 from local resources
[AIRFLOW-815] Add previous/next execution dates to available default variables.
[AIRFLOW-813] Fix unterminated unit tests in tests.job (tests/job.py)
[AIRFLOW-812] Scheduler job terminates when there is no dag file
[AIRFLOW-806] UI should properly ignore DAG doc when it is None
[AIRFLOW-794] Consistent access to DAGS_FOLDER and SQL_ALCHEMY_CONN
[AIRFLOW-785] ImportError if cgroupspy is not installed