Rflow-tfx

Major Features and Improvements

* TFX CLI now supports runtime parameter on Kubeflow, Vertex, and Airflow.
Use it with '--runtime_parameter=<parameter_name>=<parameter_value>' flag.
In the case of multiple runtime parameters, format is as follows:
'--runtime_parameter=<parameter_name>=<parameter_value> --runtime_parameter
=<parameter_name>=<parameter_value>'
* Added Manual node in the experimental orchestrator.
* Placeholders support index access and JSON serialization for list type execution properties.
* Added `ImportSchemaGen` which is a dedicated component to import a
pre-defined schema file. ImportSchemaGen will replace `Importer` with
simpler syntax and less constraints. You have to pass the file path to the
schema file instead of the parent directory unlike `Importer`.
* Updated GCP Vertex Client to support EncryptionSpec and Cloud Labels.

Breaking Changes

For Pipeline Authors

* N/A

For Component Authors

* N/A

Deprecations

* The import name of KerasTuner has been changed from `kerastuner`
to `keras_tuner`. The import name of `kerastuner` is still supported.
A warning will occur when import from `kerastuner`, but does not affect
the usage.
* **Upcoming deprecation** - TFX 1.3.0 will be the last release to support
Python 3.6. Starting with TFX 1.4.0 Python 3.6 will no longer be supported.

Bug Fixes and Other Changes
* The default job name for Google Cloud AI Training jobs was changed from
'tfx_YYYYmmddHHMMSS' to 'tfx_YYYYmmddHHMMSS_xxxxxxxx', where 'xxxxxxxx' is
a random 8 digit hexadecimal string.
* Fix component to raise error if its input required channel (specified from
ComponentSpec) has no artifacts in it.
* Fixed an issue where ClientOptions with regional endpoint was
incorrectly left out in Vertex AI pusher.
* CLI now hides passed flags from user python files in "--pipeline-path". This
will prevent errors when user python file tries reading and parsing flags.
* Fixed missing type information marker file 'py.typed'.
* Fixed handling of artifacts with no PROPERTIES in scripts/run_component.py
* Fixed passing non-string execution properties and artifact properties in
scripts/run_component.py* Depends on `apache-beam[gcp]>=2.32,<3`.
* Depends on `google-cloud-bigquery>=1.28.0,<3`.
* Depends on `jinja2>=2.7.3,<4`, i.e. now supports Jinja 3.x.
* Depends on `keras-tuner>=1.0.4,<2`.
* Depends on `kfp>=1.6.1,!=1.7.2,<1.8.2` in \[kfp\] extra.
* Depends on `kfp-pipeline-spec>=>=0.1.10,<0.2`.
* Depends on `ml-metadata>=1.3.0,<1.4.0`.
* Depends on `struct2tensor>=0.34.0,<0.35.0`.
* Depends on `tensorflow>=1.15.2,!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,<3`.
* Depends on `tensorflow-data-validation>=1.3.0,<1.4.0`.
* Depends on `tensorflow-model-analysis>=0.34.1,<0.35.0`.
* Depends on `tensorflow-serving-api>=1.15,!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,<3`.
* Depends on `tensorflow-transform>=1.3.0,<1.4.0`.
* Depends on `tfx-bsl>=1.3.0,<1.4.0`.
* Depends on 'google-cloud-aiplatform>=0.5.0,<2'.

Documentation Updates

* N/A

Major Features and Improvements

* N/A

Breaking Changes

For Pipeline Authors

* N/A

For Component Authors

* N/A

Deprecations

* N/A

Bug Fixes and Other Changes

* Added support for a custom metadata-ui-json filename in KubeflowDagRunner.
* Fixed missing type information marker file 'py.typed'.

Documentation Updates

* N/A

Major Features and Improvements

* Added RuntimeParam support for Trainer's custom_config.
* TFX Trainer and Pusher now support Vertex, which can be enabled with
`ENABLE_VERTEX_KEY` key in `custom_config`.

Breaking Changes

For Pipeline Authors

* N/A

For Component Authors

* N/A

Deprecations

* N/A

Bug Fixes and Other Changes

* Fixed the issue that kfp_pod_name is not generated as an execution property
for Kubeflow Pipelines.
* Fixed issue when InputValuePlaceholder is used as component parameter in
container based component.
* Depends on `kubernetes>=10.0.1,<13`
* `CsvToExample` now supports multi-line strings.
* `tfx.benchmarks` package was removed from the Python TFX wheel. This package
is used only for benchmarking and not useful for end users.
* Fixed the issue for fairness_indicator_thresholds support of Evaluator.
* Depends on `apache-beam[gcp]>=2.31,<3`.
* Depends on `kfp-pipeline-spec>=0.1.8,<0.2`.
* Depends on `ml-metadata>=1.2.0,<1.3.0`.
* Depends on `struct2tensor>=0.33.0,<0.34.0`.
* Depends on `tensorflow-data-validation>=1.2.0,<1.3.0`.
* Depends on `tensorflow-model-analysis>=0.33.0,<0.34.0`.
* Depends on `tensorflow-transform>=1.2.0,<1.3.0`.
* Depends on `tfx-bsl>=1.2.0,<1.3.0`.

Documentation Updates

* N/A

Version 1.1.x (skipped)

To maintain version consistency among TFX Family libraries we skipped
the 1.1.x release for TFX library.

Major Features and Improvements

* Added tfx.v1 Public APIs, please refer to
[API doc](https://www.tensorflow.org/tfx/api_docs/python/tfx/v1) for details.
* Transform component now computes pre-transform and post-transform statistics
and stores them in new, indvidual outputs ('pre_transform_schema',
'pre_transform_stats', 'post_transform_schema', 'post_transform_stats',
'post_transform_anomalies'). This can be disabled by setting
`disable_statistics=True` in the Transform component.
* BERT cola and mrpc examples now demonstrate how to calculate statistics for
NLP features.
* TFX CLI now supports
[Vertex Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines/introduction).
use it with `--engine=vertex` flag.
* Telemetry: Only first-party tfx component's executor telemetry will be
collected. All other executors will be recorded as `third_party_executor`.
For labels longer than 63, keep first 63 characters (instead of last 63
characters before).
* Supports text type (use proto json string format) RuntimeParam for protos.
* Combined/moved taxi's runtime_parameter, kubeflow_local and kubleflow_gcp
example pipelines into one penguin_pipeline_kubeflow example
* Transform component now supports passing `stats_options_updater_fn` directly
as well as through the module file.
* Placeholders support accessing artifact property and custom property.
* Removed the extra node information in IR for KubeflowDagRunner, to reduce
size of generated IR.

Breaking Changes

* Removed unneccessary default values for required component input Channels.
* The `_PropertyDictWrapper` internal wrapper for `component.inputs` and
`component.outputs` was removed: `component.inputs` and `component.outputs`
are now unwrapped dictionaries, and the attribute accessor syntax (e.g.
`components.outputs.output_name`) is no longer supported. Please use the
dictionary indexing syntax (e.g. `components.outputs['output_name']`)
instead.

For Pipeline Authors

* N/A

For Component Authors

* Apache Beam support is migrated from TFX Base Components and Executors to
dedicated Beam Components and Executors. `BaseExecutor` will no longer embed
`beam_pipeline_args`. Custom executors for Beam powered components should
now extend BaseBeamExecutor instead of BaseExecutor.

Deprecations

* Deprecated nested RuntimeParam for Proto, Please use text type (proto json
string) RuntimeParam instead of Proto dict with nested RuntimeParam in it.

Bug Fixes and Other Changes

* Forces keyword arguments for AirflowComponent to make it compatible with
Apache Airflow 2.1.0 and later.
* Fixed issue where passing `analyzer_cache` to `tfx.components.Transform`
before there are any Transform cache artifacts published would fail.
* Included type information according to PEP-561. However, protobuf generated
files don't have type information, and you might need to ignore errors from
them. For example, if you are using `mypy`, see
[the related doc](https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-type-hints-for-third-party-library).
* Removed `six` dependency.
* Depends on `apache-beam[gcp]>=2.29,<3`.
* Depends on `google-cloud-bigquery>=1.28.0,<2.21`
* Depends on `ml-metadata>=1.0.0,<1.1.0`.
* Depends on `protobuf>=3.13,<4`.
* Depends on `struct2tensor>=0.31.0,<0.32.0`.
* Depends on `tensorflow>=1.15.2,!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,<3`.
* Depends on `tensorflow-data-validation>=1.0.0,<1.1.0`.
* Depends on `tensorflow-hub>=0.9.0,<0.13`.
* Depends on `tensorflowjs>=3.6.0,<4`.
* Depends on `tensorflow-model-analysis>=0.31.0,<0.32.0`.
* Depends on `tensorflow-serving-api>=1.15,!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,<3`.
* Depends on `tensorflow-transform>=1.0.0,<1.1.0`.
* Depends on `tfx-bsl>=1.0.0,<1.1.0`.

Documentation Updates

* Update the Guide of TFX to adopt 1.0 API.
* TFT and TFDV component documentation now describes how to
configure pre-transform and post-transform statistics, which can be used for
validating text features.

| `tfx-bsl` | `~=1.6.0` | `~=1.5.0` | Synced release train |
| `tensorflow-data-validation` | `~=1.6.0` | `~=1.5.0` | Synced release train |
| `tensorflow-transform` | `~=1.6.0` | `~=1.5.0` | Synced release train |
| `ml-metadata` | `~=1.6.0` | `~=1.5.0` | Synced release train |
| `tensorflow-model-analysis` | `~=0.37.0` | `~=0.36.0` | Synced release train |
| `struct2tensor` | `~=0.37.0` | `~=0.36.0` | Synced release train |

Documentation Updates

* N/A

Major Features and Improvements

* N/A

Breaking Changes

For Pipeline Authors

* N/A

For Component Authors

* N/A

Deprecations

* N/A

Bug Fixes and Other Changes

* Update resolver query in TFX -> KFP IR compiler with vertex placeholder
syntax.

Documentation Updates

* N/A