Tensorflow-model-analysis

Major Features and Improvements

* Added support for specifying weighted vs unweighted metrics. The setting is
available in the `tfma.MetricsSpec(
example_weights=tfma.ExampleWeightOptions(weighted=True, unweighted=True))`.
If no options are provided then TFMA will default to weighted provided the
associated `tfma.ModelSpec` has an example weight key configured, otherwise
unweighted will be used.

Bug fixes and other Changes

* Added support for example_weights that are arrays.

* Reads baseUrl in JupyterLab to support TFMA rendering:
https://github.com/tensorflow/model-analysis/issues/112

* Fixing couple of issues with CIDerivedMetricComputation:

* no CI derived metric, deriving from private metrics such as
binary_confusion_matrices, was being computed
* convert_slice_metrics_to_proto method didn't have support for bounded
values metrics.

* Depends on `tfx-bsl>=1.4.0,<1.5.0`.

* Depends on `tensorflow-metadata>=1.4.0,<1.5.0`.

* Depends on `apache-beam[gcp]>=2.33,<3`.

Breaking Changes

* Confidence intervals for scalar metrics are no longer stored in the
`MetricValue.bounded_value`. Instead, the confidence interval for a metric
can be found under `MetricKeysAndValues.confidence_interval`.
* MetricKeys now require specifying whether they are weighted (
`tfma.metrics.MetricKey(..., example_weighted=True)`) or unweighted (the
default). If the weighting is unknown then `example_weighted` will be None.
Any metric computed outside of a `tfma.metrics.MetricConfig` setting (i.e.
metrics loaded from a saved model) will have the weighting set to None.
* `ExampleCount` is now weighted based on `tfma.MetricsSpec.example_weights`
settings. `WeightedExampleCount` has been deprecated (use `ExampleCount`
instead). To get unweighted example counts (i.e. the previous implementation
of `ExampleCount`), `ExampleCount` must now be added to a `MetricsSpec`
where `example_weights.unweighted` is true. To get a weighted example count
(i.e. what was previously `WeightedExampleCount`), `ExampleCount` must now
be added to a `MetricsSpec` where `example_weights.weighted` is true.

Deprecations

* Deprecated python3.6 support.

Major Features and Improvements

* N/A

Bug fixes and other Changes

* Correctly skips non-numeric numpy array type metrics for confidence interval
computations.
* Depends on `apache-beam[gcp]>=2.32,<3`.
* Depends on `tfx-bsl>=1.3.0,<1.4.0`.

Breaking Changes

* In preparation for TFMA 1.0, the following imports have been moved (note
that other modules were also moved, but TFMA only supports types that are
explicitly declared inside of `__init__.py` files):
* `tfma.CombineFnWithModels` -> `tfma.utils.CombineFnWithModels`
* `tfma.DoFnWithModels` -> `tfma.utils.DoFnWithModels`
* `tfma.get_baseline_model_spec` -> `tfma.utils.get_baseline_model_spec`
* `tfma.get_model_type` -> `tfma.utils.get_model_type`
* `tfma.get_model_spec` -> `tfma.utils.get_model_spec`
* `tfma.get_non_baseline_model_specs` ->
`tfma.utils.get_non_baseline_model_specs`
* `tfma.verify_eval_config` -> `tfma.utils.verify_eval_config`
* `tfma.update_eval_config_with_defaults` ->
`tfma.utils.update_eval_config_with_defaults`
* `tfma.verify_and_update_eval_shared_models` ->
`tfma.utils.verify_and_update_eval_shared_models`
* `tfma.create_keys_key` -> `tfma.utils.create_keys_key`
* `tfma.create_values_key` -> `tfma.utils.create_values_key`
* `tfma.compound_key` -> `tfma.utils.compound_key`
* `tfma.unique_key` -> `tfma.utils.unique_key`

Deprecations

* N/A

Major Features and Improvements

* Added `SparseTensorValue` and `RaggedTensorValue` types for storing
in-memory versions of sparse and ragged tensor values used in extracts.
Tensor values used for features, etc should now be based on either
`np.ndarray`, `SparseTensorValue`, or `RaggedTensorValue` and not
tf.compat.v1 value types.
* Add `CIDerivedMetricComputation` metric type.

Bug fixes and other Changes

* Depends on `pyarrow>=1,<6`.
* Fixes bug when computing confidence intervals for
`binary_confusion_metrics.ConfusionMatrixAtThresholds` (or any other
structured metric).
* Fixed bug where example_count post_export_metric is added even if
include_default_metrics is False.
* Depends on `apache-beam[gcp]>=2.31,<2.32`.
* Depends on
`tensorflow>=1.15.2,!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,<3`.
* Depends on `tfx-bsl>=1.3.1,<1.4.0`.
* Fixes issue with jackknife confidence interval method that resulted in
erroneously large intervals.
* Fixes bug where calls to `_apply_binary_op_elementwise` could fail on
objects of types `binary_confusion_matrices.Matrices` and
`multi_class_confusion_matrix_metrics.Matrices` due to differing thresholds.

Breaking Changes

* Missing baseline model when change thresholds are present is not allowed
anymore, an exception will be raised unless the rubber_stamp flag is True.

Deprecations

* N/A

Major Features and Improvements

* Provided functionality for `slice_keys_sql` config. It's not available under
Windows.
* The `confidence_interval` field within `metrics_for_slice_pb2.MetricValue`
has been removed and the tag number reserved. This information now lives in
`metrics_for_slice_pb2.MetricKeyAndValue.confidence_interval`.

Bug fixes and other Changes

* Improve rendering of HTML stubs for TFMA and Fairness Indicators UI.
* Update README for JupyterLab 3
* Provide implementation of ExactMatch metric.
* Jackknife CI method now works with cross-slice metrics.
* Depends on `apache-beam[gcp]>=2.31,<3`.
* Depends on `tensorflow-metadata>=1.2.0,<1.3.0`.
* Depends on `tfx-bsl>=1.2.0,<1.3.0`.

Breaking Changes

* The binary_confusion_matrices metric formerly returned confusion matrix
counts (i.e number of {true,false} {positives,negatives}) and optionally a
set of representative examples in a single object. Now, this metric class
generates two separate metrics values when examples are configured: one
containing just the counts, and the other just examples. This should only
affect users who created a custom derived metric that used
binary_confusion_matrices metric as an input.

Deprecations

* N/A

Major Features and Improvements

* N/A

Bug fixes and other Changes

* Depends on `google-cloud-bigquery>>=1.28.0,<2.21`.
* Depends on `tfx-bsl>=1.1.0,<1.2.0`.

Breaking Changes

* N/A

Deprecations

* N/A

Major Features and Improvements

* N/A

Bug fixes and other Changes

* Depends on `protobuf>=3.13,<4`.
* Depends on `tensorflow-metadata>=1.1.0,<1.2.0`.
* Depends on `tfx-bsl>=1.1.0,<1.2.0`.

Breaking Changes

* N/A

Deprecations

* N/A