Tensorflow-model-analysis

Major Features and Improvements

Bug fixes and other changes

* Updated vulcanized_tfma.js with UI changes in addons/fairness_indicators.

Breaking changes

Deprecations

Major Features and Improvements

Bug fixes and other changes

* Updated to use tf.io.gfile for reading config files (fixes issue with
reading from GCS/HDFS in 0.15.0 and 0.15.1 releases).

Breaking changes

Deprecations

Major Features and Improvements

* Added support for defaulting to using class IDs when classes are not present
in outputs for multi-class metrics (for use in keras model_to_estimator).
* Added example count metrics (`tfma.metrics.ExampleCount` and
`tfma.metrics.WeightedExampleCount`) for use with V2 metrics API.
* Added calibration metrics (`tfma.metrics.MeanLabel`,
`tfma.metrics.MeanPrediction`, and `tfma.metrics.Calibration`) for use with
V2 metrics API.
* Added `tfma.metrics.ConfusionMatrixAtThresholds` for use with V2 metrics
API.
* Added `tfma.metrics.CalibrationPlot` and `tfma.metrics.AUCPlot` for use with
V2 metrics API.
* Added multi_class / multi_label plots (
`tfma.metrics.MultiClassConfusionMatrixAtThresholds`,
`tfma.metrics.MultiLabelConfusionMatrixAtThresholds`) for use with V2
metrics API.
* Added `tfma.metrics.NDCG` metric for use with V2 metrics API.
* Added `calibration` as a post export metric.

Bug fixes and other changes

* Depends on `tensorflow>=1.15,<3.0`.
* Starting from 1.15, package `tensorflow` comes with GPU support. Users
won't need to choose between `tensorflow` and `tensorflow-gpu`.
* Caveat: `tensorflow` 2.0.0 is an exception and does not have GPU
support. If `tensorflow-gpu` 2.0.0 is installed before installing
`tensorflow_model_analysis`, it will be replaced with `tensorflow`
2.0.0. Re-install `tensorflow-gpu` 2.0.0 if needed.

Breaking changes

Deprecations

Major Features and Improvements

* Added V2 of PredictExtractor that uses TF 2.0 signature APIs and supports
keras models (note: keras model evaluation not fully supported yet).
* `tfma.run_model_analysis`, `tfma.default_extractors`,
`tfma.default_evaluators`, and `tfma.default_writers` now allow settings to
be passed as an `EvalConfig`.
* `tfma.run_model_analysis`, `tfma.default_extractors`,
`tfma.default_evaluators`, and `tfma.default_writers` now allow multiple
models to be passed (note: multi-model support not fully implemented yet).
* Added InputExtractor for extracting labels, features, and example weights
from tf.Examples.
* Added Fairness Indicator as an addon.

Bug fixes and other changes

* Enabled TF 2.0 support using compat.v1.
* Added support for slicing on native dicts of features in addition to FPL
types.
* For multi-output and / or multi-class models, please provide output_name and
/ or class_id to tfma.view.render_plot.
* Replaced dependency on `tensorflow-transform` with `tfx-bsl`. If running
with latest master, `tfx-bsl` must also be latest master.
* Depends on `tfx-bsl>=0.15,<0.16`.
* Slicing now supports conversion between int/floats and strings.
* Depends on `apache-beam[gcp]>=2.16,<3`.
* Depends on `six==1.12`.

Breaking changes

* tfma.EvalResult.slicing_metrics now contains nested dictionaries of output,
class id and then metric names.
* Update config serialization to use JSON instead of pickling and reformat
config to include input_data_specs, model_specs, output_data_specs, and
metrics_specs.
* Requires pre-installed TensorFlow >=1.15,<3.

Deprecations

Major Features and Improvements

* Added documentation on architecture.
* Added an `adapt_to_remove_metrics` function to `tfma.exporter` which can be
used to remove metrics incompatible with TFMA (e.g. `py_func` or streaming
metrics) before exporting the TFMA EvalSavedModel.
* Added support for passing sparse int64 tensors to precision/recallk.
* Added support for binarization of multiclass metrics that use labels of the
from (N) in addition to (N, 1).
* Added support for using iterators with EvalInputReceiver.
* Improved performance of confidence interval computations by modifying the
pipeline shape.
* Added QueryBasedMetricsEvaluator which supports computing query-based
metrics (e.g. normalized discounted cumulative gain).
* Added support for merging metrics produced by different evaluators.
* Added support for blacklisting specified features from fetches.
* Added functionality to the FeatureExtractor to specify the features dict as
a possible destination.
* Added support for label vocabularies for binary and multi-class estimators
that support the new ALL_CLASSES prediction output.
* Move example parsing in aggregation into the graph for performance
improvements in both standard and model_agnostic evaluation modes.
* Created separate ModelLoader type for loading the EvalSavedModel.

Bug fixes and other changes

* Upgraded codebase for TF 2.0 compatibility.
* Make metrics-related operations thread-safe by wrapping them with locks.
This eliminates race conditions that were previously possible in
multi-threaded runners which could result in incorrect metric values.
* More flexible `FanoutSlices`.
* Limit the number of sampling buckets to 20.
* Improved performance in Confidence Interval computation.
* Refactored poisson bootstrap code to be re-usable in other evaluators.
* Refactored k-anonymity code to be re-usable in other evaluators.
* Fixed slicer feature string value handling in Python3.
* Added support for example weight keys for multi-output models.
* Added option to set the desired batch size when calling run_model_analysis.
* Changed TFRecord compression type from UNCOMPRESSED to AUTO.
* Depends on `apache-beam[gcp]>=2.14,<3`.
* Depends on `numpy>=1.16,<2`.
* Depends on `protobuf>=3.7,<4`.
* Depends on `scipy==1.1.0`.
* Added support to change k_anonymization_count value via EvalConfig.

Breaking changes

* Removed uses of deprecated tf.contrib packages (where possible).
* `tfma.default_writers` now requires the `eval_saved_model` to be passed as
an argument.
* Requires pre-installed TensorFlow >=1.14,<2.

Deprecations

Major Features and Improvements

* Added support for squared pearson correlation (R squared) post export
metric.
* Added support for mean absolute error post export metric.
* Added support for mean squared error and root mean squared error post export
metric.
* Added support for not computing metrics for slices with less than a given
number of examples.

Bug fixes and other changes

* Cast / convert labels for precision / recall at K so that they work even if
the label and the classes Tensors have different types, as long as the types
are compatible.
* Post export metrics will now also search for prediction keys prefixed by
metric_tag if it is specified.
* Added support for precision/recall k using canned estimators provided
label vocab not used.
* Preserve unicode type of slice keys when serialising to and deserialising
from disk, instead of always converting them to bytes.
* Use `__slots__` in accumulators.

Breaking changes

* Expose Python 3 types in the code (this will break Python 2 compatibility)

Deprecations