Mediapipe

Major Features and Improvements

iOS
- Published MediaPipeTasksText and MediaPipeTasksVision CocoaPods at version `0.10.1-alpha-2`
- Add FaceDetector iOS API
- Add Gesture Recognizer iOS API

Web
- Add iOS support for GPU processing for Segmentation Tasks
- Add .close() method to ImageSegmenterResult/InteractiveSegmenterResult/PoseLandmarkerResult
- Add quality scores to Segmenter tasks
- Make FaceLandmarker result non-optional

Bug Fixes and Other Changes

Android
- Remove unused MediaPipe Tasks Android sample

iOS
- Added validation of C++ image classification result packet in MPPImageClassifierResult+Helpers.mm
- Fixed deps in iOS task BUILD file
- Reverted addition of flow limiter calculator in image classifier iOS
- Added delegates in iOS gesture recognizer options
- Added MPPGestureRecognizerOptionsHelpers, MPPGestureRecognizerResultHelpers, MPPGestureRecognizer header
- Updated the vision task runner to split the method that creates normalized rect based on ROI
- Added C++ utils for parsing protos from text files for iOS tests
- Added hand landmarker result, hand landmarker options, hand landmarker options helpers
- Add FaceLandmarkerOptions and FaceLandmarker Result API
- Added utils of containers and core to MPPTaskCommon to avoid warnings in xcode
- Updated error tests to use XCTAssertEqualObjects

Javascript
- Update WASM files for 0.10.1 release

<br/>
<br/>

Framework and Core Calculator Improvements

Bug Fixes and Other Changes
- Added clearing of all graph options protos in MPPGestureRecognizerOptions Helpers, support to set delegates in MPPBaseOptions
- Added method to create unique dispatch queue names in MPPVisionTaskRunner
- Updated MPPImageClassifier to use delegates instead of completion blocks for callback
- Updated documentation
- Updated time out for image classifier async tests
- Updated time out for object detector
- Added flow limiter calculator in MediaPipeTasksCommon
- Added clearing of all graph options protos in MPPGestureRecognizerOptions Helpers
- Update base_options.py
- Update Dockerfile
- Add some helpful error messages in case GL texture creation fails
- Updated Image classifier result to return empty results if packet can't be validated
- Updated MPPObjectDetectorResult Helpers to return empty result instead of nil
- Add needed enum type for choose fuse pipeline
- Added C++ utils for parsing protos from text files for iOS tests
- Updated face detector to use new methods from vision task runner
- Updated variable names in MPPHandLandmarkerOptionsHelpers
- Add MultiLandmarksSmoothingCalculator
- Add MP_DISABLE_GPU to .so target
- Updated CVPixelBuffer to support pixel format type of 32RGBA
- Added support to set delegates in MPPBaseOptions

MediaPipe Dependencies
- Removed opencv dependency from MPPVIsionTaskRunner
- Update MediaPipe to RE2 release `2023-06-01`

Major Features and Improvements
- Released MediaPipe Solutions APIs for Java, Python and Web that offer advanced end-to-end solutions for end-to-end on-device ML.

Bug Fixes and Other Changes
Bazel changes
- Updated bazelrc with required config

Framework and core calculator improvements
- Added Language Detector Python API and fixed a typo in Interactive Segmenter Options' docstring, Update CalculatorOptions to encourage proto3 options
- Updated roi not allowed check in ios vision task runner
- Updated normalized rect calculation for some angles in MPPVisionTaskRunner
- Added shell script for building cocoapods archive
- Added more pose landmarker tests and updated face landmarker tests to cover all the results
- Added Language Detector Python API and fixed a typo in Interactive Segmenter Options' docstring
- Add nullable annotation to AudioDataProducersetAudioConsumer
- Add a default_applicable_licenses to model_maker/python/vision/core
- Added podspec for CommonObjects and Vision tasks
- Add customizable face stylizer module in MediaPipe model maker
- Add custom metadata for object detection model with out-of-graph nms
- Update MPImage to use containers
- Update the face stylizer config to match the latest encoder and detector config
- Add nose in facemesh drawing
- Added config for fat simulator builds
- Added http_archive to download opencv sources
- Added config settings to select building iOS xcframework from source for certain configs
- Updated BUILD files to use the open sourced Language Detector model
- Add the TFLite conversion API to BlazeFaceStylizer in model maker
- Add the "FACE_ALIGNMENT" output stream to the face stylizer graph
- Add an extra op to rescale face stylizer generation output from [-1, 1] to [0, 1]
- Add TransformerParameters proto
- Updated docuemntation of MPPObjectDetector
- Added hash implementation for iOS normalized keypoint
- Updated wait time for object detector tests
- Updated pixel format types in object detector
- Added flow limiter calculator and conditionally selected xcframework in iOS framework targets
- Added conditional building of opencv xc framework to test targets
- Update CalculatorOptions to encourage proto3 options
- Add support for single-channel images to MPImage

MediaPipe solutions update

Android
- Move Java Connections arrays to Task class

iOS
- Updated roi not allowed check in iOS vision task runner
- Removed roi apis from iOS object detector
- Added iOS Object Detector Objective D tests
- Removed detect in image with region of interest api from iOS Object Detector
- Updated iOS tests to reflect the new orientation calculation
- Updated iOS Image Classifier to reflect new calculation for normalized rect
- Updated build rules for iOS frameworks to duplicate symbols
- Updated iOS cocoapods build script
- Updated iOS framework names
- Added build file for iOS opencv from sources
- Updated iOS object detector to use delegates instead of callbacks for async calls
- Added hash implementation for iOS normalized keypoint
- Added flow limiter calculator and conditionally selected xcframework in iOS framework targets
- Updated deps names in iOS test targets
- Added iOS task text cocoapods podspec
- Added targets for iOS text frameworks
- Added method for creating unique dispatch queue names in MPPVisionTaskRunner

Javascript
- Add scribble support to InteractiveSegmenter Web API
- Update WASM files for Alpha 14
- Add .close() method to ImageSegmenterResult/InteractiveSegmenterResult/PoseLandmarkerResult
- Update FaceStylizer, ImageSegmenter, InteractiveSegmenter, PoseLandmarker to return MPImage

Python
- Added the PoseLandmarker Python API and a simple test
- Populate labels using model metadata for the ImageSegmenter Python API
- Added the Face Aligner Python API
- Expose PoseLandmarker as a public MediaPipe Tasks Python API
- Expose FaceAligner and LanguageDetector to be public MediaPipe Tasks Python API
- Add image_segmenter_metadata_schema and object_detector_metadata_schema python files to the mediapipe python wheels
- Add HAND_CONNECTIONS to HandLandmarker and GestureRecognizer


MediaPipe Dependencies
- Aded version of dependency to podspec template
- Updated common dependencies to link in helpers

Bazel changes
- **Bazel version upgrade to v6.1.1**
- Update Halide build rules for MediaPipe to use Halide v15.0.1
- Use "x86_32" instead of "i386" for Bazel CPU ID

Framework and core calculator improvements
- Added MPPImageClassifierOptionsHelpers, TensorsToSegmentationCalculatorOptionsProto.java into tasks core's maven package, MPPObjectDetectorOptions, MPPObjectDetectorOptionsHelpers, MPPClassifierOptions, MPPGestureRecognizerOptions, MPPGestureRecognizerOptions.m, support for more standard scaling options in GlSurfaceViewRenderer
- Updated cosine similarity utility
- Added method to send packet map to C++ task runner
- Added methods to MPPVisionTaskRunner
- Added methods to MPPVisionPacketCreator
- Updated build targets of vision packet creator and task runner
- Added MPPImageClassifierResultHelpers, MPPImageClassifierOptionsHelpers
- Added MPPImageClassifier
- Updated method signature in MPPTaskRunner
- Added Face Detector implementation and tests
- Added the AudioRecord API
- Update audio_record_test.py
- Add FaceLandmarker C++ API
- Updated models
- Add the dataset module for face stylizer in model maker
- Update Node version to 16.19.0
- Add metadata writer for image segmentation
- Add Interactive Segmenter MediaPipe Task
- Add label_map filtering into filter_detection drishti calculator
- Add the source code TensorsToSegmentationCalculatorOptionsProto.java into tasks core's maven package
- Add ImageData output to GraphRunner
- Added MPPImage Utils for tests
- Added stream info for some modes in MPPImageClassifier
- Added flow limiting for live stream mode in MPPImageClassifier
- Add WebGLTexture output for ImageSegmenter
- Add face_landmarker to vision types
- Add a function to convert CoreAudio buffers into a MediaPipe time series matrix
- Add the model configuration and training hyperparameters for BlazeFaceStylizer
- Add landmarks smoothing filter when requested face num is 1
- Added MPPDetection
- Added MPPObjectDetectionResult
- Added MPPObjectDetectorOptions
- Added MPPObjectDetectorOptionsHelpers, MPPObjectDetectionResultHelpers, MPPDetectionHelpers
- Added MPPObjectDetector
- Add FrameBuffer view on ImageFrame
- Add EDGETPU_NNAPI delegate option in MediaPipe tasks API
- Added MPPLandmark
- Added MPPLandmarkHelpers
- Added MPPGestureRecognizerResult
- Added MPPGestureRecognizerOptions, MPPClassifierOptions
- Added EndLoopImageCalculator and FaceToRectCalculator
- Updated FaceStylizer API to align with the new Base Vision Task API changes
- Added some face landmarks constants
- Added pose landmarker C++ API
- Update TF version to 2023-04-12
- Added CoreAudio and MediaToolbox to BUILD file
- Update Flatbuffers to 23.1.21
- Updated error with info about unsupported mirrored orientations in MPPVisionTaskRunner
- Add VEC32F4 support to ImageFrame
- Add shaders that support better landscape rendering with GlSurfaceViewRenderer
- Update TensorsToFaceLandmarksGraph to support face mesh v2 model
- Add support for more standard scaling options in GlSurfaceViewRenderer

MediaPipe solutions update
This section should highlight the changes that are done specifically for any platform and don't propagate to
other platforms.

Android
- Add FaceDetector, Pose Landmarker, FaceLandmarker and FaceStylizer Java API
- Add getLabels to ImageSegmeter Java API
- Fix the vision tasks aar build rule to solve the "cannot find symbol" error:
- Add LabelMapProto.java source code to MediaPipe AAR
- Add interactive segmenter java API
- Add face landmarker and face geometry java lite proto source code into mediapipe tasks AAR
- Switch to use the `isPresent()` API since the `isEmpty()` is only available since java 11: https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/Optional.html#isEmpty()
- Update java image segmenter to always output confidence masks and optionally output category mask
- Adds a LanguageDetector Java API
- Update Java interactive segmenter to output both confidence masks and category mask optionally

iOS
- Changed method Updated method calls to process packet map in iOS text tasks
- Solve iOS build error for gpu_buffer.cc
- Fixed iOS running mode display strings
- Linked in Opencv iOS framework with vision tasks
- Added flow limiter calculator to iOS vision tasks

Javascript
- Add FaceLandmarker Web API
- Add the FaceStylizer Web API
- Add InteractiveSegmenter Web API

Python
- **Python support for M1**
- Added Interactive Segmenter Python API and some tests
- Expose face detector, face landmarker, face stylizer and interactive segmenter as MediaPipe Tasks Python API
- Enable TextClassifier and TextEmbedder on Windows Python
- Gracefully fail resource path lookup for Python on Windows
- Expose as mediapipe python API
- Make AudioTools compile when build from python:framework_bindings

Bug fixes
- Upgrades and fixes for image segmentation category mask on GPU

MediaPipe Dependencies
- Added dependency for image format
- Disable OpenCL dependency for OpenCV
- Add missing dependency library targets to mediapipe_task_aar

Bazel changes
- Add `` to all references to files in `WORKSPACE.bazel`

Framework and core calculator improvements
- Added MPPTextEmbedderOptions, MPPTextEmbedderOptionsHelpers, MPPImageClassifierOptions
- Added volume_gain_db option into AudioToTensorCalculator
- Added MPPEmbedding, MPPEmbeddingResult, MPPTextEmbedderResult
- Added iOS text embedder result files
- Update test to reflect the recommended graph construction style:
- Add FrameBuffer format
- Updated documentation of embedding containers
- Add YuvImage as a GpuBuffer storage backend
- Updated to types of float and quantized embedding
- Add Text Embedder tests for text with different themes
- Added MPPEmbeddingHelpers, MPPEmbeddingResultHelpers, MPPTextEmbedderOptionsHelpers, MPPTextEmbedderResultHelpers, MPPTextEmbedder
- Add "noasan" to MPPTextClassifierObjcTest
- Added MPPCosineSimilarity and cosine similarity to MPPTextEmbedder
- Added text embedder objective c tests
- Add ViewProvider<FrameBuffer> to YuvImage storage backend
- Update MP Tasks to observe timestamp bounds
- Updated swift name for ImageSource Type
- Updated list of designated initializers
- Update TensorFlow to latest
- Add more filtering methods to detection filter calculator
- Update WASM files for 0.1.0-alpha-4 release
- Updated the Begin/EndLoopCalculator to be able to handle mediapipe::Tensor
- Add location info in registry (debug mode only)
- Added vision task runner
- Added designated initializer in vision task runner
- Updated MPPImageUtils with methods to create image frame
- Updated MPPVisionTaskRunner
- Add mediapipe tasks face blendshapes graph
- Add "java_package" and "java_outer_classname" to ImageTransformationCalculatorOptions
- Updated method name in MPPVisionPacketCreator
- Update MediaPipe TFLite code to use generic "shim" symbols and headers
- Update detection result to include optional keypoints
- Update face detector graph for downstream face landmarks graph
- Add Bitmap image capture capability to GlSurfaceViewRenderer
- Update ImageSegmenter API for image/video mode to have both callback API and returned result API
- Small fixes to TensorsToImageCalculator
- Add optional face blendshapes to face landmarks detector graph
- Add a CHECK for the cases when null service is accessed unconditionally
- Add FaceLandmarkerResult for FaceLandmarker API
- Add ViewProvider for ImageFrame in GpuBufferStorageYuvImage
- Add GetInputImageTensorSpecs into BaseVisionTaskApi for tasks api users to get input image tensor specifications
- Add custom metadata in metadata_schema
- Add FaceDetectorResult
- Add volume_gain_db option to TensorsToAudioCalculator
- Add build system for Halide and expose FrameBufferUtils
- Add requiredInputBufferSize as an input argument of createAudioRecord
- Update ImageFrameToGpuBufferCalculator to use api2 and GpuBuffer conversions
- Add Empty Packet support to GraphRunner
- Add support for [xmin, ymin, xmax, ymax] style of bbox output
- Add TensorsToFaceLandmarksGraph to support two types of face mesh models

MediaPipe solutions update
This section should highlight the changes that are done specifically for any platform and don't propagate to
other platforms.

Android
- Remove usage of var for ImageSegmenter.java
- When "--define=MEDIAPIPE_NO_JNI=1" used in compilation, no implementation in libandroid.so is used

iOS
- Added iOS text embedder result files
- Added iOS test for different themes in text embedder
- Added iOS test for quantized embedding
- Added a note about swift test coverage in iOS text embedder tests
- Added MPPTaskImage for iOS vision tasks
- Open visibility of iOS TextClassifier & TextEmbedder
- Solve Linking error for Hello World iOS example
- Added swift tests for text embedder

Javascript
- Fix incorrect uint8 -> int8 conversion in JS cosine similarity
- Add MediaPipe Image Segmenter task for Web

Python
- Enable Python Audio Classifier & Embedder on Windows

Bug fixes
- Bug fixes in MPPImage
- Ssd anchors calculator add fixed anchors

MediaPipe Dependencies
- Bump Halide version from 14.0.0 to 15.0.0 and add MacOS Halide dependency

Build changes
- Allow split_vector_calculator to be build with iOS and MEDIAPIPE_DISABLE_GPU
- Update mediapipe_aar.bzl to put more mediapipe framework java proto classes into AARs.

Bazel changes
Update Bazel dependencies for Apple

Framework and core calculator improvements
- Add HandLandmarkerGraph which connect HandDetectorGraph and HandLandmarkerSubgraph with landmarks tracking.
- Updated image classifier to use a region of interest parameter
- Add support for input image rotation in ImageClassifier and ObjectDetector C++ API
- Adding BypassCalculator for use with SwitchContainer.
- Add MergeDetectionsToVectorCalculator, CombinedPredictionCalculator, EndLoopMatrixCalculator, ConcatenateClassificationListCalculator, RegexPreprocessingCalculator and BERTPreprocessorCalculator, TextToTensorCalculator and UniversalSentenceEncoderPreprocessorCalculator
- Added the TextClassifier C++ API, the TextPreprocessingSubgraph.
- Rename "Bound" struct to "Rect" and remove unused "Landmark" struct.
- Add tensor_index and tensor_name fields to ClassificationList
- Replace numpy.float with the builtin float type as numpy removes its own float type in v1.24.
- Add BGR -> RGB color conversion to ColorConvertCalculator.
- Add SQRT_HANN window type to both SpectrogramCalculator and InverseSpectrogramCalculator.
- Allow conversion of GlTextureBuffer to CVPixelBufferRef. This means that, if an iOS application sends in a GlTextureBuffer but expects a CVPixelBufferRef as output, everything will work even if the graph just forwards the same input. Also, access by Metal calculators will also work transparently.
- Allowing BypassCalculator to accept InputSidePackets.
- Enable unsigned quantized infererence using XNNPACK.
- Adds a preprocessor for Universal Sentence Encoder models.

MediaPipe solutions update
Android
- Enable creating MediaPipe Image c++ packet directly from an Android media image object when its format is RGBA_8888.
- Add Java ImageEmbedder API and TextEmbedder API.
- Fix aar breakage caused by missing "//mediapipe/tasks/java/com/google/mediapipe/tasks/components/containers:normalized_landmark".
- Fix aar breakage caused by missing "//mediapipe/tasks/cc/vision/image_segmenter/proto:segmenter_options_java_proto_lite".

Web
- Hand Landmarker Web API
- Allow Web developers to opt into CPU or GPU processing
- Add support for browsers without SIMD
- Add pre-compiled WASM files to NPM packages

Bug fixes
- Fix RGBA vs RGB selection when creating GLTexture.
- Fix accidental suppressions of GLSL linker error reporting
- Fix for CHECK failure due to pointer description sometimes being larger than allocated string space
- ClassificationAggregationCalculator and EmbeddingAggregationCalculator now fill in the `timestamp_ms` field of the classification results in the stream mode.
- Fix ObjectDetector C++ flow limiter and improve documentation.
- Better handling of empty packets in vector calculators.

MediaPipe Dependencies
- Bump up the dependency library pybind11's version to 2.10.1.

Build changes
- We are no longer adding *.tflite model files and other large binaries to our GitHub repository. Instead, these models are downloaded from Google Cloud Storage. This should speed up your getting started experience with MediaPipe (especially if you can work of a shallow clone of the repository) and allows us to expand our feature set without significantly increasing the size of the repository. Please update your Python binaries if they are fetching models from GitHub (see [download_utils.py](https://github.com/google/mediapipe/blob/d3f98334bfe533a57ec2c2fa85294c21e25fe0af/mediapipe/python/solutions/download_utils.py)).
- We have made the build targets `//mediapipe/objc:mediapipe_framework_ios`, `//mediapipe/objc:mediapipe_input_sources_ios`, `//mediapipe/objc:mediapipe_layer_renderer` publicly visible. These targets can now be used in external iOS applications.