Projectaria-tools

🎉 We’re excited to announce Project Aria Tools v1.5.2. 🎉

**[API]**
- VrsDataProvider
- Add support of VRS metadata/file tags and time sync mode
- Calibration
- Add better support for [rotating calibration data](https://github.com/facebookresearch/projectaria_tools/commit/616766ba0a86c1bc31341019d97987f67411bb18) (intrinsics, extrinsics)
- See usage either on [wiki](https://facebookresearch.github.io/projectaria_tools/docs/data_utilities/advanced_code_snippets/image_utilities#rotated-image-clockwise-90-degrees) or in the [AEA](https://github.com/facebookresearch/projectaria_tools/pull/90) dataset viewer
Expose [camera calibration rescaling function](https://github.com/facebookresearch/projectaria_tools/commit/8692a230b76539dc0e3583a2ced1f65f41c34e8d)

**[Code Samples]**
- C++
- [VRS Image mutation](https://github.com/facebookresearch/projectaria_tools/commit/6133b26f9c1f0ada4e33bbb32956b9621c475823)
- Python
- TicSync notebook [tutorial](https://github.com/facebookresearch/projectaria_tools/blob/b4599e1b69b17629e2c0c7e1ae74c94493710f78/core/examples/ticsync_tutorial.ipynb)
- MPS [EyeGaze Depth Estimation](https://github.com/facebookresearch/projectaria_tools/commit/ff0ec97b17e830617f417f14f263f31b84311e3b)
- ADT - [Notebook on converting depth maps to point clouds](https://github.com/facebookresearch/projectaria_tools/commit/093bd1c85e605ba292f0a93384054a7c52641ecf)

**[Tools - Python]**
- [VRS to MP4](https://facebookresearch.github.io/projectaria_tools/docs/data_utilities/advanced_code_snippets/vrs_to_mp4)
- Improved timing accuracy during export
- Updated to include timestamps in the metadata

**[MPS-CLI]**
- API update to mps-cli to support Aria Studio

**[Build]**
- Continuous Integration (adapt to changing software/hardware environment)
- C++ - [Fix for Pangolin](https://github.com/facebookresearch/projectaria_tools/commit/63c42728af405e01c697b6d4fe373592f1b5928f)
- Python wheels - [Fix cibuildwheel version](https://github.com/facebookresearch/projectaria_tools/commit/f8481609a4d4622962d00149dd9104f09db5bf89)
- Python wheels - Fix GitHub runner to be Intel or Mac Silicon

**[Visualization]**
- MPS Viewer - [Enable showing RGB images upright, undistorted, or unchanged](https://github.com/facebookresearch/projectaria_tools/commit/daa89dba94df36b2c83f3117f6d12045a4d58d37)

**[Projects/Dataset]**
- ADT
- Data & Tooling Updated to v2.0
- We have removed the notion of subsequences. All subsequence are now separated into their own sequence folder
- Tooling including visualization and downloaders have been updated to new file structure, but are still backwards compatible
- We have removed access to challenge data now that the challenge has completed
- Visualization
- Python - Add option to add image rectification to the viewer
- C++ - Update Pangolin viewer to better aspect ratio
- AEA
- Visualization - Python - [Ease image rotation/undistortion](https://github.com/facebookresearch/projectaria_tools/pull/90)

**[Documentation]**
- Various improvements, including:
- New [Pairing Additional Glasses and Troubleshooting](https://facebookresearch.github.io/projectaria_tools/docs/ARK/glasses_manual/pair_glasses)
- New [Project Aria Glasses Cable Clip instructions](https://facebookresearch.github.io/projectaria_tools/docs/ARK/glasses_manual/cable_clip)
- Updated Aria Digital Twin (ADT) v2.0 updated to support [Aria Dataset Explorer](https://explorer.projectaria.com/)
- Updated Info page and About ARK page

🎉 **We’re excited to announce Project Aria Tools v1.5.0.** 🎉

**[Tools - Python]**
- ASR code sample shows how to use [Faster Whisper](https://github.com/SYSTRAN/faster-whisper) to run Whisper Speech Recognition on an Aria audio stream. The ASR outputs are time aligned with [Aria Device Time](https://facebookresearch.github.io/projectaria_tools/docs/data_formats/aria_vrs/timestamps_in_aria_vrs#vrs-timestamps-single-device). Go to the [Automated Speech Recognition Readme](https://github.com/facebookresearch/projectaria_tools/blob/main/tools/samples/python/automated_speech_recognition/Readme.md) for how to get started.

**[MPS CLI]**
- Support for new MPS service, [hand tracking](https://facebookresearch.github.io/projectaria_tools/docs/ARK/sw_release_notes#new-hand-tracking-mps-wrist-and-palm-tracking)
- Updated Eye Tracking to support [depth estimation](https://facebookresearch.github.io/projectaria_tools/docs/ARK/sw_release_notes#update-to-eye-gaze-outputs---depth-estimations)

**[Documentation]**
- Various improvements, including:
- New [MPS Troubleshooting](https://facebookresearch.github.io/projectaria_tools/docs/ARK/mps/mps_troubleshooting) page
- New [Collaborative Tools Page](https://facebookresearch.github.io/projectaria_tools/docs/collaborative_tools), did you know you can use Aria data with Nerfstudio?
- New [Hand Tracking Data Formats](https://facebookresearch.github.io/projectaria_tools/docs/data_formats/mps/hand_tracking)
- Updated [MPS Google Colab tutorials](https://facebookresearch.github.io/projectaria_tools/docs/data_utilities/getting_started#running--jupyter-notebooks-on-google-colab)

🎉 **We are excited to announce the release of our new 1.4.0 version!** 🎉

- MPS CLI ([ARK](https://facebookresearch.github.io/projectaria_tools/docs/ARK/about_ARK)|Project Aria Research partners)
- New (and recommended) workflow for requesting Machine Perception services.
- MPS Multi-SLAM can only be requested by the MPS CLI.
- MPS Multi-SLAM ([ARK](https://facebookresearch.github.io/projectaria_tools/docs/ARK/about_ARK)|Project Aria Research partners)
- Compute SLAM MPS outputs in a shared coordinate frame for multiple VRS files

Here is the complete changelog:

**[API]**
- DataProvider - [{Feature} - Make DeliverQueue start and end timestamp logic more robust](https://github.com/facebookresearch/projectaria_tools/commit/599409591b434416c5d532a44b3efdb75abb79a5)
- Calibration - [{Feature} calibration - expose imu rectification matrix and bias vector](https://github.com/facebookresearch/projectaria_tools/commit/c735661436c0c6bc0b05ccd46902f56611413dca)
- Viewer MPS - [{Visualizer} Python - viewer_mps - Add mps_folder option](https://github.com/facebookresearch/projectaria_tools/commit/900bd35366786309734ce6e5836bfacfd306dc4f)

**[MPS CLI]**
- [aria_mps](https://facebookresearch.github.io/projectaria_tools/docs/ARK/mps/request_mps/mps_cli) - Addition of a Python CLI for
- Project Aria Research partners to manage their Machine Perception Services requests (run and monitor upload, run and results retrieval). Improvements on the Desktop app include:
- Auto run health checks prior to upload
- Recordings will not be uploaded if they are not valid for any of the MPS services requested
- Resumable uploads
- Concurrent processing
- Automatically downloads outputs once processing is complete
- Recordings are processed once
- Uploaded data is stored for 24 hours
- Additional MPS can be requested without needing to upload again
- Data can be reprocessed without needing to upload again
- CLI can be integrated into automated workflows
- Request Multi-Recording outputs
- Compute SLAM MPS outputs in a shared coordinate frame for multiple VRS files
- [Projectaria_tools](https://facebookresearch.github.io/projectaria_tools/docs/data_utilities/installation/installation_python) must be installed using pip to access the CLI

**[Documentation]**
- [aria_mps](https://facebookresearch.github.io/projectaria_tools/docs/ARK/mps/request_mps/mps_cli) CLI documentation
- [MPS Data Formats documentation](https://facebookresearch.github.io/projectaria_tools/docs/data_formats/mps/mps_summary) refactored to align with MPS CLI output structure
Minor improvements to other documentation

🎉 **We are excited to announce the release of version 1.3.3** 🎉
This release includes several new features and improvements, including:

- Features
* A MpsDataPathsProvider & MPSDataProvider API
- Dataset support
* Aria Everyday Activities (AEA) Dataset support
- **NEW** - Code Sample
- Demo to run [EfficientSAM with eye gaze prompt](https://github.com/facebookresearch/projectaria_tools/commit/019deb241e1a3c89c6f0f8f073397cef80d5480c)
- CI/CD/Build
- Improved our PyPI Python wheel generation workflow

Here is the complete changelog:

**[Visualization]**
- [viewer_mps](https://github.com/facebookresearch/projectaria_tools/commit/856ecad9733fe64b9c8f9b239661752bfe9a2c57) update to load multi sequence datasets (AEA, ADT)
- [viewer_projects_aea](https://facebookresearch.github.io/projectaria_tools/docs/open_datasets/aria_everyday_activities_dataset/aea_visualizers#python-visualizer)

**[API]**
- MPS
- Add [MpsDataPathsProvider](https://github.com/facebookresearch/projectaria_tools/blob/5bd15f6c7869ab9e020ee5f0e5ac3d247e754a4a/core/python/test/mpsPyBindTest.py#L148) & [MPSDataProvider](https://github.com/facebookresearch/projectaria_tools/blob/5bd15f6c7869ab9e020ee5f0e5ac3d247e754a4a/core/python/test/mpsPyBindTest.py#L142) API
Allowing to easily retrieve MPS file assets and query them their metadata by timestamp

**[Dataset]**
[AEA Aria EveryDay Activities](https://facebookresearch.github.io/projectaria_tools/docs/open_datasets/aria_everyday_activities_dataset) dataset
[ADT Aria Digital Twin](https://facebookresearch.github.io/projectaria_tools/docs/open_datasets/aria_digital_twin_dataset/data_format) dataset is updated with MPS data

**[Build/CI/CD]**
- [{Build} Improve FMT compatibility (](https://github.com/facebookresearch/projectaria_tools/commit/205e36ee10f3f9e8350a20566434ea2968e5b7ec)[#54](https://github.com/facebookresearch/projectaria_tools/pull/54)[)](https://github.com/facebookresearch/projectaria_tools/commit/205e36ee10f3f9e8350a20566434ea2968e5b7ec)
- [{Build} GitHub CI - Python Wheels generation](https://github.com/facebookresearch/projectaria_tools/commit/8c924babd5060c4111178d08a3e1f263ba7917f5)
- [{Build} Fix python wheel build workflow for PyPI](https://github.com/facebookresearch/projectaria_tools/commit/454ffc2d345782dc95971efcd3b7e13e63f60238)

**[Documentation]**
- [AEA documentation](https://facebookresearch.github.io/projectaria_tools/docs/open_datasets/aria_everyday_activities_dataset)
Minor improvements to other documentation

[Thank you to our new contributors]
selcuk-meta
eric-fb

🎉 **We are excited to announce the release of version 1.3.0** 🎉

This release includes several major new features and improvements, including:
- New python visualization samples
- A VRS_to_MP4 tool to help quickly review (RGB + Sound) for data collection
- New C++ and Python tutorials on MPS point cloud colorization & how to use ADT depth map to generate point cloud
…

Here is the complete changelog:

**[Visualization]**
- [Python] New [rerun](https://facebookresearch.github.io/projectaria_tools/docs/data_utilities/visualization/visualization_python) visualization samples to help debug and visualize Aria and MPS temporal data & states
- [viewer_aria_sensors](https://facebookresearch.github.io/projectaria_tools/docs/data_utilities/visualization/visualization_python#run-aria-sensor-viewer)
- [viewer_mps](https://facebookresearch.github.io/projectaria_tools/docs/data_utilities/visualization/visualization_python#run-mps-viewer)
- [viewer_projects_adt](https://facebookresearch.github.io/projectaria_tools/docs/open_datasets/aria_digital_twin_dataset/visualizers#python-visualizer)
- [viewer_projects_ase](https://facebookresearch.github.io/projectaria_tools/docs/open_datasets/aria_synthetic_environments_dataset/ase_data_tools#python-sample)

**[API]**
- [Python] Introduce an `projectaria_tools.mps.utils` module to help query and filter loaded MPS data

import projectaria_tools.mps.utils

Retrieve Pose/Eye Gaze data by timestamp
get_nearest_eye_gaze, get_nearest_pose

Reproject eyegaze vector in image
get_gaze_vector_reprojection

Filter Point Cloud data
filter_points_from_confidence
filter_points_from_count


- [Python - C++] Image undistortion update `distort_by_calibration`
- API update to perform bilinear or nearest neighbor [multithread](https://github.com/facebookresearch/projectaria_tools/commit/01dff693b8feec3074906336b05b5de0684389c2) interpolation to better select the right interpolation for depth (bilinear) or segmentation mask (nearest) -> see (`distort_by_calibration`, `distort_depth_by_calibration` & `distort_label_by_calibration`)
- [Python - C++] Calibration rotation
- Ease accessibility to upright pinhole calibration data for RGB/SLAM images [rotate_camera_calib_cw90deg](https://facebookresearch.github.io/projectaria_tools/docs/data_utilities/advanced_code_snippets/image_utilities#rotated-image-clockwise-90-degrees)

**[Tools]**
- [Python] Tool to create an MP4 file from VRS RGB and audio data ([code](https://github.com/facebookresearch/projectaria_tools/commit/89478f68af7dce922427647e325023b6e6b6c0d5), [documentation](https://facebookresearch.github.io/projectaria_tools/docs/data_utilities/advanced_code_snippets/vrs_to_mp4))
- [C++] [PointCloud Colorization Sample](https://github.com/facebookresearch/projectaria_tools/tree/main/tools/samples/pointcloud_colorization)
- [C++] Aria Viewer - Enable plot buffer for AriaViewer to Avoids slow down/ speed up when audio is enabled/disabled
- [C++/Python] Vrs health check
add health check for VRS streams (audio, barometer, bluetooth, camera, gps, imu, wifi) -> Reads all records from all streams and check the health of each record in each stream
- [Python] Projects/Aria Digital Twin - [Add notebook tutorial to create and merge point clouds from depth maps data](https://github.com/facebookresearch/projectaria_tools/commit/305c218fd4d5223dfe23d75351172c6f31772225)


**[Continuous integration - GitHub]**
- Various cleanup in GitHub actions
- Improved CI code coverage by adding [testing python notebooks](https://github.com/facebookresearch/projectaria_tools/commit/33d45024bb7dfae3456fa0ac71895f451754b7b8)

**[BugFix]**
- [Core] fix support of multiple gps streams (coming from Aria and cell phone)

**[Known Issues]**

- [Machine Perception Services (MPS) outputs have been renamed](https://facebookresearch.github.io/projectaria_tools/docs/ARK/sw_release_notes#122023_features), so that they more clearly communicate what is in the outputs:

SLAM/Trajectory
- global_points.csv.gz -> semidense_points.csv.gz
Eye Gaze
- generalized_eye_gaze.csv -> general_eye_gaze.csv
- calibrated_eye_gaze.csv -> personalized_eye_gaze.csv


**[Documentation]**
- Aria Digital Twin, new [landing](https://facebookresearch.github.io/projectaria_tools/docs/open_datasets/aria_digital_twin_dataset) and [challenge page](https://facebookresearch.github.io/projectaria_tools/docs/open_datasets/aria_digital_twin_dataset/adt_challenges)
- Aria Synthetic Environments, new [landing](https://facebookresearch.github.io/projectaria_tools/docs/open_datasets/aria_synthetic_environments_dataset) and [challenge page](https://facebookresearch.github.io/projectaria_tools/docs/open_datasets/aria_synthetic_environments_dataset/ase_challenges)
- Data Formats - [how Project Aria uses VRS](https://facebookresearch.github.io/projectaria_tools/docs/data_formats/aria_vrs)
- Data Utilities - refactored our visualization guide to have a[Python](https://facebookresearch.github.io/projectaria_tools/docs/data_utilities/visualization/visualization_python) and[ a C++](https://facebookresearch.github.io/projectaria_tools/docs/data_utilities/visualization/visualization_cpp) page
- Refactored downloading MPS sample data into a single [download](https://facebookresearch.github.io/projectaria_tools/docs/data_utilities/installation/download_mps_sample_data) page


[Thank you to our new contributors]
baderouaich