Projectaria-tools

🎉 **We are excited to announce the release of our new 1.4.0 version!** 🎉

- MPS CLI ([ARK](https://facebookresearch.github.io/projectaria_tools/docs/ARK/about_ARK)|Project Aria Research partners)
- New (and recommended) workflow for requesting Machine Perception services.
- MPS Multi-SLAM can only be requested by the MPS CLI.
- MPS Multi-SLAM ([ARK](https://facebookresearch.github.io/projectaria_tools/docs/ARK/about_ARK)|Project Aria Research partners)
- Compute SLAM MPS outputs in a shared coordinate frame for multiple VRS files

Here is the complete changelog:

**[API]**
- DataProvider - [{Feature} - Make DeliverQueue start and end timestamp logic more robust](https://github.com/facebookresearch/projectaria_tools/commit/599409591b434416c5d532a44b3efdb75abb79a5)
- Calibration - [{Feature} calibration - expose imu rectification matrix and bias vector](https://github.com/facebookresearch/projectaria_tools/commit/c735661436c0c6bc0b05ccd46902f56611413dca)
- Viewer MPS - [{Visualizer} Python - viewer_mps - Add mps_folder option](https://github.com/facebookresearch/projectaria_tools/commit/900bd35366786309734ce6e5836bfacfd306dc4f)

**[MPS CLI]**
- [aria_mps](https://facebookresearch.github.io/projectaria_tools/docs/ARK/mps/request_mps/mps_cli) - Addition of a Python CLI for
- Project Aria Research partners to manage their Machine Perception Services requests (run and monitor upload, run and results retrieval). Improvements on the Desktop app include:
- Auto run health checks prior to upload
- Recordings will not be uploaded if they are not valid for any of the MPS services requested
- Resumable uploads
- Concurrent processing
- Automatically downloads outputs once processing is complete
- Recordings are processed once
- Uploaded data is stored for 24 hours
- Additional MPS can be requested without needing to upload again
- Data can be reprocessed without needing to upload again
- CLI can be integrated into automated workflows
- Request Multi-Recording outputs
- Compute SLAM MPS outputs in a shared coordinate frame for multiple VRS files
- [Projectaria_tools](https://facebookresearch.github.io/projectaria_tools/docs/data_utilities/installation/installation_python) must be installed using pip to access the CLI

**[Documentation]**
- [aria_mps](https://facebookresearch.github.io/projectaria_tools/docs/ARK/mps/request_mps/mps_cli) CLI documentation
- [MPS Data Formats documentation](https://facebookresearch.github.io/projectaria_tools/docs/data_formats/mps/mps_summary) refactored to align with MPS CLI output structure
Minor improvements to other documentation

🎉 **We are excited to announce the release of version 1.3.3** 🎉
This release includes several new features and improvements, including:

- Features
* A MpsDataPathsProvider & MPSDataProvider API
- Dataset support
* Aria Everyday Activities (AEA) Dataset support
- **NEW** - Code Sample
- Demo to run [EfficientSAM with eye gaze prompt](https://github.com/facebookresearch/projectaria_tools/commit/019deb241e1a3c89c6f0f8f073397cef80d5480c)
- CI/CD/Build
- Improved our PyPI Python wheel generation workflow

Here is the complete changelog:

**[Visualization]**
- [viewer_mps](https://github.com/facebookresearch/projectaria_tools/commit/856ecad9733fe64b9c8f9b239661752bfe9a2c57) update to load multi sequence datasets (AEA, ADT)
- [viewer_projects_aea](https://facebookresearch.github.io/projectaria_tools/docs/open_datasets/aria_everyday_activities_dataset/aea_visualizers#python-visualizer)

**[API]**
- MPS
- Add [MpsDataPathsProvider](https://github.com/facebookresearch/projectaria_tools/blob/5bd15f6c7869ab9e020ee5f0e5ac3d247e754a4a/core/python/test/mpsPyBindTest.py#L148) & [MPSDataProvider](https://github.com/facebookresearch/projectaria_tools/blob/5bd15f6c7869ab9e020ee5f0e5ac3d247e754a4a/core/python/test/mpsPyBindTest.py#L142) API
Allowing to easily retrieve MPS file assets and query them their metadata by timestamp

**[Dataset]**
[AEA Aria EveryDay Activities](https://facebookresearch.github.io/projectaria_tools/docs/open_datasets/aria_everyday_activities_dataset) dataset
[ADT Aria Digital Twin](https://facebookresearch.github.io/projectaria_tools/docs/open_datasets/aria_digital_twin_dataset/data_format) dataset is updated with MPS data

**[Build/CI/CD]**
- [{Build} Improve FMT compatibility (](https://github.com/facebookresearch/projectaria_tools/commit/205e36ee10f3f9e8350a20566434ea2968e5b7ec)[#54](https://github.com/facebookresearch/projectaria_tools/pull/54)[)](https://github.com/facebookresearch/projectaria_tools/commit/205e36ee10f3f9e8350a20566434ea2968e5b7ec)
- [{Build} GitHub CI - Python Wheels generation](https://github.com/facebookresearch/projectaria_tools/commit/8c924babd5060c4111178d08a3e1f263ba7917f5)
- [{Build} Fix python wheel build workflow for PyPI](https://github.com/facebookresearch/projectaria_tools/commit/454ffc2d345782dc95971efcd3b7e13e63f60238)

**[Documentation]**
- [AEA documentation](https://facebookresearch.github.io/projectaria_tools/docs/open_datasets/aria_everyday_activities_dataset)
Minor improvements to other documentation

[Thank you to our new contributors]
selcuk-meta
eric-fb

🎉 **We are excited to announce the release of version 1.3.0** 🎉

This release includes several major new features and improvements, including:
- New python visualization samples
- A VRS_to_MP4 tool to help quickly review (RGB + Sound) for data collection
- New C++ and Python tutorials on MPS point cloud colorization & how to use ADT depth map to generate point cloud
…

Here is the complete changelog:

**[Visualization]**
- [Python] New [rerun](https://facebookresearch.github.io/projectaria_tools/docs/data_utilities/visualization/visualization_python) visualization samples to help debug and visualize Aria and MPS temporal data & states
- [viewer_aria_sensors](https://facebookresearch.github.io/projectaria_tools/docs/data_utilities/visualization/visualization_python#run-aria-sensor-viewer)
- [viewer_mps](https://facebookresearch.github.io/projectaria_tools/docs/data_utilities/visualization/visualization_python#run-mps-viewer)
- [viewer_projects_adt](https://facebookresearch.github.io/projectaria_tools/docs/open_datasets/aria_digital_twin_dataset/visualizers#python-visualizer)
- [viewer_projects_ase](https://facebookresearch.github.io/projectaria_tools/docs/open_datasets/aria_synthetic_environments_dataset/ase_data_tools#python-sample)

**[API]**
- [Python] Introduce an `projectaria_tools.mps.utils` module to help query and filter loaded MPS data

import projectaria_tools.mps.utils

Retrieve Pose/Eye Gaze data by timestamp
get_nearest_eye_gaze, get_nearest_pose

Reproject eyegaze vector in image
get_gaze_vector_reprojection

Filter Point Cloud data
filter_points_from_confidence
filter_points_from_count


- [Python - C++] Image undistortion update `distort_by_calibration`
- API update to perform bilinear or nearest neighbor [multithread](https://github.com/facebookresearch/projectaria_tools/commit/01dff693b8feec3074906336b05b5de0684389c2) interpolation to better select the right interpolation for depth (bilinear) or segmentation mask (nearest) -> see (`distort_by_calibration`, `distort_depth_by_calibration` & `distort_label_by_calibration`)
- [Python - C++] Calibration rotation
- Ease accessibility to upright pinhole calibration data for RGB/SLAM images [rotate_camera_calib_cw90deg](https://facebookresearch.github.io/projectaria_tools/docs/data_utilities/advanced_code_snippets/image_utilities#rotated-image-clockwise-90-degrees)

**[Tools]**
- [Python] Tool to create an MP4 file from VRS RGB and audio data ([code](https://github.com/facebookresearch/projectaria_tools/commit/89478f68af7dce922427647e325023b6e6b6c0d5), [documentation](https://facebookresearch.github.io/projectaria_tools/docs/data_utilities/advanced_code_snippets/vrs_to_mp4))
- [C++] [PointCloud Colorization Sample](https://github.com/facebookresearch/projectaria_tools/tree/main/tools/samples/pointcloud_colorization)
- [C++] Aria Viewer - Enable plot buffer for AriaViewer to Avoids slow down/ speed up when audio is enabled/disabled
- [C++/Python] Vrs health check
add health check for VRS streams (audio, barometer, bluetooth, camera, gps, imu, wifi) -> Reads all records from all streams and check the health of each record in each stream
- [Python] Projects/Aria Digital Twin - [Add notebook tutorial to create and merge point clouds from depth maps data](https://github.com/facebookresearch/projectaria_tools/commit/305c218fd4d5223dfe23d75351172c6f31772225)


**[Continuous integration - GitHub]**
- Various cleanup in GitHub actions
- Improved CI code coverage by adding [testing python notebooks](https://github.com/facebookresearch/projectaria_tools/commit/33d45024bb7dfae3456fa0ac71895f451754b7b8)

**[BugFix]**
- [Core] fix support of multiple gps streams (coming from Aria and cell phone)

**[Known Issues]**

- [Machine Perception Services (MPS) outputs have been renamed](https://facebookresearch.github.io/projectaria_tools/docs/ARK/sw_release_notes#122023_features), so that they more clearly communicate what is in the outputs:

SLAM/Trajectory
- global_points.csv.gz -> semidense_points.csv.gz
Eye Gaze
- generalized_eye_gaze.csv -> general_eye_gaze.csv
- calibrated_eye_gaze.csv -> personalized_eye_gaze.csv


**[Documentation]**
- Aria Digital Twin, new [landing](https://facebookresearch.github.io/projectaria_tools/docs/open_datasets/aria_digital_twin_dataset) and [challenge page](https://facebookresearch.github.io/projectaria_tools/docs/open_datasets/aria_digital_twin_dataset/adt_challenges)
- Aria Synthetic Environments, new [landing](https://facebookresearch.github.io/projectaria_tools/docs/open_datasets/aria_synthetic_environments_dataset) and [challenge page](https://facebookresearch.github.io/projectaria_tools/docs/open_datasets/aria_synthetic_environments_dataset/ase_challenges)
- Data Formats - [how Project Aria uses VRS](https://facebookresearch.github.io/projectaria_tools/docs/data_formats/aria_vrs)
- Data Utilities - refactored our visualization guide to have a[Python](https://facebookresearch.github.io/projectaria_tools/docs/data_utilities/visualization/visualization_python) and[ a C++](https://facebookresearch.github.io/projectaria_tools/docs/data_utilities/visualization/visualization_cpp) page
- Refactored downloading MPS sample data into a single [download](https://facebookresearch.github.io/projectaria_tools/docs/data_utilities/installation/download_mps_sample_data) page


[Thank you to our new contributors]
baderouaich

**[Features]**
- **[Core - Python]**
- Sophus python binding
- Add [SO3, SE3 interface](https://facebookresearch.github.io/projectaria_tools/docs/data_utilities/core_code_snippets/calibration#python-binding-for-sophus-library) in python based on Sophus library. Example code is provided in [sophus_quickstart_tutorial notebook](https://github.com/facebookresearch/projectaria_tools/blob/main/core/examples/sophus_quickstart_tutorial.ipynb)

- [Python type annotation](https://facebookresearch.github.io/projectaria_tools/docs/data_utilities/installation/type_hinting)
- Python type hinting/ stubs are automatically generated as part of the pypi package when installing projectaria_tools with pip install. Users can also generate them on their own using the `generate_stubs.py` script.

- [Google Colab runnable notebooks](https://facebookresearch.github.io/projectaria_tools/docs/data_utilities/getting_started#running-the-jupyter-notebooks-on-google-colab)
- Python notebooks can now be run in Google Colab -> [Dataprovider Quickstart Tutorial](https://colab.research.google.com/github/facebookresearch/projectaria_tools/blob/main/core/examples/dataprovider_quickstart_tutorial.ipynb) | [Machine Perception Services Tutorial](https://colab.research.google.com/github/facebookresearch/projectaria_tools/blob/main/core/examples/mps_quickstart_tutorial.ipynb)
- No installation on local machine required to test and play with projectaria_tools

- **[Core]**
- Add `cameraId` to `ImageDataRecord`
- Allow the `ImageDataRecord` to list from which camera the data came from

- Continuous integration
- GitHub Actions runs Python Unit test

- Dependencies
- Update to use VRS v1.1.0
- Remove cereal dependency and use directly rapidjson

- **[MPS]**
- Calibrated and generalized EyeGaze
- Support of calibrated eye gaze via [in-session calibration](https://facebookresearch.github.io/projectaria_tools/docs/ARK/mps/eye_gaze_calibration)
- Support for multiple wearers in a single Aria capture. The eye gaze output will contain a `session_uid` field that will help distinguish between different wearers.

- Python type format
- `print(X)` will now display object content

- **[Tools]**
- [MPS Replay Viewer {C++}](https://facebookresearch.github.io/projectaria_tools/docs/data_utilities/visualization_guide#mps-3d-replay-visualizer)
- Renders static scene and dynamic elements: 2D/3D observations rays + eye gaze data

**[BugFix]**
- [Core]
- [{bug fix} update crop and rescale to SensorCalibration](https://github.com/facebookresearch/projectaria_tools/commit/feaa489303ddf7fee3e74ed1fdb3001df142f541)
- Update the API to make calibration data to match from the sensor and device access point: `get_sensor_calibration(stream_id).camera_calibration()` and `provider.get_device_calibration().get_camera_calib(name)` to match.

**[Known Issues]**
- [Core]
- The Sophus API has been updated, if you encounter issues, please update to v1.2 of Project Aria Tools
- Here is how to update your existing code following the API change for SO3/SE3:
- `.matrix()` to `.to_matrix()`
- `.quaternion()` -> `.rotation().to_quat()[0]` or `to_quat_and_translation()[0]`

**[Documentation]**
- [Core]
- [VRS to MP4](https://facebookresearch.github.io/projectaria_tools/docs/data_utilities/advanced_code_snippets/vrs_to_mp4) Tutorial showing how to export VRS RGB images to a MP4 video.
- Additional information added to [3D Coordinate Frame Conventions](https://facebookresearch.github.io/projectaria_tools/docs/data_formats/coordinate_convention/3d_coordinate_frame_convention)
- [MPS]
- [Eye Gaze Data Formats](https://facebookresearch.github.io/projectaria_tools/docs/data_formats/mps/mps_eye_gaze) updated to include `calibrated_eye_gaze.csv` and `summary.json`
- [Eye Gaze Calibration](https://facebookresearch.github.io/projectaria_tools/docs/ARK/mps/eye_gaze_calibration)

**[Thank you to our new contributors]**
brentyi
[Seanwarren-meta](https://github.com/facebookresearch/projectaria_tools/commits?author=seanwarren-meta)
Selcuk Karakas
Przemyslaw Szczepanski
[Guru Somasundaram](https://github.com/facebookresearch/projectaria_tools/commits?author=gsomas)

**Full Changelog**: https://github.com/facebookresearch/projectaria_tools/compare/1.1.0...1.2.0

- The ADT library has been updated to support dataset versioning.
- Data schema update
- Fix quaternion order in ‘aria_trajectory.csv’
Corrected to `qw, qx, qy, qz` from `qx, qy, qz, qw`
- Fix gravity field names are now called `gravity_x/y/z_world` to align with MPS layout
- Change `SkeletonMetaData.json` to `skeleton_aria_association.json` to better reflect the file content
- Change `gt-metadata.json` to `metadata.json`
- Users are STRONGLY ADVISED to pull from the release branch and [follow ADT download instructions](https://facebookresearch.github.io/projectaria_tools/docs/open_datasets/aria_digital_twin_dataset/dataset_download) to update their ADT datasets to v1.1.
- [ASE]
- Released a more accurate set of camera FishEye model calibration parameter

- [Documentation]
- Minor updates

[BugFix]
- [Core]
- AriaViewer (reset line plots when new timestamp is requested)
- [ADT]