Highlights
We are excited to announce the release of MMAction2 1.0.0 as a part of the OpenMMLab 2.0 project! MMAction2 1.0.0 introduces an updated framework structure for the core package and a new section called `Projects`. This section showcases various engaging and versatile applications built upon the MMAction2 foundation.
In this latest release, we have significantly refactored the core package's code to make it clearer, more comprehensible, and disentangled. This has resulted in improved performance for several existing algorithms, ensuring that they now outperform their previous versions. Additionally, we have incorporated some cutting-edge algorithms, such as VideoSwin and VideoMAE, to further enhance the capabilities of MMAction2 and provide users with a more comprehensive and powerful toolkit. The new `Projects` section serves as an essential addition to MMAction2, created to foster innovation and collaboration among users. This section offers the following attractive features:
- **`Flexible code contribution`**: Unlike the core package, the `Projects` section allows for a more flexible environment for code contributions, enabling faster integration of state-of-the-art models and features.
- **`Showcase of diverse applications`**: Explore various projects built upon the MMAction2 foundation, such as deployment examples and combinations of video recognition with other tasks.
- **`Fostering creativity and collaboration`**: Encourages users to experiment, build upon the MMAction2 platform, and share their innovative applications and techniques, creating an active community of developers and researchers. Discover the possibilities within the "Projects" section and join the vibrant MMAction2 community in pushing the boundaries of video understanding applications!
Exciting Features
RGBPoseConv3D
RGBPoseConv3D is a framework that jointly uses 2D human skeletons and RGB appearance for human action recognition. It is a 3D CNN with two streams, with the architecture borrowed from SlowFast. In [RGBPoseConv3D](https://github.com/open-mmlab/mmaction2/tree/main/configs/skeleton/posec3d/rgbpose_conv3d#readme):
- The RGB stream corresponds to the `slow` stream in SlowFast; The Skeleton stream corresponds to the `fast` stream in SlowFast.
- The input resolution of RGB frames is `4x` larger than the pseudo heatmaps.
- Bilateral connections are used for early feature fusion between the two modalities.
<div align=center>
<img src="https://user-images.githubusercontent.com/34324155/209961351-6def0074-9b05-43fc-8210-a1cdaaed6536.png" width=50%/>
</div>
- Supported by Dai-Wenxun in https://github.com/open-mmlab/mmaction2/pull/2182
Inferencer
In this release, we introduce the MMAction2Inferencer, which is a versatile API for the inference that supports multiple input types. The API enables users to easily specify and customize action recognition models, streamlining the process of performing video prediction using MMAction2.
Usage:
shell
python demo/demo_inferencer.py ${INPUTS} [OPTIONS]
- The `INPUTS` can be a video path or rawframes folder. For more detailed information on `OPTIONS`, please refer to [Inferencer](https://github.com/open-mmlab/mmaction2/tree/main/demo#inferencer).
Example:
shell
python demo/demo_inferencer.py zelda.mp4 --rec tsn --vid-out-dir zelda_out --label-file tools/data/kinetics/label_map_k400.txt
You can find the `zelda.mp4` [here](https://user-images.githubusercontent.com/58767402/232312124-4d237e57-7671-4d86-9e50-f588de007377.mp4). The output video is displayed below:
https://user-images.githubusercontent.com/58767402/232312742-f5eb2e8c-f015-459c-8a4d-99c331a65735.mp4
- Supported by cir7 in https://github.com/open-mmlab/mmaction2/pull/2164
List of Novel Features
MMAction2 V1.0 introduces support for new models and datasets in the field of video understanding, including [MSG3D [Project]](https://github.com/open-mmlab/mmaction2/tree/main/projects/msg3d#readme) (CVPR'2020), [CTRGCN [Project]](https://github.com/open-mmlab/mmaction2/tree/main/projects/ctrgcn#readme) (CVPR'2021), [STGCN++](https://github.com/open-mmlab/mmaction2/tree/main/configs/skeleton/stgcnpp#readme) (Arxiv'2022), [Video Swin Transformer](https://github.com/open-mmlab/mmaction2/tree/main/configs/recognition/swin#readme) (CVPR'2022), [VideoMAE](https://github.com/open-mmlab/mmaction2/tree/main/configs/recognition/videomae#readme) (NeurIPS'2022), [C2D](https://github.com/open-mmlab/mmaction2/tree/main/configs/recognition/c2d#readme) (CVPR'2018), [MViT V2](https://github.com/open-mmlab/mmaction2/tree/main/configs/recognition/mvit#readme) (CVPR'2022), [UniFormer V1](https://github.com/open-mmlab/mmaction2/tree/main/configs/recognition/uniformer#readme) (ICLR'2022), and [UniFormer V2](https://github.com/open-mmlab/mmaction2/tree/main/configs/recognition/uniformerv2#readme) (Arxiv'2022), as well as the spatiotemporal action detection dataset [AVA-Kinetics](https://github.com/open-mmlab/mmaction2/tree/main/tools/data/ava_kinetics#readme) (Arxiv'2022).
- **[Enhanced Omni-Source](https://github.com/open-mmlab/mmaction2/tree/main/configs/recognition/omnisource#readme):** We enhanced the original [omni-source](https://arxiv.org/abs/2003.13042) technique by dynamically adjusting 3D convolutional network architecture to simultaneously utilize videos and images for training. Taking the `SlowOnlyR50 8x8` as an example, the Top-1 accuracy comparison of the three training methods illustrates that our omni-source training effectively employs the additional `ImageNet` dataset, significantly boosting performance on `Kinetics400`.
<div align=center>
<img src="https://user-images.githubusercontent.com/58767402/233539833-98c8d731-3b9f-4a5e-ad4c-dddf6e1afd68.png" width=50%/>
</div>
- **Mulit-Stream Skeleton Pipeline:** In light of MMAction2's prior support for only `joint` and `bone` modalities, we have extended support to `joint motion` and `bone motion` modalities in MMAction2 V1.0. Furthermore, we have conducted training and evaluation for these four modalities using **NTU60 2D and 3D** keypoint data on [STGCN](https://github.com/open-mmlab/mmaction2/tree/main/configs/skeleton/stgcn#readme), [2s-AGCN](https://github.com/open-mmlab/mmaction2/tree/main/configs/skeleton/2s-agcn#readme), and [STGCN++](https://github.com/open-mmlab/mmaction2/tree/main/configs/skeleton/stgcnpp#readme).
<div align=center>
<img src="https://user-images.githubusercontent.com/58767402/233548094-3ed8b98f-55a0-477c-801f-0b119648416b.png" width=50%/>
</div>
- **Repeat Augment** was initially proposed as a data augmentation method for `ImageNet` training and has been employed in recent Video Transformer works. **Whenever a video is read during training, we use multiple (typically 2-4) random samples from the video for training.** This approach not only enhances the model's generalization capability but also reduces the IO pressure of video reading. We support Repeat Augment in MMAction2 V1.0 and utilize this technique in [MViT V2](https://github.com/open-mmlab/mmaction2/tree/main/configs/recognition/mvit#readme) training. The table below compares the Top-1 accuracy on `Kinetics400` before and after employing Repeat Augment:
<div align=center>
<img src="https://user-images.githubusercontent.com/58767402/233551634-4022b21a-3978-48c5-8276-18cdcaf64879.png" width=50%/>
</div>
Bug Fixes
- [Fix] Fix flip config of TSM for sth2sth v1/v2 dataset by cir7 in https://github.com/open-mmlab/mmaction2/pull/2247
- [Fix] Fix circle ci by cir7 in https://github.com/open-mmlab/mmaction2/pull/2336 and https://github.com/open-mmlab/mmaction2/pull/2334
- [Fix] Fix accepting an unexpected argument local-rank in PyTorch 2.0 by cir7 in https://github.com/open-mmlab/mmaction2/pull/2320
- [Fix] Fix TSM config link by zyx-cv in https://github.com/open-mmlab/mmaction2/pull/2315
- [Fix] Fix numpy version requirement in CI by hukkai in https://github.com/open-mmlab/mmaction2/pull/2284
- [Fix] Fix NTU pose extraction script by cir7 in https://github.com/open-mmlab/mmaction2/pull/2246
- [Fix] Fix TSM-MobileNet V2 by cir7 in https://github.com/open-mmlab/mmaction2/pull/2332
- [Fix] Fix command bugs in localization tasks' README by hukkai in https://github.com/open-mmlab/mmaction2/pull/2244
- [Fix] Fix duplicate name in DecordInit and SampleAVAFrame by cir7 in https://github.com/open-mmlab/mmaction2/pull/2251
- [Fix] Fix channel order when showing video by cir7 in https://github.com/open-mmlab/mmaction2/pull/2308
- [Fix] Specify map_location to cpu when using _load_checkpoint by Zheng-LinXiao in https://github.com/open-mmlab/mmaction2/pull/2252
New Contributors
* Andy1621 made their first contribution in https://github.com/open-mmlab/mmaction2/pull/2153
* zoe08 made their first contribution in https://github.com/open-mmlab/mmaction2/pull/2188
* vansin made their first contribution in https://github.com/open-mmlab/mmaction2/pull/2228
* Zheng-LinXiao made their first contribution in https://github.com/open-mmlab/mmaction2/pull/2252
**Full Changelog**: https://github.com/open-mmlab/mmaction2/compare/v0.24.0...v1.0.0