Datumaro

New features
- Ability to import a video as frames with the `video_frames` format and
to split a video into frames with the `datum util split_video` command
(<https://github.com/openvinotoolkit/datumaro/pull/555>)
- `--subset` parameter in the `image_dir` format
(<https://github.com/openvinotoolkit/datumaro/pull/555>)
- `MediaManager` API to control loaded media resources at runtime
(<https://github.com/openvinotoolkit/datumaro/pull/555>)
- Command to detect the format of a dataset
(<https://github.com/openvinotoolkit/datumaro/pull/576>)
- More comfortable access to library API via `import datumaro`
(<https://github.com/openvinotoolkit/datumaro/pull/630>)
- CLI command-like free functions (`export`, `transform`, ...)
(<https://github.com/openvinotoolkit/datumaro/pull/630>)
- Reading specific annotation files for train dataset in Cityscapes
(<https://github.com/openvinotoolkit/datumaro/pull/632>)
- Random sampling transforms (`random_sampler`, `label_random_sampler`)
to create smaller datasets from bigger ones
(<https://github.com/openvinotoolkit/datumaro/pull/636>,
<https://github.com/openvinotoolkit/datumaro/pull/640>)
- API to report dataset import and export progress;
API to report dataset import and export errors and take action (skip, fail)
(supported in COCO, VOC and YOLO formats)
(<https://github.com/openvinotoolkit/datumaro/pull/650>)
- Support for downloading the ImageNetV2 and COCO datasets
(<https://github.com/openvinotoolkit/datumaro/pull/653>,
<https://github.com/openvinotoolkit/datumaro/pull/659>)
- A way for formats to signal that they don't support detection
(<https://github.com/openvinotoolkit/datumaro/pull/665>)
- Removal transforms to remove items/annoations/attributes from dataset
(`remove_items`, `remove_annotations`, `remove_attributes`)
(<https://github.com/openvinotoolkit/datumaro/pull/670>)

Enhancements
- Allowed direct file paths in `datum import`. Such sources are imported like
when the `rpath` parameter is specified, however, only the selected path
is copied into the project
(<https://github.com/openvinotoolkit/datumaro/pull/555>)
- Improved `stats` performance, added new filtering parameters,
image stats (`unique`, `repeated`) moved to the `dataset` section,
removed `mean` and `std` from the `dataset` section
(<https://github.com/openvinotoolkit/datumaro/pull/621>)
- Allowed `Image` creation from just `size` info
(<https://github.com/openvinotoolkit/datumaro/pull/634>)
- Added image search in VOC XML-based subformats
(<https://github.com/openvinotoolkit/datumaro/pull/634>)
- Added image path equality checks in simple merge, when applicable
(<https://github.com/openvinotoolkit/datumaro/pull/634>)
- Supported saving box attributes when downloading the TFDS version of VOC
(<https://github.com/openvinotoolkit/datumaro/pull/668>)
- Switched to a `pyproject.toml`-based build
(<https://github.com/openvinotoolkit/datumaro/pull/671>)

Deprecated
- TBD

Removed
- Official support of Python 3.6 (due to it's EOL)
(<https://github.com/openvinotoolkit/datumaro/pull/617>)
- Backward compatibility annotation symbols in `components.extractor`
(<https://github.com/openvinotoolkit/datumaro/pull/630>)

Bug fixes
- Prohibited calling `add`, `import` and `export` commands without a project
(<https://github.com/openvinotoolkit/datumaro/pull/555>)
- Calling `make_dataset` on empty project tree now produces the error properly
(<https://github.com/openvinotoolkit/datumaro/pull/555>)
- Saving (overwriting) a dataset in a project when rpath is used
(<https://github.com/openvinotoolkit/datumaro/pull/613>)
- Output image extension preserving in the `Resize` transform
(<https://github.com/openvinotoolkit/datumaro/issues/606>)
- Memory overuse in the `Resize` transform
(<https://github.com/openvinotoolkit/datumaro/issues/607>)
- Invalid image pixels produced by the `Resize` transform
(<https://github.com/openvinotoolkit/datumaro/issues/618>)
- Numeric warnings that sometimes occurred in `stats` command
(e.g. <https://github.com/openvinotoolkit/datumaro/issues/607>)
(<https://github.com/openvinotoolkit/datumaro/pull/621>)
- Added missing item attribute merging in simple merge
(<https://github.com/openvinotoolkit/datumaro/pull/634>)
- Inability to disambiguate VOC from LabelMe in some cases
(<https://github.com/openvinotoolkit/datumaro/issues/658>)

Security
- TBD

New features
- Command to download public datasets
(<https://github.com/openvinotoolkit/datumaro/pull/582>)
- Extension autodetection in `ByteImage`
(<https://github.com/openvinotoolkit/datumaro/pull/595>)
- MPII Human Pose Dataset (import-only) (.mat and .json)
(<https://github.com/openvinotoolkit/datumaro/pull/584>)
- MARS format (import-only)
(<https://github.com/openvinotoolkit/datumaro/pull/585>)

Enhancements
- The `pycocotools` dependency lower bound is raised to `2.0.4`.
(<https://github.com/openvinotoolkit/datumaro/pull/449>)
- `smooth_line` from `datumaro.util.annotation_util` - the function
is renamed to `approximate_line` and has updated interface
(<https://github.com/openvinotoolkit/datumaro/pull/592>)

Deprecated
- Python 3.6 support

Removed
- TBD

Bug fixes
- Fails in multimerge when lines are not approximated and when there are no
label categories (<https://github.com/openvinotoolkit/datumaro/pull/592>)
- Cannot convert LabelMe dataset, that has no subsets
(<https://github.com/openvinotoolkit/datumaro/pull/600>)

Security
- TBD

New features
- Video reading API
(<https://github.com/openvinotoolkit/datumaro/pull/521>)
- Python API documentation
(<https://github.com/openvinotoolkit/datumaro/pull/526>)
- Mapillary Vistas dataset format (Import-only)
(<https://github.com/openvinotoolkit/datumaro/pull/537>)
- Datumaro can now be installed on Windows on Python 3.9
(<https://github.com/openvinotoolkit/datumaro/pull/547>)
- Import for SYNTHIA dataset format
(<https://github.com/openvinotoolkit/datumaro/pull/532>)
- Support of `score` attribute in KITTI detetion
(<https://github.com/openvinotoolkit/datumaro/pull/571>)
- Support for Accuracy Checker dataset meta files in formats
(<https://github.com/openvinotoolkit/datumaro/pull/553>,
<https://github.com/openvinotoolkit/datumaro/pull/569>,
<https://github.com/openvinotoolkit/datumaro/pull/575>)
- Import for VoTT dataset format
(<https://github.com/openvinotoolkit/datumaro/pull/573>)
- Image resizing transform
(<https://github.com/openvinotoolkit/datumaro/pull/581>)

Enhancements
- The following formats can now be detected unambiguously:
`ade20k2017`, `ade20k2020`, `camvid`, `coco`, `cvat`, `datumaro`,
`icdar_text_localization`, `icdar_text_segmentation`,
`icdar_word_recognition`, `imagenet_txt`, `kitti_raw`, `label_me`, `lfw`,
`mot_seq`, `open_images`, `vgg_face2`, `voc`, `widerface`, `yolo`
(<https://github.com/openvinotoolkit/datumaro/pull/531>,
<https://github.com/openvinotoolkit/datumaro/pull/536>,
<https://github.com/openvinotoolkit/datumaro/pull/550>,
<https://github.com/openvinotoolkit/datumaro/pull/557>,
<https://github.com/openvinotoolkit/datumaro/pull/558>)
- Allowed Pytest-native tests
(<https://github.com/openvinotoolkit/datumaro/pull/563>)
- Allowed export options in the `datum merge` command
(<https://github.com/openvinotoolkit/datumaro/pull/545>)

Deprecated
- Using `Image`, `ByteImage` from `datumaro.util.image` - these classes
are moved to `datumaro.components.media`
(<https://github.com/openvinotoolkit/datumaro/pull/538>)

Removed
- Equality comparison support between `datumaro.components.media.Image`
and `numpy.ndarray`
(<https://github.com/openvinotoolkit/datumaro/pull/568>)

Bug fixes
- Bug 560: import issue with MOT dataset when using seqinfo.ini file
(<https://github.com/openvinotoolkit/datumaro/pull/564>)
- Empty lines in VOC subset lists are not ignored
(<https://github.com/openvinotoolkit/datumaro/pull/587>)

Security
- TBD

New features
- Import for CelebA dataset format.
(<https://github.com/openvinotoolkit/datumaro/pull/484>)

Enhancements
- File `people.txt` became optional in LFW
(<https://github.com/openvinotoolkit/datumaro/pull/509>)
- File `image_ids_and_rotation.csv` became optional Open Images
(<https://github.com/openvinotoolkit/datumaro/pull/509>)
- Allowed underscores (`_`) in subset names in COCO
(<https://github.com/openvinotoolkit/datumaro/pull/509>)
- Allowed annotation files with arbitrary names in COCO
(<https://github.com/openvinotoolkit/datumaro/pull/509>)
- The `icdar_text_localization` format is no longer detected in every directory
(<https://github.com/openvinotoolkit/datumaro/pull/531>)
- Updated `pycocotools` version to 2.0.2
(<https://github.com/openvinotoolkit/datumaro/pull/534>)

Deprecated
- TBD

Removed
- TBD

Bug fixes
- Unhandled exception when a file is specified as the source for a COCO or
MOTS dataset
(<https://github.com/openvinotoolkit/datumaro/pull/530>)
- Exporting dataset without `color` attribute into the
`icdar_text_segmentation` format
(<https://github.com/openvinotoolkit/datumaro/pull/556>)
Security
- TBD

New features
- A new installation target: `pip install datumaro[default]`, which should
be used by default. The simple `datumaro` is supposed for library users.
(<https://github.com/openvinotoolkit/datumaro/pull/238>)
- Dataset and project versioning capabilities (Git-like)
(<https://github.com/openvinotoolkit/datumaro/pull/238>)
- "dataset revpath" concept in CLI, allowing to pass a dataset path with
the dataset format in `diff`, `merge`, `explain` and `info` CLI commands
(<https://github.com/openvinotoolkit/datumaro/pull/238>)
- `import`, `remove`, `commit`, `checkout`, `log`, `status`, `info` CLI commands
(<https://github.com/openvinotoolkit/datumaro/pull/238>)
- `Coco*Extractor` classes now have an option to preserve label IDs from the
original annotation file
(<https://github.com/openvinotoolkit/datumaro/pull/453>)
- `patch` CLI command to patch datasets
(<https://github.com/openvinotoolkit/datumaro/pull/401>)
- `ProjectLabels` transform to change dataset labels for merging etc.
(<https://github.com/openvinotoolkit/datumaro/pull/401>,
<https://github.com/openvinotoolkit/datumaro/pull/478>)
- Support for custom labels in the KITTI detection format
(<https://github.com/openvinotoolkit/datumaro/pull/481>)
- Type annotations and docs for Annotation classes
(<https://github.com/openvinotoolkit/datumaro/pull/493>)
- Options to control label loading behavior in `imagenet_txt` import
(<https://github.com/openvinotoolkit/datumaro/pull/434>,
<https://github.com/openvinotoolkit/datumaro/pull/489>)

Enhancements
- A project can contain and manage multiple datasets instead of a single one.
CLI operations can be applied to the whole project, or to separate datasets.
Datasets are modified inplace, by default
(<https://github.com/openvinotoolkit/datumaro/issues/328>)
- CLI help for builtin plugins doesn't require project
(<https://github.com/openvinotoolkit/datumaro/issues/328>)
- Annotation-related classes were moved into a new module,
`datumaro.components.annotation`
(<https://github.com/openvinotoolkit/datumaro/pull/439>)
- Rollback utilities replaced with Scope utilities
(<https://github.com/openvinotoolkit/datumaro/pull/444>)
- The `Project` class from `datumaro.components` is changed completely
(<https://github.com/openvinotoolkit/datumaro/pull/238>)
- `diff` and `ediff` are joined into a single `diff` CLI command
(<https://github.com/openvinotoolkit/datumaro/pull/238>)
- Projects use new file layout, incompatible with old projects.
An old project can be updated with `datum project migrate`
(<https://github.com/openvinotoolkit/datumaro/pull/238>)
- Inheriting `CliPlugin` is not required in plugin classes
(<https://github.com/openvinotoolkit/datumaro/pull/238>)
- `Importer`s do not create `Project`s anymore and just return a list of
extractor configurations
(<https://github.com/openvinotoolkit/datumaro/pull/238>)

Deprecated
- TBD

Removed
- `import`, `project merge` CLI commands
(<https://github.com/openvinotoolkit/datumaro/pull/238>)
- Support for project hierarchies. A project cannot be a source anymore
(<https://github.com/openvinotoolkit/datumaro/pull/238>)
- Project cannot have independent internal dataset anymore. All the project
data must be stored in the project data sources
(<https://github.com/openvinotoolkit/datumaro/pull/238>)
- `datumaro_project` format
(<https://github.com/openvinotoolkit/datumaro/pull/238>)
- Unused `path` field of `DatasetItem`
(<https://github.com/openvinotoolkit/datumaro/pull/455>)

Bug fixes
- Deprecation warning in `open_images_format.py`
(<https://github.com/openvinotoolkit/datumaro/pull/440>)
- `lazy_image` returning unrelated data sometimes
(<https://github.com/openvinotoolkit/datumaro/issues/409>)
- Invalid call to `pycocotools.mask.iou`
(<https://github.com/openvinotoolkit/datumaro/pull/450>)
- Importing of Open Images datasets without image data
(<https://github.com/openvinotoolkit/datumaro/pull/463>)
- Return value type in `Dataset.is_modified`
(<https://github.com/openvinotoolkit/datumaro/pull/401>)
- Remapping of secondary categories in `RemapLabels`
(<https://github.com/openvinotoolkit/datumaro/pull/401>)
- VOC dataset patching for classification and segmentation tasks
(<https://github.com/openvinotoolkit/datumaro/pull/478>)
- Exported mask label ids in KITTI segmentation
(<https://github.com/openvinotoolkit/datumaro/pull/481>)
- Missing `label` for `Points` read in the LFW format
(<https://github.com/openvinotoolkit/datumaro/pull/494>)

Security
- TBD

New features
- The Open Images format now supports bounding box
and segmentation mask annotations
(<https://github.com/openvinotoolkit/datumaro/pull/352>,
<https://github.com/openvinotoolkit/datumaro/pull/388>).
- Bounding boxes values decrement transform (<https://github.com/openvinotoolkit/datumaro/pull/366>)
- Improved error reporting in `Dataset` (<https://github.com/openvinotoolkit/datumaro/pull/386>)
- Support ADE20K format (import only) (<https://github.com/openvinotoolkit/datumaro/pull/400>)
- Documentation website at <https://openvinotoolkit.github.io/datumaro> (<https://github.com/openvinotoolkit/datumaro/pull/420>)

Enhancements
- Datumaro no longer depends on scikit-image
(<https://github.com/openvinotoolkit/datumaro/pull/379>)
- `Dataset` remembers export options on saving / exporting for the first time (<https://github.com/openvinotoolkit/datumaro/pull/386>)

Deprecated
- TBD

Removed
- TBD

Bug fixes
- Application of `remap_labels` to dataset categories of different length (<https://github.com/openvinotoolkit/datumaro/issues/314>)
- Patching of datasets in formats (<https://github.com/openvinotoolkit/datumaro/issues/348>)
- Improved Cityscapes export performance (<https://github.com/openvinotoolkit/datumaro/pull/367>)
- Incorrect format of `*_labelIds.png` in Cityscapes export (<https://github.com/openvinotoolkit/datumaro/issues/325>, <https://github.com/openvinotoolkit/datumaro/issues/342>)
- Item id in ImageNet format (<https://github.com/openvinotoolkit/datumaro/pull/371>)
- Double quotes for ICDAR Word Recognition (<https://github.com/openvinotoolkit/datumaro/pull/375>)
- Wrong display of builtin formats in CLI (<https://github.com/openvinotoolkit/datumaro/issues/332>)
- Non utf-8 encoding of annotation files in Market-1501 export (<https://github.com/openvinotoolkit/datumaro/pull/392>)
- Import of ICDAR, PASCAL VOC and VGGFace2 images from subdirectories on WIndows
(<https://github.com/openvinotoolkit/datumaro/pull/392>)
- Saving of images with Unicode paths on Windows (<https://github.com/openvinotoolkit/datumaro/pull/392>)
- Calling `ProjectDataset.transform()` with a string argument (<https://github.com/openvinotoolkit/datumaro/issues/402>)
- Attributes casting for CVAT format (<https://github.com/openvinotoolkit/datumaro/pull/403>)
- Loading of custom project plugins (<https://github.com/openvinotoolkit/datumaro/issues/404>)
- Reading, writing anno file and saving name of the subset for test subset
(<https://github.com/openvinotoolkit/datumaro/pull/447>)

Security
- Fixed unsafe unpickling in CIFAR import (<https://github.com/openvinotoolkit/datumaro/pull/362>)