Lhotse

What's Changed

Features

* Support for OPUS encoding in Lhotse Shar format by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1238
* Perform CutSet.mix() lazily by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1244
* `CutSampler.map()` for transforming `CutSet` mini-batches by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1246
* Support multiplexing with a limited number of open streams by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1248

Recipes

* support icmc eval track 1 by yuekaizhang in https://github.com/lhotse-speech/lhotse/pull/1235
* updating the voxpopuli recipe by vesis84 in https://github.com/lhotse-speech/lhotse/pull/1243
* Allowing downloading Edin. ver. of VCTK by JinZr in https://github.com/lhotse-speech/lhotse/pull/1247

Other improvements
* Micro-optimization for LazyJsonlIterator len() by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1237
* Drop python3.7 support by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1245
* Fix `normalize_loudness` for MixedCuts with PaddingCuts by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1249


**Full Changelog**: https://github.com/lhotse-speech/lhotse/compare/v1.18...v1.19

What's Changed

New features

* MMS forced alignment backend by flyingleafe in https://github.com/lhotse-speech/lhotse/pull/1185
* Two new options: `CutSet.from_shar(seed="trng")` and `DynamicCutSampler(quadratic_duration=...)` by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1199
* Faster initialization option in `DynamicBucketingSampler` + various fixes by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1210
* CLI to estimate and print bucket bins for a cut set by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1214
* More flexible setting of audio backends by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1219

Recipes

* Add recipe for Medical Corpus by yfyeung in https://github.com/lhotse-speech/lhotse/pull/1212
* minor fix for the AMI recipe by JinZr in https://github.com/lhotse-speech/lhotse/pull/1178
* fixes compatibility with Edin. ver. VCTK dataset by JinZr in https://github.com/lhotse-speech/lhotse/pull/1182
* Minor bug fix for eval2000 recipe by JinZr in https://github.com/lhotse-speech/lhotse/pull/1127
* support far field data for icmcasr challenge by yuekaizhang in https://github.com/lhotse-speech/lhotse/pull/1189
* fixed text norm for `tal_csasr` by JinZr in https://github.com/lhotse-speech/lhotse/pull/1198 https://github.com/lhotse-speech/lhotse/pull/1213

Other improvements

* `MixedCut.truncate`: fix the case when only `PaddingCut`s are left by flyingleafe in https://github.com/lhotse-speech/lhotse/pull/1157
* Fix some potential problems in OPUS file reading by yangb05 in https://github.com/lhotse-speech/lhotse/pull/1181
* fix an issue where 404 exception leaves 0 byte placeholder by JinZr in https://github.com/lhotse-speech/lhotse/pull/1190
* Prevent accidental renaming when using with_suffix by chiiyeh in https://github.com/lhotse-speech/lhotse/pull/1192
* Fix shar export for `num_jobs>1` and recordings with transforms by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1196
* fix speaker error by yzmyyff in https://github.com/lhotse-speech/lhotse/pull/1197
* Fix for `trim_to_alignments` issue by desh2608 in https://github.com/lhotse-speech/lhotse/pull/1193
* Add `deterministic_rng` to more flaky tests by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1200
* update_recipes by vesis84 in https://github.com/lhotse-speech/lhotse/pull/1208
* SpeechSynthesisDataset returns `speaker_ids` by JinZr in https://github.com/lhotse-speech/lhotse/pull/1206
* Fix audio backend selection by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1216
* save sdm files into a single mdm file to do gss by yuekaizhang in https://github.com/lhotse-speech/lhotse/pull/1221
* Modify SpeechSynthesisDataset class, make it return text by yaozengwei in https://github.com/lhotse-speech/lhotse/pull/1205
* Allow lhotse installation without torchaudio for a limited set of features by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1231
* Use `attacut` module for Thai word tokenization (in MMS forced alignment) by flyingleafe in https://github.com/lhotse-speech/lhotse/pull/1232

New Contributors
* yangb05 made their first contribution in https://github.com/lhotse-speech/lhotse/pull/1181
* chiiyeh made their first contribution in https://github.com/lhotse-speech/lhotse/pull/1192
* yzmyyff made their first contribution in https://github.com/lhotse-speech/lhotse/pull/1197
* yaozengwei made their first contribution in https://github.com/lhotse-speech/lhotse/pull/1205

**Full Changelog**: https://github.com/lhotse-speech/lhotse/compare/v1.17...v1.18

What's Changed

New supported datasets

* Speech to text translation utilizing 3-way data by AmirHussein96 in https://github.com/lhotse-speech/lhotse/pull/1099
* "This American Life" dataset recipe by flyingleafe in https://github.com/lhotse-speech/lhotse/pull/1140
* Add VoxConverse recipe by flyingleafe in https://github.com/lhotse-speech/lhotse/pull/1142
* Add recipe for ICASSP2024 ICMC-ASR Grand Challenge by yfyeung in https://github.com/lhotse-speech/lhotse/pull/1172

New features

* Initial support for video by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1151
* `copy_data`: copy `CutSet` + its data to a new location by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1130
* Add whisper feature extractor by yuekaizhang in https://github.com/lhotse-speech/lhotse/pull/1159
* VAD workflow with Silero by rilshok in https://github.com/lhotse-speech/lhotse/pull/1160

Enhancements and fixes
* Fix feature extraction for lhotse shar CLI by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1123
* Add m4a to special cases for num samples determination by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1124
* making the kaldi import more robust by vesis84 in https://github.com/lhotse-speech/lhotse/pull/1129
* Tutorial materials in main readme page by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1133
* optimize save_audios() by vesis84 in https://github.com/lhotse-speech/lhotse/pull/1131
* Fix bugs in `resumable_download` by flyingleafe in https://github.com/lhotse-speech/lhotse/pull/1135
* Arxiv badge by desh2608 in https://github.com/lhotse-speech/lhotse/pull/1136
* Fix docs build by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1137
* Fix failing tests after repairing docs build by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1138
* Remove deprecated code, make minor cleanups by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1139
* Enforce deterministic RNG behavior in repeatedly flaky tests by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1143
* Refactor `audio.py` into smaller modules by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1144
* Fix broken `save_audio` by flyingleafe in https://github.com/lhotse-speech/lhotse/pull/1147
* Optimize `cut_into_windows` for long cuts by flyingleafe in https://github.com/lhotse-speech/lhotse/pull/1150
* Fixes for 1152 1153 and 1154 by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1156
* fix bugs in downloading voxpopuli corpus by DongjiGao in https://github.com/lhotse-speech/lhotse/pull/1165
* Support `export_to_kaldi` on resampled recordings by sih4sing5hong5 in https://github.com/lhotse-speech/lhotse/pull/1162
* Refactor `CutSet.describe` to enable parallel statistics computation by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1168
* Allow dashes in feat CLI by desh2608 in https://github.com/lhotse-speech/lhotse/pull/1169
* Apply deterministic RNG to more unit tests by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1173
* Add `fix_manifests` in all recipes by desh2608 in https://github.com/lhotse-speech/lhotse/pull/1128
* Fix small bug in eval2000 by desh2608 in https://github.com/lhotse-speech/lhotse/pull/1126
* Fix download in LibriCSS recipe by desh2608 in https://github.com/lhotse-speech/lhotse/pull/1148

New Contributors
* sih4sing5hong5 made their first contribution in https://github.com/lhotse-speech/lhotse/pull/1162
* rilshok made their first contribution in https://github.com/lhotse-speech/lhotse/pull/1160

**Full Changelog**: https://github.com/lhotse-speech/lhotse/compare/v1.16...v1.17

What's Changed

Recipes
New:
* Add speech translation corpus MuST-C by csukuangfj in https://github.com/lhotse-speech/lhotse/pull/1079
* Extend LibriTTS recipe to support LibriTTS-R by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1082
* SURT dataset by desh2608 in https://github.com/lhotse-speech/lhotse/pull/951
* [Recipe] VoxPopuli by desh2608 in https://github.com/lhotse-speech/lhotse/pull/1089
* Air Traffic Control (ATC) corpora - various improvements 2 by rouseabout in https://github.com/lhotse-speech/lhotse/pull/1090
* Add Bengali.AI Speech corpus for Kaggle Research Code Competition by yfyeung in https://github.com/lhotse-speech/lhotse/pull/1108
* Support AudioMNIST by csukuangfj in https://github.com/lhotse-speech/lhotse/pull/1093

Improvements:
* Add multithread to peoples_speech by yfyeung in https://github.com/lhotse-speech/lhotse/pull/1078
* Add multiprocess mechanism for Common Voice by yfyeung in https://github.com/lhotse-speech/lhotse/pull/1025
* text normalization for aishell4 by JinZr in https://github.com/lhotse-speech/lhotse/pull/1103
* updated text norm for aishell2 by JinZr in https://github.com/lhotse-speech/lhotse/pull/1104
* updated text norm for magicdata by JinZr in https://github.com/lhotse-speech/lhotse/pull/1105
* PR for the KeSpeech recipe by JinZr in https://github.com/lhotse-speech/lhotse/pull/1106
* Small changes in some existing recipes by desh2608 in https://github.com/lhotse-speech/lhotse/pull/1110

New features
* `CutSet.from_files` constructor for random order multi-file cutsets by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1085
* Infinite random-file random-line stateless sampler by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1102
* `StatelessSampler`: remove TRNG, make `base_seed` a mandatory parameter by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1109

Other improvements
* Loudness fix by desh2608 in https://github.com/lhotse-speech/lhotse/pull/1087
* Update SpecAug custom state_dict to be compatible with PyTorch by osadj in https://github.com/lhotse-speech/lhotse/pull/1091
* Adding log spectrogram by Tomiinek in https://github.com/lhotse-speech/lhotse/pull/1094
* add user defined kaldi feature type by ZihanLiao in https://github.com/lhotse-speech/lhotse/pull/1101
* Update cuts.rst by taras-sereda in https://github.com/lhotse-speech/lhotse/pull/1112
* Add alignment scores from CTM by desh2608 in https://github.com/lhotse-speech/lhotse/pull/1116
* Fix and enhance TIMIT recipe by flyingleafe in https://github.com/lhotse-speech/lhotse/pull/1072
* Fixing shar reader assert by Tomiinek in https://github.com/lhotse-speech/lhotse/pull/1117
* Support `del (cut|supervision).custom_field` by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1113
* Exposing `tolerance` for matching supervisions with features in Kaldi imported data dirs by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1119
* Some additional options for supervision-related methods by desh2608 in https://github.com/lhotse-speech/lhotse/pull/1115

New Contributors
* osadj made their first contribution in https://github.com/lhotse-speech/lhotse/pull/1091
* ZihanLiao made their first contribution in https://github.com/lhotse-speech/lhotse/pull/1101
* taras-sereda made their first contribution in https://github.com/lhotse-speech/lhotse/pull/1112

**Full Changelog**: https://github.com/lhotse-speech/lhotse/compare/v1.15...v1.16

What's Changed
* Resumable dataset downloads by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1045
* AMI beamformed mic option by desh2608 in https://github.com/lhotse-speech/lhotse/pull/1048
* Add options to prepare data according to CHiME-7 by desh2608 in https://github.com/lhotse-speech/lhotse/pull/1051
* Support Shar export of multi-channel, multi-source recording and cuts with start>0 by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1053
* Add `mono_downmix` option for `MultiCut.to_mono()` by desh2608 in https://github.com/lhotse-speech/lhotse/pull/1052
* Support audio duration mismatch tolerance in `MixedCut.load_audio()` by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1054
* AudioCache: caching for "command" type of audio files by vesis84 in https://github.com/lhotse-speech/lhotse/pull/1050
* Small changes in some cut methods by desh2608 in https://github.com/lhotse-speech/lhotse/pull/1059
* Enhancements and bug fixes for AMI and ICSI by desh2608 in https://github.com/lhotse-speech/lhotse/pull/1058
* Add GigaST corpus by yfyeung in https://github.com/lhotse-speech/lhotse/pull/1062
* Air Traffic Control (ATC) corpora by rouseabout in https://github.com/lhotse-speech/lhotse/pull/1061
* Fix `resumable_download` for fully downloaded files by flyingleafe in https://github.com/lhotse-speech/lhotse/pull/1060
* Fix for audio loading optimization to return the expected number of samples by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1071
* Support preparing almost all NSC data part except PART3_SameBoundaryMic by trunglebka in https://github.com/lhotse-speech/lhotse/pull/1066
* Fix bugs in `MixedCut` logic by flyingleafe in https://github.com/lhotse-speech/lhotse/pull/1073
* Air Traffic Control (ATC) corpora - various improvements by rouseabout in https://github.com/lhotse-speech/lhotse/pull/1070

New Contributors
* rouseabout made their first contribution in https://github.com/lhotse-speech/lhotse/pull/1061
* flyingleafe made their first contribution in https://github.com/lhotse-speech/lhotse/pull/1060

**Full Changelog**: https://github.com/lhotse-speech/lhotse/compare/v1.14...v1.15

What's Changed

New features

* Add CLIs for creating Lhotse Shar directories and computing features by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1042
* Integrate torchaudio's 2.0 ffmpeg backend for audio loading + add some optimizations by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1043
* (note: with PyTorch 2.0 set the following env var: `TORCHAUDIO_USE_BACKEND_DISPATCHER=1`)
* Loudness normalization with `pyloudnorm` by desh2608 in https://github.com/lhotse-speech/lhotse/pull/1016

Recipes

New

* LibriLight dataset by yfyeung in https://github.com/lhotse-speech/lhotse/pull/1014
* EDACC recipe by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1022
* [Recipe] BUT Reverb DB by desh2608 in https://github.com/lhotse-speech/lhotse/pull/1028

Improvements

* Aishell3 speaker, gender, and tone labels by zjwang21 in https://github.com/lhotse-speech/lhotse/pull/1027
* Small fix for speechcommands.py by yfyeung in https://github.com/lhotse-speech/lhotse/pull/1012
* Minor fix in LibriCSS recipe by desh2608 in https://github.com/lhotse-speech/lhotse/pull/1021
* Add `parts` and `num_jobs` options for tedlium by desh2608 in https://github.com/lhotse-speech/lhotse/pull/1030

Other enhancements

* Bump version to 1.14.0.dev and fix .dev version suffix handling by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1010
* Update recording id in the supervision after applying WPE by desh2608 in https://github.com/lhotse-speech/lhotse/pull/1015
* Specify what formats are expected when using the pipe: prefix to load cuts by srdecny in https://github.com/lhotse-speech/lhotse/pull/1013
* Function to drop alignments from cut by desh2608 in https://github.com/lhotse-speech/lhotse/pull/1019
* Fix bug in lazy CutSet subset with last by desh2608 in https://github.com/lhotse-speech/lhotse/pull/1023
* Fix LoudnessNormalization by lifeiteng in https://github.com/lhotse-speech/lhotse/pull/1029
* API to enable/disable ffmpeg-torchaudio by desh2608 in https://github.com/lhotse-speech/lhotse/pull/1032
* Ensure RIR has same sampling rate as audio by desh2608 in https://github.com/lhotse-speech/lhotse/pull/1037
* Add `transforms` attribute for MixedCut by desh2608 in https://github.com/lhotse-speech/lhotse/pull/1035
* Fix 1038 and 1039 by pzelasko in https://github.com/lhotse-speech/lhotse/pull/1040

New Contributors
* srdecny made their first contribution in https://github.com/lhotse-speech/lhotse/pull/1013

**Full Changelog**: https://github.com/lhotse-speech/lhotse/compare/v1.13...v1.14