Sign-language-translator

This release contains sign language videos **embedded as csv files** inside zip archives. The landmarks are rounded to 4 decimal places which give a precision of 0.1mm in world coordinates and 1 pixel in a 10k resolution image.

Text transcription of the signs is present in the file names. More synonyms and translations that map to these signs can be seen in the json data in the repo. The dataset has three categories:

- **Standard Dictionary**: (788)
Standard sign language dictionaries obtained from recognized organizations. The names are `country-organization-groupNumber_featureType-subtype.zip`

- **Dictionary Recordings**: (788 * 12 * 4 = 37,824) (coming soon!)
Manually recorded sign language videos that are replication of the reference clips. The names are `country-organization-groupNumber_featureType-subtype_personCode-cameraAngle.zip`

- **Miscellaneous Sentences**:
These are labeled sign language videos scraped from the internet. The names are `languageName-source-serialNumber_featureType-subtype.zip`.

Mediapipe landmarks Header


wp_x0, wp_y0, wp_z0, wp_v0, wp_p0, ..., wp_x32, wp_y32, wp_z32, wp_v32, wp_p32,
wlh_x0, wlh_y0, wlh_z0, wlh_v0, wlh_p0, ..., wlh_x20, wlh_y20, wlh_z20, wlh_v20, wlh_p20,
wrh_x0, wrh_y0, wrh_z0, wrh_v0, wrh_p0, ..., wrh_x20, wrh_y20, wrh_z20, wrh_v20, wrh_p20,
ip_x0, ip_y0, ip_z0, ip_v0, ip_p0, ..., ip_x32, ip_y32, ip_z32, ip_v32, ip_p32,
ilh_x0, ilh_y0, ilh_z0, ilh_v0, ilh_p0, ..., ilh_x20, ilh_y20, ilh_z20, ilh_v20, ilh_p20,
irh_x0, irh_y0, irh_z0, irh_v0, irh_p0, ..., irh_x20, irh_y20, irh_z20, irh_v20, irh_p20,



key:

- w: world
- i: image
- p: pose
- h: hand
- l: left
- r: right
_
- x: x-coordinate
- y: y-coordinate
- z: depth/distance from camera
- v: visibility
- p: presence

total_columns = (33 + 21 + 21) * 5 * 2 = 750
total_rows = number_of_frames in source video

This release contains sign language videos inside zip archives. Text transcription of the videos is present in the file names. More synonyms and translations that map to these videos can be seen in the json data in the repo. The dataset has three categories:

- **Standard Dictionary**: (788)
Standard sign language dictionaries obtained from recognized organizations. The names are `country-organization-groupNumber_featureType-subtype.zip`

- **Dictionary Recordings**: (788 * 12 * 4 = 37,824) (coming soon!)
Manually recorded sign language videos that are replication of the reference clips. The names are `country-organization-groupNumber_featureType-subtype_personCode-cameraAngle.zip`

- **Miscellaneous Sentences**:
These are labeled sign language videos scraped from the internet. The names are `languageName-source-serialNumber_featureType-subtype.zip`.

This release contains all base videos featuring someone performing sign language. They were also called reference_clips because they were shown to other performers to replicate sign language to create a diverse dataset.

The purpose of this release is to provide a URL to individual video files so the rule-based translator can fetch only the required files. If you want this data, see the video_datasets release instead for bundled archives.

The file labels had to be converted to english via translation or transliteration so these filenames may not correspond to the json word-mapping data. Although urls.json & extra_urls.json provide a mapping from the standard filename to these individual assets.

The filename format is as follows:

country-organization-session_text-equivalent-of-video[.disambiguation].mp4

e.g. pk-hfad-1_spring.season.mp4

This release includes language models to write text that can be translated by a rule-based text-to-sign translator.

The tlm_14.0.pt (sign_language_translator.models.TransformerLanguageModel) is a custom transformer trained on ~800 MB of text composed only of the words for which PakistanSignLanguage signs are available (see sign_recordings/collection_to_label_to_language_to_words.json). The tokenizer used is sign_language_translator.languages.text.urdu.Urdu().tokenizer + the digits in numbers and letter in acronyms are split apart as individual tokens to limit the vocab size. Later update will generate disambiguated words. The start & end token are "<" & ">".

The *-mixed-*.pkl model is trained on unambiguous supported urdu words from a corpus of around 10 MB (2.4 million tokens). It is a mix of 6 ngram models with context window size from 1 to 6. It cannot handle any longer range dependencies so concept drift can be observed in the longer generations. The tokenizer used is slt.languages.text.urdu.Urdu().tokenizer. The start & end token are "<" & ">".

The *.json models are made to demonstrate the functionality of n-gram models. The training data is text_preprocessing.json:person_names.
- Contains n-gram based statistical language models trained on 366 Urdu and 366 English names commonly used in Pakistan.
- The models predict the next character based on previous 1-3 characters.
- The start and end of sequence tokens are "[" and "]".