Utoken

Latest version: v0.1.8

The latest version of utoken with no known security vulnerabilities is 0.1.8. We recommend installing version 0.1.8.

The information on this page was curated by experts in our Cybersecurity Intelligence Team.

Latest release: v0.1.8 at Oct. 20, 2021
License: Apache-2.0 (Apache License 2.0)

Description

utoken is a universal tokenizer (multilingual word segmenter) that divides text into words, punctuation and special tokens such as numbers, URLs, XML tags, email-addresses and hashtags. It comes with a companion detokenizer.

Resources

GitHub
PyPi

Vulnerabilities

See all vulnerabilities

No known vulnerabilities found

Versions (6)

See all versions

Has known vulnerabilities

0.1.8
0.1.7
0.1.6
0.1.3
0.1.2
0.1.1