Workflow

Tokenicer

Latest version: v0.0.4

The latest version of tokenicer with no known security vulnerabilities is 0.0.4. We recommend installing version 0.0.4.

The information on this page was curated by experts in our Cybersecurity Intelligence Team.

Latest release
v0.0.4 at Feb. 21, 2025
License
Apache-2.0 (Apache License 2.0)

Description

A (nicer) tokenizer you want to use for model `inference` and `training`: with all known peventable `gotchas` normalized or auto-fixed.

Resources

Vulnerabilities

See all vulnerabilities

No known vulnerabilities found

Versions (5)

See all versions

Has known vulnerabilities

  • 0.0.4
  • 0.0.3
  • 0.0.2
  • 0.0.1
  • 0.0.1.dev0