Stream2sentence

Latest version: v0.2.3

Safety actively analyzes 627551 Python packages for vulnerabilities to keep your Python projects secure.

Scan your dependencies

0.2.3

- new parameters:
sentence_fragment_delimiters: Characters considered as sentence delimiters for yielding quick fragment.
force_first_fragment_after_words: Forces the first sentence fragment to yield after a specified number of words. Default is 10 words.

0.2.2

- enable early tokenizer initialization with `init_tokenizer`

0.2.1

Minor bugfix
- print message exchanged with logging.info for more clean and customizable output

0.2.0

- added stanza tokenizer to support sentence splitting for more languages (for example chinese)
python
text = "我喜欢读书。天气很好。我们去公园吧。今天是星期五。早上好。这是我的朋友。请帮我。吃饭了吗?我在学中文。晚安。"
expected = ["我喜欢读书。", "天气很好。", "我们去公园吧。", "今天是星期五。", "早上好。", "这是我的朋友。", "请帮我。", "吃饭了吗?", "我在学中文。晚安。"]
sentences = list(generate_sentences(text, minimum_sentence_length = 2, context_size=2, tokenizer="stanza", language="zh"))
self.assertEqual(sentences, expected)

- emoji library added to filter emojis more precisely out of the stream (the previous emoji filter method would not work well with some languages)

Links

Releases

Has known vulnerabilities

© 2024 Safety CLI Cybersecurity Inc. All Rights Reserved.