What's Changed
* **New feature** Tiktoken usage for estimating number of tokens in prompt / response, fitting semantic search results to target token number
Examples:
python
// limit semantic search results
mc.texts.search("test_collection", "query", n_results=20).fit_to_token_size(max_tokens=5, min_documents=3)
// extended string now have .to_tokens() and .num_tokens() methods:
prompt= mc.tpl('my_template.j2')
tokens = prompt.to_tokens(for_model="gpt-4")
prompt.num_tokens()
mc.llm(prompt).num_tokens()
For usage with any regular strings, see [microcore.tokenizing](https://github.com/Nayjest/ai-microcore/blob/main/microcore/tokenizing.py)
* ui improvements:
* ui.ask_yn() now have "default" argument and also handles KeyboardInterrupt
* new function: ui.warninig()
* Other minor fixes
**Full Changelog**: https://github.com/Nayjest/ai-microcore/compare/v3.9.1...v3.10.0