Llm

- Fix for `No module named 'readline'` error on Windows. [407](https://github.com/simonw/llm/issues/407)

See also [LLM 0.13: The annotated release notes](https://simonwillison.net/2024/Jan/26/llm/).

- Added support for new OpenAI embedding models: `3-small` and `3-large` and three variants of those with different dimension sizes, `3-small-512`, `3-large-256` and `3-large-1024`. See [OpenAI embedding models](https://llm.datasette.io/en/stable/openai-models.html#openai-models-embedding) for details. [394](https://github.com/simonw/llm/issues/394)
- The default `gpt-4-turbo` model alias now points to `gpt-4-turbo-preview`, which uses the most recent OpenAI GPT-4 turbo model (currently `gpt-4-0125-preview`). [396](https://github.com/simonw/llm/issues/396)
- New OpenAI model aliases `gpt-4-1106-preview` and `gpt-4-0125-preview`.
- OpenAI models now support a `-o json_object 1` option which will cause their output to be returned as a valid JSON object. [373](https://github.com/simonw/llm/issues/373)
- New [plugins](https://llm.datasette.io/en/stable/plugins/directory.html#plugin-directory) since the last release include [llm-mistral](https://github.com/simonw/llm-mistral), [llm-gemini](https://github.com/simonw/llm-gemini), [llm-ollama](https://github.com/taketwo/llm-ollama) and [llm-bedrock-meta](https://github.com/flabat/llm-bedrock-meta).
- The `keys.json` file for storing API keys is now created with `600` file permissions. [351](https://github.com/simonw/llm/issues/351)
- Documented [a pattern](https://llm.datasette.io/en/stable/setup.html#homebrew-warning) for installing plugins that depend on PyTorch using the Homebrew version of LLM, despite Homebrew using Python 3.12 when PyTorch have not yet released a stable package for that Python version. [397](https://github.com/simonw/llm/issues/397)
- Underlying OpenAI Python library has been upgraded to `>1.0`. It is possible this could cause compatibility issues with LLM plugins that also depend on that library. [325](https://github.com/simonw/llm/issues/325)
- Arrow keys now work inside the `llm chat` command. [376](https://github.com/simonw/llm/issues/376)
- `LLM_OPENAI_SHOW_RESPONSES=1` environment variable now outputs much more detailed information about the HTTP request and response made to OpenAI (and OpenAI-compatible) APIs. [404](https://github.com/simonw/llm/issues/404)
- Dropped support for Python 3.7.

- Support for the [new GPT-4 Turbo model](https://openai.com/blog/new-models-and-developer-products-announced-at-devday) from OpenAI. Try it using `llm chat -m gpt-4-turbo` or `llm chat -m 4t`. [#323](https://github.com/simonw/llm/issues/323)
- New `-o seed 1` option for OpenAI models which sets a seed that can attempt to evaluate the prompt deterministically. [324](https://github.com/simonw/llm/issues/324)

- Pin to version of OpenAI Python library prior to 1.0 to avoid breaking. [327](https://github.com/simonw/llm/issues/327)

- Fixed a bug where `llm embed -c "text"` did not correctly pick up the configured [default embedding model](https://llm.datasette.io/en/stable/embeddings/cli.html#embeddings-cli-embed-models-default). [317](https://github.com/simonw/llm/issues/317)
- New plugins: [llm-python](https://github.com/simonw/llm-python), [llm-bedrock-anthropic](https://github.com/sblakey/llm-bedrock-anthropic) and [llm-embed-jina](https://github.com/simonw/llm-embed-jina) (described in [Execute Jina embeddings with a CLI using llm-embed-jina](https://simonwillison.net/2023/Oct/26/llm-embed-jina/)).
- [llm-gpt4all](https://github.com/simonw/llm-gpt4all) now uses the new GGUF model format. [simonw/llm-gpt4all#16](https://github.com/simonw/llm-gpt4all/issues/16)

LLM now supports the new OpenAI `gpt-3.5-turbo-instruct` model, and OpenAI completion (as opposed to chat completion) models in general. [284](https://github.com/simonw/llm/issues/284)

bash
llm -m gpt-3.5-turbo-instruct 'Reasons to tame a wild beaver:'

OpenAI completion models like this support a `-o logprobs 3` option, which accepts a number between 1 and 5 and will include the log probabilities (for each produced token, what were the top 3 options considered by the model) in the logged response.

bash
llm -m gpt-3.5-turbo-instruct 'Say hello succinctly' -o logprobs 3

You can then view the `logprobs` that were recorded in the SQLite logs database like this:
bash
sqlite-utils "$(llm logs path)" \
'select * from responses order by id desc limit 1' | \
jq '.[0].response_json' -r | jq

Truncated output looks like this:

[
{
"text": "Hi",
"top_logprobs": [
{
"Hi": -0.13706253,
"Hello": -2.3714375,
"Hey": -3.3714373
}
]
},
{
"text": " there",
"top_logprobs": [
{
" there": -0.96057636,
"!\"": -0.5855763,
".\"": -3.2574513
}
]
}
]

Also in this release:

- The `llm.user_dir()` function, used by plugins, now ensures the directory exists before returning it. [275](https://github.com/simonw/llm/issues/275)
- New `LLM_OPENAI_SHOW_RESPONSES=1` environment variable for displaying the full HTTP response returned by OpenAI compatible APIs. [286](https://github.com/simonw/llm/issues/286)
- The `llm embed-multi` command now has a `--batch-size X` option for setting the batch size to use when processing embeddings - useful if you have limited memory available. [273](https://github.com/simonw/llm/issues/273)
- The `collection.embed_multi()` method also now accepts an optional `batch_size=int` argument.
- Fixed two bugs with `llm embed-multi --files` relating to handling of directories. Thanks, [ealvar3z](https://github.com/ealvar3z). [#274](https://github.com/simonw/llm/issues/274), [#280](https://github.com/simonw/llm/issues/280)