Openllm

Usage

All available models: openllm models

To start a LLM: python -m openllm start opt

To run OpenLLM within a container environment (requires GPUs): docker run --gpus all -it -P ghcr.io/bentoml/openllm:0.3.9 start opt

To run OpenLLM Clojure UI (community-maintained): docker run -p 8420:80 ghcr.io/bentoml/openllm-ui-clojure:0.3.9

Find more information about this release in the [CHANGELOG.md](https://github.com/bentoml/OpenLLM/blob/main/CHANGELOG.md)



**Full Changelog**: https://github.com/bentoml/OpenLLM/compare/v0.3.8...v0.3.9

Usage

All available models: openllm models

To start a LLM: python -m openllm start opt

To run OpenLLM within a container environment (requires GPUs): docker run --gpus all -it -P ghcr.io/bentoml/openllm:0.3.8 start opt

To run OpenLLM Clojure UI (community-maintained): docker run -p 8420:80 ghcr.io/bentoml/openllm-ui-clojure:0.3.8

Find more information about this release in the [CHANGELOG.md](https://github.com/bentoml/OpenLLM/blob/main/CHANGELOG.md)



What's Changed
* docs: Add OpenLLM tutorial Google Colab link by Sherlock113 in https://github.com/bentoml/OpenLLM/pull/497
* chore(deps-dev): bump types/react-dom from 18.0.6 to 18.2.13 by dependabot in https://github.com/bentoml/OpenLLM/pull/492
* chore(deps-dev): bump types/react from 18.2.20 to 18.2.28 by dependabot in https://github.com/bentoml/OpenLLM/pull/493
* chore(deps-dev): bump typescript from 5.1.6 to 5.2.2 by dependabot in https://github.com/bentoml/OpenLLM/pull/494
* chore(deps-dev): bump types/node from 20.5.3 to 20.8.5 by dependabot in https://github.com/bentoml/OpenLLM/pull/495
* fix(breaking): remove embeddings and update client implementation by aarnphm in https://github.com/bentoml/OpenLLM/pull/500
* feat: openai.Model.list() by XunchaoZ in https://github.com/bentoml/OpenLLM/pull/499
* chore(deps): bump taiki-e/install-action from 2.20.2 to 2.20.3 by dependabot in https://github.com/bentoml/OpenLLM/pull/502
* chore(deps): bump github/codeql-action from 2.22.2 to 2.22.3 by dependabot in https://github.com/bentoml/OpenLLM/pull/501
* ci: pre-commit autoupdate [pre-commit.ci] by pre-commit-ci in https://github.com/bentoml/OpenLLM/pull/506
* fix: check for parity by aarnphm in https://github.com/bentoml/OpenLLM/pull/508
* chore(deps-dev): bump prettier-plugin-tailwindcss from 0.5.5 to 0.5.6 by dependabot in https://github.com/bentoml/OpenLLM/pull/504
* chore(deps-dev): bump types/node from 20.5.3 to 20.8.6 by dependabot in https://github.com/bentoml/OpenLLM/pull/503


**Full Changelog**: https://github.com/bentoml/OpenLLM/compare/v0.3.7...v0.3.8

Usage

All available models: openllm models

To start a LLM: python -m openllm start opt

To run OpenLLM within a container environment (requires GPUs): docker run --gpus all -it -P ghcr.io/bentoml/openllm:0.3.7 start opt

To run OpenLLM Clojure UI (community-maintained): docker run -p 8420:80 ghcr.io/bentoml/openllm-ui-clojure:0.3.7

Find more information about this release in the [CHANGELOG.md](https://github.com/bentoml/OpenLLM/blob/main/CHANGELOG.md)



What's Changed
* chore(deps): bump taiki-e/install-action from 2.18.9 to 2.18.13 by dependabot in https://github.com/bentoml/OpenLLM/pull/372
* chore(deps): bump pypa/cibuildwheel from 2.15.0 to 2.16.0 by dependabot in https://github.com/bentoml/OpenLLM/pull/376
* chore(deps): bump github/codeql-action from 2.21.5 to 2.21.8 by dependabot in https://github.com/bentoml/OpenLLM/pull/378
* chore(deps): bump mui/material from 5.14.8 to 5.14.9 by dependabot in https://github.com/bentoml/OpenLLM/pull/368
* chore(deps-dev): bump edge-runtime/vm from 3.1.0 to 3.1.3 by dependabot in https://github.com/bentoml/OpenLLM/pull/370
* chore(deps): bump floating-ui/dom from 1.5.1 to 1.5.3 by dependabot in https://github.com/bentoml/OpenLLM/pull/366
* chore(deps-dev): bump typescript-eslint/parser from 6.4.1 to 6.7.0 by dependabot in https://github.com/bentoml/OpenLLM/pull/365
* chore(deps): bump vercel/og from 0.5.11 to 0.5.13 by dependabot in https://github.com/bentoml/OpenLLM/pull/356
* chore(deps-dev): bump postcss from 8.4.23 to 8.4.29 by dependabot in https://github.com/bentoml/OpenLLM/pull/360
* chore(deps-dev): bump turbo from 1.10.13 to 1.10.14 by dependabot in https://github.com/bentoml/OpenLLM/pull/364
* chore(bench): add more prompts by aarnphm in https://github.com/bentoml/OpenLLM/pull/380
* chore(deps): bump mui/styled-engine from 5.13.2 to 5.14.10 by dependabot in https://github.com/bentoml/OpenLLM/pull/379
* chore(deps): bump mui/base from 5.0.0-beta.14 to 5.0.0-beta.16 by dependabot in https://github.com/bentoml/OpenLLM/pull/384
* chore(deps): bump babel/runtime from 7.22.10 to 7.22.15 by dependabot in https://github.com/bentoml/OpenLLM/pull/362
* chore(deps): bump mui/system from 5.14.8 to 5.14.10 by dependabot in https://github.com/bentoml/OpenLLM/pull/383
* chore(deps): bump mui/x-data-grid from 6.13.0 to 6.14.0 by dependabot in https://github.com/bentoml/OpenLLM/pull/361
* chore(deps): bump mui/private-theming from 5.14.8 to 5.14.10 by dependabot in https://github.com/bentoml/OpenLLM/pull/382
* chore(deps): bump mui/icons-material from 5.14.8 to 5.14.9 by dependabot in https://github.com/bentoml/OpenLLM/pull/367
* chore(deps): bump mui/utils from 5.14.8 to 5.14.10 by dependabot in https://github.com/bentoml/OpenLLM/pull/381
* ci: pre-commit autoupdate [pre-commit.ci] by pre-commit-ci in https://github.com/bentoml/OpenLLM/pull/406
* chore(deps): bump actions/checkout from 4.0.0 to 4.1.0 by dependabot in https://github.com/bentoml/OpenLLM/pull/405
* chore(deps): bump taiki-e/install-action from 2.18.13 to 2.18.16 by dependabot in https://github.com/bentoml/OpenLLM/pull/404
* chore(deps-dev): bump eslint from 8.47.0 to 8.50.0 by dependabot in https://github.com/bentoml/OpenLLM/pull/400
* chore(deps): bump nextra from 2.11.1 to 2.13.0 by dependabot in https://github.com/bentoml/OpenLLM/pull/393
* chore(deps-dev): bump types/react from 18.2.20 to 18.2.22 by dependabot in https://github.com/bentoml/OpenLLM/pull/403
* chore(deps-dev): bump tailwindcss from 3.3.2 to 3.3.3 by dependabot in https://github.com/bentoml/OpenLLM/pull/398
* chore(deps-dev): bump autoprefixer from 10.4.12 to 10.4.16 by dependabot in https://github.com/bentoml/OpenLLM/pull/397
* chore(deps): bump vercel/og from 0.5.13 to 0.5.17 by dependabot in https://github.com/bentoml/OpenLLM/pull/396
* chore(deps-dev): bump next/eslint-plugin-next from 13.4.19 to 13.5.2 by dependabot in https://github.com/bentoml/OpenLLM/pull/392
* chore(deps): bump floating-ui/core from 1.4.1 to 1.5.0 by dependabot in https://github.com/bentoml/OpenLLM/pull/402
* chore(deps-dev): bump svgr/webpack from 2.0.0-alpha.26fa501a to 8.1.0 by dependabot in https://github.com/bentoml/OpenLLM/pull/389
* chore(deps): bump nextra-theme-docs from 2.12.3 to 2.13.1 by dependabot in https://github.com/bentoml/OpenLLM/pull/411
* chore(deps-dev): bump typescript-eslint/eslint-plugin from 6.7.0 to 6.7.3 by dependabot in https://github.com/bentoml/OpenLLM/pull/414
* chore(deps-dev): bump shadow-cljs from 2.25.4 to 2.25.6 by dependabot in https://github.com/bentoml/OpenLLM/pull/394
* chore(deps): bump mui/x-data-grid from 6.14.0 to 6.15.0 by dependabot in https://github.com/bentoml/OpenLLM/pull/395
* chore(deps-dev): bump typescript-eslint/parser from 6.7.0 to 6.7.3 by dependabot in https://github.com/bentoml/OpenLLM/pull/412
* chore(deps-dev): bump types/react from 18.2.20 to 18.2.23 by dependabot in https://github.com/bentoml/OpenLLM/pull/413
* chore(deps-dev): bump postcss from 8.4.29 to 8.4.30 by dependabot in https://github.com/bentoml/OpenLLM/pull/390
* chore(deps): bump mui/x-data-grid from 6.15.0 to 6.16.0 by dependabot in https://github.com/bentoml/OpenLLM/pull/434
* chore(deps-dev): bump postcss from 8.4.30 to 8.4.31 by dependabot in https://github.com/bentoml/OpenLLM/pull/430
* chore(deps): bump pypa/cibuildwheel from 2.16.0 to 2.16.1 by dependabot in https://github.com/bentoml/OpenLLM/pull/440
* chore(deps): bump github/codeql-action from 2.21.8 to 2.21.9 by dependabot in https://github.com/bentoml/OpenLLM/pull/439
* chore(deps): bump taiki-e/install-action from 2.18.16 to 2.19.2 by dependabot in https://github.com/bentoml/OpenLLM/pull/438
* chore(deps): bump mui/x-date-pickers from 6.13.0 to 6.16.0 by dependabot in https://github.com/bentoml/OpenLLM/pull/428
* chore(deps-dev): bump types/react from 18.2.20 to 18.2.24 by dependabot in https://github.com/bentoml/OpenLLM/pull/427
* chore(deps-dev): bump types/node from 20.5.3 to 20.8.0 by dependabot in https://github.com/bentoml/OpenLLM/pull/435
* ci: pre-commit autoupdate [pre-commit.ci] by pre-commit-ci in https://github.com/bentoml/OpenLLM/pull/441
* chore(deps): bump babel/runtime from 7.22.15 to 7.23.1 by dependabot in https://github.com/bentoml/OpenLLM/pull/436
* chore(deps): bump nextra from 2.13.0 to 2.13.1 by dependabot in https://github.com/bentoml/OpenLLM/pull/425
* feat: PromptTemplate and system prompt support by MingLiangDai in https://github.com/bentoml/OpenLLM/pull/407
* feat: OpenAI compatible API by XunchaoZ in https://github.com/bentoml/OpenLLM/pull/417
* chore(deps): bump pypa/cibuildwheel from 2.16.1 to 2.16.2 by dependabot in https://github.com/bentoml/OpenLLM/pull/474
* chore(deps): bump github/codeql-action from 2.21.9 to 2.22.0 by dependabot in https://github.com/bentoml/OpenLLM/pull/473
* chore(deps): bump aws-actions/configure-aws-credentials from 4.0.0 to 4.0.1 by dependabot in https://github.com/bentoml/OpenLLM/pull/471
* chore(deps): bump actions/setup-python from 4.7.0 to 4.7.1 by dependabot in https://github.com/bentoml/OpenLLM/pull/470
* chore(deps): bump taiki-e/install-action from 2.19.2 to 2.20.1 by dependabot in https://github.com/bentoml/OpenLLM/pull/472
* ci: pre-commit autoupdate [pre-commit.ci] by pre-commit-ci in https://github.com/bentoml/OpenLLM/pull/475
* fix: do not reply on env var for built bento/docker by larme in https://github.com/bentoml/OpenLLM/pull/477
* fix: asyncio stalling inside notebooks by xianml in https://github.com/bentoml/OpenLLM/pull/478
* fix: fix client HTTPS by sauyon in https://github.com/bentoml/OpenLLM/pull/480
* feat: add 1 openllm llama2 notebook demo by xianml in https://github.com/bentoml/OpenLLM/pull/479
* chore(deps-dev): bump eslint from 8.47.0 to 8.51.0 by dependabot in https://github.com/bentoml/OpenLLM/pull/469
* chore(deps-dev): bump types/node from 20.5.3 to 20.8.3 by dependabot in https://github.com/bentoml/OpenLLM/pull/467
* chore(deps-dev): bump turbo from 1.10.14 to 1.10.15 by dependabot in https://github.com/bentoml/OpenLLM/pull/461
* chore(deps): bump nextra from 2.13.1 to 2.13.2 by dependabot in https://github.com/bentoml/OpenLLM/pull/458
* chore(deps): bump nextra-theme-docs from 2.13.1 to 2.13.2 by dependabot in https://github.com/bentoml/OpenLLM/pull/457
* chore(deps-dev): bump typescript-eslint/eslint-plugin from 6.7.3 to 6.7.5 by dependabot in https://github.com/bentoml/OpenLLM/pull/483
* chore(deps): bump highlight.js from 11.8.0 to 11.9.0 by dependabot in https://github.com/bentoml/OpenLLM/pull/456
* chore(deps-dev): bump next/eslint-plugin-next from 13.5.2 to 13.5.4 by dependabot in https://github.com/bentoml/OpenLLM/pull/464
* chore(deps-dev): bump typescript-eslint/parser from 6.7.3 to 6.7.5 by dependabot in https://github.com/bentoml/OpenLLM/pull/482
* chore(deps-dev): bump prettier-plugin-tailwindcss from 0.5.4 to 0.5.5 by dependabot in https://github.com/bentoml/OpenLLM/pull/455
* chore(deps-dev): bump types/react-transition-group from 4.4.6 to 4.4.7 by dependabot in https://github.com/bentoml/OpenLLM/pull/460
* chore(deps-dev): bump edge-runtime/vm from 3.1.3 to 3.1.4 by dependabot in https://github.com/bentoml/OpenLLM/pull/466
* chore(deps): bump github/codeql-action from 2.22.0 to 2.22.2 by dependabot in https://github.com/bentoml/OpenLLM/pull/484
* chore(deps): bump taiki-e/install-action from 2.20.1 to 2.20.2 by dependabot in https://github.com/bentoml/OpenLLM/pull/485
* chore(deps-dev): bump types/react from 18.2.20 to 18.2.28 by dependabot in https://github.com/bentoml/OpenLLM/pull/486
* chore(deps): bump mui/x-data-grid from 6.16.0 to 6.16.2 by dependabot in https://github.com/bentoml/OpenLLM/pull/487
* chore(deps): bump vercel/og from 0.5.17 to 0.5.18 by dependabot in https://github.com/bentoml/OpenLLM/pull/488
* chore(deps-dev): bump types/node from 20.5.3 to 20.8.4 by dependabot in https://github.com/bentoml/OpenLLM/pull/491
* chore(deps): bump babel/runtime from 7.23.1 to 7.23.2 by dependabot in https://github.com/bentoml/OpenLLM/pull/490
* feat(client): simple implementation and streaming by aarnphm in https://github.com/bentoml/OpenLLM/pull/256

New Contributors
* MingLiangDai made their first contribution in https://github.com/bentoml/OpenLLM/pull/407
* XunchaoZ made their first contribution in https://github.com/bentoml/OpenLLM/pull/417
* xianml made their first contribution in https://github.com/bentoml/OpenLLM/pull/478
* sauyon made their first contribution in https://github.com/bentoml/OpenLLM/pull/480

**Full Changelog**: https://github.com/bentoml/OpenLLM/compare/v0.3.6...v0.3.7

Usage

All available models: openllm models

To start a LLM: python -m openllm start opt

To run OpenLLM within a container environment (requires GPUs): docker run --gpus all -it -P ghcr.io/bentoml/openllm:0.3.6 start opt

To run OpenLLM Clojure UI (community-maintained): docker run -p 8420:80 ghcr.io/bentoml/openllm-ui-clojure:0.3.6

Find more information about this release in the [CHANGELOG.md](https://github.com/bentoml/OpenLLM/blob/main/CHANGELOG.md)



What's Changed
* ci: pre-commit autoupdate [pre-commit.ci] by pre-commit-ci in https://github.com/bentoml/OpenLLM/pull/374
* chore(deps): bump peter-evans/create-pull-request from 4.2.4 to 5.0.2 by dependabot in https://github.com/bentoml/OpenLLM/pull/373
* feat: support continuous batching on `generate` by aarnphm in https://github.com/bentoml/OpenLLM/pull/375


**Full Changelog**: https://github.com/bentoml/OpenLLM/compare/v0.3.5...v0.3.6

Usage

All available models: openllm models

To start a LLM: python -m openllm start opt

To run OpenLLM within a container environment (requires GPUs): docker run --gpus all -it -P ghcr.io/bentoml/openllm:0.3.5 start opt

To run OpenLLM Clojure UI (community-maintained): docker run -p 8420:80 ghcr.io/bentoml/openllm-ui-clojure:0.3.5

Find more information about this release in the [CHANGELOG.md](https://github.com/bentoml/OpenLLM/blob/main/CHANGELOG.md)



What's Changed
* fix: set default serialisation methods by aarnphm in https://github.com/bentoml/OpenLLM/pull/355


**Full Changelog**: https://github.com/bentoml/OpenLLM/compare/v0.3.4...v0.3.5

Usage

All available models: openllm models

To start a LLM: python -m openllm start opt

To run OpenLLM within a container environment (requires GPUs): docker run --gpus all -it -P ghcr.io/bentoml/openllm:0.3.4 start opt

To run OpenLLM Clojure UI (community-maintained): docker run -p 8420:80 ghcr.io/bentoml/openllm-ui-clojure:0.3.4

Find more information about this release in the [CHANGELOG.md](https://github.com/bentoml/OpenLLM/blob/main/CHANGELOG.md)



What's Changed
* dos: fix typo by Sherlock113 in https://github.com/bentoml/OpenLLM/pull/305
* fix(serving): vllm bad num_gpus by alanpoulain in https://github.com/bentoml/OpenLLM/pull/326
* fix(serialisation): vllm ignore by aarnphm in https://github.com/bentoml/OpenLLM/pull/324
* feat: continuous batching with vLLM by aarnphm in https://github.com/bentoml/OpenLLM/pull/349
* fix(prompt): correct export extra objects items by aarnphm in https://github.com/bentoml/OpenLLM/pull/351

New Contributors
* alanpoulain made their first contribution in https://github.com/bentoml/OpenLLM/pull/326

**Full Changelog**: https://github.com/bentoml/OpenLLM/compare/v0.3.3...v0.3.4