● The New Computer Update
This is the most significant upgrade to Open Interpreter since `0.1.0`. Almost every component has been rewritten to support our project's objective—building a standard interface between language models and computers.
**Note:** This update is not backwards compatible. If you use Open Interpreter in Python, please read our [migration guide](https://github.com/KillianLucas/open-interpreter/blob/main/docs/NCU_MIGRATION_GUIDE.md).
1. The Computer API
In 2023, Open Interpreter started building the world's first real-time code execution environment for language models.
Now, we're creating an API for language models to use in that environment, starting with basic I/O controls like `display`, `mouse`, and `keyboard`:
python
computer.display.view() Returns a screenshot to vision models
computer.mouse.click("On-screen text") Locates text on-screen and clicks it
computer.mouse.move(icon="Description of icon") Locates icon on-screen and hovers over it
computer.clipboard.view() Returns the contents of the user's clipboard
Full reference: https://docs.openinterpreter.com/computer-api/reference
We are also launching a free preview of the hosted tools that power `computer.mouse` at [api.openinterpreter.com](https://api.openinterpreter.com/).
2. `OS` Mode
You can instruct Open Interpreter to use the Computer API to control your computer graphically:
interpreter --os
Even local vision models running via `.llamafile`, `LM-Studio`, or `Jan.ai` are supported.
3. LMC Messages
To support the incoming `L`anguage `M`odel `C`omputer architecture, the new messaging format that powers Open Interpreter extends OpenAI's [messages](https://platform.openai.com/docs/guides/text-generation/chat-completions-api) format to include additional information, and a new role called `computer`:
python
[
{
"role": "assistant",
"type": "code",
"format": "python",
"content": "plot = create_plot_from_data('data')\ndisplay_as_image(plot)\ndisplay_as_html(plot)"
},
{
"role": "computer",
"type": "image",
"format": "base64.png",
"content": "base64"
},
{
"role": "assistant",
"type": "message",
"content": "Plot generated successfully."
}
]
Read about `LMC Messages` [here](https://github.com/KillianLucas/open-interpreter/blob/main/docs/NCU_MIGRATION_GUIDE.md#new-static-messages-structure).
And more...
In addition to these major updates, `0.2.0` comes with a suite of fixes and enhancement from our growing open-source community:
* Fixes crash UnboundLocalError active_block by CyanideByte in https://github.com/KillianLucas/open-interpreter/pull/818
* Inserts dummy api key if missing by CyanideByte in https://github.com/KillianLucas/open-interpreter/pull/808
* Fix README_JA.md by tegnike in https://github.com/KillianLucas/open-interpreter/pull/810
* return empty dict if config file is empty by sbendary25 in https://github.com/KillianLucas/open-interpreter/pull/811
* Add pyautogui mouse click functions and parameters by Arrendy in https://github.com/KillianLucas/open-interpreter/pull/843
* Fixed the error when using Azure OpenAI API by wsbao in https://github.com/KillianLucas/open-interpreter/pull/840
* Add package mismatches info to system_debug_info by tegnike in https://github.com/KillianLucas/open-interpreter/pull/800
* Update keyboard control functions for better input handling by Arrendy in https://github.com/KillianLucas/open-interpreter/pull/845
* implement mouse position function by Arrendy in https://github.com/KillianLucas/open-interpreter/pull/850
* Fixed another a few bugs in using OpenAI API/Azure OpenAI API/OpenAI compatible custom API by wsbao in https://github.com/KillianLucas/open-interpreter/pull/848
* Added new docs for litellm hosted models by tyfiero in https://github.com/KillianLucas/open-interpreter/pull/858
* Added refreshed docs for terminal arguments and python streaming responses by tyfiero in https://github.com/KillianLucas/open-interpreter/pull/864
* Add os docs by tyfiero in https://github.com/KillianLucas/open-interpreter/pull/868
* Fixed the case where UnicodeDecodeError by Moonlight-YS in https://github.com/KillianLucas/open-interpreter/pull/863
New Contributors
* tegnike made their first contribution in https://github.com/KillianLucas/open-interpreter/pull/810
* sbendary25 made their first contribution in https://github.com/KillianLucas/open-interpreter/pull/811
* Arrendy made their first contribution in https://github.com/KillianLucas/open-interpreter/pull/843
* wsbao made their first contribution in https://github.com/KillianLucas/open-interpreter/pull/840
* tyfiero made their first contribution in https://github.com/KillianLucas/open-interpreter/pull/858
* Moonlight-YS made their first contribution in https://github.com/KillianLucas/open-interpreter/pull/863
**Full Changelog**: https://github.com/KillianLucas/open-interpreter/compare/v0.1.17...v0.2.0
This is only the beginning. Happy 2024. ●