This release includes several important updates and improvements:
* Updates and fixes to existing integrations, including LangChain, Vertex AI, ChromaDB, MindsDB Inference, and more.
* Enhanced test coverage for greater reliability.
* A schema refactor that includes changes to the structure for storing metadata of all objects like models, views, tables, etc.
* Bug fixes to the SQL API, including improvements to the LAST keyword.
All relevant updates have been accompanied by corresponding documentation updates.
What's Changed
* removing redundant package lists and sentence-transformers library by QuantumPlumber in https://github.com/mindsdb/mindsdb/pull/8971
* Safe extraction for all tar files by StpMax in https://github.com/mindsdb/mindsdb/pull/8928
* Fix tarfile safe extract by StpMax in https://github.com/mindsdb/mindsdb/pull/9000
* Refactor Langchain Handler by tmichaeldb in https://github.com/mindsdb/mindsdb/pull/8998
* Tests for sql features by ea-rus in https://github.com/mindsdb/mindsdb/pull/8952
* Update query format in quickstart-tutorial.mdx by vlad-ro in https://github.com/mindsdb/mindsdb/pull/9011
* [404] fix url in huggingface_inference_api.mdx by ArnavK-09 in https://github.com/mindsdb/mindsdb/pull/9016
* Fix hardcoded API base by ZoranPandovski in https://github.com/mindsdb/mindsdb/pull/9019
* Added Check for Import Errors When Running CREATE MODEL Statement by MinuraPunchihewa in https://github.com/mindsdb/mindsdb/pull/9020
* updated hostname of sample dbs by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9025
* Fixed Installation of Dependencies for Integrations That Refer to Other requirements.txt Files by MinuraPunchihewa in https://github.com/mindsdb/mindsdb/pull/9022
* fixed slack chatbot tutorial by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9026
* Fix delete from table in database by ea-rus in https://github.com/mindsdb/mindsdb/pull/9017
* Added File Path and URL Support to the Vertex AI Integration by MinuraPunchihewa in https://github.com/mindsdb/mindsdb/pull/9007
* clean up handler install log messages by hamishfagg in https://github.com/mindsdb/mindsdb/pull/8720
* Docker build improvements by hamishfagg in https://github.com/mindsdb/mindsdb/pull/8999
* Add 'seed db' function to MSSQL tests by hamishfagg in https://github.com/mindsdb/mindsdb/pull/8742
* Fixed Installation of Dependencies for Integrations That Refer to requirements.txt Files (Docker-supported) by MinuraPunchihewa in https://github.com/mindsdb/mindsdb/pull/9034
* Fixed Installation of Dependencies When requirements.txt Files Contain Comments by MinuraPunchihewa in https://github.com/mindsdb/mindsdb/pull/9038
* Fixed the Initializer for the EventStoreDB Integration by MinuraPunchihewa in https://github.com/mindsdb/mindsdb/pull/9029
* Fixed Unused Imports and Dependencies in Rockset Integration by MinuraPunchihewa in https://github.com/mindsdb/mindsdb/pull/9031
* updated email integration docs by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9035
* Make alembic use mindsdb log level by hamishfagg in https://github.com/mindsdb/mindsdb/pull/9028
* Bump version for release by hamishfagg in https://github.com/mindsdb/mindsdb/pull/9041
* Fix LAST with slack by ea-rus in https://github.com/mindsdb/mindsdb/pull/9036
* Updated LightFM Integration Docs with Tip to Install Linux Dev Pakcages by MinuraPunchihewa in https://github.com/mindsdb/mindsdb/pull/9042
* updated ollama docs by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/8995
* Updated Snowflake Integration Docs with Tip to Install Git by MinuraPunchihewa in https://github.com/mindsdb/mindsdb/pull/9032
* Add RAG model and utilities for Mindsdb integration by dusvyat in https://github.com/mindsdb/mindsdb/pull/8996
* Update requirements.txt by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9048
* Update requirements.txt by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9050
* update sampledatabases to new k8s server by lucas-koontz in https://github.com/mindsdb/mindsdb/pull/8986
* The "database" field in the form is changed to not required. by alejandrovillegas in https://github.com/mindsdb/mindsdb/pull/9052
* Fixed Column Structure of chat_messages Table by MinuraPunchihewa in https://github.com/mindsdb/mindsdb/pull/9015
* Fix BYOM engine create by StpMax in https://github.com/mindsdb/mindsdb/pull/9054
* updated agents docs by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9058
* updated broken links in mongo docs by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9059
* Bump version to 24.4.2.1 by StpMax in https://github.com/mindsdb/mindsdb/pull/9061
* Feature/update dd docs by MinuraPunchihewa in https://github.com/mindsdb/mindsdb/pull/9062
* Clear LAST value in editor by ea-rus in https://github.com/mindsdb/mindsdb/pull/9049
* bump litellm version by lucas-koontz in https://github.com/mindsdb/mindsdb/pull/9064
* Managing knowledge base columns by ea-rus in https://github.com/mindsdb/mindsdb/pull/9005
* Fix typo by vlad-ro in https://github.com/mindsdb/mindsdb/pull/9065
* Initial RAG 2.0 integration by dusvyat in https://github.com/mindsdb/mindsdb/pull/9068
* Mindsdb inference endpoints by ZoranPandovski in https://github.com/mindsdb/mindsdb/pull/9044
* Disable storage backup by StpMax in https://github.com/mindsdb/mindsdb/pull/9073
* Bump version to 24.4.3.0 by StpMax in https://github.com/mindsdb/mindsdb/pull/9074
* Kb updates 04.2024 by ea-rus in https://github.com/mindsdb/mindsdb/pull/9069
* 9078-bug-cannot-create-a-langchain-engine by dusvyat in https://github.com/mindsdb/mindsdb/pull/9080
* Remove custom GH actions and use shared ones by hamishfagg in https://github.com/mindsdb/mindsdb/pull/9072
* updated agent skills in docs by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9082
* Сhanged default agent type for supporting conversational history by ea-rus in https://github.com/mindsdb/mindsdb/pull/9087
* Switch to new GHA runners by hamishfagg in https://github.com/mindsdb/mindsdb/pull/9085
* Remove prometheus dir from dockerfile by hamishfagg in https://github.com/mindsdb/mindsdb/pull/9092
* Mysql improvments by ZoranPandovski in https://github.com/mindsdb/mindsdb/pull/9081
* Infernece docs by ZoranPandovski in https://github.com/mindsdb/mindsdb/pull/9100
* updated mindsdb readme by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9055
* Fixed Load Tests and Test Docs (README) by MinuraPunchihewa in https://github.com/mindsdb/mindsdb/pull/9094
* use cases docs plus docs updates by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9095
* Minor tweaks to retrieval skill and file ingestion by dusvyat in https://github.com/mindsdb/mindsdb/pull/9093
* Fixes for RAG Integration to Python SDK by tmichaeldb in https://github.com/mindsdb/mindsdb/pull/9098
* mysql and postgres docs + readme updates by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9102
* MS SQL Handler Improvements by MinuraPunchihewa in https://github.com/mindsdb/mindsdb/pull/9083
* Fix REST API Endpoints URL by ZoranPandovski in https://github.com/mindsdb/mindsdb/pull/9108
* docs cleanup by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9112
* add bigquery to cloud image by hamishfagg in https://github.com/mindsdb/mindsdb/pull/9114
* updated triggers docs by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9116
* Mindsdb provider for langchain by ea-rus in https://github.com/mindsdb/mindsdb/pull/9046
* Update README.md by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9125
* removed mindsdb cloud ref in docs by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9118
* openapi docs for agents/skills/chatbots by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9113
* updated use cases docs by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9110
* [fix] dataframe encode in byom describe by paxcema in https://github.com/mindsdb/mindsdb/pull/9124
* Fixed Script for Running Handler Unit Tests in Workflow by MinuraPunchihewa in https://github.com/mindsdb/mindsdb/pull/9103
* [docs] Improved readme for anomaly detection handler by paxcema in https://github.com/mindsdb/mindsdb/pull/9130
* MindsDB Inference Handler improvements by MinuraPunchihewa in https://github.com/mindsdb/mindsdb/pull/9099
* Drop `semaphor` table by StpMax in https://github.com/mindsdb/mindsdb/pull/9134
* Support New Retrieval Skill for RAG by tmichaeldb in https://github.com/mindsdb/mindsdb/pull/9119
* Update Web Handler README by ZoranPandovski in https://github.com/mindsdb/mindsdb/pull/9138
* Schema refactor by ea-rus in https://github.com/mindsdb/mindsdb/pull/9001
* Remove permanent ml handlers by ea-rus in https://github.com/mindsdb/mindsdb/pull/9128
* Update Avaiability of handlers by ZoranPandovski in https://github.com/mindsdb/mindsdb/pull/9117
* updated docs for LAST by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9147
* Clickhouse improvments by ZoranPandovski in https://github.com/mindsdb/mindsdb/pull/9111
* updated docs links in completed readmes&docs for integrations by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9150
* added info to langchain about langfuse support by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9153
* updated lightwood docs by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9148
* Chromadb: sync handler storage on rw operations by ea-rus in https://github.com/mindsdb/mindsdb/pull/9140
* Use SQLDatabaseToolkit for Agent sql Skill by tmichaeldb in https://github.com/mindsdb/mindsdb/pull/9132
* Fully Support Agents File & Web Retrieval by tmichaeldb in https://github.com/mindsdb/mindsdb/pull/9137
* updated a link in openai finetuning tutorial by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9156
* remove random file by hamishfagg in https://github.com/mindsdb/mindsdb/pull/9157
* Prepare to rename default branch to "main" by hamishfagg in https://github.com/mindsdb/mindsdb/pull/9089
* Update doc links for branch rename by hamishfagg in https://github.com/mindsdb/mindsdb/pull/9158
* Added Tip for Enabling Dev Mode (Windows) Before Installing Extension by MinuraPunchihewa in https://github.com/mindsdb/mindsdb/pull/9146
* Fix drop ml-engine by ea-rus in https://github.com/mindsdb/mindsdb/pull/9143
* Project and database can't have same name by ea-rus in https://github.com/mindsdb/mindsdb/pull/9127
* removed ref to mindsdb (cloud) pro by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9154
* updated use cases diagrams by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9167
* Anomaly detection docs by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9152
* Moved the File Splitter Imports to the Function They are Used in by MinuraPunchihewa in https://github.com/mindsdb/mindsdb/pull/9162
* Mongo shell: describe model error shows right command by ea-rus in https://github.com/mindsdb/mindsdb/pull/9126
* Model column mapping by ea-rus in https://github.com/mindsdb/mindsdb/pull/9166
* remove references to removed table by hamishfagg in https://github.com/mindsdb/mindsdb/pull/9165
* Fix mysql connection status tracking by hamishfagg in https://github.com/mindsdb/mindsdb/pull/9120
* [WIP] Fix langchain handler install by hamishfagg in https://github.com/mindsdb/mindsdb/pull/9115
* Restrict concurrent gha workflows by hamishfagg in https://github.com/mindsdb/mindsdb/pull/9091
* Rest api for jobs by ea-rus in https://github.com/mindsdb/mindsdb/pull/9142
* Bump type_infer to 0.0.20 by hamishfagg in https://github.com/mindsdb/mindsdb/pull/9170
* add docker commands to makefile by hamishfagg in https://github.com/mindsdb/mindsdb/pull/9173
* Update pydantic to 2.0 by hamishfagg in https://github.com/mindsdb/mindsdb/pull/9121
* updated handler link by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9175
* Added the Troubleshooting Guide for the MindsDB Inference Handler by MinuraPunchihewa in https://github.com/mindsdb/mindsdb/pull/9171
* MindsDB Inference - Fixed Validation of API Keys on Model/Engine Creation by MinuraPunchihewa in https://github.com/mindsdb/mindsdb/pull/9174
* AltibaseHandler improvements by colehaanlee in https://github.com/mindsdb/mindsdb/pull/9159
* python sdk docs for agents and jobs by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9168
* Snowflake Integration Improvements by MinuraPunchihewa in https://github.com/mindsdb/mindsdb/pull/9139
* Remove unused files by ZoranPandovski in https://github.com/mindsdb/mindsdb/pull/9002
* added feedback question to all docs by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9182
* Prepare Agents for Release by tmichaeldb in https://github.com/mindsdb/mindsdb/pull/9172
* Fix concurrency settings by hamishfagg in https://github.com/mindsdb/mindsdb/pull/9177
* updated mongo syntax by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9183
* removing agents from python sdk docs by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9184
* Fixed Authentication Errors Raised by Anyscale Handler by MinuraPunchihewa in https://github.com/mindsdb/mindsdb/pull/9191
* Fix pydantic warnings by hamishfagg in https://github.com/mindsdb/mindsdb/pull/9186
* Improve caching of docker builds by hamishfagg in https://github.com/mindsdb/mindsdb/pull/9194
* Twelvelabs fix by ZoranPandovski in https://github.com/mindsdb/mindsdb/pull/9199
* Fix extras arg in docker builds by hamishfagg in https://github.com/mindsdb/mindsdb/pull/9202
* updates to sql api docs by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9200
* removed rest api for agents by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9201
* Pydantic V2 migration by hamishfagg in https://github.com/mindsdb/mindsdb/pull/9193
* unindexed agents docs for rest api and python sdk by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9206
* updated automated finetuning tutorials by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9207
* Verify web by ZoranPandovski in https://github.com/mindsdb/mindsdb/pull/9190
* enhanced user journey through docs by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9204
* updated mql tutorials by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9214
* Hide sensitive connection args by StpMax in https://github.com/mindsdb/mindsdb/pull/9192
* bump lightwood version by hamishfagg in https://github.com/mindsdb/mindsdb/pull/9212
* Anyscale Endpoints Integration Improvements by MinuraPunchihewa in https://github.com/mindsdb/mindsdb/pull/9205
* Update docker-compose.yml by ZoranPandovski in https://github.com/mindsdb/mindsdb/pull/9213
* Minor updates to contributors docs by ZoranPandovski in https://github.com/mindsdb/mindsdb/pull/9224
* Project as database for create/retrain/finetune by ea-rus in https://github.com/mindsdb/mindsdb/pull/9135
* Access All Tables of Given Database Only for SQL Completions (fix) by tmichaeldb in https://github.com/mindsdb/mindsdb/pull/9223
* Upgrade Werkzeug to 2.3.8 by ZoranPandovski in https://github.com/mindsdb/mindsdb/pull/9217
* Upgrade mintlify in docs by ZoranPandovski in https://github.com/mindsdb/mindsdb/pull/9230
* Upgrade llama-hub version by ZoranPandovski in https://github.com/mindsdb/mindsdb/pull/9229
* Model Validation Fixes for Twelve Labs and ChromaDB by MinuraPunchihewa in https://github.com/mindsdb/mindsdb/pull/9228
* updated trigger docs by martyna-mindsdb in https://github.com/mindsdb/mindsdb/pull/9219
* Fixed Auth Mechanism for ScyllaDB/Cassandra by MinuraPunchihewa in https://github.com/mindsdb/mindsdb/pull/9231
* fix wrong `sensitive_info` arg by StpMax in https://github.com/mindsdb/mindsdb/pull/9233
* New release by ZoranPandovski in https://github.com/mindsdb/mindsdb/pull/9235
* Fixed OpenAI Conersational Model by MinuraPunchihewa in https://github.com/mindsdb/mindsdb/pull/9227
* Add gpt-4o by ZoranPandovski in https://github.com/mindsdb/mindsdb/pull/9211
* Fix tests by ZoranPandovski in https://github.com/mindsdb/mindsdb/pull/9236
New Contributors
* vlad-ro made their first contribution in https://github.com/mindsdb/mindsdb/pull/9011
* ArnavK-09 made their first contribution in https://github.com/mindsdb/mindsdb/pull/9016
* colehaanlee made their first contribution in https://github.com/mindsdb/mindsdb/pull/9159
**Full Changelog**: https://github.com/mindsdb/mindsdb/compare/v24.4.2.0...v24.5.4.0