Marqo

New features
- Add `IN` operator to the query filter string DSL (https://github.com/marqo-ai/marqo/pull/790, https://github.com/marqo-ai/marqo/pull/793, & https://github.com/marqo-ai/marqo/pull/795).
For structured indexes, you can now use the `IN` keyword to restrict text and integer fields to be within a list of values. See usage [here](https://docs.marqo.ai/2.4.0/Guides/query_dsl/#in-queries).

- Add `no_model` option for index creation (https://github.com/marqo-ai/marqo/pull/789). This allows for indexes that do no vectorisation,
providing easy use of custom vectors with no risk of accidentally mixing them up with Marqo-generated vectors. See usage [here](https://docs.marqo.ai/2.4.0/Guides/Models-Reference/list_of_models/#no-model).
- Optional `q` parameter for the search endpoint if context vectors are provided. (https://github.com/marqo-ai/marqo/pull/789).
This is particularly useful when using context vectors to search across your documents that have custom vector fields. See usage [here](https://docs.marqo.ai/2.4.0/API-Reference/Search/search/#query-q).

Bug fixes and minor changes
- Improve error message for defining `tensorFields` when adding documents to a structured index (https://github.com/marqo-ai/marqo/pull/788).

Contributor shout-outs
- A huge thank you to all our 4.1k stargazers! We appreciate all of you continuing to use our product and helping Marqo grow.
- Thanks for sharing your questions and feedback on our [forum](https://community.marqo.ai/) and
[Slack channel](https://marqo-community.slack.com/join/shared_invite/zt-2b4nsvbd2-TDf8agPszzWH5hYKBMIgDA#/shared-invite/email)!
If you have any more inquiries or thoughts, please don’t hesitate to reach out.

New features
- New `update_documents` API (https://github.com/marqo-ai/marqo/pull/773). Structured indexes now support high throughput partial updates to non-tensor fields. Unstructured indexes do not support partial updates. See usages [here](https://docs.marqo.ai/2.3.0/API-Reference/Documents/update_documents/)
- The custom vectors feature is now supported again for both structured and unstructured indexes (https://github.com/marqo-ai/marqo/pull/777). You can now add externally generated vectors to Marqo documents. See usages [here](https://docs.marqo.ai/2.3.0/API-Reference/Documents/add_or_replace_documents/#mappings)

Bug fixes and minor changes
- Fix an issue where non-default distance metrics are not configured correctly with unstructured indexes (https://github.com/marqo-ai/marqo/pull/772).
- Introduce a guide for running Marqo open source in production environments, offering insights and best practices (https://github.com/marqo-ai/marqo/pull/775).
- Remove outdated examples from the README to improve clarity and relevance (https://github.com/marqo-ai/marqo/pull/766).

Contributor shout-outs
- A huge thank you to all our 4k stargazers! This is a new milestone for Marqo!
- Stay connected and share your thoughts on our [forum](https://community.marqo.ai/) and [Slack channel](https://marqo-community.slack.com/join/shared_invite/zt-2b4nsvbd2-TDf8agPszzWH5hYKBMIgDA#/shared-invite/email)! Your insights, questions, and feedback are always welcome and highly appreciated.

Bug fixes and minor changes
- Fix response code for vector store timeout, change it from 429 to 504 (https://github.com/marqo-ai/marqo/pull/763)

New features
- Support filtering on document ID with structured indexes. This was already supported with unstructured indexes ([749](https://github.com/marqo-ai/marqo/pull/749))
- New structured index data types: `long`, `double`, `array<long>` and `array<double>` for a higher precision and range of values Available for indexes created with Marqo 2.2+ ([722](https://github.com/marqo-ai/marqo/pull/722))
- Higher precision numeric fields for unstructured indexes. Unstructured indexes created with Marqo 2.2+ will use double precision floats and longs for a higher precision and wider range of values ([722](https://github.com/marqo-ai/marqo/pull/722))
- Numeric value range validation. Values that are out of range for the field type will now receive a 400 validation error when adding documents. ([722](https://github.com/marqo-ai/marqo/pull/722))

Bug fixes and minor changes
- Fix unstructured index bug where filtering for boolean-like strings (e.g., `"true"`) would not work as expected ([709](https://github.com/marqo-ai/marqo/pull/709))
- Better handling of vector store timeouts. Marqo will now return a 429 (throttled) error message when the backend vector store is receiving more traffic than it can handle([758](https://github.com/marqo-ai/marqo/pull/758))
- Improved error logging. Stack trace will now always be logged ([745](https://github.com/marqo-ai/marqo/pull/745))
- Better API 500 error message. Marqo will no longer return verbose error messages in the API response ([751](https://github.com/marqo-ai/marqo/pull/751))
- Default index model is now hf/e5-base-v2 ([710](https://github.com/marqo-ai/marqo/pull/710))
- Improve error messages ([746](https://github.com/marqo-ai/marqo/pull/746), [#747](https://github.com/marqo-ai/marqo/pull/747))
- Improve error handling at startup when vector store is not ready. Marqo will now start and wait for vector store to become available ([752](https://github.com/marqo-ai/marqo/pull/752))

Contributor shout-outs
- A huge thank you to all our 3.9k stargazers!
- Thank you [Dmitri](https://marqo-community.slack.com/team/U06GL2R5NMT) for helping us identify the issue with running Marqo on older AMD64 processors!

New features
- Search result maximum limit and offset greatly increased. Maximum `limit` parameter increased from 400 to 1,000, `offset` increased from 1,000 to 10,000. Maximum value for `MARQO_MAX_RETRIEVABLE_DOCS` configuration is now 10,000 ([735](https://github.com/marqo-ai/marqo/pull/735)​​, [#737](https://github.com/marqo-ai/marqo/pull/737)​​). See search `limit` and `offset` usage [here](https://docs.marqo.ai/2.1.0/API-Reference/Search/search/#limit)

Bug fixes and minor changes
- Improved the Marqo bootstrapping process to address unexpected API behaviour when no index has been created yet (https://github.com/marqo-ai/marqo/pull/730).
- Improved validation for `create_index` settings (https://github.com/marqo-ai/marqo/pull/717, https://github.com/marqo-ai/marqo/pull/734). Using `dependent_fields` as a request body parameter will now raise a 400 validation error.
- Improved data parsing for documents in unstructured indexes (https://github.com/marqo-ai/marqo/pull/732).
- Made vector store layer config upgrades and rollbacks easier ([735](https://github.com/marqo-ai/marqo/pull/735)​​, [#736](https://github.com/marqo-ai/marqo/pull/736)​​).
- Readme improvements (https://github.com/marqo-ai/marqo/pull/729).

Bug fixes and minor changes
- Improved stability of `use_existing_tensors` feature in `add_documents` (https://github.com/marqo-ai/marqo/pull/725).
- Improved readability of Marqo start-up logs (https://github.com/marqo-ai/marqo/pull/719).
- Removed obsolete examples ([721](https://github.com/marqo-ai/marqo/pull/721), [#723](https://github.com/marqo-ai/marqo/pull/723)).