Marqo

New features

- Hybrid Search for unstructured indexes (`"searchMethod": "HYBRID”`) ([912](https://github.com/marqo-ai/marqo/pull/912)). Marqo now supports hybrid search for unstructured indexes, combining lexical and tensor search (e.g., using reciprocal rank fusion - RRF) to provide the best relevance possible. See usage [here](https://docs.marqo.ai/2.11/API-Reference/Search/search/#example-2-creating-and-searching-an-unstructured-index-hybrid-search-with-model-deployed-within-marqo). Please note that hybrid search only works on a fresh Marqo 2.11.0 instance without state transfer for now. This is a limitation that we will address in the next release.
- [Marqo Terraform provider](https://github.com/marqo-ai/terraform-provider-marqo) is now available on both [OpenTofu Registry](https://github.com/opentofu/registry/blob/main/providers/m/marqo-ai/marqo.json) and [Terraform Registry](https://registry.terraform.io/providers/marqo-ai/marqo/latest). See usage [here](https://docs.marqo.ai/2.11/Cloud-Reference/opentofu_provider/)

Bug fixes and minor changes

- Improve the error handling of batch add/update/get documents API ([911](https://github.com/marqo-ai/marqo/pull/911)). Now each document in a batch request has its individual response status with detailed error message. See details [here](https://docs.marqo.ai/2.11/API-Reference/Documents/add_or_replace_documents/#response)
- Fix incorrect or missing prefixes for some models in the registry ([917](https://github.com/marqo-ai/marqo/pull/917)). This change improves all BGE models, all Snowflake models, and multilingual-e5-large-instruct. For example, `snowflake-arctic-embed-l` model has 34% improvement in NDCG10 on the Arguana benchmark and 153% improvement in NDCG10 on the FIQA benchmark.
- Increase the maxHits and maxOffset limit to 1,000,000 in the default query profile. ([914](https://github.com/marqo-ai/marqo/pull/914)). This allows user to override `MARQO_MAX_SEARCH_LIMIT` and `MARQO_MAX_SEARCH_OFFSET` environment variables to large values up to one million. Please note that this is an advanced setting and very large values aren’t normally recommended.
- Fix a bug that causes 400 error when using hybrid search with LEXICAL retrieval method and TENSOR ranking method and `scoreModifiersLexical` ([922](https://github.com/marqo-ai/marqo/pull/922)).

Contributor shout-outs

- Huge shoutout to all our 4.4k stargazers! We’ve come a long way as a team and as a community, so a huge thanks to everyone who continues to support Marqo.
- Feel free to keep on sharing questions and feedback on our [forum](https://community.marqo.ai/) and [Slack channel](https://marqo-community.slack.com/join/shared_invite/zt-2b4nsvbd2-TDf8agPszzWH5hYKBMIgDA#/shared-invite/email)! If you have any more inquiries or thoughts, please don’t hesitate to reach out.

Bug fixes and minor changes
- Fix an issue where CUDA was not automatically selected as the default device for the `embed` endpoint, even when available [941](https://github.com/marqo-ai/marqo/pull/941).

Bug fixes and minor changes
- Improve the clarity of the error message when Marqo can not download the provided image ([905](https://github.com/marqo-ai/marqo/pull/905)).
- Improve the error message in hybrid search to avoid confusion ([900](https://github.com/marqo-ai/marqo/pull/900)).
- Fix a bug where a `500` error is returned when an unsupported search method is provided. Marqo now correctly returns a `400` error ([899](https://github.com/marqo-ai/marqo/pull/899)).
- Fix a bug where a `500` error is returned when an invalid image URL with non-ASCII characters is provided. Marqo now encodes the image URL correctly ([908](https://github.com/marqo-ai/marqo/pull/908)).

New features

- Hybrid Search (`"searchMethod": "HYBRID”`) (https://github.com/marqo-ai/marqo/pull/845). Marqo now supports hybrid search, combining lexical and tensor search (using reciprocal rank fusion) to provide the best relevance possible. See usage [here](https://docs.marqo.ai/2.10/API-Reference/Search/search/#hybrid-parameters).
- Lexical Search score modifiers (https://github.com/marqo-ai/marqo/pull/884). Score modifiers are now supported for lexical search. Score modifiers are applied on all matches, not just the top k retrieved, resulting in more relevant hits. See usage [here](https://docs.marqo.ai/2.10/API-Reference/Search/search/#score-modifiers).

Bug fixes and minor changes

- Increase unstructured index default `filterStringMaxLength` to 50, from 20 (https://github.com/marqo-ai/marqo/pull/887). Maximum length of string fields to be used in query filters now defaults to 50 characters long.

Contributor shout-outs

- Huge shoutout to all our 4.3k stargazers! We’ve come a long way as a team and as a community, so a huge thanks to everyone who continues to support Marqo.
- Feel free to keep on sharing questions and feedback on our [forum](https://community.marqo.ai/) and [Slack channel](https://marqo-community.slack.com/join/shared_invite/zt-2b4nsvbd2-TDf8agPszzWH5hYKBMIgDA#/shared-invite/email)! If you have any more inquiries or thoughts, please don’t hesitate to reach out.

New features
- Numeric map data type. Add numeric map data types, available for filtering and score modification ([851](https://github.com/marqo-ai/marqo/pull/851)). You can now store a map/dictionary of numeric value and use these in your filters and score modifiers, or simply retrieve these with your documents. See the new types [here](https://docs.marqo.ai/2.9/API-Reference/Indexes/create_structured_index/?h=struct#fields). For usage in search, see [here](https://docs.marqo.ai/2.9/API-Reference/Search/search/?h=map#example_4). This is supported only for indexes created with Marqo 2.9 or later.
- Double and long score modifier fields. Support double and long in map and standard numeric fields for score modifiers in both structured and unstructured indexes ([851](https://github.com/marqo-ai/marqo/pull/851)). You can now use double values with full precision as score modifiers, as well as integers with guaranteed precision up to `2^53 - 1` (increased from `2^24 - 1`), with only negligible precision loss for larger values. For details on these new types, see the documentation [here](https://docs.marqo.ai/2.9/API-Reference/Indexes/create_structured_index/?h=struct#fields).

Bug fixes and minor changes
- Fix the bug in score modifiers where missing score modifiers in docs used in `multiply_score_by` lead to the multiplication of scores by `0` instead of by `1` ([851](https://github.com/marqo-ai/marqo/pull/851)).
- Improve upgrade stability ([874](https://github.com/marqo-ai/marqo/pull/874)). Fix failure of state transfer between some versions of Marqo due to Vespa binaries being copied with state. For more information, see the documentation [here](https://docs.marqo.ai/2.9/Guides/Advanced-Usage/transferring_state/?h=transfer)
- Improve the model warmup strategy on instances with CUDA ([877](https://github.com/marqo-ai/marqo/pull/877)). Marqo now requires less memory to warmup the models when spinning up .
- Improve create/delete index resilience to partial failures ([866](https://github.com/marqo-ai/marqo/pull/866)). You can now bring Marqo to a consistent state by repeating the operation until getting a `200` response.

Contributor shout-outs
- Shoutouts to our valuable 4.3k stargazers!
- Thanks a lot for the discussion and suggestions in our community. We love to hear your thoughts and requests. Join our [Slack channel](https://join.slack.com/t/marqo-community/shared_invite/zt-2jm456s90-1pFxdE5kDQt5imqddXUIcw) and [forum](https://community.marqo.ai/) now.

Bug fixes and minor changes
- Fix an issue in Marqo where loading some models (e.g., `open_clip/xlm-roberta-base-ViT-B-32/laion5b_s13b_b90k`) is unsuccessful.
This was resolved by upgrading the `transformers` and `optimum` packages. ([868](https://github.com/marqo-ai/marqo/pull/868))