Marqo

New features
* Significant queries-per-second (QPS) and latency improvements in addition to reduced memory and storage requirements.
Get a higher QPS and a lower latency for the same infrastructure cost, or get the same performance for much cheaper!
In our large-scale experiments, we have achieved 2x QPS improvement, 2x speed-up in P50 search latency and 2.3x
speed-up in P99 search latency, compared to previous Marqo versions.
* Significantly improved recall. You can now get up to 99% recall (depending on your dataset and configuration) without
sacrificing performance.
* Support for bfloat16 numeric type. Index 2x more vectors with the same amount of memory for a minimal reduction in
recall and performance.
* Structured index. You can now create structured indexes, which provide better data validation, higher performance,
better recall and better memory efficiency.
* New API search parameter `efSearch`. Search API now accepts an optional `efSearch` parameter which allows you to
fine-tune the underlying HNSW search. Increase efSearch to improve recall at a minor cost of QPS and latency. See
[here](https://docs.marqo.ai/2.0.0/API-Reference/Search/search/#body) for usage.
* Exact nearest neighbour search. Set `"approximate": false` in the Search API body to perform an exact nearest
neighbour search. This is useful for calculating recall and finding the best `efSearch` for your dataset.
See [here](https://docs.marqo.ai/2.0.0/API-Reference/Search/search/#body) for usage.
* New approximate nearest neighbour space types. Marqo now supports `euclidean`, `angular`, `dotproduct`,
`prenormalized-angular`, and `hamming` distance metrics. L1, L2 and Linf distance metrics are no longer supported.
The distance metric determines how Marqo calculates the closeness between indexed documents and search queries.
* Easier local runs. Simply run `docker run -p 8882:8882 marqoai/marqo:2.0.0` to start Marqo locally on both
ARM64 (M-series Macs) and AMD64 machines.

Breaking changes
* Create index API no longer accept the `index_defaults` parameter. Attributes previously defined in this object,
like `textPreprocessing`, are now moved out to the top level settings object.
See [here](https://docs.marqo.ai/2.0.0/API-Reference/Indexes/create_index/) for details.
* Create index API's `filterStringMaxLength` parameter determines the maximum length of strings that are indexed for
filtering (default value 20 characters). This limitation does not apply to structured indexes.
See [here](https://docs.marqo.ai/2.0.0/API-Reference/Indexes/create_index/) for details.
* Most APIs now require camel case request bodies and return camel case responses. See
[create index](https://docs.marqo.ai/2.0.0/API-Reference/Indexes/create_index/),
[search](https://docs.marqo.ai/2.0.0/API-Reference/Search/search/) and
[add documents](https://docs.marqo.ai/2.0.0/API-Reference/Documents/add_or_replace_documents/) for a few examples.
* New Marqo configuration parameters See [here](https://docs.marqo.ai/2.0.0/Guides/Advanced-Usage/configuration/) for
usage.
* Search response `_highlights` attribute is now a list of dictionaries.
See [here](https://docs.marqo.ai/2.0.0/API-Reference/Search/search/#response-200-ok) for new usage.
* Add documents multimodal fields are defined as normal fields and not dictionaries. Furthermore, the mappings object
is optional for structured indexes. See [here](https://docs.marqo.ai/2.0.0/API-Reference/Documents/add_or_replace_documents/) for usage.
* Add documents does not accept the `refresh` parameter anymore.
* The following features are available in Marqo 1.5, but are not supported by Marqo 2.0 and will be added in future
releases:
* Separate models for search and add documents
* Prefixes for text chunks and queries
* Configurable document count limit for add documents. There is a non-configurable limit of 128 in Marqo 2.0.
* Custom (externally generated) vectors and `no_model` option for index creation.
* Optional Search API `q` parameter when searching with context vectors.

Contributor shout-outs
* Thank you to the community for your invaluable feedback, which drove the prioritisation for this major release.
* A warm thank you to all our 3.9k stargazers.

Bug fixes and minor changes
- Adding `no_model` to `MARQO_MODELS_TO_PRELOAD` no longer causes an error on startup. Preloading process is simply skipped for this model [657](https://github.com/marqo-ai/marqo/pull/657).

New Features
- Separate model for search and add documents (https://github.com/marqo-ai/marqo/pull/633). Using the `search_model` and `search_model_properties` key in `index_defaults` allows you to specify a model specifically to be used for searching. This is useful for using a different model for search than what is used for add_documents. Learn how to use `search_model` [here](https://docs.marqo.ai/1.5.0/API-Reference/Indexes/create_index/#search-model).
- Prefixes for text chunks and queries enabled to improve retrieval for specific models (https://github.com/marqo-ai/marqo/pull/643). These prefixes are defined at the `model_properties` level, but can be overriden at index creation, add documents, or search time. Learn how to use prefixes for `add_documents` [here](https://docs.marqo.ai/1.5.0/API-Reference/Documents/add_or_replace_documents/#text-chunk-prefix) and `search` [here](https://docs.marqo.ai/1.5.0/API-Reference/Search/search/#text-query-prefix).

Bug fixes and minor changes
- Upgraded `open_clip_torch`, `timm`, and `safetensors` for access to new models (https://github.com/marqo-ai/marqo/pull/646).
- Documents containing multimodal objects that encounter errors in processing are rejected with a 400 error (https://github.com/marqo-ai/marqo/pull/631).
- Updated README: More detailed explanations, fixed formatting issues (https://github.com/marqo-ai/marqo/pull/629/files, https://github.com/marqo-ai/marqo/pull/642/files).

Contributor shout-outs
- A huge thank you to all our 3.7k stargazers!
- Thanks everyone for continuing to participate in our [forum](https://community.marqo.ai/)! Keep all your insights, questions, and feedback coming!

Breaking Changes
- Configurable document count limit for `add_documents()` calls (https://github.com/marqo-ai/marqo/pull/592). This mitigates Marqo getting overloaded
due to add_documents requests with a very high number of documents. If you are adding documents in batches larger than the default (64), you will now
receive an error. You can ensure your add_documents request complies to this limit by setting the Python client’s `client_batch_size` or changing this
limit via the `MARQO_MAX_ADD_DOCS_COUNT` variable. Read more on configuring the doc count limit [here](https://marqo.pages.dev/1.4.0/Guides/Advanced-Usage/configuration/#configuring-usage-limits).
- Default `refresh` value for `add_documents()` and `delete_documents()` set to `false` (https://github.com/marqo-ai/marqo/pull/601). This prevents
unnecessary refreshes, which can negatively impact search and add_documents performance, especially for applications that are
constantly adding or deleting documents. If you search or get documents immediately after adding or deleting documents, you may still get some extra
or missing documents. To see results of these operations more immediately, simply set the `refresh` parameter to `true`. Read more on this parameter
[here](https://marqo.pages.dev/1.4.0/API-Reference/Documents/add_or_replace_documents/#query-parameters).

New Features
- Custom vector field type added (https://github.com/marqo-ai/marqo/pull/610). You can now add externally generated vectors to Marqo documents! See
usage [here](https://marqo.pages.dev/1.4.0/Guides/Advanced-Usage/document_fields/#custom-vector-object).
- `no_model` option added for index creation (https://github.com/marqo-ai/marqo/pull/617). This allows for indexes that do no vectorisation, providing
easy use of custom vectors with no risk of accidentally mixing them up with Marqo-generated vectors. See usage [here](https://marqo.pages.dev/1.4.0/API-Reference/Indexes/create_index/#no-model).
- The search endpoint's `q` parameter is now optional if `context` vectors are provided. (https://github.com/marqo-ai/marqo/pull/617). This is
particularly useful when using context vectors to search across your documents that have custom vector fields. See usage [here](https://marqo.pages.dev/1.4.0/API-Reference/Search/search/#context).
- Configurable retries added to backend requests (https://github.com/marqo-ai/marqo/pull/623). This makes `add_documents()` and `search()` requests
more resilient to transient network errors. Use with caution, as retries in Marqo will change the consistency guarantees for these endpoints. For more
control over retry error handling, you can leave retry attempts at the default value (0) and implement your own backend communication error handling.
See retry configuration instructions and how it impacts these endpoints' behaviour [here](https://marqo.pages.dev/1.4.0/Guides/Advanced-Usage/configuration/#configuring-marqo-os-request-retries).
- More informative `delete_documents()` response (https://github.com/marqo-ai/marqo/pull/619). The response object now includes a list of document
ids, status codes, and results (success or reason for failure). See delete documents usage [here](https://marqo.pages.dev/1.4.0/API-Reference/Documents/delete_documents/).
- Friendlier startup experience (https://github.com/marqo-ai/marqo/pull/600). Startup output has been condensed, with unhelpful log messages removed.
More detailed logs can be accessed by setting `MARQO_LOG_LEVEL` to `debug`.

Bug fixes and minor changes
- Updated README: added Haystack integration, tips, and fixed links (https://github.com/marqo-ai/marqo/pull/593, https://github.com/marqo-ai/marqo/pull/602, https://github.com/marqo-ai/marqo/pull/616).
- Stabilized test suite by adding score modifiers search tests (​​https://github.com/marqo-ai/marqo/pull/596) and migrating test images to S3 (https://github.com/marqo-ai/marqo/pull/594).
- `bulk` added as an illegal index name (https://github.com/marqo-ai/marqo/pull/598). This prevents conflicts with the `/bulk` endpoint.
- Unnecessary `reputation` field removed from backend call (https://github.com/marqo-ai/marqo/pull/609).
- Fixed typo in error message (https://github.com/marqo-ai/marqo/pull/615).

Contributor shout-outs
- A huge thank you to all our 3.7k stargazers!
- Shoutout to TuanaCelik for helping out with the Haystack integration!
- Thanks everyone for keeping our [forum](https://community.marqo.ai/) busy. Don't hesitate to keep posting your insights, questions, and feedback!

New features

- New E5 models added to model registry (https://github.com/marqo-ai/marqo/pull/568). E5 V2 and Multilingual E5 models are now available for use. The new E5 V2 models outperform their E5 counterparts in the BEIR benchmark, as seen [here](https://github.com/microsoft/unilm/tree/master/e5#english-pre-trained-models). See all available models [here](https://marqo.pages.dev/1.2.0/Models-Reference/dense_retrieval/).
- Dockerfile optimisation (https://github.com/marqo-ai/marqo/pull/569). A pre-built Marqo base image results in reduced image layers and increased build speed, meaning neater docker pulls and an overall better development experience.


Bug fixes and minor changes

- Major README overhaul (https://github.com/marqo-ai/marqo/pull/573). The README has been revamped with up-to-date examples and easier to follow instructions.
- New security policy (https://github.com/marqo-ai/marqo/pull/574).
- Improved testing pipeline (https://github.com/marqo-ai/marqo/pull/582 & https://github.com/marqo-ai/marqo/pull/586). Tests now trigger on pull request updates. This results in safer and easier merges to mainline.
- Updated requirements files. Now the `requirements.dev.txt` should be used to install requirements for development environments (https://github.com/marqo-ai/marqo/pull/569). Version pins for `protobuf` & `onnx` have been removed while a version pin for `anyio` has been added (https://github.com/marqo-ai/marqo/pull/581, & https://github.com/marqo-ai/marqo/pull/589).
- General readability improvements (https://github.com/marqo-ai/marqo/pull/577, https://github.com/marqo-ai/marqo/pull/578, https://github.com/marqo-ai/marqo/pull/587, & https://github.com/marqo-ai/marqo/pull/580)

Contributor shout-outs

- A huge thank you to all our 3.5k stargazers!
- Shoutout to vladdoster for all the useful spelling and grammar edits!
- Thanks everyone for keeping our [forum](https://community.marqo.ai/) bustling. Don't hesitate to keep posting your insights, questions, and feedback!

New features

- Storage status in health check endpoint (https://github.com/marqo-ai/marqo/pull/555 & https://github.com/marqo-ai/marqo/pull/559). The `GET /indexes/{index-name}/health` endpoint's `backend` object will now return the boolean `storage_is_available`, to indicate if there is remaining storage space. If space is not available, health status will now return `yellow`. See [here](https://marqo.pages.dev/1.2.0/API-Reference/health/) for detailed usage.

- Score Modifiers search optimization (https://github.com/marqo-ai/marqo/pull/566). This optimization reduces latency for searches with the `score_modifiers` parameter when field names or weights are changed. See [here](https://marqo.pages.dev/1.2.0/API-Reference/search/#score-modifiers) for detailed usage.

Bug fixes and minor changes

- Improved error message for full storage (https://github.com/marqo-ai/marqo/pull/555 & https://github.com/marqo-ai/marqo/pull/559). When storage is full, Marqo will return `400 Bad Request` instead of `429 Too Many Requests`.
- Searching with a zero vector now returns an empty list instead of an internal error (https://github.com/marqo-ai/marqo/pull/562).

Contributor shout-outs

- A huge thank you to all our 3.3k stargazers!
- Thank you for all the continued discussion in our [forum](https://community.marqo.ai/). Keep all the insights, questions, and feedback coming!