Inference

🔨 Fixed

New VLM as Classifier `Workflows` block had bug - multi-label classification results were generated with "class_name" instead of "class" field in prediction details: https://github.com/roboflow/inference/pull/637

🌱 Changed

* Increase timeout to 30 minutes for .github/workflows/test_package_install_inference_with_extras.yml by grzegorz-roboflow in https://github.com/roboflow/inference/pull/635

**Full Changelog**: https://github.com/roboflow/inference/compare/v0.18.0...v0.18.1

🚀 Added

💪 New VLMs in `Workflows`

We've shipped blocks to integrate with [**Google Gemini**](https://gemini.google.com/) and [**Anthropic Claude**](https://www.anthropic.com/claude), but that's not everything! OpenAI block got updated. New "VLM Interface" of the block assumes that it can be prompted using pre-configured options and model output can be processed by set of formatter blocs to achieve desired end. It is now possible to:
* use `classification` prompting in VLM block and apply `VLM as Classifier` block to turn output string into classification result and process further using other blocks from ecosystem
* the same can be achieved for `object-detection` prompting and `VLM as Detector` block, which converts text produced by model into `sv.Detections(...)`

From now one, VLMs are much easier to integrate.

🧑‍🦱 USE CASE: PII protection when prompting VLM
Detect faces first, apply blur prediction visualisation and ask VLMs to tell what is the person eye colour - they won't be able to tell 🙃
<table>
<tr>
<td width="40%"> <img src="https://github.com/user-attachments/assets/f33873a8-cde4-4249-83c8-64d16dd5912e" /></td>
<td> <img src="https://github.com/user-attachments/assets/bec0c964-7b51-44e7-b7e0-144bca1c68a7" /></td>
</tr>
</table>

👨‍🎨 USE CASE: VLM as object detection model

<table>
<tr>
<td width="40%"> <img src="https://github.com/user-attachments/assets/a1f0686f-3a62-43cd-9ea0-e7cd6e3d0892" /></td>
<td> <img src="https://github.com/user-attachments/assets/1915ee9c-ce4d-4d20-892c-b7740c8d254b" /></td>
</tr>
</table>

👓 USE CASE: VLM as secondary classifier

Turn VLM output into classification results and process using downstream blocks - here we ask Gemini to classify crops of dogs to tell what is dog's breed - then we extract top class as property.


🤯 `Workflows` previews in documentation 📖

Thanks to joaomarcoscrs we can embed Workflows into documentation pages. Just take a [look](https://inference.roboflow.com/workflows/create_and_run/) how amazing it is ❗

<div align="center">
<img src="https://github.com/user-attachments/assets/c002d105-9324-43c2-aafb-6baab72f48fd" width="60%" />
</div>



🌱 Changed
* E2E tests for `workflows` on hosted platform by PawelPeczek-Roboflow in https://github.com/roboflow/inference/pull/622
* Allow model type "yolov8" without size by SolomonLake in https://github.com/roboflow/inference/pull/627
* Fix `GDAL` issue in GHA by PawelPeczek-Roboflow in https://github.com/roboflow/inference/pull/628
* Add support for DELETE in sqlite wrapper by grzegorz-roboflow in https://github.com/roboflow/inference/pull/631
* Keep distinct exec sessions for inf pipeline usage by grzegorz-roboflow in https://github.com/roboflow/inference/pull/632
* Insert usage payloads into redis and to sorted set atomically by grzegorz-roboflow in https://github.com/roboflow/inference/pull/633
* remove height from workflow example by capjamesg in https://github.com/roboflow/inference/pull/634


❗ **BREAKING** ❗ `Batch[X]` kinds removed from Workflows

What was changed and why?

In `inference` release `0.18.0` we decided to make drastic move to heal the ecosystem from the problem with ambiguous kinds names (`Batch[X]` vs `X` - see more [here](https://github.com/roboflow/inference/issues/608)).

The change is breaking only for non-Roboflow Workflow plugins depending on imports from `inference.core.workflows.execution_engine.entities.types` module. **To the best of our knowledge, there is no such plugin**.

The change **is not breaking** in terms of running Workflows on Roboflow platform and on-prem given that external plugins were not used.

Migration guide

Migration should be relatively easy - in the code of a Workflow block, all instances of
python
from inference.core.workflows.execution_engine.entities.types import BATCH_OF_{{KIND_NAME}}

should be replaced with
python
from inference.core.workflows.execution_engine.entities.types import {{KIND_NAME}}


**PR with changes as reference:** https://github.com/roboflow/inference/pull/618


**Full Changelog**: https://github.com/roboflow/inference/compare/v0.17.1...v0.18.0

❗IMPORTANT ❗Security issue in `opencv-python`

This PR provides fix for the following security issue:

opencv-python versions before v4.8.1.78 bundled libwebp binaries in wheels that are vulnerable to https://github.com/advisories/GHSA-j7hp-h8jx-5ppr. opencv-python v4.8.1.78 upgrades the bundled libwebp binary to v1.3.2.

We advise all clients using `inference` to migrate, especially in production environments.

**Full Changelog**: https://github.com/roboflow/inference/compare/v0.17.0...v0.17.1

🚀 Added

💪 More Classical Computer Vision blocks in `workflows`

**Good news for the fans of classical computer vision!**
We heard you – and we’ve added a bunch of new blocks to enhance your workflows.

Basic operations on images
<table>
<tr>
<th>Workflow Definition</th>
<th>Preview</th>
</td>
<tr>
<td width="70%"><img src="https://github.com/user-attachments/assets/23fa85a6-a019-4679-ab08-3690dabca1f0" /></td>
<td><img src="https://github.com/user-attachments/assets/8c10e498-031d-4964-beb8-64dc8117ecc1" /></td>
</tr>
</table>

Camera focus check
<table>
<tr>
<th>Workflow Definition</th>
<th>Preview</th>
</td>
<tr>
<td width="40%"><img src="https://github.com/user-attachments/assets/ff3e64d2-46bf-44d9-b813-aaf4fbc11972" /></td>
<td><img src="https://github.com/user-attachments/assets/681e2f01-553d-4a34-9128-5349c7229944" /></td>
</tr>
</table>

🚀 Upgrade of `CLIP Comparison` and `Roboflow Dataset Upload` blocks

We’ve made it even more versatile. The new outputs allow seamless integration with many other blocks, enabling powerful workflows like:

**detection → crop → CLIP classification (on crops) → detection class replacement**

Get ready to streamline your processes with enhanced compatibility and new possibilities!


For **`Roboflow Dataset Upload v2`** there is now possibility to sample percentage of data to upload and we changed the default sizes of saved images to be bigger.

❗ **Do not worry!** All your old Workflows using mentioned blocks are not affected with the change thanks to versioning 😄


💥 New version of [📖 Workflow docs](https://inference.roboflow.com/workflows/about/) 🔥

<div align="center"><img src="https://github.com/user-attachments/assets/cc0ce88c-e7dc-49cf-8e16-62420f62f2c8" width="80%" /></div>

The Wait is Over – Our Workflows Documentation is Finally Here!

We’ve revamped and expanded the documentation to make your experience smoother. It’s now organized into three clear sections:
* **General Overview:** Perfect for getting you up and running quickly.
* **Mid-Level User Guide:** Gain a solid understanding of the ecosystem without diving too deep into the technical details.
* **Detailed Developer Guide:** Designed for contributors, packed with everything you need to develop within the ecosystem.

Check it out and let us know what you think of the new docs!

🌱 Changed
* Record resource details in each usage payload by grzegorz-roboflow in https://github.com/roboflow/inference/pull/607
* json.dumps resource_details when adding to usage payload by grzegorz-roboflow in https://github.com/roboflow/inference/pull/610
* enhancement in CLIP docs by venkatram-dev in https://github.com/roboflow/inference/pull/599 - thanks for contribution 🏅
* Sam2 multi polygons by probicheaux in https://github.com/roboflow/inference/pull/593
* Abstract sqlite3 wrapper in usage collector sqlite queue by grzegorz-roboflow in https://github.com/roboflow/inference/pull/619

🔨 Fixed
* Dynamic Crop block was buggy in some contexts - see details: https://github.com/roboflow/inference/pull/604
* Bug in integration tests by PawelPeczek-Roboflow in https://github.com/roboflow/inference/pull/600
* Bugfix sam2 logits cache and add test by probicheaux in https://github.com/roboflow/inference/pull/606
* Fix bug with detections offset and cover problem with additional test by PawelPeczek-Roboflow in https://github.com/roboflow/inference/pull/611
* only warn about version when its lower than latest release by hansent in https://github.com/roboflow/inference/pull/609
* Add inference_ids to model blocks by robiscoding in https://github.com/roboflow/inference/pull/615
* Fix Detection Offset Bug by NickHerrig in https://github.com/roboflow/inference/pull/621
* Bump micromatch from 4.0.5 to 4.0.8 in /inference/landing by dependabot in https://github.com/roboflow/inference/pull/617

🏅 New Contributors
* reedajohns made their first contribution in https://github.com/roboflow/inference/pull/602
* venkatram-dev made their first contribution in https://github.com/roboflow/inference/pull/599

**Full Changelog**: https://github.com/roboflow/inference/compare/v0.16.3...v0.17.0

🔨 Fixed

* [Roboflow Dataset Upload](https://inference.roboflow.com/workflows/blocks/roboflow_dataset_upload/) block now can be dynamically configured with labelling batch prefixes passed as workflow inputs. Additionally `predictions` parameter is now optional - which means that you can upload images without predictions in your workflows - all thanks to NickHerrig https://github.com/roboflow/inference/pull/586
* chandlersupple fixed [Perspective Correction Block](https://inference.roboflow.com/workflows/blocks/perspective_correction/) in https://github.com/roboflow/inference/pull/589
* robiscoding added missing `inference_id` for classification responses in https://github.com/roboflow/inference/pull/591
* PacificDou in https://github.com/roboflow/inference/pull/592 introduced changes to `inference` server routing that were required to enhance [Dedicated Deployment](https://docs.roboflow.com/roboflow-cli/dedicated-deployments) functionality
* Added limits for execution time of GHA in https://github.com/roboflow/inference/pull/596


🚀 Added

SAM2 extension

While making inference from SAM2 model you may request `inference` package and `inference` server to cache prompts and low-resolution masks from your inputs to be re-used later on upon request. You are given two parameters (both in [SAM2 request](https://github.com/roboflow/inference/blob/main/inference/core/entities/requests/sam2.py#L125) payload and [`SegmentAnything2.segment_image(...)`](https://github.com/roboflow/inference/blob/main/inference/models/sam2/segment_anything2.py#L216) method:
* `save_logits_to_cache`
* `load_logits_from_cache`
which decide how the functionality should work. Saving logits masks to cache will make it possible, to re-use them for consecutive inferences agains the same image. Enabling loading triggers search through cache intended to find the most similar prompt cached for this specific image to retrieve its mask. The mechanism is useful when the same image is segmented multiple times with slightly different sets of prompts - as injecting previous masks in that scenario may lead to better results:

<table>
<tr>
<th>Before</th>
<th>After</th>
</td>
<tr>
<td><img src="https://github.com/user-attachments/assets/9c925e6d-16dd-453b-9e43-8fa290691e53" /></td>
<td><img src="https://github.com/user-attachments/assets/945d4507-054f-4e24-bfcf-6819b2b4db2e" /></td>
</tr>
</table>

Please note that **this feature is different than cache for image embeddings which speed consecutive requests with the same image up** and if you don't wish the feature to be enabled, set `DISABLE_SAM2_LOGITS_CACHE=True` in your env.

🏅 probicheaux and tonylampada added the functionality in https://github.com/roboflow/inference/pull/582

Remaining changes
* EmilyGavrilenko added Workflow block search metadata to improve UI experience in https://github.com/roboflow/inference/pull/588
* grzegorz-roboflow added internal parameter for workflows request denoting preview in UI https://github.com/roboflow/inference/pull/595
* grzegorz-roboflow improved usage tracking extending it to models in https://github.com/roboflow/inference/pull/601 and https://github.com/roboflow/inference/pull/548
* `workflows` equipped with new batch-oriented input - `VideoFrameMetadata` letting blocks to process videos statefully see https://github.com/roboflow/inference/pull/590, https://github.com/roboflow/inference/pull/597- more docs will come soon

**Full Changelog**: https://github.com/roboflow/inference/compare/v0.16.2...v0.16.3

🚀 Added
Segment Anything 2 in `workflows` 🥳

We prepared great amount of changes to `workflows`, could not really decide which update to start with, but at the end we found onboarding of [SAM 2 model](https://ai.meta.com/sam2/) as most exciting.

Thanks to hansent effort we have introduced SAM 2 `workflow` block.

You can use SAM2 standalone, or you can ground it's predictions with other detection models which is the true power of `workflows`. Thanks to grounding, you can generate instance segmentation masks for each bounding box predicted by your object detection model

<div align="center">
<img src="https://github.com/user-attachments/assets/6cc9d2c7-1fe5-4d7e-b788-b5c89f535afa" width="60%"/>
</div>

❗ We do not support SAM2 at Roboflow Hosted Platform yet, but it is possible to use `inference server start` command to run local server supporting SAM2 model and connect it to `workflows` UI to run examples.

`workflows` 🤝 SAHI

We've added set of blocks that let people apply [SAHI technique](https://blog.roboflow.com/how-to-use-sahi-to-detect-small-objects/) based on utilities provided by [`supervision`](https://github.com/roboflow/supervision).

We are going to work to simplify SAHI usage in `workflows` UI, but now you need to use three blocks to effectively apply technique:

<div align="center">
<img src="https://github.com/user-attachments/assets/5d0849be-717e-4256-8a31-c55d848081a9" width="60%"/>
</div>

Classical Computer Vision methods in `workflows` 🔥

We do not forget about old good friends - that's why we also added bunch of blocks with classical Computer Vision algorithms:
* Dominant Color block by NickHerrig in https://github.com/roboflow/inference/pull/578
* SIFT, SIFT matching, classical pattern matching and others by ryanjball in https://github.com/roboflow/inference/pull/581

🌱 Changed
* Added encoding='utf-8' to setup.py by Bhavay-2001 in https://github.com/roboflow/inference/pull/556
* Move landing static assets to `/static/` namespace by iurisilvio in https://github.com/roboflow/inference/pull/577
* add exclusion rules for dedicated deployment authorizer by PacificDou in https://github.com/roboflow/inference/pull/576
* Workflow private block properties by EmilyGavrilenko in https://github.com/roboflow/inference/pull/579


🔨 Fixed
* Fix security issues with landing page by PawelPeczek-Roboflow in https://github.com/roboflow/inference/pull/584
* Fixed a Regression in the Custom Metadata Block that was introduced in `v0.16.0` - PR with fix by chandlersupple (https://github.com/roboflow/inference/pull/573) - we kindly ask clients relying on the Custom Metadata Block, running their workflows on-prem to update `inference` or `inference` server
* Bug in `workflows` Execution Engine that was making it impossible to feed the same block with two identical selectors (fixed in https://github.com/roboflow/inference/pull/581)

❗ In release `0.16.0` we introduced bug impacting `workflows` and `inference_sdk`

The mistake was introduced in https://github.com/roboflow/inference/pull/565 and fixed in https://github.com/roboflow/inference/pull/585 (both by PawelPeczek-Roboflow 😢 ) and was causing issues with order of results - regarding specific `workflows` blocks:
* blocks with Roboflow models, whenever used with batch input (for instance when workflow was run against multiple images, or Dynamic Crop was used) were mismatching order of predictions with respect to order of images
* the same was true for OpenAI block and GPT-4V block
* the problem was also introduced into `inference_sdk`, so whenever client was called with multiple images - results may have been missmatched

**We advice all our clients to upgrade to new release and abandon usage `inference=0.16.0`**


🏅 New Contributors
* Bhavay-2001 made their first contribution in https://github.com/roboflow/inference/pull/556

**Full Changelog**: https://github.com/roboflow/inference/compare/v0.16.0...v0.16.1