Inference

🚀 Added

🔥 YOLOv11 in `inference` 🔥

We’re excited to announce that YOLOv11 has been added to `inference`! 🚀 You can now use both `inference` and the `inference` server to get predictions from the latest YOLOv11 model. 🔥

All thanks to probicheaux and SolomonLake 🏅


https://github.com/user-attachments/assets/ed6aa0db-cb20-4bb9-bd67-a79930cc6baf

<details>
<summary>Try the model in<code>inference</code> Python package</summary>

python
import cv2
from inference import get_model

image = cv2.imread("<your-image>")
model = get_model("yolov11n-640")
predictions = model.infer(image)

print(predictions)


</details>

💪 Workflows update

Google Vision OCR in workflows

Thanks to **open source contribution** from brunopicinin we have Google Vision OCR integrated into workflow ecosystem. Great to see open source community contribution 🏅

https://github.com/user-attachments/assets/0acf8db4-2fba-4f5b-b99e-93bcbbdda8fd

See [📖 documentation](https://inference.roboflow.com/workflows/blocks/google_vision_ocr/) of the new block to explore it's capabilities.

Images stitch Workflow block

📷 Your camera is not able to cover the whole area you want to observe? Don't worry! grzegorz-roboflow just added the Workflow block which would be able to combine the POV of multiple cameras into a single image that can be further processed in your Workflow.

<table>
<tr><th>image 1</th><th>image 2</th><th>stitched image</th></tr>
<tr>
<td><img src="https://github.com/user-attachments/assets/f6d6954b-99ed-446f-a9e3-662afc7ff2e0" /> </td>
<td><img src="https://github.com/user-attachments/assets/087c22ff-c560-412e-9061-d972f7e65ef8" /> </td>
<td><img src="https://github.com/user-attachments/assets/eaff81ce-7b57-4cad-a214-b83a38f115da" /> </td>
</tr>
</table>

📏 Size measurement block

Thanks to chandlersupple, we can now measure actual size of objects with Workflows! Take a look at [📖 documentation](https://inference.roboflow.com/workflows/blocks/size_measurement/) to discover how the block works.


Workflows profiler and Execution Engine speedup 🏇

We've added Workflows Profiler - ecosystem extension to profile the execution of your workflow. It works for [`inference` server requests](https://inference.roboflow.com/inference_helpers/inference_sdk/#inference-workflows) (both self-hosted and on Roboflow platform) as well as for [`InferencePipeline`](https://inference.roboflow.com/using_inference/inference_pipeline/#inferencepipeline-and-roboflow-workflows).


The cool thing about profiler is that it is compatible with `chrome://tracing` - so you can easily grab profiler output and render it in Google Chrome browser.

To profile your Workflow execution use the following code snippet - traces are saved in `./inference_profiling` directory by default.

python
from inference_sdk import InferenceHTTPClient

client = InferenceHTTPClient(
api_url="https://detect.roboflow.com",
api_key="<YOUR-API-KEY>"
)
results = client.run_workflow(
workspace_name="<your-workspace>",
workflow_id="<your-workflow-id>",
images={
"image": "<YOUR-IMAGE>",
},
enable_profiling=True,
)


See detailed report regarding speed optimisations in the PR https://github.com/roboflow/inference/pull/710

❗ Important note
As part of speed optimisation we enabled server-side caching for workflows definitions saved on Roboflow Platform - if you frequently change and your Workflow, to see results immediately you need to specify `use_cache=False` parameter of `client.run_workflow(...)` method

🔧 Fixed
* Fix prometheus scraping by robiscoding in https://github.com/roboflow/inference/pull/712
* Fix the problem with VLMs on batch inference by PawelPeczek-Roboflow in https://github.com/roboflow/inference/pull/718

🌱 Changed
* Docker auto-reload configuration by EmilyGavrilenko in https://github.com/roboflow/inference/pull/703
* Multi-Label Classification UQL Operations by EmilyGavrilenko in https://github.com/roboflow/inference/pull/714
* Add port forward to Notebook Landing Page message by hansent in https://github.com/roboflow/inference/pull/711
* Add optional descriptions to dynamic blocks by EmilyGavrilenko in https://github.com/roboflow/inference/pull/702
* Add descriptions to task types in `VLM as Detector` by PawelPeczek-Roboflow in https://github.com/roboflow/inference/pull/704
* Add tests for Google Vision OCR by PawelPeczek-Roboflow in https://github.com/roboflow/inference/pull/715
* Improvements regarding custom python blocks by PawelPeczek-Roboflow in https://github.com/roboflow/inference/pull/716


🏅 New Contributors
We do want to honor brunopicinin who made their first contribution to `inference` in https://github.com/roboflow/inference/pull/709 as a part of Hacktoberfest 2024. We invite other open-source community members to contribute 😄

**Full Changelog**: https://github.com/roboflow/inference/compare/v0.21.1...v0.22.0

What's Changed
* Improvements in contribution guides by PawelPeczek-Roboflow in https://github.com/roboflow/inference/pull/691
* Fix issue with SAM2 producing segmentation masks as points and lines by PawelPeczek-Roboflow in https://github.com/roboflow/inference/pull/697


**Full Changelog**: https://github.com/roboflow/inference/compare/v0.21.0...v0.21.1

🚀 Added

👩‍🎨 Become an artist with Workflows 👨‍🎨

Ever wanted to be an artist but felt like you lacked the skills? No worries! We’ve just added the StabilityAI Inpainting block to the Workflows ecosystem. Now, you can effortlessly add whatever you envision into your images! 🌟🖼️

Credits to Fafruch for origin idea 💪

https://github.com/user-attachments/assets/c7a4c107-1fa8-46a7-a429-3b4fbe39184b

<details>
<summary>📖 docs</summary>

* Model [📖 docs](https://platform.stability.ai/docs/legacy/grpc-api/features/inpainting)
* Workflow block [📖 docs](https://inference.roboflow.com/workflows/blocks/stability_ai_inpainting/)

</details>

🤯 Workflows + video + `inference` server - Experimental feature preview 🔬

Imagine creating a Workflow in our UI, tuning it to understand what happens in your video. So far, video processing with `InferencePipeline` required a bit of glue code and setup in your environment. We hope that soon, you won’t need any custom scripts for video processing! You’ll be able to ship your Workflow directly to the `inference` server, just pointing which video source to process.

We're thrilled to announce that we’ve taken the first step toward making this idea a reality! Check out our experimental feature for video processing, now controlled by the inference server with a user-friendly REST API for easy integration.


https://github.com/user-attachments/assets/c1afa7d6-507a-48cb-93a8-c067bd9c9ccf


🔍 We encourage you to try it out! The feature is available in the inference server Docker images that you can self-host. Please note that this feature is experimental, and breaking changes are to be expected. Check out our [📖 docs](https://inference.roboflow.com/workflows/video_processing/overview/) to learn more.

🙃 Flips, Rotations and Resizing in Workflows

Tired of dealing with image orientation problems while building demos with Workflows? Whether it's resizing, rotating, or flipping, those headaches end today with our new features for seamless image adjustments!


All thanks to EmilyGavrilenko and PR https://github.com/roboflow/inference/pull/683
<div align="center"><img src="https://github.com/user-attachments/assets/445db635-75e8-4455-98f2-edf578e2d542" width="60%" /></div>

✨ Ensure Your Tracked Objects Stay on Course! 🛰️
Wondering if the objects you're tracking follow the path you intended? We’ve got you covered in Workflows! Thanks to shantanubala, we now offer **Fréchet Distance Analysis** as a Workflow block. Simply specify the desired trajectory, and Workflow calculates the deviation for each tracked box. 📊
See details: https://github.com/roboflow/inference/pull/682

> **What’s Fréchet Distance?**
> It’s a mathematical measure that compares the similarity between two curves—perfect for analyzing how closely your tracked objects follow the path you’ve set.
>

🆕 Background removal in Dynamic Crop and updated UI for VLMs

Let’s be honest—VLMs in Workflows still had room for improvement, especially when integrating model outputs with other blocks. Well, we've made it better! 🎉 Now, each model task comes with a clear description and a suggested parser to follow the block, helping you get the most out of your model predictions with ease. 🛠️

Additionally, you can now remove background while performing Dynamic Crop on Instance Segmentation model results 🤯

https://github.com/user-attachments/assets/25881769-4e6d-4ca1-8880-d15471c56f78




🔧 Fixed
* Fix lora by probicheaux in https://github.com/roboflow/inference/pull/676
* Add ability to parametrise class remap UQL operation by PawelPeczek-Roboflow in https://github.com/roboflow/inference/pull/674
* Fix license link in docs by NickHerrig in https://github.com/roboflow/inference/pull/680
* Add `transformers` extra to dev setup py by PawelPeczek-Roboflow in https://github.com/roboflow/inference/pull/678

🌱 Changed
* `gstreamer` backend in `inference` by PawelPeczek-Roboflow in https://github.com/roboflow/inference/pull/646
* Add workflow schema api docs by NickHerrig in https://github.com/roboflow/inference/pull/677
* Add Florence-2 [📖 documentation](https://inference.roboflow.com/foundation/florence2/) by capjamesg in https://github.com/roboflow/inference/pull/626

🏅 New Contributors
* shantanubala made their first contribution in https://github.com/roboflow/inference/pull/682

**Full Changelog**: https://github.com/roboflow/inference/compare/v0.20.1...v0.21.0

What's Changed
* Fix workflows gallery previews by PawelPeczek-Roboflow in https://github.com/roboflow/inference/pull/668
* Add option to reset timer when detection falls outside of zone by grzegorz-roboflow in https://github.com/roboflow/inference/pull/671
* Line counter improvements by EmilyGavrilenko in https://github.com/roboflow/inference/pull/670


**Full Changelog**: https://github.com/roboflow/inference/compare/v0.20.0...v0.20.1

🚀 Added

🌟 Florence 2 🤝 `Workflows`

Thanks to probicheaux, the Workflows ecosystem just got better with the addition of the Florence 2 block. Florence 2, one of the top open-source releases this year, is a powerful Visual Language Model capable of tasks like object detection, segmentation, image captioning, OCR, and more. Now, you can use it directly in your workflows!

Florence 2 and SAM 2 - zero shot grounded segmentation

Ever wished for precise segmentation but didn’t have the data to train your model? Now you don’t need it! With Florence 2 and SAM 2, you can achieve stunning segmentation results effortlessly — without a single annotation.

Discover how to combine these powerful models and get top-tier segmentation quality for free!

https://github.com/user-attachments/assets/965297e0-375f-4f0d-9448-2232a578542f

Florence 2 as OCR model

Need Text Layout Detection and OCR? Florence 2 Has You Covered!

https://github.com/user-attachments/assets/84bef0ae-66a9-41a1-81cc-f09dcbd09e37

Zero-shot object detection needed?

Do not hesitate and try out Florence 2 as object detection model - the quality of results is surprisingly good 🔥


https://github.com/user-attachments/assets/b08f54d3-53d1-4a58-9c80-5d95a6b99df1

🔔 Additional notes
* Florence 2 requires either Roboflow Dedicated Deployment or self-hosted `inference` server - it is not available on Roboflow Hosted Platform
* To discover full potential of Florence 2 - read the [paper](https://arxiv.org/pdf/2311.06242)
* Visit [📖 documentation](https://inference.roboflow.com/workflows/blocks/florence2_model/) of Florence 2 Workflow block

New version of SIFT block

Tired using SIFT descriptors calculation block followed by SIFT comparison? This is no longer needed. Check out [SIFT Comparison v2](https://inference.roboflow.com/workflows/blocks/sift_comparison/#version-v2) block. PR: https://github.com/roboflow/inference/pull/657
<div align="center"><img width="640" alt="sift_updated" src="https://github.com/user-attachments/assets/29c39ef1-dc60-479f-8c5d-f6efac84fd2b"></div>

Workflows UQL extended with new operations

You may not even be aware, but Universal Query Language powers Workflows operations that can be fully customised in UI. There are two new features shipped:
* selecting prediction by confidence by EmilyGavrilenko in https://github.com/roboflow/inference/pull/655
* class names remapping operation by NickHerrig in https://github.com/roboflow/inference/pull/656

Instance Segmentation ⏩ oriented rectangle
Thanks to chandlersupple, Instance Segmentation results can be turned into oriented bounding boxes - check out [📖 docs](https://inference.roboflow.com/workflows/blocks/bounding_rectangle/)

🔧 Fixed

* Broken links removed from docs in https://github.com/roboflow/inference/pull/663

🚀 Added

🎥 Video processing in `workflows` 🤯

We’re excited to announce that, thanks to the contributions of grzegorz-roboflow, our Workflows ecosystem now extends to video processing! Dive in and explore the new possibilities:

https://github.com/user-attachments/assets/b4778f79-7630-4393-8abd-7d3a5ba64f5c

New blocks:
* [Time In Zone](https://inference.roboflow.com/workflows/blocks/timeinzone/) to analyse dwell time
* [Line Counter](https://inference.roboflow.com/workflows/blocks/line_counter/) to detect objects passing line
* Visualisation for [zone](https://inference.roboflow.com/workflows/blocks/polygon_zone_visualization/) and [line counter](https://inference.roboflow.com/workflows/blocks/line_counter_visualization/)

We've introduced minimal support for video processing in the Workflows UI, with plans to expand to more advanced features soon. To get started, you can create a Python script using the InferencePipeline, similar to the provided [example](https://gist.github.com/PawelPeczek-Roboflow/585bb4dcb3ecde090537e7e7e9e17281).

*Video source [YT | Karol Majek](https://www.youtube.com/watch?v=MNn9qKG2UFI&t=0s)*

🔥 OWLv2 🤝 `inference`

Thanks to probicheaux we have [OWLv2](https://arxiv.org/abs/2306.09683) model in `inference`. OWLv2 was primarily trained to detect objects from text. The implementation in Inference currently only supports detecting objects from visual examples of that object.

You can use model in `inference` server - both CPU and GPU, as well as in Python package. Visit our [📖 docs](https://inference.roboflow.com/foundation/owlv2/) to learn more.

https://github.com/user-attachments/assets/d4933ea6-3ea1-4259-ba5a-9f1f6113e5a7


👓 TROCR 🤝 `inference`

stellasphere shipped TROCR model to expand OCR models offering in `inference` 🔥


You can use model in `inference` server - both CPU and GPU, as well as in Python package. Visit our [📖 docs](https://inference.roboflow.com/foundation/trocr/) to learn more.

🧑‍🎓 Workflows - endpoint to discover interface

Guessing the data format for Workflow inputs and outputs was a challange as for now, but thanks to EmilyGavrilenko this is no longer the case. We offer two new endpoints (for [workflows registered on the platform](https://detect.roboflow.com/docs#/default/describe_predefined_workflow_interface__workspace_name__workflows__workflow_id__describe_interface_post) and for [workflows submitted in payload](https://detect.roboflow.com/docs#/default/describe_workflow_interface_workflows_describe_interface_post)). Details in https://github.com/roboflow/inference/pull/644.

<details>
<summary>🔔 Example response</summary>

json
{
"inputs": {
"image": ["image"],
"model_id": ["roboflow_model_id"],
},
"outputs": {
"detections": ["object_detection_prediction"],
"crops": ["image"],
"classification": {
"inference_id": ["string"],
"predictions": ["classification_prediction"],
},
},
"typing_hints": {
"image": "dict",
"roboflow_model_id": "str",
"object_detection_prediction": "dict",
"string": "str",
"classification_prediction": "dict",
},
"kinds_schemas": {
"image": {},
"object_detection_prediction": {"dict": "with OpenAPI 3.0 schema of result"},
"classification_prediction": {"dict": "with OpenAPI 3.0 schema of result"}
}
}


</details>



🌱 Changed
* Bring back transformers extras by probicheaux in https://github.com/roboflow/inference/pull/639
* Move xformers to paligemma code by probicheaux in https://github.com/roboflow/inference/pull/641
* Cache plan details by grzegorz-roboflow in https://github.com/roboflow/inference/pull/636
* Bump next from 14.2.7 to 14.2.12 in /inference/landing by dependabot in https://github.com/roboflow/inference/pull/649

🔧 Fixed
* Fixed bug with Workflows Execution Engine causing bug when conditional execution discards inputs of a step that changes dimensionality - see details in https://github.com/roboflow/inference/pull/645


♻️ Removed
* Remove unmaintained device management code by robiscoding in https://github.com/roboflow/inference/pull/647

**Full Changelog**: https://github.com/roboflow/inference/compare/v0.18.1...v0.19.0