Inference

🚀 Added

🤯 Next-level `workflows`

Better integration with Roboflow platform
From now on, we have much better alignment with UI workflow creator available in [`Roboflow app`](https://app.roboflow.com/). Just take a look how nice it presents itself thanks to hansent EmilyGavrilenko casmwenger kresetar jchens


But great look is not the only feature, the team has added tons of functionalities, including:
* operations on processed by `workflow` Execution Engine - including filtering and conditions are now possible to be build with UI creators
* Roboflow models and projects available to be used are suggested automatically
* Preview option to run workflow that is under development is now available
* ... and much more - check out yourself!

`workflows` Universal Query Language (UQL)

We've added Universal Query Language as extension to `workflows` eco-system. We've discovered that it would be extremely helpful for users to be able to build chains of transformations (like filtering, selecting only specific bounding boxes, aggregating results etc) or expressions evaluating into booleans. UQL powers UI extensions like the one presented below:


Yes, we know that `UQL` is not the best name, but as majority engineers we are struggling to find names for things we create. Please help us in that regards!

`workflows` 🤝 `sv.Detections`
From now on, the default representation of predictions from `object-detection`, `instance-segmentation` and `keypoint-detection` models is [`sv.Detections`](https://supervision.roboflow.com/latest/detection/core/). That has a lot of practical implications for blocks creators. Take a look how easy it is to add a block that makes prediction from your custom model. This was mainly possible thanks to grzegorz-roboflow

<details>
<summary>👉 Code snippet with your custom model block fitting our eco-system </summary>

python
from typing import Literal, Type

import supervision as sv

from inference.core.workflows.entities.base import (
Batch,
OutputDefinition,
WorkflowImageData,
)
from inference.core.workflows.entities.types import (
BATCH_OF_OBJECT_DETECTION_PREDICTION_KIND,
ImageInputField,
StepOutputImageSelector,
WorkflowImageSelector,
)
from inference.core.workflows.prototypes.block import (
BlockResult,
WorkflowBlock,
WorkflowBlockManifest,
)


class BlockManifest(WorkflowBlockManifest):
type: Literal["MyModel"]
images: Union[WorkflowImageSelector, StepOutputImageSelector] = ImageInputField

classmethod
def describe_outputs(cls) -> List[OutputDefinition]:
return [
OutputDefinition(
name="predictions", kind=[BATCH_OF_OBJECT_DETECTION_PREDICTION_KIND]
)
]


class MyModelBlock(WorkflowBlock):

def __init__(self):
self._model = load_my_model(...)

classmethod
def get_manifest(cls) -> Type[WorkflowBlockManifest]:
return BlockManifest

async def run(self, image: WorkflowImageData) -> BlockResult:
result = self._model(image)
detections = sv.Detections(...) here you need to convert results into sv.Detections - there is a need to add couple of keys into .data property - docs covering that will come soon, in questions - do not hesitate to ask
return {"predictions": detections}


</details>

True conditional branching for SIMD operations in `workflows`
We had a serious technical limitation in previous iterations of `workflows` Execution Engine - lack of ability to simulate different execution branches for each element of data processed`. This is no longer the case! Now it is possible to detect high-level objects, make crops based on detections and then for each cropped image independently decide whether or not to save in Roboflow project - based on condition stated in UQL 🤯


But this is not everything! As technical preview we prepared rock-paper-scissor game in `workflows`. Check it out [here](https://github.com/roboflow/inference/blob/main/tests/workflows/integration_tests/execution/test_workflow_solving_rock_paper_scissor.py)

Advancements in video processing with `workflows`
This feature is still **experimental**, but we are making progress - now it is possible to process multiple videos at once with `InferencePipeline` and `workflows`:

https://github.com/roboflow/inference/assets/146137186/a0bbd55a-d7a9-4ee7-ad14-aac8c03df2ba



<details>
<summary>👉 Code snippet </summary>

python
from typing import List, Optional

import cv2
import supervision as sv

from inference import InferencePipeline
from inference.core.interfaces.camera.entities import VideoFrame
from inference.core.utils.drawing import create_tiles

STOP = False
ANNOTATOR = sv.BoundingBoxAnnotator()


def main() -> None:
workflow_specification = {
"version": "1.0",
"inputs": [
{"type": "WorkflowImage", "name": "image"},
],
"steps": [
{
"type": "ObjectDetectionModel",
"name": "step_1",
"image": "$inputs.image",
"model_id": "yolov8n-640",
"confidence": 0.5,
}
],
"outputs": [
{"type": "JsonField", "name": "predictions", "selector": "$steps.step_1.predictions"},
],
}
pipeline = InferencePipeline.init_with_workflow(
video_reference=[
"<YOUR-VIDEO>",
"<YOUR-VIDEO>",
],
workflow_specification=workflow_specification,
on_prediction=workflows_sink,
)
pipeline.start()
pipeline.join()

def workflows_sink(
predictions: List[Optional[dict]],
video_frames: List[Optional[VideoFrame]],
) -> None:
images_to_show = []
for prediction, frame in zip(predictions, video_frames):
if prediction is None or frame is None:
continue
detections: sv.Detections = prediction["predictions"]
visualised = ANNOTATOR.annotate(frame.image.copy(), detections)
images_to_show.append(visualised)
tiles = create_tiles(images=images_to_show)
cv2.imshow(f"Predictions", tiles)
cv2.waitKey(1)


if __name__ == '__main__':
main()


</details>

Other changes:
* Step Name Property Copy Changes by yeldarby in https://github.com/roboflow/inference/pull/444
* Abstract ImageInputField and RoboflowModelField + Copy Changes by yeldarby in https://github.com/roboflow/inference/pull/445
* Allow CORS by default by yeldarby in https://github.com/roboflow/inference/pull/485
* Add PerspectiveCorrectionBlock and PolygonSimplificationBlock by grzegorz-roboflow in https://github.com/roboflow/inference/pull/441

List of contributors: EmilyGavrilenko, casmwenger, kresetar, jchens, yeldarby, grzegorz-roboflow, hansent, SkalskiP, PawelPeczek-Roboflow

Predictions JSON ➕ visualisation Roboflow hosted platform
Previously clients needed to choose between visualisation of predictions and Predictions JSON returned from `inference` server running at Roboflow hosted platform. This is no longer the case thanks to SolomonLake and https://github.com/roboflow/inference/pull/467

python
from inference_sdk import InferenceHTTPClient, InferenceConfiguration

CLIENT = InferenceHTTPClient(
api_url="https://detect.roboflow.com/",
api_key="<YOUR-API-KEY>"
).configure(InferenceConfiguration(
format="image_and_json",
))

response = CLIENT.infer("<your_image>.jpg", model_id="yolov8n-640")

check out
response["predictions"]
and
response["visualisation"]


🌱 Changed
* Fixing yolov10 documentation by nathan-marraccini in https://github.com/roboflow/inference/pull/480
* Supervision updates for Predict on a Video, Webcam or RTSP Stream Page by nathan-marraccini in https://github.com/roboflow/inference/pull/477
* Add paligemma aliases for newly uploaded models by probicheaux in https://github.com/roboflow/inference/pull/463
* Add PaliGemma LoRA by probicheaux in https://github.com/roboflow/inference/pull/464
* Bump braces from 3.0.2 to 3.0.3 in /inference/landing by dependabot in https://github.com/roboflow/inference/pull/466
* Fix security vulnerabilities by PawelPeczek-Roboflow in https://github.com/roboflow/inference/pull/483


🥇 New Contributors
* nathan-marraccini made their first contribution in https://github.com/roboflow/inference/pull/480

**Full Changelog**: https://github.com/roboflow/inference/compare/v0.12.1...v0.13.0

or
pip install "inference-cli>=0.12.1"
or
pip install "inference-sdk>=0.12.1"

</details>
<details>
<summary>👉 Downgrade numpy </summary>

bash
in your Python environment hosting inference library
pip install "numpy<2.0.0"

</details>

We are sorry for inconvenience.

❗ Planned deprecations
* `np_image_to_base64(...)` to be replaced with `encode_image_to_jpeg_bytes(...)` in the future - grzegorz-roboflow in https://github.com/roboflow/inference/pull/469

🌱 Changed
* Remove sv.FPSMonitor deprecation warnings by grzegorz-roboflow in https://github.com/roboflow/inference/pull/461
* Loose boto3 requirements by iurisilvio in https://github.com/roboflow/inference/pull/457 - `inference` should install faster now 🤗
* Fix paligemma generation bug by probicheaux in https://github.com/roboflow/inference/pull/459
* Add support for a tunnel to expose inference server to remote calls by iurisilvio in https://github.com/roboflow/inference/pull/451
* Workflow documentation additions, add YOLOv10 docs by capjamesg in https://github.com/roboflow/inference/pull/475
* fix Docker Getting Started link in docs returns 404 by grzegorz-roboflow in https://github.com/roboflow/inference/pull/478

🏅 New Contributors
* iurisilvio made their first contribution in https://github.com/roboflow/inference/pull/457

**Full Changelog**: https://github.com/roboflow/inference/compare/v0.12.0...v0.12.1

🔨 Fixed

🔥 `YOLOv10` in `inference` now has pre- and post-processing issues solved

Thanks to jameslahm we have inconsistencies in results from `YOLOv10` model in `inference` package sorted out. PR https://github.com/roboflow/inference/pull/437

<p align="center">
<img src="https://github.com/roboflow/inference/assets/146137186/beaafb53-8c99-433c-8a53-45ca002238e6" width="70%"/>
</p>

🌱 Changed

❗`breaking change`❗Inference from PaliGemma
PaliGemma models changes model category from foundation one into Roboflow model. That implies the following change in a way how it is exposed by `inference server`:

**Before:**
python
def do_gemma_request(prompt: str, image_path: str):
infer_payload = {
"image": {
"type": "base64",
"value": encode_bas64(image_path),
},
"api_key": "<ROBOFLOW-API-KEY>",
"prompt": prompt,
}
response = requests.post(
f'http://localhost:{PORT}/llm/paligemma',
json=infer_payload,
)
resp = response.json()


**Now:**
python
def do_gemma_request(prompt: str, image_path: str):
infer_payload = {
"image": {
"type": "base64",
"value": encode_bas64(image_path),
},
"prompt": prompt,
"model_id": "paligemma-3b-mix-224",
}
response = requests.post(
f'http://localhost:{PORT}/infer/lmm',
json=infer_payload,
)
resp = response.json()


PR https://github.com/roboflow/inference/pull/436
Other changes
* Replaced `sv.BoxAnnotator` with `sv.BoundingBoxAnnotator` combined with `sv.LabelAnnotator` to be prepare for `sv.BoxAnnotator` deprecation by grzegorz-roboflow in https://github.com/roboflow/inference/pull/434
* Add PaliGemma documentation, update table of contents by capjamesg in https://github.com/roboflow/inference/pull/429
* Add http get support for legacy model inference by PacificDou in https://github.com/roboflow/inference/pull/449
* Fix dead supported blocks link by LinasKo in https://github.com/roboflow/inference/pull/448
* Docs: Remove banner saying Sv Keypoint annotators are experimental by LinasKo in https://github.com/roboflow/inference/pull/450

🥇 New Contributors
* jameslahm made their first contribution in https://github.com/roboflow/inference/pull/437

**Full Changelog**: https://github.com/roboflow/inference/compare/v0.11.2...v0.12.0

What's Changed
* Add YOLOv10 Object Detection Support by NickHerrig and probicheaux in https://github.com/roboflow/inference/pull/431

New Contributors
* NickHerrig made their first contribution in https://github.com/roboflow/inference/pull/431

**Full Changelog**: https://github.com/roboflow/inference/compare/v0.11.1...v0.11.2

🔨 Fixed

❗ `setuptools>=70.0.0` breaks `CLIP` and `YoloWorld` models in `inference`

Using `setuptools` in version `70.0.0` and above breaks usage of Clip and YoloWorld models. That impacts historical version of inference package installed in python environments with newest `setuptools`. Problem may affect clients using `inference` as Python package in their environments, docker builds are not impacted.

**Symptoms of the problem:**
* `ImportError` while attempting `from inference.models import YOLOWorld`, despite previous `pip install inference[yolo-world]`
* `ImportError` while attempting `from inference.models import Clip`


We release change pinning `setuptools` version into compatible ones. This is not the ultimate solution for that problem (as some time in the future it may be needed to unblock `setuptools`), that's why we will need to take actions in the future releases - stay tuned.

As a solution for now, we recommend enforcing `setuptools<70.0.0` in all environments using `inference`, so if you are impacted restrict `setuptools` in your build:

pip install setuptools>=65.5.1,<70.0.0


🏗️ docker image for Jetson with Jetpack 4.5 is now fixed
We had issues with builds on Jetpack 4.5 which should be solved now. Details: https://github.com/roboflow/inference/pull/393

🌱 Changed
* In `workflows`, one can now define selectors to runtime inputs (`$inputs.<name>`) in outputs definitions, making it possible to pass input data through the `workflow`.

**Full Changelog**: https://github.com/roboflow/inference/compare/v0.11.0...v0.11.1

🚀 Added

🎉 PaliGemma in `inference`! 🎉

You've probably heard about [new PaliGemma model](https://blog.roboflow.com/paligemma-multimodal-vision/), right? We have it supported in new release of `inference` thanks to probicheaux.

To run the model, you need to build and `inference` server your GPU machine using the following commands:
bash
clone the inference repo
git clone https://github.com/roboflow/inference.git

navigate into repository root
cd inference

build inference server with PaliGemma dependencies
docker build -t roboflow/roboflow-inference-server-paligemma -f docker/dockerfiles/Dockerfile.paligemma .

run server
docker run -p 9001:9001 roboflow/roboflow-inference-server-paligemma


<details>
<summary>👉 To prompt the model visit our <a href="https://github.com/roboflow/inference/blob/main/examples/paligemma/paligemma_client.py">examples 📖 <a/> or use the following code snippet:</summary>

python
import base64
import requests
import os

PORT = 9001
API_KEY = os.environ["ROBOFLOW_API_KEY"]
IMAGE_PATH = "<PATH-TO-YOUR>/image.jpg"

def encode_bas64(image_path: str):
with open(image_path, "rb") as image:
x = image.read()
image_string = base64.b64encode(x)
return image_string.decode("ascii")

def do_gemma_request(image_path: str, prompt: str):
infer_payload = {
"image": {
"type": "base64",
"value": encode_bas64(image_path),
},
"api_key": API_KEY,
"prompt": prompt
}
response = requests.post(
f'http://localhost:{PORT}/llm/paligemma',
json=infer_payload,
)
return response.json()


print(do_gemma_request(
image_path=IMAGE_PATH,
prompt="Describe the image"
))


</details>

🌱 Changed
* documentations updates:
* document source_id parameter of VideoFrame by sberan in https://github.com/roboflow/inference/pull/395
* fix workflows specification URL and other docs updates by SolomonLake in https://github.com/roboflow/inference/pull/398
* add link to Roboflow licensing by capjamesg in https://github.com/roboflow/inference/pull/403

🔨 Fixed
* Bug introduced into `InferencePipeline.init_with_workflow(...)` in `v0.10.0` causing import errors yielding misleading error message informing about broken dependencies:

inference.core.exceptions.CannotInitialiseModelError: Could not initialise workflow processing due to lack of dependencies required. Please provide an issue report under https://github.com/roboflow/inference/issues

Fixed with this PR https://github.com/roboflow/inference/pull/407


**Full Changelog**: https://github.com/roboflow/inference/compare/v0.10.0...v0.11.0