Inference

🚀 Added

`inference` is ready for `Florence-2` 🤩

Thanks to probicheaux we have `inference` package ready for `Florence-2`. It is Large Multimodal Model capable of processing both image and text input handling wide range of generic vision and language-vision tasks.

We are excited to add it to the collection of models offered by `inference`. Due to the complexity of build, model is shipped only within
[docker image 🐋 ](https://hub.docker.com/layers/roboflow/roboflow-inference-server-gpu/0.14.0/images/sha256-ef5516297b6ce4a34c5e74f5e7c9a4bb80f3da22e9e18f7fad6b4e3d297e75de?context=explore). Everything within our official `inference` server build for GPU 🤯 . To fully utilise the new models you need to **wait on the release in Roboflow platform**.

You should be able to spin up your container via `inference-cli`:

inference server start


<details>
<summary> ❗ What is required to run the container and what has changed in the build? </summary>

We've needed to bump required CUDA version in docker build for GPU server from `11.7` to `11.8`. That is why now, you may not be able to run
the container on servers having older CUDA. We've run the server experimentally on machine with CUDA 11.6 and it worked, but we cannot guarantee that to work on older builds.

</details>

<details>
<summary> 🤔 How to run new model? </summary>

python
import requests

payload = {
"api_key": "<YOUR-ROBOFLOW-API-KEY>,
"image": {
"type": "url",
"value": "https://media.roboflow.com/dog.jpeg",
},
"prompt": "<CAPTION>",
"model_id": "<model-id-available-when-roboflow-platform-starts-the-support>"
}

response = requests.post(
f"{server_url}/infer/lmm",
json=payload,
)

print(response.json())

</details>


New blocks in `workflows` 🥹

We have added the following block to `workflows` ecosystem:
* `Property Definition` which let you to use specific attribute of data as an input for next step or as output
* `Detections Classes Replacement` to replace classes of bounding boxes in scenario when you first run general object-detection model, then crop image based on predictions and you apply secondary classification model. Results of secondary model replaces originally predicted classes
* and few others - explore our [collection of blocks ✨](https://inference.roboflow.com/workflows/blocks/)

Blocks that were added are still **in refinement** - we may want to improve them over time - so stay tuned!

🌱 Changed

🔐 Mitigation for security vulnerabilities ❗ **BREAKING** 🚧
To two mitigate security vulnerabilities:
* unsafe deserialisation of pickled inputs enabled by default for self-hosted `inference`
* Server-side request forgery (SSRF)

we needed to add couple of changes, among which one is **breaking**. From now on **default value for env variable: `ALLOW_NUMPY_INPUT` is `False`**.

**Implications:**
* if you rely on pickled numpy images passed to `inference` Python package or sent to `inference` server - you need to set this env variable explicitly into `ALLOW_NUMPY_INPUT=true` in your environment or start a server with this env variable (see [how](https://inference.roboflow.com/inference_helpers/inference_cli/#inference-server-start))
* there are also other changes which you can optionally tune to run `inference` server safer - see our [docs 📖 ](https://inference.roboflow.com/server_configuration/accepted_input_formats/)

🔨 Fixed

❗ Removed bug in inference post-processing

Some models trained at Roboflow platform experienced problem with predictions post-processing when there was padding as
the option selected while creating dataset. Thanks to grzegorz-roboflow it was fixed in https://github.com/roboflow/inference/pull/495

Other minor fixes
* fixed malformed `workflow` outputs in https://github.com/roboflow/inference/pull/499
* replace match statement with if-else for Python 3.9 compatibility by natserract in https://github.com/roboflow/inference/pull/488
* InferencePipeline: allow it to run offline even if Active Learning enabled by sberan in https://github.com/roboflow/inference/pull/491
* Import `sky` only when required because it is slow by iurisilvio in https://github.com/roboflow/inference/pull/494
* Change GPT-4 default model into GPT-4o by PawelPeczek-Roboflow in https://github.com/roboflow/inference/pull/500
* Monitoring improvements robiscoding in https://github.com/roboflow/inference/pull/490, robiscoding in https://github.com/roboflow/inference/pull/492
* Extend perspective correction to warp image by grzegorz-roboflow in https://github.com/roboflow/inference/pull/503
* Show block name in error message thrown by steps_initialiser by grzegorz-roboflow in https://github.com/roboflow/inference/pull/504
* Fix issue with workflows blocks after adding request id to response by PawelPeczek-Roboflow in https://github.com/roboflow/inference/pull/505
* Follow config to import core models by iurisilvio in https://github.com/roboflow/inference/pull/508
* Rename workflow blocks by EmilyGavrilenko in https://github.com/roboflow/inference/pull/511
* Update upload weights list by capjamesg in https://github.com/roboflow/inference/pull/512
* Default to Local Workflows Execution by yeldarby in https://github.com/roboflow/inference/pull/515

🏅 New Contributors
* natserract made their first contribution in https://github.com/roboflow/inference/pull/488
* EmilyGavrilenko made their first contribution in https://github.com/roboflow/inference/pull/511

**Full Changelog**: https://github.com/roboflow/inference/compare/v0.13.0...v0.14.0

🚀 Added

🤯 Next-level `workflows`

Better integration with Roboflow platform
From now on, we have much better alignment with UI workflow creator available in [`Roboflow app`](https://app.roboflow.com/). Just take a look how nice it presents itself thanks to hansent EmilyGavrilenko casmwenger kresetar jchens


But great look is not the only feature, the team has added tons of functionalities, including:
* operations on processed by `workflow` Execution Engine - including filtering and conditions are now possible to be build with UI creators
* Roboflow models and projects available to be used are suggested automatically
* Preview option to run workflow that is under development is now available
* ... and much more - check out yourself!

`workflows` Universal Query Language (UQL)

We've added Universal Query Language as extension to `workflows` eco-system. We've discovered that it would be extremely helpful for users to be able to build chains of transformations (like filtering, selecting only specific bounding boxes, aggregating results etc) or expressions evaluating into booleans. UQL powers UI extensions like the one presented below:


Yes, we know that `UQL` is not the best name, but as majority engineers we are struggling to find names for things we create. Please help us in that regards!

`workflows` 🤝 `sv.Detections`
From now on, the default representation of predictions from `object-detection`, `instance-segmentation` and `keypoint-detection` models is [`sv.Detections`](https://supervision.roboflow.com/latest/detection/core/). That has a lot of practical implications for blocks creators. Take a look how easy it is to add a block that makes prediction from your custom model. This was mainly possible thanks to grzegorz-roboflow

<details>
<summary>👉 Code snippet with your custom model block fitting our eco-system </summary>

python
from typing import Literal, Type

import supervision as sv

from inference.core.workflows.entities.base import (
Batch,
OutputDefinition,
WorkflowImageData,
)
from inference.core.workflows.entities.types import (
BATCH_OF_OBJECT_DETECTION_PREDICTION_KIND,
ImageInputField,
StepOutputImageSelector,
WorkflowImageSelector,
)
from inference.core.workflows.prototypes.block import (
BlockResult,
WorkflowBlock,
WorkflowBlockManifest,
)


class BlockManifest(WorkflowBlockManifest):
type: Literal["MyModel"]
images: Union[WorkflowImageSelector, StepOutputImageSelector] = ImageInputField

classmethod
def describe_outputs(cls) -> List[OutputDefinition]:
return [
OutputDefinition(
name="predictions", kind=[BATCH_OF_OBJECT_DETECTION_PREDICTION_KIND]
)
]


class MyModelBlock(WorkflowBlock):

def __init__(self):
self._model = load_my_model(...)

classmethod
def get_manifest(cls) -> Type[WorkflowBlockManifest]:
return BlockManifest

async def run(self, image: WorkflowImageData) -> BlockResult:
result = self._model(image)
detections = sv.Detections(...) here you need to convert results into sv.Detections - there is a need to add couple of keys into .data property - docs covering that will come soon, in questions - do not hesitate to ask
return {"predictions": detections}


</details>

True conditional branching for SIMD operations in `workflows`
We had a serious technical limitation in previous iterations of `workflows` Execution Engine - lack of ability to simulate different execution branches for each element of data processed`. This is no longer the case! Now it is possible to detect high-level objects, make crops based on detections and then for each cropped image independently decide whether or not to save in Roboflow project - based on condition stated in UQL 🤯


But this is not everything! As technical preview we prepared rock-paper-scissor game in `workflows`. Check it out [here](https://github.com/roboflow/inference/blob/main/tests/workflows/integration_tests/execution/test_workflow_solving_rock_paper_scissor.py)

Advancements in video processing with `workflows`
This feature is still **experimental**, but we are making progress - now it is possible to process multiple videos at once with `InferencePipeline` and `workflows`:

https://github.com/roboflow/inference/assets/146137186/a0bbd55a-d7a9-4ee7-ad14-aac8c03df2ba



<details>
<summary>👉 Code snippet </summary>

python
from typing import List, Optional

import cv2
import supervision as sv

from inference import InferencePipeline
from inference.core.interfaces.camera.entities import VideoFrame
from inference.core.utils.drawing import create_tiles

STOP = False
ANNOTATOR = sv.BoundingBoxAnnotator()


def main() -> None:
workflow_specification = {
"version": "1.0",
"inputs": [
{"type": "WorkflowImage", "name": "image"},
],
"steps": [
{
"type": "ObjectDetectionModel",
"name": "step_1",
"image": "$inputs.image",
"model_id": "yolov8n-640",
"confidence": 0.5,
}
],
"outputs": [
{"type": "JsonField", "name": "predictions", "selector": "$steps.step_1.predictions"},
],
}
pipeline = InferencePipeline.init_with_workflow(
video_reference=[
"<YOUR-VIDEO>",
"<YOUR-VIDEO>",
],
workflow_specification=workflow_specification,
on_prediction=workflows_sink,
)
pipeline.start()
pipeline.join()

def workflows_sink(
predictions: List[Optional[dict]],
video_frames: List[Optional[VideoFrame]],
) -> None:
images_to_show = []
for prediction, frame in zip(predictions, video_frames):
if prediction is None or frame is None:
continue
detections: sv.Detections = prediction["predictions"]
visualised = ANNOTATOR.annotate(frame.image.copy(), detections)
images_to_show.append(visualised)
tiles = create_tiles(images=images_to_show)
cv2.imshow(f"Predictions", tiles)
cv2.waitKey(1)


if __name__ == '__main__':
main()


</details>

Other changes:
* Step Name Property Copy Changes by yeldarby in https://github.com/roboflow/inference/pull/444
* Abstract ImageInputField and RoboflowModelField + Copy Changes by yeldarby in https://github.com/roboflow/inference/pull/445
* Allow CORS by default by yeldarby in https://github.com/roboflow/inference/pull/485
* Add PerspectiveCorrectionBlock and PolygonSimplificationBlock by grzegorz-roboflow in https://github.com/roboflow/inference/pull/441

List of contributors: EmilyGavrilenko, casmwenger, kresetar, jchens, yeldarby, grzegorz-roboflow, hansent, SkalskiP, PawelPeczek-Roboflow

Predictions JSON ➕ visualisation Roboflow hosted platform
Previously clients needed to choose between visualisation of predictions and Predictions JSON returned from `inference` server running at Roboflow hosted platform. This is no longer the case thanks to SolomonLake and https://github.com/roboflow/inference/pull/467

python
from inference_sdk import InferenceHTTPClient, InferenceConfiguration

CLIENT = InferenceHTTPClient(
api_url="https://detect.roboflow.com/",
api_key="<YOUR-API-KEY>"
).configure(InferenceConfiguration(
format="image_and_json",
))

response = CLIENT.infer("<your_image>.jpg", model_id="yolov8n-640")

check out
response["predictions"]
and
response["visualisation"]


🌱 Changed
* Fixing yolov10 documentation by nathan-marraccini in https://github.com/roboflow/inference/pull/480
* Supervision updates for Predict on a Video, Webcam or RTSP Stream Page by nathan-marraccini in https://github.com/roboflow/inference/pull/477
* Add paligemma aliases for newly uploaded models by probicheaux in https://github.com/roboflow/inference/pull/463
* Add PaliGemma LoRA by probicheaux in https://github.com/roboflow/inference/pull/464
* Bump braces from 3.0.2 to 3.0.3 in /inference/landing by dependabot in https://github.com/roboflow/inference/pull/466
* Fix security vulnerabilities by PawelPeczek-Roboflow in https://github.com/roboflow/inference/pull/483


🥇 New Contributors
* nathan-marraccini made their first contribution in https://github.com/roboflow/inference/pull/480

**Full Changelog**: https://github.com/roboflow/inference/compare/v0.12.1...v0.13.0

or
pip install "inference-cli>=0.12.1"
or
pip install "inference-sdk>=0.12.1"

</details>
<details>
<summary>👉 Downgrade numpy </summary>

bash
in your Python environment hosting inference library
pip install "numpy<2.0.0"

</details>

We are sorry for inconvenience.

❗ Planned deprecations
* `np_image_to_base64(...)` to be replaced with `encode_image_to_jpeg_bytes(...)` in the future - grzegorz-roboflow in https://github.com/roboflow/inference/pull/469

🌱 Changed
* Remove sv.FPSMonitor deprecation warnings by grzegorz-roboflow in https://github.com/roboflow/inference/pull/461
* Loose boto3 requirements by iurisilvio in https://github.com/roboflow/inference/pull/457 - `inference` should install faster now 🤗
* Fix paligemma generation bug by probicheaux in https://github.com/roboflow/inference/pull/459
* Add support for a tunnel to expose inference server to remote calls by iurisilvio in https://github.com/roboflow/inference/pull/451
* Workflow documentation additions, add YOLOv10 docs by capjamesg in https://github.com/roboflow/inference/pull/475
* fix Docker Getting Started link in docs returns 404 by grzegorz-roboflow in https://github.com/roboflow/inference/pull/478

🏅 New Contributors
* iurisilvio made their first contribution in https://github.com/roboflow/inference/pull/457

**Full Changelog**: https://github.com/roboflow/inference/compare/v0.12.0...v0.12.1

🔨 Fixed

🔥 `YOLOv10` in `inference` now has pre- and post-processing issues solved

Thanks to jameslahm we have inconsistencies in results from `YOLOv10` model in `inference` package sorted out. PR https://github.com/roboflow/inference/pull/437

<p align="center">
<img src="https://github.com/roboflow/inference/assets/146137186/beaafb53-8c99-433c-8a53-45ca002238e6" width="70%"/>
</p>

🌱 Changed

❗`breaking change`❗Inference from PaliGemma
PaliGemma models changes model category from foundation one into Roboflow model. That implies the following change in a way how it is exposed by `inference server`:

**Before:**
python
def do_gemma_request(prompt: str, image_path: str):
infer_payload = {
"image": {
"type": "base64",
"value": encode_bas64(image_path),
},
"api_key": "<ROBOFLOW-API-KEY>",
"prompt": prompt,
}
response = requests.post(
f'http://localhost:{PORT}/llm/paligemma',
json=infer_payload,
)
resp = response.json()


**Now:**
python
def do_gemma_request(prompt: str, image_path: str):
infer_payload = {
"image": {
"type": "base64",
"value": encode_bas64(image_path),
},
"prompt": prompt,
"model_id": "paligemma-3b-mix-224",
}
response = requests.post(
f'http://localhost:{PORT}/infer/lmm',
json=infer_payload,
)
resp = response.json()


PR https://github.com/roboflow/inference/pull/436
Other changes
* Replaced `sv.BoxAnnotator` with `sv.BoundingBoxAnnotator` combined with `sv.LabelAnnotator` to be prepare for `sv.BoxAnnotator` deprecation by grzegorz-roboflow in https://github.com/roboflow/inference/pull/434
* Add PaliGemma documentation, update table of contents by capjamesg in https://github.com/roboflow/inference/pull/429
* Add http get support for legacy model inference by PacificDou in https://github.com/roboflow/inference/pull/449
* Fix dead supported blocks link by LinasKo in https://github.com/roboflow/inference/pull/448
* Docs: Remove banner saying Sv Keypoint annotators are experimental by LinasKo in https://github.com/roboflow/inference/pull/450

🥇 New Contributors
* jameslahm made their first contribution in https://github.com/roboflow/inference/pull/437

**Full Changelog**: https://github.com/roboflow/inference/compare/v0.11.2...v0.12.0

What's Changed
* Add YOLOv10 Object Detection Support by NickHerrig and probicheaux in https://github.com/roboflow/inference/pull/431

New Contributors
* NickHerrig made their first contribution in https://github.com/roboflow/inference/pull/431

**Full Changelog**: https://github.com/roboflow/inference/compare/v0.11.1...v0.11.2

🔨 Fixed

❗ `setuptools>=70.0.0` breaks `CLIP` and `YoloWorld` models in `inference`

Using `setuptools` in version `70.0.0` and above breaks usage of Clip and YoloWorld models. That impacts historical version of inference package installed in python environments with newest `setuptools`. Problem may affect clients using `inference` as Python package in their environments, docker builds are not impacted.

**Symptoms of the problem:**
* `ImportError` while attempting `from inference.models import YOLOWorld`, despite previous `pip install inference[yolo-world]`
* `ImportError` while attempting `from inference.models import Clip`


We release change pinning `setuptools` version into compatible ones. This is not the ultimate solution for that problem (as some time in the future it may be needed to unblock `setuptools`), that's why we will need to take actions in the future releases - stay tuned.

As a solution for now, we recommend enforcing `setuptools<70.0.0` in all environments using `inference`, so if you are impacted restrict `setuptools` in your build:

pip install setuptools>=65.5.1,<70.0.0


🏗️ docker image for Jetson with Jetpack 4.5 is now fixed
We had issues with builds on Jetpack 4.5 which should be solved now. Details: https://github.com/roboflow/inference/pull/393

🌱 Changed
* In `workflows`, one can now define selectors to runtime inputs (`$inputs.<name>`) in outputs definitions, making it possible to pass input data through the `workflow`.

**Full Changelog**: https://github.com/roboflow/inference/compare/v0.11.0...v0.11.1