Haystack-experimental

🧪 New Experiments

New `Agent` component
[`Agent`](https://github.com/deepset-ai/haystack-experimental/blob/main/haystack_experimental/components/agents/agent.py) component enables tool-calling functionality with provider-agnostic chat model support and can be used as a standalone component or within a pipeline.
👉 See the `Agent` in action: [🧑‍🍳 Build a GitHub Issue Resolver Agent](https://haystack.deepset.ai/cookbook/github_issue_resolver_agent)


python
from haystack.dataclasses import ChatMessage
from haystack.components.websearch import SerperDevWebSearch
from haystack_experimental.tools.component_tool import ComponentTool
from haystack_experimental.components.agents import Agent
from haystack.components.generators.chat import OpenAIChatGenerator

web_tool = ComponentTool(
component=SerperDevWebSearch(),
)

agent = Agent(
chat_generator=OpenAIChatGenerator(model="gpt-4o-mini"),
tools=[web_tool],
exit_condition="text",
)

result = agent.run(
messages=[ChatMessage.from_user("Find information about Haystack")]
)

Improved `ComponentTool` and `tool` Decorator
The [`ComponentTool`](https://github.com/deepset-ai/haystack-experimental/blob/main/haystack_experimental/tools/component_tool.py) and [`tool`](https://github.com/deepset-ai/haystack-experimental/blob/7e88f2a4e4310359e6c76a6ce8ff920d1bb841f7/haystack_experimental/tools/from_function.py#L137) decorator are extended for better integration with the new `Agent` component

New Ready-Made `SuperComponents`
Introducing new `SuperComponent`s that bundle commonly used components and logic for indexing pipelines: [`MultiFileConverter`](https://github.com/deepset-ai/haystack-experimental/blob/main/haystack_experimental/super_components/converters/multi_file_converter.py), [`SentenceTransformersDocumentIndexer`](https://github.com/deepset-ai/haystack-experimental/blob/main/haystack_experimental/super_components/indexers/sentence_transformers_document_indexer.py), [`DocumentPreprocessor`](https://github.com/deepset-ai/haystack-experimental/blob/main/haystack_experimental/super_components/preprocessors/document_preprocessor.py)

python
from haystack_experimental.super_components.converters import MultiFileConverter

process all common file types (.csv, .docx, .html, .json, .md, .txt, .pdf, .pptx, .xlsx) with one component
converter = MultiFileConverter()
converter.run(sources=["test.txt", "test.pdf"], meta={})


What's Changed
* docs: add Supercomponent pydoc, delete outdated by dfokina in https://github.com/deepset-ai/haystack-experimental/pull/193
* docs: updating trace comparison tool README.md by davidsbatista in https://github.com/deepset-ai/haystack-experimental/pull/195
* chore: Create issue templates for adding, removing, moving an experiment by julian-risch in https://github.com/deepset-ai/haystack-experimental/pull/192
* chore: remove OpenSearch from experimental by anakin87 in https://github.com/deepset-ai/haystack-experimental/pull/200
* fix: fixing auto-merging tests, removing hard-coded doc ids by davidsbatista in https://github.com/deepset-ai/haystack-experimental/pull/202
* chore: add tool related code to prepare Agent PR by mathislucka in https://github.com/deepset-ai/haystack-experimental/pull/203
* feat: add file and indexing related super components by mathislucka in https://github.com/deepset-ai/haystack-experimental/pull/184
* docs: Add SuperComponent to catalog by julian-risch in https://github.com/deepset-ai/haystack-experimental/pull/190
* feat: Introduce Agent by mathislucka in https://github.com/deepset-ai/haystack-experimental/pull/175
* docs: add pydoc config for Agent component by julian-risch in https://github.com/deepset-ai/haystack-experimental/pull/208
* docs: Notebook for Agent component by mathislucka in https://github.com/deepset-ai/haystack-experimental/pull/204
* docs: add MultiFileConverter, SentenceTransformerrsDocumentIndexer, and DocumentPreprocessor to docs by dfokina in https://github.com/deepset-ai/haystack-experimental/pull/210


**Full Changelog**: https://github.com/deepset-ai/haystack-experimental/compare/v0.6.0...v0.7.0

New Experiments
- New `SuperComponent` abstraction that allows to wrap any pipeline into a friendly component interface and to create your own super components [1]
python
from haystack_experimental import SuperComponent

rag_pipeline = basic RAG pipeline with retriever, prompt builder, generator and answer builder components

input_mapping = {
"search_query": ["retriever.query", "prompt_builder.query", "answer_builder.query"]
}
output_mapping = {
"answer_builder.answers": "final_answers"
}

wrapper = SuperComponent(
pipeline=rag_pipeline,
input_mapping=input_mapping,
output_mapping=output_mapping
)

result = wrapper.run(search_query="What is the capital of France?")
print(result["final_answers"][0])

- New `AsyncPipeline` that can schedule components to run concurrently [2]

Other Updates:
* Added a debug/tracing script to compare two pipeline runs with the old and new pipeline run logic [3]
* Changed `LLMMetadaExtractor` to use `ChatGenerator` instead of `Generator` [4]

[1]: https://github.com/deepset-ai/haystack-experimental/pull/183
[2]: https://github.com/deepset-ai/haystack-experimental/pull/180
[3]: https://github.com/deepset-ai/haystack-experimental/pull/186
[4]: https://github.com/deepset-ai/haystack-experimental/pull/188


**Full Changelog**: https://github.com/deepset-ai/haystack-experimental/compare/v0.5.0...v0.6.0

New Experiments
- New `Pipeline` class with new pipeline run logic - [`Pipeline` example][1]

[1]: https://colab.research.google.com/github/deepset-ai/haystack-experimental/blob/main/examples/pipeline.ipynb

**Full Changelog**: https://github.com/deepset-ai/haystack-experimental/compare/v0.4.0...v0.5.0

----

🧬 New Pipeline Logic


This release introduces a reimplementation of the pipeline-run logic to resolve multiple issues, improving reliability and performance. These changes will also be included in **Haystack 2.10**.

Fixed Issues:

1. **Exceptions in pipelines with two cycles**
- Pipelines with two cycles sharing an optional (like in `PromptBuilder`) or a greedy variadic edge (e.g., in `BranchJoiner`) might raise exceptions. [Details here](https://github.com/deepset-ai/haystack/issues/8657).

2. **Incorrect execution in cycles with multiple optional or variadic edges**
- Entry points for cycles were non-deterministic, causing components to run with unexpected inputs or multiple times. This impacted execution time and final outputs.

3. **Missing intermediate outputs in cycles**
- Outputs produced within a cycle were overwritten, preventing downstream components from receiving them.

4. **Premature execution of lazy variadic components**
- Components like `DocumentJoiner` sometimes executed before receiving all inputs, leading to repeated partial executions that affected downstream results.

5. **Order-sensitive behavior in `add_component` and `connect`**
- Some bugs above occurred due to specific orderings of `add_component` and `connect` in pipeline creation, causing non-deterministic behavior in cyclic pipelines.

**Am I Affected by this Change?**

- **Non-cyclic pipelines without lazy variadic components**:
No impact—your pipelines should function as before.

- **Non-cyclic pipelines with lazy variadic components**:
Check inputs and outputs of components like `DocumentJoiner` for issues `4` and `5`. Use `LoggingTracer` with content tracing to validate behavior. Component execution order now uses lexicographical sorting; rename upstream components if necessary.

- **Pipelines with cycles**:
Review your pipeline outputs as well as the component input and outputs to ensure expected behavior, as you may encounter any of the above issues.

Share your comments in discussion 177

New Experiments
- AsyncPipeline and async-enabled components - [`AsyncPipeline` example][1]

[1]: https://colab.research.google.com/github/deepset-ai/haystack-experimental/blob/main/examples/async_pipeline.ipynb



**Full Changelog**: https://github.com/deepset-ai/haystack-experimental/compare/v0.3.0...v0.4.0

New Experiments
- Metadata extraction with LLM - [`LLMetadataExtractor`][1]
- Support for tools in [`OllamaChatGenerator`][2] , [`HuggingFaceAPIChatGenerator`][3], [`AnthropicChatGenerator`][4]

[1]: https://github.com/deepset-ai/haystack-experimental/blob/4e1b37aa862147cf3411d9ba9ae72c5d16b5366d/haystack_experimental/components/extractors/llm_metadata_extractor.py
[2]: https://github.com/deepset-ai/haystack-experimental/blob/4e1b37aa862147cf3411d9ba9ae72c5d16b5366d/haystack_experimental/components/generators/ollama/chat/chat_generator.py
[3]: https://github.com/deepset-ai/haystack-experimental/blob/4e1b37aa862147cf3411d9ba9ae72c5d16b5366d/haystack_experimental/components/generators/chat/hugging_face_api.py
[4]: https://github.com/deepset-ai/haystack-experimental/blob/4e1b37aa862147cf3411d9ba9ae72c5d16b5366d/haystack_experimental/components/generators/anthropic/chat/chat_generator.py


**Full Changelog**: https://github.com/deepset-ai/haystack-experimental/compare/v0.2.0...v0.3.0

New experiments
- Tool Calling - [refactored `ChatMessage` dataclass][10], [`Tool` dataclass][4], [refactored `OpenAIChatGenerator`][11], [`ToolInvoker` component][12]
- Memory Components - [`ChatMessageWriter`][5], [`ChatMessageRetriever`][6], [`InMemoryChatMessageStore`][7]
- Document Splitting & Retrieval Techniques - [`Auto-Merging Retriever`][8], [`HierarchicalDocumentSplitter`][9]
- Metadata extraction with LLM - [`LLMetadataExtractor`][13]


[4]: https://github.com/deepset-ai/haystack-experimental/tree/main/haystack_experimental/dataclasses/tool.py
[5]: https://github.com/deepset-ai/haystack-experimental/blob/main/haystack_experimental/components/writers/chat_message_writer.py
[6]: https://github.com/deepset-ai/haystack-experimental/blob/main/haystack_experimental/components/retrievers/chat_message_retriever.py
[7]: https://github.com/deepset-ai/haystack-experimental/blob/main/haystack_experimental/chat_message_stores/in_memory.py
[8]: https://github.com/deepset-ai/haystack-experimental/blob/main/haystack_experimental/components/retrievers/auto_merging_retriever.py
[9]: https://github.com/deepset-ai/haystack-experimental/blob/main/haystack_experimental/components/splitters/hierarchical_doc_splitter.py
[10]: https://github.com/deepset-ai/haystack-experimental/blob/main/haystack_experimental/dataclasses/chat_message.py
[11]: https://github.com/deepset-ai/haystack-experimental/blob/main/haystack_experimental/components/generators/chat/openai.py
[12]: https://github.com/deepset-ai/haystack-experimental/blob/main/haystack_experimental/components/tools/tool_invoker.py
[13]: https://github.com/deepset-ai/haystack-experimental/blob/main/haystack_experimental/components/extractors/llm_metadata_extractor.py

New Contributors
* dfokina made their first contribution in https://github.com/deepset-ai/haystack-experimental/pull/81

**Full Changelog**: https://github.com/deepset-ai/haystack-experimental/compare/v0.1.1...v0.2.0