Breaking Changes
Note that 2 methods from the `Sessions` class have been deprecated.
- `Sessions.preset_parameters`
- `Sessions.run_conversation`
For each of these methods, you can use `Sessions.detect_intent` instead, which is fully backwards compatible.
New Features
Agent Tasks Generator
This is a specialized tool that allows a user to evaluate any arbitrary agent to determine what the Agent is capable of accomplishing from a task perspective.
This is a pre-release feature that will be accompanied by more automated testing features in the future.
For now, you can use this as a way to analyze arbitrary Agents to see if they are set up to perform the tasks you believe you configured them to do.
py
from dfcx_scrapi.tools.agent_task_generator import AgentTaskGenerator
atg = AgentTaskGenerator(agent_id=agent_id)
atg.get_agent_tasks()
_Output_
{'tasks': [{'name': 'Greeting and Intent Understanding',
'description': 'The agent greets the user and attempts to understand their intent. It can provide basic information like translations or virtual money, and direct the user to appropriate tools or flows based on their request.'},
{'name': 'Product and Company Information Retrieval',
'description': 'The agent can access a data store containing information from the YETI website to answer user queries about YETI products and the company.'},
{'name': 'Trip Planning Assistance',
'description': 'The agent can collect basic information from the user to assist with trip planning, including destination, travel dates, and preferences. It can then pass this information to a separate flow for further processing.'}]}
Evaluation Dataset from Conversation History
You can now quickly create an Evaluation dataset format using pre-selected conversations from the Conversation History in your Agent.
Simply select the list of `conversation_ids` that you want, and pass that to the `Evals.create_dataset_from_conv_ids` method, which will provide a Pandas Dataframe in return.
This can be saved as a CSV, Google Sheet, or used locally to run Evals on your Agent.
py
from dfcx_scrapi.core.conversation_history import ConversationHistory
from dfcx_scrapi.tools.evaluations import Evaluations
ch = ConversationHistory()
evals = Evaluations(agent_id=agent_id)
all_convos = ch.list_conversations(agent_id)
convo_ids = [convo.name for convo in all_convos[:5]]
evals.create_dataset_from_conv_ids(convo_ids)
_Output_
| eval_id | action_id | action_type | action_input | action_input_parameters | tool_action | notes |
|---|---|---|---|---|---|---|
| 0 | 001 | 1 | User Utterance | what items do you have for dogs? | | |
| 1 | 001 | 2 | Tool Invocation | yeti-website | {'requestBody': {'query': 'what items do you h... | yeti-website | |
| 2 | 001 | 3 | Agent Response | YETI offers dog bowls and dog beds. The Boomer... | | |
| 3 | 002 | 1 | User Utterance | who is the ceo? | | |
| 4 | 002 | 2 | Tool Invocation | yeti-website | {'requestBody': {'query': 'who is the ceo?'}} | yeti-website | |
| 5 | 002 | 3 | Agent Response | The CEO of YETI is Matt Reintjes. | | |
| 6 | 003 | 1 | User Utterance | I want to speak to an operator | | |
| 7 | 003 | 2 | Agent Response | Just a moment while I connect you... | | |
| 8 | 004 | 1 | User Utterance | where is yeti hq at? | | |
| 9 | 004 | 2 | Tool Invocation | yeti-website | {'requestBody': {'query': 'where is yeti hq at... | yeti-website | |
| 10 | 004 | 3 | Agent Response | YETI's headquarters is located in Austin, Texa... | | |
| 11 | 005 | 1 | User Utterance | what is the smallest cup I can buy? | | |
| 12 | 005 | 2 | Tool Invocation | yeti-website | {'requestBody': {'query': 'what is the smalles... | yeti-website | |
| 13 | 005 | 3 | Agent Response | The smallest cup you can buy is the 4oz cup. I... | | |
CICD Workflow Example
We've added an example CICD workflow for anyone that is curious to see how a CICD workflow could be set up using SCRAPI.
Fair warning, it's very involved! 😄
However, it provides some good pointers on how you can set up these types of complex pipelines using this library.
Enhancements
* Added support for `environment_id` when calling `Sessions.build_session_id`, which allows you to now use `Sessions.detect_intent` with a `session_id` that includes an Environment
* Added `language_code` support throughout Evaluations class
* Added support for setting BigQuery logging and interaction settings
* Added support for new lint rules in `ruff`
* Updated some out of date Example notebooks and fixed broken links
* Added support for Session Parameters in Evaluations
Bug Fix
* Fixed several issues in Evaluations where dataframe prep and parsing where failing
What's Changed
* Feat/agent tasks generator by kmaphoenix in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/259
* feat: Implement Python requests-based get/set for BigQuery Interaction Logging Settings by justin-oos in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/258
* Feat/bq update sdk by kmaphoenix in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/263
* Fill Golden Template from Conv_Ids by gmchueh in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/261
* Fix/update lint rules by kmaphoenix in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/264
* Clean example notebooks (bot builder / vertex ai / google sheets) by ethanknights in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/256
* Adding session parameters to evaluations function by AAMEHROTRA1230 in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/262
* Feature/dfcxcicd by sridharvikram in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/239
* fix: add lang_code support for DataLoader by kmaphoenix in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/265
* fix/evaluations multiple tool pairing and empty utterance pairing by gmchueh in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/267
* Update google doc reference in nlu_evaluation_testing.ipynb by ethanknights in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/266
* Fix/session id with env by kmaphoenix in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/271
New Contributors
* justin-oos made their first contribution in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/258
* gmchueh made their first contribution in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/261
* ethanknights made their first contribution in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/256
* AAMEHROTRA1230 made their first contribution in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/262
* sridharvikram made their first contribution in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/239
**Full Changelog**: https://github.com/GoogleCloudPlatform/dfcx-scrapi/compare/1.12.5...1.13.0