New Features
Builder Classes
In this release, we're introducing a new concept called `Builders`. 👷🏼♂️🏗 🛠
**Motivation:**
All Dialogflow CX resources are built as [Protocol Buffers](https://developers.google.com/protocol-buffers) or "protos" for short. When you call `get_agent` or `get_intent`, you're receiving that resource as an `Agent Proto` or `Intent Proto` that you can then manipulate, edit, and push back to Dialogflow CX.
Protos can contain other protos. Protos can be difficult to work with at times!
We built this class to simplify working with Protos!
Each Builder class offers a set of CRUD functions that can be performed **_100% offline_**, without any calls to the Dialogflow CX APIs.
This allows you to programmatically build 100's or 1000's of resources offline, iterate, experiment, and perfect until you're ready to actually push them to your Dialogflow CX Agent via the SCRAPI `core` classes.
We've even included a [Builders 101 Colab Notebook](https://github.com/GoogleCloudPlatform/dfcx-scrapi/blob/main/examples/bot_building_series/builders_101.ipynb) to kickstart your Builders journey!
Happy Building! 😄 🏗️
---
NLU Utils - Semantic Similarity and Clustering
We've introduced a new [NaturalLanguageUnderstandingUtil](https://github.com/GoogleCloudPlatform/dfcx-scrapi/blob/main/src/dfcx_scrapi/tools/nlu_util.py) Class that is packed full of some amazing NLU Analysis tools to supercharge your bot tuning workflows. Powered by [Tensorflow](https://www.tensorflow.org/), [1][USEv4](https://tfhub.dev/google/universal-sentence-encoder/4), and [2][ScaNN](https://github.com/google-research/google-research/tree/master/scann), the class includes analysis methods to perform the following tasks:
* Identifying Conflicting Training Phrases Across Intents
* Finding the Most Similar Training Phrases for a Provided Set of Utterances
* Clustering Utterances that Don't Match Training Phrases
We've provided an [NLU Analysis](https://github.com/GoogleCloudPlatform/dfcx-scrapi/blob/main/examples/nlu_analysis_series/nlu_semantic_similarity_demo.ipynb) notebook that allows you to dive right in and start using these features with your Dialogflow CX Agents.
[1] [USEv4]() provides us with high quality embeddings that we can utilize for performing Semantic Similarity and Classification tasks.
[2] The [Scalable Nearest Neighbors (ScaNN)](https://github.com/google-research/google-research/tree/master/scann) method from Google Research team provides us with SOTA performance clustering.
---
Agent Assist
We're _super excited_ to announce the release of our `Agent Assist` class! This is our first non-DFCX class to make it into the library as well. After careful consideration, we realized that there are many `CCAI Ecosystem` APIs that a typical bot developer might need to access that are immediately adjacent to Dialogflow CX. The Agent Assist APIs are used heavily in conjunction with DFCX APIs, so it was a logical choice for us to include them.
With the new Agent Assist class, you can perform several functions like:
* CRUD operations for Conversation Profiles
* Extremely useful if you're using [AnalyzeContent](https://cloud.google.com/dialogflow/es/docs/reference/rest/v2beta1/projects.conversations.participants/analyzeContent)
* CRUD operations on Participants
* Creating and Completing Conversations
We'll continue to build out this class to include other AA features like Smart Reply, FAQ Suggestion, and Article Suggestion over time.
You can find the top level folder `agent_assist` adjacent to the `dfcx_scrapi` folder.
---
Session Entity Types
If you're familiar with [System Entity Types](https://cloud.google.com/dialogflow/cx/docs/concept/entity-system) and [Custom Entity Types](https://cloud.google.com/dialogflow/cx/docs/concept/entity-custom) but haven't used [Session Entity Types](https://cloud.google.com/dialogflow/cx/docs/concept/entity-session) yet, then you're in for a treat! 🍬
Session Entity Types are a powerful way to enhance your bot's NLU capabilities at a very granular level - Session by Session. Unlike System and Custom Entity Types which can only be modified during _Design Time_, Session Entity Types can be modified dynamically _during Runtime_.
Once you've instantiated a user Session, you can call the Session Entity Types API to create _Session specific entity types_ that have a TTL for the length of the active Session.
This allows you to handle use cases like:
* Loading user-specific data into a session for NLU recognition
* Performing Entity Expansion on existing Entity Types based on user feedback _during the conversation_
* Overwriting existing Entity Types for a single user session
---
Enhancements
Intents to DataFrame Transposed
We've introduced a new `Intents` method that allows you to export your Intents / Training Phrases transposed from what the normal export achieves. Typically the data is exported row by row like this:
| intent | training_phrase|
|---|---|
|head_intent.billing| I need to pay my bill|
|head_intent.billing| How do I make a new payment for the statement I received?|
|head_intent.billing| What's the minimum I can pay right now for my phone bill?|
In the transposed setup, we provide the data with the `Intent Name` as the column header, and all of the training phrases below it:
| head_intent.billing| confirmation.yes |
|---|---|
|I need to pay my bill| yes|
|How do I make a new payment for the statement I received?| yeah|
|What's the minimum I can pay right now for my phone bill?| for sure|
The motivation behind this feature is to provide the exported data for your team's linguists, data labelers, and QA team to review from a different perspective. This transposed format can help with use cases like:
* Identifying "weak" Intents that need more training phrases
* Identifying "strong" Intents that perhaps have too _many_ training phrases
* Overall Intent and Training Phrase Balancing and Distribution
---
Notebooks
We've provided 5 new example notebooks in this release.
1. [SCRAPI Prebuilt Development Notebook](https://github.com/GoogleCloudPlatform/dfcx-scrapi/blob/main/examples/scrapi_prebuilt.ipynb), This is a a great starter notebook if you're unsure where to begin with SCRAPI. It provides some foundational imports and ends with a blank canvas to allow you to explore, script, and build based on your needs.
2. [Builder 101 - Using Builder Classes to Build Bots Programmatically](https://github.com/GoogleCloudPlatform/dfcx-scrapi/blob/main/examples/bot_building_series/builders_101.ipynb), A primer on the new Builder classes that will get you up to speed in no time!
3. [Exporting Custom Entities to Google Sheets](https://github.com/GoogleCloudPlatform/dfcx-scrapi/blob/main/examples/google_sheets_series/custom_entities_to_sheets.ipynb), a quick how-to guide on getting Custom Entity Types and Synonyms out of your bot, and into Google Sheets to share, analyze, and iterate with your team.
4. [NLU Analysis - Semantic Similarity and Clustering](https://github.com/GoogleCloudPlatform/dfcx-scrapi/blob/main/examples/nlu_analysis_series/nlu_semantic_similarity_demo.ipynb), using machine learning to improve machine learning! This notebook provides 3 great ML-based approaches to tuning your Intents and Training Phrases.
5. [Intent Analysis Using Levenshtein Ratios](https://github.com/GoogleCloudPlatform/dfcx-scrapi/blob/main/examples/nlu_analysis_series/intent_levenshtein_demo.ipynb), another great analysis notebook that leverages Levenshtein Distance to identify conflicts between intents.
---
Bug Fixes
- Fixed a bug in the `Experiments` class that incorrectly set the `client_options` variable
- Fixed a bug in `entity_type_proto_to_dataframe` that was incorrectly concatenating a Pandas DataFrame with a Dictionary, instead of 2 Pandas DataFrames
- Fixed a bug in the `Levenshtein` class that referenced non-existent keys in an upstream DataFrame
---
Misc
- Lots of style updates across the library for consistency
- Doc string updates across the library to provide better explanation of methods and functions
---
What's Changed
* Feature/intent to df transposed by MRyderOC in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/73
* Correctly set region using environment_path by cbradgoog in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/77
* entity to dataframe fix by DtStry in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/76
* Adding methods for extracting test coverage by hkhaitan1 in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/75
* Feature/intent builder by MRyderOC in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/74
* Feature/session entity types by kmaphoenix in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/84
* Feature/agent and entity builder by MRyderOC in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/80
* 81 fr static analysis class and notebooks by cgibson6279 in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/83
* Feature/notebook examples by kmaphoenix in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/86
* Feature/style cleanup by SeanScripts in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/82
New Contributors
* cbradgoog made their first contribution in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/77
* DtStry made their first contribution in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/76
* hkhaitan1 made their first contribution in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/75
**Full Changelog**: https://github.com/GoogleCloudPlatform/dfcx-scrapi/compare/1.4.0...1.5.0