Dfcx-scrapi

What's Changed
* Conversation Clean Up and Minor Feature Adds by kmaphoenix in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/36


**Full Changelog**: https://github.com/GoogleCloudPlatform/dfcx-scrapi/compare/1.2.0...1.2.1

Features

Versions
You can now manage DFCX [Versions](https://cloud.google.com/dialogflow/cx/docs/concept/version) directly from SCRAPI which includes all basic CRUD functions.

py
from dfcx_scrapi.core.versions import Versions
from dfcx_scrapi.core.flows import Flows

f = Flows(creds_path)
flows_map = f.get_flows_map(agent_id, reverse=True)

List Versions
all_versions = v.list_versions(flows_map['Default Start Flow'])

Get Version by Display Name
ver = v.get_version_by_display_name(display_name='dsf v2', flow_id=flows_map['Default Start Flow'])

Load Version
lro = v.load_version(ver)

Create Version
lro = v.create_version(flows_map['Default Start Flow'], 'patrick api ver', 'this came from SCRAPI')

Compare Version
res = v.compare_versions(v0, v1, flows_map['Default Start Flow'])

Delete Version
v.delete_version(v1)

* Feature/versions by Omerside in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/23

Environments
As a compliment to Versions, the DFCX [Environments](https://cloud.google.com/dialogflow/cx/docs/concept/version#manage-environments) functionality has also been added to SCRAPI.
Automatically manage the promotion of Flow Versions into various Environments within DFCX with this Class.

py
from dfcx_scrapi.core.environments import Environments
from dfcx_scrapi.core.operations import Operations

ops = Operations(creds_path)
envs = Environments(creds_path)

List Envs
all_envs = envs.list_environments(agent_id)
prod_env = all_envs[0]

Get Env
prod_env = envs.get_environment(prod_env.name)

Get Env by Display Name
demo_env = envs.get_environment_by_display_name('PROD', agent_id)

Create Enviornment
lro = envs.create_environment_by_display_name(
display_name='DEMO2',
version_configs=[
('Default Start Flow', 2),
('Confidence Demo',3),
('Sentiment Demo',1),
('Response Map Demo',1),
('Lists Demo',1),
('Multi Param Demo',1),
('Proper Names Demo',4),
('Entity Routing',1),
('[Redaction] Demo',1)
],
description='This is my DEMO environment',
agent_id=agent_id)

Update Environment
lro = envs.update_environment(demo_env.name, display_name='DEMO_API_UPDATE')

Delete Environment
envs.delete_environment(demo_env.name)

Deploy Flow to Environment
lro = envs.deploy_flow_to_environment(demo_env.name, flow_ver_id))

Lookup Environment History
history = envs.lookup_environment_history(demo_env.name)

List Continuous Test Results
ct_res = envs.list_continuous_test_results(demo_env.name)

* Feature/environments by Omerside in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/26

Security Settings
You can now manage [CCAI Security Settings](https://cloud.google.com/dialogflow/cx/docs/concept/security-settings) directly from SCRAPI! The CCAI Security Settings feature allows you to apply Data Loss Prevention (DLP) templates,Redaction Strategies and Retention Policies for conversations that are handled by Dialogflow CX.

py
from dfcx_scrapi.core.security_settings import SecuritySettings

ss = SecuritySettings(creds_path)

List Security Settings
all_ss = ss.list_security_settings(location_id)

Get Security Settings
my_ss = ss.get_security_settings(my_ss.name)

Create Security Settings from Obj
my_new_ss = my_ss
my_new_ss.display_name = "SCRAPI SS UPDATE"
my_new_ss.retention_window_days = 10
result = ss.create_security_settings(location_id, my_new_ss)

Update Security Settings
result = ss.update_security_settings(my_ss.name, retention_window_days=30, display_name='SCRAPI SS API UPDATE')

Delete Security Settings
result = ss.delete_security_settings(my_ss.name)

* Feature/security settings by kmaphoenix in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/28

Utterance Generator Utils
This class is an extension of the UtteranceGenerator ML class introduced in [Release v1.1.0](https://github.com/GoogleCloudPlatform/dfcx-scrapi/releases/tag/1.1.0).
This class provides additional methods that wrap the base class and allow the user to perform tasks like:
- Create New Training Phrases
- Create Test Dataset
- Create Synthetic Dataset

Each of these are ML-powered, synthetically generated datasets meant to bootstrap your bot building experience or further automate the testing and QA of your bots.

py
from dfcx_scrapi.tools.utterance_generator_util import UtteranceGeneratorUtils
ug = UtteranceGeneratorUtils(creds_path=creds_path)

Create New Training Phrases
df = ug.create_new_training_phrases(agent_id, ['context.people_names'], 10)
df.head(2)

Create Test Dataset
df = ug.create_test_dataset(agent_id, ['context.people_names'], 10)
df.head(2)

Create Synthetic Dataset
df = ug.create_synthetic_dataset(agent_id, ['context.people_names'], 10)
df.head(2)

* Feature/utterance_gen_util by Hgithubacct in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/15

Bug Fixes
- adding src/dfcx_scrapi/core_async/__init__.py by jmound in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/21

Enhancements
- Deprecated various methods across classes that were dependent on Python `Requests` library and implemented the methods based on the core Dialogflow CX library
- Updated Dataframe outputs in `Intents.bulk_intent_to_df()` to provide a more consistent naming schema throughout the library

Docs
- Updated licensing throughout library
- Various Doc strings and Styles updated throughout the library
- Renamed `tools/utterance_generator_utils.py` -> `tools/utterance_generator_util.py` for name space consistency
- Renamed `tools/webhook_utils.py` -> `tools/webhook_util.py` for name space consistency
- Moved `soop/project.py` -> `core/project.py`
- Deleted unused `soop` folder
- Renamed `tools/analysis_util` -> `tools/levenshtein.py` for more appropriate visibility

New Contributors
* jmound made their first contribution in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/21
* Hgithubacct made their first contribution in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/15
* Greenford made their first contribution in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/15

Minor Patch - What's Changed
* Adding bug report template by kmaphoenix in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/16
* updating doc strings in create_intent by kmaphoenix in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/17
* add language_code to bulk create intent from df by zya-codes in https://github.com/GoogleCloudPlatform/dfcx-scrapi/pull/14

**Full Changelog**: https://github.com/GoogleCloudPlatform/dfcx-scrapi/compare/1.1.0...1.1.1

Features
Semantic Clustering
We've introduced a new class that provides the ability to perform Semantic Clustering on Training Phrases that are currently in Dialogflow CX, or that you plan to put in Dialogflow CX. The motivation behind this class is to provide a base "engine" for building tools that allow Bot Builders and Designers with easier methods for determining where their Intents may be "heavy" or "light" with training data. By performing Semantic Clustering, you can quickly pinpoint clusters of Training Data to be groomed from your CX Agent that could be causing conflicts with other Intents.

We'll be extending this class with more tools in the future.
Try out the new features here: [SemanticClustering](https://github.com/GoogleCloudPlatform/dfcx-scrapi/blob/main/src/dfcx_scrapi/tools/semantic_clustering.py)

Utterance Generator
We're excited to introduce another new class that enables users to quickly generate synthetic Training Phrases based on a single input phrase. The motivation behind this class was to provide a way for Bot Builders and Designers to "bootstrap" their Dialogflow CX Agents with an appropriate amount of training data to move to a Production deployment. Once in Production, synthetic data could be swapped out with "real-world" data as it is collected and analyzed for a more fine tuned experience.

In addition, this class can be used to "bolster" Intents that have very few Training Phrases. This is especially helpful when you have high variance in your Training Phrase distribution among Intents, in particular when the of Training Phrases in your smallest Intents is not at least 10% of the of Training Phrases in your largest intents.

We'll be extending this class with more tools in the future.
Try out the new features here: [UtteranceGenerator](https://github.com/GoogleCloudPlatform/dfcx-scrapi/blob/main/src/dfcx_scrapi/core_ml/utterance_generator.py)

Test Cases (Sync and Async)
Test Cases are finally here! SCRAPI now has support for all Dialogflow CX Test Case features and functionality in both synchronous and asynchronous classes. The motivation behind including the Async class was primarily due to the large scale at which Test Cases can grow to, which can cause running a large set of Test Cases to take quite some time. Allowing these to be run Asynchronously can provide the developer more flexibility to retrieve the results when they are ready, rather than waiting for a large batch of them to return all at once.

You can find the new classes here:
[TestCases](https://github.com/GoogleCloudPlatform/dfcx-scrapi/blob/main/src/dfcx_scrapi/core/test_cases.py)
[TestCasesAsync](https://github.com/GoogleCloudPlatform/dfcx-scrapi/blob/main/src/dfcx_scrapi/core_async/test_cases.py)

Bulk Update Entity From DataFrame
Added a ability to provide bulk updates to Entities via DataFrame, similar to the existing bulk create functionality

Bug Fixes
- Fixed a bug in `core/conversations.py` that forced the input of `creds_path` arg

Enhancements
- Added `language_code` support to `TransitionRouteGroups.update_transition_route_group()`
- Added `language_code` support to `EntityTypes.create_entity_type()`
- Added `language_code` support to `DataframeFunctions.bulk_create_entity_from_dataframe()`
- Added `language_code` support to `EntityTypes.update_entity_type()`

Docs
- Updated docs for `EntityType.create_entity_type()`
- Updated docs for `DataframeFunctions.bulk_create_entity_from_dataframe()`
- Updated `requirements.txt`

Misc.
- README updates
- Minor .gitignore updates

Features
Multi Threaded Intent Detection
The `DialogflowConversation` Class now supports Multi Threaded Intent Detection via Python [threading](https://docs.python.org/3/library/threading.html)!
This will enable hundreds or thousands of tests to be run in parallel, significantly expediting Intent Detection tasks for QA and Operation teams. See `DialogflowConversation.run_intent_detection()` for more information.

Intents Language Code Support
All methods in the `Intents` Class have been updated to include `language_code` as an optional arg (where applicable) to allow for Intent CRUD operations across multi-lingual CX Agents.

Bug Fixes
- Fixed an issue in `StatsUtil.stats()` that caused the `agent_id` to not be passed into the underlying functions appropriately
- Fixed broken link in `examples/template.ipynb`

Enhancements
- Additional error handling in `ChangeHistory.change_history_to_dataframe()` to account for the scenario where no Change History results are present in the provided CX Agent
- Added `chunk_size` and `rate_limit` args to `DialogflowConversation.run_intent_detection()` to allow for more granular control over the amount of data sent to the Detect Intent API in parallel
- Added Boolean flag for `checkpoints` in `DialogflowConversation.reply()` to provide an option for users to obtain timestamp information for debugging purposes

Docs
- Updated docstrings for `DataframeFunctions.bulk_update_intent_from_dataframe()` which was missing `language_code` as an optional arg

Misc.
- Updated Docker image to be specific vs. latest due to issues in `3.10.x`
- Updated Makefile

Features
- Added `AnalysisUtils`, a Class for building methods and functions related to the in-depth comparison and analysis of Intents and Training Phrases in Dialogflow CX Agents
- Added `calc_tp_distances()` function in `AnalysisUtils` which performs Levenshtein Distance and Ratio comparison between 2 intents

Bug Fixes
- Fixed an issue in `SearchUtil.find_list_parameters()` that was causing a failure when iterating through the Flows map

Enhancements
- Restructured the `examples/` folder to contain a hierarchy based on the intended use of the Notebooks and scripts
- Added an `.ipynb` template to use as a guideline and best practice for authoring new examples to include in the repo
- Added several new Python Notebook examples with instructions in the `examples/` folder
- Updated all Python Notebooks to adhere to new template structure