Convokit

We're excited to announce the public release of **Convokit 3.0**!

The new version of ConvoKit now supports MongoDB as a backend choice for working with corpus data. This update provides several benefits, such as taking advantage of MongoDB's lazy loading to handle extremely large corpora, and ensuring resilience to unexpected crashes by continuously writing all changes to the database.

To learn more about using MongoDB as a backend choice, refer to our documentation at https://convokit.cornell.edu/documentation/storage_options.html.

Database Backend
Historically, ConvoKit allows you to work with conversational data directly in program memory through the Corpus class. Moreover, long term storage is provided by dumping the contents of a Corpus onto disk using the JSON format. This paradigm works well for distributing and storing static datasets, and for doing computations on conversational data that follow the pattern of doing computations on some or all of the data over a short time period and optionally storing these results on disk. For example, ConvoKit distributes datasets included with the library in JSON format, which you can load into program memory to explore and compute with.

In ConvoKit version 3.0.0, we introduce a new option for working with conversational data: the MongoDB backend. Consider a use case where you want to collect conversational data over a long time period and ensure you maintain a persistent representation of the dataset if your data collection program unexpectedly crashes. In the memory backend paradigm, this would require regularly dumping your corpus to JSON files, requiring repeated expensive write operations. On the other hand, with the new database backend, all your data is automatically saved for long term storage in the database as it is added to the corpus.

Documentation
Please refer to [this database setup document](https://convokit.cornell.edu/documentation/db_setup.html) to setup a mongoDB database and [this storage document](https://convokit.cornell.edu/documentation/storage_options.html) for a further explanation of how the database backend option works.

Tests
Updated tests to include db_mode testing.

Examples
Updated examples to include demonstration of db_mode usage.

Bug Fixes
- Fixed issue where `corpus.utterances` throws an error in `politenessAPI` as it should call `corpus.iter_utterances()` instead. Corpus items should not access their private variables and should use the public "getters" for access.
- Fixed bug in `coordination.py` for the usage of metadata mutability.
- Fixed issue in Pairer with `pair_mode` set to `maximize` causing the pairing function to return an integer, which causes an error in pairing objects.

Breaking Changes
Modified `ConvoKit.Metadata` to disallow any mutability to metadata fields. Implemented by returning deepcopy of metadata field storage every time the field is accessed. It is intended to align the behaviors between memory and DB modes. 197

Change Log
**Added:**
- Added DB backend mode to allow working with corpora using database as a supporting backend. 175 184
- Extended `__init__` in `model/corpus.py` with parameters for DB functionality. 175
- Updated `model/backendMapper` to separate memory and DB transactions. 175
- Introduces a new layer of abstraction between Corpus components (Utterance, Speaker, Conversation, ConvoKitMeta) and concrete data mapping. Data mapping is now handled by a BackendMapper instance variable in the Corpus. 169

**Changed:**
- Modified files in the ConvoKit model to support both memory mode and DB mode backends. 175
- Removed deprecated arguments and functions from ConvoKit model. 176
- Updated demo examples with older version of ConvoKit object references. 192

**Fixed:**
- Fixed usage of the mutability of metadata within `coordination.py`. 197
- Fixed issue in the Pairer module when `pair_mode` was set to `maximize`, causing the pairing function to return an integer and subsequently leading to an error. 197
- Fixed issue that caused `corpus.utterances` to throw an error within `politenessAPI`. 170
- Fixed FightingWords to allow overlapping classes. 189

**Python Version Requirement Update:**
- With Python 3.7 reached EOL (end of life) on June 27, 2023, ConvoKit now requires Python 3.8 or above.

This release adds support for Chinese politeness strategy extraction. Currently, ConvoKit's [politenessStrategies](https://convokit.cornell.edu/documentation/politenessStrategies.html) supports three politeness strategy collections covering two languages.

This release includes a new method `from_pandas` in the Corpus class that should simplify the Corpus creation process.

It generates a ConvoKit corpus from pandas dataframes of speakers, utterances, and conversations.

A notebook demonstrating the use of this method can be found [here](https://github.com/CornellNLP/Cornell-Conversational-Analysis-Toolkit/blob/master/examples/corpus_from_pandas.ipynb).

This release contains an implementation of the [Expected Conversational Context Framework](https://convokit.cornell.edu/documentation/expected_context_model.html), and [associated demos](https://github.com/CornellNLP/Cornell-Conversational-Analysis-Toolkit/tree/master/convokit/expected_context_framework/demos).

This release describes changes that have been implemented as part of the v2.4 release.

Public-facing functionality

ConvoKitMatrix and Vectors

Vectors and Matrices now get first-class treatment in ConvoKit. Vector data can now be stored in a ConvoKitMatrix object that is integrated with the Corpus and its objects, allowing for straightforward access from Corpus component objects, user-friendly display of vectors data, and more. Read our [introduction to vectors](https://github.com/CornellNLP/Cornell-Conversational-Analysis-Toolkit/blob/master/examples/vectors/vector_demo.ipynb) for more details.

Accordingly, we have re-implemented the relevant Transformers that were already using array or vector-like data to leverage on this new data structure, namely:
- PromptTypes
- HyperConvo
- BoWTransformer
- BoWClassifier - now renamed to VectorClassifier
- PairedBoW - now renamed to PairedVectorClassifier

The last two Transformers can now be used for any general vector data, as opposed to just bag-of-words vector data.

Metadata deletion
We have implemented a formal way to delete metadata attributes from a Corpus component object. Prior to this, metadata attributes were deleted from objects individually -- leading to possible inconsistencies between the ConvoKitIndex (that tracks what metadata attributes currently exist) and the Corpus component objects. To rectify this, we now **disallow deletion of metadata attributes from objects individually.** Such deletion should instead be carried out using the Corpus method `delete_metadata()`.

Other changes
- FightingWords and BoWTransformer now have default `text_func` values for the three main component types: utterance, speaker, and conversation.
- `corpus.iterate_by()` is now deprecated.
- The API of PromptTypes has been modified: rather than selecting types of prompt and response utterances to use in the constructor, we now give users the option to select prompts and responses as arguments to the `fit` and `transform` calls.

Other internal changes
- In light of SIGDIAL 2020, we have a new [video introduction](https://www.youtube.com/watch?v=nofzyxM4h1k) and [Jupyter notebook tutorial](https://github.com/CornellNLP/Cornell-Conversational-Analysis-Toolkit/blob/master/examples/Introduction_to_ConvoKit.ipynb) introducing new users to ConvoKit.
- ConvoKitIndex now tracks a list of class types for each metadata attribute, instead of a single class type. This will lead to changes in `index.json` during dumps of any currently existing corpora, but will have no compatibility issues with loading from existing corpora.
- We updated the following demos that make use of Vectors and PromptTypes: [PromptTypes](https://github.com/CornellNLP/Cornell-Conversational-Analysis-Toolkit/blob/master/examples/prompt-types/prompt-type-demo.ipynb) and [Predicting conversations gone awry](https://github.com/CornellNLP/Cornell-Conversational-Analysis-Toolkit/blob/master/examples/conversations-gone-awry/Conversations_Gone_Awry_Prediction.ipynb)