Functionality
Transformers new summarize() functionality
Some Transformers now have a summarize() function that summarizes the annotated corpus (i.e. annotated by a transform() call) in a way that gives the user a high-level view / interpretation of the annotated metadata.
New Transformers
We introduce several new Transformers: Classifier, Bag-of-Words Classifier, Ranker, Pairer, Paired Prediction, Paired Bag-of-Words Prediction, Fighting Words, and (Conversational) Forecaster (with variants: Bag-of-Words and CRAFT).
New TextProcessor
We introduce TextCleaner, which does text cleaning for online text data. This cleaner depends on the *clean-text* package.
Enhanced Conversation functionality
- Conversation.check_integrity() can be used to check if a conversation has a valid and intact reply-to chain (i.e. only one root utterance, every utterance specified by reply-to exists, etc)
- Conversation.print_conversation_structure() is a way of pretty-printing a Conversation's thread structure (whether displaying just its utterances' ids, texts, or other details is customizable)
- Conversation.get_chronological_utterance_list() provides a list of the Conversation's utterances sorted from earliest to latest timestamp
**Tree operations**
- Conversation.traverse() allows for Conversations to be traversed as a tree structure, e.g. breadth-first, depth-first, pre-order, post-order. Specifically, traverse() returns an iterator of Utterances or UtteranceNodes (a wrapper class for working with Utterances in a conversational tree setting)
- Conversation allows for subtree extraction using any arbitrary utterance in the Conversation as the new root
- Conversation.get_root_to_leaf_paths() returns all the root to leaf paths in the conversation tree
Other changes
Public-facing interface changes
- All Corpus objects now support a full set of all possible object iterators (e.g. User.iter_utterances() or Corpus.iter_users()) with selector functions (i.e. filters that select for the corpus object to be generated)
- Corpus has new methods for checking for the presence of corpus objects, e.g. corpus.has_utterance(), corpus.has_conversation(), corpus.has_user()
- A random User / Utterance / Conversation can be obtained from a Corpus with corpus.random_user() / corpus.random_utterance() / corpus.random_conversation()
- User objects now have ids, not names. Corpus.get_usernames() and User.name are deprecated (in favor of Corpus.get_user_ids() and User.id respectively) and print a warning when used.
- Corpora can be mutated to only include specific Conversations by using Corpus.filter_conversations_by()
- Corpus filtering by utterance is no longer supported to avoid encouraging Corpus mutations that break Conversation reply-to chains. Corpus.filter_utterances_by() is now deprecated and no longer usable.
- Corpus object (i.e. User, Utterance, Conversation) ids and metadata keys must now be strings or None. It used to be that any Hashable object could be used, but this posed problems for corpus dumping to and loading from jsons.
- Deletion of a metadata key for one object results in deletion of that metadata key for all objects of that object type
- Corpus.dump() automatically increments the version number of the Corpus by 1.
- Corpus.download() now has a *use_local* boolean parameter that allows offline users to skip the online check for a new dataset version and uses the local version by default.
- Fixed a bug where specified conversation and user metadata were not getting excluded correctly during Corpus initialisation step
- \_\_str\_\_ is now implemented to provide a concise human-readable string display of the Corpus object (that hides private variables)
- Fixed some bugs with Hypergraph motif counting
Internal changes
- Corpus initialisation and dumping have been heavily refactored to improve future maintainability.
- There is a new CorpusObject parent class that User, Utterance, and Conversation inherit from. This parent class implements some shared functionality for all Corpus objects.
- Corpus now uses a ConvokitIndex object to correctly track the metadata state of itself and its Corpus objects. Previously, this index was computed on the spot when Corpus.dump() was called, and referred to when loading a Corpus. However, any changes to a loaded Corpus object would not update the internal index of the Corpus, meaning the index could be inconsistent with the Corpus state.
- Corpus objects (Corpus, User, Utterance, Conversation) all use a ConvokitMeta object instead of a simple dict() for their metadata. This change is necessary to ensure that updates to the metadata (key additions / deletions) are reflected in ConvokitIndex. However, because ConvokitMeta inherits from the dict class, there is no change to how users should work with the .meta attribute.
- Users and Utterances now have 'owner' attributes to indicate the Corpus they belong to. This change is necessary for the maintaining of a consistent index. (Conversations have always had this attribute.)
- Introduces optional dependencies on the *clean-text* and *torch* packages for sanitizing text under the FightingWords Transformer and running a neural network as part of the Forecaster-CRAFT Transformer respectively.
- A single script for running all existing test suites has been created to speed up testing before deployment: *tests/run_all_tests.py*