Rasa

Deprecations and Removals
- [6410](https://github.com/rasahq/rasa/issues/6410): `Domain.random_template_for` is deprecated and will be removed in Rasa Open Source
3.0.0. You can alternatively use the `TemplatedNaturalLanguageGenerator`.

`Domain.action_names` is deprecated and will be removed in Rasa Open Source
3.0.0. Please use `Domain.action_names_or_texts` instead.
- [7458](https://github.com/rasahq/rasa/issues/7458): Interfaces for `Policy.__init__` and `Policy.load` have changed.
See [migration guide](./migration-guide.mdxrasa-21-to-rasa-22) for details.
- [7495](https://github.com/rasahq/rasa/issues/7495): Deprecate training and test data in Markdown format. This includes:
- reading and writing of story files in Markdown format
- reading and writing of NLU data in Markdown format
- reading and writing of retrieval intent data in Markdown format

Support for Markdown data will be removed entirely in Rasa Open Source 3.0.0.

Please convert your existing Markdown data by using the commands
from the [migration guide](./migration-guide.mdxrasa-21-to-rasa-22):

bash
rasa data convert nlu -f yaml --data={SOURCE_DIR} --out={TARGET_DIR}
rasa data convert nlg -f yaml --data={SOURCE_DIR} --out={TARGET_DIR}
rasa data convert core -f yaml --data={SOURCE_DIR} --out={TARGET_DIR}

- [7529](https://github.com/rasahq/rasa/issues/7529): `Domain.add_categorical_slot_default_value`, `Domain.add_requested_slot`
and `Domain.add_knowledge_base_slots` are deprecated and will be removed in Rasa Open
Source 3.0.0. Their internal versions are now called during the Domain creation.
Calling them manually is no longer required.

Features
- [6971](https://github.com/rasahq/rasa/issues/6971): Incremental training of models in a pipeline is now supported.

If you have added new NLU training examples or new stories/rules for
dialogue manager, you don't need to train the pipeline from scratch.
Instead, you can initialize the pipeline with a previously trained model
and continue finetuning the model on the complete dataset consisting of
new training examples. To do so, use `rasa train --finetune`. For more
detailed explanation of the command, check out the docs on [incremental
training](./command-line-interface.mdxincremental-training).

Added a configuration parameter `additional_vocabulary_size` to
[`CountVectorsFeaturizer`](./components.mdxcountvectorsfeaturizer)
and `number_additional_patterns` to [`RegexFeaturizer`](./components.mdxregexfeaturizer).
These parameters are useful to configure when using incremental training for your pipelines.
- [7408](https://github.com/rasahq/rasa/issues/7408): Add the option to use cross-validation to the
[`POST /model/test/intents`](/pages/http-apioperation/testModelIntent) endpoint.
To use cross-validation specify the query parameter `cross_validation_folds` in addition
to the training data in YAML format.

Add option to run NLU evaluation
([`POST /model/test/intents`](/pages/http-apioperation/testModelIntent)) and
model training ([`POST /model/train`](/pages/http-apioperation/trainModel))
asynchronously.
To trigger asynchronous processing specify
a callback URL in the query parameter `callback_url` which Rasa Open Source should send
the results to. This URL will also be called in case of errors.
- [7496](https://github.com/rasahq/rasa/issues/7496): Make [TED Policy](./policies.mdx#ted-policy) an end-to-end policy. Namely, make it possible to train TED on stories that contain
intent and entities or user text and bot actions or bot text.
If you don't have text in your stories, TED will behave the same way as before.
Add possibility to predict entities using TED.

Here's an example of a dialogue in the Rasa story format:

rasa-yaml
stories:
- story: collect restaurant booking info name of the story - just for debugging
steps:
- intent: greet user message with no entities
- action: utter_ask_howcanhelp action that the bot should execute
- intent: inform user message with entities
entities:
- location: "rome"
- price: "cheap"
- bot: On it actual text that bot can output
- action: utter_ask_cuisine
- user: I would like [spanish](cuisine). actual text that user input
- action: utter_ask_num_people


Some model options for `TEDPolicy` got renamed.
Please update your configuration files using the following mapping:

| Old model option | New model option |
|-----------------------------|--------------------------------------------------------|
|transformer_size |dictionary “transformer_size” with keys |
| |“text”, “action_text”, “label_action_text”, “dialogue” |
|number_of_transformer_layers |dictionary “number_of_transformer_layers” with keys |
| |“text”, “action_text”, “label_action_text”, “dialogue” |
|dense_dimension |dictionary “dense_dimension” with keys |
| |“text”, “action_text”, “label_action_text”, “intent”, |
| |“action_name”, “label_action_name”, “entities”, “slots”,|
| |“active_loop” |

Improvements
- [3998](https://github.com/rasahq/rasa/issues/3998): Added a message showing the location where the failed stories file was saved.
- [7232](https://github.com/rasahq/rasa/issues/7232): Add support for the top-level response keys `quick_replies`, `attachment` and `elements` refered to in `rasa.core.channels.OutputChannel.send_reponse`, as well as `metadata`.
- [7257](https://github.com/rasahq/rasa/issues/7257): Changed the format of the histogram of confidence values for both correct and incorrect predictions produced by running `rasa test`.
- [7284](https://github.com/rasahq/rasa/issues/7284): Run [`bandit`](https://bandit.readthedocs.io/en/latest/) checks on pull requests.
Introduce `make static-checks` command to run all static checks locally.
- [7397](https://github.com/rasahq/rasa/issues/7397): Add `rasa train --dry-run` command that allows to check if training needs to be performed
and what exactly needs to be retrained.
- [7408](https://github.com/rasahq/rasa/issues/7408): [`POST /model/test/intents`](/pages/http-api#operation/testModelIntent) now returns
the `report` field for `intent_evaluation`, `entity_evaluation` and
`response_selection_evaluation` as machine-readable JSON payload instead of string.
- [7436](https://github.com/rasahq/rasa/issues/7436): Make `rasa data validate stories` work for end-to-end.

The `rasa data validate stories` function now considers the tokenized user text instead of the plain text that is part of a state.
This is closer to what Rasa Core actually uses to distinguish states and thus captures more story structure problems.

Bugfixes
- [6804](https://github.com/rasahq/rasa/issues/6804): Rename `language_list` to `supported_language_list` for `JiebaTokenizer`.
- [7244](https://github.com/rasahq/rasa/issues/7244): A `float` slot returns unambiguous values - `[1.0, <value>]` if successfully converted, `[0.0, 0.0]` if not.
This makes it possible to distinguish an empty float slot from a slot set to `0.0`.
:::caution
This change is model-breaking. Please retrain your models.
:::
- [7306](https://github.com/rasahq/rasa/issues/7306): Fix an erroneous attribute for Redis key prefix in `rasa.core.tracker_store.RedisTrackerStore`: 'RedisTrackerStore' object has no attribute 'prefix'.
- [7407](https://github.com/rasahq/rasa/issues/7407): Remove token when its text (for example, whitespace) can't be tokenized by LM tokenizer (from `LanguageModelFeaturizer`).
- [7408](https://github.com/rasahq/rasa/issues/7408): Temporary directories which were created during requests to the [HTTP API](http-api.mdx)
are now cleaned up correctly once the request was processed.
- [7422](https://github.com/rasahq/rasa/issues/7422): Add option `use_word_boundaries` for `RegexFeaturizer` and `RegexEntityExtractor`. To correctly process languages such as Chinese that don't use whitespace for word separation, the user needs to add the `use_word_boundaries: False` option to those two components.
- [7529](https://github.com/rasahq/rasa/issues/7529): Correctly fingerprint the default domain slots. Previously this led to the issue
that `rasa train core` would always retrain the model even if the training data hasn't
changed.

Improved Documentation
- [7313](https://github.com/rasahq/rasa/issues/7313): Return the "Migrate from" entry to the docs sidebar.

Miscellaneous internal changes
- [7167](https://github.com/rasahq/rasa/issues/7167)

_Pre-release version_

Improvements
- [7426](https://github.com/rasahq/rasa/issues/7426): Removed `multidict` from the project dependencies. `multidict` continues to be a second
order dependency of Rasa Open Source but will be determined by the dependencies which
use it instead of by Rasa Open Source directly.

This resolves issues like the following:

bash
sanic 20.9.1 has requirement multidict==5.0.0, but you'll have multidict 4.6.0 which is incompatible.


Bugfixes
- [7316](https://github.com/rasahq/rasa/issues/7316): `SingleStateFeaturizer` checks whether it was trained with `RegexInterpreter` as
nlu interpreter. If that is the case, `RegexInterpreter` is used during prediction.
- [7390](https://github.com/rasahq/rasa/issues/7390): Make sure the `responses` are synced between NLU training data and the Domain even if there're no retrieval intents in the NLU training data.
- [7417](https://github.com/rasahq/rasa/issues/7417): Categorical slots will have a default value set when just updating nlg data in the domain.

Previously this resulted in `InvalidDomain` being thrown.
- [7418](https://github.com/rasahq/rasa/issues/7418): - Preserve `domain` slot ordering while dumping it back to the file.
- Preserve multiline `text` examples of `responses` defined in `domain` and `NLU` training data.

Bugfixes
- [7235](https://github.com/rasahq/rasa/issues/7235): Slots that use `initial_value` won't cause rule contradiction errors when `conversation_start: true` is used. Previously, two rules that differed only in their use of `conversation_start` would be flagged as contradicting when a slot used `initial_value`.

In checking for incomplete rules, an action will be required to have set _only_ those slots that the same action has set in another rule. Previously, an action was expected to have set also slots which, despite being present after this action in another rule, were not actually set by this action.
- [7345](https://github.com/rasahq/rasa/issues/7345): Fixed Rasa Open Source not being able to fetch models from certain URLs.

Bugfixes
- [7338](https://github.com/rasahq/rasa/issues/7338): Sender ID is correctly set when copying the tracker and sending it to the action server (instead of sending the `default` value). This fixes a problem where the action server would only retrieve trackers with a `sender_id` `default`.

Deprecations and Removals
- [7136](https://github.com/rasahq/rasa/issues/7136): The [`Policy`](policies.mdx) interface was changed to return a `PolicyPrediction` object when
`predict_action_probabilities` is called. Returning a list of probabilities directly
is deprecated and support for this will be removed in Rasa Open Source 3.0.

You can adapt your custom policy by wrapping your probabilities in a `PolicyPrediction`
object:

python
from rasa.core.policies.policy import Policy, PolicyPrediction
... other imports

def predict_action_probabilities(
self,
tracker: DialogueStateTracker,
domain: Domain,
interpreter: NaturalLanguageInterpreter,
**kwargs: Any,
) -> PolicyPrediction:
probabilities = ... an action prediction of your policy
return PolicyPrediction(probabilities, "policy_name", policy_priority=self.priority)


The same change was applied to the `PolicyEnsemble` interface. Instead of returning
a tuple of action probabilities and policy name, it is now returning a
`PolicyPrediction` object. Support for the old `PolicyEnsemble` interface will be
removed in Rasa Open Source 3.0.

:::caution
This change is model-breaking. Please retrain your models.

:::
- [7263](https://github.com/rasahq/rasa/issues/7263): The [Pika Event Broker](event-brokers.mdx#pika-event-broker) no longer supports
the environment variables `RABBITMQ_SSL_CA_FILE` and `RABBITMQ_SSL_KEY_PASSWORD`.
You can alternatively specify `RABBITMQ_SSL_CA_FILE` in the RabbitMQ connection URL as
described in the
[RabbitMQ documentation](https://www.rabbitmq.com/uri-query-parameters.html).

yaml-rasa title="endpoints.yml
event_broker:
type: pika
url: "amqps://user:passwordhost?cacertfile=path_to_ca_cert&password=private_key_password"
queues:
- my_queue



Support for `RABBITMQ_SSL_KEY_PASSWORD` was removed entirely.

The method [`Event Broker.close`](event-brokers.mdx) was changed to be asynchronous.
Support for synchronous implementations will be removed in Rasa Open Source 2.2.0.
To adapt your implementation add the `async` keyword:

python
from rasa.core.brokers.broker import EventBroker

class MyEventBroker(EventBroker):

async def close(self) -> None:
clean up event broker resources


Features
- [7136](https://github.com/rasahq/rasa/issues/7136): [Policies](policies.mdx) can now return obligatory and optional events as part of their
prediction. Obligatory events are always applied to the current conversation tracker.
Optional events are only applied to the conversation tracker in case the policy wins.

Improvements
- [4341](https://github.com/rasahq/rasa/issues/4341): Changed `Agent.load` method to support `pathlib` paths.
- [5715](https://github.com/rasahq/rasa/issues/5715): If you are using the feature [Entity Roles and Groups](nlu-training-data.mdx#entities-roles-and-groups), you should now also list the roles and groups
in your domain file if you want roles and groups to influence your conversations. For example:
yaml-rasa
entities:
- city:
roles:
- from
- to
- name
- topping:
groups:
- 1
- 2
- size:
groups:
- 1
- 2


Entity roles and groups can now influence dialogue predictions. For more information see the section
[Entity Roles and Groups influencing dialogue predictions](nlu-training-data.mdxentity-roles-and-groups-influencing-dialogue-predictions).
- [6285](https://github.com/rasahq/rasa/issues/6285): Predictions of the [`FallbackClassifier`](components.mdx#fallbackclassifier) are
ignored when
[evaluating the NLU model](testing-your-assistant.mdxevaluating-an-nlu-model)
Note that the `FallbackClassifier` predictions still apply to
[test stories](testing-your-assistant.mdxwriting-test-stories).
- [6474](https://github.com/rasahq/rasa/issues/6474): Adapt the training data reader and emulator for wit.ai to their latest format.
Update the instructions in the
[migrate from wit.ai documentation](migrate-from/facebook-wit-ai-to-rasa.mdx)
to run Rasa Open Source in wit.ai emulation mode.
- [6498](https://github.com/rasahq/rasa/issues/6498): Adding configurable prefixes to Redis [Tracker](./tracker-stores.mdx) and [Lock Stores](./lock-stores.mdx) so that a single Redis instance (and logical DB) can support multiple conversation trackers and locks.
By default, conversations will be prefixed with `tracker:...` and all locks prefixed with `lock:...`. Additionally, you can add an alphanumeric-only `prefix: value` in `endpoints.yml` such that keys in redis will take the form `value:tracker:...` and `value:lock:...` respectively.
- [6571](https://github.com/rasahq/rasa/issues/6571): Log the model's relative path when using CLI commands.
- [6852](https://github.com/rasahq/rasa/issues/6852): Adds the option to configure whether extracted entities should be split by comma (`","`) or not. The default behaviour is `True` - i.e. split any list of extracted entities by comma. This makes sense for a list of ingredients in a recipie, for example `"avocado, tofu, cauliflower"`, however doesn't make sense for an address such as `"Schönhauser Allee 175, 10119 Berlin, Germany"`.

In the latter case, add a new option to your config, e.g. if you are using the `DIETClassifier` this becomes:

yaml
...
- name: DIETClassifier
split_entities_by_comma: False
...


in which case, none of the extracted entities will be split by comma. To switch it on/off for specific entity types you can use:

yaml
...
- name: DIETClassifier
split_entities_by_comma:
address: True
ingredient: False
...


where both `address` and `ingredient` are two entity types.

This feature is also available for `CRFEntityExtractor`.
- [6860](https://github.com/rasahq/rasa/issues/6860): Fetching test stories from the HTTP API endpoint
`GET /conversations/<conversation_id>/story` no longer triggers an update
of the
[conversation session](./domain.mdxsession-configuration).

Added a new boolean query parameter `all_sessions` (default: `false`) to the
[HTTP API](./http-api.mdx) endpoint for fetching test stories
(`GET /conversations/<conversation_id>/story`).

When setting `?all_sessions=true`, the endpoint returns test stories for all
conversation sessions for `conversation_id`.
When setting `?all_sessions=all_sessions`, or when omitting the `all_sessions`
parameter, a single test story is returned for `conversation_id`. In cases where
multiple conversation sessions exist, only the last story is returned.

Specifying the `retrieve_events_from_previous_conversation_sessions`
kwarg for the [Tracker Store](./tracker-stores.mdx) class is deprecated and will be
removed in Rasa Open Source 3.0. Please use the `retrieve_full_tracker()` method
instead.
- [6865](https://github.com/rasahq/rasa/issues/6865): Improve the `rasa data convert nlg` command and introduce the `rasa data convert responses` command
to simplify the migration from pre-2.0 response selector format to the new format.
- [6966](https://github.com/rasahq/rasa/issues/6966): Added warning for when an option is provided for a [component](components.mdx) that is not listed as a key in the defaults for that component.
- [6977](https://github.com/rasahq/rasa/issues/6977): [Forms](forms.mdx) no longer reject their execution before a potential custom
action for validating / extracting slots was executed.
Forms continue to reject in two cases automatically:
- A slot was requested to be filled, but no slot mapping applied to the latest user
message and there was no custom action for potentially extracting other slots.
- A slot was requested to be filled, but the custom action for validating / extracting
slots didn't return any slot event.

Additionally you can also reject the form execution manually by returning a
`ActionExecutionRejected` event within your custom action for validating / extracting
slots.
- [7027](https://github.com/rasahq/rasa/issues/7027): Remove dependency between `ConveRTTokenizer` and `ConveRTFeaturizer`. The `ConveRTTokenizer` is now deprecated, and the
`ConveRTFeaturizer` can be used with any other `Tokenizer`.

Remove dependency between `HFTransformersNLP`, `LanguageModelTokenizer`, and `LanguageModelFeaturizer`. Both
`HFTransformersNLP` and `LanguageModelTokenizer` are now deprecated. `LanguageModelFeaturizer` implements the behavior
of the stack and can be used with any other `Tokenizer`.
- [7061](https://github.com/rasahq/rasa/issues/7061): Gray out "Download" button in Rasa Playground when the project is not yet ready to be downloaded.
- [7068](https://github.com/rasahq/rasa/issues/7068): Slot mappings for [Forms](forms.mdx) in the domain are now optional. If you do not
provide any slot mappings as part of the domain, you need to provide
[custom slot mappings](forms.mdxcustom-slot-mappings) through a custom action.
A form without slot mappings is specified as follows:

rasa-yaml
forms:
my_form:
no mappings


The action for [forms](forms.mdx) can now be overridden by defining a custom action
with the same name as the form. This can be used to keep using the deprecated
Rasa Open Source `FormAction` which is implemented within the Rasa SDK. Note that it is
**not** recommended to override the form action for anything else than using the
deprecated Rasa SDK `FormAction`.
- [7102](https://github.com/rasahq/rasa/issues/7102): Changed the default model weights loaded for `HFTransformersNLP` component.

Use a [language agnostic sentence embedding model](https://tfhub.dev/google/LaBSE/1)
as the default model. These model weights should help improve performance on
intent classification and response selection.
- [7122](https://github.com/rasahq/rasa/issues/7122): Add validations for [slot mappings](forms.mdx#slot-mappings).
If a slot mapping is not valid, an `InvalidDomain` error is raised.
- [7132](https://github.com/rasahq/rasa/issues/7132): Adapt the training data reader and emulator for LUIS to
[their latest format](https://westcentralus.dev.cognitive.microsoft.com/docs/services/luis-endpoint-api-v3-0/)
and add support for roles.
Update the instructions in the
["Migrate from LUIS" documentation page](migrate-from/microsoft-luis-to-rasa.mdx)
to reflect the recent changes made to the UI of LUIS.
- [7160](https://github.com/rasahq/rasa/issues/7160): Adapt the [training data reader and emulator for DialogFlow](migrate-from/google-dialogflow-to-rasa.mdx) to
[their latest format](https://cloud.google.com/dialogflow/es/docs/reference/rest/v2/DetectIntentResponse)
and add support for regex entities.
- [7263](https://github.com/rasahq/rasa/issues/7263): The [Pika Event Broker](event-brokers.mdx#pika-event-broker) was reimplemented with
the `[aio-pika` library[(https://aio-pika.readthedocs.io/en/latest/). Messages will
now be published to RabbitMQ asynchronously which improves the prediction performance.
- [7278](https://github.com/rasahq/rasa/issues/7278): The confidence of the [`FallbackClassifier`](components.mdx#fallbackclassifier)
predictions is set to `1 - top intent confidence`.

Bugfixes
- [5974](https://github.com/rasahq/rasa/issues/5974): `ActionRestart` will now trigger `ActionSessionStart` as a followup action.
- [6582](https://github.com/rasahq/rasa/issues/6582): Fixed a bug with `rasa data split nlu` which caused the resulting train / test ratio to sometimes differ from the ratio specified by the user or by default.

The splitting algorithm ensures that every intent and response class appears in both the training and the test set. This means that each split must contain at least as many examples as there are classes, which for small datasets can contradict the requested training fraction. When this happens, the command issues a warning to the user that the requested training fraction can't be satisfied.
- [6721](https://github.com/rasahq/rasa/issues/6721): Fixed bug where slots with `influence_conversation=false` affected the action
prediction if they were set manually using the
`POST /conversations/<conversation_id/tracker/events` endpoint in the
[HTTP API](./http-api.mdx).
- [6760](https://github.com/rasahq/rasa/issues/6760): Update Pika event broker to be a separate process and make it use a
`multiprocessing.Queue` to send and process messages. This change should help
avoid situations when events stop being sent after a while.
- [6973](https://github.com/rasahq/rasa/issues/6973): Ignore rules when validating stories
- [6986](https://github.com/rasahq/rasa/issues/6986): - Updated Slack Connector for new Slack Events API
- [7001](https://github.com/rasahq/rasa/issues/7001): Update Rasa Playground "Download" button to work correctly depending on the current chat state.
- [7002](https://github.com/rasahq/rasa/issues/7002): Test stories can now contain both: normal intents and retrieval intents. The `failed_test_stories.yml`, generated by `rasa test`, also specifies the full retrieval intent now.
Previously `rasa test` would fail on test stories that specified retrieval intents.
- [7031](https://github.com/rasahq/rasa/issues/7031): The converter tool is now able to convert test stories that contain a number as entity type.
- [7034](https://github.com/rasahq/rasa/issues/7034): The converter tool now converts test stories and stories that contain full retrieval intents correctly.
Previously the response keys were deleted during conversion to YAML.
- [7204](https://github.com/rasahq/rasa/issues/7204): The slack connector requires a configuration for `slack_signing_secret` to make
the connector more secure. The configuration value needs to be added to your
`credentials.yml` if you are using the slack connector.
- [7246](https://github.com/rasahq/rasa/issues/7246): Fixed model fingerprinting - it should avoid some more unecessary retrainings now.
- [7253](https://github.com/rasahq/rasa/issues/7253): Fixed a problem when slots of type `text` or `list` were referenced by name only in
the training data and this was treated as an empty value. This means that the two
following stories are equivalent in case the slot type is `text`:

yaml
stories:
- story: Story referencing slot by name
steps:
- intent: greet
- slot_was_set:
- name

- story: Story referencing slot with name and value
steps:
- intent: greet
- slot_was_set:
- name: "some name"



Note that you still need to specify values for all other slot types as only `text`
and `list` slots are featurized in a binary fashion.

Improved Documentation
- [6973](https://github.com/rasahq/rasa/issues/6973): Correct data validation docs

Miscellaneous internal changes
- [6470](https://github.com/rasahq/rasa/issues/6470), [#7015](https://github.com/rasahq/rasa/issues/7015), [#7090](https://github.com/rasahq/rasa/issues/7090)