Features
========
- [\3765](https://github.com/rasahq/rasa/issues/3765): Add support for entities with roles and grouping of entities in Rasa NLU.
You can now define a role and/or group label in addition to the entity type for entities.
Use the role label if an entity can play different roles in your assistant.
For example, a city can be a destination or a departure city.
The group label can be used to group multiple entities together.
For example, you could group different pizza orders, so that you know what toppings goes with which pizza and
what size which pizza has.
For more details see entities-roles-groups.
To fill slots from entities with a specific role/group, you need to either use forms or use a custom action.
We updated the tracker method `get_latest_entity_values` to take an optional role/group label.
If you want to use a form, you can add the specific role/group label of interest to the slot mapping function
`from_entity` (see forms).
<div class="admonition note">
Composite entities are currently just supported by the \`diet-classifier\` and \`CRFEntityExtractor\`.
</div>
- [\5465](https://github.com/rasahq/rasa/issues/5465): Update training data format for NLU to support entities with a role or group label.
You can now specify synonyms, roles, and groups of entities using the following data format:
Markdown:
{.sourceCode .none}
[LA]{"entity": "location", "role": "city", "group": "CA", "value": "Los Angeles"}
JSON:
{.sourceCode .none}
"entities": [
{
"start": 10,
"end": 12,
"value": "Los Angeles",
"entity": "location",
"role": "city",
"group": "CA",
}
]
The markdown format `[LA](location:Los Angeles)` is deprecated. To update your training data file just
execute the following command on the terminal of your choice:
`sed -i .deprecated -E 's/\[(.*)\]\((.*):(.*)\)/\[\1\]\{"entity": "\2", "value": "\3"\}/g' nlu.md`
For more information about the new data format see training-data-format.
Improvements
============
- [\2224](https://github.com/rasahq/rasa/issues/2224): Suppressed `pika` logs when establishing the connection. These log messages
mostly happened when Rasa X and RabbitMQ were started at the same time. Since RabbitMQ
can take a few seconds to initialize, Rasa X has to re-try until the connection is
established.
In case you suspect a different problem (such as failing authentication) you can
re-enable the `pika` logs by setting the log level to `DEBUG`. To run Rasa Open
Source in debug mode, use the `--debug` flag. To run Rasa X in debug mode, set the
environment variable `DEBUG_MODE` to `true`.
- [\3419](https://github.com/rasahq/rasa/issues/3419): Include the source filename of a story in the failed stories
Include the source filename of a story in the failed stories to make it easier to identify the file which contains the failed story.
- [\5544](https://github.com/rasahq/rasa/issues/5544): Add confusion matrix and "confused\_with" to response selection evaluation
If you are using ResponseSelectors, they now produce similiar outputs during NLU evaluation. Misclassfied responses are listed in a "confused\_with" attribute in the evaluation report. Similiarily, a confusion matrix of all responses is plotted.
- [\5578](https://github.com/rasahq/rasa/issues/5578): Added `socketio` to the compatible channels for reminders-and-external-events.
- [\5595](https://github.com/rasahq/rasa/issues/5595): Update `POST /model/train` endpoint to accept retrieval action responses
at the `responses` key of the JSON payload.
- [\5627](https://github.com/rasahq/rasa/issues/5627): All Rasa Open Source images are now using Python 3.7 instead of Python 3.6.
- [\5635](https://github.com/rasahq/rasa/issues/5635): Update dependencies based on the `dependabot` check.
- [\5636](https://github.com/rasahq/rasa/issues/5636): Add dropout between `FFNN` and `DenseForSparse` layers in `DIETClassifier`,
`ResponseSelector` and `EmbeddingIntentClassifier` controlled by `use_dense_input_dropout` config parameter.
- [\5646](https://github.com/rasahq/rasa/issues/5646): `DIETClassifier` only counts as extractor in `rasa test` if it was actually trained for entity recognition.
- [\5669](https://github.com/rasahq/rasa/issues/5669): Remove regularization gradient for variables that don't have prediction gradient.
- [\5672](https://github.com/rasahq/rasa/issues/5672): Raise a warning in `CRFEntityExtractor` and `DIETClassifier` if entities are not correctly annotated in the
training data, e.g. their start and end values do not match any start and end values of tokens.
- [\5690](https://github.com/rasahq/rasa/issues/5690): Add `full_retrieval_intent` property to `ResponseSelector` rankings
- [\5717](https://github.com/rasahq/rasa/issues/5717): Change default values for hyper-parameters in `EmbeddingIntentClassifier` and `DIETClassifier`
Use `scale_loss=False` in `DIETClassifier`. Reduce the number of dense dimensions for sparse features of text from 512 to 256 in `EmbeddingIntentClassifier`.
Bugfixes
========
- [\5230](https://github.com/rasahq/rasa/issues/5230): Fixed issue where posting to certain callback channel URLs would return a 500 error on successful posts due to invalid response format.
- [\5475](https://github.com/rasahq/rasa/issues/5475): One word can just have one entity label.
If you are using, for example, `ConveRTTokenizer` words can be split into multiple tokens.
Our entity extractors assign entity labels per token. So, it might happen, that a word, that was split into two tokens,
got assigned two different entity labels. This is now fixed. One word can just have one entity label at a time.
- [\5509](https://github.com/rasahq/rasa/issues/5509): An entity label should always cover a complete word.
If you are using, for example, `ConveRTTokenizer` words can be split into multiple tokens.
Our entity extractors assign entity labels per token. So, it might happen, that just a part of a word has
an entity label. This is now fixed. An entity label always covers a complete word.
- [\5574](https://github.com/rasahq/rasa/issues/5574): Fixed an issue that happened when metadata is passed in a new session.
Now the metadata is correctly passed to the ActionSessionStart.
- [\5672](https://github.com/rasahq/rasa/issues/5672): Updated Python dependency `ruamel.yaml` to `>=0.16`. We recommend to use at least
`0.16.10` due to the security issue
[CVE-2019-20478](https://nvd.nist.gov/vuln/detail/CVE-2019-20478) which is present in
in prior versions.
Miscellaneous internal changes
==============================
- \5556, \5587, \5614, \5631, \5633