[0.17.0](https://github.com/recognai/rubrix/compare/v0.16.1...v0.17.0) (2022-08-22)
⚡ Highlights
Preparing a training set in the spaCy DocBin format
`prepare_for_training` is a method that prepares a dataset for training. Before `prepare_for_training` prepared the data for easily training Hugginface Transformers.
Now, you can prepare your training data for `spaCy` NER pipelines, thanks to our great community contributor ignacioct !
With the example below, you can export your Rubrix dataset into a Docbin, save it to disk, and then use it with the spacy train [command](https://spacy.io/api/cli#train).
python
import spacy
import rubrix as rb
from datasets import load_dataset
Load annotated dataset from Rubrix
rb_dataset = rb.load("ner_dataset")
Loading an spaCy blank language model to create the Docbin, as it works faster
nlp = spacy.blank("en")
After this line, the file will be stored in disk
rb_dataset.prepare_for_training(framework="spacy", lang=nlp).to_disk("train.spacy")
You can find a full example at: https://rubrix.readthedocs.io/en/v0.17.0/guides/cookbook.html#Train-a-spaCy-model-by-exporting-to-Docbin
Load large datasets using batches
Before this release, the `rb.load` method to read datasets from Python retrieved the full dataset. For large datasets, this could cause high memory consumption, network timeouts, and the inability to read datasets larger than the available memory.
Thanks to the awesome work by maxserras. Now it's possible to optimize memory consumption and avoid network timeouts when working with large datasets. To that end, a simple batch-iteration over the whole database can be done employing the `from_id` parameter in the `rb.load` method.
An example of reading the first 1000 records and the next batch of up to 1000 records:
python
import rubrix as rb
dataset_batch_1 = rb.load(name="example-dataset", limit=1000)
dataset_batch_2 = rb.load(name="example-dataset", limit=1000, id_from=dataset_batch_1[-1].id)
The reference to the `rb.load` method can be found at: https://rubrix.readthedocs.io/en/v0.17.0/reference/python/python_client.html#rubrix.load
Larger pagination sizes for faster bulk review and annotation
Using filters and search for data annotation and review, some users are able to filter and quickly review dozens of records in one go. To serve those users, it's now possible to see and bulk annotate 50 and 100 records in each page.
<img width="1679" alt="Screenshot 2022-08-25 at 10 33 58" src="https://user-images.githubusercontent.com/1107111/186616549-09280798-8a5e-4153-9680-febafd48d65c.png">
Copy record text to clipboard
Sometimes is useful to copy the text in records to use inspect it or process it with another application. Now, this is possible thanks to the feature request by our great community member and contributor Ankush-Chander !
<img width="1028" alt="Screenshot 2022-08-25 at 10 38 19" src="https://user-images.githubusercontent.com/1107111/186617710-f91a9be5-ad0b-4ee9-bbc9-449395a5a260.png">
Better error logging for generic errors
Thanks to work done by Ankush-Chander and frascuchon we now have more meaningful messages for generic server errors!
Features
* Add new pagination size ranges ([1667](https://github.com/recognai/rubrix/issues/1667)) ([5b4f1f2](https://github.com/recognai/rubrix/commit/5b4f1f290d8a1307186059be3df330a5136a9b69)), closes [#1578](https://github.com/recognai/rubrix/issues/1578)
* Allow `rb.load` fetch records in batches passing the `from_id` argument ([3e6344a](https://github.com/recognai/rubrix/commit/3e6344a7b2a6adce4a9289753cfb268f7bb9a0a8))
* Copy to clipboard the record text ([1625](https://github.com/recognai/rubrix/issues/1625)) ([d634a7b](https://github.com/recognai/rubrix/commit/d634a7bfc1bd972b3fb1e82de0a1a0fda4eb7e3b)), closes [#1616](https://github.com/recognai/rubrix/issues/1616)
* **Error Logging:** send error detail in response for generic server errors ([1648](https://github.com/recognai/rubrix/issues/1648)) ([ad17631](https://github.com/recognai/rubrix/commit/ad17631107a108010ad27895e0b47246205d0378))
* **Listeners:** allow using query params in the condition through search parameter ([1627](https://github.com/recognai/rubrix/issues/1627)) ([a0a245d](https://github.com/recognai/rubrix/commit/a0a245d936c139cd7a229c17246cd3f4ed2233e6)), closes [#1622](https://github.com/recognai/rubrix/issues/1622)
* `prepare_for_training` supports spacy ([1635](https://github.com/recognai/rubrix/issues/1635)) ([8587808](https://github.com/recognai/rubrix/commit/85878087144723695568ebde0d04e16fc9de3c1c))
Bug Fixes
* **Client:** reusing the inner `httpx` client ([1640](https://github.com/recognai/rubrix/issues/1640)) ([854a972](https://github.com/recognai/rubrix/commit/854a972e3ad0e3beac9bdc62d79fdf8d53e15f84)), closes [#1646](https://github.com/recognai/rubrix/issues/1646)
* **docker-compose.yaml:** default volume and disable disk threshold ([1656](https://github.com/recognai/rubrix/issues/1656)) ([05ae688](https://github.com/recognai/rubrix/commit/05ae6883e681a773cef280fc6cc1c3d115337924)), closes [#1275](https://github.com/recognai/rubrix/issues/1275)
* Encode rule name in Weak Labeling API requests ([1649](https://github.com/recognai/rubrix/issues/1649)) ([4634df8](https://github.com/recognai/rubrix/commit/4634df8a74b06020f0ab6cf386ad3bbd419b7cbd)), closes [#1645](https://github.com/recognai/rubrix/issues/1645)
* handle stream api connection errors gracefully ([1636](https://github.com/recognai/rubrix/issues/1636)) ([a106ec4](https://github.com/recognai/rubrix/commit/a106ec4541ae9c90e61c835173c11876d249e775)), closes [#1559](https://github.com/recognai/rubrix/issues/1559)
* Update progress bar when refreshing after adding new records ([1666](https://github.com/recognai/rubrix/issues/1666)) ([7e0d915](https://github.com/recognai/rubrix/commit/7e0d9156d614787996b6a5212bd4715132616a12)), closes [#1590](https://github.com/recognai/rubrix/issues/1590)
Documentation
* Add Slack support link in README's get started ([1688](https://github.com/recognai/rubrix/issues/1688)) ([bef010c](https://github.com/recognai/rubrix/commit/bef010cd6fb1c772c21a04c205cde64a7991bd9a))
* Adding Elasticsearch persistence to docker compose section ([1643](https://github.com/recognai/rubrix/issues/1643)) ([ecdc854](https://github.com/recognai/rubrix/commit/ecdc854a5ee5714a85ef3f87202eb102954667d8))
* spacy `DocBin` cookbook ([1642](https://github.com/recognai/rubrix/issues/1642)) ([bb98278](https://github.com/recognai/rubrix/commit/bb9827845053ff47e3575103a1f51097ace9f396)), closes [#420](https://github.com/recognai/rubrix/issues/420)
Visual enhancements
* Small visual adjustments for Text2Text record card ([1632](https://github.com/recognai/rubrix/issues/1632)) ([9c87cf1](https://github.com/recognai/rubrix/commit/9c87cf191093f1779f894f5fb55b2e1febe27123)), closes [#1138](https://github.com/recognai/rubrix/issues/1138)
* Improve card spacing ([1638](https://github.com/recognai/rubrix/issues/1638)) ([fd4016a](https://github.com/recognai/rubrix/commit/fd4016a6a654db54999ff91a3974d4616722c634)), closes [#1624](https://github.com/recognai/rubrix/issues/1624)
You can see all work included in the release here
- fix: Update progress bar when refreshing after adding new records (1666) by leiyre
- chore: configure miniconda for readthedocs builder by frascuchon
- style: Small visual adjustments for Text2Text record card (1632) by leiyre
- feat: Copy to clipboard the record text (1625) by leiyre
- docs: Add Slack support link in README's get started (1688) by dvsrepo
- chore: update version by frascuchon
- feat: Add new pagination size ranges (1667) by leiyre
- fix: handle stream api connection errors gracefully (1636) by Ankush-Chander
- feat: allow `rb.load` fetch records in batches passing the `from_id` argument by maxserras
- fix(Client): reusing the inner `httpx` client (1640) by frascuchon
- feat(Error Logging): send error detail in response for generic server errors (1648) by frascuchon
- docs: spacy `DocBin` cookbook (1642) by ignacioct
- feat: prepare_for_training supports spacy (1635) by frascuchon
- style: Improve card spacing (1638) by leiyre
- docs: Adding Elasticsearch persistence to docker compose section (1643) by maxserras
- chore: remove old rubrix client class (1639) by frascuchon
- feat(Listeners): allow using query params in the condition through search parameter (1627) by frascuchon
- doc: show metric graphs in documentation (1669) by leiyre
- fix(docker-compose.yaml): default volume and disable disk threshold (1656) by frascuchon
- fix: Encode rule name in Weak Labeling API requests (1649) by leiyre