Lilac

<!-- Release notes generated using configuration in .github/release.yml at v0.2.3 -->

What's Changed

We now have 2 CLI scripts for sharing Lilac datasets (via huggingface):

bash
lilac upload local/Capybara --url_or_repo=lilacai/Capybara


To download the dataset to a local project directory:
bash
lilac download lilacai/Capybara


For more details on sharing datasets, see the [Sharing Guide](https://docs.lilacml.com//datasets/dataset_share.html)

With this change, we added a new environment variable `USE_TABLE_INDEX`, useful for frozen demos. This will dramatically improve the performance of queries as we use a cached DuckDB table. This will slow down labeling, or any edits, as the table will get re-computed upon each change.

Upload / Download
* Add an upload dataset script. Some other cleanups. by nsthorat in https://github.com/lilacai/lilac/pull/1059

Bug fixes
* Fix a bug with CSV source reader for TSV files, and named columns. by nsthorat in https://github.com/lilacai/lilac/pull/1040
* Progress bar by brilee in https://github.com/lilacai/lilac/pull/1043
* Fix bug with ItemMedia not rendering media fields that are deeply nested siblings. by nsthorat in https://github.com/lilacai/lilac/pull/1044
* Fix clustering an enriched field by dsmilkov in https://github.com/lilacai/lilac/pull/1048
* Propagate filters in the group by panel by dsmilkov in https://github.com/lilacai/lilac/pull/1041

Performance
* Add indexing on database startup, flag-guarded by brilee in https://github.com/lilacai/lilac/pull/1052

UI
* Add clustering in the UI by dsmilkov in https://github.com/lilacai/lilac/pull/1045
* Add search to the cluster UI. Add some polish. by nsthorat in https://github.com/lilacai/lilac/pull/1054
* Add clusters to the schema menu. Migrate to a custom carousel component so the page doesn't freeze. by nsthorat in https://github.com/lilacai/lilac/pull/1050
Clustering
* Add `dataset.cluster(input)` where `input` can be any lambda func by dsmilkov in https://github.com/lilacai/lilac/pull/1042
* `dataset.cluster()` flattens any repeated before clustering by dsmilkov in https://github.com/lilacai/lilac/pull/1051

Lilac Garden
* Add remote computation for the jina embedding by dsmilkov in https://github.com/lilacai/lilac/pull/1049

Other Changes
* Move the import of .env.local in publish_pip to the top of the file. by nsthorat in https://github.com/lilacai/lilac/pull/1039
* fix: migrate embeddings by azure openai to openai > 1.0.0 by dechantoine in https://github.com/lilacai/lilac/pull/1053
* Streamline lilac deployment by brilee in https://github.com/lilacai/lilac/pull/1057
* Add a notebook for working with concepts from python. by nsthorat in https://github.com/lilacai/lilac/pull/1055

**Full Changelog**: https://github.com/lilacai/lilac/compare/v0.2.2...v0.2.3

<!-- Release notes generated using configuration in .github/release.yml at v0.2.2 -->

Bug fixes
* Fix a bug with OpenAI embeddings after upgrading. by nsthorat in https://github.com/lilacai/lilac/pull/1038
* Remove an extra temporary column at the end of clustering by dsmilkov in https://github.com/lilacai/lilac/pull/1035

Other Changes
* Convert the pivot viewer to a bunch of carousels. by nsthorat in https://github.com/lilacai/lilac/pull/1034


**Full Changelog**: https://github.com/lilacai/lilac/compare/v0.2.1...v0.2.2

<!-- Release notes generated using configuration in .github/release.yml at v0.2.1 -->

Keyboard shortcuts are now available for deleting, and labeling!

To delete a row: use backspace or delete.
To label, go to dataset settings, and configure key-bindings for each label.

https://github.com/lilacai/lilac/assets/1100749/4e9e014c-c8f3-4433-816b-0754ad0ce0ec

What's Changed

Features
* Add keyboard shortcuts for fast labeling. by nsthorat in https://github.com/lilacai/lilac/pull/1028

Bug fixes
* Allows non folder exports by hynky1999 in https://github.com/lilacai/lilac/pull/1026
* Fixes incorrect destructuring by hynky1999 in https://github.com/lilacai/lilac/pull/1025
* Improve auto-binning, and sorting of histograms. by nsthorat in https://github.com/lilacai/lilac/pull/1033
* Fix lilac deployer for slashed datasets. by nsthorat in https://github.com/lilacai/lilac/pull/1021

Docs
* Update documentation for labels, keyboard shortcuts, deleting rows. by nsthorat in https://github.com/lilacai/lilac/pull/1030
* Add documentation that points to the lilac deployer UI. by nsthorat in https://github.com/lilacai/lilac/pull/1020

UI
* Improve the UI around deleting. by nsthorat in https://github.com/lilacai/lilac/pull/1024
* Add a 2-feature pivot view, allowing you to view a hierarchy of 2 features by nsthorat in https://github.com/lilacai/lilac/pull/1023

Other Changes
* Improve the title generation in clustering by dsmilkov in https://github.com/lilacai/lilac/pull/1022
* Fix some `map(overwrite=True)` bugs by dsmilkov in https://github.com/lilacai/lilac/pull/1031
* Add superclusters (categories) by dsmilkov in https://github.com/lilacai/lilac/pull/1032

New Contributors
* hynky1999 made their first contribution in https://github.com/lilacai/lilac/pull/1026

**Full Changelog**: https://github.com/lilacai/lilac/compare/v0.2.0...v0.2.1

<!-- Release notes generated using configuration in .github/release.yml at v0.2.0 -->

What's Changed

The UI now supports deleting row(s), viewing the trash & undeleting. Exporting will now automatically drop deleted rows.

Breaking changes
* Merge `output_column` and `nest_under` --> `dataset.map(output_path=...)` by dsmilkov in https://github.com/lilacai/lilac/pull/1001

UI
* Add the ability to delete and restore rows from the UI. by nsthorat in https://github.com/lilacai/lilac/pull/1011
* Fix signal configs to use ClassVar by dsmilkov in https://github.com/lilacai/lilac/pull/1016

Performance
* Fix jina to also run on CUDA if available by dsmilkov in https://github.com/lilacai/lilac/pull/996
* Use CUDA when available for sentence transformers. by nsthorat in https://github.com/lilacai/lilac/pull/991
* Use the yaml CLoader loader if it's available. by nsthorat in https://github.com/lilacai/lilac/pull/995
* Use cuml for clustering when possible by dsmilkov in https://github.com/lilacai/lilac/pull/997
* Fix map by brilee in https://github.com/lilacai/lilac/pull/994
* Add Jina (Small) on Garden signal by dsmilkov in https://github.com/lilacai/lilac/pull/1009

Bug fixes
* Fix some small UI bugs. by nsthorat in https://github.com/lilacai/lilac/pull/987
* Fix issue with repeated of string rendering. by nsthorat in https://github.com/lilacai/lilac/pull/1015
* Load datasets in a separate thread from the UI. by nsthorat in https://github.com/lilacai/lilac/pull/1014
* Fix issue where we don't block on the server thread from the CLI. by nsthorat in https://github.com/lilacai/lilac/pull/1013

Clustering (coming soon)
* Make `ds.cluster()` have resumable title generation by dsmilkov in https://github.com/lilacai/lilac/pull/1000
* `dataset.cluster()` now uses `transform()` which uses `map()` by dsmilkov in https://github.com/lilacai/lilac/pull/1002
* Add topic clustering in `dataset.cluster()` by dsmilkov in https://github.com/lilacai/lilac/pull/993
* Allow clustering of a nested path by dsmilkov in https://github.com/lilacai/lilac/pull/1007
* Add `dataset.cluster(remote=True)` bit by dsmilkov in https://github.com/lilacai/lilac/pull/1010

Map & signal changes
* Add signal.map customization by brilee in https://github.com/lilacai/lilac/pull/1004
* Allow map to be called for arbitrary depth by dsmilkov in https://github.com/lilacai/lilac/pull/998
* remove VectorCompute path in dispatch_workers by brilee in https://github.com/lilacai/lilac/pull/1008
* Implement signals on top of the map infrastructure by brilee in https://github.com/lilacai/lilac/pull/1006
* `dataset.map` can now nest_under any repeated by dsmilkov in https://github.com/lilacai/lilac/pull/999
* Remove TaskShardId by brilee in https://github.com/lilacai/lilac/pull/1003

Other Changes
* Update the Dockerfile to use port 80 so we can use it on GCE. by nsthorat in https://github.com/lilacai/lilac/pull/992
* Make OpenAI calls threaded with exponential backoff by dsmilkov in https://github.com/lilacai/lilac/pull/1005


**Full Changelog**: https://github.com/lilacai/lilac/compare/v0.1.26...v0.2.0

<!-- Release notes generated using configuration in .github/release.yml at v0.1.26 -->

This release adds a markdown code block extractor signal, highlighting markdown code blocks and their languages.

<img width="390" alt="image" src="https://github.com/lilacai/lilac/assets/1100749/49afd2b8-2bb4-4085-b385-67fe17ecf571">

What's Changed
* Add markdown extractor signal. by nsthorat in https://github.com/lilacai/lilac/pull/983
* Add `dataset.cluster()` by dsmilkov in https://github.com/lilacai/lilac/pull/981

Bug fixes
* Emit membership prob in HDBScan, and fix "group by" UI bugs by dsmilkov in https://github.com/lilacai/lilac/pull/976
* Fix `ll.start_server()` and add a test for full end-to-end server startup by dsmilkov in https://github.com/lilacai/lilac/pull/984
* Add CLI integration tests. by nsthorat in https://github.com/lilacai/lilac/pull/985
* Make `ll.start_server()` blocking outside an event loop by dsmilkov in https://github.com/lilacai/lilac/pull/986

Other Changes



**Full Changelog**: https://github.com/lilacai/lilac/compare/v0.1.25...v0.1.26

<!-- Release notes generated using configuration in .github/release.yml at v0.1.25 -->

This release drops dask for a thin multi-processing client, and comes with lots of performance improvements, namely the slow import time of lilac.

We have also added a simple API for loading from HuggingFace
py
import lilac as ll
from datasets import load_dataset
hf_ds = load_dataset('Open-Orca/SlimOrca-Dedup')
ds = ll.from_huggingface(hf_ds)


And a simple API for getting embeddings:
py
answer_emb = ds.get_embeddings('jina-v2-small', rowid, 'answer')[0]['vector']


We've also added some color to the UI, and organized components a little better
<img width="1264" alt="image" src="https://github.com/lilacai/lilac/assets/1100749/f1e835f3-e77b-49f0-a1fb-3d555c224520">

Features
* Add Jina V2 embeddings by dsmilkov in https://github.com/lilacai/lilac/pull/966
* Add sugar for `ll.from_huggingface()` by dsmilkov in https://github.com/lilacai/lilac/pull/962
* Improve the row header to give us space for deleting. by nsthorat in https://github.com/lilacai/lilac/pull/965

Performance
* Reduce import times by brilee in https://github.com/lilacai/lilac/pull/961
* Using `loky` (thin wrapper around `multiprocessing`) instead of dask by dsmilkov in https://github.com/lilacai/lilac/pull/947
* fix iterable robustness by brilee in https://github.com/lilacai/lilac/pull/977

Bug fixes
* Fix memory leak caused by Iterable/Iterator mixups by brilee in https://github.com/lilacai/lilac/pull/974
* Fix broken doc links. by nsthorat in https://github.com/lilacai/lilac/pull/964
* Add color scales for semantic / concept search. Add openchat format. by nsthorat in https://github.com/lilacai/lilac/pull/975

Other Changes
* Remove legal-termination concept. by nsthorat in https://github.com/lilacai/lilac/pull/980

**Full Changelog**: https://github.com/lilacai/lilac/compare/v0.1.24...v0.1.25