Lilac

<!-- Release notes generated using configuration in .github/release.yml at v0.1.24 -->

This release changes the text media visualizer to Monaco (the engine that powers VSCode).

Monaco allows us to:
- Deep-link to any line within a document.
- Add right click menus to text.
- Add "thumbs up" and "thumbs down" to concepts from the menu, for any text.
- Search any text from the right click-menu, with semantic similarity or keyword search.

Here is a video explaining the changes: https://www.youtube.com/watch?v=83Rj006tVIk

This release also has custom support for the ShareGPT format in the UI:
<img width="1478" alt="image" src="https://github.com/lilacai/lilac/assets/1100749/181d2ca7-94d5-490a-9641-3d4e8727edbd">

Features
* Add special support for a DELETED label by brilee in https://github.com/lilacai/lilac/pull/951
* Switch to monaco for the main viewer. by nsthorat in https://github.com/lilacai/lilac/pull/952
* Simplify monaco viewer. Add support for deep linking. by nsthorat in https://github.com/lilacai/lilac/pull/956
* Infer dataset formats. Start with just ShareGPT. by nsthorat in https://github.com/lilacai/lilac/pull/948
* Add UI for title slots for ShareGPT. by nsthorat in https://github.com/lilacai/lilac/pull/950

Bug fixes
* Make the signal "try it" page work for signals w/o schema by dsmilkov in https://github.com/lilacai/lilac/pull/944
* Fix UI bugs: monaco scroll, hash state forgotten, compare non-media fields by nsthorat in https://github.com/lilacai/lilac/pull/946
* Eliminate setup count call from parquet_source. by brilee in https://github.com/lilacai/lilac/pull/959
* Fix a bug where we highlighted all concept spans regardless of their score. by nsthorat in https://github.com/lilacai/lilac/pull/958
* Fix bug with loading dataset and settings. by nsthorat in https://github.com/lilacai/lilac/pull/957

Docs
* Add limit/filter docs by brilee in https://github.com/lilacai/lilac/pull/943

Other Changes
* Add youtube video for the blog post by dsmilkov in https://github.com/lilacai/lilac/pull/942
* Drive-by cleanup of schema.py code by brilee in https://github.com/lilacai/lilac/pull/955

**Full Changelog**: https://github.com/lilacai/lilac/compare/v0.1.23...v0.1.24

<!-- Release notes generated using configuration in .github/release.yml at v0.1.23 -->

High-level
Lilac is now moving towards editing data directly in the tool. The first vehicle for this is `Dataset.map`.

New blog post on curating data with the new `Dataset.map` feature:
https://docs.lilacml.com/blog/curate-coding-dataset.html

Documentation on Dataset.map:
https://docs.lilacml.com/datasets/dataset_edit.html

Features
* Add dataset.map support for limit/filter by brilee in https://github.com/lilacai/lilac/pull/933
* Add support for arbitrary value type `v` in `map<k, v>` in parquet by dsmilkov in https://github.com/lilacai/lilac/pull/935
* Add batch size support and collapse transform impl by brilee in https://github.com/lilacai/lilac/pull/934

Improvements
* Improve the UI for repeated values. by nsthorat in https://github.com/lilacai/lilac/pull/904
* Small ergonomic fixes while writing the "code formatting" blog post by dsmilkov in https://github.com/lilacai/lilac/pull/909
* Merge multiple shards of the same task into the same progress bar. by nsthorat in https://github.com/lilacai/lilac/pull/910
* Add threaded task execution. by nsthorat in https://github.com/lilacai/lilac/pull/920
* Fix css style for markdown tables by dsmilkov in https://github.com/lilacai/lilac/pull/931
* Fix tqdm progress bars by separating report_progress from show_progress. by nsthorat in https://github.com/lilacai/lilac/pull/929
* Make parquet the default source by dsmilkov in https://github.com/lilacai/lilac/pull/941

Bug fixes
* Fix keyword search to work with apostrophe `'` by dsmilkov in https://github.com/lilacai/lilac/pull/907
* Make sure the results of dataset.map() always returns an iterable. by nsthorat in https://github.com/lilacai/lilac/pull/925
* Remove position= in tqdm. by nsthorat in https://github.com/lilacai/lilac/pull/913

Docs
* Add a guide for iterating on dataset by dsmilkov in https://github.com/lilacai/lilac/pull/923
* Add blog post for diffing and `dataset.map` by dsmilkov in https://github.com/lilacai/lilac/pull/912
* Redo the docs.lilacml.com landing page by dsmilkov in https://github.com/lilacai/lilac/pull/932
* Small tweaks to improve the glaive dataset blog post. by nsthorat in https://github.com/lilacai/lilac/pull/938
* Rename the guide to edit a dataset by dsmilkov in https://github.com/lilacai/lilac/pull/930
* Revamp welcome/intro pages by brilee in https://github.com/lilacai/lilac/pull/908

Other
* Refactor dataset/signal endpoints into separate module by brilee in https://github.com/lilacai/lilac/pull/900
* Add memray dep and instructions by brilee in https://github.com/lilacai/lilac/pull/917
* Add spec for select options by brilee in https://github.com/lilacai/lilac/pull/918
* Simplify helper methods to closer align to API for select options by brilee in https://github.com/lilacai/lilac/pull/919
* Start writing the query options compiler by brilee in https://github.com/lilacai/lilac/pull/924

Coming soon
* Add server-side RAG python code. by nsthorat in https://github.com/lilacai/lilac/pull/911
* Migrate the UI to the server-side python RAG. by nsthorat in https://github.com/lilacai/lilac/pull/914
* Improve the RAG UI by nsthorat in https://github.com/lilacai/lilac/pull/916

**Full Changelog**: https://github.com/lilacai/lilac/compare/v0.1.22...v0.1.23

<!-- Release notes generated using configuration in .github/release.yml at v0.1.22 -->
High-level
- Excluding a tag from the UI is now an option from the searchbox, enabling the workflow of keeping that filter on, and progressively tagging new data to be removed.
<img width="705" alt="image" src="https://github.com/lilacai/lilac/assets/1100749/44b4aa0d-b127-4ff6-b414-cef849ac1cc8">
- Signals can now be written without defining a schema.

Features
* Add `dataset.transform()` where we pass the entire input as iterable by dsmilkov in https://github.com/lilacai/lilac/pull/897
* Add support for input paths to dataset.map. by nsthorat in https://github.com/lilacai/lilac/pull/882
* Improve ergonomics of map, relaxing the exact requirement of kwargs={row, job_id} by nsthorat in https://github.com/lilacai/lilac/pull/883
* Add a second option in searchbox dropdown to exclude a tag by brilee in https://github.com/lilacai/lilac/pull/889
* Add rendering of string spans that were derived from a map with input path by dsmilkov in https://github.com/lilacai/lilac/pull/888
* Make schema in signals optional by dsmilkov in https://github.com/lilacai/lilac/pull/895
* Add string filters by brilee in https://github.com/lilacai/lilac/pull/892

Bug fixes
* Fix a few issues with batching, prefetching, and searches. by nsthorat in https://github.com/lilacai/lilac/pull/881
* Upgrade duckdb to 0.9.2, fixing a crash in a dask process with fetch_df_chunk. by nsthorat in https://github.com/lilacai/lilac/pull/884
* Fix UI bugs with span rendering of maps. by nsthorat in https://github.com/lilacai/lilac/pull/894
* Fix span resolving for map outputs by dsmilkov in https://github.com/lilacai/lilac/pull/886
* Prefer existing embedding in embedding retrieval function by brilee in https://github.com/lilacai/lilac/pull/890
* Allow lilac to run tasks outside a running event loop. by nsthorat in https://github.com/lilacai/lilac/pull/899

Other Changes
* Pass explicit schema during jsonl -> parquet conversion by dsmilkov in https://github.com/lilacai/lilac/pull/885
* Rename `lilac.lilac_span` to `lilac.span` by dsmilkov in https://github.com/lilacai/lilac/pull/887
* Make the tags & namespaces in the dataset panel expandable. by nsthorat in https://github.com/lilacai/lilac/pull/893
* Fix trailing error with tests. by nsthorat in https://github.com/lilacai/lilac/pull/901
* Make the tag expandables serializable in the URL for sharing. by nsthorat in https://github.com/lilacai/lilac/pull/898
* Add the navigation store to the URL hash. by nsthorat in https://github.com/lilacai/lilac/pull/896


**Full Changelog**: https://github.com/lilacai/lilac/compare/v0.1.21...v0.1.22

<!-- Release notes generated using configuration in .github/release.yml at v0.1.21 -->

Features
* Signal computations are now cached. If a signal fails half-way through, it will be resumed.
* Source loading is much faster, up to 40x faster for some sources (e.g. HuggingFace)
* Map dtype is now supported for parquet sources.

Details
* Add jsonl intermediate caching to signals. Introduce a central spot for this cache abstraction. by nsthorat in https://github.com/lilacai/lilac/pull/858
* Rename fast_process to load_to_parquet by brilee in https://github.com/lilacai/lilac/pull/862
* Implement fast_process for parquet sources by brilee in https://github.com/lilacai/lilac/pull/860
* Implement CSV direct to parquet by brilee in https://github.com/lilacai/lilac/pull/863
* Implement fast json source by brilee in https://github.com/lilacai/lilac/pull/865
* Add `map<key, value>` dtype. No support in the UI yet. by dsmilkov in https://github.com/lilacai/lilac/pull/870
* Implement fast processing for huggingface datasets by brilee in https://github.com/lilacai/lilac/pull/869

Bug Fixes & Other Changes
* add development docs on profiling by brilee in https://github.com/lilacai/lilac/pull/861
* Add docs for settings and compare mode by dsmilkov in https://github.com/lilacai/lilac/pull/859
* Add a nest_under field to dataset.map(). by nsthorat in https://github.com/lilacai/lilac/pull/866
* Avoid computing stats for every single field on page load by dsmilkov in https://github.com/lilacai/lilac/pull/873
* Fix a sample_size yaml bug by dsmilkov in https://github.com/lilacai/lilac/pull/874
* UI fixes for expanding long rows. by nsthorat in https://github.com/lilacai/lilac/pull/875
* Fix small bug with compute signal / concepts and filtering by valid dtypes. by nsthorat in https://github.com/lilacai/lilac/pull/877
* Add support for `map` field in the schema and UI by dsmilkov in https://github.com/lilacai/lilac/pull/878
* Fix a bug with previewing and comparing on repeated values. by nsthorat in https://github.com/lilacai/lilac/pull/879
* Allow custom signals to work with dask processes. by nsthorat in https://github.com/lilacai/lilac/pull/880


**Full Changelog**: https://github.com/lilacai/lilac/compare/v0.1.20...v0.1.21

<!-- Release notes generated using configuration in .github/release.yml at v0.1.20 -->

Bug fixes
* Small fix with rendering `MetadataSearch` in the schema view by dsmilkov in https://github.com/lilacai/lilac/pull/855
* Fast dataset load by brilee in https://github.com/lilacai/lilac/pull/854
* Fix a bug with single item mode and monaco diff not updating by dsmilkov in https://github.com/lilacai/lilac/pull/856


**Full Changelog**: https://github.com/lilacai/lilac/compare/v0.1.19...v0.1.20

<!-- Release notes generated using configuration in .github/release.yml at v0.1.19 -->

Bug fixes
* Fix thread bug in hnswlib, which should fix CI python tests by dsmilkov in https://github.com/lilacai/lilac/pull/852
* Fix bugs with the media fields selector where no fields showed up. by nsthorat in https://github.com/lilacai/lilac/pull/853


**Full Changelog**: https://github.com/lilacai/lilac/compare/v0.1.18...v0.1.19