Instructlab-sdg

⚠️ Introducing v3 knowledge format - no backwards compat for v1/v2 ⚠️

The newly introduced **v3 knowledge format** is **incompatible** with the previous v1 and v2 formats. As a result, all existing knowledge contributions must be **re-formatted** to comply with the v3 specifications.

For detailed information and guidelines on how to re-format your contributions, please refer to the [issue discussion on GitHub](https://github.com/instructlab/sdg/issues/160).

---

What's Changed

* Add v3 knowledge schema support by abhi1092, shivchander, aakankshaduggal, russellb, markmc, derekhiggins in https://github.com/instructlab/sdg/pull/161

**Full Changelog**: https://github.com/instructlab/sdg/compare/v0.1.3...v0.2.0

What's Changed
* Add a YAML based file format for pipelines by markmc in https://github.com/instructlab/sdg/pull/86
* llmblock: Set a more reasonable default for num_tokens by russellb in https://github.com/instructlab/sdg/pull/125
* pipeline: Fail explicitly on an empty dataset by russellb in https://github.com/instructlab/sdg/pull/127
* Automate validation of pipeline configs by russellb in https://github.com/instructlab/sdg/pull/132
* Update grounded_skills.yaml to add seed value by aakankshaduggal in https://github.com/instructlab/sdg/pull/137
* ci: Run lint job if pipeline configs change by russellb in https://github.com/instructlab/sdg/pull/140
* Set gen_kwargs['n'] dynamically in the simple pipelines by russellb in https://github.com/instructlab/sdg/pull/144
* Add `model_prompt` config param for LLMBlock by russellb in https://github.com/instructlab/sdg/pull/141
* filterblock: add default_value for use with convert_dtype by markmc in https://github.com/instructlab/sdg/pull/143
* Export public APIs in top-level package by tiran in https://github.com/instructlab/sdg/pull/73
* Indent simple pipeline "principle" content by derekhiggins in https://github.com/instructlab/sdg/pull/150
* Move `gen_kwargs` down to `LLMBlock` by markmc in https://github.com/instructlab/sdg/pull/146
* ci: run e2e on pipeline config related changes by russellb in https://github.com/instructlab/sdg/pull/151
* importblock: resolve circular import issue by markmc in https://github.com/instructlab/sdg/pull/153
* Remove unused requirements by russellb in https://github.com/instructlab/sdg/pull/152
* Drop `__index_level_0__` columns by aakankshaduggal in https://github.com/instructlab/sdg/pull/142
* Fix SamplePopulatorBlock by markmc in https://github.com/instructlab/sdg/pull/156
* Block Name In Errors by gabe-l-hart in https://github.com/instructlab/sdg/pull/155
* Load custom pipelines from shared data dir by derekhiggins in https://github.com/instructlab/sdg/pull/166
* LLMBlock concurrency by gabe-l-hart in https://github.com/instructlab/sdg/pull/157

New Contributors
* tiran made their first contribution in https://github.com/instructlab/sdg/pull/73
* derekhiggins made their first contribution in https://github.com/instructlab/sdg/pull/150
* gabe-l-hart made their first contribution in https://github.com/instructlab/sdg/pull/155

**Full Changelog**: https://github.com/instructlab/sdg/compare/v0.1.2...v0.1.3

What's Changed
* Update messages datafile extension to be .jsonl by Maxusmusti in https://github.com/instructlab/sdg/pull/115

New Contributors
* Maxusmusti made their first contribution in https://github.com/instructlab/sdg/pull/115

**Full Changelog**: https://github.com/instructlab/sdg/compare/v0.1.1...v0.1.2

What's Changed
* Update generate_data.py to capture context key by aakankshaduggal in https://github.com/instructlab/sdg/pull/98
* Add CI workflow that runs the full SDG pipeline by russellb in https://github.com/instructlab/sdg/pull/93
* Remove two files that are now unused by russellb in https://github.com/instructlab/sdg/pull/104
* Batch support with vllm by aakankshaduggal in https://github.com/instructlab/sdg/pull/105
* converts dataset format messages required for training by oindrillac in https://github.com/instructlab/sdg/pull/94


**Full Changelog**: https://github.com/instructlab/sdg/compare/v0.1.0...v0.1.1

This version introduces an effective rewrite of the library. There is a `simple` pipeline aimed at maintaining compatibility with small environments supported by the `ilab` CLI. There is also a new `full` pipeline that is much more extensive and can produce higher quality results for environments capable of running it, along with the required teacher model, Mixtral-8x7b-instruct.

What's Changed
* Update e2e config to optimize pip caching by nathan-weinberg in https://github.com/instructlab/sdg/pull/44
* github: Automate some labels with mergify by russellb in https://github.com/instructlab/sdg/pull/40
* Add SDG library code by shivchander, aakankshaduggal, oindrillac, et. al. in https://github.com/instructlab/sdg/pull/42
* 📚 Adding Knowledge llm blocks by abhi1092 in https://github.com/instructlab/sdg/pull/50
* e2e: Fix permissions error by russellb in https://github.com/instructlab/sdg/pull/51
* Initial CLI integration with new SDG interfaces by russellb in https://github.com/instructlab/sdg/pull/46
* Fix dataset formatting for pipeline differences by russellb in https://github.com/instructlab/sdg/pull/57
* updates to grounded flow by oindrillac, shivchanderm, oindrillac in https://github.com/instructlab/sdg/pull/53
* e2e: Only run one job at a time for a given PR by russellb in https://github.com/instructlab/sdg/pull/68
* Fix prompt file paths for an installed library by russellb in https://github.com/instructlab/sdg/pull/67
* Resolve some trivial TODOs in generate_data() by markmc in https://github.com/instructlab/sdg/pull/74
* Fix mismatch in full pipeline outputs by russellb in https://github.com/instructlab/sdg/pull/75
* Updated chunking_document. by PalmPalm7 in https://github.com/instructlab/sdg/pull/65
* Handle type conversion errors in FilterByValueBlock by russellb in https://github.com/instructlab/sdg/pull/78
* Make SynthSkillsFlow honor the num_iters parameter by russellb in https://github.com/instructlab/sdg/pull/82
* Bump actions/download-artifact from 4.1.7 to 4.1.8 by dependabot in https://github.com/instructlab/sdg/pull/91
* Drop remaining import from main instructlab package by russellb in https://github.com/instructlab/sdg/pull/89
* generate_data: Fix check for `output` in results by russellb in https://github.com/instructlab/sdg/pull/71
* generate_data: fix support for multiple leaf nodes by russellb in https://github.com/instructlab/sdg/pull/85
* Allow FilterByValueBlock to handle one or many values by russellb in https://github.com/instructlab/sdg/pull/81
* Bump pypa/gh-action-pypi-publish from 1.8.14 to 1.9.0 by dependabot in https://github.com/instructlab/sdg/pull/24
* iterblock: remove duplicate line of code by russellb in https://github.com/instructlab/sdg/pull/83

New Contributors
* shivchander made their first contribution in https://github.com/instructlab/sdg/pull/42
* aakankshaduggal made their first contribution in https://github.com/instructlab/sdg/pull/42
* abhi1092 made their first contribution in https://github.com/instructlab/sdg/pull/50
* oindrillac made their first contribution in https://github.com/instructlab/sdg/pull/53
* markmc made their first contribution in https://github.com/instructlab/sdg/pull/74
* PalmPalm7 made their first contribution in https://github.com/instructlab/sdg/pull/65

**Full Changelog**: https://github.com/instructlab/sdg/compare/v0.0.4...v0.1.0

**Full Changelog**: https://github.com/instructlab/sdg/compare/v0.0.4...v0.0.4.1