Llm-foundry

What's Changed
- Torch has been bumped to `2.6.0` (in 1740)
- Sparse support has been disabled in the latest megablocks version (as part of the latest torch upgrade) and we cascaded those disables to llm-foundry as well (for more details, view the [megablocks release](https://github.com/databricks/megablocks/releases/tag/v0.8.0))
- `TransformerEngine` has been removed from the `all` dependency group due to version compatibility issues (in 1742). We expect to add this back in a future release.
- Transformers has been bumped to `v4.49.0` (in 1735) and this would result in the master weights being `torch.bfloat16` (view https://github.com/huggingface/transformers/issues/36567 for more context). `llm-foundry` doesn't support master weights in lower precision, so we manually hardcoded this to `torch.float32` when loading in #1734.

Detailed Changes
* remove deprecated param by bigning in https://github.com/mosaicml/llm-foundry/pull/1727
* Bump TE for FA 2.7.1.post1 bump by KuuCi in https://github.com/mosaicml/llm-foundry/pull/1730
* Fix dtype issue in transformers by dakinggg in https://github.com/mosaicml/llm-foundry/pull/1734
* Bump composer to 0.29.0 by rithwik-db in https://github.com/mosaicml/llm-foundry/pull/1733
* Bump Transformer v4.49.0 by KuuCi in https://github.com/mosaicml/llm-foundry/pull/1735
* Bump FA2 to 2.7.4.post1 by KuuCi in https://github.com/mosaicml/llm-foundry/pull/1728
* Comment GHCR Image Upload by KuuCi in https://github.com/mosaicml/llm-foundry/pull/1739
* Remove TE from all dependency group by dakinggg in https://github.com/mosaicml/llm-foundry/pull/1742
* Bump torch to 2.6 by rithwik-db in https://github.com/mosaicml/llm-foundry/pull/1740
* Update Makefile to use WORLD_SIZE by irenedea in https://github.com/mosaicml/llm-foundry/pull/1751

New Contributors
* rithwik-db made their first contribution in https://github.com/mosaicml/llm-foundry/pull/1733

**Full Changelog**: https://github.com/mosaicml/llm-foundry/compare/v0.17.1...v0.18.0

What's New
Datasets version upgrade (1724)
We've upgraded the version of Hugging Face datasets library to include a fix for a common issue of the multiprocessing pool hanging after tokenization or filtering.

What's Changed
* Update accelerate requirement from <1.2,>=0.25 to >=0.25,<1.4 by dependabot in https://github.com/mosaicml/llm-foundry/pull/1714
* Bump datasets version by dakinggg in https://github.com/mosaicml/llm-foundry/pull/1724


**Full Changelog**: https://github.com/mosaicml/llm-foundry/compare/v0.17.0...v0.17.1

What's Changed
* Update mcli examples to use 0.16.0 by irenedea in https://github.com/mosaicml/llm-foundry/pull/1713
* Refactor HF checkpointer by milocress in https://github.com/mosaicml/llm-foundry/pull/1690
Previously, MlFlow required PEFT models to be specified as a special "flavor" distinct from Transformers models. This workaround is no longer necessary, allowing us to simplify the codepath and cleanly abstract uploading the HuggingFace checkpoints from registering trained models.
* Bump version to 0.18.0.dev by milocress in https://github.com/mosaicml/llm-foundry/pull/1717
Removes the deprecated `sample_weighing_factor` argument from `mpt` loss calculations.

**Full Changelog**: https://github.com/mosaicml/llm-foundry/compare/v0.16.0...v0.17.0

What's New
Streaming 0.11.0 🚀 (1711)
We've upgraded [streaming](https://github.com/mosaicml/streaming) to 0.11.0. StreamingDataset can now be used with custom Stream implementations via a registry. See [the documentation page](https://docs.mosaicml.com/projects/streaming/en/stable/dataset_configuration/mixing_data_sources.html) for example usage.

What's Changed
* Fix llama3 example yamls by j316chuck in https://github.com/mosaicml/llm-foundry/pull/1688
* Update example yamls to use newest foundry version by snarayan21 in https://github.com/mosaicml/llm-foundry/pull/1689
* Update datasets requirement from <2.21,>=2.20.0 to >=2.20.0,<3.2 by dependabot in https://github.com/mosaicml/llm-foundry/pull/1670
* Catch multiple slashes in source dataset into one slash by KuuCi in https://github.com/mosaicml/llm-foundry/pull/1697
* Make loaded peft adapters optionally trainable by snarayan21 in https://github.com/mosaicml/llm-foundry/pull/1701
* Adding preprocessors for QA and messages datasets by ShashankMosaicML in https://github.com/mosaicml/llm-foundry/pull/1700
* Update pycln by b-chu in https://github.com/mosaicml/llm-foundry/pull/1704
* Add permission error by b-chu in https://github.com/mosaicml/llm-foundry/pull/1703
* Update datasets requirement from <3.2,>=2.20.0 to >=2.20.0,<3.3 by dependabot in https://github.com/mosaicml/llm-foundry/pull/1698
* Bump coverage[toml] from 7.6.4 to 7.6.10 by dependabot in https://github.com/mosaicml/llm-foundry/pull/1702
* Update mosaicml-streaming to 0.11.0 by es94129 in https://github.com/mosaicml/llm-foundry/pull/1711
* Bump version to 0.17.0.dev0 by irenedea in https://github.com/mosaicml/llm-foundry/pull/1712


**Full Changelog**: https://github.com/mosaicml/llm-foundry/compare/v0.15.1...v0.16.0

What's Changed
* Bump version 0.16.0.dev0 by j316chuck in https://github.com/mosaicml/llm-foundry/pull/1667
* Update mlflow requirement from <2.18,>=2.14.1 to >=2.14.1,<2.19 by dependabot in https://github.com/mosaicml/llm-foundry/pull/1673
* Speed up embedding tests by dakinggg in https://github.com/mosaicml/llm-foundry/pull/1668
* Add mcli yaml version bump by j316chuck in https://github.com/mosaicml/llm-foundry/pull/1674
* Bump Openai version by snarayan21 in https://github.com/mosaicml/llm-foundry/pull/1684
* Bump Streaming to v0.10.0 by snarayan21 in https://github.com/mosaicml/llm-foundry/pull/1685
* Bugfix auto packing with streams + no remote path by mattyding in https://github.com/mosaicml/llm-foundry/pull/1679
* Bump Composer to v0.28.0 by snarayan21 in https://github.com/mosaicml/llm-foundry/pull/1687
* Expose `DistributedSampler` RNG seed argument by janEbert in https://github.com/mosaicml/llm-foundry/pull/1677
* Add llama3 ft example yamls by j316chuck in https://github.com/mosaicml/llm-foundry/pull/1686

New Contributors
* janEbert made their first contribution in https://github.com/mosaicml/llm-foundry/pull/1677

**Full Changelog**: https://github.com/mosaicml/llm-foundry/compare/v0.15.0...v0.15.1

New Features

Open Source Embedding + Contrastive Code (https://github.com/mosaicml/llm-foundry/pull/1615)

LLM foundry now supports finetuning embedding models with contrastive loss. Foundry now supports various approaches to selecting negative passages for contrastive loss which can be either randomly selected or pre-defined. For more information, please view the the [readme](https://github.com/mosaicml/llm-foundry/blob/main/llmfoundry/models/llm_embed/README.md).

PyTorch 2.5.1 (https://github.com/mosaicml/llm-foundry/pull/1665)
This release updates LLM Foundry to the PyTorch 2.5.1 release, bringing with it support for the new features and optimizations in PyTorch 2.5.1.

Improved error messages (https://github.com/mosaicml/llm-foundry/pull/1657, https://github.com/mosaicml/llm-foundry/pull/1660, https://github.com/mosaicml/llm-foundry/pull/1623, https://github.com/mosaicml/llm-foundry/pull/1625)
Various improved error messages, making debugging user errors more clear.


What's Changed
* Update mcli examples to use 0.14.0 by irenedea in https://github.com/mosaicml/llm-foundry/pull/1624
* Open Source Embedding + Contrastive Code by KuuCi in https://github.com/mosaicml/llm-foundry/pull/1615
* Catch delta table not found error by milocress in https://github.com/mosaicml/llm-foundry/pull/1625
* Add Mlflow 403 PL UserError by mattyding in https://github.com/mosaicml/llm-foundry/pull/1623
* Catches when data prep cluster fails to start by milocress in https://github.com/mosaicml/llm-foundry/pull/1628
* Bump mlflow max version by dakinggg in https://github.com/mosaicml/llm-foundry/pull/1629
* add another cluster connection failure wrapper by milocress in https://github.com/mosaicml/llm-foundry/pull/1630
* Add MLflow `log_model` option by nancyhung in https://github.com/mosaicml/llm-foundry/pull/1544
* Move loss generating token counting to the dataloader by dakinggg in https://github.com/mosaicml/llm-foundry/pull/1632
* Bump databricks-connect from 14.1.0 to 15.4.3 by dependabot in https://github.com/mosaicml/llm-foundry/pull/1636
* Fix dataset download location by dakinggg in https://github.com/mosaicml/llm-foundry/pull/1639
* Revert "Bump databricks-connect from 14.1.0 to 15.4.3" by XiaohanZhangCMU in https://github.com/mosaicml/llm-foundry/pull/1640
* Bump transformers version by dakinggg in https://github.com/mosaicml/llm-foundry/pull/1631
* Fix gpu tests test_tp_train and test_huggingface_conversion_callback_interval by irenedea in https://github.com/mosaicml/llm-foundry/pull/1642
* Update datasets requirement from <2.20,>=2.19 to >=2.20.0,<2.21 by dependabot in https://github.com/mosaicml/llm-foundry/pull/1330
* Add max shard size to transformers save_pretrained by b-chu in https://github.com/mosaicml/llm-foundry/pull/1648
* Update huggingface-hub requirement from <0.25,>=0.19.0 to >=0.19.0,<0.27 by dependabot in https://github.com/mosaicml/llm-foundry/pull/1652
* Update accelerate requirement from <0.34,>=0.25 to >=0.25,<1.2 by dependabot in https://github.com/mosaicml/llm-foundry/pull/1633
* Catch Delta Table Not Found by KuuCi in https://github.com/mosaicml/llm-foundry/pull/1653
* Add Exception for missing UC column by milocress in https://github.com/mosaicml/llm-foundry/pull/1654
* Infer step size for Embeddings by KuuCi in https://github.com/mosaicml/llm-foundry/pull/1647
* Pin FAv2 by mvpatel2000 in https://github.com/mosaicml/llm-foundry/pull/1656
* Retry catching BlockingIOError by KuuCi in https://github.com/mosaicml/llm-foundry/pull/1657
* Catch bad data prep by milocress in https://github.com/mosaicml/llm-foundry/pull/1644
* Update pytest-cov requirement from <6,>=4 to >=4,<7 by dependabot in https://github.com/mosaicml/llm-foundry/pull/1663
* Bump coverage[toml] from 7.6.1 to 7.6.4 by dependabot in https://github.com/mosaicml/llm-foundry/pull/1650
* Move transform_model_pre_registration in hf_checkpointer by irenedea in https://github.com/mosaicml/llm-foundry/pull/1664
* Catch Cluster Permissions Error by KuuCi in https://github.com/mosaicml/llm-foundry/pull/1660
* Mosaicml version bump by j316chuck in https://github.com/mosaicml/llm-foundry/pull/1661
* Changes for removing unused terms in CE loss fn by gupta-abhay in https://github.com/mosaicml/llm-foundry/pull/1643
* Update setuptools requirement from <68.0.0 to <76.0.0 by dependabot in https://github.com/mosaicml/llm-foundry/pull/1662
* Bump docker version to torch 2.5.1 by j316chuck in https://github.com/mosaicml/llm-foundry/pull/1665
* Bump ubuntu 22.04 + torch 2.5.1 by KuuCi in https://github.com/mosaicml/llm-foundry/pull/1666

New Contributors
* mattyding made their first contribution in https://github.com/mosaicml/llm-foundry/pull/1623

**Full Changelog**: https://github.com/mosaicml/llm-foundry/compare/v0.14.5...v0.15.0