Mosaicml

New Features

1. **𐄷 Weight Standardization**

Weight Standardization reparametrizes convolutional weights such that the fan-in dimensions have zero mean and unit standard deviation. This could slightly improve performance at the expensive of 5% lower throughput. This has been used in several papers to train with smaller batch sizes, with normalization layers besides batch norm, and for transfer learning.


Using Weight Standardization with the Composer Trainer:

python
import composer

Apply Weight Standardization (when training is initialized)
weight_std = composer.algorithms.WeightStandardization()

Train with Weight Standardization
trainer = composer.trainer.Trainer(
...
algorithms=[weight_std]
)
trainer.fit()


Using Weight Standardization with the Composer functional interface:

python
import composer
from torchvision.models import resnet50

my_model = resnet50()

Apply weight standardization to model
my_model = composer.functional.weight_standardization(my_model)


Please see the [Weight Standardization Method Card](https://docs.mosaicml.com/en/stable/method_cards/weight_standardization.html) for more details.

Bug Fixes

* Fix for checkpoints not being saved automatically at the end of a run (1552)
* Fix Onnx export for Composer HuggingFaceModels (1557)
* Fix for MIoU metric producing NaN's (1558)
* CometML logger documentation updates and fixes (1567, 1570, 1571)
* WandB image visualizer fix (1591)

What's Changed
* Update evaluate_periodically() when eval interval is of type Duration by karan6181 in https://github.com/mosaicml/composer/pull/1523
* Quality of life updates to EMA by coryMosaicML in https://github.com/mosaicml/composer/pull/1524
* Add ADE20K and COCO v2 dataset behind a version flag by karan6181 in https://github.com/mosaicml/composer/pull/1528
* Pinned setuptools version to fix distutils version error by karan6181 in https://github.com/mosaicml/composer/pull/1536
* Less strict name formatting by hanlint in https://github.com/mosaicml/composer/pull/1535
* Defaulting streaming dataset version to 1 and add a deprecation warning by karan6181 in https://github.com/mosaicml/composer/pull/1532
* Changing 'stable' to 'latest' in notebooks in examples by bcui19 in https://github.com/mosaicml/composer/pull/1534
* Bump furo from 2022.6.21 to 2022.9.15 by dependabot in https://github.com/mosaicml/composer/pull/1540
* Bump fasteners from 0.17.3 to 0.18 by dependabot in https://github.com/mosaicml/composer/pull/1538
* Add Pandoc to Docker images, bump version to 2.19.2 by bandish-shah in https://github.com/mosaicml/composer/pull/1550
* Removed streaming version 2 from yaml since version 1 is default by karan6181 in https://github.com/mosaicml/composer/pull/1551
* Bump ipykernel from 6.15.2 to 6.15.3 by dependabot in https://github.com/mosaicml/composer/pull/1548
* Bump yamllint from 1.27.1 to 1.28.0 by dependabot in https://github.com/mosaicml/composer/pull/1546
* Bump traitlets from 5.3.0 to 5.4.0 by dependabot in https://github.com/mosaicml/composer/pull/1539
* Object Store Logger Race Condition + EMA Fix by mvpatel2000 in https://github.com/mosaicml/composer/pull/1552
* Adding in erroring for when using GradMonitor and DeepSpeed by bcui19 in https://github.com/mosaicml/composer/pull/1555
* Bump pypandoc from 1.8.1 to 1.9 by dependabot in https://github.com/mosaicml/composer/pull/1559
* Update context to raise errror by mvpatel2000 in https://github.com/mosaicml/composer/pull/1561
* Fix MIoU metric when `self.total_union==0` by abhi-mosaic in https://github.com/mosaicml/composer/pull/1558
* Move dataloader `initialize_object` to factory methods by hanlint in https://github.com/mosaicml/composer/pull/1510
* Weight Standardization method by Landanjs in https://github.com/mosaicml/composer/pull/1562
* Update comet links to include query params and point to main site by dakinggg in https://github.com/mosaicml/composer/pull/1567
* remove dead line in alibi by mvpatel2000 in https://github.com/mosaicml/composer/pull/1568
* GLU Fixes by mvpatel2000 in https://github.com/mosaicml/composer/pull/1564
* Add FSDP strategy by abhi-mosaic in https://github.com/mosaicml/composer/pull/1553
* Comet example by dakinggg in https://github.com/mosaicml/composer/pull/1570
* Add missing _enabled flag, post_close, and clean up comet ml tests by dakinggg in https://github.com/mosaicml/composer/pull/1571
* Consistent Method Card Style by growlix in https://github.com/mosaicml/composer/pull/1407
* add missing return in context by mvpatel2000 in https://github.com/mosaicml/composer/pull/1574
* Remove eval batch split by mvpatel2000 in https://github.com/mosaicml/composer/pull/1576
* Fix Onnx Export for Composer HuggingFaceModels by nik-mosaic in https://github.com/mosaicml/composer/pull/1557
* Revert checkpoint rename by hanlint in https://github.com/mosaicml/composer/pull/1579

New Contributors
* bcui19 made their first contribution in https://github.com/mosaicml/composer/pull/1534

**Full Changelog**: https://github.com/mosaicml/composer/compare/v0.10.0...v0.10.1

Streaming `v0.10.0` is released! Install via `pip`:


pip install --upgrade mosaicml-streaming==0.10.0


Improvements

1. Reusable cloud download clients (https://github.com/mosaicml/streaming/pull/817)
* Streaming now reuses cloud download clients when downloading shard files instead of creating a new client for each download.
* This avoids run failures that sometimes occur with too many open sockets or excessive cloud authentication requests.

2: `py1b` shuffle algorithm deprecation (https://github.com/mosaicml/streaming/pull/837)
* The `py1b` shuffle algorithm has now been deprecated. Please use the improved `py1e` (default) or the `py1br` shuffle algorithms instead.

What's Changed
* Update FAQs to indicate wrapping not supported by milocress in https://github.com/mosaicml/streaming/pull/822
* refactored the download module to have reusable clients by ethantang-db in https://github.com/mosaicml/streaming/pull/817
* Update pytest-cov requirement from <6,>=4 to >=4,<7 by dependabot in https://github.com/mosaicml/streaming/pull/821
* Consistent errors for unused streams in batching methods by snarayan21 in https://github.com/mosaicml/streaming/pull/826
* Update setuptools requirement from <68.0.0 to <76.0.0 by dependabot in https://github.com/mosaicml/streaming/pull/825
* fix f string by XiaohanZhangCMU in https://github.com/mosaicml/streaming/pull/829
* Bump fastapi from 0.115.4 to 0.115.5 by dependabot in https://github.com/mosaicml/streaming/pull/830
* Bump uvicorn from 0.32.0 to 0.32.1 by dependabot in https://github.com/mosaicml/streaming/pull/834
* Bump pydantic from 2.9.2 to 2.10.1 by dependabot in https://github.com/mosaicml/streaming/pull/833
* Bump pytest from 8.3.3 to 8.3.4 by dependabot in https://github.com/mosaicml/streaming/pull/836
* Bump pydantic from 2.10.1 to 2.10.2 by dependabot in https://github.com/mosaicml/streaming/pull/835
* Version bump to 0.11.0.dev0, including deprecations by snarayan21 in https://github.com/mosaicml/streaming/pull/837

New Contributors
* ethantang-db made their first contribution in https://github.com/mosaicml/streaming/pull/817

**Full Changelog**: https://github.com/mosaicml/streaming/compare/v0.9.1...v0.10.0

Streaming `v0.9.1` is released! Install via `pip`:


pip install --upgrade mosaicml-streaming==0.9.1


What's New
1. Streaming is added to Gurubase (https://github.com/mosaicml/streaming/pull/805)
* Streaming now has an AI assistant available to help users with their questions! Try out Streaming Guru which uses the data from this repo and data from the [docs](https://docs.mosaicml.com/projects/streaming/en/stable/) to answer questions by leveraging the LLM.

Improvements
1. Permission Issue Resolution (https://github.com/mosaicml/streaming/pull/813)
* Resolved read permission issues occurring when shared memory files are created in shared computing environments. We added retry conditions to allow the creation of new shared memory files upon encountering permission errors.
* Prefix Integrity for Shared Memory Files: When creating shared memory files, both LOCALS and FILELOCKS are now validated to ensure no overlap with existing files, and they are matched with consistent prefix identifiers.
* Handling Non-Normal Program Exits: Enhanced cleanup procedures to address cases where non-normal program exits left some shared memory files uncleared. All files in SHM_TO_CLEAN are now checked to prevent duplicates.
These changes improve shared memory management and reliability in shared environments.

2. Fix Shard Eviction Hanging (https://github.com/mosaicml/streaming/pull/795)
* Changed the search for coldest shard to avoid looping over remote shards by considering local shards only as possible candidates for eviction.




What's Changed
* Bump pydantic from 2.9.1 to 2.9.2 by dependabot in https://github.com/mosaicml/streaming/pull/785
* Bump fastapi from 0.114.2 to 0.115.0 by dependabot in https://github.com/mosaicml/streaming/pull/786
* Bump uvicorn from 0.30.6 to 0.31.0 by dependabot in https://github.com/mosaicml/streaming/pull/793
* Fixed broken links in README.md by LukaszSztukiewicz in https://github.com/mosaicml/streaming/pull/794
* Shard evict fix by snarayan21 in https://github.com/mosaicml/streaming/pull/795
* Update huggingface-hub requirement from <0.25,>=0.23.4 to >=0.23.4,<0.26 by dependabot in https://github.com/mosaicml/streaming/pull/787
* Fix dataset.size() typo in docs by snarayan21 in https://github.com/mosaicml/streaming/pull/798
* Warning -> info about defaults from v0.7.0 by snarayan21 in https://github.com/mosaicml/streaming/pull/799
* Bump uvicorn from 0.31.0 to 0.31.1 by dependabot in https://github.com/mosaicml/streaming/pull/803
* Bump fastapi from 0.115.0 to 0.115.2 by dependabot in https://github.com/mosaicml/streaming/pull/804
* Introducing Streaming Guru on Gurubase.io by kursataktas in https://github.com/mosaicml/streaming/pull/805
* Add better error message for shared prefix by XiaohanZhangCMU in https://github.com/mosaicml/streaming/pull/806
* Bump uvicorn from 0.31.1 to 0.32.0 by dependabot in https://github.com/mosaicml/streaming/pull/809
* Bump pytest-split from 0.9.0 to 0.10.0 by dependabot in https://github.com/mosaicml/streaming/pull/810
* Fix logo png by XiaohanZhangCMU in https://github.com/mosaicml/streaming/pull/808
* Update huggingface-hub requirement from <0.26,>=0.23.4 to >=0.23.4,<0.27 by dependabot in https://github.com/mosaicml/streaming/pull/814
* Bump fastapi from 0.115.2 to 0.115.4 by dependabot in https://github.com/mosaicml/streaming/pull/815
* Fix shared memory permission issue in a shared pod environment by XiaohanZhangCMU in https://github.com/mosaicml/streaming/pull/813

New Contributors
* LukaszSztukiewicz made their first contribution in https://github.com/mosaicml/streaming/pull/794
* kursataktas made their first contribution in https://github.com/mosaicml/streaming/pull/805

**Full Changelog**: https://github.com/mosaicml/streaming/compare/v0.9.0...v0.9.1

Streaming `v0.9.0` is released! Install via `pip`:


pip install --upgrade mosaicml-streaming==0.9.0


Whats new
1. Improved compatibility for ndarray and json types (776, 777)
It is now possible to have columns including a map type successfully convert to JSON in an MDS file if the given type for the column is specified as 'json', and allows the JSON encoder to handle ndarray types.

What's Changed
* Bump fastapi from 0.112.1 to 0.112.2 by dependabot in https://github.com/mosaicml/streaming/pull/768
* Bump ci testing by snarayan21 in https://github.com/mosaicml/streaming/pull/770
* Bump jupyter from 1.0.0 to 1.1.1 by dependabot in https://github.com/mosaicml/streaming/pull/772
* Bump fastapi from 0.112.2 to 0.114.0 by dependabot in https://github.com/mosaicml/streaming/pull/779
* Bump pydantic from 2.8.2 to 2.9.1 by dependabot in https://github.com/mosaicml/streaming/pull/778
* Allow JSON encoder to handle ndarray by srowen in https://github.com/mosaicml/streaming/pull/777
* Add MapType as JSON-compatible by srowen in https://github.com/mosaicml/streaming/pull/776
* Bump fastapi from 0.114.0 to 0.114.2 by dependabot in https://github.com/mosaicml/streaming/pull/783
* Update datasets requirement from <3,>=2.4.0 to >=2.4.0,<4 by dependabot in https://github.com/mosaicml/streaming/pull/784
* Bump pytest from 8.3.2 to 8.3.3 by dependabot in https://github.com/mosaicml/streaming/pull/782
* Bump main branch to 0.10.0.dev0 by dakinggg in https://github.com/mosaicml/streaming/pull/790


**Full Changelog**: https://github.com/mosaicml/streaming/compare/v0.8.1...v0.9.0

🐛 Bug Fixes

1. **Fixed Notebook Progress Bars in Colab**

Fixes a bug introduced by 1264 which causes Composer running in Colab notebooks to error out with:
UnsupportedOperation: fileno.

Closes 1312. Fixed in PR 1314.

Changelog

https://github.com/mosaicml/composer/compare/v0.8.1...v0.8.2

Streaming `v0.8.1` is released! Install via `pip`:


pip install --upgrade mosaicml-streaming==0.8.1


🔧 Improvements
**Dataloader hanging between epochs has now been resolved!** We've seen training time improvements of up to 40% for some many-epoch training jobs. If this was impacting your runs and has now been fixed, please let us know!
* Fix dataloader hang at the end of an epoch by XiaohanZhangCMU in https://github.com/mosaicml/streaming/pull/741
* Add default compression, and warning about local paths to dataframe_to_mds by srowen in https://github.com/mosaicml/streaming/pull/748
* Throw exception when event.is_set() after write()s by srowen in https://github.com/mosaicml/streaming/pull/754

🐛 Bug Fixes
* Ensure deterministic sample order between epochs when `shuffle=False` by snarayan21 in https://github.com/mosaicml/streaming/pull/750

What's Changed
* Make Pytest log in color in Github Action by eitanturok in https://github.com/mosaicml/streaming/pull/739
* fix azure container name and blob name in download_from_azure by jaehwana2z in https://github.com/mosaicml/streaming/pull/733
* Bump uvicorn from 0.30.3 to 0.30.5 by dependabot in https://github.com/mosaicml/streaming/pull/743
* Update huggingface-hub requirement from <0.24,>=0.23.4 to >=0.23.4,<0.25 by dependabot in https://github.com/mosaicml/streaming/pull/729
* Bump fastapi from 0.111.1 to 0.112.0 by dependabot in https://github.com/mosaicml/streaming/pull/744
* Bump ci-testing to v0.1.0 by snarayan21 in https://github.com/mosaicml/streaming/pull/745
* Patching conf.py due to Sphinx deprecating config manipulation by snarayan21 in https://github.com/mosaicml/streaming/pull/746
* Bump ci-testing to v0.1.2 by snarayan21 in https://github.com/mosaicml/streaming/pull/747
* Type hints conformant with pep 585 by snarayan21 in https://github.com/mosaicml/streaming/pull/752
* Ruff rule to remove unused imports by snarayan21 in https://github.com/mosaicml/streaming/pull/756
* Fix linting for numpy 2.1.0 by snarayan21 in https://github.com/mosaicml/streaming/pull/764
* Bump fastapi from 0.112.0 to 0.112.1 by dependabot in https://github.com/mosaicml/streaming/pull/760
* Bump uvicorn from 0.30.5 to 0.30.6 by dependabot in https://github.com/mosaicml/streaming/pull/762
* Version 0.8.1 bump! by snarayan21 in https://github.com/mosaicml/streaming/pull/766

New Contributors
* eitanturok made their first contribution in https://github.com/mosaicml/streaming/pull/739
* jaehwana2z made their first contribution in https://github.com/mosaicml/streaming/pull/733
* srowen made their first contribution in https://github.com/mosaicml/streaming/pull/748

**Full Changelog**: https://github.com/mosaicml/streaming/compare/v0.8.0...v0.8.1