Mosaicml

New Features

1. **🪵 Logging and ObjectStore Enhancements**

There are multiple improvements to our logging and object store support in this release.

- *Image visualization using our `CometMLLogger` ([1710](https://github.com/mosaicml/composer/pull/1710))*

We've added support for using our `ImageVisualizer` callback with [CometML](https://www.comet.com/site/) to log images and segmentation masks to CometML.
python
from composer.trainer import Trainer

trainer = Trainer(...,
callbacks=[ImageVisualizer()],
loggers=[CometMLLogger()]
)


- *Added direct support for [Oracle Cloud Infrastructure (OCI)](https://www.oracle.com/cloud/storage/object-storage/) as an `ObjectStore` ([#1774](https://github.com/mosaicml/composer/pull/1774)) and support for Google Cloud Storage (GCS) via URI ([#1833](https://github.com/mosaicml/composer/pull/1833))*

To use, you can simply set your `save_folder` or `load_path` to a URI beginning with `oci://` or `gs://`, to save and load with OCI and GCS respectively.
python
from composer.trainer import Trainer

Checkpoint saving to Google Cloud Storage.
trainer = Trainer(
model=model,
save_folder="gs://my-bucket/{run_name}/checkpoints",
run_name='my-run',
save_interval="1ep",
save_filename="ep{epoch}.pt",
save_num_checkpoints_to_keep=0, delete all checkpoints locally
...
)

trainer.fit()


- *Added basic support for logging with [MLFlow](https://www.mlflow.org/docs/latest/tracking.html) ([#1795](https://github.com/mosaicml/composer/pull/1795))*

We've added basic support for using MLFlow to log experiment metrics.
python
from composer.loggers import MLFlowLogger
from composer.trainer import Trainer

mlflow_logger = MLFlowLogger(experiment_name=mlflow_exp_name,
run_name=mlflow_run_name,
tracking_uri=mlflow_uri)
trainer = Trainer(..., loggers=[mlflow_logger])


- *Simplified console and progress bar logging ([1694](https://github.com/mosaicml/composer/pull/1694))*

To turn off the progress bar, set `progress_bar=False`. To turn on logging directly to the console, set `log_to_console=True`. To control the frequency of logging to console, set `console_log_interval` (e.g. to `1ep` or `1ba`).

- *[`getfile`](https://docs.mosaicml.com/en/latest/apireference/generated/composer.utils.getfile.html) supports URIs ([#1750](https://github.com/mosaicml/composer/pull/1750))*

Our `get_file` utility now supports URIs directly (`s3://`, `oci://`, and `gs://`) for downloading files.

1. **🏃‍♀️ Support for Mid-Epoch Resumption with the latest release of [Streaming](https://github.com/mosaicml/streaming)**

We've added support in Composer for the latest release of our Streaming library. This includes awesome new features like instant mid epoch resumption and deterministic shuffling, regardless of the number of nodes. See the [Streaming release notes](https://github.com/mosaicml/streaming/releases/tag/v0.2.0) for more!

1. **🚨 New algorithm - `GyroDropout`!**

Thanks to [jelite](https://github.com/jelite) for adding a new algorithm, `GyroDropout` to Composer! Please see the [method card](https://docs.mosaicml.com/en/latest/method_cards/gyro_dropout.html) for more details.

1. **🤗 HuggingFace + Composer improvements**

We've added a new utility to load a 🤗 HuggingFace model and tokenizer out of a Composer checkpoint ([1754](https://github.com/mosaicml/composer/pull/1754)), making the pretraining -> finetuning workflow even easier in Composer. Check out the [docs](https://docs.mosaicml.com/en/latest/api_reference/generated/composer.models.HuggingFaceModel.html#composer.models.HuggingFaceModel.hf_from_composer_checkpoint) for more details, and our [example notebook](https://docs.mosaicml.com/en/latest/examples/pretrain_finetune_huggingface.html) for a full tutorial ([#1775](https://github.com/mosaicml/composer/pull/1775))!

1. **🎓 GradMonitor -> OptimizerMonitor**

Renames our `GradMonitor` callback to `OptimizerMonitor`, and adds the ability to track optimizer specific metrics. Check out the [docs](https://docs.mosaicml.com/en/latest/api_reference/generated/composer.callbacks.OptimizerMonitor.html) for more details, and add to your code just like any other callback!
python
from composer.callbacks import OptimizerMonitor
from composer.trainer import Trainer

trainer = Trainer(
...,
callbacks=[OptimizerMonitor(log_optimizer_metrics=log_optimizer_metrics)]
)


1. **🐳 New PyTorch and CUDA versions**

We've expanded our library of Docker images with support for PyTorch 1.13 + CUDA 11.7:
- `mosaicml/pytorch:1.13.0_cu117-python3.10-ubuntu20.04`
- `mosaicml/pytorch:1.13.0_cpu-python3.10-ubuntu20.04`

The `mosaicml/pytorch:latest`, `mosaicml/pytorch:cpu_latest` and `mosaicml/composer:0.12.0` tags are now built from PyTorch 1.13 based images. Please see our [DockerHub repository](https://hub.docker.com/orgs/mosaicml/repositories) for additional details.

API changes

1. **Replace `grad_accum` with `device_train_microbatch_size`** ([1749](https://github.com/mosaicml/composer/pull/1749), [#1776](https://github.com/mosaicml/composer/pull/1776))

We're deprecating the `grad_accum` Trainer argument in favor of the more intuitive `device_train_microbatch_size`. Instead of thinking about how to divide your specified minibatch into microbatches, simply specify the size of your microbatch. For example, let's say you want to split your minibatch of 2048 into two microbatches of 1024:

python
from composer import Trainer

trainer = Trainer(
...,
device_train_microbatch_size=1024,
)


If you want Composer to tune the microbatch for you automatically, enable automatic microbatching as follows:

python
from composer import Trainer

trainer = Trainer(
...,
device_train_microbatch_size='auto',
)


The `grad_accum` argument is still supported but will be deprecated in the next Composer release.

1. **Renamed precisions** ([1761](https://github.com/mosaicml/composer/pull/1761))

We've renamed precision attributes for clarity. The following values have been removed: ``['amp', 'fp16', bf16']``.

We have added the following values, prefixed with 'amp' to clarify when an Automatic Mixed Precision type is being used: ``['amp_fp16', 'amp_bf16']``.

The `fp32` precision value remains unchanged.


Deprecations

1. Removed support for YAHP ([1512](https://github.com/mosaicml/composer/pull/1512))
1. Removed COCO and SSD datasets ([1717](https://github.com/mosaicml/composer/pull/1717))
1. Fully removed Streaming v1 support, please see the [mosaicml/streaming](https://github.com/mosaicml/streaming) project for our next-gen streaming datasets ([#1787](https://github.com/mosaicml/composer/pull/1787))
1. Deprecated `FusedLayerNorm` algorithm ([1789](https://github.com/mosaicml/composer/pull/1789))
1. Fully removed `grad_clip_norm` training argument, please use the `GradientClipping` algorithm instead ([1768](https://github.com/mosaicml/composer/pull/1768))
1. Removed `data_fit`, `data_epoch`, and `data_batch` from `Logger` ([1826](https://github.com/mosaicml/composer/pull/1826))

Bug Fixes

- Fix FSDP checkpoint strategy ([1734](https://github.com/mosaicml/composer/pull/1734))
- Fix gradient clipping with FSDP ([1740](https://github.com/mosaicml/composer/pull/1740))
- Adds more supported FSDP config flags (`sync_module_states`, `forward_prefecth`, `limit_all_gathers`) ([1794](https://github.com/mosaicml/composer/pull/1794))
- Allow `FULL` precision with FSDP ([1796](https://github.com/mosaicml/composer/pull/1796))
- Fix `eval_microbatch` modification on `EVAL_BEFORE_FORWARD` event ([1739](https://github.com/mosaicml/composer/pull/1739))
- Fix algorithm API backwards compatibility in checkpoints ([1741](https://github.com/mosaicml/composer/pull/1741))
- Fixes a bad `None` check preventing setting `device_id` to `0` ([1767](https://github.com/mosaicml/composer/pull/1767))
- Unregister engine to make cleaning up memory easier ([1769](https://github.com/mosaicml/composer/pull/1769))
- Fix issue if `metric_names` is not a list ([1798](https://github.com/mosaicml/composer/pull/1798))
- Match implementation for list and tensor batch splitting ([1804](https://github.com/mosaicml/composer/pull/1804))
- Fixes infinite eval issue ([1815](https://github.com/mosaicml/composer/pull/1815))


What's Changed
* Update installation constraints for streaming by karan6181 in https://github.com/mosaicml/composer/pull/1661
* Update decoupled_weight_decay.md by jacobfulano in https://github.com/mosaicml/composer/pull/1672
* Notebooks part 2 by dakinggg in https://github.com/mosaicml/composer/pull/1659
* Add trainer arg for engine passes by mvpatel2000 in https://github.com/mosaicml/composer/pull/1673
* Autoload algorithms by mvpatel2000 in https://github.com/mosaicml/composer/pull/1658
* Faster metrics calculations + Fix warnings added by the new version of torchmetrics by dskhudia in https://github.com/mosaicml/composer/pull/1674
* Update coolname requirement from <2,>=1.1.0 to >=1.1.0,<3 by dependabot in https://github.com/mosaicml/composer/pull/1666
* Bump ipykernel from 6.16.0 to 6.16.1 by dependabot in https://github.com/mosaicml/composer/pull/1667
* Bump traitlets from 5.4.0 to 5.5.0 by dependabot in https://github.com/mosaicml/composer/pull/1668
* Image viz by dakinggg in https://github.com/mosaicml/composer/pull/1676
* Update checks for Gated Linear Units Method by jacobfulano in https://github.com/mosaicml/composer/pull/1575
* ADE20k streaming factory method by Landanjs in https://github.com/mosaicml/composer/pull/1626
* Deyahpify cifar10 by growlix in https://github.com/mosaicml/composer/pull/1677
* Nuke YAHP by hanlint in https://github.com/mosaicml/composer/pull/1512
* Imagenet streaming factory method by codestar12 in https://github.com/mosaicml/composer/pull/1649
* Bump ipykernel from 6.16.1 to 6.16.2 by dependabot in https://github.com/mosaicml/composer/pull/1683
* Bump pytest from 7.1.3 to 7.2.0 by dependabot in https://github.com/mosaicml/composer/pull/1684
* Bump pypandoc from 1.9 to 1.10 by dependabot in https://github.com/mosaicml/composer/pull/1680
* Update py-cpuinfo requirement from <9,>=8.0.0 to >=8.0.0,<10 by dependabot in https://github.com/mosaicml/composer/pull/1681
* Uncomment and clean up algorithms documentation by growlix in https://github.com/mosaicml/composer/pull/1685
* Update glu check by mvpatel2000 in https://github.com/mosaicml/composer/pull/1689
* fix backwards compatability by mvpatel2000 in https://github.com/mosaicml/composer/pull/1693
* Fix engine pass registration by mvpatel2000 in https://github.com/mosaicml/composer/pull/1692
* Add Low Precision LayerNorm by nik-mosaic in https://github.com/mosaicml/composer/pull/1525
* Update codeowners by mvpatel2000 in https://github.com/mosaicml/composer/pull/1691
* Add nccl env var by mvpatel2000 in https://github.com/mosaicml/composer/pull/1695
* Fix eval timestamp by mvpatel2000 in https://github.com/mosaicml/composer/pull/1697
* Update distributed docs by mvpatel2000 in https://github.com/mosaicml/composer/pull/1696
* Return empty dict if wandb disabled by dakinggg in https://github.com/mosaicml/composer/pull/1698
* Autoresume related error messages by dakinggg in https://github.com/mosaicml/composer/pull/1687
* Add log_image to wandb, cometml, and LoggerDestination by eracah in https://github.com/mosaicml/composer/pull/1675
* Pin PyTorch and supporting package versions by bandish-shah in https://github.com/mosaicml/composer/pull/1688
* Add in unit tests for log_image function for CometMLLogger and WandBLogger by eracah in https://github.com/mosaicml/composer/pull/1701
* refactor devices by mvpatel2000 in https://github.com/mosaicml/composer/pull/1699
* remove as in device by mvpatel2000 in https://github.com/mosaicml/composer/pull/1704
* Fix device imports by mvpatel2000 in https://github.com/mosaicml/composer/pull/1705
* Fix typing in EMA's _move_params_to_device() by coryMosaicML in https://github.com/mosaicml/composer/pull/1707
* Add docs for saving and loading checkpoints with GCS by eracah in https://github.com/mosaicml/composer/pull/1702
* Clean up imports by mvpatel2000 in https://github.com/mosaicml/composer/pull/1700
* Add rud docs by eracah in https://github.com/mosaicml/composer/pull/1709
* Bump cryptography from 38.0.1 to 38.0.3 by dependabot in https://github.com/mosaicml/composer/pull/1712
* GHA workflow for code quality checks by bandish-shah in https://github.com/mosaicml/composer/pull/1719
* Add support for Path in CheckpointSaver by cojennin in https://github.com/mosaicml/composer/pull/1721
* Docs Typo by mvpatel2000 in https://github.com/mosaicml/composer/pull/1723
* Bump nbsphinx from 0.8.9 to 0.8.10 by dependabot in https://github.com/mosaicml/composer/pull/1725
* Bump sphinx-argparse from 0.3.2 to 0.4.0 by dependabot in https://github.com/mosaicml/composer/pull/1726
* Simple nlp tests by dakinggg in https://github.com/mosaicml/composer/pull/1716
* Build Streaming CIFAR10 Factory Function by growlix in https://github.com/mosaicml/composer/pull/1729
* Change `build_streaming_cifar10_dataloader()` to use v2 by default by growlix in https://github.com/mosaicml/composer/pull/1730
* Clear the Optimizer before wrapping with FSDP by bcui19 in https://github.com/mosaicml/composer/pull/1732
* Add inf eval check by mvpatel2000 in https://github.com/mosaicml/composer/pull/1733
* Fix fsdp checkpoint strategy by bcui19 in https://github.com/mosaicml/composer/pull/1734
* Assign eval microbatch to self.state.batch by dakinggg in https://github.com/mosaicml/composer/pull/1739
* Add masks to wandblogger.log_image and cometmllogger.log_image and refactor ImageVisualizer to use log_image [WIP] by eracah in https://github.com/mosaicml/composer/pull/1710
* Protect backwards compatability by mvpatel2000 in https://github.com/mosaicml/composer/pull/1741
* Add composer version state by dakinggg in https://github.com/mosaicml/composer/pull/1742
* Adds auto object store creation to `get_file` by dakinggg in https://github.com/mosaicml/composer/pull/1750
* Log console interval by eracah in https://github.com/mosaicml/composer/pull/1694
* Bump sphinxcontrib-katex from 0.9.0 to 0.9.3 by dependabot in https://github.com/mosaicml/composer/pull/1757
* Bump pandoc from 2.2 to 2.3 by dependabot in https://github.com/mosaicml/composer/pull/1756
* Bump cryptography from 38.0.3 to 38.0.4 by dependabot in https://github.com/mosaicml/composer/pull/1755
* Add more event tests by mvpatel2000 in https://github.com/mosaicml/composer/pull/1762
* Add python 3.10, pytorch 1.13, cuda 11.7 by mvpatel2000 in https://github.com/mosaicml/composer/pull/1735
* Add huggingface info to state dict by dakinggg in https://github.com/mosaicml/composer/pull/1744
* Global batch size by mvpatel2000 in https://github.com/mosaicml/composer/pull/1746
* Add device to state by mvpatel2000 in https://github.com/mosaicml/composer/pull/1765
* Rename precisions by mvpatel2000 in https://github.com/mosaicml/composer/pull/1761
* Device id none by dakinggg in https://github.com/mosaicml/composer/pull/1767
* Autoload HuggingFace model/tokenizer by dakinggg in https://github.com/mosaicml/composer/pull/1754
* Supporting `train_device_microbatch_size` by mvpatel2000 in https://github.com/mosaicml/composer/pull/1749
* Switch flash attention to tag by mvpatel2000 in https://github.com/mosaicml/composer/pull/1766
* remove grad clip norm by mvpatel2000 in https://github.com/mosaicml/composer/pull/1768
* unregister engine for memory cleanup by mvpatel2000 in https://github.com/mosaicml/composer/pull/1769
* Fix hf tokenizer test for new hf version by dakinggg in https://github.com/mosaicml/composer/pull/1772
* Decrease microbatch size if batch size is smaller by mvpatel2000 in https://github.com/mosaicml/composer/pull/1771
* remove deprecated code by mvpatel2000 in https://github.com/mosaicml/composer/pull/1773
* cache call to cpuinfo by dakinggg in https://github.com/mosaicml/composer/pull/1778
* device train microbatch size pt 2 by mvpatel2000 in https://github.com/mosaicml/composer/pull/1776
* Huggingface pretrain + finetune notebook by dakinggg in https://github.com/mosaicml/composer/pull/1775
* Bump traitlets from 5.5.0 to 5.6.0 by dependabot in https://github.com/mosaicml/composer/pull/1781
* Bump deepspeed from 0.7.5 to 0.7.6 by dependabot in https://github.com/mosaicml/composer/pull/1780
* Minor docs fix for deepspeed typo by mvpatel2000 in https://github.com/mosaicml/composer/pull/1784
* Update Auto Microbatching by mvpatel2000 in https://github.com/mosaicml/composer/pull/1785
* Adding GyroDropout as an algorithm to Composer by jelite in https://github.com/mosaicml/composer/pull/1718
* Add Deprecation warning for Fused LayerNorm by nik-mosaic in https://github.com/mosaicml/composer/pull/1789
* Update error msgs by mvpatel2000 in https://github.com/mosaicml/composer/pull/1791
* Change gyro emoji by nik-mosaic in https://github.com/mosaicml/composer/pull/1792
* Speeding up tests by dakinggg in https://github.com/mosaicml/composer/pull/1779
* Add durations arg to pytest by dakinggg in https://github.com/mosaicml/composer/pull/1793
* Properly implement gradient clipping for FSDP by bcui19 in https://github.com/mosaicml/composer/pull/1740
* Updating FSDP supported config flags by bcui19 in https://github.com/mosaicml/composer/pull/1794
* Remove streaming v1 datasets. by knighton in https://github.com/mosaicml/composer/pull/1787
* Remove references to validate in docs by dakinggg in https://github.com/mosaicml/composer/pull/1800
* Install latest Git in Docker images by bandish-shah in https://github.com/mosaicml/composer/pull/1770
* move to pypi release for flash attn by mvpatel2000 in https://github.com/mosaicml/composer/pull/1777
* Check and make sure that metric names is a list of strings by dakinggg in https://github.com/mosaicml/composer/pull/1798
* Adding in the possibility of 'None' for MixedPrecision FSDP by bcui19 in https://github.com/mosaicml/composer/pull/1796
* Updating assertion check for gradient clipping and updating gradient clip tests for FSDP by bcui19 in https://github.com/mosaicml/composer/pull/1802
* Moving Pytest CPU to GHA by mvpatel2000 in https://github.com/mosaicml/composer/pull/1790
* Bump sphinxext-opengraph from 0.6.3 to 0.7.3 by dependabot in https://github.com/mosaicml/composer/pull/1760
* Update distributed_training.rst by lupesko in https://github.com/mosaicml/composer/pull/1731
* Use streaming v3 by knighton in https://github.com/mosaicml/composer/pull/1797
* Bump traitlets from 5.6.0 to 5.7.0 by dependabot in https://github.com/mosaicml/composer/pull/1806
* Bump ipykernel from 6.16.2 to 6.19.2 by dependabot in https://github.com/mosaicml/composer/pull/1810
* Update packaging requirement from <22,>=21.3.0 to >=21.3.0,<23 by dependabot in https://github.com/mosaicml/composer/pull/1808
* match list batch splitting and tensor batch splitting by dakinggg in https://github.com/mosaicml/composer/pull/1804
* Add type ignore for onnx import by mvpatel2000 in https://github.com/mosaicml/composer/pull/1811
* Remove pip install all from coverage action by dakinggg in https://github.com/mosaicml/composer/pull/1805
* Remove coco and ssd by growlix in https://github.com/mosaicml/composer/pull/1717
* Rename matrix by mvpatel2000 in https://github.com/mosaicml/composer/pull/1813
* Add OCI ObjectStore by eracah in https://github.com/mosaicml/composer/pull/1774
* Add MLFlowLogger by eracah in https://github.com/mosaicml/composer/pull/1795
* Object store docs by dakinggg in https://github.com/mosaicml/composer/pull/1817
* fix inf eval by mvpatel2000 in https://github.com/mosaicml/composer/pull/1815
* Add `fsdp_config` to `state` and add fsdp_config to trainer docstring by growlix in https://github.com/mosaicml/composer/pull/1821
* Add SHARP support to docker by mvpatel2000 in https://github.com/mosaicml/composer/pull/1818
* Testing Infra Cleanup by mvpatel2000 in https://github.com/mosaicml/composer/pull/1822
* Remove dead code in dockerfile by mvpatel2000 in https://github.com/mosaicml/composer/pull/1823
* Fix Export Docs by mvpatel2000 in https://github.com/mosaicml/composer/pull/1824
* Remove old deprecated logger methods by eracah in https://github.com/mosaicml/composer/pull/1826
* NLP metrics tests by dakinggg in https://github.com/mosaicml/composer/pull/1830
* Nlp pipeline test by dakinggg in https://github.com/mosaicml/composer/pull/1828
* Add tests for uri helper functions by eracah in https://github.com/mosaicml/composer/pull/1827
* Add pip targets to installation.rst docs by eracah in https://github.com/mosaicml/composer/pull/1829

New Contributors
* cojennin made their first contribution in https://github.com/mosaicml/composer/pull/1721
* jelite made their first contribution in https://github.com/mosaicml/composer/pull/1718

**Full Changelog**: https://github.com/mosaicml/composer/compare/v0.11.1...v0.12.0

Bug Fixes

* Fixes for Notebooks (1659)
* Documentation updates and fixes (1685, 1696, 1702, 1709)
* Addressed warnings and speed improvements for Torchmetrics (1674)
* Fixes to Gated Linear Units method (1575, 1689)
* Set `NCCL_ASYNC_ERROR_HANDLING` ENV variable in Composer launcher to enable distributed timeout (1695)
* Fix epoch count when `eval` is called before `fit` (1697)
* Constrain PyTorch package versions to avoid unintended upgrades (1688)
* Fix Optimizer state sharding issue with FSDP (1732)
* Rase `ValueError` with if evaluation dataloader of infinite length is specified

**Full Changelog**: https://github.com/mosaicml/composer/compare/v0.11.0...v0.11.1

New Features

1. **🧰 FSDP Beta Support**

Composer now supports [PyTorch FSDP](https://pytorch.org/docs/stable/fsdp.html)! PyTorch FSDP is a strategy for distributed training, similar to PyTorch DDP, that distributes work using data-parallelism only. On top of this, FSDP uses model, gradient, and optimizer sharding to dramatically reduce device memory requirements, and enables users to easily scale and train large models.

Here's how easy it is to use FSDP with Composer:
python
import torch.nn as nn
from composer import Trainer

class Block (nn.Module):
...

Your custom model
class Model(nn.Module):
def __init__(self, n_layers):
super().__init__()
self.blocks = nn.ModuleList([
Block(...) for _ in range(n_layers)
]),
self.head = nn.Linear(...)
def forward(self, inputs):
...

FSDP Wrap Function
def fsdp_wrap_fn(self, module):
return isinstance(module, Block)

Activation Checkpointing Function
def activation_checkpointing_fn(self, module):
return isinstance(module, Block)

ComposerModel wrapper, used by the Trainer
to compute loss, metrics, etc.
class MyComposerModel(ComposerModel):

def __init__(self, n_layers):
super().__init__()
self.model = Model(n_layers)
...

def forward(self, batch):
...

def eval_forward(self, batch, outputs=None):
...

def loss(self, outputs, batch):
...

Pass your ComposerModel and fsdp_config into the Trainer
composer_model = MyComposerModel(n_layers=3)
fsdp_config = {
'sharding_strategy': 'FULL_SHARD',
'min_params': 1e8,
'cpu_offload': False, Not supported yet
'mixed_precision': 'DEFAULT',
'backward_prefetch': 'BACKWARD_POST',
'activation_checkpointing': False,
'activation_cpu_offload': False,
'verbose': True
}

trainer = Trainer(
model=composer_model,
fsdp_config=fsdp_config,
...
)

trainer.fit()



For more information, please see our [FSDP docs](https://docs.mosaicml.com/en/v0.11.0/notes/distributed_training.html#fullyshardeddataparallel-fsdp).

New Features

1. **𐄷 Weight Standardization**

Weight Standardization reparametrizes convolutional weights such that the fan-in dimensions have zero mean and unit standard deviation. This could slightly improve performance at the expensive of 5% lower throughput. This has been used in several papers to train with smaller batch sizes, with normalization layers besides batch norm, and for transfer learning.


Using Weight Standardization with the Composer Trainer:

python
import composer

Apply Weight Standardization (when training is initialized)
weight_std = composer.algorithms.WeightStandardization()

Train with Weight Standardization
trainer = composer.trainer.Trainer(
...
algorithms=[weight_std]
)
trainer.fit()


Using Weight Standardization with the Composer functional interface:

python
import composer
from torchvision.models import resnet50

my_model = resnet50()

Apply weight standardization to model
my_model = composer.functional.weight_standardization(my_model)


Please see the [Weight Standardization Method Card](https://docs.mosaicml.com/en/stable/method_cards/weight_standardization.html) for more details.

Bug Fixes

* Fix for checkpoints not being saved automatically at the end of a run (1552)
* Fix Onnx export for Composer HuggingFaceModels (1557)
* Fix for MIoU metric producing NaN's (1558)
* CometML logger documentation updates and fixes (1567, 1570, 1571)
* WandB image visualizer fix (1591)

What's Changed
* Update evaluate_periodically() when eval interval is of type Duration by karan6181 in https://github.com/mosaicml/composer/pull/1523
* Quality of life updates to EMA by coryMosaicML in https://github.com/mosaicml/composer/pull/1524
* Add ADE20K and COCO v2 dataset behind a version flag by karan6181 in https://github.com/mosaicml/composer/pull/1528
* Pinned setuptools version to fix distutils version error by karan6181 in https://github.com/mosaicml/composer/pull/1536
* Less strict name formatting by hanlint in https://github.com/mosaicml/composer/pull/1535
* Defaulting streaming dataset version to 1 and add a deprecation warning by karan6181 in https://github.com/mosaicml/composer/pull/1532
* Changing 'stable' to 'latest' in notebooks in examples by bcui19 in https://github.com/mosaicml/composer/pull/1534
* Bump furo from 2022.6.21 to 2022.9.15 by dependabot in https://github.com/mosaicml/composer/pull/1540
* Bump fasteners from 0.17.3 to 0.18 by dependabot in https://github.com/mosaicml/composer/pull/1538
* Add Pandoc to Docker images, bump version to 2.19.2 by bandish-shah in https://github.com/mosaicml/composer/pull/1550
* Removed streaming version 2 from yaml since version 1 is default by karan6181 in https://github.com/mosaicml/composer/pull/1551
* Bump ipykernel from 6.15.2 to 6.15.3 by dependabot in https://github.com/mosaicml/composer/pull/1548
* Bump yamllint from 1.27.1 to 1.28.0 by dependabot in https://github.com/mosaicml/composer/pull/1546
* Bump traitlets from 5.3.0 to 5.4.0 by dependabot in https://github.com/mosaicml/composer/pull/1539
* Object Store Logger Race Condition + EMA Fix by mvpatel2000 in https://github.com/mosaicml/composer/pull/1552
* Adding in erroring for when using GradMonitor and DeepSpeed by bcui19 in https://github.com/mosaicml/composer/pull/1555
* Bump pypandoc from 1.8.1 to 1.9 by dependabot in https://github.com/mosaicml/composer/pull/1559
* Update context to raise errror by mvpatel2000 in https://github.com/mosaicml/composer/pull/1561
* Fix MIoU metric when `self.total_union==0` by abhi-mosaic in https://github.com/mosaicml/composer/pull/1558
* Move dataloader `initialize_object` to factory methods by hanlint in https://github.com/mosaicml/composer/pull/1510
* Weight Standardization method by Landanjs in https://github.com/mosaicml/composer/pull/1562
* Update comet links to include query params and point to main site by dakinggg in https://github.com/mosaicml/composer/pull/1567
* remove dead line in alibi by mvpatel2000 in https://github.com/mosaicml/composer/pull/1568
* GLU Fixes by mvpatel2000 in https://github.com/mosaicml/composer/pull/1564
* Add FSDP strategy by abhi-mosaic in https://github.com/mosaicml/composer/pull/1553
* Comet example by dakinggg in https://github.com/mosaicml/composer/pull/1570
* Add missing _enabled flag, post_close, and clean up comet ml tests by dakinggg in https://github.com/mosaicml/composer/pull/1571
* Consistent Method Card Style by growlix in https://github.com/mosaicml/composer/pull/1407
* add missing return in context by mvpatel2000 in https://github.com/mosaicml/composer/pull/1574
* Remove eval batch split by mvpatel2000 in https://github.com/mosaicml/composer/pull/1576
* Fix Onnx Export for Composer HuggingFaceModels by nik-mosaic in https://github.com/mosaicml/composer/pull/1557
* Revert checkpoint rename by hanlint in https://github.com/mosaicml/composer/pull/1579

New Contributors
* bcui19 made their first contribution in https://github.com/mosaicml/composer/pull/1534

**Full Changelog**: https://github.com/mosaicml/composer/compare/v0.10.0...v0.10.1

New Features

1. **:comet: Comet Experiment Tracking (1490)**

We've added support for the popular [Comet](https://www.comet.com) experiment tracker! To enable, simply create the logger and pass it to the `Trainer` object at initialization:

python
from composer import Trainer
from composer.loggers import CometMLLogger

cometml_logger = CometMLLogger()

trainer = Trainer(
...
loggers=[cometml_logger],
)


Please see our [Logging](https://docs.mosaicml.com/en/stable/trainer/logging.html) and [CometMLLogger](https://docs.mosaicml.com/en/stable/api_reference/generated/composer.loggers.CometMLLogger.html#composer.loggers.CometMLLogger) docs pages for details on usage.


1. **:magic_wand: Automatic Evaluation Batch Size Selection (1417)**

Composer now supports `eval_batch_size='auto'`, which will choose the right evaluation batch size to avoid CUDA OOMs! Now, in conjunction with `grad_accum='auto'`, you can run the same code on any hardware with no changes necessary. This makes it easy to add evaluation to a training script without having to pick and choose the right batch sizes to avoid CUDA OOMs.

1. **:dart: Evaluation API Changes (1479)**

The Evaluation API has been updated to be consistent with the Trainer API. If the `eval_dataloader` was provided to the Trainer during initialization, `eval` can be invoked without needing to provide anything additional:

python
trainer = Trainer(
eval_dataloader=...
)
trainer.eval()


Alternatively, the `eval_dataloader` can be passed directly to the `eval()` method:

python
trainer = Trainer(
...
)
trainer.eval(
eval_dataloader=...
)


The `eval_dataloader` can be a pytorch dataloader, or for multiple metrics, a list of `Evaluator` objects.

1. :wood: **Simplified Logging (1416)**

We've significantly simplified our internal logging interface:
- Removed the use of `LogLevel` throughout the logging, which was a mostly unused feature. Filtering logs are the responsibility of the logger.
- For better compatibility with external logging interfaces such as CometML or Weights & Biases, loggers now support the following methods: `log_metrics`, `log_hyperparameters`, and `log_artifacts`. Previous calls to `data_fit, data_epeoch, ..` have been removed.

1. :dart: **validate --> eval_forward (1411 , 1419)**

Previously, `ComposerModel` implemented the `validate(batch: Any) -> Tuple[Any, Any]` method which returns an `(input, target)` tuple, and the Trainer handles updating the metrics. In `v0.10`, we return the metrics updating control to the user.

Now, models instead implement `def eval_forward(batch: Any)` which returns the outputs of evaluation, and also `def update_metric(batch, outputs, metric)` which updates the metric.

An example implementation for classification can be found in our `ComposerClassifer` base class:

python
def update_metric(self, batch: Any, outputs: Any, metric: Metric) -> None:
_, targets = batch
metric.update(outputs, targets)

def eval_forward(self, batch: Any, outputs: Optional[Any] = None) -> Any:
return outputs if outputs is not None else self.forward(batch)


1. :female_detective: **Evaluator changes**

The `Evaluator` class now stores evaluation metric _names_ instead of metric _instances_. For example:

python
glue_mrpc_task = Evaluator(
label='glue_mrpc',
dataloader=mrpc_dataloader,
metric_names=['BinaryF1Score', 'Accuracy']
)


These metric names are matched against the metrics returned by the `ComposerModel`. The metric _instances_ are now stored as deep copies in the `State` class as `state.train_metrics` or `state.eval_metrics`.

1. **:construction: Streaming Datasets Repository Preview**

We're in the process of splitting out streaming datasets into it's own repository! Streaming datasets is a high-performance drop-in replacement for Torch `IterableDataset` objects and enables you to stream your training data from cloud based object stores. For an early preview, please checkout the [Streaming repo](https://github.com/mosaicml/streaming).

1. :x: **YAHP deprecation**

We are deprecating support for [yahp](https://github.com/mosaicml/yahp), our hyperparameter configuration tool. Support for this will be removed in the following minor version release of Composer. We recommend users migrate to OmegaConf, or Hydra as tools.

Bug Fixes

* Documentation fixes (1408, 1422, 1425, 1413, 1432, 1403, 1426, 1396, 1446, 1466, 1443)
* Upgrade WandB version (1440)
* fix import (1442)
* fix wrong extra deps group (1449)
* wandb bug fix (1488)
* Reset train metrics every batch (1496)
* fix auto grad accum (1515)
* Fix compression file remote download exception handling (1526)
* Add Pandoc to Docker images, bump version to 2.19.2 (1550)

What's Changed
* current metrics docs by A-Jacobson in https://github.com/mosaicml/composer/pull/1402
* merge nlp+hf notebooks by A-Jacobson in https://github.com/mosaicml/composer/pull/1406
* Add break epoch exception by mvpatel2000 in https://github.com/mosaicml/composer/pull/1415
* Upgrade to torch 1.12.1 by abhi-mosaic in https://github.com/mosaicml/composer/pull/1409
* Metrics refactor pt1 by ishanashastri in https://github.com/mosaicml/composer/pull/1411
* Use state algos by mvpatel2000 in https://github.com/mosaicml/composer/pull/1412
* Add default ignore index by moinnadeem in https://github.com/mosaicml/composer/pull/1421
* Update default hparams for ResNet model card by abhi-mosaic in https://github.com/mosaicml/composer/pull/1423
* update colout link in custom speedup notebook by A-Jacobson in https://github.com/mosaicml/composer/pull/1408
* Clean up prose in key files by dblalock in https://github.com/mosaicml/composer/pull/1422
* Relax codeowners by bandish-shah in https://github.com/mosaicml/composer/pull/1424
* Fix typo by Landanjs in https://github.com/mosaicml/composer/pull/1425
* Fix pre-commit checks failing on fresh checkout of dev by dblalock in https://github.com/mosaicml/composer/pull/1414
* Have docs use preferred import paths, not longest import paths by dblalock in https://github.com/mosaicml/composer/pull/1413
* Fix missing indent by Landanjs in https://github.com/mosaicml/composer/pull/1432
* eval_batch_size=auto by mvpatel2000 in https://github.com/mosaicml/composer/pull/1417
* Simplify helper for conflicting files by hanlint in https://github.com/mosaicml/composer/pull/1427
* add install from dev instructions by A-Jacobson in https://github.com/mosaicml/composer/pull/1403
* Style/tone consistency update for tutorial notebooks by alextrott16 in https://github.com/mosaicml/composer/pull/1426
* Dynamic quantization + minor improvements in inference APIs by dskhudia in https://github.com/mosaicml/composer/pull/1433
* Upgrade WandB version by moinnadeem in https://github.com/mosaicml/composer/pull/1440
* Log multiple losses by Landanjs in https://github.com/mosaicml/composer/pull/1375
* Fix attribute by mvpatel2000 in https://github.com/mosaicml/composer/pull/1442
* Expand evaluation doc by alextrott16 in https://github.com/mosaicml/composer/pull/1396
* Metrics Refactor Part 2 by ishanashastri in https://github.com/mosaicml/composer/pull/1419
* Create dependabot.yml by mvpatel2000 in https://github.com/mosaicml/composer/pull/1448
* Methods overview fix by growlix in https://github.com/mosaicml/composer/pull/1446
* Bump custom-inherit from 2.3.2 to 2.4.0 by dependabot in https://github.com/mosaicml/composer/pull/1451
* Bump junitparser from 2.4.3 to 2.8.0 by dependabot in https://github.com/mosaicml/composer/pull/1453
* Update moto[s3] requirement from <3.2,>=3.1.12 to >=4.0.1,<5 by dependabot in https://github.com/mosaicml/composer/pull/1450
* Update monai requirement from <0.9,>=0.8.0 to >=0.9.0,<0.10 by dependabot in https://github.com/mosaicml/composer/pull/1452
* Update torch-optimizer requirement from <0.2,>=0.1.0 to >=0.3.0,<0.4 by dependabot in https://github.com/mosaicml/composer/pull/1454
* Bump cryptography from 37.0.2 to 37.0.4 by dependabot in https://github.com/mosaicml/composer/pull/1457
* Bump sphinxext-opengraph from 0.6.1 to 0.6.3 by dependabot in https://github.com/mosaicml/composer/pull/1458
* Bump coverage[toml] from 6.3.2 to 6.4.4 by dependabot in https://github.com/mosaicml/composer/pull/1460
* Bump nbsphinx from 0.8.8 to 0.8.9 by dependabot in https://github.com/mosaicml/composer/pull/1459
* Fix incorrect deps group in `streaming` requirement by hanlint in https://github.com/mosaicml/composer/pull/1449
* Logger Destination Refactor by eracah in https://github.com/mosaicml/composer/pull/1416
* Bump sphinx-markdown-tables from 0.0.15 to 0.0.17 by dependabot in https://github.com/mosaicml/composer/pull/1463
* Bump traitlets from 5.1.1 to 5.3.0 by dependabot in https://github.com/mosaicml/composer/pull/1462
* Bump vit-pytorch from 0.27 to 0.35.8 by dependabot in https://github.com/mosaicml/composer/pull/1465
* Bump furo from 2022.3.4 to 2022.6.21 by dependabot in https://github.com/mosaicml/composer/pull/1467
* Bump ipykernel from 6.9.2 to 6.15.1 by dependabot in https://github.com/mosaicml/composer/pull/1470
* Bump pytest from 7.1.0 to 7.1.2 by dependabot in https://github.com/mosaicml/composer/pull/1469
* Bump sphinxcontrib-katex from 0.8.6 to 0.9.0 by dependabot in https://github.com/mosaicml/composer/pull/1476
* Bump tabulate from 0.8.9 to 0.8.10 by dependabot in https://github.com/mosaicml/composer/pull/1478
* Bump yamllint from 1.26.3 to 1.27.1 by dependabot in https://github.com/mosaicml/composer/pull/1481
* Bump ipykernel from 6.15.1 to 6.15.2 by dependabot in https://github.com/mosaicml/composer/pull/1482
* Refactor CheckpointSaver by hanlint in https://github.com/mosaicml/composer/pull/1428
* Clean up docs Makefile by eracah in https://github.com/mosaicml/composer/pull/1466
* Model surgery info -> debug by mvpatel2000 in https://github.com/mosaicml/composer/pull/1485
* Docker image with Flash Attention by abhi-mosaic in https://github.com/mosaicml/composer/pull/1471
* Fix WandBLogger bug with inaccurate step count by eracah in https://github.com/mosaicml/composer/pull/1488
* Update Eval API by hanlint in https://github.com/mosaicml/composer/pull/1479
* Random Names with Fixed Seed by mvpatel2000 in https://github.com/mosaicml/composer/pull/1487
* ResNet50 on ImageNet training script example by Landanjs in https://github.com/mosaicml/composer/pull/1434
* Remove hparams from `test_precision` and `test_state` by hanlint in https://github.com/mosaicml/composer/pull/1486
* Clean up `save_checkpoint` by hanlint in https://github.com/mosaicml/composer/pull/1484
* Remove hparams from test_ddp by hanlint in https://github.com/mosaicml/composer/pull/1489
* update model token embeddings according to tokenizer len by ananyahjha93 in https://github.com/mosaicml/composer/pull/1493
* BERT classifier metrics depend on num_labels by alextrott16 in https://github.com/mosaicml/composer/pull/1495
* Reset train metrics every batch by abhi-mosaic in https://github.com/mosaicml/composer/pull/1496
* Algolia doc search by nqn in https://github.com/mosaicml/composer/pull/1443
* Squelch Engine debug logs by hanlint in https://github.com/mosaicml/composer/pull/1497
* Remove TODO by mvpatel2000 in https://github.com/mosaicml/composer/pull/1499
* Remove hparams from checkpoint tests by hanlint in https://github.com/mosaicml/composer/pull/1491
* [Docs] Training ResNet-50 on AWS tutorial by bandish-shah in https://github.com/mosaicml/composer/pull/1444
* Refactor hparams in tests by hanlint in https://github.com/mosaicml/composer/pull/1498
* Bump pytest from 7.1.2 to 7.1.3 by dependabot in https://github.com/mosaicml/composer/pull/1500
* Improved comments and improved test code by karan6181 in https://github.com/mosaicml/composer/pull/1502
* Refactor GLUE fine-tune queuing to improve efficiency and add task-specific seed sweeps by alextrott16 in https://github.com/mosaicml/composer/pull/1363
* Raise ValueError for Profiler + Auto Grad Accum by mvpatel2000 in https://github.com/mosaicml/composer/pull/1504
* add yahp deprecation warnings by hanlint in https://github.com/mosaicml/composer/pull/1505
* Move logic from `initialize_object` to object store class by hanlint in https://github.com/mosaicml/composer/pull/1508
* Fix run name comment by mvpatel2000 in https://github.com/mosaicml/composer/pull/1509
* Add CometML Support by eracah in https://github.com/mosaicml/composer/pull/1490
* Raise ValueError if missing a surgery algorithm by mvpatel2000 in https://github.com/mosaicml/composer/pull/1506
* remove datasets from gitignore by hanlint in https://github.com/mosaicml/composer/pull/1513
* fix auto grad accum by mvpatel2000 in https://github.com/mosaicml/composer/pull/1515
* Use eval context by mvpatel2000 in https://github.com/mosaicml/composer/pull/1516
* Update tensorflow-io requirement from <0.27,>=0.26.0 to >=0.26.0,<0.28 by dependabot in https://github.com/mosaicml/composer/pull/1522
* Bump cryptography from 37.0.4 to 38.0.1 by dependabot in https://github.com/mosaicml/composer/pull/1521
* Fix SAM loss by mvpatel2000 in https://github.com/mosaicml/composer/pull/1518
* Fixed remote path in streaming dataloader facesynthetics jupyter notebook by karan6181 in https://github.com/mosaicml/composer/pull/1519
* Rework auto grad accum checks by mvpatel2000 in https://github.com/mosaicml/composer/pull/1517
* [xs] remove libcloudhparams from `test_filehelpers.py` by hanlint in https://github.com/mosaicml/composer/pull/1514
* Add v2 datasets behind a version flag by knighton in https://github.com/mosaicml/composer/pull/1507
* Fix compression file remote download exception handling. by knighton in https://github.com/mosaicml/composer/pull/1526

New Contributors
* ananyahjha93 made their first contribution in https://github.com/mosaicml/composer/pull/1493

**Full Changelog**: https://github.com/mosaicml/composer/compare/v0.9.0...v0.10.0

Streaming `v0.9.1` is released! Install via `pip`:


pip install --upgrade mosaicml-streaming==0.9.1


What's New
1. Streaming is added to Gurubase (https://github.com/mosaicml/streaming/pull/805)
* Streaming now has an AI assistant available to help users with their questions! Try out Streaming Guru which uses the data from this repo and data from the [docs](https://docs.mosaicml.com/projects/streaming/en/stable/) to answer questions by leveraging the LLM.

Improvements
1. Permission Issue Resolution (https://github.com/mosaicml/streaming/pull/813)
* Resolved read permission issues occurring when shared memory files are created in shared computing environments. We added retry conditions to allow the creation of new shared memory files upon encountering permission errors.
* Prefix Integrity for Shared Memory Files: When creating shared memory files, both LOCALS and FILELOCKS are now validated to ensure no overlap with existing files, and they are matched with consistent prefix identifiers.
* Handling Non-Normal Program Exits: Enhanced cleanup procedures to address cases where non-normal program exits left some shared memory files uncleared. All files in SHM_TO_CLEAN are now checked to prevent duplicates.
These changes improve shared memory management and reliability in shared environments.

2. Fix Shard Eviction Hanging (https://github.com/mosaicml/streaming/pull/795)
* Changed the search for coldest shard to avoid looping over remote shards by considering local shards only as possible candidates for eviction.




What's Changed
* Bump pydantic from 2.9.1 to 2.9.2 by dependabot in https://github.com/mosaicml/streaming/pull/785
* Bump fastapi from 0.114.2 to 0.115.0 by dependabot in https://github.com/mosaicml/streaming/pull/786
* Bump uvicorn from 0.30.6 to 0.31.0 by dependabot in https://github.com/mosaicml/streaming/pull/793
* Fixed broken links in README.md by LukaszSztukiewicz in https://github.com/mosaicml/streaming/pull/794
* Shard evict fix by snarayan21 in https://github.com/mosaicml/streaming/pull/795
* Update huggingface-hub requirement from <0.25,>=0.23.4 to >=0.23.4,<0.26 by dependabot in https://github.com/mosaicml/streaming/pull/787
* Fix dataset.size() typo in docs by snarayan21 in https://github.com/mosaicml/streaming/pull/798
* Warning -> info about defaults from v0.7.0 by snarayan21 in https://github.com/mosaicml/streaming/pull/799
* Bump uvicorn from 0.31.0 to 0.31.1 by dependabot in https://github.com/mosaicml/streaming/pull/803
* Bump fastapi from 0.115.0 to 0.115.2 by dependabot in https://github.com/mosaicml/streaming/pull/804
* Introducing Streaming Guru on Gurubase.io by kursataktas in https://github.com/mosaicml/streaming/pull/805
* Add better error message for shared prefix by XiaohanZhangCMU in https://github.com/mosaicml/streaming/pull/806
* Bump uvicorn from 0.31.1 to 0.32.0 by dependabot in https://github.com/mosaicml/streaming/pull/809
* Bump pytest-split from 0.9.0 to 0.10.0 by dependabot in https://github.com/mosaicml/streaming/pull/810
* Fix logo png by XiaohanZhangCMU in https://github.com/mosaicml/streaming/pull/808
* Update huggingface-hub requirement from <0.26,>=0.23.4 to >=0.23.4,<0.27 by dependabot in https://github.com/mosaicml/streaming/pull/814
* Bump fastapi from 0.115.2 to 0.115.4 by dependabot in https://github.com/mosaicml/streaming/pull/815
* Fix shared memory permission issue in a shared pod environment by XiaohanZhangCMU in https://github.com/mosaicml/streaming/pull/813

New Contributors
* LukaszSztukiewicz made their first contribution in https://github.com/mosaicml/streaming/pull/794
* kursataktas made their first contribution in https://github.com/mosaicml/streaming/pull/805

**Full Changelog**: https://github.com/mosaicml/streaming/compare/v0.9.0...v0.9.1