Composer

What's New

**1. Add passk for HumanEval**

HumanEval now supports passk. We also support first-class integration with the MosaicML platform for secure code evaluation.

**2. `log_model` with MLFlow**

The MLFlow integration now supports `log_model` at the end of the run.

What's Changed
* Update checkpoint.py by b-chu in https://github.com/mosaicml/composer/pull/2540
* Add log image to mlflow by eracah in https://github.com/mosaicml/composer/pull/2416
* Log runtime estimator units by mvpatel2000 in https://github.com/mosaicml/composer/pull/2542
* Bump traitlets from 5.9.0 to 5.10.0 by dependabot in https://github.com/mosaicml/composer/pull/2547
* Bump gitpython from 3.1.35 to 3.1.36 by dependabot in https://github.com/mosaicml/composer/pull/2546
* Bump ipykernel from 6.25.1 to 6.25.2 by dependabot in https://github.com/mosaicml/composer/pull/2544
* Add providers param to ONNX Session in tests by nik-mosaic in https://github.com/mosaicml/composer/pull/2553
* Bump flash attn by mvpatel2000 in https://github.com/mosaicml/composer/pull/2551
* Remove pin by mvpatel2000 in https://github.com/mosaicml/composer/pull/2554
* Change filter to include pull_request_target by mvpatel2000 in https://github.com/mosaicml/composer/pull/2557
* Downgrade nightly to previous version by mvpatel2000 in https://github.com/mosaicml/composer/pull/2556
* MCLI Code Eval by rishab-partha in https://github.com/mosaicml/composer/pull/2479
* Bump cryptography from 41.0.3 to 41.0.4 by dependabot in https://github.com/mosaicml/composer/pull/2559
* Bump gitpython from 3.1.36 to 3.1.37 by dependabot in https://github.com/mosaicml/composer/pull/2560
* Update numpy requirement from <1.26.0,>=1.21.5 to >=1.21.5,<1.27.0 by dependabot in https://github.com/mosaicml/composer/pull/2561
* Update support for HumanEval by mcarbin in https://github.com/mosaicml/composer/pull/2550
* Add log_model to MLFlowLogger by dakinggg in https://github.com/mosaicml/composer/pull/2541
* Bump version to 0.16.3 by mvpatel2000 in https://github.com/mosaicml/composer/pull/2566

New Contributors
* mcarbin made their first contribution in https://github.com/mosaicml/composer/pull/2550

**Full Changelog**: https://github.com/mosaicml/composer/compare/v0.16.2...v0.16.3

What's New

**1. PyTorch Nightly Support**

Composer now supports PyTorch Nightly and Cuda 12! Along with new docker images based on nightly PyTorch versions and release candidates, we've updated our PyTorch monkeypatches to support the latest version of PyTorch. These monkeypatches add additional functionality in finer-grain FSDP wrapping and patch bugs related to sharded checkpoints. We are in the process of upstreaming these changes into PyTorch.

Bug Fixes

**1. MosaicML Logger Robustness**

MosaicML logger now is robust to platform timeouts and other errors. Additionally, it can now be disabled by setting the environment variable `MOSAICML_PLATFORM` to `'False'` when training on the MosaicML platform.

**2. GCS Integration**

GCS authentication is now supported with HMAC keys, patching a bug in the previous implementation.

**3. Optimizer Monitor Norm Calculation (https://github.com/mosaicml/composer/pull/2531)**

Previously, the optimizer monitor incorrectly reduced norms across GPUs. It now correctly computes norms in a distributed setting.

What's Changed
* fix: when there is no train_metrics, do not checkpoint by furkanbiten in https://github.com/mosaicml/composer/pull/2502
* Remove metric saving by mvpatel2000 in https://github.com/mosaicml/composer/pull/2514
* Fix daily tests by removing gpu marker by j316chuck in https://github.com/mosaicml/composer/pull/2515
* Refactor mosaic_fsdp.py by b-chu in https://github.com/mosaicml/composer/pull/2506
* Disable slack notifications for PRs by mvpatel2000 in https://github.com/mosaicml/composer/pull/2517
* Add custom sharding to ChunkShardingSpec by b-chu in https://github.com/mosaicml/composer/pull/2507
* Update nightly docker image to torch nightly 09-03-23 by j316chuck in https://github.com/mosaicml/composer/pull/2518
* Update pre-commit in setup.py by b-chu in https://github.com/mosaicml/composer/pull/2522
* Add FSDP custom wrap with torch 2.1 by mvpatel2000 in https://github.com/mosaicml/composer/pull/2460
* Fix GCSObjectStore bug where hmac keys auth doesn't work by eracah in https://github.com/mosaicml/composer/pull/2519
* Bump gitpython from 3.1.34 to 3.1.35 by dependabot in https://github.com/mosaicml/composer/pull/2525
* Bump pytest from 7.4.0 to 7.4.2 by dependabot in https://github.com/mosaicml/composer/pull/2523
* Upgrade to MLFlow version 2.5.0 by ngcgarcia in https://github.com/mosaicml/composer/pull/2528
* Disable cifar daily test by mvpatel2000 in https://github.com/mosaicml/composer/pull/2527
* Mosaicml logger robustness improvements by mvpatel2000 in https://github.com/mosaicml/composer/pull/2530
* Fix metrics keys sort in DecoupledAdamW for OptimizerMonitor FSDP metric agreggation by m1kol in https://github.com/mosaicml/composer/pull/2531
* Fix github actions for GCS integration testing by mvpatel2000 in https://github.com/mosaicml/composer/pull/2532
* Fix GCS tests by mvpatel2000 in https://github.com/mosaicml/composer/pull/2535
* Change cast for mosaicml logger by mvpatel2000 in https://github.com/mosaicml/composer/pull/2538
* Bump Version to 0.16.2 by mvpatel2000 in https://github.com/mosaicml/composer/pull/2537
* Bump transformers version by dakinggg in https://github.com/mosaicml/composer/pull/2539

New Contributors
* ngcgarcia made their first contribution in https://github.com/mosaicml/composer/pull/2528
* m1kol made their first contribution in https://github.com/mosaicml/composer/pull/2531

**Full Changelog**: https://github.com/mosaicml/composer/compare/v0.16.1...v0.16.2

New Features

**1. HPU (Habana Gaudi) Support (https://github.com/mosaicml/composer/pull/2444)**

Composer now supports Habana Gaudi chips! To enable HPUs, `device` needs to be specified as `'hpu'`:


composer_model = MyComposerModel(n_layers=3)

trainer = Trainer(
model=composer_model,
device='hpu',
...
)


**2. Generate Callback (https://github.com/mosaicml/composer/pull/2449)**

We've added a new callback which runs generate on a language model at a given frequency to visualize outputs:


from composer.callbacks import Generate

composer_model = MyComposerModel(n_layers=3)
generate_callback = Generate(prompts=['How good is my model?'], interval='5ba')

trainer = Trainer(
model=composer_model,
callbacks = generate_callback,
...
)


Bug Fixes

**1. Checkpoint Fixes**

Elastic sharded checkpointing now disables torchmetric saving to avoid issues with torchmetrics tensors being sharded. Additionally, checkpointing now falls back on the old path which does not convert torchmetrics tensors to `numpy`. Checkpointing also no longer materializes optimizer state when saving weights only.

**2. MLFlow Performance Improvements**

MLFlow integration has significant performance improvements in logging frequency and system metrics collected.


What's Changed
* Hpu support by vivekgoe in https://github.com/mosaicml/composer/pull/2444
* Change `input_ids` to a kwarg in `HuggingFaceModel.generate` by dakinggg in https://github.com/mosaicml/composer/pull/2459
* Add log_table by irenedea in https://github.com/mosaicml/composer/pull/2437
* Enable composer to work with torch nightly builds, torch 2.1.0, and cuda 12.1. by j316chuck in https://github.com/mosaicml/composer/pull/2463
* Materialize only model state_dict in memory for `save_weights_only` by eracah in https://github.com/mosaicml/composer/pull/2450
* Improve performance of MLflow logging by dbczumar in https://github.com/mosaicml/composer/pull/2442
* Fail fast if scheduler warmup and max duration are incompatible by dakinggg in https://github.com/mosaicml/composer/pull/2458
* Add nightly docker image by j316chuck in https://github.com/mosaicml/composer/pull/2452
* Fix local eval by rishab-partha in https://github.com/mosaicml/composer/pull/2465
* Add torch 2.1.0 args for github release-docker workflow by j316chuck in https://github.com/mosaicml/composer/pull/2470
* Log system metrics on each event by prithvikannan in https://github.com/mosaicml/composer/pull/2412
* Fix torch 2.1.0 docker tag by j316chuck in https://github.com/mosaicml/composer/pull/2472
* Upstream Generate Callback by irenedea in https://github.com/mosaicml/composer/pull/2449
* Bump torch nightly docker image by j316chuck in https://github.com/mosaicml/composer/pull/2476
* Test pytorch 2.1.0 docker images on ci/cd by j316chuck in https://github.com/mosaicml/composer/pull/2469
* Fix huggingface tokenizer loading for slow tokenizers by dakinggg in https://github.com/mosaicml/composer/pull/2483
* Deprecate Fused LayerNorm by nik-mosaic in https://github.com/mosaicml/composer/pull/2475
* Transformers upgrade by dakinggg in https://github.com/mosaicml/composer/pull/2489
* Update RTD build config with build.os by bandish-shah in https://github.com/mosaicml/composer/pull/2490
* Upgrade torch docker version and tests by j316chuck in https://github.com/mosaicml/composer/pull/2488
* upgrade node by j316chuck in https://github.com/mosaicml/composer/pull/2492
* Gating tying modules w/ FSDP for torch 2.0 by bcui19 in https://github.com/mosaicml/composer/pull/2467
* Removing min_params by bcui19 in https://github.com/mosaicml/composer/pull/2494
* Fix torchmetrics backwards compatibility issue by eracah in https://github.com/mosaicml/composer/pull/2468
* Adding some fixes to FSDP tests by bcui19 in https://github.com/mosaicml/composer/pull/2495
* Fail count on mosaicml logger by mvpatel2000 in https://github.com/mosaicml/composer/pull/2496
* Remove PR curve metrics from backward compatibility test and skip torch 1.13 by eracah in https://github.com/mosaicml/composer/pull/2497
* filter warning by mvpatel2000 in https://github.com/mosaicml/composer/pull/2500
* Bump version to 0.16.1 by mvpatel2000 in https://github.com/mosaicml/composer/pull/2498
* Skip metrics in state dict by mvpatel2000 in https://github.com/mosaicml/composer/pull/2501
* Add peak memory stats by mvpatel2000 in https://github.com/mosaicml/composer/pull/2504
* Fix sharded ckpt by mvpatel2000 in https://github.com/mosaicml/composer/pull/2505
* Bump gitpython from 3.1.31 to 3.1.34 by dependabot in https://github.com/mosaicml/composer/pull/2509
* Annotate `torch_prof_remote_file_name` as Optional by srstevenson in https://github.com/mosaicml/composer/pull/2512

New Contributors
* vivekgoe made their first contribution in https://github.com/mosaicml/composer/pull/2444
* irenedea made their first contribution in https://github.com/mosaicml/composer/pull/2437
* j316chuck made their first contribution in https://github.com/mosaicml/composer/pull/2463
* dbczumar made their first contribution in https://github.com/mosaicml/composer/pull/2442

**Full Changelog**: https://github.com/mosaicml/composer/compare/v0.16.0...v0.16.1

What's New

**1. New Events (2264)**

Composer now has the events `EVAL_BEFORE_ALL` and `EVAL_AFTER_ALL`, which lets users control logging of certain bespoke evaluation information across all evalutors.

**2. Elastic Sharded Checkpointing**

Traditionally, checkpoints are stored as giant monoliths. For large model training, moving the entire model to 1 node may be infeasible and writing one large file from 1 node may be slow. Composer now supports elastic sharded checkpoints with FSDP, where every rank writes a single shard of the checkpoint. This checkpointing strategy is elastic, which means even if you resume on a different number of GPUs, Composer will handle resumption. To enable sharded checkpointing, it must be specified in the FSDP Config as `'state_dict_type': 'sharded'`:


composer_model = MyComposerModel(n_layers=3)

fsdp_config = {
'sharding_strategy': 'FULL_SHARD',
'state_dict_type': 'sharded',
'sharded_ckpt_prefix_dir': 'ba{batch}-shards' will save each set of shards checkpoint to a unique folder based on batch
}

trainer = Trainer(
model=composer_model,
max_duration='4ba'
fsdp_config=fsdp_config,
save_folder='checkpoints',
save_interval='2ba',
...
)


See the [docs](https://docs.mosaicml.com/projects/composer/en/latest/notes/distributed_training.html#saving-and-loading-sharded-checkpoints-with-fsdp) for more information in how to integrate this with your project.

Bug Fixes
* Fixes runtime estimator when using multiple evaluators in https://github.com/mosaicml/composer/pull/2331
* Fix autoresume docs link in https://github.com/mosaicml/composer/pull/2332
* Use Enum value when logging hyper-parameters in https://github.com/mosaicml/composer/pull/2386
* Fix GCSObjectStore to match function signatures of other object stores in https://github.com/mosaicml/composer/pull/2445
* Cast to float32 before numpy() to avoid bf16 errors in https://github.com/mosaicml/composer/pull/2441

What's Changed
* Update numpy requirement from <1.25.0,>=1.21.5 to >=1.21.5,<1.26.0 by dependabot in https://github.com/mosaicml/composer/pull/2316
* Bump ipykernel from 6.23.1 to 6.23.2 by dependabot in https://github.com/mosaicml/composer/pull/2317
* Bump sphinxcontrib-katex from 0.9.5 to 0.9.6 by dependabot in https://github.com/mosaicml/composer/pull/2319
* Pin Apex by mvpatel2000 in https://github.com/mosaicml/composer/pull/2322
* CodeQL on PRs by mvpatel2000 in https://github.com/mosaicml/composer/pull/2323
* Add secrets check as part of pre-commit by karan6181 in https://github.com/mosaicml/composer/pull/2324
* Update local rank 0 to be elastic by mvpatel2000 in https://github.com/mosaicml/composer/pull/2321
* Bump pytest from 7.3.1 to 7.4.0 by dependabot in https://github.com/mosaicml/composer/pull/2330
* Bump ipykernel from 6.23.2 to 6.23.3 by dependabot in https://github.com/mosaicml/composer/pull/2329
* Auto add mosaicml logger by mvpatel2000 in https://github.com/mosaicml/composer/pull/2325
* Add precision config arg for FP8 by julian-q in https://github.com/mosaicml/composer/pull/2335
* Fixes daily test failures with respect to autoadd mosaicml logger by mvpatel2000 in https://github.com/mosaicml/composer/pull/2339
* In-line group to avoid OOM by mvpatel2000 in https://github.com/mosaicml/composer/pull/2320
* Set offload_to_cpu True for state_dict_type=sharded by eracah in https://github.com/mosaicml/composer/pull/2338
* Update version to 15.1 by mvpatel2000 in https://github.com/mosaicml/composer/pull/2341
* Fix mapi mocking by mvpatel2000 in https://github.com/mosaicml/composer/pull/2342
* Change gpu timeout by rishab-partha in https://github.com/mosaicml/composer/pull/2343
* Fix test_fsdp_load_old_checkpoint test to fix daily tests by eracah in https://github.com/mosaicml/composer/pull/2347
* Add spaces between sentences in eval label warning by srstevenson in https://github.com/mosaicml/composer/pull/2327
* Avoid overwriting seed==0 by tbenthompson in https://github.com/mosaicml/composer/pull/2352
* Small Documentation Typo Fixes by sarthak-314 in https://github.com/mosaicml/composer/pull/2349
* Fix wandb errror with autoresume issue by eracah in https://github.com/mosaicml/composer/pull/2353
* Bump ipykernel from 6.23.3 to 6.24.0 by dependabot in https://github.com/mosaicml/composer/pull/2360
* raise min mcli by mvpatel2000 in https://github.com/mosaicml/composer/pull/2362
* Add node rank to signal files by mvpatel2000 in https://github.com/mosaicml/composer/pull/2363
* Move pydantic pin to deepspeed by mvpatel2000 in https://github.com/mosaicml/composer/pull/2366
* Batch log metrics calls in speed_monitor.py by prithvikannan in https://github.com/mosaicml/composer/pull/2367
* Read Composer run name env var by mvpatel2000 in https://github.com/mosaicml/composer/pull/2372
* Fix typing for args in streaming by dakinggg in https://github.com/mosaicml/composer/pull/2373
* Add distributed sync during wait_for_workers to avoid timeout for large checkpoints by dakinggg in https://github.com/mosaicml/composer/pull/2368
* Update torchmetrics requirement from <0.12,>=0.10.0 to >=0.10.0,<1.1 by dependabot in https://github.com/mosaicml/composer/pull/2358
* Add code eval dataset and metric by rishab-partha in https://github.com/mosaicml/composer/pull/2301
* Isolate env var in unit tests by mvpatel2000 in https://github.com/mosaicml/composer/pull/2379
* Add extra steps for space free up by XiaohanZhangCMU in https://github.com/mosaicml/composer/pull/2382
* regex changed in time.py by megha95 in https://github.com/mosaicml/composer/pull/2378
* Support no param models by making optimizer optional by mvpatel2000 in https://github.com/mosaicml/composer/pull/2374
* pin identify version to resolve codequality failures by XiaohanZhangCMU in https://github.com/mosaicml/composer/pull/2391
* Add ls to object stores by dakinggg in https://github.com/mosaicml/composer/pull/2376
* Change transformers by rishab-partha in https://github.com/mosaicml/composer/pull/2383
* Respect MLFLow experiment environment variable by aspfohl in https://github.com/mosaicml/composer/pull/2377
* Change code eval apikey by rishab-partha in https://github.com/mosaicml/composer/pull/2394
* Moves pytest-cpu slack notifications to issues from helpdesk by mvpatel2000 in https://github.com/mosaicml/composer/pull/2398
* Add code eval docs by rishab-partha in https://github.com/mosaicml/composer/pull/2397
* fixed pre-commit issues with modifications to pretty-format-json args. by snarayan21 in https://github.com/mosaicml/composer/pull/2392
* Fix LOCAL_WORLD_SIZE in pytest by rishab-partha in https://github.com/mosaicml/composer/pull/2407
* Add code eval secrets to workflows by rishab-partha in https://github.com/mosaicml/composer/pull/2399
* Enable Elastic Sharded Checkpointing by eracah in https://github.com/mosaicml/composer/pull/2262
* Remove compute_on_step from MAP by priba in https://github.com/mosaicml/composer/pull/2390
* Save metadata and integration when save_weights_only is set by eracah in https://github.com/mosaicml/composer/pull/2396
* remove unused Trainer docstring arg load_fsdp_monolith_rank0_only by eracah in https://github.com/mosaicml/composer/pull/2408
* torch2.0.1 custom auto wrap by vchiley in https://github.com/mosaicml/composer/pull/2400
* Add ruff pre-commit by Skylion007 in https://github.com/mosaicml/composer/pull/2414
* Switch google cloud backend from libcloud to google cloud storage API by XiaohanZhangCMU in https://github.com/mosaicml/composer/pull/2340
* Updates GPU test timeout to use mcloud flag by mvpatel2000 in https://github.com/mosaicml/composer/pull/2420
* Add a `EVAL_STANDALONE_START` and `EVAL_STANDALONE_END` events and change RUD to not `wait_for_workers` every eval by dakinggg in https://github.com/mosaicml/composer/pull/2418
* Throttle optimizer monitor by mvpatel2000 in https://github.com/mosaicml/composer/pull/2419
* Adding extra condition to avoid running eval_train_metrics by furkanbiten in https://github.com/mosaicml/composer/pull/2411
* fp8 on Ada by dskhudia in https://github.com/mosaicml/composer/pull/2424
* Bump coverage[toml] from 7.2.7 to 7.3.0 by dependabot in https://github.com/mosaicml/composer/pull/2432
* Bump cryptography from 38.0.4 to 41.0.3 by dependabot in https://github.com/mosaicml/composer/pull/2436
* Bump ipykernel from 6.24.0 to 6.25.1 by dependabot in https://github.com/mosaicml/composer/pull/2434
* Multilingual compatibility and batching for Code Evaluation by rishab-partha in https://github.com/mosaicml/composer/pull/2410
* Update max duration on tests by mvpatel2000 in https://github.com/mosaicml/composer/pull/2429
* Update timeout by rishab-partha in https://github.com/mosaicml/composer/pull/2438
* add dist.barrier to rotate_checkpoints by eracah in https://github.com/mosaicml/composer/pull/2440
* Bump version to 0.16 by mvpatel2000 in https://github.com/mosaicml/composer/pull/2439
* Fix notebooks by rishab-partha in https://github.com/mosaicml/composer/pull/2446
* Fix notebooks v2 by rishab-partha in https://github.com/mosaicml/composer/pull/2448

New Contributors
* eltociear made their first contribution in https://github.com/mosaicml/composer/pull/2333
* antoinebrl made their first contribution in https://github.com/mosaicml/composer/pull/2334
* julian-q made their first contribution in https://github.com/mosaicml/composer/pull/2335
* srstevenson made their first contribution in https://github.com/mosaicml/composer/pull/2327
* tbenthompson made their first contribution in https://github.com/mosaicml/composer/pull/2352
* sarthak-314 made their first contribution in https://github.com/mosaicml/composer/pull/2349
* prithvikannan made their first contribution in https://github.com/mosaicml/composer/pull/2367
* XiaohanZhangCMU made their first contribution in https://github.com/mosaicml/composer/pull/2382
* megha95 made their first contribution in https://github.com/mosaicml/composer/pull/2378
* snarayan21 made their first contribution in https://github.com/mosaicml/composer/pull/2392
* priba made their first contribution in https://github.com/mosaicml/composer/pull/2390
* Skylion007 made their first contribution in https://github.com/mosaicml/composer/pull/2414
* furkanbiten made their first contribution in https://github.com/mosaicml/composer/pull/2411

**Full Changelog**: https://github.com/mosaicml/composer/compare/v0.15.0...v0.16.0

Bug Fixes

This is a patch release that mainly fixes a bug related to autoresume, and changes the default to `offload_to_cpu` for PyTorch version >2 sharded checkpoints.

What's Changed

* Fixes daily test failures with respect to autoadd mosaicml logger by mvpatel2000 in https://github.com/mosaicml/composer/pull/2339
* Set offload_to_cpu True for state_dict_type=sharded by eracah in https://github.com/mosaicml/composer/pull/2338
* Update version by mvpatel2000 in https://github.com/mosaicml/composer/pull/2341
* Fix MAPI mocking by mvpatel2000 in https://github.com/mosaicml/composer/pull/2342
* Change GPU timeout by rishab-partha in https://github.com/mosaicml/composer/pull/2343
* Add cpu call by eracah in https://github.com/mosaicml/composer/pull/2347
* Add spaces between sentences in eval label warning by srstevenson in https://github.com/mosaicml/composer/pull/2327
* Avoid overwriting seed=0 by tbenthompson in https://github.com/mosaicml/composer/pull/2352
* Small documentation typo fixes by sarthak-314 in https://github.com/mosaicml/composer/pull/2349
* Fix wandb errror with autoresume issue by eracah in https://github.com/mosaicml/composer/pull/2353


**Full Changelog**: https://github.com/mosaicml/composer/compare/v0.15.0...v0.15.1

What's New

1. **Exact Eval (https://github.com/mosaicml/composer/pull/2218)**

Composer now supports exact evaluation! Now, evaluation will give the exact same results regardless of the number of GPUs by removing any duplicated samples from the dataloader.

2. **Monolithic Checkpoint Loading (https://github.com/mosaicml/composer/pull/2288)**

When training large models, loading the model and optimizer on every rank can use up all the system memory. With FSDP, Composer can now load the model and optimizer on only rank 0 and broadcast it to all other ranks. To enable:

python
from composer import Trainer

Construct Trainer
trainer = Trainer(
...,
fsdp_config={
load_monolith_rank0_only: True
},
)

Train!
trainer.fit()


and ensure the model on rank 0 is on CPU/GPU (as opposed to meta).

3. **Spin Dataloaders**

By default, Composer spins dataloaders back to the current timestamp to ensure deterministic resumption. However, dataloader spinning can be very slow, so `Trainer` now has a new flag to disable spinning if determinism is not required. To enable:

python
from composer import Trainer

Construct Trainer
trainer = Trainer(
...,
spin_dataloaders=False,
)

Train!
trainer.fit()


Deprecations

* `HealthChecker` is now deprecated and will be removed in `v0.17.0`

Bug Fixes

* Add support for saving HF info in state dict when using DDP by dakinggg in https://github.com/mosaicml/composer/pull/2206
* Change state dict loading default to strict by dakinggg in https://github.com/mosaicml/composer/pull/2216
* CE loss vs CE metric equivalence by dakinggg in https://github.com/mosaicml/composer/pull/2241
* Move sharded checkpoints into their own intermediate prefix folder by eracah in https://github.com/mosaicml/composer/pull/2205
* Fix typo depricated -> deprecated by eracah in https://github.com/mosaicml/composer/pull/2270
* Spin dataloader arg by mvpatel2000 in https://github.com/mosaicml/composer/pull/2267
* Confirming the output variable has two dimensions before confirming the shape of the second element. by jimmiemunyi in https://github.com/mosaicml/composer/pull/2275
* Add loss_dict keyword to closure lambda function by Landanjs in https://github.com/mosaicml/composer/pull/1952
* Strip spacing icl by bmosaicml in https://github.com/mosaicml/composer/pull/2306

What's Changed
* Update FFCV by mvpatel2000 in https://github.com/mosaicml/composer/pull/2197
* Add support for saving HF info in state dict when using DDP by dakinggg in https://github.com/mosaicml/composer/pull/2206
* Bump junitparser from 3.0.0 to 3.1.0 by dependabot in https://github.com/mosaicml/composer/pull/2212
* Bump sentencepiece from 0.1.98 to 0.1.99 by dependabot in https://github.com/mosaicml/composer/pull/2208
* Add docs for Checkpointing with Cloudflare R2 by eracah in https://github.com/mosaicml/composer/pull/2215
* Working slack link by growlix in https://github.com/mosaicml/composer/pull/2217
* Change state dict loading default to strict by dakinggg in https://github.com/mosaicml/composer/pull/2216
* Fix typo in evaluation docs by dakinggg in https://github.com/mosaicml/composer/pull/2225
* Clean soft cross entropy by mvpatel2000 in https://github.com/mosaicml/composer/pull/2227
* add cmake by dakinggg in https://github.com/mosaicml/composer/pull/2229
* Upgrade to mcli0.4, smaller mcli improvements by aspfohl in https://github.com/mosaicml/composer/pull/2226
* Bump to torch 2.0.1 by mvpatel2000 in https://github.com/mosaicml/composer/pull/2235
* Deprecate healthchecker by mvpatel2000 in https://github.com/mosaicml/composer/pull/2236
* Update torch 2.0.1 workflows by mvpatel2000 in https://github.com/mosaicml/composer/pull/2239
* Log wandb URL to metadata by mvpatel2000 in https://github.com/mosaicml/composer/pull/2240
* Bump ipykernel from 6.22.0 to 6.23.1 by dependabot in https://github.com/mosaicml/composer/pull/2244
* Update transformers requirement from <4.29,>=4.11 to >=4.11,<4.30 by dependabot in https://github.com/mosaicml/composer/pull/2245
* CE loss vs CE metric equivalence by dakinggg in https://github.com/mosaicml/composer/pull/2241
* Exact Eval by mvpatel2000 in https://github.com/mosaicml/composer/pull/2218
* bump torchmetrics pin by nik-mosaic in https://github.com/mosaicml/composer/pull/2247
* Remove deprecated code / torch 1.11 / torch 1.12 by mvpatel2000 in https://github.com/mosaicml/composer/pull/2234
* Rename `backwards_create_graph` description by mvpatel2000 in https://github.com/mosaicml/composer/pull/2248
* Move sharded checkpoints into their own intermediate prefix folder by eracah in https://github.com/mosaicml/composer/pull/2205
* Fix daily tests by fixing test_fsdp_load_old_checkpoint by eracah in https://github.com/mosaicml/composer/pull/2249
* Support for multiple optimizer groups in torch 2.0 + FSDP by sashaDoubov in https://github.com/mosaicml/composer/pull/2230
* Change AdamW step to a tensor instead of an int by eracah in https://github.com/mosaicml/composer/pull/2237
* Update to cuda 11.8 by mvpatel2000 in https://github.com/mosaicml/composer/pull/2250
* Fix daily tests by adding s3 secrets to daily-gpu tests by eracah in https://github.com/mosaicml/composer/pull/2254
* Typo in s3_prefix: epemeral -> ephemeral 🤦‍♂️ by eracah in https://github.com/mosaicml/composer/pull/2255
* Bump yamllint from 1.31.0 to 1.32.0 by dependabot in https://github.com/mosaicml/composer/pull/2256
* Bump coverage[toml] from 7.2.5 to 7.2.6 by dependabot in https://github.com/mosaicml/composer/pull/2258
* Add callbacks for EVAL_BEFORE_ALL and EVAL_AFTER_ALL by rishab-partha in https://github.com/mosaicml/composer/pull/2264
* Update torch device naming convention for h100 gpus by vchiley in https://github.com/mosaicml/composer/pull/2265
* Fix typo depricated -> deprecated by eracah in https://github.com/mosaicml/composer/pull/2270
* alerts for daily tests by mvpatel2000 in https://github.com/mosaicml/composer/pull/2272
* Fix daily tests by patching cupy version by mvpatel2000 in https://github.com/mosaicml/composer/pull/2274
* Skip ffcv notebook by mvpatel2000 in https://github.com/mosaicml/composer/pull/2277
* Spin dataloader arg by mvpatel2000 in https://github.com/mosaicml/composer/pull/2267
* Confirming the output variable has two dimensions before confirming the shape of the second element. by jimmiemunyi in https://github.com/mosaicml/composer/pull/2275
* Bump coverage[toml] from 7.2.6 to 7.2.7 by dependabot in https://github.com/mosaicml/composer/pull/2282
* Patch for tokenizers that have python files in save_pretrained output by dakinggg in https://github.com/mosaicml/composer/pull/2279
* fix get file(overwite=True) to properly handle pre-existing files by bmosaicml in https://github.com/mosaicml/composer/pull/2284
* Fix Checkpointing Docs Link by rishab-partha in https://github.com/mosaicml/composer/pull/2278
* Add errors for Mixed Dataloader Eval by rishab-partha in https://github.com/mosaicml/composer/pull/2269
* Fix autoresume with slashed directory by rishab-partha in https://github.com/mosaicml/composer/pull/2287
* Delete symlinks when not saving checkpoints locally by rishab-partha in https://github.com/mosaicml/composer/pull/2285
* fixed adding tokenizer to hf by KuuCi in https://github.com/mosaicml/composer/pull/2290
* New Console Logger Test + Discard before Eval by rishab-partha in https://github.com/mosaicml/composer/pull/2273
* Enabled kv caching during generate to speed up QA Task by bmosaicml in https://github.com/mosaicml/composer/pull/2293
* Update monai requirement from <1.2,>=0.9.1 to >=0.9.1,<1.3 by dependabot in https://github.com/mosaicml/composer/pull/2298
* Bump sphinxcontrib-katex from 0.9.4 to 0.9.5 by dependabot in https://github.com/mosaicml/composer/pull/2296
* Training Checkpoint Fix by KuuCi in https://github.com/mosaicml/composer/pull/2294
* Update transformers requirement from <4.30,>=4.11 to >=4.11,<4.31 by dependabot in https://github.com/mosaicml/composer/pull/2295
* Fixed how save_checkpoint_to_save_folder called CheckpointSaver object to save state and logger by KuuCi in https://github.com/mosaicml/composer/pull/2300
* Update Slack link in README.md by ejyuen in https://github.com/mosaicml/composer/pull/2261
* Change progress bar logger to print all eval metrics by rishab-partha in https://github.com/mosaicml/composer/pull/2286
* Add pytest clear cache by rishab-partha in https://github.com/mosaicml/composer/pull/2305
* Fix tests for wandb and mlflow loggers by b-chu in https://github.com/mosaicml/composer/pull/2302
* Monolithic Loading by mvpatel2000 in https://github.com/mosaicml/composer/pull/2288
* Add loss_dict keyword to closure lambda function by Landanjs in https://github.com/mosaicml/composer/pull/1952
* Strip spacing icl by bmosaicml in https://github.com/mosaicml/composer/pull/2306
* Add additional error with auto microbatching by mvpatel2000 in https://github.com/mosaicml/composer/pull/2308
* Group autoresume messages by mvpatel2000 in https://github.com/mosaicml/composer/pull/2307
* Move deepspeed enabled to state by mvpatel2000 in https://github.com/mosaicml/composer/pull/2309
* Jiggling tests and adding gc collect by bcui19 in https://github.com/mosaicml/composer/pull/2312
* Monolithic loading improvements by mvpatel2000 in https://github.com/mosaicml/composer/pull/2313
* Update version to 0.15 by mvpatel2000 in https://github.com/mosaicml/composer/pull/2315

New Contributors
* aspfohl made their first contribution in https://github.com/mosaicml/composer/pull/2226
* sashaDoubov made their first contribution in https://github.com/mosaicml/composer/pull/2230
* rishab-partha made their first contribution in https://github.com/mosaicml/composer/pull/2264
* jimmiemunyi made their first contribution in https://github.com/mosaicml/composer/pull/2275
* KuuCi made their first contribution in https://github.com/mosaicml/composer/pull/2290
* b-chu made their first contribution in https://github.com/mosaicml/composer/pull/2302

**Full Changelog**: https://github.com/mosaicml/composer/compare/v0.14.1...v0.15.0