Dlio-benchmark

What's Changed
* Add docker image with CPU only dependencies by johnugeorge in https://github.com/argonne-lcf/dlio_benchmark/pull/8
* Add dlio fixes by johnugeorge in https://github.com/argonne-lcf/dlio_benchmark/pull/10
* Fixed issues related to checkpointing and profiling by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/13
* Config parameters fixes by johnugeorge in https://github.com/argonne-lcf/dlio_benchmark/pull/11
* Fixing folder number for evaluation by johnugeorge in https://github.com/argonne-lcf/dlio_benchmark/pull/14
* fixed checkpoint issues by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/16
* Adding PR unit tests for testing different data format and fixing issues for reading png and jpeg with pytorch data folder. by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/17
* A bunch of minor fixes by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/18
* Minor fixes by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/22
* Add ckpting to UNET3D workload, remove old prefetch param by lhovon in https://github.com/argonne-lcf/dlio_benchmark/pull/23
* Minor modification of configuration options to remove some confusion by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/25
* Adding Storage interface for supporting multiple storage backends by johnugeorge in https://github.com/argonne-lcf/dlio_benchmark/pull/20
* Code Fixes by johnugeorge in https://github.com/argonne-lcf/dlio_benchmark/pull/26
* Add the UNET3D sleep time for V100 32GB batch size 4 by lhovon in https://github.com/argonne-lcf/dlio_benchmark/pull/29
* Minor config changes by johnugeorge in https://github.com/argonne-lcf/dlio_benchmark/pull/31
* Make hydra config folder configurable by johnugeorge in https://github.com/argonne-lcf/dlio_benchmark/pull/32
* Mlperf storage v0.5 by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/33
* Changes to support segregation of data loader and reader by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/37
* Added application-level profile support for DLIO by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/39
* Multithreading issue with TensorFlow and PyTorch dataloader by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/44
* bug fix to free memory once file is completely read by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/51
* Pull changes from mlperf_storage_v0.5.1 by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/52
* Improved tracing utility added preprocessing support by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/53
* Trace improvement. by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/48
* Moved resize image to config by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/55
* instead of using direct methods using enter and exit. by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/54
* Reorganizing output files by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/56
* Generator fixed random seed by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/58
* Merging branch mlperf_storage_v0.5.1 by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/57
* fixing mistakes in calculating total number of steps by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/59
* Mlperf storage v0.5.1 by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/60
* Added support for Dali data loader by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/49
* Changed datatype to be np.uint8 universally in the call by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/61
* Adding support for training on a subset of dataset by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/63
* DLIO profiler integration by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/62
* Added Support Power9PC by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/65
* Update unet3d.yaml to correct the sample size for unet3d by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/68
* For X86 and AMD machines, we can create a pip based dlio installations by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/66
* Added validation to check enough core available for reading by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/73
* Added custom plugin code for custom data loader and reader. by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/74
* Changes required within DLIO Benchmark for creating a pip wheel by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/77
* Update bert.yaml to be consistent with mlperf storage by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/79
* Fixing subfolder issues and added subset tests by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/82
* Documentation: Instructions to compile and run on Lassen machine. by OlgaKogiou in https://github.com/argonne-lcf/dlio_benchmark/pull/85
* Changes to improve documentation by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/89
* Fixed dali data loader execution. by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/91
* Enhancing Dali data loader support by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/94
* Fixing Dali Data loader Parallelism and Pipelining. by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/93
* Update typo which gives issue for pytorch 1.3.1 by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/103
* Added documentation for the JPEG generator issue by kaushikvelusamy in https://github.com/argonne-lcf/dlio_benchmark/pull/100
* Workloads by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/97
* Added Info logging for profiler and removed unnecessary bracket calls. by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/104
* Fix the data dir path by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/108
* Making DLIO Profiler default for dlio_benchmark. by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/111
* Adding dlp logger. by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/109
* Workloads by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/112
* fixed readthedoc build issue by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/115
* fix Docker file to use venv. by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/119
* Switch dlio_profiler to use pypi instead of github by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/120
* Added force install for profiler for avoiding caching issues by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/123
* Update README.md by venkat-1 in https://github.com/argonne-lcf/dlio_benchmark/pull/121
* torch checkpoint creation should use storage class methods by krehm in https://github.com/argonne-lcf/dlio_benchmark/pull/126
* Reducing Github actions time by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/128
* Create output_folder using os.makedirs() by krehm in https://github.com/argonne-lcf/dlio_benchmark/pull/124
* Adding Native Dali Data Loader support for TFRecord, Images, and NPZ files by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/118
* Add support for pytorch spawn and forkserver multiprocessing_context by krehm in https://github.com/argonne-lcf/dlio_benchmark/pull/129
* Reopen dlio.log in non-fork reader_threads child processes by krehm in https://github.com/argonne-lcf/dlio_benchmark/pull/130
* added checkpointing to support LLMs by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/114
* added dlp for spawned workers pytorch by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/136
* Fix MPI finalization. by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/139
* Adding dlio_profiler to requirements.txt by johnugeorge in https://github.com/argonne-lcf/dlio_benchmark/pull/144
* Fix dataloader initialization to only happen once. Not on every epoch. by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/143
* Fix random sampling pytorch non-determinism. by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/145
* Fixed printing for DLIO output. by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/142
* Doc changes to fix DLIO profiler and remove IOStat by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/146
* Support for custom checkpointing. by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/137
* Feature/parallel io generator by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/148
* fix random bugs and printing by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/147
* Release for v2.0 by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/113
* Fix requirements file by johnugeorge in https://github.com/argonne-lcf/dlio_benchmark/pull/150
* fixed sample distribution bugs by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/152
* Fix sample shuffling by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/154
* Optimization to sample distribution by TheAssembler1 in https://github.com/argonne-lcf/dlio_benchmark/pull/156
* DALI data loader fix and configuration files update for new batch sizes by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/158
* Fixing github action issues by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/162
* Fixing github action issues (162) by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/163
* Fixed random samples issue and added more github actions to test the configuration files by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/164
* Various bug fixes by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/166
* Fixed global_index issue and redundant shuffling in DALI by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/168
* merge main by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/172
* Adding support to include host cpu and memory info into the json files by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/174
* Changed from PyTorch to Tensorflow for ResNet50 and CosmoFlow by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/183
* Fixing action failure issue by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/184
* Fixed Performance issue in TF Data loader by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/185
* Merge from main by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/186
* Synthetic data support by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/188
* Added doc for synthetic data loader and data reader by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/189
* Packaging by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/190
* Packaging by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/191
* generating indexed_binary files causes kernel OOM to kill process (181) by krehm in https://github.com/argonne-lcf/dlio_benchmark/pull/182
* reduced tensorflow version by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/192
* Improve tfreader parsing performance (batch) by LouisDDN in https://github.com/argonne-lcf/dlio_benchmark/pull/194
* Update config.py by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/196
* Shard filenames instead of images (tfreader) by LouisDDN in https://github.com/argonne-lcf/dlio_benchmark/pull/197
* Request changes from MLPerf Storage by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/199
* Fixed potential insufficient samples due to num_files is not divisible by comm.size by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/200
* Mlperf requests by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/201
* sync up mlperf_storage_v1.0 by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/203
* Fix requirements file by johnugeorge in https://github.com/argonne-lcf/dlio_benchmark/pull/204
* Mlperf storage v1.0 by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/206
* Fixed the MPI initialization issue by zhenghh04 in https://github.com/argonne-lcf/dlio_benchmark/pull/207
* Switch DLIO Profiler to DFTracer. by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/208
* Fix README CI badge by izzet in https://github.com/argonne-lcf/dlio_benchmark/pull/212
* Adding version fix restricts matching on python 3.9 environment. by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/218
* Only intialize and finalize on DLIOMPI by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/214
* Ignore file indexing for native data loader. by hariharan-devarajan in https://github.com/argonne-lcf/dlio_benchmark/pull/215

New Contributors
* johnugeorge made their first contribution in https://github.com/argonne-lcf/dlio_benchmark/pull/8
* hariharan-devarajan made their first contribution in https://github.com/argonne-lcf/dlio_benchmark/pull/37
* OlgaKogiou made their first contribution in https://github.com/argonne-lcf/dlio_benchmark/pull/85
* kaushikvelusamy made their first contribution in https://github.com/argonne-lcf/dlio_benchmark/pull/100
* venkat-1 made their first contribution in https://github.com/argonne-lcf/dlio_benchmark/pull/121
* krehm made their first contribution in https://github.com/argonne-lcf/dlio_benchmark/pull/126
* TheAssembler1 made their first contribution in https://github.com/argonne-lcf/dlio_benchmark/pull/156
* LouisDDN made their first contribution in https://github.com/argonne-lcf/dlio_benchmark/pull/194

**Full Changelog**: https://github.com/argonne-lcf/dlio_benchmark/compare/v1.0.0...v2.0.0

In this new release, we have the following changes and new enhancements
1. Added support for S3 storage
2. Updated config files for MLPerf Storage workloads: UNet3D and Bert.
3. Changes on configuration options:
- added variability support for sample size, training and validation computation time.
- changes on shuffling, prefetching setting.
- moved batch_size, batch_size_eval to reader session

This release is correspondence to MLPerf storage v0.5 prerelease: https://github.com/mlcommons/storage/releases/tag/v0.5-rc0

DLIO v1.0 Release Notes
We are excited to announce the release of DLIO 1.0! There are many new features and new enhancements compared to previous 0.0.1 version:
* Using YAML file to configure DLIO in Hydra.cc framework; The configuration options are organized in a hierarchical way, including model, framework, workflow, dataset, train, evaluation, checkpoint, profiling. a set of YAML files for some workloads are included.
* Data loader support enhancement:
- Added data loader layer above data format to allow user to choose data loader and data format independently.
- Added PyTorch data loader support. We have full PyTorch data loader support for one sample per file dataset
- Enhanced TensorFlow tf.data loader to support for generic file format beyond tfrecord format (currently only support one sample per file case for generic data format)
* New dataset support
- Added support for png and jpeg formats
- Supporting multiple subfolders for training and validation datasets.
- Supporting generating validation dataset
* Profiling and logging
- Added support for iostat profiling
- Added detailed logging info
* Added support for validation.
* Added post processing python script
* Added unit tests and GitHub Actions tests.
* User and developer documentation in github.io: https://argonne-lcf.github.io/dlio_benchmark