Fms-hf-tuning

Mamba Model Support
- Add mamba and cuda dependencies to support tuning Mamba and Jamba models.

Configuration Updates
- New `add_special_tokens` argument for special tokens to be added to the tokenizer's vocabulary.

Data Preprocessor Updates
- Enable `streaming` in `DataSetConfig` to load large datasets by utilizing HF `IterableDatasets`.
- Add support for `chat_template` in `DataPreProcessorConfig`. Fix parsing issue for passing `chat_template` via CLI args.
- Allow a tokenizer data handler for users to have direct control over dataset and truncating samples.
- Fix multi-processes broadcast error when running data preprocessor.
- Update data handler backend to support additional types of handlers beyond Map style. Adds support for Filter type data handlers.
- Fix data handler to string conversion bug.
- Allow users to use `tools` and `documents` in chat templates and pass in additional data arguments to the data handler to enable processing sub field of a data sample.
- Add data preprocessing script, decoupling data preprocessor from the tuning. Users can execute the script [`offline_data_processing.py`](https://github.com/foundation-model-stack/fms-hf-tuning/blob/main/scripts/offline_data_processing.py) to run data preprocessor standalone.

Dependency Updates
- simpleeval from <1.0 to <2.0
- transformers updated from 4.48.1 to 4.49
- datasets from <3.0 to <4.0

Additional changes
- Support automatic HuggingFace checkpointing for ScatterMoE. Converted checkpoint can be found at `hf_converted_checkpoint` folder within every saved checkpoint directory.
- Add Github Action to free up disk space.
- Remove deprecated `push_to_hub_token` to resolve warning.

Full list of Changes
* build(deps): Update simpleeval requirement from <1.0,>=0.9.13 to >=0.9.13,<2.0 by dependabot in https://github.com/foundation-model-stack/fms-hf-tuning/pull/369
* feat: Enable streaming in data preprocessor by willmj in https://github.com/foundation-model-stack/fms-hf-tuning/pull/437
* feat: Support for add special tokens via cli args by YashasviChaurasia in https://github.com/foundation-model-stack/fms-hf-tuning/pull/473
* feat: add support for chat template from data config by YashasviChaurasia in https://github.com/foundation-model-stack/fms-hf-tuning/pull/474
* feat: Add tokenizer data handler. by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/487
* build(deps): upgrade transformers to 4.49 by anhuong in https://github.com/foundation-model-stack/fms-hf-tuning/pull/485
* fix: use main_process_first instead of broadcast_object_list by willmj in https://github.com/foundation-model-stack/fms-hf-tuning/pull/458
* feat: Update data handler backend and introduce filter based handlers. by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/488
* fix: data handler to string conversion bug by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/490
* feat: add sum op for trainer controller by kmehant in https://github.com/foundation-model-stack/fms-hf-tuning/pull/491
* feat: support moe hf chkpt by kmehant in https://github.com/foundation-model-stack/fms-hf-tuning/pull/486
* chore: Add a GH runner to free up disk space by aluu317 in https://github.com/foundation-model-stack/fms-hf-tuning/pull/496
* build(deps): Update datasets requirement from <3.0,>=2.15.0 to >=2.15.0,<4.0 by dependabot in https://github.com/foundation-model-stack/fms-hf-tuning/pull/340
* fix: Remove deprecated push_to_hub_token to resolve warning by Luka-D in https://github.com/foundation-model-stack/fms-hf-tuning/pull/419
* feat: Add tools and documents usage in chat template by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/495
* test:Addition of data preprocessing script, decoupling data preprocessor from the tuning by Abhishek-TAMU in https://github.com/foundation-model-stack/fms-hf-tuning/pull/459
* fix: bug which caused test case failures after PR merge by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/500
* build(deps): changes needed to support mamba/jamba model by anhuong in https://github.com/foundation-model-stack/fms-hf-tuning/pull/400


**Full Changelog**: https://github.com/foundation-model-stack/fms-hf-tuning/compare/v2.6.0...v2.7.1

Recommend to use v2.7.1. Needed additional dependency update for Pypi to publish wheel release. To view set of changes see v2.7.1.

Summary of changes

Data Preprocessor Updates:
* Renaming of Data handlers to make it self explanatory:
`apply_dataset_formatting` renamed to `add_tokenizer_eos_token`
`apply_custom_data_formatting_jinja_template` renamed to `apply_custom_jinja_template`
* Added support for Jinja template rendering in the dataset.
* Introduced a data handler for the EPT use case.
* Added support for renaming and retaining dataset columns in Data handlers.
* Introduced an EOS token flag to disable its use in data handlers.

Documentation Additions:
* Added documentation on running EPT with our library.

Dependency Updates:
* Updated the upper limit of `TRL` from `v0.11` to `v0.14` and the lower limit from `v0.9.3`to `v0.13`.
* Updated the upper limit of `Transformers` to `v4.48.1` and the lower limit from `v4.45` to `v4.46`.

Additional Changes:
* Setting `Legacy=True` in `AutoTokenizer` to suppress warnings.
* Bug fixed in Aim tracker where server-based tracking was not being picked up.

Full list of Changes
* chore: bump python version by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/449
* fix: Hardcode Legacy behavior to True to resolve warning. by Luka-D in https://github.com/foundation-model-stack/fms-hf-tuning/pull/446
* feat: Add support for jinja based template rendering of the dataset by Abhishek-TAMU in https://github.com/foundation-model-stack/fms-hf-tuning/pull/438
* fix: bug in aim tracker where the server based tracking was not picked by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/454
* fix: remove duplicate main_process_port entry by prince14322 in https://github.com/foundation-model-stack/fms-hf-tuning/pull/444
* fix: space missing from data_formatter_template causing mismatch with response_template by HarikrishnanBalagopal in https://github.com/foundation-model-stack/fms-hf-tuning/pull/455
* fix: Using Sandboxed Environment in handler rendering Jinja template by Abhishek-TAMU in https://github.com/foundation-model-stack/fms-hf-tuning/pull/456
* docs: Add documentation on how to do EPT runs with our library. by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/461
* feat: Rename data handlers and add a new one for EPT scenarios by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/460
* fix:Retrieval of tokenizer_kwargs in data handler tokenize_and_apply_input_masking by Abhishek-TAMU in https://github.com/foundation-model-stack/fms-hf-tuning/pull/465
* feat: Add support for renaming and retaining columns in data preprocessor by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/466
* chore(deps): upgrade trl and transformers by willmj in https://github.com/foundation-model-stack/fms-hf-tuning/pull/448
* feat: adding eos token to be made a flag so we don't force it on every handler by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/467
* chore:Merge set of changes for v2.6.0 by Abhishek-TAMU in https://github.com/foundation-model-stack/fms-hf-tuning/pull/472

New Contributors
* prince14322 made their first contribution in https://github.com/foundation-model-stack/fms-hf-tuning/pull/444

**Full Changelog**: https://github.com/foundation-model-stack/fms-hf-tuning/compare/v2.5.0...v2.6.0

What's Changed
* docs: EOS token support by willmj in https://github.com/foundation-model-stack/fms-hf-tuning/pull/443
* feat: add scanner tracker by aluu317 in https://github.com/foundation-model-stack/fms-hf-tuning/pull/422
* docs: add note to note that file extension is required in training data path by willmj in https://github.com/foundation-model-stack/fms-hf-tuning/pull/447
* feat: updates documentation with chat template guide flowchart by YashasviChaurasia in https://github.com/foundation-model-stack/fms-hf-tuning/pull/445
* chore: bump python version by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/449
* fix: Hardcode Legacy behavior to True to resolve warning. by Luka-D in https://github.com/foundation-model-stack/fms-hf-tuning/pull/446
* feat: Add support for jinja based template rendering of the dataset by Abhishek-TAMU in https://github.com/foundation-model-stack/fms-hf-tuning/pull/438
* fix: bug in aim tracker where the server based tracking was not picked by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/454
* fix: remove duplicate main_process_port entry by prince14322 in https://github.com/foundation-model-stack/fms-hf-tuning/pull/444
* fix: space missing from data_formatter_template causing mismatch with response_template by HarikrishnanBalagopal in https://github.com/foundation-model-stack/fms-hf-tuning/pull/455
* fix: Using Sandboxed Environment in handler rendering Jinja template by Abhishek-TAMU in https://github.com/foundation-model-stack/fms-hf-tuning/pull/456
* docs: Add documentation on how to do EPT runs with our library. by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/461
* feat: Rename data handlers and add a new one for EPT scenarios by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/460
* fix:Retrieval of tokenizer_kwargs in data handler tokenize_and_apply_input_masking by Abhishek-TAMU in https://github.com/foundation-model-stack/fms-hf-tuning/pull/465
* feat: Add support for renaming and retaining columns in data preprocessor by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/466
* chore(deps): upgrade trl and transformers by willmj in https://github.com/foundation-model-stack/fms-hf-tuning/pull/448
* feat: adding eos token to be made a flag so we don't force it on every handler by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/467

New Contributors
* YashasviChaurasia made their first contribution in https://github.com/foundation-model-stack/fms-hf-tuning/pull/445
* prince14322 made their first contribution in https://github.com/foundation-model-stack/fms-hf-tuning/pull/444

**Full Changelog**: https://github.com/foundation-model-stack/fms-hf-tuning/compare/v2.4.0-rc.2...v2.6.0-rc.1

In v2.5.0, `fms-hf-tuning` library is now built with python 3.12. See more on support update below.
Other note-worthy updates in this release:

**New tracker:**
- New tracker using [HFResourceScanner](https://github.com/foundation-model-stack/hf-resource-scanner/tree/main/HFResourceScanner) to enable lightweight tracking of memory usage and train time during training.

**Support update:**
- We have tested and extended the support for python 3.12. `fms-hf-tuning` can now run with py 3.9, 3.10, 3.11 and 3.12.
- `Dockerfile` is updated to use python 3.12 as default.

What's Changed
* docs: EOS token support by willmj in https://github.com/foundation-model-stack/fms-hf-tuning/pull/443
* feat: add scanner tracker by aluu317 in https://github.com/foundation-model-stack/fms-hf-tuning/pull/422
* docs: add note to note that file extension is required in training data path by willmj in https://github.com/foundation-model-stack/fms-hf-tuning/pull/447
* feat: updates documentation with chat template guide flowchart by YashasviChaurasia in https://github.com/foundation-model-stack/fms-hf-tuning/pull/445
* chore: bump python version by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/449

New Contributors
* YashasviChaurasia made their first contribution in https://github.com/foundation-model-stack/fms-hf-tuning/pull/445

**Full Changelog**: https://github.com/foundation-model-stack/fms-hf-tuning/compare/v2.4.0...v2.5.0

Summary of Changes
Acceleration Updates:
- Dataclass args added for accelerated MoE tuning, which can be activated using the new int flag `fast_moe` for the number of expert parallel sharding.
- Update function name from `requires_agumentation` to `requires_augmentation`.
- Note: the lower limit of the `fms-acceleration` library has been increased to 0.6.0.
Data Preprocessor Updates:
- Allows for padding free plugin to be used without response template.
- Allows HF dataset IDs to be passed via the `training_data_path flag`.
Additional Changes:
- Add pad_token to special_tokens_dict when pad_token == eos_token, which improves granite 3.0 + 3.1 quality on the tuning stack.
For full details of changes, see the [release notes](https://github.com/foundation-model-stack/fms-hf-tuning/releases/tag/v2.4.0).
(edited)


Full List of Change
* fix: broken README.md link by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/429
* feat: Allow hf dataset id to be passed via training_data_path by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/431
* feat: dataclass args for accelerated MoE tuning by willmj in https://github.com/foundation-model-stack/fms-hf-tuning/pull/390
* feat: allow for padding free plugin to be used without response template by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/430
* fix: function name from `requires_agumentation` to `requires_augmentation` by willmj in https://github.com/foundation-model-stack/fms-hf-tuning/pull/434
* fix: Add pad_token to special_tokens_dict when pad_token == eos_token by Abhishek-TAMU in https://github.com/foundation-model-stack/fms-hf-tuning/pull/436
* chore(deps): upgrade fms-acceleration to >= 0.6 by willmj in https://github.com/foundation-model-stack/fms-hf-tuning/pull/440
* docs: update granite3 model support by anhuong in https://github.com/foundation-model-stack/fms-hf-tuning/pull/441


**Full Changelog**: https://github.com/foundation-model-stack/fms-hf-tuning/compare/v2.3.1...v2.4.0