Fms-hf-tuning

Summary of changes

Data Preprocessor Updates:
* Renaming of Data handlers to make it self explanatory:
`apply_dataset_formatting` renamed to `add_tokenizer_eos_token`
`apply_custom_data_formatting_jinja_template` renamed to `apply_custom_jinja_template`
* Added support for Jinja template rendering in the dataset.
* Introduced a data handler for the EPT use case.
* Added support for renaming and retaining dataset columns in Data handlers.
* Introduced an EOS token flag to disable its use in data handlers.

Documentation Additions:
* Added documentation on running EPT with our library.

Dependency Updates:
* Updated the upper limit of `TRL` from `v0.11` to `v0.14` and the lower limit from `v0.9.3`to `v0.13`.
* Updated the upper limit of `Transformers` to `v4.48.1` and the lower limit from `v4.45` to `v4.46`.

Additional Changes:
* Setting `Legacy=True` in `AutoTokenizer` to suppress warnings.
* Bug fixed in Aim tracker where server-based tracking was not being picked up.

Full list of Changes
* chore: bump python version by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/449
* fix: Hardcode Legacy behavior to True to resolve warning. by Luka-D in https://github.com/foundation-model-stack/fms-hf-tuning/pull/446
* feat: Add support for jinja based template rendering of the dataset by Abhishek-TAMU in https://github.com/foundation-model-stack/fms-hf-tuning/pull/438
* fix: bug in aim tracker where the server based tracking was not picked by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/454
* fix: remove duplicate main_process_port entry by prince14322 in https://github.com/foundation-model-stack/fms-hf-tuning/pull/444
* fix: space missing from data_formatter_template causing mismatch with response_template by HarikrishnanBalagopal in https://github.com/foundation-model-stack/fms-hf-tuning/pull/455
* fix: Using Sandboxed Environment in handler rendering Jinja template by Abhishek-TAMU in https://github.com/foundation-model-stack/fms-hf-tuning/pull/456
* docs: Add documentation on how to do EPT runs with our library. by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/461
* feat: Rename data handlers and add a new one for EPT scenarios by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/460
* fix:Retrieval of tokenizer_kwargs in data handler tokenize_and_apply_input_masking by Abhishek-TAMU in https://github.com/foundation-model-stack/fms-hf-tuning/pull/465
* feat: Add support for renaming and retaining columns in data preprocessor by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/466
* chore(deps): upgrade trl and transformers by willmj in https://github.com/foundation-model-stack/fms-hf-tuning/pull/448
* feat: adding eos token to be made a flag so we don't force it on every handler by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/467
* chore:Merge set of changes for v2.6.0 by Abhishek-TAMU in https://github.com/foundation-model-stack/fms-hf-tuning/pull/472

New Contributors
* prince14322 made their first contribution in https://github.com/foundation-model-stack/fms-hf-tuning/pull/444

**Full Changelog**: https://github.com/foundation-model-stack/fms-hf-tuning/compare/v2.5.0...v2.6.0

What's Changed
* docs: EOS token support by willmj in https://github.com/foundation-model-stack/fms-hf-tuning/pull/443
* feat: add scanner tracker by aluu317 in https://github.com/foundation-model-stack/fms-hf-tuning/pull/422
* docs: add note to note that file extension is required in training data path by willmj in https://github.com/foundation-model-stack/fms-hf-tuning/pull/447
* feat: updates documentation with chat template guide flowchart by YashasviChaurasia in https://github.com/foundation-model-stack/fms-hf-tuning/pull/445
* chore: bump python version by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/449
* fix: Hardcode Legacy behavior to True to resolve warning. by Luka-D in https://github.com/foundation-model-stack/fms-hf-tuning/pull/446
* feat: Add support for jinja based template rendering of the dataset by Abhishek-TAMU in https://github.com/foundation-model-stack/fms-hf-tuning/pull/438
* fix: bug in aim tracker where the server based tracking was not picked by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/454
* fix: remove duplicate main_process_port entry by prince14322 in https://github.com/foundation-model-stack/fms-hf-tuning/pull/444
* fix: space missing from data_formatter_template causing mismatch with response_template by HarikrishnanBalagopal in https://github.com/foundation-model-stack/fms-hf-tuning/pull/455
* fix: Using Sandboxed Environment in handler rendering Jinja template by Abhishek-TAMU in https://github.com/foundation-model-stack/fms-hf-tuning/pull/456
* docs: Add documentation on how to do EPT runs with our library. by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/461
* feat: Rename data handlers and add a new one for EPT scenarios by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/460
* fix:Retrieval of tokenizer_kwargs in data handler tokenize_and_apply_input_masking by Abhishek-TAMU in https://github.com/foundation-model-stack/fms-hf-tuning/pull/465
* feat: Add support for renaming and retaining columns in data preprocessor by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/466
* chore(deps): upgrade trl and transformers by willmj in https://github.com/foundation-model-stack/fms-hf-tuning/pull/448
* feat: adding eos token to be made a flag so we don't force it on every handler by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/467

New Contributors
* YashasviChaurasia made their first contribution in https://github.com/foundation-model-stack/fms-hf-tuning/pull/445
* prince14322 made their first contribution in https://github.com/foundation-model-stack/fms-hf-tuning/pull/444

**Full Changelog**: https://github.com/foundation-model-stack/fms-hf-tuning/compare/v2.4.0-rc.2...v2.6.0-rc.1

In v2.5.0, `fms-hf-tuning` library is now built with python 3.12. See more on support update below.
Other note-worthy updates in this release:

**New tracker:**
- New tracker using [HFResourceScanner](https://github.com/foundation-model-stack/hf-resource-scanner/tree/main/HFResourceScanner) to enable lightweight tracking of memory usage and train time during training.

**Support update:**
- We have tested and extended the support for python 3.12. `fms-hf-tuning` can now run with py 3.9, 3.10, 3.11 and 3.12.
- `Dockerfile` is updated to use python 3.12 as default.

What's Changed
* docs: EOS token support by willmj in https://github.com/foundation-model-stack/fms-hf-tuning/pull/443
* feat: add scanner tracker by aluu317 in https://github.com/foundation-model-stack/fms-hf-tuning/pull/422
* docs: add note to note that file extension is required in training data path by willmj in https://github.com/foundation-model-stack/fms-hf-tuning/pull/447
* feat: updates documentation with chat template guide flowchart by YashasviChaurasia in https://github.com/foundation-model-stack/fms-hf-tuning/pull/445
* chore: bump python version by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/449

New Contributors
* YashasviChaurasia made their first contribution in https://github.com/foundation-model-stack/fms-hf-tuning/pull/445

**Full Changelog**: https://github.com/foundation-model-stack/fms-hf-tuning/compare/v2.4.0...v2.5.0

Summary of Changes
Acceleration Updates:
- Dataclass args added for accelerated MoE tuning, which can be activated using the new int flag `fast_moe` for the number of expert parallel sharding.
- Update function name from `requires_agumentation` to `requires_augmentation`.
- Note: the lower limit of the `fms-acceleration` library has been increased to 0.6.0.
Data Preprocessor Updates:
- Allows for padding free plugin to be used without response template.
- Allows HF dataset IDs to be passed via the `training_data_path flag`.
Additional Changes:
- Add pad_token to special_tokens_dict when pad_token == eos_token, which improves granite 3.0 + 3.1 quality on the tuning stack.
For full details of changes, see the [release notes](https://github.com/foundation-model-stack/fms-hf-tuning/releases/tag/v2.4.0).
(edited)


Full List of Change
* fix: broken README.md link by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/429
* feat: Allow hf dataset id to be passed via training_data_path by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/431
* feat: dataclass args for accelerated MoE tuning by willmj in https://github.com/foundation-model-stack/fms-hf-tuning/pull/390
* feat: allow for padding free plugin to be used without response template by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/430
* fix: function name from `requires_agumentation` to `requires_augmentation` by willmj in https://github.com/foundation-model-stack/fms-hf-tuning/pull/434
* fix: Add pad_token to special_tokens_dict when pad_token == eos_token by Abhishek-TAMU in https://github.com/foundation-model-stack/fms-hf-tuning/pull/436
* chore(deps): upgrade fms-acceleration to >= 0.6 by willmj in https://github.com/foundation-model-stack/fms-hf-tuning/pull/440
* docs: update granite3 model support by anhuong in https://github.com/foundation-model-stack/fms-hf-tuning/pull/441


**Full Changelog**: https://github.com/foundation-model-stack/fms-hf-tuning/compare/v2.3.1...v2.4.0

What's Changed
* fix: broken README.md link by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/429
* feat: Allow hf dataset id to be passed via training_data_path by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/431
* feat: dataclass args for accelerated MoE tuning by willmj in https://github.com/foundation-model-stack/fms-hf-tuning/pull/390
* feat: allow for padding free plugin to be used without response template by dushyantbehl in https://github.com/foundation-model-stack/fms-hf-tuning/pull/430
* fix: function name from `requires_agumentation` to `requires_augmentation` by willmj in https://github.com/foundation-model-stack/fms-hf-tuning/pull/434
* fix: Add pad_token to special_tokens_dict when pad_token == eos_token by Abhishek-TAMU in https://github.com/foundation-model-stack/fms-hf-tuning/pull/436
* chore(deps): upgrade fms-acceleration to >= 0.6 by willmj in https://github.com/foundation-model-stack/fms-hf-tuning/pull/440
* docs: update granite3 model support by anhuong in https://github.com/foundation-model-stack/fms-hf-tuning/pull/441


**Full Changelog**: https://github.com/foundation-model-stack/fms-hf-tuning/compare/v2.3.0...v2.4.0-rc.2