Huggingface-hub

Documentation

Version v0.5.0 is the first version which features an API reference. It is still a work in progress with features lacking, some images not rendering, and a documentation reorg coming up, but should already provide significantly simpler access to the `huggingface_hub` API.

The documentation is visible [here](https://huggingface.co/docs/huggingface_hub/index).

* API reference documentation by LysandreJik in https://github.com/huggingface/huggingface_hub/pull/782
* [API Reference docs] Remove git references from GitHub Action templates by LysandreJik in https://github.com/huggingface/huggingface_hub/pull/813
* DOC API docstring improvements by adrinjalali in https://github.com/huggingface/huggingface_hub/pull/731

Model & datasets list improvements

The `list_models` and `list_datasets` methods have been improved in several ways.

List private models

These two methods now accept the `token` keyword to specify your token. Specifying the token will include your private models and datasets in the returned list.

* Support list_models and list_datasets with token arg by muellerzr in https://github.com/huggingface/huggingface_hub/pull/638

Modelcard metadata

These two methods now accept the `cardData` boolean argument. If set to `True`, the modelcard metadata will also be returned when using these two methods.

* Include cardData in list_models and list_datasets by muellerzr in https://github.com/huggingface/huggingface_hub/pull/639

Filtering by carbon emissions

The `list_models` method now also accepts an `emissions_trehsholds` parameter to filter by carbon emissions.

* Enable filtering by carbon emission by muellerzr in https://github.com/huggingface/huggingface_hub/pull/668

Keras improvements

The Keras serialization and upload methods have been worked on to provide better support for models:
- All parameters are now included in the saved model when using `push_to_hub_keras`
- `log_dir` parameter for TensorBoard logs, which will automatically spawn a TensorBoard instance on the Hub.
- Automatic model card

* Introduce `include_optimizer` parameter to `push_to_hub_keras()` by merveenoyan in https://github.com/huggingface/huggingface_hub/pull/616
* Add TensorBoard for Keras models by merveenoyan in https://github.com/huggingface/huggingface_hub/pull/651
* Create Automatic Keras model card by merveenoyan in https://github.com/huggingface/huggingface_hub/pull/679
* Allow TensorBoard Override for same Repository by merveenoyan in https://github.com/huggingface/huggingface_hub/pull/709
* Add tempfile for tensorboard logs in tensorboard tests in `test_keras_integration.py` by merveenoyan in https://github.com/huggingface/huggingface_hub/pull/761

Contributing guide

A contributing guide is now available for the `huggingface_hub` repository. For any and all information related to contributing to the repository, please check it out!

Read more about it here: [CONTRIBUTING.md](https://github.com/huggingface/huggingface_hub/blob/main/CONTRIBUTING.md).

Pre-commit hooks

The `huggingface_hub` GitHub repository has several checks to ensure that the code respects code quality standards. Opt-in pre-commit hooks have been added in order to make it simpler for contributors to leverage them.

Read more about it in the aforementionned [CONTRIBUTING guide](https://github.com/huggingface/huggingface_hub/blob/main/CONTRIBUTING.md).

* MNT Add pre-commit hooks by adrinjalali in https://github.com/huggingface/huggingface_hub/pull/807

Renaming and transferring repositories

Repositories can now be renamed and transferred programmatically using `move_repo`.

* Allow renaming and transferring repos programmatically by osanseviero in https://github.com/huggingface/huggingface_hub/pull/704

Breaking changes & deprecation

⛔ The following methods have now been removed following a deprecation cycle

`list_repos_objs`

The `list_repos_objs` and the accompanying CLI utility `huggingface-cli repo ls-files` have been removed.
The same can be done using the `model_info` and `dataset_info` methods.

* Remove deprecated `list_repos_objs` and `huggingface-cli repo ls-files` by julien-c in https://github.com/huggingface/huggingface_hub/pull/702

Python 3.6

Python 3.6 support is now dropped as end of life. Using Python 3.6 and installing `huggingface_hub` will result in version v0.4.0 being installed.

* CI support python 3.7-3.10 - remove 3.6 support by adrinjalali in https://github.com/huggingface/huggingface_hub/pull/790

⚠️ Items below are now deprecated and will be removed in a future version

* API deprecate positional args in file_download and hf_api by adrinjalali in https://github.com/huggingface/huggingface_hub/pull/745
* MNT deprecate name and organization in favor of repo_id by adrinjalali in https://github.com/huggingface/huggingface_hub/pull/733

What's Changed

* Include "model" in repo_type to keep consistency by muellerzr in https://github.com/huggingface/huggingface_hub/pull/620
* Hotfix for repo_type by muellerzr in https://github.com/huggingface/huggingface_hub/pull/623
* fix: typo in docstring by ariG23498 in https://github.com/huggingface/huggingface_hub/pull/647
* {upload|delete}_file: Remove client-side filename validation by SBrandeis in https://github.com/huggingface/huggingface_hub/pull/669
* Ensure `post_method` is only executed once by sgugger in https://github.com/huggingface/huggingface_hub/pull/676
* Remove paying subscription mention from docstring by cakiki in https://github.com/huggingface/huggingface_hub/pull/653
* Improve tests and logging by muellerzr in https://github.com/huggingface/huggingface_hub/pull/682
* docs(links): Update `settings/token` to `settings/tokens` by ronvoluted in https://github.com/huggingface/huggingface_hub/pull/699
* Add support for private hub by juliensimon in https://github.com/huggingface/huggingface_hub/pull/703
* Add retry_endpoint for test stability by osanseviero in https://github.com/huggingface/huggingface_hub/pull/719
* FIX fix a bug in _filter_emissions to accept numbers w/o decimal and dict emissions by adrinjalali in https://github.com/huggingface/huggingface_hub/pull/753
* Logging fix for `hf_api`, logging documentation by LysandreJik in https://github.com/huggingface/huggingface_hub/pull/748
* Contributing guide & code of conduct by LysandreJik in https://github.com/huggingface/huggingface_hub/pull/692
* Fix pytorch and tensorflow python matrix by osanseviero in https://github.com/huggingface/huggingface_hub/pull/760
* MNT add links to related projects and the forum on issue template by adrinjalali in https://github.com/huggingface/huggingface_hub/pull/773
* Note on the README by LysandreJik in https://github.com/huggingface/huggingface_hub/pull/772
* Remove autoreviewers by muellerzr in https://github.com/huggingface/huggingface_hub/pull/793
* CI Error on FutureWarning by adrinjalali in https://github.com/huggingface/huggingface_hub/pull/787
* MNT more informative message on error in `Hf.Api.delete_repo` by adrinjalali in https://github.com/huggingface/huggingface_hub/pull/783
* Add security status by McPatate in https://github.com/huggingface/huggingface_hub/pull/654
* Remove redundant part of security test by osanseviero in https://github.com/huggingface/huggingface_hub/pull/802
* Changed test repository names to fix tests by merveenoyan in https://github.com/huggingface/huggingface_hub/pull/803
* TST calling delete_repo under tempfile for fixing the test by merveenoyan in https://github.com/huggingface/huggingface_hub/pull/804
* Disable logging in with organization token by merveenoyan in https://github.com/huggingface/huggingface_hub/pull/780
* MNT change dev version to 0.5, 0.4 is already released by adrinjalali in https://github.com/huggingface/huggingface_hub/pull/810
* 👨‍💻 Configure HF Hub URL with environment variable by SBrandeis in https://github.com/huggingface/huggingface_hub/pull/815
* MNT support oder requests versions by adrinjalali in https://github.com/huggingface/huggingface_hub/pull/817
* Rename the env variable `HF_ENDPOINT`. by Narsil in https://github.com/huggingface/huggingface_hub/pull/819

New Contributors

* McPatate made their first contribution in https://github.com/huggingface/huggingface_hub/pull/583
* FremyCompany made their first contribution in https://github.com/huggingface/huggingface_hub/pull/606
* simoninithomas made their first contribution in https://github.com/huggingface/huggingface_hub/pull/633
* mlonaws made their first contribution in https://github.com/huggingface/huggingface_hub/pull/630
* ariG23498 made their first contribution in https://github.com/huggingface/huggingface_hub/pull/647
* J-Petiot made their first contribution in https://github.com/huggingface/huggingface_hub/pull/660
* ronvoluted made their first contribution in https://github.com/huggingface/huggingface_hub/pull/699
* juliensimon made their first contribution in https://github.com/huggingface/huggingface_hub/pull/703
* allendorf made their first contribution in https://github.com/huggingface/huggingface_hub/pull/742
* frgfm made their first contribution in https://github.com/huggingface/huggingface_hub/pull/747
* hbredin made their first contribution in https://github.com/huggingface/huggingface_hub/pull/688

**Full Changelog**: https://github.com/huggingface/huggingface_hub/compare/v0.4.0...v0.5.0

Tag listing

* Introduce Tag Listing by muellerzr in https://github.com/huggingface/huggingface_hub/pull/537

This PR introduces the ability to fetch all available tags for models or datasets and returns them as a nested namespace object, for example:

py
>>> from huggingface_hub import HfApi

>>> api = HfApi()
>>> tags = api.get_model_tags()
>>> print(tags)
Available Attributes:
* benchmark
* language_creators
* languages
* licenses
* multilinguality
* size_categories
* task_categories
* task_ids

>>> print(tags.benchmark)
Available Attributes:
* raft
* superb
* test


Namespace objects

* Namespace Objects for Search Parameters by muellerzr in https://github.com/huggingface/huggingface_hub/pull/556

With a goal of adding more tab-completion to the library, this PR introduces two objects:
- `DatasetSearchArguments`
- `ModelSearchArguments`

These two `AttributeDictionary` objects contain all the valid information we can extract from a model as tab-complete parameters. We also include the `author_or_organization` and `dataset` (or `model`) `_name` as well through careful string splitting.

Model Filter

* Implement a Model Filter class by muellerzr in https://github.com/huggingface/huggingface_hub/pull/553

This PR introduces a new way to search the hub: the `ModelFilter` class.

It is a simple Enum at first to the user, allowing them to specify what they want to search for, such as:
python
f = ModelFilter(author="microsoft", model_name="wavlm-base-sd", framework="pytorch")

From there, they can pass in this filter to the new `list_models_by_filter` function in `HfApi` to search through it:
python
models = api.list_modes(filter=f)


The API may then be used for complex queries:
py
args = ModelSearchArguments()
f = ModelFilter(framework=[args.library.pytorch, args.library.TensorFlow], model_name="bert", tasks=[args.pipeline_tag.Summarization, args.pipeline_tag.TokenClassification])

api.list_models_from_filter(f)


Ignoring filenames in snapshot_download

This PR introduces a way to limit the files that will be fetched by the `snapshot_download`. This is useful when you want to download and cache an entire repository without using git, and that you want to skip files according to their filenames.

* [Snapshot download] allow some filenames to be ignored by patrickvonplaten in https://github.com/huggingface/huggingface_hub/pull/566


What's Changed
* [Hotfix][API] card_data => cardData on /api/datasets by julien-c in https://github.com/huggingface/huggingface_hub/pull/530
* Fix the progress bars when cloning a repository by LysandreJik in https://github.com/huggingface/huggingface_hub/pull/517
* Update Hugging Face Hub documentation README and Endpoints by muellerzr in https://github.com/huggingface/huggingface_hub/pull/527
* Convert string functions to f-string by muellerzr in https://github.com/huggingface/huggingface_hub/pull/536
* Fixing FS for `espnet`. by Narsil in https://github.com/huggingface/huggingface_hub/pull/542
* [snapshot_download] upgrade to canonical separator by julien-c in https://github.com/huggingface/huggingface_hub/pull/545
* Add test directions by muellerzr in https://github.com/huggingface/huggingface_hub/pull/547
* [HOTFIX] Change test for missing_input to reflect back-end redirect changes by muellerzr in https://github.com/huggingface/huggingface_hub/pull/552
* Bring consistency to download and upload APIs by muellerzr in https://github.com/huggingface/huggingface_hub/pull/574
* Search by authors and string by FrancescoSaverioZuppichini in https://github.com/huggingface/huggingface_hub/pull/531
* Quick typo by muellerzr in https://github.com/huggingface/huggingface_hub/pull/575

New Contributors
* kahne made their first contribution in https://github.com/huggingface/huggingface_hub/pull/569
* FrancescoSaverioZuppichini made their first contribution in https://github.com/huggingface/huggingface_hub/pull/531

**Full Changelog**: https://github.com/huggingface/huggingface_hub/compare/v0.2.1...v0.4.0

This is a patch release fixing an issue with the notebook login.

https://github.com/huggingface/huggingface_hub/commit/5e2da9bae95ed4c99683e9572ecc32c9e0da5e15#diff-fb1696cbcf008dd89dde5e8c1da9d4be5a8f7d809bc32f07d4453caba40df15f


The notebook login is adapted to work with the access tokens.

Skipping large files

The `Repository` class now has an additional parameter, `skip_lfs_files`, which allows cloning the repository while skipping the large file download.

https://github.com/huggingface/huggingface_hub/pull/472

Local files only for `snapshot_download`

The `snapshot_download` method can now take `local_files_only` as a parameter to enable leveraging previously downloaded files.

https://github.com/huggingface/huggingface_hub/pull/505

What's Changed

* clean_ok should be True by default by LysandreJik in https://github.com/huggingface/huggingface_hub/pull/462


**Full Changelog**: https://github.com/huggingface/huggingface_hub/compare/v0.1.1...v0.1.2