Huggingface-hub

Repocard metadata

This PR adds a metadata_update function that allows the user to update the metadata in a repository on the hub. The function accepts a dict with metadata (following the same pattern as the YAML in the README) and behaves as follows for all top level fields except model-index.

Examples:

Starting from


existing_results = [{
'dataset': {'name': 'IMDb', 'type': 'imdb'},
'metrics': [{'name': 'Accuracy', 'type': 'accuracy', 'value': 0.995}],
'task': {'name': 'Text Classification', 'type': 'text-classification'}
}]


1. Overwrite existing metric value in existing result
py
new_results = deepcopy(existing_results)

*Disclaimer*: This release was initially released with advertised support for 844. It was not released in this release and will be in v0.7.

fastai support

v0.6.0 introduces downstream (download) and upstream (upload) support for the fastai libraries. It supports fastai versions above 2.4.
The integration is detailed in the following [blog](https://huggingface.co/blog/fastai).

* Add fastai upstream and downstream capacities for fastai>=2.4 and fastcore>=1.3.27 versions by omarespejel in 678

Automatic binary file tracking in `Repository`

Binary files are now rejected by default by the Hub. v0.6.0 introduces automatic binary file tracking through the `auto_lfs_track` argument of the `Repository.git_add` method. It also introduces the `Repository.auto_track_binary_files` method which can be used independently of other methods.

* ENH Auto track binary files in Repository by LysandreJik in 828

`skip_lfs_file` is now added to mixins

The parameter `skip_lfs_files` is now added to the different mixins. This will enable pushing files to the hub without first downloading the files above 10MB. This should drammatically reduce the time needed when updating a modelcard, a configuration file, and others.

* :sparkles: add skip_lfs_files to mixins' push_to_hub by nateraw in 858

Keras support improvement

The support for Keras model is greatly improved through several additions:

- The `save_pretrained_keras` method now accepts a list of tags that will automatically be added to the repository.
- Download statistics are now available on Keras models

* Introducing list of tags to Keras model card by merveenoyan in 806
* Enable keras download stats by merveenoyan in 860

Bugfixes and improvements

* FIX don't raise if name/organizaiton are passed postionally by adrinjalali in 822
* ENH Use provided token from HUGGING_FACE_HUB_TOKEN env variable if available by FrancescoSaverioZuppichini in 794
* tests(hf_api): remove infectionTypes field by McPatate in 834
* Remove docs, tasks and inference API from huggingface_hub by osanseviero in 833
* FEAT Uniformize `hf_api` a bit and add support for Spaces by julien-c in 792
* Add a bug report template by osanseviero in 832
* clean up formatting by stevhliu in 839
* Release guide by LysandreJik in 820
* Fix keras test by osanseviero in 855
* DOC Add quick start guide by stevhliu in 850
* MNT refactor: subprocess.run -> run_subprocess by LysandreJik in 352
* MNT enable preview on black by adrinjalali in 849
* Update how to guides by stevhliu in 840
* Update contribution guide for merging PRs by stevhliu in 856
* DOC Update landing page by stevhliu in 854
* space after uri by leondz in 866

This is a patch release fixing a breaking backward compatibility issue.

Linked PR: https://github.com/huggingface/huggingface_hub/pull/822

Documentation

Version v0.5.0 is the first version which features an API reference. It is still a work in progress with features lacking, some images not rendering, and a documentation reorg coming up, but should already provide significantly simpler access to the `huggingface_hub` API.

The documentation is visible [here](https://huggingface.co/docs/huggingface_hub/index).

* API reference documentation by LysandreJik in https://github.com/huggingface/huggingface_hub/pull/782
* [API Reference docs] Remove git references from GitHub Action templates by LysandreJik in https://github.com/huggingface/huggingface_hub/pull/813
* DOC API docstring improvements by adrinjalali in https://github.com/huggingface/huggingface_hub/pull/731

Model & datasets list improvements

The `list_models` and `list_datasets` methods have been improved in several ways.

List private models

These two methods now accept the `token` keyword to specify your token. Specifying the token will include your private models and datasets in the returned list.

* Support list_models and list_datasets with token arg by muellerzr in https://github.com/huggingface/huggingface_hub/pull/638

Modelcard metadata

These two methods now accept the `cardData` boolean argument. If set to `True`, the modelcard metadata will also be returned when using these two methods.

* Include cardData in list_models and list_datasets by muellerzr in https://github.com/huggingface/huggingface_hub/pull/639

Filtering by carbon emissions

The `list_models` method now also accepts an `emissions_trehsholds` parameter to filter by carbon emissions.

* Enable filtering by carbon emission by muellerzr in https://github.com/huggingface/huggingface_hub/pull/668

Keras improvements

The Keras serialization and upload methods have been worked on to provide better support for models:
- All parameters are now included in the saved model when using `push_to_hub_keras`
- `log_dir` parameter for TensorBoard logs, which will automatically spawn a TensorBoard instance on the Hub.
- Automatic model card

* Introduce `include_optimizer` parameter to `push_to_hub_keras()` by merveenoyan in https://github.com/huggingface/huggingface_hub/pull/616
* Add TensorBoard for Keras models by merveenoyan in https://github.com/huggingface/huggingface_hub/pull/651
* Create Automatic Keras model card by merveenoyan in https://github.com/huggingface/huggingface_hub/pull/679
* Allow TensorBoard Override for same Repository by merveenoyan in https://github.com/huggingface/huggingface_hub/pull/709
* Add tempfile for tensorboard logs in tensorboard tests in `test_keras_integration.py` by merveenoyan in https://github.com/huggingface/huggingface_hub/pull/761

Contributing guide

A contributing guide is now available for the `huggingface_hub` repository. For any and all information related to contributing to the repository, please check it out!

Read more about it here: [CONTRIBUTING.md](https://github.com/huggingface/huggingface_hub/blob/main/CONTRIBUTING.md).

Pre-commit hooks

The `huggingface_hub` GitHub repository has several checks to ensure that the code respects code quality standards. Opt-in pre-commit hooks have been added in order to make it simpler for contributors to leverage them.

Read more about it in the aforementionned [CONTRIBUTING guide](https://github.com/huggingface/huggingface_hub/blob/main/CONTRIBUTING.md).

* MNT Add pre-commit hooks by adrinjalali in https://github.com/huggingface/huggingface_hub/pull/807

Renaming and transferring repositories

Repositories can now be renamed and transferred programmatically using `move_repo`.

* Allow renaming and transferring repos programmatically by osanseviero in https://github.com/huggingface/huggingface_hub/pull/704

Breaking changes & deprecation

⛔ The following methods have now been removed following a deprecation cycle

`list_repos_objs`

The `list_repos_objs` and the accompanying CLI utility `huggingface-cli repo ls-files` have been removed.
The same can be done using the `model_info` and `dataset_info` methods.

* Remove deprecated `list_repos_objs` and `huggingface-cli repo ls-files` by julien-c in https://github.com/huggingface/huggingface_hub/pull/702

Python 3.6

Python 3.6 support is now dropped as end of life. Using Python 3.6 and installing `huggingface_hub` will result in version v0.4.0 being installed.

* CI support python 3.7-3.10 - remove 3.6 support by adrinjalali in https://github.com/huggingface/huggingface_hub/pull/790

⚠️ Items below are now deprecated and will be removed in a future version

* API deprecate positional args in file_download and hf_api by adrinjalali in https://github.com/huggingface/huggingface_hub/pull/745
* MNT deprecate name and organization in favor of repo_id by adrinjalali in https://github.com/huggingface/huggingface_hub/pull/733

What's Changed

* Include "model" in repo_type to keep consistency by muellerzr in https://github.com/huggingface/huggingface_hub/pull/620
* Hotfix for repo_type by muellerzr in https://github.com/huggingface/huggingface_hub/pull/623
* fix: typo in docstring by ariG23498 in https://github.com/huggingface/huggingface_hub/pull/647
* {upload|delete}_file: Remove client-side filename validation by SBrandeis in https://github.com/huggingface/huggingface_hub/pull/669
* Ensure `post_method` is only executed once by sgugger in https://github.com/huggingface/huggingface_hub/pull/676
* Remove paying subscription mention from docstring by cakiki in https://github.com/huggingface/huggingface_hub/pull/653
* Improve tests and logging by muellerzr in https://github.com/huggingface/huggingface_hub/pull/682
* docs(links): Update `settings/token` to `settings/tokens` by ronvoluted in https://github.com/huggingface/huggingface_hub/pull/699
* Add support for private hub by juliensimon in https://github.com/huggingface/huggingface_hub/pull/703
* Add retry_endpoint for test stability by osanseviero in https://github.com/huggingface/huggingface_hub/pull/719
* FIX fix a bug in _filter_emissions to accept numbers w/o decimal and dict emissions by adrinjalali in https://github.com/huggingface/huggingface_hub/pull/753
* Logging fix for `hf_api`, logging documentation by LysandreJik in https://github.com/huggingface/huggingface_hub/pull/748
* Contributing guide & code of conduct by LysandreJik in https://github.com/huggingface/huggingface_hub/pull/692
* Fix pytorch and tensorflow python matrix by osanseviero in https://github.com/huggingface/huggingface_hub/pull/760
* MNT add links to related projects and the forum on issue template by adrinjalali in https://github.com/huggingface/huggingface_hub/pull/773
* Note on the README by LysandreJik in https://github.com/huggingface/huggingface_hub/pull/772
* Remove autoreviewers by muellerzr in https://github.com/huggingface/huggingface_hub/pull/793
* CI Error on FutureWarning by adrinjalali in https://github.com/huggingface/huggingface_hub/pull/787
* MNT more informative message on error in `Hf.Api.delete_repo` by adrinjalali in https://github.com/huggingface/huggingface_hub/pull/783
* Add security status by McPatate in https://github.com/huggingface/huggingface_hub/pull/654
* Remove redundant part of security test by osanseviero in https://github.com/huggingface/huggingface_hub/pull/802
* Changed test repository names to fix tests by merveenoyan in https://github.com/huggingface/huggingface_hub/pull/803
* TST calling delete_repo under tempfile for fixing the test by merveenoyan in https://github.com/huggingface/huggingface_hub/pull/804
* Disable logging in with organization token by merveenoyan in https://github.com/huggingface/huggingface_hub/pull/780
* MNT change dev version to 0.5, 0.4 is already released by adrinjalali in https://github.com/huggingface/huggingface_hub/pull/810
* 👨‍💻 Configure HF Hub URL with environment variable by SBrandeis in https://github.com/huggingface/huggingface_hub/pull/815
* MNT support oder requests versions by adrinjalali in https://github.com/huggingface/huggingface_hub/pull/817
* Rename the env variable `HF_ENDPOINT`. by Narsil in https://github.com/huggingface/huggingface_hub/pull/819

New Contributors

* McPatate made their first contribution in https://github.com/huggingface/huggingface_hub/pull/583
* FremyCompany made their first contribution in https://github.com/huggingface/huggingface_hub/pull/606
* simoninithomas made their first contribution in https://github.com/huggingface/huggingface_hub/pull/633
* mlonaws made their first contribution in https://github.com/huggingface/huggingface_hub/pull/630
* ariG23498 made their first contribution in https://github.com/huggingface/huggingface_hub/pull/647
* J-Petiot made their first contribution in https://github.com/huggingface/huggingface_hub/pull/660
* ronvoluted made their first contribution in https://github.com/huggingface/huggingface_hub/pull/699
* juliensimon made their first contribution in https://github.com/huggingface/huggingface_hub/pull/703
* allendorf made their first contribution in https://github.com/huggingface/huggingface_hub/pull/742
* frgfm made their first contribution in https://github.com/huggingface/huggingface_hub/pull/747
* hbredin made their first contribution in https://github.com/huggingface/huggingface_hub/pull/688

**Full Changelog**: https://github.com/huggingface/huggingface_hub/compare/v0.4.0...v0.5.0

Tag listing

* Introduce Tag Listing by muellerzr in https://github.com/huggingface/huggingface_hub/pull/537

This PR introduces the ability to fetch all available tags for models or datasets and returns them as a nested namespace object, for example:

py
>>> from huggingface_hub import HfApi

>>> api = HfApi()
>>> tags = api.get_model_tags()
>>> print(tags)
Available Attributes:
* benchmark
* language_creators
* languages
* licenses
* multilinguality
* size_categories
* task_categories
* task_ids

>>> print(tags.benchmark)
Available Attributes:
* raft
* superb
* test


Namespace objects

* Namespace Objects for Search Parameters by muellerzr in https://github.com/huggingface/huggingface_hub/pull/556

With a goal of adding more tab-completion to the library, this PR introduces two objects:
- `DatasetSearchArguments`
- `ModelSearchArguments`

These two `AttributeDictionary` objects contain all the valid information we can extract from a model as tab-complete parameters. We also include the `author_or_organization` and `dataset` (or `model`) `_name` as well through careful string splitting.

Model Filter

* Implement a Model Filter class by muellerzr in https://github.com/huggingface/huggingface_hub/pull/553

This PR introduces a new way to search the hub: the `ModelFilter` class.

It is a simple Enum at first to the user, allowing them to specify what they want to search for, such as:
python
f = ModelFilter(author="microsoft", model_name="wavlm-base-sd", framework="pytorch")

From there, they can pass in this filter to the new `list_models_by_filter` function in `HfApi` to search through it:
python
models = api.list_modes(filter=f)


The API may then be used for complex queries:
py
args = ModelSearchArguments()
f = ModelFilter(framework=[args.library.pytorch, args.library.TensorFlow], model_name="bert", tasks=[args.pipeline_tag.Summarization, args.pipeline_tag.TokenClassification])

api.list_models_from_filter(f)


Ignoring filenames in snapshot_download

This PR introduces a way to limit the files that will be fetched by the `snapshot_download`. This is useful when you want to download and cache an entire repository without using git, and that you want to skip files according to their filenames.

* [Snapshot download] allow some filenames to be ignored by patrickvonplaten in https://github.com/huggingface/huggingface_hub/pull/566


What's Changed
* [Hotfix][API] card_data => cardData on /api/datasets by julien-c in https://github.com/huggingface/huggingface_hub/pull/530
* Fix the progress bars when cloning a repository by LysandreJik in https://github.com/huggingface/huggingface_hub/pull/517
* Update Hugging Face Hub documentation README and Endpoints by muellerzr in https://github.com/huggingface/huggingface_hub/pull/527
* Convert string functions to f-string by muellerzr in https://github.com/huggingface/huggingface_hub/pull/536
* Fixing FS for `espnet`. by Narsil in https://github.com/huggingface/huggingface_hub/pull/542
* [snapshot_download] upgrade to canonical separator by julien-c in https://github.com/huggingface/huggingface_hub/pull/545
* Add test directions by muellerzr in https://github.com/huggingface/huggingface_hub/pull/547
* [HOTFIX] Change test for missing_input to reflect back-end redirect changes by muellerzr in https://github.com/huggingface/huggingface_hub/pull/552
* Bring consistency to download and upload APIs by muellerzr in https://github.com/huggingface/huggingface_hub/pull/574
* Search by authors and string by FrancescoSaverioZuppichini in https://github.com/huggingface/huggingface_hub/pull/531
* Quick typo by muellerzr in https://github.com/huggingface/huggingface_hub/pull/575

New Contributors
* kahne made their first contribution in https://github.com/huggingface/huggingface_hub/pull/569
* FrancescoSaverioZuppichini made their first contribution in https://github.com/huggingface/huggingface_hub/pull/531

**Full Changelog**: https://github.com/huggingface/huggingface_hub/compare/v0.2.1...v0.4.0