Huggingface-hub

Spaces on steroids 🚀

Spaces support has been substantially enhanced. You can now:
- Request hardware for your Space !
- Configure your Space secrets
- Get runtime information about your Space

py
Assign hardware when creating the Space
api.create_repo(repo_id=repo_id, repo_type="space", space_sdk="gradio", space_hardware="cpu-upgrade")

Configure some secrets
api.add_space_secret(repo_id=repo_id, key="HF_TOKEN", value="hf_api_***")

Request hardware on the fly
api.request_space_hardware(repo_id=repo_id, hardware="t4-medium")

Get Space runtime (state, hardware, sdk,...)
api.get_space_runtime(repo_id=repo_id)


Visit the [docs](https://huggingface.co/docs/huggingface_hub/main/en/guides/manage_spaces) for more details.

* Manage space hardware and secrets programmatically by Wauplin in 1243
* Mock Space tests requiring GPUs by Wauplin in 1263
* Remove `get_space_secrets` endpoint by Wauplin in 1264

**And in bonus:** Spaces now support Dockerfile natively !

py
api.create_repo(repo_id=repo_id, repo_type="space", space_sdk="docker")


Check out [the docs](https://huggingface.co/docs/hub/spaces-sdks-docker) for more details.

New features and improvements for `HfApi`

List all branches and tags from a repo

It's now possible to list branches/tags from a repo, getting exact `ref` and `target_commit`.
More details [in the docs](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/hf_api#huggingface_hub.HfApi.list_repo_refs).

py
>>> api.list_repo_refs("bigcode/the-stack", repo_type='dataset')
GitRefs(
branches=[
GitRefInfo(name='main', ref='refs/heads/main', target_commit='18edc1591d9ce72aa82f56c4431b3c969b210ae3'),
GitRefInfo(name='v1.1.a1', ref='refs/heads/v1.1.a1', target_commit='f9826b862d1567f3822d3d25649b0d6d22ace714')],
converts=[],
tags=[
GitRefInfo(name='v1.0', ref='refs/tags/v1.0', target_commit='c37a8cd1e382064d8aced5e05543c5f7753834da')
]
)


* List branches and tags from a repo by Wauplin in 1276

Manage your "favorite repos" list

New endpoints to [like](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/hf_api#huggingface_hub.HfApi.like) a repo, [unlike](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/hf_api#huggingface_hub.HfApi.unlike) it and [list_liked_repos](https://huggingface.co/docs/huggingface_hub/main/package_reference/hf_api#huggingface_hub.HfApi.list_liked_repos).

* Add method to list liked repos by Wauplin in 1254
* Fix list_liked_repos (only public likes are returned) by Wauplin in 1273
* Fix like test by Wauplin (direct commit on main)

Get the "real" `repo_id` when creating a repo

When using [create_repo](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/hf_api#huggingface_hub.HfApi.create_repo), one can provide a simple `repo_name` without specifying a namespace (example: `"my-cool-model"` instead of `"Wauplin/my-cool-model"`). This was annoying for as one could easily know if a namespace has been added to the provided repo_id. To facilitate this, the return value of `create_repo` is now an instance of [RepoUrl](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/hf_api#huggingface_hub.RepoUrl) which contains information like the endpoint, namespace, repo_id and repo_type.

* Extend create_repo return value by Wauplin in 1268

Create branch from any revision

By default, new branches start from main `HEAD`. It's now possible to specify any branch, tag or commit id to start from.

* Create branch from given revision by Wauplin in 1293

Modelcards

Modelcards module is getting some adjustments to better integrate with the Hub. The scope of this work is larger than "just" `huggingface_hub` and resulted in the launch of the [HF Model Card Guidebook](https://huggingface.co/docs/hub/model-card-guidebook) to help operationalize model cards in the ML community.

* Clarify what "More Resources" is by breaking it into its parts. by meg-huggingface in 1238
* Adding "summary" variables by meg-huggingface in 1248
* `datasetcard_template`: I think linking to a GH user does not make sense anymore now that dataset repos are fully on the Hub by julien-c in 1257
* Push empty model card and dataset card to the Hub by Wauplin in 1261
* [Dataset | Model card] When pushing to template repos, work on actual raw contents by julien-c in 1282
* Update `REGEX_YAML_BLOCK` by julien-c in 1285
* Make model use more visible by osanseviero in 1286

Documentation updates

Quite some effort has been put into the documentation in the past few weeks:
- a new [landing page](https://huggingface.co/docs/huggingface_hub/main/en/index)
- a more detailed [installation page](https://huggingface.co/docs/huggingface_hub/main/en/installation)
- a new page explaining the difference between [HTTP vs GIT approaches](https://huggingface.co/docs/huggingface_hub/main/en/concepts/git_vs_http)
- revisited ["Searching the Hub" page ](https://huggingface.co/docs/huggingface_hub/searching-the-hub)
- small updates in the existing docs to add some examples and consistency

* Revisit documentation part 1: divio-like landing page, installation page, configuration page by Wauplin in 1233
* Updates in docs by Wauplin in 1235
* Doc page: git vs http by Wauplin in 1245
* Minor suggestions post 1245 by osanseviero in 1249
* fix broken urls in docs by Wauplin in 1253
* [Minor doc tweak] *currently... by julien-c in 1256
* Update Repository docstring by Wauplin in 1297
* add link to create_repo by Wauplin in 1299
* Document .no_exist folder by Wauplin in 1308
* Revisit "search the hub" page (+HfApi reference page) by Wauplin in 1303

Test suite for dependent libraries

`huggingface_hub` is getting more and more mature but you might still have some friction if you are maintainer of a library depending on `huggingface_hub`. To help detect breaking changes that would affect third-party libraries, we built a framework to run simple end-to-end tests in our CI. This is still preliminary work but the hope is make `hfh` ecosystem more and more robust over time. Check out [our README](https://github.com/huggingface/huggingface_hub/tree/main/contrib) for more details.

* [RFC] Contrib test suite + tests for timm and sentence_transformers by Wauplin in 1200

Faster download

Goal is to download files faster. First step has been to increase the chunk size by which the files are uploaded. Second step has been to add an optional Rust extension. This is not officially documented for now as we are internally assessing its benefits and limits. Installing and activating `hf_transfer` is purely optional.

* Enabling hf_transfer use. by Narsil in 1272
* Increase the chunk size for faster download by Narsil in 1267

Breaking changes

- `Repository` "clone_from" feature do not create the remote repository if it doesn't exist on the Hub. Please use [`create_repo`](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/hf_api#huggingface_hub.HfApi.create_repo) first before cloning it locally. The `private` init argument has also been removed as it was not needed anymore.
- `allow_regex` and `ignore_regex` have been removed from [`snapshot_download`](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/file_download#huggingface_hub.snapshot_download) in favor `allow_patterns` and `ignore_patterns`.
- git-based implementation of [`push_to_hub_fastai`](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/mixins#huggingface_hub.push_to_hub_fastai) has been removed in favor of the HTTP-based approach. Same for [`ModelHubMixin`](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/mixins#huggingface_hub.ModelHubMixin), [`PyTorchModelHubMixin`](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/mixins#huggingface_hub.PyTorchModelHubMixin), `push_to_hub_keras` and [`KerasModelHubMixin`](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/mixins#huggingface_hub.KerasModelHubMixin).
- `create_repo` is now forced to use keyword-arguments. Same for `metadata_eval_result`.

* Remove deprecated code in v0.12 by Wauplin in 1196

QOL improvements

Not really some features, not really some fixes. But definitely a quality of life improvement for users 🙂
* Progress bar when uploading LFS files by Wauplin in 1266
* tell user how to paste token on Windows by fristhon in 1289
* fixes 1290 - check for existing token during interpreter login by lonnen in 1291
* Store token in configurable HF_HOME (in HfFolder) by Wauplin in 1236
* Compute sha by chunks when uploading by Wauplin in 1296

Internal

CI runs on Windows !

* Add windows machine for CI by Wauplin in 1112

Misc

* Remove obsolete comment by severo in 1265
* fix: authorization header should never forwarded by rtrompier in 1269
* refacto Offline repository tests by Wauplin (direct commit on main)
* Revert "refacto Offline repository tests" by Wauplin (direct commit on main)
* 1278 More robust temporary directory by Wauplin in 1279
* Refacto Repository tests by Wauplin in 1277
* skip flake8 B028 by Wauplin (direct commit on main)

Bugfixes & small improvements

* Fix metadata_update for verified evaluations by lewtun in 1214
* Accept pathlib.Path in upload_file by Wauplin in 1247
* Skip failing tests related to DatasetFilter by Wauplin in 1251
* fix tasks ids test by Wauplin (direct commit on main)
* Fix InferenceAPI on image task by Wauplin in 1270
* Update "move_repo" doc url + fix Bigcode test in CI by Wauplin in 1292
* Fix test_comment_discussion by Wauplin in 1295
* Handle `hf://` urls + raise `ValueError` if repo type is unknown by Wauplin in 1298
* Make ModelSearchArguments and DatasetSearchArguments more robust by Wauplin in 1300
* Make tests more robust (no pre-existing repo on staging Hub) by Wauplin in 1302
* fix: use cache for snapshots even if refs does not exist by brycedrennan Wauplin in 1306
* Fix caching error when offline and no refs dir by Wauplin in 1307

Hot-fix to fix permission issues when downloading with `hf_hub_download` or `snapshot_download`. For more details, see https://github.com/huggingface/huggingface_hub/pull/1220, https://github.com/huggingface/huggingface_hub/issues/1141 and https://github.com/huggingface/huggingface_hub/issues/1215.

**Full changelog:** https://github.com/huggingface/huggingface_hub/compare/v0.11.0...v0.11.1

delete_tag(repo_id, tag="something") If you created a tag by mistake

create_branch(repo_id, branch="experiment-154")
delete_branch(repo_id, branch="experiment-1") Clean some old branches


* Add a `create_tag` method to create tags from the HTTP endpoint by Wauplin in 1089
* Add `delete_tag` method to `HfApi` by Wauplin in 1128
* Create tag twice doesn't work by Wauplin in 1149
* Add "create_branch" and "delete_branch" endpoints by Wauplin 1181

Upload lots of files in a single commit
Making a very large commit was previously tedious. Files are now processed by chunks which makes it possible to upload 25k files in a single commit (and 1Gb payload limitation if uploading only non-LFS files). This should make it easier to upload large datasets.

* Create commit by streaming a ndjson payload (allow lots of file in single commit) by Wauplin in 1117

Delete an entire folder
py
from huggingface_hub import CommitOperationDelete, create_commit, delete_folder

Delete a single folder
delete_folder(repo_id=repo_id, path_in_repo="logs/")

Alternatively, use the low-level `create_commit`
create_commit(
repo_id,
operations=[
CommitOperationDelete(path_in_repo="old_config.json") Delete a file
CommitOperationDelete(path_in_repo="logs/") Delete a folder
],
commit_message=...,
)

* Delete folder with commit endpoint by Wauplin in 1163

Support pagination when listing repos
In the future, listing models, datasets and spaces will be paginated on the Hub by default. To avoid breaking changes, `huggingface_hub` follows already pagination. Output type is currently a list (deprecated), will become a generator in `v0.14`.

* Add support for pagination in list_models list_datasets and list_spaces by Wauplin 1176
* Deprecate output in list_models by Wauplin in 1143

Misc

* Allow create PR against non-main branch by Wauplin in 1168
* 1162 Reorder operations correctly in commit endpoint by Wauplin in 1175

Login, tokens and authentication

Authentication has been revisited to make it as easy as possible for the users.

Unified `login` and `logout` methods

py
from huggingface_hub import login, logout

`login` detects automatically if you are running in a notebook or a script
Launch widgets or TUI accordingly
login()

Now possible to login with a hardcoded token (non-blocking)
login(token="hf_***")

If you want to bypass the auto-detection of `login`
notebook_login() still available
interpreter_login() to login from a script

Logout programmatically
logout()


sh
Still possible to login from CLI
huggingface-cli login


* Unified login/logout methods by Wauplin in 1111

Set token only for a `HfApi` session

py
from huggingface_hub import HfApi

Token will be sent in every request but not stored on machine
api = HfApi(token="hf_***")

* Add token attribute to HfApi by Wauplin in 1116

Stop using `use_auth_token` in favor of `token`, everywhere

`token` parameter can now be passed to every method in `huggingface_hub`. `use_auth_token` is still accepted where it previously existed but the mid-term goal (~6 months) is to deprecate and remove it.

* Replace `use_auth_token` arg by `token` everywhere by Wauplin in 1122

Respect git credential helper from the user

Previously, token was stored in the `git credential store`. Can now be in any helper configured by the user -keychain, cache,...-.

* Refactor git credential handling in login workflow by Wauplin in 1138

Better error handling

Helper to dump machine information

sh
Dump all relevant information. To be used when reporting an issue.
➜ huggingface-cli env

Copy-and-paste the text below in your GitHub issue.

- huggingface_hub version: 0.11.0.dev0
- Platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
...

* 1173 Add dump env helper by Wauplin in 1177

Misc

* Cache not found is not an error by singingwolfboy in 1101
* Propagate error messages when multiple on BadRequest by Wauplin in 1115
* Add error message from `x-error-message` header if exists by Wauplin in 1121

Modelcards

Few improvements/fixes in the modelcard module:

* :art: make repocard content a property by nateraw in 1147
* :white_check_mark: fix content string in repocard tests by nateraw in 1155
* Add Hub verification token to evaluation metadata by lewtun in 1142
* Use default `model_name` in `metadata_update` by lvwerra in 1157
* Refer to modelcard creator app from doc by Wauplin in 1184
* Parent Model --> Finetuned from model by meg-huggingface 1191
* FIX overwriting metadata when both verified and unverified reported values by Wauplin in 1186

Cache assets

New feature to provide a path in the cache where any downstream library can store assets (processed data, files from the web, extracted data, rendered images,...)

* [RFC] Proposal for a way to cache files in downstream libraries by Wauplin in 1088

Documentation updates
* Fixing a typo in the doc. by Narsil in 1113
* Fix docstring of list_datasets by albertvillanova in 1125
* Add repo_type=dataset possibility to guide by Wauplin in 1134
* Fix PyTorch & Keras mixin doc by lewtun in 1139
* Update how-to-manage.mdx by severo in 1150
* Typo fix by meg-huggingface in 1166
* Adds link to model card metadata spec by meg-huggingface in 1171
* Removing "Related Models" & just asking for "Parent Model" by meg-huggingface in 1178

Breaking changes

- Cannot provide an organization to `create_repo`
- `identical_ok` removed in `upload_file`
- Breaking changes in arguments for `validate_preupload_info`, `prepare_commit_payload`, `_upload_lfs_object` (internal helpers for the commit API)
- `huggingface_hub.snapshot_download` is not exposed as a public module anymore

Deprecations
* Remove deprecated code from v0.9, v0.10 and v0.11 by Wauplin in 1092
* Rename languages to langage + remove duplicate code in tests by Wauplin in 1169
* Deprecate output in list_models by Wauplin in 1143
* Set back feature to create a repo when using clone_from by Wauplin in 1187

Internal
* Configure pytest to run on staging by default + flags in config by Wauplin in 1093
* fix search models test by Wauplin in 1106
* Add mypy in the CI (and fix existing type issues) by Wauplin in 1097
* Fix deprecation warnings for assertEquals in tests by Wauplin in 1135
* Skip failing test in ci by Wauplin in 1148
* :green_heart: fix mypy ci by nateraw in 1167
* Update pr docs actions by mishig25 in 1170
* Revert "Update pr docs actions" by mishig25 1192

Bugfixes & small improvements
* Expose list_spaces by osanseviero in 1132
* respect NO_COLOR env var by singingwolfboy in 1103
* Fix list_models bool parameters by Wauplin in 1152
* FIX url encoding in hf_hub_url by Wauplin in 1164
* Fix cannot create pr on foreign repo by Wauplin 1183
* Fix `HfApi.move_repo(...)` and complete tests by Wauplin in 1136
* Commit empty files as regular and warn user by Wauplin in 1180
* Parse file size in get_hf_file_metadata by Wauplin 1179
* Fix get file size on lfs by Wauplin 1188
* More robust create relative symlink in cache by Wauplin in 1109
* Test running CI on Python 3.11 1189

New features and improvements for `HfApi`

`HfApi` is the central point to interact with the Hub API (manage repos, create commits,...). The goal is to propose more and more git-related features using HTTP endpoints to allow users to interact with the Hub without cloning locally a repo.

Create/delete tags and branches
py
from huggingface_hub import create_branch, create_tag, delete_branch, delete_tag

Hot-fix to force utf-8 encoding in modelcards. See https://github.com/huggingface/huggingface_hub/pull/1102 and https://github.com/skops-dev/skops/pull/162#issuecomment-1263516507 for context.

**Full Changelog**: https://github.com/huggingface/huggingface_hub/compare/v0.10.0...v0.10.1

Modelcards

Contribution from nateraw to integrate the work done on Modelcards and DatasetCards (from [nateraw/modelcards](https://github.com/nateraw/modelcards)) directly in `huggingface_hub`.

py
>>> from huggingface_hub import ModelCard

>>> card = ModelCard.load('nateraw/vit-base-beans')
>>> card.data.to_dict()
{'language': 'en', 'license': 'apache-2.0', 'tags': ['generated_from_trainer', 'image-classification'],...}


Related commits

* Add additional repo card utils from `modelcards` repo by nateraw in 940
* Add regression test for empty modelcard update by Wauplin in 1060
* Add template variables to dataset card template by nateraw in 1068
* Further clarifying Model Card sections by meg-huggingface in 1052
* Create modelcard if doesn't exist on `update_metadata` by Wauplin in 1061

Related documentation
- [Creating and Sharing Model Cards](https://huggingface.co/docs/huggingface_hub/main/en/how-to-model-cards)
- [Repository Cards](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/cards)

Cache management (`huggingface-cli scan-cache` and `huggingface-cli delete-cache`)

New commands in `huggingface-cli` to scan and delete parts of the cache. Goal is to manage the cache-system the same way for any dependent library that uses `huggingface_hub`. Only the new cache-system format is supported.

txt
➜ huggingface-cli scan-cache
REPO ID REPO TYPE SIZE ON DISK NB FILES LAST_ACCESSED LAST_MODIFIED REFS LOCAL PATH
--------------------------- --------- ------------ -------- ------------- ------------- ------------------- -------------------------------------------------------------------------
glue dataset 116.3K 15 4 days ago 4 days ago 2.4.0, main, 1.17.0 /home/wauplin/.cache/huggingface/hub/datasets--glue
google/fleurs dataset 64.9M 6 1 week ago 1 week ago refs/pr/1, main /home/wauplin/.cache/
(...)

Done in 0.0s. Scanned 6 repo(s) for a total of 3.4G.
Got 1 warning(s) while scanning. Use -vvv to print details.


Related commits

* Feature: add an utility to scan cache by Wauplin in 990
* Utility to delete revisions by Wauplin in 1035
* 1025 add time details to scan cache by Wauplin in 1045
* Fix scan cache failure when cached snapshot is empty by Wauplin in 1054
* 1025 `huggingface-cli delete-cache` command by Wauplin in 1046
* Sort repos/revisions by age in `delete-cache` by Wauplin in 1063


Related documentation

- [Manage huggingface_hub cache-system](https://huggingface.co/docs/huggingface_hub/main/en/how-to-cache)
- [Cache-system reference](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/cache)

Better error handling (and http-related stuff)

HTTP calls to the Hub have been harmonized to behave the same across the library.

Major differences are:

- Unified way to handle HTTP errors using `hf_raise_for_status` (more informative error message)
- Auth token is always sent by default when a user is logged in (see [documentation](https://huggingface.co/docs/huggingface_hub/main/en/quick-start#login)).
- package versions are sent as user-agent header for telemetry (python, huggingface_hub, tensorflow, torch,...). It was already the case for `hf_hub_download`.

Related commits

* Always send the cached token when user is logged in by Wauplin in 1064
* Add user agent to all requests with huggingface_hub version (and other) by Wauplin in 1075
* [Repository] Add better error message by patrickvonplaten in 993
* Clearer HTTP error messages in `huggingface_hub` by Wauplin in 1019
* Handle backoff on HTTP 503 error when pushing repeatedly by Wauplin in 1038

Breaking changes

1. For consistency, the return type of `create_commit` has been modified. This is a breaking change, but we hope the return type of this method was never used (quite recent and niche output type).

* Return more information in create_commit output by Wauplin in 1066

2. Since `repo_id` is now validated using `validate_hf_hub_args` (see below), a breaking change can be caused if `repo_id` was previously miused. A `HFValidationError` is now raised if `repo_id` is not valid.

Miscellaneous improvements

Add support for autocomplete

* Add autocomplete + tests + type checking by Wauplin in 1041

http-based `push_to_hub_fastai`

* Add changes for push_to_hub_fastai to use the new http-based approach. by nandwalritik in 1040

Check if a file is cached

* try_to_load_from_cache returns cached non-existence by sgugger in 1039

Get file metadata (commit hash, etag, location) without downloading
* Add get_hf_file_metadata to fetch metadata from the Hub by Wauplin in 1058

Validate arguments using `validate_hf_hub_args`
* Add validator for repo id + decorator to validate arguments in `huggingface_hub` by Wauplin in 1029
* Remove repo_id validation in hf_hub_url and hf_hub_download by Wauplin in 1031

:warning: This is a breaking change if `repo_id` was previously misused :warning:

Related documentation:
- [UtilitiesValidators](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/utilities#validators)

Documentation updates
* Fix raise syntax: remove markdown bullet point by mishig25 in 1034
* docs render tree correctly by mishig25 in 1070

Deprecations
* ENH Deprecate clone_from behavior by merveenoyan in 952
* 🗑 Deprecate `token` in read-only methods of `HfApi` in favor of `use_auth_token` by SBrandeis in 928
* Remove legacy helper 'install_lfs_in_userspace' by Wauplin in 1059
* 1055 deprecate private and repo type in repository class by Wauplin in 1057

Bugfixes & small improvements
* Consider empty subfolder as None in hf_hub_url and hf_hub_download by Wauplin in 1021
* enable http request retry under proxy by MrZhengXin in 1022
* Add securityStatus to ModelInfo object with default value None. by Wauplin in 1026
* 👽️ Add size parameter for lfsFiles when committing on the hub by coyotte508 in 1048
* Use `/models/` path for api call to update settings by Wauplin in 1049
* Globally set git credential.helper to `store` in google colab by Wauplin in 1053
* FIX notebook login by Wauplin in 1073

Windows-specific bug fixes
* Fix default cache on windows by thomwolf in 1069
* Degraded but fully working cache-system when symlinks are not supported by Wauplin in 1067
* Check symlinks support per directory instead of globally by Wauplin in 1077