Huggingface-hub

py
from huggingface_hub import Repository

repo = Repository("local_directory", clone_from="<user>/<model_id>")

save large files in `local_directory`
repo.git_add()
repo.auto_track_large_files()
repo.git_commit("Add large files")
repo.git_push()
No push rejected error anymore!


It is automatically used when leveraging the `commit` context manager:

py
from huggingface_hub import Repository

repo = Repository("local_directory", clone_from="<user>/<model_id>")
with repo.commit("Add large files"):
add large files

No push rejected error anymore!


- Auto track with LFS 177 (LysandreJik)

Documentation

- Update docs structure 145 (Pierrci)
- Update links to docs 147 (LysandreJik)
- Add new repo guide 153 (osanseviero)
- Add documentation for endpoints 155 (osanseviero)
- Document hf.co webhook publicly 156 (julien-c)
- docs: ✏️ mention the Training metrics tab 193 (severo)
- doc for Spaces 189 (julien-c)

Breaking changes

Reminder: the `huggingface_hub` library follows semantic versioning and is undergoing active development. While the first major version is not out (v1.0.0), you should expect breaking changes and we strongly recommend pinning the library to a specific version.

Two breaking changes are introduced with version v0.0.14.

The `whoami` return changes from a tuple to a dictionary

- Allow obtaining Inference API tokens with whoami 157 (osanseviero)

The `whoami` method changes its returned value from a tuple of `(<user>, [<organisations>])` to a dictionary containing a lot more information:

In versions v0.0.13 and below, here was the behavior of the `whoami` method from the `HfApi` class:
py
from huggingface_hub import HfFolder, HfApi
api = HfApi()
api.whoami(HfFolder.get_token())
('<user>', ['<org_0>', '<org_1>'])


In version v0.0.14, this is updated to the following:

py
from huggingface_hub import HfFolder, HfApi
api = HfApi()
api.whoami(HfFolder.get_token())
{
'type': str,
'name': str,
'fullname': str,
'email': str,
'emailVerified': bool,
'apiToken': str,
`plan': str,
'avatarUrl': str,
'orgs': List[str]
}


The `Repository`'s `use_auth_token` initialization parameter now defaults to `True`.

The `use_auth_token` initialization parameter of the `Repository` class now defaults to `True`. The behavior is unchanged if users are not logged in, at which point `Repository` remains agnostic to the `huggingface_hub`.

- Set use_auth_token to True by default 204 (LysandreJik)


Improvements and bugfixes

- Add sklearn code snippet 133 (osanseviero)
- Allow passing only model ID to clone when authenticated 150 (LysandreJik)
- More robust endpoint with toggled staging endpoint 148 (LysandreJik)
- Add config to list_models 152 (osanseviero)
- Fix audio-to-audio widget and add icon 142 (osanseviero)
- Upgrade spaCy to api 0.0.12 and remove allowlist 161 (osanseviero)
- docs: fix webhook response format 162 (severo)
- Update link in README.md 163 (nateraw)
- Revert "docs: fix webhook response format (162)" 165 (severo)
- Add Keras docker image 117 (osanseviero)
- Allow multiple models when testing a pipeline 124 (osanseviero)
- scikit rebased 170 (Narsil)
- Upgrading community frameworks to `audio-to-audio`. 94 (Narsil)
- Add sagemaker docs 173 (philschmid)
- Add Structured Data Classification as task 172 (osanseviero)
- Fixing keras outputs (widgets was ignoring because of type mismatch, now testing for it) 176 (Narsil)
- Updating spacy. 179 (Narsil)
- Create initial superb docker image structure 181 (osanseviero)
- Upgrading asteroid image. 175 (Narsil)
- Removing tests on huggingface_hub for unrelated changes in api-inference-community 180 (Narsil)
- Fixing audio-to-audio validation. 184 (Narsil)
- `rmdir api-inference-community/src/sentence-transformers` 188 (Pierrci)
- Allow generic inference for ASR for superb 185 (osanseviero)
- Add timestamp to snapshot download tests 201 (LysandreJik)
- No need for token to understand HF urls 203 (LysandreJik)
- Remove `--no_renames` argument to list deleted files. 205 (LysandreJik)

Version 0.0.13 introduces a context manager to save files directly to the Hub. See below for some examples.

Example with a single file

py
from huggingface_hub import Repository

repo = Repository("text-files", clone_from="<user>/text-files", use_auth_token=True)

with repo.commit("My first file."):
with open("file.txt", "w+") as f:
f.write(json.dumps({"key": "value"}))


Example with a `torch.save` statement:

py
import torch
from huggingface_hub import Repository

model = torch.nn.Transformer()

repo = Repository("torch-files", clone_from="<user>/torch-files", use_auth_token=True)

with repo.commit("Adding my cool model!"):
torch.save(model.state_dict(), "model.pt")


Example with a Flax/JAX seralization statement

py
from flax import serialization
from jax import random
from flax import linen as nn
from huggingface_hub import Repository

model = nn.Dense(features=5)

key1, key2 = random.split(random.PRNGKey(0))
x = random.normal(key1, (10,))
params = model.init(key2, x)

bytes_output = serialization.to_bytes(params)

repo = Repository("flax-model", clone_from="<user>/flax-model", use_auth_token=True)

with repo.commit("Adding my cool Flax model!"):
with open("flax_model.msgpack", "wb") as f:
f.write(bytes_output)

Patches an issue when cloning a repository twice.

Improved documentation

The `huggingface_hub` documentation is now available on [hf.co/docs](https://hf.co/docs)! Additionally, a new step-by-step guide to adding libraries is available.

- New documentation for 🤗 Hub 71 (osanseviero)
- Step by step guide on adding Model Hub support to libraries 86 (LysandreJik)

New method: `hf_hub_download`

A new method is introduced: `hf_hub_download`. It is the equivalent of doing `cached_download(hf_hub_url())`, in a single method.

- HF Hub download 137 (LysandreJik)

`Repository` power-up

The `Repository` class is updated to behave more similarly to git. It is now impossible to clone a repository in a folder that already contains files.

The PyTorch Mixin contributed by vasudevgupta7 is slightly updated to have the `push_to_hub` method manage a repository as one would from the command line.

- Repository power-up 132 (LysandreJik)

Improvement & Fixes

- Adding `audio-to-audio` task. 93 (Narsil)
- When pipelines fail to load in framework code, for whatever reason 96 (Narsil)
- Solve `rmtree` issue on windows 105 (SBrandeis)
- Add identical_ok option to HfApi.upload_file method 102 (SBrandeis)
- Solve compatibility issues when calling `subprocess.run` 104 (SBrandeis)
- Open source Inference widgets + optimize for community contributions 87 (julien-c)
- model `tags` can be `undefined` 107 (Pierrci)
- Doc tweaks 109 (julien-c)
- [huggingface_hub] Support for spaces 108 (julien-c)
- speechbrain library tag + code snippet 73 (osanseviero)
- Allow batching for feature-extraction 106 (osanseviero)
- adding audio-to-audio widget. 95 (Narsil)
- Add image to text (for image captioning) 114 (osanseviero)
- Add formatting and upgrade Sentence Transformers api version for better error messages 119 (osanseviero)
- Change videos in docs so they are played directly in our site 120 (osanseviero)
- Fix inference API GitHub actions 125 (osanseviero)
- Fixing sentence-transformers CACHE value for docker + functools (docker needs Py3.8) 123 (Narsil)
- Load errors with flair should now be generating proper API errors. 121 (Narsil)
- Simplify manage to autodetect task+framework if possible. 122 (Narsil)
- Change sentence transformers source to original repo 128 (osanseviero)
- Allow Python versions with letters in the minor version suffix 82 (ulf1)
- Update `upload_file` docs 136 (LysandreJik)
- Reformat repo README 130 (osanseviero)
- Add config to model info 135 (osanseviero)
- Add input validation for structured-data-classification 97 (osanseviero)

v0.0.10 Signs the merging of three components of the HuggingFace stack: the `huggingface_hub` repository is now the central platform to contribute new libraries to be supported on the hub.

It regroups three previously separated components:

- The [`huggingface_hub` Python library](https://github.com/huggingface/huggingface_hub/tree/main/src/huggingface_hub), as the Python library to download, upload, and retrieve information from the hub.
- The [`api-inference-community`](https://github.com/huggingface/huggingface_hub/tree/main/api-inference-community), as the platform where libraries wishing for hub support may be added.
- The [`interfaces`](https://github.com/huggingface/huggingface_hub/tree/main/interfaces), as the definition for pipeline types as well as default widget inputs and definitions/UI elements for third-party libraries.

Future efforts will be focused on further easing contributing third-party libraries to the Hugging Face Hub

Improvement & Fixes

- Add typing extensions to conda yaml file 49 (LysandreJik)
- Alignment on modelcard metadata specification 39 (LysandreJik)
- Bring interfaces from `widgets-server` 50 (julien-c)
- Sentence similarity default widget and pipeline type 52 (osanseviero)
- [interfaces] Expose configuration options for external libraries 51 (julien-c)
- Adding `api-inference-community` to `huggingface_hub`. 48 (Narsil)
- Add TensorFlowTTS as library + code snippet 55 (osanseviero)
- Add protobuf as a dependency to handle tokenizers that require it: 58 (Narsil)
- Update validation for NLP tasks 59 (osanseviero)
- spaCy code snippet and language tag 57 (osanseviero)
- SpaCy fixes 60 (osanseviero)
- Allow changing repo visibility programmatically 61 (osanseviero)
- Add Adapter Transformers snippet 62 (osanseviero)
- Change order in spaCy snippet 66 (osanseviero)
- Add validation to check all rows in table question answering have same length 67 (osanseviero)
- added question-answering part for Bengali language 68 (sagorbrur)
- Add spaCy to inference API 63 (osanseviero)
- AllenNLP library tag + code snippet 72 (osanseviero)
- Fix AllenNLP QA example 80 (epwalsh)
- do not crash even if this config isn't set 81 (julien-c)
- Mark model config as optional 83 (Pierrci)
- Add repr() to ModelFile and RepoObj 75 (lewtun)
- Refactor create_repo 84 (SBrandeis)

Support for large file uploads

Implementation of an endpoint to programmatically upload (large) files to any repo on the hub, without the need for git, using HTTP POST requests.

- [API] Support for the file upload endpoint 42 (SBrandeis)

The `HfApi.model_list` method now allows multiple filters

Models may now be filtered using several filters:

py
Example usage:

>>> from huggingface_hub import HfApi
>>> api = HfApi()

>>> List all models
>>> api.list_models()

>>> List only the text classification models
>>> api.list_models(filter="text-classification")

>>> List only the russian models compatible with pytorch
>>> api.list_models(filter=("ru", "pytorch"))

>>> List only the models trained on the "common_voice" dataset
>>> api.list_models(filter="dataset:common_voice")

>>> List only the models from the AllenNLP library
>>> api.list_models(filter="allennlp")


- Document the `filter` argument 41 (LysandreJik)

`ModelInfo` now has a readable representation

Improvement of the `ModelInfo` class so that it displays information about the object.

- Include a readable repr for ModelInfo 32 (muellerzr)

Improvements and bugfixes

- Fix conda by specifying python version + add tests to main branch 28 (LysandreJik)
- Improve Mixin 34 (LysandreJik)
- Enable `library_name` and `library_version` in `snapshot_download` 38 (LysandreJik)
- [Windows support] Very long filenames 40 (LysandreJik)
- Make error message more verbose when creating a repo 44 (osanseviero)
- Open-source /docs 46 (julien-c)