Huggingface-hub

What's Changed

* Fix typing-extensions minimum version by lhoestq in https://github.com/huggingface/huggingface_hub/pull/453
* Fix argument order in `create_repo` for `Repository.clone_from` by sgugger in https://github.com/huggingface/huggingface_hub/pull/459


**Full Changelog**: https://github.com/huggingface/huggingface_hub/compare/v0.1.0...v0.1.1

What's Changed

Version v0.1.0 is the first minor release of the `huggingface_hub` package, which promises better stability for the incoming versions. This update comes with big quality of life improvements.

Make token optional in all HfApi methods. by sgugger in https://github.com/huggingface/huggingface_hub/pull/379

Previously, most methods of the `HfApi` class required the token to be explicitly passed. This is changed in this version, where it defaults to the token stored in the cache. This results in a re-ordering of arguments, but backward compatibility is preserved in most cases. Where it is not preserved, an explicit error is thrown.

Root methods instead of `HfApi` by LysandreJik in https://github.com/huggingface/huggingface_hub/pull/388

The `HfApi` class now exposes its methods through the `hf_api` file, reducing the friction to access these helpers. See the example below:

py
Previously
from huggingface_hub import HfApi

api = HfApi()
user = api.whoami()

Now
from huggingface_hub.hf_api import whoami

user = whoami()


The `HfApi` can still be imported and works as before for backward compatibility.

Add `list_repo_files` util by sgugger in https://github.com/huggingface/huggingface_hub/pull/395

Offers a `list_repo_files` to ... list the repo files! Supports both model repositories and dataset repositories

Add helper to generate an eval result `model-index`, with proper typing by julien-c in https://github.com/huggingface/huggingface_hub/pull/382

Offers a `metadata_eval_result` in order to generate a YAML block to put in model cards according to evaluation results.

Add metrics to API by mariosasko in https://github.com/huggingface/huggingface_hub/pull/429

Adds a list_metrics method to HfApi!

Git prune by LysandreJik in https://github.com/huggingface/huggingface_hub/pull/450

Adds a `git_prune` method to the `Repository` class. This prunes local files which are unneeded as already pushed to a remote.
It adds the argument `auto_lfs_prune` to `git_push` and the `commit` context-manager for simpler handling.

Bug fixes
* Fix HfApi.create_repo when repo_type is 'space' by nateraw in https://github.com/huggingface/huggingface_hub/pull/394
* Last fixes for `datasets`' `push_to_hub` method by LysandreJik in https://github.com/huggingface/huggingface_hub/pull/415


**Full Changelog**: https://github.com/huggingface/huggingface_hub/compare/v0.0.19...v0.1.0

Repository metadata (julien-c)

The version v0.0.18 of the `huggingface_hub` includes tools to manage repository metadata. The following example reads metadata from a repository:

python
from huggingface_hub import Repository

repo = Repository("xxx", clone_from="yyy")
data = repo.repocard_metadata_load()


The following example completes that metadata before writing it to the repository locally.

python
data["license"] = "apache-2.0"
repo.repocard_metadata_save(data)


- Repo metadata load and save 339 (julien-c)

Git tags (AngledLuffa)

Tag management is now available! Add, check, delete tags locally or remotely directly from the `Repository` utility.

- Tags 323 (AngledLuffa)

Revisited Keras support (nateraw)

The Keras mixin has been revisited:
- It now saves models as `SavedModel` objects rather than `.h5` files.
- It now offers methods that can be leveraged simply as a functional API, instead of having to use the Mixin as an actual mixin.

Improvements and bug fixes

- Better error message for bad token. 362 (sgugger)
- Add utility to get repo name 364 (sgugger)
- Improve save and load repocard metadata 355 (elishowk)
- Update Keras Mixin 284 (nateraw)
- Add timeout to dataset_info 373 (lhoestq)

Non-blocking git-push

The pushing methods now have access to a `blocking` boolean parameter to indicate whether the push should happen
asynchronously.

In order to see if the push has finished or its status code (to spot a failure), one should use the `command_queue`
property on the `Repository` object.

For example:

python
from huggingface_hub import Repository

repo = Repository("<local_folder>", clone_from="<user>/<model_name>")

with repo.commit("Commit message", blocking=False):
Save data

last_command = repo.command_queue[-1]

Status of the push command
last_command.status
Will return the status code
-> -1 will indicate the push is still ongoing
-> 0 will indicate the push has completed successfully
-> non-zero code indicates the error code if there was an error

if there was an error, the stderr may be inspected
last_command.stderr

Whether the command finished or if it is still ongoing
last_command.is_done

Whether the command errored-out.
last_command.failed


When using `blocking=False`, the commands will be tracked and your script will exit only when all pushes are done, even
if other errors happen in your script (a failed push counts as done).

- Non blocking git push 315 (LysandreJik)

Notebook login (sgugger)

The `huggingface_hub` library now has a `notebook_login` method which can be used to login on notebooks with no access to the shell. In a notebook, login with the following:

py
from huggingface_hub import notebook_login

notebook_login()


- Add a widget to login in notebook 329 (sgugger)

Improvements and bugfixes

- added option to create private repo 319 (philschmid)
- display git push warnings 326 (elishowk)
- Allow specifying data with the Inference API wrapper 271 (osanseviero)
- Add auth to snapshot download 340 (lewtun)

The `huggingface_hub` version v0.0.16 introduces several quality of life improvements.

Progress bars in `Repository`

Progress bars are now visible with many git operations, such as pulling, cloning and pushing:

python
>>> from huggingface_hub import Repository
>>> repo = Repository("local_folder", clone_from="huggingface/CodeBERTa-small-v1")


Cloning https://huggingface.co/huggingface/CodeBERTa-small-v1 into local empty directory.
Download file pytorch_model.bin: 45%|████████████████████████████▋ | 144M/321M [00:13<00:12, 14.7MB/s]
Download file flax_model.msgpack: 42%|██████████████████████████▌ | 134M/319M [00:13<00:13, 14.4MB/s]

Branching support

There is now branching support in `Repository`. This will clone the `xxx` repository and checkout the `new-branch` revision. If it is an existing branch on the remote, it will checkout that branch. If it is another revision, such as a commit or a tag, it will also checkout that revision.

If the revision does not exist, it will create a branch from the latest commit on the `main` branch.

py
>>> from huggingface_hub import Repository
>>> repo = Repository("local", clone_from="xxx", revision="new-branch")


Once the repository is instantiated, it is possible to manually checkout revisions using the `git_checkout` method. If the revision already exists:

>>> repo.git_checkout("main")

If a branch should be created from the current head in the case that it does not exist:
py
>>> repo.git_checkout("brand-new-branch", create_branch_ok=True)


Revision `brand-new-branch` does not exist. Created and checked out branch `brand-new-branch`


Finally, the `commit` context manager has a new `branch` parameter to specify to which branch the utility should push:

py
>>> with repo.commit("New commit on branch brand-new-branch", branch="brand-new-branch"):
... Save any file or model here, it will be committed to that branch.
... torch.save(model.state_dict())


Git credentials

The login system has been redesigned to leverage `git-credential` instead of a token-based authentication system. It leverages the `git-credential store` helper. If you're unaware of what this is, you may see the following when logging in with `huggingface_hub`:

_| _| _| _| _|_|_| _|_|_| _|_|_| _| _| _|_|_| _|_|_|_| _|_| _|_|_| _|_|_|_|
_| _| _| _| _| _| _| _|_| _| _| _| _| _| _| _|
_|_|_|_| _| _| _| _|_| _| _|_| _| _| _| _| _| _|_| _|_|_| _|_|_|_| _| _|_|_|
_| _| _| _| _| _| _| _| _| _| _|_| _| _| _| _| _| _| _|
_| _| _|_| _|_|_| _|_|_| _|_|_| _| _| _|_|_| _| _| _| _|_|_| _|_|_|_|


Username:
Password:
Login successful
Your token has been saved to /root/.huggingface/token
Authenticated through git-crendential store but this isn't the helper defined on your machine.
You will have to re-authenticate when pushing to the Hugging Face Hub. Run the following command in your terminal to set it as the default

git config --global credential.helper store


Running the command `git config --global credential.helper store` will set this as the default way to handle credentials for git authentication. All repositories instantiated with the `Repository` utility will have this helper set by default, so no action is required from your part when leveraging it.

Improved logging

The logging system is now similar to the existing logging system in `transformers` and `datasets`, based on a `logging` module that controls the entire library's logging level:

py
>>> from huggingface_hub import logging
>>> logging.set_verbosity_error()
>>> logging.set_verbosity_info()


Bug fixes and improvements

- Add documentation to GitHub and the Hub docs about the Inference client wrapper 253 (osanseviero)
- Have large files enabled by default when using `Repository` 219 (LysandreJik)
- Clarify/specify/document model card metadata, `model-index`, and pipeline/task types 265 (julien-c)
- [model_card][metadata] Actually, lets make dataset.name required 267 (julien-c)
- Progress bars 261 (LysandreJik)
- Add keras mixin 230 (nateraw)
- Open source code related to the repo type (tag icon, display order, snippets) 273 (osanseviero)
- Branch push to hub 276 (LysandreJik)
- Git credentials 277 (LysandreJik)
- Push to hub/commit with branches 282 (LysandreJik)
- Better logging 262 (LysandreJik)
- Remove custom language pack behavior 291 (LysandreJik)
- Update Hub and huggingface_hub docs 293 (osanseviero)
- Adding a handler 292 (LysandreJik)

Improvements and bugfixes
- [Docs] Update link to Gradio documentation 206 (abidlabs)
- Fix title typo (Cliet -> Client) 207 (cakiki)
- add _from_pretrained hook 159 (nateraw)
- Add `filename` option to `lfs_track` 212 (LysandreJik)
- Repository fixes 213 (LysandreJik)
- Repository documentation 214 (LysandreJik)
- Add datasets filtering and sorting 194 (lhoestq)
- doc: sync github to spaces 221 (borisdayma)
- added batch transform documentation & model archive documentation 224 (philschmid)
- Sync with hf internal 228 (mishig25)
- Adding batching support for superb 215 (Narsil)
- Adding SD for superb (speech-classification). 225 (Narsil)
- Use Hugging Face fork for s3prl 229 (lewtun)
- Mv `interfaces` -> `widgets/lib/interfaces` 227 (mishig25)
- Tweak to prevent accidental sharing of token 226 (julien-c)
- Fix CLI-based repo creation 234 (osanseviero)
- Add proxify util function 235 (mishig25)