This release addresses incompatibilities related to saving and loading models across a range of `transformers` versions following changes in `transformers` v4.31.
* Extend support to transformers v4.33 (402).
* Add fallback to load `state_dict` with `strict=False`, due to incompatibilities related to `state_dict` keys between `transformers` v4.30 and v4.31 (398).
Options for improved pipeline package compatibility
If you are packaging models with `spacy package` and you'd like to avoid having your users run into the fallback `strict=False` loading and warnings related to `state_dict` keys, you may want to add a stricter `transformers` requirement that corresponds to the version used during training.
If you're training with the newest supported `transformers` version (v4.33.x), you could currently add `transformers>=4.31` to your package requirements. Note that this would restrict your users to Python 3.8+ and PyTorch 1.9+ due to `transformers` requirements.
Create `meta.json` with your additional requirements:
json
{
"requirements": ["transformers>=4.31"]
}
Package with [`spacy package`](https://spacy.io/api/cli#package):
shell
spacy package --meta meta.json input_dir output_dir
The requirement `transformers>=4.31` will be added to any other package requirements that are automatically determined based on your pipeline config.