Cappr

Breaking changes

None

New features

None

Bug fixes

* [`cappr.huggingface`](https://cappr.readthedocs.io/en/latest/cappr.huggingface.classify.html) is compatible with Llama 3/3.1's tokenizer. It works around [this issue](https://github.com/huggingface/transformers/issues/30947) using code from [this PR](https://github.com/huggingface/transformers/pull/31316) (with small modifications). See the updated list of supported architectures [here](https://cappr.readthedocs.io/en/latest/select_a_language_model.html#hugging-face).

Breaking changes

None

New features

* [`cappr.huggingface.classify`](https://cappr.readthedocs.io/en/latest/cappr.huggingface.classify.html) internally passes in a [`DynamicCache`](https://huggingface.co/docs/transformers/main/en/kv_cache) object if possible. This change gets rid of a warning you might see when running previous versions of CAPPr:


We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)


Bug fixes

None

Breaking changes

None

New features

* [`cappr.huggingface.classify`](https://cappr.readthedocs.io/en/latest/cappr.huggingface.classify.html) doesn't copy the prompt's KVs when broadcasting the prompt to completions if `batch_size=1` or if you pass in a single prompt. Instead, it repeats a view of it. This change saves memory for tasks where there are many completions. For example, in the [Banking 77 demo](https://github.com/kddubey/cappr/blob/main/demos/huggingface/banking_77_classes.ipynb), peak reserved CUDA memory goes from 13.8 GB to 8.3 GB (~40% decrease), and peak allocated CUDA memory goes from 9.3 GB to 7.7 GB (~17% decrease).

Bug fixes

None

Breaking changes

None

New features

* The [`agg_log_probs`](https://cappr.readthedocs.io/en/latest/cappr.utils.classify.html#cappr.utils.classify.agg_log_probs) function returns a numpy array instead of a list of numpy arrays if there are a constant number of completions.

Bug fixes

None

Breaking changes

None

New features

* The [`agg_log_probs`](https://cappr.readthedocs.io/en/latest/cappr.utils.classify.html#cappr.utils.classify.agg_log_probs) function applies `func` instead of `np.exp ∘ func`. So if you want average token log-probabilities, set `func=np.mean`

Bug fixes

None

Breaking changes

* There's no `setup.py` file, in case you were relying on that.

New features

None

Bug fixes

None