Mlx

Highlights:
- Faster quantized matrix-vector multiplies
- [Benchmarks](https://github.com/ml-explore/mlx/pull/786#issue-2168529706)
- `mx.fast.scaled_dot_product_attention` fused op

Core
- Memory allocation API improvements
- Faster GPU reductions for smaller sizes (between 2 and 7x)
- [Benchmarks](https://github.com/ml-explore/mlx/pull/783#issuecomment-1977460059)
- `mx.fast.scaled_dot_product_attention` fused op
- Faster quantized matrix-vector multiplications
- Pickle support for `mx.array`

NN
- Dilation on convolution layers

Bugfixes
- Fix `mx.topk`
- Fix reshape for zero sizes

Highlights:

- Faster convolutions.
- Up to 14x faster for some common sizes.
- See [benchmarks](https://github.com/ml-explore/mlx/pull/651#issuecomment-1969759609)

Core
- `mx.where` properly handles `inf`
- Faster and more general convolutions
- Input and kernel dilation
- Asymmetric padding
- Support for cross-correlation and convolution
- `atleast_{1,2,3}d` accept any number of arrays


NN
- `nn.Upsample` layer
- Supports nearest neighbor and linear interpolation
- Any number of dimensions

Optimizers
- Linear schedule and schedule joiner:
- Use for e.g. linear warmup + cosine decay

Bugfixes
- `arange` throws on `inf` inputs
- Fix Cmake build with MLX
- Fix `logsumexp` `inf` edge case
- Fix grad of power w.r.t. to exponent edge case
- Fix compile with `inf` constants
- Bug temporary bug in convolution

Highlights:

- Partial shapeless compilation
- Default shapeless compilation for all activations
- Can be more than 5x faster than uncompiled versions
- CPU kernel fusion
- Some functions can be up to [10x faster](https://github.com/ml-explore/mlx/pull/691#issuecomment-1947708207)

Core

- CPU compilation
- Shapeless compilation for some cases
- `mx.compile(function, shapeless=True)`
- Up to 10x faster scatter: [benchmarks](https://github.com/ml-explore/mlx/pull/709#issuecomment-1951733610)
- `mx.atleast_1d`, `mx.atleast_2d`, `mx.atleast_3d`

Bugfixes

- Bug with `tolist` with `bfloat16` and `float16`
- Bug with `argmax` on M3

Highlights:

- `mx.fast` subpackage
- Custom `mx.fast.rope` up to [20x faster](https://github.com/ml-explore/mlx/pull/676#issuecomment-1940672929)

Core

- Support metadata with `safetensors`
- Up to 5x faster scatter and 30% faster gather
- 40% faster `bfloat16` quantizated matrix-vector multiplies
- `mx.fast` subpackage with a fast RoPE
- Context manager `mx.stream` to set the default device

NN

- Average and Max pooling layers for 1D and 2D inputs

Optimizers

- Support schedulers for e.g. learning rates
- A few basic schedulers:
- `optimizers.step_decay`
- `optimizers.cosine_decay`
- `opimtizers.exponential_decay`

Bugfixes

- Fix bug in remainder with negative numerators and integers
- Fix bug with slicing into softmax
- Fix quantized matmuls with non 32 multiples

Highlights:

- `mx.compile` **makes stuff go fast**
- Some functions are up to 10x faster ([benchmarks](https://github.com/ml-explore/mlx/pull/614#issuecomment-1929056286))
- Training models anywhere from 10% to twice as fast ([benchmarks](https://github.com/ml-explore/mlx-examples/pull/420#issuecomment-1932422605))
- Simple syntax for compiling full training steps

Core

- `mx.compile` function transformation
- Find devices properly for iOS
- Up to 10x faster GPU gather
- `__abs__` overload for `abs` on arrays
- `loc` and `scale` in parameter for `mx.random.normal`

NN

- Margin ranking loss
- BCE loss with weights

Bugfixes

- Fix for broken eval during function transformations
- Fix `mx.var` to give `inf` with `doff >= nelem`
- Fix loading empty modules in `nn.Sequential`

Highlights

- Memory use improvements:
- Gradient checkpointing for training with `mx.checkpoint`
- Better graph execution order
- Buffer donation

Core

- Gradient checkpointing with `mx.checkpoint`
- CPU only QR factorization `mx.linalg.qr`
- Release Python GIL during `mx.eval`
- Depth-based graph execution order
- Lazy loading arrays from files
- Buffer donation for reduced memory use
- `mx.diag`, `mx.diagonal`
- Breaking: `array.shape` is a Python tuple
- GPU support for `int64` and `uint64` reductions
- vmap over reductions and arg reduction:
- `sum`, `prod`, `max`, `min`, `all`, `any`
- `argmax`, `argmin`

NN

- Softshrink activation

Bugfixes

- Comparisons with `inf` work, and fix `mx.isinf`
- Bug fix with RoPE cache
- Handle empty Matmul on the CPU
- Negative shape checking for `mx.full`
- Correctly propagate `NaN` in some binary ops
- `mx.logaddexp`, `mx.maximum`, `mx.minimum`
- Fix > 4D non-contiguous binary ops
- Fix `mx.log1p` with `inf` input
- Fix SGD to apply weight decay even with 0 momentum