Summary
This release is the culmination of a month of work in the `refresh` branch to re-architect TurnkeyML for the future.
The high-level changes are:
- `Stages` have been replaced by `Tools`, which each have their own arguments parser to enable maximum configurability
- `Sequences` are no longer hardcoded lists of `Stages` - now users can enter any list of `Tools` they like
- Everything is a `Tool`: model discovery, benchmarking, and even cache management functionality (report, list, delete, etc.) are all now modular `Tools`.
- `Plugins` now primarily add new `Tools`. These new `Tools` can do a lot more than `Stages` used to be able to, such as:
- Add a new frontend into the sequence, as a replacement for model discovery. For example, load an LLM checkpoint.
- Replace the backend of the sequence with something other than our standard benchmarking flow. For example, run an accuracy test.
- Add utility functions, such as managing the cache or built-in models.
- There is a brand new command line interface and API to expose the new way of doing things.
- The tools documentation has been completely re-written.
Breaking Changes
There are almost too many to list since the whole project has been re-architected. Here are the major things to be aware of:
- `Stage` classes need to be ported to on to the `Tool` class.
- Any reference to `stage` should be changed to `tool`, for example the `stage_status::` stat is now `tool_status::`
- The names of most `Tools` have been renamed to fit our new naming scheme (see contribution guide).
- There is no longer a notion of `benchmark_status`, since benchmarking is now a `Tool`. It's now `tool_status::benchmark`.
- The `build_model()` API has been removed. Instead, `Tools` should be called as their own standalone APIs.
- The `benchmark_files()` API has been re-named to `evaluate_files()`.
- The repository directory structure has been redone, which changes a lot of imports.
- For example, `turnkeyml.state` has `State` and `load_state()`.
- See the code organization doc for the full set of changes.