This new release brings ~400 new commits with improvements all over the place: docs, features, fixes... QA-Board is used in many projects at Samsung, and we don't have plans to break backward compatibility. Therefore we feel ready to call it this version `v1`.
> If you run into issues with QA-Board, or want to know more about its use-cases, know it's actively developed. Don't hesitate to contact [me](arthur.flamsamsung.com).
New Features
- **Display run badges** to e.g. show inference results in QA-Board and link to model training logs
- **Compare versus runs from different projects**
- **Soft-deletes** to keep run metadata but remove all/some files
- **Parallel sampling for auto-tuning**
- **User accounts**, required to trigger experiments from the app.
- **Per-user custom tests** in the web application.
- **Move runs** to an existing or new batch
- **Built-in distributed workers** with `celery`, and monitoring with [`flower`](https://flower.readthedocs.io/en/latest/).
- **Redo**: all a batch's run, or just failed ones, or per-run... and with the same parameters as the original.
- **Usage outside of git repos**: support `GIT_MESSAGE` / `GIT_AUTHORED_DATETIME` / `GIT_COMMIT` / `GIT_BRANCH` in ENV.
- Support for default `database` per _batches.yaml_ file
- Use indexing when using matrix configurations, e.g. `${matrix.value[0]}`
- **Input manifests** support, and a `--strict` mode in bit-accuracy checks that forbids unexpected output files
- distributed workers via Jenkins
- **Custom bit-accuracy `compare` function** to e.g. ignore metadata
- More flexible _per-input_ configurations in batches
- Can use globs to define in _qaboard.yaml_ `inputs.batches`. In this case it can be helpful to enable the optional flag `allow_duplicate_batches`.
- Internal: for "user-side" DAG/pipelines make it possible to call e.g. `CI=true GIT_COMMIT=xxxxx qa batch --action-on-existing=assert-exists`
UX improvements
- Tons of improvements to the logs, status indicators and error messages. Both in the CLI and web application.
- We tried to make the defaults of everything (UI, config...) more natural.
- **Tuning analysis / tradeoffs** tab enabled with runs started on the CLI, not just with experiments from the app
- **UI:** layout improvements in many places
- **Image viewer:** No animations to make the diff feel smoother, tooltip with real image values, hide the reference if same as new. Allow ref images with different dimensions, and behave nicely.
- **File viewer icons**
- **Filter list of projects**
- **Scroll to bottom** button in logs
- **`target_passfail` metrics** to show in green/red instead of a gradient
- Text viewer: toogle side-by-side / inline
- Indicator when identical files are compared
- **Filter milestones** in the selection menu. Direct links to milestones in the project sidebar.
- Remove columns without metrics from the table view
- Mark pending runs as finished from the UI
- Highlight commit that have milestones and prevent deleting them
- Avoid recoloring plotly plots if we don't overlay new/reference
- _...and many minor things_
Performance
- Support for custom `postgresql` configuration
- Removed re-renders and overhead in the web application
- Transparent pre-compressed gzip files (`.gz`) to limit storage usage with custom viewers
- Memoize string edit distances
Fixes
- Many minor issues all over the place...
- Gitlab not needed anymore. But if used, added a proxy to get project avatars, and support triggering manual jobs in CI pipelines with tons of jobs...
- hardcoded ports, URL...
- Don't _rely_ on conventions for the output directory locations
- Better uswgi setup: no startup failures, "cheaper" algo, set queue size...
- Fix imports when not running under a QA-Board directory
Upgrades
- **Upgraded dependencies** of the web application and backend
- Website updated to Docusaurus v2