Pioreactor

Enhancements
- **New faster ADC firmware with less noise**
- Upgraded ADC firmware improves signal processing speed and reduces measurement noise, leading to more reliable readings for all sensors.
- **`led_intensity` is now registered in our database**
- This means that running `pio kill --all-jobs` (and related `pio kill` commands) will now also turn off all LEDs, ensuring a complete shutdown of active processes.
- **New option in `pio workers add` to specify an IPv4 address**
- When adding a new worker, you can now explicitly provide an IPv4 address instead of relying on the default `hostname.local`. This is useful in networks where mDNS resolution is unreliable or unavailable. Ex: `pio workers add <name> -a 192.168.0.3`
- **New time option on the Overview page: "Now" for only real-time data**
- The UI now has a “Now” option that filters out historical data, displaying only real-time sensor readings and status updates.
- **Logs for experiment profiles now include an action step number**
- Each log entry related to an experiment profile now contains a step number, making it easier to track progress and diagnose issues in multi-step workflows.
- **Improved outlier detection in nOD and growth rates**
- Our outlier detection algorithms for normal optical density (nOD) and growth rates have been refined, reducing false positives and improving tracking accuracy during experiments.
- **Changing OD readings interval programmatically**
- OD Reading job now exposes `interval` as a editable published_setting. For example, you can PATCH to `http://pioreactor.local/api/workers/<pioreactor_unit>/jobs/update/job_name/od_reading/experiments/<experiment>` with body:

{
"settings": {
"interval": 10
},
}

to change OD readings interval to 10s.

Breaking Changes
- **`id` → `job_id` in `pio_metadata_settings` table**
- Database schema update: The primary identifier column in `pio_metadata_settings` has been renamed to `job_id`.
- **Changed the scaling of `smoothing_penalizer`**
- The `smoothing_penalizer` parameter now operates on a scale that is **~100x lower** than before.
- **Deprecation of `/unit_api/jobs/stop/...` endpoint**
- The `/unit_api/jobs/stop/...` API endpoint is being deprecated in favor of using query parameters:
- Instead of `/unit_api/jobs/stop/job_name`, use `/unit_api/jobs/stop/?job_name=...`.
- The special case `/unit_api/jobs/stop/all` remains valid and unchanged.
- **Timestamp precision change: From `xxxxx` milliseconds → `xxx` milliseconds**
- All timestamps will now be stored with three-digit millisecond precision instead of five. This change optimizes storage efficiency and speeds up queries while maintaining sufficient accuracy for most use cases.

Bug Fixes
- **Fix for "More" button in the Logs UI page**
- Previously, clicking "More" in the Logs UI would default to "Standard" log level instead of retaining the selected filter. Now, it correctly uses the log level chosen by the user.
- **Multiple experiment profiles no longer overwrite each other in MQTT**
- Previously, running multiple experiment profiles could cause MQTT messages to overwrite each other. This is now fixed, but note that the new MQTT topic format introduces `job_id`, deviating from our usual topic structure.
- **Fix for API returning incorrect responses for Huey-related tasks**
- API responses related to background tasks (e.g., adding a new Pioreactor, syncing configs, updating firmware) were sometimes incorrect or missing details. This has been fixed.
- **Correction to `od_reading.config`'s `smoothing_penalizer` scaling error**
- A miscalculated scaling factor in `od_reading.config` caused the smoothing factor to be larger than intended. This has been corrected, and your `config.ini` file has been automatically updated to reflect the new values.
- **Fix for missing log events in Event Logs after worker unassignment**
- Some log events (such as clean-up and assignment events) were missing when they occurred after a worker was unassigned. These now appear correctly. Additionally, some unrelated log entries that were mistakenly displayed have been removed.
- **Scaling bug fix in the extended Kalman filter affecting nOD detection**
- A bug in the extended Kalman filter was **causing outlier detections too frequently**. In extreme cases, these detections could compound, driving nOD values negative and corrupting the filter’s internal state. This issue has been fixed with a new, more stable filtering algorithm that significantly improves robustness.

**Important**, any OD calibrations made on software versions 25.1.21 and 25.2.10 have incorrect metadata, and it needs to be updated. Use this hot fix release to fix them. You'll still need to rerun:


pio calibrations analyze --device od --name <cal name>


to recreate the calibration curves.


- Hot fix for OD Calibrations bug.

Enhancements

- **New OD Calibration**: Introduced a new OD calibration using standards (requires multiple vials). Run:

pio calibrations run --device od

Inspired by the plugin by odcambc.

- **UI Improvements**:
- Improved chart colors.
- Added the ability to choose the level of detail on the new Event Logs page.

- **New OD Reading Option**: The OD reading CLI now includes a `--snapshot` option to start the job, take a single reading, and exit. This is useful for scripting.

- **New Pump Control CLI**: Introduced a new CLI for pumps:

pio run pumps --media 1 --waste 2

This command will add 1ml of media and remove 2ml of waste. The order matters, and pumps can be specified multiple times:

pio run pumps --waste 2 --media 1 --waste 2

This new CLI is really useful for experiment profiles. For example, a chemostat can be "programmed" as (but don't actually do this, using a dosing automation):
yaml
common:
jobs:
pumps:
actions:
- type: repeat
hours_elapsed: 0 start immediately
repeat_every_hours: 0.5 every 30m, run the following actions
actions:
- type: start
hours_elapsed: 0
options:
media: 1
waste: 2



- **Experiment & System Enhancements**:
- Initial support for 40ml model.
- Ability to run multiple experiment profiles per experiment.
- Users can now specify which Pioreactor to update on the Updates page (available only with release archives).
- Stirring calibration is now included as part of the self-test.
- Improved stirring job handling when OD readings have long pauses.

- Previously, if a worker’s web server was down, an update would be blocked. Now, the leader will first attempt the web server, and if a 5xx error is observed, it will attempt SSH communication instead.

---

Web API Changes

- Introduced:
- `GET /unit_api/jobs/running/<job>`
- `GET /api/experiment_profiles/running/experiments/<experiment>`

---

Breaking Changes

- **Calibration Structs**:
- `predict` → `x_to_y`
- `ipredict` → `y_to_x`
- This change makes naming clearer.

- **Plugin Migration** (Upcoming):
- Plugins should migrate from `click_some_name` to auto-discovered plugins by importing `run`.
- Example migration:
python
import click
from pioreactor.cli.run import run

run.command("my_name")
click.option("--my_option")
def my_name(my_option):
...



Bug Fixes

- Fixed UI not displaying third-party calibrations.
- Experiment profiles now directly use `unit_api/`, potentially mitigating Huey worker stampedes when starting multiple jobs.
- Fixed `pio calibrations run ... -y` not saving as active.
- Fixed manual dosing issues in the UI.
- Fixed manual log recording in the UI.
- There was a race condition between `monitor` and a db creation job that was preventing the `monitor` job from starting. Awkwardly, this only seemed to happen on power cycles, and only _sometimes_. This escaped our testing. We've fixed it by improving how we initialize the dbs, and how we connect to them.

Highlights
- New UI updates:
- An `Event Logs` page for seeing the logs generated by your Pioreactors
- A detailed overview of your cluster's leader-specific duties on the new `Leader`'s page.
- See the Leader's filesystem, logs, update cluster clocks, and view important running jobs.
- View different Pioreactors' plugins on the `Plugins` page, and install to specific Pioreactor vs entire cluster.
- Manage your calibrations from the UI's new `Calibrations` page.
- View existing calibrations, set active calibrations, and download calibration files.
- New calibrations API. A calibration now creates a YAML file as an artifact, stored in `~/.pioreactor/calibrations`. This makes editing, creating, sharing, and transferring calibrations much easier.
- There's also a new CLI for calibrations:

Usage: pio calibrations [OPTIONS] COMMAND [ARGS]...

interface for all calibration types.

Options:
--help Show this message and exit.

Commands:
delete Delete a calibration file from local storage.
display Display the contents of a calibration YAML file.
list List existing calibrations for the given device.
run Run an interactive calibration assistant for a specific device.
set-active Mark a specific calibration as 'active' for that device
analyze Analyze the data from a calibration.


For example, to run a pump calibration, use `pio calibrations run --device media_pump`. View all your media pump calibrations with: `pio calibrations list --device media_pump`.

- For now, the actual calibrations are the same protocol as before, but in the near future, we'll be updating them with new features. Adding this unified CLI and YAML format was the first step.


Web API changes
- New API to retrieve and set clocks on Pioreactors
- GET `/api/units/<pioreactor_unit>/system/utc_clock`
- GET `/unit_api/system/utc_clock`
- POST `/api/system/utc_clock`
- POST `/unit_api/system/utc_clock`
- New log APIs
- GET `/api/experiments/<experiment>/recent_logs`
- GET `/api/experiments/<experiment>/logs`
- GET `/api/logs`
- GET `/api/workers/<pioreactor_unit>/experiments/<experiment>/recent_logs`
- GET `/api/workers/<pioreactor_unit>/experiments/<experiment>/logs`
- GET `/api/units/<pioreactor_unit>/logs`
- POST `/workers/<pioreactor_unit>/experiments/<experiment>/logs`
- New calibrations APIs
- GET `/api/workers/<pioreactor_unit>/calibrations`
- GET `/unit_api/calibrations`
- GET `/unit_api/active_calibrations`
- GET `/api/workers/<pioreactor_unit>/calibrations/<device>`
- GET `/unit_api/calibrations/<device>`
- PATCH `/api/workers/<pioreactor_unit>/active_calibrations/<device>/<cal_name>`
- PATCH `/unit_api/active_calibrations/<device>/<cal_name>`
- DELETE `/api/workers/<pioreactor_unit>/active_calibrations/<device>/<cal_name>`
- DELETE `/api/workers/<pioreactor_unit>/calibrations/<device>/<cal_name>`
- DELETE `/unit_api/active_calibrations/<device>/<cal_name>`
- DELETE `/unit_api/calibrations/<device>/<cal_name>`
- POST `/unit_api/calibrations/<device>`
- New API for plugins
- GET `/api/units/<pioreactor_unit>/plugins/installed`
- PATCH `/api/units/<pioreactor_unit>/plugins/install`
- PATCH `/api/units/<pioreactor_unit>/plugins/uninstall`
- Changed the `settings` API (see docs).
- New `/api/units` that returns a list of units (this is workers & leader). If leader is also a worker, then it's identical to `/api/workers`
- New `/api/experiments/<experiment>/historical_worker_assignments` that stores historical assignments to experiments
- New Path API for getting the dir structure of `~/.pioreactor`:
- `/unit_api/system/path/<path>`

Enhancements
- new SQL table for `historical_experiment_assignments` that stores historical assignments to experiments.
- UI performance improvements
- Better terminal plots
- Customs charts in the UI are now downsampled like the other charts.
- More logging in experiment profiles

Breaking changes
- `use_calibration` under `od_reading.config` is deprecated. Use the calibrations "active" state instead.
- **Note**: by default, all calibrations are not active, even if they were "current" before. You must set them to be active.
- removed Python library `diskcache`.
- any stirring calibrations needs to be redone. On the command line, run `pio calibration run --device stirring` to start the calibration assistant.
- fixed typo `utils.local_persistant_storage` to `utils.local_persistent_storage`.
- Kalman Filter database table is no longer populated. There is a way to re-add it, lmk.
- moved intermittent cache location to `/tmp/pioreactor_cache/local_intermittent_pioreactor_metadata.sqlite`. This also determined by your configuration, see `[storage]`.
- removed `pioreactor.utils.gpio_helpers`
- removed `calibrations` export dataset. Use the export option on the /Calibrations page instead.
- persistent storage is now on single sqlite3 database in `/home/pioreactor/.pioreactor/storage/local_persistent_pioreactor_metadata.sqlite`. This is configurable in your configuration.
- When checking for calibrations in custom Dosing automations, users may have added:
python
with local_persistant_storage("current_pump_calibration") as cache:
if "media" not in cache:
...

This should be updated to (**Note** the spelling in `local_persistant_storage` changed, too!):

python
with local_persistent_storage("active_calibrations") as cache:
if "media_pump" not in cache:
...


Bug fixes
- Fix PWM3 not cleaning up correctly
- Fixed Stirring not updating to best DC % when using a calibration after changing target RPM
- Fixed a bug that could cause OD calibrations to map a small voltage value to a max OD.
- Fixed bug where dataset exports were not sorted correctly.
- em-dashes are now replaced in config.ini on save.
- Fixed a bug where errors on the Experiment Profiles page weren't properly displayed.

- Hotfix for UI settings bug

Highlights
- New export datasets improvements!
- new export dataset API. The datasets on the Export Data UI page are now provided via YAML files on the leader's disk. This makes it easy to add new datasets to that UI to be exported. These YAML files can be added to `~/.pioreactor/exportable_datasets`.
- new Export Data page in the UI. Preview datasets before you export them, and new partition options for the exported CSVs.
- Plugins can now add datasets to the Export Data page. The plugin's datasets are automatically added to the Export Data page when installed.
- Stirring can now pause itself during an OD reading. This is accomplished by "dodging OD readings". You can activate this feature by setting the `enable_dodging_od` to `True` in config.ini, under `[stirring.config]`. The replaces an older, less reliable plugin that was on our forums. Users have wanted this feature to have a very fast RPM between OD measurements (to get more aeration), and avoid noisy OD measurements. There's no reason to believe this will decrease the noise if using a "moderate" RPM though.

Enhancements
- improvements to Dodging background job code, including the ability to initialize the class based on dodging or not.
- better error handling for failed OD blank action.
- better button state management in the UI.
- a job YAMLs' published_settings can have a new field, `editable` (bool), which controls whether it shows up on the Settings dialog or not. (False means it won't show up since it's not editable!). Default is true. This _should_ align with the `published_setting` in Python's job classes.
- you can add IPv4 addresses to the (new) `[cluster.addresses]` section to specify IPs for pioreactors. Example:

[cluster.addresses]
pio01=10.42.0.2
pio02=10.42.0.3


Note that the leader's address is automatically added in our software.
- new installs only: updated RPiOS to version 2024-11-19
- improvements to correlation self-tests

Bug fixes
- Fixed "circulate X" actions in the Manage All dialog in the UI.

Breaking changes
- moved all the temporary caches, which previously where their own sqlite3 dbs in `/tmp/` to `/tmp/local_intermittent_pioreactor_metadata.sqlite`. This shouldn't break anything unless you update _during_ an experiment - don't do that!