Pioreactor

Highlights
- New UI updates:
- An `Event Logs` page for seeing the logs generated by your Pioreactors
- A detailed overview of your cluster's leader-specific duties on the new `Leader`'s page.
- See the Leader's filesystem, logs, update cluster clocks, and view important running jobs.
- View different Pioreactors' plugins on the `Plugins` page, and install to specific Pioreactor vs entire cluster.
- Manage your calibrations from the UI's new `Calibrations` page.
- View existing calibrations, set active calibrations, and download calibration files.
- New calibrations API. A calibration now creates a YAML file as an artifact, stored in `~/.pioreactor/calibrations`. This makes editing, creating, sharing, and transferring calibrations much easier.
- There's also a new CLI for calibrations:

Usage: pio calibrations [OPTIONS] COMMAND [ARGS]...

interface for all calibration types.

Options:
--help Show this message and exit.

Commands:
delete Delete a calibration file from local storage.
display Display the contents of a calibration YAML file.
list List existing calibrations for the given device.
run Run an interactive calibration assistant for a specific device.
set-active Mark a specific calibration as 'active' for that device
analyze Analyze the data from a calibration.


For example, to run a pump calibration, use `pio calibrations run --device media_pump`. View all your media pump calibrations with: `pio calibrations list --device media_pump`. And to duplicate a

- For now, the actual calibrations are the same protocol as before, but in the near future, we'll be updating them with new features. Adding this unified CLI and YAML format was the first step.


Web API changes
- New API to retrieve and set clocks on Pioreactors
- GET `/api/units/<pioreactor_unit>/system/utc_clock`
- GET `/unit_api/system/utc_clock`
- POST `/api/system/utc_clock`
- POST `/unit_api/system/utc_clock`
- New log APIs
- GET `/api/experiments/<experiment>/recent_logs`
- GET `/api/experiments/<experiment>/logs`
- GET `/api/logs`
- GET `/api/workers/<pioreactor_unit>/experiments/<experiment>/recent_logs`
- GET `/api/workers/<pioreactor_unit>/experiments/<experiment>/logs`
- GET `/api/units/<pioreactor_unit>/logs`
- POST `/workers/<pioreactor_unit>/experiments/<experiment>/logs`
- New calibrations APIs
- GET `/api/workers/<pioreactor_unit>/calibrations`
- GET `/unit_api/calibrations`
- GET `/unit_api/active_calibrations`
- GET `/api/workers/<pioreactor_unit>/calibrations/<device>`
- GET `/unit_api/calibrations/<device>`
- PATCH `/api/workers/<pioreactor_unit>/active_calibrations/<device>/<cal_name>`
- PATCH `/unit_api/active_calibrations/<device>/<cal_name>`
- DELETE `/api/workers/<pioreactor_unit>/active_calibrations/<device>/<cal_name>`
- DELETE `/api/workers/<pioreactor_unit>/calibrations/<device>/<cal_name>`
- DELETE `/unit_api/active_calibrations/<device>/<cal_name>`
- DELETE `/unit_api/calibrations/<device>/<cal_name>`
- POST `/unit_api/calibrations/<device>`
- New API for plugins
- GET `/api/units/<pioreactor_unit>/plugins/installed`
- PATCH `/api/units/<pioreactor_unit>/plugins/install`
- PATCH `/api/units/<pioreactor_unit>/plugins/uninstall`
- Changed the `settings` API (see docs).
- New `/api/units` that returns a list of units (this is workers & leader). If leader is also a worker, then it's identical to `/api/workers`
- New `/api/experiments/<experiment>/historical_worker_assignments` that stores historical assignments to experiments
- New Path API for getting the dir structure of `~/.pioreactor`:
- `/unit_api/system/path/<path>`

Enhancements
- new SQL table for `historical_experiment_assignments` that stores historical assignments to experiments.
- UI performance improvements
- Better terminal plots
- Customs charts in the UI are now downsampled like the other charts.
- More logging in experiment profiles

Breaking changes
- `use_calibration` under `od_reading.config` is deprecated. Use the calibrations "active" state instead.
- **Note**: by default, all calibrations are not active, even if they were "current" before. You must set them to be active.
- removed Python library `diskcache`.
- any stirring calibrations needs to be redone. On the command line, run `pio calibration run --device stirring` to start the calibration assistant.
- fixed typo `utils.local_persistant_storage` to `utils.local_persistent_storage`.
- Kalman Filter database table is no longer populated. There is a way to re-add it, lmk.
- moved intermittent cache location to `/tmp/pioreactor_cache/local_intermittent_pioreactor_metadata.sqlite`. This also determined by your configuration, see `[storage]`.
- removed `pioreactor.utils.gpio_helpers`
- removed `calibrations` export dataset. Use the export option on the /Calibrations page instead.
- persistent storage is now on single sqlite3 database in `/home/pioreactor/.pioreactor/storage/local_persistent_pioreactor_metadata.sqlite`. This is configurable in your configuration.
- When checking for calibrations in custom Dosing automations, users may have added:
python
with local_persistant_storage("current_pump_calibration") as cache:
if "media" not in cache:
...

This should be updated to (**Note** the spelling in `local_persistant_storage` changed, too!):

python
with local_persistent_storage("active_calibrations") as cache:
if "media_pump" not in cache:
...


Bug fixes
- Fix PWM3 not cleaning up correctly
- Fixed Stirring not updating to best DC % when using a calibration after changing target RPM
- Fixed a bug that could cause OD calibrations to map a small voltage value to a max OD.
- Fixed bug where dataset exports were not sorted correctly.
- em-dashes are now replaced in config.ini on save.
- Fixed a bug where errors on the Experiment Profiles page weren't properly displayed.

- Hotfix for UI settings bug

Highlights
- New export datasets improvements!
- new export dataset API. The datasets on the Export Data UI page are now provided via YAML files on the leader's disk. This makes it easy to add new datasets to that UI to be exported. These YAML files can be added to `~/.pioreactor/exportable_datasets`.
- new Export Data page in the UI. Preview datasets before you export them, and new partition options for the exported CSVs.
- Plugins can now add datasets to the Export Data page. The plugin's datasets are automatically added to the Export Data page when installed.
- Stirring can now pause itself during an OD reading. This is accomplished by "dodging OD readings". You can activate this feature by setting the `enable_dodging_od` to `True` in config.ini, under `[stirring.config]`. The replaces an older, less reliable plugin that was on our forums. Users have wanted this feature to have a very fast RPM between OD measurements (to get more aeration), and avoid noisy OD measurements. There's no reason to believe this will decrease the noise if using a "moderate" RPM though.

Enhancements
- improvements to Dodging background job code, including the ability to initialize the class based on dodging or not.
- better error handling for failed OD blank action.
- better button state management in the UI.
- a job YAMLs' published_settings can have a new field, `editable` (bool), which controls whether it shows up on the Settings dialog or not. (False means it won't show up since it's not editable!). Default is true. This _should_ align with the `published_setting` in Python's job classes.
- you can add IPv4 addresses to the (new) `[cluster.addresses]` section to specify IPs for pioreactors. Example:

[cluster.addresses]
pio01=10.42.0.2
pio02=10.42.0.3


Note that the leader's address is automatically added in our software.
- new installs only: updated RPiOS to version 2024-11-19
- improvements to correlation self-tests

Bug fixes
- Fixed "circulate X" actions in the Manage All dialog in the UI.

Breaking changes
- moved all the temporary caches, which previously where their own sqlite3 dbs in `/tmp/` to `/tmp/local_intermittent_pioreactor_metadata.sqlite`. This shouldn't break anything unless you update _during_ an experiment - don't do that!

Enhancements
- `dosing_automation.vial_volume` replaced with `dosing_automation.liquid_volume`. You can see the values by watching `pio mqtt -t "pioreactor/+/+/dosing_automation/liquid_volume"` after starting a dosing automation.
- Adding a SQL table for tracking `liquid_volume`.
- Because we are now storing `liquid_volume` in the database, you can add charts in the UI that track the volume over time:
1. Add the following yaml contents to `~/.pioreactor/plugins/ui/contrib/charts/liquid_volume.yaml`: https://gist.github.com/CamDavidsonPilon/95eef30189101da69f706d02ef28d972
2. In your config.ini, under `ui.overview.charts`, add the line `liquid_volume=1`.
- New dataset exports from the Export data page in the UI: calibrations and liquid-volumes.
- Added a "partition by unit" option to the Export data page that will create a csv per Pioreactor in the export, instead of grouping them all together.
- od calibrations can use the `--json-file` to edit calibration polynomial coefficients. In the json file, specify `curve_data_` fields with values of the curve's polynomial coefficients (leading term first), and set `curve_type` as `"poly"`. The routine will begin with that calibration curve displayed.
- faster UI response times when starting jobs.
- faster syncing configs.
- faster copying files across cluster via `pio cp`.
- faster clean up of jobs using PWMs.
- new installs only: updated base RPiOS to 2024-10-22.
- new database table in `/tmp/local_intermittent_pioreactor_metadata.sqlite` called `pio_job_published_settings` that stores the published settings for each job. This powers the next API endpoints:
- New API endpoints for getting the current settings of a _running_ job:
- Per pioreactor:
- GET: `/unit_api/jobs/settings/job_name/<job_name>`
- GET: `/unit_api/jobs/settings/job_name/<job_name>/setting/<setting>`
- Across the cluster:
- GET: `/api/jobs/settings/job_name/<job_name>/setting/<setting>`
- GET: `/api/jobs/settings/job_name/<job_name>/experiments/<experiment>`
- GET: `/api/jobs/settings/job_name/<job_name>/experiments/<experiment>/setting/<setting>`
- GET: `/api/jobs/settings/workers/<pioreactor_unit>/job_name/<job_name>`
- GET: `/api/jobs/settings/workers/<pioreactor_unit>/job_name/<job_name>/setting/<setting>`
Ex: query the temperature of a Pioreactor: `curl http://pio01.local/unit_api/jobs/settings/job_name/temperature_automation/setting/temperature`


Breaking changes
- `pio kill --name x` is now `pio kill --job-name x`
- removed publishing published_settings metadata to mqtt. Ex `$properties`, `$settable`, `$unit`, `$datatype` are no longer being sent mqtt. This was never used, and just a bandwidth suck.

Bug fixes
- fix for OD calibration graph showing "two lines" in the terminal display
- fix for updating over the internet when a Pioreactor is on a `A.devX` or `B.rcY` release
- `pio kill --all-jobs` will no longer kill long-running jobs from plugins (specifically, `logs2x` jobs.)
- updating the UI software won't prematurely stop any currently running activities
- correct ethernet mac address on RPi5s
- We weren't passing all the OS environment variables when jobs were started from the UI. This is fixed now.
- Fixed circulate media / alt. media in the UI.
- Fixed manual dosing updates in the UI.

Enhancements
- amount of data shown on charts is now a function of the OD sampling rate
- allow for showing more than 16 workers in a chart.

Bug fixes
- Bug fix for "Manage all" that would start activities in all Pioreactors, whether they were in the experiment or not.
- Fix for bug when clicking a legend element it not hiding
- `led_intensity` (i.e. changes to LEDs) now respect whether a worker is active or not.
- Fix bug for UI crashing with "colors" error.
- If a worker is referenced in a profile, but is not part of the current experiment, the actions will not be schedualed for it.

Enhancements
- UI improvements to the experiment select box.
- Better clean up of configs when a worker is removed from the cluster.
- Improved UI loading time

Bug fixes
- only show ipv4 in UI and in avahi aliases.
- fixed experiment profile plugin checks.
- fixed experiment profile display crashing the UI when editing plugins section.

Breaking changes
- `pio clear-cache` renamed to `pio cache clear`
- `pio view-cache` renamed to `pio cache view`
- some more web API changes to endpoints that manage updates
- We no longer use `monitor` to start jobs. This has a slowdown when changing LEDs or starting pumps, unfortunately, but generally better performance elsewhere.
- `watchdog` job has been merged with `monitor`. `watchdog` no longer exists.