Pioreactor

Highlights

- initial support for Pioreactor 20ml v1.1! This is our latest iteration of Pioreactor. Even though it's a minor 0.x release, there's lots of positives about it. We encourage you to check out the upgrade kit [here](https://pioreactor.com/collections/upgrade-kits/products/pioreactor-20ml-v1-1-upgrade-kit).
- some further support for tracking the model and version of the Pioreactor you are using. Users can change the version in the config file. For example:

[pioreactor]
model=pioreactor_20ml
version=1.1

If you have a mixed cluster (some 1.0, some 1.1), then you should put this configuration in the _unit specific_ config files.
- For v1.1: New temperature inference algorithm makes reaching the `thermostat` setpoint quicker, and the Pioreactor can reach higher temperatures (our internal testing could easily reach up to 45C in a cool room). This algorithm uses the magic of ✨statistics✨. We may update the themostat PID values in the future, but the default ones work okay for now. A Pioreactor v1.0 update for this algorithm should come out soon, too.

Enhancements

- When using `turbidostat`, there is now a small moving average filter on the raw OD readings. This will prevent the turbidostat from firing when an OD outlier occurs.
- MQTT data is no long persisted between leader power-cycles. This was the cause of a lot of bad UI state issues where users couldn't interact with the Pioreactor via the UI after a power-cycle (intentional or not). We originally persisted the data since we previously used MQTT as more like a database, but our engineering style has moved away from that idea, and we now only use MQTT for "ephemeral" data. Taking out the persistent MQTT data forces this style change. Users shouldn't notice anything different.
- The leader is now the source-of-truth for the cluster's clocks. For example, when a worker boots up, it will ask the leader what the time is, and will periodically continue asking. If the leader has access to the internet, it will pull the correct time (and periodically continue asking). If the leader doesn't have access to the internet, it will use the default time on the Pi. This solves the problem of workers' clocks getting out of sync when powered down, especially in a local-access-point network.
![https://i.imgur.com/vt5gxyy.png]
- Lots of small UI improvements, including accessibility, empty-state, and loading improvements.
- Previously, we would "kick" stirring by forcing the DC% to 100% for a moment, and then increasing the running DC% slightly. Going forward, we'll actually try the following when the
sensor fails to read a signal: _DC% to 0%_, then _DC% to 100%_, and then a slight increase in the DC%. Why?
- If the mixing fan has stalled, setting the DC% to 0% does nothing, since the fan is already stopped.
- If the mixing fan is running, but the stir bar isn't in sync, this step will align the stir bar and fan again.
- If the mixing fan is running _too fast_, but the sensor isn't reading it, this allows for a small pause.
- The recommend way to upgrade Pioreactors and clusters is now using _release archives_. We have more control over the upgrade process this way. However, users are still welcome use the command line, `pio update`, which is what we use in house.
- A chart legend's in the UI now displays the entire name of the worker, if there is enough room.

Breaking changes

- Temporary Pioreactor labels, set in the UI, are now unique across an experiment.
- config `max_volume_to_warn` was removed, it's now hardcoded as 90% of `max_volume_to_stop`

Bug fixes

- Fix `pio ...` commands that displayed the CLI options not working on workers.
- Potential fix for heater continuing to be on after requested to be turned off.

Enhancements
- Faster app start-up performance, which should translate to faster response times.
- Log when workers change experiment assignments.
- Log when workers change active status.
- Adding `[pioreactor]` section to config.inis
- improvements to calibration charts

Breaking changes
- `pio install-plugin` is now `pio plugins install`. Likewise for `uninstall`.
- `pio list-plugins` is now `pio plugins list`.
- `pios install-plugin` is now `pios plugins install`. Likewise for `uninstall`.

Bug fixes
- fixed Log table in the UI not showing all entries.
- fixed HAT button response in the UI.

Highlights

- The Pioreactor leader can now support multiple experiments! If you have more than one Pioreactor, this change allows you to run multiple experiments simultaneously, assign Pioreactors to different experiments, and manage all experiments concurrently. No more multi-leader set ups - all you need is a single leader and multiple workers! See video [here](https://www.youtube.com/watch?v=7SuR26BQG5c).
- Ability to delete experiments from the UI.
- Better control over your cluster, using the Inventory page in the UI.
- Ship with network configuration of local-link connections: plug in an ethernet from your Pioreactor to your computer, and after invoking `sudo nmcli c PioreactorLocalLink up`, you should be able to visit `http://pioreactor.local` in your browser.

Enhancements
- replace the `ip` file that is written to on startup with a new `network_info.txt` file that contains the hostname, IPv4 address, and MAC addresses.
- Adding the ethernet (wired) mac address to the system tab.
- new Python module for controlling workers: `pioreactor.cluster_management`
- by default, for new installs, the local-access-point SSID is now `pioreactor_<hostname>`.
- UI performance improvements
- New database tables to handle workers (`workers`) and experiments assignments (`experiment_assignments`).
- New `pio workers` CLI to mange your inventory. Try `pio workers --help` to see all the commands available.
- Better error messages when a self-test fails.
- `pio kill` has new options to kill specific actions. Ex: `pio kill --experiment this-exp`, `pio kill --job-source experiment_profile`


Breaking changes
- When a experiment profile **stops early** (i.e. via "stop early" in the UI), it now will halt any jobs that it started. This is a change from how they worked previously, but this new behaviour is less of a surprise to users.
- `pio add-pioreactor <name>` is now `pio workers add <name>`
- `pio cluster-status` is now `pio workers status`
- `utils.publish_ready_to_disconnected_state` changed names to `utils.managed_lifecycle`
- `config.inventory` in the config.ini is no longer used. All that data is now handled in the database on the leader, and managed in the UI or CLI.
- `pio kill <job_name>` is removed, use `pio kill --name <job_name>`.

Bug fixes
- fix for not being able to access `http://pioreactor.local` reliably.
- fix for multiple exporting datasets when selecting "All experiments"

Known bugs
- removing a Pioreactor leader from an experiment will stop any experiment profiles running that are associated to that experiment.

Enhancements
- For better consistency between Pioreactors, we've introduced a new configuration option that will automatically adjust the IR LED intensity to match a target value in the reference photodiode, at the start of OD reading. This means that if your IR LEDs are slightly different between Pioreactors, the IR LED output will be adjusted to match a hardcoded value. To enable this feature, change the `[od_config]` config parameter `ir_led_intensity` value to `auto`. For new installs, this is the default configuration. This _shouldn't_ change your actual OD readings very much (since we normalize raw PD by REF, and increase or decrease in REF is balanced by increase or decrease in PD), but it will make analysis easier.
- Significant UI performance improvements: we are use less MQTT clients, which should mean faster loading, less network overhead, and overall lower resource-usage.

Bug fixes
- Fixes updating automations in experiment profiles

Enhancements
- reusing more MQTT clients internally => faster job startup and less network overhead

Bug fixes
- using the archive upload method to update Pioreactors had a bug when distributing the archive to workers on the cluster. That has been fixed. The first time, you archive update may fail. But it should succeed the second time.
- fix UI bug that was preventing real-time data from showing up in some custom charts.
- fix UI bug that was causing a stale datum to appear in charts.
- To avoid downstream permission issues, `pio` can't be run as root. That is, `sudo pio ...` will fail.
- a typo prevented `od_config.smoothing_penalizer` from being used internally. This is fixed.
- some retry logic for fixing "lost" state in the UI.
- fixed numerous MQTT connections from accumulating in the UI

Highlights
- **Experimental** introducing outlier filtering in growth rate calculations. This is tunable with the new `ekf_outlier_std_threshold` parameter under `[growth_rate_calculating.config]`. To turn off outlier filtering, set this parameter to some very large number (1000s). Don't put it less than 3.0 - that's silly.
- With this new filtering, we can provide more reasonable values for the parameters of the growth rate Kalman filter. We previously had to artificially _increase_ the measurement std. deviation (`obs_std`) to allow for some outliers. This had the knock-on effect of hiding growth-rate changes, so we had to also increase that parameter `rate_std`. With better outlier protection in the model, we can move these values back. New installs will have the following parameters, and we encourage existing users to try these values if you plan to use the outlier filtering.

[growth_rate_kalman]
acc_std=0.0008
obs_std=1.5
od_std=0.0025
rate_std=0.1

- added configuration for alternative mqtt brokers with the new configuration

[mqtt]
username=pioreactor
password=raspberry
broker_address=
broker_ws_port=9001
broker_port=1883
ws_protocol=ws
use_tls=0


Enhancements
- clear the growth-rate cache with `pio run growth_rate_cacluating clear_cache`
- added Pioreactor specific software version to the UI: Page *Pioreactors -> Manage -> System -> Version*. **this requires a restart to display correctly**
- new UI MQTT library. Is it faster? Maybe!
- increased the default `max_subdose` to 1.0.

Bug fixes
- fixed a case where dosing automation IO execution would not run due to a floating point rounding error. Sorry!
- fixed a memory leak in long running dosing automations that had thousands of dosing events. Sorry!
- fixed a race condition that caused an error to occur when a software PWM channel was closed too quickly. Sorry!
- fixed bug that was partially crashing the UI if some bad syntax was entered into a custom yaml file. Sorry!
- fixed bug that was causing bad json from the server, causing empty / non-loading areas in the UI. Sorry!
- fixed `datum` bug in the Overview that was crashing the UI. Sorry!