Pyde1

--------------------

Overview
========

Resolves shutdown issue with MQTT unconnected, DE1 config-file values,
improves some logging, updates FeatureFlag for FW 1293,
improves compatability with Manjaro (OS),
fixes documentation-generation issue.

Changed
=======

* Reduce log severity for unimplemented MMRs 0x3820 and 0x3824 – ``0125b72``
* ``FeatureFlag`` includes ``sched_idle`` flag, active for FW 1293 and later –
``64ee7f7``
* Timeouts on CUUID request/notify log changed wording to state
that it could also be the write or the lack of a notify received that
caused the timeout – ``a675f50``
* Removed stray comment from ``20-create-dirs.sh`` – ``6070984``
* Link ``README.rst`` for documentation generation – ``bb640f3``

Fixed
=====

* Shutdown without an MQTT connection does not try (and fail) to close it –
``adda65e``
* DE1 is initialized with config-file values, rather than default –
``f7d6393``
* HTTP API now returns a more descriptive error if the payload data type is
incorrect – ``43614df``
* `disconnect-btid.sh` should no longer cause `sh` errors with Manjaro OS –
``d3a3c65``
* Service definitions updated to use ``StandardError=journal`` – ``ac0ead7``


------------------

------------------

Overview
========

First release version.

Changed
=======

* Allow request of Idle from a refill state
(apparently not acted on by the DE1) - ``55d81bb``
* Allow "force" of DE1 Idle from any state, enabled through config -
``05adc93``
* Prereqs updated to current versions - ``5d320cb``

-------------------

Overview
========

Documentation, including installation, added. Installation scripts,
tested with Raspberry Pi OS Lite (Release date: October 30th 2021,
Kernel version: 5.10) available in the source repo.

New
===

* Documentation viewable at https://pyde1.readthedocs.io/en/latest/
* Install scripts in the source repo in the ``install`` directory
* Provide config for TLS for MQTT clients - ``427b3e0``

Changed
=======

* Documentation reorganized and consolidated into the ``docs`` directory
* ``disconnect-btid.sh`` is now expected at
``/usr/local/bin/pyde1-disconnect-btid.sh`` by ``pyde1.service``

------------------

Overview
========

Functionality for the beta release completed and tested.

New
===

- The flush-control features of *experimental* Firmware 1283 were
implemented and include control of target duration, temperature, and
flow. - ``46c0481``

- Clean, Descale, and Transport functionality is now available through
the API. - ``65f2ac9``

- Provide asynchronous firmware upload through API. - ``d6a2dbc``, ``32436a9``

- GET of DE1\_STATE enabled. - ``2b4435e``

- Rewrite of logging and logging configuration. "Early" logging is
captured and routed to the log file, once it is opened. Log levels
and formatters can be easily configured through the YAML config
files. - ``b759168``, ``39c714d``, ``7df0397``, ``d3e128c``, ``cabab97``

- Provide logging over MQTT for client use (in addition to console and
log file). - ``019bed0``

- Profile frame logging provides "not" names for unset FrameFlags to
clarify log messages. For example, the absence of ``CtrlF`` is now
rendered as ``CtrlP``. - ``c842565``

- MQTT "Will" implemented, reporting unexpected MQTT disconnects.
- ``22d06b4``

- Feature flags have been added to formalize access to DE1 and firmware
abilities. - ``d7405b0``

Changed
=======

- ``c_api`` was updated with new information. - ``46c0481``

- The firmware version is read early in the DE1 initialization to
determine the range of valid MMRs and how to efficiently read them.
- ``46c0481``

- The ``ModeControl`` class was refactored into ``flow_sequencer``.
- ``46c0481``

- MMRs that are not able to be decoded (such as not implemented), are
logged along with the value received. - ``2d0fa24``

- Return 400 Bad Request for PATCH/PUT with no content. - ``d00bd24``

- Change MQTT to not request retaining messages from pyDE1. - ``8a8ba5e``

- Logging level and wording changes. - ``99ec22f``, ``b31c850``

- Rework imports to remove order dependencies and simplify. - ``c895f7d``,
- ``b31c850``

- Improve reconnection algorithm for DE1 and Scale. - ``6be3e5a``

- Improve camelcase\_from\_underscore(). - ``0b40fe9``

- Do not try to reconnect DE1 or Scale while shutting down. - ``bd21a93``

- Inbound (HTTP) API: Check DE1 and scale is\_ready instead of
is\_connected. - ``5de28e7``

------------------

Overview
========
This release focused on converting command-line executables to robust,
self-starting, and supervised services. Both the core pyDE1 controller
and the Visualizer uploader now can be started with ``systemd``
automatically at boot. Configuration of many parameters can be done
through YAML files (simple, human-friendly syntax), by default in
``/usr/local/pyde1/``. Command-line parameters, usable by the service
unit files, can be used to override the config-file location.

Logging configuration may change prior to "beta". At this time it is
only configurable in the output format and level for the *stderr* and
*file* loggers.

By default, the *stderr* logger is at the WARNING level abd without
timestamps, as it is managed through ``systemd`` when being run as a
service. A command-line parameter allows for timestamped output at the
DEBUG level for interactive use.

New
===

- Services run under ``systemd``

- Service ("unit") files for ``pyde1.service`` and
``pyde1-visualizer.service``
- Config files in YAML form

- Auto-off, configurable
- Track the IDs of connected Bluetooth devices for cleanup under Linux
and disconnect them at the Bluez level in the case of a non-graceful
exit
- MQTT supports authorization and access-control lists
- Visualizer: Don't upload short "shots", such as for flushing
(configurable)
- Stop-at-weight offset configurable through ``pyde1.conf``
- Database:

- Self-initialize, if needed
- Check for the proper schema at start

- Replay: config file and command-line switches allow easier
configuration, including sequence ID and MQTT topic root

Changed
=======

.. warning::
SIGHUP is no longer used for log rotation. It is a
termination signal.

- Paths changed to ``/var/log/pyde1`` and
``/var/lib/pyde1/pyde1.sqlite`` by default (configurable)
- Refactored and unified shutdown processes
- Refactored supervised processes to handle uncaught exceptions and
properly terminate for automated restart
- Visualizer: log to ``pyde1-visualizer.log`` by default
- Stop-at-weight internally includes 170 ms to account for the
"fall-time" from the basket to the cup.
- Logging:

- Switched to a file-watcher handler so that log rotation should be
transparent, without the need of a signal
- Provide better control of formatting and level for use with
``systemd`` (service) infrastructure
- Change default file name to ``pyde1.log``
- Add ``--console`` command-line flag to provide timestamped,
DEBUG-level output to assist in development and debugging
- Adjust some log levels so that INFO-level logs are more meaningful
- Removed last usages of ``aiologger``

- The outbound API reports "disconnected" for the DE1 and scale when
initialized

Fixed
=====

- MQTT (outbound) API will now detect connection or authentication
failures with the broker and terminate pyDE1
- FlowSequencer no longer raises exception when trying to report that
the steam time is not managed directly by the software. (It is
managed by the DE1 firmware.)
- Mass-flow estimates had an off-by-one error that was corrected
- Replay now properly reports sequence\_id on gate notifications

Deprecated
==========

- ``find_first_and_load.py`` (Use the APIs. It would have already been
removed if previously deprecated)

Removed
=======

- ``ugly_bits.py`` (previously deprecated)
- ``try_de1.py`` (previously deprecated)
- ``DE1._recorder_active`` and dependencies, including ``shot_file.py``
(previously deprecated)
- Profile ``from_json_file()`` (previously deprecated)
- ``replay_vis_test.py`` -- Use ``replay.py`` with config or
command-line options


------------------

------------------

Schema Upgrade Required
=======================

.. warning::
Backup your database before updating the schema.

See SQLite ``.backup`` for details if you are not familiar.

This adds columns for the ``id`` and ``name`` fields that are now being
sent with ``ConnectivityUpdate``

New
===

- Stand-alone app automatically uploads to Visualizer on shot
completion
- PUT and GET of DE1\_PROFILE\_ID allows setting of profile by ID
- A stand-alone "replay" utility can be used to exercise clients, such
as web apps
- Both the DE1 and scale will try to reconnect on unexpected disconnect
- Add ``DE1IncompleteSequenceRecordError`` for when write is not yet
complete
- Variants of the EB6 profile at different temperatures

Changed
=======

- Better logging when waiting for a sequence to complete times out
- Capture pre-sequence history at all times so "sync" is possible on
replay
- Removed read-back of CUUID.RequestedState as StateInfo provides
current state
- Removed "extra" last-drops check
- Allow more API requests when DE1 or scale is not ready
- Use "ready" and not just "connected" to determine if the DE1 or scale
can be queried
- Allow [dis]connect while [dis]connected
- ``ConnectivityChange`` notification includes ``id`` and ``name`` to
remove the need to call the API for them
- Improve error message on JSON decode by including a snippet around
the error
- Set the default first-drops threshold to 0.0 for fast-flowing shots