Lvmgort

🚀 New

* Overwatcher now reports error events via notifications. If the error happens while a tile is being observed, a comment in the night log is added.

✨ Improved

* Roll over the GORT log when the SJD changes.
* Improve the logic handling how the Overwatcher observer decides when to open or close the dome near evening or morning twilight.
* Run a clean-up first in pre-observing in case the spectrographs are not in a good state.
* Run some pre-observing checks before calling each `GortObserver.observe_tile()` in the `ObserverOverwatcher`. Currently only checks if the spectrographs have an error state and resets them.
* Handle `SPECTROGRAPH_NOT_IDLE` errors in the troubleshooter.
* Disable the Overwatcher and cancel observations if the dome fails to move.
* Add retries for safe enclosure operations.

🔧 Fixed

* Fixed a bug that would prevent a new SJD to trigger an update of the ephemeris and calibrations.
* Fix a bug that would leave the Overwatcher in cancelling mode if `start_observing` failed.

✨ Improved

* Take AG darks during the pre-observing task.
* Add `retry_without_parking` option to the shutdown recipe.
* Emit events for dome opening and closing and report them as notifications.
* Modified `emergency_shutdown()` to close the dome if the shutdown recipe fails.

🔧 Fixed

* Prevent the calibrations module from trying to close the dome when a calibration is retrying.
* Prevent a case in which failing to park the telescopes could have caused the dome to not be closed even if `retry_without_parking` was set to `True`.

🚀 New

* [38](https://github.com/sdss/lvmgort/pull/38) Add a post-observing daily task that runs 15 minutes after morning twilight and will do a few check (make sure the dome is closed, park the telescopes, etc.) and retry safe calibrations that failed during the normal sequence.

✨ Improved

* Add a comment to the night log when a calibration fails.
* Do not start exposure if we are within 10 minutes of twilight.

🔧 Fixed

* Prevent the Overwatcher observer from opening the dome while calibrations are ongoing.
* Fixed a bug in the twilight flats recipe related to the extra exposures.

⚙️ Engineering

* Use API to send notifications.

🔥 Breaking change

* `GortObserver.observe_tile` now default to `async_readout=False`. This will block until the exposure is done, which is a more natural behaviour for an external user that is not trying to over-optimise things. The code that uses `observe_tile` in GORT (`Gort.observe()` and `ObserverOverwatcher.observe_loop_task()`) have been updated to explicitly use `async_readout=True`.

🚀 New

* [37](https://github.com/sdss/lvmgort/pull/37) Basic implementation of the `Troubleshooter` class for the Overwatcher. Currently only very broad troubleshooting checks and recipes are implemented.

🏷️ Changed

* Removed morning twilight flats.

🔧 Fixed

* Temporary fix in the cleanup recipe for a bug in `lvmscp` caused by a quick reset after reading out a pending exposure.

⚙️ Engineering

* Added a very basic test to confirm the Overwatcher can be initialised.

🚀 New

* [34](https://github.com/sdss/lvmgort/pull/34) Adds a `safety` module to the Overwatcher that will monitor the alerts independently and close the dome at a very low level if the main task fails to do it after 5 minutes.

✨ Improved

* [35](https://github.com/sdss/lvmgort/pull/35) Refactor dither observing to allow finer control of when to reacquire a tile and when to keep observing and adjust the science telescope dither position.
* [36](https://github.com/sdss/lvmgort/pull/36) Modity the evening twilight recipe to continue cycling the standards mask and taking flats until the exposure time reaches 100s.
* GORT will fail to initialise if the Overwatcher is running. This can be overridden by passing `override_overwatcher=True` to the `Gort` constructor or `--override-overwatcher` in the CLI.
* Rearranged the Overwatcher helpers and make the pre- and post-observing scripts recipes.
* Added a framework for run daily tasks and a pre-observing task.
* Add option to disable the overwatcher to the shutdown recipe.

✨ Improved

* Overwatcher: cancel the current cancellation if conditions are unsafe.
* Overwatcher: lock the dome for 30 minutes if it's closed due to unsafe conditions.
* Always reset the spectrographs before an exposure. Add some checks to ensure the shutters are closed during a cleanup.

🏷️ Changed

* Overwatcher: rename allow dome calibrations to allow calibrations, which enabled/disables all calibration (not only in-dome calibrations).

🔧 Fixes

* Overwatcher: fixes to daytime logic.
* Add the current GORT version to the Overwatcher actor.
* Set the correct dither position in `GortObserver`.