Openmsistream

This release includes a fix for a crash that would appear when logging the final shutdown message for an S3TransferStreamProcessor.

This release edits the "ControlledProcess" logic to allow all long-running programs to periodically produce "heartbeat" messages to a topic to let users remotely and asynchronously monitor which programs are still alive and running. The heartbeat messages have keys like "[program_id]_heartbeat" (where "program_id" is set on the command line) and json-formatted string values that are dictionaries with timestamps and information on how many messages/bytes have been produced/read/processed since the previous heartbeat.

Three command line arguments are added to specify the name of the topic that should accept the heartbeat messages, the ID of the program (to uniquely identify its heartbeat messages if multiple programs and sending heartbeats to the same topic), and the interval at which heartbeat messages should be produced.

Heartbeat messages can be sent to a different broker than their main programs are interacting with by adding broker configurations in the new "[heartbeat]" section of the configuration file. That section can also include configurations for the producer that will produce the heartbeat messages.

The PR also includes new CI tests and documentation for this new behavior.

This release fixes a rare bug that would potentially cause problems when running a DataFileUploadDirectory (especially if the files in that directory were very large in size). The fix is to make the EventHandlerActiveFile utility dataclass hashable by setting eq and frozen both True in the decorator; EventHandlerActiveFile objects are not mutable so this is fine.

This release makes use of new command-line argument propagation logic in OpenMSIToolbox, which greatly simplifies the process of propagating command line arguments through to class constructors.

This release fixes parsing of KafkaCrypto config files, allowing full use of any parameters there in the KafkaCrypto implementation. Users with already-running workflows relying on encryption will want to double check that everything still works after upgrading to this version, if necessary, since before now the KafkaCrypto config files weren't being read fully. (This will only affect users that have edited the KafkaCrypto config files themselves, otherwise everything will be the same.)

This release adds a new flag ("--treat_undecryptable_as_plaintext") for encrypted Consumer-side programs to allow more efficient processing of messages that will never be decrypted. Useful for cases where encrypted and non-encrypted messages get mixed in topics, or when enabling/disabling encryption across a deployment.