Openc3

This release fixes the download_file, get_target_file, and put_target_file APIs. It implements the DISABLED keyword and adds API methods to enable_cmd and disable_cmd (admins only).

Breaking Changes and Upgrade Notes since 5.14.2 - Please Read
* The internal Traefik port was changed to 2900 to match our standard external port and to better support unprivileged runtime environments
* The change requires updating .env, compose.yaml and the traefik configuration files.
* Stop the previous release, update the files, and restart

Bug Fixes
* Implement the DISABLED keyword
* Fix redis cluster in COSMOS Enterprise
* Fix download_file, get_target_file, put_target_file in Ruby and Python

All Pull Requests in this Release
* Bump pypa/gh-action-pypi-publish from 1.8.12 to 1.8.14 by dependabot in https://github.com/OpenC3/cosmos/pull/1141
* Implement DISABLED for commands by jmthomas in https://github.com/OpenC3/cosmos/pull/1138
* Fix redis cluster with StoredQueue by ryanmelt in https://github.com/OpenC3/cosmos/pull/1146
* Bump alpine build by jmthomas in https://github.com/OpenC3/cosmos/pull/1145
* Fix download_file and get_target_file by jmthomas in https://github.com/OpenC3/cosmos/pull/1143

Prerequisites:
Docker - Running OpenC3 requires a working Docker or Podman installation. Typically Docker Desktop on Windows / Mac. Plain Docker or Podman also works on linux. We actively develop and run with Docker Desktop on Mac/Windows, and Linux on Raspberry Pi, so if you have any issues on another platform, please let us know by submitting a ticket!

Minimum Resources allocated to Docker: 4GB RAM, 1 CPU, 80GB Disk
Recommended Resources allocated to Docker: 16GB RAM, 2+ CPUs, 100GB Disk

This release brings major improvements to interface throughput with a new strategy of batching writes to Redis and an additional option to interfaces to even further improvement throughput at the cost of slightly higher processing latency. If you have interfaces receiving data at greater than 10Hz, try adding OPTION OPTIMIZE_THROUGHPUT 0.1 to your interfaces.

Breaking Changes and Upgrade Notes - Please Read
* The internal Traefik port was changed to 2900 to match our standard external port and to better support unprivileged runtime environments
* The change requires updating .env, compose.yaml and the traefik configuration files.
* Stop the previous release, update the files, and restart

Greatly Improved TlmViewer Code Completion in Screen Editor
The screen editor is now fully aware of the telemetry database, making it even easier to quickly build out telemetry screens!

Other Improvements
* Item background colors now alternate in Admin lists
* ScriptRunner Suite report filenames now indicate what was run
* Grapher nows goes to using scientific notation for labels with very large and very small numbers
* Grapher can now add user configured horizontal lines
* Bucket Explorer can now create files in new folders

Bug Fixes
* Fixed python target file loading APIs
* The HIDDEN command keyword is now supported
* Generating presigned urls now checks targets_modified as well as the targets folder
* Fixed importing pandas in Python scripts
* Renewed support for item names that include square brackets
* Fixed cycle time support for low data rate targets
* Fixed upgrading plugins when multiple of the same plugin are installed

All Pull Requests in this Release
* Alternate list item colors by jmthomas in https://github.com/OpenC3/cosmos/pull/1112
* Bump ip from 2.0.0 to 2.0.1 in /openc3-cosmos-init/plugins by dependabot in https://github.com/OpenC3/cosmos/pull/1114
* Fix _get_uri by ryanmelt in https://github.com/OpenC3/cosmos/pull/1116
* Allow override AWS ARN for gov cloud, change traefik ports by jmthomas in https://github.com/OpenC3/cosmos/pull/1119
* Add suite, group, script to report file name by jmthomas in https://github.com/OpenC3/cosmos/pull/1111
* Graph high & low values with scientific notation by jmthomas in https://github.com/OpenC3/cosmos/pull/1108
* Hide HIDDEN commands and packets by jmthomas in https://github.com/OpenC3/cosmos/pull/1109
* Download presigned requests check targets_modified by jmthomas in https://github.com/OpenC3/cosmos/pull/1113
* Add horizontal lines to graphs by jmthomas in https://github.com/OpenC3/cosmos/pull/1107
* Allow setting path when uploading file by jmthomas in https://github.com/OpenC3/cosmos/pull/1120
* Screen completion by jmthomas in https://github.com/OpenC3/cosmos/pull/1117
* Fix import of pandas, and use of PYPI_URL by ryanmelt in https://github.com/OpenC3/cosmos/pull/1126
* Bump pypa/gh-action-pypi-publish from 1.8.11 to 1.8.12 by dependabot in https://github.com/OpenC3/cosmos/pull/1129
* Output warnings as WARN and import openc3-enterprise by jmthomas in https://github.com/OpenC3/cosmos/pull/1131
* Add option to Interfaces to Improve Throughput Through Redis Write Queueing by ryanmelt in https://github.com/OpenC3/cosmos/pull/1123
* Support item names with brackets by jmthomas in https://github.com/OpenC3/cosmos/pull/1134
* Handle upgrade with multiple installed same plugins by ryanmelt in https://github.com/OpenC3/cosmos/pull/1135

Prerequisites:
Docker - Running OpenC3 requires a working Docker or Podman installation. Typically Docker Desktop on Windows / Mac. Plain Docker or Podman also works on linux. We actively develop and run with Docker Desktop on Mac/Windows, and Linux on Raspberry Pi, so if you have any issues on another platform, please let us know by submitting a ticket!

Minimum Resources allocated to Docker: 4GB RAM, 1 CPU, 80GB Disk
Recommended Resources allocated to Docker: 16GB RAM, 2+ CPUs, 100GB Disk

This release mainly fixes issues with our new Python support. Thanks to everyone who is trying it out and submitting tickets!

Python Bug Fixes

* Custom Conversions, Limits Responses, etc our now imported successfully
* Generators now support --python fully
* UBI pip3 is now symlinked to pip
* Pip will now retry with --no-index if the attempt at an online install fails
* Handle String edge case that was breaking INST2 ASCIICMD

Other Improvements

* openc3.sh cli rake will now fail with a warning if no Rakefile is found
* The Raw dialogs in CmdTlmServer are now non-modal, allowing multiple to open and be moved around
* CmdTlmServer now has a Clear Counters option

All Pull Requests in this Release
* Handle custom python classes by ryanmelt in [1080](https://github.com/OpenC3/cosmos/pull/1080)
* Python generators by jmthomas in [1082](https://github.com/OpenC3/cosmos/pull/1082)
* Format developing page as code by jmthomas in [1083](https://github.com/OpenC3/cosmos/pull/1083)
* Warn about rake without Rakefile by jmthomas in [1084](https://github.com/OpenC3/cosmos/pull/1084)
* Allow multiple positionable raw dialogs by jmthomas in [1088](https://github.com/OpenC3/cosmos/pull/1088)
* CmdTlmServer clear counters by jmthomas in [1089](https://github.com/OpenC3/cosmos/pull/1089)
* Link pip3 to pip in ubi by ryanmelt in [1092](https://github.com/OpenC3/cosmos/pull/1092)
* Pip install retry with no-index by ryanmelt in [1093](https://github.com/OpenC3/cosmos/pull/1093)
* Handle bad string types and rescue AttributeError by jmthomas in [1090](https://github.com/OpenC3/cosmos/pull/1090)

Prerequisites:
Docker - Running OpenC3 requires a working Docker or Podman installation. Typically Docker Desktop on Windows / Mac. Plain Docker or Podman also works on linux. We actively develop and run with Docker Desktop on Mac/Windows, and Linux on Raspberry Pi, so if you have any issues on another platform, please let us know by submitting a ticket!

Minimum Resources allocated to Docker: 4GB RAM, 1 CPU, 80GB Disk
Recommended Resources allocated to Docker: 16GB RAM, 2+ CPUs, 100GB Disk

Important Bug Fixes Release for ScriptRunner
We squashed a couple bugs that could cause ScriptRunner to write over the contents of the running script with the contents of another file. Pressing Ctrl-S while running in a different file than original started or some cases where after stopping a different file was still shown could cause the original file to be overwritten. These have all been corrected and all users are encouraged to upgrade.

Further Enhancements to Dynamic Packet Creation
Dynamic packet creation in Ruby now works fully including logging and decom picking up the new packets.

All Pull Requests in this Release
* Clear SR file cache when stopped by jmthomas in [1069](https://github.com/OpenC3/cosmos/pull/1069)
* ARG MINIO/MC release by jmthomas in [1066](https://github.com/OpenC3/cosmos/pull/1066)
* Improve dynamic packet support by ryanmelt in [1070](https://github.com/OpenC3/cosmos/pull/1070)
* Add additional save protections to ScriptRunner by ryanmelt in [1072](https://github.com/OpenC3/cosmos/pull/1072)

Prerequisites:
Docker - Running OpenC3 requires a working Docker or Podman installation. Typically Docker Desktop on Windows / Mac. Plain Docker or Podman also works on linux. We actively develop and run with Docker Desktop on Mac/Windows, and Linux on Raspberry Pi, so if you have any issues on another platform, please let us know by submitting a ticket!

Minimum Resources allocated to Docker: 4GB RAM, 1 CPU, 80GB Disk
Recommended Resources allocated to Docker: 16GB RAM, 2+ CPUs, 100GB Disk

Related Telemetry Items and Screens for Commands
Commands have a new keyword for associating related telemetry items to the command: RELATED_ITEM. This automatically builds a telemetry screen that is shown alongside the command in CmdSender. If you want to get fancy, you can also associate a full custom built telemetry screen with the SCREEN keyword.

Generic Command / Response Protocol
The command response architecture is a command pattern for a lot of different devices, so we took the legacy TEMPLATED protocol and extracted the command response pattern from it into its own CmdResponseProtocol class. Adding this protocol to your Interface will put it into a command/response mode for any commands that have an associated RESPONSE keyword defined.

Dynamic Packet Definition
We've added the ability to dynamically define packets within the COSMOS framework at runtime. This will typically be done by an Accessor or a Processor after receipt of a packet to define its contents for access by our fronted tools.

BucketExplorer Folder Size
BucketExplorer now calculates and displays the size of all files in the currently displayed folder.

New ping endpoints for our API containers
We added a couple new ping endpoints to our API containers that can be used for health/aliveness checks.

PROCESSORs now only run in the Decom microservice
Previous processors were being run both in the interface and in decom (twice total). Processors can now be used reliably run once on each packet reception, so they can safely perform side effects if needed. This is a potentially breaking change if the multiple execution was expected in previous versions.

ScriptRunner Performance Greatly Enhanced
ScriptRunner was updated to process script events in batches, rather than one at a time. This greatly improved ScriptRunner performance and should keep it from ever falling behind when flying through lots of lines that would have to be highlighted one by one.

Important Bug Fixes
1. Accessors now enforce data type on read/write
2. wait() fixed in Python scripts
3. Continue even if there are errors in local_init in the init container. This removes many use cases where the init container can get stuck looping, and will allow for manual recovery by users.
4. Fixed diff logic on plugin upgrade
5. Fixed Python Tcp/IP Client Interface
6. Made tlm item cache default to no caching, unless you specifically want it
7. Fixed a bug where Interfacedisconnect would not be called if an exception was raised during Interfaceconnect
8. Fixed BucketExplorer routes with special characters
9. Fixed writing Python strings without first converting them to bytes type
10. Fixed loading TlmGrapher configurations with more than one graph specified
11. Support upload/download from TableManager in Python
12. Fixed a bug with the rux-status component handling of fatal
13. Updated DataExtractor to handle receiving null values

All Pull Requests in this Release
* Bump github/codeql-action from 2 to 3 by dependabot in [992](https://github.com/OpenC3/cosmos/pull/992)
* Bump actions/upload-artifact from 3 to 4 by dependabot in [993](https://github.com/OpenC3/cosmos/pull/993)
* Tool Screenshots for AstroUX by jmthomas in [996](https://github.com/OpenC3/cosmos/pull/996)
* update logging doc. Add license discussion. by ryanmelt in [1016](https://github.com/OpenC3/cosmos/pull/1016)
* Create ping endpoint in cmd-tlm and scripts api by jmthomas in [1009](https://github.com/OpenC3/cosmos/pull/1009)
* Add folder size by jmthomas in [1000](https://github.com/OpenC3/cosmos/pull/1000)
* Make json, cbor, xml, html enforce data types by jmthomas in [1006](https://github.com/OpenC3/cosmos/pull/1006)
* Move openc3_script_sleep and fix wait() in python by jmthomas in [1004](https://github.com/OpenC3/cosmos/pull/1004)
* New cmd keywords for related telemetry and remove legacy snmp by ryanmelt in [1005](https://github.com/OpenC3/cosmos/pull/1005)
* continue if local init fails by ryanmelt in [1021](https://github.com/OpenC3/cosmos/pull/1021)
* upcase microservice target name by ryanmelt in [1020](https://github.com/OpenC3/cosmos/pull/1020)
* Only run processors in decom microservice by ryanmelt in [1019](https://github.com/OpenC3/cosmos/pull/1019)
* Add TCPIP keepalive options by ryanmelt in [1018](https://github.com/OpenC3/cosmos/pull/1018)
* Add Generic Cmd Response Protocol and Template Accessor by ryanmelt in [1017](https://github.com/OpenC3/cosmos/pull/1017)
* Add warning about unmerged diffs and fix diff logic by jmthomas in [1027](https://github.com/OpenC3/cosmos/pull/1027)
* Fix python tpcip client connection by jmthomas in [1034](https://github.com/OpenC3/cosmos/pull/1034)
* Bump follow-redirects from 1.15.2 to 1.15.4 in /openc3-cosmos-init/plugins/openc3-tool-base by dependabot in [1028](https://github.com/OpenC3/cosmos/pull/1028)
* Bump follow-redirects from 1.15.2 to 1.15.4 in /openc3-cosmos-init/plugins by dependabot in [1029](https://github.com/OpenC3/cosmos/pull/1029)
* Bump follow-redirects from 1.15.2 to 1.15.4 in /openc3/templates/tool_react by dependabot in [1031](https://github.com/OpenC3/cosmos/pull/1031)
* Bump follow-redirects from 1.15.3 to 1.15.4 in /docs.openc3.com by dependabot in [1030](https://github.com/OpenC3/cosmos/pull/1030)
* Default to no tlm cache by ryanmelt in [1037](https://github.com/OpenC3/cosmos/pull/1037)
* Process ScriptRunner events in batches by ryanmelt in [1038](https://github.com/OpenC3/cosmos/pull/1038)
* Dependencies by jmthomas in [1035](https://github.com/OpenC3/cosmos/pull/1035)
* Ensure disconnect called at least once on interface. by ryanmelt in [1039](https://github.com/OpenC3/cosmos/pull/1039)
* Fix bucket explorer route by jmthomas in [1047](https://github.com/OpenC3/cosmos/pull/1047)
* Write python strings by jmthomas in [1041](https://github.com/OpenC3/cosmos/pull/1041)
* Fix Grapher load config by jmthomas in [1046](https://github.com/OpenC3/cosmos/pull/1046)
* Allow python upload/download in TableManager by jmthomas in [1052](https://github.com/OpenC3/cosmos/pull/1052)
* Map fatal to critical by jmthomas in [1050](https://github.com/OpenC3/cosmos/pull/1050)
* Add requirements.txt to python gems by jmthomas in [1061](https://github.com/OpenC3/cosmos/pull/1061)
* Add Script Runner to logger source. Fix docs by jmthomas in [1057](https://github.com/OpenC3/cosmos/pull/1057)
* Data Extractor handle null by ryanmelt in [1063](https://github.com/OpenC3/cosmos/pull/1063)
* Test command endianness by jmthomas in [1051](https://github.com/OpenC3/cosmos/pull/1051)
* Handle 404 errors in Script Runner by jmthomas in [1064](https://github.com/OpenC3/cosmos/pull/1064)
* Add queued store and display active users by jmthomas in [1045](https://github.com/OpenC3/cosmos/pull/1045)
* Dynamic target packet definition creation support. by ryanmelt in [1048](https://github.com/OpenC3/cosmos/pull/1048)

Prerequisites:
Docker - Running OpenC3 requires a working Docker or Podman installation. Typically Docker Desktop on Windows / Mac. Plain Docker or Podman also works on linux. We actively develop and run with Docker Desktop on Mac/Windows, and Linux on Raspberry Pi, so if you have any issues on another platform, please let us know by submitting a ticket!

Minimum Resources allocated to Docker: 4GB RAM, 1 CPU, 80GB Disk
Recommended Resources allocated to Docker: 16GB RAM, 2+ CPUs, 100GB Disk

IMPORTANT: Breaking change to Non-Root Containers and Renamed Minio Volume

Previously all of our non-COSMOS unique containers ran as received from Docker Hub (so Traefik and Minio ran as root). Going forward all of our containers are configured to run as non-root users (except with rootless podman - which isn't really running as root).

This is a breaking change as earlier versions have volume mounted data that is owned by other user ids. Additionally we renamed the openc3-minio-v volume to openc3-bucket-v to ease upgrading to Enterprise Edition. Note that significant changes were made to compose.yaml and all traefik config files.

To migrate from COSMOS 5.12.0 to 5.13.0 please run the following scripts from your COSMOS project folder.

MacOS/Linux: https://github.com/OpenC3/cosmos-project/blob/05c95a24627ac0694fd4af605533de1eb13683f4/migrate_5_12_to_5_13.sh

Windows: https://github.com/OpenC3/cosmos-project/blob/05c95a24627ac0694fd4af605533de1eb13683f4/migrate_5_12_to_5_13.bat

Note: you should always review shell scripts before running them on your computer.

AstroUx 7 Styling

We've gone through a review with Rocket Communications and all of our tools have a fresh new update to AstroUx version 7! You'll notice more consistent styling and layout throughout and a fresh new look. Also, all value widgets for items with limits now show an icon to indicate limits state improving color blind support.

TlmGrapher Array Items

You can now graph individual items from arrays in TlmGrapher!

ScriptRunner Open Recent

ScriptRunner now keeps track of recently opened files in the File menu

More Persistent config when leaving tools and coming back

Whenever you leave a COSMOS tool and then return, it will generally now continue with the same settings as when you left, rather than refreshing.

New MqttStreamInterface

New interface that supports using two Mqtt topics as byte streams for sending and receiving data.

Python Streaming API

Added python code to support all of our Websocket Apis including the StreamingApi.

Calendar and Autonomic Now Enterprise Only

These tools have been moved to COSMOS Enterprise Edition going forward. Thanks to all of our Enterprise customers for supporting us!

Python code generators now available

Our code generators now support either Ruby or Python. Just add --python to the end for Python support.

Important Bug Fixes

1. The cmd() log_message parameter now works again
2. Fixed commanding with binary data from Python
3. Fixed the XTCE converter and added XTCE support for ContainerRefEntry
4. Python suite analysis now handles load_utility calls

All Pull Requests in this Release
* Bump axios to latest by jmthomas in [930](https://github.com/OpenC3/cosmos/pull/930)
* Bump actions/github-script from 6 to 7 by dependabot in [932](https://github.com/OpenC3/cosmos/pull/932)
* Graph array by ryanmelt in [933](https://github.com/OpenC3/cosmos/pull/933)
* Remove Logger from script files. Fix cmd log_message param by jmthomas in [931](https://github.com/OpenC3/cosmos/pull/931)
* Change severity to level by jmthomas in [936](https://github.com/OpenC3/cosmos/pull/936)
* Prepend astro status to valuewidget by jmthomas in [938](https://github.com/OpenC3/cosmos/pull/938)
* Bump got, docusaurus/core, docusaurus/plugin-client-redirects, docusaurus/preset-classic and docusaurus-lunr-search in /docs.openc3.com by dependabot in [945](https://github.com/OpenC3/cosmos/pull/945)
* Updates for docusaurus 3 by ryanmelt in [947](https://github.com/OpenC3/cosmos/pull/947)
* Fix python encoding of binary by jmthomas in [946](https://github.com/OpenC3/cosmos/pull/946)
* Protocol api docs by jmthomas in [949](https://github.com/OpenC3/cosmos/pull/949)
* Add Open Recent to ScriptRunner, fix file icons by jmthomas in [951](https://github.com/OpenC3/cosmos/pull/951)
* Bump adobe/css-tools from 4.3.1 to 4.3.2 in /openc3/templates/tool_react by dependabot in [962](https://github.com/OpenC3/cosmos/pull/962)
* Bump pypa/gh-action-pypi-publish from 1.8.10 to 1.8.11 by dependabot in [963](https://github.com/OpenC3/cosmos/pull/963)
* Default config by jmthomas in [960](https://github.com/OpenC3/cosmos/pull/960)
* Mqtt Stream and Stream Interface. Plus fixes and faraday multipart by ryanmelt in [955](https://github.com/OpenC3/cosmos/pull/955)
* Astro by jmthomas in [943](https://github.com/OpenC3/cosmos/pull/943)
* Add Key Concepts page by jmthomas in [950](https://github.com/OpenC3/cosmos/pull/950)
* Bump actions/setup-python from 4 to 5 by dependabot in [969](https://github.com/OpenC3/cosmos/pull/969)
* Add OPENC3_IMAGE_SUFFIX to the cli and cliroot Calls by stephen-ritter in [957](https://github.com/OpenC3/cosmos/pull/957)
* Cleanup cmd apis by jmthomas in [967](https://github.com/OpenC3/cosmos/pull/967)
* Cleanup tlm apis by jmthomas in [970](https://github.com/OpenC3/cosmos/pull/970)
* XTCE - Add ContainerRefEntry support - Ignore Ancillary by ryanmelt in [972](https://github.com/OpenC3/cosmos/pull/972)
* Dependencies by jmthomas in [971](https://github.com/OpenC3/cosmos/pull/971)
* Fix Details Dialog when limits are 0 by jmthomas in [973](https://github.com/OpenC3/cosmos/pull/973)
* Create python streaming api by jmthomas in [974](https://github.com/OpenC3/cosmos/pull/974)
* Move Calendar and Autonomic to Enterprise by jmthomas in [976](https://github.com/OpenC3/cosmos/pull/976)
* Further astro refinements by jmthomas in [980](https://github.com/OpenC3/cosmos/pull/980)
* run all containers as non-root by ryanmelt in [979](https://github.com/OpenC3/cosmos/pull/979)
* Docusaurus fixes by ryanmelt in [983](https://github.com/OpenC3/cosmos/pull/983)
* Add python generators by jmthomas in [982](https://github.com/OpenC3/cosmos/pull/982)
* python suite analysis handle load_utility require_utility by ryanmelt in [984](https://github.com/OpenC3/cosmos/pull/984)
* Packet Viewer remember last packet by ryanmelt in [985](https://github.com/OpenC3/cosmos/pull/985)
* Add checks to plugin undeploy by ryanmelt in [986](https://github.com/OpenC3/cosmos/pull/986)

Prerequisites:
Docker - Running OpenC3 requires a working Docker or Podman installation. Typically Docker Desktop on Windows / Mac. Plain Docker or Podman also works on linux. We actively develop and run with Docker Desktop on Mac/Windows, and Linux on Raspberry Pi, so if you have any issues on another platform, please let us know by submitting a ticket!

Minimum Resources allocated to Docker: 4GB RAM, 1 CPU, 80GB Disk
Recommended Resources allocated to Docker: 16GB RAM, 2+ CPUs, 100GB Disk