Assemblyline

Service base

- Remove unnecessary code from v4 base package

Release candidate for 4.2 release. We need to bump the minor version to synchronize the service and the core component release.

- Assemblyline will be more restrictive in terms of what service versions are allowed to run because of platform incompatibility (ie. a service on 4.1 may not necessarily work on a 4.0 system)
This means if you upgrade the system to 4.1 before your services have been built for 4.1, Assemblyline will start to disable those services until an update comes in with a matching '4.1' prefix.
- System will be overall more responsive to changes made to the UI with thanks to using events rather than polling to drive core actions.
- Service Base & Dependencies:
- Persistent Service Updaters: updaters will now run as Deployments rather than Jobs in Kubernetes. This entails a change in updater code for existing services under 4.0 where this would now be considered a dependency container. See the docs for more information.
- Sync Interval: There is now pre-defined buckets for setting the sync interval in the UI. You will still have the option of setting the sync interval at a granular level as in 4.0.
- Ingestion: You're now able to increase the number of ingest, complete, and submit threads in Ingester by setting the corresponding environment variables. This can be done by using coreEnv in the values.yaml when deploying.
- Updater: Improved support for Harbor registries as well as for insecure local registries (ie. for development)
- Helm:
- Storage: ReadWriteMany (RWX) is no longer a requirement
- Unified tags for releases:
4.0: You have to set the tag version for core, ui, frontend, and service-server (if defaults aren't taken)
4.1: You will only need to set one tag version that will apply to all components

Rebuild

Base

- Add optimized path for file information updates

Base

- Add an optimized path for file information updates

v4.2.0.stable0
Base

- Use Redis native operator instead of Lua script when scripts are unnecessary
- Reduce the size on Redis messages when possible
- Stop over-using the Redis EXPIRE command to reduce load on Redis
- Add support for privileged services in the ODM

Core

- Reduce the size on Redis messages when possible
- Use Max classification between CART file and submission classification during submit
- Stop using Mocks during test and use real integration testing
- Add support for privileged service by creating Tasking and Safelist clients
- Cache connections to redis structures

Service Server

- Cleanup service status when finishing a task
- Use Tasking and Safelist clients to share code with privileged services

UI - API Server

- Use Max classification between CART file and submission classification during ingest
- Reduce UI start memory footprint
- Add support for privileged services during service registration

Service Base

- Add a run_privileged_service script that will perform the job of both service server and task handler so services can connect directly to the infrastructure
- Add a PrivilegedServiceAPI so privileged service can have direct access to the safelisting API
- Add new ExtraFeature sample service to test Redis message reduction and to show people how to request certain parts of the service task
- Update find_base64 and find_pe_file common functions

UI Frontend

- Add support for privileged services in the UI