Scancodeio

--------------------

- Synchronize QUEUED and RUNNING pipeline runs with their related worker jobs during
worker maintenance tasks scheduled every 10 minutes.
If a container was taken down while a pipeline was running, or if pipeline process
was killed unexpectedly, that pipeline run status will be updated to a FAILED state
during the next maintenance tasks.
QUEUED pipeline will be restored in the queue as the worker redis cache backend data
is now persistent and reloaded on starting the image.
Note that internaly, a running job emits a "heartbeat" every 60 seconds to let all the
workers know that it is properly running.
After 90 seconds without any heartbeats, a worker will determine that the job is not
active anymore and that job will be moved to the failed registry during the worker
maintenance tasks. The pipeline run will be updated as well to reflect this failure
in the Web UI, the REST API, and the command line interface.
https://github.com/nexB/scancode.io/issues/130

- Enable redis data persistence using the "Append Only File" with the default policy of
fsync every second in the docker-compose.
https://github.com/nexB/scancode.io/issues/130

- Add a new tutorial chapter about license policies and compliance alerts.
https://github.com/nexB/scancode.io/issues/337

- Include layers in docker image data.
https://github.com/nexB/scancode.io/issues/175

- Fix a server error on resource details view when the compliance alert is "missing".
https://github.com/nexB/scancode.io/issues/344

- Migrate the ScanCodebase pipeline from `scancode.run_scancode` subprocess to
`scancode.scan_for_application_packages` and `scancode.scan_for_files`.
https://github.com/nexB/scancode.io/issues/340

--------------------

- Fix a build failure related to dependency conflict.
https://github.com/nexB/scancode.io/issues/342

-------------------

- Upgrade ScanCode-toolkit to version 30.1.0

- Replace the task queue system, from Celery to RQ.
https://github.com/nexB/scancode.io/issues/176

- Add ability to delete "not started" and "queued" pipeline tasks.
https://github.com/nexB/scancode.io/issues/176

- Add ability to stop "running" pipeline tasks.
https://github.com/nexB/scancode.io/issues/176

- Refactor the "execute" management command and add support for --async mode.
https://github.com/nexB/scancode.io/issues/130

- Include codebase resource data in the details of package creation project errors.
https://github.com/nexB/scancode.io/issues/208

- Add a SCANCODEIO_REST_API_PAGE_SIZE setting to control the number of objects
returned per page in the REST API.
https://github.com/nexB/scancode.io/issues/328

- Provide an "add input" action on the Project endpoint of the REST API.
https://github.com/nexB/scancode.io/issues/318

-------

- Add ability to "archive" projects, from the Web UI, API and command line interface.
Data cleanup of the project's input, codebase, and output directories is available
during the archive operation.
Archived projects cannot be modified anymore and are hidden by default from the
project list.
A project cannot be archived if one of its related run is queued or already running.
https://github.com/nexB/scancode.io/issues/312

- Remove the run_extractcode pipe in favor of extractcode API.
https://github.com/nexB/scancode.io/issues/312

- The `scancode.run_scancode` pipe now uses an optimal number of available CPUs for
multiprocessing by default.
The exact number of parallel processes available to ScanCode.io can be defined
using the SCANCODEIO_PROCESSES setting.
https://github.com/nexB/scancode.io/issues/302

- Renamed the SCANCODE_DEFAULT_OPTIONS setting to SCANCODE_TOOLKIT_CLI_OPTIONS.
https://github.com/nexB/scancode.io/issues/302

- Log the outputs of run_scancode as progress indication.
https://github.com/nexB/scancode.io/issues/300

-------

- Upgrade ScanCode-toolkit to version 21.7.30

- Add new documentation chapters and tutorials on the usage of the Web User Interface.
https://github.com/nexB/scancode.io/issues/241

- Add ability to register custom pipelines through a new SCANCODEIO_PIPELINES_DIRS
setting.
https://github.com/nexB/scancode.io/issues/237

- Add a pipeline `scan_package.ScanPackage` to scan a single package archive with
ScanCode-toolkit.
https://github.com/nexB/scancode.io/issues/25

- Detected Package dependencies are not created as Package instance anymore but stored
on the Package model itself in a new `dependencies` field.
https://github.com/nexB/scancode.io/issues/228

- Add the extra_data field on the DiscoveredPackage model.
https://github.com/nexB/scancode.io/issues/191

- Improve XLSX creation. We now check that the content is correctly added before
calling XlsxWriter and report and error if the truncated can be truncated.
https://github.com/nexB/scancode.io/issues/206

- Add support for VMWare Photon-based Docker images and rootfs. This is an RPM-based
Linux distribution

--------

- Add support for VM image formats extraction such as VMDK, VDI and QCOW.
See https://github.com/nexB/extractcode#archive-format-kind-file_system for the full
list of supported extensions.
The new extraction feature requires the installation of `libguestfs-tools`,
see https://github.com/nexB/extractcode#adding-support-for-vm-images-extraction for
installation details.
https://github.com/nexB/scancode.io/issues/132

- Add the ability to disable multiprocessing and threading entirely through the
SCANCODEIO_PROCESSES setting. Use 0 to disable multiprocessing and use -1 to also
disable threading.
https://github.com/nexB/scancode.io/issues/185

- Missing project workspace are restored on reports (xlsx, json) creation. This allow
to download reports even if the project workspace (input, codebase) was deleted.
https://github.com/nexB/scancode.io/issues/154

- Add ability to search on all list views.
https://github.com/nexB/scancode.io/issues/184

- Add the is_binary, is_text, and is_archive fields to the CodebaseResource model.
https://github.com/nexB/scancode.io/issues/75