Scancodeio

-------

- Upgrade ScanCode-toolkit to version 21.7.30

- Add new documentation chapters and tutorials on the usage of the Web User Interface.
https://github.com/nexB/scancode.io/issues/241

- Add ability to register custom pipelines through a new SCANCODEIO_PIPELINES_DIRS
setting.
https://github.com/nexB/scancode.io/issues/237

- Add a pipeline `scan_package.ScanPackage` to scan a single package archive with
ScanCode-toolkit.
https://github.com/nexB/scancode.io/issues/25

- Detected Package dependencies are not created as Package instance anymore but stored
on the Package model itself in a new `dependencies` field.
https://github.com/nexB/scancode.io/issues/228

- Add the extra_data field on the DiscoveredPackage model.
https://github.com/nexB/scancode.io/issues/191

- Improve XLSX creation. We now check that the content is correctly added before
calling XlsxWriter and report and error if the truncated can be truncated.
https://github.com/nexB/scancode.io/issues/206

- Add support for VMWare Photon-based Docker images and rootfs. This is an RPM-based
Linux distribution

--------

- Add support for VM image formats extraction such as VMDK, VDI and QCOW.
See https://github.com/nexB/extractcode#archive-format-kind-file_system for the full
list of supported extensions.
The new extraction feature requires the installation of `libguestfs-tools`,
see https://github.com/nexB/extractcode#adding-support-for-vm-images-extraction for
installation details.
https://github.com/nexB/scancode.io/issues/132

- Add the ability to disable multiprocessing and threading entirely through the
SCANCODEIO_PROCESSES setting. Use 0 to disable multiprocessing and use -1 to also
disable threading.
https://github.com/nexB/scancode.io/issues/185

- Missing project workspace are restored on reports (xlsx, json) creation. This allow
to download reports even if the project workspace (input, codebase) was deleted.
https://github.com/nexB/scancode.io/issues/154

- Add ability to search on all list views.
https://github.com/nexB/scancode.io/issues/184

- Add the is_binary, is_text, and is_archive fields to the CodebaseResource model.
https://github.com/nexB/scancode.io/issues/75

--------

- Adds a new way to fetch docker images using skopeo provided as a
plugin using docker:// reference URL-like pointers to a docker image.
The syntax is docker://<docker image> where <docker image> is the string
that would be used in a "docker pull <docker image>" command.
Also rename scanpipe.pipes.fetch.download() to fetch_http()
https://github.com/nexB/scancode.io/issues/174

- Pipeline status modals are now loaded asynchronously and available from the
project list view.

- Fix an issue accessing codebase resource content using the scan_codebase and
load_inventory pipelines.
https://github.com/nexB/scancode.io/issues/147

--------

- The installation local timezone can be configured using the TIME_ZONE setting.
The current timezone in now included in the dates representation in the web UI.
https://github.com/nexB/scancode.io/issues/142

- Fix pipeline failure issue related to the assignment of un-saved (not valid) packages.
https://github.com/nexB/scancode.io/issues/162

- Add a new QUEUED status to differentiate a pipeline that is in the queue for execution
from a pipeline execution not requested yet.
https://github.com/nexB/scancode.io/issues/130

- Refactor the multiprocessing code for file and package scanning.
All database related operation are now executed in the main process as forking the
existing database connection in sub-processes is a source of issues.
Add progress logging for scan_for_files and scan_for_application_packages pipes.
https://github.com/nexB/scancode.io/issues/145

- Links from the charts to the resources list are now also filtered by
in_package/not_in_package if enabled on the project details view.
https://github.com/nexB/scancode.io/issues/124

- Add ability to filter on codebase resource detected values such as licenses,
copyrights, holders, authors, emails, and urls.
https://github.com/nexB/scancode.io/issues/153

- Filtered list views from a click on chart sections can now be opened in a new tab
using ctrl/meta + click.
https://github.com/nexB/scancode.io/issues/125

- Add links to codebase resource and to discovered packages in list views.

--------

- Implement timeout on the scan functions, default to 120 seconds per resources.
https://github.com/nexB/scancode.io/issues/135

- Fix issue with closing modal buttons in the web UI.
https://github.com/nexB/scancode.io/issues/116
https://github.com/nexB/scancode.io/issues/141

-------

- Add support for Docker and VM images using RPMs such as Fedora, CentOS, RHEL,
and openSUSE linux distributions.
https://github.com/nexB/scancode.io/issues/6

- Add a compliance alert system based on license policies provided through a
policies.yml file. The compliance alerts are computed from the license_expression and
stored on the codebase resource. When the policy feature is enabled, the compliance
alert values are displayed in the UI and returned in all the downloadable results.
The enable and setup the policy feature, refer to
https://scancodeio.readthedocs.io/en/latest/scancodeio-settings.html#scancode-io-settings
https://github.com/nexB/scancode.io/issues/90

- Add a new codebase resource detail view including the file content.
Detected value can be displayed as annotation in the file source.
https://github.com/nexB/scancode.io/issues/102

- Download URLs can be provided as inputs on the project form.
Each URL is fetched and added to the project input directory.
https://github.com/nexB/scancode.io/issues/100

- Run celery worker with the "threads" pool implementation.
Implement parallelization with ProcessPoolExecutor for file and package scans.
Add a SCANCODEIO_PROCESSES settings to control the multiprocessing CPUs count.
https://github.com/nexB/scancode.io/issues/70

- Optimize "tag" type pipes using the update() API in place of save() on the QuerySet
iteration.
https://github.com/nexB/scancode.io/issues/70

- Use the extractcode API for the Docker pipeline.
This change helps with performance and results consistency between pipelines.
https://github.com/nexB/scancode.io/issues/70

- Implement cache to prevent scanning multiple times a duplicated codebase resource.
https://github.com/nexB/scancode.io/issues/70

- Create the virtualenv using the virtualenv.pyz app in place of the bundled "venv".
https://github.com/nexB/scancode.io/issues/104

- Consistent ordering for the pipelines, now sorted alphabetically.