Scancodeio

--------------------

- Associate resolved packages with their source codebase resource.
https://github.com/nexB/scancode.io/issues/1140

- Add a new `CollectSourceStrings` pipeline (addon) for collecting source string using
xgettext.
https://github.com/nexB/scancode.io/pull/1160

--------------------

- Add support for Python 3.12 and upgrade to Python 3.12 in the Dockerfile.
https://github.com/nexB/scancode.io/pull/1138

- Add support for CycloneDX XML inputs.
https://github.com/nexB/scancode.io/issues/1136

- Upgrade the SPDX schema to v2.3.1
https://github.com/nexB/scancode.io/issues/1130

--------------------

- Add support for importing CycloneDX SBOM 1.2, 1.3, 1.4 and 1.5 spec formats.
https://github.com/nexB/scancode.io/issues/1045

- The pipeline help modal is now available from all project views: form, list, details.
The docstring are converted from markdown to html for proper rendering.
https://github.com/nexB/scancode.io/pull/1105

- Add a new `CollectSymbols` pipeline (addon) for collecting codebase symbols using
Universal Ctags.
https://github.com/nexB/scancode.io/pull/1116

- Capture errors during the `inspect_elf_binaries` pipeline execution.
Errors on resource inspection are stored as project error message instead of global
pipeline failure.
The problematic resource path is stored in the message details and displayed in the
message list UI as a link to the resource details view.
https://github.com/nexB/scancode.io/issues/1121
https://github.com/nexB/scancode.io/issues/1122

- Use the `package_only` option in scancode `get_package_data` API in
`inspect_packages` pipeline, to skip license and copyright detection in
extracted license and copyright statements found in package metadata.
https://github.com/nexB/scancode-toolkit/pull/3689

- Rename the ``match_to_purldb`` pipeline to ``match_to_matchcode``, and add
MatchCode.io API settings to ScanCode.io settings.

- In the DiscoveredPackage model, rename the "datasource_id" attribute to
"datasource_ids" and add a new attribute "datafile_paths". This is aligned
with the scancode-toolkit Package model, and package detection information
is now stored correctly. Also update the UI for discovered packages to
show the corresponding package datafiles and their datasource IDs.
A data migration is included to facilitate the migration of existing data.
https://github.com/nexB/scancode.io/issues/1099

- Add PurlDB tab, displayed when the PURLDB_URL settings is configured.
When loading the package details view, a request is made on the PurlDB to fetch and
and display any available data.
https://github.com/nexB/scancode.io/issues/1125

- Create a new management command `purldb-scan-queue-worker`, that runs
scancode.io as a Package scan queue worker for PurlDB.
`purldb-scan-queue-worker` gets the next available Package to be scanned and
the list of pipeline names to be run on the Package from PurlDB, creates a
Project, fetches the Package, runs the specified pipelines, and returns the
results to PurlDB.
https://github.com/nexB/scancode.io/pull/1078
https://github.com/nexB/purldb/issues/236

- Update matchcode-toolkit to v4.0.0

--------------------

- Add ability to "group" pipeline steps to control their inclusion in a pipeline run.
The groups can be selected in the UI, or provided using the
"pipeline_name:group1,group2" syntax in CLI and REST API.
https://github.com/nexB/scancode.io/issues/1045

- Refine pipeline choices in the "Add pipeline" modal based on the project context.
* When there is at least one existing pipeline in the project, the modal now includes
all addon pipelines along with the existing pipeline for selection.
* In cases where no pipelines are assigned to the project, the modal displays all
base (non-addon) pipelines for user selection.

https://github.com/nexB/scancode.io/issues/1071

- Rename pipeline for consistency and precision:
* scan_codebase_packages: inspect_packages

Restructure the inspect_manifest pipeline into:
* load_sbom: for loading SPDX/CycloneDX SBOMs and ABOUT files
* resolve_dependencies: for resolving package dependencies
* inspect_packages: gets package data from package manifests/lockfiles

A data migration is included to facilitate the migration of existing data.
Only the new names are available in the web UI but the REST API and CLI are backward
compatible with the old names.
https://github.com/nexB/scancode.io/issues/1034
https://github.com/nexB/scancode.io/discussions/1035

- Remove "packageFileName" entry from SPDX output.
https://github.com/nexB/scancode.io/issues/1076

- Add an add-on pipeline for collecting DWARF debug symbol compilation
unit paths when available from elfs.
https://github.com/nexB/purldb/issues/260

- Extract all archives recursively in the `scan_single_package` pipeline.
https://github.com/nexB/scancode.io/issues/1081

- Add URL scheme validation with explicit error messages for input URLs.
https://github.com/nexB/scancode.io/issues/1047

- All supported `output_format` can now be downloaded using the results_download API
action providing a value for the new `output_format` parameter.
https://github.com/nexB/scancode.io/issues/1091

- Add settings related to fetching private files. Those settings allow to
define credentials for various authentication types.
https://github.com/nexB/scancode.io/issues/620
https://github.com/nexB/scancode.io/issues/203

- Update matchcode-toolkit to v3.0.0

--------------------

- Rename multiple pipelines for consistency and precision:
* docker: analyze_docker_image
* root_filesystems: analyze_root_filesystem_or_vm_image
* docker_windows: analyze_windows_docker_image
* inspect_manifest: inspect_packages
* deploy_to_develop: map_deploy_to_develop
* scan_package: scan_single_package

A data migration is included to facilitate the migration of existing data.
Only the new names are available in the web UI but the REST API and CLI are backward
compatible with the old names.
https://github.com/nexB/scancode.io/issues/1044

- Generate CycloneDX SBOM in 1.5 spec format, migrated from 1.4 previously.
The Package vulnerabilities are now included in the CycloneDX SBOM when available.
https://github.com/nexB/scancode.io/issues/807

- Improve the inspect_manifest pipeline to accept archives as inputs.
https://github.com/nexB/scancode.io/issues/1034

- Add support for "tagging" download URL inputs using the "<fragment>" section of URLs.
This feature is particularly useful in the map_develop_to_deploy pipeline when
download URLs are utilized as inputs. Tags such as "from" and "to" can be specified
by adding "from" or "to" fragments at the end of the download URLs.
Using the CLI, the uploaded files can be tagged using the "filename:tag" syntax
while using the `--input-file` arguments.
In the UI, tags can be edited from the Project details view "Inputs" panel.
On the REST API, a new `upload_file_tag` field is available to use along the
`upload_file`.
https://github.com/nexB/scancode.io/issues/708

--------------------

- Upgrade Django to version 5.0 and drop support for Python 3.8 and 3.9
https://github.com/nexB/scancode.io/issues/1020

- Fetching "Download URL" inputs is now delegated to an initial pipeline step that is
always run as the start of a pipeline.
This allows to run pipelines on workers running from a remote location, external to
the main ScanCode.io app server.
https://github.com/nexB/scancode.io/issues/410

- Migrate the Project.input_sources field into a InputSource model.
https://github.com/nexB/scancode.io/issues/410

- Refactor run_scancode to not fail on scan errors happening at the resource level,
such as a timeout. Project error message are created instead.
https://github.com/nexB/scancode.io/issues/1018

- Add support for the SCANCODEIO_SCAN_FILE_TIMEOUT setting in the scan_package pipeline.
https://github.com/nexB/scancode.io/issues/1018

- Add support for non-archive single file in the scan_package pipeline.
https://github.com/nexB/scancode.io/issues/1009

- Do not include "add-on" pipelines in the "New project" form choices.
https://github.com/nexB/scancode.io/issues/1041

- Display a "Run pipelines" button in the "Pipelines" panel.
Remove the ability to run a single pipeline in favor of running all "not started"
project pipeline.
https://github.com/nexB/scancode.io/issues/997

- In "map_deploy_to_develop" pipeline, add support for path patterns
in About file attributes documenting resource paths.
https://github.com/nexB/scancode.io/issues/1004

- Fix an issue where the pipeline details cannot be fetched when using URLs that
include credentials such as "user:passdomain".
https://github.com/nexB/scancode.io/issues/998

- Add a new pipeline, ``match_to_purldb``, that check CodebaseResources of a
Project against PurlDB for Package matches.