Scancodeio

--------------------

- Add support for ScanCode.io results in the "load_inventory" pipeline.
https://github.com/nexB/scancode.io/issues/609

- Add support for CycloneDX 1.4 to the "inspect-manifest" pipeline to import SBOM into
a Project.
https://github.com/nexB/scancode.io/issues/583

- Add fields in CycloneDX BOM output using the component properties.
See registered properties at https://github.com/nexB/aboutcode-cyclonedx-taxonomy
https://github.com/nexB/scancode.io/issues/637

- Upgrade to Python 3.11 in the Dockerfile.
https://github.com/nexB/scancode.io/pull/611

- Refine the "Command Line Interface" documentation about the ``scanpipe`` command
usages in the Docker context.
Add the /app workdir in the "PYTHONPATH" env of the Docker file to make the
``scanpipe`` entry point available while running ``docker compose`` commands.
https://github.com/nexB/scancode.io/issues/616

- Add new tutorial about the "find vulnerabilities" pipeline and the vulnerablecode
integration in the documentation.
https://github.com/nexB/scancode.io/issues/600

- Rewrite the CLI tutorials for a Docker-based installation.
https://github.com/nexB/scancode.io/issues/440

- Use CodebaseResource ``path`` instead of ``id`` as slug_field in URL navigation.
https://github.com/nexB/scancode.io/issues/242

- Remove dead code related to the project_tree view
https://github.com/nexB/scancode.io/issues/623

- Update ``scanpipe.pipes.ProjectCodebase`` and related code to work properly
with current Project/CodebaseResource path scheme.
https://github.com/nexB/scancode.io/pull/624

- Add ``SCANCODEIO_PAGINATE_BY`` setting to customize the number of items displayed per
page for each object type.
https://github.com/nexB/scancode.io/issues/563

- Add setting for per-file timeout. The maximum time allowed for a file to be
analyzed when scanning a codebase is configurable with SCANCODEIO_SCAN_FILE_TIMEOUT
while the maximum time allowed for a pipeline to complete can be defined using
SCANCODEIO_TASK_TIMEOUT.
https://github.com/nexB/scancode.io/issues/593

--------------------

- Upgrade ScanCode-toolkit and related dependencies to solve installation issues.
https://github.com/nexB/scancode.io/pull/586

- Add support for Python 3.11
https://github.com/nexB/scancode.io/pull/611

- Populate ``documentDescribes`` field with Package and Dependency SPDX IDs in
SPDX BOM output.
https://github.com/nexB/scancode.io/issues/564

--------------------

- Add a new "find vulnerabilities" pipeline to lookup vulnerabilities in the
VulnerableCode database for all project discovered packages.
Vulnerability data is stored in the extra_data field of each package.
More details about VulnerableCode at https://github.com/nexB/vulnerablecode/
https://github.com/nexB/scancode.io/issues/101

- Add a new "inspect manifest" pipeline to resolve packages from manifest, lockfile,
and SBOM. The resolved packages are created as discovered packages.
Support PyPI "requirements.txt" files, SPDX document as JSON ".spdx.json",
and AboutCode ".ABOUT" files.
https://github.com/nexB/scancode.io/issues/284

- Generate SBOM (Software Bill of Materials) compliant with the SPDX 2.3 specification
as a new downloadable output.
https://github.com/nexB/scancode.io/issues/389

- Generate CycloneDX SBOM (Software Bill of Materials) as a new downloadable output.
https://github.com/nexB/scancode.io/issues/389

- Display Webhook status in the Run modal.
The WebhookSubscription model was refined to capture delivery data.
https://github.com/nexB/scancode.io/issues/389

- Display the current active step of a running pipeline in the "Pipeline" section of
the project details view, inside the run status tag.
https://github.com/nexB/scancode.io/issues/300

- Add proper pagination for API actions: resources, packages, dependencies, and errors.

- Refine the fields ordering in API Serializers based on the toolkit order.
https://github.com/nexB/scancode.io/issues/546

- Keep the current filters state when submitting a search in list views.
https://github.com/nexB/scancode.io/issues/541

- Improve the performances of the project details view to load faster by deferring the
the charts rendering. This is especially noticeable on projects with a large amount
of codebase resources and discovered packages.
https://github.com/nexB/scancode.io/issues/193

- Add support for filtering by "Other" values when filtering from the charts in the
Project details view.
https://github.com/nexB/scancode.io/issues/526

- ``CodebaseResource.for_packages`` now returns a list of
``DiscoveredPackage.package_uid`` or ``DiscoveredPackage.package_url`` if
``DiscoveredPackage.package_uid`` is not present. This is done to reflect the
how scancode-toolkit's JSON output returns ``package_uid``s in the
``for_packages`` field for Resources.

- Add the model DiscoveredDependency. This represents Package dependencies
discovered in a Project. The ``scan_codebase`` and ``scan_packages`` pipelines
have been updated to create DiscoveredDepdendency objects. The Project API has
been updated with new fields:

- ``dependency_count``
- The number of DiscoveredDependencies associated with the project.

- ``discovered_dependencies_summary``
- A mapping that contains following fields:

- ``total``
- The number of DiscoveredDependencies associated with the project.
- ``is_runtime``
- The number of runtime dependencies.
- ``is_optional``
- The number of optional dependencies.
- ``is_resolved``
- The number of resolved dependencies.

These values are also available on the Project view.
https://github.com/nexB/scancode.io/issues/447

- The ``dependencies`` field has been removed from the DiscoveredPackage model.

- Create directory CodebaseResources in the rootfs pipeline.
https://github.com/nexB/scancode.io/issues/515

- Add ProjectErrors when the DiscoveredPackage could not be fetched using the
provided `package_uid` during the `assemble_package` step instead of failing the whole
pipeline.
https://github.com/nexB/scancode.io/issues/525

- Escape paths before using them in regular expressions in ``CodebaseResource.walk()``.
https://github.com/nexB/scancode.io/issues/525

- Disable multiprocessing and threading by default on macOS ("spawn" start method).
https://github.com/nexB/scancode.io/issues/522

--------------------

- WARNING: Drop support for Python 3.6 and 3.7. Add support for Python 3.10.
Upgrade Django to version 4.1 series.

- Upgrade ScanCode-toolkit to version 31.0.x.
See https://github.com/nexB/scancode-toolkit/blob/develop/CHANGELOG.rst for an
overview of the changes in the v31 compared to v30.

- Implement run status auto-refresh using the htmx JavaScript library.
The statuses of queued and running pipeline are now automatically refreshed
in the project list and project details views every 10 seconds.
A new "toast" type of notification is displayed along the status update.
https://github.com/nexB/scancode.io/issues/390

- Ensure the worker service waits for migrations completion before starting.
To solve this issue we install the wait-for-it script available in
Debian by vishnubob and as suggested in the Docker documentation.
In the docker-compose.yml, we let the worker wait for the web processing
to be complete when gunicorn exposes port 8000 and web container is available.
Reference: https://docs.docker.com/compose/startup-order/
Reference: https://github.com/vishnubob/wait-for-it
Reference: https://tracker.debian.org/pkg/wait-for-it
https://github.com/nexB/scancode.io/issues/387

- Add a "create-user" management command to create new user with its API key.
https://github.com/nexB/scancode.io/issues/458

- Add a "tag" field on the CodebaseResource model.
The layer details are stored in this field in the "docker" pipeline.
https://github.com/nexB/scancode.io/issues/443

- Add support for multiple inputs in the LoadInventory pipeline.
https://github.com/nexB/scancode.io/issues/451

- Add new SCANCODEIO_REDIS_PASSWORD environment variable and setting
to optionally set Redis instance password.

- Ensure a project cannot be deleted through the API while a pipeline is running.
https://github.com/nexB/scancode.io/issues/402

- Display "License clarity" and "Scan summary" values as new panel in the project
details view. The summary is generated during the `scan_package` pipeline.
https://github.com/nexB/scancode.io/issues/411

- Enhance Project list view page:

- 20 projects are now displayed per page
- Creation date displayed under the project name
- Add ability to sort by date and name
- Add ability to filter by pipeline type
- Add ability to filter by run status

https://github.com/nexB/scancode.io/issues/413

- Correctly extract symlinks in docker images. We now use the latest
container-inspector to fix symlinks extraction in docker image tarballs.
In particular broken symlinks are not treated as an error anymore
and symlinks are extracted correctly.
https://github.com/nexB/scancode.io/issues/471
https://github.com/nexB/scancode.io/issues/407

- Add a Package details view including all model fields and resources.
Display only 5 resources per package in the list view.
https://github.com/nexB/scancode.io/issues/164
https://github.com/nexB/scancode.io/issues/464

- Add the ability to filter by empty and none values providing the
"EMPTY" magic value to any filters.
https://github.com/nexB/scancode.io/issues/296

- CodebaseResource.name now contains both the bare file name with extension, as
opposed to just the bare file name without extension.
Using a name stripped from its extension was something that was not used in
other AboutCode project or tools.
https://github.com/nexB/scancode.io/issues/467

- Export current results as XLSX for resource, packages, and errors list views.
https://github.com/nexB/scancode.io/issues/48

- Add support for .tgz extension for input files in Docker pipeline
https://github.com/nexB/scancode.io/issues/499

- Add support for resource missing file content in details view.
Refine the annotation using the new className instead of type.
https://github.com/nexB/scancode.io/issues/495

- Change the worksheet names in XLSX output, using the
"PACKAGES", "RESOURCES", "DEPENDENCIES", and "ERRORS" names.
https://github.com/nexB/scancode.io/issues/511

- Update application Package scanning step to reflect the updates in
scancode-toolkit package scanning.

- Package data detected from a file are now stored on the
CodebaseResource.package_data field.
- A second processing step is now done after scanning for Package data, where
Package Resources are determined and DiscoveredPackages and
DiscoveredDependencies are created.

https://github.com/nexB/scancode.io/issues/444

--------------------

- Add authentication for the Web UI views and REST API endpoint.
The autentication is disabled by default and can be enabled using the
SCANCODEIO_REQUIRE_AUTHENTICATION settings.
When enabled, users have to authenticate through a login form in the Web UI, or using
their API Key in the REST API.
The API Key can be viewed in the Web UI "Profile settings" view ince logged-in.
Users can be created using the Django "createsuperuser" management command.
https://github.com/nexB/scancode.io/issues/359

- Include project errors in XLSX results output.
https://github.com/nexB/scancode.io/issues/364

- Add input_sources used to fetch inputs to JSON results output.
https://github.com/nexB/scancode.io/issues/351

- Refactor the update_or_create_package pipe to support the ProjectError system
and fix a database transaction error.
https://github.com/nexB/scancode.io/issues/381

- Add webhook subscription available when creating project from REST API.
https://github.com/nexB/scancode.io/issues/98

- Add the project "reset" feature in the UI, CLI, and REST API.
https://github.com/nexB/scancode.io/issues/375

- Add a new GitHub action that build the docker-compose images and run the test suite.
This ensure that the app is properly working and tested when running with Docker.
https://github.com/nexB/scancode.io/issues/367

- Add --no-install-recommends in the Dockerfile apt-get install and add the
`linux-image-amd64` package. This packages makes available the kernels
required by extractcode and libguestfs for proper VM images extraction.
https://github.com/nexB/scancode.io/issues/367

- Add a new `list-project` CLI command to list projects.
https://github.com/nexB/scancode.io/issues/365

--------------------

- Remove the --no-install-recommends in the Dockerfile apt-get install to include
required dependencies for proper VM extraction.
https://github.com/nexB/scancode.io/issues/367