Scancodeio

--------------------

- Upgrade ScanCode-toolkit to latest v32.0.x
Warning: This upgrade requires schema and data migrations (both included).
It is recommended to reset and re-run the pipelines to benefit from the latest
ScanCode detection improvements.
Refer to https://github.com/nexB/scancode-toolkit/blob/develop/CHANGELOG.rst#v3200-next-roadmap
for the full list of changes.
https://github.com/nexB/scancode.io/issues/569

- Add a new ``deploy_to_develop`` pipeline specialized in creating relations between
the development source code and binaries or deployed code.
This pipeline is expecting 2 archive files with "from-" and "to-" filename prefixes
as inputs:
1. "from-[FILENAME]" archive containing the development source code
2. "to-[FILENAME]" archive containing the deployment compiled code
https://github.com/nexB/scancode.io/issues/659

- Add ability to configure a Project through a new "Settings" form in the UI or by
providing a ".scancode-config.yml" configuration file as one of the Project inputs.
The "Settings" form allows to rename a Project, add and edit the notes, as well
as providing a list of patterns to be ignored during pipeline runs, the choice of
extracting archives recursively, and the ability to provide a custom template for
attribution.
https://github.com/nexB/scancode.io/issues/685
https://github.com/nexB/scancode.io/issues/764

- Add ``notes`` field on the Project model. Notes can be updated from the Project
settings form. Also, notes can be provided while creating a project through the CLI
using the a new ``--notes`` option.
https://github.com/nexB/scancode.io/issues/709

- Add a mapper function to relate .ABOUT files during the d2d pipeline.
https://github.com/nexB/scancode.io/issues/740

- Enhance the file viewer UI of the resource details view.
A new search for the file content was added.
Also, it is now possible to expand the file viewer in full screen mode.
https://github.com/nexB/scancode.io/issues/724

- Refine the breadcrumb UI for details view.
https://github.com/nexB/scancode.io/issues/717

- Move the "Resources status" panel from the run modal to the project details view.
https://github.com/nexB/scancode.io/issues/370

- Improve the speed of Project ``reset`` and ``delete`` using the _raw_delete model API.
https://github.com/nexB/scancode.io/issues/729

- Specify ``update_fields`` during each ``save()`` related to Run tasks,
to force a SQL UPDATE in order to avoid any data loss when the model fields are
updated during the task execution.
https://github.com/nexB/scancode.io/issues/726

- Add support for XLSX input in the ``load_inventory`` pipeline.
https://github.com/nexB/scancode.io/issues/735

- Add support for unknown licenses in attribution output.
https://github.com/nexB/scancode.io/issues/749

- Add ``License`` objects to each of the package for attribution generation.
https://github.com/nexB/scancode.io/issues/775

- The "Codebase" panel can now be used to browse the Project's codebase/ directory
and open related resources details view.
https://github.com/nexB/scancode.io/issues/744

--------------------

- Enhance the ``update_or_create_package`` pipe and add the ability to assign multiple
codebase resources at once.
https://github.com/nexB/scancode.io/issues/681

- Add new command line option to create-project and add-input management commands to
copy the content of a local source directory to the project codebase work directory.
https://github.com/nexB/scancode.io/pull/672

- Include the ScanCode-toolkit version in the output headers.
https://github.com/nexB/scancode.io/pull/670

- Enhance the ``output`` management command to support providing multiple formats at
once.
https://github.com/nexB/scancode.io/issues/646

- Improve the resolution of CycloneDX BOM and SPDX document when the file extension is
simply ``.json``.
https://github.com/nexB/scancode.io/pull/688

- Add support for manifest types using ScanCode-toolkit handlers.
https://github.com/nexB/scancode.io/issues/658

- Enhance the Resource details view to use the tabset system and display all
available data including the content viewer.
https://github.com/nexB/scancode.io/issues/215

- Add a "layers" data sheet in the xlsx output for docker pipeline run.
https://github.com/nexB/scancode.io/issues/578

- Move the ``cyclonedx`` and ``spdx`` root modules into the ``pipes`` module.
https://github.com/nexB/scancode.io/issues/657

- Remove the admin app and views.
https://github.com/nexB/scancode.io/issues/645

- Enhance the ``resolve_about_packages`` pipe to handle filename and checksum values.

- Split the pipes unit tests into their own related submodule.

- Upgrade ScanCode Toolkit to v31.2.6
https://github.com/nexB/scancode.io/issues/693

--------------------

- Add support for ScanCode.io results in the "load_inventory" pipeline.
https://github.com/nexB/scancode.io/issues/609

- Add support for CycloneDX 1.4 to the "inspect-manifest" pipeline to import SBOM into
a Project.
https://github.com/nexB/scancode.io/issues/583

- Add fields in CycloneDX BOM output using the component properties.
See registered properties at https://github.com/nexB/aboutcode-cyclonedx-taxonomy
https://github.com/nexB/scancode.io/issues/637

- Upgrade to Python 3.11 in the Dockerfile.
https://github.com/nexB/scancode.io/pull/611

- Refine the "Command Line Interface" documentation about the ``scanpipe`` command
usages in the Docker context.
Add the /app workdir in the "PYTHONPATH" env of the Docker file to make the
``scanpipe`` entry point available while running ``docker compose`` commands.
https://github.com/nexB/scancode.io/issues/616

- Add new tutorial about the "find vulnerabilities" pipeline and the vulnerablecode
integration in the documentation.
https://github.com/nexB/scancode.io/issues/600

- Rewrite the CLI tutorials for a Docker-based installation.
https://github.com/nexB/scancode.io/issues/440

- Use CodebaseResource ``path`` instead of ``id`` as slug_field in URL navigation.
https://github.com/nexB/scancode.io/issues/242

- Remove dead code related to the project_tree view
https://github.com/nexB/scancode.io/issues/623

- Update ``scanpipe.pipes.ProjectCodebase`` and related code to work properly
with current Project/CodebaseResource path scheme.
https://github.com/nexB/scancode.io/pull/624

- Add ``SCANCODEIO_PAGINATE_BY`` setting to customize the number of items displayed per
page for each object type.
https://github.com/nexB/scancode.io/issues/563

- Add setting for per-file timeout. The maximum time allowed for a file to be
analyzed when scanning a codebase is configurable with SCANCODEIO_SCAN_FILE_TIMEOUT
while the maximum time allowed for a pipeline to complete can be defined using
SCANCODEIO_TASK_TIMEOUT.
https://github.com/nexB/scancode.io/issues/593

--------------------

- Upgrade ScanCode-toolkit and related dependencies to solve installation issues.
https://github.com/nexB/scancode.io/pull/586

- Add support for Python 3.11
https://github.com/nexB/scancode.io/pull/611

- Populate ``documentDescribes`` field with Package and Dependency SPDX IDs in
SPDX BOM output.
https://github.com/nexB/scancode.io/issues/564

--------------------

- Add a new "find vulnerabilities" pipeline to lookup vulnerabilities in the
VulnerableCode database for all project discovered packages.
Vulnerability data is stored in the extra_data field of each package.
More details about VulnerableCode at https://github.com/nexB/vulnerablecode/
https://github.com/nexB/scancode.io/issues/101

- Add a new "inspect manifest" pipeline to resolve packages from manifest, lockfile,
and SBOM. The resolved packages are created as discovered packages.
Support PyPI "requirements.txt" files, SPDX document as JSON ".spdx.json",
and AboutCode ".ABOUT" files.
https://github.com/nexB/scancode.io/issues/284

- Generate SBOM (Software Bill of Materials) compliant with the SPDX 2.3 specification
as a new downloadable output.
https://github.com/nexB/scancode.io/issues/389

- Generate CycloneDX SBOM (Software Bill of Materials) as a new downloadable output.
https://github.com/nexB/scancode.io/issues/389

- Display Webhook status in the Run modal.
The WebhookSubscription model was refined to capture delivery data.
https://github.com/nexB/scancode.io/issues/389

- Display the current active step of a running pipeline in the "Pipeline" section of
the project details view, inside the run status tag.
https://github.com/nexB/scancode.io/issues/300

- Add proper pagination for API actions: resources, packages, dependencies, and errors.

- Refine the fields ordering in API Serializers based on the toolkit order.
https://github.com/nexB/scancode.io/issues/546

- Keep the current filters state when submitting a search in list views.
https://github.com/nexB/scancode.io/issues/541

- Improve the performances of the project details view to load faster by deferring the
the charts rendering. This is especially noticeable on projects with a large amount
of codebase resources and discovered packages.
https://github.com/nexB/scancode.io/issues/193

- Add support for filtering by "Other" values when filtering from the charts in the
Project details view.
https://github.com/nexB/scancode.io/issues/526

- ``CodebaseResource.for_packages`` now returns a list of
``DiscoveredPackage.package_uid`` or ``DiscoveredPackage.package_url`` if
``DiscoveredPackage.package_uid`` is not present. This is done to reflect the
how scancode-toolkit's JSON output returns ``package_uid``s in the
``for_packages`` field for Resources.

- Add the model DiscoveredDependency. This represents Package dependencies
discovered in a Project. The ``scan_codebase`` and ``scan_packages`` pipelines
have been updated to create DiscoveredDepdendency objects. The Project API has
been updated with new fields:

- ``dependency_count``
- The number of DiscoveredDependencies associated with the project.

- ``discovered_dependencies_summary``
- A mapping that contains following fields:

- ``total``
- The number of DiscoveredDependencies associated with the project.
- ``is_runtime``
- The number of runtime dependencies.
- ``is_optional``
- The number of optional dependencies.
- ``is_resolved``
- The number of resolved dependencies.

These values are also available on the Project view.
https://github.com/nexB/scancode.io/issues/447

- The ``dependencies`` field has been removed from the DiscoveredPackage model.

- Create directory CodebaseResources in the rootfs pipeline.
https://github.com/nexB/scancode.io/issues/515

- Add ProjectErrors when the DiscoveredPackage could not be fetched using the
provided `package_uid` during the `assemble_package` step instead of failing the whole
pipeline.
https://github.com/nexB/scancode.io/issues/525

- Escape paths before using them in regular expressions in ``CodebaseResource.walk()``.
https://github.com/nexB/scancode.io/issues/525

- Disable multiprocessing and threading by default on macOS ("spawn" start method).
https://github.com/nexB/scancode.io/issues/522

--------------------

- WARNING: Drop support for Python 3.6 and 3.7. Add support for Python 3.10.
Upgrade Django to version 4.1 series.

- Upgrade ScanCode-toolkit to version 31.0.x.
See https://github.com/nexB/scancode-toolkit/blob/develop/CHANGELOG.rst for an
overview of the changes in the v31 compared to v30.

- Implement run status auto-refresh using the htmx JavaScript library.
The statuses of queued and running pipeline are now automatically refreshed
in the project list and project details views every 10 seconds.
A new "toast" type of notification is displayed along the status update.
https://github.com/nexB/scancode.io/issues/390

- Ensure the worker service waits for migrations completion before starting.
To solve this issue we install the wait-for-it script available in
Debian by vishnubob and as suggested in the Docker documentation.
In the docker-compose.yml, we let the worker wait for the web processing
to be complete when gunicorn exposes port 8000 and web container is available.
Reference: https://docs.docker.com/compose/startup-order/
Reference: https://github.com/vishnubob/wait-for-it
Reference: https://tracker.debian.org/pkg/wait-for-it
https://github.com/nexB/scancode.io/issues/387

- Add a "create-user" management command to create new user with its API key.
https://github.com/nexB/scancode.io/issues/458

- Add a "tag" field on the CodebaseResource model.
The layer details are stored in this field in the "docker" pipeline.
https://github.com/nexB/scancode.io/issues/443

- Add support for multiple inputs in the LoadInventory pipeline.
https://github.com/nexB/scancode.io/issues/451

- Add new SCANCODEIO_REDIS_PASSWORD environment variable and setting
to optionally set Redis instance password.

- Ensure a project cannot be deleted through the API while a pipeline is running.
https://github.com/nexB/scancode.io/issues/402

- Display "License clarity" and "Scan summary" values as new panel in the project
details view. The summary is generated during the `scan_package` pipeline.
https://github.com/nexB/scancode.io/issues/411

- Enhance Project list view page:

- 20 projects are now displayed per page
- Creation date displayed under the project name
- Add ability to sort by date and name
- Add ability to filter by pipeline type
- Add ability to filter by run status

https://github.com/nexB/scancode.io/issues/413

- Correctly extract symlinks in docker images. We now use the latest
container-inspector to fix symlinks extraction in docker image tarballs.
In particular broken symlinks are not treated as an error anymore
and symlinks are extracted correctly.
https://github.com/nexB/scancode.io/issues/471
https://github.com/nexB/scancode.io/issues/407

- Add a Package details view including all model fields and resources.
Display only 5 resources per package in the list view.
https://github.com/nexB/scancode.io/issues/164
https://github.com/nexB/scancode.io/issues/464

- Add the ability to filter by empty and none values providing the
"EMPTY" magic value to any filters.
https://github.com/nexB/scancode.io/issues/296

- CodebaseResource.name now contains both the bare file name with extension, as
opposed to just the bare file name without extension.
Using a name stripped from its extension was something that was not used in
other AboutCode project or tools.
https://github.com/nexB/scancode.io/issues/467

- Export current results as XLSX for resource, packages, and errors list views.
https://github.com/nexB/scancode.io/issues/48

- Add support for .tgz extension for input files in Docker pipeline
https://github.com/nexB/scancode.io/issues/499

- Add support for resource missing file content in details view.
Refine the annotation using the new className instead of type.
https://github.com/nexB/scancode.io/issues/495

- Change the worksheet names in XLSX output, using the
"PACKAGES", "RESOURCES", "DEPENDENCIES", and "ERRORS" names.
https://github.com/nexB/scancode.io/issues/511

- Update application Package scanning step to reflect the updates in
scancode-toolkit package scanning.

- Package data detected from a file are now stored on the
CodebaseResource.package_data field.
- A second processing step is now done after scanning for Package data, where
Package Resources are determined and DiscoveredPackages and
DiscoveredDependencies are created.

https://github.com/nexB/scancode.io/issues/444