Scancode-toolkit

---------

This is a minor release with several bug fixes, major performance improvements
and support for new and improved package formats


Many thanks to every contributors that made this possible and in particular:

- Abhigya Verma abhi27-web
- Ayan Sinha Mahapatra AyanSinhaMahapatra
- Dennis Clark DennisClark
- Jono Yang JonoYang
- Mayur Agarwal mrmayurgithub
- Philippe Ombredanne pombredanne
- Pierre Tardy tardyp


Outputs:
~~~~~~~~

- Add new YAML-formatted output. This is exactly the same data structure as for
the JSON output
- Add new Debian machine readable copyright output.
- The CSV output "Resource" column has been renamed to "path".
- The SPDX output now has the mandatory DocumentNamespace attribute per SPDX specs 2344


Copyright detection:
~~~~~~~~~~~~~~~~~~~~

- The copyright detection speed has been significantly improved with the tests
taking roughly 1/2 of the time to run. This is achieved mostly by replacing
NLTK with a the minimal and simplified subset we need in a new library named
pygmars.

License detection:
~~~~~~~~~~~~~~~~~~~

- Add new licenses: now tracking 1763 licenses
- Add new license detection rules: now tracking 29475 license detection rules
- We have also improved license expression parsing and processing


Package detection:
~~~~~~~~~~~~~~~~~~

- The Debian packages declared license detection has been significantly improved.
- The Alpine packages declared license detection has been significantly improved.
- There is new support for shell parsing and Alpine packages APKBUILD data collection.
- There is new support for various Windows packages detection using multiple
techniques including MSI, Windows registry and several more.
- There is new support for Distroless Debian-like installed packages.
- There is new support for Dart Pub package manifests.

--------

This is a major new release with important security and bug fixes, as well as
significant improvement in license detection.


Many thanks to every contributors that made this possible and in particular:

- Akanksha Garg akugarg
- Ayan Sinha Mahapatra AyanSinhaMahapatra
- Dennis Clark DennisClark
- François Granade farialima
- Hanna Modica hanna-modica
- Jelmer Vernooij jelmer
- Jono Yang JonoYang
- Konrad Weihmann priv-kweihmann
- Philippe Ombredanne pombredanne
- Pierre Tardy tardyp
- Sarita Singh itssingh
- Sebastian Thomas sebathomas
- Steven Esser majurg
- Till Jaeger LeChasseur
- Thomas Druez tdruez



Breaking API changes:
~~~~~~~~~~~~~~~~~~~~~

- The configure scripts for Linux, macOS and Windows have been entirely
refactored and should be considered as new. These are now only native scripts
(.bat on Windows and .sh on POSIX) and the Python script etc/configure.py
has been removed. Use the PYTHON_EXECUTABLE environment variable to point to
alternative non-default Python executable and this on all OSes.


Security updates:
~~~~~~~~~~~~~~~~~

- Update minimum versions and pinned version of thirdparty dependencies
to benefit from latest improvements and security fixes. This includes in
particular this issues:

- pkg:pypi/pygments: (low severity, limited impact) CVE-2021-20270, CVE-2021-27291
- pkg:pypi/lxml: (low severity, likely no impact) CVE-2021-28957
- pkg:pypi/nltk: (low severity, likely no impact) CVE-2019-14751
- pkg:pypi/jinja2: (low severity, likely no impact) CVE-2020-28493, CVE-2019-10906
- pkg:pypi/pycryptodome: (high severity) CVE-2018-15560 (dropped since no
longer used by pdfminer)


Outputs:
~~~~~~~~

- The JSON output packages section has a new "extra_data" attributes which is
a JSON object that can contain arbitrary data that are specific to a package
type.


License detection:
~~~~~~~~~~~~~~~~~~~

- The SPDX license list has been update to 3.13

- Add 42 new and update 45 existing licenses.

- Over 14,300 new and improved license detection rules have been added. A large
number of these (~13,400) are to avoid false positive detection.


Copyright detection:
~~~~~~~~~~~~~~~~~~~~

- Improved speed and fixed some timeout issues. Fixed minor misc. bugs.

- Allow calling copyright detection from text lines to ease integration


Package detection:
~~~~~~~~~~~~~~~~~~

- A new "extra_data" dictionary is now part of the "packages" data in the
returned JSON. This is used to store arbitrary type-specific data that do
cannot be fit in the Package data structure.

- The Debian copyright files license detection has been reworked and
significantly improved.

- The PyPI package detection and manifest parsing has been reworked and
significantly improved.

- The detection of Windows executables and DLLs metadata has been enabled.
These metadata are returned as packages.


Other:
~~~~~~~
- Most third-party libraries have been updated to their newer versions. Some
dependency constraints have been relaxed to help some usage as a library.

- The on-commit CI tests now validate that we can install from PyPI without
problem.

- Fix several installation issues.

- Add new function to detect copyrights from lines.

--------

This is a major version with no breaking API changes. Heads-up: the next version
will bring up some significant API changes summarized above.


Security:
~~~~~~~~~

- Update dependency versions for security fixes.


License scanning:
~~~~~~~~~~~~~~~~~

- Add 22 new licenses and update 71 existing licenses

- Update licenses to include the SPDX license list 3.12

- Improve license detection accuracy with over 2,300 new and updated license
detection rules

- Undeprecate the regexp license and deprecate the hs-regexp-orig license

- Improve license db initial load time with caching for faster scancode
start time

- Add experimental SCANCODE_LICENSE_INDEX_CACHE environment variable to point
to an alternative directory where the license index cache is stored (as
opposed to store this as package data.)

- Ensure that license short names are not more than 50 characters long

- Thank you to:
- Dennis Clark DennisClark
- Chin-Yeung Li chinyeungli
- Armijn Hemmel armijnhemel
- Sarita Singh itssingh
- Akanksha Garg akugarg


Copyright scanning:
~~~~~~~~~~~~~~~~~~~

- Detect SPDX-FileCopyrightText as defined by the FSFE Reuse project
Thank you to Daniel Eder daniel-eder

- Fix bug when using the --filter-clues command line option
Thank you to Van Lindberg VanL

- Fixed copyright truncation bug
Thank you to Akanksha Garg akugarg


Package scanning:
~~~~~~~~~~~~~~~~~

- Add support for installed RPMs detection internally (not wired to scans)
Thank you to Chin-Yeung Li chinyeungli

- Improve handling of Debian copyright files with faster and more
accurate license detection
Thank you to Thomas Druez tdruez

- Add new built-in support for installed_files report. Only available when
used as a library.

- Improve support for RPM, npm, Debian, build scripts (Bazel) and Go packages
Thank you to:
- Divyansh Sharma Divyansh2512
- Jonothan Yang JonoYang
- Steven Esser majurg

- Add new support to collect information from semi-structured Readme files
and related metadata files.
Thank you to Jonothan Yang JonoYang and Steven Esser majurg


Outputs:
~~~~~~~~~

- Add new Debian copyright-formatted output.
Thank you to Jelmer Vernooij jelmer

- Fix bug in --include where directories where not skipped correctly
Thank you to Pierre Tardy tardyp


Misc. and documentation improvements:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- Update the way tests assertions are made
Thank you to Aditya Viki adityaviki

- Thank you to Aryan Kenchappagol aryanxk02

--------

Installation:
~~~~~~~~~~~~~

- Resolve reported installation issues on macOS, Windows and Linux
- Stop using extras for a default wheel installation
- Build new scancode-toolkit-mini package with limited dependencies for use
when packaging in distros and similar
- The new Dockerfile will create smaller images and containers.
Thank you to Viktor Tiulpin tiulpin

License scanning:
~~~~~~~~~~~~~~~~~

- Over 150 new and updated licenses
- Support the latest SPDX license list v3.11
- Improve license detection accuracy with over 740 new and improved license
detection rules
- Fix license cache handling issues

Misc.:
~~~~~~
- Update extractcode, typecode and their native dependencies for better support
of latests versions of macOS.

-------

Security:
~~~~~~~~~

- Update vulnerable LXML to version 4.6.2 to fix
https://nvd.nist.gov/vuln/detail/CVE-2020-27783
This was detected thanks to https://github.com/nexb/vulnerablecode

Operating system support:
~~~~~~~~~~~~~~~~~~~~~~~~~

- Drop support for Python 2 295
- Drop support for 32 bits on Windows 335
- Add support for Python 64 bits on Windows 64 bits 335
- Add support for Python 3.6, 37, 3.8 and 3.9 on Linux, Windows and macOS.
These are now tested on Azure.
- Add deprecation message for native Windows support 2366

License scanning:
~~~~~~~~~~~~~~~~~

- Improve license detection accuracy with over 8400 new license detection rules
added or updated
- Remove the previously deprecated --license-diag option
- Include pre-built license index in release archives to speed up start 988
- Use SPDX LicenseRef-scancode namespace for all licenses keys not in SPDX
- Replace DEJACODE_LICENSE_URL with SCANCODE_LICENSEDB_URL at
https://scancode-licensedb.aboutcode.org #2165
- Add new license flag in license detection results "is_license_intro" that
is used to indicate that a license rule is a short license introduction
statement (that typically may be reported as some unknown license)

Package scanning:
~~~~~~~~~~~~~~~~~

- Add detection of package-installed files
- Add analysis of system package installed databases for Debian, OpenWRT and
Alpine Linux packages
- Add support for Alpine Linux, Debian, OpenWRT.

Copyright scanning:
~~~~~~~~~~~~~~~~~~~

- Improve detection with minor grammar fixes

Misc.:
~~~~~~

- Adopt a new calendar date-based versioning for scancode-toolkit version numbers
- Update thirdparty dependencies and built-in plugins
- Allow installation without extractcode and typecode native plugins. Instead
one can elect to install these or not to have a lighter footprint if needed.
- Update configuration and bootstrap scripts to support a new PyPI-like
repository at https://thirdparty.aboutcode.org/pypi/
- Create new release scripts to populate released archives with just the
required wheels of a given OS and Python version.
- Updated scancode.bat to handle % signs in the arguments 1876

-------------------

Notable changes:
~~~~~~~~~~~~~~~~

- Collect Windows executable metadata 652
- Fix minor bugs
- Add Dockerfile to build docker image from ScanCode sources 2265