Scons

From Joseph Brill:
- Verify that a user specified msvc script (via MSVC_USE_SCRIPT) exists and raise an exception
when the user specified msvc script does not exist.
- Fix issue where if you only had mingw installed on a Windows system and no MSVC compiler, and
did not explicitly request the mingw tool, mingw tool initialization would fail and set the
default compiler to MSVC which wasn't installed, yielding broken build.
Updated mingw tool so that the generate and exists methods use the same mingw search paths
(issue 4134).
- Update the debug output written to stdout for MSVC initialization which is enabled by setting
SCONS_MSCOMMON_DEBUG=- to use the logging module. Also changed the debug output format
written to stdout to include more information about the source for each message of MSVC
initialization debugging output. A single space was added before the message for all
debugging output records written to stdout and to files.
- Refactor the data definitions for msvc configurations to allow derived data structures to be
constructed during initialization that removes the need for special case handling during
runtime execution. Special case handling of host/target combinations is eliminated and
replaced with pre-computed search lists that implicitly handle the differences between full
versions and express versions of msvc. This fixes an issue where Express versions of the MSVC
compiler were not detected due to differences in initial msvc detection and msvc batch file
determination when configuring the build environment. This could lead to build failures when
only an MSVC Express instance is installed and the MSVC version is not explicitly specified
(issue 2668 and issue 2697).
- Added MSVC_USE_SETTINGS construction variable to pass a dictionary to configure the msvc compiler
system environment as an alternative to bypassing Visual Studio autodetection entirely.
- Added MSVC_SDK_VERSION construction variable which allows building with a specific Microsoft
SDK version. This variable is used with the msvc batch file determined via autodetection subject
to validation constraints. Refer to the documentation for additional requirements and validation
details.
- Added MSVC_TOOLSET_VERSION construction variable which allows building with a specific toolset
version. This variable is used with the msvc batch file determined via autodetection subject to
validation constraints. This variable does not affect the autodetection and selection of msvc
instances. The toolset version is applied after an msvc instance is selected. This could be the
default version of msvc. Refer to the documentation for additional requirements and validation
details. Addresses issue 3265, issue 3664, and pull request 4149.
- Added MSVC_SPECTRE_LIBS construction variable which allows building with spectre-mitigated
Visual C++ libraries. This variable is used with the msvc batch file determined via autodetection
subject to validation constraints. Refer to the documentation for additional requirements and
validation details.
- Added MSVC_SCRIPT_ARGS construction variable which specifies command line arguments that are
passed to the msvc batch file determined via autodetection subject to validation constraints.
Refer to the documentation for additional requirements and validation details. Addresses
enhancement issue 4106.
- An exception is raised when MSVC_UWP_APP is enabled for Visual Studio 2013 and earlier.
Previous behavior was to silently ignore MSVC_UWP_APP when enabled for Visual Studio 2013
and earlier. Refer to the documentation for additional requirements and validation details.
MSVC_UWP_APP was extended to accept True, False, and None in addition to '1' and '0'.
- The imported system environment variable names for MSVC 7.0 and 6.0 have been changed to the
names set by their respective installers. Prior to this change, bypassing MSVC detection by
specifying the MSVC 7.0 batch file directly would fail due to using an erroneous environment
variable name. Arguments are no longer passed to the MSVC 6.0 to 7.1 batch files as no
arguments are required and could improve the effectiveness of the internal MSVC cache.
- Propagate the OS and windir environment variables from the system environment to the msvc
environment. The OS and windir environment variables are used in the MSVC 6.0 batch file
and the SDK 6.0-7.1 SetEnv.cmd batch files. Inclusion of the OS and windir environment
variables eliminates some partial paths and warnings generated by the MSVC 6.0 and SDK
6.0-7.1 batch files when the variables are not defined.
Note: Attempting to run the SDK 6.0-7.1 batch files directly via MSVC_USE_SCRIPT can lead to
build failures and/or incomplete build environments. The SDK 6.0-7.1 batch files
require delayed expansion to be enabled which is currently not supported and is
typically not enabled by default on the host system. The batch files may also require
environment variables that are not included by default in the msvc environment.
- Suppress issuing a warning when there are no installed Visual Studio instances for the default
tools configuration (issue 2813). When msvc is the default compiler because there are no
compilers installed, a build may fail due to the cl.exe command not being recognized. At
present, there is no easy way to detect during msvc initialization if the default environment
will be used later to build a program and/or library. There is no error/warning issued for the
default tools as there are legitimate SCons uses that do not require a c compiler.
- Added a global policy setting and an environment construction variable for specifying the
action to be taken when an msvc request cannot be satisfied. The available options are "error",
"exception", "warning", "warn", "ignore", and "suppress". The global policy variable may be
set and retrieved via the functions msvc_set_notfound_policy and msvc_get_notfound_policy,
respectively. These two methods may be imported from SCons.Tool.MSCommon. The environment
construction variable is MSVC_NOTFOUND_POLICY. When defined, the environment construction
variable overrides the global policy setting for a given environment. When the active policy
is "error" or "exception", an MSVCVersionNotFound exception is raised. When the active policy
is "warning" or "warn", a VisualCMissingWarning warning is issued and the constructed
environment is likely incomplete. When the active policy is "ignore" or "suppress", no action
is taken and the constructed environment is likely incomplete. As implemented, the default
global policy is "warning". The ability to set the global policy via an SCons command-line
option may be added in a future enhancement.
- Added a global policy setting and an environment construction variable for specifying the
action to be taken when msvc script errors are detected. The available options are "error",
"exception", "warning", "warn", "ignore", and "suppress". The global policy variable may be
set and retrieved via the functions msvc_set_scripterror_policy and msvc_get_scripterror_policy,
respectively. These two methods may be imported from SCons.Tool.MSCommon. The environment
construction variable is MSVC_SCRIPTERROR_POLICY. When defined, the environment construction
variable overrides the global policy setting for a given environment. When the active policy
is "error" or "exception", an MSVCScriptExecutionError exception is raised when msvc batch file
errors are detected. When the active policy is "warning" or "warn", an MSVCScriptExecutionWarning
warning is issued when msvc batch file errors are detected. When the active policy is "ignore" or
"suppress", msvc batch error messages are suppressed. As implemented, the default global policy
is "ignore". The ability to set the global policy via an SCons command-line option may be added
in a future enhancement.
- Added experimental function msvc_query_version_toolset to SCons.Tool.MSCommon. Given a version
specification, this function will return an msvc version and an msvc toolset version. The msvc
toolset version may be None. The msvc version and msvc toolset version can be used in the
environment construction variables MSVC_VERSION and MSVC_TOOLSET_VERSION, respectively. The
version specification may be an msvc version or an msvc toolset version. This is a proxy for
using an msvc toolset version to select an msvc instance. This function may be removed when an
msvc toolset version is used during msvc instance selection.
- Modify the MSCommon logger configuration to be independent of the root logger. This fixes an issue
when multiple loggers are created and the MSCommon logger added computed fields to the root logger
that are not present in other logging instances.
- Modify the MSVC_USE_SCRIPT_ARGS test fixture to disable the msvc cache. This fixes an issue where
the MSVC_USE_SCRIPT_ARGS test for success relied on a debug log message that was not produced when
the msvc cache file exists and the test keys are already in the cache as the msvc script invocation
was bypassed.

From William Deegan:
- Fix check for unsupported Python version. It was broken. Also now the error message
will include what is the minimum supported version of Python
- Fix ActionTests to work with python 3.10.1 (and higher)
NOTE: If you build with Python 3.10.0 and then rebuild with 3.10.1 (or higher), you may
see unexpected rebuilds. This is due to Python internals changing which changed
the signature of a Python Action Function.
- Fix a number of Python ResourceWarnings which are issued when running SCons and/or it's tests
with python 3.9 (or higher)
- Action._subproc() can now be used as a python context manager to ensure that the
POpen object is properly closed.
(Thanks to Mats Wichmann for catching that DummyPopen needed additional logic)
- Added project_url for mailing lists and Discord
- Updated project url in steup.cfg to be https instead of http
- Updated setup.cfg to remove Python 3.5 and add Python 3.10
- Added default values for source and target arguments to _defines() function. This
is used to expand CPPDEFINES (and others). Previous change added those arguments
with no defaults, so old usage where _defines() was called without source and target
arguments would yield an exception. This issue was found via qt4 and qt5 tools in
scons-contrib https://github.com/SCons/scons-contrib/issues/45

From David H:
- Add JavaScanner to include JAVACLASSPATH as a dependency when using the Java tool.
- Fix incorrect Java classpath generation when a NodeList is used as part of any JAVA*PATH variables.

From Daniel Moody:
- Add cache-debug messages for push failures.
- Ninja: Changed generated build.ninja file to run SCons only build Actions via
a SCons Deamon. Added logic for starting and connecting to SCons daemon (currently
only used for ninja)
- Ninja: Fix issue where Configure files weren't being properly processed when build run
via ninja.
- Ninja: Added ninja mingw support and improved ninja CommandGeneratorAction support.
- Ninja: Update ninja file generation to only create response files for build commands
which exceed MAXLINELENGTH
- Ninja: Added NINJA_GENERATED_SOURCE_ALIAS_NAME which allows user to specify an
Alias() which the ninja tool can use to determine which files are generated sources.
If this is not set by the user then the ninja tool will still dynamically determine
which files are generated sources based on NINJA_GENERATED_SOURCE_SUFFIXES, and create
a phony target _ninja_generated_sources. Generated sources will be built first by
ninja. This is needed because ninja cannot determine which generated sources are
required by other build targets. Code contributed by MongoDB
The downstream commit is here:
https://github.com/mongodb/mongo/commit/2fef432fa6e7cf3fd4f22ba3b193222c2887f14f
- Ninja: Added special case for ninja scons daemon to work in win32 python3.6 environments.
This particular environment does a bad job managing popen standard file handles, so
some special workarounds are needed.
- Ninja:Added user configurable setting of ninja depfile format via NINJA_DEPFILE_PARSE_FORMAT.
Now setting NINJA_DEPFILE_PARSE_FORMAT to [msvc,gcc,clang] can force the ninja expected
format. Compiler tools will also configure the variable automatically.
- Ninja: Made ninja tool force the ninja file as the only target.
- Ninja: Improved the default targets setup and made sure there is always a default target for
the ninja file, which excludes targets that start and stop the daemon.
- Ninja: Update ninja tool so targets passed to SCons are propagated to ninja when scons
automatically executes ninja.
- Small refactor of scons daemons using a shared StateInfo class for communication
between the scons interactive thread and the http server thread. Added error handling
for scons interactive failing to startup.
- Ninja: Updated ninja scons daemon scripts to output errors to stderr as well as the daemon log.
- Ninja: Fix typo in ninja scons daemon startup which causes ConnectionRefusedError to not retry
- Added SHELL_ENV_GENERATORS construction variable. This variable should be set to a list
(or an iterable) which contains functions to be called in order
when constructing the execution environment (Generally this is the shell environment
variables). This allows the user to customize how (for example) PATH is constructed.
Note that these are called for every build command run by SCons. It could have considerable
performance impact if not used carefully.
to connect to the server during start up.
- lex: Fixed an issue with the lex tool where file arguments specified to either "--header-file="
or "--tables-file=" which included a space in the path to the file would be processed incorrectly
- Ninja: added option "--skip-ninja-regen" to enable skipping regeneration of the ninja file
if scons can determine the ninja file doesnot need to be regenerated, which will also
skip restarting the scons daemon. Note this option is could result in incorrect rebuilds
if scons Glob or scons generated files are used in ninja build target's command lines.
- Ninja: Added new alias "shutdown-ninja-scons-daemon" to allow ninja to shutdown the daemon.
Also added cleanup to test framework to kill ninja scons daemons and clean ip daemon logs.
NOTE: Test for this requires python psutil module. It will be skipped if not present.
- Ninja: Added command line variable NINJA_CMD_ARGS that allows to pass through ninja command line args.
This can also be set in your Environment().

From Mats Wichmann:
- Tweak the way default site_scons paths on Windows are expressed to
conform to conventions (what they actually resolve to is unchanged),
drop a Py2 workaround, and pick a better "system" path, old one
remains supported (%AllUsersProfile%\scons\site_scons vs old
%AllUsersProfile%\Application Data\scons\site_scons).
- Fix testsuite to work on Windows systems where there is no usable
association for running .py files directly. There are a few tests where
we need to do this for internal reasons, those are skipped in that case.
Bad association could mean some other tool took it over (Visual
Studio Code is known to do this), or no association at all.
- Updated debug code in MSVC and MSVS tools to conform to the
suggested "lazy interpolation" use of the Python logging module.
Calls now look like 'debug("template %s", text)' rather than
'debug("template %s" % text)' so the logging system does the
interpolation only when/if needed (was a pylint warning).
- Update Help (-H) output a bit. Drop "ignored for compat" entry.
Pass window size to formatter so it formats for wider displays too.
- runtest.py now accepts -j 0 to auto-detect number of usable
processors for testing threads.
- Fixed crash in C scanner's dictify_CPPDEFINES() function which happens if
AppendUnique is called on CPPPATH. (Issue 4108).
- The MSVC script_env_cache now contains a sanity check: if the retrieved
tools path does not exist, the entry is invalidated so it will
be recomputed, in an attempt to avoid scons failing when certain
compiler version bumps have taken place. The dictionary key (uses
the name of a batch file and any arguments which may have been
passes), is now computed a bit differently: the dashes are left
off if there are no arguments. The default cachefile is changed
to have a .json suffix, for better recognition on Windows since
the contents are json.
- As "code modernization" all of SCons now uses the current super()
zero-argument syntax instead of direct calls to a parent class method
or the super() two-argument syntax.
- Renamed ParseFlag's internal data structure to "mapping" instead of
"dict" (avoid redefining builtin)
- Fix an old use-before-set bug in tex tool (issue 2888)
- Fix a test harness exception returning stderr if a wait_for timed out.
- ParseConfig now correctly passes the *unique* flag to a user-supplied
flag-merging function.
- Restore the ability of the content-timestamp decider to see that a
a source which is a symlink has changed if the file-system target of
that link has been modified (issue 3880)
- Modernize a few tests that use now-deprecated unittest.getTestCaseNames
and unittest.makeSuite - Python itself suggests the replacements.
- SCons.Tool.find_program_path now takes an optional add_path argument
to add a path to the execution environment if it was discovered in
default_paths. Previously, the routine, called by many tool modules,
never altered the execution environment, leaving it to the tools.
- A new construction variable FORTRANCOMMONFLAGS is added which is
applied to all Fortran dialects, in case someone needs to set some
flags globally. FORTRANFLAGS looked like it was intended for that,
but was not applied to other dialects, and e2e tests explicitly checked
that FORTRANFLAGS did not propagate outside the FORTRAN dialect,
so the conclusion is that behavior is intentional (issue 2257)
- SCons programmatic importing (tool modules and platform modules)
no longer uses the deprecated (since Py 3.10) importlib.load_module
routine, shifting to the preferred exec_module. Old Python 2 compatible
import fallback (using the imp module) in tool module loading is dropped.
Tool module loading no longer special-cases Jython, which is a dead
project as far as SCons (no timeline in sight for Python 3 support).
- Improvements to lex and yacc tools: better documentation of
extra-file options, add test for extra-file behavior.
- Two new construction variables are introduced for lex (LEX_HEADER_FILE
and LEX_TABLES_FILE) as the preferred way of specifying these extra-file
options.
- Two new construction variables are introduced for yacc
(YACC_HEADER_FILE and YACC_GRAPH_FILE) as the preferred way of
specifying these extra-file options.


From Zhichang Yu:
- Added MSVC_USE_SCRIPT_ARGS variable to pass arguments to MSVC_USE_SCRIPT.
- Added Configure.CheckMember() checker to check if struct/class has the specified member.

From Ivan Kravets, PlatformIO:
- Conditional C/C++ Preprocessor: Strip shell's backslashes from the computed include (-DFOO_H=\"foo.h\")

From Jacob Cassagnol:
- Default hash algorithm check updated for SCons FIPS compliance. Now checks for hash viability
first and then walks the tree to use the first viable hash as the default one. This typically
selects SHA1 on FIPS-enabled systems less than Python 3.9 as the new default instead of MD5,
unless SHA1 has also been disabled by security policy, at which point SCons selects SHA256
as the default. For systems running Python 3.9 and later, the hashlib bug has been fixed,
and SCons will once again default to MD5 as the preferred algorithm.

From Joseph Brill:
- Fix MSVS tests (vs-N.N-exec.py) for MSVS 6.0, 7.0, and 7.1 (import missing module).
- Add support for Visual Studio 2022.

From William Deegan:
- Fix reproducible builds. Restore logic respecting SOURCE_DATE_EPOCH when set.
- Fix version tests to work with updated scons --version output. (Date format changed)
- Fix issue 4021. Change the way subst() is used in Textfile() to not evaluate '$$(' -> '$',
but instead it should yield '$('.
- Change SCons.Platform.win32.get_architecture() to return platform.platform() when run in an
environment where neither: PROCESSOR_ARCHITEW6432 nor PROCESSOR_ARCHITECTURE is set.
This should fix platform tests which started failing when HOST_OS/HOST_ARCH changes
introduced by Aaron Franke (listed below) were merged.
- Further PCH updates. It's now recommended that env['PCH'] should always be a File node.
Either via return value from env.PCH() or by explicitly using File('StdAfx.pch').
- Added --no-ignore-skips to runtest.py. Changed default to ignore skips when setting
runtest.py's exit status. Previously would exit 2 if any tests were skipped.
Now will only exit 2 if user specifies --no-ignore-skips and some tests were skipped.

From Ryan Egesdahl:
- Small fix to ensure CLVar default value is an empty list.
See MongoDB bug report: https://jira.mongodb.org/browse/SERVER-59656
Code contributed by MongoDB.
- Ninja - Fixed an issue where if you control-c and/or killed ninja while it was running scons to
regenerate build.ninja you would end up with no build.ninja file and have to rerun scons from scratch.
Code contributed by MongoDB.

From Aaron Franke:
- Define HOST_OS and HOST_ARCH in the environment for all platforms.
Before this change, these were only defined for Win32 and OS/2.

From Daniel Moody:
- Fix ninja tool to never use for_sig substitution because ninja does not use signatures. This
issue affected CommandGeneratorAction function actions specifically.
- Expanded ninja Mkdir to also support Mkdir actions.
- Added support for the PCH environment variable to support subst generators.
- Fix command line escaping for ninja dollar sign escape. Without escaping ninja properly,
the ninja file scons regenerate and callback invocations will lose the $ characters used in
the scons command line which ninja uses itself for escaping. For Example:
scons BUILD=xyz OTHERVAR=$BUILD
Prior to this fix, it would cause ninja to fail to escape the dollar sign, leading to the
single dollar sign being used as a ninja escape character in the ninja file.

From Daniel Moody:
- Added ninja API 'NINJA_FORCE_SCONS_BUILD' to force a node to callback to scons.

From Mats Wichmann:
- Two small Python 3.10 fixes: one more docstring turned into raw
because it contained an escape; updated "helpful" syntax error message
from 3.10 was not expected by SubstTests.py and test/Subst/Syntax.py
- EmitterProxy rich comparison set is completed (checker warning).
Added __le__, __gt__, __ge__.
- Fix gcc/g++ tool failing if "gcc --version" returns text which fails
to_String conversion (i.e., not UTF-8) - failure happens when tool
initialization checks version. For gcc, the initial version string is
not translated, for the rest, don't convert, just consume raw and discard.
- Maintenance and doc: modernize some usage in Scanner package,
calling super(), switching some imitialization to comprehensions,
and code formatting. Docstring for scanner Base moved from
init-method to class-level so it's picked up by Sphinx.
Added new sconsign filenames to skip_entry_list in Scanner/Dir.py
- Change SCons.Scanner.Base to ScannerBase. Old name kept as an alias
but is now unused in SCons itself.
- Call Variables option converter consistently - the converter should
have access to the env if it needs to (issue 2064).
- Fixed the variables Add() method to accept a tuple for the variable
name the same way AddVariables() does (issue 3869).
- The premade validator PathIsDirCreate for for PathVariable now catches
the case where the directory could not be created due to permission
problems, allowing a more helpful error to be emitted (issue 2828)
- Maintenance: Python thread.setDaemon is deprecated in favor of
directly updating daemon attribute - update SCons to do this.
- Make sure when subst'ing a callable, the callable is called with
the correct for_signature value, previously it would be true even
if doing SUBST_RAW (issue 4037)
- Update Util/NodeList implementation to get rid of a workaround for
early Python 3 slicing issue that is no longer a problem.
- Rework some Java tests to skip rather than fail on CI systems, where
the working java is > v9, but a 1.8 or 9 was also found.
- Java updates: on Windows, detect more default JDK install locations.
On all platforms, more Java versions (up to 17.0 now). Add more information
on version selection to docs.
Update docs on JavaH tool in light of javah command dropped since 10.0.
Try to be better about preserving user's passed-in JAVA* construction vars.
- Start the deprecation of the qt tool, which refers to Qt3 (usupported
since around 2006). There's a deprecation warning added, initially
defaulting to disabled.

From Brian Quistorff:
- Fix crash when scons is run from a python environement where a signal
is set from outside Python.

From Byron Platt:
- Fix Install() issue when copytree recursion gives bad arguments that can
lead to install side-effects including keeping dangling symlinks and
silently failing to copy directories (and their subdirectories) when the
directory already exists in the target.

From Joseph Brill:
- Internal MSVS update: Remove unnecessary calls to find all installed versions of msvc
when constructing the installed visual studios list.

From William Deegan:
- Improve Subst()'s logic to check for proper callable function or class's argument list.
It will now allow callables with expected args, and any extra args as long as they
have default arguments. Additionally functions with no defaults for extra arguments
as long as they are set using functools.partial to create a new callable which set them.
- Fix Issue 3035 - mingw with SHLIBVERSION set fails with either not a dll error or
"Multiple ways to build the same target were specified for:". Now mingw will disable
creating the symlinks (and adding version string to ) dlls. It sets SHLIBNOVERSIONSYMLINKS,
IMPLIBNOVERSIONSYMLINKS and LDMODULENOVERSIONSYMLINKS to True.
- Added --experimental flag, to enable various experimental features/tools. You can specify
'all', 'none', or any combination of available experimental features.
- Fix Issue 3933 - Remove unguarded print of debug information in SharedLibrary logic when
SHLIBVERSION is specified.
- Fix versioned shared library naming for MacOS platform. (Previously was libxyz.dylib.1.2.3,
has been fixed to libxyz.1.2.3.dylib. Additionally the sonamed symlink had the same issue,
that is now resolved as well)
- Add experimental ninja builder. (Contributed by MongoDB, Daniel Moody and many others).
- Fix 3955 - _LIBDIRFLAGS leaving $( and $) in *COMSTR output. Added affect_signature flag to
_concat function. If set to False, it will prepend and append $( and $). That way the various
Environment variables can use that rather than "$( _concat(...) $)".
- Fix issue with exparimental ninja tool which would fail on windows or when ninja package wasn't
installed but --experimental=ninja was specified.
- As part of experimental ninja tool, allow SetOption() to set both disable_execute_ninja and
disable_ninja.

From David H:
- Fix Issue 3906 - `IMPLICIT_COMMAND_DEPENDENCIES` was not properly disabled when
set to any string value (For example ['none','false','no','off'])
Also previously 'All' wouldn't have the desired affect.

From Ivan Kravets:
- Provide a custom argument escape function for `TempFileMunge` using a new
`TEMPFILEARGESCFUNC` variable. Useful if you need to apply extra operations on
a command argument before writing to a temporary file (fix Windows slashes,
normalize paths, etc.)

From Henrik Maier:
- DocbookXslt tool: The XSLT stylesheet file is now initialized to an env.File() Node,
such that dependencies work correctly in hierarchical builds (eg when using
DocbookXslt in SConscript('subdir/SConscript') context.

From Daniel Moody:
- Update CacheDir to use uuid for tmpfile uniqueness instead of pid.
This fixes cases for shared cache where two systems write to the same
cache tmpfile at the same time because the happened to get the same pid.
- Added support for passing custom CacheDir derived classes to SCons. Moved
copy_from_cache attribute from the Environment class to CacheDir class.
Code contributed by MongoDB.
- Update BuildTask to pass all targets to the progress object fixing an issue
where multi-target build nodes only got the first target passed to the progress
object.
- Fix a potential race condition in shared cache environments where the permissions are
not writeable for a moment after the file has been renamed and other builds (users) will copy
it out of the cache. Small reorganization of logic to copy files from cachedir. Moved CacheDir
writeable permission code for copy to cache behind the atomic rename operation.
- Added marking of intermediate and and multi target nodes generated from SConf tests so that
is_conftest() is more accurate.
- Added test for configure check failing to ensure it didn't break generating and running ninja.


From Mats Wichmann:
- Initial support in tests for Python 3.10 - expected bytecode and
one changed expected exception message. Change some more regexes
to be specified as rawstrings in response to DeprecationWarnings.
- Add an example of adding an emitter to User Guide (concept
from Jeremy Elson)
- Add timing information for sconsign database dump when --debug=time
is selected. Also switch to generally using time.perf_counter,
which is the Python recommended way for timing short durations.
- Drop remaining definitions of dict-like has_key methods, since
Python 3 doesn't have a dictionary has_key (maintenance)
- Do not treat --site-dir=DIR and --no-site-dir as distinct options.
Allows a later instance to override an earlier one.
- Ignore empty cmdline arguments when computing targets (issue 2986)
- Remove long-deprecated construction variables PDFCOM, WIN32_INSERT_DEF,
WIN32DEFPREFIX, WIN32DEFSUFFIX, WIN32EXPPREFIX, WIN32EXPSUFFIX.
All have been replaced by other names since at least 1.0.
- Add a __iadd__ method to the CLVar class so that inplace adds
(+=) also work as expected (issue 2399)
- Remove local copy of CLVar in EnvironmentTests unittest file -
should be testing against the production version, and they
didn't really differ.
- Don't strip spaces in INSTALLSTR by using raw subst (issue 2018)
- Deprecate Python 3.5 as a supported version.
- CPPDEFINES now expands construction variable references (issue 2363)
- Restore behavior that Install()'d files are writable (issue 3927)
- Simplified Mkdir(), the internal mkdir_func no longer needs to handle
existing directories, it can now pass exist_ok=True to os.makedirs().
- Avoid WhereIs exception if user set a tool name to empty (from issue 1742)
- Maintenance: remove obsolete __getslice__ definitions (Py3 never calls);
add Node.fs.scandir to call new (Py3.5) os.scandir; Node.fs.makedirs
now passes the exist_ok flag; Cachedir creation now uses this flag.
- Maintenance: remove unneeded imports and reorganize some. Fix uses
of warnings in some tools which instantiated the class but did nothing
with them, need to instead call SCons.Warnings.warn with the warn class.
- Drop overridden changed_since_last_build method in Value class.
- Resync the SetOption implementation and the manpage, making sure new
options are available and adding a notes column for misc information.
SetOption equivalents to --hash-chunksize, --implicit-deps-unchanged
and --implicit-deps-changed are enabled.
- Add tests for SetOption failing on disallowed options and value types.
- Maintenance: eliminate lots of checker complaints about Util.py.
- Maintenance: fix checker-spotted issues in Environment (apply_tools)
and EnvironmentTests (asserts comparing with self).
For consistency, env.Tool() now returns a tool object the same way
Tool() has done.
- Change SConscript() missing SConscript behavior - if must_exist=False,
the warning is suppressed.
- Make sure TEMPFILEPREFIX can be set to an empty string (issue 3964)

From Dillan Mills:
- Add support for the (TARGET,SOURCE,TARGETS,SOURCES,CHANGED_TARGETS,CHANGED_SOURCES}.relpath property.
This will provide a path relative to the top of the build tree (where the SConstruct is located)
Fixes 396

From Andrew Morrow:
- Fix issue 3790: Generators in CPPDEFINES now have access to populated source
and target lists

From James Benton:
- Add COMPILATIONDB_PATH_FILTER env option for CompilationDatabase() builder which allows
filtering of entries based on the output file paths using glob style file matching (issue 3742).

From Joseph Brill:
- Internal MSVC and test updates: Rework the msvc installed versions cache so that it
is not exposed externally and update external references accordingly.
- Modify the MSCommon internal-use only debug logging records to contain the correct relative
file path when the debug function is called from outside the MSCommon module.

From William Deegan:
- Fix yacc tool, not respecting YACC set at time of tool initialization.
- Refactor SCons.Tool to move all common shared and loadable module linking logic to SCons.Tool.linkCommon
- Remove pywin32 imports from SCons.Script.Main. No longer needed.
- Switch to use ctypes instead of pywin32 (requiring an extra pip install) - Fixes Github Issue 2291
- pywin32 no longer necessary for SCons install. (pip install SCons will no longer also require pywin32 on win32)
- Remove pywin32 usage from SCons.Util where it was used for accessing the registry. Python native winreg
library already includes this functionality.
- Remove using pywin32 to retrieve peak memory usage on Win32 for `--debug=memory`
- Fix Issue 3759 - include scons.1, sconsign.1, scons-time.1 manpages in sdist and wheel packages.
- Change SCons's build so the generated `SCons/__init__.py` is no longer removed by `scons -c`
- Completely rewrote versioned shared libraries logic. Added support for SOVERSION via dmoody's initial PR 3733
- No longer automatically disable setting SONAME on shared libraries on OpenBSD.
- Fix race condition bug when initializing a scons cache directory at the
same time from multiple threads or processes. Problem described in PR 3114.
This is a simpler fix which should avoid some problems identified with the initial PR.
(Credit to Fredrik Medley for reporting the issue, the initial PR, and discussing and testing
this solution)
- Minor edits to User Guide and manpage's header with copyright, released date changed.

From Michał Górny:
- Fix dvipdf test failure due to passing incorrect flag to dvipdf.

From Adam Gross:
- Fix minor bug affecting SCons.Node.FS.File.get_csig()'s usage of the MD5 chunksize.
User-facing behavior does not change with this fix (GH Issue 3726).
- Fix occasional test failures caused by not being able to find a file or directory fixture
when running multiple tests with multiple jobs.
- Added support for a new command-line parameter `--hash-format` to override the default
hash format that SCons uses. It can also be set via `SetOption('hash_format')`. Supported
values are: `md5`, `sha1`, and `sha256`. For all hash formats other than
the default of `md5`, the SConsign database will include the name of the hash format.
For example, `--hash-format=sha256` will create a SConsign with name
`.sconsign_sha256.dblite.`.
- Fix incorrect cache hits and/or misses when running in interactive mode by having
SCons.Node.Node.clear() clear out all caching-related state.
- Change Environment.SideEffect() to not add duplicate side effects.
NOTE: The list of returned side effect Nodes will not include any duplicate side effect Nodes.
- Add support to the Python scanner for finding dynamically generated dependencies.
Previously the scanner only found imports if they existed on disk at scanning time.

From David H:
- Add ZIP_OVERRIDE_TIMESTAMP env option to Zip builder which allows for overriding of the file
modification times in the archive.
- Fix Zip builder not rebuilding when ZIPROOT env option was changed.

From Jason Kenny
- Fix python3 crash when Value node get_text_content when child content does not have decode()
NOTE: If you depend on Value node's get_text_content returning concatenated contents of it's
children. This may break your code. It now concatenates the csig() of all children.

From Joachim Kuebart:
- Suppress missing SConscript deprecation warning if `must_exist=False`
is used.

From Rocco Matano:
- Fix Zip tool to respect ZIPCOMSTR. Previously all zip builder calls would yield something
like zip(["test.zip"], ["zip_scons.py"]) and ignore ZIPCOMSTR if ZIPCOM and ZIPCOMSTR
weren't set after the Environment/Tool is initialized. (Explained in PR 3659)

From Daniel Moody:
- Fix issue where java parsed a class incorrectly from lambdas used after a new.

From Simon Tegelid
- Fix using TEMPFILE in multiple actions in an action list. Previously a builder, or command
with an action list like this:
['${TEMPFILE("xxx.py -otempfile $SOURCE")}', '${TEMPFILE("yyy.py -o$TARGET tempfile")}']
Could yield a single tempfile with the first TEMPFILE's contents, used by both steps
in the action list.

From Mats Wichmann:
- Complete tests for Dictionary, env.keys() and env.values() for
OverrideEnvironment. Enable env.setdefault() method, add tests.
- Raise an error if an option (not otherwise consumed) is used which
looks like an abbreviation of one one added by AddOption. (3653)
- Tool module not found will now raise a UserError to more clearly indicate this is
probably an SConscript problem, and to make the traceback more relevant.
- Fix three issues with MergeFlags:
- Signature/return did not match documentation or existing usage - the implementation
now no longer returns the passed env
- merging --param arguments did not work (issue 3107);
- passing a dict to merge where the values are strings failed (issue 2961).
- Include previously-excluded SideEffect section in User Guide.
- Clean up unneeded imports (autoflake tool).
- Make sure cProfile is used if profiling - SCons was expecting
the Util module to monkeypatch in cProfile as profile if available,
but this is no longer being done.
- Cleanup in SCons.Util.AddMethod. If called with an environment instance
as the object to modify, the method would not be correctly set up in
any Clone of that instance. Now tries to detect this and calls
MethodWrapper to set up the method the same way env.AddMethod does.
MethodWrapper moved to Util to avoid a circular import. Fixes 3028.
- Some Python 2 compatibility code dropped
- Rework runtest.py to use argparse for arg handling (was a mix
of hand-coded and optparse, with a stated intent to "gradually port").
- Add options to runtest to generate/not generate a log of failed tests,
and to rerun such tests. Useful when an error cascades through several
tests, can quickly try if a change improves all the fails. Dropped
runtest test for fallback from qmtest, not needed; added new tests.
- Eliminate tex tool usage of "for foo in range(len(iterable))"
- Restore internal Trace function to functional state.
- Only try to initialize the wix tool by default (or when tool `default` is explicitly installed)
on Windows based systems.
- Pick a better "Topic" Trove classifier for SCons: SW Dev / Build Tools
- Use os.replace instead of os.rename in dblite so don't need to
special-case Windows here. dblite is the default storage engine for the SConsign file(s).
- Fix cut-n-paste error in msvc debug printout and make some debug output
in msvs and msvsTests.py be off until needed (uncomment to use)
- Fix Issue 3014 - Empty file and missing file have same csig
- Refactor env.Append/Prepend to remove Py 1.5 era need to nest
try blocks, can now "continue" at the appropriate places.
- Add /snap/bin to env['PATH'] on POSIX, although this is only
really useful for a subset of POSIX systems that use snaps.
Was needed for CI builds, which run on Ubuntu LTS images.
- Eliminate Py2-ism __nonzero__ (now __bool__). Work around issue 3860
where a check for BuilderBase raising exc. on __bool__ was optimized out.
This issue was found due to a bug in Python 3.10.0a4. See issue 3860 for details.

From Rob Boehne:
- Fix fortran tools to set SHFORTRAN variables to $FORTRAN, similarly SHF77, SHF90, SHF95,
SHF03 and SHF08 will default to the variables $F77, $F90, $F95, $F03 and $F08 respectively.
If you were depending on changing the value of FORTRAN (or $F[0-9][0-9]) having no effect
on the value of SHFORTRAN, this change will break that. The values of FORTRAN, F77, F90,
F95, F03, F08 and SHFORTRAN, SHF77 (etc.) now are not overridden in generate if alredy set
by the user.
- Fix subprocess execution of 'lslpp' on AIX to produce text standard i/o.
- Re-do the fix for suncxx tool (Oracle Studio compiler) now that only Python 3 is supported,
to avoid decoding errors.

From William Deegan:
- Added Environment() variable TEMPFILEDIR which allows setting the directory which temp
files createdby TEMPFILEMUNGE are created in.

From Daniel Moody:
- Added method on Node to test if its node used in SConf. (Github Issue 3626)

From Dirk Baechle:
- Updated documentation toolchain to work properly under Python3, also
removed libxslt support from the Docbook Tool. (issue 3580)
- Added Docker images for building and testing SCons. (issue 3585)


From James Benton:
- Improve Visual Studio solution/project generation code to add support
for a per-variant cppflags. Intellisense can be affected by cppflags,
this is especially important when it comes to /std:c++* which specifies
what C++ standard version to target. SCons will append /Zc:__cplusplus
to the project's cppflags when a /std:c++* flag is found as this is
required for intellisense to use the C++ standard version from cppflags.

From Rob Boehne
- Specify UTF-8 encoding when opening Java source file as text. By default, encoding is the output
of locale.getpreferredencoding(False), and varies by platform.

From Joseph Brill:
- MSVC updates: When there are multiple product installations (e.g, Community and
Build Tools) of MSVC 2017 or MSVC 2019, an Enterprise, Professional,
or Community installation will be selected before a Build Tools installation when
"14.1" or "14.2" is requested, respectively. (GH Issue 3699).
- MSVC updates: When there are multiple product installations of MSVC 2017 (e.g.,
Community and Express), 2017 Express is no longer returned when "14.1" is
requested. Only 2017 Express will be returned when "14.1Exp" is requested.
(GH Issue 3699).
- MSVC updates: An MSVC 6.0 installation now appears in the installed versions list
when msvc debug output is enabled (GH Issue 3699).
- MSVS test updates: Tests for building a program using generated MSVS project and
solution files using MSVS 2015 and later now work as expected on x86 hosts.
- Test update: Reduce the number of "false negative" test failures for the interactive
configuration test (test/interactive/configure.py).
- MSVS update: Fix the development environment path for MSVS 7.0.

From William Deegan:
- Fix broken clang + MSVC 2019 combination by using MSVC configuration logic to
propagate'VCINSTALLDIR' and 'VCToolsInstallDir' which clang tools use to locate
header files and libraries from MSVC install. (Fixes GH Issue 3480)
- Added C:\msys64\mingw64\bin to default mingw and clang windows PATH's. This
is a reasonable default and also aligns with changes in Appveyor's VS2019 image.
- Drop support for Python 2.7. SCons will be Python 3.5+ going forward.
- Change SCons.Node.ValueWithMemo to consider any name passed when memoizing Value() nodes
- Fix Github Issue 3550 - When using Substfile() with a value like Z:\mongo\build\install\bin
the implementation using re.sub() would end up interpreting the string and finding regex escape
characters where it should have been simply replacing existing text. Switched to use string.replace().
- Fix Github Issue 2904 - Provide useful error message when more than one Configure Contexts are opened.
Only one open is allowed. You must call conf.Finish() to complete the currently open one before creating another
- Add msys2 installed mingw default path to PATH for mingw tool.
- C:\msys64\mingw64\bin
- Purge obsolete internal build and tooling scripts
- Allow user specified location for vswhere.exe specified by VSWHERE.
NOTE: This must be set at the time the 'msvc' 'msvs' and/or 'mslink' tool(s) are initialized to have any effect.
- Resolve Issue 3451 and Issue 3450 - Rewrite SCons setup.py and packaging. Move script logic to entry points so
package can create scripts which use the correct version of Python.
- Resolve Issue 3248 - Removing '-Wl,-Bsymbolic' from SHLIBVERSIONFLAGS
NOTE: If your build depends on the above you must now add to your SHLIBVERSIONFLAGS
- Speedup bin/docs-update-generated by caching parsed docbook schema. (60x speedup)
- Reorganized source tree. Moved src/engine/SCons to SCons to be more in line with current Python source
tree organization practices.
- Renamed as.py to asm.py and left redirecting tool. 'as' is a reserved word and so
changing the name was required as we wanted to import symbols for use in compilation_db
tool.
- Add CompilationDatabase() builder in compilation_db tool. Contributed by MongoDB.
Setting COMPILATIONDB_USE_ABSPATH to True|False controls whether the files are absolute or relative
paths. Address Issue 3693 and 3694 found during development.
- Fixed Github Issue 3628 - Hardcoding pickle protocol to 4 (supports python 3.4+)
and skipping Python 3.8's new pickle protocol 5 whose main advantage is for out-of-band data buffers.
NOTE: If you used Python 3.8 with SCons 3.0.0 or above, you may get a a pickle protocol error. Remove your
.sconsign.dblite. You will end up with a full rebuild.

From Andrii Doroshenko:
- Extended `Environment.Dump()` to select a format to serialize construction variables (pretty, json).

From Jeremy Elson:
- Updated design doc to use the correct syntax for Depends()

From Adam Gross:
- Added support for scanning multiple entries in an action string if
IMPLICIT_COMMAND_DEPENDENCIES is set to 2 or 'all'. This enables more thorough
action scanning where every item in each command line is scanned to determine
if it is a non-source and non-target path and added to the list of implicit dependencies
for the target.
- Added support for taking instances of the Value class as implicit
dependencies.
- Added new module SCons.Scanner.Python to allow scanning .py files.
- Added support for explicitly passing a name when creating Value() nodes. This may be useful
when the value can't be converted to a string or if having a name is otherwise desirable.
- Fixed usage of abspath and path for RootDir objects on Windows. Previously
env.fs.Dir("T:").abspath would return "T:\T:" and now it correctly returns "T:".

From Ivan Kravets, PlatformIO
- New conditional C Scanner (`SCons.Scanner.C.CConditionalScanner()`)
which interprets C/C Preprocessor conditional syntax (ifdef, if, else,
elif, define, etc.)
- Improvements for virtual C Pre-Processor:
* Handle UNSIGNED LONG and LONG numeric constants in DEC (keep support for HEX)
* Skip unrecognized directives, such as `if( defined ...)`
* Ignore `include DYNAMIC_INCLUDE` directive that depends on a dynamic
macro which is not located in a state TABLE.
* Cleanup CPP expressions before evaluating (strip comments, carriage returns)

From Iosif Kurazs:
- Added a new flag called "linedraw" for the command line argument "--tree"
that instructs scons to use single line drawing characters to draw the dependency tree.

From Daniel Moody:
- Add no_progress (-Q) option as a set-able option. However, setting it in the
SConstruct/SConscript will still cause "scons: Reading SConscript files ..." to be
printed, since the option is not set when the build scripts first get read.
- Added check for SONAME in environment to setup symlinks correctly (Github Issue 3246)
- User callable's called during substition expansion could possibly throw a TypeError
exception, however SCons was using TypeError to detect if the callable had a different
signature than expected, and would silently fail to report user's exceptions. Fixed to
use signature module to detect function signature instead of TypeError. (Github Issue 3654)
- Added storage of SConstructs and SConscripts nodes into global set for checking
if a given node is a SConstruct/SConscript.
Added new node function SCons.Node.is_sconscript(self) (Github Issue 3625)

From Andrew Morrow:
- Fix Issue 3469 - Fixed improper reuse of temporary and compiled files by Configure when changing
the order and/or number of tests. This is done by using the hash of the generated temporary files
content and (For the target files) the hash of the action.
So where previously files would be named:
- config_1.c, config_1.o, config_1
The will now be named (For example)
- conftest_68b375d16e812c43e6d72d6e93401e7c_0.c,
conftest_68b375d16e812c43e6d72d6e93401e7c_0_5713f09fc605f46b2ab2f7950455f187.o
or
conftest_68b375d16e812c43e6d72d6e93401e7c_0.o
conftest_68b375d16e812c43e6d72d6e93401e7c_0_5713f09fc605f46b2ab2f7950455f187 (for executable)

From Mathew Robinson:
- Improve performance of Subst by preventing unnecessary frame
allocations by no longer defining the *Subber classes inside of their
respective function calls.
- Improve performance of Subst in some cases by preventing
unnecessary calls to eval when a token is surrounded in braces
but is not a function call.
- Improve performance of subst by removing unnecessary recursion.
- Cleanup dangling symlinks before running builders (Issue 3516)

From Mats Wichmann:
- Remove deprecated SourceCode
- str.format syntax errors fixed
- a bunch of linter/checker syntax fixups
- Convert remaining uses of insecure/deprecated mktemp method.
- Clean up some duplications in manpage. Clarify portion of manpage on Dir and File nodes.
- Reduce needless list conversions.
- Fixed regex in Python scanner.
- Accommodate VS 2017 Express - it's got a more liberal license then VS
Community, so some people prefer it (from 2019, no more Express)
- vswhere call should also now work even if programs aren't on the C: drive.
- Add an alternate warning message if cl.exe is not found and msvc config
cache is in use (SCONS_CACHE_MSVC_CONFIG was given) - config cache
may be out of date.
- Fixed bug where changing TEXTFILESUFFIX would cause Substfile() to rebuild. (Github Issue 3540)
- Script/Main.py now uses importlib instead of imp module.
- Drop some Python 2-isms.
- MSVC updates: pass on VSCMD_DEBUG and VSCMD_SKIP_SENDTELEMETRY to msvc
tool setup if set in environment. Add Powershell to default env
(used to call telemetry script).
- Microsoft Visual Studio - switch to using uuid module to generate GUIDs rather than hand rolled
method using md5 directly.
NOTE: This change affects the following builders' output. If your build depends on the output of these builders
you will likely see a rebuild.
* Package() (with PACKAGETYPE='msi')
* MSVSSolution()
* MSVSProject()
- Docbook builder provides a fallback if lxml fails to generate
a document with tostring().
- Fix description of ARCOMSTR constr. var. (issue 3636). Previously the text was a copy of ASCOMSTR which
has different function.
- Update xml files in SCons to reflect changed relative paths after
code restructuring (src/engine/SCons -> SCons)
- Preliminary Python 3.9 support - elimination of some warnings.
- Drop the with_metaclass jig which was designed to let class
definitions using a metaclass be written the same for Py2/Py3.
- Bump python_version_unsupported (and deprecated) to indicate 3.5
is lowest supported Python.
- ParseFlags should not modify the user's passed in dict in case it's
a compound data structure (e.g. values are lists) (issue 3665)
- In Py3 classes no longer need to be listed as deriving from object.
- Remove deprecated check for Task subclasses needing a needs_execute
method - this is now enforced via an abstract base class, so the
check and test is no longer needed.
- Close various logfiles (trace, cache, taskmastertrace, configure)
when done using atexit calls.
- Rebase forked copy of shutil.copytree to Python 3.7 stlib version.
- Significant rework of documentation: API docs are now generated
using Sphinx; manpage and user guide now use more "standard"
markup elements (which could facilitate later conversion to a
different doc format, should that choice be made); significant
rewordings in manpage. Manpage Examples moved to an external
repository / website (scons-cookbook.readthedocs.io).
- Clean up test harness and tests' use of subdir, file_fixture and
dir_fixture.
- SubstitutionEnvironment and OverrideEnvironment now have keys()
and values() methods to better emulate a dict (already had items()).
- Rename internal Warning base class to SConsWarning to avoid any
possible confusion with Python's own Warning class.