Pysys

including support for running a test in multiple modes, and
``pysysdirconfig.xml`` files that allow you to specify defaults that apply to
all testcases under a particular directory - such as groups, modes, a prefix
to add to the start of each test id, and a numeric hint to help define the
execution order of your tests.

There is also new support for collecting files from each test output
directory (e.g. code coverage files), new features in the `pysys run` and
`pysys print` command lines, and a host of small additions to the API to make
test creation easier e.g. `BaseTest.assertEval`, `BaseTest.copy` (with filtering of each copied
line) and `BaseTest.write_text` (for easy programmatic creation of files in the output
directory).

This is a major release and therefore there are a few significant changes
that could required changes in existing projects; please review the
compatibility section of this document and perform an initial test run using
the new PySys version to check for issues before switching over.

Miscellaneous new features:

- Added support for running tests in multiple modes from within a single PySys
execution. To make use of this, add the following property to your
`pysysproject.xml`::

<property name="supportMultipleModesPerRun" value="true"/>

The old concept of modes within PySys is now deprecated in favor of the
more powerful features of `supportMultipleModesPerRun=True` so we recommend
all users to add this project setting when possible. Please note though that
it will result in slightly different behaviour (e.g. different output
directory names) if you have any tests with `<mode>...</mode>` in their
descriptor. See the user guide for detailed information about running tests
in multiple modes.

- Added a project configuration option that collects a copy of all test output
files matching a specified pattern into a single directory. This is useful
for collecting together code coverage files from all tests into one place,
and could also be used for collating other outputs such as performance or
memory usage graphs. Files are copied from the output directory at the
end of each test's execution, and before any files are purged. The sample
project file shows how to use this feature to collect Python code
coverage files::

<property name="pythonCoverageDir" value="__pysys_coverage_python_OUTDIR"/>
<collect-test-output pattern=".coverage*" outputDir="${pythonCoverageDir}" outputPattern="FILENAME_TESTID_UNIQUE"/>

The output directory is wiped clean at the start of each test run to prevent
unwanted interference between test runs, and is created on demand when the
first matching output file is found, so the directory will not be created if
there is no matching output.

- Added support for generating code coverage reports for programs written in
Python, using the coverage.py library. To enable this, ensure the coverage
library is installed (``pip install coverage``), add collecting of test output
files named ``.coverage*`` to a directory stored in the ``pythonCoverageDir``
project property (see above example), and run the tests with
``-X pythonCoverage=true``. You can optionally set a project property
``pythonCoverageArgs`` to pass arguments to the coverage tool, such as which
modules/files to include or omit. After all tests have been executed, the
runner calls a new method `processCoverageData` which combines all the
collected coverage files into a single file and produces an HTML report
from it, within the pythonCoverageDir directory. If you wish to produce
coverage reports using other tools or languages (such as Java), this
should be easy to achieve by following the same pattern - using
`<collect-test-output>` to gather the coverage files and providing a
custom implementation of `pysys.baserunner.BaseRunner.processCoverageData`.

- Added `BaseTest.assertEval` method which supersedes `BaseTest.assertThat` and provides
a convenient way to assert an arbitrary Python expression, with generation of
a clear outcome reason that is easy to understand and debug.

- Added `BaseTest.copy` method for copying a binary or text file, with
optional transformation of the contents by a series of mapping functions.
This can be used to extract information of interest from a log file before
diff-ing with a reference copy, for example by stripping out timestamps
and irrelevant information.

- Added `BaseTest.write_text` method for writing characters to a text file
in the output directory using a single line of Python.

- Added `expectedExitStatus` parameter to `BaseTest.startProcess()` method
which can be used to assert that a command returns a non-zero exit code,
for example ``self.startProcess(..., expectedExitStatus='==5')``.
This is simpler and more intuitive than setting `ignoreExitStatus=True` and
then checking the exit status separately.

- Added ``quiet`` parameter to `BaseTest.startProcess()` method
which disable INFO/WARN level logging (unless a failure outcome is appended),
which is useful when calling a process repeatedly to poll for completion of
some operation.

- Added `BaseTest.startPython` method with similar options to `BaseTest.startProcess`
that should be used for starting Python processes. Supports functionality
such as Python code coverage.

- Added `BaseTest.disableCoverage` attribute which can be used to globally
disable all code coverage (in all languages) for a specific test. For example
if you apply a group called 'performance' to all performance tests, you could
disable coverage for those tests by adding this line to your BaseTest::

if 'performance' in self.descriptor.groups: self.disableCoverage = True

- Added ``hostname``, ``startTime`` and ``startDate`` project properties which can be
used in any ``pysysproject.xml`` configuration file. The start time/date
gives the UTC time when the test run began, using the yyyy-mm-dd HH.MM.SS
format which is suitable for inclusion in file/directory names.

- Added `BaseTest.getBoolProperty()` helper method which provides a simple way to
get a True/False value indicating whether a setting is enabled, either
directly using a ``-X prop=value`` argument, or with a property set in the
``pysysproject.xml`` configuration file.

- Added environment variable ``PYSYS_PORTS_FILE`` which if present will be read
as a UTF-8/ASCII file with one port number on each line, and used to populate
the pool of ports for `BaseTest.getNextAvailableTCPPort()`. This can be used to
avoid port conflicts when invoking PySys from an environment where some ports
are taken up by other processes.

- Added ``TIMEOUTS['WaitForAvailableTCPPort']`` which controls how long
`BaseTest.getNextAvailableTCPPort()` will wait before throwing an exception.
Previously ``getNextAvailableTCPPort()`` would have thrown an exception if
other tests were using up all ports from the available pool; the new
behaviour is to block and retry until this timeout is reached.

Improvements to the XML descriptors that provide information about tests:

- Added support for disabling search for testcases in part of a directory tree
by adding a ``.pysysignore`` or ``pysysignore`` file. This is just an empty file
that prevents searching inside the directory tree that contains it for tests.
This could be useful for reducing time taken to locate testcase and also for
avoiding errors if a subdirectory of your PySys project directory contains
any non-PySys files with filenames that PySys would normally interpret
as a testcase such as ``descriptor.xml``.

- Added a new XML file called ``pysysdirconfig.xml`` which is similar to
``pysystest.xml`` and allows setting configuration options that affect all
tests under the directory containing the ``pysysdirconfig.xml`` file.

This allows setting things like groups, test id prefix, execution order,
and skipping of tests for a set of related testcases without needing to
add the options to each and every individual ``pysystest.xml`` file. For
example, if you have a couple of directories containing performance tests
you could add ``pysysdirconfig.xml`` files to each with a
``<group>performance</group>`` element so it's easy to include/exclude all
your performance when you invoke ``pysys.py run``. You could also include
a ``<execution-order hint="+100"/>`` to specify that performance
tests should be run after your other tests(the default order hint is 0.0).

The ``pysysdirconfig.xml`` file can contain any option that's valid in
a ``pysystest.xml`` file except the ``description/title/purpose``. a sample
``pysysdirconfig.xml`` file is provided in ``pysys/config/templates/dirconfig``.

See the PySys User Guide for more information.

- Added support for specifying a prefix that will be added to start of the
testcase directory name to form the testcase identifier. This can be
specified in ``pysystest.xml`` testcase descriptor files and/or in
directory-level ``pysysdirconfig.xml`` files like this:

<id-prefix>MyComponent.Performance.</id-prefix>

Large test projects may benefit from setting prefixes in ``pysysdirconfig.xml``
files to provide automatic namespacing of testcases, ensuring there are no name
clashes across different test directories, and providing a way to group
together related test ids without the need to use very long names for
each individual testcase directory. Prefixes can be specified cumulatively,
so with the final testcase id generated from adding the prefix from each
parent directory, finishing with the name of the testcase directory itself.

We recommend using an underscore or dot character for separating test
prefixes.

- Added support for specifying the order in which testcases are run. To do
this, specify a floating point value in any ``pysystest.xml`` testcase
descriptor, or ``pysysdirconfig.xml`` descriptor (which provides a default for
all testcases under that directory)::

<execution-order hint="+100.0"/>

Tests with a higher ordering hint are executed after tests with lower
values. The default order value is 0.0, and values can be positive or
negative. Tests with the same order hint are executed based on the
sort order of the testcase directories. It is also possible to configure
hints at a project level for specific modes or groups. See the user guide
for more information.

You might want to specify a large order hint for long-running performance or
robustness tests to ensure they execute after more important unit/correctness
tests. You might want to specify a negative hint for individual tests that
are known to take a long time (if you're running with multiple threads), to
ensure they get an early start and don't hold up the completion of the test
run.

- Added a new way to skip tests, by adding this element to the `pysystest.xml`
descriptor::

<skipped reason="Skipped due to open bug ABC-123"/>

Although tests can still be skipped by setting the ``state="skipped"``
attribute, the use of the ``skipped`` element is recommended as it provides a
way to specify the reason the test has been skipped, and also allows a
whole directory of tests to be skipped by adding the element to a
``pysysdirconfig.xml`` file. The default ``pysystest.xml`` template generated
for new testcases now contains a commented-out ``skipped`` element instead of
a `state=` attribute.

- Added a new API for overriding the way test descriptors are loaded from a
directory on the file system. This allows for programmatic customization
of descriptor settings such as the supported modes for each testcase, and
also provides a way to make PySys capable of finding and running non-PySys
tests (by programmatically creating PySys TestDescriptor objects for them).
See the `pysys.config.descriptor.DescriptorLoader` class for more details.

Improvements to the ``pysys.py`` command line tool:

- Added support for running tests by specifying just a (non-numeric) suffix
without needing to include the entire id. Although support for specifying a
pure numeric suffix (e.g. ``pysys.py run 10``) has been around for a long time,
you can now do the same with strings such as ``pysys.py run foo_10``.

- Added ``--sort`` option to ``pysys.py print``. This allows sorting by ``title``
which is very helpful for displaying related testcases together (especially
if the titles are written carefully with common information at the beginning
of each one) and therefore for more easily locating testcases of interest.
It can also sort by ``id`` or ``executionOrderHint`` which indicates the order
in which the testcases will be executed. The default sort order if none of
these options is specified continues to be based on the full path of the
``pysystest.xml`` files.

- Added ``--grep``/``-G`` filtering option to ``pysys.py print`` and ``pysys.py run``
which selects testcases that have the specific regular expression (matched
case insensitively) in their ``id`` or ``title``. This can be a convenient way
to quickly run a set of tests related to a particular feature area.

- Added a concise summary of the test ids for any non-passes in a format that's
easy to copy and paste into a new command, such as for re-running the failed
tests. This can be disabled using the `pysys.writer.console.ConsoleSummaryResultsWriter` property
``showTestIdList`` if desired.

- Added an environment variable ``PYSYS_DEFAULT_THREADS`` which can be used to set
the number of threads to use with ``--threads auto`` is specified on a
per-machine or per-user basis.

- Added the ability to set logging verbosity for specific ``pysys.*`` categories
individually using ``-vCAT=LEVEL``. For example to enable just DEBUG logging
related to process starting, use ``-vprocess=DEBUG``. Detailed DEBUG logging
related to assertions including the processed version of the input files uses
the category "assertions" and is no longer included by default when the
root log level is specified using ``-vDEBUG`` since it tends to be excessively
verbose and slow to generate; if required, it can be enabled using
``-vassertions=DEBUG``.

- Argument parsing now permits mixing of ``-OPTION`` and non-option (e.g. test
id) arguments, rather than requiring that the test ids be specified
only at the end of the command line. For example::

pysys run --threads auto MyTest_001 -vDEBUG

- Added automatic conversion of strings specified on the command line with
``-Xkey=value`` to int, float or bool if there's a static variable of the
same name and one of those types defined on the test `BaseTest` class. This makes it
easier to write tests that have their parameters overridden from the command
line. For example, if a test class has a static variable ``iterations=1000``
to control how many iterations it performs, it can be run with
``pysys run -Xiterations=10`` during test development to override the number
of iterations to a much lower number without any changes to ``run.py``.
Note that this doesn't apply to BaseRunner, only BaseTest.

- Added ``--json`` output mode to ``pysys.py print`` which dumps full information
about the available tests in JSON format suitable for reading in from other
programs.

- Changed ``makeproject`` so that when a template is to be specified, it is now
necessary to use an explicit ``--template`` argument, e.g ``--template=NAME``.

Bug fixes:

- PySys now uses ``Test outcome reason:`` rather than ``Test failure reason:``
to display the outcome, since there is sometimes a reason for non-failure
outcomes such as SKIPPED.

- Fixed ``--purge`` to delete files in nested subdirectories of the output
directory not just direct children of the output directory.

- Previous versions of PySys did not complain if you created multiple tests
with the same id (in different parent directories under the same project).
This was dangerous as the results would overwrite each other, so in this
version PySys checks for this condition and will terminate with an error
if it is detected. If you intentionally have multiple tests with the same
name in different directories, add an ``<id-prefix>`` element to the
``pysystest.xml`` or (better) to a ``pysysdirconfig.xml`` file to provide
separate namespaces for the tests in each directory and avoid colliding ids.

- The Ant JUnit writer now includes the test duration.

- Improved `BaseTest.assertGrep` outcome reason to include the entire matching string
when a ``contains=False`` test fails since ``ERROR - The bad thing happened`` is
a much more useful outcome reason than just ``ERROR``.

- Fixed CSV performance reporter runDetails which was including each item
twice.

- On Windows, paths within the testcase are now normalized so that the drive
letter is always capitalized (e.g. ``C:`` not ``c:``). Previously the
capitalization of the drive letter would vary depending on how exactly PySys
was launched, which could occasionally lead to inconsistent behaviour if
testing an application that relies on the ASCII sort order of paths.

Upgrade guide and compatibility:

Occasionally it is necessary for a new PySys release to include changes that
might change or break the behaviour of existing test suites. As 1.5.0 is a
major release it is possible that some users might need to make changes:

- Errors and typos in ``pysystest.xml`` XML descriptors will now prevent any tests
from running, whereas previously they would just be logged. Since an invalid
descriptor prevents the associated testcase from reporting a result, the
new behaviour ensures such mistakes will be spotted and fixed promptly.
If you have any non-PySys files under your PySys project root directory
with names such as ``descriptor.xml`` which PySys would normally recognise
as testcases, you can avoid errors by adding a ``.pysysignore`` file to prevent
PySys looking in that part of the directory tree.

- `BaseTest.mkdir` now returns the absolute path (including the output
directory) instead of just the relative path passed in. This make it easier
to use in-line while performing operations such as creating a file in the
new directory. Code that relied on the old behaviour of returning the
path passed in may need to be updated to avoid having the output directory
specified twice. If you're using ``os.path.join`` then no change will be
required.

- The ``self.output`` variable in `pysys.baserunner.BaseRunner` is no longer set to the current
directory, but instead to a ``pysys-runner-OUTDIR`` subdirectory of the
test root (or to ``OUTDIR/pysys-runner`` if ``OUTDIR`` is an absolute path).
This ensures that any files created by the runner go into a known location
that is isolated from other runs using a different `OUTDIR`. The runner's
``self.output`` directory is often not actually used for anything since
most logic that writes output files lives in `BaseTest` subclasses, so
most users won't be affected. For the same reason, the runner output
directory is not created (or cleaned) automatically.
If you have a custom ``BaseRunner`` that writes files to its output directory
then you should add a call to ``self.deleteDir <BaseTest.deleteDir>`` and then
`self.mkdir <BaseTest.mkdir>` to
clean previous output and then create the new output directory.

- The behaviour of `BaseTest.getDefaultEnvirons` has changed compared to
PySys 1.4.0, but only when the command being launched is ``sys.executable``,
i.e. another instance of the current Python process (``getDefaultEnvirons`` is
used by `BaseTest.startProcess` when ``environs=`` is not explicitly provided).

In 1.4.0 the returned environment always set the ``PYTHONHOME`` environment
variable, and on Windows would add a copy of the` `PATH`` environment from the
parent process. In PySys 1.5.0 this is no longer the case, as the 1.4.0
behaviour was found to cause subtle problems when running from a virtualenv
installation or when the child Python itself launches another Python process
of a different version. The new behaviour is that `BaseTest.getDefaultEnvirons` adds
the directory containing the Python executable to ``PATH`` (on all OSes), and
copies the ``LD_LIBRARY_PATH`` from the parent process only on Unix (where it
is necessary to reliably load the required libraries). `getDefaultEnvirons`
no longer sets the ``PYTHONHOME`` environment variable.

- The format of ``pysys print`` has changed to use a ``|`` character instead of a
colon to separate the test id and titles. This makes it easier to copy and
paste test ids from ``pysys print`` into the command line.

- Several fields in the `pysys.config.descriptor.TestDescriptor` (aka ``XMLDescriptorContainer``) class
that used to contain absolute paths now contain paths relative to
the newly introduced `testDir` member. These are: `module`, `output`,
`input`, `reference`. The values of `BaseTest.output/input/reference`
have not changed (these are still absolute paths), so this change is unlikely
to affect many users.

- The ``PROJECT`` variable in the `constants` module is deprecated. Use
`self.project` instead (which is defined on classes such as `BaseTest`,
`pysys.baserunner.BaseRunner` etc).

Installation:

- The available options for installing PySys have been reworked and modernised.
The recommended way to install PySys is by running `pip install PySys`.

- A binary `.whl` wheel is now available for the first time, which is more
efficient, reliable and lightweight than other installation methods, and
is used by the pip installer. The `tar.gz` source distribution is still
available but is no longer a recommended installation mechanism. The Windows
GUI installer is no longer published as this is superseded by the simpler
installation experience provided by `pip`.

- HTML documentation of the PySys API is no longer installed locally by default,
but is available on https://pysys-test.github.io/pysys-test website or as a
separate zip file available from
https://github.com/pysys-test/pysys-test/releases.

Improvements to the `pysys.py` tool:

- `pysys.py` has a new `makeproject` command that generates a default
`pysysproject.xml` with some recommended defaults to make it easy to start a
new project without needing to download the samples.

- As an alternative to the usual `pysys.py` executable script, it is now also
possible to launch PySys using::

python -m pysys

- Added new command line option `--printLogs all|failures|none` (default value
is `all`) which allows user to avoid the printing of run.log to the stdout
console either for all tests, or for tests that pass. This is useful to
avoid generating huge amounts of output during large test runs (which can
be problematic when stdout is captured by a Continuous Integration system),
or to show detailed information only for failing tests which makes it easier
for a user to locate the diagnostic information they care about more quickly.
The specified value is stored in `runner.printLogs` and can be changed by
custom writer implementations if desired, for example to avoid duplicating
information already being printed to stdout by the writer in a different
format.

- PySys will now automatically enable colored output if there is no color
setting in the `pysysproject.xml` or `PYSYS_COLOR` environment - provided
PySys is running in an interactive terminal. On Windows the `colorama`
library is now a dependency to ensure colored output is always possible.

- Added `--threads auto` which is equivalent to `--threads 0` and provides
a clearer way to indicate that PySys will automatically determine how many
threads to run tests with based on the number of available CPUs.

- The outcome reason string now has a suffix specifying how many additional
failure outcomes were logged (so if you have a complex test you can see at a
glance if there's just one problem to resolve, or 5, or 20!).


New project options:

- Added support for running PySys tests under Travis CI(R) to the sample
`pysysproject.xml` file. Travis support includes by default only printing
`run.log` output for failed tests, and containing that detailed output within
a folded section that can be expanded if needed. To enable this just ensure
that the Travis CI writer is enabled in your project configuration file,
which you can copy from the sample project configuration file if you already
have an existing project file.

- Added support for configuring the default encodings to use for common file
patterns in the `pysysproject.xml` configuration, e.g. ::

<default-file-encoding pattern="*.yaml" encoding="utf-8"/>.

The sample project configuration file now
sets utf-8 as the default encoding for XML, json and yaml files, and also
for testcase run.log files (though run.log continues to be written in local
encoding unless the project file is updated). For more information on this
feature, see comments in `pysysproject.xml` and in
`ProcessUser.getDefaultFileEncoding()`.

- Use of ``print()`` rather than ``self.log`` is a common mistake that results in
essential diagnostic information showing up on the console but not
stored in ``run.log``. A new project option `redirectPrintToLogger`
can optionally be enabled to instruct PySys to catch output written using
``print()`` statements or to ``sys.stdout`` and redirect it to the logging
framework, so it will show up in ``run.log``. Writers that genuinely need
the ability to write directly to stdout should be changed to use
`pysys.utils.logutils.stdoutPrint`.

- There are new settings for customizing the default environment used by
`BaseTest.startProcess`::

<property name="defaultEnvironsDefaultLang" value="en_US.UTF-8"/>
<property name="defaultEnvironsTempDir" value="self.output'"/>

See `BaseTest.getDefaultEnvirons()` for more information on these.

Main API improvements:

- Added `BaseTest.skipTest()` method, which can be used to avoid running the
rest of the `BaseTest.execute()` or `BaseTest.validate()` method, if it is not appropriate for
the test to execute on this platform/mode.

- Added boolean `pysys.constants.IS_WINDOWS` constant, since conditionalizing logic for Windows
versus all other Operating Systems is very common; this avoids the need for
error-prone matching against string literals.

- Added `BaseTest.startProcess()` argument `stdouterr` which allows
specifying the base prefix to use for writing process standard output and
error using a single parameter, either as a string or from a tuple such
as that returned from `allocateUniqueStdOutErr()`. As as result there is no
longer a need to save the generated stdout and stderr to local variables
before passing to startProcess; you can simply specify::

self.startProcess(..., stdouterr=self.allocateUniqueStdOutErr('myprocess'))

Alternatively if you don't care about allocating unique names (perhaps
because you have only one instance of the process) a simple string prefix
can be specified instead. The final `stdout` and `stderr` paths are available
on the returned `ProcessWrapper` object.

If no displayName is provided, `startProcess` will generate one based on
the `stdouterr` prefix so it's easy to identify which process is being
started.

- Added `BaseTest.getDefaultEnvirons()` method which is now used by
`BaseTest.startProcess()` to provide a minimal but clean set of environment variables
for launching a given process, and can also be used as a basis for creating
customized environments using the new `BaseTest.createEnvirons()` helper method.
There are some new project properties to control how this works, which
you may wish to consider using for new projects, but are not enabled by
default in existing projects to maintain compatibility::

<property name="defaultEnvironsDefaultLang" value="en_US.UTF-8"/>
<property name="defaultEnvironsTempDir" value="self.output'"/>

See `BaseTest.getDefaultEnvirons()` for more information on these.
If needed you can further customize the environment by overriding
`getDefaultEnvirons`.

- Extended the writers API:
- Added `runLogOutput=` parameter to the `processResult()` method of
the `BaseResultsWriter` class so that writers such as the
`JUnitXMLResultsWriter` can include the test output with no loss of unicode
character information.
- Added `testoutdir=` parameter to the `setup()` method so writers have
a way to identify different test runs on the same machine.
- Added `runner=` parameter to the `setup()` method so writers have
access to the runner instance for reading/modifying configuration
settings.
- Added `isEnabled()` method that can optionally be used by a writer to
disable itself based on the environment in which it is running, or
to enable itself even when `--record` isn't specified, which is useful
for writers that produce output for a CI system.

- Rewrote the process monitoring API to make it easier to add extra monitoring
statistics (by subclassing the OS-specific `DEFAULT_PROCESS_MONITOR` or the
superclass `BaseProcessMonitor`, or to add a custom handler for the
generated statistics, by subclassing `BaseProcessMonitorHandler`.

- Added `BaseTest.startBackgroundThread` method which takes care of ensuring
threads are stopped and joined during cleanup, that exceptions from threads
result in BLOCKED outcomes and that logging output from background threads
goes to the same handlers as foreground logging. The thread target can
be either a simple function or an instance method (e.g. on the testcase).
A Python `threading.Event` object called `stopped` is passed to the
background thread to make it easy to determine when it should finish
executing. The `ProcessUser.addOutcome()` method is now thread-safe
(though most of the `ProcessUser` and `BaseTest` should still not be accessed
from multiple threads without locking).

- Added `BaseTest.pythonDocTest()` method for executing the doctests in a
Python file.

Minor API additions:

- Added `PerformanceUnit.NANO_SECONDS` (with alias `ns`) which is now
recommended when measuring the peformance of operations that take less than a
second.

- Added `__str__` implementations for BaseTest and BaseRunner, which uniquely
identify the test (and cycle, in multi-cycle runs). This may be useful for
diagnostic and logging purposes.

- Performance reporter classes can now make use of `self.runner` to access
information such as the mode in which the test is running for reporting
purposes.

- Added `BaseTest.assertPathExists` for checking that a file exists (or not).

- The default implementation of `BaseTest.getDefaultFileEncoding()` now
delegates to the runner's implementation, allowing customizations to be
performed in just one place if neede for both `BaseTest` and runner class.

- Added `BaseTest.compareVersions()` static helper method for
comparing two alphanumeric dotted version strings.

- Added `BaseTest.deletedir` which is more convenient that the associated
`fileutils.deletedir` for paths under the `self.output` directory.

- Added `BaseTest.addOutcome(override=...)` argument which can be used to
specify a new test outcome that replaces any existing outcomes even if
they have a higher precedence.

- Added `ignores=` argument to `BaseTest.waitForSignal()` method which
excludes lines matching the specified expression from matching both the
main `expr` match expression and any `errorExpr` expressions.

- Added `pysys.utils.fileutils.toLongPathSafe/fromLongPathSafe` which on Windows performs
the necessary magic to allow Python to access paths longer than 256
characters (and on other platforms are a no-op), and `pathexists` which
is a long path-safe version of `os.path.exists`. PySys will now handle long
paths in the most critical places, such as `deletedir`, `logFileContents`,
`openfile`, `assertPathExists`, when enumerating available tests, and during
test cleanup. Test authors can make use of `toLongPathSafe` as needed in
their own test cases.

- Added `pysys.utils.logutils.stdoutPrint` for writers that genuinely need
the ability to write directly to stdout without using a logger.


Upgrade guide and compatibility:

It is pretty rare for a new PySys release to include changes that might change
or break the behaviour of existing test suites, but occasionally it is
necessary in order to fix bugs or allow us to provide new functionality. In
this release there are a few such changes:

- In the previous release unknown or invalid keyword arguments passed to
assert* methods would be silently ignored (potentially masking mistakes);
now it is an error to specify an invalid argument.

- The environment `BaseTest.startProcess` uses by default if no `environs=`
parameter was specified has changed. Although the documentation states that
a clean environment is used if no `environs` dictionary is specified, in
PySys v1.1, 1.2 and 1.3 the Windows behaviour changed to include a copy of
all environment variables in the parent PySys process (typically a very
large set of variables), which could cause tests to unintentionally
be affected by the environment it was run from. This is now fixed, so that
a small minimal set of environment variables are always returned, as returned
by the new `ProcessUser.getDefaultEnvirons()` method. As a result on Windows
a much smaller set of environment variables and PATH/LD_LIBRARY_PATH
components will be used, and on Unix instead of a completely empty
environment, a few variables will now be set. If this causes problems you can
temporarily go back to the legacy behaviour by setting this
`pysysproject.xml` option::

<property name="defaultEnvironsLegacyMode" value="true"/>

See https://github.com/pysys-test/pysys-test/issues/9 for more information.

- The default process monitor file format has changed in this release to
provide consistency across all operating systems, and because the
Windows-specific statistics private/thread/handle count were not correct and
cannot easily be obtained in a robust way. If you need these, or wish to
use a wider set of monitoring statistics than PySys provides in the box, it
is easy to create a custom `BaseProcessMonitor` subclass, perhaps using a
cross-platform Python library such as `psutil` to gather the data.

Previously there was no header line, and on Windows the columns were::

dd/mm/yy HH:MM:SS, CPU, Resident, Virtual, Private, Threads, Handles

and on Linux::

mm/dd/yy HH:MM:SS, CPU, Resident, Virtual

In this release there is a header line comment at the start of the file
beginning with `` indicating the column headings. Also a standard date
format is used, and only the columns supported on all operating systems are
included::

yyyy-mm-dd HH:MM:SS, CPU, Resident, Virtual

This behaviour can be customized for all your testcases from your runner's
`setup` method. For example to go back to the previous file format (although
without the Windows-specific columns, which are no longer supported), add::

ProcessMonitorTextFileHandler.setDefaults(
[
ProcessMonitorKey.DATE_TIME_LEGACY,
ProcessMonitorKey.CPU_CORE_UTILIZATION,
ProcessMonitorKey.MEMORY_RESIDENT_KB,
ProcessMonitorKey.MEMORY_VIRTUAL_KB,
], writeHeaderLine=False)

Also note that the numProcessors keyword argument to `startProcessMonitor` is
deprecated. For now it can still be used to scale down the
`CPU_CORE_UTILIZATION` value but it is not recommended for use and may be
removed in a future release. Use `CPU_TOTAL_UTILIZATION` if you wish to see
total CPU usage across all cores.

In the previous release, the Linux process monitor also gathered data
from child processes (that were running at the moment the monitor was
started). As this functionality was Linux-specific, not documented, and
generated incorrect results this has been removed. Optional support for
monitoring child processes may be re-added in a future PySys release.
Although child process are not included in the statistics for each process,
the contributions from its child threads are included.

- If you have created a custom subclass of `ProcessMonitor` you will need to
rework it, as this class no longer exists and the API has been rewritten in
order to make it easier to maintain and extend.
For example it is now easier to add extra monitoring statistics (by
subclassing `BaseProcessMonitor`), or provide custom handlers for the data
for different file formats or automated checking of results (by subclassing
`BaseProcessMonitorHandler`; no longer requires subclassing the process
monitor itself). If you have written a custom subclass of ProcessMonitor
to customize what data is gathered you will need to rework it when moving to
this version of PySys. If you need to provide custom code to handle the
generated statistics, you can now do that by passing a
`BaseProcessMonitorHandler` subclass to `BaseTest.startProcessMonitor`.

- Fixed bug in which symbols (classes, constants, imports) defined in one
`run.py` could be seen by other run.py files, potentially causing test
behaviour to vary based on what other tests had previously run, and/or
race conditions seen only during parallel execution. Now every `run.py` file
has its own independent namespace. It is possible some previously passing
tests might fail as a result of this change, if they were relying on
the buggy behaviour to implicitly import symbols.

- Although most real PySys projects had a `pysysproject.xml` file in the root
directory specifying the configuration, PySys v1.3.0 and earlier treated
this file as optional, resulting in confusing error messages, and
long and sometimes disruptive searching of non-test directories if a user
tried to run PySys from a non-test directory (e.g. from `c:`). To avoid
user confusion, by default PySys will now terminate with an error if you
try to run it from a directory which doesn't have a project file. Any users
who found the ability to use it without a project file useful can enable
it by setting the `PYSYS_PERMIT_NO_PROJECTFILE=true` environment variable.

- Removed `pysys.utils.smtpserver` which was never used by any part of PySys,
does not really belong in this project, and adds little over Python's
built-in `smtpd` module.

- Removed `DEFAULT_STYLESHEET` `pysys-log.xsl` as referenced in
`XMLResultsWriter`, as it does not work in most modern browsers
(e.g. Chrome, Firefox) for security reasons and is not widely used. If you
need this functionality, the ability to specify a custom .xsl stylesheet for
the `XMLResultsWriter` is still available as a configuration option in
`pysysproject.xml`.

- Any custom performance reporter classes created using PySys 1.3.0
and which provided a custom constructor should be updated to include the
`**kwargs` parameter added in this version of PySys, as the old constructor
signature is now deprecated. As this API was added in 1.3.0 no other versions
are affected.


Bug fixes:

- Fixed bug in which random log lines might not be written to `run.log` and/or
stdout when running tests multi-threaded (as a result of an underlying
python bug https://bugs.python.org/issue35185).

- Fixed bug in which symbols (classes, constants, imports) defined in one
`run.py` could be seen by other run.py files, potentially causing test
behaviour to vary based on what other tests had previously run, and/or
race conditions seen only during parallel execution. Now every `run.py` file
has its own independent namespace. It is possible some previously passing
tests might fail as a result of this change, if they were relying on
the buggy behaviour to implicitly import symbols.

- Fixed `startProcess()` to use a clean and minimal set of environment
variables on Windows if no `environs=` parameter was specified, rather than
copying all environment variables from the parent PySys process to the child
process. See https://github.com/pysys-test/pysys-test/issues/9 for more
information.

- Fixed `startProcess()` to add a `BLOCKED` test outcome when a process fails
to start due to a `ProcessError`, unless `ignoreExitStatus=True`. Previously
this flag only affected non-zero exit codes, resulting in `ProcessError`
failures getting ignored.

- Fixed `startProcess()` to correctly handle passing empty arguments,
and arguments containing spaces, quotes and glob characters on Windows.
Previously, empty arguments were skipped, and arguments containing spaces
were only handled correctly if first character was not a space.

- Fixed a number of errors in the statistics reported by process monitors,
especially on Windows where negative values were sometimes returned
(due to integer overflow), incorrect (and very time-consuming) aggregation
based on the child threads that existed at the time the process monitor was
first started, lack of support for non-English Windows installations
(which have localized counter names) and that the statistics might be
returned for the wrong process due to the way the performance counter API
changes which process is being monitored when processes of the same name
terminate.
On Linux the statistics were sometimes wrong due to undocumented and
in some cases incorrect aggregation across child processes, which has now
been removed. The values are now correct on all operating systems.

- Fix bug in which non-ASCII characters in test outcome reasons could
prevent the test log being written to disk if executed in multi-threaded
mode. Only affects Python 2.

- Significant improvements to robustness when testing support for international
(I18N) characters. This includes implementing fully safe logging of unicode
strings (with `?` replacements for any unsupported characters) that works
regardless of what encoding is in use for stdout and `run.log`. Also fixed
exception when logging unicode characters in Python 2 if a formatter was not
configured in `pysysproject.xml`, by ensuring it is always stored as a
unicode character string not a byte string (which used to happen in Python 2
if it was not mentioned in the project config). Fixed `logFileContents` to
more robustly handle files containing I18N/non-ASCII characters.

- `JUnitXMLResultsWriter` and `XMLResultsWriter` now write using UTF-8
encoding rather than local/default encoding, and also include the
`encoding="utf-8"` header in the XML header. Since previously there was no
`encoding` header many tools would have interpreted them as UTF-8 already,
and now the behaviour is consistent with that expectation.

- Added `pysys.writers.replaceIllegalXMLCharacters()` utility function, and use
it to avoid `XMLResultsWriter` and `JUnitXMLResultsWriter` from generating
invalid XML if `run.log` or outcome reason contain characters not permitted
by XML. Also ASCII control characters (e.g. coloring instructions
from other tools) are now stripped out of all outcome reason strings
(including in run.log and non-XML based writers) since such characters
are not useful and make summary test results harder to read.

- Fixed rare condition in which performance result reporting would be prevented
due to spurious error about `resultKey` already being used.

--------------

--------------
Changes affecting compatibility:

- Fixed `BaseTest.assertDiff` (and filediff) handling of "include" expressions
list to filter out lines if no include expressions match (as documented)
rather than if any include expressions match. This fix may cause tests to fail
that had previously - and incorrectly - passed as a result of all lines
being filtered out before the comparison. There is also now a message
logged at warn level when every line in a file comparison is filtered
out, since in most cases this is not desirable behaviour.
- Changed `pysys.py run` to return a non-zero exit code if any tests
failed, whereas previously it would return 0.

Other fixes and new features:

- PySys now provides 'single-source' support for both Python 2.7 and
Python 3.x, without the need for the 2to3.py script to be run at
installation time for use with Python 3.
- Added support for specifying what file encoding is to be used for reading
and writing text files (for example in `waitForSignal` and various
assertions). This is especially important for Python 3 where text files
are processed using unicode character strings rather than Python 2
byte "str" objects. The encoding can be specified explicitly on
individual methods the open files, or globally based on file names
or extensions by overriding the new `ProcessUser.getDefaultFileEncoding()`
method. For example, `getDefaultFileEncoding` could be overridden to
specify that .xml files should be treated as UTF-8 by default. If
the encoding is not specified explicitly or through
`getDefaultFileEncoding()`, Python selects the preferred encoding based
on the locale that it is running in.
- Changed the way multiple cycles are executed in multi-threaded mode to
allow tests from different cycles to execute in parallel instead of waiting
for each cycle to fully complete before starting the next cycle. This
improved parallelism makes it much easier to reproduce race
conditions demonstrated by a single testcase, which was not possible
with the previous threading behaviour. To maintain existing
behaviour for users who have provided a `runner.cycleComplete()` method,
concurrent cycle execution will be disabled if `cycleComplete()` is overridden.
Anybody affected by this is encouraged to transition away from use of
`cycleComplete()` and perform any required cleanup tasks in
`BaseTest.cleanup()` or `BaseRunner.testComplete()` instead.
- Added `<requires-python>` and `<requires-pysys>` elements to the project XML
file which allow checking for the specified minimum python or pysys
version, resulting in a clear error if attempting to use the wrong
version.
- Added support for coloring console output to highlight passes, fails,
warnings and more. This is configured in the project configuration file.
Coloring can also be enabled or disabled for a particular user and/or
machine using the `PYSYS_COLOR=true/false` environment variable override.
Coloring works on any terminal that supports ANSI escape sequences (e.g.
most Unix terminals). On Windows, which does not, it is possible to get
colored output by installing a package such as "colorama", which PySys will
load if it is present on the python path. It is possible to customize the
colors used or to use alternative libraries for coloring on windows by
providing a custom ColorLogFormatter class. The colors used for each
category of log messages can be customized in the project configuration
file, e.g. ::

<formatter><property name="color:timed out" value="MAGENTA"/></formatter>

- Added `ProcessUser.getExprFromFile` helper method to automate the common task
of retrieving some text from a file, for example to capture information
such as a process identifier from a log file, or to extract some
performance results that were logged.
- Added `BaseTest.reportPerformanceResult()` and a flexible framework for
recording performance results (e.g. throughput, latency etc) measured
by PySys tests, including storage of results in a human-readable and
machine-parsable CSV file together with run-specific information
such as the host where the test was executed. The CSV files can be
aggregated across multiple test runs and/or cycles and imported into
any spreadsheet for comparisons and more detailed analysis. The standard
CSVPerformanceReporter can be subclassed and replaced with an alternative
recording mechanism if desired (e.g. writing directly to a database or
other file format). Fibonacci_test_005 demonstrates how performance
results can be reported using this framework.
- Added support for providing a custom class to implement formatting of
log messages, for both run.log and stdout. Errors in the `<formatters>` XML
node will now be treated as errors rather than being silently ignored.
- Changed pysys.py to ignore trailing slash characters on test ids, which
makes it easier to use shell tab completion to select a specific test.
- Fixed pysys.py command line parser to give a clear error if requested to
execute a non-existent test id, or if the test descriptor XML could
not be parsed. Previously invalid test ids would be either silently
ignored without an error, or would result in other test ids being
executed more than once.
- Fixed `ProcessUser.startProcess` to use the test output directory (rather
than the current working directory) as the root when a relative path is
specified for the workingDir argument.
- Fixed bug in which log level and exception tracebacks were being
inadvertently suppressed from the stdout console output when executing
from multiple threads.
- Fixed manual tester thread to report a BLOCKED outcome instead of hanging
if a fatal error occurs (e.g. Tck does not load due to DISPLAY not being
configured correctly).
- Added `BaseResultsWriter` class and associated docstring documentation to
make it easier to create new results writers.
- Changed standard record writers to report the number of cycles starting
from 1 rather than from 0 (which is consistent with how cycles are
displayed by the rest of PySys).
- Extended the concept of "writers" to include not just "record" writers
(which are enabled only when `--record` is specified) but also "summary"
writers which are always enabled and log a summary at the end of test
execution (if none is explicitly configured a default
`ConsoleSummaryResultsWriter` is instantiated), and "progress" writers
which are enabled only when `--progress` is specified and log progress
information throughout a run.
- The monolithic logic for writing a summary to the console at the end of
test execution has been refactored out of baserunner and into
the configurable and separately extendable `ConsoleSummaryResultsWriter` class.
Any baserunner subclasses that are currently overriding the summary printing
functionality and/or making use of the results dictionary returned by
`start()` should now switch to using "summary" writers instead. This
functionality will be removed in a future release and is now deprecated.
- The default summary results writer now has a configuration parameter
`showOutcomeReason` which causes the outcome reason string to be included
underneath each failure outcome, to provide a quick summary of what went
wrong.
- The default summary results writer now has a configuration parameter
`showOutputDir` which causes the path to the test's output directory
to be printed underneath each failure outcome, to make it easy to quickly
find and open the relevant files to debug the failure.
- Added a `--progress` command line option (can also be switched on using
the `PYSYS_PROGRESS=true` environment variable), which logs a summary of
how many test have executed, outcomes, a list of most recent failure
reasons and a list of what other tests are currently executing. This
provides very helpful feedback to the user while executing a long
test run. The progress reporting is implemented in a fully extensible
way using a new kind of 'progress' result writer. A custom progress
result writer class can be configured for a project; if none is
specified the default `ConsoleProgressResultsWriter` is added automatically.
- Fixed unexpected DEBUG logging on standard output after any of the
Python `logging.info/warn/error()` methods is called. This behaviour was
triggered if certain libraries (e.g SSL libraries) were not available
when python starts.
- Added `defaultIgnoreExitStatus` project property which controls whether
non-zero return codes from `startProcess()` result in test failures, when the
`ignoreExitStatus` flag is not explicitly specified. To retain the same
behaviour for existing projects, `defaultIgnoreExitStatus` is set to True if
the property is not configured in the project configuration. However to
promote best practice for new PySys projects, the example pysys project
configuration file sets `defaultIgnoreExitStatus` to False, which ensures
that processes that return failure codes are not ignored unless explicitly
intended by the author of the testcase.
- Fixed `waitForSocket`, which in previous versions immediately returned
success instead of waiting a valid socket connection as documented.
- If the test run is interrupted from the keyboard, the prompt that asks
whether to continue to run tests is no longer displayed if there are no more
tests left to run. The prompt can also be completely disabled using an
environment variable `PYSYS_DISABLE_KBRD_INTERRUPT_PROMPT=true`, for users who
prefer Ctrl+C to immediately terminate the test run in all cases.
- Added `pysys.utils.pycompat` module containing a small set of helpers for
writing code that works with Python 2 and Python 3.
- Fixed writing to process stdin so that if a character string is passed in it
will be converted to a byte object automatically, using the default
encoding. Previously, it was not possible to write character strings in
Python 3, and in Python 2 it would only work if they contained only ascii
characters.

--------------
- Added the errorExpr argument to the waitForSignal method. Occurrence of any
matches to expressions in this argument will terminate the waitForSignal
loop, allowing early exit prior to the timeout.
- Refactored reconfiguration of global logging out of the pysys __init__.py
class into the pysys.py launcher. This allows other applications making
use of the PySys framework to make their own logging decisions.
- Improved usability of the assertDiff method by writing out the unified
diff to a file in the output subdirectory so failures are easier to triage.
- Added the literal argument to the assertGrep method to avoid having to
escape regular expressions.
- Added the utils.fileutils module for miscellaneous file related utilities.

--------------
- The validateOnly option has been added to the pysys.py run launcher
task. When set the purge output subdirectory, setup and execute methods
on the test will not be invoked. This makes it easier to fix validation
errors in the test without the need to re-run the entire test.
- The logFileContents() method has been added to the pysys.basetest.BaseTest
class to allow logging of file contents to the run.log. This can be used
to provide additional diagnostic information to the run.log to assist
the triage of test failures.
- The CSVResultsWriter has been added to the set of test summary writers.
See the sample pysysproject.xml file for more details.
- It is now possible to specify a regex for matching in the test selection.
See the run usage for more details (pysys.py run -h).