Pysys

This release improves usability of the ``pysys`` command line, gives a better experience editing/running tests from IDEs,
improves performance and Ctrl+C cancellation handling, and adds a few new methods and parameters you can use in your tests.
This release also adds support for Python 3.11 and 3.12, and removes support for the end-of-life 3.6 and 3.7 Python releases.

Some new features you might want to try out include:

- Instead of running ``pysys print``, use the equivalent ``pysys ls`` to save some keystrokes. Try out the new ``--verbose``
flag which groups related tests (sorting by title), and prints the full path of each test file (e.g. ``XXX/pysystest.py``) so you can
easily open up any tests of interest directly from your IDE or shell.

- Now that ``pysys run`` accepts directory paths (not only test ids), you can use your shell's directory tab completion to help specify
the tests to run.

- Get better code completion and avoid warnings when editing tests inside your Python-enabled IDE/editor by following the
new best practices described in the PySys documentation. For example use mix-in helper classes (not ``<test-plugin>``) for
sharing logic between tests, use ``import pysys.basetest, pysys.mappers`` not ``import pysys`` at the start of
all tests, and make sure references to ``pysys.basetest.BaseTest`` in your test class class inheritance list are fully qualified.

To use a Python IDE with PySys, either configure it to use a venv/Python installation that has PySys installed, or
specify the path to the PySys package with a ``PYTHONPATH`` environment variable (e.g. using a ``.env`` file).
If your project has any custom Python extensions configured with a ``<pythonpath>`` element you should add that to your
editor's ``PYTHONPATH`` too.

- Use the new `BaseTest.assertGrepOfGrep()` method if you're extracting a line of text from a file and
then validating it with a regular expression. You'll get much more descriptive messages if a test fails using
this approach rather than doing it all in a single ``assertGrep``.

- See also the new `BaseTest.createThreadPoolExecutor`, `BaseTest.listDirContents` and `pysys.mappers.IncludeMatches` methods.

- Considering switching to ``.tar.gz`` or ``.tar.xz`` instead of ``.zip`` format to get smaller (better compressed) test output
directory archives when using `pysys.writer.testoutput.TestOutputArchiveWriter`.

- Add a `pysys.writer.outcomes.TextResultsWriter` with ``verbose`` set to true to your project configuration, and enable it
with an environment variable (e.g ``PYSYS_DEFAULT_ARGS=--writer verboseTextResultsWriter``) so that whenever you kick off a long test
run you can check on the failures while it executes (or after it's completed - even if you close your shell/window) just by browsing
through the text file. You can also pipe the file through ``sort`` to group related failures together.
Example configuration::

<writer classname="TextResultsWriter" module="pysys.writer" alias="verboseTextResultsWriter">
<property name="file" value="${testRootDir}/__pysys_results.${outDirName}.log"/>
<property name="verbose" value="true"/>
</writer>

- Use the environment variable ``PYSYS_PROJECT_APPEND`` to run PySys with any user-specific settings you like (e.g. custom writers)
without modifying the main project file.

Command line usability
----------------------

This release adds several enhancements aimed at making it easier to use the ``pysys`` command line:

- Added command line support for running and printing tests by their (absolute or relative) directory rather than their test id.
This allows use of shell completion, and also makes it possible to run all the tests found under a set of named subdirectories.
For example::

pysys run performance-tests correctness-tests/foo/Test_001

- Added ``pysys ls`` as a short and convenient alias for ``pysys print``.

- Added the pysys print sort option ``--sort dirAndTitle`` which first groups tests whose parent/base directory is the same
and then sorts by title. In many cases this is the most useful way of recursively listing a large set of test titles
(since titles in different parent directories often do not sort together very cleanly when using ``--sort title``).

- Added ``--dir`` (``-D``), ``--testfile`` (``-F``) and ``--title`` (``-T``) options to ``pysys print`` to allow listing the directories,
files (e.g. ``pysystest.py``) and optionally also the titles for each matching test (without the far more verbose output of ``--full``).

- Added pysys print option ``-v`` (``--verbose``) which provides a convenient way to list tests with a bit more
information for each one. The results are sorted by ``dirAndTitle`` (so that similar tests are grouped together), and both the title
and the main test file are displayed (equivalent to specifying ``--title --testfile``). The verbose listing is especially
useful for finding out what tests you have covering different areas. Displaying the test file paths makes it
easy to jump to any tests of interest (either when running from an IDE that supports hyperlinks or by copying the path
into a shell/command prompt). If you wish to display the directory instead of the test file, use ``-vD``.

- Added colouring of the test title when printing a test listing with both title and ``testfile``/``dir`` (e.g. ``--verbose``),
so that it stands out better from the other content.

- Changed pysys print ``--grep`` to print the test file path (not only the title), and to sort by ``dirAndTitle``.
Also, the matching part of each title is now highlighted in red (unless console colouring is disabled).

- Added ``verbose`` configuration option to `pysys.writer.outcomes.TextResultsWriter` which produces gives a lot more information about
each test result including the outcome reason, test title, and absolute path of the output directory (across two lines).

The new format could be useful for monitoring in-progress test runs (maybe you wish to start solving some of the failures while
the test run is still executing), or for performing bulk triage of a large set of failures at the end of a long test run. Opening
the log file in an IDE that supports jump-to-file would allow clicking the output directories for further investigation of failures
where needed.

Although in multi-threaded mode the order of test results in the file will be non-deterministic (whatever order the scheduler picks for executing
the tests), the format is designed so that tests are grouped by failure mode (outcome and reason) when the log file is run through a
shell ``sort`` command.

Example configuration::

<writer classname="TextResultsWriter" module="pysys.writer" alias="verboseTextResultsWriter">
<property name="file" value="${testRootDir}/__pysys_results.${outDirName}.log"/>
<property name="verbose" value="true"/>
</writer>

- Added ``--writer CLASSNAME|ALIAS`` argument to ``pysys run`` which can be used to selectively enable an individual writer that would not
otherwise be enabled. For example, use ``--writer TextResultsWriter`` to enable the text summary log without using ``--record`` which might enable
several other writers you don't need. This option could be used in a ``PYSYS_DEFAULT_ARGS`` environment variable to enable a writer
just for the current user. To enable a specific writer (if present in a given project) without enabling all writers of the same
class, set the ``alias=`` attribute on the writer configuration and specify the writer.

- Added ``--preserveEmptyOutputs`` argument to ``pysys run`` which disables the usual behaviour of deleting empty (zero-length) files and directories.
This could be useful for comparing successful vs failure runs of the same test if the presence of empty files/dirs is relevant to the outcome.
The default test output cleanup behaviour was also changed to delete empty directories (not just files), and to permit the deletion of
non-zero/purgeable files after a ``SKIPPED`` outcome (previously only a ``PASSED`` outcome would delete non-zero/purgeable files).

- Added environment variable ``PYSYS_PROJECT_APPEND`` which treats the main ``pysysproject.xml`` file as if it had additional XML from
the specified file appended to it. This allows user-specific settings such as custom writers to be added dynamically without
modifying the main project file. The XML file to append must have ``<pysysproject>`` as its root XML node.

IDE/editor support
------------------

Several changes make it easier to write and execute PySys tests from Python-enabled IDEs/editors:

- Changed the imports of the default "new testcase" templates to allow Python IDEs to correctly locate the PySys ``BaseTest`` class.
This allows for code assist/navigation/completion which would otherwise not work. To apply this change to existing tests,
change ``import pysys`` to ``import pysys.basetest, pysys.mappers``, and make sure references to `pysys.basetest.BaseTest`
in the class inheritance list are fully qualified.

To use a Python IDE with PySys, either configure it to use a Python installation or ``venv`` that has PySys installed, or
specify the path to the PySys classes with a ``PYTHONPATH`` environment variable (e.g. using a ``.env`` file).
If your project has any custom Python extensions configured with a ``<pythonpath>`` element, you should add that to your
editor's ``PYTHONPATH`` too.

- Defined a new recommended pattern for reusing logic across tests. Previously, users created multiple custom ``BaseTest`` subclasses or
used the ``<test-plugin>`` element in the project configuration to define reusable peices. Both are now discouraged.
``BaseTest`` subclassing creates the possibility of unpredictable bugs due to clashes in field/method names (and other multiple
inheritance dangers). The test plugin concept has the big downside that Python IDEs and editors cannot detect the ``self.PLUGIN_ALIAS``
configuration from the PySys project and therefore code completion and navigation are impossible.

The preferred approach is now to use standard Python for reuse, by creating "mix-in" helper classes that expose functionality using inheritance.
A key constraint is that the helper class you inherit from contain only a single field holding an instance that encapsulates all the functionality
to avoid name clashes.

In the getting started sample there is a ``MyServerHelper`` mix-in class that provides a field called ``self.myserver`` through
which all of the real functionality is encapsulated and exposed to individual tests for reuse. To use it all you need to do
is inherit the helper in any tests that need it::

from myorg.myserverhelper import MyServerHelper
class PySysTest(MyServerHelper, pysys.basetest.BaseTest):

def execute(self):
server = self.myserver.startServer(name="my_server")
...

Since this approach uses standard Python, any IDE will be able to give assistance for the ``myserver`` methods (provided your extension
classes are on its configured ``PYTHONPATH``).

Any number of helpers can be added to each test that needs them. Just ensure that the ``BaseTest`` class is listed last in the list of
classes your test inherits from.

- Changed the header at the start of each test's console output to always include ``Dir:``, which now gives the full absolute
path of the test directory (not just its parent). This is helpful for quickly navigating between tests when running inside
an IDE.

- Added project property ``pysysLogAbsolutePaths`` which can be set to a true/false value to control whether absolute paths will be logged
for assertion failure locations and in the summary of failed test directories.
To avoid clutter this remains false by default, unless the ``PYSYS_LOG_ABSOLUTE_PATHS`` environment variable is set or PySys detects it is being executed inside
an IDE. Currently only Visual Studio Code is supported automatically, though you can add support for the environment variables of other
IDEs using a project property like ``<property name="pysysLogAbsolutePaths" value="${eval: os.getenv('TERM_PROGRAM','')=='vscode' }"/>``.

Ctrl+C/SIGTERM handling
-----------------------

- Reimplemented the handling of signals such as Ctrl+C, SIGINT and SIGTERM to provide increased robustness and ensure child processes
are terminated and custom cleanup logic executed wherever possible. All these signals are now handled the same way, by setting
the new global flag `pysys.process.user.ProcessUser.isRunnerAborting` which informs all tests they should
complete as quickly as possible - but in an orderly way to allow for essential cleanup. The built-in PySys methods that perform
waiting all check this flag periodically, but if you have any custom code that performs polling or waiting using Python functions
like ``time.sleep`` it is recommended to change to using the `BaseTest.pollWait` method instead, which checks this flag and
raises a ``KeyboardInterrupt`` to abort the test.
In addition, a new method `pysys.process.user.ProcessUser.handleRunnerAbort` is called for each test that is currently executing, to provide a
means of unblocking tests that are waiting for something (such as a TCP socket) not covered by the above mechanism.
The default implementation simply terminates all processes, but this method can be overridden to provide additional
logic or to avoid killing any processes and leave all terminations to the test's synchronous cleanup method.

To ensure a rapid exit, starting new processes is disabled once termination
has begun. Once the cleanup phase of a test or runner beings, the restrictions are removed since it may be necessary to
run processes to perform cleanup (e.g. to terminate containers etc). The previous PySys behaviour of prompting interactively
once Ctrl+C is sent has been removed.

For advanced users, in addition to the boolean flag, there is an event on Windows `pysys.process.user.ProcessUser.isRunnerAbortingEvent`
and a file handle on Linux `pysys.process.user.ProcessUser.isRunnerAbortingHandle`
that can be waited on in combination with other Windows objects or Unix file handles to abort if a termination request happens.

If a second Ctrl+C or signal is sent more than 6 seconds after the original abort request, the PySys process will immediately
perform a hard/unclean exit, without further attempts to cleanup or to print information about test failures.

Helper functions to use in your tests
-------------------------------------

- Added `BaseTest.assertGrepOfGrep()` to simplify the common case of extracting a value using a grep
expression and then checks that value using another regular expression. This is a small wrapper method around
`BaseTest.assertThatGrep()`. This approach is superior to putting both extraction and validation into
a single `BaseTest.assertGrep()` regular expression due to improved messages on failure which make
tests easier to write and debug.

- Added a `BaseTest.createThreadPoolExecutor` method that creates a PySys-friendly thread pool to parallelize
slow operations (e.g. HTTP requests) in your tests. This should be used instead of Python's own thread pool
in order to provide correct logging, cleanup, and a more useful default maximum number of worker threads.

- Added `BaseTest.listDirContents` to provide easier access to the existing `pysys.utils.fileutils.listDirContents` method.

- Added `pysys.mappers.IncludeMatches` for including just substrings matching a regular expression
when doing a file copy or diff.

- Added ``mustExist`` option to `BaseTest.grepOrNone` and `BaseTest.grepAll` to avoid an exception when files do no exist,
and instead treat it the same as if they existed but did not contain the matching expression.

Minor improvements:

- Change the debug format when starting a process to only show environment variables whose values differ from the default environment.

- Improved `BaseTest.waitForGrep` robustness when the file being checked is big or contains long lines, by checking for timeout expiry during
each pass through the file not just at the end, and logging a warning if excessively large lines are found (given that evaluating
regular expressions on lines with 10,000s of characters can sometimes take a very long time).
If reading from a file that has long lines, consider adding the new `pysys.mappers.TruncateLongLines` to the mappers list to shorten them,
or setting the new ``self.grepTruncateIfLineLongerThan`` field of ``BaseTest``.
The ``waitForGrep`` method now logs warnings if dangerously long lines are detected (as defined by ``self.grepWarnIfLineLongerThan``).

- Added ``self.testStartTime`` to BaseTest, which gives the time when the test object was created.

- Improved outcome reason for `BaseTest.assertDiff` to include some of the differing lines.

- Added `pysys.utils.osutils.getUsableCPUCount` to provide the number of CPUs that are usable by PySys which may be less than
are available in the machine. This implementation attempts to use cgroups v1 or v2 configuration (``cpu.cfs_quota/period_us`` or ``cpu.max`` -
but not ``cpu.shares``) if present. The current implementation assumes cgroups is mounted at ``/sys/fs/cgroup``.

- Added ``encodingReplaceOnError`` boolean parameter to `BaseTest.assertGrep()`, `BaseTest.assertThatGrep()`, `BaseTest.assertGrepOfGrep()` and
`BaseTest.waitForGrep()`, which can be used to prevent exceptions while reading file content
when the file contains invalid/unexpected characters (or the encoding is set incorrectly).
This ensures detection of error conditions etc can still complete reliably even if some unexpected characters are found elsewhere in the file.

- Added ``message=`` for customizing the introductory log message in `BaseTest.logFileContents`.

Performance improvements
------------------------

- Improved performance of waiting for processes (on Windows by blocking with ``WaitForSingleObject`` instead of sleep polling,
and on Linux if kernel version is >=5.3 and Python>=3.9 using ``pidfd_open``).

- Improved the algorithm for selecting the default number of worker threads when running with ``--threads=auto`` or ``--ci``
to provide a default upper limit on the number of workers.

This is based on the `pysys.utils.osutils.getUsableCPUCount` value, but also follows the example of Python's
``ThreadPoolExecutor`` class in capping the number of workers, to avoid
taking over very wide servers which is likely to be pointless and/or counter-productive for performance due to the
Python Global Interpreter Lock. Initially this cap is set to the high value of 100 but it may be reduced in
future PySys releases. The upper limit can be overridden on a per-project basis by setting the project property ``pysysMaxWorkerThreads``.
The default number of threads can also be further lowered for specific machines or users by setting the ``PYSYS_DEFAULT_THREADS``
environment variable.

Project and test configuration
------------------------------

- Enhanced how user-provided ``userData`` in test descriptors is processed, to include automatic substitution of ``${...}`` project properties,
and to support specifying user data keys independently using ``_`` or ``.`` syntax as an alternative to providing an entire dictionary,
for example::

__pysys_user_data.myDataKey1__ = "a, ${PROJECT_PROP}, b"
__pysys_user_data_myDataKey2__ = 12345

- Added project property attribute ``path=`` as an alternative to ``value=`` for properties containing a path.
When ``path`` is used, paths are normalized to remove any ``/../`` components.

- Added a ``pysysdirconfig`` option for setting the default ``pysys make`` template to one already defined at project level or
in a parent ``pysysdirconfig``::

<set-default-maker-template name="my-test"/>

See :ref:`pysys/TestDescriptors:Sample pysysdirconfig.xml`.

- Improved test descriptor loading for shared Python test classes so that it is no longer necessary to set ``__pysys_python_module__ = "PYTHONPATH"`` when
setting a ``__pysys_python_class__`` that contains at least one dot character, for example ``mypackage.MyTestClass``.

Writers
-------

- Added support for ``.tar.gz`` and ``.tar.xz`` to `pysys.writer.testoutput.TestOutputArchiveWriter` which are both smaller
in many cases than ``.zip`` files. See the new ``format`` option to control this.

- Added configuration options to `pysys.writer.outcomes.JUnitXMLResultsWriter` for customizing how PySys test descriptor concepts
(test id, mode etc) are mapped to JUnit concepts, to suit the many different tools that have their own interpretations
of this file format. The default mapping produces suboptimal results in some tools, due to duplicating the test id in the ``<testsuite>``
and ``<testcase>`` names. A mapping that works better in many tools can be configured using::

<writer classname="JUnitXMLResultsWriter" module="pysys.writer">
...
<property name="testsuiteName" value="TESTID_PACKAGE"/>
<property name="testcaseName" value="TESTID_NO_PACKAGE_OR_MODE~MODE"/>
<property name="testcaseClassname" value=""/>
<writer>

- Added ``includeTestIf`` option to `pysys.writer.coverage.PythonCoverageWriter` and `pysys.writer.testoutput.CollectTestOutputWriter`
which allows creating multiple coverage writers that collect data from different subsets of your tests, for example to separate out
a full report from all tests from a smaller report from just the unit tests or smoke tests. However note this is not intended to solve
the problem of disabling coverage generation for performance/robustness tests - for that you should set the ``disableCoverage`` group
on the relevant tests or test directories.

- Added ``artifacts`` dictionary to `pysys.writer.outcomes.JSONResultsWriter` which records artifact paths published during execution of the tests, for
example code coverage and performance reports.

- Added `pysys.baserunner.BaseRunner.getInProgressTests` for getting a list of the tests that are currently executing across
all worker threads. If you suspect a test failure could be related to other tests executing at the same time, logging
this list could be helpful for debugging the problem.

Fixes
-----

- Fixed the GitHub Actions support to stop using the recently deprecated ``::set-output`` mechanism for publishing
output artifacts and instead use the ``GITHUB_OUTPUT`` variable.
- Upgraded the sample GitHub Actions to latest versions ready for upcoming github.com upgrade of Node version.
- Prevented duplicate tests passed on the command line resulting in the same test executing more than once.
- Fixed sensitivity to system clock changes by using ``time.monotonic`` rather than ``time.time`` for timeouts
and time measurements that do not need wall clock time.
- Added ``time=`` attribute to JUnit XML output in the ``<testcase>`` node (in addition to the ``<testsuite>`` node where
it already existed), which is required by some tools.

Minor behaviour changes
-----------------------

There should be no breaking changes in this release, however the following behaviour changes might be observed by some users:

- Changed `BaseTest.abort` to not override any existing failure outcomes. However, if the abort outcome is ``SKIPPED`` or some other
non-failure, the previous behaviour of overriding existing outcomes is preserved. To precisely control when to override previous
failure outcomes, use `BaseTest.addOutcome` instead, which has an ``override=`` option.
- Fixed `BaseTest.assertThatGrep` to strip trailing newlines from the matched part of the line in the rare cases where
the regular expression unintentionally captures newline characters (e.g. ``[^xyz]``). If you have tests that rely on
this behaviour it is recommended to fix them, but if that is impractical set the project property
``pysysLegacyAssertThatGrepNewLineBehaviour`` to true to retain the previous behaviour. This property may be removed in a future
major release.
- Improved the ``pysystestxml_upgrader.py`` sample script to accept one or more directory arguments instead of upgrading the current
working directory.
- Removed the ``authors`` field from the default ``pysys make`` template. If you find it useful feel free to add it to your
own per-project maker template.
- User-provided ``userData`` in test descriptors is now subject to automatic substitution of ``${...}`` project properties.
Such properties can be escaped using ``${$}`` if needed.
- Test descriptor loading now assumes modules are on the ``PYTHONPATH`` rather than in a ``run.py`` file if the module is not explicitly specified
and ``__pysys_python_class__`` contains at least one dot character, for example ``mypackage.MyTestClass``.
- The process exit status returned by ``pysys run`` if no tests were found matching the specified options is now ``9``
(instead of the default fatal error code of ``10`` as in previous versions).
- Added a fallback mechanism for writing log messages from unregistered threads (not created with a PySys
thread pool or background thread) to stdout, although with a warning to discourage it.
Previously such logging did not go anywhere. However. it is still best to
use a standard PySys mechanism for creating background threads since otherwise the logging is not written to
``run.log`` files for the test that generated it.

-----------------

The main changes are new features to help with triaging performance results, and improved usability for the new
``pysystest.py`` descriptor and modes support that was added in version 2.0.

This release also adds support for Python 3.10, upgrades to the sample GitHub Actions workflows, and includes a lot of
minor enhancements and fixes.

Usability enhancements to the ``pysystest.py`` descriptors:

- Added a ``__pysys_parameterized_test_modes__`` field to the ``pysystest.py`` descriptor which makes it easier to
configure a list of modes for a parameterized test that uses the same Python logic to cover multiple testing
scenarios. See :doc:`/pysys/UserGuide` for detailed information about modes.
- Renamed ``combineModeDimensions`` to `pysys.config.descriptor.TestModesConfigHelper.createModeCombinations` for
improved usability.
- Made the ``__pysys_groups__`` inheritance specifier ``inherit=true/false`` optional (defaults to ``true``) since in
most cases users would prefer not to worry about it.
- Test descriptors with a ``.py`` extension are now loaded using Python's own parser instead of the regular expression
approach used for non-Python ``pysys.*`` files. This allows normal Python syntax to be used for things like
the ``lambda`` expressions (in ``__pysys_modes__``) which is more intuitive compared to the PySys 2.0 approach
of nesting them inside multi-line strings. The samples and the default ``pysys make`` test template have been updated
accordingly.
All descriptor values should go at the start of the file, before any ``import`` statements - this is important for
efficient parsing. For optimum parsing performance, make sure your first import is an ``import XXX`` rather than
a ``from XXX import YYY`` statement.
- Added ``pysys make`` template substitution variable ``DEFAULT_DESCRIPTOR_MINIMAL`` as an alternative to
``DEFAULT_DESCRIPTOR`` for cases where you want to exclude most of the commented example lines.

New features related to performance testing:

- Added detailed documentation to the :doc:`/pysys/UserGuide` about how to use PySys for performance testing.
- Added a new performance reporter class `pysys.perf.reporters.JSONPerformanceReporter` which writes performance
results in a format that's easy to machine-read for handling by other systems. To use this instead of (or as well)
as the default CSV reporter, add ``<performance-reporter classname="..."/>`` elements to your project configuration.
- Added a new performance reporter class `pysys.perf.reporters.PrintSummaryPerformanceReporter` which prints a
summary of performance results (including mean and standard deviation calculation if aggregating across multiple
cycles) at the end of the test run.
This performance reporter is now added by default (alongside ``CSVPerformanceReporter``), if
no explicit list of ``<performance-reporters>`` has been configured in ``pysysproject.xml``.
- The default performance reporter class `pysys.perf.reporters.CSVPerformanceReporter` has a new property
called ``aggregateCycles`` which means automatically rewriting the summary file at the end of a run where you
have executed multiple cycles, to record aggregate statistics such as mean and standard deviation (with
``samples=cycles``) instead of individual results for each cycle. This is useful when cycling tests locally to
generate stable numbers for comparisons while optimizing your application.
- Extended the performance reporter API. Added a new class `pysys.perf.api.BasePerformanceReporter` for creating
custom reporters. Now you can have multiple performance reporters in the same project, and configure properties for
each using the same ``<property>`` or ``"key"="value"`` XML syntax as for writers. Performance reporters now have an
additional ``setup`` method which is called just after `pysys.baserunner.BaseRunner.setup`, and is now the best place
for any initialization (it is recommended to move any such code out of the ``__init__`` constructor which should
no longer be used in most cases).
- Moved the performance classes from ``pysys.utils.perfreporters`` to `pysys.perf`. The old module is deprecated but
will continue to work, so this is not a breaking change.
- Added ``cpuCount`` to the default ``runDetails`` dictionary, since it's useful information to have available,
especially in performance test summary files. This is the value returned by Python's ``os.cpu_count()`` function.
- Added a ``reportPerformanceResult`` boolean variable to ``BaseTest``, which can be set to ``True`` by any commands
that would make recording of performance results pointless (such as enablement of profiling).
- If no ``resultDetails`` are explicitly specified when reporting a result in a test that has modes, then the name and
parameters from the test's mode are recorded as the ``resultDetails``.
- Added a ``failureOutcome`` parameter to `BaseTest.assertThat` which could be used to add a `pysys.constants.BADPERF`
outcome for a performance assertion.
- Added ``pysys run`` option ``--sort=random`` which randomly sorts/shuffles the order of tests and modes within each
cycle. This is useful for reducing systematic performance interactions between different tests/modes when running
multiple cycles.
- Added a new test outcome `pysys.constants.BADPERF` which can be used instead of ``FAILED`` to indicate the measured
performance was deemed insufficient. Unlike other failure outcomes, ``BADPERF`` does not prevent subsequent numeric
results from being recorded by `BaseTest.reportPerformanceResult`.

Miscellaneous new features:

- Upgraded the sample GitHub workflows to use the new CodeCov uploader as the old v1 uploader was deprecated.
If you use this functionality in your own GitHub Actions workflow, you should make the same change.
- Added support for Python 3.10. Note that on Windows the following deprecation warning may be seen until PyWin32
releases version 304::

DeprecationWarning: getargs: The 'u' format is deprecated. Use 'U' instead.

- Changed the behavior of the ``assertMessage`` in all assertion methods (e.g. `BaseTest.assertGrep`) so that
instead of replacing the default PySys message (e.g. ``Grep on foo.txt contains "Bar"``), it is added before the
default message, when the assertion fails. This means there is no loss of information when using ``assertMessage=``,
making it easier to justify using it to provide a more high-level explanation of what each assertion should
achieve. For example::

self.assertLineCount('server.log', 'ERROR ', condition='<= 10',
assertMessage='Assert that throttling of error messages keeps them below configured limit')
- Added `pysys.writer.outcomes.JSONResultsWriter` which writes test outcomes (and the runner's ``runDetails``) to a
single machine-readable ``.json`` file.
- Added a ``json`` substitution value to the `pysys.writer.console.ConsoleFailureAnnotationsWriter` class which can
be used to provide comprehensive machine-readable information about each test result.
- Changed the behavior of `pysys.writer.console.ConsoleFailureAnnotationsWriter` so that if a format is provided by
the ``PYSYS_CONSOLE_FAILURE_ANNOTATIONS`` environment variable it will always override any ``format`` in the
``pysysproject.xml`` configuration.
- Added ``timeout`` and ``hard=True/False`` flags to `pysys.process.Process.stop`. Also added logic on Linux which
automatically attempts a SIGKILL if the SIGTERM times out (though it still raises an exception in this case).
- Added a ``closeStdinAfterWrite`` parameter to `pysys.process.Process.write` which can be used for child processes
that wait for "End Of File" before completing.

- The ``detailMessage`` passed to `BaseTest.waitForGrep` is now added at the beginning rather than the end of the
log line, to give more prominence to the user's high-level description of what is being waited for::

self.waitForGrep('server.log', 'Ready for .*', detailMessage='Waiting until server is up')

- Simplified how PySys interacts with Python's ``logging`` library. PySys now records log messages from any Python
logger category to ``run.log`` and the console, whereas previously only messages from log categories starting with
``pysys.*`` would be included. The log level for any Python logger can be changed using the
``-vcategory=DEBUG`` argument to ``pysys run``, and the category may be any Python log category, or may be a
category under the ``pysys.`` logger such as ``-vprocess=DEBUG``.
- Added ``<input-dir>!INPUT_DIR_IF_PRESENT_ELSE_TEST_DIR!</input-dir>`` as alternative syntax for
``<input-dir>!Input_dir_if_present_else_testDir!</input-dir>``.
- The `BaseTest.deleteDir` method (and the test output directory cleanup code) now changes permissions and file
attributes to permit deletion if possible. This is useful when tests execute tools that create read-only files.

Fixes:

- Added missing ``<skipped message="..."/>`` element in JUnit XML reports when a test is skipped.
- Ignore common editor swap/temporary file extensions such as ``~`` and ``.swp`` when identifying ``pysystest.*``
files. The environment variable ``PYSYS_IGNORED_PYSYSTEST_SUFFIXES`` allows additional exclusions to be added if
needed.
- Fixed ``IndexError`` during handling of a non-matching ``assertThat``.
- Fixed bug in which a directory named ``!Input_dir_if_present_else_testDir!`` could be created by ``pysys make``.
- Fixed a rare circular dependency import issue with ``pysys.constants.Project`` / ``PROJECT``.
- Fixed display of duplicate newlines when setting ``stripWhitespace=False`` in `BaseTest.logFileContents`.
- Removed the normal logging prefix from PySys in each `BaseTest.logFileContents` line to avoid distracting from the
contents of the file being displayed.
- When using ``--threads=auto``, the number of available CPUs is now based on the number available to the PySys
process (``len(os.sched_getaffinity(0))`` - on operating systems that support this concept) rather than the total
number of physical CPUs on the machine.
- Fixed the `pysys.writer.console.ConsoleFailureAnnotationsWriter` ``testFile`` fallback to point to the Python file
when there was no failure outcome.

Deprecations:

- Moved the performance classes from ``pysys.utils.perfreporters`` to `pysys.perf`. The old module is deprecated but
is not planned for removal so this is not a breaking change.

-----------------

------------
- Fixed methods such as `BaseTest.assertGrep` to treat ``ignores='a string'`` as a list containing that string,
rather than as separate expressions containing each letter in the string which could lead to ignoring lines
that shoudl not be ignored.
- Fixed the project property ``defaultEnvirons.ENVVAR`` added in 1.6.0 which did not in fact set the environment
variable as described (due to an additional unwanted ``.`` character); now it does.
- Avoid creating unnecessary runner output directory as a result of ``mkdir(runner.output+'/../xxx')`` by
normalizing paths before calling ``mkdir``.
- Fixed `BaseTest.assertLineCount` bug in which ``reFlags`` parameter was not honored.
- Fixed numerous Python warnings.
- Fixed bug in which `pysys.utils.fileutils.toLongPathSafe` and `pysys.utils.fileutils.mkdir` would incorrectly
capitalize the first letter when passed a relative path.
- Improved the formatting of ``pysys print --full`` so it is easier to read. Most items with empty or default values
are no longer shown, so you can focus on the information that's actually interesting.
- Fixed bug in which ``--modes`` argument would not be honored if running tests with ``--ci``.

Migration notes
---------------

Breaking changes
~~~~~~~~~~~~~~~~

The main changes that might require changes to existing projects/tests are:

- Removal of Python 2 and 3.5 support; the minimum supported Python version is now 3.6.
- When user-defined ``mappers=`` are used (for example during ``self.copy``; see also `pysys.mappers`), it is now an
error for a mapper to strip off the trailing ``\\n`` character at the end of each line, as failure to do so can have
unintended consequences on later mappers. This requirement is also more clearly documented.
- Some mistakes in the ``pysystest.xml`` structure that were previously tolerated will now produce stderr warning
messages (such as incorrectly nesting ``<modes>`` inside ``<groups>``) and others will produce a fatal error
(for example multiple occurrences of the same element). To find out if any tests need fixing up, just execute
``pysys print`` in your PySys project directory and act on any warning or error messages.
- The deprecated ``supportMultipleModesPerRun=false`` project property (only used in very old PySys projects) can no
longer be used - please change your tests to use the modern modes approach instead.
- On Windows the ``testDir`` (and the input/output/reference directories) no longer start with the ``\\?\``
long path prefix; instead this can be added for operations where it is needed using
`pysys.utils.fileutils.toLongPathSafe` (as the standard PySys methods already do, for example ``self.copy``).
Where possible it is recommended to avoid nesting tests and output directories so deeply that long path support is
needed.

The remaining breaking changes are unlikely edge cases or in rarely used APIs that are unlikely to affect many users:

- The ``pysys.xml`` package has been renamed to `pysys.config` to provide a more logical home for test descriptors
and project configuration. Aliases exist so nothing should break, however if you have added extra files to the
``pysys/xml/templates`` directory such as customized ``pysys makeproject`` templates these should now be moved to
the ``pysys/config/templates`` directory. It is also recommended to find/rename your framework extensions to use the
new name as the ``pysys.xml`` module name is deprecated and will be removed in a future
release.
- The deprecated ``pysys.process._stringToUnicode`` method is now removed, since in Python 3 it is a no-op.
- If you created a custom `pysys.config.descriptor.DescriptorLoader` subclass to manipulate modes, you need to change
it to work with `pysys.config.descriptor.TestMode` objects instead of strings, and to set at least one of them
to be a primary mode.
- It is now an error to have multiple ``pysystest.*`` filenames in a single directory, for example ``pysystest.py``
and ``pysystest.xml``.
- If a test's title ends with ``"goes here TODO"`` then the test will report a ``BLOCKED`` outcome, to encourage
test authors to remember to fill it in. This could cause some existing tests to start blocking, though only if
you have added a title ending with ``"goes here TODO"``.
- Removed undocumented internal module ``pysys.utils.loader``; no-one should be using this; if you are, use Python's
``importlib.import_module()`` instead.
- The ``pysys run --ci`` flag now excludes tests tagged with group ``manual`` (in addition to excluding the
``manual`` test type, since ``pysystest.py`` descriptors use groups for this rather than test type).
- The ``--json`` output of ``pysys.py print`` now has a dict representing the modes and their parameters
for the ``modes`` value instead of a simple list, and the ``xmlDescriptor`` field was renamed to ``descriptorFile``.
Also the non-JSON ``pysys print`` output has changed slightly, especially around modes; use ``--json`` instead of
parsing the non-JSON output directly .
- Removed the ``primaryMode`` attribute from `pysys.config.descriptor.TestDescriptor`, as this information is now
stored in the `pysys.config.descriptor.TestMode` object.

Deprecations
~~~~~~~~~~~~

- It is strongly recommended to use the new `pysys.constants.PREFERRED_ENCODING` constant instead of
Python's built-in ``locale.getpreferredencoding()`` function, to avoid thread-safety issues in your tests - use of
that function within tests should be considered as deprecated.
- If you have a custom `pysys.utils.perfreporter.CSVPerformanceReporter` subclass, the signatures for
`pysys.utils.perfreporter.CSVPerformanceReporter.getRunDetails` and
`pysys.utils.perfreporter.CSVPerformanceReporter.getRunHeader` have changed to include a ``testobj`` parameter.
Although this should not immediately break existing applications, to avoid future breaking changes you should
update the signatures of those methods if you override them to accept a ``testobj`` parameter and also any arbitrary
``**kwargs`` that may be added in future.
- The ``pysys.xml`` module is deprecated; rename any imports to use `pysys.config` instead.
- The `pysys.utils.fileunzip` module is deprecated; use `BaseTest.unpackArchive` instead. For example, replace
``unzip(gzfilename, binary=True)`` with ``self.unpackArchive(gzfilename, gzfilename[:-3])``.
- The (undocumented) ``DEFAULT_DESCRIPTOR`` constant is now deprecated and should not be used.
- The old ``<mode>`` elements are deprecated in favor of the new Python lambda syntax
(support for these won't be removed any time soon, but are discouraged for new tests).
- The `pysys.utils.pycompat` module is now deprecated; see the documentation inside that module for details on
how to upgrade code that is using it.
- The ``ConsoleMakeTestHelper`` class is now deprecated in favor of `pysys.launcher.console_make.DefaultTestMaker`.

A quick way to check for the removed and deprecated items using a regular expression is shown in the following grep
command::

grep -r "\(supportMultipleModesPerRun.*alse\|DescriptorLoader\|pysys.utils.loader\|_stringToUnicode\|pysys[.]xml\|pysys.utils.fileunzip\|[^_]DEFAULT_DESCRIPTOR\|pysys.utils.pycompat\|PY2\|string_types\|binary_type\|isstring[(]\|quotestring[(]\|openfile[(]\|ConsoleMakeTestHelper\|def getRunDetails\|def getRunHeader\|locale.getpreferredencoding\|addResource\|CommonProcessWrapper\|TEST_TEMPLATE\|DESCRIPTOR_TEMPLATE\|ThreadFilter\)" .

(This expression also contains some removed/deprecated items from the previous 1.6.0 release, though does not attempt to cover
any earlier releases).

Optional steps
~~~~~~~~~~~~~~
As the default may change in a future release, existing PySys projects are recommended to explicitly specify what
directory they wish to use to store test input by specifying one of the following 3 ``<input-dir>`` configurations::

<pysysproject>

<pysysdirconfig>

<!-- The default for PySys projects created before 2.0 -->
<input-dir>Input</input-dir>

<!-- Recommended for new projects - input files are stored in the testDir alongside pysystest.py -->
<input-dir>.</input-dir>

<!-- Special option added in PySys 2.0 that auto-detects based on presence of an Input/ dir; useful for getting
the new behaviour for new tests without the need to update or potentially create bugs in existing tests
-->
<input-dir>!Input_dir_if_present_else_testDir!</input-dir>

<!-- The following is preferred from 2.1 onwards:
<input-dir>!INPUT_DIR_IF_PRESENT_ELSE_TEST_DIR!</input-dir>
-->


</pysysdirconfig>

</pysysproject>

Many users will prefer to use the new ``pysystest.py`` style for newly created tests alongside older tests using
the ``pysystest.xml`` style. However for anyone who wants to switch entirely to the new style, a utility script for
automatically converting ``pysystest.xml`` + ``run.py`` tests to ``pysystest.py`` (without losing
version control history) is provided as part of the cookbook sample
at https://github.com/pysys-test/sample-cookbook/tree/main/util_scripts/pysystestxml_upgrader.py

By default ``pysys make`` will generate tests with a new-style ``pysystest.py`` file, but if you prefer to keep your
project using the previous ``pysystest.xml`` and ``run.py`` structure, just add this to your ``pysysdirconfig.xml`` to
configure ``pysys make`` to use a template that based around ``pysystest.xml`` instead::

<pysysdirconfig>

<maker-template name="pysys-xml-test" description="a pre-v2.0 PySys test with pysystest.xml and run.py files"
copy="${pysysTemplatesDir}/pysystest-xml-test/*"/>

</pysysdirconfig>

Some users may wish to run their tests with the ``PYTHONWARNINGS=error`` environment variable or ``-Werror`` command
line argument, which prevents use of language features that Python itself has deprecated or which are likely to
result in test bugs.

-------------------

when running on GitHub(R) Actions:

- Improved detection of the server (non-ephemeral/dynamic) port range on Windows(R) as used by
`BaseTest.getNextAvailableTCPPort()`. This was previously incorrect on recent Windows versions leading to
potential clashes with ephemeral/dynamic/local ports or an insufficient pool of server ports. In addition,
a warning is now logged if a machine is configured with no ports available for starting server processes,
and falls back to using the IANA server port range in this case. If you get this warning on Windows you can
it by reconfiguring your system (e.g. ``netsh int ipv4 set dynamicportrange tcp ...``) or if that's not possible,
by setting the ``PYSYS_PORTS`` environment variable.
- Fixed a `BaseTest.waitForSocket()` bug on macOS(R) in which the wait never succeeds although the socket is
listening.
- Reduced the ``TIMEOUTS['WaitForAvailableTCPPort']`` constant from 20 minutes to 5 minutes since a properly
configured system should not spend significant amounts of time waiting for ports and it is better to
know sooner if the port pool is exhausted.

-------------------

The significant new features of PySys 1.6.0 are grouped around a few themes:

- a new "plugins" concept to encourage a more modular style when sharing functionality between tests;
- easier validation with the new `BaseTest.assertThatGrep()` method, which extracts a value using a grep
expression and then checks its value is as expected. For extract-and-assert use cases this approach gives much
clearer messages when the assert fails than using assertGrep;
- new writers for recording test results, including GitHub(R) Actions support and a writer that produces .zip
archives of test output directories, plus new APIs to allow writers to publish artifacts, and to visit each of
the test's output files;
- a library of line mappers for more powerful copy and grep line pre-processing;
- process starting enhancements such as `BaseTest.waitForBackgroundProcesses()`, automatic logging of stderr when
a process fails, and `BaseTest.waitForGrep()` can now abort based on error messages in a different file;
- several pysys.py and project configuration enhancements that make running and configuring PySys easier.
- a new "getting started" `sample <https://github.com/pysys-test/sample-getting-started>`_ project which can be
easily forked from GitHub(R) to create new PySys-based projects. The sample also demonstrates common techniques
and best practices for writing tests in PySys.

As this is a major release of PySys there are also some changes in this release that may require changes to your
project configuration file and/or runner/basetest/writer framework extension classes you've written (though in most
cases it won't be necessary to change individual tests). These breaking changes are either to reduce the chance of
errors going undetected, or to support bug fixes and implementation simplification. So be sure to look at the upgrade
guide below if you want to switch an existing project to use the new version.

New Plugin API
--------------
This release introduces a new concept: test and runner "plugins" which provide shared functionality available for
use in testcases.

Existing users will be familiar with the pattern of creating one or more BaseTest framework subclasses to provide a
convenient place for functionality needed by many tests, such as launching the applications you're testing, or
starting compilation or deployment tools. This traditional approach of using *inheritance* to share functionality does
have some merits, but in many projects it can lead to unhelpful complexity because:

a) it's not always clear what functionality is provided by your custom subclasses rather than by PySys itself
(which makes it hard to know which documentation to look at)
b) there is no automatic namespacing to prevent custom functionality clashing with methods PySys may add in future
c) sometimes a test needs functionality from more than one base class, and it's easy to get multiple inheritance
wrong
d) none of this really lends itself well to third parties implementing and distributing additional PySys
capabilities to support additional tools/languages etc

So, in this release we introduce the concept of "plugins" which use *composition* rather than *inheritance* to
provide a simpler way to share functionality across tests. There are currently 3 kinds of plugin:

- **test plugins** NB: as of PySys 2.2 these are no longer recommended - instead the "helper class" paradigm is preferred.

- **runner plugins**; these are instantiated just once per invocation of PySys, by the BaseRunner,
before `pysys.baserunner.BaseRunner.setup()` is called. Unlike test plugins, any processes or state they maintain are
shared across all tests. These can be used to start servers/VMs that are shared across tests.
Runner plugins are configured with ``<runner-plugin classname="..." alias="..."/>`` and can be any Python
class provided it has a method ``setup(self, runner)`` (and no constructor arguments).

- **writer plugins**: this kind of plugin has existed in PySys for many releases and are effectively a special kind of
runner plugin with extra callbacks to allow them to write test results and/or output files to a variety of
destinations. Writers must implement a similar but different interface to other runner plugins; see `pysys.writer`
for details. They can be used for everything from writing test outcome to an XML file, to archiving output files, to
collecting files from each test output and using them to generate a code coverage report during cleanup at the end
of the run.

A test plugin could look like this::

class MyTestPlugin(object):
myPluginProperty = 'default value'
"""
Example of a plugin configuration property. The value for this plugin instance can be overridden using ``<property .../>``.
Types such as boolean/list[str]/int/float will be automatically converted from string.
"""

def setup(self, testObj):
self.owner = self.testObj = testObj
self.log = logging.getLogger('pysys.myorg.MyRunnerPlugin')
self.log.info('Created MyTestPlugin instance with myPluginProperty=%s', self.myPluginProperty)

testObj.addCleanupFunction(self.__myPluginCleanup)

def __myPluginCleanup(self):
self.log.info('Cleaning up MyTestPlugin instance')

An example of providing a method that can be accessed from each test
def getPythonVersion(self):
self.owner.startProcess(sys.executable, arguments=['--version'], stdouterr='MyTestPlugin.pythonVersion')
return self.owner.waitForGrep('MyTestPlugin.pythonVersion.out', '(?P<output>.+)')['output'].strip()

With configuration like this::

<pysysproject>
<test-plugin classname="myorg.testplugin.MyTestPlugin" alias="myalias"/>
</pysysproject>

... you can now access methods defined by the plugin from your tests using ``self.myalias.getPythonVersion()``.

You can add any number of test and/or runner plugins to your project, perhaps a mixture of custom plugins specific
to your application, and third party PySys plugins supporting standard tools and languages.

In addition to the alias-based lookup, plugins can get a list of the other plugin instances added through the XML
using ``self.testPlugins`` (from `BaseTest`) or ``self.runnerPlugins`` (from `pysys.baserunner.BaseRunner`), which
provides a way for plugins to reference each other without depending on the aliases that may be in use in a
particular project configuration.

For examples of the project configuration, including how to set plugin-specific properties that will be passed to
its constructor, see the sample ``pysysproject.xml`` file.

New and improved result writers
-------------------------------
- Added `pysys.writer.testoutput.TestOutputArchiveWriter` that creates zip archives of each failed test's output directory,
producing artifacts that could be uploaded to a CI system or file share to allow the failures to be analysed.
Properties are provided to allow detailed control of the maximum number and size of archives generated, and the
files to include/exclude.

- Added `pysys.writer.ci.GitHubActionsCIWriter` which if added to your pysysproject.xml will automatically enable
various features when run from GitHub(R) Actions including annotations summarizing failures, grouping/folding of
detailed test output, and setting output variables for published artifacts (e.g. performance .csv files, archived
test output etc) which can be used to upload the artifacts when present.

See `https://github.com/pysys-test/sample-getting-started` for an example workflow file you can copy into your
own project.

This uses the new `pysys.writer.api.TestOutcomeSummaryGenerator` mix-in class that can be used when implementing CI
writers to get a summary of test outcomes.

- Added `pysys.writer.api.ArtifactPublisher` interface which can be implemented by writers that support some concept of
artifact publishing, for example CI providers that 'upload' artifacts. Artifacts are published by
various `pysys.utils.perfreporter.CSVPerformanceReporter` and various writers
including `pysys.writer.testoutput.TestOutputArchiveWriter`.

- Added `pysys.writer.testoutput.CollectTestOutputWriter` which supercedes the ``collect-test-output`` feature,
providing a more powerful way to collect files of interest (e.g. performance graphs, code coverage files, etc) from
all tests and collate them into a single directory and optionally a .zip archive.
This uses the new `pysys.writer.api.TestOutputVisitor` writer interface which can be implemented by writers that wish
to visit each (non-zero) file in the test output directory after each test.

The CollectTestOutputWriter can be used standalone, or as a base class for writers that collect a particular kind
of file (e.g. code coverage) and then do something with it during the runner cleanup phase when all tests have
completed.

- Moved Python code coverage generation out to ``pysys.writer.testoutput.PythonCoverageWriter`` (as of 2.0,
it's now in `pysys.writer.coverage.PythonCoverageWriter`) as an example of how to use a plugin to add
code coverage support without subclassing the runner. Existing projects use this behind the scenes, but new projects
should add the writer to their configuration explicitly if they need it (see sample project).

- Added `pysys.writer.console.ConsoleFailureAnnotationsWriter` that prints a single annotation line to stdout for each test
failure, for the benefit of IDEs and CI providers that can highlight failures found by regular expression stdout
parsing. An instance of this writer is automatically added to every project, and enables itself if
the ``PYSYS_CONSOLE_FAILURE_ANNOTATIONS`` environment variable is set, producing make-style console output::

C:\project\test\MyTest_001\pysystest.py:12: error: TIMED OUT - Reason for timed out outcome is general tardiness (MyTest_001 [CYCLE 02])

The format can be customized using the ``PYSYS_CONSOLE_FAILURE_ANNOTATIONS`` environment variable, or alternatively
additional instances can be added to the project writers configuration and configured using the properties
described in the writer class.

- Added a ``runDetails`` dictionary to `pysys.baserunner.BaseRunner`. This is a dictionary of string metadata about
this test run, and is included in performance summary CSV reports and by some writers. The console summary writer
logs the runDetails when executing 2 or more tests.

The default runDetails contains a few standard values (currently these include ``outDirName``, ``hostname``, ``os``
and ``startTime``). Additional items can be added by runner subclasses in the `pysys.baserunner.BaseRunner.setup()`
method - for example you could add the build number of your application (perhaps read
using `pysys.utils.fileutils.loadProperties()`).

If you had previously created a custom `pysys.utils.perfreporter.CSVPerformanceReporter.getRunDetails()` method it
is recommended to remove it and instead provide the same information in the runner ``runDetails``.

- Added property ``versionControlGetCommitCommand`` which if set results in the specified command line
being executed (in the testRootDir) when the test run starts and used to populate the ``vcsCommit`` key in the
runner's ``runDetails`` with a commit/revision number from your version control system. This is a convenient way to
ensure writers and performance reports include the version of the application you're testing with.

There are also some more minor enhancements to the writers:

- The `pysys.writer` module has been split up into separate submodules. However the writers module imports all symbols
from the new submodules, so no change is required in your code or projects that reference pysys.writer.XXX classes.

- Added `pysys.writer.console.ConsoleSummaryResultsWriter` property for ``showTestTitle`` (default=False) as sometimes seeing
the titles of tests can be helpful when triaging results. There is also a new ``showTestDir`` which allows the
testDir to be displayed in addition to the output dir in cases where the output dir is not located underneath
the test dir (due to --outdir). Also changed the defaults for some other properties to
showOutcomeReason=True and showOutputDir=True, which are recommended for better visibility into why tests failed.
They can be disabled if desired in the project configuration.

- Added a summary of INSPECT and NOTVERIFIED outcomes at the end of test execution (similar to the existing failures
summary), since often these outcomes do require human attention. This can be disabled using the properties on
`pysys.writer.console.ConsoleSummaryResultsWriter` if desired.

- Added `pysys.utils.logutils.stripANSIEscapeCodes()` which can be used to remove ANSI escape codes such as console
color instructions from the ``runLogOutput=`` parameter of a custom writer (`pysys.writer.api.BaseResultsWriter`),
since usually you wouldn't want these if writing the output to a file.

More powerful copy and line mapping
-----------------------------------
Manipulating the contents of text files is a very common task in system tests, and this version of PySys has
several improvements that make this easier:

- PySys now comes with some predefined mappers for common pre-processing tasks such as selecting multiple lines of
interest between two regular expressions, and stripping out timestamps and other regular expressions.

These can be found in the new `pysys.mappers` module and are particularly useful when using `BaseTest.copy()` to
pre-process a file before calling `BaseTest.assertDiff` to compare it to a reference file. For example::

self.assertDiff(self.copy('myfile.txt', 'myfile-processed.txt', mappers=[
pysys.mappers.IncludeLinesBetween('Error message .*:', stopBefore='^$'),
pysys.mappers.RegexReplace(pysys.mappers.RegexReplace.DATETIME_REGEX, '<timestamp>'),
]),
'reference-myfile-processed.txt')

(Note that for convenience we use the fact that copy() returns the destination path to allow passing it directly
as the first file for assertDiff to work on).

- `BaseTest.assertGrep` has a new mappers= argument that can be used to pre-process the lines of a file before
grepping using any mapper function. The main use of this is to allow grepping within a range of lines, as defined by
the `pysys.mappers.IncludeLinesBetween` mapper::

self.assertGrep('example.log', expr=r'MyClass', mappers=[
pysys.mappers.IncludeLinesBetween('Error message.* - stack trace is:', stopBefore='^$') ])

This is more reliable than trying to achieve the same effect with `BaseTest.assertOrderedGrep` (which can give
incorrect results if the section markers appear more than once in the file). Therefore, in most cases it's best to
avoid assertOrderedGrep() and instead try to use `BaseTest.assertDiff` or `BaseTest.assertGrep`.

- `BaseTest.waitForGrep` and `BaseTest.getExprFromFile` also now support a mappers= argument.

- When used from `BaseTest.copy` there is also support for line mappers to be notified when starting/finishing a new
file, which allows for complex and stateful transformation of file contents based on file types/path if needed.

- `BaseTest.copy` can now be used to copy directories in addition to individual files.

It is recommended to use this method instead of ``shutil.copytree`` as it provides a number of benefits including
better error safety, long path support, and the ability to copy over an existing directory.

- `BaseTest.copy` now permits the source and destination to be the same (except for directory copies) which allows it
to be used for in-place transformations.

- `BaseTest.copy` now copies all file attributes including date/time, not just the Unix permissions/mode.

Assertion improvements
----------------------

- Added `BaseTest.assertThatGrep()` which makes it easier to do the common operation of extracting a value using grep
and then performing a validation on it using `BaseTest.assertThat`.

This is essentially a simplified wrapper around the functionality added in 1.5.1, but avoids the need for slightly
complex syntax and hopefully will encourage people to use the extract-then-assert paradigm rather than trying to do
them both at the same time with a single `BaseTest.assertGrep` which is less powerful and produces much less
informative messages when there's a failure.

The new method is very easy to use::

self.assertThatGrep('myserver.log', r'Successfully authenticated user "([^"]*)"',
"value == expected", expected='myuser')

In cases where you need multiple regex groups for matching purpose, name the one containing the value using (?P<value>...)
self.assertThatGrep('myserver.log', r'Successfully authenticated user "([^"]*)" in (?P<value>[^ ]+) seconds',
"0.0 <= float(value) <= 60.0")


- All assertion methods that have the (deprecated and unnecessary) ``filedir`` as their second positional (non-keyword)
argument now support the more natural pattern of giving the expr/exprList as the second positional argument,
so instead of doing ``self.assertGrep('file', expr='Foo.*')`` you can also now use the more
natural ``self.assertGrep('file', 'Foo.*')``. For compatibility with existing testcases, the old signature of
``self.assertGrep('file', 'filedir', [expr=]'expr')`` continues to behave as before, but the recommended usage
in new tests is now to avoid all use of filedir as a positional argument for consistency and readability. (If you
need to set the filedir, you can use the keyword argument or just add it as a prefix to the ``file`` argument).

Simpler process handling
------------------------

- `BaseTest.startProcess()` now logs the last few lines of stderr before aborting the test when a process fails. This
behaviour can be customized with a new ``onError=`` parameter::

Log stdout instead of stderr
self.startProcess(..., onError=lambda process: self.logFileContents(process.stdout, tail=True))

Unless stderr is empty, log it and then use it to extract an error message (which will appear in the outcome reason)
self.startProcess(..., onError=lambda process: self.logFileContents(process.stderr, tail=True) and self.getExprFromFile(process.stderr, 'Error: (.*)')

Do nothing on error
self.startProcess(..., onError=lambda process: None)

- `BaseTest.waitForGrep` has a new optional ``errorIf=`` parameter that accepts a function which can trigger an abort
if it detects an error condition (not only in the file being waited on, as ``errorExpr=`` does). For example::

self.waitForGrep('myoutput.txt', expr='My message', encoding='utf-8',
process=myprocess, errorIf=lambda: self.getExprFromFile('myprocess.log', ' ERROR .*', returnNoneIfMissing=True))

- `BaseTest.waitProcess()` now has a ``checkExitStatus=`` argument that can be used to check the return code of the
process for success.

- Added `BaseTest.waitForBackgroundProcesses()` which waits for completion of all background processes and optionally
checks for the expected exit status. This is especially useful when you have a test that needs to execute
lots of processes but doesn't care about the order they execute in, since having them all execute concurrently in the
background and then calling waitForBackgroundProcesses() will be a lot quicker than executing them serially in the
foreground.

- Added a way to set global defaults for environment variables that will be used by `BaseTest.startProcess()`, using
project properties. For example, to set the ``JAVA_TOOL_OPTIONS`` environment variable that Java(R) uses for JVM
arguments::

<property name="defaultEnvirons.JAVA_TOOL_OPTIONS" value="-Xmx512M"/>

When you want to set environment variables globally to affect all processes in all tests, this is simpler than
providing a custom override of `BaseTest.getDefaultEnvirons()`.

- `BaseTest.startProcess()` now accepts an ``info={}`` argument which can hold a dictionary of user-defined metadata
about the process such as port numbers, log file paths etc.

pysys.py and project configuration improvements
-----------------------------------------------

- Added environment variable ``PYSYS_DEFAULT_ARGS`` which can be used to specify default arguments that the current
user/machine should use with pysys run, to avoid the need to explicitly provide them on the command line
each time, for example::

PYSYS_DEFAULT_ARGS=--progress --outdir __pysys_outdir
pysys.py run

- The sample project file and project defaults introduce a new naming convention of ``__pysys_*`` for output
directories and files created by PySys (for example, by writers). This helps avoid outputs getting mixed up with
testcase directories and also allows for easier ignore rules for version control systems.

- Added command line option ``-j`` as an alias for ``--threads`` (to control the number of jobs/threads). The old
command line option ``-n`` continues to work, but ``-j`` is the main short name that's documented for it.
As an alternative to specifying an absolute number of threads, a multiplier of the number of cores in the machine
can be provided e.g. ``-j x1.5``. This could be useful in CI and other automated testing environments.
Finally, if only one test is selected it will single-threaded regardless of the ``--threads`` argument.

- Added support for including Python log messages for categories other than pysys.* in the PySys test output,
using a "python:" prefix on the category name, e.g.::

pysys run -v python:myorg.mycategory=debug

Note that this ``python:`` prefix is deprecated and no longer required from PySys 2.1 onwards.

- Added ``pysys run --ci`` option which automatically sets the best defaults for non-interactive execution of PySys
to make it easier to run in CI jobs. See ``pysys run --help`` for more information.

- Added convention of having a ``-XcodeCoverage`` command line option that enables coverage for all supported
languages. You may wish to add support for this is you have a plugin providing support for a different language.

- Added a standard property ``${os}`` to the project file for finer-grained control of platform-specific properties.
The new ``${os}`` property gets its value from Python's ``platform.system().lower()``, and has values such
as ``windows``, ``linux``, ``darwin``, etc. For comparison the existing ``${osfamily}`` is always either
``windows`` or ``unix``.

- Added a standard property ``${outDirName}`` to the project file which is the basename from the ``-outdir``, giving
a user-customizable "name" for the current test run that can be used in project property paths to keep test
runs separate, for example, this could be used to label performance CSV files from separate test runs with
``--outdir perf_baseline`` and ``--outdir after_perf_improvements``.

- The standard project property ``testRootDir`` is now defined automatically without the need to
add the boilerplate ``<property root="testRootDir"/>`` to your project configuration. The old property name ``root``
continues to be defined for compatibility with older projects.

- When importing a properties file using ``<property file=... />" there are some new attributes available for
controlling how the properties are imported: ``includes=`` and ``excludes=`` allow a regular expression to be
specified to control which properties keys in the file will be imported, and ``prefix=`` allows a string prefix to
be added onto every imported property, which provides namespacing so you know where each property came from and a
way to ensure there is no clash with other properties.

- Added a handler for notifications from Python's ''warnings'' module so that any warnings are logged to run.log with
a stack trace (rather than just in stderr which is hard to track down). There is also a summary WARN log message at
the end of the test run if any Python warnings were encountered. There is however no error so users can choose when
and whether to deal with the warnings.

- Colored output is disabled if the ``NO_COLOR`` environment variable is set; this is a cross-product standard
(https://no-color.org/). The ``PYSYS_COLOR`` variable take precedence if set.

- Code coverage can now be disabled automatically for tests where it is not wanted (e.g. performance tests) by adding
the ``disableCoverage`` group to the ``pysystest.*`` descriptor, or the ``pysysdirconfig.xml`` for a whole
directory. This is equivalent to setting the ``self.disableCoverage`` attribute on the base test.

- `Python code coverage <pysys.writer.coverage.PythonCoverageWriter>` now produces an XML ``coverage.xml`` report
in addition to the ``.coverage`` file and HTML report. This is useful for some code coverage UI/aggregation services.

- The prefix "__" is now used for many files and directories PySys creates, to make it easier to spot which are
generated artifacts rather than checked in files. You may want to add ``__pysys_*`` and possibly ``__coverage_*``
to your version control system's ignore patterns so that paths created by the PySys runner and performance/writer
log files don't show up in your local changes.

Miscellaneous test API improvements
-----------------------------------

- Added `pysys.utils.fileutils.loadProperties()` for reading .properties files, and `pysys.utils.fileutils.loadJSON()`
for loading .json files.

- `BaseTest.logFileContents` now has a global variable ``self.logFileContentsDefaultExcludes`` (default ``[]``) which
it uses to specify the line exclusion regular expressions if no ``excludes=[...]`` is passed as a parameter. This
provides a convenient way to filter out lines that you usually don't care about at a global level (e.g. from a
`BaseTest.setup` method shared by all tests), such as unimportant lines logged to stderr during startup of
commonly used processes which would otherwise be logged by `BaseTest.startProcess` when a process fails to start.

- Added `BaseTest.disableLogging()` for cases where you need to pause logging (e.g. while repeatedly polling) to avoid
cluttering the run log.

- Added `pysys.config.project.Project.getProperty()` which is a convenient and safe way to get a project property
of bool/int/float/list[str] type. Also added `pysys.baserunner.BaseRunner.getXArg()` which does the same thing for
``-Xkey=value`` arguments.

- `BaseTest.getExprFromFile` now supports ``(?P<groupName>...)`` named regular expression groups, and will return
a dictionary containing the matched groups if any are present in the regular expression. For example::

authInfo = self.getExprFromFile('myserver.log', expr=r'Successfully authenticated user "(?P<username>[^"]*)" in (?P<authSecs>[^ ]+) seconds\.'))

- Added `BaseTest.getOutcomeLocation()` which can be used from custom writers to record the file and line number
corresponding to the outcome, if known.

Bug fixes
---------

- In some cases foreground processes could be left running after timing out; this is now fixed.

- Ensure ANSI escape codes (e.g. for console coloring) do not appear in JUnit XML writer output files, or in test
outcome reasons.

- Setting the project property ``redirectPrintToLogger`` to any value (including ``false``) was treated as if
it had been set to ``true``; this is now fixed.

Upgrade guide and compatibility
-------------------------------

As this is a major version release of PySys we have taken the opportunity to clean up some aspects which could
cause new errors or require changes. In many cases it will be necessary to make changes to your project configuration,
and code changes if you have created custom BaseRunner/BaseTest/writer subclasses - though individual tests will
generally not require changes, so the total migration effort should be small.

The changes that everyone should pay attention to are:

- The default values of several project properties have been changed to reflect best practice.

If you are migrating an existing project we recommend sticking with the current behaviour to start with, by adding
the following properties to your project configuration (except for any that you already define ``<property .../>``
overrides for). Then once the PySys upgrade is complete and all tests passing you can switch to some of the new
defaults (by removing these properties) if and when convenient.

The properties you should set to keep the same behaviour as pre-1.6.0 versions of PySys are::

<!-- Whether tests will by default report a failure outcome when a process completes with a non-zero return code.
The default value as specified below will be used when the ignoreExitStatus= parameter to the function is not
specified. The default was changed to false in PySys 1.6.0. -->
<property name="defaultIgnoreExitStatus" value="true"/>

<!-- Whether tests will abort as soon as a process or wait operation completes with errors, rather than attempting
to limp on. The default value as specified below will be used when the abortOnError parameter to the function
is not specified. Default was changed to true in PySys 1.6.0. -->
<property name="defaultAbortOnError" value="false"/>

<!-- Recommended behaviour is to NOT strip whitespace unless explicitly requested with the stripWhitespace=
option; this option exists to keep compatibility for old projects. The default was changed to false
in PySys 1.6.0. -->
<property name="defaultAssertDiffStripWhitespace" value="true"/>

<!-- Overrides the default name use to for the runner's ``self.output`` directory (which may be used for things
like code coverage reports, temporary files etc).
The default was changed to "__pysys_runner.${outDirName}" in PySys 1.6.0.
If a relative path is specified, it is relative to the testRootDir, or if an absolute --outdir was specified,
relative to that directory.
-->
<property name="pysysRunnerDirName" value="pysys-runner-${outDirName}"/>

<!-- Overrides the default name use to for the performance summary .csv file. The default was changed to
"__pysys_performance/${outDirName}_${hostname}/perf_${startDate}_${startTime}.${outDirName}.csv" in PySys 1.6.0.
-->
<property name="csvPerformanceReporterSummaryFile" value="performance_output/${outDirName}_${hostname}/perf_${startDate}_${startTime}.csv"/>

<!-- Set this to true unless you used the "mode" feature before it was redesigned in PySys 1.4.1. -->
<property name="supportMultipleModesPerRun" value="false"/>

<!-- Set temporary directory end var for child processes to the testcase output directory to avoid cluttering up
common file locations. Empty string means don't do this. "self.output" is recommended.
-->
<property name="defaultEnvironsTempDir" value=""/>

<!-- Controls whether print() and sys.stdout.write() statements will be automatically converted into logger.info()
calls. If redirection is disabled, output from print() statements will not be captured in run.log files and will
often not appear in the correct place on the console when running multi-threaded.

Note that this affects custom writers as well as testcases. If you have a custom writer, use
pysys.utils.logutils.stdoutPrint() to write to stdout without any redirection. -->
<property name="redirectPrintToLogger" value="false"/>

<!-- Produces more informative messages from waitForGrep/Signal. Can be set to false for the terser behaviour if
preferred. -->
<property name="verboseWaitForGrep" value="false"/>

The list is ordered with the properties most likely to break existing tests at the top of the list, so you may wish
to start with the easier ones at the bottom of the list.

- If you have testcases using the non-standard descriptor filenames ``.pysystest`` or ``descriptor.xml`` (rather
than the usual ``pysystest.xml``) they will not be found by this version of PySys by default, so action is required
to have them execute as normal. If you wish to avoid renaming the files, just set the new project
property ``pysysTestDescriptorFileNames`` to a comma-separated list of the names you want to use,
e.g. "pysystest.xml, .pysystest, descriptor.xml".

If you use the non-standard filename ``.pysysproject`` rather than ``pysysproject.xml`` for your project
configuration file you will need to rename it.

- If your BaseTest or BaseRunner makes use of ``-Xkey[=value]`` command line overrides with int/float/bool/list types, you
should review your code and/or test thoroughly as there are now automatic conversions from string to int/float/bool/list[str]
in some cases where previously the string type would have been retained.
a) -Xkey and -Xkey=true/false now consistently produce a boolean True/False
(previously -Xkey=true would produce a string ``"true"`` whereas -Xkey would produce a boolean ``True``) and
b) -X attributes set on BaseRunner now undergo conversion from string to match the bool/int/float/list type of the
default value if a static field of that name already exists on the runner class (which brings BaseRunner into line
with the behaviour that BaseTest has had since 1.5.0, and also adds support for the ``list`` type). This applies to
the attributes set on the object, but not to the contents of the xargs dictionary.

The same type conversion applies to any custom `pysys.writer` classes, so if you have a static variable providing a
default value, then in this version the variable will be set to the type of that bool/int/float/list rather than to
string.

So, as well as checking your tests still pass you should test that the configuration of your writers
and ``pysys.py run -X`` handling is also working as expected.

- Since `BaseTest.startProcess` now logs stderr/out automatically before aborting, if you previously wrote extensions
that manually log stderr/out after process failures (in a try...except/finally block), you may wish to remove them
to avoid duplication, or change them to use the new ``onError=`` mechanism.

- The default directory for performance output is now under ``__pysys_performance/`` rather than
``performance_output/``, so if you have any tooling that picks up these files you will need to redirect it, or set the
``csvPerformanceReporterSummaryFile`` project property described above. The default filename also includes
the ``${outDirName}``. See `pysys.utils.perfreporter`.

Be sure to remove use of the following deprecated items at your earliest convenience:

- Deprecated the ``ThreadFilter`` class. Usually it is not recommended
to suppress log output and better alternatives are available, e.g. the quiet=True option for `BaseTest.startProcess`,
and the `BaseTest.disableLogging()` method.
Please remove uses of ThreadFilter from your code as it will be removed in a future release.

- The method `pysys.basetest.BaseTest.addResource` is deprecated and will be removed in a future release, so please
change tests to stop using it; use `pysys.basetest.BaseTest.addCleanupFunction` instead.

- The ``pysys.process.commonwrapper.CommonProcessWrapper`` class is now renamed to `pysys.process.Process`. A
redirection module exists, so any code that depends on the old location will still work, but please change references
to the new name the old one will be removed in a future release.

- If you need code coverage of a Python application, instead of the built-in python coverage support e.g.::

<property name="pythonCoverageDir" value="__coverage_python.${outDirName}"/>
<property name="pythonCoverageArgs" value="--rcfile=${testRootDir}/python_coveragerc"/>
<collect-test-output pattern=".coverage*" outputDir="${pythonCoverageDir}" outputPattern="FILENAME_TESTID_UNIQUE"/>

change to using the new writer, e.g.::

<writer classname="pysys.writer.testoutput.PythonCoverageWriter">
<property name="destDir" value="__coverage_python.${outDirName}"/>
<property name="pythonCoverageArgs" value="--rcfile=${testRootDir}/python_coveragerc"/>
</writer>

(if using 2.0+, use `pysys.writer.coverage.PythonCoverageWriter` instead of
``pysys.writer.testoutput.PythonCoverageWriter``.

Finally there are also some fixes, cleanup, and better error checking that *could* require changes (typically to
extension/framework classes rather than individual tests) but in most cases will not be noticed. Most users can ignore
the following list and consult it only if you get new test failures after upgrading PySys:

- Timestamps in process monitor output, writers, performance reporter and similar places are now in local time instead
of UTC.
This means these timestamps will match up with the times in run.log output which have always been local time.
- Performance CSV files contain some details about the test run. A couple of these have been renamed: ``time`` is
now ``startTime`` and ``outdir`` is now ``outDirName``. The keys and values can be changed as needed using
the ``runDetails`` field of `pysys.baserunner.BaseRunner`. It is encouraged to use this rather than the previous
mechanism of `pysys.utils.perfreporter.CSVPerformanceReporter.getRunDetails()`.
- Exceptions from cleanup functions will now lead to test failures whereas before they were only logged, so may have
easily gone unnoticed. You can disable this using the new "ignoreErrors=True" argument to
`BaseTest.addCleanupFunction` if desired.
- Properties files referenced in the project configuration are now read using UTF-8 encoding if possible, falling back
to ISO8859-1 if they contain invalid UTF-8. This follows Java(R) 9+ behaviour and provides for more stable results
than the previous PySys behaviour of using whatever the default locale encoding is, which does not conform to any
standard for .properties file and makes it impossible to share a .properties file across tests running in different
locales. The PySys implementation still does not claim to fully implement the .properties file format, for example
``\`` are treated as literals not escape sequences. See `pysys.utils.fileutils.loadProperties()` for details.
- Duplicate ``<property name="..." .../>`` project properties now produce an error to avoid unintentional mistakes.
However it is still permitted to overwrite project properties from a .properties file.
You can also use the new ``includes``/``excludes`` attributes when importing a .properties file to avoid clashes.
- PySys used to silently ignore project and writer properties that use a missing (or typo'd) property or environment
variable, setting it to "" (or the default value if specified). To ensure errors are noticed up-front, it is now a
fatal error if a property's value value cannot be resolved - unless a ``default=`` value is provided in which case
the default is used (but it would be an error if the default also references a non-existent variable). This is
unlikely to cause problems for working projects, however if you have some unused properties with invalid values you
may have to remove them. The new behaviour only applies to ``<property name="..." value="..." [default="..."]/>``
elements, it does not apply to properties read from .properties files, which still default to "" if unresolved.
Run your tests with ``-vDEBUG`` logging if you need help debugging properties problems.
- The ``PYSYS_PERMIT_NO_PROJECTFILE`` option is no longer supported - you must now have a pysysproject.xml file for
all projects.
- Writer, performance and code coverage logs now go under ``--outdir`` if an absolute ``--outdir`` path is specified
on the command line rather than the usual location under ``testDirRoot/``.
- On Windows the default output directory is now ``win`` rather than the (somewhat misleading) ``win32``.
There is no change to the value of PySys constants such as PLATFORM, just the default output directory. If you
prefer a different output directory on your machine you could customize it by setting environment variable
``PYSYS_DEFAULT_ARGS=--outdir __myoutputdir``.
- If you created a custom subclass of `pysys.utils.perfreporter.CSVPerformanceReporter` using the 1.3.0 release and
it does not yet have (and pass through to the superclass) a ``runner`` and/or ``**kwargs`` argument you will need
to add these, as an exception will be generated otherwise.
- Made it an error to change project properties after the project has been loaded. This was never intended, as projects
are immutable. In the unlikely event you do this, change to storing user-defined cross-test/global state in your
runner class instead.
- Project properties whose name clashes with one of the pre-defined fields of `pysys.config.project.Project`
(e.g. "properties" or "root") will no longer override those fields - which would most likely not work correctly
anyway. If you need to get a property whose name clashes with a built-in member, use
`pysys.config.project.Project.properties`.
- PySys now checks that its working directory (``os.chdir()``) and environment (``os.environ``) have not been modified
during execution of tests (after `pysys.baserunner.BaseRunner.setup()'). Sometimes test authors do this by mistake
and it's extremely dangerous as it causes behaviour changes (and potentially file system race conditions) in
subsequent tests that can be very hard to debug.
The environment and working directory should only be modified for child processes not for PySys itself -
calling or overriding `BaseTest.getDefaultEnvirons()` is a good way to do this.
- Attempting to write to ``runDetails`` or ``pysys.constants.TIMEOUTS`` after `pysys.baserunner.BaseRunner.setup()`
has completed (e.g. from individual tests) is no longer permitted in the interests of safety.
- Changed the implementation of the outcome constants such as `pysys.constants.FAILED` to be an instance of class
`pysys.constants.Outcome` rather than an integer. It is unlikely this change will affect existing code (unless you
have created any custom outcome types, which is not documented). The use of objects to represent outcomes allows for
simpler and more efficient conversion to display name using a ``%s`` format string or ``str()`` without the need for
the LOOKUP dictionary (which still works, but is now deprecated). It also allows easier checking if an outcome
represents a failure using `pysys.constants.Outcome.isFailure()`. The `pysys.constants.PRECEDENT` constant is
deprecated in favor of `pysys.constants.OUTCOMES` which has an identical value.
- There is no longer a default writer so if you choose delete the <writers> element from your project you won't
have any writers.
- Removed undocumented ``TEST_TEMPLATE`` constant from ``pysys.basetest`` and ``DESCRIPTOR_TEMPLATE``
from `pysys.config.descriptor` (they're now constants on `pysys.launcher.console_make.ConsoleMakeTestHelper` if you
really need them, but this is unlikely and they are not part of the public PySys API).
- Removed deprecated and unused constant ``DTD`` from `pysys.config.project` and `pysys.config.descriptor`.
- Removed deprecated method ``purgeDirectory()`` from `pysys.baserunner.BaseRunner`
and `pysys.writer.outcomes.JUnitXMLResultsWriter`. Use `pysys.utils.fileutils.deletedir` instead.
- Removed deprecated classes ``ThreadedStreamHandler`` and ``ThreadedFileHandler`` from the
``pysys.`` module as there is no reason for PySys to provide these. These are trivial to implement using the
Python logging API if anyone does need similar functionality.
- `pysys.process.user.ProcessUser` no longer sets ``self.output``, and it sets ``self.input`` to the project's
testRootDir instead of the current directory. Since these are overridden by `pysys.basetest.BaseTest` and
`pysys.baserunner.BaseRunner` it is unlikely this will affect anyone.
- Changed the log messages at the end of a test run to say "THERE WERE NO FAILURES" instead of
"THERE WERE NO NON PASSES", and similarly for the "Summary of non passes:".
- `pysys.process.Process.wait` now raises an error if the specified timeout isn't a positive
number (giving the same behaviour as `BaseTest.waitProcess`) rather than the dangerous behaviour of waiting without
a timeout.

---------------
Release History
---------------

Documentation improvements:

PySys now uses Sphinx to build its documentation (instead of epydoc), and new content has also been written resulting
in a significantly larger set of HTML documentation that is also easier to navigate, and brings together
the detailed API reference with information on usage and how to get started with PySys. The main ``.rst``
documentation source files are shipped inside the binary distribution of PySys so that users can view and
potentially even re-package the documentation combined with their own extensions.

Assertion and waitForGrep improvements:

- `BaseTest.assertThat` has been radically overhauled with a powerful mechanism that uses named parameters (e.g.
``actualXXX=`` and ``expected=``) to produce self-describing log messages and outcome reasons, and even the ability to
evaluate arbitrary Python expressions in the parameters, for example::

self.assertThat("actualStartupMessage == expected", expected='Started successfully', actualStartupMessage=msg)
self.assertThat('actualUser == expected', expected='myuser', actualUser=user)

self.assertThat("actual == expected", actual__eval="myDataStructure['item1'][-1].getId()", expected="foo")
self.assertThat("actual == expected", actual__eval="myDataStructure['item2'][-1].getId()", expected="bar")
self.assertThat("actual == expected", actual__eval="myDataStructure['item3'][-1].getId()", expected="baz")

This automatically produces informative log messages such as::

Assert that (actual == expected) with actual (myDataStructure['item1'][-1].getId()) ='foo', expected='foo' ... passed
Assert that (actual == expected) with actual (myDataStructure['item2'][-1].getId()) ='bar', expected='bar' ... passed
Assert that (actual == expected) with actual (myDataStructure['item3'][-1].getId()) ='baZaar', expected='baz' ... failed
actual: 'baZaar'
expected: 'baz'
^

Note that when two named parameters are provided and the condition string is a simple equality
comparison (``==`` or ``is``), additional lines are logged when the assertion fails to show at what point the
two arguments differ. For best results make sure you have colours turned on.

As a result of these changes to assertThat, the less powerful `BaseTest.assertEval` method is now deprecated and
new tests should use assertThat instead.

Both methods also now allow the condition/eval string to make use of some additional standard Python modules such as
``math`` and ``re``, and to use ``import_module('...').XXX`` to dynamically import additional modules.

- `BaseTest.assertGrep` (and `BaseTest.assertLastGrep`) now return the regular expression match object, or if any
``(?P<groupName>...)`` named groups are present in the regular expression, a dictionary containing the matched values.
This allows matching values from within the regular expression in a way that produces nicely descriptive error
messages, and also enables more sophisticated checking (e.g. by casting numeric types to float). For example::

self.assertThat('username == expected', expected='myuser',
**self.assertGrep('myserver.log', expr=r'Successfully authenticated user "(?P<username>[^"]*)"'))

self.assertThat('0 <= float(authSecs) < max', max=MAX_AUTH_TIME,
**self.assertGrep('myserver.log', expr=r'Successfully authenticated user "[^"]*" in (?P<authSecs>[^ ]+) seconds\.'))

`BaseTest.waitForGrep` now provides the same dictionary return value when given a regular expression with named
groups, so the above trick can also be used during execution of the test when convenient.

- `BaseTest.waitForGrep()` has been added as a new and clearer name for `BaseTest.waitForSignal()`, and we recommend
using waitForGrep in new tests from now on (see upgrade section for more information about this change).

- `BaseTest.waitForGrep` (and `BaseTest.waitForSignal`) now logs more useful information if the
``verboseWaitForGrep`` (or its alias, ``verboseWaitForSignal``) is set to true in the ``pysysproject.xml``
properties. This includes logging at the start of waiting rather than at the end of waiting (to make it easier to
debug hangs during test development or when triaging an automated test run). In addition, if a non-default timeout
was specified this is included in the log message, and for the (small proportion of) waits that take longer than
30 seconds an additional message is logged to indicate how long was actually spent, which makes it easier to debug
tests that sometimes timeout and sometimes complete just before they would have timed out. All of this new
functionality only applies if you have ``verboseWaitForGrep=true`` so will not affect existing projects, but this
is now enabled for newly created projects.

- `BaseTest.waitForGrep` (and `BaseTest.waitForSignal`) now has a ``detailMessage`` parameter that can
be used to provide some extra information to explain more about the wait condition.

- All ``assertXXX`` methods in `BaseTest` now return a value to indicate the result of the assertion. In most
cases this is a boolean ``True``/``False``. This creates an opportunity to gather or log additional diagnostic
information (e.g. using `BaseTest.logFileContents`) after an assertion fails.

- Regular expression behaviour can now be customized by a ``reFlags=`` parameter on methods such as
`BaseTest.assertGrep`, `BaseTest.waitForGrep`, etc. This allows for ignoring case, and use of verbose regular
expression syntax, for example::

self.assertGrep('myserver.log', reFlags=re.VERBOSE | re.IGNORECASE, expr=r\"""
in\
\d + the integral part
\. the decimal point
\d * some fractional digits
\ seconds\. in verbose regex mode we escape spaces with a slash
\""")

- `BaseTest.assertDiff` now has colour-coding of the added/removed lines when logging a diff to the console on failure.

- `BaseTest.assertDiff` usability was improved by including the relative path to each file
in the assertion messages, so you can now use the same basename for the file to be compared and the reference
file without losing track of which is which. This also makes it easier to manually diff the output directory against
the ``Reference`` directory using GUI diff tools when debugging test failures.

- `BaseTest.assertDiff` has a new advanced feature, *autoUpdateAssertDiffReferences*, to help when you
have a large set of test reference files which need to be updated after a behaviour or output formatting change.
If you run the tests with ``-XautoUpdateAssertDiffReferences`` any diff failures will result in PySys overwriting
the reference file with the contents of the comparison file, providing an easy way to quickly update a large set
of references. Use this feature with caution, since it overwrites reference files with no backup. In
particular, make sure you have committed all reference files to version control before running the command, and
then afterwards be sure to carefully check the resulting diff to make sure the changes were as expected before
committing. From PySys 2.2 onwards the shorter option ``-XupdateDiffReferences`` may be used instead.

Improvements to the ``pysys.py`` tool:
- PySys now supports v3.8 of Python.

- Added ``Test directory`` to ``pysys print --full``. The directory is given as a path relative to the directory
PySys was run from.

New project options:

- The ``pysysproject.xml`` project configuration has a new ``<project-help>...</project-help>`` element which can be
used to provide project-specific text to be appended to the ``pysys run --help`` usage message. This could be useful
for documenting ``-Xkey=value`` options that are relevant for this project, and general usage information. A
``Project Help`` heading is automatically added if no other heading is present, and PySys will intelligently add or
remove indentation from the specified content so that it aligns with the built-in options.

- ``pysysproject.xml`` has a new property ``defaultAssertDiffStripWhitespace`` which controls whether
`BaseTest.assertDiff` ignores whitespace (and blank lines at the end of a file). The recommended
value is False, but to maintain compatibility with existing projects the default if not specified in the project file
is True.

- The ``<property name=.../>`` and ``<property file=.../>`` elements have a new optional attribute
called ``pathMustExist="true/false"`` that can be set to true to indicate that the project should not load (and no
tests be run) if the .properties file does not exist, or in the case of ``<property name=.../>``, if the property
value does not exist (either as an absolute path or as a path relative to the project root directory). We recommend
setting using ``pathMustExist`` on all ``<property file=.../>`` elements to be explicit about whether the file is
optional or mandatory.

- ``<pythonpath>`` can now be used (and is recommended) instead of ``<path>`` to add items to the PYTHONPATH. There is
no plan to remove support for ``<path>`` but this should increase clarity for new users.

Port allocation improvements:

- `BaseTest.getNextAvailableTCPPort` and `BaseTest.waitForSocket` now support IPv6, via the new
``socketAddressFamily`` argument (IPv4 remains the default). It is also possible now to control which host
address/interface is used to check that an allocated port isn't in use using the new ``hosts`` argument.

- A new environment variable ``PYSYS_PORTS=minport-maxport,port,...`` can be used to override the set of possible
server ports allocated from `BaseTest.getNextAvailableTCPPort()`. This avoids the usual logic which uses
`pysys.utils.portalloc.getEphemeralTCPPortRange()` to detect the local/client-side ports which should be avoided
for server-side use. In addition, the default behaviour of getEphemeralTCPPortRange has changed on Linux, so that
if ``/proc/sys/net/ipv4/ip_local_port_range`` is missing, PySys will fall back to using the default IANA ephemeral
port range (with a warning). This makes it possible to use PySys in environments such as
Windows Subsystem for Linux (WSL) v1 which may not have the usual Linux network stack.

Advanced pysystest.xml additions:

- It is now possible to use ``${...}`` project properties when specifying the Python module to load for a given test,
for example::

<data>
<class name="PySysTest" module="${testRootDir}/test-utils/custom_run_module.py"/>
</data>

- User-defined key/value data can be added to ``pysystest.xml`` (and will be inherited from any parent
``pysysdirconfig.xml`` files)::

<data>
<user-data name="myThing" value="foobar"/>
</data>

Any user-defined data is available as a string in the ``userData`` field of `self.descriptor <pysys.config.descriptor.TestDescriptor>`,
and each named value will be set as a variable on the `BaseTest` class. If a static (non-instance) variable of the same name
exists on the test class at construction then the ``<user-data>`` will override it, but its type will be coerced
automatically to an int/float/bool to match the type of the variable. A ``pysys.py run -Xname=value`` argument can be
specified to provide a temporary override for any items in the test's user data.
(from v2.2 onwards there is also automatic substituting of ``${...}`` properties in user data values).

Bug fixes:

- Handling of errors deleting previous test output has been improved. In 1.5.0, there was a usability regression in
which a test would fail to run if any part of its output directory could not be deleted due
to a shell or tool (e.g. tail) keeping it locked. Now, although error deleting files will still cause the test to
fail (since this has a high chance of affecting correctness), directory deletion errors are logged at WARN in the
test output but do not cause an error.

- Fixed bug in which ``BaseTest.assertDiff`` was not logging the diff to the console after a failure.

- Fixed bug in which a ``pysysdirconfig.xml`` in the same directory as a ``pysysproject.xml`` would be read twice,
potentially resulting in duplicated a ``id-prefix``.

- Fixed some bugs in the selection of test ids on the command line. Now we always prefer an exact match over any
possible suffix matches, and give an error if there are multiple matching suffixes rather than just picking one.

- Fixed 1.5.0 bug in which a ``-Xkey=value`` command line value of ``1`` or ``0`` would be converted to a boolean
True or False value instead of an int, when the `BaseTest` object has a field named ``key`` of type int.

- Fixed reading .properties file values that contain an equals ``=`` symbol.

- Replace new line characters in test outcome reasons to avoid confusing tools.

- Changed `BaseTest.getNextAvailableTCPPort` to check the allocated port isn't in use on ``localhost`` (previously
we only checked ``INADDR_ANY`` which doesn't include the ``localhost`` interface).

Upgrade guide and compatibility:
This is a minor release so is not expected to break existing tests, however we recommend reading the notes
below and making any 'recommended' changes at a convenient time after upgrading (to avoid problems in future major
upgrades), and also running your tests with the new version before upgrading to confirm everything still works as
expected.

- Default project property ``defaultAssertDiffStripWhitespace`` was added. It is recommended to add this to
your ``pysysproject.xml`` file set to false, but it is likely some test reference files may need fixing, so the
default value is True which maintains pre-1.5.1 behaviour.

- `BaseTest.waitForSignal()` is now just an alias for the newly added `BaseTest.waitForGrep()`, which is the
preferred method to use for waiting until a regular expression is found in a file. This is a bit of API cleanup that
provides consistency with widely-used `BaseTest.assertGrep()`, and increases clarity for new users who could
otherwise be unsure of the meaning of the term "signal".

The two methods are identical except for a small usability improvement in the method signature to avoid a common
mistake in which the (rarely used, and never needed) ``filedir`` was given a prominent position as the second
positional argument and therefore sometimes incorrectly given the value intended for the ``expr`` expression to be
searched, as can be seen from the two signatures::

def waitForSignal( self, file, filedir, expr='', ... )
def waitForGrep( self, file, expr='', ..., filedir=None )

In the new waitForGrep method, ``filedir`` can only be specified as a ``filedir=`` keyword argument, permitting the
more natural positional usage::

self.waitForGrep('file', 'expr', ...)

There is no plan to actually remove waitForSignal, however in the interests of consistency we'd recommend doing a
find-replace ``self.waitForSignal -> self.waitForGrep`` on your tests at a convenient time, bearing in mind that it
could result in test failures in the unlikely event you are setting ``filedir`` and doing so positionally rather
than with ``filedir=``.

If you use the ``verboseWaitForSignal`` project property, we recommend you transition to the new
``verboseWaitForGrep`` property, though both work on both methods for now.

- In `BaseTest.startProcess()`, ``background=True/False`` has been added as an alternative and simpler equivalent of
``state=BACKGROUND``. It is preferred to use ``background=True`` in new tests (although there is no plan to
remove ``state`` so it is not mandatory to change existing tests).

- The global namespace available for use in eval() methods such as `BaseTest.assertThat`, `BaseTest.assertEval`,
`BaseTest.assertLineCount` and `BaseTest.waitForGrep` has been cut down to remove some functions and modules
(e.g. ``filegrep``) that no-one is likely to be using. If you find you need anything that is no longer available,
just use ``import_module('modulename').member`` in your eval string to add it, but it is highly unlikely this will
affect anyone as none of the removed symbols were documented. Also `BaseTest.assertEval` is deprecated in
favor of `BaseTest.assertThat` which provides more powerful capabilities (note that `BaseTest.assertThat` was itself
previously deprecated, but after recent changes is now the preferred way to perform general-purpose assertions).

- There are some deprecations in this release, to remove some items that no-one is likely to be using from the API.
We encourage users to check for and remove any references to the following to be ready for future removal:

- ``pysys.utils.filecopy`` and its functions ``copyfileobj`` and ``filecopy`` are now deprecated (and hidden from the
documentation) as there are functions in Python's standard library module ``shutil`` that do the same thing.
- ``pysys.utils.threadpool`` is also deprecated and hidden from the public API as it was never really
intended for general purpose use and Python 3 contains similar functionality.
- The ``DTD`` constants in `pysys.config.project` and `pysys.config.descriptor`.
- ``pysys.config.descriptor.XMLDescriptorParser`` (replaced by `pysys.config.descriptor.DescriptorLoader`)
- ``pysys.config.descriptor.XMLDescriptorContainer`` (replaced by `pysys.config.descriptor.TestDescriptor`)
- ``pysys.config.descriptor.XMLDescriptorCreator`` and ``DESCRIPTOR_TEMPLATE`` (create descriptors manually if needed)