Benchexec

This release works only on Python 3.6 and newer!

- Add possibility to have a `close()` method to tool-info modules.
- The `test_tool_info` utility now has a `--debug` argument
that will show the debug log, e.g., from the tool-info module.
- Fix bug in detection of CPU throttling during benchmarking,
for which we warn if we detect it.
This did not work for cases with core numbers longer than one digit.
- Properly escape the suggested command line for running table-generator
that is shown by benchexec before it terminates.
- Allow specifying path to libseccomp via environment variable `LIBSECCOMP`.
This is useful for environments like NixOS.
- Fix handling of empty result files in table-generator,
the generated HTML tables will work again.
- Fix handling of empty run results when filtering rows in HTML tables.
- Make filters for status and textual columns in HTML tables work
if filter string contains a colon.
- When entering a filter in the HTML tables' filter row,
do not loose focus on the input field when applying the filter,
such that users can continue typing.
- Fix invalid values shown in score-based quantile plot
if some runs of a table do not have a score value.
Now a run set is only shown in a score-based quantile plot
if all values have scores (otherwise the plot would be misleading).
- Text selection now works as expected while an overlay window is open
in the HTML tables (only text in the overlay will be selected).

This is expected to be the last BenchExec release that supports Python 3.5,
newer releases will require Python 3.6 or newer.
Please [cf. issue 717](https://github.com/sosy-lab/benchexec/issues/717)
for our plan on dropping support for further Python versions.

We would like to note that Linux kernel 5.11
brings a major improvement for BenchExec users not on Ubuntu:
Now it should be possible to use the overlayfs feature as a regular user,
no need to pass `--read-only-dir /` and similar parameters.
We updated our [installation instructions](https://github.com/sosy-lab/benchexec/blob/master/doc/INSTALL.md)
accordingly and also clarified that BenchExec requires x86 or ARM machines
and recommend Linux kernel 4.14 or newer due to reduced cgroups overhead.

Changes in this release:

- In HTML tables, the following settings are now stored in the hash part of the URL:
- Column sorting
- Page size of the table, i.e., how many rows are shown
- Filters for task names that are defined by entering text
into the left-most input field of the filter row of the table.
Previously this would only work for task-name filters defined in the filter sidebar.
This means that using the back/forward navigation of the browser
will change these settings and that they can be present in shared links.
- Fix a few cases of printing of statistics information in HTML tables.
This affects corner cases like the number of visible decimal digits for `0`
and trailing zeroes for the standard deviation in the tooltip.
- When a user requests rounding to a certain number of decimal digits,
the filtering functionality of the HTML tables will now use the raw values,
not the rounded values.
This is consistent with the behavior when rounding is not explicitly requested
and BenchExec applies the default rounding rules.
- Fix harmless stack trace printed at end of `benchexec` execution
in cases like of early termination, e.g., if the tool could not be found.
- Some improvements to tool-info modules.
- Several updates of JS libraries, but this should not bring user-visible changes.

- One tool-info module improved.

- One tool-info module improved.

- BenchExec is now available in a PPA for easy installation on Ubuntu.
Just run the following commands

sudo add-apt-repository ppa:sosy-lab/benchmarking
sudo apt install benchexec

- Column filters are now reflected in the URL of HTML tables.
This makes it possible to open a table, configure some filters,
and share a link with others that will apply these filters on load.
Furthermore, using the back and forward buttons of the browser
will now also update the applied filters.
- Add parameter `--initial-table-state` to `table-generator`,
which allows to define the default state of HTML tables
(e.g., filters, opened tab, etc.).
- Category-specific statistics are shown more often again on first table tab.
Since BenchExec 3.0 these were removed in some cases where we cannot compute
them, but this accidentally removed them from more than the desired cases.
- Improved rounding in table-generator.
- SV-COMP scoring schema updated according to rules of SV-COMP'21.
- Many tool-info modules updated to use the new API from BenchExec 3.3
and improvements for SV-COMP'21 and Test-Comp'21.
- Improved warnings in certain cases where a benchmark definition
does not make sense (e.g., `<exclude>` tags that do not match anything).
- HTML tables now show a proper error message if the browser is not supported
and also a loading message.
- Several smaller bug fixes like avoiding crashes in corner cases.

- New API for tool-info modules (needed by `benchexec` for getting information
about the benchmarked tool). The new API is defined by class
[`benchexec.tools.template.BaseTool2`](https://github.com/sosy-lab/benchexec/blob/master/benchexec/tools/template.py)
and is similar to the old API, but more convenient to use and provides more
useful information to the tool-info module.
The old API is still supported
and will be removed no sooner than in BenchExec 4.0. We also provide a
[migration guide](https://github.com/sosy-lab/benchexec/blob/master/doc/tool-integration.md#migrating-tool-info-modules-to-new-api).
- A new parameter `--tool-directory` for `benchexec` allows to specify
the installation directory of the benchmarked tool easily
without having to modify `PATH` or change into the tool's directory.
Note that this only works if the respective tool-info module
makes use of the new `BaseTool2` API.
- New version 2.0 of the
[task-definition format](https://github.com/sosy-lab/benchexec/blob/master/doc/benchexec.md#task-definition-files)
for `benchexec`.
This format allows to specify arbitrary additional information in a key
named `options` and `benchexec` will pass everything in this key
to the tool-info module, but note that this only works
if the respective tool-info module makes use of the new `BaseTool2` API.
This is useful to add domain-specific information about tasks, for example
in the [SV-Benchmarks](https://github.com/sosy-lab/sv-benchmarks#task-definitions)
repository it is used to declare the program language.
BenchExec also still supports version 1.0 of the format.

- `table-generator` is now defined to work on Windows
and we test this in continuous integration.
Previously, it probably was working on Windows most of the time
but we did not systematically test this.
- Fix a crash in `benchexec` for task with property
but without task-definition file.