Avocado-framework

Users/Test Writers
==================

* ``INSTRUMENTED`` tests using the
:func:`avocado.core.test.Test.fetch_asset` can take advantage of
plugins that will attempt to download (and cache) assets before the
test execution. This should make the overall test execution more
reliable, and give better test execution times as the download
time will be excluded. Users can also manually execute the
``avocado assets`` command to manually fetch assets from tests.

* The still experimental "N(ext) Runner" support for Avocado
Instrumented tests is more complete and supports tag filtering and
passing tags to the tests.

* A new architecture for "finding" tests has been introduced as an
alternative to the :mod:`avocado.core.loader` code. It's based
around the :mod:`avocado.core.resolver`, and it's currently used in
the still experimental "N(ext) Runner". It currently supports tests
of the following types: ``avocado-instrumented``, ``exec-test``,
``glib``, ``golang``, ``python-unittest`` and ``robot``. You can
experiment it by running ``avocado nlist``, similarly to how ``avocado
list`` is used.

* Avocado ``sysinfo`` feature file will now work out of the box on
``pip`` based installations. Previously, it would require
configuration files tweaks to adjust installation paths.

* A massive documentation overhaul, now designed around guides to
different target audiences. The "User's Guide", "Test Writer's
Guide" and "Contributor's Guide" can be easily found as first lever
sections contain curated content for those audiences.

Bug Fixes
=========

* Content supposed to be UI only could leak into TAP files, making
them invalid.

* Avocado's ``sysinfo`` feature will now run commands without a
shell, resulting in more proper captured output, without shell
related content.

* :func:`avocado.utils.process.SubProcess.send_signal` will now send a
signal to itself correctly even when using ``sudo`` mode.

Utility APIs
============

* The :mod:`avocado.utils.vmimage` library now allows a user to
define the ``qemu-img`` binary that will be used for creating
snapshot images via the :data:`avocado.utils.vmimage.QEMU_IMG`
variable.

* The :mod:`avocado.utils.configure_network` module introduced
a number of utilities, including MTU configuration support,
a method for validating network among peers, IPv6 support, etc.

* The :func:`avocado.utils.configure_network.set_ip` function now
supports different interface types through a ``interface_type``
parameter, while still defaulting to ``Ethernet``.

Internal Changes
================

* Package support for Enterprise Linux 8.

* Increased CI coverage, having tests now run on four different
hardware architectures: amd64 (x86_64), arm64 (aarch64), ppc64le and
s390x.

* Packit support adding extended CI coverage, with RPM packages being
built for Pull Requests and results shown on GitHub.

* Pylint checks for w0703 were enabled.

* Runners, such as the remote runner, vm runner, docker runner, and
the default local runner now conform to a "runner" interface and
can be seen as proper plugins with ``avocado plugins``.

* Avocado's configuration parser will now treat values with relative
paths as a special value, and evaluate their content in relation
to the Python's distribution directory where Avocado is installed.

For more information, please check out the complete
`Avocado changelog
<https://github.com/avocado-framework/avocado/compare/72.0...73.0>`_.

Users/Test Writers
==================

* Avocado can now run on systems with nothing but Python 3 (and
"quasi-standard-library" module ``setuptools``). This means that it
won't require extra packages, and should be easier to deploy on
containers, embedded systems, etc. Optional plugins may have
additional requirements.

* A new and still experimental test runner implementation, known as
"N(ext) Runner" has been introduced. It brings a number of
different concepts, increasing the decoupling between a test (and
its runner) and the job. For more information, please refer to
:ref:`the early documentation <nrunner>`.

* The new :func:`avocado.cancel_on` decorator has been added to the
Test APIs, allowing you to define the conditions for a test to be
considered canceled. See one example :ref:`here
<turning_errors_into_cancels>`.

* The ``glib`` plugin got a configuration option its safe/unsafe
operation, that is, whether it will execute binaries in an attempt to
find the whole list of tests. Look for the ``glib.conf`` shipped
with the plugin to enable the unsafe mode.

* Avocado can now use tags inside Python Unittests, and not only on
its own Instrumented tests. It's expected that other forms or
providing tags for other types of tests will also be introduced in
the near future.

* The HTML report will now show, as a handy pop-up, the contents of the
test whiteboard. If you set, say, performance metrics there, you'll
able to see straight from the report.

* The HTML report now has filtering support by test status, and can
show all records in the table.

* The :mod:`avocado.utils.runtime` module, a badly designed mechanism
for sharing Avocado runtime settings with the utility libraries, has
been removed.

* The test runner feature that would allow binaries to be run
transparently inside GDB was removed. The reason for dropping such
a feature have to do with how it limits the test runner to run one
test at a time, and the use of the :mod:`avocado.utils.runtime`
mechanism, also removed.

* Initial examples for writing custom jobs, using the so called Job
API, have been added to ``examples/jobs``. These APIs are still
non-public (under core), but they're supposed to become public
and supported soon.

* By means of a new plugin (``merge_files``, of type ``job.prepost``),
when using the :ref:`output check record <output_check_record>`
features, duplicate files created by different tests/variants will
be consolidated into unique files.

Bug Fixes
=========

* The HTML plugin now correctly shows the date for tests that were
never executed because of interrupted jobs.

* A temporarily workaround for a stack overflow problem in Python 3.7
has been addressed.

* The pict plugin (a varianter implementaion) now properly yields the
variants paths as a list.

* A Python 3 related fix to mod:`avocado.utils.software_manager`, that
was using Python 2 ``next`` on ``get_source``.

* A Python 3 related fix to the docker plugin, that wasn't caught
earlier.

Utility APIs
============

* :mod:`avocado.utils.partition` now allows ``mkfs`` and ``mount``
flags to be set.

* :func:`avocado.utils.cpu.get_cpu_vendor_name` now returns the CPU
vendor name for POWER9.

* :mod:`avocado.utils.asset` now allows a given location, as well as a
list, to be given, simplifying the most common use case.

* :meth:`avocado.utils.process.SubProcess.stop` now supports setting
a timeout. Please refer to the documentation for the important
details on its behavior.

* :mod:`avocado.utils.memory` now properly handles hugepages for POWER
platform.

Internal Changes
================

* Removal of the ``stevedore`` library dependency (previously used for
the dispatcher/plugins infrastructure).

* ``make check`` now runs selftests using the experimental N(ext)
Runner.

* Formal support for Python 3.7, which is now on our CI checks,
documentation and module information.

* The Yaml to Mux plugin now uses a safe version of the Yaml loader,
so that the execution of arbitrary Python code from Yaml input is
now no longer possible.

* Codecov coverage reports for have been enabled for Avocado, and
can be seen on every pull request.

* New tests have been added to many of the optional plugins.

* Various pylint compliance improvements, including w0231, w0235,
w0706, w0715 and w0221.

* Avocado's selftests now use tempfile.TemporaryDirectory instead of
mkdtemp and shutil.rmtree.

* :class:`avocado.core.job.Job` instantiation now takes a ``config``
dictionary parameter, instead of a ``argparse.Namespace`` instance,
and keeps it in a ``config`` attribute.

* :class:`avocado.core.job.Job` instances don't have a ``references``
attribute anymore. That information is available in the ``config``
attribute, that is, ``myjob.config['references']``.

* Basic checks for Fedora and RHEL 8 using Cirrus CI have been added,
and will be shown on every pull request.

For more information, please check out the complete
`Avocado changelog
<https://github.com/avocado-framework/avocado/compare/70.0...71.0>`_.

The major changes introduced on this version are listed below.
- Support for running tests in Docker container. Now, in addition to
running tests on a (libvirt based) Virtual Machine or on a remote host,
you can now run tests in transient Docker containers. The usage is as
simple as::

$ avocado run mytests.py --docker ldoktor/fedora-avocado

The container will be started, using `ldoktor/fedora-avocado` as
the image. This image contains a Fedora based system with Avocado
already installed, and it's provided at the official Docker hub.
- Introduction of the "Fail Fast" feature.

By running a job with the `--failfast` flag, the job will be
interrupted after the very first test failure. If your job only
makes sense if it's a complete PASS, this feature can save you a lot
of time.
- Avocado supports replaying previous jobs, selected by using their
Job IDs. Now, it's also possible to use the special keyword
`latest`, which will cause Avocado to rerun the very last job.
- Python's standard signal handling is restored for SIGPIPE, and thus
for all tests running on Avocado.

In previous releases, Avocado introduced a change that set the
default handler to SIGPIPE, which caused the application to be
terminated. This seemed to be the right approach when testing how
the Avocado app would behave on broken pipes on the command line,
but it introduced side effects to a lot of Python code. Instead of
exceptions, the affected Python code would receive the signal themselves.

This is now reverted to the Python standard, and the signal behavior
of Python based tests running on Avocado should not surprise anyone.
- The project release notes are now part of the official
documentation. That means that users can quickly find when a given
change was introduced.

Together with those changes listed, a total of 38 changes made into
this release. For more information, please check out the complete
`Avocado changelog
<https://github.com/avocado-framework/avocado/compare/38.0...39.0>`_.

Sprint Theme

After so much love that we had on the previous version, let's twist
things a bit with an antagonist title. Info on this pretty good movie
by Tarantino can be found at:

http://www.imdb.com/title/tt3460252/?ref_=nm_flmg_wr_2

https://www.youtube.com/watch?v=6_UI1GzaWv0

The story line:

In the dead of a Wyoming winter, a bounty hunter and his prisoner
find shelter in a cabin currently inhabited by a collection of
nefarious characters.

Release Meeting

The Avocado release meetings are now open to the community via
Hangouts on Air. The meetings are recorded and made available on the
`Avocado Test Framework YouTube channel
<https://www.youtube.com/channel/UC-RVZ_HFTbEztDM7wNY4NfA>`_.

For this release, you can watch the meeting on `this link
<https://www.youtube.com/watch?v=GotEH7SmHSw>`_.

Another development cycle has just finished, and our community will
receive this new release containing a nice assortment of bug fixes and
new features.
- The download of assets in tests now allow for an expiration time.
This means that tests that need to download any kind of external
asset, say a tarball, can now automatically benefit from the
download cache, but can also keep receiving new versions
automatically.

Suppose your asset uses an asset named `myproject-daily.tar.bz2`,
and that your test runs 50 times a day. By setting the expire time
to `1d` (1 day), your test will benefit from cache on most runs, but
will still fetch the new version when the the 24 hours from the
first download have passed.

For more information, please check out the documentation on the
`expire` parameter to the `fetch_asset()` method[1].
- Environment variables can be propagated into tests running on remote
systems. It's a known fact that one way to influence application behavior,
including test, is to set environment variables. A command line such as::


$ MYAPP_DEBUG=1 avocado run myapp_test.py


Will work as expected on a local system. But Avocado also allows
running tests on remote machines, and up until now, it has been
lacking a way to propagate environment variables to the remote
system.

Now, you can use::


$ MYAPP_DEBUG=1 avocado run --env-keep MYAPP_DEBUG \
--remote-host test-machine myapp_test.py

- The plugin interfaces have been moved into the
`avocado.core.plugin_interfaces` module. This means that plugin
writers now have to import the interface definitions this namespace,
example::


...
from avocado.core.plugin_interfaces import CLICmd

class MyCommand(CLICmd):
...


This is a way to keep ourselves honest, and say that there's no
difference from plugin interfaces to Avocado's core implementation,
that is, they may change at will. For greater stability, one should
be tracking the LTS releases.

Also, it effectively makes all plugins the same, whether they're
implemented and shipped as part of Avocado, or as part of external
projects.
- A contrib script for running kvm-unit-tests. As some people are
aware, Avocado has indeed a close relation to virtualization
testing. Avocado-VT is one obvious example, but there are other
virtualization related test suites can Avocado can run.

This release adds a contrib script that will fetch, download,
compile and run kvm-unit-tests using Avocado's external runner
feature. This gives results in a better granularity than the
support that exists in Avocado-VT, which gives only a single
PASS/FAIL for the entire test suite execution.

For more information, please check out the Avocado changelog[2].

Also, while we focused on Avocado, let's also not forget that
Avocado-VT maintains it's own fast pace of incoming niceties.
- s390 support: Avocado-VT is breaking into new grounds, and now has
support for the s390 architecture. Fedora 23 for s390 has been added
as a valid guest OS, and s390-virtio has been added as a new machine
type.
- Avocado-VT is now more resilient against failures to persist its
environment file, and will only give warnings instead of errors when
it fails to save it.
- An improved implementation of the "job lock" plugin, which prevents
multiple Avocado jobs with VT tests to run simultaneously. Since
there's no finer grained resource locking in Avocado-VT, this is a
global lock that will prevent issues such as image corruption when
two jobs are run at the same time.

This new implementation will now check if existing lock files are
stale, that is, they are leftovers from previous run. If the
processes associated with these files are not present, the stale
lock files are deleted, removing the need to clean them up manually.
It also outputs better debugging information when failures to
acquire lock.

The complete list of changes to Avocado-VT are available on its
changelog[3].

While not officially part of this release, this development cycle saw
the introduction of new tests on our avocado-misc-tests. Go check it
out!

Finally, since Avocado and Avocado-VT are not newly born anymore, we
decided to update information mentioning KVM-Autotest, virt-test on so
on around the web. This will hopefully redirect new users to the
Avocado community and avoid confusion.

Install avocado

Instructions are available in our documentation on how to install
either with packages or from source[4].

Updated RPM packages are be available in the project repos for EPEL 6,
EPEL 7, Fedora 22, Fedora 23 and the newly released Fedora 24.

Please note that on the next release, we'll drop support for Fedora 22
packages.

Happy hacking and testing!

---

[1] http://avocado-framework.readthedocs.io/en/38.0/WritingTests.html
[2] https://github.com/avocado-framework/avocado/compare/37.0...38.0
[3] https://github.com/avocado-framework/avocado-vt/compare/37.0...38.0
[4] http://avocado-framework.readthedocs.io/en/38.0/GetStartedGuide.html#installing-avocado

details on the specific RFC[9].

The main changes in Avocado for this release are:
- A big round of fixes and on machine readable output formats, such
as xunit (aka JUnit) and JSON. The xunit output, for instance,
now includes tests with schema checking. This should make sure
interoperability is even better on this release.
- Much more robust handling of test references, aka test URLs.
Avocado now properly handles very long test references, and also
test references with non-ascii characters.
- The avocado command line application now provides richer exit
status[1]. If your application or custom script depends on the
avocado exit status code, you should be fine as avocado still
returns zero for success and non-zero for errors. On error
conditions, though, the exit status code are richer and made of
combinable (ORable) codes. This way it's possible to detect that,
say, both a test failure and a job timeout occurred in a single
execution.
- [SECURITY RELATED] The remote execution of tests (including in
Virtual Machines) now allows for proper checks of host keys[2].
Without these checks, avocado is susceptible to a man-in-the-middle
attack, by connecting and sending credentials to the wrong machine.
This check is _disabled_ by default, because users depend on this
behavior when using machines without any prior knowledge such as
cloud based virtual machines. Also, a bug in the underlying SSH
library may prevent existing keys to be used if these are in ECDSA
format[3]. There's an automated check in place to check for the
resolution of the third party library bug. Expect this feature to
be _enabled_ by default in the upcoming releases.
- Pre/Post Job hooks. Avocado now defines a proper interface for
extension/plugin writers to execute actions while a Job is runnning.
Both Pre and Post hooks have access to the Job state (actually, the
complete Job instance). Pre job hooks are called before tests are
run, and post job hooks are called at the very end of the job (after
tests would have usually finished executing).
- Pre/Post job scripts[4]. As a feature built on top of the Pre/Post job
hooks described earlier, it's now possible to put executable scripts
in a configurable location, such as `/etc/avocado/scripts/job/pre.d`
and have them called by Avocado before the execution of tests. The
executed scripts will receive some information about the job via
environment variables[5].
- The implementation of proper Test-IDs[6] in the test result
directory.

Also, while not everything is (yet) translated into code, this release
saw various and major RFCs, which are definitely shaping the future of
Avocado. Among those:
- Introduce proper test IDs[6]
- Pre/Post _test_ hooks[7]
- Multi-stream tests[8]
- Avocado maintainability and integration with avocado-vt[9]
- Improvements to job status (completely implemented)[10]

For a complete list of changes please check the Avocado changelog[11].

For Avocado-VT, please check the full Avocado-VT changelog[12].

Install avocado

Instructions are available in our documentation on how to install
either with packages or from source[13].

Updated RPM packages are be available in the project repos for
Fedora 22, Fedora 23, EPEL 6 and EPEL 7.

Packages

As a heads up, we still package the latest version of the various
Avocado sub projects, such as the very popular Avocado-VT and the
pretty much experimental Avocado-Virt and Avocado-Server projects.

For the upcoming releases, there will be changes in our package
offers, with a greater focus on long term stability packages for
Avocado. Other packages may still be offered as a convenience, or
may see a change of ownership. All in the best interest of our users.
If you have any concerns or questions, please let us know.

Happy hacking and testing!

---

[1] http://avocado-framework.readthedocs.org/en/35.0/ResultFormats.html#exit-codes
[2] https://github.com/avocado-framework/avocado/blob/35.0/etc/avocado/avocado.conf#L41
[3] https://github.com/avocado-framework/avocado/blob/35.0/selftests/functional/test_thirdparty_bugs.py#L17
[4] http://avocado-framework.readthedocs.org/en/35.0/ReferenceGuide.html#job-pre-and-post-scripts
[5] http://avocado-framework.readthedocs.org/en/35.0/ReferenceGuide.html#script-execution-environment
[6] https://www.redhat.com/archives/avocado-devel/2016-March/msg00024.html
[7] https://www.redhat.com/archives/avocado-devel/2016-April/msg00000.html
[8] https://www.redhat.com/archives/avocado-devel/2016-April/msg00042.html
[9] https://www.redhat.com/archives/avocado-devel/2016-April/msg00038.html
[10] https://www.redhat.com/archives/avocado-devel/2016-April/msg00010.html
[11] https://github.com/avocado-framework/avocado/compare/0.34.0...35.0
[13] https://github.com/avocado-framework/avocado-vt/compare/0.34.0...35.0
[12] http://avocado-framework.readthedocs.org/en/35.0/GetStartedGuide.html#installing-avocado

This is another proud announcement: Avocado release 35.0, aka "Mr
Robot", is now out!

This release, while a "regular" release, will also serve as a beta for
our first "long term stability" (aka "lts") release. That means that
the next release, will be version "36.0lts" and will receive only bug
fixes and minor improvements. So, expect release 35.0 to be pretty
much like "36.0lts" feature-wise. New features will make into the