RFC Avocado-based functional tests

Background
==========

A few months ago, Cleber started a thread[1] about the system for Avocado-based tests. He invited people to take a look at it, we got some interesting feedback.

However, I saw no feedback on the actual code itself, and I don't know what needs to be done to get this merged.

To make sure we're having a discussion about the implementation of avocado_qemu and to figure out what's still necessary to get it included, I am sending all the commits from Amador's avocado_qemu branch[2] as a RFC.

Note that I didn't review most of this code yet, and I will probably send comments and questions as replies to this RFC later.

Also, many of the patches in this series probably can be squashed together, but I decided to not do that before getting some feedback on the changes.

[1] https://www.mail-archive.com/[email protected]/msg506859.html
[2] https://github.com/apahim/qemu/commits/avocado_qemu

README
======

Below, I'm copying the raw contents of tests/avocado/README.rst, for reference:

----------------------------------------------------------------------

========================================
QEMU tests using the Avocado Framework
========================================

This directory [tests/avocado] hosts functional tests written using Avocado Testing Framework.

Installation
============

To install Avocado and the dependencies needed for these tests, run::

    pip install --user avocado-framework avocado-framework-plugin-varianter-yaml-to-mux aexpect

Alternatively, follow the instructions on this link::

http://avocado-framework.readthedocs.io/en/latest/GetStartedGuide.html#installing-avocado

Overview
========

In this directory [tests/avocado], an ``avocado_qemu`` package is provided, containing the ``test`` module, which inherits from ``avocado.Test`` and provides a builtin and easy-to-use Qemu virtual machine. Here's a template that can be used as reference to start writing your own tests::

    from avocado_qemu import test

    class MyTest(test.QemuTest):
        """
        :avocado: enable
        """

        def setUp(self):
            self.vm.args.extend(['-m', '512'])
            self.vm.launch()

        def test_01(self):
            res = self.vm.qmp('human-monitor-command',
                              command_line='info version')
            self.assertIn('v2.9.0', res['return'])

        def tearDown(self):
            self.vm.shutdown()

To execute your test, run::

avocado run test_my_test.py

To execute all tests, run::

avocado run .

If you don't specify the Qemu binary to use, the ``avocado_qemu`` package will automatically probe it. The probe will try to use the Qemu binary from the git tree build directory, using the same architecture as the local system (if the architecture is not specified). If the Qemu binary is not available in the git tree build directory, the next try is to use the system installed Qemu binary.

You can define a number of optional parameters, providing them via YAML file using the Avocado parameters system:

- ``qemu_bin``: Use a given Qemu binary, skipping the automatic probe. Example: ``qemu_bin: /usr/libexec/qemu-kvm``.
- ``qemu_dst_bin``: Use a given Qemu binary to create the destination VM when the migration process takes place. If it's not provided, the same binary used in the source VM will be used for the destination VM.
Example: ``qemu_dst_bin: /usr/libexec/qemu-kvm-binary2``.
- ``arch``: Probe the Qemu binary from a given architecture. It has no effect if ``qemu_bin`` is specified. If not provided, the binary probe will use the system architecture. Example: ``arch: x86_64``
- ``image_path``: When a test requires (usually a bootable) image, this parameter is used to define where the image is located. When undefined it uses ``$QEMU_ROOT/bootable_image_$arch.qcow2``. The image is added to the qemu command __only__ when the test requires an image. By default ``,snapshot=on`` is used, but it can be altered by ``image_snapshot`` parameter.
- ``image_user`` and ``image_pass``: When using a ``image_path``, if you want to get the console from the Guest OS you have to define the Guest OS credentials. Example: ``image_user: avocado`` and ``image_pass: p4ssw0rd``. Both parameters have defaults to ``avocado``.
- ``machine_type``: Use this option to define a machine type for the VM.
Example: ``machine_type: pc``
- ``machine_accel``: Use this option to define a machine acceleration for the VM. Example: ``machine_accel: kvm``.
- ``machine_kvm_type``: Use this option to select the KVM type when the ``accel`` is ``kvm`` and there are more than one KVM types available.
Example: ``machine_kvm_type: PR``

Run the test with::

    $ avocado run test_my_test.py -m parameters.yaml

Additionally, you can use a variants file to to set different values for each parameter. Using the YAML tag ``!mux`` Avocado will execute the tests once per combination of parameters. Example::

    $ cat variants.yaml
    architecture: !mux
        x86_64:
            arch: x86_64
        i386:
            arch: i386

Run it the with::

    $ avocado run test_my_test.py -m variants.yaml

You can use both the parameters file and the variants file in the same command line::

    $ avocado run test_my_test.py -m parameters.yaml variants.yaml

Avocado will then merge the parameters from both files and create the proper variants.

See ``avocado run --help`` and ``man avocado`` for several other options, such as ``--filter-by-tags``, ``--show-job-log``, ``--failfast``, etc.

Uninstallation
==============

If you've followed the installation instructions above, you can easily uninstall Avocado. Start by listing the packages you have installed::

    pip list --user

And remove any package you want with::

pip uninstall <package_name>

Amador Pahim (12):
qemu.py: Introduce _create_console() method Introduce the basic framework to run Avocado tests
avocado_qemu: Fix exception name in caller
avocado_qemu: Improve migration error message
avocado_qemu: Functional test for RHBZ#1431939
avocado_qemu: Functional test for RHBZ#1447027
avocado_qemu: Functional test for RHBZ#1436616
avocado_qemu: Clean unneeded 'pass'
avocado_qemu: Set QMP log level to INFO
avocado_qemu: Introduce the add_image() VM API
avocado_qemu: Tests fixes
avocado_qemu: Force vmimage distro

Cleber Rosa (3):
avocado_qemu: Remove duplicate PortTracker implementation
avocado_qemu: Simplify the installation instructions
avocado_qemu: Add a few VNC related tests

Lukáš Doktor (9):
avocado_qemu: Improve handle_prompts to allow login after booted vm
avocado_qemu: Be lenient towards poluted serial console
avocado_qemu: Increase the login timeout to 60s
avocado_qemu: Add " " after the default prompt regexp
avocado_qemu: Store "arch" in VM
avocado_qemu: Provide defaults for user and pass
avocado_qemu: Ignore kernel messages on get_console
avocado_qemu: Add support to request image for testing
avocado_qemu: Functional test for RHBZ1473203

 scripts/qemu.py                                    |  59 ++-
 tests/avocado/README.rst                           | 132 +++++++
 tests/avocado/avocado_qemu/__init__.py             |   0
 tests/avocado/avocado_qemu/test.py                 | 418 +++++++++++++++++++++
 tests/avocado/parameters.yaml                      |  19 +
 tests/avocado/test_info_memdev_host_nodes.py       |  66 ++++
 tests/avocado/test_nec-usb-xhci.py                 |  63 ++++
 .../test_nec-usb-xhci.py.data/parameters.yaml      |   4 +
 tests/avocado/test_numa_hotplug.py                 | 120 ++++++
 tests/avocado/test_ovmf_with_240_vcpus.py          |  70 ++++
 .../parameters.yaml                                |   2 +
 tests/avocado/test_vnc.py                          |  58 +++
 tests/avocado/variants.yaml                        |  62 +++
 tests/qemu-iotests/iotests.py                      |  28 +-
 14 files changed, 1077 insertions(+), 24 deletions(-)
 create mode 100644 tests/avocado/README.rst
 create mode 100644 tests/avocado/avocado_qemu/__init__.py
 create mode 100644 tests/avocado/avocado_qemu/test.py
 create mode 100644 tests/avocado/parameters.yaml
 create mode 100644 tests/avocado/test_info_memdev_host_nodes.py
 create mode 100644 tests/avocado/test_nec-usb-xhci.py
 create mode 100644 tests/avocado/test_nec-usb-xhci.py.data/parameters.yaml
 create mode 100644 tests/avocado/test_numa_hotplug.py
 create mode 100644 tests/avocado/test_ovmf_with_240_vcpus.py
 create mode 100644 tests/avocado/test_ovmf_with_240_vcpus.py.data/parameters.yaml
 create mode 100644 tests/avocado/test_vnc.py
 create mode 100644 tests/avocado/variants.yaml

2.14.3