RFC QEMU validator: A method to specify QEMU crash-test cases

Rationale
---------

Today we have a few test cases in the QEMU tree that just run QEMU with a few command-line options, run some QMP or HMP commands, and check if QEMU didn't crash.

Some examples:
* scripts/device-crash-test
* The test case suggested by Thomas at:
Subject: [RFC PATCH] tests: Add a device_add/del HMP test
* Some test cases in tests/device-introspect-test.c
* tests/cpu-plug-test.c
* tests/display-vga-test.c
* tests/drive_del-test.c
* tests/machine-none-test.c
* tests/test-hmp.c
* Some device test skeletons: ac97-test.c, e1000-test.c, eepro100-test.c, es1370-test.c, i82801b11-test.c, intel-hda-test.c, ioh3420-test.c, ipoctal232-test.c, ne2000-test.c, nvme-test.c, pcnet-test.c, spapr-phb-test.c, test-netfilter.c, tpci200-test.c, usb-hcd-ohci-test.c, usb-hcd-xhci-test.c, virtio-balloon-test.c, virtio-console-test.c, virtio-serial-test.c, vmxnet3-test.c.

All of those test cases require lots of boilerplate code just to run some combinations of QEMU command-line options and monitor commands.

In addition to that, writing and reviewing test code like the examples above is more complex than it could be. Some past examples from qemu-devel:
* Subject: [RFC 5/6] device-crash-test: Basic device_add support
* Subject: [RFC 6/6] device-crash-test: Multi-device device_add test
* Subject: [RFC PATCH] tests/device-introspect: Test devices with all machines, not only with "none"

The Proposal
------------

This series introduces the scripts/validator.py test runner, and a set of example test specifications in tests/validator.

Each test specification is a YAML file containing the QEMU command-line and monitor commands to run, and can contain variables like $MACHINE, $DEVICE, $CPU, that can be expanded automatically by the test runner.

This way, scripts/device-crash-test can be replaced by a one-line test specification:

command-line: '$QEMU -S -machine $MACHINE,accel=$ACCEL -device $DEVICE'

The device_add/device_del test case written by Thomas can be specified as:

command-line: '$QEMU -S -machine $MACHINE,accel=$ACCEL' monitor-commands:
- qmp:
- execute: 'device_add'
arguments:
driver: '$DEVICE'
id: 'test-device-for-$DEVICE'
- execute: 'device_del'
arguments:
id: 'test-device-for-$DEVICE'

and the test runner will take care of running test cases with with all valid machine-types, device-types, and accelerators.

For other examples, see the last patch in this series ("Collection of validator.py test cases"). Most of the examples replace existing C or Python test cases with simple declarative test specifications.

I expect this to be very useful when somebody is fixing a QEMU crash and just want to add a very simple reproducer to ensure the crash will never be introduced again.

(I actually suggest we _require_ people to include a tests/validator test specifications every time they fix a crash that can be reproduced using just QMP commands).

Features
--------

Current features:
* Automatic variable expansion, as described above
* A simple method to specify defaults (e.g. to test only "-machine none" or only "accel=kvm:tcg" unless running in --full mode)
* A simple method to represent expected failures (it can't replace the device-crash-test whitelist mechanism, yet)

Caveats
-------

The test runner code still needs some clean up. I'm not happy with the current variable enumeration/expansion code, and plan to make it cleaner.

How to test
-----------

You can run all example test cases on /usr/bin/qemu-system-x86_64 with:

  $ ./scripts/validator.py tests/validator/* -V QEMU=/usr/bin/qemu-system-x86_64

However, the device-crash-test.yaml test specification still takes too long to run and reports too many false positives. So I suggest doing this instead:

  $ ./scripts/validator.py $(ls tests/validator/* | grep -v device-crash-test) \
        -V QEMU=/usr/bin/qemu-system-x86_64

On my system, this runs ~700 test cases in ~15 seconds.

What's out of scope?
--------------------

The following are _not_ goals of this system:
* Replacing the work done by the Avocado team to support Avocado-based tests.
* Making the YAML test specification a fully-featured imperative language. Anything that requires extra logic still needs to be written in C or Python.

Future Plans
------------

Short term:
* Choose a better name for the test runner
* Support a list of known-failures similar to the whitelist in device-crash-test.
* Replace device-crash-test completely
* "make check" rules

Long term:
* stdout and QMP command output validation.
I plan to use this to write machine-type/guest-ABI compatibility test cases that will detect guest ABI breakage as soon as possible.
* See if the Avocado multiplexer API can be used to implement variable expansion.
* See if we can make some of the features (e.g. automatic variable expansion) available to Python test cases
* Fuzzing QMP commands and QEMU command-line arguments.
* See if code can be shared with the iotests test runner (especially the variable expansion logic)

Eduardo Habkost (18):
qmp.py: Make it safe to call close() any time qmp.py: Fix error handling for Python 3 qmp.py: Cleanly handle unexpectedly closed socket qemu.py: Make _vm_monitor a method
qemu.py: Split _base_args()
qemu.py: Move _load_io_log() call to _post_shutdown() qemu.py: Use wait() logic inside shutdown() qemu.py: Close _qmp inside _post_shutdown() qemu.py: Make monitor optional
qemu.py: Set _launched = False on _post_shutdown qemu.py: Log crashes inside _post_shutdown() qemu.py: Only wait for process if it's still running qemu.py: 'force' parameter on shutdown() qemu.py: Don't try to quit cleanly on exceptions qemu.py: qmp_obj() method
qemu.py: is_launched() method
validator.py script
Collection of validator.py test cases

 scripts/qemu.py                        | 112 +++--
 scripts/qmp/qmp.py                     |  16 +-
 scripts/validator.py                   | 724 +++++++++++++++++++++++++++++++++
 tests/validator/cpu.yaml               |   5 +
 tests/validator/device-add.yaml        |  17 +
 tests/validator/device-crash-test.yaml |   7 +
 tests/validator/device-help.yaml       |   3 +
 tests/validator/device-introspect.yaml |  23 ++
 tests/validator/device-list.yaml       |  10 +
 tests/validator/just-help.yaml         |   3 +
 10 files changed, 876 insertions(+), 44 deletions(-)
 create mode 100755 scripts/validator.py
 create mode 100644 tests/validator/cpu.yaml
 create mode 100644 tests/validator/device-add.yaml
 create mode 100644 tests/validator/device-crash-test.yaml
 create mode 100644 tests/validator/device-help.yaml
 create mode 100644 tests/validator/device-introspect.yaml
 create mode 100644 tests/validator/device-list.yaml
 create mode 100644 tests/validator/just-help.yaml

2.14.3