Skip to contentSkip to conversation

Acceptance tests: exclude "flaky" tests and introduce SPICE test

PATCH In review Cleber Rosa 3 participants 10 messages Jun 21, 2019 – Jul 5, 2019

lore
Share & tools
lore
#0 lore raw

cover Acceptance tests: exclude "flaky" tests and introduce SPICE test

These two seemingly unrelated topics have been added together in the same patch series because both deal with Avocado tags, and how their use is expand here to exclude tests.

For tests which sometimes can fail or hang, this adds the "flaky" tag, and doesn't run them as part of "make check-acceptance".

For the SPICE test, it depends on QEMU being configured with it, and if not, it will exclude it from the set of tests.

Cleber Rosa (2):
Acceptance tests: exclude "flaky" tests
Acceptance tests: add SPICE protocol check

 .travis.yml                              |  5 ++-
 docs/devel/testing.rst                   | 17 ++++++++
 tests/Makefile.include                   | 12 +++++-
 tests/acceptance/boot_linux_console.py   |  2 +
 tests/acceptance/linux_ssh_mips_malta.py |  2 +
 tests/acceptance/spice.py                | 54 ++++++++++++++++++++++++
 tests/requirements.txt                   |  2 +-
 7 files changed, 91 insertions(+), 3 deletions(-)
 create mode 100644 tests/acceptance/spice.py
Signature · 1 line
2.21.0
#1 lore raw

PATCH 1/2 Acceptance tests: exclude "flaky" tests

It's a fact that some tests may not be 100% reliable in all environments. While it's a tough call to remove a useful test that from the tree because it may fail every 1/100th time (or so), having human attention drawn to known issues is very bad for humans and for the projects they manage.

As a compromise solution, this marks tests that are known to have issues, or that exercises known issues in QEMU or other components, and excludes them from the entry point. As a consequence, tests marked as "flaky" will not be executed as part of "make check-acceptance".

Because such tests should be forgiven but never be forgotten, it's possible to list them with (assuming "make check-venv" or "make check-acceptance" has already initiatilized the venv):

  $ ./tests/venv/bin/avocado list -t flaky tests/acceptance

The current list of tests marked as flaky are a result of running the entire set of acceptance tests around 20 times. The results were then processed with a helper script[1]. That either confirmed known issues (in the case of aarch64 and arm)[2] or revealed new ones (mips).

This also bumps the Avocado version to one that includes a fix to the parsing of multiple and mix "key:val" and simple tag values.

[1] https://raw.githubusercontent.com/avocado-framework/avocado/master/contrib/scripts/summarize-job-failures.py
[2] https://bugs.launchpad.net/qemu/+bug/1829779

---
 docs/devel/testing.rst                   | 17 +++++++++++++++++
 tests/Makefile.include                   |  6 +++++-
 tests/acceptance/boot_linux_console.py   |  2 ++
 tests/acceptance/linux_ssh_mips_malta.py |  2 ++
 tests/requirements.txt                   |  2 +-
 5 files changed, 27 insertions(+), 2 deletions(-)
docs/devel/testing.rst +17 -0
index da2d0fc964..ff4d8e2e1c 100644
@@ -574,6 +574,23 @@ may be invoked by running:
 
   tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/
 
+Tagging tests
+-------------
+
+flaky
+~~~~~
+
+If a test is known to fail intermittently, even if only every one
+hundredth time, it's highly advisable to mark it as a flaky test.
+This will prevent these individual tests from failing much larger
+jobs, will avoid human interaction and time wasted to verify a known
+issue, and worse of all, can lead to the discredit of automated
+testing.
+
+To mark a test as flaky, add to its docstring.::
+
+  :avocado: tags=flaky
+
 Manual Installation
 -------------------
 
tests/Makefile.include +5 -1
index db750dd6d0..4c97da2878 100644
@@ -1125,7 +1125,11 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
 # Any number of command separated loggers are accepted.  For more
 # information please refer to "avocado --help".
 AVOCADO_SHOW=app
-AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGET_DIRS)))
+
+# Additional tags that are added to each occurence of "--filter-by-tags"
+AVOCADO_EXTRA_TAGS := ,-flaky
+
+AVOCADO_TAGS=$(patsubst %-softmmu,--filter-by-tags=arch:%$(AVOCADO_EXTRA_TAGS), $(filter %-softmmu,$(TARGET_DIRS)))
 
 ifneq ($(findstring v2,"v$(PYTHON_VERSION)"),v2)
 $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
tests/acceptance/boot_linux_console.py +2 -0
index 32159503e9..6bd5c1ab53 100644
@@ -249,6 +249,7 @@ class BootLinuxConsole(Test):
         """
         :avocado: tags=arch:aarch64
         :avocado: tags=machine:virt
+        :avocado: tags=flaky
         """
         kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/'
                       'releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz')
@@ -270,6 +271,7 @@ class BootLinuxConsole(Test):
         """
         :avocado: tags=arch:arm
         :avocado: tags=machine:virt
+        :avocado: tags=flaky
         """
         kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/'
                       'releases/29/Everything/armhfp/os/images/pxeboot/vmlinuz')
tests/acceptance/linux_ssh_mips_malta.py +2 -0
index aafb0c39f6..ae70b658e0 100644
@@ -208,6 +208,7 @@ class LinuxSSH(Test):
         :avocado: tags=machine:malta
         :avocado: tags=endian:big
         :avocado: tags=device:pcnet32
+        :avocado: tags=flaky
         """
         kernel_url = ('https://people.debian.org/~aurel32/qemu/mips/'
                       'vmlinux-3.2.0-4-5kc-malta')
@@ -222,6 +223,7 @@ class LinuxSSH(Test):
         :avocado: tags=machine:malta
         :avocado: tags=endian:little
         :avocado: tags=device:pcnet32
+        :avocado: tags=flaky
         """
         kernel_url = ('https://people.debian.org/~aurel32/qemu/mipsel/'
                       'vmlinux-3.2.0-4-5kc-malta')
tests/requirements.txt +1 -1
index 3ae0e29ad7..58d63d171f 100644
@@ -1,5 +1,5 @@
 # Add Python module requirements, one per line, to be installed
 # in the tests/venv Python virtual environment. For more info,
 # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
-avocado-framework==68.0
+avocado-framework==69.1
 paramiko
Signature · 1 line
2.21.0
#2 lore raw

Re: [Qemu-devel] [PATCH 1/2] Acceptance tests: exclude "flaky" tests

Quoted 53 lines · Cleber Rosa

It's a fact that some tests may not be 100% reliable in all environments. While it's a tough call to remove a useful test that from the tree because it may fail every 1/100th time (or so), having human attention drawn to known issues is very bad for humans and for the projects they manage.

As a compromise solution, this marks tests that are known to have issues, or that exercises known issues in QEMU or other components, and excludes them from the entry point. As a consequence, tests marked as "flaky" will not be executed as part of "make check-acceptance".

Because such tests should be forgiven but never be forgotten, it's possible to list them with (assuming "make check-venv" or "make check-acceptance" has already initiatilized the venv):

  $ ./tests/venv/bin/avocado list -t flaky tests/acceptance

The current list of tests marked as flaky are a result of running the entire set of acceptance tests around 20 times. The results were then processed with a helper script[1]. That either confirmed known issues (in the case of aarch64 and arm)[2] or revealed new ones (mips).

This also bumps the Avocado version to one that includes a fix to the parsing of multiple and mix "key:val" and simple tag values.

[1] https://raw.githubusercontent.com/avocado-framework/avocado/master/contrib/scripts/summarize-job-failures.py
[2] https://bugs.launchpad.net/qemu/+bug/1829779

---
 docs/devel/testing.rst                   | 17 +++++++++++++++++
 tests/Makefile.include                   |  6 +++++-
 tests/acceptance/boot_linux_console.py   |  2 ++
 tests/acceptance/linux_ssh_mips_malta.py |  2 ++
 tests/requirements.txt                   |  2 +-
 5 files changed, 27 insertions(+), 2 deletions(-)
docs/devel/testing.rst +6 -0
index da2d0fc964..ff4d8e2e1c 100644
@@ -574,6 +574,23 @@ may be invoked by running:
 
   tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/
 
+Tagging tests
+-------------
+
+flaky
+~~~~~
+

docs/devel/testing.rst may be invoked by running: · line 592

Quoted 10 lines
docs/devel/testing.rst +10 -0
+If a test is known to fail intermittently, even if only every one
+hundredth time, it's highly advisable to mark it as a flaky test.
+This will prevent these individual tests from failing much larger
+jobs, will avoid human interaction and time wasted to verify a known
+issue, and worse of all, can lead to the discredit of automated
+testing.
+
+To mark a test as flaky, add to its docstring.::
+
+  :avocado: tags=flaky

I certainly disagree with this patch, failing tests have to be fixed. Why not tag all the codebase flaky and sing "happy coding"?

Anyway if this get accepted, 'flaky' tags must have the intermittent failure well described, and a Launchpad/Bugzilla tracking ticket referenced.

Quoted 73 lines
docs/devel/testing.rst +1 -0
+
 Manual Installation
 -------------------
 
tests/Makefile.include +5 -1
index db750dd6d0..4c97da2878 100644
@@ -1125,7 +1125,11 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
 # Any number of command separated loggers are accepted.  For more
 # information please refer to "avocado --help".
 AVOCADO_SHOW=app
-AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGET_DIRS)))
+
+# Additional tags that are added to each occurence of "--filter-by-tags"
+AVOCADO_EXTRA_TAGS := ,-flaky
+
+AVOCADO_TAGS=$(patsubst %-softmmu,--filter-by-tags=arch:%$(AVOCADO_EXTRA_TAGS), $(filter %-softmmu,$(TARGET_DIRS)))
 
 ifneq ($(findstring v2,"v$(PYTHON_VERSION)"),v2)
 $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
tests/acceptance/boot_linux_console.py +2 -0
index 32159503e9..6bd5c1ab53 100644
@@ -249,6 +249,7 @@ class BootLinuxConsole(Test):
         """
         :avocado: tags=arch:aarch64
         :avocado: tags=machine:virt
+        :avocado: tags=flaky
         """
         kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/'
                       'releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz')
@@ -270,6 +271,7 @@ class BootLinuxConsole(Test):
         """
         :avocado: tags=arch:arm
         :avocado: tags=machine:virt
+        :avocado: tags=flaky
         """
         kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/'
                       'releases/29/Everything/armhfp/os/images/pxeboot/vmlinuz')
tests/acceptance/linux_ssh_mips_malta.py +2 -0
index aafb0c39f6..ae70b658e0 100644
@@ -208,6 +208,7 @@ class LinuxSSH(Test):
         :avocado: tags=machine:malta
         :avocado: tags=endian:big
         :avocado: tags=device:pcnet32
+        :avocado: tags=flaky
         """
         kernel_url = ('https://people.debian.org/~aurel32/qemu/mips/'
                       'vmlinux-3.2.0-4-5kc-malta')
@@ -222,6 +223,7 @@ class LinuxSSH(Test):
         :avocado: tags=machine:malta
         :avocado: tags=endian:little
         :avocado: tags=device:pcnet32
+        :avocado: tags=flaky
         """
         kernel_url = ('https://people.debian.org/~aurel32/qemu/mipsel/'
                       'vmlinux-3.2.0-4-5kc-malta')
tests/requirements.txt +1 -1
index 3ae0e29ad7..58d63d171f 100644
@@ -1,5 +1,5 @@
 # Add Python module requirements, one per line, to be installed
 # in the tests/venv Python virtual environment. For more info,
 # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
-avocado-framework==68.0
+avocado-framework==69.1
 paramiko
#3 lore raw

Re: [Qemu-devel] [PATCH 1/2] Acceptance tests: exclude "flaky" tests

Quoted 59 lines · Philippe Mathieu-Daudé
Quoted 58 lines · Cleber Rosa

It's a fact that some tests may not be 100% reliable in all environments. While it's a tough call to remove a useful test that from the tree because it may fail every 1/100th time (or so), having human attention drawn to known issues is very bad for humans and for the projects they manage.

As a compromise solution, this marks tests that are known to have issues, or that exercises known issues in QEMU or other components, and excludes them from the entry point. As a consequence, tests marked as "flaky" will not be executed as part of "make check-acceptance".

Because such tests should be forgiven but never be forgotten, it's possible to list them with (assuming "make check-venv" or "make check-acceptance" has already initiatilized the venv):

  $ ./tests/venv/bin/avocado list -t flaky tests/acceptance

The current list of tests marked as flaky are a result of running the entire set of acceptance tests around 20 times. The results were then processed with a helper script[1]. That either confirmed known issues (in the case of aarch64 and arm)[2] or revealed new ones (mips).

This also bumps the Avocado version to one that includes a fix to the parsing of multiple and mix "key:val" and simple tag values.

[1] https://raw.githubusercontent.com/avocado-framework/avocado/master/contrib/scripts/summarize-job-failures.py
[2] https://bugs.launchpad.net/qemu/+bug/1829779

---
 docs/devel/testing.rst                   | 17 +++++++++++++++++
 tests/Makefile.include                   |  6 +++++-
 tests/acceptance/boot_linux_console.py   |  2 ++
 tests/acceptance/linux_ssh_mips_malta.py |  2 ++
 tests/requirements.txt                   |  2 +-
 5 files changed, 27 insertions(+), 2 deletions(-)
docs/devel/testing.rst +11 -0
index da2d0fc964..ff4d8e2e1c 100644
@@ -574,6 +574,23 @@ may be invoked by running:
 
   tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/
 
+Tagging tests
+-------------
+
+flaky
+~~~~~
+
+If a test is known to fail intermittently, even if only every one
+hundredth time, it's highly advisable to mark it as a flaky test.
+This will prevent these individual tests from failing much larger
+jobs, will avoid human interaction and time wasted to verify a known
+issue, and worse of all, can lead to the discredit of automated
Quoted 9 lines

docs/devel/testing.rst may be invoked by running: · line 592

Quoted 5 lines
docs/devel/testing.rst +5 -0
+testing.
+
+To mark a test as flaky, add to its docstring.::
+
+  :avocado: tags=flaky

I certainly disagree with this patch, failing tests have to be fixed. Why not tag all the codebase flaky and sing "happy coding"?

That's a great idea! :)

Now, seriously, I also resisted this for quite a long time. The reality, though, is that intermittent failures will continue to appear, and letting tests (and jobs, and CI pipelines, and whatnot) fail is a very bad idea. We all agree that real fixes are better than this, but many times they don't come quickly.

Quoted 3 lines

Anyway if this get accepted, 'flaky' tags must have the intermittent failure well described, and a Launchpad/Bugzilla tracking ticket referenced.

And here you have a key point that I absolutely agree with. The "flaky" approach can either poison a lot of tests, and be seen as quick way out of a difficult issue revealed by a test. Or, it can serve as an effective tool to keep track of these very important issues.

If we add:

# https://bugs.launchpad.net/qemu/+bug/1829779
:avocado: flaky

Topped with some human, I believe this can be very effective. This goes without saying, but comments here are very much welcome.

- Cleber.

Quoted 74 lines
Quoted 73 lines
docs/devel/testing.rst +1 -0
+
 Manual Installation
 -------------------
 
tests/Makefile.include +5 -1
index db750dd6d0..4c97da2878 100644
@@ -1125,7 +1125,11 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
 # Any number of command separated loggers are accepted.  For more
 # information please refer to "avocado --help".
 AVOCADO_SHOW=app
-AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGET_DIRS)))
+
+# Additional tags that are added to each occurence of "--filter-by-tags"
+AVOCADO_EXTRA_TAGS := ,-flaky
+
+AVOCADO_TAGS=$(patsubst %-softmmu,--filter-by-tags=arch:%$(AVOCADO_EXTRA_TAGS), $(filter %-softmmu,$(TARGET_DIRS)))
 
 ifneq ($(findstring v2,"v$(PYTHON_VERSION)"),v2)
 $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
tests/acceptance/boot_linux_console.py +2 -0
index 32159503e9..6bd5c1ab53 100644
@@ -249,6 +249,7 @@ class BootLinuxConsole(Test):
         """
         :avocado: tags=arch:aarch64
         :avocado: tags=machine:virt
+        :avocado: tags=flaky
         """
         kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/'
                       'releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz')
@@ -270,6 +271,7 @@ class BootLinuxConsole(Test):
         """
         :avocado: tags=arch:arm
         :avocado: tags=machine:virt
+        :avocado: tags=flaky
         """
         kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/'
                       'releases/29/Everything/armhfp/os/images/pxeboot/vmlinuz')
tests/acceptance/linux_ssh_mips_malta.py +2 -0
index aafb0c39f6..ae70b658e0 100644
@@ -208,6 +208,7 @@ class LinuxSSH(Test):
         :avocado: tags=machine:malta
         :avocado: tags=endian:big
         :avocado: tags=device:pcnet32
+        :avocado: tags=flaky
         """
         kernel_url = ('https://people.debian.org/~aurel32/qemu/mips/'
                       'vmlinux-3.2.0-4-5kc-malta')
@@ -222,6 +223,7 @@ class LinuxSSH(Test):
         :avocado: tags=machine:malta
         :avocado: tags=endian:little
         :avocado: tags=device:pcnet32
+        :avocado: tags=flaky
         """
         kernel_url = ('https://people.debian.org/~aurel32/qemu/mipsel/'
                       'vmlinux-3.2.0-4-5kc-malta')
tests/requirements.txt +1 -1
index 3ae0e29ad7..58d63d171f 100644
@@ -1,5 +1,5 @@
 # Add Python module requirements, one per line, to be installed
 # in the tests/venv Python virtual environment. For more info,
 # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
-avocado-framework==68.0
+avocado-framework==69.1
 paramiko
#4 lore raw

Re: [Qemu-devel] [PATCH 1/2] Acceptance tests: exclude "flaky" tests

Quoted 14 lines · Cleber Rosa
Quoted 13 lines · Philippe Mathieu-Daudé
Quoted 12 lines · Cleber Rosa

It's a fact that some tests may not be 100% reliable in all environments. While it's a tough call to remove a useful test that from the tree because it may fail every 1/100th time (or so), having human attention drawn to known issues is very bad for humans and for the projects they manage.

As a compromise solution, this marks tests that are known to have issues, or that exercises known issues in QEMU or other components, and excludes them from the entry point. As a consequence, tests marked as "flaky" will not be executed as part of "make check-acceptance".

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Because such tests should be forgiven but never be forgotten, it's possible to list them with (assuming "make check-venv" or "make check-acceptance" has already initiatilized the venv):

   $ ./tests/venv/bin/avocado list -t flaky tests/acceptance

It needs a Make target to run those flaky tests (If we ever agree on this idea of flaky tests). Other Avocado flags are passed (e.g. -t for tags) that can happen to fail tests on their absent. One clear example is the spice test on patch 02 of this series...

Side note: check-acceptance seems to get growing in complexity that I worry will end up in pitfalls. is a Make target the proper way to implement complex test runs (I don't think so). Perhaps Avocado runner concept could help somehow?

Quoted 49 lines
Quoted 40 lines
Quoted 40 lines

The current list of tests marked as flaky are a result of running the entire set of acceptance tests around 20 times. The results were then processed with a helper script[1]. That either confirmed known issues (in the case of aarch64 and arm)[2] or revealed new ones (mips).

This also bumps the Avocado version to one that includes a fix to the parsing of multiple and mix "key:val" and simple tag values.

[1] https://raw.githubusercontent.com/avocado-framework/avocado/master/contrib/scripts/summarize-job-failures.py
[2] https://bugs.launchpad.net/qemu/+bug/1829779

---
  docs/devel/testing.rst                   | 17 +++++++++++++++++
  tests/Makefile.include                   |  6 +++++-
  tests/acceptance/boot_linux_console.py   |  2 ++
  tests/acceptance/linux_ssh_mips_malta.py |  2 ++
  tests/requirements.txt                   |  2 +-
  5 files changed, 27 insertions(+), 2 deletions(-)
docs/devel/testing.rst +10 -0
index da2d0fc964..ff4d8e2e1c 100644
@@ -574,6 +574,23 @@ may be invoked by running:
 
    tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/
 
+Tagging tests
+-------------
+
+flaky
+~~~~~
+
+If a test is known to fail intermittently, even if only every one
+hundredth time, it's highly advisable to mark it as a flaky test.
+This will prevent these individual tests from failing much larger
+jobs, will avoid human interaction and time wasted to verify a known
Quoted 9 lines

docs/devel/testing.rst may be invoked by running: · line 592

Quoted 6 lines
docs/devel/testing.rst +6 -0
+issue, and worse of all, can lead to the discredit of automated
+testing.
+
+To mark a test as flaky, add to its docstring.::
+
+  :avocado: tags=flaky

I certainly disagree with this patch, failing tests have to be fixed. Why not tag all the codebase flaky and sing "happy coding"?

Quoted 7 lines

That's a great idea! :)

Now, seriously, I also resisted this for quite a long time. The reality, though, is that intermittent failures will continue to appear, and letting tests (and jobs, and CI pipelines, and whatnot) fail is a very bad idea. We all agree that real fixes are better than this, but many times they don't come quickly.

It seems to me that flaky test is just a case in a broaden scenario: run (or not) grouped tests. You may have tests indeed broken or that takes considerable time (those tagged "slow") which one may fairly want to exclude from make check-acceptance as well. Thus some way to group tests plus define run inclusion/exclusion patterns seems the ultimate goal here.

Quoted 10 lines
Quoted 4 lines

Anyway if this get accepted, 'flaky' tags must have the intermittent failure well described, and a Launchpad/Bugzilla tracking ticket referenced.

And here you have a key point that I absolutely agree with. The "flaky" approach can either poison a lot of tests, and be seen as quick way out of a difficult issue revealed by a test. Or, it can serve as an effective tool to keep track of these very important issues.

Quoted 7 lines

If we add:

    # https://bugs.launchpad.net/qemu/+bug/1829779
    :avocado: flaky

Topped with some human, I believe this can be very effective. This goes without saying, but comments here are very much welcome.

I agree that all flaky test should have a tracking bug. In the end it represents a technical debit that we should address.

- Wainer

Quoted 76 lines

- Cleber.

Quoted 73 lines
Quoted 73 lines
docs/devel/testing.rst +1 -0
+
  Manual Installation
  -------------------
 
tests/Makefile.include +5 -1
index db750dd6d0..4c97da2878 100644
@@ -1125,7 +1125,11 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
  # Any number of command separated loggers are accepted.  For more
  # information please refer to "avocado --help".
  AVOCADO_SHOW=app
-AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGET_DIRS)))
+
+# Additional tags that are added to each occurence of "--filter-by-tags"
+AVOCADO_EXTRA_TAGS := ,-flaky
+
+AVOCADO_TAGS=$(patsubst %-softmmu,--filter-by-tags=arch:%$(AVOCADO_EXTRA_TAGS), $(filter %-softmmu,$(TARGET_DIRS)))
 
  ifneq ($(findstring v2,"v$(PYTHON_VERSION)"),v2)
  $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
tests/acceptance/boot_linux_console.py +2 -0
index 32159503e9..6bd5c1ab53 100644
@@ -249,6 +249,7 @@ class BootLinuxConsole(Test):
          """
          :avocado: tags=arch:aarch64
          :avocado: tags=machine:virt
+        :avocado: tags=flaky
          """
          kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/'
                        'releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz')
@@ -270,6 +271,7 @@ class BootLinuxConsole(Test):
          """
          :avocado: tags=arch:arm
          :avocado: tags=machine:virt
+        :avocado: tags=flaky
          """
          kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/'
                        'releases/29/Everything/armhfp/os/images/pxeboot/vmlinuz')
tests/acceptance/linux_ssh_mips_malta.py +2 -0
index aafb0c39f6..ae70b658e0 100644
@@ -208,6 +208,7 @@ class LinuxSSH(Test):
          :avocado: tags=machine:malta
          :avocado: tags=endian:big
          :avocado: tags=device:pcnet32
+        :avocado: tags=flaky
          """
          kernel_url = ('https://people.debian.org/~aurel32/qemu/mips/'
                        'vmlinux-3.2.0-4-5kc-malta')
@@ -222,6 +223,7 @@ class LinuxSSH(Test):
          :avocado: tags=machine:malta
          :avocado: tags=endian:little
          :avocado: tags=device:pcnet32
+        :avocado: tags=flaky
          """
          kernel_url = ('https://people.debian.org/~aurel32/qemu/mipsel/'
                        'vmlinux-3.2.0-4-5kc-malta')
tests/requirements.txt +1 -1
index 3ae0e29ad7..58d63d171f 100644
@@ -1,5 +1,5 @@
  # Add Python module requirements, one per line, to be installed
  # in the tests/venv Python virtual environment. For more info,
  # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
-avocado-framework==68.0
+avocado-framework==69.1
  paramiko
#5 lore raw

Re: [Qemu-devel] [PATCH 1/2] Acceptance tests: exclude "flaky" tests

Quoted 20 lines · Wainer dos Santos Moschetta
Quoted 18 lines · Cleber Rosa
Quoted 17 lines · Philippe Mathieu-Daudé
Quoted 16 lines · Cleber Rosa

It's a fact that some tests may not be 100% reliable in all environments. While it's a tough call to remove a useful test that from the tree because it may fail every 1/100th time (or so), having human attention drawn to known issues is very bad for humans and for the projects they manage.

As a compromise solution, this marks tests that are known to have issues, or that exercises known issues in QEMU or other components, and excludes them from the entry point. As a consequence, tests marked as "flaky" will not be executed as part of "make check-acceptance".

Because such tests should be forgiven but never be forgotten, it's possible to list them with (assuming "make check-venv" or "make check-acceptance" has already initiatilized the venv):

Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
   $ ./tests/venv/bin/avocado list -t flaky tests/acceptance

It needs a Make target to run those flaky tests (If we ever agree on this idea of flaky tests). Other Avocado flags are passed (e.g. -t for tags) that can happen to fail tests on their absent. One clear example is the spice test on patch 02 of this series...

I was trying to avoid having so make "check-acceptance-*" rules that just choosing one would be harder than writing an Avocado command line from scratch... but I think you have a point here. For once, this can be used in a Travis job with an special "allow_failures" option set.

Quoted 5 lines

Side note: check-acceptance seems to get growing in complexity that I worry will end up in pitfalls. is a Make target the proper way to implement complex test runs (I don't think so). Perhaps Avocado runner concept could help somehow?

I guess you mean the Avocado Job concept, and writing your own runner based on those APIs. If so, then absolutely yes. I've shared with Eduardo some of the use cases that we can solve much easily. But, we need to finish the last bits on the Avocado side, properly document and support the API before attempting to use it here on QEMU.

Quoted 55 lines
Quoted 47 lines
Quoted 38 lines
Quoted 38 lines

The current list of tests marked as flaky are a result of running the entire set of acceptance tests around 20 times. The results were then processed with a helper script[1]. That either confirmed known issues (in the case of aarch64 and arm)[2] or revealed new ones (mips).

This also bumps the Avocado version to one that includes a fix to the parsing of multiple and mix "key:val" and simple tag values.

[1] https://raw.githubusercontent.com/avocado-framework/avocado/master/contrib/scripts/summarize-job-failures.py
[2] https://bugs.launchpad.net/qemu/+bug/1829779

---
  docs/devel/testing.rst                   | 17 +++++++++++++++++
  tests/Makefile.include                   |  6 +++++-
  tests/acceptance/boot_linux_console.py   |  2 ++
  tests/acceptance/linux_ssh_mips_malta.py |  2 ++
  tests/requirements.txt                   |  2 +-
  5 files changed, 27 insertions(+), 2 deletions(-)
docs/devel/testing.rst +10 -0
index da2d0fc964..ff4d8e2e1c 100644
@@ -574,6 +574,23 @@ may be invoked by running:
    tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/
+Tagging tests
+-------------
+
+flaky
+~~~~~
+
+If a test is known to fail intermittently, even if only every one
+hundredth time, it's highly advisable to mark it as a flaky test.
+This will prevent these individual tests from failing much larger
+jobs, will avoid human interaction and time wasted to verify a known
Quoted 9 lines

docs/devel/testing.rst may be invoked by running: · line 592

Quoted 6 lines
docs/devel/testing.rst +6 -0
+issue, and worse of all, can lead to the discredit of automated
+testing.
+
+To mark a test as flaky, add to its docstring.::
+
+  :avocado: tags=flaky

I certainly disagree with this patch, failing tests have to be fixed. Why not tag all the codebase flaky and sing "happy coding"?

Quoted 7 lines

That's a great idea! :)

Now, seriously, I also resisted this for quite a long time. The reality, though, is that intermittent failures will continue to appear, and letting tests (and jobs, and CI pipelines, and whatnot) fail is a very bad idea. We all agree that real fixes are better than this, but many times they don't come quickly.

Quoted 6 lines

It seems to me that flaky test is just a case in a broaden scenario: run (or not) grouped tests. You may have tests indeed broken or that takes considerable time (those tagged "slow") which one may fairly want to exclude from make check-acceptance as well. Thus some way to group tests plus define run inclusion/exclusion patterns seems the ultimate goal here.

Yes, you have a point about "yet another set of tests". I think that, whenever we break the limit of expressiveness with something like the current incarnation of tags (which shouldn't need much explanation) then it's time to rely on something that has all the expressiveness and doesn't impose any other restrictions.

I'm refering to the previous idea about using the Job API, and creating customized runner that, with the expressiveness of Python code, can choose tests for different scenarios. What I don't think we should try to do, is to come up with yet another language, or YAML parser, or anything along those lines.

Quoted 15 lines
Quoted 15 lines
Quoted 4 lines

Anyway if this get accepted, 'flaky' tags must have the intermittent failure well described, and a Launchpad/Bugzilla tracking ticket referenced.

And here you have a key point that I absolutely agree with. The "flaky" approach can either poison a lot of tests, and be seen as quick way out of a difficult issue revealed by a test. Or, it can serve as an effective tool to keep track of these very important issues.

If we add:

    # https://bugs.launchpad.net/qemu/+bug/1829779
    :avocado: flaky
Quoted 8 lines
Quoted 2 lines

Topped with some human, I believe this can be very effective. This goes without saying, but comments here are very much welcome.

I agree that all flaky test should have a tracking bug. In the end it represents a technical debit that we should address.

- Wainer

Yep, that I also agree 100%.
- Cleber.

Quoted 75 lines
Quoted 74 lines

- Cleber.

Quoted 71 lines
Quoted 71 lines
docs/devel/testing.rst +1 -0
+
  Manual Installation
  -------------------
tests/Makefile.include +5 -1
index db750dd6d0..4c97da2878 100644
@@ -1125,7 +1125,11 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
  # Any number of command separated loggers are accepted.  For more
  # information please refer to "avocado --help".
  AVOCADO_SHOW=app
-AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGET_DIRS)))
+
+# Additional tags that are added to each occurence of "--filter-by-tags"
+AVOCADO_EXTRA_TAGS := ,-flaky
+
+AVOCADO_TAGS=$(patsubst %-softmmu,--filter-by-tags=arch:%$(AVOCADO_EXTRA_TAGS), $(filter %-softmmu,$(TARGET_DIRS)))
  ifneq ($(findstring v2,"v$(PYTHON_VERSION)"),v2)
  $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
tests/acceptance/boot_linux_console.py +2 -0
index 32159503e9..6bd5c1ab53 100644
@@ -249,6 +249,7 @@ class BootLinuxConsole(Test):
          """
          :avocado: tags=arch:aarch64
          :avocado: tags=machine:virt
+        :avocado: tags=flaky
          """
          kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/'
                        'releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz')
@@ -270,6 +271,7 @@ class BootLinuxConsole(Test):
          """
          :avocado: tags=arch:arm
          :avocado: tags=machine:virt
+        :avocado: tags=flaky
          """
          kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/'
                        'releases/29/Everything/armhfp/os/images/pxeboot/vmlinuz')
tests/acceptance/linux_ssh_mips_malta.py +2 -0
index aafb0c39f6..ae70b658e0 100644
@@ -208,6 +208,7 @@ class LinuxSSH(Test):
          :avocado: tags=machine:malta
          :avocado: tags=endian:big
          :avocado: tags=device:pcnet32
+        :avocado: tags=flaky
          """
          kernel_url = ('https://people.debian.org/~aurel32/qemu/mips/'
                        'vmlinux-3.2.0-4-5kc-malta')
@@ -222,6 +223,7 @@ class LinuxSSH(Test):
          :avocado: tags=machine:malta
          :avocado: tags=endian:little
          :avocado: tags=device:pcnet32
+        :avocado: tags=flaky
          """
          kernel_url = ('https://people.debian.org/~aurel32/qemu/mipsel/'
                        'vmlinux-3.2.0-4-5kc-malta')
tests/requirements.txt +1 -1
index 3ae0e29ad7..58d63d171f 100644
@@ -1,5 +1,5 @@
  # Add Python module requirements, one per line, to be installed
  # in the tests/venv Python virtual environment. For more info,
  # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
-avocado-framework==68.0
+avocado-framework==69.1
  paramiko
#6 lore raw

Re: [Qemu-devel] [PATCH 1/2] Acceptance tests: exclude "flaky" tests

Quoted 26 lines · Cleber Rosa
Quoted 19 lines · Wainer dos Santos Moschetta
Quoted 18 lines · Cleber Rosa
Quoted 17 lines · Philippe Mathieu-Daudé
Quoted 16 lines · Cleber Rosa

It's a fact that some tests may not be 100% reliable in all environments. While it's a tough call to remove a useful test that from the tree because it may fail every 1/100th time (or so), having human attention drawn to known issues is very bad for humans and for the projects they manage.

As a compromise solution, this marks tests that are known to have issues, or that exercises known issues in QEMU or other components, and excludes them from the entry point. As a consequence, tests marked as "flaky" will not be executed as part of "make check-acceptance".

Because such tests should be forgiven but never be forgotten, it's possible to list them with (assuming "make check-venv" or "make check-acceptance" has already initiatilized the venv):

Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
    $ ./tests/venv/bin/avocado list -t flaky tests/acceptance

It needs a Make target to run those flaky tests (If we ever agree on this idea of flaky tests). Other Avocado flags are passed (e.g. -t for tags) that can happen to fail tests on their absent. One clear example is the spice test on patch 02 of this series...

Quoted 4 lines

I was trying to avoid having so make "check-acceptance-*" rules that just choosing one would be harder than writing an Avocado command line from scratch... but I think you have a point here. For once, this can be used in a Travis job with an special "allow_failures" option set.

Checking if I understood: you are proposing to keep running the flaky on Travis which might get failing most of time (but Travis won't flag error). Until someone fix them and remove the "flaky" tag so putting them on the (virtual) "stable" tests group? If so, I am ok with this approach while we don't have a better solution in place.

Quoted 11 lines
Quoted 6 lines

Side note: check-acceptance seems to get growing in complexity that I worry will end up in pitfalls. is a Make target the proper way to implement complex test runs (I don't think so). Perhaps Avocado runner concept could help somehow?

I guess you mean the Avocado Job concept, and writing your own runner based on those APIs. If so, then absolutely yes. I've shared with Eduardo some of the use cases that we can solve much easily. But, we need to finish the last bits on the Avocado side, properly document and support the API before attempting to use it here on QEMU.

I just realized that test runner is another thing in Avocado's jargon:
https://avocado-framework.readthedocs.io/en/70.0/GetStartedGuide.html#running-tests-with-an-external-runner

Ok, I'm looking forward those improvements on Avocado API. :)

- Wainer

Quoted 169 lines
Quoted 54 lines
Quoted 47 lines
Quoted 38 lines
Quoted 38 lines

The current list of tests marked as flaky are a result of running the entire set of acceptance tests around 20 times. The results were then processed with a helper script[1]. That either confirmed known issues (in the case of aarch64 and arm)[2] or revealed new ones (mips).

This also bumps the Avocado version to one that includes a fix to the parsing of multiple and mix "key:val" and simple tag values.

[1] https://raw.githubusercontent.com/avocado-framework/avocado/master/contrib/scripts/summarize-job-failures.py
[2] https://bugs.launchpad.net/qemu/+bug/1829779

---
   docs/devel/testing.rst                   | 17 +++++++++++++++++
   tests/Makefile.include                   |  6 +++++-
   tests/acceptance/boot_linux_console.py   |  2 ++
   tests/acceptance/linux_ssh_mips_malta.py |  2 ++
   tests/requirements.txt                   |  2 +-
   5 files changed, 27 insertions(+), 2 deletions(-)
docs/devel/testing.rst +10 -0
index da2d0fc964..ff4d8e2e1c 100644
@@ -574,6 +574,23 @@ may be invoked by running:
     tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/
+Tagging tests
+-------------
+
+flaky
+~~~~~
+
+If a test is known to fail intermittently, even if only every one
+hundredth time, it's highly advisable to mark it as a flaky test.
+This will prevent these individual tests from failing much larger
+jobs, will avoid human interaction and time wasted to verify a known
Quoted 9 lines

docs/devel/testing.rst may be invoked by running: · line 592

Quoted 6 lines
docs/devel/testing.rst +6 -0
+issue, and worse of all, can lead to the discredit of automated
+testing.
+
+To mark a test as flaky, add to its docstring.::
+
+  :avocado: tags=flaky

I certainly disagree with this patch, failing tests have to be fixed. Why not tag all the codebase flaky and sing "happy coding"?

Quoted 7 lines

That's a great idea! :)

Now, seriously, I also resisted this for quite a long time. The reality, though, is that intermittent failures will continue to appear, and letting tests (and jobs, and CI pipelines, and whatnot) fail is a very bad idea. We all agree that real fixes are better than this, but many times they don't come quickly.

Quoted 6 lines

It seems to me that flaky test is just a case in a broaden scenario: run (or not) grouped tests. You may have tests indeed broken or that takes considerable time (those tagged "slow") which one may fairly want to exclude from make check-acceptance as well. Thus some way to group tests plus define run inclusion/exclusion patterns seems the ultimate goal here.

Yes, you have a point about "yet another set of tests". I think that, whenever we break the limit of expressiveness with something like the current incarnation of tags (which shouldn't need much explanation) then it's time to rely on something that has all the expressiveness and doesn't impose any other restrictions.

I'm refering to the previous idea about using the Job API, and creating customized runner that, with the expressiveness of Python code, can choose tests for different scenarios. What I don't think we should try to do, is to come up with yet another language, or YAML parser, or anything along those lines.

Quoted 14 lines
Quoted 14 lines
Quoted 3 lines

Anyway if this get accepted, 'flaky' tags must have the intermittent failure well described, and a Launchpad/Bugzilla tracking ticket referenced.

And here you have a key point that I absolutely agree with. The "flaky" approach can either poison a lot of tests, and be seen as quick way out of a difficult issue revealed by a test. Or, it can serve as an effective tool to keep track of these very important issues.

If we add:

     # https://bugs.launchpad.net/qemu/+bug/1829779
     :avocado: flaky
Quoted 7 lines
Quoted 2 lines

Topped with some human, I believe this can be very effective. This goes without saying, but comments here are very much welcome.

I agree that all flaky test should have a tracking bug. In the end it represents a technical debit that we should address.

- Wainer

Yep, that I also agree 100%.
- Cleber.

Quoted 73 lines
Quoted 73 lines

- Cleber.

Quoted 71 lines
Quoted 71 lines
docs/devel/testing.rst +1 -0
+
   Manual Installation
   -------------------
tests/Makefile.include +5 -1
index db750dd6d0..4c97da2878 100644
@@ -1125,7 +1125,11 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
   # Any number of command separated loggers are accepted.  For more
   # information please refer to "avocado --help".
   AVOCADO_SHOW=app
-AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGET_DIRS)))
+
+# Additional tags that are added to each occurence of "--filter-by-tags"
+AVOCADO_EXTRA_TAGS := ,-flaky
+
+AVOCADO_TAGS=$(patsubst %-softmmu,--filter-by-tags=arch:%$(AVOCADO_EXTRA_TAGS), $(filter %-softmmu,$(TARGET_DIRS)))
   ifneq ($(findstring v2,"v$(PYTHON_VERSION)"),v2)
   $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
tests/acceptance/boot_linux_console.py +2 -0
index 32159503e9..6bd5c1ab53 100644
@@ -249,6 +249,7 @@ class BootLinuxConsole(Test):
           """
           :avocado: tags=arch:aarch64
           :avocado: tags=machine:virt
+        :avocado: tags=flaky
           """
           kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/'
                         'releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz')
@@ -270,6 +271,7 @@ class BootLinuxConsole(Test):
           """
           :avocado: tags=arch:arm
           :avocado: tags=machine:virt
+        :avocado: tags=flaky
           """
           kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/'
                         'releases/29/Everything/armhfp/os/images/pxeboot/vmlinuz')
tests/acceptance/linux_ssh_mips_malta.py +2 -0
index aafb0c39f6..ae70b658e0 100644
@@ -208,6 +208,7 @@ class LinuxSSH(Test):
           :avocado: tags=machine:malta
           :avocado: tags=endian:big
           :avocado: tags=device:pcnet32
+        :avocado: tags=flaky
           """
           kernel_url = ('https://people.debian.org/~aurel32/qemu/mips/'
                         'vmlinux-3.2.0-4-5kc-malta')
@@ -222,6 +223,7 @@ class LinuxSSH(Test):
           :avocado: tags=machine:malta
           :avocado: tags=endian:little
           :avocado: tags=device:pcnet32
+        :avocado: tags=flaky
           """
           kernel_url = ('https://people.debian.org/~aurel32/qemu/mipsel/'
                         'vmlinux-3.2.0-4-5kc-malta')
tests/requirements.txt +1 -1
index 3ae0e29ad7..58d63d171f 100644
@@ -1,5 +1,5 @@
   # Add Python module requirements, one per line, to be installed
   # in the tests/venv Python virtual environment. For more info,
   # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
-avocado-framework==68.0
+avocado-framework==69.1
   paramiko
#7 lore raw

PATCH 2/2 Acceptance tests: add SPICE protocol check

In reply to Cleber Rosa

This fires a QEMU binary with SPICE enabled, and does a basic handshake, doing a basic client/server interaction and protocol validation.

---
 .travis.yml               |  5 +++-
 tests/Makefile.include    |  6 +++++
 tests/acceptance/spice.py | 54 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 64 insertions(+), 1 deletion(-)
 create mode 100644 tests/acceptance/spice.py
.travis.yml +4 -1
index aeb9b211cd..6c9257a459 100644
@@ -231,7 +231,7 @@ matrix:
 
     # Acceptance (Functional) tests
     - env:
-        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu"
+        - CONFIG="--python=/usr/bin/python3 --enable-spice --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu"
         - TEST_CMD="make check-acceptance"
       after_failure:
         - cat tests/results/latest/job.log
@@ -240,6 +240,9 @@ matrix:
           packages:
             - python3-pip
             - python3.5-venv
+            - libspice-protocol-dev
+            - libspice-server-dev
+
     # Using newer GCC with sanitizers
     - addons:
         apt:
tests/Makefile.include +6 -0
index 4c97da2878..7fc2d28099 100644
@@ -1129,6 +1129,12 @@ AVOCADO_SHOW=app
 # Additional tags that are added to each occurence of "--filter-by-tags"
 AVOCADO_EXTRA_TAGS := ,-flaky
 
+# At last one test require spice to be enabled, allow it to be excluded
+# if it's not enabled
+ifneq ($(findstring y,"$(CONFIG_SPICE)"),y)
+AVOCADO_EXTRA_TAGS := $(AVOCADO_EXTRA_TAGS),-spice
+endif
+
 AVOCADO_TAGS=$(patsubst %-softmmu,--filter-by-tags=arch:%$(AVOCADO_EXTRA_TAGS), $(filter %-softmmu,$(TARGET_DIRS)))
 
 ifneq ($(findstring v2,"v$(PYTHON_VERSION)"),v2)
tests/acceptance/spice.py new file +54 -0

54 lines not on this page · raw message

Signature · 1 line
2.21.0
#8 lore raw

Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add SPICE protocol check

Quoted 40 lines · Cleber Rosa

This fires a QEMU binary with SPICE enabled, and does a basic handshake, doing a basic client/server interaction and protocol validation.

---
  .travis.yml               |  5 +++-
  tests/Makefile.include    |  6 +++++
  tests/acceptance/spice.py | 54 +++++++++++++++++++++++++++++++++++++++
  3 files changed, 64 insertions(+), 1 deletion(-)

create mode 100644 tests/acceptance/spice.py

.travis.yml +4 -1
index aeb9b211cd..6c9257a459 100644
@@ -231,7 +231,7 @@ matrix:
 
      # Acceptance (Functional) tests
      - env:
-        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu"
+        - CONFIG="--python=/usr/bin/python3 --enable-spice --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu"
          - TEST_CMD="make check-acceptance"
        after_failure:
          - cat tests/results/latest/job.log
@@ -240,6 +240,9 @@ matrix:
            packages:
              - python3-pip
              - python3.5-venv
+            - libspice-protocol-dev
+            - libspice-server-dev
+
      # Using newer GCC with sanitizers
      - addons:
          apt:
tests/Makefile.include +0 -0
index 4c97da2878..7fc2d28099 100644

tests/Makefile.include AVOCADO_SHOW=app · line 1137

Quoted 9 lines
tests/Makefile.include +6 -0
@@ -1129,6 +1129,12 @@ AVOCADO_SHOW=app
  # Additional tags that are added to each occurence of "--filter-by-tags"
  AVOCADO_EXTRA_TAGS := ,-flaky
 
+# At last one test require spice to be enabled, allow it to be excluded
+# if it's not enabled
+ifneq ($(findstring y,"$(CONFIG_SPICE)"),y)
+AVOCADO_EXTRA_TAGS := $(AVOCADO_EXTRA_TAGS),-spice
+endif
+

Cleber, what about that improvement to avocado_qemu you were developing to parse the configure files then expose the enabled/disabled features to test code? Do you still plan to push it and so this proposal is just temporary?

Quoted 53 lines
tests/Makefile.include +0 -0
  AVOCADO_TAGS=$(patsubst %-softmmu,--filter-by-tags=arch:%$(AVOCADO_EXTRA_TAGS), $(filter %-softmmu,$(TARGET_DIRS)))
 
  ifneq ($(findstring v2,"v$(PYTHON_VERSION)"),v2)
tests/acceptance/spice.py new file +44 -0
new file mode 100644
index 0000000000..aa22b1992d
@@ -0,0 +1,54 @@
+# Simple functional tests for SPICE functionality
+#
+# Copyright (c) 2019 Red Hat, Inc.
+#
+# Author:
+#  Cleber Rosa <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import socket
+import struct
+
+from avocado_qemu import Test
+from avocado.utils.network import find_free_port
+
+
+class Spice(Test):
+
+    def test_protocol(self):
+        """
+        :avocado: tags=quick
+        :avocado: tags=spice
+        """
+        port = find_free_port(5001, 5500, sequent=False)
+        self.vm.add_args('-nodefaults', '-S',
+                         '-spice', 'port=%d,disable-ticketing' % port)
+        self.vm.launch()
+
+        RED_MAGIC = 0x51444552
+        MAJOR_VERSION = 0x2
+
+        client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+        client.connect(('127.0.0.1', port))
+        red_link_mess = struct.pack('<10I',
+                                    RED_MAGIC,      # magic
+                                    MAJOR_VERSION,  # major version
+                                    0x0,            # minor version
+                                    0x18,           # size in bytes from here
+                                    0x0,            # connection id
+                                    0x1,            # channel type RED_CHANNEL_MAIN
+                                    0x0,            # channel id
+                                    0x0,            # number of common caps
+                                    0x0,            # number of channel caps

tests/acceptance/spice.py line 54

Quoted 10 lines
tests/acceptance/spice.py new file +10 -0
+                                    0x14)           # caps offset from size
+        client.send(red_link_mess)
+
+        RED_LINK_REPLY_BASE_FMT = '<5I'  # magic, major, minor, size, error
+        red_link_reply = client.recv(struct.calcsize(RED_LINK_REPLY_BASE_FMT))
+        (magic, major, minor, size, error) = struct.unpack_from(RED_LINK_REPLY_BASE_FMT,
+                                                                red_link_reply)
+        self.assertEqual(magic, RED_MAGIC, "Mismatch of MAGIC number")
+        self.assertEqual(major, MAJOR_VERSION, "Mismatch of major protocol version")
+        self.assertEqual(error, 0x0, "Unexpected error reported by server")

That test case looks good to me.

- Wainer

#9 lore raw

Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add SPICE protocol check

Quoted 45 lines · Wainer dos Santos Moschetta
Quoted 43 lines · Cleber Rosa

This fires a QEMU binary with SPICE enabled, and does a basic handshake, doing a basic client/server interaction and protocol validation.

---
  .travis.yml               |  5 +++-
  tests/Makefile.include    |  6 +++++
  tests/acceptance/spice.py | 54 +++++++++++++++++++++++++++++++++++++++
  3 files changed, 64 insertions(+), 1 deletion(-)

create mode 100644 tests/acceptance/spice.py

.travis.yml +4 -1
index aeb9b211cd..6c9257a459 100644
@@ -231,7 +231,7 @@ matrix:
      # Acceptance (Functional) tests
      - env:
-        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu"
+        - CONFIG="--python=/usr/bin/python3 --enable-spice --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu"
          - TEST_CMD="make check-acceptance"
        after_failure:
          - cat tests/results/latest/job.log
@@ -240,6 +240,9 @@ matrix:
            packages:
              - python3-pip
              - python3.5-venv
+            - libspice-protocol-dev
+            - libspice-server-dev
+
      # Using newer GCC with sanitizers
      - addons:
          apt:
tests/Makefile.include +2 -0
index 4c97da2878..7fc2d28099 100644
@@ -1129,6 +1129,12 @@ AVOCADO_SHOW=app
  # Additional tags that are added to each occurence of "--filter-by-tags"
  AVOCADO_EXTRA_TAGS := ,-flaky
+# At last one test require spice to be enabled, allow it to be excluded
+# if it's not enabled
Quoted 9 lines

tests/Makefile.include AVOCADO_SHOW=app · line 1137

Quoted 4 lines
tests/Makefile.include +4 -0
+ifneq ($(findstring y,"$(CONFIG_SPICE)"),y)
+AVOCADO_EXTRA_TAGS := $(AVOCADO_EXTRA_TAGS),-spice
+endif
+

Cleber, what about that improvement to avocado_qemu you were developing to parse the configure files then expose the enabled/disabled features to test code? Do you still plan to push it and so this proposal is just temporary?

That was actually a prototype that was done *before* the days of "avocado_qemu"[1]. While the main reason for it to not have moved forward back then was the requirement of a build environment, I believe we can adapt some of the lessons learned there into a generic set of features for the test runner.

Basically:

* a generic capability mechanism should be present, with possibly many implementations (looking at the build environment is clearly one)

* Jobs should be able to include/exlude tests based on capabilities (akin to how we're using tags)

* for some other cases, tests should also be given a chance to loop at capabilities and decided to abort (cancel) at run time.

Having said that, I think we can start with the tools that we have, which should serve to make the scope of those future enhancements and features even clearer and better defined.

Regards,
- Cleber.

[1] - https://lists.gnu.org/archive/html/qemu-devel/2017-07/msg06757.html

Quoted 66 lines
Quoted 52 lines
tests/Makefile.include +0 -0
  AVOCADO_TAGS=$(patsubst %-softmmu,--filter-by-tags=arch:%$(AVOCADO_EXTRA_TAGS), $(filter %-softmmu,$(TARGET_DIRS)))
  ifneq ($(findstring v2,"v$(PYTHON_VERSION)"),v2)
tests/acceptance/spice.py new file +44 -0
new file mode 100644
index 0000000000..aa22b1992d
@@ -0,0 +1,54 @@
+# Simple functional tests for SPICE functionality
+#
+# Copyright (c) 2019 Red Hat, Inc.
+#
+# Author:
+#  Cleber Rosa <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import socket
+import struct
+
+from avocado_qemu import Test
+from avocado.utils.network import find_free_port
+
+
+class Spice(Test):
+
+    def test_protocol(self):
+        """
+        :avocado: tags=quick
+        :avocado: tags=spice
+        """
+        port = find_free_port(5001, 5500, sequent=False)
+        self.vm.add_args('-nodefaults', '-S',
+                         '-spice', 'port=%d,disable-ticketing' % port)
+        self.vm.launch()
+
+        RED_MAGIC = 0x51444552
+        MAJOR_VERSION = 0x2
+
+        client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+        client.connect(('127.0.0.1', port))
+        red_link_mess = struct.pack('<10I',
+                                    RED_MAGIC,      # magic
+                                    MAJOR_VERSION,  # major version
+                                    0x0,            # minor version
+                                    0x18,           # size in bytes from here
+                                    0x0,            # connection id
+                                    0x1,            # channel type RED_CHANNEL_MAIN
+                                    0x0,            # channel id
+                                    0x0,            # number of common caps
+                                    0x0,            # number of channel caps

tests/acceptance/spice.py line 54

Quoted 10 lines
tests/acceptance/spice.py new file +10 -0
+                                    0x14)           # caps offset from size
+        client.send(red_link_mess)
+
+        RED_LINK_REPLY_BASE_FMT = '<5I'  # magic, major, minor, size, error
+        red_link_reply = client.recv(struct.calcsize(RED_LINK_REPLY_BASE_FMT))
+        (magic, major, minor, size, error) = struct.unpack_from(RED_LINK_REPLY_BASE_FMT,
+                                                                red_link_reply)
+        self.assertEqual(magic, RED_MAGIC, "Mismatch of MAGIC number")
+        self.assertEqual(major, MAJOR_VERSION, "Mismatch of major protocol version")
+        self.assertEqual(error, 0x0, "Unexpected error reported by server")

That test case looks good to me.

- Wainer