Acceptance Tests: basic architecture support

On a number of different scenarios, such as when choosing a QEMU binary to be used on tests (or a image to use to boot a test VM), it's useful to define the architecture that should be used.

This introduces both a test parameter and a test instance attribute, that will contain such a value.

The selection of the default QEMU binary, if one is not given explicitly, has also been changed to look at the arch attribute.

Signed-off-by Cleber Rosa <[email protected]>

---
 docs/devel/testing.rst                    | 18 ++++++++++++++++++
 tests/acceptance/avocado_qemu/__init__.py | 13 ++++++++++---
 2 files changed, 28 insertions(+), 3 deletions(-)

docs/devel/testing.rst +18 -0

index 727c4019b5..4178ae5022 100644
@@ -659,6 +659,17 @@ vm
 A QEMUMachine instance, initially configured according to the given
 ``qemu_bin`` parameter.
 
+arch
+~~~~
+
+The architecture that will be used on a number of different
+scenarios.  For instance, when a QEMU binary is not explicitly given,
+the one selected will depend on this attribute.
+
+The ``arch`` attribute will be set to the test parameter of the same
+name, and if one is not given explicitly, it will default to the
+system architecture (as given by ``uname``).
+
 qemu_bin
 ~~~~~~~~
 

@@ -681,6 +692,13 @@ like the following:
 
   PARAMS (key=qemu_bin, path=*, default=x86_64-softmmu/qemu-system-x86_64) => 'x86_64-softmmu/qemu-system-x86_64
 
+arch
+~~~~
+
+The architecture that will be used on a number of different scenarios.
+This parameter has a direct relation with the ``arch`` attribute.  If
+not given, it will default to None.
+
 qemu_bin
 ~~~~~~~~
 

tests/acceptance/avocado_qemu/__init__.py +10 -3

index d8d5b48dac..b8d934382d 100644
@@ -23,16 +23,22 @@ def is_readable_executable_file(path):
     return os.path.isfile(path) and os.access(path, os.R_OK | os.X_OK)
 
 
-def pick_default_qemu_bin():
+def pick_default_qemu_bin(arch=None):
     """
     Picks the path of a QEMU binary, starting either in the current working
     directory or in the source tree root directory.
 
+    :param arch: the arch to use when looking for a QEMU binary (the target
+                 will match the arch given).  If None (the default) arch
+                 will be the current host system arch (as given by
+                 :func:`os.uname`).
+    :type arch: str
     :returns: the path to the default QEMU binary or None if one could not
               be found
     :rtype: str or None
     """
-    arch = os.uname()[4]
+    if arch is None:
+        arch = os.uname()[4]
     qemu_bin_relative_path = os.path.join("%s-softmmu" % arch,
                                           "qemu-system-%s" % arch)
     if is_readable_executable_file(qemu_bin_relative_path):

@@ -47,8 +53,9 @@ def pick_default_qemu_bin():
 class Test(avocado.Test):
     def setUp(self):
         self.vm = None
+        self.arch = self.params.get('arch', default=os.uname()[4])
         self.qemu_bin = self.params.get('qemu_bin',
-                                        default=pick_default_qemu_bin())
+                                        default=pick_default_qemu_bin(self.arch))
         if self.qemu_bin is None:
             self.cancel("No QEMU binary defined or found in the source tree")
         self.vm = QEMUMachine(self.qemu_bin)

Because some sane defaults may require the knowledge of the arch, let's give the QEMUMachine the opportunity to hold that information.

Signed-off-by Cleber Rosa <[email protected]>

Reviewed-by Philippe Mathieu-Daudé <[email protected]>

---
 scripts/qemu.py | 7 +++++++
 1 file changed, 7 insertions(+)

scripts/qemu.py +7 -0

index f099ce7278..d9e24a0c1a 100644
@@ -113,6 +113,7 @@ class QEMUMachine(object):
         self._test_dir = test_dir
         self._temp_dir = None
         self._launched = False
+        self._arch = None
         self._machine = None
         self._console_device_type = None
         self._console_address = None

@@ -406,6 +407,12 @@ class QEMUMachine(object):
         '''
         self._args.extend(args)
 
+    def set_arch(self, arch):
+        """
+        Sets the architecture of this machine
+        """
+        self._arch = arch
+
     def set_machine(self, machine_type):
         '''
         Sets the machine type

Some targets require a machine type to be set, as there's no default (aarch64 is one example). To give a consistent interface to users of this API, this changes set_machine() so that a predefined default can be used, if one is not given. The approach used is exactly the same with the console device type.

Also, even when there's a default machine type, for some purposes, testing included, it's better if outside code is explicit about the machine type, instead of relying on whatever is set internally.

Signed-off-by Cleber Rosa <[email protected]>

---
 scripts/qemu.py | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

scripts/qemu.py +0 -0

index d9e24a0c1a..fca9b76990 100644

scripts/qemu.py +14 -1

+    }
+
 
 class QEMUMachineError(Exception):
     """

@@ -413,13 +422,24 @@ class QEMUMachine(object):
         """
         self._arch = arch
 
-    def set_machine(self, machine_type):
+    def set_machine(self, machine_type=None):
         '''
         Sets the machine type
 
         If set, the machine type will be added to the base arguments
         of the resulting QEMU command line.
         '''
+        if machine_type is None:
+            if self._arch is None:
+                raise QEMUMachineError("Can not set a default machine type: "
+                                       "QEMU instance without a defined arch")
+            for regex, machine in MACHINE_TYPES.items():
+                if re.match(regex, self._arch):
+                    machine_type = machine
+                    break
+            if machine_type is None:
+                raise QEMUMachineError("Can not set a machine type: no "
+                                       "matching machine type definition")
         self._machine = machine_type
 
     def set_console(self, device_type=None):

Quoted 20 lines · Cleber Rosa

Some targets require a machine type to be set, as there's no default (aarch64 is one example). To give a consistent interface to users of this API, this changes set_machine() so that a predefined default can be used, if one is not given. The approach used is exactly the same with the console device type.

Also, even when there's a default machine type, for some purposes, testing included, it's better if outside code is explicit about the machine type, instead of relying on whatever is set internally.

Signed-off-by Cleber Rosa <[email protected]>

---
 scripts/qemu.py | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

scripts/qemu.py +0 -0

index d9e24a0c1a..fca9b76990 100644

scripts/qemu.py CONSOLE_DEV_TYPES = { · line 45

Quoted 10 lines

scripts/qemu.py +7 -0

@@ -36,6 +36,15 @@ CONSOLE_DEV_TYPES = {
     r'^s390-ccw-virtio.*': 'sclpconsole',
     }
 
+#: Maps archictures to the preferred machine type
+MACHINE_TYPES = {
+    r'^aarch64$': 'virt',
+    r'^ppc$': 'g3beige',
+    r'^ppc64$': 'pseries',
+    r'^s390x$': 's390-ccw-virtio',
+    r'^x86_64$': 'q35',

Why choose Q35 rather than PC (the default)?

I was wondering about how to generate variants/machines.json but this is definitively something we want to do via a QMP query.

Eduardo what do you think?

Quoted 32 lines

scripts/qemu.py +14 -1

+    }
+
 
 class QEMUMachineError(Exception):
     """

@@ -413,13 +422,24 @@ class QEMUMachine(object):
         """
         self._arch = arch
 
-    def set_machine(self, machine_type):
+    def set_machine(self, machine_type=None):
         '''
         Sets the machine type
 
         If set, the machine type will be added to the base arguments
         of the resulting QEMU command line.
         '''
+        if machine_type is None:
+            if self._arch is None:
+                raise QEMUMachineError("Can not set a default machine type: "
+                                       "QEMU instance without a defined arch")
+            for regex, machine in MACHINE_TYPES.items():
+                if re.match(regex, self._arch):
+                    machine_type = machine
+                    break
+            if machine_type is None:
+                raise QEMUMachineError("Can not set a machine type: no "
+                                       "matching machine type definition")
         self._machine = machine_type
 
     def set_console(self, device_type=None):

Quoted 34 lines · Philippe Mathieu-Daudé

Quoted 20 lines · Cleber Rosa

Some targets require a machine type to be set, as there's no default (aarch64 is one example). To give a consistent interface to users of this API, this changes set_machine() so that a predefined default can be used, if one is not given. The approach used is exactly the same with the console device type.

Also, even when there's a default machine type, for some purposes, testing included, it's better if outside code is explicit about the machine type, instead of relying on whatever is set internally.

Signed-off-by Cleber Rosa <[email protected]>

---
 scripts/qemu.py | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

scripts/qemu.py +0 -0

index d9e24a0c1a..fca9b76990 100644

scripts/qemu.py CONSOLE_DEV_TYPES = { · line 45

Quoted 10 lines

scripts/qemu.py +7 -0

@@ -36,6 +36,15 @@ CONSOLE_DEV_TYPES = {
     r'^s390-ccw-virtio.*': 'sclpconsole',
     }
 
+#: Maps archictures to the preferred machine type
+MACHINE_TYPES = {
+    r'^aarch64$': 'virt',
+    r'^ppc$': 'g3beige',
+    r'^ppc64$': 'pseries',
+    r'^s390x$': 's390-ccw-virtio',
+    r'^x86_64$': 'q35',

Why choose Q35 rather than PC (the default)?

Quoted 5 lines

I was wondering about how to generate variants/machines.json but this is definitively something we want to do via a QMP query.

Eduardo what do you think?

It was motivated by Eduardo's initiative to make q35 the default "across the board". He can confirm and give more details.

Quoted 43 lines · Cleber Rosa

Quoted 39 lines · Philippe Mathieu-Daudé

Quoted 20 lines · Cleber Rosa

Some targets require a machine type to be set, as there's no default (aarch64 is one example). To give a consistent interface to users of this API, this changes set_machine() so that a predefined default can be used, if one is not given. The approach used is exactly the same with the console device type.

Also, even when there's a default machine type, for some purposes, testing included, it's better if outside code is explicit about the machine type, instead of relying on whatever is set internally.

Signed-off-by Cleber Rosa <[email protected]>

---
 scripts/qemu.py | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

scripts/qemu.py +0 -0

index d9e24a0c1a..fca9b76990 100644

scripts/qemu.py CONSOLE_DEV_TYPES = { · line 45

Quoted 10 lines

scripts/qemu.py +7 -0

@@ -36,6 +36,15 @@ CONSOLE_DEV_TYPES = {
     r'^s390-ccw-virtio.*': 'sclpconsole',
     }
 
+#: Maps archictures to the preferred machine type
+MACHINE_TYPES = {
+    r'^aarch64$': 'virt',
+    r'^ppc$': 'g3beige',
+    r'^ppc64$': 'pseries',
+    r'^s390x$': 's390-ccw-virtio',
+    r'^x86_64$': 'q35',

Why choose Q35 rather than PC (the default)?

I was wondering about how to generate variants/machines.json but this is definitively something we want to do via a QMP query.

Eduardo what do you think?

Quoted 2 lines

It was motivated by Eduardo's initiative to make q35 the default "across the board". He can confirm and give more details.

Making Q35 the default on applications using QEMU and libvirt is something I'd like to happen. But I think the simplest way to do that is to change the QEMU default. This way you won't need this table on qemu.py: you can just use the default provided by QEMU.

Quoted 44 lines · Eduardo Habkost

Quoted 43 lines · Cleber Rosa

Quoted 39 lines · Philippe Mathieu-Daudé

Quoted 20 lines · Cleber Rosa

Some targets require a machine type to be set, as there's no default (aarch64 is one example). To give a consistent interface to users of this API, this changes set_machine() so that a predefined default can be used, if one is not given. The approach used is exactly the same with the console device type.

Also, even when there's a default machine type, for some purposes, testing included, it's better if outside code is explicit about the machine type, instead of relying on whatever is set internally.

Signed-off-by Cleber Rosa <[email protected]>

---
 scripts/qemu.py | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

scripts/qemu.py +0 -0

index d9e24a0c1a..fca9b76990 100644

scripts/qemu.py CONSOLE_DEV_TYPES = { · line 45

Quoted 10 lines

scripts/qemu.py +7 -0

@@ -36,6 +36,15 @@ CONSOLE_DEV_TYPES = {
     r'^s390-ccw-virtio.*': 'sclpconsole',
     }
 
+#: Maps archictures to the preferred machine type
+MACHINE_TYPES = {
+    r'^aarch64$': 'virt',
+    r'^ppc$': 'g3beige',
+    r'^ppc64$': 'pseries',
+    r'^s390x$': 's390-ccw-virtio',
+    r'^x86_64$': 'q35',

Why choose Q35 rather than PC (the default)?

I was wondering about how to generate variants/machines.json but this is definitively something we want to do via a QMP query.

Eduardo what do you think?

Quoted 8 lines

Quoted 2 lines

It was motivated by Eduardo's initiative to make q35 the default "across the board". He can confirm and give more details.

Making Q35 the default on applications using QEMU and libvirt is something I'd like to happen. But I think the simplest way to do that is to change the QEMU default. This way you won't need this table on qemu.py: you can just use the default provided by QEMU.

The idea is to bring consistency on how we're calling "qemu-system-$(ARCH)", and at the same time apply the "explicit is better than implicit" rule.

The most important fact is that some targets do not (currently) have "the default provided by QEMU", aarch64 is one of them.

- Cleber.

Quoted 60 lines · Cleber Rosa

Quoted 44 lines · Eduardo Habkost

Quoted 43 lines · Cleber Rosa

Quoted 39 lines · Philippe Mathieu-Daudé

Quoted 20 lines · Cleber Rosa

Some targets require a machine type to be set, as there's no default (aarch64 is one example). To give a consistent interface to users of this API, this changes set_machine() so that a predefined default can be used, if one is not given. The approach used is exactly the same with the console device type.

Also, even when there's a default machine type, for some purposes, testing included, it's better if outside code is explicit about the machine type, instead of relying on whatever is set internally.

Signed-off-by Cleber Rosa <[email protected]>

---
 scripts/qemu.py | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

scripts/qemu.py +0 -0

index d9e24a0c1a..fca9b76990 100644

scripts/qemu.py CONSOLE_DEV_TYPES = { · line 45

Quoted 10 lines

scripts/qemu.py +7 -0

@@ -36,6 +36,15 @@ CONSOLE_DEV_TYPES = {
     r'^s390-ccw-virtio.*': 'sclpconsole',
     }
 
+#: Maps archictures to the preferred machine type
+MACHINE_TYPES = {
+    r'^aarch64$': 'virt',
+    r'^ppc$': 'g3beige',
+    r'^ppc64$': 'pseries',
+    r'^s390x$': 's390-ccw-virtio',
+    r'^x86_64$': 'q35',

Why choose Q35 rather than PC (the default)?

I was wondering about how to generate variants/machines.json but this is definitively something we want to do via a QMP query.

Eduardo what do you think?

Quoted 8 lines

Quoted 2 lines

It was motivated by Eduardo's initiative to make q35 the default "across the board". He can confirm and give more details.

Making Q35 the default on applications using QEMU and libvirt is something I'd like to happen. But I think the simplest way to do that is to change the QEMU default. This way you won't need this table on qemu.py: you can just use the default provided by QEMU.

The idea is to bring consistency on how we're calling "qemu-system-$(ARCH)", and at the same time apply the "explicit is better than implicit" rule.

Quoted 5 lines

The most important fact is that some targets do not (currently) have "the default provided by QEMU", aarch64 is one of them.

- Cleber.

So I ended up not relaying the question properly: should we default (even if explicitly adding "-machine") to "pc"?

Quoted 63 lines · Cleber Rosa

Quoted 60 lines · Cleber Rosa

Quoted 44 lines · Eduardo Habkost

Quoted 43 lines · Cleber Rosa

Quoted 39 lines · Philippe Mathieu-Daudé

Quoted 20 lines · Cleber Rosa

Some targets require a machine type to be set, as there's no default (aarch64 is one example). To give a consistent interface to users of this API, this changes set_machine() so that a predefined default can be used, if one is not given. The approach used is exactly the same with the console device type.

Also, even when there's a default machine type, for some purposes, testing included, it's better if outside code is explicit about the machine type, instead of relying on whatever is set internally.

Signed-off-by Cleber Rosa <[email protected]>

---
 scripts/qemu.py | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

scripts/qemu.py +0 -0

index d9e24a0c1a..fca9b76990 100644

scripts/qemu.py CONSOLE_DEV_TYPES = { · line 45

Quoted 10 lines

scripts/qemu.py +7 -0

@@ -36,6 +36,15 @@ CONSOLE_DEV_TYPES = {
     r'^s390-ccw-virtio.*': 'sclpconsole',
     }
 
+#: Maps archictures to the preferred machine type
+MACHINE_TYPES = {
+    r'^aarch64$': 'virt',
+    r'^ppc$': 'g3beige',
+    r'^ppc64$': 'pseries',
+    r'^s390x$': 's390-ccw-virtio',
+    r'^x86_64$': 'q35',

Why choose Q35 rather than PC (the default)?

I was wondering about how to generate variants/machines.json but this is definitively something we want to do via a QMP query.

Eduardo what do you think?

Quoted 8 lines

Quoted 2 lines

It was motivated by Eduardo's initiative to make q35 the default "across the board". He can confirm and give more details.

Making Q35 the default on applications using QEMU and libvirt is something I'd like to happen. But I think the simplest way to do that is to change the QEMU default. This way you won't need this table on qemu.py: you can just use the default provided by QEMU.

The idea is to bring consistency on how we're calling "qemu-system-$(ARCH)", and at the same time apply the "explicit is better than implicit" rule.

Quoted 8 lines

Quoted 5 lines

The most important fact is that some targets do not (currently) have "the default provided by QEMU", aarch64 is one of them.

- Cleber.

So I ended up not relaying the question properly: should we default (even if explicitly adding "-machine") to "pc"?

I think using the default machine-type (when QEMU has a default) would be less surprising for users of the qemu.py API.

Implicitly adding -machine when there's no default is also surprising, but then it's a nice surprise: instead of crashing you get a running VM.

Now, there are two other questions related to this:

If using 'pc' as default, should we always add -machine, or just omit the machine-type name? I think we should omit it unless the caller asked for a specific machine-type name (because it would be less surprising for users of the API).

About our default testing configuration for acceptance tests:
should acceptance tests run against PC by default? Should it test Q35? Should we test both PC and Q35? I'm not sure what's the answer, but I think these decisions shouldn't affect the qemu.py API at all.

Quoted 80 lines · Eduardo Habkost

Quoted 63 lines · Cleber Rosa

Quoted 60 lines · Cleber Rosa

Quoted 44 lines · Eduardo Habkost

Quoted 43 lines · Cleber Rosa

Quoted 39 lines · Philippe Mathieu-Daudé

Quoted 20 lines · Cleber Rosa

Some targets require a machine type to be set, as there's no default (aarch64 is one example). To give a consistent interface to users of this API, this changes set_machine() so that a predefined default can be used, if one is not given. The approach used is exactly the same with the console device type.

Also, even when there's a default machine type, for some purposes, testing included, it's better if outside code is explicit about the machine type, instead of relying on whatever is set internally.

Signed-off-by Cleber Rosa <[email protected]>

---
 scripts/qemu.py | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

scripts/qemu.py +0 -0

index d9e24a0c1a..fca9b76990 100644

scripts/qemu.py CONSOLE_DEV_TYPES = { · line 45

Quoted 10 lines

scripts/qemu.py +7 -0

@@ -36,6 +36,15 @@ CONSOLE_DEV_TYPES = {
     r'^s390-ccw-virtio.*': 'sclpconsole',
     }
 
+#: Maps archictures to the preferred machine type
+MACHINE_TYPES = {
+    r'^aarch64$': 'virt',
+    r'^ppc$': 'g3beige',
+    r'^ppc64$': 'pseries',
+    r'^s390x$': 's390-ccw-virtio',
+    r'^x86_64$': 'q35',

Why choose Q35 rather than PC (the default)?

I was wondering about how to generate variants/machines.json but this is definitively something we want to do via a QMP query.

Eduardo what do you think?

Quoted 8 lines

Quoted 2 lines

It was motivated by Eduardo's initiative to make q35 the default "across the board". He can confirm and give more details.

Making Q35 the default on applications using QEMU and libvirt is something I'd like to happen. But I think the simplest way to do that is to change the QEMU default. This way you won't need this table on qemu.py: you can just use the default provided by QEMU.

The idea is to bring consistency on how we're calling "qemu-system-$(ARCH)", and at the same time apply the "explicit is better than implicit" rule.

Quoted 8 lines

Quoted 5 lines

The most important fact is that some targets do not (currently) have "the default provided by QEMU", aarch64 is one of them.

- Cleber.

So I ended up not relaying the question properly: should we default (even if explicitly adding "-machine") to "pc"?

I think using the default machine-type (when QEMU has a default) would be less surprising for users of the qemu.py API.

Implicitly adding -machine when there's no default is also surprising, but then it's a nice surprise: instead of crashing you get a running VM.

Quoted 6 lines

Now, there are two other questions related to this:

If using 'pc' as default, should we always add -machine, or just omit the machine-type name? I think we should omit it unless the caller asked for a specific machine-type name (because it would be less surprising for users of the API).

I agree with that.

Quoted 6 lines

About our default testing configuration for acceptance tests:
should acceptance tests run against PC by default? Should it test Q35? Should we test both PC and Q35? I'm not sure what's the answer, but I think these decisions shouldn't affect the qemu.py API at all.

If I'm going to submit contributions to some subsystem, I'd like to run all the tests that cover this subsystem, previous to annoy the maintainer.

For example if a series target the "X86 Machines" subsystem, then I'd expect the JSON variant to test both PC and Q35.

Regards,

Phil.

Quoted 81 lines · Philippe Mathieu-Daudé

Quoted 80 lines · Eduardo Habkost

Quoted 63 lines · Cleber Rosa

Quoted 60 lines · Cleber Rosa

Quoted 44 lines · Eduardo Habkost

Quoted 43 lines · Cleber Rosa

Quoted 39 lines · Philippe Mathieu-Daudé

Quoted 20 lines · Cleber Rosa

Some targets require a machine type to be set, as there's no default (aarch64 is one example). To give a consistent interface to users of this API, this changes set_machine() so that a predefined default can be used, if one is not given. The approach used is exactly the same with the console device type.

Also, even when there's a default machine type, for some purposes, testing included, it's better if outside code is explicit about the machine type, instead of relying on whatever is set internally.

Signed-off-by Cleber Rosa <[email protected]>

---
 scripts/qemu.py | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

scripts/qemu.py +0 -0

index d9e24a0c1a..fca9b76990 100644

scripts/qemu.py CONSOLE_DEV_TYPES = { · line 45

Quoted 10 lines

scripts/qemu.py +7 -0

@@ -36,6 +36,15 @@ CONSOLE_DEV_TYPES = {
     r'^s390-ccw-virtio.*': 'sclpconsole',
     }
 
+#: Maps archictures to the preferred machine type
+MACHINE_TYPES = {
+    r'^aarch64$': 'virt',
+    r'^ppc$': 'g3beige',
+    r'^ppc64$': 'pseries',
+    r'^s390x$': 's390-ccw-virtio',
+    r'^x86_64$': 'q35',

Why choose Q35 rather than PC (the default)?

I was wondering about how to generate variants/machines.json but this is definitively something we want to do via a QMP query.

Eduardo what do you think?

Quoted 8 lines

Quoted 2 lines

It was motivated by Eduardo's initiative to make q35 the default "across the board". He can confirm and give more details.

Making Q35 the default on applications using QEMU and libvirt is something I'd like to happen. But I think the simplest way to do that is to change the QEMU default. This way you won't need this table on qemu.py: you can just use the default provided by QEMU.

The idea is to bring consistency on how we're calling "qemu-system-$(ARCH)", and at the same time apply the "explicit is better than implicit" rule.

Quoted 8 lines

Quoted 5 lines

The most important fact is that some targets do not (currently) have "the default provided by QEMU", aarch64 is one of them.

- Cleber.

So I ended up not relaying the question properly: should we default (even if explicitly adding "-machine") to "pc"?

I think using the default machine-type (when QEMU has a default) would be less surprising for users of the qemu.py API.

Implicitly adding -machine when there's no default is also surprising, but then it's a nice surprise: instead of crashing you get a running VM.

Quoted 9 lines

Quoted 6 lines

Now, there are two other questions related to this:

If using 'pc' as default, should we always add -machine, or just omit the machine-type name? I think we should omit it unless the caller asked for a specific machine-type name (because it would be less surprising for users of the API).

I agree with that.

OK!

Quoted 7 lines

Quoted 6 lines

About our default testing configuration for acceptance tests:
should acceptance tests run against PC by default? Should it test Q35? Should we test both PC and Q35? I'm not sure what's the answer, but I think these decisions shouldn't affect the qemu.py API at all.

Quoted 6 lines

If I'm going to submit contributions to some subsystem, I'd like to run all the tests that cover this subsystem, previous to annoy the maintainer.

For example if a series target the "X86 Machines" subsystem, then I'd expect the JSON variant to test both PC and Q35.

I agree, and we'll get there, but I'd rather do it in small steps.

The reason is that we want every single FAIL/ERROR on the acceptance tests to really flag a regression, so we need careful execution and validation prior to increasing the "test matrix".

At the same time, we need to be careful to not grow the default acceptance tests execution to a point that people won't run it. I've just heard similar feedback regarding Avocado-VT, that has *too many* tests that make it impractical for the individual developer to run.

From this description it should be easy to add filtering rules.

From a developer point of view, I'll run the tests covering the areas I

The plans we have for that is to define some sane test sets (probably based on tags and/or variants files), and at the same time, rely on combinatorial independent testing to reduce the number of test variations to make it practical for the regular developer to run on his environment.

Quoted 103 lines · Cleber Rosa

Quoted 81 lines · Philippe Mathieu-Daudé

Quoted 80 lines · Eduardo Habkost

Quoted 63 lines · Cleber Rosa

Quoted 60 lines · Cleber Rosa

Quoted 44 lines · Eduardo Habkost

Quoted 43 lines · Cleber Rosa

Quoted 39 lines · Philippe Mathieu-Daudé

Quoted 20 lines · Cleber Rosa

Some targets require a machine type to be set, as there's no default (aarch64 is one example). To give a consistent interface to users of this API, this changes set_machine() so that a predefined default can be used, if one is not given. The approach used is exactly the same with the console device type.

Also, even when there's a default machine type, for some purposes, testing included, it's better if outside code is explicit about the machine type, instead of relying on whatever is set internally.

Signed-off-by Cleber Rosa <[email protected]>

---
 scripts/qemu.py | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

scripts/qemu.py +0 -0

index d9e24a0c1a..fca9b76990 100644

scripts/qemu.py CONSOLE_DEV_TYPES = { · line 45

Quoted 10 lines

scripts/qemu.py +7 -0

@@ -36,6 +36,15 @@ CONSOLE_DEV_TYPES = {
     r'^s390-ccw-virtio.*': 'sclpconsole',
     }
 
+#: Maps archictures to the preferred machine type
+MACHINE_TYPES = {
+    r'^aarch64$': 'virt',
+    r'^ppc$': 'g3beige',
+    r'^ppc64$': 'pseries',
+    r'^s390x$': 's390-ccw-virtio',
+    r'^x86_64$': 'q35',

Why choose Q35 rather than PC (the default)?

I was wondering about how to generate variants/machines.json but this is definitively something we want to do via a QMP query.

Eduardo what do you think?

Quoted 8 lines

Quoted 2 lines

It was motivated by Eduardo's initiative to make q35 the default "across the board". He can confirm and give more details.

Making Q35 the default on applications using QEMU and libvirt is something I'd like to happen. But I think the simplest way to do that is to change the QEMU default. This way you won't need this table on qemu.py: you can just use the default provided by QEMU.

The idea is to bring consistency on how we're calling "qemu-system-$(ARCH)", and at the same time apply the "explicit is better than implicit" rule.

Quoted 8 lines

Quoted 5 lines

The most important fact is that some targets do not (currently) have "the default provided by QEMU", aarch64 is one of them.

- Cleber.

So I ended up not relaying the question properly: should we default (even if explicitly adding "-machine") to "pc"?

I think using the default machine-type (when QEMU has a default) would be less surprising for users of the qemu.py API.

Implicitly adding -machine when there's no default is also surprising, but then it's a nice surprise: instead of crashing you get a running VM.

Quoted 9 lines

Quoted 6 lines

Now, there are two other questions related to this:

If using 'pc' as default, should we always add -machine, or just omit the machine-type name? I think we should omit it unless the caller asked for a specific machine-type name (because it would be less surprising for users of the API).

I agree with that.

OK!

Quoted 7 lines

Quoted 6 lines

About our default testing configuration for acceptance tests:
should acceptance tests run against PC by default? Should it test Q35? Should we test both PC and Q35? I'm not sure what's the answer, but I think these decisions shouldn't affect the qemu.py API at all.

Quoted 8 lines

Quoted 6 lines

If I'm going to submit contributions to some subsystem, I'd like to run all the tests that cover this subsystem, previous to annoy the maintainer.

For example if a series target the "X86 Machines" subsystem, then I'd expect the JSON variant to test both PC and Q35.

I agree, and we'll get there, but I'd rather do it in small steps.

Sure.

Quoted 9 lines

The reason is that we want every single FAIL/ERROR on the acceptance tests to really flag a regression, so we need careful execution and validation prior to increasing the "test matrix".

At the same time, we need to be careful to not grow the default acceptance tests execution to a point that people won't run it. I've just heard similar feedback regarding Avocado-VT, that has *too many* tests that make it impractical for the individual developer to run.

The problem here is not the number of tests but the set of tests selected.

A test should have a description of what it is testing, the covered devices/modes/...
From this description it should be easy to add filtering rules.

From a developer point of view, I'll run the tests covering the areas I modified.
A maintainer runs more extensive tests on his subsystems.

Now if you plan a release, you are not an individual developer :)

Quoted 6 lines

The plans we have for that is to define some sane test sets (probably based on tags and/or variants files), and at the same time, rely on combinatorial independent testing to reduce the number of test variations to make it practical for the regular developer to run on his environment.

OK, we'll get there! :)

(I don't want to reject people to contribute tests because "we already have too many".)

Regards,

Phil.

Quoted 63 lines · Cleber Rosa

Quoted 60 lines · Cleber Rosa

Quoted 44 lines · Eduardo Habkost

Quoted 43 lines · Cleber Rosa

Quoted 39 lines · Philippe Mathieu-Daudé

Quoted 20 lines · Cleber Rosa

Some targets require a machine type to be set, as there's no default (aarch64 is one example). To give a consistent interface to users of this API, this changes set_machine() so that a predefined default can be used, if one is not given. The approach used is exactly the same with the console device type.

Also, even when there's a default machine type, for some purposes, testing included, it's better if outside code is explicit about the machine type, instead of relying on whatever is set internally.

Signed-off-by Cleber Rosa <[email protected]>

---
 scripts/qemu.py | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

scripts/qemu.py +0 -0

index d9e24a0c1a..fca9b76990 100644

scripts/qemu.py CONSOLE_DEV_TYPES = { · line 45

Quoted 10 lines

scripts/qemu.py +7 -0

@@ -36,6 +36,15 @@ CONSOLE_DEV_TYPES = {
     r'^s390-ccw-virtio.*': 'sclpconsole',
     }
 
+#: Maps archictures to the preferred machine type
+MACHINE_TYPES = {
+    r'^aarch64$': 'virt',
+    r'^ppc$': 'g3beige',
+    r'^ppc64$': 'pseries',
+    r'^s390x$': 's390-ccw-virtio',
+    r'^x86_64$': 'q35',

Why choose Q35 rather than PC (the default)?

I was wondering about how to generate variants/machines.json but this is definitively something we want to do via a QMP query.

Eduardo what do you think?

Quoted 8 lines

Quoted 2 lines

It was motivated by Eduardo's initiative to make q35 the default "across the board". He can confirm and give more details.

Making Q35 the default on applications using QEMU and libvirt is something I'd like to happen. But I think the simplest way to do that is to change the QEMU default. This way you won't need this table on qemu.py: you can just use the default provided by QEMU.

The idea is to bring consistency on how we're calling "qemu-system-$(ARCH)", and at the same time apply the "explicit is better than implicit" rule.

Quoted 8 lines

Quoted 5 lines

The most important fact is that some targets do not (currently) have "the default provided by QEMU", aarch64 is one of them.

- Cleber.

So I ended up not relaying the question properly: should we default (even if explicitly adding "-machine") to "pc"?

I think using the default machine-type (when QEMU has a default) would be less surprising for users of the qemu.py API.

Implicitly adding -machine when there's no default is also surprising, but then it's a nice surprise: instead of crashing you get a running VM.

Now, there are two other questions related to this:

If using 'pc' as default, should we always add -machine, or just omit the machine-type name? I think we should omit it unless the caller asked for a specific machine-type name (because it would be less surprising for users of the API).

About our default testing configuration for acceptance tests:
should acceptance tests run against PC by default? Should it test Q35? Should we test both PC and Q35? I'm not sure what's the answer, but I think these decisions shouldn't affect the qemu.py API at all.

Quoted 87 lines · Eduardo Habkost

Quoted 63 lines · Cleber Rosa

Quoted 60 lines · Cleber Rosa

Quoted 44 lines · Eduardo Habkost

Quoted 43 lines · Cleber Rosa

Quoted 39 lines · Philippe Mathieu-Daudé

Quoted 20 lines · Cleber Rosa

Some targets require a machine type to be set, as there's no default (aarch64 is one example). To give a consistent interface to users of this API, this changes set_machine() so that a predefined default can be used, if one is not given. The approach used is exactly the same with the console device type.

Also, even when there's a default machine type, for some purposes, testing included, it's better if outside code is explicit about the machine type, instead of relying on whatever is set internally.

Signed-off-by Cleber Rosa <[email protected]>

---
 scripts/qemu.py | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

scripts/qemu.py +0 -0

index d9e24a0c1a..fca9b76990 100644

scripts/qemu.py CONSOLE_DEV_TYPES = { · line 45

Quoted 10 lines

scripts/qemu.py +7 -0

@@ -36,6 +36,15 @@ CONSOLE_DEV_TYPES = {
     r'^s390-ccw-virtio.*': 'sclpconsole',
     }
 
+#: Maps archictures to the preferred machine type
+MACHINE_TYPES = {
+    r'^aarch64$': 'virt',
+    r'^ppc$': 'g3beige',
+    r'^ppc64$': 'pseries',
+    r'^s390x$': 's390-ccw-virtio',
+    r'^x86_64$': 'q35',

Why choose Q35 rather than PC (the default)?

I was wondering about how to generate variants/machines.json but this is definitively something we want to do via a QMP query.

Eduardo what do you think?

Quoted 8 lines

Quoted 2 lines

It was motivated by Eduardo's initiative to make q35 the default "across the board". He can confirm and give more details.

Making Q35 the default on applications using QEMU and libvirt is something I'd like to happen. But I think the simplest way to do that is to change the QEMU default. This way you won't need this table on qemu.py: you can just use the default provided by QEMU.

The idea is to bring consistency on how we're calling "qemu-system-$(ARCH)", and at the same time apply the "explicit is better than implicit" rule.

Quoted 8 lines

Quoted 5 lines

The most important fact is that some targets do not (currently) have "the default provided by QEMU", aarch64 is one of them.

- Cleber.

So I ended up not relaying the question properly: should we default (even if explicitly adding "-machine") to "pc"?

I think using the default machine-type (when QEMU has a default) would be less surprising for users of the qemu.py API.

Implicitly adding -machine when there's no default is also surprising, but then it's a nice surprise: instead of crashing you get a running VM.

Now, there are two other questions related to this:

If using 'pc' as default, should we always add -machine, or just omit the machine-type name? I think we should omit it unless the caller asked for a specific machine-type name (because it would be less surprising for users of the API).

Quoted 5 lines

About our default testing configuration for acceptance tests:
should acceptance tests run against PC by default? Should it test Q35? Should we test both PC and Q35? I'm not sure what's the answer, but I think these decisions shouldn't affect the qemu.py API at all.

As a general goal we should be aiming to test everything we provide to users if, we expect them to actually use it and not have it break later :-)

Given that 'pc' has been the default for entire existance of QEMU, 99% of guests are using 'pc' and thus we definitely want to be testing it.

Equally we'd like to encourage use of 'q35' so we should also be testing that, even though it has very little usage right now.

Thus, we really need to be running any tests against both machine types. If we do that, then the question of default doesn't really matter as one of them is always not the default and needs testing regardless.

Regards,
Daniel

Quoted 66 lines · Cleber Rosa

Quoted 63 lines · Cleber Rosa

Quoted 44 lines · Eduardo Habkost

Quoted 43 lines · Cleber Rosa

Quoted 39 lines · Philippe Mathieu-Daudé

Quoted 20 lines · Cleber Rosa

Some targets require a machine type to be set, as there's no default (aarch64 is one example). To give a consistent interface to users of this API, this changes set_machine() so that a predefined default can be used, if one is not given. The approach used is exactly the same with the console device type.

Also, even when there's a default machine type, for some purposes, testing included, it's better if outside code is explicit about the machine type, instead of relying on whatever is set internally.

Signed-off-by Cleber Rosa <[email protected]>

---
 scripts/qemu.py | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

scripts/qemu.py +0 -0

index d9e24a0c1a..fca9b76990 100644

scripts/qemu.py CONSOLE_DEV_TYPES = { · line 45

Quoted 10 lines

scripts/qemu.py +7 -0

@@ -36,6 +36,15 @@ CONSOLE_DEV_TYPES = {
     r'^s390-ccw-virtio.*': 'sclpconsole',
     }
 
+#: Maps archictures to the preferred machine type
+MACHINE_TYPES = {
+    r'^aarch64$': 'virt',
+    r'^ppc$': 'g3beige',
+    r'^ppc64$': 'pseries',
+    r'^s390x$': 's390-ccw-virtio',
+    r'^x86_64$': 'q35',

Why choose Q35 rather than PC (the default)?

I was wondering about how to generate variants/machines.json but this is definitively something we want to do via a QMP query.

Eduardo what do you think?

Quoted 8 lines

Quoted 2 lines

It was motivated by Eduardo's initiative to make q35 the default "across the board". He can confirm and give more details.

Making Q35 the default on applications using QEMU and libvirt is something I'd like to happen. But I think the simplest way to do that is to change the QEMU default. This way you won't need this table on qemu.py: you can just use the default provided by QEMU.

The idea is to bring consistency on how we're calling "qemu-system-$(ARCH)", and at the same time apply the "explicit is better than implicit" rule.

The most important fact is that some targets do not (currently) have "the default provided by QEMU", aarch64 is one of them.

Quoted 5 lines

Quoted 2 lines

- Cleber.

So I ended up not relaying the question properly: should we default (even if explicitly adding "-machine") to "pc"?

I think using the default machine-type (when QEMU has a default) would be less surprising for users of the qemu.py API.

Implicitly adding -machine when there's no default is also surprising, but then it's a nice surprise: instead of crashing you get a running VM.

Now, there are two other questions related to this:

If using 'pc' as default, should we always add -machine, or just omit the machine-type name? I think we should omit it unless the caller asked for a specific machine-type name (because it would be less surprising for users of the API).

About our default testing configuration for acceptance tests:
should acceptance tests run against PC by default? Should it test Q35? Should we test both PC and Q35? I'm not sure what's the answer, but I think these decisions shouldn't affect the qemu.py API at all.

To make sure we're on the same page, we're still going to have default machine types, based on the arch, for those targets that don't provide one (aarch64 is one example). Right?

Quoted 3 lines · Cleber Rosa

To make sure we're on the same page, we're still going to have default machine types, based on the arch, for those targets that don't provide one (aarch64 is one example). Right?

Does it make sense to define a default? The reason arm doesn't specify a default machine type is because you can't just run any old guest on any old machine type. You need to know "this guest image will run on machine type X", and run it on machine type X. This is like knowing you need to run a test on x86 PC and not on PPC spapr.

From a test writer perspective, working on these higher level tests, it

Would it make more sense for each test to specify which machine types it can work on?

thanks
-- PMM

Quoted 5 lines · Peter Maydell

Quoted 3 lines · Cleber Rosa

To make sure we're on the same page, we're still going to have default machine types, based on the arch, for those targets that don't provide one (aarch64 is one example). Right?

Quoted 8 lines

Does it make sense to define a default? The reason arm doesn't specify a default machine type is because you can't just run any old guest on any old machine type. You need to know "this guest image will run on machine type X", and run it on machine type X. This is like knowing you need to run a test on x86 PC and not on PPC spapr.

While requiring tests to specify every single aspect of the environment that will be used may be OK for low level unit tests, it puts a lot of burden on higher level tests (which is supposed to be the vast majority under tests/acceptance).

From a test writer perspective, working on these higher level tests, it may want to make sure that feature "X", unrelated to the target arch, machine type, etc, "just works". You man want to look at the "vnc.py" test for a real world example.

Eduardo has suggested that "make check-acceptance" runs all (possible) tests on all target archs by default.

Quoted 3 lines

Would it make more sense for each test to specify which machine types it can work on?

I think it does, but I believe in the black list approach, instead of the white list.

The reason for that is that I believe that majority of the tests under "tests/acceptance" can be made to work on every target (which would be the default). So far, I've made sure tests behave correctly on the 5 arches included in the "archs.json" file in this series (x86_64, ppc64, ppc, aarch64, s390x).

To give a full disclosure, "boot_linux.py" (boots a linux kernel) is x86_64 specific, and CANCELS when asked to be run on other archs. But, on the work I've done top of these series, it already works with ppc64 and aarch64. Also, "boot_linux.py" sent in another series, (which boots a full linux guest) is also being adapted to work on most of the target archs.

Quoted 22 lines · Cleber Rosa

Quoted 5 lines · Peter Maydell

Quoted 3 lines · Cleber Rosa

To make sure we're on the same page, we're still going to have default machine types, based on the arch, for those targets that don't provide one (aarch64 is one example). Right?

Quoted 8 lines

Does it make sense to define a default? The reason arm doesn't specify a default machine type is because you can't just run any old guest on any old machine type. You need to know "this guest image will run on machine type X", and run it on machine type X. This is like knowing you need to run a test on x86 PC and not on PPC spapr.

While requiring tests to specify every single aspect of the environment that will be used may be OK for low level unit tests, it puts a lot of burden on higher level tests (which is supposed to be the vast majority under tests/acceptance).

Quoted 4 lines

From a test writer perspective, working on these higher level tests, it may want to make sure that feature "X", unrelated to the target arch, machine type, etc, "just works". You man want to look at the "vnc.py" test for a real world example.

OK, if it doesn't have a dependency on machine at all, it should state that somehow.

Quoted 2 lines

Eduardo has suggested that "make check-acceptance" runs all (possible) tests on all target archs by default.

Yeah; or we have some mechanism for trimming down the matrix of what we run. But I think it's better coverage if we have 3 tests ABC that don't depend on machine and 3 machines XYZ to run AX BY CZ than AX BX CX by specifying X as an arbitrary "default".

It looks like the 'vnc' test is just testing QEMU functionality, not anything that involves interacting with the guest or machine model? There's a good argument that that only really needs to be run once, not once per architecture.

You might also want to consider the "none" machine, which exists for bits of test infrastructure that aren't actually trying to run guests.

Quoted 13 lines

Quoted 3 lines

Would it make more sense for each test to specify which machine types it can work on?

I think it does, but I believe in the black list approach, instead of the white list.

The reason for that is that I believe that majority of the tests under "tests/acceptance" can be made to work on every target (which would be the default). So far, I've made sure tests behave correctly on the 5 arches included in the "archs.json" file in this series (x86_64, ppc64, ppc, aarch64, s390x).

Quoted 6 lines

To give a full disclosure, "boot_linux.py" (boots a linux kernel) is x86_64 specific, and CANCELS when asked to be run on other archs. But, on the work I've done top of these series, it already works with ppc64 and aarch64. Also, "boot_linux.py" sent in another series, (which boots a full linux guest) is also being adapted to work on most of the target archs.

Right, "boot Linux" is machine specific. The kernel/disk /etc that boots on aarch64 virt is probably not going to boot on the 64-bit xilinx board; and on 32-bit arm you definitely are going to want a different kernel in some places. This is likely to be true of most tests that actually try to run code in the guest.

We should aim to test the machines we care about (regardless of what architectures they are), rather than thinking about it in terms of "testing architectures X, Y, Z", I think.

I think you're going to need at least some whitelist functionality; otherwise half the tests are going to break every time we add a new machine (and "add every new machine to the blacklist for half the tests" doesn't scale very well).

thanks
-- PMM

Quoted 4 lines · Peter Maydell

We should aim to test the machines we care about (regardless of what architectures they are), rather than thinking about it in terms of "testing architectures X, Y, Z", I think.

To me it's clear that:

1) I lack a complete understanding of what "we care about"
2) It's easier to start with something, and tweak it to taste

Quoted 70 lines · Eduardo Habkost

Quoted 69 lines · Cleber Rosa

Quoted 65 lines · Cleber Rosa

Quoted 44 lines · Eduardo Habkost

Quoted 43 lines · Cleber Rosa

Quoted 39 lines · Philippe Mathieu-Daudé

Quoted 20 lines · Cleber Rosa

Some targets require a machine type to be set, as there's no default (aarch64 is one example). To give a consistent interface to users of this API, this changes set_machine() so that a predefined default can be used, if one is not given. The approach used is exactly the same with the console device type.

Also, even when there's a default machine type, for some purposes, testing included, it's better if outside code is explicit about the machine type, instead of relying on whatever is set internally.

Signed-off-by Cleber Rosa <[email protected]>

---
 scripts/qemu.py | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

scripts/qemu.py +0 -0

index d9e24a0c1a..fca9b76990 100644

scripts/qemu.py CONSOLE_DEV_TYPES = { · line 45

Quoted 10 lines

scripts/qemu.py +7 -0

@@ -36,6 +36,15 @@ CONSOLE_DEV_TYPES = {
     r'^s390-ccw-virtio.*': 'sclpconsole',
     }
 
+#: Maps archictures to the preferred machine type
+MACHINE_TYPES = {
+    r'^aarch64$': 'virt',
+    r'^ppc$': 'g3beige',
+    r'^ppc64$': 'pseries',
+    r'^s390x$': 's390-ccw-virtio',
+    r'^x86_64$': 'q35',

Why choose Q35 rather than PC (the default)?

I was wondering about how to generate variants/machines.json but this is definitively something we want to do via a QMP query.

Eduardo what do you think?

Quoted 8 lines

Quoted 2 lines

It was motivated by Eduardo's initiative to make q35 the default "across the board". He can confirm and give more details.

Making Q35 the default on applications using QEMU and libvirt is something I'd like to happen. But I think the simplest way to do that is to change the QEMU default. This way you won't need this table on qemu.py: you can just use the default provided by QEMU.

The idea is to bring consistency on how we're calling "qemu-system-$(ARCH)", and at the same time apply the "explicit is better than implicit" rule.

The most important fact is that some targets do not (currently) have "the default provided by QEMU", aarch64 is one of them.

- Cleber.

Quoted 6 lines

Quoted 2 lines

So I ended up not relaying the question properly: should we default (even if explicitly adding "-machine") to "pc"?

I think using the default machine-type (when QEMU has a default) would be less surprising for users of the qemu.py API.

OK, agreed.

Quoted 11 lines

Implicitly adding -machine when there's no default is also surprising, but then it's a nice surprise: instead of crashing you get a running VM.

Now, there are two other questions related to this:

If using 'pc' as default, should we always add -machine, or just omit the machine-type name? I think we should omit it unless the caller asked for a specific machine-type name (because it would be less surprising for users of the API).

OK.

Quoted 6 lines

About our default testing configuration for acceptance tests:
should acceptance tests run against PC by default? Should it test Q35? Should we test both PC and Q35? I'm not sure what's the answer, but I think these decisions shouldn't affect the qemu.py API at all.

OK.

To make sure we're on the same page, we're still going to have default machine types, based on the arch, for those targets that don't provide one (aarch64 is one example). Right?

- Cleber.

Quoted 73 lines · Cleber Rosa

Quoted 70 lines · Eduardo Habkost

Quoted 69 lines · Cleber Rosa

Quoted 65 lines · Cleber Rosa

Quoted 44 lines · Eduardo Habkost

Quoted 43 lines · Cleber Rosa

Quoted 39 lines · Philippe Mathieu-Daudé

Quoted 20 lines · Cleber Rosa

Some targets require a machine type to be set, as there's no default (aarch64 is one example). To give a consistent interface to users of this API, this changes set_machine() so that a predefined default can be used, if one is not given. The approach used is exactly the same with the console device type.

Also, even when there's a default machine type, for some purposes, testing included, it's better if outside code is explicit about the machine type, instead of relying on whatever is set internally.

Signed-off-by Cleber Rosa <[email protected]>

---
 scripts/qemu.py | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

scripts/qemu.py +0 -0

index d9e24a0c1a..fca9b76990 100644

scripts/qemu.py CONSOLE_DEV_TYPES = { · line 45

Quoted 10 lines

scripts/qemu.py +7 -0

@@ -36,6 +36,15 @@ CONSOLE_DEV_TYPES = {
     r'^s390-ccw-virtio.*': 'sclpconsole',
     }
 
+#: Maps archictures to the preferred machine type
+MACHINE_TYPES = {
+    r'^aarch64$': 'virt',
+    r'^ppc$': 'g3beige',
+    r'^ppc64$': 'pseries',
+    r'^s390x$': 's390-ccw-virtio',
+    r'^x86_64$': 'q35',

Why choose Q35 rather than PC (the default)?

I was wondering about how to generate variants/machines.json but this is definitively something we want to do via a QMP query.

Eduardo what do you think?

Quoted 8 lines

Quoted 2 lines

It was motivated by Eduardo's initiative to make q35 the default "across the board". He can confirm and give more details.

Making Q35 the default on applications using QEMU and libvirt is something I'd like to happen. But I think the simplest way to do that is to change the QEMU default. This way you won't need this table on qemu.py: you can just use the default provided by QEMU.

The idea is to bring consistency on how we're calling "qemu-system-$(ARCH)", and at the same time apply the "explicit is better than implicit" rule.

The most important fact is that some targets do not (currently) have "the default provided by QEMU", aarch64 is one of them.

- Cleber.

Quoted 21 lines

Quoted 6 lines

Quoted 2 lines

So I ended up not relaying the question properly: should we default (even if explicitly adding "-machine") to "pc"?

I think using the default machine-type (when QEMU has a default) would be less surprising for users of the qemu.py API.

OK, agreed.

Quoted 11 lines

Implicitly adding -machine when there's no default is also surprising, but then it's a nice surprise: instead of crashing you get a running VM.

Now, there are two other questions related to this:

If using 'pc' as default, should we always add -machine, or just omit the machine-type name? I think we should omit it unless the caller asked for a specific machine-type name (because it would be less surprising for users of the API).

Getting down to business, trying to apply those changes, I was faced with a situation. Actually, the same situation I faced a few months ago. Handling it was defered until it was *really* a blocker. Basically the issue is: the set_console() method, which gives tests a ready to use console, depends on knowing the machine type (see CONSOLE_DEV_TYPES).

As a case study, let's look at "boot_console_linux.py":
1) it sets the machine type explicitly
2) it has nothing to do with the specific machine type
3) the setting of a machine type is boiler plate code to set a console
4) the console is used on the test's real purpose: verifying the Linux
kernel booted

Now, to be able to run the same test -- booting a Linux kernel -- on *other target archs*, we need the same machinery. Even more important:
to have similar tests we'll need to either abstract those features or duplicate them. This can be seen, at least in part, on the firmware tests that Philippe sent to the list: they would also benefit from having a console device ready to be used on the configured machine type[1]:

Assuming that we want to provide this type of machinery for free (or as close as that) to the acceptance/functional tests, we need some source of "known good" configuration for the targets we aim to support.

Let's restrict the discussion to the issue at hand, machine types, while keeping in mind that the same pattern happened with devices types to use as console before, and my experience running into default network device types in further work (tests that interact with the guest by ssh'ing into it).

The solutions I can think of are:

1) run the target binary previous to the "real" run, and query
information -- this is what Avocado-VT does[2], and something I tried on earlier versions of the acceptance tests infrastructure code

2) attempt to get this information from the build system[3]

3) hard code the "known" good configuration

I've previously worked on solutions along the lines of #1 and #2, but the general feedback wasn't that positive, for valid reasons. Eduardo probably remembers this.

So, I'm tempted to try solution #3. As much as duplicating target defaults in qemu.py doesn't sound perfect, it seems to be the more predictable and attainable solution at this point.

Thoughts?

Thanks!
- Cleber.

[1] - https://lists.gnu.org/archive/html/qemu-devel/2018-10/msg00591.html
[2] -
https://github.com/avocado-framework/avocado-vt/blob/65.0/virttest/utils_misc.py#L2105
[3] - http://lists.gnu.org/archive/html/qemu-devel/2017-07/msg06757.html

Quoted 17 lines

OK.

Quoted 6 lines

About our default testing configuration for acceptance tests:
should acceptance tests run against PC by default? Should it test Q35? Should we test both PC and Q35? I'm not sure what's the answer, but I think these decisions shouldn't affect the qemu.py API at all.

OK.

To make sure we're on the same page, we're still going to have default machine types, based on the arch, for those targets that don't provide one (aarch64 is one example). Right?

- Cleber.

Quoted 137 lines · Cleber Rosa

Quoted 73 lines · Cleber Rosa

Quoted 70 lines · Eduardo Habkost

Quoted 69 lines · Cleber Rosa

Quoted 65 lines · Cleber Rosa

Quoted 44 lines · Eduardo Habkost

Quoted 43 lines · Cleber Rosa

Quoted 39 lines · Philippe Mathieu-Daudé

Quoted 20 lines · Cleber Rosa

Some targets require a machine type to be set, as there's no default (aarch64 is one example). To give a consistent interface to users of this API, this changes set_machine() so that a predefined default can be used, if one is not given. The approach used is exactly the same with the console device type.

Also, even when there's a default machine type, for some purposes, testing included, it's better if outside code is explicit about the machine type, instead of relying on whatever is set internally.

Signed-off-by Cleber Rosa <[email protected]>

---
 scripts/qemu.py | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

scripts/qemu.py +0 -0

index d9e24a0c1a..fca9b76990 100644

scripts/qemu.py CONSOLE_DEV_TYPES = { · line 45

Quoted 10 lines

scripts/qemu.py +7 -0

@@ -36,6 +36,15 @@ CONSOLE_DEV_TYPES = {
     r'^s390-ccw-virtio.*': 'sclpconsole',
     }
 
+#: Maps archictures to the preferred machine type
+MACHINE_TYPES = {
+    r'^aarch64$': 'virt',
+    r'^ppc$': 'g3beige',
+    r'^ppc64$': 'pseries',
+    r'^s390x$': 's390-ccw-virtio',
+    r'^x86_64$': 'q35',

Why choose Q35 rather than PC (the default)?

I was wondering about how to generate variants/machines.json but this is definitively something we want to do via a QMP query.

Eduardo what do you think?

Quoted 8 lines

Quoted 2 lines

It was motivated by Eduardo's initiative to make q35 the default "across the board". He can confirm and give more details.

Making Q35 the default on applications using QEMU and libvirt is something I'd like to happen. But I think the simplest way to do that is to change the QEMU default. This way you won't need this table on qemu.py: you can just use the default provided by QEMU.

The idea is to bring consistency on how we're calling "qemu-system-$(ARCH)", and at the same time apply the "explicit is better than implicit" rule.

The most important fact is that some targets do not (currently) have "the default provided by QEMU", aarch64 is one of them.

- Cleber.

Quoted 21 lines

Quoted 6 lines

Quoted 2 lines

So I ended up not relaying the question properly: should we default (even if explicitly adding "-machine") to "pc"?

I think using the default machine-type (when QEMU has a default) would be less surprising for users of the qemu.py API.

OK, agreed.

Quoted 11 lines

Implicitly adding -machine when there's no default is also surprising, but then it's a nice surprise: instead of crashing you get a running VM.

Now, there are two other questions related to this:

If using 'pc' as default, should we always add -machine, or just omit the machine-type name? I think we should omit it unless the caller asked for a specific machine-type name (because it would be less surprising for users of the API).

Getting down to business, trying to apply those changes, I was faced with a situation. Actually, the same situation I faced a few months ago. Handling it was defered until it was *really* a blocker. Basically the issue is: the set_console() method, which gives tests a ready to use console, depends on knowing the machine type (see CONSOLE_DEV_TYPES).

As a case study, let's look at "boot_console_linux.py":
1) it sets the machine type explicitly
2) it has nothing to do with the specific machine type
3) the setting of a machine type is boiler plate code to set a console
4) the console is used on the test's real purpose: verifying the Linux
kernel booted

Now, to be able to run the same test -- booting a Linux kernel -- on *other target archs*, we need the same machinery. Even more important:
to have similar tests we'll need to either abstract those features or duplicate them. This can be seen, at least in part, on the firmware tests that Philippe sent to the list: they would also benefit from having a console device ready to be used on the configured machine type[1]:

Assuming that we want to provide this type of machinery for free (or as close as that) to the acceptance/functional tests, we need some source of "known good" configuration for the targets we aim to support.

Let's restrict the discussion to the issue at hand, machine types, while keeping in mind that the same pattern happened with devices types to use as console before, and my experience running into default network device types in further work (tests that interact with the guest by ssh'ing into it).

The solutions I can think of are:

1) run the target binary previous to the "real" run, and query
information -- this is what Avocado-VT does[2], and something I tried on earlier versions of the acceptance tests infrastructure code

2) attempt to get this information from the build system[3]

Quoted 5 lines

3) hard code the "known" good configuration

I've previously worked on solutions along the lines of #1 and #2, but the general feedback wasn't that positive, for valid reasons. Eduardo probably remembers this.

I don't remember seeing negative feedback for #1. It sounds like the best solution.

Quoted 4 lines

So, I'm tempted to try solution #3. As much as duplicating target defaults in qemu.py doesn't sound perfect, it seems to be the more predictable and attainable solution at this point.

I consider #3 to be acceptable just as a temporary solution until we implement #1.

Quoted 28 lines

Thoughts?

Thanks!
- Cleber.

[1] - https://lists.gnu.org/archive/html/qemu-devel/2018-10/msg00591.html
[2] -
https://github.com/avocado-framework/avocado-vt/blob/65.0/virttest/utils_misc.py#L2105
[3] - http://lists.gnu.org/archive/html/qemu-devel/2017-07/msg06757.html

Quoted 17 lines

OK.

Quoted 6 lines

About our default testing configuration for acceptance tests:
should acceptance tests run against PC by default? Should it test Q35? Should we test both PC and Q35? I'm not sure what's the answer, but I think these decisions shouldn't affect the qemu.py API at all.

OK.

To make sure we're on the same page, we're still going to have default machine types, based on the arch, for those targets that don't provide one (aarch64 is one example). Right?

- Cleber.

Quoted 138 lines · Eduardo Habkost

Quoted 137 lines · Cleber Rosa

Quoted 73 lines · Cleber Rosa

Quoted 70 lines · Eduardo Habkost

Quoted 69 lines · Cleber Rosa

Quoted 65 lines · Cleber Rosa

Quoted 44 lines · Eduardo Habkost

Quoted 43 lines · Cleber Rosa

Quoted 39 lines · Philippe Mathieu-Daudé

Quoted 20 lines · Cleber Rosa

Some targets require a machine type to be set, as there's no default (aarch64 is one example). To give a consistent interface to users of this API, this changes set_machine() so that a predefined default can be used, if one is not given. The approach used is exactly the same with the console device type.

Also, even when there's a default machine type, for some purposes, testing included, it's better if outside code is explicit about the machine type, instead of relying on whatever is set internally.

Signed-off-by Cleber Rosa <[email protected]>

---
 scripts/qemu.py | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

scripts/qemu.py +0 -0

index d9e24a0c1a..fca9b76990 100644

scripts/qemu.py CONSOLE_DEV_TYPES = { · line 45

Quoted 10 lines

scripts/qemu.py +7 -0

@@ -36,6 +36,15 @@ CONSOLE_DEV_TYPES = {
     r'^s390-ccw-virtio.*': 'sclpconsole',
     }
 
+#: Maps archictures to the preferred machine type
+MACHINE_TYPES = {
+    r'^aarch64$': 'virt',
+    r'^ppc$': 'g3beige',
+    r'^ppc64$': 'pseries',
+    r'^s390x$': 's390-ccw-virtio',
+    r'^x86_64$': 'q35',

Why choose Q35 rather than PC (the default)?

I was wondering about how to generate variants/machines.json but this is definitively something we want to do via a QMP query.

Eduardo what do you think?

Quoted 8 lines

Quoted 2 lines

It was motivated by Eduardo's initiative to make q35 the default "across the board". He can confirm and give more details.

Making Q35 the default on applications using QEMU and libvirt is something I'd like to happen. But I think the simplest way to do that is to change the QEMU default. This way you won't need this table on qemu.py: you can just use the default provided by QEMU.

The idea is to bring consistency on how we're calling "qemu-system-$(ARCH)", and at the same time apply the "explicit is better than implicit" rule.

The most important fact is that some targets do not (currently) have "the default provided by QEMU", aarch64 is one of them.

- Cleber.

Quoted 21 lines

Quoted 6 lines

Quoted 2 lines

So I ended up not relaying the question properly: should we default (even if explicitly adding "-machine") to "pc"?

I think using the default machine-type (when QEMU has a default) would be less surprising for users of the qemu.py API.

OK, agreed.

Quoted 11 lines

Implicitly adding -machine when there's no default is also surprising, but then it's a nice surprise: instead of crashing you get a running VM.

Now, there are two other questions related to this:

If using 'pc' as default, should we always add -machine, or just omit the machine-type name? I think we should omit it unless the caller asked for a specific machine-type name (because it would be less surprising for users of the API).

Getting down to business, trying to apply those changes, I was faced with a situation. Actually, the same situation I faced a few months ago. Handling it was defered until it was *really* a blocker. Basically the issue is: the set_console() method, which gives tests a ready to use console, depends on knowing the machine type (see CONSOLE_DEV_TYPES).

As a case study, let's look at "boot_console_linux.py":
1) it sets the machine type explicitly
2) it has nothing to do with the specific machine type
3) the setting of a machine type is boiler plate code to set a console
4) the console is used on the test's real purpose: verifying the Linux
kernel booted

Now, to be able to run the same test -- booting a Linux kernel -- on *other target archs*, we need the same machinery. Even more important:
to have similar tests we'll need to either abstract those features or duplicate them. This can be seen, at least in part, on the firmware tests that Philippe sent to the list: they would also benefit from having a console device ready to be used on the configured machine type[1]:

Assuming that we want to provide this type of machinery for free (or as close as that) to the acceptance/functional tests, we need some source of "known good" configuration for the targets we aim to support.

Let's restrict the discussion to the issue at hand, machine types, while keeping in mind that the same pattern happened with devices types to use as console before, and my experience running into default network device types in further work (tests that interact with the guest by ssh'ing into it).

The solutions I can think of are:

1) run the target binary previous to the "real" run, and query
information -- this is what Avocado-VT does[2], and something I tried on earlier versions of the acceptance tests infrastructure code

2) attempt to get this information from the build system[3]

Quoted 9 lines

Quoted 5 lines

3) hard code the "known" good configuration

I've previously worked on solutions along the lines of #1 and #2, but the general feedback wasn't that positive, for valid reasons. Eduardo probably remembers this.

I don't remember seeing negative feedback for #1. It sounds like the best solution.

IIRC, it was mostly related to the fact that the reliable way to query a target information (instead of parsing a human oriented output) would be to use QMP.

Then, the question of using QEMUMachine for that (and the possible chicken-and-egg situations, having a different set of base args, etc) led me into prototyping a simplified version:

https://github.com/clebergnu/qemu/commit/b769b3d969b9e29c644d12c30cd0bb7f312e4fbc#diff-07b9b989eb85128c2c8d2a8f17685698R46

Which would still be duplicating some code. I'm not particularly happy about either approaches TBH.

Quoted 5 lines

Quoted 4 lines

So, I'm tempted to try solution #3. As much as duplicating target defaults in qemu.py doesn't sound perfect, it seems to be the more predictable and attainable solution at this point.

Quoted 4 lines

I consider #3 to be acceptable just as a temporary solution until we implement #1.

So, with the extra information given here, would you recommend doing #3? Or pause this, and work on a #1 of sorts?

One of the Avocado features relevant to virtualization testing is the ability to reuse tests in different scenarios, known as variants. This adds a JSON based variants file, that can be used to run most tests in a number of different architectures. It can be run with:

   $ avocado run \
     --json-variants-load=tests/acceptance/variants/arch.json \
     --filter-by-tags='-x86_64' -- tests/acceptance/

Currently this covers 5 architectures, resulting in the execution of 25 different combinations.

Signed-off-by Cleber Rosa <[email protected]>

---
 tests/acceptance/variants/arch.json | 92 +++++++++++++++++++++++++++++
 1 file changed, 92 insertions(+)
 create mode 100644 tests/acceptance/variants/arch.json

tests/acceptance/variants/arch.json new file +92 -0

new file mode 100644
index 0000000000..64ff6ad278
@@ -0,0 +1,92 @@
+[
+    {
+        "paths": [
+            "/run/*"
+        ],
+        "variant": [
+            [
+                "/run/aarch64",
+                [
+                    [
+                        "/run/aarch64",
+                        "arch",
+                        "aarch64"
+                    ]
+                ]
+            ]
+        ],
+        "variant_id": "aarch64"
+    },
+    {
+        "paths": [
+            "/run/*"
+        ],
+        "variant": [
+            [
+                "/run/ppc",
+                [
+                    [
+                        "/run/ppc",
+                        "arch",
+                        "ppc"
+                    ]
+                ]
+            ]
+        ],
+        "variant_id": "ppc"
+    },
+    {
+        "paths": [
+            "/run/*"
+        ],
+        "variant": [
+            [
+                "/run/ppc64",
+                [
+                    [
+                        "/run/ppc64",
+                        "arch",
+                        "ppc64"
+                    ]
+                ]
+            ]
+        ],
+        "variant_id": "ppc64"
+    },
+    {
+        "paths": [
+            "/run/*"
+        ],
+        "variant": [
+            [
+                "/run/s390x",
+                [
+                    [
+                        "/run/s390x",
+                        "arch",
+                        "s390x"
+                    ]
+                ]
+            ]
+        ],
+        "variant_id": "s390x"
+    },
+    {
+        "paths": [
+            "/run/*"
+        ],
+        "variant": [
+            [
+                "/run/x86_64",
+                [
+                    [
+                        "/run/x86_64",
+                        "arch",
+                        "x86_64"
+                    ]
+                ]
+            ]
+        ],
+        "variant_id": "x86_64"
+    }
+]

Hi Cleber,

Quoted 5 lines · Cleber Rosa

One of the Avocado features relevant to virtualization testing is the ability to reuse tests in different scenarios, known as variants. This adds a JSON based variants file, that can be used to run most tests in a number of different architectures. It can be run with:

Quoted 6 lines

   $ avocado run \
     --json-variants-load=tests/acceptance/variants/arch.json \
     --filter-by-tags='-x86_64' -- tests/acceptance/

Currently this covers 5 architectures, resulting in the execution of 25 different combinations.

Can you provide here the command you used to generate this JSON?

Thanks,

Phil.

Quoted 106 lines

Signed-off-by Cleber Rosa <[email protected]>

---
 tests/acceptance/variants/arch.json | 92 +++++++++++++++++++++++++++++
 1 file changed, 92 insertions(+)
 create mode 100644 tests/acceptance/variants/arch.json

tests/acceptance/variants/arch.json new file +92 -0

new file mode 100644
index 0000000000..64ff6ad278
@@ -0,0 +1,92 @@
+[
+    {
+        "paths": [
+            "/run/*"
+        ],
+        "variant": [
+            [
+                "/run/aarch64",
+                [
+                    [
+                        "/run/aarch64",
+                        "arch",
+                        "aarch64"
+                    ]
+                ]
+            ]
+        ],
+        "variant_id": "aarch64"
+    },
+    {
+        "paths": [
+            "/run/*"
+        ],
+        "variant": [
+            [
+                "/run/ppc",
+                [
+                    [
+                        "/run/ppc",
+                        "arch",
+                        "ppc"
+                    ]
+                ]
+            ]
+        ],
+        "variant_id": "ppc"
+    },
+    {
+        "paths": [
+            "/run/*"
+        ],
+        "variant": [
+            [
+                "/run/ppc64",
+                [
+                    [
+                        "/run/ppc64",
+                        "arch",
+                        "ppc64"
+                    ]
+                ]
+            ]
+        ],
+        "variant_id": "ppc64"
+    },
+    {
+        "paths": [
+            "/run/*"
+        ],
+        "variant": [
+            [
+                "/run/s390x",
+                [
+                    [
+                        "/run/s390x",
+                        "arch",
+                        "s390x"
+                    ]
+                ]
+            ]
+        ],
+        "variant_id": "s390x"
+    },
+    {
+        "paths": [
+            "/run/*"
+        ],
+        "variant": [
+            [
+                "/run/x86_64",
+                [
+                    [
+                        "/run/x86_64",
+                        "arch",
+                        "x86_64"
+                    ]
+                ]
+            ]
+        ],
+        "variant_id": "x86_64"
+    }
+]

One of the Avocado features relevant to virtualization testing is the ability to reuse tests in different scenarios, known as variants. This adds a JSON based variants file, that can be used to run most tests in a number of different architectures. It can be run with:

   $ avocado run \
     --json-variants-load=tests/acceptance/variants/arch.json \
     --filter-by-tags='-x86_64' -- tests/acceptance/

Currently this covers 5 architectures, resulting in the execution of 25 different combinations.

Signed-off-by Cleber Rosa <[email protected]>

---
 tests/acceptance/variants/arch.json | 92 +++++++++++++++++++++++++++++
 1 file changed, 92 insertions(+)
 create mode 100644 tests/acceptance/variants/arch.json

Quoted 9 lines · Cleber Rosa

One of the Avocado features relevant to virtualization testing is the ability to reuse tests in different scenarios, known as variants. This adds a JSON based variants file, that can be used to run most tests in a number of different architectures. It can be run with:

   $ avocado run \
     --json-variants-load=tests/acceptance/variants/arch.json \
     --filter-by-tags='-x86_64' -- tests/acceptance/

Quoted 2 lines

Currently this covers 5 architectures, resulting in the execution of 25 different combinations.

IMHO we should add this file with all supported archs, and CANCEL a job if there is no preferred machine in Qemu::MACHINE_TYPES.

Quoted 6 lines

Signed-off-by Cleber Rosa <[email protected]>

---
 tests/acceptance/variants/arch.json | 92 +++++++++++++++++++++++++++++
 1 file changed, 92 insertions(+)
 create mode 100644 tests/acceptance/variants/arch.json

With the introduction of a variants file that can run the same tests on various architectures, it makes sense to make most tests to be reusable on those environments. The exception should be when a test is really testing a specific architecture feature.

With the change proposed here, on a command line such as:

  $ avocado run \
     --json-variants-load=tests/acceptance/variants/arch.json \
     -- tests/acceptance/

The boot_linux_console.py tests will appear as "CANCELED: Currently specific to the x86_64 arch", which is as a good thing when compared to being ignored by tags because:

* The architecture specific parts can be addressed
* It will be run on the matching architecture (as opposed to always being filtered out by the tags mechanism)
* CANCELED tests do no influence negatively the overall job results, they're not considered an error or failure

Signed-off-by Cleber Rosa <[email protected]>

---
 tests/acceptance/boot_linux_console.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

tests/acceptance/boot_linux_console.py +0 -1

index 58032f971c..ba3ac036da 100644
@@ -19,12 +19,13 @@ class BootLinuxConsole(Test):
     and the kernel command line is properly passed from QEMU to the kernel
 
     :avocado: enable
-    :avocado: tags=x86_64

Quoted 36 lines · Cleber Rosa

With the introduction of a variants file that can run the same tests on various architectures, it makes sense to make most tests to be reusable on those environments. The exception should be when a test is really testing a specific architecture feature.

With the change proposed here, on a command line such as:

  $ avocado run \
     --json-variants-load=tests/acceptance/variants/arch.json \
     -- tests/acceptance/

The boot_linux_console.py tests will appear as "CANCELED: Currently specific to the x86_64 arch", which is as a good thing when compared to being ignored by tags because:

* The architecture specific parts can be addressed
* It will be run on the matching architecture (as opposed to always being filtered out by the tags mechanism)
* CANCELED tests do no influence negatively the overall job results, they're not considered an error or failure

Signed-off-by Cleber Rosa <[email protected]>

---
 tests/acceptance/boot_linux_console.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

tests/acceptance/boot_linux_console.py +0 -1

index 58032f971c..ba3ac036da 100644
@@ -19,12 +19,13 @@ class BootLinuxConsole(Test):
     and the kernel command line is properly passed from QEMU to the kernel
 
     :avocado: enable
-    :avocado: tags=x86_64

tests/acceptance/boot_linux_console.py class BootLinuxConsole(Test): · line 31

Quoted 10 lines

tests/acceptance/boot_linux_console.py +2 -0

     """
 
     timeout = 60
 
     def test(self):
+        if self.arch != 'x86_64':
+            self.cancel('Currently specific to the x86_64 target arch')
         kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/'
                       'Everything/x86_64/os/images/pxeboot/vmlinuz')
         kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a'

For some reason this test run quicker on a aarch64 host than my x86 laptop:

ThunderX 88XX (aarch64 Little Endian):

(05/30)
tests/acceptance/boot_linux_console.py:BootLinuxConsole.test;x86_64:
PASS (12.88 s)

Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz:

(05/30)
tests/acceptance/boot_linux_console.py:BootLinuxConsole.test;x86_64:
PASS (31.13 s)

Reviewed-by Philippe Mathieu-Daudé <[email protected]>

Tested-by Philippe Mathieu-Daudé <[email protected]>

With the introduction of a variants file that can run the same tests on various architectures, it makes sense to make most tests to be reusable on those environments. The exception should be when a test is really testing a specific architecture feature.

With the change proposed here, on a command line such as:

  $ avocado run \
     --json-variants-load=tests/acceptance/variants/arch.json \
     -- tests/acceptance/

The boot_linux_console.py tests will appear as "CANCELED: Currently specific to the x86_64 arch", which is as a good thing when compared to being ignored by tags because:

* The architecture specific parts can be addressed
* It will be run on the matching architecture (as opposed to always being filtered out by the tags mechanism)
* CANCELED tests do no influence negatively the overall job results, they're not considered an error or failure

Signed-off-by Cleber Rosa <[email protected]>

---
 tests/acceptance/boot_linux_console.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

tests/acceptance/boot_linux_console.py +0 -0

index 58032f971c..ba3ac036da 100644

tests/acceptance/boot_linux_console.py +0 -0

         kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/'
                       'Everything/x86_64/os/images/pxeboot/vmlinuz')
         kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a'

Hi Cleber,

Quoted 33 lines · Cleber Rosa

With the introduction of a variants file that can run the same tests on various architectures, it makes sense to make most tests to be reusable on those environments. The exception should be when a test is really testing a specific architecture feature.

With the change proposed here, on a command line such as:

  $ avocado run \
     --json-variants-load=tests/acceptance/variants/arch.json \
     -- tests/acceptance/

The boot_linux_console.py tests will appear as "CANCELED: Currently specific to the x86_64 arch", which is as a good thing when compared to being ignored by tags because:

* The architecture specific parts can be addressed
* It will be run on the matching architecture (as opposed to always being filtered out by the tags mechanism)
* CANCELED tests do no influence negatively the overall job results, they're not considered an error or failure

Signed-off-by Cleber Rosa <[email protected]>

---
 tests/acceptance/boot_linux_console.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

tests/acceptance/boot_linux_console.py +0 -0

index 58032f971c..ba3ac036da 100644
@@ -19,12 +19,13 @@ class BootLinuxConsole(Test):
     and the kernel command line is properly passed from QEMU to the kernel
 

tests/acceptance/boot_linux_console.py class BootLinuxConsole(Test): · line 22

Quoted 2 lines

tests/acceptance/boot_linux_console.py +0 -1

     :avocado: enable
-    :avocado: tags=x86_64

Can we keep a such tag (in format 'arch:x86_64') ...

tests/acceptance/boot_linux_console.py class BootLinuxConsole(Test): · line 28

Quoted 7 lines

tests/acceptance/boot_linux_console.py +2 -0

     """
 
     timeout = 60
 
     def test(self):
+        if self.arch != 'x86_64':
+            self.cancel('Currently specific to the x86_64 target arch')

... and have this check generic? (Eventually via a QemuTest(Test) parent class).

Quoted 4 lines

tests/acceptance/boot_linux_console.py +0 -0

         kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/'
                       'Everything/x86_64/os/images/pxeboot/vmlinuz')
         kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a'