Skip to contentSkip to conversation

tests/avocado: Pass parameters via Makefile

PATCH In review Fabiano Rosas 2 participants 5 messages Jan 20, 2023 – Feb 2, 2023

lore
Share & tools
lore
#0 lore raw

cover tests/avocado: Pass parameters via Makefile

This is intended to replace the last two patches of Daniel's series:
https://lore.kernel.org/r/[email protected]

Currently, the initialization code in setUp() infers properties of the tests & host/target machine by looking at the avocado tags present in the test. If there are no tags some best-effort fallbacks are chosen.

This means that for generic tests they end up always choosing the QEMU target that matches the host architecture, which is not always desirable. If we allow command line parameters to override the fallback, we can alter this behavior.

patch 1 - change precedence to tags > params to avoid changing tests

          that do have tags;

patch 2 - wires up the new AVOCADO_PARAMS variable to the '-p' avocado

          command line option;

Fabiano Rosas (2):
tests/avocado: Invert parameter vs. tag precedence during setUp tests/avocado: Allow passing command line parameters via Makefile

 tests/Makefile.include                 |  6 ++++-
 tests/avocado/avocado_qemu/__init__.py | 32 +++++++++++++++-----------
 2 files changed, 24 insertions(+), 14 deletions(-)
Signature · 1 line
2.35.3
#1 lore raw

PATCH 1/2 tests/avocado: Invert parameter vs. tag precedence during setUp

We currently never pass parameters to the avocado process via Makefile. To start doing so we need to invert the precedence between command line parameters and tags, otherwise a command line parameter would override values for all the tests, which is unlikely to be a common use-case.

A more likely use-case is to force certain values for the tests that have no tags. For instance, if a test has no 'arch' tags and therefore can run for all targets, one could possibly force it to run on a certain target with an arch=foo parameter.

This applies to the variables set during setUp(): arch, machine, cpu, distro_name, distro_version. Parameters used directly in tests or read via self.params.get are left unchanged.

---
 tests/avocado/avocado_qemu/__init__.py | 32 +++++++++++++++-----------
 1 file changed, 19 insertions(+), 13 deletions(-)
tests/avocado/avocado_qemu/__init__.py +19 -13
index 910f3ba1ea..a181cac383 100644
@@ -240,12 +240,23 @@ def _get_unique_tag_val(self, tag_name):
             return vals.pop()
         return None
 
+    def _get_prop(self, name):
+        """
+        Infer test properties based on tags. If no tag is present,
+        look for a command line parameter of the same name.
+        """
+        val = self._get_unique_tag_val(name)
+        if not val:
+            # If there's no tag, look for a command line
+            # parameter. This allows the user to override any defaults
+            # the caller of this function would choose if we were to
+            # return None.
+            val = self.params.get(name)
+        return val
+
     def setUp(self, bin_prefix):
-        self.arch = self.params.get('arch',
-                                    default=self._get_unique_tag_val('arch'))
-
-        self.cpu = self.params.get('cpu',
-                                   default=self._get_unique_tag_val('cpu'))
+        self.arch = self._get_prop('arch')
+        self.cpu = self._get_prop('cpu')
 
         default_qemu_bin = pick_default_qemu_bin(bin_prefix, arch=self.arch)
         self.qemu_bin = self.params.get('qemu_bin',
@@ -274,8 +285,7 @@ def setUp(self):
 
         super().setUp('qemu-system-')
 
-        self.machine = self.params.get('machine',
-                                       default=self._get_unique_tag_val('machine'))
+        self.machine = self._get_prop('machine')
 
     def require_accelerator(self, accelerator):
         """
@@ -529,15 +539,11 @@ class LinuxTest(LinuxSSHMixIn, QemuSystemTest):
     memory = '1024'
 
     def _set_distro(self):
-        distro_name = self.params.get(
-            'distro',
-            default=self._get_unique_tag_val('distro'))
+        distro_name = self._get_prop('distro')
         if not distro_name:
             distro_name = 'fedora'
 
-        distro_version = self.params.get(
-            'distro_version',
-            default=self._get_unique_tag_val('distro_version'))
+        distro_version = self._get_prop('distro_version')
         if not distro_version:
             distro_version = '31'
 
Signature · 1 line
2.35.3
Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] tests/avocado: Invert parameter vs. tag precedence during setUp

Quoted 11 lines · Fabiano Rosas

We currently never pass parameters to the avocado process via Makefile. To start doing so we need to invert the precedence between command line parameters and tags, otherwise a command line parameter would override values for all the tests, which is unlikely to be a common use-case.

A more likely use-case is to force certain values for the tests that have no tags. For instance, if a test has no 'arch' tags and therefore can run for all targets, one could possibly force it to run on a certain target with an arch=foo parameter.

Quoted 6 lines

This applies to the variables set during setUp(): arch, machine, cpu, distro_name, distro_version. Parameters used directly in tests or read via self.params.get are left unchanged.

---

Quoted 64 lines
  tests/avocado/avocado_qemu/__init__.py | 32 +++++++++++++++-----------
  1 file changed, 19 insertions(+), 13 deletions(-)
tests/avocado/avocado_qemu/__init__.py +19 -13
index 910f3ba1ea..a181cac383 100644
@@ -240,12 +240,23 @@ def _get_unique_tag_val(self, tag_name):
              return vals.pop()
          return None
 
+    def _get_prop(self, name):
+        """
+        Infer test properties based on tags. If no tag is present,
+        look for a command line parameter of the same name.
+        """
+        val = self._get_unique_tag_val(name)
+        if not val:
+            # If there's no tag, look for a command line
+            # parameter. This allows the user to override any defaults
+            # the caller of this function would choose if we were to
+            # return None.
+            val = self.params.get(name)
+        return val
+
      def setUp(self, bin_prefix):
-        self.arch = self.params.get('arch',
-                                    default=self._get_unique_tag_val('arch'))
-
-        self.cpu = self.params.get('cpu',
-                                   default=self._get_unique_tag_val('cpu'))
+        self.arch = self._get_prop('arch')
+        self.cpu = self._get_prop('cpu')
 
          default_qemu_bin = pick_default_qemu_bin(bin_prefix, arch=self.arch)
          self.qemu_bin = self.params.get('qemu_bin',
@@ -274,8 +285,7 @@ def setUp(self):
 
          super().setUp('qemu-system-')
 
-        self.machine = self.params.get('machine',
-                                       default=self._get_unique_tag_val('machine'))
+        self.machine = self._get_prop('machine')
 
      def require_accelerator(self, accelerator):
          """
@@ -529,15 +539,11 @@ class LinuxTest(LinuxSSHMixIn, QemuSystemTest):
      memory = '1024'
 
      def _set_distro(self):
-        distro_name = self.params.get(
-            'distro',
-            default=self._get_unique_tag_val('distro'))
+        distro_name = self._get_prop('distro')
          if not distro_name:
              distro_name = 'fedora'
 
-        distro_version = self.params.get(
-            'distro_version',
-            default=self._get_unique_tag_val('distro_version'))
+        distro_version = self._get_prop('distro_version')
          if not distro_version:
              distro_version = '31'
 
#3 lore raw

PATCH 2/2 tests/avocado: Allow passing command line parameters via Makefile

In reply to Fabiano Rosas

Add support for the 'avocado run' "-p" option, which allows us to pass parameters in the form key=value to be applied to all tests selected for a given run. This is useful to force generic tests to use a specific machine, cpu or qemu-binary where otherwise the defaults would be used.

E.g.:

 $ make check-avocado AVOCADO_PARAMS="machine=virt arch=riscv64"

<runs all tests replacing machine and arch in those that do not have
these parameters already set via tags>

---
 tests/Makefile.include | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
tests/Makefile.include +5 -1
index 9422ddaece..f92e730aa0 100644
@@ -107,6 +107,10 @@ else
 	AVOCADO_CMDLINE_TAGS=$(addprefix -t , $(AVOCADO_TAGS))
 endif
 
+ifdef AVOCADO_PARAMS
+	AVOCADO_CMDLINE_PARAMS=$(addprefix -p , $(AVOCADO_PARAMS))
+endif
+
 quiet-venv-pip = $(quiet-@)$(call quiet-command-run, \
     $(TESTS_PYTHON) -m pip -q --disable-pip-version-check $1, \
     "VENVPIP","$1")
@@ -144,7 +148,7 @@ check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
             --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
             $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
 			--filter-by-tags-include-empty-key) \
-            $(AVOCADO_CMDLINE_TAGS) \
+            $(AVOCADO_CMDLINE_TAGS) $(AVOCADO_CMDLINE_PARAMS) \
             $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
             "AVOCADO", "tests/avocado")
 
Signature · 1 line
2.35.3
Reviewed-by on v1 2/2
#4 lore raw

Re: [PATCH 2/2] tests/avocado: Allow passing command line parameters via Makefile

Quoted 6 lines · Fabiano Rosas

Add support for the 'avocado run' "-p" option, which allows us to pass parameters in the form key=value to be applied to all tests selected for a given run. This is useful to force generic tests to use a specific machine, cpu or qemu-binary where otherwise the defaults would be used.

Quoted 8 lines

E.g.:

  $ make check-avocado AVOCADO_PARAMS="machine=virt arch=riscv64"

<runs all tests replacing machine and arch in those that do not have
these parameters already set via tags>

---

Quoted 27 lines
  tests/Makefile.include | 6 +++++-
  1 file changed, 5 insertions(+), 1 deletion(-)
tests/Makefile.include +5 -1
index 9422ddaece..f92e730aa0 100644
@@ -107,6 +107,10 @@ else
  	AVOCADO_CMDLINE_TAGS=$(addprefix -t , $(AVOCADO_TAGS))
  endif
 
+ifdef AVOCADO_PARAMS
+	AVOCADO_CMDLINE_PARAMS=$(addprefix -p , $(AVOCADO_PARAMS))
+endif
+
  quiet-venv-pip = $(quiet-@)$(call quiet-command-run, \
      $(TESTS_PYTHON) -m pip -q --disable-pip-version-check $1, \
      "VENVPIP","$1")
@@ -144,7 +148,7 @@ check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
              --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
              $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
  			--filter-by-tags-include-empty-key) \
-            $(AVOCADO_CMDLINE_TAGS) \
+            $(AVOCADO_CMDLINE_TAGS) $(AVOCADO_CMDLINE_PARAMS) \
              $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
              "AVOCADO", "tests/avocado")