tests/avocado: Pass parameters via Makefile
cover tests/avocado: Pass parameters via Makefile
This is intended to replace the last two patches of Daniel's series:
https://lore.kernel.org/r/[email protected]
Currently, the initialization code in setUp() infers properties of the tests & host/target machine by looking at the avocado tags present in the test. If there are no tags some best-effort fallbacks are chosen.
This means that for generic tests they end up always choosing the QEMU target that matches the host architecture, which is not always desirable. If we allow command line parameters to override the fallback, we can alter this behavior.
patch 1 - change precedence to tags > params to avoid changing tests
that do have tags;
patch 2 - wires up the new AVOCADO_PARAMS variable to the '-p' avocado
command line option;
Fabiano Rosas (2):
tests/avocado: Invert parameter vs. tag precedence during setUp tests/avocado: Allow passing command line parameters via Makefile
tests/Makefile.include | 6 ++++- tests/avocado/avocado_qemu/__init__.py | 32 +++++++++++++++----------- 2 files changed, 24 insertions(+), 14 deletions(-)
Signature · 1 line
2.35.3
PATCH 1/2 tests/avocado: Invert parameter vs. tag precedence during setUp
We currently never pass parameters to the avocado process via Makefile. To start doing so we need to invert the precedence between command line parameters and tags, otherwise a command line parameter would override values for all the tests, which is unlikely to be a common use-case.
A more likely use-case is to force certain values for the tests that have no tags. For instance, if a test has no 'arch' tags and therefore can run for all targets, one could possibly force it to run on a certain target with an arch=foo parameter.
This applies to the variables set during setUp(): arch, machine, cpu, distro_name, distro_version. Parameters used directly in tests or read via self.params.get are left unchanged.
--- tests/avocado/avocado_qemu/__init__.py | 32 +++++++++++++++----------- 1 file changed, 19 insertions(+), 13 deletions(-)
tests/avocado/avocado_qemu/__init__.py +19 -13
@@ -240,12 +240,23 @@ def _get_unique_tag_val(self, tag_name): return vals.pop() return None + def _get_prop(self, name): + """ + Infer test properties based on tags. If no tag is present, + look for a command line parameter of the same name. + """ + val = self._get_unique_tag_val(name) + if not val: + # If there's no tag, look for a command line + # parameter. This allows the user to override any defaults + # the caller of this function would choose if we were to + # return None. + val = self.params.get(name) + return val + def setUp(self, bin_prefix): - self.arch = self.params.get('arch', - default=self._get_unique_tag_val('arch')) - - self.cpu = self.params.get('cpu', - default=self._get_unique_tag_val('cpu')) + self.arch = self._get_prop('arch') + self.cpu = self._get_prop('cpu') default_qemu_bin = pick_default_qemu_bin(bin_prefix, arch=self.arch) self.qemu_bin = self.params.get('qemu_bin',
@@ -274,8 +285,7 @@ def setUp(self): super().setUp('qemu-system-') - self.machine = self.params.get('machine', - default=self._get_unique_tag_val('machine')) + self.machine = self._get_prop('machine') def require_accelerator(self, accelerator): """
@@ -529,15 +539,11 @@ class LinuxTest(LinuxSSHMixIn, QemuSystemTest): memory = '1024' def _set_distro(self): - distro_name = self.params.get( - 'distro', - default=self._get_unique_tag_val('distro')) + distro_name = self._get_prop('distro') if not distro_name: distro_name = 'fedora' - distro_version = self.params.get( - 'distro_version', - default=self._get_unique_tag_val('distro_version')) + distro_version = self._get_prop('distro_version') if not distro_version: distro_version = '31'
Signature · 1 line
2.35.3
Re: [PATCH 1/2] tests/avocado: Invert parameter vs. tag precedence during setUp
Quoted 11 lines · Fabiano Rosas
We currently never pass parameters to the avocado process via
Makefile. To start doing so we need to invert the precedence between command line parameters and tags, otherwise a command line parameter would override values for all the tests, which is unlikely to be a common use-case.A more likely use-case is to force certain values for the tests that have no tags. For instance, if a test has no 'arch' tags and therefore can run for all targets, one could possibly force it to run on a certain target with an arch=foo parameter.
Quoted 6 lines
This applies to the variables set during
setUp(): arch, machine, cpu,distro_name,distro_version. Parameters used directly in tests or read via self.params.get are left unchanged.Signed-off-by Fabiano Rosas <[email protected]>---
Quoted 64 lines
tests/avocado/avocado_qemu/__init__.py | 32 +++++++++++++++----------- 1 file changed, 19 insertions(+), 13 deletions(-)tests/avocado/avocado_qemu/__init__.py +19
-13@@ -240,12 +240,23 @@ def _get_unique_tag_val(self, tag_name): return vals.pop() return None + def _get_prop(self, name): + """ + Infer test properties based on tags. If no tag is present, + look for a command line parameter of the same name. + """ + val = self._get_unique_tag_val(name) + if not val: + # If there's no tag, look for a command line + # parameter. This allows the user to override any defaults + # the caller of this function would choose if we were to + # return None. + val = self.params.get(name) + return val + def setUp(self, bin_prefix): - self.arch = self.params.get('arch', - default=self._get_unique_tag_val('arch')) - - self.cpu = self.params.get('cpu', - default=self._get_unique_tag_val('cpu')) + self.arch = self._get_prop('arch') + self.cpu = self._get_prop('cpu') default_qemu_bin = pick_default_qemu_bin(bin_prefix, arch=self.arch) self.qemu_bin = self.params.get('qemu_bin',@@ -274,8 +285,7 @@ def setUp(self): super().setUp('qemu-system-') - self.machine = self.params.get('machine', - default=self._get_unique_tag_val('machine')) + self.machine = self._get_prop('machine') def require_accelerator(self, accelerator): """@@ -529,15 +539,11 @@ class LinuxTest(LinuxSSHMixIn, QemuSystemTest): memory = '1024' def _set_distro(self): - distro_name = self.params.get( - 'distro', - default=self._get_unique_tag_val('distro')) + distro_name = self._get_prop('distro') if not distro_name: distro_name = 'fedora' - distro_version = self.params.get( - 'distro_version', - default=self._get_unique_tag_val('distro_version')) + distro_version = self._get_prop('distro_version') if not distro_version: distro_version = '31'
PATCH 2/2 tests/avocado: Allow passing command line parameters via Makefile
In reply to Fabiano RosasAdd support for the 'avocado run' "-p" option, which allows us to pass parameters in the form key=value to be applied to all tests selected for a given run. This is useful to force generic tests to use a specific machine, cpu or qemu-binary where otherwise the defaults would be used.
E.g.:
$ make check-avocado AVOCADO_PARAMS="machine=virt arch=riscv64"
<runs all tests replacing machine and arch in those that do not have
these parameters already set via tags>
--- tests/Makefile.include | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
tests/Makefile.include +5 -1
@@ -107,6 +107,10 @@ else AVOCADO_CMDLINE_TAGS=$(addprefix -t , $(AVOCADO_TAGS)) endif +ifdef AVOCADO_PARAMS + AVOCADO_CMDLINE_PARAMS=$(addprefix -p , $(AVOCADO_PARAMS)) +endif + quiet-venv-pip = $(quiet-@)$(call quiet-command-run, \ $(TESTS_PYTHON) -m pip -q --disable-pip-version-check $1, \ "VENVPIP","$1")
@@ -144,7 +148,7 @@ check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \ --filter-by-tags-include-empty-key) \ - $(AVOCADO_CMDLINE_TAGS) \ + $(AVOCADO_CMDLINE_TAGS) $(AVOCADO_CMDLINE_PARAMS) \ $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \ "AVOCADO", "tests/avocado")
Signature · 1 line
2.35.3
Re: [PATCH 2/2] tests/avocado: Allow passing command line parameters via Makefile
Quoted 6 lines · Fabiano Rosas
Add support for the 'avocado run' "-p" option, which allows us to pass parameters in the form key=value to be applied to all tests selected for a given run. This is useful to force generic tests to use a specific machine, cpu or qemu-binary where otherwise the defaults would be used.
Quoted 8 lines
E.g.:
$ make check-avocado AVOCADO_PARAMS="machine=virt arch=riscv64"<runs all tests replacing machine and arch in those that do not have
these parameters already set via tags>Signed-off-by Fabiano Rosas <[email protected]>---
Quoted 27 lines
tests/Makefile.include | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)tests/Makefile.include +5
-1@@ -107,6 +107,10 @@ else AVOCADO_CMDLINE_TAGS=$(addprefix -t , $(AVOCADO_TAGS)) endif +ifdef AVOCADO_PARAMS + AVOCADO_CMDLINE_PARAMS=$(addprefix -p , $(AVOCADO_PARAMS)) +endif + quiet-venv-pip = $(quiet-@)$(call quiet-command-run, \ $(TESTS_PYTHON) -m pip -q --disable-pip-version-check $1, \ "VENVPIP","$1")@@ -144,7 +148,7 @@ check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \ --filter-by-tags-include-empty-key) \ - $(AVOCADO_CMDLINE_TAGS) \ + $(AVOCADO_CMDLINE_TAGS) $(AVOCADO_CMDLINE_PARAMS) \ $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \ "AVOCADO", "tests/avocado")