avocado-qemu: New SMMUv3 tests
cover avocado-qemu: New SMMUv3 tests
This patch adds a first set of SMMU functional tests using a Fedora cloud-init image. Given the kernel in use, range invalidation is not tested yet. However different guest kernel configurations are tested: standard, strict=0 and passthrough mode.
The patch applies on top of Cleber's series:
PATCH v2 00/10] Acceptance Test: introduce base class for Linux based tests.
Special thanks to Cleber for his support on this first trial.
Best Regards
Eric
Eric Auger (1):
avocado_qemu: Add SMMUv3 tests
tests/acceptance/smmu.py | 104 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 104 insertions(+) create mode 100644 tests/acceptance/smmu.py
Signature · 1 line
2.26.2
PATCH 1/1 avocado_qemu: Add SMMUv3 tests
Add new tests checking the good behavior of the SMMUv3 protecting 2 virtio pci devices (block and net). We check the guest boots and we are able to install a package. Different guest configs are tested:
standard, passthrough an strict=0. Given the version of the guest kernel in use (5.3.7 at this moment), range invalidation is not yet tested. This will be handled separately.
--- tests/acceptance/smmu.py | 104 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 104 insertions(+) create mode 100644 tests/acceptance/smmu.py
tests/acceptance/smmu.py new file +104 -0
@@ -0,0 +1,104 @@ +# SMMUv3 Functional tests +# +# Copyright (c) 2021 Red Hat, Inc. +# +# Author: +# Eric Auger <[email protected]> +# +# This work is licensed under the terms of the GNU GPL, version 2 or +# later. See the COPYING file in the top-level directory. + +import os + +from avocado_qemu import LinuxTest, BUILD_DIR +from avocado.utils import ssh + +class SMMU(LinuxTest): + + KERNEL_COMMON_PARAMS = ("root=UUID=b6950a44-9f3c-4076-a9c2-355e8475b0a7 ro " + "earlyprintk=pl011,0x9000000 ignore_loglevel " + "no_timer_check printk.time=1 rd_NO_PLYMOUTH " + "console=ttyAMA0 ") + IOMMU_ADDON = ',iommu_platform=on,disable-modern=off,disable-legacy=on' + IMAGE = ("https://archives.fedoraproject.org/pub/archive/fedora/" + "linux/releases/31/Everything/aarch64/os/images/pxeboot/") + kernel_path = None + initrd_path = None + kernel_params = None + + def set_up_boot(self): + path = self.download_boot() + self.vm.add_args('-device', 'virtio-blk-pci,bus=pcie.0,scsi=off,' + + 'drive=drv0,id=virtio-disk0,bootindex=1,' + 'werror=stop,rerror=stop' + self.IOMMU_ADDON) + self.vm.add_args('-drive', + 'file=%s,if=none,cache=writethrough,id=drv0' % path) + + def setUp(self): + super(SMMU, self).setUp(None, 'virtio-net-pci' + self.IOMMU_ADDON) + + def add_common_args(self): + self.vm.add_args("-machine", "virt") + self.vm.add_args('-bios', os.path.join(BUILD_DIR, 'pc-bios', + 'edk2-aarch64-code.fd')) + self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0') + self.vm.add_args('-object', + 'rng-random,id=rng0,filename=/dev/urandom') + + def common_vm_setup(self, custom_kernel=None): + self.require_accelerator("kvm") + self.add_common_args() + self.vm.add_args("-accel", "kvm") + self.vm.add_args("-cpu", "host") + self.vm.add_args("-machine", "iommu=smmuv3") + + if custom_kernel is None: + return + + kernel_url = self.IMAGE + 'vmlinuz' + initrd_url = self.IMAGE + 'initrd.img' + self.kernel_path = self.fetch_asset(kernel_url) + self.initrd_path = self.fetch_asset(initrd_url) + + def run_and_check(self): + if self.kernel_path: + self.vm.add_args('-kernel', self.kernel_path, + '-append', self.kernel_params, + '-initrd', self.initrd_path) + self.launch_and_wait() + self.ssh_command('cat /proc/cmdline') + self.ssh_command('dnf -y install numactl-devel') + + def test_smmu(self): + """ + :avocado: tags=accel:kvm + :avocado: tags=cpu:host + :avocado: tags=smmu + """ + + self.common_vm_setup() + self.run_and_check() + + def test_smmu_passthrough(self): + """ + :avocado: tags=accel:kvm + :avocado: tags=cpu:host + :avocado: tags=smmu + """ + self.common_vm_setup(True) + + self.kernel_params = self.KERNEL_COMMON_PARAMS + 'iommu.passthrough=on' + + self.run_and_check() + + def test_smmu_nostrict(self): + """ + :avocado: tags=accel:kvm + :avocado: tags=cpu:host + :avocado: tags=smmu + """ + self.common_vm_setup(True) + + self.kernel_params = self.KERNEL_COMMON_PARAMS + 'iommu.strict=0' + + self.run_and_check()
Signature · 1 line
2.26.2
Re: [PATCH 1/1] avocado_qemu: Add SMMUv3 tests
Quoted 23 lines · Eric Auger
Add new tests checking the good behavior of the SMMUv3 protecting 2 virtio pci devices (block and net). We check the guest boots and we are able to install a package. Different guest configs are tested:
standard, passthrough an strict=0. Given the version of the guest kernel in use (5.3.7 at this moment), range invalidation is not yet tested. This will be handled separately.Signed-off-by Eric Auger <[email protected]>--- tests/acceptance/smmu.py | 104 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 104 insertions(+) create mode 100644 tests/acceptance/smmu.pytests/acceptance/smmu.py new file +4
-0@@ -0,0 +1,104 @@ +# SMMUv3 Functional tests +# +# Copyright (c) 2021 Red Hat, Inc. +#
tests/acceptance/smmu.py line 14
Quoted 10 lines
tests/acceptance/smmu.py new file +10
-0+# Author: +# Eric Auger <[email protected]> +# +# This work is licensed under the terms of the GNU GPL, version 2 or +# later. See the COPYING file in the top-level directory. + +import os + +from avocado_qemu import LinuxTest, BUILD_DIR +from avocado.utils import ssh
This import is not needed, given that the you're not using them directly, but only using the LinuxTest methods that wrap them.
Quoted 26 lines
tests/acceptance/smmu.py new file +26
-0+ +class SMMU(LinuxTest): + + KERNEL_COMMON_PARAMS = ("root=UUID=b6950a44-9f3c-4076-a9c2-355e8475b0a7 ro " + "earlyprintk=pl011,0x9000000 ignore_loglevel " + "no_timer_check printk.time=1 rd_NO_PLYMOUTH " + "console=ttyAMA0 ") + IOMMU_ADDON = ',iommu_platform=on,disable-modern=off,disable-legacy=on' + IMAGE = ("https://archives.fedoraproject.org/pub/archive/fedora/" + "linux/releases/31/Everything/aarch64/os/images/pxeboot/") + kernel_path = None + initrd_path = None + kernel_params = None + + def set_up_boot(self): + path = self.download_boot() + self.vm.add_args('-device', 'virtio-blk-pci,bus=pcie.0,scsi=off,' + + 'drive=drv0,id=virtio-disk0,bootindex=1,' + 'werror=stop,rerror=stop' + self.IOMMU_ADDON) + self.vm.add_args('-drive', + 'file=%s,if=none,cache=writethrough,id=drv0' % path) + + def setUp(self): + super(SMMU, self).setUp(None, 'virtio-net-pci' + self.IOMMU_ADDON) + + def add_common_args(self):
tests/acceptance/smmu.py line 50
Quoted 10 lines
tests/acceptance/smmu.py new file +10
-0+ self.vm.add_args("-machine", "virt") + self.vm.add_args('-bios', os.path.join(BUILD_DIR, 'pc-bios', + 'edk2-aarch64-code.fd')) + self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0') + self.vm.add_args('-object', + 'rng-random,id=rng0,filename=/dev/urandom') + + def common_vm_setup(self, custom_kernel=None): + self.require_accelerator("kvm") + self.add_common_args()
I know you're following the previous test pattern/template, but maybe combine add_command_args() and common_vm_setup()? They seem to be doing the same thing.
Quoted 10 lines
tests/acceptance/smmu.py new file +10
-0+ self.vm.add_args("-accel", "kvm") + self.vm.add_args("-cpu", "host") + self.vm.add_args("-machine", "iommu=smmuv3") + + if custom_kernel is None: + return + + kernel_url = self.IMAGE + 'vmlinuz' + initrd_url = self.IMAGE + 'initrd.img' + self.kernel_path = self.fetch_asset(kernel_url)
tests/acceptance/smmu.py line 70
Quoted 10 lines
tests/acceptance/smmu.py new file +10
-0+ self.initrd_path = self.fetch_asset(initrd_url) + + def run_and_check(self): + if self.kernel_path: + self.vm.add_args('-kernel', self.kernel_path, + '-append', self.kernel_params, + '-initrd', self.initrd_path) + self.launch_and_wait() + self.ssh_command('cat /proc/cmdline') + self.ssh_command('dnf -y install numactl-devel')
Would you expect the package installation to cover significant more than, say, a package removal? Not relying on the distro's package repos (and external networking) would be an improvement to the test's stability, but I wonder how much functional coverage would be lost.
FIY, I've tried it with 'dnf -y remove yum' instead, and test times are also considerably lower.
tests/acceptance/smmu.py line 77
Quoted 7 lines
tests/acceptance/smmu.py new file +7
-0+ + def test_smmu(self): + """ + :avocado: tags=accel:kvm + :avocado: tags=cpu:host + :avocado: tags=smmu + """
These tags are common across all tests, so you can move them to the class' docstring. Also, you need to add ":avocado: tags=arch:aarch64" or else these will be attempted to be executed with other targets.
Quoted 30 lines
tests/acceptance/smmu.py new file +27
-0+ + self.common_vm_setup() + self.run_and_check() + + def test_smmu_passthrough(self): + """ + :avocado: tags=accel:kvm + :avocado: tags=cpu:host + :avocado: tags=smmu + """ + self.common_vm_setup(True) + + self.kernel_params = self.KERNEL_COMMON_PARAMS + 'iommu.passthrough=on' + + self.run_and_check() + + def test_smmu_nostrict(self): + """ + :avocado: tags=accel:kvm + :avocado: tags=cpu:host + :avocado: tags=smmu + """ + self.common_vm_setup(True) + + self.kernel_params = self.KERNEL_COMMON_PARAMS + 'iommu.strict=0' + + self.run_and_check()Signature · 1 line
2.26.2
Other than those comments, the tests work as expected:
$ ./tests/venv/bin/avocado run tests/acceptance/smmu.py JOB ID : 8b1f1bb775d41e5c593e727ec1907710f076b161 JOB LOG : /home/cleber/avocado/job-results/job-2021-03-25T10.31-8b1f1bb/job.log (1/3) tests/acceptance/smmu.py:SMMU.test_smmu: PASS (42.99 s) (2/3) tests/acceptance/smmu.py:SMMU.test_smmu_passthrough: PASS (47.06 s) (3/3) tests/acceptance/smmu.py:SMMU.test_smmu_nostrict: PASS (47.54 s) RESULTS : PASS 3 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 138.56 s
And even with a bit of stress (running them in parallel) proved to be quite stable:
./tests/venv/bin/avocado run --test-runner=nrunner tests/acceptance/smmu.py
JOB ID : bb2c3a006bb491a7086612dfa1ef980b55db4105
JOB LOG : /home/cleber/avocado/job-results/job-2021-03-25T10.30-bb2c3a0/job.log
(2/3) tests/acceptance/smmu.py:SMMU.test_smmu_passthrough: STARTED (1/3) tests/acceptance/smmu.py:SMMU.test_smmu: STARTED
(3/3) tests/acceptance/smmu.py:SMMU.test_smmu_nostrict: STARTED (1/3) tests/acceptance/smmu.py:SMMU.test_smmu: PASS (50.22 s) (2/3) tests/acceptance/smmu.py:SMMU.test_smmu_passthrough: PASS (57.98 s) (3/3) tests/acceptance/smmu.py:SMMU.test_smmu_nostrict: PASS (58.28 s)
RESULTS : PASS 3 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME : 60.35 s
Regards,
- Cleber.
Re: [PATCH 1/1] avocado_qemu: Add SMMUv3 tests
Hi Cleber,
Quoted 29 lines · Cleber Rosa
Quoted 28 lines · Eric Auger
Add new tests checking the good behavior of the SMMUv3 protecting 2 virtio pci devices (block and net). We check the guest boots and we are able to install a package. Different guest configs are tested:
standard, passthrough an strict=0. Given the version of the guest kernel in use (5.3.7 at this moment), range invalidation is not yet tested. This will be handled separately.Signed-off-by Eric Auger <[email protected]>--- tests/acceptance/smmu.py | 104 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 104 insertions(+) create mode 100644 tests/acceptance/smmu.pytests/acceptance/smmu.py new file +9
-0@@ -0,0 +1,104 @@ +# SMMUv3 Functional tests +# +# Copyright (c) 2021 Red Hat, Inc. +# +# Author: +# Eric Auger <[email protected]> +# +# This work is licensed under the terms of the GNU GPL, version 2 or +# later. See the COPYING file in the top-level directory.
Quoted 8 lines
tests/acceptance/smmu.py line 14
Quoted 5 lines
tests/acceptance/smmu.py new file +5
-0+ +import os + +from avocado_qemu import LinuxTest, BUILD_DIR +from avocado.utils import sshThis import is not needed, given that the you're not using them directly, but only using the LinuxTest methods that wrap them.
Sure I will remove it
Quoted 33 lines
Quoted 33 lines
tests/acceptance/smmu.py new file +32
-0+ +class SMMU(LinuxTest): + + KERNEL_COMMON_PARAMS = ("root=UUID=b6950a44-9f3c-4076-a9c2-355e8475b0a7 ro " + "earlyprintk=pl011,0x9000000 ignore_loglevel " + "no_timer_check printk.time=1 rd_NO_PLYMOUTH " + "console=ttyAMA0 ") + IOMMU_ADDON = ',iommu_platform=on,disable-modern=off,disable-legacy=on' + IMAGE = ("https://archives.fedoraproject.org/pub/archive/fedora/" + "linux/releases/31/Everything/aarch64/os/images/pxeboot/") + kernel_path = None + initrd_path = None + kernel_params = None + + def set_up_boot(self): + path = self.download_boot() + self.vm.add_args('-device', 'virtio-blk-pci,bus=pcie.0,scsi=off,' + + 'drive=drv0,id=virtio-disk0,bootindex=1,' + 'werror=stop,rerror=stop' + self.IOMMU_ADDON) + self.vm.add_args('-drive', + 'file=%s,if=none,cache=writethrough,id=drv0' % path) + + def setUp(self): + super(SMMU, self).setUp(None, 'virtio-net-pci' + self.IOMMU_ADDON) + + def add_common_args(self): + self.vm.add_args("-machine", "virt") + self.vm.add_args('-bios', os.path.join(BUILD_DIR, 'pc-bios', + 'edk2-aarch64-code.fd')) + self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0') + self.vm.add_args('-object', + 'rng-random,id=rng0,filename=/dev/urandom')
Quoted 8 lines
tests/acceptance/smmu.py line 50
Quoted 4 lines
tests/acceptance/smmu.py new file +4
-0+ + def common_vm_setup(self, custom_kernel=None): + self.require_accelerator("kvm") + self.add_common_args()I know you're following the previous test pattern/template, but maybe combine
add_command_args()andcommon_vm_setup()? They seem to be doing the same thing.
yep
Quoted 18 lines
Quoted 18 lines
tests/acceptance/smmu.py new file +17
-0+ self.vm.add_args("-accel", "kvm") + self.vm.add_args("-cpu", "host") + self.vm.add_args("-machine", "iommu=smmuv3") + + if custom_kernel is None: + return + + kernel_url = self.IMAGE + 'vmlinuz' + initrd_url = self.IMAGE + 'initrd.img' + self.kernel_path = self.fetch_asset(kernel_url) + self.initrd_path = self.fetch_asset(initrd_url) + + def run_and_check(self): + if self.kernel_path: + self.vm.add_args('-kernel', self.kernel_path, + '-append', self.kernel_params, + '-initrd', self.initrd_path)
Quoted 8 lines
tests/acceptance/smmu.py line 70
Quoted 3 lines
tests/acceptance/smmu.py new file +3
-0+ self.launch_and_wait() + self.ssh_command('cat /proc/cmdline') + self.ssh_command('dnf -y install numactl-devel')Would you expect the package installation to cover significant more than, say, a package removal? Not relying on the distro's package repos (and external networking) would be an improvement to the test's stability, but I wonder how much functional coverage would be lost.
I noticed this package installed caused an issue in a specific case (range invalidation) which is not handled here due to the kernel being too old. I image that doing the package install is more stressful wrt network than the remove?
Quoted 12 lines
FIY, I've tried it with 'dnf -y remove yum' instead, and test times are also considerably lower.
tests/acceptance/smmu.py line 77
Quoted 7 lines
tests/acceptance/smmu.py new file +7
-0+ + def test_smmu(self): + """ + :avocado: tags=accel:kvm + :avocado: tags=cpu:host + :avocado: tags=smmu + """
Quoted 3 lines
These tags are common across all tests, so you can move them to the class' docstring. Also, you need to add ":avocado: tags=arch:aarch64" or else these will be attempted to be executed with other targets.
sure
Quoted 48 lines
Quoted 31 lines
tests/acceptance/smmu.py new file +27
-0+ + self.common_vm_setup() + self.run_and_check() + + def test_smmu_passthrough(self): + """ + :avocado: tags=accel:kvm + :avocado: tags=cpu:host + :avocado: tags=smmu + """ + self.common_vm_setup(True) + + self.kernel_params = self.KERNEL_COMMON_PARAMS + 'iommu.passthrough=on' + + self.run_and_check() + + def test_smmu_nostrict(self): + """ + :avocado: tags=accel:kvm + :avocado: tags=cpu:host + :avocado: tags=smmu + """ + self.common_vm_setup(True) + + self.kernel_params = self.KERNEL_COMMON_PARAMS + 'iommu.strict=0' + + self.run_and_check()Signature · 1 line
2.26.2Other than those comments, the tests work as expected:
$ ./tests/venv/bin/avocado run tests/acceptance/smmu.py JOB ID : 8b1f1bb775d41e5c593e727ec1907710f076b161 JOB LOG : /home/cleber/avocado/job-results/job-2021-03-25T10.31-8b1f1bb/job.log (1/3) tests/acceptance/smmu.py:SMMU.test_smmu: PASS (42.99 s) (2/3) tests/acceptance/smmu.py:SMMU.test_smmu_passthrough: PASS (47.06 s) (3/3) tests/acceptance/smmu.py:SMMU.test_smmu_nostrict: PASS (47.54 s) RESULTS : PASS 3 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 138.56 sAnd even with a bit of stress (running them in parallel) proved to be quite stable:
./tests/venv/bin/avocado run --test-runner=nrunner
tests/acceptance/smmu.py
Quoted 10 lines
JOB ID :
bb2c3a006bb491a7086612dfa1ef980b55db4105
JOB LOG :/home/cleber/avocado/job-results/job-2021-03-25T10.30-bb2c3a0/job.log
(2/3) tests/acceptance/smmu.py:SMMU.test_smmu_passthrough: STARTED (1/3) tests/acceptance/smmu.py:SMMU.test_smmu: STARTED
(3/3) tests/acceptance/smmu.py:SMMU.test_smmu_nostrict: STARTED (1/3) tests/acceptance/smmu.py:SMMU.test_smmu: PASS (50.22 s) (2/3) tests/acceptance/smmu.py:SMMU.test_smmu_passthrough: PASS (57.98 s) (3/3) tests/acceptance/smmu.py:SMMU.test_smmu_nostrict: PASS (58.28 s)
RESULTS : PASS 3 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME : 60.35 s
Thank you very much for this testing!
Eric
Quoted 4 lines
Regards,
- Cleber.