acceptance tests: Test firmware checking debug console output
v2 cover acceptance tests: Test firmware checking debug console output
Hi,
This RFC series add simple acceptance tests which boot SeaBIOS and EDK2 on the pc and virt/aarch64 default machines
Still PoC but can be useful for the Avocado team to test the multi-arch targets.
Since v1: https://lists.gnu.org/archive/html/qemu-devel/2018-09/msg03780.html
- more Pythonic context managers (Cleber)
- use wait_for (Cleber)
- use workdir (Cleber)
- use Avocado 65.0 "-p" option for aarch64 (Cleber)
- fixed Q35 incorrect description in cover (Laszlo)
- use ArmVirtQemu name (Laszlo)
Next:
- address Laszlo comments about correct QEMU options and flash images
- use FDDrainer
- refactor common code in setUp()
- use new avocado.utils.archive gzip features (Cleber)
- test x86_64/aarch64 'arch' tags (Cleber)
- build EDK2 flash images (Laszlo)
- include variable store flash image (Laszlo)
- use virtual disk with UEFI shell script (Laszlo)
- improve OVMF binary selection (releases / snapshots) (Laszlo)
- check ISO images (Laszlo -> Cleber)
- add aexpect tests (Cleber?)
Regards,
Phil.
Philippe Mathieu-Daudé (3):
acceptance tests: Add SeaBIOS boot and debug console checking test acceptance tests: Add EDK2 OVMF boot and debug console checking test acceptance tests: Add EDK2 ArmVirtQemu boot and console checking test
tests/acceptance/boot_firmware.py | 159 ++++++++++++++++++++++++++++++ 1 file changed, 159 insertions(+) create mode 100644 tests/acceptance/boot_firmware.py
Signature · 1 line
2.17.1
RFC PATCH v2 1/3 acceptance tests: Add SeaBIOS boot and debug console checking test
This test boots SeaBIOS and check the debug console (I/O port on the ISA bus) reports enough information on the initialized devices.
Example:
$ avocado run tests/acceptance/boot_firmware.py JOB ID : 3dac2e738c941747ec01f043092b882a8370a92f JOB LOG : /home/phil/avocado/job-results/job-2018-10-03T19.42-3dac2e7/job.log (1/1) tests/acceptance/boot_firmware.py:BootFirmware.test_seabios: PASS (0.27 s) RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 0.56 s
--- tests/acceptance/boot_firmware.py | 74 +++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 tests/acceptance/boot_firmware.py
tests/acceptance/boot_firmware.py new file +74 -0
@@ -0,0 +1,74 @@ +# coding=utf-8 +# +# Functional test that boots SeaBIOS and checks the console +# +# Copyright (c) 2018 Red Hat, Inc. +# +# Author: +# Philippe Mathieu-Daudé <[email protected]> +# +# SPDX-License-Identifier: GPL-2.0-or-later + +import os +import logging + +from avocado_qemu import Test +from avocado.utils.wait import wait_for + + +def read_console_for_string(console, expected_string, logger, ignore_ansi=True): + msg = console.readline() + if len(msg) == 0: + return False + if logger: + logger.debug(msg.strip() if ignore_ansi and not '\x1b' in msg else msg) + return expected_string in msg + + +class BootFirmware(Test): + """ + Boots a firmware and checks via a console it is operational + + :avocado: enable + """ + + def test_seabios(self): + """ + Boots SeaBIOS on a default PC machine, checks the debug console + + :avocado: tags=arch:x86_64 + :avocado: tags=maxtime:5s + :avocado: tags=quick + """ + debugcon_path = os.path.join(self.workdir, 'debugconsole.log') + serial_logger = logging.getLogger('serial') + debugcon_logger = logging.getLogger('debugcon') + + self.vm.set_machine('pc') + self.vm.set_console() + self.vm.add_args('-nographic', + '-net', 'none', + '-global', 'isa-debugcon.iobase=0x402', + '-debugcon', 'file:%s' % debugcon_path) + self.vm.launch() + console = self.vm.console_socket.makefile() + + # serial console checks + if not wait_for(read_console_for_string, timeout=5, step=0, + args=(console, 'No bootable device.', serial_logger)): + self.fail("SeaBIOS failed to boot") + + # debug console checks + expected = [ + 'Running on QEMU (i440fx)', + 'Turning on vga text mode console', + 'Found 1 lpt ports', + 'Found 1 serial ports', + 'PS2 keyboard initialized', + ] + with open(debugcon_path) as debugcon: + content = debugcon.readlines() + for line in content: # TODO use FDDrainer + debugcon_logger.debug(line.strip()) + for exp in expected: + self.assertIn(exp + '\n', content)
Signature · 1 line
2.17.1
RFC PATCH v2 2/3 acceptance tests: Add EDK2 OVMF boot and debug console checking test
In reply to Philippe Mathieu-DaudéThis test boots OVMF and check the debug console (I/O port on the ISA bus) report enough information on the initialized devices.
$ avocado --show=app,debugcon run tests/acceptance/boot_firmware.py (1/1) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_pc: debugcon: SecCoreStartupWithStack(0xFFFCC000, 0x820000) debugcon: SEC: Normal boot ... debugcon: [Bds]Booting EFI Internal Shell PASS (5.27 s) JOB TIME : 5.60 s
--- tests/acceptance/boot_firmware.py | 49 +++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+)
tests/acceptance/boot_firmware.py +49 -0
@@ -72,3 +72,52 @@ class BootFirmware(Test): debugcon_logger.debug(line.strip()) for exp in expected: self.assertIn(exp + '\n', content) + + def test_ovmf_pc(self): + """ + Boots OVMF on the default PC machine, checks the debug console + + :avocado: tags=arch:x86_64 + :avocado: tags=maxtime:10s + """ + debugcon_path = os.path.join(self.workdir, 'debugconsole.log') + serial_logger = logging.getLogger('serial') + debugcon_logger = logging.getLogger('debugcon') + + self.vm.set_machine('pc') + self.vm.set_console() + self.vm.add_args('-nographic', + '-net', 'none', + '-global', 'isa-debugcon.iobase=0x402', + '-debugcon', 'file:%s' % debugcon_path, + '--bios', '/usr/share/OVMF/OVMF_CODE.fd') + self.vm.launch() + console = self.vm.console_socket.makefile() + + # serial console checks + if not wait_for(read_console_for_string, timeout=10, step=0, + args=(console, 'EDK II', serial_logger)): + self.fail("OVMF failed to boot") + + # debug console checks + expected = [ + 'SEC: Normal boot', + 'S3 support was detected on QEMU', + 'Platform PEI Firmware Volume Initialization', + 'DXE IPL Entry', + 'Installing FVB for EMU Variable support', + 'SmbiosCreateTable: Initialize 32-bit entry point structure', + 'PlatformBootManagerBeforeConsole', + 'OnRootBridgesConnected: root bridges have been connected, ' + 'installing ACPI tables', + 'Found LPC Bridge device', + 'PlatformBootManagerAfterConsole', + 'EfiBootManagerConnectAll', + '[Bds]Booting EFI Internal Shell', + ] + with open(debugcon_path) as debugcon: + content = debugcon.readlines() + for line in content: # TODO use FDDrainer + debugcon_logger.debug(line.strip()) + for exp in expected: + self.assertIn(exp + '\r\n', content)
Signature · 1 line
2.17.1
RFC PATCH v2 3/3 acceptance tests: Add EDK2 ArmVirtQemu boot and console checking test
In reply to Philippe Mathieu-DaudéThis test boots EDK2 ArmVirtQemu and check the debug console (PL011) reports enough information on the initialized devices.
$ avocado run -p qemu_bin=aarch64-softmmu/qemu-system-aarch64 tests/acceptance/boot_firmware.py JOB ID : cb1c5bd9e0312483eabeffbb37885a5273ef23bf JOB LOG : /home/phil/avocado/job-results/job-2018-10-03T19.39-cb1c5bd/job.log (1/1) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_virt: PASS (5.02 s) RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 5.30 s
---
The '-p' option requires Avocado >= 65.0
--- tests/acceptance/boot_firmware.py | 36 +++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+)
tests/acceptance/boot_firmware.py +36 -0
@@ -121,3 +121,39 @@ class BootFirmware(Test): debugcon_logger.debug(line.strip()) for exp in expected: self.assertIn(exp + '\r\n', content) + + def test_ovmf_virt(self): + """ + Boots OVMF on the default virt machine, checks the debug console + + :avocado: tags=arch:aarch64 + :avocado: tags=maxtime:20s + """ + image_url = ('http://snapshots.linaro.org/components/kernel/' + 'leg-virt-tianocore-edk2-upstream/latest/' + 'QEMU-AARCH64/DEBUG_GCC5/QEMU_EFI.img.gz') + image_path_gz = self.fetch_asset(image_url) + image_path = os.path.join(self.workdir, 'flash.img') + + # kludge until Avocado support gzip files + import gzip, shutil + with gzip.open(image_path_gz) as gz, open(image_path, 'wb') as img: + shutil.copyfileobj(gz, img) + + serial_path = os.path.join(self.workdir, 'serial.log') + self.vm.set_machine('virt') + self.vm.add_args('-nographic', + '-cpu', 'cortex-a57', + '-m', '1G', # 1GB min to boot fw? + '-drive', 'file=%s,format=raw,if=pflash' % image_path, + '-chardev', 'file,path=%s,id=console' % serial_path, + '-serial', 'chardev:console') + self.vm.launch() + serial_logger = logging.getLogger('serial') + + # serial console checks + if not wait_for(read_console_for_string, timeout=15, step=0, + args=(open(serial_path), + 'Start PXE over IPv4', + serial_logger)): + self.fail("OVMF failed to boot")
Signature · 1 line
2.17.1
Re: [Qemu-devel] [RFC PATCH v2 3/3] acceptance tests: Add EDK2 ArmVirtQemu boot and console checking test
Quoted 52 lines · Philippe Mathieu-Daudé
This test boots EDK2 ArmVirtQemu and check the debug console (PL011) reports enough information on the initialized devices.
$ avocado run -p qemu_bin=aarch64-softmmu/qemu-system-aarch64 tests/acceptance/boot_firmware.py JOB ID : cb1c5bd9e0312483eabeffbb37885a5273ef23bf JOB LOG : /home/phil/avocado/job-results/job-2018-10-03T19.39-cb1c5bd/job.log (1/1) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_virt: PASS (5.02 s) RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 5.30 sSigned-off-by Philippe Mathieu-Daudé <[email protected]>---
The '-p' option requires Avocado >= 65.0--- tests/acceptance/boot_firmware.py | 36 +++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+)tests/acceptance/boot_firmware.py +26
-0@@ -121,3 +121,39 @@ class BootFirmware(Test): debugcon_logger.debug(line.strip()) for exp in expected: self.assertIn(exp + '\r\n', content) + + def test_ovmf_virt(self): + """ + Boots OVMF on the default virt machine, checks the debug console + + :avocado: tags=arch:aarch64 + :avocado: tags=maxtime:20s + """ + image_url = ('http://snapshots.linaro.org/components/kernel/' + 'leg-virt-tianocore-edk2-upstream/latest/' + 'QEMU-AARCH64/DEBUG_GCC5/QEMU_EFI.img.gz') + image_path_gz = self.fetch_asset(image_url) + image_path = os.path.join(self.workdir, 'flash.img') + + # kludge until Avocado support gzip files + import gzip, shutil + with gzip.open(image_path_gz) as gz, open(image_path, 'wb') as img: + shutil.copyfileobj(gz, img) + + serial_path = os.path.join(self.workdir, 'serial.log') + self.vm.set_machine('virt') + self.vm.add_args('-nographic', + '-cpu', 'cortex-a57', + '-m', '1G', # 1GB min to boot fw? + '-drive', 'file=%s,format=raw,if=pflash' % image_path, + '-chardev', 'file,path=%s,id=console' % serial_path,
tests/acceptance/boot_firmware.py class BootFirmware(Test): · line 159
Quoted 10 lines
tests/acceptance/boot_firmware.py +10
-0+ '-serial', 'chardev:console') + self.vm.launch() + serial_logger = logging.getLogger('serial') + + # serial console checks + if not wait_for(read_console_for_string, timeout=15, step=0, + args=(open(serial_path), + 'Start PXE over IPv4', + serial_logger)): + self.fail("OVMF failed to boot")
I suggest replacing "OVMF" with "ArmVirtQemu" in comments and strings as well.
I'd also suggest renaming the method test_ovmf_virt() to test_armvirtqemu_virt().
If that seems baroque, then I suggest "test_uefi_<machtype>" in both patches #2 and #3.
(Sorry about my obsession with these names; just trying to capture the existing "vernacular".)
Thanks
Laszlo
Re: [Qemu-devel] [RFC PATCH v2 3/3] acceptance tests: Add EDK2 ArmVirtQemu boot and console checking test
Quoted 58 lines · Laszlo Ersek
Quoted 57 lines · Philippe Mathieu-Daudé
This test boots EDK2 ArmVirtQemu and check the debug console (PL011) reports enough information on the initialized devices.
$ avocado run -p qemu_bin=aarch64-softmmu/qemu-system-aarch64 tests/acceptance/boot_firmware.py JOB ID : cb1c5bd9e0312483eabeffbb37885a5273ef23bf JOB LOG : /home/phil/avocado/job-results/job-2018-10-03T19.39-cb1c5bd/job.log (1/1) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_virt: PASS (5.02 s) RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 5.30 sSigned-off-by Philippe Mathieu-Daudé <[email protected]>---
The '-p' option requires Avocado >= 65.0--- tests/acceptance/boot_firmware.py | 36 +++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+)tests/acceptance/boot_firmware.py +32
-0@@ -121,3 +121,39 @@ class BootFirmware(Test): debugcon_logger.debug(line.strip()) for exp in expected: self.assertIn(exp + '\r\n', content) + + def test_ovmf_virt(self): + """ + Boots OVMF on the default virt machine, checks the debug console + + :avocado: tags=arch:aarch64 + :avocado: tags=maxtime:20s + """ + image_url = ('http://snapshots.linaro.org/components/kernel/' + 'leg-virt-tianocore-edk2-upstream/latest/' + 'QEMU-AARCH64/DEBUG_GCC5/QEMU_EFI.img.gz') + image_path_gz = self.fetch_asset(image_url) + image_path = os.path.join(self.workdir, 'flash.img') + + # kludge until Avocado support gzip files + import gzip, shutil + with gzip.open(image_path_gz) as gz, open(image_path, 'wb') as img: + shutil.copyfileobj(gz, img) + + serial_path = os.path.join(self.workdir, 'serial.log') + self.vm.set_machine('virt') + self.vm.add_args('-nographic', + '-cpu', 'cortex-a57', + '-m', '1G', # 1GB min to boot fw? + '-drive', 'file=%s,format=raw,if=pflash' % image_path, + '-chardev', 'file,path=%s,id=console' % serial_path, + '-serial', 'chardev:console') + self.vm.launch() + serial_logger = logging.getLogger('serial') + + # serial console checks + if not wait_for(read_console_for_string, timeout=15, step=0,
Quoted 8 lines
tests/acceptance/boot_firmware.py class BootFirmware(Test): · line 159
Quoted 5 lines
tests/acceptance/boot_firmware.py +4
-0+ args=(open(serial_path), + 'Start PXE over IPv4', + serial_logger)): + self.fail("OVMF failed to boot")I suggest replacing "OVMF" with "ArmVirtQemu" in comments and strings as well.
Oops OK.
Quoted 6 lines
I'd also suggest renaming the method
test_ovmf_virt()totest_armvirtqemu_virt().If that seems baroque, then I suggest "test_uefi_<machtype>" in both patches #2 and #3.
What about this?
- test_bios_seabios_x86_64_pc()
- test_uefi_ovmf_x86_64_pc()
- test_uefi_armvirtqemu_aarch64_virt()
Quoted 7 lines
(Sorry about my obsession with these names; just trying to capture the existing "vernacular".)
Thanks
Laszlo
Re: [Qemu-devel] [RFC PATCH v2 3/3] acceptance tests: Add EDK2 ArmVirtQemu boot and console checking test
Quoted 74 lines · Philippe Mathieu-Daudé
Quoted 58 lines · Laszlo Ersek
Quoted 57 lines · Philippe Mathieu-Daudé
This test boots EDK2 ArmVirtQemu and check the debug console (PL011) reports enough information on the initialized devices.
$ avocado run -p qemu_bin=aarch64-softmmu/qemu-system-aarch64 tests/acceptance/boot_firmware.py JOB ID : cb1c5bd9e0312483eabeffbb37885a5273ef23bf JOB LOG : /home/phil/avocado/job-results/job-2018-10-03T19.39-cb1c5bd/job.log (1/1) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_virt: PASS (5.02 s) RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 5.30 sSigned-off-by Philippe Mathieu-Daudé <[email protected]>---
The '-p' option requires Avocado >= 65.0--- tests/acceptance/boot_firmware.py | 36 +++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+)tests/acceptance/boot_firmware.py +32
-0@@ -121,3 +121,39 @@ class BootFirmware(Test): debugcon_logger.debug(line.strip()) for exp in expected: self.assertIn(exp + '\r\n', content) + + def test_ovmf_virt(self): + """ + Boots OVMF on the default virt machine, checks the debug console + + :avocado: tags=arch:aarch64 + :avocado: tags=maxtime:20s + """ + image_url = ('http://snapshots.linaro.org/components/kernel/' + 'leg-virt-tianocore-edk2-upstream/latest/' + 'QEMU-AARCH64/DEBUG_GCC5/QEMU_EFI.img.gz') + image_path_gz = self.fetch_asset(image_url) + image_path = os.path.join(self.workdir, 'flash.img') + + # kludge until Avocado support gzip files + import gzip, shutil + with gzip.open(image_path_gz) as gz, open(image_path, 'wb') as img: + shutil.copyfileobj(gz, img) + + serial_path = os.path.join(self.workdir, 'serial.log') + self.vm.set_machine('virt') + self.vm.add_args('-nographic', + '-cpu', 'cortex-a57', + '-m', '1G', # 1GB min to boot fw? + '-drive', 'file=%s,format=raw,if=pflash' % image_path, + '-chardev', 'file,path=%s,id=console' % serial_path, + '-serial', 'chardev:console') + self.vm.launch() + serial_logger = logging.getLogger('serial') + + # serial console checks + if not wait_for(read_console_for_string, timeout=15, step=0,Quoted 8 lines
tests/acceptance/boot_firmware.py class BootFirmware(Test): · line 159
Quoted 5 lines
tests/acceptance/boot_firmware.py +4
-0+ args=(open(serial_path), + 'Start PXE over IPv4', + serial_logger)): + self.fail("OVMF failed to boot")I suggest replacing "OVMF" with "ArmVirtQemu" in comments and strings as well.
Oops OK.
Quoted 4 lines
I'd also suggest renaming the method
test_ovmf_virt()totest_armvirtqemu_virt().
Quoted 8 lines
Quoted 2 lines
If that seems baroque, then I suggest "test_uefi_<machtype>" in both patches #2 and #3.
What about this?
-
test_bios_seabios_x86_64_pc()
-test_uefi_ovmf_x86_64_pc()
-test_uefi_armvirtqemu_aarch64_virt()
Those work for me :)
Laszlo
Quoted 7 lines · Philippe Mathieu-Daudé
Hi,
This RFC series add simple acceptance tests which boot SeaBIOS and EDK2 on the pc and
virt/aarch64default machinesStill PoC but can be useful for the Avocado team to test the multi-arch targets.
I couldn't get this to work on qemu-test (aarch64):
12:52:56 [alex@qemu-test:~/l/qemu.git] review/acceptance-rfc-v2 + avocado run tests/acceptance
JOB ID : b6377b8d4196903846c7d57e2b234c523d6c6ba1
JOB LOG : /home/alex/avocado/job-results/job-2018-10-04T12.53-b6377b8/job.log
(1/9) tests/acceptance/boot_firmware.py:BootFirmware.test_seabios: ERROR: timed out (15.14 s) (2/9) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_pc: ERROR: timed out (15.14 s) (3/9) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_virt: FAIL: OVMF failed to boot (17.97 s) (4/9) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test: ERROR: timed out (17.36 s) (5/9) tests/acceptance/version.py:Version.test_qmp_human_info_version: ERROR: timed out (15.14 s) (6/9) tests/acceptance/vnc.py:Vnc.test_no_vnc: ERROR: timed out (15.14 s)
(7/9) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password: ERROR: timed out (15.14 s) (8/9) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password: ERROR: timed out (15.14 s) (9/9) tests/acceptance/vnc.py:Vnc.test_vnc_change_password: ERROR: timed out (15.14 s)
RESULTS : PASS 0 | ERROR 8 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME : 142.37 s
The error log: https://transfer.sh/lK71v/avocado-errors.log
Quoted 26 lines
Since v1: https://lists.gnu.org/archive/html/qemu-devel/2018-09/msg03780.html
- more Pythonic context managers (Cleber)
- usewait_for(Cleber)
- use workdir (Cleber)
- use Avocado 65.0 "-p" option for aarch64 (Cleber)
- fixed Q35 incorrect description in cover (Laszlo)
- use ArmVirtQemu name (Laszlo)Next:
- address Laszlo comments about correct QEMU options and flash images
- use FDDrainer
- refactor common code insetUp()
- use new avocado.utils.archive gzip features (Cleber)
- test x86_64/aarch64 'arch' tags (Cleber)
- build EDK2 flash images (Laszlo)
- include variable store flash image (Laszlo)
- use virtual disk with UEFI shell script (Laszlo)
- improve OVMF binary selection (releases / snapshots) (Laszlo)
- check ISO images (Laszlo -> Cleber)
- add aexpect tests (Cleber?)Regards,
Phil.
Quoted 8 lines
Philippe Mathieu-Daudé (3):
acceptance tests: Add SeaBIOS boot and debug console checking test acceptance tests: Add EDK2 OVMF boot and debug console checking test acceptance tests: Add EDK2 ArmVirtQemu boot and console checking testtests/acceptance/boot_firmware.py | 159 ++++++++++++++++++++++++++++++ 1 file changed, 159 insertions(+) create mode 100644 tests/acceptance/boot_firmware.py
--
Alex Bennée
Quoted 27 lines · Alex Bennée
Quoted 7 lines · Philippe Mathieu-Daudé
Hi,
This RFC series add simple acceptance tests which boot SeaBIOS and EDK2 on the pc and
virt/aarch64default machinesStill PoC but can be useful for the Avocado team to test the multi-arch targets.
I couldn't get this to work on qemu-test (aarch64):
12:52:56 [alex@qemu-test:~/l/qemu.git] review/acceptance-rfc-v2 + avocado run tests/acceptance
JOB ID :b6377b8d4196903846c7d57e2b234c523d6c6ba1
JOB LOG :/home/alex/avocado/job-results/job-2018-10-04T12.53-b6377b8/job.log
(1/9) tests/acceptance/boot_firmware.py:BootFirmware.test_seabios: ERROR: timed out (15.14 s) (2/9) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_pc: ERROR: timed out (15.14 s) (3/9) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_virt: FAIL: OVMF failed to boot (17.97 s) (4/9) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test: ERROR: timed out (17.36 s) (5/9) tests/acceptance/version.py:Version.test_qmp_human_info_version: ERROR: timed out (15.14 s) (6/9) tests/acceptance/vnc.py:Vnc.test_no_vnc: ERROR: timed out (15.14 s)
(7/9) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password: ERROR: timed out (15.14 s) (8/9) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password: ERROR: timed out (15.14 s) (9/9) tests/acceptance/vnc.py:Vnc.test_vnc_change_password: ERROR: timed out (15.14 s)
RESULTS : PASS 0 | ERROR 8 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME : 142.37 s
Quoted 1 line
The error log: https://transfer.sh/lK71v/avocado-errors.log
It looks like it's trying to launch a pc (rather than virt) machine:
2018-10-04 12:53:16,339 qemu L0270 DEBUG| Output: 'qemu-system-aarch64: -machine pc: unsupported machine type\nUse -machine help to list supported machines\n'
Quoted 27 lines
Quoted 27 lines
Since v1: https://lists.gnu.org/archive/html/qemu-devel/2018-09/msg03780.html
- more Pythonic context managers (Cleber)
- usewait_for(Cleber)
- use workdir (Cleber)
- use Avocado 65.0 "-p" option for aarch64 (Cleber)
- fixed Q35 incorrect description in cover (Laszlo)
- use ArmVirtQemu name (Laszlo)Next:
- address Laszlo comments about correct QEMU options and flash images
- use FDDrainer
- refactor common code insetUp()
- use new avocado.utils.archive gzip features (Cleber)
- test x86_64/aarch64 'arch' tags (Cleber)
- build EDK2 flash images (Laszlo)
- include variable store flash image (Laszlo)
- use virtual disk with UEFI shell script (Laszlo)
- improve OVMF binary selection (releases / snapshots) (Laszlo)
- check ISO images (Laszlo -> Cleber)
- add aexpect tests (Cleber?)Regards,
Phil.
Quoted 8 lines
Quoted 8 lines
Philippe Mathieu-Daudé (3):
acceptance tests: Add SeaBIOS boot and debug console checking test acceptance tests: Add EDK2 OVMF boot and debug console checking test acceptance tests: Add EDK2 ArmVirtQemu boot and console checking testtests/acceptance/boot_firmware.py | 159 ++++++++++++++++++++++++++++++ 1 file changed, 159 insertions(+) create mode 100644 tests/acceptance/boot_firmware.py
--
Alex Bennée
Quoted 30 lines · Alex Bennée
Quoted 27 lines · Alex Bennée
Quoted 7 lines · Philippe Mathieu-Daudé
Hi,
This RFC series add simple acceptance tests which boot SeaBIOS and EDK2 on the pc and
virt/aarch64default machinesStill PoC but can be useful for the Avocado team to test the multi-arch targets.
I couldn't get this to work on qemu-test (aarch64):
12:52:56 [alex@qemu-test:~/l/qemu.git] review/acceptance-rfc-v2 + avocado run tests/acceptance
JOB ID :b6377b8d4196903846c7d57e2b234c523d6c6ba1
JOB LOG :/home/alex/avocado/job-results/job-2018-10-04T12.53-b6377b8/job.log
(1/9) tests/acceptance/boot_firmware.py:BootFirmware.test_seabios: ERROR: timed out (15.14 s) (2/9) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_pc: ERROR: timed out (15.14 s) (3/9) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_virt: FAIL: OVMF failed to boot (17.97 s) (4/9) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test: ERROR: timed out (17.36 s) (5/9) tests/acceptance/version.py:Version.test_qmp_human_info_version: ERROR: timed out (15.14 s) (6/9) tests/acceptance/vnc.py:Vnc.test_no_vnc: ERROR: timed out (15.14 s)
(7/9) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password: ERROR: timed out (15.14 s) (8/9) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password: ERROR: timed out (15.14 s) (9/9) tests/acceptance/vnc.py:Vnc.test_vnc_change_password: ERROR: timed out (15.14 s)
RESULTS : PASS 0 | ERROR 8 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME : 142.37 s
Quoted 5 lines
Quoted 1 line
The error log: https://transfer.sh/lK71v/avocado-errors.log
It looks like it's trying to launch a pc (rather than virt) machine:
2018-10-04 12:53:16,339 qemu L0270 DEBUG| Output: 'qemu-system-aarch64: -machine pc: unsupported machine type\nUse -machine help to list supported machines\n'
That also suggests a secondary bug somewhere: if QEMU exits immediately with an error, then the test should fail immediately, not wait for a 15s timeout before it notices.
thanks
-- PMM
Quoted 33 lines · Peter Maydell
Quoted 32 lines · Alex Bennée
Quoted 28 lines · Alex Bennée
Quoted 7 lines · Philippe Mathieu-Daudé
Hi,
This RFC series add simple acceptance tests which boot SeaBIOS and EDK2 on the pc and
virt/aarch64default machinesStill PoC but can be useful for the Avocado team to test the multi-arch targets.
I couldn't get this to work on qemu-test (aarch64):
12:52:56 [alex@qemu-test:~/l/qemu.git] review/acceptance-rfc-v2 + avocado run tests/acceptance
JOB ID :b6377b8d4196903846c7d57e2b234c523d6c6ba1
JOB LOG :/home/alex/avocado/job-results/job-2018-10-04T12.53-b6377b8/job.log
(1/9) tests/acceptance/boot_firmware.py:BootFirmware.test_seabios: ERROR: timed out (15.14 s) (2/9) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_pc: ERROR: timed out (15.14 s) (3/9) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_virt: FAIL: OVMF failed to boot (17.97 s) (4/9) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test: ERROR: timed out (17.36 s) (5/9) tests/acceptance/version.py:Version.test_qmp_human_info_version: ERROR: timed out (15.14 s) (6/9) tests/acceptance/vnc.py:Vnc.test_no_vnc: ERROR: timed out (15.14 s)
(7/9) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password: ERROR: timed out (15.14 s) (8/9) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password: ERROR: timed out (15.14 s) (9/9) tests/acceptance/vnc.py:Vnc.test_vnc_change_password: ERROR: timed out (15.14 s)
RESULTS : PASS 0 | ERROR 8 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME : 142.37 sThe error log: https://transfer.sh/lK71v/avocado-errors.log
Quoted 9 lines
Quoted 3 lines
It looks like it's trying to launch a pc (rather than virt) machine:
2018-10-04 12:53:16,339 qemu L0270 DEBUG| Output: 'qemu-system-aarch64: -machine pc: unsupported machine type\nUse -machine help to list supported machines\n'That also suggests a secondary bug somewhere: if QEMU exits immediately with an error, then the test should fail immediately, not wait for a 15s timeout before it notices.
That's true, and that's due to a connection timeout to the QMP socket. It's a known issue that we're tracking here:
https://trello.com/c/O882Vulm/44-qemumachine-even-if-we-expect-qemu-to-exit-with-an-error
And Wainer Moschetta (+cc) is working on it.
Regards,
- Cleber.
Quoted 3 lines
thanks
-- PMM
Hi Alex,
Quoted 7 lines · Alex Bennée
Quoted 4 lines · Alex Bennée
Quoted 2 lines · Philippe Mathieu-Daudé
Hi,
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
This RFC series add simple acceptance tests which boot SeaBIOS and EDK2 on the pc and
virt/aarch64default machinesStill PoC but can be useful for the Avocado team to test the multi-arch targets.
I couldn't get this to work on qemu-test (aarch64):
Thanks for giving it a try!
Sorry, I only tested this on x86_64 :/
Quoted 16 lines
Quoted 16 lines
12:52:56 [alex@qemu-test:~/l/qemu.git] review/acceptance-rfc-v2 + avocado run tests/acceptance
JOB ID :b6377b8d4196903846c7d57e2b234c523d6c6ba1
JOB LOG :/home/alex/avocado/job-results/job-2018-10-04T12.53-b6377b8/job.log
(1/9) tests/acceptance/boot_firmware.py:BootFirmware.test_seabios: ERROR: timed out (15.14 s) (2/9) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_pc: ERROR: timed out (15.14 s) (3/9) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_virt: FAIL: OVMF failed to boot (17.97 s) (4/9) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test: ERROR: timed out (17.36 s) (5/9) tests/acceptance/version.py:Version.test_qmp_human_info_version: ERROR: timed out (15.14 s) (6/9) tests/acceptance/vnc.py:Vnc.test_no_vnc: ERROR: timed out (15.14 s)
(7/9) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password: ERROR: timed out (15.14 s) (8/9) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password: ERROR: timed out (15.14 s) (9/9) tests/acceptance/vnc.py:Vnc.test_vnc_change_password: ERROR: timed out (15.14 s)
RESULTS : PASS 0 | ERROR 8 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME : 142.37 s
Quoted 5 lines
Quoted 1 line
The error log: https://transfer.sh/lK71v/avocado-errors.log
It looks like it's trying to launch a pc (rather than virt) machine:
2018-10-04 12:53:16,339 qemu L0270 DEBUG| Output: 'qemu-system-aarch64: -machine pc: unsupported machine type\nUse -machine help to list supported machines\n'
Due to some limitation (Cleber is aware of it) you have to use the -p option as in patch #3 of this series:
$ avocado run --filter-by-tags=arch:x86_64 -p qemu_bin=x86_64-softmmu/qemu-system-x86_64 tests/acceptance/boot_firmware.py JOB ID : 5f9078796ef749e0606bbdf9409e91443ab2bad1 JOB LOG : /home/phil/avocado/job-results/job-2018-10-04T14.30-5f90787/job.log (1/2) tests/acceptance/boot_firmware.py:BootFirmware.test_seabios: PASS (0.90 s) (2/2) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_pc: FAIL: 'SEC: Normal boot\r\n' not found in [] (9.48 s) RESULTS : PASS 1 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 10.72 s
The error "FAIL: 'SEC: Normal boot\r\n' not found" means the serial test worked, but not the debugconsole.
You can use verbose Avocado:
$ avocado --show=app,debugcon,serial run --filter-by-tags=arch:x86_64 -p qemu_bin=x86_64-softmmu/qemu-system-x86_64 tests/acceptance/boot_firmware.py (2/2) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_pc: \ serial: UEFI Interactive Shell v2.2
serial: EDK II
FAIL: 'SEC: Normal boot\r\n' not found in [] (9.58 s)
I'll look at it.
$ avocado run --filter-by-tags=arch:aarch64 -p qemu_bin=aarch64-softmmu/qemu-system-aarch64 tests/acceptance/boot_firmware.py JOB ID : 678d83c0b10d6daadb5fa177318b774281c346b7 JOB LOG : /home/phil/avocado/job-results/job-2018-10-04T14.29-678d83c/job.log (1/1) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_virt: FAIL: OVMF failed to boot (16.59 s) RESULTS : PASS 0 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 16.83 s
$ avocado --show=app,serial run --filter-by-tags=arch:aarch64 -p qemu_bin=aarch64-softmmu/qemu-system-aarch64 tests/acceptance/boot_firmware.py JOB LOG : /home/phil/avocado/job-results/job-2018-10-04T14.40-dab2e26/job.log (1/1) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_virt: serial: a serial: dd-symbol- serial: file /home/ serial: buildslave/ serial: workspace/l serial: eg-virt-tia serial: nocore-edk2 serial: -upstream/e serial: dk2/Build/A serial: rmVirtQemu- serial: AARCH64/DEB serial: UG_GCC5/AA serial: RCH64/ArmPl serial: atformPkg/P serial: rePeiCore/P serial: rePeiCoreUn serial: iCore/DEBUG serial: /ArmPlatfor serial: mPrePeiCore serial: .dll 0x1800 ... serial: Boot00 serial: 04: EFI In serial: ternal S serial: h serial: ell 0x00 serial: 01 serial: serial: PlatformRe serial: covery Opti serial: ons: FAIL: OVMF failed to boot (16.44 s) RESULTS : PASS 0 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 16.68 s
So this image boots, this is my ugly way of handling virt console with avocado which failed :)
QEMU console arg:
'-chardev', 'file,path=%s,id=console' % serial_path, '-serial', 'chardev:console')
then I'd like to use the FDDrainer class Cleber suggested instead of the current read_console_for_string().
Thanks,
Phil.
Quoted 28 lines · Alex Bennée
Quoted 7 lines · Philippe Mathieu-Daudé
Hi,
This RFC series add simple acceptance tests which boot SeaBIOS and EDK2 on the pc and
virt/aarch64default machinesStill PoC but can be useful for the Avocado team to test the multi-arch targets.
I couldn't get this to work on qemu-test (aarch64):
12:52:56 [alex@qemu-test:~/l/qemu.git] review/acceptance-rfc-v2 + avocado run tests/acceptance
JOB ID :b6377b8d4196903846c7d57e2b234c523d6c6ba1
JOB LOG :/home/alex/avocado/job-results/job-2018-10-04T12.53-b6377b8/job.log
(1/9) tests/acceptance/boot_firmware.py:BootFirmware.test_seabios: ERROR: timed out (15.14 s) (2/9) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_pc: ERROR: timed out (15.14 s) (3/9) tests/acceptance/boot_firmware.py:BootFirmware.test_ovmf_virt: FAIL: OVMF failed to boot (17.97 s) (4/9) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test: ERROR: timed out (17.36 s) (5/9) tests/acceptance/version.py:Version.test_qmp_human_info_version: ERROR: timed out (15.14 s) (6/9) tests/acceptance/vnc.py:Vnc.test_no_vnc: ERROR: timed out (15.14 s)
(7/9) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password: ERROR: timed out (15.14 s) (8/9) tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password: ERROR: timed out (15.14 s) (9/9) tests/acceptance/vnc.py:Vnc.test_vnc_change_password: ERROR: timed out (15.14 s)
RESULTS : PASS 0 | ERROR 8 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME : 142.37 s
Quoted 2 lines
The error log: https://transfer.sh/lK71v/avocado-errors.log
The core reason is:
2018-10-04 12:53:16,339 qemu L0270 DEBUG| Output:
'qemu-system-aarch64: -machine pc: unsupported machine type\nUse -machine help to list supported machines\n'
I'm minutes away from sending a series that adds support for arch and machine type defaults.
- Cleber.
Quoted 39 lines
Quoted 26 lines
Since v1: https://lists.gnu.org/archive/html/qemu-devel/2018-09/msg03780.html
- more Pythonic context managers (Cleber)
- usewait_for(Cleber)
- use workdir (Cleber)
- use Avocado 65.0 "-p" option for aarch64 (Cleber)
- fixed Q35 incorrect description in cover (Laszlo)
- use ArmVirtQemu name (Laszlo)Next:
- address Laszlo comments about correct QEMU options and flash images
- use FDDrainer
- refactor common code insetUp()
- use new avocado.utils.archive gzip features (Cleber)
- test x86_64/aarch64 'arch' tags (Cleber)
- build EDK2 flash images (Laszlo)
- include variable store flash image (Laszlo)
- use virtual disk with UEFI shell script (Laszlo)
- improve OVMF binary selection (releases / snapshots) (Laszlo)
- check ISO images (Laszlo -> Cleber)
- add aexpect tests (Cleber?)Regards,
Phil.
Quoted 8 lines
Philippe Mathieu-Daudé (3):
acceptance tests: Add SeaBIOS boot and debug console checking test acceptance tests: Add EDK2 OVMF boot and debug console checking test acceptance tests: Add EDK2 ArmVirtQemu boot and console checking testtests/acceptance/boot_firmware.py | 159 ++++++++++++++++++++++++++++++ 1 file changed, 159 insertions(+) create mode 100644 tests/acceptance/boot_firmware.py--
Alex Bennée
Quoted 2 lines · Cleber Rosa
Quoted 1 line · Alex Bennée
<snip>
Quoted 4 lines
Quoted 3 lines
The error log: https://transfer.sh/lK71v/avocado-errors.log
Quoted 8 lines
The core reason is:
2018-10-04 12:53:16,339 qemu L0270 DEBUG| Output:'qemu-system-aarch64: -machine pc: unsupported machine type\nUse -machine help to list supported machines\n'
I'm minutes away from sending a series that adds support for arch and machine type defaults.
\o/
I see it now, I shall have a look.
Quoted 42 lines
- Cleber.
Quoted 39 lines
Quoted 26 lines
Since v1: https://lists.gnu.org/archive/html/qemu-devel/2018-09/msg03780.html
- more Pythonic context managers (Cleber)
- usewait_for(Cleber)
- use workdir (Cleber)
- use Avocado 65.0 "-p" option for aarch64 (Cleber)
- fixed Q35 incorrect description in cover (Laszlo)
- use ArmVirtQemu name (Laszlo)Next:
- address Laszlo comments about correct QEMU options and flash images
- use FDDrainer
- refactor common code insetUp()
- use new avocado.utils.archive gzip features (Cleber)
- test x86_64/aarch64 'arch' tags (Cleber)
- build EDK2 flash images (Laszlo)
- include variable store flash image (Laszlo)
- use virtual disk with UEFI shell script (Laszlo)
- improve OVMF binary selection (releases / snapshots) (Laszlo)
- check ISO images (Laszlo -> Cleber)
- add aexpect tests (Cleber?)Regards,
Phil.
Quoted 8 lines
Philippe Mathieu-Daudé (3):
acceptance tests: Add SeaBIOS boot and debug console checking test acceptance tests: Add EDK2 OVMF boot and debug console checking test acceptance tests: Add EDK2 ArmVirtQemu boot and console checking testtests/acceptance/boot_firmware.py | 159 ++++++++++++++++++++++++++++++ 1 file changed, 159 insertions(+) create mode 100644 tests/acceptance/boot_firmware.py--
Alex Bennée
--
Alex Bennée