tests/acceptance: Add test of NeXTcube framebuffer using OCR
v3 cover tests/acceptance: Add test of NeXTcube framebuffer using OCR
Hi,
I was looking at Thomas' last series [*] where he adds the NeXTcube machine, thinking about enforcing a new rule "new machines must have tests". Then I realized the UART is not emitting characters, so our current sample tests are not helpful.
Since the framebuffer is working, I gave a try at dumping the screen content via the HMP 'screendump' command, then parsing the screenshot with an OCR tool.
The default ROM dump the bootlog to a console. Using the old good tesseract tool we can recover some useful words to be sure the guest is sane, its framebuffer is definitively working.
This test takes less than 6s on Travis-CI:
https://travis-ci.org/philmd/qemu/builds/552174983#L1836
AVOCADO tests/acceptance
(3/9) /home/travis/build/philmd/qemu/tests/acceptance/machine_m68k_nextcube.py:NextCubeMachine.test_bootrom_framebuffer: PASS (5.69 s)
Since v2: https://lists.gnu.org/archive/html/qemu-devel/2019-07/msg00249.html
- addressed Thomas and Cleber review comments (detailed in each patch)
- add RFC to interract sending VNC keys
Since v1: https://lists.gnu.org/archive/html/qemu-devel/2019-06/msg06514.html
- use the English dictionary (Thomas)
- support tesseract v3 and v4 (much better results with v4, but not all distros provide it)
- add a test of the framebuffer width/height
Regards,
Phil.
Based-on: [email protected] [*] "m68k: Add basic support for the NeXTcube machine"
https://lists.gnu.org/archive/html/qemu-devel/2019-07/msg02152.html
Philippe Mathieu-Daudé (3):
tests/acceptance: Add test of NeXTcube framebuffer using OCR tests/acceptance: Run commands sending VNC keys .travis.yml: Let the avocado job run the NeXTcube tests
.travis.yml | 7 +- tests/acceptance/machine_m68k_nextcube.py | 170 ++++++++++++++++++++++ tests/requirements.txt | 1 + 3 files changed, 177 insertions(+), 1 deletion(-) create mode 100644 tests/acceptance/machine_m68k_nextcube.py
Signature · 1 line
2.20.1
PATCH v3 1/3 tests/acceptance: Add test of NeXTcube framebuffer using OCR
From: Philippe Mathieu-Daudé <[email protected]>
Add a test of the NeXTcube framebuffer using the Tesseract OCR engine on a screenshot of the framebuffer device.
The test is very quick:
$ avocado --show=app,console run tests/acceptance/machine_m68k_nextcube.py JOB ID : 78844a92424cc495bd068c3874d542d1e20f24bc JOB LOG : /home/phil/avocado/job-results/job-2019-08-13T13.16-78844a9/job.log (1/3) tests/acceptance/machine_m68k_nextcube.py:NextCubeMachine.test_bootrom_framebuffer_size: PASS (2.16 s) (2/3) tests/acceptance/machine_m68k_nextcube.py:NextCubeMachine.test_bootrom_framebuffer_ocr_with_tesseract_v3: - ue r pun Honl'flx ; 5‘ 55‘ avg ncaaaaa 25 MHZ, memary jag m Backplane slat «a Ethernet address a a r a r3 2 Memgry sackets aea canflqured far 16MB Darlly page made stMs but have 16MB page made stMs )nstalled Memgry sackets a and 1 canflqured far 16MB Darlly page made stMs but have 16MB page made stMs )nstalled [...] Yestlnq the rpu, 5:: system test raneg Errar egge 51 Egg: cammand Default pggc devlce nut fauna NEXY>I PASS (2.64 s) (3/3) tests/acceptance/machine_m68k_nextcube.py:NextCubeMachine.test_bootrom_framebuffer_ocr_with_tesseract_v4: SKIP: tesseract v4 OCR tool not available RESULTS : PASS 2 | ERROR 0 | FAIL 0 | SKIP 1 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 5.35 s
Documentation on how to install tesseract:
https://github.com/tesseract-ocr/tesseract/wiki#installation
---
v2:
- test fb sizes
- handle 2 versions of teseract
v3:
- sort imports
- use of ALL_CAPS variables if they are at the module level (PIL_AVAILABLE)
- use find_command from avocado.utils.path
- check tesseract version using regex to avoid exceptions
- do not use '-' in avocado tags
- use use os.path.join()
- add few comments
--- tests/acceptance/machine_m68k_nextcube.py | 121 ++++++++++++++++++++++ 1 file changed, 121 insertions(+) create mode 100644 tests/acceptance/machine_m68k_nextcube.py
tests/acceptance/machine_m68k_nextcube.py new file +121 -0
@@ -0,0 +1,121 @@ +# Functional test that boots a VM and run OCR on the framebuffer +# +# Copyright (c) Philippe Mathieu-Daudé <[email protected]> +# +# This work is licensed under the terms of the GNU GPL, version 2 or +# later. See the COPYING file in the top-level directory. + +import os +import re +import time +import logging +import distutils.spawn + +from avocado_qemu import Test +from avocado import skipUnless +from avocado.utils import process +from avocado.utils.path import find_command, CmdNotFoundError + +PIL_AVAILABLE = True +try: + from PIL import Image +except ImportError: + PIL_AVAILABLE = False + + +def tesseract_available(expected_version): + try: + find_command('tesseract') + except CmdNotFoundError: + return False + res = process.run('tesseract --version') + try: + version = res.stdout_text.split()[1] + except IndexError: + version = res.stderr_text.split()[1] + return int(version.split('.')[0]) == expected_version + + match = re.match(r'tesseract\s(\d)', res) + if match is None: + return False + # now this is guaranteed to be a digit + return int(match.groups()[0]) == expected_version + + +class NextCubeMachine(Test): + + timeout = 15 + + def check_bootrom_framebuffer(self, screenshot_path): + rom_url = ('http://www.nextcomputers.org/NeXTfiles/Software/ROM_Files/' + '68040_Non-Turbo_Chipset/Rev_2.5_v66.BIN') + rom_hash = 'b3534796abae238a0111299fc406a9349f7fee24' + rom_path = self.fetch_asset(rom_url, asset_hash=rom_hash) + + self.vm.set_machine('next-cube') + self.vm.add_args('-bios', rom_path) + self.vm.launch() + + self.log.info('VM launched, waiting for display') + # TODO: Use avocado.utils.wait.wait_for to catch the + # 'displaysurface_create 1120x832' trace-event. + time.sleep(2) + + self.vm.command('human-monitor-command', + command_line='screendump %s' % screenshot_path) + + @skipUnless(PIL_AVAILABLE, 'Python PIL not installed') + def test_bootrom_framebuffer_size(self): + """ + :avocado: tags=arch:m68k + :avocado: tags=machine:next_cube + :avocado: tags=device:framebuffer + """ + screenshot_path = os.path.join(self.workdir, "dump.png") + self.check_bootrom_framebuffer(screenshot_path) + + width, height = Image.open(screenshot_path).size + self.assertEqual(width, 1120) + self.assertEqual(height, 832) + + @skipUnless(tesseract_available(3), 'tesseract v3 OCR tool not available') + def test_bootrom_framebuffer_ocr_with_tesseract_v3(self): + """ + :avocado: tags=arch:m68k + :avocado: tags=machine:next_cube + :avocado: tags=device:framebuffer + """ + screenshot_path = os.path.join(self.workdir, "dump.png") + self.check_bootrom_framebuffer(screenshot_path) + + console_logger = logging.getLogger('console') + text = process.run("tesseract %s stdout" % screenshot_path).stdout_text + for line in text.split('\n'): + if len(line): + console_logger.debug(line) + self.assertIn('Backplane', text) + self.assertIn('Ethernet address', text) + + # Tesseract 4 adds a new OCR engine based on LSTM neural networks. The + # new version is faster and more accurate than version 3. The drawback is + # that it is still alpha-level software. + @skipUnless(tesseract_available(4), 'tesseract v4 OCR tool not available') + def test_bootrom_framebuffer_ocr_with_tesseract_v4(self): + """ + :avocado: tags=arch:m68k + :avocado: tags=machine:next_cube + :avocado: tags=device:framebuffer + """ + screenshot_path = os.path.join(self.workdir, "dump.png") + self.check_bootrom_framebuffer(screenshot_path) + + console_logger = logging.getLogger('console') + proc = process.run("tesseract --oem 1 %s stdout" % screenshot_path) + text = proc.stdout_text + for line in text.split('\n'): + if len(line): + console_logger.debug(line) + self.assertIn('Testing the FPU, SCC', text) + self.assertIn('System test failed. Error code 51', text) + self.assertIn('Boot command', text) + self.assertIn('Next>', text)
Signature · 1 line
2.20.1
PATCH v3 2/3 .travis.yml: Let the avocado job run the NeXTcube tests
In reply to Philippe Mathieu-DaudéFrom: Philippe Mathieu-Daudé <[email protected]>
---
v2:
- install tesseract English package (Thomas)
- install PIL
--- .travis.yml | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
.travis.yml +6 -1
@@ -232,15 +232,20 @@ matrix: # Acceptance (Functional) tests - env: - - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu" + - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,m68k-softmmu" - TEST_CMD="make check-acceptance" after_failure: - cat tests/results/latest/job.log addons: apt: packages: + - python3-pil - python3-pip - python3.5-venv + - tesseract-ocr + - tesseract-ocr-eng + + # Using newer GCC with sanitizers - addons: apt:
Signature · 1 line
2.20.1
RFC PATCH v3 3/3 tests/acceptance: Run commands sending VNC keys
In reply to Philippe Mathieu-DaudéProof of concept to interract with a framebuffer via VNC.
This test send 'mem' and 'help' to the firmware prompt within the single X terminal opened.
---
This test is limited because the '-vnc' option only allow to bind TCP port. To allow parallel testing, I should try to bind within a range instead of ':0'.
--- tests/acceptance/machine_m68k_nextcube.py | 49 +++++++++++++++++++++++ tests/requirements.txt | 1 + 2 files changed, 50 insertions(+)
tests/acceptance/machine_m68k_nextcube.py +49 -0
@@ -13,6 +13,7 @@ import distutils.spawn from avocado_qemu import Test from avocado import skipUnless +from vncdotool import api as vnc from avocado.utils import process from avocado.utils.path import find_command, CmdNotFoundError
@@ -119,3 +120,51 @@ class NextCubeMachine(Test): self.assertIn('System test failed. Error code 51', text) self.assertIn('Boot command', text) self.assertIn('Next>', text) + + + @staticmethod + def vnc_send_string(client, string, eol=True): + for key in string: + client.keyPress(key) + time.sleep(0.02) + if eol: + client.keyPress('enter') + time.sleep(0.02) + + @skipUnless(tesseract_available(3), 'tesseract v3 OCR tool not available') + def test_bootrom_via_vnc_with_tesseract_v3(self): + """ + :avocado: tags=arch:m68k + :avocado: tags=machine:next_cube + :avocado: tags=device:framebuffer + """ + screenshot_path = os.path.join(self.workdir, "dump.png") + + rom_url = ('http://www.nextcomputers.org/NeXTfiles/Software/ROM_Files/' + '68040_Non-Turbo_Chipset/Rev_2.5_v66.BIN') + rom_hash = 'b3534796abae238a0111299fc406a9349f7fee24' + rom_path = self.fetch_asset(rom_url, asset_hash=rom_hash) + + self.vm.set_machine('next-cube') + self.vm.add_args('-bios', rom_path, + '-vnc', ':0') # XXX do not use static TCP port... + self.vm.launch() + + self.log.info('VM launched, waiting for display') + # TODO: Use avocado.utils.wait.wait_for to catch the + # 'displaysurface_create 1120x832' trace-event. + time.sleep(2) + + with vnc.connect('127.0.0.1') as client: + self.vnc_send_string(client, 'help') + self.vnc_send_string(client, 'mem') + client.captureScreen(screenshot_path) + client.disconnect() + + console_logger = logging.getLogger('console') + text = process.run("tesseract %s stdout" % screenshot_path).stdout_text + for line in text.split('\n'): + if len(line): + console_logger.debug(line) + self.assertIn('address space', text) + self.assertIn('System', text)
tests/requirements.txt +1 -0
@@ -3,3 +3,4 @@ # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 avocado-framework==68.0 paramiko +vncdotool
Signature · 1 line
2.20.1
Re: [Qemu-devel] [RFC PATCH v3 3/3] tests/acceptance: Run commands sending VNC keys
Am Tue, 13 Aug 2019 15:49:21 +0200
schrieb Philippe Mathieu-Daudé <[email protected]>:
Quoted 10 lines
Proof of concept to interract with a framebuffer via VNC.
This test send 'mem' and 'help' to the firmware prompt within the single X terminal opened.
Signed-off-by Philippe Mathieu-Daudé <[email protected]>---
This test is limited because the '-vnc' option only allow to bind TCP port. To allow parallel testing, I should try to bind within a range instead of ':0'.
Maybe it would be better to use the "sendkey" HMP command here?
Thomas