Skip to contentSkip to conversation

tests/acceptance: Add test of NeXTcube framebuffer using OCR

PATCH In review Philippe Mathieu-Daudé 2 participants 5 messages Aug 13, 2019 – Sep 8, 2019

lore
Share & tools
lore
#0 lore raw

v3 cover tests/acceptance: Add test of NeXTcube framebuffer using OCR

Hi,

I was looking at Thomas' last series [*] where he adds the NeXTcube machine, thinking about enforcing a new rule "new machines must have tests". Then I realized the UART is not emitting characters, so our current sample tests are not helpful.

Since the framebuffer is working, I gave a try at dumping the screen content via the HMP 'screendump' command, then parsing the screenshot with an OCR tool.

The default ROM dump the bootlog to a console. Using the old good tesseract tool we can recover some useful words to be sure the guest is sane, its framebuffer is definitively working.

This test takes less than 6s on Travis-CI:
https://travis-ci.org/philmd/qemu/builds/552174983#L1836

AVOCADO tests/acceptance
(3/9) /home/travis/build/philmd/qemu/tests/acceptance/machine_m68k_nextcube.py:NextCubeMachine.test_bootrom_framebuffer: PASS (5.69 s)

Since v2: https://lists.gnu.org/archive/html/qemu-devel/2019-07/msg00249.html
- addressed Thomas and Cleber review comments (detailed in each patch)
- add RFC to interract sending VNC keys

Since v1: https://lists.gnu.org/archive/html/qemu-devel/2019-06/msg06514.html
- use the English dictionary (Thomas)
- support tesseract v3 and v4 (much better results with v4, but not all distros provide it)
- add a test of the framebuffer width/height

Regards,

Phil.

Based-on: [email protected] [*] "m68k: Add basic support for the NeXTcube machine"
https://lists.gnu.org/archive/html/qemu-devel/2019-07/msg02152.html

Philippe Mathieu-Daudé (3):
tests/acceptance: Add test of NeXTcube framebuffer using OCR tests/acceptance: Run commands sending VNC keys .travis.yml: Let the avocado job run the NeXTcube tests

 .travis.yml                               |   7 +-
 tests/acceptance/machine_m68k_nextcube.py | 170 ++++++++++++++++++++++
 tests/requirements.txt                    |   1 +
 3 files changed, 177 insertions(+), 1 deletion(-)
 create mode 100644 tests/acceptance/machine_m68k_nextcube.py
Signature · 1 line
2.20.1
#1 lore raw

PATCH v3 1/3 tests/acceptance: Add test of NeXTcube framebuffer using OCR

From: Philippe Mathieu-Daudé <[email protected]>

Add a test of the NeXTcube framebuffer using the Tesseract OCR engine on a screenshot of the framebuffer device.

The test is very quick:

  $ avocado --show=app,console run tests/acceptance/machine_m68k_nextcube.py
  JOB ID     : 78844a92424cc495bd068c3874d542d1e20f24bc
  JOB LOG    : /home/phil/avocado/job-results/job-2019-08-13T13.16-78844a9/job.log
   (1/3) tests/acceptance/machine_m68k_nextcube.py:NextCubeMachine.test_bootrom_framebuffer_size: PASS (2.16 s)
   (2/3) tests/acceptance/machine_m68k_nextcube.py:NextCubeMachine.test_bootrom_framebuffer_ocr_with_tesseract_v3: -
  ue r pun Honl'flx ; 5‘ 55‘
  avg ncaaaaa 25 MHZ, memary jag m
  Backplane slat «a
  Ethernet address a a r a r3 2
  Memgry sackets aea canflqured far 16MB Darlly page made stMs but have 16MB page made stMs )nstalled
  Memgry sackets a and 1 canflqured far 16MB Darlly page made stMs but have 16MB page made stMs )nstalled
  [...]
  Yestlnq the rpu, 5::
  system test raneg Errar egge 51
  Egg: cammand
  Default pggc devlce nut fauna
  NEXY>I
  PASS (2.64 s)
   (3/3) tests/acceptance/machine_m68k_nextcube.py:NextCubeMachine.test_bootrom_framebuffer_ocr_with_tesseract_v4: SKIP: tesseract v4 OCR tool not available
  RESULTS    : PASS 2 | ERROR 0 | FAIL 0 | SKIP 1 | WARN 0 | INTERRUPT 0 | CANCEL 0
  JOB TIME   : 5.35 s

Documentation on how to install tesseract:
https://github.com/tesseract-ocr/tesseract/wiki#installation

---
v2:
- test fb sizes
- handle 2 versions of teseract
v3:
- sort imports
- use of ALL_CAPS variables if they are at the module level (PIL_AVAILABLE)
- use find_command from avocado.utils.path
- check tesseract version using regex to avoid exceptions
- do not use '-' in avocado tags
- use use os.path.join()
- add few comments

---
 tests/acceptance/machine_m68k_nextcube.py | 121 ++++++++++++++++++++++
 1 file changed, 121 insertions(+)
 create mode 100644 tests/acceptance/machine_m68k_nextcube.py
tests/acceptance/machine_m68k_nextcube.py new file +121 -0
new file mode 100644
index 0000000000..e09cab9f20
@@ -0,0 +1,121 @@
+# Functional test that boots a VM and run OCR on the framebuffer
+#
+# Copyright (c) Philippe Mathieu-Daudé <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import os
+import re
+import time
+import logging
+import distutils.spawn
+
+from avocado_qemu import Test
+from avocado import skipUnless
+from avocado.utils import process
+from avocado.utils.path import find_command, CmdNotFoundError
+
+PIL_AVAILABLE = True
+try:
+    from PIL import Image
+except ImportError:
+    PIL_AVAILABLE = False
+
+
+def tesseract_available(expected_version):
+    try:
+        find_command('tesseract')
+    except CmdNotFoundError:
+        return False
+    res = process.run('tesseract --version')
+    try:
+        version = res.stdout_text.split()[1]
+    except IndexError:
+        version = res.stderr_text.split()[1]
+    return int(version.split('.')[0]) == expected_version
+
+    match = re.match(r'tesseract\s(\d)', res)
+    if match is None:
+        return False
+    # now this is guaranteed to be a digit
+    return int(match.groups()[0]) == expected_version
+
+
+class NextCubeMachine(Test):
+
+    timeout = 15
+
+    def check_bootrom_framebuffer(self, screenshot_path):
+        rom_url = ('http://www.nextcomputers.org/NeXTfiles/Software/ROM_Files/'
+                   '68040_Non-Turbo_Chipset/Rev_2.5_v66.BIN')
+        rom_hash = 'b3534796abae238a0111299fc406a9349f7fee24'
+        rom_path = self.fetch_asset(rom_url, asset_hash=rom_hash)
+
+        self.vm.set_machine('next-cube')
+        self.vm.add_args('-bios', rom_path)
+        self.vm.launch()
+
+        self.log.info('VM launched, waiting for display')
+        # TODO: Use avocado.utils.wait.wait_for to catch the
+        #       'displaysurface_create 1120x832' trace-event.
+        time.sleep(2)
+
+        self.vm.command('human-monitor-command',
+                        command_line='screendump %s' % screenshot_path)
+
+    @skipUnless(PIL_AVAILABLE, 'Python PIL not installed')
+    def test_bootrom_framebuffer_size(self):
+        """
+        :avocado: tags=arch:m68k
+        :avocado: tags=machine:next_cube
+        :avocado: tags=device:framebuffer
+        """
+        screenshot_path = os.path.join(self.workdir, "dump.png")
+        self.check_bootrom_framebuffer(screenshot_path)
+
+        width, height = Image.open(screenshot_path).size
+        self.assertEqual(width, 1120)
+        self.assertEqual(height, 832)
+
+    @skipUnless(tesseract_available(3), 'tesseract v3 OCR tool not available')
+    def test_bootrom_framebuffer_ocr_with_tesseract_v3(self):
+        """
+        :avocado: tags=arch:m68k
+        :avocado: tags=machine:next_cube
+        :avocado: tags=device:framebuffer
+        """
+        screenshot_path = os.path.join(self.workdir, "dump.png")
+        self.check_bootrom_framebuffer(screenshot_path)
+
+        console_logger = logging.getLogger('console')
+        text = process.run("tesseract %s stdout" % screenshot_path).stdout_text
+        for line in text.split('\n'):
+            if len(line):
+                console_logger.debug(line)
+        self.assertIn('Backplane', text)
+        self.assertIn('Ethernet address', text)
+
+    # Tesseract 4 adds a new OCR engine based on LSTM neural networks. The
+    # new version is faster and more accurate than version 3. The drawback is
+    # that it is still alpha-level software.
+    @skipUnless(tesseract_available(4), 'tesseract v4 OCR tool not available')
+    def test_bootrom_framebuffer_ocr_with_tesseract_v4(self):
+        """
+        :avocado: tags=arch:m68k
+        :avocado: tags=machine:next_cube
+        :avocado: tags=device:framebuffer
+        """
+        screenshot_path = os.path.join(self.workdir, "dump.png")
+        self.check_bootrom_framebuffer(screenshot_path)
+
+        console_logger = logging.getLogger('console')
+        proc = process.run("tesseract --oem 1 %s stdout" % screenshot_path)
+        text = proc.stdout_text
+        for line in text.split('\n'):
+            if len(line):
+                console_logger.debug(line)
+        self.assertIn('Testing the FPU, SCC', text)
+        self.assertIn('System test failed. Error code 51', text)
+        self.assertIn('Boot command', text)
+        self.assertIn('Next>', text)
Signature · 1 line
2.20.1
#2 lore raw

PATCH v3 2/3 .travis.yml: Let the avocado job run the NeXTcube tests

In reply to Philippe Mathieu-Daudé

From: Philippe Mathieu-Daudé <[email protected]>

---
v2:
- install tesseract English package (Thomas)
- install PIL

---
 .travis.yml | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
.travis.yml +6 -1
index caf0a1f8fa..f6dea2492c 100644
@@ -232,15 +232,20 @@ matrix:
 
     # Acceptance (Functional) tests
     - env:
-        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu"
+        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,m68k-softmmu"
         - TEST_CMD="make check-acceptance"
       after_failure:
         - cat tests/results/latest/job.log
       addons:
         apt:
           packages:
+            - python3-pil
             - python3-pip
             - python3.5-venv
+            - tesseract-ocr
+            - tesseract-ocr-eng
+
+
     # Using newer GCC with sanitizers
     - addons:
         apt:
Signature · 1 line
2.20.1
#3 lore raw

RFC PATCH v3 3/3 tests/acceptance: Run commands sending VNC keys

In reply to Philippe Mathieu-Daudé

Proof of concept to interract with a framebuffer via VNC.

This test send 'mem' and 'help' to the firmware prompt within the single X terminal opened.

---
This test is limited because the '-vnc' option only allow to bind TCP port. To allow parallel testing, I should try to bind within a range instead of ':0'.

---
 tests/acceptance/machine_m68k_nextcube.py | 49 +++++++++++++++++++++++
 tests/requirements.txt                    |  1 +
 2 files changed, 50 insertions(+)
tests/acceptance/machine_m68k_nextcube.py +49 -0
index e09cab9f20..f4f853dfa1 100644
@@ -13,6 +13,7 @@ import distutils.spawn
 
 from avocado_qemu import Test
 from avocado import skipUnless
+from vncdotool import api as vnc
 from avocado.utils import process
 from avocado.utils.path import find_command, CmdNotFoundError
 
@@ -119,3 +120,51 @@ class NextCubeMachine(Test):
         self.assertIn('System test failed. Error code 51', text)
         self.assertIn('Boot command', text)
         self.assertIn('Next>', text)
+
+
+    @staticmethod
+    def vnc_send_string(client, string, eol=True):
+        for key in string:
+            client.keyPress(key)
+            time.sleep(0.02)
+        if eol:
+            client.keyPress('enter')
+            time.sleep(0.02)
+
+    @skipUnless(tesseract_available(3), 'tesseract v3 OCR tool not available')
+    def test_bootrom_via_vnc_with_tesseract_v3(self):
+        """
+        :avocado: tags=arch:m68k
+        :avocado: tags=machine:next_cube
+        :avocado: tags=device:framebuffer
+        """
+        screenshot_path = os.path.join(self.workdir, "dump.png")
+
+        rom_url = ('http://www.nextcomputers.org/NeXTfiles/Software/ROM_Files/'
+                   '68040_Non-Turbo_Chipset/Rev_2.5_v66.BIN')
+        rom_hash = 'b3534796abae238a0111299fc406a9349f7fee24'
+        rom_path = self.fetch_asset(rom_url, asset_hash=rom_hash)
+
+        self.vm.set_machine('next-cube')
+        self.vm.add_args('-bios', rom_path,
+                         '-vnc', ':0') # XXX do not use static TCP port...
+        self.vm.launch()
+
+        self.log.info('VM launched, waiting for display')
+        # TODO: Use avocado.utils.wait.wait_for to catch the
+        #       'displaysurface_create 1120x832' trace-event.
+        time.sleep(2)
+
+        with vnc.connect('127.0.0.1') as client:
+            self.vnc_send_string(client, 'help')
+            self.vnc_send_string(client, 'mem')
+            client.captureScreen(screenshot_path)
+            client.disconnect()
+
+        console_logger = logging.getLogger('console')
+        text = process.run("tesseract %s stdout" % screenshot_path).stdout_text
+        for line in text.split('\n'):
+            if len(line):
+                console_logger.debug(line)
+        self.assertIn('address space', text)
+        self.assertIn('System', text)
tests/requirements.txt +1 -0
index 3ae0e29ad7..f7467287a1 100644
@@ -3,3 +3,4 @@
 # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
 avocado-framework==68.0
 paramiko
+vncdotool
Signature · 1 line
2.20.1
#4 lore raw

Re: [Qemu-devel] [RFC PATCH v3 3/3] tests/acceptance: Run commands sending VNC keys

Am Tue, 13 Aug 2019 15:49:21 +0200
schrieb Philippe Mathieu-Daudé <[email protected]>:

Quoted 10 lines

Proof of concept to interract with a framebuffer via VNC.

This test send 'mem' and 'help' to the firmware prompt within the single X terminal opened.

---
This test is limited because the '-vnc' option only allow to bind TCP port. To allow parallel testing, I should try to bind within a range instead of ':0'.

Maybe it would be better to use the "sendkey" HMP command here?

Thomas