Skip to contentSkip to conversation

Proof of concept for screenshot testcases

PATCH Superseded Eilís 'pidge' Ní Fhlannagáin 3 participants 11 messages Dec 13, 2023 – Dec 14, 2023

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover Proof of concept for screenshot testcases

This takes the work rburton did on image screenshot testing and expands it.

Right now this only works for qemux86-64. Some standardization of screensize/resolution needs to happen with runqemu params for other machines. There is an issue in qemux86-64 (and possibly others) where the screenswitch icon is only half present. This causes the test to fail.

This test takes a screendump of a qemu image (for now, just core-image-sato for qemux86-64), and compares it to an image we have on record. Some normalisation of the different qemu configs need to happen to be able to support all machines. Example, the qemuarm64 screen size is much larger than the qemux86-64.

The image we have on record contains a blanked out clock. We do the same blanking out process for the screenshot, so the images should have zero differences. If they do, we fail.

In order to enable this test, you will need meta-openembedded/meta-oe in your bblayers.conf and the following in local.conf:

IMAGE_CLASSES += "testimage"
TEST_SUITES = "login"
IMAGE_INSTALL:append = " python3-qemu-qmp " TESTIMAGEDEPENDS:append:qemuall = " imagemagick-native:do_populate_sysroot "

Eilís 'pidge' Ní Fhlannagáin (1):
login.py: Proof of concept for screenshot testcases

Ross Burton (3):
qemurunner: remove unused import
python3-qemu-qmp: Add recipe
runqemu: add qmp socket support

 .../core-image-sato-qemux86-64.png            | Bin 0 -> 46986 bytes
 meta/lib/oeqa/runtime/cases/login.py          |  34 ++++++++++++++++++
 meta/lib/oeqa/utils/qemurunner.py             |   1 -
 .../python/python3-qemu-qmp_0.0.2.bb          |  14 ++++++++
 scripts/runqemu                               |  11 ++++++
 5 files changed, 59 insertions(+), 1 deletion(-)
 create mode 100644 meta/files/image-tests/core-image-sato-qemux86-64.png
 create mode 100644 meta/lib/oeqa/runtime/cases/login.py
 create mode 100644 meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb
Signature · 1 line
2.34.1
#1 lore raw

PATCH v2 1/4 qemurunner: remove unused import

From: Ross Burton <[email protected]>

Removes unused logging import

---
 meta/lib/oeqa/utils/qemurunner.py | 1 -
 1 file changed, 1 deletion(-)
meta/lib/oeqa/utils/qemurunner.py +0 -1
index 29fe2719762..ee367078000 100644
@@ -19,7 +19,6 @@ import errno
 import string
 import threading
 import codecs
-import logging
 import tempfile
 from collections import defaultdict
 import importlib
Signature · 1 line
2.34.1
#2 lore raw

PATCH v2 2/4 python3-qemu-qmp: Add recipe

In reply to Eilís 'pidge' Ní Fhlannagáin

From: Ross Burton <[email protected]>

Add a recipe for python QEMU Monitor Protocol. This will be needed for qemu screendumps for qemu screendump testing.

---
 .../python/python3-qemu-qmp_0.0.2.bb               | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb
meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb new file +14 -0
new file mode 100644
index 00000000000..faa0cc69738
@@ -0,0 +1,14 @@
+SUMMARY = "asyncio library for communicating with QEMU Monitor Protocol (QMP) servers"
+LICENSE = "LGPLv2+ & GPLv2"
+LIC_FILES_CHKSUM = "file://LICENSE;md5=4cf66a4984120007c9881cc871cf49db \
+                    file://LICENSE_GPL2;md5=441c28d2cf86e15a37fa47e15a72fbac"
+
+SRC_URI[sha256sum] = "c918e9e3ae09abdf70c7ece67637a93ac4583d940bbf48d24ff77987f74f1b8b"
+
+inherit pypi python_setuptools_build_meta
+
+PYPI_PACKAGE = "qemu.qmp"
+
+DEPENDS += "python3-setuptools-scm-native"
+
+BBCLASSEXTEND = "native nativesdk"
Signature · 1 line
2.34.1
#4 lore raw

PATCH v2 3/4 runqemu: add qmp socket support

In reply to Eilís 'pidge' Ní Fhlannagáin

From: Ross Burton <[email protected]>

Add support for qmp sockets and defaults to unix:qmp.sock if unspecified

---
 scripts/runqemu | 11 +++++++++++
 1 file changed, 11 insertions(+)
scripts/runqemu +11 -0
index 18aeb7f5f0c..6a5a6451daf 100755
@@ -84,6 +84,7 @@ of the following environment variables (in any order):
     publicvnc - enable a VNC server open to all hosts
     audio - enable audio
     guestagent - enable guest agent communication
+    qmp=<path> - create a QMP socket (defaults to unix:qmp.sock if unspecified)
     [*/]ovmf* - OVMF firmware file or base name for booting with UEFI
   tcpserial=<port> - specify tcp serial port number
   qemuparams=<xyz> - specify custom parameters to QEMU
@@ -221,6 +222,7 @@ class BaseConfig(object):
         self.cleaned = False
         # Files to cleanup after run
         self.cleanup_files = []
+        self.qmp = None
         self.guest_agent = False
         self.guest_agent_sockpath = '/tmp/qga.sock'
 
@@ -536,6 +538,10 @@ to your build configuration.
                 self.qemu_opt_script += ' -vnc :0'
             elif arg == 'guestagent':
                 self.guest_agent = True
+            elif arg == "qmp":
+                self.qmp = "unix:qmp.sock"
+            elif arg.startswith("qmp="):
+                self.qmp = arg[len('qmp='):]
             elif arg.startswith('guestagent-sockpath='):
                 self.guest_agent_sockpath = '%s' % arg[len('guestagent-sockpath='):]
             elif arg.startswith('tcpserial='):
@@ -1406,6 +1412,10 @@ to your build configuration.
             self.qemu_opt += ' -device virtio-serial '
             self.qemu_opt += ' -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 '
 
+    def setup_qmp(self):
+        if self.qmp:
+            self.qemu_opt += " -qmp %s,server,nowait" % self.qmp
+
     def setup_vga(self):
         if self.nographic == True:
             if self.sdl == True:
@@ -1547,6 +1557,7 @@ to your build configuration.
             self.qemu_opt += " -snapshot"
 
         self.setup_guest_agent()
+        self.setup_qmp()
         self.setup_serial()
         self.setup_vga()
 
Signature · 1 line
2.34.1
#5 lore raw

PATCH v2 4/4 login.py: Proof of concept for screenshot testcases

In reply to Eilís 'pidge' Ní Fhlannagáin

This takes the work rburton did on image screenshot testing and expands it.

Right now this only works for qemux86-64. Some standardization of screensize/resolution needs to happen with runqemu params for other machines. There is an issue in qemux86-64 (and possibly others) where the screenswitch icon is only half present. This causes the test to fail.

This test takes a screendump of a qemu image (for now, just core-image-sato for qemux86-64), and compares it to an image we have on record. Some normalisation of the different qemu configs need to happen to be able to support all machines. Example, the qemuarm64 screen size is much larger than the qemux86-64.

The image we have on record contains a blanked out clock. We do the same blanking out process for the screenshot, so the images should have zero differences. If they do, we fail.

In order to enable this test, you will need meta-openembedded/meta-oe in your bblayers.conf and the following in local.conf:

IMAGE_CLASSES += "testimage"
TEST_SUITES = "login"
IMAGE_INSTALL:append = " python3-qemu-qmp " TESTIMAGEDEPENDS:append:qemuall = " imagemagick-native:do_populate_sysroot "

---
 .../core-image-sato-qemux86-64.png            | Bin 0 -> 46986 bytes
 meta/lib/oeqa/runtime/cases/login.py          |  34 ++++++++++++++++++
 2 files changed, 34 insertions(+)
 create mode 100644 meta/files/image-tests/core-image-sato-qemux86-64.png
 create mode 100644 meta/lib/oeqa/runtime/cases/login.py
meta/files/image-tests/core-image-sato-qemux86-64.png new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..5a3e46169dca6d118c003d29a59c59588cee7463
GIT binary patch
meta/lib/oeqa/runtime/cases/login.py new file +34 -0
new file mode 100644
index 00000000000..9e351fdeb29
@@ -0,0 +1,34 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+import subprocess
+from oeqa.runtime.case import OERuntimeTestCase
+import tempfile
+from oeqa.runtime.decorator.package import OEHasPackage
+
+class LoginTest(OERuntimeTestCase):
+
+    @OEHasPackage(['python3-qemu-qmp'])
+    def test_screenshot(self):
+        if self.td.get('MACHINE') != "qemux86-64":
+            self.fail
+
+        if bb.utils.which(os.getenv('PATH'), "convert") is not None and bb.utils.which(os.getenv('PATH'), "compare") is not None:
+            with tempfile.NamedTemporaryFile(prefix="oeqa-screenshot-login", suffix=".png") as t:
+                ret = self.target.runner.run_monitor("screendump", args={"filename": t.name, "format":"png"})
+                # Use the meta-oe version of convert, along with it's suffix
+                cmd = "convert.im7 {0} -fill white -draw 'rectangle 600,10 640,22' {1}".format(t.name, t.name)
+                proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+                output, error = proc.communicate()
+
+                # Use the meta-oe version of compare, along with it's suffix
+                cmd = "compare.im7 -metric MSE {0} {1}/meta/files/image-tests/core-image-sato-{2}.png /dev/null".format(t.name, self.td.get('COREBASE'), self.td.get('MACHINE'))
+                proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+                output, error = proc.communicate()
+                diff=float(error.decode('utf-8').replace("(", "").replace(")","").split()[1])
+                self.assertEqual(0, diff, "Screenshot diff is %s." % (str(diff)))
+        else:
+            self.fail
Signature · 1 line
2.34.1
#6 lore raw

Re: [OE-core] [PATCHv2 4/4] login.py: Proof of concept for screenshot testcases

Quoted 47 lines · Eilís 'pidge' Ní Fhlannagáin

This takes the work rburton did on image screenshot testing and expands it.

Right now this only works for qemux86-64. Some standardization of screensize/resolution needs to happen with runqemu params for other machines. There is an issue in qemux86-64 (and possibly others) where the screenswitch icon is only half present. This causes the test to fail.

This test takes a screendump of a qemu image (for now, just core-image-sato for qemux86-64), and compares it to an image we have on record. Some normalisation of the different qemu configs need to happen to be able to support all machines. Example, the qemuarm64 screen size is much larger than the qemux86-64.

The image we have on record contains a blanked out clock. We do the same blanking out process for the screenshot, so the images should have zero differences. If they do, we fail.

In order to enable this test, you will need meta-openembedded/meta-oe in your bblayers.conf and the following in local.conf:

IMAGE_CLASSES += "testimage"
TEST_SUITES = "login"
IMAGE_INSTALL:append = " python3-qemu-qmp " TESTIMAGEDEPENDS:append:qemuall = " imagemagick-native:do_populate_sysroot "

---
 .../core-image-sato-qemux86-64.png            | Bin 0 -> 46986 bytes
 meta/lib/oeqa/runtime/cases/login.py          |  34 ++++++++++++++++++
 2 files changed, 34 insertions(+)
 create mode 100644 meta/files/image-tests/core-image-sato-qemux86-64.png
 create mode 100644 meta/lib/oeqa/runtime/cases/login.py
meta/lib/oeqa/runtime/cases/login.py new file +4 -0
new file mode 100644
index 00000000000..9e351fdeb29
@@ -0,0 +1,34 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT

meta/lib/oeqa/runtime/cases/login.py line 14

Quoted 10 lines
meta/lib/oeqa/runtime/cases/login.py new file +10 -0
+#
+
+import subprocess
+from oeqa.runtime.case import OERuntimeTestCase
+import tempfile
+from oeqa.runtime.decorator.package import OEHasPackage
+
+class LoginTest(OERuntimeTestCase):
+
+    @OEHasPackage(['python3-qemu-qmp'])

This only works if we have python3-qemu-qmp in the target? I thought the monitor was external?

meta/lib/oeqa/runtime/cases/login.py line 17

Quoted 3 lines
meta/lib/oeqa/runtime/cases/login.py new file +3 -0
+    def test_screenshot(self):
+        if self.td.get('MACHINE') != "qemux86-64":
+            self.fail

So we're going to throw failures on all other machines? Wouldn't a skip be more appropriate? Can we provide images for other machines?

Have a look at things like:
@skipIfNotMachine("qemux86-64", "tests are qemux86-64 specific currently")

meta/lib/oeqa/runtime/cases/login.py line 19

Quoted 2 lines
meta/lib/oeqa/runtime/cases/login.py new file +2 -0
+
+        if bb.utils.which(os.getenv('PATH'), "convert") is not None and bb.utils.which(os.getenv('PATH'), "compare") is not None:

This works for nativesdk-imagemagick but it would need convert+compare in HOSTTOOLS to work for a host binary. I'm torn on the best way to find that otherwise.

meta/lib/oeqa/runtime/cases/login.py line 25

Quoted 6 lines
meta/lib/oeqa/runtime/cases/login.py new file +6 -0
+            with tempfile.NamedTemporaryFile(prefix="oeqa-screenshot-login", suffix=".png") as t:
+                ret = self.target.runner.run_monitor("screendump", args={"filename": t.name, "format":"png"})
+                # Use the meta-oe version of convert, along with it's suffix
+                cmd = "convert.im7 {0} -fill white -draw 'rectangle 600,10 640,22' {1}".format(t.name, t.name)
+                proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+                output, error = proc.communicate()

You probably want subprocess.check_result()

meta/lib/oeqa/runtime/cases/login.py line 30

Quoted 5 lines
meta/lib/oeqa/runtime/cases/login.py new file +5 -0
+
+                # Use the meta-oe version of compare, along with it's suffix
+                cmd = "compare.im7 -metric MSE {0} {1}/meta/files/image-tests/core-image-sato-{2}.png /dev/null".format(t.name, self.td.get('COREBASE'), self.td.get('MACHINE'))
+                proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+                output, error = proc.communicate()

and check_output()

meta/lib/oeqa/runtime/cases/login.py line 34

Quoted 4 lines
meta/lib/oeqa/runtime/cases/login.py new file +4 -0
+                diff=float(error.decode('utf-8').replace("(", "").replace(")","").split()[1])
+                self.assertEqual(0, diff, "Screenshot diff is %s." % (str(diff)))
+        else:
+            self.fail

Wouldn't:

self.skipTest("could not find convert tool from imagemagick")

be better?

Right now I can't merge this. With skip()s I can at least merge whilst we sort the other pieces.

Cheers,

Richard

#3 lore raw

Re: [OE-core] [PATCHv2 2/4] python3-qemu-qmp: Add recipe

In reply to Eilís 'pidge' Ní Fhlannagáin
Quoted 22 lines · Eilís 'pidge' Ní Fhlannagáin

From: Ross Burton <[email protected]>

Add a recipe for python QEMU Monitor Protocol. This will be needed for qemu screendumps for qemu screendump testing.

---
 .../python/python3-qemu-qmp_0.0.2.bb               | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb
meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb new file +4 -0
new file mode 100644
index 00000000000..faa0cc69738
@@ -0,0 +1,14 @@
+SUMMARY = "asyncio library for communicating with QEMU Monitor Protocol (QMP) servers"
+LICENSE = "LGPLv2+ & GPLv2"
+LIC_FILES_CHKSUM = "file://LICENSE;md5=4cf66a4984120007c9881cc871cf49db \
+                    file://LICENSE_GPL2;md5=441c28d2cf86e15a37fa47e15a72fbac"

meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb line 14

Quoted 10 lines
meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb new file +10 -0
+
+SRC_URI[sha256sum] = "c918e9e3ae09abdf70c7ece67637a93ac4583d940bbf48d24ff77987f74f1b8b"
+
+inherit pypi python_setuptools_build_meta
+
+PYPI_PACKAGE = "qemu.qmp"
+
+DEPENDS += "python3-setuptools-scm-native"
+
+BBCLASSEXTEND = "native nativesdk"

None of the errors Alexandre reported originally from the autobuilder have been fixed here :(

(HOMEPAGE, missing maintainer entry and so on)

Also, I'm not sure what qmp is doing on the target with this recipe? Does qmp trigger remote command execution inside the guest? I'd previously assumed the framedump was from the external presentation rather than internal to the guest.

Cheers,

Richard

#7 lore raw

Re: [OE-core] [PATCHv2 4/4] login.py: Proof of concept for screenshot testcases

In reply to Richard Purdie
Quoted 5 lines · Richard Purdie
Quoted 2 lines

+ @OEHasPackage(['python3-qemu-qmp'])

This only works if we have python3-qemu-qmp in the target? I thought the monitor was external?

Yeah, that’s not needed. That recipe was in my prototype branch but it isn’t required when using run_monitor() as that handles the QMP itself. The recipe can be removed (or added to meta-python, as it is still useful to have).

Ross

#8 lore raw

Re: [OE-core] [PATCHv2 4/4] login.py: Proof of concept for screenshot testcases

In reply to Richard Purdie
Quoted 62 lines · Richard Purdie
Quoted 47 lines · Eilís 'pidge' Ní Fhlannagáin

This takes the work rburton did on image screenshot testing and expands it.

Right now this only works for qemux86-64. Some standardization of screensize/resolution needs to happen with runqemu params for other machines. There is an issue in qemux86-64 (and possibly others) where the screenswitch icon is only half present. This causes the test to fail.

This test takes a screendump of a qemu image (for now, just core-image-sato for qemux86-64), and compares it to an image we have on record. Some normalisation of the different qemu configs need to happen to be able to support all machines. Example, the qemuarm64 screen size is much larger than the qemux86-64.

The image we have on record contains a blanked out clock. We do the same blanking out process for the screenshot, so the images should have zero differences. If they do, we fail.

In order to enable this test, you will need meta-openembedded/meta-oe in your bblayers.conf and the following in local.conf:

IMAGE_CLASSES += "testimage"
TEST_SUITES = "login"
IMAGE_INSTALL:append = " python3-qemu-qmp " TESTIMAGEDEPENDS:append:qemuall = " imagemagick-native:do_populate_sysroot "

---
  .../core-image-sato-qemux86-64.png            | Bin 0 -> 46986 bytes
  meta/lib/oeqa/runtime/cases/login.py          |  34 ++++++++++++++++++
  2 files changed, 34 insertions(+)

create mode 100644 meta/files/image-tests/core-image-sato-qemux86-64.png create mode 100644 meta/lib/oeqa/runtime/cases/login.py

meta/lib/oeqa/runtime/cases/login.py new file +4 -0
new file mode 100644
index 00000000000..9e351fdeb29
@@ -0,0 +1,34 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT

meta/lib/oeqa/runtime/cases/login.py line 14

Quoted 10 lines
meta/lib/oeqa/runtime/cases/login.py new file +10 -0
+#
+
+import subprocess
+from oeqa.runtime.case import OERuntimeTestCase
+import tempfile
+from oeqa.runtime.decorator.package import OEHasPackage
+
+class LoginTest(OERuntimeTestCase):
+
+    @OEHasPackage(['python3-qemu-qmp'])

This only works if we have python3-qemu-qmp in the target? I thought the monitor was external?

Quoted 6 lines

meta/lib/oeqa/runtime/cases/login.py line 17

Quoted 3 lines
meta/lib/oeqa/runtime/cases/login.py new file +3 -0
+    def test_screenshot(self):
+        if self.td.get('MACHINE') != "qemux86-64":
+            self.fail

So we're going to throw failures on all other machines? Wouldn't a skip be more appropriate? Can we provide images for other machines?

I can get images for other machines, but it's going to take a bit to run everything. I've debug code to grab those images. The other option is to ensure the other qemu machines have something like QB_GRAPHICS = "-device virtio-gpu-pci, xres=800,yres=600" to normalise the resolutions

Quoted 10 lines

Have a look at things like:
@skipIfNotMachine("qemux86-64", "tests are qemux86-64 specific currently")

meta/lib/oeqa/runtime/cases/login.py line 19

Quoted 2 lines
meta/lib/oeqa/runtime/cases/login.py new file +2 -0
+
+        if bb.utils.which(os.getenv('PATH'), "convert") is not None and bb.utils.which(os.getenv('PATH'), "compare") is not None:

This works for nativesdk-imagemagick but it would need convert+compare in HOSTTOOLS to work for a host binary. I'm torn on the best way to find that otherwise.

Fixed to rely on the imagemagicks provided by TESTIMAGEDEPENDS:append.

Quoted 10 lines

meta/lib/oeqa/runtime/cases/login.py line 25

Quoted 7 lines
meta/lib/oeqa/runtime/cases/login.py new file +6 -0
+            with tempfile.NamedTemporaryFile(prefix="oeqa-screenshot-login", suffix=".png") as t:
+                ret = self.target.runner.run_monitor("screendump", args={"filename": t.name, "format":"png"})
+                # Use the meta-oe version of convert, along with it's suffix
+                cmd = "convert.im7 {0} -fill white -draw 'rectangle 600,10 640,22' {1}".format(t.name, t.name)
+                proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+                output, error = proc.communicate()

You probably want subprocess.check_result()

check_output, but yes, that works here.... but...

Quoted 7 lines

meta/lib/oeqa/runtime/cases/login.py line 30

Quoted 5 lines
meta/lib/oeqa/runtime/cases/login.py new file +5 -0
+
+                # Use the meta-oe version of compare, along with it's suffix
+                cmd = "compare.im7 -metric MSE {0} {1}/meta/files/image-tests/core-image-sato-{2}.png /dev/null".format(t.name, self.td.get('COREBASE'), self.td.get('MACHINE'))
+                proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+                output, error = proc.communicate()

and check_output()

...doesn't here. In this case, we should probably either use run() or Popen() because imagemagick returns non-failing output on stderr (which is what we need) as well stdout (which we don't really want). Using Popen allows us to do this a bit cleaner as opposed to check_output.

Quoted 20 lines

meta/lib/oeqa/runtime/cases/login.py line 34

Quoted 5 lines
meta/lib/oeqa/runtime/cases/login.py new file +4 -0
+                diff=float(error.decode('utf-8').replace("(", "").replace(")","").split()[1])
+                self.assertEqual(0, diff, "Screenshot diff is %s." % (str(diff)))
+        else:
+            self.fail

Wouldn't:

self.skipTest("could not find convert tool from imagemagick")

be better?

Right now I can't merge this. With skip()s I can at least merge whilst we sort the other pieces.

Quoted 1 line

Cheers,

Richard

Signature · 3 lines
Eilís 'pidge' Ní Fhlannagáin
BayLibre - At the Heart of Embedded Linux
www.baylibre.com
#9 lore raw

Re: [OE-core] [PATCHv2 4/4] login.py: Proof of concept for screenshot testcases

Quoted 67 lines · Eilís 'pidge' Ní Fhlannagáin
Quoted 66 lines · Richard Purdie
Quoted 47 lines · Eilís 'pidge' Ní Fhlannagáin

This takes the work rburton did on image screenshot testing and expands it.

Right now this only works for qemux86-64. Some standardization of screensize/resolution needs to happen with runqemu params for other machines. There is an issue in qemux86-64 (and possibly others) where the screenswitch icon is only half present. This causes the test to fail.

This test takes a screendump of a qemu image (for now, just core-image-sato for qemux86-64), and compares it to an image we have on record. Some normalisation of the different qemu configs need to happen to be able to support all machines. Example, the qemuarm64 screen size is much larger than the qemux86-64.

The image we have on record contains a blanked out clock. We do the same blanking out process for the screenshot, so the images should have zero differences. If they do, we fail.

In order to enable this test, you will need meta-openembedded/meta-oe in your bblayers.conf and the following in local.conf:

IMAGE_CLASSES += "testimage"
TEST_SUITES = "login"
IMAGE_INSTALL:append = " python3-qemu-qmp " TESTIMAGEDEPENDS:append:qemuall = " imagemagick-native:do_populate_sysroot "

---
  .../core-image-sato-qemux86-64.png            | Bin 0 -> 46986 bytes
  meta/lib/oeqa/runtime/cases/login.py          |  34 ++++++++++++++++++
  2 files changed, 34 insertions(+)

create mode 100644 meta/files/image-tests/core-image-sato-qemux86-64.png create mode 100644 meta/lib/oeqa/runtime/cases/login.py

meta/lib/oeqa/runtime/cases/login.py new file +4 -0
new file mode 100644
index 00000000000..9e351fdeb29
@@ -0,0 +1,34 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT

meta/lib/oeqa/runtime/cases/login.py line 14

Quoted 10 lines
meta/lib/oeqa/runtime/cases/login.py new file +10 -0
+#
+
+import subprocess
+from oeqa.runtime.case import OERuntimeTestCase
+import tempfile
+from oeqa.runtime.decorator.package import OEHasPackage
+
+class LoginTest(OERuntimeTestCase):
+
+    @OEHasPackage(['python3-qemu-qmp'])

This only works if we have python3-qemu-qmp in the target? I thought the monitor was external?

Quoted 3 lines
meta/lib/oeqa/runtime/cases/login.py new file +3 -0
+    def test_screenshot(self):
+        if self.td.get('MACHINE') != "qemux86-64":
+            self.fail
Quoted 7 lines
Quoted 2 lines

So we're going to throw failures on all other machines? Wouldn't a skip be more appropriate? Can we provide images for other machines?

I can get images for other machines, but it's going to take a bit to run everything. I've debug code to grab those images. The other option is to ensure the other qemu machines have something like QB_GRAPHICS = "-device virtio-gpu-pci, xres=800,yres=600" to normalise the resolutions

One step at a time I guess but I would like this to ultimately work with our commonly used machines.

Quoted 21 lines
Quoted 10 lines

Have a look at things like:
@skipIfNotMachine("qemux86-64", "tests are qemux86-64 specific currently")

meta/lib/oeqa/runtime/cases/login.py line 19

Quoted 2 lines
meta/lib/oeqa/runtime/cases/login.py new file +2 -0
+
+        if bb.utils.which(os.getenv('PATH'), "convert") is not None and bb.utils.which(os.getenv('PATH'), "compare") is not None:

This works for nativesdk-imagemagick but it would need convert+compare in HOSTTOOLS to work for a host binary. I'm torn on the best way to find that otherwise.

Fixed to rely on the imagemagicks provided by TESTIMAGEDEPENDS:append.

Quoted 8 lines
Quoted 7 lines
meta/lib/oeqa/runtime/cases/login.py new file +6 -0
+            with tempfile.NamedTemporaryFile(prefix="oeqa-screenshot-login", suffix=".png") as t:
+                ret = self.target.runner.run_monitor("screendump", args={"filename": t.name, "format":"png"})
+                # Use the meta-oe version of convert, along with it's suffix
+                cmd = "convert.im7 {0} -fill white -draw 'rectangle 600,10 640,22' {1}".format(t.name, t.name)
+                proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+                output, error = proc.communicate()
Quoted 4 lines
Quoted 2 lines

You probably want subprocess.check_result()

check_output, but yes, that works here.... but...

I meant check_call() since you don't use the output.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
meta/lib/oeqa/runtime/cases/login.py new file +5 -0
+
+                # Use the meta-oe version of compare, along with it's suffix
+                cmd = "compare.im7 -metric MSE {0} {1}/meta/files/image-tests/core-image-sato-{2}.png /dev/null".format(t.name, self.td.get('COREBASE'), self.td.get('MACHINE'))
+                proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+                output, error = proc.communicate()
Quoted 6 lines
Quoted 1 line

and check_output()

...doesn't here. In this case, we should probably either use run() or Popen() because imagemagick returns non-failing output on stderr (which is what we need) as well stdout (which we don't really want). Using Popen allows us to do this a bit cleaner as opposed to check_output.

You can use stderr=subprocess.STDOUT with check_output?

Cheers,

Richard

#10 lore raw

Re: [OE-core] [PATCHv2 4/4] login.py: Proof of concept for screenshot testcases

Quoted 71 lines · Richard Purdie
Quoted 70 lines · Eilís 'pidge' Ní Fhlannagáin
Quoted 68 lines · Richard Purdie
Quoted 47 lines · Eilís 'pidge' Ní Fhlannagáin

This takes the work rburton did on image screenshot testing and expands it.

Right now this only works for qemux86-64. Some standardization of screensize/resolution needs to happen with runqemu params for other machines. There is an issue in qemux86-64 (and possibly others) where the screenswitch icon is only half present. This causes the test to fail.

This test takes a screendump of a qemu image (for now, just core-image-sato for qemux86-64), and compares it to an image we have on record. Some normalisation of the different qemu configs need to happen to be able to support all machines. Example, the qemuarm64 screen size is much larger than the qemux86-64.

The image we have on record contains a blanked out clock. We do the same blanking out process for the screenshot, so the images should have zero differences. If they do, we fail.

In order to enable this test, you will need meta-openembedded/meta-oe in your bblayers.conf and the following in local.conf:

IMAGE_CLASSES += "testimage"
TEST_SUITES = "login"
IMAGE_INSTALL:append = " python3-qemu-qmp " TESTIMAGEDEPENDS:append:qemuall = " imagemagick-native:do_populate_sysroot "

---
   .../core-image-sato-qemux86-64.png            | Bin 0 -> 46986 bytes
   meta/lib/oeqa/runtime/cases/login.py          |  34 ++++++++++++++++++
   2 files changed, 34 insertions(+)

create mode 100644 meta/files/image-tests/core-image-sato-qemux86-64.png create mode 100644 meta/lib/oeqa/runtime/cases/login.py

meta/lib/oeqa/runtime/cases/login.py new file +4 -0
new file mode 100644
index 00000000000..9e351fdeb29
@@ -0,0 +1,34 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT

meta/lib/oeqa/runtime/cases/login.py line 14

Quoted 10 lines
meta/lib/oeqa/runtime/cases/login.py new file +10 -0
+#
+
+import subprocess
+from oeqa.runtime.case import OERuntimeTestCase
+import tempfile
+from oeqa.runtime.decorator.package import OEHasPackage
+
+class LoginTest(OERuntimeTestCase):
+
+    @OEHasPackage(['python3-qemu-qmp'])

This only works if we have python3-qemu-qmp in the target? I thought the monitor was external?

meta/lib/oeqa/runtime/cases/login.py line 17

Quoted 3 lines
meta/lib/oeqa/runtime/cases/login.py new file +3 -0
+    def test_screenshot(self):
+        if self.td.get('MACHINE') != "qemux86-64":
+            self.fail

So we're going to throw failures on all other machines? Wouldn't a skip be more appropriate? Can we provide images for other machines?

Quoted 7 lines
Quoted 4 lines

I can get images for other machines, but it's going to take a bit to run everything. I've debug code to grab those images. The other option is to ensure the other qemu machines have something like QB_GRAPHICS = "-device virtio-gpu-pci, xres=800,yres=600" to normalise the resolutions

One step at a time I guess but I would like this to ultimately work with our commonly used machines.

Agreed. I've some patches to the qemu configs that should get us what we want, but I need to test them.

Quoted 22 lines
Quoted 22 lines
Quoted 11 lines

Have a look at things like:
@skipIfNotMachine("qemux86-64", "tests are qemux86-64 specific currently")

meta/lib/oeqa/runtime/cases/login.py line 19

Quoted 2 lines
meta/lib/oeqa/runtime/cases/login.py new file +2 -0
+
+        if bb.utils.which(os.getenv('PATH'), "convert") is not None and bb.utils.which(os.getenv('PATH'), "compare") is not None:

This works for nativesdk-imagemagick but it would need convert+compare in HOSTTOOLS to work for a host binary. I'm torn on the best way to find that otherwise.

Fixed to rely on the imagemagicks provided by TESTIMAGEDEPENDS:append.

Quoted 8 lines
Quoted 7 lines
meta/lib/oeqa/runtime/cases/login.py new file +6 -0
+            with tempfile.NamedTemporaryFile(prefix="oeqa-screenshot-login", suffix=".png") as t:
+                ret = self.target.runner.run_monitor("screendump", args={"filename": t.name, "format":"png"})
+                # Use the meta-oe version of convert, along with it's suffix
+                cmd = "convert.im7 {0} -fill white -draw 'rectangle 600,10 640,22' {1}".format(t.name, t.name)
+                proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+                output, error = proc.communicate()
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

You probably want subprocess.check_result()

check_output, but yes, that works here.... but...

I meant check_call() since you don't use the output.

Ah, I have it as check_output. I can submit a patch to move it to that function with my qemu patches, if what we have is ok.

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

meta/lib/oeqa/runtime/cases/login.py line 30

Quoted 6 lines
meta/lib/oeqa/runtime/cases/login.py new file +5 -0
+
+                # Use the meta-oe version of compare, along with it's suffix
+                cmd = "compare.im7 -metric MSE {0} {1}/meta/files/image-tests/core-image-sato-{2}.png /dev/null".format(t.name, self.td.get('COREBASE'), self.td.get('MACHINE'))
+                proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+                output, error = proc.communicate()

and check_output()

Quoted 7 lines
Quoted 4 lines

...doesn't here. In this case, we should probably either use run() or Popen() because imagemagick returns non-failing output on stderr (which is what we need) as well stdout (which we don't really want). Using Popen allows us to do this a bit cleaner as opposed to check_output.

You can use stderr=subprocess.STDOUT with check_output?

Yes, but doing so mixes stdout and stderr from my recollection. With compare (and a few other imagemagick tools) stdout is used for image output and stderr is (as in this case), the numerical diff of the image, which I wanted to avoid.

In this case, we don't happen to see the output because we're directing to /dev/null, but something like:

compare -metric MSE /home/pidge/oeqa-screenshot-loginzm8bokga.png ../meta/files/image-tests/core-image-sato-qemux86-64.png png:-

makes a stdout/stderr mess if we use check_output. I eventually want to play with piping the output to other things, to get some more info (identify is an option), for error tracking and such.

-e

Quoted 5 lines

Cheers,

Richard

Signature · 3 lines
Eilís 'pidge' Ní Fhlannagáin
BayLibre - At the Heart of Embedded Linux
www.baylibre.com