Skip to contentSkip to conversation

Image screenshot testing proof of concept

PATCH Superseded Eilís 'pidge' Ní Fhlannagáin 3 participants 7 messages Dec 7, 2023 – Dec 10, 2023

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover Image screenshot testing proof of concept

This patch series is based off of earlier work from Ross Burton to enable QMP based screenshot testing.

Eilís 'pidge' Ní Fhlannagáin (1):
login.py: Proof of concept for screenshot testcases

Ross Burton (3):
qemurunner: remove unused import
python3-qemu-qmp: Add recipe
runqemu: add qmp socket support

 .../core-image-sato-qemux86-64.png            | Bin 0 -> 46986 bytes
 meta/lib/oeqa/runtime/cases/login.py          |  39 ++++++++++++++++++
 meta/lib/oeqa/utils/qemurunner.py             |   1 -
 .../python/python3-qemu-qmp_0.0.2.bb          |  14 +++++++
 scripts/runqemu                               |  11 +++++
 5 files changed, 64 insertions(+), 1 deletion(-)
 create mode 100644 meta/files/image-tests/core-image-sato-qemux86-64.png
 create mode 100644 meta/lib/oeqa/runtime/cases/login.py
 create mode 100644 meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb
Signature · 1 line
2.34.1
#1 lore raw

PATCH 1/4 qemurunner: remove unused import

From: Ross Burton <[email protected]>

Removes unused logging import

---
 meta/lib/oeqa/utils/qemurunner.py | 1 -
 1 file changed, 1 deletion(-)
meta/lib/oeqa/utils/qemurunner.py +0 -1
index 29fe2719762..ee367078000 100644
@@ -19,7 +19,6 @@ import errno
 import string
 import threading
 import codecs
-import logging
 import tempfile
 from collections import defaultdict
 import importlib
Signature · 1 line
2.34.1
#2 lore raw

PATCH 2/4 python3-qemu-qmp: Add recipe

In reply to Eilís 'pidge' Ní Fhlannagáin

From: Ross Burton <[email protected]>

Add a recipe for python QEMU Monitor Protocol. This will be needed for qemu screendumps for qemu screendump testing.

---
 .../python/python3-qemu-qmp_0.0.2.bb               | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb
meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb new file +14 -0
new file mode 100644
index 00000000000..faa0cc69738
@@ -0,0 +1,14 @@
+SUMMARY = "asyncio library for communicating with QEMU Monitor Protocol (QMP) servers"
+LICENSE = "LGPLv2+ & GPLv2"
+LIC_FILES_CHKSUM = "file://LICENSE;md5=4cf66a4984120007c9881cc871cf49db \
+                    file://LICENSE_GPL2;md5=441c28d2cf86e15a37fa47e15a72fbac"
+
+SRC_URI[sha256sum] = "c918e9e3ae09abdf70c7ece67637a93ac4583d940bbf48d24ff77987f74f1b8b"
+
+inherit pypi python_setuptools_build_meta
+
+PYPI_PACKAGE = "qemu.qmp"
+
+DEPENDS += "python3-setuptools-scm-native"
+
+BBCLASSEXTEND = "native nativesdk"
Signature · 1 line
2.34.1
#3 lore raw

Re: [OE-core] [PATCH 2/4] python3-qemu-qmp: Add recipe

This will also need a maintainer entry in:
https://git.openembedded.org/openembedded-core/tree/meta/conf/distro/include/maintainers.inc

Quoted 51 lines · Eilís 'pidge' Ní Fhlannagáin

From: Ross Burton <[email protected]>

Add a recipe for python QEMU Monitor Protocol. This will be needed for qemu screendumps for qemu screendump testing.

---
 .../python/python3-qemu-qmp_0.0.2.bb               | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb
meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb new file +1 -0
new file mode 100644
index 00000000000..faa0cc69738
@@ -0,0 +1,14 @@
+SUMMARY = "asyncio library for communicating with QEMU Monitor Protocol

(QMP) servers"

meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb new file +3 -0
+LICENSE = "LGPLv2+ & GPLv2"
+LIC_FILES_CHKSUM = "file://LICENSE;md5=4cf66a4984120007c9881cc871cf49db \
+

file://LICENSE_GPL2;md5=441c28d2cf86e15a37fa47e15a72fbac"

meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb new file +2 -0
+
+SRC_URI[sha256sum] =

"c918e9e3ae09abdf70c7ece67637a93ac4583d940bbf48d24ff77987f74f1b8b"

meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb new file +8 -1
+
+inherit pypi python_setuptools_build_meta
+
+PYPI_PACKAGE = "qemu.qmp"
+
+DEPENDS += "python3-setuptools-scm-native"
+
+BBCLASSEXTEND = "native nativesdk"
--

2.34.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#191960):
https://lists.openembedded.org/g/openembedded-core/message/191960
Mute This Topic: https://lists.openembedded.org/mt/103035077/924729
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#4 lore raw

Re: [OE-core] [PATCH 2/4] python3-qemu-qmp: Add recipe

Quoted 3 lines · Tim Orling

This will also need a maintainer entry in:
https://git.openembedded.org/openembedded-core/tree/meta/conf/distro/include/maintainers.inc

Also:
stdio: WARNING: python3-qemu-qmp-0.0.2-r0 do_recipe_qa: QA Issue: Recipe python3-qemu-qmp in /home/pokybuild/yocto-worker/qemuarm64-armhost/build/meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb does not contain a HOMEPAGE. Please add an entry. [missing-metadata]
stdio: WARNING: python3-qemu-qmp-0.0.2-r0 do_recipe_qa: QA Issue: Recipe python3-qemu-qmp in /home/pokybuild/yocto-worker/qemuarm64-armhost/build/meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb does not have an assigned maintainer. Please add an entry into meta/conf/distro/include/maintainers.inc. [missing-maintainer]
stdio: WARNING: python3-qemu-qmp-native-0.0.2-r0 do_recipe_qa: QA Issue: Recipe python3-qemu-qmp-native in /home/pokybuild/yocto-worker/qemuarm64-armhost/build/meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb does not contain a HOMEPAGE. Please add an entry. [missing-metadata]
WARNING: python3-qemu-qmp-0.0.2-r0 do_package_qa: QA Issue: Recipe LICENSE includes obsolete licenses GPLv2 LGPLv2+ [obsolete-license]

Quoted 56 lines
Quoted 37 lines · Eilís 'pidge' Ní Fhlannagáin

From: Ross Burton <[email protected]>

Add a recipe for python QEMU Monitor Protocol. This will be needed for qemu screendumps for qemu screendump testing.

---
 .../python/python3-qemu-qmp_0.0.2.bb               | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb
meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb new file +1 -0
new file mode 100644
index 00000000000..faa0cc69738
@@ -0,0 +1,14 @@
+SUMMARY = "asyncio library for communicating with QEMU Monitor Protocol

(QMP) servers"

meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb new file +3 -0
+LICENSE = "LGPLv2+ & GPLv2"
+LIC_FILES_CHKSUM = "file://LICENSE;md5=4cf66a4984120007c9881cc871cf49db \
+

file://LICENSE_GPL2;md5=441c28d2cf86e15a37fa47e15a72fbac"

meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb new file +2 -0
+
+SRC_URI[sha256sum] =

"c918e9e3ae09abdf70c7ece67637a93ac4583d940bbf48d24ff77987f74f1b8b"

meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb new file +8 -1
+
+inherit pypi python_setuptools_build_meta
+
+PYPI_PACKAGE = "qemu.qmp"
+
+DEPENDS += "python3-setuptools-scm-native"
+
+BBCLASSEXTEND = "native nativesdk"
--
Quoted 6 lines

2.34.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#191969): https://lists.openembedded.org/g/openembedded-core/message/191969
Mute This Topic: https://lists.openembedded.org/mt/103035077/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#5 lore raw

PATCH 3/4 runqemu: add qmp socket support

In reply to Eilís 'pidge' Ní Fhlannagáin

From: Ross Burton <[email protected]>

Add support for qmp sockets and defaults to unix:qmp.sock if unspecified

---
 scripts/runqemu | 11 +++++++++++
 1 file changed, 11 insertions(+)
scripts/runqemu +11 -0
index 18aeb7f5f0c..6a5a6451daf 100755
@@ -84,6 +84,7 @@ of the following environment variables (in any order):
     publicvnc - enable a VNC server open to all hosts
     audio - enable audio
     guestagent - enable guest agent communication
+    qmp=<path> - create a QMP socket (defaults to unix:qmp.sock if unspecified)
     [*/]ovmf* - OVMF firmware file or base name for booting with UEFI
   tcpserial=<port> - specify tcp serial port number
   qemuparams=<xyz> - specify custom parameters to QEMU
@@ -221,6 +222,7 @@ class BaseConfig(object):
         self.cleaned = False
         # Files to cleanup after run
         self.cleanup_files = []
+        self.qmp = None
         self.guest_agent = False
         self.guest_agent_sockpath = '/tmp/qga.sock'
 
@@ -536,6 +538,10 @@ to your build configuration.
                 self.qemu_opt_script += ' -vnc :0'
             elif arg == 'guestagent':
                 self.guest_agent = True
+            elif arg == "qmp":
+                self.qmp = "unix:qmp.sock"
+            elif arg.startswith("qmp="):
+                self.qmp = arg[len('qmp='):]
             elif arg.startswith('guestagent-sockpath='):
                 self.guest_agent_sockpath = '%s' % arg[len('guestagent-sockpath='):]
             elif arg.startswith('tcpserial='):
@@ -1406,6 +1412,10 @@ to your build configuration.
             self.qemu_opt += ' -device virtio-serial '
             self.qemu_opt += ' -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 '
 
+    def setup_qmp(self):
+        if self.qmp:
+            self.qemu_opt += " -qmp %s,server,nowait" % self.qmp
+
     def setup_vga(self):
         if self.nographic == True:
             if self.sdl == True:
@@ -1547,6 +1557,7 @@ to your build configuration.
             self.qemu_opt += " -snapshot"
 
         self.setup_guest_agent()
+        self.setup_qmp()
         self.setup_serial()
         self.setup_vga()
 
Signature · 1 line
2.34.1
#6 lore raw

PATCH 4/4 login.py: Proof of concept for screenshot testcases

In reply to Eilís 'pidge' Ní Fhlannagáin

This takes the work rburton did on image screenshot testing and expands it. The two caveats here:

1. We expect ImageMagick as a host tool for the time being. There is a question as to do we bring ImageMagick into core for this or do we require meta-oe for these tests.

2. Right now this only works for qemux86-64. Some standardization of screensize/resolution needs to happen with runqemu params for other machines.

This test takes a screendump of a qemu image (for now, just core-image-sato for qemux86-64), and compares it to an image we have on record. Some normalisation of the different qemu configs need to happen to be able to support all machines. Example, the qemuarm64 screen size is much larger than the qemux86-64.

The image we have on record contains a blanked out clock. We do the same blanking out process for the screenshot, so the images should have zero differences. If they do, we fail.

---
 .../core-image-sato-qemux86-64.png            | Bin 0 -> 46986 bytes
 meta/lib/oeqa/runtime/cases/login.py          |  39 ++++++++++++++++++
 2 files changed, 39 insertions(+)
 create mode 100644 meta/files/image-tests/core-image-sato-qemux86-64.png
 create mode 100644 meta/lib/oeqa/runtime/cases/login.py
meta/files/image-tests/core-image-sato-qemux86-64.png new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..5a3e46169dca6d118c003d29a59c59588cee7463
GIT binary patch
meta/lib/oeqa/runtime/cases/login.py new file +39 -0
new file mode 100644
index 00000000000..198d02caa5e
@@ -0,0 +1,39 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+import subprocess
+from oeqa.runtime.case import OERuntimeTestCase
+from oeqa.utils.commands import get_bb_vars
+import tempfile
+
+class LoginTest(OERuntimeTestCase):
+
+
+    @classmethod
+    def setUpClass(cls):
+        cls.bbvars = get_bb_vars(['COREBASE', 'MACHINE'])
+
+
+    def test_screenshot(self):
+        with tempfile.NamedTemporaryFile(prefix="oeqa-screenshot-login", suffix=".png") as t:
+            ret = self.target.runner.run_monitor("screendump", args={"filename": t.name, "format":"png"})
+            
+            if self.bbvars['MACHINE'] == "qemux86-64":
+                cmd = "/usr/bin/convert {0} -fill white -draw \"rectangle 600,10 640,22\" {0}".format(t.name)
+                subprocess.run(cmd.split())
+            else:
+                self.fail
+            
+            cmd = "/usr/bin/compare -metric MSE {0} {1}/meta/files/image-tests/core-image-sato-{2}.png /dev/null".format(t.name, self.bbvars['COREBASE'], self.bbvars['MACHINE'])
+            proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+            output, error = proc.communicate()
+
+            diff=error.decode('utf-8').replace("(", "").replace(")","").split()[1]
+            bb.warn(diff)
+            if float(diff) == 0:
+                bb.note("Screenshot diff is %s." % (diff))
+            else:
+                self.fail
Signature · 1 line
2.34.1