Image screenshot testing proof of concept
cover Image screenshot testing proof of concept
This patch series is based off of earlier work from Ross Burton to enable QMP based screenshot testing.
Eilís 'pidge' Ní Fhlannagáin (1):
login.py: Proof of concept for screenshot testcases
Ross Burton (3):
qemurunner: remove unused import
python3-qemu-qmp: Add recipe
runqemu: add qmp socket support
.../core-image-sato-qemux86-64.png | Bin 0 -> 46986 bytes meta/lib/oeqa/runtime/cases/login.py | 39 ++++++++++++++++++ meta/lib/oeqa/utils/qemurunner.py | 1 - .../python/python3-qemu-qmp_0.0.2.bb | 14 +++++++ scripts/runqemu | 11 +++++ 5 files changed, 64 insertions(+), 1 deletion(-) create mode 100644 meta/files/image-tests/core-image-sato-qemux86-64.png create mode 100644 meta/lib/oeqa/runtime/cases/login.py create mode 100644 meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb
Signature · 1 line
2.34.1
PATCH 1/4 qemurunner: remove unused import
From: Ross Burton <[email protected]>
Removes unused logging import
--- meta/lib/oeqa/utils/qemurunner.py | 1 - 1 file changed, 1 deletion(-)
meta/lib/oeqa/utils/qemurunner.py +0 -1
@@ -19,7 +19,6 @@ import errno import string import threading import codecs -import logging import tempfile from collections import defaultdict import importlib
Signature · 1 line
2.34.1
PATCH 2/4 python3-qemu-qmp: Add recipe
In reply to Eilís 'pidge' Ní FhlannagáinFrom: Ross Burton <[email protected]>
Add a recipe for python QEMU Monitor Protocol. This will be needed for qemu screendumps for qemu screendump testing.
--- .../python/python3-qemu-qmp_0.0.2.bb | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb
meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb new file +14 -0
@@ -0,0 +1,14 @@ +SUMMARY = "asyncio library for communicating with QEMU Monitor Protocol (QMP) servers" +LICENSE = "LGPLv2+ & GPLv2" +LIC_FILES_CHKSUM = "file://LICENSE;md5=4cf66a4984120007c9881cc871cf49db \ + file://LICENSE_GPL2;md5=441c28d2cf86e15a37fa47e15a72fbac" + +SRC_URI[sha256sum] = "c918e9e3ae09abdf70c7ece67637a93ac4583d940bbf48d24ff77987f74f1b8b" + +inherit pypi python_setuptools_build_meta + +PYPI_PACKAGE = "qemu.qmp" + +DEPENDS += "python3-setuptools-scm-native" + +BBCLASSEXTEND = "native nativesdk"
Signature · 1 line
2.34.1
Re: [OE-core] [PATCH 2/4] python3-qemu-qmp: Add recipe
This will also need a maintainer entry in:
https://git.openembedded.org/openembedded-core/tree/meta/conf/distro/include/maintainers.inc
Quoted 51 lines · Eilís 'pidge' Ní Fhlannagáin
From: Ross Burton <[email protected]>
Add a recipe for python QEMU Monitor Protocol. This will be needed for qemu screendumps for qemu screendump testing.
Signed-off-by Ross Burton <[email protected]>Signed-off-by Eilís 'pidge' Ní Fhlannagáin <[email protected]>--- .../python/python3-qemu-qmp_0.0.2.bb | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bbmeta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb new file +1
-0@@ -0,0 +1,14 @@ +SUMMARY = "asyncio library for communicating with QEMU Monitor Protocol(QMP) servers"
meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb new file +3
-0+LICENSE = "LGPLv2+ & GPLv2" +LIC_FILES_CHKSUM = "file://LICENSE;md5=4cf66a4984120007c9881cc871cf49db \ +file://LICENSE_GPL2;md5=441c28d2cf86e15a37fa47e15a72fbac"
meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb new file +2
-0+ +SRC_URI[sha256sum] ="
c918e9e3ae09abdf70c7ece67637a93ac4583d940bbf48d24ff77987f74f1b8b"meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb new file +8
-1+ +inherit pypi python_setuptools_build_meta + +PYPI_PACKAGE = "qemu.qmp" + +DEPENDS += "python3-setuptools-scm-native" + +BBCLASSEXTEND = "native nativesdk" --2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#191960):
https://lists.openembedded.org/g/openembedded-core/message/191960
Mute This Topic: https://lists.openembedded.org/mt/103035077/924729
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH 2/4] python3-qemu-qmp: Add recipe
Quoted 3 lines · Tim Orling
This will also need a maintainer entry in:
https://git.openembedded.org/openembedded-core/tree/meta/conf/distro/include/maintainers.inc
Also:
stdio: WARNING: python3-qemu-qmp-0.0.2-r0 do_recipe_qa: QA Issue: Recipe python3-qemu-qmp in /home/pokybuild/yocto-worker/qemuarm64-armhost/build/meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb does not contain a HOMEPAGE. Please add an entry. [missing-metadata]
stdio: WARNING: python3-qemu-qmp-0.0.2-r0 do_recipe_qa: QA Issue: Recipe python3-qemu-qmp in /home/pokybuild/yocto-worker/qemuarm64-armhost/build/meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb does not have an assigned maintainer. Please add an entry into meta/conf/distro/include/maintainers.inc. [missing-maintainer]
stdio: WARNING: python3-qemu-qmp-native-0.0.2-r0 do_recipe_qa: QA Issue: Recipe python3-qemu-qmp-native in /home/pokybuild/yocto-worker/qemuarm64-armhost/build/meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb does not contain a HOMEPAGE. Please add an entry. [missing-metadata]
WARNING: python3-qemu-qmp-0.0.2-r0 do_package_qa: QA Issue: Recipe LICENSE includes obsolete licenses GPLv2 LGPLv2+ [obsolete-license]
Quoted 56 lines
Quoted 37 lines · Eilís 'pidge' Ní Fhlannagáin
From: Ross Burton <[email protected]>
Add a recipe for python QEMU Monitor Protocol. This will be needed for qemu screendumps for qemu screendump testing.
Signed-off-by Ross Burton <[email protected]>Signed-off-by Eilís 'pidge' Ní Fhlannagáin <[email protected]>--- .../python/python3-qemu-qmp_0.0.2.bb | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bbmeta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb new file +1
-0@@ -0,0 +1,14 @@ +SUMMARY = "asyncio library for communicating with QEMU Monitor Protocol(QMP) servers"
meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb new file +3
-0+LICENSE = "LGPLv2+ & GPLv2" +LIC_FILES_CHKSUM = "file://LICENSE;md5=4cf66a4984120007c9881cc871cf49db \ +file://LICENSE_GPL2;md5=441c28d2cf86e15a37fa47e15a72fbac"
meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb new file +2
-0+ +SRC_URI[sha256sum] ="
c918e9e3ae09abdf70c7ece67637a93ac4583d940bbf48d24ff77987f74f1b8b"meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb new file +8
-1+ +inherit pypi python_setuptools_build_meta + +PYPI_PACKAGE = "qemu.qmp" + +DEPENDS += "python3-setuptools-scm-native" + +BBCLASSEXTEND = "native nativesdk" --Quoted 6 lines
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#191969): https://lists.openembedded.org/g/openembedded-core/message/191969
Mute This Topic: https://lists.openembedded.org/mt/103035077/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 3/4 runqemu: add qmp socket support
In reply to Eilís 'pidge' Ní FhlannagáinFrom: Ross Burton <[email protected]>
Add support for qmp sockets and defaults to unix:qmp.sock if unspecified
--- scripts/runqemu | 11 +++++++++++ 1 file changed, 11 insertions(+)
scripts/runqemu +11 -0
@@ -84,6 +84,7 @@ of the following environment variables (in any order): publicvnc - enable a VNC server open to all hosts audio - enable audio guestagent - enable guest agent communication + qmp=<path> - create a QMP socket (defaults to unix:qmp.sock if unspecified) [*/]ovmf* - OVMF firmware file or base name for booting with UEFI tcpserial=<port> - specify tcp serial port number qemuparams=<xyz> - specify custom parameters to QEMU
@@ -221,6 +222,7 @@ class BaseConfig(object): self.cleaned = False # Files to cleanup after run self.cleanup_files = [] + self.qmp = None self.guest_agent = False self.guest_agent_sockpath = '/tmp/qga.sock'
@@ -536,6 +538,10 @@ to your build configuration. self.qemu_opt_script += ' -vnc :0' elif arg == 'guestagent': self.guest_agent = True + elif arg == "qmp": + self.qmp = "unix:qmp.sock" + elif arg.startswith("qmp="): + self.qmp = arg[len('qmp='):] elif arg.startswith('guestagent-sockpath='): self.guest_agent_sockpath = '%s' % arg[len('guestagent-sockpath='):] elif arg.startswith('tcpserial='):
@@ -1406,6 +1412,10 @@ to your build configuration. self.qemu_opt += ' -device virtio-serial ' self.qemu_opt += ' -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 ' + def setup_qmp(self): + if self.qmp: + self.qemu_opt += " -qmp %s,server,nowait" % self.qmp + def setup_vga(self): if self.nographic == True: if self.sdl == True:
@@ -1547,6 +1557,7 @@ to your build configuration. self.qemu_opt += " -snapshot" self.setup_guest_agent() + self.setup_qmp() self.setup_serial() self.setup_vga()
Signature · 1 line
2.34.1
PATCH 4/4 login.py: Proof of concept for screenshot testcases
In reply to Eilís 'pidge' Ní FhlannagáinThis takes the work rburton did on image screenshot testing and expands it. The two caveats here:
1. We expect ImageMagick as a host tool for the time being. There is a question as to do we bring ImageMagick into core for this or do we require meta-oe for these tests.
2. Right now this only works for qemux86-64. Some standardization of screensize/resolution needs to happen with runqemu params for other machines.
This test takes a screendump of a qemu image (for now, just core-image-sato for qemux86-64), and compares it to an image we have on record. Some normalisation of the different qemu configs need to happen to be able to support all machines. Example, the qemuarm64 screen size is much larger than the qemux86-64.
The image we have on record contains a blanked out clock. We do the same blanking out process for the screenshot, so the images should have zero differences. If they do, we fail.
--- .../core-image-sato-qemux86-64.png | Bin 0 -> 46986 bytes meta/lib/oeqa/runtime/cases/login.py | 39 ++++++++++++++++++ 2 files changed, 39 insertions(+) create mode 100644 meta/files/image-tests/core-image-sato-qemux86-64.png create mode 100644 meta/lib/oeqa/runtime/cases/login.py
meta/files/image-tests/core-image-sato-qemux86-64.png new file binary +0 -0
meta/lib/oeqa/runtime/cases/login.py new file +39 -0
@@ -0,0 +1,39 @@ +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: MIT +# + +import subprocess +from oeqa.runtime.case import OERuntimeTestCase +from oeqa.utils.commands import get_bb_vars +import tempfile + +class LoginTest(OERuntimeTestCase): + + + @classmethod + def setUpClass(cls): + cls.bbvars = get_bb_vars(['COREBASE', 'MACHINE']) + + + def test_screenshot(self): + with tempfile.NamedTemporaryFile(prefix="oeqa-screenshot-login", suffix=".png") as t: + ret = self.target.runner.run_monitor("screendump", args={"filename": t.name, "format":"png"}) + + if self.bbvars['MACHINE'] == "qemux86-64": + cmd = "/usr/bin/convert {0} -fill white -draw \"rectangle 600,10 640,22\" {0}".format(t.name) + subprocess.run(cmd.split()) + else: + self.fail + + cmd = "/usr/bin/compare -metric MSE {0} {1}/meta/files/image-tests/core-image-sato-{2}.png /dev/null".format(t.name, self.bbvars['COREBASE'], self.bbvars['MACHINE']) + proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + output, error = proc.communicate() + + diff=error.decode('utf-8').replace("(", "").replace(")","").split()[1] + bb.warn(diff) + if float(diff) == 0: + bb.note("Screenshot diff is %s." % (diff)) + else: + self.fail
Signature · 1 line
2.34.1