Proof of concept for screenshot testcases
v2 cover Proof of concept for screenshot testcases
This takes the work rburton did on image screenshot testing and expands it.
Right now this only works for qemux86-64. Some standardization of screensize/resolution needs to happen with runqemu params for other machines. There is an issue in qemux86-64 (and possibly others) where the screenswitch icon is only half present. This causes the test to fail.
This test takes a screendump of a qemu image (for now, just core-image-sato for qemux86-64), and compares it to an image we have on record. Some normalisation of the different qemu configs need to happen to be able to support all machines. Example, the qemuarm64 screen size is much larger than the qemux86-64.
The image we have on record contains a blanked out clock. We do the same blanking out process for the screenshot, so the images should have zero differences. If they do, we fail.
In order to enable this test, you will need meta-openembedded/meta-oe in your bblayers.conf and the following in local.conf:
IMAGE_CLASSES += "testimage"TEST_SUITES = "login"
IMAGE_INSTALL:append = " python3-qemu-qmp " TESTIMAGEDEPENDS:append:qemuall = " imagemagick-native:do_populate_sysroot "
Eilís 'pidge' Ní Fhlannagáin (1):
login.py: Proof of concept for screenshot testcases
Ross Burton (3):
qemurunner: remove unused import
python3-qemu-qmp: Add recipe
runqemu: add qmp socket support
.../core-image-sato-qemux86-64.png | Bin 0 -> 46986 bytes meta/lib/oeqa/runtime/cases/login.py | 34 ++++++++++++++++++ meta/lib/oeqa/utils/qemurunner.py | 1 - .../python/python3-qemu-qmp_0.0.2.bb | 14 ++++++++ scripts/runqemu | 11 ++++++ 5 files changed, 59 insertions(+), 1 deletion(-) create mode 100644 meta/files/image-tests/core-image-sato-qemux86-64.png create mode 100644 meta/lib/oeqa/runtime/cases/login.py create mode 100644 meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb
Signature · 1 line
2.34.1
PATCH v2 1/4 qemurunner: remove unused import
From: Ross Burton <[email protected]>
Removes unused logging import
--- meta/lib/oeqa/utils/qemurunner.py | 1 - 1 file changed, 1 deletion(-)
meta/lib/oeqa/utils/qemurunner.py +0 -1
@@ -19,7 +19,6 @@ import errno import string import threading import codecs -import logging import tempfile from collections import defaultdict import importlib
Signature · 1 line
2.34.1
PATCH v2 2/4 python3-qemu-qmp: Add recipe
In reply to Eilís 'pidge' Ní FhlannagáinFrom: Ross Burton <[email protected]>
Add a recipe for python QEMU Monitor Protocol. This will be needed for qemu screendumps for qemu screendump testing.
--- .../python/python3-qemu-qmp_0.0.2.bb | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb
meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb new file +14 -0
@@ -0,0 +1,14 @@ +SUMMARY = "asyncio library for communicating with QEMU Monitor Protocol (QMP) servers" +LICENSE = "LGPLv2+ & GPLv2" +LIC_FILES_CHKSUM = "file://LICENSE;md5=4cf66a4984120007c9881cc871cf49db \ + file://LICENSE_GPL2;md5=441c28d2cf86e15a37fa47e15a72fbac" + +SRC_URI[sha256sum] = "c918e9e3ae09abdf70c7ece67637a93ac4583d940bbf48d24ff77987f74f1b8b" + +inherit pypi python_setuptools_build_meta + +PYPI_PACKAGE = "qemu.qmp" + +DEPENDS += "python3-setuptools-scm-native" + +BBCLASSEXTEND = "native nativesdk"
Signature · 1 line
2.34.1
Re: [OE-core] [PATCHv2 2/4] python3-qemu-qmp: Add recipe
Quoted 22 lines · Eilís 'pidge' Ní Fhlannagáin
From: Ross Burton <[email protected]>
Add a recipe for python QEMU Monitor Protocol. This will be needed for qemu screendumps for qemu screendump testing.
Signed-off-by Ross Burton <[email protected]>Signed-off-by Eilís 'pidge' Ní Fhlannagáin <[email protected]>--- .../python/python3-qemu-qmp_0.0.2.bb | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bbmeta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb new file +4
-0@@ -0,0 +1,14 @@ +SUMMARY = "asyncio library for communicating with QEMU Monitor Protocol (QMP) servers" +LICENSE = "LGPLv2+ & GPLv2" +LIC_FILES_CHKSUM = "file://LICENSE;md5=4cf66a4984120007c9881cc871cf49db \ + file://LICENSE_GPL2;md5=441c28d2cf86e15a37fa47e15a72fbac"
meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb line 14
Quoted 10 lines
meta/recipes-devtools/python/python3-qemu-qmp_0.0.2.bb new file +10
-0+ +SRC_URI[sha256sum] = "c918e9e3ae09abdf70c7ece67637a93ac4583d940bbf48d24ff77987f74f1b8b" + +inherit pypi python_setuptools_build_meta + +PYPI_PACKAGE = "qemu.qmp" + +DEPENDS += "python3-setuptools-scm-native" + +BBCLASSEXTEND = "native nativesdk"
None of the errors Alexandre reported originally from the autobuilder have been fixed here :(
(HOMEPAGE, missing maintainer entry and so on)
Also, I'm not sure what qmp is doing on the target with this recipe? Does qmp trigger remote command execution inside the guest? I'd previously assumed the framedump was from the external presentation rather than internal to the guest.
Cheers,
Richard
PATCH v2 3/4 runqemu: add qmp socket support
In reply to Eilís 'pidge' Ní FhlannagáinFrom: Ross Burton <[email protected]>
Add support for qmp sockets and defaults to unix:qmp.sock if unspecified
--- scripts/runqemu | 11 +++++++++++ 1 file changed, 11 insertions(+)
scripts/runqemu +11 -0
@@ -84,6 +84,7 @@ of the following environment variables (in any order): publicvnc - enable a VNC server open to all hosts audio - enable audio guestagent - enable guest agent communication + qmp=<path> - create a QMP socket (defaults to unix:qmp.sock if unspecified) [*/]ovmf* - OVMF firmware file or base name for booting with UEFI tcpserial=<port> - specify tcp serial port number qemuparams=<xyz> - specify custom parameters to QEMU
@@ -221,6 +222,7 @@ class BaseConfig(object): self.cleaned = False # Files to cleanup after run self.cleanup_files = [] + self.qmp = None self.guest_agent = False self.guest_agent_sockpath = '/tmp/qga.sock'
@@ -536,6 +538,10 @@ to your build configuration. self.qemu_opt_script += ' -vnc :0' elif arg == 'guestagent': self.guest_agent = True + elif arg == "qmp": + self.qmp = "unix:qmp.sock" + elif arg.startswith("qmp="): + self.qmp = arg[len('qmp='):] elif arg.startswith('guestagent-sockpath='): self.guest_agent_sockpath = '%s' % arg[len('guestagent-sockpath='):] elif arg.startswith('tcpserial='):
@@ -1406,6 +1412,10 @@ to your build configuration. self.qemu_opt += ' -device virtio-serial ' self.qemu_opt += ' -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 ' + def setup_qmp(self): + if self.qmp: + self.qemu_opt += " -qmp %s,server,nowait" % self.qmp + def setup_vga(self): if self.nographic == True: if self.sdl == True:
@@ -1547,6 +1557,7 @@ to your build configuration. self.qemu_opt += " -snapshot" self.setup_guest_agent() + self.setup_qmp() self.setup_serial() self.setup_vga()
Signature · 1 line
2.34.1
PATCH v2 4/4 login.py: Proof of concept for screenshot testcases
In reply to Eilís 'pidge' Ní FhlannagáinThis takes the work rburton did on image screenshot testing and expands it.
Right now this only works for qemux86-64. Some standardization of screensize/resolution needs to happen with runqemu params for other machines. There is an issue in qemux86-64 (and possibly others) where the screenswitch icon is only half present. This causes the test to fail.
This test takes a screendump of a qemu image (for now, just core-image-sato for qemux86-64), and compares it to an image we have on record. Some normalisation of the different qemu configs need to happen to be able to support all machines. Example, the qemuarm64 screen size is much larger than the qemux86-64.
The image we have on record contains a blanked out clock. We do the same blanking out process for the screenshot, so the images should have zero differences. If they do, we fail.
In order to enable this test, you will need meta-openembedded/meta-oe in your bblayers.conf and the following in local.conf:
IMAGE_CLASSES += "testimage"TEST_SUITES = "login"
IMAGE_INSTALL:append = " python3-qemu-qmp " TESTIMAGEDEPENDS:append:qemuall = " imagemagick-native:do_populate_sysroot "
--- .../core-image-sato-qemux86-64.png | Bin 0 -> 46986 bytes meta/lib/oeqa/runtime/cases/login.py | 34 ++++++++++++++++++ 2 files changed, 34 insertions(+) create mode 100644 meta/files/image-tests/core-image-sato-qemux86-64.png create mode 100644 meta/lib/oeqa/runtime/cases/login.py
meta/files/image-tests/core-image-sato-qemux86-64.png new file binary +0 -0
meta/lib/oeqa/runtime/cases/login.py new file +34 -0
@@ -0,0 +1,34 @@ +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: MIT +# + +import subprocess +from oeqa.runtime.case import OERuntimeTestCase +import tempfile +from oeqa.runtime.decorator.package import OEHasPackage + +class LoginTest(OERuntimeTestCase): + + @OEHasPackage(['python3-qemu-qmp']) + def test_screenshot(self): + if self.td.get('MACHINE') != "qemux86-64": + self.fail + + if bb.utils.which(os.getenv('PATH'), "convert") is not None and bb.utils.which(os.getenv('PATH'), "compare") is not None: + with tempfile.NamedTemporaryFile(prefix="oeqa-screenshot-login", suffix=".png") as t: + ret = self.target.runner.run_monitor("screendump", args={"filename": t.name, "format":"png"}) + # Use the meta-oe version of convert, along with it's suffix + cmd = "convert.im7 {0} -fill white -draw 'rectangle 600,10 640,22' {1}".format(t.name, t.name) + proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + output, error = proc.communicate() + + # Use the meta-oe version of compare, along with it's suffix + cmd = "compare.im7 -metric MSE {0} {1}/meta/files/image-tests/core-image-sato-{2}.png /dev/null".format(t.name, self.td.get('COREBASE'), self.td.get('MACHINE')) + proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + output, error = proc.communicate() + diff=float(error.decode('utf-8').replace("(", "").replace(")","").split()[1]) + self.assertEqual(0, diff, "Screenshot diff is %s." % (str(diff))) + else: + self.fail
Signature · 1 line
2.34.1
Re: [OE-core] [PATCHv2 4/4] login.py: Proof of concept for screenshot testcases
Quoted 47 lines · Eilís 'pidge' Ní Fhlannagáin
This takes the work rburton did on image screenshot testing and expands it.
Right now this only works for qemux86-64. Some standardization of screensize/resolution needs to happen with runqemu params for other machines. There is an issue in qemux86-64 (and possibly others) where the screenswitch icon is only half present. This causes the test to fail.
This test takes a screendump of a qemu image (for now, just core-image-sato for qemux86-64), and compares it to an image we have on record. Some normalisation of the different qemu configs need to happen to be able to support all machines. Example, the qemuarm64 screen size is much larger than the qemux86-64.
The image we have on record contains a blanked out clock. We do the same blanking out process for the screenshot, so the images should have zero differences. If they do, we fail.
In order to enable this test, you will need meta-openembedded/meta-oe in your bblayers.conf and the following in local.conf:
IMAGE_CLASSES+= "testimage"TEST_SUITES= "login"
IMAGE_INSTALL:append = " python3-qemu-qmp " TESTIMAGEDEPENDS:append:qemuall = " imagemagick-native:do_populate_sysroot "Signed-off-by Eilís 'pidge' Ní Fhlannagáin <[email protected]>Co-authored-by Ross Burton <[email protected]>Co-authored-by Eilís 'pidge' Ní Fhlannagáin <[email protected]>--- .../core-image-sato-qemux86-64.png | Bin 0 -> 46986 bytes meta/lib/oeqa/runtime/cases/login.py | 34 ++++++++++++++++++ 2 files changed, 34 insertions(+) create mode 100644 meta/files/image-tests/core-image-sato-qemux86-64.png create mode 100644 meta/lib/oeqa/runtime/cases/login.pymeta/lib/oeqa/runtime/cases/login.py new file +4
-0@@ -0,0 +1,34 @@ +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: MIT
meta/lib/oeqa/runtime/cases/login.py line 14
Quoted 10 lines
meta/lib/oeqa/runtime/cases/login.py new file +10
-0+# + +import subprocess +from oeqa.runtime.case import OERuntimeTestCase +import tempfile +from oeqa.runtime.decorator.package import OEHasPackage + +class LoginTest(OERuntimeTestCase): + + @OEHasPackage(['python3-qemu-qmp'])
This only works if we have python3-qemu-qmp in the target? I thought the monitor was external?
meta/lib/oeqa/runtime/cases/login.py line 17
Quoted 3 lines
meta/lib/oeqa/runtime/cases/login.py new file +3
-0+ def test_screenshot(self): + if self.td.get('MACHINE') != "qemux86-64": + self.fail
So we're going to throw failures on all other machines? Wouldn't a skip be more appropriate? Can we provide images for other machines?
Have a look at things like:
@skipIfNotMachine("qemux86-64", "tests are qemux86-64 specific currently")
meta/lib/oeqa/runtime/cases/login.py line 19
Quoted 2 lines
meta/lib/oeqa/runtime/cases/login.py new file +2
-0+ + if bb.utils.which(os.getenv('PATH'), "convert") is not None and bb.utils.which(os.getenv('PATH'), "compare") is not None:
This works for nativesdk-imagemagick but it would need convert+compare in HOSTTOOLS to work for a host binary. I'm torn on the best way to find that otherwise.
meta/lib/oeqa/runtime/cases/login.py line 25
Quoted 6 lines
meta/lib/oeqa/runtime/cases/login.py new file +6
-0+ with tempfile.NamedTemporaryFile(prefix="oeqa-screenshot-login", suffix=".png") as t: + ret = self.target.runner.run_monitor("screendump", args={"filename": t.name, "format":"png"}) + # Use the meta-oe version of convert, along with it's suffix + cmd = "convert.im7 {0} -fill white -draw 'rectangle 600,10 640,22' {1}".format(t.name, t.name) + proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + output, error = proc.communicate()
You probably want subprocess.check_result()
meta/lib/oeqa/runtime/cases/login.py line 30
Quoted 5 lines
meta/lib/oeqa/runtime/cases/login.py new file +5
-0+ + # Use the meta-oe version of compare, along with it's suffix + cmd = "compare.im7 -metric MSE {0} {1}/meta/files/image-tests/core-image-sato-{2}.png /dev/null".format(t.name, self.td.get('COREBASE'), self.td.get('MACHINE')) + proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + output, error = proc.communicate()
and check_output()
meta/lib/oeqa/runtime/cases/login.py line 34
Quoted 4 lines
meta/lib/oeqa/runtime/cases/login.py new file +4
-0+ diff=float(error.decode('utf-8').replace("(", "").replace(")","").split()[1]) + self.assertEqual(0, diff, "Screenshot diff is %s." % (str(diff))) + else: + self.fail
Wouldn't:
self.skipTest("could not find convert tool from imagemagick")
be better?
Right now I can't merge this. With skip()s I can at least merge whilst we sort the other pieces.
Cheers,
Richard
Re: [OE-core] [PATCHv2 4/4] login.py: Proof of concept for screenshot testcases
Quoted 5 lines · Richard Purdie
Quoted 2 lines
+ @OEHasPackage(['python3-qemu-qmp'])
This only works if we have python3-qemu-qmp in the target? I thought the monitor was external?
Yeah, that’s not needed. That recipe was in my prototype branch but it isn’t required when using run_monitor() as that handles the QMP itself. The recipe can be removed (or added to meta-python, as it is still useful to have).
Ross
Re: [OE-core] [PATCHv2 4/4] login.py: Proof of concept for screenshot testcases
In reply to Richard PurdieQuoted 62 lines · Richard Purdie
Quoted 47 lines · Eilís 'pidge' Ní Fhlannagáin
This takes the work rburton did on image screenshot testing and expands it.
Right now this only works for qemux86-64. Some standardization of screensize/resolution needs to happen with runqemu params for other machines. There is an issue in qemux86-64 (and possibly others) where the screenswitch icon is only half present. This causes the test to fail.
This test takes a screendump of a qemu image (for now, just core-image-sato for qemux86-64), and compares it to an image we have on record. Some normalisation of the different qemu configs need to happen to be able to support all machines. Example, the qemuarm64 screen size is much larger than the qemux86-64.
The image we have on record contains a blanked out clock. We do the same blanking out process for the screenshot, so the images should have zero differences. If they do, we fail.
In order to enable this test, you will need meta-openembedded/meta-oe in your bblayers.conf and the following in local.conf:
IMAGE_CLASSES+= "testimage"TEST_SUITES= "login"
IMAGE_INSTALL:append = " python3-qemu-qmp " TESTIMAGEDEPENDS:append:qemuall = " imagemagick-native:do_populate_sysroot "Signed-off-by Eilís 'pidge' Ní Fhlannagáin <[email protected]>Co-authored-by Ross Burton <[email protected]>Co-authored-by Eilís 'pidge' Ní Fhlannagáin <[email protected]>--- .../core-image-sato-qemux86-64.png | Bin 0 -> 46986 bytes meta/lib/oeqa/runtime/cases/login.py | 34 ++++++++++++++++++ 2 files changed, 34 insertions(+)create mode 100644 meta/files/image-tests/core-image-sato-qemux86-64.png create mode 100644
meta/lib/oeqa/runtime/cases/login.pymeta/lib/oeqa/runtime/cases/login.py new file +4
-0@@ -0,0 +1,34 @@ +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: MITmeta/lib/oeqa/runtime/cases/login.py line 14
Quoted 10 lines
meta/lib/oeqa/runtime/cases/login.py new file +10
-0+# + +import subprocess +from oeqa.runtime.case import OERuntimeTestCase +import tempfile +from oeqa.runtime.decorator.package import OEHasPackage + +class LoginTest(OERuntimeTestCase): + + @OEHasPackage(['python3-qemu-qmp'])This only works if we have python3-qemu-qmp in the target? I thought the monitor was external?
Quoted 6 lines
meta/lib/oeqa/runtime/cases/login.py line 17
Quoted 3 lines
meta/lib/oeqa/runtime/cases/login.py new file +3
-0+ def test_screenshot(self): + if self.td.get('MACHINE') != "qemux86-64": + self.failSo we're going to throw failures on all other machines? Wouldn't a skip be more appropriate? Can we provide images for other machines?
I can get images for other machines, but it's going to take a bit to run everything. I've debug code to grab those images. The other option is to ensure the other qemu machines have something like QB_GRAPHICS = "-device virtio-gpu-pci, xres=800,yres=600" to normalise the resolutions
Quoted 10 lines
Have a look at things like:
@skipIfNotMachine("qemux86-64", "tests are qemux86-64 specific currently")meta/lib/oeqa/runtime/cases/login.py line 19
Quoted 2 lines
meta/lib/oeqa/runtime/cases/login.py new file +2
-0+ + if bb.utils.which(os.getenv('PATH'), "convert") is not None and bb.utils.which(os.getenv('PATH'), "compare") is not None:This works for nativesdk-imagemagick but it would need convert+compare in HOSTTOOLS to work for a host binary. I'm torn on the best way to find that otherwise.
Fixed to rely on the imagemagicks provided by TESTIMAGEDEPENDS:append.
Quoted 10 lines
meta/lib/oeqa/runtime/cases/login.py line 25
Quoted 7 lines
meta/lib/oeqa/runtime/cases/login.py new file +6
-0+ with tempfile.NamedTemporaryFile(prefix="oeqa-screenshot-login", suffix=".png") as t: + ret = self.target.runner.run_monitor("screendump", args={"filename": t.name, "format":"png"}) + # Use the meta-oe version of convert, along with it's suffix + cmd = "convert.im7 {0} -fill white -draw 'rectangle 600,10 640,22' {1}".format(t.name, t.name) + proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + output, error = proc.communicate()You probably want
subprocess.check_result()
check_output, but yes, that works here.... but...
Quoted 7 lines
meta/lib/oeqa/runtime/cases/login.py line 30
Quoted 5 lines
meta/lib/oeqa/runtime/cases/login.py new file +5
-0+ + # Use the meta-oe version of compare, along with it's suffix + cmd = "compare.im7 -metric MSE {0} {1}/meta/files/image-tests/core-image-sato-{2}.png /dev/null".format(t.name, self.td.get('COREBASE'), self.td.get('MACHINE')) + proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + output, error = proc.communicate()and
check_output()
...doesn't here. In this case, we should probably either use run() or Popen() because imagemagick returns non-failing output on stderr (which is what we need) as well stdout (which we don't really want). Using Popen allows us to do this a bit cleaner as opposed to check_output.
Quoted 20 lines
meta/lib/oeqa/runtime/cases/login.py line 34
Quoted 5 lines
meta/lib/oeqa/runtime/cases/login.py new file +4
-0+ diff=float(error.decode('utf-8').replace("(", "").replace(")","").split()[1]) + self.assertEqual(0, diff, "Screenshot diff is %s." % (str(diff))) + else: + self.failWouldn't:
self.skipTest("could not find convert tool from imagemagick")be better?
Right now I can't merge this. With skip()s I can at least merge whilst we sort the other pieces.
Quoted 1 line
Cheers,
Richard
Signature · 3 lines
Eilís 'pidge' Ní Fhlannagáin BayLibre - At the Heart of Embedded Linux www.baylibre.com
Re: [OE-core] [PATCHv2 4/4] login.py: Proof of concept for screenshot testcases
Quoted 67 lines · Eilís 'pidge' Ní Fhlannagáin
Quoted 66 lines · Richard Purdie
Quoted 47 lines · Eilís 'pidge' Ní Fhlannagáin
This takes the work rburton did on image screenshot testing and expands it.
Right now this only works for qemux86-64. Some standardization of screensize/resolution needs to happen with runqemu params for other machines. There is an issue in qemux86-64 (and possibly others) where the screenswitch icon is only half present. This causes the test to fail.
This test takes a screendump of a qemu image (for now, just core-image-sato for qemux86-64), and compares it to an image we have on record. Some normalisation of the different qemu configs need to happen to be able to support all machines. Example, the qemuarm64 screen size is much larger than the qemux86-64.
The image we have on record contains a blanked out clock. We do the same blanking out process for the screenshot, so the images should have zero differences. If they do, we fail.
In order to enable this test, you will need meta-openembedded/meta-oe in your bblayers.conf and the following in local.conf:
IMAGE_CLASSES+= "testimage"TEST_SUITES= "login"
IMAGE_INSTALL:append = " python3-qemu-qmp " TESTIMAGEDEPENDS:append:qemuall = " imagemagick-native:do_populate_sysroot "Signed-off-by Eilís 'pidge' Ní Fhlannagáin <[email protected]>Co-authored-by Ross Burton <[email protected]>Co-authored-by Eilís 'pidge' Ní Fhlannagáin <[email protected]>--- .../core-image-sato-qemux86-64.png | Bin 0 -> 46986 bytes meta/lib/oeqa/runtime/cases/login.py | 34 ++++++++++++++++++ 2 files changed, 34 insertions(+)create mode 100644 meta/files/image-tests/core-image-sato-qemux86-64.png create mode 100644
meta/lib/oeqa/runtime/cases/login.pymeta/lib/oeqa/runtime/cases/login.py new file +4
-0@@ -0,0 +1,34 @@ +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: MITmeta/lib/oeqa/runtime/cases/login.py line 14
Quoted 10 lines
meta/lib/oeqa/runtime/cases/login.py new file +10
-0+# + +import subprocess +from oeqa.runtime.case import OERuntimeTestCase +import tempfile +from oeqa.runtime.decorator.package import OEHasPackage + +class LoginTest(OERuntimeTestCase): + + @OEHasPackage(['python3-qemu-qmp'])This only works if we have python3-qemu-qmp in the target? I thought the monitor was external?
Quoted 3 lines
meta/lib/oeqa/runtime/cases/login.py new file +3
-0+ def test_screenshot(self): + if self.td.get('MACHINE') != "qemux86-64": + self.fail
Quoted 7 lines
Quoted 2 lines
So we're going to throw failures on all other machines? Wouldn't a skip be more appropriate? Can we provide images for other machines?
I can get images for other machines, but it's going to take a bit to run everything. I've debug code to grab those images. The other option is to ensure the other qemu machines have something like
QB_GRAPHICS= "-device virtio-gpu-pci, xres=800,yres=600" to normalise the resolutions
One step at a time I guess but I would like this to ultimately work with our commonly used machines.
Quoted 21 lines
Quoted 10 lines
Have a look at things like:
@skipIfNotMachine("qemux86-64", "tests are qemux86-64 specific currently")meta/lib/oeqa/runtime/cases/login.py line 19
Quoted 2 lines
meta/lib/oeqa/runtime/cases/login.py new file +2
-0+ + if bb.utils.which(os.getenv('PATH'), "convert") is not None and bb.utils.which(os.getenv('PATH'), "compare") is not None:This works for nativesdk-imagemagick but it would need convert+compare in HOSTTOOLS to work for a host binary. I'm torn on the best way to find that otherwise.
Fixed to rely on the imagemagicks provided by TESTIMAGEDEPENDS:append.
Quoted 8 lines
Quoted 7 lines
meta/lib/oeqa/runtime/cases/login.py new file +6
-0+ with tempfile.NamedTemporaryFile(prefix="oeqa-screenshot-login", suffix=".png") as t: + ret = self.target.runner.run_monitor("screendump", args={"filename": t.name, "format":"png"}) + # Use the meta-oe version of convert, along with it's suffix + cmd = "convert.im7 {0} -fill white -draw 'rectangle 600,10 640,22' {1}".format(t.name, t.name) + proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + output, error = proc.communicate()
Quoted 4 lines
Quoted 2 lines
You probably want
subprocess.check_result()
check_output, but yes, that works here.... but...
I meant check_call() since you don't use the output.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
meta/lib/oeqa/runtime/cases/login.py new file +5
-0+ + # Use the meta-oe version of compare, along with it's suffix + cmd = "compare.im7 -metric MSE {0} {1}/meta/files/image-tests/core-image-sato-{2}.png /dev/null".format(t.name, self.td.get('COREBASE'), self.td.get('MACHINE')) + proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + output, error = proc.communicate()
Quoted 6 lines
Quoted 1 line
and
check_output()...doesn't here. In this case, we should probably either use
run()orPopen()because imagemagick returns non-failing output on stderr (which is what we need) as well stdout (which we don't really want). Using Popen allows us to do this a bit cleaner as opposed tocheck_output.
You can use stderr=subprocess.STDOUT with check_output?
Cheers,
Richard
Re: [OE-core] [PATCHv2 4/4] login.py: Proof of concept for screenshot testcases
Quoted 71 lines · Richard Purdie
Quoted 70 lines · Eilís 'pidge' Ní Fhlannagáin
Quoted 68 lines · Richard Purdie
Quoted 47 lines · Eilís 'pidge' Ní Fhlannagáin
This takes the work rburton did on image screenshot testing and expands it.
Right now this only works for qemux86-64. Some standardization of screensize/resolution needs to happen with runqemu params for other machines. There is an issue in qemux86-64 (and possibly others) where the screenswitch icon is only half present. This causes the test to fail.
This test takes a screendump of a qemu image (for now, just core-image-sato for qemux86-64), and compares it to an image we have on record. Some normalisation of the different qemu configs need to happen to be able to support all machines. Example, the qemuarm64 screen size is much larger than the qemux86-64.
The image we have on record contains a blanked out clock. We do the same blanking out process for the screenshot, so the images should have zero differences. If they do, we fail.
In order to enable this test, you will need meta-openembedded/meta-oe in your bblayers.conf and the following in local.conf:
IMAGE_CLASSES+= "testimage"TEST_SUITES= "login"
IMAGE_INSTALL:append = " python3-qemu-qmp " TESTIMAGEDEPENDS:append:qemuall = " imagemagick-native:do_populate_sysroot "Signed-off-by Eilís 'pidge' Ní Fhlannagáin <[email protected]>Co-authored-by Ross Burton <[email protected]>Co-authored-by Eilís 'pidge' Ní Fhlannagáin <[email protected]>--- .../core-image-sato-qemux86-64.png | Bin 0 -> 46986 bytes meta/lib/oeqa/runtime/cases/login.py | 34 ++++++++++++++++++ 2 files changed, 34 insertions(+)create mode 100644 meta/files/image-tests/core-image-sato-qemux86-64.png create mode 100644
meta/lib/oeqa/runtime/cases/login.pymeta/lib/oeqa/runtime/cases/login.py new file +4
-0@@ -0,0 +1,34 @@ +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: MITmeta/lib/oeqa/runtime/cases/login.py line 14
Quoted 10 lines
meta/lib/oeqa/runtime/cases/login.py new file +10
-0+# + +import subprocess +from oeqa.runtime.case import OERuntimeTestCase +import tempfile +from oeqa.runtime.decorator.package import OEHasPackage + +class LoginTest(OERuntimeTestCase): + + @OEHasPackage(['python3-qemu-qmp'])This only works if we have python3-qemu-qmp in the target? I thought the monitor was external?
meta/lib/oeqa/runtime/cases/login.py line 17
Quoted 3 lines
meta/lib/oeqa/runtime/cases/login.py new file +3
-0+ def test_screenshot(self): + if self.td.get('MACHINE') != "qemux86-64": + self.failSo we're going to throw failures on all other machines? Wouldn't a skip be more appropriate? Can we provide images for other machines?
Quoted 7 lines
Quoted 4 lines
I can get images for other machines, but it's going to take a bit to run everything. I've debug code to grab those images. The other option is to ensure the other qemu machines have something like
QB_GRAPHICS= "-device virtio-gpu-pci, xres=800,yres=600" to normalise the resolutionsOne step at a time I guess but I would like this to ultimately work with our commonly used machines.
Agreed. I've some patches to the qemu configs that should get us what we want, but I need to test them.
Quoted 22 lines
Quoted 22 lines
Quoted 11 lines
Have a look at things like:
@skipIfNotMachine("qemux86-64", "tests are qemux86-64 specific currently")meta/lib/oeqa/runtime/cases/login.py line 19
Quoted 2 lines
meta/lib/oeqa/runtime/cases/login.py new file +2
-0+ + if bb.utils.which(os.getenv('PATH'), "convert") is not None and bb.utils.which(os.getenv('PATH'), "compare") is not None:This works for nativesdk-imagemagick but it would need convert+compare in HOSTTOOLS to work for a host binary. I'm torn on the best way to find that otherwise.
Fixed to rely on the imagemagicks provided by TESTIMAGEDEPENDS:append.
Quoted 8 lines
Quoted 7 lines
meta/lib/oeqa/runtime/cases/login.py new file +6
-0+ with tempfile.NamedTemporaryFile(prefix="oeqa-screenshot-login", suffix=".png") as t: + ret = self.target.runner.run_monitor("screendump", args={"filename": t.name, "format":"png"}) + # Use the meta-oe version of convert, along with it's suffix + cmd = "convert.im7 {0} -fill white -draw 'rectangle 600,10 640,22' {1}".format(t.name, t.name) + proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + output, error = proc.communicate()
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
You probably want
subprocess.check_result()
check_output, but yes, that works here.... but...I meant
check_call()since you don't use the output.
Ah, I have it as check_output. I can submit a patch to move it to that function with my qemu patches, if what we have is ok.
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
meta/lib/oeqa/runtime/cases/login.py line 30
Quoted 6 lines
meta/lib/oeqa/runtime/cases/login.py new file +5
-0+ + # Use the meta-oe version of compare, along with it's suffix + cmd = "compare.im7 -metric MSE {0} {1}/meta/files/image-tests/core-image-sato-{2}.png /dev/null".format(t.name, self.td.get('COREBASE'), self.td.get('MACHINE')) + proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + output, error = proc.communicate()and
check_output()
Quoted 7 lines
Quoted 4 lines
...doesn't here. In this case, we should probably either use
run()orPopen()because imagemagick returns non-failing output on stderr (which is what we need) as well stdout (which we don't really want). Using Popen allows us to do this a bit cleaner as opposed tocheck_output.You can use stderr=subprocess.STDOUT with
check_output?
Yes, but doing so mixes stdout and stderr from my recollection. With compare (and a few other imagemagick tools) stdout is used for image output and stderr is (as in this case), the numerical diff of the image, which I wanted to avoid.
In this case, we don't happen to see the output because we're directing to /dev/null, but something like:
compare -metric MSE /home/pidge/oeqa-screenshot-loginzm8bokga.png ../meta/files/image-tests/core-image-sato-qemux86-64.png png:-
makes a stdout/stderr mess if we use check_output. I eventually want to play with piping the output to other things, to get some more info (identify is an option), for error tracking and such.
-e
Quoted 5 lines
Cheers,
Richard
Signature · 3 lines
Eilís 'pidge' Ní Fhlannagáin BayLibre - At the Heart of Embedded Linux www.baylibre.com