login.py: Proof of concept for screenshot testcases
PATCH v3 1/1 login.py: Proof of concept for screenshot testcases
This takes the work rburton did on image screenshot testing and expands it.
Right now this only works for qemux86-64. Some standardization of screensize/resolution needs to happen with runqemu params for other machines. There is an issue in qemux86-64 (and possibly others) where the screenswitch/home icon is only half present. This causes the test to fail. I'm unsure if this is a QMP/qemu issue but I've certainly seen the half-rendered icon, so it's not Imagemagick.
This test takes a screendump of a qemu image (for now, just core-image-sato for qemux86-64), and compares it to an image we have on record. Some normalisation of the different qemu configs need to happen to be able to support all machines. Example, the qemuarm64 screen size is much larger than the qemux86-64.
The image we have on record contains a blanked out clock. We do the same blanking out process for the screenshot, so the images should have zero differences. If they do, we fail.
In order to enable this test, you will need meta-openembedded/meta-oe in your bblayers.conf and the following in local.conf:
IMAGE_CLASSES += "testimage"TEST_SUITES = "login"
TESTIMAGEDEPENDS:append:qemuall = " imagemagick-native:do_populate_sysroot "
--- .../core-image-sato-qemux86-64.png | Bin 0 -> 46986 bytes meta/lib/oeqa/runtime/cases/login.py | 42 ++++++++++++++++++ 2 files changed, 42 insertions(+) create mode 100644 meta/files/image-tests/core-image-sato-qemux86-64.png create mode 100644 meta/lib/oeqa/runtime/cases/login.py
meta/files/image-tests/core-image-sato-qemux86-64.png new file binary +0 -0
meta/lib/oeqa/runtime/cases/login.py new file +42 -0
42 lines not on this page · raw message
Signature · 1 line
2.34.1
Hello,
Please also fix 2/4 and always resend the whole series, else the burden to cherry-pick from the multiple version of the series and my memory is not what it used to be.
Quoted 971 lines · Eil�s 'pidge' N� Fhlannag�in
This takes the work rburton did on image screenshot testing and expands it.
Right now this only works for qemux86-64. Some standardization of screensize/resolution needs to happen with runqemu params for other machines. There is an issue in qemux86-64 (and possibly others) where the screenswitch/home icon is only half present. This causes the test to fail. I'm unsure if this is a QMP/qemu issue but I've certainly seen the half-rendered icon, so it's not Imagemagick.
This test takes a screendump of a qemu image (for now, just core-image-sato for qemux86-64), and compares it to an image we have on record. Some normalisation of the different qemu configs need to happen to be able to support all machines. Example, the qemuarm64 screen size is much larger than the qemux86-64.
The image we have on record contains a blanked out clock. We do the same blanking out process for the screenshot, so the images should have zero differences. If they do, we fail.
In order to enable this test, you will need meta-openembedded/meta-oe in your bblayers.conf and the following in local.conf:
IMAGE_CLASSES+= "testimage"TEST_SUITES= "login"
TESTIMAGEDEPENDS:append:qemuall = " imagemagick-native:do_populate_sysroot "Signed-off-by Eil�s 'pidge' N� Fhlannag�in <[email protected]>Co-authored-by Ross Burton <[email protected]>Co-authored-by Eil�s 'pidge' N� Fhlannag�in <[email protected]>--- .../core-image-sato-qemux86-64.png | Bin 0 -> 46986 bytes meta/lib/oeqa/runtime/cases/login.py | 42 ++++++++++++++++++ 2 files changed, 42 insertions(+) create mode 100644 meta/files/image-tests/core-image-sato-qemux86-64.png create mode 100644 meta/lib/oeqa/runtime/cases/login.pymeta/files/image-tests/core-image-sato-qemux86-64.png new file binary +0
-0meta/lib/oeqa/runtime/cases/login.py new file +42
-0@@ -0,0 +1,42 @@ +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: MIT +# + +import subprocess +from oeqa.runtime.case import OERuntimeTestCase +import tempfile +from oeqa.runtime.decorator.package import OEHasPackage +import shutil + +class LoginTest(OERuntimeTestCase): + + def test_screenshot(self): + # Leaving this limited to qemux86-64, until I can get the same resolution + # on all the qemu machines. + if self.td.get('MACHINE') != "qemux86-64": + self.skipTest("") + + if bb.utils.which(os.getenv('PATH'), "convert.im7") is not None and bb.utils.which(os.getenv('PATH'), "compare.im7") is not None: + with tempfile.NamedTemporaryFile(prefix="oeqa-screenshot-login", suffix=".png") as t: + ret = self.target.runner.run_monitor("screendump", args={"filename": t.name, "format":"png"}) + # Use the meta-oe version of convert, along with it's suffix + cmd = "convert.im7 {0} -fill white -draw 'rectangle 600,10 640,22' {1}".format(t.name, t.name) + subprocess.check_output(cmd.split(), shell=True, stderr=subprocess.PIPE) + + # Uncomment for debugging/capturing other machine images. + # shutil.copy(t.name, os.getenv('HOME')+"/"+self.td.get('MACHINE')+".png") + + # Use the meta-oe version of compare, along with it's suffix + cmd = "compare.im7 -metric MSE {0} {1}/meta/files/image-tests/core-image-sato-{2}.png /dev/null".format(t.name, self.td.get('COREBASE'), self.td.get('MACHINE')) + # We use Popen here because compare outputs the values we care about on stderr. + proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + output, error = proc.communicate() + + # In theory this should be 0, however I've seen some odd things with the Home icon/screen switcher. + # We should as some point, log the diff image for debugging, but for now, just report that it's different. + diff=float(error.decode('utf-8').replace("(", "").replace(")","").split()[1]) + self.assertEqual(0, diff, "Screenshot diff is %s." % (str(diff))) + else: + self.skipTest("Could not find imagemagick tools.")Signature · 11 lines
2.34.1 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#192394): https://lists.openembedded.org/g/openembedded-core/message/192394 Mute This Topic: https://lists.openembedded.org/mt/103173636/3617179 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Quoted 6 lines · Eilís 'pidge' Ní Fhlannagáin
This takes the work rburton did on image screenshot testing and expands it.
Right now this only works for qemux86-64. Some standardization of screensize/resolution needs to happen with runqemu params for other machines.
I suspect that we don't want to make everything the same so we may need to allow the code to adapt to the sizes we have, I think there are only a small number of different ones. The annoying part may be just working out the clock mask sizes for the different ones since we can easily have different machine specific references.
Quoted 4 lines
There is an issue in qemux86-64 (and possibly others) where
the screenswitch/home icon is only half present. This causes the test to fail. I'm unsure if this is a QMP/qemu issue but I've certainly seen the half-rendered icon, so it's not Imagemagick.
These are the kinds of issues we want to start catching and why this is important!
Quoted 10 lines
This test takes a screendump of a qemu image (for now, just core-image-sato for qemux86-64), and compares it to an image we have on record. Some normalisation of the different qemu configs need to happen to be able to support all machines. Example, the qemuarm64 screen size is much larger than the qemux86-64.
The image we have on record contains a blanked out clock. We do the same blanking out process for the screenshot, so the images should have zero differences. If they do, we fail.
Quoted 6 lines
In order to enable this test, you will need meta-openembedded/meta-oe in your bblayers.conf and the following in local.conf:
IMAGE_CLASSES+= "testimage"TEST_SUITES= "login"
TESTIMAGEDEPENDS:append:qemuall = " imagemagick-native:do_populate_sysroot "
or HOSTTOOLS += "convert compare"
?
meta/lib/oeqa/runtime/cases/login.py
Quoted 1 line
meta/lib/oeqa/runtime/cases/login.py +0
-0
I just noticed the name. Does "login" make sense for a desktop screenshot? I can kind of see the logic but I'd not be looking in that file for this :/.
Quoted 14 lines
new file mode 100644
index 00000000000..30074f71b66meta/lib/oeqa/runtime/cases/login.py new file +9
-0@@ -0,0 +1,42 @@ +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: MIT +# + +import subprocess +from oeqa.runtime.case import OERuntimeTestCase +import tempfile
meta/lib/oeqa/runtime/cases/login.py line 19
Quoted 10 lines
meta/lib/oeqa/runtime/cases/login.py new file +10
-0+from oeqa.runtime.decorator.package import OEHasPackage +import shutil + +class LoginTest(OERuntimeTestCase): + + def test_screenshot(self): + # Leaving this limited to qemux86-64, until I can get the same resolution + # on all the qemu machines. + if self.td.get('MACHINE') != "qemux86-64": + self.skipTest("")
You mean
self.skipTest("Test only compatible with qemux86-64")
right? ;-)
Quoted 13 lines
meta/lib/oeqa/runtime/cases/login.py new file +13
-0+ + if bb.utils.which(os.getenv('PATH'), "convert.im7") is not None and bb.utils.which(os.getenv('PATH'), "compare.im7") is not None: + with tempfile.NamedTemporaryFile(prefix="oeqa-screenshot-login", suffix=".png") as t: + ret = self.target.runner.run_monitor("screendump", args={"filename": t.name, "format":"png"}) + # Use the meta-oe version of convert, along with it's suffix + cmd = "convert.im7 {0} -fill white -draw 'rectangle 600,10 640,22' {1}".format(t.name, t.name) + subprocess.check_output(cmd.split(), shell=True, stderr=subprocess.PIPE) + + # Uncomment for debugging/capturing other machine images. + # shutil.copy(t.name, os.getenv('HOME')+"/"+self.td.get('MACHINE')+".png") + + # Use the meta-oe version of compare, along with it's suffix + cmd = "compare.im7 -metric MSE {0} {1}/meta/files/image-tests/core-image-sato-{2}.png /dev/null".format(t.name, self.td.get('COREBASE'), self.td.get('MACHINE'))
meta/lib/oeqa/runtime/cases/login.py line 42
Quoted 10 lines
meta/lib/oeqa/runtime/cases/login.py new file +10
-0+ # We use Popen here because compare outputs the values we care about on stderr. + proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + output, error = proc.communicate() + + # In theory this should be 0, however I've seen some odd things with the Home icon/screen switcher. + # We should as some point, log the diff image for debugging, but for now, just report that it's different. + diff=float(error.decode('utf-8').replace("(", "").replace(")","").split()[1]) + self.assertEqual(0, diff, "Screenshot diff is %s." % (str(diff))) + else: + self.skipTest("Could not find imagemagick tools.")
Upon failure do we want to copy to some randomly named file in /tmp/ so we can at least have a chance to see what went wrong (printing the name in the test failure message)? I think currently the image would be lost making debugging hard?
The selftests use OEQA_DEBUGGING_SAVED_OUTPUT but I'm not sure that is available in the runtime test environment at present unfortunately.
Cheers,
Richard