Skip to contentSkip to conversation

login.py: Proof of concept for screenshot testcases

PATCH In review Eilís 'pidge' Ní Fhlannagáin 3 participants 3 messages Dec 14, 2023 – Dec 15, 2023

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 login.py: Proof of concept for screenshot testcases

This takes the work rburton did on image screenshot testing and expands it.

Right now this only works for qemux86-64. Some standardization of screensize/resolution needs to happen with runqemu params for other machines. There is an issue in qemux86-64 (and possibly others) where the screenswitch/home icon is only half present. This causes the test to fail. I'm unsure if this is a QMP/qemu issue but I've certainly seen the half-rendered icon, so it's not Imagemagick.

This test takes a screendump of a qemu image (for now, just core-image-sato for qemux86-64), and compares it to an image we have on record. Some normalisation of the different qemu configs need to happen to be able to support all machines. Example, the qemuarm64 screen size is much larger than the qemux86-64.

The image we have on record contains a blanked out clock. We do the same blanking out process for the screenshot, so the images should have zero differences. If they do, we fail.

In order to enable this test, you will need meta-openembedded/meta-oe in your bblayers.conf and the following in local.conf:

IMAGE_CLASSES += "testimage"
TEST_SUITES = "login"
TESTIMAGEDEPENDS:append:qemuall = " imagemagick-native:do_populate_sysroot "

---
 .../core-image-sato-qemux86-64.png            | Bin 0 -> 46986 bytes
 meta/lib/oeqa/runtime/cases/login.py          |  42 ++++++++++++++++++
 2 files changed, 42 insertions(+)
 create mode 100644 meta/files/image-tests/core-image-sato-qemux86-64.png
 create mode 100644 meta/lib/oeqa/runtime/cases/login.py
meta/files/image-tests/core-image-sato-qemux86-64.png new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..5a3e46169dca6d118c003d29a59c59588cee7463
GIT binary patch
meta/lib/oeqa/runtime/cases/login.py new file +42 -0

42 lines not on this page · raw message

Signature · 1 line
2.34.1
#1 lore raw

Hello,

Please also fix 2/4 and always resend the whole series, else the burden to cherry-pick from the multiple version of the series and my memory is not what it used to be.

Quoted 971 lines · Eil�s 'pidge' N� Fhlannag�in

This takes the work rburton did on image screenshot testing and expands it.

Right now this only works for qemux86-64. Some standardization of screensize/resolution needs to happen with runqemu params for other machines. There is an issue in qemux86-64 (and possibly others) where the screenswitch/home icon is only half present. This causes the test to fail. I'm unsure if this is a QMP/qemu issue but I've certainly seen the half-rendered icon, so it's not Imagemagick.

This test takes a screendump of a qemu image (for now, just core-image-sato for qemux86-64), and compares it to an image we have on record. Some normalisation of the different qemu configs need to happen to be able to support all machines. Example, the qemuarm64 screen size is much larger than the qemux86-64.

The image we have on record contains a blanked out clock. We do the same blanking out process for the screenshot, so the images should have zero differences. If they do, we fail.

In order to enable this test, you will need meta-openembedded/meta-oe in your bblayers.conf and the following in local.conf:

IMAGE_CLASSES += "testimage"
TEST_SUITES = "login"
TESTIMAGEDEPENDS:append:qemuall = " imagemagick-native:do_populate_sysroot "

---
 .../core-image-sato-qemux86-64.png            | Bin 0 -> 46986 bytes
 meta/lib/oeqa/runtime/cases/login.py          |  42 ++++++++++++++++++
 2 files changed, 42 insertions(+)
 create mode 100644 meta/files/image-tests/core-image-sato-qemux86-64.png
 create mode 100644 meta/lib/oeqa/runtime/cases/login.py
meta/files/image-tests/core-image-sato-qemux86-64.png new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..5a3e46169dca6d118c003d29a59c59588cee7463
GIT binary patch
meta/lib/oeqa/runtime/cases/login.py new file +42 -0
new file mode 100644
index 00000000000..30074f71b66
@@ -0,0 +1,42 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+import subprocess
+from oeqa.runtime.case import OERuntimeTestCase
+import tempfile
+from oeqa.runtime.decorator.package import OEHasPackage
+import shutil
+
+class LoginTest(OERuntimeTestCase):
+
+    def test_screenshot(self):
+        # Leaving this limited to qemux86-64, until I can get the same resolution
+        # on all the qemu machines.
+        if self.td.get('MACHINE') != "qemux86-64":
+            self.skipTest("")
+
+        if bb.utils.which(os.getenv('PATH'), "convert.im7") is not None and bb.utils.which(os.getenv('PATH'), "compare.im7") is not None:
+            with tempfile.NamedTemporaryFile(prefix="oeqa-screenshot-login", suffix=".png") as t:
+                ret = self.target.runner.run_monitor("screendump", args={"filename": t.name, "format":"png"})
+                # Use the meta-oe version of convert, along with it's suffix
+                cmd = "convert.im7 {0} -fill white -draw 'rectangle 600,10 640,22' {1}".format(t.name, t.name)
+                subprocess.check_output(cmd.split(), shell=True, stderr=subprocess.PIPE)
+                
+                # Uncomment for debugging/capturing other machine images.
+                # shutil.copy(t.name, os.getenv('HOME')+"/"+self.td.get('MACHINE')+".png")
+
+                # Use the meta-oe version of compare, along with it's suffix               
+                cmd = "compare.im7 -metric MSE {0} {1}/meta/files/image-tests/core-image-sato-{2}.png /dev/null".format(t.name, self.td.get('COREBASE'), self.td.get('MACHINE'))
+                # We use Popen here because compare outputs the values we care about on stderr. 
+                proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+                output, error = proc.communicate()
+                
+                # In theory this should be 0, however I've seen some odd things with the Home icon/screen switcher.
+                # We should as some point, log the diff image for debugging, but for now, just report that it's different.
+                diff=float(error.decode('utf-8').replace("(", "").replace(")","").split()[1])
+                self.assertEqual(0, diff, "Screenshot diff is %s." % (str(diff)))
+        else:
+            self.skipTest("Could not find imagemagick tools.")
Signature · 11 lines
2.34.1



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#192394): https://lists.openembedded.org/g/openembedded-core/message/192394
Mute This Topic: https://lists.openembedded.org/mt/103173636/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#2 lore raw
In reply to Eilís 'pidge' Ní Fhlannagáin
Quoted 6 lines · Eilís 'pidge' Ní Fhlannagáin

This takes the work rburton did on image screenshot testing and expands it.

Right now this only works for qemux86-64. Some standardization of screensize/resolution needs to happen with runqemu params for other machines.

I suspect that we don't want to make everything the same so we may need to allow the code to adapt to the sizes we have, I think there are only a small number of different ones. The annoying part may be just working out the clock mask sizes for the different ones since we can easily have different machine specific references.

Quoted 4 lines

There is an issue in qemux86-64 (and possibly others) where
the screenswitch/home icon is only half present. This causes the test to fail. I'm unsure if this is a QMP/qemu issue but I've certainly seen the half-rendered icon, so it's not Imagemagick.

These are the kinds of issues we want to start catching and why this is important!

Quoted 10 lines

This test takes a screendump of a qemu image (for now, just core-image-sato for qemux86-64), and compares it to an image we have on record. Some normalisation of the different qemu configs need to happen to be able to support all machines. Example, the qemuarm64 screen size is much larger than the qemux86-64.

The image we have on record contains a blanked out clock. We do the same blanking out process for the screenshot, so the images should have zero differences. If they do, we fail.

Quoted 6 lines

In order to enable this test, you will need meta-openembedded/meta-oe in your bblayers.conf and the following in local.conf:

IMAGE_CLASSES += "testimage"
TEST_SUITES = "login"
TESTIMAGEDEPENDS:append:qemuall = " imagemagick-native:do_populate_sysroot "

or HOSTTOOLS += "convert compare"

?

meta/lib/oeqa/runtime/cases/login.py

Quoted 1 line
meta/lib/oeqa/runtime/cases/login.py +0 -0

I just noticed the name. Does "login" make sense for a desktop screenshot? I can kind of see the logic but I'd not be looking in that file for this :/.

Quoted 14 lines

new file mode 100644
index 00000000000..30074f71b66

meta/lib/oeqa/runtime/cases/login.py new file +9 -0
@@ -0,0 +1,42 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+import subprocess
+from oeqa.runtime.case import OERuntimeTestCase
+import tempfile

meta/lib/oeqa/runtime/cases/login.py line 19

Quoted 10 lines
meta/lib/oeqa/runtime/cases/login.py new file +10 -0
+from oeqa.runtime.decorator.package import OEHasPackage
+import shutil
+
+class LoginTest(OERuntimeTestCase):
+
+    def test_screenshot(self):
+        # Leaving this limited to qemux86-64, until I can get the same resolution
+        # on all the qemu machines.
+        if self.td.get('MACHINE') != "qemux86-64":
+            self.skipTest("")

You mean

self.skipTest("Test only compatible with qemux86-64")

right? ;-)

Quoted 13 lines
meta/lib/oeqa/runtime/cases/login.py new file +13 -0
+
+        if bb.utils.which(os.getenv('PATH'), "convert.im7") is not None and bb.utils.which(os.getenv('PATH'), "compare.im7") is not None:
+            with tempfile.NamedTemporaryFile(prefix="oeqa-screenshot-login", suffix=".png") as t:
+                ret = self.target.runner.run_monitor("screendump", args={"filename": t.name, "format":"png"})
+                # Use the meta-oe version of convert, along with it's suffix
+                cmd = "convert.im7 {0} -fill white -draw 'rectangle 600,10 640,22' {1}".format(t.name, t.name)
+                subprocess.check_output(cmd.split(), shell=True, stderr=subprocess.PIPE)
+                
+                # Uncomment for debugging/capturing other machine images.
+                # shutil.copy(t.name, os.getenv('HOME')+"/"+self.td.get('MACHINE')+".png")
+
+                # Use the meta-oe version of compare, along with it's suffix               
+                cmd = "compare.im7 -metric MSE {0} {1}/meta/files/image-tests/core-image-sato-{2}.png /dev/null".format(t.name, self.td.get('COREBASE'), self.td.get('MACHINE'))

meta/lib/oeqa/runtime/cases/login.py line 42

Quoted 10 lines
meta/lib/oeqa/runtime/cases/login.py new file +10 -0
+                # We use Popen here because compare outputs the values we care about on stderr. 
+                proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+                output, error = proc.communicate()
+                
+                # In theory this should be 0, however I've seen some odd things with the Home icon/screen switcher.
+                # We should as some point, log the diff image for debugging, but for now, just report that it's different.
+                diff=float(error.decode('utf-8').replace("(", "").replace(")","").split()[1])
+                self.assertEqual(0, diff, "Screenshot diff is %s." % (str(diff)))
+        else:
+            self.skipTest("Could not find imagemagick tools.")

Upon failure do we want to copy to some randomly named file in /tmp/ so we can at least have a chance to see what went wrong (printing the name in the test failure message)? I think currently the image would be lost making debugging hard?

The selftests use OEQA_DEBUGGING_SAVED_OUTPUT but I'm not sure that is available in the runtime test environment at present unfortunately.

Cheers,

Richard