Skip to contentSkip to conversation

image_types_wic: make do_rootfs_wicenv hash-safe

PATCH In review Sam Kent 3 participants 5 messages Apr 30 – May 26

lore
Share & tools
lore

Sam Kent

author
#0 lore raw

PATCH 1/2 image_types_wic: make do_rootfs_wicenv hash-safe

BBLAYERS and BBPATH contain host-specific paths which contaminate the hash and result in unnecessary cache misses.

Exclude BBLAYERS and BBPATH from vardeps and instead enumerate the wic plugin .py files tracking them via file-checksums.

The hash now only changes when a plugin file is actually added, removed, or modified.

Fixes [YOCTO #15662]

---
 meta/classes-recipe/image_types_wic.bbclass | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
meta/classes-recipe/image_types_wic.bbclass +21 -0
index ea8c1c9..e6f77e9 100644
@@ -68,6 +68,22 @@ def wks_checksums(files, search_path):
             ret = ret + " " + found + ":True"
     return ret
 
+def wic_plugin_checksums(bblayers, bbpath):
+    entries = []
+    seen = set()
+    paths = (bblayers or "").split() + [p for p in (bbpath or "").split(":") if p]
+    for path in paths:
+        if path in seen:
+            continue
+        seen.add(path)
+        plugin_dir = os.path.join(path, "lib", "wic", "plugins")
+        if os.path.isdir(plugin_dir):
+            for root, dirs, files in os.walk(plugin_dir):
+                dirs.sort()
+                for f in sorted(files):
+                    if f.endswith(".py"):
+                        entries.append(os.path.join(root, f) + ":True")
+    return " ".join(entries)
 
 WIC_CREATE_EXTRA_ARGS ?= ""
 
@@ -228,8 +244,13 @@ python do_rootfs_wicenv () {
 addtask do_flush_pseudodb after do_rootfs before do_image do_image_qa
 addtask do_rootfs_wicenv after do_image before do_image_wic do_image_wicenv
 do_rootfs_wicenv[vardeps] += "${WICVARS}"
+do_rootfs_wicenv[vardepsexclude] += "BBLAYERS BBPATH"
 do_rootfs_wicenv[prefuncs] = 'set_image_size'
 
+# Track actual plugin files instead of host paths (BBLAYERS/BBPATH excluded above).
+WIC_PLUGIN_CHECKSUMS = "${@wic_plugin_checksums(d.getVar('BBLAYERS'), d.getVar('BBPATH')) if d.getVar('USING_WIC') else ''}"
+do_rootfs_wicenv[file-checksums] += "${WIC_PLUGIN_CHECKSUMS}"
+
 IMAGE_CMD:wicenv () {
 	cp "${STAGING_DIR}/${MACHINE}/imgdata/${IMAGE_BASENAME}.env" \
 	   "${IMGDEPLOYDIR}/${IMAGE_BASENAME}.env"
Signature · 1 line
2.34.1
author
#1 lore raw

PATCH 2/2 image_types_wic: add selftests for do_rootfs_wicenv hash safety

Add WicEnvHashTests to the wic selftest suite to cover the fix for bug 15662:

- test_bblayers_excluded_from_task_hash: static check via tinfoil that BBLAYERS appears in do_rootfs_wicenv[vardepsexclude].

- test_hash_does_not_change_when_empty_layer_added_to_bblayers: runs with -S none, adds a layer containing no wic plugins, and asserts the sigdata hash is unchanged.

- test_hash_changes_when_wic_plugin_added_to_layer: the inverse — adds a layer that contains a wic plugin and asserts the hash does change.

---
 meta/lib/oeqa/selftest/cases/wic.py | 111 +++++++++++++++++++++++++++-
 1 file changed, 108 insertions(+), 3 deletions(-)
meta/lib/oeqa/selftest/cases/wic.py +108 -3
index 4e94f4d..174e57d 100644
@@ -17,14 +17,13 @@ import filecmp
 
 from glob import glob
 from shutil import rmtree, copy
-from tempfile import NamedTemporaryFile
-from tempfile import TemporaryDirectory
+from tempfile import NamedTemporaryFile, TemporaryDirectory, mkdtemp
 from textwrap import dedent
 
 from oeqa.selftest.case import OESelftestTestCase
 from oeqa.core.decorator import OETestTag
 from oeqa.core.decorator.data import skipIfNotArch
-from oeqa.utils.commands import runCmd, bitbake, get_bb_var, get_bb_vars, runqemu
+from oeqa.utils.commands import runCmd, bitbake, get_bb_var, get_bb_vars, runqemu, create_temp_layer
 
 
 def extract_files(debugfs_output):
@@ -2300,3 +2299,109 @@ class ModifyTests(WicTestCase):
         # check if it's removed
         result = runCmd("wic ls %s:2/ -n %s" % (images[0], sysroot))
         self.assertNotIn('etc', [line.split()[-1] for line in result.output.split('\n') if line])
+
+
+class WicEnvHashTests(OESelftestTestCase):
+    """Tests for do_rootfs_wicenv task hash safety (bug 15662)."""
+
+    def setUpLocal(self):
+        super(WicEnvHashTests, self).setUpLocal()
+        self.topdir = get_bb_var('TOPDIR')
+
+    def _fresh_tmpdir(self, name):
+        """Return a clean TMPDIR path under TOPDIR, removing any prior contents."""
+        import shutil
+        path = os.path.join(self.topdir, name)
+        if os.path.exists(path):
+            shutil.rmtree(path)
+        self.track_for_cleanup(path)
+        return path
+
+    def _get_task_sigdata_hash(self, tmpdir, taskname):
+        """Return the sigdata hash for taskname; fail if no match found."""
+        matches = []
+        for root, _, files in os.walk(os.path.join(tmpdir, 'stamps')):
+            for f in files:
+                if taskname in f and '.sigdata.' in f:
+                    matches.append(f.rsplit('.sigdata.', 1)[-1])
+        self.assertGreater(len(matches), 0,
+                           "No %s sigdata file found in %s" % (taskname, tmpdir))
+        return matches[0]
+
+    def test_bblayers_excluded_from_task_hash(self):
+        """BBLAYERS must appear in do_rootfs_wicenv[vardepsexclude]."""
+        import bb.tinfoil
+        with bb.tinfoil.Tinfoil() as tinfoil:
+            tinfoil.prepare(config_only=False, quiet=2)
+            d = tinfoil.parse_recipe('core-image-minimal')
+            vardepsexclude = (d.getVarFlag('do_rootfs_wicenv', 'vardepsexclude') or '').split()
+
+        self.assertIn('BBLAYERS', vardepsexclude,
+                      "BBLAYERS is not in do_rootfs_wicenv[vardepsexclude]; "
+                      "host paths will change the task hash across workspaces")
+
+    def test_hash_does_not_change_when_empty_layer_added_to_bblayers(self):
+        """Adding an empty layer to BBLAYERS must not change the do_rootfs_wicenv hash."""
+        tmpdir1 = self._fresh_tmpdir('tmp-wicenv-hash1')
+        self.write_config(
+            'TMPDIR = "%s"\n'
+            'BB_SIGNATURE_HANDLER = "OEBasicHash"\n'
+            'IMAGE_FSTYPES += "wic"\n' % tmpdir1
+        )
+        bitbake('core-image-minimal -c do_rootfs_wicenv -S none')
+        hash_before = self._get_task_sigdata_hash(tmpdir1, 'do_rootfs_wicenv')
+
+        # Add a layer with no wic plugins — changes BBLAYERS but must not change the hash.
+        templayerdir = mkdtemp(prefix='selftest-wicenv-')
+        self.track_for_cleanup(templayerdir)
+        create_temp_layer(templayerdir, 'selftestwicenvhash')
+        runCmd('bitbake-layers add-layer %s' % templayerdir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer %s' % templayerdir)
+
+        tmpdir2 = self._fresh_tmpdir('tmp-wicenv-hash2')
+        self.write_config(
+            'TMPDIR = "%s"\n'
+            'BB_SIGNATURE_HANDLER = "OEBasicHash"\n'
+            'IMAGE_FSTYPES += "wic"\n' % tmpdir2
+        )
+        bitbake('core-image-minimal -c do_rootfs_wicenv -S none')
+        hash_after = self._get_task_sigdata_hash(tmpdir2, 'do_rootfs_wicenv')
+
+        self.assertEqual(hash_before, hash_after,
+                         "do_rootfs_wicenv hash changed after adding an empty layer to "
+                         "BBLAYERS even though no wic plugins changed (bug 15662)")
+
+    def test_hash_changes_when_wic_plugin_added_to_layer(self):
+        """Adding a layer that contains a wic plugin must change the do_rootfs_wicenv hash."""
+        tmpdir3 = self._fresh_tmpdir('tmp-wicenv-hash3')
+        self.write_config(
+            'TMPDIR = "%s"\n'
+            'BB_SIGNATURE_HANDLER = "OEBasicHash"\n'
+            'IMAGE_FSTYPES += "wic"\n' % tmpdir3
+        )
+        bitbake('core-image-minimal -c do_rootfs_wicenv -S none')
+        hash_before = self._get_task_sigdata_hash(tmpdir3, 'do_rootfs_wicenv')
+
+        # Add a layer that contains a wic plugin — must change the hash.
+        templayerdir = mkdtemp(prefix='selftest-wicenv-plugin-')
+        self.track_for_cleanup(templayerdir)
+        create_temp_layer(templayerdir, 'selftestwicenvplugin')
+        plugin_dir = os.path.join(templayerdir, 'lib', 'wic', 'plugins', 'source')
+        os.makedirs(plugin_dir)
+        with open(os.path.join(plugin_dir, 'selftest_dummy.py'), 'w') as f:
+            f.write('# selftest dummy wic source plugin\n')
+        runCmd('bitbake-layers add-layer %s' % templayerdir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer %s' % templayerdir)
+
+        tmpdir4 = self._fresh_tmpdir('tmp-wicenv-hash4')
+        self.write_config(
+            'TMPDIR = "%s"\n'
+            'BB_SIGNATURE_HANDLER = "OEBasicHash"\n'
+            'IMAGE_FSTYPES += "wic"\n' % tmpdir4
+        )
+        bitbake('core-image-minimal -c do_rootfs_wicenv -S none')
+        hash_after = self._get_task_sigdata_hash(tmpdir4, 'do_rootfs_wicenv')
+
+        self.assertNotEqual(hash_before, hash_after,
+                            "do_rootfs_wicenv hash did not change after adding a layer "
+                            "with a wic plugin — file-checksums tracking is not working")
Signature · 1 line
2.34.1
#2 lore raw

Re: [PATCH 2/2] image_types_wic: add selftests for do_rootfs_wicenv hash safety

Quoted 73 lines · Sam Kent

Add WicEnvHashTests to the wic selftest suite to cover the fix for bug 15662:

- test_bblayers_excluded_from_task_hash: static check via tinfoil that BBLAYERS appears in do_rootfs_wicenv[vardepsexclude].

- test_hash_does_not_change_when_empty_layer_added_to_bblayers: runs with -S none, adds a layer containing no wic plugins, and asserts the sigdata hash is unchanged.

- test_hash_changes_when_wic_plugin_added_to_layer: the inverse — adds a layer that contains a wic plugin and asserts the hash does change.

---
 meta/lib/oeqa/selftest/cases/wic.py | 111 +++++++++++++++++++++++++++-
 1 file changed, 108 insertions(+), 3 deletions(-)
meta/lib/oeqa/selftest/cases/wic.py +32 -3
index 4e94f4d..174e57d 100644
@@ -17,14 +17,13 @@ import filecmp
 
 from glob import glob
 from shutil import rmtree, copy
-from tempfile import NamedTemporaryFile
-from tempfile import TemporaryDirectory
+from tempfile import NamedTemporaryFile, TemporaryDirectory, mkdtemp
 from textwrap import dedent
 
 from oeqa.selftest.case import OESelftestTestCase
 from oeqa.core.decorator import OETestTag
 from oeqa.core.decorator.data import skipIfNotArch
-from oeqa.utils.commands import runCmd, bitbake, get_bb_var, get_bb_vars, runqemu
+from oeqa.utils.commands import runCmd, bitbake, get_bb_var, get_bb_vars, runqemu, create_temp_layer
 
 
 def extract_files(debugfs_output):
@@ -2300,3 +2299,109 @@ class ModifyTests(WicTestCase):
         # check if it's removed
         result = runCmd("wic ls %s:2/ -n %s" % (images[0], sysroot))
         self.assertNotIn('etc', [line.split()[-1] for line in result.output.split('\n') if line])
+
+
+class WicEnvHashTests(OESelftestTestCase):
+    """Tests for do_rootfs_wicenv task hash safety (bug 15662)."""
+
+    def setUpLocal(self):
+        super(WicEnvHashTests, self).setUpLocal()
+        self.topdir = get_bb_var('TOPDIR')
+
+    def _fresh_tmpdir(self, name):
+        """Return a clean TMPDIR path under TOPDIR, removing any prior contents."""
+        import shutil
+        path = os.path.join(self.topdir, name)
+        if os.path.exists(path):
+            shutil.rmtree(path)
+        self.track_for_cleanup(path)
+        return path
+
+    def _get_task_sigdata_hash(self, tmpdir, taskname):
+        """Return the sigdata hash for taskname; fail if no match found."""
+        matches = []
+        for root, _, files in os.walk(os.path.join(tmpdir, 'stamps')):
+            for f in files:
+                if taskname in f and '.sigdata.' in f:
+                    matches.append(f.rsplit('.sigdata.', 1)[-1])
+        self.assertGreater(len(matches), 0,
+                           "No %s sigdata file found in %s" % (taskname, tmpdir))
+        return matches[0]
+
+    def test_bblayers_excluded_from_task_hash(self):

meta/lib/oeqa/selftest/cases/wic.py class ModifyTests(WicTestCase): · line 2341

Quoted 10 lines
meta/lib/oeqa/selftest/cases/wic.py +10 -0
+        """BBLAYERS must appear in do_rootfs_wicenv[vardepsexclude]."""
+        import bb.tinfoil
+        with bb.tinfoil.Tinfoil() as tinfoil:
+            tinfoil.prepare(config_only=False, quiet=2)
+            d = tinfoil.parse_recipe('core-image-minimal')
+            vardepsexclude = (d.getVarFlag('do_rootfs_wicenv', 'vardepsexclude') or '').split()
+
+        self.assertIn('BBLAYERS', vardepsexclude,
+                      "BBLAYERS is not in do_rootfs_wicenv[vardepsexclude]; "
+                      "host paths will change the task hash across workspaces")

I don't think we need this first test case, the other two confirm that BBLAYERS is excluded based on the observed behaviour.

Quoted 56 lines
meta/lib/oeqa/selftest/cases/wic.py +56 -0
+
+    def test_hash_does_not_change_when_empty_layer_added_to_bblayers(self):
+        """Adding an empty layer to BBLAYERS must not change the do_rootfs_wicenv hash."""
+        tmpdir1 = self._fresh_tmpdir('tmp-wicenv-hash1')
+        self.write_config(
+            'TMPDIR = "%s"\n'
+            'BB_SIGNATURE_HANDLER = "OEBasicHash"\n'
+            'IMAGE_FSTYPES += "wic"\n' % tmpdir1
+        )
+        bitbake('core-image-minimal -c do_rootfs_wicenv -S none')
+        hash_before = self._get_task_sigdata_hash(tmpdir1, 'do_rootfs_wicenv')
+
+        # Add a layer with no wic plugins — changes BBLAYERS but must not change the hash.
+        templayerdir = mkdtemp(prefix='selftest-wicenv-')
+        self.track_for_cleanup(templayerdir)
+        create_temp_layer(templayerdir, 'selftestwicenvhash')
+        runCmd('bitbake-layers add-layer %s' % templayerdir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer %s' % templayerdir)
+
+        tmpdir2 = self._fresh_tmpdir('tmp-wicenv-hash2')
+        self.write_config(
+            'TMPDIR = "%s"\n'
+            'BB_SIGNATURE_HANDLER = "OEBasicHash"\n'
+            'IMAGE_FSTYPES += "wic"\n' % tmpdir2
+        )
+        bitbake('core-image-minimal -c do_rootfs_wicenv -S none')
+        hash_after = self._get_task_sigdata_hash(tmpdir2, 'do_rootfs_wicenv')
+
+        self.assertEqual(hash_before, hash_after,
+                         "do_rootfs_wicenv hash changed after adding an empty layer to "
+                         "BBLAYERS even though no wic plugins changed (bug 15662)")
+
+    def test_hash_changes_when_wic_plugin_added_to_layer(self):
+        """Adding a layer that contains a wic plugin must change the do_rootfs_wicenv hash."""
+        tmpdir3 = self._fresh_tmpdir('tmp-wicenv-hash3')
+        self.write_config(
+            'TMPDIR = "%s"\n'
+            'BB_SIGNATURE_HANDLER = "OEBasicHash"\n'
+            'IMAGE_FSTYPES += "wic"\n' % tmpdir3
+        )
+        bitbake('core-image-minimal -c do_rootfs_wicenv -S none')
+        hash_before = self._get_task_sigdata_hash(tmpdir3, 'do_rootfs_wicenv')
+
+        # Add a layer that contains a wic plugin — must change the hash.
+        templayerdir = mkdtemp(prefix='selftest-wicenv-plugin-')
+        self.track_for_cleanup(templayerdir)
+        create_temp_layer(templayerdir, 'selftestwicenvplugin')
+        plugin_dir = os.path.join(templayerdir, 'lib', 'wic', 'plugins', 'source')
+        os.makedirs(plugin_dir)
+        with open(os.path.join(plugin_dir, 'selftest_dummy.py'), 'w') as f:
+            f.write('# selftest dummy wic source plugin\n')
+        runCmd('bitbake-layers add-layer %s' % templayerdir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer %s' % templayerdir)
+
+        tmpdir4 = self._fresh_tmpdir('tmp-wicenv-hash4')
+        self.write_config(

meta/lib/oeqa/selftest/cases/wic.py class ModifyTests(WicTestCase): · line 2407

Quoted 10 lines
meta/lib/oeqa/selftest/cases/wic.py +10 -0
+            'TMPDIR = "%s"\n'
+            'BB_SIGNATURE_HANDLER = "OEBasicHash"\n'
+            'IMAGE_FSTYPES += "wic"\n' % tmpdir4
+        )
+        bitbake('core-image-minimal -c do_rootfs_wicenv -S none')
+        hash_after = self._get_task_sigdata_hash(tmpdir4, 'do_rootfs_wicenv')
+
+        self.assertNotEqual(hash_before, hash_after,
+                            "do_rootfs_wicenv hash did not change after adding a layer "
+                            "with a wic plugin — file-checksums tracking is not working")

Two questions:

- Why do we need a fresh TMPDIR for each build?

- Can we combine these two test cases into one so that we're not unnecessarily re-calculating hash_before?

Best regards,

Signature · 1 line
Paul Barker
#3 lore raw
In reply to Sam Kent
Quoted 30 lines · Sam Kent

BBLAYERS and BBPATH contain host-specific paths which contaminate the hash and result in unnecessary cache misses.

Exclude BBLAYERS and BBPATH from vardeps and instead enumerate the wic plugin .py files tracking them via file-checksums.

The hash now only changes when a plugin file is actually added, removed, or modified.

Fixes [YOCTO #15662]

---
 meta/classes-recipe/image_types_wic.bbclass | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
meta/classes-recipe/image_types_wic.bbclass +6 -0
index ea8c1c9..e6f77e9 100644
@@ -68,6 +68,22 @@ def wks_checksums(files, search_path):
             ret = ret + " " + found + ":True"
     return ret
 
+def wic_plugin_checksums(bblayers, bbpath):
+    entries = []
+    seen = set()
+    paths = (bblayers or "").split() + [p for p in (bbpath or "").split(":") if p]
+    for path in paths:
+        if path in seen:

meta/classes-recipe/image_types_wic.bbclass def wks_checksums(files, search_path): · line 86

Quoted 10 lines
meta/classes-recipe/image_types_wic.bbclass +10 -0
+            continue
+        seen.add(path)
+        plugin_dir = os.path.join(path, "lib", "wic", "plugins")
+        if os.path.isdir(plugin_dir):
+            for root, dirs, files in os.walk(plugin_dir):
+                dirs.sort()
+                for f in sorted(files):
+                    if f.endswith(".py"):
+                        entries.append(os.path.join(root, f) + ":True")
+    return " ".join(entries)

Re-implementing the wic plugin search logic here is likely to lead to inconsistencies. For example, wic currently searches for plugins based on BBLAYERS but not BBPATH, and it looks in both scripts/lib/wic/plugins and lib/wic/plugins. Even if we align the two implementations now, it will be very easy for them to drift apart again in the future.

Could we instead rely on wic to perform the plugin search? There is currently a wic list source-plugins command, we could add a wic list imager-plugins command or perhaps a wic list plugins command to list all plugins. We could then call that here to get the list.

We should also move this to an event handler so that we don't unnecessarily re-run this search whenever the bitbake data store is updated. See image-postinst-intercepts.bbclass for an example of how to do this.

Best regards,

Signature · 1 line
Paul Barker
#4 lore raw
Quoted 53 lines · Paul Barker via lists.openembedded.org
Quoted 32 lines · Sam Kent

BBLAYERS and BBPATH contain host-specific paths which contaminate the hash and result in unnecessary cache misses.

Exclude BBLAYERS and BBPATH from vardeps and instead enumerate the wic plugin .py files tracking them via file-checksums.

The hash now only changes when a plugin file is actually added, removed, or modified.

Fixes [YOCTO #15662]

---
 meta/classes-recipe/image_types_wic.bbclass | 21 +++++++++++++++++++++

 1 file changed, 21 insertions(+)
meta/classes-recipe/image_types_wic.bbclass +0 -0
index ea8c1c9..e6f77e9 100644
@@ -68,6 +68,22 @@ def wks_checksums(files, search_path):

             ret = ret + " " + found + ":True"
     return ret

+def wic_plugin_checksums(bblayers, bbpath):
+    entries = []
+    seen = set()
+    paths = (bblayers or "").split() + [p for p in (bbpath or "").split(":") if p] +    for path in paths:
+        if path in seen:
+            continue
+        seen.add(path)

Quoted 8 lines

+        plugin_dir = os.path.join(path, "lib", "wic", "plugins") +        if os.path.isdir(plugin_dir):
+            for root, dirs, files in os.walk(plugin_dir):
+                dirs.sort()
+                for f in sorted(files):
+                    if f.endswith(".py"):
+                        entries.append(os.path.join(root, f) + ":True") +    return " ".join(entries)

Re-implementing the wic plugin search logic here is likely to lead to inconsistencies. For example, wic currently searches for plugins based on BBLAYERS but not BBPATH, and it looks in both scripts/lib/wic/plugins and lib/wic/plugins. Even if we align the two implementations now, it will be very easy for them to drift apart again in the future.

Could we instead rely on wic to perform the plugin search? There is currently a wic list source-plugins command, we could add a wic list imager-plugins command or perhaps a wic list plugins command to list all plugins. We could then call that here to get the list.

Quoted 4 lines

We should also move this to an event handler so that we don't unnecessarily re-run this search whenever the bitbake data store is updated. See image-postinst-intercepts.bbclass for an example of how to do this.

We need to run this at parse time and since wic is now an external tool, calling it in these contexts is effectively not possible...

Cheers,

Richard