sstate.bbclass: remove previous version's stamp

There is a potential problem if we don't remove the previous version's stamp, for example:

The depend chain is:
libtool-native -> autoconf-native -> m4-native
We have two m4-native: 1.4.9 and 1.4.7

1) Clean all of them to make a fresh build so that we can reproduce the problem

$ bitbake m4-native autoconf-native libtool-native -ccleansstate

2) Build libtool-native so that the m4-native_1.4.17 will be built

$ bitbake libtool-native

3) Set PREFERRED_VERSION_m4-native = "1.4.9" and build again

$ bitbake libtool-native

4) Set PREFERRED_VERSION_m4-native = "1.4.17" and build again

$ bitbake libtool-native -ccleansstate && bitbake libtool-native

Then the build will fail:
[snip]

| m4: unrecognized option '--gnu'
| Try `m4 --help' for more information.
| autom4te: m4 failed with exit status: 1

[snip]

The is because when we change m4-native to 1.4.17 and build libtool-native again:
5) libtool-native depends on autoconf-native, and autoconf-native's version isn't change, so it can remove the current stamp and mirror the sstate (the one depends on m4-native_1.4.9) from the SSTATE_DIR correctly.

6) The mirrored autoconf-native depends on m4-native_1.4.17's do_populate_sysroot, and the stamp is already there (which is made by step 2), so it would do nothing, but this is incorrect, since the one that really in the sysroot is m4-native_1.4.9, then the error happens.

Remove previous version's stamp in sstate_clean() will fix the problem.

[YOCTO #5422]

Signed-off-by Robert Yang <[email protected]>

---
 meta/classes/sstate.bbclass | 22 +++++++++++++---------
 1 file changed, 13 insertions(+), 9 deletions(-)

meta/classes/sstate.bbclass +7 -5

index 35c3f85..bad8b78 100644
@@ -331,11 +331,18 @@ def sstate_clean_manifest(manifest, d):
 
 def sstate_clean(ss, d):
     import oe.path
+    import glob
+    import re
 
     d2 = d.createCopy()
+    stamp_clean = d.getVar("STAMPCLEAN", True)
     extrainf = d.getVarFlag("do_" + ss['task'], 'stamp-extra-info', True)
     if extrainf:
         d2.setVar("SSTATE_MANMACH", extrainf)
+        wildcard_stfile = "%s.do_%s*.%s" % (stamp_clean, ss['task'], extrainf)
+    else:
+        wildcard_stfile = "%s.do_%s*" % (stamp_clean, ss['task'])
+
     manifest = d2.expand("${SSTATE_MANFILEPREFIX}.%s" % ss['name'])
 
     if os.path.exists(manifest):

@@ -350,15 +357,12 @@ def sstate_clean(ss, d):
         for lock in locks:
             bb.utils.unlockfile(lock)
 
-    stfile = d.getVar("STAMP", True) + ".do_" + ss['task']
-    oe.path.remove(stfile)
-    oe.path.remove(stfile + "_setscene")
-    if extrainf:
-        oe.path.remove(stfile + ".*" + extrainf)

Review

Quoted 3 lines · Robert Yang

(quoted diff) +3 -0

+        # Keep the sigdata
+        if not re.match(re_sigdata, stfile):
+            oe.path.remove(stfile)

I'm not sure its worth the overhead of using regexps here. Can we not do something simple like:

if ".sigdata." not in stfile:

Cheers,

Richard

Quoted 10 lines · Richard Purdie

meta/classes/sstate.bbclass def sstate_clean(ss, d): · line 363

Quoted 3 lines · Robert Yang

meta/classes/sstate.bbclass +3 -0

+        # Keep the sigdata
+        if not re.match(re_sigdata, stfile):
+            oe.path.remove(stfile)

I'm not sure its worth the overhead of using regexps here. Can we not do something simple like:

if ".sigdata." not in stfile:

Sounds better, I've updated the PULL:

git://git.pokylinux.org/poky-contrib rbt/clean_stamp

and here is the patch:

meta/classes/sstate.bbclass +6 -4

index 35c3f85..d4e0dcb 100644
@@ -331,11 +331,17 @@ def sstate_clean_manifest(manifest, d):
 
  def sstate_clean(ss, d):
      import oe.path
+    import glob
 
      d2 = d.createCopy()
+    stamp_clean = d.getVar("STAMPCLEAN", True)
      extrainf = d.getVarFlag("do_" + ss['task'], 'stamp-extra-info', True)
      if extrainf:
          d2.setVar("SSTATE_MANMACH", extrainf)
+        wildcard_stfile = "%s.do_%s*.%s" % (stamp_clean, ss['task'], extrainf)
+    else:
+        wildcard_stfile = "%s.do_%s*" % (stamp_clean, ss['task'])
+
      manifest = d2.expand("${SSTATE_MANFILEPREFIX}.%s" % ss['name'])
 
      if os.path.exists(manifest):

@@ -350,15 +356,11 @@ def sstate_clean(ss, d):
          for lock in locks:
              bb.utils.unlockfile(lock)
 
-    stfile = d.getVar("STAMP", True) + ".do_" + ss['task']
-    oe.path.remove(stfile)
-    oe.path.remove(stfile + "_setscene")
-    if extrainf:

Quoted 55 lines · Robert Yang

Quoted 10 lines · Richard Purdie

meta/classes/sstate.bbclass def sstate_clean(ss, d): · line 363

Quoted 3 lines · Robert Yang

meta/classes/sstate.bbclass +3 -0

+        # Keep the sigdata
+        if not re.match(re_sigdata, stfile):
+            oe.path.remove(stfile)

I'm not sure its worth the overhead of using regexps here. Can we not do something simple like:

if ".sigdata." not in stfile:

Sounds better, I've updated the PULL:

git://git.pokylinux.org/poky-contrib rbt/clean_stamp

and here is the patch:

meta/classes/sstate.bbclass +7 -9

index 35c3f85..d4e0dcb 100644
@@ -331,11 +331,17 @@ def sstate_clean_manifest(manifest, d):
 
   def sstate_clean(ss, d):
       import oe.path
+    import glob
 
       d2 = d.createCopy()
+    stamp_clean = d.getVar("STAMPCLEAN", True)
       extrainf = d.getVarFlag("do_" + ss['task'], 'stamp-extra-info', True)
       if extrainf:
           d2.setVar("SSTATE_MANMACH", extrainf)
+        wildcard_stfile = "%s.do_%s*.%s" % (stamp_clean, ss['task'], extrainf)
+    else:
+        wildcard_stfile = "%s.do_%s*" % (stamp_clean, ss['task'])
+
       manifest = d2.expand("${SSTATE_MANFILEPREFIX}.%s" % ss['name'])
 
       if os.path.exists(manifest):

@@ -350,15 +356,11 @@ def sstate_clean(ss, d):
           for lock in locks:
               bb.utils.unlockfile(lock)
 
-    stfile = d.getVar("STAMP", True) + ".do_" + ss['task']
-    oe.path.remove(stfile)
-    oe.path.remove(stfile + "_setscene")
-    if extrainf:
-        oe.path.remove(stfile + ".*" + extrainf)
-        oe.path.remove(stfile + "_setscene" + ".*" + extrainf)
-    else:
-        oe.path.remove(stfile + ".*")
-        oe.path.remove(stfile + "_setscene" + ".*")
+    # Remove the current and previous stamps, but keep the sigdata

meta/classes/sstate.bbclass def sstate_clean(ss, d): · line 363

Quoted 4 lines

meta/classes/sstate.bbclass +4 -0

+    for stfile in glob.glob(wildcard_stfile):
+        # Keep the sigdata
+        if ".sigdata." not in stfile:
+            oe.path.remove(stfile)

Sorry Robert, there is one more issue here. You're doing "do_%s*" but this can match multiple tasks, e.g. do_package* would also wipe out do_package_write_ipk, do_package_write_rpm etc.

The easiest way to handle this is probably in the for loop, making sure that do_<task>. or do_<task>_setscene. are in the filename.

It might be worth putting some comments in the code about these subtleties.

Cheers,

Richard