Skip to contentSkip to conversation

sstatesig: Fix pn and taskname derivation in find_siginfo

PATCH Superseded [email protected] 2 participants 7 messages Jul 13, 2023 – Aug 8, 2023 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 sstatesig: Fix pn and taskname derivation in find_siginfo

From: Yang Xu <[email protected]>

The bb.siggen.compare_sigfiles method transforms the key format from [mc:<mc_name>:][virtual:][native:]<recipe path>:<taskname> to <recipe dir>/<recipe name>:<taskname>[:virtual][:native][:mc:<mc_name>] by clean_basepaths. However, find_siginfo uses the original format to get the package name (pn) and task name.

This commit corrects the method for deriving the pn and task name in find_siginfo and adds handling for multilib name.
And add test for compare_sigfiles and find_siginfo working together.

---

Notes:
v1: correct handling for pn and taskname for native target in find_siginfo
v2: add handling for multilib target in find_siginfo
v3: add testcase for compare_sigfiles and find_siginfo work together.
v4: optimize testcase to avoid non-deterministic fail

 .../recipes-test/binutils/binutils_%.bbappend |  2 +
 meta/lib/oe/sstatesig.py                      | 17 ++--
 meta/lib/oeqa/selftest/cases/sstatetests.py   | 83 +++++++++++++++++++
 3 files changed, 97 insertions(+), 5 deletions(-)
 create mode 100644 meta-selftest/recipes-test/binutils/binutils_%.bbappend
meta-selftest/recipes-test/binutils/binutils_%.bbappend new file +2 -0
new file mode 100644
index 0000000000..205720982c
@@ -0,0 +1,2 @@
+# This bbappend is used to alter the recipe using the test_recipe.inc file created by tests.
+include test_recipe.inc
meta/lib/oe/sstatesig.py +12 -5
index f943df181e..f041a0c430 100644
@@ -321,11 +321,18 @@ def find_siginfo(pn, taskname, taskhashlist, d):
     if not taskname:
         # We have to derive pn and taskname
         key = pn
-        splitit = key.split('.bb:')
-        taskname = splitit[1]
-        pn = os.path.basename(splitit[0]).split('_')[0]
-        if key.startswith('virtual:native:'):
-            pn = pn + '-native'
+        if key.count(':') >= 2:
+            splitit, taskname, affix = key.split(':', 2)
+        else:
+            splitit, taskname = key.split(':', 1)
+            affix = ''
+        pn = os.path.splitext(os.path.basename(splitit))[0].split('_')[0]
+        affixitems = affix.split(':')
+        if affixitems[0] == 'virtual':
+            if affixitems[1] == 'native':
+                pn = pn + '-native'
+            if affixitems[1] == 'multilib':
+                pn = affixitems[2] + '-' + pn
 
     hashfiles = {}
     filedates = {}
meta/lib/oeqa/selftest/cases/sstatetests.py +83 -0

86 lines not on this page · raw message

Signature · 1 line
2.18.0
#1 lore raw

Dear Sirs,

Sorry, could you please help to review patch v4?
In patch v4, I optimize the testcase to avoid non-deterministic fail.

Thank you

Quoted 175 lines · On Thu, 2023-07-13 at 02:16 +0000, [email protected] wrote:

From: Yang Xu <[email protected]>

The bb.siggen.compare_sigfiles method transforms the key format from
[mc:<mc_name>:][virtual:][native:]<recipe path>:<taskname> to <recipe dir>/<recipe name>:<taskname>[:virtual][:native][:mc:<mc_name>]
by clean_basepaths. However, find_siginfo uses the original format
to get the package name (pn) and task name.

This commit corrects the method for deriving the pn and task name in find_siginfo and adds handling for multilib name.
And add test for compare_sigfiles and find_siginfo working together.

---

Notes:
v1: correct handling for pn and taskname for native target in
find_siginfo
v2: add handling for multilib target in find_siginfo
v3: add testcase for compare_sigfiles and find_siginfo work
together.
v4: optimize testcase to avoid non-deterministic fail

 .../recipes-test/binutils/binutils_%.bbappend |  2 +
 meta/lib/oe/sstatesig.py                      | 17 ++--
 meta/lib/oeqa/selftest/cases/sstatetests.py   | 83
+++++++++++++++++++
 3 files changed, 97 insertions(+), 5 deletions(-)
 create mode 100644 meta-selftest/recipes-

test/binutils/binutils_%.bbappend

meta-selftest/recipes-test/binutils/binutils_%.bbappend new file +1 -0
new file mode 100644
index 0000000000..205720982c
@@ -0,0 +1,2 @@
+# This bbappend is used to alter the recipe using the

test_recipe.inc file created by tests.

meta-selftest/recipes-test/binutils/binutils_%.bbappend new file +1 -0
+include test_recipe.inc
meta/lib/oe/sstatesig.py +0 -0
index f943df181e..f041a0c430 100644
@@ -321,11 +321,18 @@ def find_siginfo(pn, taskname, taskhashlist,

d):
if not taskname:
# We have to derive pn and taskname
key = pn

meta/lib/oe/sstatesig.py +6 -5
-        splitit = key.split('.bb:')
-        taskname = splitit[1]
-        pn = os.path.basename(splitit[0]).split('_')[0]
-        if key.startswith('virtual:native:'):
-            pn = pn + '-native'
+        if key.count(':') >= 2:
+            splitit, taskname, affix = key.split(':', 2)
+        else:
+            splitit, taskname = key.split(':', 1)
+            affix = ''
+        pn =

os.path.splitext(os.path.basename(splitit))[0].split('_')[0]

meta/lib/oe/sstatesig.py +6 -0
+        affixitems = affix.split(':')
+        if affixitems[0] == 'virtual':
+            if affixitems[1] == 'native':
+                pn = pn + '-native'
+            if affixitems[1] == 'multilib':
+                pn = affixitems[2] + '-' + pn
 
     hashfiles = {}
     filedates = {}
meta/lib/oeqa/selftest/cases/sstatetests.py +4 -0
index febafdb2f7..3fa3038218 100644
@@ -691,3 +691,86 @@ TMPDIR = "${TOPDIR}/tmp-sstatesamehash2"
         self.maxDiff = None
         self.assertCountEqual(files1, files2)
 
+class SStateFindSiginfo(SStateBase):
+    def test_sstate_compare_sigfiles_and_find_siginfo(self):
+        """
+        Test the functionality of the find_siginfo: basic function

and callback in compare_sigfiles

meta/lib/oeqa/selftest/cases/sstatetests.py +11 -0
+        """
+        self.write_config("""
+TMPDIR = \"${TOPDIR}/tmp-sstates-findsiginfo\"
+TCLIBCAPPEND = \"\"
+MACHINE = \"qemux86-64\"
+require conf/multilib.conf
+MULTILIBS = "multilib:lib32"
+DEFAULTTUNE:virtclass-multilib-lib32 = "x86"
+BB_SIGNATURE_HANDLER = "OEBasicHash"
+""")
+        self.track_for_cleanup(self.topdir + "/tmp-sstates-

findsiginfo")

meta/lib/oeqa/selftest/cases/sstatetests.py +41 -0
+
+        pns = ["binutils", "binutils-native", "lib32-binutils"]
+        target_configs = [
+"""
+TMPVAL1 = "tmpval1"
+TMPVAL2 = "tmpval2"
+do_tmptask1() {
+    echo ${TMPVAL1}
+}
+do_tmptask2() {
+    echo ${TMPVAL2}
+}
+addtask do_tmptask1
+addtask tmptask2 before do_tmptask1
+""",
+"""
+TMPVAL3 = "tmpval3"
+TMPVAL4 = "tmpval4"
+do_tmptask1() {
+    echo ${TMPVAL3}
+}
+do_tmptask2() {
+    echo ${TMPVAL4}
+}
+addtask do_tmptask1
+addtask tmptask2 before do_tmptask1
+"""
+        ]
+
+        for target_config in target_configs:
+            self.write_recipeinc("binutils", target_config)
+            for pn in pns:
+                bitbake("%s -c do_tmptask1 -S none" % pn)
+            self.delete_recipeinc("binutils")
+
+        with bb.tinfoil.Tinfoil() as tinfoil:
+            tinfoil.prepare(config_only=True)
+
+            def find_siginfo(pn, taskname, sigs=None):
+                result = None
+                tinfoil.set_event_mask(["bb.event.FindSigInfoResult"

,

meta/lib/oeqa/selftest/cases/sstatetests.py +2 -0
+                                "bb.command.CommandCompleted"])
+                ret = tinfoil.run_command("findSigInfo", pn,

taskname, sigs)

meta/lib/oeqa/selftest/cases/sstatetests.py +5 -0
+                if ret:
+                    while True:
+                        event = tinfoil.wait_event(1)
+                        if event:
+                            if isinstance(event,

bb.command.CommandCompleted):

meta/lib/oeqa/selftest/cases/sstatetests.py +2 -0
+                                break
+                            elif isinstance(event,

bb.event.FindSigInfoResult):

meta/lib/oeqa/selftest/cases/sstatetests.py +10 -0
+                                result = event.result
+                return result
+
+            def recursecb(key, hash1, hash2):
+                nonlocal recursecb_count
+                recursecb_count += 1
+                hashes = [hash1, hash2]
+                hashfiles = find_siginfo(key, None, hashes)
+                self.assertCountEqual(hashes, hashfiles)
+                bb.siggen.compare_sigfiles(hashfiles[hash1],

hashfiles[hash2], recursecb)

meta/lib/oeqa/selftest/cases/sstatetests.py +6 -0
+
+            for pn in pns:
+                recursecb_count = 0
+                filedates = find_siginfo(pn, "do_tmptask1")
+                self.assertGreaterEqual(len(filedates), 2)
+                latestfiles = sorted(filedates.keys(), key=lambda f:

filedates[f])[-2:]
+ bb.siggen.compare_sigfiles(latestfiles[-2],
latestfiles[-1], recursecb)
+ self.assertEqual(recursecb_count,1)

#2 lore raw
In reply to [email protected]
Quoted 53 lines · Yang Xu via lists.openembedded.org

From: Yang Xu <[email protected]>

The bb.siggen.compare_sigfiles method transforms the key format from [mc:<mc_name>:][virtual:][native:]<recipe path>:<taskname> to <recipe dir>/<recipe name>:<taskname>[:virtual][:native][:mc:<mc_name>] by clean_basepaths. However, find_siginfo uses the original format to get the package name (pn) and task name.

This commit corrects the method for deriving the pn and task name in find_siginfo and adds handling for multilib name.
And add test for compare_sigfiles and find_siginfo working together.

---

Notes:
v1: correct handling for pn and taskname for native target in find_siginfo
v2: add handling for multilib target in find_siginfo
v3: add testcase for compare_sigfiles and find_siginfo work together.
v4: optimize testcase to avoid non-deterministic fail

 .../recipes-test/binutils/binutils_%.bbappend |  2 +
 meta/lib/oe/sstatesig.py                      | 17 ++--
 meta/lib/oeqa/selftest/cases/sstatetests.py   | 83 +++++++++++++++++++
 3 files changed, 97 insertions(+), 5 deletions(-)
 create mode 100644 meta-selftest/recipes-test/binutils/binutils_%.bbappend
meta-selftest/recipes-test/binutils/binutils_%.bbappend new file +2 -0
new file mode 100644
index 0000000000..205720982c
@@ -0,0 +1,2 @@
+# This bbappend is used to alter the recipe using the test_recipe.inc file created by tests.
+include test_recipe.inc
meta/lib/oe/sstatesig.py +5 -5
index f943df181e..f041a0c430 100644
@@ -321,11 +321,18 @@ def find_siginfo(pn, taskname, taskhashlist, d):
     if not taskname:
         # We have to derive pn and taskname
         key = pn
-        splitit = key.split('.bb:')
-        taskname = splitit[1]
-        pn = os.path.basename(splitit[0]).split('_')[0]
-        if key.startswith('virtual:native:'):
-            pn = pn + '-native'
+        if key.count(':') >= 2:
+            splitit, taskname, affix = key.split(':', 2)
+        else:
+            splitit, taskname = key.split(':', 1)
+            affix = ''

meta/lib/oe/sstatesig.py def find_siginfo(pn, taskname, taskhashlist, d): · line 338

Quoted 10 lines
meta/lib/oe/sstatesig.py +7 -0
+        pn = os.path.splitext(os.path.basename(splitit))[0].split('_')[0]
+        affixitems = affix.split(':')
+        if affixitems[0] == 'virtual':
+            if affixitems[1] == 'native':
+                pn = pn + '-native'
+            if affixitems[1] == 'multilib':
+                pn = affixitems[2] + '-' + pn
 
     hashfiles = {}
     filedates = {}

I've stared at this patch long and hard and I'm coming to the conclusion that whilst what you're doing improves things, there are more corner cases remaining and we're just moving the problem to new ones down the road. Having to hardcode in each of the class names and special case them is a big warning sign.

I started wondering why we encode pathnames into the siginfo files. The reason is that is how bitbake handles them within runqueue. In earlier times when this was being built, that was fine but things have been extended many times over since that decision was made.

The issue is that those "internal" representations don't map onto other systems, so sstate writes files in a different format. There is no easy way to map the original representations to the format used in the sstate file names.

My conclusion is that we should change the siginfo files to have the data already in "sstate" format. That does potentially break a few things but is probably worth doing for the simplicity gains in code like this.

Cheers,

Richard

A message that is not in the archive (sent off-list or not yet received): [email protected]

#4 lore raw
Quoted 83 lines · Richard Purdie via lists.openembedded.org
Quoted 53 lines · Yang Xu via lists.openembedded.org

From: Yang Xu <[email protected]>

The bb.siggen.compare_sigfiles method transforms the key format from [mc:<mc_name>:][virtual:][native:]<recipe path>:<taskname> to <recipe dir>/<recipe name>:<taskname>[:virtual][:native][:mc:<mc_name>] by clean_basepaths. However, find_siginfo uses the original format to get the package name (pn) and task name.

This commit corrects the method for deriving the pn and task name in find_siginfo and adds handling for multilib name.
And add test for compare_sigfiles and find_siginfo working together.

---

Notes:
v1: correct handling for pn and taskname for native target in find_siginfo
v2: add handling for multilib target in find_siginfo
v3: add testcase for compare_sigfiles and find_siginfo work together.
v4: optimize testcase to avoid non-deterministic fail

 .../recipes-test/binutils/binutils_%.bbappend |  2 +
 meta/lib/oe/sstatesig.py                      | 17 ++--
 meta/lib/oeqa/selftest/cases/sstatetests.py   | 83 +++++++++++++++++++
 3 files changed, 97 insertions(+), 5 deletions(-)
 create mode 100644 meta-selftest/recipes-test/binutils/binutils_%.bbappend
meta-selftest/recipes-test/binutils/binutils_%.bbappend new file +2 -0
new file mode 100644
index 0000000000..205720982c
@@ -0,0 +1,2 @@
+# This bbappend is used to alter the recipe using the test_recipe.inc file created by tests.
+include test_recipe.inc
meta/lib/oe/sstatesig.py +5 -5
index f943df181e..f041a0c430 100644
@@ -321,11 +321,18 @@ def find_siginfo(pn, taskname, taskhashlist, d):
     if not taskname:
         # We have to derive pn and taskname
         key = pn
-        splitit = key.split('.bb:')
-        taskname = splitit[1]
-        pn = os.path.basename(splitit[0]).split('_')[0]
-        if key.startswith('virtual:native:'):
-            pn = pn + '-native'
+        if key.count(':') >= 2:
+            splitit, taskname, affix = key.split(':', 2)
+        else:
+            splitit, taskname = key.split(':', 1)
+            affix = ''

meta/lib/oe/sstatesig.py def find_siginfo(pn, taskname, taskhashlist, d): · line 338

Quoted 10 lines
meta/lib/oe/sstatesig.py +7 -0
+        pn = os.path.splitext(os.path.basename(splitit))[0].split('_')[0]
+        affixitems = affix.split(':')
+        if affixitems[0] == 'virtual':
+            if affixitems[1] == 'native':
+                pn = pn + '-native'
+            if affixitems[1] == 'multilib':
+                pn = affixitems[2] + '-' + pn
 
     hashfiles = {}
     filedates = {}

I've stared at this patch long and hard and I'm coming to the conclusion that whilst what you're doing improves things, there are more corner cases remaining and we're just moving the problem to new ones down the road. Having to hardcode in each of the class names and special case them is a big warning sign.

I started wondering why we encode pathnames into the siginfo files. The reason is that is how bitbake handles them within runqueue. In earlier times when this was being built, that was fine but things have been extended many times over since that decision was made.

The issue is that those "internal" representations don't map onto other systems, so sstate writes files in a different format. There is no easy way to map the original representations to the format used in the sstate file names.

Quoted 4 lines

My conclusion is that we should change the siginfo files to have the data already in "sstate" format. That does potentially break a few things but is probably worth doing for the simplicity gains in code like this.

I've sent a couple of patches, one to bitbake and one to OE-Core which change the format of the data in runtaskdeps to be PN based rather than recipe filename based which then makes the above change unneeded, we no longer need to hardcode classes into it.

With those changes, I did confirm that the testcase added in this pattch does pass.

We need to review those other changes but assuming testing shows they work out ok could you update this patch to simply add the test case? I'd also appreciate it if you could check those patches do resolve the issue you were seeing.

Cheers,

Richard

#5 lore raw

It is a elegant solution for current issue. I will test and update my patch to only include testcase part soon.

Thank you

Yang

Quoted 137 lines · Richard Purdie

External email : Please do not click links or open attachments until you have verified the sender or the content.

Quoted 1 line · Richard Purdie via lists.openembedded.org

On Thu, 2023-07-13 at 02:16 +0000, Yang Xu via

lists.openembedded.org

Quoted 4 lines
Quoted 3 lines · wrote:

From: Yang Xu <[email protected]>

The bb.siggen.compare_sigfiles method transforms the key format

from

Quoted 2 lines
Quoted 2 lines

[mc:<mc_name>:][virtual:][native:]<recipe path>:<taskname> to `<recipe dir>/<recipe

name>:<taskname>[:virtual][:native][:mc:<mc_name>]`

Quoted 1 line
Quoted 1 line

by clean_basepaths. However, find_siginfo uses the original

format

Quoted 3 lines
Quoted 3 lines

to get the package name (pn) and task name.

This commit corrects the method for deriving the pn and task name

in

Quoted 2 lines
Quoted 2 lines

find_siginfo and adds handling for multilib name. And add test for compare_sigfiles and find_siginfo working

together.

Quoted 6 lines
Quoted 6 lines

---

Notes:
v1: correct handling for pn and taskname for native target in

find_siginfo

Quoted 2 lines
Quoted 2 lines

v2: add handling for multilib target in find_siginfo
v3: add testcase for compare_sigfiles and find_siginfo work

together.

Quoted 5 lines
Quoted 5 lines

v4: optimize testcase to avoid non-deterministic fail

 .../recipes-test/binutils/binutils_%.bbappend |  2 +
 meta/lib/oe/sstatesig.py                      | 17 ++--
 meta/lib/oeqa/selftest/cases/sstatetests.py   | 83

+++++++++++++++++++

Quoted 2 lines
Quoted 2 lines
 3 files changed, 97 insertions(+), 5 deletions(-)
 create mode 100644 meta-selftest/recipes-

test/binutils/binutils_%.bbappend

Quoted 2 lines

Review

Quoted 2 lines
(quoted diff) +0 -0

test/binutils/binutils_%.bbappend b/meta-selftest/recipes- test/binutils/binutils_%.bbappend

Quoted 6 lines

meta-selftest/recipes-test/binutils/binutils_%.bbappend line 1

Quoted 6 lines

new file mode 100644
index 0000000000..205720982c

meta-selftest/recipes-test/binutils/binutils_%.bbappend new file +1 -0
@@ -0,0 +1,2 @@
+# This bbappend is used to alter the recipe using the

test_recipe.inc file created by tests.

Quoted 6 lines

meta/lib/oe/sstatesig.py def find_siginfo(pn, taskname,

Quoted 6 lines
meta-selftest/recipes-test/binutils/binutils_%.bbappend new file +1 -0
+include test_recipe.inc
meta/lib/oe/sstatesig.py +0 -0
index f943df181e..f041a0c430 100644
@@ -321,11 +321,18 @@ def find_siginfo(pn, taskname,

taskhashlist, d):

Quoted 6 lines
Quoted 6 lines
meta/lib/oe/sstatesig.py +0 -3
     if not taskname:
         # We have to derive pn and taskname
         key = pn
-        splitit = key.split('.bb:')
-        taskname = splitit[1]
-        pn = os.path.basename(splitit[0]).split('_')[0]
Quoted 8 lines

meta/lib/oe/sstatesig.py def find_siginfo(pn, taskname, taskhashlist, d):

Quoted 8 lines
meta/lib/oe/sstatesig.py +6 -2
-        if key.startswith('virtual:native:'):
-            pn = pn + '-native'
+        if key.count(':') >= 2:
+            splitit, taskname, affix = key.split(':', 2)
+        else:
+            splitit, taskname = key.split(':', 1)
+            affix = ''
+        pn =

os.path.splitext(os.path.basename(splitit))[0].split('_')[0]

Quoted 7 lines
Quoted 7 lines
meta/lib/oe/sstatesig.py +6 -0
+        affixitems = affix.split(':')
+        if affixitems[0] == 'virtual':
+            if affixitems[1] == 'native':
+                pn = pn + '-native'
+            if affixitems[1] == 'multilib':
+                pn = affixitems[2] + '-' + pn
 
Quoted 7 lines

meta/lib/oe/sstatesig.py def find_siginfo(pn, taskname, taskhashlist, d): · line 338

Quoted 2 lines
meta/lib/oe/sstatesig.py +0 -0
     hashfiles = {}
     filedates = {}

I've stared at this patch long and hard and I'm coming to the conclusion that whilst what you're doing improves things, there are more corner cases remaining and we're just moving the problem to

new

Quoted 1 line

ones down the road. Having to hardcode in each of the class names

and

Quoted 3 lines

special case them is a big warning sign.

I started wondering why we encode pathnames into the siginfo files.

The

Quoted 1 line

reason is that is how bitbake handles them within runqueue. In

earlier

Quoted 4 lines

times when this was being built, that was fine but things have been extended many times over since that decision was made.

The issue is that those "internal" representations don't map onto

other

Quoted 1 line

systems, so sstate writes files in a different format. There is no

easy

Quoted 4 lines

way to map the original representations to the format used in the sstate file names.

My conclusion is that we should change the siginfo files to have

the

Quoted 3 lines

data already in "sstate" format. That does potentially break a few things but is probably worth doing for the simplicity gains in code like this.

I've sent a couple of patches, one to bitbake and one to OE-Core which
change the format of the data in runtaskdeps to be PN based rather than
recipe filename based which then makes the above change unneeded, we no
longer need to hardcode classes into it.

With those changes, I did confirm that the testcase added in this pattch does pass.

We need to review those other changes but assuming testing shows they work out ok could you update this patch to simply add the test case? I'd also appreciate it if you could check those patches do resolve the
issue you were seeing.

Cheers,

Richard

#7 lore raw

I got it, thank you :)

Yang

Quoted 4 lines · Richard Purdie

External email : Please do not click links or open attachments until you have verified the sender or the content.

Quoted 10 lines
Quoted 1 line · Yang Xu (徐扬)

It is a elegant solution for current issue. I will test and update

my

Quoted 3 lines

patch to only include testcase part soon.

Thank you

I did end up tweaking your patch to this:

https://git.yoctoproject.org/poky/commit/?h=master-next&id=3a79452507846756f6d78cab4dc4adafa0cb2be7

Quoted 4 lines

Cheers,

Richard