Skip to contentSkip to conversation

siggen: clean_basepath leaks full recipe file path

RFC New Jean-Francois Dagenais 3 participants 17 messages Aug 20, 2020 – Sep 23, 2020 2 messages not in the archive

lore
Share & tools
lore
#0 lore raw

Hi all,

While investigating constant mismatches of hashes from our SSTATE_MIRROR depot, Maxime and I came across a bitbake-diffsigs which showed a suspicious cause for the mismatch. It appeared to be an underlying taskdeptasks which contains the full path of the recipe, which is totally a local build thing.

After searching for a little while. We came accross the clean_basepath(a) function in siggen.py. We added prints on it:

def clean_basepath(a):

    mc = None print('a: ' + a) if a.startswith("mc:"):
        _, mc, a = a.split(":", 2)
    b = a.rsplit("/", 2)[1] + '/' + a.rsplit("/", 2)[2] print('b: ' + b) if
    a.startswith("virtual:"):
        b = b + ":" + a.rsplit(":", 1)[0]
    if mc:
        b = b + ":mc:" + mc
    print('b+: ' + b) return b

Sure enough, this looks like the culprit. Consider the following trace:
a: virtual:native:/full/path/to/poky/meta/recipes-devtools/gnu-config/gnu-config_git.bb:do_populate_sysroot
b: gnu-config/gnu-config_git.bb:do_populate_sysroot
b+: gnu-config/gnu-config_git.bb:do_populate_sysroot:virtual:native:/full/path/to/poky/meta/recipes-devtools/gnu-config/gnu-config_git.bb

Looking at this, it really seems like the intent of the if a.startswith("virtual:") case is to insert only the :virtual:native at the end.

The simplest fix we suggest is simply changing a.rsplit(":", 1)[0] to a.rsplit(":", 2)[0] and causes b+ to become:

b+: gnu-config/gnu-config_git.bb:do_populate_sysroot:virtual:native

Since this is so profound and core to the way signatures work, coupled with the fact that this code hasn't changed in years, we feel being the ones to find and fix this is unlikely.

Perpahps the content/format of the passed a variable has changed recently?

The following email is a patch which could be applied if deemed correct. But please consider this re-written version:

def clean_basepath(basepath):

    # basepath is a string like '/full/path/to/recipes-whatever/helloworld/helloworld_0.1.2.bb:do_compile'
    # or 'virtual:native:/full/path/to/recipes-whatever/helloworld/helloworld_0.1.2.bb:do_compile'
    # or 'mc:somemachine:/full/path/to/recipes-whatever/helloworld/helloworld_0.1.2.bb:do_compile'

_, parent_dir, recipe_task = basepath.rsplit("/", 2)
# drop what is left of the second slash, searching from right to left cleaned = parent_dir + '/' + recipe_task

# if there was a 'abc:xyz' prefix, add it to the end if not basepath.startswith('/'):
cleaned += ':' + basepath.split('/', 1)[0].rstrip(':')

return cleaned

Both simple fix and this rewrite were tested with oe-selftest -r sstatetests.

We feel this version has these advantages:
* improved readablility
* improved efficiency (the split is not repeated twice and function it has fewer ifs)
* honours the open/close principle better by not hardcoding "virtual:" nor "mc:"

Thanks for your feedback! I will submit this version as a patch if it is positive.

#1 lore raw

PATCH 1/1 siggen: clean_basepath: remove recipe full path when virtual:xyz present

Before this fix, this example basepath (a):
virtual:native:/full/path/to/recipes-example/helloworld/helloworld_1.2.3.bb:do_compile
would get incorrectly "cleaned" into:
helloworld/helloworld_1.2.3.bb:do_compile:virtual:native:/full/path/to/recipes-example/helloworld/helloworld_1.2.3.bb

When searching backwards in a trying to isolate the 'virtual:xyz' to add it to the end of the string, we need to consider a still has the recipe path and taskname. So stoping the rsplit after only 1 split is not enough. We want to reach the second ':' from the end.

This way, we obtain:
helloworld/helloworld_1.2.3.bb:do_compile:virtual:native

---
 lib/bb/siggen.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/bb/siggen.py +1 -1
index 4c63b0ba..ad49d1e2 100644
@@ -752,7 +752,7 @@ def clean_basepath(a):
         _, mc, a = a.split(":", 2)
     b = a.rsplit("/", 2)[1] + '/' + a.rsplit("/", 2)[2]
     if a.startswith("virtual:"):
-        b = b + ":" + a.rsplit(":", 1)[0]
+        b = b + ":" + a.rsplit(":", 2)[0]
     if mc:
         b = b + ":mc:" + mc
     return b
Signature · 1 line
2.28.0
#2 lore raw
In reply to Jean-Francois Dagenais

Hi,

Quoted 85 lines · Jean-Francois Dagenais

While investigating constant mismatches of hashes from our SSTATE_MIRROR depot,
Maxime and I came across a bitbake-diffsigs which showed a suspicious cause for
the mismatch. It appeared to be an underlying taskdeptasks which contains the
full path of the recipe, which is totally a local build thing.

After searching for a little while. We came accross the clean_basepath(a)
function in siggen.py. We added prints on it:

def clean_basepath(a):

    mc = None print('a: ' + a) if a.startswith("mc:"):
        _, mc, a = a.split(":", 2)
    b = a.rsplit("/", 2)[1] + '/' + a.rsplit("/", 2)[2] print('b: ' +
b) if
    a.startswith("virtual:"):
        b = b + ":" + a.rsplit(":", 1)[0]
    if mc:
        b = b + ":mc:" + mc
    print('b+: ' + b) return b

Sure enough, this looks like the culprit. Consider the following trace:
a: virtual:native:/full/path/to/poky/meta/recipes-devtools/gnu- config/gnu-config_git.bb:do_populate_sysroot
b: gnu-config/gnu-config_git.bb:do_populate_sysroot
b+: gnu-config/gnu- config_git.bb:do_populate_sysroot:virtual:native:/full/path/to/poky/m eta/recipes-devtools/gnu-config/gnu-config_git.bb

Looking at this, it really seems like the intent of the if a.startswith("virtual:") case is to insert only the :virtual:native at the
end.

The simplest fix we suggest is simply changing a.rsplit(":", 1)[0] to
a.rsplit(":", 2)[0] and causes b+ to become:

b+: gnu-config/gnu-config_git.bb:do_populate_sysroot:virtual:native

Since this is so profound and core to the way signatures work, coupled with the
fact that this code hasn't changed in years, we feel being the ones to find and
fix this is unlikely.

Perpahps the content/format of the passed a variable has changed recently?

The following email is a patch which could be applied if deemed correct. But please
consider this re-written version:

def clean_basepath(basepath):

    # basepath is a string like '/full/path/to/recipes-
whatever/helloworld/helloworld_0.1.2.bb:do_compile'
    # or 'virtual:native:/full/path/to/recipes-
whatever/helloworld/helloworld_0.1.2.bb:do_compile'
    # or 'mc:somemachine:/full/path/to/recipes-
whatever/helloworld/helloworld_0.1.2.bb:do_compile'

_, parent_dir, recipe_task = basepath.rsplit("/", 2) # drop what is left of the second slash, searching from right to
left
cleaned = parent_dir + '/' + recipe_task

# if there was a 'abc:xyz' prefix, add it to the end if not basepath.startswith('/'):
cleaned += ':' + basepath.split('/', 1)[0].rstrip(':')

return cleaned

Both simple fix and this rewrite were tested with oe-selftest -r sstatetests.

We feel this version has these advantages:
* improved readablility
* improved efficiency (the split is not repeated twice and function it has fewer ifs)
* honours the open/close principle better by not hardcoding "virtual:" nor "mc:"

Quoted 2 lines

Thanks for your feedback! I will submit this version as a patch if it is positive.

Well done finding this, the bug has clearly been there a while! I've had a feeling there was something going on somewhere but never found it.

I've taken the opportunity to make the code slightly clearer so have written and sent a modified version of your patch. I hope that is ok! Please take a look and let me know if I've missed anything too.

Cheers,

Richard

#3 lore raw

Hi Richard, all,

Quoted 35 lines · Richard Purdie
Quoted 34 lines

The following email is a patch which could be applied if deemed correct. But please
consider this re-written version:

def clean_basepath(basepath):

   # basepath is a string like '/full/path/to/recipes-
whatever/helloworld/helloworld_0.1.2.bb:do_compile'
   # or 'virtual:native:/full/path/to/recipes-
whatever/helloworld/helloworld_0.1.2.bb:do_compile'
   # or 'mc:somemachine:/full/path/to/recipes-
whatever/helloworld/helloworld_0.1.2.bb:do_compile'

_, parent_dir, recipe_task = basepath.rsplit("/", 2)

   # drop what is left of the second slash, searching from right to
left
   cleaned = parent_dir + '/' + recipe_task
   # if there was a 'abc:xyz' prefix, add it to the end
   if not basepath.startswith('/'):
       cleaned += ':' + basepath.split('/', 1)[0].rstrip(':')

return cleaned

Both simple fix and this rewrite were tested with oe-selftest -r sstatetests.

We feel this version has these advantages:
* improved readablility
* improved efficiency (the split is not repeated twice and function it has fewer ifs)
* honours the open/close principle better by not hardcoding "virtual:" nor "mc:"

Quoted 6 lines
Quoted 2 lines

Thanks for your feedback! I will submit this version as a patch if it is positive.

Well done finding this, the bug has clearly been there a while! I've had a feeling there was something going on somewhere but never found it.

Cache misses/extra tasks running has always bugged me but it never became a high priority target until this week. We started using the hash equivalence server a few months back and just recently, CI pipelines of what should have otherwise been all cached work started talking hours and hours. This current issue might not be completely responsible for what we were rebuilding since all CI runners experience the same recipe space checkout path. Which means even though the full recipe paths are in the sstate cache objects, they are the same for all runners.

Your confirmation that we did find something so core and were able to fix it gives us more confidence to take a crack at the other issues we're seing. ;)

Quoted 5 lines

I've taken the opportunity to make the code slightly clearer so have written and sent a modified version of your patch. I hope that is ok! Please take a look and let me know if I've missed anything too.

It looks good, just using a slightly different method than the patch I sent:
https://lists.openembedded.org/g/bitbake-devel/topic/patch_siggen/76318034

Can you share why you remade it?

Also, my colleague Maxime mentioned test cases... It would be nice to write actual unit tests targetted directly on this function so I would then be able to submit with more confidence the refactored version of the function I included in the RFC just above here. (I still would like the community's review on it and comments on the advantages I submitted BTW.)

Perhaps if someone more familiar with these parts could send a list of possible formats the function encounters in all the different scenarios and give dummy before -> after strings. I could bundle this into unit tests along with my refactored function in an upcoming series.

Example: * 'virtual:something:/full/path/to/poky/meta/recipes-whatever/helloworld/helloworld_1.2.3.bb:do_sometask' --> 'helloworld/helloworld_1.2.3.bb:do_sometask:virtual:something' * 'mc:...' ->
'...' * etc.

Thanks for your thoughts and the rapid feedback!

#4 lore raw
Quoted 41 lines · Jean-Francois Dagenais
Quoted 33 lines · Richard Purdie

The following email is a patch which could be applied if deemed

Quoted 32 lines

correct. But please
consider this re-written version:

def clean_basepath(basepath):

   # basepath is a string like '/full/path/to/recipes-
whatever/helloworld/helloworld_0.1.2.bb:do_compile'
   # or 'virtual:native:/full/path/to/recipes-
whatever/helloworld/helloworld_0.1.2.bb:do_compile'
   # or 'mc:somemachine:/full/path/to/recipes-
whatever/helloworld/helloworld_0.1.2.bb:do_compile'

_, parent_dir, recipe_task = basepath.rsplit("/", 2)

   # drop what is left of the second slash, searching from right to
left
   cleaned = parent_dir + '/' + recipe_task
   # if there was a 'abc:xyz' prefix, add it to the end
   if not basepath.startswith('/'):
       cleaned += ':' + basepath.split('/', 1)[0].rstrip(':')

return cleaned

Both simple fix and this rewrite were tested with oe-selftest -r sstatetests.

We feel this version has these advantages:
* improved readablility
* improved efficiency (the split is not repeated twice and function it has fewer ifs)
* honours the open/close principle better by not hardcoding "virtual:" nor "mc:"

Quoted 6 lines
Quoted 2 lines

Thanks for your feedback! I will submit this version as a patch if it is positive.

Well done finding this, the bug has clearly been there a while! I've had a feeling there was something going on somewhere but never found it.

Quoted 7 lines

Cache misses/extra tasks running has always bugged me but it never became a high priority target until this week. We started using the hash equivalence server a few months back and just recently, CI pipelines of what should have otherwise been all cached work started talking hours and hours. This current issue might not be completely responsible for what we were rebuilding since all CI runners experience the same recipe space checkout path. Which means even though the full recipe paths are in the sstate cache objects, they are the same for all runners.

FWIW there was a recent change to package.bbclass to add a sorted() from Mark Hatle which would also have the above kind of affect, in that case based on disk directory entry ordering.

Quoted 2 lines

Your confirmation that we did find something so core and were able to fix it gives us more confidence to take a crack at the other issues we're seing. ;)

Please do try and figure these out! If I can help I will but obviously its difficult with no data...

Quoted 9 lines
Quoted 4 lines

I've taken the opportunity to make the code slightly clearer so have written and sent a modified version of your patch. I hope that is ok! Please take a look and let me know if I've missed anything too.

It looks good, just using a slightly different method than the patch I sent:
https://lists.openembedded.org/g/bitbake-devel/topic/patch_siggen/76318034

Can you share why you remade it?

I was thinking something slightly clearer was possible, I'd forgotten why it works the way it does though.

Quoted 11 lines

Also, my colleague Maxime mentioned test cases... It would be nice to write actual unit tests targetted directly on this function so I would then be able to submit with more confidence the refactored version of the function I included in the RFC just above here. (I still would like the community's review on it and comments on the advantages I submitted BTW.)

Perhaps if someone more familiar with these parts could send a list of possible formats the function encounters in all the different scenarios and give dummy before -> after strings. I could bundle this into unit tests along with my refactored function in an upcoming series.

Quoted 4 lines

Example: * 'virtual:something:/full/path/to/poky/meta/recipes-whatever/helloworld/helloworld_1.2.3.bb:do_sometask' --> 'helloworld/helloworld_1.2.3.bb:do_sometask:virtual:something' * 'mc:...' ->
'...' * etc.

As far as I can think offhand, the function accepts:

/path/recipedir/recipe_verison.bb:do_task virtual:A:/path/recipedir/recipe_verison.bb:do_task virtual:A:B:/path/recipedir/recipe_verison.bb:do_task virtual:A:B:C:/path/recipedir/recipe_verison.bb:do_task mc:X:virtual:A:/path/recipedir/recipe_verison.bb:do_task mc:X:virtual:A:B:/path/recipedir/recipe_verison.bb:do_task mc:X:virtual:A:B:C:/path/recipedir/recipe_verison.bb:do_task

and what we care about particularly is that given a list of these, the sorted() result of the return values is consistent, independent of path.

For example, we should get the same value back for:

/short/recipedir/recipe_version.bb
and
/this/is/a/long/path/recipedir/recipe_version.bb

so perhaps a unittest which tested that, iterating the above formats?

Cheers,

Richard

#5 lore raw
Quoted 15 lines · Richard Purdie

/path/recipedir/recipe_verison.bb:do_task virtual:A:/path/recipedir/recipe_verison.bb:do_task virtual:A:B:/path/recipedir/recipe_verison.bb:do_task virtual:A:B:C:/path/recipedir/recipe_verison.bb:do_task mc:X:virtual:A:/path/recipedir/recipe_verison.bb:do_task mc:X:virtual:A:B:/path/recipedir/recipe_verison.bb:do_task mc:X:virtual:A:B:C:/path/recipedir/recipe_verison.bb:do_task

and what we care about particularly is that given a list of these, the sorted() result of the return values is consistent, independent of path.

Quoted 7 lines

For example, we should get the same value back for:

/short/recipedir/recipe_version.bb
and
/this/is/a/long/path/recipedir/recipe_version.bb

so perhaps a unittest which tested that, iterating the above formats?

Would it be true to say then, that the ONLY purpose of clean_basepath() is to remove the full path of the recipe?

If true, then could we not simply only snip the path itself, and leave the rest in the same order?

So 'mc:X:virtual:A:B:C:/this/is/a/long/path/recipedir/recipe_verison.bb:do_task' would become 'mc:X:virtual:A:B:C:recipedir/recipe_verison.bb:do_task' ?

That would definitely simplify the implementation and be more future proof, would it not?

#6 lore raw
Quoted 22 lines · Jean-Francois Dagenais
Quoted 22 lines

On Aug 21, 2020, at 11:10, Richard Purdie <
[email protected]> wrote:

/path/recipedir/recipe_verison.bb:do_task virtual:A:/path/recipedir/recipe_verison.bb:do_task virtual:A:B:/path/recipedir/recipe_verison.bb:do_task virtual:A:B:C:/path/recipedir/recipe_verison.bb:do_task mc:X:virtual:A:/path/recipedir/recipe_verison.bb:do_task mc:X:virtual:A:B:/path/recipedir/recipe_verison.bb:do_task mc:X:virtual:A:B:C:/path/recipedir/recipe_verison.bb:do_task

and what we care about particularly is that given a list of these, the
sorted() result of the return values is consistent, independent of path.

For example, we should get the same value back for:

/short/recipedir/recipe_version.bb
and
/this/is/a/long/path/recipedir/recipe_version.bb

Quoted 5 lines
Quoted 2 lines

so perhaps a unittest which tested that, iterating the above formats?

Would it be true to say then, that the ONLY purpose of clean_basepath() is to remove the full path of the recipe?

No, although I was struggling to remember why it does this. The answer is:

http://git.yoctoproject.org/cgit.cgi/poky/commit/bitbake/lib/bb/siggen.py?id=5a3a7e0f746ad5dd160bc297d8c8b0e770630d89

although these days I don't think we use shared work tasks.

Quoted 11 lines

If true, then could we not simply only snip the path itself, and leave the rest in the same order?

So 'mc:X:virtual:A:B:C:/this/is/a/long/path/recipedir/recipe_verison.bb:
do_task'
would become 'mc:X:virtual:A:B:C:recipedir/recipe_verison.bb:do_task' ?

That would definitely simplify the implementation and be more future proof, would it not?

Well, there was a reason it does this. That reason may no longer be heavily used but there is a bit of work in deprecating and removing that shared work support which in turn would allow this to be cleaned up.

Cheers,

Richard

#7 lore raw

Hey again!

Quoted 6 lines · Richard Purdie

No, although I was struggling to remember why it does this. The answer is:

http://git.yoctoproject.org/cgit.cgi/poky/commit/bitbake/lib/bb/siggen.py?id=5a3a7e0f746ad5dd160bc297d8c8b0e770630d89

I understand. Thank you for this.

Quoted 12 lines

although these days I don't think we use shared work tasks.

Quoted 9 lines

If true, then could we not simply only snip the path itself, and leave the rest in the same order?

So 'mc:X:virtual:A:B:C:/this/is/a/long/path/recipedir/recipe_verison.bb:
do_task'
would become 'mc:X:virtual:A:B:C:recipedir/recipe_verison.bb:do_task' ?

Quoted 7 lines
Quoted 2 lines

That would definitely simplify the implementation and be more future proof, would it not?

Well, there was a reason it does this. That reason may no longer be heavily used but there is a bit of work in deprecating and removing that shared work support which in turn would allow this to be cleaned up.

I am writing the unit tests now.

From what I see though, it seems like the 'virtual:xyz' and 'mc:abc' are mirrored in the output. Meaning that:

'mc:X:virtual:A:B:C:/path/recipedir/recipe_version.bb:do_task'

becomes:

'recipedir/recipe_version.bb:do_task:virtual:A:B:C:mc:X'

Is this completely wanted and necessary?

Sorry for the counter examination! Just seeing what the actual real requirements are. Once the unit tests are in place they will carry that torch onward. ;)

#8 lore raw

PATCH 1/2 bitbake: tests/siggen: introduce clean_basepath testcases

In reply to Richard Purdie

While discussing with Richard we thought these might help document and safeguard the basic requirements of clean_basepath.

A 'bonus' performance testcase is added but commented out since its runtime is long and test machine specific. It is intended for developers to test before and after their changes to the target function as a due diligence verification.

---
 bin/bitbake-selftest   |  1 +
 lib/bb/tests/siggen.py | 91 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 92 insertions(+)
 create mode 100644 lib/bb/tests/siggen.py
bin/bitbake-selftest +1 -0
index e84d6a55..6c073741 100755
@@ -27,6 +27,7 @@ tests = ["bb.tests.codeparser",
          "bb.tests.parse",
          "bb.tests.persist_data",
          "bb.tests.runqueue",
+         "bb.tests.siggen",
          "bb.tests.utils",
          "hashserv.tests",
          "layerindexlib.tests.layerindexobj",
lib/bb/tests/siggen.py new file +91 -0

91 lines not on this page · raw message

Signature · 1 line
2.28.0
#9 lore raw

PATCH 2/2 bitbake: siggen: clean_basepath: improve perfo and readability

This change improves performance by reducing runtime about 33% for typical inputs. (using test_clean_basepath_performance)

It is also easier to read, and slightly more resilient to future changes since it doesn't mention 'virtual' anymore.

---
 lib/bb/siggen.py | 30 ++++++++++++++++++++----------
 1 file changed, 20 insertions(+), 10 deletions(-)
lib/bb/siggen.py +20 -10
index ad49d1e2..1456324a 100644
@@ -746,16 +746,26 @@ def list_inline_diff(oldlist, newlist, colors=None):
             ret.append(item)
     return '[%s]' % (', '.join(ret))
 
-def clean_basepath(a):
-    mc = None
-    if a.startswith("mc:"):
-        _, mc, a = a.split(":", 2)
-    b = a.rsplit("/", 2)[1] + '/' + a.rsplit("/", 2)[2]
-    if a.startswith("virtual:"):
-        b = b + ":" + a.rsplit(":", 2)[0]
-    if mc:
-        b = b + ":mc:" + mc
-    return b
+def clean_basepath(basepath):
+    basepath, dir, recipe_task = basepath.rsplit("/", 2)
+    cleaned = dir + '/' + recipe_task
+
+    if basepath[0] == '/':
+        return cleaned
+
+    if basepath.startswith("mc:"):
+        mc, mc_name, basepath = basepath.split(":", 2)
+        mc_suffix = ':mc:' + mc_name
+    else:
+        mc_suffix = ''
+
+    # mc stuff now removed from basepath. Whatever was next, if present will be the first
+    # suffix. ':/', recipe path start, marks the end of this. Something like
+    # 'virtual:a[:b[:c]]:/path...' (b and c being optional)
+    if basepath[0] != '/':
+        cleaned += ':' + basepath.split(':/', 1)[0]
+
+    return cleaned + mc_suffix
 
 def clean_basepaths(a):
     b = {}
Signature · 1 line
2.28.0
#11 lore raw

PATCH 1/2 bitbake: tests/siggen: introduce clean_basepath testcases

In reply to Richard Purdie

While discussing with Richard we thought these might help document and safeguard the basic requirements of clean_basepath.

A 'bonus' performance testcase is added but commented out since its runtime is long and test machine specific. It is intended for developers to test before and after their changes to the target function as a due diligence verification.

---
 bin/bitbake-selftest   |  1 +
 lib/bb/tests/siggen.py | 91 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 92 insertions(+)
 create mode 100644 lib/bb/tests/siggen.py
bin/bitbake-selftest +1 -0
index e84d6a55..6c073741 100755
@@ -27,6 +27,7 @@ tests = ["bb.tests.codeparser",
          "bb.tests.parse",
          "bb.tests.persist_data",
          "bb.tests.runqueue",
+         "bb.tests.siggen",
          "bb.tests.utils",
          "hashserv.tests",
          "layerindexlib.tests.layerindexobj",
lib/bb/tests/siggen.py new file +91 -0

91 lines not on this page · raw message

Signature · 1 line
2.28.0
#12 lore raw

PATCH 2/2 bitbake: siggen: clean_basepath: improve perfo and readability

This change improves performance by reducing runtime about 33% for typical inputs. (using test_clean_basepath_performance)

It is also easier to read, and slightly more resilient to future changes since it doesn't mention 'virtual' anymore.

---
 lib/bb/siggen.py | 30 ++++++++++++++++++++----------
 1 file changed, 20 insertions(+), 10 deletions(-)
lib/bb/siggen.py +20 -10
index ad49d1e2..1456324a 100644
@@ -746,16 +746,26 @@ def list_inline_diff(oldlist, newlist, colors=None):
             ret.append(item)
     return '[%s]' % (', '.join(ret))
 
-def clean_basepath(a):
-    mc = None
-    if a.startswith("mc:"):
-        _, mc, a = a.split(":", 2)
-    b = a.rsplit("/", 2)[1] + '/' + a.rsplit("/", 2)[2]
-    if a.startswith("virtual:"):
-        b = b + ":" + a.rsplit(":", 2)[0]
-    if mc:
-        b = b + ":mc:" + mc
-    return b
+def clean_basepath(basepath):
+    basepath, dir, recipe_task = basepath.rsplit("/", 2)
+    cleaned = dir + '/' + recipe_task
+
+    if basepath[0] == '/':
+        return cleaned
+
+    if basepath.startswith("mc:"):
+        mc, mc_name, basepath = basepath.split(":", 2)
+        mc_suffix = ':mc:' + mc_name
+    else:
+        mc_suffix = ''
+
+    # mc stuff now removed from basepath. Whatever was next, if present will be the first
+    # suffix. ':/', recipe path start, marks the end of this. Something like
+    # 'virtual:a[:b[:c]]:/path...' (b and c being optional)
+    if basepath[0] != '/':
+        cleaned += ':' + basepath.split(':/', 1)[0]
+
+    return cleaned + mc_suffix
 
 def clean_basepaths(a):
     b = {}
Signature · 1 line
2.28.0

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#16 lore raw
Quoted 97 lines · Richard Purdie via lists.openembedded.org

Hi,

Quoted 90 lines · Jean-Francois Dagenais

While investigating constant mismatches of hashes from our SSTATE_MIRROR depot,
Maxime and I came across a bitbake-diffsigs which showed a suspicious
cause for
the mismatch. It appeared to be an underlying taskdeptasks which contains the
full path of the recipe, which is totally a local build thing.

After searching for a little while. We came accross the clean_basepath(a)
function in siggen.py. We added prints on it:

def clean_basepath(a):

    mc = None print('a: ' + a) if a.startswith("mc:"):
        _, mc, a = a.split(":", 2)
    b = a.rsplit("/", 2)[1] + '/' + a.rsplit("/", 2)[2] print('b: '
+
b) if
    a.startswith("virtual:"):
        b = b + ":" + a.rsplit(":", 1)[0]
    if mc:
        b = b + ":mc:" + mc
    print('b+: ' + b) return b

Sure enough, this looks like the culprit. Consider the following trace:
a: virtual:native:/full/path/to/poky/meta/recipes-devtools/gnu- config/gnu-config_git.bb:do_populate_sysroot
b: gnu-config/gnu-config_git.bb:do_populate_sysroot
b+: gnu-config/gnu- config_git.bb:do_populate_sysroot:virtual:native:/full/path/to/poky /m
eta/recipes-devtools/gnu-config/gnu-config_git.bb

Looking at this, it really seems like the intent of the if a.startswith("virtual:") case is to insert only the :virtual:native at the
end.

The simplest fix we suggest is simply changing a.rsplit(":", 1)[0]
to
a.rsplit(":", 2)[0] and causes b+ to become:

b+: gnu-config/gnu-config_git.bb:do_populate_sysroot:virtual:native

Since this is so profound and core to the way signatures work, coupled with the
fact that this code hasn't changed in years, we feel being the ones to find and
fix this is unlikely.

Perpahps the content/format of the passed a variable has changed recently?

The following email is a patch which could be applied if deemed correct. But please
consider this re-written version:

def clean_basepath(basepath):

    # basepath is a string like '/full/path/to/recipes-
whatever/helloworld/helloworld_0.1.2.bb:do_compile'
    # or 'virtual:native:/full/path/to/recipes-
whatever/helloworld/helloworld_0.1.2.bb:do_compile'
    # or 'mc:somemachine:/full/path/to/recipes-
whatever/helloworld/helloworld_0.1.2.bb:do_compile'

_, parent_dir, recipe_task = basepath.rsplit("/", 2) # drop what is left of the second slash, searching from right
to
left
cleaned = parent_dir + '/' + recipe_task

# if there was a 'abc:xyz' prefix, add it to the end if not basepath.startswith('/'):
cleaned += ':' + basepath.split('/', 1)[0].rstrip(':')

return cleaned

Both simple fix and this rewrite were tested with oe-selftest -r sstatetests.

We feel this version has these advantages:
* improved readablility
* improved efficiency (the split is not repeated twice and function it has fewer ifs)
* honours the open/close principle better by not hardcoding "virtual:" nor "mc:"

Quoted 3 lines

Thanks for your feedback! I will submit this version as a patch if it
is positive.

Quoted 7 lines

Well done finding this, the bug has clearly been there a while! I've had a feeling there was something going on somewhere but never found it.

I've taken the opportunity to make the code slightly clearer so have written and sent a modified version of your patch. I hope that is ok! Please take a look and let me know if I've missed anything too.

Of course as soon as I send that I realised my patch is wrong since virtuals can have multiple entries, which is why this is written searching backwards.

Your patch is therefore the correct one!

Cheers,

Richard

#17 lore raw

Hello!

Quoted 104 lines · Richard Purdie
Quoted 101 lines · Richard Purdie via lists.openembedded.org

Hi,

Quoted 90 lines · Jean-Francois Dagenais

While investigating constant mismatches of hashes from our SSTATE_MIRROR depot,
Maxime and I came across a bitbake-diffsigs which showed a suspicious
cause for
the mismatch. It appeared to be an underlying taskdeptasks which contains the
full path of the recipe, which is totally a local build thing.

After searching for a little while. We came accross the clean_basepath(a)
function in siggen.py. We added prints on it:

def clean_basepath(a):

    mc = None print('a: ' + a) if a.startswith("mc:"):
        _, mc, a = a.split(":", 2)
    b = a.rsplit("/", 2)[1] + '/' + a.rsplit("/", 2)[2] print('b: '
+
b) if
    a.startswith("virtual:"):
        b = b + ":" + a.rsplit(":", 1)[0]
    if mc:
        b = b + ":mc:" + mc
    print('b+: ' + b) return b

Sure enough, this looks like the culprit. Consider the following trace:
a: virtual:native:/full/path/to/poky/meta/recipes-devtools/gnu- config/gnu-config_git.bb:do_populate_sysroot
b: gnu-config/gnu-config_git.bb:do_populate_sysroot
b+: gnu-config/gnu- config_git.bb:do_populate_sysroot:virtual:native:/full/path/to/poky /m
eta/recipes-devtools/gnu-config/gnu-config_git.bb

Looking at this, it really seems like the intent of the if a.startswith("virtual:") case is to insert only the :virtual:native at the
end.

The simplest fix we suggest is simply changing a.rsplit(":", 1)[0]
to
a.rsplit(":", 2)[0] and causes b+ to become:

b+: gnu-config/gnu-config_git.bb:do_populate_sysroot:virtual:native

Since this is so profound and core to the way signatures work, coupled with the
fact that this code hasn't changed in years, we feel being the ones to find and
fix this is unlikely.

Perpahps the content/format of the passed a variable has changed recently?

The following email is a patch which could be applied if deemed correct. But please
consider this re-written version:

def clean_basepath(basepath):

    # basepath is a string like '/full/path/to/recipes-
whatever/helloworld/helloworld_0.1.2.bb:do_compile'
    # or 'virtual:native:/full/path/to/recipes-
whatever/helloworld/helloworld_0.1.2.bb:do_compile'
    # or 'mc:somemachine:/full/path/to/recipes-
whatever/helloworld/helloworld_0.1.2.bb:do_compile'

_, parent_dir, recipe_task = basepath.rsplit("/", 2) # drop what is left of the second slash, searching from right
to
left
cleaned = parent_dir + '/' + recipe_task

# if there was a 'abc:xyz' prefix, add it to the end if not basepath.startswith('/'):
cleaned += ':' + basepath.split('/', 1)[0].rstrip(':')

return cleaned

Both simple fix and this rewrite were tested with oe-selftest -r sstatetests.

We feel this version has these advantages:
* improved readablility
* improved efficiency (the split is not repeated twice and function it has fewer ifs)
* honours the open/close principle better by not hardcoding "virtual:" nor "mc:"

Quoted 3 lines

Thanks for your feedback! I will submit this version as a patch if it
is positive.

Well done finding this, the bug has clearly been there a while! I've had a feeling there was something going on somewhere but never found it.

Quoted 7 lines
Quoted 3 lines

I've taken the opportunity to make the code slightly clearer so have written and sent a modified version of your patch. I hope that is ok! Please take a look and let me know if I've missed anything too.

Of course as soon as I send that I realised my patch is wrong since virtuals can have multiple entries, which is why this is written searching backwards.

Is there a way to add test cases for this? We would like to be able to write tests for this.

Thanks,
Maxime.

Quoted 6 lines

Your patch is therefore the correct one!

Cheers,

Richard

#18 lore raw
Quoted 123 lines · Maxime Roussin-Bélanger
Quoted 5 lines · Richard Purdie
Quoted 2 lines · Richard Purdie via lists.openembedded.org

Hi,

Quoted 7 lines
Quoted 7 lines · Jean-Francois Dagenais
Quoted 6 lines

While investigating constant mismatches of hashes from our SSTATE_MIRROR depot,
Maxime and I came across a bitbake-diffsigs which showed a suspicious
cause for
the mismatch. It appeared to be an underlying taskdeptasks

which

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

contains the
full path of the recipe, which is totally a local build thing.

After searching for a little while. We came accross the clean_basepath(a)
function in siggen.py. We added prints on it:

def clean_basepath(a):

    mc = None print('a: ' + a) if a.startswith("mc:"):
        _, mc, a = a.split(":", 2)
    b = a.rsplit("/", 2)[1] + '/' + a.rsplit("/", 2)[2]

print('b: '

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

+
b) if

    a.startswith("virtual:"):
        b = b + ":" + a.rsplit(":", 1)[0]
    if mc:
        b = b + ":mc:" + mc
    print('b+: ' + b) return b

Sure enough, this looks like the culprit. Consider the

following

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

trace:
a: virtual:native:/full/path/to/poky/meta/recipes-devtools/gnu- config/gnu-config_git.bb:do_populate_sysroot
b: gnu-config/gnu-config_git.bb:do_populate_sysroot b+: gnu-config/gnu-

config_git.bb:do_populate_sysroot:virtual:native:/full/path/to/poky

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

/m
eta/recipes-devtools/gnu-config/gnu-config_git.bb

Looking at this, it really seems like the intent of the if a.startswith("virtual:") case is to insert only the :virtual:native at the
end.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

The simplest fix we suggest is simply changing a.rsplit(":", 1)[0]
to
a.rsplit(":", 2)[0] and causes b+ to become:

b+: gnu-config/gnu-

config_git.bb:do_populate_sysroot:virtual:native

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Since this is so profound and core to the way signatures work, coupled with the
fact that this code hasn't changed in years, we feel being the

ones

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

to find and
fix this is unlikely.

Perpahps the content/format of the passed a variable has

changed

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines

recently?

The following email is a patch which could be applied if deemed correct. But please
consider this re-written version:

def clean_basepath(basepath):

    # basepath is a string like '/full/path/to/recipes-
whatever/helloworld/helloworld_0.1.2.bb:do_compile'
    # or 'virtual:native:/full/path/to/recipes-
whatever/helloworld/helloworld_0.1.2.bb:do_compile'
    # or 'mc:somemachine:/full/path/to/recipes-
whatever/helloworld/helloworld_0.1.2.bb:do_compile'
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

_, parent_dir, recipe_task = basepath.rsplit("/", 2) # drop what is left of the second slash, searching from

right

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

to
left
cleaned = parent_dir + '/' + recipe_task

# if there was a 'abc:xyz' prefix, add it to the end if not basepath.startswith('/'):
cleaned += ':' + basepath.split('/', 1)[0].rstrip(':')

return cleaned

Both simple fix and this rewrite were tested with `oe-selftest

-r

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

sstatetests`.

We feel this version has these advantages:
* improved readablility
* improved efficiency (the split is not repeated twice and

function

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

it has fewer ifs)
* honours the open/close principle better by not hardcoding "virtual:" nor "mc:"

Thanks for your feedback! I will submit this version as a patch

if

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

it
is positive.

Well done finding this, the bug has clearly been there a while!

I've

Quoted 1 line
Quoted 1 line

had a feeling there was something going on somewhere but never

found

Quoted 3 lines
Quoted 3 lines

it.

I've taken the opportunity to make the code slightly clearer so

have

Quoted 1 line
Quoted 1 line

written and sent a modified version of your patch. I hope that is

ok!

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Please take a look and let me know if I've missed anything too.

Of course as soon as I send that I realised my patch is wrong since virtuals can have multiple entries, which is why this is written searching backwards.

Is there a way to add test cases for this? We would like to be able to write tests for this.

Yes, some simple unit tests added to lib/bb/tests/ somewhere would be run by bitbake-selftest, I'd love to see those!

Cheers,

Richard