siggen: clean_basepath leaks full recipe file path
Hi all,
While investigating constant mismatches of hashes from our SSTATE_MIRROR depot, Maxime and I came across a bitbake-diffsigs which showed a suspicious cause for the mismatch. It appeared to be an underlying taskdeptasks which contains the full path of the recipe, which is totally a local build thing.
After searching for a little while. We came accross the clean_basepath(a) function in siggen.py. We added prints on it:
def clean_basepath(a):
mc = None print('a: ' + a) if a.startswith("mc:"):
_, mc, a = a.split(":", 2)
b = a.rsplit("/", 2)[1] + '/' + a.rsplit("/", 2)[2] print('b: ' + b) if
a.startswith("virtual:"):
b = b + ":" + a.rsplit(":", 1)[0]
if mc:
b = b + ":mc:" + mc
print('b+: ' + b) return bSure enough, this looks like the culprit. Consider the following trace:
a: virtual:native:/full/path/to/poky/meta/recipes-devtools/gnu-config/gnu-config_git.bb:do_populate_sysroot
b: gnu-config/gnu-config_git.bb:do_populate_sysroot
b+: gnu-config/gnu-config_git.bb:do_populate_sysroot:virtual:native:/full/path/to/poky/meta/recipes-devtools/gnu-config/gnu-config_git.bb
Looking at this, it really seems like the intent of the if a.startswith("virtual:") case is to insert only the :virtual:native at the end.
The simplest fix we suggest is simply changing a.rsplit(":", 1)[0] to a.rsplit(":", 2)[0] and causes b+ to become:
b+: gnu-config/gnu-config_git.bb:do_populate_sysroot:virtual:native
Since this is so profound and core to the way signatures work, coupled with the fact that this code hasn't changed in years, we feel being the ones to find and fix this is unlikely.
Perpahps the content/format of the passed a variable has changed recently?
The following email is a patch which could be applied if deemed correct. But please consider this re-written version:
def clean_basepath(basepath):
# basepath is a string like '/full/path/to/recipes-whatever/helloworld/helloworld_0.1.2.bb:do_compile'
# or 'virtual:native:/full/path/to/recipes-whatever/helloworld/helloworld_0.1.2.bb:do_compile'
# or 'mc:somemachine:/full/path/to/recipes-whatever/helloworld/helloworld_0.1.2.bb:do_compile' _, parent_dir, recipe_task = basepath.rsplit("/", 2)
# drop what is left of the second slash, searching from right to left cleaned = parent_dir + '/' + recipe_task
# if there was a 'abc:xyz' prefix, add it to the end if not basepath.startswith('/'):
cleaned += ':' + basepath.split('/', 1)[0].rstrip(':')
return cleaned
Both simple fix and this rewrite were tested with oe-selftest -r sstatetests.
We feel this version has these advantages:
* improved readablility
* improved efficiency (the split is not repeated twice and function it has fewer ifs)
* honours the open/close principle better by not hardcoding "virtual:" nor "mc:"
Thanks for your feedback! I will submit this version as a patch if it is positive.
PATCH 1/1 siggen: clean_basepath: remove recipe full path when virtual:xyz present
Before this fix, this example basepath (a):
virtual:native:/full/path/to/recipes-example/helloworld/helloworld_1.2.3.bb:do_compile
would get incorrectly "cleaned" into:
helloworld/helloworld_1.2.3.bb:do_compile:virtual:native:/full/path/to/recipes-example/helloworld/helloworld_1.2.3.bb
When searching backwards in a trying to isolate the 'virtual:xyz' to add it to the end of the string, we need to consider a still has the recipe path and taskname. So stoping the rsplit after only 1 split is not enough. We want to reach the second ':' from the end.
This way, we obtain:
helloworld/helloworld_1.2.3.bb:do_compile:virtual:native
--- lib/bb/siggen.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/bb/siggen.py +1 -1
@@ -752,7 +752,7 @@ def clean_basepath(a): _, mc, a = a.split(":", 2) b = a.rsplit("/", 2)[1] + '/' + a.rsplit("/", 2)[2] if a.startswith("virtual:"): - b = b + ":" + a.rsplit(":", 1)[0] + b = b + ":" + a.rsplit(":", 2)[0] if mc: b = b + ":mc:" + mc return b
Signature · 1 line
2.28.0
Hi,
Quoted 85 lines · Jean-Francois Dagenais
While investigating constant mismatches of hashes from our
SSTATE_MIRRORdepot,
Maxime and I came across a bitbake-diffsigs which showed a suspicious cause for
the mismatch. It appeared to be an underlying taskdeptasks which contains the
full path of the recipe, which is totally a local build thing.After searching for a little while. We came accross the
clean_basepath(a)
function insiggen.py. We added prints on it:def
clean_basepath(a):mc = None print('a: ' + a) if a.startswith("mc:"): _, mc, a = a.split(":", 2) b = a.rsplit("/", 2)[1] + '/' + a.rsplit("/", 2)[2] print('b: ' + b) if a.startswith("virtual:"): b = b + ":" + a.rsplit(":", 1)[0] if mc: b = b + ":mc:" + mc print('b+: ' + b) return bSure enough, this looks like the culprit. Consider the following trace:
a: virtual:native:/full/path/to/poky/meta/recipes-devtools/gnu- config/gnu-config_git.bb:do_populate_sysroot
b: gnu-config/gnu-config_git.bb:do_populate_sysroot
b+: gnu-config/gnu- config_git.bb:do_populate_sysroot:virtual:native:/full/path/to/poky/m eta/recipes-devtools/gnu-config/gnu-config_git.bbLooking at this, it really seems like the intent of the
if a.startswith("virtual:")case is to insert only the:virtual:nativeat the
end.The simplest fix we suggest is simply changing
a.rsplit(":", 1)[0]toa.rsplit(":", 2)[0]and causes b+ to become:b+: gnu-config/gnu-config_git.bb:do_populate_sysroot:virtual:native
Since this is so profound and core to the way signatures work, coupled with the
fact that this code hasn't changed in years, we feel being the ones to find and
fix this is unlikely.Perpahps the content/format of the passed
avariable has changed recently?The following email is a patch which could be applied if deemed correct. But please
consider this re-written version:def
clean_basepath(basepath):# basepath is a string like '/full/path/to/recipes- whatever/helloworld/helloworld_0.1.2.bb:do_compile' # or 'virtual:native:/full/path/to/recipes- whatever/helloworld/helloworld_0.1.2.bb:do_compile' # or 'mc:somemachine:/full/path/to/recipes- whatever/helloworld/helloworld_0.1.2.bb:do_compile'_,
parent_dir,recipe_task=basepath.rsplit("/", 2)# drop what is left of the second slash, searching from right to
left
cleaned =parent_dir+ '/' +recipe_task# if there was a 'abc:xyz' prefix, add it to the end if not
basepath.startswith('/'):
cleaned += ':' +basepath.split('/', 1)[0].rstrip(':')return cleaned
Both simple fix and this rewrite were tested with
oe-selftest -r sstatetests.We feel this version has these advantages:
* improved readablility
* improved efficiency (the split is not repeated twice and function it has fewerifs)
* honours the open/close principle better by not hardcoding "virtual:" nor "mc:"
Quoted 2 lines
Thanks for your feedback! I will submit this version as a patch if it is positive.
Well done finding this, the bug has clearly been there a while! I've had a feeling there was something going on somewhere but never found it.
I've taken the opportunity to make the code slightly clearer so have written and sent a modified version of your patch. I hope that is ok! Please take a look and let me know if I've missed anything too.
Cheers,
Richard
Hi Richard, all,
Quoted 35 lines · Richard Purdie
Quoted 34 lines
The following email is a patch which could be applied if deemed correct. But please
consider this re-written version:def
clean_basepath(basepath):# basepath is a string like '/full/path/to/recipes- whatever/helloworld/helloworld_0.1.2.bb:do_compile' # or 'virtual:native:/full/path/to/recipes- whatever/helloworld/helloworld_0.1.2.bb:do_compile' # or 'mc:somemachine:/full/path/to/recipes- whatever/helloworld/helloworld_0.1.2.bb:do_compile'_,
parent_dir,recipe_task=basepath.rsplit("/", 2)# drop what is left of the second slash, searching from right to left cleaned = parent_dir + '/' + recipe_task# if there was a 'abc:xyz' prefix, add it to the end if not basepath.startswith('/'): cleaned += ':' + basepath.split('/', 1)[0].rstrip(':')return cleaned
Both simple fix and this rewrite were tested with
oe-selftest -r sstatetests.We feel this version has these advantages:
* improved readablility
* improved efficiency (the split is not repeated twice and function it has fewerifs)
* honours the open/close principle better by not hardcoding "virtual:" nor "mc:"
Quoted 6 lines
Quoted 2 lines
Thanks for your feedback! I will submit this version as a patch if it is positive.
Well done finding this, the bug has clearly been there a while! I've had a feeling there was something going on somewhere but never found it.
Cache misses/extra tasks running has always bugged me but it never became a high priority target until this week. We started using the hash equivalence server a few months back and just recently, CI pipelines of what should have otherwise been all cached work started talking hours and hours. This current issue might not be completely responsible for what we were rebuilding since all CI runners experience the same recipe space checkout path. Which means even though the full recipe paths are in the sstate cache objects, they are the same for all runners.
Your confirmation that we did find something so core and were able to fix it gives us more confidence to take a crack at the other issues we're seing. ;)
Quoted 5 lines
I've taken the opportunity to make the code slightly clearer so have written and sent a modified version of your patch. I hope that is ok! Please take a look and let me know if I've missed anything too.
It looks good, just using a slightly different method than the patch I sent:
https://lists.openembedded.org/g/bitbake-devel/topic/patch_siggen/76318034
Can you share why you remade it?
Also, my colleague Maxime mentioned test cases... It would be nice to write actual unit tests targetted directly on this function so I would then be able to submit with more confidence the refactored version of the function I included in the RFC just above here. (I still would like the community's review on it and comments on the advantages I submitted BTW.)
Perhaps if someone more familiar with these parts could send a list of possible formats the function encounters in all the different scenarios and give dummy before -> after strings. I could bundle this into unit tests along with my refactored function in an upcoming series.
Example: * 'virtual:something:/full/path/to/poky/meta/recipes-whatever/helloworld/helloworld_1.2.3.bb:do_sometask' --> 'helloworld/helloworld_1.2.3.bb:do_sometask:virtual:something' * 'mc:...' ->
'...' * etc.
Thanks for your thoughts and the rapid feedback!
Quoted 41 lines · Jean-Francois Dagenais
Quoted 33 lines · Richard Purdie
The following email is a patch which could be applied if deemed
Quoted 32 lines
correct. But please
consider this re-written version:def
clean_basepath(basepath):# basepath is a string like '/full/path/to/recipes- whatever/helloworld/helloworld_0.1.2.bb:do_compile' # or 'virtual:native:/full/path/to/recipes- whatever/helloworld/helloworld_0.1.2.bb:do_compile' # or 'mc:somemachine:/full/path/to/recipes- whatever/helloworld/helloworld_0.1.2.bb:do_compile'_,
parent_dir,recipe_task=basepath.rsplit("/", 2)# drop what is left of the second slash, searching from right to left cleaned = parent_dir + '/' + recipe_task# if there was a 'abc:xyz' prefix, add it to the end if not basepath.startswith('/'): cleaned += ':' + basepath.split('/', 1)[0].rstrip(':')return cleaned
Both simple fix and this rewrite were tested with
oe-selftest -r sstatetests.We feel this version has these advantages:
* improved readablility
* improved efficiency (the split is not repeated twice and function it has fewerifs)
* honours the open/close principle better by not hardcoding "virtual:" nor "mc:"Quoted 6 lines
Quoted 2 lines
Thanks for your feedback! I will submit this version as a patch if it is positive.
Well done finding this, the bug has clearly been there a while! I've had a feeling there was something going on somewhere but never found it.
Quoted 7 lines
Cache misses/extra tasks running has always bugged me but it never became a high priority target until this week. We started using the hash equivalence server a few months back and just recently, CI pipelines of what should have otherwise been all cached work started talking hours and hours. This current issue might not be completely responsible for what we were rebuilding since all CI runners experience the same recipe space checkout path. Which means even though the full recipe paths are in the sstate cache objects, they are the same for all runners.
FWIW there was a recent change to package.bbclass to add a sorted() from Mark Hatle which would also have the above kind of affect, in that case based on disk directory entry ordering.
Quoted 2 lines
Your confirmation that we did find something so core and were able to fix it gives us more confidence to take a crack at the other issues we're seing. ;)
Please do try and figure these out! If I can help I will but obviously its difficult with no data...
Quoted 9 lines
Quoted 4 lines
I've taken the opportunity to make the code slightly clearer so have written and sent a modified version of your patch. I hope that is ok! Please take a look and let me know if I've missed anything too.
It looks good, just using a slightly different method than the patch I sent:
https://lists.openembedded.org/g/bitbake-devel/topic/patch_siggen/76318034Can you share why you remade it?
I was thinking something slightly clearer was possible, I'd forgotten why it works the way it does though.
Quoted 11 lines
Also, my colleague Maxime mentioned test cases... It would be nice to write actual unit tests targetted directly on this function so I would then be able to submit with more confidence the refactored version of the function I included in the RFC just above here. (I still would like the community's review on it and comments on the advantages I submitted BTW.)
Perhaps if someone more familiar with these parts could send a list of possible formats the function encounters in all the different scenarios and give dummy before -> after strings. I could bundle this into unit tests along with my refactored function in an upcoming series.
Quoted 4 lines
Example: * 'virtual:something:/full/path/to/poky/meta/recipes-whatever/helloworld/helloworld_1.2.3.bb:do_sometask' --> 'helloworld/helloworld_1.2.3.bb:do_sometask:virtual:something' * 'mc:...' ->
'...' * etc.
As far as I can think offhand, the function accepts:
/path/recipedir/recipe_verison.bb:do_task virtual:A:/path/recipedir/recipe_verison.bb:do_task virtual:A:B:/path/recipedir/recipe_verison.bb:do_task virtual:A:B:C:/path/recipedir/recipe_verison.bb:do_task mc:X:virtual:A:/path/recipedir/recipe_verison.bb:do_task mc:X:virtual:A:B:/path/recipedir/recipe_verison.bb:do_task mc:X:virtual:A:B:C:/path/recipedir/recipe_verison.bb:do_task
and what we care about particularly is that given a list of these, the sorted() result of the return values is consistent, independent of path.
For example, we should get the same value back for:
/short/recipedir/recipe_version.bb
and
/this/is/a/long/path/recipedir/recipe_version.bb
so perhaps a unittest which tested that, iterating the above formats?
Cheers,
Richard
Quoted 15 lines · Richard Purdie
/path/recipedir/recipe_verison.bb:do_task virtual:A:/path/recipedir/recipe_verison.bb:do_task virtual:A:B:/path/recipedir/recipe_verison.bb:do_task virtual:A:B:C:/path/recipedir/recipe_verison.bb:do_task mc:X:virtual:A:/path/recipedir/recipe_verison.bb:do_task mc:X:virtual:A:B:/path/recipedir/recipe_verison.bb:do_task mc:X:virtual:A:B:C:/path/recipedir/recipe_verison.bb:do_task
and what we care about particularly is that given a list of these, the
sorted()result of the return values is consistent, independent of path.
Quoted 7 lines
For example, we should get the same value back for:
/short/recipedir/recipe_version.bb
and
/this/is/a/long/path/recipedir/recipe_version.bbso perhaps a unittest which tested that, iterating the above formats?
Would it be true to say then, that the ONLY purpose of clean_basepath() is to remove the full path of the recipe?
If true, then could we not simply only snip the path itself, and leave the rest in the same order?
So 'mc:X:virtual:A:B:C:/this/is/a/long/path/recipedir/recipe_verison.bb:do_task' would become 'mc:X:virtual:A:B:C:recipedir/recipe_verison.bb:do_task' ?
That would definitely simplify the implementation and be more future proof, would it not?
Quoted 22 lines · Jean-Francois Dagenais
Quoted 22 lines
On Aug 21, 2020, at 11:10, Richard Purdie <
[email protected]> wrote:/path/recipedir/recipe_verison.bb:do_task virtual:A:/path/recipedir/recipe_verison.bb:do_task virtual:A:B:/path/recipedir/recipe_verison.bb:do_task virtual:A:B:C:/path/recipedir/recipe_verison.bb:do_task mc:X:virtual:A:/path/recipedir/recipe_verison.bb:do_task mc:X:virtual:A:B:/path/recipedir/recipe_verison.bb:do_task mc:X:virtual:A:B:C:/path/recipedir/recipe_verison.bb:do_task
and what we care about particularly is that given a list of these, the
sorted()result of the return values is consistent, independent of path.For example, we should get the same value back for:
/short/recipedir/recipe_version.bb
and
/this/is/a/long/path/recipedir/recipe_version.bb
Quoted 5 lines
Quoted 2 lines
so perhaps a unittest which tested that, iterating the above formats?
Would it be true to say then, that the ONLY purpose of
clean_basepath()is to remove the full path of the recipe?
No, although I was struggling to remember why it does this. The answer is:
although these days I don't think we use shared work tasks.
Quoted 11 lines
If true, then could we not simply only snip the path itself, and leave the rest in the same order?
So 'mc:X:virtual:A:B:C:/this/is/a/long/path/recipedir/recipe_verison.bb:
do_task'
would become 'mc:X:virtual:A:B:C:recipedir/recipe_verison.bb:do_task' ?That would definitely simplify the implementation and be more future proof, would it not?
Well, there was a reason it does this. That reason may no longer be heavily used but there is a bit of work in deprecating and removing that shared work support which in turn would allow this to be cleaned up.
Cheers,
Richard
Hey again!
Quoted 6 lines · Richard Purdie
No, although I was struggling to remember why it does this. The answer is:
I understand. Thank you for this.
Quoted 12 lines
although these days I don't think we use shared work tasks.
Quoted 9 lines
If true, then could we not simply only snip the path itself, and leave the rest in the same order?
So 'mc:X:virtual:A:B:C:/this/is/a/long/path/recipedir/recipe_verison.bb:
do_task'
would become 'mc:X:virtual:A:B:C:recipedir/recipe_verison.bb:do_task' ?
Quoted 7 lines
Quoted 2 lines
That would definitely simplify the implementation and be more future proof, would it not?
Well, there was a reason it does this. That reason may no longer be heavily used but there is a bit of work in deprecating and removing that shared work support which in turn would allow this to be cleaned up.
I am writing the unit tests now.
From what I see though, it seems like the 'virtual:xyz' and 'mc:abc' are mirrored in the output. Meaning that:
'mc:X:virtual:A:B:C:/path/recipedir/recipe_version.bb:do_task'
becomes:
'recipedir/recipe_version.bb:do_task:virtual:A:B:C:mc:X'
Is this completely wanted and necessary?
Sorry for the counter examination! Just seeing what the actual real requirements are. Once the unit tests are in place they will carry that torch onward. ;)
PATCH 1/2 bitbake: tests/siggen: introduce clean_basepath testcases
In reply to Richard PurdieWhile discussing with Richard we thought these might help document and safeguard the basic requirements of clean_basepath.
A 'bonus' performance testcase is added but commented out since its runtime is long and test machine specific. It is intended for developers to test before and after their changes to the target function as a due diligence verification.
--- bin/bitbake-selftest | 1 + lib/bb/tests/siggen.py | 91 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 92 insertions(+) create mode 100644 lib/bb/tests/siggen.py
bin/bitbake-selftest +1 -0
@@ -27,6 +27,7 @@ tests = ["bb.tests.codeparser", "bb.tests.parse", "bb.tests.persist_data", "bb.tests.runqueue", + "bb.tests.siggen", "bb.tests.utils", "hashserv.tests", "layerindexlib.tests.layerindexobj",
lib/bb/tests/siggen.py new file +91 -0
91 lines not on this page · raw message
Signature · 1 line
2.28.0
PATCH 2/2 bitbake: siggen: clean_basepath: improve perfo and readability
This change improves performance by reducing runtime about 33% for typical inputs. (using test_clean_basepath_performance)
It is also easier to read, and slightly more resilient to future changes since it doesn't mention 'virtual' anymore.
--- lib/bb/siggen.py | 30 ++++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-)
lib/bb/siggen.py +20 -10
@@ -746,16 +746,26 @@ def list_inline_diff(oldlist, newlist, colors=None): ret.append(item) return '[%s]' % (', '.join(ret)) -def clean_basepath(a): - mc = None - if a.startswith("mc:"): - _, mc, a = a.split(":", 2) - b = a.rsplit("/", 2)[1] + '/' + a.rsplit("/", 2)[2] - if a.startswith("virtual:"): - b = b + ":" + a.rsplit(":", 2)[0] - if mc: - b = b + ":mc:" + mc - return b +def clean_basepath(basepath): + basepath, dir, recipe_task = basepath.rsplit("/", 2) + cleaned = dir + '/' + recipe_task + + if basepath[0] == '/': + return cleaned + + if basepath.startswith("mc:"): + mc, mc_name, basepath = basepath.split(":", 2) + mc_suffix = ':mc:' + mc_name + else: + mc_suffix = '' + + # mc stuff now removed from basepath. Whatever was next, if present will be the first + # suffix. ':/', recipe path start, marks the end of this. Something like + # 'virtual:a[:b[:c]]:/path...' (b and c being optional) + if basepath[0] != '/': + cleaned += ':' + basepath.split(':/', 1)[0] + + return cleaned + mc_suffix def clean_basepaths(a): b = {}
Signature · 1 line
2.28.0
Re: [PATCH 2/2] bitbake: siggen: clean_basepath: improve perfo and readability
Quoted 5 lines · Jean-Francois Dagenais
Sigend-off-by Maxime Roussin-Bélanger <[email protected]>
Typo here... sorry. Resend?
PATCH 1/2 bitbake: tests/siggen: introduce clean_basepath testcases
In reply to Richard PurdieWhile discussing with Richard we thought these might help document and safeguard the basic requirements of clean_basepath.
A 'bonus' performance testcase is added but commented out since its runtime is long and test machine specific. It is intended for developers to test before and after their changes to the target function as a due diligence verification.
--- bin/bitbake-selftest | 1 + lib/bb/tests/siggen.py | 91 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 92 insertions(+) create mode 100644 lib/bb/tests/siggen.py
bin/bitbake-selftest +1 -0
@@ -27,6 +27,7 @@ tests = ["bb.tests.codeparser", "bb.tests.parse", "bb.tests.persist_data", "bb.tests.runqueue", + "bb.tests.siggen", "bb.tests.utils", "hashserv.tests", "layerindexlib.tests.layerindexobj",
lib/bb/tests/siggen.py new file +91 -0
91 lines not on this page · raw message
Signature · 1 line
2.28.0
PATCH 2/2 bitbake: siggen: clean_basepath: improve perfo and readability
This change improves performance by reducing runtime about 33% for typical inputs. (using test_clean_basepath_performance)
It is also easier to read, and slightly more resilient to future changes since it doesn't mention 'virtual' anymore.
--- lib/bb/siggen.py | 30 ++++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-)
lib/bb/siggen.py +20 -10
@@ -746,16 +746,26 @@ def list_inline_diff(oldlist, newlist, colors=None): ret.append(item) return '[%s]' % (', '.join(ret)) -def clean_basepath(a): - mc = None - if a.startswith("mc:"): - _, mc, a = a.split(":", 2) - b = a.rsplit("/", 2)[1] + '/' + a.rsplit("/", 2)[2] - if a.startswith("virtual:"): - b = b + ":" + a.rsplit(":", 2)[0] - if mc: - b = b + ":mc:" + mc - return b +def clean_basepath(basepath): + basepath, dir, recipe_task = basepath.rsplit("/", 2) + cleaned = dir + '/' + recipe_task + + if basepath[0] == '/': + return cleaned + + if basepath.startswith("mc:"): + mc, mc_name, basepath = basepath.split(":", 2) + mc_suffix = ':mc:' + mc_name + else: + mc_suffix = '' + + # mc stuff now removed from basepath. Whatever was next, if present will be the first + # suffix. ':/', recipe path start, marks the end of this. Something like + # 'virtual:a[:b[:c]]:/path...' (b and c being optional) + if basepath[0] != '/': + cleaned += ':' + basepath.split(':/', 1)[0] + + return cleaned + mc_suffix def clean_basepaths(a): b = {}
Signature · 1 line
2.28.0
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 9 lines · Jean-Francois Dagenais via lists.openembedded.org
It looks good, just using a slightly different method than the patch I sent:
https://lists.openembedded.org/g/bitbake-devel/topic/patch_siggen/76318034Can you share why you remade it?
Never mind this part, just saw you replied about this while I was writing! ;)
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 97 lines · Richard Purdie via lists.openembedded.org
Hi,
Quoted 90 lines · Jean-Francois Dagenais
While investigating constant mismatches of hashes from our
SSTATE_MIRRORdepot,
Maxime and I came across a bitbake-diffsigs which showed a suspicious
cause for
the mismatch. It appeared to be an underlying taskdeptasks which contains the
full path of the recipe, which is totally a local build thing.After searching for a little while. We came accross the
clean_basepath(a)
function insiggen.py. We added prints on it:def
clean_basepath(a):mc = None print('a: ' + a) if a.startswith("mc:"): _, mc, a = a.split(":", 2) b = a.rsplit("/", 2)[1] + '/' + a.rsplit("/", 2)[2] print('b: ' + b) if a.startswith("virtual:"): b = b + ":" + a.rsplit(":", 1)[0] if mc: b = b + ":mc:" + mc print('b+: ' + b) return bSure enough, this looks like the culprit. Consider the following trace:
a: virtual:native:/full/path/to/poky/meta/recipes-devtools/gnu- config/gnu-config_git.bb:do_populate_sysroot
b: gnu-config/gnu-config_git.bb:do_populate_sysroot
b+: gnu-config/gnu- config_git.bb:do_populate_sysroot:virtual:native:/full/path/to/poky /m
eta/recipes-devtools/gnu-config/gnu-config_git.bbLooking at this, it really seems like the intent of the
if a.startswith("virtual:")case is to insert only the:virtual:nativeat the
end.The simplest fix we suggest is simply changing
a.rsplit(":", 1)[0]
toa.rsplit(":", 2)[0]and causes b+ to become:b+: gnu-config/gnu-config_git.bb:do_populate_sysroot:virtual:native
Since this is so profound and core to the way signatures work, coupled with the
fact that this code hasn't changed in years, we feel being the ones to find and
fix this is unlikely.Perpahps the content/format of the passed
avariable has changed recently?The following email is a patch which could be applied if deemed correct. But please
consider this re-written version:def
clean_basepath(basepath):# basepath is a string like '/full/path/to/recipes- whatever/helloworld/helloworld_0.1.2.bb:do_compile' # or 'virtual:native:/full/path/to/recipes- whatever/helloworld/helloworld_0.1.2.bb:do_compile' # or 'mc:somemachine:/full/path/to/recipes- whatever/helloworld/helloworld_0.1.2.bb:do_compile'_,
parent_dir,recipe_task=basepath.rsplit("/", 2)# drop what is left of the second slash, searching from right
to
left
cleaned =parent_dir+ '/' +recipe_task# if there was a 'abc:xyz' prefix, add it to the end if not
basepath.startswith('/'):
cleaned += ':' +basepath.split('/', 1)[0].rstrip(':')return cleaned
Both simple fix and this rewrite were tested with
oe-selftest -r sstatetests.We feel this version has these advantages:
* improved readablility
* improved efficiency (the split is not repeated twice and function it has fewerifs)
* honours the open/close principle better by not hardcoding "virtual:" nor "mc:"Quoted 3 lines
Thanks for your feedback! I will submit this version as a patch if it
is positive.
Quoted 7 lines
Well done finding this, the bug has clearly been there a while! I've had a feeling there was something going on somewhere but never found it.
I've taken the opportunity to make the code slightly clearer so have written and sent a modified version of your patch. I hope that is ok! Please take a look and let me know if I've missed anything too.
Of course as soon as I send that I realised my patch is wrong since virtuals can have multiple entries, which is why this is written searching backwards.
Your patch is therefore the correct one!
Cheers,
Richard
Hello!
Quoted 104 lines · Richard Purdie
Quoted 101 lines · Richard Purdie via lists.openembedded.org
Hi,
Quoted 90 lines · Jean-Francois Dagenais
While investigating constant mismatches of hashes from our
SSTATE_MIRRORdepot,
Maxime and I came across a bitbake-diffsigs which showed a suspicious
cause for
the mismatch. It appeared to be an underlying taskdeptasks which contains the
full path of the recipe, which is totally a local build thing.After searching for a little while. We came accross the
clean_basepath(a)
function insiggen.py. We added prints on it:def
clean_basepath(a):mc = None print('a: ' + a) if a.startswith("mc:"): _, mc, a = a.split(":", 2) b = a.rsplit("/", 2)[1] + '/' + a.rsplit("/", 2)[2] print('b: ' + b) if a.startswith("virtual:"): b = b + ":" + a.rsplit(":", 1)[0] if mc: b = b + ":mc:" + mc print('b+: ' + b) return bSure enough, this looks like the culprit. Consider the following trace:
a: virtual:native:/full/path/to/poky/meta/recipes-devtools/gnu- config/gnu-config_git.bb:do_populate_sysroot
b: gnu-config/gnu-config_git.bb:do_populate_sysroot
b+: gnu-config/gnu- config_git.bb:do_populate_sysroot:virtual:native:/full/path/to/poky /m
eta/recipes-devtools/gnu-config/gnu-config_git.bbLooking at this, it really seems like the intent of the
if a.startswith("virtual:")case is to insert only the:virtual:nativeat the
end.The simplest fix we suggest is simply changing
a.rsplit(":", 1)[0]
toa.rsplit(":", 2)[0]and causes b+ to become:b+: gnu-config/gnu-config_git.bb:do_populate_sysroot:virtual:native
Since this is so profound and core to the way signatures work, coupled with the
fact that this code hasn't changed in years, we feel being the ones to find and
fix this is unlikely.Perpahps the content/format of the passed
avariable has changed recently?The following email is a patch which could be applied if deemed correct. But please
consider this re-written version:def
clean_basepath(basepath):# basepath is a string like '/full/path/to/recipes- whatever/helloworld/helloworld_0.1.2.bb:do_compile' # or 'virtual:native:/full/path/to/recipes- whatever/helloworld/helloworld_0.1.2.bb:do_compile' # or 'mc:somemachine:/full/path/to/recipes- whatever/helloworld/helloworld_0.1.2.bb:do_compile'_,
parent_dir,recipe_task=basepath.rsplit("/", 2)# drop what is left of the second slash, searching from right
to
left
cleaned =parent_dir+ '/' +recipe_task# if there was a 'abc:xyz' prefix, add it to the end if not
basepath.startswith('/'):
cleaned += ':' +basepath.split('/', 1)[0].rstrip(':')return cleaned
Both simple fix and this rewrite were tested with
oe-selftest -r sstatetests.We feel this version has these advantages:
* improved readablility
* improved efficiency (the split is not repeated twice and function it has fewerifs)
* honours the open/close principle better by not hardcoding "virtual:" nor "mc:"Quoted 3 lines
Thanks for your feedback! I will submit this version as a patch if it
is positive.Well done finding this, the bug has clearly been there a while! I've had a feeling there was something going on somewhere but never found it.
Quoted 7 lines
Quoted 3 lines
I've taken the opportunity to make the code slightly clearer so have written and sent a modified version of your patch. I hope that is ok! Please take a look and let me know if I've missed anything too.
Of course as soon as I send that I realised my patch is wrong since virtuals can have multiple entries, which is why this is written searching backwards.
Is there a way to add test cases for this? We would like to be able to write tests for this.
Thanks,
Maxime.
Quoted 6 lines
Your patch is therefore the correct one!
Cheers,
Richard
Quoted 123 lines · Maxime Roussin-Bélanger
Quoted 5 lines · Richard Purdie
Quoted 2 lines · Richard Purdie via lists.openembedded.org
Hi,
Quoted 7 lines
Quoted 7 lines · Jean-Francois Dagenais
Quoted 6 lines
While investigating constant mismatches of hashes from our
SSTATE_MIRRORdepot,
Maxime and I came across a bitbake-diffsigs which showed a suspicious
cause for
the mismatch. It appeared to be an underlying taskdeptaskswhich
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
contains the
full path of the recipe, which is totally a local build thing.After searching for a little while. We came accross the
clean_basepath(a)
function insiggen.py. We added prints on it:def
clean_basepath(a):mc = None print('a: ' + a) if a.startswith("mc:"): _, mc, a = a.split(":", 2) b = a.rsplit("/", 2)[1] + '/' + a.rsplit("/", 2)[2]print('b: '
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
+
b) ifa.startswith("virtual:"): b = b + ":" + a.rsplit(":", 1)[0] if mc: b = b + ":mc:" + mc print('b+: ' + b) return bSure enough, this looks like the culprit. Consider the
following
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
trace:
a: virtual:native:/full/path/to/poky/meta/recipes-devtools/gnu- config/gnu-config_git.bb:do_populate_sysroot
b: gnu-config/gnu-config_git.bb:do_populate_sysroot b+: gnu-config/gnu-config_git.bb:do_populate_sysroot:virtual:native:/full/path/to/poky
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
/m
eta/recipes-devtools/gnu-config/gnu-config_git.bbLooking at this, it really seems like the intent of the
if a.startswith("virtual:")case is to insert only the:virtual:nativeat the
end.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
The simplest fix we suggest is simply changing
a.rsplit(":", 1)[0]
toa.rsplit(":", 2)[0]and causes b+ to become:b+: gnu-config/gnu-
config_git.bb:do_populate_sysroot:virtual:native
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Since this is so profound and core to the way signatures work, coupled with the
fact that this code hasn't changed in years, we feel being theones
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
to find and
fix this is unlikely.Perpahps the content/format of the passed
avariable haschanged
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
recently?
The following email is a patch which could be applied if deemed correct. But please
consider this re-written version:def
clean_basepath(basepath):# basepath is a string like '/full/path/to/recipes- whatever/helloworld/helloworld_0.1.2.bb:do_compile' # or 'virtual:native:/full/path/to/recipes- whatever/helloworld/helloworld_0.1.2.bb:do_compile' # or 'mc:somemachine:/full/path/to/recipes- whatever/helloworld/helloworld_0.1.2.bb:do_compile'Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
_,
parent_dir,recipe_task=basepath.rsplit("/", 2)# drop what is left of the second slash, searching fromright
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
to
left
cleaned =parent_dir+ '/' +recipe_task# if there was a 'abc:xyz' prefix, add it to the end if not
basepath.startswith('/'):
cleaned += ':' +basepath.split('/', 1)[0].rstrip(':')return cleaned
Both simple fix and this rewrite were tested with `oe-selftest
-r
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
sstatetests`.
We feel this version has these advantages:
* improved readablility
* improved efficiency (the split is not repeated twice andfunction
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
it has fewer
ifs)
* honours the open/close principle better by not hardcoding "virtual:" nor "mc:"Thanks for your feedback! I will submit this version as a patch
if
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
it
is positive.Well done finding this, the bug has clearly been there a while!
I've
Quoted 1 line
Quoted 1 line
had a feeling there was something going on somewhere but never
found
Quoted 3 lines
Quoted 3 lines
it.
I've taken the opportunity to make the code slightly clearer so
have
Quoted 1 line
Quoted 1 line
written and sent a modified version of your patch. I hope that is
ok!
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Please take a look and let me know if I've missed anything too.
Of course as soon as I send that I realised my patch is wrong since virtuals can have multiple entries, which is why this is written searching backwards.
Is there a way to add test cases for this? We would like to be able to write tests for this.
Yes, some simple unit tests added to lib/bb/tests/ somewhere would be run by bitbake-selftest, I'd love to see those!
Cheers,
Richard