externalsrc.bbclass: dynamical change of DEBUG_PREFIX_MAP
PATCH 1/2 externalsrc.bbclass: dynamical change of DEBUG_PREFIX_MAP
When using external source manager (devtool), the debug symbol generated by the compilation doesn't point to the right directory. This is normally handled by gcc options that are defined
in DEBUG_PREFIX_MAP in conf/bitbake.conf.
But the path in it are hardcoded and point to WORKDIR which is not overloaded by devtool.
This patch takes the parent directory of external source directory and prepend correct path to DEBUG_PREFIX_MAP.
Moreover, to avoid wrong path resolution during dwarfsrcfiles step in splitdebuginfo, it make B variable point to the same structure as EXTERNALSRC (if EXTERNALSRC_BUILD is not defined).
--- meta/classes/externalsrc.bbclass | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)
meta/classes/externalsrc.bbclass +13 -1
@@ -53,14 +53,26 @@ python () { d.setVar('BB_DONT_CACHE', '1') if externalsrc: + import os import oe.recipeutils import oe.path + pn = d.getVar('PN') + # taken from bitbake.conf + debugsrcdir = "/usr/src/debug/%s/%s%s-%s" % (pn, d.getVar('EXTENDPE'), d.getVar('PV'), d.getVar('PR')) + externalsrc_parentdir = os.path.dirname(externalsrc) + + debug_prefix_map_ext = "-fmacro-prefix-map=%s=%s " % (externalsrc_parentdir, debugsrcdir) + macro_prefix_map_ext = "-fdebug-prefix-map=%s=%s " % (externalsrc_parentdir, debugsrcdir) + d.prependVar('DEBUG_PREFIX_MAP', debug_prefix_map_ext) + d.prependVar('DEBUG_PREFIX_MAP', macro_prefix_map_ext) + d.setVar('S', externalsrc) if externalsrcbuild: d.setVar('B', externalsrcbuild) else: - d.setVar('B', '${WORKDIR}/${BPN}-${PV}/') + builddir = os.path.join(externalsrc_parentdir, "%s-build" % pn) + d.setVar('B', builddir) local_srcuri = [] fetch = bb.fetch2.Fetch((d.getVar('SRC_URI') or '').split(), d)
Signature · 1 line
2.25.1
PATCH 2/2 package.bbclass: manage external sources for debug source file packaging
--- meta/classes/package.bbclass | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
meta/classes/package.bbclass +8 -1
@@ -560,13 +560,20 @@ def copydebugsources(debugsrcdir, sources, d): strip = d.getVar("STRIP") objcopy = d.getVar("OBJCOPY") workdir = d.getVar("WORKDIR") - workparentdir = os.path.dirname(os.path.dirname(workdir)) + externalsrc = d.getVar('EXTERNALSRC') + if externalsrc: + workparentdir = os.path.dirname(externalsrc) + else: + workparentdir = os.path.dirname(os.path.dirname(workdir)) + workbasedir = os.path.basename(os.path.dirname(workdir)) + "/" + os.path.basename(workdir) # If build path exists in sourcefile, it means toolchain did not use # -fdebug-prefix-map to compile if checkbuildpath(sourcefile, d): localsrc_prefix = workparentdir + "/" + elif externalsrc: + localsrc_prefix = os.path.join("/usr/src/debug/", workbasedir) + "/" else: localsrc_prefix = "/usr/src/debug/"
Signature · 1 line
2.25.1
Re: [PATCH 2/2] package.bbclass: manage external sources for debug source file packaging
Hello, I would like to add a link to a yocto mail where I expose my problematic that led to this patch series:
https://lists.yoctoproject.org/g/yocto/topic/83622035?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,83622035
After further research, I think these patch can be simplified by modified only WORKDIR variable to be os.path.dirname(EXTERNALSRC) but I fear of consequence to have WORKDIR outside of TMPDIR so I didn't take this path.
Quoted 38 lines · Frederic Martinsons
Signed-off-by Frederic Martinsons <[email protected]>--- meta/classes/package.bbclass | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)meta/classes/package.bbclass +6
-1@@ -560,13 +560,20 @@ def copydebugsources(debugsrcdir, sources, d): strip = d.getVar("STRIP") objcopy = d.getVar("OBJCOPY") workdir = d.getVar("WORKDIR") - workparentdir = os.path.dirname(os.path.dirname(workdir)) + externalsrc = d.getVar('EXTERNALSRC') + if externalsrc: + workparentdir = os.path.dirname(externalsrc) + else: + workparentdir = os.path.dirname(os.path.dirname(workdir)) + workbasedir = os.path.basename(os.path.dirname(workdir)) + "/" + os.path.basename(workdir) # If build path exists in sourcefile, it means toolchain did notuse
# -fdebug-prefix-map to compile ifcheckbuildpath(sourcefile, d):
localsrc_prefix= workparentdir + "/"meta/classes/package.bbclass +2
-0+ elif externalsrc: + localsrc_prefix = os.path.join("/usr/src/debug/",workbasedir) + "/"
else:
localsrc_prefix= "/usr/src/debug/"--
2.25.1
Re: [OE-core] [PATCH 2/2] package.bbclass: manage external sources for debug source file packaging
Quoted 7 lines · Frederic Martinsons
Hello, I would like to add a link to a yocto mail where I expose my problematic that led to this patch series:
https://lists.yoctoproject.org/g/yocto/topic/83622035?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,83622035After further research, I think these patch can be simplified by modified only WORKDIR variable to be
os.path.dirname(EXTERNALSRC)but I fear of consequence to have WORKDIR outside of TMPDIR so I didn't take this path.
Changing WORKDIR would no doubt solve your immediate problem but would create a ton of others :(.
The issue is that the class wants to re-declare S but some of our code makes assumptions about the location of WORKDIR with regard to S (and B). There is more inside WORKDIR than just S and EXTERNALSRC does really correspond to S, not WORKDIR...
Cheers,
Richard
Re: [OE-core] [PATCH 2/2] package.bbclass: manage external sources for debug source file packaging
Hello Richard,
First, sorry for the commit message (I gave details in the first patch on externalsrc.bbclass but not on this one). I agree with you that it seems an hack but I don't know to make package;bbclass find the sources correctly. The "culprit" is the following command (
https://github.com/openembedded/openembedded-core/blob/master/meta/classes/package.bbclass#L591)
and especially the last part of it:
processdebugsrc += "(cd '%s' ; cpio -pd0mlL --no-preserve-owner '%s%s'
Quoted 1 line
2>/dev/null)"
this command explicitely goes into *workparentdir *and wait for the sources to be in it thanks to the input list that was calculated as the output of dwarfsrcfiles earlier call (with *localsrc_prefix *remove from them).
Have you an idea (or even a lead that I can explore) on how make things correct here ?
Quoted 31 lines · Richard Purdie
Quoted 1 line · Frederic Martinsons
Hello, I would like to add a link to a yocto mail where I expose my
problematic that led
Quoted 2 lines
to this patch series:
Quoted 2 lines
After further research, I think these patch can be simplified by
modified only
Quoted 1 line
WORKDIR variable to be
os.path.dirname(EXTERNALSRC)but I fear ofconsequence
Quoted 1 line
to have WORKDIR outside of TMPDIR so I didn't take this path.
Changing WORKDIR would no doubt solve your immediate problem but would create
a ton of others :(.The issue is that the class wants to re-declare S but some of our code makes
assumptions about the location of WORKDIR with regard to S (and B). There is more inside WORKDIR than just S and EXTERNALSRC does really correspond to S,
not WORKDIR...Cheers,
Richard
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [OE-core] [PATCH 2/2] package.bbclass: manage external sources for debug source file packaging
Hello again,
Since my patches doesn't meet the requirements and considering that you acknowledge the issue (please correct if I mistaken), should I open a bug?
Of course I'll give as many details as I can in this report.
Thanks.
Quoted 58 lines · Frederic Martinsons via lists.openembedded.org
Hello Richard,
First, sorry for the commit message (I gave details in the first patch on externalsrc.bbclass but not on this one). I agree with you that it seems an hack but I don't know to make package;bbclass find the sources correctly. The "culprit" is the following command (
https://github.com/openembedded/openembedded-core/blob/master/meta/classes/package.bbclass#L591)
and especially the last part of it:processdebugsrc += "(cd '%s' ; cpio -pd0mlL --no-preserve-owner '%s%s'
Quoted 1 line
2>/dev/null)"
this command explicitely goes into *workparentdir *and wait for the sources to be in it thanks to the input list that was calculated as the output of dwarfsrcfiles earlier call (with
*localsrc_prefix*remove from them).Have you an idea (or even a lead that I can explore) on how make things correct here ?
Quoted 31 lines · Richard Purdie
Quoted 1 line · Frederic Martinsons
Hello, I would like to add a link to a yocto mail where I expose my
problematic that led
Quoted 2 lines
to this patch series:
Quoted 2 lines
After further research, I think these patch can be simplified by
modified only
Quoted 1 line
WORKDIR variable to be
os.path.dirname(EXTERNALSRC)but I fear ofconsequence
Quoted 1 line
to have WORKDIR outside of TMPDIR so I didn't take this path.
Changing WORKDIR would no doubt solve your immediate problem but would create
a ton of others :(.The issue is that the class wants to re-declare S but some of our code makes
assumptions about the location of WORKDIR with regard to S (and B). There is more inside WORKDIR than just S and EXTERNALSRC does really correspond to S,
not WORKDIR...Cheers,
Richard
Re: [OE-core] [PATCH 2/2] package.bbclass: manage external sources for debug source file packaging
In reply to Frederic MartinsonsQuoted 29 lines · Frederic Martinsons
Signed-off-by Frederic Martinsons <[email protected]>---
meta/classes/package.bbclass | 9 ++++++++-1 file changed, 8 insertions(+), 1 deletion(-)meta/classes/package.bbclass +6
-1@@ -560,13 +560,20 @@ def copydebugsources(debugsrcdir, sources, d): strip = d.getVar("STRIP") objcopy = d.getVar("OBJCOPY") workdir = d.getVar("WORKDIR") - workparentdir = os.path.dirname(os.path.dirname(workdir)) + externalsrc = d.getVar('EXTERNALSRC') + if externalsrc: + workparentdir = os.path.dirname(externalsrc) + else: + workparentdir = os.path.dirname(os.path.dirname(workdir)) + workbasedir = os.path.basename(os.path.dirname(workdir)) + "/" + os.path.basename(workdir)# If build path exists in sourcefile, it means toolchain did not use # -fdebug-prefix-map to compile
ifcheckbuildpath(sourcefile, d):
localsrc_prefix = workparentdir + "/"meta/classes/package.bbclass +2
-0+ elif externalsrc: + localsrc_prefix = os.path.join("/usr/src/debug/", workbasedir) + "/"
Quoted 3 lines
else:
localsrc_prefix = "/usr/src/debug/"
Having to change package.bbbclass to work around whatever externalsrc is doing feels rather wrong. This looks/feels like we're adding hacks on top of hacks and is going to end up with an unmaintainable mess. I'd note there are also no test cases monitoring whether this is working or regressing.
The patch doesn't have a commit message and explaining anything about the problem being solved either. I don't doubt there is one but I don't think this patch is right or ready to go in...
Cheers,
Richard
Quoted 51 lines · Frederic Martinsons
When using external source manager (devtool), the debug symbol generated by the compilation doesn't point to the right directory. This is normally handled by gcc options that are defined
inDEBUG_PREFIX_MAPinconf/bitbake.conf.But the path in it are hardcoded and point to WORKDIR which is not overloaded by devtool.
This patch takes the parent directory of external source directory and prepend correct path to
DEBUG_PREFIX_MAP.Moreover, to avoid wrong path resolution during dwarfsrcfiles step in splitdebuginfo, it make B variable point to the same structure as EXTERNALSRC (if
EXTERNALSRC_BUILDis not defined).Signed-off-by Frederic Martinsons <[email protected]>---
meta/classes/externalsrc.bbclass | 14 +++++++++++++-1 file changed, 13 insertions(+), 1 deletion(-)meta/classes/externalsrc.bbclass +0
-0@@ -53,14 +53,26 @@ python () { d.setVar('BB_DONT_CACHE', '1')if externalsrc:
+ import os
import oe.recipeutils
import oe.pathmeta/classes/externalsrc.bbclass +10
-0+ pn = d.getVar('PN') + # taken from bitbake.conf + debugsrcdir = "/usr/src/debug/%s/%s%s-%s" % (pn, d.getVar('EXTENDPE'), d.getVar('PV'), d.getVar('PR')) + externalsrc_parentdir = os.path.dirname(externalsrc) + + debug_prefix_map_ext = "-fmacro-prefix-map=%s=%s " % (externalsrc_parentdir, debugsrcdir) + macro_prefix_map_ext = "-fdebug-prefix-map=%s=%s " % (externalsrc_parentdir, debugsrcdir) + d.prependVar('DEBUG_PREFIX_MAP', debug_prefix_map_ext) + d.prependVar('DEBUG_PREFIX_MAP', macro_prefix_map_ext) +d.setVar('S', externalsrc)
if externalsrcbuild:
d.setVar('B', externalsrcbuild)
else:meta/classes/externalsrc.bbclass +2
-1- d.setVar('B', '${WORKDIR}/${BPN}-${PV}/') + builddir = os.path.join(externalsrc_parentdir, "%s-build" % pn) + d.setVar('B', builddir)
Quoted 1 line
I'm ok with adding some prefix mappings however the commit message doesn't really make it clear that you're actually inventing a new build directory here too. We can't just poke around the parent directory of the externalsrc like this since we have no idea what that directory is and we've not been asked to "touch" it.
Cheers,
Richard
Hello Richard,
Thanks for your answer, I didn't think about poking parent directory issue but you're right. What about setting B to ${EXTERNALSRC}/${PN}-build instead ?
The important thing here (from what I understood) is to avoid having too much relative path (that is to say a number of '../' greater than the number of directory required to reach BDIR starting from EXTERNALSRC.
Quoted 74 lines · Richard Purdie
Quoted 15 lines · Frederic Martinsons
When using external source manager (devtool), the debug symbol generated by the compilation doesn't point to the right directory. This is normally handled by gcc options that are defined
inDEBUG_PREFIX_MAPinconf/bitbake.conf.But the path in it are hardcoded and point to WORKDIR which is not overloaded by devtool.
This patch takes the parent directory of external source directory and prepend correct path to
DEBUG_PREFIX_MAP.Moreover, to avoid wrong path resolution during dwarfsrcfiles step in splitdebuginfo, it make B variable point to the same structure as EXTERNALSRC (if
EXTERNALSRC_BUILDis not defined).Review
Quoted 6 lines
Signed-off-by Frederic Martinsons <[email protected]>--- meta/classes/externalsrc.bbclass | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)(quoted diff) +0
-0b/meta/classes/externalsrc.bbclass
Quoted 6 lines
index
3d6b80bee2..de5ef714b4100644meta/classes/externalsrc.bbclass +0
-0@@ -53,14 +53,26 @@ python () { d.setVar('BB_DONT_CACHE', '1')meta/classes/externalsrc.bbclass python () { · line 64
Quoted 10 lines
meta/classes/externalsrc.bbclass +4
-0if externalsrc: + import os import oe.recipeutils import oe.path + pn = d.getVar('PN') + # taken from bitbake.conf + debugsrcdir = "/usr/src/debug/%s/%s%s-%s" % (pn,
d.getVar('EXTENDPE'),d.getVar('PV'),d.getVar('PR'))meta/classes/externalsrc.bbclass python () {
Quoted 3 lines
meta/classes/externalsrc.bbclass +3
-0+ externalsrc_parentdir = os.path.dirname(externalsrc) + + debug_prefix_map_ext = "-fmacro-prefix-map=%s=%s " %(
externalsrc_parentdir, debugsrcdir)meta/classes/externalsrc.bbclass python () { · line 68
Quoted 1 line
meta/classes/externalsrc.bbclass +1
-0+ macro_prefix_map_ext = "-fdebug-prefix-map=%s=%s " %(
externalsrc_parentdir, debugsrcdir)meta/classes/externalsrc.bbclass python () {
Quoted 9 lines
meta/classes/externalsrc.bbclass +4
-1+ d.prependVar('DEBUG_PREFIX_MAP', debug_prefix_map_ext) + d.prependVar('DEBUG_PREFIX_MAP', macro_prefix_map_ext) + d.setVar('S', externalsrc) if externalsrcbuild: d.setVar('B', externalsrcbuild) else: - d.setVar('B', '${WORKDIR}/${BPN}-${PV}/') + builddir = os.path.join(externalsrc_parentdir, "%s-build" %pn)
meta/classes/externalsrc.bbclass python () { · line 76
Quoted 2 lines
meta/classes/externalsrc.bbclass +1
-0+ d.setVar('B', builddir)I'm ok with adding some prefix mappings however the commit message doesn't really
make it clear that you're actually inventing a new build directory here too. We
can't just poke around the parent directory of the externalsrc like this since we
have no idea what that directory is and we've not been asked to "touch" it.Cheers,
Richard