Skip to contentSkip to conversation

package.bbclass: Fix base directory for debugsource files when using externalsrc

PATCH In review Alejandro Enedino Hernandez Samaniego 3 participants 5 messages Jul 1, 2022 – Jul 11, 2022

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 package.bbclass: Fix base directory for debugsource files when using externalsrc

While executing do_package, bitbake checks for a list of debug source files and uses a pattern to match the ones to be included in copydebugsources.

Previously when externalsrc was in use either directly or by using devtool, the source location changed and this pattern no longer matched, hence debug source files failed to be included in the corresponding package.

Check when the source directory isnt the default (based on WORKDIR), and change the pattern used to match debug source files if that is the case, allowing us to perform do_package properly.

Workaround debugsource.list containing paths from the host by moving debug source files away from the host directory structure to avoid host contamination (this seems to happen when packages use $TMPDIR/work-shared and externalsrc is in use).

Test matrix included using:
- devtool to use externalsrc automatically
- externalsrc with a non-devtool based source directory
- No externalsrc at all
Tested the following packages to be working:
- glibc ($TMPDIR/work-shared based)
- libxcrypt ($TMPDIR/work based)

[YOCTO 8015]

---
 meta/classes/package.bbclass | 31 ++++++++++++++++++++++++++++---
 1 file changed, 28 insertions(+), 3 deletions(-)
meta/classes/package.bbclass +28 -3
index 62050a18b8..4850134022 100644
@@ -553,13 +553,25 @@ def copydebugsources(debugsrcdir, sources, d):
         strip = d.getVar("STRIP")
         objcopy = d.getVar("OBJCOPY")
         workdir = d.getVar("WORKDIR")
+        sdir = d.getVar("S")
+        sparentdir = os.path.dirname(os.path.dirname(sdir))
+        sbasedir = os.path.basename(os.path.dirname(sdir)) + "/" + os.path.basename(sdir)
         workparentdir = os.path.dirname(os.path.dirname(workdir))
         workbasedir = os.path.basename(os.path.dirname(workdir)) + "/" + os.path.basename(workdir)
 
+        # If S isnt based on WORKDIR we can infer our sources are located elsewhere,
+        # e.g. using externalsrc; use S as base for our dirs
+        if workdir in sdir:
+            basedir = workbasedir
+            parentdir = workparentdir
+        else:
+            basedir = sbasedir
+            parentdir = sparentdir
+
         # If build path exists in sourcefile, it means toolchain did not use
         # -fdebug-prefix-map to compile
         if checkbuildpath(sourcefile, d):
-            localsrc_prefix = workparentdir + "/"
+            localsrc_prefix = parentdir + "/"
         else:
             localsrc_prefix = "/usr/src/debug/"
 
@@ -581,7 +593,7 @@ def copydebugsources(debugsrcdir, sources, d):
         processdebugsrc += "sed 's#%s##g' | "
         processdebugsrc += "(cd '%s' ; cpio -pd0mlL --no-preserve-owner '%s%s' 2>/dev/null)"
 
-        cmd = processdebugsrc % (sourcefile, workbasedir, localsrc_prefix, workparentdir, dvar, debugsrcdir)
+        cmd = processdebugsrc % (sourcefile, basedir, localsrc_prefix, parentdir, dvar, debugsrcdir)
         try:
             subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
         except subprocess.CalledProcessError:
@@ -591,9 +603,22 @@ def copydebugsources(debugsrcdir, sources, d):
         # cpio seems to have a bug with -lL together and symbolic links are just copied, not dereferenced.
         # Work around this by manually finding and copying any symbolic links that made it through.
         cmd = "find %s%s -type l -print0 -delete | sed s#%s%s/##g | (cd '%s' ; cpio -pd0mL --no-preserve-owner '%s%s')" % \
-                (dvar, debugsrcdir, dvar, debugsrcdir, workparentdir, dvar, debugsrcdir)
+                (dvar, debugsrcdir, dvar, debugsrcdir, parentdir, dvar, debugsrcdir)
         subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
 
+
+        # debugsources.list may be polluted from the host if we used externalsrc,
+        # cpio uses copy-pass and may have just created a directory structure
+        # matching the one from the host, if thats the case move those files to
+        # debugsrcdir to avoid host contamination.
+        # Empty dir structure will be deleted in the next step.
+
+        # Same check as above for externalsrc
+        if workdir not in sdir:
+            if os.path.exists(dvar + debugsrcdir + sdir):
+                cmd = "mv %s%s%s/* %s%s" % (dvar, debugsrcdir, sdir, dvar,debugsrcdir)
+                subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
+
         # The copy by cpio may have resulted in some empty directories!  Remove these
         cmd = "find %s%s -empty -type d -delete" % (dvar, debugsrcdir)
         subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
Signature · 1 line
2.25.1
#1 lore raw

Hi Alejandro,

Alejandro Hernandez Samaniego <[email protected]> escreveu no dia sexta, 1/07/2022 à(s) 03:09:

Quoted 46 lines

While executing do_package, bitbake checks for a list of debug source files and uses a pattern to match the ones to be included in copydebugsources.

Previously when externalsrc was in use either directly or by using devtool, the source location changed and this pattern no longer matched, hence debug source files failed to be included in the corresponding package.

Check when the source directory isnt the default (based on WORKDIR), and change the pattern used to match debug source files if that is the case, allowing us to perform do_package properly.

Workaround debugsource.list containing paths from the host by moving debug source files away from the host directory structure to avoid host contamination (this seems to happen when packages use $TMPDIR/work-shared and externalsrc is in use).

Test matrix included using:
- devtool to use externalsrc automatically
- externalsrc with a non-devtool based source directory
- No externalsrc at all
Tested the following packages to be working:
- glibc ($TMPDIR/work-shared based)
- libxcrypt ($TMPDIR/work based)

[YOCTO 8015]

Signed-off-by: Alejandro Enedino Hernandez Samaniego < [email protected]>

---
 meta/classes/package.bbclass | 31 ++++++++++++++++++++++++++++---
 1 file changed, 28 insertions(+), 3 deletions(-)
meta/classes/package.bbclass +2 -0
index 62050a18b8..4850134022 100644
@@ -553,13 +553,25 @@ def copydebugsources(debugsrcdir, sources, d):
         strip = d.getVar("STRIP")
         objcopy = d.getVar("OBJCOPY")
         workdir = d.getVar("WORKDIR")
+        sdir = d.getVar("S")
+        sparentdir = os.path.dirname(os.path.dirname(sdir))

meta/classes/package.bbclass def copydebugsources(debugsrcdir, sources, d): · line 567

Quoted 10 lines
meta/classes/package.bbclass +3 -0
+        sbasedir = os.path.basename(os.path.dirname(sdir)) + "/" +
 os.path.basename(sdir)
         workparentdir = os.path.dirname(os.path.dirname(workdir))
         workbasedir = os.path.basename(os.path.dirname(workdir)) + "/" +
 os.path.basename(workdir)
 
+        # If S isnt based on WORKDIR we can infer our sources are located
 elsewhere,
+        # e.g. using externalsrc; use S as base for our dirs
 

This doesn't seem very safe imo, if the idea is to check if the externalsrc in use it is better using:

if bb.data.inherits_class('externalsrc', d):

Jose

+ if workdir in sdir:

Quoted 78 lines
meta/classes/package.bbclass +6 -0
+            basedir = workbasedir
+            parentdir = workparentdir
+        else:
+            basedir = sbasedir
+            parentdir = sparentdir
+
         # If build path exists in sourcefile, it means toolchain did not

use
# -fdebug-prefix-map to compile
if checkbuildpath(sourcefile, d):

meta/classes/package.bbclass +1 -1
-            localsrc_prefix = workparentdir + "/"
+            localsrc_prefix = parentdir + "/"
         else:
             localsrc_prefix = "/usr/src/debug/"
@@ -581,7 +593,7 @@ def copydebugsources(debugsrcdir, sources, d):
         processdebugsrc += "sed 's#%s##g' | "
         processdebugsrc += "(cd '%s' ; cpio -pd0mlL --no-preserve-owner

'%s%s' 2>/dev/null)"

- cmd = processdebugsrc % (sourcefile, workbasedir,
localsrc_prefix, workparentdir, dvar, debugsrcdir)
+ cmd = processdebugsrc % (sourcefile, basedir, localsrc_prefix,
parentdir, dvar, debugsrcdir)
try:
subprocess.check_output(cmd, shell=True,
stderr=subprocess.STDOUT)
except subprocess.CalledProcessError:

meta/classes/package.bbclass +0 -0
@@ -591,9 +603,22 @@ def copydebugsources(debugsrcdir, sources, d):
         # cpio seems to have a bug with -lL together and symbolic links

are just copied, not dereferenced.
# Work around this by manually finding and copying any symbolic
links that made it through.

         cmd = "find %s%s -type l -print0 -delete | sed s#%s%s/##g | (cd

'%s' ; cpio -pd0mL --no-preserve-owner '%s%s')" % \
- (dvar, debugsrcdir, dvar, debugsrcdir, workparentdir,
dvar, debugsrcdir)
+ (dvar, debugsrcdir, dvar, debugsrcdir, parentdir, dvar,
debugsrcdir)
subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)

meta/classes/package.bbclass +2 -0
+
+        # debugsources.list may be polluted from the host if we used

externalsrc,
+ # cpio uses copy-pass and may have just created a directory
structure
+ # matching the one from the host, if thats the case move those
files to

meta/classes/package.bbclass +7 -0
+        # debugsrcdir to avoid host contamination.
+        # Empty dir structure will be deleted in the next step.
+
+        # Same check as above for externalsrc
+        if workdir not in sdir:
+            if os.path.exists(dvar + debugsrcdir + sdir):
+                cmd = "mv %s%s%s/* %s%s" % (dvar, debugsrcdir, sdir,

dvar,debugsrcdir)
+ subprocess.check_output(cmd, shell=True,
stderr=subprocess.STDOUT)
+
# The copy by cpio may have resulted in some empty directories!
Remove these
cmd = "find %s%s -empty -type d -delete" % (dvar, debugsrcdir) subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
--
2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#167448):
https://lists.openembedded.org/g/openembedded-core/message/167448
Mute This Topic: https://lists.openembedded.org/mt/92100558/5052612
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 3 lines
Best regards,

José Quaresma
#2 lore raw
Quoted 52 lines · Jose Quaresma

Hi Alejandro,

Alejandro Hernandez Samaniego <[email protected]> escreveu no dia sexta, 1/07/2022 à(s) 03:09:

Quoted 49 lines

While executing do_package, bitbake checks for a list of debug source files and uses a pattern to match the ones to be included in copydebugsources.

Previously when externalsrc was in use either directly or by using devtool, the source location changed and this pattern no longer matched, hence debug source files failed to be included in the corresponding package.

Check when the source directory isnt the default (based on WORKDIR), and change the pattern used to match debug source files if that is the case, allowing us to perform do_package properly.

Workaround debugsource.list containing paths from the host by moving debug source files away from the host directory structure  to avoid host contamination (this seems to happen when packages use $TMPDIR/work-shared and externalsrc is in use).

Test matrix included using:
- devtool to use externalsrc automatically
- externalsrc with a non-devtool based source directory
- No externalsrc at all
Tested the following packages to be working:
- glibc ($TMPDIR/work-shared based)
- libxcrypt ($TMPDIR/work based)

[YOCTO 8015]

---
 meta/classes/package.bbclass | 31 ++++++++++++++++++++++++++++---

 1 file changed, 28 insertions(+), 3 deletions(-)
meta/classes/package.bbclass +3 -0
index 62050a18b8..4850134022 100644
@@ -553,13 +553,25 @@ def copydebugsources(debugsrcdir, sources, d):
          strip = d.getVar("STRIP")
          objcopy = d.getVar("OBJCOPY")
          workdir = d.getVar("WORKDIR")
+        sdir = d.getVar("S")
+        sparentdir = os.path.dirname(os.path.dirname(sdir))
+        sbasedir = os.path.basename(os.path.dirname(sdir)) + "/" + os.path.basename(sdir)
          workparentdir = os.path.dirname(os.path.dirname(workdir))
          workbasedir = os.path.basename(os.path.dirname(workdir)) + "/" + os.path.basename(workdir)
 
Quoted 8 lines

meta/classes/package.bbclass def copydebugsources(debugsrcdir, sources, d): · line 577

Quoted 3 lines
meta/classes/package.bbclass +2 -0
+        # If S isnt based on WORKDIR we can infer our sources are located elsewhere,
+        # e.g. using externalsrc; use S as base for our dirs
 

This doesn't seem very safe imo, if the idea is to check if the externalsrc in use it is better using:

if bb.data.inherits_class('externalsrc', d):

The code is generic enough that if the source is outside WORKDIR, it should work so I'm not sure it makes sense to gate this on externalsrc only though...

Cheers,

Richard

#3 lore raw

Richard Purdie <[email protected]> escreveu no dia sexta, 1/07/2022 à(s) 12:19:

Quoted 64 lines · Jose Quaresma
Quoted 3 lines

Hi Alejandro,

Alejandro Hernandez Samaniego <[email protected]> escreveu no

dia sexta, 1/07/2022 à(s) 03:09:

Quoted 28 lines
Quoted 28 lines

While executing do_package, bitbake checks for a list of debug source files and uses a pattern to match the ones to be included in copydebugsources.

Previously when externalsrc was in use either directly or by using devtool, the source location changed and this pattern no longer matched, hence debug source files failed to be included in the corresponding package.

Check when the source directory isnt the default (based on WORKDIR), and change the pattern used to match debug source files if that is the case, allowing us to perform do_package properly.

Workaround debugsource.list containing paths from the host by moving debug source files away from the host directory structure to avoid host contamination (this seems to happen when packages use $TMPDIR/work-shared and externalsrc is in use).

Test matrix included using:
- devtool to use externalsrc automatically
- externalsrc with a non-devtool based source directory
- No externalsrc at all
Tested the following packages to be working:
- glibc ($TMPDIR/work-shared based)
- libxcrypt ($TMPDIR/work based)

Quoted 3 lines
Quoted 3 lines

[YOCTO 8015]

Signed-off-by Alejandro Enedino Hernandez Samaniego <

[email protected]>

Quoted 5 lines

Review

Quoted 5 lines
---
 meta/classes/package.bbclass | 31 ++++++++++++++++++++++++++++---
 1 file changed, 28 insertions(+), 3 deletions(-)
(quoted diff) +0 -0

b/meta/classes/package.bbclass

Quoted 10 lines

meta/classes/package.bbclass def copydebugsources(debugsrcdir, sources, d): · line 558

Quoted 10 lines

index 62050a18b8..4850134022 100644

meta/classes/package.bbclass +3 -0
@@ -553,13 +553,25 @@ def copydebugsources(debugsrcdir, sources, d):
         strip = d.getVar("STRIP")
         objcopy = d.getVar("OBJCOPY")
         workdir = d.getVar("WORKDIR")
+        sdir = d.getVar("S")
+        sparentdir = os.path.dirname(os.path.dirname(sdir))
+        sbasedir = os.path.basename(os.path.dirname(sdir)) + "/" +

os.path.basename(sdir)

Quoted 2 lines

meta/classes/package.bbclass def copydebugsources(debugsrcdir, sources, d): · line 560

Quoted 2 lines
meta/classes/package.bbclass +0 -0
         workparentdir = os.path.dirname(os.path.dirname(workdir))
         workbasedir = os.path.basename(os.path.dirname(workdir)) +

"/" + os.path.basename(workdir)

Quoted 2 lines

meta/classes/package.bbclass def copydebugsources(debugsrcdir, sources, d): · line 561

Quoted 2 lines
meta/classes/package.bbclass +1 -0
+        # If S isnt based on WORKDIR we can infer our sources are

located elsewhere,

Quoted 4 lines
Quoted 2 lines
meta/classes/package.bbclass +1 -0
+        # e.g. using externalsrc; use S as base for our dirs
 
Quoted 9 lines
Quoted 1 line

This doesn't seem very safe imo, if the idea is to check if the

externalsrc in use it is better using:

Quoted 2 lines

if bb.data.inherits_class('externalsrc', d):

The code is generic enough that if the source is outside WORKDIR, it should work so I'm not sure it makes sense to gate this on externalsrc only though...

My comment is mainly because I see the externalsrc everywhere in this patch. But as said to make it generic it's better as it is.

Jose

Quoted 6 lines

Cheers,

Richard

Signature · 3 lines
Best regards,

José Quaresma
#4 lore raw
In reply to Alejandro Enedino Hernandez Samaniego
Quoted 89 lines · Alejandro Hernandez Samaniego

While executing do_package, bitbake checks for a list of debug source files and uses a pattern to match the ones to be included in copydebugsources.

Previously when externalsrc was in use either directly or by using devtool, the source location changed and this pattern no longer matched, hence debug source files failed to be included in the corresponding package.

Check when the source directory isnt the default (based on WORKDIR), and change the pattern used to match debug source files if that is the case, allowing us to perform do_package properly.

Workaround debugsource.list containing paths from the host by moving debug source files away from the host directory structure to avoid host contamination (this seems to happen when packages use $TMPDIR/work-shared and externalsrc is in use).

Test matrix included using:
- devtool to use externalsrc automatically
- externalsrc with a non-devtool based source directory
- No externalsrc at all
Tested the following packages to be working:
- glibc ($TMPDIR/work-shared based)
- libxcrypt ($TMPDIR/work based)

[YOCTO 8015]

---
 meta/classes/package.bbclass | 31 ++++++++++++++++++++++++++++---
 1 file changed, 28 insertions(+), 3 deletions(-)
meta/classes/package.bbclass +21 -3
index 62050a18b8..4850134022 100644
@@ -553,13 +553,25 @@ def copydebugsources(debugsrcdir, sources, d):
         strip = d.getVar("STRIP")
         objcopy = d.getVar("OBJCOPY")
         workdir = d.getVar("WORKDIR")
+        sdir = d.getVar("S")
+        sparentdir = os.path.dirname(os.path.dirname(sdir))
+        sbasedir = os.path.basename(os.path.dirname(sdir)) + "/" + os.path.basename(sdir)
         workparentdir = os.path.dirname(os.path.dirname(workdir))
         workbasedir = os.path.basename(os.path.dirname(workdir)) + "/" + os.path.basename(workdir)
 
+        # If S isnt based on WORKDIR we can infer our sources are located elsewhere,
+        # e.g. using externalsrc; use S as base for our dirs
+        if workdir in sdir:
+            basedir = workbasedir
+            parentdir = workparentdir
+        else:
+            basedir = sbasedir
+            parentdir = sparentdir
+
         # If build path exists in sourcefile, it means toolchain did not use
         # -fdebug-prefix-map to compile
         if checkbuildpath(sourcefile, d):
-            localsrc_prefix = workparentdir + "/"
+            localsrc_prefix = parentdir + "/"
         else:
             localsrc_prefix = "/usr/src/debug/"
 
@@ -581,7 +593,7 @@ def copydebugsources(debugsrcdir, sources, d):
         processdebugsrc += "sed 's#%s##g' | "
         processdebugsrc += "(cd '%s' ; cpio -pd0mlL --no-preserve-owner '%s%s' 2>/dev/null)"
 
-        cmd = processdebugsrc % (sourcefile, workbasedir, localsrc_prefix, workparentdir, dvar, debugsrcdir)
+        cmd = processdebugsrc % (sourcefile, basedir, localsrc_prefix, parentdir, dvar, debugsrcdir)
         try:
             subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
         except subprocess.CalledProcessError:
@@ -591,9 +603,22 @@ def copydebugsources(debugsrcdir, sources, d):
         # cpio seems to have a bug with -lL together and symbolic links are just copied, not dereferenced.
         # Work around this by manually finding and copying any symbolic links that made it through.
         cmd = "find %s%s -type l -print0 -delete | sed s#%s%s/##g | (cd '%s' ; cpio -pd0mL --no-preserve-owner '%s%s')" % \
-                (dvar, debugsrcdir, dvar, debugsrcdir, workparentdir, dvar, debugsrcdir)
+                (dvar, debugsrcdir, dvar, debugsrcdir, parentdir, dvar, debugsrcdir)
         subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
 
+
+        # debugsources.list may be polluted from the host if we used externalsrc,
+        # cpio uses copy-pass and may have just created a directory structure
+        # matching the one from the host, if thats the case move those files to
+        # debugsrcdir to avoid host contamination.
+        # Empty dir structure will be deleted in the next step.

meta/classes/package.bbclass def copydebugsources(debugsrcdir, sources, d): · line 624

Quoted 10 lines
meta/classes/package.bbclass +7 -0
+
+        # Same check as above for externalsrc
+        if workdir not in sdir:
+            if os.path.exists(dvar + debugsrcdir + sdir):
+                cmd = "mv %s%s%s/* %s%s" % (dvar, debugsrcdir, sdir, dvar,debugsrcdir)
+                subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
+
         # The copy by cpio may have resulted in some empty directories!  Remove these
         cmd = "find %s%s -empty -type d -delete" % (dvar, debugsrcdir)
         subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)

This has merged but unfortunately it introduces some kind of regression. If you build a kernel after this is applied, the kernel source in /usr/src/ for debug information is no longer present.

This is why I was seeing build failures for the buildpaths QA test addition whilst Bruce was not since I didn't have this applied in that particular build.

We need to find out what broke here and how to fix it.

Clearly we need some better QA tests as well to detect this.

Cheers,

Richard