Skip to contentSkip to conversation

package: export debugsources in PKGDESTWORK as json

PATCH Superseded [email protected] 3 participants 7 messages Jun 5, 2025 – Jun 9, 2025

v7 of 8 Superseded Latest is v8 →

lore
Share & tools
lore
#0 lore raw

PATCH v7 1/1 package: export debugsources in PKGDESTWORK as json

From: Daniel Turull <[email protected]>

The source information used during packaging can be use from other tasks to have more detailed information on the files used during the compilation and improve SPDX accuracy.

Source files used during compilation are store as compressed zstd json in pkgdata/debugsources/$PN-debugsources.json.zstd
Format:
{ binary1: [src1, src2, ...], binary2: [src1, src2, ...] }

I checked the sstate size, and it slightly increase using core-image-full-cmdline:
Before: 2454884 B
After: 2456860 B (+1976 B or 0,08%)

---
 meta/conf/bitbake.conf |  2 ++
 meta/lib/oe/package.py | 46 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 48 insertions(+)
meta/conf/bitbake.conf +2 -0
index 13d55923b7..8c0c71d76d 100644
@@ -991,5 +991,7 @@ oe.sstatesig.find_sstate_manifest[vardepsexclude] = "BBEXTENDCURR BBEXTENDVARIAN
 oe.utils.get_multilib_datastore[vardepsexclude] = "DEFAULTTUNE_MULTILIB_ORIGINAL OVERRIDES"
 oe.path.format_display[vardepsexclude] = "TOPDIR"
 oe.utils.get_bb_number_threads[vardepsexclude] = "BB_NUMBER_THREADS"
+oe.package.save_debugsources_info[vardepsexclude] = "BB_NUMBER_THREADS"
+oe.package.read_debugsources_file[vardepsexclude] = "BB_NUMBER_THREADS"
 oe.packagedata.emit_pkgdata[vardepsexclude] = "BB_NUMBER_THREADS"
 oe.packagedata.read_subpkgdata_extended[vardepsexclude] = "BB_NUMBER_THREADS"
meta/lib/oe/package.py +46 -0

58 lines not on this page · raw message

#1 lore raw
Quoted 4 lines · Daniel Turull via lists.openembedded.org

I checked the sstate size, and it slightly increase using core-image-full-cmdline:
Before: 2454884 B
After: 2456860 B (+1976 B or 0,08%)

2000 _bytes_ for adding all of the source information to the core-image-full-cmdline sstate? I know zstd is good but that seems a little low?

Ross

#2 lore raw

Hi, now I noticed that I put the wrong unit when reading du -s. It is Kbytes. So the increase is 1976 KB, not 1976 Bytes

Best regards,
Daniel

Quoted 20 lines

-----Original Message-----
From: Ross Burton <[email protected]>
Sent: Thursday, 5 June 2025 15:06
To: Daniel Turull <[email protected]>
Cc: [email protected]; Richard Purdie <[email protected]>
Subject: Re: [OE-core] [PATCH v7] package: export debugsources in PKGDESTWORK as json

Quoted 4 lines · Daniel Turull via lists.openembedded.org

I checked the sstate size, and it slightly increase using core-image-full-cmdline:
Before: 2454884 B
After: 2456860 B (+1976 B or 0,08%)

2000 _bytes_ for adding all of the source information to the core-image-full- cmdline sstate? I know zstd is good but that seems a little low?

Ross

#3 lore raw

Hi Daniel,

Quoted 3 lines · Daniel Turull

Hi, now I noticed that I put the wrong unit when reading du -s. It is Kbytes. So the increase is 1976 KB, not 1976 Bytes

That makes a lot more sense and is an acceptable cost for that useful data.

Thanks,
Ross

#4 lore raw
In reply to [email protected]
Quoted 74 lines

-----Original Message-----
From: Daniel Turull <[email protected]>
Sent: Thursday, 5 June 2025 14:03
To: [email protected]
Cc: Daniel Turull <[email protected]>; Richard Purdie <[email protected]>
Subject: [PATCH v7] package: export debugsources in PKGDESTWORK as json

From: Daniel Turull <[email protected]>

The source information used during packaging can be use from other tasks to have more detailed information on the files used during the compilation and improve SPDX accuracy.

Source files used during compilation are store as compressed zstd json in pkgdata/debugsources/$PN-debugsources.json.zstd
Format:
{ binary1: [src1, src2, ...], binary2: [src1, src2, ...] }

I checked the sstate size, and it slightly increase using core-image-full-cmdline:
Before: 2454884 B
After: 2456860 B (+1976 B or 0,08%)

---
 meta/conf/bitbake.conf |  2 ++
 meta/lib/oe/package.py | 46
++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 48 insertions(+)
meta/conf/bitbake.conf index renamed from meta/conf/bitbake.conf +0 -0

13d55923b7..8c0c71d76d 100644

meta/conf/bitbake.conf +0 -0
@@ -991,5 +991,7 @@ oe.sstatesig.find_sstate_manifest[vardepsexclude] =

"BBEXTENDCURR BBEXTENDVARIAN oe.utils.get_multilib_datastore[vardepsexclude] = "DEFAULTTUNE_MULTILIB_ORIGINAL OVERRIDES"
oe.path.format_display[vardepsexclude] = "TOPDIR" oe.utils.get_bb_number_threads[vardepsexclude] =
"BB_NUMBER_THREADS" +oe.package.save_debugsources_info[vardepsexclude] = "BB_NUMBER_THREADS" +oe.package.read_debugsources_file[vardepsexclude] = "BB_NUMBER_THREADS"
oe.packagedata.emit_pkgdata[vardepsexclude] = "BB_NUMBER_THREADS" oe.packagedata.read_subpkgdata_extended[vardepsexclude] =
"BB_NUMBER_THREADS"

meta/lib/oe/package.py index renamed from meta/lib/oe/package.py +0 -0

0bcc04ea54..c5acb7d205 100644

meta/lib/oe/package.py +14 -0
@@ -1049,6 +1049,49 @@ def copydebugsources(debugsrcdir, sources, d):
             if os.path.exists(p) and not os.listdir(p):
                 os.rmdir(p)
 
+def save_debugsources_info(debugsrcdir, sources_raw, d):
+    import json
+    import bb.compress.zstd
+    if debugsrcdir and sources_raw:
+        debugsources_file = d.expand("${PKGDESTWORK}/debugsources/${PN}-
 debugsources.json.zstd")
+        debugsources_dir = os.path.dirname(debugsources_file)
+        if not os.path.isdir(debugsources_dir):
+            bb.utils.mkdirhier(debugsources_dir)
+        bb.utils.remove(debugsources_file)
+
+        workdir = d.getVar("WORKDIR")
+        sdir = d.getVar("S")
+        pn = d.getVar('PN')
+        pv = d.getVar('PV')

meta/lib/oe/package.py def copydebugsources(debugsrcdir, sources, d): · line 1076

Quoted 10 lines
meta/lib/oe/package.py +10 -0
+        sources_dict = {}
+        for file, src_files in sources_raw:
+            file_clean = file.replace(f"{workdir}/package/","")
+            sources_clean = [
+                src.replace(f"/usr/src/debug/{pn}/", "")
+                   .replace(f"{sdir}/", "")
+                   .replace(f"/usr/src/kernel/", "")
+                   .replace(f"/usr/src/{pn}/", "")
+                   .replace(f"{pn}-{pv}/", "")
+                   .replace(f"{pv}/", "")

When looking at Joshua's comments, I relalized that I missed to remove "git". I'll send a new version.

Quoted 36 lines
meta/lib/oe/package.py +6 -0
+                for src in src_files
+                if not any(keyword in src for keyword in ("<internal>", "<built-in>"))
 and not src.endswith("/")
+            ]
+            sources_dict[file_clean] = sorted(sources_clean)
+        num_threads = int(d.getVar("BB_NUMBER_THREADS"))
+        with bb.compress.zstd.open(debugsources_file, "wt", encoding="utf-8",

num_threads=num_threads) as f:

meta/lib/oe/package.py +7 -0
+            json.dump(sources_dict, f, sort_keys=True)
+
+def read_debugsources_file(d):
+    import json
+    import bb.compress.zstd
+    try:
+        fn = d.expand("${PKGDESTWORK}/debugsources/${PN}-

debugsources.json.zstd")

meta/lib/oe/package.py +2 -0
+        num_threads = int(d.getVar("BB_NUMBER_THREADS"))
+        with bb.compress.zstd.open(fn, "rt", encoding="utf-8",

num_threads=num_threads) as f:

meta/lib/oe/package.py +7 -0
+            return json.load(f)
+    except FileNotFoundError:
+        bb.debug(1, f"File not found: {fn}")
+        return None
 
 def process_split_and_strip_files(d):
     cpath = oe.cachedpath.CachedPath()
@@ -1280,6 +1323,9 @@ def process_split_and_strip_files(d):
         # Process the dv["srcdir"] if requested...
         # This copies and places the referenced sources for later debugging...
         copydebugsources(dv["srcdir"], sources, d)
+
+        # Save source info to be accessible to other tasks
+        save_debugsources_info(dv["srcdir"], results, d)
     #
     # End of debug splitting
     #
#5 lore raw
Quoted 78 lines · Daniel Turull
Quoted 78 lines

-----Original Message-----
From: Daniel Turull <[email protected]>
Sent: Thursday, 5 June 2025 14:03
To: [email protected]
Cc: Daniel Turull <[email protected]>; Richard Purdie <[email protected]>
Subject: [PATCH v7] package: export debugsources in PKGDESTWORK as json

From: Daniel Turull <[email protected]>

The source information used during packaging can be use from other tasks to have more detailed information on the files used during the compilation and improve SPDX accuracy.

Source files used during compilation are store as compressed zstd json in pkgdata/debugsources/$PN-debugsources.json.zstd
Format:
{ binary1: [src1, src2, ...], binary2: [src1, src2, ...] }

I checked the sstate size, and it slightly increase using core-image-full-cmdline:
Before: 2454884 B
After: 2456860 B (+1976 B or 0,08%)

---
 meta/conf/bitbake.conf |  2 ++
 meta/lib/oe/package.py | 46
++++++++++++++++++++++++++++++++++++++++++

 2 files changed, 48 insertions(+)
meta/conf/bitbake.conf index renamed from meta/conf/bitbake.conf +0 -0

13d55923b7..8c0c71d76d 100644

meta/conf/bitbake.conf +0 -0
@@ -991,5 +991,7 @@ oe.sstatesig.find_sstate_manifest[vardepsexclude] =

"BBEXTENDCURR BBEXTENDVARIAN oe.utils.get_multilib_datastore[vardepsexclude] = "DEFAULTTUNE_MULTILIB_ORIGINAL OVERRIDES" oe.path.format_display[vardepsexclude] = "TOPDIR" oe.utils.get_bb_number_threads[vardepsexclude] = "BB_NUMBER_THREADS" +oe.package.save_debugsources_info[vardepsexclude] = "BB_NUMBER_THREADS" +oe.package.read_debugsources_file[vardepsexclude] = "BB_NUMBER_THREADS" oe.packagedata.emit_pkgdata[vardepsexclude] = "BB_NUMBER_THREADS" oe.packagedata.read_subpkgdata_extended[vardepsexclude] = "BB_NUMBER_THREADS"

meta/lib/oe/package.py index renamed from meta/lib/oe/package.py +0 -0

0bcc04ea54..c5acb7d205 100644

meta/lib/oe/package.py +0 -0
@@ -1049,6 +1049,49 @@ def copydebugsources(debugsrcdir, sources, d):

             if os.path.exists(p) and not os.listdir(p):
                 os.rmdir(p)

+def save_debugsources_info(debugsrcdir, sources_raw, d):
+    import json
+    import bb.compress.zstd
+    if debugsrcdir and sources_raw:
+        debugsources_file = d.expand("${PKGDESTWORK}/debugsources/${PN}- debugsources.json.zstd")
+        debugsources_dir = os.path.dirname(debugsources_file) +        if not os.path.isdir(debugsources_dir):
+            bb.utils.mkdirhier(debugsources_dir)
+        bb.utils.remove(debugsources_file)
+
+        workdir = d.getVar("WORKDIR")
+        sdir = d.getVar("S")
+        pn = d.getVar('PN')
+        pv = d.getVar('PV')
+        sources_dict = {}
+        for file, src_files in sources_raw:
+            file_clean = file.replace(f"{workdir}/package/","") +            sources_clean = [

Quoted 8 lines
Quoted 6 lines

+                src.replace(f"/usr/src/debug/{pn}/", "") +                   .replace(f"{sdir}/", "")
+                   .replace(f"/usr/src/kernel/", "") +                   .replace(f"/usr/src/{pn}/", "") +                   .replace(f"{pn}-{pv}/", "") +                   .replace(f"{pv}/", "")

When looking at Joshua's comments, I relalized that I missed to remove "git". I'll send a new version.

I'm a bit apprehensive about this "magic" set of replacements. In general, sdir (S) is going to represent what you're after. B (the build directory) will also show up here and can probably be mapped the same way as S for the purposes of what you're doing. Files in B will almost always be generated.

/usr/src/xxx should also be possible to be read from a variable rather than hardcoded. I can understand why the kernel ends up special but I'm not sure why there would be two other usr/src options.

I'm therefore left wondering why the other pieces come from and if we can remove more of the magic values here.

Cheers,

Richard

#6 lore raw
Quoted 110 lines

-----Original Message-----
From: Richard Purdie <[email protected]>
Sent: Monday, 9 June 2025 10:57
To: Daniel Turull <[email protected]>; openembedded- [email protected]
Subject: Re: [PATCH v7] package: export debugsources in PKGDESTWORK as json

Quoted 17 lines · Daniel Turull
Quoted 17 lines

-----Original Message-----
From: Daniel Turull <[email protected]>
Sent: Thursday, 5 June 2025 14:03
To: [email protected]
Cc: Daniel Turull <[email protected]>; Richard Purdie <[email protected]>
Subject: [PATCH v7] package: export debugsources in PKGDESTWORK as json

From: Daniel Turull <[email protected]>

The source information used during packaging can be use from other tasks to have more detailed information on the files used during the compilation and improve SPDX accuracy.

Quoted 6 lines
Quoted 6 lines

Source files used during compilation are store as compressed zstd json in pkgdata/debugsources/$PN-debugsources.json.zstd Format:
{ binary1: [src1, src2, ...], binary2: [src1, src2, ...] }

I checked the sstate size, and it slightly increase using core-image-full-

cmdline:

Quoted 11 lines
Quoted 11 lines

Before: 2454884 B
After: 2456860 B (+1976 B or 0,08%)

---
 meta/conf/bitbake.conf |  2 ++
 meta/lib/oe/package.py | 46
++++++++++++++++++++++++++++++++++++++++++

 2 files changed, 48 insertions(+)
Quoted 16 lines
Quoted 8 lines
meta/conf/bitbake.conf index renamed from meta/conf/bitbake.conf +0 -0

13d55923b7..8c0c71d76d 100644

meta/conf/bitbake.conf +0 -0
@@ -991,5 +991,7 @@

oe.sstatesig.find_sstate_manifest[vardepsexclude] = "BBEXTENDCURR BBEXTENDVARIAN oe.utils.get_multilib_datastore[vardepsexclude] = "DEFAULTTUNE_MULTILIB_ORIGINAL OVERRIDES"

Quoted 8 lines

 oe.path.format_display[vardepsexclude] = "TOPDIR" oe.utils.get_bb_number_threads[vardepsexclude] = "BB_NUMBER_THREADS" +oe.package.save_debugsources_info[vardepsexclude] = "BB_NUMBER_THREADS" +oe.package.read_debugsources_file[vardepsexclude] = "BB_NUMBER_THREADS" oe.packagedata.emit_pkgdata[vardepsexclude] =

"BB_NUMBER_THREADS"

Quoted 7 lines

meta/lib/oe/package.py def copydebugsources(debugsrcdir, sources,

Quoted 7 lines

 oe.packagedata.read_subpkgdata_extended[vardepsexclude] = "BB_NUMBER_THREADS"

meta/lib/oe/package.py index renamed from meta/lib/oe/package.py +0 -0

0bcc04ea54..c5acb7d205 100644

meta/lib/oe/package.py +0 -0
@@ -1049,6 +1049,49 @@ def copydebugsources(debugsrcdir, sources,

d):

Quoted 23 lines
Quoted 23 lines

             if os.path.exists(p) and not os.listdir(p):
                 os.rmdir(p)

meta/lib/oe/package.py +6 -0
+def save_debugsources_info(debugsrcdir, sources_raw, d):
+    import json
+    import bb.compress.zstd
+    if debugsrcdir and sources_raw:
+        debugsources_file =
+d.expand("${PKGDESTWORK}/debugsources/${PN}-

debugsources.json.zstd")
+        debugsources_dir = os.path.dirname(debugsources_file) +        if not os.path.isdir(debugsources_dir):
+            bb.utils.mkdirhier(debugsources_dir)
+        bb.utils.remove(debugsources_file)
+
+        workdir = d.getVar("WORKDIR")
+        sdir = d.getVar("S")
+        pn = d.getVar('PN')
+        pv = d.getVar('PV')
+        sources_dict = {}
+        for file, src_files in sources_raw:
+            file_clean = file.replace(f"{workdir}/package/","") +            sources_clean = [

Quoted 8 lines
Quoted 6 lines

+                src.replace(f"/usr/src/debug/{pn}/", "") +                   .replace(f"{sdir}/", "")
+                   .replace(f"/usr/src/kernel/", "") +                   .replace(f"/usr/src/{pn}/", "") +                   .replace(f"{pn}-{pv}/", "") +                   .replace(f"{pv}/", "")

When looking at Joshua's comments, I relalized that I missed to remove

"git". I'll send a new version.

I'm a bit apprehensive about this "magic" set of replacements. In general, sdir
(S) is going to represent what you're after. B (the build directory) will also show up here and can probably be mapped the same way as S for the purposes of what you're doing. Files in B will almost always be generated.

Quoted 6 lines

/usr/src/xxx should also be possible to be read from a variable rather than hardcoded. I can understand why the kernel ends up special but I'm not sure why there would be two other usr/src options.

I'm therefore left wondering why the other pieces come from and if we can remove more of the magic values here.

I'll look at the "magic" and see if we can do the same with variables and remove it also from the SPDX patch.

I'll try to send a new version later this week, without "magic".

Thanks for taking the time to look at it. I feel the series has been improved quite a lot since my first version.

Daniel

Quoted 3 lines

Cheers,

Richard