package: export debugsources in PKGDESTWORK as json
PATCH v7 1/1 package: export debugsources in PKGDESTWORK as json
From: Daniel Turull <[email protected]>
The source information used during packaging can be use from other tasks to have more detailed information on the files used during the compilation and improve SPDX accuracy.
Source files used during compilation are store as compressed zstd json in pkgdata/debugsources/$PN-debugsources.json.zstd
Format:
{ binary1: [src1, src2, ...], binary2: [src1, src2, ...] }
I checked the sstate size, and it slightly increase using core-image-full-cmdline:
Before: 2454884 B
After: 2456860 B (+1976 B or 0,08%)
--- meta/conf/bitbake.conf | 2 ++ meta/lib/oe/package.py | 46 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+)
meta/conf/bitbake.conf +2 -0
@@ -991,5 +991,7 @@ oe.sstatesig.find_sstate_manifest[vardepsexclude] = "BBEXTENDCURR BBEXTENDVARIAN oe.utils.get_multilib_datastore[vardepsexclude] = "DEFAULTTUNE_MULTILIB_ORIGINAL OVERRIDES" oe.path.format_display[vardepsexclude] = "TOPDIR" oe.utils.get_bb_number_threads[vardepsexclude] = "BB_NUMBER_THREADS" +oe.package.save_debugsources_info[vardepsexclude] = "BB_NUMBER_THREADS" +oe.package.read_debugsources_file[vardepsexclude] = "BB_NUMBER_THREADS" oe.packagedata.emit_pkgdata[vardepsexclude] = "BB_NUMBER_THREADS" oe.packagedata.read_subpkgdata_extended[vardepsexclude] = "BB_NUMBER_THREADS"
meta/lib/oe/package.py +46 -0
58 lines not on this page · raw message
Quoted 4 lines · Daniel Turull via lists.openembedded.org
I checked the sstate size, and it slightly increase using core-image-full-cmdline:
Before: 2454884 B
After: 2456860 B (+1976 B or 0,08%)
2000 _bytes_ for adding all of the source information to the core-image-full-cmdline sstate? I know zstd is good but that seems a little low?
Ross
Hi, now I noticed that I put the wrong unit when reading du -s. It is Kbytes. So the increase is 1976 KB, not 1976 Bytes
Best regards,
Daniel
Quoted 20 lines
-----Original Message-----
From: Ross Burton <[email protected]>
Sent: Thursday, 5 June 2025 15:06
To: Daniel Turull <[email protected]>
Cc: [email protected]; Richard Purdie <[email protected]>
Subject: Re: [OE-core] [PATCH v7] package: export debugsources in PKGDESTWORK as jsonQuoted 4 lines · Daniel Turull via lists.openembedded.org
I checked the sstate size, and it slightly increase using core-image-full-cmdline:
Before: 2454884 B
After: 2456860 B (+1976 B or 0,08%)2000 _bytes_ for adding all of the source information to the core-image-full- cmdline sstate? I know zstd is good but that seems a little low?
Ross
Hi Daniel,
Quoted 3 lines · Daniel Turull
Hi, now I noticed that I put the wrong unit when reading du -s. It is Kbytes. So the increase is 1976 KB, not 1976 Bytes
That makes a lot more sense and is an acceptable cost for that useful data.
Thanks,
Ross
Quoted 74 lines
-----Original Message-----
From: Daniel Turull <[email protected]>
Sent: Thursday, 5 June 2025 14:03
To: [email protected]
Cc: Daniel Turull <[email protected]>; Richard Purdie <[email protected]>
Subject: [PATCH v7] package: export debugsources in PKGDESTWORK as jsonFrom: Daniel Turull <[email protected]>
The source information used during packaging can be use from other tasks to have more detailed information on the files used during the compilation and improve SPDX accuracy.
Source files used during compilation are store as compressed zstd json in pkgdata/debugsources/$PN-debugsources.json.zstd
Format:
{ binary1: [src1, src2, ...], binary2: [src1, src2, ...] }I checked the sstate size, and it slightly increase using core-image-full-cmdline:
Before: 2454884 B
After: 2456860 B (+1976 B or 0,08%)CC Richard Purdie <[email protected]>Signed-off-by Daniel Turull <[email protected]>--- meta/conf/bitbake.conf | 2 ++ meta/lib/oe/package.py | 46 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+)meta/conf/bitbake.conf index renamed from meta/conf/bitbake.conf +0
-0
13d55923b7..8c0c71d76d100644meta/conf/bitbake.conf +0
-0@@ -991,5 +991,7 @@ oe.sstatesig.find_sstate_manifest[vardepsexclude] ="BBEXTENDCURR BBEXTENDVARIAN oe.utils.get_multilib_datastore[vardepsexclude] = "
DEFAULTTUNE_MULTILIB_ORIGINALOVERRIDES"
oe.path.format_display[vardepsexclude] = "TOPDIR" oe.utils.get_bb_number_threads[vardepsexclude] =
"BB_NUMBER_THREADS" +oe.package.save_debugsources_info[vardepsexclude] = "BB_NUMBER_THREADS" +oe.package.read_debugsources_file[vardepsexclude] = "BB_NUMBER_THREADS"
oe.packagedata.emit_pkgdata[vardepsexclude] = "BB_NUMBER_THREADS" oe.packagedata.read_subpkgdata_extended[vardepsexclude] =
"BB_NUMBER_THREADS"meta/lib/oe/package.py index renamed from meta/lib/oe/package.py +0
-0
0bcc04ea54..c5acb7d205100644meta/lib/oe/package.py +14
-0@@ -1049,6 +1049,49 @@ def copydebugsources(debugsrcdir, sources, d): if os.path.exists(p) and not os.listdir(p): os.rmdir(p) +def save_debugsources_info(debugsrcdir, sources_raw, d): + import json + import bb.compress.zstd + if debugsrcdir and sources_raw: + debugsources_file = d.expand("${PKGDESTWORK}/debugsources/${PN}- debugsources.json.zstd") + debugsources_dir = os.path.dirname(debugsources_file) + if not os.path.isdir(debugsources_dir): + bb.utils.mkdirhier(debugsources_dir) + bb.utils.remove(debugsources_file) + + workdir = d.getVar("WORKDIR") + sdir = d.getVar("S") + pn = d.getVar('PN') + pv = d.getVar('PV')
meta/lib/oe/package.py def copydebugsources(debugsrcdir, sources, d): · line 1076
Quoted 10 lines
meta/lib/oe/package.py +10
-0+ sources_dict = {} + for file, src_files in sources_raw: + file_clean = file.replace(f"{workdir}/package/","") + sources_clean = [ + src.replace(f"/usr/src/debug/{pn}/", "") + .replace(f"{sdir}/", "") + .replace(f"/usr/src/kernel/", "") + .replace(f"/usr/src/{pn}/", "") + .replace(f"{pn}-{pv}/", "") + .replace(f"{pv}/", "")
When looking at Joshua's comments, I relalized that I missed to remove "git". I'll send a new version.
Quoted 36 lines
meta/lib/oe/package.py +6
-0+ for src in src_files + if not any(keyword in src for keyword in ("<internal>", "<built-in>")) and not src.endswith("/") + ] + sources_dict[file_clean] = sorted(sources_clean) + num_threads = int(d.getVar("BB_NUMBER_THREADS")) + with bb.compress.zstd.open(debugsources_file, "wt", encoding="utf-8",num_threads=num_threads) as f:
meta/lib/oe/package.py +7
-0+ json.dump(sources_dict, f, sort_keys=True) + +def read_debugsources_file(d): + import json + import bb.compress.zstd + try: + fn = d.expand("${PKGDESTWORK}/debugsources/${PN}-debugsources.json.zstd")
meta/lib/oe/package.py +2
-0+ num_threads = int(d.getVar("BB_NUMBER_THREADS")) + with bb.compress.zstd.open(fn, "rt", encoding="utf-8",num_threads=num_threads) as f:
meta/lib/oe/package.py +7
-0+ return json.load(f) + except FileNotFoundError: + bb.debug(1, f"File not found: {fn}") + return None def process_split_and_strip_files(d): cpath = oe.cachedpath.CachedPath()@@ -1280,6 +1323,9 @@ def process_split_and_strip_files(d): # Process the dv["srcdir"] if requested... # This copies and places the referenced sources for later debugging... copydebugsources(dv["srcdir"], sources, d) + + # Save source info to be accessible to other tasks + save_debugsources_info(dv["srcdir"], results, d) # # End of debug splitting #
Quoted 78 lines · Daniel Turull
Quoted 78 lines
-----Original Message-----
From: Daniel Turull <[email protected]>
Sent: Thursday, 5 June 2025 14:03
To: [email protected]
Cc: Daniel Turull <[email protected]>; Richard Purdie <[email protected]>
Subject: [PATCH v7] package: export debugsources in PKGDESTWORK as jsonFrom: Daniel Turull <[email protected]>
The source information used during packaging can be use from other tasks to have more detailed information on the files used during the compilation and improve SPDX accuracy.
Source files used during compilation are store as compressed zstd json in pkgdata/debugsources/$PN-debugsources.json.zstd
Format:
{ binary1: [src1, src2, ...], binary2: [src1, src2, ...] }I checked the sstate size, and it slightly increase using core-image-full-cmdline:
Before: 2454884 B
After: 2456860 B (+1976 B or 0,08%)CC Richard Purdie <[email protected]>Signed-off-by Daniel Turull <[email protected]>---
meta/conf/bitbake.conf | 2 ++
meta/lib/oe/package.py | 46
++++++++++++++++++++++++++++++++++++++++++2 files changed, 48 insertions(+)meta/conf/bitbake.conf index renamed from meta/conf/bitbake.conf +0
-0
13d55923b7..8c0c71d76d100644meta/conf/bitbake.conf +0
-0@@ -991,5 +991,7 @@ oe.sstatesig.find_sstate_manifest[vardepsexclude] ="BBEXTENDCURR BBEXTENDVARIAN oe.utils.get_multilib_datastore[vardepsexclude] = "
DEFAULTTUNE_MULTILIB_ORIGINALOVERRIDES" oe.path.format_display[vardepsexclude] = "TOPDIR" oe.utils.get_bb_number_threads[vardepsexclude] = "BB_NUMBER_THREADS" +oe.package.save_debugsources_info[vardepsexclude] = "BB_NUMBER_THREADS" +oe.package.read_debugsources_file[vardepsexclude] = "BB_NUMBER_THREADS" oe.packagedata.emit_pkgdata[vardepsexclude] = "BB_NUMBER_THREADS" oe.packagedata.read_subpkgdata_extended[vardepsexclude] = "BB_NUMBER_THREADS"meta/lib/oe/package.py index renamed from meta/lib/oe/package.py +0
-0
0bcc04ea54..c5acb7d205100644meta/lib/oe/package.py +0
-0@@ -1049,6 +1049,49 @@ def copydebugsources(debugsrcdir, sources, d):if
os.path.exists(p)and notos.listdir(p):
os.rmdir(p)+def
save_debugsources_info(debugsrcdir, sources_raw, d):
+ import json
+ import bb.compress.zstd
+ if debugsrcdir andsources_raw:
+debugsources_file=d.expand("${PKGDESTWORK}/debugsources/${PN}- debugsources.json.zstd")
+debugsources_dir=os.path.dirname(debugsources_file)+ if notos.path.isdir(debugsources_dir):
+bb.utils.mkdirhier(debugsources_dir)
+bb.utils.remove(debugsources_file)
+
+ workdir =d.getVar("WORKDIR")
+ sdir =d.getVar("S")
+ pn =d.getVar('PN')
+ pv =d.getVar('PV')
+sources_dict= {}
+ for file,src_filesinsources_raw:
+file_clean=file.replace(f"{workdir}/package/","")+sources_clean= [
Quoted 8 lines
Quoted 6 lines
+
src.replace(f"/usr/src/debug/{pn}/", "")+ .replace(f"{sdir}/", "")
+ .replace(f"/usr/src/kernel/", "") + .replace(f"/usr/src/{pn}/", "") + .replace(f"{pn}-{pv}/", "") + .replace(f"{pv}/", "")When looking at Joshua's comments, I relalized that I missed to remove "git". I'll send a new version.
I'm a bit apprehensive about this "magic" set of replacements. In general, sdir (S) is going to represent what you're after. B (the build directory) will also show up here and can probably be mapped the same way as S for the purposes of what you're doing. Files in B will almost always be generated.
/usr/src/xxx should also be possible to be read from a variable rather than hardcoded. I can understand why the kernel ends up special but I'm not sure why there would be two other usr/src options.
I'm therefore left wondering why the other pieces come from and if we can remove more of the magic values here.
Cheers,
Richard
Quoted 110 lines
-----Original Message-----
From: Richard Purdie <[email protected]>
Sent: Monday, 9 June 2025 10:57
To: Daniel Turull <[email protected]>; openembedded- [email protected]
Subject: Re: [PATCH v7] package: export debugsources in PKGDESTWORK as jsonQuoted 17 lines · Daniel Turull
Quoted 17 lines
-----Original Message-----
From: Daniel Turull <[email protected]>
Sent: Thursday, 5 June 2025 14:03
To: [email protected]
Cc: Daniel Turull <[email protected]>; Richard Purdie <[email protected]>
Subject: [PATCH v7] package: export debugsources in PKGDESTWORK as jsonFrom: Daniel Turull <[email protected]>
The source information used during packaging can be use from other tasks to have more detailed information on the files used during the compilation and improve SPDX accuracy.
Quoted 6 lines
Quoted 6 lines
Source files used during compilation are store as compressed zstd json in pkgdata/debugsources/$PN-debugsources.json.zstd Format:
{ binary1: [src1, src2, ...], binary2: [src1, src2, ...] }I checked the sstate size, and it slightly increase using core-image-full-
cmdline:
Quoted 11 lines
Quoted 11 lines
Before: 2454884 B
After: 2456860 B (+1976 B or 0,08%)CC Richard Purdie <[email protected]>Signed-off-by Daniel Turull <[email protected]>---
meta/conf/bitbake.conf | 2 ++
meta/lib/oe/package.py | 46
++++++++++++++++++++++++++++++++++++++++++2 files changed, 48 insertions(+)Quoted 16 lines
Quoted 8 lines
meta/conf/bitbake.conf index renamed from meta/conf/bitbake.conf +0
-0
13d55923b7..8c0c71d76d100644meta/conf/bitbake.conf +0
-0@@ -991,5 +991,7 @@oe.sstatesig.find_sstate_manifest[vardepsexclude] = "BBEXTENDCURR BBEXTENDVARIAN oe.utils.get_multilib_datastore[vardepsexclude] = "
DEFAULTTUNE_MULTILIB_ORIGINALOVERRIDES"Quoted 8 lines
oe.path.format_display[vardepsexclude] = "TOPDIR" oe.utils.get_bb_number_threads[vardepsexclude] = "
BB_NUMBER_THREADS" +oe.package.save_debugsources_info[vardepsexclude] = "BB_NUMBER_THREADS" +oe.package.read_debugsources_file[vardepsexclude] = "BB_NUMBER_THREADS" oe.packagedata.emit_pkgdata[vardepsexclude] ="
BB_NUMBER_THREADS"Quoted 7 lines
meta/lib/oe/package.py def copydebugsources(debugsrcdir, sources,
Quoted 7 lines
oe.packagedata.read_subpkgdata_extended[vardepsexclude] = "
BB_NUMBER_THREADS"meta/lib/oe/package.py index renamed from meta/lib/oe/package.py +0
-0
0bcc04ea54..c5acb7d205100644meta/lib/oe/package.py +0
-0@@ -1049,6 +1049,49 @@ def copydebugsources(debugsrcdir, sources,d):
Quoted 23 lines
Quoted 23 lines
if
os.path.exists(p)and notos.listdir(p):
os.rmdir(p)meta/lib/oe/package.py +6
-0+def save_debugsources_info(debugsrcdir, sources_raw, d): + import json + import bb.compress.zstd + if debugsrcdir and sources_raw: + debugsources_file = +d.expand("${PKGDESTWORK}/debugsources/${PN}-debugsources.json.zstd")
+debugsources_dir=os.path.dirname(debugsources_file)+ if notos.path.isdir(debugsources_dir):
+bb.utils.mkdirhier(debugsources_dir)
+bb.utils.remove(debugsources_file)
+
+ workdir =d.getVar("WORKDIR")
+ sdir =d.getVar("S")
+ pn =d.getVar('PN')
+ pv =d.getVar('PV')
+sources_dict= {}
+ for file,src_filesinsources_raw:
+file_clean=file.replace(f"{workdir}/package/","")+sources_clean= [Quoted 8 lines
Quoted 6 lines
+
src.replace(f"/usr/src/debug/{pn}/", "")+ .replace(f"{sdir}/", "")
+ .replace(f"/usr/src/kernel/", "") + .replace(f"/usr/src/{pn}/", "") + .replace(f"{pn}-{pv}/", "") + .replace(f"{pv}/", "")When looking at Joshua's comments, I relalized that I missed to remove
"git". I'll send a new version.
I'm a bit apprehensive about this "magic" set of replacements. In general, sdir
(S) is going to represent what you're after. B (the build directory) will also show up here and can probably be mapped the same way as S for the purposes of what you're doing. Files in B will almost always be generated.
Quoted 6 lines
/usr/src/xxxshould also be possible to be read from a variable rather than hardcoded. I can understand why the kernel ends up special but I'm not sure why there would be two other usr/src options.I'm therefore left wondering why the other pieces come from and if we can remove more of the magic values here.
I'll look at the "magic" and see if we can do the same with variables and remove it also from the SPDX patch.
I'll try to send a new version later this week, without "magic".
Thanks for taking the time to look at it. I feel the series has been improved quite a lot since my first version.
Daniel
Quoted 3 lines
Cheers,
Richard