package: export debugsources in PKGDESTWORK as json
PATCH 1/2 package: export debugsources in PKGDESTWORK as json
From: Daniel Turull <[email protected]>
The source information used during packaging can be use from other tasks to have more detailed information on the files used during the compilation and improve SPDX accuracy.
Source files used during compilation are store as compressed zstd json in pkgdata/debugsources/$PN-debugsources.json.zstd
Format:
{ binary1: [src1, src2, ...], binary2: [src1, src2, ...] }
I checked the sstate size, and it slightly increases using core-image-full-cmdline:
without patch: 2456792 KB sstate-cache/ with patch: 2460028 KB sstate-cache/
(4236 KB or 0.17%)
(From OE-Core rev: c507dcb8a8780a42bfe68b1ebaff0909b4236e6b) Adaptations to match spdx in scarthgap: change BP to PF
--- meta/conf/bitbake.conf | 2 ++ meta/lib/oe/package.py | 46 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+)
meta/conf/bitbake.conf +2 -0
@@ -989,5 +989,7 @@ oe.sstatesig.find_sstate_manifest[vardepsexclude] = "BBEXTENDCURR BBEXTENDVARIAN oe.utils.get_multilib_datastore[vardepsexclude] = "DEFAULTTUNE_MULTILIB_ORIGINAL OVERRIDES" oe.path.format_display[vardepsexclude] = "TOPDIR" oe.utils.get_bb_number_threads[vardepsexclude] = "BB_NUMBER_THREADS" +oe.package.save_debugsources_info[vardepsexclude] = "BB_NUMBER_THREADS" +oe.package.read_debugsources_info[vardepsexclude] = "BB_NUMBER_THREADS" oe.packagedata.emit_pkgdata[vardepsexclude] = "BB_NUMBER_THREADS" oe.packagedata.read_subpkgdata_extended[vardepsexclude] = "BB_NUMBER_THREADS"
meta/lib/oe/package.py +46 -0
58 lines not on this page · raw message
PATCH 2/2 spdx: add option to include only compiled sources
From: Daniel Turull <[email protected]>
When SPDX_INCLUDE_COMPILED_SOURCES is enabled, only include the source code files that are used during compilation.
It uses debugsource information generated during do_package.
This enables an external tool to use the SPDX information to disregard vulnerabilities that are not compiled.
As example, when used with the default config with linux-yocto, the spdx size is reduced from 156MB to 61MB.
Tested with bitbake world on oe-core.
(From OE-Core rev: c6a2f1fca76fae4c3ea471a0c63d0b453beea968) Adapted to existing files for create-spdx-2.2
--- meta/classes/create-spdx-2.2.bbclass | 12 ++++++++ meta/lib/oe/spdx.py | 42 ++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+)
meta/classes/create-spdx-2.2.bbclass +12 -0
@@ -100,6 +100,9 @@ python() { # Transform the license array to a dictionary data["licenses"] = {l["licenseId"]: l for l in data["licenses"]} d.setVar("SPDX_LICENSE_DATA", data) + + if d.getVar("SPDX_INCLUDE_COMPILED_SOURCES") == "1": + d.setVar("SPDX_INCLUDE_SOURCES", "1") } def convert_license_to_spdx(lic, document, d, existing={}):
@@ -215,6 +218,11 @@ def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *, archiv spdx_files = [] file_counter = 1 + + check_compiled_sources = d.getVar("SPDX_INCLUDE_COMPILED_SOURCES") == "1" + if check_compiled_sources: + compiled_sources, types = oe.spdx.get_compiled_sources(d) + bb.debug(1, f"Total compiled files: {len(compiled_sources)}") for subdir, dirs, files in os.walk(topdir): dirs[:] = [d for d in dirs if d not in ignore_dirs] if subdir == str(topdir):
@@ -225,6 +233,10 @@ def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *, archiv filename = str(filepath.relative_to(topdir)) if not filepath.is_symlink() and filepath.is_file(): + # Check if file is compiled + if check_compiled_sources: + if not oe.spdx.is_compiled_source(filename, compiled_sources, types): + continue spdx_file = oe.spdx.SPDXFile() spdx_file.SPDXID = get_spdxid(file_counter) for t in get_types(filepath):
meta/lib/oe/spdx.py +42 -0
45 lines not on this page · raw message
Hi Daniel,
Since this is a feature addition the TSC will need to approve it before I can take this backport.
I've copied the TSC so they can add this to the agenda for their next meeting.
Thanks,
Steve
Quoted 113 lines · Daniel Turull via lists.openembedded.org
From: Daniel Turull <[email protected]>
The source information used during packaging can be use from other tasks to have more detailed information on the files used during the compilation and improve SPDX accuracy.
Source files used during compilation are store as compressed zstd json in pkgdata/debugsources/$PN-debugsources.json.zstd
Format:
{ binary1: [src1, src2, ...], binary2: [src1, src2, ...] }I checked the sstate size, and it slightly increases using core-image-full-cmdline:
without patch: 2456792 KB sstate-cache/ with patch: 2460028 KB sstate-cache/(4236 KB or 0.17%)
(From OE-Core rev:
c507dcb8a8780a42bfe68b1ebaff0909b4236e6b) Adaptations to match spdx in scarthgap: change BP to PFCC Mathieu Dubois-Briand <[email protected]>CC Richard Purdie <[email protected]>Signed-off-by Daniel Turull <[email protected]>--- meta/conf/bitbake.conf | 2 ++ meta/lib/oe/package.py | 46 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+)meta/conf/bitbake.conf +2
-0@@ -989,5 +989,7 @@ oe.sstatesig.find_sstate_manifest[vardepsexclude] = "BBEXTENDCURR BBEXTENDVARIAN oe.utils.get_multilib_datastore[vardepsexclude] = "DEFAULTTUNE_MULTILIB_ORIGINAL OVERRIDES" oe.path.format_display[vardepsexclude] = "TOPDIR" oe.utils.get_bb_number_threads[vardepsexclude] = "BB_NUMBER_THREADS" +oe.package.save_debugsources_info[vardepsexclude] = "BB_NUMBER_THREADS" +oe.package.read_debugsources_info[vardepsexclude] = "BB_NUMBER_THREADS" oe.packagedata.emit_pkgdata[vardepsexclude] = "BB_NUMBER_THREADS" oe.packagedata.read_subpkgdata_extended[vardepsexclude] = "BB_NUMBER_THREADS"meta/lib/oe/package.py +46
-1@@ -1038,6 +1038,49 @@ def copydebugsources(debugsrcdir, sources, d): if os.path.exists(p) and not os.listdir(p): os.rmdir(p) +def save_debugsources_info(debugsrcdir, sources_raw, d): + import json + import bb.compress.zstd + if debugsrcdir and sources_raw: + debugsources_file = d.expand("${PKGDESTWORK}/debugsources/${PN}-debugsources.json.zstd") + debugsources_dir = os.path.dirname(debugsources_file) + if not os.path.isdir(debugsources_dir): + bb.utils.mkdirhier(debugsources_dir) + bb.utils.remove(debugsources_file) + + workdir = d.getVar("WORKDIR") + pn = d.getVar('PN') + + # Kernel sources are in a different directory and are special case + # we format the sources as expected by spdx by replacing /usr/src/kernel/ + # into BP/ + kernel_src = d.getVar('KERNEL_SRC_PATH') + pf = d.getVar('PF') + sources_dict = {} + for file, src_files in sources_raw: + file_clean = file.replace(f"{workdir}/package/","") + sources_clean = [ + src.replace(f"{debugsrcdir}/{pn}/", "") + if not kernel_src else src.replace(f"{kernel_src}/", f"{pf}/") + for src in src_files + if not any(keyword in src for keyword in ("<internal>", "<built-in>")) and not src.endswith("/") + ] + sources_dict[file_clean] = sorted(sources_clean) + num_threads = int(d.getVar("BB_NUMBER_THREADS")) + with bb.compress.zstd.open(debugsources_file, "wt", encoding="utf-8", num_threads=num_threads) as f: + json.dump(sources_dict, f, sort_keys=True) + +def read_debugsources_info(d): + import json + import bb.compress.zstd + try: + fn = d.expand("${PKGDESTWORK}/debugsources/${PN}-debugsources.json.zstd") + num_threads = int(d.getVar("BB_NUMBER_THREADS")) + with bb.compress.zstd.open(fn, "rt", encoding="utf-8", num_threads=num_threads) as f: + return json.load(f) + except FileNotFoundError: + bb.debug(1, f"File not found: {fn}") + return None def process_split_and_strip_files(d): cpath = oe.cachedpath.CachedPath()@@ -1269,6 +1312,9 @@ def process_split_and_strip_files(d): # Process the dv["srcdir"] if requested... # This copies and places the referenced sources for later debugging... copydebugsources(dv["srcdir"], sources, d) + + # Save source info to be accessible to other tasks + save_debugsources_info(dv["srcdir"], results, d) # # End of debug splitting # -=-=-=-=-=-=-=-=-=-=-=-Links: You receive all messages sent to this group.
View/Reply Online (#219060): https://lists.openembedded.org/g/openembedded-core/message/219060
Mute This Topic: https://lists.openembedded.org/mt/113722386/3620601
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Thanks Steve for letting me know. I understand. Please let me know if you need anything from me.
Best regards,
Daniel
Quoted 196 lines
-----Original Message-----
From: Steve Sakoman <[email protected]>
Sent: Thursday, 19 June 2025 17:24
To: Daniel Turull <[email protected]>
Cc: [email protected]; Mathieu Dubois-Briand <[email protected]>; Richard Purdie <[email protected]>; Yocto TSC <[email protected]>
Subject: Re: [OE-core] [scarthgap][PATCH 1/2] package: export debugsources in PKGDESTWORK as jsonHi Daniel,
Since this is a feature addition the TSC will need to approve it before I can take this backport.
I've copied the TSC so they can add this to the agenda for their next meeting.
Thanks,
Steve
Quoted 34 lines · Daniel Turull via lists.openembedded.org
From: Daniel Turull <[email protected]>
The source information used during packaging can be use from other tasks to have more detailed information on the files used during the compilation and improve SPDX accuracy.
Source files used during compilation are store as compressed zstd json in pkgdata/debugsources/$PN-debugsources.json.zstd
Format:
{ binary1: [src1, src2, ...], binary2: [src1, src2, ...] }I checked the sstate size, and it slightly increases using core-image-full-cmdline:
without patch: 2456792 KB sstate-cache/ with patch: 2460028 KB sstate-cache/(4236 KB or 0.17%)
(From OE-Core rev:
c507dcb8a8780a42bfe68b1ebaff0909b4236e6b) Adaptations to match spdx in scarthgap: change BP to PFCC Mathieu Dubois-Briand <[email protected]>CC Richard Purdie <[email protected]>Signed-off-by Daniel Turull <[email protected]>--- meta/conf/bitbake.conf | 2 ++ meta/lib/oe/package.py | 46 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+)meta/conf/bitbake.conf index renamed from meta/conf/bitbake.conf +0
-0
78f15b76ae..acf4e2d153100644meta/conf/bitbake.conf +0
-0@@ -989,5 +989,7 @@ oe.sstatesig.find_sstate_manifest[vardepsexclude]Quoted 1 line
= "BBEXTENDCURR BBEXTENDVARIAN
oe.utils.get_multilib_datastore[vardepsexclude] = "
DEFAULTTUNE_MULTILIB_ORIGINALOVERRIDES"Quoted 3 lines
oe.path.format_display[vardepsexclude] = "TOPDIR" oe.utils.get_bb_number_threads[vardepsexclude] = "
BB_NUMBER_THREADS"
+oe.package.save_debugsources_info[vardepsexclude] ="
BB_NUMBER_THREADS"Quoted 1 line
+oe.package.read_debugsources_info[vardepsexclude] =
"
BB_NUMBER_THREADS"Quoted 2 lines
oe.packagedata.emit_pkgdata[vardepsexclude] = "
BB_NUMBER_THREADS" oe.packagedata.read_subpkgdata_extended[vardepsexclude] ="
BB_NUMBER_THREADS"meta/lib/oe/package.py def copydebugsources(debugsrcdir, sources, d): · line 1045
Quoted 13 lines
meta/lib/oe/package.py index renamed from meta/lib/oe/package.py +0
-0
af0923a63f..ba0d326781100644meta/lib/oe/package.py +5
-0@@ -1038,6 +1038,49 @@ def copydebugsources(debugsrcdir, sources, d): if os.path.exists(p) and not os.listdir(p): os.rmdir(p) +def save_debugsources_info(debugsrcdir, sources_raw, d): + import json + import bb.compress.zstd + if debugsrcdir and sources_raw: + debugsources_file = d.expand("${PKGDESTWORK}/debugsources/${PN}-debugsources.json.zstd")
Quoted 11 lines
meta/lib/oe/package.py +11
-0+ debugsources_dir = os.path.dirname(debugsources_file) + if not os.path.isdir(debugsources_dir): + bb.utils.mkdirhier(debugsources_dir) + bb.utils.remove(debugsources_file) + + workdir = d.getVar("WORKDIR") + pn = d.getVar('PN') + + # Kernel sources are in a different directory and are special case + # we format the sources as expected by spdx by replacing /usr/src/kernel/ + # into BP/meta/lib/oe/package.py def copydebugsources(debugsrcdir, sources, d):
Quoted 10 lines
meta/lib/oe/package.py +10
-0+ kernel_src = d.getVar('KERNEL_SRC_PATH') + pf = d.getVar('PF') + sources_dict = {} + for file, src_files in sources_raw: + file_clean = file.replace(f"{workdir}/package/","") + sources_clean = [ + src.replace(f"{debugsrcdir}/{pn}/", "") + if not kernel_src else src.replace(f"{kernel_src}/", f"{pf}/") + for src in src_files + if not any(keyword in src for keyword in ("<internal>", "<built-in>"))and not
src.endswith("/")meta/lib/oe/package.py def copydebugsources(debugsrcdir, sources, d):
Quoted 4 lines
meta/lib/oe/package.py +4
-0+ ] + sources_dict[file_clean] = sorted(sources_clean) + num_threads = int(d.getVar("BB_NUMBER_THREADS")) + with bb.compress.zstd.open(debugsources_file, "wt", encoding="utf-8",num_threads=num_threads) as f:
meta/lib/oe/package.py def copydebugsources(debugsrcdir, sources, d):
Quoted 7 lines
meta/lib/oe/package.py +7
-0+ json.dump(sources_dict, f, sort_keys=True) + +def read_debugsources_info(d): + import json + import bb.compress.zstd + try: + fn = d.expand("${PKGDESTWORK}/debugsources/${PN}-debugsources.json.zstd")
meta/lib/oe/package.py def copydebugsources(debugsrcdir, sources, d): · line 1079
Quoted 2 lines
meta/lib/oe/package.py +2
-0+ num_threads = int(d.getVar("BB_NUMBER_THREADS")) + with bb.compress.zstd.open(fn, "rt", encoding="utf-8",num_threads=num_threads) as f:
Quoted 19 lines
meta/lib/oe/package.py +4
-0+ return json.load(f) + except FileNotFoundError: + bb.debug(1, f"File not found: {fn}") + return None def process_split_and_strip_files(d): cpath = oe.cachedpath.CachedPath() @@ -1269,6 +1312,9 @@ def
process_split_and_strip_files(d):
# Process the dv["srcdir"] if requested...
# This copies and places the referenced sources for later debugging...copydebugsources(dv["srcdir"], sources, d)meta/lib/oe/package.py +3
-1+ + # Save source info to be accessible to other tasks + save_debugsources_info(dv["srcdir"], results, d) # # End of debug splitting # -=-=-=-=-=-=-=-=-=-=-=-Quoted 4 lines
Links: You receive all messages sent to this group. View/Reply Online (#219060):
https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist
s.openembedded.org%2Fg%2Fopenembedded-core%2Fmessage%2F219060&data=05%
Quoted 1 line
7C02%7Cdaniel.turull%40ericsson.com%7C91edce15ba0d48a507bd08ddaf45505 1
Quoted 1 line
%7C92e84cebfbfd47abbe52080c6b87953f%7C0%7C0%7C638859434418210300 %7CUnk
Quoted 1 line
nown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIl AiOiJ
Quoted 1 line
XaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Nd6J wfT8Z
Quoted 4 lines
I7PrsTzldSmYXIscMdi%2Ff6NGA6mLD3eVZ0%3D&reserved=0 Mute This Topic:
https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.openembedded.org%2Fmt%2F113722386%2F3620601&data=05%7C02%7Cdan iel.tu
Quoted 1 line
rull%40ericsson.com%7C91edce15ba0d48a507bd08ddaf455051%7C92e84cebfbf d4
Quoted 1 line
7abbe52080c6b87953f%7C0%7C0%7C638859434418229205%7CUnknown%7CT WFpbGZsb
Quoted 1 line
3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIj o
Quoted 1 line
iTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=OcMMd%2Fjagn8lY2Si% 2Fd0DK
Quoted 5 lines
NQ9EcRUo75nmXST96JFT0Y%3D&reserved=0
Group Owner: [email protected] Unsubscribe:
https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist
s.openembedded.org%2Fg%2Fopenembedded-core%2Funsub&data=05%7C02%7Cdani
Quoted 1 line
el.turull%40ericsson.com%7C91edce15ba0d48a507bd08ddaf455051%7C92e84c eb
Quoted 1 line
fbfd47abbe52080c6b87953f%7C0%7C0%7C638859434418241703%7CUnknown %7CTWFp
Quoted 1 line
bGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiI sIk
Quoted 1 line
FOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=OI7it65hjB%2BkO pZboP
Quoted 3 lines
tFKEe2Zv1cWNF9FWu5jUVsEJw%3D&reserved=0 [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 5 lines · Steve Sakoman
Since this is a feature addition the TSC will need to approve it before I can take this backport.
I've copied the TSC so they can add this to the agenda for their next meeting.
The TSC discussed this today. The change adds a single data file to packagedata based on existing information we already calculate during the build so the potential risk and impact on existing users is low.
Whilst this is a feature change, the uses for security and handling CVEs are extremely useful so on balance the TSC thinks this can be merged.
Cheers,
Richard
(on behalf of the YP TSC)
Thanks 😊
Could I submit to scarthgap the standalone script, or people can take it from master?
e60b1759c1aea5b8f5317e46608f0a3e782ecf57 improve_kernel_cve_report: add script for postprocesing of kernel CVE data
Best regards,
Daniel
Quoted 28 lines
-----Original Message-----
From: Richard Purdie <[email protected]>
Sent: Monday, 23 June 2025 23:33
To: Steve Sakoman <[email protected]>; Daniel Turull <[email protected]>
Cc: [email protected]; Mathieu Dubois-Briand <[email protected]>; Yocto TSC <[email protected]>
Subject: Re: [OE-core] [scarthgap][PATCH 1/2] package: export debugsources in PKGDESTWORK as jsonQuoted 5 lines · Steve Sakoman
Since this is a feature addition the TSC will need to approve it before I can take this backport.
I've copied the TSC so they can add this to the agenda for their next meeting.
The TSC discussed this today. The change adds a single data file to packagedata based on existing information we already calculate during the build so the potential risk and impact on existing users is low.
Whilst this is a feature change, the uses for security and handling CVEs are extremely useful so on balance the TSC thinks this can be merged.
Cheers,
Richard
(on behalf of the YP TSC)
Quoted 8 lines · Richard Purdie
Quoted 5 lines · Steve Sakoman
Since this is a feature addition the TSC will need to approve it before I can take this backport.
I've copied the TSC so they can add this to the agenda for their next meeting.
Quoted 7 lines
The TSC discussed this today. The change adds a single data file to packagedata based on existing information we already calculate during the build so the potential risk and impact on existing users is low.
Whilst this is a feature change, the uses for security and handling CVEs are extremely useful so on balance the TSC thinks this can be merged.
Unfortunately this patch is causing oe-selftest failures on the autobuilder, here's a sample log:
https://errors.yoctoproject.org/Errors/Details/862388/
This will need to be addressed before I can merge.
Thanks,
Steve
Ok.
I'll take a look and see what changed from scarthgap to master. I'll resend the patch when it is fixed. I'll test with the oe-selftest
Daniel
Quoted 45 lines
-----Original Message-----
From: Steve Sakoman <[email protected]>
Sent: Wednesday, 25 June 2025 17:27
To: Daniel Turull <[email protected]>
Cc: [email protected]; Mathieu Dubois-Briand <[email protected]>; Yocto TSC <[email protected]>; Richard Purdie <[email protected]>
Subject: Re: [OE-core] [scarthgap][PATCH 1/2] package: export debugsources in PKGDESTWORK as jsonQuoted 8 lines · Richard Purdie
Quoted 5 lines · Steve Sakoman
Since this is a feature addition the TSC will need to approve it before I can take this backport.
I've copied the TSC so they can add this to the agenda for their next meeting.
Quoted 7 lines
The TSC discussed this today. The change adds a single data file to packagedata based on existing information we already calculate during the build so the potential risk and impact on existing users is low.
Whilst this is a feature change, the uses for security and handling CVEs are extremely useful so on balance the TSC thinks this can be merged.
Unfortunately this patch is causing oe-selftest failures on the autobuilder, here's a sample log:
https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ferrors.
yoctoproject.org%2FErrors%2FDetails%2F862388%2F&data=05%7C02%7Cdan iel.turull%40ericsson.com%7Cdefa9e6dee2e4d7b179308ddb3fcc9fd%7C92e8 4cebfbfd47abbe52080c6b87953f%7C0%7C0%7C638864620472989564%7CUn known%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAw MCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C &sdata=yfloJxqQl8rqqBSwuFZdlk62PfAx0RD%2ByfILVfmTgI8%3D&reserved=0This will need to be addressed before I can merge.
Thanks,
Steve
Hi Steve,
I looked at the log and it is a bit strange. The failure in the log seems to be related to sstate. From what I see in the log sstatetests.SStatePrintdiff.test_gcc_runtime_vs_gcc_source is failing but my patch is not touching gcc neither sstate. It is only creating a new zstd file with existing data in do_package.
2025-06-24 22:32:42,764 - oe-selftest - INFO - sstatetests.SStatePrintdiff.test_gcc_runtime_vs_gcc_source (subunit.RemotedTestCase) 2025-06-24 22:32:42,764 - oe-selftest - INFO - ... FAIL
File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/sstatetests.py", line 884, in test_gcc_runtime_vs_gcc_source
self.run_test_printdiff_changerecipe("gcc-runtime", "gcc-source", "-c do_preconfigure {}".format(gcc_source_pn),
File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/sstatetests.py", line 818, in run_test_printdiff_changerecipe
result_sametmp = bitbake("-S printdiff {}".format(target))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/utils/commands.py", line 234, in bitbake
return runCmd(cmd, ignore_status, timeout, output_log=output_log, **options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/utils/commands.py", line 212, in runCmd
raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command 'bitbake -S printdiff gcc-runtime' returned non-zero exit status 1:
When I run the same failing test in my local machine, it passes.
oe-selftest -r sstatetests.SStatePrintdiff.test_gcc_runtime_vs_gcc_source -j 20 -T machine -T toolchain-user -T toolchain-system
2025-06-26 09:44:35,119 - oe-selftest - INFO - Adding layer libraries: 2025-06-26 09:44:35,119 - oe-selftest - INFO - /local/x/yocto-scarthgap/meta-yocto/meta-poky/lib 2025-06-26 09:44:35,119 - oe-selftest - INFO - /local/x/yocto-scarthgap/meta/lib 2025-06-26 09:44:35,120 - oe-selftest - INFO - /repo/x/yocto-scarthgap/meta-yocto/meta-yocto-bsp/lib 2025-06-26 09:44:35,120 - oe-selftest - INFO - /repo/x/yocto-scarthgap/openembedded-core/meta-skeleton/lib 2025-06-26 09:44:35,120 - oe-selftest - INFO - /local/x/yocto-scarthgap/openembedded-core/meta-selftest/lib 2025-06-26 09:44:35,121 - oe-selftest - INFO - Checking base configuration is valid/parsable NOTE: Starting bitbake server... 2025-06-26 09:46:05,270 - oe-selftest - INFO - sstatetests.SStatePrintdiff.test_gcc_runtime_vs_gcc_source (subunit.RemotedTestCase) 2025-06-26 09:46:05,271 - oe-selftest - INFO - ... ok 2025-06-26 09:46:05,271 - oe-selftest - INFO - 0: 1/1 1/1 (89.19s) (0 failed) (sstatetests.SStatePrintdiff.test_gcc_runtime_vs_gcc_source) 2025-06-26 09:46:14,303 - oe-selftest - INFO - ---------------------------------------------------------------------- 2025-06-26 09:46:14,303 - oe-selftest - INFO - Ran 1 test in 98.415s 2025-06-26 09:46:14,303 - oe-selftest - INFO - OK 2025-06-26 09:46:18,464 - oe-selftest - INFO - RESULTS: 2025-06-26 09:46:18,464 - oe-selftest - INFO - RESULTS - sstatetests.SStatePrintdiff.test_gcc_runtime_vs_gcc_source: PASSED (89.20s) 2025-06-26 09:46:18,465 - oe-selftest - INFO - SUMMARY: 2025-06-26 09:46:18,465 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 98.418s 2025-06-26 09:46:18,466 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=1, skipped=0, failures=0, errors=0)
Is it possible that was a temporally glitch in the CI, or related to the latest update to gcc 13.4? Best regards,
Daniel
Quoted 44 lines
-----Original Message-----
From: Steve Sakoman <[email protected]>
Sent: Wednesday, 25 June 2025 17:27
To: Daniel Turull <[email protected]>
Cc: [email protected]; Mathieu Dubois-Briand <[email protected]>; Yocto TSC <[email protected]>; Richard Purdie <[email protected]>
Subject: Re: [OE-core] [scarthgap][PATCH 1/2] package: export debugsources in PKGDESTWORK as jsonQuoted 8 lines · Richard Purdie
Quoted 5 lines · Steve Sakoman
Since this is a feature addition the TSC will need to approve it before I can take this backport.
I've copied the TSC so they can add this to the agenda for their next meeting.
Quoted 7 lines
The TSC discussed this today. The change adds a single data file to packagedata based on existing information we already calculate during the build so the potential risk and impact on existing users is low.
Whilst this is a feature change, the uses for security and handling CVEs are extremely useful so on balance the TSC thinks this can be merged.
Unfortunately this patch is causing oe-selftest failures on the autobuilder, here's a sample log:
https://errors.yoct/
oproject.org%2FErrors%2FDetails%2F862388%2F&data=05%7C02%7Cdaniel.turu ll%40ericsson.com%7Cdefa9e6dee2e4d7b179308ddb3fcc9fd%7C92e84cebfbfd4 7abbe52080c6b87953f%7C0%7C0%7C638864620472989564%7CUnknown%7CT WFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4 zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=yfloJxqQl8r qqBSwuFZdlk62PfAx0RD%2ByfILVfmTgI8%3D&reserved=0This will need to be addressed before I can merge.
Thanks,
Steve
Hi Daniel,
Quoted 43 lines · Daniel Turull
Hi Steve,
I looked at the log and it is a bit strange. The failure in the log seems to be related to sstate. From what I see in the log sstatetests.SStatePrintdiff.test_gcc_runtime_vs_gcc_source is failing but my patch is not touching gcc neither sstate. It is only creating a new zstd file with existing data indo_package.2025-06-24 22:32:42,764 - oe-selftest - INFO - sstatetests.SStatePrintdiff.test_gcc_runtime_vs_gcc_source (subunit.RemotedTestCase) 2025-06-24 22:32:42,764 - oe-selftest - INFO - ... FAILFile "
/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/sstatetests.py", line 884, intest_gcc_runtime_vs_gcc_source
self.run_test_printdiff_changerecipe("gcc-runtime", "gcc-source", "-cdo_preconfigure{}".format(gcc_source_pn),
File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/sstatetests.py", line 818, inrun_test_printdiff_changerecipe
result_sametmp=bitbake("-S printdiff {}".format(target))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/utils/commands.py", line 234, in bitbake
returnrunCmd(cmd, ignore_status, timeout, output_log=output_log, **options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/utils/commands.py", line 212, in runCmd
raiseAssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command 'bitbake -S printdiff gcc-runtime' returned non-zero exit status 1:When I run the same failing test in my local machine, it passes.
oe-selftest -r sstatetests.SStatePrintdiff.test_gcc_runtime_vs_gcc_source -j 20 -T machine -T toolchain-user -T toolchain-system2025-06-26 09:44:35,119 - oe-selftest - INFO - Adding layer libraries: 2025-06-26 09:44:35,119 - oe-selftest - INFO - /local/x/yocto-scarthgap/meta-yocto/meta-poky/lib 2025-06-26 09:44:35,119 - oe-selftest - INFO - /local/x/yocto-scarthgap/meta/lib 2025-06-26 09:44:35,120 - oe-selftest - INFO - /repo/x/yocto-scarthgap/meta-yocto/meta-yocto-bsp/lib 2025-06-26 09:44:35,120 - oe-selftest - INFO - /repo/x/yocto-scarthgap/openembedded-core/meta-skeleton/lib 2025-06-26 09:44:35,120 - oe-selftest - INFO - /local/x/yocto-scarthgap/openembedded-core/meta-selftest/lib 2025-06-26 09:44:35,121 - oe-selftest - INFO - Checking base configuration is valid/parsable NOTE: Starting bitbake server... 2025-06-26 09:46:05,270 - oe-selftest - INFO - sstatetests.SStatePrintdiff.test_gcc_runtime_vs_gcc_source (subunit.RemotedTestCase) 2025-06-26 09:46:05,271 - oe-selftest - INFO - ... ok 2025-06-26 09:46:05,271 - oe-selftest - INFO - 0: 1/1 1/1 (89.19s) (0 failed) (sstatetests.SStatePrintdiff.test_gcc_runtime_vs_gcc_source) 2025-06-26 09:46:14,303 - oe-selftest - INFO - ---------------------------------------------------------------------- 2025-06-26 09:46:14,303 - oe-selftest - INFO - Ran 1 test in 98.415s 2025-06-26 09:46:14,303 - oe-selftest - INFO - OK 2025-06-26 09:46:18,464 - oe-selftest - INFO - RESULTS: 2025-06-26 09:46:18,464 - oe-selftest - INFO - RESULTS - sstatetests.SStatePrintdiff.test_gcc_runtime_vs_gcc_source: PASSED (89.20s) 2025-06-26 09:46:18,465 - oe-selftest - INFO - SUMMARY: 2025-06-26 09:46:18,465 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 98.418s 2025-06-26 09:46:18,466 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=1, skipped=0, failures=0, errors=0)
Quoted 1 line
Is it possible that was a temporally glitch in the CI, or related to the latest update to gcc 13.4?
It is possible! I ran the test 4 times on that first day, both with and without your patches. Failed both times with and passed both times without. And oe-selftest failed on all distros tested.
So my assumption was that it was patch related.
I added back your patches and tried again this morning. All oe-selftests passed!
So I'm going to leave them in my test queue for a few days to see if the issue reappears. If not, I'll merge them.
Steve
Quoted 44 lines
Quoted 44 lines
-----Original Message-----
From: Steve Sakoman <[email protected]>
Sent: Wednesday, 25 June 2025 17:27
To: Daniel Turull <[email protected]>
Cc: [email protected]; Mathieu Dubois-Briand <[email protected]>; Yocto TSC <[email protected]>; Richard Purdie <[email protected]>
Subject: Re: [OE-core] [scarthgap][PATCH 1/2] package: export debugsources in PKGDESTWORK as jsonQuoted 8 lines · Richard Purdie
Quoted 5 lines · Steve Sakoman
Since this is a feature addition the TSC will need to approve it before I can take this backport.
I've copied the TSC so they can add this to the agenda for their next meeting.
Quoted 7 lines
The TSC discussed this today. The change adds a single data file to packagedata based on existing information we already calculate during the build so the potential risk and impact on existing users is low.
Whilst this is a feature change, the uses for security and handling CVEs are extremely useful so on balance the TSC thinks this can be merged.
Unfortunately this patch is causing oe-selftest failures on the autobuilder, here's a sample log:
https://errors.yoct/
oproject.org%2FErrors%2FDetails%2F862388%2F&data=05%7C02%7Cdaniel.turu ll%40ericsson.com%7Cdefa9e6dee2e4d7b179308ddb3fcc9fd%7C92e84cebfbfd4 7abbe52080c6b87953f%7C0%7C0%7C638864620472989564%7CUnknown%7CT WFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4 zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=yfloJxqQl8r qqBSwuFZdlk62PfAx0RD%2ByfILVfmTgI8%3D&reserved=0This will need to be addressed before I can merge.
Thanks,
Steve