Skip to contentSkip to conversation

package: export debugsources in PKGDESTWORK as json

PATCH In review [email protected] 3 participants 10 messages Jun 19, 2025 – Jun 26, 2025

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 package: export debugsources in PKGDESTWORK as json

From: Daniel Turull <[email protected]>

The source information used during packaging can be use from other tasks to have more detailed information on the files used during the compilation and improve SPDX accuracy.

Source files used during compilation are store as compressed zstd json in pkgdata/debugsources/$PN-debugsources.json.zstd
Format:
{ binary1: [src1, src2, ...], binary2: [src1, src2, ...] }

I checked the sstate size, and it slightly increases using core-image-full-cmdline:

without patch: 2456792 KB sstate-cache/
with patch:    2460028 KB sstate-cache/

(4236 KB or 0.17%)

(From OE-Core rev: c507dcb8a8780a42bfe68b1ebaff0909b4236e6b) Adaptations to match spdx in scarthgap: change BP to PF

CC Mathieu Dubois-Briand <[email protected]>
CC Richard Purdie <[email protected]>
---
 meta/conf/bitbake.conf |  2 ++
 meta/lib/oe/package.py | 46 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 48 insertions(+)
meta/conf/bitbake.conf +2 -0
index 78f15b76ae..acf4e2d153 100644
@@ -989,5 +989,7 @@ oe.sstatesig.find_sstate_manifest[vardepsexclude] = "BBEXTENDCURR BBEXTENDVARIAN
 oe.utils.get_multilib_datastore[vardepsexclude] = "DEFAULTTUNE_MULTILIB_ORIGINAL OVERRIDES"
 oe.path.format_display[vardepsexclude] = "TOPDIR"
 oe.utils.get_bb_number_threads[vardepsexclude] = "BB_NUMBER_THREADS"
+oe.package.save_debugsources_info[vardepsexclude] = "BB_NUMBER_THREADS"
+oe.package.read_debugsources_info[vardepsexclude] = "BB_NUMBER_THREADS"
 oe.packagedata.emit_pkgdata[vardepsexclude] = "BB_NUMBER_THREADS"
 oe.packagedata.read_subpkgdata_extended[vardepsexclude] = "BB_NUMBER_THREADS"
meta/lib/oe/package.py +46 -0

58 lines not on this page · raw message

#1 lore raw

PATCH 2/2 spdx: add option to include only compiled sources

From: Daniel Turull <[email protected]>

When SPDX_INCLUDE_COMPILED_SOURCES is enabled, only include the source code files that are used during compilation.

It uses debugsource information generated during do_package.

This enables an external tool to use the SPDX information to disregard vulnerabilities that are not compiled.

As example, when used with the default config with linux-yocto, the spdx size is reduced from 156MB to 61MB.

Tested with bitbake world on oe-core.

(From OE-Core rev: c6a2f1fca76fae4c3ea471a0c63d0b453beea968) Adapted to existing files for create-spdx-2.2

CC Mathieu Dubois-Briand <[email protected]>
CC Joshua Watt <[email protected]>
---
 meta/classes/create-spdx-2.2.bbclass | 12 ++++++++
 meta/lib/oe/spdx.py                  | 42 ++++++++++++++++++++++++++++
 2 files changed, 54 insertions(+)
meta/classes/create-spdx-2.2.bbclass +12 -0
index ade1a04be3..1fc11ad7ac 100644
@@ -100,6 +100,9 @@ python() {
         # Transform the license array to a dictionary
         data["licenses"] = {l["licenseId"]: l for l in data["licenses"]}
         d.setVar("SPDX_LICENSE_DATA", data)
+
+    if d.getVar("SPDX_INCLUDE_COMPILED_SOURCES") == "1":
+        d.setVar("SPDX_INCLUDE_SOURCES", "1")
 }
 
 def convert_license_to_spdx(lic, document, d, existing={}):
@@ -215,6 +218,11 @@ def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *, archiv
     spdx_files = []
 
     file_counter = 1
+
+    check_compiled_sources = d.getVar("SPDX_INCLUDE_COMPILED_SOURCES") == "1"
+    if check_compiled_sources:
+        compiled_sources, types = oe.spdx.get_compiled_sources(d)
+        bb.debug(1, f"Total compiled files: {len(compiled_sources)}")
     for subdir, dirs, files in os.walk(topdir):
         dirs[:] = [d for d in dirs if d not in ignore_dirs]
         if subdir == str(topdir):
@@ -225,6 +233,10 @@ def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *, archiv
             filename = str(filepath.relative_to(topdir))
 
             if not filepath.is_symlink() and filepath.is_file():
+                # Check if file is compiled
+                if check_compiled_sources:
+                     if not oe.spdx.is_compiled_source(filename, compiled_sources, types):
+                          continue
                 spdx_file = oe.spdx.SPDXFile()
                 spdx_file.SPDXID = get_spdxid(file_counter)
                 for t in get_types(filepath):
meta/lib/oe/spdx.py +42 -0

45 lines not on this page · raw message

#2 lore raw
In reply to [email protected]

Hi Daniel,

Since this is a feature addition the TSC will need to approve it before I can take this backport.

I've copied the TSC so they can add this to the agenda for their next meeting.

Thanks,

Steve

Quoted 113 lines · Daniel Turull via lists.openembedded.org

From: Daniel Turull <[email protected]>

The source information used during packaging can be use from other tasks to have more detailed information on the files used during the compilation and improve SPDX accuracy.

Source files used during compilation are store as compressed zstd json in pkgdata/debugsources/$PN-debugsources.json.zstd
Format:
{ binary1: [src1, src2, ...], binary2: [src1, src2, ...] }

I checked the sstate size, and it slightly increases using core-image-full-cmdline:

without patch: 2456792 KB sstate-cache/
with patch:    2460028 KB sstate-cache/

(4236 KB or 0.17%)

(From OE-Core rev: c507dcb8a8780a42bfe68b1ebaff0909b4236e6b) Adaptations to match spdx in scarthgap: change BP to PF

CC Mathieu Dubois-Briand <[email protected]>
CC Richard Purdie <[email protected]>
---
 meta/conf/bitbake.conf |  2 ++
 meta/lib/oe/package.py | 46 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 48 insertions(+)
meta/conf/bitbake.conf +2 -0
index 78f15b76ae..acf4e2d153 100644
@@ -989,5 +989,7 @@ oe.sstatesig.find_sstate_manifest[vardepsexclude] = "BBEXTENDCURR BBEXTENDVARIAN
 oe.utils.get_multilib_datastore[vardepsexclude] = "DEFAULTTUNE_MULTILIB_ORIGINAL OVERRIDES"
 oe.path.format_display[vardepsexclude] = "TOPDIR"
 oe.utils.get_bb_number_threads[vardepsexclude] = "BB_NUMBER_THREADS"
+oe.package.save_debugsources_info[vardepsexclude] = "BB_NUMBER_THREADS"
+oe.package.read_debugsources_info[vardepsexclude] = "BB_NUMBER_THREADS"
 oe.packagedata.emit_pkgdata[vardepsexclude] = "BB_NUMBER_THREADS"
 oe.packagedata.read_subpkgdata_extended[vardepsexclude] = "BB_NUMBER_THREADS"
meta/lib/oe/package.py +46 -1
index af0923a63f..ba0d326781 100644
@@ -1038,6 +1038,49 @@ def copydebugsources(debugsrcdir, sources, d):
             if os.path.exists(p) and not os.listdir(p):
                 os.rmdir(p)
 
+def save_debugsources_info(debugsrcdir, sources_raw, d):
+    import json
+    import bb.compress.zstd
+    if debugsrcdir and sources_raw:
+        debugsources_file = d.expand("${PKGDESTWORK}/debugsources/${PN}-debugsources.json.zstd")
+        debugsources_dir = os.path.dirname(debugsources_file)
+        if not os.path.isdir(debugsources_dir):
+            bb.utils.mkdirhier(debugsources_dir)
+        bb.utils.remove(debugsources_file)
+
+        workdir = d.getVar("WORKDIR")
+        pn = d.getVar('PN')
+
+        # Kernel sources are in a different directory and are special case
+        # we format the sources as expected by spdx by replacing /usr/src/kernel/
+        # into BP/
+        kernel_src = d.getVar('KERNEL_SRC_PATH')
+        pf = d.getVar('PF')
+        sources_dict = {}
+        for file, src_files in sources_raw:
+            file_clean = file.replace(f"{workdir}/package/","")
+            sources_clean = [
+                src.replace(f"{debugsrcdir}/{pn}/", "")
+                if not kernel_src else src.replace(f"{kernel_src}/", f"{pf}/")
+                for src in src_files
+                if not any(keyword in src for keyword in ("<internal>", "<built-in>")) and not src.endswith("/")
+            ]
+            sources_dict[file_clean] = sorted(sources_clean)
+        num_threads = int(d.getVar("BB_NUMBER_THREADS"))
+        with bb.compress.zstd.open(debugsources_file, "wt", encoding="utf-8", num_threads=num_threads) as f:
+            json.dump(sources_dict, f, sort_keys=True)
+
+def read_debugsources_info(d):
+    import json
+    import bb.compress.zstd
+    try:
+        fn = d.expand("${PKGDESTWORK}/debugsources/${PN}-debugsources.json.zstd")
+        num_threads = int(d.getVar("BB_NUMBER_THREADS"))
+        with bb.compress.zstd.open(fn, "rt", encoding="utf-8", num_threads=num_threads) as f:
+            return json.load(f)
+    except FileNotFoundError:
+        bb.debug(1, f"File not found: {fn}")
+        return None
 
 def process_split_and_strip_files(d):
     cpath = oe.cachedpath.CachedPath()
@@ -1269,6 +1312,9 @@ def process_split_and_strip_files(d):
         # Process the dv["srcdir"] if requested...
         # This copies and places the referenced sources for later debugging...
         copydebugsources(dv["srcdir"], sources, d)
+
+        # Save source info to be accessible to other tasks
+        save_debugsources_info(dv["srcdir"], results, d)
     #
     # End of debug splitting
     #
 
-=-=-=-=-=-=-=-=-=-=-=-

Links: You receive all messages sent to this group.
View/Reply Online (#219060): https://lists.openembedded.org/g/openembedded-core/message/219060
Mute This Topic: https://lists.openembedded.org/mt/113722386/3620601
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#3 lore raw

Thanks Steve for letting me know. I understand. Please let me know if you need anything from me.

Best regards,
Daniel

Quoted 196 lines

-----Original Message-----
From: Steve Sakoman <[email protected]>
Sent: Thursday, 19 June 2025 17:24
To: Daniel Turull <[email protected]>
Cc: [email protected]; Mathieu Dubois-Briand <[email protected]>; Richard Purdie <[email protected]>; Yocto TSC <[email protected]>
Subject: Re: [OE-core] [scarthgap][PATCH 1/2] package: export debugsources in PKGDESTWORK as json

Hi Daniel,

Since this is a feature addition the TSC will need to approve it before I can take this backport.

I've copied the TSC so they can add this to the agenda for their next meeting.

Thanks,

Steve

Quoted 34 lines · Daniel Turull via lists.openembedded.org

From: Daniel Turull <[email protected]>

The source information used during packaging can be use from other tasks to have more detailed information on the files used during the compilation and improve SPDX accuracy.

Source files used during compilation are store as compressed zstd json in pkgdata/debugsources/$PN-debugsources.json.zstd
Format:
{ binary1: [src1, src2, ...], binary2: [src1, src2, ...] }

I checked the sstate size, and it slightly increases using core-image-full-cmdline:

without patch: 2456792 KB sstate-cache/
with patch:    2460028 KB sstate-cache/

(4236 KB or 0.17%)

(From OE-Core rev: c507dcb8a8780a42bfe68b1ebaff0909b4236e6b) Adaptations to match spdx in scarthgap: change BP to PF

CC Mathieu Dubois-Briand <[email protected]>
CC Richard Purdie <[email protected]>
---
 meta/conf/bitbake.conf |  2 ++
 meta/lib/oe/package.py | 46
++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 48 insertions(+)
meta/conf/bitbake.conf index renamed from meta/conf/bitbake.conf +0 -0

78f15b76ae..acf4e2d153 100644

meta/conf/bitbake.conf +0 -0
@@ -989,5 +989,7 @@ oe.sstatesig.find_sstate_manifest[vardepsexclude]
Quoted 1 line

= "BBEXTENDCURR BBEXTENDVARIAN

oe.utils.get_multilib_datastore[vardepsexclude] = "DEFAULTTUNE_MULTILIB_ORIGINAL OVERRIDES"

Quoted 3 lines

oe.path.format_display[vardepsexclude] = "TOPDIR" oe.utils.get_bb_number_threads[vardepsexclude] = "BB_NUMBER_THREADS"
+oe.package.save_debugsources_info[vardepsexclude] =

"BB_NUMBER_THREADS"

Quoted 1 line

+oe.package.read_debugsources_info[vardepsexclude] =

"BB_NUMBER_THREADS"

Quoted 2 lines

oe.packagedata.emit_pkgdata[vardepsexclude] = "BB_NUMBER_THREADS" oe.packagedata.read_subpkgdata_extended[vardepsexclude] =

"BB_NUMBER_THREADS"

meta/lib/oe/package.py def copydebugsources(debugsrcdir, sources, d): · line 1045

Quoted 13 lines
meta/lib/oe/package.py index renamed from meta/lib/oe/package.py +0 -0

af0923a63f..ba0d326781 100644

meta/lib/oe/package.py +5 -0
@@ -1038,6 +1038,49 @@ def copydebugsources(debugsrcdir, sources, d):
             if os.path.exists(p) and not os.listdir(p):
                 os.rmdir(p)
 
+def save_debugsources_info(debugsrcdir, sources_raw, d):
+    import json
+    import bb.compress.zstd
+    if debugsrcdir and sources_raw:
+        debugsources_file = d.expand("${PKGDESTWORK}/debugsources/${PN}-

debugsources.json.zstd")

Quoted 11 lines
meta/lib/oe/package.py +11 -0
+        debugsources_dir = os.path.dirname(debugsources_file)
+        if not os.path.isdir(debugsources_dir):
+            bb.utils.mkdirhier(debugsources_dir)
+        bb.utils.remove(debugsources_file)
+
+        workdir = d.getVar("WORKDIR")
+        pn = d.getVar('PN')
+
+        # Kernel sources are in a different directory and are special case
+        # we format the sources as expected by spdx by replacing /usr/src/kernel/
+        # into BP/

meta/lib/oe/package.py def copydebugsources(debugsrcdir, sources, d):

Quoted 10 lines
meta/lib/oe/package.py +10 -0
+        kernel_src = d.getVar('KERNEL_SRC_PATH')
+        pf = d.getVar('PF')
+        sources_dict = {}
+        for file, src_files in sources_raw:
+            file_clean = file.replace(f"{workdir}/package/","")
+            sources_clean = [
+                src.replace(f"{debugsrcdir}/{pn}/", "")
+                if not kernel_src else src.replace(f"{kernel_src}/", f"{pf}/")
+                for src in src_files
+                if not any(keyword in src for keyword in ("<internal>", "<built-in>"))

and not src.endswith("/")

meta/lib/oe/package.py def copydebugsources(debugsrcdir, sources, d):

Quoted 4 lines
meta/lib/oe/package.py +4 -0
+            ]
+            sources_dict[file_clean] = sorted(sources_clean)
+        num_threads = int(d.getVar("BB_NUMBER_THREADS"))
+        with bb.compress.zstd.open(debugsources_file, "wt", encoding="utf-8",

num_threads=num_threads) as f:

meta/lib/oe/package.py def copydebugsources(debugsrcdir, sources, d):

Quoted 7 lines
meta/lib/oe/package.py +7 -0
+            json.dump(sources_dict, f, sort_keys=True)
+
+def read_debugsources_info(d):
+    import json
+    import bb.compress.zstd
+    try:
+        fn = d.expand("${PKGDESTWORK}/debugsources/${PN}-

debugsources.json.zstd")

meta/lib/oe/package.py def copydebugsources(debugsrcdir, sources, d): · line 1079

Quoted 2 lines
meta/lib/oe/package.py +2 -0
+        num_threads = int(d.getVar("BB_NUMBER_THREADS"))
+        with bb.compress.zstd.open(fn, "rt", encoding="utf-8",

num_threads=num_threads) as f:

Quoted 19 lines
meta/lib/oe/package.py +4 -0
+            return json.load(f)
+    except FileNotFoundError:
+        bb.debug(1, f"File not found: {fn}")
+        return None
 
 def process_split_and_strip_files(d):
     cpath = oe.cachedpath.CachedPath() @@ -1269,6 +1312,9 @@ def

process_split_and_strip_files(d):
# Process the dv["srcdir"] if requested...
# This copies and places the referenced sources for later debugging... copydebugsources(dv["srcdir"], sources, d)

meta/lib/oe/package.py +3 -1
+
+        # Save source info to be accessible to other tasks
+        save_debugsources_info(dv["srcdir"], results, d)
     #
     # End of debug splitting
     #
 
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 4 lines

Links: You receive all messages sent to this group. View/Reply Online (#219060):
https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist
s.openembedded.org%2Fg%2Fopenembedded-

core%2Fmessage%2F219060&data=05%

Quoted 1 line

7C02%7Cdaniel.turull%40ericsson.com%7C91edce15ba0d48a507bd08ddaf45505 1

Quoted 1 line

%7C92e84cebfbfd47abbe52080c6b87953f%7C0%7C0%7C638859434418210300 %7CUnk

Quoted 1 line

nown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIl AiOiJ

Quoted 1 line

XaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Nd6J wfT8Z

Quoted 4 lines

I7PrsTzldSmYXIscMdi%2Ff6NGA6mLD3eVZ0%3D&reserved=0 Mute This Topic:
https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist

s.openembedded.org%2Fmt%2F113722386%2F3620601&data=05%7C02%7Cdan iel.tu

Quoted 1 line

rull%40ericsson.com%7C91edce15ba0d48a507bd08ddaf455051%7C92e84cebfbf d4

Quoted 1 line

7abbe52080c6b87953f%7C0%7C0%7C638859434418229205%7CUnknown%7CT WFpbGZsb

Quoted 1 line

3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIj o

Quoted 1 line

iTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=OcMMd%2Fjagn8lY2Si% 2Fd0DK

Quoted 5 lines

NQ9EcRUo75nmXST96JFT0Y%3D&reserved=0
Group Owner: [email protected] Unsubscribe:
https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist
s.openembedded.org%2Fg%2Fopenembedded-

core%2Funsub&data=05%7C02%7Cdani

Quoted 1 line

el.turull%40ericsson.com%7C91edce15ba0d48a507bd08ddaf455051%7C92e84c eb

Quoted 1 line

fbfd47abbe52080c6b87953f%7C0%7C0%7C638859434418241703%7CUnknown %7CTWFp

Quoted 1 line

bGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiI sIk

Quoted 1 line

FOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=OI7it65hjB%2BkO pZboP

Quoted 3 lines

tFKEe2Zv1cWNF9FWu5jUVsEJw%3D&reserved=0 [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#4 lore raw
In reply to Steve Sakoman
Quoted 5 lines · Steve Sakoman

Since this is a feature addition the TSC will need to approve it before I can take this backport.

I've copied the TSC so they can add this to the agenda for their next meeting.

The TSC discussed this today. The change adds a single data file to packagedata based on existing information we already calculate during the build so the potential risk and impact on existing users is low.

Whilst this is a feature change, the uses for security and handling CVEs are extremely useful so on balance the TSC thinks this can be merged.

Cheers,

Richard
(on behalf of the YP TSC)

#5 lore raw

Thanks 😊

Could I submit to scarthgap the standalone script, or people can take it from master?

e60b1759c1aea5b8f5317e46608f0a3e782ecf57 improve_kernel_cve_report: add script for postprocesing of kernel CVE data

Best regards,
Daniel

Quoted 28 lines

-----Original Message-----
From: Richard Purdie <[email protected]>
Sent: Monday, 23 June 2025 23:33
To: Steve Sakoman <[email protected]>; Daniel Turull <[email protected]>
Cc: [email protected]; Mathieu Dubois-Briand <[email protected]>; Yocto TSC <[email protected]>
Subject: Re: [OE-core] [scarthgap][PATCH 1/2] package: export debugsources in PKGDESTWORK as json

Quoted 5 lines · Steve Sakoman

Since this is a feature addition the TSC will need to approve it before I can take this backport.

I've copied the TSC so they can add this to the agenda for their next meeting.

The TSC discussed this today. The change adds a single data file to packagedata based on existing information we already calculate during the build so the potential risk and impact on existing users is low.

Whilst this is a feature change, the uses for security and handling CVEs are extremely useful so on balance the TSC thinks this can be merged.

Cheers,

Richard
(on behalf of the YP TSC)

#6 lore raw
In reply to Richard Purdie
Quoted 8 lines · Richard Purdie
Quoted 5 lines · Steve Sakoman

Since this is a feature addition the TSC will need to approve it before I can take this backport.

I've copied the TSC so they can add this to the agenda for their next meeting.

Quoted 7 lines

The TSC discussed this today. The change adds a single data file to packagedata based on existing information we already calculate during the build so the potential risk and impact on existing users is low.

Whilst this is a feature change, the uses for security and handling CVEs are extremely useful so on balance the TSC thinks this can be merged.

Unfortunately this patch is causing oe-selftest failures on the autobuilder, here's a sample log:

https://errors.yoctoproject.org/Errors/Details/862388/

This will need to be addressed before I can merge.

Thanks,

Steve

#7 lore raw

Ok.
I'll take a look and see what changed from scarthgap to master. I'll resend the patch when it is fixed. I'll test with the oe-selftest

Daniel

Quoted 45 lines

-----Original Message-----
From: Steve Sakoman <[email protected]>
Sent: Wednesday, 25 June 2025 17:27
To: Daniel Turull <[email protected]>
Cc: [email protected]; Mathieu Dubois-Briand <[email protected]>; Yocto TSC <[email protected]>; Richard Purdie <[email protected]>
Subject: Re: [OE-core] [scarthgap][PATCH 1/2] package: export debugsources in PKGDESTWORK as json

Quoted 8 lines · Richard Purdie
Quoted 5 lines · Steve Sakoman

Since this is a feature addition the TSC will need to approve it before I can take this backport.

I've copied the TSC so they can add this to the agenda for their next meeting.

Quoted 7 lines

The TSC discussed this today. The change adds a single data file to packagedata based on existing information we already calculate during the build so the potential risk and impact on existing users is low.

Whilst this is a feature change, the uses for security and handling CVEs are extremely useful so on balance the TSC thinks this can be merged.

Unfortunately this patch is causing oe-selftest failures on the autobuilder, here's a sample log:

https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ferrors.
yoctoproject.org%2FErrors%2FDetails%2F862388%2F&data=05%7C02%7Cdan iel.turull%40ericsson.com%7Cdefa9e6dee2e4d7b179308ddb3fcc9fd%7C92e8 4cebfbfd47abbe52080c6b87953f%7C0%7C0%7C638864620472989564%7CUn known%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAw MCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C &sdata=yfloJxqQl8rqqBSwuFZdlk62PfAx0RD%2ByfILVfmTgI8%3D&reserved=0

This will need to be addressed before I can merge.

Thanks,

Steve

#8 lore raw
In reply to Steve Sakoman

Hi Steve,
I looked at the log and it is a bit strange. The failure in the log seems to be related to sstate. From what I see in the log sstatetests.SStatePrintdiff.test_gcc_runtime_vs_gcc_source is failing but my patch is not touching gcc neither sstate. It is only creating a new zstd file with existing data in do_package.

2025-06-24 22:32:42,764 - oe-selftest - INFO - sstatetests.SStatePrintdiff.test_gcc_runtime_vs_gcc_source (subunit.RemotedTestCase)
2025-06-24 22:32:42,764 - oe-selftest - INFO -  ... FAIL

File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/sstatetests.py", line 884, in test_gcc_runtime_vs_gcc_source
self.run_test_printdiff_changerecipe("gcc-runtime", "gcc-source", "-c do_preconfigure {}".format(gcc_source_pn),
File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/sstatetests.py", line 818, in run_test_printdiff_changerecipe
result_sametmp = bitbake("-S printdiff {}".format(target))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/utils/commands.py", line 234, in bitbake
return runCmd(cmd, ignore_status, timeout, output_log=output_log, **options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/utils/commands.py", line 212, in runCmd
raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command 'bitbake -S printdiff gcc-runtime' returned non-zero exit status 1:

When I run the same failing test in my local machine, it passes.
oe-selftest -r sstatetests.SStatePrintdiff.test_gcc_runtime_vs_gcc_source -j 20 -T machine -T toolchain-user -T toolchain-system

2025-06-26 09:44:35,119 - oe-selftest - INFO - Adding layer libraries:
2025-06-26 09:44:35,119 - oe-selftest - INFO -  /local/x/yocto-scarthgap/meta-yocto/meta-poky/lib
2025-06-26 09:44:35,119 - oe-selftest - INFO -  /local/x/yocto-scarthgap/meta/lib
2025-06-26 09:44:35,120 - oe-selftest - INFO -  /repo/x/yocto-scarthgap/meta-yocto/meta-yocto-bsp/lib
2025-06-26 09:44:35,120 - oe-selftest - INFO -  /repo/x/yocto-scarthgap/openembedded-core/meta-skeleton/lib
2025-06-26 09:44:35,120 - oe-selftest - INFO -  /local/x/yocto-scarthgap/openembedded-core/meta-selftest/lib
2025-06-26 09:44:35,121 - oe-selftest - INFO - Checking base configuration is valid/parsable
NOTE: Starting bitbake server...
2025-06-26 09:46:05,270 - oe-selftest - INFO - sstatetests.SStatePrintdiff.test_gcc_runtime_vs_gcc_source (subunit.RemotedTestCase)
2025-06-26 09:46:05,271 - oe-selftest - INFO -  ... ok
2025-06-26 09:46:05,271 - oe-selftest - INFO - 0: 1/1 1/1 (89.19s) (0 failed) (sstatetests.SStatePrintdiff.test_gcc_runtime_vs_gcc_source)
2025-06-26 09:46:14,303 - oe-selftest - INFO - ----------------------------------------------------------------------
2025-06-26 09:46:14,303 - oe-selftest - INFO - Ran 1 test in 98.415s
2025-06-26 09:46:14,303 - oe-selftest - INFO - OK
2025-06-26 09:46:18,464 - oe-selftest - INFO - RESULTS:
2025-06-26 09:46:18,464 - oe-selftest - INFO - RESULTS - sstatetests.SStatePrintdiff.test_gcc_runtime_vs_gcc_source: PASSED (89.20s)
2025-06-26 09:46:18,465 - oe-selftest - INFO - SUMMARY:
2025-06-26 09:46:18,465 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 98.418s
2025-06-26 09:46:18,466 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=1, skipped=0, failures=0, errors=0)

Is it possible that was a temporally glitch in the CI, or related to the latest update to gcc 13.4? Best regards,
Daniel

Quoted 44 lines

-----Original Message-----
From: Steve Sakoman <[email protected]>
Sent: Wednesday, 25 June 2025 17:27
To: Daniel Turull <[email protected]>
Cc: [email protected]; Mathieu Dubois-Briand <[email protected]>; Yocto TSC <[email protected]>; Richard Purdie <[email protected]>
Subject: Re: [OE-core] [scarthgap][PATCH 1/2] package: export debugsources in PKGDESTWORK as json

Quoted 8 lines · Richard Purdie
Quoted 5 lines · Steve Sakoman

Since this is a feature addition the TSC will need to approve it before I can take this backport.

I've copied the TSC so they can add this to the agenda for their next meeting.

Quoted 7 lines

The TSC discussed this today. The change adds a single data file to packagedata based on existing information we already calculate during the build so the potential risk and impact on existing users is low.

Whilst this is a feature change, the uses for security and handling CVEs are extremely useful so on balance the TSC thinks this can be merged.

Unfortunately this patch is causing oe-selftest failures on the autobuilder, here's a sample log:

https://errors.yoct/
oproject.org%2FErrors%2FDetails%2F862388%2F&data=05%7C02%7Cdaniel.turu ll%40ericsson.com%7Cdefa9e6dee2e4d7b179308ddb3fcc9fd%7C92e84cebfbfd4 7abbe52080c6b87953f%7C0%7C0%7C638864620472989564%7CUnknown%7CT WFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4 zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=yfloJxqQl8r qqBSwuFZdlk62PfAx0RD%2ByfILVfmTgI8%3D&reserved=0

This will need to be addressed before I can merge.

Thanks,

Steve

#9 lore raw

Hi Daniel,

Quoted 43 lines · Daniel Turull

Hi Steve,
I looked at the log and it is a bit strange. The failure in the log seems to be related to sstate. From what I see in the log sstatetests.SStatePrintdiff.test_gcc_runtime_vs_gcc_source is failing but my patch is not touching gcc neither sstate. It is only creating a new zstd file with existing data in do_package.

2025-06-24 22:32:42,764 - oe-selftest - INFO - sstatetests.SStatePrintdiff.test_gcc_runtime_vs_gcc_source (subunit.RemotedTestCase)
2025-06-24 22:32:42,764 - oe-selftest - INFO -  ... FAIL

File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/sstatetests.py", line 884, in test_gcc_runtime_vs_gcc_source
self.run_test_printdiff_changerecipe("gcc-runtime", "gcc-source", "-c do_preconfigure {}".format(gcc_source_pn),
File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/sstatetests.py", line 818, in run_test_printdiff_changerecipe
result_sametmp = bitbake("-S printdiff {}".format(target))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/utils/commands.py", line 234, in bitbake
return runCmd(cmd, ignore_status, timeout, output_log=output_log, **options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/utils/commands.py", line 212, in runCmd
raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command 'bitbake -S printdiff gcc-runtime' returned non-zero exit status 1:

When I run the same failing test in my local machine, it passes.
oe-selftest -r sstatetests.SStatePrintdiff.test_gcc_runtime_vs_gcc_source -j 20 -T machine -T toolchain-user -T toolchain-system

2025-06-26 09:44:35,119 - oe-selftest - INFO - Adding layer libraries:
2025-06-26 09:44:35,119 - oe-selftest - INFO -  /local/x/yocto-scarthgap/meta-yocto/meta-poky/lib
2025-06-26 09:44:35,119 - oe-selftest - INFO -  /local/x/yocto-scarthgap/meta/lib
2025-06-26 09:44:35,120 - oe-selftest - INFO -  /repo/x/yocto-scarthgap/meta-yocto/meta-yocto-bsp/lib
2025-06-26 09:44:35,120 - oe-selftest - INFO -  /repo/x/yocto-scarthgap/openembedded-core/meta-skeleton/lib
2025-06-26 09:44:35,120 - oe-selftest - INFO -  /local/x/yocto-scarthgap/openembedded-core/meta-selftest/lib
2025-06-26 09:44:35,121 - oe-selftest - INFO - Checking base configuration is valid/parsable
NOTE: Starting bitbake server...
2025-06-26 09:46:05,270 - oe-selftest - INFO - sstatetests.SStatePrintdiff.test_gcc_runtime_vs_gcc_source (subunit.RemotedTestCase)
2025-06-26 09:46:05,271 - oe-selftest - INFO -  ... ok
2025-06-26 09:46:05,271 - oe-selftest - INFO - 0: 1/1 1/1 (89.19s) (0 failed) (sstatetests.SStatePrintdiff.test_gcc_runtime_vs_gcc_source)
2025-06-26 09:46:14,303 - oe-selftest - INFO - ----------------------------------------------------------------------
2025-06-26 09:46:14,303 - oe-selftest - INFO - Ran 1 test in 98.415s
2025-06-26 09:46:14,303 - oe-selftest - INFO - OK
2025-06-26 09:46:18,464 - oe-selftest - INFO - RESULTS:
2025-06-26 09:46:18,464 - oe-selftest - INFO - RESULTS - sstatetests.SStatePrintdiff.test_gcc_runtime_vs_gcc_source: PASSED (89.20s)
2025-06-26 09:46:18,465 - oe-selftest - INFO - SUMMARY:
2025-06-26 09:46:18,465 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 98.418s
2025-06-26 09:46:18,466 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=1, skipped=0, failures=0, errors=0)
Quoted 1 line

Is it possible that was a temporally glitch in the CI, or related to the latest update to gcc 13.4?

It is possible! I ran the test 4 times on that first day, both with and without your patches. Failed both times with and passed both times without. And oe-selftest failed on all distros tested.

So my assumption was that it was patch related.

I added back your patches and tried again this morning. All oe-selftests passed!

So I'm going to leave them in my test queue for a few days to see if the issue reappears. If not, I'll merge them.

Steve

Quoted 44 lines
Quoted 44 lines

-----Original Message-----
From: Steve Sakoman <[email protected]>
Sent: Wednesday, 25 June 2025 17:27
To: Daniel Turull <[email protected]>
Cc: [email protected]; Mathieu Dubois-Briand <[email protected]>; Yocto TSC <[email protected]>; Richard Purdie <[email protected]>
Subject: Re: [OE-core] [scarthgap][PATCH 1/2] package: export debugsources in PKGDESTWORK as json

Quoted 8 lines · Richard Purdie
Quoted 5 lines · Steve Sakoman

Since this is a feature addition the TSC will need to approve it before I can take this backport.

I've copied the TSC so they can add this to the agenda for their next meeting.

Quoted 7 lines

The TSC discussed this today. The change adds a single data file to packagedata based on existing information we already calculate during the build so the potential risk and impact on existing users is low.

Whilst this is a feature change, the uses for security and handling CVEs are extremely useful so on balance the TSC thinks this can be merged.

Unfortunately this patch is causing oe-selftest failures on the autobuilder, here's a sample log:

https://errors.yoct/
oproject.org%2FErrors%2FDetails%2F862388%2F&data=05%7C02%7Cdaniel.turu ll%40ericsson.com%7Cdefa9e6dee2e4d7b179308ddb3fcc9fd%7C92e84cebfbfd4 7abbe52080c6b87953f%7C0%7C0%7C638864620472989564%7CUnknown%7CT WFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4 zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=yfloJxqQl8r qqBSwuFZdlk62PfAx0RD%2ByfILVfmTgI8%3D&reserved=0

This will need to be addressed before I can merge.

Thanks,

Steve