Skip to contentSkip to conversation

lib: Fix dependencies on SPDX code

PATCH Superseded Joshua Watt 2 participants 3 messages Mar 7, 2025 – Mar 11, 2025

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 lib: Fix dependencies on SPDX code

The SPDX library code was being ignored from taskhash calculations due to accidentally being omitted from BBIMPORTS. This meant that changes in the code or dependent variables would not cause the task to rebuild correctly.

In order to add spdx_common, convert the Dep object from a named tuple to a frozen dataclass. These function more or less equivalently, but the bitbake code parser cannot handle named tuples.

---
 meta/lib/oe/__init__.py    | 2 +-
 meta/lib/oe/spdx_common.py | 8 ++++++--
 2 files changed, 7 insertions(+), 3 deletions(-)
meta/lib/oe/__init__.py +1 -1
index d7604812834..3179a3f3d2c 100644
@@ -11,4 +11,4 @@ __path__ = extend_path(__path__, __name__)
 # processed correctly (e.g. qa)
 BBIMPORTS = ["qa", "data", "path", "utils", "types", "package", "packagedata", \
              "packagegroup", "sstatesig", "lsb", "cachedpath", "license", \
-             "reproducible", "rust", "buildcfg", "go"]
+             "reproducible", "rust", "buildcfg", "go", "spdx30_tasks", "spdx_common"]
meta/lib/oe/spdx_common.py +6 -2
index 23a17271d6c..e1b26edaaf6 100644
@@ -12,7 +12,7 @@ import re
 import shutil
 
 from pathlib import Path
-
+from dataclasses import dataclass
 
 LIC_REGEX = re.compile(
     rb"^\W*SPDX-License-Identifier:\s*([ \w\d.()+-]+?)(?:\s+\W*)?$",
@@ -77,7 +77,11 @@ def process_sources(d):
     return True
 
 
-Dep = collections.namedtuple("Dep", ["pn", "hashfn", "in_taskhash"])
+@dataclass(frozen=True)
+class Dep(object):
+    pn: str
+    hashfn: str
+    in_taskhash: bool
 
 
 def collect_direct_deps(d, dep_task):
Signature · 1 line
2.47.1
#1 lore raw
Quoted 11 lines · Joshua Watt via lists.openembedded.org

The SPDX library code was being ignored from taskhash calculations due to accidentally being omitted from BBIMPORTS. This meant that changes in the code or dependent variables would not cause the task to rebuild correctly.

In order to add spdx_common, convert the Dep object from a named tuple to a frozen dataclass. These function more or less equivalently, but the bitbake code parser cannot handle named tuples.

---

Hi Joshua,

I believe this is triggering some issue on the autobuilder:

2025-03-10 09:45:19,401 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/srv/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/selftest/cases/sstatetests.py", line 482, in test_sstate_allarch_samesigs
    self.sstate_common_samesigs(configA, configB, allarch=True)
  File "/srv/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/selftest/cases/sstatetests.py", line 213, in sstate_common_samesigs
    self.assertEqual(files1, files2)
  File "/usr/lib/python3.12/unittest/case.py", line 885, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib/python3.12/unittest/case.py", line 1184, in assertDictEqual
    self.fail(self._formatMessage(msg, standardMsg))
  File "/usr/lib/python3.12/unittest/case.py", line 715, in fail
    raise self.failureException(msg)

AssertionError: {'nat[215727 chars]x': 'd8ed538990588f9d37804e867be98c9b4809de8f2[88942 chars]d1a'} != {'nat[215727 chars]x': '481bab8ce0c89a635d3ec3f3c4a28c0b5a67b2c23[88942 chars]d1a'}

https://autobuilder.yoctoproject.org/valkyrie/api/v2/logs/1715394/raw_inline

Can you have a look at this error please?

Signature · 3 lines
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#2 lore raw

PATCH v2 1/1 lib: Fix dependencies on SPDX code

In reply to Joshua Watt

The SPDX library code was being ignored from taskhash calculations due to accidentally being omitted from BBIMPORTS. This meant that changes in the code or dependent variables would not cause the task to rebuild correctly.

In order to add spdx_common, convert the Dep object from a named tuple to a frozen dataclass. These function more or less equivalently, but the bitbake code parser cannot handle named tuples.

Finally, the vardepsexclude that used to be present on the recipe tasks needs to be moved to the python code in order for the variables to be correctly ignored. Several unused exclusions were removed

---
 meta/classes/create-spdx-3.0.bbclass | 3 +--
 meta/lib/oe/__init__.py              | 2 +-
 meta/lib/oe/spdx_common.py           | 8 ++++++--
 3 files changed, 8 insertions(+), 5 deletions(-)
meta/classes/create-spdx-3.0.bbclass +1 -2
index 25f3aa5f433..b4a5156e709 100644
@@ -134,7 +134,6 @@ python do_create_spdx() {
     import oe.spdx30_tasks
     oe.spdx30_tasks.create_spdx(d)
 }
-do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS SPDX_BUILD_HOST"
 do_create_spdx[vardeps] += "\
     SPDX_INCLUDE_BITBAKE_PARENT_BUILD \
     SPDX_PACKAGE_ADDITIONAL_PURPOSE \
@@ -170,7 +169,7 @@ python do_create_package_spdx() {
     import oe.spdx30_tasks
     oe.spdx30_tasks.create_package_spdx(d)
 }
-do_create_package_spdx[vardepsexclude] += "OVERRIDES SPDX_MULTILIB_SSTATE_ARCHS"
+oe.spdx30_tasks.create_package_spdx[vardepsexclude] = "OVERRIDES"
 
 addtask do_create_package_spdx after do_create_spdx before do_build do_rm_work
 SSTATETASKS += "do_create_package_spdx"
meta/lib/oe/__init__.py +1 -1
index d7604812834..3179a3f3d2c 100644
@@ -11,4 +11,4 @@ __path__ = extend_path(__path__, __name__)
 # processed correctly (e.g. qa)
 BBIMPORTS = ["qa", "data", "path", "utils", "types", "package", "packagedata", \
              "packagegroup", "sstatesig", "lsb", "cachedpath", "license", \
-             "reproducible", "rust", "buildcfg", "go"]
+             "reproducible", "rust", "buildcfg", "go", "spdx30_tasks", "spdx_common"]
meta/lib/oe/spdx_common.py +6 -2
index 23a17271d6c..e1b26edaaf6 100644
@@ -12,7 +12,7 @@ import re
 import shutil
 
 from pathlib import Path
-
+from dataclasses import dataclass
 
 LIC_REGEX = re.compile(
     rb"^\W*SPDX-License-Identifier:\s*([ \w\d.()+-]+?)(?:\s+\W*)?$",
@@ -77,7 +77,11 @@ def process_sources(d):
     return True
 
 
-Dep = collections.namedtuple("Dep", ["pn", "hashfn", "in_taskhash"])
+@dataclass(frozen=True)
+class Dep(object):
+    pn: str
+    hashfn: str
+    in_taskhash: bool
 
 
 def collect_direct_deps(d, dep_task):
Signature · 1 line
2.47.1