Skip to contentSkip to conversation

Add missing SRCREV check for SCM URIs

PATCH Superseded Sai Sneha 2 participants 6 messages May 19 – May 20

v2 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Sai Sneha

author
#0 lore raw

v2 cover Add missing SRCREV check for SCM URIs

This patch series adds a check for git:// (and other SCM) URIs in SRC_URI that are missing a corresponding SRCREV, addressing Bugzilla #16051 [1] and building on Corentin Guillevic's RFC series [2].

== Problem ==

A recipe with a git:// URI and no SRCREV causes two problems:

1. Under BB_NO_NETWORK=1, parsing fails with a FetchError that halts parsing of ALL recipes, not just the offending one. This breaks CI pipelines that use "bitbake --parse-only" as a sanity check, which was the original motivation reported by Ross Burton [3].

2. Without BB_NO_NETWORK, BitBake silently queries the remote repository on every parse, breaking reproducibility.

== Why the trivial fix doesn't work ==

The obvious fix of checking SRCREV in insane.bbclass do_recipe_qa was attempted by Corentin Guillevic [2] but fails because fetcher_hashes_dummyfunc[vardepvalue] expands SRCREV at parse time BEFORE QA checks run. This triggers get_autorev() which causes:

"AUTOREV/SRCPV set too late for the fetcher to work properly"

This was confirmed by Mathieu Dubois-Briand's autobuilder testing [2]. Corentin's series needed 4 patches including devtool/recipetool workarounds to paper over the expansion crash.

== This series' approach ==

We intercept at the vardepvalue expansion point itself in base.bbclass:

  fetcher_hashes_dummyfunc[vardepvalue] = \
    "${@bb.fetch.get_hashvalue(d) if check_srcrev_set(d) else ''}"

check_srcrev_set() uses d.getVar(candidate, False) — the False flag prevents expansion entirely, so get_autorev() is never triggered and no network access occurs. This solves the root cause in one patch without needing devtool/recipetool changes.

== Design decisions ==

1. Why not use srcrev_internal_helper()?

srcrev_internal_helper() expands SRCREV internally, which triggers get_autorev() and live git ls-remote calls — exactly what we are trying to prevent. We intentionally use d.getVar(candidate, False) instead. Our fallback chain mirrors BitBake's resolution order exactly, as confirmed by BitBake's own error messages:
SRCREV_default:pn-X, SRCREV_default, SRCREV:pn-X, SRCREV

2. Why duplicate warnings at parse time AND QA time?

Parse-time warnings (base.bbclass) are necessary for CI pipelines running "bitbake --parse-only" with BB_NO_NETWORK=1 — they would miss the issue entirely if only QA-time warnings existed. QA-time warnings (insane.bbclass) are necessary for proper WARN_QA/ERROR_QA layer control and QA framework integration. This pattern is already established — src-uri-bad fires at both parse and QA time.

3. Why hardcode git/gitsm/hg/svn schemes?

These are the fetchers that require SRCREV. This is a known limitation — if new SCM fetchers are added in future, this list will need updating. Noted as a follow-on improvement.

== Enforcement levels ==

  Normal builds, all layers  →  WARN_QA   (warning, build continues)
  oe-core layer              →  ERROR_QA  (error, strict enforcement)
  yocto-check-layer runs     →  ERROR     (layer validation fails)

This matches the graduated enforcement requested in Bug 16051 Comments 2 and 3.

== Testing ==

Tested on openembedded-core with:

- Recipe with missing SRCREV → WARNING/ERROR at parse time ✓
- Recipe with SRCREV = "${AUTOREV}" → silent passthrough ✓
- Recipe with rev= in URI → no false positive ✓
- Recipe with multiple SCM URIs → all missing SRCREVs reported ✓
- oe-core recipe → ERROR instead of WARNING ✓
- yocto-check-layer on layer with missing SRCREV → FAIL ✓
- 952 other recipes parse cleanly → 0 errors ✓

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=16051
[2] https://lore.kernel.org/openembedded-core/[email protected]/
[3] https://lists.openembedded.org/g/openembedded-devel/topic/115911777

Sai Sneha (3):
base.bbclass: warn when SRCREV is missing for SCM URIs at parse time insane.bbclass: add missing-srcrev QA check for SCM URIs yocto-check-layer.bbclass: enforce missing SRCREV check during layer validation

 meta/classes-global/base.bbclass             | 40 +++++++++++++++++++
 meta/classes-global/insane.bbclass           | 32 +++++++++++++++
 meta/classes-global/yocto-check-layer.bbclass| 32 +++++++++++++++
 3 files changed, 104 insertions(+), 2 deletions(-)
Signature · 4 lines
Note: This patch series was developed with assistance from Anthropic
Claude (AI). All code has been reviewed, tested, and verified by the
contributor. The Signed-off-by line represents the contributor's own
certification per the Developer's Certificate of Origin.
author
#1 lore raw

PATCH v3 1/3 base.bbclass: warn when SRCREV is missing for SCM URIs at parse time

A recipe with a git:// (or other SCM) URI in SRC_URI must have a corresponding SRCREV set. Without it, BitBake performs a live query on the remote repository at every parse, breaking reproducibility and causing parse failures under BB_NO_NETWORK=1.

The trivial fix of checking SRCREV in insane.bbclass do_recipe_qa fails because fetcher_hashes_dummyfunc[vardepvalue] expands SRCREV at parse time before QA checks run, causing a FetchError that halts parsing entirely. This was identified by Corentin Guillevic's RFC series and confirmed by Mathieu Dubois-Briand's autobuilder testing.

The fix intercepts at the vardepvalue expansion point by introducing check_srcrev_set() which inspects SRC_URI and SRCREV using unexpanded getVar() calls (False flag). This avoids triggering get_autorev() or any live network fetch. The function is called conditionally:

  fetcher_hashes_dummyfunc[vardepvalue] =     "${@bb.fetch.get_hashvalue(d) if check_srcrev_set(d) else ''}"

Note on design choice: we intentionally use d.getVar(candidate, False) rather than srcrev_internal_helper() because the latter expands SRCREV, triggering get_autorev() and live network fetches -- the exact problem this patch fixes. Our fallback chain mirrors BitBake's internal resolution order exactly, as confirmed by the error messages BitBake itself produces:
SRCREV_default:pn-X, SRCREV_default, SRCREV:pn-X, SRCREV

The check handles:
- Full SRCREV resolution fallback chain matching BitBake's behavior
- Multiple SCM URIs in one recipe (all reported, not just the first)
- Inline rev= parameter in URI (skipped, no SRCREV needed)
- AUTOREV passthrough (legitimate devtool usage, let through)
- Parse-time severity respecting WARN_QA/ERROR_QA layer config

Note: parse-time warnings intentionally duplicate the QA-time warnings added in insane.bbclass. This is necessary because CI pipelines running 'bitbake --parse-only' with BB_NO_NETWORK=1 (the original motivation from Ross Burton) would miss the issue entirely if only QA-time warnings existed. This pattern is already established in the codebase (e.g. src-uri-bad fires at both parse and QA time).

AI-Generated: Developed with assistance from Anthropic Claude

---
 meta/classes-global/base.bbclass | 40 ++++++++++++++++++++++++++++++--
 1 file changed, 38 insertions(+), 2 deletions(-)
meta/classes-global/base.bbclass +38 -2
index 62f2814bb7..66b2e25f71 100644
@@ -165,13 +165,49 @@ def setup_hosttools_dir(dest, toolsvar, d, fatal=True):
     if notfound and fatal:
         bb.fatal("The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed:\n  %s" % " ".join(notfound))
 
+def check_srcrev_set(d):
+    import bb.fetch2
+    src_uri = (d.getVar('SRC_URI', False) or '').split()
+    pn = d.getVar('PN')
+    found_issue = False
+    for uri in src_uri:
+        try:
+            (scheme, _, _, _, _, params) = bb.fetch2.decodeurl(uri)
+        except Exception:
+            continue
+        if scheme not in ('git', 'gitsm', 'hg', 'svn'):
+            continue
+        if params.get('rev', ''):
+            continue
+        name = params.get('name', '') or 'default'
+        candidates = [
+            'SRCREV_%s:pn-%s' % (name, pn),
+            'SRCREV_%s' % name,
+            'SRCREV:pn-%s' % pn,
+            'SRCREV',
+        ]
+        raw = None
+        for candidate in candidates:
+            raw = d.getVar(candidate, False)
+            if raw:
+                break
+        if not raw or raw == 'INVALID':
+            found_issue = True
+            if bb.utils.contains('ERROR_QA', 'missing-srcrev', True, False, d):
+                bb.error("%s: %s not set for %s" % (pn, candidates[-1], uri))
+            elif bb.utils.contains('WARN_QA', 'missing-srcrev', True, False, d):
+                bb.warn("%s: %s not set for %s" % (pn, candidates[-1], uri))
+        elif '${AUTOREV}' in raw:
+            return True
+    return not found_issue
+        
+
 # We can't use vardepvalue against do_fetch directly since that would overwrite
 # the other task dependencies so we use an indirect function.
 python fetcher_hashes_dummyfunc() {
     return
 }
-fetcher_hashes_dummyfunc[vardepvalue] = "${@bb.fetch.get_hashvalue(d)}"
-
+fetcher_hashes_dummyfunc[vardepvalue] = "${@bb.fetch.get_hashvalue(d) if check_srcrev_set(d) else ''}"
 addtask fetch
 do_fetch[dirs] = "${DL_DIR}"
 do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}"
Signature · 1 line
2.34.1
author
#2 lore raw

PATCH v3 2/3 insane.bbclass: add missing-srcrev QA check for SCM URIs

In reply to Sai Sneha

Add a proper QA check test_missing_srcrev() in do_recipe_qa that fires at build time via oe.qa.handle_error(), complementing the parse-time check in base.bbclass.

The check uses unexpanded getVar() calls (False flag) to safely inspect SRCREV without triggering AUTOREV expansion, and mirrors BitBake's full SRCREV resolution fallback chain.

Severity is controlled per layer:
- WARN_QA:append = " missing-srcrev" for all layers (warning by default)
- ERROR_QA:append:layer-core = " missing-srcrev" for oe-core (strict)

This follows the same pattern as missing-metadata and missing-maintainer which are warnings globally but errors for oe-core.

AI-Generated: Developed with assistance from Anthropic Claude

---
 meta/classes-global/insane.bbclass | 32 ++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
meta/classes-global/insane.bbclass +32 -0
index 04700be71c..296cfba0fe 100644
@@ -49,6 +49,9 @@ ERROR_QA ?= "\
 ERROR_QA:append = " ${@bb.utils.filter('DISTRO_FEATURES', 'usrmerge', d)}"
 WARN_QA:append:layer-core = " missing-metadata missing-maintainer"
 
+WARN_QA:append = " missing-srcrev"
+ERROR_QA:append:layer-core = " missing-srcrev"
+
 FAKEROOT_QA = "host-user-contaminated"
 FAKEROOT_QA[doc] = "QA tests which need to run under fakeroot. If any \
 enabled tests are listed here, the do_package_qa task will run under fakeroot."
@@ -1455,6 +1458,34 @@ python do_qa_unpack() {
 python do_recipe_qa() {
     import re
 
+    def test_missing_srcrev(pn, d):
+        import bb.fetch2
+        src_uri = (d.getVar('SRC_URI', False) or '').split()
+        for uri in src_uri:
+            try:
+                (scheme, _, _, _, _, params) = bb.fetch2.decodeurl(uri)
+            except Exception:
+                continue
+            if scheme not in ('git', 'gitsm', 'hg', 'svn'):
+                continue
+            if params.get('rev', ''):
+                continue
+            name = params.get('name', '') or 'default'
+            candidates = [
+                'SRCREV_%s:pn-%s' % (name, pn),
+                'SRCREV_%s' % name,
+                'SRCREV:pn-%s' % pn,
+                'SRCREV',
+            ]
+            raw = None
+            for candidate in candidates:
+                raw = d.getVar(candidate, False)
+                if raw:
+                    break
+            if not raw or raw == 'INVALID':
+                oe.qa.handle_error("missing-srcrev",
+                    "%s: %s not set for SCM URI %s" % (pn, candidates[-1], uri), d)
+
     def test_naming(pn, d):
         if pn.endswith("-native") and not bb.data.inherits_class("native", d):
             oe.qa.handle_error("recipe-naming", "Recipe %s appears native but is not, should inherit native" % pn, d)
@@ -1497,6 +1528,7 @@ python do_recipe_qa() {
 
     pn = d.getVar('PN')
     test_naming(pn, d)
+    test_missing_srcrev(pn, d)
     test_missing_metadata(pn, d)
     test_missing_maintainer(pn, d)
     test_srcuri(pn, d)
Signature · 1 line
2.34.1
author
#3 lore raw

PATCH v3 3/3 yocto-check-layer.bbclass: enforce missing SRCREV check during layer validation

In reply to Sai Sneha

Add check_missing_srcrev() to yocto-check-layer.bbclass to enforce the missing-srcrev check as an error during yocto-check-layer runs.

This ensures layer maintainers submitting layers to the Yocto ecosystem must fix all missing SRCREVs before their layer passes validation, even if their layer only gets WARN_QA during normal builds.

This satisfies the requirement from Bug 16051 Comment 2:
'add a QA check outside of checklayer then add it to checklayer'

AI-Generated: Developed with assistance from Anthropic Claude

---
 meta/classes-global/yocto-check-layer.bbclass | 32 +++++++++++++++++++
 1 file changed, 32 insertions(+)
meta/classes-global/yocto-check-layer.bbclass +32 -0
index ba93085325..1ef5424083 100644
@@ -56,7 +56,39 @@ def check_network_flag(d):
         if network and not is_allowed(bpn, task):
             bb.error(f"QA Issue: task {task} has network enabled")
 
+def check_missing_srcrev(d):
+    import bb.fetch2
+    src_uri = (d.getVar('SRC_URI', False) or '').split()
+    pn = d.getVar('PN')
+    for uri in src_uri:
+        try:
+            (scheme, _, _, _, _, params) = bb.fetch2.decodeurl(uri)
+        except Exception:
+            continue
+        if scheme not in ('git', 'gitsm', 'hg', 'svn'):
+            continue
+        if params.get('rev', ''):
+            continue
+        name = params.get('name', '') or 'default'
+        candidates = [
+            'SRCREV_%s:pn-%s' % (name, pn),
+            'SRCREV_%s' % name,
+            'SRCREV:pn-%s' % pn,
+            'SRCREV',
+        ]
+        raw = None
+        for candidate in candidates:
+            raw = d.getVar(candidate, False)
+            if raw:
+                break
+        if not raw or raw == 'INVALID':
+            bb.error("QA Issue: %s: %s not set for SCM URI %s [missing-srcrev]" % (
+                d.getVar('PN'), candidates[-1], uri))
+            d.setVar("QA_ERRORS_FOUND", "True")
+
+
 python () {
     check_insane_skip(d)
     check_network_flag(d)
+    check_missing_srcrev(d)
 }
Signature · 1 line
2.34.1
#4 lore raw
In reply to Sai Sneha

Hi Sai,

Please add a changelog to the patches, so we can follow the evolution of the patches.

And remember to update the version in the cover letter subject too.

// Martin

Quoted 128 lines · Sai Sneha via lists.openembedded.org

This patch series adds a check for git:// (and other SCM) URIs in SRC_URI that are missing a corresponding SRCREV, addressing Bugzilla #16051 [1] and building on Corentin Guillevic's RFC series [2].

== Problem ==

A recipe with a git:// URI and no SRCREV causes two problems:

1. Under BB_NO_NETWORK=1, parsing fails with a FetchError that halts parsing of ALL recipes, not just the offending one. This breaks CI pipelines that use "bitbake --parse-only" as a sanity check, which was the original motivation reported by Ross Burton [3].

2. Without BB_NO_NETWORK, BitBake silently queries the remote repository on every parse, breaking reproducibility.

== Why the trivial fix doesn't work ==

The obvious fix of checking SRCREV in insane.bbclass do_recipe_qa was attempted by Corentin Guillevic [2] but fails because fetcher_hashes_dummyfunc[vardepvalue] expands SRCREV at parse time BEFORE QA checks run. This triggers get_autorev() which causes:

  "AUTOREV/SRCPV set too late for the fetcher to work properly"

This was confirmed by Mathieu Dubois-Briand's autobuilder testing [2].
Corentin's series needed 4 patches including devtool/recipetool workarounds to paper over the expansion crash.

== This series' approach ==

We intercept at the vardepvalue expansion point itself in base.bbclass:

  fetcher_hashes_dummyfunc[vardepvalue] = \
    "${@bb.fetch.get_hashvalue(d) if check_srcrev_set(d) else ''}"

check_srcrev_set() uses d.getVar(candidate, False) — the False flag prevents expansion entirely, so get_autorev() is never triggered and no network access occurs. This solves the root cause in one patch without needing devtool/recipetool changes.

== Design decisions ==

1. Why not use srcrev_internal_helper()?

   srcrev_internal_helper() expands SRCREV internally, which triggers get_autorev() and live git ls-remote calls — exactly what we are trying to prevent. We intentionally use d.getVar(candidate, False) instead. Our fallback chain mirrors BitBake's resolution order exactly, as confirmed by BitBake's own error messages:
     SRCREV_default:pn-X, SRCREV_default, SRCREV:pn-X, SRCREV

2. Why duplicate warnings at parse time AND QA time?

   Parse-time warnings (base.bbclass) are necessary for CI pipelines running "bitbake --parse-only" with BB_NO_NETWORK=1 — they would miss the issue entirely if only QA-time warnings existed. QA-time warnings (insane.bbclass) are necessary for proper WARN_QA/ERROR_QA layer control and QA framework integration. This pattern is already established — src-uri-bad fires at both parse and QA time.

3. Why hardcode git/gitsm/hg/svn schemes?

   These are the fetchers that require SRCREV. This is a known limitation — if new SCM fetchers are added in future, this list will need updating. Noted as a follow-on improvement.

== Enforcement levels ==

  Normal builds, all layers  →  WARN_QA   (warning, build continues) oe-core layer              →  ERROR_QA  (error, strict enforcement) yocto-check-layer runs     →  ERROR     (layer validation fails)

This matches the graduated enforcement requested in Bug 16051 Comments 2 and 3.

== Testing ==

Tested on openembedded-core with:

- Recipe with missing SRCREV → WARNING/ERROR at parse time ✓
- Recipe with SRCREV = "${AUTOREV}" → silent passthrough ✓
- Recipe with rev= in URI → no false positive ✓
- Recipe with multiple SCM URIs → all missing SRCREVs reported ✓
- oe-core recipe → ERROR instead of WARNING ✓
- yocto-check-layer on layer with missing SRCREV → FAIL ✓
- 952 other recipes parse cleanly → 0 errors ✓

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=16051
[2]
https://lore.kernel.org/openembedded-core/[email protected]/
[3]
https://lists.openembedded.org/g/openembedded-devel/topic/115911777

Sai Sneha (3):
  base.bbclass: warn when SRCREV is missing for SCM URIs at parse time
  insane.bbclass: add missing-srcrev QA check for SCM URIs yocto-check-layer.bbclass: enforce missing SRCREV check during layer validation

 meta/classes-global/base.bbclass             | 40
+++++++++++++++++++
 meta/classes-global/insane.bbclass           | 32 +++++++++++++++ meta/classes-global/yocto-check-layer.bbclass| 32 +++++++++++++++

 3 files changed, 104 insertions(+), 2 deletions(-)

--
Note: This patch series was developed with assistance from Anthropic Claude (AI). All code has been reviewed, tested, and verified by the contributor. The Signed-off-by line represents the contributor's own certification per the Developer's Certificate of Origin.

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#237306):
https://lists.openembedded.org/g/openembedded-core/message/237306
Mute This Topic: https://lists.openembedded.org/mt/119387995/3618414
Group Owner: [email protected] Unsubscribe:
https://lists.openembedded.org/g/openembedded-core/unsub [
[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

author
#5 lore raw

Hi Paul, Martin,

Thank you for the detailed review and suggestions.

You are right about the duplicated logic across the three patches. I will rework the series to factor the SRCREV detection into a shared helper and integrate the yocto-check-layer enforcement through CHECKLAYER_REQUIRED_TESTS instead of reimplementing the check separately.

I will also address the style and cleanup comments:

* simplify the control flow and return logic
* improve variable naming
* use f-strings where appropriate
* add changelogs for the next revision
* clarify the wording around Corentin's earlier work

I’ll send an updated revision shortly.

Thanks,
Sai Sneha