Add missing SRCREV check for SCM URIs
v5 cover Add missing SRCREV check for SCM URIs
This patch series adds a check for git:// (and other SCM) URIs in SRC_URI that are missing a corresponding SRCREV, addressing Bugzilla #16051 [1] and referencing Corentin Guillevic's RFC series [2].
== Changes since v4 ==
- Add tag= to skip condition in oe.qa.check_uri_srcrev() alongside the existing rev= handling. This fixes a selftest breakage (test_git_unpack_nonetwork) reported by Mathieu Dubois-Briand where recipes using tag= in SRC_URI intentionally leave SRCREV empty.
- Update docstring to mention tag= parameter
- Verified by running full bbtests.BitbakeTests selftest suite:
32/33 passed, 1 failure (test_image_manifest) is unrelated as it requires building a full image and fails due to network constraints
== Note on CHECKLAYER_REQUIRED_TESTS ==
Paul suggested using CHECKLAYER_REQUIRED_TESTS instead of Patch 4. We tested this but found it promotes missing-srcrev to ERROR_QA for ALL layers globally during normal builds, because CHECKLAYER_REQUIRED_TESTS feeds directly into ERROR_QA. This contradicts Bug 16051 Comment 3 which requests keeping it as a warning for external layers by default. The direct implementation in yocto-check-layer.bbclass enforces it strictly only during checklayer validation without affecting normal builds.
== Note on candidates list order ==
Martin suggested reversing the candidates list so plain SRCREV is tested first. We kept most-specific-first order because it mirrors BitBake internal srcrev_internal_helper() resolution order exactly, as confirmed by BitBake own error messages:
SRCREV_default:pn-X, SRCREV_default, SRCREV:pn-X, SRCREV
== Problem ==
A recipe with a git:// URI and no SRCREV causes two problems:
1. Under BB_NO_NETWORK=1, parsing fails with a FetchError that halts parsing of ALL recipes. This breaks CI pipelines that use bitbake --parse-only as a sanity check.
2. Without BB_NO_NETWORK, BitBake silently queries the remote repository on every parse, breaking reproducibility.
== Why the trivial fix does not work ==
The obvious fix of checking SRCREV in insane.bbclass do_recipe_qa was attempted by Corentin Guillevic [2] but fails because fetcher_hashes_dummyfunc[vardepvalue] expands SRCREV at parse time BEFORE QA checks run. Corentin patches 1 and 2 add the QA check, while patches 3 and 4 ensure devtool and recipetool only use AUTOREV when appropriate.
== This series approach ==
We intercept at the vardepvalue expansion point in base.bbclass using a shared helper oe.qa.check_uri_srcrev() that uses d.getVar(candidate, False) preventing expansion entirely.
== Design decisions ==
1. Why not use srcrev_internal_helper()
It expands SRCREV internally, triggering get_autorev() and live network fetches. Our fallback chain mirrors BitBake resolution order exactly without expansion.
2. Why duplicate warnings at parse time AND QA time
Parse-time catches CI parse-only pipelines. QA-time gives proper layer control. This pattern is established - src-uri-bad does same.
3. Why hardcode git/gitsm/hg/svn schemes
Known limitation, noted as follow-on improvement.
== Enforcement levels ==
Normal builds, all layers -> WARN_QA oe-core layer -> ERROR_QA yocto-check-layer runs -> ERROR
== Testing ==
- Recipe with missing SRCREV -> WARNING/ERROR at parse time
- Recipe with SRCREV = AUTOREV -> silent passthrough
- Recipe with rev= in URI -> no false positive
- Recipe with multiple SCM URIs -> all missing SRCREVs reported
- oe-core recipe -> ERROR instead of WARNING
- yocto-check-layer on layer with missing SRCREV -> FAIL
- 952 other recipes parse cleanly -> 0 errors
[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=16051
[2] https://lore.kernel.org/openembedded-core/[email protected]/
[3] https://lists.openembedded.org/g/openembedded-devel/topic/115911777
Sai Sneha (4):
oe/qa.py: add check_uri_srcrev() helper for SCM URI SRCREV validation base.bbclass: warn when SRCREV is missing for SCM URIs at parse time insane.bbclass: add missing-srcrev QA check for SCM URIs yocto-check-layer.bbclass: enforce missing SRCREV check during layer validation
meta/classes-global/base.bbclass | 28 ++++++++++ meta/classes-global/insane.bbclass | 13 +++++ meta/classes-global/yocto-check-layer.bbclass | 15 ++++++ meta/lib/oe/qa.py | 30 +++++++++++ 4 files changed, 86 insertions(+), 2 deletions(-)
--
Note: This patch series was developed with assistance from Anthropic Claude (AI). All code has been reviewed, tested, and verified by the contributor. The Signed-off-by line represents the contributor own certification per the Developer Certificate of Origin.
PATCH v5 1/4 oe/qa.py: add check_uri_srcrev() helper for SCM URI SRCREV validation
Add a shared helper function check_uri_srcrev() that checks whether a single SCM URI has a valid SRCREV set in the datastore.
The function uses unexpanded getVar() calls (False flag) to safely inspect SRCREV without triggering AUTOREV expansion or live network fetches. It mirrors BitBake's internal SRCREV resolution fallback chain exactly:
SRCREV_<name>:pn-<recipe>, SRCREV_<name>, SRCREV:pn-<recipe>, SRCREV
Returns:
- The revision string if a valid SRCREV is found
- '' for non-SCM URIs or URIs with inline rev= or tag= parameter (skip)
- None if SRCREV is missing or INVALID
This shared helper is used by base.bbclass, insane.bbclass and yocto-check-layer.bbclass to avoid duplicating the same logic.
AI-Generated: Developed with assistance from Anthropic Claude
---
Changes in v5:
- Add tag= to skip condition alongside rev= to fix selftest breakage (test_git_unpack_nonetwork) reported by Mathieu Dubois-Briand
- Update docstring to mention tag= parameter
Changes in v4:
- New patch: factored common SRCREV checking logic into shared helper
- Eliminates code duplication across base.bbclass, insane.bbclass and yocto-check-layer.bbclass
- Uses for-else pattern, f-strings, simplified name lookup
- Adds docstring explaining return values
Changes in v3:
- Added AI-Generated disclosure and Reported-by tag
Changes in v2:
- Initial public submission
meta/lib/oe/qa.py | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+)
meta/lib/oe/qa.py +30 -0
@@ -240,6 +240,36 @@ def check_upstream_status(fullpath): else: return "Missing Upstream-Status in patch\n%s\nPlease add according to %s ." % (fullpath, guidelines) +def check_uri_srcrev(pn, uri, d): + """ + Check that a single SCM URI has a valid SRCREV set. + + Returns the resolved revision string if valid (including '${AUTOREV}'). + Returns '' for non-SCM URIs or URIs with inline rev= parameter (skip). + Returns None if SRCREV is missing or INVALID. + """ + import bb.fetch2 + try: + (scheme, _, _, _, _, params) = bb.fetch2.decodeurl(uri) + except Exception: + return '' + if scheme not in ('git', 'gitsm', 'hg', 'svn'): + return '' + if params.get('rev', '') or params.get('tag', ''): + return '' + name = params.get('name', 'default') + candidates = [ + f'SRCREV_{name}:pn-{pn}', + f'SRCREV_{name}', + f'SRCREV:pn-{pn}', + 'SRCREV', + ] + for candidate in candidates: + rev = d.getVar(candidate, False) + if rev and rev != 'INVALID': + return rev + return None + if __name__ == "__main__": import sys
Signature · 1 line
2.34.1
Re: [PATCH v5 1/4] oe/qa.py: add check_uri_srcrev() helper for SCM URI SRCREV validation
Hello,
Quoted 8 lines · Sai Sneha
Add a shared helper function
check_uri_srcrev()that checks whether a single SCM URI has a valid SRCREV set in the datastore.The function uses unexpanded
getVar()calls (False flag) to safely inspect SRCREV without triggering AUTOREV expansion or live network fetches. It mirrors BitBake's internal SRCREV resolution fallback chain exactly:
SRCREV_<name>:pn-<recipe>, SRCREV_<name>, SRCREV:pn-<recipe>, SRCREV
That sounds wrong. Bitbake uses the *expanded* value:lib/bb/fetch2/__init__.py:
1256 srcrev = d.getVar(a)
It needs the expansion to handle every possible overrides (for example, MACHINE or DISTRO, but those are not the only one)
Quoted 50 lines
Returns:
- The revision string if a valid SRCREV is found
- '' for non-SCM URIs or URIs with inline rev= or tag= parameter (skip)
- None if SRCREV is missing or INVALIDThis shared helper is used by base.bbclass, insane.bbclass and yocto-check-layer.bbclass to avoid duplicating the same logic.
Reported-by Yoann Congal <[email protected]>AI-Generated: Developed with assistance from Anthropic Claude
Signed-off-by Sai Sneha <[email protected]>---
Changes in v5:
- Add tag= to skip condition alongside rev= to fix selftest breakage (test_git_unpack_nonetwork) reported by Mathieu Dubois-Briand
- Update docstring to mention tag= parameterChanges in v4:
- New patch: factored common SRCREV checking logic into shared helper
- Eliminates code duplication across base.bbclass, insane.bbclass and yocto-check-layer.bbclass
- Uses for-else pattern, f-strings, simplified name lookup
- Adds docstring explaining return valuesChanges in v3:
- Added AI-Generated disclosure and Reported-by tagChanges in v2:
- Initial public submissionmeta/lib/oe/qa.py | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+)meta/lib/oe/qa.py +7
-0@@ -240,6 +240,36 @@ def check_upstream_status(fullpath): else: return "Missing Upstream-Status in patch\n%s\nPlease add according to %s ." % (fullpath, guidelines) +def check_uri_srcrev(pn, uri, d): + """ + Check that a single SCM URI has a valid SRCREV set. + + Returns the resolved revision string if valid (including '${AUTOREV}'). + Returns '' for non-SCM URIs or URIs with inline rev= parameter (skip). + Returns None if SRCREV is missing or INVALID.
meta/lib/oe/qa.py def check_upstream_status(fullpath): · line 259
Quoted 10 lines
meta/lib/oe/qa.py +10
-0+ """ + import bb.fetch2 + try: + (scheme, _, _, _, _, params) = bb.fetch2.decodeurl(uri) + except Exception: + return '' + if scheme not in ('git', 'gitsm', 'hg', 'svn'): + return '' + if params.get('rev', '') or params.get('tag', ''): + return ''
Sorry but I'm not sure "skipping the check for tag=... SRC_URI" is the right fix.
A SRC_URI with tag=... but no SRCREV is a situation we want to avoid in OE-core so the check must remain enabled by default.
How about disabling the check in the test instead?
meta/lib/oe/qa.py def check_upstream_status(fullpath): · line 268
Quoted 9 lines
meta/lib/oe/qa.py +9
-0+ name = params.get('name', 'default') + candidates = [ + f'SRCREV_{name}:pn-{pn}', + f'SRCREV_{name}', + f'SRCREV:pn-{pn}', + 'SRCREV', + ] + for candidate in candidates: + rev = d.getVar(candidate, False)
This code will miss many overrides (e.g MACHINE or DISTRO)
Quoted 7 lines
meta/lib/oe/qa.py +4
-0+ if rev and rev != 'INVALID': + return rev + return None + if __name__ == "__main__": import sys
Signature · 2 lines
Yoann Congal Smile ECS
Re: [OE-core] [PATCH v5 1/4] oe/qa.py: add check_uri_srcrev() helper for SCM URI SRCREV validation
Quoted 11 lines · Yoann Congal via lists.openembedded.org
Hello,
Quoted 8 lines · Sai Sneha
Add a shared helper function
check_uri_srcrev()that checks whether a single SCM URI has a valid SRCREV set in the datastore.The function uses unexpanded
getVar()calls (False flag) to safely inspect SRCREV without triggering AUTOREV expansion or live network fetches. It mirrors BitBake's internal SRCREV resolution fallback chain exactly:
SRCREV_<name>:pn-<recipe>, SRCREV_<name>, SRCREV:pn-<recipe>, SRCREV
Quoted 6 lines
That sounds wrong. Bitbake uses the *expanded* value:
lib/bb/fetch2/__init__.py:
1256 srcrev =d.getVar(a)It needs the expansion to handle every possible overrides (for example, MACHINE or DISTRO, but those are not the only one)
I don't think we should be handling overrides here at all, that's the job of the bitbake parser.
Best regards,
Signature · 1 line
Paul Barker
Re: [OE-core] [PATCH v5 1/4] oe/qa.py: add check_uri_srcrev() helper for SCM URI SRCREV validation
In reply to Sai SnehaQuoted 56 lines · Sai Sneha via lists.openembedded.org
Add a shared helper function
check_uri_srcrev()that checks whether a single SCM URI has a valid SRCREV set in the datastore.The function uses unexpanded
getVar()calls (False flag) to safely inspect SRCREV without triggering AUTOREV expansion or live network fetches. It mirrors BitBake's internal SRCREV resolution fallback chain exactly:
SRCREV_<name>:pn-<recipe>, SRCREV_<name>, SRCREV:pn-<recipe>, SRCREVReturns:
- The revision string if a valid SRCREV is found
- '' for non-SCM URIs or URIs with inline rev= or tag= parameter (skip)
- None if SRCREV is missing or INVALIDThis shared helper is used by base.bbclass, insane.bbclass and yocto-check-layer.bbclass to avoid duplicating the same logic.
Reported-by Yoann Congal <[email protected]>AI-Generated: Developed with assistance from Anthropic Claude
Signed-off-by Sai Sneha <[email protected]>---
Changes in v5:
- Add tag= to skip condition alongside rev= to fix selftest breakage (test_git_unpack_nonetwork) reported by Mathieu Dubois-Briand
- Update docstring to mention tag= parameterChanges in v4:
- New patch: factored common SRCREV checking logic into shared helper
- Eliminates code duplication across base.bbclass, insane.bbclass and yocto-check-layer.bbclass
- Uses for-else pattern, f-strings, simplified name lookup
- Adds docstring explaining return valuesChanges in v3:
- Added AI-Generated disclosure and Reported-by tagChanges in v2:
- Initial public submissionmeta/lib/oe/qa.py | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+)meta/lib/oe/qa.py +5
-0@@ -240,6 +240,36 @@ def check_upstream_status(fullpath): else: return "Missing Upstream-Status in patch\n%s\nPlease add according to %s ." % (fullpath, guidelines) +def check_uri_srcrev(pn, uri, d): + """ + Check that a single SCM URI has a valid SRCREV set. + + Returns the resolved revision string if valid (including '${AUTOREV}').
meta/lib/oe/qa.py def check_upstream_status(fullpath): · line 257
Quoted 10 lines
meta/lib/oe/qa.py +10
-0+ Returns '' for non-SCM URIs or URIs with inline rev= parameter (skip). + Returns None if SRCREV is missing or INVALID. + """ + import bb.fetch2 + try: + (scheme, _, _, _, _, params) = bb.fetch2.decodeurl(uri) + except Exception: + return '' + if scheme not in ('git', 'gitsm', 'hg', 'svn'): + return ''
Each fetcher class has a supports_srcrev() method we can call instead of trying to hard-code a list here.
Quoted 4 lines
meta/lib/oe/qa.py +4
-0+ if params.get('rev', '') or params.get('tag', ''): + return '' + name = params.get('name', 'default') + candidates = [
meta/lib/oe/qa.py def check_upstream_status(fullpath): · line 271
Quoted 10 lines
meta/lib/oe/qa.py +10
-0+ f'SRCREV_{name}:pn-{pn}', + f'SRCREV_{name}', + f'SRCREV:pn-{pn}', + 'SRCREV', + ] + for candidate in candidates: + rev = d.getVar(candidate, False) + if rev and rev != 'INVALID': + return rev + return None
If there is no 'name' param, this code will look for SRCREV_default before SRCREV, which is not correct. Off the top of my head, I think this should be looking at SRCREV_${name} if name is set, else SRCREV. You should check how the existing fetcher code determines the SRCREV value to use.
Please also drop overrides handling as I mentioned in the other email I just sent.
Signature · 1 line
Paul Barker
PATCH v5 2/4 base.bbclass: warn when SRCREV is missing for SCM URIs at parse time
In reply to Sai SnehaA recipe with a git:// (or other SCM) URI in SRC_URI must have a corresponding SRCREV set. Without it, BitBake performs a live query on the remote repository at every parse, breaking reproducibility and causing parse failures under BB_NO_NETWORK=1.
The trivial fix of checking SRCREV in insane.bbclass do_recipe_qa fails because fetcher_hashes_dummyfunc[vardepvalue] expands SRCREV at parse time before QA checks run, causing a FetchError that halts parsing entirely. This was identified in Corentin Guillevic's RFC series and confirmed by Mathieu Dubois-Briand's autobuilder testing.
The fix intercepts at the vardepvalue expansion point by introducing check_srcrev_set() which uses the shared oe.qa.check_uri_srcrev() helper. The function is called conditionally:
fetcher_hashes_dummyfunc[vardepvalue] =
"${@bb.fetch.get_hashvalue(d) if check_srcrev_set(d) else ''}"Note on design choice: we intentionally use d.getVar(candidate, False) in oe.qa.check_uri_srcrev() rather than srcrev_internal_helper() because the latter expands SRCREV, triggering get_autorev() and live network fetches -- the exact problem this patch fixes.
Note: parse-time warnings intentionally complement the QA-time warnings added in insane.bbclass. This is necessary because CI pipelines running 'bitbake --parse-only' with BB_NO_NETWORK=1 (the original motivation from Ross Burton) would miss the issue entirely if only QA-time warnings existed. This pattern is already established in the codebase (e.g. src-uri-bad fires at both parse and QA time).
AI-Generated: Developed with assistance from Anthropic Claude
---
Changes in v5:
- No changes to this patch
Changes in v4:
- Refactored to use shared oe.qa.check_uri_srcrev() helper
- Eliminates duplicated URI parsing logic
- Added docstring explaining True/False return values
Changes in v3:
- Added AI-Generated disclosure and Reported-by tag
Changes in v2:
- Initial public submission
meta/classes-global/base.bbclass | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-)
meta/classes-global/base.bbclass +28 -2
@@ -164,14 +164,40 @@ def setup_hosttools_dir(dest, toolsvar, d, fatal=True): if notfound and fatal: bb.fatal("The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed:\n %s" % " ".join(notfound)) + +def check_srcrev_set(d): + """ + Inspect SRC_URI for SCM URIs missing a valid SRCREV. + + Returns True if all SCM URIs have a valid SRCREV or AUTOREV set, + allowing get_hashvalue(d) to proceed normally. + + Returns False if any SCM URI is missing SRCREV, suppressing + get_hashvalue(d) to prevent a FetchError parse crash. + """ + import oe.qa + src_uri = (d.getVar('SRC_URI', False) or '').split() + pn = d.getVar('PN') + for uri in src_uri: + rev = oe.qa.check_uri_srcrev(pn, uri, d) + if rev is None: + # SRCREV missing — warn and suppress hash expansion + if bb.utils.contains('ERROR_QA', 'missing-srcrev', True, False, d): + bb.error("%s: SRCREV not set for %s" % (pn, uri)) + elif bb.utils.contains('WARN_QA', 'missing-srcrev', True, False, d): + bb.warn("%s: SRCREV not set for %s" % (pn, uri)) + return False + if rev and '${AUTOREV}' in rev: + return True + return True + # We can't use vardepvalue against do_fetch directly since that would overwrite # the other task dependencies so we use an indirect function. python fetcher_hashes_dummyfunc() { return } -fetcher_hashes_dummyfunc[vardepvalue] = "${@bb.fetch.get_hashvalue(d)}" - +fetcher_hashes_dummyfunc[vardepvalue] = "${@bb.fetch.get_hashvalue(d) if check_srcrev_set(d) else ''}" addtask fetch do_fetch[dirs] = "${DL_DIR}" do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}"
Signature · 1 line
2.34.1
Re: [OE-core] [PATCH v5 2/4] base.bbclass: warn when SRCREV is missing for SCM URIs at parse time
Quoted 86 lines · Sai Sneha via lists.openembedded.org
A recipe with a
git://(or other SCM) URI inSRC_URImust have a corresponding SRCREV set. Without it, BitBake performs a live query on the remote repository at every parse, breaking reproducibility and causing parse failures under BB_NO_NETWORK=1.The trivial fix of checking SRCREV in insane.bbclass
do_recipe_qafails because fetcher_hashes_dummyfunc[vardepvalue] expands SRCREV at parse time before QA checks run, causing a FetchError that halts parsing entirely. This was identified in Corentin Guillevic's RFC series and confirmed by Mathieu Dubois-Briand's autobuilder testing.The fix intercepts at the vardepvalue expansion point by introducing
check_srcrev_set()which uses the sharedoe.qa.check_uri_srcrev()helper. The function is called conditionally:fetcher_hashes_dummyfunc[vardepvalue] = "${@bb.fetch.get_hashvalue(d) if check_srcrev_set(d) else ''}"Note on design choice: we intentionally use
d.getVar(candidate, False)inoe.qa.check_uri_srcrev()rather thansrcrev_internal_helper()because the latter expands SRCREV, triggeringget_autorev()and live network fetches -- the exact problem this patch fixes.Note: parse-time warnings intentionally complement the QA-time warnings added in insane.bbclass. This is necessary because CI pipelines running 'bitbake --parse-only' with BB_NO_NETWORK=1 (the original motivation from Ross Burton) would miss the issue entirely if only QA-time warnings existed. This pattern is already established in the codebase (e.g. src-uri-bad fires at both parse and QA time).
Reported-by Yoann Congal <[email protected]>AI-Generated: Developed with assistance from Anthropic Claude
Signed-off-by Sai Sneha <[email protected]>---
Changes in v5:
- No changes to this patchChanges in v4:
- Refactored to use sharedoe.qa.check_uri_srcrev()helper
- Eliminates duplicated URI parsing logic
- Added docstring explaining True/False return valuesChanges in v3:
- Added AI-Generated disclosure and Reported-by tagChanges in v2:
- Initial public submissionmeta/classes-global/base.bbclass | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-)meta/classes-global/base.bbclass +26
-0@@ -164,14 +164,40 @@ def setup_hosttools_dir(dest, toolsvar, d, fatal=True): if notfound and fatal: bb.fatal("The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed:\n %s" % " ".join(notfound)) + +def check_srcrev_set(d): + """ + Inspect SRC_URI for SCM URIs missing a valid SRCREV. + + Returns True if all SCM URIs have a valid SRCREV or AUTOREV set, + allowing get_hashvalue(d) to proceed normally. + + Returns False if any SCM URI is missing SRCREV, suppressing + get_hashvalue(d) to prevent a FetchError parse crash. + """ + import oe.qa + src_uri = (d.getVar('SRC_URI', False) or '').split() + pn = d.getVar('PN') + for uri in src_uri: + rev = oe.qa.check_uri_srcrev(pn, uri, d) + if rev is None: + # SRCREV missing — warn and suppress hash expansion + if bb.utils.contains('ERROR_QA', 'missing-srcrev', True, False, d): + bb.error("%s: SRCREV not set for %s" % (pn, uri)) + elif bb.utils.contains('WARN_QA', 'missing-srcrev', True, False, d): + bb.warn("%s: SRCREV not set for %s" % (pn, uri)) + return False + if rev and '${AUTOREV}' in rev: + return True + return True
meta/classes-global/base.bbclass def setup_hosttools_dir(dest, toolsvar, d, fatal=True): · line 200
Quoted 10 lines
meta/classes-global/base.bbclass +2
-2+ # We can't use vardepvalue against do_fetch directly since that would overwrite # the other task dependencies so we use an indirect function. python fetcher_hashes_dummyfunc() { return } -fetcher_hashes_dummyfunc[vardepvalue] = "${@bb.fetch.get_hashvalue(d)}" - +fetcher_hashes_dummyfunc[vardepvalue] = "${@bb.fetch.get_hashvalue(d) if check_srcrev_set(d) else ''}"
I think the overall approach here is wrong. This code attempts to mask the error that would occur if network access is unavailable but the SRCREV cannot be resolved without network access. However, it fails to do this properly - what about SRCREV set to a tag instead of a commit hash? In that case check_uri_srcrev() would return a non-empty string, check_srcrev_set() would return True and we would still end up looking up the tag in the remote git repository during parsing.
For the cases where one SRC_URI entry specifies a branch or tag, and has no SRCREV set, all SRCREV hashes are removed from the vardeps of the do_fetch function. This means that the fetcher will no longer be re-ran when the branches or tags are changed upstream. That is breakage of expected fetcher behaviour.
I recommend discussing the issues further on the bug tracker or mailing list, making sure you have a full understanding of the behaviour and the necessary change. Then come back to the implementation.
Best regards,
Signature · 1 line
Paul Barker
PATCH v5 3/4 insane.bbclass: add missing-srcrev QA check for SCM URIs
In reply to Sai SnehaAdd a proper QA check test_missing_srcrev() in do_recipe_qa that fires at build time via oe.qa.handle_error(), using the shared oe.qa.check_uri_srcrev() helper to avoid code duplication.
Severity is controlled per layer:
- WARN_QA:append = " missing-srcrev" for all layers (warning by default)
- ERROR_QA:append:layer-core = " missing-srcrev" for oe-core (strict)
This follows the same pattern as missing-metadata and missing-maintainer which are warnings globally but errors for oe-core.
Note on CHECKLAYER_REQUIRED_TESTS: missing-srcrev is intentionally not added to CHECKLAYER_REQUIRED_TESTS because that would promote it to ERROR_QA for all layers globally during normal builds, contradicting the layer-specific enforcement requested in Bug 16051 Comment 3. Instead, yocto-check-layer.bbclass enforces it directly during checklayer validation runs.
AI-Generated: Developed with assistance from Anthropic Claude
---
Changes in v5:
- No changes to this patch
Changes in v4:
- Refactored to use shared oe.qa.check_uri_srcrev() helper
- Eliminates duplicated URI parsing logic
- Added note explaining why CHECKLAYER_REQUIRED_TESTS is not used
Changes in v3:
- Added AI-Generated disclosure and Reported-by tag
Changes in v2:
- Initial public submission
meta/classes-global/insane.bbclass | 13 +++++++++++++ 1 file changed, 13 insertions(+)
meta/classes-global/insane.bbclass +13 -0
@@ -49,6 +49,9 @@ ERROR_QA ?= "\ ERROR_QA:append = " ${@bb.utils.filter('DISTRO_FEATURES', 'usrmerge', d)}" WARN_QA:append:layer-core = " missing-metadata missing-maintainer" +WARN_QA:append = " missing-srcrev" +ERROR_QA:append:layer-core = " missing-srcrev" + FAKEROOT_QA = "host-user-contaminated" FAKEROOT_QA[doc] = "QA tests which need to run under fakeroot. If any \ enabled tests are listed here, the do_package_qa task will run under fakeroot."
@@ -1455,6 +1458,15 @@ python do_qa_unpack() { python do_recipe_qa() { import re + def test_missing_srcrev(pn, d): + import oe.qa + src_uri = (d.getVar('SRC_URI', False) or '').split() + for uri in src_uri: + rev = oe.qa.check_uri_srcrev(pn, uri, d) + if rev is None: + oe.qa.handle_error("missing-srcrev", + "%s: SRCREV not set for SCM URI %s" % (pn, uri), d) + def test_naming(pn, d): if pn.endswith("-native") and not bb.data.inherits_class("native", d): oe.qa.handle_error("recipe-naming", "Recipe %s appears native but is not, should inherit native" % pn, d)
@@ -1497,6 +1509,7 @@ python do_recipe_qa() { pn = d.getVar('PN') test_naming(pn, d) + test_missing_srcrev(pn, d) test_missing_metadata(pn, d) test_missing_maintainer(pn, d) test_srcuri(pn, d)
Signature · 1 line
2.34.1
Re: [PATCH v5 3/4] insane.bbclass: add missing-srcrev QA check for SCM URIs
Quoted 11 lines · Sai Sneha
Add a proper QA check
test_missing_srcrev()indo_recipe_qathat fires at build time viaoe.qa.handle_error(), using the sharedoe.qa.check_uri_srcrev()helper to avoid code duplication.Severity is controlled per layer:
- WARN_QA:append = " missing-srcrev" for all layers (warning by default)
- ERROR_QA:append:layer-core = " missing-srcrev" for oe-core (strict)This follows the same pattern as missing-metadata and missing-maintainer which are warnings globally but errors for oe-core.
Quoted 6 lines
Note on
CHECKLAYER_REQUIRED_TESTS: missing-srcrev is intentionally not added toCHECKLAYER_REQUIRED_TESTSbecause that would promote it toERROR_QAfor all layers globally during normal builds, contradicting the layer-specific enforcement requested in Bug 16051 Comment 3. Instead, yocto-check-layer.bbclass enforces it directly during checklayer validation runs.
Neither CHECKLAYER_REQUIRED_TESTS nor yocto-check-layer does not appear in this patch. Maybe move this note to the next patch?
Quoted 61 lines
Reported-by Yoann Congal <[email protected]>AI-Generated: Developed with assistance from Anthropic Claude
Signed-off-by Sai Sneha <[email protected]>---
Changes in v5:
- No changes to this patchChanges in v4:
- Refactored to use sharedoe.qa.check_uri_srcrev()helper
- Eliminates duplicated URI parsing logic
- Added note explaining whyCHECKLAYER_REQUIRED_TESTSis not usedChanges in v3:
- Added AI-Generated disclosure and Reported-by tagChanges in v2:
- Initial public submissionmeta/classes-global/insane.bbclass | 13 +++++++++++++ 1 file changed, 13 insertions(+)meta/classes-global/insane.bbclass +13
-0@@ -49,6 +49,9 @@ ERROR_QA ?= "\ ERROR_QA:append = " ${@bb.utils.filter('DISTRO_FEATURES', 'usrmerge', d)}" WARN_QA:append:layer-core = " missing-metadata missing-maintainer" +WARN_QA:append = " missing-srcrev" +ERROR_QA:append:layer-core = " missing-srcrev" + FAKEROOT_QA = "host-user-contaminated" FAKEROOT_QA[doc] = "QA tests which need to run under fakeroot. If any \ enabled tests are listed here, the do_package_qa task will run under fakeroot."@@ -1455,6 +1458,15 @@ python do_qa_unpack() { python do_recipe_qa() { import re + def test_missing_srcrev(pn, d): + import oe.qa + src_uri = (d.getVar('SRC_URI', False) or '').split() + for uri in src_uri: + rev = oe.qa.check_uri_srcrev(pn, uri, d) + if rev is None: + oe.qa.handle_error("missing-srcrev", + "%s: SRCREV not set for SCM URI %s" % (pn, uri), d) + def test_naming(pn, d): if pn.endswith("-native") and not bb.data.inherits_class("native", d): oe.qa.handle_error("recipe-naming", "Recipe %s appears native but is not, should inherit native" % pn, d)@@ -1497,6 +1509,7 @@ python do_recipe_qa() { pn = d.getVar('PN') test_naming(pn, d) + test_missing_srcrev(pn, d) test_missing_metadata(pn, d) test_missing_maintainer(pn, d) test_srcuri(pn, d)
Signature · 2 lines
Yoann Congal Smile ECS
Re: [PATCH v5 3/4] insane.bbclass: add missing-srcrev QA check for SCM URIs
Hi all,
After reading through all the mailing list feedback carefully and spending time studying the bitbake fetcher code, I think I now have a better picture of what went wrong with my approach. I wanted to summarize my understanding before attempting v6.
The core problem with v5:
The bigger issue is that intercepting at the vardepvalue level in base.bbclass was the wrong approach entirely. when I suppress the hash computation for recipes with missing SRCREV I am also accidentally breaking something important bitbake uses that hash to know when to re-run do_fetch. So if a branch or tag changes upstream do_fetch would silently not re-run which is worse than the original problem.
Other issues I missed:
- Using d.getVar(candidate, False) misses MACHINE/DISTRO overrides. Paul is right that override resolution is the bitbake parser's job, not something OE-core should be reimplementing.
- Hardcoding git/gitsm/hg/svn was fragile. bitbake already has supports_srcrev() on each fetcher class for exactly this purpose.
- The parse-time warning was wrong. src-uri-bad is purely a QA-time check I should not have added parse-time warnings.
- The problem is broader than just missing SRCREV as a SRCREV set to a branch or tag name instead of a commit hash causes the same reproducibility issues.
- The tag= skip was wrong since the selftest recipe should use INSANE_SKIP instead of skipping tag= globally.
- The CHECKLAYER_REQUIRED_TESTS decision needs proper discussion not a unilateral call in the cover letter.
Where I think the fix should go:
Richard suggested fixing this in the bitbake fetchers themselves using supports_srcrev(). I am not sure if that means changes to the BitBake repository itself, or just using supports_srcrev() within OE-core. Could someone clarify the right direction?
My current thinking for OE-core is a simple QA check in insane.bbclass using oe.qa.handle_error() with supports_srcrev() to identify which URIs need checking, along with fixing the gitunpackoffline selftest with INSANE_SKIP. But I want to make sure this aligns with what the maintainers have in mind before writing more code.
Thanks again for the thorough review it was genuinely helpful.
Sai Sneha
Quoted 95 lines · Yoann Congal
Quoted 11 lines · Sai Sneha
Add a proper QA check
test_missing_srcrev()indo_recipe_qathat fires at build time viaoe.qa.handle_error(), using the sharedoe.qa.check_uri_srcrev()helper to avoid code duplication.Severity is controlled per layer:
- WARN_QA:append = " missing-srcrev" for all layers (warning by default)
- ERROR_QA:append:layer-core = " missing-srcrev" for oe-core (strict)This follows the same pattern as missing-metadata and missing-maintainer which are warnings globally but errors for oe-core.
Quoted 6 lines
Note on
CHECKLAYER_REQUIRED_TESTS: missing-srcrev is intentionally not added toCHECKLAYER_REQUIRED_TESTSbecause that would promote it toERROR_QAfor all layers globally during normal builds, contradicting the layer-specific enforcement requested in Bug 16051 Comment 3. Instead, yocto-check-layer.bbclass enforces it directly during checklayer validation runs.Neither
CHECKLAYER_REQUIRED_TESTSnor yocto-check-layer does not appear in this patch. Maybe move this note to the next patch?Quoted 18 lines
Reported-by Yoann Congal <[email protected]>AI-Generated: Developed with assistance from Anthropic Claude
Signed-off-by Sai Sneha <[email protected]>---
Changes in v5:
- No changes to this patchChanges in v4:
- Refactored to use sharedoe.qa.check_uri_srcrev()helper
- Eliminates duplicated URI parsing logic
- Added note explaining whyCHECKLAYER_REQUIRED_TESTSis not usedChanges in v3:
- Added AI-Generated disclosure and Reported-by tagReview
Quoted 6 lines
Changes in v2:
- Initial public submissionmeta/classes-global/insane.bbclass | 13 +++++++++++++ 1 file changed, 13 insertions(+)(quoted diff) +0
-0b/meta/classes-global/insane.bbclass
meta/classes-global/insane.bbclass ERROR_QA ?= "\ · line 49
Quoted 5 lines
index
04700be71c..1fe426646d100644meta/classes-global/insane.bbclass +0
-0@@ -49,6 +49,9 @@ ERROR_QA ?= "\ ERROR_QA:append = " ${@bb.utils.filter('DISTRO_FEATURES', 'usrmerge',d)}"
meta/classes-global/insane.bbclass ERROR_QA ?= "\
Quoted 8 lines
meta/classes-global/insane.bbclass +3
-0WARN_QA:append:layer-core = " missing-metadata missing-maintainer" +WARN_QA:append = " missing-srcrev" +ERROR_QA:append:layer-core = " missing-srcrev" + FAKEROOT_QA = "host-user-contaminated" FAKEROOT_QA[doc] = "QA tests which need to run under fakeroot. If any \ enabled tests are listed here, the do_package_qa task will run underfakeroot."
Quoted 5 lines
meta/classes-global/insane.bbclass +1
-0@@ -1455,6 +1458,15 @@ python do_qa_unpack() { python do_recipe_qa() { import re + def test_missing_srcrev(pn, d):meta/classes-global/insane.bbclass python do_qa_unpack() { · line 1471
Quoted 10 lines
meta/classes-global/insane.bbclass +8
-0+ import oe.qa + src_uri = (d.getVar('SRC_URI', False) or '').split() + for uri in src_uri: + rev = oe.qa.check_uri_srcrev(pn, uri, d) + if rev is None: + oe.qa.handle_error("missing-srcrev", + "%s: SRCREV not set for SCM URI %s" % (pn, uri), d) + def test_naming(pn, d): if pn.endswith("-native") and not
bb.data.inherits_class("native", d):meta/classes-global/insane.bbclass python do_qa_unpack() { · line 1472
Quoted 1 line
meta/classes-global/insane.bbclass +0
-0oe.qa.handle_error("recipe-naming", "Recipe %s appearsnative but is not, should inherit native" % pn, d)
meta/classes-global/insane.bbclass python do_recipe_qa() { · line 1515
Quoted 8 lines
meta/classes-global/insane.bbclass +1
-0@@ -1497,6 +1509,7 @@ python do_recipe_qa() { pn = d.getVar('PN') test_naming(pn, d) + test_missing_srcrev(pn, d) test_missing_metadata(pn, d) test_missing_maintainer(pn, d) test_srcuri(pn, d)--
Yoann Congal
Smile ECS
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [OE-core] insane.bbclass: add missing-srcrev QA check for SCM URIs
Hi all,
Following up on the suggestions to fix this in the BitBake fetchers using supports_srcrev(), I have tried implementing that approach locally. Here is what I did and how it works.
The fix has two parts:
Part 1 — BitBake fix (prevent parse crash):
The crash happens in setup_revisions() when srcrev_internal_helper() raises a FetchError for missing SRCREV. This propagates up through expandWithRefs() as an ExpansionError which halts parsing for all recipes. The fix catches FetchError in setup_revisions() and sets a missing_srcrev flag on the FetchData object:
def setup_revisions(self, d):
try:
self.revision = srcrev_internal_helper(self, d, self.name) except FetchError:
self.revision = None
self.missing_srcrev = True
git.py and hg.py also needed a guard to prevent calling latest_revision() when missing_srcrev is set, since that would trigger a live network fetch under BB_NO_NETWORK=1.
Part 2 — OE-core QA check (insane.bbclass):
A simple test_missing_srcrev() in do_recipe_qa uses supports_srcrev() to identify which URIs need checking:
def test_missing_srcrev(pn, d):
import bb.fetch2
try:
fetcher = bb.fetch2.Fetch( (d.getVar('SRC_URI') or '').split(), d)
for url in fetcher.ud:
ud = fetcher.ud[url]
if ud.method.supports_srcrev() and \getattr(ud, 'missing_srcrev', False):
oe.qa.handle_error("missing-srcrev",
"%s: SRCREV not set for SCM URI %s" % (pn, url), d) except bb.fetch2.FetchError:
pass
Test results:
$ bitbake -p 2>&1 | grep -E "(WARNING|ERROR|SRCREV)" WARNING: You are running bitbake under WSLv2... Summary: There was 1 WARNING message. (no parse crash, all recipes parse cleanly)
$ bitbake -c recipe_qa -f test-fetcher 2>&1 | grep -E "(WARNING|ERROR|missing-srcrev)" WARNING: test-fetcher-1.0-r0 do_recipe_qa: QA Issue: test-fetcher: SRCREV not set for SCM URI git://github.com/example/project.git [missing-srcrev]
$ bitbake -c recipe_qa test-srcrev 2>&1 | grep -E "(WARNING|ERROR|missing-srcrev)" ERROR: test-srcrev-1.0-r0 do_recipe_qa: QA Issue: test-srcrev: SRCREV not set for SCM URI git://github.com/example/project.git [missing-srcrev]
Selftest results:test_git_unpack_nonetwork PASSEDtest_git_unpack_nonetwork_fail PASSEDtest_git_patchtool PASSEDtest_git_patchtool2 PASSEDtest_just_parse PASSEDtest_warnings_errors PASSEDtest_invalid_recipe_src_uri PASSED
Changes are minimal:
BitBake: 3 files, 8 insertions, 3 deletions
OE-core: 1 file, 16 insertions, 1 deletion
Does this approach align with what was suggested? I am happy to refine it further based on the feedback.
Best regards,
Sai Sneha
Quoted 166 lines · Sai Sneha via lists.openembedded.org
Hi all,
After reading through all the mailing list feedback carefully and spending time studying the bitbake fetcher code, I think I now have a better picture of what went wrong with my approach. I wanted to summarize my understanding before attempting v6.
The core problem with v5:
The bigger issue is that intercepting at the vardepvalue level in base.bbclass was the wrong approach entirely. when I suppress the hash computation for recipes with missing SRCREV I am also accidentally breaking something important bitbake uses that hash to know when to re-run
do_fetch. So if a branch or tag changes upstreamdo_fetchwould silently not re-run which is worse than the original problem.Other issues I missed:
- Using
d.getVar(candidate, False)misses MACHINE/DISTRO overrides. Paul is right that override resolution is the bitbake parser's job, not something OE-core should be reimplementing.- Hardcoding git/gitsm/hg/svn was fragile. bitbake already has
supports_srcrev()on each fetcher class for exactly this purpose.- The parse-time warning was wrong. src-uri-bad is purely a QA-time check I should not have added parse-time warnings.
- The problem is broader than just missing SRCREV as a SRCREV set to a branch or tag name instead of a commit hash causes the same reproducibility issues.
- The tag= skip was wrong since the selftest recipe should use
INSANE_SKIPinstead of skipping tag= globally.- The
CHECKLAYER_REQUIRED_TESTSdecision needs proper discussion not a unilateral call in the cover letter.Where I think the fix should go:
Richard suggested fixing this in the bitbake fetchers themselves using
supports_srcrev(). I am not sure if that means changes to the BitBake repository itself, or just usingsupports_srcrev()within OE-core. Could someone clarify the right direction?My current thinking for OE-core is a simple QA check in insane.bbclass using
oe.qa.handle_error()withsupports_srcrev()to identify which URIs need checking, along with fixing the gitunpackoffline selftest withINSANE_SKIP. But I want to make sure this aligns with what the maintainers have in mind before writing more code.Thanks again for the thorough review it was genuinely helpful.
Sai Sneha
Quoted 82 lines · Yoann Congal
Quoted 11 lines · Sai Sneha
Add a proper QA check
test_missing_srcrev()indo_recipe_qathat fires at build time viaoe.qa.handle_error(), using the sharedoe.qa.check_uri_srcrev()helper to avoid code duplication.Severity is controlled per layer:
- WARN_QA:append = " missing-srcrev" for all layers (warning by default)
- ERROR_QA:append:layer-core = " missing-srcrev" for oe-core (strict)This follows the same pattern as missing-metadata and missing-maintainer which are warnings globally but errors for oe-core.
Quoted 6 lines
Note on
CHECKLAYER_REQUIRED_TESTS: missing-srcrev is intentionally not added toCHECKLAYER_REQUIRED_TESTSbecause that would promote it toERROR_QAfor all layers globally during normal builds, contradicting the layer-specific enforcement requested in Bug 16051 Comment 3. Instead, yocto-check-layer.bbclass enforces it directly during checklayer validation runs.Neither
CHECKLAYER_REQUIRED_TESTSnor yocto-check-layer does not appear in this patch. Maybe move this note to the next patch?Quoted 18 lines
Reported-by Yoann Congal <[email protected]>AI-Generated: Developed with assistance from Anthropic Claude
Signed-off-by Sai Sneha <[email protected]>---
Changes in v5:
- No changes to this patchChanges in v4:
- Refactored to use sharedoe.qa.check_uri_srcrev()helper
- Eliminates duplicated URI parsing logic
- Added note explaining whyCHECKLAYER_REQUIRED_TESTSis not usedChanges in v3:
- Added AI-Generated disclosure and Reported-by tagReview
Quoted 6 lines
Changes in v2:
- Initial public submissionmeta/classes-global/insane.bbclass | 13 +++++++++++++ 1 file changed, 13 insertions(+)(quoted diff) +0
-0b/meta/classes-global/insane.bbclass
meta/classes-global/insane.bbclass ERROR_QA ?= "\ · line 49
Quoted 5 lines
index
04700be71c..1fe426646d100644meta/classes-global/insane.bbclass +0
-0@@ -49,6 +49,9 @@ ERROR_QA ?= "\ ERROR_QA:append = " ${@bb.utils.filter('DISTRO_FEATURES', 'usrmerge',d)}"
meta/classes-global/insane.bbclass ERROR_QA ?= "\
Quoted 8 lines
meta/classes-global/insane.bbclass +3
-0WARN_QA:append:layer-core = " missing-metadata missing-maintainer" +WARN_QA:append = " missing-srcrev" +ERROR_QA:append:layer-core = " missing-srcrev" + FAKEROOT_QA = "host-user-contaminated" FAKEROOT_QA[doc] = "QA tests which need to run under fakeroot. If any \ enabled tests are listed here, the do_package_qa task will run underfakeroot."
Quoted 5 lines
meta/classes-global/insane.bbclass +1
-0@@ -1455,6 +1458,15 @@ python do_qa_unpack() { python do_recipe_qa() { import re + def test_missing_srcrev(pn, d):meta/classes-global/insane.bbclass python do_qa_unpack() { · line 1471
Quoted 10 lines
meta/classes-global/insane.bbclass +8
-0+ import oe.qa + src_uri = (d.getVar('SRC_URI', False) or '').split() + for uri in src_uri: + rev = oe.qa.check_uri_srcrev(pn, uri, d) + if rev is None: + oe.qa.handle_error("missing-srcrev", + "%s: SRCREV not set for SCM URI %s" % (pn, uri), d) + def test_naming(pn, d): if pn.endswith("-native") and not
bb.data.inherits_class("native", d):meta/classes-global/insane.bbclass python do_qa_unpack() { · line 1472
Quoted 1 line
meta/classes-global/insane.bbclass +0
-0oe.qa.handle_error("recipe-naming", "Recipe %s appearsnative but is not, should inherit native" % pn, d)
Quoted 2 lines
meta/classes-global/insane.bbclass +0
-0@@ -1497,6 +1509,7 @@ python do_recipe_qa() {Quoted 13 lines
meta/classes-global/insane.bbclass python do_recipe_qa() { · line 1515
Quoted 6 lines
meta/classes-global/insane.bbclass +1
-0pn = d.getVar('PN') test_naming(pn, d) + test_missing_srcrev(pn, d) test_missing_metadata(pn, d) test_missing_maintainer(pn, d) test_srcuri(pn, d)--
Yoann Congal
Smile ECS-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#238140):
https://lists.openembedded.org/g/openembedded-core/message/238140
Mute This Topic: https://lists.openembedded.org/mt/119607376/10302268
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] insane.bbclass: add missing-srcrev QA check for SCM URIs
Quoted 20 lines · Sai Sneha via lists.openembedded.org
Following up on the suggestions to fix this in the BitBake fetchers using
supports_srcrev(), I have tried implementing that approach locally. Here is what I did and how it works.The fix has two parts:
Part 1 — BitBake fix (prevent parse crash):
The crash happens insetup_revisions()whensrcrev_internal_helper()raises a FetchError for missing SRCREV. This propagates up throughexpandWithRefs()as an ExpansionError which halts parsing for all recipes. The fix catches FetchError insetup_revisions()and sets amissing_srcrevflag on the FetchData object:def
setup_revisions(self, d):
try:
self.revision =srcrev_internal_helper(self, d, self.name)except FetchError:
self.revision = None
self.missing_srcrev = True
Quoted 3 lines
git.pyandhg.pyalso needed a guard to prevent callinglatest_revision()whenmissing_srcrevis set, since that would trigger a live network fetch under BB_NO_NETWORK=1.
That is going to break some of our other protections for invalid SRCREV setups and swallow other errors.
How about instead we set a flag in OE-Core on a per layer basis:
BB_FETCH_REQUIRE_DETERMINISM:layer-core = "1"
and this simply changes the fetcher behaviour to error if it would touch the network?
Cheers,
Richard
Re: [OE-core] insane.bbclass: add missing-srcrev QA check for SCM URIs
Hi Richard,
That makes much more sense, thank you. Setting BB_FETCH_REQUIRE_DETERMINISM as a per-layer flag in OE-core and having BitBake enforce it is a cleaner approach than catching FetchError in setup_revisions().
could you please point me to where in BitBake the network access would be triggered so I know the right place to check this flag? I believe it would be in latest_revision() or srcrev_internal_helper(), but I want to confirm before implementing.
Best regards,
Sai Sneha
Quoted 42 lines · Richard Purdie
On Tue, 2026-06-09 at 14:49 +0530, Sai Sneha via lists.openembedded.org wrote:
Following up on the suggestions to fix this in the BitBake fetchers using
supports_srcrev(), I have tried implementing that approach locally. Here is what I did and how it works.The fix has two parts:
Part 1 — BitBake fix (prevent parse crash):
The crash happens insetup_revisions()whensrcrev_internal_helper()raises a FetchError for missing SRCREV. This propagates up throughexpandWithRefs()as an ExpansionError which halts parsing for all recipes. The fix catches FetchError insetup_revisions()and sets amissing_srcrevflag on the FetchData object:def
setup_revisions(self, d):
try:
self.revision =srcrev_internal_helper(self, d, self.name)except FetchError:
self.revision = None
self.missing_srcrev = True
git.pyandhg.pyalso needed a guard to prevent callinglatest_revision()whenmissing_srcrevis set, since that would trigger a live network fetch under BB_NO_NETWORK=1.That is going to break some of our other protections for invalid SRCREV setups and swallow other errors.
How about instead we set a flag in OE-Core on a per layer basis:
BB_FETCH_REQUIRE_DETERMINISM:layer-core = "1"
and this simply changes the fetcher behaviour to error if it would touch the network?
Cheers,
Richard
Re: [OE-core] insane.bbclass: add missing-srcrev QA check for SCM URIs
Quoted 7 lines · Sai Sneha
Hi Richard,
That makes much more sense, thank you. Setting
BB_FETCH_REQUIRE_DETERMINISMas a per-layer flag in OE-core and having BitBake enforce it is a cleaner approach than catching FetchError insetup_revisions().
Quoted 5 lines
could you please point me to where in BitBake the network access would
be triggered so I know the right place to check this flag? I believe it would be inlatest_revision()orsrcrev_internal_helper(), but I want to confirm before implementing.
Thinking more about this, does:
BB_NO_NETWORK:layer-core = "1"
already do what we need?
Cheers,
Richard
Re: [OE-core] insane.bbclass: add missing-srcrev QA check for SCM URIs
Hi Richard,
Please correct me if am wrong.
BB_NO_NETWORK:layer-core = "1"
would work for the case where SRCREV is set to a floating branch or tag, since _lsremote() calls check_network_access() which raises NetworkAccess when BB_NO_NETWORK is set.
However for the case where SRCREV is completely missing the crash happens earlier in srcrev_internal_helper() which raises FetchError before latest_revision() is even called so BB_NO_NETWORK wouldn’t catch that case.
Best regards,
Sai Sneha
Quoted 24 lines · Richard Purdie
Quoted 7 lines · Sai Sneha
Hi Richard,
That makes much more sense, thank you. Setting
BB_FETCH_REQUIRE_DETERMINISMas a per-layer flag in OE-core and having BitBake enforce it is a cleaner approach than catching FetchError insetup_revisions().Quoted 5 lines
could you please point me to where in BitBake the network access would
be triggered so I know the right place to check this flag? I believe it would be inlatest_revision()orsrcrev_internal_helper(), but I want to confirm before implementing.Thinking more about this, does:
BB_NO_NETWORK:layer-core = "1"
already do what we need?
Cheers,
Richard
Re: [OE-core] insane.bbclass: add missing-srcrev QA check for SCM URIs
Quoted 8 lines · Sai Sneha
Hi Richard,
Please correct me if am wrong. BB_NO_NETWORK:layer-core = "1"
would work for the case where SRCREV is set to a floating branch or tag, since_lsremote()callscheck_network_access()which raises NetworkAccess whenBB_NO_NETWORKis set.
Quoted 4 lines
However for the case where SRCREV is completely missing the crash happens earlier in
srcrev_internal_helper()which raises FetchError beforelatest_revision()is even called soBB_NO_NETWORKwouldn’t catch that case.
In that case the build fails to parse and is unbuildable? That is probably fine as check-layer tests whether it parses?
Cheers,
Richard
Re: [OE-core] insane.bbclass: add missing-srcrev QA check for SCM URIs
Hi Richard,
That makes sense if SRCREV is completely missing the recipe is unbuildable anyway, so the parse failure is an acceptable outcome for that case.
So the complete solution would be:
- BB_NO_NETWORK:layer-core = "1" in OE-core to prevent floating branch/tag lookups
- A QA check in insane.bbclass to warn about missing SRCREV for external layers
- yocto-check-layer catches parse failures during validation
Is this the right direction?
Best regards,
Sai Sneha
Quoted 21 lines · Richard Purdie
Quoted 8 lines · Sai Sneha
Hi Richard,
Please correct me if am wrong.
BB_NO_NETWORK:layer-core = "1"
would work for the case where SRCREV is set to a floating branch or tag, since_lsremote()callscheck_network_access()which raises NetworkAccess whenBB_NO_NETWORKis set.Quoted 4 lines
However for the case where SRCREV is completely missing the crash happens earlier in
srcrev_internal_helper()which raises FetchError beforelatest_revision()is even called soBB_NO_NETWORKwouldn’t catch that case.In that case the build fails to parse and is unbuildable? That is probably fine as check-layer tests whether it parses?
Cheers,
Richard
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [OE-core] insane.bbclass: add missing-srcrev QA check for SCM URIs
I tested BB_NO_NETWORK:layer-core = "1" and found that it does not prevent the parse crash for the case where SRCREV is completely missing.
The crash happens in srcrev_internal_helper() which raises FetchError before any network access is attempted — so BB_NO_NETWORK never gets checked. The BB_NO_NETWORK flag would only help for the case where SRCREV is set to a floating branch/tag and latest_revision() is called, since _lsremote() calls check_network_access() there.
So it seems like two separate fixes are needed:
1. For missing SRCREV — something needs to prevent the parse crash in srcrev_internal_helper(). Would it be appropriate to check BB_NO_NETWORK in srcrev_internal_helper() itself before raising FetchError?
2. For floating branch/tag — BB_NO_NETWORK:layer-core = "1" would work.
On the OE-core side I have a simple QA check in insane.bbclass using supports_srcrev() that fires at build time, which handles the warning/error reporting cleanly.
Does this analysis make sense?
Best regards,
Sai Sneha
Quoted 55 lines · Sai Sneha via lists.openembedded.org
Hi Richard,
That makes sense if SRCREV is completely missing the recipe is unbuildable anyway, so the parse failure is an acceptable outcome for that case.
So the complete solution would be:
- BB_NO_NETWORK:layer-core = "1" in OE-core to prevent floating branch/tag lookups
- A QA check in insane.bbclass to warn about missing SRCREV for external layers
- yocto-check-layer catches parse failures during validationIs this the right direction?
Best regards,
Sai SnehaQuoted 14 lines · Richard Purdie
Quoted 8 lines · Sai Sneha
Hi Richard,
Please correct me if am wrong.
BB_NO_NETWORK:layer-core = "1"
would work for the case where SRCREV is set to a floating branch or tag, since_lsremote()callscheck_network_access()which raises NetworkAccess whenBB_NO_NETWORKis set.Quoted 4 lines
However for the case where SRCREV is completely missing the crash happens earlier in
srcrev_internal_helper()which raises FetchError beforelatest_revision()is even called soBB_NO_NETWORKwouldn’t catch that case.Quoted 7 lines
In that case the build fails to parse and is unbuildable? That is probably fine as check-layer tests whether it parses?
Cheers,
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#238312):
https://lists.openembedded.org/g/openembedded-core/message/238312
Mute This Topic: https://lists.openembedded.org/mt/119719972/10302268
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] insane.bbclass: add missing-srcrev QA check for SCM URIs
Quoted 11 lines · Sai Sneha
I tested BB_NO_NETWORK:layer-core = "1" and found that it does not prevent the parse crash for the case where SRCREV is completely missing.
The crash happens in
srcrev_internal_helper()which raises FetchError before any network access is attempted — soBB_NO_NETWORKnever gets checked. TheBB_NO_NETWORKflag would only help for the case where SRCREV is set to a floating branch/tag andlatest_revision()is called, since_lsremote()callscheck_network_access()there.
FetchError isn't a crash, the code is finding there is a problem and raising an exception. The exception should say what the issue is.
Doesn't that mean that case is already covered and we don't need to do anything further?
Quoted 6 lines
So it seems like two separate fixes are needed:
1. For missing SRCREV — something needs to prevent the parse crash in
srcrev_internal_helper(). Would it be appropriate to checkBB_NO_NETWORKinsrcrev_internal_helper()itself before raising FetchError?
I don't understand why we need to trap this, then handle it again later?
Cheers,
Richard
Re: [OE-core] insane.bbclass: add missing-srcrev QA check for SCM URIs
Hi Richard,
I tested this locally. You are right that the error message is already clear:
"Please set a valid SRCREV for url..."
However parsing halts for ALL recipes with:
"ERROR: Parsing halted due to errors"
Not just the offending recipe. Is that acceptable behavior or does it need fixing?
Best regards,
Sai Sneha
Quoted 33 lines · Richard Purdie
Quoted 11 lines · Sai Sneha
I tested BB_NO_NETWORK:layer-core = "1" and found that it does not prevent the parse crash for the case where SRCREV is completely missing.
The crash happens in
srcrev_internal_helper()which raises FetchError before any network access is attempted — soBB_NO_NETWORKnever gets checked. TheBB_NO_NETWORKflag would only help for the case where SRCREV is set to a floating branch/tag andlatest_revision()is called, since_lsremote()callscheck_network_access()there.FetchError isn't a crash, the code is finding there is a problem and raising an exception. The exception should say what the issue is.
Doesn't that mean that case is already covered and we don't need to do anything further?
Quoted 6 lines
So it seems like two separate fixes are needed:
1. For missing SRCREV — something needs to prevent the parse crash in
srcrev_internal_helper(). Would it be appropriate to checkBB_NO_NETWORKinsrcrev_internal_helper()itself before raising FetchError?I don't understand why we need to trap this, then handle it again later?
Cheers,
Richard
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [OE-core] insane.bbclass: add missing-srcrev QA check for SCM URIs
While testing I have also noticed something in the error message when no name= parameter is set in the URI, the candidates list includes SRCREV_default and SRCREV_default:pn-X even though no name was explicitly specified.looking at the code, self.name defaults to 'default' when no name= param exists (line 1353 of __init__.py) which is what causes this.
Is this intentional behavior or is the candidates list supposed to only include SRCREV:pn-X and SRCREV when no name= is set?
Best regards,
Sai Sneha
Quoted 66 lines · Sai Sneha via lists.openembedded.org
Hi Richard,
I tested this locally. You are right that the error message is already clear:
"Please set a valid SRCREV for url..."
However parsing halts for ALL recipes with:
"ERROR: Parsing halted due to errors"
Not just the offending recipe. Is that acceptable behavior or does it need fixing?
Best regards,
Sai SnehaQuoted 26 lines · Richard Purdie
Quoted 11 lines · Sai Sneha
I tested BB_NO_NETWORK:layer-core = "1" and found that it does not prevent the parse crash for the case where SRCREV is completely missing.
The crash happens in
srcrev_internal_helper()which raises FetchError before any network access is attempted — soBB_NO_NETWORKnever gets checked. TheBB_NO_NETWORKflag would only help for the case where SRCREV is set to a floating branch/tag andlatest_revision()is called, since_lsremote()callscheck_network_access()there.FetchError isn't a crash, the code is finding there is a problem and raising an exception. The exception should say what the issue is.
Doesn't that mean that case is already covered and we don't need to do anything further?
Quoted 6 lines
So it seems like two separate fixes are needed:
1. For missing SRCREV — something needs to prevent the parse crash in
srcrev_internal_helper(). Would it be appropriate to checkBB_NO_NETWORKinsrcrev_internal_helper()itself before raising FetchError?Quoted 7 lines
I don't understand why we need to trap this, then handle it again later?
Cheers,
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#238324):
https://lists.openembedded.org/g/openembedded-core/message/238324
Mute This Topic: https://lists.openembedded.org/mt/119719972/10302268
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
PATCH v5 4/4 yocto-check-layer.bbclass: enforce missing SRCREV check during layer validation
In reply to Sai SnehaAdd check_missing_srcrev() to yocto-check-layer.bbclass to enforce the missing-srcrev check as an error during yocto-check-layer runs, using the shared oe.qa.check_uri_srcrev() helper.
This ensures layer maintainers submitting layers to the Yocto ecosystem must fix all missing SRCREVs before their layer passes validation, even if their layer only gets WARN_QA during normal builds.
Note on design: missing-srcrev is not added to CHECKLAYER_REQUIRED_TESTS because that would promote it to ERROR_QA for all layers globally during normal builds, contradicting the layer-specific enforcement requested in Bug 16051 Comment 3. This direct implementation in yocto-check-layer.bbclass enforces it strictly during checklayer validation without affecting normal builds.
This satisfies the requirement from Bug 16051 Comment 2:
'add a QA check outside of checklayer then add it to checklayer'
AI-Generated: Developed with assistance from Anthropic Claude
---
Changes in v5:
- No changes to this patch
Changes in v4:
- Refactored to use shared oe.qa.check_uri_srcrev() helper
- Eliminates duplicated URI parsing logic
- Added docstring
Changes in v3:
- Added AI-Generated disclosure and Reported-by tag
Changes in v2:
- Initial public submission
meta/classes-global/yocto-check-layer.bbclass | 15 +++++++++++++++ meta/lib/oe/qa.py | 2 +- 2 files changed, 16 insertions(+), 1 deletion(-)
meta/classes-global/yocto-check-layer.bbclass +15 -0
@@ -56,7 +56,22 @@ def check_network_flag(d): if network and not is_allowed(bpn, task): bb.error(f"QA Issue: task {task} has network enabled") +def check_missing_srcrev(d): + """ + Enforce missing-srcrev check as error during yocto-check-layer runs. + Uses the shared oe.qa.check_uri_srcrev() helper. + """ + import oe.qa + src_uri = (d.getVar('SRC_URI', False) or '').split() + pn = d.getVar('PN') + for uri in src_uri: + rev = oe.qa.check_uri_srcrev(pn, uri, d) + if rev is None: + bb.error("QA Issue: %s: SRCREV not set for SCM URI %s [missing-srcrev]" % (pn, uri)) + d.setVar("QA_ERRORS_FOUND", "True") + python () { check_insane_skip(d) check_network_flag(d) + check_missing_srcrev(d) }
meta/lib/oe/qa.py +1 -1
@@ -245,7 +245,7 @@ def check_uri_srcrev(pn, uri, d): Check that a single SCM URI has a valid SRCREV set. Returns the resolved revision string if valid (including '${AUTOREV}'). - Returns '' for non-SCM URIs or URIs with inline rev= parameter (skip). + Returns '' for non-SCM URIs or URIs with inline rev= or tag= parameter (skip). Returns None if SRCREV is missing or INVALID. """ import bb.fetch2
Signature · 1 line
2.34.1
Quoted 10 lines · Sai Sneha via lists.openembedded.org
This patch series adds a check for
git://(and other SCM) URIs inSRC_URIthat are missing a corresponding SRCREV, addressing Bugzilla #16051 [1] and referencing Corentin Guillevic's RFC series [2].== Changes since v4 ==
- Add tag= to skip condition in
oe.qa.check_uri_srcrev()alongside the existing rev= handling. This fixes a selftest breakage (test_git_unpack_nonetwork) reported by Mathieu Dubois-Briand where recipes using tag= inSRC_URIintentionally leave SRCREV empty.
We cannot just skip when we see a tag=... parameters. The tag would still need to be looked up in the remote repository at parse time.
Quoted 7 lines
- Update docstring to mention tag= parameter
- Verified by running full bbtests.BitbakeTests selftest suite:
32/33 passed, 1 failure (test_image_manifest) is unrelated as it requires building a full image and fails due to network constraints== Note on
CHECKLAYER_REQUIRED_TESTS==
Quoted 7 lines
Paul suggested using
CHECKLAYER_REQUIRED_TESTSinstead of Patch 4. We tested this but found it promotes missing-srcrev toERROR_QAfor ALL layers globally during normal builds, becauseCHECKLAYER_REQUIRED_TESTSfeeds directly intoERROR_QA. This contradicts Bug 16051 Comment 3 which requests keeping it as a warning for external layers by default. The direct implementation in yocto-check-layer.bbclass enforces it strictly only during checklayer validation without affecting normal builds.
The correct way to handle this is via a discussion instead of just re-prompting an LLM. The comments in bugzilla are not requirements, and some human discussion is probably needed to arrive at a resolution.
Quoted 11 lines
== Note on candidates list order ==
Martin suggested reversing the candidates list so plain SRCREV is tested first. We kept most-specific-first order because it mirrors BitBake internal
srcrev_internal_helper()resolution order exactly, as confirmed by BitBake own error messages:
SRCREV_default:pn-X,SRCREV_default, SRCREV:pn-X, SRCREV== Problem ==
Quoted 8 lines
A recipe with a
git://URI and no SRCREV causes two problems:1. Under BB_NO_NETWORK=1, parsing fails with a FetchError that halts parsing of ALL recipes. This breaks CI pipelines that use bitbake --parse-only as a sanity check.
2. Without
BB_NO_NETWORK, BitBake silently queries the remote repository on every parse, breaking reproducibility.
These issues also occur when SRCREV is set to a branch or tag instead of an explicit commit hash.
Quoted 10 lines
== Why the trivial fix does not work ==
The obvious fix of checking SRCREV in insane.bbclass
do_recipe_qawas attempted by Corentin Guillevic [2] but fails because fetcher_hashes_dummyfunc[vardepvalue] expands SRCREV at parse time BEFORE QA checks run. Corentin patches 1 and 2 add the QA check, while patches 3 and 4 ensure devtool and recipetool only use AUTOREV when appropriate.
Quoted 5 lines
== This series approach ==
We intercept at the vardepvalue expansion point in base.bbclass using a shared helper
oe.qa.check_uri_srcrev()that usesd.getVar(candidate, False)preventing expansion entirely.
I recommend being highly skeptical of AI generating code which modifies fairly low-level things like vardeps or vardepvalue flags. They are powerful, but should not be the first thing you reach for when trying to solve a problem unless you understand fully what they are doing.
Quoted 5 lines
== Design decisions ==
1. Why not use
srcrev_internal_helper()
Quoted 8 lines
It expands SRCREV internally, triggering
get_autorev()and live network fetches. Our fallback chain mirrors BitBake resolution order exactly without expansion.2. Why duplicate warnings at parse time AND QA time
Parse-time catches CI parse-only pipelines. QA-time gives proper layer control. This pattern is established - src-uri-bad does same.
Where is this pattern established? src-uri-bad is just handled as a regular QA check in do_recipe_qa().
Quoted 4 lines
3. Why hardcode git/gitsm/hg/svn schemes
Known limitation, noted as follow-on improvement.
"Known limitation" is not an answer to the question "Why?".
Quoted 33 lines
== Enforcement levels ==
Normal builds, all layers -> WARN_QA oe-core layer -> ERROR_QA yocto-check-layer runs -> ERROR== Testing ==
- Recipe with missing SRCREV -> WARNING/ERROR at parse time
- Recipe with SRCREV = AUTOREV -> silent passthrough
- Recipe with rev= in URI -> no false positive
- Recipe with multiple SCM URIs -> all missing SRCREVs reported
- oe-core recipe -> ERROR instead of WARNING
- yocto-check-layer on layer with missing SRCREV -> FAIL
- 952 other recipes parse cleanly -> 0 errors[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=16051
[2] https://lore.kernel.org/openembedded-core/[email protected]/
[3] https://lists.openembedded.org/g/openembedded-devel/topic/115911777Sai Sneha (4):
oe/qa.py: addcheck_uri_srcrev()helper for SCM URI SRCREV validation base.bbclass: warn when SRCREV is missing for SCM URIs at parse time insane.bbclass: add missing-srcrev QA check for SCM URIs yocto-check-layer.bbclass: enforce missing SRCREV check during layer validationmeta/classes-global/base.bbclass | 28 ++++++++++ meta/classes-global/insane.bbclass | 13 +++++ meta/classes-global/yocto-check-layer.bbclass | 15 ++++++ meta/lib/oe/qa.py | 30 +++++++++++ 4 files changed, 86 insertions(+), 2 deletions(-)
Quoted 5 lines
--
Note: This patch series was developed with assistance from Anthropic Claude (AI). All code has been reviewed, tested, and verified by the contributor. The Signed-off-by line represents the contributor own certification per the Developer Certificate of Origin.
Best regards,
Signature · 1 line
Paul Barker
Quoted 67 lines · Sai Sneha via lists.openembedded.org
This patch series adds a check for
git://(and other SCM) URIs inSRC_URIthat are missing a corresponding SRCREV, addressing Bugzilla #16051 [1] and referencing Corentin Guillevic's RFC series [2].== Changes since v4 ==
- Add tag= to skip condition in
oe.qa.check_uri_srcrev()alongside the existing rev= handling. This fixes a selftest breakage (test_git_unpack_nonetwork) reported by Mathieu Dubois-Briand where recipes using tag= inSRC_URIintentionally leave SRCREV empty.
- Update docstring to mention tag= parameter
- Verified by running full bbtests.BitbakeTests selftest suite:
32/33 passed, 1 failure (test_image_manifest) is unrelated as it requires building a full image and fails due to network constraints== Note on
CHECKLAYER_REQUIRED_TESTS==Paul suggested using
CHECKLAYER_REQUIRED_TESTSinstead of Patch 4. We tested this but found it promotes missing-srcrev toERROR_QAfor ALL layers globally during normal builds, becauseCHECKLAYER_REQUIRED_TESTSfeeds directly intoERROR_QA. This contradicts Bug 16051 Comment 3 which requests keeping it as a warning for external layers by default. The direct implementation in yocto-check-layer.bbclass enforces it strictly only during checklayer validation without affecting normal builds.== Note on candidates list order ==
Martin suggested reversing the candidates list so plain SRCREV is tested first. We kept most-specific-first order because it mirrors BitBake internal
srcrev_internal_helper()resolution order exactly, as confirmed by BitBake own error messages:
SRCREV_default:pn-X,SRCREV_default, SRCREV:pn-X, SRCREV== Problem ==
A recipe with a
git://URI and no SRCREV causes two problems:1. Under BB_NO_NETWORK=1, parsing fails with a FetchError that halts parsing of ALL recipes. This breaks CI pipelines that use bitbake --parse-only as a sanity check.
2. Without
BB_NO_NETWORK, BitBake silently queries the remote repository on every parse, breaking reproducibility.== Why the trivial fix does not work ==
The obvious fix of checking SRCREV in insane.bbclass
do_recipe_qawas attempted by Corentin Guillevic [2] but fails because fetcher_hashes_dummyfunc[vardepvalue] expands SRCREV at parse time BEFORE QA checks run. Corentin patches 1 and 2 add the QA check, while patches 3 and 4 ensure devtool and recipetool only use AUTOREV when appropriate.== This series approach ==
We intercept at the vardepvalue expansion point in base.bbclass using a shared helper
oe.qa.check_uri_srcrev()that usesd.getVar(candidate, False)preventing expansion entirely.== Design decisions ==
1. Why not use
srcrev_internal_helper()It expands SRCREV internally, triggering
get_autorev()and live network fetches. Our fallback chain mirrors BitBake resolution order exactly without expansion.
Quoted 8 lines
2. Why duplicate warnings at parse time AND QA time
Parse-time catches CI parse-only pipelines. QA-time gives proper layer control. This pattern is established - src-uri-bad does same.
3. Why hardcode git/gitsm/hg/svn schemes
Known limitation, noted as follow-on improvement.
Why don't we do something in the bitbake fetchers which would check if they would need a srcrev lookup and fail if that were the case?
We could set something with a layer override which would then trigger any recipe in core to fail to parse if they weren't set deterministically.
Looking at the patches, I'm worried we're duplicating fetcher logic in OE-Core and we definitely don't want to be doing that!
Cheers,
Richard