Skip to contentSkip to conversation

devtool: fix standalone clone conversion for nested git repos

PATCH In review Jamin Lin 2 participants 14 messages Jul 31 – Aug 17

lore
Share & tools
lore

Jamin Lin

author
#0 lore raw

v4 cover devtool: fix standalone clone conversion for nested git repos

When a recipe has multiple git SRC_URI entries with destsuffix values nested inside S, devtool modify left all but the top-level repo with a .git/objects/info/alternates file pointing into the downloads cache.

Running 'bitbake -c cleanall' then removed those shared objects, breaking all subsequent git operations in the workspace.
Fix git_convert_standalone_clone() to walk all git repos nested inside S and repack each one to a fully standalone clone.

v1:
- Fix git_convert_standalone_clone() to walk all git repos nested inside S and repack each one to a standalone clone
v2:
- Add selftest recipe devtool-test-multi-destsuffix with six nested git SRC_URI entries to reproduce the scenario
- Add test_devtool_modify_multi_git_destsuffix_standalone to verify all nested repos have their alternates removed after devtool modify and remain usable after 'bitbake -c cleanall'
v3:
- Read SRC_URI directly instead of walking the tree for '.git' dirs - also fixes repos nested inside another repo's own working tree
- Fix setup_git_repo() to register nested repos as submodules before the initial commit (was dead code), processing bottom-up to avoid stale submodule references
- Fix commitIgnored() to skip committing when nothing is staged, so do_patch no longer fails on a submodule that's dirty only because of its own nested content
- Simplify test recipe to three genuinely-nested destsuffix entries (level1/level1/level2/level1/level2/level3), update selftest to match
v4:
- Fix regression from v3:
setup_git_repo() now detects nested repos both for fresh workspace repos and for recipes whose source is already a git repo (fixes test_devtool_modify_nested_gitsm).

Jamin Lin (5):
oe/patch: Skip commitIgnored when nothing is actually staged
devtool: Register nested git repos before the initial commit
devtool-source: Make nested destsuffix git repos standalone
meta-selftest: Add devtool-test-multi-destsuffix recipe oeqa/selftest/devtool: Add test for multiple nested git destsuffix
repos

 .../devtool-test-multi-destsuffix_git.bb      | 27 +++++++
 meta/classes/devtool-source.bbclass           | 16 ++++
 meta/lib/oe/patch.py                          | 12 +++
 meta/lib/oeqa/selftest/cases/devtool.py       | 64 +++++++++++++++
 scripts/lib/devtool/__init__.py               | 79 +++++++++++++------
 5 files changed, 175 insertions(+), 23 deletions(-)
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb
Signature · 1 line
2.43.0
author
#1 lore raw

PATCH v4 1/5 oe/patch: Skip commitIgnored when nothing is actually staged

patch.bbclass's patch_task_postfunc calls GitApplyTree.commitIgnored() whenever 'git status --porcelain .' reports the source tree as dirty after do_patch, to snapshot those changes into the devtool tracking repo with 'git add' + 'git commit'.

A path can be reported as dirty by git status purely because it is a submodule (or an unregistered embedded git repo) that itself has modified or untracked content - for example a recipe with multiple git SRC_URI entries where one destsuffix places a repo inside another repo's own working tree. The outer repo's tracked commit hash for that submodule hasn't changed, so 'git add' has nothing new to stage for it, and git refuses to fold the submodule's own dirty state into a plain commit without it being resolved first:

  $ git commit -m ... --no-verify --no-gpg-sign
  Changes not staged for commit:
    (commit or discard the untracked or modified content in submodules)
          modified:   level1 (modified content)
  no changes added to commit (use "git add" and/or "git commit -a")

'git commit' then exits non-zero with nothing to commit, and commitIgnored() propagates that failure straight up, taking the whole do_patch task down with it.

Fix by checking 'git diff --cached --name-only' after 'git add': if nothing was actually staged, there is nothing meaningful to snapshot, so skip the commit (and the note it would otherwise add) instead of failing.

---
 meta/lib/oe/patch.py | 12 ++++++++++++
 1 file changed, 12 insertions(+)
meta/lib/oe/patch.py +12 -0
index 1d50e83ab7..944dde1e03 100644
@@ -516,6 +516,18 @@ class GitApplyTree(PatchTree):
     def commitIgnored(subject, dir=None, files=None, d=None):
         if files:
             runcmd(['git', 'add'] + files, dir)
+
+        # 'git add' can leave nothing actually staged even though the caller
+        # saw a dirty status: a path can show as modified purely because it
+        # is a submodule/embedded git repository with modified or untracked
+        # content of its own (e.g. a further nested git repo from another
+        # destsuffix SRC_URI entry) - git refuses to record that via a plain
+        # 'git add'/'git commit' without resolving the submodule's own state,
+        # so the commit below would fail with "no changes added to commit".
+        # Skip the commit if there is nothing actually staged.
+        if not runcmd(['git', 'diff', '--cached', '--name-only'], dir).strip():
+            return
+
         cmd = ["git"]
         GitApplyTree.gitCommandUserOptions(cmd, d=d)
         cmd += ["commit", "-m", subject, "--no-verify", "--no-gpg-sign"]
Signature · 1 line
2.43.0
#2 lore raw

Re: [PATCH v4 1/5] oe/patch: Skip commitIgnored when nothing is actually staged

Quoted 43 lines · Jamin Lin

patch.bbclass's patch_task_postfunc calls GitApplyTree.commitIgnored() whenever 'git status --porcelain .' reports the source tree as dirty after do_patch, to snapshot those changes into the devtool tracking repo with 'git add' + 'git commit'.

A path can be reported as dirty by git status purely because it is a submodule (or an unregistered embedded git repo) that itself has modified or untracked content - for example a recipe with multiple git SRC_URI entries where one destsuffix places a repo inside another repo's own working tree. The outer repo's tracked commit hash for that submodule hasn't changed, so 'git add' has nothing new to stage for it, and git refuses to fold the submodule's own dirty state into a plain commit without it being resolved first:

  $ git commit -m ... --no-verify --no-gpg-sign
  Changes not staged for commit:
    (commit or discard the untracked or modified content in submodules)
          modified:   level1 (modified content)
  no changes added to commit (use "git add" and/or "git commit -a")

'git commit' then exits non-zero with nothing to commit, and commitIgnored() propagates that failure straight up, taking the whole do_patch task down with it.

Fix by checking 'git diff --cached --name-only' after 'git add': if nothing was actually staged, there is nothing meaningful to snapshot, so skip the commit (and the note it would otherwise add) instead of failing.

---
 meta/lib/oe/patch.py | 12 ++++++++++++
 1 file changed, 12 insertions(+)
meta/lib/oe/patch.py +1 -0
index 1d50e83ab7..944dde1e03 100644
@@ -516,6 +516,18 @@ class GitApplyTree(PatchTree):
     def commitIgnored(subject, dir=None, files=None, d=None):
         if files:
             runcmd(['git', 'add'] + files, dir)
+

meta/lib/oe/patch.py class GitApplyTree(PatchTree): · line 529

Quoted 10 lines
meta/lib/oe/patch.py +10 -0
+        # 'git add' can leave nothing actually staged even though the caller
+        # saw a dirty status: a path can show as modified purely because it
+        # is a submodule/embedded git repository with modified or untracked
+        # content of its own (e.g. a further nested git repo from another
+        # destsuffix SRC_URI entry) - git refuses to record that via a plain
+        # 'git add'/'git commit' without resolving the submodule's own state,
+        # so the commit below would fail with "no changes added to commit".
+        # Skip the commit if there is nothing actually staged.
+        if not runcmd(['git', 'diff', '--cached', '--name-only'], dir).strip():
+            return

This is a lot of text for a simple message, was this AI generated?

We can just say "Skip the commit if there is nothing actually staged" here.

Best regards,

Signature · 1 line
Paul Barker
author
#3 lore raw

RE: [PATCH v4 1/5] oe/patch: Skip commitIgnored when nothing is actually staged

Quoted 63 lines

-----Original Message-----
From: Paul Barker <[email protected]>
Sent: Sunday, August 16, 2026 6:50 PM
To: Jamin Lin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Troy Lee <[email protected]>
Subject: Re: [PATCH v4 1/5] oe/patch: Skip commitIgnored when nothing is actually staged

Quoted 39 lines · Jamin Lin

patch.bbclass's patch_task_postfunc calls GitApplyTree.commitIgnored() whenever 'git status --porcelain .' reports the source tree as dirty after do_patch, to snapshot those changes into the devtool tracking repo with 'git add' + 'git commit'.

A path can be reported as dirty by git status purely because it is a submodule (or an unregistered embedded git repo) that itself has modified or untracked content - for example a recipe with multiple git SRC_URI entries where one destsuffix places a repo inside another repo's own working tree. The outer repo's tracked commit hash for that submodule hasn't changed, so 'git add' has nothing new to stage for it, and git refuses to fold the submodule's own dirty state into a plain commit without it being resolved first:

  $ git commit -m ... --no-verify --no-gpg-sign
  Changes not staged for commit:
    (commit or discard the untracked or modified content in submodules)
          modified:   level1 (modified content)
  no changes added to commit (use "git add" and/or "git commit -a")

'git commit' then exits non-zero with nothing to commit, and commitIgnored() propagates that failure straight up, taking the whole do_patch task down with it.

Fix by checking 'git diff --cached --name-only' after 'git add': if nothing was actually staged, there is nothing meaningful to snapshot, so skip the commit (and the note it would otherwise add) instead of failing.

---
 meta/lib/oe/patch.py | 12 ++++++++++++
 1 file changed, 12 insertions(+)
meta/lib/oe/patch.py index renamed from meta/lib/oe/patch.py +0 -0

1d50e83ab7..944dde1e03 100644

meta/lib/oe/patch.py class GitApplyTree(PatchTree): · line 520

Quoted 5 lines
meta/lib/oe/patch.py +2 -0
@@ -516,6 +516,18 @@ class GitApplyTree(PatchTree):
     def commitIgnored(subject, dir=None, files=None, d=None):
         if files:
             runcmd(['git', 'add'] + files, dir)
+
+        # 'git add' can leave nothing actually staged even though the

caller

meta/lib/oe/patch.py class GitApplyTree(PatchTree): · line 521

Quoted 1 line
meta/lib/oe/patch.py +1 -0
+        # saw a dirty status: a path can show as modified purely because

it

meta/lib/oe/patch.py class GitApplyTree(PatchTree): · line 522

Quoted 1 line
meta/lib/oe/patch.py +1 -0
+        # is a submodule/embedded git repository with modified or

untracked

meta/lib/oe/patch.py class GitApplyTree(PatchTree):

Quoted 3 lines
meta/lib/oe/patch.py +3 -0
+        # content of its own (e.g. a further nested git repo from another
+        # destsuffix SRC_URI entry) - git refuses to record that via a plain
+        # 'git add'/'git commit' without resolving the submodule's own
Quoted 9 lines

state,

meta/lib/oe/patch.py class GitApplyTree(PatchTree): · line 526

Quoted 1 line
meta/lib/oe/patch.py +1 -0
+        # so the commit below would fail with "no changes added to

commit".

meta/lib/oe/patch.py class GitApplyTree(PatchTree): · line 529

Quoted 3 lines
meta/lib/oe/patch.py +3 -0
+        # Skip the commit if there is nothing actually staged.
+        if not runcmd(['git', 'diff', '--cached', '--name-only'], dir).strip():
+            return

This is a lot of text for a simple message, was this AI generated?

Yes, will add AI-Generated: Uses Claud

Quoted 3 lines

We can just say "Skip the commit if there is nothing actually staged" here.

Will do.

Quoted 4 lines

Best regards,

--
Paul Barker

author
#4 lore raw

PATCH v4 2/5 devtool: Register nested git repos before the initial commit

In reply to Jamin Lin

setup_git_repo() is meant to convert a git repo that a recipe unpacks inside S (e.g. via multiple git SRC_URI entries with different destsuffix values) into a regular git submodule, so devtool can later tag branches on it and extract patches from it via finish/update.

That detection never actually triggered, because of the order the function ran things in when it had to create the workspace repo itself:

1. 'git init'
2. 'git add -A .' + initial commit <- commits the nested repo as a bare, unregistered gitlink
3. checkout devbranch, tag basetag
4. scan 'git status --porcelain' for still-untracked directories ("?? <dir>/") and convert any that are git repos into submodules

By the time step 4 ran, the nested repo had already been swept up by step 2's 'git add -A .': git treats a directory containing its own .git as an embedded repo and stages it as a gitlink pointing at its current HEAD, without registering it as a submodule. Once that gitlink is committed, 'git status --porcelain' reports it as e.g. " M <dir>" (already tracked) rather than "?? <dir>/" (untracked), so step 4's "line.endswith('/')" check could never match it and the conversion to a real submodule silently never happened.

There is also a second entry path with the same root cause: when the recipe's top-level source is itself fetched via git://, repodir is already a git repo, so the 'if not .git' block above (init + initial commit) is skipped entirely - and so was the detection that lived inside it. In that case the nested repo instead gets committed as a bare gitlink later, by patch_task_postfunc's 'git add' after do_patch.

Fix this by extracting the detection into a helper and calling it before anything can commit the nested repo as a bare gitlink, in both cases:
- freshly-created workspace repo: right after 'git init', before

    'git add -A .' and the initial commit;

- repodir already a git repo: at function entry, before the later 'git add' in patch_task_postfunc.
At those points the nested repo is still untracked and reported with a trailing "/", so it is correctly picked up and registered via 'git submodule add'.

Nested repos are discovered top-down (so a repo that manages its own submodules via .gitmodules can be skipped rather than descended into), but registered bottom-up (deepest first): a parent's commit recording its child's HEAD must happen after that child is fully finalized, otherwise registering a still-deeper repo afterwards moves the child's HEAD forward again and leaves the parent pointing at a stale revision.

---
 scripts/lib/devtool/__init__.py | 79 +++++++++++++++++++++++----------
 1 file changed, 56 insertions(+), 23 deletions(-)
scripts/lib/devtool/__init__.py +56 -23
index 58b02eb460..030ba7edd9 100644
@@ -197,9 +197,59 @@ def setup_git_repo(repodir, version, devbranch, basetag='devtool-base', d=None):
     """
     import bb.process
     import oe.patch
+
+    def register_nested_git_submodules():
+        # If the recipe unpacks another git repo inside S (e.g. multiple git
+        # SRC_URI entries with destsuffix), declare it as a regular git
+        # submodule now, so we will be able to tag branches on it and extract
+        # patches when doing finish/update on the recipe. This must happen
+        # before anything else commits that nested repo as a bare,
+        # unregistered gitlink: once that happens 'git status' no longer
+        # reports it as untracked ("?? <dir>/"), so this detection can never
+        # find it. That can happen either from 'git add -A .' below (for a
+        # freshly-initialized repo) or, when repodir is already its own git
+        # repo (e.g. a recipe fetched via plain git://), from a later 'git
+        # add' done elsewhere (patch_task_postfunc, after do_patch) - so this
+        # is called both from the fresh-repo branch below and from the
+        # already-a-repo branch, before either has a chance to do that.
+        #
+        # Discover nested repos top-down (so we can still skip descending into
+        # a repo that manages its own submodules via .gitmodules), but do the
+        # actual 'git submodule add' + commit bottom-up (deepest repo first):
+        # a parent's commit recording its child's current HEAD must happen
+        # after that child is fully finalized, otherwise a deeper repo added
+        # later on gets its own registration commit, moving the child's HEAD
+        # forward again and leaving the parent's already-made commit pointing
+        # at a stale, superseded revision of it.
+        stdout, _ = bb.process.run("git status --porcelain", cwd=repodir)
+        nested_repos = []
+        for line in stdout.splitlines():
+            if line.endswith("/"):
+                new_dir = line.split()[1]
+                for root, dirs, files in os.walk(os.path.join(repodir, new_dir)):
+                    if ".git" in dirs + files:
+                        nested_repos.append(root)
+                        # Do not descend into nested git repos that have submodules themselves.
+                        if ".gitmodules" in files:
+                            logger.warning('Nested git repository with submodules %s; devtool will not recurse into it', root)
+                            dirs[:] = []
+
+        for root in reversed(nested_repos):
+            parentdir = os.path.join(root, "..")
+            (stdout, _) = bb.process.run('git remote', cwd=root)
+            remote = stdout.splitlines()[0]
+            (stdout, _) = bb.process.run('git remote get-url %s' % remote, cwd=root)
+            remote_url = stdout.splitlines()[0]
+            logger.error(os.path.relpath(parentdir, root))
+            bb.process.run('git submodule add %s %s' % (remote_url, os.path.relpath(root, parentdir)), cwd=parentdir)
+            oe.patch.GitApplyTree.commitIgnored("Add additional submodule from SRC_URI", dir=parentdir, d=d)
+
     if not os.path.exists(os.path.join(repodir, '.git')):
         bb.process.run('git init', cwd=repodir)
         bb.process.run('git config --local gc.autodetach 0', cwd=repodir)
+
+        register_nested_git_submodules()
+
         bb.process.run('git add -f -A .', cwd=repodir)
         commit_cmd = ['git']
         oe.patch.GitApplyTree.gitCommandUserOptions(commit_cmd, d=d)
@@ -214,6 +264,12 @@ def setup_git_repo(repodir, version, devbranch, basetag='devtool-base', d=None):
             commitmsg = "Initial commit from upstream"
         commit_cmd += ['-m', commitmsg]
         bb.process.run(commit_cmd, cwd=repodir)
+    else:
+        # repodir is already a git repo in its own right (e.g. a recipe whose
+        # top-level source is fetched via plain git://), so there was no
+        # fresh init/initial commit above to interfere with detecting nested
+        # repos - do it here instead, at the earliest point available.
+        register_nested_git_submodules()
 
     # Ensure singletask.lock (as used by externalsrc.bbclass) is ignored by git
     gitinfodir = os.path.join(repodir, '.git', 'info')
@@ -237,29 +293,6 @@ def setup_git_repo(repodir, version, devbranch, basetag='devtool-base', d=None):
     bb.process.run('git checkout -b %s' % devbranch, cwd=repodir)
     bb.process.run('git tag -f --no-sign %s' % basetag, cwd=repodir)
 
-    # if recipe unpacks another git repo inside S, we need to declare it as a regular git submodule now,
-    # so we will be able to tag branches on it and extract patches when doing finish/update on the recipe
-    stdout, _ = bb.process.run("git status --porcelain", cwd=repodir)
-    found = False
-    for line in stdout.splitlines():
-        if line.endswith("/"):
-            new_dir = line.split()[1]
-            for root, dirs, files in os.walk(os.path.join(repodir, new_dir)):
-                if ".git" in dirs + files:
-                    (stdout, _) = bb.process.run('git remote', cwd=root)
-                    remote = stdout.splitlines()[0]
-                    (stdout, _) = bb.process.run('git remote get-url %s' % remote, cwd=root)
-                    remote_url = stdout.splitlines()[0]
-                    logger.error(os.path.relpath(os.path.join(root, ".."), root))
-                    bb.process.run('git submodule add %s %s' % (remote_url, os.path.relpath(root, os.path.join(root, ".."))), cwd=os.path.join(root, ".."))
-                    # Do not descend into nested git repos that have submodules themselves.
-                    if ".gitmodules" in files:
-                        logger.warning('Nested git repository with submodules %s; devtool will not recurse into it', root)
-                        dirs[:] = []
-                    found = True
-                if found:
-                    oe.patch.GitApplyTree.commitIgnored("Add additional submodule from SRC_URI", dir=os.path.join(root, ".."), d=d)
-                    found = False
     if os.path.exists(os.path.join(repodir, '.gitmodules')):
         bb.process.run('git submodule foreach --recursive  "git tag -f --no-sign %s"' % basetag, cwd=repodir)
 
Signature · 1 line
2.43.0
#5 lore raw

Re: [PATCH v4 2/5] devtool: Register nested git repos before the initial commit

Quoted 41 lines · Jamin Lin

setup_git_repo() is meant to convert a git repo that a recipe unpacks inside S (e.g. via multiple git SRC_URI entries with different destsuffix values) into a regular git submodule, so devtool can later tag branches on it and extract patches from it via finish/update.

That detection never actually triggered, because of the order the function ran things in when it had to create the workspace repo itself:

1. 'git init'
2. 'git add -A .' + initial commit <- commits the nested repo as a bare, unregistered gitlink
3. checkout devbranch, tag basetag
4. scan 'git status --porcelain' for still-untracked directories ("?? <dir>/") and convert any that are git repos into submodules

By the time step 4 ran, the nested repo had already been swept up by step 2's 'git add -A .': git treats a directory containing its own .git as an embedded repo and stages it as a gitlink pointing at its current HEAD, without registering it as a submodule. Once that gitlink is committed, 'git status --porcelain' reports it as e.g. " M <dir>" (already tracked) rather than "?? <dir>/" (untracked), so step 4's "line.endswith('/')" check could never match it and the conversion to a real submodule silently never happened.

There is also a second entry path with the same root cause: when the recipe's top-level source is itself fetched via git://, repodir is already a git repo, so the 'if not .git' block above (init + initial commit) is skipped entirely - and so was the detection that lived inside it. In that case the nested repo instead gets committed as a bare gitlink later, by patch_task_postfunc's 'git add' after do_patch.

Fix this by extracting the detection into a helper and calling it before anything can commit the nested repo as a bare gitlink, in both cases:
- freshly-created workspace repo: right after 'git init', before

    'git add -A .' and the initial commit;

- repodir already a git repo: at function entry, before the later 'git add' in patch_task_postfunc.
At those points the nested repo is still untracked and reported with a trailing "/", so it is correctly picked up and registered via 'git submodule add'.

Quoted 8 lines

Nested repos are discovered top-down (so a repo that manages its own submodules via .gitmodules can be skipped rather than descended into), but registered bottom-up (deepest first): a parent's commit recording its child's HEAD must happen after that child is fully finalized, otherwise registering a still-deeper repo afterwards moves the child's HEAD forward again and leaves the parent pointing at a stale revision.

This commit is doing three things:
- Refactoring existing code into register_nested_git_submodules()
- Modifying the refactored code
- Moving the call site earlier

It's very hard to review this commit and be confident that it is correct. This should be split into a two or three logical steps with simpler commit messages.

The wall-of-text commit message and comment suggests to me this may be generated with AI. Have you read the contributor guide [1]?

[1]: https://docs.yoctoproject.org/contributor-guide/submit-changes.html#acceptance-of-ai-generated-code

Best regards,

Signature · 1 line
Paul Barker
author
#6 lore raw

RE: [PATCH v4 2/5] devtool: Register nested git repos before the initial commit

Quoted 59 lines

Subject: Re: [PATCH v4 2/5] devtool: Register nested git repos before the initial commit

Quoted 41 lines · Jamin Lin

setup_git_repo() is meant to convert a git repo that a recipe unpacks inside S (e.g. via multiple git SRC_URI entries with different destsuffix values) into a regular git submodule, so devtool can later tag branches on it and extract patches from it via finish/update.

That detection never actually triggered, because of the order the function ran things in when it had to create the workspace repo itself:

1. 'git init'
2. 'git add -A .' + initial commit <- commits the nested repo as a bare, unregistered gitlink
3. checkout devbranch, tag basetag
4. scan 'git status --porcelain' for still-untracked directories ("?? <dir>/") and convert any that are git repos into submodules

By the time step 4 ran, the nested repo had already been swept up by step 2's 'git add -A .': git treats a directory containing its own .git as an embedded repo and stages it as a gitlink pointing at its current HEAD, without registering it as a submodule. Once that gitlink is committed, 'git status --porcelain' reports it as e.g. " M <dir>" (already tracked) rather than "?? <dir>/" (untracked), so step 4's "line.endswith('/')" check could never match it and the conversion to a real submodule silently never happened.

There is also a second entry path with the same root cause: when the recipe's top-level source is itself fetched via git://, repodir is already a git repo, so the 'if not .git' block above (init + initial commit) is skipped entirely - and so was the detection that lived inside it. In that case the nested repo instead gets committed as a bare gitlink later, by patch_task_postfunc's 'git add' after do_patch.

Fix this by extracting the detection into a helper and calling it before anything can commit the nested repo as a bare gitlink, in both cases:
- freshly-created workspace repo: right after 'git init', before

    'git add -A .' and the initial commit;

- repodir already a git repo: at function entry, before the later 'git add' in patch_task_postfunc.
At those points the nested repo is still untracked and reported with a trailing "/", so it is correctly picked up and registered via 'git submodule add'.

Quoted 8 lines

Nested repos are discovered top-down (so a repo that manages its own submodules via .gitmodules can be skipped rather than descended into), but registered bottom-up (deepest first): a parent's commit recording its child's HEAD must happen after that child is fully finalized, otherwise registering a still-deeper repo afterwards moves the child's HEAD forward again and leaves the parent pointing at a stale revision.

This commit is doing three things:
- Refactoring existing code into register_nested_git_submodules()
- Modifying the refactored code
- Moving the call site earlier

Quoted 6 lines

It's very hard to review this commit and be confident that it is correct. This should be split into a two or three logical steps with simpler commit messages.

The wall-of-text commit message and comment suggests to me this may be generated with AI. Have you read the contributor guide [1]?

After I read this contribution guide, will add
AI-Generated: Uses Claud in commit message.

Quoted 8 lines

[1]:
https://docs.yoctoproject.org/contributor-guide/submit-changes.html#acceptan
ce-of-ai-generated-code

Best regards,

--
Paul Barker

author
#7 lore raw

PATCH v4 3/5 devtool-source: Make nested destsuffix git repos standalone

In reply to Jamin Lin

When a recipe uses multiple git SRC_URI entries with different destsuffix values (e.g. recipes with separate repositories for the kernel, modules and application), do_unpack clones each source tree with
'git clone -n -s'.

The -s flag uses git's shared-object mechanism:
instead of copying objects locally it writes a .git/objects/info/alternates file pointing back to the bare repository under the downloads directory (DL_DIR/git2/).

scriptutils.git_convert_standalone_clone() is called by devtool_post_unpack to make the top-level source directory standalone: it runs 'git repack -a' to copy all objects into the local object store and then removes the alternates file.

However it only processes the top-level source directory. Each nested git repo created by a separate SRC_URI entry retains its own alternates file still pointing into downloads/.

Steps to reproduce:
1. devtool modify <recipe-with-multiple-git-SRC_URI>
2. bitbake -c cleanall <recipe>
3. bitbake <recipe>

At step 2, 'bitbake -c cleanall' calls fetcher.clean() which deletes the bare repositories from downloads/git2/. The top-level workspace repo is standalone (alternates already removed by the original code), but the nested repos still hold alternates pointing to the now-deleted paths.

At step 3, srctree_hash_files() runs 'git add -A .' with a custom GIT_INDEX_FILE. Git internally calls 'git status --porcelain=2' on each nested repo to check for changes; this fails with exit 128 because the nested alternates are broken:
error: unable to normalize alternate object path:
.../downloads/git2/github.com.example.module//objects
fatal: bad object HEAD
fatal: 'git status --porcelain=2' failed in submodule modules/lib/module

This halts the BitBake parse phase with a CalledProcessError and leaves the workspace in an unrecoverable state without manual intervention.

Fix by having devtool_post_unpack() look at the recipe's SRC_URI directly:
any git entry with an explicit destsuffix param names an additional checkout nested under the source tree, so convert each of those to a standalone clone the same way as the top-level tree.

This is deliberately metadata-driven rather than walking the unpacked source tree looking for '.git' directories: a directory walk has no way to tell a nested checkout from an ordinary subdirectory of one, so it either has to stop at the first git repo it finds - which then misses a destsuffix repo nested inside another repo's own working tree - or keep walking into every repo's contents, which is wasted work for large trees. Reading SRC_URI instead gives the exact, authoritative set of paths that need converting, regardless of how they happen to be nested on disk.

Only entries with an explicit destsuffix are handled, since that is the only way a recipe ends up with more than one git checkout under S; this also avoids having to duplicate the git fetcher's internal logic for computing an implicit default destsuffix (which depends on the subdir/ subpath params and BB_GIT_DEFAULT_DESTSUFFIX).

---
 meta/classes/devtool-source.bbclass | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
meta/classes/devtool-source.bbclass +16 -0
index f29f40588f..940cdedbac 100644
@@ -97,6 +97,22 @@ python devtool_post_unpack() {
 
     scriptutils.git_convert_standalone_clone(srcsubdir)
 
+    # Recipes can use multiple git SRC_URI entries with an explicit destsuffix to
+    # unpack several repositories as nested subdirectories of the source tree
+    # (e.g. recipes with separate repos for the kernel, modules and
+    # application). Each such entry is unpacked as its own 'git clone -s' and
+    # needs the same standalone conversion as srcsubdir above, otherwise it keeps
+    # referencing objects in the downloads dir that 'bitbake -c cleanall' removes.
+    # We only look at entries with an explicit destsuffix param, since that's the
+    # only way a recipe ends up with more than one git checkout under S - this
+    # avoids having to duplicate the fetcher's internal default-destsuffix logic.
+    import bb.fetch2
+    fetch = bb.fetch2.Fetch(d.getVar('SRC_URI').split(), d)
+    for url in fetch.urls:
+        ud = fetch.ud[url]
+        if ud.type == 'git' and ud.parm.get('destsuffix'):
+            scriptutils.git_convert_standalone_clone(os.path.join(unpackdir, ud.parm['destsuffix']))
+
     # Make sure that srcsubdir exists
     bb.utils.mkdirhier(srcsubdir)
     if not os.listdir(srcsubdir):
Signature · 1 line
2.43.0
#8 lore raw

Re: [PATCH v4 3/5] devtool-source: Make nested destsuffix git repos standalone

Quoted 82 lines · Jamin Lin

When a recipe uses multiple git SRC_URI entries with different destsuffix values (e.g. recipes with separate repositories for the kernel, modules and application), do_unpack clones each source tree with
'git clone -n -s'.

The -s flag uses git's shared-object mechanism:
instead of copying objects locally it writes a .git/objects/info/alternates file pointing back to the bare repository under the downloads directory (DL_DIR/git2/).

scriptutils.git_convert_standalone_clone() is called by devtool_post_unpack to make the top-level source directory standalone: it runs 'git repack -a' to copy all objects into the local object store and then removes the alternates file.

However it only processes the top-level source directory. Each nested git repo created by a separate SRC_URI entry retains its own alternates file still pointing into downloads/.

Steps to reproduce:
1. devtool modify <recipe-with-multiple-git-SRC_URI>
2. bitbake -c cleanall <recipe>
3. bitbake <recipe>

At step 2, 'bitbake -c cleanall' calls fetcher.clean() which deletes the bare repositories from downloads/git2/. The top-level workspace repo is standalone (alternates already removed by the original code), but the nested repos still hold alternates pointing to the now-deleted paths.

At step 3, srctree_hash_files() runs 'git add -A .' with a custom GIT_INDEX_FILE. Git internally calls 'git status --porcelain=2' on each nested repo to check for changes; this fails with exit 128 because the nested alternates are broken:
error: unable to normalize alternate object path:
.../downloads/git2/github.com.example.module//objects
fatal: bad object HEAD
fatal: 'git status --porcelain=2' failed in submodule modules/lib/module

This halts the BitBake parse phase with a CalledProcessError and leaves the workspace in an unrecoverable state without manual intervention.

Fix by having devtool_post_unpack() look at the recipe's SRC_URI directly:
any git entry with an explicit destsuffix param names an additional checkout nested under the source tree, so convert each of those to a standalone clone the same way as the top-level tree.

This is deliberately metadata-driven rather than walking the unpacked source tree looking for '.git' directories: a directory walk has no way to tell a nested checkout from an ordinary subdirectory of one, so it either has to stop at the first git repo it finds - which then misses a destsuffix repo nested inside another repo's own working tree - or keep walking into every repo's contents, which is wasted work for large trees. Reading SRC_URI instead gives the exact, authoritative set of paths that need converting, regardless of how they happen to be nested on disk.

Only entries with an explicit destsuffix are handled, since that is the only way a recipe ends up with more than one git checkout under S; this also avoids having to duplicate the git fetcher's internal logic for computing an implicit default destsuffix (which depends on the subdir/ subpath params and BB_GIT_DEFAULT_DESTSUFFIX).

---
 meta/classes/devtool-source.bbclass | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
meta/classes/devtool-source.bbclass +9 -0
index f29f40588f..940cdedbac 100644
@@ -97,6 +97,22 @@ python devtool_post_unpack() {
 
     scriptutils.git_convert_standalone_clone(srcsubdir)
 
+    # Recipes can use multiple git SRC_URI entries with an explicit destsuffix to
+    # unpack several repositories as nested subdirectories of the source tree
+    # (e.g. recipes with separate repos for the kernel, modules and
+    # application). Each such entry is unpacked as its own 'git clone -s' and
+    # needs the same standalone conversion as srcsubdir above, otherwise it keeps
+    # referencing objects in the downloads dir that 'bitbake -c cleanall' removes.
+    # We only look at entries with an explicit destsuffix param, since that's the
+    # only way a recipe ends up with more than one git checkout under S - this
+    # avoids having to duplicate the fetcher's internal default-destsuffix logic.

meta/classes/devtool-source.bbclass python devtool_post_unpack() { · line 118

Quoted 10 lines
meta/classes/devtool-source.bbclass +7 -0
+    import bb.fetch2
+    fetch = bb.fetch2.Fetch(d.getVar('SRC_URI').split(), d)
+    for url in fetch.urls:
+        ud = fetch.ud[url]
+        if ud.type == 'git' and ud.parm.get('destsuffix'):
+            scriptutils.git_convert_standalone_clone(os.path.join(unpackdir, ud.parm['destsuffix']))
+
     # Make sure that srcsubdir exists
     bb.utils.mkdirhier(srcsubdir)
     if not os.listdir(srcsubdir):

Why still handle srcsubdir separately above this loop?

What about SRC_URI containing multiple git repositories that unpack side-by-side instead of nested?

It may be better to just iterate through all git repositories in SRC_URI rather than just the nested ones.

Best regards,

Signature · 1 line
Paul Barker
author
#9 lore raw

PATCH v4 4/5 meta-selftest: Add devtool-test-multi-destsuffix recipe

In reply to Jamin Lin

Add a test recipe with three git SRC_URI entries using nested destsuffix values, where each repo's checkout lives inside the working tree of the previous one (level1, then level1/level2, then level1/level2/level3). This recipe is used by the devtool selftest to verify that devtool modify correctly converts all nested git repos to standalone clones, including the case where a repo is nested inside another repo's own working tree rather than merely under a shared plain directory.

---
 .../devtool-test-multi-destsuffix_git.bb      | 27 +++++++++++++++++++
 1 file changed, 27 insertions(+)
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb
meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb new file +27 -0
new file mode 100644
index 0000000000..880f0f0bb4
@@ -0,0 +1,27 @@
+SUMMARY = "Test recipe for multiple git SRC_URI entries with nested destsuffix values"
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
+
+# Three git entries genuinely nested inside each other's working tree (not just
+# nested by path under a shared plain directory): level2's destsuffix places it
+# inside level1's checkout, and level3's destsuffix places it inside level2's.
+# This exercises the devtool code path that must convert every nested git repo
+# to a standalone clone even when one repo's checkout lives inside another
+# repo's working tree: the initial fetch uses a shared clone whose alternates
+# point into downloads/git2/; git repack copies those objects locally so the
+# workspace survives 'bitbake -c cleanall'.
+SRC_URI = "git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level1;destsuffix=level1 \
+    git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level2;destsuffix=level1/level2 \
+    git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level3;destsuffix=level1/level2/level3 \
+"
+
+SRCREV_level1 = "64bc7c8fae61ded0c4e555aa775911f84c56e438"
+SRCREV_level2 = "64bc7c8fae61ded0c4e555aa775911f84c56e438"
+SRCREV_level3 = "64bc7c8fae61ded0c4e555aa775911f84c56e438"
+SRCREV_FORMAT = "level1_level2_level3"
+
+S = "${UNPACKDIR}"
+
+do_configure[noexec] = "1"
+do_compile[noexec] = "1"
+do_install[noexec] = "1"
Signature · 1 line
2.43.0
author
#10 lore raw

PATCH v4 5/5 oeqa/selftest/devtool: Add test for multiple nested git destsuffix repos

In reply to Jamin Lin

Add test_devtool_modify_multi_git_destsuffix_standalone to verify that devtool modify converts all nested git repos (from multiple SRC_URI git entries with different destsuffix values, including a repo nested inside another repo's own working tree) to standalone clones so the workspace survives 'bitbake -c cleanall'.

---
 meta/lib/oeqa/selftest/cases/devtool.py | 64 +++++++++++++++++++++++++
 1 file changed, 64 insertions(+)
meta/lib/oeqa/selftest/cases/devtool.py +64 -0
index a10eb0c784..de73a2e620 100644
@@ -1265,6 +1265,70 @@ class DevtoolModifyTests(DevtoolBase):
         self.assertExists(os.path.join(source_repo_gitsm_gitmodules, 'bitbake'), 'Submodule not found')
         self.assertExists(os.path.join(source_repo_gitsm_gitmodules, 'bitbake-gitsm-test1'), 'Submodule not found')
 
+    def test_devtool_modify_multi_git_destsuffix_standalone(self):
+        """
+        Verify that devtool modify converts all nested git repos (from multiple
+        SRC_URI git entries with different destsuffix values) to standalone clones
+        so that 'bitbake -c cleanall' does not break the devtool workspace.
+
+        The recipe (devtool-test-multi-destsuffix) has three git SRC_URI entries
+        with S = ${UNPACKDIR}, each nested inside the previous repo's own
+        working tree:
+            destsuffix=level1               -> srcdir/level1/
+            destsuffix=level1/level2        -> srcdir/level1/level2/
+            destsuffix=level1/level2/level3 -> srcdir/level1/level2/level3/
+
+        This mirrors real-world recipes that embed multiple module repos
+        as nested subdirectories of the primary source tree, including the
+        case where one repo's checkout lives inside another repo's working
+        tree rather than merely under a shared plain directory.
+        """
+        testrecipe = 'devtool-test-multi-destsuffix'
+        src_uri = get_bb_var('SRC_URI', testrecipe)
+        self.assertIn('git://', src_uri,
+                      'This test expects %s to have git SRC_URI entries' % testrecipe)
+        self.track_for_cleanup(self.workspacedir)
+        self.add_command_to_tearDown('devtool reset %s' % testrecipe)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        result = runCmd('devtool modify %s' % testrecipe)
+        self.assertEqual(result.status, 0,
+                         'devtool modify failed: %s' % result.output)
+        srcdir = os.path.join(self.workspacedir, 'sources', testrecipe)
+        nested_paths = [
+            ('level1', 'level1'),
+            ('level2', 'level1/level2'),
+            ('level3', 'level1/level2/level3'),
+        ]
+
+        for name, subpath in nested_paths:
+            repo_path = os.path.join(srcdir, subpath)
+            self.assertExists(os.path.join(repo_path, '.git'),
+                              'Repo %s (.git) not found in devtool workspace' % name)
+
+        # Key assertion: no nested repo should retain a git alternates file.
+        # devtool modify must repack objects locally so the workspace does not
+        # depend on the downloads cache, which 'bitbake -c cleanall' will delete.
+        for name, subpath in nested_paths:
+            repo_path = os.path.join(srcdir, subpath)
+            alternates_file = os.path.join(repo_path, '.git', 'objects',
+                                           'info', 'alternates')
+            self.assertNotExists(alternates_file,
+                                 'Repo %s still has a git alternates file after '
+                                 'devtool modify' % name)
+
+        # Verify the workspace survives cleanall, which removes the shared
+        # objects in the downloads cache that alternates would reference.
+        bitbake('%s -c cleanall' % testrecipe)
+
+        # After cleanall all repos must still be usable.
+        # A broken alternates file would cause git operations to fail.
+        for name, subpath in nested_paths:
+            repo_path = os.path.join(srcdir, subpath)
+            result = runCmd('git status', cwd=repo_path)
+            self.assertEqual(result.status, 0,
+                             'git status failed in repo %s after cleanall: %s'
+                             % (name, result.output))
+
 class DevtoolUpdateTests(DevtoolBase):
 
     def test_devtool_update_recipe(self):
Signature · 1 line
2.43.0
#11 lore raw

Re: [PATCH v4 5/5] oeqa/selftest/devtool: Add test for multiple nested git destsuffix repos

Quoted 63 lines · Jamin Lin

Add test_devtool_modify_multi_git_destsuffix_standalone to verify that devtool modify converts all nested git repos (from multiple SRC_URI git entries with different destsuffix values, including a repo nested inside another repo's own working tree) to standalone clones so the workspace survives 'bitbake -c cleanall'.

---
 meta/lib/oeqa/selftest/cases/devtool.py | 64 +++++++++++++++++++++++++
 1 file changed, 64 insertions(+)
meta/lib/oeqa/selftest/cases/devtool.py +44 -0
index a10eb0c784..de73a2e620 100644
@@ -1265,6 +1265,70 @@ class DevtoolModifyTests(DevtoolBase):
         self.assertExists(os.path.join(source_repo_gitsm_gitmodules, 'bitbake'), 'Submodule not found')
         self.assertExists(os.path.join(source_repo_gitsm_gitmodules, 'bitbake-gitsm-test1'), 'Submodule not found')
 
+    def test_devtool_modify_multi_git_destsuffix_standalone(self):
+        """
+        Verify that devtool modify converts all nested git repos (from multiple
+        SRC_URI git entries with different destsuffix values) to standalone clones
+        so that 'bitbake -c cleanall' does not break the devtool workspace.
+
+        The recipe (devtool-test-multi-destsuffix) has three git SRC_URI entries
+        with S = ${UNPACKDIR}, each nested inside the previous repo's own
+        working tree:
+            destsuffix=level1               -> srcdir/level1/
+            destsuffix=level1/level2        -> srcdir/level1/level2/
+            destsuffix=level1/level2/level3 -> srcdir/level1/level2/level3/
+
+        This mirrors real-world recipes that embed multiple module repos
+        as nested subdirectories of the primary source tree, including the
+        case where one repo's checkout lives inside another repo's working
+        tree rather than merely under a shared plain directory.
+        """
+        testrecipe = 'devtool-test-multi-destsuffix'
+        src_uri = get_bb_var('SRC_URI', testrecipe)
+        self.assertIn('git://', src_uri,
+                      'This test expects %s to have git SRC_URI entries' % testrecipe)
+        self.track_for_cleanup(self.workspacedir)
+        self.add_command_to_tearDown('devtool reset %s' % testrecipe)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        result = runCmd('devtool modify %s' % testrecipe)
+        self.assertEqual(result.status, 0,
+                         'devtool modify failed: %s' % result.output)
+        srcdir = os.path.join(self.workspacedir, 'sources', testrecipe)
+        nested_paths = [
+            ('level1', 'level1'),
+            ('level2', 'level1/level2'),
+            ('level3', 'level1/level2/level3'),
+        ]
+
+        for name, subpath in nested_paths:
+            repo_path = os.path.join(srcdir, subpath)
+            self.assertExists(os.path.join(repo_path, '.git'),
+                              'Repo %s (.git) not found in devtool workspace' % name)
+
+        # Key assertion: no nested repo should retain a git alternates file.
+        # devtool modify must repack objects locally so the workspace does not
+        # depend on the downloads cache, which 'bitbake -c cleanall' will delete.
+        for name, subpath in nested_paths:

meta/lib/oeqa/selftest/cases/devtool.py class DevtoolModifyTests(DevtoolBase): · line 1321

Quoted 10 lines
meta/lib/oeqa/selftest/cases/devtool.py +10 -0
+            repo_path = os.path.join(srcdir, subpath)
+            alternates_file = os.path.join(repo_path, '.git', 'objects',
+                                           'info', 'alternates')
+            self.assertNotExists(alternates_file,
+                                 'Repo %s still has a git alternates file after '
+                                 'devtool modify' % name)
+
+        # Verify the workspace survives cleanall, which removes the shared
+        # objects in the downloads cache that alternates would reference.
+        bitbake('%s -c cleanall' % testrecipe)

We can't use cleanall here as it deletes data from the main downloads directory. Is the above check for an alternates file not sufficient? If not then this test needs to use an isolated downloads directory.

Best regards,

Signature · 1 line
Paul Barker
author
#12 lore raw

RE: [PATCH v4 5/5] oeqa/selftest/devtool: Add test for multiple nested git destsuffix repos

Quoted 85 lines · Jamin Lin
Quoted 18 lines

Add test_devtool_modify_multi_git_destsuffix_standalone to verify that devtool modify converts all nested git repos (from multiple SRC_URI git entries with different destsuffix values, including a repo nested inside another repo's own working tree) to standalone clones so the workspace survives 'bitbake -c cleanall'.

---
 meta/lib/oeqa/selftest/cases/devtool.py | 64
+++++++++++++++++++++++++
 1 file changed, 64 insertions(+)
meta/lib/oeqa/selftest/cases/devtool.py +0 -0
index a10eb0c784..de73a2e620 100644

meta/lib/oeqa/selftest/cases/devtool.py class DevtoolModifyTests(DevtoolBase): · line 1265

Quoted 1 line
meta/lib/oeqa/selftest/cases/devtool.py +0 -0
@@ -1265,6 +1265,70 @@ class DevtoolModifyTests(DevtoolBase):
         self.assertExists(os.path.join(source_repo_gitsm_gitmodules,

'bitbake'), 'Submodule not found')

meta/lib/oeqa/selftest/cases/devtool.py class DevtoolModifyTests(DevtoolBase):

Quoted 6 lines
meta/lib/oeqa/selftest/cases/devtool.py +3 -0
         self.assertExists(os.path.join(source_repo_gitsm_gitmodules,
 'bitbake-gitsm-test1'), 'Submodule not found')
 
+    def test_devtool_modify_multi_git_destsuffix_standalone(self):
+        """
+        Verify that devtool modify converts all nested git repos (from

multiple

meta/lib/oeqa/selftest/cases/devtool.py class DevtoolModifyTests(DevtoolBase): · line 1272

Quoted 1 line
meta/lib/oeqa/selftest/cases/devtool.py +1 -0
+        SRC_URI git entries with different destsuffix values) to standalone

clones

meta/lib/oeqa/selftest/cases/devtool.py class DevtoolModifyTests(DevtoolBase):

Quoted 3 lines
meta/lib/oeqa/selftest/cases/devtool.py +3 -0
+        so that 'bitbake -c cleanall' does not break the devtool workspace.
+
+        The recipe (devtool-test-multi-destsuffix) has three git SRC_URI

entries

meta/lib/oeqa/selftest/cases/devtool.py class DevtoolModifyTests(DevtoolBase): · line 1276

Quoted 1 line
meta/lib/oeqa/selftest/cases/devtool.py +1 -0
+        with S = ${UNPACKDIR}, each nested inside the previous repo's

own

meta/lib/oeqa/selftest/cases/devtool.py class DevtoolModifyTests(DevtoolBase):

Quoted 9 lines
meta/lib/oeqa/selftest/cases/devtool.py +9 -0
+        working tree:
+            destsuffix=level1               -> srcdir/level1/
+            destsuffix=level1/level2        -> srcdir/level1/level2/
+            destsuffix=level1/level2/level3 ->
+ srcdir/level1/level2/level3/
+
+        This mirrors real-world recipes that embed multiple module repos
+        as nested subdirectories of the primary source tree, including the
+        case where one repo's checkout lives inside another repo's

working

meta/lib/oeqa/selftest/cases/devtool.py class DevtoolModifyTests(DevtoolBase):

Quoted 6 lines
meta/lib/oeqa/selftest/cases/devtool.py +6 -0
+        tree rather than merely under a shared plain directory.
+        """
+        testrecipe = 'devtool-test-multi-destsuffix'
+        src_uri = get_bb_var('SRC_URI', testrecipe)
+        self.assertIn('git://', src_uri,
+                      'This test expects %s to have git SRC_URI entries'

% testrecipe)

meta/lib/oeqa/selftest/cases/devtool.py class DevtoolModifyTests(DevtoolBase):

Quoted 3 lines
meta/lib/oeqa/selftest/cases/devtool.py +3 -0
+        self.track_for_cleanup(self.workspacedir)
+        self.add_command_to_tearDown('devtool reset %s' % testrecipe)
+        self.add_command_to_tearDown('bitbake-layers remove-layer

*/workspace')

Quoted 8 lines
meta/lib/oeqa/selftest/cases/devtool.py +8 -0
+        result = runCmd('devtool modify %s' % testrecipe)
+        self.assertEqual(result.status, 0,
+                         'devtool modify failed: %s' % result.output)
+        srcdir = os.path.join(self.workspacedir, 'sources', testrecipe)
+        nested_paths = [
+            ('level1', 'level1'),
+            ('level2', 'level1/level2'),
+            ('level3', 'level1/level2/level3'),

meta/lib/oeqa/selftest/cases/devtool.py class DevtoolModifyTests(DevtoolBase):

Quoted 10 lines
meta/lib/oeqa/selftest/cases/devtool.py +10 -0
+        ]
+
+        for name, subpath in nested_paths:
+            repo_path = os.path.join(srcdir, subpath)
+            self.assertExists(os.path.join(repo_path, '.git'),
+                              'Repo %s (.git) not found in devtool
+ workspace' % name)
+
+        # Key assertion: no nested repo should retain a git alternates file.
+        # devtool modify must repack objects locally so the workspace

does not

meta/lib/oeqa/selftest/cases/devtool.py class DevtoolModifyTests(DevtoolBase): · line 1313

Quoted 1 line
meta/lib/oeqa/selftest/cases/devtool.py +1 -0
+        # depend on the downloads cache, which 'bitbake -c cleanall' will

delete.

meta/lib/oeqa/selftest/cases/devtool.py class DevtoolModifyTests(DevtoolBase):

Quoted 6 lines
meta/lib/oeqa/selftest/cases/devtool.py +6 -0
+        for name, subpath in nested_paths:
+            repo_path = os.path.join(srcdir, subpath)
+            alternates_file = os.path.join(repo_path, '.git', 'objects',
+                                           'info', 'alternates')
+            self.assertNotExists(alternates_file,
+                                 'Repo %s still has a git alternates
Quoted 8 lines

file after '

meta/lib/oeqa/selftest/cases/devtool.py class DevtoolModifyTests(DevtoolBase):

Quoted 3 lines
meta/lib/oeqa/selftest/cases/devtool.py +3 -0
+                                 'devtool modify' % name)
+
+        # Verify the workspace survives cleanall, which removes the

shared

Quoted 2 lines
meta/lib/oeqa/selftest/cases/devtool.py +2 -0
+        # objects in the downloads cache that alternates would reference.
+        bitbake('%s -c cleanall' % testrecipe)

Will remove cleanall test.

Quoted 8 lines

We can't use cleanall here as it deletes data from the main downloads directory. Is the above check for an alternates file not sufficient? If not then this test needs to use an isolated downloads directory.

Best regards,

--
Paul Barker

author
#13 lore raw
In reply to Jamin Lin

All,

Thanks all for the review and the time you spent on this.

I originally ran into this problem myself and used AI to help put together the fix. However, the design questions raised here have gone beyond my own understanding of this code and its original intent, and I don't feel I can properly stand behind the patches or address the review comments with confidence.

So I'm going to drop this series rather than submit something I can't fully explain.

For anyone who hits the same issue: as a workaround, use 'bitbake -c cleansstate <recipe>' instead of 'bitbake -c cleanall'. cleansstate does not delete the bare repositories under DL_DIR/git2/, so the nested repos' alternates stay valid and the devtool workspace keeps working.

Thanks again for the review.

Jamin

Quoted 63 lines

-----Original Message-----
From: Jamin Lin
Sent: Friday, July 31, 2026 5:27 PM
To: [email protected]; [email protected]; [email protected]; [email protected]
Cc: Troy Lee <[email protected]>; Jamin Lin <[email protected]>
Subject: [PATCH v4 0/5] devtool: fix standalone clone conversion for nested git repos

When a recipe has multiple git SRC_URI entries with destsuffix values nested inside S, devtool modify left all but the top-level repo with
a .git/objects/info/alternates file pointing into the downloads cache.

Running 'bitbake -c cleanall' then removed those shared objects, breaking all subsequent git operations in the workspace.
Fix git_convert_standalone_clone() to walk all git repos nested inside S and repack each one to a fully standalone clone.

v1:
- Fix git_convert_standalone_clone() to walk all git repos nested inside S and repack each one to a standalone clone
v2:
- Add selftest recipe devtool-test-multi-destsuffix with six nested git SRC_URI entries to reproduce the scenario
- Add test_devtool_modify_multi_git_destsuffix_standalone to verify all nested repos have their alternates removed after devtool modify and remain usable after 'bitbake -c cleanall'
v3:
- Read SRC_URI directly instead of walking the tree for '.git' dirs - also fixes repos nested inside another repo's own working tree
- Fix setup_git_repo() to register nested repos as submodules before the initial commit (was dead code), processing bottom-up to avoid stale submodule references
- Fix commitIgnored() to skip committing when nothing is staged, so do_patch no longer fails on a submodule that's dirty only because of its own nested content
- Simplify test recipe to three genuinely-nested destsuffix entries (level1/level1/level2/level1/level2/level3), update selftest to match
v4:
- Fix regression from v3:
setup_git_repo() now detects nested repos both for fresh workspace repos and for recipes whose source is already a git repo (fixes test_devtool_modify_nested_gitsm).

Jamin Lin (5):
oe/patch: Skip commitIgnored when nothing is actually staged
devtool: Register nested git repos before the initial commit
devtool-source: Make nested destsuffix git repos standalone
meta-selftest: Add devtool-test-multi-destsuffix recipe oeqa/selftest/devtool: Add test for multiple nested git destsuffix
repos

 .../devtool-test-multi-destsuffix_git.bb      | 27 +++++++
 meta/classes/devtool-source.bbclass           | 16 ++++
 meta/lib/oe/patch.py                          | 12 +++
 meta/lib/oeqa/selftest/cases/devtool.py       | 64 +++++++++++++++
 scripts/lib/devtool/__init__.py               | 79 +++++++++++++------
 5 files changed, 175 insertions(+), 23 deletions(-)  create mode 100644

meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb

--
2.43.0