Skip to contentSkip to conversation

scripts/scriptutils: Convert nested git repos to standalone clones in devtool workspace

PATCH Superseded Jamin Lin 2 participants 3 messages May 20 – May 21

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Jamin Lin

author
#0 lore raw

PATCH 1/1 scripts/scriptutils: Convert nested git repos to standalone clones in devtool workspace

When a recipe uses multiple git SRC_URI entries with different destsuffix values (e.g. Zephyr-based recipes with separate repositories for the kernel, modules and application), do_unpack clones each source tree with 'git clone -n -s'.

The -s flag uses git's shared-object mechanism:
instead of copying objects locally it writes a .git/objects/info/alternates file pointing back to the bare repository under the downloads directory (DL_DIR/git2/).

git_convert_standalone_clone() is called by devtool_post_unpack to make the workspace standalone: it runs 'git repack -a' to copy all objects into the local object store and then removes the alternates file.

However it only processes the top-level source directory. Each nested git repo created by a separate SRC_URI entry retains its own alternates file still pointing into downloads/.

Steps to reproduce:
1. devtool modify <recipe-with-multiple-git-SRC_URI>
2. bitbake -c cleanall <recipe>
3. bitbake <recipe>

At step 2, 'bitbake -c cleanall' calls fetcher.clean() which deletes the bare repositories from downloads/git2/. The top-level workspace repo is standalone (alternates already removed by the original code), but the nested repos still hold alternates pointing to the now-deleted paths.

At step 3, srctree_hash_files() runs 'git add -A .' with a custom GIT_INDEX_FILE. Git internally calls 'git status --porcelain=2' on each nested repo to check for changes; this fails with exit 128 because the nested alternates are broken:
error: unable to normalize alternate object path:
.../downloads/git2/github.com.zephyrproject-rtos.acpica//objects
fatal: bad object HEAD
fatal: 'git status --porcelain=2' failed in submodule modules/lib/acpica

This halts the BitBake parse phase with a CalledProcessError and leaves the workspace in an unrecoverable state without manual intervention.

Fix by extending git_convert_standalone_clone() to walk the source tree and apply the same 'git repack -a' + remove-alternates treatment to every nested git repository found, making the entire workspace fully standalone at devtool modify time. The walk uses dirs[:] = [] to stop at each git boundary so it never descends into already-converted repos.

---
 scripts/lib/scriptutils.py | 28 ++++++++++++++++++++++++----
 1 file changed, 24 insertions(+), 4 deletions(-)
scripts/lib/scriptutils.py +24 -4
index 32e749dbb1..0a83470373 100644
@@ -100,16 +100,36 @@ def load_plugins(logger, plugins, pluginpath):
 
 
 def git_convert_standalone_clone(repodir):
-    """If specified directory is a git repository, ensure it's a standalone clone"""
+    """
+    If specified directory is a git repository, ensure it's a standalone clone.
+    Also converts any nested git repositories (created by multiple SRC_URI git
+    entries with different destsuffix values) so that none of their contents
+    depend on the shared downloads directory via alternates.
+    """
+
     import bb.process
-    if os.path.exists(os.path.join(repodir, '.git')):
-        alternatesfile = os.path.join(repodir, '.git', 'objects', 'info', 'alternates')
+
+    def _convert(gitdir, workdir):
+        alternatesfile = os.path.join(gitdir, 'objects', 'info', 'alternates')
         if os.path.exists(alternatesfile):
             # This will have been cloned with -s, so we need to convert it so none
             # of the contents is shared
-            bb.process.run('git repack -a', cwd=repodir)
+            bb.process.run('git repack -a', cwd=workdir)
             os.remove(alternatesfile)
 
+    if os.path.exists(os.path.join(repodir, '.git')):
+        _convert(os.path.join(repodir, '.git'), repodir)
+
+    # Also handle nested git repos created by multiple SRC_URI git entries
+    # with different destsuffix values. Each nested repo is cloned with -s
+    # and has its own alternates pointing to the downloads directory.
+    for root, dirs, files in os.walk(repodir):
+        if root == repodir:
+            continue
+        if '.git' in dirs:
+            _convert(os.path.join(root, '.git'), root)
+            dirs[:] = []  # don't recurse into nested repos
+
 def _get_temp_recipe_dir(d):
     # This is a little bit hacky but we need to find a place where we can put
     # the recipe so that bitbake can find it. We're going to delete it at the
Signature · 1 line
2.43.0
#1 lore raw

There have been patches fixing similar issues, for which oe-core has no tests or ways to reproduce. Rather than ask reviewers to set up zephyr builds so they can observe the issue, can you add a testcase for this to devtool's oe-selftest?

Alex

Quoted 108 lines · Jamin Lin via lists.openembedded.org

When a recipe uses multiple git SRC_URI entries with different destsuffix values (e.g. Zephyr-based recipes with separate repositories for the kernel, modules and application), do_unpack clones each source tree with 'git clone -n -s'.

The -s flag uses git's shared-object mechanism:
instead of copying objects locally it writes a .git/objects/info/alternates file pointing back to the bare repository under the downloads directory (DL_DIR/git2/).

git_convert_standalone_clone() is called by devtool_post_unpack to make the workspace standalone: it runs 'git repack -a' to copy all objects into the local object store and then removes the alternates file.

However it only processes the top-level source directory. Each nested git repo created by a separate SRC_URI entry retains its own alternates file still pointing into downloads/.

Steps to reproduce:
1. devtool modify <recipe-with-multiple-git-SRC_URI>
2. bitbake -c cleanall <recipe>
3. bitbake <recipe>

At step 2, 'bitbake -c cleanall' calls fetcher.clean() which deletes the bare repositories from downloads/git2/. The top-level workspace repo is standalone (alternates already removed by the original code), but the nested repos still hold alternates pointing to the now-deleted paths.

At step 3, srctree_hash_files() runs 'git add -A .' with a custom GIT_INDEX_FILE. Git internally calls 'git status --porcelain=2' on each nested repo to check for changes; this fails with exit 128 because the nested alternates are broken:
error: unable to normalize alternate object path:
.../downloads/git2/github.com.zephyrproject-rtos.acpica//objects
fatal: bad object HEAD
fatal: 'git status --porcelain=2' failed in submodule modules/lib/acpica

This halts the BitBake parse phase with a CalledProcessError and leaves the workspace in an unrecoverable state without manual intervention.

Fix by extending git_convert_standalone_clone() to walk the source tree and apply the same 'git repack -a' + remove-alternates treatment to every nested git repository found, making the entire workspace fully standalone at devtool modify time. The walk uses dirs[:] = [] to stop at each git boundary so it never descends into already-converted repos.

---
 scripts/lib/scriptutils.py | 28 ++++++++++++++++++++++++----
 1 file changed, 24 insertions(+), 4 deletions(-)
scripts/lib/scriptutils.py +24 -5
index 32e749dbb1..0a83470373 100644
@@ -100,16 +100,36 @@ def load_plugins(logger, plugins, pluginpath):
 
 
 def git_convert_standalone_clone(repodir):
-    """If specified directory is a git repository, ensure it's a standalone clone"""
+    """
+    If specified directory is a git repository, ensure it's a standalone clone.
+    Also converts any nested git repositories (created by multiple SRC_URI git
+    entries with different destsuffix values) so that none of their contents
+    depend on the shared downloads directory via alternates.
+    """
+
     import bb.process
-    if os.path.exists(os.path.join(repodir, '.git')):
-        alternatesfile = os.path.join(repodir, '.git', 'objects', 'info', 'alternates')
+
+    def _convert(gitdir, workdir):
+        alternatesfile = os.path.join(gitdir, 'objects', 'info', 'alternates')
         if os.path.exists(alternatesfile):
             # This will have been cloned with -s, so we need to convert it so none
             # of the contents is shared
-            bb.process.run('git repack -a', cwd=repodir)
+            bb.process.run('git repack -a', cwd=workdir)
             os.remove(alternatesfile)
 
+    if os.path.exists(os.path.join(repodir, '.git')):
+        _convert(os.path.join(repodir, '.git'), repodir)
+
+    # Also handle nested git repos created by multiple SRC_URI git entries
+    # with different destsuffix values. Each nested repo is cloned with -s
+    # and has its own alternates pointing to the downloads directory.
+    for root, dirs, files in os.walk(repodir):
+        if root == repodir:
+            continue
+        if '.git' in dirs:
+            _convert(os.path.join(root, '.git'), root)
+            dirs[:] = []  # don't recurse into nested repos
+
 def _get_temp_recipe_dir(d):
     # This is a little bit hacky but we need to find a place where we can put
     # the recipe so that bitbake can find it. We're going to delete it at the
--

2.43.0

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#237382): https://lists.openembedded.org/g/openembedded-core/message/237382
Mute This Topic: https://lists.openembedded.org/mt/119403462/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

author
#2 lore raw

Hi Alexander

Thanks for the review and suggestion.
Will add and resend v2.

Thanks,
Jamin

Quoted 138 lines

-----Original Message-----
From: Alexander Kanavin <[email protected]>
Sent: Wednesday, May 20, 2026 5:24 PM
To: Jamin Lin <[email protected]>
Cc: [email protected]; Troy Lee <[email protected]>; Vince Chang <[email protected]>; [email protected]
Subject: Re: [OE-core] [PATCH v1] scripts/scriptutils: Convert nested git repos to standalone clones in devtool workspace

There have been patches fixing similar issues, for which oe-core has no tests or ways to reproduce. Rather than ask reviewers to set up zephyr builds so they can observe the issue, can you add a testcase for this to devtool's oe-selftest?

Alex

Quoted 9 lines · Jamin Lin via lists.openembedded.org

When a recipe uses multiple git SRC_URI entries with different destsuffix values (e.g. Zephyr-based recipes with separate repositories for the kernel, modules and application), do_unpack clones each source tree with 'git clone -n -s'.

The -s flag uses git's shared-object mechanism:
instead of copying objects locally it writes a .git/objects/info/alternates file pointing back to the bare repository under

the downloads directory (DL_DIR/git2/).

Quoted 30 lines

git_convert_standalone_clone() is called by devtool_post_unpack to make the workspace standalone: it runs 'git repack -a' to copy all objects into the local object store and then removes the alternates file.

However it only processes the top-level source directory. Each nested git repo created by a separate SRC_URI entry retains its own alternates file still pointing into downloads/.

Steps to reproduce:
1. devtool modify <recipe-with-multiple-git-SRC_URI>
2. bitbake -c cleanall <recipe>
3. bitbake <recipe>

At step 2, 'bitbake -c cleanall' calls fetcher.clean() which deletes the bare repositories from downloads/git2/. The top-level workspace repo is standalone (alternates already removed by the original code), but the nested repos still hold alternates pointing to the now-deleted paths.

At step 3, srctree_hash_files() runs 'git add -A .' with a custom GIT_INDEX_FILE. Git internally calls 'git status --porcelain=2' on each nested repo to check for changes; this fails with exit 128 because the nested alternates are broken:
error: unable to normalize alternate object path:
.../downloads/git2/github.com.zephyrproject-rtos.acpica//objects
fatal: bad object HEAD
fatal: 'git status --porcelain=2' failed in submodule
modules/lib/acpica

Quoted 8 lines

This halts the BitBake parse phase with a CalledProcessError and leaves the workspace in an unrecoverable state without manual intervention.

Fix by extending git_convert_standalone_clone() to walk the source tree and apply the same 'git repack -a' + remove-alternates treatment to every nested git repository found, making the entire workspace fully standalone at devtool modify time. The walk uses dirs[:] = [] to stop at each git boundary so it never descends into already-converted

repos.

Quoted 11 lines
---
 scripts/lib/scriptutils.py | 28 ++++++++++++++++++++++++----
 1 file changed, 24 insertions(+), 4 deletions(-)
scripts/lib/scriptutils.py +0 -0
index 32e749dbb1..0a83470373 100644

scripts/lib/scriptutils.py def load_plugins(logger, plugins, pluginpath): · line 103

Quoted 4 lines
scripts/lib/scriptutils.py +0 -1
@@ -100,16 +100,36 @@ def load_plugins(logger, plugins, pluginpath):
 
 
 def git_convert_standalone_clone(repodir):
-    """If specified directory is a git repository, ensure it's a standalone

clone"""

scripts/lib/scriptutils.py def load_plugins(logger, plugins, pluginpath):

Quoted 3 lines
scripts/lib/scriptutils.py +3 -0
+    """
+    If specified directory is a git repository, ensure it's a standalone clone.
+    Also converts any nested git repositories (created by multiple SRC_URI

git

scripts/lib/scriptutils.py def load_plugins(logger, plugins, pluginpath):

Quoted 7 lines
scripts/lib/scriptutils.py +4 -2
+    entries with different destsuffix values) so that none of their contents
+    depend on the shared downloads directory via alternates.
+    """
+
     import bb.process
-    if os.path.exists(os.path.join(repodir, '.git')):
-        alternatesfile = os.path.join(repodir, '.git', 'objects', 'info',

'alternates')

scripts/lib/scriptutils.py def load_plugins(logger, plugins, pluginpath):

Quoted 6 lines
scripts/lib/scriptutils.py +4 -0
+
+    def _convert(gitdir, workdir):
+        alternatesfile = os.path.join(gitdir, 'objects', 'info',
+ 'alternates')
         if os.path.exists(alternatesfile):
             # This will have been cloned with -s, so we need to convert it

so none

Quoted 10 lines
scripts/lib/scriptutils.py +6 -1
             # of the contents is shared
-            bb.process.run('git repack -a', cwd=repodir)
+            bb.process.run('git repack -a', cwd=workdir)
             os.remove(alternatesfile)
 
+    if os.path.exists(os.path.join(repodir, '.git')):
+        _convert(os.path.join(repodir, '.git'), repodir)
+
+    # Also handle nested git repos created by multiple SRC_URI git entries
+    # with different destsuffix values. Each nested repo is cloned with -s

scripts/lib/scriptutils.py def load_plugins(logger, plugins, pluginpath):

Quoted 10 lines
scripts/lib/scriptutils.py +8 -0
+    # and has its own alternates pointing to the downloads directory.
+    for root, dirs, files in os.walk(repodir):
+        if root == repodir:
+            continue
+        if '.git' in dirs:
+            _convert(os.path.join(root, '.git'), root)
+            dirs[:] = []  # don't recurse into nested repos
+
 def _get_temp_recipe_dir(d):
     # This is a little bit hacky but we need to find a place where we can

put

Quoted 15 lines
     # the recipe so that bitbake can find it. We're going to delete
it at the
--
2.43.0

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#237382):
https://lists.openembedded.org/g/openembedded-core/message/237382
Mute This Topic: https://lists.openembedded.org/mt/119403462/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-