scripts/scriptutils: Convert nested git repos to standalone clones in devtool workspace
PATCH 1/1 scripts/scriptutils: Convert nested git repos to standalone clones in devtool workspace
When a recipe uses multiple git SRC_URI entries with different destsuffix values (e.g. Zephyr-based recipes with separate repositories for the kernel, modules and application), do_unpack clones each source tree with 'git clone -n -s'.
The -s flag uses git's shared-object mechanism:
instead of copying objects locally it writes a .git/objects/info/alternates file pointing back to the bare repository under the downloads directory (DL_DIR/git2/).
git_convert_standalone_clone() is called by devtool_post_unpack to make the workspace standalone: it runs 'git repack -a' to copy all objects into the local object store and then removes the alternates file.
However it only processes the top-level source directory. Each nested git repo created by a separate SRC_URI entry retains its own alternates file still pointing into downloads/.
Steps to reproduce:
1. devtool modify <recipe-with-multiple-git-SRC_URI>
2. bitbake -c cleanall <recipe>
3. bitbake <recipe>
At step 2, 'bitbake -c cleanall' calls fetcher.clean() which deletes the bare repositories from downloads/git2/. The top-level workspace repo is standalone (alternates already removed by the original code), but the nested repos still hold alternates pointing to the now-deleted paths.
At step 3, srctree_hash_files() runs 'git add -A .' with a custom GIT_INDEX_FILE. Git internally calls 'git status --porcelain=2' on each nested repo to check for changes; this fails with exit 128 because the nested alternates are broken:
error: unable to normalize alternate object path:
.../downloads/git2/github.com.zephyrproject-rtos.acpica//objects
fatal: bad object HEAD
fatal: 'git status --porcelain=2' failed in submodule modules/lib/acpica
This halts the BitBake parse phase with a CalledProcessError and leaves the workspace in an unrecoverable state without manual intervention.
Fix by extending git_convert_standalone_clone() to walk the source tree and apply the same 'git repack -a' + remove-alternates treatment to every nested git repository found, making the entire workspace fully standalone at devtool modify time. The walk uses dirs[:] = [] to stop at each git boundary so it never descends into already-converted repos.
--- scripts/lib/scriptutils.py | 28 ++++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-)
scripts/lib/scriptutils.py +24 -4
@@ -100,16 +100,36 @@ def load_plugins(logger, plugins, pluginpath): def git_convert_standalone_clone(repodir): - """If specified directory is a git repository, ensure it's a standalone clone""" + """ + If specified directory is a git repository, ensure it's a standalone clone. + Also converts any nested git repositories (created by multiple SRC_URI git + entries with different destsuffix values) so that none of their contents + depend on the shared downloads directory via alternates. + """ + import bb.process - if os.path.exists(os.path.join(repodir, '.git')): - alternatesfile = os.path.join(repodir, '.git', 'objects', 'info', 'alternates') + + def _convert(gitdir, workdir): + alternatesfile = os.path.join(gitdir, 'objects', 'info', 'alternates') if os.path.exists(alternatesfile): # This will have been cloned with -s, so we need to convert it so none # of the contents is shared - bb.process.run('git repack -a', cwd=repodir) + bb.process.run('git repack -a', cwd=workdir) os.remove(alternatesfile) + if os.path.exists(os.path.join(repodir, '.git')): + _convert(os.path.join(repodir, '.git'), repodir) + + # Also handle nested git repos created by multiple SRC_URI git entries + # with different destsuffix values. Each nested repo is cloned with -s + # and has its own alternates pointing to the downloads directory. + for root, dirs, files in os.walk(repodir): + if root == repodir: + continue + if '.git' in dirs: + _convert(os.path.join(root, '.git'), root) + dirs[:] = [] # don't recurse into nested repos + def _get_temp_recipe_dir(d): # This is a little bit hacky but we need to find a place where we can put # the recipe so that bitbake can find it. We're going to delete it at the
Signature · 1 line
2.43.0
There have been patches fixing similar issues, for which oe-core has no tests or ways to reproduce. Rather than ask reviewers to set up zephyr builds so they can observe the issue, can you add a testcase for this to devtool's oe-selftest?
Alex
Quoted 108 lines · Jamin Lin via lists.openembedded.org
When a recipe uses multiple git
SRC_URIentries with different destsuffix values (e.g. Zephyr-based recipes with separate repositories for the kernel, modules and application),do_unpackclones each source tree with 'git clone -n -s'.The -s flag uses git's shared-object mechanism:
instead of copying objects locally it writes a .git/objects/info/alternates file pointing back to the bare repository under the downloads directory (DL_DIR/git2/).
git_convert_standalone_clone()is called bydevtool_post_unpackto make the workspace standalone: it runs 'git repack -a' to copy all objects into the local object store and then removes the alternates file.However it only processes the top-level source directory. Each nested git repo created by a separate
SRC_URIentry retains its own alternates file still pointing into downloads/.Steps to reproduce:
1. devtool modify <recipe-with-multiple-git-SRC_URI>
2. bitbake -c cleanall <recipe>
3. bitbake <recipe>At step 2, 'bitbake -c cleanall' calls
fetcher.clean()which deletes the bare repositories from downloads/git2/. The top-level workspace repo is standalone (alternates already removed by the original code), but the nested repos still hold alternates pointing to the now-deleted paths.At step 3,
srctree_hash_files()runs 'git add -A .' with a customGIT_INDEX_FILE. Git internally calls 'git status --porcelain=2' on each nested repo to check for changes; this fails with exit 128 because the nested alternates are broken:
error: unable to normalize alternate object path:
.../downloads/git2/github.com.zephyrproject-rtos.acpica//objects
fatal: bad object HEAD
fatal: 'git status --porcelain=2' failed in submodule modules/lib/acpicaThis halts the BitBake parse phase with a CalledProcessError and leaves the workspace in an unrecoverable state without manual intervention.
Fix by extending
git_convert_standalone_clone()to walk the source tree and apply the same 'git repack -a' + remove-alternates treatment to every nested git repository found, making the entire workspace fully standalone at devtool modify time. The walk uses dirs[:] = [] to stop at each git boundary so it never descends into already-converted repos.Signed-off-by Jamin Lin <[email protected]>--- scripts/lib/scriptutils.py | 28 ++++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-)scripts/lib/scriptutils.py +24
-5@@ -100,16 +100,36 @@ def load_plugins(logger, plugins, pluginpath): def git_convert_standalone_clone(repodir): - """If specified directory is a git repository, ensure it's a standalone clone""" + """ + If specified directory is a git repository, ensure it's a standalone clone. + Also converts any nested git repositories (created by multiple SRC_URI git + entries with different destsuffix values) so that none of their contents + depend on the shared downloads directory via alternates. + """ + import bb.process - if os.path.exists(os.path.join(repodir, '.git')): - alternatesfile = os.path.join(repodir, '.git', 'objects', 'info', 'alternates') + + def _convert(gitdir, workdir): + alternatesfile = os.path.join(gitdir, 'objects', 'info', 'alternates') if os.path.exists(alternatesfile): # This will have been cloned with -s, so we need to convert it so none # of the contents is shared - bb.process.run('git repack -a', cwd=repodir) + bb.process.run('git repack -a', cwd=workdir) os.remove(alternatesfile) + if os.path.exists(os.path.join(repodir, '.git')): + _convert(os.path.join(repodir, '.git'), repodir) + + # Also handle nested git repos created by multiple SRC_URI git entries + # with different destsuffix values. Each nested repo is cloned with -s + # and has its own alternates pointing to the downloads directory. + for root, dirs, files in os.walk(repodir): + if root == repodir: + continue + if '.git' in dirs: + _convert(os.path.join(root, '.git'), root) + dirs[:] = [] # don't recurse into nested repos + def _get_temp_recipe_dir(d): # This is a little bit hacky but we need to find a place where we can put # the recipe so that bitbake can find it. We're going to delete it at the --2.43.0
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#237382): https://lists.openembedded.org/g/openembedded-core/message/237382
Mute This Topic: https://lists.openembedded.org/mt/119403462/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Hi Alexander
Thanks for the review and suggestion.
Will add and resend v2.
Thanks,
Jamin
Quoted 138 lines
-----Original Message-----
From: Alexander Kanavin <[email protected]>
Sent: Wednesday, May 20, 2026 5:24 PM
To: Jamin Lin <[email protected]>
Cc: [email protected]; Troy Lee <[email protected]>; Vince Chang <[email protected]>; [email protected]
Subject: Re: [OE-core] [PATCH v1]scripts/scriptutils: Convert nested git repos to standalone clones in devtool workspaceThere have been patches fixing similar issues, for which oe-core has no tests or ways to reproduce. Rather than ask reviewers to set up zephyr builds so they can observe the issue, can you add a testcase for this to devtool's oe-selftest?
Alex
Quoted 9 lines · Jamin Lin via lists.openembedded.org
When a recipe uses multiple git
SRC_URIentries with different destsuffix values (e.g. Zephyr-based recipes with separate repositories for the kernel, modules and application),do_unpackclones each source tree with 'git clone -n -s'.The -s flag uses git's shared-object mechanism:
instead of copying objects locally it writes a .git/objects/info/alternates file pointing back to the bare repository underthe downloads directory (DL_DIR/git2/).
Quoted 30 lines
git_convert_standalone_clone()is called bydevtool_post_unpackto make the workspace standalone: it runs 'git repack -a' to copy all objects into the local object store and then removes the alternates file.However it only processes the top-level source directory. Each nested git repo created by a separate
SRC_URIentry retains its own alternates file still pointing into downloads/.Steps to reproduce:
1. devtool modify <recipe-with-multiple-git-SRC_URI>
2. bitbake -c cleanall <recipe>
3. bitbake <recipe>At step 2, 'bitbake -c cleanall' calls
fetcher.clean()which deletes the bare repositories from downloads/git2/. The top-level workspace repo is standalone (alternates already removed by the original code), but the nested repos still hold alternates pointing to the now-deleted paths.At step 3,
srctree_hash_files()runs 'git add -A .' with a customGIT_INDEX_FILE. Git internally calls 'git status --porcelain=2' on each nested repo to check for changes; this fails with exit 128 because the nested alternates are broken:
error: unable to normalize alternate object path:
.../downloads/git2/github.com.zephyrproject-rtos.acpica//objects
fatal: bad object HEAD
fatal: 'git status --porcelain=2' failed in submodule
modules/lib/acpicaQuoted 8 lines
This halts the BitBake parse phase with a CalledProcessError and leaves the workspace in an unrecoverable state without manual intervention.
Fix by extending
git_convert_standalone_clone()to walk the source tree and apply the same 'git repack -a' + remove-alternates treatment to every nested git repository found, making the entire workspace fully standalone at devtool modify time. The walk uses dirs[:] = [] to stop at each git boundary so it never descends into already-convertedrepos.
Quoted 11 lines
Signed-off-by Jamin Lin <[email protected]>--- scripts/lib/scriptutils.py | 28 ++++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-)scripts/lib/scriptutils.py +0
-0scripts/lib/scriptutils.py def load_plugins(logger, plugins, pluginpath): · line 103
Quoted 4 lines
scripts/lib/scriptutils.py +0
-1@@ -100,16 +100,36 @@ def load_plugins(logger, plugins, pluginpath): def git_convert_standalone_clone(repodir): - """If specified directory is a git repository, ensure it's a standaloneclone"""
scripts/lib/scriptutils.py def load_plugins(logger, plugins, pluginpath):
Quoted 3 lines
scripts/lib/scriptutils.py +3
-0+ """ + If specified directory is a git repository, ensure it's a standalone clone. + Also converts any nested git repositories (created by multiple SRC_URIgit
scripts/lib/scriptutils.py def load_plugins(logger, plugins, pluginpath):
Quoted 7 lines
scripts/lib/scriptutils.py +4
-2+ entries with different destsuffix values) so that none of their contents + depend on the shared downloads directory via alternates. + """ + import bb.process - if os.path.exists(os.path.join(repodir, '.git')): - alternatesfile = os.path.join(repodir, '.git', 'objects', 'info','alternates')
scripts/lib/scriptutils.py def load_plugins(logger, plugins, pluginpath):
Quoted 6 lines
scripts/lib/scriptutils.py +4
-0+ + def _convert(gitdir, workdir): + alternatesfile = os.path.join(gitdir, 'objects', 'info', + 'alternates') if os.path.exists(alternatesfile): # This will have been cloned with -s, so we need to convert itso none
Quoted 10 lines
scripts/lib/scriptutils.py +6
-1# of the contents is shared - bb.process.run('git repack -a', cwd=repodir) + bb.process.run('git repack -a', cwd=workdir) os.remove(alternatesfile) + if os.path.exists(os.path.join(repodir, '.git')): + _convert(os.path.join(repodir, '.git'), repodir) + + # Also handle nested git repos created by multiple SRC_URI git entries + # with different destsuffix values. Each nested repo is cloned with -sscripts/lib/scriptutils.py def load_plugins(logger, plugins, pluginpath):
Quoted 10 lines
scripts/lib/scriptutils.py +8
-0+ # and has its own alternates pointing to the downloads directory. + for root, dirs, files in os.walk(repodir): + if root == repodir: + continue + if '.git' in dirs: + _convert(os.path.join(root, '.git'), root) + dirs[:] = [] # don't recurse into nested repos + def _get_temp_recipe_dir(d): # This is a little bit hacky but we need to find a place where we canput
Quoted 15 lines
# the recipe so that bitbake can find it. We're going to delete it at the -- 2.43.0-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#237382):
https://lists.openembedded.org/g/openembedded-core/message/237382
Mute This Topic: https://lists.openembedded.org/mt/119403462/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-