devtool: fix standalone clone conversion for nested git repos
v2 cover devtool: fix standalone clone conversion for nested git repos
When a recipe has multiple git SRC_URI entries with destsuffix values nested inside S, devtool modify left all but the top-level repo with a .git/objects/info/alternates file pointing into the downloads cache.
Running 'bitbake -c cleanall' then removed those shared objects, breaking all subsequent git operations in the workspace.
Fix git_convert_standalone_clone() to walk all git repos nested inside S and repack each one to a fully standalone clone.
v1:
- Fix git_convert_standalone_clone() to walk all git repos nested inside S and repack each one to a standalone clone
v2:
- Add selftest recipe devtool-test-multi-destsuffix with six nested git SRC_URI entries to reproduce the scenario
- Add test_devtool_modify_multi_git_destsuffix_standalone to verify all nested repos have their alternates removed after devtool modify and remain usable after 'bitbake -c cleanall'
Jamin Lin (3):
scripts/scriptutils: Convert nested git repos to standalone clones in
devtool workspace
meta-selftest: Add devtool-test-multi-destsuffix recipe oeqa/selftest/devtool: Add test for multiple nested git destsuffix
repos
.../devtool-test-multi-destsuffix_git.bb | 29 ++++++++ meta/lib/oeqa/selftest/cases/devtool.py | 67 +++++++++++++++++++ scripts/lib/scriptutils.py | 28 ++++++-- 3 files changed, 120 insertions(+), 4 deletions(-) create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb
Signature · 1 line
2.43.0
PATCH v2 1/3 scripts/scriptutils: Convert nested git repos to standalone clones in devtool workspace
When a recipe uses multiple git SRC_URI entries with different destsuffix values (e.g. Zephyr-based recipes with separate repositories for the kernel, modules and application), do_unpack clones each source tree with 'git clone -n -s'.
The -s flag uses git's shared-object mechanism:
instead of copying objects locally it writes a .git/objects/info/alternates file pointing back to the bare repository under the downloads directory (DL_DIR/git2/).
git_convert_standalone_clone() is called by devtool_post_unpack to make the workspace standalone: it runs 'git repack -a' to copy all objects into the local object store and then removes the alternates file.
However it only processes the top-level source directory. Each nested git repo created by a separate SRC_URI entry retains its own alternates file still pointing into downloads/.
Steps to reproduce:
1. devtool modify <recipe-with-multiple-git-SRC_URI>
2. bitbake -c cleanall <recipe>
3. bitbake <recipe>
At step 2, 'bitbake -c cleanall' calls fetcher.clean() which deletes the bare repositories from downloads/git2/. The top-level workspace repo is standalone (alternates already removed by the original code), but the nested repos still hold alternates pointing to the now-deleted paths.
At step 3, srctree_hash_files() runs 'git add -A .' with a custom GIT_INDEX_FILE. Git internally calls 'git status --porcelain=2' on each nested repo to check for changes; this fails with exit 128 because the nested alternates are broken:
error: unable to normalize alternate object path:
.../downloads/git2/github.com.zephyrproject-rtos.acpica//objects
fatal: bad object HEAD
fatal: 'git status --porcelain=2' failed in submodule modules/lib/acpica
This halts the BitBake parse phase with a CalledProcessError and leaves the workspace in an unrecoverable state without manual intervention.
Fix by extending git_convert_standalone_clone() to walk the source tree and apply the same 'git repack -a' + remove-alternates treatment to every nested git repository found, making the entire workspace fully standalone at devtool modify time. The walk uses dirs[:] = [] to stop at each git boundary so it never descends into already-converted repos.
--- scripts/lib/scriptutils.py | 28 ++++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-)
scripts/lib/scriptutils.py +24 -4
@@ -100,16 +100,36 @@ def load_plugins(logger, plugins, pluginpath): def git_convert_standalone_clone(repodir): - """If specified directory is a git repository, ensure it's a standalone clone""" + """ + If specified directory is a git repository, ensure it's a standalone clone. + Also converts any nested git repositories (created by multiple SRC_URI git + entries with different destsuffix values) so that none of their contents + depend on the shared downloads directory via alternates. + """ + import bb.process - if os.path.exists(os.path.join(repodir, '.git')): - alternatesfile = os.path.join(repodir, '.git', 'objects', 'info', 'alternates') + + def _convert(gitdir, workdir): + alternatesfile = os.path.join(gitdir, 'objects', 'info', 'alternates') if os.path.exists(alternatesfile): # This will have been cloned with -s, so we need to convert it so none # of the contents is shared - bb.process.run('git repack -a', cwd=repodir) + bb.process.run('git repack -a', cwd=workdir) os.remove(alternatesfile) + if os.path.exists(os.path.join(repodir, '.git')): + _convert(os.path.join(repodir, '.git'), repodir) + + # Also handle nested git repos created by multiple SRC_URI git entries + # with different destsuffix values. Each nested repo is cloned with -s + # and has its own alternates pointing to the downloads directory. + for root, dirs, files in os.walk(repodir): + if root == repodir: + continue + if '.git' in dirs: + _convert(os.path.join(root, '.git'), root) + dirs[:] = [] # don't recurse into nested repos + def _get_temp_recipe_dir(d): # This is a little bit hacky but we need to find a place where we can put # the recipe so that bitbake can find it. We're going to delete it at the
Signature · 1 line
2.43.0
Re: [PATCH v2 1/3] scripts/scriptutils: Convert nested git repos to standalone clones in devtool workspace
Quoted 82 lines · Jamin Lin
When a recipe uses multiple git
SRC_URIentries with different destsuffix values (e.g. Zephyr-based recipes with separate repositories for the kernel, modules and application),do_unpackclones each source tree with 'git clone -n -s'.The -s flag uses git's shared-object mechanism:
instead of copying objects locally it writes a .git/objects/info/alternates file pointing back to the bare repository under the downloads directory (DL_DIR/git2/).
git_convert_standalone_clone()is called bydevtool_post_unpackto make the workspace standalone: it runs 'git repack -a' to copy all objects into the local object store and then removes the alternates file.However it only processes the top-level source directory. Each nested git repo created by a separate
SRC_URIentry retains its own alternates file still pointing into downloads/.Steps to reproduce:
1. devtool modify <recipe-with-multiple-git-SRC_URI>
2. bitbake -c cleanall <recipe>
3. bitbake <recipe>At step 2, 'bitbake -c cleanall' calls
fetcher.clean()which deletes the bare repositories from downloads/git2/. The top-level workspace repo is standalone (alternates already removed by the original code), but the nested repos still hold alternates pointing to the now-deleted paths.At step 3,
srctree_hash_files()runs 'git add -A .' with a customGIT_INDEX_FILE. Git internally calls 'git status --porcelain=2' on each nested repo to check for changes; this fails with exit 128 because the nested alternates are broken:
error: unable to normalize alternate object path:
.../downloads/git2/github.com.zephyrproject-rtos.acpica//objects
fatal: bad object HEAD
fatal: 'git status --porcelain=2' failed in submodule modules/lib/acpicaThis halts the BitBake parse phase with a CalledProcessError and leaves the workspace in an unrecoverable state without manual intervention.
Fix by extending
git_convert_standalone_clone()to walk the source tree and apply the same 'git repack -a' + remove-alternates treatment to every nested git repository found, making the entire workspace fully standalone at devtool modify time. The walk uses dirs[:] = [] to stop at each git boundary so it never descends into already-converted repos.Signed-off-by Jamin Lin <[email protected]>--- scripts/lib/scriptutils.py | 28 ++++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-)scripts/lib/scriptutils.py +13
-4@@ -100,16 +100,36 @@ def load_plugins(logger, plugins, pluginpath): def git_convert_standalone_clone(repodir): - """If specified directory is a git repository, ensure it's a standalone clone""" + """ + If specified directory is a git repository, ensure it's a standalone clone. + Also converts any nested git repositories (created by multiple SRC_URI git + entries with different destsuffix values) so that none of their contents + depend on the shared downloads directory via alternates. + """ + import bb.process - if os.path.exists(os.path.join(repodir, '.git')): - alternatesfile = os.path.join(repodir, '.git', 'objects', 'info', 'alternates') + + def _convert(gitdir, workdir): + alternatesfile = os.path.join(gitdir, 'objects', 'info', 'alternates') if os.path.exists(alternatesfile): # This will have been cloned with -s, so we need to convert it so none # of the contents is shared - bb.process.run('git repack -a', cwd=repodir) + bb.process.run('git repack -a', cwd=workdir) os.remove(alternatesfile) + if os.path.exists(os.path.join(repodir, '.git')): + _convert(os.path.join(repodir, '.git'), repodir)
scripts/lib/scriptutils.py def load_plugins(logger, plugins, pluginpath): · line 131
Quoted 10 lines
scripts/lib/scriptutils.py +10
-0+ + # Also handle nested git repos created by multiple SRC_URI git entries + # with different destsuffix values. Each nested repo is cloned with -s + # and has its own alternates pointing to the downloads directory. + for root, dirs, files in os.walk(repodir): + if root == repodir: + continue + if '.git' in dirs: + _convert(os.path.join(root, '.git'), root) + dirs[:] = [] # don't recurse into nested repos
Unfortunately, I don't think you can skip nested repos here. What about a recipe which unpacks a git repository as a subdirectory of another?
This makes me think, instead of walking the directory tree under ${S} looking for git repositories, could we walk the list of SRC_URI entries looking for 'git://' URIs? That should be a much shorter list. If that works, you'd probably want to do it in devtool-source.bbclass instead of here.
Best regards,
Signature · 1 line
Paul Barker
PATCH v2 2/3 meta-selftest: Add devtool-test-multi-destsuffix recipe
In reply to Jamin LinAdd a test recipe with six git SRC_URI entries using nested destsuffix values. This recipe is used by the devtool selftest
to verify that devtool modify correctly converts all nested git repos to standalone clones.
--- .../devtool-test-multi-destsuffix_git.bb | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb
meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb new file +29 -0
@@ -0,0 +1,29 @@ +SUMMARY = "Test recipe for multiple git SRC_URI entries with nested destsuffix values" +LICENSE = "CLOSED" + +# Six git entries all nested inside S (${UNPACKDIR}). This exercises the +# devtool code path that must convert every nested git repo to a standalone +# clone: the initial fetch uses a shared clone whose alternates point into +# downloads/git2/; git repack copies those objects locally so the workspace +# survives 'bitbake -c cleanall'. +SRC_URI = "git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level1_a;destsuffix=level1_a \ + git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level1_b;destsuffix=level1_b \ + git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level2_a;destsuffix=level1/level2_a \ + git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level2_b;destsuffix=level1/level2_b \ + git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level3_a;destsuffix=level1/level2/level3_a \ + git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level3_b;destsuffix=level1/level2/level3_b \ +" + +SRCREV_level1_a = "64bc7c8fae61ded0c4e555aa775911f84c56e438" +SRCREV_level1_b = "64bc7c8fae61ded0c4e555aa775911f84c56e438" +SRCREV_level2_a = "64bc7c8fae61ded0c4e555aa775911f84c56e438" +SRCREV_level2_b = "64bc7c8fae61ded0c4e555aa775911f84c56e438" +SRCREV_level3_a = "64bc7c8fae61ded0c4e555aa775911f84c56e438" +SRCREV_level3_b = "64bc7c8fae61ded0c4e555aa775911f84c56e438" +SRCREV_FORMAT = "level1_a_level1_b_level2_a_level2_b_level3_a_level3_b" + +S = "${UNPACKDIR}" + +do_configure[noexec] = "1" +do_compile[noexec] = "1" +do_install[noexec] = "1"
Signature · 1 line
2.43.0
Re: [PATCH v2 2/3] meta-selftest: Add devtool-test-multi-destsuffix recipe
Quoted 22 lines · Jamin Lin
Add a test recipe with six git
SRC_URIentries using nested destsuffix values. This recipe is used by the devtool selftest
to verify that devtool modify correctly converts all nested git repos to standalone clones.Signed-off-by Jamin Lin <[email protected]>--- .../devtool-test-multi-destsuffix_git.bb | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bbmeta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb new file +5
-0@@ -0,0 +1,29 @@ +SUMMARY = "Test recipe for multiple git SRC_URI entries with nested destsuffix values" +LICENSE = "CLOSED" + +# Six git entries all nested inside S (${UNPACKDIR}). This exercises the +# devtool code path that must convert every nested git repo to a standalone
meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb line 15
Quoted 10 lines
meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb new file +10
-0+# clone: the initial fetch uses a shared clone whose alternates point into +# downloads/git2/; git repack copies those objects locally so the workspace +# survives 'bitbake -c cleanall'. +SRC_URI = "git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level1_a;destsuffix=level1_a \ + git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level1_b;destsuffix=level1_b \ + git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level2_a;destsuffix=level1/level2_a \ + git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level2_b;destsuffix=level1/level2_b \ + git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level3_a;destsuffix=level1/level2/level3_a \ + git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level3_b;destsuffix=level1/level2/level3_b \ +"
Thanks for including a test case and recipe!
These do not appear to be "nested" git repositories though, as none is contained within another. We also don't need all 6 of them. Perhaps just three is enough, nested like this:
- destsuffix=level1
- destsuffix=level1/level2
- destsuffix=level1/level2/level3
Best regards,
Signature · 1 line
Paul Barker
PATCH v2 3/3 oeqa/selftest/devtool: Add test for multiple nested git destsuffix repos
In reply to Jamin LinAdd test_devtool_modify_multi_git_destsuffix_standalone to verify that devtool modify converts all nested git repos (from multiple SRC_URI git entries with different destsuffix values) to standalone clones so the workspace survives 'bitbake -c cleanall'.
--- meta/lib/oeqa/selftest/cases/devtool.py | 67 +++++++++++++++++++++++++ 1 file changed, 67 insertions(+)
meta/lib/oeqa/selftest/cases/devtool.py +67 -0
@@ -1182,6 +1182,73 @@ class DevtoolModifyTests(DevtoolBase): self.assertExists(os.path.join(source_repo_gitsm_gitmodules, 'bitbake'), 'Submodule not found') self.assertExists(os.path.join(source_repo_gitsm_gitmodules, 'bitbake-gitsm-test1'), 'Submodule not found') + def test_devtool_modify_multi_git_destsuffix_standalone(self): + """ + Verify that devtool modify converts all nested git repos (from multiple + SRC_URI git entries with different destsuffix values) to standalone clones + so that 'bitbake -c cleanall' does not break the devtool workspace. + + The recipe (devtool-test-multi-destsuffix) has six git SRC_URI entries + with S = ${UNPACKDIR}. All repos are nested inside S: + destsuffix=level1_a -> srcdir/level1_a/ + destsuffix=level1_b -> srcdir/level1_b/ + destsuffix=level1/level2_a -> srcdir/level1/level2_a/ + destsuffix=level1/level2_b -> srcdir/level1/level2_b/ + destsuffix=level1/level2/level3_a -> srcdir/level1/level2/level3_a/ + destsuffix=level1/level2/level3_b -> srcdir/level1/level2/level3_b/ + + This mirrors real-world recipes that embed multiple module repos + as nested subdirectories of the primary source tree. + """ + testrecipe = 'devtool-test-multi-destsuffix' + src_uri = get_bb_var('SRC_URI', testrecipe) + self.assertIn('git://', src_uri, + 'This test expects %s to have git SRC_URI entries' % testrecipe) + self.track_for_cleanup(self.workspacedir) + self.add_command_to_tearDown('devtool reset %s' % testrecipe) + self.add_command_to_tearDown('bitbake-layers remove-layer */workspace') + result = runCmd('devtool modify %s' % testrecipe) + self.assertEqual(result.status, 0, + 'devtool modify failed: %s' % result.output) + srcdir = os.path.join(self.workspacedir, 'sources', testrecipe) + nested_paths = [ + ('level1_a', 'level1_a'), + ('level1_b', 'level1_b'), + ('level2_a', 'level1/level2_a'), + ('level2_b', 'level1/level2_b'), + ('level3_a', 'level1/level2/level3_a'), + ('level3_b', 'level1/level2/level3_b'), + ] + + for name, subpath in nested_paths: + repo_path = os.path.join(srcdir, subpath) + self.assertExists(os.path.join(repo_path, '.git'), + 'Repo %s (.git) not found in devtool workspace' % name) + + # Key assertion: no nested repo should retain a git alternates file. + # devtool modify must repack objects locally so the workspace does not + # depend on the downloads cache, which 'bitbake -c cleanall' will delete. + for name, subpath in nested_paths: + repo_path = os.path.join(srcdir, subpath) + alternates_file = os.path.join(repo_path, '.git', 'objects', + 'info', 'alternates') + self.assertNotExists(alternates_file, + 'Repo %s still has a git alternates file after ' + 'devtool modify' % name) + + # Verify the workspace survives cleanall, which removes the shared + # objects in the downloads cache that alternates would reference. + bitbake('%s -c cleanall' % testrecipe) + + # After cleanall all repos must still be usable. + # A broken alternates file would cause git operations to fail. + for name, subpath in nested_paths: + repo_path = os.path.join(srcdir, subpath) + result = runCmd('git status', cwd=repo_path) + self.assertEqual(result.status, 0, + 'git status failed in repo %s after cleanall: %s' + % (name, result.output)) + class DevtoolUpdateTests(DevtoolBase): def test_devtool_update_recipe(self):
Signature · 1 line
2.43.0