Skip to contentSkip to conversation

devtool: fix standalone clone conversion for nested git repos

PATCH Superseded Jamin Lin 2 participants 6 messages May 22

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Jamin Lin

author
#0 lore raw

v2 cover devtool: fix standalone clone conversion for nested git repos

When a recipe has multiple git SRC_URI entries with destsuffix values nested inside S, devtool modify left all but the top-level repo with a .git/objects/info/alternates file pointing into the downloads cache.

Running 'bitbake -c cleanall' then removed those shared objects, breaking all subsequent git operations in the workspace.
Fix git_convert_standalone_clone() to walk all git repos nested inside S and repack each one to a fully standalone clone.

v1:
- Fix git_convert_standalone_clone() to walk all git repos nested inside S and repack each one to a standalone clone
v2:
- Add selftest recipe devtool-test-multi-destsuffix with six nested git SRC_URI entries to reproduce the scenario
- Add test_devtool_modify_multi_git_destsuffix_standalone to verify all nested repos have their alternates removed after devtool modify and remain usable after 'bitbake -c cleanall'

Jamin Lin (3):
scripts/scriptutils: Convert nested git repos to standalone clones in
devtool workspace
meta-selftest: Add devtool-test-multi-destsuffix recipe oeqa/selftest/devtool: Add test for multiple nested git destsuffix
repos

 .../devtool-test-multi-destsuffix_git.bb      | 29 ++++++++
 meta/lib/oeqa/selftest/cases/devtool.py       | 67 +++++++++++++++++++
 scripts/lib/scriptutils.py                    | 28 ++++++--
 3 files changed, 120 insertions(+), 4 deletions(-)
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb
Signature · 1 line
2.43.0
author
#1 lore raw

PATCH v2 1/3 scripts/scriptutils: Convert nested git repos to standalone clones in devtool workspace

When a recipe uses multiple git SRC_URI entries with different destsuffix values (e.g. Zephyr-based recipes with separate repositories for the kernel, modules and application), do_unpack clones each source tree with 'git clone -n -s'.

The -s flag uses git's shared-object mechanism:
instead of copying objects locally it writes a .git/objects/info/alternates file pointing back to the bare repository under the downloads directory (DL_DIR/git2/).

git_convert_standalone_clone() is called by devtool_post_unpack to make the workspace standalone: it runs 'git repack -a' to copy all objects into the local object store and then removes the alternates file.

However it only processes the top-level source directory. Each nested git repo created by a separate SRC_URI entry retains its own alternates file still pointing into downloads/.

Steps to reproduce:
1. devtool modify <recipe-with-multiple-git-SRC_URI>
2. bitbake -c cleanall <recipe>
3. bitbake <recipe>

At step 2, 'bitbake -c cleanall' calls fetcher.clean() which deletes the bare repositories from downloads/git2/. The top-level workspace repo is standalone (alternates already removed by the original code), but the nested repos still hold alternates pointing to the now-deleted paths.

At step 3, srctree_hash_files() runs 'git add -A .' with a custom GIT_INDEX_FILE. Git internally calls 'git status --porcelain=2' on each nested repo to check for changes; this fails with exit 128 because the nested alternates are broken:
error: unable to normalize alternate object path:
.../downloads/git2/github.com.zephyrproject-rtos.acpica//objects
fatal: bad object HEAD
fatal: 'git status --porcelain=2' failed in submodule modules/lib/acpica

This halts the BitBake parse phase with a CalledProcessError and leaves the workspace in an unrecoverable state without manual intervention.

Fix by extending git_convert_standalone_clone() to walk the source tree and apply the same 'git repack -a' + remove-alternates treatment to every nested git repository found, making the entire workspace fully standalone at devtool modify time. The walk uses dirs[:] = [] to stop at each git boundary so it never descends into already-converted repos.

---
 scripts/lib/scriptutils.py | 28 ++++++++++++++++++++++++----
 1 file changed, 24 insertions(+), 4 deletions(-)
scripts/lib/scriptutils.py +24 -4
index 32e749dbb1..0a83470373 100644
@@ -100,16 +100,36 @@ def load_plugins(logger, plugins, pluginpath):
 
 
 def git_convert_standalone_clone(repodir):
-    """If specified directory is a git repository, ensure it's a standalone clone"""
+    """
+    If specified directory is a git repository, ensure it's a standalone clone.
+    Also converts any nested git repositories (created by multiple SRC_URI git
+    entries with different destsuffix values) so that none of their contents
+    depend on the shared downloads directory via alternates.
+    """
+
     import bb.process
-    if os.path.exists(os.path.join(repodir, '.git')):
-        alternatesfile = os.path.join(repodir, '.git', 'objects', 'info', 'alternates')
+
+    def _convert(gitdir, workdir):
+        alternatesfile = os.path.join(gitdir, 'objects', 'info', 'alternates')
         if os.path.exists(alternatesfile):
             # This will have been cloned with -s, so we need to convert it so none
             # of the contents is shared
-            bb.process.run('git repack -a', cwd=repodir)
+            bb.process.run('git repack -a', cwd=workdir)
             os.remove(alternatesfile)
 
+    if os.path.exists(os.path.join(repodir, '.git')):
+        _convert(os.path.join(repodir, '.git'), repodir)
+
+    # Also handle nested git repos created by multiple SRC_URI git entries
+    # with different destsuffix values. Each nested repo is cloned with -s
+    # and has its own alternates pointing to the downloads directory.
+    for root, dirs, files in os.walk(repodir):
+        if root == repodir:
+            continue
+        if '.git' in dirs:
+            _convert(os.path.join(root, '.git'), root)
+            dirs[:] = []  # don't recurse into nested repos
+
 def _get_temp_recipe_dir(d):
     # This is a little bit hacky but we need to find a place where we can put
     # the recipe so that bitbake can find it. We're going to delete it at the
Signature · 1 line
2.43.0
author
#3 lore raw

PATCH v2 2/3 meta-selftest: Add devtool-test-multi-destsuffix recipe

In reply to Jamin Lin

Add a test recipe with six git SRC_URI entries using nested destsuffix values. This recipe is used by the devtool selftest
to verify that devtool modify correctly converts all nested git repos to standalone clones.

---
 .../devtool-test-multi-destsuffix_git.bb      | 29 +++++++++++++++++++
 1 file changed, 29 insertions(+)
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb
meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb new file +29 -0
new file mode 100644
index 0000000000..0702b13f3a
@@ -0,0 +1,29 @@
+SUMMARY = "Test recipe for multiple git SRC_URI entries with nested destsuffix values"
+LICENSE = "CLOSED"
+
+# Six git entries all nested inside S (${UNPACKDIR}). This exercises the
+# devtool code path that must convert every nested git repo to a standalone
+# clone: the initial fetch uses a shared clone whose alternates point into
+# downloads/git2/; git repack copies those objects locally so the workspace
+# survives 'bitbake -c cleanall'.
+SRC_URI = "git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level1_a;destsuffix=level1_a \
+    git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level1_b;destsuffix=level1_b \
+    git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level2_a;destsuffix=level1/level2_a \
+    git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level2_b;destsuffix=level1/level2_b \
+    git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level3_a;destsuffix=level1/level2/level3_a \
+    git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level3_b;destsuffix=level1/level2/level3_b \
+"
+
+SRCREV_level1_a = "64bc7c8fae61ded0c4e555aa775911f84c56e438"
+SRCREV_level1_b = "64bc7c8fae61ded0c4e555aa775911f84c56e438"
+SRCREV_level2_a = "64bc7c8fae61ded0c4e555aa775911f84c56e438"
+SRCREV_level2_b = "64bc7c8fae61ded0c4e555aa775911f84c56e438"
+SRCREV_level3_a = "64bc7c8fae61ded0c4e555aa775911f84c56e438"
+SRCREV_level3_b = "64bc7c8fae61ded0c4e555aa775911f84c56e438"
+SRCREV_FORMAT = "level1_a_level1_b_level2_a_level2_b_level3_a_level3_b"
+
+S = "${UNPACKDIR}"
+
+do_configure[noexec] = "1"
+do_compile[noexec] = "1"
+do_install[noexec] = "1"
Signature · 1 line
2.43.0
author
#5 lore raw

PATCH v2 3/3 oeqa/selftest/devtool: Add test for multiple nested git destsuffix repos

In reply to Jamin Lin

Add test_devtool_modify_multi_git_destsuffix_standalone to verify that devtool modify converts all nested git repos (from multiple SRC_URI git entries with different destsuffix values) to standalone clones so the workspace survives 'bitbake -c cleanall'.

---
 meta/lib/oeqa/selftest/cases/devtool.py | 67 +++++++++++++++++++++++++
 1 file changed, 67 insertions(+)
meta/lib/oeqa/selftest/cases/devtool.py +67 -0
index 5ed69aee1b..b927bbf8f0 100644
@@ -1182,6 +1182,73 @@ class DevtoolModifyTests(DevtoolBase):
         self.assertExists(os.path.join(source_repo_gitsm_gitmodules, 'bitbake'), 'Submodule not found')
         self.assertExists(os.path.join(source_repo_gitsm_gitmodules, 'bitbake-gitsm-test1'), 'Submodule not found')
 
+    def test_devtool_modify_multi_git_destsuffix_standalone(self):
+        """
+        Verify that devtool modify converts all nested git repos (from multiple
+        SRC_URI git entries with different destsuffix values) to standalone clones
+        so that 'bitbake -c cleanall' does not break the devtool workspace.
+
+        The recipe (devtool-test-multi-destsuffix) has six git SRC_URI entries
+        with S = ${UNPACKDIR}. All repos are nested inside S:
+            destsuffix=level1_a              -> srcdir/level1_a/
+            destsuffix=level1_b              -> srcdir/level1_b/
+            destsuffix=level1/level2_a       -> srcdir/level1/level2_a/
+            destsuffix=level1/level2_b       -> srcdir/level1/level2_b/
+            destsuffix=level1/level2/level3_a -> srcdir/level1/level2/level3_a/
+            destsuffix=level1/level2/level3_b -> srcdir/level1/level2/level3_b/
+
+        This mirrors real-world recipes that embed multiple module repos
+        as nested subdirectories of the primary source tree.
+        """
+        testrecipe = 'devtool-test-multi-destsuffix'
+        src_uri = get_bb_var('SRC_URI', testrecipe)
+        self.assertIn('git://', src_uri,
+                      'This test expects %s to have git SRC_URI entries' % testrecipe)
+        self.track_for_cleanup(self.workspacedir)
+        self.add_command_to_tearDown('devtool reset %s' % testrecipe)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        result = runCmd('devtool modify %s' % testrecipe)
+        self.assertEqual(result.status, 0,
+                         'devtool modify failed: %s' % result.output)
+        srcdir = os.path.join(self.workspacedir, 'sources', testrecipe)
+        nested_paths = [
+            ('level1_a', 'level1_a'),
+            ('level1_b', 'level1_b'),
+            ('level2_a', 'level1/level2_a'),
+            ('level2_b', 'level1/level2_b'),
+            ('level3_a', 'level1/level2/level3_a'),
+            ('level3_b', 'level1/level2/level3_b'),
+        ]
+
+        for name, subpath in nested_paths:
+            repo_path = os.path.join(srcdir, subpath)
+            self.assertExists(os.path.join(repo_path, '.git'),
+                              'Repo %s (.git) not found in devtool workspace' % name)
+
+        # Key assertion: no nested repo should retain a git alternates file.
+        # devtool modify must repack objects locally so the workspace does not
+        # depend on the downloads cache, which 'bitbake -c cleanall' will delete.
+        for name, subpath in nested_paths:
+            repo_path = os.path.join(srcdir, subpath)
+            alternates_file = os.path.join(repo_path, '.git', 'objects',
+                                           'info', 'alternates')
+            self.assertNotExists(alternates_file,
+                                 'Repo %s still has a git alternates file after '
+                                 'devtool modify' % name)
+
+        # Verify the workspace survives cleanall, which removes the shared
+        # objects in the downloads cache that alternates would reference.
+        bitbake('%s -c cleanall' % testrecipe)
+
+        # After cleanall all repos must still be usable.
+        # A broken alternates file would cause git operations to fail.
+        for name, subpath in nested_paths:
+            repo_path = os.path.join(srcdir, subpath)
+            result = runCmd('git status', cwd=repo_path)
+            self.assertEqual(result.status, 0,
+                             'git status failed in repo %s after cleanall: %s'
+                             % (name, result.output))
+
 class DevtoolUpdateTests(DevtoolBase):
 
     def test_devtool_update_recipe(self):
Signature · 1 line
2.43.0
#2 lore raw

Re: [PATCH v2 1/3] scripts/scriptutils: Convert nested git repos to standalone clones in devtool workspace

In reply to Jamin Lin
Quoted 82 lines · Jamin Lin

When a recipe uses multiple git SRC_URI entries with different destsuffix values (e.g. Zephyr-based recipes with separate repositories for the kernel, modules and application), do_unpack clones each source tree with 'git clone -n -s'.

The -s flag uses git's shared-object mechanism:
instead of copying objects locally it writes a .git/objects/info/alternates file pointing back to the bare repository under the downloads directory (DL_DIR/git2/).

git_convert_standalone_clone() is called by devtool_post_unpack to make the workspace standalone: it runs 'git repack -a' to copy all objects into the local object store and then removes the alternates file.

However it only processes the top-level source directory. Each nested git repo created by a separate SRC_URI entry retains its own alternates file still pointing into downloads/.

Steps to reproduce:
1. devtool modify <recipe-with-multiple-git-SRC_URI>
2. bitbake -c cleanall <recipe>
3. bitbake <recipe>

At step 2, 'bitbake -c cleanall' calls fetcher.clean() which deletes the bare repositories from downloads/git2/. The top-level workspace repo is standalone (alternates already removed by the original code), but the nested repos still hold alternates pointing to the now-deleted paths.

At step 3, srctree_hash_files() runs 'git add -A .' with a custom GIT_INDEX_FILE. Git internally calls 'git status --porcelain=2' on each nested repo to check for changes; this fails with exit 128 because the nested alternates are broken:
error: unable to normalize alternate object path:
.../downloads/git2/github.com.zephyrproject-rtos.acpica//objects
fatal: bad object HEAD
fatal: 'git status --porcelain=2' failed in submodule modules/lib/acpica

This halts the BitBake parse phase with a CalledProcessError and leaves the workspace in an unrecoverable state without manual intervention.

Fix by extending git_convert_standalone_clone() to walk the source tree and apply the same 'git repack -a' + remove-alternates treatment to every nested git repository found, making the entire workspace fully standalone at devtool modify time. The walk uses dirs[:] = [] to stop at each git boundary so it never descends into already-converted repos.

---
 scripts/lib/scriptutils.py | 28 ++++++++++++++++++++++++----
 1 file changed, 24 insertions(+), 4 deletions(-)
scripts/lib/scriptutils.py +13 -4
index 32e749dbb1..0a83470373 100644
@@ -100,16 +100,36 @@ def load_plugins(logger, plugins, pluginpath):
 
 
 def git_convert_standalone_clone(repodir):
-    """If specified directory is a git repository, ensure it's a standalone clone"""
+    """
+    If specified directory is a git repository, ensure it's a standalone clone.
+    Also converts any nested git repositories (created by multiple SRC_URI git
+    entries with different destsuffix values) so that none of their contents
+    depend on the shared downloads directory via alternates.
+    """
+
     import bb.process
-    if os.path.exists(os.path.join(repodir, '.git')):
-        alternatesfile = os.path.join(repodir, '.git', 'objects', 'info', 'alternates')
+
+    def _convert(gitdir, workdir):
+        alternatesfile = os.path.join(gitdir, 'objects', 'info', 'alternates')
         if os.path.exists(alternatesfile):
             # This will have been cloned with -s, so we need to convert it so none
             # of the contents is shared
-            bb.process.run('git repack -a', cwd=repodir)
+            bb.process.run('git repack -a', cwd=workdir)
             os.remove(alternatesfile)
 
+    if os.path.exists(os.path.join(repodir, '.git')):
+        _convert(os.path.join(repodir, '.git'), repodir)

scripts/lib/scriptutils.py def load_plugins(logger, plugins, pluginpath): · line 131

Quoted 10 lines
scripts/lib/scriptutils.py +10 -0
+
+    # Also handle nested git repos created by multiple SRC_URI git entries
+    # with different destsuffix values. Each nested repo is cloned with -s
+    # and has its own alternates pointing to the downloads directory.
+    for root, dirs, files in os.walk(repodir):
+        if root == repodir:
+            continue
+        if '.git' in dirs:
+            _convert(os.path.join(root, '.git'), root)
+            dirs[:] = []  # don't recurse into nested repos

Unfortunately, I don't think you can skip nested repos here. What about a recipe which unpacks a git repository as a subdirectory of another?

This makes me think, instead of walking the directory tree under ${S} looking for git repositories, could we walk the list of SRC_URI entries looking for 'git://' URIs? That should be a much shorter list. If that works, you'd probably want to do it in devtool-source.bbclass instead of here.

Best regards,

Signature · 1 line
Paul Barker
#4 lore raw

Re: [PATCH v2 2/3] meta-selftest: Add devtool-test-multi-destsuffix recipe

In reply to Jamin Lin
Quoted 22 lines · Jamin Lin

Add a test recipe with six git SRC_URI entries using nested destsuffix values. This recipe is used by the devtool selftest
to verify that devtool modify correctly converts all nested git repos to standalone clones.

---
 .../devtool-test-multi-destsuffix_git.bb      | 29 +++++++++++++++++++
 1 file changed, 29 insertions(+)
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb
meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb new file +5 -0
new file mode 100644
index 0000000000..0702b13f3a
@@ -0,0 +1,29 @@
+SUMMARY = "Test recipe for multiple git SRC_URI entries with nested destsuffix values"
+LICENSE = "CLOSED"
+
+# Six git entries all nested inside S (${UNPACKDIR}). This exercises the
+# devtool code path that must convert every nested git repo to a standalone

meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb line 15

Quoted 10 lines
meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb new file +10 -0
+# clone: the initial fetch uses a shared clone whose alternates point into
+# downloads/git2/; git repack copies those objects locally so the workspace
+# survives 'bitbake -c cleanall'.
+SRC_URI = "git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level1_a;destsuffix=level1_a \
+    git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level1_b;destsuffix=level1_b \
+    git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level2_a;destsuffix=level1/level2_a \
+    git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level2_b;destsuffix=level1/level2_b \
+    git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level3_a;destsuffix=level1/level2/level3_a \
+    git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level3_b;destsuffix=level1/level2/level3_b \
+"

Thanks for including a test case and recipe!

These do not appear to be "nested" git repositories though, as none is contained within another. We also don't need all 6 of them. Perhaps just three is enough, nested like this:

- destsuffix=level1
- destsuffix=level1/level2
- destsuffix=level1/level2/level3

Best regards,

Signature · 1 line
Paul Barker