Skip to contentSkip to conversation

oe: Fix build failures with multiple git SRC_URI entries

PATCH In review Jamin Lin 2 participants 11 messages May 15 – May 20

lore
Share & tools
lore

Jamin Lin

author
#0 lore raw

cover oe: Fix build failures with multiple git SRC_URI entries

Some recipes (e.g. Zephyr-based) use multiple git SRC_URI entries with different destsuffix values, causing each source to be unpacked into a separate subdirectory of EXTERNALSRC that retains its own .git directory.

These nested git repositories trigger two independent failures:

1. externalsrc.bbclass: 'git add -A .' exits with code 128 during srctree_hash_files(), halting the bitbake parse phase.

2. oe/reproducible.py: 'git log -1' exits with code 128 inside a nested repo found by find_git_repositories(), aborting do_unpack.

Jamin Lin (2):
externalsrc: Handle nested git repos from multiple SRC_URI entries
reproducible: Handle nested git repos in find_git_repositories

 meta/classes/externalsrc.bbclass | 37 +++++++++++++++++++++++++++++++-
 meta/lib/oe/reproducible.py      |  6 +++++-
 2 files changed, 41 insertions(+), 2 deletions(-)
Signature · 1 line
2.43.0
author
#1 lore raw

PATCH 1/2 externalsrc: Handle nested git repos from multiple SRC_URI entries

When a recipe uses multiple git SRC_URI entries with different destsuffix values (e.g. Zephyr-based recipes with separate repos for the kernel, modules, and application), each source is unpacked into a subdirectory of EXTERNALSRC that retains its own .git directory.

srctree_hash_files() calls 'git add -A .' at the EXTERNALSRC root, which fails with exit code 128 when git encounters these unregistered nested git repositories, halting the bitbake parse phase.

Fix by scanning for nested git repos before the add. If any are found, exclude them from the top-level 'git add' using pathspec magic ':(exclude)<path>' and hash each nested repo independently using a temporary index. This ensures changes in any nested repo still trigger do_compile/do_configure to re-run.

---
 meta/classes/externalsrc.bbclass | 37 +++++++++++++++++++++++++++++++-
 1 file changed, 36 insertions(+), 1 deletion(-)
meta/classes/externalsrc.bbclass +36 -1
index 902ff2604f..0dd57af668 100644
@@ -234,8 +234,43 @@ def srctree_hash_files(d, srcdir=None):
             # Update our custom index
             env = os.environ.copy()
             env['GIT_INDEX_FILE'] = tmp_index.name
-            subprocess.check_output(['git', 'add', '-A', '.'], cwd=s_dir, env=env)
+            # Find nested git repos created by multiple SRC_URI git entries with
+            # different destsuffix values. git add -A . exits 128 when it encounters
+            # these unregistered nested repos.
+            nested_git_dirs = []
+            for root, dirs, files in os.walk(s_dir):
+                if root == s_dir:
+                    continue
+                if '.git' in dirs or '.git' in files:
+                    nested_git_dirs.append(root)
+                    dirs[:] = []  # don't recurse into nested repos
+            if nested_git_dirs:
+                excludes = [':(exclude)' + os.path.relpath(n, s_dir) for n in nested_git_dirs]
+                subprocess.check_output(['git', 'add', '-A', '.'] + excludes, cwd=s_dir, env=env)
+            else:
+                subprocess.check_output(['git', 'add', '-A', '.'], cwd=s_dir, env=env)
             git_sha1 = subprocess.check_output(['git', 'write-tree'], cwd=s_dir, env=env).decode("utf-8")
+            # Hash each nested git repo separately so source changes there still
+            # trigger do_compile/do_configure to re-run.
+            for nested in nested_git_dirs:
+                nested_git = os.path.join(nested, '.git')
+                if not os.path.isdir(nested_git):
+                    continue
+                with tempfile.NamedTemporaryFile(prefix='oe-devtool-nested-index') as nested_tmp:
+                    nested_index = os.path.join(nested_git, 'index')
+                    if os.path.exists(nested_index):
+                        shutil.copyfile(nested_index, nested_tmp.name)
+                    nested_env = os.environ.copy()
+                    nested_env['GIT_INDEX_FILE'] = nested_tmp.name
+                    proc = subprocess.Popen(['git', 'add', '-A', '.'], cwd=nested,
+                                           env=nested_env, stdout=subprocess.DEVNULL,
+                                           stderr=subprocess.DEVNULL)
+                    proc.communicate()
+                    proc = subprocess.Popen(['git', 'write-tree'], cwd=nested,
+                                           env=nested_env, stdout=subprocess.PIPE,
+                                           stderr=subprocess.DEVNULL)
+                    stdout, _ = proc.communicate()
+                    git_sha1 += stdout.decode("utf-8")
             if os.path.exists(os.path.join(s_dir, ".gitmodules")) and os.path.getsize(os.path.join(s_dir, ".gitmodules")) > 0:
                 submodule_helper = subprocess.check_output(["git", "config", "--file", ".gitmodules", "--get-regexp", "path"], cwd=s_dir, env=env).decode("utf-8")
                 for line in submodule_helper.splitlines():
Signature · 1 line
2.43.0
#2 lore raw

Re: [PATCH v1 1/2] externalsrc: Handle nested git repos from multiple SRC_URI entries

Quoted 8 lines · Jamin Lin

When a recipe uses multiple git SRC_URI entries with different destsuffix values (e.g. Zephyr-based recipes with separate repos for the kernel, modules, and application), each source is unpacked into a subdirectory of EXTERNALSRC that retains its own .git directory.

srctree_hash_files() calls 'git add -A .' at the EXTERNALSRC root, which fails with exit code 128 when git encounters these unregistered nested git repositories, halting the bitbake parse phase.

Is this true? The documentation for git add [1] talks about issuing a warning when this occurs, not an error, and in some quick local testing I get a successful exit (exit code 0) when I try this.

Quoted 20 lines

Fix by scanning for nested git repos before the add. If any are found, exclude them from the top-level 'git add' using pathspec magic ':(exclude)<path>' and hash each nested repo independently using a temporary index. This ensures changes in any nested repo still trigger do_compile/do_configure to re-run.

---
 meta/classes/externalsrc.bbclass | 37 +++++++++++++++++++++++++++++++-
 1 file changed, 36 insertions(+), 1 deletion(-)
meta/classes/externalsrc.bbclass +0 -1
index 902ff2604f..0dd57af668 100644
@@ -234,8 +234,43 @@ def srctree_hash_files(d, srcdir=None):
             # Update our custom index
             env = os.environ.copy()
             env['GIT_INDEX_FILE'] = tmp_index.name
-            subprocess.check_output(['git', 'add', '-A', '.'], cwd=s_dir, env=env)

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 246

Quoted 10 lines
meta/classes/externalsrc.bbclass +10 -0
+            # Find nested git repos created by multiple SRC_URI git entries with
+            # different destsuffix values. git add -A . exits 128 when it encounters
+            # these unregistered nested repos.
+            nested_git_dirs = []
+            for root, dirs, files in os.walk(s_dir):
+                if root == s_dir:
+                    continue
+                if '.git' in dirs or '.git' in files:
+                    nested_git_dirs.append(root)
+                    dirs[:] = []  # don't recurse into nested repos

This os.walk() loop is expensive, is there an alternative way to handle this?

The code has also become difficult to parse. My rule of thumb is that if a group of lines needs a leading comment, it also needs an empty line before the comment to visually separate things.

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 251

Quoted 5 lines
meta/classes/externalsrc.bbclass +5 -0
+            if nested_git_dirs:
+                excludes = [':(exclude)' + os.path.relpath(n, s_dir) for n in nested_git_dirs]
+                subprocess.check_output(['git', 'add', '-A', '.'] + excludes, cwd=s_dir, env=env)
+            else:
+                subprocess.check_output(['git', 'add', '-A', '.'], cwd=s_dir, env=env)

To simplify the code, construct a cmd variable and call subprocess.check_output(cmd, ...) once.

Quoted 12 lines
meta/classes/externalsrc.bbclass +11 -0
             git_sha1 = subprocess.check_output(['git', 'write-tree'], cwd=s_dir, env=env).decode("utf-8")
+            # Hash each nested git repo separately so source changes there still
+            # trigger do_compile/do_configure to re-run.
+            for nested in nested_git_dirs:
+                nested_git = os.path.join(nested, '.git')
+                if not os.path.isdir(nested_git):
+                    continue
+                with tempfile.NamedTemporaryFile(prefix='oe-devtool-nested-index') as nested_tmp:
+                    nested_index = os.path.join(nested_git, 'index')
+                    if os.path.exists(nested_index):
+                        shutil.copyfile(nested_index, nested_tmp.name)
+                    nested_env = os.environ.copy()

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 273

Quoted 10 lines
meta/classes/externalsrc.bbclass +10 -0
+                    nested_env['GIT_INDEX_FILE'] = nested_tmp.name
+                    proc = subprocess.Popen(['git', 'add', '-A', '.'], cwd=nested,
+                                           env=nested_env, stdout=subprocess.DEVNULL,
+                                           stderr=subprocess.DEVNULL)
+                    proc.communicate()
+                    proc = subprocess.Popen(['git', 'write-tree'], cwd=nested,
+                                           env=nested_env, stdout=subprocess.PIPE,
+                                           stderr=subprocess.DEVNULL)
+                    stdout, _ = proc.communicate()
+                    git_sha1 += stdout.decode("utf-8")

We should re-use the code from the following block which handles submodules instead of re-implementing the behaviour. Perhaps the common code needs to be refactored out.

Best regards,

Signature · 1 line
Paul Barker
author
#3 lore raw

RE: [PATCH v1 1/2] externalsrc: Handle nested git repos from multiple SRC_URI entries

Hi Paul,

Quoted 9 lines

Subject: Re: [PATCH v1 1/2] externalsrc: Handle nested git repos from multiple SRC_URI entries

Quoted 5 lines · Jamin Lin

When a recipe uses multiple git SRC_URI entries with different destsuffix values (e.g. Zephyr-based recipes with separate repos for the kernel, modules, and application), each source is unpacked into a subdirectory of EXTERNALSRC that retains its own .git directory.

Quoted 8 lines
Quoted 3 lines

srctree_hash_files() calls 'git add -A .' at the EXTERNALSRC root, which fails with exit code 128 when git encounters these unregistered nested git repositories, halting the bitbake parse phase.

Is this true? The documentation for git add [1] talks about issuing a warning when this occurs, not an error, and in some quick local testing I get a successful exit (exit code 0) when I try this.

Please try the "zephyr-helloworld" package.

This recipe uses multiple Git repositories in "SRC_URI", and I can reproduce the issue with this package:

https://git.yoctoproject.org/meta-zephyr/tree/meta-zephyr-core/recipes-kernel/zephyr-kernel/zephyr-kernel-src-4.3.0.inc

Steps to reproduce:

1. git clone https://github.com/openembedded/bitbake.git
2. git clone https://github.com/openembedded/openembedded-core.git
3. git clone https://git.yoctoproject.org/meta-zephyr
4. git clone https://github.com/openembedded/meta-openembedded.git
5. source openembedded-core/oe-init-build-env build
6. Edit build/conf/bblayers.conf and add the following layers:

* meta-openembedded/meta-python
* meta-zephyr/meta-zephyr-core
* meta-openembedded/meta-oe
7. devtool modify zephyr-helloworld
8. bitbake -c cleanall zephyr-helloworld
9. bitbake zephyr-helloworld

After that, the following parser error occurs, and the package can no longer be built:

```text
jamin_lin@aspeed-fw02:~/oe-review/build$ bitbake zephyr-helloworld
Loading cache: 100% |#########################################################################################################################################################################################################| Time: 0:00:01
Loaded 4949 entries from dependency cache.
ERROR: ExpansionError during parsing /home/jamin_lin/oe-review/meta-zephyr/meta-zephyr-core/recipes-kernel/zephyr-kernel/zephyr-helloworld.bb
bb.data_smart.ExpansionError: Failure expanding variable do_compile[file-checksums], expression was ${@srctree_hash_files(d)} which triggered exception CalledProcessError: Command '['git', 'add', '-A', '.']' returned non-zero exit status 128.
The variable dependency chain for the failure is: do_compile[file-checksums]

ERROR: Parsing halted due to errors, see error messages above
Parsing recipes:   0% |                                                                                                                                                                                                      | ETA:  --:--:--
Summary: There were 2 ERROR messages, returning a non-zero exit code.
```

I can consistently reproduce this issue with the steps above.

Thanks,
Jamin

Quoted 96 lines
Quoted 18 lines

Fix by scanning for nested git repos before the add. If any are found, exclude them from the top-level 'git add' using pathspec magic ':(exclude)<path>' and hash each nested repo independently using a temporary index. This ensures changes in any nested repo still trigger do_compile/do_configure to re-run.

---
 meta/classes/externalsrc.bbclass | 37
+++++++++++++++++++++++++++++++-
 1 file changed, 36 insertions(+), 1 deletion(-)
meta/classes/externalsrc.bbclass +0 -0
index 902ff2604f..0dd57af668 100644

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 237

Quoted 4 lines
meta/classes/externalsrc.bbclass +0 -1
@@ -234,8 +234,43 @@ def srctree_hash_files(d, srcdir=None):
             # Update our custom index
             env = os.environ.copy()
             env['GIT_INDEX_FILE'] = tmp_index.name
-            subprocess.check_output(['git', 'add', '-A', '.'], cwd=s_dir,

env=env)

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 237

Quoted 1 line
meta/classes/externalsrc.bbclass +1 -0
+            # Find nested git repos created by multiple SRC_URI git

entries with

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 238

Quoted 1 line
meta/classes/externalsrc.bbclass +1 -0
+            # different destsuffix values. git add -A . exits 128 when it

encounters

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 246

Quoted 8 lines
meta/classes/externalsrc.bbclass +8 -0
+            # these unregistered nested repos.
+            nested_git_dirs = []
+            for root, dirs, files in os.walk(s_dir):
+                if root == s_dir:
+                    continue
+                if '.git' in dirs or '.git' in files:
+                    nested_git_dirs.append(root)
+                    dirs[:] = []  # don't recurse into nested repos

This os.walk() loop is expensive, is there an alternative way to handle this?

The code has also become difficult to parse. My rule of thumb is that if a group of lines needs a leading comment, it also needs an empty line before the comment to visually separate things.

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 248

Quoted 2 lines
meta/classes/externalsrc.bbclass +2 -0
+            if nested_git_dirs:
+                excludes = [':(exclude)' + os.path.relpath(n, s_dir) for n

in nested_git_dirs]

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 249

Quoted 1 line
meta/classes/externalsrc.bbclass +1 -0
+                subprocess.check_output(['git', 'add', '-A', '.'] + excludes,

cwd=s_dir, env=env)

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 252

Quoted 3 lines
meta/classes/externalsrc.bbclass +3 -0
+            else:
+                subprocess.check_output(['git', 'add', '-A', '.'],
+ cwd=s_dir, env=env)

To simplify the code, construct a cmd variable and call subprocess.check_output(cmd, ...) once.

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 255

Quoted 3 lines
meta/classes/externalsrc.bbclass +1 -0
             git_sha1 = subprocess.check_output(['git', 'write-tree'],
 cwd=s_dir, env=env).decode("utf-8")
+            # Hash each nested git repo separately so source changes

there still

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None):

Quoted 6 lines
meta/classes/externalsrc.bbclass +6 -0
+            # trigger do_compile/do_configure to re-run.
+            for nested in nested_git_dirs:
+                nested_git = os.path.join(nested, '.git')
+                if not os.path.isdir(nested_git):
+                    continue
+                with

tempfile.NamedTemporaryFile(prefix='oe-devtool-nested-index') as nested_tmp:

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None):

Quoted 3 lines
meta/classes/externalsrc.bbclass +3 -0
+                    nested_index = os.path.join(nested_git, 'index')
+                    if os.path.exists(nested_index):
+                        shutil.copyfile(nested_index,

nested_tmp.name)

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None):

Quoted 3 lines
meta/classes/externalsrc.bbclass +3 -0
+                    nested_env = os.environ.copy()
+                    nested_env['GIT_INDEX_FILE'] = nested_tmp.name
+                    proc = subprocess.Popen(['git', 'add', '-A', '.'],

cwd=nested,

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 268

Quoted 1 line
meta/classes/externalsrc.bbclass +1 -0
+                                           env=nested_env,

stdout=subprocess.DEVNULL,

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 269

Quoted 1 line
meta/classes/externalsrc.bbclass +1 -0
+

stderr=subprocess.DEVNULL)

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 271

Quoted 2 lines
meta/classes/externalsrc.bbclass +2 -0
+                    proc.communicate()
+                    proc = subprocess.Popen(['git', 'write-tree'],

cwd=nested,

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 272

Quoted 1 line
meta/classes/externalsrc.bbclass +1 -0
+                                           env=nested_env,

stdout=subprocess.PIPE,

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 273

Quoted 1 line
meta/classes/externalsrc.bbclass +1 -0
+

stderr=subprocess.DEVNULL)

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 273

Quoted 2 lines
meta/classes/externalsrc.bbclass +2 -0
+                    stdout, _ = proc.communicate()
+                    git_sha1 += stdout.decode("utf-8")

We should re-use the code from the following block which handles submodules instead of re-implementing the behaviour. Perhaps the common code needs to be refactored out.

Best regards,

--
Paul Barker

#4 lore raw

Re: [OE-core] [PATCH v1 1/2] externalsrc: Handle nested git repos from multiple SRC_URI entries

Quoted 44 lines · Jamin Lin via lists.openembedded.org

Hi Paul,

Quoted 9 lines

Subject: Re: [PATCH v1 1/2] externalsrc: Handle nested git repos from multiple SRC_URI entries

Quoted 5 lines · Jamin Lin

When a recipe uses multiple git SRC_URI entries with different destsuffix values (e.g. Zephyr-based recipes with separate repos for the kernel, modules, and application), each source is unpacked into a subdirectory of EXTERNALSRC that retains its own .git directory.

Quoted 8 lines
Quoted 3 lines

srctree_hash_files() calls 'git add -A .' at the EXTERNALSRC root, which fails with exit code 128 when git encounters these unregistered nested git repositories, halting the bitbake parse phase.

Is this true? The documentation for git add [1] talks about issuing a warning when this occurs, not an error, and in some quick local testing I get a successful exit (exit code 0) when I try this.

Please try the "zephyr-helloworld" package.

This recipe uses multiple Git repositories in "SRC_URI", and I can reproduce the issue with this package:

https://git.yoctoproject.org/meta-zephyr/tree/meta-zephyr-core/recipes-kernel/zephyr-kernel/zephyr-kernel-src-4.3.0.inc

Steps to reproduce:

1. git clone https://github.com/openembedded/bitbake.git
2. git clone https://github.com/openembedded/openembedded-core.git
3. git clone https://git.yoctoproject.org/meta-zephyr
4. git clone https://github.com/openembedded/meta-openembedded.git
5. source openembedded-core/oe-init-build-env build
6. Edit build/conf/bblayers.conf and add the following layers:

* meta-openembedded/meta-python
* meta-zephyr/meta-zephyr-core
* meta-openembedded/meta-oe
7. devtool modify zephyr-helloworld
8. bitbake -c cleanall zephyr-helloworld
9. bitbake zephyr-helloworld

After that, the following parser error occurs, and the package can no longer be built:

Quoted 12 lines
```text
jamin_lin@aspeed-fw02:~/oe-review/build$ bitbake zephyr-helloworld
Loading cache: 100% |#########################################################################################################################################################################################################| Time: 0:00:01
Loaded 4949 entries from dependency cache.
ERROR: ExpansionError during parsing /home/jamin_lin/oe-review/meta-zephyr/meta-zephyr-core/recipes-kernel/zephyr-kernel/zephyr-helloworld.bb
bb.data_smart.ExpansionError: Failure expanding variable do_compile[file-checksums], expression was ${@srctree_hash_files(d)} which triggered exception CalledProcessError: Command '['git', 'add', '-A', '.']' returned non-zero exit status 128.
The variable dependency chain for the failure is: do_compile[file-checksums]

ERROR: Parsing halted due to errors, see error messages above
Parsing recipes:   0% |                                                                                                                                                                                                      | ETA:  --:--:--
Summary: There were 2 ERROR messages, returning a non-zero exit code.
```

Following these steps I see the same error. But the cause is not the nested git repositories. bitbake -c cleanall ... has deleted the sources from the downloads directory, but the devtool workspace is still referencing the deleted sources. Even a git status command in the zephyr-helloworld sources directory fails:

  $ git -C workspace/sources/zephyr-helloworld status
  error: unable to normalize alternate object path: /home/pbarker/bitbake-builds/.bitbake-setup-downloads/git2/github.com.zephyrproject-rtos.mcuboot//objects
  error: unable to normalize alternate object path: /home/pbarker/bitbake-builds/.bitbake-setup-downloads/git2/github.com.zephyrproject-rtos.mcuboot//objects
  error: unable to normalize alternate object path: /home/pbarker/bitbake-builds/.bitbake-setup-downloads/git2/github.com.zephyrproject-rtos.mcuboot//objects
  error: unable to normalize alternate object path: /home/pbarker/bitbake-builds/.bitbake-setup-downloads/git2/github.com.zephyrproject-rtos.mcuboot//objects
  fatal: bad object HEAD
  fatal: 'git status --porcelain=2' failed in submodule bootloader/mcuboot

If I replace step 8 with bitbake -c cleansstate ... then there is no parse error.

What we need to fix is the interaction between devtool modify and bitbake -c cleanall. Perhaps devtool needs to detect this broken state and tell the user how to fix it - simply running bitbake -c fetch zephyr-helloworld isn't possible as the metadata can't be parsed.

Best regards,

Signature · 1 line
Paul Barker
author
#5 lore raw

RE: [OE-core] [PATCH v1 1/2] externalsrc: Handle nested git repos from multiple SRC_URI entries

Hi Paul,

Quoted 104 lines

Subject: Re: [OE-core] [PATCH v1 1/2] externalsrc: Handle nested git repos from multiple SRC_URI entries

Quoted 15 lines · Jamin Lin via lists.openembedded.org

Hi Paul,

Quoted 13 lines

Subject: Re: [PATCH v1 1/2] externalsrc: Handle nested git repos from multiple SRC_URI entries

Quoted 8 lines · Jamin Lin

When a recipe uses multiple git SRC_URI entries with different destsuffix values (e.g. Zephyr-based recipes with separate repos for the kernel, modules, and application), each source is unpacked into a subdirectory of EXTERNALSRC that retains its own .git directory.

srctree_hash_files() calls 'git add -A .' at the EXTERNALSRC root, which fails with exit code 128 when git encounters these unregistered nested git repositories, halting the bitbake parse phase.

Quoted 8 lines
Quoted 4 lines

Is this true? The documentation for git add [1] talks about issuing a warning when this occurs, not an error, and in some quick local testing I get a successful exit (exit code 0) when I try this.

Please try the "zephyr-helloworld" package.

This recipe uses multiple Git repositories in "SRC_URI", and I can reproduce

the issue with this package:

Quoted 13 lines

https://git.yoctoproject.org/meta-zephyr/tree/meta-zephyr-core/recipes
-kernel/zephyr-kernel/zephyr-kernel-src-4.3.0.inc

Steps to reproduce:

1. git clone https://github.com/openembedded/bitbake.git
2. git clone https://github.com/openembedded/openembedded-core.git
3. git clone https://git.yoctoproject.org/meta-zephyr
4. git clone https://github.com/openembedded/meta-openembedded.git
5. source openembedded-core/oe-init-build-env build 6. Edit build/conf/bblayers.conf and add the following layers:

Quoted 7 lines

* meta-openembedded/meta-python
* meta-zephyr/meta-zephyr-core
* meta-openembedded/meta-oe
7. devtool modify zephyr-helloworld
8. bitbake -c cleanall zephyr-helloworld 9. bitbake zephyr-helloworld

After that, the following parser error occurs, and the package can no longer

be built:

Quoted 5 lines

```text

jamin_lin@aspeed-fw02:~/oe-review/build$ bitbake zephyr-helloworld
Loading cache: 100%

|###############################################################
################################################################
################################################################
##########| Time: 0:00:01 Loaded 4949 entries from dependency cache.

Quoted 4 lines

ERROR: ExpansionError during parsing /home/jamin_lin/oe-review/meta-zephyr/meta-zephyr-core/recipes-kernel/ zephyr-kernel/zephyr-helloworld.bb bb.data_smart.ExpansionError: Failure expanding variable

do_compile[file-checksums], expression was ${@srctree_hash_files(d)} which triggered exception CalledProcessError: Command '['git', 'add', '-A', '.']' returned non-zero exit status 128.

Quoted 5 lines

The variable dependency chain for the failure is:
do_compile[file-checksums]

ERROR: Parsing halted due to errors, see error messages above
Parsing recipes: 0% |

| ETA: --:--:--

Quoted 2 lines

Summary: There were 2 ERROR messages, returning a non-zero exit code. ```

Following these steps I see the same error. But the cause is not the nested git repositories. bitbake -c cleanall ... has deleted the sources from the downloads directory, but the devtool workspace is still referencing the deleted sources. Even a git status command in the zephyr-helloworld sources directory fails:

  $ git -C workspace/sources/zephyr-helloworld status
  error: unable to normalize alternate object path:
/home/pbarker/bitbake-builds/.bitbake-setup-downloads/git2/github.com.zeph
yrproject-rtos.mcuboot//objects
  error: unable to normalize alternate object path:
/home/pbarker/bitbake-builds/.bitbake-setup-downloads/git2/github.com.zeph
yrproject-rtos.mcuboot//objects
  error: unable to normalize alternate object path:
/home/pbarker/bitbake-builds/.bitbake-setup-downloads/git2/github.com.zeph
yrproject-rtos.mcuboot//objects
  error: unable to normalize alternate object path:
/home/pbarker/bitbake-builds/.bitbake-setup-downloads/git2/github.com.zeph
yrproject-rtos.mcuboot//objects
  fatal: bad object HEAD
  fatal: 'git status --porcelain=2' failed in submodule bootloader/mcuboot

If I replace step 8 with bitbake -c cleansstate ... then there is no parse error.

What we need to fix is the interaction between devtool modify and bitbake -c cleanall. Perhaps devtool needs to detect this broken state and tell the user how to fix it - simply running bitbake -c fetch zephyr-helloworld isn't possible as the metadata can't be parsed.

Quoted 4 lines

Best regards,

--
Paul Barker

Hi Paul,

Thanks for identifying the root cause.

After studying the devtool design, I now understand what happens: when devtool modify creates the workspace, it does not copy the full git object store locally.
Instead, it clones the source tree using git's alternate object store mechanism - the workspace's .git/objects/info/alternates file points back to the downloaded bare repositories under the downloads/ directory. This way the workspace shares the git objects with the download cache rather than duplicating them on disk.

When bitbake -c cleanall is run, it deletes the downloaded bare repositories from downloads/. The workspace source tree still exists, but its alternates now point to missing paths. Any git operation on the workspace (including git status, git add, etc.) then fails because git cannot resolve the object references, producing errors like:

error: unable to normalize alternate object path: .../downloads/git2/.../objects
fatal: bad object HEAD

This is what causes the parser to halt with exit 128.

The fix on my side is straightforward: replace bitbake -c cleanall with bitbake -c cleansstate in my devtool workflow. cleansstate clears the sstate cache without touching the downloads directory, so the workspace alternates remain intact.

That said, I am not sure whether some improvement on the OE side would still be worthwhile. Currently, when this breakage occurs, the user sees only a cryptic CalledProcessError: exit 128 with no indication of what went wrong or how to recover.
The situation is also not obvious: running bitbake -c cleanall on a single-repo recipe with a devtool workspace does not trigger this problem, so users working with multi-repo recipes (such as Zephyr-based recipes that fetch multiple git repositories into separate destsuffix directories) are likely to hit this unexpectedly. A clearer error message pointing to the root cause and the recovery steps (devtool reset -n followed by devtool modify) might save other users from the same confusion.

Please drop this patch series because it is not a normal test case for devtool.

Below patch that may help improve the user experience, though I leave it to the community to decide whether they are worth reviewing:

Best regards,
Jamin

meta/classes/externalsrc.bbclass +51 -0
index 902ff2604f..3e6c9d937e 100644
@@ -227,6 +227,57 @@ def srctree_hash_files(d, srcdir=None):
 
     ret = " "
     if git_dir is not None:
+        # Find nested git repos created by multiple SRC_URI git entries with
+        # different destsuffix values, so their alternates can also be checked
+        # for breakage below.
+        nested_git_dirs = []
+        for root, dirs, files in os.walk(s_dir):
+            if root == s_dir:
+                continue
+            if '.git' in dirs or '.git' in files:
+                nested_git_dirs.append(root)
+                dirs[:] = []
+
+        # Check for broken git alternates in the top-level repo and all nested
+        # repos. This can happen when 'bitbake -c cleanall' deletes the
+        # downloads directory that the devtool workspace references via
+        # alternates, causing all subsequent git operations to exit 128 and
+        # halt the parse phase.  Relative alternate paths are resolved against
+        # the repo's objects directory; double-slash paths (a known devtool
+        # artefact) are normalised before the existence check.
+        def has_broken_alternates(dot_git):
+            obj_dir = os.path.join(dot_git, 'objects')
+            alt_file = os.path.join(obj_dir, 'info', 'alternates')
+            if not os.path.exists(alt_file):
+                return False
+            with open(alt_file) as f:
+                for line in f:
+                    path = line.strip()
+                    if not path or path.startswith('#'):
+                        continue
+                    if not os.path.isabs(path):
+                        path = os.path.join(obj_dir, path)
+                    if not os.path.exists(os.path.normpath(path)):
+                        return True
+            return False
+
+        broken = has_broken_alternates(git_dir)
+        if not broken:
+            for nested in nested_git_dirs:
+                nested_dot_git = os.path.join(nested, '.git')
+                if os.path.isdir(nested_dot_git) and has_broken_alternates(nested_dot_git):
+                    broken = True
+                    break
+
+        if broken:
+            bb.warn('%s: devtool workspace has broken git alternates, '
+                    'likely caused by "bitbake -c cleanall" removing the '
+                    'downloads directory. '
+                    'To repair the workspace, run: '
+                    'devtool reset %s && devtool modify %s'
+                    % (d.getVar('PN'), d.getVar('PN'), d.getVar('PN')))
+            return s_dir + '/*:True'
+
 
 
 
author
#6 lore raw

RE: [OE-core] [PATCH v1 1/2] externalsrc: Handle nested git repos from multiple SRC_URI entries

Hi Paul

Quoted 103 lines

Hi Paul,

Thanks for identifying the root cause.

After studying the devtool design, I now understand what happens: when devtool modify creates the workspace, it does not copy the full git object store locally.
Instead, it clones the source tree using git's alternate object store mechanism - the workspace's .git/objects/info/alternates file points back to the downloaded bare repositories under the downloads/ directory.
This way the workspace shares the git objects with the download cache rather than duplicating them on disk.

When bitbake -c cleanall is run, it deletes the downloaded bare repositories from downloads/. The workspace source tree still exists, but its alternates now point to missing paths.
Any git operation on the workspace (including git status, git add, etc.) then fails because git cannot resolve the object references, producing errors like:

error: unable to normalize alternate object path: .../downloads/git2/.../objects
fatal: bad object HEAD

This is what causes the parser to halt with exit 128.

The fix on my side is straightforward: replace bitbake -c cleanall with bitbake -c cleansstate in my devtool workflow. cleansstate clears the sstate cache without touching the downloads directory, so the workspace alternates remain intact.

That said, I am not sure whether some improvement on the OE side would still be worthwhile. Currently, when this breakage occurs, the user sees only a cryptic CalledProcessError: exit 128 with no indication of what went wrong or how to recover.
The situation is also not obvious: running bitbake -c cleanall on a single-repo recipe with a devtool workspace does not trigger this problem, so users working with multi-repo recipes (such as Zephyr-based recipes that fetch multiple git repositories into separate destsuffix directories) are likely to hit this unexpectedly. A clearer error message pointing to the root cause and the recovery steps (devtool reset -n followed by devtool modify) might save other users from the same confusion.

Please drop this patch series because it is not a normal test case for devtool.

Below patch that may help improve the user experience, though I leave it to the community to decide whether they are worth reviewing:

Best regards,
Jamin

meta/classes/externalsrc.bbclass +12 -0
index 902ff2604f..3e6c9d937e 100644
@@ -227,6 +227,57 @@ def srctree_hash_files(d, srcdir=None):
 
     ret = " "
     if git_dir is not None:
+        # Find nested git repos created by multiple SRC_URI git entries with
+        # different destsuffix values, so their alternates can also be checked
+        # for breakage below.
+        nested_git_dirs = []
+        for root, dirs, files in os.walk(s_dir):
+            if root == s_dir:
+                continue
+            if '.git' in dirs or '.git' in files:
+                nested_git_dirs.append(root)
+                dirs[:] = []
+
+        # Check for broken git alternates in the top-level repo and all

nested

meta/classes/externalsrc.bbclass +4 -0
+        # repos. This can happen when 'bitbake -c cleanall' deletes the
+        # downloads directory that the devtool workspace references via
+        # alternates, causing all subsequent git operations to exit 128 and
+        # halt the parse phase.  Relative alternate paths are resolved

against

meta/classes/externalsrc.bbclass +23 -0
+        # the repo's objects directory; double-slash paths (a known devtool
+        # artefact) are normalised before the existence check.
+        def has_broken_alternates(dot_git):
+            obj_dir = os.path.join(dot_git, 'objects')
+            alt_file = os.path.join(obj_dir, 'info', 'alternates')
+            if not os.path.exists(alt_file):
+                return False
+            with open(alt_file) as f:
+                for line in f:
+                    path = line.strip()
+                    if not path or path.startswith('#'):
+                        continue
+                    if not os.path.isabs(path):
+                        path = os.path.join(obj_dir, path)
+                    if not os.path.exists(os.path.normpath(path)):
+                        return True
+            return False
+
+        broken = has_broken_alternates(git_dir)
+        if not broken:
+            for nested in nested_git_dirs:
+                nested_dot_git = os.path.join(nested, '.git')
+                if os.path.isdir(nested_dot_git) and

has_broken_alternates(nested_dot_git):

meta/classes/externalsrc.bbclass +2 -0
+                    broken = True
+                    break

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 280

Quoted 10 lines
meta/classes/externalsrc.bbclass +10 -0
+
+        if broken:
+            bb.warn('%s: devtool workspace has broken git alternates, '
+                    'likely caused by "bitbake -c cleanall" removing the '
+                    'downloads directory. '
+                    'To repair the workspace, run: '
+                    'devtool reset %s && devtool modify %s'
+                    % (d.getVar('PN'), d.getVar('PN'), d.getVar('PN')))
+            return s_dir + '/*:True'
+

Please ignore the comments above.
I believe I have identified the root cause. Please see my updated solution here:
https://patchwork.yoctoproject.org/project/oe-core/patch/[email protected]/

Thanks,
Jamin

Quoted 3 lines
author
#7 lore raw

RE: [PATCH v1 1/2] externalsrc: Handle nested git repos from multiple SRC_URI entries

In reply to Paul Barker

Hi Paul,

Quoted 53 lines

Subject: Re: [PATCH v1 1/2] externalsrc: Handle nested git repos from multiple SRC_URI entries

Quoted 8 lines · Jamin Lin

When a recipe uses multiple git SRC_URI entries with different destsuffix values (e.g. Zephyr-based recipes with separate repos for the kernel, modules, and application), each source is unpacked into a subdirectory of EXTERNALSRC that retains its own .git directory.

srctree_hash_files() calls 'git add -A .' at the EXTERNALSRC root, which fails with exit code 128 when git encounters these unregistered nested git repositories, halting the bitbake parse phase.

Is this true? The documentation for git add [1] talks about issuing a warning when this occurs, not an error, and in some quick local testing I get a successful exit (exit code 0) when I try this.

Quoted 18 lines

Fix by scanning for nested git repos before the add. If any are found, exclude them from the top-level 'git add' using pathspec magic ':(exclude)<path>' and hash each nested repo independently using a temporary index. This ensures changes in any nested repo still trigger do_compile/do_configure to re-run.

---
 meta/classes/externalsrc.bbclass | 37
+++++++++++++++++++++++++++++++-
 1 file changed, 36 insertions(+), 1 deletion(-)
meta/classes/externalsrc.bbclass +0 -0
index 902ff2604f..0dd57af668 100644

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 237

Quoted 4 lines
meta/classes/externalsrc.bbclass +0 -1
@@ -234,8 +234,43 @@ def srctree_hash_files(d, srcdir=None):
             # Update our custom index
             env = os.environ.copy()
             env['GIT_INDEX_FILE'] = tmp_index.name
-            subprocess.check_output(['git', 'add', '-A', '.'], cwd=s_dir,

env=env)

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 237

Quoted 1 line
meta/classes/externalsrc.bbclass +1 -0
+            # Find nested git repos created by multiple SRC_URI git

entries with

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 238

Quoted 1 line
meta/classes/externalsrc.bbclass +1 -0
+            # different destsuffix values. git add -A . exits 128 when it

encounters

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 246

Quoted 8 lines
meta/classes/externalsrc.bbclass +8 -0
+            # these unregistered nested repos.
+            nested_git_dirs = []
+            for root, dirs, files in os.walk(s_dir):
+                if root == s_dir:
+                    continue
+                if '.git' in dirs or '.git' in files:
+                    nested_git_dirs.append(root)
+                    dirs[:] = []  # don't recurse into nested repos
Quoted 2 lines

This os.walk() loop is expensive, is there an alternative way to handle this?

os.scandir() was considered but rejected: destsuffix allows arbitrarily
deep paths (e.g. destsuffix=${P}/modules/tee/tf-m/trusted-firmware-m), so a depth-1 scan would silently miss nested repos and stop hashing their content — source
changes there would no longer trigger do_compile to re-run.

The os.walk() loop uses dirs[:] = [] to stop recursing as soon as a .git entry is found, so we never descend into the nested repos themselves (which may contain tens of thousands of files). The walk only traverses the shallow skeleton of intermediate directories between EXTERNALSRC and each nested repo.

Please see the use case here:
https://git.yoctoproject.org/meta-zephyr/tree/meta-zephyr-core/recipes-kernel/zephyr-kernel/zephyr-kernel-src-4.3.0.inc

${SRC_URI_ZEPHYR_OPEN_AMP};name=open-amp;nobranch=1;destsuffix=${P}/modules/lib/open-amp \
${SRC_URI_ZEPHYR_TRUSTED_FIRMWARE_M};name=trusted-firmware-m;nobranch=1;destsuffix=${P}/modules/tee/tf-m/trusted-firmware-m \

Thanks-Jamin

Quoted 58 lines

The code has also become difficult to parse. My rule of thumb is that if a group of lines needs a leading comment, it also needs an empty line before the comment to visually separate things.

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 248

Quoted 2 lines
meta/classes/externalsrc.bbclass +2 -0
+            if nested_git_dirs:
+                excludes = [':(exclude)' + os.path.relpath(n, s_dir) for n

in nested_git_dirs]

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 249

Quoted 1 line
meta/classes/externalsrc.bbclass +1 -0
+                subprocess.check_output(['git', 'add', '-A', '.'] + excludes,

cwd=s_dir, env=env)

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 252

Quoted 3 lines
meta/classes/externalsrc.bbclass +3 -0
+            else:
+                subprocess.check_output(['git', 'add', '-A', '.'],
+ cwd=s_dir, env=env)

To simplify the code, construct a cmd variable and call subprocess.check_output(cmd, ...) once.

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 255

Quoted 3 lines
meta/classes/externalsrc.bbclass +1 -0
             git_sha1 = subprocess.check_output(['git', 'write-tree'],
 cwd=s_dir, env=env).decode("utf-8")
+            # Hash each nested git repo separately so source changes

there still

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None):

Quoted 6 lines
meta/classes/externalsrc.bbclass +6 -0
+            # trigger do_compile/do_configure to re-run.
+            for nested in nested_git_dirs:
+                nested_git = os.path.join(nested, '.git')
+                if not os.path.isdir(nested_git):
+                    continue
+                with

tempfile.NamedTemporaryFile(prefix='oe-devtool-nested-index') as nested_tmp:

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None):

Quoted 3 lines
meta/classes/externalsrc.bbclass +3 -0
+                    nested_index = os.path.join(nested_git, 'index')
+                    if os.path.exists(nested_index):
+                        shutil.copyfile(nested_index,

nested_tmp.name)

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None):

Quoted 3 lines
meta/classes/externalsrc.bbclass +3 -0
+                    nested_env = os.environ.copy()
+                    nested_env['GIT_INDEX_FILE'] = nested_tmp.name
+                    proc = subprocess.Popen(['git', 'add', '-A', '.'],

cwd=nested,

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 268

Quoted 1 line
meta/classes/externalsrc.bbclass +1 -0
+                                           env=nested_env,

stdout=subprocess.DEVNULL,

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 269

Quoted 1 line
meta/classes/externalsrc.bbclass +1 -0
+

stderr=subprocess.DEVNULL)

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 271

Quoted 2 lines
meta/classes/externalsrc.bbclass +2 -0
+                    proc.communicate()
+                    proc = subprocess.Popen(['git', 'write-tree'],

cwd=nested,

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 272

Quoted 1 line
meta/classes/externalsrc.bbclass +1 -0
+                                           env=nested_env,

stdout=subprocess.PIPE,

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 273

Quoted 1 line
meta/classes/externalsrc.bbclass +1 -0
+

stderr=subprocess.DEVNULL)

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 273

Quoted 2 lines
meta/classes/externalsrc.bbclass +2 -0
+                    stdout, _ = proc.communicate()
+                    git_sha1 += stdout.decode("utf-8")

We should re-use the code from the following block which handles submodules instead of re-implementing the behaviour. Perhaps the common code needs to be refactored out.

Best regards,

--
Paul Barker

author
#8 lore raw

RE: [PATCH v1 1/2] externalsrc: Handle nested git repos from multiple SRC_URI entries

Hi Paul,

Quoted 113 lines

Subject: RE: [PATCH v1 1/2] externalsrc: Handle nested git repos from multiple SRC_URI entries

Hi Paul,

Quoted 54 lines

Subject: Re: [PATCH v1 1/2] externalsrc: Handle nested git repos from multiple SRC_URI entries

Quoted 8 lines · Jamin Lin

When a recipe uses multiple git SRC_URI entries with different destsuffix values (e.g. Zephyr-based recipes with separate repos for the kernel, modules, and application), each source is unpacked into a subdirectory of EXTERNALSRC that retains its own .git directory.

srctree_hash_files() calls 'git add -A .' at the EXTERNALSRC root, which fails with exit code 128 when git encounters these unregistered nested git repositories, halting the bitbake parse phase.

Is this true? The documentation for git add [1] talks about issuing a warning when this occurs, not an error, and in some quick local testing I get a successful exit (exit code 0) when I try this.

Quoted 18 lines

Fix by scanning for nested git repos before the add. If any are found, exclude them from the top-level 'git add' using pathspec magic ':(exclude)<path>' and hash each nested repo independently using a temporary index. This ensures changes in any nested repo still trigger do_compile/do_configure to re-run.

---
 meta/classes/externalsrc.bbclass | 37
+++++++++++++++++++++++++++++++-
 1 file changed, 36 insertions(+), 1 deletion(-)
meta/classes/externalsrc.bbclass +0 -0
index 902ff2604f..0dd57af668 100644

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 237

Quoted 4 lines
meta/classes/externalsrc.bbclass +0 -1
@@ -234,8 +234,43 @@ def srctree_hash_files(d, srcdir=None):
             # Update our custom index
             env = os.environ.copy()
             env['GIT_INDEX_FILE'] = tmp_index.name
-            subprocess.check_output(['git', 'add', '-A', '.'], cwd=s_dir,

env=env)

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 237

Quoted 1 line
meta/classes/externalsrc.bbclass +1 -0
+            # Find nested git repos created by multiple SRC_URI git

entries with

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 239

Quoted 2 lines
meta/classes/externalsrc.bbclass +2 -0
+            # different destsuffix values. git add -A . exits 128
+ when it

encounters

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 246

Quoted 8 lines
meta/classes/externalsrc.bbclass +8 -0
+            # these unregistered nested repos.
+            nested_git_dirs = []
+            for root, dirs, files in os.walk(s_dir):
+                if root == s_dir:
+                    continue
+                if '.git' in dirs or '.git' in files:
+                    nested_git_dirs.append(root)
+                    dirs[:] = []  # don't recurse into nested repos
Quoted 2 lines

This os.walk() loop is expensive, is there an alternative way to handle this?

os.scandir() was considered but rejected: destsuffix allows arbitrarily deep paths (e.g. destsuffix=${P}/modules/tee/tf-m/trusted-firmware-m), so a depth-1 scan would silently miss nested repos and stop hashing their content — source changes there would no longer trigger do_compile to re-run.

The os.walk() loop uses dirs[:] = [] to stop recursing as soon as a .git entry is found, so we never descend into the nested repos themselves (which may contain tens of thousands of files). The walk only traverses the shallow skeleton of intermediate directories between EXTERNALSRC and each nested repo.

Please see the use case here:
https://git.yoctoproject.org/meta-zephyr/tree/meta-zephyr-core/recipes-kernel
/zephyr-kernel/zephyr-kernel-src-4.3.0.inc

${SRC_URI_ZEPHYR_OPEN_AMP};name=open-amp;nobranch=1;destsuffix=${P} /modules/lib/open-amp \
${SRC_URI_ZEPHYR_TRUSTED_FIRMWARE_M};name=trusted-firmware-m;nob ranch=1;destsuffix=${P}/modules/tee/tf-m/trusted-firmware-m \

Thanks-Jamin

Quoted 31 lines

The code has also become difficult to parse. My rule of thumb is that if a group of lines needs a leading comment, it also needs an empty line before the comment to visually separate things.

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 250

Quoted 3 lines
meta/classes/externalsrc.bbclass +3 -0
+            if nested_git_dirs:
+                excludes = [':(exclude)' + os.path.relpath(n,
+ s_dir) for n

in nested_git_dirs]

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 252

Quoted 2 lines
meta/classes/externalsrc.bbclass +2 -0
+                subprocess.check_output(['git', 'add', '-A', '.'] +
+ excludes,

cwd=s_dir, env=env)

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 255

Quoted 3 lines
meta/classes/externalsrc.bbclass +3 -0
+            else:
+                subprocess.check_output(['git', 'add', '-A', '.'],
+ cwd=s_dir, env=env)

To simplify the code, construct a cmd variable and call subprocess.check_output(cmd, ...) once.

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 259

Quoted 4 lines
meta/classes/externalsrc.bbclass +2 -0
             git_sha1 = subprocess.check_output(['git',
 'write-tree'], cwd=s_dir, env=env).decode("utf-8")
+            # Hash each nested git repo separately so source
+ changes

there still

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None):

Quoted 6 lines
meta/classes/externalsrc.bbclass +6 -0
+            # trigger do_compile/do_configure to re-run.
+            for nested in nested_git_dirs:
+                nested_git = os.path.join(nested, '.git')
+                if not os.path.isdir(nested_git):
+                    continue
+                with

tempfile.NamedTemporaryFile(prefix='oe-devtool-nested-index') as nested_tmp:

Quoted 32 lines
Quoted 6 lines

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None):

Quoted 3 lines
meta/classes/externalsrc.bbclass +3 -0
+                    nested_index = os.path.join(nested_git, 'index')
+                    if os.path.exists(nested_index):
+                        shutil.copyfile(nested_index,

nested_tmp.name)

Quoted 2 lines
meta/classes/externalsrc.bbclass +2 -0
+                    nested_env = os.environ.copy()
+                    nested_env['GIT_INDEX_FILE'] =

nested_tmp.name

Quoted 25 lines

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 272

Quoted 2 lines
meta/classes/externalsrc.bbclass +2 -0
+                    proc = subprocess.Popen(['git', 'add', '-A',
+ '.'],

cwd=nested,

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 273

Quoted 1 line
meta/classes/externalsrc.bbclass +1 -0
+                                           env=nested_env,

stdout=subprocess.DEVNULL,

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 274

Quoted 1 line
meta/classes/externalsrc.bbclass +1 -0
+

stderr=subprocess.DEVNULL)

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 276

Quoted 2 lines
meta/classes/externalsrc.bbclass +2 -0
+                    proc.communicate()
+                    proc = subprocess.Popen(['git', 'write-tree'],

cwd=nested,

Quoted 1 line

+ env=nested_env,

stdout=subprocess.PIPE,

Quoted 1 line

+

stderr=subprocess.DEVNULL)

meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 273

Quoted 2 lines
meta/classes/externalsrc.bbclass +2 -0
+                    stdout, _ = proc.communicate()
+                    git_sha1 += stdout.decode("utf-8")

We should re-use the code from the following block which handles submodules instead of re-implementing the behaviour. Perhaps the common code needs to be refactored out.

Best regards,

--
Paul Barker

Thanks for the review and suggestion.

Will send v2 as below.

```
diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
index 902ff2604f..f2e0812eea 100644
--- a/meta/classes/externalsrc.bbclass
+++ b/meta/classes/externalsrc.bbclass
@@ -234,18 +234,47 @@ def srctree_hash_files(d, srcdir=None):
             # Update our custom index
             env = os.environ.copy()
             env['GIT_INDEX_FILE'] = tmp_index.name
-            subprocess.check_output(['git', 'add', '-A', '.'], cwd=s_dir, env=env)
+
+            # Find nested git repos created by multiple SRC_URI git entries with
+            # different destsuffix values. When GIT_INDEX_FILE is set, git add -A .
+            # exits 128 instead of warning when it encounters these unregistered
+            # nested repos, halting the bitbake parse phase.
+            nested_git_dirs = []
+            for root, dirs, files in os.walk(s_dir):
+                if root == s_dir:
+                    continue
+                if '.git' in dirs or '.git' in files:
+                    nested_git_dirs.append(root)
+                    dirs[:] = []
+
+            cmd = ['git', 'add', '-A', '.']
+            if nested_git_dirs:
+                cmd += [':(exclude)' + os.path.relpath(n, s_dir) for n in nested_git_dirs]
+            subprocess.check_output(cmd, cwd=s_dir, env=env)
+
             git_sha1 = subprocess.check_output(['git', 'write-tree'], cwd=s_dir, env=env).decode("utf-8")
-            if os.path.exists(os.path.join(s_dir, ".gitmodules")) and os.path.getsize(os.path.join(s_dir, ".gitmodules")) > 0:
+
+            # Hash nested git repos and submodules together so changes in any of
+            # them still trigger do_compile/do_configure to re-run.
+            subdirs_to_hash = list(nested_git_dirs)
+            if os.path.exists(os.path.join(s_dir, ".gitmodules")) and \
+                    os.path.getsize(os.path.join(s_dir, ".gitmodules")) > 0:
                 submodule_helper = subprocess.check_output(["git", "config", "--file", ".gitmodules", "--get-regexp", "path"], cwd=s_dir, env=env).decode("utf-8")
                 for line in submodule_helper.splitlines():
                     module_dir = os.path.join(s_dir, line.rsplit(maxsplit=1)[1])
                     if os.path.isdir(module_dir):
-                        proc = subprocess.Popen(['git', 'add', '-A', '.'], cwd=module_dir, env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
-                        proc.communicate()
-                        proc = subprocess.Popen(['git', 'write-tree'], cwd=module_dir, env=env, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
-                        stdout, _ = proc.communicate()
-                        git_sha1 += stdout.decode("utf-8")
+                        subdirs_to_hash.append(module_dir)
+
+            for subdir in subdirs_to_hash:
+                proc = subprocess.Popen(['git', 'add', '-A', '.'], cwd=subdir,
+                                        env=env, stdout=subprocess.DEVNULL,
+                                        stderr=subprocess.DEVNULL)
+                proc.communicate()
+                proc = subprocess.Popen(['git', 'write-tree'], cwd=subdir,
+                                        env=env, stdout=subprocess.PIPE,
+                                        stderr=subprocess.DEVNULL)
+                stdout, _ = proc.communicate()
+                git_sha1 += stdout.decode("utf-8")
             sha1 = hashlib.sha1(git_sha1.encode("utf-8")).hexdigest()
         with open(oe_hash_file, 'w') as fobj:
             fobj.write(sha1)
```

Thanks,
Jamin

author
#9 lore raw

PATCH 2/2 reproducible: Handle nested git repos in find_git_repositories

In reply to Jamin Lin

When EXTERNALSRC contains multiple nested git repositories (from multiple SRC_URI git entries with different destsuffix values), find_git_repositories() walks into sub-repos and get_source_date_epoch_from_git() subsequently fails with exit code 128 when running 'git log -1' inside them.

Two fixes:
- Stop os.walk recursion when a .git entry is found (dirs[:] = []) to avoid descending into nested repos.
- Change 'git log -1' from check=True to check=False with explicit error handling, so a failing nested repo is skipped gracefully instead of raising CalledProcessError and aborting do_unpack.

---
 meta/lib/oe/reproducible.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
meta/lib/oe/reproducible.py +5 -1
index a80376010a..6bb25da55a 100644
@@ -82,6 +82,7 @@ def find_git_repositories(d, sourcedir):
         for root, dirs, files in os.walk(mainpath, topdown=True):
             if '.git' in dirs or '.git' in files:
                 git_repositories.append(root)
+                dirs[:] = []  # don't recurse into nested git repos (multiple SRC_URI destsuffix)
 
     if not git_repositories:
         bb.warn('Failed to find any git repositories in UNPACKDIR or S')
@@ -105,7 +106,10 @@ def get_source_date_epoch_from_git(d, sourcedir):
 
         bb.debug(1, "git repository: %s" % repo_path)
         p = subprocess.run(['git', '-C', repo_path, 'log', '-1', '--pretty=%ct'],
-                           check=True, stdout=subprocess.PIPE)
+                           check=False, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+        if p.returncode != 0:
+            bb.debug(1, "git log failed for %s (exit %d): %s" % (repo_path, p.returncode, p.stdout.decode('utf-8')))
+            continue
         source_dates.append(int(p.stdout.decode('utf-8')))
 
     if source_dates:
Signature · 1 line
2.43.0
#10 lore raw

Re: [PATCH v1 2/2] reproducible: Handle nested git repos in find_git_repositories

Quoted 31 lines · Jamin Lin

When EXTERNALSRC contains multiple nested git repositories (from multiple SRC_URI git entries with different destsuffix values), find_git_repositories() walks into sub-repos and get_source_date_epoch_from_git() subsequently fails with exit code 128 when running 'git log -1' inside them.

Two fixes:
- Stop os.walk recursion when a .git entry is found (dirs[:] = []) to avoid descending into nested repos.
- Change 'git log -1' from check=True to check=False with explicit error handling, so a failing nested repo is skipped gracefully instead of raising CalledProcessError and aborting do_unpack.

---
 meta/lib/oe/reproducible.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
meta/lib/oe/reproducible.py +1 -0
index a80376010a..6bb25da55a 100644
@@ -82,6 +82,7 @@ def find_git_repositories(d, sourcedir):
         for root, dirs, files in os.walk(mainpath, topdown=True):
             if '.git' in dirs or '.git' in files:
                 git_repositories.append(root)
+                dirs[:] = []  # don't recurse into nested git repos (multiple SRC_URI destsuffix)
 
     if not git_repositories:
         bb.warn('Failed to find any git repositories in UNPACKDIR or S')

meta/lib/oe/reproducible.py def get_source_date_epoch_from_git(d, sourcedir): · line 113

Quoted 9 lines
meta/lib/oe/reproducible.py +4 -1
@@ -105,7 +106,10 @@ def get_source_date_epoch_from_git(d, sourcedir):
 
         bb.debug(1, "git repository: %s" % repo_path)
         p = subprocess.run(['git', '-C', repo_path, 'log', '-1', '--pretty=%ct'],
-                           check=True, stdout=subprocess.PIPE)
+                           check=False, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+        if p.returncode != 0:
+            bb.debug(1, "git log failed for %s (exit %d): %s" % (repo_path, p.returncode, p.stdout.decode('utf-8')))
+            continue
         source_dates.append(int(p.stdout.decode('utf-8')))

This causes the repository to be silently (in most cases, since bb.debug() output is not usually printed) dropped from the source_dates array. Does this invalidate the returned source date epoch?

We may want a warning here instead of a debug print.

Also note that there may be other reasons for the git log command to fail, and the output that identifies why it failed is likely to be in stderr rather than stdout so we don't want to throw that way.

Best regards,

Signature · 1 line
Paul Barker