oe: Fix build failures with multiple git SRC_URI entries
cover oe: Fix build failures with multiple git SRC_URI entries
Some recipes (e.g. Zephyr-based) use multiple git SRC_URI entries with different destsuffix values, causing each source to be unpacked into a separate subdirectory of EXTERNALSRC that retains its own .git directory.
These nested git repositories trigger two independent failures:
1. externalsrc.bbclass: 'git add -A .' exits with code 128 during srctree_hash_files(), halting the bitbake parse phase.
2. oe/reproducible.py: 'git log -1' exits with code 128 inside a nested repo found by find_git_repositories(), aborting do_unpack.
Jamin Lin (2):
externalsrc: Handle nested git repos from multiple SRC_URI entries
reproducible: Handle nested git repos in find_git_repositories
meta/classes/externalsrc.bbclass | 37 +++++++++++++++++++++++++++++++- meta/lib/oe/reproducible.py | 6 +++++- 2 files changed, 41 insertions(+), 2 deletions(-)
Signature · 1 line
2.43.0
PATCH 1/2 externalsrc: Handle nested git repos from multiple SRC_URI entries
When a recipe uses multiple git SRC_URI entries with different destsuffix values (e.g. Zephyr-based recipes with separate repos for the kernel, modules, and application), each source is unpacked into a subdirectory of EXTERNALSRC that retains its own .git directory.
srctree_hash_files() calls 'git add -A .' at the EXTERNALSRC root, which fails with exit code 128 when git encounters these unregistered nested git repositories, halting the bitbake parse phase.
Fix by scanning for nested git repos before the add. If any are found, exclude them from the top-level 'git add' using pathspec magic ':(exclude)<path>' and hash each nested repo independently using a temporary index. This ensures changes in any nested repo still trigger do_compile/do_configure to re-run.
--- meta/classes/externalsrc.bbclass | 37 +++++++++++++++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-)
meta/classes/externalsrc.bbclass +36 -1
@@ -234,8 +234,43 @@ def srctree_hash_files(d, srcdir=None): # Update our custom index env = os.environ.copy() env['GIT_INDEX_FILE'] = tmp_index.name - subprocess.check_output(['git', 'add', '-A', '.'], cwd=s_dir, env=env) + # Find nested git repos created by multiple SRC_URI git entries with + # different destsuffix values. git add -A . exits 128 when it encounters + # these unregistered nested repos. + nested_git_dirs = [] + for root, dirs, files in os.walk(s_dir): + if root == s_dir: + continue + if '.git' in dirs or '.git' in files: + nested_git_dirs.append(root) + dirs[:] = [] # don't recurse into nested repos + if nested_git_dirs: + excludes = [':(exclude)' + os.path.relpath(n, s_dir) for n in nested_git_dirs] + subprocess.check_output(['git', 'add', '-A', '.'] + excludes, cwd=s_dir, env=env) + else: + subprocess.check_output(['git', 'add', '-A', '.'], cwd=s_dir, env=env) git_sha1 = subprocess.check_output(['git', 'write-tree'], cwd=s_dir, env=env).decode("utf-8") + # Hash each nested git repo separately so source changes there still + # trigger do_compile/do_configure to re-run. + for nested in nested_git_dirs: + nested_git = os.path.join(nested, '.git') + if not os.path.isdir(nested_git): + continue + with tempfile.NamedTemporaryFile(prefix='oe-devtool-nested-index') as nested_tmp: + nested_index = os.path.join(nested_git, 'index') + if os.path.exists(nested_index): + shutil.copyfile(nested_index, nested_tmp.name) + nested_env = os.environ.copy() + nested_env['GIT_INDEX_FILE'] = nested_tmp.name + proc = subprocess.Popen(['git', 'add', '-A', '.'], cwd=nested, + env=nested_env, stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL) + proc.communicate() + proc = subprocess.Popen(['git', 'write-tree'], cwd=nested, + env=nested_env, stdout=subprocess.PIPE, + stderr=subprocess.DEVNULL) + stdout, _ = proc.communicate() + git_sha1 += stdout.decode("utf-8") if os.path.exists(os.path.join(s_dir, ".gitmodules")) and os.path.getsize(os.path.join(s_dir, ".gitmodules")) > 0: submodule_helper = subprocess.check_output(["git", "config", "--file", ".gitmodules", "--get-regexp", "path"], cwd=s_dir, env=env).decode("utf-8") for line in submodule_helper.splitlines():
Signature · 1 line
2.43.0
Re: [PATCH v1 1/2] externalsrc: Handle nested git repos from multiple SRC_URI entries
Quoted 8 lines · Jamin Lin
When a recipe uses multiple git
SRC_URIentries with different destsuffix values (e.g. Zephyr-based recipes with separate repos for the kernel, modules, and application), each source is unpacked into a subdirectory of EXTERNALSRC that retains its own .git directory.
srctree_hash_files()calls 'git add -A .' at the EXTERNALSRC root, which fails with exit code 128 when git encounters these unregistered nested git repositories, halting the bitbake parse phase.
Is this true? The documentation for git add [1] talks about issuing a warning when this occurs, not an error, and in some quick local testing I get a successful exit (exit code 0) when I try this.
Quoted 20 lines
Fix by scanning for nested git repos before the add. If any are found, exclude them from the top-level 'git add' using pathspec magic ':(exclude)<path>' and hash each nested repo independently using a temporary index. This ensures changes in any nested repo still trigger do_compile/do_configure to re-run.
Signed-off-by Jamin Lin <[email protected]>--- meta/classes/externalsrc.bbclass | 37 +++++++++++++++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-)meta/classes/externalsrc.bbclass +0
-1@@ -234,8 +234,43 @@ def srctree_hash_files(d, srcdir=None): # Update our custom index env = os.environ.copy() env['GIT_INDEX_FILE'] = tmp_index.name - subprocess.check_output(['git', 'add', '-A', '.'], cwd=s_dir, env=env)
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 246
Quoted 10 lines
meta/classes/externalsrc.bbclass +10
-0+ # Find nested git repos created by multiple SRC_URI git entries with + # different destsuffix values. git add -A . exits 128 when it encounters + # these unregistered nested repos. + nested_git_dirs = [] + for root, dirs, files in os.walk(s_dir): + if root == s_dir: + continue + if '.git' in dirs or '.git' in files: + nested_git_dirs.append(root) + dirs[:] = [] # don't recurse into nested repos
This os.walk() loop is expensive, is there an alternative way to handle this?
The code has also become difficult to parse. My rule of thumb is that if a group of lines needs a leading comment, it also needs an empty line before the comment to visually separate things.
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 251
Quoted 5 lines
meta/classes/externalsrc.bbclass +5
-0+ if nested_git_dirs: + excludes = [':(exclude)' + os.path.relpath(n, s_dir) for n in nested_git_dirs] + subprocess.check_output(['git', 'add', '-A', '.'] + excludes, cwd=s_dir, env=env) + else: + subprocess.check_output(['git', 'add', '-A', '.'], cwd=s_dir, env=env)
To simplify the code, construct a cmd variable and call subprocess.check_output(cmd, ...) once.
Quoted 12 lines
meta/classes/externalsrc.bbclass +11
-0git_sha1 = subprocess.check_output(['git', 'write-tree'], cwd=s_dir, env=env).decode("utf-8") + # Hash each nested git repo separately so source changes there still + # trigger do_compile/do_configure to re-run. + for nested in nested_git_dirs: + nested_git = os.path.join(nested, '.git') + if not os.path.isdir(nested_git): + continue + with tempfile.NamedTemporaryFile(prefix='oe-devtool-nested-index') as nested_tmp: + nested_index = os.path.join(nested_git, 'index') + if os.path.exists(nested_index): + shutil.copyfile(nested_index, nested_tmp.name) + nested_env = os.environ.copy()
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 273
Quoted 10 lines
meta/classes/externalsrc.bbclass +10
-0+ nested_env['GIT_INDEX_FILE'] = nested_tmp.name + proc = subprocess.Popen(['git', 'add', '-A', '.'], cwd=nested, + env=nested_env, stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL) + proc.communicate() + proc = subprocess.Popen(['git', 'write-tree'], cwd=nested, + env=nested_env, stdout=subprocess.PIPE, + stderr=subprocess.DEVNULL) + stdout, _ = proc.communicate() + git_sha1 += stdout.decode("utf-8")
We should re-use the code from the following block which handles submodules instead of re-implementing the behaviour. Perhaps the common code needs to be refactored out.
Best regards,
Signature · 1 line
Paul Barker
RE: [PATCH v1 1/2] externalsrc: Handle nested git repos from multiple SRC_URI entries
Hi Paul,
Quoted 9 lines
Subject: Re: [PATCH v1 1/2] externalsrc: Handle nested git repos from multiple
SRC_URIentriesQuoted 5 lines · Jamin Lin
When a recipe uses multiple git
SRC_URIentries with different destsuffix values (e.g. Zephyr-based recipes with separate repos for the kernel, modules, and application), each source is unpacked into a subdirectory of EXTERNALSRC that retains its own .git directory.
Quoted 8 lines
Quoted 3 lines
srctree_hash_files()calls 'git add -A .' at the EXTERNALSRC root, which fails with exit code 128 when git encounters these unregistered nested git repositories, halting the bitbake parse phase.Is this true? The documentation for
git add[1] talks about issuing a warning when this occurs, not an error, and in some quick local testing I get a successful exit (exit code 0) when I try this.
Please try the "zephyr-helloworld" package.
This recipe uses multiple Git repositories in "SRC_URI", and I can reproduce the issue with this package:
Steps to reproduce:
1. git clone https://github.com/openembedded/bitbake.git
2. git clone https://github.com/openembedded/openembedded-core.git
3. git clone https://git.yoctoproject.org/meta-zephyr
4. git clone https://github.com/openembedded/meta-openembedded.git
5. source openembedded-core/oe-init-build-env build
6. Edit build/conf/bblayers.conf and add the following layers:
* meta-openembedded/meta-python
* meta-zephyr/meta-zephyr-core
* meta-openembedded/meta-oe
7. devtool modify zephyr-helloworld
8. bitbake -c cleanall zephyr-helloworld
9. bitbake zephyr-helloworld
After that, the following parser error occurs, and the package can no longer be built:
```text
jamin_lin@aspeed-fw02:~/oe-review/build$ bitbake zephyr-helloworld
Loading cache: 100% |#########################################################################################################################################################################################################| Time: 0:00:01
Loaded 4949 entries from dependency cache.
ERROR: ExpansionError during parsing /home/jamin_lin/oe-review/meta-zephyr/meta-zephyr-core/recipes-kernel/zephyr-kernel/zephyr-helloworld.bb
bb.data_smart.ExpansionError: Failure expanding variable do_compile[file-checksums], expression was ${@srctree_hash_files(d)} which triggered exception CalledProcessError: Command '['git', 'add', '-A', '.']' returned non-zero exit status 128.
The variable dependency chain for the failure is: do_compile[file-checksums]
ERROR: Parsing halted due to errors, see error messages above
Parsing recipes: 0% | | ETA: --:--:--
Summary: There were 2 ERROR messages, returning a non-zero exit code.
```I can consistently reproduce this issue with the steps above.
Thanks,
Jamin
Quoted 96 lines
Quoted 18 lines
Fix by scanning for nested git repos before the add. If any are found, exclude them from the top-level 'git add' using pathspec magic ':(exclude)<path>' and hash each nested repo independently using a temporary index. This ensures changes in any nested repo still trigger do_compile/do_configure to re-run.
Signed-off-by Jamin Lin <[email protected]>--- meta/classes/externalsrc.bbclass | 37 +++++++++++++++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-)meta/classes/externalsrc.bbclass +0
-0meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 237
Quoted 4 lines
meta/classes/externalsrc.bbclass +0
-1@@ -234,8 +234,43 @@ def srctree_hash_files(d, srcdir=None): # Update our custom index env = os.environ.copy() env['GIT_INDEX_FILE'] = tmp_index.name - subprocess.check_output(['git', 'add', '-A', '.'], cwd=s_dir,env=env)
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 237
Quoted 1 line
meta/classes/externalsrc.bbclass +1
-0+ # Find nested git repos created by multiple SRC_URI gitentries with
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 238
Quoted 1 line
meta/classes/externalsrc.bbclass +1
-0+ # different destsuffix values. git add -A . exits 128 when itencounters
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 246
Quoted 8 lines
meta/classes/externalsrc.bbclass +8
-0+ # these unregistered nested repos. + nested_git_dirs = [] + for root, dirs, files in os.walk(s_dir): + if root == s_dir: + continue + if '.git' in dirs or '.git' in files: + nested_git_dirs.append(root) + dirs[:] = [] # don't recurse into nested reposThis
os.walk()loop is expensive, is there an alternative way to handle this?The code has also become difficult to parse. My rule of thumb is that if a group of lines needs a leading comment, it also needs an empty line before the comment to visually separate things.
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 248
Quoted 2 lines
meta/classes/externalsrc.bbclass +2
-0+ if nested_git_dirs: + excludes = [':(exclude)' + os.path.relpath(n, s_dir) for nin
nested_git_dirs]meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 249
Quoted 1 line
meta/classes/externalsrc.bbclass +1
-0+ subprocess.check_output(['git', 'add', '-A', '.'] + excludes,cwd=s_dir, env=env)
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 252
Quoted 3 lines
meta/classes/externalsrc.bbclass +3
-0+ else: + subprocess.check_output(['git', 'add', '-A', '.'], + cwd=s_dir, env=env)To simplify the code, construct a cmd variable and call
subprocess.check_output(cmd, ...)once.meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 255
Quoted 3 lines
meta/classes/externalsrc.bbclass +1
-0git_sha1 = subprocess.check_output(['git', 'write-tree'], cwd=s_dir, env=env).decode("utf-8") + # Hash each nested git repo separately so source changesthere still
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None):
Quoted 6 lines
meta/classes/externalsrc.bbclass +6
-0+ # trigger do_compile/do_configure to re-run. + for nested in nested_git_dirs: + nested_git = os.path.join(nested, '.git') + if not os.path.isdir(nested_git): + continue + with
tempfile.NamedTemporaryFile(prefix='oe-devtool-nested-index')asnested_tmp:meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None):
Quoted 3 lines
meta/classes/externalsrc.bbclass +3
-0+ nested_index = os.path.join(nested_git, 'index') + if os.path.exists(nested_index): + shutil.copyfile(nested_index,nested_tmp.name)
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None):
Quoted 3 lines
meta/classes/externalsrc.bbclass +3
-0+ nested_env = os.environ.copy() + nested_env['GIT_INDEX_FILE'] = nested_tmp.name + proc = subprocess.Popen(['git', 'add', '-A', '.'],cwd=nested,
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 268
Quoted 1 line
meta/classes/externalsrc.bbclass +1
-0+ env=nested_env,stdout=subprocess.DEVNULL,
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 269
Quoted 1 line
meta/classes/externalsrc.bbclass +1
-0+stderr=subprocess.DEVNULL)
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 271
Quoted 2 lines
meta/classes/externalsrc.bbclass +2
-0+ proc.communicate() + proc = subprocess.Popen(['git', 'write-tree'],cwd=nested,
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 272
Quoted 1 line
meta/classes/externalsrc.bbclass +1
-0+ env=nested_env,stdout=subprocess.PIPE,
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 273
Quoted 1 line
meta/classes/externalsrc.bbclass +1
-0+stderr=subprocess.DEVNULL)
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 273
Quoted 2 lines
meta/classes/externalsrc.bbclass +2
-0+ stdout, _ = proc.communicate() + git_sha1 += stdout.decode("utf-8")We should re-use the code from the following block which handles submodules instead of re-implementing the behaviour. Perhaps the common code needs to be refactored out.
Best regards,
--
Paul Barker
Re: [OE-core] [PATCH v1 1/2] externalsrc: Handle nested git repos from multiple SRC_URI entries
Quoted 44 lines · Jamin Lin via lists.openembedded.org
Hi Paul,
Quoted 9 lines
Subject: Re: [PATCH v1 1/2] externalsrc: Handle nested git repos from multiple
SRC_URIentriesQuoted 5 lines · Jamin Lin
When a recipe uses multiple git
SRC_URIentries with different destsuffix values (e.g. Zephyr-based recipes with separate repos for the kernel, modules, and application), each source is unpacked into a subdirectory of EXTERNALSRC that retains its own .git directory.Quoted 8 lines
Quoted 3 lines
srctree_hash_files()calls 'git add -A .' at the EXTERNALSRC root, which fails with exit code 128 when git encounters these unregistered nested git repositories, halting the bitbake parse phase.Is this true? The documentation for
git add[1] talks about issuing a warning when this occurs, not an error, and in some quick local testing I get a successful exit (exit code 0) when I try this.Please try the "zephyr-helloworld" package.
This recipe uses multiple Git repositories in "
SRC_URI", and I can reproduce the issue with this package:Steps to reproduce:
1. git clone https://github.com/openembedded/bitbake.git
2. git clone https://github.com/openembedded/openembedded-core.git
3. git clone https://git.yoctoproject.org/meta-zephyr
4. git clone https://github.com/openembedded/meta-openembedded.git
5. source openembedded-core/oe-init-build-env build
6. Editbuild/conf/bblayers.confand add the following layers:* meta-openembedded/meta-python
* meta-zephyr/meta-zephyr-core
* meta-openembedded/meta-oe
7. devtool modify zephyr-helloworld
8. bitbake -c cleanall zephyr-helloworld
9. bitbake zephyr-helloworldAfter that, the following parser error occurs, and the package can no longer be built:
Quoted 12 lines
```text
jamin_lin@aspeed-fw02:~/oe-review/build$ bitbake zephyr-helloworld
Loading cache: 100% |#########################################################################################################################################################################################################| Time: 0:00:01
Loaded 4949 entries from dependency cache.
ERROR: ExpansionError during parsing /home/jamin_lin/oe-review/meta-zephyr/meta-zephyr-core/recipes-kernel/zephyr-kernel/zephyr-helloworld.bb
bb.data_smart.ExpansionError: Failure expanding variable do_compile[file-checksums], expression was ${@srctree_hash_files(d)} which triggered exception CalledProcessError: Command '['git', 'add', '-A', '.']' returned non-zero exit status 128.
The variable dependency chain for the failure is: do_compile[file-checksums]
ERROR: Parsing halted due to errors, see error messages above
Parsing recipes: 0% | | ETA: --:--:--
Summary: There were 2 ERROR messages, returning a non-zero exit code.
```Following these steps I see the same error. But the cause is not the nested git repositories. bitbake -c cleanall ... has deleted the sources from the downloads directory, but the devtool workspace is still referencing the deleted sources. Even a git status command in the zephyr-helloworld sources directory fails:
$ git -C workspace/sources/zephyr-helloworld status error: unable to normalize alternate object path: /home/pbarker/bitbake-builds/.bitbake-setup-downloads/git2/github.com.zephyrproject-rtos.mcuboot//objects error: unable to normalize alternate object path: /home/pbarker/bitbake-builds/.bitbake-setup-downloads/git2/github.com.zephyrproject-rtos.mcuboot//objects error: unable to normalize alternate object path: /home/pbarker/bitbake-builds/.bitbake-setup-downloads/git2/github.com.zephyrproject-rtos.mcuboot//objects error: unable to normalize alternate object path: /home/pbarker/bitbake-builds/.bitbake-setup-downloads/git2/github.com.zephyrproject-rtos.mcuboot//objects fatal: bad object HEAD fatal: 'git status --porcelain=2' failed in submodule bootloader/mcuboot
If I replace step 8 with bitbake -c cleansstate ... then there is no parse error.
What we need to fix is the interaction between devtool modify and bitbake -c cleanall. Perhaps devtool needs to detect this broken state and tell the user how to fix it - simply running bitbake -c fetch zephyr-helloworld isn't possible as the metadata can't be parsed.
Best regards,
Signature · 1 line
Paul Barker
RE: [OE-core] [PATCH v1 1/2] externalsrc: Handle nested git repos from multiple SRC_URI entries
Hi Paul,
Quoted 104 lines
Subject: Re: [OE-core] [PATCH v1 1/2] externalsrc: Handle nested git repos from multiple
SRC_URIentriesQuoted 15 lines · Jamin Lin via lists.openembedded.org
Hi Paul,
Quoted 13 lines
Subject: Re: [PATCH v1 1/2] externalsrc: Handle nested git repos from multiple
SRC_URIentriesQuoted 8 lines · Jamin Lin
When a recipe uses multiple git
SRC_URIentries with different destsuffix values (e.g. Zephyr-based recipes with separate repos for the kernel, modules, and application), each source is unpacked into a subdirectory of EXTERNALSRC that retains its own .git directory.
srctree_hash_files()calls 'git add -A .' at the EXTERNALSRC root, which fails with exit code 128 when git encounters these unregistered nested git repositories, halting the bitbake parse phase.Quoted 8 lines
Quoted 4 lines
Is this true? The documentation for
git add[1] talks about issuing a warning when this occurs, not an error, and in some quick local testing I get a successful exit (exit code 0) when I try this.Please try the "zephyr-helloworld" package.
This recipe uses multiple Git repositories in "
SRC_URI", and I can reproducethe issue with this package:
Quoted 13 lines
https://git.yoctoproject.org/meta-zephyr/tree/meta-zephyr-core/recipes
-kernel/zephyr-kernel/zephyr-kernel-src-4.3.0.incSteps to reproduce:
1. git clone https://github.com/openembedded/bitbake.git
2. git clone https://github.com/openembedded/openembedded-core.git
3. git clone https://git.yoctoproject.org/meta-zephyr
4. git clone https://github.com/openembedded/meta-openembedded.git
5. source openembedded-core/oe-init-build-env build 6. Editbuild/conf/bblayers.confand add the following layers:Quoted 7 lines
* meta-openembedded/meta-python
* meta-zephyr/meta-zephyr-core
* meta-openembedded/meta-oe
7. devtool modify zephyr-helloworld
8. bitbake -c cleanall zephyr-helloworld 9. bitbake zephyr-helloworldAfter that, the following parser error occurs, and the package can no longer
be built:
Quoted 5 lines
```text
jamin_lin@aspeed-fw02:~/oe-review/build$ bitbake zephyr-helloworld Loading cache: 100%|###############################################################
################################################################
################################################################
##########| Time: 0:00:01 Loaded 4949 entries from dependency cache.Quoted 4 lines
ERROR: ExpansionError during parsing
/home/jamin_lin/oe-review/meta-zephyr/meta-zephyr-core/recipes-kernel/zephyr-kernel/zephyr-helloworld.bb bb.data_smart.ExpansionError: Failure expanding variabledo_compile[file-checksums], expression was ${@srctree_hash_files(d)} which triggered exception CalledProcessError: Command '['git', 'add', '-A', '.']' returned non-zero exit status 128.
Quoted 5 lines
The variable dependency chain for the failure is:
do_compile[file-checksums]ERROR: Parsing halted due to errors, see error messages above
Parsing recipes: 0% || ETA: --:--:--
Quoted 2 lines
Summary: There were 2 ERROR messages, returning a non-zero exit code. ```
Following these steps I see the same error. But the cause is not the nested git repositories.
bitbake -c cleanall ...has deleted the sources from the downloads directory, but the devtool workspace is still referencing the deleted sources. Even agit statuscommand in the zephyr-helloworld sources directory fails:$ git -C workspace/sources/zephyr-helloworld status error: unable to normalize alternate object path: /home/pbarker/bitbake-builds/.bitbake-setup-downloads/git2/github.com.zeph yrproject-rtos.mcuboot//objects error: unable to normalize alternate object path: /home/pbarker/bitbake-builds/.bitbake-setup-downloads/git2/github.com.zeph yrproject-rtos.mcuboot//objects error: unable to normalize alternate object path: /home/pbarker/bitbake-builds/.bitbake-setup-downloads/git2/github.com.zeph yrproject-rtos.mcuboot//objects error: unable to normalize alternate object path: /home/pbarker/bitbake-builds/.bitbake-setup-downloads/git2/github.com.zeph yrproject-rtos.mcuboot//objects fatal: bad object HEAD fatal: 'git status --porcelain=2' failed in submodule bootloader/mcubootIf I replace step 8 with
bitbake -c cleansstate ...then there is no parse error.What we need to fix is the interaction between
devtool modifyandbitbake -c cleanall. Perhaps devtool needs to detect this broken state and tell the user how to fix it - simply runningbitbake -c fetch zephyr-helloworldisn't possible as the metadata can't be parsed.
Quoted 4 lines
Best regards,
--
Paul Barker
Hi Paul,
Thanks for identifying the root cause.
After studying the devtool design, I now understand what happens: when devtool modify creates the workspace, it does not copy the full git object store locally.
Instead, it clones the source tree using git's alternate object store mechanism - the workspace's .git/objects/info/alternates file points back to the downloaded bare repositories under the downloads/ directory. This way the workspace shares the git objects with the download cache rather than duplicating them on disk.
When bitbake -c cleanall is run, it deletes the downloaded bare repositories from downloads/. The workspace source tree still exists, but its alternates now point to missing paths. Any git operation on the workspace (including git status, git add, etc.) then fails because git cannot resolve the object references, producing errors like:
error: unable to normalize alternate object path: .../downloads/git2/.../objects
fatal: bad object HEAD
This is what causes the parser to halt with exit 128.
The fix on my side is straightforward: replace bitbake -c cleanall with bitbake -c cleansstate in my devtool workflow. cleansstate clears the sstate cache without touching the downloads directory, so the workspace alternates remain intact.
That said, I am not sure whether some improvement on the OE side would still be worthwhile. Currently, when this breakage occurs, the user sees only a cryptic CalledProcessError: exit 128 with no indication of what went wrong or how to recover.
The situation is also not obvious: running bitbake -c cleanall on a single-repo recipe with a devtool workspace does not trigger this problem, so users working with multi-repo recipes (such as Zephyr-based recipes that fetch multiple git repositories into separate destsuffix directories) are likely to hit this unexpectedly. A clearer error message pointing to the root cause and the recovery steps (devtool reset -n followed by devtool modify) might save other users from the same confusion.
Please drop this patch series because it is not a normal test case for devtool.
Below patch that may help improve the user experience, though I leave it to the community to decide whether they are worth reviewing:
Best regards,
Jamin
meta/classes/externalsrc.bbclass +51 -0
@@ -227,6 +227,57 @@ def srctree_hash_files(d, srcdir=None): ret = " " if git_dir is not None: + # Find nested git repos created by multiple SRC_URI git entries with + # different destsuffix values, so their alternates can also be checked + # for breakage below. + nested_git_dirs = [] + for root, dirs, files in os.walk(s_dir): + if root == s_dir: + continue + if '.git' in dirs or '.git' in files: + nested_git_dirs.append(root) + dirs[:] = [] + + # Check for broken git alternates in the top-level repo and all nested + # repos. This can happen when 'bitbake -c cleanall' deletes the + # downloads directory that the devtool workspace references via + # alternates, causing all subsequent git operations to exit 128 and + # halt the parse phase. Relative alternate paths are resolved against + # the repo's objects directory; double-slash paths (a known devtool + # artefact) are normalised before the existence check. + def has_broken_alternates(dot_git): + obj_dir = os.path.join(dot_git, 'objects') + alt_file = os.path.join(obj_dir, 'info', 'alternates') + if not os.path.exists(alt_file): + return False + with open(alt_file) as f: + for line in f: + path = line.strip() + if not path or path.startswith('#'): + continue + if not os.path.isabs(path): + path = os.path.join(obj_dir, path) + if not os.path.exists(os.path.normpath(path)): + return True + return False + + broken = has_broken_alternates(git_dir) + if not broken: + for nested in nested_git_dirs: + nested_dot_git = os.path.join(nested, '.git') + if os.path.isdir(nested_dot_git) and has_broken_alternates(nested_dot_git): + broken = True + break + + if broken: + bb.warn('%s: devtool workspace has broken git alternates, ' + 'likely caused by "bitbake -c cleanall" removing the ' + 'downloads directory. ' + 'To repair the workspace, run: ' + 'devtool reset %s && devtool modify %s' + % (d.getVar('PN'), d.getVar('PN'), d.getVar('PN'))) + return s_dir + '/*:True' +
RE: [OE-core] [PATCH v1 1/2] externalsrc: Handle nested git repos from multiple SRC_URI entries
Hi Paul
Quoted 103 lines
Hi Paul,
Thanks for identifying the root cause.
After studying the devtool design, I now understand what happens: when devtool modify creates the workspace, it does not copy the full git object store locally.
Instead, it clones the source tree using git's alternate object store mechanism - the workspace's .git/objects/info/alternates file points back to the downloaded bare repositories under the downloads/ directory.
This way the workspace shares the git objects with the download cache rather than duplicating them on disk.When bitbake -c cleanall is run, it deletes the downloaded bare repositories from downloads/. The workspace source tree still exists, but its alternates now point to missing paths.
Any git operation on the workspace (including git status, git add, etc.) then fails because git cannot resolve the object references, producing errors like:error: unable to normalize alternate object path: .../downloads/git2/.../objects
fatal: bad object HEADThis is what causes the parser to halt with exit 128.
The fix on my side is straightforward: replace bitbake -c cleanall with bitbake -c cleansstate in my devtool workflow. cleansstate clears the sstate cache without touching the downloads directory, so the workspace alternates remain intact.
That said, I am not sure whether some improvement on the OE side would still be worthwhile. Currently, when this breakage occurs, the user sees only a cryptic CalledProcessError: exit 128 with no indication of what went wrong or how to recover.
The situation is also not obvious: running bitbake -c cleanall on a single-repo recipe with a devtool workspace does not trigger this problem, so users working with multi-repo recipes (such as Zephyr-based recipes that fetch multiple git repositories into separate destsuffix directories) are likely to hit this unexpectedly. A clearer error message pointing to the root cause and the recovery steps (devtool reset -n followed by devtool modify) might save other users from the same confusion.Please drop this patch series because it is not a normal test case for devtool.
Below patch that may help improve the user experience, though I leave it to the community to decide whether they are worth reviewing:
Best regards,
Jaminmeta/classes/externalsrc.bbclass +12
-0@@ -227,6 +227,57 @@ def srctree_hash_files(d, srcdir=None): ret = " " if git_dir is not None: + # Find nested git repos created by multiple SRC_URI git entries with + # different destsuffix values, so their alternates can also be checked + # for breakage below. + nested_git_dirs = [] + for root, dirs, files in os.walk(s_dir): + if root == s_dir: + continue + if '.git' in dirs or '.git' in files: + nested_git_dirs.append(root) + dirs[:] = [] + + # Check for broken git alternates in the top-level repo and allnested
meta/classes/externalsrc.bbclass +4
-0+ # repos. This can happen when 'bitbake -c cleanall' deletes the + # downloads directory that the devtool workspace references via + # alternates, causing all subsequent git operations to exit 128 and + # halt the parse phase. Relative alternate paths are resolvedagainst
meta/classes/externalsrc.bbclass +23
-0+ # the repo's objects directory; double-slash paths (a known devtool + # artefact) are normalised before the existence check. + def has_broken_alternates(dot_git): + obj_dir = os.path.join(dot_git, 'objects') + alt_file = os.path.join(obj_dir, 'info', 'alternates') + if not os.path.exists(alt_file): + return False + with open(alt_file) as f: + for line in f: + path = line.strip() + if not path or path.startswith('#'): + continue + if not os.path.isabs(path): + path = os.path.join(obj_dir, path) + if not os.path.exists(os.path.normpath(path)): + return True + return False + + broken = has_broken_alternates(git_dir) + if not broken: + for nested in nested_git_dirs: + nested_dot_git = os.path.join(nested, '.git') + if os.path.isdir(nested_dot_git) and
has_broken_alternates(nested_dot_git):meta/classes/externalsrc.bbclass +2
-0+ broken = True + break
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 280
Quoted 10 lines
meta/classes/externalsrc.bbclass +10
-0+ + if broken: + bb.warn('%s: devtool workspace has broken git alternates, ' + 'likely caused by "bitbake -c cleanall" removing the ' + 'downloads directory. ' + 'To repair the workspace, run: ' + 'devtool reset %s && devtool modify %s' + % (d.getVar('PN'), d.getVar('PN'), d.getVar('PN'))) + return s_dir + '/*:True' +
Please ignore the comments above.
I believe I have identified the root cause. Please see my updated solution here:
https://patchwork.yoctoproject.org/project/oe-core/patch/[email protected]/
Thanks,
Jamin
Quoted 3 lines
RE: [PATCH v1 1/2] externalsrc: Handle nested git repos from multiple SRC_URI entries
In reply to Paul BarkerHi Paul,
Quoted 53 lines
Subject: Re: [PATCH v1 1/2] externalsrc: Handle nested git repos from multiple
SRC_URIentriesQuoted 8 lines · Jamin Lin
When a recipe uses multiple git
SRC_URIentries with different destsuffix values (e.g. Zephyr-based recipes with separate repos for the kernel, modules, and application), each source is unpacked into a subdirectory of EXTERNALSRC that retains its own .git directory.
srctree_hash_files()calls 'git add -A .' at the EXTERNALSRC root, which fails with exit code 128 when git encounters these unregistered nested git repositories, halting the bitbake parse phase.Is this true? The documentation for
git add[1] talks about issuing a warning when this occurs, not an error, and in some quick local testing I get a successful exit (exit code 0) when I try this.Quoted 18 lines
Fix by scanning for nested git repos before the add. If any are found, exclude them from the top-level 'git add' using pathspec magic ':(exclude)<path>' and hash each nested repo independently using a temporary index. This ensures changes in any nested repo still trigger do_compile/do_configure to re-run.
Signed-off-by Jamin Lin <[email protected]>--- meta/classes/externalsrc.bbclass | 37 +++++++++++++++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-)meta/classes/externalsrc.bbclass +0
-0meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 237
Quoted 4 lines
meta/classes/externalsrc.bbclass +0
-1@@ -234,8 +234,43 @@ def srctree_hash_files(d, srcdir=None): # Update our custom index env = os.environ.copy() env['GIT_INDEX_FILE'] = tmp_index.name - subprocess.check_output(['git', 'add', '-A', '.'], cwd=s_dir,env=env)
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 237
Quoted 1 line
meta/classes/externalsrc.bbclass +1
-0+ # Find nested git repos created by multiple SRC_URI gitentries with
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 238
Quoted 1 line
meta/classes/externalsrc.bbclass +1
-0+ # different destsuffix values. git add -A . exits 128 when itencounters
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 246
Quoted 8 lines
meta/classes/externalsrc.bbclass +8
-0+ # these unregistered nested repos. + nested_git_dirs = [] + for root, dirs, files in os.walk(s_dir): + if root == s_dir: + continue + if '.git' in dirs or '.git' in files: + nested_git_dirs.append(root) + dirs[:] = [] # don't recurse into nested repos
Quoted 2 lines
This
os.walk()loop is expensive, is there an alternative way to handle this?
os.scandir() was considered but rejected: destsuffix allows arbitrarily
deep paths (e.g. destsuffix=${P}/modules/tee/tf-m/trusted-firmware-m), so a depth-1 scan would silently miss nested repos and stop hashing their content — source
changes there would no longer trigger do_compile to re-run.
The os.walk() loop uses dirs[:] = [] to stop recursing as soon as a .git entry is found, so we never descend into the nested repos themselves (which may contain tens of thousands of files). The walk only traverses the shallow skeleton of intermediate directories between EXTERNALSRC and each nested repo.
Please see the use case here:
https://git.yoctoproject.org/meta-zephyr/tree/meta-zephyr-core/recipes-kernel/zephyr-kernel/zephyr-kernel-src-4.3.0.inc
${SRC_URI_ZEPHYR_OPEN_AMP};name=open-amp;nobranch=1;destsuffix=${P}/modules/lib/open-amp \
${SRC_URI_ZEPHYR_TRUSTED_FIRMWARE_M};name=trusted-firmware-m;nobranch=1;destsuffix=${P}/modules/tee/tf-m/trusted-firmware-m \
Thanks-Jamin
Quoted 58 lines
The code has also become difficult to parse. My rule of thumb is that if a group of lines needs a leading comment, it also needs an empty line before the comment to visually separate things.
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 248
Quoted 2 lines
meta/classes/externalsrc.bbclass +2
-0+ if nested_git_dirs: + excludes = [':(exclude)' + os.path.relpath(n, s_dir) for nin
nested_git_dirs]meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 249
Quoted 1 line
meta/classes/externalsrc.bbclass +1
-0+ subprocess.check_output(['git', 'add', '-A', '.'] + excludes,cwd=s_dir, env=env)
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 252
Quoted 3 lines
meta/classes/externalsrc.bbclass +3
-0+ else: + subprocess.check_output(['git', 'add', '-A', '.'], + cwd=s_dir, env=env)To simplify the code, construct a cmd variable and call
subprocess.check_output(cmd, ...)once.meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 255
Quoted 3 lines
meta/classes/externalsrc.bbclass +1
-0git_sha1 = subprocess.check_output(['git', 'write-tree'], cwd=s_dir, env=env).decode("utf-8") + # Hash each nested git repo separately so source changesthere still
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None):
Quoted 6 lines
meta/classes/externalsrc.bbclass +6
-0+ # trigger do_compile/do_configure to re-run. + for nested in nested_git_dirs: + nested_git = os.path.join(nested, '.git') + if not os.path.isdir(nested_git): + continue + with
tempfile.NamedTemporaryFile(prefix='oe-devtool-nested-index')asnested_tmp:meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None):
Quoted 3 lines
meta/classes/externalsrc.bbclass +3
-0+ nested_index = os.path.join(nested_git, 'index') + if os.path.exists(nested_index): + shutil.copyfile(nested_index,nested_tmp.name)
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None):
Quoted 3 lines
meta/classes/externalsrc.bbclass +3
-0+ nested_env = os.environ.copy() + nested_env['GIT_INDEX_FILE'] = nested_tmp.name + proc = subprocess.Popen(['git', 'add', '-A', '.'],cwd=nested,
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 268
Quoted 1 line
meta/classes/externalsrc.bbclass +1
-0+ env=nested_env,stdout=subprocess.DEVNULL,
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 269
Quoted 1 line
meta/classes/externalsrc.bbclass +1
-0+stderr=subprocess.DEVNULL)
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 271
Quoted 2 lines
meta/classes/externalsrc.bbclass +2
-0+ proc.communicate() + proc = subprocess.Popen(['git', 'write-tree'],cwd=nested,
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 272
Quoted 1 line
meta/classes/externalsrc.bbclass +1
-0+ env=nested_env,stdout=subprocess.PIPE,
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 273
Quoted 1 line
meta/classes/externalsrc.bbclass +1
-0+stderr=subprocess.DEVNULL)
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 273
Quoted 2 lines
meta/classes/externalsrc.bbclass +2
-0+ stdout, _ = proc.communicate() + git_sha1 += stdout.decode("utf-8")We should re-use the code from the following block which handles submodules instead of re-implementing the behaviour. Perhaps the common code needs to be refactored out.
Best regards,
--
Paul Barker
RE: [PATCH v1 1/2] externalsrc: Handle nested git repos from multiple SRC_URI entries
Hi Paul,
Quoted 113 lines
Subject: RE: [PATCH v1 1/2] externalsrc: Handle nested git repos from multiple
SRC_URIentriesHi Paul,
Quoted 54 lines
Subject: Re: [PATCH v1 1/2] externalsrc: Handle nested git repos from multiple
SRC_URIentriesQuoted 8 lines · Jamin Lin
When a recipe uses multiple git
SRC_URIentries with different destsuffix values (e.g. Zephyr-based recipes with separate repos for the kernel, modules, and application), each source is unpacked into a subdirectory of EXTERNALSRC that retains its own .git directory.
srctree_hash_files()calls 'git add -A .' at the EXTERNALSRC root, which fails with exit code 128 when git encounters these unregistered nested git repositories, halting the bitbake parse phase.Is this true? The documentation for
git add[1] talks about issuing a warning when this occurs, not an error, and in some quick local testing I get a successful exit (exit code 0) when I try this.Quoted 18 lines
Fix by scanning for nested git repos before the add. If any are found, exclude them from the top-level 'git add' using pathspec magic ':(exclude)<path>' and hash each nested repo independently using a temporary index. This ensures changes in any nested repo still trigger do_compile/do_configure to re-run.
Signed-off-by Jamin Lin <[email protected]>--- meta/classes/externalsrc.bbclass | 37 +++++++++++++++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-)meta/classes/externalsrc.bbclass +0
-0meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 237
Quoted 4 lines
meta/classes/externalsrc.bbclass +0
-1@@ -234,8 +234,43 @@ def srctree_hash_files(d, srcdir=None): # Update our custom index env = os.environ.copy() env['GIT_INDEX_FILE'] = tmp_index.name - subprocess.check_output(['git', 'add', '-A', '.'], cwd=s_dir,env=env)
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 237
Quoted 1 line
meta/classes/externalsrc.bbclass +1
-0+ # Find nested git repos created by multiple SRC_URI gitentries with
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 239
Quoted 2 lines
meta/classes/externalsrc.bbclass +2
-0+ # different destsuffix values. git add -A . exits 128 + when itencounters
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 246
Quoted 8 lines
meta/classes/externalsrc.bbclass +8
-0+ # these unregistered nested repos. + nested_git_dirs = [] + for root, dirs, files in os.walk(s_dir): + if root == s_dir: + continue + if '.git' in dirs or '.git' in files: + nested_git_dirs.append(root) + dirs[:] = [] # don't recurse into nested reposQuoted 2 lines
This
os.walk()loop is expensive, is there an alternative way to handle this?
os.scandir()was considered but rejected: destsuffix allows arbitrarily deep paths (e.g. destsuffix=${P}/modules/tee/tf-m/trusted-firmware-m), so a depth-1 scan would silently miss nested repos and stop hashing their content — source changes there would no longer triggerdo_compileto re-run.The
os.walk()loop uses dirs[:] = [] to stop recursing as soon as a .git entry is found, so we never descend into the nested repos themselves (which may contain tens of thousands of files). The walk only traverses the shallow skeleton of intermediate directories between EXTERNALSRC and each nested repo.Please see the use case here:
https://git.yoctoproject.org/meta-zephyr/tree/meta-zephyr-core/recipes-kernel
/zephyr-kernel/zephyr-kernel-src-4.3.0.inc${SRC_URI_ZEPHYR_OPEN_AMP};name=open-amp;nobranch=1;destsuffix=${P} /modules/lib/open-amp \
${SRC_URI_ZEPHYR_TRUSTED_FIRMWARE_M};name=trusted-firmware-m;nob ranch=1;destsuffix=${P}/modules/tee/tf-m/trusted-firmware-m \Thanks-Jamin
Quoted 31 lines
The code has also become difficult to parse. My rule of thumb is that if a group of lines needs a leading comment, it also needs an empty line before the comment to visually separate things.
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 250
Quoted 3 lines
meta/classes/externalsrc.bbclass +3
-0+ if nested_git_dirs: + excludes = [':(exclude)' + os.path.relpath(n, + s_dir) for nin
nested_git_dirs]meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 252
Quoted 2 lines
meta/classes/externalsrc.bbclass +2
-0+ subprocess.check_output(['git', 'add', '-A', '.'] + + excludes,cwd=s_dir, env=env)
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 255
Quoted 3 lines
meta/classes/externalsrc.bbclass +3
-0+ else: + subprocess.check_output(['git', 'add', '-A', '.'], + cwd=s_dir, env=env)To simplify the code, construct a cmd variable and call
subprocess.check_output(cmd, ...)once.meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 259
Quoted 4 lines
meta/classes/externalsrc.bbclass +2
-0git_sha1 = subprocess.check_output(['git', 'write-tree'], cwd=s_dir, env=env).decode("utf-8") + # Hash each nested git repo separately so source + changesthere still
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None):
Quoted 6 lines
meta/classes/externalsrc.bbclass +6
-0+ # trigger do_compile/do_configure to re-run. + for nested in nested_git_dirs: + nested_git = os.path.join(nested, '.git') + if not os.path.isdir(nested_git): + continue + with
tempfile.NamedTemporaryFile(prefix='oe-devtool-nested-index')asnested_tmp:
Quoted 32 lines
Quoted 6 lines
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None):
Quoted 3 lines
meta/classes/externalsrc.bbclass +3
-0+ nested_index = os.path.join(nested_git, 'index') + if os.path.exists(nested_index): + shutil.copyfile(nested_index,nested_tmp.name)
Quoted 2 lines
meta/classes/externalsrc.bbclass +2
-0+ nested_env = os.environ.copy() + nested_env['GIT_INDEX_FILE'] =nested_tmp.name
Quoted 25 lines
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 272
Quoted 2 lines
meta/classes/externalsrc.bbclass +2
-0+ proc = subprocess.Popen(['git', 'add', '-A', + '.'],cwd=nested,
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 273
Quoted 1 line
meta/classes/externalsrc.bbclass +1
-0+ env=nested_env,stdout=subprocess.DEVNULL,
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 274
Quoted 1 line
meta/classes/externalsrc.bbclass +1
-0+stderr=subprocess.DEVNULL)
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 276
Quoted 2 lines
meta/classes/externalsrc.bbclass +2
-0+ proc.communicate() + proc = subprocess.Popen(['git', 'write-tree'],cwd=nested,
Quoted 1 line
+ env=nested_env,
stdout=subprocess.PIPE,
Quoted 1 line
+
stderr=subprocess.DEVNULL)
meta/classes/externalsrc.bbclass def srctree_hash_files(d, srcdir=None): · line 273
Quoted 2 lines
meta/classes/externalsrc.bbclass +2
-0+ stdout, _ = proc.communicate() + git_sha1 += stdout.decode("utf-8")We should re-use the code from the following block which handles submodules instead of re-implementing the behaviour. Perhaps the common code needs to be refactored out.
Best regards,
--
Paul Barker
Thanks for the review and suggestion.
Will send v2 as below.
```
diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
index 902ff2604f..f2e0812eea 100644
--- a/meta/classes/externalsrc.bbclass
+++ b/meta/classes/externalsrc.bbclass
@@ -234,18 +234,47 @@ def srctree_hash_files(d, srcdir=None):
# Update our custom index
env = os.environ.copy()
env['GIT_INDEX_FILE'] = tmp_index.name
- subprocess.check_output(['git', 'add', '-A', '.'], cwd=s_dir, env=env)
+
+ # Find nested git repos created by multiple SRC_URI git entries with
+ # different destsuffix values. When GIT_INDEX_FILE is set, git add -A .
+ # exits 128 instead of warning when it encounters these unregistered
+ # nested repos, halting the bitbake parse phase.
+ nested_git_dirs = []
+ for root, dirs, files in os.walk(s_dir):
+ if root == s_dir:
+ continue
+ if '.git' in dirs or '.git' in files:
+ nested_git_dirs.append(root)
+ dirs[:] = []
+
+ cmd = ['git', 'add', '-A', '.']
+ if nested_git_dirs:
+ cmd += [':(exclude)' + os.path.relpath(n, s_dir) for n in nested_git_dirs]
+ subprocess.check_output(cmd, cwd=s_dir, env=env)
+
git_sha1 = subprocess.check_output(['git', 'write-tree'], cwd=s_dir, env=env).decode("utf-8")
- if os.path.exists(os.path.join(s_dir, ".gitmodules")) and os.path.getsize(os.path.join(s_dir, ".gitmodules")) > 0:
+
+ # Hash nested git repos and submodules together so changes in any of
+ # them still trigger do_compile/do_configure to re-run.
+ subdirs_to_hash = list(nested_git_dirs)
+ if os.path.exists(os.path.join(s_dir, ".gitmodules")) and \
+ os.path.getsize(os.path.join(s_dir, ".gitmodules")) > 0:
submodule_helper = subprocess.check_output(["git", "config", "--file", ".gitmodules", "--get-regexp", "path"], cwd=s_dir, env=env).decode("utf-8")
for line in submodule_helper.splitlines():
module_dir = os.path.join(s_dir, line.rsplit(maxsplit=1)[1])
if os.path.isdir(module_dir):
- proc = subprocess.Popen(['git', 'add', '-A', '.'], cwd=module_dir, env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
- proc.communicate()
- proc = subprocess.Popen(['git', 'write-tree'], cwd=module_dir, env=env, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
- stdout, _ = proc.communicate()
- git_sha1 += stdout.decode("utf-8")
+ subdirs_to_hash.append(module_dir)
+
+ for subdir in subdirs_to_hash:
+ proc = subprocess.Popen(['git', 'add', '-A', '.'], cwd=subdir,
+ env=env, stdout=subprocess.DEVNULL,
+ stderr=subprocess.DEVNULL)
+ proc.communicate()
+ proc = subprocess.Popen(['git', 'write-tree'], cwd=subdir,
+ env=env, stdout=subprocess.PIPE,
+ stderr=subprocess.DEVNULL)
+ stdout, _ = proc.communicate()
+ git_sha1 += stdout.decode("utf-8")
sha1 = hashlib.sha1(git_sha1.encode("utf-8")).hexdigest()
with open(oe_hash_file, 'w') as fobj:
fobj.write(sha1)
```Thanks,
Jamin
PATCH 2/2 reproducible: Handle nested git repos in find_git_repositories
In reply to Jamin LinWhen EXTERNALSRC contains multiple nested git repositories (from multiple SRC_URI git entries with different destsuffix values), find_git_repositories() walks into sub-repos and get_source_date_epoch_from_git() subsequently fails with exit code 128 when running 'git log -1' inside them.
Two fixes:
- Stop os.walk recursion when a .git entry is found (dirs[:] = []) to avoid descending into nested repos.
- Change 'git log -1' from check=True to check=False with explicit error handling, so a failing nested repo is skipped gracefully instead of raising CalledProcessError and aborting do_unpack.
--- meta/lib/oe/reproducible.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
meta/lib/oe/reproducible.py +5 -1
@@ -82,6 +82,7 @@ def find_git_repositories(d, sourcedir): for root, dirs, files in os.walk(mainpath, topdown=True): if '.git' in dirs or '.git' in files: git_repositories.append(root) + dirs[:] = [] # don't recurse into nested git repos (multiple SRC_URI destsuffix) if not git_repositories: bb.warn('Failed to find any git repositories in UNPACKDIR or S')
@@ -105,7 +106,10 @@ def get_source_date_epoch_from_git(d, sourcedir): bb.debug(1, "git repository: %s" % repo_path) p = subprocess.run(['git', '-C', repo_path, 'log', '-1', '--pretty=%ct'], - check=True, stdout=subprocess.PIPE) + check=False, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) + if p.returncode != 0: + bb.debug(1, "git log failed for %s (exit %d): %s" % (repo_path, p.returncode, p.stdout.decode('utf-8'))) + continue source_dates.append(int(p.stdout.decode('utf-8'))) if source_dates:
Signature · 1 line
2.43.0
Re: [PATCH v1 2/2] reproducible: Handle nested git repos in find_git_repositories
Quoted 31 lines · Jamin Lin
When EXTERNALSRC contains multiple nested git repositories (from multiple
SRC_URIgit entries with different destsuffix values),find_git_repositories()walks into sub-repos andget_source_date_epoch_from_git()subsequently fails with exit code 128 when running 'git log -1' inside them.Two fixes:
- Stop os.walk recursion when a .git entry is found (dirs[:] = []) to avoid descending into nested repos.
- Change 'git log -1' from check=True to check=False with explicit error handling, so a failing nested repo is skipped gracefully instead of raising CalledProcessError and abortingdo_unpack.Signed-off-by Jamin Lin <[email protected]>--- meta/lib/oe/reproducible.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)meta/lib/oe/reproducible.py +1
-0@@ -82,6 +82,7 @@ def find_git_repositories(d, sourcedir): for root, dirs, files in os.walk(mainpath, topdown=True): if '.git' in dirs or '.git' in files: git_repositories.append(root) + dirs[:] = [] # don't recurse into nested git repos (multiple SRC_URI destsuffix) if not git_repositories: bb.warn('Failed to find any git repositories in UNPACKDIR or S')
meta/lib/oe/reproducible.py def get_source_date_epoch_from_git(d, sourcedir): · line 113
Quoted 9 lines
meta/lib/oe/reproducible.py +4
-1@@ -105,7 +106,10 @@ def get_source_date_epoch_from_git(d, sourcedir): bb.debug(1, "git repository: %s" % repo_path) p = subprocess.run(['git', '-C', repo_path, 'log', '-1', '--pretty=%ct'], - check=True, stdout=subprocess.PIPE) + check=False, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) + if p.returncode != 0: + bb.debug(1, "git log failed for %s (exit %d): %s" % (repo_path, p.returncode, p.stdout.decode('utf-8'))) + continue source_dates.append(int(p.stdout.decode('utf-8')))
This causes the repository to be silently (in most cases, since bb.debug() output is not usually printed) dropped from the source_dates array. Does this invalidate the returned source date epoch?
We may want a warning here instead of a debug print.
Also note that there may be other reasons for the git log command to fail, and the output that identifies why it failed is likely to be in stderr rather than stdout so we don't want to throw that way.
Best regards,
Signature · 1 line
Paul Barker