linux (git/curl-native) and autorev
From: Max Krummenacher <[email protected]>
Hello
If one builds a kernel using AUTOREV invoking bitbake only works once. Any subsequent bitbake invocation fails parsing the meta data.
Reproducable with:
- latest poky, b5624ee564
- Kernel with SRCREV = "AUTOREV", e.g. in local.conf SRCREV_machine:pn-linux-yocto:forcevariable = "${AUTOREV}"
- bitbake virtual/kernel; bitbake virtual/kernel
On the second invocation parsing fails when the fetcher tries to evaluate the latest SRCREV:
| ERROR: ExpansionError during parsing meta/recipes-kernel/linux/linux-yocto_6.6.bb
| Traceback (most recent call last):
| File "bitbake/lib/bb/fetch2/__init__.py", line 1245, in srcrev_internal_helper(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
| d.setVar("__BBAUTOREV_ACTED_UPON", True)
| > srcrev = ud.method.latest_revision(ud, d, name)
|
| File "bitbake/lib/bb/fetch2/__init__.py", line 1667, in Git.latest_revision(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
| except KeyError:
| > revs[key] = rev = self._latest_revision(ud, d, name)
| return rev
| File "bitbake/lib/bb/fetch2/git.py", line 850, in Git._latest_revision(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
|
| > output = self._lsremote(ud, d, "")
| # Tags of the form ^{} may not work, need to fallback to other form
| File "bitbake/lib/bb/fetch2/git.py", line 833, in Git._lsremote(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, search=''):
| bb.fetch2.check_network_access(d, cmd, repourl)
| > output = runfetchcmd(cmd, d, True)
| if not output:
| File "bitbake/lib/bb/fetch2/__init__.py", line 957, in runfetchcmd(cmd='export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64-linux/usr/bin:scripts:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all ls-remote https://git.yoctoproject.org/linux-yocto.git ', d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, quiet=True, cleanup=[], log=None, workdir=None):
|
| > raise FetchError(error_message)
|
| bb.data_smart.ExpansionError: Failure expanding variable fetcher_hashes_dummyfunc[vardepvalue], expression was ${@bb.fetch.get_hashvalue(d)} which triggered exception FetchError: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64-linux/usr/bin:scripts:/var/home/krm/build/poky/build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all ls-remote https://git.yoctoproject.org/linux-yocto.git failed with exit code 128, output:
| fatal: unable to access 'https://git.yoctoproject.org/linux-yocto.git/': error setting certificate file: build/tmp/work/x86_64-linux/curl-native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca-certificates.crt
|
| The variable dependency chain for the failure is: fetcher_hashes_dummyfunc[vardepvalue]Note:
One gets out of that parser error by deleting the git binary in the
kernel's work recipe-sysroot-nativerm tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin/git
Bisecting poky leads to commitf7fa98cca8 ("kern-tools: depend on git-replacement-native") Reverting it on top of b5624ee564 makes the parsing pass.
I assume that git-replacement-native does not work with https, the fetch error also goes away if changing in SRC_URI from https to git.
Any comments?
Cheers
Max
Quoted 49 lines · On Sat, Mar 9, 2024 at 12:58 PM <[email protected]> wrote:
From: Max Krummenacher <[email protected]>
Hello
If one builds a kernel using AUTOREV invoking bitbake only works once. Any subsequent bitbake invocation fails parsing the meta data.
Reproducable with:
- latest poky,b5624ee564
- Kernel with SRCREV = "AUTOREV", e.g. in local.confSRCREV_machine:pn-linux-yocto:forcevariable = "${AUTOREV}"
- bitbake virtual/kernel; bitbake virtual/kernelOn the second invocation parsing fails when the fetcher tries to evaluate the latest SRCREV:
| ERROR: ExpansionError during parsing meta/recipes-kernel/linux/linux-yocto_6.6.bb | Traceback (most recent call last): | File "bitbake/lib/bb/fetch2/__init__.py", line 1245, in srcrev_internal_helper(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'): | d.setVar("__BBAUTOREV_ACTED_UPON", True) | > srcrev = ud.method.latest_revision(ud, d, name) | | File "bitbake/lib/bb/fetch2/__init__.py", line 1667, in Git.latest_revision(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'): | except KeyError: | > revs[key] = rev = self._latest_revision(ud, d, name) | return rev | File "bitbake/lib/bb/fetch2/git.py", line 850, in Git._latest_revision(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'): | | > output = self._lsremote(ud, d, "") | # Tags of the form ^{} may not work, need to fallback to other form | File "bitbake/lib/bb/fetch2/git.py", line 833, in Git._lsremote(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, search=''): | bb.fetch2.check_network_access(d, cmd, repourl) | > output = runfetchcmd(cmd, d, True) | if not output: | File "bitbake/lib/bb/fetch2/__init__.py", line 957, in runfetchcmd(cmd='export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64-linux/usr/bin:scripts:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all ls-remote https://git.yoctoproject.org/linux-yocto.git ', d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, quiet=True, cleanup=[], log=None, workdir=None): | | > raise FetchError(error_message) | | bb.data_smart.ExpansionError: Failure expanding variable fetcher_hashes_dummyfunc[vardepvalue], expression was ${@bb.fetch.get_hashvalue(d)} which triggered exception FetchError: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64-linux/usr/bin:scripts:/var/home/krm/build/poky/build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all ls-remote https://git.yoctoproject.org/linux-yocto.git failed with exit code 128, output: | fatal: unable to access 'https://git.yoctoproject.org/linux-yocto.git/': error setting certificate file: build/tmp/work/x86_64-linux/curl-native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca-certificates.crt | | The variable dependency chain for the failure is: fetcher_hashes_dummyfunc[vardepvalue]Note:
One gets out of that parser error by deleting the git binary in the
kernel's work recipe-sysroot-nativerm tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin/git
Quoted 8 lines
Bisecting poky leads to commit
f7fa98cca8("kern-tools: depend on git-replacement-native") Reverting it on top ofb5624ee564makes the parsing pass.I assume that
git-replacement-nativedoes not work with https, the fetch error also goes away if changing inSRC_URIfrom https to git.Any comments?
I didn't even know that curl was coming into play :)
Adding DEPENDS:class-native += "ca-certificates" to the curl recipe should resolve the issue.
I also tried exporting the SSL_CERT_DIR in the git-native wrapper, but that didn't help
since of course it is curl that is looking for the certs.
I also thought we could export the SSL_CERT_DIR from within the kern-tools, but really,
it does seem like curl should know that this is a dependency when used as a native
tool (it already has ca-certficates as a RRECOMMENDS).
I haven't done a patch for it, because the intricacies of the native classes and where
we should actually add that depends is someone lost on me.
Richard will likely know the right place to fix it .. and then I'm happy to spin a patch.
Bruce
Quoted 3 lines
Cheers
Max
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
Am Samstag, dem 09.03.2024 um 13:04 -0800 schrieb Bruce Ashfield:
Quoted 57 lines · On Sat, Mar 9, 2024 at 12:58 PM <[email protected]> wrote:
Quoted 56 lines
From: Max Krummenacher <[email protected]>
Hello
If one builds a kernel using AUTOREV invoking bitbake only works once. Any subsequent bitbake invocation fails parsing the meta data.
Reproducable with:
- latest poky,b5624ee564
- Kernel with SRCREV = "AUTOREV", e.g. in local.confSRCREV_machine:pn-linux-yocto:forcevariable = "${AUTOREV}"
- bitbake virtual/kernel; bitbake virtual/kernelOn the second invocation parsing fails when the fetcher tries to evaluate the latest SRCREV:
Quoted 20 lines
ERROR: ExpansionError during parsing meta/recipes-kernel/linux/linux-yocto_6.6.bb
Traceback (most recent call last):
File "bitbake/lib/bb/fetch2/__init__.py", line 1245, in srcrev_internal_helper(ud=<bb.fetch2.FetchData object at0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at0x7f8e26195890>, name='machine'):
d.setVar("__BBAUTOREV_ACTED_UPON", True)
> srcrev =ud.method.latest_revision(ud, d, name)File "
bitbake/lib/bb/fetch2/__init__.py", line 1667, in Git.latest_revision(ud=<bb.fetch2.FetchData object at0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at0x7f8e26195890>, name='machine'):
except KeyError:
> revs[key] = rev =self._latest_revision(ud, d, name)
return rev
File "bitbake/lib/bb/fetch2/git.py", line 850, in Git._latest_revision(ud=<bb.fetch2.FetchData object at0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at0x7f8e26195890>, name='machine'):Quoted 1 line
output = self._lsremote(ud, d, "")# Tags of the form ^{} may not work, need to fallback to other form
File "bitbake/lib/bb/fetch2/git.py", line 833, in Git._lsremote(ud=<bb.fetch2.FetchData object at0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at0x7f8e26195890>, search=''):
bb.fetch2.check_network_access(d, cmd, repourl)
> output =runfetchcmd(cmd, d, True)
if not output:
File "bitbake/lib/bb/fetch2/__init__.py", line 957, in runfetchcmd(cmd='export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64-linux/usr/bin:scripts:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all ls-remote https://git.yoctoproject.org/linux-yocto.git ', d=<bb.data_smart.DataSmart object at0x7f8e26195890>, quiet=True, cleanup=[], log=None, workdir=None):Quoted 6 lines
Quoted 1 line
raise
FetchError(error_message)bb.data_smart.ExpansionError: Failure expanding variable fetcher_hashes_dummyfunc[vardepvalue], expression was ${@bb.fetch.get_hashvalue(d)} which triggered exception FetchError: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64-linux/usr/bin:scripts:/var/home/krm/build/poky/build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all ls-remote https://git.yoctoproject.org/linux-yocto.git failed with exit code 128, output:
fatal: unable to access 'https://git.yoctoproject.org/linux-yocto.git/': error setting certificate file: build/tmp/work/x86_64-linux/curl-native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca-certificates.crtThe variable dependency chain for the failure is: fetcher_hashes_dummyfunc[vardepvalue]
Note:
One gets out of that parser error by deleting the git binary in the
kernel's work recipe-sysroot-nativerm tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin/gitBisecting poky leads to commit
f7fa98cca8("kern-tools: depend on git-replacement-native") Reverting it on top ofb5624ee564makes the parsing pass.I assume that
git-replacement-nativedoes not work with https, the fetch error also goes away if changing inSRC_URIfrom https to git.
Quoted 6 lines
Quoted 1 line
Any comments?
I didn't even know that curl was coming into play :)
Adding DEPENDS:class-native += "ca-certificates" to the curl recipe should resolve the issue.
Looks like curl-native resp. libcurl hardcodes the lookup to its own work directory, i.e.:
x86_64-linux/curl-native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca-certificates.crt
So even if DEPENDS/RDEPENDS will install ca-certificates in the kernel's recipe-sysroot-native the parsing will fail if the curl-native directory is not/no longer populated, e.g. because curl-native came from sstate or rm_work is in INHERIT.
Max
Quoted 39 lines
I also tried exporting the
SSL_CERT_DIRin the git-native wrapper, but that didn't help
since of course it is curl that is looking for the certs.I also thought we could export the
SSL_CERT_DIRfrom within the kern-tools, but really,
it does seem like curl should know that this is a dependency when used as a native
tool (it already has ca-certficates as a RRECOMMENDS).I haven't done a patch for it, because the intricacies of the native classes and where
we should actually add that depends is someone lost on me.Richard will likely know the right place to fix it .. and then I'm happy to spin a patch.
Bruce
Quoted 3 lines
Cheers
Max--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#196873): https://lists.openembedded.org/g/openembedded-core/message/196873
Mute This Topic: https://lists.openembedded.org/mt/104831542/3617484
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 140 lines · Max
Am Samstag, dem 09.03.2024 um 13:04 -0800 schrieb Bruce Ashfield:
Quoted 127 lines · On Sat, Mar 9, 2024 at 12:58 PM <[email protected]> wrote:
Quoted 126 lines
From: Max Krummenacher <[email protected]>
Hello
If one builds a kernel using AUTOREV invoking bitbake only works once.
Any subsequent bitbake invocation fails parsing the meta data.Reproducable with:
- latest poky,b5624ee564
- Kernel with SRCREV = "AUTOREV", e.g. in local.confSRCREV_machine:pn-linux-yocto:forcevariable = "${AUTOREV}"
- bitbake virtual/kernel; bitbake virtual/kernelOn the second invocation parsing fails when the fetcher tries to evaluate the latest SRCREV:
Quoted 89 lines
ERROR: ExpansionError during parsing meta/recipes-
kernel/linux/linux-yocto_6.6.bb
Traceback (most recent call last):
File "bitbake/lib/bb/fetch2/__init__.py", line 1245, in srcrev_internal_helper(ud=<bb.fetch2.FetchData object at0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at0x7f8e26195890>, name='machine'):
d.setVar("__BBAUTOREV_ACTED_UPON", True)> srcrev =ud.method.latest_revision(ud, d, name)File "
bitbake/lib/bb/fetch2/__init__.py", line 1667, in Git.latest_revision(ud=<bb.fetch2.FetchData object at0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at0x7f8e26195890>, name='machine'):
except KeyError:
> revs[key] = rev =self._latest_revision(ud, d, name)
return rev
File "bitbake/lib/bb/fetch2/git.py", line 850, in Git._latest_revision(ud=<bb.fetch2.FetchData object at0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at0x7f8e26195890>, name='machine'):> output =
self._lsremote(ud, d, "")
# Tags of the form ^{} may not work, need to fallback to other form
File "bitbake/lib/bb/fetch2/git.py", line 833, in Git._lsremote(ud=<bb.fetch2.FetchData object at0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at0x7f8e26195890>, search=''):
bb.fetch2.check_network_access(d, cmd, repourl)
> output =runfetchcmd(cmd, d, True)if not output:
File "bitbake/lib/bb/fetch2/__init__.py", line 957, in runfetchcmd(cmd='export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64- linux/usr/bin:scripts:build/tmp/work/qemux86_64-poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64- poky-linux/linux-yocto/6.6.20+git/recipe- sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all ls-remote
https://git.yoctoproject.org/linux-yocto.git ',
d=<bb.data_smart.DataSmart object at0x7f8e26195890>, quiet=True, cleanup=[], log=None, workdir=None):> raise
FetchError(error_message)bb.data_smart.ExpansionError: Failure expanding variable fetcher_hashes_dummyfunc[vardepvalue], expression was ${@bb.fetch.get_hashvalue(d)} which triggered exception
FetchError: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64- linux/usr/bin:scripts:/var/home/krm/build/poky/build/tmp/work/q emux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64- poky-linux/linux-yocto/6.6.20+git/recipe- sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all ls-remote
https://git.yoctoproject.org/linux-yocto.git failed with exit code 128, output:
fatal: unable to access 'https://git.yoctoproject.org/linux-yocto.git/': error setting certificate file: build/tmp/work/x86_64-linux/curl- native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca- certificates.crtQuoted 2 lines
The variable dependency chain for the failure is:
fetcher_hashes_dummyfunc[vardepvalue]Note:
One gets out of that parser error by deleting the git binary in the
kernel's work recipe-sysroot-nativerm tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe- sysroot-native/usr/bin/gitBisecting poky leads to commit
f7fa98cca8("kern-tools: depend on git-replacement-native") Reverting it on top ofb5624ee564makes the parsing pass.I assume that
git-replacement-nativedoes not work with https, the
fetch error also goes away if changing inSRC_URIfrom https to git.Quoted 6 lines
Quoted 1 line
Any comments?
I didn't even know that curl was coming into play :)
Adding DEPENDS:class-native += "ca-certificates" to the curl recipe should resolve the issue.
Looks like curl-native resp. libcurl hardcodes the lookup to its own work directory, i.e.:
x86_64-linux/curl-native/8.6.0/recipe-sysroot- native/etc/ssl/certs/ca-certificates.crt
Quoted 7 lines
So even if DEPENDS/RDEPENDS will install ca-certificates in the kernel's
recipe-sysroot-native the parsing will fail if the curl-native directory
is not/no longer populated, e.g. because curl-native came from sstate orrm_workis in INHERIT.
This all gets a bit messy.
We've relied upon scripts that use openssl to set variables like:
export SSL_CERT_DIR="XXXX/etc/ssl/certs/
so in theory we might be able to set an environment variable in a wrapper around the git commands.
It may be better if we teach curl a relative path to the certs...
I suspect this isn't going to be an easy/neat fix unfortunately.
Cheers,
Richard
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 168 lines · Richard Purdie via lists.openembedded.org
Quoted 154 lines · Max
Am Samstag, dem 09.03.2024 um 13:04 -0800 schrieb Bruce Ashfield:
Quoted 139 lines · On Sat, Mar 9, 2024 at 12:58 PM <[email protected]> wrote:
Quoted 138 lines
From: Max Krummenacher <[email protected]>
Hello
If one builds a kernel using AUTOREV invoking bitbake only works
once.
Any subsequent bitbake invocation fails parsing the meta data.Reproducable with:
- latest poky,b5624ee564
- Kernel with SRCREV = "AUTOREV", e.g. in local.confSRCREV_machine:pn-linux-yocto:forcevariable = "${AUTOREV}"
- bitbake virtual/kernel; bitbake virtual/kernelOn the second invocation parsing fails when the fetcher tries to
evaluate the latest SRCREV:Quoted 97 lines
ERROR: ExpansionError during parsing meta/recipes-
kernel/linux/linux-yocto_6.6.bb
Traceback (most recent call last):
File "bitbake/lib/bb/fetch2/__init__.py", line 1245, in srcrev_internal_helper(ud=<bb.fetch2.FetchData object at0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at0x7f8e26195890>, name='machine'):
d.setVar("__BBAUTOREV_ACTED_UPON", True)> srcrev =ud.method.latest_revision(ud, d, name)File "
bitbake/lib/bb/fetch2/__init__.py", line 1667, in Git.latest_revision(ud=<bb.fetch2.FetchData object at0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at0x7f8e26195890>, name='machine'):
except KeyError:
> revs[key] = rev = self._latest_revision(ud, d,
name)
return rev
File "bitbake/lib/bb/fetch2/git.py", line 850, in Git._latest_revision(ud=<bb.fetch2.FetchData object at0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at0x7f8e26195890>, name='machine'):> output =
self._lsremote(ud, d, "")
# Tags of the form ^{} may not work, need to fallback to other form
File "bitbake/lib/bb/fetch2/git.py", line 833, in Git._lsremote(ud=<bb.fetch2.FetchData object at0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at0x7f8e26195890>, search=''):
bb.fetch2.check_network_access(d, cmd, repourl)
> output =runfetchcmd(cmd, d, True)if not output:
File "bitbake/lib/bb/fetch2/__init__.py", line 957, in runfetchcmd(cmd='export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export
PATH="build/tmp/sysroots-uninative/x86_64- linux/usr/bin:scripts:build/tmp/work/qemux86_64-poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64- poky-linux/linux-yocto/6.6.20+git/recipe- sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat
-c safe.bareRepository=all ls-remote
https://git.yoctoproject.org/linux-yocto.git ',
d=<bb.data_smart.DataSmart object at0x7f8e26195890>, quiet=True, cleanup=[], log=None, workdir=None):> raise
FetchError(error_message)bb.data_smart.ExpansionError: Failure expanding variable fetcher_hashes_dummyfunc[vardepvalue], expression was ${@bb.fetch.get_hashvalue(d)} which triggered exception
FetchError: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export
PATH="build/tmp/sysroots-uninative/x86_64- linux/usr/bin:scripts:/var/home/krm/build/poky/build/tmp/work /q emux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64- poky-linux/linux-yocto/6.6.20+git/recipe- sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat
-c safe.bareRepository=all ls-remote
https://git.yoctoproject.org/linux-yocto.git failed with exit
code 128, output:
fatal: unable to access 'https://git.yoctoproject.org/linux-yocto.git/': error setting
certificate file: build/tmp/work/x86_64-linux/curl- native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca- certificates.crtQuoted 2 lines
The variable dependency chain for the failure is:
fetcher_hashes_dummyfunc[vardepvalue]Note:
One gets out of that parser error by deleting the git binary in the
kernel's work recipe-sysroot-native
`rm tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-
sysroot-native/usr/bin/git`Bisecting poky leads to commit
f7fa98cca8("kern-tools: depend on git-replacement-native") Reverting it on top ofb5624ee564makes the parsing pass.I assume that
git-replacement-nativedoes not work with https,
the
fetch error also goes away if changing inSRC_URIfrom https to git.Quoted 7 lines
Quoted 1 line
Any comments?
I didn't even know that curl was coming into play :)
Adding DEPENDS:class-native += "ca-certificates" to the curl recipe
should resolve the issue.Looks like curl-native resp. libcurl hardcodes the lookup to its own
work directory, i.e.:
x86_64-linux/curl-native/8.6.0/recipe-sysroot- native/etc/ssl/certs/ca-certificates.crtQuoted 8 lines
So even if DEPENDS/RDEPENDS will install ca-certificates in the kernel's
recipe-sysroot-native the parsing will fail if the curl-native directory
is not/no longer populated, e.g. because curl-native came from sstate
orrm_workis in INHERIT.This all gets a bit messy.
We've relied upon scripts that use openssl to set variables like:
Quoted 8 lines
export SSL_CERT_DIR="XXXX/etc/ssl/certs/
so in theory we might be able to set an environment variable in a wrapper around the git commands.
It may be better if we teach curl a relative path to the certs...
I suspect this isn't going to be an easy/neat fix unfortunately.
I'm trying not to get sucked into "work" today however I realised that relative paths won't work without some implementation of "$ORIGIN" support into these paths. Given it ultimately ends up in openssl, it would probably be best there.
I'd not be against writing a $ORIGIN support patch and seeing what upstream think about it. It would still mean finding a way to find the path to the library file somehow.
For purposes of the release, setting the right envvars in the git wrapper is probably the way forward for now, much as I dislike the requirement to do that.
This does feel like a problem it would be good to solve for wider Linux in general for relocatable (and reproducible) binaries.
Cheers,
Richard
Quoted 185 lines · Richard Purdie
Quoted 168 lines · Richard Purdie via lists.openembedded.org
Quoted 154 lines · Max
Am Samstag, dem 09.03.2024 um 13:04 -0800 schrieb Bruce Ashfield:
Quoted 139 lines · On Sat, Mar 9, 2024 at 12:58 PM <[email protected]> wrote:
Quoted 138 lines
From: Max Krummenacher <[email protected]>
Hello
If one builds a kernel using AUTOREV invoking bitbake only works
once.
Any subsequent bitbake invocation fails parsing the meta data.Reproducable with:
- latest poky,b5624ee564
- Kernel with SRCREV = "AUTOREV", e.g. in local.confSRCREV_machine:pn-linux-yocto:forcevariable = "${AUTOREV}"
- bitbake virtual/kernel; bitbake virtual/kernelOn the second invocation parsing fails when the fetcher tries to
evaluate the latest SRCREV:Quoted 97 lines
ERROR: ExpansionError during parsing meta/recipes-
kernel/linux/linux-yocto_6.6.bb
Traceback (most recent call last):
File "bitbake/lib/bb/fetch2/__init__.py", line 1245, in
srcrev_internal_helper(ud=<bb.fetch2.FetchData object at0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at0x7f8e26195890>, name='machine'):
d.setVar("__BBAUTOREV_ACTED_UPON", True)
> srcrev =ud.method.latest_revision(ud, d, name)File "
bitbake/lib/bb/fetch2/__init__.py", line 1667, in
Git.latest_revision(ud=<bb.fetch2.FetchData object at0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at0x7f8e26195890>, name='machine'):
except KeyError:
> revs[key] = rev = self._latest_revision(ud,
d,
name)
return rev
File "bitbake/lib/bb/fetch2/git.py", line 850, in
Git._latest_revision(ud=<bb.fetch2.FetchData object at0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at0x7f8e26195890>, name='machine'):Quoted 1 line
output = self._lsremote(ud, d, "")# Tags of the form ^{} may not work, need to
fallback to other form
File "bitbake/lib/bb/fetch2/git.py", line 833, in
Git._lsremote(ud=<bb.fetch2.FetchData object at0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at0x7f8e26195890>, search=''):
bb.fetch2.check_network_access(d, cmd,
repourl)Quoted 1 line
output =
runfetchcmd(cmd, d, True)if not output:
File "bitbake/lib/bb/fetch2/__init__.py", line 957, in
runfetchcmd(cmd='export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export
PATH="build/tmp/sysroots-uninative/x86_64- linux/usr/bin:scripts:build/tmp/work/qemux86_64-poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64- poky-linux/linux-yocto/6.6.20+git/recipe- sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat
-c safe.bareRepository=all ls-remote
https://git.yoctoproject.org/linux-yocto.git ', d=<bb.data_smart.DataSmart object at0x7f8e26195890>, quiet=True, cleanup=[], log=None, workdir=None):Quoted 1 line
raise
FetchError(error_message)bb.data_smart.ExpansionError: Failure expanding variable fetcher_hashes_dummyfunc[vardepvalue], expression was ${@bb.fetch.get_hashvalue(d)} which triggered exception
FetchError: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export
PATH="build/tmp/sysroots-uninative/x86_64- linux/usr/bin:scripts:/var/home/krm/build/poky/build/tmp/work /q emux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64- poky-linux/linux-yocto/6.6.20+git/recipe- sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat
-c safe.bareRepository=all ls-remote
https://git.yoctoproject.org/linux-yocto.git failed with exit
code 128, output:
fatal: unable to access 'https://git.yoctoproject.org/linux-yocto.git/': error setting
certificate file: build/tmp/work/x86_64-linux/curl- native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca- certificates.crtQuoted 2 lines
The variable dependency chain for the failure is:
fetcher_hashes_dummyfunc[vardepvalue]Note:
One gets out of that parser error by deleting the git binary in the
kernel's work recipe-sysroot-native
`rm tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-
sysroot-native/usr/bin/git`Bisecting poky leads to commit
f7fa98cca8("kern-tools: depend on git-replacement-native") Reverting it on top ofb5624ee564makes the parsing pass.I assume that
git-replacement-nativedoes not work with https,
the
fetch error also goes away if changing inSRC_URIfrom https to git.Quoted 7 lines
Quoted 1 line
Any comments?
I didn't even know that curl was coming into play :)
Adding DEPENDS:class-native += "ca-certificates" to the curl recipe
should resolve the issue.Looks like curl-native resp. libcurl hardcodes the lookup to its own
work directory, i.e.:
x86_64-linux/curl-native/8.6.0/recipe-sysroot- native/etc/ssl/certs/ca-certificates.crtQuoted 8 lines
So even if DEPENDS/RDEPENDS will install ca-certificates in the kernel's
recipe-sysroot-native the parsing will fail if the curl-native directory
is not/no longer populated, e.g. because curl-native came from sstate
orrm_workis in INHERIT.This all gets a bit messy.
We've relied upon scripts that use openssl to set variables like:
Quoted 8 lines
export SSL_CERT_DIR="XXXX/etc/ssl/certs/
so in theory we might be able to set an environment variable in a wrapper around the git commands.
It may be better if we teach curl a relative path to the certs...
I suspect this isn't going to be an easy/neat fix unfortunately.
I'm trying not to get sucked into "work" today however I realised that relative paths won't work without some implementation of "$ORIGIN" support into these paths. Given it ultimately ends up in openssl, it would probably be best there.
Quoted 7 lines
I'd not be against writing a $ORIGIN support patch and seeing what upstream think about it. It would still mean finding a way to find the path to the library file somehow.
For purposes of the release, setting the right envvars in the git wrapper is probably the way forward for now, much as I dislike the requirement to do that.
I tried this, but curl didn't seem to use it to locate the cert file.
i.e., I tried this:
---------------
(quoted diff) +0 -0
b/meta/recipes-devtools/git/git_2.44.0.bb
index e6d1470873..f6b06ec601 100644
meta/recipes-devtools/git/git_2.44.0.bb +2 -2
@@ -31,7 +31,7 @@ PACKAGECONFIG ??= "expat curl" PACKAGECONFIG[cvsserver] = "" PACKAGECONFIG[svn] = "" PACKAGECONFIG[manpages] = ",,asciidoc-native xmlto-native" -PACKAGECONFIG[curl] = "--with-curl,--without-curl,curl" +PACKAGECONFIG[curl] = "--with-curl,--without-curl,curl ca-certificates" PACKAGECONFIG[expat] = "--with-expat,--without-expat,expat" EXTRA_OECONF = "--with-perl=${STAGING_BINDIR_NATIVE}/perl-native/perl \
@@ -103,13 +103,15 @@ do_install:append:class-target () { do_install:append:class-native() { create_wrapper ${D}${bindir}/git \ GIT_EXEC_PATH='`dirname $''realpath`'/${REL_GIT_EXEC_PATH} \ - GIT_TEMPLATE_DIR='`dirname $''realpath`'/${REL_GIT_TEMPLATE_DIR} + GIT_TEMPLATE_DIR='`dirname
$''realpath`'/${REL_GIT_TEMPLATE_DIR} \
+ SSL_CERT_DIR='dirname $''realpath'/../../etc/ssl/certs/
}
do_install:append:class-nativesdk() {
create_wrapper ${D}${bindir}/git \
GIT_EXEC_PATH='`dirname $''realpath`'/${REL_GIT_EXEC_PATH} \
GIT_TEMPLATE_DIR='`dirname $''realpath`'/${REL_GIT_TEMPLATE_DIR}+ SSL_CERT_DIR='dirname $''realpath'/../../etc/ssl/certs/
perl_native_fixup
}
---------
Probably because the exec of curl from git isn't getting the environment ? Either
that, or I did it wrong. I didn't try it as an explicit export, and that is probably it,
I can try that later tonight.
Only when I added the depends on ca-certifications to curl itself was it able to fetch the autorevs.
Bruce
Quoted 9 lines
This does feel like a problem it would be good to solve for wider Linux in general for relocatable (and reproducible) binaries.
Cheers,
Richard
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Quoted 272 lines · Bruce Ashfield
Quoted 207 lines · Richard Purdie
Quoted 188 lines · Richard Purdie via lists.openembedded.org
Quoted 174 lines · Max
Am Samstag, dem 09.03.2024 um 13:04 -0800 schrieb Bruce Ashfield:
Quoted 157 lines · On Sat, Mar 9, 2024 at 12:58 PM <[email protected]> wrote:
Quoted 156 lines
From: Max Krummenacher <[email protected]>
Hello
If one builds a kernel using AUTOREV invoking bitbake only works
once.
Any subsequent bitbake invocation fails parsing the meta data.Reproducable with:
- latest poky,b5624ee564
- Kernel with SRCREV = "AUTOREV", e.g. in local.confSRCREV_machine:pn-linux-yocto:forcevariable = "${AUTOREV}"
- bitbake virtual/kernel; bitbake virtual/kernelOn the second invocation parsing fails when the fetcher tries
to
evaluate the latest SRCREV:Quoted 110 lines
ERROR: ExpansionError during parsing meta/recipes-
kernel/linux/linux-yocto_6.6.bb
Traceback (most recent call last):
File "bitbake/lib/bb/fetch2/__init__.py", line 1245, in srcrev_internal_helper(ud=<bb.fetch2.FetchData object at0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at0x7f8e26195890>, name='machine'):
d.setVar("__BBAUTOREV_ACTED_UPON", True)> srcrev =ud.method.latest_revision(ud, d, name)File "
bitbake/lib/bb/fetch2/__init__.py", line 1667, in Git.latest_revision(ud=<bb.fetch2.FetchData object at0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at0x7f8e26195890>, name='machine'):
except KeyError:
> revs[key] = rev = self._latest_revision(ud,
d,
name)
return rev
File "bitbake/lib/bb/fetch2/git.py", line 850, in Git._latest_revision(ud=<bb.fetch2.FetchData object at0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at0x7f8e26195890>, name='machine'):> output =
self._lsremote(ud, d, "")# Tags of the form ^{} may not work, need to fallback to other form
File "bitbake/lib/bb/fetch2/git.py", line 833, in Git._lsremote(ud=<bb.fetch2.FetchData object at0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at0x7f8e26195890>, search=''):
bb.fetch2.check_network_access(d, cmd,
repourl)
> output =runfetchcmd(cmd, d, True)if not output:
File "bitbake/lib/bb/fetch2/__init__.py", line 957, in runfetchcmd(cmd='export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64- linux/usr/bin:scripts:build/tmp/work/qemux86_64-poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/bin/x86_64-poky- linux:build/tmp/work/qemux86_64- poky-linux/linux-yocto/6.6.20+git/recipe- sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64- poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/sbin:build/tmp/work/qemux86_64-poky- linux/linux-
yocto/6.6.20+git/recipe-sysroot- native/usr/bin:build/tmp/work/qemux86_64-poky- linux/linux-
yocto/6.6.20+git/recipe-sysroot- native/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat
-c safe.bareRepository=all ls-remote
https://git.yoctoproject.org/linux-yocto.git ',
d=<bb.data_smart.DataSmart object at0x7f8e26195890>, quiet=True, cleanup=[], log=None, workdir=None):> raise
FetchError(error_message)bb.data_smart.ExpansionError: Failure expanding variable fetcher_hashes_dummyfunc[vardepvalue], expression was ${@bb.fetch.get_hashvalue(d)} which triggered exception
FetchError: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64- linux/usr/bin:scripts:/var/home/krm/build/poky/build/tmp/ work
/q emux86_64-poky-linux/linux-yocto/6.6.20+git/recipe- sysroot-
native/usr/bin/x86_64-poky- linux:build/tmp/work/qemux86_64- poky-linux/linux-yocto/6.6.20+git/recipe- sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64- poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/sbin:build/tmp/work/qemux86_64-poky- linux/linux-
yocto/6.6.20+git/recipe-sysroot- native/usr/bin:build/tmp/work/qemux86_64-poky- linux/linux-
yocto/6.6.20+git/recipe-sysroot- native/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat
-c safe.bareRepository=all ls-remote
https://git.yoctoproject.org/linux-yocto.git failed with exit
code 128, output:
fatal: unable to access 'https://git.yoctoproject.org/linux-yocto.git/': error setting
certificate file: build/tmp/work/x86_64-linux/curl- native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca- certificates.crtQuoted 2 lines
The variable dependency chain for the failure is:
fetcher_hashes_dummyfunc[vardepvalue]Note:
One gets out of that parser error by deleting the git binary in
the
kernel's work recipe-sysroot-nativerm tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe- sysroot-native/usr/bin/gitBisecting poky leads to commit
f7fa98cca8("kern-tools: depend on git-replacement-native") Reverting it on top ofb5624ee564makes the parsing pass.I assume that
git-replacement-nativedoes not work with https,
the
fetch error also goes away if changing inSRC_URIfrom https to
git.Quoted 7 lines
Quoted 1 line
Any comments?
I didn't even know that curl was coming into play :)
Adding DEPENDS:class-native += "ca-certificates" to the curl recipe
should resolve the issue.Looks like curl-native resp. libcurl hardcodes the lookup to its
own
work directory, i.e.:
x86_64-linux/curl-native/8.6.0/recipe-sysroot- native/etc/ssl/certs/ca-certificates.crtQuoted 8 lines
So even if DEPENDS/RDEPENDS will install ca-certificates in the kernel's
recipe-sysroot-native the parsing will fail if the curl-native directory
is not/no longer populated, e.g. because curl-native came from sstate
orrm_workis in INHERIT.This all gets a bit messy.
We've relied upon scripts that use openssl to set variables like:
Quoted 8 lines
export SSL_CERT_DIR="XXXX/etc/ssl/certs/
so in theory we might be able to set an environment variable in a wrapper around the git commands.
It may be better if we teach curl a relative path to the certs...
I suspect this isn't going to be an easy/neat fix unfortunately.
I'm trying not to get sucked into "work" today however I realised that
relative paths won't work without some implementation of "$ORIGIN" support into these paths. Given it ultimately ends up in openssl, it
would probably be best there.Quoted 8 lines
I'd not be against writing a $ORIGIN support patch and seeing what upstream think about it. It would still mean finding a way to find the
path to the library file somehow.For purposes of the release, setting the right envvars in the git wrapper is probably the way forward for now, much as I dislike the requirement to do that.
I tried this, but curl didn't seem to use it to locate the cert file.
i.e., I tried this:
---------------
meta/recipes-devtools/git/git_2.44.0.bb +0
-0@@ -31,7 +31,7 @@ PACKAGECONFIG ??= "expat curl"PACKAGECONFIG[cvsserver] = ""
PACKAGECONFIG[svn] = ""
PACKAGECONFIG[manpages] = ",,asciidoc-native xmlto-native"meta/recipes-devtools/git/git_2.44.0.bb +1
-1-PACKAGECONFIG[curl] = "--with-curl,--without-curl,curl" +PACKAGECONFIG[curl] = "--with-curl,--without-curl,curl ca-certificates"
PACKAGECONFIG[expat] = "--with-expat,--without-expat,expat"EXTRA_OECONF = "--with-perl=${STAGING_BINDIR_NATIVE}/perl- native/perl \
meta/recipes-devtools/git/git_2.44.0.bb +0
-0@@ -103,13 +103,15 @@ do_install:append:class-target () { do_install:append:class-native() { create_wrapper ${D}${bindir}/git \GIT_EXEC_PATH='`dirname
$''realpath`'/${REL_GIT_EXEC_PATH} \
- GIT_TEMPLATE_DIR='`dirname
$''realpath`'/${REL_GIT_TEMPLATE_DIR}
+ GIT_TEMPLATE_DIR='`dirname
$''realpath`'/${REL_GIT_TEMPLATE_DIR} \
+ SSL_CERT_DIR='`dirname
$''realpath`'/../../etc/ssl/certs/
}do_install:append:class-nativesdk() {
create_wrapper${D}${bindir}/git \
GIT_EXEC_PATH='`dirname
$''realpath`'/${REL_GIT_EXEC_PATH} \
GIT_TEMPLATE_DIR='`dirname
$''realpath`'/${REL_GIT_TEMPLATE_DIR}
+ SSL_CERT_DIR='`dirname
$''realpath`'/../../etc/ssl/certs/
perl_native_fixup
}---------
Probably because the exec of curl from git isn't getting the environment ? Either
that, or I did it wrong. I didn't try it as an explicit export, and that is probably it,
I can try that later tonight.
Quoted 3 lines
Only when I added the depends on ca-certifications to curl itself was it able to
fetch the autorevs.
We probably need both the variable (to avoid the workdir removal issue) and the dependency.
Cheers,
Richard
Quoted 280 lines · Richard Purdie
Quoted 272 lines · Bruce Ashfield
Quoted 207 lines · Richard Purdie
Quoted 188 lines · Richard Purdie via lists.openembedded.org
Quoted 174 lines · Max
Am Samstag, dem 09.03.2024 um 13:04 -0800 schrieb Bruce Ashfield:
Quoted 157 lines · On Sat, Mar 9, 2024 at 12:58 PM <[email protected]> wrote:
Quoted 156 lines
From: Max Krummenacher <[email protected]>
Hello
If one builds a kernel using AUTOREV invoking bitbake only works
once.
Any subsequent bitbake invocation fails parsing the meta data.Reproducable with:
- latest poky,b5624ee564
- Kernel with SRCREV = "AUTOREV", e.g. in local.confSRCREV_machine:pn-linux-yocto:forcevariable = "${AUTOREV}"
- bitbake virtual/kernel; bitbake virtual/kernelOn the second invocation parsing fails when the fetcher tries
to
evaluate the latest SRCREV:Quoted 110 lines
ERROR: ExpansionError during parsing meta/recipes-
kernel/linux/linux-yocto_6.6.bb
Traceback (most recent call last):
File "bitbake/lib/bb/fetch2/__init__.py", line 1245, in srcrev_internal_helper(ud=<bb.fetch2.FetchData object at0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at0x7f8e26195890>, name='machine'):
d.setVar("__BBAUTOREV_ACTED_UPON", True)> srcrev =ud.method.latest_revision(ud, d, name)File "
bitbake/lib/bb/fetch2/__init__.py", line 1667, in Git.latest_revision(ud=<bb.fetch2.FetchData object at0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at0x7f8e26195890>, name='machine'):
except KeyError:
> revs[key] = rev = self._latest_revision(ud,
d,
name)
return rev
File "bitbake/lib/bb/fetch2/git.py", line 850, in Git._latest_revision(ud=<bb.fetch2.FetchData object at0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at0x7f8e26195890>, name='machine'):> output =
self._lsremote(ud, d, "")# Tags of the form ^{} may not work, need to fallback to other form
File "bitbake/lib/bb/fetch2/git.py", line 833, in Git._lsremote(ud=<bb.fetch2.FetchData object at0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at0x7f8e26195890>, search=''):
bb.fetch2.check_network_access(d, cmd,
repourl)
> output =runfetchcmd(cmd, d, True)if not output:
File "bitbake/lib/bb/fetch2/__init__.py", line 957, in runfetchcmd(cmd='export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64- linux/usr/bin:scripts:build/tmp/work/qemux86_64-poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/bin/x86_64-poky- linux:build/tmp/work/qemux86_64- poky-linux/linux-yocto/6.6.20+git/recipe- sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64- poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/sbin:build/tmp/work/qemux86_64-poky- linux/linux-
yocto/6.6.20+git/recipe-sysroot- native/usr/bin:build/tmp/work/qemux86_64-poky- linux/linux-
yocto/6.6.20+git/recipe-sysroot- native/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat
-c safe.bareRepository=all ls-remote
https://git.yoctoproject.org/linux-yocto.git ',
d=<bb.data_smart.DataSmart object at0x7f8e26195890>, quiet=True, cleanup=[], log=None, workdir=None):> raise
FetchError(error_message)bb.data_smart.ExpansionError: Failure expanding variable fetcher_hashes_dummyfunc[vardepvalue], expression was ${@bb.fetch.get_hashvalue(d)} which triggered exception
FetchError: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64- linux/usr/bin:scripts:/var/home/krm/build/poky/build/tmp/ work
/q emux86_64-poky-linux/linux-yocto/6.6.20+git/recipe- sysroot-
native/usr/bin/x86_64-poky- linux:build/tmp/work/qemux86_64- poky-linux/linux-yocto/6.6.20+git/recipe- sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64- poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/sbin:build/tmp/work/qemux86_64-poky- linux/linux-
yocto/6.6.20+git/recipe-sysroot- native/usr/bin:build/tmp/work/qemux86_64-poky- linux/linux-
yocto/6.6.20+git/recipe-sysroot- native/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat
-c safe.bareRepository=all ls-remote
https://git.yoctoproject.org/linux-yocto.git failed with exit
code 128, output:
fatal: unable to access 'https://git.yoctoproject.org/linux-yocto.git/': error setting
certificate file: build/tmp/work/x86_64-linux/curl- native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca- certificates.crtQuoted 2 lines
The variable dependency chain for the failure is:
fetcher_hashes_dummyfunc[vardepvalue]Note:
One gets out of that parser error by deleting the git binary in
the
kernel's work recipe-sysroot-nativerm tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe- sysroot-native/usr/bin/gitBisecting poky leads to commit
f7fa98cca8("kern-tools: depend on git-replacement-native") Reverting it on top ofb5624ee564makes the parsing pass.I assume that
git-replacement-nativedoes not work with https,
the
fetch error also goes away if changing inSRC_URIfrom https to
git.Quoted 7 lines
Quoted 1 line
Any comments?
I didn't even know that curl was coming into play :)
Adding DEPENDS:class-native += "ca-certificates" to the curl recipe
should resolve the issue.Looks like curl-native resp. libcurl hardcodes the lookup to its
own
work directory, i.e.:
x86_64-linux/curl-native/8.6.0/recipe-sysroot- native/etc/ssl/certs/ca-certificates.crtQuoted 8 lines
So even if DEPENDS/RDEPENDS will install ca-certificates in the kernel's
recipe-sysroot-native the parsing will fail if the curl-native directory
is not/no longer populated, e.g. because curl-native came from sstate
orrm_workis in INHERIT.This all gets a bit messy.
We've relied upon scripts that use openssl to set variables like:
Quoted 8 lines
export SSL_CERT_DIR="XXXX/etc/ssl/certs/
so in theory we might be able to set an environment variable in a wrapper around the git commands.
It may be better if we teach curl a relative path to the certs...
I suspect this isn't going to be an easy/neat fix unfortunately.
I'm trying not to get sucked into "work" today however I realised that
relative paths won't work without some implementation of "$ORIGIN" support into these paths. Given it ultimately ends up in openssl, it
would probably be best there.Quoted 8 lines
I'd not be against writing a $ORIGIN support patch and seeing what upstream think about it. It would still mean finding a way to find the
path to the library file somehow.For purposes of the release, setting the right envvars in the git wrapper is probably the way forward for now, much as I dislike the requirement to do that.
I tried this, but curl didn't seem to use it to locate the cert file.
i.e., I tried this:
---------------
meta/recipes-devtools/git/git_2.44.0.bb +0
-0@@ -31,7 +31,7 @@ PACKAGECONFIG ??= "expat curl"PACKAGECONFIG[cvsserver] = ""
PACKAGECONFIG[svn] = ""
PACKAGECONFIG[manpages] = ",,asciidoc-native xmlto-native"meta/recipes-devtools/git/git_2.44.0.bb +1
-1-PACKAGECONFIG[curl] = "--with-curl,--without-curl,curl" +PACKAGECONFIG[curl] = "--with-curl,--without-curl,curl ca-certificates"
PACKAGECONFIG[expat] = "--with-expat,--without-expat,expat"EXTRA_OECONF = "--with-perl=${STAGING_BINDIR_NATIVE}/perl- native/perl \
meta/recipes-devtools/git/git_2.44.0.bb +0
-0@@ -103,13 +103,15 @@ do_install:append:class-target () { do_install:append:class-native() { create_wrapper ${D}${bindir}/git \GIT_EXEC_PATH='`dirname
$''realpath`'/${REL_GIT_EXEC_PATH} \
- GIT_TEMPLATE_DIR='`dirname
$''realpath`'/${REL_GIT_TEMPLATE_DIR}
+ GIT_TEMPLATE_DIR='`dirname
$''realpath`'/${REL_GIT_TEMPLATE_DIR} \
+ SSL_CERT_DIR='`dirname
$''realpath`'/../../etc/ssl/certs/
}do_install:append:class-nativesdk() {
create_wrapper${D}${bindir}/git \
GIT_EXEC_PATH='`dirname
$''realpath`'/${REL_GIT_EXEC_PATH} \
GIT_TEMPLATE_DIR='`dirname
$''realpath`'/${REL_GIT_TEMPLATE_DIR}
+ SSL_CERT_DIR='`dirname
$''realpath`'/../../etc/ssl/certs/
perl_native_fixup
}---------
Probably because the exec of curl from git isn't getting the environment ? Either
that, or I did it wrong. I didn't try it as an explicit export, and that is probably it,
I can try that later tonight.Quoted 3 lines
Only when I added the depends on ca-certifications to curl itself was it able to
fetch the autorevs.We probably need both the variable (to avoid the workdir removal issue) and the dependency.
Quoted 4 lines
Cheers,
Richard
Looks like git has its own cainfo path env variable, SSL_CERT_DIR seems to not work. [1]
The following changes did make AUTOREV work for me:
meta/recipes-devtools/git/git_2.44.0.bb +3 -0
@@ -4,6 +4,7 @@ DESCRIPTION = "Git is a free and open source distributed version control system SECTION = "console/utils" LICENSE = "GPL-2.0-only & GPL-2.0-or-later & BSD-3-Clause & MIT & BSL-1.0 & LGPL-2.1-or-later" DEPENDS = "openssl zlib" +DEPENDS:class-native += "ca-certificates" PROVIDES:append:class-native = " git-replacement-native"
@@ -95,6 +96,7 @@ perl_native_fixup () { REL_GIT_EXEC_PATH = "${@os.path.relpath(libexecdir, bindir)}/git-core" REL_GIT_TEMPLATE_DIR = "${@os.path.relpath(datadir, bindir)}/git-core/templates" +REL_GIT_SSL_CAINFO = "${@os.path.relpath(sysconfdir, bindir)}/ssl/certs/ca-certificates.crt" do_install:append:class-target () { perl_native_fixup
@@ -103,6 +105,7 @@ do_install:append:class-target () { do_install:append:class-native() { create_wrapper ${D}${bindir}/git \ GIT_EXEC_PATH='`dirname $''realpath`'/${REL_GIT_EXEC_PATH} \ + GIT_SSL_CAINFO='`dirname $''realpath`'/${REL_GIT_SSL_CAINFO} \ GIT_TEMPLATE_DIR='`dirname $''realpath`'/${REL_GIT_TEMPLATE_DIR} }
if that is acceptable I could send a proper patch for review.
Max
[1] https://stackoverflow.com/questions/11621768/how-can-i-make-git-accept-a-self-signed-certificate
Quoted 282 lines · Max Krummenacher
Quoted 280 lines · Richard Purdie
Quoted 272 lines · Bruce Ashfield
Quoted 207 lines · Richard Purdie
Quoted 188 lines · Richard Purdie via lists.openembedded.org
Quoted 174 lines · Max
Am Samstag, dem 09.03.2024 um 13:04 -0800 schrieb Bruce Ashfield:
Quoted 157 lines · On Sat, Mar 9, 2024 at 12:58 PM <[email protected]> wrote:
Quoted 156 lines
From: Max Krummenacher <[email protected]>
Hello
If one builds a kernel using AUTOREV invoking bitbake only works
once.
Any subsequent bitbake invocation fails parsing the meta data.Reproducable with:
- latest poky,b5624ee564
- Kernel with SRCREV = "AUTOREV", e.g. in local.conf `SRCREV_machine:pn-linux-yocto:forcevariable =
"${AUTOREV}"`
- bitbake virtual/kernel; bitbake virtual/kernelOn the second invocation parsing fails when the fetcher tries
to
evaluate the latest SRCREV:Quoted 110 lines
ERROR: ExpansionError during parsing meta/recipes-
kernel/linux/linux-yocto_6.6.bb
Traceback (most recent call last):
File "bitbake/lib/bb/fetch2/__init__.py", line 1245, in
srcrev_internal_helper(ud=<bb.fetch2.FetchData object at0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at0x7f8e26195890>, name='machine'):
d.setVar("__BBAUTOREV_ACTED_UPON", True)
> srcrev = ud.method.latest_revision(ud, d,
name)File "
bitbake/lib/bb/fetch2/__init__.py", line 1667, in
Git.latest_revision(ud=<bb.fetch2.FetchData object at0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at0x7f8e26195890>, name='machine'):
except KeyError:
> revs[key] = rev =
self._latest_revision(ud,
d,
name)
return rev
File "bitbake/lib/bb/fetch2/git.py", line 850, in
Git._latest_revision(ud=<bb.fetch2.FetchData object at0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at0x7f8e26195890>, name='machine'):Quoted 1 line
output = self._lsremote(ud, d, "")# Tags of the form ^{} may not work, need to
fallback to other form
File "bitbake/lib/bb/fetch2/git.py", line 833, in
Git._lsremote(ud=<bb.fetch2.FetchData object at0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at0x7f8e26195890>, search=''):
bb.fetch2.check_network_access(d,
cmd,
repourl)Quoted 1 line
output =
runfetchcmd(cmd, d, True)if not output:
File "bitbake/lib/bb/fetch2/__init__.py", line 957, in
runfetchcmd(cmd='export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64- linux/usr/bin:scripts:build/tmp/work/qemux86_64-poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/bin/x86_64-poky- linux:build/tmp/work/qemux86_64- poky-linux/linux-yocto/6.6.20+git/recipe- sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64- poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/sbin:build/tmp/work/qemux86_64-poky- linux/linux-
yocto/6.6.20+git/recipe-sysroot- native/usr/bin:build/tmp/work/qemux86_64-poky- linux/linux-
yocto/6.6.20+git/recipe-sysroot- native/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat
-c safe.bareRepository=all ls-remote
https://git.yoctoproject.org/linux-yocto.git ', d=<bb.data_smart.DataSmart object at0x7f8e26195890>, quiet=True, cleanup=[], log=None, workdir=None):Quoted 1 line
raise
FetchError(error_message)bb.data_smart.ExpansionError: Failure expanding variable fetcher_hashes_dummyfunc[vardepvalue], expression was ${@bb.fetch.get_hashvalue(d)} which triggered exception
FetchError: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64- linux/usr/bin:scripts:/var/home/krm/build/poky/build/tmp/ work
/q emux86_64-poky-linux/linux-yocto/6.6.20+git/recipe- sysroot-
native/usr/bin/x86_64-poky- linux:build/tmp/work/qemux86_64- poky-linux/linux-yocto/6.6.20+git/recipe- sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64- poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/sbin:build/tmp/work/qemux86_64-poky- linux/linux-
yocto/6.6.20+git/recipe-sysroot- native/usr/bin:build/tmp/work/qemux86_64-poky- linux/linux-
yocto/6.6.20+git/recipe-sysroot- native/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat
-c safe.bareRepository=all ls-remote
https://git.yoctoproject.org/linux-yocto.git failed with exit
code 128, output:
fatal: unable to access 'https://git.yoctoproject.org/linux-yocto.git/': error setting
certificate file: build/tmp/work/x86_64-linux/curl- native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca- certificates.crtQuoted 2 lines
The variable dependency chain for the failure is:
fetcher_hashes_dummyfunc[vardepvalue]Note:
One gets out of that parser error by deleting the git binary in
the
kernel's work recipe-sysroot-nativerm tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe- sysroot-native/usr/bin/gitBisecting poky leads to commit
f7fa98cca8("kern-tools: depend on git-replacement-native") Reverting it on top ofb5624ee564makes the parsing pass.I assume that
git-replacement-nativedoes not work with https,
the
fetch error also goes away if changing inSRC_URIfrom https to
git.Quoted 7 lines
Quoted 1 line
Any comments?
I didn't even know that curl was coming into play :)
Adding DEPENDS:class-native += "ca-certificates" to the curl recipe
should resolve the issue.Looks like curl-native resp. libcurl hardcodes the lookup to its
own
work directory, i.e.:
x86_64-linux/curl-native/8.6.0/recipe-sysroot- native/etc/ssl/certs/ca-certificates.crtQuoted 8 lines
So even if DEPENDS/RDEPENDS will install ca-certificates in the kernel's
recipe-sysroot-native the parsing will fail if the curl-native directory
is not/no longer populated, e.g. because curl-native came from sstate
orrm_workis in INHERIT.This all gets a bit messy.
We've relied upon scripts that use openssl to set variables like:
Quoted 8 lines
export SSL_CERT_DIR="XXXX/etc/ssl/certs/
so in theory we might be able to set an environment variable in a wrapper around the git commands.
It may be better if we teach curl a relative path to the certs...
I suspect this isn't going to be an easy/neat fix unfortunately.
I'm trying not to get sucked into "work" today however I realised that
relative paths won't work without some implementation of "$ORIGIN" support into these paths. Given it ultimately ends up in openssl, it
would probably be best there.Quoted 8 lines
I'd not be against writing a $ORIGIN support patch and seeing what upstream think about it. It would still mean finding a way to find the
path to the library file somehow.For purposes of the release, setting the right envvars in the git wrapper is probably the way forward for now, much as I dislike the requirement to do that.
I tried this, but curl didn't seem to use it to locate the cert file.
i.e., I tried this:
---------------
meta/recipes-devtools/git/git_2.44.0.bb +1
-1@@ -31,7 +31,7 @@ PACKAGECONFIG ??= "expat curl" PACKAGECONFIG[cvsserver] = "" PACKAGECONFIG[svn] = "" PACKAGECONFIG[manpages] = ",,asciidoc-native xmlto-native" -PACKAGECONFIG[curl] = "--with-curl,--without-curl,curl" +PACKAGECONFIG[curl] = "--with-curl,--without-curl,curl ca-certificates"
PACKAGECONFIG[expat] = "--with-expat,--without-expat,expat"
EXTRA_OECONF= "--with-perl=${STAGING_BINDIR_NATIVE}/perl-
native/perl \meta/recipes-devtools/git/git_2.44.0.bb +2
-1@@ -103,13 +103,15 @@ do_install:append:class-target () { do_install:append:class-native() { create_wrapper ${D}${bindir}/git \ GIT_EXEC_PATH='`dirname $''realpath`'/${REL_GIT_EXEC_PATH} \ - GIT_TEMPLATE_DIR='`dirname $''realpath`'/${REL_GIT_TEMPLATE_DIR} + GIT_TEMPLATE_DIR='`dirname $''realpath`'/${REL_GIT_TEMPLATE_DIR} \ + SSL_CERT_DIR='`dirname$''realpath`'/../../etc/ssl/certs/
}do_install:append:class-nativesdk() { create_wrapper ${D}${bindir}/git \ GIT_EXEC_PATH='`dirname$''realpath`'/${REL_GIT_EXEC_PATH} \
GIT_TEMPLATE_DIR='`dirname
$''realpath`'/${REL_GIT_TEMPLATE_DIR}
+ SSL_CERT_DIR='`dirname
$''realpath`'/../../etc/ssl/certs/
perl_native_fixup
}---------
Probably because the exec of curl from git isn't getting the environment ? Either
that, or I did it wrong. I didn't try it as an explicit export, and that is probably it,
I can try that later tonight.Quoted 3 lines
Only when I added the depends on ca-certifications to curl itself was it able to
fetch the autorevs.We probably need both the variable (to avoid the workdir removal issue) and the dependency.
Quoted 8 lines
Quoted 4 lines
Cheers,
Richard
Looks like git has its own cainfo path env variable,
SSL_CERT_DIRseems to not work. [1]
Aha! That explains why it was ignoring my attempts :)
Quoted 28 lines
The following changes did make AUTOREV work for me:
meta/recipes-devtools/git/git_2.44.0.bb +3
-0@@ -4,6 +4,7 @@ DESCRIPTION = "Git is a free and open source distributed version control system SECTION = "console/utils" LICENSE = "GPL-2.0-only & GPL-2.0-or-later & BSD-3-Clause & MIT & BSL-1.0 & LGPL-2.1-or-later" DEPENDS = "openssl zlib" +DEPENDS:class-native += "ca-certificates" PROVIDES:append:class-native = " git-replacement-native"@@ -95,6 +96,7 @@ perl_native_fixup () { REL_GIT_EXEC_PATH = "${@os.path.relpath(libexecdir, bindir)}/git-core" REL_GIT_TEMPLATE_DIR = "${@os.path.relpath(datadir, bindir)}/git-core/templates" +REL_GIT_SSL_CAINFO = "${@os.path.relpath(sysconfdir, bindir)}/ssl/certs/ca-certificates.crt" do_install:append:class-target () { perl_native_fixup@@ -103,6 +105,7 @@ do_install:append:class-target () { do_install:append:class-native() { create_wrapper ${D}${bindir}/git \ GIT_EXEC_PATH='`dirname $''realpath`'/${REL_GIT_EXEC_PATH} \ + GIT_SSL_CAINFO='`dirname $''realpath`'/${REL_GIT_SSL_CAINFO} \ GIT_TEMPLATE_DIR='`dirname $''realpath`'/${REL_GIT_TEMPLATE_DIR} }
Quoted 1 line
if that is acceptable I could send a proper patch for review.
It looks fine to me. It follows the structure of the other variables.
I don't see any reason for a conditional or packageconfig dependency on ca-certificates
(since https isn't exactly uncommon), and keeping everything contained within the git
recipe is better this close to release.
That being said, I still wonder / think that curl should also just provide its own dependency on ca-certificates, since it is the application that is actually looking for them. I don't know if this way we'll run into some sort of issue with them not being available after sstate, cleanup or something else.
Definitely send it for review, hopefully those that are well versed in those questions I have will see it and comment :)
Bruce
Quoted 4 lines
Max
[1] https://stackoverflow.com/questions/11621768/how-can-i-make-git-accept-a-self-signed-certificate
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
PATCH 1/1 git: git-replacement-native: depend on ca-certificate
From: Max Krummenacher <[email protected]>
git is delegating webacces for URLs using TLS to libcurl. However our native libcurl build does not find a ca-certificate.crt unless its curl-native work dir still exists and thus git will fail.
If a recipe uses AUTOREV with a git repo using https as its protocol parsing of that recipe will fail fetching the latest HEAD.
Fix that by depending on ca-certificate and give its location to libcurl via git's envrironment variable GIT_SSL_CAINFO.
--- meta/recipes-devtools/git/git_2.44.0.bb | 3 +++ 1 file changed, 3 insertions(+)
See also the ML discussion:
https://lore.kernel.org/all/[email protected]/
meta/recipes-devtools/git/git_2.44.0.bb +3 -0
@@ -4,6 +4,7 @@ DESCRIPTION = "Git is a free and open source distributed version control system SECTION = "console/utils" LICENSE = "GPL-2.0-only & GPL-2.0-or-later & BSD-3-Clause & MIT & BSL-1.0 & LGPL-2.1-or-later" DEPENDS = "openssl zlib" +DEPENDS:class-native += "ca-certificates" PROVIDES:append:class-native = " git-replacement-native"
@@ -95,6 +96,7 @@ perl_native_fixup () { REL_GIT_EXEC_PATH = "${@os.path.relpath(libexecdir, bindir)}/git-core" REL_GIT_TEMPLATE_DIR = "${@os.path.relpath(datadir, bindir)}/git-core/templates" +REL_GIT_SSL_CAINFO = "${@os.path.relpath(sysconfdir, bindir)}/ssl/certs/ca-certificates.crt" do_install:append:class-target () { perl_native_fixup
@@ -103,6 +105,7 @@ do_install:append:class-target () { do_install:append:class-native() { create_wrapper ${D}${bindir}/git \ GIT_EXEC_PATH='`dirname $''realpath`'/${REL_GIT_EXEC_PATH} \ + GIT_SSL_CAINFO='`dirname $''realpath`'/${REL_GIT_SSL_CAINFO} \ GIT_TEMPLATE_DIR='`dirname $''realpath`'/${REL_GIT_TEMPLATE_DIR} }
Signature · 1 line
2.42.0