Skip to contentSkip to conversation

linux (git/curl-native) and autorev

REGRESSION New [email protected] 3 participants 10 messages Mar 9, 2024 – Mar 11, 2024 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

From: Max Krummenacher <[email protected]>

Hello

If one builds a kernel using AUTOREV invoking bitbake only works once. Any subsequent bitbake invocation fails parsing the meta data.

Reproducable with:
- latest poky, b5624ee564
- Kernel with SRCREV = "AUTOREV", e.g. in local.conf SRCREV_machine:pn-linux-yocto:forcevariable = "${AUTOREV}"
- bitbake virtual/kernel; bitbake virtual/kernel

On the second invocation parsing fails when the fetcher tries to evaluate the latest SRCREV:

| ERROR: ExpansionError during parsing meta/recipes-kernel/linux/linux-yocto_6.6.bb
| Traceback (most recent call last):
|   File "bitbake/lib/bb/fetch2/__init__.py", line 1245, in srcrev_internal_helper(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
|              d.setVar("__BBAUTOREV_ACTED_UPON", True)
|     >        srcrev = ud.method.latest_revision(ud, d, name)
|      
|   File "bitbake/lib/bb/fetch2/__init__.py", line 1667, in Git.latest_revision(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
|              except KeyError:
|     >            revs[key] = rev = self._latest_revision(ud, d, name)
|                  return rev
|   File "bitbake/lib/bb/fetch2/git.py", line 850, in Git._latest_revision(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
|      
|     >        output = self._lsremote(ud, d, "")
|              # Tags of the form ^{} may not work, need to fallback to other form
|   File "bitbake/lib/bb/fetch2/git.py", line 833, in Git._lsremote(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, search=''):
|                      bb.fetch2.check_network_access(d, cmd, repourl)
|     >            output = runfetchcmd(cmd, d, True)
|                  if not output:
|   File "bitbake/lib/bb/fetch2/__init__.py", line 957, in runfetchcmd(cmd='export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64-linux/usr/bin:scripts:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all ls-remote https://git.yoctoproject.org/linux-yocto.git ', d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, quiet=True, cleanup=[], log=None, workdir=None):
|      
|     >        raise FetchError(error_message)
|      
| bb.data_smart.ExpansionError: Failure expanding variable fetcher_hashes_dummyfunc[vardepvalue], expression was ${@bb.fetch.get_hashvalue(d)} which triggered exception FetchError: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64-linux/usr/bin:scripts:/var/home/krm/build/poky/build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all ls-remote https://git.yoctoproject.org/linux-yocto.git  failed with exit code 128, output:
| fatal: unable to access 'https://git.yoctoproject.org/linux-yocto.git/': error setting certificate file: build/tmp/work/x86_64-linux/curl-native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca-certificates.crt
| 
| The variable dependency chain for the failure is: fetcher_hashes_dummyfunc[vardepvalue]

Note:
One gets out of that parser error by deleting the git binary in the
kernel's work recipe-sysroot-native
rm tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin/git

Bisecting poky leads to commit
f7fa98cca8 ("kern-tools: depend on git-replacement-native") Reverting it on top of b5624ee564 makes the parsing pass.

I assume that git-replacement-native does not work with https, the fetch error also goes away if changing in SRC_URI from https to git.

Any comments?

Cheers
Max

#1 lore raw
Quoted 49 lines · On Sat, Mar 9, 2024 at 12:58 PM <[email protected]> wrote:

From: Max Krummenacher <[email protected]>

Hello

If one builds a kernel using AUTOREV invoking bitbake only works once. Any subsequent bitbake invocation fails parsing the meta data.

Reproducable with:
- latest poky, b5624ee564
- Kernel with SRCREV = "AUTOREV", e.g. in local.conf SRCREV_machine:pn-linux-yocto:forcevariable = "${AUTOREV}"
- bitbake virtual/kernel; bitbake virtual/kernel

On the second invocation parsing fails when the fetcher tries to evaluate the latest SRCREV:

| ERROR: ExpansionError during parsing meta/recipes-kernel/linux/linux-yocto_6.6.bb
| Traceback (most recent call last):
|   File "bitbake/lib/bb/fetch2/__init__.py", line 1245, in srcrev_internal_helper(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
|              d.setVar("__BBAUTOREV_ACTED_UPON", True)
|     >        srcrev = ud.method.latest_revision(ud, d, name)
|
|   File "bitbake/lib/bb/fetch2/__init__.py", line 1667, in Git.latest_revision(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
|              except KeyError:
|     >            revs[key] = rev = self._latest_revision(ud, d, name)
|                  return rev
|   File "bitbake/lib/bb/fetch2/git.py", line 850, in Git._latest_revision(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
|
|     >        output = self._lsremote(ud, d, "")
|              # Tags of the form ^{} may not work, need to fallback to other form
|   File "bitbake/lib/bb/fetch2/git.py", line 833, in Git._lsremote(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, search=''):
|                      bb.fetch2.check_network_access(d, cmd, repourl)
|     >            output = runfetchcmd(cmd, d, True)
|                  if not output:
|   File "bitbake/lib/bb/fetch2/__init__.py", line 957, in runfetchcmd(cmd='export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64-linux/usr/bin:scripts:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all ls-remote https://git.yoctoproject.org/linux-yocto.git ', d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, quiet=True, cleanup=[], log=None, workdir=None):
|
|     >        raise FetchError(error_message)
|
| bb.data_smart.ExpansionError: Failure expanding variable fetcher_hashes_dummyfunc[vardepvalue], expression was ${@bb.fetch.get_hashvalue(d)} which triggered exception FetchError: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64-linux/usr/bin:scripts:/var/home/krm/build/poky/build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all ls-remote https://git.yoctoproject.org/linux-yocto.git  failed with exit code 128, output:
| fatal: unable to access 'https://git.yoctoproject.org/linux-yocto.git/': error setting certificate file: build/tmp/work/x86_64-linux/curl-native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca-certificates.crt
|
| The variable dependency chain for the failure is: fetcher_hashes_dummyfunc[vardepvalue]

Note:
One gets out of that parser error by deleting the git binary in the
kernel's work recipe-sysroot-native
rm tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin/git

Quoted 8 lines

Bisecting poky leads to commit
f7fa98cca8 ("kern-tools: depend on git-replacement-native") Reverting it on top of b5624ee564 makes the parsing pass.

I assume that git-replacement-native does not work with https, the fetch error also goes away if changing in SRC_URI from https to git.

Any comments?

I didn't even know that curl was coming into play :)

Adding DEPENDS:class-native += "ca-certificates" to the curl recipe should resolve the issue.

I also tried exporting the SSL_CERT_DIR in the git-native wrapper, but that didn't help
since of course it is curl that is looking for the certs.

I also thought we could export the SSL_CERT_DIR from within the kern-tools, but really,
it does seem like curl should know that this is a dependency when used as a native
tool (it already has ca-certficates as a RRECOMMENDS).

I haven't done a patch for it, because the intricacies of the native classes and where
we should actually add that depends is someone lost on me.

Richard will likely know the right place to fix it .. and then I'm happy to spin a patch.

Bruce

Quoted 3 lines

Cheers
Max

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

author
#2 lore raw

Am Samstag, dem 09.03.2024 um 13:04 -0800 schrieb Bruce Ashfield:

Quoted 57 lines · On Sat, Mar 9, 2024 at 12:58 PM <[email protected]> wrote:
Quoted 56 lines

From: Max Krummenacher <[email protected]>

Hello

If one builds a kernel using AUTOREV invoking bitbake only works once. Any subsequent bitbake invocation fails parsing the meta data.

Reproducable with:
- latest poky, b5624ee564
- Kernel with SRCREV = "AUTOREV", e.g. in local.conf SRCREV_machine:pn-linux-yocto:forcevariable = "${AUTOREV}"
- bitbake virtual/kernel; bitbake virtual/kernel

On the second invocation parsing fails when the fetcher tries to evaluate the latest SRCREV:

Quoted 20 lines

ERROR: ExpansionError during parsing meta/recipes-kernel/linux/linux-yocto_6.6.bb
Traceback (most recent call last):
File "bitbake/lib/bb/fetch2/__init__.py", line 1245, in srcrev_internal_helper(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
d.setVar("__BBAUTOREV_ACTED_UPON", True)
> srcrev = ud.method.latest_revision(ud, d, name)

File "bitbake/lib/bb/fetch2/__init__.py", line 1667, in Git.latest_revision(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
except KeyError:
> revs[key] = rev = self._latest_revision(ud, d, name)
return rev
File "bitbake/lib/bb/fetch2/git.py", line 850, in Git._latest_revision(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):

Quoted 1 line
       output = self._lsremote(ud, d, "")

# Tags of the form ^{} may not work, need to fallback to other form
File "bitbake/lib/bb/fetch2/git.py", line 833, in Git._lsremote(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, search=''):
bb.fetch2.check_network_access(d, cmd, repourl)
> output = runfetchcmd(cmd, d, True)
if not output:
File "bitbake/lib/bb/fetch2/__init__.py", line 957, in runfetchcmd(cmd='export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64-linux/usr/bin:scripts:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all ls-remote https://git.yoctoproject.org/linux-yocto.git ', d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, quiet=True, cleanup=[], log=None, workdir=None):

Quoted 6 lines
Quoted 1 line

raise FetchError(error_message)

bb.data_smart.ExpansionError: Failure expanding variable fetcher_hashes_dummyfunc[vardepvalue], expression was ${@bb.fetch.get_hashvalue(d)} which triggered exception FetchError: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64-linux/usr/bin:scripts:/var/home/krm/build/poky/build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all ls-remote https://git.yoctoproject.org/linux-yocto.git failed with exit code 128, output:
fatal: unable to access 'https://git.yoctoproject.org/linux-yocto.git/': error setting certificate file: build/tmp/work/x86_64-linux/curl-native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca-certificates.crt

The variable dependency chain for the failure is: fetcher_hashes_dummyfunc[vardepvalue]

Note:
One gets out of that parser error by deleting the git binary in the
kernel's work recipe-sysroot-native
rm tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-native/usr/bin/git

Bisecting poky leads to commit
f7fa98cca8 ("kern-tools: depend on git-replacement-native") Reverting it on top of b5624ee564 makes the parsing pass.

I assume that git-replacement-native does not work with https, the fetch error also goes away if changing in SRC_URI from https to git.

Quoted 6 lines
Quoted 1 line

Any comments?

I didn't even know that curl was coming into play :)

Adding DEPENDS:class-native += "ca-certificates" to the curl recipe should resolve the issue.

Looks like curl-native resp. libcurl hardcodes the lookup to its own work directory, i.e.:
x86_64-linux/curl-native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca-certificates.crt

So even if DEPENDS/RDEPENDS will install ca-certificates in the kernel's recipe-sysroot-native the parsing will fail if the curl-native directory is not/no longer populated, e.g. because curl-native came from sstate or rm_work is in INHERIT.

Max

Quoted 39 lines

I also tried exporting the SSL_CERT_DIR in the git-native wrapper, but that didn't help
since of course it is curl that is looking for the certs.

I also thought we could export the SSL_CERT_DIR from within the kern-tools, but really,
it does seem like curl should know that this is a dependency when used as a native
tool (it already has ca-certficates as a RRECOMMENDS).

I haven't done a patch for it, because the intricacies of the native classes and where
we should actually add that depends is someone lost on me.

Richard will likely know the right place to fix it .. and then I'm happy to spin a patch.

Bruce

Quoted 3 lines

Cheers
Max

--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#196873): https://lists.openembedded.org/g/openembedded-core/message/196873
Mute This Topic: https://lists.openembedded.org/mt/104831542/3617484
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#3 lore raw
Quoted 140 lines · Max

Am Samstag, dem 09.03.2024 um 13:04 -0800 schrieb Bruce Ashfield:

Quoted 127 lines · On Sat, Mar 9, 2024 at 12:58 PM <[email protected]> wrote:
Quoted 126 lines

From: Max Krummenacher <[email protected]>

Hello

If one builds a kernel using AUTOREV invoking bitbake only works once.
Any subsequent bitbake invocation fails parsing the meta data.

Reproducable with:
- latest poky, b5624ee564
- Kernel with SRCREV = "AUTOREV", e.g. in local.conf SRCREV_machine:pn-linux-yocto:forcevariable = "${AUTOREV}"
- bitbake virtual/kernel; bitbake virtual/kernel

On the second invocation parsing fails when the fetcher tries to evaluate the latest SRCREV:

Quoted 89 lines

ERROR: ExpansionError during parsing meta/recipes- kernel/linux/linux-yocto_6.6.bb
Traceback (most recent call last):
  File "bitbake/lib/bb/fetch2/__init__.py", line 1245, in srcrev_internal_helper(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
             d.setVar("__BBAUTOREV_ACTED_UPON", True) >        srcrev = ud.method.latest_revision(ud, d, name)

  File "bitbake/lib/bb/fetch2/__init__.py", line 1667, in Git.latest_revision(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
             except KeyError:
    >            revs[key] = rev = self._latest_revision(ud, d, name)
                 return rev
  File "bitbake/lib/bb/fetch2/git.py", line 850, in Git._latest_revision(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):

    >        output = self._lsremote(ud, d, "")
             # Tags of the form ^{} may not work, need to fallback to other form
  File "bitbake/lib/bb/fetch2/git.py", line 833, in Git._lsremote(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, search=''):
                     bb.fetch2.check_network_access(d, cmd, repourl)
    >            output = runfetchcmd(cmd, d, True) if not output:
  File "bitbake/lib/bb/fetch2/__init__.py", line 957, in runfetchcmd(cmd='export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64- linux/usr/bin:scripts:build/tmp/work/qemux86_64-poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64- poky-linux/linux-yocto/6.6.20+git/recipe- sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all ls-remote
https://git.yoctoproject.org/linux-yocto.git ',
d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, quiet=True, cleanup=[], log=None, workdir=None):

    >        raise FetchError(error_message)

bb.data_smart.ExpansionError: Failure expanding variable fetcher_hashes_dummyfunc[vardepvalue], expression was ${@bb.fetch.get_hashvalue(d)} which triggered exception
FetchError: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64- linux/usr/bin:scripts:/var/home/krm/build/poky/build/tmp/work/q emux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64- poky-linux/linux-yocto/6.6.20+git/recipe- sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all ls-remote
https://git.yoctoproject.org/linux-yocto.git  failed with exit code 128, output:
fatal: unable to access 'https://git.yoctoproject.org/linux-yocto.git/': error setting certificate file: build/tmp/work/x86_64-linux/curl- native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca- certificates.crt

Quoted 2 lines

The variable dependency chain for the failure is:
fetcher_hashes_dummyfunc[vardepvalue]

Note:
One gets out of that parser error by deleting the git binary in the
kernel's work recipe-sysroot-native
rm tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.20+git/recipe- sysroot-native/usr/bin/git

Bisecting poky leads to commit
f7fa98cca8 ("kern-tools: depend on git-replacement-native") Reverting it on top of b5624ee564 makes the parsing pass.

I assume that git-replacement-native does not work with https, the
fetch error also goes away if changing in SRC_URI from https to git.

Quoted 6 lines
Quoted 1 line

Any comments?

I didn't even know that curl was coming into play :)

Adding DEPENDS:class-native += "ca-certificates" to the curl recipe should resolve the issue.

Looks like curl-native resp. libcurl hardcodes the lookup to its own work directory, i.e.:
x86_64-linux/curl-native/8.6.0/recipe-sysroot- native/etc/ssl/certs/ca-certificates.crt

Quoted 7 lines

So even if DEPENDS/RDEPENDS will install ca-certificates in the kernel's
recipe-sysroot-native the parsing will fail if the curl-native directory
is not/no longer populated, e.g. because curl-native came from sstate or
rm_work is in INHERIT.

This all gets a bit messy.

We've relied upon scripts that use openssl to set variables like:

export SSL_CERT_DIR="XXXX/etc/ssl/certs/

so in theory we might be able to set an environment variable in a wrapper around the git commands.

It may be better if we teach curl a relative path to the certs...

I suspect this isn't going to be an easy/neat fix unfortunately.

Cheers,

Richard

A message that is not in the archive (sent off-list or not yet received): [email protected]

#5 lore raw
Quoted 168 lines · Richard Purdie via lists.openembedded.org
Quoted 154 lines · Max

Am Samstag, dem 09.03.2024 um 13:04 -0800 schrieb Bruce Ashfield:

Quoted 139 lines · On Sat, Mar 9, 2024 at 12:58 PM <[email protected]> wrote:
Quoted 138 lines

From: Max Krummenacher <[email protected]>

Hello

If one builds a kernel using AUTOREV invoking bitbake only works
once.
Any subsequent bitbake invocation fails parsing the meta data.

Reproducable with:
- latest poky, b5624ee564
- Kernel with SRCREV = "AUTOREV", e.g. in local.conf SRCREV_machine:pn-linux-yocto:forcevariable = "${AUTOREV}"
- bitbake virtual/kernel; bitbake virtual/kernel

On the second invocation parsing fails when the fetcher tries to
evaluate the latest SRCREV:

Quoted 97 lines

ERROR: ExpansionError during parsing meta/recipes- kernel/linux/linux-yocto_6.6.bb
Traceback (most recent call last):
  File "bitbake/lib/bb/fetch2/__init__.py", line 1245, in srcrev_internal_helper(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
             d.setVar("__BBAUTOREV_ACTED_UPON", True) >        srcrev = ud.method.latest_revision(ud, d, name)

  File "bitbake/lib/bb/fetch2/__init__.py", line 1667, in Git.latest_revision(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
             except KeyError:
    >            revs[key] = rev = self._latest_revision(ud, d,
name)
                 return rev
  File "bitbake/lib/bb/fetch2/git.py", line 850, in Git._latest_revision(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):

    >        output = self._lsremote(ud, d, "")
             # Tags of the form ^{} may not work, need to fallback to other form
  File "bitbake/lib/bb/fetch2/git.py", line 833, in Git._lsremote(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, search=''):
                     bb.fetch2.check_network_access(d, cmd, repourl)
    >            output = runfetchcmd(cmd, d, True) if not output:
  File "bitbake/lib/bb/fetch2/__init__.py", line 957, in runfetchcmd(cmd='export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export
PATH="build/tmp/sysroots-uninative/x86_64- linux/usr/bin:scripts:build/tmp/work/qemux86_64-poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64- poky-linux/linux-yocto/6.6.20+git/recipe- sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat
-c safe.bareRepository=all ls-remote
https://git.yoctoproject.org/linux-yocto.git ',
d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, quiet=True, cleanup=[], log=None, workdir=None):

    >        raise FetchError(error_message)

bb.data_smart.ExpansionError: Failure expanding variable fetcher_hashes_dummyfunc[vardepvalue], expression was ${@bb.fetch.get_hashvalue(d)} which triggered exception
FetchError: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export
PATH="build/tmp/sysroots-uninative/x86_64- linux/usr/bin:scripts:/var/home/krm/build/poky/build/tmp/work /q emux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64- poky-linux/linux-yocto/6.6.20+git/recipe- sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat
-c safe.bareRepository=all ls-remote
https://git.yoctoproject.org/linux-yocto.git  failed with exit
code 128, output:
fatal: unable to access 'https://git.yoctoproject.org/linux-yocto.git/': error setting
certificate file: build/tmp/work/x86_64-linux/curl- native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca- certificates.crt

Quoted 2 lines

The variable dependency chain for the failure is:
fetcher_hashes_dummyfunc[vardepvalue]

Note:
One gets out of that parser error by deleting the git binary in the
kernel's work recipe-sysroot-native
`rm tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-
sysroot-native/usr/bin/git`

Bisecting poky leads to commit
f7fa98cca8 ("kern-tools: depend on git-replacement-native") Reverting it on top of b5624ee564 makes the parsing pass.

I assume that git-replacement-native does not work with https,
the
fetch error also goes away if changing in SRC_URI from https to git.

Quoted 7 lines
Quoted 1 line

Any comments?

I didn't even know that curl was coming into play :)

Adding DEPENDS:class-native += "ca-certificates" to the curl recipe
should resolve the issue.

Looks like curl-native resp. libcurl hardcodes the lookup to its own
work directory, i.e.:
x86_64-linux/curl-native/8.6.0/recipe-sysroot- native/etc/ssl/certs/ca-certificates.crt

Quoted 8 lines

So even if DEPENDS/RDEPENDS will install ca-certificates in the kernel's
recipe-sysroot-native the parsing will fail if the curl-native directory
is not/no longer populated, e.g. because curl-native came from sstate
or
rm_work is in INHERIT.

This all gets a bit messy.

We've relied upon scripts that use openssl to set variables like:

Quoted 8 lines

export SSL_CERT_DIR="XXXX/etc/ssl/certs/

so in theory we might be able to set an environment variable in a wrapper around the git commands.

It may be better if we teach curl a relative path to the certs...

I suspect this isn't going to be an easy/neat fix unfortunately.

I'm trying not to get sucked into "work" today however I realised that relative paths won't work without some implementation of "$ORIGIN" support into these paths. Given it ultimately ends up in openssl, it would probably be best there.

I'd not be against writing a $ORIGIN support patch and seeing what upstream think about it. It would still mean finding a way to find the path to the library file somehow.

For purposes of the release, setting the right envvars in the git wrapper is probably the way forward for now, much as I dislike the requirement to do that.

This does feel like a problem it would be good to solve for wider Linux in general for relocatable (and reproducible) binaries.

Cheers,

Richard

#6 lore raw
Quoted 185 lines · Richard Purdie
Quoted 168 lines · Richard Purdie via lists.openembedded.org
Quoted 154 lines · Max

Am Samstag, dem 09.03.2024 um 13:04 -0800 schrieb Bruce Ashfield:

Quoted 139 lines · On Sat, Mar 9, 2024 at 12:58 PM <[email protected]> wrote:
Quoted 138 lines

From: Max Krummenacher <[email protected]>

Hello

If one builds a kernel using AUTOREV invoking bitbake only works
once.
Any subsequent bitbake invocation fails parsing the meta data.

Reproducable with:
- latest poky, b5624ee564
- Kernel with SRCREV = "AUTOREV", e.g. in local.conf SRCREV_machine:pn-linux-yocto:forcevariable = "${AUTOREV}"
- bitbake virtual/kernel; bitbake virtual/kernel

On the second invocation parsing fails when the fetcher tries to
evaluate the latest SRCREV:

Quoted 97 lines

ERROR: ExpansionError during parsing meta/recipes- kernel/linux/linux-yocto_6.6.bb
Traceback (most recent call last):
File "bitbake/lib/bb/fetch2/__init__.py", line 1245, in
srcrev_internal_helper(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
d.setVar("__BBAUTOREV_ACTED_UPON", True)
> srcrev = ud.method.latest_revision(ud, d, name)

File "bitbake/lib/bb/fetch2/__init__.py", line 1667, in
Git.latest_revision(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
except KeyError:
> revs[key] = rev = self._latest_revision(ud,
d,
name)
return rev
File "bitbake/lib/bb/fetch2/git.py", line 850, in
Git._latest_revision(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):

Quoted 1 line
       output = self._lsremote(ud, d, "")

# Tags of the form ^{} may not work, need to
fallback to other form
File "bitbake/lib/bb/fetch2/git.py", line 833, in
Git._lsremote(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, search=''):
bb.fetch2.check_network_access(d, cmd,
repourl)

Quoted 1 line

output = runfetchcmd(cmd, d, True)

if not output:
File "bitbake/lib/bb/fetch2/__init__.py", line 957, in
runfetchcmd(cmd='export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export
PATH="build/tmp/sysroots-uninative/x86_64- linux/usr/bin:scripts:build/tmp/work/qemux86_64-poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64- poky-linux/linux-yocto/6.6.20+git/recipe- sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat
-c safe.bareRepository=all ls-remote
https://git.yoctoproject.org/linux-yocto.git ', d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, quiet=True, cleanup=[], log=None, workdir=None):

Quoted 1 line

raise FetchError(error_message)

bb.data_smart.ExpansionError: Failure expanding variable fetcher_hashes_dummyfunc[vardepvalue], expression was ${@bb.fetch.get_hashvalue(d)} which triggered exception
FetchError: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export
PATH="build/tmp/sysroots-uninative/x86_64- linux/usr/bin:scripts:/var/home/krm/build/poky/build/tmp/work /q emux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64- poky-linux/linux-yocto/6.6.20+git/recipe- sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat
-c safe.bareRepository=all ls-remote
https://git.yoctoproject.org/linux-yocto.git failed with exit
code 128, output:
fatal: unable to access 'https://git.yoctoproject.org/linux-yocto.git/': error setting
certificate file: build/tmp/work/x86_64-linux/curl- native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca- certificates.crt

Quoted 2 lines

The variable dependency chain for the failure is:
fetcher_hashes_dummyfunc[vardepvalue]

Note:
One gets out of that parser error by deleting the git binary in the
kernel's work recipe-sysroot-native
`rm tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-
sysroot-native/usr/bin/git`

Bisecting poky leads to commit
f7fa98cca8 ("kern-tools: depend on git-replacement-native") Reverting it on top of b5624ee564 makes the parsing pass.

I assume that git-replacement-native does not work with https,
the
fetch error also goes away if changing in SRC_URI from https to git.

Quoted 7 lines
Quoted 1 line

Any comments?

I didn't even know that curl was coming into play :)

Adding DEPENDS:class-native += "ca-certificates" to the curl recipe
should resolve the issue.

Looks like curl-native resp. libcurl hardcodes the lookup to its own
work directory, i.e.:
x86_64-linux/curl-native/8.6.0/recipe-sysroot- native/etc/ssl/certs/ca-certificates.crt

Quoted 8 lines

So even if DEPENDS/RDEPENDS will install ca-certificates in the kernel's
recipe-sysroot-native the parsing will fail if the curl-native directory
is not/no longer populated, e.g. because curl-native came from sstate
or
rm_work is in INHERIT.

This all gets a bit messy.

We've relied upon scripts that use openssl to set variables like:

Quoted 8 lines

export SSL_CERT_DIR="XXXX/etc/ssl/certs/

so in theory we might be able to set an environment variable in a wrapper around the git commands.

It may be better if we teach curl a relative path to the certs...

I suspect this isn't going to be an easy/neat fix unfortunately.

I'm trying not to get sucked into "work" today however I realised that relative paths won't work without some implementation of "$ORIGIN" support into these paths. Given it ultimately ends up in openssl, it would probably be best there.

Quoted 7 lines

I'd not be against writing a $ORIGIN support patch and seeing what upstream think about it. It would still mean finding a way to find the path to the library file somehow.

For purposes of the release, setting the right envvars in the git wrapper is probably the way forward for now, much as I dislike the requirement to do that.

I tried this, but curl didn't seem to use it to locate the cert file.

i.e., I tried this:

---------------

(quoted diff) +0 -0

b/meta/recipes-devtools/git/git_2.44.0.bb
index e6d1470873..f6b06ec601 100644

meta/recipes-devtools/git/git_2.44.0.bb +2 -2
@@ -31,7 +31,7 @@ PACKAGECONFIG ??= "expat curl"
 PACKAGECONFIG[cvsserver] = ""
 PACKAGECONFIG[svn] = ""
 PACKAGECONFIG[manpages] = ",,asciidoc-native xmlto-native"
-PACKAGECONFIG[curl] = "--with-curl,--without-curl,curl"
+PACKAGECONFIG[curl] = "--with-curl,--without-curl,curl ca-certificates"
 PACKAGECONFIG[expat] = "--with-expat,--without-expat,expat"
 
 EXTRA_OECONF = "--with-perl=${STAGING_BINDIR_NATIVE}/perl-native/perl \
@@ -103,13 +103,15 @@ do_install:append:class-target () {
 do_install:append:class-native() {
        create_wrapper ${D}${bindir}/git \
                GIT_EXEC_PATH='`dirname $''realpath`'/${REL_GIT_EXEC_PATH} \
-               GIT_TEMPLATE_DIR='`dirname $''realpath`'/${REL_GIT_TEMPLATE_DIR}
+               GIT_TEMPLATE_DIR='`dirname

$''realpath`'/${REL_GIT_TEMPLATE_DIR} \
+ SSL_CERT_DIR='dirname $''realpath'/../../etc/ssl/certs/
}

 do_install:append:class-nativesdk() {
        create_wrapper ${D}${bindir}/git \
                GIT_EXEC_PATH='`dirname $''realpath`'/${REL_GIT_EXEC_PATH} \
                GIT_TEMPLATE_DIR='`dirname $''realpath`'/${REL_GIT_TEMPLATE_DIR}

+ SSL_CERT_DIR='dirname $''realpath'/../../etc/ssl/certs/
perl_native_fixup
}

---------

Probably because the exec of curl from git isn't getting the environment ? Either
that, or I did it wrong. I didn't try it as an explicit export, and that is probably it,
I can try that later tonight.

Only when I added the depends on ca-certifications to curl itself was it able to fetch the autorevs.

Bruce

Quoted 9 lines

This does feel like a problem it would be good to solve for wider Linux in general for relocatable (and reproducible) binaries.

Cheers,

Richard

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#7 lore raw
Quoted 272 lines · Bruce Ashfield
Quoted 207 lines · Richard Purdie
Quoted 188 lines · Richard Purdie via lists.openembedded.org
Quoted 174 lines · Max

Am Samstag, dem 09.03.2024 um 13:04 -0800 schrieb Bruce Ashfield:

Quoted 157 lines · On Sat, Mar 9, 2024 at 12:58 PM <[email protected]> wrote:
Quoted 156 lines

From: Max Krummenacher <[email protected]>

Hello

If one builds a kernel using AUTOREV invoking bitbake only works
once.
Any subsequent bitbake invocation fails parsing the meta data.

Reproducable with:
- latest poky, b5624ee564
- Kernel with SRCREV = "AUTOREV", e.g. in local.conf SRCREV_machine:pn-linux-yocto:forcevariable = "${AUTOREV}"
- bitbake virtual/kernel; bitbake virtual/kernel

On the second invocation parsing fails when the fetcher tries
to
evaluate the latest SRCREV:

Quoted 110 lines

ERROR: ExpansionError during parsing meta/recipes- kernel/linux/linux-yocto_6.6.bb
Traceback (most recent call last):
  File "bitbake/lib/bb/fetch2/__init__.py", line 1245, in srcrev_internal_helper(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
             d.setVar("__BBAUTOREV_ACTED_UPON", True) >        srcrev = ud.method.latest_revision(ud, d, name)

  File "bitbake/lib/bb/fetch2/__init__.py", line 1667, in Git.latest_revision(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
             except KeyError:
    >            revs[key] = rev = self._latest_revision(ud,
d,
name)
                 return rev
  File "bitbake/lib/bb/fetch2/git.py", line 850, in Git._latest_revision(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):

    >        output = self._lsremote(ud, d, "") # Tags of the form ^{} may not work, need to fallback to other form
  File "bitbake/lib/bb/fetch2/git.py", line 833, in Git._lsremote(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, search=''):
                     bb.fetch2.check_network_access(d, cmd,
repourl)
    >            output = runfetchcmd(cmd, d, True) if not output:
  File "bitbake/lib/bb/fetch2/__init__.py", line 957, in runfetchcmd(cmd='export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64- linux/usr/bin:scripts:build/tmp/work/qemux86_64-poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/bin/x86_64-poky- linux:build/tmp/work/qemux86_64- poky-linux/linux-yocto/6.6.20+git/recipe- sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64- poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/sbin:build/tmp/work/qemux86_64-poky- linux/linux-
yocto/6.6.20+git/recipe-sysroot- native/usr/bin:build/tmp/work/qemux86_64-poky- linux/linux-
yocto/6.6.20+git/recipe-sysroot- native/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat
-c safe.bareRepository=all ls-remote
https://git.yoctoproject.org/linux-yocto.git ',
d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, quiet=True, cleanup=[], log=None, workdir=None):

    >        raise FetchError(error_message)

bb.data_smart.ExpansionError: Failure expanding variable fetcher_hashes_dummyfunc[vardepvalue], expression was ${@bb.fetch.get_hashvalue(d)} which triggered exception
FetchError: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64- linux/usr/bin:scripts:/var/home/krm/build/poky/build/tmp/ work
/q emux86_64-poky-linux/linux-yocto/6.6.20+git/recipe- sysroot-
native/usr/bin/x86_64-poky- linux:build/tmp/work/qemux86_64- poky-linux/linux-yocto/6.6.20+git/recipe- sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64- poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/sbin:build/tmp/work/qemux86_64-poky- linux/linux-
yocto/6.6.20+git/recipe-sysroot- native/usr/bin:build/tmp/work/qemux86_64-poky- linux/linux-
yocto/6.6.20+git/recipe-sysroot- native/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat
-c safe.bareRepository=all ls-remote
https://git.yoctoproject.org/linux-yocto.git  failed with exit
code 128, output:
fatal: unable to access 'https://git.yoctoproject.org/linux-yocto.git/': error setting
certificate file: build/tmp/work/x86_64-linux/curl- native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca- certificates.crt

Quoted 2 lines

The variable dependency chain for the failure is:
fetcher_hashes_dummyfunc[vardepvalue]

Note:
One gets out of that parser error by deleting the git binary in
the
kernel's work recipe-sysroot-native
rm tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe- sysroot-native/usr/bin/git

Bisecting poky leads to commit
f7fa98cca8 ("kern-tools: depend on git-replacement-native") Reverting it on top of b5624ee564 makes the parsing pass.

I assume that git-replacement-native does not work with https,
the
fetch error also goes away if changing in SRC_URI from https to
git.

Quoted 7 lines
Quoted 1 line

Any comments?

I didn't even know that curl was coming into play :)

Adding DEPENDS:class-native += "ca-certificates" to the curl recipe
should resolve the issue.

Looks like curl-native resp. libcurl hardcodes the lookup to its
own
work directory, i.e.:
x86_64-linux/curl-native/8.6.0/recipe-sysroot- native/etc/ssl/certs/ca-certificates.crt

Quoted 8 lines

So even if DEPENDS/RDEPENDS will install ca-certificates in the kernel's
recipe-sysroot-native the parsing will fail if the curl-native directory
is not/no longer populated, e.g. because curl-native came from sstate
or
rm_work is in INHERIT.

This all gets a bit messy.

We've relied upon scripts that use openssl to set variables like:

Quoted 8 lines

export SSL_CERT_DIR="XXXX/etc/ssl/certs/

so in theory we might be able to set an environment variable in a wrapper around the git commands.

It may be better if we teach curl a relative path to the certs...

I suspect this isn't going to be an easy/neat fix unfortunately.

I'm trying not to get sucked into "work" today however I realised that
relative paths won't work without some implementation of "$ORIGIN" support into these paths. Given it ultimately ends up in openssl, it
would probably be best there.

Quoted 8 lines

I'd not be against writing a $ORIGIN support patch and seeing what upstream think about it. It would still mean finding a way to find the
path to the library file somehow.

For purposes of the release, setting the right envvars in the git wrapper is probably the way forward for now, much as I dislike the requirement to do that.

I tried this, but curl didn't seem to use it to locate the cert file.

i.e., I tried this:

---------------

meta/recipes-devtools/git/git_2.44.0.bb +0 -0
index e6d1470873..f6b06ec601 100644
@@ -31,7 +31,7 @@ PACKAGECONFIG ??= "expat curl"

 PACKAGECONFIG[cvsserver] = ""
 PACKAGECONFIG[svn] = ""
 PACKAGECONFIG[manpages] = ",,asciidoc-native xmlto-native"

meta/recipes-devtools/git/git_2.44.0.bb +1 -1
-PACKAGECONFIG[curl] = "--with-curl,--without-curl,curl"
+PACKAGECONFIG[curl] = "--with-curl,--without-curl,curl ca-

certificates"
 PACKAGECONFIG[expat] = "--with-expat,--without-expat,expat"

 EXTRA_OECONF = "--with-perl=${STAGING_BINDIR_NATIVE}/perl- native/perl \

meta/recipes-devtools/git/git_2.44.0.bb +0 -0
@@ -103,13 +103,15 @@ do_install:append:class-target () {
  do_install:append:class-native() {
         create_wrapper ${D}${bindir}/git \

                GIT_EXEC_PATH='`dirname
$''realpath`'/${REL_GIT_EXEC_PATH} \
-               GIT_TEMPLATE_DIR='`dirname
$''realpath`'/${REL_GIT_TEMPLATE_DIR}
+               GIT_TEMPLATE_DIR='`dirname
$''realpath`'/${REL_GIT_TEMPLATE_DIR} \
+               SSL_CERT_DIR='`dirname
$''realpath`'/../../etc/ssl/certs/
 }

 do_install:append:class-nativesdk() {
        create_wrapper ${D}${bindir}/git \
                GIT_EXEC_PATH='`dirname
$''realpath`'/${REL_GIT_EXEC_PATH} \
                GIT_TEMPLATE_DIR='`dirname
$''realpath`'/${REL_GIT_TEMPLATE_DIR}
+               SSL_CERT_DIR='`dirname
$''realpath`'/../../etc/ssl/certs/
        perl_native_fixup
 }

---------

Probably because the exec of curl from git isn't getting the environment ? Either
that, or I did it wrong. I didn't try it as an explicit export, and that is probably it,
I can try that later tonight.

Quoted 3 lines

Only when I added the depends on ca-certifications to curl itself was it able to
fetch the autorevs.

We probably need both the variable (to avoid the workdir removal issue) and the dependency.

Cheers,

Richard

#8 lore raw
Quoted 280 lines · Richard Purdie
Quoted 272 lines · Bruce Ashfield
Quoted 207 lines · Richard Purdie
Quoted 188 lines · Richard Purdie via lists.openembedded.org
Quoted 174 lines · Max

Am Samstag, dem 09.03.2024 um 13:04 -0800 schrieb Bruce Ashfield:

Quoted 157 lines · On Sat, Mar 9, 2024 at 12:58 PM <[email protected]> wrote:
Quoted 156 lines

From: Max Krummenacher <[email protected]>

Hello

If one builds a kernel using AUTOREV invoking bitbake only works
once.
Any subsequent bitbake invocation fails parsing the meta data.

Reproducable with:
- latest poky, b5624ee564
- Kernel with SRCREV = "AUTOREV", e.g. in local.conf SRCREV_machine:pn-linux-yocto:forcevariable = "${AUTOREV}"
- bitbake virtual/kernel; bitbake virtual/kernel

On the second invocation parsing fails when the fetcher tries
to
evaluate the latest SRCREV:

Quoted 110 lines

ERROR: ExpansionError during parsing meta/recipes- kernel/linux/linux-yocto_6.6.bb
Traceback (most recent call last):
  File "bitbake/lib/bb/fetch2/__init__.py", line 1245, in srcrev_internal_helper(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
             d.setVar("__BBAUTOREV_ACTED_UPON", True) >        srcrev = ud.method.latest_revision(ud, d, name)

  File "bitbake/lib/bb/fetch2/__init__.py", line 1667, in Git.latest_revision(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
             except KeyError:
    >            revs[key] = rev = self._latest_revision(ud,
d,
name)
                 return rev
  File "bitbake/lib/bb/fetch2/git.py", line 850, in Git._latest_revision(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):

    >        output = self._lsremote(ud, d, "") # Tags of the form ^{} may not work, need to fallback to other form
  File "bitbake/lib/bb/fetch2/git.py", line 833, in Git._lsremote(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, search=''):
                     bb.fetch2.check_network_access(d, cmd,
repourl)
    >            output = runfetchcmd(cmd, d, True) if not output:
  File "bitbake/lib/bb/fetch2/__init__.py", line 957, in runfetchcmd(cmd='export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64- linux/usr/bin:scripts:build/tmp/work/qemux86_64-poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/bin/x86_64-poky- linux:build/tmp/work/qemux86_64- poky-linux/linux-yocto/6.6.20+git/recipe- sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64- poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/sbin:build/tmp/work/qemux86_64-poky- linux/linux-
yocto/6.6.20+git/recipe-sysroot- native/usr/bin:build/tmp/work/qemux86_64-poky- linux/linux-
yocto/6.6.20+git/recipe-sysroot- native/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat
-c safe.bareRepository=all ls-remote
https://git.yoctoproject.org/linux-yocto.git ',
d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, quiet=True, cleanup=[], log=None, workdir=None):

    >        raise FetchError(error_message)

bb.data_smart.ExpansionError: Failure expanding variable fetcher_hashes_dummyfunc[vardepvalue], expression was ${@bb.fetch.get_hashvalue(d)} which triggered exception
FetchError: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64- linux/usr/bin:scripts:/var/home/krm/build/poky/build/tmp/ work
/q emux86_64-poky-linux/linux-yocto/6.6.20+git/recipe- sysroot-
native/usr/bin/x86_64-poky- linux:build/tmp/work/qemux86_64- poky-linux/linux-yocto/6.6.20+git/recipe- sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64- poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/sbin:build/tmp/work/qemux86_64-poky- linux/linux-
yocto/6.6.20+git/recipe-sysroot- native/usr/bin:build/tmp/work/qemux86_64-poky- linux/linux-
yocto/6.6.20+git/recipe-sysroot- native/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat
-c safe.bareRepository=all ls-remote
https://git.yoctoproject.org/linux-yocto.git  failed with exit
code 128, output:
fatal: unable to access 'https://git.yoctoproject.org/linux-yocto.git/': error setting
certificate file: build/tmp/work/x86_64-linux/curl- native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca- certificates.crt

Quoted 2 lines

The variable dependency chain for the failure is:
fetcher_hashes_dummyfunc[vardepvalue]

Note:
One gets out of that parser error by deleting the git binary in
the
kernel's work recipe-sysroot-native
rm tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe- sysroot-native/usr/bin/git

Bisecting poky leads to commit
f7fa98cca8 ("kern-tools: depend on git-replacement-native") Reverting it on top of b5624ee564 makes the parsing pass.

I assume that git-replacement-native does not work with https,
the
fetch error also goes away if changing in SRC_URI from https to
git.

Quoted 7 lines
Quoted 1 line

Any comments?

I didn't even know that curl was coming into play :)

Adding DEPENDS:class-native += "ca-certificates" to the curl recipe
should resolve the issue.

Looks like curl-native resp. libcurl hardcodes the lookup to its
own
work directory, i.e.:
x86_64-linux/curl-native/8.6.0/recipe-sysroot- native/etc/ssl/certs/ca-certificates.crt

Quoted 8 lines

So even if DEPENDS/RDEPENDS will install ca-certificates in the kernel's
recipe-sysroot-native the parsing will fail if the curl-native directory
is not/no longer populated, e.g. because curl-native came from sstate
or
rm_work is in INHERIT.

This all gets a bit messy.

We've relied upon scripts that use openssl to set variables like:

Quoted 8 lines

export SSL_CERT_DIR="XXXX/etc/ssl/certs/

so in theory we might be able to set an environment variable in a wrapper around the git commands.

It may be better if we teach curl a relative path to the certs...

I suspect this isn't going to be an easy/neat fix unfortunately.

I'm trying not to get sucked into "work" today however I realised that
relative paths won't work without some implementation of "$ORIGIN" support into these paths. Given it ultimately ends up in openssl, it
would probably be best there.

Quoted 8 lines

I'd not be against writing a $ORIGIN support patch and seeing what upstream think about it. It would still mean finding a way to find the
path to the library file somehow.

For purposes of the release, setting the right envvars in the git wrapper is probably the way forward for now, much as I dislike the requirement to do that.

I tried this, but curl didn't seem to use it to locate the cert file.

i.e., I tried this:

---------------

meta/recipes-devtools/git/git_2.44.0.bb +0 -0
index e6d1470873..f6b06ec601 100644
@@ -31,7 +31,7 @@ PACKAGECONFIG ??= "expat curl"

 PACKAGECONFIG[cvsserver] = ""
 PACKAGECONFIG[svn] = ""
 PACKAGECONFIG[manpages] = ",,asciidoc-native xmlto-native"

meta/recipes-devtools/git/git_2.44.0.bb +1 -1
-PACKAGECONFIG[curl] = "--with-curl,--without-curl,curl"
+PACKAGECONFIG[curl] = "--with-curl,--without-curl,curl ca-

certificates"
 PACKAGECONFIG[expat] = "--with-expat,--without-expat,expat"

 EXTRA_OECONF = "--with-perl=${STAGING_BINDIR_NATIVE}/perl- native/perl \

meta/recipes-devtools/git/git_2.44.0.bb +0 -0
@@ -103,13 +103,15 @@ do_install:append:class-target () {
  do_install:append:class-native() {
         create_wrapper ${D}${bindir}/git \

                GIT_EXEC_PATH='`dirname
$''realpath`'/${REL_GIT_EXEC_PATH} \
-               GIT_TEMPLATE_DIR='`dirname
$''realpath`'/${REL_GIT_TEMPLATE_DIR}
+               GIT_TEMPLATE_DIR='`dirname
$''realpath`'/${REL_GIT_TEMPLATE_DIR} \
+               SSL_CERT_DIR='`dirname
$''realpath`'/../../etc/ssl/certs/
 }

 do_install:append:class-nativesdk() {
        create_wrapper ${D}${bindir}/git \
                GIT_EXEC_PATH='`dirname
$''realpath`'/${REL_GIT_EXEC_PATH} \
                GIT_TEMPLATE_DIR='`dirname
$''realpath`'/${REL_GIT_TEMPLATE_DIR}
+               SSL_CERT_DIR='`dirname
$''realpath`'/../../etc/ssl/certs/
        perl_native_fixup
 }

---------

Probably because the exec of curl from git isn't getting the environment ? Either
that, or I did it wrong. I didn't try it as an explicit export, and that is probably it,
I can try that later tonight.

Quoted 3 lines

Only when I added the depends on ca-certifications to curl itself was it able to
fetch the autorevs.

We probably need both the variable (to avoid the workdir removal issue) and the dependency.

Quoted 4 lines

Cheers,

Richard

Looks like git has its own cainfo path env variable, SSL_CERT_DIR seems to not work. [1]

The following changes did make AUTOREV work for me:

meta/recipes-devtools/git/git_2.44.0.bb +3 -0
@@ -4,6 +4,7 @@ DESCRIPTION = "Git is a free and open source distributed version control system
 SECTION = "console/utils"
 LICENSE = "GPL-2.0-only & GPL-2.0-or-later & BSD-3-Clause & MIT & BSL-1.0 & LGPL-2.1-or-later"
 DEPENDS = "openssl zlib"
+DEPENDS:class-native += "ca-certificates"
 
 PROVIDES:append:class-native = " git-replacement-native"
 
@@ -95,6 +96,7 @@ perl_native_fixup () {
 
 REL_GIT_EXEC_PATH = "${@os.path.relpath(libexecdir, bindir)}/git-core"
 REL_GIT_TEMPLATE_DIR = "${@os.path.relpath(datadir, bindir)}/git-core/templates"
+REL_GIT_SSL_CAINFO = "${@os.path.relpath(sysconfdir, bindir)}/ssl/certs/ca-certificates.crt"
 
 do_install:append:class-target () {
 	perl_native_fixup
@@ -103,6 +105,7 @@ do_install:append:class-target () {
 do_install:append:class-native() {
 	create_wrapper ${D}${bindir}/git \
 		GIT_EXEC_PATH='`dirname $''realpath`'/${REL_GIT_EXEC_PATH} \
+		GIT_SSL_CAINFO='`dirname $''realpath`'/${REL_GIT_SSL_CAINFO} \
 		GIT_TEMPLATE_DIR='`dirname $''realpath`'/${REL_GIT_TEMPLATE_DIR}
 }
 

if that is acceptable I could send a proper patch for review.

Max

[1] https://stackoverflow.com/questions/11621768/how-can-i-make-git-accept-a-self-signed-certificate

#9 lore raw
Quoted 282 lines · Max Krummenacher
Quoted 280 lines · Richard Purdie
Quoted 272 lines · Bruce Ashfield
Quoted 207 lines · Richard Purdie
Quoted 188 lines · Richard Purdie via lists.openembedded.org
Quoted 174 lines · Max

Am Samstag, dem 09.03.2024 um 13:04 -0800 schrieb Bruce Ashfield:

Quoted 157 lines · On Sat, Mar 9, 2024 at 12:58 PM <[email protected]> wrote:
Quoted 156 lines

From: Max Krummenacher <[email protected]>

Hello

If one builds a kernel using AUTOREV invoking bitbake only works
once.
Any subsequent bitbake invocation fails parsing the meta data.

Reproducable with:
- latest poky, b5624ee564
- Kernel with SRCREV = "AUTOREV", e.g. in local.conf `SRCREV_machine:pn-linux-yocto:forcevariable =
"${AUTOREV}"`
- bitbake virtual/kernel; bitbake virtual/kernel

On the second invocation parsing fails when the fetcher tries
to
evaluate the latest SRCREV:

Quoted 110 lines

ERROR: ExpansionError during parsing meta/recipes- kernel/linux/linux-yocto_6.6.bb
Traceback (most recent call last):
File "bitbake/lib/bb/fetch2/__init__.py", line 1245, in
srcrev_internal_helper(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
d.setVar("__BBAUTOREV_ACTED_UPON", True)
> srcrev = ud.method.latest_revision(ud, d,
name)

File "bitbake/lib/bb/fetch2/__init__.py", line 1667, in
Git.latest_revision(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):
except KeyError:
> revs[key] = rev =
self._latest_revision(ud,
d,
name)
return rev
File "bitbake/lib/bb/fetch2/git.py", line 850, in
Git._latest_revision(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, name='machine'):

Quoted 1 line
       output = self._lsremote(ud, d, "")

# Tags of the form ^{} may not work, need to
fallback to other form
File "bitbake/lib/bb/fetch2/git.py", line 833, in
Git._lsremote(ud=<bb.fetch2.FetchData object at 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, search=''):
bb.fetch2.check_network_access(d,
cmd,
repourl)

Quoted 1 line

output = runfetchcmd(cmd, d, True)

if not output:
File "bitbake/lib/bb/fetch2/__init__.py", line 957, in
runfetchcmd(cmd='export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64- linux/usr/bin:scripts:build/tmp/work/qemux86_64-poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/bin/x86_64-poky- linux:build/tmp/work/qemux86_64- poky-linux/linux-yocto/6.6.20+git/recipe- sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64- poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/sbin:build/tmp/work/qemux86_64-poky- linux/linux-
yocto/6.6.20+git/recipe-sysroot- native/usr/bin:build/tmp/work/qemux86_64-poky- linux/linux-
yocto/6.6.20+git/recipe-sysroot- native/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat
-c safe.bareRepository=all ls-remote
https://git.yoctoproject.org/linux-yocto.git ', d=<bb.data_smart.DataSmart object at 0x7f8e26195890>, quiet=True, cleanup=[], log=None, workdir=None):

Quoted 1 line

raise FetchError(error_message)

bb.data_smart.ExpansionError: Failure expanding variable fetcher_hashes_dummyfunc[vardepvalue], expression was ${@bb.fetch.get_hashvalue(d)} which triggered exception
FetchError: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus"; export PATH="build/tmp/sysroots-uninative/x86_64- linux/usr/bin:scripts:/var/home/krm/build/poky/build/tmp/ work
/q emux86_64-poky-linux/linux-yocto/6.6.20+git/recipe- sysroot-
native/usr/bin/x86_64-poky- linux:build/tmp/work/qemux86_64- poky-linux/linux-yocto/6.6.20+git/recipe- sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64- poky- linux/linux-yocto/6.6.20+git/recipe-sysroot- native/usr/sbin:build/tmp/work/qemux86_64-poky- linux/linux-
yocto/6.6.20+git/recipe-sysroot- native/usr/bin:build/tmp/work/qemux86_64-poky- linux/linux-
yocto/6.6.20+git/recipe-sysroot- native/sbin:build/tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe-sysroot- native/bin:bitbake/bin:build/tmp/hosttools"; export HOME="/home/krm"; git -c gc.autoDetach=false -c core.pager=cat
-c safe.bareRepository=all ls-remote
https://git.yoctoproject.org/linux-yocto.git failed with exit
code 128, output:
fatal: unable to access 'https://git.yoctoproject.org/linux-yocto.git/': error setting
certificate file: build/tmp/work/x86_64-linux/curl- native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca- certificates.crt

Quoted 2 lines

The variable dependency chain for the failure is:
fetcher_hashes_dummyfunc[vardepvalue]

Note:
One gets out of that parser error by deleting the git binary in
the
kernel's work recipe-sysroot-native
rm tmp/work/qemux86_64-poky-linux/linux- yocto/6.6.20+git/recipe- sysroot-native/usr/bin/git

Bisecting poky leads to commit
f7fa98cca8 ("kern-tools: depend on git-replacement-native") Reverting it on top of b5624ee564 makes the parsing pass.

I assume that git-replacement-native does not work with https,
the
fetch error also goes away if changing in SRC_URI from https to
git.

Quoted 7 lines
Quoted 1 line

Any comments?

I didn't even know that curl was coming into play :)

Adding DEPENDS:class-native += "ca-certificates" to the curl recipe
should resolve the issue.

Looks like curl-native resp. libcurl hardcodes the lookup to its
own
work directory, i.e.:
x86_64-linux/curl-native/8.6.0/recipe-sysroot- native/etc/ssl/certs/ca-certificates.crt

Quoted 8 lines

So even if DEPENDS/RDEPENDS will install ca-certificates in the kernel's
recipe-sysroot-native the parsing will fail if the curl-native directory
is not/no longer populated, e.g. because curl-native came from sstate
or
rm_work is in INHERIT.

This all gets a bit messy.

We've relied upon scripts that use openssl to set variables like:

Quoted 8 lines

export SSL_CERT_DIR="XXXX/etc/ssl/certs/

so in theory we might be able to set an environment variable in a wrapper around the git commands.

It may be better if we teach curl a relative path to the certs...

I suspect this isn't going to be an easy/neat fix unfortunately.

I'm trying not to get sucked into "work" today however I realised that
relative paths won't work without some implementation of "$ORIGIN" support into these paths. Given it ultimately ends up in openssl, it
would probably be best there.

Quoted 8 lines

I'd not be against writing a $ORIGIN support patch and seeing what upstream think about it. It would still mean finding a way to find the
path to the library file somehow.

For purposes of the release, setting the right envvars in the git wrapper is probably the way forward for now, much as I dislike the requirement to do that.

I tried this, but curl didn't seem to use it to locate the cert file.

i.e., I tried this:

---------------

meta/recipes-devtools/git/git_2.44.0.bb +1 -1
index e6d1470873..f6b06ec601 100644
@@ -31,7 +31,7 @@ PACKAGECONFIG ??= "expat curl"
 PACKAGECONFIG[cvsserver] = ""
 PACKAGECONFIG[svn] = ""
 PACKAGECONFIG[manpages] = ",,asciidoc-native xmlto-native"
-PACKAGECONFIG[curl] = "--with-curl,--without-curl,curl"
+PACKAGECONFIG[curl] = "--with-curl,--without-curl,curl ca-

certificates"
PACKAGECONFIG[expat] = "--with-expat,--without-expat,expat"

EXTRA_OECONF = "--with-perl=${STAGING_BINDIR_NATIVE}/perl-
native/perl \

meta/recipes-devtools/git/git_2.44.0.bb +2 -1
@@ -103,13 +103,15 @@ do_install:append:class-target () {
 do_install:append:class-native() {
        create_wrapper ${D}${bindir}/git \
                GIT_EXEC_PATH='`dirname
 $''realpath`'/${REL_GIT_EXEC_PATH} \
-               GIT_TEMPLATE_DIR='`dirname
 $''realpath`'/${REL_GIT_TEMPLATE_DIR}
+               GIT_TEMPLATE_DIR='`dirname
 $''realpath`'/${REL_GIT_TEMPLATE_DIR} \
+               SSL_CERT_DIR='`dirname

$''realpath`'/../../etc/ssl/certs/
}

 do_install:append:class-nativesdk() {
        create_wrapper ${D}${bindir}/git \
                GIT_EXEC_PATH='`dirname

$''realpath`'/${REL_GIT_EXEC_PATH} \
GIT_TEMPLATE_DIR='`dirname
$''realpath`'/${REL_GIT_TEMPLATE_DIR}
+ SSL_CERT_DIR='`dirname
$''realpath`'/../../etc/ssl/certs/
perl_native_fixup
}

---------

Probably because the exec of curl from git isn't getting the environment ? Either
that, or I did it wrong. I didn't try it as an explicit export, and that is probably it,
I can try that later tonight.

Quoted 3 lines

Only when I added the depends on ca-certifications to curl itself was it able to
fetch the autorevs.

We probably need both the variable (to avoid the workdir removal issue) and the dependency.

Quoted 8 lines
Quoted 4 lines

Cheers,

Richard

Looks like git has its own cainfo path env variable, SSL_CERT_DIR seems to not work. [1]

Aha! That explains why it was ignoring my attempts :)

Quoted 28 lines

The following changes did make AUTOREV work for me:

meta/recipes-devtools/git/git_2.44.0.bb +3 -0
@@ -4,6 +4,7 @@ DESCRIPTION = "Git is a free and open source distributed version control system
 SECTION = "console/utils"
 LICENSE = "GPL-2.0-only & GPL-2.0-or-later & BSD-3-Clause & MIT & BSL-1.0 & LGPL-2.1-or-later"
 DEPENDS = "openssl zlib"
+DEPENDS:class-native += "ca-certificates"
 
 PROVIDES:append:class-native = " git-replacement-native"
 
@@ -95,6 +96,7 @@ perl_native_fixup () {
 
 REL_GIT_EXEC_PATH = "${@os.path.relpath(libexecdir, bindir)}/git-core"
 REL_GIT_TEMPLATE_DIR = "${@os.path.relpath(datadir, bindir)}/git-core/templates"
+REL_GIT_SSL_CAINFO = "${@os.path.relpath(sysconfdir, bindir)}/ssl/certs/ca-certificates.crt"
 
 do_install:append:class-target () {
        perl_native_fixup
@@ -103,6 +105,7 @@ do_install:append:class-target () {
 do_install:append:class-native() {
        create_wrapper ${D}${bindir}/git \
                GIT_EXEC_PATH='`dirname $''realpath`'/${REL_GIT_EXEC_PATH} \
+               GIT_SSL_CAINFO='`dirname $''realpath`'/${REL_GIT_SSL_CAINFO} \
                GIT_TEMPLATE_DIR='`dirname $''realpath`'/${REL_GIT_TEMPLATE_DIR}
 }
 
Quoted 1 line

if that is acceptable I could send a proper patch for review.

It looks fine to me. It follows the structure of the other variables.

I don't see any reason for a conditional or packageconfig dependency on ca-certificates
(since https isn't exactly uncommon), and keeping everything contained within the git
recipe is better this close to release.

That being said, I still wonder / think that curl should also just provide its own dependency on ca-certificates, since it is the application that is actually looking for them. I don't know if this way we'll run into some sort of issue with them not being available after sstate, cleanup or something else.

Definitely send it for review, hopefully those that are well versed in those questions I have will see it and comment :)

Bruce

Quoted 4 lines

Max

[1] https://stackoverflow.com/questions/11621768/how-can-i-make-git-accept-a-self-signed-certificate

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#10 lore raw

PATCH 1/1 git: git-replacement-native: depend on ca-certificate

From: Max Krummenacher <[email protected]>

git is delegating webacces for URLs using TLS to libcurl. However our native libcurl build does not find a ca-certificate.crt unless its curl-native work dir still exists and thus git will fail.
If a recipe uses AUTOREV with a git repo using https as its protocol parsing of that recipe will fail fetching the latest HEAD.

Fix that by depending on ca-certificate and give its location to libcurl via git's envrironment variable GIT_SSL_CAINFO.

---
 meta/recipes-devtools/git/git_2.44.0.bb | 3 +++
 1 file changed, 3 insertions(+)

See also the ML discussion:
https://lore.kernel.org/all/[email protected]/

meta/recipes-devtools/git/git_2.44.0.bb +3 -0
index e6d1470873..90e555eba7 100644
@@ -4,6 +4,7 @@ DESCRIPTION = "Git is a free and open source distributed version control system
 SECTION = "console/utils"
 LICENSE = "GPL-2.0-only & GPL-2.0-or-later & BSD-3-Clause & MIT & BSL-1.0 & LGPL-2.1-or-later"
 DEPENDS = "openssl zlib"
+DEPENDS:class-native += "ca-certificates"
 
 PROVIDES:append:class-native = " git-replacement-native"
 
@@ -95,6 +96,7 @@ perl_native_fixup () {
 
 REL_GIT_EXEC_PATH = "${@os.path.relpath(libexecdir, bindir)}/git-core"
 REL_GIT_TEMPLATE_DIR = "${@os.path.relpath(datadir, bindir)}/git-core/templates"
+REL_GIT_SSL_CAINFO = "${@os.path.relpath(sysconfdir, bindir)}/ssl/certs/ca-certificates.crt"
 
 do_install:append:class-target () {
 	perl_native_fixup
@@ -103,6 +105,7 @@ do_install:append:class-target () {
 do_install:append:class-native() {
 	create_wrapper ${D}${bindir}/git \
 		GIT_EXEC_PATH='`dirname $''realpath`'/${REL_GIT_EXEC_PATH} \
+		GIT_SSL_CAINFO='`dirname $''realpath`'/${REL_GIT_SSL_CAINFO} \
 		GIT_TEMPLATE_DIR='`dirname $''realpath`'/${REL_GIT_TEMPLATE_DIR}
 }
 
Signature · 1 line
2.42.0