git do_unpack
Recently it has come to my attention that in morty, and maybe later, do_unpack for git does not use the downloads directory, and attempts to contact the git server.
This is unfortunate, as we often do bitbake -c fetchall and give tarballs of our workspace to customers, thinking it would save something, but it appears the savings are limited, and now I have a customer complaint about this, since I have one recipe that uses a private repository.
What is the logic of not using the downloads directory for the source of the unpack?
I recently fixed a recipe by writing an unpack that does not use networking and does a clone of the downloads directory to get around this. Why is this not done by openembedded-core?
Thanks.
It would help if you can:
a) provide log.do_unpack that shows that network access is indeed happening;
b) demonstrate the behavior with oe-core master, as morty is soon eligible for retro computing museum, and hardly anyone wants to look into that :)
Alex
Quoted 19 lines · John Klug
Recently it has come to my attention that in morty, and maybe later,
do_unpackfor git does not use the downloads directory, and attempts to contact the git server.This is unfortunate, as we often do bitbake -c fetchall and give tarballs of our workspace to customers, thinking it would save something, but it appears the savings are limited, and now I have a customer complaint about this, since I have one recipe that uses a private repository.
What is the logic of not using the downloads directory for the source of the unpack?
I recently fixed a recipe by writing an unpack that does not use networking and does a clone of the downloads directory to get around this. Why is this not done by openembedded-core?
Thanks.
a:
(fetch was already run successfully while connected with a VPN).
VPN then shut down.
$ bitbake -c unpack dhq-client
...
NOTE: Executing RunQueue Tasks
ERROR: dhq-client-0.32-r11 do_unpack: Fetcher failure: Fetch command export DBUS_SESSION_BUS_ADDRESS="unix:abstract=/tmp/dbus-CbyhEfsmW1"; export SSH_AGENT_PID="2444"; export SSH_AUTH_SOCK="/tmp/ssh-gHgQLFqq7E5R/agent.2443"; export PATH="/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin/python-native:/home/klugja/mlinux-up-14/layers/openembedded-core/scripts:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin/arm-mlinux-linux-gnueabi:/home/klugja/mlinux-up-14/build/tmp/sysroots/mt100eocg/usr/bin/crossscripts:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/sbin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/sbin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/bin:/home/klugja/mlinux-up-14/scripts:/home/klugja/mlinux-up-14/bitbake/bin:/home/klugja/bin:/home/klugja/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"; export HOME="/home/klugja"; git -c core.fsyncobjectfiles=0 ls-remote ssh://[email protected]/devicehq/mAPI-dhq-client.git failed with exit code 128, output:
ssh: connect to host gitlab.multitech.net port 22: Connection timed out
fatal: Could not read from remote repository.
Please make sure you have the correct access rights and the repository exists.
ERROR: dhq-client-0.32-r11 do_unpack: Function failed: base_do_unpack
ERROR: Logfile of failure stored in: /home/klugja/mlinux-up-14/build/tmp/work/mt100eocg-mlinux-linux-gnueabi/dhq-client/0.32-r11/temp/log.do_unpack.10175
ERROR: Task (/home/klugja/mlinux-up-14/build/../layers/meta-mlinux/recipes-core/dhq-client/dhq-client_0.32.bb:do_unpack) failed with exit code '1'
NOTE: Tasks Summary: Attempted 2 tasks of which 1 didn't need to be rerun and 1 failed.
Summary: 1 task failed:
/home/klugja/mlinux-up-14/build/../layers/meta-mlinux/recipes-core/dhq-client/dhq-client_0.32.bb:do_unpack
Summary: There was 1 WARNING message shown.
Summary: There were 2 ERROR messages shown, returning a non-zero exit code.
Line 3 of the do_unpack.log:
DEBUG: Fetcher accessed the network with the command git -c core.fsyncobjectfiles=0 ls-remote ssh://[email protected]/devicehq/mAPI-dhq-client.git
Testing the download of the git (download is verifiable):
klugja@peru:~/mlinux-up-14/build/downloads/git2/gitlab.multitech.net.devicehq.mAPI-dhq-client.git$ git fsck Checking object directories: 100% (256/256), done. Checking objects: 100% (485/485), done. klugja@peru:~/mlinux-up-14/build/downloads/git2/gitlab.multitech.net.devicehq.mAPI-dhq-client.git$
b. Too busy writing my own git commands in my own do_unpack to fix this. Maybe someday.
Hi John,
Quoted 24 lines · John Klug
a:
(fetch was already run successfully while connected with a VPN).
VPN then shut down.
$ bitbake -c unpack dhq-client...NOTE: Executing RunQueue Tasks
ERROR: dhq-client-0.32-r11do_unpack: Fetcher failure: Fetch command export DBUS_SESSION_BUS_ADDRESS="unix:abstract=/tmp/dbus-CbyhEfsmW1"; export SSH_AGENT_PID="2444"; export SSH_AUTH_SOCK="/tmp/ssh-gHgQLFqq7E5R/agent.2443"; export PATH="/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin/python-native:/home/klugja/mlinux-up-14/layers/openembedded-core/scripts:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin/arm-mlinux-linux-gnueabi:/home/klugja/mlinux-up-14/build/tmp/sysroots/mt100eocg/usr/bin/crossscripts:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/sbin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/sbin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/bin:/home/klugja/mlinux-up-14/scripts:/home/klugja/mlinux-up-14/bitbake/bin:/home/klugja/bin:/home/klugja/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"; export HOME="/home/klugja"; git -c core.fsyncobjectfiles=0 ls-remote ssh://[email protected]/devicehq/mAPI-dhq-client.git failed with exit code 128, output:
ssh: connect to host gitlab.multitech.net port 22: Connection timed out
fatal: Could not read from remote repository.Please make sure you have the correct access rights and the repository exists.
ERROR: dhq-client-0.32-r11
do_unpack: Function failed:base_do_unpack
ERROR: Logfile of failure stored in:/home/klugja/mlinux-up-14/build/tmp/work/mt100eocg-mlinux-linux-gnueabi/dhq-client/0.32-r11/temp/log.do_unpack.10175
ERROR: Task (/home/klugja/mlinux-up-14/build/../layers/meta-mlinux/recipes-core/dhq-client/dhq-client_0.32.bb:do_unpack) failed with exit code '1'
NOTE: Tasks Summary: Attempted 2 tasks of which 1 didn't need to be rerun and 1 failed.Summary: 1 task failed:
Quoted 9 lines
/home/klugja/mlinux-up-14/build/../layers/meta-mlinux/recipes-core/dhq-client/dhq-client_0.32.bb:do_unpackSummary: There was 1 WARNING message shown.
Summary: There were 2 ERROR messages shown, returning a non-zero exit code.Line 3 of the do_unpack.log:
DEBUG: Fetcher accessed the network with the command git -c core.fsyncobjectfiles=0 ls-remote ssh://[email protected]/devicehq/mAPI-dhq-client.git
Any chance you're using a tag in your SRC_URI?
https://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#git-fetcher
or AUTOREV? Which I guess needs network access to know if there's something to pull or not.
Quentin
This happens if the recipe does not specify the exact revision hash to use in SRCREV. If it says "use latest revision" or "use this tag", then bitbake will do ls-remote on the original repo to resolve those to up-to-date revisions, which is not unreasonable.
What does the recipe look like?
Alex
Quoted 62 lines · John Klug
a:
(fetch was already run successfully while connected with a VPN).
VPN then shut down.
$ bitbake -c unpack dhq-client...NOTE: Executing RunQueue Tasks
ERROR: dhq-client-0.32-r11do_unpack: Fetcher failure: Fetch command
export DBUS_SESSION_BUS_ADDRESS="unix:abstract=/tmp/dbus-CbyhEfsmW1";
export SSH_AGENT_PID="2444"; export
SSH_AUTH_SOCK="/tmp/ssh-gHgQLFqq7E5R/agent.2443"; export PATH="/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin/python-native:/home/klugja/mlinux-up-14/layers/openembedded-core/scripts:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin/arm-mlinux-linux-gnueabi:/home/klugja/mlinux-up-14/build/tmp/sysroots/mt100eocg/usr/bin/crossscripts:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/sbin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/sbin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/bin:/home/klugja/mlinux-up-14/scripts:/home/klugja/mlinux-up-14/bitbake/bin:/home/klugja/bin:/home/klugja/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"; export HOME="/home/klugja"; git -c core.fsyncobjectfiles=0 ls-remote ssh://
[email protected]/devicehq/mAPI-dhq-client.git failed with exit
code 128, output:
ssh: connect to host gitlab.multitech.net port 22: Connection timed out
fatal: Could not read from remote repository.Please make sure you have the correct access rights and the repository exists.
ERROR: dhq-client-0.32-r11
do_unpack: Function failed:base_do_unpack
ERROR: Logfile of failure stored in:/home/klugja/mlinux-up-14/build/tmp/work/mt100eocg-mlinux-linux-gnueabi/dhq-client/0.32-r11/temp/log.do_unpack.10175
ERROR: Task
(/home/klugja/mlinux-up-14/build/../layers/meta-mlinux/recipes-core/dhq-client/dhq-client_0.32.bb:do_unpack) failed with exit code '1'
NOTE: Tasks Summary: Attempted 2 tasks of which 1 didn't need to be
rerun and 1 failed.Summary: 1 task failed:
/home/klugja/mlinux-up-14/build/../layers/meta-mlinux/recipes-core/dhq-client/dhq-client_0.32.bb:do_unpack
Summary: There was 1 WARNING message shown.
Summary: There were 2 ERROR messages shown, returning a non-zero exit
code.Line 3 of the do_unpack.log:
DEBUG: Fetcher accessed the network with the command git -c
core.fsyncobjectfiles=0 ls-remote ssh:// [email protected]/devicehq/mAPI-dhq-client.gitTesting the download of the git (download is verifiable):
klugja@peru:~/mlinux-up-14/build/downloads/git2/gitlab.multitech.net.devicehq.mAPI-dhq-client.git$
git fsck
Checking object directories: 100% (256/256), done.
Checking objects: 100% (485/485), done.
klugja@peru
:~/mlinux-up-14/build/downloads/git2/gitlab.multitech.net.devicehq.mAPI-dhq-client.git$b. Too busy writing my own git commands in my own
do_unpackto fix this. Maybe someday.
We typically use tags to match PV and name our recipes NAME_TAG.bb
Recipe name:
dhq-client_0.39.bb
from bitbake -e dhq-client output:
# pre-expansion value:
# "${PN}-${PV}"
P="dhq-client-0.39"In recipe:
SRCREV = "${PV}"SRC_URI = "git://[email protected]/devicehq/mAPI-dhq-client.git;protocol=ssh;nobranch=1 \ file://config.json.sample \ file://dhqclient.init \ file://dhqclient \ file://dhq-rootCA.pem \ "
Sanity check:
jklug@trustytahrdm1:~$ git clone [email protected]:devicehq/mAPI-dhq-client Cloning into 'mAPI-dhq-client'... remote: Counting objects: 485, done. remote: Compressing objects: 100% (199/199), done. remote: Total 485 (delta 285), reused 450 (delta 265) Receiving objects: 100% (485/485), 84.73 KiB | 14.12 MiB/s, done. Resolving deltas: 100% (285/285), done. jklug@trustytahrdm1:~$ cd mAPI-dhq-client/ jklug@trustytahrdm1:~/mAPI-dhq-client$ git log -1 --format=%ai 0.39 | head 2020-05-05 09:47:03 -0500 jklug@trustytahrdm1:~/mAPI-dhq-client$ git checkout 0.39 Note: checking out '0.39'.
You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
HEAD is now at d21aab3 Merge branch 'vp/fix_logs' into 'master'
SRCREV should be the actual commit hash that matches the tag, then do_unpack will not do git ls-remote. Tags can, and do move around.
Alex
Quoted 47 lines · John Klug
We typically use tags to match PV and name our recipes NAME_TAG.bb
Recipe name:
dhq-client_0.39.bb
from bitbake -e dhq-client output:
# pre-expansion value: # "${PN}-${PV}" P="dhq-client-0.39"In recipe:
SRCREV = "${PV}"
SRC_URI= "git://[email protected]/devicehq/mAPI-dhq-client.git;protocol=ssh;nobranch=1
\ file://config.json.sample \ file://dhqclient.init
\ file://dhqclient \ file://dhq-rootCA.pem \ "Sanity check:
jklug@trustytahrdm1:~$ git clone [email protected]: devicehq/mAPI-dhq-client Cloning into 'mAPI-dhq-client'... remote: Counting objects: 485, done. remote: Compressing objects: 100% (199/199), done. remote: Total 485 (delta 285), reused 450 (delta 265) Receiving objects: 100% (485/485), 84.73 KiB | 14.12 MiB/s, done. Resolving deltas: 100% (285/285), done. jklug@trustytahrdm1:~$ cd mAPI-dhq-client/ jklug@trustytahrdm1:~/mAPI-dhq-client$ git log -1 --format=%ai 0.39 | head 2020-05-05 09:47:03 -0500 jklug@trustytahrdm1:~/mAPI-dhq-client$ git checkout 0.39 Note: checking out '0.39'.You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
HEAD is now at d21aab3 Merge branch 'vp/fix_logs' into 'master'