Skip to contentSkip to conversation

git do_unpack

DISCUSSION John Klug 3 participants 7 messages Jun 28, 2020 – Jun 30, 2020

lore
Share & tools
lore

John Klug

author
#0 lore raw

Recently it has come to my attention that in morty, and maybe later, do_unpack for git does not use the downloads directory, and attempts to contact the git server.

This is unfortunate, as we often do bitbake -c fetchall and give tarballs of our workspace to customers, thinking it would save something, but it appears the savings are limited, and now I have a customer complaint about this, since I have one recipe that uses a private repository.

What is the logic of not using the downloads directory for the source of the unpack?

I recently fixed a recipe by writing an unpack that does not use networking and does a clone of the downloads directory to get around this.  Why is this not done by openembedded-core?

Thanks.

#1 lore raw

It would help if you can:

a) provide log.do_unpack that shows that network access is indeed happening;

b) demonstrate the behavior with oe-core master, as morty is soon eligible for retro computing museum, and hardly anyone wants to look into that :)

Alex

Quoted 19 lines · John Klug

Recently it has come to my attention that in morty, and maybe later, do_unpack for git does not use the downloads directory, and attempts to contact the git server.

This is unfortunate, as we often do bitbake -c fetchall and give tarballs of our workspace to customers, thinking it would save something, but it appears the savings are limited, and now I have a customer complaint about this, since I have one recipe that uses a private repository.

What is the logic of not using the downloads directory for the source of the unpack?

I recently fixed a recipe by writing an unpack that does not use networking and does a clone of the downloads directory to get around this. Why is this not done by openembedded-core?

Thanks.

author
#2 lore raw

a:

(fetch was already run successfully while connected with a VPN).

VPN then shut down.

  $  bitbake -c unpack dhq-client
     ...

NOTE: Executing RunQueue Tasks
ERROR: dhq-client-0.32-r11 do_unpack: Fetcher failure: Fetch command export DBUS_SESSION_BUS_ADDRESS="unix:abstract=/tmp/dbus-CbyhEfsmW1"; export SSH_AGENT_PID="2444"; export SSH_AUTH_SOCK="/tmp/ssh-gHgQLFqq7E5R/agent.2443"; export PATH="/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin/python-native:/home/klugja/mlinux-up-14/layers/openembedded-core/scripts:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin/arm-mlinux-linux-gnueabi:/home/klugja/mlinux-up-14/build/tmp/sysroots/mt100eocg/usr/bin/crossscripts:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/sbin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/sbin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/bin:/home/klugja/mlinux-up-14/scripts:/home/klugja/mlinux-up-14/bitbake/bin:/home/klugja/bin:/home/klugja/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"; export HOME="/home/klugja"; git -c core.fsyncobjectfiles=0 ls-remote ssh://[email protected]/devicehq/mAPI-dhq-client.git failed with exit code 128, output:
ssh: connect to host gitlab.multitech.net port 22: Connection timed out
fatal: Could not read from remote repository.

Please make sure you have the correct access rights and the repository exists.

ERROR: dhq-client-0.32-r11 do_unpack: Function failed: base_do_unpack
ERROR: Logfile of failure stored in: /home/klugja/mlinux-up-14/build/tmp/work/mt100eocg-mlinux-linux-gnueabi/dhq-client/0.32-r11/temp/log.do_unpack.10175
ERROR: Task (/home/klugja/mlinux-up-14/build/../layers/meta-mlinux/recipes-core/dhq-client/dhq-client_0.32.bb:do_unpack) failed with exit code '1'
NOTE: Tasks Summary: Attempted 2 tasks of which 1 didn't need to be rerun and 1 failed.

Summary: 1 task failed:

    /home/klugja/mlinux-up-14/build/../layers/meta-mlinux/recipes-core/dhq-client/dhq-client_0.32.bb:do_unpack

Summary: There was 1 WARNING message shown.
Summary: There were 2 ERROR messages shown, returning a non-zero exit code.

Line 3 of the do_unpack.log:

DEBUG: Fetcher accessed the network with the command git -c core.fsyncobjectfiles=0 ls-remote ssh://[email protected]/devicehq/mAPI-dhq-client.git

Testing the download of the git (download is verifiable):

  klugja@peru:~/mlinux-up-14/build/downloads/git2/gitlab.multitech.net.devicehq.mAPI-dhq-client.git$ git fsck
  Checking object directories: 100% (256/256), done.
  Checking objects: 100% (485/485), done.
  klugja@peru:~/mlinux-up-14/build/downloads/git2/gitlab.multitech.net.devicehq.mAPI-dhq-client.git$

b. Too busy writing my own git commands in my own do_unpack to fix this. Maybe someday.

#3 lore raw

Hi John,

Quoted 24 lines · John Klug

a:

(fetch was already run successfully while connected with a VPN).

VPN then shut down.

  $  bitbake -c unpack dhq-client
     ...

NOTE: Executing RunQueue Tasks
ERROR: dhq-client-0.32-r11 do_unpack: Fetcher failure: Fetch command export DBUS_SESSION_BUS_ADDRESS="unix:abstract=/tmp/dbus-CbyhEfsmW1"; export SSH_AGENT_PID="2444"; export SSH_AUTH_SOCK="/tmp/ssh-gHgQLFqq7E5R/agent.2443"; export PATH="/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin/python-native:/home/klugja/mlinux-up-14/layers/openembedded-core/scripts:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin/arm-mlinux-linux-gnueabi:/home/klugja/mlinux-up-14/build/tmp/sysroots/mt100eocg/usr/bin/crossscripts:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/sbin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/sbin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/bin:/home/klugja/mlinux-up-14/scripts:/home/klugja/mlinux-up-14/bitbake/bin:/home/klugja/bin:/home/klugja/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"; export HOME="/home/klugja"; git -c core.fsyncobjectfiles=0 ls-remote ssh://[email protected]/devicehq/mAPI-dhq-client.git failed with exit code 128, output:
ssh: connect to host gitlab.multitech.net port 22: Connection timed out
fatal: Could not read from remote repository.

Please make sure you have the correct access rights and the repository exists.

ERROR: dhq-client-0.32-r11 do_unpack: Function failed: base_do_unpack
ERROR: Logfile of failure stored in: /home/klugja/mlinux-up-14/build/tmp/work/mt100eocg-mlinux-linux-gnueabi/dhq-client/0.32-r11/temp/log.do_unpack.10175
ERROR: Task (/home/klugja/mlinux-up-14/build/../layers/meta-mlinux/recipes-core/dhq-client/dhq-client_0.32.bb:do_unpack) failed with exit code '1'
NOTE: Tasks Summary: Attempted 2 tasks of which 1 didn't need to be rerun and 1 failed.

Summary: 1 task failed:

Quoted 9 lines
    /home/klugja/mlinux-up-14/build/../layers/meta-mlinux/recipes-core/dhq-client/dhq-client_0.32.bb:do_unpack

Summary: There was 1 WARNING message shown.
Summary: There were 2 ERROR messages shown, returning a non-zero exit code.

Line 3 of the do_unpack.log:

DEBUG: Fetcher accessed the network with the command git -c core.fsyncobjectfiles=0 ls-remote ssh://[email protected]/devicehq/mAPI-dhq-client.git

Any chance you're using a tag in your SRC_URI?

https://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#git-fetcher

or AUTOREV? Which I guess needs network access to know if there's something to pull or not.

Quentin

#4 lore raw
In reply to John Klug

This happens if the recipe does not specify the exact revision hash to use in SRCREV. If it says "use latest revision" or "use this tag", then bitbake will do ls-remote on the original repo to resolve those to up-to-date revisions, which is not unreasonable.

What does the recipe look like?

Alex

Quoted 62 lines · John Klug

a:

(fetch was already run successfully while connected with a VPN).

VPN then shut down.

  $  bitbake -c unpack dhq-client
     ...

NOTE: Executing RunQueue Tasks
ERROR: dhq-client-0.32-r11 do_unpack: Fetcher failure: Fetch command
export DBUS_SESSION_BUS_ADDRESS="unix:abstract=/tmp/dbus-CbyhEfsmW1";
export SSH_AGENT_PID="2444"; export
SSH_AUTH_SOCK="/tmp/ssh-gHgQLFqq7E5R/agent.2443"; export PATH="/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin/python-native:/home/klugja/mlinux-up-14/layers/openembedded-core/scripts:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin/arm-mlinux-linux-gnueabi:/home/klugja/mlinux-up-14/build/tmp/sysroots/mt100eocg/usr/bin/crossscripts:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/sbin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/sbin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/bin:/home/klugja/mlinux-up-14/scripts:/home/klugja/mlinux-up-14/bitbake/bin:/home/klugja/bin:/home/klugja/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"; export HOME="/home/klugja"; git -c core.fsyncobjectfiles=0 ls-remote ssh://
[email protected]/devicehq/mAPI-dhq-client.git failed with exit
code 128, output:
ssh: connect to host gitlab.multitech.net port 22: Connection timed out
fatal: Could not read from remote repository.

Please make sure you have the correct access rights and the repository exists.

ERROR: dhq-client-0.32-r11 do_unpack: Function failed: base_do_unpack
ERROR: Logfile of failure stored in:
/home/klugja/mlinux-up-14/build/tmp/work/mt100eocg-mlinux-linux-gnueabi/dhq-client/0.32-r11/temp/log.do_unpack.10175
ERROR: Task
(/home/klugja/mlinux-up-14/build/../layers/meta-mlinux/recipes-core/dhq-client/dhq-client_0.32.bb:do_unpack) failed with exit code '1'
NOTE: Tasks Summary: Attempted 2 tasks of which 1 didn't need to be
rerun and 1 failed.

Summary: 1 task failed:

/home/klugja/mlinux-up-14/build/../layers/meta-mlinux/recipes-core/dhq-client/dhq-client_0.32.bb:
do_unpack
Summary: There was 1 WARNING message shown.
Summary: There were 2 ERROR messages shown, returning a non-zero exit
code.

Line 3 of the do_unpack.log:

DEBUG: Fetcher accessed the network with the command git -c
core.fsyncobjectfiles=0 ls-remote ssh:// [email protected]/devicehq/mAPI-dhq-client.git

Testing the download of the git (download is verifiable):
klugja@peru:~/mlinux-up-14/build/downloads/git2/gitlab.multitech.net.devicehq.mAPI-dhq-client.git$
git fsck
Checking object directories: 100% (256/256), done.
Checking objects: 100% (485/485), done.
klugja@peru
:~/mlinux-up-14/build/downloads/git2/gitlab.multitech.net.devicehq.mAPI-dhq-client.git$

b. Too busy writing my own git commands in my own do_unpack to fix this. Maybe someday.

author
#5 lore raw

We typically use tags to match PV and name our recipes NAME_TAG.bb

Recipe name:

dhq-client_0.39.bb

from bitbake -e dhq-client output:

# pre-expansion value:
#   "${PN}-${PV}"
P="dhq-client-0.39"

In recipe:

SRCREV = "${PV}"
SRC_URI = "git://[email protected]/devicehq/mAPI-dhq-client.git;protocol=ssh;nobranch=1 \ file://config.json.sample \ file://dhqclient.init \ file://dhqclient \ file://dhq-rootCA.pem \ "

Sanity check:

jklug@trustytahrdm1:~$ git clone [email protected]:devicehq/mAPI-dhq-client
Cloning into 'mAPI-dhq-client'...
remote: Counting objects: 485, done.
remote: Compressing objects: 100% (199/199), done.
remote: Total 485 (delta 285), reused 450 (delta 265)
Receiving objects: 100% (485/485), 84.73 KiB | 14.12 MiB/s, done.
Resolving deltas: 100% (285/285), done.
jklug@trustytahrdm1:~$ cd mAPI-dhq-client/
jklug@trustytahrdm1:~/mAPI-dhq-client$ git log -1 --format=%ai 0.39 | head
2020-05-05 09:47:03 -0500
jklug@trustytahrdm1:~/mAPI-dhq-client$ git checkout 0.39
Note: checking out '0.39'.

You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example:

git checkout -b <new-branch-name>

HEAD is now at d21aab3 Merge branch 'vp/fix_logs' into 'master'

#6 lore raw

SRCREV should be the actual commit hash that matches the tag, then do_unpack will not do git ls-remote. Tags can, and do move around.

Alex

Quoted 47 lines · John Klug

We typically use tags to match PV and name our recipes NAME_TAG.bb

Recipe name:

dhq-client_0.39.bb

from bitbake -e dhq-client output:

# pre-expansion value:
#   "${PN}-${PV}"
P="dhq-client-0.39"

In recipe:

SRCREV = "${PV}"
SRC_URI = "git:// [email protected]/devicehq/mAPI-dhq-client.git;protocol=ssh;nobranch=1
\ file://config.json.sample \ file://dhqclient.init
\ file://dhqclient \ file://dhq-rootCA.pem \ "

Sanity check:

jklug@trustytahrdm1:~$ git clone [email protected]:
devicehq/mAPI-dhq-client
Cloning into 'mAPI-dhq-client'...
remote: Counting objects: 485, done.
remote: Compressing objects: 100% (199/199), done.
remote: Total 485 (delta 285), reused 450 (delta 265)
Receiving objects: 100% (485/485), 84.73 KiB | 14.12 MiB/s, done.
Resolving deltas: 100% (285/285), done.
jklug@trustytahrdm1:~$ cd mAPI-dhq-client/
jklug@trustytahrdm1:~/mAPI-dhq-client$ git log -1 --format=%ai 0.39 | head
2020-05-05 09:47:03 -0500
jklug@trustytahrdm1:~/mAPI-dhq-client$ git checkout 0.39
Note: checking out '0.39'.

You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example:

git checkout -b <new-branch-name>

HEAD is now at d21aab3 Merge branch 'vp/fix_logs' into 'master'