Skip to contentSkip to conversation

NO INTERNET ENVIRONMENT - USING PREMIRROR fails -> gstreamer1.0_1.12.2.bb:do_unpack failed due to a fetch issue

DISCUSSION [email protected] 2 participants 6 messages Jul 23, 2020 – Aug 17, 2020

lore
Share & tools
lore
#0 lore raw

Hi,

I have a problem. I am trying to set up development environment that runs offline. For this purpose i set up a repository folder by the following local.conf file:

"
MACHINE ??= 'imx6dlsabresd'
DISTRO ?= 'fsl-imx-fb'
PACKAGE_CLASSES ?= "package_rpm"
EXTRA_IMAGE_FEATURES ?= "debug-tweaks dbg-pkgs dev-pkgs package-management ssh-server-dropbear ssh-server-openssh staticdev-pkgs " USER_CLASSES ?= "buildstats image-mklibs image-prelink"
PATCHRESOLVE = "noop"
BB_DISKMON_DIRS ??= "\
STOPTASKS,${TMPDIR},1G,100K \
STOPTASKS,${DL_DIR},1G,100K \
STOPTASKS,${SSTATE_DIR},1G,100K \
STOPTASKS,/tmp,100M,100K \
ABORT,${TMPDIR},100M,1K \
ABORT,${DL_DIR},100M,1K \
ABORT,${SSTATE_DIR},100M,1K \
ABORT,/tmp,10M,1K"
PACKAGECONFIG_append_pn-qemu-native = " sdl"
PACKAGECONFIG_append_pn-nativesdk-qemu = " sdl"
CONF_VERSION = "1"

IMAGE_INSTALL_append = " gstreamer imx-test"

DL_DIR ?= "${BSPDIR}/downloads/"
SSTATE_DIR ?= "${BSPDIR}/sstate-cache/"
ACCEPT_FSL_EULA = "1"

# ADDED
BB_NO_NETWORK = "1"

INHERIT += "own-mirrors"
BB_GENERATE_MIRROR_TARBALLS = "1"

SOURCE_MIRROR_URL = "file://${BSPDIR}/repository"

PREMIRRORS_prepend = "\
git://.*/.* file://${BSPDIR}/repository/ \n \
svn://.*/.* file://${BSPDIR}/repository/ \n \
cvs://.*/.* file://${BSPDIR}/repository/ \n \
ftp://.*/.* file://${BSPDIR}/repository/ \n \
http://.*/.* file://${BSPDIR}/repository/ \n \
https://.*/.* file://${BSPDIR}/repository/ \n "
"

initially i run the conf above with "BB_NO_NETWORK = "0" to generate the tarballs in downloads dir. Run the command: bitbake fsl-image-multimedia-full

After initial run I copied the content of the downloads folder into repository folder. removed build/tmp folder, downloads folder and sstate-cache folder to be on the clear page. Then I set [BB_NO_NETWORK = "1" ] and execute "tar -czvf ... " to compress all the imx-yocto-bsp folder.

Then I port the imx-tar.gz file to a crops environment on Windows.

The set environment is crops/poky and no internet [unplugged ethernet].

then I run the command bitbake fsl-image-multimedia-full again.
I expect that the build will be successful as the repository shall be enough. But I get the following error:

"
ERROR: gstreamer1.0-1.12.2-r0 do_unpack: Fetcher failure: Fetch command export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1000/bus"; export SSH_AGENT_PID="2184"; export SSH_AUTH_SOCK="/run/user/1000/keyring/ssh"; export all_proxy="http://ikitis:[email protected]:8080"; export ALL_PROXY="http://ikitis:[email protected]:8080"; export ftp_proxy="http://ikitis:[email protected]:8080"; export FTP_PROXY="http://10.1.1.51:8080/"; export https_proxy="http://ikitis:[email protected]:8080"; export HTTPS_PROXY="http://10.1.1.51:8080/"; export http_proxy="http://ikitis:[email protected]:8080"; export HTTP_PROXY="http://10.1.1.51:8080/"; export PATH="/home/aselsan/imx-yocto-bsp/build-image-minimal/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/gstreamer1.0/1.12.2-r0/recipe-sysroot-native/usr/bin/python3-native:/home/aselsan/imx-yocto-bsp/sources/poky/scripts:/home/aselsan/imx-yocto-bsp/build-image-minimal/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/gstreamer1.0/1.12.2-r0/recipe-sysroot-native/usr/bin/arm-poky-linux-gnueabi:/home/aselsan/imx-yocto-bsp/build-image-minimal/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/gstreamer1.0/1.12.2-r0/recipe-sysroot/usr/bin/crossscripts:/home/aselsan/imx-yocto-bsp/build-image-minimal/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/gstreamer1.0/1.12.2-r0/recipe-sysroot-native/usr/sbin:/home/aselsan/imx-yocto-bsp/build-image-minimal/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/gstreamer1.0/1.12.2-r0/recipe-sysroot-native/usr/bin:/home/aselsan/imx-yocto-bsp/build-image-minimal/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/gstreamer1.0/1.12.2-r0/recipe-sysroot-native/sbin:/home/aselsan/imx-yocto-bsp/build-image-minimal/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/gstreamer1.0/1.12.2-r0/recipe-sysroot-native/bin:/home/aselsan/imx-yocto-bsp/sources/poky/bitbake/bin:/home/aselsan/imx-yocto-bsp/build-image-minimal/tmp/hosttools"; export HOME="/home/aselsan"; git -c core.fsyncobjectfiles=0 submodule update --init --recursive failed with exit code 1, output:
Submodule 'common' (https://anongit.freedesktop.org/git/gstreamer/common.git) registered for path 'common'
Cloning into '/home/aselsan/imx-yocto-bsp/build-image-minimal/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/gstreamer1.0/1.12.2-r0/git/common'...
fatal: unable to access 'https://anongit.freedesktop.org/git/gstreamer/common.git/': Received HTTP code 407 from proxy after CONNECT
fatal: clone of 'https://anongit.freedesktop.org/git/gstreamer/common.git' into submodule path '/home/aselsan/imx-yocto-bsp/build-image-minimal/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/gstreamer1.0/1.12.2-r0/git/common' failed
Failed to clone 'common'. Retry scheduled
Cloning into '/home/aselsan/imx-yocto-bsp/build-image-minimal/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/gstreamer1.0/1.12.2-r0/git/common'...
fatal: unable to access 'https://anongit.freedesktop.org/git/gstreamer/common.git/': Received HTTP code 407 from proxy after CONNECT
fatal: clone of 'https://anongit.freedesktop.org/git/gstreamer/common.git' into submodule path '/home/aselsan/imx-yocto-bsp/build-image-minimal/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/gstreamer1.0/1.12.2-r0/git/common' failed
Failed to clone 'common' a second time, aborting

ERROR: gstreamer1.0-1.12.2-r0 do_unpack: Function failed: base_do_unpack
ERROR: Logfile of failure stored in: /home/aselsan/imx-yocto-bsp/build-image-minimal/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/gstreamer1.0/1.12.2-r0/temp/log.do_unpack.54012
ERROR: Task (/home/aselsan/imx-yocto-bsp/sources/poky/meta/recipes-multimedia/gstreamer/gstreamer1.0_1.12.2.bb:do_unpack) failed with exit code '1' "

I checked out but could not find a reason

I would highly appreciate if you be kind enough to give support on the issue.

Best Regards
B. A.

#1 lore raw

Add "common" git repo to SRC_URI and set SRCREV + SRCREV_FORMAT for it, instead of using git submodules to fetch it.

Quoted 126 lines · On Thu, Jul 23, 2020 at 2:40 PM <[email protected]> wrote:

Hi,

I have a problem. I am trying to set up development environment that runs offline. For this purpose i set up a repository folder by the following local.conf file:

"
MACHINE ??= 'imx6dlsabresd'
DISTRO ?= 'fsl-imx-fb'
PACKAGE_CLASSES ?= "package_rpm"
EXTRA_IMAGE_FEATURES ?= "debug-tweaks dbg-pkgs dev-pkgs package-management ssh-server-dropbear ssh-server-openssh staticdev-pkgs " USER_CLASSES ?= "buildstats image-mklibs image-prelink" PATCHRESOLVE = "noop"

BB_DISKMON_DIRS ??= "\
    STOPTASKS,${TMPDIR},1G,100K \
    STOPTASKS,${DL_DIR},1G,100K \
    STOPTASKS,${SSTATE_DIR},1G,100K \
    STOPTASKS,/tmp,100M,100K \
    ABORT,${TMPDIR},100M,1K \
    ABORT,${DL_DIR},100M,1K \
    ABORT,${SSTATE_DIR},100M,1K \
    ABORT,/tmp,10M,1K"

PACKAGECONFIG_append_pn-qemu-native = " sdl"
PACKAGECONFIG_append_pn-nativesdk-qemu = " sdl"
CONF_VERSION = "1"

IMAGE_INSTALL_append = " gstreamer imx-test"

DL_DIR ?= "${BSPDIR}/downloads/"
SSTATE_DIR ?= "${BSPDIR}/sstate-cache/"
ACCEPT_FSL_EULA = "1"

# ADDED
BB_NO_NETWORK = "1"

INHERIT += "own-mirrors"
BB_GENERATE_MIRROR_TARBALLS = "1"

SOURCE_MIRROR_URL = "file://${BSPDIR}/repository"

PREMIRRORS_prepend = "\
git://.*/.* file://${BSPDIR}/repository/ \n \
svn://.*/.* file://${BSPDIR}/repository/ \n \
cvs://.*/.* file://${BSPDIR}/repository/ \n \
ftp://.*/.* file://${BSPDIR}/repository/ \n \
http://.*/.* file://${BSPDIR}/repository/ \n \
https://.*/.* file://${BSPDIR}/repository/ \n "
"

initially i run the conf above with "BB_NO_NETWORK = "0" to generate the tarballs in downloads dir.
Run the command: bitbake fsl-image-multimedia-full

After initial run I copied the content of the downloads folder into repository folder. removed build/tmp folder, downloads folder and sstate-cache folder to be on the clear page.
Then I set [BB_NO_NETWORK = "1" ] and execute "tar -czvf ... " to compress all the imx-yocto-bsp folder.

Then I port the imx-tar.gz file to a crops environment on Windows.

The set environment is crops/poky and no internet [unplugged ethernet].

then I run the command bitbake fsl-image-multimedia-full again. I expect that the build will be successful as the repository shall be enough.
But I get the following error:

"
ERROR: gstreamer1.0-1.12.2-r0 do_unpack: Fetcher failure: Fetch command
export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1000/bus"; export
SSH_AGENT_PID="2184"; export SSH_AUTH_SOCK="/run/user/1000/keyring/ssh";
export all_proxy="http://ikitis:[email protected]:8080"; export
ALL_PROXY="http://ikitis:[email protected]:8080"; export ftp_proxy="
http://ikitis:[email protected]:8080"; export FTP_PROXY="
http://10.1.1.51:8080/"; export https_proxy="
http://ikitis:[email protected]:8080"; export HTTPS_PROXY="
http://10.1.1.51:8080/"; export http_proxy="
http://ikitis:[email protected]:8080"; export HTTP_PROXY="
http://10.1.1.51:8080/"; export PATH="/home/aselsan/imx-yocto-bsp/build-image-minimal/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/gstreamer1.0/1.12.2-r0/recipe-sysroot-native/usr/bin/python3-native:/home/aselsan/imx-yocto-bsp/sources/poky/scripts:/home/aselsan/imx-yocto-bsp/build-image-minimal/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/gstreamer1.0/1.12.2-r0/recipe-sysroot-native/usr/bin/arm-poky-linux-gnueabi:/home/aselsan/imx-yocto-bsp/build-image-minimal/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/gstreamer1.0/1.12.2-r0/recipe-sysroot/usr/bin/crossscripts:/home/aselsan/imx-yocto-bsp/build-image-minimal/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/gstreamer1.0/1.12.2-r0/recipe-sysroot-native/usr/sbin:/home/aselsan/imx-yocto-bsp/build-image-minimal/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/gstreamer1.0/1.12.2-r0/recipe-sysroot-native/usr/bin:/home/aselsan/imx-yocto-bsp/build-image-minimal/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/gstreamer1.0/1.12.2-r0/recipe-sysroot-native/sbin:/home/aselsan/imx-yocto-bsp/build-image-minimal/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/gstreamer1.0/1.12.2-r0/recipe-sysroot-native/bin:/home/aselsan/imx-yocto-bsp/sources/poky/bitbake/bin:/home/aselsan/imx-yocto-bsp/build-image-minimal/tmp/hosttools"; export HOME="/home/aselsan"; git -c core.fsyncobjectfiles=0 submodule
update --init --recursive failed with exit code 1, output:
Submodule 'common' (
https://anongit.freedesktop.org/git/gstreamer/common.git) registered for
path 'common'
Cloning into
'/home/aselsan/imx-yocto-bsp/build-image-minimal/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/gstreamer1.0/1.12.2-r0/git/common'...
fatal: unable to access '
https://anongit.freedesktop.org/git/gstreamer/common.git/': Received HTTP
code 407 from proxy after CONNECT
fatal: clone of 'https://anongit.freedesktop.org/git/gstreamer/common.git'
into submodule path
'/home/aselsan/imx-yocto-bsp/build-image-minimal/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/gstreamer1.0/1.12.2-r0/git/common'
failed
Failed to clone 'common'. Retry scheduled
Cloning into
'/home/aselsan/imx-yocto-bsp/build-image-minimal/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/gstreamer1.0/1.12.2-r0/git/common'...
fatal: unable to access '
https://anongit.freedesktop.org/git/gstreamer/common.git/': Received HTTP
code 407 from proxy after CONNECT
fatal: clone of 'https://anongit.freedesktop.org/git/gstreamer/common.git'
into submodule path
'/home/aselsan/imx-yocto-bsp/build-image-minimal/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/gstreamer1.0/1.12.2-r0/git/common'
failed
Failed to clone 'common' a second time, aborting

ERROR: gstreamer1.0-1.12.2-r0 do_unpack: Function failed: base_do_unpack
ERROR: Logfile of failure stored in:
/home/aselsan/imx-yocto-bsp/build-image-minimal/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/gstreamer1.0/1.12.2-r0/temp/log.do_unpack.54012
ERROR: Task (/home/aselsan/imx-yocto-bsp/sources/poky/meta/recipes-multimedia/gstreamer/gstreamer1.0_1.12.2.bb:do_unpack)
failed with exit code '1'
"

I checked out but could not find a reason

I would highly appreciate if you be kind enough to give support on the issue.

Best Regards
B. A.

#2 lore raw

Martin  sorry that am im not quite sure if i get you right

gstreamer1.0_1.12.2.bb is below

"
require gstreamer1.0.inc

LIC_FILES_CHKSUM = "file://COPYING;md5=6762ed442b3822387a51c92d928ead0d \
file://gst/gst.h;beginline=1;endline=21;md5=e059138481205ee2c6fc1c079c016d0d"

SRC_URI = " \
http://gstreamer.freedesktop.org/src/gstreamer/gstreamer-${PV}.tar.xz \
file://0001-configure-Add-switches-for-enabling-disabling-libdw-.patch \
"
SRC_URI[md5sum] = "4748860621607ffd96244fb79c86c238"
SRC_URI[sha256sum] = "9fde3f39a2ea984f9e07ce09250285ce91f6e3619d186889f75b5154ecf994ba"

S = "${WORKDIR}/gstreamer-${PV}"

"
I have no git repo. so how shall i set the "common" git repo? And also how can i set SRCREV + SRCREV_FORMAT for it

I am quite new so can you be little bit more specific.

Regards.

#3 lore raw

Are you sure this is the whole SRC_URI and that there aren't some other parts added through some .bbappend?

Always use "bitbake -e" to see whatever bitbake sees when building the recipe to make sure.

"common" is git repository gstreamer components use in 1.12 it is git submodule as you can see:
https://gitlab.freedesktop.org/gstreamer/gstreamer/-/tree/1.12

autogen was than checking out this git submodule during do_configure when it wasn't there already, see:
https://gitlab.freedesktop.org/gstreamer/gstreamer/-/blob/1.12/autogen.sh#L19

Usually it was failing in do_configure when building without access to this git repo, we were adding "common" git repo to SRC_URI explicitly to make sure that the right revision is always there with the gstreamer sources and managed with bitbake fetcher (so that it's used from downloads directory instead of the fetch from internet every single time gst* recipes are rebuilt for whatever reason).

In your error log it shows that it was failing already in do_unpack, so I guess your recipe has some modifications to work around this issue as well, but the part you've shown doesn't include it.

Regards,

Quoted 32 lines · On Thu, Jul 23, 2020 at 2:56 PM <[email protected]> wrote:

Martin sorry that am im not quite sure if i get you right

gstreamer1.0_1.12.2.bb is below

"
require gstreamer1.0.inc

LIC_FILES_CHKSUM = "file://COPYING;md5=6762ed442b3822387a51c92d928ead0d \

file://gst/gst.h;beginline=1;endline=21;md5=e059138481205ee2c6fc1c079c016d0d"

SRC_URI = " \
    http://gstreamer.freedesktop.org/src/gstreamer/gstreamer-${PV}.tar.xz

\
file://0001-configure-Add-switches-for-enabling-disabling-libdw-.patch
\
"
SRC_URI[md5sum] = "4748860621607ffd96244fb79c86c238" SRC_URI[sha256sum] = "9fde3f39a2ea984f9e07ce09250285ce91f6e3619d186889f75b5154ecf994ba"

S = "${WORKDIR}/gstreamer-${PV}"

"
I have no git repo. so how shall i set the "common" git repo? And also how can i set SRCREV + SRCREV_FORMAT for it

I am quite new so can you be little bit more specific.

Regards.

#4 lore raw

a search under the source shown that there is a suspicious file ./meta-fsl-bsp-release/imx/meta-bsp/recipes-multimedia/gstreamer/gstreamer1.0_1.12.%.bbappend

I guess this recipe is appending something to "gstreamer1.0_1.12.2.bb" but i am not sure. How can be sure that this recipe is in effect ???

The recipe has the following content:
"
# Use i.MX fork of GST for customizations
GST1.0_SRC ?= "gitsm://source.codeaurora.org/external/imx/gstreamer.git;protocol=https" SRCBRANCH = "imx-1.12.x"

SRC_URI_remove_imx = "http://gstreamer.freedesktop.org/src/gstreamer/gstreamer-${PV}.tar.xz " SRC_URI_append_imx = " ${GST1.0_SRC};branch=${SRCBRANCH}"
SRCREV_imx = "cca1ebf223bd423ed5fc21031014ee1f3ce497a1"

EXTRA_AUTORECONF = ""

DEPENDS += " elfutils"

# Unrecognised options, need to remove them EXTRA_OECONF_remove = " --disable-docbook --disable-trace"

S_imx = "${WORKDIR}/git"

"

#5 lore raw

Hi Martin,

I am still stuck with this issue,
I understood that the autogen.sh is the point where it wants to update the common submodule.

*Even if BB_NO_NETWORK is set it still tries to access the* https://anongit.freedesktop.org/git/gstreamer/common.git ( https://anongit.freedesktop.org/git/gstreamer/common.git ) Which I think is quite surprising!

But about the following lines, I don't know how to do the described thing.

Quoted 7 lines

Usually it was failing in do_configure when building without access to this git repo, *we were adding "common" git repo to SRC_URI explicitly to make sure that the right revision is always there with the gstreamer sources and managed with bitbake fetcher (so that it's used from downloads directory instead of the fetch from internet every single time gst* recipes are rebuilt for whatever reason)*.

* What should i copy to my local? the content of https://anongit.freedesktop.org/git/gstreamer/common.git ??? shall i copy one file or complete directory
* And then in which file shall i add this url? /home/aselsan/imx-yocto-bsp/sources/poky/meta/recipes-multimedia/gstreamer/gstreamer1.0_1.12.2.bb ? or in a new recipe
* And is it enough just to add the path to SRC_URI or shall i also add the file also ?
Regards,