Skip to contentSkip to conversation

Fetch problems with PREMIRRORS

DISCUSSION Gary Thomas 2 participants 4 messages Feb 9, 2011 – Feb 11, 2011

lore
Share & tools
lore
#0 lore raw

Per Richard's suggestion, I have added this to my DISTRO.conf:

# Provide pre-staged sources
PREMIRRORS = "\
http://.*/.* file:///work/misc/Poky/sources/ \n \
https://.*/.* file:///work/misc/Poky/sources/ \n \

ftp://.*/.*   file:///work/misc/Poky/sources/ \n \
bzr://.*/.*   file:///work/misc/Poky/sources/ \n \
cvs://.*/.*   file:///work/misc/Poky/sources/ \n \
git://.*/.*   file:///work/misc/Poky/sources/ \n \
hg://.*/.*    file:///work/misc/Poky/sources/ \n \
osc://.*/.*   file:///work/misc/Poky/sources/ \n \
p4://.*/.*    file:///work/misc/Poky/sources/ \n \
svk://.*/.*   file:///work/misc/Poky/sources/ \n \
svn://.*/.*   file:///work/misc/Poky/sources/ \n"

The tree '/work/misc/Poky/sources' is a fully populated source cache.

This all works great as long as the needed sources are in the cache. However, there seems to be no fallback if they are not present. I get errors like this:

| NOTE: fetch http://pokylinux.org/sources/git2_github.com.wrpseudo.pseudo.git.tar.gz
| NOTE: fetch http://autobuilder.pokylinux.org/sources/git2_github.com.wrpseudo.pseudo.git.tar.gz
| ERROR: Function 'Fetcher failure for URL: 'git://github.com/wrpseudo/pseudo.git;protocol=git'. Unable to fetch URL git://github.com/wrpseudo/pseudo.git;protocol=git from any source.' failed

NOTE: package pseudo-native-0.0+git1+5434325fc887817ebb2bad36313d8277992cef1d-r18: task Fetcher failure for URL: 'git://github.com/wrpseudo/pseudo.git;protocol=git'. Unable to fetch URL git://github.com/wrpseudo/pseudo.git;protocol=git from any source.: Failed

In order to actually fetch, I have to remove the PREMIRRORS from my conf.

Is there some way to get it to use the source cache but still fetch when needed?

Thanks

Note: I also normally have BB_NO_NETWORK="1", but that is not applicable here (no message about it) and setting it false does not change the behaviour.

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#1 lore raw
Quoted 32 lines · Gary Thomas

Per Richard's suggestion, I have added this to my DISTRO.conf:

# Provide pre-staged sources
PREMIRRORS = "\
http://.*/.* file:///work/misc/Poky/sources/ \n \
https://.*/.* file:///work/misc/Poky/sources/ \n \

ftp://.*/.*   file:///work/misc/Poky/sources/ \n \
bzr://.*/.*   file:///work/misc/Poky/sources/ \n \
cvs://.*/.*   file:///work/misc/Poky/sources/ \n \
git://.*/.*   file:///work/misc/Poky/sources/ \n \
hg://.*/.*    file:///work/misc/Poky/sources/ \n \
osc://.*/.*   file:///work/misc/Poky/sources/ \n \
p4://.*/.*    file:///work/misc/Poky/sources/ \n \
svk://.*/.*   file:///work/misc/Poky/sources/ \n \
svn://.*/.*   file:///work/misc/Poky/sources/ \n"

The tree '/work/misc/Poky/sources' is a fully populated source cache.

This all works great as long as the needed sources are in the cache. However, there seems to be no fallback if they are not present. I get errors like this:

| NOTE: fetch http://pokylinux.org/sources/git2_github.com.wrpseudo.pseudo.git.tar.gz
| NOTE: fetch http://autobuilder.pokylinux.org/sources/git2_github.com.wrpseudo.pseudo.git.tar.gz
| ERROR: Function 'Fetcher failure for URL: 'git://github.com/wrpseudo/pseudo.git;protocol=git'. Unable to fetch URL git://github.com/wrpseudo/pseudo.git;protocol=git from any 

source.' failed
NOTE: package pseudo-native-0.0+git1+5434325fc887817ebb2bad36313d8277992cef1d-r18: task Fetcher failure for URL: 'git://github.com/wrpseudo/pseudo.git;protocol=git'. Unable to fetch URL git://github.com/wrpseudo/pseudo.git;protocol=git from any source.: Failed

In order to actually fetch, I have to remove the PREMIRRORS from my conf.

Quoted 8 lines

Is there some way to get it to use the source cache but still fetch when needed?

Thanks

Note: I also normally have BB_NO_NETWORK="1", but that is not applicable here (no message about it) and setting it false does not change the behaviour.

It is applicable.

Try not setting that variable at all, not setting it false. I suspect there are two issues here:

a) BB_NO_NETWORK is having an effect but the log messages are being swallowed...
b) "false" handling for the variable might not be working properly so leave it unset.

We can fix those things but I'd like to confirm these are the problems first.

Cheers,

Richard

#2 lore raw
Quoted 47 lines · Richard Purdie
Quoted 32 lines · Gary Thomas

Per Richard's suggestion, I have added this to my DISTRO.conf:

# Provide pre-staged sources
PREMIRRORS = "\
http://.*/.* file:///work/misc/Poky/sources/ \n \
https://.*/.* file:///work/misc/Poky/sources/ \n \

ftp://.*/.*   file:///work/misc/Poky/sources/ \n \
bzr://.*/.*   file:///work/misc/Poky/sources/ \n \
cvs://.*/.*   file:///work/misc/Poky/sources/ \n \
git://.*/.*   file:///work/misc/Poky/sources/ \n \
hg://.*/.*    file:///work/misc/Poky/sources/ \n \
osc://.*/.*   file:///work/misc/Poky/sources/ \n \
p4://.*/.*    file:///work/misc/Poky/sources/ \n \
svk://.*/.*   file:///work/misc/Poky/sources/ \n \
svn://.*/.*   file:///work/misc/Poky/sources/ \n"

The tree '/work/misc/Poky/sources' is a fully populated source cache.

This all works great as long as the needed sources are in the cache. However, there seems to be no fallback if they are not present. I get errors like this:

| NOTE: fetch http://pokylinux.org/sources/git2_github.com.wrpseudo.pseudo.git.tar.gz
| NOTE: fetch http://autobuilder.pokylinux.org/sources/git2_github.com.wrpseudo.pseudo.git.tar.gz
| ERROR: Function 'Fetcher failure for URL: 'git://github.com/wrpseudo/pseudo.git;protocol=git'. Unable to fetch URL git://github.com/wrpseudo/pseudo.git;protocol=git from any

source.' failed
NOTE: package pseudo-native-0.0+git1+5434325fc887817ebb2bad36313d8277992cef1d-r18: task Fetcher failure for URL: 'git://github.com/wrpseudo/pseudo.git;protocol=git'. Unable to fetch URL git://github.com/wrpseudo/pseudo.git;protocol=git from any source.: Failed

In order to actually fetch, I have to remove the PREMIRRORS from my conf.

Quoted 8 lines

Is there some way to get it to use the source cache but still fetch when needed?

Thanks

Note: I also normally have BB_NO_NETWORK="1", but that is not applicable here (no message about it) and setting it false does not change the behaviour.

It is applicable.

Try not setting that variable at all, not setting it false. I suspect there are two issues here:

Quoted 7 lines

a) BB_NO_NETWORK is having an effect but the log messages are being swallowed...
b) "false" handling for the variable might not be working properly so leave it unset.

We can fix those things but I'd like to confirm these are the problems first.

Correct, it does seem to be the case. When BB_NO_NETWORK="0", the fetcher works properly. When set to "1", it fails, but no explicit error about why is generated, neither on the console nor in the log.do_fetch file.

I made this small change to check it out:

bitbake/lib/bb/fetch2/__init__.py +1 -0
index ef9d75f..57c4f20 100644
@@ -395,6 +395,7 @@ def check_network_access(d, info = ""):
     log remote network access, and error if BB_NO_NETWORK is set
     """
     if bb.data.getVar("BB_NO_NETWORK", d, True) == "1":
+        bb.error("BB_NO_NETWORK is set, but the fetcher code attempted network access with the command %s" % info)
         raise FetchError("BB_NO_NETWORK is set, but the fetcher code attempted network access with the command %s" % inf
     else:
         logger.debug(1, "Fetcher accessed the network with the command %s" % info)

This shows that this _is_ the error case, with the messages making it to the console and log file. I can see that this exception is being thrown on line 886, but the reason is lost in the exception data & I'm unclear how to retrieve it and present it to the user.

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#3 lore raw
Quoted 30 lines · Gary Thomas
Quoted 10 lines

Try not setting that variable at all, not setting it false. I suspect there are two issues here:

a) BB_NO_NETWORK is having an effect but the log messages are being swallowed...
b) "false" handling for the variable might not be working properly so leave it unset.

We can fix those things but I'd like to confirm these are the problems first.

Correct, it does seem to be the case. When BB_NO_NETWORK="0", the fetcher works properly. When set to "1", it fails, but no explicit error about why is generated, neither on the console nor in the log.do_fetch file.

I made this small change to check it out:

bitbake/lib/bb/fetch2/__init__.py +1 -0
index ef9d75f..57c4f20 100644
@@ -395,6 +395,7 @@ def check_network_access(d, info = ""):
      log remote network access, and error if BB_NO_NETWORK is set
      """
      if bb.data.getVar("BB_NO_NETWORK", d, True) == "1":
+        bb.error("BB_NO_NETWORK is set, but the fetcher code attempted network access with the command %s" % info)
          raise FetchError("BB_NO_NETWORK is set, but the fetcher code attempted network access with the command %s" % inf
      else:
          logger.debug(1, "Fetcher accessed the network with the command %s" % info)
Quoted 4 lines

This shows that this _is_ the error case, with the messages making it to the console and log file. I can see that this exception is being thrown on line 886, but the reason is lost in the exception data & I'm unclear how to retrieve it and present it to the user.

This should be at least partly addressed with some of the recent commits such as:

http://git.pokylinux.org/cgit.cgi/poky/commit/?id=ea70c4362fdb81bc9467975c145c48196c45e3af

so the debug logs (with -D) at least show whats happening.

Not perfect but better at least...

Cheers,

Richard