Fetch problems with PREMIRRORS
Per Richard's suggestion, I have added this to my DISTRO.conf:
# Provide pre-staged sources
PREMIRRORS = "\
http://.*/.* file:///work/misc/Poky/sources/ \n \
https://.*/.* file:///work/misc/Poky/sources/ \n \
ftp://.*/.* file:///work/misc/Poky/sources/ \n \ bzr://.*/.* file:///work/misc/Poky/sources/ \n \ cvs://.*/.* file:///work/misc/Poky/sources/ \n \ git://.*/.* file:///work/misc/Poky/sources/ \n \ hg://.*/.* file:///work/misc/Poky/sources/ \n \ osc://.*/.* file:///work/misc/Poky/sources/ \n \ p4://.*/.* file:///work/misc/Poky/sources/ \n \ svk://.*/.* file:///work/misc/Poky/sources/ \n \ svn://.*/.* file:///work/misc/Poky/sources/ \n"
The tree '/work/misc/Poky/sources' is a fully populated source cache.
This all works great as long as the needed sources are in the cache. However, there seems to be no fallback if they are not present. I get errors like this:
| NOTE: fetch http://pokylinux.org/sources/git2_github.com.wrpseudo.pseudo.git.tar.gz | NOTE: fetch http://autobuilder.pokylinux.org/sources/git2_github.com.wrpseudo.pseudo.git.tar.gz | ERROR: Function 'Fetcher failure for URL: 'git://github.com/wrpseudo/pseudo.git;protocol=git'. Unable to fetch URL git://github.com/wrpseudo/pseudo.git;protocol=git from any source.' failed
NOTE: package pseudo-native-0.0+git1+5434325fc887817ebb2bad36313d8277992cef1d-r18: task Fetcher failure for URL: 'git://github.com/wrpseudo/pseudo.git;protocol=git'. Unable to fetch URL git://github.com/wrpseudo/pseudo.git;protocol=git from any source.: Failed
In order to actually fetch, I have to remove the PREMIRRORS from my conf.
Is there some way to get it to use the source cache but still fetch when needed?
Thanks
Note: I also normally have BB_NO_NETWORK="1", but that is not applicable here (no message about it) and setting it false does not change the behaviour.
Signature · 4 lines
------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------
Quoted 32 lines · Gary Thomas
Per Richard's suggestion, I have added this to my DISTRO.conf:
# Provide pre-staged sources
PREMIRRORS = "\
http://.*/.* file:///work/misc/Poky/sources/ \n \
https://.*/.* file:///work/misc/Poky/sources/ \n \ftp://.*/.* file:///work/misc/Poky/sources/ \n \ bzr://.*/.* file:///work/misc/Poky/sources/ \n \ cvs://.*/.* file:///work/misc/Poky/sources/ \n \ git://.*/.* file:///work/misc/Poky/sources/ \n \ hg://.*/.* file:///work/misc/Poky/sources/ \n \ osc://.*/.* file:///work/misc/Poky/sources/ \n \ p4://.*/.* file:///work/misc/Poky/sources/ \n \ svk://.*/.* file:///work/misc/Poky/sources/ \n \ svn://.*/.* file:///work/misc/Poky/sources/ \n"The tree '/work/misc/Poky/sources' is a fully populated source cache.
This all works great as long as the needed sources are in the cache. However, there seems to be no fallback if they are not present. I get errors like this:
| NOTE: fetch http://pokylinux.org/sources/git2_github.com.wrpseudo.pseudo.git.tar.gz | NOTE: fetch http://autobuilder.pokylinux.org/sources/git2_github.com.wrpseudo.pseudo.git.tar.gz | ERROR: Function 'Fetcher failure for URL: 'git://github.com/wrpseudo/pseudo.git;protocol=git'. Unable to fetch URL git://github.com/wrpseudo/pseudo.git;protocol=git from anysource.' failed
NOTE: package pseudo-native-0.0+git1+5434325fc887817ebb2bad36313d8277992cef1d-r18: task Fetcher failure for URL: 'git://github.com/wrpseudo/pseudo.git;protocol=git'. Unable to fetch URLgit://github.com/wrpseudo/pseudo.git;protocol=gitfrom any source.: FailedIn order to actually fetch, I have to remove the PREMIRRORS from my conf.
Quoted 8 lines
Is there some way to get it to use the source cache but still fetch when needed?
Thanks
Note: I also normally have BB_NO_NETWORK="1", but that is not applicable here (no message about it) and setting it false does not change the behaviour.
It is applicable.
Try not setting that variable at all, not setting it false. I suspect there are two issues here:
a) BB_NO_NETWORK is having an effect but the log messages are being swallowed...
b) "false" handling for the variable might not be working properly so leave it unset.
We can fix those things but I'd like to confirm these are the problems first.
Cheers,
Richard
Quoted 47 lines · Richard Purdie
Quoted 32 lines · Gary Thomas
Per Richard's suggestion, I have added this to my DISTRO.conf:
# Provide pre-staged sources
PREMIRRORS = "\
http://.*/.* file:///work/misc/Poky/sources/ \n \
https://.*/.* file:///work/misc/Poky/sources/ \n \ftp://.*/.* file:///work/misc/Poky/sources/ \n \ bzr://.*/.* file:///work/misc/Poky/sources/ \n \ cvs://.*/.* file:///work/misc/Poky/sources/ \n \ git://.*/.* file:///work/misc/Poky/sources/ \n \ hg://.*/.* file:///work/misc/Poky/sources/ \n \ osc://.*/.* file:///work/misc/Poky/sources/ \n \ p4://.*/.* file:///work/misc/Poky/sources/ \n \ svk://.*/.* file:///work/misc/Poky/sources/ \n \ svn://.*/.* file:///work/misc/Poky/sources/ \n"The tree '/work/misc/Poky/sources' is a fully populated source cache.
This all works great as long as the needed sources are in the cache. However, there seems to be no fallback if they are not present. I get errors like this:
| NOTE: fetch http://pokylinux.org/sources/git2_github.com.wrpseudo.pseudo.git.tar.gz | NOTE: fetch http://autobuilder.pokylinux.org/sources/git2_github.com.wrpseudo.pseudo.git.tar.gz | ERROR: Function 'Fetcher failure for URL: 'git://github.com/wrpseudo/pseudo.git;protocol=git'. Unable to fetch URL git://github.com/wrpseudo/pseudo.git;protocol=git from anysource.' failed
NOTE: package pseudo-native-0.0+git1+5434325fc887817ebb2bad36313d8277992cef1d-r18: task Fetcher failure for URL: 'git://github.com/wrpseudo/pseudo.git;protocol=git'. Unable to fetch URLgit://github.com/wrpseudo/pseudo.git;protocol=gitfrom any source.: FailedIn order to actually fetch, I have to remove the PREMIRRORS from my conf.
Quoted 8 lines
Is there some way to get it to use the source cache but still fetch when needed?
Thanks
Note: I also normally have BB_NO_NETWORK="1", but that is not applicable here (no message about it) and setting it false does not change the behaviour.
It is applicable.
Try not setting that variable at all, not setting it false. I suspect there are two issues here:
Quoted 7 lines
a)
BB_NO_NETWORKis having an effect but the log messages are being swallowed...
b) "false" handling for the variable might not be working properly so leave it unset.We can fix those things but I'd like to confirm these are the problems first.
Correct, it does seem to be the case. When BB_NO_NETWORK="0", the fetcher works properly. When set to "1", it fails, but no explicit error about why is generated, neither on the console nor in the log.do_fetch file.
I made this small change to check it out:
bitbake/lib/bb/fetch2/__init__.py +1 -0
@@ -395,6 +395,7 @@ def check_network_access(d, info = ""): log remote network access, and error if BB_NO_NETWORK is set """ if bb.data.getVar("BB_NO_NETWORK", d, True) == "1": + bb.error("BB_NO_NETWORK is set, but the fetcher code attempted network access with the command %s" % info) raise FetchError("BB_NO_NETWORK is set, but the fetcher code attempted network access with the command %s" % inf else: logger.debug(1, "Fetcher accessed the network with the command %s" % info)
This shows that this _is_ the error case, with the messages making it to the console and log file. I can see that this exception is being thrown on line 886, but the reason is lost in the exception data & I'm unclear how to retrieve it and present it to the user.
Signature · 4 lines
------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------
Quoted 30 lines · Gary Thomas
Quoted 10 lines
Try not setting that variable at all, not setting it false. I suspect there are two issues here:
a)
BB_NO_NETWORKis having an effect but the log messages are being swallowed...
b) "false" handling for the variable might not be working properly so leave it unset.We can fix those things but I'd like to confirm these are the problems first.
Correct, it does seem to be the case. When BB_NO_NETWORK="0", the fetcher works properly. When set to "1", it fails, but no explicit error about why is generated, neither on the console nor in the log.do_fetch file.
I made this small change to check it out:
bitbake/lib/bb/fetch2/__init__.py +1
-0@@ -395,6 +395,7 @@ def check_network_access(d, info = ""): log remote network access, and error if BB_NO_NETWORK is set """ if bb.data.getVar("BB_NO_NETWORK", d, True) == "1": + bb.error("BB_NO_NETWORK is set, but the fetcher code attempted network access with the command %s" % info) raise FetchError("BB_NO_NETWORK is set, but the fetcher code attempted network access with the command %s" % inf else: logger.debug(1, "Fetcher accessed the network with the command %s" % info)
Quoted 4 lines
This shows that this _is_ the error case, with the messages making it to the console and log file. I can see that this exception is being thrown on line 886, but the reason is lost in the exception data & I'm unclear how to retrieve it and present it to the user.
This should be at least partly addressed with some of the recent commits such as:
http://git.pokylinux.org/cgit.cgi/poky/commit/?id=ea70c4362fdb81bc9467975c145c48196c45e3af
so the debug logs (with -D) at least show whats happening.
Not perfect but better at least...
Cheers,
Richard