Suggestion to improving buildroot download infra
Hi,
Is there a way to integrate parallel wget download into buildroot?
That would greatly speed the downloading process when doing "make source" and maybe even later all the downloading could be done in the background, even
when compiling some big package (for example, gentoo linux does that).
If I understand it correctly, there are at least two (if not including the crude & fork method)
ways to do parallel wget:
1. use xarg:
echo $URL_LIST | xargs -n 1 -P ${NUMBER_OF_PARALLEL_DOWNLOADS} wget
or 2. use GNU parallel
parallel -a $URL_LIST --jobs ${NUMBER_OF_PARALLEL_DOWNLOADS} wget
How exactly, does buildroot collect the download urls when doing example, "make source" ? Does it collect them first into some list and then do wget -i list ?
Or does it just go each selected packet one-by-one and fetch the url for each?
-S-
Hello,
Quoted 1 line · Stefan Fr?berg
Is there a way to integrate parallel wget download into buildroot?
This will be a direct consequence of top-level parallel build support. Not only parallel download, but downloading package A, while package B is being built, etc.
Quoted 15 lines
That would greatly speed the downloading process when doing "make source" and maybe even later all the downloading could be done in the background, even
when compiling some big package (for example, gentoo linux does that).If I understand it correctly, there are at least two (if not including the crude & fork method)
ways to do parallel wget:1. use xarg:
echo $URL_LIST | xargs -n 1 -P ${NUMBER_OF_PARALLEL_DOWNLOADS} wget
or 2. use GNU parallel
Quoted 7 lines
parallel -a $URL_LIST --jobs ${NUMBER_OF_PARALLEL_DOWNLOADS} wget
How exactly, does buildroot collect the download urls when doing example, "make source" ? Does it collect them first into some list and then do wget -i list ?
Or does it just go each selected packet one-by-one and fetch the url for each?
For the moment, the easiest is to comment the .NOTPARALLEL: statement in the main Makefile, and then run:
make -j8 source
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
Hi
Ah, okay. I will try it soon and see what packages will break. Thanks :-)
-S-
15.11.2017, 16:50, Thomas Petazzoni kirjoitti:
Quoted 39 lines
Hello,
Quoted 1 line · Stefan Fr?berg
Is there a way to integrate parallel wget download into buildroot?
This will be a direct consequence of top-level parallel build support. Not only parallel download, but downloading package A, while package B is being built, etc.
Quoted 15 lines
That would greatly speed the downloading process when doing "make source" and maybe even later all the downloading could be done in the background, even
when compiling some big package (for example, gentoo linux does that).If I understand it correctly, there are at least two (if not including the crude & fork method)
ways to do parallel wget:1. use xarg:
echo $URL_LIST | xargs -n 1 -P ${NUMBER_OF_PARALLEL_DOWNLOADS} wget
or 2. use GNU parallel
Quoted 7 lines
parallel -a $URL_LIST --jobs ${NUMBER_OF_PARALLEL_DOWNLOADS} wget
How exactly, does buildroot collect the download urls when doing example, "make source" ? Does it collect them first into some list and then do wget -i list ?
Or does it just go each selected packet one-by-one and fetch the url for each?For the moment, the easiest is to comment the .NOTPARALLEL: statement in the main
Makefile, and then run:make -j8 source
Best regards,
Thomas
Quoted 16 lines · Stefan Fr?berg
Hi,
Is there a way to integrate parallel wget download into buildroot?
That would greatly speed the downloading process when doing "make source" and maybe even later all the downloading could be done in the background, even when compiling some big package (for example, gentoo linux does that).
If I understand it correctly, there are at least two (if not including the crude & fork method)
ways to do parallel wget:1. use xarg:
echo $URL_LIST | xargs -n 1 -P ${NUMBER_OF_PARALLEL_DOWNLOADS} wget
Quoted 8 lines
or 2. use GNU parallel
parallel -a $URL_LIST --jobs ${NUMBER_OF_PARALLEL_DOWNLOADS} wget
How exactly, does buildroot collect the download urls when doing example, "make source" ? Does it collect them first into some list and then do wget -i list ?
Or does it just go each selected packet one-by-one and fetch the url for each?
For each package, the <pkg>-source target is triggered. This target depends on
<pkg-build-dir>/.stamp_downloaded. If that stamp file does not exist yet, the download is triggered.
Since it is all done with make rules, there is no way to do things in parallel
except for doing the complete make invocation in parallel (with -j).
Regards,
Arnout
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF