Fetch time optimization (svn : gcc/eglibc - git : linux-yocto)
Hi,
I noticed in from scratch builds for qemuarm that the longest time is taken in fetching sources, especially those fetched using git (linux-yocto for example) & svn (gcc, eglibc & co).
To reduce the fetch time would that make sense to
- fetch gcc/glibc & co from the archive of a stable version and then apply patches on top of it (maybe patches stored in an archive fetched from oe's website and applied in bulk or patches stored in OE)
- do the same thing for the linux-yocto kernel or add a --reference option to the git fetcher so that we can provide a local tree as a reference ?
Do you have other ideas (appart from using a local mirror) to optimize the fetch time ?
Thanks,
Eric
Quoted 3 lines · Eric Bénard
I noticed in from scratch builds for qemuarm that the longest time is taken in fetching sources, especially those fetched using git (linux-yocto for example) & svn (gcc, eglibc & co).
Are you timing these as fetches from the source control systems or from the mirror tarballs of the repositories. The tarballs should be faster...
Quoted 4 lines
To reduce the fetch time would that make sense to
- fetch gcc/glibc & co from the archive of a stable version and then apply patches on top of it (maybe patches stored in an archive fetched from oe's website and applied in bulk or patches stored in OE)
Unfortunately the patches tend to get unwieldy. The tarballs of the svn repos on the mirror should be about equal in size to the upstream archive in this case.
Quoted 3 lines
- do the same thing for the linux-yocto kernel or add a --reference option to the git fetcher so that we can provide a local tree as a reference ?
This is effectively how the repositories in DL_DIR are used. If you place a tree in the right place there, it should reuse references...
Quoted 2 lines
Do you have other ideas (appart from using a local mirror) to optimize the fetch time ?
I'd be interested firstly to understand if you're using the SCM directly or using the mirror tarballs as that should make a big difference. In the standard configuration it should be using mirror tarballs...
Cheers,
Richard
Quoted 18 lines · Richard Purdie
Quoted 3 lines · Eric Bénard
I noticed in from scratch builds for qemuarm that the longest time is taken in fetching sources, especially those fetched using git (linux-yocto for example) & svn (gcc, eglibc & co).
Are you timing these as fetches from the source control systems or from the mirror tarballs of the repositories. The tarballs should be faster...
Quoted 4 lines
To reduce the fetch time would that make sense to
- fetch gcc/glibc & co from the archive of a stable version and then apply patches on top of it (maybe patches stored in an archive fetched from oe's website and applied in bulk or patches stored in OE)Unfortunately the patches tend to get unwieldy. The tarballs of the svn repos on the mirror should be about equal in size to the upstream archive in this case.
Quoted 6 lines
Quoted 3 lines
- do the same thing for the linux-yocto kernel or add a --reference option to the git fetcher so that we can provide a local tree as a reference ?
This is effectively how the repositories in
DL_DIRare used. If you place a tree in the right place there, it should reuse references...
Agreed .. they definitely do here.
Richard probably recalls me asking for a --reference option several years ago as well .. but in the end, at some point the initial fetch happens and then the blobs are re-used. So setting up local mirrors, or pre-fetching are options to make sure that the first download is primed and ready to go. For most builds I do, any time fetching just happens in the background and doesn't get in the way.
Quoted 7 lines
Quoted 3 lines
Do you have other ideas (appart from using a local mirror) to optimize the fetch time ?
I'd be interested firstly to understand if you're using the SCM directly or using the mirror tarballs as that should make a big difference. In the standard configuration it should be using mirror tarballs...
As would I, since there are some ideas, but they either break workflows, don't follow best practices or compromise the completeness of the data.
Cheers,
Bruce
Quoted 10 lines
Cheers,
Richard
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"
2012/3/30 Bruce Ashfield <[email protected]>:
Quoted 64 lines
Quoted 18 lines · Richard Purdie
Quoted 3 lines · Eric Bénard
I noticed in from scratch builds for qemuarm that the longest time is taken in fetching sources, especially those fetched using git (linux-yocto for example) & svn (gcc, eglibc & co).
Are you timing these as fetches from the source control systems or from the mirror tarballs of the repositories. The tarballs should be faster...
Quoted 4 lines
To reduce the fetch time would that make sense to
- fetch gcc/glibc & co from the archive of a stable version and then apply patches on top of it (maybe patches stored in an archive fetched from oe's website and applied in bulk or patches stored in OE)Unfortunately the patches tend to get unwieldy. The tarballs of the svn repos on the mirror should be about equal in size to the upstream archive in this case.
Quoted 6 lines
Quoted 3 lines
- do the same thing for the linux-yocto kernel or add a --reference option to the git fetcher so that we can provide a local tree as a reference ?
This is effectively how the repositories in
DL_DIRare used. If you place a tree in the right place there, it should reuse references...Agreed .. they definitely do here.
Richard probably recalls me asking for a --reference option several years ago as well .. but in the end, at some point the initial fetch happens and then the blobs are re-used. So setting up local mirrors, or pre-fetching are options to make sure that the first download is primed and ready to go. For most builds I do, any time fetching just happens in the background and doesn't get in the way.
Quoted 7 lines
Quoted 3 lines
Do you have other ideas (appart from using a local mirror) to optimize the fetch time ?
I'd be interested firstly to understand if you're using the SCM directly or using the mirror tarballs as that should make a big difference. In the standard configuration it should be using mirror tarballs...
As would I, since there are some ideas, but they either break workflows, don't follow best practices or compromise the completeness of the data.
Cheers,
Bruce
Quoted 10 lines
Cheers,
Richard
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Quoted 8 lines
--
"Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Hi,
this might be a bit off-topic, but another idea would be to add a separate threading mechanism for fetching.
Current threading can help to use the CPU and memory load to it's optimum, but sometimes you have to wait for a download to finish.. Instead there could be a separate set of threads that only download the sources and make optimal use of the bandwidth too.
This would also allow to fetch files when the normal threads are busy with configuring/building/packaging recipes.
The downside would be that it requires some sort of inter process communication.
Or it could be regulated with a simple check if the download is finished..
How does this idea sound to you?
Signature · 2 lines
Regards Samuel
Quoted 10 lines · On Friday 30 March 2012 08:44:56 Samuel Stirtzel wrote:
this might be a bit off-topic, but another idea would be to add a separate threading mechanism for fetching.
Current threading can help to use the CPU and memory load to it's optimum, but sometimes you have to wait for a download to finish.. Instead there could be a separate set of threads that only download the sources and make optimal use of the bandwidth too.
This would also allow to fetch files when the normal threads are busy with configuring/building/packaging recipes.
What you're really suggesting here is a modified BitBake scheduler that understands that fetch tasks that require network bandwidth are different from other tasks such as compile ones which stress the CPU. It sounds like it might be worth investigating at least. FYI, BitBake's schedulers are pluggable and not particularly complicated (see bitbake/lib/bb/runqueue.py).
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 12 lines · Samuel Stirtzel
this might be a bit off-topic, but another idea would be to add a separate threading mechanism for fetching.
Current threading can help to use the CPU and memory load to it's optimum, but sometimes you have to wait for a download to finish.. Instead there could be a separate set of threads that only download the sources and make optimal use of the bandwidth too.
This would also allow to fetch files when the normal threads are busy with configuring/building/packaging recipes.
Quoted 5 lines
The downside would be that it requires some sort of inter process communication.
Or it could be regulated with a simple check if the download is finished..How does this idea sound to you?
Its easier than you think to do this, bitbake has a plugable scheduler implementation so you'd just have to write one which ignores "fetch" operations from the total thread count.
Sadly this isn't really the place most people have a bottleneck in day to day usage of the system. People have tried various algorithms for enhancing the scheduler and as far as I know never found anything that makes a significant difference, much to everyone's surprise :/.
Cheers,
Richard
2012/3/30 Richard Purdie <[email protected]>:
Quoted 30 lines · Samuel Stirtzel
Quoted 12 lines
this might be a bit off-topic, but another idea would be to add a separate threading mechanism for fetching.
Current threading can help to use the CPU and memory load to it's optimum, but sometimes you have to wait for a download to finish.. Instead there could be a separate set of threads that only download the sources and make optimal use of the bandwidth too.
This would also allow to fetch files when the normal threads are busy with configuring/building/packaging recipes.
Quoted 5 lines
The downside would be that it requires some sort of inter process communication.
Or it could be regulated with a simple check if the download is finished..How does this idea sound to you?
Its easier than you think to do this, bitbake has a plugable scheduler implementation so you'd just have to write one which ignores "fetch" operations from the total thread count.
Sadly this isn't really the place most people have a bottleneck in day to day usage of the system. People have tried various algorithms for enhancing the scheduler and as far as I know never found anything that makes a significant difference, much to everyone's surprise :/.
Cheers,
Quoted 8 lines
Richard
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Of course this will only reduce the time of recipes if they are build for the first time,
or when the version/URL changes.
It is not that important, I agree,
but it would improve the situation for first time users, or new installations.
Example for 2 threads:
http://pastebin.com/kviwQZJ3
It is very likely that the current situation also uses cpu and network resources at the same time,
but it might occur that the build-task has to wait for a download to finish or vice versa.
Ignoring fetch tasks from the thread count would only do half of the job and _could_ cause network bottlenecks ;)
Fetching should be "independent" from the dependency chain.
E.g.: it should not wait with the downloads for dependencies to finish building, the download sequence should still match the dependency chain sequence.
If it is really that easy, then I will look into it.
Signature · 2 lines
Regards Samuel
Quoted 13 lines · Samuel Stirtzel
Of course this will only reduce the time of recipes if they are build for the first time,
or when the version/URL changes.
It is not that important, I agree,
but it would improve the situation for first time users, or new installations.Example for 2 threads:
http://pastebin.com/kviwQZJ3
It is very likely that the current situation also uses cpu and network resources at the same time,
but it might occur that the build-task has to wait for a download to finish or vice versa.
Quoted 3 lines
Ignoring fetch tasks from the thread count would only do half of the job and _could_ cause network bottlenecks ;)
Fetching should be "independent" from the dependency chain.
This simply isn't true and there is also no benefit to splitting them to be independent. The fetch tasks have dependencies just like any other task (for example git:// urls depend on git-native being built unless its in ASSUME_PROVIDED).
Quoted 2 lines
E.g.: it should not wait with the downloads for dependencies to finish building, the download sequence should still match the dependency chain sequence.
I'm afraid I don't understand what you mean. I think you will find that if you exclude the "fetch" tasks from the normal "cpu" thread count you will get the behaviour you are describing.
Cheers,
Richard
2012/3/30 Richard Purdie <[email protected]>:
Quoted 14 lines · Samuel Stirtzel
Quoted 13 lines
Of course this will only reduce the time of recipes if they are build for the first time,
or when the version/URL changes.
It is not that important, I agree,
but it would improve the situation for first time users, or new installations.Example for 2 threads:
http://pastebin.com/kviwQZJ3
It is very likely that the current situation also uses cpu and network resources at the same time,
but it might occur that the build-task has to wait for a download to finish or vice versa.
Quoted 8 lines
Quoted 3 lines
Ignoring fetch tasks from the thread count would only do half of the job and _could_ cause network bottlenecks ;)
Fetching should be "independent" from the dependency chain.This simply isn't true and there is also no benefit to splitting them to be independent. The fetch tasks have dependencies just like any other task (for example
git://urls depend on git-native being built unless its inASSUME_PROVIDED).
You are right, my mistake.
Adding some line like PARALLEL_FETCH to the config will do the rest.
Quoted 7 lines
Quoted 3 lines
E.g.: it should not wait with the downloads for dependencies to finish building, the download sequence should still match the dependency chain sequence.
I'm afraid I don't understand what you mean. I think you will find that if you exclude the "fetch" tasks from the normal "cpu" thread count you will get the behaviour you are describing.
I was erroneously assuming that the download only starts after all dependencies finished building,
but of course this was only derivated as the threads where blocked by the build tasks.
So yes the method you mentioned will work.
Quoted 10 lines
Cheers,
Richard
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Signature · 2 lines
Regards Samuel
Quoted 20 lines · Bruce Ashfield
Quoted 18 lines · Richard Purdie
Quoted 3 lines · Eric Bénard
I noticed in from scratch builds for qemuarm that the longest time is taken in fetching sources, especially those fetched using git (linux-yocto for example) & svn (gcc, eglibc & co).
Are you timing these as fetches from the source control systems or from the mirror tarballs of the repositories. The tarballs should be faster...
Quoted 4 lines
To reduce the fetch time would that make sense to
- fetch gcc/glibc & co from the archive of a stable version and then apply patches on top of it (maybe patches stored in an archive fetched from oe's website and applied in bulk or patches stored in OE)Unfortunately the patches tend to get unwieldy. The tarballs of the svn repos on the mirror should be about equal in size to the upstream archive in this case.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
- do the same thing for the linux-yocto kernel or add a --reference option to the git fetcher so that we can provide a local tree as a reference ?
This is effectively how the repositories in
DL_DIRare used. If you place a tree in the right place there, it should reuse references...Agreed .. they definitely do here.
What's right place?
I guess the idea was to use --reference for e.g. some other kernel recipe sources checkout.
And I guess that building linux-foo won't notice that there is e.g. /OE/downloads/git2/gitorious.org.shr.linux.git:
from which it can share a lot of objects using --reference
Bob Ham (rah on #oe) said that he is working on some sort of support for --reference with bitbake after I've refused to add just another linux-bar recipe to meta-smartphone, but not sure how he plans to implement it to be usefull and working oob in different env with different sources available in downloads dir.
Cheers,
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
Quoted 21 lines · Martin Jansa
Quoted 10 lines · Bruce Ashfield
Quoted 6 lines · Richard Purdie
Quoted 3 lines
- do the same thing for the linux-yocto kernel or add a --reference option to the git fetcher so that we can provide a local tree as a reference ?
This is effectively how the repositories in
DL_DIRare used. If you place a tree in the right place there, it should reuse references...Agreed .. they definitely do here.
What's right place?
I guess the idea was to use --reference for e.g. some other kernel recipe sources checkout.
And I guess that building linux-foo won't notice that there is e.g. /OE/downloads/git2/gitorious.org.shr.linux.git:
from which it can share a lot of objects using --reference
Quoted 5 lines
Bob Ham (rah on #oe) said that he is working on some sort of support for --reference with bitbake after I've refused to add just another linux-bar recipe to meta-smartphone, but not sure how he plans to implement it to be usefull and working oob in different env with different sources available in downloads dir.
You could conceivably symlink all your different kernel directories together within git2/. As far as I can tell, the fetcher simply wouldn't care in most cases. The branch structures could get a little mangled I guess and you'd not want to share the resulting mirror tarballs.
There is an argument for using one large shared container for all the git objects as another way of solving this. I don't know how well git has that supported but at the object level its a non-issue at least.
Cheers,
Richard
Quoted 9 lines · Richard Purdie
Quoted 3 lines · Eric Bénard
I noticed in from scratch builds for qemuarm that the longest time is taken in fetching sources, especially those fetched using git (linux-yocto for example) & svn (gcc, eglibc & co).
Are you timing these as fetches from the source control systems or from the mirror tarballs of the repositories. The tarballs should be faster...
the default configuration seems to fetch from source control systems as I always see very long time to fetch gcc/eglibc/linux-yocto (despite having a 2.2 MBytes/s downlink DSL line).
Quoted 9 lines
Quoted 4 lines
To reduce the fetch time would that make sense to
- fetch gcc/glibc & co from the archive of a stable version and then apply patches on top of it (maybe patches stored in an archive fetched from oe's website and applied in bulk or patches stored in OE)Unfortunately the patches tend to get unwieldy. The tarballs of the svn repos on the mirror should be about equal in size to the upstream archive in this case.
I don't think that's a size problem but that fetching through svn or git is far less efficient than http or ftp especially from gnu's svn which may be overloaded.
Morover in a pure OE context we have no interest of all the source history provided by svn or git and that makes a very big volume to download.
Quoted 7 lines
Quoted 3 lines
- do the same thing for the linux-yocto kernel or add a --reference option to the git fetcher so that we can provide a local tree as a reference ?
This is effectively how the repositories in
DL_DIRare used. If you place a tree in the right place there, it should reuse references...
Quoted 7 lines
Quoted 2 lines
Do you have other ideas (appart from using a local mirror) to optimize the fetch time ?
I'd be interested firstly to understand if you're using the SCM directly or using the mirror tarballs as that should make a big difference. In the standard configuration it should be using mirror tarballs...
that doesn't seems to be the case :
from a clean oe-core + bitbake clone :
. ./openembedded-core/oe-init-build-env
edit local.conf to select qemuarm & BBTHREAD to 8
bitbake core-image-minimal -c fetchall
and then I see bitbake stops at around 209 or 214 tasks waiting and
I see that in ps :/home/ebenard/OE-CORE/build/tmp-eglibc/sysroots/x86_64-linux/usr/bin/git.real
clone --bare --mirrorgit://git.yoctoproject.org/linux-yocto-3.2 /home/ebenard/OE-CORE/build/downloads/git2/git.yoctoproject.org.linux-yocto-3.2 and
svn co -r 184847
http://gcc.gnu.org/svn/gcc/branches/gcc-4_6-branch@184847
... and both are actually fetching at only around 200 KiB/s which last for quite a long time as (from an other downloads dir) the final tree size are huge :
du -s git.yoctoproject.org.linux-yocto-3.2/
610824 git.yoctoproject.org.linux-yocto-3.2/
du -s gcc.gnu.org/
1602496 gcc.gnu.org/
du -s www.eglibc.org/
625048 www.eglibc.org
If I launch at the same time :
wget ftp://ftp.gnu.org/gnu/gcc/gcc-4.6.3/gcc-4.6.3.tar.bz2 I get a download speed close to 1MB/s and the file to download is only 64MB which would save bandwidth.
Eric
Quoted 8 lines · Eric Bénard
Quoted 5 lines · Richard Purdie
Quoted 3 lines · Eric Bénard
I noticed in from scratch builds for qemuarm that the longest time is taken in fetching sources, especially those fetched using git (linux-yocto for example) & svn (gcc, eglibc & co).
Quoted 7 lines
Quoted 4 lines
Are you timing these as fetches from the source control systems or from the mirror tarballs of the repositories. The tarballs should be faster...
the default configuration seems to fetch from source control systems as I always see very long time to fetch gcc/eglibc/linux-yocto (despite having a 2.2 MBytes/s downlink DSL line).
If you're hitting the SCMs I can understand the frustration.
Quoted 3 lines
I don't think that's a size problem but that fetching through svn or git is far less efficient than http or ftp especially from gnu's svn which may be overloaded.
Agreed.
Quoted 3 lines
Morover in a pure OE context we have no interest of all the source history provided by svn or git and that makes a very big volume to download.
The fetcher will deal with this well in the svn case. In the git case, we made a decision to include history since its not that more expensive. Both these assumptions are based on a working up to date mirror.
Quoted 13 lines
. ./openembedded-core/oe-init-build-env
edit local.conf to select qemuarm & BBTHREAD to 8
bitbake core-image-minimal -c fetchalland then I see bitbake stops at around 209 or 214 tasks waiting and
I see that in ps :/home/ebenard/OE-CORE/build/tmp-eglibc/sysroots/x86_64-linux/usr/bin/git.real
clone --bare --mirrorgit://git.yoctoproject.org/linux-yocto-3.2/home/ebenard/OE-CORE/build/downloads/git2/git.yoctoproject.org.linux-yocto-3.2and
svn co -r 184847
http://gcc.gnu.org/svn/gcc/branches/gcc-4_6-branch@184847
Quoted 13 lines
... and both are actually fetching at only around 200 KiB/s which last for quite a long time as (from an other downloads dir) the final tree size are huge :
du -s git.yoctoproject.org.linux-yocto-3.2/
610824 git.yoctoproject.org.linux-yocto-3.2/
du -s gcc.gnu.org/
1602496 gcc.gnu.org/
du -s www.eglibc.org/
625048 www.eglibc.org
If I launch at the same time :
wget ftp://ftp.gnu.org/gnu/gcc/gcc-4.6.3/gcc-4.6.3.tar.bz2 I get a download speed close to 1MB/s and the file to download is only 64MB which would save bandwidth.
Try adding this to your configuration:
PREMIRRORS = "\git://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \
svn://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n"
and see if that helps the performance. It might be we consider making this the default for OE-Core although some people are nervous about doing this...
Cheers,
Richard
Quoted 7 lines · Richard Purdie
Quoted 3 lines · Eric Bénard
the default configuration seems to fetch from source control systems as I always see very long time to fetch gcc/eglibc/linux-yocto (despite having a 2.2 MBytes/s downlink DSL line).
If you're hitting the SCMs I can understand the frustration.
that's not a frustration, that's a feedback on the default behaviour. But I agree with you that could be a frustration for someone trying OE-core for the first time ;-)
Quoted 10 lines
Try adding this to your configuration:
PREMIRRORS = "\
git://.*/.*http://downloads.yoctoproject.org/mirror/sources/ \n \
svn://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n"and see if that helps the performance. It might be we consider making this the default for OE-Core although some people are nervous about doing this...
sure that will help : in my work setup I have my own mirrors configured but here again, that's not what a new user will have and in that case, I'm testing the plain default configuration to help finding bugs or things to improve the release.
I think fetching from git or svn should not be the first thing to do in recipes like gcc, eglibc, linux & co where we are based on a stable released version : this doesn't bring real added value to the user in OE context and this wastes bandwidth (a tbz2 kernel is around 75MB, a git one is around 600MB).
Eric
Quoted 28 lines · Eric Bénard
Quoted 7 lines · Richard Purdie
Quoted 3 lines · Eric Bénard
the default configuration seems to fetch from source control systems as I always see very long time to fetch gcc/eglibc/linux-yocto (despite having a 2.2 MBytes/s downlink DSL line).
If you're hitting the SCMs I can understand the frustration.
that's not a frustration, that's a feedback on the default behaviour. But I agree with you that could be a frustration for someone trying OE-core for the first time ;-)
Quoted 10 lines
Try adding this to your configuration:
PREMIRRORS = "\
git://.*/.*http://downloads.yoctoproject.org/mirror/sources/ \n \
svn://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n"and see if that helps the performance. It might be we consider making this the default for OE-Core although some people are nervous about doing this...
sure that will help : in my work setup I have my own mirrors configured but here again, that's not what a new user will have and in that case, I'm testing the plain default configuration to help finding bugs or things to improve the release.
Quoted 4 lines
I think fetching from git or svn should not be the first thing to do in recipes like gcc, eglibc, linux & co where we are based on a stable released version : this doesn't bring real added value to the user in OE context and this wastes bandwidth (a tbz2 kernel is around
s/user/developer/ and there is value in having git history. I know we'd never do without it in our shop.
I suggested shallow clones and some other options to Richard a few weeks ago, or some other hybrid models. They all vary in terms of nastiness and have some good and bad points.
But from a kernel guy's point of view, you definitely want to work inside git, but
I can see from non-kernel point of view, build and boot is all that really matters.
Cheers,
Bruce
Quoted 8 lines
75MB, a git one is around 600MB).
Eric
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"
Quoted 9 lines · Bruce Ashfield
Quoted 4 lines · Eric Bénard
I think fetching from git or svn should not be the first thing to do in recipes like gcc, eglibc, linux & co where we are based on a stable released version : this doesn't bring real added value to the user in OE context and this wastes bandwidth (a tbz2 kernel is around
s/user/developer/ and there is value in having git history. I know we'd never do without it in our shop.
Quoted 6 lines
I suggested shallow clones and some other options to Richard a few weeks ago, or some other hybrid models. They all vary in terms of nastiness and have some good and bad points.
But from a kernel guy's point of view, you definitely want to work inside git, but
Do you mean you work in the git tree of linux-yocto *directly inside* OE's sources / downloads ?
Eric
Quoted 28 lines · Eric Bénard
Quoted 7 lines · Richard Purdie
Quoted 3 lines · Eric Bénard
the default configuration seems to fetch from source control systems as I always see very long time to fetch gcc/eglibc/linux-yocto (despite having a 2.2 MBytes/s downlink DSL line).
If you're hitting the SCMs I can understand the frustration.
that's not a frustration, that's a feedback on the default behaviour. But I agree with you that could be a frustration for someone trying OE-core for the first time ;-)
Quoted 10 lines
Try adding this to your configuration:
PREMIRRORS = "\
git://.*/.*http://downloads.yoctoproject.org/mirror/sources/ \n \
svn://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n"and see if that helps the performance. It might be we consider making this the default for OE-Core although some people are nervous about doing this...
sure that will help : in my work setup I have my own mirrors configured but here again, that's not what a new user will have and in that case, I'm testing the plain default configuration to help finding bugs or things to improve the release.
Quoted 5 lines
I think fetching from git or svn should not be the first thing to do in recipes like gcc, eglibc, linux & co where we are based on a stable released version : this doesn't bring real added value to the user in OE context and this wastes bandwidth (a tbz2 kernel is around 75MB, a git one is around 600MB).
We've gone around in circles on this. We did use tarballs for gcc, people complained. We switched to svn, you're not happy and probably others aren't. We can't win.
Adding the PREMIRRORS makes the situation better, I agree its not perfect. The original question was how can we speed it up and this is an easy way to do so for the default user case without changing anything major.
I hear what you're saying on the tarball vs. SCM issue but using tarballs does break use cases some users do use, the opposite is not true.
Its also ultimately down to the maintainers of recipes. The gcc issue is more maintainable the way its set up now compared to large numbers of patches (which took an age to apply) and doesn't have much of an additional bandwidth cost.
The linux-yocto kernel recipe heavily uses the SCM to do things so whilst it does have a higher download cost, it as adds value and is ultimately a maintainers choice too.
So whilst I hear what you're saying, I don't think we can change anything other than the PREMIRROR...
Cheers,
Richard
Quoted 4 lines · Richard Purdie
The linux-yocto kernel recipe heavily uses the SCM to do things so whilst it does have a higher download cost, it as adds value and is ultimately a maintainers choice too.
OK now that I've given a closer look at the linux-yocto recipes & bbclass I understand better why you need it in that recipe and that this recipe is heavily based on git's features.
Quoted 3 lines
So whilst I hear what you're saying, I don't think we can change anything other than the PREMIRROR...
then maybe for the new users testing OE, having PREMIRRORs set in the default configuration would be a great thing so that they don't believe OE is a big slow beast just because they have to wait hours for git or svn to fetch sources.
Eric
Quoted 9 lines · Eric Bénard
Quoted 4 lines · Richard Purdie
The linux-yocto kernel recipe heavily uses the SCM to do things so whilst it does have a higher download cost, it as adds value and is ultimately a maintainers choice too.
OK now that I've given a closer look at the linux-yocto recipes & bbclass I understand better why you need it in that recipe and that this recipe is heavily based on git's features.
There are alternatives that I'm going to be exploring going forward, just nothing
that we can bring in during the stabilization cycle. The recipes manipulate git and use it to construct what you build, they don't absolutely require a full git history, so there are some potential savings to be had.
It just obviously limits flexibility if a derived recipe wants to merge branches and histories to construct what is built. So having a simple/shallow history for basic builds while not breaking more complex cases probably hits the sweet spot.
Cheers,
Bruce
Quoted 15 lines
Quoted 4 lines
So whilst I hear what you're saying, I don't think we can change anything other than the PREMIRROR...
then maybe for the new users testing OE, having PREMIRRORs set in the default configuration would be a great thing so that they don't believe OE is a big slow beast just because they have to wait hours for git or svn to fetch sources.
Eric
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"
Quoted 11 lines · Bruce Ashfield
There are alternatives that I'm going to be exploring going forward, just nothing
that we can bring in during the stabilization cycle. The recipes manipulate git and use it to construct what you build, they don't absolutely require a full git history, so there are some potential savings to be had.It just obviously limits flexibility if a derived recipe wants to merge branches and histories to construct what is built. So having a simple/shallow history for basic builds while not breaking more complex cases probably hits the sweet spot.
OK in the end all the slow download problems I met while testing oe-core & qemuarm from scratch were due to a problem on the server hosting yocto's git and mirror services (so setting PREMIRROR to use yocto's mirror didn't improve the situation).
Now that this problem is fixed on the yocto server, the time to download linux-yocto kernel went from 90-120 minutes down to 20-30 minutes which seems more reasonnable !
So there was really a problem but I was not looking in the right direction to fix it :-(
Eric