Performance Issue: Build time increases

RFC Robert Yang 4 participants 8 messages Aug 12, 2011 – Aug 16, 2011

lore

Share & tools

lore

Robert Yang

author

2011-08-12 11:10 UTC #0 lore raw

Hi folks,

The build time of core-image-sato increases about 5 ~ 10 minutes than the following commit:

commit 5af197b55a4b779f1ec93186f0723026949ba2b5

Author: Liping Ke <[email protected]>
Date:   Fri Jun 3 08:22:40 2011 +0800

cache: Implement multiple extra cache fields request support

On my host,
1) the build time of 5af197b55a4b779f1ec93186f0723026949ba2b5 is:

real    208m26.133s
user    241m29.280s
sys     47m0.630s

2) and: 068839698fe192d8846c0ed4db65861448e8e524 is:

real    217m39.687s
user    255m34.150s
sys     48m21.510s

I use the bisect build method to find out which patch causes the time
increases, but the build time is not stable on my host(Ubuntu 11.04 64bit),
e.g., the build time of 1) is 208m at the first build, then "git co other_commit" and build it, after about 10 builds, then go back to build 5af197b55a4b779f1ec, the build time will increases about 8 minutes, I have
stopped the X and cron, at. This may have relationship with the linux
distribution and disk.

So I have to restart the build, reboot the machine by two days, and go on the build. It would be better if anyone has other good method.

I think that for the next release(e.g., yocto 1.2), we can find a clean and stable machine(for the distribution, maybe RHEL is more stable than Ubuntu) to check the build time weekly, so that we can notice the performance issue early.

Signature · 3 lines

Thanks

Robert

Lu, Lianhao

2011-08-12 12:23 UTC #1 lore raw

Robert Yang wrote on 2011-08-12:

Quoted 11 lines

Hi folks,
The build time of core-image-sato increases about 5 ~ 10 minutes than the following commit:
commit 5af197b55a4b779f1ec93186f0723026949ba2b5
Author: Liping Ke <[email protected]>
Date:   Fri Jun 3 08:22:40 2011 +0800
cache: Implement multiple extra cache fields request support

I think this patch only stores extra information in an extra cache. It only affects the parsing time, and I don't think there will be 5 min difference. How about comparing the "bitbake -e" time with and without the patch(by deleting all the cache files before running bitbake).

Best Regards,
-Lianhao

Quoted 28 lines

On my host,
1) the build time of 5af197b55a4b779f1ec93186f0723026949ba2b5 is:
real    208m26.133s
user    241m29.280s
sys     47m0.630s
2) and: 068839698fe192d8846c0ed4db65861448e8e524 is:
real    217m39.687s
user    255m34.150s
sys     48m21.510s
I use the bisect build method to find out which patch causes the time
increases, but the build time is not stable on my host(Ubuntu 11.04
64bit), e.g., the build time of 1) is 208m at the first build, then
"git co other_commit" and build it, after about 10 builds, then go
back to build 5af197b55a4b779f1ec, the build time will increases about
8 minutes, I have stopped the X and cron, at. This may have relationship with the linux distribution and disk.
So I have to restart the build, reboot the machine by two days, and go on the build. It would be better if anyone has other good method.
I think that for the next release(e.g., yocto 1.2), we can find a
clean and stable machine(for the distribution, maybe RHEL is more
stable than Ubuntu) to check the build time weekly, so that we can notice the performance issue early.

Richard Purdie

2011-08-15 14:36 UTC #3 lore raw

In reply to Robert Yang

Quoted 33 lines · Robert Yang

Hi folks,
The build time of core-image-sato increases about 5 ~ 10 minutes than the following commit:
commit 5af197b55a4b779f1ec93186f0723026949ba2b5
Author: Liping Ke <[email protected]>
Date:   Fri Jun 3 08:22:40 2011 +0800
cache: Implement multiple extra cache fields request support
On my host,
1) the build time of 5af197b55a4b779f1ec93186f0723026949ba2b5 is:
real    208m26.133s
user    241m29.280s
sys     47m0.630s
2) and: 068839698fe192d8846c0ed4db65861448e8e524 is:
real    217m39.687s
user    255m34.150s
sys     48m21.510s
I use the bisect build method to find out which patch causes the time increases, but the build time is not stable on my host(Ubuntu 11.04 64bit), e.g., the build time of 1) is 208m at the first build, then "git co other_commit" and build it, after about 10 builds, then go back to build 5af197b55a4b779f1ec, the build time will increases about 8 minutes, I have stopped the X and cron, at. This may have relationship with the linux distribution and disk.

Quoted 7 lines

So I have to restart the build, reboot the machine by two days, and go on the build. It would be better if anyone has other good method.
I think that for the next release(e.g., yocto 1.2), we can find a clean and stable machine(for the distribution, maybe RHEL is more stable than Ubuntu) to check the build time weekly, so that we can notice the performance issue early.

I think we really need to get to the bottom of why the build times are so variable. Do we need to start doing these on a clean distro install? is a fresh boot good enough? is there a way we can clear out the VM caches and get reproducible times? reformat the build partition?

If that really is just because of the nature of a chaotic system, can we get some better representation of build time to use as a benchmark? We could really do with finding a faster test (which would hopefully still be representative)...

Cheers,

Richard

Robert Yang

author

2011-08-16 08:42 UTC #2 lore raw

In reply to Lu, Lianhao

Quoted 7 lines · Lianhao

Robert Yang wrote on 2011-08-12:
Quoted 6 lines
Hi folks,
The build time of core-image-sato increases about 5 ~ 10 minutes than the following commit:

Quoted 8 lines

Quoted 5 lines
commit 5af197b55a4b779f1ec93186f0723026949ba2b5
Author: Liping Ke<[email protected]>
Date:   Fri Jun 3 08:22:40 2011 +0800
cache: Implement multiple extra cache fields request support
I think this patch only stores extra information in an extra cache. It only affects the parsing time, and I don't think there will be 5 min difference. How about comparing the "bitbake -e" time with and without the patch(by deleting all the cache files before running bitbake).

I'm sorry that I didn't describe it clearly, what I meant was that: This patch worked correctly, it didn't increase the build time, but other patches after this one made the build time increased, we just use this commit as a reference point.

// Robert

Quoted 39 lines

Best Regards,
-Lianhao
Quoted 21 lines
On my host,
1) the build time of 5af197b55a4b779f1ec93186f0723026949ba2b5 is:
real    208m26.133s
user    241m29.280s
sys     47m0.630s
2) and: 068839698fe192d8846c0ed4db65861448e8e524 is:
real    217m39.687s
user    255m34.150s
sys     48m21.510s
I use the bisect build method to find out which patch causes the time
increases, but the build time is not stable on my host(Ubuntu 11.04
64bit), e.g., the build time of 1) is 208m at the first build, then
"git co other_commit" and build it, after about 10 builds, then go
back to build 5af197b55a4b779f1ec, the build time will increases about
8 minutes, I have stopped the X and cron, at. This may have relationship with the linux distribution and disk.
Quoted 7 lines
So I have to restart the build, reboot the machine by two days, and go on the build. It would be better if anyone has other good method.
I think that for the next release(e.g., yocto 1.2), we can find a
clean and stable machine(for the distribution, maybe RHEL is more
stable than Ubuntu) to check the build time weekly, so that we can notice the performance issue early.
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Robert Yang

author

2011-08-16 08:44 UTC #4 lore raw

In reply to Richard Purdie

Quoted 37 lines · Richard Purdie

Quoted 36 lines · Robert Yang
Hi folks,
The build time of core-image-sato increases about 5 ~ 10 minutes than the following commit:
commit 5af197b55a4b779f1ec93186f0723026949ba2b5
Author: Liping Ke<[email protected]>
Date:   Fri Jun 3 08:22:40 2011 +0800
cache: Implement multiple extra cache fields request support
On my host,
1) the build time of 5af197b55a4b779f1ec93186f0723026949ba2b5 is:
real    208m26.133s
user    241m29.280s
sys     47m0.630s
2) and: 068839698fe192d8846c0ed4db65861448e8e524 is:
real    217m39.687s
user    255m34.150s
sys     48m21.510s
I use the bisect build method to find out which patch causes the time increases, but the build time is not stable on my host(Ubuntu 11.04 64bit), e.g., the build time of 1) is 208m at the first build, then "git co other_commit" and build it, after about 10 builds, then go back to build 5af197b55a4b779f1ec, the build time will increases about 8 minutes, I have stopped the X and cron, at. This may have relationship with the linux distribution and disk.
So I have to restart the build, reboot the machine by two days, and go on the build. It would be better if anyone has other good method.

Quoted 8 lines

Quoted 4 lines
I think that for the next release(e.g., yocto 1.2), we can find a clean and stable machine(for the distribution, maybe RHEL is more stable than Ubuntu) to check the build time weekly, so that we can notice the performance issue early.
I think we really need to get to the bottom of why the build times are so variable. Do we need to start doing these on a clean distro install? is a fresh boot good enough? is there a way we can clear out the VM

Hi Richard,

Thanks for you reply, yes, one build one restart(bitbake core-image-sato, then restart the computer, and bitbake again) makes the build time stable. I'm doing this now.

// Robert

Quoted 22 lines

caches and get reproducible times? reformat the build partition?
If that really is just because of the nature of a chaotic system, can we get some better representation of build time to use as a benchmark? We could really do with finding a faster test (which would hopefully still be representative)...
Cheers,
Richard
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Paul Eggleton

2011-08-16 09:00 UTC #5 lore raw

Quoted 3 lines · On Tuesday 16 August 2011 09:40:14 Robert Yang wrote:

Thanks for you reply, yes, one build one restart(bitbake core-image-sato, then restart the computer, and bitbake again) makes the build time stable. I'm doing this now.

Are you using my scripts (scripts/contrib/test_build_time.sh) ? Much of the initial part of the script is ensuring that caches get cleared out; thus rebooting should not be necessary in order to get stable results.

Cheers,
Paul

Signature · 2 lines

Paul Eggleton
Intel Open Source Technology Centre

Robert Yang

author

2011-08-16 10:42 UTC #6 lore raw

Quoted 8 lines · Paul Eggleton

Quoted 3 lines · On Tuesday 16 August 2011 09:40:14 Robert Yang wrote:
Thanks for you reply, yes, one build one restart(bitbake core-image-sato, then restart the computer, and bitbake again) makes the build time stable. I'm doing this now.
Are you using my scripts (scripts/contrib/test_build_time.sh) ? Much of the initial part of the script is ensuring that caches get cleared out; thus rebooting should not be necessary in order to get stable results.

I'm sorry to say that I have tried it, but I don't know its principle, and it made my git repo have too many untracked or modified files. So I built it manually again:-)

// Robert

Quoted 4 lines

Cheers,
Paul

Paul Eggleton

2011-08-16 11:04 UTC #7 lore raw

Quoted 2 lines · On Tuesday 16 August 2011 11:37:24 Robert Yang wrote:

I'm sorry to say that I have tried it, but I don't know its principle, and it made my git repo have too many untracked or modified files.

Is this when you specified a list of revisions to be applied to the repository prior to building? If so, I'm wondering if the script needs to do these in a different way - perhaps using a branch?

I'd like to know how you ran the script when you tried it. If you have any questions I'm happy to answer them, and I'd also be happy to spend some more time improving the scripts. Personally I think it would be useful to do this rather than using manual methods that are difficult to reproduce. The major benefit of the scripts is that you can start a bisect operation and "walk away", rather than having to monitor it constantly.

Cheers,
Paul

Signature · 2 lines

Paul Eggleton
Intel Open Source Technology Centre