Skip to contentSkip to conversation

Performance measurement: Building openembedded-core with and without overclocking on C-i7 2600K

DISCUSSION Ulf Samuelsson 6 participants 13 messages Oct 17, 2011 – Oct 29, 2011

lore
Share & tools
lore
#0 lore raw

If anyone is interested...

Same machine in two different configurations.

.........................................................#1...........................#2 P8Z68V Pro
Core-i7 2600K @ 3.4 GHz..................4.4 GHz
8 GB @ ...........................................1333 MHz...............1640 MHz 1TB WD Black 7200 RPM
Ubuntu 11-10

Almost all source code already in source directory

Building meta-toolchain is 13% faster with overclocking.

#1
build: meta-toolchain
START 10:46:23 55:52
FINISH 11:42:15
build: core-image-minimal
START 11:42:15 17:34
FINISH 11:59:49
build: core-image-sato
START 13:40:47 01:19
FINISH 13:42:06
Build fails, since one package could not be downloaded.
build: meta-toolchain-sdk
START 13:42:06 00:38
FINISH 13:42:44
build: adt-installer
START 13:42:44 00:03
FINISH 13:42:51
build: meta-ide-support
START 13:42:51 01:13
FINISH 13:44:04

#2
build: meta-toolchain
START 20:18:51 49:19 (vs 55:52)
FINISH 21:08:10
build: core-image-minimal
START 21:08:10 16:38 (vs 17:34)
FINISH 21:24:48
build: core-image-sato
START 21:24:48 15:38 (vs 01:19 - I think I have a problem in the first build)
FINISH 21:40:26
build: meta-toolchain-sdk
START 21:40:26 00:59 (vs 00:38)
FINISH 21:41:25
build: adt-installer
START 21:41:25 00:03 (vs 00:03)
FINISH 21:41:28
build: meta-ide-support
START 21:41:28 01:03 (vs 01:13)
FINISH 21:42:31

Signature · 2 lines
Best Regards
Ulf Samuelsson
#1 lore raw
Quoted 2 lines · Ulf Samuelsson

If anyone is interested...

thanks for numbers. Is there statistics on 64bit host Vs. 32bit build host available too ?

Quoted 62 lines

Same machine in two different configurations.

.........................................................#1...........................#2

P8Z68V Pro
Core-i7 2600K @ 3.4 GHz..................4.4 GHz 8 GB @ ...........................................1333 MHz...............1640 MHz
1TB WD Black 7200 RPM
Ubuntu 11-10

Almost all source code already in source directory

Building meta-toolchain is 13% faster with overclocking.

#1
build: meta-toolchain
START 10:46:23 55:52
FINISH 11:42:15
build: core-image-minimal
START 11:42:15 17:34
FINISH 11:59:49
build: core-image-sato
START 13:40:47 01:19
FINISH 13:42:06
Build fails, since one package could not be downloaded.
build: meta-toolchain-sdk
START 13:42:06 00:38
FINISH 13:42:44
build: adt-installer
START 13:42:44 00:03
FINISH 13:42:51
build: meta-ide-support
START 13:42:51 01:13
FINISH 13:44:04

#2
build: meta-toolchain
START 20:18:51 49:19 (vs 55:52)
FINISH 21:08:10
build: core-image-minimal
START 21:08:10 16:38 (vs 17:34)
FINISH 21:24:48
build: core-image-sato
START 21:24:48 15:38 (vs 01:19 - I think I have a problem in the first build)
FINISH 21:40:26
build: meta-toolchain-sdk
START 21:40:26 00:59 (vs 00:38)
FINISH 21:41:25
build: adt-installer
START 21:41:25 00:03 (vs 00:03)
FINISH 21:41:28
build: meta-ide-support
START 21:41:28 01:03 (vs 01:13)
FINISH 21:42:31

#2 lore raw

2011-10-18 00:13, Khem Raj skrev:

Quoted 5 lines · Ulf Samuelsson
Quoted 2 lines

If anyone is interested...

thanks for numbers. Is there statistics on 64bit host Vs. 32bit build host available too ?

We discussed that a few days ago.

My comparision between 2 similar machines indicate that 32 bit 11.04 with Psyco-Dist is 20% faster than 64 bit with 11.04

The machine running 64 bit 11.04 was tested with 32 bit 11.04 using an external disk (connected through e-SATA) and results were very close to the numbers for the other 32 bit machine

I didn't save those numbers but I think that is good enough
to draw the conclusion that adding JIT for 64 bit (PyPy?) will reduce build time significantly.

It would be interesting to know the effect of faster disks. I run SATA-II disks at the moment, and upgrading to SATA-III might improve things.

Since you have multiple tasks, it might be worthwhile having several disks to the disk heads spending most of their time moving around.
Have a lot of computer parts including a chassi mounting 6+ disks.
* /
* /home
* <OE_DIR>/source
* <OE_DIR>/build/tmp

Also what happens if it it built on a fast SSD, or even a RAMdisk?

Building the stuff below adds 25 GB + a 2 GB source directory. With the Z68 chipset, you can get up to 32 GB.
It would be interesting to know how well the linux caching system performs vs having <OE_DIR>/build/tmp etc. on a ramdisk.

The 8 GB DIMMs are not really available without ECC, but Corsair is about to release a 32 GB kit which will set you back $1000 so I wont do that test for the forseeable future. Maybe someone else which will have such a machine anyway would be interested enough to do a test.

One issue with this is that even 32 GB may not be enough. With the current build using 27 GB, there is not much headroom to grow.

If using a ramdisk, proves to be significantly faster,
an alternative would be smaller ramdisk with some background task moving packages to a real disk after the build of that package
has completed.

Quoted 69 lines
Quoted 52 lines

Same machine in two different configurations.

.........................................................#1...........................#2

P8Z68V Pro
Core-i7 2600K @ 3.4 GHz..................4.4 GHz 8 GB @ ...........................................1333 MHz...............1640 MHz
1TB WD Black 7200 RPM
Ubuntu 11-10

Almost all source code already in source directory

Building meta-toolchain is 13% faster with overclocking.

#1
build: meta-toolchain
START 10:46:23 55:52
FINISH 11:42:15
build: core-image-minimal
START 11:42:15 17:34
FINISH 11:59:49
build: core-image-sato
START 13:40:47 01:19
FINISH 13:42:06
Build fails, since one package could not be downloaded.
build: meta-toolchain-sdk
START 13:42:06 00:38
FINISH 13:42:44
build: adt-installer
START 13:42:44 00:03
FINISH 13:42:51
build: meta-ide-support
START 13:42:51 01:13
FINISH 13:44:04

#2
build: meta-toolchain
START 20:18:51 49:19 (vs 55:52)
FINISH 21:08:10
build: core-image-minimal
START 21:08:10 16:38 (vs 17:34)
FINISH 21:24:48
build: core-image-sato
START 21:24:48 15:38 (vs 01:19 - I think I have a problem in the first build)
FINISH 21:40:26
build: meta-toolchain-sdk

Quoted 11 lines

START 21:40:26 00:59 (vs 00:38)
FINISH 21:41:25
build: adt-installer
START 21:41:25 00:03 (vs 00:03)
FINISH 21:41:28
build: meta-ide-support
START 21:41:28 01:03 (vs 01:13)
FINISH 21:42:31

_______________________________________________
Openembedded-devel mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel

Signature · 2 lines
Best Regards
Ulf Samuelsson
#9 lore raw
In reply to Ulf Samuelsson
Quoted 1 line · Ulf Samuelsson

If anyone is interested...

Build stats are always interesting... ;)

Quoted 11 lines

Same machine in two different configurations.

.........................................................#1...........................#2 P8Z68V Pro
Core-i7 2600K @ 3.4 GHz..................4.4 GHz
8 GB @ ...........................................1333 MHz...............1640 MHz
1TB WD Black 7200 RPM
Ubuntu 11-10

Almost all source code already in source directory

This would severely restrict the usefulness of the comparison. I'd guess that this means that in one case the source will be downloaded, while it'll already be downloaded in the second test. Could you redo the statistics with all sources downloaded?

Quoted 1 line

Building meta-toolchain is 13% faster with overclocking.

Was the overclocking test run after the non-overclocking?

Cheers,
Anders

Signature · 2 lines
Anders Darander
ChargeStorm AB / eStorm AB
#10 lore raw

2011-10-18 08:28, Anders Darander skrev:

Quoted 13 lines · Ulf Samuelsson
Quoted 1 line

If anyone is interested...

Build stats are always interesting... ;)

Quoted 9 lines

Same machine in two different configurations. .........................................................#1...........................#2 P8Z68V Pro
Core-i7 2600K @ 3.4 GHz..................4.4 GHz
8 GB @ ...........................................1333 MHz...............1640 MHz
1TB WD Black 7200 RPM
Ubuntu 11-10
Almost all source code already in source directory

Quoted 4 lines

This would severely restrict the usefulness of the comparison. I'd guess that this means that in one case the source will be downloaded, while it'll already be downloaded in the second test. Could you redo the statistics with all sources downloaded?

I had already built it once before running the first test then updated and I think one additional package was downloaded over a 100 MBps line so it affects a little, but not too much.

I think the conclusion:
Increasing the CPU speed by 30% and the memory frequency by 20% gives 10-15% faster build time is OK.

Quoted 6 lines
Quoted 1 line

Building meta-toolchain is 13% faster with overclocking.

Was the overclocking test run after the non-overclocking?

Cheers,
Anders

Signature · 2 lines
Best Regards
Ulf Samuelsson
#11 lore raw
Quoted 14 lines · Ulf Samuelsson

2011-10-18 08:28, Anders Darander skrev:

Quoted 13 lines · Ulf Samuelsson
Quoted 1 line

If anyone is interested...

Build stats are always interesting... ;)

Quoted 9 lines

Same machine in two different configurations. .........................................................#1...........................#2 P8Z68V Pro
Core-i7 2600K @ 3.4 GHz..................4.4 GHz
8 GB @ ...........................................1333 MHz...............1640 MHz
1TB WD Black 7200 RPM
Ubuntu 11-10
Almost all source code already in source directory

Quoted 7 lines
Quoted 4 lines

This would severely restrict the usefulness of the comparison. I'd guess that this means that in one case the source will be downloaded, while it'll already be downloaded in the second test. Could you redo the statistics with all sources downloaded?

I had already built it once before running the first test then updated and I think one additional package was downloaded over a 100 MBps line so it affects a little, but not too much.

Ok, just curious. Not least as you had one image fail in the first configuration, which you attributed to a failed download. That remark made me wonder...

Cheers,
Anders

Quoted 6 lines

I think the conclusion:
Increasing the CPU speed by 30% and the memory frequency by 20% gives 10-15% faster build time is OK.

Quoted 2 lines
Quoted 1 line

Building meta-toolchain is 13% faster with overclocking.

Was the overclocking test run after the non-overclocking?

Signature · 2 lines
Anders Darander
ChargeStorm AB / eStorm AB
#12 lore raw

2011-10-18 14:19, Anders Darander skrev:

Quoted 18 lines · Ulf Samuelsson
Quoted 17 lines

2011-10-18 08:28, Anders Darander skrev:

Quoted 16 lines · Ulf Samuelsson
Quoted 1 line

If anyone is interested...

Build stats are always interesting... ;)

Quoted 9 lines

Same machine in two different configurations. .........................................................#1...........................#2 P8Z68V Pro
Core-i7 2600K @ 3.4 GHz..................4.4 GHz
8 GB @ ...........................................1333 MHz...............1640 MHz
1TB WD Black 7200 RPM
Ubuntu 11-10
Almost all source code already in source directory

This would severely restrict the usefulness of the comparison. I'd guess that this means that in one case the source will be downloaded, while it'll already be downloaded in the second test. Could you redo the statistics with all sources downloaded?

Quoted 6 lines
Quoted 3 lines

I had already built it once before running the first test then updated and I think one additional package was downloaded over a 100 MBps line so it affects a little, but not too much.

Ok, just curious. Not least as you had one image fail in the first configuration, which you attributed to a failed download. That remark made me wonder...

OK, To be 100% correct:
I built meta-toolchain first so all packages for that build should be there. This is much longer than anything else, so this is a good comparision.

Then the build #1 was started and measruements was taken. This failed for core-image-sato due to one missing package, which After build #1 was completed, I downloaded the missing package and started build #2.

Quoted 10 lines

Cheers,
Anders

Quoted 5 lines

I think the conclusion:
Increasing the CPU speed by 30% and the memory frequency by 20% gives 10-15% faster build time is OK.

Quoted 2 lines
Quoted 1 line

Building meta-toolchain is 13% faster with overclocking.

Was the overclocking test run after the non-overclocking?

Signature · 2 lines
Best Regards
Ulf Samuelsson
#3 lore raw
In reply to Ulf Samuelsson
Quoted 3 lines · Ulf Samuelsson

My comparision between 2 similar machines indicate that 32 bit 11.04 with Psyco-Dist is 20% faster than 64 bit with 11.04

thats why I wanted to know if they were exactly same machine. Better to reinstall may be. I dont rule out that 32bit cant be faster but we need to measure it

Quoted 9 lines

The machine running 64 bit 11.04 was tested with 32 bit 11.04 using an external disk (connected through e-SATA) and results were very close to the numbers for the other 32 bit machine

I didn't save those numbers but I think that is good enough
to draw the conclusion that adding JIT for 64 bit (PyPy?) will reduce build time significantly.

Thats what I am interested in. If pypy would make it faster than we could recommend it.

#4 lore raw
In reply to Ulf Samuelsson
Quoted 10 lines · Ulf Samuelsson

My comparision between 2 similar machines indicate that 32 bit 11.04 with Psyco-Dist is 20% faster than 64 bit with 11.04

The machine running 64 bit 11.04 was tested with 32 bit 11.04 using an external disk (connected through e-SATA) and results were very close to the numbers for the other 32 bit machine

I didn't save those numbers but I think that is good enough
to draw the conclusion that adding JIT for 64 bit (PyPy?) will reduce build time significantly.

You seem to have changed two things at once there (or perhaps just made a bit too much of a leap in your conclusion). Did you test 32-bit without the JIT?

p.

#5 lore raw

2011-10-18 16:47, Phil Blundell skrev:

Quoted 8 lines · Ulf Samuelsson
Quoted 7 lines

My comparision between 2 similar machines indicate that 32 bit 11.04 with Psyco-Dist is 20% faster than 64 bit with 11.04

The machine running 64 bit 11.04 was tested with 32 bit 11.04 using an external disk (connected through e-SATA) and results were very close to the numbers for the other 32 bit machine

Quoted 6 lines
Quoted 3 lines

I didn't save those numbers but I think that is good enough
to draw the conclusion that adding JIT for 64 bit (PyPy?) will reduce build time significantly.

You seem to have changed two things at once there (or perhaps just made a bit too much of a leap in your conclusion). Did you test 32-bit without the JIT?

The word indicate is not soo strong as you indicate ;-) Just saying that if it looks like a duck, walks like a duck it is probably a duck.

It takes too long to reinstall for that to be an alternative for me. More data from other people could be an alternative.

Quoted 9 lines

p.

_______________________________________________
Openembedded-devel mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel

Signature · 2 lines
Best Regards
Ulf Samuelsson
#6 lore raw
In reply to Ulf Samuelsson

В сообщении от 18 октября 2011 10:03:25 автор Ulf Samuelsson написал:

Quoted 1 line

Also what happens if it it built on a fast SSD, or even a RAMdisk?

Our main build server has two Xeons X5670 with 96 GB of RAM. It's running 24×7 cyclicly doing clean isolated builds of all our OE projects (currently five). "Isolated" means that the the actual build is done inside a container (LXC) with no network available at all, that is a requirement for all our builds, to be able to take one tarball of sources, one tarball of OE tree and build the project somewhere in a nuclear bunker.

The system is configured in such way that container is located on tmpfs (with size of 50 GB). The most complex build takes about hour and a half on this system occupying about 45 GB of space in a ramdisk.

So today I tried to get some RAM vs. Linux cache statistics and switched this mount point over to newly created 60 GB LVM partition with ext4 on RAID0 array consisting of two SAS 15K drives.

The system made builds for three projects is this configuration and I see no difference at all, usual 1-2 minutes deviation. Granted, the system has quite powerful disks (RAID array gives about 380 MB/sec on hdparm) and things might be a little different on plain SATA drives, but frankly I'd expected to see the difference anyway since there are lots of small files involved in a build.

Maybe I should try to further degrade the disk system by creating some encrypted volume inside LVM, but still from what I see Linux caching and buffering works good enough, just give it as much RAM as you can.

But then also what you'll get from RAM or disk or even CPU upgrade depends on what type of build you have. Upgrading developers build servers from pair of 4-core Opterons (don't remember exact model) with 8 GB of RAM to pair of Xeons E5620 with 24 GB of RAM with comparable disks gave about 20-30% of build time reduction for one project and 50% for another. But that builds are not isolated and use icecc cluster with all build servers available to the cluster, maybe that helps also in our situation.

Signature · 3 lines
 http://roman.khimov.ru
mailto: [email protected]
gpg --keyserver hkp://subkeys.pgp.net --recv-keys 0xE5E055C3
#7 lore raw

2011-10-23 19:50, Roman Khimov skrev:

Quoted 27 lines

? ????????? ?? 18 ??????? 2011 10:03:25 ????? Ulf Samuelsson ???????:

Quoted 1 line

Also what happens if it it built on a fast SSD, or even a RAMdisk?

Our main build server has two Xeons X5670 with 96 GB of RAM. It's running 24×7 cyclicly doing clean isolated builds of all our OE projects (currently five). "Isolated" means that the the actual build is done inside a container (LXC) with no network available at all, that is a requirement for all our builds, to be able to take one tarball of sources, one tarball of OE tree and build the project somewhere in a nuclear bunker.

The system is configured in such way that container is located on tmpfs (with size of 50 GB). The most complex build takes about hour and a half on this system occupying about 45 GB of space in a ramdisk.

So today I tried to get some RAM vs. Linux cache statistics and switched this mount point over to newly created 60 GB LVM partition with ext4 on RAID0 array consisting of two SAS 15K drives.

The system made builds for three projects is this configuration and I see no difference at all, usual 1-2 minutes deviation. Granted, the system has quite powerful disks (RAID array gives about 380 MB/sec on hdparm) and things might be a little different on plain SATA drives, but frankly I'd expected to see the difference anyway since there are lots of small files involved in a build.

Maybe I should try to further degrade the disk system by creating some encrypted volume inside LVM, but still from what I see Linux caching and buffering works good enough, just give it as much RAM as you can.

Quoted 8 lines

But then also what you'll get from RAM or disk or even CPU upgrade depends on what type of build you have. Upgrading developers build servers from pair of 4-core Opterons (don't remember exact model) with 8 GB of RAM to pair of Xeons E5620 with 24 GB of RAM with comparable disks gave about 20-30% of build time reduction for one project and 50% for another. But that builds are not isolated and use icecc cluster with all build servers available to the cluster, maybe that helps also in our situation.

Thanks,

I tried installing 32 bit Ubuntu 11.10 on a HDD and got results, which are very close to the results of 64 bit ubuntu 11.10, but when I tried hdparm the 32 bit system had 50 MB/s and the 64 bit system
had 100-110 MB/s so they are not comparable.

Also tried putting tmp/sysroots on a 3 GB tmpfs, and that only made the build a couple of minutes faster on a 1 hour build.

Quoted 6 lines

_______________________________________________
Openembedded-devel mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel

Signature · 2 lines
Best Regards
Ulf Samuelsson
#8 lore raw
In reply to Roman Khimov

Hi,

Quoted 6 lines

Our main build server has two Xeons X5670 with 96 GB of RAM. It's running 24×7 cyclicly doing clean isolated builds of all our OE projects (currently five). "Isolated" means that the the actual build is done inside a container (LXC) with no network available at all, that is a requirement for all our builds, to be able to take one tarball of sources, one tarball of OE tree and build the project somewhere in a nuclear bunker.

So, LXC sounds a better way than qemu for performances, isn't it? Portability issues (next years operating systems) moving around the OE setup?

Quoted 3 lines

The system is configured in such way that container is located on tmpfs (with size of 50 GB). The most complex build takes about hour and a half on this system occupying about 45 GB of space in a ramdisk.

I'm now building on xeon 5530 8 core (HP Z800) with 24GB DDR3. 20GB for tmpfs where I put only the tmp directory. Recipes dir is on ssd 480MB/S writing speed.

Quoted 9 lines

So today I tried to get some RAM vs. Linux cache statistics and switched this mount point over to newly created 60 GB LVM partition with ext4 on RAID0 array consisting of two SAS 15K drives.

The system made builds for three projects is this configuration and I see no difference at all, usual 1-2 minutes deviation. Granted, the system has quite powerful disks (RAID array gives about 380 MB/sec on hdparm) and things might be a little different on plain SATA drives, but frankly I'd expected to see the difference anyway since there are lots of small files involved in a build.

You say you have no big differences between all tmpfs and SAS one? I have instead SAS 15k RAID1, but I don't remember now their speed. Moving from a complete SAS 15K setup to tmp dir in RAM I've instead reduced from 4h to 1.5h for my image.

Quoted 11 lines

Maybe I should try to further degrade the disk system by creating some encrypted volume inside LVM, but still from what I see Linux caching and buffering works good enough, just give it as much RAM as you can.

But then also what you'll get from RAM or disk or even CPU upgrade depends on what type of build you have. Upgrading developers build servers from pair of 4-core Opterons (don't remember exact model) with 8 GB of RAM to pair of Xeons E5620 with 24 GB of RAM with comparable disks gave about 20-30% of build time reduction for one project and 50% for another. But that builds are not isolated and use icecc cluster with all build servers available to the cluster, maybe that helps also in our situation.

In my opinion the best setup is the one that I can move from a box to another one without problem,
to be safe in case a box stops working.
If it stops I go to another one, I rsync download from the backup server and restart.

Redundancy of cpu power should a different approach, but you "nuclear bunker" contraints doesn't include this possibility.
I was in the past thinking to get a 15k€ building system with 2 computational nodes and a fast storage, with a standard debian on it. A "small" blade center.
A developer ssh into it, he can do meld through nfs mount, start it's own tagged qemu and compile his image.
If one core crash, the other one works anyway.

But now we are instead moving from 8 cores to 16 cores (IBM I don't remember the model, something about 7000k€ off full price w/o discount) with 64GB ram in order to be able to compile at least the compilation at the same time.

Anyway it is difficult to convince developers to use central machine, instead of using their own awful pc.
It's an hard job, but it's needed to speed up the stupid compilation job.

To complete the redundancy idea I'll have soon the 16cores with arch linux and qemu through virtual manager.
If I have 16cores pb I'll get the used qemu image to the 8cores and I go on.

Bye,
Raffaele

Signature · 1 line
www.opensurf.it