Skip to contentSkip to conversation

Recommended Hardware for building

DISCUSSION Oliver Novakovic 6 participants 6 messages Oct 2, 2014 – Oct 8, 2014

lore
Share & tools
lore
#0 lore raw

Dear all,

since we are using Yocto in our projects, I'd like to set up a dedicated Build PC or server for the Yocto build tasks (including Jenkins).

Can anyone recommend a reasonable performant hardware setup to use ?

What should be considered ? Are there any pitfalls ? What about bottlenecks in the build system ?

Specifically:

How many cores are recommended ? And how much cache is necessary ? How much of the main memory does Yocto really use ? Is 32 GB sufficient or should I go for 64 ?

Does it make sense to use two SSDs as Raid0 to get builds faster ?

Any hints, recommendations or links to further reading will be highly appreciated.

Thanks in advance,

Oliver

Alpine Electronics R&D Europe GmbH, Vor dem Lauch 14, 70567 Stuttgart, Registergericht Stuttgart, HRB 25265
Geschaeftsfuehrer: Toshinori Kobayashi, Shuji Taguchi Working office: Vor dem Lauch 14, 70567 Stuttgart

ATTENTION: This e-mail and any files transmitted with it may be privileged and/or confidential, and are intended solely for the use of the individual or entity to whom this e-mail is addressed. If you are not one of the named recipients or otherwise have reason to believe that you have received this e-mail in error, please notify the sender and delete this message immediately from your computer. We thank you for your cooperation.

#1 lore raw
Quoted 5 lines · Oliver Novakovic

Can anyone recommend a reasonable performant hardware setup to use ?

What should be considered ? Are there any pitfalls ? What about bottlenecks in the build system ?

Quoted 7 lines

Specifically:

How many cores are recommended ? And how much cache is necessary ? How much of the main memory does Yocto really use ? Is 32 GB sufficient or should I go for 64 ?

Does it make sense to use two SSDs as Raid0 to get builds faster ?

As much of everything as you can afford. :) The build isn't heavy in any particular metric, so don't sacrifice RAM for SSDs for example.

RAID 0 over SSD would be nice and fast, but I prefer having a good amount of RAM and a tuned ext4 (no journal, long commit delay) so data doesn't actually hit the disk as frequently. Keeping the actual build directories on a separate disk is good for performance and not causing data loss when you lose a disk.

There are people that have 64GB in machines and then set TMPDIR to a tmpfs. Surprisingly this isn't that much faster (5% or so), but it's a lot easier on the hardware and power consumption.

Ross

#2 lore raw
Quoted 23 lines · Ross
Quoted 5 lines · Oliver Novakovic

Can anyone recommend a reasonable performant hardware setup to use ?

What should be considered ? Are there any pitfalls ? What about bottlenecks in the build system ?

Quoted 7 lines

Specifically:

How many cores are recommended ? And how much cache is necessary ? How much of the main memory does Yocto really use ? Is 32 GB sufficient or should I go for 64 ?

Does it make sense to use two SSDs as Raid0 to get builds faster ?

As much of everything as you can afford. :) The build isn't heavy in any particular metric, so don't sacrifice RAM for SSDs for example.

RAID 0 over SSD would be nice and fast, but I prefer having a good amount of RAM and a tuned ext4 (no journal, long commit delay) so data doesn't actually hit the disk as frequently. Keeping the actual build directories on a separate disk is good for performance and not causing data loss when you lose a disk.

Quoted 3 lines

There are people that have 64GB in machines and then set TMPDIR to a tmpfs. Surprisingly this isn't that much faster (5% or so), but it's a lot easier on the hardware and power consumption.

My experience:

I've got a quad core with hyper-threading (so 8 usable cores) running at about 3.8 GHz, 16GB of RAM and use multiple SSDs - one to hold the meta data, downloads and top level build areas (local.conf, etc) and have the TMPDIR on a second SSD (so, as Ross says, I don't get a surprise when it wears out!).

I can build my images (basically an x11 image) in just under 60 minutes (once all the files have been fetched). I run with BB_NUMBER_THREADS and PARALLEL_MAKE both set to 16 to make sure the cores are fully loaded as much as possible (other says that should be 8 and 8 to reduce scheduling overhead).

During the build the system is CPU bound quite a bit of the time (so more cores should help), but there are significant periods where the build dependency chain means this isn't the case and only two or three cores are active. Previously I recall comparing results with someone else and finding that having lots more cores (24, I think) didn't give a significant improvement in build time (certainly not for the 3x system build cost).

I've never seen peak memory usage go much above 9 GB during a build, and the peaks generally coincide with linking activities for "big" items (gcc, eglibc). This is likely to go higher with more active threads.

I started out with a RAID-0 SSD build array, but I didn't really see any difference over a single high-spec (consumer) SSD. As Ross said, running a fast file system on the disk is a good idea.

--

Chris Tapp
[email protected]
www.keylevel.com

----
You can tell you're getting older when your car insurance gets real cheap!

#3 lore raw
Quoted 8 lines · Chris Tapp
Quoted 5 lines · Ross
Quoted 4 lines · Oliver Novakovic

Can anyone recommend a reasonable performant hardware setup to use ?

What should be considered ? Are there any pitfalls ? What about bottlenecks in the build system ?

you should start by saying what you're going to build, my experience is quite different when building "small" images like console-image or even x11-image and "big" images/feeds which contain whole qt5 stack, 3 webkits and 2 chromium builds.

In general: bitbake will better utilize all available performance with bigger image (e.g. build time for console image won't change so much if you go from 8 cores to 24, but building e.g. just webkit alone will be more than twice faster on 24 cores).

Regards,

Quoted 47 lines
Quoted 17 lines
Quoted 7 lines

Specifically:

How many cores are recommended ? And how much cache is necessary ? How much of the main memory does Yocto really use ? Is 32 GB sufficient or should I go for 64 ?

Does it make sense to use two SSDs as Raid0 to get builds faster ?

As much of everything as you can afford. :) The build isn't heavy in any particular metric, so don't sacrifice RAM for SSDs for example.

RAID 0 over SSD would be nice and fast, but I prefer having a good amount of RAM and a tuned ext4 (no journal, long commit delay) so data doesn't actually hit the disk as frequently. Keeping the actual build directories on a separate disk is good for performance and not causing data loss when you lose a disk.

Quoted 3 lines

There are people that have 64GB in machines and then set TMPDIR to a tmpfs. Surprisingly this isn't that much faster (5% or so), but it's a lot easier on the hardware and power consumption.

My experience:

I've got a quad core with hyper-threading (so 8 usable cores) running at about 3.8 GHz, 16GB of RAM and use multiple SSDs - one to hold the meta data, downloads and top level build areas (local.conf, etc) and have the TMPDIR on a second SSD (so, as Ross says, I don't get a surprise when it wears out!).

I can build my images (basically an x11 image) in just under 60 minutes (once all the files have been fetched). I run with BB_NUMBER_THREADS and PARALLEL_MAKE both set to 16 to make sure the cores are fully loaded as much as possible (other says that should be 8 and 8 to reduce scheduling overhead).

During the build the system is CPU bound quite a bit of the time (so more cores should help), but there are significant periods where the build dependency chain means this isn't the case and only two or three cores are active. Previously I recall comparing results with someone else and finding that having lots more cores (24, I think) didn't give a significant improvement in build time (certainly not for the 3x system build cost).

I've never seen peak memory usage go much above 9 GB during a build, and the peaks generally coincide with linking activities for "big" items (gcc, eglibc). This is likely to go higher with more active threads.

I started out with a RAID-0 SSD build array, but I didn't really see any difference over a single high-spec (consumer) SSD. As Ross said, running a fast file system on the disk is a good idea.

--

Chris Tapp
[email protected]
www.keylevel.com

----
You can tell you're getting older when your car insurance gets real cheap!

Signature · 4 lines
_______________________________________________
yocto mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/yocto
Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]
#4 lore raw

Oliver,

Quoted 17 lines

-----Original Message-----
From: [email protected] [mailto:yocto- [email protected]] On Behalf Of Martin Jansa
Sent: Thursday, October 02, 2014 3:09 PM
To: Chris Tapp
Cc: [email protected]
Subject: Re: [yocto] Recommended Hardware for building

Quoted 4 lines · Chris Tapp
Quoted 1 line · Ross

On 2 October 2014 10:36, Oliver Novakovic <[email protected]>

Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line

Can anyone recommend a reasonable performant hardware setup to

use ?

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

What should be considered ? Are there any pitfalls ? What about bottlenecks in the build system ?

you should start by saying what you're going to build, my experience is quite different when building "small" images like console-image or even x11-image and "big" images/feeds which contain whole qt5 stack, 3 webkits and 2 chromium builds.

Agreed, what you are building and what your goals are makes a difference in what you need. I have a build machine setup that is mainly used to verify everything builds correctly after committing changes. It's an Intel i3-3220 with 8GB RAM. The autobuilder is setup on a Linux VM which is given 4GB RAM and does not recognize the extra Hyper-threaded cores, meaning it acts as a dual core machine. Rebuild of my console image typically takes under 20 minutes, and most of that time is packaging/install. After the initial build, there really isn't much for my system that needs to get rebuilt between commits.

So if you are looking for a build machine that is outside your normal workflow, a $400 PC may be enough for you. If this machine is for your development build and you have a have a lot of graphic applications that you need to build, you may want something more in line with what other people are suggesting.

Regards,
Bryan

Quoted 76 lines

In general: bitbake will better utilize all available performance with bigger image (e.g. build time for console image won't change so much if you go from 8 cores to 24, but building e.g. just webkit alone will be more than twice faster on 24 cores).

Regards,

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Specifically:

How many cores are recommended ? And how much cache is necessary

?

Quoted 14 lines
Quoted 14 lines
Quoted 4 lines

How much of the main memory does Yocto really use ? Is 32 GB sufficient or should I go for 64 ?

Does it make sense to use two SSDs as Raid0 to get builds faster ?

As much of everything as you can afford. :) The build isn't heavy in any particular metric, so don't sacrifice RAM for SSDs for example.

RAID 0 over SSD would be nice and fast, but I prefer having a good amount of RAM and a tuned ext4 (no journal, long commit delay) so data doesn't actually hit the disk as frequently. Keeping the actual build directories on a separate disk is good for performance and not causing data loss when you lose a disk.

Quoted 7 lines
Quoted 3 lines

There are people that have 64GB in machines and then set TMPDIR to a tmpfs. Surprisingly this isn't that much faster (5% or so), but it's a lot easier on the hardware and power consumption.

My experience:

I've got a quad core with hyper-threading (so 8 usable cores) running at

about 3.8 GHz, 16GB of RAM and use multiple SSDs - one to hold the meta data, downloads and top level build areas (local.conf, etc) and have the TMPDIR on a second SSD (so, as Ross says, I don't get a surprise when it wears out!).

Quoted 2 lines

I can build my images (basically an x11 image) in just under 60 minutes

(once all the files have been fetched). I run with BB_NUMBER_THREADS and PARALLEL_MAKE both set to 16 to make sure the cores are fully loaded as much as possible (other says that should be 8 and 8 to reduce scheduling overhead).

Quoted 2 lines

During the build the system is CPU bound quite a bit of the time (so more

cores should help), but there are significant periods where the build dependency chain means this isn't the case and only two or three cores are active. Previously I recall comparing results with someone else and finding that having lots more cores (24, I think) didn't give a significant improvement in build time (certainly not for the 3x system build cost).

Quoted 2 lines

I've never seen peak memory usage go much above 9 GB during a build,

and the peaks generally coincide with linking activities for "big" items (gcc, eglibc). This is likely to go higher with more active threads.

Quoted 2 lines

I started out with a RAID-0 SSD build array, but I didn't really see any

difference over a single high-spec (consumer) SSD. As Ross said, running a fast file system on the disk is a good idea.

Quoted 7 lines

--

Chris Tapp
[email protected]
www.keylevel.com

Quoted 8 lines

----
You can tell you're getting older when your car insurance gets real cheap!

--
_______________________________________________
yocto mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/yocto

--
Martin 'JaMa' Jansa jabber: [email protected]

#5 lore raw
In reply to Burton, Ross
Quoted 23 lines · Ross
Quoted 5 lines · Oliver Novakovic

Can anyone recommend a reasonable performant hardware setup to use ?

What should be considered ? Are there any pitfalls ? What about bottlenecks in the build system ?

Quoted 7 lines

Specifically:

How many cores are recommended ? And how much cache is necessary ? How much of the main memory does Yocto really use ? Is 32 GB sufficient or should I go for 64 ?

Does it make sense to use two SSDs as Raid0 to get builds faster ?

As much of everything as you can afford. :) The build isn't heavy in any particular metric, so don't sacrifice RAM for SSDs for example.

RAID 0 over SSD would be nice and fast, but I prefer having a good amount of RAM and a tuned ext4 (no journal, long commit delay) so data doesn't actually hit the disk as frequently. Keeping the actual build directories on a separate disk is good for performance and not causing data loss when you lose a disk.

Quoted 3 lines

There are people that have 64GB in machines and then set TMPDIR to a tmpfs. Surprisingly this isn't that much faster (5% or so), but it's a lot easier on the hardware and power consumption.

That's how I roll - after I lost few SSD drives by constantly building and rebuilding stuff, I ended up shoving 64GB of RAM into my gear and building into RAM-based tmpfs. Of course, there are size limits to what can be built in one go, but there are workarounds and ways to serialize builds...

Signature · 1 line
Denys