Skip to contentSkip to conversation

Tricks to speed up kernel builds

DISCUSSION Ozan Çağlayan 3 participants 3 messages Sep 15, 2009 – Sep 16, 2009

lore
Share & tools
lore
#0 lore raw

Hi,

I'd googled a lot about the $subject but couldn't find any detailed guide about it. So, the followings are what I understand from what I've found:

1. Build inside a tmpfs to avoid I/O bottlenecks,
2. Use ccache,
3. Switch governor to performance if supported,
4. Pass -j to gnu make to parallelize the build process,
5. Don't build debug symbols a.k.a set CONFIG_DEBUG_INFO=n
6. There are a bunch of sched patches in tip which apparently improves things during kbuild
7. Distribute the work into the local network using icecream, distcc.

For the -j part, how many jobs should we pass to it? I've seen people passing

- NR_of_cores+1
- NR_of_cores x 2 (I think that this is useful when there's 2 threads of execution per core like HyperThreading?)
- 64, 128, etc. regardless of the number of cores (Is it really useful regardless of the CPU)

The curious question is *what build times can you get* with your typical systems using your speedup tricks (quad core, dual core, etc.) ? What are other tricks that I can use to even more speed up the build?

Do you really use icecream/distcc on your daily test builds?

How about nicing/ionicing make on an idle system? Will it make any difference?

Thanks a lot
Ozan Caglayan

#1 lore raw
Quoted 24 lines · Ozan Çağlayan

Hi,

I'd googled a lot about the $subject but couldn't find any detailed guide about it. So, the followings are what I understand from what I've found:

1. Build inside a tmpfs to avoid I/O bottlenecks,
2. Use ccache,
3. Switch governor to performance if supported,
4. Pass -j to gnu make to parallelize the build process,
5. Don't build debug symbols a.k.a set CONFIG_DEBUG_INFO=n
6. There are a bunch of sched patches in tip which apparently improves things during kbuild
7. Distribute the work into the local network using icecream, distcc.

For the -j part, how many jobs should we pass to it? I've seen people passing

- NR_of_cores+1
- NR_of_cores x 2 (I think that this is useful when there's 2 threads of execution per core like HyperThreading?)
- 64, 128, etc. regardless of the number of cores (Is it really useful regardless of the CPU)

Quoted 8 lines

The curious question is *what build times can you get* with your typical systems using your speedup tricks (quad core, dual core, etc.) ? What are other tricks that I can use to even more speed up the build?

Do you really use icecream/distcc on your daily test builds?

How about nicing/ionicing make on an idle system? Will it make any difference?

I've only done a few compiles recently, and I only really use icecream and make -jX to speed up compile times.

I have a 4 node icecc cluster, three quad cores (one limited to 2 cores), and one dual core (limited to a single core), which gives 11 total cores I can distribute to, and I usually pass: make -j12

Any higher and I don't really see any speedups, any lower, same difference.

My last kernel compile (2.6.31-git4) took about 9 minutes.

Quoted 12 lines

Thanks a lot
Ozan Caglayan

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 2 lines
Thomas Fjellstrom
[email protected]
#2 lore raw
Quoted 8 lines · Thomas Fjellstrom
Quoted 7 lines · Ozan ?a?layan

Hi,

I'd googled a lot about the $subject but couldn't find any detailed guide about it. So, the followings are what I understand from what I've found:

1. Build inside a tmpfs to avoid I/O bottlenecks,

if you have the ram the system will use it to cache the data, avoiding the I/O bottlenecks.

the first time you compile you have to wait for the source to be read into ram, but you would have to copy the source to the tmpfs anyway

Quoted 8 lines
Quoted 8 lines

2. Use ccache,
3. Switch governor to performance if supported,
4. Pass -j to gnu make to parallelize the build process,
5. Don't build debug symbols a.k.a set CONFIG_DEBUG_INFO=n
6. There are a bunch of sched patches in tip which apparently improves things during kbuild
7. Distribute the work into the local network using icecream, distcc.

Quoted 6 lines
Quoted 6 lines

For the -j part, how many jobs should we pass to it? I've seen people passing

- NR_of_cores+1
- NR_of_cores x 2 (I think that this is useful when there's 2 threads of execution per core like HyperThreading?)

this is not due to hyperthreading.

if you do not have the data cached in ram from a prior build, individual compile threads will stall waiting for the source to be read from disk. by having more threads than you have cores you can have some stalled waiting for data while others run and use the cpu cycles.

exactly how much benfit you get, and where things taper off is very dependant on your system (how fast is your I/O, how effectivly does it handle parallel requests)

it's probably higher than NR_of_cores +1, but if you have a 8-core or 16 core box it's probably less than NR_of_cores x 2 (unless you have a storage subsystem that works well in parallel, say something like a 8 or 16 disk raid array)

Quoted 2 lines
Quoted 2 lines

- 64, 128, etc. regardless of the number of cores (Is it really useful regardless of the CPU)

this will overload your system with lots of processes competing for the CPU. it's a good stress test of the system, but inefficiant in processor utilization due to the fact that each time a new thread takes the CPU the cache has been wiped out by other threads.

Quoted 5 lines
Quoted 5 lines

The curious question is *what build times can you get* with your typical systems using your speedup tricks (quad core, dual core, etc.) ? What are other tricks that I can use to even more speed up the build?

Do you really use icecream/distcc on your daily test builds?

most people don't, some people do. some people have systems that can build the kernel in a handful of seconds (I think the record was 7.5 seconds several years ago) if it is in the memory disk cache and don't need to. others build on architecures that don't have systems that fast and use distcc all the time to get acceptable speed

since many virtualization options will not let a guest OS use more than one real CPU at a time, people who build in virtual systems commonly use this sort of thing to use all the real cores on their system (same as the -j option above)

Linus has been heard to complain that it takes longer to boot a system with a new kernel than to compile it (this was on an enterprise-class system that took a long time for the bios/raid/etc initialization)

Quoted 2 lines
Quoted 2 lines

How about nicing/ionicing make on an idle system? Will it make any difference?

if there is nothing to compete with the kernel build, these don't help.

David Lang

Quoted 26 lines

I've only done a few compiles recently, and I only really use icecream and make -jX to speed up compile times.

I have a 4 node icecc cluster, three quad cores (one limited to 2 cores), and one dual core (limited to a single core), which gives 11 total cores I can distribute to, and I usually pass: make -j12

Any higher and I don't really see any speedups, any lower, same difference.

My last kernel compile (2.6.31-git4) took about 9 minutes.

Quoted 12 lines

Thanks a lot
Ozan Caglayan

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/