Parallel compilation performance regression
Hi David,
I've been experiencing a performance regression when running a parallel compilation (eg, make -j72) on recent kernels.
I bisected it to this commit:
commit b667b867344301e24f21d4a4c844675ff61d89e1
Author: David Howells <[email protected]> Date: Tue Sep 24 16:09:04 2019 +0100
pipe: Advance tail pointer inside of wait spinlock in pipe_read()
Advance the pipe ring tail pointer inside of wait spinlock in pipe_read() so that the pipe can be written into with kernel notifications from contexts where pipe->mutex cannot be taken.
Prior to this commit I got 70% or so thread saturation of cc1 and after it rarely gets above 15% and would often drop down to 1-2 threads. It doesn't look like a clean revert either. Looking at upstream, it seems that some later code changed the wakeup. I'm not really sure how this all fits into parallelized make.
Best
Jon
Quoted 4 lines · Jonathan
Hi David,
I've been experiencing a performance regression when running a parallel compilation (eg, make -j72) on recent kernels.
I bet you're using a version of make which predates 4.3:
Quoted 10 lines · Matthew Wilcox
Quoted 4 lines · Jonathan
Hi David,
I've been experiencing a performance regression when running a parallel compilation (eg, make -j72) on recent kernels.
I bet you're using a version of make which predates 4.3:
I am!
# make --version GNU Make 4.2.1
Thank you Matthew!
Quoted 13 lines · Jonathan
Quoted 10 lines · Matthew Wilcox
Quoted 4 lines · Jonathan
Hi David,
I've been experiencing a performance regression when running a parallel compilation (eg, make -j72) on recent kernels.
I bet you're using a version of make which predates 4.3:
Quoted 7 lines
I am!
# make --version GNU Make 4.2.1Thank you Matthew!
Check your distribution and included patches on top of a vanilla make v4.2. Debian had some important ones I described in [1].
When proposing "make version 4.3" for tc-build - an opportunity to build a llvm-toolchain the easy way - we saw different numbers. I pointed in [1] to the Linus patch Matthew did here.
Personally, with switching to Debian's make version 4.3-3 I have seen no big differences when using "make -j3" to build Linux v5.7+. That might be different with "make -j72"...
...can I have SSH access to this machine, please :-),
You forgot to tell which Linux version you use.
If you are interested please look at closed tc-build issue #72 for our analysis. For tc-build it did not matter - (Debian's) make v4.2 had some slightly better performance.
BTW, with that pipe improvements in Linux v5.7 I see some better numbers when using pipebench to benchmark my devices:
Example: SanDisk iSSD 16GB
root# cat /dev/sdb | pipebench > /dev/null
Summary:
Piped 14.91 GB in 00h01m25.20s: 179.23 MB/second
Before: approx. 100MB/s
BTW, I heard of hyperfine benchmark tool the first time when dealing with make performance (see [2]) in ClangBuiltLinux.
I would like to see some benchmark numbers with hyperfine from you if you do not mind.
Stay OPEN minded and curious.
Thanks.
- Sedat -
[1] https://github.com/ClangBuiltLinux/tc-build/issues/72
[2] https://github.com/sharkdp/hyperfine
[3] https://github.com/sharkdp/hyperfine/releases