CONFIG_NO_HZ_FULL restricts cpu usage to the equivalent of one in 4.2
New title, I originally posted this last night but I've now made a little progress in identifying what changed. Previous thread was labelled for AMD Phenom, but it is more general. CC'ing Jeff because he replied to the original, I guess he probably won't be interested after this.
Yesterday was the first day I had done any real compiling with 4.2-rc. I started out using -rc1, and make -j4 on a recent LFS/BLFS system (gcc-5.1.0, make-4.1, etc). I wanted to start trying to build kde5, and I expected a lot of issues. What I did not expect was that qt5 would build as if I was using -j1.
Examination eventually identified that with 4.2-rc1 and 4.2-rc2, make ran the number of jobs I had specified, but the total of the CPU percentages ('top' from procps-ng-3.3.10) maxed out at 100%. On 4.1 kernels the percentage with -j4 maxes out at 400% (my machine has 4 cores). I suspected either an unfortunate choice in 'make oldconfig', or something specific to an AMD Phenom / gcc-5.1.
Today I have tried make -j4 on two other machines with 4.2-rc kernels [ building the current git stable release ]. On my i3 SandyBridge everything was fine, CPU usage approached 400%. On my AMD A10-7850K I have the same problem as on the phenom. Not surprising, I began by using the Phenom config when I got the A10, then adapted it to suit, whereas the i3 has much less memory so I haven't made many changes since I got it.
Comparing the configs for the i3 and the A10, the first thing which looked as if it might be relevant was the _CPU_ACCOUNTING choices. Those on the A10 seem to be driven from CONFIG_NO_HZ_FULL so I began by changing that to CONFIG_NO_HZ_IDLE. Payday ;-) make -j4 now approaches 400% CPU usage.
The config differences follow. Perhaps it is actually one of the subsequent choices that is the problem. And I guess it could still be a gcc-5.1 issue.
config-4.2-speed-ok renamed from config-4.2-initial +4 -6
@@ -104,11 +104,8 @@ CONFIG_TICK_ONESHOT=y CONFIG_NO_HZ_COMMON=y # CONFIG_HZ_PERIODIC is not set -# CONFIG_NO_HZ_IDLE is not set -CONFIG_NO_HZ_FULL=y -CONFIG_NO_HZ_FULL_ALL=y -CONFIG_NO_HZ_FULL_SYSIDLE=y -CONFIG_NO_HZ_FULL_SYSIDLE_SMALL=4 +CONFIG_NO_HZ_IDLE=y +# CONFIG_NO_HZ_FULL is not set # CONFIG_NO_HZ is not set CONFIG_HIGH_RES_TIMERS=y
@@ -116,7 +113,9 @@ # CPU/Task time and stats accounting # CONFIG_VIRT_CPU_ACCOUNTING=y +# CONFIG_TICK_CPU_ACCOUNTING is not set CONFIG_VIRT_CPU_ACCOUNTING_GEN=y +# CONFIG_IRQ_TIME_ACCOUNTING is not set # CONFIG_BSD_PROCESS_ACCT is not set CONFIG_TASKSTATS=y CONFIG_TASK_DELAY_ACCT=y
@@ -131,7 +130,6 @@ # CONFIG_TASKS_RCU is not set CONFIG_RCU_STALL_COMMON=y CONFIG_CONTEXT_TRACKING=y -CONFIG_RCU_USER_QS=y CONFIG_CONTEXT_TRACKING_FORCE=y # CONFIG_TREE_RCU_TRACE is not set CONFIG_RCU_NOCB_CPU=y
Anyway, I'll start a bisection. But it might take me a few days, this is not a convenient time (somehow, kernel issues which need bisection always come at a bad time for me).
ĸen
Signature · 1 line
This one goes up to eleven!
Quoted 33 lines · Ken Moffat
New title, I originally posted this last night but I've now made a little progress in identifying what changed. Previous thread was labelled for AMD Phenom, but it is more general. CC'ing Jeff because he replied to the original, I guess he probably won't be interested after this.
Yesterday was the first day I had done any real compiling with 4.2-rc. I started out using -rc1, and make -j4 on a recent LFS/BLFS system (gcc-5.1.0, make-4.1, etc). I wanted to start trying to build kde5, and I expected a lot of issues. What I did not expect was that qt5 would build as if I was using -j1.
Examination eventually identified that with 4.2-rc1 and 4.2-rc2, make ran the number of jobs I had specified, but the total of the CPU percentages ('top' from procps-ng-3.3.10) maxed out at 100%. On 4.1 kernels the percentage with -j4 maxes out at 400% (my machine has 4 cores). I suspected either an unfortunate choice in 'make oldconfig', or something specific to an AMD Phenom / gcc-5.1.
Today I have tried make -j4 on two other machines with 4.2-rc kernels [ building the current git stable release ]. On my i3 SandyBridge everything was fine, CPU usage approached 400%. On my AMD A10-7850K I have the same problem as on the phenom. Not surprising, I began by using the Phenom config when I got the A10, then adapted it to suit, whereas the i3 has much less memory so I haven't made many changes since I got it.
Comparing the configs for the i3 and the A10, the first thing which looked as if it might be relevant was the
_CPU_ACCOUNTINGchoices. Those on the A10 seem to be driven fromCONFIG_NO_HZ_FULLso I began by changing that toCONFIG_NO_HZ_IDLE. Payday ;-) make -j4 now approaches 400% CPU usage.
Quoted 4 lines
The config differences follow. Perhaps it is actually one of the subsequent choices that is the problem. And I guess it could still be a gcc-5.1 issue.
Before going nuts bisecting, it could be worth running perf record -a -g -e cycles (or perhaps -e task-clock instead of -e cycles). It could also be worth manually sampling /proc/PID/stack a few times for a process that isn't making as much progress as it should.
--Andy
Quoted 7 lines · Andy Lutomirski
Before going nuts bisecting, it could be worth running perf record -a -g -e cycles (or perhaps -e task-clock instead of -e cycles). It could also be worth manually sampling
/proc/PID/stacka few times for a process that isn't making as much progress as it should.--Andy
I think Frederic has already spotted the problem - in the early bisects, everything was building on CPU 0. But thanks for the suggestion.
ĸen
Signature · 1 line
This one goes up to eleven!
2015-07-15 18:27 GMT+02:00 Ken Moffat <[email protected]>:
Quoted 40 lines
New title, I originally posted this last night but I've now made a little progress in identifying what changed. Previous thread was labelled for AMD Phenom, but it is more general. CC'ing Jeff because he replied to the original, I guess he probably won't be interested after this.
Yesterday was the first day I had done any real compiling with 4.2-rc. I started out using -rc1, and make -j4 on a recent LFS/BLFS system (gcc-5.1.0, make-4.1, etc). I wanted to start trying to build kde5, and I expected a lot of issues. What I did not expect was that qt5 would build as if I was using -j1.
Examination eventually identified that with 4.2-rc1 and 4.2-rc2, make ran the number of jobs I had specified, but the total of the CPU percentages ('top' from procps-ng-3.3.10) maxed out at 100%. On 4.1 kernels the percentage with -j4 maxes out at 400% (my machine has 4 cores). I suspected either an unfortunate choice in 'make oldconfig', or something specific to an AMD Phenom / gcc-5.1.
Today I have tried make -j4 on two other machines with 4.2-rc kernels [ building the current git stable release ]. On my i3 SandyBridge everything was fine, CPU usage approached 400%. On my AMD A10-7850K I have the same problem as on the phenom. Not surprising, I began by using the Phenom config when I got the A10, then adapted it to suit, whereas the i3 has much less memory so I haven't made many changes since I got it.
Comparing the configs for the i3 and the A10, the first thing which looked as if it might be relevant was the
_CPU_ACCOUNTINGchoices. Those on the A10 seem to be driven fromCONFIG_NO_HZ_FULLso I began by changing that toCONFIG_NO_HZ_IDLE. Payday ;-) make -j4 now approaches 400% CPU usage.The config differences follow. Perhaps it is actually one of the subsequent choices that is the problem. And I guess it could still be a gcc-5.1 issue.
config-4.2-speed-ok 2015-07-15 17:00:50.919998703 +0100 line 109
Quoted 6 lines
config-4.2-speed-ok 2015-07-15 17:00:50.919998703 +0100 renamed from config-4.2-initial 2015-07-15 16:25:12.548005751 +0100 +0
-3@@ -104,11 +104,8 @@ CONFIG_TICK_ONESHOT=y CONFIG_NO_HZ_COMMON=y # CONFIG_HZ_PERIODIC is not set -# CONFIG_NO_HZ_IDLE is not set -CONFIG_NO_HZ_FULL=y -CONFIG_NO_HZ_FULL_ALL=y
You had CONFIG_NO_HZ_FULL_ALL enabled? Because that would indeed produce that effect since it isolates all CPUs but 0 off sched domains.
Which means that basically only CPU 0 runs user tasks unless you forces these otherwise.
Quoted 20 lines · Frederic Weisbecker
2015-07-15 18:27 GMT+02:00 Ken Moffat <[email protected]>:
Quoted 8 lines
The config differences follow. Perhaps it is actually one of the subsequent choices that is the problem. And I guess it could still be a gcc-5.1 issue.
config-4.2-speed-ok 2015-07-15 17:00:50.919998703 +0100 line 109
Quoted 6 lines
config-4.2-speed-ok 2015-07-15 17:00:50.919998703 +0100 renamed from config-4.2-initial 2015-07-15 16:25:12.548005751 +0100 +0
-3@@ -104,11 +104,8 @@ CONFIG_TICK_ONESHOT=y CONFIG_NO_HZ_COMMON=y # CONFIG_HZ_PERIODIC is not set -# CONFIG_NO_HZ_IDLE is not set -CONFIG_NO_HZ_FULL=y -CONFIG_NO_HZ_FULL_ALL=yYou had
CONFIG_NO_HZ_FULL_ALLenabled? Because that would indeed produce that effect since it isolates all CPUs but 0 off sched domains.
Quoted 2 lines
Which means that basically only CPU 0 runs user tasks unless you forces these otherwise.
Thanks. I'll put it down to a bad .config choice, although it was fine on early 4.1. While I was starting to bisect, I noticed that on the A10 everything was happening on CPU 0 - not sure if that was happening on the original box, but for the moment it sounds likely.
ĸen
Signature · 1 line
This one goes up to eleven!
Quoted 21 lines · Ken Moffat
Quoted 20 lines · Frederic Weisbecker
2015-07-15 18:27 GMT+02:00 Ken Moffat <[email protected]>:
Quoted 8 lines
The config differences follow. Perhaps it is actually one of the subsequent choices that is the problem. And I guess it could still be a gcc-5.1 issue.
config-4.2-speed-ok 2015-07-15 17:00:50.919998703 +0100 line 109
Quoted 6 lines
config-4.2-speed-ok 2015-07-15 17:00:50.919998703 +0100 renamed from config-4.2-initial 2015-07-15 16:25:12.548005751 +0100 +0
-3@@ -104,11 +104,8 @@ CONFIG_TICK_ONESHOT=y CONFIG_NO_HZ_COMMON=y # CONFIG_HZ_PERIODIC is not set -# CONFIG_NO_HZ_IDLE is not set -CONFIG_NO_HZ_FULL=y -CONFIG_NO_HZ_FULL_ALL=yYou had
CONFIG_NO_HZ_FULL_ALLenabled? Because that would indeed produce that effect since it isolates all CPUs but 0 off sched domains.
Quoted 5 lines
Quoted 2 lines
Which means that basically only CPU 0 runs user tasks unless you forces these otherwise.
Thanks. I'll put it down to a bad .config choice, although it was fine on early 4.1.
Yeah we decided to include the nohz_full on cpu_isolated_map recently, except CPU 0.
Quoted 7 lines
While I was starting to bisect, I noticed that
on the A10 everything was happening on CPU 0 - not sure if that was happening on the original box, but for the moment it sounds likely.ĸen
Signature · 1 line
This one goes up to eleven!