SMP Affinity and nice
I am trying to come to an understanding as to why 1 process is getting less cpu time than identical processes with a higher "nice" value. Server has 2 physical processors with hyperthreading (cpu 0,1,2,3)
I am starting 5 processes that perform a square root loop to max out a cpu. They use the exact same code but are renamed for identification:
cpumax1, cpumax2, cpumax3, cpumax4, cpumax5
I start in order:
1. nice -n 10 cpumax1 2. nice -n 10 cpumax2 3. nice -n 10 cpumax3 4. nice -n 10 cpumax4 5. nice -n 0 cpumax5
Here is the top output:
PR NI S %CPU TIME+ P COMMAND 35 10 R 99.9 1:46.90 3 cpumax1 35 10 R 99.9 1:41.01 1 cpumax3 35 10 R 99.9 1:39.48 0 cpumax4
25 0 R 66.9 1:03.13 2 cpumax5
35 10 R 33.0 0:39.30 2 cpumax2
cpumax1 is using processor 3, 99%
cpumax2 is using processor 2, 33%
cpumax3 is using processor 1, 99%
cpumax4 is using processor 0, 99%
cpumax5 is using processor 2, 66%
So since cpumax5 has a lower nice value and thus a higher priority (25 in
this case), shouldn't it be given it's own cpu. If I give cpumax5 a nice
value of -20, it does start using it's own cpu. I don't want to manage cpu affinity via taskset command.
My explanation would be that since the scheduler tries to limit cpu affinity, the nice value of 0 isn't enough to get the scheduler to move this process to another processors run queue. I could be totally wrong here though.
I should also note here that this test is totally dependent on the order of startup. If I start cpumax5 first with a nice value of 0 before the other 4, it will get it's own cpu:
PR NI S %CPU TIME+ P COMMAND
35 10 R 99.9 1:00.03 3 cpumax2 25 0 R 99.9 1:08.01 1 cpumax5 35 10 R 99.9 1:03.69 2 cpumax1 35 10 R 50.3 0:29.02 0 cpumax3 35 10 R 49.6 0:26.37 0 cpumax4
I just want to understand this better. Thanks.
Quoted 9 lines
Subject: SMP Affinity and nice
I am trying to come to an understanding as to why 1 process is getting less cpu time than identical processes with a higher "nice" value. Server has 2 physical processors with hyperthreading (cpu 0,1,2,3)
I am starting 5 processes that perform a square root loop to max out a cpu. They use the exact same code but are renamed for identification:
cpumax1, cpumax2, cpumax3, cpumax4, cpumax5
[...]
What you describe should be addressed in the -ck patchset (smpnice-...diff) Not sure if it is in mainline already, though.
Jan Engelhardt
Quoted 6 lines · Jan Engelhardt
Quoted 6 lines
Subject: SMP Affinity and nice
I am trying to come to an understanding as to why 1 process is getting less cpu time than identical processes with a higher "nice" value. Server has 2 physical processors with hyperthreading (cpu 0,1,2,3)
Quoted 6 lines
Quoted 3 lines
I am starting 5 processes that perform a square root loop to max out a cpu. They use the exact same code but are renamed for identification:
cpumax1, cpumax2, cpumax3, cpumax4, cpumax5[...]
What you describe should be addressed in the -ck patchset (smpnice-...diff) Not sure if it is in mainline already, though.
It's coming in 2.6.18.
Rich,
What kernel version are you using when you see this phenomenon?
Peter
Signature · 4 lines
Peter Williams [email protected] "Learning, n. The kind of ignorance distinguishing the studious." -- Ambrose Bierce
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 2 lines · Rich Paredes
Hi Peter. 2.6.5
OK. That version does not take tasks' nice values into account when doing load balancing and just tries to evenly distribute the tasks among the CPUs. The smpnice patches (to be included in the 2.6.18 kernel) change this so that nice is taken into account when load balancing is done and should fix your problem.
Peter
Signature · 4 lines
Peter Williams [email protected] "Learning, n. The kind of ignorance distinguishing the studious." -- Ambrose Bierce
Quoted 8 lines · Rich Paredes
So since cpumax5 has a lower nice value and thus a higher priority (25 in
this case), shouldn't it be given it's own cpu. If I give cpumax5 a nice
value of -20, it does start using it's own cpu.My explanation would be that since the scheduler tries to limit cpu affinity, the nice value of 0 isn't enough to get the scheduler to move this process to another processors run queue. I could be totally wrong here though.
I think you are correct. The load balancer doesn't think that this is enough of an imbalance to go through the effort of swapping two processes around.
Chris
Quoted 6 lines · Chris Friesen
Quoted 4 lines · Rich Paredes
So since cpumax5 has a lower nice value and thus a higher priority (25 in
this case), shouldn't it be given it's own cpu. If I give cpumax5 a nice
value of -20, it does start using it's own cpu.
Quoted 6 lines
Quoted 4 lines
My explanation would be that since the scheduler tries to limit cpu affinity, the nice value of 0 isn't enough to get the scheduler to move this process to another processors run queue. I could be totally wrong here though.
I think you are correct. The load balancer doesn't think that this is enough of an imbalance to go through the effort of swapping two processes around.
The kernel in use (2.6.5) doesn't take nice into account during load balancing and just allocates the 5 tasks among the 4 CPUs in a way that tries to give each CPU the same number of tasks. It also tries not to move tasks around too much so when it has found a solution that satisfies that criterion it leaves the tasks there.
5 tasks among 4 CPUs means 1 task each for 3 of the CPUs and 2 tasks for the other CPU. As nice isn't taken into account it is purely down to chance whether or not the high priority task ends up being one of those that gets a CPU to itself or has to share with another task. Some elementary probability theory should enable the probability of a "good" outcome (i.e. the high priority task not having to share) to be calculated.
This is an example of the type of situation that the smpnice patches were designed to handle. They take nice into account and should ensure that the high priority does get a CPU to itself in this scenario. They are scheduled for release in the 2.6.18 kernel.
Peter
Signature · 4 lines
Peter Williams [email protected] "Learning, n. The kind of ignorance distinguishing the studious." -- Ambrose Bierce