One core idles with 4 nice and 1 regular process
First, a description of the problem, and a fairly simple way to test it:
All of this is running on my quad-core Q6600. First, I start four nice -n 19 processes, each of which does nothing but busy-wait. (In my case, I used a simple python script, but any CPU-bound process will do.) At this point, top shows all four cores being utilized, at approximately 100% nice each.
At this point, I start another of these busy-waiting processes, except with no nice adjustment, so it runs at regular priority. After doing so, top now reports the following CPU usage: one core is 100% user, two cores are 100% nice, and finally, one last core is 100% idle, doing nothing, despite a total of five processes being available to run.
I thought I first noticed this running a 2.6.29 kernel, and thought about bisecting, but my test 2.6.28 kernel also exhibited the behavior. I couldn't test any previous kernels due to a lack of ext4 support. Both my current 2.6.30 kernel and the torvalds/linux-2.6.git HEAD also show the behavior.
Now, I'm not sure if this is a bug or a strange feature (though it seems very bug-like to me) or perhaps a configuration problem, but I'd like to find out nonetheless.
If I can provide any additional information to help, I'll be happy to oblige.
Quoted 2 lines · Jason Lynch
If I can provide any additional information to help, I'll be happy to oblige.
I discovered a post on lkml from last October, wherein a user reports what sounds like the same issue on 2.6.25.9.
http://article.gmane.org/gmane.linux.kernel/748316/
In addition, Ubuntu's current 2.6.28 kernel appears to be immune from the problem, suggesting a possible configuration issue, or at least an option that can trigger/avoid the problem. However, comparing the configs is difficult due to the number of added, changed, and removed options between kernel versions. The differences that stood out in my initial comparisons were CONFIG_RT_GROUP_SCHED and CONFIG_CPUSETS, but I haven't yet had an opportunity to test either of them, and neither sounds especially promising.
I won't personally be able to do any more extensive kernel testing for perhaps a couple of weeks. However, if this is merely a configuration issue, it seems rather opaque to me, as I can't imagine this behavior being desirable in any case.