BUG in ht-aware scheduler/nice in 2.6.7-rc2 on dual xeon

Con, Ingo, List,

I'm overjoyed with decent ht-aware scheduling in 2.6.7-rc2 and it does mostly the right thing. However, the 'nice' work by Con shows some slight problems.

Please find attached program 'eat-time.cc'. Make sure not to compile it with -O which might confuse things as this program basically does nothing.

Run it without arguments to determine the speed of 1 cpu, it outputs a number (megaloops/second). Then start it with that number as a parameter:

Sample:

$ ./eat-time
592
$ ./eat-time 592
99%
99%
100%
etc

Now starting four of these at the same time gives the desired result:

$ ./eat-time 592 & ./eat-time 592 & ./eat-time 592 & ./eat-time 592
50%
50%
50%
50% 
etc

This however:

$ ./eat-time 592 & ./eat-time 592 & 
100%
99%
In another xterm:
$ nice -n +19 ./eat-time 592 & nice -n +19 ./eat-time 592
5%
5%
5%

Fails sometimes, with all processes getting 50%. The above 'screenshot' is from the working and expected situation, which happens most of the time.

When it goes wrong, top shows me that Cpu0 and Cpu1 are 100% user, while Cpu2 and Cpu3 are both 100% nice. The niced processes show up in top as PRiority 39, the unniced ones (NI = 0) as PR 25.

I've also seen it that Cpu2 and Cpu3 are 100% busy, and 0 and 1 are 100% nice.

I'd say this situation happens once every 5 or 8 invocations, and perhaps somewhat more when first starting the niced processes.

Perhaps related, when running the above 'nice -n +19' line on its own, I see all CPUs getting load over time, the two processes are wandering. After a while they settle down, only to go on wandering again some time later, also touching configurations where a physical cpu suddenly hosts two processes.

Without nice, two processes get firmly pegged to different physical CPUs.

Anything I can do to help resolve this, just yell.

Thanks!

http://www.PowerDNS.com      Open source, database driven DNS Software 
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO

This is just because the scheduler balancing is not aware of nice and when two same niceness tasks are on the same physical core they get equal shares. The ht-aware nice only works at keeping different nice values on the same physical core fair. There is no more that can be done using the current ht aware mechanism; a far more complicated balancing algorithm that takes nice into account would be required.

Con

Ahh, that is because we don't do nice-aware SMP balancing. It is orthogonal to the HT-nice work.

Fixing it properly requires someone to sit down and do some careful design of nice-aware balancing that should be general enough to handle the HT problems as a matter of course.