Cpufreq regression in 2.6.38rc3 on i7 SMP

I noticed this regression wrt powersavings/cpufreq between 2.6.32 and 2.6.37 up to 38rc3. I originally thought I am looking at a BIOS problem but I don't think so anymore.

I have this dual i7 that has cpufreq clock up the wrong CPU.

I have seen this problem in 2.6.37, 2.6.38rc2 2.6.38rc3 but I ran on 2.6.32 between .12 and .25 and I have not seen it on there.

It's 2x 6-core 2.93 GHz Nehalem, Supermicro X8DTV. According to i7z the CPUs idle at 1.6 GHz. When putting on a single CPU using process I can see that the CPU load ends up on the first core in the first CPU. But the core that is then upped to 2.9 GHz is the first core in the *second* CPU.

Cpu speed from cpuinfo 2933.00Mhz
True Frequency (without accounting Turbo) 2933 MHz

Socket [0] - [physical cores=6, logical cores=12, max online cores ever=6]
CPU Multiplier 22x || Bus clock frequency (BCLK) 133.32 MHz
TURBO ENABLED on 6 Cores, Hyper Threading ON
True Frequency 3066.32 MHz (133.32 x [23])
Max TURBO Multiplier (if Enabled) with 1/2/3/4/5/6 cores is 25x/25x/24x/24x/24x/24x Current Frequency 1626.05 MHz (Max of below)

        Core [core-id]  :Actual Freq (Mult.)      C0%   Halt(C1)%  C3 %   C6 %  Temp(in *C)
        Core 1 [0]:       1599.82 (12.00x)       100    45.5       0       0    37
        Core 2 [1]:       1599.84 (12.00x)         1    0.551      0    99.4    27
        Core 3 [2]:       1599.98 (12.00x)         0    0.00654    0     100    27
        Core 4 [3]:       1535.95 (11.52x)         0    0.00688    0     100    31
        Core 5 [4]:       1626.05 (12.20x)         0    0.0321     0     100    27
        Core 6 [5]:       1599.99 (12.00x)         0    0.00673    0     100    29

29
Socket [1] - [physical cores=6, logical cores=12, max online cores ever=6]
CPU Multiplier 22x || Bus clock frequency (BCLK) 133.32 MHz
TURBO ENABLED on 6 Cores, Hyper Threading ON
True Frequency 3066.32 MHz (133.32 x [23])
Max TURBO Multiplier (if Enabled) with 1/2/3/4/5/6 cores is 25x/25x/24x/24x/24x/24x Current Frequency 2856.88 MHz (Max of below)

        Core [core-id]  :Actual Freq (Mult.)      C0%   Halt(C1)%  C3 %   C6 %  Temp(in *C)
        Core 1 [6]:       2856.88 (21.43x)      1.42       0       1    97.7    29
        Core 2 [7]:       1967.00 (14.75x)         1    0.196      0    99.8    24
        Core 3 [8]:       2139.22 (16.05x)         1    0.0745     0    99.9    24
        Core 4 [9]:       1951.43 (14.64x)         0    0.0272     0     100    25
        Core 5 [10]:      2013.44 (15.10x)         0    0.0276     0     100    27
        Core 6 [11]:      1971.25 (14.79x)         0    0.0298     0     100    27

C0 = Processor running without halting
C1 = Processor running with halts (States >C0 are power saver) C3 = Cores running with PLL turned off and core cache turned off C6 = Everything in C3 + core state saved to last level cache
Above values in table are in percentage over the last 1 sec
[core-id] refers to core-id number in /proc/cpuinfo 'Garbage Values' message printed when garbage values are read
Ctrl+C to exit

Interestingly it is specific to the application. If you replace my application with drystones things work as they should. My application uses a lot of memory mappings (both file based and anonymous) but it doesn't slack off with I/O, it will run at 99.9+% CPU as reported by rusage after pages are in. It also has code generated by a compiler that certainly doesn't do a good job at microoptimizations, and that is what I think is confusing the kernel here. A hypothesis is that I don't spend enough time in C0% for cpufreq to notice. In any case, it is a regression wrt 2.6.32 and it wouldn't explain why that other CPU core does get clocked up.

This problem doesn't show right after boot. It mostly appears after a couple minutes of this process consuming close to 100% CPU (but sometimes never). I can run my application just fine after boot and you get the expected fast runtime and then after a while it gets wedged like the above and never comes out.

Even on runs that are mostly successful (doesn't get permanently wedged) I can see in i7z that for some periods the core that the process is on isn't clocked at 2.93. This usually happens when the process was on core 1/12 in cpu 1/2, correctly clocked there but sometimes it gets moved to 3/12 and that one fails to go up until the scheduler picks cpu 1/12 again.

Kernel config, dmesg on
http://www.cons.org/linux-kernel-201102/

Is anyone else seeing this?

Martin

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Martin Cracauer <[email protected]>   http://www.cons.org/cracauer/