Skip to contentSkip to conversation

top/htop and all other CPU usage

REGRESSION In review [email protected] 4 participants 23 messages Nov 28, 2011 – Dec 5, 2011 Proc filesystem

lore
Share & tools
lore
#0 lore raw

Hi,

All this time i have been thinking i'm the only one - and i've been to loaded with work during working hours and tired when home =P

Anyways, I've neen seeing this since -rc1 on:
Intel(R) Core(TM) i5 CPU U520 @ 1.07GHz - x86-64 AMD Phenom(tm) II X6 1090T - x86-64
AMD Athlon(tm) 64 X2 Dual Core Processor 4400+ - x86-64 Intel(R) Core(TM)2 CPU T7200 @ 2.00GHz - i686

Configs available on demand - I've been running the same config for quite some time though.

#1 lore raw
Quoted 5 lines · On Mon 28-11-11 23:28:03, [email protected] wrote:

Hi,

All this time i have been thinking i'm the only one - and i've been to loaded with work during working hours and tired when home =P

Quoted 8 lines

Anyways, I've neen seeing this since -rc1 on:
Intel(R) Core(TM) i5 CPU U520 @ 1.07GHz - x86-64 AMD Phenom(tm) II X6 1090T - x86-64
AMD Athlon(tm) 64 X2 Dual Core Processor 4400+ - x86-64 Intel(R) Core(TM)2 CPU T7200 @ 2.00GHz - i686

Configs available on demand - I've been running the same config for quite some time though.

As I have written in other email could you post your config and collect the following data?
for i in seq 30;
do

        cat /proc/stat > `date +'%s'`
        sleep 1

done
export old_user=0 old_nice=0 old_sys=0 old_idle=0 old_iowait=0;

# for all your available CPUs
grep cpu0 * | while read cpu user nice sys idle iowait rest; 
do 
        echo $cpu $(($user-$old_user)) $(($nice-$old_nice)) $(($sys-$old_sys)) $(($idle-$old_idle)) $(($iowait-$old_iowait))
        old_user=$user old_nice=$nice old_sys=$sys old_idle=$idle old_iowait=$iowait
done

Thanks

Signature · 6 lines
Michal Hocko
SUSE Labs
SUSE LINUX s.r.o.
Lihovarska 1060/12
190 00 Praha 9    
Czech Republic
#2 lore raw
Quoted 9 lines · Michal Hocko

As I have written in other email could you post your config and collect the following data?
for i in seq 30;
do

        cat /proc/stat > `date +'%s'`
        sleep 1

done
export old_user=0 old_nice=0 old_sys=0 old_idle=0 old_iowait=0;

Quoted 6 lines
# for all your available CPUs
grep cpu0 * | while read cpu user nice sys idle iowait rest; 
do 
        echo $cpu $(($user-$old_user)) $(($nice-$old_nice)) $(($sys-$old_sys)) $(($idle-$old_idle)) $(($iowait-$old_iowait))
        old_user=$user old_nice=$nice old_sys=$sys old_idle=$idle old_iowait=$iowait
done

1322566208:cpu0 5199 0 2931 357890604 2541
1322566209:cpu0 0 0 1 0 0
1322566210:cpu0 0 0 0 0 0
1322566211:cpu0 0 0 0 0 0
1322566212:cpu0 0 0 0 0 0
1322566213:cpu0 0 0 0 0 0
1322566214:cpu0 1 0 0 0 0
1322566215:cpu0 2 0 0 0 0
1322566216:cpu0 3 0 0 0 0
1322566217:cpu0 2 0 0 0 0
1322566218:cpu0 4 0 0 0 0
1322566219:cpu0 1 0 0 0 0
1322566220:cpu0 2 0 0 0 0
1322566221:cpu0 2 0 1 0 0
1322566222:cpu0 1 0 0 0 0
1322566223:cpu0 2 0 0 0 0
1322566224:cpu0 1 0 1 0 0
1322566225:cpu0 1 0 0 0 0
1322566226:cpu0 2 0 0 0 0
1322566227:cpu0 1 0 1 0 0
1322566228:cpu0 2 0 0 0 0
1322566229:cpu0 2 0 0 0 0
1322566230:cpu0 6 0 3 0 0
1322566231:cpu0 1 0 0 0 0
1322566232:cpu0 2 0 0 0 0
1322566233:cpu0 3 0 0 0 0
1322566234:cpu0 2 0 0 0 0
1322566235:cpu0 2 0 2 0 0
1322566236:cpu0 0 0 1 0 0
1322566237:cpu0 1 0 0 0 0

$ grep . -r /sys/devices/system/cpu/cpuidle/
/sys/devices/system/cpu/cpuidle/current_driver:intel_idle
/sys/devices/system/cpu/cpuidle/current_governor_ro:menu
$ grep . -r /sys/devices/system/cpu/cpufreq/
/sys/devices/system/cpu/cpufreq/ondemand/sampling_rate_min:10000
/sys/devices/system/cpu/cpufreq/ondemand/sampling_rate:10000
/sys/devices/system/cpu/cpufreq/ondemand/up_threshold:95
/sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor:1
/sys/devices/system/cpu/cpufreq/ondemand/ignore_nice_load:0
/sys/devices/system/cpu/cpufreq/ondemand/powersave_bias:0
/sys/devices/system/cpu/cpufreq/ondemand/io_is_busy:1

One thing I have to note, it takes some time (from 30 seconds to 10 minutes) before this bug starts manifesting itself.

#3 lore raw
Quoted 40 lines · Artem S. Tashkinov
Quoted 9 lines · Michal Hocko

As I have written in other email could you post your config and collect the following data?
for i in seq 30;
do

        cat /proc/stat > `date +'%s'`
        sleep 1

done
export old_user=0 old_nice=0 old_sys=0 old_idle=0 old_iowait=0;

Quoted 6 lines
# for all your available CPUs
grep cpu0 * | while read cpu user nice sys idle iowait rest; 
do 
        echo $cpu $(($user-$old_user)) $(($nice-$old_nice)) $(($sys-$old_sys)) $(($idle-$old_idle)) $(($iowait-$old_iowait))
        old_user=$user old_nice=$nice old_sys=$sys old_idle=$idle old_iowait=$iowait
done

1322566208:cpu0 5199 0 2931 357890604 2541
1322566209:cpu0 0 0 1 0 0
1322566210:cpu0 0 0 0 0 0
1322566211:cpu0 0 0 0 0 0
1322566212:cpu0 0 0 0 0 0
1322566213:cpu0 0 0 0 0 0
1322566214:cpu0 1 0 0 0 0
1322566215:cpu0 2 0 0 0 0
1322566216:cpu0 3 0 0 0 0
1322566217:cpu0 2 0 0 0 0
1322566218:cpu0 4 0 0 0 0
1322566219:cpu0 1 0 0 0 0
1322566220:cpu0 2 0 0 0 0
1322566221:cpu0 2 0 1 0 0
1322566222:cpu0 1 0 0 0 0
1322566223:cpu0 2 0 0 0 0
1322566224:cpu0 1 0 1 0 0
1322566225:cpu0 1 0 0 0 0
1322566226:cpu0 2 0 0 0 0
1322566227:cpu0 1 0 1 0 0
1322566228:cpu0 2 0 0 0 0
1322566229:cpu0 2 0 0 0 0

Quoted 8 lines

1322566230:cpu0 6 0 3 0 0
1322566231:cpu0 1 0 0 0 0
1322566232:cpu0 2 0 0 0 0
1322566233:cpu0 3 0 0 0 0
1322566234:cpu0 2 0 0 0 0
1322566235:cpu0 2 0 2 0 0
1322566236:cpu0 0 0 1 0 0
1322566237:cpu0 1 0 0 0 0

Hmm, really strange. It looks that idle/iowait is not accounted at all. Which would explain why the numbers you are seeing are so weird.

Quoted 4 lines
$ grep . -r /sys/devices/system/cpu/cpuidle/
/sys/devices/system/cpu/cpuidle/current_driver:intel_idle
/sys/devices/system/cpu/cpuidle/current_governor_ro:menu

I will check whether I have a machine with intel_idle somewhere around.

Quoted 12 lines
$ grep . -r /sys/devices/system/cpu/cpufreq/
/sys/devices/system/cpu/cpufreq/ondemand/sampling_rate_min:10000
/sys/devices/system/cpu/cpufreq/ondemand/sampling_rate:10000
/sys/devices/system/cpu/cpufreq/ondemand/up_threshold:95
/sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor:1
/sys/devices/system/cpu/cpufreq/ondemand/ignore_nice_load:0
/sys/devices/system/cpu/cpufreq/ondemand/powersave_bias:0
/sys/devices/system/cpu/cpufreq/ondemand/io_is_busy:1

One thing I have to note, it takes some time (from 30 seconds to 10 minutes) before this bug starts manifesting itself.

Signature · 6 lines
Michal Hocko
SUSE Labs
SUSE LINUX s.r.o.
Lihovarska 1060/12
190 00 Praha 9    
Czech Republic
#4 lore raw
Quoted 1 line · Michal Hocko

On Tue 29-11-11 11:38:47, Artem S. Tashkinov wrote:

[...]

Quoted 3 lines
Quoted 3 lines
$ grep . -r /sys/devices/system/cpu/cpuidle/
/sys/devices/system/cpu/cpuidle/current_driver:intel_idle
/sys/devices/system/cpu/cpuidle/current_governor_ro:menu

Could you try with acpi_idle driver?
echo acpi_idle > /sys/devices/system/cpu/cpuidle/current_driver

Signature · 6 lines
Michal Hocko
SUSE Labs
SUSE LINUX s.r.o.
Lihovarska 1060/12
190 00 Praha 9    
Czech Republic
#5 lore raw
Quoted 11 lines · Michal Hocko
Quoted 1 line · Michal Hocko

On Tue 29-11-11 11:38:47, Artem S. Tashkinov wrote:

[...]

Quoted 3 lines
Quoted 3 lines
$ grep . -r /sys/devices/system/cpu/cpuidle/
/sys/devices/system/cpu/cpuidle/current_driver:intel_idle
/sys/devices/system/cpu/cpuidle/current_governor_ro:menu

Could you try with acpi_idle driver?
echo acpi_idle > /sys/devices/system/cpu/cpuidle/current_driver

[root@localhost ~]# echo acpi_idle > /sys/devices/system/cpu/cpuidle/current_driver -bash: /sys/devices/system/cpu/cpuidle/current_driver: Permission denied

#6 lore raw
Quoted 9 lines · Artem S. Tashkinov
Quoted 9 lines · Michal Hocko
Quoted 1 line · Michal Hocko

On Tue 29-11-11 11:38:47, Artem S. Tashkinov wrote:

[...]

Quoted 3 lines
Quoted 3 lines
$ grep . -r /sys/devices/system/cpu/cpuidle/
/sys/devices/system/cpu/cpuidle/current_driver:intel_idle
/sys/devices/system/cpu/cpuidle/current_governor_ro:menu
Quoted 5 lines
Quoted 2 lines

Could you try with acpi_idle driver?
echo acpi_idle > /sys/devices/system/cpu/cpuidle/current_driver

[root@localhost ~]# echo acpi_idle > /sys/devices/system/cpu/cpuidle/current_driver -bash: /sys/devices/system/cpu/cpuidle/current_driver: Permission denied

It seems that this is cannot be set in runtime:
SYSDEV_CLASS_ATTR(current_driver, 0444, show_current_driver, NULL);

Or maybe you need to boot with cpuidle_sysfs_switch. According to the documentation you might be able to change the governor. I have no idea whether this can help somehow but let's try that.

I haven't found any intel_idle machine in my lab so far and all other acpi_idle machines seem to work (or at least randomly selected ones) so this smells like a major difference in the setup.

Signature · 6 lines
Michal Hocko
SUSE Labs
SUSE LINUX s.r.o.
Lihovarska 1060/12
190 00 Praha 9    
Czech Republic
#7 lore raw
Quoted 10 lines · Michal Hocko

It seems that this is cannot be set in runtime:
SYSDEV_CLASS_ATTR(current_driver, 0444, show_current_driver, NULL);

Or maybe you need to boot with cpuidle_sysfs_switch. According to the documentation you might be able to change the governor. I have no idea whether this can help somehow but let's try that.

I haven't found any intel_idle machine in my lab so far and all other acpi_idle machines seem to work (or at least randomly selected ones) so this smells like a major difference in the setup.

# echo acpi_idle > /sys/devices/system/cpu/cpuidle/current_driver
bash: /sys/devices/system/cpu/cpuidle/current_driver: Permission denied

BTW

# cat  /sys/devices/system/cpu/cpuidle/available_governors
ladder menu

dmesg | grep -i idle

[    0.000000] Kernel command line: root=/dev/sda1 ro cpuidle_sysfs_switch
[    0.000126] using mwait in idle threads.
[    1.083872] intel_idle: MWAIT substates: 0x1120
[    1.083873] intel_idle: v0.4 model 0x2A
[    1.083874] intel_idle: lapic_timer_reliable_states 0xffffffff
[    1.159043] cpuidle: using governor ladder
[    1.168842] cpuidle: using governor menu
#8 lore raw
In reply to Michal Hocko
Quoted 19 lines · Michal Hocko
Quoted 9 lines · Artem S. Tashkinov
Quoted 9 lines · Michal Hocko
Quoted 1 line · Michal Hocko

On Tue 29-11-11 11:38:47, Artem S. Tashkinov wrote:

[...]

Quoted 3 lines
Quoted 3 lines
$ grep . -r /sys/devices/system/cpu/cpuidle/
/sys/devices/system/cpu/cpuidle/current_driver:intel_idle
/sys/devices/system/cpu/cpuidle/current_governor_ro:menu
Quoted 5 lines
Quoted 2 lines

Could you try with acpi_idle driver?
echo acpi_idle > /sys/devices/system/cpu/cpuidle/current_driver

[root@localhost ~]# echo acpi_idle > /sys/devices/system/cpu/cpuidle/current_driver -bash: /sys/devices/system/cpu/cpuidle/current_driver: Permission denied

It seems that this is cannot be set in runtime:
SYSDEV_CLASS_ATTR(current_driver, 0444, show_current_driver, NULL);

Quoted 7 lines

Or maybe you need to boot with cpuidle_sysfs_switch. According to the documentation you might be able to change the governor. I have no idea whether this can help somehow but let's try that.

I haven't found any intel_idle machine in my lab so far and all other acpi_idle machines seem to work (or at least randomly selected ones) so this smells like a major difference in the setup.

I'm able to reproduce that with acpi_driver on one box, but not on demand.

Thanks,
Rafael

#9 lore raw
Quoted 24 lines · Rafael J. Wysocki
Quoted 23 lines · Michal Hocko
Quoted 9 lines · Artem S. Tashkinov
Quoted 9 lines · Michal Hocko
Quoted 1 line · Michal Hocko

On Tue 29-11-11 11:38:47, Artem S. Tashkinov wrote:

[...]

Quoted 3 lines
Quoted 3 lines
$ grep . -r /sys/devices/system/cpu/cpuidle/
/sys/devices/system/cpu/cpuidle/current_driver:intel_idle
/sys/devices/system/cpu/cpuidle/current_governor_ro:menu
Quoted 5 lines
Quoted 2 lines

Could you try with acpi_idle driver?
echo acpi_idle > /sys/devices/system/cpu/cpuidle/current_driver

[root@localhost ~]# echo acpi_idle > /sys/devices/system/cpu/cpuidle/current_driver -bash: /sys/devices/system/cpu/cpuidle/current_driver: Permission denied

It seems that this is cannot be set in runtime:
SYSDEV_CLASS_ATTR(current_driver, 0444, show_current_driver, NULL);

Or maybe you need to boot with cpuidle_sysfs_switch. According to the documentation you might be able to change the governor. I have no idea whether this can help somehow but let's try that.

Quoted 5 lines
Quoted 3 lines

I haven't found any intel_idle machine in my lab so far and all other acpi_idle machines seem to work (or at least randomly selected ones) so this smells like a major difference in the setup.

I'm able to reproduce that with acpi_driver on one box, but not on demand.

And do you see the same thing (no idle/io_wait) updates?

Quoted 3 lines

Thanks,
Rafael

Signature · 6 lines
Michal Hocko
SUSE Labs
SUSE LINUX s.r.o.
Lihovarska 1060/12
190 00 Praha 9    
Czech Republic
#10 lore raw
Quoted 25 lines · Michal Hocko
Quoted 24 lines · Rafael J. Wysocki
Quoted 23 lines · Michal Hocko
Quoted 9 lines · Artem S. Tashkinov
Quoted 9 lines · Michal Hocko
Quoted 1 line · Michal Hocko

On Tue 29-11-11 11:38:47, Artem S. Tashkinov wrote:

[...]

Quoted 3 lines
Quoted 3 lines
$ grep . -r /sys/devices/system/cpu/cpuidle/
/sys/devices/system/cpu/cpuidle/current_driver:intel_idle
/sys/devices/system/cpu/cpuidle/current_governor_ro:menu
Quoted 5 lines
Quoted 2 lines

Could you try with acpi_idle driver?
echo acpi_idle > /sys/devices/system/cpu/cpuidle/current_driver

[root@localhost ~]# echo acpi_idle > /sys/devices/system/cpu/cpuidle/current_driver -bash: /sys/devices/system/cpu/cpuidle/current_driver: Permission denied

It seems that this is cannot be set in runtime:
SYSDEV_CLASS_ATTR(current_driver, 0444, show_current_driver, NULL);

Or maybe you need to boot with cpuidle_sysfs_switch. According to the documentation you might be able to change the governor. I have no idea whether this can help somehow but let's try that.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

I haven't found any intel_idle machine in my lab so far and all other acpi_idle machines seem to work (or at least randomly selected ones) so this smells like a major difference in the setup.

I'm able to reproduce that with acpi_driver on one box, but not on demand.

And do you see the same thing (no idle/io_wait) updates?

Actaully, I was wrong. The box I'm seeing the issue on also has "none" in /sys/devices/system/cpu/cpuidle/current_driver. Sorry for the confusion.

Thanks,
Rafael

#11 lore raw

[Let's add Len to the CC for idle driver]

Quoted 3 lines · Rafael J. Wysocki
Quoted 2 lines · Michal Hocko
Quoted 1 line · Rafael J. Wysocki

On Tuesday, November 29, 2011, Michal Hocko wrote:

[...]

Quoted 10 lines
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

I haven't found any intel_idle machine in my lab so far and all other acpi_idle machines seem to work (or at least randomly selected ones) so this smells like a major difference in the setup.

I'm able to reproduce that with acpi_driver on one box, but not on demand.

And do you see the same thing (no idle/io_wait) updates?

Actaully, I was wrong. The box I'm seeing the issue on also has "none" in /sys/devices/system/cpu/cpuidle/current_driver. Sorry for the confusion.

OK. So we have seen the issue only with intel_idle and none drivers so far. acpi_idle which is at my machines works just fine.
I think we should focus on those drivers.

To summarize this issue.
Users are seeing weird values reported by [h]top. CPUs seem to be at 100% even though there is nothing hogging them. /proc/stat collected data on the affected system shown that idle/io_wait are not accounted properly.
It has been identified that problem disappears if a25cac51 [proc:
Consider NO_HZ when printing idle and iowait times] is reverted. This patch fixes a bug when idle/io_wait times are not repororted properly when a CPU is tickless. It relies on get_cpu_idle_time_us which either reports sched_time idle_sleeptime or (idle_sleeptime + now-idle_entrytime) if we are idle at the moment.

While implementation is not race free (we better not use locks in that path...) so we might race:

E.g.

CPU1 CPU2
now = ktime_get

			tick_nohz_start_idle
				ts->idle_entrytime = now;
if (ts->idle_active)
				ts->idle_active = 1

[...]
return idle_sleeptime

But this is OK because sleeptime will be more or less accurate. We just skip few ticks.

It would be worse if we had a race like:
CPU1 CPU2
now = ktime_get

			tick_nohz_start_idle
				now = ktime_get
				update_ts_time_stats()
				ts->idle_entrytime = now;
				ts->idle_active = 1

if (ts->idle_active)
delta = ktime_sub(now, idle_entrytime)
ktime_add(idle_sleeptime, delta)

In this case we might get an overflow from ktime_sub but AFAIU the ktime_* magic the overflow should cause to get smaller idle_sleeptime in the end after ktime_add (we do not add a small number but rather subtract it), right?
So it shouldn't be a big deal as well.

So the question is. What is the role of the idle driver here?

Thanks

Signature · 6 lines
Michal Hocko
SUSE Labs
SUSE LINUX s.r.o.
Lihovarska 1060/12
190 00 Praha 9    
Czech Republic
#12 lore raw

On Thu 01-12-11 15:07:49, Michal Hocko wrote:
[...]

Quoted 25 lines

While implementation is not race free (we better not use locks in that path...) so we might race:

E.g.

CPU1 CPU2
now = ktime_get

			tick_nohz_start_idle
				ts->idle_entrytime = now;
if (ts->idle_active)
				ts->idle_active = 1

[...]
return idle_sleeptime

But this is OK because sleeptime will be more or less accurate. We just skip few ticks.

It would be worse if we had a race like:
CPU1 CPU2
now = ktime_get

			tick_nohz_start_idle
				now = ktime_get
				update_ts_time_stats()
				ts->idle_entrytime = now;
				ts->idle_active = 1
Quoted 8 lines

if (ts->idle_active)
delta = ktime_sub(now, idle_entrytime)
ktime_add(idle_sleeptime, delta)

In this case we might get an overflow from ktime_sub but AFAIU the ktime_* magic the overflow should cause to get smaller idle_sleeptime in the end after ktime_add (we do not add a small number but rather subtract it), right?

Scratch that. Dunno why but I thought that ktime_t has unsigned values but it is apparently not true (tv64 is s64). Anyway the above races should be safe.

Signature · 6 lines
Michal Hocko
SUSE Labs
SUSE LINUX s.r.o.
Lihovarska 1060/12
190 00 Praha 9    
Czech Republic
#13 lore raw
In reply to Artem S. Tashkinov
Quoted 40 lines · Artem S. Tashkinov
Quoted 9 lines · Michal Hocko

As I have written in other email could you post your config and collect the following data?
for i in seq 30;
do

        cat /proc/stat > `date +'%s'`
        sleep 1

done
export old_user=0 old_nice=0 old_sys=0 old_idle=0 old_iowait=0;

Quoted 6 lines
# for all your available CPUs
grep cpu0 * | while read cpu user nice sys idle iowait rest; 
do 
        echo $cpu $(($user-$old_user)) $(($nice-$old_nice)) $(($sys-$old_sys)) $(($idle-$old_idle)) $(($iowait-$old_iowait))
        old_user=$user old_nice=$nice old_sys=$sys old_idle=$idle old_iowait=$iowait
done

1322566208:cpu0 5199 0 2931 357890604 2541
1322566209:cpu0 0 0 1 0 0
1322566210:cpu0 0 0 0 0 0
1322566211:cpu0 0 0 0 0 0
1322566212:cpu0 0 0 0 0 0
1322566213:cpu0 0 0 0 0 0
1322566214:cpu0 1 0 0 0 0
1322566215:cpu0 2 0 0 0 0
1322566216:cpu0 3 0 0 0 0
1322566217:cpu0 2 0 0 0 0
1322566218:cpu0 4 0 0 0 0
1322566219:cpu0 1 0 0 0 0
1322566220:cpu0 2 0 0 0 0
1322566221:cpu0 2 0 1 0 0
1322566222:cpu0 1 0 0 0 0
1322566223:cpu0 2 0 0 0 0
1322566224:cpu0 1 0 1 0 0
1322566225:cpu0 1 0 0 0 0
1322566226:cpu0 2 0 0 0 0
1322566227:cpu0 1 0 1 0 0
1322566228:cpu0 2 0 0 0 0
1322566229:cpu0 2 0 0 0 0

Quoted 8 lines

1322566230:cpu0 6 0 3 0 0
1322566231:cpu0 1 0 0 0 0
1322566232:cpu0 2 0 0 0 0
1322566233:cpu0 3 0 0 0 0
1322566234:cpu0 2 0 0 0 0
1322566235:cpu0 2 0 2 0 0
1322566236:cpu0 0 0 1 0 0
1322566237:cpu0 1 0 0 0 0

Could you post raw data as well? Ideally starting right after boot and collected for more than 30s (longer better...)

Thanks!

Signature · 6 lines
Michal Hocko
SUSE Labs
SUSE LINUX s.r.o.
Lihovarska 1060/12
190 00 Praha 9    
Czech Republic
#14 lore raw

PATCH 1/1 proc: Do not overflow get_{idle,iowait}_time for nohz (was: Re: Re: [REGRESSION] [Linux 3.2] top/htop and all other CPU usage)

Quoted 23 lines · Michal Hocko
Quoted 22 lines · Artem S. Tashkinov
Quoted 9 lines · Michal Hocko

As I have written in other email could you post your config and collect the following data?
for i in seq 30;
do

        cat /proc/stat > `date +'%s'`
        sleep 1

done
export old_user=0 old_nice=0 old_sys=0 old_idle=0 old_iowait=0;

Quoted 6 lines
# for all your available CPUs
grep cpu0 * | while read cpu user nice sys idle iowait rest; 
do 
        echo $cpu $(($user-$old_user)) $(($nice-$old_nice)) $(($sys-$old_sys)) $(($idle-$old_idle)) $(($iowait-$old_iowait))
        old_user=$user old_nice=$nice old_sys=$sys old_idle=$idle old_iowait=$iowait
done

1322566208:cpu0 5199 0 2931 357890604 2541
1322566209:cpu0 0 0 1 0 0
1322566210:cpu0 0 0 0 0 0
1322566211:cpu0 0 0 0 0 0

[...]

Quoted 3 lines

Could you post raw data as well? Ideally starting right after boot and collected for more than 30s (longer better...)

Ahh, missed that you attached data. And also noticed that you are using CONFIG_HZ_300 which explains the problem and why I do cannot reproduce it.

get_{idle,iowait}_time translates us to cputime64_t and it uses usecs_to_cputime which is just an alias for usecs_to_jiffies and it does

	if (u > jiffies_to_usecs(MAX_JIFFY_OFFSET))
                return MAX_JIFFY_OFFSET;

which in your case (HZ=300) means that we overflow much more often than for HZ==100. The patch below should fix this:
---

Quoted 1 line

From 23882e2aabe27934df4d23b0ed52749fd4f61ab4 Mon Sep 17 00:00:00 2001

From: Michal Hocko <[email protected]>
Date: Fri, 2 Dec 2011 16:17:03 +0100
Subject: [PATCH] proc: Do not overflow get_{idle,iowait}_time for nohz

get_{idle,iowait}_time use usecs_to_cputime to translate micro seconds time to cputime64_t. This is just an alias to usecs_to_jiffies which reduces the data type from u64 to unsigned int and also checks whether the given paramerer overflows jiffies_to_usecs(MAX_JIFFY_OFFSET) and returns MAX_JIFFY_OFFSET in that case. How much we overflow depends on CONFIG_HZ and especially for CONFIG_HZ_300 it is quite low (1431649781).

This results in a bug when people saw [h]top going mad reporting 100% CPU usage even though there was basically no CPU load at all. The reason was simply that /proc/stat stopped reporting idle/io_wait changes (and reported MAX_JIFFY_OFFSET) and so the only change happenning was for user system time.

Let's use nsecs_to_jiffies64 instead as it doesn't overflow.

---
 fs/proc/stat.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)
fs/proc/stat.c +2 -2
index 42b274d..2a30d67 100644
@@ -32,7 +32,7 @@ static cputime64_t get_idle_time(int cpu)
 		idle = kstat_cpu(cpu).cpustat.idle;
 		idle = cputime64_add(idle, arch_idle_time(cpu));
 	} else
-		idle = usecs_to_cputime(idle_time);
+		idle = nsecs_to_jiffies64(1000 * idle_time);
 
 	return idle;
 }
@@ -46,7 +46,7 @@ static cputime64_t get_iowait_time(int cpu)
 		/* !NO_HZ so we can rely on cpustat.iowait */
 		iowait = kstat_cpu(cpu).cpustat.iowait;
 	else
-		iowait = usecs_to_cputime(iowait_time);
+		iowait = nsecs_to_jiffies64(1000 * iowait_time);
 
 	return iowait;
 }
Signature · 10 lines
1.7.7.3


-- 
Michal Hocko
SUSE Labs
SUSE LINUX s.r.o.
Lihovarska 1060/12
190 00 Praha 9    
Czech Republic
#15 lore raw

Re: [PATCH] proc: Do not overflow get_{idle,iowait}_time for nohz (was: Re: Re: [REGRESSION] [Linux 3.2] top/htop and all other CPU usage)

Quoted 33 lines · Michal Hocko
Quoted 23 lines · Michal Hocko
Quoted 22 lines · Artem S. Tashkinov
Quoted 9 lines · Michal Hocko

As I have written in other email could you post your config and collect the following data?
for i in seq 30;
do

        cat /proc/stat > `date +'%s'`
        sleep 1

done
export old_user=0 old_nice=0 old_sys=0 old_idle=0 old_iowait=0;

Quoted 6 lines
# for all your available CPUs
grep cpu0 * | while read cpu user nice sys idle iowait rest; 
do 
        echo $cpu $(($user-$old_user)) $(($nice-$old_nice)) $(($sys-$old_sys)) $(($idle-$old_idle)) $(($iowait-$old_iowait))
        old_user=$user old_nice=$nice old_sys=$sys old_idle=$idle old_iowait=$iowait
done

1322566208:cpu0 5199 0 2931 357890604 2541
1322566209:cpu0 0 0 1 0 0
1322566210:cpu0 0 0 0 0 0
1322566211:cpu0 0 0 0 0 0

[...]

Quoted 3 lines

Could you post raw data as well? Ideally starting right after boot and collected for more than 30s (longer better...)

Ahh, missed that you attached data. And also noticed that you are using CONFIG_HZ_300 which explains the problem and why I do cannot reproduce it.

Quoted 6 lines

get_{idle,iowait}_time translates us to cputime64_t and it uses usecs_to_cputime which is just an alias for usecs_to_jiffies and it does

	if (u > jiffies_to_usecs(MAX_JIFFY_OFFSET))
                return MAX_JIFFY_OFFSET;

which in your case (HZ=300) means that we overflow much more often than for HZ==100. The patch below should fix this:

And the one with a more cleaned up changelog. No functional changes
---

Quoted 1 line

From 107887016b91de59194a93c751d040b05d5e37fe Mon Sep 17 00:00:00 2001

From: Michal Hocko <[email protected]>
Date: Fri, 2 Dec 2011 16:17:03 +0100
Subject: [PATCH] proc: Do not overflow get_{idle,iowait}_time for nohz

Since a25cac51 [proc: Consider NO_HZ when printing idle and iowait times] we are reporting idle/io_wait time also while a CPU is tickless. We rely on get_{idle,iowait}_time functions to retrieve proper data.

These functions, however, use usecs_to_cputime to translate micro seconds time to cputime64_t. This is just an alias to usecs_to_jiffies which reduces the data type from u64 to unsigned int and also checks whether the given parameter overflows jiffies_to_usecs(MAX_JIFFY_OFFSET) and returns MAX_JIFFY_OFFSET in that case.

When do we overflow depends on CONFIG_HZ but especially for CONFIG_HZ_300 it is quite low (1431649781) so we are getting MAX_JIFFY_OFFSET for >3000s! until we overflow unsigned int. Just for reference CONFIG_100 has an overflow window around 20s, CONFIG_250 ~8s and CONFIG_1000 ~2s.

This results in a bug when people saw [h]top going mad reporting 100% CPU usage even though there was basically no CPU load. The reason was simply that /proc/stat stopped reporting idle/io_wait changes (and reported MAX_JIFFY_OFFSET) and so the only change happening was for user system time.

Let's use nsecs_to_jiffies64 instead which doesn't reduce the precision to 32b type and it is much more appropriate for cumulative time values (unlike usecs_to_jiffies which intended for timeout calculations).

---
 fs/proc/stat.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)
fs/proc/stat.c +2 -2
index 42b274d..2a30d67 100644
@@ -32,7 +32,7 @@ static cputime64_t get_idle_time(int cpu)
 		idle = kstat_cpu(cpu).cpustat.idle;
 		idle = cputime64_add(idle, arch_idle_time(cpu));
 	} else
-		idle = usecs_to_cputime(idle_time);
+		idle = nsecs_to_jiffies64(1000 * idle_time);
 
 	return idle;
 }
@@ -46,7 +46,7 @@ static cputime64_t get_iowait_time(int cpu)
 		/* !NO_HZ so we can rely on cpustat.iowait */
 		iowait = kstat_cpu(cpu).cpustat.iowait;
 	else
-		iowait = usecs_to_cputime(iowait_time);
+		iowait = nsecs_to_jiffies64(1000 * iowait_time);
 
 	return iowait;
 }
Signature · 10 lines
1.7.7.3


-- 
Michal Hocko
SUSE Labs
SUSE LINUX s.r.o.
Lihovarska 1060/12
190 00 Praha 9    
Czech Republic
#16 lore raw

Re: Re: [PATCH] proc: Do not overflow get_{idle,iowait}_time for nohz (was: Re: Re: [REGRESSION] [Linux 3.2] top/htop and all other CPU usage)

Quoted 62 lines · Michal Hocko

And the one with a more cleaned up changelog. No functional changes
---
From 107887016b91de59194a93c751d040b05d5e37fe Mon Sep 17 00:00:00 2001
From: Michal Hocko <>
Date: Fri, 2 Dec 2011 16:17:03 +0100
Subject: [PATCH] proc: Do not overflow get_{idle,iowait}_time for nohz

Since a25cac51 [proc: Consider NO_HZ when printing idle and iowait times] we are reporting idle/io_wait time also while a CPU is tickless. We rely on get_{idle,iowait}_time functions to retrieve proper data.

These functions, however, use usecs_to_cputime to translate micro seconds time to cputime64_t. This is just an alias to usecs_to_jiffies which reduces the data type from u64 to unsigned int and also checks whether the given parameter overflows jiffies_to_usecs(MAX_JIFFY_OFFSET) and returns MAX_JIFFY_OFFSET in that case.

When do we overflow depends on CONFIG_HZ but especially for CONFIG_HZ_300 it is quite low (1431649781) so we are getting MAX_JIFFY_OFFSET for >3000s! until we overflow unsigned int. Just for reference CONFIG_100 has an overflow window around 20s, CONFIG_250 ~8s and CONFIG_1000 ~2s.

This results in a bug when people saw [h]top going mad reporting 100% CPU usage even though there was basically no CPU load. The reason was simply that /proc/stat stopped reporting idle/io_wait changes (and reported MAX_JIFFY_OFFSET) and so the only change happening was for user system time.

Let's use nsecs_to_jiffies64 instead which doesn't reduce the precision to 32b type and it is much more appropriate for cumulative time values (unlike usecs_to_jiffies which intended for timeout calculations).

---
 fs/proc/stat.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)
fs/proc/stat.c +2 -2
index 42b274d..2a30d67 100644
@@ -32,7 +32,7 @@ static cputime64_t get_idle_time(int cpu)
 		idle = kstat_cpu(cpu).cpustat.idle;
 		idle = cputime64_add(idle, arch_idle_time(cpu));
 	} else
-		idle = usecs_to_cputime(idle_time);
+		idle = nsecs_to_jiffies64(1000 * idle_time);
 
 	return idle;
 }
@@ -46,7 +46,7 @@ static cputime64_t get_iowait_time(int cpu)
 		/* !NO_HZ so we can rely on cpustat.iowait */
 		iowait = kstat_cpu(cpu).cpustat.iowait;
 	else
-		iowait = usecs_to_cputime(iowait_time);
+		iowait = nsecs_to_jiffies64(1000 * iowait_time);
 
 	return iowait;
 }
Signature · 1 line
1.7.7.3

Thank you, this patch has fixed the issue for me.

#17 lore raw

Re: Re: [PATCH] proc: Do not overflow get_{idle,iowait}_time for nohz (was: Re: Re: [REGRESSION] [Linux 3.2] top/htop and all other CPU usage)

Quoted 60 lines · Artem S. Tashkinov
Quoted 58 lines · Michal Hocko

And the one with a more cleaned up changelog. No functional changes
---
From 107887016b91de59194a93c751d040b05d5e37fe Mon Sep 17 00:00:00 2001
From: Michal Hocko <>
Date: Fri, 2 Dec 2011 16:17:03 +0100
Subject: [PATCH] proc: Do not overflow get_{idle,iowait}_time for nohz

Since a25cac51 [proc: Consider NO_HZ when printing idle and iowait times] we are reporting idle/io_wait time also while a CPU is tickless. We rely on get_{idle,iowait}_time functions to retrieve proper data.

These functions, however, use usecs_to_cputime to translate micro seconds time to cputime64_t. This is just an alias to usecs_to_jiffies which reduces the data type from u64 to unsigned int and also checks whether the given parameter overflows jiffies_to_usecs(MAX_JIFFY_OFFSET) and returns MAX_JIFFY_OFFSET in that case.

When do we overflow depends on CONFIG_HZ but especially for CONFIG_HZ_300 it is quite low (1431649781) so we are getting MAX_JIFFY_OFFSET for >3000s! until we overflow unsigned int. Just for reference CONFIG_100 has an overflow window around 20s, CONFIG_250 ~8s and CONFIG_1000 ~2s.

This results in a bug when people saw [h]top going mad reporting 100% CPU usage even though there was basically no CPU load. The reason was simply that /proc/stat stopped reporting idle/io_wait changes (and reported MAX_JIFFY_OFFSET) and so the only change happening was for user system time.

Let's use nsecs_to_jiffies64 instead which doesn't reduce the precision to 32b type and it is much more appropriate for cumulative time values (unlike usecs_to_jiffies which intended for timeout calculations).

---
 fs/proc/stat.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)
fs/proc/stat.c +2 -2
index 42b274d..2a30d67 100644
@@ -32,7 +32,7 @@ static cputime64_t get_idle_time(int cpu)
 		idle = kstat_cpu(cpu).cpustat.idle;
 		idle = cputime64_add(idle, arch_idle_time(cpu));
 	} else
-		idle = usecs_to_cputime(idle_time);
+		idle = nsecs_to_jiffies64(1000 * idle_time);
 
 	return idle;
 }
@@ -46,7 +46,7 @@ static cputime64_t get_iowait_time(int cpu)
 		/* !NO_HZ so we can rely on cpustat.iowait */
 		iowait = kstat_cpu(cpu).cpustat.iowait;
 	else
-		iowait = usecs_to_cputime(iowait_time);
+		iowait = nsecs_to_jiffies64(1000 * iowait_time);
 
Quoted 8 lines
Quoted 4 lines
fs/proc/stat.c +0 -0
 	return iowait;
 }
Signature · 1 line
1.7.7.3

Thank you, this patch has fixed the issue for me.

Thanks for retesting!

Signature · 6 lines
Michal Hocko
SUSE Labs
SUSE LINUX s.r.o.
Lihovarska 1060/12
190 00 Praha 9    
Czech Republic
#19 lore raw
In reply to Michal Hocko
Quoted 24 lines · Michal Hocko
Quoted 5 lines · On Mon 28-11-11 23:28:03, [email protected] wrote:

Hi,

All this time i have been thinking i'm the only one - and i've been to loaded with work during working hours and tired when home =P

Quoted 8 lines

Anyways, I've neen seeing this since -rc1 on:
Intel(R) Core(TM) i5 CPU U520 @ 1.07GHz - x86-64 AMD Phenom(tm) II X6 1090T - x86-64
AMD Athlon(tm) 64 X2 Dual Core Processor 4400+ - x86-64 Intel(R) Core(TM)2 CPU T7200 @ 2.00GHz - i686

Configs available on demand - I've been running the same config for quite some time though.

As I have written in other email could you post your config and collect the following data?
for i in seq 30;
do

        cat /proc/stat > `date +'%s'`
        sleep 1

done
export old_user=0 old_nice=0 old_sys=0 old_idle=0 old_iowait=0;

Quoted 6 lines
# for all your available CPUs
grep cpu0 * | while read cpu user nice sys idle iowait rest; 
do 
        echo $cpu $(($user-$old_user)) $(($nice-$old_nice)) $(($sys-$old_sys)) $(($idle-$old_idle)) $(($iowait-$old_iowait))
        old_user=$user old_nice=$nice old_sys=$sys old_idle=$idle old_iowait=$iowait
done

1322587011:cpu0 5838553 164 1252921 1844674407370 78234 1322587012:cpu0 4 0 1 0 0
1322587013:cpu0 4 0 2 0 0
1322587014:cpu0 2 0 1 0 0
1322587015:cpu0 3 0 2 0 0
1322587016:cpu0 7 0 2 0 4
1322587017:cpu0 2 0 2 0 0
1322587018:cpu0 1 0 1 0 0
1322587019:cpu0 4 0 2 0 1
1322587020:cpu0 5 0 3 0 5
1322587021:cpu0 7 0 1 0 3
1322587022:cpu0 4 0 2 0 0
1322587023:cpu0 4 0 2 0 0
1322587024:cpu0 3 0 3 0 0
1322587025:cpu0 3 0 1 0 0
1322587026:cpu0 4 0 2 0 4
1322587027:cpu0 2 0 2 0 0
1322587028:cpu0 2 0 1 0 0
1322587029:cpu0 4 0 1 0 0
1322587030:cpu0 5 0 1 0 0
1322587032:cpu0 5 0 1 0 4
1322587033:cpu0 6 0 1 0 0
1322587034:cpu0 1 0 2 0 0
1322587035:cpu0 3 0 2 0 0
1322587036:cpu0 3 0 1 0 0
1322587037:cpu0 4 0 2 0 2
1322587038:cpu0 4 0 1 0 0
1322587039:cpu0 2 0 1 0 0
1322587040:cpu0 3 0 1 0 0
1322587041:cpu0 4 0 3 0 0

1322587011:cpu1 5944677 172 1287816 1844674407370 871 1322587012:cpu1 4 0 2 0 0
1322587013:cpu1 2 0 2 0 0
1322587014:cpu1 3 0 0 0 0
1322587015:cpu1 3 0 1 0 9
1322587016:cpu1 22 0 3 0 18
1322587017:cpu1 10 0 3 0 13
1322587018:cpu1 3 0 1 0 0
1322587019:cpu1 4 0 3 0 14
1322587020:cpu1 14 0 2 0 7
1322587021:cpu1 16 0 1 0 2
1322587022:cpu1 5 0 0 0 0
1322587023:cpu1 7 0 1 0 0
1322587024:cpu1 8 0 1 0 0
1322587025:cpu1 2 0 2 0 0
1322587026:cpu1 4 0 2 0 5
1322587027:cpu1 7 0 2 0 0
1322587028:cpu1 5 0 0 0 0
1322587029:cpu1 4 0 2 0 0
1322587030:cpu1 6 0 0 0 0
1322587032:cpu1 3 0 2 0 0
1322587033:cpu1 6 0 1 0 0
1322587034:cpu1 2 0 1 0 0
1322587035:cpu1 4 0 2 0 0
1322587036:cpu1 3 0 1 0 0
1322587037:cpu1 4 0 0 0 0
1322587038:cpu1 8 0 1 0 0
1322587039:cpu1 4 0 2 0 0
1322587040:cpu1 4 0 0 0 0
1322587041:cpu1 8 0 1 0 0

For me the cpu usage is sporadic and only shows up as user or kernel, nothing else.

Signature · 1 line
Ian Kumlien  -- http://demius.net || http://pomac.netswarm.net
#20 lore raw
In reply to Michal Hocko
Quoted 24 lines · Michal Hocko
Quoted 5 lines · On Mon 28-11-11 23:28:03, [email protected] wrote:

Hi,

All this time i have been thinking i'm the only one - and i've been to loaded with work during working hours and tired when home =P

Quoted 8 lines

Anyways, I've neen seeing this since -rc1 on:
Intel(R) Core(TM) i5 CPU U520 @ 1.07GHz - x86-64 AMD Phenom(tm) II X6 1090T - x86-64
AMD Athlon(tm) 64 X2 Dual Core Processor 4400+ - x86-64 Intel(R) Core(TM)2 CPU T7200 @ 2.00GHz - i686

Configs available on demand - I've been running the same config for quite some time though.

As I have written in other email could you post your config and collect the following data?
for i in seq 30;
do

        cat /proc/stat > `date +'%s'`
        sleep 1

done
export old_user=0 old_nice=0 old_sys=0 old_idle=0 old_iowait=0;

Quoted 6 lines
# for all your available CPUs
grep cpu0 * | while read cpu user nice sys idle iowait rest; 
do 
        echo $cpu $(($user-$old_user)) $(($nice-$old_nice)) $(($sys-$old_sys)) $(($idle-$old_idle)) $(($iowait-$old_iowait))
        old_user=$user old_nice=$nice old_sys=$sys old_idle=$idle old_iowait=$iowait
done

Sorry, the previous one was AMD X2 4400+ and now the T7200:

1322587696:cpu0 1410110 148 368843 357890604 357890604 1322587697:cpu0 0 0 1 0 0
1322587698:cpu0 1 0 1 0 0
1322587699:cpu0 2 0 1 0 0
1322587700:cpu0 0 0 1 0 0
1322587701:cpu0 0 0 1 0 0
1322587702:cpu0 0 0 1 0 0
1322587703:cpu0 1 0 0 0 0
1322587704:cpu0 0 0 0 0 0
1322587705:cpu0 0 0 1 0 0
1322587706:cpu0 1 0 1 0 0
1322587707:cpu0 0 0 1 0 0
1322587708:cpu0 1 0 0 0 0
1322587709:cpu0 1 0 0 0 0
1322587711:cpu0 0 0 2 0 0
1322587712:cpu0 0 0 0 0 0
1322587713:cpu0 1 0 1 0 0
1322587714:cpu0 1 0 0 0 0
1322587715:cpu0 1 0 1 0 0
1322587716:cpu0 1 0 0 0 0
1322587717:cpu0 0 0 1 0 0
1322587718:cpu0 1 0 1 0 0
1322587719:cpu0 0 0 1 0 0
1322587720:cpu0 0 0 0 0 0
1322587721:cpu0 2 0 1 0 0
1322587722:cpu0 1 0 0 0 0
1322587723:cpu0 0 0 3 0 0
1322587724:cpu0 1 0 0 0 0
1322587725:cpu0 0 0 0 0 0
1322587726:cpu0 0 0 1 0 0

1322587696:cpu1 1395249 509 284978 357890604 95640 1322587697:cpu1 0 0 0 0 0
1322587698:cpu1 0 0 0 0 0
1322587699:cpu1 0 0 0 0 0
1322587700:cpu1 1 0 0 0 0
1322587701:cpu1 1 0 0 0 0
1322587702:cpu1 0 0 0 0 0
1322587703:cpu1 0 0 1 0 0
1322587704:cpu1 1 0 0 0 0
1322587705:cpu1 0 0 1 0 0
1322587706:cpu1 1 0 0 0 0
1322587707:cpu1 0 0 0 0 0
1322587708:cpu1 1 0 0 0 0
1322587709:cpu1 1 0 1 0 0
1322587711:cpu1 0 0 0 0 0
1322587712:cpu1 0 0 0 0 0
1322587713:cpu1 0 0 0 0 0
1322587714:cpu1 0 0 0 0 0
1322587715:cpu1 0 0 0 0 0
1322587716:cpu1 0 0 1 0 0
1322587717:cpu1 0 0 0 0 0
1322587718:cpu1 0 0 0 0 0
1322587719:cpu1 0 0 1 0 0
1322587720:cpu1 1 0 0 0 0
1322587721:cpu1 2 0 0 0 0
1322587722:cpu1 0 0 0 0 0
1322587723:cpu1 0 0 1 0 0
1322587724:cpu1 0 0 0 0 0
1322587725:cpu1 1 0 0 0 0
1322587726:cpu1 0 0 0 0 0

Btw, perf doesn't seem to yield anything..

Signature · 1 line
Ian Kumlien  -- http://demius.net || http://pomac.netswarm.net
#21 lore raw
Quoted 56 lines · Ian Kumlien
Quoted 24 lines · Michal Hocko
Quoted 5 lines · On Mon 28-11-11 23:28:03, [email protected] wrote:

Hi,

All this time i have been thinking i'm the only one - and i've been to loaded with work during working hours and tired when home =P

Quoted 8 lines

Anyways, I've neen seeing this since -rc1 on:
Intel(R) Core(TM) i5 CPU U520 @ 1.07GHz - x86-64 AMD Phenom(tm) II X6 1090T - x86-64
AMD Athlon(tm) 64 X2 Dual Core Processor 4400+ - x86-64 Intel(R) Core(TM)2 CPU T7200 @ 2.00GHz - i686

Configs available on demand - I've been running the same config for quite some time though.

As I have written in other email could you post your config and collect the following data?
for i in seq 30;
do

        cat /proc/stat > `date +'%s'`
        sleep 1

done
export old_user=0 old_nice=0 old_sys=0 old_idle=0 old_iowait=0;

Quoted 6 lines
# for all your available CPUs
grep cpu0 * | while read cpu user nice sys idle iowait rest; 
do 
        echo $cpu $(($user-$old_user)) $(($nice-$old_nice)) $(($sys-$old_sys)) $(($idle-$old_idle)) $(($iowait-$old_iowait))
        old_user=$user old_nice=$nice old_sys=$sys old_idle=$idle old_iowait=$iowait
done

Sorry, the previous one was AMD X2 4400+ and now the T7200:

1322587696:cpu0 1410110 148 368843 357890604 357890604 1322587697:cpu0 0 0 1 0 0
1322587698:cpu0 1 0 1 0 0
1322587699:cpu0 2 0 1 0 0
1322587700:cpu0 0 0 1 0 0
1322587701:cpu0 0 0 1 0 0
1322587702:cpu0 0 0 1 0 0
1322587703:cpu0 1 0 0 0 0
1322587704:cpu0 0 0 0 0 0
1322587705:cpu0 0 0 1 0 0
1322587706:cpu0 1 0 1 0 0
1322587707:cpu0 0 0 1 0 0
1322587708:cpu0 1 0 0 0 0
1322587709:cpu0 1 0 0 0 0
1322587711:cpu0 0 0 2 0 0
1322587712:cpu0 0 0 0 0 0
1322587713:cpu0 1 0 1 0 0
1322587714:cpu0 1 0 0 0 0
1322587715:cpu0 1 0 1 0 0
1322587716:cpu0 1 0 0 0 0
1322587717:cpu0 0 0 1 0 0
1322587718:cpu0 1 0 1 0 0

Quoted 8 lines

1322587719:cpu0 0 0 1 0 0
1322587720:cpu0 0 0 0 0 0
1322587721:cpu0 2 0 1 0 0
1322587722:cpu0 1 0 0 0 0
1322587723:cpu0 0 0 3 0 0
1322587724:cpu0 1 0 0 0 0
1322587725:cpu0 0 0 0 0 0
1322587726:cpu0 0 0 1 0 0

OK, so the same thing as in another email in the thread (no idle/io_wait accounting).
Could you double check what kind of idle driver are you using? cat /sys/devices/system/cpu/cpuidle/current_driver

Signature · 6 lines
Michal Hocko
SUSE Labs
SUSE LINUX s.r.o.
Lihovarska 1060/12
190 00 Praha 9    
Czech Republic
#22 lore raw
Quoted 26 lines · Michal Hocko
Quoted 25 lines · Ian Kumlien
Quoted 24 lines · Michal Hocko
Quoted 5 lines · On Mon 28-11-11 23:28:03, [email protected] wrote:

Hi,

All this time i have been thinking i'm the only one - and i've been to loaded with work during working hours and tired when home =P

Quoted 8 lines

Anyways, I've neen seeing this since -rc1 on:
Intel(R) Core(TM) i5 CPU U520 @ 1.07GHz - x86-64 AMD Phenom(tm) II X6 1090T - x86-64
AMD Athlon(tm) 64 X2 Dual Core Processor 4400+ - x86-64 Intel(R) Core(TM)2 CPU T7200 @ 2.00GHz - i686

Configs available on demand - I've been running the same config for quite some time though.

As I have written in other email could you post your config and collect the following data?
for i in seq 30;
do

        cat /proc/stat > `date +'%s'`
        sleep 1

done
export old_user=0 old_nice=0 old_sys=0 old_idle=0 old_iowait=0;

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
# for all your available CPUs
grep cpu0 * | while read cpu user nice sys idle iowait rest; 
do 
        echo $cpu $(($user-$old_user)) $(($nice-$old_nice)) $(($sys-$old_sys)) $(($idle-$old_idle)) $(($iowait-$old_iowait))
        old_user=$user old_nice=$nice old_sys=$sys old_idle=$idle old_iowait=$iowait
done

Sorry, the previous one was AMD X2 4400+ and now the T7200:

--8<-- [data] --8<--

Quoted 4 lines

OK, so the same thing as in another email in the thread (no idle/io_wait accounting).
Could you double check what kind of idle driver are you using? cat /sys/devices/system/cpu/cpuidle/current_driver

"none", on both machines

Signature · 1 line
Ian Kumlien  -- http://demius.net || http://pomac.netswarm.net