top/htop and all other CPU usage
Hi,
All this time i have been thinking i'm the only one - and i've been to loaded with work during working hours and tired when home =P
Anyways, I've neen seeing this since -rc1 on:
Intel(R) Core(TM) i5 CPU U520 @ 1.07GHz - x86-64 AMD Phenom(tm) II X6 1090T - x86-64
AMD Athlon(tm) 64 X2 Dual Core Processor 4400+ - x86-64 Intel(R) Core(TM)2 CPU T7200 @ 2.00GHz - i686
Configs available on demand - I've been running the same config for quite some time though.
Quoted 5 lines · On Mon 28-11-11 23:28:03, [email protected] wrote:
Hi,
All this time i have been thinking i'm the only one - and i've been to loaded with work during working hours and tired when home =P
Quoted 8 lines
Anyways, I've neen seeing this since -rc1 on:
Intel(R) Core(TM) i5 CPU U520 @ 1.07GHz - x86-64 AMD Phenom(tm) II X6 1090T - x86-64
AMD Athlon(tm) 64 X2 Dual Core Processor 4400+ - x86-64 Intel(R) Core(TM)2 CPU T7200 @ 2.00GHz - i686Configs available on demand - I've been running the same config for quite some time though.
As I have written in other email could you post your config and collect the following data?
for i in seq 30;
do
cat /proc/stat > `date +'%s'`
sleep 1done
export old_user=0 old_nice=0 old_sys=0 old_idle=0 old_iowait=0;
# for all your available CPUs
grep cpu0 * | while read cpu user nice sys idle iowait rest;
do
echo $cpu $(($user-$old_user)) $(($nice-$old_nice)) $(($sys-$old_sys)) $(($idle-$old_idle)) $(($iowait-$old_iowait))
old_user=$user old_nice=$nice old_sys=$sys old_idle=$idle old_iowait=$iowait
doneThanks
Signature · 6 lines
Michal Hocko SUSE Labs SUSE LINUX s.r.o. Lihovarska 1060/12 190 00 Praha 9 Czech Republic
Quoted 9 lines · Michal Hocko
As I have written in other email could you post your config and collect the following data?
for i inseq 30;
docat /proc/stat > `date +'%s'` sleep 1done
export old_user=0 old_nice=0 old_sys=0 old_idle=0 old_iowait=0;
Quoted 6 lines
# for all your available CPUs
grep cpu0 * | while read cpu user nice sys idle iowait rest;
do
echo $cpu $(($user-$old_user)) $(($nice-$old_nice)) $(($sys-$old_sys)) $(($idle-$old_idle)) $(($iowait-$old_iowait))
old_user=$user old_nice=$nice old_sys=$sys old_idle=$idle old_iowait=$iowait
done1322566208:cpu0 5199 0 2931 357890604 2541
1322566209:cpu0 0 0 1 0 0
1322566210:cpu0 0 0 0 0 0
1322566211:cpu0 0 0 0 0 0
1322566212:cpu0 0 0 0 0 0
1322566213:cpu0 0 0 0 0 0
1322566214:cpu0 1 0 0 0 0
1322566215:cpu0 2 0 0 0 0
1322566216:cpu0 3 0 0 0 0
1322566217:cpu0 2 0 0 0 0
1322566218:cpu0 4 0 0 0 0
1322566219:cpu0 1 0 0 0 0
1322566220:cpu0 2 0 0 0 0
1322566221:cpu0 2 0 1 0 0
1322566222:cpu0 1 0 0 0 0
1322566223:cpu0 2 0 0 0 0
1322566224:cpu0 1 0 1 0 0
1322566225:cpu0 1 0 0 0 0
1322566226:cpu0 2 0 0 0 0
1322566227:cpu0 1 0 1 0 0
1322566228:cpu0 2 0 0 0 0
1322566229:cpu0 2 0 0 0 0
1322566230:cpu0 6 0 3 0 0
1322566231:cpu0 1 0 0 0 0
1322566232:cpu0 2 0 0 0 0
1322566233:cpu0 3 0 0 0 0
1322566234:cpu0 2 0 0 0 0
1322566235:cpu0 2 0 2 0 0
1322566236:cpu0 0 0 1 0 0
1322566237:cpu0 1 0 0 0 0
$ grep . -r /sys/devices/system/cpu/cpuidle/ /sys/devices/system/cpu/cpuidle/current_driver:intel_idle /sys/devices/system/cpu/cpuidle/current_governor_ro:menu
$ grep . -r /sys/devices/system/cpu/cpufreq/ /sys/devices/system/cpu/cpufreq/ondemand/sampling_rate_min:10000 /sys/devices/system/cpu/cpufreq/ondemand/sampling_rate:10000 /sys/devices/system/cpu/cpufreq/ondemand/up_threshold:95 /sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor:1 /sys/devices/system/cpu/cpufreq/ondemand/ignore_nice_load:0 /sys/devices/system/cpu/cpufreq/ondemand/powersave_bias:0 /sys/devices/system/cpu/cpufreq/ondemand/io_is_busy:1
One thing I have to note, it takes some time (from 30 seconds to 10 minutes) before this bug starts manifesting itself.
Quoted 40 lines · Artem S. Tashkinov
Quoted 9 lines · Michal Hocko
As I have written in other email could you post your config and collect the following data?
for i inseq 30;
docat /proc/stat > `date +'%s'` sleep 1done
export old_user=0 old_nice=0 old_sys=0 old_idle=0 old_iowait=0;Quoted 6 lines
# for all your available CPUs grep cpu0 * | while read cpu user nice sys idle iowait rest; do echo $cpu $(($user-$old_user)) $(($nice-$old_nice)) $(($sys-$old_sys)) $(($idle-$old_idle)) $(($iowait-$old_iowait)) old_user=$user old_nice=$nice old_sys=$sys old_idle=$idle old_iowait=$iowait done1322566208:cpu0 5199 0 2931 357890604 2541
1322566209:cpu0 0 0 1 0 0
1322566210:cpu0 0 0 0 0 0
1322566211:cpu0 0 0 0 0 0
1322566212:cpu0 0 0 0 0 0
1322566213:cpu0 0 0 0 0 0
1322566214:cpu0 1 0 0 0 0
1322566215:cpu0 2 0 0 0 0
1322566216:cpu0 3 0 0 0 0
1322566217:cpu0 2 0 0 0 0
1322566218:cpu0 4 0 0 0 0
1322566219:cpu0 1 0 0 0 0
1322566220:cpu0 2 0 0 0 0
1322566221:cpu0 2 0 1 0 0
1322566222:cpu0 1 0 0 0 0
1322566223:cpu0 2 0 0 0 0
1322566224:cpu0 1 0 1 0 0
1322566225:cpu0 1 0 0 0 0
1322566226:cpu0 2 0 0 0 0
1322566227:cpu0 1 0 1 0 0
1322566228:cpu0 2 0 0 0 0
1322566229:cpu0 2 0 0 0 0
Quoted 8 lines
1322566230:cpu0 6 0 3 0 0
1322566231:cpu0 1 0 0 0 0
1322566232:cpu0 2 0 0 0 0
1322566233:cpu0 3 0 0 0 0
1322566234:cpu0 2 0 0 0 0
1322566235:cpu0 2 0 2 0 0
1322566236:cpu0 0 0 1 0 0
1322566237:cpu0 1 0 0 0 0
Hmm, really strange. It looks that idle/iowait is not accounted at all. Which would explain why the numbers you are seeing are so weird.
Quoted 4 lines
$ grep . -r /sys/devices/system/cpu/cpuidle/ /sys/devices/system/cpu/cpuidle/current_driver:intel_idle /sys/devices/system/cpu/cpuidle/current_governor_ro:menu
I will check whether I have a machine with intel_idle somewhere around.
Quoted 12 lines
$ grep . -r /sys/devices/system/cpu/cpufreq/ /sys/devices/system/cpu/cpufreq/ondemand/sampling_rate_min:10000 /sys/devices/system/cpu/cpufreq/ondemand/sampling_rate:10000 /sys/devices/system/cpu/cpufreq/ondemand/up_threshold:95 /sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor:1 /sys/devices/system/cpu/cpufreq/ondemand/ignore_nice_load:0 /sys/devices/system/cpu/cpufreq/ondemand/powersave_bias:0 /sys/devices/system/cpu/cpufreq/ondemand/io_is_busy:1One thing I have to note, it takes some time (from 30 seconds to 10 minutes) before this bug starts manifesting itself.
Signature · 6 lines
Michal Hocko SUSE Labs SUSE LINUX s.r.o. Lihovarska 1060/12 190 00 Praha 9 Czech Republic
Quoted 1 line · Michal Hocko
On Tue 29-11-11 11:38:47, Artem S. Tashkinov wrote:
[...]
Quoted 3 lines
Quoted 3 lines
$ grep . -r /sys/devices/system/cpu/cpuidle/ /sys/devices/system/cpu/cpuidle/current_driver:intel_idle /sys/devices/system/cpu/cpuidle/current_governor_ro:menu
Could you try with acpi_idle driver?
echo acpi_idle > /sys/devices/system/cpu/cpuidle/current_driver
Signature · 6 lines
Michal Hocko SUSE Labs SUSE LINUX s.r.o. Lihovarska 1060/12 190 00 Praha 9 Czech Republic
Quoted 11 lines · Michal Hocko
Quoted 1 line · Michal Hocko
On Tue 29-11-11 11:38:47, Artem S. Tashkinov wrote:
[...]
Quoted 3 lines
Quoted 3 lines
$ grep . -r /sys/devices/system/cpu/cpuidle/ /sys/devices/system/cpu/cpuidle/current_driver:intel_idle /sys/devices/system/cpu/cpuidle/current_governor_ro:menuCould you try with
acpi_idledriver?
echoacpi_idle>/sys/devices/system/cpu/cpuidle/current_driver
[root@localhost ~]# echo acpi_idle > /sys/devices/system/cpu/cpuidle/current_driver -bash: /sys/devices/system/cpu/cpuidle/current_driver: Permission denied
Quoted 9 lines · Artem S. Tashkinov
Quoted 9 lines · Michal Hocko
Quoted 1 line · Michal Hocko
On Tue 29-11-11 11:38:47, Artem S. Tashkinov wrote:
[...]
Quoted 3 lines
Quoted 3 lines
$ grep . -r /sys/devices/system/cpu/cpuidle/ /sys/devices/system/cpu/cpuidle/current_driver:intel_idle /sys/devices/system/cpu/cpuidle/current_governor_ro:menu
Quoted 5 lines
Quoted 2 lines
Could you try with
acpi_idledriver?
echoacpi_idle>/sys/devices/system/cpu/cpuidle/current_driver[root@localhost ~]# echo
acpi_idle>/sys/devices/system/cpu/cpuidle/current_driver-bash:/sys/devices/system/cpu/cpuidle/current_driver: Permission denied
It seems that this is cannot be set in runtime:SYSDEV_CLASS_ATTR(current_driver, 0444, show_current_driver, NULL);
Or maybe you need to boot with cpuidle_sysfs_switch. According to the documentation you might be able to change the governor. I have no idea whether this can help somehow but let's try that.
I haven't found any intel_idle machine in my lab so far and all other acpi_idle machines seem to work (or at least randomly selected ones) so this smells like a major difference in the setup.
Signature · 6 lines
Michal Hocko SUSE Labs SUSE LINUX s.r.o. Lihovarska 1060/12 190 00 Praha 9 Czech Republic
Quoted 10 lines · Michal Hocko
It seems that this is cannot be set in runtime:
SYSDEV_CLASS_ATTR(current_driver, 0444, show_current_driver, NULL);Or maybe you need to boot with
cpuidle_sysfs_switch. According to the documentation you might be able to change the governor. I have no idea whether this can help somehow but let's try that.I haven't found any
intel_idlemachine in my lab so far and all otheracpi_idlemachines seem to work (or at least randomly selected ones) so this smells like a major difference in the setup.
# echo acpi_idle > /sys/devices/system/cpu/cpuidle/current_driver bash: /sys/devices/system/cpu/cpuidle/current_driver: Permission denied
BTW
# cat /sys/devices/system/cpu/cpuidle/available_governors ladder menu
dmesg | grep -i idle
[ 0.000000] Kernel command line: root=/dev/sda1 ro cpuidle_sysfs_switch [ 0.000126] using mwait in idle threads. [ 1.083872] intel_idle: MWAIT substates: 0x1120 [ 1.083873] intel_idle: v0.4 model 0x2A [ 1.083874] intel_idle: lapic_timer_reliable_states 0xffffffff [ 1.159043] cpuidle: using governor ladder [ 1.168842] cpuidle: using governor menu
Quoted 19 lines · Michal Hocko
Quoted 9 lines · Artem S. Tashkinov
Quoted 9 lines · Michal Hocko
Quoted 1 line · Michal Hocko
On Tue 29-11-11 11:38:47, Artem S. Tashkinov wrote:
[...]
Quoted 3 lines
Quoted 3 lines
$ grep . -r /sys/devices/system/cpu/cpuidle/ /sys/devices/system/cpu/cpuidle/current_driver:intel_idle /sys/devices/system/cpu/cpuidle/current_governor_ro:menuQuoted 5 lines
Quoted 2 lines
Could you try with
acpi_idledriver?
echoacpi_idle>/sys/devices/system/cpu/cpuidle/current_driver[root@localhost ~]# echo
acpi_idle>/sys/devices/system/cpu/cpuidle/current_driver-bash:/sys/devices/system/cpu/cpuidle/current_driver: Permission deniedIt seems that this is cannot be set in runtime:
SYSDEV_CLASS_ATTR(current_driver, 0444, show_current_driver, NULL);
Quoted 7 lines
Or maybe you need to boot with
cpuidle_sysfs_switch. According to the documentation you might be able to change the governor. I have no idea whether this can help somehow but let's try that.I haven't found any
intel_idlemachine in my lab so far and all otheracpi_idlemachines seem to work (or at least randomly selected ones) so this smells like a major difference in the setup.
I'm able to reproduce that with acpi_driver on one box, but not on demand.
Thanks,
Rafael
Quoted 24 lines · Rafael J. Wysocki
Quoted 23 lines · Michal Hocko
Quoted 9 lines · Artem S. Tashkinov
Quoted 9 lines · Michal Hocko
Quoted 1 line · Michal Hocko
On Tue 29-11-11 11:38:47, Artem S. Tashkinov wrote:
[...]
Quoted 3 lines
Quoted 3 lines
$ grep . -r /sys/devices/system/cpu/cpuidle/ /sys/devices/system/cpu/cpuidle/current_driver:intel_idle /sys/devices/system/cpu/cpuidle/current_governor_ro:menuQuoted 5 lines
Quoted 2 lines
Could you try with
acpi_idledriver?
echoacpi_idle>/sys/devices/system/cpu/cpuidle/current_driver[root@localhost ~]# echo
acpi_idle>/sys/devices/system/cpu/cpuidle/current_driver-bash:/sys/devices/system/cpu/cpuidle/current_driver: Permission deniedIt seems that this is cannot be set in runtime:
SYSDEV_CLASS_ATTR(current_driver, 0444, show_current_driver, NULL);Or maybe you need to boot with
cpuidle_sysfs_switch. According to the documentation you might be able to change the governor. I have no idea whether this can help somehow but let's try that.
Quoted 5 lines
Quoted 3 lines
I haven't found any
intel_idlemachine in my lab so far and all otheracpi_idlemachines seem to work (or at least randomly selected ones) so this smells like a major difference in the setup.I'm able to reproduce that with
acpi_driveron one box, but not on demand.
And do you see the same thing (no idle/io_wait) updates?
Quoted 3 lines
Thanks,
Rafael
Signature · 6 lines
Michal Hocko SUSE Labs SUSE LINUX s.r.o. Lihovarska 1060/12 190 00 Praha 9 Czech Republic
Quoted 25 lines · Michal Hocko
Quoted 24 lines · Rafael J. Wysocki
Quoted 23 lines · Michal Hocko
Quoted 9 lines · Artem S. Tashkinov
Quoted 9 lines · Michal Hocko
Quoted 1 line · Michal Hocko
On Tue 29-11-11 11:38:47, Artem S. Tashkinov wrote:
[...]
Quoted 3 lines
Quoted 3 lines
$ grep . -r /sys/devices/system/cpu/cpuidle/ /sys/devices/system/cpu/cpuidle/current_driver:intel_idle /sys/devices/system/cpu/cpuidle/current_governor_ro:menuQuoted 5 lines
Quoted 2 lines
Could you try with
acpi_idledriver?
echoacpi_idle>/sys/devices/system/cpu/cpuidle/current_driver[root@localhost ~]# echo
acpi_idle>/sys/devices/system/cpu/cpuidle/current_driver-bash:/sys/devices/system/cpu/cpuidle/current_driver: Permission deniedIt seems that this is cannot be set in runtime:
SYSDEV_CLASS_ATTR(current_driver, 0444, show_current_driver, NULL);Or maybe you need to boot with
cpuidle_sysfs_switch. According to the documentation you might be able to change the governor. I have no idea whether this can help somehow but let's try that.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
I haven't found any
intel_idlemachine in my lab so far and all otheracpi_idlemachines seem to work (or at least randomly selected ones) so this smells like a major difference in the setup.I'm able to reproduce that with
acpi_driveron one box, but not on demand.And do you see the same thing (no idle/io_wait) updates?
Actaully, I was wrong. The box I'm seeing the issue on also has "none" in /sys/devices/system/cpu/cpuidle/current_driver. Sorry for the confusion.
Thanks,
Rafael
[Let's add Len to the CC for idle driver]
Quoted 3 lines · Rafael J. Wysocki
Quoted 2 lines · Michal Hocko
Quoted 1 line · Rafael J. Wysocki
On Tuesday, November 29, 2011, Michal Hocko wrote:
[...]
Quoted 10 lines
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
I haven't found any
intel_idlemachine in my lab so far and all otheracpi_idlemachines seem to work (or at least randomly selected ones) so this smells like a major difference in the setup.I'm able to reproduce that with
acpi_driveron one box, but not on demand.And do you see the same thing (no idle/io_wait) updates?
Actaully, I was wrong. The box I'm seeing the issue on also has "none" in
/sys/devices/system/cpu/cpuidle/current_driver. Sorry for the confusion.
OK. So we have seen the issue only with intel_idle and none drivers so far. acpi_idle which is at my machines works just fine.
I think we should focus on those drivers.
To summarize this issue.
Users are seeing weird values reported by [h]top. CPUs seem to be at 100% even though there is nothing hogging them. /proc/stat collected data on the affected system shown that idle/io_wait are not accounted properly.
It has been identified that problem disappears if a25cac51 [proc:
Consider NO_HZ when printing idle and iowait times] is reverted. This patch fixes a bug when idle/io_wait times are not repororted properly when a CPU is tickless. It relies on get_cpu_idle_time_us which either reports sched_time idle_sleeptime or (idle_sleeptime + now-idle_entrytime) if we are idle at the moment.
While implementation is not race free (we better not use locks in that path...) so we might race:
E.g.
CPU1 CPU2
now = ktime_get
tick_nohz_start_idle ts->idle_entrytime = now; if (ts->idle_active) ts->idle_active = 1
[...]
return idle_sleeptime
But this is OK because sleeptime will be more or less accurate. We just skip few ticks.
It would be worse if we had a race like:
CPU1 CPU2
now = ktime_get
tick_nohz_start_idle now = ktime_get update_ts_time_stats() ts->idle_entrytime = now; ts->idle_active = 1
if (ts->idle_active)
delta = ktime_sub(now, idle_entrytime)
ktime_add(idle_sleeptime, delta)
In this case we might get an overflow from ktime_sub but AFAIU the ktime_* magic the overflow should cause to get smaller idle_sleeptime in the end after ktime_add (we do not add a small number but rather subtract it), right?
So it shouldn't be a big deal as well.
So the question is. What is the role of the idle driver here?
Thanks
Signature · 6 lines
Michal Hocko SUSE Labs SUSE LINUX s.r.o. Lihovarska 1060/12 190 00 Praha 9 Czech Republic
On Thu 01-12-11 15:07:49, Michal Hocko wrote:
[...]
Quoted 25 lines
While implementation is not race free (we better not use locks in that path...) so we might race:
E.g.
CPU1 CPU2
now =ktime_gettick_nohz_start_idle ts->idle_entrytime = now; if (ts->idle_active) ts->idle_active = 1[...]
returnidle_sleeptimeBut this is OK because sleeptime will be more or less accurate. We just skip few ticks.
It would be worse if we had a race like:
CPU1 CPU2
now =ktime_gettick_nohz_start_idle now = ktime_get update_ts_time_stats() ts->idle_entrytime = now; ts->idle_active = 1
Quoted 8 lines
if (
ts->idle_active)
delta =ktime_sub(now, idle_entrytime)
ktime_add(idle_sleeptime, delta)In this case we might get an overflow from
ktime_subbut AFAIU the ktime_* magic the overflow should cause to get smalleridle_sleeptimein the end afterktime_add(we do not add a small number but rather subtract it), right?
Scratch that. Dunno why but I thought that ktime_t has unsigned values but it is apparently not true (tv64 is s64). Anyway the above races should be safe.
Signature · 6 lines
Michal Hocko SUSE Labs SUSE LINUX s.r.o. Lihovarska 1060/12 190 00 Praha 9 Czech Republic
Quoted 40 lines · Artem S. Tashkinov
Quoted 9 lines · Michal Hocko
As I have written in other email could you post your config and collect the following data?
for i inseq 30;
docat /proc/stat > `date +'%s'` sleep 1done
export old_user=0 old_nice=0 old_sys=0 old_idle=0 old_iowait=0;Quoted 6 lines
# for all your available CPUs grep cpu0 * | while read cpu user nice sys idle iowait rest; do echo $cpu $(($user-$old_user)) $(($nice-$old_nice)) $(($sys-$old_sys)) $(($idle-$old_idle)) $(($iowait-$old_iowait)) old_user=$user old_nice=$nice old_sys=$sys old_idle=$idle old_iowait=$iowait done1322566208:cpu0 5199 0 2931 357890604 2541
1322566209:cpu0 0 0 1 0 0
1322566210:cpu0 0 0 0 0 0
1322566211:cpu0 0 0 0 0 0
1322566212:cpu0 0 0 0 0 0
1322566213:cpu0 0 0 0 0 0
1322566214:cpu0 1 0 0 0 0
1322566215:cpu0 2 0 0 0 0
1322566216:cpu0 3 0 0 0 0
1322566217:cpu0 2 0 0 0 0
1322566218:cpu0 4 0 0 0 0
1322566219:cpu0 1 0 0 0 0
1322566220:cpu0 2 0 0 0 0
1322566221:cpu0 2 0 1 0 0
1322566222:cpu0 1 0 0 0 0
1322566223:cpu0 2 0 0 0 0
1322566224:cpu0 1 0 1 0 0
1322566225:cpu0 1 0 0 0 0
1322566226:cpu0 2 0 0 0 0
1322566227:cpu0 1 0 1 0 0
1322566228:cpu0 2 0 0 0 0
1322566229:cpu0 2 0 0 0 0
Quoted 8 lines
1322566230:cpu0 6 0 3 0 0
1322566231:cpu0 1 0 0 0 0
1322566232:cpu0 2 0 0 0 0
1322566233:cpu0 3 0 0 0 0
1322566234:cpu0 2 0 0 0 0
1322566235:cpu0 2 0 2 0 0
1322566236:cpu0 0 0 1 0 0
1322566237:cpu0 1 0 0 0 0
Could you post raw data as well? Ideally starting right after boot and collected for more than 30s (longer better...)
Thanks!
Signature · 6 lines
Michal Hocko SUSE Labs SUSE LINUX s.r.o. Lihovarska 1060/12 190 00 Praha 9 Czech Republic
PATCH 1/1 proc: Do not overflow get_{idle,iowait}_time for nohz (was: Re: Re: [REGRESSION] [Linux 3.2] top/htop and all other CPU usage)
Quoted 23 lines · Michal Hocko
Quoted 22 lines · Artem S. Tashkinov
Quoted 9 lines · Michal Hocko
As I have written in other email could you post your config and collect the following data?
for i inseq 30;
docat /proc/stat > `date +'%s'` sleep 1done
export old_user=0 old_nice=0 old_sys=0 old_idle=0 old_iowait=0;Quoted 6 lines
# for all your available CPUs grep cpu0 * | while read cpu user nice sys idle iowait rest; do echo $cpu $(($user-$old_user)) $(($nice-$old_nice)) $(($sys-$old_sys)) $(($idle-$old_idle)) $(($iowait-$old_iowait)) old_user=$user old_nice=$nice old_sys=$sys old_idle=$idle old_iowait=$iowait done1322566208:cpu0 5199 0 2931 357890604 2541
1322566209:cpu0 0 0 1 0 0
1322566210:cpu0 0 0 0 0 0
1322566211:cpu0 0 0 0 0 0
[...]
Quoted 3 lines
Could you post raw data as well? Ideally starting right after boot and collected for more than 30s (longer better...)
Ahh, missed that you attached data. And also noticed that you are using CONFIG_HZ_300 which explains the problem and why I do cannot reproduce it.
get_{idle,iowait}_time translates us to cputime64_t and it uses usecs_to_cputime which is just an alias for usecs_to_jiffies and it does
if (u > jiffies_to_usecs(MAX_JIFFY_OFFSET))
return MAX_JIFFY_OFFSET;which in your case (HZ=300) means that we overflow much more often than for HZ==100. The patch below should fix this:
---
Quoted 1 line
From
23882e2aabe27934df4d23b0ed52749fd4f61ab4Mon Sep 17 00:00:00 2001
From: Michal Hocko <[email protected]>
Date: Fri, 2 Dec 2011 16:17:03 +0100
Subject: [PATCH] proc: Do not overflow get_{idle,iowait}_time for nohz
get_{idle,iowait}_time use usecs_to_cputime to translate micro seconds time to cputime64_t. This is just an alias to usecs_to_jiffies which reduces the data type from u64 to unsigned int and also checks whether the given paramerer overflows jiffies_to_usecs(MAX_JIFFY_OFFSET) and returns MAX_JIFFY_OFFSET in that case. How much we overflow depends on CONFIG_HZ and especially for CONFIG_HZ_300 it is quite low (1431649781).
This results in a bug when people saw [h]top going mad reporting 100% CPU usage even though there was basically no CPU load at all. The reason was simply that /proc/stat stopped reporting idle/io_wait changes (and reported MAX_JIFFY_OFFSET) and so the only change happenning was for user system time.
Let's use nsecs_to_jiffies64 instead as it doesn't overflow.
--- fs/proc/stat.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
fs/proc/stat.c +2 -2
@@ -32,7 +32,7 @@ static cputime64_t get_idle_time(int cpu) idle =kstat_cpu (cpu).cpustat.idle; idle =cputime64_add (idle,arch_idle_time (cpu)); }else - idle =usecs_to_cputime (idle_time); + idle =nsecs_to_jiffies64 (1000 * idle_time);return idle; }
@@ -46,7 +46,7 @@ static cputime64_t get_iowait_time(int cpu)/* !NO_HZ so we can rely on cpustat.iowait */ iowait =kstat_cpu (cpu).cpustat.iowait; else - iowait =usecs_to_cputime (iowait_time); + iowait =nsecs_to_jiffies64 (1000 * iowait_time);return iowait; }
Signature · 10 lines
1.7.7.3 -- Michal Hocko SUSE Labs SUSE LINUX s.r.o. Lihovarska 1060/12 190 00 Praha 9 Czech Republic
Re: [PATCH] proc: Do not overflow get_{idle,iowait}_time for nohz (was: Re: Re: [REGRESSION] [Linux 3.2] top/htop and all other CPU usage)
Quoted 33 lines · Michal Hocko
Quoted 23 lines · Michal Hocko
Quoted 22 lines · Artem S. Tashkinov
Quoted 9 lines · Michal Hocko
As I have written in other email could you post your config and collect the following data?
for i inseq 30;
docat /proc/stat > `date +'%s'` sleep 1done
export old_user=0 old_nice=0 old_sys=0 old_idle=0 old_iowait=0;Quoted 6 lines
# for all your available CPUs grep cpu0 * | while read cpu user nice sys idle iowait rest; do echo $cpu $(($user-$old_user)) $(($nice-$old_nice)) $(($sys-$old_sys)) $(($idle-$old_idle)) $(($iowait-$old_iowait)) old_user=$user old_nice=$nice old_sys=$sys old_idle=$idle old_iowait=$iowait done1322566208:cpu0 5199 0 2931 357890604 2541
1322566209:cpu0 0 0 1 0 0
1322566210:cpu0 0 0 0 0 0
1322566211:cpu0 0 0 0 0 0[...]
Quoted 3 lines
Could you post raw data as well? Ideally starting right after boot and collected for more than 30s (longer better...)
Ahh, missed that you attached data. And also noticed that you are using
CONFIG_HZ_300which explains the problem and why I do cannot reproduce it.
Quoted 6 lines
get_{idle,iowait}_time translates us to
cputime64_tand it usesusecs_to_cputimewhich is just an alias forusecs_to_jiffiesand it doesif (u > jiffies_to_usecs(MAX_JIFFY_OFFSET)) return MAX_JIFFY_OFFSET;which in your case (HZ=300) means that we overflow much more often than for HZ==100. The patch below should fix this:
And the one with a more cleaned up changelog. No functional changes
---
Quoted 1 line
From
107887016b91de59194a93c751d040b05d5e37feMon Sep 17 00:00:00 2001
From: Michal Hocko <[email protected]>
Date: Fri, 2 Dec 2011 16:17:03 +0100
Subject: [PATCH] proc: Do not overflow get_{idle,iowait}_time for nohz
Since a25cac51 [proc: Consider NO_HZ when printing idle and iowait times] we are reporting idle/io_wait time also while a CPU is tickless. We rely on get_{idle,iowait}_time functions to retrieve proper data.
These functions, however, use usecs_to_cputime to translate micro seconds time to cputime64_t. This is just an alias to usecs_to_jiffies which reduces the data type from u64 to unsigned int and also checks whether the given parameter overflows jiffies_to_usecs(MAX_JIFFY_OFFSET) and returns MAX_JIFFY_OFFSET in that case.
When do we overflow depends on CONFIG_HZ but especially for CONFIG_HZ_300 it is quite low (1431649781) so we are getting MAX_JIFFY_OFFSET for >3000s! until we overflow unsigned int. Just for reference CONFIG_100 has an overflow window around 20s, CONFIG_250 ~8s and CONFIG_1000 ~2s.
This results in a bug when people saw [h]top going mad reporting 100% CPU usage even though there was basically no CPU load. The reason was simply that /proc/stat stopped reporting idle/io_wait changes (and reported MAX_JIFFY_OFFSET) and so the only change happening was for user system time.
Let's use nsecs_to_jiffies64 instead which doesn't reduce the precision to 32b type and it is much more appropriate for cumulative time values (unlike usecs_to_jiffies which intended for timeout calculations).
--- fs/proc/stat.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
fs/proc/stat.c +2 -2
@@ -32,7 +32,7 @@ static cputime64_t get_idle_time(int cpu) idle =kstat_cpu (cpu).cpustat.idle; idle =cputime64_add (idle,arch_idle_time (cpu)); }else - idle =usecs_to_cputime (idle_time); + idle =nsecs_to_jiffies64 (1000 * idle_time);return idle; }
@@ -46,7 +46,7 @@ static cputime64_t get_iowait_time(int cpu)/* !NO_HZ so we can rely on cpustat.iowait */ iowait =kstat_cpu (cpu).cpustat.iowait; else - iowait =usecs_to_cputime (iowait_time); + iowait =nsecs_to_jiffies64 (1000 * iowait_time);return iowait; }
Signature · 10 lines
1.7.7.3 -- Michal Hocko SUSE Labs SUSE LINUX s.r.o. Lihovarska 1060/12 190 00 Praha 9 Czech Republic
Artem S. Tashkinov
Tested-byRe: Re: [PATCH] proc: Do not overflow get_{idle,iowait}_time for nohz (was: Re: Re: [REGRESSION] [Linux 3.2] top/htop and all other CPU usage)
Quoted 62 lines · Michal Hocko
And the one with a more cleaned up changelog. No functional changes
---
From107887016b91de59194a93c751d040b05d5e37feMon Sep 17 00:00:00 2001
From: Michal Hocko <>
Date: Fri, 2 Dec 2011 16:17:03 +0100
Subject: [PATCH] proc: Do not overflow get_{idle,iowait}_time for nohzSince
a25cac51[proc: ConsiderNO_HZwhen printing idle and iowait times] we are reporting idle/io_wait time also while a CPU is tickless. We rely on get_{idle,iowait}_time functions to retrieve proper data.These functions, however, use
usecs_to_cputimeto translate micro seconds time tocputime64_t. This is just an alias tousecs_to_jiffieswhich reduces the data type from u64 to unsigned int and also checks whether the given parameter overflowsjiffies_to_usecs(MAX_JIFFY_OFFSET)and returnsMAX_JIFFY_OFFSETin that case.When do we overflow depends on
CONFIG_HZbut especially forCONFIG_HZ_300it is quite low (1431649781) so we are gettingMAX_JIFFY_OFFSETfor >3000s! until we overflow unsigned int. Just for referenceCONFIG_100has an overflow window around 20s,CONFIG_250~8s andCONFIG_1000~2s.This results in a bug when people saw [h]top going mad reporting 100% CPU usage even though there was basically no CPU load. The reason was simply that
/proc/statstopped reporting idle/io_wait changes (and reportedMAX_JIFFY_OFFSET) and so the only change happening was for user system time.Let's use
nsecs_to_jiffies64instead which doesn't reduce the precision to 32b type and it is much more appropriate for cumulative time values (unlikeusecs_to_jiffieswhich intended for timeout calculations).Signed-off-by Michal Hocko <[email protected]>--- fs/proc/stat.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)fs/proc/stat.c +2
-2@@ -32,7 +32,7 @@ static cputime64_t get_idle_time(int cpu) idle =kstat_cpu (cpu).cpustat.idle; idle =cputime64_add (idle,arch_idle_time (cpu)); }else - idle =usecs_to_cputime (idle_time); + idle =nsecs_to_jiffies64 (1000 * idle_time);return idle; }@@ -46,7 +46,7 @@ static cputime64_t get_iowait_time(int cpu)/* !NO_HZ so we can rely on cpustat.iowait */ iowait =kstat_cpu (cpu).cpustat.iowait; else - iowait =usecs_to_cputime (iowait_time); + iowait =nsecs_to_jiffies64 (1000 * iowait_time);return iowait; }Signature · 1 line
1.7.7.3
Thank you, this patch has fixed the issue for me.
Re: Re: [PATCH] proc: Do not overflow get_{idle,iowait}_time for nohz (was: Re: Re: [REGRESSION] [Linux 3.2] top/htop and all other CPU usage)
Quoted 60 lines · Artem S. Tashkinov
Quoted 58 lines · Michal Hocko
And the one with a more cleaned up changelog. No functional changes
---
From107887016b91de59194a93c751d040b05d5e37feMon Sep 17 00:00:00 2001
From: Michal Hocko <>
Date: Fri, 2 Dec 2011 16:17:03 +0100
Subject: [PATCH] proc: Do not overflow get_{idle,iowait}_time for nohzSince
a25cac51[proc: ConsiderNO_HZwhen printing idle and iowait times] we are reporting idle/io_wait time also while a CPU is tickless. We rely on get_{idle,iowait}_time functions to retrieve proper data.These functions, however, use
usecs_to_cputimeto translate micro seconds time tocputime64_t. This is just an alias tousecs_to_jiffieswhich reduces the data type from u64 to unsigned int and also checks whether the given parameter overflowsjiffies_to_usecs(MAX_JIFFY_OFFSET)and returnsMAX_JIFFY_OFFSETin that case.When do we overflow depends on
CONFIG_HZbut especially forCONFIG_HZ_300it is quite low (1431649781) so we are gettingMAX_JIFFY_OFFSETfor >3000s! until we overflow unsigned int. Just for referenceCONFIG_100has an overflow window around 20s,CONFIG_250~8s andCONFIG_1000~2s.This results in a bug when people saw [h]top going mad reporting 100% CPU usage even though there was basically no CPU load. The reason was simply that
/proc/statstopped reporting idle/io_wait changes (and reportedMAX_JIFFY_OFFSET) and so the only change happening was for user system time.Let's use
nsecs_to_jiffies64instead which doesn't reduce the precision to 32b type and it is much more appropriate for cumulative time values (unlikeusecs_to_jiffieswhich intended for timeout calculations).Signed-off-by Michal Hocko <[email protected]>--- fs/proc/stat.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)fs/proc/stat.c +2
-2@@ -32,7 +32,7 @@ static cputime64_t get_idle_time(int cpu) idle =kstat_cpu (cpu).cpustat.idle; idle =cputime64_add (idle,arch_idle_time (cpu)); }else - idle =usecs_to_cputime (idle_time); + idle =nsecs_to_jiffies64 (1000 * idle_time);return idle; }@@ -46,7 +46,7 @@ static cputime64_t get_iowait_time(int cpu)/* !NO_HZ so we can rely on cpustat.iowait */ iowait =kstat_cpu (cpu).cpustat.iowait; else - iowait =usecs_to_cputime (iowait_time); + iowait =nsecs_to_jiffies64 (1000 * iowait_time);
Quoted 8 lines
Quoted 4 lines
fs/proc/stat.c +0
-0return iowait; }Signature · 1 line
1.7.7.3Thank you, this patch has fixed the issue for me.
Tested-by Artem S. Tashkinov <[email protected]>
Thanks for retesting!
Signature · 6 lines
Michal Hocko SUSE Labs SUSE LINUX s.r.o. Lihovarska 1060/12 190 00 Praha 9 Czech Republic
Quoted 2 lines · Michal Hocko
Could you post raw data as well? Ideally starting right after boot and collected for more than 30s (longer better...)
Already posted that under the "out.tar.xz" filename - just grep your mail agent history.
If it's not there, here's a copy: http://www.gossamer-threads.com/lists/linux/kernel/1460716#1460716
Best wishes,
Artem
Quoted 24 lines · Michal Hocko
Quoted 5 lines · On Mon 28-11-11 23:28:03, [email protected] wrote:
Hi,
All this time i have been thinking i'm the only one - and i've been to loaded with work during working hours and tired when home =P
Quoted 8 lines
Anyways, I've neen seeing this since -rc1 on:
Intel(R) Core(TM) i5 CPU U520 @ 1.07GHz - x86-64 AMD Phenom(tm) II X6 1090T - x86-64
AMD Athlon(tm) 64 X2 Dual Core Processor 4400+ - x86-64 Intel(R) Core(TM)2 CPU T7200 @ 2.00GHz - i686Configs available on demand - I've been running the same config for quite some time though.
As I have written in other email could you post your config and collect the following data?
for i inseq 30;
docat /proc/stat > `date +'%s'` sleep 1done
export old_user=0 old_nice=0 old_sys=0 old_idle=0 old_iowait=0;
Quoted 6 lines
# for all your available CPUs
grep cpu0 * | while read cpu user nice sys idle iowait rest;
do
echo $cpu $(($user-$old_user)) $(($nice-$old_nice)) $(($sys-$old_sys)) $(($idle-$old_idle)) $(($iowait-$old_iowait))
old_user=$user old_nice=$nice old_sys=$sys old_idle=$idle old_iowait=$iowait
done1322587011:cpu0 5838553 164 1252921 1844674407370 78234 1322587012:cpu0 4 0 1 0 0
1322587013:cpu0 4 0 2 0 0
1322587014:cpu0 2 0 1 0 0
1322587015:cpu0 3 0 2 0 0
1322587016:cpu0 7 0 2 0 4
1322587017:cpu0 2 0 2 0 0
1322587018:cpu0 1 0 1 0 0
1322587019:cpu0 4 0 2 0 1
1322587020:cpu0 5 0 3 0 5
1322587021:cpu0 7 0 1 0 3
1322587022:cpu0 4 0 2 0 0
1322587023:cpu0 4 0 2 0 0
1322587024:cpu0 3 0 3 0 0
1322587025:cpu0 3 0 1 0 0
1322587026:cpu0 4 0 2 0 4
1322587027:cpu0 2 0 2 0 0
1322587028:cpu0 2 0 1 0 0
1322587029:cpu0 4 0 1 0 0
1322587030:cpu0 5 0 1 0 0
1322587032:cpu0 5 0 1 0 4
1322587033:cpu0 6 0 1 0 0
1322587034:cpu0 1 0 2 0 0
1322587035:cpu0 3 0 2 0 0
1322587036:cpu0 3 0 1 0 0
1322587037:cpu0 4 0 2 0 2
1322587038:cpu0 4 0 1 0 0
1322587039:cpu0 2 0 1 0 0
1322587040:cpu0 3 0 1 0 0
1322587041:cpu0 4 0 3 0 0
1322587011:cpu1 5944677 172 1287816 1844674407370 871 1322587012:cpu1 4 0 2 0 0
1322587013:cpu1 2 0 2 0 0
1322587014:cpu1 3 0 0 0 0
1322587015:cpu1 3 0 1 0 9
1322587016:cpu1 22 0 3 0 18
1322587017:cpu1 10 0 3 0 13
1322587018:cpu1 3 0 1 0 0
1322587019:cpu1 4 0 3 0 14
1322587020:cpu1 14 0 2 0 7
1322587021:cpu1 16 0 1 0 2
1322587022:cpu1 5 0 0 0 0
1322587023:cpu1 7 0 1 0 0
1322587024:cpu1 8 0 1 0 0
1322587025:cpu1 2 0 2 0 0
1322587026:cpu1 4 0 2 0 5
1322587027:cpu1 7 0 2 0 0
1322587028:cpu1 5 0 0 0 0
1322587029:cpu1 4 0 2 0 0
1322587030:cpu1 6 0 0 0 0
1322587032:cpu1 3 0 2 0 0
1322587033:cpu1 6 0 1 0 0
1322587034:cpu1 2 0 1 0 0
1322587035:cpu1 4 0 2 0 0
1322587036:cpu1 3 0 1 0 0
1322587037:cpu1 4 0 0 0 0
1322587038:cpu1 8 0 1 0 0
1322587039:cpu1 4 0 2 0 0
1322587040:cpu1 4 0 0 0 0
1322587041:cpu1 8 0 1 0 0
For me the cpu usage is sporadic and only shows up as user or kernel, nothing else.
Signature · 1 line
Ian Kumlien -- http://demius.net || http://pomac.netswarm.net
Quoted 24 lines · Michal Hocko
Quoted 5 lines · On Mon 28-11-11 23:28:03, [email protected] wrote:
Hi,
All this time i have been thinking i'm the only one - and i've been to loaded with work during working hours and tired when home =P
Quoted 8 lines
Anyways, I've neen seeing this since -rc1 on:
Intel(R) Core(TM) i5 CPU U520 @ 1.07GHz - x86-64 AMD Phenom(tm) II X6 1090T - x86-64
AMD Athlon(tm) 64 X2 Dual Core Processor 4400+ - x86-64 Intel(R) Core(TM)2 CPU T7200 @ 2.00GHz - i686Configs available on demand - I've been running the same config for quite some time though.
As I have written in other email could you post your config and collect the following data?
for i inseq 30;
docat /proc/stat > `date +'%s'` sleep 1done
export old_user=0 old_nice=0 old_sys=0 old_idle=0 old_iowait=0;
Quoted 6 lines
# for all your available CPUs
grep cpu0 * | while read cpu user nice sys idle iowait rest;
do
echo $cpu $(($user-$old_user)) $(($nice-$old_nice)) $(($sys-$old_sys)) $(($idle-$old_idle)) $(($iowait-$old_iowait))
old_user=$user old_nice=$nice old_sys=$sys old_idle=$idle old_iowait=$iowait
doneSorry, the previous one was AMD X2 4400+ and now the T7200:
1322587696:cpu0 1410110 148 368843 357890604 357890604 1322587697:cpu0 0 0 1 0 0
1322587698:cpu0 1 0 1 0 0
1322587699:cpu0 2 0 1 0 0
1322587700:cpu0 0 0 1 0 0
1322587701:cpu0 0 0 1 0 0
1322587702:cpu0 0 0 1 0 0
1322587703:cpu0 1 0 0 0 0
1322587704:cpu0 0 0 0 0 0
1322587705:cpu0 0 0 1 0 0
1322587706:cpu0 1 0 1 0 0
1322587707:cpu0 0 0 1 0 0
1322587708:cpu0 1 0 0 0 0
1322587709:cpu0 1 0 0 0 0
1322587711:cpu0 0 0 2 0 0
1322587712:cpu0 0 0 0 0 0
1322587713:cpu0 1 0 1 0 0
1322587714:cpu0 1 0 0 0 0
1322587715:cpu0 1 0 1 0 0
1322587716:cpu0 1 0 0 0 0
1322587717:cpu0 0 0 1 0 0
1322587718:cpu0 1 0 1 0 0
1322587719:cpu0 0 0 1 0 0
1322587720:cpu0 0 0 0 0 0
1322587721:cpu0 2 0 1 0 0
1322587722:cpu0 1 0 0 0 0
1322587723:cpu0 0 0 3 0 0
1322587724:cpu0 1 0 0 0 0
1322587725:cpu0 0 0 0 0 0
1322587726:cpu0 0 0 1 0 0
1322587696:cpu1 1395249 509 284978 357890604 95640 1322587697:cpu1 0 0 0 0 0
1322587698:cpu1 0 0 0 0 0
1322587699:cpu1 0 0 0 0 0
1322587700:cpu1 1 0 0 0 0
1322587701:cpu1 1 0 0 0 0
1322587702:cpu1 0 0 0 0 0
1322587703:cpu1 0 0 1 0 0
1322587704:cpu1 1 0 0 0 0
1322587705:cpu1 0 0 1 0 0
1322587706:cpu1 1 0 0 0 0
1322587707:cpu1 0 0 0 0 0
1322587708:cpu1 1 0 0 0 0
1322587709:cpu1 1 0 1 0 0
1322587711:cpu1 0 0 0 0 0
1322587712:cpu1 0 0 0 0 0
1322587713:cpu1 0 0 0 0 0
1322587714:cpu1 0 0 0 0 0
1322587715:cpu1 0 0 0 0 0
1322587716:cpu1 0 0 1 0 0
1322587717:cpu1 0 0 0 0 0
1322587718:cpu1 0 0 0 0 0
1322587719:cpu1 0 0 1 0 0
1322587720:cpu1 1 0 0 0 0
1322587721:cpu1 2 0 0 0 0
1322587722:cpu1 0 0 0 0 0
1322587723:cpu1 0 0 1 0 0
1322587724:cpu1 0 0 0 0 0
1322587725:cpu1 1 0 0 0 0
1322587726:cpu1 0 0 0 0 0
Btw, perf doesn't seem to yield anything..
Signature · 1 line
Ian Kumlien -- http://demius.net || http://pomac.netswarm.net
Quoted 56 lines · Ian Kumlien
Quoted 24 lines · Michal Hocko
Quoted 5 lines · On Mon 28-11-11 23:28:03, [email protected] wrote:
Hi,
All this time i have been thinking i'm the only one - and i've been to loaded with work during working hours and tired when home =P
Quoted 8 lines
Anyways, I've neen seeing this since -rc1 on:
Intel(R) Core(TM) i5 CPU U520 @ 1.07GHz - x86-64 AMD Phenom(tm) II X6 1090T - x86-64
AMD Athlon(tm) 64 X2 Dual Core Processor 4400+ - x86-64 Intel(R) Core(TM)2 CPU T7200 @ 2.00GHz - i686Configs available on demand - I've been running the same config for quite some time though.
As I have written in other email could you post your config and collect the following data?
for i inseq 30;
docat /proc/stat > `date +'%s'` sleep 1done
export old_user=0 old_nice=0 old_sys=0 old_idle=0 old_iowait=0;Quoted 6 lines
# for all your available CPUs grep cpu0 * | while read cpu user nice sys idle iowait rest; do echo $cpu $(($user-$old_user)) $(($nice-$old_nice)) $(($sys-$old_sys)) $(($idle-$old_idle)) $(($iowait-$old_iowait)) old_user=$user old_nice=$nice old_sys=$sys old_idle=$idle old_iowait=$iowait doneSorry, the previous one was AMD X2 4400+ and now the T7200:
1322587696:cpu0 1410110 148 368843 357890604 357890604 1322587697:cpu0 0 0 1 0 0
1322587698:cpu0 1 0 1 0 0
1322587699:cpu0 2 0 1 0 0
1322587700:cpu0 0 0 1 0 0
1322587701:cpu0 0 0 1 0 0
1322587702:cpu0 0 0 1 0 0
1322587703:cpu0 1 0 0 0 0
1322587704:cpu0 0 0 0 0 0
1322587705:cpu0 0 0 1 0 0
1322587706:cpu0 1 0 1 0 0
1322587707:cpu0 0 0 1 0 0
1322587708:cpu0 1 0 0 0 0
1322587709:cpu0 1 0 0 0 0
1322587711:cpu0 0 0 2 0 0
1322587712:cpu0 0 0 0 0 0
1322587713:cpu0 1 0 1 0 0
1322587714:cpu0 1 0 0 0 0
1322587715:cpu0 1 0 1 0 0
1322587716:cpu0 1 0 0 0 0
1322587717:cpu0 0 0 1 0 0
1322587718:cpu0 1 0 1 0 0
Quoted 8 lines
1322587719:cpu0 0 0 1 0 0
1322587720:cpu0 0 0 0 0 0
1322587721:cpu0 2 0 1 0 0
1322587722:cpu0 1 0 0 0 0
1322587723:cpu0 0 0 3 0 0
1322587724:cpu0 1 0 0 0 0
1322587725:cpu0 0 0 0 0 0
1322587726:cpu0 0 0 1 0 0
OK, so the same thing as in another email in the thread (no idle/io_wait accounting).
Could you double check what kind of idle driver are you using? cat /sys/devices/system/cpu/cpuidle/current_driver
Signature · 6 lines
Michal Hocko SUSE Labs SUSE LINUX s.r.o. Lihovarska 1060/12 190 00 Praha 9 Czech Republic
Quoted 26 lines · Michal Hocko
Quoted 25 lines · Ian Kumlien
Quoted 24 lines · Michal Hocko
Quoted 5 lines · On Mon 28-11-11 23:28:03, [email protected] wrote:
Hi,
All this time i have been thinking i'm the only one - and i've been to loaded with work during working hours and tired when home =P
Quoted 8 lines
Anyways, I've neen seeing this since -rc1 on:
Intel(R) Core(TM) i5 CPU U520 @ 1.07GHz - x86-64 AMD Phenom(tm) II X6 1090T - x86-64
AMD Athlon(tm) 64 X2 Dual Core Processor 4400+ - x86-64 Intel(R) Core(TM)2 CPU T7200 @ 2.00GHz - i686Configs available on demand - I've been running the same config for quite some time though.
As I have written in other email could you post your config and collect the following data?
for i inseq 30;
docat /proc/stat > `date +'%s'` sleep 1done
export old_user=0 old_nice=0 old_sys=0 old_idle=0 old_iowait=0;
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
# for all your available CPUs grep cpu0 * | while read cpu user nice sys idle iowait rest; do echo $cpu $(($user-$old_user)) $(($nice-$old_nice)) $(($sys-$old_sys)) $(($idle-$old_idle)) $(($iowait-$old_iowait)) old_user=$user old_nice=$nice old_sys=$sys old_idle=$idle old_iowait=$iowait doneSorry, the previous one was AMD X2 4400+ and now the T7200:
--8<-- [data] --8<--
Quoted 4 lines
OK, so the same thing as in another email in the thread (no idle/io_wait accounting).
Could you double check what kind of idle driver are you using? cat/sys/devices/system/cpu/cpuidle/current_driver
"none", on both machines
Signature · 1 line
Ian Kumlien -- http://demius.net || http://pomac.netswarm.net