ksoftirqd using mysteriously high amounts of CPU time
Greetings, all
I'm running into an issue where ksoftirqd/0 consumes a considerably larger amount of CPU time than it should, and begins to actively consume 99% CPU time during network operations.
I've done my best to diagnose this, but it has followed me from kernel 2.4.20 to 2.6.3 and all versions in between, on three different server systems with distinct hardware configurations.
The latest system to be affected by this takes considerably longer to have trouble, but this is only due to its sheer muscle to begin with (2x MP 2800+), but does indeed appear. After 55 days of uptime:
5 root 34 19 0 0 0 S 0.0 0.0 0:21.40 ksoftirqd_CPU0 6 root 34 19 0 0 0 S 0.0 0.0 0:36.31 ksoftirqd_CPU1
Compare this to another machine which I don't run myself:
3 root 19 19 0 0 0 S 0.0 0.0 0:09.86 ksoftirqd_CPU0
There are peculiarities to my configuration which are shared on all the affected machines - the use of PPPoE to connect is one example. However on one of the affected systems a second network interface was subjected to heavy NFS traffic over simple ethernet and exhibited the same behavior. Another shared trait is a firewall ruleset I've written for the machines, which I can attach if someone would like to see them.
The three configurations include:
K6-2 500MHz
Ne2K, RealTek 8139
Duron 900MHz
SiS 900, RealTek 8139
2x Athlon MP 2800+
3com 3c905b 10/100+
The issue begins with ksoftirqd consuming more time than usual, and ends after four to twelve weeks depending on how powerful the machine, with the machine crippled and unable to sustain network traffic of any kind. Any assistance on where to look for solutions to this kind of problem would be GREATLY appreciated.
Thanks in advance!
Brad
Quoted 4 lines · Brad Laue
I'm running into an issue where ksoftirqd/0 consumes a considerably larger amount of CPU time than it should, and begins to actively consume 99% CPU time during network operations.
Please ensure that the machine was booted with `profile=1' on the kernel boot command line. The cost of this is negligible.
When the problem starts happening, run:
sudo readprofile -r
sleep 10
sudo readprofile -n -v -m /boot/System.map | sort -n +2 | tail -40
(make sure that /boot/System.map refers to the currently-running kernel)
Quoted 10 lines · Andrew Morton
Please ensure that the machine was booted with `profile=1' on the kernel boot command line. The cost of this is negligible.
When the problem starts happening, run:
sudo readprofile -r
sleep 10
sudo readprofile -n -v -m/boot/System.map| sort -n +2 | tail -40(make sure that
/boot/System.maprefers to the currently-running kernel)
Gave the machine a bit of a workout to create the problem sooner than usual, albeit on a smaller scale. ksoftirqd consumed about 4.7% of the CPU according to top after several concurrent wget operations, and had accumulated about 6 seconds of runtime, unheard of for a system uptime of 15 minutes; the output attached is during this.
Hopefully I'm right in assuming that; on all other systems I've experimented with ksoftirqd doesn't even flinch unless subjected to extremes.
Did some further testing with a machine of identical configuration (suffers the same issue too); removed iptables from the suspects list, as the issue exists sans those modules.
Hopefully the attached shows some irregularity. If not, I'll have to reply back in a few weeks when the problem recurs over the course of time.
Cheers,
Brad
Quoted 1 line · Brad Laue
Hopefully the attached shows some irregularity. If not, I'll have to reply back in a few weeks when the problem recurs over the course of time.
And without further ado, the attachment. It's been a long day. :)
Brad
c0122bf0 current_kernel_time 2 0.0312 c0122d20 local_bh_enable 2 0.0139 c0137800 unlock_page 2 0.0208 c013ef10 __kmalloc 2 0.0156 c0155630 mark_buffer_dirty 2 0.0250 c0165760 poll_freewait 2 0.0250 c01df2b0 xfs_ichgtime 2 0.0075 c0202780 linvfs_get_block_core 2 0.0027 c0205310 linvfs_write 2 0.0069 c0216e90 fast_copy_page 2 0.0078 c0244b90 tty_write 2 0.0030 c0246020 tty_poll 2 0.0156 c0248670 n_tty_receive_buf 2 0.0005 c024b1e0 pty_unthrottle 2 0.0208 c024b240 pty_write 2 0.0043 c0109354 system_call 3 0.0682 c0152e70 vfs_write 3 0.0099 c0157cd0 alloc_buffer_head 3 0.0312 c01657b0 __pollwait 3 0.0144 c016e100 dnotify_parent 3 0.0156 c01b1e50 xfs_bmapi 3 0.0006 c024a3a0 normal_poll 3 0.0088 c0107090 cpu_idle 4 0.0625 c0153e60 fget 4 0.0625 c0165c60 sys_select 4 0.0032 c0216e30 fast_clear_page 4 0.0417 c0139280 generic_file_aio_write_nolock 5 0.0019 c0209d60 xfs_write 6 0.0028 c011ba10 __wake_up 7 0.0729 c0165960 do_select 8 0.0111 c0249920 read_chan 9 0.0042 c0156170 __block_prepare_write 12 0.0117 c02173c0 __copy_from_user_ll 12 0.0938 c010b570 handle_IRQ_event 13 0.1161 c0187210 write_profile 14 0.2188 c011b3c0 schedule 15 0.0107 c0217350 __copy_to_user_ll 47 0.4196 c0122c80 do_softirq 136 0.8500
c0107030 default_idle 9222 192.1250
00000000 total 9611 0.0063
Quoted 6 lines · Brad Laue
Quoted 2 lines · Brad Laue
Hopefully the attached shows some irregularity. If not, I'll have to reply back in a few weeks when the problem recurs over the course of time.
And without further ado, the attachment. It's been a long day. :)
It beats me. Something must be waking up ksoftirqd all the time.
If you have time, could you please apply the below, then wait for ksoftirqd to go bad again and then run:
dmesg -c echo 1 > /proc/sys/debug/0 ; sleep 1; echo 0 > /proc/sys/debug/0 dmesg -s 1000000 > /tmp/foo
and then send foo?
include/linux/kernel.h renamed from include/linux/kernel.h~proc-sys-debug +2 -0
@@ -214,6 +214,8 @@ extern void dump_stack(void);1 ; \ }) +extern int proc_sys_debug[8 ]; +#endif /* __KERNEL__ */ #define SI_LOAD_SHIFT 16
diff -puN -L kernel/ksyms.c /dev/null /dev/null
kernel/sysctl.c renamed from kernel/sysctl.c~proc-sys-debug +19 -0
@@ -849,7 +849,26 @@ static ctl_table fs_table[] = { { .ctl_name = 0 } }; +int proc_sys_debug[8 ]; +EXPORT_SYMBOL (proc_sys_debug); +static ctl_table debug_table[] = { + {1 ,"0" , &proc_sys_debug[0 ],sizeof (int ),0644 ,NULL , + &proc_dointvec_minmax, &sysctl_intvec,NULL ,NULL ,NULL }, + {2 ,"1" , &proc_sys_debug[1 ],sizeof (int ),0644 ,NULL , + &proc_dointvec_minmax, &sysctl_intvec,NULL ,NULL ,NULL }, + {3 ,"2" , &proc_sys_debug[2 ],sizeof (int ),0644 ,NULL , + &proc_dointvec_minmax, &sysctl_intvec,NULL ,NULL ,NULL }, + {4 ,"3" , &proc_sys_debug[3 ],sizeof (int ),0644 ,NULL , + &proc_dointvec_minmax, &sysctl_intvec,NULL ,NULL ,NULL }, + {5 ,"4" , &proc_sys_debug[4 ],sizeof (int ),0644 ,NULL , + &proc_dointvec_minmax, &sysctl_intvec,NULL ,NULL ,NULL }, + {6 ,"5" , &proc_sys_debug[5 ],sizeof (int ),0644 ,NULL , + &proc_dointvec_minmax, &sysctl_intvec,NULL ,NULL ,NULL }, + {7 ,"6" , &proc_sys_debug[6 ],sizeof (int ),0644 ,NULL , + &proc_dointvec_minmax, &sysctl_intvec,NULL ,NULL ,NULL }, + {8 ,"7" , &proc_sys_debug[7 ],sizeof (int ),0644 ,NULL , + &proc_dointvec_minmax, &sysctl_intvec,NULL ,NULL ,NULL }, { .ctl_name =0 } };
_
kernel/softirq.c | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletion(-)
kernel/softirq.c renamed from kernel/softirq.c~a +6 -1
@@ -54,8 +54,13 @@ static inline void wakeup_softirqd(void)/* Interrupts are disabled: no need to stop preemption */ struct task_struct *tsk =__get_cpu_var (ksoftirqd); -if (tsk && tsk->state !=TASK_RUNNING ) +if (tsk && tsk->state !=TASK_RUNNING ) { +if (proc_sys_debug[0 ]) { +printk ("%s wakes ksoftirqd\n" , current->comm); +dump_stack (); + }wake_up_process (tsk); + } }/*
_
Quoted 9 lines · Andrew Morton
Quoted 6 lines · Brad Laue
Quoted 2 lines · Brad Laue
Hopefully the attached shows some irregularity. If not, I'll have to reply back in a few weeks when the problem recurs over the course of time.
And without further ado, the attachment. It's been a long day. :)
It beats me. Something must be waking up ksoftirqd all the time.
Hello,
I have impression, that it is somehow related to ACPI and CPU temperature. When CPU gets more hot ksoftirqd starts to eat 99% of CPU.
It may be checked by disabling ACPI (if enabled) and/or monitoring /proc/acpi/thermal_zone/THRM/temperature (if any).
Quoted 86 lines
If you have time, could you please apply the below, then wait for ksoftirqd to go bad again and then run:
dmesg -c echo 1 > /proc/sys/debug/0 ; sleep 1; echo 0 > /proc/sys/debug/0 dmesg -s 1000000 > /tmp/fooand then send foo?
diff -puN
include/linux/kernel.h~proc-sys-debuginclude/linux/kernel.hinclude/linux/kernel.h renamed from include/linux/kernel.h~proc-sys-debug +2
-0@@ -214,6 +214,8 @@ extern void dump_stack(void);1 ; \ }) +extern int proc_sys_debug[8 ]; +#endif /* __KERNEL__ */ #define SI_LOAD_SHIFT 16diff -puN -L
kernel/ksyms.c/dev/null/dev/nulldiff -puNkernel/sysctl.c~proc-sys-debugkernel/sysctl.ckernel/sysctl.c renamed from kernel/sysctl.c~proc-sys-debug +19
-0@@ -849,7 +849,26 @@ static ctl_table fs_table[] = { { .ctl_name = 0 } }; +int proc_sys_debug[8 ]; +EXPORT_SYMBOL (proc_sys_debug); +static ctl_table debug_table[] = { + {1 ,"0" , &proc_sys_debug[0 ],sizeof (int ),0644 ,NULL , + &proc_dointvec_minmax, &sysctl_intvec,NULL ,NULL ,NULL }, + {2 ,"1" , &proc_sys_debug[1 ],sizeof (int ),0644 ,NULL , + &proc_dointvec_minmax, &sysctl_intvec,NULL ,NULL ,NULL }, + {3 ,"2" , &proc_sys_debug[2 ],sizeof (int ),0644 ,NULL , + &proc_dointvec_minmax, &sysctl_intvec,NULL ,NULL ,NULL }, + {4 ,"3" , &proc_sys_debug[3 ],sizeof (int ),0644 ,NULL , + &proc_dointvec_minmax, &sysctl_intvec,NULL ,NULL ,NULL }, + {5 ,"4" , &proc_sys_debug[4 ],sizeof (int ),0644 ,NULL , + &proc_dointvec_minmax, &sysctl_intvec,NULL ,NULL ,NULL }, + {6 ,"5" , &proc_sys_debug[5 ],sizeof (int ),0644 ,NULL , + &proc_dointvec_minmax, &sysctl_intvec,NULL ,NULL ,NULL }, + {7 ,"6" , &proc_sys_debug[6 ],sizeof (int ),0644 ,NULL , + &proc_dointvec_minmax, &sysctl_intvec,NULL ,NULL ,NULL }, + {8 ,"7" , &proc_sys_debug[7 ],sizeof (int ),0644 ,NULL , + &proc_dointvec_minmax, &sysctl_intvec,NULL ,NULL ,NULL }, { .ctl_name =0 } };_
kernel/softirq.c | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletion(-)diff -puN
kernel/softirq.c~akernel/softirq.ckernel/softirq.c renamed from kernel/softirq.c~a +6
-1@@ -54,8 +54,13 @@ static inline void wakeup_softirqd(void)/* Interrupts are disabled: no need to stop preemption */ struct task_struct *tsk =__get_cpu_var (ksoftirqd); -if (tsk && tsk->state !=TASK_RUNNING ) +if (tsk && tsk->state !=TASK_RUNNING ) { +if (proc_sys_debug[0 ]) { +printk ("%s wakes ksoftirqd\n" , current->comm); +dump_stack (); + }wake_up_process (tsk); + } }/* _
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 1 line
umka
Quoted 7 lines · Yury V. Umanets
Hello,
I have impression, that it is somehow related to ACPI and CPU temperature. When CPU gets more hot ksoftirqd starts to eat 99% of CPU.
It may be checked by disabling ACPI (if enabled) and/or monitoring
/proc/acpi/thermal_zone/THRM/temperature(if any).
Happens on a system without ACPI or Power Management of any kind enabled though.
Brad
Quoted 6 lines · Brad Laue
Quoted 5 lines · Yury V. Umanets
Hello,
I have impression, that it is somehow related to ACPI and CPU temperature. When CPU gets more hot ksoftirqd starts to eat 99% of CPU.
Quoted 7 lines
Quoted 2 lines
It may be checked by disabling ACPI (if enabled) and/or monitoring
/proc/acpi/thermal_zone/THRM/temperature(if any).Happens on a system without ACPI or Power Management of any kind enabled though.
Brad
Then have you seen clear dependence of ksoftirqd getting crazy on system load? Or something else?
Quoted 5 lines
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 1 line
umka
Quoted 12 lines · Yury V. Umanets
Quoted 10 lines · Brad Laue
Quoted 7 lines · Yury V. Umanets
Hello,
I have impression, that it is somehow related to ACPI and CPU temperature. When CPU gets more hot ksoftirqd starts to eat 99% of CPU.
It may be checked by disabling ACPI (if enabled) and/or monitoring
/proc/acpi/thermal_zone/THRM/temperature(if any).
Quoted 6 lines
Quoted 3 lines
Happens on a system without ACPI or Power Management of any kind enabled though.
Brad
Then have you seen clear dependence of ksoftirqd getting crazy on system load? Or something else?
Not sure what you mean; when the problem begins to get bad, anything that tries to use a network interface causes ksoftirqd to jump to 99% CPU, and eventually it just stays there at idle. The process trying to use the network uses abnormal amounts of CPU time as well.
Brad
Quoted 14 lines · Andrew Morton
Quoted 4 lines · Brad Laue
Quoted 1 line · Brad Laue
Hopefully the attached shows some irregularity. If not, I'll have to reply back in a few weeks when the problem recurs over the course of time.
And without further ado, the attachment. It's been a long day. :)
It beats me. Something must be waking up ksoftirqd all the time.
If you have time, could you please apply the below, then wait for ksoftirqd to go bad again and then run:
Quoted 5 lines
dmesg -c echo 1 > /proc/sys/debug/0 ; sleep 1; echo 0 > /proc/sys/debug/0 dmesg -s 1000000 > /tmp/fooand then send foo?
Will do. I think the profile output I generated was a bit premature. I'll wait a few days until the problem resurfaces in earnest and attach some more output then.
For now, it looks like the 'pppoe' process itself is waking up ksoftirqd the most according to the output of dmesg, but that may change with time.
Will keep you posted.
Brad
Quoted 14 lines · Andrew Morton
Quoted 4 lines · Brad Laue
Quoted 1 line · Brad Laue
Hopefully the attached shows some irregularity. If not, I'll have to reply back in a few weeks when the problem recurs over the course of time.
And without further ado, the attachment. It's been a long day. :)
It beats me. Something must be waking up ksoftirqd all the time.
If you have time, could you please apply the below, then wait for ksoftirqd to go bad again and then run:
Quoted 6 lines
dmesg -c echo 1 > /proc/sys/debug/0 ; sleep 1; echo 0 > /proc/sys/debug/0 dmesg -s 1000000 > /tmp/fooand then send foo?
Okay, it looks like the 'pppoe' process is definitely causing all the soft interrupts. Puzzling, because on other systems with similar hardware this doesn't happen. Ran the machine without iptables running (eek!!!) just to make sure, and the same result occurred, so it doesn't seem to be the fact that packets are going through an elaborate ruleset that's causing it. Kernel mode PPPoE doesn't seem to change things much.
Hard to figure out from a kernel perspective why this many soft interrupts are being used, but I'm just getting a grasp on what their purpose is to begin with.
Hope this offers some insight.
Brad
ction+0x0/0x20 [<c011a910>] default_wake_function+0x0/0x20 [<c0243d38>] tty_write+0x1e8/0x290 [<c0249140>] write_chan+0x0/0x220 [<c0243b50>] tty_write+0x0/0x290 [<c0151fa8>] vfs_write+0xb8/0x130 [<c01520d2>] sys_write+0x42/0x70 [<c0109387>] syscall_call+0x7/0xb
pppoe wakes ksoftirqd
Call Trace: [<c0121e6c>] __tasklet_schedule+0x7c/0x80 [<d084656c>] ppp_asynctty_receive+0xbc/0xd0 [ppp_async] [<c024a342>] pty_write+0x142/0x1d0 [<c0249333>] write_chan+0x1f3/0x220 [<c011a910>] default_wake_function+0x0/0x20 [<c011a910>] default_wake_function+0x0/0x20 [<c0243d38>] tty_write+0x1e8/0x290 [<c0249140>] write_chan+0x0/0x220 [<c0243b50>] tty_write+0x0/0x290 [<c0151fa8>] vfs_write+0xb8/0x130 [<c01520d2>] sys_write+0x42/0x70 [<c0109387>] syscall_call+0x7/0xb
pppoe wakes ksoftirqd
Call Trace: [<c0121e6c>] __tasklet_schedule+0x7c/0x80 [<d084656c>] ppp_asynctty_receive+0xbc/0xd0 [ppp_async] [<c024a342>] pty_write+0x142/0x1d0 [<c0249333>] write_chan+0x1f3/0x220 [<c011a910>] default_wake_function+0x0/0x20 [<c011a910>] default_wake_function+0x0/0x20 [<c0243d38>] tty_write+0x1e8/0x290 [<c0249140>] write_chan+0x0/0x220 [<c0243b50>] tty_write+0x0/0x290 [<c0151fa8>] vfs_write+0xb8/0x130 [<c01520d2>] sys_write+0x42/0x70 [<c0109387>] syscall_call+0x7/0xb
pppoe wakes ksoftirqd
Call Trace: [<c0121e6c>] __tasklet_schedule+0x7c/0x80 [<d084656c>] ppp_asynctty_receive+0xbc/0xd0 [ppp_async] [<c024a342>] pty_write+0x142/0x1d0 [<c0249333>] write_chan+0x1f3/0x220 [<c011a910>] default_wake_function+0x0/0x20 [<c011a910>] default_wake_function+0x0/0x20 [<c0243d38>] tty_write+0x1e8/0x290 [<c0249140>] write_chan+0x0/0x220 [<c0243b50>] tty_write+0x0/0x290 [<c0151fa8>] vfs_write+0xb8/0x130 [<c01520d2>] sys_write+0x42/0x70 [<c0109387>] syscall_call+0x7/0xb
pppoe wakes ksoftirqd
Call Trace: [<c0121e6c>] __tasklet_schedule+0x7c/0x80 [<d084656c>] ppp_asynctty_receive+0xbc/0xd0 [ppp_async] [<c024a342>] pty_write+0x142/0x1d0 [<c0249333>] write_chan+0x1f3/0x220 [<c011a910>] default_wake_function+0x0/0x20 [<c011a910>] default_wake_function+0x0/0x20 [<c0243d38>] tty_write+0x1e8/0x290 [<c0249140>] write_chan+0x0/0x220 [<c0243b50>] tty_write+0x0/0x290 [<c0151fa8>] vfs_write+0xb8/0x130 [<c01520d2>] sys_write+0x42/0x70 [<c0109387>] syscall_call+0x7/0xb
pppoe wakes ksoftirqd
Call Trace: [<c0121e6c>] __tasklet_schedule+0x7c/0x80 [<d084656c>] ppp_asynctty_receive+0xbc/0xd0 [ppp_async] [<c024a342>] pty_write+0x142/0x1d0 [<c0249333>] write_chan+0x1f3/0x220 [<c011a910>] default_wake_function+0x0/0x20 [<c011a910>] default_wake_function+0x0/0x20 [<c0243d38>] tty_write+0x1e8/0x290 [<c0249140>] write_chan+0x0/0x220 [<c02978b3>] net_tx_action+0x83/0xd0 [<c0243b50>] tty_write+0x0/0x290 [<c0151fa8>] vfs_write+0xb8/0x130 [<c01520d2>] sys_write+0x42/0x70 [<c0109387>] syscall_call+0x7/0xb
pppoe wakes ksoftirqd
Call Trace: [<c0121e6c>] __tasklet_schedule+0x7c/0x80 [<d084656c>] ppp_asynctty_receive+0xbc/0xd0 [ppp_async] [<c024a342>] pty_write+0x142/0x1d0 [<c0249333>] write_chan+0x1f3/0x220 [<c011a910>] default_wake_function+0x0/0x20 [<c011a910>] default_wake_function+0x0/0x20 [<c0243d38>] tty_write+0x1e8/0x290 [<c0249140>] write_chan+0x0/0x220 [<c0243b50>] tty_write+0x0/0x290 [<c0151fa8>] vfs_write+0xb8/0x130 [<c01520d2>] sys_write+0x42/0x70 [<c0109387>] syscall_call+0x7/0xb
pppoe wakes ksoftirqd
Call Trace: [<c0121e6c>] __tasklet_schedule+0x7c/0x80 [<d084656c>] ppp_asynctty_receive+0xbc/0xd0 [ppp_async] [<c024a342>] pty_write+0x142/0x1d0 [<c0249333>] write_chan+0x1f3/0x220 [<c011a910>] default_wake_function+0x0/0x20 [<c011a910>] default_wake_function+0x0/0x20 [<c0243d38>] tty_write+0x1e8/0x290 [<c0249140>] write_chan+0x0/0x220 [<c0243b50>] tty_write+0x0/0x290 [<c0151fa8>] vfs_write+0xb8/0x130 [<c01520d2>] sys_write+0x42/0x70 [<c0109387>] syscall_call+0x7/0xb
pppoe wakes ksoftirqd
Call Trace: [<c0121e6c>] __tasklet_schedule+0x7c/0x80 [<d084656c>] ppp_asynctty_receive+0xbc/0xd0 [ppp_async] [<c024a342>] pty_write+0x142/0x1d0 [<c0249333>] write_chan+0x1f3/0x220 [<c011a910>] default_wake_function+0x0/0x20 [<c011a910>] default_wake_function+0x0/0x20 [<c0243d38>] tty_write+0x1e8/0x290 [<c0249140>] write_chan+0x0/0x220 [<c0243b50>] tty_write+0x0/0x290 [<c0151fa8>] vfs_write+0xb8/0x130 [<c01520d2>] sys_write+0x42/0x70 [<c0109387>] syscall_call+0x7/0xb
pppoe wakes ksoftirqd
Call Trace: [<c0121e6c>] __tasklet_schedule+0x7c/0x80 [<d084656c>] ppp_asynctty_receive+0xbc/0xd0 [ppp_async] [<c024a342>] pty_write+0x142/0x1d0 [<c0249333>] write_chan+0x1f3/0x220 [<c011a910>] default_wake_function+0x0/0x20 [<c011a910>] default_wake_function+0x0/0x20 [<c0243d38>] tty_write+0x1e8/0x290 [<c0249140>] write_chan+0x0/0x220 [<c02978b3>] net_tx_action+0x83/0xd0 [<c0243b50>] tty_write+0x0/0x290 [<c0151fa8>] vfs_write+0xb8/0x130 [<c01520d2>] sys_write+0x42/0x70 [<c0109387>] syscall_call+0x7/0xb
pppoe wakes ksoftirqd
Call Trace: [<c0121e6c>] __tasklet_schedule+0x7c/0x80 [<d084656c>] ppp_asynctty_receive+0xbc/0xd0 [ppp_async] [<c024a342>] pty_write+0x142/0x1d0 [<c0249333>] write_chan+0x1f3/0x220 [<c011a910>] default_wake_function+0x0/0x20 [<c011a910>] default_wake_function+0x0/0x20 [<c0243d38>] tty_write+0x1e8/0x290 [<c0249140>] write_chan+0x0/0x220 [<c0243b50>] tty_write+0x0/0x290 [<c0151fa8>] vfs_write+0xb8/0x130 [<c01520d2>] sys_write+0x42/0x70 [<c0109387>] syscall_call+0x7/0xb
pppoe wakes ksoftirqd
Call Trace: [<c0121e6c>] __tasklet_schedule+0x7c/0x80 [<d084656c>] ppp_asynctty_receive+0xbc/0xd0 [ppp_async] [<c024a342>] pty_write+0x142/0x1d0 [<c0249333>] write_chan+0x1f3/0x220 [<c011a910>] default_wake_function+0x0/0x20 [<c011a910>] default_wake_function+0x0/0x20 [<c0243d38>] tty_write+0x1e8/0x290 [<c0249140>] write_chan+0x0/0x220 [<c0243b50>] tty_write+0x0/0x290 [<c0151fa8>] vfs_write+0xb8/0x130 [<c01520d2>] sys_write+0x42/0x70 [<c0109387>] syscall_call+0x7/0xb
pppoe wakes ksoftirqd
Call Trace: [<c0121e6c>] __tasklet_schedule+0x7c/0x80 [<d084656c>] ppp_asynctty_receive+0xbc/0xd0 [ppp_async] [<c024a342>] pty_write+0x142/0x1d0 [<c0249333>] write_chan+0x1f3/0x220 [<c011a910>] default_wake_function+0x0/0x20 [<c011a910>] default_wake_function+0x0/0x20 [<c0243d38>] tty_write+0x1e8/0x290 [<c0249140>] write_chan+0x0/0x220 [<c0243b50>] tty_write+0x0/0x290 [<c0151fa8>] vfs_write+0xb8/0x130 [<c01520d2>] sys_write+0x42/0x70 [<c0109387>] syscall_call+0x7/0xb
pppoe wakes ksoftirqd
Call Trace: [<c0121e6c>] __tasklet_schedule+0x7c/0x80 [<d084656c>] ppp_asynctty_receive+0xbc/0xd0 [ppp_async] [<c024a342>] pty_write+0x142/0x1d0 [<c0249333>] write_chan+0x1f3/0x220 [<c011a910>] default_wake_function+0x0/0x20 [<c011a910>] default_wake_function+0x0/0x20 [<c0243d38>] tty_write+0x1e8/0x290 [<c0249140>] write_chan+0x0/0x220 [<c0243b50>] tty_write+0x0/0x290 [<c0151fa8>] vfs_write+0xb8/0x130 [<c01520d2>] sys_write+0x42/0x70 [<c0109387>] syscall_call+0x7/0xb
pppoe wakes ksoftirqd
Call Trace: [<c0121e6c>] __tasklet_schedule+0x7c/0x80 [<d084656c>] ppp_asynctty_receive+0xbc/0xd0 [ppp_async] [<c024a342>] pty_write+0x142/0x1d0 [<c0249333>] write_chan+0x1f3/0x220 [<c011a910>] default_wake_function+0x0/0x20 [<c011a910>] default_wake_function+0x0/0x20 [<c0243d38>] tty_write+0x1e8/0x290 [<c0249140>] write_chan+0x0/0x220 [<c0243b50>] tty_write+0x0/0x290 [<c0151fa8>] vfs_write+0xb8/0x130 [<c01520d2>] sys_write+0x42/0x70 [<c0109387>] syscall_call+0x7/0xb
pppoe wakes ksoftirqd
Call Trace: [<c0121e6c>] __tasklet_schedule+0x7c/0x80 [<d084656c>] ppp_asynctty_receive+0xbc/0xd0 [ppp_async] [<c024a342>] pty_write+0x142/0x1d0 [<c0249333>] write_chan+0x1f3/0x220 [<c011a910>] default_wake_function+0x0/0x20 [<c011a910>] default_wake_function+0x0/0x20 [<c0243d38>] tty_write+0x1e8/0x290 [<c0249140>] write_chan+0x0/0x220 [<c0243b50>] tty_write+0x0/0x290 [<c0151fa8>] vfs_write+0xb8/0x130 [<c01520d2>] sys_write+0x42/0x70 [<c0109387>] syscall_call+0x7/0xb
pppoe wakes ksoftirqd
Call Trace: [<c0121e6c>] __tasklet_schedule+0x7c/0x80 [<d084656c>] ppp_asynctty_receive+0xbc/0xd0 [ppp_async] [<c024a342>] pty_write+0x142/0x1d0 [<c0249333>] write_chan+0x1f3/0x220 [<c011a910>] default_wake_function+0x0/0x20 [<c011a910>] default_wake_function+0x0/0x20 [<c0243d38>] tty_write+0x1e8/0x290 [<c0249140>] write_chan+0x0/0x220 [<c0243b50>] tty_write+0x0/0x290 [<c0151fa8>] vfs_write+0xb8/0x130 [<c01520d2>] sys_write+0x42/0x70 [<c0109387>] syscall_call+0x7/0xb
pppoe wakes ksoftirqd
Call Trace: [<c0121e6c>] __tasklet_schedule+0x7c/0x80 [<d084656c>] ppp_asynctty_receive+0xbc/0xd0 [ppp_async] [<c024a342>] pty_write+0x142/0x1d0 [<c0249333>] write_chan+0x1f3/0x220 [<c011a910>] default_wake_function+0x0/0x20 [<c011a910>] default_wake_function+0x0/0x20 [<c0243d38>] tty_write+0x1e8/0x290 [<c0249140>] write_chan+0x0/0x220 [<c0243b50>] tty_write+0x0/0x290 [<c0151fa8>] vfs_write+0xb8/0x130 [<c01520d2>] sys_write+0x42/0x70 [<c0109387>] syscall_call+0x7/0xb
pppoe wakes ksoftirqd
Call Trace: [<c0121e6c>] __tasklet_schedule+0x7c/0x80 [<d084656c>] ppp_asynctty_receive+0xbc/0xd0 [ppp_async] [<c024a342>] pty_write+0x142/0x1d0 [<c0249333>] write_chan+0x1f3/0x220 [<c011a910>] default_wake_function+0x0/0x20 [<c011a910>] default_wake_function+0x0/0x20 [<c0243d38>] tty_write+0x1e8/0x290 [<c0249140>] write_chan+0x0/0x220 [<c0243b50>] tty_write+0x0/0x290 [<c0151fa8>] vfs_write+0xb8/0x130 [<c01520d2>] sys_write+0x42/0x70 [<c0109387>] syscall_call+0x7/0xb
pppoe wakes ksoftirqd
Call Trace: [<c0121e6c>] __tasklet_schedule+0x7c/0x80 [<d084656c>] ppp_asynctty_receive+0xbc/0xd0 [ppp_async] [<c024a342>] pty_write+0x142/0x1d0 [<c0249333>] write_chan+0x1f3/0x220 [<c011a910>] default_wake_function+0x0/0x20 [<c011a910>] default_wake_function+0x0/0x20 [<c0243d38>] tty_write+0x1e8/0x290 [<c0249140>] write_chan+0x0/0x220 [<c0243b50>] tty_write+0x0/0x290 [<c0151fa8>] vfs_write+0xb8/0x130 [<c01520d2>] sys_write+0x42/0x70 [<c0109387>] syscall_call+0x7/0xb
pppoe wakes ksoftirqd
Call Trace: [<c0121e6c>] __tasklet_schedule+0x7c/0x80 [<d084656c>] ppp_asynctty_receive+0xbc/0xd0 [ppp_async] [<c024a342>] pty_write+0x142/0x1d0 [<c0249333>] write_chan+0x1f3/0x220 [<c011a910>] default_wake_function+0x0/0x20 [<c011a910>] default_wake_function+0x0/0x20 [<c0243d38>] tty_write+0x1e8/0x290 [<c0249140>] write_chan+0x0/0x220 [<c0243b50>] tty_write+0x0/0x290 [<c0151fa8>] vfs_write+0xb8/0x130 [<c01520d2>] sys_write+0x42/0x70 [<c0109387>] syscall_call+0x7/0xb
pppoe wakes ksoftirqd
Call Trace: [<c0121e6c>] __tasklet_schedule+0x7c/0x80 [<d084656c>] ppp_asynctty_receive+0xbc/0xd0 [ppp_async] [<c024a342>] pty_write+0x142/0x1d0 [<c0249333>] write_chan+0x1f3/0x220 [<c011a910>] default_wake_function+0x0/0x20 [<c011a910>] default_wake_function+0x0/0x20 [<c0243d38>] tty_write+0x1e8/0x290 [<c0249140>] write_chan+0x0/0x220 [<c0243b50>] tty_write+0x0/0x290 [<c0151fa8>] vfs_write+0xb8/0x130 [<c01520d2>] sys_write+0x42/0x70 [<c0109387>] syscall_call+0x7/0xb
pppoe wakes ksoftirqd
Call Trace: [<c0121e6c>] __tasklet_schedule+0x7c/0x80 [<d084656c>] ppp_asynctty_receive+0xbc/0xd0 [ppp_async] [<c024a342>] pty_write+0x142/0x1d0 [<c0249333>] write_chan+0x1f3/0x220 [<c011a910>] default_wake_function+0x0/0x20 [<c011a910>] default_wake_function+0x0/0x20 [<c0243d38>] tty_write+0x1e8/0x290 [<c0249140>] write_chan+0x0/0x220 [<c0243b50>] tty_write+0x0/0x290 [<c0151fa8>] vfs_write+0xb8/0x130 [<c01520d2>] sys_write+0x42/0x70 [<c0109387>] syscall_call+0x7/0xb
pppoe wakes ksoftirqd
Call Trace: [<c0121e6c>] __tasklet_schedule+0x7c/0x80 [<d084656c>] ppp_asynctty_receive+0xbc/0xd0 [ppp_async] [<c024a342>] pty_write+0x142/0x1d0 [<c0249333>] write_chan+0x1f3/0x220 [<c011a910>] default_wake_function+0x0/0x20 [<c011a910>] default_wake_function+0x0/0x20 [<c0243d38>] tty_write+0x1e8/0x290 [<c0249140>] write_chan+0x0/0x220 [<c0243b50>] tty_write+0x0/0x290 [<c0151fa8>] vfs_write+0xb8/0x130 [<c01520d2>] sys_write+0x42/0x70 [<c0109387>] syscall_call+0x7/0xb
pppoe wakes ksoftirqd
Call Trace: [<c0121e6c>] __tasklet_schedule+0x7c/0x80 [<d084656c>] ppp_asynctty_receive+0xbc/0xd0 [ppp_async] [<c024a342>] pty_write+0x142/0x1d0 [<c0249333>] write_chan+0x1f3/0x220 [<c011a910>] default_wake_function+0x0/0x20 [<c011a910>] default_wake_function+0x0/0x20 [<c0243d38>] tty_write+0x1e8/0x290 [<c0249140>] write_chan+0x0/0x220 [<c0243b50>] tty_write+0x0/0x290 [<c0151fa8>] vfs_write+0xb8/0x130 [<c01520d2>] sys_write+0x42/0x70 [<c0109387>] syscall_call+0x7/0xb
pppoe wakes ksoftirqd
Call Trace: [<c0121e6c>] __tasklet_schedule+0x7c/0x80 [<d084656c>] ppp_asynctty_receive+0xbc/0xd0 [ppp_async] [<c024a342>] pty_write+0x142/0x1d0 [<c0249333>] write_chan+0x1f3/0x220 [<c011a910>] default_wake_function+0x0/0x20 [<c011a910>] default_wake_function+0x0/0x20 [<c0243d38>] tty_write+0x1e8/0x290 [<c0249140>] write_chan+0x0/0x220 [<c0243b50>] tty_write+0x0/0x290 [<c0151fa8>] vfs_write+0xb8/0x130 [<c01520d2>] sys_write+0x42/0x70 [<c0109387>] syscall_call+0x7/0xb
pppoe wakes ksoftirqd
Call Trace: [<c0121e6c>] __tasklet_schedule+0x7c/0x80 [<d084656c>] ppp_asynctty_receive+0xbc/0xd0 [ppp_async] [<c024a342>] pty_write+0x142/0x1d0 [<c0249333>] write_chan+0x1f3/0x220 [<c011a910>] default_wake_function+0x0/0x20 [<c011a910>] default_wake_function+0x0/0x20 [<c0243d38>] tty_write+0x1e8/0x290 [<c0249140>] write_chan+0x0/0x220 [<c0243b50>] tty_write+0x0/0x290 [<c0151fa8>] vfs_write+0xb8/0x130 [<c01520d2>] sys_write+0x42/0x70 [<c0109387>] syscall_call+0x7/0xb
pppoe wakes ksoftirqd
Call Trace: [<c0121e6c>] __tasklet_schedule+0x7c/0x80 [<d084656c>] ppp_asynctty_receive+0xbc/0xd0 [ppp_async] [<c024a342>] pty_write+0x142/0x1d0 [<c0249333>] write_chan+0x1f3/0x220 [<c011a910>] default_wake_function+0x0/0x20 [<c011a910>] default_wake_function+0x0/0x20 [<c0243d38>] tty_write+0x1e8/0x290 [<c0249140>] write_chan+0x0/0x220 [<c0243b50>] tty_write+0x0/0x290 [<c0151fa8>] vfs_write+0xb8/0x130 [<c01520d2>] sys_write+0x42/0x70 [<c0109387>] syscall_call+0x7/0xb
pppoe wakes ksoftirqd
Call Trace: [<c0121e6c>] __tasklet_schedule+0x7c/0x80 [<d084656c>] ppp_asynctty_receive+0xbc/0xd0 [ppp_async] [<c024a342>] pty_write+0x142/0x1d0 [<c0249333>] write_chan+0x1f3/0x220 [<c011a910>] default_wake_function+0x0/0x20 [<c011a910>] default_wake_function+0x0/0x20 [<c0243d38>] tty_write+0x1e8/0x290 [<c0249140>] write_chan+0x0/0x220 [<c0243b50>] tty_write+0x0/0x290 [<c0151fa8>] vfs_write+0xb8/0x130 [<c01520d2>] sys_write+0x42/0x70 [<c0109387>] syscall_call+0x7/0xb
pppoe wakes ksoftirqd
Call Trace: [<c0121e6c>] __tasklet_schedule+0x7c/0x80 [<d084656c>] ppp_asynctty_receive+0xbc/0xd0 [ppp_async] [<c024a342>] pty_write+0x142/0x1d0 [<c0249333>] write_chan+0x1f3/0x220 [<c011a910>] default_wake_function+0x0/0x20 [<c011a910>] default_wake_function+0x0/0x20 [<c0243d38>] tty_write+0x1e8/0x290 [<c0249140>] write_chan+0x0/0x220 [<c0243b50>] tty_write+0x0/0x290 [<c0151fa8>] vfs_write+0xb8/0x130 [<c01520d2>] sys_write+0x42/0x70 [<c0109387>] syscall_call+0x7/0xb
c013df00 kmem_cache_alloc 2 0.0312 c013df90 __kmalloc 2 0.0156 c01d8900 xfs_ilock 2 0.0114 c0204390 linvfs_write 2 0.0069 c02475f0 n_tty_write_wakeup 2 0.0312 c0247630 n_tty_receive_buf 2 0.0005 c024a200 pty_write 2 0.0043 c0121cc0 local_bh_enable 3 0.0208 c016d180 dnotify_parent 3 0.0156 c0215eb0 fast_clear_page 3 0.0312 c0109354 system_call 4 0.0909 c0138300 generic_file_aio_write_nolock 4 0.0015 c0151e30 do_sync_write 4 0.0208 c0151ef0 vfs_write 4 0.0132 c0155e10 generic_commit_write 4 0.0227 c0164ce0 sys_select 4 0.0032 c016bde0 inode_update_time 4 0.0192 c0243a00 tty_read 4 0.0119 c0243b50 tty_write 4 0.0061 c0126040 run_timer_softirq 5 0.0116 c013a580 free_hot_cold_page 5 0.0195 c0164900 max_select_fd 5 0.0223 c0215974 csum_partial 5 0.0174 c02488e0 read_chan 5 0.0023 c0152ee0 fget 6 0.0938 c01551f0 __block_prepare_write 6 0.0059 c01649e0 do_select 6 0.0083 c0213930 radix_tree_lookup 6 0.0750 c011a990 __wake_up 8 0.0833 c013e060 kfree 9 0.0804 c01b0ed0 xfs_bmapi 9 0.0018 c01862a0 write_profile 10 0.1562 c0208de0 xfs_write 10 0.0047 c0216440 __copy_from_user_ll 17 0.1328 c011a340 schedule 18 0.0128 c010ad50 handle_IRQ_event 21 0.1875 c02163d0 __copy_to_user_ll 27 0.2411 c0121c00 do_softirq 144 0.7500
c0107030 default_idle 8696 181.1667
00000000 total 9144 0.0060
Quoted 47 lines · Brad Laue
Quoted 2 lines · Brad Laue
Hopefully the attached shows some irregularity. If not, I'll have to reply back in a few weeks when the problem recurs over the course of time.
And without further ado, the attachment. It's been a long day. :)
Brad
______________________________________________________________________
c0122bf0 current_kernel_time 2 0.0312 c0122d20 local_bh_enable 2 0.0139 c0137800 unlock_page 2 0.0208 c013ef10 __kmalloc 2 0.0156 c0155630 mark_buffer_dirty 2 0.0250 c0165760 poll_freewait 2 0.0250 c01df2b0 xfs_ichgtime 2 0.0075 c0202780 linvfs_get_block_core 2 0.0027 c0205310 linvfs_write 2 0.0069 c0216e90 fast_copy_page 2 0.0078 c0244b90 tty_write 2 0.0030 c0246020 tty_poll 2 0.0156 c0248670 n_tty_receive_buf 2 0.0005 c024b1e0 pty_unthrottle 2 0.0208 c024b240 pty_write 2 0.0043 c0109354 system_call 3 0.0682 c0152e70 vfs_write 3 0.0099 c0157cd0 alloc_buffer_head 3 0.0312 c01657b0 __pollwait 3 0.0144 c016e100 dnotify_parent 3 0.0156 c01b1e50 xfs_bmapi 3 0.0006 c024a3a0 normal_poll 3 0.0088 c0107090 cpu_idle 4 0.0625 c0153e60 fget 4 0.0625 c0165c60 sys_select 4 0.0032 c0216e30 fast_clear_page 4 0.0417 c0139280 generic_file_aio_write_nolock 5 0.0019 c0209d60 xfs_write 6 0.0028 c011ba10 __wake_up 7 0.0729 c0165960 do_select 8 0.0111 c0249920 read_chan 9 0.0042 c0156170 __block_prepare_write 12 0.0117 c02173c0 __copy_from_user_ll 12 0.0938 c010b570 handle_IRQ_event 13 0.1161 c0187210 write_profile 14 0.2188 c011b3c0 schedule 15 0.0107 c0217350 __copy_to_user_ll 47 0.4196 c0122c80 do_softirq 136 0.8500
Quoted 2 lines
c0107030default_idle9222 192.1250
00000000 total
Hello,
IMHO this does not look like profile results related to the problem. Or it is inside big critical section with disabled irqs and profiler has no hits there.
I guess it might be useful to try SysRq to catch the problem.
Do not forget to enable it in kernel config and run
echo "1" > /proc/sys/kernel/sysrq
then press Alt-SysRq-t when ksoftirqd starts to eat cpu time
This will dump a list of current tasks and their information to console. Probably there will be some hint...
Quoted 1 line
9611 0.0063
Signature · 1 line
umka