New: Kernel locks up after a few minutes of heavy surfing
Subsystems (3)
Begin forwarded message:
Date: Sun, 6 Dec 2009 13:40:19 GMT
From: [email protected]
To: [email protected]
Subject: [Bug 14749] New: Kernel locks up after a few minutes of heavy surfing
http://bugzilla.kernel.org/show_bug.cgi?id=14749
Summary: Kernel locks up after a few minutes of heavy surfing
Product: Networking
Version: 2.5
Kernel Version: 2.6.31.6
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: high
Priority: P1
Component: IPV4
AssignedTo: [email protected]
ReportedBy: [email protected]
Regression: YesCreated an attachment (id=24049)
--> (http://bugzilla.kernel.org/attachment.cgi?id=24049)
Warnings found in kernel, relating to network corruption.
This bug is new as of 2.6.31.x kernels. After a short period of heavy surfing (e.g. lots of tabs open in Firefox), the kernel will suddenly stop responding. Nothing is written to the serial console, and the machine stops responding to pings. My only clue so far has been a warning which I found once in my dmesg log (attached).
I have already tried manually applying this patch from the upcoming -stable queue:
net-fix-sk_forward_alloc-corruption.patch
to no effect.
I am currently switching back to Fedora's 2.6.31.6-145.fc12.i686 kernel to see if it is more stable. (I cannot trust 2.6.31.6 any more.)
Signature · 6 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug. --
Quoted 41 lines · On Mon, 7 Dec 2009 09:01:54 -0800 Stephen Hemminger <[email protected]>…
Begin forwarded message:
Date: Sun, 6 Dec 2009 13:40:19 GMT
From: [email protected]
To: [email protected]
Subject: [Bug 14749] New: Kernel locks up after a few minutes of heavy surfinghttp://bugzilla.kernel.org/show_bug.cgi?id=14749
Summary: Kernel locks up after a few minutes of heavy surfing Product: Networking Version: 2.5 Kernel Version: 2.6.31.6 Platform: All OS/Version: Linux Tree: Mainline Status: NEW Severity: high Priority: P1 Component: IPV4 AssignedTo: [email protected] ReportedBy: [email protected] Regression: YesCreated an attachment (id=24049)
--> (http://bugzilla.kernel.org/attachment.cgi?id=24049)
Warnings found in kernel, relating to network corruption.This bug is new as of 2.6.31.x kernels. After a short period of heavy surfing (e.g. lots of tabs open in Firefox), the kernel will suddenly stop responding. Nothing is written to the serial console, and the machine stops responding to pings. My only clue so far has been a warning which I found once in my dmesg log (attached).
I have already tried manually applying this patch from the upcoming -stable queue:
Quoted 7 lines
net-fix-sk_forward_alloc-corruption.patch
to no effect.
I am currently switching back to Fedora's 2.6.31.6-145.fc12.i686 kernel to see if it is more stable. (I cannot trust 2.6.31.6 any more.)
Putting attachment inline since then developers are more likely to read it
-----------[ cut here ]------------
WARNING: at /home/chris/LINUX/linux-2.6.31/net/core/stream.c:202 inet_csk_destroy_sock+0x77/0xd3() Hardware name: Precision WorkStation 650 Modules linked in: tun snd_seq_oss snd_seq_midi snd_seq_dummy fuse nfsd lockd auth_rpcgss exportfs sunrpc autofs4 af_packet ipt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_LOG nf_conntrack_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables x_tables ipv6 p4_clockmod speedstep_lib binfmt_misc dm_mirror dm_region_hash dm_log dm_mod uinput snd_emu10k1_synth snd_emux_synth snd_seq_virmidi snd_seq_midi_event snd_seq_midi_emul snd_emu10k1 snd_ac97_codec snd_usb_audio ac97_bus snd_seq snd_pcm snd_usb_lib snd_rawmidi snd_seq_device snd_timer firewire_ohci ppdev uvcvideo floppy firewire_core snd_page_alloc snd_util_mem snd_hwdep parport_pc pwc psmouse videodev parport v4l1_compat crc_itu_t pcspkr snd sg i2c_i801 serio_raw soundcore dcdbas ext3 jbd mbcache sr_mod cdrom sd_mo d pata_acpi sata_sil uhci_hcd ata_piix libata scsi_mod ehci_hcd e1000 usbcore thermal button radeon intel_agp ttm drm agpgart i2c_algo_bit cfbcopyarea cfbimgblt cfbfillrect [last unloaded: processor] Pid: 32056, comm: rpm Not tainted 2.6.31.6 #1 Call Trace: [<c1023ba8>] ? warn_slowpath_common+0x5d/0x70 [<c1023bc6>] ? warn_slowpath_null+0xb/0xd [<c11871ca>] ? inet_csk_destroy_sock+0x77/0xd3 [<c119188f>] ? tcp_rcv_state_process+0x81f/0x9e8 [<c11966c3>] ? tcp_v4_do_rcv+0x128/0x16d [<c1196b0d>] ? tcp_v4_rcv+0x405/0x640 [<c118003e>] ? ip_local_deliver_finish+0xf3/0x1ab [<c117fcd9>] ? ip_rcv_finish+0x2a9/0x2cf [<c117fa30>] ? ip_rcv_finish+0x0/0x2cf [<c116b7c5>] ? netif_receive_skb+0x261/0x281 [<f8527bfc>] ? e1000_clean_rx_irq+0x31c/0x3c3 [e1000] [<f852a6fa>] ? e1000_clean+0x2a7/0x3f5 [e1000] [<c11c783c>] ? _spin_unlock_irqrestore+0xe/0x21 [<c10354c0>] ? hrtimer_run_pending+0xd/0xa5 [<c11c769b>] ? _spin_lock_irq+0xe/0x24 [<c116bce5>] ? net_rx_action+0x57/0xfd [<c1027ea3>] ? __do_softirq+0x7a/0xe3 [<c1027e29>] ? __do_softirq+0x0/0xe3 <IRQ> [<c1027c3c>] ? irq_exit+0x29/0x63 [<c1004320>] ? do_IRQ+0x7c/0x8d [<c1002f29>] ? common_interrupt+0x29/0x30 ---[ end trace e643d9455a26ccf3 ]--- ------------[ cut here ]------------ WARNING: at /home/chris/LINUX/linux-2.6.31/net/ipv4/af_inet.c:151 inet_sock_destruct+0xd8/0x138() Hardware name: Precision WorkStation 650 Modules linked in: tun snd_seq_oss snd_seq_midi snd_seq_dummy fuse nfsd lockd auth_rpcgss exportfs sunrpc autofs4 af_packet ipt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_LOG nf_conntrack_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables x_tables ipv6 p4_clockmod speedstep_lib binfmt_misc dm_mirror dm_region_hash dm_log dm_mod uinput snd_emu10k1_synth snd_emux_synth snd_seq_virmidi snd_seq_midi_event snd_seq_midi_emul snd_emu10k1 snd_ac97_codec snd_usb_audio ac97_bus snd_seq snd_pcm snd_usb_lib snd_rawmidi snd_seq_device snd_timer firewire_ohci ppdev uvcvideo floppy firewire_core snd_page_alloc snd_util_mem snd_hwdep parport_pc pwc psmouse videodev parport v4l1_compat crc_itu_t pcspkr snd sg i2c_i801 serio_raw soundcore dcdbas ext3 jbd mbcache sr_mod cdrom sd_mo d pata_acpi sata_sil uhci_hcd ata_piix libata scsi_mod ehci_hcd e1000 usbcore thermal button radeon intel_agp ttm drm agpgart i2c_algo_bit cfbcopyarea cfbimgblt cfbfillrect [last unloaded: processor] Pid: 32056, comm: rpm Tainted: G W 2.6.31.6 #1 Call Trace: [<c1023ba8>] ? warn_slowpath_common+0x5d/0x70 [<c1023bc6>] ? warn_slowpath_null+0xb/0xd [<c11a1414>] ? inet_sock_destruct+0xd8/0x138 [<c1163243>] ? __sk_free+0x10/0xa2 [<c1196b4a>] ? tcp_v4_rcv+0x442/0x640 [<c118003e>] ? ip_local_deliver_finish+0xf3/0x1ab [<c117fcd9>] ? ip_rcv_finish+0x2a9/0x2cf [<c117fa30>] ? ip_rcv_finish+0x0/0x2cf [<c116b7c5>] ? netif_receive_skb+0x261/0x281 [<f8527bfc>] ? e1000_clean_rx_irq+0x31c/0x3c3 [e1000] [<f852a6fa>] ? e1000_clean+0x2a7/0x3f5 [e1000] [<c11c783c>] ? _spin_unlock_irqrestore+0xe/0x21 [<c10354c0>] ? hrtimer_run_pending+0xd/0xa5 [<c11c769b>] ? _spin_lock_irq+0xe/0x24 [<c116bce5>] ? net_rx_action+0x57/0xfd [<c1027ea3>] ? __do_softirq+0x7a/0xe3 [<c1027e29>] ? __do_softirq+0x0/0xe3 <IRQ> [<c1027c3c>] ? irq_exit+0x29/0x63 [<c1004320>] ? do_IRQ+0x7c/0x8d [<c1002f29>] ? common_interrupt+0x29/0x30 ---[ end trace e643d9455a26ccf4 ]--- ------------[ cut here ]------------ WARNING: at /home/chris/LINUX/linux-2.6.31/net/ipv4/af_inet.c:154 inet_sock_destruct+0x11e/0x138() Hardware name: Precision WorkStation 650 Modules linked in: tun snd_seq_oss snd_seq_midi snd_seq_dummy fuse nfsd lockd auth_rpcgss exportfs sunrpc autofs4 af_packet ipt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_LOG nf_conntrack_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables x_tables ipv6 p4_clockmod speedstep_lib binfmt_misc dm_mirror dm_region_hash dm_log dm_mod uinput snd_emu10k1_synth snd_emux_synth snd_seq_virmidi snd_seq_midi_event snd_seq_midi_emul snd_emu10k1 snd_ac97_codec snd_usb_audio ac97_bus snd_seq snd_pcm snd_usb_lib snd_rawmidi snd_seq_device snd_timer firewire_ohci ppdev uvcvideo floppy firewire_core snd_page_alloc snd_util_mem snd_hwdep parport_pc pwc psmouse videodev parport v4l1_compat crc_itu_t pcspkr snd sg i2c_i801 serio_raw soundcore dcdbas ext3 jbd mbcache sr_mod cdrom sd_mo d pata_acpi sata_sil uhci_hcd ata_piix libata scsi_mod ehci_hcd e1000 usbcore thermal button radeon intel_agp ttm drm agpgart i2c_algo_bit cfbcopyarea cfbimgblt cfbfillrect [last unloaded: processor] Pid: 32056, comm: rpm Tainted: G W 2.6.31.6 #1 Call Trace: [<c1023ba8>] ? warn_slowpath_common+0x5d/0x70 [<c1023bc6>] ? warn_slowpath_null+0xb/0xd [<c11a145a>] ? inet_sock_destruct+0x11e/0x138 [<c1163243>] ? __sk_free+0x10/0xa2 [<c1196b4a>] ? tcp_v4_rcv+0x442/0x640 [<c118003e>] ? ip_local_deliver_finish+0xf3/0x1ab [<c117fcd9>] ? ip_rcv_finish+0x2a9/0x2cf [<c117fa30>] ? ip_rcv_finish+0x0/0x2cf [<c116b7c5>] ? netif_receive_skb+0x261/0x281 [<f8527bfc>] ? e1000_clean_rx_irq+0x31c/0x3c3 [e1000] [<f852a6fa>] ? e1000_clean+0x2a7/0x3f5 [e1000] [<c11c783c>] ? _spin_unlock_irqrestore+0xe/0x21 [<c10354c0>] ? hrtimer_run_pending+0xd/0xa5 [<c11c769b>] ? _spin_lock_irq+0xe/0x24 [<c116bce5>] ? net_rx_action+0x57/0xfd [<c1027ea3>] ? __do_softirq+0x7a/0xe3 [<c1027e29>] ? __do_softirq+0x0/0xe3 <IRQ> [<c1027c3c>] ? irq_exit+0x29/0x63 [<c1004320>] ? do_IRQ+0x7c/0x8d [<c1002f29>] ? common_interrupt+0x29/0x30 ---[ end trace e643d9455a26ccf5 ]---
RFC PATCH 1/1 net: sock_queue_err_skb() and sk_forward_alloc corruption
In reply to Stephen HemmingerWhile investigating on sk_forward_alloc corruptions, I found two problems :
1) skb_tstamp_tx() is calling sock_queue_err_skb().
This is not good as is, because we need sock lock
before calling sock_queue_err_skb().
Problem is skb_tstamp_rx() wont be able to lock sock...
skb_tstamp_rx() ->
sock_queue_err_skb() -> sk_mem_charge(sk, skb->truesize) -> // PROBLEM : sk->sk_forward_alloc -= size; // MUST BE PROTECTED
2) UDP (again) sk_forward_alloc corruption
__udp4_lib_err ->
if (inet->recverr) ip_icmp_error() -> sock_queue_err_skb() // PROBLEM
Oh well...
I wonder if we could use a special version of skb_set_owner_r()/sock_rfree() *without* sk_mem_charge()/sk_mem_uncharge() calls for this error queue.
(We dont call sk_rmem_schedule() anyway, so I guess current usage is not correct, even with sock locked ?)
Something like this (untested but compiled) patch ?
--- include/net/sock.h | 11 ++++++++++- net/core/sock.c | 8 ++++++++ 2 files changed, 18 insertions(+), 1 deletion(-)
include/net/sock.h +10 -1
@@ -961,6 +961,7 @@ extern struct sk_buff *sock_rmalloc(struct sock *sk, gfp_t priority);extern void sock_wfree (struct sk_buff *skb);extern void sock_rfree (struct sk_buff *skb); +extern void sock_nocharge_rfree (struct sk_buff *skb); extern int sock_setsockopt(struct socket *sock, int level, int op, char __user *optval,
@@ -1383,6 +1384,14 @@ static inline void skb_set_owner_r(struct sk_buff *skb, struct sock *sk)sk_mem_charge (sk, skb->truesize); } +static inline void skb_set_owner_nocharge_r (struct sk_buff *skb,struct sock *sk) +{ +skb_orphan (skb); + skb->sk = sk; + skb->destructor = sock_nocharge_rfree; +atomic_add (skb->truesize, &sk->sk_rmem_alloc); +} +extern void sk_reset_timer (struct sock *sk,struct timer_list * timer,unsigned long expires);
@@ -1398,7 +1407,7 @@ static inline int sock_queue_err_skb(struct sock *sk, struct sk_buff *skb)if (atomic_read (&sk->sk_rmem_alloc) + skb->truesize >= (unsigned )sk->sk_rcvbuf)return -ENOMEM ; -skb_set_owner_r (skb, sk); +skb_set_owner_nocharge_r (skb, sk);skb_queue_tail (&sk->sk_error_queue, skb);if (!sock_flag (sk,SOCK_DEAD )) sk->sk_data_ready (sk, skb->len);
net/core/sock.c +8 -0
@@ -1284,6 +1284,14 @@ void sock_rfree(struct sk_buff *skb) }EXPORT_SYMBOL (sock_rfree); +void sock_nocharge_rfree (struct sk_buff *skb) +{ +struct sock *sk = skb->sk; + +atomic_sub (skb->truesize, &sk->sk_rmem_alloc); +} +EXPORT_SYMBOL (sock_nocharge_rfree); +int sock_i_uid (struct sock *sk) {
David Miller
maintainerRe: [RFC, PATCH] net: sock_queue_err_skb() and sk_forward_alloc corruption
From: Eric Dumazet <[email protected]>
Date: Mon, 07 Dec 2009 23:32:16 +0100
Quoted 9 lines
I wonder if we could use a special version of skb_set_owner_r()/sock_rfree() *without* sk_mem_charge()/sk_mem_uncharge() calls for this error queue.
(We dont call
sk_rmem_schedule()anyway, so I guess current usage is not correct, even with sock locked ?)Something like this (untested but compiled) patch ?
Signed-off-by Eric Dumazet <[email protected]>
I think this is legitimate in exactly this kind of case.
The paths where we do these non-charging add, we already just made sure the receive queue is not over the limit. Therefore we won't have possible paths where we can queue error skbs endlessly and without any controls.
So I'm ok with this approach to fix these bugs.