Skip to contentSkip to conversation

kernel BUG at kernel/timer.c:748!

DISCUSSION Dave Jones 7 participants 20 messages Sep 5, 2012 – Sep 24, 2012

Subsystems (4)
lore
Share & tools
lore

Dave Jones

author
#0 lore raw

Just hit this bug on 3.6-rc4.

The BUG is..

	BUG_ON(!timer->function);

Not much to go on... Any thoughts on what I could add to get more debug info on which protocol etc this was ?

Dave

kernel BUG at kernel/timer.c:748!
invalid opcode: 0000 [#1] SMP

Modules linked in: tun fuse ipt_ULOG binfmt_misc nfnetlink nfc caif_socket caif phonet can llc2 pppoe pppox ppp_generic slhc irda crc_ccitt rds af_key decnet rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 nfsv3 nfs_acl nfs fscache lockd sunrpc bluetooth rfkill ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack kvm_intel kvm crc32c_intel ghash_clmulni_intel microcode pcspkr i2c_i801 e1000e uinput i915 video i2c_algo_bit drm_kms_helper drm i2c_core
CPU 3 
Pid: 12330, comm: trinity-child3 Not tainted 3.6.0-rc4+ #36
RIP: 0010:[<ffffffff810813f5>]  [<ffffffff810813f5>] mod_timer+0x2c5/0x2f0
RSP: 0018:ffff88000dfd7e08  EFLAGS: 00010246
RAX: 000000000000001a RBX: ffff880122d62948 RCX: 000000000000001a
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88000dfd7e10
RBP: ffff88000dfd7e48 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000001517000 R11: 0000000000000246 R12: 000000016c000000
R13: 000000016c12bcb1 R14: ffff8801236cee00 R15: 00000000ffffff01
FS:  00007fa96745f740(0000) GS:ffff880148200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000100ff000 CR3: 0000000099344000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process trinity-child3 (pid: 12330, threadinfo ffff88000dfd6000, task ffff880090890000)
Stack:
 ffffffff8154cb6d 0000000007b5edf7 ffff88000dfd7e28 ffff880122d62520
 0000000000000009 0000000000000004 ffff8801236cee00 00000000ffffff01
 ffff88000dfd7e68 ffffffff8154c79c ffffffff81550e6c ffff880122d62520
Call Trace:
 [<ffffffff8154cb6d>] ? lock_sock_nested+0x8d/0xa0
 [<ffffffff8154c79c>] sk_reset_timer+0x1c/0x30
 [<ffffffff81550e6c>] ? sock_setsockopt+0x8c/0x960
 [<ffffffff815a84a0>] inet_csk_reset_keepalive_timer+0x20/0x30
 [<ffffffff815c018d>] tcp_set_keepalive+0x3d/0x50
 [<ffffffff81551703>] sock_setsockopt+0x923/0x960
 [<ffffffff810ddf76>] ? trace_hardirqs_on_caller+0x16/0x1e0
 [<ffffffff811db0ac>] ? fget_light+0x24c/0x520
 [<ffffffff8154af86>] sys_setsockopt+0xc6/0xe0
 [<ffffffff816a50ed>] system_call_fastpath+0x1a/0x1f
Code: 00 74 43 9c 58 0f 1f 44 00 00 f6 c4 02 0f 84 14 ff ff ff eb 93 48 c7 c7 20 48 c3 81 e8 f5 70 05 00 85 c0 0f 85 fe fe ff ff eb b7 <0f> 0b 48 8b 75 08 48 89 df e8 3d f6 ff ff e9 b2 fd ff ff 4d 89 
RIP  [<ffffffff810813f5>] mod_timer+0x2c5/0x2f0
 RSP <ffff88000dfd7e08>
---[ end trace 7e7b5910138e49a3 ]---
#1 lore raw
Quoted 5 lines · Dave Jones

Just hit this bug on 3.6-rc4.

The BUG is..

        BUG_ON(!timer->function);

TCP keepalive timer is setup when the socket is created.

__sock_create
inet_create
tcp_v4_init_sock
tcp_init_sock
tcp_init_xmit_timers
inet_csk_init_xmit_timers

timer->function should not be NULL when set keepalive option.

Strange...have bug somewhere.

Lin Ming

Quoted 50 lines

Not much to go on... Any thoughts on what I could add to get more debug info on which protocol etc this was ?

Dave

kernel BUG at kernel/timer.c:748!
invalid opcode: 0000 [#1] SMP

Modules linked in: tun fuse ipt_ULOG binfmt_misc nfnetlink nfc caif_socket caif phonet can llc2 pppoe pppox ppp_generic slhc irda crc_ccitt rds af_key decnet rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 nfsv3 nfs_acl nfs fscache lockd sunrpc bluetooth rfkill ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack kvm_intel kvm crc32c_intel ghash_clmulni_intel microcode pcspkr i2c_i801 e1000e uinput i915 video i2c_algo_bit drm_kms_helper drm i2c_core
CPU 3
Pid: 12330, comm: trinity-child3 Not tainted 3.6.0-rc4+ #36
RIP: 0010:[<ffffffff810813f5>]  [<ffffffff810813f5>] mod_timer+0x2c5/0x2f0
RSP: 0018:ffff88000dfd7e08  EFLAGS: 00010246
RAX: 000000000000001a RBX: ffff880122d62948 RCX: 000000000000001a
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88000dfd7e10
RBP: ffff88000dfd7e48 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000001517000 R11: 0000000000000246 R12: 000000016c000000
R13: 000000016c12bcb1 R14: ffff8801236cee00 R15: 00000000ffffff01
FS:  00007fa96745f740(0000) GS:ffff880148200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000100ff000 CR3: 0000000099344000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process trinity-child3 (pid: 12330, threadinfo ffff88000dfd6000, task ffff880090890000)
Stack:
 ffffffff8154cb6d 0000000007b5edf7 ffff88000dfd7e28 ffff880122d62520
 0000000000000009 0000000000000004 ffff8801236cee00 00000000ffffff01
 ffff88000dfd7e68 ffffffff8154c79c ffffffff81550e6c ffff880122d62520
Call Trace:
 [<ffffffff8154cb6d>] ? lock_sock_nested+0x8d/0xa0
 [<ffffffff8154c79c>] sk_reset_timer+0x1c/0x30
 [<ffffffff81550e6c>] ? sock_setsockopt+0x8c/0x960
 [<ffffffff815a84a0>] inet_csk_reset_keepalive_timer+0x20/0x30
 [<ffffffff815c018d>] tcp_set_keepalive+0x3d/0x50
 [<ffffffff81551703>] sock_setsockopt+0x923/0x960
 [<ffffffff810ddf76>] ? trace_hardirqs_on_caller+0x16/0x1e0
 [<ffffffff811db0ac>] ? fget_light+0x24c/0x520
 [<ffffffff8154af86>] sys_setsockopt+0xc6/0xe0
 [<ffffffff816a50ed>] system_call_fastpath+0x1a/0x1f
Code: 00 74 43 9c 58 0f 1f 44 00 00 f6 c4 02 0f 84 14 ff ff ff eb 93 48 c7 c7 20 48 c3 81 e8 f5 70 05 00 85 c0 0f 85 fe fe ff ff eb b7 <0f> 0b 48 8b 75 08 48 89 df e8 3d f6 ff ff e9 b2 fd ff ff 4d 89
RIP  [<ffffffff810813f5>] mod_timer+0x2c5/0x2f0
 RSP <ffff88000dfd7e08>
---[ end trace 7e7b5910138e49a3 ]---

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw
Quoted 16 lines · Lin Ming
Quoted 5 lines · Dave Jones

Just hit this bug on 3.6-rc4.

The BUG is..

        BUG_ON(!timer->function);

TCP keepalive timer is setup when the socket is created.

__sock_create
inet_create
tcp_v4_init_sock
tcp_init_sock
tcp_init_xmit_timers
inet_csk_init_xmit_timers

Quoted 3 lines

timer->function should not be NULL when set keepalive option.

Strange...have bug somewhere.

is this a passively opened socket or actively opened one?

Quoted 57 lines

Lin Ming

Quoted 46 lines

Not much to go on... Any thoughts on what I could add to get more debug info on which protocol etc this was ?

Dave

kernel BUG at kernel/timer.c:748!
invalid opcode: 0000 [#1] SMP

Modules linked in: tun fuse ipt_ULOG binfmt_misc nfnetlink nfc caif_socket caif phonet can llc2 pppoe pppox ppp_generic slhc irda crc_ccitt rds af_key decnet rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 nfsv3 nfs_acl nfs fscache lockd sunrpc bluetooth rfkill ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack kvm_intel kvm crc32c_intel ghash_clmulni_intel microcode pcspkr i2c_i801 e1000e uinput i915 video i2c_algo_bit drm_kms_helper drm i2c_core
CPU 3
Pid: 12330, comm: trinity-child3 Not tainted 3.6.0-rc4+ #36
RIP: 0010:[<ffffffff810813f5>]  [<ffffffff810813f5>] mod_timer+0x2c5/0x2f0
RSP: 0018:ffff88000dfd7e08  EFLAGS: 00010246
RAX: 000000000000001a RBX: ffff880122d62948 RCX: 000000000000001a
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88000dfd7e10
RBP: ffff88000dfd7e48 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000001517000 R11: 0000000000000246 R12: 000000016c000000
R13: 000000016c12bcb1 R14: ffff8801236cee00 R15: 00000000ffffff01
FS:  00007fa96745f740(0000) GS:ffff880148200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000100ff000 CR3: 0000000099344000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process trinity-child3 (pid: 12330, threadinfo ffff88000dfd6000, task ffff880090890000)
Stack:
 ffffffff8154cb6d 0000000007b5edf7 ffff88000dfd7e28 ffff880122d62520
 0000000000000009 0000000000000004 ffff8801236cee00 00000000ffffff01
 ffff88000dfd7e68 ffffffff8154c79c ffffffff81550e6c ffff880122d62520
Call Trace:
 [<ffffffff8154cb6d>] ? lock_sock_nested+0x8d/0xa0
 [<ffffffff8154c79c>] sk_reset_timer+0x1c/0x30
 [<ffffffff81550e6c>] ? sock_setsockopt+0x8c/0x960
 [<ffffffff815a84a0>] inet_csk_reset_keepalive_timer+0x20/0x30
 [<ffffffff815c018d>] tcp_set_keepalive+0x3d/0x50
 [<ffffffff81551703>] sock_setsockopt+0x923/0x960
 [<ffffffff810ddf76>] ? trace_hardirqs_on_caller+0x16/0x1e0
 [<ffffffff811db0ac>] ? fget_light+0x24c/0x520
 [<ffffffff8154af86>] sys_setsockopt+0xc6/0xe0
 [<ffffffff816a50ed>] system_call_fastpath+0x1a/0x1f
Code: 00 74 43 9c 58 0f 1f 44 00 00 f6 c4 02 0f 84 14 ff ff ff eb 93 48 c7 c7 20 48 c3 81 e8 f5 70 05 00 85 c0 0f 85 fe fe ff ff eb b7 <0f> 0b 48 8b 75 08 48 89 df e8 3d f6 ff ff e9 b2 fd ff ff 4d 89
RIP  [<ffffffff810813f5>] mod_timer+0x2c5/0x2f0
 RSP <ffff88000dfd7e08>
---[ end trace 7e7b5910138e49a3 ]---
Quoted 4 lines

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#3 lore raw
Quoted 17 lines · Yuchung Cheng
Quoted 16 lines · Lin Ming
Quoted 5 lines · Dave Jones

Just hit this bug on 3.6-rc4.

The BUG is..

        BUG_ON(!timer->function);

TCP keepalive timer is setup when the socket is created.

__sock_create
inet_create
tcp_v4_init_sock
tcp_init_sock
tcp_init_xmit_timers
inet_csk_init_xmit_timers

Quoted 5 lines
Quoted 3 lines

timer->function should not be NULL when set keepalive option.

Strange...have bug somewhere.

is this a passively opened socket or actively opened one?

I have no idea. That fuzzer ran for around 10 hours before it triggered this. In that time, millions of syscalls were done.

Even though I had logs of every syscall, the volume of data to sift through to try and map back to the socket that caused this is firmly in 'needle in haystick' territory.

Dave

#5 lore raw
In reply to Dave Jones

Hello,

Quoted 42 lines · Dave Jones

Just hit this bug on 3.6-rc4.

The BUG is..

	BUG_ON(!timer->function);

Not much to go on... Any thoughts on what I could add to get more debug info on which protocol etc this was ?

Dave

kernel BUG at kernel/timer.c:748!
invalid opcode: 0000 [#1] SMP

Modules linked in: tun fuse ipt_ULOG binfmt_misc nfnetlink nfc caif_socket caif phonet can llc2 pppoe pppox ppp_generic slhc irda crc_ccitt rds af_key decnet rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 nfsv3 nfs_acl nfs fscache lockd sunrpc bluetooth rfkill ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack kvm_intel kvm crc32c_intel ghash_clmulni_intel microcode pcspkr i2c_i801 e1000e uinput i915 video i2c_algo_bit drm_kms_helper drm i2c_core
CPU 3 
Pid: 12330, comm: trinity-child3 Not tainted 3.6.0-rc4+ #36
RIP: 0010:[<ffffffff810813f5>]  [<ffffffff810813f5>] mod_timer+0x2c5/0x2f0
RSP: 0018:ffff88000dfd7e08  EFLAGS: 00010246
RAX: 000000000000001a RBX: ffff880122d62948 RCX: 000000000000001a
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88000dfd7e10
RBP: ffff88000dfd7e48 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000001517000 R11: 0000000000000246 R12: 000000016c000000
R13: 000000016c12bcb1 R14: ffff8801236cee00 R15: 00000000ffffff01
FS:  00007fa96745f740(0000) GS:ffff880148200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000100ff000 CR3: 0000000099344000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process trinity-child3 (pid: 12330, threadinfo ffff88000dfd6000, task ffff880090890000)
Stack:
 ffffffff8154cb6d 0000000007b5edf7 ffff88000dfd7e28 ffff880122d62520
 0000000000000009 0000000000000004 ffff8801236cee00 00000000ffffff01
 ffff88000dfd7e68 ffffffff8154c79c ffffffff81550e6c ffff880122d62520
Call Trace:
 [<ffffffff8154cb6d>] ? lock_sock_nested+0x8d/0xa0
 [<ffffffff8154c79c>] sk_reset_timer+0x1c/0x30
 [<ffffffff81550e6c>] ? sock_setsockopt+0x8c/0x960
 [<ffffffff815a84a0>] inet_csk_reset_keepalive_timer+0x20/0x30
 [<ffffffff815c018d>] tcp_set_keepalive+0x3d/0x50
 [<ffffffff81551703>] sock_setsockopt+0x923/0x960
Quoted 8 lines
 [<ffffffff810ddf76>] ? trace_hardirqs_on_caller+0x16/0x1e0
 [<ffffffff811db0ac>] ? fget_light+0x24c/0x520
 [<ffffffff8154af86>] sys_setsockopt+0xc6/0xe0
 [<ffffffff816a50ed>] system_call_fastpath+0x1a/0x1f
Code: 00 74 43 9c 58 0f 1f 44 00 00 f6 c4 02 0f 84 14 ff ff ff eb 93 48 c7 c7 20 48 c3 81 e8 f5 70 05 00 85 c0 0f 85 fe fe ff ff eb b7 <0f> 0b 48 8b 75 08 48 89 df e8 3d f6 ff ff e9 b2 fd ff ff 4d 89 
RIP  [<ffffffff810813f5>] mod_timer+0x2c5/0x2f0
 RSP <ffff88000dfd7e08>
---[ end trace 7e7b5910138e49a3 ]---

Can this help? In case you see ICMPV6_PKT_TOOBIG...

[PATCH] tcp: fix possible socket refcount problem for ipv6

	commit 144d56e91044181ec0ef67aeca91e9a8b5718348

("tcp: fix possible socket refcount problem") is missing the IPv6 part. As tcp_release_cb is shared by both protocols we should hold sock reference for the TCP_MTU_REDUCED_DEFERRED bit.

---
 net/ipv6/tcp_ipv6.c |    5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)
net/ipv6/tcp_ipv6.c +3 -2
index 09078b9..f3bfb8b 100644
@@ -403,8 +403,9 @@ static void tcp_v6_err(struct sk_buff *skb, struct inet6_skb_parm *opt,
 		tp->mtu_info = ntohl(info);
 		if (!sock_owned_by_user(sk))
 			tcp_v6_mtu_reduced(sk);
-		else
-			set_bit(TCP_MTU_REDUCED_DEFERRED, &tp->tsq_flags);
+		else if (!test_and_set_bit(TCP_MTU_REDUCED_DEFERRED,
+					   &tp->tsq_flags))
+			sock_hold(sk);
 		goto out;
 	}
 
Signature · 1 line
1.7.3.4
#4 lore raw
In reply to Lin Ming
Quoted 9 lines · Lin Ming
Quoted 5 lines · Dave Jones

Just hit this bug on 3.6-rc4.

The BUG is..

        BUG_ON(!timer->function);

TCP keepalive timer is setup when the socket is created.

Quoted 8 lines

__sock_create
inet_create
tcp_v4_init_sock
tcp_init_sock
tcp_init_xmit_timers
inet_csk_init_xmit_timers

timer->function should not be NULL when set keepalive option.

And tcp_init_xmit_timers() is called on the passive open side as well, v4 as well as v6. I don't see any code explicitly set timer->function back to NULL (unless through set_timer(..., NULL,...). This may be a corrupted sock (already released?)

Jerry

Quoted 59 lines

Strange...have bug somewhere.

Lin Ming

Quoted 46 lines

Not much to go on... Any thoughts on what I could add to get more debug info on which protocol etc this was ?

Dave

kernel BUG at kernel/timer.c:748!
invalid opcode: 0000 [#1] SMP

Modules linked in: tun fuse ipt_ULOG binfmt_misc nfnetlink nfc caif_socket caif phonet can llc2 pppoe pppox ppp_generic slhc irda crc_ccitt rds af_key decnet rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 nfsv3 nfs_acl nfs fscache lockd sunrpc bluetooth rfkill ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack kvm_intel kvm crc32c_intel ghash_clmulni_intel microcode pcspkr i2c_i801 e1000e uinput i915 video i2c_algo_bit drm_kms_helper drm i2c_core
CPU 3
Pid: 12330, comm: trinity-child3 Not tainted 3.6.0-rc4+ #36
RIP: 0010:[<ffffffff810813f5>]  [<ffffffff810813f5>] mod_timer+0x2c5/0x2f0
RSP: 0018:ffff88000dfd7e08  EFLAGS: 00010246
RAX: 000000000000001a RBX: ffff880122d62948 RCX: 000000000000001a
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88000dfd7e10
RBP: ffff88000dfd7e48 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000001517000 R11: 0000000000000246 R12: 000000016c000000
R13: 000000016c12bcb1 R14: ffff8801236cee00 R15: 00000000ffffff01
FS:  00007fa96745f740(0000) GS:ffff880148200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000100ff000 CR3: 0000000099344000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process trinity-child3 (pid: 12330, threadinfo ffff88000dfd6000, task ffff880090890000)
Stack:
 ffffffff8154cb6d 0000000007b5edf7 ffff88000dfd7e28 ffff880122d62520
 0000000000000009 0000000000000004 ffff8801236cee00 00000000ffffff01
 ffff88000dfd7e68 ffffffff8154c79c ffffffff81550e6c ffff880122d62520
Call Trace:
 [<ffffffff8154cb6d>] ? lock_sock_nested+0x8d/0xa0
 [<ffffffff8154c79c>] sk_reset_timer+0x1c/0x30
 [<ffffffff81550e6c>] ? sock_setsockopt+0x8c/0x960
 [<ffffffff815a84a0>] inet_csk_reset_keepalive_timer+0x20/0x30
 [<ffffffff815c018d>] tcp_set_keepalive+0x3d/0x50
 [<ffffffff81551703>] sock_setsockopt+0x923/0x960
 [<ffffffff810ddf76>] ? trace_hardirqs_on_caller+0x16/0x1e0
 [<ffffffff811db0ac>] ? fget_light+0x24c/0x520
 [<ffffffff8154af86>] sys_setsockopt+0xc6/0xe0
 [<ffffffff816a50ed>] system_call_fastpath+0x1a/0x1f
Code: 00 74 43 9c 58 0f 1f 44 00 00 f6 c4 02 0f 84 14 ff ff ff eb 93 48 c7 c7 20 48 c3 81 e8 f5 70 05 00 85 c0 0f 85 fe fe ff ff eb b7 <0f> 0b 48 8b 75 08 48 89 df e8 3d f6 ff ff e9 b2 fd ff ff 4d 89
RIP  [<ffffffff810813f5>] mod_timer+0x2c5/0x2f0
 RSP <ffff88000dfd7e08>
---[ end trace 7e7b5910138e49a3 ]---
Quoted 4 lines

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#6 lore raw
In reply to Julian Anastasov
Quoted 13 lines · Julian Anastasov
Quoted 12 lines

kernel BUG at kernel/timer.c:748!

Call Trace:
 ? lock_sock_nested+0x8d/0xa0
 sk_reset_timer+0x1c/0x30
 ? sock_setsockopt+0x8c/0x960
 inet_csk_reset_keepalive_timer+0x20/0x30
 tcp_set_keepalive+0x3d/0x50
 sock_setsockopt+0x923/0x960
 ? trace_hardirqs_on_caller+0x16/0x1e0
 ? fget_light+0x24c/0x520
 sys_setsockopt+0xc6/0xe0
 system_call_fastpath+0x1a/0x1f
Quoted 3 lines

Can this help? In case you see ICMPV6_PKT_TOOBIG...

[PATCH] tcp: fix possible socket refcount problem for ipv6

I just managed to reproduce this bug on rc5 with this patch, so it doesn't seem to help.

Dave

#7 lore raw
Quoted 15 lines · Dave Jones
Quoted 13 lines · Julian Anastasov
Quoted 12 lines

kernel BUG at kernel/timer.c:748!

Call Trace:
 ? lock_sock_nested+0x8d/0xa0
 sk_reset_timer+0x1c/0x30
 ? sock_setsockopt+0x8c/0x960
 inet_csk_reset_keepalive_timer+0x20/0x30
 tcp_set_keepalive+0x3d/0x50
 sock_setsockopt+0x923/0x960
 ? trace_hardirqs_on_caller+0x16/0x1e0
 ? fget_light+0x24c/0x520
 sys_setsockopt+0xc6/0xe0
 system_call_fastpath+0x1a/0x1f
Quoted 6 lines
Quoted 3 lines

Can this help? In case you see ICMPV6_PKT_TOOBIG...

[PATCH] tcp: fix possible socket refcount problem for ipv6

I just managed to reproduce this bug on rc5 with this patch, so it doesn't seem to help.

Could you post some tcpdump traces?

Quoted 7 lines

Dave

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#8 lore raw
Quoted 16 lines · Yuchung Cheng
Quoted 15 lines · Dave Jones
Quoted 13 lines · Julian Anastasov
Quoted 12 lines

kernel BUG at kernel/timer.c:748!

Call Trace:
 ? lock_sock_nested+0x8d/0xa0
 sk_reset_timer+0x1c/0x30
 ? sock_setsockopt+0x8c/0x960
 inet_csk_reset_keepalive_timer+0x20/0x30
 tcp_set_keepalive+0x3d/0x50
 sock_setsockopt+0x923/0x960
 ? trace_hardirqs_on_caller+0x16/0x1e0
 ? fget_light+0x24c/0x520
 sys_setsockopt+0xc6/0xe0
 system_call_fastpath+0x1a/0x1f
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

Can this help? In case you see ICMPV6_PKT_TOOBIG...

[PATCH] tcp: fix possible socket refcount problem for ipv6

I just managed to reproduce this bug on rc5 with this patch, so it doesn't seem to help.

Could you post some tcpdump traces?

It's likely that there aren't any packets. The fuzzer isn't smart enough (yet) to do anything too clever to the sockets it creates.

More likely is that this is some race where thread A is doing a setsockopt while thread B is doing a tear-down of the same socket.

Dave

#9 lore raw
Quoted 23 lines · Dave Jones
Quoted 22 lines · Yuchung Cheng
Quoted 21 lines · Dave Jones
Quoted 13 lines · Julian Anastasov
Quoted 12 lines

kernel BUG at kernel/timer.c:748!

Call Trace:
 ? lock_sock_nested+0x8d/0xa0
 sk_reset_timer+0x1c/0x30
 ? sock_setsockopt+0x8c/0x960
 inet_csk_reset_keepalive_timer+0x20/0x30
 tcp_set_keepalive+0x3d/0x50
 sock_setsockopt+0x923/0x960
 ? trace_hardirqs_on_caller+0x16/0x1e0
 ? fget_light+0x24c/0x520
 sys_setsockopt+0xc6/0xe0
 system_call_fastpath+0x1a/0x1f
Quoted 3 lines

Can this help? In case you see ICMPV6_PKT_TOOBIG...

[PATCH] tcp: fix possible socket refcount problem for ipv6

I just managed to reproduce this bug on rc5 with this patch, so it doesn't seem to help.

Quoted 7 lines
Quoted 1 line

Could you post some tcpdump traces?

It's likely that there aren't any packets. The fuzzer isn't smart enough (yet) to do anything too clever to the sockets it creates.

More likely is that this is some race where thread A is doing a setsockopt while thread B is doing a tear-down of the same socket.

I spent some time trying to track this bug, but found nothing so far.

The timer->function are never cleared by TCP stack at tear down, and should be set before fd is installed and can be caught by other threads.

Most likely its a refcounting issue...

Following debugging patch might trigger a bug sooner ?

include/net/sock.h +9 -6
index 84bdaec..5d3ad5b 100644
@@ -511,18 +511,18 @@ static inline void sock_hold(struct sock *sk)
  */
 static inline void __sock_put(struct sock *sk)
 {
-	atomic_dec(&sk->sk_refcnt);
+	int newcnt = atomic_dec_return(&sk->sk_refcnt);
+
+	WARN_ON(newcnt <= 0);
 }
 
 static inline bool sk_del_node_init(struct sock *sk)
 {
 	bool rc = __sk_del_node_init(sk);
 
-	if (rc) {
-		/* paranoid for a while -acme */
-		WARN_ON(atomic_read(&sk->sk_refcnt) == 1);
+	if (rc)
 		__sock_put(sk);
-	}
+
 	return rc;
 }
 #define sk_del_node_init_rcu(sk)	sk_del_node_init(sk)
@@ -1620,7 +1620,10 @@ static inline void sk_filter_charge(struct sock *sk, struct sk_filter *fp)
 /* Ungrab socket and destroy it, if it was the last reference. */
 static inline void sock_put(struct sock *sk)
 {
-	if (atomic_dec_and_test(&sk->sk_refcnt))
+	int newcnt = atomic_dec_return(&sk->sk_refcnt);
+
+	WARN_ON(newcnt < 0);
+	if (newcnt == 0)
 		sk_free(sk);
 }
 
author
#10 lore raw
Quoted 8 lines · Eric Dumazet

I spent some time trying to track this bug, but found nothing so far.

The timer->function are never cleared by TCP stack at tear down, and should be set before fd is installed and can be caught by other threads.

Most likely its a refcounting issue...

Following debugging patch might trigger a bug sooner ?

4 hours so far, nothing.. I'll leave it run overnight, but it doesn't seem to have made much difference if any.

Dave

author
#11 lore raw
Quoted 7 lines · Dave Jones
Quoted 5 lines · Eric Dumazet

I spent some time trying to track this bug, but found nothing so far.

The timer->function are never cleared by TCP stack at tear down, and should be set before fd is installed and can be caught by other threads.

Quoted 6 lines
Quoted 3 lines

Most likely its a refcounting issue...

Following debugging patch might trigger a bug sooner ?

4 hours so far, nothing.. I'll leave it run overnight, but it doesn't seem to have made much difference if any.

One of my over-weekend runs hit this again, but it didn't trigger the WARN that your patch added :-/

Dave

#12 lore raw
Quoted 12 lines · Dave Jones
Quoted 11 lines · Dave Jones
Quoted 8 lines · Eric Dumazet

I spent some time trying to track this bug, but found nothing so far.

The timer->function are never cleared by TCP stack at tear down, and should be set before fd is installed and can be caught by other threads.

Most likely its a refcounting issue...

Following debugging patch might trigger a bug sooner ?

Quoted 8 lines
Quoted 2 lines

4 hours so far, nothing.. I'll leave it run overnight, but it doesn't seem to have made much difference if any.

One of my over-weekend runs hit this again, but it didn't trigger the WARN that your patch added :-/

Dave

OK, I believe I found the reason. I Will post a patch.

open a raw socket AF_INET, TCP_PROTO
+ connect() ->sk_state set to TCP_ESTABLISHED
+ setsockopt( SO_KEEPALIVE, &on) -> crash

Thanks

#13 lore raw
Quoted 5 lines · Eric Dumazet

OK, I believe I found the reason. I Will post a patch.

open a raw socket AF_INET, TCP_PROTO
+ connect() ->sk_state set to TCP_ESTABLISHED
+ setsockopt( SO_KEEPALIVE, &on) -> crash

I confirm following patch fixes the problem for me.

Thanks again

[PATCH] net: guard tcp_set_keepalive() to tcp sockets

Its possible to use RAW sockets to get a crash in
tcp_set_keepalive() / sk_reset_timer()

Fix is to make sure socket is a SOCK_STREAM one.

---
 net/core/sock.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
net/core/sock.c +2 -1
index 3057920..a6000fb 100644
@@ -691,7 +691,8 @@ set_rcvbuf:
 
 	case SO_KEEPALIVE:
 #ifdef CONFIG_INET
-		if (sk->sk_protocol == IPPROTO_TCP)
+		if (sk->sk_protocol == IPPROTO_TCP &&
+		    sk->sk_type == SOCK_STREAM)
 			tcp_set_keepalive(sk, valbool);
 #endif
 		sock_valbool_flag(sk, SOCK_KEEPOPEN, valbool);
author
#14 lore raw
Quoted 12 lines · Eric Dumazet
Quoted 5 lines · Eric Dumazet

OK, I believe I found the reason. I Will post a patch.

open a raw socket AF_INET, TCP_PROTO
+ connect() ->sk_state set to TCP_ESTABLISHED
+ setsockopt( SO_KEEPALIVE, &on) -> crash

I confirm following patch fixes the problem for me.

Quoted 8 lines

Thanks again

[PATCH] net: guard tcp_set_keepalive() to tcp sockets

Its possible to use RAW sockets to get a crash in
tcp_set_keepalive() / sk_reset_timer()

Fix is to make sure socket is a SOCK_STREAM one.

Great, I'll give this a shot.

Any idea why this only just started triggering ?
(Ie, do we need this for stable too?)

thanks,

Dave

#15 lore raw
Quoted 4 lines · Dave Jones

Great, I'll give this a shot.

Any idea why this only just started triggering ?
(Ie, do we need this for stable too?)

Seems this is a very old bug.

I guess your trinity tool gets better ?

I used following program to trigger the bug :

#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <stdio.h>
#include <string.h>

int main(int argc, char *argv[])

{
	int on = 1;
	struct sockaddr_in addr;
	int raw_sock = socket(AF_INET, SOCK_RAW, IPPROTO_TCP);

	memset(&addr, 0, sizeof(addr));
	addr.sin_family = AF_INET;
	addr.sin_addr.s_addr = htonl(0x7f000001);
	addr.sin_port = 123;
	connect(raw_sock, (struct sockaddr *)&addr, sizeof(addr));
	setsockopt(raw_sock, SOL_SOCKET, SO_KEEPALIVE, &on, sizeof(on));
	return 0;
}
author
#16 lore raw
Quoted 8 lines · Eric Dumazet
Quoted 4 lines · Dave Jones

Great, I'll give this a shot.

Any idea why this only just started triggering ?
(Ie, do we need this for stable too?)

Seems this is a very old bug.

Indeed. As far as I can tell, this goes back to 2.1.8, back in November 1996.

Quoted 1 line

I guess your trinity tool gets better ?

Possible. Or it may be that it was finding other bugs first that are now fixed, that prevented us getting far enough along to try this.

Quoted 1 line

I used following program to trigger the bug :

I'm sure reproducers like that will be appreciated by anyone running QA tests on older kernels.

tangent: Is there any kind of networking correctness suite other than fuzz testers like isic etc ?

thanks,

Dave

maintainer
#18 lore raw
In reply to Eric Dumazet

From: Eric Dumazet <[email protected]>
Date: Mon, 24 Sep 2012 19:00:11 +0200

Quoted 1 line

I know you meant "From: Eric Dumazet <[email protected]>" here :-)

Quoted 9 lines

[PATCH] net: guard tcp_set_keepalive() to tcp sockets

Its possible to use RAW sockets to get a crash in
tcp_set_keepalive() / sk_reset_timer()

Fix is to make sure socket is a SOCK_STREAM one.

Applied and queued up for -stable, thanks Eric.

maintainer
#17 lore raw
In reply to Dave Jones

From: Dave Jones <[email protected]>
Date: Mon, 24 Sep 2012 14:11:52 -0400

Quoted 2 lines

tangent: Is there any kind of networking correctness suite other than fuzz testers like isic etc ?

If people want to work on this I'm willing to maintain the tree.