Skip to contentSkip to conversation

BUG: using __this_cpu_read() in preemptible code in ip6_finish_output

PATCH Applied syzbot 6 participants 8 messages Apr 2, 2019 – Apr 5, 2019

Subsystems (3)
lore
Share & tools
lore

syzbot

author syzbot
#0 lore raw

Hello,

syzbot found the following crash on:

HEAD commit:    e1427237 macsec: add noinline tag to avoid a frame size wa..
git tree:       net-next
console output: https://syzkaller.appspot.com/x/log.txt?x=15594553200000
kernel config:  https://syzkaller.appspot.com/x/.config?x=8e9bc94c16d346a6
dashboard link: https://syzkaller.appspot.com/bug?extid=51471b4aae195285a4a3
compiler:       gcc (GCC) 9.0.0 20181231 (experimental)
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=156ffb07200000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=14412673200000

IMPORTANT: if you fix the bug, please add the following tag to the commit:

Reported-by [email protected]
BUG: using __this_cpu_read() in preemptible [00000000] code: syz-executor222/7596
caller is dev_recursion_level include/linux/netdevice.h:3052 [inline]
caller is ip6_skb_dst_mtu include/net/ip6_route.h:245 [inline]
caller is ip6_finish_output+0x335/0xdc0 net/ipv6/ip6_output.c:149
CPU: 1 PID: 7596 Comm: syz-executor222 Not tainted 5.1.0-rc2+ #118
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x172/0x1f0 lib/dump_stack.c:113
 __this_cpu_preempt_check+0x246/0x270 lib/smp_processor_id.c:47
 dev_recursion_level include/linux/netdevice.h:3052 [inline]
 ip6_skb_dst_mtu include/net/ip6_route.h:245 [inline]
 ip6_finish_output+0x335/0xdc0 net/ipv6/ip6_output.c:149
 NF_HOOK_COND include/linux/netfilter.h:278 [inline]
 ip6_output+0x235/0x7f0 net/ipv6/ip6_output.c:171
 dst_output include/net/dst.h:433 [inline]
 NF_HOOK include/linux/netfilter.h:289 [inline]
 NF_HOOK include/linux/netfilter.h:283 [inline]
 ip6_xmit+0xe41/0x20c0 net/ipv6/ip6_output.c:275
 inet6_csk_xmit+0x2fb/0x5d0 net/ipv6/inet6_connection_sock.c:139
 __tcp_transmit_skb+0x1a32/0x3750 net/ipv4/tcp_output.c:1155
 tcp_transmit_skb net/ipv4/tcp_output.c:1171 [inline]
 tcp_send_syn_data net/ipv4/tcp_output.c:3494 [inline]
 tcp_connect+0x1e47/0x4280 net/ipv4/tcp_output.c:3533
 tcp_v6_connect+0x150b/0x20a0 net/ipv6/tcp_ipv6.c:331
 __inet_stream_connect+0x83f/0xea0 net/ipv4/af_inet.c:659
 tcp_sendmsg_fastopen net/ipv4/tcp.c:1155 [inline]
 tcp_sendmsg_locked+0x231f/0x37f0 net/ipv4/tcp.c:1197
 tcp_sendmsg+0x30/0x50 net/ipv4/tcp.c:1431
 inet_sendmsg+0x147/0x5e0 net/ipv4/af_inet.c:802
 sock_sendmsg_nosec net/socket.c:651 [inline]
 sock_sendmsg+0xdd/0x130 net/socket.c:661
 __sys_sendto+0x262/0x380 net/socket.c:1932
 __do_sys_sendto net/socket.c:1944 [inline]
 __se_sys_sendto net/socket.c:1940 [inline]
 __x64_sys_sendto+0xe1/0x1a0 net/socket.c:1940
 do_syscall_64+0x103/0x610 arch/x86/entry/common.c:290
 entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x440189
Code: 18 89 d0 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 fb 13 fc ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007ffed7abd1a8 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
RAX: ffffffffffffffda RBX: 00000000004002c8 RCX: 0000000000440189
RDX: 0000000000000000 RSI: 0000000000000000 RD

---
This bug is generated by a bot. It may contain errors. See https://goo.gl/tpsmEJ for more information about syzbot. syzbot engineers can be reached at [email protected].

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot. syzbot can test patches for this bug, for details see:
https://goo.gl/tpsmEJ#testing-patches

author syzbot
#1 lore raw
syzbot has bisected this bug to:

syzbot has bisected this bug to:

commit 97cdcf37b57e3f204be3000b9eab9686f38b4356

Author: Florian Westphal <[email protected]>
Date:   Mon Apr 1 14:42:13 2019 +0000

net: place xmit recursion in softnet data

bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=11c04b0f200000
start commit:   e1427237 macsec: add noinline tag to avoid a frame size wa..
git tree:       net-next
final crash:    https://syzkaller.appspot.com/x/report.txt?x=13c04b0f200000
console output: https://syzkaller.appspot.com/x/log.txt?x=15c04b0f200000
kernel config:  https://syzkaller.appspot.com/x/.config?x=8e9bc94c16d346a6
dashboard link: https://syzkaller.appspot.com/bug?extid=51471b4aae195285a4a3
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=156ffb07200000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=14412673200000
Reported-by [email protected]
Fixes 97cdcf37b57e ("net: place xmit recursion in softnet data")

For information about bisection process see: https://goo.gl/tpsmEJ#bisection

#2 lore raw

PATCH 1/1 net: use correct this_cpu primitive in dev_recursion_level

syzbot reports:

BUG: using __this_cpu_read() in preemptible code:
caller is dev_recursion_level include/linux/netdevice.h:3052 [inline]
 __this_cpu_preempt_check+0x246/0x270 lib/smp_processor_id.c:47
 dev_recursion_level include/linux/netdevice.h:3052 [inline]
 ip6_skb_dst_mtu include/net/ip6_route.h:245 [inline]

I erronously downgraded a this_cpu_read to __this_cpu_read when moving dev_recursion_level() around.

Reported-by [email protected]
Fixes 97cdcf37b57e ("net: place xmit recursion in softnet data")
---
 include/linux/netdevice.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/netdevice.h +1 -1
index eb9f05e0863d..521eb869555e 100644
@@ -3049,7 +3049,7 @@ DECLARE_PER_CPU_ALIGNED(struct softnet_data, softnet_data);
 
 static inline int dev_recursion_level(void)
 {
-	return __this_cpu_read(softnet_data.xmit.recursion);
+	return this_cpu_read(softnet_data.xmit.recursion);
 }
 
 #define XMIT_RECURSION_LIMIT	10
Signature · 1 line
2.21.0
Reviewed-by on v1 1/1
#3 lore raw

Re: [PATCH net-next] net: use correct this_cpu primitive in dev_recursion_level

Quoted 10 lines · Florian Westphal

syzbot reports:

BUG: using __this_cpu_read() in preemptible code:
caller is dev_recursion_level include/linux/netdevice.h:3052 [inline]
 __this_cpu_preempt_check+0x246/0x270 lib/smp_processor_id.c:47
 dev_recursion_level include/linux/netdevice.h:3052 [inline]
 ip6_skb_dst_mtu include/net/ip6_route.h:245 [inline]

I erronously downgraded a this_cpu_read to __this_cpu_read when moving dev_recursion_level() around.

maintainer Applied
#4 lore raw

Re: [PATCH net-next] net: use correct this_cpu primitive in dev_recursion_level

In reply to Florian Westphal

From: Florian Westphal <[email protected]>
Date: Wed, 3 Apr 2019 08:28:35 +0200

Quoted 7 lines

syzbot reports:

BUG: using __this_cpu_read() in preemptible code:
caller is dev_recursion_level include/linux/netdevice.h:3052 [inline]
 __this_cpu_preempt_check+0x246/0x270 lib/smp_processor_id.c:47
 dev_recursion_level include/linux/netdevice.h:3052 [inline]
 ip6_skb_dst_mtu include/net/ip6_route.h:245 [inline]
Quoted 6 lines

I erronously downgraded a this_cpu_read to __this_cpu_read when moving dev_recursion_level() around.

Reported-by [email protected]
Fixes 97cdcf37b57e ("net: place xmit recursion in softnet data")

Applied, thanks Florian.

#5 lore raw
In reply to syzbot
Quoted 8 lines · syzbot

syzbot has bisected this bug to:

commit 97cdcf37b57e3f204be3000b9eab9686f38b4356

Author: Florian Westphal <[email protected]>
Date:   Mon Apr 1 14:42:13 2019 +0000

net: place xmit recursion in softnet data

I am seeing this as well on ARMv7 board booted from NFS root (exynos_defconfig):

[ 30.221238] BUG: using __this_cpu_read() in preemptible [00000000]
code: systemd-network/236
[ 30.228576] caller is ip6_output+0x68/0x3e8
[ 30.232578] CPU: 1 PID: 236 Comm: systemd-network Not tainted
5.1.0-rc3-next-20190405 #2
[ 30.240657] Hardware name: SAMSUNG EXYNOS (Flattened Device Tree)
[ 30.246719] [<c011238c>] (unwind_backtrace) from [<c010df50>]
(show_stack+0x10/0x14)
[ 30.254447] [<c010df50>] (show_stack) from [<c0a87270>] (dump_stack+0x98/0xc4)
[ 30.261638] [<c0a87270>] (dump_stack) from [<c0494a94>]
(__this_cpu_preempt_check+0x124/0x128)
[ 30.270238] [<c0494a94>] (__this_cpu_preempt_check) from [<c08e486c>]
(ip6_output+0x68/0x3e8)
[ 30.278730] [<c08e486c>] (ip6_output) from [<c08e52ac>]
(ip6_send_skb+0x30/0x1d0)
[ 30.286180] [<c08e52ac>] (ip6_send_skb) from [<c0912798>]
(rawv6_sendmsg+0x824/0x9c0)
[ 30.294005] [<c0912798>] (rawv6_sendmsg) from [<c07d8ff8>]
(sock_sendmsg+0x14/0x24)
[ 30.301628] [<c07d8ff8>] (sock_sendmsg) from [<c07d97cc>]
(___sys_sendmsg+0x230/0x244)
[ 30.309514] [<c07d97cc>] (___sys_sendmsg) from [<c07da9f8>]
(__sys_sendmsg+0x50/0x8c)
[ 30.317316] [<c07da9f8>] (__sys_sendmsg) from [<c01011ac>]
(__sys_trace_return+0x0/0x14)
[ 30.325376] Exception stack(0xe79f9fa8 to 0xe79f9ff0)
[ 30.330377] 9fa0: 00000000 b6f3da58 0000000b beeb2a6c 00000000 00000000
[ 30.338560] 9fc0: 00000000 b6f3da58 004015ce 00000128 004b10c8
00000000 020d28f9 00000000
[ 30.346708] 9fe0: 004aeee0 beeb2a40 0047cbdc b6d27684

Full log:
https://krzk.eu/#/builders/22/builds/1055/steps/13/logs/serial0

Best regards,
Krzysztof Kozlowski

#6 lore raw
Quoted 5 lines · Krzysztof Kozlowski
Quoted 3 lines · syzbot

syzbot has bisected this bug to:

Quoted 7 lines
Quoted 5 lines

commit 97cdcf37b57e3f204be3000b9eab9686f38b4356

Author: Florian Westphal <[email protected]>
Date:   Mon Apr 1 14:42:13 2019 +0000

net: place xmit recursion in softnet data

I am seeing this as well on ARMv7 board booted from NFS root (exynos_defconfig):

Sorry about this, the fix is now in net-next though:
28b05b92886871bdd ("net: use correct this_cpu primitive in dev_recursion_level")

#7 lore raw
Quoted 11 lines · Florian Westphal
Quoted 9 lines · Krzysztof Kozlowski
Quoted 7 lines · syzbot

syzbot has bisected this bug to:

commit 97cdcf37b57e3f204be3000b9eab9686f38b4356

Author: Florian Westphal <[email protected]>
Date:   Mon Apr 1 14:42:13 2019 +0000
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

net: place xmit recursion in softnet data

I am seeing this as well on ARMv7 board booted from NFS root (exynos_defconfig):

Sorry about this, the fix is now in net-next though:
28b05b92886871bdd ("net: use correct this_cpu primitive in dev_recursion_level")

Krzysztof, just in case, you can see the current bug status on dashboard as well. E.g. for this one it says:

Fix commit: net: use correct this_cpu primitive in dev_recursion_level