IPv6 FIB related crash with MACVLANs in 3.9.11+ kernel.
Subsystems (2)
The kernel has some additional patches, but not much to IPv6.
The bug is that when we have lots of mac-vlans on some ixgbe ports (500 per interface in this case), and boot up the system with the ports unplugged, we get this crash almost every time. Boot-up is going to do normal bootup stuff plus create and configure the 1000 mac-vlans, dump their routing tables, etc.
We are using one routing table per network device, and some ip rules.
If we plug in the ixgbe ports, we do not ever see a crash.
We have not yet tried reproducing it on other drivers, but I suspect the issue is not related to ixgbe.
Any ideas on this one?
Reading symbols from /home/greearb/kernel/2.6/linux-3.9.x64/net/ipv6/ipv6.ko...done. (gdb) l *(fib6_walk_continue+0xd3)0x105c0 is in fib6_walk_continue (/home/greearb/git/linux-3.9.dev.y/net/ipv6/ip6_fib.c:1423).
1418 if (fn == w->root)
1419 return 0;
1420 pn = fn->parent;
1421 w->node = pn;
1422 #ifdef CONFIG_IPV6_SUBTREES
1423 if (FIB6_SUBTREE(pn) == fn) {
1424 WARN_ON(!(fn->fn_flags & RTN_ROOT));
1425 w->state = FWS_L;
1426 continue;
1427 }
(gdb)
[root@lanforge-13100125 ~]# BUG: unable to handle kernel NULL pointer dereference at 0000000000000018
IP: [<ffffffffa00a75c0>] fib6_walk_continue+0xd3/0x13c [ipv6] PGD 4017c4067 PUD 3f3a94067 PMD 0
Oops: 0000 [#1] PREEMPT SMP Modules linked in: nf_nat_ipv4 nf_nat fuse macvlan wanlink(O) pktgen
ip6table_filter ip6_tables ebtable_nat ebtables coretemp mperf intel_powerclamp kvm_intel kvm iTCO_wdt iTCO_vendor_support microcode serio_raw joydev pcspkr i2c_i801 lpc_ich e1000e ixgbe ptp pps_core mdio hwmon dca video shpchp uinput ipv6 mgag200 i2c_algo_bit drm_kms_helper ttm drm i2c_core [last unloaded:iptable_nat]
CPU 7
Pid: 26961, comm: ip Tainted: G C O 3.9.11+ #134 Supermicro
X9SCI/X9SCA/X9SCI/X9SCA
RIP: 0010:[<ffffffffa00a75c0>] [<ffffffffa00a75c0>] fib6_walk_continue+0xd3/0x13c [ipv6] RSP: 0018:ffff880400677a48 EFLAGS: 00010283 RAX: ffff8803f8b08698 RBX: ffff8803f88ea6c0 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff880400677918 RDI: ffff8803f3dde058 RBP: ffff880400677a58 R08: ffff8803f3dde034 R09: ffff8803f3dde000 R10: ffffffff810ca37a R11: ffff88041d5adef8 R12: ffff8803f3a34500 R13: ffffffff81ab5780 R14: ffff8803f88ea6c0 R15: ffff88041bcfc200 FS: 00007f054b30b740(0000) GS:ffff88042fdc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000018 CR3: 00000003f3c8c000 CR4: 00000000001407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process ip (pid: 26961, threadinfo ffff880400676000, task ffff8803ff90aee0) Stack: ffff880400677aa8 ffff8803f248dd00 ffff880400677ad8 ffffffffa00a7815 ffff8803ff90aee0 ffff88041bcfc214 0000000200000000 0000000200000020 ffff8803f3a34500 ffff8803f248dd00 ffffffff81ab5780 0000000000000e70 Call Trace: [<ffffffffa00a7815>] inet6_dump_fib+0x179/0x211 [ipv6] [<ffffffff81535b19>] netlink_dump+0x6b/0x1b2 [<ffffffff81535e2c>] netlink_recvmsg+0x1cc/0x322 [<ffffffff815205d9>] ? rtnetlink_rcv+0x2b/0x2d [<ffffffff814ff3f5>] __sock_recvmsg+0x6a/0x77 [<ffffffff814ff473>] sock_recvmsg+0x71/0x8a [<ffffffff8150aea1>] ? copy_from_user+0x9/0xb [<ffffffff8150b207>] ? verify_iovec+0x54/0xa8 [<ffffffff81500f59>] ___sys_recvmsg+0x13b/0x20d [<ffffffff811602ca>] ? handle_mm_fault+0x536/0x550 [<ffffffff815ce8a6>] ? __do_page_fault+0x307/0x389 [<ffffffff81162789>] ? remove_vma+0x5d/0x65 [<ffffffff8116467d>] ? do_munmap+0x332/0x34c [<ffffffff81501323>] __sys_recvmsg+0x42/0x60 [<ffffffff8150135a>] sys_recvmsg+0x19/0x1b [<ffffffff815d1c99>] system_call_fastpath+0x16/0x1b Code: 89 43 2c e9 61 ff ff ff 48 89 df ff 53 38 85 c0 75 7d ff 43 30 e9 4f ff
ff ff c6 43 28 04 48 3b 43 10 74 69 48 8b 10 48 89 53 18 <48> 39 42 18 75 20 f6 40 2a 02 75 11 be 90 05 00 00 48 c7 c7 2a
RIP [<ffffffffa00a75c0>] fib6_walk_continue+0xd3/0x13c [ipv6]
RSP <ffff880400677a48>
CR2: 0000000000000018
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Hi Ben,
Quoted 11 lines · Ben Greear
The kernel has some additional patches, but not much to IPv6.
The bug is that when we have lots of mac-vlans on some ixgbe ports (500 per interface in this case), and boot up the system with the ports unplugged, we get this crash almost every time. Boot-up is going to do normal bootup stuff plus create and configure the 1000 mac-vlans, dump their routing tables, etc.
We are using one routing table per network device, and some ip rules.
Quoted 6 lines
If we plug in the ixgbe ports, we do not ever see a crash.
We have not yet tried reproducing it on other drivers, but I suspect the issue is not related to ixgbe.
Any ideas on this one?
Could you bring the machine to a panic again with enabling RT6_DEBUG at the top of ip6_fib.c and send a dump of the trace?
Thanks,
Hannes
Quoted 16 lines · Hannes Frederic Sowa
Hi Ben,
Quoted 13 lines · Ben Greear
The kernel has some additional patches, but not much to IPv6.
The bug is that when we have lots of mac-vlans on some ixgbe ports (500 per interface in this case), and boot up the system with the ports unplugged, we get this crash almost every time. Boot-up is going to do normal bootup stuff plus create and configure the 1000 mac-vlans, dump their routing tables, etc.
We are using one routing table per network device, and some ip rules.
If we plug in the ixgbe ports, we do not ever see a crash.
Quoted 7 lines
Quoted 4 lines
We have not yet tried reproducing it on other drivers, but I suspect the issue is not related to ixgbe.
Any ideas on this one?
Could you bring the machine to a panic again with enabling
RT6_DEBUGat the top ofip6_fib.cand send a dump of the trace?
Yes, but it will be a bit until we can create a duplicate machine. We ended up delivering the machine with a note to make sure the interfaces were plugged in (we found the bug hours before shipping the system, of course).
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 22 lines · Ben Greear
Quoted 21 lines · Hannes Frederic Sowa
Hi Ben,
Quoted 17 lines · Ben Greear
The kernel has some additional patches, but not much to IPv6.
The bug is that when we have lots of mac-vlans on some ixgbe ports (500 per interface in this case), and boot up the system with the ports unplugged, we get this crash almost every time. Boot-up is going to do normal bootup stuff plus create and configure the 1000 mac-vlans, dump their routing tables, etc.
We are using one routing table per network device, and some ip rules.
If we plug in the ixgbe ports, we do not ever see a crash.
We have not yet tried reproducing it on other drivers, but I suspect the issue is not related to ixgbe.
Any ideas on this one?
Quoted 7 lines
Quoted 2 lines
Could you bring the machine to a panic again with enabling
RT6_DEBUGat the top ofip6_fib.cand send a dump of the trace?Yes, but it will be a bit until we can create a duplicate machine. We ended up delivering the machine with a note to make sure the interfaces were plugged in (we found the bug hours before shipping the system, of course).
According to my system test guy, it took a lot longer to reproduce the problem with the debug enabled kernel, but I do not see any extra debug messages on the serial console logging or in /var/log/messages
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 26 lines · Ben Greear
Quoted 25 lines · Ben Greear
Quoted 23 lines · Hannes Frederic Sowa
Hi Ben,
Quoted 11 lines · Ben Greear
The kernel has some additional patches, but not much to IPv6.
The bug is that when we have lots of mac-vlans on some ixgbe ports (500 per interface in this case), and boot up the system with the ports unplugged, we get this crash almost every time. Boot-up is going to do normal bootup stuff plus create and configure the 1000 mac-vlans, dump their routing tables, etc.
We are using one routing table per network device, and some ip rules.
Quoted 6 lines
If we plug in the ixgbe ports, we do not ever see a crash.
We have not yet tried reproducing it on other drivers, but I suspect the issue is not related to ixgbe.
Any ideas on this one?
Could you bring the machine to a panic again with enabling
RT6_DEBUGat the top ofip6_fib.cand send a dump of the trace?
Quoted 8 lines
Quoted 4 lines
Yes, but it will be a bit until we can create a duplicate machine. We ended up delivering the machine with a note to make sure the interfaces were plugged in (we found the bug hours before shipping the system, of course).
According to my system test guy, it took a lot longer to reproduce the problem with the debug enabled kernel, but I do not see any extra debug messages on the serial console logging or in
/var/log/messages
Sounds like a race, then, like I thought.
I forgot, those are pr_debugs, I usually enable them with
$ echo file net/ipv6/ip6_fib.c +p > /sys/kernel/debug/dynamic_debug/control
RT6_TRACE is pretty noisy so you should see output immediatley if you do ipv6 traffic. Other way is to specify dyndbg="file net/ipv6/ip6_fib.c +p" on the kernel command line.
Try before doing to play with that until you can confirm the output showes up on the console.
Thanks again,
Hannes
Hi Hannes and Ben,
I encounter the same kernel panic.
Do you guys have any progress on this issue?
Thanks and Best Regards,
Hongmei
Hi Hongmei,
Quoted 4 lines · Hongmei Li
Hi Hannes and Ben,
I encounter the same kernel panic.
Do you guys have any progress on this issue?
Unfortunately not, not looked into it for a long time.
Can you send me a fast reproducer script? I have better machines to reproduce this issue nowadays.
Thanks,
Hannes
Thanks Hannes for your prompt response!
We just encountered this issue two times in our product stability test so far, and my panic backtrace is the exactly same with the one reported here. I don't know how to reproduce the issue till now.
I tried several method, unfortunately, I can not reproduce it by myself. :(
Thanks,
Hongmei
Quoted 6 lines · Hongmei Li
Thanks Hannes for your prompt response!
We just encountered this issue two times in our product stability test so far, and my panic backtrace is the exactly same with the one reported here. I don't know how to reproduce the issue till now.
I tried several method, unfortunately, I can not reproduce it by myself. :(
We could reproduce it easily using our user-space tool that creates 1000 mac-vlans, configures them, dumps routing tables, etc. But, we could only reproduce it when we had the ixgbe ports unplugged. If they were properly connected to a switch, we did not see any crashes.
My original email on 2/3/2014 has more details, and the thread that follows has some info on debugging we did at the time.
We will be happy to test patches...
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 8 lines · Ben Greear
Quoted 6 lines · Hongmei Li
Thanks Hannes for your prompt response!
We just encountered this issue two times in our product stability test so far, and my panic backtrace is the exactly same with the one reported here. I don't know how to reproduce the issue till now.
I tried several method, unfortunately, I can not reproduce it by myself. :(
Quoted 7 lines
We could reproduce it easily using our user-space tool that creates 1000 mac-vlans, configures them, dumps routing tables, etc. But, we could only reproduce it when we had the ixgbe ports unplugged. If they were properly connected to a switch, we did not see any crashes.
My original email on 2/3/2014 has more details, and the thread that follows has some info on debugging we did at the time.
I just tried to reproduce the problem, disabling the port on a switch and setting up 1000 macvlans on a ixgbe, enabling ipv6 and dumping routing tables. Unluckily I had no success. :(
Any more hints?
Bye,
Hannes
Quoted 14 lines · Hannes Frederic Sowa
Quoted 13 lines · Ben Greear
Quoted 6 lines · Hongmei Li
Thanks Hannes for your prompt response!
We just encountered this issue two times in our product stability test so far, and my panic backtrace is the exactly same with the one reported here. I don't know how to reproduce the issue till now.
I tried several method, unfortunately, I can not reproduce it by myself. :(We could reproduce it easily using our user-space tool that creates 1000 mac-vlans, configures them, dumps routing tables, etc. But, we could only reproduce it when we had the ixgbe ports unplugged. If they were properly connected to a switch, we did not see any crashes.
Quoted 8 lines
Quoted 2 lines
My original email on 2/3/2014 has more details, and the thread that follows has some info on debugging we did at the time.
I just tried to reproduce the problem, disabling the port on a switch and setting up 1000 macvlans on a ixgbe, enabling ipv6 and dumping routing tables. Unluckily I had no success. :(
Any more hints?
We are going to be running up to 20 concurrent scripts that will be dumping routes & ips, and configuring ip addresses and routes during bringup.
I don't have a simple stand-alone way to reproduce this, but at least when we reported the problem is was very easy for us to reproduce with our tool.
Maybe 20 scripts running in parallel that randomly configured and dumped routes and ip addresses on random interfaces would do the trick?
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 5 lines · Ben Greear
We are going to be running up to 20 concurrent scripts that will be dumping routes & ips, and configuring ip addresses and routes during bringup.
Quoted 7 lines
I don't have a simple stand-alone way to reproduce this, but at least when we reported the problem is was very easy for us to reproduce with our tool.
Maybe 20 scripts running in parallel that randomly configured and dumped routes and ip addresses on random interfaces would do the trick?
I think the most important question is why is this related with macvlan? IPv6 is L3 while macvlan pure L2, if this is a IPv6 routing bug, it should not be limited to macvlan.
What does your IPv6 routing table look like? And how do you configure those macvlan interfaces?
Quoted 10 lines · Cong Wang
Quoted 9 lines · Ben Greear
We are going to be running up to 20 concurrent scripts that will be dumping routes & ips, and configuring ip addresses and routes during bringup.
I don't have a simple stand-alone way to reproduce this, but at least when we reported the problem is was very easy for us to reproduce with our tool.
Quoted 7 lines
Quoted 3 lines
Maybe 20 scripts running in parallel that randomly configured and dumped routes and ip addresses on random interfaces would do the trick?
I think the most important question is why is this related with macvlan? IPv6 is L3 while macvlan pure L2, if this is a IPv6 routing bug, it should not be limited to macvlan.
We saw it using mac-vlans on top of ixgbe. Probably it can be reproduced elsewhere, but since we had a good test case, we didn't bother trying lots of other combinations.
Just enabling some debug code caused the problem to be much harder to reproduce, so it is probably some sort of race. Maybe lots of mac-vlans make it easier to hit.
Quoted 2 lines
What does your IPv6 routing table look like? And how do you configure those macvlan interfaces?
Each interface would have it's own routing table, with at least as subnet route. I'm not sure we were adding a default gateway or not.
The main routing table would also have some auto-created routes I think.
It is configured using 'ip' to add and dump routes.
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 30 lines · Ben Greear
Quoted 10 lines · Cong Wang
Quoted 9 lines · Ben Greear
We are going to be running up to 20 concurrent scripts that will be dumping routes & ips, and configuring ip addresses and routes during bringup.
I don't have a simple stand-alone way to reproduce this, but at least when we reported the problem is was very easy for us to reproduce with our tool.
Quoted 7 lines
Quoted 3 lines
Maybe 20 scripts running in parallel that randomly configured and dumped routes and ip addresses on random interfaces would do the trick?
I think the most important question is why is this related with macvlan? IPv6 is L3 while macvlan pure L2, if this is a IPv6 routing bug, it should not be limited to macvlan.
We saw it using mac-vlans on top of ixgbe. Probably it can be reproduced elsewhere, but since we had a good test case, we didn't bother trying lots of other combinations.
Just enabling some debug code caused the problem to be much harder to reproduce, so it is probably some sort of race. Maybe lots of mac-vlans make it easier to hit.
Quoted 2 lines
What does your IPv6 routing table look like? And how do you configure those macvlan interfaces?
Quoted 8 lines
Each interface would have it's own routing table, with at least as subnet route. I'm not sure we were adding a default gateway or not.
The main routing table would also have some auto-created routes I think.
It is configured using 'ip' to add and dump routes.
Ben
Just saw this thread. Can you check if this commit makes a difference:
commit 40b8fe45d1f094e3babe7b2dc2b71557ab71401d
Author: Vlad Yasevich <[email protected]> Date: Mon Sep 22 16:34:17 2014 -0400
macvtap: Fix race between device delete and open.
Thanks
-vlad
Quoted 11 lines
Thanks,
Ben--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 40 lines · Vladislav Yasevich
Quoted 30 lines · Ben Greear
Quoted 10 lines · Cong Wang
Quoted 9 lines · Ben Greear
We are going to be running up to 20 concurrent scripts that will be dumping routes & ips, and configuring ip addresses and routes during bringup.
I don't have a simple stand-alone way to reproduce this, but at least when we reported the problem is was very easy for us to reproduce with our tool.
Quoted 7 lines
Quoted 3 lines
Maybe 20 scripts running in parallel that randomly configured and dumped routes and ip addresses on random interfaces would do the trick?
I think the most important question is why is this related with macvlan? IPv6 is L3 while macvlan pure L2, if this is a IPv6 routing bug, it should not be limited to macvlan.
We saw it using mac-vlans on top of ixgbe. Probably it can be reproduced elsewhere, but since we had a good test case, we didn't bother trying lots of other combinations.
Just enabling some debug code caused the problem to be much harder to reproduce, so it is probably some sort of race. Maybe lots of mac-vlans make it easier to hit.
Quoted 2 lines
What does your IPv6 routing table look like? And how do you configure those macvlan interfaces?
Quoted 8 lines
Each interface would have it's own routing table, with at least as subnet route. I'm not sure we were adding a default gateway or not.
The main routing table would also have some auto-created routes I think.
It is configured using 'ip' to add and dump routes.
Quoted 8 lines
Ben
Just saw this thread. Can you check if this commit makes a difference:
commit40b8fe45d1f094e3babe7b2dc2b71557ab71401dAuthor: Vlad Yasevich <[email protected]> Date: Mon Sep 22 16:34:17 2014 -0400macvtap: Fix race between device delete and open.
We can retest this, but we do not use macvtap, and we have higher-priority things to test at the moment, so might be a while.
Thanks,
Ben
Quoted 16 lines
Thanks
-vladQuoted 11 lines
Thanks,
Ben--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com