ip6_dst_lookup_tail oops

Hit this after around 36 hours of fuzzing.

BUG: unable to handle kernel NULL pointer dereference at 000000000000017e
IP: [<ffffffff81626308>] ip6_dst_lookup_tail+0xe8/0x200
PGD be5fe067 PUD 93459067 PMD 0 
Oops: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
Modules linked in: fuse l2tp_ppp l2tp_core 8021q garp bridge stp dlci binfmt_misc hidp bnep rfcomm ipt_ULOG scsi_transport_iscsi can_raw nfnetlink can_bcm can llc2 af_key netrom af_rxrpc phonet rose caif_socket caif pppoe pppox ax25 nfc ppp_generic decnet appletalk slhc ipx irda p8023 atm psnap x25 crc_ccitt p8022 llc lockd sunrpc ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack nf_conntrack ip6table_filter ip6_tables btusb bluetooth snd_hda_codec_realtek snd_hda_intel snd_hda_codec usb_debug rfkill microcode snd_pcm snd_page_alloc serio_raw snd_timer snd pcspkr edac_core soundcore r8169 mii vhost_net tun macvtap macvlan kvm_amd kvm
CPU 2 
Pid: 10098, comm: trinity-child2 Not tainted 3.8.0-rc3+ #52 Gigabyte Technology Co., Ltd. GA-MA78GM-S2H/GA-MA78GM-S2H
RIP: 0010:[<ffffffff81626308>]  [<ffffffff81626308>] ip6_dst_lookup_tail+0xe8/0x200
RSP: 0018:ffff88008b5b3978  EFLAGS: 00010206
RAX: 0000000000000011 RBX: 0000000000000000 RCX: 0000000000000000
RDX: ffff88008b5b3b10 RSI: ffff880124067600 RDI: ffff880065013800
RBP: ffff88008b5b3a08 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000000 R12: ffff88008b5b3a20
R13: ffff88008b5b3b10 R14: ffff880065013800 R15: ffffffff81cb1980
FS:  00007fd98131d740(0000) GS:ffff88012f200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000000017e CR3: 000000009bef5000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process trinity-child2 (pid: 10098, threadinfo ffff88008b5b2000, task ffff880089e3a490)
Stack:
 ffff880065013800 0000000000000002 0000000000000001 0000000000000000
 ffff88008b5b39a8 ffffffff81332a28 ffff88008b5b39c8 ffffffff8106bdfc
 0000000000000001 ffff880124067600 ffff88008b5b3a08 0000000086d9021e
Call Trace:
 [<ffffffff81332a28>] ? __const_udelay+0x28/0x30
 [<ffffffff8106bdfc>] ? __rcu_read_unlock+0x5c/0xa0
 [<ffffffff8154f065>] ? sk_dst_check+0x5/0x260
 [<ffffffff816266cb>] ip6_sk_dst_lookup_flow+0xcb/0x1b0
 [<ffffffff8164803e>] udpv6_sendmsg+0x66e/0xb80
 [<ffffffff81332a28>] ? __const_udelay+0x28/0x30
 [<ffffffff8106bdfc>] ? __rcu_read_unlock+0x5c/0xa0
 [<ffffffff815e6391>] inet_sendmsg+0x111/0x220
 [<ffffffff815e6285>] ? inet_sendmsg+0x5/0x220
 [<ffffffff81547220>] sock_sendmsg+0xb0/0xe0
 [<ffffffff810b228e>] ? put_lock_stats.isra.23+0xe/0x40
 [<ffffffff810b73a7>] ? lock_release_non_nested+0x2b7/0x2f0
 [<ffffffff815486bc>] __sys_sendmsg+0x3ac/0x3c0
 [<ffffffff810b1ef8>] ? trace_hardirqs_off_caller+0x28/0xc0
 [<ffffffff810b1e22>] ? get_lock_stats+0x22/0x70
 [<ffffffff810b228e>] ? put_lock_stats.isra.23+0xe/0x40
 [<ffffffff8100a1b6>] ? native_sched_clock+0x26/0x90
 [<ffffffff810b1ef8>] ? trace_hardirqs_off_caller+0x28/0xc0
 [<ffffffff8104c494>] ? do_setitimer+0x1c4/0x300
 [<ffffffff810b228e>] ? put_lock_stats.isra.23+0xe/0x40
 [<ffffffff811cbe3a>] ? fget_light+0x3ca/0x500
 [<ffffffff810b830d>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffff8154afb9>] sys_sendmsg+0x49/0x90
 [<ffffffff816a6802>] system_call_fastpath+0x16/0x1b
Code: 00 00 48 8b 5d d8 4c 8b 65 e0 4c 8b 6d e8 4c 8b 75 f0 4c 8b 7d f8 c9 c3 0f 1f 00 49 8b 34 24 48 8b 86 98 00 00 00 48 85 c0 74 c2 <f6> 80 6d 01 00 00 de 75 b9 48 8b 56 18 49 8d 75 24 b9 01 00 00 
RIP  [<ffffffff81626308>] ip6_dst_lookup_tail+0xe8/0x200
 RSP <ffff88008b5b3978>
CR2: 000000000000017e
---[ end trace 439801e1c30eed47 ]---

   0:	f6 80 6d 01 00 00 de 	testb  $0xde,0x16d(%rax)
   7:	75 b9                	jne    0xffffffffffffffc2
   9:	48 8b 56 18          	mov    0x18(%rsi),%rdx
   d:	49 8d 75 24          	lea    0x24(%r13),%rsi
  11:	b9                   	.byte 0xb9
  12:	01 00                	add    %eax,(%rax)
	...

This looks like the GPF in this function I reported last September.
http://www.spinics.net/lists/netdev/msg211894.html

In those reports, I ended up with an rt->n == 0x8000000000000011, but this time, it's just 0x11.

Dave

Now I've hit it with rt->n = 2000000000000010. So I'm starting to think this is getting passed in directly from userspace somehow, as these values look like the output of my 'set a few random bits' routine that sometimes gets called for params.

I'm having trouble mapping a corrupt sendmsg parameter to a messed up rt->n though.

Any clues ?

Dave

Well, neighbor table entries for ipv6 get added over rtnetlink, but it seems you would have to be able to memory map the socket to get a pointer like that into place. Trinity doesn't record the syscalls it makes does it? That might be helpful in tracking this down.
Neil

ah, that's helpful. That explains why just fuzzing sendmsg alone won't hit this.

It does, but when it takes two days to hit something like this, you end up with gigabytes of data to look through.

Dave

Yeah, I expect the failure is going to be preceded by a RTM_NEWNEIGH message at some point, or possibly some combination of RTM_NEWNEIGH and RTM_ROUTE that erroneously sets up that bogus neighbor pointer.

Yeah, thats a bit much. I suppose at this point I'd suggest running a stap script or just modifying the kernel to dump the neighbour table every time you add a neighbor entry in __neigh_create. That might at least get us closer to the point at which we introduce the problem.

Best
Neil