Skip to contentSkip to conversation

NFS related bug in 3.14.54

DISCUSSION Hans-Peter Budek 2 participants 2 messages Oct 21, 2015 2 messages not in the archive

Subsystems (2)
lore
Share & tools
lore
#2 lore raw
In reply to a message not in the archive
Quoted 7 lines · J. Bruce Fields
Quoted 2 lines · Hans-Peter Budek

A diskless station which mounts my NFS server as root device (nolock,nfsvers=3,vers=3)causes a kernel bug on my server:

I don't have the bandwidth to handle all bugs myself, would you mind resending this with a cc: to [email protected]?

Quoted 6 lines

Doesn't ring any bells off the top of my head.

I'd also be curious to know whether this is reproduceable (e.g., if it happens every (or most) times you boot this client, and whether there was some previous kernel version where this didn't happen (so, did this just start happening on an upgrade of your server's kernel).

I checked this about 10 times. This happend every time with a similar kernel log. The contents of some registers and the stack was slightly different but the call trace and the dereference of 0x8 remained the same.
I had to reboot my server each time because it doesn't respond to nfs mount requests after the trap.
The previous kernel version was 3.14.35 which seems to work.

Cheers,
Peter

Quoted 99 lines

--b.

Quoted 95 lines
Oct 21 12:44:34 falco kernel: [64002.882741] NFSD: the nfsdcld client tracking
upcall will be removed in 3.10. Please transition to using nfsdcltrack.
Oct 21 12:46:34 falco kernel: [64123.231017] NFSD: Unable to end grace period: -110
...
Oct 21 13:00:12 falco kernel: [64941.364757] BUG: unable to handle kernel NULL
pointer dereference at 0000000000000008
Oct 21 13:00:12 falco kernel: [64941.364782] IP: [<ffffffff812de742>]
skb_copy_and_csum_datagram_iovec+0x22/0x110
Oct 21 13:00:12 falco kernel: [64941.364804] PGD 3d321067 PUD d9915067 PMD 0
Oct 21 13:00:12 falco kernel: [64941.364819] Oops: 0000 [#1] SMP
Oct 21 13:00:12 falco kernel: [64941.364831] Modules linked in: nfsd auth_rpcgss

oid_registry nfs_acl lockd sunrpc ipv6 sg usb_storage ahci libahci rtc_cmos floppy evdev coretemp it87 hwmon_vid hwmon i2c_i801 acpi_cpufreq processor r8169 mii pcspkr usbhid xhci_hcd uhci_hcd ehci_pci ehci_hcd usbcore usb_common rr2310_00(PO)

Oct 21 13:00:12 falco kernel: [64941.364933] CPU: 0 PID: 23000 Comm: nfsd
Tainted: P           O 3.14.54 #7
Oct 21 13:00:12 falco kernel: [64941.364948] Hardware name: Gigabyte Technology
Co., Ltd. EP35C-DS3R/EP35C-DS3R, BIOS F3 07/17/2008
Oct 21 13:00:12 falco kernel: [64941.364966] task: ffff8801fe3fb170 ti:
ffff88003cd84000 task.ti: ffff88003cd84000
Oct 21 13:00:12 falco kernel: [64941.364982] RIP: 0010:[<ffffffff812de742>]
[<ffffffff812de742>] skb_copy_and_csum_datagram_iovec+0x22/0x110
Oct 21 13:00:12 falco kernel: [64941.365005] RSP: 0018:ffff88003cd85bd0  EFLAGS:
00010202
Oct 21 13:00:12 falco kernel: [64941.365016] RAX: 0000000000000000 RBX:
ffff8800e5e31880 RCX: 00000000000004f8
Oct 21 13:00:12 falco kernel: [64941.365031] RDX: 0000000000000000 RSI:
0000000000001088 RDI: ffff8800d98e9e00
Oct 21 13:00:12 falco kernel: [64941.365046] RBP: 0000000000000008 R08:
0000000000000000 R09: 00000000744d8bb2
Oct 21 13:00:12 falco kernel: [64941.365108] R10: 00000000000004c0 R11:
0000000000000005 R12: ffff8800d98e9e00
Oct 21 13:00:12 falco kernel: [64941.365171] R13: 0000000000001080 R14:
0000000000001080 R15: ffff8800d98e9e00
Oct 21 13:00:12 falco kernel: [64941.365233] FS:  0000000000000000(0000)
GS:ffff880213c00000(0000) knlGS:0000000000000000
Oct 21 13:00:12 falco kernel: [64941.365344] CS:  0010 DS: 0000 ES: 0000 CR0:
000000008005003b
Oct 21 13:00:12 falco kernel: [64941.365403] CR2: 0000000000000008 CR3:
00000000d80dd000 CR4: 00000000000007f0
Oct 21 13:00:12 falco kernel: [64941.365465] Stack:
Oct 21 13:00:12 falco kernel: [64941.365517]  ffff8800d98e9e00 ffffffff812d7f3d
ffff8800e5e31880 ffff88003cd85dc8
Oct 21 13:00:12 falco kernel: [64941.365632]  0000000000000000 0000000000000000
0000000000001080 ffffffff81331f82
Oct 21 13:00:12 falco kernel: [64941.365740]  0000004213c0c130 0000000000000002
ffff8800e5e318f0 ffff880208ee4028
Oct 21 13:00:12 falco kernel: [64941.365740] Call Trace:
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffff812d7f3d>] ?
skb_checksum+0x1d/0x30
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffff81331f82>] ?
udp_recvmsg+0x1e2/0x350
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffff8133c028>] ?
inet_recvmsg+0x48/0x80
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffff812cf802>] ?
sock_recvmsg+0x72/0x90
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffff81042811>] ?
lock_timer_base.isra.31+0x21/0x50
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffff812cf850>] ?
kernel_recvmsg+0x30/0x40
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffffa01c29a4>] ?
svc_udp_recvfrom+0x84/0x3e0 [sunrpc]
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffff81042cca>] ?
del_timer_sync+0x4a/0x60
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffff81371fbf>] ?
schedule_timeout+0x12f/0x1d0
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffffa01ce231>] ?
svc_recv+0x961/0x970 [sunrpc]
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffff8105d240>] ?
wake_up_process+0x30/0x30
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffffa021f66d>] ?
nfsd+0x9d/0x120 [nfsd]
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffffa021f5d0>] ?
nfsd_destroy+0x70/0x70 [nfsd]
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffff810535f8>] ?
kthread+0xb8/0xd0
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffff81053540>] ?
kthread_create_on_node+0x170/0x170
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffff8137b258>] ?
ret_from_fork+0x58/0x90
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffff81053540>] ?
kthread_create_on_node+0x170/0x170
Oct 21 13:00:12 falco kernel: [64941.365740] Code: a5 fe ff ff 0f 1f 44 00 00 41
56 31 c0 41 55 41 54 49 89 fc 55 89 f5 53 48 83 ec 10 8b 77 68 41 89 f5 41 29 ed
0f 84 88 00 00 00 <48> 8b 42 08 48 89 d3 48 85 c0 75 0f 66 90 48 83 c3 10 48 8b 43
Oct 21 13:00:12 falco kernel: [64941.365740] RIP  [<ffffffff812de742>]
skb_copy_and_csum_datagram_iovec+0x22/0x110
Oct 21 13:00:12 falco kernel: [64941.365740]  RSP <ffff88003cd85bd0>
Oct 21 13:00:12 falco kernel: [64941.365740] CR2: 0000000000000008
Oct 21 13:00:12 falco kernel: [64941.367743] ---[ end trace 874f0a58b4dbd906 ]---

Cheers,
Peter

#3 lore raw
Quoted 15 lines · Hans-Peter Budek
Quoted 7 lines · J. Bruce Fields
Quoted 2 lines · Hans-Peter Budek

A diskless station which mounts my NFS server as root device (nolock,nfsvers=3,vers=3)causes a kernel bug on my server:

I don't have the bandwidth to handle all bugs myself, would you mind resending this with a cc: to [email protected]?

Quoted 6 lines

Doesn't ring any bells off the top of my head.

I'd also be curious to know whether this is reproduceable (e.g., if it happens every (or most) times you boot this client, and whether there was some previous kernel version where this didn't happen (so, did this just start happening on an upgrade of your server's kernel).

Quoted 6 lines

I checked this about 10 times. This happend every time with a similar kernel log. The contents of some registers and the stack was slightly different but the call trace and the dereference of 0x8 remained the same.
I had to reboot my server each time because it doesn't respond to nfs mount requests after the trap.
The previous kernel version was 3.14.35 which seems to work.

Between 3.14.35 and 3.14.54 I don't see any interesting changes in fs/nfsd or net/sunrpc.

Given where it happened this could be due to a networking change.

Unless someone comes up with a better idea, another option may just be to a search by bisection between those two versions.

--b.

Quoted 102 lines

Cheers,
Peter

Quoted 99 lines

--b.

Quoted 95 lines
Oct 21 12:44:34 falco kernel: [64002.882741] NFSD: the nfsdcld client tracking
upcall will be removed in 3.10. Please transition to using nfsdcltrack.
Oct 21 12:46:34 falco kernel: [64123.231017] NFSD: Unable to end grace period: -110
...
Oct 21 13:00:12 falco kernel: [64941.364757] BUG: unable to handle kernel NULL
pointer dereference at 0000000000000008
Oct 21 13:00:12 falco kernel: [64941.364782] IP: [<ffffffff812de742>]
skb_copy_and_csum_datagram_iovec+0x22/0x110
Oct 21 13:00:12 falco kernel: [64941.364804] PGD 3d321067 PUD d9915067 PMD 0
Oct 21 13:00:12 falco kernel: [64941.364819] Oops: 0000 [#1] SMP
Oct 21 13:00:12 falco kernel: [64941.364831] Modules linked in: nfsd auth_rpcgss

oid_registry nfs_acl lockd sunrpc ipv6 sg usb_storage ahci libahci rtc_cmos floppy evdev coretemp it87 hwmon_vid hwmon i2c_i801 acpi_cpufreq processor r8169 mii pcspkr usbhid xhci_hcd uhci_hcd ehci_pci ehci_hcd usbcore usb_common rr2310_00(PO)

Oct 21 13:00:12 falco kernel: [64941.364933] CPU: 0 PID: 23000 Comm: nfsd
Tainted: P           O 3.14.54 #7
Oct 21 13:00:12 falco kernel: [64941.364948] Hardware name: Gigabyte Technology
Co., Ltd. EP35C-DS3R/EP35C-DS3R, BIOS F3 07/17/2008
Oct 21 13:00:12 falco kernel: [64941.364966] task: ffff8801fe3fb170 ti:
ffff88003cd84000 task.ti: ffff88003cd84000
Oct 21 13:00:12 falco kernel: [64941.364982] RIP: 0010:[<ffffffff812de742>]
[<ffffffff812de742>] skb_copy_and_csum_datagram_iovec+0x22/0x110
Oct 21 13:00:12 falco kernel: [64941.365005] RSP: 0018:ffff88003cd85bd0  EFLAGS:
00010202
Oct 21 13:00:12 falco kernel: [64941.365016] RAX: 0000000000000000 RBX:
ffff8800e5e31880 RCX: 00000000000004f8
Oct 21 13:00:12 falco kernel: [64941.365031] RDX: 0000000000000000 RSI:
0000000000001088 RDI: ffff8800d98e9e00
Oct 21 13:00:12 falco kernel: [64941.365046] RBP: 0000000000000008 R08:
0000000000000000 R09: 00000000744d8bb2
Oct 21 13:00:12 falco kernel: [64941.365108] R10: 00000000000004c0 R11:
0000000000000005 R12: ffff8800d98e9e00
Oct 21 13:00:12 falco kernel: [64941.365171] R13: 0000000000001080 R14:
0000000000001080 R15: ffff8800d98e9e00
Oct 21 13:00:12 falco kernel: [64941.365233] FS:  0000000000000000(0000)
GS:ffff880213c00000(0000) knlGS:0000000000000000
Oct 21 13:00:12 falco kernel: [64941.365344] CS:  0010 DS: 0000 ES: 0000 CR0:
000000008005003b
Oct 21 13:00:12 falco kernel: [64941.365403] CR2: 0000000000000008 CR3:
00000000d80dd000 CR4: 00000000000007f0
Oct 21 13:00:12 falco kernel: [64941.365465] Stack:
Oct 21 13:00:12 falco kernel: [64941.365517]  ffff8800d98e9e00 ffffffff812d7f3d
ffff8800e5e31880 ffff88003cd85dc8
Oct 21 13:00:12 falco kernel: [64941.365632]  0000000000000000 0000000000000000
0000000000001080 ffffffff81331f82
Oct 21 13:00:12 falco kernel: [64941.365740]  0000004213c0c130 0000000000000002
ffff8800e5e318f0 ffff880208ee4028
Oct 21 13:00:12 falco kernel: [64941.365740] Call Trace:
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffff812d7f3d>] ?
skb_checksum+0x1d/0x30
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffff81331f82>] ?
udp_recvmsg+0x1e2/0x350
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffff8133c028>] ?
inet_recvmsg+0x48/0x80
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffff812cf802>] ?
sock_recvmsg+0x72/0x90
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffff81042811>] ?
lock_timer_base.isra.31+0x21/0x50
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffff812cf850>] ?
kernel_recvmsg+0x30/0x40
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffffa01c29a4>] ?
svc_udp_recvfrom+0x84/0x3e0 [sunrpc]
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffff81042cca>] ?
del_timer_sync+0x4a/0x60
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffff81371fbf>] ?
schedule_timeout+0x12f/0x1d0
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffffa01ce231>] ?
svc_recv+0x961/0x970 [sunrpc]
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffff8105d240>] ?
wake_up_process+0x30/0x30
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffffa021f66d>] ?
nfsd+0x9d/0x120 [nfsd]
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffffa021f5d0>] ?
nfsd_destroy+0x70/0x70 [nfsd]
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffff810535f8>] ?
kthread+0xb8/0xd0
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffff81053540>] ?
kthread_create_on_node+0x170/0x170
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffff8137b258>] ?
ret_from_fork+0x58/0x90
Oct 21 13:00:12 falco kernel: [64941.365740]  [<ffffffff81053540>] ?
kthread_create_on_node+0x170/0x170
Oct 21 13:00:12 falco kernel: [64941.365740] Code: a5 fe ff ff 0f 1f 44 00 00 41
56 31 c0 41 55 41 54 49 89 fc 55 89 f5 53 48 83 ec 10 8b 77 68 41 89 f5 41 29 ed
0f 84 88 00 00 00 <48> 8b 42 08 48 89 d3 48 85 c0 75 0f 66 90 48 83 c3 10 48 8b 43
Oct 21 13:00:12 falco kernel: [64941.365740] RIP  [<ffffffff812de742>]
skb_copy_and_csum_datagram_iovec+0x22/0x110
Oct 21 13:00:12 falco kernel: [64941.365740]  RSP <ffff88003cd85bd0>
Oct 21 13:00:12 falco kernel: [64941.365740] CR2: 0000000000000008
Oct 21 13:00:12 falco kernel: [64941.367743] ---[ end trace 874f0a58b4dbd906 ]---

Cheers,
Peter