[3.10.4] NFS locking panic, plus persisting NFS shutdown panic from 3.9.*
Subsystems (3)
I just got this panic on 3.10.4, in the middle of a large parallel compilation (of Chromium, as it happens) over NFSv3:
[16364.527516] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 [16364.527571] IP: [<ffffffff81245157>] nlmclnt_setlockargs+0x55/0xcf [16364.527611] PGD 0 [16364.527626] Oops: 0000 [#1] PREEMPT SMP [16364.527656] Modules linked in: [last unloaded: microcode] [16364.527690] CPU: 0 PID: 17034 Comm: flock Not tainted 3.10.4-05315-gf4ce424-dirty #1 [16364.527730] Hardware name: System manufacturer System Product Name/P8H61-MX USB3, BIOS 0506 08/10/2012 [16364.527775] task: ffff88041a97ad60 ti: ffff8803501d4000 task.ti: ffff8803501d4000 [16364.527813] RIP: 0010:[<ffffffff81245157>] [<ffffffff81245157>] nlmclnt_setlockargs+0x55/0xcf [16364.527860] RSP: 0018:ffff8803501d5c58 EFLAGS: 00010282 [16364.527889] RAX: ffff88041a97ad60 RBX: ffff8803e49c8800 RCX: 0000000000000000 [16364.527926] RDX: 0000000000000000 RSI: 000000000000004a RDI: ffff8803e49c8b54 [16364.527962] RBP: ffff8803501d5c68 R08: 0000000000015720 R09: 0000000000000000 [16364.527998] R10: 00007ffffffff000 R11: ffff8803501d5d58 R12: ffff8803501d5d58 [16364.528034] R13: ffff88041bd2bc00 R14: 0000000000000000 R15: ffff8803fc9e2900 [16364.528070] FS: 0000000000000000(0000) GS:ffff88042fa00000(0000) knlGS:0000000000000000 [16364.528111] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16364.528142] CR2: 0000000000000008 CR3: 0000000001c0b000 CR4: 00000000001407f0 [16364.528177] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16364.528214] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16364.528303] Stack: [16364.528316] ffff8803501d5d58 ffff8803e49c8800 ffff8803501d5cd8 ffffffff81245418 [16364.528369] 0000000000000000 ffff8803516f0bc0 ffff8803d7b7b6c0 ffffffff81215c81 [16364.528418] ffff880300000007 ffff88041bd2bdc8 ffff8801aabe9650 ffff8803fc9e2900 [16364.528467] Call Trace: [16364.528485] [<ffffffff81245418>] nlmclnt_proc+0x148/0x5fb [16364.528516] [<ffffffff81215c81>] ? nfs_put_lock_context+0x69/0x6e [16364.528550] [<ffffffff812209a2>] nfs3_proc_lock+0x21/0x23 [16364.528581] [<ffffffff812149dd>] do_unlk+0x96/0xb2 [16364.528608] [<ffffffff81214b41>] nfs_flock+0x5a/0x71 [16364.528637] [<ffffffff8119a747>] locks_remove_flock+0x9e/0x113 [16364.528668] [<ffffffff8115cc68>] __fput+0xb6/0x1e6 [16364.528695] [<ffffffff8115cda6>] ____fput+0xe/0x10 [16364.528724] [<ffffffff810998da>] task_work_run+0x7e/0x98 [16364.528754] [<ffffffff81082bc5>] do_exit+0x3cc/0x8fa [16364.528782] [<ffffffff81083501>] ? SyS_wait4+0xa5/0xc2 [16364.528811] [<ffffffff8108328d>] do_group_exit+0x6f/0xa2 [16364.528843] [<ffffffff810832d7>] SyS_exit_group+0x17/0x17 [16364.528876] [<ffffffff81613e92>] system_call_fastpath+0x16/0x1b [16364.528907] Code: 00 00 65 48 8b 04 25 c0 b8 00 00 48 8b 72 20 48 81 ee c0 01 00 00 f3 a4 48 8d bb 54 03 00 00 be 4a 00 00 00 48 8b 90 68 05 00 00 <48> 8b 52 08 48 89 bb d0 00 00 00 48 83 c2 45 48 89 53 38 48 8b [16364.529176] RIP [<ffffffff81245157>] nlmclnt_setlockargs+0x55/0xcf [16364.529264] RSP <ffff8803501d5c58> [16364.529283] CR2: 0000000000000008 [16364.539039] ---[ end trace 5a73fddf23441377 ]---
This is the same machine on which this panic has been occurring on shutdown since 3.9.x: Al Viro has previously pointed out the problem and nothing has happened:
[50618.993226] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 [50618.993904] IP: [<ffffffff81165e76>] path_init+0x11c/0x36f [50618.994609] PGD 0 [50618.995329] Oops: 0000 [#1] PREEMPT SMP [50618.996027] Modules linked in: [last unloaded: microcode] [50618.996758] CPU: 3 PID: 1262 Comm: pulseaudio Not tainted 3.10.4-05315-gf4ce424-dirty #1 [50618.997506] Hardware name: System manufacturer System Product Name/P8H61-MX USB3, BIOS 0506 08/10/2012 [50618.998268] task: ffff88041bf1ad60 ti: ffff88041b19e000 task.ti: ffff88041b19e000 [50618.999017] RIP: 0010:[<ffffffff81165e76>] [<ffffffff81165e76>] path_init+0x11c/0x36f [50618.999804] RSP: 0018:ffff88041b19f508 EFLAGS: 00010246 [50619.000592] RAX: 0000000000000000 RBX: ffff88041b19f658 RCX: 000000000000005c [50619.001398] RDX: 0000000000005c5c RSI: ffff880419b3781a RDI: ffffffff81c34a10 [50619.002198] RBP: ffff88041b19f558 R08: ffff88041b19f588 R09: ffff88041b19f7c4 [50619.002999] R10: 00000000ffffff9c R11: ffff88041b19f658 R12: 0000000000000041 [50619.003816] R13: 0000000000000040 R14: ffff880419b3781a R15: ffff88041b19f7c4 [50619.004638] FS: 00007fca19bc2740(0000) GS:ffff88042fac0000(0000) knlGS:0000000000000000 [50619.005465] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [50619.006284] CR2: 0000000000000008 CR3: 0000000001c0b000 CR4: 00000000001407e0 [50619.007092] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [50619.007922] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [50619.008750] Stack: [50619.009576] ffff88041b19f518 00000000ffbfaa5e 0000000000000000 ffffffff8151e735 [50619.010437] ffffc900080ae000 ffff88041b19f658 0000000000000041 ffff880419b3781a [50619.011292] ffff88041b19f628 ffff88041b19f7c4 ffff88041b19f5e8 ffffffff811660fc [50619.012119] Call Trace: [50619.012947] [<ffffffff8151e735>] ? skb_checksum+0x4f/0x25b [50619.013782] [<ffffffff811660fc>] path_lookupat+0x33/0x6c5 [50619.014618] [<ffffffff8152c623>] ? dev_hard_start_xmit+0x2e5/0x50b [50619.015457] [<ffffffff811667b4>] filename_lookup.isra.27+0x26/0x5c [50619.016298] [<ffffffff8116687e>] do_path_lookup+0x33/0x35 [50619.017123] [<ffffffff81166aac>] kern_path+0x2a/0x4d [50619.017973] [<ffffffff815203d8>] ? __alloc_skb+0x75/0x186 [50619.018832] [<ffffffff81520324>] ? __kmalloc_reserve.isra.42+0x2d/0x6c [50619.019702] [<ffffffff815871a3>] unix_find_other+0x38/0x1b9 [50619.020568] [<ffffffff81589043>] unix_stream_connect+0x102/0x3ed [50619.021429] [<ffffffff81518cbc>] ? __sock_create+0x168/0x1c0 [50619.022301] [<ffffffff815de7e3>] ? call_refreshresult+0x91/0x91 [50619.023170] [<ffffffff81516531>] kernel_connect+0x10/0x12 [50619.024047] [<ffffffff815e1d36>] xs_local_setup_socket+0x122/0x191 [50619.024945] [<ffffffff815e2f50>] xs_local_connect+0x2c/0x48 [50619.025849] [<ffffffff815e01f6>] xprt_connect+0x112/0x11b [50619.026756] [<ffffffff815de81c>] call_connect+0x39/0x3b [50619.027662] [<ffffffff815e4e68>] __rpc_execute+0xe8/0x2ca [50619.028567] [<ffffffff815e5109>] rpc_execute+0x76/0x9d [50619.029473] [<ffffffff815debd1>] rpc_run_task+0x78/0x80 [50619.030376] [<ffffffff815ded0f>] rpc_call_sync+0x88/0x9e [50619.031270] [<ffffffff815ebd2f>] rpcb_register_call+0x1f/0x2e [50619.032143] [<ffffffff815ec216>] rpcb_v4_register+0xb2/0x13c [50619.033031] [<ffffffff8108addb>] ? call_timer_fn+0x15e/0x15e [50619.033918] [<ffffffff815e7816>] svc_unregister.isra.11+0x5a/0xcb [50619.034804] [<ffffffff815e789b>] svc_rpcb_cleanup+0x14/0x21 [50619.035706] [<ffffffff815e70ef>] svc_shutdown_net+0x2b/0x30 [50619.036586] [<ffffffff812471c5>] lockd_down_net+0x7f/0xa3 [50619.037465] [<ffffffff81247413>] lockd_down+0x30/0xb2 [50619.038346] [<ffffffff8124439f>] nlmclnt_done+0x1f/0x23 [50619.039227] [<ffffffff8120fd72>] ? nfs_start_lockd+0xc8/0xc8 [50619.040086] [<ffffffff8120fd89>] nfs_destroy_server+0x17/0x19 [50619.040962] [<ffffffff8121024b>] nfs_free_server+0xeb/0x15c [50619.041947] [<ffffffff812172c3>] nfs_kill_super+0x1f/0x23 [50619.042824] [<ffffffff8115da33>] deactivate_locked_super+0x26/0x52 [50619.043696] [<ffffffff8115e73d>] deactivate_super+0x42/0x47 [50619.044562] [<ffffffff8117453e>] mntput_no_expire+0x135/0x13e [50619.045424] [<ffffffff81174574>] mntput+0x2d/0x2f [50619.046287] [<ffffffff8115cd78>] __fput+0x1c6/0x1e6 [50619.047111] [<ffffffff8115cda6>] ____fput+0xe/0x10 [50619.047943] [<ffffffff810998da>] task_work_run+0x7e/0x98 [50619.048764] [<ffffffff81082bc5>] do_exit+0x3cc/0x8fa [50619.049580] [<ffffffff81174449>] ? mntput_no_expire+0x40/0x13e [50619.050399] [<ffffffff8108ca8b>] ? __dequeue_signal+0x1a/0x118 [50619.051215] [<ffffffff8108328d>] do_group_exit+0x6f/0xa2 [50619.052000] [<ffffffff8108f0e7>] get_signal_to_deliver+0x4f2/0x530 [50619.052797] [<ffffffff81036a39>] do_signal+0x4d/0x4a4 [50619.053577] [<ffffffff810f2810>] ? call_rcu+0x17/0x19 [50619.054344] [<ffffffff81036ebc>] do_notify_resume+0x2c/0x6b [50619.055084] [<ffffffff81614098>] int_signal+0x12/0x17 [50619.055852] Code: c7 c7 10 4a c3 81 e8 79 c4 f3 ff e8 99 3a f3 ff 48 83 7b 20 00 0f 85 8d 00 00 00 65 48 8b 04 25 c0 b8 00 00 48 8b 80 58 05 00 00 <8b> 50 08 f6 c2 01 74 04 f3 90 eb f4 48 8b 48 18 48 89 4b 20 48 [50619.057735] RIP [<ffffffff81165e76>] path_init+0x11c/0x36f [50619.058586] RSP <ffff88041b19f508> [50619.059429] CR2: 0000000000000008
.config available on request, but it seems like I've been posting it to l-k with various crashes too often and I don't want to be accused of spamming!
Quoted 39 lines · On Sun, 04 Aug 2013 16:40:58 +0100 Nix <[email protected]> wrote:
I just got this panic on 3.10.4, in the middle of a large parallel compilation (of Chromium, as it happens) over NFSv3:
[16364.527516] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 [16364.527571] IP: [<ffffffff81245157>] nlmclnt_setlockargs+0x55/0xcf [16364.527611] PGD 0 [16364.527626] Oops: 0000 [#1] PREEMPT SMP [16364.527656] Modules linked in: [last unloaded: microcode] [16364.527690] CPU: 0 PID: 17034 Comm: flock Not tainted 3.10.4-05315-gf4ce424-dirty #1 [16364.527730] Hardware name: System manufacturer System Product Name/P8H61-MX USB3, BIOS 0506 08/10/2012 [16364.527775] task: ffff88041a97ad60 ti: ffff8803501d4000 task.ti: ffff8803501d4000 [16364.527813] RIP: 0010:[<ffffffff81245157>] [<ffffffff81245157>] nlmclnt_setlockargs+0x55/0xcf [16364.527860] RSP: 0018:ffff8803501d5c58 EFLAGS: 00010282 [16364.527889] RAX: ffff88041a97ad60 RBX: ffff8803e49c8800 RCX: 0000000000000000 [16364.527926] RDX: 0000000000000000 RSI: 000000000000004a RDI: ffff8803e49c8b54 [16364.527962] RBP: ffff8803501d5c68 R08: 0000000000015720 R09: 0000000000000000 [16364.527998] R10: 00007ffffffff000 R11: ffff8803501d5d58 R12: ffff8803501d5d58 [16364.528034] R13: ffff88041bd2bc00 R14: 0000000000000000 R15: ffff8803fc9e2900 [16364.528070] FS: 0000000000000000(0000) GS:ffff88042fa00000(0000) knlGS:0000000000000000 [16364.528111] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16364.528142] CR2: 0000000000000008 CR3: 0000000001c0b000 CR4: 00000000001407f0 [16364.528177] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16364.528214] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16364.528303] Stack: [16364.528316] ffff8803501d5d58 ffff8803e49c8800 ffff8803501d5cd8 ffffffff81245418 [16364.528369] 0000000000000000 ffff8803516f0bc0 ffff8803d7b7b6c0 ffffffff81215c81 [16364.528418] ffff880300000007 ffff88041bd2bdc8 ffff8801aabe9650 ffff8803fc9e2900 [16364.528467] Call Trace: [16364.528485] [<ffffffff81245418>] nlmclnt_proc+0x148/0x5fb [16364.528516] [<ffffffff81215c81>] ? nfs_put_lock_context+0x69/0x6e [16364.528550] [<ffffffff812209a2>] nfs3_proc_lock+0x21/0x23 [16364.528581] [<ffffffff812149dd>] do_unlk+0x96/0xb2 [16364.528608] [<ffffffff81214b41>] nfs_flock+0x5a/0x71 [16364.528637] [<ffffffff8119a747>] locks_remove_flock+0x9e/0x113 [16364.528668] [<ffffffff8115cc68>] __fput+0xb6/0x1e6 [16364.528695] [<ffffffff8115cda6>] ____fput+0xe/0x10 [16364.528724] [<ffffffff810998da>] task_work_run+0x7e/0x98 [16364.528754] [<ffffffff81082bc5>] do_exit+0x3cc/0x8fa [16364.528782] [<ffffffff81083501>] ? SyS_wait4+0xa5/0xc2
Quoted 9 lines
[16364.528811] [<ffffffff8108328d>] do_group_exit+0x6f/0xa2 [16364.528843] [<ffffffff810832d7>] SyS_exit_group+0x17/0x17 [16364.528876] [<ffffffff81613e92>] system_call_fastpath+0x16/0x1b [16364.528907] Code: 00 00 65 48 8b 04 25 c0 b8 00 00 48 8b 72 20 48 81 ee c0 01 00 00 f3 a4 48 8d bb 54 03 00 00 be 4a 00 00 00 48 8b 90 68 05 00 00 <48> 8b 52 08 48 89 bb d0 00 00 00 48 83 c2 45 48 89 53 38 48 8b [16364.529176] RIP [<ffffffff81245157>] nlmclnt_setlockargs+0x55/0xcf [16364.529264] RSP <ffff8803501d5c58> [16364.529283] CR2: 0000000000000008 [16364.539039] ---[ end trace 5a73fddf23441377 ]---
What might be most helpful is to figure out exactly where the above panic occurred. The instructions here may be helpful:
http://wiki.samba.org/index.php/LinuxCIFS_troubleshooting#Oopses
..but you'll need to replace cifs.ko with lockd.ko in the gdb command.
Quoted 83 lines
This is the same machine on which this panic has been occurring on shutdown since 3.9.x: Al Viro has previously pointed out the problem and nothing has happened:
[50618.993226] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 [50618.993904] IP: [<ffffffff81165e76>] path_init+0x11c/0x36f [50618.994609] PGD 0 [50618.995329] Oops: 0000 [#1] PREEMPT SMP [50618.996027] Modules linked in: [last unloaded: microcode] [50618.996758] CPU: 3 PID: 1262 Comm: pulseaudio Not tainted 3.10.4-05315-gf4ce424-dirty #1 [50618.997506] Hardware name: System manufacturer System Product Name/P8H61-MX USB3, BIOS 0506 08/10/2012 [50618.998268] task: ffff88041bf1ad60 ti: ffff88041b19e000 task.ti: ffff88041b19e000 [50618.999017] RIP: 0010:[<ffffffff81165e76>] [<ffffffff81165e76>] path_init+0x11c/0x36f [50618.999804] RSP: 0018:ffff88041b19f508 EFLAGS: 00010246 [50619.000592] RAX: 0000000000000000 RBX: ffff88041b19f658 RCX: 000000000000005c [50619.001398] RDX: 0000000000005c5c RSI: ffff880419b3781a RDI: ffffffff81c34a10 [50619.002198] RBP: ffff88041b19f558 R08: ffff88041b19f588 R09: ffff88041b19f7c4 [50619.002999] R10: 00000000ffffff9c R11: ffff88041b19f658 R12: 0000000000000041 [50619.003816] R13: 0000000000000040 R14: ffff880419b3781a R15: ffff88041b19f7c4 [50619.004638] FS: 00007fca19bc2740(0000) GS:ffff88042fac0000(0000) knlGS:0000000000000000 [50619.005465] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [50619.006284] CR2: 0000000000000008 CR3: 0000000001c0b000 CR4: 00000000001407e0 [50619.007092] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [50619.007922] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [50619.008750] Stack: [50619.009576] ffff88041b19f518 00000000ffbfaa5e 0000000000000000 ffffffff8151e735 [50619.010437] ffffc900080ae000 ffff88041b19f658 0000000000000041 ffff880419b3781a [50619.011292] ffff88041b19f628 ffff88041b19f7c4 ffff88041b19f5e8 ffffffff811660fc [50619.012119] Call Trace: [50619.012947] [<ffffffff8151e735>] ? skb_checksum+0x4f/0x25b [50619.013782] [<ffffffff811660fc>] path_lookupat+0x33/0x6c5 [50619.014618] [<ffffffff8152c623>] ? dev_hard_start_xmit+0x2e5/0x50b [50619.015457] [<ffffffff811667b4>] filename_lookup.isra.27+0x26/0x5c [50619.016298] [<ffffffff8116687e>] do_path_lookup+0x33/0x35 [50619.017123] [<ffffffff81166aac>] kern_path+0x2a/0x4d [50619.017973] [<ffffffff815203d8>] ? __alloc_skb+0x75/0x186 [50619.018832] [<ffffffff81520324>] ? __kmalloc_reserve.isra.42+0x2d/0x6c [50619.019702] [<ffffffff815871a3>] unix_find_other+0x38/0x1b9 [50619.020568] [<ffffffff81589043>] unix_stream_connect+0x102/0x3ed [50619.021429] [<ffffffff81518cbc>] ? __sock_create+0x168/0x1c0 [50619.022301] [<ffffffff815de7e3>] ? call_refreshresult+0x91/0x91 [50619.023170] [<ffffffff81516531>] kernel_connect+0x10/0x12 [50619.024047] [<ffffffff815e1d36>] xs_local_setup_socket+0x122/0x191 [50619.024945] [<ffffffff815e2f50>] xs_local_connect+0x2c/0x48 [50619.025849] [<ffffffff815e01f6>] xprt_connect+0x112/0x11b [50619.026756] [<ffffffff815de81c>] call_connect+0x39/0x3b [50619.027662] [<ffffffff815e4e68>] __rpc_execute+0xe8/0x2ca [50619.028567] [<ffffffff815e5109>] rpc_execute+0x76/0x9d [50619.029473] [<ffffffff815debd1>] rpc_run_task+0x78/0x80 [50619.030376] [<ffffffff815ded0f>] rpc_call_sync+0x88/0x9e [50619.031270] [<ffffffff815ebd2f>] rpcb_register_call+0x1f/0x2e [50619.032143] [<ffffffff815ec216>] rpcb_v4_register+0xb2/0x13c [50619.033031] [<ffffffff8108addb>] ? call_timer_fn+0x15e/0x15e [50619.033918] [<ffffffff815e7816>] svc_unregister.isra.11+0x5a/0xcb [50619.034804] [<ffffffff815e789b>] svc_rpcb_cleanup+0x14/0x21 [50619.035706] [<ffffffff815e70ef>] svc_shutdown_net+0x2b/0x30 [50619.036586] [<ffffffff812471c5>] lockd_down_net+0x7f/0xa3 [50619.037465] [<ffffffff81247413>] lockd_down+0x30/0xb2 [50619.038346] [<ffffffff8124439f>] nlmclnt_done+0x1f/0x23 [50619.039227] [<ffffffff8120fd72>] ? nfs_start_lockd+0xc8/0xc8 [50619.040086] [<ffffffff8120fd89>] nfs_destroy_server+0x17/0x19 [50619.040962] [<ffffffff8121024b>] nfs_free_server+0xeb/0x15c [50619.041947] [<ffffffff812172c3>] nfs_kill_super+0x1f/0x23 [50619.042824] [<ffffffff8115da33>] deactivate_locked_super+0x26/0x52 [50619.043696] [<ffffffff8115e73d>] deactivate_super+0x42/0x47 [50619.044562] [<ffffffff8117453e>] mntput_no_expire+0x135/0x13e [50619.045424] [<ffffffff81174574>] mntput+0x2d/0x2f [50619.046287] [<ffffffff8115cd78>] __fput+0x1c6/0x1e6 [50619.047111] [<ffffffff8115cda6>] ____fput+0xe/0x10 [50619.047943] [<ffffffff810998da>] task_work_run+0x7e/0x98 [50619.048764] [<ffffffff81082bc5>] do_exit+0x3cc/0x8fa [50619.049580] [<ffffffff81174449>] ? mntput_no_expire+0x40/0x13e [50619.050399] [<ffffffff8108ca8b>] ? __dequeue_signal+0x1a/0x118 [50619.051215] [<ffffffff8108328d>] do_group_exit+0x6f/0xa2 [50619.052000] [<ffffffff8108f0e7>] get_signal_to_deliver+0x4f2/0x530 [50619.052797] [<ffffffff81036a39>] do_signal+0x4d/0x4a4 [50619.053577] [<ffffffff810f2810>] ? call_rcu+0x17/0x19 [50619.054344] [<ffffffff81036ebc>] do_notify_resume+0x2c/0x6b [50619.055084] [<ffffffff81614098>] int_signal+0x12/0x17 [50619.055852] Code: c7 c7 10 4a c3 81 e8 79 c4 f3 ff e8 99 3a f3 ff 48 83 7b 20 00 0f 85 8d 00 00 00 65 48 8b 04 25 c0 b8 00 00 48 8b 80 58 05 00 00 <8b> 50 08 f6 c2 01 74 04 f3 90 eb f4 48 8b 48 18 48 89 4b 20 48 [50619.057735] RIP [<ffffffff81165e76>] path_init+0x11c/0x36f [50619.058586] RSP <ffff88041b19f508> [50619.059429] CR2: 0000000000000008
Quoted 3 lines
.config available on request, but it seems like I've been posting it to l-k with various crashes too often and I don't want to be accused of spamming!
Prob would have been a good idea to cc linux-nfs. It can be easy to
Quoted 5 lines · here's what Al
Quoted 1 line
[ 251.256556] EIP is at path_init+0xc7/0x27fApparently that's
set_root_rcu()withcurrent->fsbeing NULL. Which comes fromAF_UNIXconnect done by some twisted call chain in context of hell knows what.
...and then:
Quoted 3 lines
Why is it done in essentially random process context, anyway? There's such thing as chroot, after all, which would screw that sucker as hard as NULL ->fs, but in a less visible way...
Having not studied the problem, I can't offer up much of an idea on how to fix it at this point.
Signature · 1 line
Jeff Layton <[email protected]>
On 5 Aug 2013, Jeff Layton stated:
Quoted 51 lines
Quoted 39 lines · On Sun, 04 Aug 2013 16:40:58 +0100 Nix <[email protected]> wrote:
I just got this panic on 3.10.4, in the middle of a large parallel compilation (of Chromium, as it happens) over NFSv3:
[16364.527516] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 [16364.527571] IP: [<ffffffff81245157>] nlmclnt_setlockargs+0x55/0xcf [16364.527611] PGD 0 [16364.527626] Oops: 0000 [#1] PREEMPT SMP [16364.527656] Modules linked in: [last unloaded: microcode] [16364.527690] CPU: 0 PID: 17034 Comm: flock Not tainted 3.10.4-05315-gf4ce424-dirty #1 [16364.527730] Hardware name: System manufacturer System Product Name/P8H61-MX USB3, BIOS 0506 08/10/2012 [16364.527775] task: ffff88041a97ad60 ti: ffff8803501d4000 task.ti: ffff8803501d4000 [16364.527813] RIP: 0010:[<ffffffff81245157>] [<ffffffff81245157>] nlmclnt_setlockargs+0x55/0xcf [16364.527860] RSP: 0018:ffff8803501d5c58 EFLAGS: 00010282 [16364.527889] RAX: ffff88041a97ad60 RBX: ffff8803e49c8800 RCX: 0000000000000000 [16364.527926] RDX: 0000000000000000 RSI: 000000000000004a RDI: ffff8803e49c8b54 [16364.527962] RBP: ffff8803501d5c68 R08: 0000000000015720 R09: 0000000000000000 [16364.527998] R10: 00007ffffffff000 R11: ffff8803501d5d58 R12: ffff8803501d5d58 [16364.528034] R13: ffff88041bd2bc00 R14: 0000000000000000 R15: ffff8803fc9e2900 [16364.528070] FS: 0000000000000000(0000) GS:ffff88042fa00000(0000) knlGS:0000000000000000 [16364.528111] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16364.528142] CR2: 0000000000000008 CR3: 0000000001c0b000 CR4: 00000000001407f0 [16364.528177] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16364.528214] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16364.528303] Stack: [16364.528316] ffff8803501d5d58 ffff8803e49c8800 ffff8803501d5cd8 ffffffff81245418 [16364.528369] 0000000000000000 ffff8803516f0bc0 ffff8803d7b7b6c0 ffffffff81215c81 [16364.528418] ffff880300000007 ffff88041bd2bdc8 ffff8801aabe9650 ffff8803fc9e2900 [16364.528467] Call Trace: [16364.528485] [<ffffffff81245418>] nlmclnt_proc+0x148/0x5fb [16364.528516] [<ffffffff81215c81>] ? nfs_put_lock_context+0x69/0x6e [16364.528550] [<ffffffff812209a2>] nfs3_proc_lock+0x21/0x23 [16364.528581] [<ffffffff812149dd>] do_unlk+0x96/0xb2 [16364.528608] [<ffffffff81214b41>] nfs_flock+0x5a/0x71 [16364.528637] [<ffffffff8119a747>] locks_remove_flock+0x9e/0x113 [16364.528668] [<ffffffff8115cc68>] __fput+0xb6/0x1e6 [16364.528695] [<ffffffff8115cda6>] ____fput+0xe/0x10 [16364.528724] [<ffffffff810998da>] task_work_run+0x7e/0x98 [16364.528754] [<ffffffff81082bc5>] do_exit+0x3cc/0x8fa [16364.528782] [<ffffffff81083501>] ? SyS_wait4+0xa5/0xc2Quoted 8 lines
[16364.528811] [<ffffffff8108328d>] do_group_exit+0x6f/0xa2 [16364.528843] [<ffffffff810832d7>] SyS_exit_group+0x17/0x17 [16364.528876] [<ffffffff81613e92>] system_call_fastpath+0x16/0x1b [16364.528907] Code: 00 00 65 48 8b 04 25 c0 b8 00 00 48 8b 72 20 48 81 ee c0 01 00 00 f3 a4 48 8d bb 54 03 00 00 be 4a 00 00 00 48 8b 90 68 05 00 00 <48> 8b 52 08 48 89 bb d0 00 00 00 48 83 c2 45 48 89 53 38 48 8b [16364.529176] RIP [<ffffffff81245157>] nlmclnt_setlockargs+0x55/0xcf [16364.529264] RSP <ffff8803501d5c58> [16364.529283] CR2: 0000000000000008 [16364.539039] ---[ end trace 5a73fddf23441377 ]---
Quoted 2 lines
What might be most helpful is to figure out exactly where the above panic occurred.
OK. My kernel is non-modular and built without debugging information:
rebuilt with debugging info, and got this:
0xffffffff81245418 is in nlmclnt_proc (fs/lockd/clntproc.c:172).
167 return -ENOMEM;
168 } 169 /* Set up the argument struct */ 170 nlmclnt_setlockargs(call, fl);
171
172 if (IS_SETLK(cmd) || IS_SETLKW(cmd)) {
173 if (fl->fl_type != F_UNLCK) {
174 call->a_args.block = IS_SETLKW(cmd) ? 1 : 0;
175 status = nlmclnt_lock(call, fl);
176 } else
That's decimal 328:
0xffffffff81245413 <+323>: callq 0xffffffff81245102 <nlmclnt_setlockargs> 0xffffffff81245418 <+328>: mov -0x40(%rbp),%eax 0xffffffff8124541b <+331>: sub $0x6,%eax 0xffffffff8124541e <+334>: cmp $0x1,%eax
nlm_alloc_call() cannot fail (we have a NULL check right there), and fl also cannot be NULL because it's dereferenced in nfs_flock(), up the call chain from where we are.
Time to stick some printk()s in, I susupect. (Not sure how to keep them from utterly flooding the log, though.)
Quoted 82 lines
Quoted 82 lines
This is the same machine on which this panic has been occurring on shutdown since 3.9.x: Al Viro has previously pointed out the problem and nothing has happened:
[50618.993226] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 [50618.993904] IP: [<ffffffff81165e76>] path_init+0x11c/0x36f [50618.994609] PGD 0 [50618.995329] Oops: 0000 [#1] PREEMPT SMP [50618.996027] Modules linked in: [last unloaded: microcode] [50618.996758] CPU: 3 PID: 1262 Comm: pulseaudio Not tainted 3.10.4-05315-gf4ce424-dirty #1 [50618.997506] Hardware name: System manufacturer System Product Name/P8H61-MX USB3, BIOS 0506 08/10/2012 [50618.998268] task: ffff88041bf1ad60 ti: ffff88041b19e000 task.ti: ffff88041b19e000 [50618.999017] RIP: 0010:[<ffffffff81165e76>] [<ffffffff81165e76>] path_init+0x11c/0x36f [50618.999804] RSP: 0018:ffff88041b19f508 EFLAGS: 00010246 [50619.000592] RAX: 0000000000000000 RBX: ffff88041b19f658 RCX: 000000000000005c [50619.001398] RDX: 0000000000005c5c RSI: ffff880419b3781a RDI: ffffffff81c34a10 [50619.002198] RBP: ffff88041b19f558 R08: ffff88041b19f588 R09: ffff88041b19f7c4 [50619.002999] R10: 00000000ffffff9c R11: ffff88041b19f658 R12: 0000000000000041 [50619.003816] R13: 0000000000000040 R14: ffff880419b3781a R15: ffff88041b19f7c4 [50619.004638] FS: 00007fca19bc2740(0000) GS:ffff88042fac0000(0000) knlGS:0000000000000000 [50619.005465] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [50619.006284] CR2: 0000000000000008 CR3: 0000000001c0b000 CR4: 00000000001407e0 [50619.007092] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [50619.007922] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [50619.008750] Stack: [50619.009576] ffff88041b19f518 00000000ffbfaa5e 0000000000000000 ffffffff8151e735 [50619.010437] ffffc900080ae000 ffff88041b19f658 0000000000000041 ffff880419b3781a [50619.011292] ffff88041b19f628 ffff88041b19f7c4 ffff88041b19f5e8 ffffffff811660fc [50619.012119] Call Trace: [50619.012947] [<ffffffff8151e735>] ? skb_checksum+0x4f/0x25b [50619.013782] [<ffffffff811660fc>] path_lookupat+0x33/0x6c5 [50619.014618] [<ffffffff8152c623>] ? dev_hard_start_xmit+0x2e5/0x50b [50619.015457] [<ffffffff811667b4>] filename_lookup.isra.27+0x26/0x5c [50619.016298] [<ffffffff8116687e>] do_path_lookup+0x33/0x35 [50619.017123] [<ffffffff81166aac>] kern_path+0x2a/0x4d [50619.017973] [<ffffffff815203d8>] ? __alloc_skb+0x75/0x186 [50619.018832] [<ffffffff81520324>] ? __kmalloc_reserve.isra.42+0x2d/0x6c [50619.019702] [<ffffffff815871a3>] unix_find_other+0x38/0x1b9 [50619.020568] [<ffffffff81589043>] unix_stream_connect+0x102/0x3ed [50619.021429] [<ffffffff81518cbc>] ? __sock_create+0x168/0x1c0 [50619.022301] [<ffffffff815de7e3>] ? call_refreshresult+0x91/0x91 [50619.023170] [<ffffffff81516531>] kernel_connect+0x10/0x12 [50619.024047] [<ffffffff815e1d36>] xs_local_setup_socket+0x122/0x191 [50619.024945] [<ffffffff815e2f50>] xs_local_connect+0x2c/0x48 [50619.025849] [<ffffffff815e01f6>] xprt_connect+0x112/0x11b [50619.026756] [<ffffffff815de81c>] call_connect+0x39/0x3b [50619.027662] [<ffffffff815e4e68>] __rpc_execute+0xe8/0x2ca [50619.028567] [<ffffffff815e5109>] rpc_execute+0x76/0x9d [50619.029473] [<ffffffff815debd1>] rpc_run_task+0x78/0x80 [50619.030376] [<ffffffff815ded0f>] rpc_call_sync+0x88/0x9e [50619.031270] [<ffffffff815ebd2f>] rpcb_register_call+0x1f/0x2e [50619.032143] [<ffffffff815ec216>] rpcb_v4_register+0xb2/0x13c [50619.033031] [<ffffffff8108addb>] ? call_timer_fn+0x15e/0x15e [50619.033918] [<ffffffff815e7816>] svc_unregister.isra.11+0x5a/0xcb [50619.034804] [<ffffffff815e789b>] svc_rpcb_cleanup+0x14/0x21 [50619.035706] [<ffffffff815e70ef>] svc_shutdown_net+0x2b/0x30 [50619.036586] [<ffffffff812471c5>] lockd_down_net+0x7f/0xa3 [50619.037465] [<ffffffff81247413>] lockd_down+0x30/0xb2 [50619.038346] [<ffffffff8124439f>] nlmclnt_done+0x1f/0x23 [50619.039227] [<ffffffff8120fd72>] ? nfs_start_lockd+0xc8/0xc8 [50619.040086] [<ffffffff8120fd89>] nfs_destroy_server+0x17/0x19 [50619.040962] [<ffffffff8121024b>] nfs_free_server+0xeb/0x15c [50619.041947] [<ffffffff812172c3>] nfs_kill_super+0x1f/0x23 [50619.042824] [<ffffffff8115da33>] deactivate_locked_super+0x26/0x52 [50619.043696] [<ffffffff8115e73d>] deactivate_super+0x42/0x47 [50619.044562] [<ffffffff8117453e>] mntput_no_expire+0x135/0x13e [50619.045424] [<ffffffff81174574>] mntput+0x2d/0x2f [50619.046287] [<ffffffff8115cd78>] __fput+0x1c6/0x1e6 [50619.047111] [<ffffffff8115cda6>] ____fput+0xe/0x10 [50619.047943] [<ffffffff810998da>] task_work_run+0x7e/0x98 [50619.048764] [<ffffffff81082bc5>] do_exit+0x3cc/0x8fa [50619.049580] [<ffffffff81174449>] ? mntput_no_expire+0x40/0x13e [50619.050399] [<ffffffff8108ca8b>] ? __dequeue_signal+0x1a/0x118 [50619.051215] [<ffffffff8108328d>] do_group_exit+0x6f/0xa2 [50619.052000] [<ffffffff8108f0e7>] get_signal_to_deliver+0x4f2/0x530 [50619.052797] [<ffffffff81036a39>] do_signal+0x4d/0x4a4 [50619.053577] [<ffffffff810f2810>] ? call_rcu+0x17/0x19 [50619.054344] [<ffffffff81036ebc>] do_notify_resume+0x2c/0x6b [50619.055084] [<ffffffff81614098>] int_signal+0x12/0x17 [50619.055852] Code: c7 c7 10 4a c3 81 e8 79 c4 f3 ff e8 99 3a f3 ff 48 83 7b 20 00 0f 85 8d 00 00 00 65 48 8b 04 25 c0 b8 00 00 48 8b 80 58 05 00 00 <8b> 50 08 f6 c2 01 74 04 f3 90 eb f4 48 8b 48 18 48 89 4b 20 48 [50619.057735] RIP [<ffffffff81165e76>] path_init+0x11c/0x36f [50619.058586] RSP <ffff88041b19f508> [50619.059429] CR2: 0000000000000008
[...]
Quoted 6 lines
Quoted 3 lines
Why is it done in essentially random process context, anyway? There's such thing as chroot, after all, which would screw that sucker as hard as NULL ->fs, but in a less visible way...
Having not studied the problem, I can't offer up much of an idea on how to fix it at this point.
What mystifies me is why a v3 server shutdown and unregistration is triggering a v4 registration. It's not really that surprising that a registration that late in shutdown, after the disconnected fs has been destroyed, would cause problems... (I have NFSv4 built in, but am not using it yet.)
My .config seems useful at this juncture, if people on this list haven't seen it:
CONFIG_64BIT=yCONFIG_X86_64=yCONFIG_X86=yCONFIG_INSTRUCTION_DECODER=yCONFIG_OUTPUT_FORMAT="elf64-x86-64"CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"CONFIG_LOCKDEP_SUPPORT=yCONFIG_STACKTRACE_SUPPORT=yCONFIG_HAVE_LATENCYTOP_SUPPORT=yCONFIG_MMU=yCONFIG_NEED_DMA_MAP_STATE=yCONFIG_NEED_SG_DMA_LENGTH=yCONFIG_GENERIC_ISA_DMA=yCONFIG_GENERIC_BUG=yCONFIG_GENERIC_BUG_RELATIVE_POINTERS=yCONFIG_GENERIC_HWEIGHT=yCONFIG_ARCH_MAY_HAVE_PC_FDC=yCONFIG_RWSEM_XCHGADD_ALGORITHM=yCONFIG_GENERIC_CALIBRATE_DELAY=yCONFIG_ARCH_HAS_CPU_RELAX=yCONFIG_ARCH_HAS_CACHE_LINE_SIZE=yCONFIG_ARCH_HAS_CPU_AUTOPROBE=yCONFIG_HAVE_SETUP_PER_CPU_AREA=yCONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=yCONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=yCONFIG_ARCH_HIBERNATION_POSSIBLE=yCONFIG_ARCH_SUSPEND_POSSIBLE=yCONFIG_ZONE_DMA32=yCONFIG_AUDIT_ARCH=yCONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=yCONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=yCONFIG_HAVE_INTEL_TXT=yCONFIG_X86_64_SMP=yCONFIG_X86_HT=yCONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-rdi -fcall-saved-rsi -fcall-saved-rdx -fcall-saved-rcx -fcall-saved-r8 -fcall-saved-r9 -fcall-saved-r10 -fcall-saved-r11" CONFIG_ARCH_CPU_PROBE_RELEASE=yCONFIG_ARCH_SUPPORTS_UPROBES=yCONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"CONFIG_IRQ_WORK=yCONFIG_BUILDTIME_EXTABLE_SORT=yCONFIG_INIT_ENV_ARG_LIMIT=32CONFIG_CROSS_COMPILE=""CONFIG_LOCALVERSION=""CONFIG_LOCALVERSION_AUTO=yCONFIG_HAVE_KERNEL_GZIP=yCONFIG_HAVE_KERNEL_BZIP2=yCONFIG_HAVE_KERNEL_LZMA=yCONFIG_HAVE_KERNEL_XZ=yCONFIG_HAVE_KERNEL_LZO=yCONFIG_KERNEL_LZMA=yCONFIG_DEFAULT_HOSTNAME="mutilate"CONFIG_SWAP=yCONFIG_SYSVIPC=yCONFIG_SYSVIPC_SYSCTL=yCONFIG_POSIX_MQUEUE=yCONFIG_POSIX_MQUEUE_SYSCTL=yCONFIG_FHANDLE=yCONFIG_AUDIT=yCONFIG_HAVE_GENERIC_HARDIRQS=yCONFIG_GENERIC_HARDIRQS=yCONFIG_GENERIC_IRQ_PROBE=yCONFIG_GENERIC_IRQ_SHOW=yCONFIG_GENERIC_PENDING_IRQ=yCONFIG_IRQ_DOMAIN=yCONFIG_IRQ_FORCED_THREADING=yCONFIG_SPARSE_IRQ=yCONFIG_CLOCKSOURCE_WATCHDOG=yCONFIG_ARCH_CLOCKSOURCE_DATA=yCONFIG_GENERIC_TIME_VSYSCALL=yCONFIG_GENERIC_CLOCKEVENTS=yCONFIG_GENERIC_CLOCKEVENTS_BUILD=yCONFIG_GENERIC_CLOCKEVENTS_BROADCAST=yCONFIG_GENERIC_CLOCKEVENTS_MIN_ADJUST=yCONFIG_GENERIC_CMOS_UPDATE=yCONFIG_TICK_ONESHOT=yCONFIG_NO_HZ_COMMON=yCONFIG_NO_HZ_IDLE=yCONFIG_NO_HZ=yCONFIG_HIGH_RES_TIMERS=yCONFIG_IRQ_TIME_ACCOUNTING=yCONFIG_BSD_PROCESS_ACCT=yCONFIG_TASKSTATS=yCONFIG_TASK_DELAY_ACCT=yCONFIG_TASK_XACCT=yCONFIG_TASK_IO_ACCOUNTING=yCONFIG_TREE_PREEMPT_RCU=yCONFIG_PREEMPT_RCU=yCONFIG_RCU_STALL_COMMON=yCONFIG_RCU_FANOUT=8CONFIG_RCU_FANOUT_LEAF=8CONFIG_RCU_BOOST=yCONFIG_RCU_BOOST_PRIO=1CONFIG_RCU_BOOST_DELAY=500CONFIG_LOG_BUF_SHIFT=18CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=yCONFIG_ARCH_SUPPORTS_NUMA_BALANCING=yCONFIG_ARCH_WANTS_PROT_NUMA_PROT_NONE=yCONFIG_CGROUPS=yCONFIG_CGROUP_SCHED=yCONFIG_FAIR_GROUP_SCHED=yCONFIG_NAMESPACES=yCONFIG_PID_NS=yCONFIG_NET_NS=yCONFIG_UIDGID_CONVERTED=yCONFIG_SCHED_AUTOGROUP=yCONFIG_RELAY=yCONFIG_BLK_DEV_INITRD=yCONFIG_INITRAMFS_SOURCE="usr/initramfs.mutilate"CONFIG_INITRAMFS_ROOT_UID=99CONFIG_INITRAMFS_ROOT_GID=101CONFIG_RD_GZIP=yCONFIG_RD_BZIP2=yCONFIG_RD_LZMA=yCONFIG_RD_XZ=yCONFIG_RD_LZO=yCONFIG_INITRAMFS_COMPRESSION_LZMA=yCONFIG_CC_OPTIMIZE_FOR_SIZE=yCONFIG_SYSCTL=yCONFIG_ANON_INODES=yCONFIG_HAVE_UID16=yCONFIG_SYSCTL_EXCEPTION_TRACE=yCONFIG_HOTPLUG=yCONFIG_HAVE_PCSPKR_PLATFORM=yCONFIG_UID16=yCONFIG_KALLSYMS=yCONFIG_KALLSYMS_ALL=yCONFIG_PRINTK=yCONFIG_BUG=yCONFIG_ELF_CORE=yCONFIG_PCSPKR_PLATFORM=yCONFIG_BASE_FULL=yCONFIG_FUTEX=yCONFIG_EPOLL=yCONFIG_SIGNALFD=yCONFIG_TIMERFD=yCONFIG_EVENTFD=yCONFIG_SHMEM=yCONFIG_AIO=yCONFIG_PCI_QUIRKS=yCONFIG_HAVE_PERF_EVENTS=yCONFIG_PERF_EVENTS=yCONFIG_VM_EVENT_COUNTERS=yCONFIG_SLUB_DEBUG=yCONFIG_SLUB=yCONFIG_TRACEPOINTS=yCONFIG_HAVE_OPROFILE=yCONFIG_OPROFILE_NMI_TIMER=yCONFIG_JUMP_LABEL=yCONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=yCONFIG_ARCH_USE_BUILTIN_BSWAP=yCONFIG_USER_RETURN_NOTIFIER=yCONFIG_HAVE_IOREMAP_PROT=yCONFIG_HAVE_KPROBES=yCONFIG_HAVE_KRETPROBES=yCONFIG_HAVE_OPTPROBES=yCONFIG_HAVE_KPROBES_ON_FTRACE=yCONFIG_HAVE_ARCH_TRACEHOOK=yCONFIG_HAVE_DMA_ATTRS=yCONFIG_USE_GENERIC_SMP_HELPERS=yCONFIG_GENERIC_SMP_IDLE_THREAD=yCONFIG_HAVE_REGS_AND_STACK_ACCESS_API=yCONFIG_HAVE_DMA_API_DEBUG=yCONFIG_HAVE_HW_BREAKPOINT=yCONFIG_HAVE_MIXED_BREAKPOINTS_REGS=yCONFIG_HAVE_USER_RETURN_NOTIFIER=yCONFIG_HAVE_PERF_EVENTS_NMI=yCONFIG_HAVE_PERF_REGS=yCONFIG_HAVE_PERF_USER_STACK_DUMP=yCONFIG_HAVE_ARCH_JUMP_LABEL=yCONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG=yCONFIG_HAVE_ALIGNED_STRUCT_PAGE=yCONFIG_HAVE_CMPXCHG_LOCAL=yCONFIG_HAVE_CMPXCHG_DOUBLE=yCONFIG_ARCH_WANT_COMPAT_IPC_PARSE_VERSION=yCONFIG_ARCH_WANT_OLD_COMPAT_IPC=yCONFIG_HAVE_ARCH_SECCOMP_FILTER=yCONFIG_SECCOMP_FILTER=yCONFIG_HAVE_CONTEXT_TRACKING=yCONFIG_HAVE_IRQ_TIME_ACCOUNTING=yCONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=yCONFIG_MODULES_USE_ELF_RELA=yCONFIG_OLD_SIGSUSPEND3=yCONFIG_COMPAT_OLD_SIGACTION=yCONFIG_SLABINFO=yCONFIG_RT_MUTEXES=yCONFIG_BASE_SMALL=0CONFIG_MODULES=yCONFIG_MODULE_UNLOAD=yCONFIG_STOP_MACHINE=yCONFIG_BLOCK=yCONFIG_BLK_DEV_BSG=yCONFIG_PARTITION_ADVANCED=yCONFIG_MSDOS_PARTITION=yCONFIG_EFI_PARTITION=yCONFIG_BLOCK_COMPAT=yCONFIG_IOSCHED_NOOP=yCONFIG_IOSCHED_DEADLINE=mCONFIG_IOSCHED_CFQ=yCONFIG_DEFAULT_CFQ=yCONFIG_DEFAULT_IOSCHED="cfq"CONFIG_PREEMPT_NOTIFIERS=yCONFIG_UNINLINE_SPIN_UNLOCK=yCONFIG_MUTEX_SPIN_ON_OWNER=yCONFIG_FREEZER=yCONFIG_ZONE_DMA=yCONFIG_SMP=yCONFIG_X86_SUPPORTS_MEMORY_FAILURE=yCONFIG_SCHED_OMIT_FRAME_POINTER=yCONFIG_NO_BOOTMEM=yCONFIG_MCORE2=yCONFIG_X86_INTERNODE_CACHE_SHIFT=6CONFIG_X86_L1_CACHE_SHIFT=6CONFIG_X86_INTEL_USERCOPY=yCONFIG_X86_USE_PPRO_CHECKSUM=yCONFIG_X86_P6_NOP=yCONFIG_X86_TSC=yCONFIG_X86_CMPXCHG64=yCONFIG_X86_CMOV=yCONFIG_X86_MINIMUM_CPU_FAMILY=64CONFIG_X86_DEBUGCTLMSR=yCONFIG_CPU_SUP_INTEL=yCONFIG_CPU_SUP_AMD=yCONFIG_CPU_SUP_CENTAUR=yCONFIG_HPET_TIMER=yCONFIG_HPET_EMULATE_RTC=yCONFIG_DMI=yCONFIG_GART_IOMMU=yCONFIG_SWIOTLB=yCONFIG_IOMMU_HELPER=yCONFIG_NR_CPUS=8CONFIG_SCHED_SMT=yCONFIG_SCHED_MC=yCONFIG_PREEMPT=yCONFIG_PREEMPT_COUNT=yCONFIG_X86_LOCAL_APIC=yCONFIG_X86_IO_APIC=yCONFIG_X86_MCE=yCONFIG_X86_MCE_INTEL=yCONFIG_X86_MCE_THRESHOLD=yCONFIG_X86_THERMAL_VECTOR=yCONFIG_MICROCODE=mCONFIG_MICROCODE_INTEL=yCONFIG_MICROCODE_OLD_INTERFACE=yCONFIG_MICROCODE_INTEL_LIB=yCONFIG_X86_MSR=mCONFIG_X86_CPUID=yCONFIG_ARCH_PHYS_ADDR_T_64BIT=yCONFIG_ARCH_DMA_ADDR_T_64BIT=yCONFIG_DIRECT_GBPAGES=yCONFIG_ARCH_SPARSEMEM_ENABLE=yCONFIG_ARCH_SPARSEMEM_DEFAULT=yCONFIG_ARCH_SELECT_MEMORY_MODEL=yCONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000CONFIG_SELECT_MEMORY_MODEL=yCONFIG_SPARSEMEM_MANUAL=yCONFIG_SPARSEMEM=yCONFIG_HAVE_MEMORY_PRESENT=yCONFIG_SPARSEMEM_EXTREME=yCONFIG_SPARSEMEM_VMEMMAP_ENABLE=yCONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=yCONFIG_SPARSEMEM_VMEMMAP=yCONFIG_HAVE_MEMBLOCK=yCONFIG_HAVE_MEMBLOCK_NODE_MAP=yCONFIG_ARCH_DISCARD_MEMBLOCK=yCONFIG_PAGEFLAGS_EXTENDED=yCONFIG_SPLIT_PTLOCK_CPUS=4CONFIG_COMPACTION=yCONFIG_MIGRATION=yCONFIG_PHYS_ADDR_T_64BIT=yCONFIG_ZONE_DMA_FLAG=1CONFIG_BOUNCE=yCONFIG_VIRT_TO_BUS=yCONFIG_MMU_NOTIFIER=yCONFIG_KSM=yCONFIG_DEFAULT_MMAP_MIN_ADDR=4096CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=yCONFIG_TRANSPARENT_HUGEPAGE=yCONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=yCONFIG_CROSS_MEMORY_ATTACH=yCONFIG_X86_CHECK_BIOS_CORRUPTION=yCONFIG_X86_BOOTPARAM_MEMORY_CORRUPTION_CHECK=yCONFIG_X86_RESERVE_LOW=64CONFIG_MTRR=yCONFIG_X86_PAT=yCONFIG_ARCH_USES_PG_UNCACHED=yCONFIG_ARCH_RANDOM=yCONFIG_X86_SMAP=yCONFIG_EFI=yCONFIG_SECCOMP=yCONFIG_HZ_1000=yCONFIG_HZ=1000CONFIG_SCHED_HRTICK=yCONFIG_PHYSICAL_START=0x1000000CONFIG_PHYSICAL_ALIGN=0x1000000CONFIG_HOTPLUG_CPU=yCONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=yCONFIG_ARCH_HIBERNATION_HEADER=yCONFIG_HIBERNATE_CALLBACKS=yCONFIG_HIBERNATION=yCONFIG_PM_STD_PARTITION=""CONFIG_TOI_CORE=yCONFIG_TOI_SWAP=yCONFIG_TOI_CRYPTO=yCONFIG_TOI_USERUI=yCONFIG_TOI_USERUI_DEFAULT_PATH="/usr/sbin/tuxoniceui_text"CONFIG_TOI_DEFAULT_IMAGE_SIZE_LIMIT=-2CONFIG_TOI_REPLACE_SWSUSP=yCONFIG_TOI_IGNORE_LATE_INITCALL=yCONFIG_TOI_DEFAULT_WAIT=-1CONFIG_TOI_DEFAULT_EXTRA_PAGES_ALLOWANCE=50000CONFIG_TOI_CHECKSUM=yCONFIG_TOI=yCONFIG_PM_SLEEP=yCONFIG_PM_SLEEP_SMP=yCONFIG_PM_RUNTIME=yCONFIG_PM=yCONFIG_ACPI=yCONFIG_ACPI_SLEEP=yCONFIG_ACPI_PROC_EVENT=yCONFIG_ACPI_BUTTON=yCONFIG_ACPI_FAN=yCONFIG_ACPI_I2C=yCONFIG_ACPI_PROCESSOR=yCONFIG_ACPI_HOTPLUG_CPU=yCONFIG_ACPI_THERMAL=yCONFIG_ACPI_CUSTOM_DSDT_FILE=""CONFIG_ACPI_BLACKLIST_YEAR=0CONFIG_X86_PM_TIMER=yCONFIG_ACPI_CONTAINER=yCONFIG_ACPI_HED=yCONFIG_ACPI_APEI=yCONFIG_ACPI_APEI_GHES=yCONFIG_ACPI_APEI_PCIEAER=yCONFIG_CPU_FREQ=yCONFIG_CPU_FREQ_TABLE=yCONFIG_CPU_FREQ_GOV_COMMON=yCONFIG_CPU_FREQ_STAT=yCONFIG_CPU_FREQ_STAT_DETAILS=yCONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=yCONFIG_CPU_FREQ_GOV_PERFORMANCE=yCONFIG_CPU_FREQ_GOV_ONDEMAND=yCONFIG_X86_INTEL_PSTATE=yCONFIG_X86_ACPI_CPUFREQ=yCONFIG_CPU_IDLE=yCONFIG_CPU_IDLE_GOV_LADDER=yCONFIG_CPU_IDLE_GOV_MENU=yCONFIG_INTEL_IDLE=yCONFIG_I7300_IDLE_IOAT_CHANNEL=yCONFIG_I7300_IDLE=yCONFIG_PCI=yCONFIG_PCI_DIRECT=yCONFIG_PCI_MMCONFIG=yCONFIG_PCI_DOMAINS=yCONFIG_PCIEPORTBUS=yCONFIG_PCIEAER=yCONFIG_PCIEASPM=yCONFIG_PCIEASPM_DEFAULT=yCONFIG_PCIE_PME=yCONFIG_ARCH_SUPPORTS_MSI=yCONFIG_PCI_MSI=yCONFIG_HT_IRQ=yCONFIG_PCI_ATS=yCONFIG_PCI_IOV=yCONFIG_PCI_IOAPIC=yCONFIG_PCI_LABEL=yCONFIG_ISA_DMA_API=yCONFIG_AMD_NB=yCONFIG_BINFMT_ELF=yCONFIG_COMPAT_BINFMT_ELF=yCONFIG_ARCH_BINFMT_ELF_RANDOMIZE_PIE=yCONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS=yCONFIG_BINFMT_SCRIPT=yCONFIG_BINFMT_MISC=yCONFIG_COREDUMP=yCONFIG_IA32_EMULATION=yCONFIG_COMPAT=yCONFIG_COMPAT_FOR_U64_ALIGNMENT=yCONFIG_SYSVIPC_COMPAT=yCONFIG_KEYS_COMPAT=yCONFIG_HAVE_TEXT_POKE_SMP=yCONFIG_X86_DEV_DMA_OPS=yCONFIG_NET=yCONFIG_PACKET=yCONFIG_PACKET_DIAG=yCONFIG_UNIX=yCONFIG_UNIX_DIAG=yCONFIG_INET=yCONFIG_IP_MULTICAST=yCONFIG_INET_LRO=yCONFIG_INET_DIAG=yCONFIG_INET_TCP_DIAG=yCONFIG_INET_UDP_DIAG=yCONFIG_TCP_CONG_CUBIC=yCONFIG_DEFAULT_TCP_CONG="cubic"CONFIG_IPV6=yCONFIG_IPV6_PRIVACY=yCONFIG_HAVE_NET_DSA=yCONFIG_DNS_RESOLVER=yCONFIG_NETLINK_DIAG=yCONFIG_RPS=yCONFIG_RFS_ACCEL=yCONFIG_XPS=yCONFIG_BQL=yCONFIG_BPF_JIT=yCONFIG_BT=yCONFIG_BT_RFCOMM=yCONFIG_BT_HCIBTUSB=yCONFIG_HAVE_BPF_JIT=yCONFIG_UEVENT_HELPER_PATH=""CONFIG_PREVENT_FIRMWARE_BUILD=yCONFIG_FW_LOADER=yCONFIG_FIRMWARE_IN_KERNEL=yCONFIG_EXTRA_FIRMWARE="radeon/BARTS_mc.bin radeon/BARTS_me.bin radeon/BARTS_pfp.bin radeon/SUMO_uvd.bin radeon/BTC_rlc.bin rtl_nic/rtl8168f-1.fw rtl_nic/rtl8168f-2.fw" CONFIG_EXTRA_FIRMWARE_DIR="/usr/src/linux/linux-firmware"CONFIG_FW_LOADER_USER_HELPER=yCONFIG_DMA_SHARED_BUFFER=yCONFIG_PNP=yCONFIG_PNPACPI=yCONFIG_BLK_DEV=yCONFIG_BLK_DEV_LOOP=yCONFIG_BLK_DEV_LOOP_MIN_COUNT=8CONFIG_BLK_DEV_CRYPTOLOOP=mCONFIG_BLK_DEV_NBD=mCONFIG_CDROM_PKTCDVD=yCONFIG_CDROM_PKTCDVD_BUFFERS=16CONFIG_HAVE_IDE=yCONFIG_SCSI_MOD=yCONFIG_SCSI=yCONFIG_SCSI_DMA=yCONFIG_BLK_DEV_SD=yCONFIG_BLK_DEV_SR=yCONFIG_SCSI_MULTI_LUN=yCONFIG_SCSI_SCAN_ASYNC=yCONFIG_ATA=yCONFIG_ATA_VERBOSE_ERROR=yCONFIG_ATA_ACPI=yCONFIG_SATA_ZPODD=yCONFIG_SATA_AHCI=yCONFIG_MD=yCONFIG_BLK_DEV_MD=yCONFIG_MD_RAID1=yCONFIG_BLK_DEV_DM=yCONFIG_DM_SNAPSHOT=yCONFIG_DM_MIRROR=yCONFIG_DM_ZERO=yCONFIG_NETDEVICES=yCONFIG_NET_CORE=yCONFIG_DUMMY=mCONFIG_MII=yCONFIG_NETCONSOLE=yCONFIG_NETCONSOLE_DYNAMIC=yCONFIG_NETPOLL=yCONFIG_NET_POLL_CONTROLLER=yCONFIG_TUN=yCONFIG_VHOST_NET=yCONFIG_VHOST_RING=yCONFIG_ETHERNET=yCONFIG_NET_VENDOR_REALTEK=yCONFIG_R8169=yCONFIG_USB_USBNET=mCONFIG_USB_NET_CDCETHER=mCONFIG_INPUT=yCONFIG_INPUT_POLLDEV=yCONFIG_INPUT_MOUSEDEV=yCONFIG_INPUT_MOUSEDEV_SCREEN_X=1680CONFIG_INPUT_MOUSEDEV_SCREEN_Y=1050CONFIG_INPUT_JOYDEV=yCONFIG_INPUT_EVDEV=yCONFIG_INPUT_KEYBOARD=yCONFIG_KEYBOARD_ATKBD=yCONFIG_INPUT_MOUSE=yCONFIG_MOUSE_PS2=yCONFIG_MOUSE_PS2_ALPS=yCONFIG_MOUSE_PS2_LOGIPS2PP=yCONFIG_MOUSE_PS2_SYNAPTICS=yCONFIG_MOUSE_PS2_CYPRESS=yCONFIG_MOUSE_PS2_LIFEBOOK=yCONFIG_MOUSE_PS2_TRACKPOINT=yCONFIG_SERIO=yCONFIG_SERIO_I8042=yCONFIG_SERIO_LIBPS2=yCONFIG_TTY=yCONFIG_VT=yCONFIG_CONSOLE_TRANSLATIONS=yCONFIG_VT_CONSOLE=yCONFIG_VT_CONSOLE_SLEEP=yCONFIG_HW_CONSOLE=yCONFIG_UNIX98_PTYS=yCONFIG_SERIAL_8250=yCONFIG_SERIAL_8250_PNP=yCONFIG_FIX_EARLYCON_MEM=yCONFIG_SERIAL_8250_PCI=yCONFIG_SERIAL_8250_NR_UARTS=4CONFIG_SERIAL_8250_RUNTIME_UARTS=4CONFIG_SERIAL_CORE=yCONFIG_HW_RANDOM=yCONFIG_HW_RANDOM_INTEL=yCONFIG_NVRAM=mCONFIG_HPET=yCONFIG_HPET_MMAP=yCONFIG_DEVPORT=yCONFIG_I2C=yCONFIG_I2C_BOARDINFO=yCONFIG_I2C_CHARDEV=yCONFIG_I2C_HELPER_AUTO=yCONFIG_I2C_ALGOBIT=yCONFIG_I2C_I801=yCONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=yCONFIG_GPIO_DEVRES=yCONFIG_POWER_SUPPLY=yCONFIG_HWMON=yCONFIG_HWMON_VID=mCONFIG_SENSORS_CORETEMP=yCONFIG_SENSORS_IT87=mCONFIG_SENSORS_W83627EHF=mCONFIG_THERMAL=yCONFIG_THERMAL_HWMON=yCONFIG_THERMAL_DEFAULT_GOV_STEP_WISE=yCONFIG_THERMAL_GOV_STEP_WISE=yCONFIG_SSB_POSSIBLE=yCONFIG_BCMA_POSSIBLE=yCONFIG_MFD_CORE=yCONFIG_LPC_ICH=yCONFIG_MEDIA_SUPPORT=yCONFIG_MEDIA_CAMERA_SUPPORT=yCONFIG_VIDEO_DEV=yCONFIG_VIDEO_V4L2=yCONFIG_VIDEOBUF2_CORE=yCONFIG_VIDEOBUF2_MEMOPS=yCONFIG_VIDEOBUF2_VMALLOC=yCONFIG_MEDIA_USB_SUPPORT=yCONFIG_USB_VIDEO_CLASS=yCONFIG_USB_VIDEO_CLASS_INPUT_EVDEV=yCONFIG_MEDIA_SUBDRV_AUTOSELECT=yCONFIG_AGP=yCONFIG_VGA_ARB=yCONFIG_VGA_ARB_MAX_GPUS=2CONFIG_DRM=yCONFIG_DRM_KMS_HELPER=yCONFIG_DRM_TTM=yCONFIG_DRM_RADEON=yCONFIG_HDMI=yCONFIG_FB=yCONFIG_FB_CFB_FILLRECT=yCONFIG_FB_CFB_COPYAREA=yCONFIG_FB_CFB_IMAGEBLIT=yCONFIG_BACKLIGHT_LCD_SUPPORT=yCONFIG_BACKLIGHT_CLASS_DEVICE=yCONFIG_VGA_CONSOLE=yCONFIG_VGACON_SOFT_SCROLLBACK=yCONFIG_VGACON_SOFT_SCROLLBACK_SIZE=512CONFIG_DUMMY_CONSOLE=yCONFIG_FRAMEBUFFER_CONSOLE=yCONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY=y
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=yCONFIG_SOUND=yCONFIG_SOUND_OSS_CORE=yCONFIG_SND=yCONFIG_SND_TIMER=yCONFIG_SND_PCM=yCONFIG_SND_JACK=yCONFIG_SND_SEQUENCER=yCONFIG_SND_SEQ_DUMMY=mCONFIG_SND_OSSEMUL=yCONFIG_SND_MIXER_OSS=yCONFIG_SND_PCM_OSS=yCONFIG_SND_PCM_OSS_PLUGINS=yCONFIG_SND_SEQUENCER_OSS=yCONFIG_SND_HRTIMER=yCONFIG_SND_SEQ_HRTIMER_DEFAULT=yCONFIG_SND_DYNAMIC_MINORS=yCONFIG_SND_VERBOSE_PROCFS=yCONFIG_SND_VMASTER=yCONFIG_SND_KCTL_JACK=yCONFIG_SND_DMA_SGBUF=yCONFIG_SND_PCI=yCONFIG_SND_HDA_INTEL=yCONFIG_SND_HDA_PREALLOC_SIZE=2048CONFIG_SND_HDA_INPUT_JACK=yCONFIG_SND_HDA_GENERIC=yCONFIG_SND_HDA_POWER_SAVE_DEFAULT=0CONFIG_HID=yCONFIG_HID_GENERIC=yCONFIG_HID_A4TECH=yCONFIG_HID_APPLE=yCONFIG_HID_BELKIN=yCONFIG_HID_CHERRY=yCONFIG_HID_CHICONY=yCONFIG_HID_CYPRESS=yCONFIG_HID_EZKEY=yCONFIG_HID_KENSINGTON=yCONFIG_HID_LOGITECH=yCONFIG_HID_MICROSOFT=yCONFIG_HID_MONTEREY=yCONFIG_USB_HID=yCONFIG_USB_ARCH_HAS_OHCI=yCONFIG_USB_ARCH_HAS_EHCI=yCONFIG_USB_ARCH_HAS_XHCI=yCONFIG_USB_SUPPORT=yCONFIG_USB_COMMON=yCONFIG_USB_ARCH_HAS_HCD=yCONFIG_USB=yCONFIG_USB_DEFAULT_PERSIST=yCONFIG_USB_DYNAMIC_MINORS=yCONFIG_USB_XHCI_HCD=yCONFIG_USB_EHCI_HCD=yCONFIG_USB_EHCI_PCI=yCONFIG_USB_STORAGE=yCONFIG_NEW_LEDS=yCONFIG_LEDS_CLASS=yCONFIG_RTC_LIB=yCONFIG_RTC_CLASS=yCONFIG_RTC_HCTOSYS=yCONFIG_RTC_SYSTOHC=yCONFIG_RTC_HCTOSYS_DEVICE="rtc0"CONFIG_RTC_INTF_SYSFS=yCONFIG_RTC_INTF_PROC=yCONFIG_RTC_INTF_DEV=yCONFIG_RTC_DRV_CMOS=yCONFIG_CLKEVT_I8253=yCONFIG_I8253_LOCK=yCONFIG_CLKBLD_I8253=yCONFIG_IOMMU_API=yCONFIG_IOMMU_SUPPORT=yCONFIG_DMAR_TABLE=yCONFIG_INTEL_IOMMU=yCONFIG_INTEL_IOMMU_DEFAULT_ON=yCONFIG_INTEL_IOMMU_FLOPPY_WA=yCONFIG_IRQ_REMAP=yCONFIG_FIRMWARE_MEMMAP=yCONFIG_DMIID=yCONFIG_EFI_VARS=yCONFIG_EFI_VARS_PSTORE=yCONFIG_DCACHE_WORD_ACCESS=yCONFIG_EXT4_FS=yCONFIG_EXT4_USE_FOR_EXT23=yCONFIG_EXT4_FS_POSIX_ACL=yCONFIG_JBD2=yCONFIG_FS_MBCACHE=yCONFIG_FS_POSIX_ACL=yCONFIG_EXPORTFS=yCONFIG_FILE_LOCKING=yCONFIG_FSNOTIFY=yCONFIG_DNOTIFY=yCONFIG_INOTIFY_USER=yCONFIG_FANOTIFY=yCONFIG_QUOTA=yCONFIG_QUOTA_NETLINK_INTERFACE=yCONFIG_PRINT_QUOTA_WARNING=yCONFIG_QUOTA_TREE=yCONFIG_QFMT_V2=yCONFIG_QUOTACTL=yCONFIG_QUOTACTL_COMPAT=yCONFIG_FUSE_FS=yCONFIG_CUSE=yCONFIG_GENERIC_ACL=yCONFIG_ISO9660_FS=yCONFIG_JOLIET=yCONFIG_UDF_FS=yCONFIG_UDF_NLS=yCONFIG_FAT_FS=mCONFIG_MSDOS_FS=mCONFIG_VFAT_FS=mCONFIG_FAT_DEFAULT_CODEPAGE=437CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"CONFIG_PROC_FS=yCONFIG_PROC_SYSCTL=yCONFIG_PROC_PAGE_MONITOR=yCONFIG_SYSFS=yCONFIG_TMPFS=yCONFIG_TMPFS_POSIX_ACL=yCONFIG_TMPFS_XATTR=yCONFIG_HUGETLBFS=yCONFIG_HUGETLB_PAGE=yCONFIG_CONFIGFS_FS=yCONFIG_MISC_FILESYSTEMS=yCONFIG_PSTORE=yCONFIG_NETWORK_FILESYSTEMS=yCONFIG_NFS_FS=yCONFIG_NFS_V3=yCONFIG_NFS_V3_ACL=yCONFIG_NFS_V4=yCONFIG_NFS_USE_KERNEL_DNS=yCONFIG_NFSD=yCONFIG_NFSD_V2_ACL=yCONFIG_NFSD_V3=yCONFIG_NFSD_V3_ACL=yCONFIG_LOCKD=yCONFIG_LOCKD_V4=yCONFIG_NFS_ACL_SUPPORT=yCONFIG_NFS_COMMON=yCONFIG_SUNRPC=yCONFIG_SUNRPC_GSS=yCONFIG_NLS=yCONFIG_NLS_DEFAULT="iso8859-1"CONFIG_NLS_CODEPAGE_437=yCONFIG_NLS_ASCII=mCONFIG_NLS_ISO8859_1=yCONFIG_NLS_ISO8859_15=mCONFIG_NLS_UTF8=mCONFIG_TRACE_IRQFLAGS_SUPPORT=yCONFIG_PRINTK_TIME=yCONFIG_DEFAULT_MESSAGE_LOGLEVEL=4CONFIG_ENABLE_WARN_DEPRECATED=yCONFIG_ENABLE_MUST_CHECK=yCONFIG_FRAME_WARN=1024CONFIG_STRIP_ASM_SYMS=yCONFIG_DEBUG_FS=yCONFIG_DEBUG_KERNEL=yCONFIG_LOCKUP_DETECTOR=yCONFIG_HARDLOCKUP_DETECTOR=yCONFIG_BOOTPARAM_HARDLOCKUP_PANIC_VALUE=0CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=0CONFIG_PANIC_ON_OOPS_VALUE=0CONFIG_DETECT_HUNG_TASK=yCONFIG_DEFAULT_HUNG_TASK_TIMEOUT=120CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0CONFIG_SCHED_DEBUG=yCONFIG_SCHEDSTATS=yCONFIG_TIMER_STATS=yCONFIG_HAVE_DEBUG_KMEMLEAK=yCONFIG_STACKTRACE=yCONFIG_DEBUG_BUGVERBOSE=yCONFIG_DEBUG_MEMORY_INIT=yCONFIG_ARCH_WANT_FRAME_POINTERS=yCONFIG_FRAME_POINTER=yCONFIG_RCU_CPU_STALL_TIMEOUT=60CONFIG_RCU_CPU_STALL_VERBOSE=yCONFIG_LATENCYTOP=yCONFIG_ARCH_HAS_DEBUG_STRICT_USER_COPY_CHECKS=yCONFIG_DEBUG_STRICT_USER_COPY_CHECKS=yCONFIG_USER_STACKTRACE_SUPPORT=yCONFIG_NOP_TRACER=yCONFIG_HAVE_FUNCTION_TRACER=yCONFIG_HAVE_FUNCTION_GRAPH_TRACER=yCONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=yCONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=yCONFIG_HAVE_DYNAMIC_FTRACE=yCONFIG_HAVE_DYNAMIC_FTRACE_WITH_REGS=yCONFIG_HAVE_FTRACE_MCOUNT_RECORD=yCONFIG_HAVE_SYSCALL_TRACEPOINTS=yCONFIG_HAVE_FENTRY=yCONFIG_HAVE_C_RECORDMCOUNT=yCONFIG_TRACE_CLOCK=yCONFIG_RING_BUFFER=yCONFIG_EVENT_TRACING=yCONFIG_CONTEXT_SWITCH_TRACER=yCONFIG_TRACING=yCONFIG_GENERIC_TRACER=yCONFIG_TRACING_SUPPORT=yCONFIG_FTRACE=yCONFIG_FUNCTION_TRACER=yCONFIG_BRANCH_PROFILE_NONE=yCONFIG_BLK_DEV_IO_TRACE=yCONFIG_DYNAMIC_FTRACE=yCONFIG_DYNAMIC_FTRACE_WITH_REGS=yCONFIG_FTRACE_MCOUNT_RECORD=yCONFIG_HAVE_ARCH_KGDB=yCONFIG_HAVE_ARCH_KMEMCHECK=yCONFIG_STRICT_DEVMEM=yCONFIG_X86_VERBOSE_BOOTUP=yCONFIG_EARLY_PRINTK=yCONFIG_DEBUG_RODATA=yCONFIG_HAVE_MMIOTRACE_SUPPORT=yCONFIG_IO_DELAY_TYPE_0X80=0CONFIG_IO_DELAY_TYPE_0XED=1CONFIG_IO_DELAY_TYPE_UDELAY=2CONFIG_IO_DELAY_TYPE_NONE=3CONFIG_IO_DELAY_0X80=yCONFIG_DEFAULT_IO_DELAY_TYPE=0CONFIG_KEYS=yCONFIG_SECURITY=yCONFIG_SECURITYFS=yCONFIG_SECURITY_NETWORK=yCONFIG_SECURITY_PATH=yCONFIG_SECURITY_APPARMOR=yCONFIG_SECURITY_APPARMOR_BOOTPARAM_VALUE=0CONFIG_DEFAULT_SECURITY_DAC=yCONFIG_DEFAULT_SECURITY=""CONFIG_CRYPTO=yCONFIG_CRYPTO_ALGAPI=yCONFIG_CRYPTO_ALGAPI2=yCONFIG_CRYPTO_AEAD2=yCONFIG_CRYPTO_BLKCIPHER=yCONFIG_CRYPTO_BLKCIPHER2=yCONFIG_CRYPTO_HASH=yCONFIG_CRYPTO_HASH2=yCONFIG_CRYPTO_RNG2=yCONFIG_CRYPTO_PCOMP2=yCONFIG_CRYPTO_MANAGER=yCONFIG_CRYPTO_MANAGER2=yCONFIG_CRYPTO_MANAGER_DISABLE_TESTS=yCONFIG_CRYPTO_GF128MUL=yCONFIG_CRYPTO_WORKQUEUE=yCONFIG_CRYPTO_CRYPTD=yCONFIG_CRYPTO_ABLK_HELPER_X86=yCONFIG_CRYPTO_GLUE_HELPER_X86=yCONFIG_CRYPTO_CBC=yCONFIG_CRYPTO_ECB=yCONFIG_CRYPTO_LRW=yCONFIG_CRYPTO_XTS=yCONFIG_CRYPTO_CRC32C=yCONFIG_CRYPTO_CRC32C_INTEL=yCONFIG_CRYPTO_CRC32_PCLMUL=yCONFIG_CRYPTO_MD4=yCONFIG_CRYPTO_SHA256=yCONFIG_CRYPTO_AES=yCONFIG_CRYPTO_SERPENT=yCONFIG_CRYPTO_SERPENT_AVX_X86_64=yCONFIG_CRYPTO_TWOFISH_COMMON=yCONFIG_CRYPTO_TWOFISH_X86_64=yCONFIG_CRYPTO_TWOFISH_X86_64_3WAY=yCONFIG_CRYPTO_TWOFISH_AVX_X86_64=yCONFIG_CRYPTO_LZO=yCONFIG_HAVE_KVM=yCONFIG_HAVE_KVM_IRQCHIP=yCONFIG_HAVE_KVM_IRQ_ROUTING=yCONFIG_HAVE_KVM_EVENTFD=yCONFIG_KVM_APIC_ARCHITECTURE=yCONFIG_KVM_MMIO=yCONFIG_KVM_ASYNC_PF=yCONFIG_HAVE_KVM_MSI=yCONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT=yCONFIG_VIRTUALIZATION=yCONFIG_KVM=yCONFIG_KVM_INTEL=yCONFIG_KVM_DEVICE_ASSIGNMENT=yCONFIG_BINARY_PRINTF=yCONFIG_BITREVERSE=yCONFIG_GENERIC_STRNCPY_FROM_USER=yCONFIG_GENERIC_STRNLEN_USER=yCONFIG_GENERIC_FIND_FIRST_BIT=yCONFIG_GENERIC_PCI_IOMAP=yCONFIG_GENERIC_IOMAP=yCONFIG_GENERIC_IO=yCONFIG_CRC16=yCONFIG_CRC_ITU_T=yCONFIG_CRC32=yCONFIG_CRC32_SLICEBY8=yCONFIG_ZLIB_INFLATE=yCONFIG_LZO_COMPRESS=yCONFIG_LZO_DECOMPRESS=yCONFIG_XZ_DEC=yCONFIG_XZ_DEC_X86=yCONFIG_XZ_DEC_POWERPC=yCONFIG_XZ_DEC_IA64=yCONFIG_XZ_DEC_ARM=yCONFIG_XZ_DEC_ARMTHUMB=yCONFIG_XZ_DEC_SPARC=yCONFIG_XZ_DEC_BCJ=yCONFIG_DECOMPRESS_GZIP=yCONFIG_DECOMPRESS_BZIP2=yCONFIG_DECOMPRESS_LZMA=yCONFIG_DECOMPRESS_XZ=yCONFIG_DECOMPRESS_LZO=yCONFIG_GENERIC_ALLOCATOR=yCONFIG_HAS_IOMEM=yCONFIG_HAS_IOPORT=yCONFIG_HAS_DMA=yCONFIG_CHECK_SIGNATURE=yCONFIG_CPU_RMAP=yCONFIG_DQL=yCONFIG_NLATTR=yCONFIG_ARCH_HAS_ATOMIC64_DEC_IF_POSITIVE=yCONFIG_OID_REGISTRY=yCONFIG_UCS2_STRING=y
Quoted 172 lines · On Mon, 05 Aug 2013 15:48:01 +0100 Nix <[email protected]> wrote:
On 5 Aug 2013, Jeff Layton stated:
Quoted 51 lines
Quoted 39 lines · On Sun, 04 Aug 2013 16:40:58 +0100 Nix <[email protected]> wrote:
I just got this panic on 3.10.4, in the middle of a large parallel compilation (of Chromium, as it happens) over NFSv3:
[16364.527516] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 [16364.527571] IP: [<ffffffff81245157>] nlmclnt_setlockargs+0x55/0xcf [16364.527611] PGD 0 [16364.527626] Oops: 0000 [#1] PREEMPT SMP [16364.527656] Modules linked in: [last unloaded: microcode] [16364.527690] CPU: 0 PID: 17034 Comm: flock Not tainted 3.10.4-05315-gf4ce424-dirty #1 [16364.527730] Hardware name: System manufacturer System Product Name/P8H61-MX USB3, BIOS 0506 08/10/2012 [16364.527775] task: ffff88041a97ad60 ti: ffff8803501d4000 task.ti: ffff8803501d4000 [16364.527813] RIP: 0010:[<ffffffff81245157>] [<ffffffff81245157>] nlmclnt_setlockargs+0x55/0xcf [16364.527860] RSP: 0018:ffff8803501d5c58 EFLAGS: 00010282 [16364.527889] RAX: ffff88041a97ad60 RBX: ffff8803e49c8800 RCX: 0000000000000000 [16364.527926] RDX: 0000000000000000 RSI: 000000000000004a RDI: ffff8803e49c8b54 [16364.527962] RBP: ffff8803501d5c68 R08: 0000000000015720 R09: 0000000000000000 [16364.527998] R10: 00007ffffffff000 R11: ffff8803501d5d58 R12: ffff8803501d5d58 [16364.528034] R13: ffff88041bd2bc00 R14: 0000000000000000 R15: ffff8803fc9e2900 [16364.528070] FS: 0000000000000000(0000) GS:ffff88042fa00000(0000) knlGS:0000000000000000 [16364.528111] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16364.528142] CR2: 0000000000000008 CR3: 0000000001c0b000 CR4: 00000000001407f0 [16364.528177] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16364.528214] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16364.528303] Stack: [16364.528316] ffff8803501d5d58 ffff8803e49c8800 ffff8803501d5cd8 ffffffff81245418 [16364.528369] 0000000000000000 ffff8803516f0bc0 ffff8803d7b7b6c0 ffffffff81215c81 [16364.528418] ffff880300000007 ffff88041bd2bdc8 ffff8801aabe9650 ffff8803fc9e2900 [16364.528467] Call Trace: [16364.528485] [<ffffffff81245418>] nlmclnt_proc+0x148/0x5fb [16364.528516] [<ffffffff81215c81>] ? nfs_put_lock_context+0x69/0x6e [16364.528550] [<ffffffff812209a2>] nfs3_proc_lock+0x21/0x23 [16364.528581] [<ffffffff812149dd>] do_unlk+0x96/0xb2 [16364.528608] [<ffffffff81214b41>] nfs_flock+0x5a/0x71 [16364.528637] [<ffffffff8119a747>] locks_remove_flock+0x9e/0x113 [16364.528668] [<ffffffff8115cc68>] __fput+0xb6/0x1e6 [16364.528695] [<ffffffff8115cda6>] ____fput+0xe/0x10 [16364.528724] [<ffffffff810998da>] task_work_run+0x7e/0x98 [16364.528754] [<ffffffff81082bc5>] do_exit+0x3cc/0x8fa [16364.528782] [<ffffffff81083501>] ? SyS_wait4+0xa5/0xc2Quoted 8 lines
[16364.528811] [<ffffffff8108328d>] do_group_exit+0x6f/0xa2 [16364.528843] [<ffffffff810832d7>] SyS_exit_group+0x17/0x17 [16364.528876] [<ffffffff81613e92>] system_call_fastpath+0x16/0x1b [16364.528907] Code: 00 00 65 48 8b 04 25 c0 b8 00 00 48 8b 72 20 48 81 ee c0 01 00 00 f3 a4 48 8d bb 54 03 00 00 be 4a 00 00 00 48 8b 90 68 05 00 00 <48> 8b 52 08 48 89 bb d0 00 00 00 48 83 c2 45 48 89 53 38 48 8b [16364.529176] RIP [<ffffffff81245157>] nlmclnt_setlockargs+0x55/0xcf [16364.529264] RSP <ffff8803501d5c58> [16364.529283] CR2: 0000000000000008 [16364.539039] ---[ end trace 5a73fddf23441377 ]---Quoted 2 lines
What might be most helpful is to figure out exactly where the above panic occurred.
OK. My kernel is non-modular and built without debugging information:
rebuilt with debugging info, and got this:
0xffffffff81245418is innlmclnt_proc(fs/lockd/clntproc.c:172).
167 return-ENOMEM;168 } 169 /* Set up the argument struct */ 170 nlmclnt_setlockargs(call, fl);171
172 if (IS_SETLK(cmd)||IS_SETLKW(cmd)) {
173 if (fl->fl_type!=F_UNLCK) {
174call->a_args.block=IS_SETLKW(cmd)? 1 : 0;
175 status =nlmclnt_lock(call, fl);
176 } elseThat's decimal 328:
0xffffffff81245413 <+323>: callq 0xffffffff81245102 <nlmclnt_setlockargs> 0xffffffff81245418 <+328>: mov -0x40(%rbp),%eax 0xffffffff8124541b <+331>: sub $0x6,%eax 0xffffffff8124541e <+334>: cmp $0x1,%eax
nlm_alloc_call()cannot fail (we have a NULL check right there), and fl also cannot be NULL because it's dereferenced innfs_flock(), up the call chain from where we are.Time to stick some printk()s in, I susupect. (Not sure how to keep them from utterly flooding the log, though.)
Quoted 82 lines
Quoted 82 lines
This is the same machine on which this panic has been occurring on shutdown since 3.9.x: Al Viro has previously pointed out the problem and nothing has happened:
[50618.993226] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 [50618.993904] IP: [<ffffffff81165e76>] path_init+0x11c/0x36f [50618.994609] PGD 0 [50618.995329] Oops: 0000 [#1] PREEMPT SMP [50618.996027] Modules linked in: [last unloaded: microcode] [50618.996758] CPU: 3 PID: 1262 Comm: pulseaudio Not tainted 3.10.4-05315-gf4ce424-dirty #1 [50618.997506] Hardware name: System manufacturer System Product Name/P8H61-MX USB3, BIOS 0506 08/10/2012 [50618.998268] task: ffff88041bf1ad60 ti: ffff88041b19e000 task.ti: ffff88041b19e000 [50618.999017] RIP: 0010:[<ffffffff81165e76>] [<ffffffff81165e76>] path_init+0x11c/0x36f [50618.999804] RSP: 0018:ffff88041b19f508 EFLAGS: 00010246 [50619.000592] RAX: 0000000000000000 RBX: ffff88041b19f658 RCX: 000000000000005c [50619.001398] RDX: 0000000000005c5c RSI: ffff880419b3781a RDI: ffffffff81c34a10 [50619.002198] RBP: ffff88041b19f558 R08: ffff88041b19f588 R09: ffff88041b19f7c4 [50619.002999] R10: 00000000ffffff9c R11: ffff88041b19f658 R12: 0000000000000041 [50619.003816] R13: 0000000000000040 R14: ffff880419b3781a R15: ffff88041b19f7c4 [50619.004638] FS: 00007fca19bc2740(0000) GS:ffff88042fac0000(0000) knlGS:0000000000000000 [50619.005465] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [50619.006284] CR2: 0000000000000008 CR3: 0000000001c0b000 CR4: 00000000001407e0 [50619.007092] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [50619.007922] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [50619.008750] Stack: [50619.009576] ffff88041b19f518 00000000ffbfaa5e 0000000000000000 ffffffff8151e735 [50619.010437] ffffc900080ae000 ffff88041b19f658 0000000000000041 ffff880419b3781a [50619.011292] ffff88041b19f628 ffff88041b19f7c4 ffff88041b19f5e8 ffffffff811660fc [50619.012119] Call Trace: [50619.012947] [<ffffffff8151e735>] ? skb_checksum+0x4f/0x25b [50619.013782] [<ffffffff811660fc>] path_lookupat+0x33/0x6c5 [50619.014618] [<ffffffff8152c623>] ? dev_hard_start_xmit+0x2e5/0x50b [50619.015457] [<ffffffff811667b4>] filename_lookup.isra.27+0x26/0x5c [50619.016298] [<ffffffff8116687e>] do_path_lookup+0x33/0x35 [50619.017123] [<ffffffff81166aac>] kern_path+0x2a/0x4d [50619.017973] [<ffffffff815203d8>] ? __alloc_skb+0x75/0x186 [50619.018832] [<ffffffff81520324>] ? __kmalloc_reserve.isra.42+0x2d/0x6c [50619.019702] [<ffffffff815871a3>] unix_find_other+0x38/0x1b9 [50619.020568] [<ffffffff81589043>] unix_stream_connect+0x102/0x3ed [50619.021429] [<ffffffff81518cbc>] ? __sock_create+0x168/0x1c0 [50619.022301] [<ffffffff815de7e3>] ? call_refreshresult+0x91/0x91 [50619.023170] [<ffffffff81516531>] kernel_connect+0x10/0x12 [50619.024047] [<ffffffff815e1d36>] xs_local_setup_socket+0x122/0x191 [50619.024945] [<ffffffff815e2f50>] xs_local_connect+0x2c/0x48 [50619.025849] [<ffffffff815e01f6>] xprt_connect+0x112/0x11b [50619.026756] [<ffffffff815de81c>] call_connect+0x39/0x3b [50619.027662] [<ffffffff815e4e68>] __rpc_execute+0xe8/0x2ca [50619.028567] [<ffffffff815e5109>] rpc_execute+0x76/0x9d [50619.029473] [<ffffffff815debd1>] rpc_run_task+0x78/0x80 [50619.030376] [<ffffffff815ded0f>] rpc_call_sync+0x88/0x9e [50619.031270] [<ffffffff815ebd2f>] rpcb_register_call+0x1f/0x2e [50619.032143] [<ffffffff815ec216>] rpcb_v4_register+0xb2/0x13c [50619.033031] [<ffffffff8108addb>] ? call_timer_fn+0x15e/0x15e [50619.033918] [<ffffffff815e7816>] svc_unregister.isra.11+0x5a/0xcb [50619.034804] [<ffffffff815e789b>] svc_rpcb_cleanup+0x14/0x21 [50619.035706] [<ffffffff815e70ef>] svc_shutdown_net+0x2b/0x30 [50619.036586] [<ffffffff812471c5>] lockd_down_net+0x7f/0xa3 [50619.037465] [<ffffffff81247413>] lockd_down+0x30/0xb2 [50619.038346] [<ffffffff8124439f>] nlmclnt_done+0x1f/0x23 [50619.039227] [<ffffffff8120fd72>] ? nfs_start_lockd+0xc8/0xc8 [50619.040086] [<ffffffff8120fd89>] nfs_destroy_server+0x17/0x19 [50619.040962] [<ffffffff8121024b>] nfs_free_server+0xeb/0x15c [50619.041947] [<ffffffff812172c3>] nfs_kill_super+0x1f/0x23 [50619.042824] [<ffffffff8115da33>] deactivate_locked_super+0x26/0x52 [50619.043696] [<ffffffff8115e73d>] deactivate_super+0x42/0x47 [50619.044562] [<ffffffff8117453e>] mntput_no_expire+0x135/0x13e [50619.045424] [<ffffffff81174574>] mntput+0x2d/0x2f [50619.046287] [<ffffffff8115cd78>] __fput+0x1c6/0x1e6 [50619.047111] [<ffffffff8115cda6>] ____fput+0xe/0x10 [50619.047943] [<ffffffff810998da>] task_work_run+0x7e/0x98 [50619.048764] [<ffffffff81082bc5>] do_exit+0x3cc/0x8fa [50619.049580] [<ffffffff81174449>] ? mntput_no_expire+0x40/0x13e [50619.050399] [<ffffffff8108ca8b>] ? __dequeue_signal+0x1a/0x118 [50619.051215] [<ffffffff8108328d>] do_group_exit+0x6f/0xa2 [50619.052000] [<ffffffff8108f0e7>] get_signal_to_deliver+0x4f2/0x530 [50619.052797] [<ffffffff81036a39>] do_signal+0x4d/0x4a4 [50619.053577] [<ffffffff810f2810>] ? call_rcu+0x17/0x19 [50619.054344] [<ffffffff81036ebc>] do_notify_resume+0x2c/0x6b [50619.055084] [<ffffffff81614098>] int_signal+0x12/0x17 [50619.055852] Code: c7 c7 10 4a c3 81 e8 79 c4 f3 ff e8 99 3a f3 ff 48 83 7b 20 00 0f 85 8d 00 00 00 65 48 8b 04 25 c0 b8 00 00 48 8b 80 58 05 00 00 <8b> 50 08 f6 c2 01 74 04 f3 90 eb f4 48 8b 48 18 48 89 4b 20 48 [50619.057735] RIP [<ffffffff81165e76>] path_init+0x11c/0x36f [50619.058586] RSP <ffff88041b19f508> [50619.059429] CR2: 0000000000000008[...]
Quoted 4 lines
Quoted 3 lines
Why is it done in essentially random process context, anyway? There's such thing as chroot, after all, which would screw that sucker as hard as NULL ->fs, but in a less visible way...
Quoted 9 lines
Quoted 2 lines
Having not studied the problem, I can't offer up much of an idea on how to fix it at this point.
What mystifies me is why a v3 server shutdown and unregistration is triggering a v4 registration. It's not really that surprising that a registration that late in shutdown, after the disconnected fs has been destroyed, would cause problems... (I have NFSv4 built in, but am not using it yet.)
It's not. This is an rpcbind v4 (de)registration -- nothing to do with NFSv4:
[50619.032143] [<ffffffff815ec216>] rpcb_v4_register+0xb2/0x13c
What's happening here is that we need to take down lockd on the last reference to the lazily-umounted NFS filesystem. That means that we need to upcall to rpcbind to tell it to remove the port registration. That requires opening the unix socket that rpcbind listens on, which involves walking a path. But, current->fs has already been torn down and set to NULL at this point since we're doing this at delayed fput time.
David Howells mentioned to me on IRC that the bug is really in do_exit() and that we ought to be calling exit_task_work() before calling exit_fs() (and maybe before exit_task_namespaces() too). I don't have enough of a feel for the delayed fput code to know whether that's the best fix, but it sounds plausible to me.
At the same time though, we probably ought to be doing this pathwalk from some root that is guaranteed to be able to reach that pathname. It's certainly possible that the socket might not be reachable to the last task that's accessing the NFS mount...
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 81 lines · On Mon, 5 Aug 2013 11:04:27 -0400 Jeff Layton <[email protected]> wrote:
Quoted 78 lines · On Mon, 05 Aug 2013 15:48:01 +0100 Nix <[email protected]> wrote:
On 5 Aug 2013, Jeff Layton stated:
Quoted 51 lines
Quoted 39 lines · On Sun, 04 Aug 2013 16:40:58 +0100 Nix <[email protected]> wrote:
I just got this panic on 3.10.4, in the middle of a large parallel compilation (of Chromium, as it happens) over NFSv3:
[16364.527516] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 [16364.527571] IP: [<ffffffff81245157>] nlmclnt_setlockargs+0x55/0xcf [16364.527611] PGD 0 [16364.527626] Oops: 0000 [#1] PREEMPT SMP [16364.527656] Modules linked in: [last unloaded: microcode] [16364.527690] CPU: 0 PID: 17034 Comm: flock Not tainted 3.10.4-05315-gf4ce424-dirty #1 [16364.527730] Hardware name: System manufacturer System Product Name/P8H61-MX USB3, BIOS 0506 08/10/2012 [16364.527775] task: ffff88041a97ad60 ti: ffff8803501d4000 task.ti: ffff8803501d4000 [16364.527813] RIP: 0010:[<ffffffff81245157>] [<ffffffff81245157>] nlmclnt_setlockargs+0x55/0xcf [16364.527860] RSP: 0018:ffff8803501d5c58 EFLAGS: 00010282 [16364.527889] RAX: ffff88041a97ad60 RBX: ffff8803e49c8800 RCX: 0000000000000000 [16364.527926] RDX: 0000000000000000 RSI: 000000000000004a RDI: ffff8803e49c8b54 [16364.527962] RBP: ffff8803501d5c68 R08: 0000000000015720 R09: 0000000000000000 [16364.527998] R10: 00007ffffffff000 R11: ffff8803501d5d58 R12: ffff8803501d5d58 [16364.528034] R13: ffff88041bd2bc00 R14: 0000000000000000 R15: ffff8803fc9e2900 [16364.528070] FS: 0000000000000000(0000) GS:ffff88042fa00000(0000) knlGS:0000000000000000 [16364.528111] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16364.528142] CR2: 0000000000000008 CR3: 0000000001c0b000 CR4: 00000000001407f0 [16364.528177] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16364.528214] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16364.528303] Stack: [16364.528316] ffff8803501d5d58 ffff8803e49c8800 ffff8803501d5cd8 ffffffff81245418 [16364.528369] 0000000000000000 ffff8803516f0bc0 ffff8803d7b7b6c0 ffffffff81215c81 [16364.528418] ffff880300000007 ffff88041bd2bdc8 ffff8801aabe9650 ffff8803fc9e2900 [16364.528467] Call Trace: [16364.528485] [<ffffffff81245418>] nlmclnt_proc+0x148/0x5fb [16364.528516] [<ffffffff81215c81>] ? nfs_put_lock_context+0x69/0x6e [16364.528550] [<ffffffff812209a2>] nfs3_proc_lock+0x21/0x23 [16364.528581] [<ffffffff812149dd>] do_unlk+0x96/0xb2 [16364.528608] [<ffffffff81214b41>] nfs_flock+0x5a/0x71 [16364.528637] [<ffffffff8119a747>] locks_remove_flock+0x9e/0x113 [16364.528668] [<ffffffff8115cc68>] __fput+0xb6/0x1e6 [16364.528695] [<ffffffff8115cda6>] ____fput+0xe/0x10 [16364.528724] [<ffffffff810998da>] task_work_run+0x7e/0x98 [16364.528754] [<ffffffff81082bc5>] do_exit+0x3cc/0x8fa [16364.528782] [<ffffffff81083501>] ? SyS_wait4+0xa5/0xc2Quoted 8 lines
[16364.528811] [<ffffffff8108328d>] do_group_exit+0x6f/0xa2 [16364.528843] [<ffffffff810832d7>] SyS_exit_group+0x17/0x17 [16364.528876] [<ffffffff81613e92>] system_call_fastpath+0x16/0x1b [16364.528907] Code: 00 00 65 48 8b 04 25 c0 b8 00 00 48 8b 72 20 48 81 ee c0 01 00 00 f3 a4 48 8d bb 54 03 00 00 be 4a 00 00 00 48 8b 90 68 05 00 00 <48> 8b 52 08 48 89 bb d0 00 00 00 48 83 c2 45 48 89 53 38 48 8b [16364.529176] RIP [<ffffffff81245157>] nlmclnt_setlockargs+0x55/0xcf [16364.529264] RSP <ffff8803501d5c58> [16364.529283] CR2: 0000000000000008 [16364.539039] ---[ end trace 5a73fddf23441377 ]---Quoted 2 lines
What might be most helpful is to figure out exactly where the above panic occurred.
OK. My kernel is non-modular and built without debugging information:
rebuilt with debugging info, and got this:
0xffffffff81245418is innlmclnt_proc(fs/lockd/clntproc.c:172).
167 return-ENOMEM;168 } 169 /* Set up the argument struct */ 170 nlmclnt_setlockargs(call, fl);171
172 if (IS_SETLK(cmd)||IS_SETLKW(cmd)) {
173 if (fl->fl_type!=F_UNLCK) {
174call->a_args.block=IS_SETLKW(cmd)? 1 : 0;
175 status =nlmclnt_lock(call, fl);
176 } elseThat's decimal 328:
0xffffffff81245413 <+323>: callq 0xffffffff81245102 <nlmclnt_setlockargs> 0xffffffff81245418 <+328>: mov -0x40(%rbp),%eax 0xffffffff8124541b <+331>: sub $0x6,%eax 0xffffffff8124541e <+334>: cmp $0x1,%eax
Quoted 7 lines
Quoted 7 lines
nlm_alloc_call()cannot fail (we have a NULL check right there), and fl also cannot be NULL because it's dereferenced innfs_flock(), up the call chain from where we are.Time to stick some printk()s in, I susupect. (Not sure how to keep them from utterly flooding the log, though.)
The listing and disassembly from nlmclnt_proc is not terribly interesting unfortunately. You really want to do the listing and disassembly of the RIP at panic time (nlmclnt_setlockargs+0x55).
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 58 lines · Jeff Layton
Quoted 55 lines · On Mon, 5 Aug 2013 11:04:27 -0400 Jeff Layton <[email protected]> wrote:
Quoted 52 lines · On Mon, 05 Aug 2013 15:48:01 +0100 Nix <[email protected]> wrote:
On 5 Aug 2013, Jeff Layton stated:
Quoted 50 lines
Quoted 47 lines · On Sun, 04 Aug 2013 16:40:58 +0100 Nix <[email protected]> wrote:
I just got this panic on 3.10.4, in the middle of a large parallel compilation (of Chromium, as it happens) over NFSv3:
[16364.527516] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 [16364.527571] IP: [<ffffffff81245157>] nlmclnt_setlockargs+0x55/0xcf [16364.527611] PGD 0 [16364.527626] Oops: 0000 [#1] PREEMPT SMP [16364.527656] Modules linked in: [last unloaded: microcode] [16364.527690] CPU: 0 PID: 17034 Comm: flock Not tainted 3.10.4-05315-gf4ce424-dirty #1 [16364.527730] Hardware name: System manufacturer System Product Name/P8H61-MX USB3, BIOS 0506 08/10/2012 [16364.527775] task: ffff88041a97ad60 ti: ffff8803501d4000 task.ti: ffff8803501d4000 [16364.527813] RIP: 0010:[<ffffffff81245157>] [<ffffffff81245157>] nlmclnt_setlockargs+0x55/0xcf [16364.527860] RSP: 0018:ffff8803501d5c58 EFLAGS: 00010282 [16364.527889] RAX: ffff88041a97ad60 RBX: ffff8803e49c8800 RCX: 0000000000000000 [16364.527926] RDX: 0000000000000000 RSI: 000000000000004a RDI: ffff8803e49c8b54 [16364.527962] RBP: ffff8803501d5c68 R08: 0000000000015720 R09: 0000000000000000 [16364.527998] R10: 00007ffffffff000 R11: ffff8803501d5d58 R12: ffff8803501d5d58 [16364.528034] R13: ffff88041bd2bc00 R14: 0000000000000000 R15: ffff8803fc9e2900 [16364.528070] FS: 0000000000000000(0000) GS:ffff88042fa00000(0000) knlGS:0000000000000000 [16364.528111] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16364.528142] CR2: 0000000000000008 CR3: 0000000001c0b000 CR4: 00000000001407f0 [16364.528177] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16364.528214] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16364.528303] Stack: [16364.528316] ffff8803501d5d58 ffff8803e49c8800 ffff8803501d5cd8 ffffffff81245418 [16364.528369] 0000000000000000 ffff8803516f0bc0 ffff8803d7b7b6c0 ffffffff81215c81 [16364.528418] ffff880300000007 ffff88041bd2bdc8 ffff8801aabe9650 ffff8803fc9e2900 [16364.528467] Call Trace: [16364.528485] [<ffffffff81245418>] nlmclnt_proc+0x148/0x5fb [16364.528516] [<ffffffff81215c81>] ? nfs_put_lock_context+0x69/0x6e [16364.528550] [<ffffffff812209a2>] nfs3_proc_lock+0x21/0x23 [16364.528581] [<ffffffff812149dd>] do_unlk+0x96/0xb2 [16364.528608] [<ffffffff81214b41>] nfs_flock+0x5a/0x71 [16364.528637] [<ffffffff8119a747>] locks_remove_flock+0x9e/0x113 [16364.528668] [<ffffffff8115cc68>] __fput+0xb6/0x1e6 [16364.528695] [<ffffffff8115cda6>] ____fput+0xe/0x10 [16364.528724] [<ffffffff810998da>] task_work_run+0x7e/0x98 [16364.528754] [<ffffffff81082bc5>] do_exit+0x3cc/0x8fa [16364.528782] [<ffffffff81083501>] ? SyS_wait4+0xa5/0xc2 [16364.528811] [<ffffffff8108328d>] do_group_exit+0x6f/0xa2 [16364.528843] [<ffffffff810832d7>] SyS_exit_group+0x17/0x17 [16364.528876] [<ffffffff81613e92>] system_call_fastpath+0x16/0x1b [16364.528907] Code: 00 00 65 48 8b 04 25 c0 b8 00 00 48 8b 72 20 48 81 ee c0 01 00 00 f3 a4 48 8d bb 54 03 00 00 be 4a 00 00 00 48 8b 90 68 05 00 00 <48> 8b 52 08 48 89 bb d0 00 00 00 48 83 c2 45 48 89 53 38 48 8b [16364.529176] RIP [<ffffffff81245157>] nlmclnt_setlockargs+0x55/0xcf [16364.529264] RSP <ffff8803501d5c58> [16364.529283] CR2: 0000000000000008 [16364.539039] ---[ end trace 5a73fddf23441377 ]---
[...]
Quoted 3 lines
The listing and disassembly from
nlmclnt_procis not terribly interesting unfortunately. You really want to do the listing and disassembly of the RIP at panic time (nlmclnt_setlockargs+0x55).
Oh, sorry! Wrong end of the oops :)
0xffffffff81245157 is in nlmclnt_setlockargs (fs/lockd/clntproc.c:131).
126 struct nlm_args *argp = &req->a_args;
127 struct nlm_lock *lock = &argp->lock;
128
129 nlmclnt_next_cookie(&argp->cookie); 130 memcpy(&lock->fh, NFS_FH(file_inode(fl->fl_file)), sizeof(struct nfs_fh)); 131 lock->caller = utsname()->nodename; 132 lock->oh.data = req->a_owner; 133 lock->oh.len = snprintf(req->a_owner, sizeof(req->a_owner), "%u@%s",
134 (unsigned int)fl->fl_u.nfs_fl.owner->pid,
135 utsname()->nodename);
0xffffffff81245102 <+0>: callq 0xffffffff81613b00 <__fentry__> 0xffffffff81245107 <+5>: push %rbp 0xffffffff81245108 <+6>: mov %rsp,%rbp 0xffffffff8124510b <+9>: push %r12 0xffffffff8124510d <+11>: mov %rsi,%r12 0xffffffff81245110 <+14>: push %rbx 0xffffffff81245111 <+15>: mov %rdi,%rbx 0xffffffff81245114 <+18>: lea 0x10(%rdi),%rdi 0xffffffff81245118 <+22>: callq 0xffffffff812450df <nlmclnt_next_cookie> 0xffffffff8124511d <+27>: mov 0x60(%r12),%rdx 0xffffffff81245122 <+32>: lea 0x44(%rbx),%rax 0xffffffff81245126 <+36>: mov %rax,%rdi 0xffffffff81245129 <+39>: mov $0x82,%ecx 0xffffffff8124512e <+44>: mov %gs:0xb8c0,%rax 0xffffffff81245137 <+53>: mov 0x20(%rdx),%rsi 0xffffffff8124513b <+57>: sub $0x1c0,%rsi 0xffffffff81245142 <+64>: rep movsb %ds:(%rsi),%es:(%rdi) 0xffffffff81245144 <+66>: lea 0x354(%rbx),%rdi 0xffffffff8124514b <+73>: mov $0x4a,%esi 0xffffffff81245150 <+78>: mov 0x568(%rax),%rdx -> 0xffffffff81245157 <+85>: mov 0x8(%rdx),%rdx 0xffffffff8124515b <+89>: mov %rdi,0xd0(%rbx) 0xffffffff81245162 <+96>: add $0x45,%rdx
(aside: wish GDB reported those offsets in hex, or the kernel reported them in decimal. Every time I look at these I forget to convert and get confused...)
I wonder if req is NULL (possible if the assignment to argp at the top of the function's been pushed down by the optimizer). Time to stick a printk() in and find out (after work is over so I can reboot this box like billy-o).
Signature · 1 line
NULL && (void)
Quoted 103 lines · Nix
Quoted 58 lines · Jeff Layton
Quoted 55 lines · On Mon, 5 Aug 2013 11:04:27 -0400 Jeff Layton <[email protected]> wrote:
Quoted 52 lines · On Mon, 05 Aug 2013 15:48:01 +0100 Nix <[email protected]> wrote:
On 5 Aug 2013, Jeff Layton stated:
Quoted 50 lines
Quoted 47 lines · On Sun, 04 Aug 2013 16:40:58 +0100 Nix <[email protected]> wrote:
I just got this panic on 3.10.4, in the middle of a large parallel compilation (of Chromium, as it happens) over NFSv3:
[16364.527516] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 [16364.527571] IP: [<ffffffff81245157>] nlmclnt_setlockargs+0x55/0xcf [16364.527611] PGD 0 [16364.527626] Oops: 0000 [#1] PREEMPT SMP [16364.527656] Modules linked in: [last unloaded: microcode] [16364.527690] CPU: 0 PID: 17034 Comm: flock Not tainted 3.10.4-05315-gf4ce424-dirty #1 [16364.527730] Hardware name: System manufacturer System Product Name/P8H61-MX USB3, BIOS 0506 08/10/2012 [16364.527775] task: ffff88041a97ad60 ti: ffff8803501d4000 task.ti: ffff8803501d4000 [16364.527813] RIP: 0010:[<ffffffff81245157>] [<ffffffff81245157>] nlmclnt_setlockargs+0x55/0xcf [16364.527860] RSP: 0018:ffff8803501d5c58 EFLAGS: 00010282 [16364.527889] RAX: ffff88041a97ad60 RBX: ffff8803e49c8800 RCX: 0000000000000000 [16364.527926] RDX: 0000000000000000 RSI: 000000000000004a RDI: ffff8803e49c8b54 [16364.527962] RBP: ffff8803501d5c68 R08: 0000000000015720 R09: 0000000000000000 [16364.527998] R10: 00007ffffffff000 R11: ffff8803501d5d58 R12: ffff8803501d5d58 [16364.528034] R13: ffff88041bd2bc00 R14: 0000000000000000 R15: ffff8803fc9e2900 [16364.528070] FS: 0000000000000000(0000) GS:ffff88042fa00000(0000) knlGS:0000000000000000 [16364.528111] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16364.528142] CR2: 0000000000000008 CR3: 0000000001c0b000 CR4: 00000000001407f0 [16364.528177] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16364.528214] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16364.528303] Stack: [16364.528316] ffff8803501d5d58 ffff8803e49c8800 ffff8803501d5cd8 ffffffff81245418 [16364.528369] 0000000000000000 ffff8803516f0bc0 ffff8803d7b7b6c0 ffffffff81215c81 [16364.528418] ffff880300000007 ffff88041bd2bdc8 ffff8801aabe9650 ffff8803fc9e2900 [16364.528467] Call Trace: [16364.528485] [<ffffffff81245418>] nlmclnt_proc+0x148/0x5fb [16364.528516] [<ffffffff81215c81>] ? nfs_put_lock_context+0x69/0x6e [16364.528550] [<ffffffff812209a2>] nfs3_proc_lock+0x21/0x23 [16364.528581] [<ffffffff812149dd>] do_unlk+0x96/0xb2 [16364.528608] [<ffffffff81214b41>] nfs_flock+0x5a/0x71 [16364.528637] [<ffffffff8119a747>] locks_remove_flock+0x9e/0x113 [16364.528668] [<ffffffff8115cc68>] __fput+0xb6/0x1e6 [16364.528695] [<ffffffff8115cda6>] ____fput+0xe/0x10 [16364.528724] [<ffffffff810998da>] task_work_run+0x7e/0x98 [16364.528754] [<ffffffff81082bc5>] do_exit+0x3cc/0x8fa [16364.528782] [<ffffffff81083501>] ? SyS_wait4+0xa5/0xc2 [16364.528811] [<ffffffff8108328d>] do_group_exit+0x6f/0xa2 [16364.528843] [<ffffffff810832d7>] SyS_exit_group+0x17/0x17 [16364.528876] [<ffffffff81613e92>] system_call_fastpath+0x16/0x1b [16364.528907] Code: 00 00 65 48 8b 04 25 c0 b8 00 00 48 8b 72 20 48 81 ee c0 01 00 00 f3 a4 48 8d bb 54 03 00 00 be 4a 00 00 00 48 8b 90 68 05 00 00 <48> 8b 52 08 48 89 bb d0 00 00 00 48 83 c2 45 48 89 53 38 48 8b [16364.529176] RIP [<ffffffff81245157>] nlmclnt_setlockargs+0x55/0xcf [16364.529264] RSP <ffff8803501d5c58> [16364.529283] CR2: 0000000000000008 [16364.539039] ---[ end trace 5a73fddf23441377 ]---[...]
Quoted 3 lines
The listing and disassembly from
nlmclnt_procis not terribly interesting unfortunately. You really want to do the listing and disassembly of the RIP at panic time (nlmclnt_setlockargs+0x55).Oh, sorry! Wrong end of the oops :)
0xffffffff81245157is innlmclnt_setlockargs(fs/lockd/clntproc.c:131).
126 structnlm_args*argp =&req->a_args;
127 structnlm_lock*lock =&argp->lock;
128129 nlmclnt_next_cookie(&argp->cookie); 130 memcpy(&lock->fh, NFS_FH(file_inode(fl->fl_file)), sizeof(struct nfs_fh)); 131 lock->caller = utsname()->nodename; 132 lock->oh.data = req->a_owner; 133 lock->oh.len = snprintf(req->a_owner, sizeof(req->a_owner), "%u@%s",134 (unsigned int)fl->fl_u.nfs_fl.owner->pid,
135 utsname()->nodename);0xffffffff81245102 <+0>: callq 0xffffffff81613b00 <__fentry__> 0xffffffff81245107 <+5>: push %rbp 0xffffffff81245108 <+6>: mov %rsp,%rbp 0xffffffff8124510b <+9>: push %r12 0xffffffff8124510d <+11>: mov %rsi,%r12 0xffffffff81245110 <+14>: push %rbx 0xffffffff81245111 <+15>: mov %rdi,%rbx 0xffffffff81245114 <+18>: lea 0x10(%rdi),%rdi 0xffffffff81245118 <+22>: callq 0xffffffff812450df <nlmclnt_next_cookie> 0xffffffff8124511d <+27>: mov 0x60(%r12),%rdx 0xffffffff81245122 <+32>: lea 0x44(%rbx),%rax 0xffffffff81245126 <+36>: mov %rax,%rdi 0xffffffff81245129 <+39>: mov $0x82,%ecx 0xffffffff8124512e <+44>: mov %gs:0xb8c0,%rax 0xffffffff81245137 <+53>: mov 0x20(%rdx),%rsi 0xffffffff8124513b <+57>: sub $0x1c0,%rsi 0xffffffff81245142 <+64>: rep movsb %ds:(%rsi),%es:(%rdi) 0xffffffff81245144 <+66>: lea 0x354(%rbx),%rdi 0xffffffff8124514b <+73>: mov $0x4a,%esi 0xffffffff81245150 <+78>: mov 0x568(%rax),%rdx -> 0xffffffff81245157 <+85>: mov 0x8(%rdx),%rdx 0xffffffff8124515b <+89>: mov %rdi,0xd0(%rbx) 0xffffffff81245162 <+96>: add $0x45,%rdx
Quoted 9 lines
(aside: wish GDB reported those offsets in hex, or the kernel reported them in decimal. Every time I look at these I forget to convert and get confused...)
I wonder if req is NULL (possible if the assignment to argp at the top of the function's been pushed down by the optimizer). Time to stick a
printk()in and find out (after work is over so I can reboot this box like billy-o).
Does the attached patch fix the problem?
Signature · 6 lines
Trond Myklebust Linux NFS client maintainer NetApp [email protected] www.netapp.com
From 3c50ba80105464a28d456d9a1e0f1d81d4af92a8 Mon Sep 17 00:00:00 2001
From: Trond Myklebust <[email protected]>
Date: Mon, 5 Aug 2013 12:06:12 -0400
Subject: [PATCH] LOCKD: Don't call utsname()->nodename from
nlmclnt_setlockargs
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Firstly, nlmclnt_setlockargs can be called from a reclaimer thread, in which case we're in entirely the wrong namespace.
Secondly, commit 8aac62706adaaf0fab02c4327761561c8bda9448 (move exit_task_namespaces() outside of exit_notify()) now means that exit_task_work() is called after exit_task_namespaces(), which triggers an Oops when we're freeing up the locks.
--- fs/lockd/clntproc.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
fs/lockd/clntproc.c +3 -2
@@ -125,14 +125,15 @@ static void nlmclnt_setlockargs(struct nlm_rqst *req, struct file_lock *fl) {struct nlm_args *argp = &req->a_args;struct nlm_lock *lock = &argp->lock; +char *nodename = req->a_host->h_rpcclnt->cl_nodename;nlmclnt_next_cookie (&argp->cookie);memcpy (&lock->fh,NFS_FH (file_inode (fl->fl_file)),sizeof (struct nfs_fh )); - lock->caller =utsname ()->nodename; + lock->caller = nodename; lock->oh.data = req->a_owner; lock->oh.len =snprintf (req->a_owner,sizeof (req->a_owner),"%u@%s" , (unsigned int )fl->fl_u.nfs_fl.owner->pid, -utsname ()->nodename); + nodename); lock->svid = fl->fl_u.nfs_fl.owner->pid; lock->fl.fl_start = fl->fl_start; lock->fl.fl_end = fl->fl_end;
Signature · 1 line
1.8.3.1
Jeff Layton
Reviewed-byQuoted 37 lines · Trond
From
3c50ba80105464a28d456d9a1e0f1d81d4af92a8Mon Sep 17 00:00:00 2001
From: Trond Myklebust <[email protected]>
Date: Mon, 5 Aug 2013 12:06:12 -0400
Subject: [PATCH] LOCKD: Don't call utsname()->nodename from
nlmclnt_setlockargs
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bitFirstly,
nlmclnt_setlockargscan be called from a reclaimer thread, in which case we're in entirely the wrong namespace.
Secondly, commit8aac62706adaaf0fab02c4327761561c8bda9448(moveexit_task_namespaces()outside ofexit_notify()) now means thatexit_task_work()is called afterexit_task_namespaces(), which triggers an Oops when we're freeing up the locks.Signed-off-by Trond Myklebust <[email protected]>Cc Toralf Förster <[email protected]>Cc Oleg Nesterov <[email protected]>Cc Nix <[email protected]>Cc Jeff Layton <[email protected]>--- fs/lockd/clntproc.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)fs/lockd/clntproc.c +1
-0@@ -125,14 +125,15 @@ static void nlmclnt_setlockargs(struct nlm_rqst *req, struct file_lock *fl) {struct nlm_args *argp = &req->a_args;struct nlm_lock *lock = &argp->lock; +char *nodename = req->a_host->h_rpcclnt->cl_nodename;nlmclnt_next_cookie (&argp->cookie);memcpy (&lock->fh,NFS_FH (file_inode (fl->fl_file)),sizeof (struct nfs_fh ));
fs/lockd/clntproc.c static void nlmclnt_setlockargs(struct nlm_rqst *req, struct file_lock *fl) · line 139
Quoted 10 lines
fs/lockd/clntproc.c +2
-2- lock->caller =utsname ()->nodename; + lock->caller = nodename; lock->oh.data = req->a_owner; lock->oh.len =snprintf (req->a_owner,sizeof (req->a_owner),"%u@%s" , (unsigned int )fl->fl_u.nfs_fl.owner->pid, -utsname ()->nodename); + nodename); lock->svid = fl->fl_u.nfs_fl.owner->pid; lock->fl.fl_start = fl->fl_start; lock->fl.fl_end = fl->fl_end;
Looks good to me...
Trond, any thoughts on the other oops that Nix posted? The issue there seems to be that we're trying to do the pathwalk to the rpcbind unix socket from exit_task_work(), but that's happening after we've already called exit_fs().
The trivial answer seems to be to simply call exit_task_work() before exit_fs() there, but it seems like we ought to be doing the upcall to rpcbind in a mount namespace from which we know we can reach the socket...
Quoted 60 lines · Jeff Layton
Quoted 37 lines · Trond
From
3c50ba80105464a28d456d9a1e0f1d81d4af92a8Mon Sep 17 00:00:00 2001
From: Trond Myklebust <[email protected]>
Date: Mon, 5 Aug 2013 12:06:12 -0400
Subject: [PATCH] LOCKD: Don't call utsname()->nodename from
nlmclnt_setlockargs
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bitFirstly,
nlmclnt_setlockargscan be called from a reclaimer thread, in which case we're in entirely the wrong namespace.
Secondly, commit8aac62706adaaf0fab02c4327761561c8bda9448(moveexit_task_namespaces()outside ofexit_notify()) now means thatexit_task_work()is called afterexit_task_namespaces(), which triggers an Oops when we're freeing up the locks.Signed-off-by Trond Myklebust <[email protected]>Cc Toralf Förster <[email protected]>Cc Oleg Nesterov <[email protected]>Cc Nix <[email protected]>Cc Jeff Layton <[email protected]>--- fs/lockd/clntproc.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)fs/lockd/clntproc.c +1
-0@@ -125,14 +125,15 @@ static void nlmclnt_setlockargs(struct nlm_rqst *req, struct file_lock *fl) {struct nlm_args *argp = &req->a_args;struct nlm_lock *lock = &argp->lock; +char *nodename = req->a_host->h_rpcclnt->cl_nodename;nlmclnt_next_cookie (&argp->cookie);memcpy (&lock->fh,NFS_FH (file_inode (fl->fl_file)),sizeof (struct nfs_fh ));fs/lockd/clntproc.c static void nlmclnt_setlockargs(struct nlm_rqst *req, struct file_lock *fl) · line 139
Quoted 10 lines
fs/lockd/clntproc.c +2
-2- lock->caller =utsname ()->nodename; + lock->caller = nodename; lock->oh.data = req->a_owner; lock->oh.len =snprintf (req->a_owner,sizeof (req->a_owner),"%u@%s" , (unsigned int )fl->fl_u.nfs_fl.owner->pid, -utsname ()->nodename); + nodename); lock->svid = fl->fl_u.nfs_fl.owner->pid; lock->fl.fl_start = fl->fl_start; lock->fl.fl_end = fl->fl_end;Looks good to me...
Reviewed-by Jeff Layton <[email protected]>Trond, any thoughts on the other oops that Nix posted? The issue there seems to be that we're trying to do the pathwalk to the rpcbind unix socket from
exit_task_work(), but that's happening after we've already calledexit_fs().
Quoted 4 lines
The trivial answer seems to be to simply call
exit_task_work()beforeexit_fs()there, but it seems like we ought to be doing the upcall to rpcbind in a mount namespace from which we know we can reach the socket...
Isn't it enough to just do the same thing as we did for gss proxy? i.e. set the RPC_CLNT_CREATE_NO_IDLE_TIMEOUT flag.
See attachment.
Signature · 6 lines
Trond Myklebust Linux NFS client maintainer NetApp [email protected] www.netapp.com
From ab56d77893815b1b9f0aaa7a89cee7c832a31cff Mon Sep 17 00:00:00 2001
From: Trond Myklebust <[email protected]>
Date: Mon, 5 Aug 2013 14:10:43 -0400
Subject: [PATCH] SUNRPC: Don't auto-disconnect from the local rpcbind socket
There is no need for the kernel to time out the AF_LOCAL connection to the rpcbind socket, and doing so is problematic because when it is time to reconnect, our process may no longer be using the same mount namespace.
--- net/sunrpc/rpcb_clnt.c | 9 +++++++++ 1 file changed, 9 insertions(+)
net/sunrpc/rpcb_clnt.c +9 -0
@@ -238,6 +238,15 @@ static int rpcb_create_local_unix(struct net *net) .program = &rpcb_program, .version = RPCBVERS_2, .authflavor = RPC_AUTH_NULL, +/* +* We turn off the idle timeout to prevent the kernel +* from automatically disconnecting the socket. +* Otherwise, we'd have to cache the mount namespace +* of the caller and somehow pass that to the socket +* reconnect code. +*/ + .flags =RPC_CLNT_CREATE_NOPING | + RPC_CLNT_CREATE_NO_IDLE_TIMEOUT, };struct rpc_clnt *clnt, *clnt4;int result =0 ;
Signature · 1 line
1.8.3.1
Quoted 66 lines · Trond
Quoted 60 lines · Jeff Layton
Quoted 37 lines · Trond
From
3c50ba80105464a28d456d9a1e0f1d81d4af92a8Mon Sep 17 00:00:00 2001
From: Trond Myklebust <[email protected]>
Date: Mon, 5 Aug 2013 12:06:12 -0400
Subject: [PATCH] LOCKD: Don't call utsname()->nodename from
nlmclnt_setlockargs
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bitFirstly,
nlmclnt_setlockargscan be called from a reclaimer thread, in which case we're in entirely the wrong namespace.
Secondly, commit8aac62706adaaf0fab02c4327761561c8bda9448(moveexit_task_namespaces()outside ofexit_notify()) now means thatexit_task_work()is called afterexit_task_namespaces(), which triggers an Oops when we're freeing up the locks.Signed-off-by Trond Myklebust <[email protected]>Cc Toralf Förster <[email protected]>Cc Oleg Nesterov <[email protected]>Cc Nix <[email protected]>Cc Jeff Layton <[email protected]>--- fs/lockd/clntproc.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)fs/lockd/clntproc.c +1
-0@@ -125,14 +125,15 @@ static void nlmclnt_setlockargs(struct nlm_rqst *req, struct file_lock *fl) {struct nlm_args *argp = &req->a_args;struct nlm_lock *lock = &argp->lock; +char *nodename = req->a_host->h_rpcclnt->cl_nodename;nlmclnt_next_cookie (&argp->cookie);memcpy (&lock->fh,NFS_FH (file_inode (fl->fl_file)),sizeof (struct nfs_fh ));fs/lockd/clntproc.c static void nlmclnt_setlockargs(struct nlm_rqst *req, struct file_lock *fl) · line 139
Quoted 10 lines
fs/lockd/clntproc.c +2
-2- lock->caller =utsname ()->nodename; + lock->caller = nodename; lock->oh.data = req->a_owner; lock->oh.len =snprintf (req->a_owner,sizeof (req->a_owner),"%u@%s" , (unsigned int )fl->fl_u.nfs_fl.owner->pid, -utsname ()->nodename); + nodename); lock->svid = fl->fl_u.nfs_fl.owner->pid; lock->fl.fl_start = fl->fl_start; lock->fl.fl_end = fl->fl_end;Looks good to me...
Reviewed-by Jeff Layton <[email protected]>Trond, any thoughts on the other oops that Nix posted? The issue there seems to be that we're trying to do the pathwalk to the rpcbind unix socket from
exit_task_work(), but that's happening after we've already calledexit_fs().Quoted 4 lines
The trivial answer seems to be to simply call
exit_task_work()beforeexit_fs()there, but it seems like we ought to be doing the upcall to rpcbind in a mount namespace from which we know we can reach the socket...
Quoted 4 lines
Isn't it enough to just do the same thing as we did for gss proxy? i.e. set the
RPC_CLNT_CREATE_NO_IDLE_TIMEOUTflag.See attachment.
Yeah, that looks like a reasonable thing to do...
OTOH, Is there any other way for a unix socket to end up disconnected other than if we were to close it? Maybe if rpcbind stopped, the socket unlinked and recreated and then started again?
If so then you still could potentially end up in this situation even if you didn't autoclose it.
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 74 lines · Jeff Layton
Quoted 66 lines · Trond
Quoted 60 lines · Jeff Layton
Quoted 37 lines · Trond
From
3c50ba80105464a28d456d9a1e0f1d81d4af92a8Mon Sep 17 00:00:00 2001
From: Trond Myklebust <[email protected]>
Date: Mon, 5 Aug 2013 12:06:12 -0400
Subject: [PATCH] LOCKD: Don't call utsname()->nodename from
nlmclnt_setlockargs
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bitFirstly,
nlmclnt_setlockargscan be called from a reclaimer thread, in which case we're in entirely the wrong namespace.
Secondly, commit8aac62706adaaf0fab02c4327761561c8bda9448(moveexit_task_namespaces()outside ofexit_notify()) now means thatexit_task_work()is called afterexit_task_namespaces(), which triggers an Oops when we're freeing up the locks.Signed-off-by Trond Myklebust <[email protected]>Cc Toralf Förster <[email protected]>Cc Oleg Nesterov <[email protected]>Cc Nix <[email protected]>Cc Jeff Layton <[email protected]>--- fs/lockd/clntproc.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)fs/lockd/clntproc.c +1
-0@@ -125,14 +125,15 @@ static void nlmclnt_setlockargs(struct nlm_rqst *req, struct file_lock *fl) {struct nlm_args *argp = &req->a_args;struct nlm_lock *lock = &argp->lock; +char *nodename = req->a_host->h_rpcclnt->cl_nodename;nlmclnt_next_cookie (&argp->cookie);memcpy (&lock->fh,NFS_FH (file_inode (fl->fl_file)),sizeof (struct nfs_fh ));fs/lockd/clntproc.c static void nlmclnt_setlockargs(struct nlm_rqst *req, struct file_lock *fl) · line 139
Quoted 10 lines
fs/lockd/clntproc.c +2
-2- lock->caller =utsname ()->nodename; + lock->caller = nodename; lock->oh.data = req->a_owner; lock->oh.len =snprintf (req->a_owner,sizeof (req->a_owner),"%u@%s" , (unsigned int )fl->fl_u.nfs_fl.owner->pid, -utsname ()->nodename); + nodename); lock->svid = fl->fl_u.nfs_fl.owner->pid; lock->fl.fl_start = fl->fl_start; lock->fl.fl_end = fl->fl_end;Looks good to me...
Reviewed-by Jeff Layton <[email protected]>Trond, any thoughts on the other oops that Nix posted? The issue there seems to be that we're trying to do the pathwalk to the rpcbind unix socket from
exit_task_work(), but that's happening after we've already calledexit_fs().Quoted 4 lines
The trivial answer seems to be to simply call
exit_task_work()beforeexit_fs()there, but it seems like we ought to be doing the upcall to rpcbind in a mount namespace from which we know we can reach the socket...Quoted 4 lines
Isn't it enough to just do the same thing as we did for gss proxy? i.e. set the
RPC_CLNT_CREATE_NO_IDLE_TIMEOUTflag.See attachment.
Quoted 8 lines
Yeah, that looks like a reasonable thing to do...
OTOH, Is there any other way for a unix socket to end up disconnected other than if we were to close it? Maybe if rpcbind stopped, the socket unlinked and recreated and then started again?
If so then you still could potentially end up in this situation even if you didn't autoclose it.
True. How about something like the following instead. Note the change to the original patch...
Signature · 6 lines
Trond Myklebust Linux NFS client maintainer NetApp [email protected] www.netapp.com
From 00326ed6442c66021cd4b5e19e80f3e2027d5d42 Mon Sep 17 00:00:00 2001
From: Trond Myklebust <[email protected]>
Date: Mon, 5 Aug 2013 14:10:43 -0400
Subject: [PATCH v2 1/2] SUNRPC: Don't auto-disconnect from the local rpcbind
socket
There is no need for the kernel to time out the AF_LOCAL connection to the rpcbind socket, and doing so is problematic because when it is time to reconnect, our process may no longer be using the same mount namespace.
--- net/sunrpc/rpcb_clnt.c | 8 ++++++++ 1 file changed, 8 insertions(+)
net/sunrpc/rpcb_clnt.c +8 -0
@@ -238,6 +238,14 @@ static int rpcb_create_local_unix(struct net *net) .program = &rpcb_program, .version = RPCBVERS_2, .authflavor = RPC_AUTH_NULL, +/* +* We turn off the idle timeout to prevent the kernel +* from automatically disconnecting the socket. +* Otherwise, we'd have to cache the mount namespace +* of the caller and somehow pass that to the socket +* reconnect code. +*/ + .flags = RPC_CLNT_CREATE_NO_IDLE_TIMEOUT, };struct rpc_clnt *clnt, *clnt4;int result =0 ;
Signature · 1 line
1.8.3.1
From d0ce48dd442e153b7fa109f48e3a3e642ae1395f Mon Sep 17 00:00:00 2001
From: Trond Myklebust <[email protected]>
Date: Mon, 5 Aug 2013 16:04:47 -0400
Subject: [PATCH v2 2/2] SUNRPC: If the rpcbind channel is disconnected, fail
the call to unregister
If rpcbind causes our connection to the AF_LOCAL socket to close after we've registered a service, then we want to be careful about reconnecting since the mount namespace may have changed.
By simply refusing to reconnect the AF_LOCAL socket in the case of unregister, we avoid the need to somehow save the mount namespace. While this may lead to some services not unregistering properly, it should be safe.
--- include/linux/sunrpc/sched.h | 1 + net/sunrpc/clnt.c | 4 ++++ net/sunrpc/netns.h | 1 + net/sunrpc/rpcb_clnt.c | 40 +++++++++++++++++++++++++++------------- 4 files changed, 33 insertions(+), 13 deletions(-)
include/linux/sunrpc/sched.h +1 -0
@@ -121,6 +121,7 @@ struct rpc_task_setup {#define RPC_TASK_SOFTCONN 0x0400/* Fail if can't connect */ #define RPC_TASK_SENT 0x0800/* message was sent */ #define RPC_TASK_TIMEOUT 0x1000/* fail with ETIMEDOUT on timeout */ +#define RPC_TASK_NOCONNECT 0x2000/* return ENOTCONN if not connected */ #define RPC_IS_ASYNC (t) ((t)->tk_flags & RPC_TASK_ASYNC)#define RPC_IS_SWAPPER (t) ((t)->tk_flags & RPC_TASK_SWAPPER)
net/sunrpc/clnt.c +4 -0
@@ -1660,6 +1660,10 @@ call_connect(struct rpc_task *task) task->tk_action = call_connect_status;if (task->tk_status <0 )return ; +if (task->tk_flags &RPC_TASK_NOCONNECT ) { +rpc_exit (task, -ENOTCONN ); +return ; + }xprt_connect (task); } }
net/sunrpc/netns.h +1 -0
@@ -23,6 +23,7 @@ struct sunrpc_net {struct rpc_clnt *rpcb_local_clnt4;spinlock_t rpcb_clnt_lock;unsigned int rpcb_users; + unsigned int rpcb_is_af_local : 1;struct mutex gssp_lock;wait_queue_head_t gssp_wq;
net/sunrpc/rpcb_clnt.c +27 -13
@@ -204,13 +204,15 @@ void rpcb_put_local(struct net *net) }static void rpcb_set_local (struct net *net,struct rpc_clnt *clnt, -struct rpc_clnt *clnt4) +struct rpc_clnt *clnt4, +bool is_af_local) {struct sunrpc_net *sn =net_generic (net, sunrpc_net_id);/* Protected by rpcb_create_local_mutex */ sn->rpcb_local_clnt = clnt; sn->rpcb_local_clnt4 = clnt4; + sn->rpcb_is_af_local = is_af_local ?1 :0 ;smp_wmb (); sn->rpcb_users =1 ; dprintk("RPC: created new rpcb local clients (rpcb_local_clnt: "
@@ -271,7 +273,7 @@ static int rpcb_create_local_unix(struct net *net) clnt4 =NULL ; } -rpcb_set_local (net, clnt, clnt4); +rpcb_set_local (net, clnt, clnt4, true); out:return result;
@@ -323,7 +325,7 @@ static int rpcb_create_local_net(struct net *net) clnt4 =NULL ; } -rpcb_set_local (net, clnt, clnt4); +rpcb_set_local (net, clnt, clnt4, false); out:return result;
@@ -384,13 +386,16 @@ static struct rpc_clnt *rpcb_create(struct net *net, const char *hostname,return rpc_create (&args); } -static int rpcb_register_call (struct rpc_clnt *clnt,struct rpc_message *msg) +static int rpcb_register_call (struct sunrpc_net *sn,struct rpc_clnt *clnt,struct rpc_message *msg,bool is_set) { -int result, error =0 ; +int flags =RPC_TASK_NOCONNECT ; +int error, result =0 ; +if (is_set || !sn->rpcb_is_af_local) + flags =RPC_TASK_SOFTCONN ; msg->rpc_resp = &result; - error =rpc_call_sync (clnt, msg,RPC_TASK_SOFTCONN ); + error =rpc_call_sync (clnt, msg, flags);if (error <0 ) {dprintk ("RPC: failed to contact local rpcbind " "server (errno %d).\n" , -error);
@@ -447,16 +452,19 @@ int rpcb_register(struct net *net, u32 prog, u32 vers, int prot, unsigned short .rpc_argp = &map, };struct sunrpc_net *sn =net_generic (net, sunrpc_net_id); +bool is_set = false;dprintk ("RPC: %sregistering (%u, %u, %d, %u) with local " "rpcbind\n" , (port ?"" :"un" ), prog, vers, prot, port); msg.rpc_proc = &rpcb_procedures2[RPCBPROC_UNSET ]; -if (port) +if (port !=0 ) { msg.rpc_proc = &rpcb_procedures2[RPCBPROC_SET ]; + is_set = true; + } -return rpcb_register_call (sn->rpcb_local_clnt, &msg); +return rpcb_register_call (sn, sn->rpcb_local_clnt, &msg, is_set); }/*
@@ -469,6 +477,7 @@ static int rpcb_register_inet4(struct sunrpc_net *sn,const struct sockaddr_in *sin = (const struct sockaddr_in *)sap;struct rpcbind_args *map = msg->rpc_argp;unsigned short port =ntohs (sin->sin_port); +bool is_set = false;int result; map->r_addr =rpc_sockaddr2uaddr (sap,GFP_KERNEL );
@@ -479,10 +488,12 @@ static int rpcb_register_inet4(struct sunrpc_net *sn, map->r_addr, map->r_netid); msg->rpc_proc = &rpcb_procedures4[RPCBPROC_UNSET ]; -if (port) +if (port !=0 ) { msg->rpc_proc = &rpcb_procedures4[RPCBPROC_SET ]; + is_set = true; + } - result =rpcb_register_call (sn->rpcb_local_clnt4, msg); + result =rpcb_register_call (sn, sn->rpcb_local_clnt4, msg, is_set);kfree (map->r_addr);return result; }
@@ -497,6 +508,7 @@ static int rpcb_register_inet6(struct sunrpc_net *sn,const struct sockaddr_in6 *sin6 = (const struct sockaddr_in6 *)sap;struct rpcbind_args *map = msg->rpc_argp;unsigned short port =ntohs (sin6->sin6_port); +bool is_set = false;int result; map->r_addr =rpc_sockaddr2uaddr (sap,GFP_KERNEL );
@@ -507,10 +519,12 @@ static int rpcb_register_inet6(struct sunrpc_net *sn, map->r_addr, map->r_netid); msg->rpc_proc = &rpcb_procedures4[RPCBPROC_UNSET ]; -if (port) +if (port !=0 ) { msg->rpc_proc = &rpcb_procedures4[RPCBPROC_SET ]; + is_set = true; + } - result =rpcb_register_call (sn->rpcb_local_clnt4, msg); + result =rpcb_register_call (sn, sn->rpcb_local_clnt4, msg, is_set);kfree (map->r_addr);return result; }
@@ -527,7 +541,7 @@ static int rpcb_unregister_all_protofamilies(struct sunrpc_net *sn, map->r_addr ="" ; msg->rpc_proc = &rpcb_procedures4[RPCBPROC_UNSET ]; -return rpcb_register_call (sn->rpcb_local_clnt4, msg); +return rpcb_register_call (sn, sn->rpcb_local_clnt4, msg, false); }/**
Signature · 1 line
1.8.3.1
Jeff Layton
Acked-byQuoted 81 lines · Trond
Quoted 80 lines · Jeff Layton
Quoted 66 lines · Trond
Quoted 60 lines · Jeff Layton
Quoted 37 lines · Trond
From
3c50ba80105464a28d456d9a1e0f1d81d4af92a8Mon Sep 17 00:00:00 2001
From: Trond Myklebust <[email protected]>
Date: Mon, 5 Aug 2013 12:06:12 -0400
Subject: [PATCH] LOCKD: Don't call utsname()->nodename from
nlmclnt_setlockargs
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bitFirstly,
nlmclnt_setlockargscan be called from a reclaimer thread, in which case we're in entirely the wrong namespace.
Secondly, commit8aac62706adaaf0fab02c4327761561c8bda9448(moveexit_task_namespaces()outside ofexit_notify()) now means thatexit_task_work()is called afterexit_task_namespaces(), which triggers an Oops when we're freeing up the locks.Signed-off-by Trond Myklebust <[email protected]>Cc Toralf Förster <[email protected]>Cc Oleg Nesterov <[email protected]>Cc Nix <[email protected]>Cc Jeff Layton <[email protected]>--- fs/lockd/clntproc.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)fs/lockd/clntproc.c +1
-0@@ -125,14 +125,15 @@ static void nlmclnt_setlockargs(struct nlm_rqst *req, struct file_lock *fl) {struct nlm_args *argp = &req->a_args;struct nlm_lock *lock = &argp->lock; +char *nodename = req->a_host->h_rpcclnt->cl_nodename;nlmclnt_next_cookie (&argp->cookie);memcpy (&lock->fh,NFS_FH (file_inode (fl->fl_file)),sizeof (struct nfs_fh ));fs/lockd/clntproc.c static void nlmclnt_setlockargs(struct nlm_rqst *req, struct file_lock *fl) · line 139
Quoted 10 lines
fs/lockd/clntproc.c +2
-2- lock->caller =utsname ()->nodename; + lock->caller = nodename; lock->oh.data = req->a_owner; lock->oh.len =snprintf (req->a_owner,sizeof (req->a_owner),"%u@%s" , (unsigned int )fl->fl_u.nfs_fl.owner->pid, -utsname ()->nodename); + nodename); lock->svid = fl->fl_u.nfs_fl.owner->pid; lock->fl.fl_start = fl->fl_start; lock->fl.fl_end = fl->fl_end;Looks good to me...
Reviewed-by Jeff Layton <[email protected]>Trond, any thoughts on the other oops that Nix posted? The issue there seems to be that we're trying to do the pathwalk to the rpcbind unix socket from
exit_task_work(), but that's happening after we've already calledexit_fs().Quoted 4 lines
The trivial answer seems to be to simply call
exit_task_work()beforeexit_fs()there, but it seems like we ought to be doing the upcall to rpcbind in a mount namespace from which we know we can reach the socket...Quoted 4 lines
Isn't it enough to just do the same thing as we did for gss proxy? i.e. set the
RPC_CLNT_CREATE_NO_IDLE_TIMEOUTflag.See attachment.
Yeah, that looks like a reasonable thing to do...
OTOH, Is there any other way for a unix socket to end up disconnected other than if we were to close it? Maybe if rpcbind stopped, the socket unlinked and recreated and then started again?
Quoted 5 lines
Quoted 2 lines
If so then you still could potentially end up in this situation even if you didn't autoclose it.
True. How about something like the following instead. Note the change to the original patch...
Looks good to me.
On 6 Aug 2013, Trond Myklebust verbalised:
Quoted 2 lines
True. How about something like the following instead. Note the change to the original patch...
Well, with those applied I could reboot without a panic for the first time since 3.8.x: looking good. I'll give it a reboot or two with a system that's not hot from booting though.
Signature · 1 line
NULL && (void)
T24gV2VkLCAyMDEzLTA4LTA3IGF0IDExOjE4ICswMTAwLCBOaXggd3JvdGU6DQo+IE9uIDYgQXVn IDIwMTMsIFRyb25kIE15a2xlYnVzdCB2ZXJiYWxpc2VkOg0KPiA+IFRydWUuIEhvdyBhYm91dCBz b21ldGhpbmcgbGlrZSB0aGUgZm9sbG93aW5nIGluc3RlYWQuIE5vdGUgdGhlIGNoYW5nZSB0bw0K PiA+IHRoZSBvcmlnaW5hbCBwYXRjaC4uLg0KPiANCj4gV2VsbCwgd2l0aCB0aG9zZSBhcHBsaWVk IEkgY291bGQgcmVib290IHdpdGhvdXQgYSBwYW5pYyBmb3IgdGhlIGZpcnN0DQo+IHRpbWUgc2lu Y2UgMy44Lng6IGxvb2tpbmcgZ29vZC4gSSdsbCBnaXZlIGl0IGEgcmVib290IG9yIHR3byB3aXRo IGENCj4gc3lzdGVtIHRoYXQncyBub3QgaG90IGZyb20gYm9vdGluZyB0aG91Z2guDQo+IA0KDQpD b3VsZCB5b3UgcGxlYXNlIGFsc28gdHJ5IGFwcGx5aW5nIG9ubHkgdGhlIDEvMiBwYXRjaCwgdG8g c2VlIGlmIHRoYXQNCnN1ZmZpY2VzIHRvIHF1ZWxsIHRoZSBzaHV0ZG93biBwYW5pYz8NCg0KLS0g DQpUcm9uZCBNeWtsZWJ1c3QNCkxpbnV4IE5GUyBjbGllbnQgbWFpbnRhaW5lcg0KDQpOZXRBcHAN ClRyb25kLk15a2xlYnVzdEBuZXRhcHAuY29tDQp3d3cubmV0YXBwLmNvbQ0K
Quoted 11 lines · Trond Myklebust
Quoted 7 lines · Nix
On 6 Aug 2013, Trond Myklebust verbalised:
Quoted 2 lines
True. How about something like the following instead. Note the change to the original patch...
Well, with those applied I could reboot without a panic for the first time since 3.8.x: looking good. I'll give it a reboot or two with a system that's not hot from booting though.
Could you please also try applying only the 1/2 patch, to see if that suffices to quell the shutdown panic?
It doesn't suffice. I see this severely truncated oops:
[ 115.799092] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 [ 115.800284] IP: [<ffffffff81165ec6>] path_init+0x11c/0x36f [ 115.801463] PGD 0 [ 115.802625] Oops: 0000 [#1] PREEMPT SMP [ 115.803805] Modules linked in: [last unloaded: microcode] [ 115.804995] CPU: 3 PID: 1191 Comm: sleep Not tainted 3.10.5-05317-g3c9f6fa-dirty #2 [ 115.806207] Hardware name: System manufacturer System Product Name/P8H61-MX USB3, BIOS 0506 08/10/2012 [ 115.807453] task: ffff8804189a0000 ti: ffff8803f74d6000 task.ti: ffff8803f74d6000
Signature · 1 line
NULL && (void)
T24gV2VkLCAyMDEzLTA4LTA3IGF0IDIyOjAxICswMTAwLCBOaXggd3JvdGU6DQo+IE9uIDcgQXVn IDIwMTMsIFRyb25kIE15a2xlYnVzdCBzYWlkOg0KPiANCj4gPiBPbiBXZWQsIDIwMTMtMDgtMDcg YXQgMTE6MTggKzAxMDAsIE5peCB3cm90ZToNCj4gPj4gT24gNiBBdWcgMjAxMywgVHJvbmQgTXlr bGVidXN0IHZlcmJhbGlzZWQ6DQo+ID4+ID4gVHJ1ZS4gSG93IGFib3V0IHNvbWV0aGluZyBsaWtl IHRoZSBmb2xsb3dpbmcgaW5zdGVhZC4gTm90ZSB0aGUgY2hhbmdlIHRvDQo+ID4+ID4gdGhlIG9y aWdpbmFsIHBhdGNoLi4uDQo+ID4+IA0KPiA+PiBXZWxsLCB3aXRoIHRob3NlIGFwcGxpZWQgSSBj b3VsZCByZWJvb3Qgd2l0aG91dCBhIHBhbmljIGZvciB0aGUgZmlyc3QNCj4gPj4gdGltZSBzaW5j ZSAzLjgueDogbG9va2luZyBnb29kLiBJJ2xsIGdpdmUgaXQgYSByZWJvb3Qgb3IgdHdvIHdpdGgg YQ0KPiA+PiBzeXN0ZW0gdGhhdCdzIG5vdCBob3QgZnJvbSBib290aW5nIHRob3VnaC4NCj4gPg0K PiA+IENvdWxkIHlvdSBwbGVhc2UgYWxzbyB0cnkgYXBwbHlpbmcgb25seSB0aGUgMS8yIHBhdGNo LCB0byBzZWUgaWYgdGhhdA0KPiA+IHN1ZmZpY2VzIHRvIHF1ZWxsIHRoZSBzaHV0ZG93biBwYW5p Yz8NCj4gDQo+IEl0IGRvZXNuJ3Qgc3VmZmljZS4gSSBzZWUgdGhpcyBzZXZlcmVseSB0cnVuY2F0 ZWQgb29wczoNCj4gDQo+IFsgIDExNS43OTkwOTJdIEJVRzogdW5hYmxlIHRvIGhhbmRsZSBrZXJu ZWwgTlVMTCBwb2ludGVyIGRlcmVmZXJlbmNlIGF0IDAwMDAwMDAwMDAwMDAwMDgNCj4gWyAgMTE1 LjgwMDI4NF0gSVA6IFs8ZmZmZmZmZmY4MTE2NWVjNj5dIHBhdGhfaW5pdCsweDExYy8weDM2Zg0K PiBbICAxMTUuODAxNDYzXSBQR0QgMCANCj4gWyAgMTE1LjgwMjYyNV0gT29wczogMDAwMCBbIzFd IFBSRUVNUFQgU01QIA0KPiBbICAxMTUuODAzODA1XSBNb2R1bGVzIGxpbmtlZCBpbjogW2xhc3Qg dW5sb2FkZWQ6IG1pY3JvY29kZV0gDQo+IFsgIDExNS44MDQ5OTVdIENQVTogMyBQSUQ6IDExOTEg Q29tbTogc2xlZXAgTm90IHRhaW50ZWQgMy4xMC41LTA1MzE3LWczYzlmNmZhLWRpcnR5ICMyDQo+ IFsgIDExNS44MDYyMDddIEhhcmR3YXJlIG5hbWU6IFN5c3RlbSBtYW51ZmFjdHVyZXIgU3lzdGVt IFByb2R1Y3QgTmFtZS9QOEg2MS1NWCBVU0IzLCBCSU9TIDA1MDYgMDgvMTAvMjAxMg0KPiBbICAx MTUuODA3NDUzXSB0YXNrOiBmZmZmODgwNDE4OWEwMDAwIHRpOiBmZmZmODgwM2Y3NGQ2MDAwIHRh c2sudGk6IGZmZmY4ODAzZjc0ZDYwMDANCj4gDQpPSy4gVGhlbiBJJ2xsIG1hcmsgdGhlbSBib3Ro IGZvciBzdGFibGUgaW5jbHVzaW9uIGluIDMuOSsuDQoNClRoYW5rcyBmb3IgdGVzdGluZyENCi0t IA0KVHJvbmQgTXlrbGVidXN0DQpMaW51eCBORlMgY2xpZW50IG1haW50YWluZXINCg0KTmV0QXBw DQpUcm9uZC5NeWtsZWJ1c3RAbmV0YXBwLmNvbQ0Kd3d3Lm5ldGFwcC5jb20NCg==
On 5 Aug 2013, Trond Myklebust told this:
Quoted 10 lines
Does the attached patch fix the problem?
From
3c50ba80105464a28d456d9a1e0f1d81d4af92a8Mon Sep 17 00:00:00 2001
From: Trond Myklebust <[email protected]>
Date: Mon, 5 Aug 2013 12:06:12 -0400
Subject: [PATCH] LOCKD: Don't call utsname()->nodename from
nlmclnt_setlockargs
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
It makes it worse. Much, much worse. From a crash every so often when I'm doing compilations over NFS, I get an immediate panic on startx, long long before I even try to replicate the earlier panic:
[ 83.432358] task: ffff88041aaa5ac0 ti: ffff8804199e2000 task.ti: ffff8804199e2000 [ 83.432428] RIP: 0010:[<ffffffff8124af69>] [<ffffffff8124af69>] encode_nlm4_lock+0x26/0xbe [ 83.432512] RSP: 0018:ffff8804199e3a78 EFLAGS: 00010286 [ 83.432564] RAX: 0000000000000000 RBX: ffff88041a577038 RCX: ffffffffffffffff [ 83.432630] RDX: ffff8804193b3098 RSI: ffff88041a577038 RDI: 000000000000008c [ 83.432697] RBP: ffff8804199e3aa8 R08: ffff8804193b3098 R09: 0000000000000001 [ 83.432763] R10: ffff88042fa12980 R11: ffff88042fa12980 R12: ffff8804199e3ae8 [ 83.432830] R13: 000000000000008c R14: ffff8804199e3fd8 R15: ffffffff815de80e [ 83.432898] FS: 00007f594b40c740(0000) GS:ffff88042fa00000(0000) knlGS:0000000000000000 [ 83.432974] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 83.433028] CR2: 000000000000008c CR3: 000000041ab3d000 CR4: 00000000001407f0 [ 83.433095] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 83.433176] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 83.433255] Stack: [ 83.433276] ffff88041a44fb70 ffff880400000004 ffff8804199e3ae8 ffff88041a577010 [ 83.433360] ffff8804188e0e00 ffff8804199e3fd8 ffff8804199e3ac8 ffffffff8124b0d7 [ 83.433443] ffff8804188e0e00 ffffffff8124b086 ffff8804199e3b38 ffffffff815e6032 [ 83.433616] Call Trace: [ 83.433646] [<ffffffff8124b0d7>] nlm4_xdr_enc_lockargs+0x51/0x76 [ 83.433707] [<ffffffff8124b086>] ? nlm4_xdr_enc_cancargs+0x56/0x56 [ 83.433769] [<ffffffff815e6032>] rpcauth_wrap_req+0x57/0x62 [ 83.433826] [<ffffffff815de98a>] call_transmit+0x17c/0x1f9 [ 83.433880] [<ffffffff815e4e58>] __rpc_execute+0xe8/0x2ca [ 83.433935] [<ffffffff815e50f9>] rpc_execute+0x76/0x9d [ 83.433986] [<ffffffff815debc1>] rpc_run_task+0x78/0x80 [ 83.434039] [<ffffffff815decff>] rpc_call_sync+0x88/0x9e [ 83.434092] [<ffffffff81244b3c>] nlmclnt_call+0xb5/0x240 [ 83.434146] [<ffffffff812454f0>] nlmclnt_proc+0x226/0x5fb [ 83.434226] [<ffffffff812209a2>] nfs3_proc_lock+0x21/0x23 [ 83.434280] [<ffffffff81214a5e>] do_setlk+0x65/0xee [ 83.434329] [<ffffffff81214ca6>] nfs_lock+0x14e/0x162 [ 83.434382] [<ffffffff81199661>] vfs_lock_file+0x29/0x35 [ 83.434435] [<ffffffff8119a51d>] fcntl_setlk+0x139/0x2c5 [ 83.434490] [<ffffffff81169621>] SyS_fcntl+0x2b6/0x47d [ 83.434543] [<ffffffff81613e92>] system_call_fastpath+0x16/0x1b [ 83.434600] Code: 5b 41 5c 5d c3 0f 1f 44 00 00 55 31 c0 48 83 c9 ff 48 89 e5 41 56 41 55 41 54 49 89 fc 53 48 89 f3 48 83 ec 10 4c 8b 2e 4c 89 ef <f2> ae 4c 89 e7 48 f7 d1 4c 8d 71 ff 41 8d 76 04 e8 9f 16 3a 00 [ 83.435077] RIP [<ffffffff8124af69>] encode_nlm4_lock+0x26/0xbe [ 83.435140] RSP <ffff8804199e3a78> [ 83.435197] CR2: 000000000000008c
That's here:
(gdb) list *(encode_nlm4_lock+0x26)0xffffffff8124af69 is in encode_nlm4_lock (fs/lockd/clnt4xdr.c:329).
324 * string caller_name<LM_MAXSTRLEN>;
325 */
326 static void encode_caller_name(struct xdr_stream *xdr, const char *name)
327 {
328 /* NB: client-side does not set lock->len */ 329 u32 length = strlen(name); 330 __be32 *p;
331
332 p = xdr_reserve_space(xdr, 4 + length);
333 xdr_encode_opaque(p, name, length);
0xffffffff8124af69 <+38>: repnz scas %es:(%rdi),%al
Pretty clearly, "name" can be NULL after this patch...
Signature · 1 line
NULL && (void)
Quoted 67 lines · Nix
On 5 Aug 2013, Trond Myklebust told this:
Quoted 10 lines
Does the attached patch fix the problem?
From
3c50ba80105464a28d456d9a1e0f1d81d4af92a8Mon Sep 17 00:00:00 2001
From: Trond Myklebust <[email protected]>
Date: Mon, 5 Aug 2013 12:06:12 -0400
Subject: [PATCH] LOCKD: Don't call utsname()->nodename from
nlmclnt_setlockargs
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bitIt makes it worse. Much, much worse. From a crash every so often when I'm doing compilations over NFS, I get an immediate panic on startx, long long before I even try to replicate the earlier panic:
[ 83.432358] task: ffff88041aaa5ac0 ti: ffff8804199e2000 task.ti: ffff8804199e2000 [ 83.432428] RIP: 0010:[<ffffffff8124af69>] [<ffffffff8124af69>] encode_nlm4_lock+0x26/0xbe [ 83.432512] RSP: 0018:ffff8804199e3a78 EFLAGS: 00010286 [ 83.432564] RAX: 0000000000000000 RBX: ffff88041a577038 RCX: ffffffffffffffff [ 83.432630] RDX: ffff8804193b3098 RSI: ffff88041a577038 RDI: 000000000000008c [ 83.432697] RBP: ffff8804199e3aa8 R08: ffff8804193b3098 R09: 0000000000000001 [ 83.432763] R10: ffff88042fa12980 R11: ffff88042fa12980 R12: ffff8804199e3ae8 [ 83.432830] R13: 000000000000008c R14: ffff8804199e3fd8 R15: ffffffff815de80e [ 83.432898] FS: 00007f594b40c740(0000) GS:ffff88042fa00000(0000) knlGS:0000000000000000 [ 83.432974] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 83.433028] CR2: 000000000000008c CR3: 000000041ab3d000 CR4: 00000000001407f0 [ 83.433095] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 83.433176] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 83.433255] Stack: [ 83.433276] ffff88041a44fb70 ffff880400000004 ffff8804199e3ae8 ffff88041a577010 [ 83.433360] ffff8804188e0e00 ffff8804199e3fd8 ffff8804199e3ac8 ffffffff8124b0d7 [ 83.433443] ffff8804188e0e00 ffffffff8124b086 ffff8804199e3b38 ffffffff815e6032 [ 83.433616] Call Trace: [ 83.433646] [<ffffffff8124b0d7>] nlm4_xdr_enc_lockargs+0x51/0x76 [ 83.433707] [<ffffffff8124b086>] ? nlm4_xdr_enc_cancargs+0x56/0x56 [ 83.433769] [<ffffffff815e6032>] rpcauth_wrap_req+0x57/0x62 [ 83.433826] [<ffffffff815de98a>] call_transmit+0x17c/0x1f9 [ 83.433880] [<ffffffff815e4e58>] __rpc_execute+0xe8/0x2ca [ 83.433935] [<ffffffff815e50f9>] rpc_execute+0x76/0x9d [ 83.433986] [<ffffffff815debc1>] rpc_run_task+0x78/0x80 [ 83.434039] [<ffffffff815decff>] rpc_call_sync+0x88/0x9e [ 83.434092] [<ffffffff81244b3c>] nlmclnt_call+0xb5/0x240 [ 83.434146] [<ffffffff812454f0>] nlmclnt_proc+0x226/0x5fb [ 83.434226] [<ffffffff812209a2>] nfs3_proc_lock+0x21/0x23 [ 83.434280] [<ffffffff81214a5e>] do_setlk+0x65/0xee [ 83.434329] [<ffffffff81214ca6>] nfs_lock+0x14e/0x162 [ 83.434382] [<ffffffff81199661>] vfs_lock_file+0x29/0x35 [ 83.434435] [<ffffffff8119a51d>] fcntl_setlk+0x139/0x2c5 [ 83.434490] [<ffffffff81169621>] SyS_fcntl+0x2b6/0x47d [ 83.434543] [<ffffffff81613e92>] system_call_fastpath+0x16/0x1b [ 83.434600] Code: 5b 41 5c 5d c3 0f 1f 44 00 00 55 31 c0 48 83 c9 ff 48 89 e5 41 56 41 55 41 54 49 89 fc 53 48 89 f3 48 83 ec 10 4c 8b 2e 4c 89 ef <f2> ae 4c 89 e7 48 f7 d1 4c 8d 71 ff 41 8d 76 04 e8 9f 16 3a 00 [ 83.435077] RIP [<ffffffff8124af69>] encode_nlm4_lock+0x26/0xbe [ 83.435140] RSP <ffff8804199e3a78> [ 83.435197] CR2: 000000000000008cThat's here:
(gdb) list *(encode_nlm4_lock+0x26)
0xffffffff8124af69is inencode_nlm4_lock(fs/lockd/clnt4xdr.c:329).
324 * string caller_name<LM_MAXSTRLEN>;
325 */
326 static voidencode_caller_name(struct xdr_stream *xdr, const char *name)
327 {328 /* NB: client-side does not set lock->len */ 329 u32 length = strlen(name); 330 __be32 *p;
Quoted 8 lines
331
332 p =xdr_reserve_space(xdr, 4 + length);
333xdr_encode_opaque(p, name, length);0xffffffff8124af69 <+38>: repnz scas %es:(%rdi),%alPretty clearly, "name" can be NULL after this patch...
Yes. This scheme will only work if we make sure that host->h_rpcclnt is initialised at mount time. Here is a v2 patch that should do the right thing.
Signature · 6 lines
Trond Myklebust Linux NFS client maintainer NetApp [email protected] www.netapp.com
From 9a1b6bf818e74bb7aabaecb59492b739f2f4d742 Mon Sep 17 00:00:00 2001
From: Trond Myklebust <[email protected]>
Date: Mon, 5 Aug 2013 12:06:12 -0400
Subject: [PATCH v2] LOCKD: Don't call utsname()->nodename from
nlmclnt_setlockargs
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Firstly, nlmclnt_setlockargs can be called from a reclaimer thread, in which case we're in entirely the wrong namespace.
Secondly, commit 8aac62706adaaf0fab02c4327761561c8bda9448 (move exit_task_namespaces() outside of exit_notify()) now means that exit_task_work() is called after exit_task_namespaces(), which triggers an Oops when we're freeing up the locks.
Fix this by ensuring that we initialise the nlm_host's rpc_client at mount time, so that the cl_nodename field is initialised to the value of utsname()->nodename that the net namespace uses. Then replace the lockd callers of utsname()->nodename.
--- fs/lockd/clntlock.c | 13 +++++++++---- fs/lockd/clntproc.c | 5 +++-- 2 files changed, 12 insertions(+), 6 deletions(-)
fs/lockd/clntlock.c +9 -4
@@ -64,12 +64,17 @@ struct nlm_host *nlmclnt_init(const struct nlmclnt_initdata *nlm_init) nlm_init->protocol, nlm_version, nlm_init->hostname, nlm_init->noresvport, nlm_init->net); -if (host ==NULL ) { -lockd_down (nlm_init->net); -return ERR_PTR (-ENOLCK ); - } +if (host ==NULL ) + goto out_nohost; +if (host->h_rpcclnt ==NULL &&nlm_bind_host (host) ==NULL ) + goto out_nobind;return host; +out_nobind: +nlmclnt_release_host (host); +out_nohost: +lockd_down (nlm_init->net); +return ERR_PTR (-ENOLCK ); }EXPORT_SYMBOL_GPL (nlmclnt_init);
fs/lockd/clntproc.c +3 -2
@@ -125,14 +125,15 @@ static void nlmclnt_setlockargs(struct nlm_rqst *req, struct file_lock *fl) {struct nlm_args *argp = &req->a_args;struct nlm_lock *lock = &argp->lock; +char *nodename = req->a_host->h_rpcclnt->cl_nodename;nlmclnt_next_cookie (&argp->cookie);memcpy (&lock->fh,NFS_FH (file_inode (fl->fl_file)),sizeof (struct nfs_fh )); - lock->caller =utsname ()->nodename; + lock->caller = nodename; lock->oh.data = req->a_owner; lock->oh.len =snprintf (req->a_owner,sizeof (req->a_owner),"%u@%s" , (unsigned int )fl->fl_u.nfs_fl.owner->pid, -utsname ()->nodename); + nodename); lock->svid = fl->fl_u.nfs_fl.owner->pid; lock->fl.fl_start = fl->fl_start; lock->fl.fl_end = fl->fl_end;
Signature · 1 line
1.8.3.1
On 5 Aug 2013, Trond Myklebust uttered the following:
Quoted 3 lines
Yes. This scheme will only work if we make sure that
host->h_rpcclntis initialised at mount time. Here is a v2 patch that should do the right thing.
Confirmed, that fixes it! I'll try your shutdown crash fix next.
Signature · 1 line
NULL && (void)
Quoted 103 lines · On Mon, 05 Aug 2013 16:50:37 +0100 Nix <[email protected]> wrote:
Quoted 58 lines · Jeff Layton
Quoted 55 lines · On Mon, 5 Aug 2013 11:04:27 -0400 Jeff Layton <[email protected]> wrote:
Quoted 52 lines · On Mon, 05 Aug 2013 15:48:01 +0100 Nix <[email protected]> wrote:
On 5 Aug 2013, Jeff Layton stated:
Quoted 50 lines
Quoted 47 lines · On Sun, 04 Aug 2013 16:40:58 +0100 Nix <[email protected]> wrote:
I just got this panic on 3.10.4, in the middle of a large parallel compilation (of Chromium, as it happens) over NFSv3:
[16364.527516] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 [16364.527571] IP: [<ffffffff81245157>] nlmclnt_setlockargs+0x55/0xcf [16364.527611] PGD 0 [16364.527626] Oops: 0000 [#1] PREEMPT SMP [16364.527656] Modules linked in: [last unloaded: microcode] [16364.527690] CPU: 0 PID: 17034 Comm: flock Not tainted 3.10.4-05315-gf4ce424-dirty #1 [16364.527730] Hardware name: System manufacturer System Product Name/P8H61-MX USB3, BIOS 0506 08/10/2012 [16364.527775] task: ffff88041a97ad60 ti: ffff8803501d4000 task.ti: ffff8803501d4000 [16364.527813] RIP: 0010:[<ffffffff81245157>] [<ffffffff81245157>] nlmclnt_setlockargs+0x55/0xcf [16364.527860] RSP: 0018:ffff8803501d5c58 EFLAGS: 00010282 [16364.527889] RAX: ffff88041a97ad60 RBX: ffff8803e49c8800 RCX: 0000000000000000 [16364.527926] RDX: 0000000000000000 RSI: 000000000000004a RDI: ffff8803e49c8b54 [16364.527962] RBP: ffff8803501d5c68 R08: 0000000000015720 R09: 0000000000000000 [16364.527998] R10: 00007ffffffff000 R11: ffff8803501d5d58 R12: ffff8803501d5d58 [16364.528034] R13: ffff88041bd2bc00 R14: 0000000000000000 R15: ffff8803fc9e2900 [16364.528070] FS: 0000000000000000(0000) GS:ffff88042fa00000(0000) knlGS:0000000000000000 [16364.528111] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16364.528142] CR2: 0000000000000008 CR3: 0000000001c0b000 CR4: 00000000001407f0 [16364.528177] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16364.528214] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16364.528303] Stack: [16364.528316] ffff8803501d5d58 ffff8803e49c8800 ffff8803501d5cd8 ffffffff81245418 [16364.528369] 0000000000000000 ffff8803516f0bc0 ffff8803d7b7b6c0 ffffffff81215c81 [16364.528418] ffff880300000007 ffff88041bd2bdc8 ffff8801aabe9650 ffff8803fc9e2900 [16364.528467] Call Trace: [16364.528485] [<ffffffff81245418>] nlmclnt_proc+0x148/0x5fb [16364.528516] [<ffffffff81215c81>] ? nfs_put_lock_context+0x69/0x6e [16364.528550] [<ffffffff812209a2>] nfs3_proc_lock+0x21/0x23 [16364.528581] [<ffffffff812149dd>] do_unlk+0x96/0xb2 [16364.528608] [<ffffffff81214b41>] nfs_flock+0x5a/0x71 [16364.528637] [<ffffffff8119a747>] locks_remove_flock+0x9e/0x113 [16364.528668] [<ffffffff8115cc68>] __fput+0xb6/0x1e6 [16364.528695] [<ffffffff8115cda6>] ____fput+0xe/0x10 [16364.528724] [<ffffffff810998da>] task_work_run+0x7e/0x98 [16364.528754] [<ffffffff81082bc5>] do_exit+0x3cc/0x8fa [16364.528782] [<ffffffff81083501>] ? SyS_wait4+0xa5/0xc2 [16364.528811] [<ffffffff8108328d>] do_group_exit+0x6f/0xa2 [16364.528843] [<ffffffff810832d7>] SyS_exit_group+0x17/0x17 [16364.528876] [<ffffffff81613e92>] system_call_fastpath+0x16/0x1b [16364.528907] Code: 00 00 65 48 8b 04 25 c0 b8 00 00 48 8b 72 20 48 81 ee c0 01 00 00 f3 a4 48 8d bb 54 03 00 00 be 4a 00 00 00 48 8b 90 68 05 00 00 <48> 8b 52 08 48 89 bb d0 00 00 00 48 83 c2 45 48 89 53 38 48 8b [16364.529176] RIP [<ffffffff81245157>] nlmclnt_setlockargs+0x55/0xcf [16364.529264] RSP <ffff8803501d5c58> [16364.529283] CR2: 0000000000000008 [16364.539039] ---[ end trace 5a73fddf23441377 ]---[...]
Quoted 3 lines
The listing and disassembly from
nlmclnt_procis not terribly interesting unfortunately. You really want to do the listing and disassembly of the RIP at panic time (nlmclnt_setlockargs+0x55).Oh, sorry! Wrong end of the oops :)
0xffffffff81245157is innlmclnt_setlockargs(fs/lockd/clntproc.c:131).
126 structnlm_args*argp =&req->a_args;
127 structnlm_lock*lock =&argp->lock;
128129 nlmclnt_next_cookie(&argp->cookie); 130 memcpy(&lock->fh, NFS_FH(file_inode(fl->fl_file)), sizeof(struct nfs_fh)); 131 lock->caller = utsname()->nodename; 132 lock->oh.data = req->a_owner; 133 lock->oh.len = snprintf(req->a_owner, sizeof(req->a_owner), "%u@%s",134 (unsigned int)fl->fl_u.nfs_fl.owner->pid,
135 utsname()->nodename);0xffffffff81245102 <+0>: callq 0xffffffff81613b00 <__fentry__> 0xffffffff81245107 <+5>: push %rbp 0xffffffff81245108 <+6>: mov %rsp,%rbp 0xffffffff8124510b <+9>: push %r12 0xffffffff8124510d <+11>: mov %rsi,%r12 0xffffffff81245110 <+14>: push %rbx 0xffffffff81245111 <+15>: mov %rdi,%rbx 0xffffffff81245114 <+18>: lea 0x10(%rdi),%rdi 0xffffffff81245118 <+22>: callq 0xffffffff812450df <nlmclnt_next_cookie> 0xffffffff8124511d <+27>: mov 0x60(%r12),%rdx 0xffffffff81245122 <+32>: lea 0x44(%rbx),%rax 0xffffffff81245126 <+36>: mov %rax,%rdi 0xffffffff81245129 <+39>: mov $0x82,%ecx 0xffffffff8124512e <+44>: mov %gs:0xb8c0,%rax 0xffffffff81245137 <+53>: mov 0x20(%rdx),%rsi 0xffffffff8124513b <+57>: sub $0x1c0,%rsi 0xffffffff81245142 <+64>: rep movsb %ds:(%rsi),%es:(%rdi) 0xffffffff81245144 <+66>: lea 0x354(%rbx),%rdi 0xffffffff8124514b <+73>: mov $0x4a,%esi 0xffffffff81245150 <+78>: mov 0x568(%rax),%rdx -> 0xffffffff81245157 <+85>: mov 0x8(%rdx),%rdx 0xffffffff8124515b <+89>: mov %rdi,0xd0(%rbx) 0xffffffff81245162 <+96>: add $0x45,%rdx
Quoted 9 lines
(aside: wish GDB reported those offsets in hex, or the kernel reported them in decimal. Every time I look at these I forget to convert and get confused...)
I wonder if req is NULL (possible if the assignment to argp at the top of the function's been pushed down by the optimizer). Time to stick a
printk()in and find out (after work is over so I can reboot this box like billy-o).
Ah-ha! That same bug was discussed earlier this week. See the thread with this title:
Subject: fuzz tested user mode linux core dumps in fs/lockd/clntproc.c:131
I haven't followed it too closely, unfortunately, but both oopses look like similar problems. The kernel is tearing things down in the fput codepath prior to running exit tasks that require those things...
Signature · 1 line
Jeff Layton <[email protected]>