NULL dereference in rpcauth_lookup_credcache
Subsystems (2)
Since -rc1 my regression tests crash my client. Is this a known problem? I'll investigate some more, I haven't even looked at the code yet or checked which test exactly is hitting this.
--b.
[ 164.109570] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 [ 164.111207] PGD 0 P4D 0 [ 164.111528] Oops: 0000 [#1] PREEMPT SMP PTI [ 164.112303] CPU: 2 PID: 2947 Comm: kworker/u8:5 Not tainted 4.20.0-rc1-13223-gafb6d1c474ef #1898 [ 164.113487] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20180531_142017-buildhw-08.phx2.fedoraproject.org-1.fc28 04/01/2014 [ 164.115301] Workqueue: rpciod rpc_async_schedule [sunrpc] [ 164.115920] RIP: 0010:rpcauth_lookup_credcache+0x3d/0x450 [sunrpc] [ 164.116700] Code: 89 f5 41 54 41 89 d4 53 48 83 ec 38 89 4d b0 4c 8b 7f 20 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8d 45 c0 48 89 45 c8 <41> 8b 77 08 48 89 45 c0 48 8b 47 10 4c 89 ef 48 8b 40 28 e8 cb d2 [ 164.119299] RSP: 0018:ffffc90001ee3cf0 EFLAGS: 00010246 [ 164.119872] RAX: ffffc90001ee3d10 RBX: ffff88007cc18180 RCX: 0000000000600040 [ 164.120800] RDX: 0000000000000001 RSI: ffffc90001ee3d60 RDI: ffff88007cafb198 [ 164.121643] RBP: ffffc90001ee3d50 R08: 0000000000000000 R09: 0000000000000000 [ 164.122464] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001 [ 164.123373] R13: ffffc90001ee3d60 R14: ffff88007cafb198 R15: 0000000000000000 [ 164.124296] FS: 0000000000000000(0000) GS:ffff88007fd00000(0000) knlGS:0000000000000000 [ 164.125322] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 164.126006] CR2: 0000000000000008 CR3: 000000007829c003 CR4: 00000000001606e0 [ 164.126860] Call Trace: [ 164.127045] ? call_retry_reserve+0x30/0x30 [sunrpc] [ 164.127622] rpcauth_lookupcred+0xa0/0xc0 [sunrpc] [ 164.128200] rpcauth_refreshcred+0x15f/0x170 [sunrpc] [ 164.128807] __rpc_execute+0xa9/0x460 [sunrpc] [ 164.129281] process_one_work+0x227/0x630 [ 164.129684] worker_thread+0x3c/0x390 [ 164.130062] ? process_one_work+0x630/0x630 [ 164.130609] kthread+0x11d/0x140 [ 164.130936] ? kthread_park+0x80/0x80 [ 164.131339] ret_from_fork+0x3a/0x50 [ 164.131676] Modules linked in: rpcsec_gss_krb5 nfsv4 nfs lockd grace auth_rpcgss sunrpc [ 164.132719] CR2: 0000000000000008 [ 164.133050] ---[ end trace b4028a6781a696ad ]---
Quoted 33 lines · J. Bruce Fields
Since -rc1 my regression tests crash my client. Is this a known problem? I'll investigate some more, I haven't even looked at the code yet or checked which test exactly is hitting this.
--b.
[ 164.109570] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 [ 164.111207] PGD 0 P4D 0 [ 164.111528] Oops: 0000 [#1] PREEMPT SMP PTI [ 164.112303] CPU: 2 PID: 2947 Comm: kworker/u8:5 Not tainted 4.20.0-rc1-13223-gafb6d1c474ef #1898 [ 164.113487] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20180531_142017-buildhw-08.phx2.fedoraproject.org-1.fc28 04/01/2014 [ 164.115301] Workqueue: rpciod rpc_async_schedule [sunrpc] [ 164.115920] RIP: 0010:rpcauth_lookup_credcache+0x3d/0x450 [sunrpc] [ 164.116700] Code: 89 f5 41 54 41 89 d4 53 48 83 ec 38 89 4d b0 4c 8b 7f 20 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8d 45 c0 48 89 45 c8 <41> 8b 77 08 48 89 45 c0 48 8b 47 10 4c 89 ef 48 8b 40 28 e8 cb d2 [ 164.119299] RSP: 0018:ffffc90001ee3cf0 EFLAGS: 00010246 [ 164.119872] RAX: ffffc90001ee3d10 RBX: ffff88007cc18180 RCX: 0000000000600040 [ 164.120800] RDX: 0000000000000001 RSI: ffffc90001ee3d60 RDI: ffff88007cafb198 [ 164.121643] RBP: ffffc90001ee3d50 R08: 0000000000000000 R09: 0000000000000000 [ 164.122464] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001 [ 164.123373] R13: ffffc90001ee3d60 R14: ffff88007cafb198 R15: 0000000000000000 [ 164.124296] FS: 0000000000000000(0000) GS:ffff88007fd00000(0000) knlGS:0000000000000000 [ 164.125322] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 164.126006] CR2: 0000000000000008 CR3: 000000007829c003 CR4: 00000000001606e0 [ 164.126860] Call Trace: [ 164.127045] ? call_retry_reserve+0x30/0x30 [sunrpc] [ 164.127622] rpcauth_lookupcred+0xa0/0xc0 [sunrpc] [ 164.128200] rpcauth_refreshcred+0x15f/0x170 [sunrpc] [ 164.128807] __rpc_execute+0xa9/0x460 [sunrpc] [ 164.129281] process_one_work+0x227/0x630
Quoted 9 lines
[ 164.129684] worker_thread+0x3c/0x390 [ 164.130062] ? process_one_work+0x630/0x630 [ 164.130609] kthread+0x11d/0x140 [ 164.130936] ? kthread_park+0x80/0x80 [ 164.131339] ret_from_fork+0x3a/0x50 [ 164.131676] Modules linked in: rpcsec_gss_krb5 nfsv4 nfs lockd grace auth_rpcgss sunrpc [ 164.132719] CR2: 0000000000000008 [ 164.133050] ---[ end trace b4028a6781a696ad ]---
I just encountered this repeatedly with cthon04 general tests.
MNTOPTIONS="rw,proto=tcp,vers=4.1,sec=sys"
--
Chuck Lever
[email protected]
Looks like it's the fault of
07d02a67b7faae "SUNRPC: Simplify lookup code"
--b.
Quoted 53 lines · Chuck Lever
Quoted 33 lines · J. Bruce Fields
Since -rc1 my regression tests crash my client. Is this a known problem? I'll investigate some more, I haven't even looked at the code yet or checked which test exactly is hitting this.
--b.
[ 164.109570] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 [ 164.111207] PGD 0 P4D 0 [ 164.111528] Oops: 0000 [#1] PREEMPT SMP PTI [ 164.112303] CPU: 2 PID: 2947 Comm: kworker/u8:5 Not tainted 4.20.0-rc1-13223-gafb6d1c474ef #1898 [ 164.113487] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20180531_142017-buildhw-08.phx2.fedoraproject.org-1.fc28 04/01/2014 [ 164.115301] Workqueue: rpciod rpc_async_schedule [sunrpc] [ 164.115920] RIP: 0010:rpcauth_lookup_credcache+0x3d/0x450 [sunrpc] [ 164.116700] Code: 89 f5 41 54 41 89 d4 53 48 83 ec 38 89 4d b0 4c 8b 7f 20 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8d 45 c0 48 89 45 c8 <41> 8b 77 08 48 89 45 c0 48 8b 47 10 4c 89 ef 48 8b 40 28 e8 cb d2 [ 164.119299] RSP: 0018:ffffc90001ee3cf0 EFLAGS: 00010246 [ 164.119872] RAX: ffffc90001ee3d10 RBX: ffff88007cc18180 RCX: 0000000000600040 [ 164.120800] RDX: 0000000000000001 RSI: ffffc90001ee3d60 RDI: ffff88007cafb198 [ 164.121643] RBP: ffffc90001ee3d50 R08: 0000000000000000 R09: 0000000000000000 [ 164.122464] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001 [ 164.123373] R13: ffffc90001ee3d60 R14: ffff88007cafb198 R15: 0000000000000000 [ 164.124296] FS: 0000000000000000(0000) GS:ffff88007fd00000(0000) knlGS:0000000000000000 [ 164.125322] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 164.126006] CR2: 0000000000000008 CR3: 000000007829c003 CR4: 00000000001606e0 [ 164.126860] Call Trace: [ 164.127045] ? call_retry_reserve+0x30/0x30 [sunrpc] [ 164.127622] rpcauth_lookupcred+0xa0/0xc0 [sunrpc] [ 164.128200] rpcauth_refreshcred+0x15f/0x170 [sunrpc] [ 164.128807] __rpc_execute+0xa9/0x460 [sunrpc] [ 164.129281] process_one_work+0x227/0x630Quoted 9 lines
[ 164.129684] worker_thread+0x3c/0x390 [ 164.130062] ? process_one_work+0x630/0x630 [ 164.130609] kthread+0x11d/0x140 [ 164.130936] ? kthread_park+0x80/0x80 [ 164.131339] ret_from_fork+0x3a/0x50 [ 164.131676] Modules linked in: rpcsec_gss_krb5 nfsv4 nfs lockd grace auth_rpcgss sunrpc [ 164.132719] CR2: 0000000000000008 [ 164.133050] ---[ end trace b4028a6781a696ad ]---I just encountered this repeatedly with cthon04 general tests.
MNTOPTIONS="rw,proto=tcp,vers=4.1,sec=sys"
--
Chuck Lever
[email protected]
Quoted 3 lines · Bruce Fields
Looks like it's the fault of
07d02a67b7faae"SUNRPC: Simplify lookup code"
I'm having trouble reproducing this bug. I've tried both cthon and xfstests in a loop, so far without success (both NFSv3 and v4.1, but only sec=sys). Is there anything else you're doing that I might try?
e.g. Are you running multiple workloads in parallel? Different users?..
Quoted 74 lines
--b.
Quoted 70 lines · Chuck Lever
Quoted 50 lines
On Nov 8, 2018, at 4:44 PM, J. Bruce Fields <[email protected]
Quoted 1 line
wrote:
Since -rc1 my regression tests crash my client. Is this a known problem? I'll investigate some more, I haven't even looked at the code
yet or checked which test exactly is hitting this.--b.
[ 164.109570] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 [ 164.111207] PGD 0 P4D 0 [ 164.111528] Oops: 0000 [#1] PREEMPT SMP PTI [ 164.112303] CPU: 2 PID: 2947 Comm: kworker/u8:5 Not tainted 4.20.0-rc1-13223-gafb6d1c474ef #1898 [ 164.113487] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20180531_142017-buildhw-08.phx2.fedoraproject.org- 1.fc28 04/01/2014 [ 164.115301] Workqueue: rpciod rpc_async_schedule [sunrpc] [ 164.115920] RIP: 0010:rpcauth_lookup_credcache+0x3d/0x450 [sunrpc] [ 164.116700] Code: 89 f5 41 54 41 89 d4 53 48 83 ec 38 89 4d b0 4c 8b 7f 20 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8d 45 c0 48 89 45 c8 <41> 8b 77 08 48 89 45 c0 48 8b 47 10 4c 89 ef 48 8b 40 28 e8 cb d2 [ 164.119299] RSP: 0018:ffffc90001ee3cf0 EFLAGS: 00010246 [ 164.119872] RAX: ffffc90001ee3d10 RBX: ffff88007cc18180 RCX: 0000000000600040 [ 164.120800] RDX: 0000000000000001 RSI: ffffc90001ee3d60 RDI: ffff88007cafb198 [ 164.121643] RBP: ffffc90001ee3d50 R08: 0000000000000000 R09: 0000000000000000 [ 164.122464] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001 [ 164.123373] R13: ffffc90001ee3d60 R14: ffff88007cafb198 R15: 0000000000000000 [ 164.124296] FS: 0000000000000000(0000) GS:ffff88007fd00000(0000) knlGS:0000000000000000 [ 164.125322] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 164.126006] CR2: 0000000000000008 CR3: 000000007829c003 CR4: 00000000001606e0 [ 164.126860] Call Trace: [ 164.127045] ? call_retry_reserve+0x30/0x30 [sunrpc] [ 164.127622] rpcauth_lookupcred+0xa0/0xc0 [sunrpc] [ 164.128200] rpcauth_refreshcred+0x15f/0x170 [sunrpc] [ 164.128807] __rpc_execute+0xa9/0x460 [sunrpc] [ 164.129281] process_one_work+0x227/0x630 [ 164.129684] worker_thread+0x3c/0x390Quoted 9 lines
[ 164.130062] ? process_one_work+0x630/0x630 [ 164.130609] kthread+0x11d/0x140 [ 164.130936] ? kthread_park+0x80/0x80 [ 164.131339] ret_from_fork+0x3a/0x50 [ 164.131676] Modules linked in: rpcsec_gss_krb5 nfsv4 nfs lockd grace auth_rpcgss sunrpc [ 164.132719] CR2: 0000000000000008 [ 164.133050] ---[ end trace b4028a6781a696ad ]---I just encountered this repeatedly with cthon04 general tests.
MNTOPTIONS="rw,proto=tcp,vers=4.1,sec=sys"
--
Chuck Lever
[email protected]
Signature · 5 lines
Trond Myklebust CTO, Hammerspace Inc 4300 El Camino Real, Suite 105 Los Altos, CA 94022 www.hammer.space
Quoted 7 lines · Trond Myklebust
Quoted 2 lines · Bruce Fields
Looks like it's the fault of
Quoted 7 lines
Quoted 1 line
07d02a67b7faae"SUNRPC: Simplify lookup code"I'm having trouble reproducing this bug. I've tried both cthon and xfstests in a loop, so far without success (both NFSv3 and v4.1, but only sec=sys). Is there anything else you're doing that I might try?
e.g. Are you running multiple workloads in parallel? Different users?..
Some observations, for what they are worth:
Single user test running with no other NFS workload.
I see the BUG fire at umount time, not during the test.
My client is a two-node NUMA system with 12 cores, which could be more likely to trigger races.
Export is tmpfs.
Quoted 65 lines
Quoted 65 lines
--b.
Quoted 62 lines · Chuck Lever
Quoted 50 lines
On Nov 8, 2018, at 4:44 PM, J. Bruce Fields <[email protected]
Quoted 1 line
wrote:
Since -rc1 my regression tests crash my client. Is this a known problem? I'll investigate some more, I haven't even looked at the code
yet or checked which test exactly is hitting this.--b.
[ 164.109570] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 [ 164.111207] PGD 0 P4D 0 [ 164.111528] Oops: 0000 [#1] PREEMPT SMP PTI [ 164.112303] CPU: 2 PID: 2947 Comm: kworker/u8:5 Not tainted 4.20.0-rc1-13223-gafb6d1c474ef #1898 [ 164.113487] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20180531_142017-buildhw-08.phx2.fedoraproject.org- 1.fc28 04/01/2014 [ 164.115301] Workqueue: rpciod rpc_async_schedule [sunrpc] [ 164.115920] RIP: 0010:rpcauth_lookup_credcache+0x3d/0x450 [sunrpc] [ 164.116700] Code: 89 f5 41 54 41 89 d4 53 48 83 ec 38 89 4d b0 4c 8b 7f 20 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8d 45 c0 48 89 45 c8 <41> 8b 77 08 48 89 45 c0 48 8b 47 10 4c 89 ef 48 8b 40 28 e8 cb d2 [ 164.119299] RSP: 0018:ffffc90001ee3cf0 EFLAGS: 00010246 [ 164.119872] RAX: ffffc90001ee3d10 RBX: ffff88007cc18180 RCX: 0000000000600040 [ 164.120800] RDX: 0000000000000001 RSI: ffffc90001ee3d60 RDI: ffff88007cafb198 [ 164.121643] RBP: ffffc90001ee3d50 R08: 0000000000000000 R09: 0000000000000000 [ 164.122464] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001 [ 164.123373] R13: ffffc90001ee3d60 R14: ffff88007cafb198 R15: 0000000000000000 [ 164.124296] FS: 0000000000000000(0000) GS:ffff88007fd00000(0000) knlGS:0000000000000000 [ 164.125322] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 164.126006] CR2: 0000000000000008 CR3: 000000007829c003 CR4: 00000000001606e0 [ 164.126860] Call Trace: [ 164.127045] ? call_retry_reserve+0x30/0x30 [sunrpc] [ 164.127622] rpcauth_lookupcred+0xa0/0xc0 [sunrpc] [ 164.128200] rpcauth_refreshcred+0x15f/0x170 [sunrpc] [ 164.128807] __rpc_execute+0xa9/0x460 [sunrpc] [ 164.129281] process_one_work+0x227/0x630 [ 164.129684] worker_thread+0x3c/0x390Quoted 9 lines
[ 164.130062] ? process_one_work+0x630/0x630 [ 164.130609] kthread+0x11d/0x140 [ 164.130936] ? kthread_park+0x80/0x80 [ 164.131339] ret_from_fork+0x3a/0x50 [ 164.131676] Modules linked in: rpcsec_gss_krb5 nfsv4 nfs lockd grace auth_rpcgss sunrpc [ 164.132719] CR2: 0000000000000008 [ 164.133050] ---[ end trace b4028a6781a696ad ]---I just encountered this repeatedly with cthon04 general tests.
Quoted 14 lines
Quoted 8 lines
Quoted 8 lines
MNTOPTIONS="rw,proto=tcp,vers=4.1,sec=sys"
--
Chuck Lever
[email protected]Signature · 5 lines
Trond Myklebust CTO, Hammerspace Inc 4300 El Camino Real, Suite 105 Los Altos, CA 94022 www.hammer.space
--
Chuck Lever
[email protected]
Quoted 19 lines · Chuck Lever
Quoted 8 lines
On Nov 12, 2018, at 9:59 AM, Trond Myklebust <
[email protected]> wrote:Quoted 3 lines · Bruce Fields
Looks like it's the fault of
07d02a67b7faae"SUNRPC: Simplify lookup code"Quoted 8 lines
I'm having trouble reproducing this bug. I've tried both cthon and xfstests in a loop, so far without success (both NFSv3 and v4.1, but
only sec=sys). Is there anything else you're doing that I might try?e.g. Are you running multiple workloads in parallel? Different users?..
Some observations, for what they are worth:
Quoted 9 lines
Single user test running with no other NFS workload.
I see the BUG fire at umount time, not during the test.
My client is a two-node NUMA system with 12 cores, which could be more likely to trigger races.
Export is tmpfs.
Thanks! That's useful info. Particularly the observation that you're seeing it at umount time...
Quoted 103 lines
Quoted 82 lines
Quoted 82 lines
--b.
Quoted 78 lines · Chuck Lever
Quoted 66 lines
On Nov 8, 2018, at 4:44 PM, J. Bruce Fields <
[email protected]Quoted 1 line
wrote:
Since -rc1 my regression tests crash my client. Is this a known
problem? I'll investigate some more, I haven't even looked at
the code
yet or checked which test exactly is hitting this.--b.
[ 164.109570] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 [ 164.111207] PGD 0 P4D 0 [ 164.111528] Oops: 0000 [#1] PREEMPT SMP PTI [ 164.112303] CPU: 2 PID: 2947 Comm: kworker/u8:5 Not tainted 4.20.0-rc1-13223-gafb6d1c474ef #1898 [ 164.113487] Hardware name: QEMU Standard PC (i440FX +PIIX,
1996), BIOS ?-20180531_142017-buildhw-
08.phx2.fedoraproject.org-
1.fc28 04/01/2014[ 164.115301] Workqueue: rpciod rpc_async_schedule [sunrpc] [ 164.115920] RIP: 0010:rpcauth_lookup_credcache+0x3d/0x450 [sunrpc] [ 164.116700] Code: 89 f5 41 54 41 89 d4 53 48 83 ec 38 894d b0
4c 8b 7f 20 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8d 45
c0 48 89 45 c8 <41> 8b 77 08 48 89 45 c0 48 8b 47 10 4c 89 ef 48
8b 40 28 e8 cb d2[ 164.119299] RSP: 0018:ffffc90001ee3cf0 EFLAGS: 00010246 [ 164.119872] RAX: ffffc90001ee3d10 RBX: ffff88007cc18180 RCX: 0000000000600040 [ 164.120800] RDX: 0000000000000001 RSI: ffffc90001ee3d60 RDI: ffff88007cafb198 [ 164.121643] RBP: ffffc90001ee3d50 R08: 0000000000000000 R09: 0000000000000000 [ 164.122464] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001 [ 164.123373] R13: ffffc90001ee3d60 R14: ffff88007cafb198 R15: 0000000000000000 [ 164.124296] FS: 0000000000000000(0000) GS:ffff88007fd00000(0000) knlGS:0000000000000000 [ 164.125322] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 164.126006] CR2: 0000000000000008 CR3: 000000007829c003 CR4: 00000000001606e0 [ 164.126860] Call Trace: [ 164.127045] ? call_retry_reserve+0x30/0x30 [sunrpc] [ 164.127622] rpcauth_lookupcred+0xa0/0xc0 [sunrpc] [ 164.128200] rpcauth_refreshcred+0x15f/0x170 [sunrpc] [ 164.128807] __rpc_execute+0xa9/0x460 [sunrpc] [ 164.129281] process_one_work+0x227/0x630 [ 164.129684] worker_thread+0x3c/0x390 [ 164.130062] ? process_one_work+0x630/0x630Quoted 9 lines
[ 164.130609] kthread+0x11d/0x140 [ 164.130936] ? kthread_park+0x80/0x80 [ 164.131339] ret_from_fork+0x3a/0x50 [ 164.131676] Modules linked in: rpcsec_gss_krb5 nfsv4 nfs lockd grace auth_rpcgss sunrpc [ 164.132719] CR2: 0000000000000008 [ 164.133050] ---[ end trace b4028a6781a696ad ]---I just encountered this repeatedly with cthon04 general tests.
Quoted 14 lines
Quoted 8 lines
Quoted 8 lines
MNTOPTIONS="rw,proto=tcp,vers=4.1,sec=sys"
--
Chuck Lever
[email protected]Signature · 5 lines
Trond Myklebust CTO, Hammerspace Inc 4300 El Camino Real, Suite 105 Los Altos, CA 94022 www.hammer.space--
Chuck Lever
[email protected]
Signature · 5 lines
Trond Myklebust CTO, Hammerspace Inc 4300 El Camino Real, Suite 105 Los Altos, CA 94022 www.hammer.space
Quoted 10 lines · Trond Myklebust
Quoted 3 lines · Bruce Fields
Looks like it's the fault of
07d02a67b7faae"SUNRPC: Simplify lookup code"I'm having trouble reproducing this bug. I've tried both cthon and xfstests in a loop, so far without success (both NFSv3 and v4.1, but only sec=sys). Is there anything else you're doing that I might try?
e.g. Are you running multiple workloads in parallel? Different users?..
Nothing that interesting. Currently it's connectathon over v4, v3, v4/krb5, v3/krb5, v4/krb5i, v4/krb5p, v4.1, v4.1/krb5, but just serially one after the other. Then some pynfs tests (which bypass the client), then xfstests over v4.2/sys. And also a few one-off locking tests of my own that probably aren't a factor here.
(Hah, I just realized I was mounting with vers=4 and assuming that meant 4.0, but actually it's changed over time depending on the defaults, so currently those "v4" runs are actually all 4.2. Gah.)
--b.
Quoted 83 lines
Quoted 75 lines
--b.
Quoted 70 lines · Chuck Lever
Quoted 50 lines
On Nov 8, 2018, at 4:44 PM, J. Bruce Fields <[email protected]
Quoted 1 line
wrote:
Since -rc1 my regression tests crash my client. Is this a known problem? I'll investigate some more, I haven't even looked at the code
yet or checked which test exactly is hitting this.--b.
[ 164.109570] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 [ 164.111207] PGD 0 P4D 0 [ 164.111528] Oops: 0000 [#1] PREEMPT SMP PTI [ 164.112303] CPU: 2 PID: 2947 Comm: kworker/u8:5 Not tainted 4.20.0-rc1-13223-gafb6d1c474ef #1898 [ 164.113487] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20180531_142017-buildhw-08.phx2.fedoraproject.org- 1.fc28 04/01/2014 [ 164.115301] Workqueue: rpciod rpc_async_schedule [sunrpc] [ 164.115920] RIP: 0010:rpcauth_lookup_credcache+0x3d/0x450 [sunrpc] [ 164.116700] Code: 89 f5 41 54 41 89 d4 53 48 83 ec 38 89 4d b0 4c 8b 7f 20 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8d 45 c0 48 89 45 c8 <41> 8b 77 08 48 89 45 c0 48 8b 47 10 4c 89 ef 48 8b 40 28 e8 cb d2 [ 164.119299] RSP: 0018:ffffc90001ee3cf0 EFLAGS: 00010246 [ 164.119872] RAX: ffffc90001ee3d10 RBX: ffff88007cc18180 RCX: 0000000000600040 [ 164.120800] RDX: 0000000000000001 RSI: ffffc90001ee3d60 RDI: ffff88007cafb198 [ 164.121643] RBP: ffffc90001ee3d50 R08: 0000000000000000 R09: 0000000000000000 [ 164.122464] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001 [ 164.123373] R13: ffffc90001ee3d60 R14: ffff88007cafb198 R15: 0000000000000000 [ 164.124296] FS: 0000000000000000(0000) GS:ffff88007fd00000(0000) knlGS:0000000000000000 [ 164.125322] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 164.126006] CR2: 0000000000000008 CR3: 000000007829c003 CR4: 00000000001606e0 [ 164.126860] Call Trace: [ 164.127045] ? call_retry_reserve+0x30/0x30 [sunrpc] [ 164.127622] rpcauth_lookupcred+0xa0/0xc0 [sunrpc] [ 164.128200] rpcauth_refreshcred+0x15f/0x170 [sunrpc] [ 164.128807] __rpc_execute+0xa9/0x460 [sunrpc] [ 164.129281] process_one_work+0x227/0x630 [ 164.129684] worker_thread+0x3c/0x390Quoted 9 lines
[ 164.130062] ? process_one_work+0x630/0x630 [ 164.130609] kthread+0x11d/0x140 [ 164.130936] ? kthread_park+0x80/0x80 [ 164.131339] ret_from_fork+0x3a/0x50 [ 164.131676] Modules linked in: rpcsec_gss_krb5 nfsv4 nfs lockd grace auth_rpcgss sunrpc [ 164.132719] CR2: 0000000000000008 [ 164.133050] ---[ end trace b4028a6781a696ad ]---I just encountered this repeatedly with cthon04 general tests.
MNTOPTIONS="rw,proto=tcp,vers=4.1,sec=sys"
--
Chuck Lever
[email protected]Signature · 5 lines
Trond Myklebust CTO, Hammerspace Inc 4300 El Camino Real, Suite 105 Los Altos, CA 94022 www.hammer.space
Quoted 24 lines · On Mon, 2018-11-12 at 13:24 -0500, [email protected] wrote:
Quoted 5 lines · Trond Myklebust
Quoted 3 lines · Bruce Fields
Looks like it's the fault of
07d02a67b7faae"SUNRPC: Simplify lookup code"Quoted 8 lines
I'm having trouble reproducing this bug. I've tried both cthon and xfstests in a loop, so far without success (both NFSv3 and v4.1, but
only sec=sys). Is there anything else you're doing that I might try?e.g. Are you running multiple workloads in parallel? Different users?..
Nothing that interesting. Currently it's connectathon over v4, v3, v4/krb5, v3/krb5, v4/krb5i, v4/krb5p, v4.1, v4.1/krb5, but just serially
one after the other. Then some pynfs tests (which bypass the client),
then xfstests over v4.2/sys. And also a few one-off locking tests of my
own that probably aren't a factor here.
Quoted 5 lines
(Hah, I just realized I was mounting with vers=4 and assuming that meant
4.0, but actually it's changed over time depending on the defaults, so
currently those "v4" runs are actually all 4.2. Gah.)
Are you perhaps both using RPCSEC_GSS w/ integrity checking for your EXCHANGE_ID authentication? The client will attempt to use that by default if rpc.gssd is running.
I ask because I think the issue might be with RPCSEC_GSS, specifically with the RPCSEC_GSS context destroy code, hence the 2 patches that I just sent out.
Cheers
Trond
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Quoted 31 lines · Trond Myklebust
Quoted 24 lines · On Mon, 2018-11-12 at 13:24 -0500, [email protected] wrote:
Quoted 5 lines · Trond Myklebust
Quoted 3 lines · Bruce Fields
Looks like it's the fault of
07d02a67b7faae"SUNRPC: Simplify lookup code"Quoted 8 lines
I'm having trouble reproducing this bug. I've tried both cthon and xfstests in a loop, so far without success (both NFSv3 and v4.1, but
only sec=sys). Is there anything else you're doing that I might try?e.g. Are you running multiple workloads in parallel? Different users?..
Nothing that interesting. Currently it's connectathon over v4, v3, v4/krb5, v3/krb5, v4/krb5i, v4/krb5p, v4.1, v4.1/krb5, but just serially
one after the other. Then some pynfs tests (which bypass the client),
then xfstests over v4.2/sys. And also a few one-off locking tests of my
own that probably aren't a factor here.Quoted 5 lines
(Hah, I just realized I was mounting with vers=4 and assuming that meant
4.0, but actually it's changed over time depending on the defaults, so
currently those "v4" runs are actually all 4.2. Gah.)
Quoted 3 lines
Are you perhaps both using
RPCSEC_GSSw/ integrity checking for yourEXCHANGE_IDauthentication? The client will attempt to use that by default if rpc.gssd is running.
Yes, in addition to the krb5i mount I'd expect the sys/krb5/krb5p mounts are using krb5i for EXCHANGE_ID.
Quoted 3 lines
I ask because I think the issue might be with
RPCSEC_GSS, specifically with theRPCSEC_GSScontext destroy code, hence the 2 patches that I just sent out.
Looks like my tests pass after applying those two patches.
--b.
Quoted 47 lines · On Mon, 2018-11-12 at 18:01 -0500, [email protected] wrote:
Quoted 37 lines · Trond Myklebust
Quoted 28 lines · On Mon, 2018-11-12 at 13:24 -0500, [email protected] wrote:
Quoted 13 lines · Trond Myklebust
Quoted 3 lines · Bruce Fields
Looks like it's the fault of
07d02a67b7faae"SUNRPC: Simplify lookup code"I'm having trouble reproducing this bug. I've tried both cthon and
xfstests in a loop, so far without success (both NFSv3 and v4.1,
but
only sec=sys). Is there anything else you're doing that I might try?Quoted 2 lines
e.g. Are you running multiple workloads in parallel? Different users?..
Nothing that interesting. Currently it's connectathon over v4, v3,
v4/krb5, v3/krb5, v4/krb5i, v4/krb5p, v4.1, v4.1/krb5, but just serially
one after the other. Then some pynfs tests (which bypass the client),
then xfstests over v4.2/sys. And also a few one-off locking tests of
my
own that probably aren't a factor here.Quoted 7 lines
(Hah, I just realized I was mounting with vers=4 and assuming that
meant
4.0, but actually it's changed over time depending on the defaults,
so
currently those "v4" runs are actually all 4.2. Gah.)Quoted 4 lines
Are you perhaps both using
RPCSEC_GSSw/ integrity checking for yourEXCHANGE_IDauthentication? The client will attempt to use that by default if rpc.gssd is running.Yes, in addition to the krb5i mount I'd expect the sys/krb5/krb5p mounts
are using krb5i forEXCHANGE_ID.
Quoted 8 lines
Quoted 5 lines
I ask because I think the issue might be with
RPCSEC_GSS, specifically
with theRPCSEC_GSScontext destroy code, hence the 2 patches that I
just sent out.Looks like my tests pass after applying those two patches.
Cool! Thanks for testing.
Chuck, do you think the above might also explain your sighting of the same Oops?
Cheers
Trond
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Quoted 57 lines · Trond Myklebust
Quoted 55 lines · On Mon, 2018-11-12 at 18:01 -0500, [email protected] wrote:
Quoted 38 lines · Trond Myklebust
Quoted 29 lines · On Mon, 2018-11-12 at 13:24 -0500, [email protected] wrote:
Quoted 14 lines · Trond Myklebust
Quoted 4 lines · Bruce Fields
Looks like it's the fault of
07d02a67b7faae"SUNRPC: Simplify lookup code"I'm having trouble reproducing this bug. I've tried both cthon and
xfstests in a loop, so far without success (both NFSv3 and v4.1,
but
only sec=sys). Is there anything else you're doing that I might try?Quoted 2 lines
e.g. Are you running multiple workloads in parallel? Different users?..
Nothing that interesting. Currently it's connectathon over v4, v3,
v4/krb5, v3/krb5, v4/krb5i, v4/krb5p, v4.1, v4.1/krb5, but just serially
one after the other. Then some pynfs tests (which bypass the client),
then xfstests over v4.2/sys. And also a few one-off locking tests of
my
own that probably aren't a factor here.Quoted 7 lines
(Hah, I just realized I was mounting with vers=4 and assuming that
meant
4.0, but actually it's changed over time depending on the defaults,
so
currently those "v4" runs are actually all 4.2. Gah.)Quoted 4 lines
Are you perhaps both using
RPCSEC_GSSw/ integrity checking for yourEXCHANGE_IDauthentication? The client will attempt to use that by default if rpc.gssd is running.Yes, in addition to the krb5i mount I'd expect the sys/krb5/krb5p mounts
are using krb5i forEXCHANGE_ID.Quoted 5 lines
I ask because I think the issue might be with
RPCSEC_GSS, specifically
with theRPCSEC_GSScontext destroy code, hence the 2 patches that I
just sent out.
Quoted 7 lines
Quoted 2 lines
Looks like my tests pass after applying those two patches.
Cool! Thanks for testing.
Chuck, do you think the above might also explain your sighting of the same Oops?
Could be, I don’t think I saw it until I started testing NFSv4. I won’t be able to confirm that until next week.
Quoted 9 lines
Cheers
TrondSignature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Quoted 70 lines · Chuck Lever
Quoted 70 lines
Quoted 66 lines · Trond Myklebust
Quoted 46 lines · On Mon, 2018-11-12 at 18:01 -0500, [email protected] wrote:
Quoted 36 lines · Trond Myklebust
Quoted 17 lines · On Mon, 2018-11-12 at 13:24 -0500, [email protected] wrote:
Quoted 4 lines · Trond Myklebust
On Sat, 2018-11-10 at 16:49 -0500, Bruce Fields wrote:
Looks like it's the fault of
07d02a67b7faae"SUNRPC: Simplify lookup code"I'm having trouble reproducing this bug. I've tried both cthon
and
xfstests in a loop, so far without success (both NFSv3 and v4.1,
but
only sec=sys). Is there anything else you're doing that I might
try?Quoted 3 lines
e.g. Are you running multiple workloads in parallel? Different
users?..Nothing that interesting. Currently it's connectathon over v4,
v3,
v4/krb5, v3/krb5, v4/krb5i, v4/krb5p, v4.1, v4.1/krb5, but just
serially
one after the other. Then some pynfs tests (which bypass the client),
then xfstests over v4.2/sys. And also a few one-off locking tests of
my
own that probably aren't a factor here.Quoted 7 lines
(Hah, I just realized I was mounting with vers=4 and assuming that
meant
4.0, but actually it's changed over time depending on the defaults,
so
currently those "v4" runs are actually all 4.2. Gah.)Quoted 5 lines
Are you perhaps both using
RPCSEC_GSSw/ integrity checking for yourEXCHANGE_IDauthentication? The client will attempt to use that by
default if rpc.gssd is running.Yes, in addition to the krb5i mount I'd expect the sys/krb5/krb5p mounts
are using krb5i forEXCHANGE_ID.Quoted 6 lines
I ask because I think the issue might be with
RPCSEC_GSS, specifically
with theRPCSEC_GSScontext destroy code, hence the 2 patches that
I
just sent out.Looks like my tests pass after applying those two patches.
Quoted 9 lines
Quoted 5 lines
Cool! Thanks for testing.
Chuck, do you think the above might also explain your sighting of the
same Oops?Could be, I don’t think I saw it until I started testing NFSv4. I won’t be able to confirm that until next week.
OK. Either way, I know that part of the GSS code needs to be fixed in order to deal with the reference count being 0, so I think it is worth merging this patch now, and then we can see if there is more to the regression when you can get back to your test rig.
Thanks
Trond
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Quoted 80 lines · Trond Myklebust
Quoted 76 lines · Chuck Lever
Quoted 75 lines
Quoted 57 lines · Trond Myklebust
Quoted 47 lines · On Mon, 2018-11-12 at 18:01 -0500, [email protected] wrote:
Quoted 36 lines · Trond Myklebust
Quoted 17 lines · On Mon, 2018-11-12 at 13:24 -0500, [email protected] wrote:
Quoted 4 lines · Trond Myklebust
On Sat, 2018-11-10 at 16:49 -0500, Bruce Fields wrote:
Looks like it's the fault of
07d02a67b7faae"SUNRPC: Simplify lookup code"I'm having trouble reproducing this bug. I've tried both cthon
and
xfstests in a loop, so far without success (both NFSv3 and v4.1,
but
only sec=sys). Is there anything else you're doing that I might
try?Quoted 3 lines
e.g. Are you running multiple workloads in parallel? Different
users?..Nothing that interesting. Currently it's connectathon over v4,
v3,
v4/krb5, v3/krb5, v4/krb5i, v4/krb5p, v4.1, v4.1/krb5, but just
serially
one after the other. Then some pynfs tests (which bypass the client),
then xfstests over v4.2/sys. And also a few one-off locking tests of
my
own that probably aren't a factor here.Quoted 7 lines
(Hah, I just realized I was mounting with vers=4 and assuming that
meant
4.0, but actually it's changed over time depending on the defaults,
so
currently those "v4" runs are actually all 4.2. Gah.)Quoted 5 lines
Are you perhaps both using
RPCSEC_GSSw/ integrity checking for yourEXCHANGE_IDauthentication? The client will attempt to use that by
default if rpc.gssd is running.Yes, in addition to the krb5i mount I'd expect the sys/krb5/krb5p mounts
are using krb5i forEXCHANGE_ID.Quoted 9 lines
Quoted 6 lines
I ask because I think the issue might be with
RPCSEC_GSS, specifically
with theRPCSEC_GSScontext destroy code, hence the 2 patches that
I
just sent out.Looks like my tests pass after applying those two patches.
Cool! Thanks for testing.
Chuck, do you think the above might also explain your sighting of the
same Oops?
Quoted 8 lines
Quoted 3 lines
Could be, I don’t think I saw it until I started testing NFSv4. I won’t be able to confirm that until next week.
OK. Either way, I know that part of the GSS code needs to be fixed in order to deal with the reference count being 0, so I think it is worth merging this patch now, and then we can see if there is more to the regression when you can get back to your test rig.
Sounds fine to me.
Quoted 8 lines
Thanks
TrondSignature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]