Skip to contentSkip to conversation

PROBLEM: NULL pointer dereference; nfsd4_remove_cld_pipe

DISCUSSION Jamie Heilman 2 participants 5 messages Nov 12, 2019

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Giving 5.4.0-rc7 a spin I hit a NULL pointer dereference and bisected it to:

commit 6ee95d1c899186c0798cafd25998d436bcdb9618

Author: Scott Mayhew <[email protected]>
Date:   Mon Sep 9 16:10:31 2019 -0400

nfsd: add support for upcall version 2

The splat against 5.3.0-rc2-00034-g6ee95d1c8991:

BUG: kernel NULL pointer dereference, address: 0000000000000036
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0 
Oops: 0000 [#1] PREEMPT SMP PTI
CPU: 0 PID: 2936 Comm: rpc.nfsd Not tainted 5.3.0-rc2-00034-g6ee95d1c8991 #1
Hardware name: Dell Inc. Precision WorkStation T3400  /0TP412, BIOS A14 04/30/2012
RIP: 0010:crypto_destroy_tfm+0x5/0x4d
Code: 78 01 00 00 48 85 c0 74 05 e9 05 05 66 00 c3 55 48 8b af 80 01 00 00 e8 d5 ff ff ff 48 89 ef 5d e9 12 f9 ef ff 48 85 ff 74 47 <48> 83 7e 30 00 41 55 4c 8b 6e 38 41 54 49 89 fc 55 48 89 f5 75 14
RSP: 0018:ffffc90000b7bd68 EFLAGS: 00010282
RAX: ffffffffa0402841 RBX: ffff888230484400 RCX: 0000000000002cd0
RDX: 0000000000002cce RSI: 0000000000000006 RDI: fffffffffffffffe
RBP: ffffffff81e68440 R08: ffff888232801800 R09: ffffffffa0402841
R10: 0000000000000200 R11: ffff88823048ae40 R12: ffff888231585100
R13: ffff88823048ae40 R14: 000000000000000b R15: ffff888230484400
FS:  00007f02102c3740(0000) GS:ffff888233a00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000036 CR3: 0000000230f94000 CR4: 00000000000406f0
Call Trace:
 nfsd4_remove_cld_pipe+0x6d/0x83 [nfsd]
 nfsd4_cld_tracking_init+0x1cf/0x295 [nfsd]
 nfsd4_client_tracking_init+0x72/0x13e [nfsd]
 nfs4_state_start_net+0x22a/0x2cf [nfsd]
 nfsd_svc+0x1c6/0x292 [nfsd]
 write_threads+0x68/0xb0 [nfsd]
 ? write_versions+0x333/0x333 [nfsd]
 nfsctl_transaction_write+0x4a/0x62 [nfsd]
 vfs_write+0xa0/0xdd
 ksys_write+0x71/0xba
 do_syscall_64+0x48/0x55
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x7f021056c904
Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb bb 0f 1f 80 00 00 00 00 48 8d 05 d9 3a 0d 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 48 83 ec 28 48 89 54 24 18 48
RSP: 002b:00007ffdc76ec618 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 000055b534955560 RCX: 00007f021056c904
RDX: 0000000000000002 RSI: 000055b534955560 RDI: 0000000000000003
RBP: 0000000000000003 R08: 0000000000000000 R09: 00007ffdc76ec4b0
R10: 00007ffdc76ec367 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000008 R14: 0000000000000000 R15: 000055b534b8a2a0
Modules linked in: cpufreq_userspace cpufreq_powersave cpufreq_ondemand cpufreq_conservative autofs4 fan nfsd auth_rpcgss nfs lockd grace fscache sunrpc bridge stp llc nhpoly1305_sse2 nhpoly1305 aes_generic chacha_x86_64 chacha_generic adiantum poly1305_generic vhost_net tun vhost tap dm_crypt snd_hda_codec_analog snd_hda_codec_generic usb_storage snd_hda_intel kvm_intel snd_hda_codec kvm snd_hwdep snd_hda_core snd_pcm dcdbas snd_timer irqbypass snd soundcore sr_mod cdrom tg3 sg floppy evdev xfs dm_mod raid1 md_mod psmouse
CR2: 0000000000000036
---[ end trace bc12bbe4cdd6319f ]---

...
NFS: Registering the id_resolver key type
Key type id_resolver registered
Key type id_legacy registered

My kernel config is at
http://audible.transient.net/~jamie/k/upcallv2.config-5.3.0-rc2-00034-g6ee95d1c8991

I don't think there's anything terribly interesting about my nfs server setup, this happens reliably on boot up, idle network, no active clients; let me know what else you need, happy to debug.

Signature · 1 line
Jamie Heilman                     http://audible.transient.net/~jamie/
#1 lore raw

Hi Jamie,

Quoted 60 lines · Jamie Heilman

Giving 5.4.0-rc7 a spin I hit a NULL pointer dereference and bisected it to:

commit 6ee95d1c899186c0798cafd25998d436bcdb9618

Author: Scott Mayhew <[email protected]>
Date:   Mon Sep 9 16:10:31 2019 -0400

nfsd: add support for upcall version 2

The splat against 5.3.0-rc2-00034-g6ee95d1c8991:

BUG: kernel NULL pointer dereference, address: 0000000000000036
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0 
Oops: 0000 [#1] PREEMPT SMP PTI
CPU: 0 PID: 2936 Comm: rpc.nfsd Not tainted 5.3.0-rc2-00034-g6ee95d1c8991 #1
Hardware name: Dell Inc. Precision WorkStation T3400  /0TP412, BIOS A14 04/30/2012
RIP: 0010:crypto_destroy_tfm+0x5/0x4d
Code: 78 01 00 00 48 85 c0 74 05 e9 05 05 66 00 c3 55 48 8b af 80 01 00 00 e8 d5 ff ff ff 48 89 ef 5d e9 12 f9 ef ff 48 85 ff 74 47 <48> 83 7e 30 00 41 55 4c 8b 6e 38 41 54 49 89 fc 55 48 89 f5 75 14
RSP: 0018:ffffc90000b7bd68 EFLAGS: 00010282
RAX: ffffffffa0402841 RBX: ffff888230484400 RCX: 0000000000002cd0
RDX: 0000000000002cce RSI: 0000000000000006 RDI: fffffffffffffffe
RBP: ffffffff81e68440 R08: ffff888232801800 R09: ffffffffa0402841
R10: 0000000000000200 R11: ffff88823048ae40 R12: ffff888231585100
R13: ffff88823048ae40 R14: 000000000000000b R15: ffff888230484400
FS:  00007f02102c3740(0000) GS:ffff888233a00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000036 CR3: 0000000230f94000 CR4: 00000000000406f0
Call Trace:
 nfsd4_remove_cld_pipe+0x6d/0x83 [nfsd]
 nfsd4_cld_tracking_init+0x1cf/0x295 [nfsd]
 nfsd4_client_tracking_init+0x72/0x13e [nfsd]
 nfs4_state_start_net+0x22a/0x2cf [nfsd]
 nfsd_svc+0x1c6/0x292 [nfsd]
 write_threads+0x68/0xb0 [nfsd]
 ? write_versions+0x333/0x333 [nfsd]
 nfsctl_transaction_write+0x4a/0x62 [nfsd]
 vfs_write+0xa0/0xdd
 ksys_write+0x71/0xba
 do_syscall_64+0x48/0x55
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x7f021056c904
Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb bb 0f 1f 80 00 00 00 00 48 8d 05 d9 3a 0d 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 48 83 ec 28 48 89 54 24 18 48
RSP: 002b:00007ffdc76ec618 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 000055b534955560 RCX: 00007f021056c904
RDX: 0000000000000002 RSI: 000055b534955560 RDI: 0000000000000003
RBP: 0000000000000003 R08: 0000000000000000 R09: 00007ffdc76ec4b0
R10: 00007ffdc76ec367 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000008 R14: 0000000000000000 R15: 000055b534b8a2a0
Modules linked in: cpufreq_userspace cpufreq_powersave cpufreq_ondemand cpufreq_conservative autofs4 fan nfsd auth_rpcgss nfs lockd grace fscache sunrpc bridge stp llc nhpoly1305_sse2 nhpoly1305 aes_generic chacha_x86_64 chacha_generic adiantum poly1305_generic vhost_net tun vhost tap dm_crypt snd_hda_codec_analog snd_hda_codec_generic usb_storage snd_hda_intel kvm_intel snd_hda_codec kvm snd_hwdep snd_hda_core snd_pcm dcdbas snd_timer irqbypass snd soundcore sr_mod cdrom tg3 sg floppy evdev xfs dm_mod raid1 md_mod psmouse
CR2: 0000000000000036
---[ end trace bc12bbe4cdd6319f ]---

...
NFS: Registering the id_resolver key type
Key type id_resolver registered
Key type id_legacy registered

Quoted 10 lines

My kernel config is at
http://audible.transient.net/~jamie/k/upcallv2.config-5.3.0-rc2-00034-g6ee95d1c8991

I don't think there's anything terribly interesting about my nfs server setup, this happens reliably on boot up, idle network, no active clients; let me know what else you need, happy to debug.

Signature · 1 line
Jamie Heilman                     http://audible.transient.net/~jamie/

Does this patch help?

-Scott

From e46430ef6ee045ec447a0796b419b8cdeee4f25f Mon Sep 17 00:00:00 2001
From: Scott Mayhew <[email protected]>
Date: Tue, 12 Nov 2019 10:10:00 -0500
Subject: [PATCH] nfsd: Fix cld_net->cn_tfm initialization

Don't assign an error pointer to cn->cn_tfm, otherwise an oops will occur in nfsd4_remove_cld_pipe().

Fixes 6ee95d1c8991 ("nfsd: add support for upcall version 2")
---
 fs/nfsd/nfs4recover.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
fs/nfsd/nfs4recover.c +5 -2
index cdc75ad4438b..af07d0f55fe3 100644
@@ -1578,6 +1578,7 @@ nfsd4_cld_tracking_init(struct net *net)
 	struct nfsd_net *nn = net_generic(net, nfsd_net_id);
 	bool running;
 	int retries = 10;
+	struct crypto_shash *tfm;
 
 	status = nfs4_cld_state_init(net);
 	if (status)
@@ -1586,11 +1587,13 @@ nfsd4_cld_tracking_init(struct net *net)
 	status = __nfsd4_init_cld_pipe(net);
 	if (status)
 		goto err_shutdown;
+	tfm = crypto_alloc_shash("sha256", 0, 0);
 	nn->cld_net->cn_tfm = crypto_alloc_shash("sha256", 0, 0);
-	if (IS_ERR(nn->cld_net->cn_tfm)) {
-		status = PTR_ERR(nn->cld_net->cn_tfm);
+	if (IS_ERR(tfm)) {
+		status = PTR_ERR(tfm);
 		goto err_remove;
 	}
+	nn->cld_net->cn_tfm = tfm;
 
 	/*
 	 * rpc pipe upcalls take 30 seconds to time out, so we don't want to
Signature · 1 line
2.17.2
#2 lore raw
In reply to Jamie Heilman

Hi Jamie,

Quoted 60 lines · Jamie Heilman

Giving 5.4.0-rc7 a spin I hit a NULL pointer dereference and bisected it to:

commit 6ee95d1c899186c0798cafd25998d436bcdb9618

Author: Scott Mayhew <[email protected]>
Date:   Mon Sep 9 16:10:31 2019 -0400

nfsd: add support for upcall version 2

The splat against 5.3.0-rc2-00034-g6ee95d1c8991:

BUG: kernel NULL pointer dereference, address: 0000000000000036
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0 
Oops: 0000 [#1] PREEMPT SMP PTI
CPU: 0 PID: 2936 Comm: rpc.nfsd Not tainted 5.3.0-rc2-00034-g6ee95d1c8991 #1
Hardware name: Dell Inc. Precision WorkStation T3400  /0TP412, BIOS A14 04/30/2012
RIP: 0010:crypto_destroy_tfm+0x5/0x4d
Code: 78 01 00 00 48 85 c0 74 05 e9 05 05 66 00 c3 55 48 8b af 80 01 00 00 e8 d5 ff ff ff 48 89 ef 5d e9 12 f9 ef ff 48 85 ff 74 47 <48> 83 7e 30 00 41 55 4c 8b 6e 38 41 54 49 89 fc 55 48 89 f5 75 14
RSP: 0018:ffffc90000b7bd68 EFLAGS: 00010282
RAX: ffffffffa0402841 RBX: ffff888230484400 RCX: 0000000000002cd0
RDX: 0000000000002cce RSI: 0000000000000006 RDI: fffffffffffffffe
RBP: ffffffff81e68440 R08: ffff888232801800 R09: ffffffffa0402841
R10: 0000000000000200 R11: ffff88823048ae40 R12: ffff888231585100
R13: ffff88823048ae40 R14: 000000000000000b R15: ffff888230484400
FS:  00007f02102c3740(0000) GS:ffff888233a00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000036 CR3: 0000000230f94000 CR4: 00000000000406f0
Call Trace:
 nfsd4_remove_cld_pipe+0x6d/0x83 [nfsd]
 nfsd4_cld_tracking_init+0x1cf/0x295 [nfsd]
 nfsd4_client_tracking_init+0x72/0x13e [nfsd]
 nfs4_state_start_net+0x22a/0x2cf [nfsd]
 nfsd_svc+0x1c6/0x292 [nfsd]
 write_threads+0x68/0xb0 [nfsd]
 ? write_versions+0x333/0x333 [nfsd]
 nfsctl_transaction_write+0x4a/0x62 [nfsd]
 vfs_write+0xa0/0xdd
 ksys_write+0x71/0xba
 do_syscall_64+0x48/0x55
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x7f021056c904
Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb bb 0f 1f 80 00 00 00 00 48 8d 05 d9 3a 0d 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 48 83 ec 28 48 89 54 24 18 48
RSP: 002b:00007ffdc76ec618 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 000055b534955560 RCX: 00007f021056c904
RDX: 0000000000000002 RSI: 000055b534955560 RDI: 0000000000000003
RBP: 0000000000000003 R08: 0000000000000000 R09: 00007ffdc76ec4b0
R10: 00007ffdc76ec367 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000008 R14: 0000000000000000 R15: 000055b534b8a2a0
Modules linked in: cpufreq_userspace cpufreq_powersave cpufreq_ondemand cpufreq_conservative autofs4 fan nfsd auth_rpcgss nfs lockd grace fscache sunrpc bridge stp llc nhpoly1305_sse2 nhpoly1305 aes_generic chacha_x86_64 chacha_generic adiantum poly1305_generic vhost_net tun vhost tap dm_crypt snd_hda_codec_analog snd_hda_codec_generic usb_storage snd_hda_intel kvm_intel snd_hda_codec kvm snd_hwdep snd_hda_core snd_pcm dcdbas snd_timer irqbypass snd soundcore sr_mod cdrom tg3 sg floppy evdev xfs dm_mod raid1 md_mod psmouse
CR2: 0000000000000036
---[ end trace bc12bbe4cdd6319f ]---

...
NFS: Registering the id_resolver key type
Key type id_resolver registered
Key type id_legacy registered

Quoted 10 lines

My kernel config is at
http://audible.transient.net/~jamie/k/upcallv2.config-5.3.0-rc2-00034-g6ee95d1c8991

I don't think there's anything terribly interesting about my nfs server setup, this happens reliably on boot up, idle network, no active clients; let me know what else you need, happy to debug.

Signature · 1 line
Jamie Heilman                     http://audible.transient.net/~jamie/

Please try this patch (v2 because I messed up the first one).

-Scott

From 34ae6455abfd81b47ab34b66ca88a29ff33c7d98 Mon Sep 17 00:00:00 2001
From: Scott Mayhew <[email protected]>
Date: Tue, 12 Nov 2019 10:10:00 -0500
Subject: [PATCH v2] nfsd: Fix cld_net->cn_tfm initialization

Don't assign an error pointer to cn->cn_tfm, otherwise an oops will occur in nfsd4_remove_cld_pipe().

Fixes 6ee95d1c8991 ("nfsd: add support for upcall version 2")
---
 fs/nfsd/nfs4recover.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
fs/nfsd/nfs4recover.c +5 -3
index cdc75ad4438b..d1bc56b2e861 100644
@@ -1578,6 +1578,7 @@ nfsd4_cld_tracking_init(struct net *net)
 	struct nfsd_net *nn = net_generic(net, nfsd_net_id);
 	bool running;
 	int retries = 10;
+	struct crypto_shash *tfm;
 
 	status = nfs4_cld_state_init(net);
 	if (status)
@@ -1586,11 +1587,12 @@ nfsd4_cld_tracking_init(struct net *net)
 	status = __nfsd4_init_cld_pipe(net);
 	if (status)
 		goto err_shutdown;
-	nn->cld_net->cn_tfm = crypto_alloc_shash("sha256", 0, 0);
-	if (IS_ERR(nn->cld_net->cn_tfm)) {
-		status = PTR_ERR(nn->cld_net->cn_tfm);
+	tfm = crypto_alloc_shash("sha256", 0, 0);
+	if (IS_ERR(tfm)) {
+		status = PTR_ERR(tfm);
 		goto err_remove;
 	}
+	nn->cld_net->cn_tfm = tfm;
 
 	/*
 	 * rpc pipe upcalls take 30 seconds to time out, so we don't want to
Signature · 1 line
2.17.2
#3 lore raw
Quoted 67 lines · Scott Mayhew

Hi Jamie,

Quoted 63 lines · Jamie Heilman

Giving 5.4.0-rc7 a spin I hit a NULL pointer dereference and bisected it to:

commit 6ee95d1c899186c0798cafd25998d436bcdb9618

Author: Scott Mayhew <[email protected]>
Date:   Mon Sep 9 16:10:31 2019 -0400

nfsd: add support for upcall version 2

The splat against 5.3.0-rc2-00034-g6ee95d1c8991:

BUG: kernel NULL pointer dereference, address: 0000000000000036
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0 
Oops: 0000 [#1] PREEMPT SMP PTI
CPU: 0 PID: 2936 Comm: rpc.nfsd Not tainted 5.3.0-rc2-00034-g6ee95d1c8991 #1
Hardware name: Dell Inc. Precision WorkStation T3400  /0TP412, BIOS A14 04/30/2012
RIP: 0010:crypto_destroy_tfm+0x5/0x4d
Code: 78 01 00 00 48 85 c0 74 05 e9 05 05 66 00 c3 55 48 8b af 80 01 00 00 e8 d5 ff ff ff 48 89 ef 5d e9 12 f9 ef ff 48 85 ff 74 47 <48> 83 7e 30 00 41 55 4c 8b 6e 38 41 54 49 89 fc 55 48 89 f5 75 14
RSP: 0018:ffffc90000b7bd68 EFLAGS: 00010282
RAX: ffffffffa0402841 RBX: ffff888230484400 RCX: 0000000000002cd0
RDX: 0000000000002cce RSI: 0000000000000006 RDI: fffffffffffffffe
RBP: ffffffff81e68440 R08: ffff888232801800 R09: ffffffffa0402841
R10: 0000000000000200 R11: ffff88823048ae40 R12: ffff888231585100
R13: ffff88823048ae40 R14: 000000000000000b R15: ffff888230484400
FS:  00007f02102c3740(0000) GS:ffff888233a00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000036 CR3: 0000000230f94000 CR4: 00000000000406f0
Call Trace:
 nfsd4_remove_cld_pipe+0x6d/0x83 [nfsd]
 nfsd4_cld_tracking_init+0x1cf/0x295 [nfsd]
 nfsd4_client_tracking_init+0x72/0x13e [nfsd]
 nfs4_state_start_net+0x22a/0x2cf [nfsd]
 nfsd_svc+0x1c6/0x292 [nfsd]
 write_threads+0x68/0xb0 [nfsd]
 ? write_versions+0x333/0x333 [nfsd]
 nfsctl_transaction_write+0x4a/0x62 [nfsd]
 vfs_write+0xa0/0xdd
 ksys_write+0x71/0xba
 do_syscall_64+0x48/0x55
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x7f021056c904
Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb bb 0f 1f 80 00 00 00 00 48 8d 05 d9 3a 0d 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 48 83 ec 28 48 89 54 24 18 48
RSP: 002b:00007ffdc76ec618 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 000055b534955560 RCX: 00007f021056c904
RDX: 0000000000000002 RSI: 000055b534955560 RDI: 0000000000000003
RBP: 0000000000000003 R08: 0000000000000000 R09: 00007ffdc76ec4b0
R10: 00007ffdc76ec367 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000008 R14: 0000000000000000 R15: 000055b534b8a2a0
Modules linked in: cpufreq_userspace cpufreq_powersave cpufreq_ondemand cpufreq_conservative autofs4 fan nfsd auth_rpcgss nfs lockd grace fscache sunrpc bridge stp llc nhpoly1305_sse2 nhpoly1305 aes_generic chacha_x86_64 chacha_generic adiantum poly1305_generic vhost_net tun vhost tap dm_crypt snd_hda_codec_analog snd_hda_codec_generic usb_storage snd_hda_intel kvm_intel snd_hda_codec kvm snd_hwdep snd_hda_core snd_pcm dcdbas snd_timer irqbypass snd soundcore sr_mod cdrom tg3 sg floppy evdev xfs dm_mod raid1 md_mod psmouse
CR2: 0000000000000036
---[ end trace bc12bbe4cdd6319f ]---

...
NFS: Registering the id_resolver key type
Key type id_resolver registered
Key type id_legacy registered

My kernel config is at
http://audible.transient.net/~jamie/k/upcallv2.config-5.3.0-rc2-00034-g6ee95d1c8991

Quoted 8 lines
Quoted 7 lines

I don't think there's anything terribly interesting about my nfs server setup, this happens reliably on boot up, idle network, no active clients; let me know what else you need, happy to debug.

Signature · 1 line
Jamie Heilman                     http://audible.transient.net/~jamie/

Please try this patch (v2 because I messed up the first one).

Yep, that seems to solve it. Is the implication that CONFIG_CRYPTO_SHA256 should be selected by nfsd? (I tested with it unset, as per my config before.)

Signature · 1 line
Jamie Heilman                     http://audible.transient.net/~jamie/
#4 lore raw
Quoted 72 lines · Jamie Heilman
Quoted 71 lines · Scott Mayhew

Hi Jamie,

Quoted 67 lines · Jamie Heilman

Giving 5.4.0-rc7 a spin I hit a NULL pointer dereference and bisected it to:

commit 6ee95d1c899186c0798cafd25998d436bcdb9618

Author: Scott Mayhew <[email protected]>
Date:   Mon Sep 9 16:10:31 2019 -0400

nfsd: add support for upcall version 2

The splat against 5.3.0-rc2-00034-g6ee95d1c8991:

BUG: kernel NULL pointer dereference, address: 0000000000000036
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0 
Oops: 0000 [#1] PREEMPT SMP PTI
CPU: 0 PID: 2936 Comm: rpc.nfsd Not tainted 5.3.0-rc2-00034-g6ee95d1c8991 #1
Hardware name: Dell Inc. Precision WorkStation T3400  /0TP412, BIOS A14 04/30/2012
RIP: 0010:crypto_destroy_tfm+0x5/0x4d
Code: 78 01 00 00 48 85 c0 74 05 e9 05 05 66 00 c3 55 48 8b af 80 01 00 00 e8 d5 ff ff ff 48 89 ef 5d e9 12 f9 ef ff 48 85 ff 74 47 <48> 83 7e 30 00 41 55 4c 8b 6e 38 41 54 49 89 fc 55 48 89 f5 75 14
RSP: 0018:ffffc90000b7bd68 EFLAGS: 00010282
RAX: ffffffffa0402841 RBX: ffff888230484400 RCX: 0000000000002cd0
RDX: 0000000000002cce RSI: 0000000000000006 RDI: fffffffffffffffe
RBP: ffffffff81e68440 R08: ffff888232801800 R09: ffffffffa0402841
R10: 0000000000000200 R11: ffff88823048ae40 R12: ffff888231585100
R13: ffff88823048ae40 R14: 000000000000000b R15: ffff888230484400
FS:  00007f02102c3740(0000) GS:ffff888233a00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000036 CR3: 0000000230f94000 CR4: 00000000000406f0
Call Trace:
 nfsd4_remove_cld_pipe+0x6d/0x83 [nfsd]
 nfsd4_cld_tracking_init+0x1cf/0x295 [nfsd]
 nfsd4_client_tracking_init+0x72/0x13e [nfsd]
 nfs4_state_start_net+0x22a/0x2cf [nfsd]
 nfsd_svc+0x1c6/0x292 [nfsd]
 write_threads+0x68/0xb0 [nfsd]
 ? write_versions+0x333/0x333 [nfsd]
 nfsctl_transaction_write+0x4a/0x62 [nfsd]
 vfs_write+0xa0/0xdd
 ksys_write+0x71/0xba
 do_syscall_64+0x48/0x55
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x7f021056c904
Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb bb 0f 1f 80 00 00 00 00 48 8d 05 d9 3a 0d 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 48 83 ec 28 48 89 54 24 18 48
RSP: 002b:00007ffdc76ec618 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 000055b534955560 RCX: 00007f021056c904
RDX: 0000000000000002 RSI: 000055b534955560 RDI: 0000000000000003
RBP: 0000000000000003 R08: 0000000000000000 R09: 00007ffdc76ec4b0
R10: 00007ffdc76ec367 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000008 R14: 0000000000000000 R15: 000055b534b8a2a0
Modules linked in: cpufreq_userspace cpufreq_powersave cpufreq_ondemand cpufreq_conservative autofs4 fan nfsd auth_rpcgss nfs lockd grace fscache sunrpc bridge stp llc nhpoly1305_sse2 nhpoly1305 aes_generic chacha_x86_64 chacha_generic adiantum poly1305_generic vhost_net tun vhost tap dm_crypt snd_hda_codec_analog snd_hda_codec_generic usb_storage snd_hda_intel kvm_intel snd_hda_codec kvm snd_hwdep snd_hda_core snd_pcm dcdbas snd_timer irqbypass snd soundcore sr_mod cdrom tg3 sg floppy evdev xfs dm_mod raid1 md_mod psmouse
CR2: 0000000000000036
---[ end trace bc12bbe4cdd6319f ]---

...
NFS: Registering the id_resolver key type
Key type id_resolver registered
Key type id_legacy registered

My kernel config is at
http://audible.transient.net/~jamie/k/upcallv2.config-5.3.0-rc2-00034-g6ee95d1c8991

I don't think there's anything terribly interesting about my nfs server setup, this happens reliably on boot up, idle network, no active clients; let me know what else you need, happy to debug.

Quoted 7 lines
Quoted 4 lines
Quoted 3 lines
Signature · 1 line
Jamie Heilman                     http://audible.transient.net/~jamie/

Please try this patch (v2 because I messed up the first one).

Yep, that seems to solve it.

Thanks! There's another small problem, in that nfsd could incorrectly fall back to using the old nfsdcld tracking ops even if nfsdcld isn't running... so a v3 patch is forthcoming.

Quoted 3 lines

Is the implication that
CONFIG_CRYPTO_SHA256 should be selected by nfsd? (I tested with it unset, as per my config before.)

Yes - I'll send a separate patch for that.

-Scott

Quoted 5 lines
Signature · 2 lines
-- 
Jamie Heilman                     http://audible.transient.net/~jamie/