Skip to contentSkip to conversation

general protection fault, probably for non-canonical address in nfsd

DISCUSSION Hans-Peter Jansen 3 participants 7 messages Jun 7, 2020 – Jun 8, 2020

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Hi,

after upgrading the kernel from 5.6.11 to 5.6.14, we suffer from regular crashes of nfsd here:

2020-06-07T01:32:43.600306+02:00 server rpc.mountd[2664]: authenticated mount request from 192.168.3.16:303 for /work (/work)
2020-06-07T01:32:43.602594+02:00 server rpc.mountd[2664]: authenticated mount request from 192.168.3.16:304 for /work/vmware (/work)
2020-06-07T01:32:43.602971+02:00 server rpc.mountd[2664]: authenticated mount request from 192.168.3.16:305 for /work/vSphere (/work)
2020-06-07T01:32:43.606276+02:00 server kernel: [51901.089211] general protection fault, probably for non-canonical address 0xb9159d506ba40000: 0000 [#1] SMP PTI
2020-06-07T01:32:43.606284+02:00 server kernel: [51901.089226] CPU: 1 PID: 3190 Comm: nfsd Tainted: G           O      5.6.14-lp151.2-default #1 openSUSE Tumbleweed (unreleased)
2020-06-07T01:32:43.606286+02:00 server kernel: [51901.089234] Hardware name: System manufacturer System Product Name/P7F-E, BIOS 0906    09/20/2010
2020-06-07T01:32:43.606287+02:00 server kernel: [51901.089247] RIP: 0010:cgroup_sk_free+0x26/0x80
2020-06-07T01:32:43.606288+02:00 server kernel: [51901.089257] Code: 00 00 00 00 66 66 66 66 90 53 48 8b 07 48 c7 c3 30 72 07 b6 a8 01 75 07 48 85 c0 48 0f 45 d8 48 8b 83 18 09 00 00 a8 03 
75 1a <65> 48 ff 08 f6 43 7c 01 74 02 5b c3 48 8b 43 18 a8 03 75 26 65 48
2020-06-07T01:32:43.606290+02:00 server kernel: [51901.089276] RSP: 0018:ffffb248c21e7e10 EFLAGS: 00010246
2020-06-07T01:32:43.606291+02:00 server kernel: [51901.089280] RAX: b91603a504000000 RBX: ffff99ab141a0000 RCX: 0000000000000021
2020-06-07T01:32:43.606292+02:00 server kernel: [51901.089284] RDX: ffffffffb6135ec4 RSI: 0000000000010080 RDI: ffff99a7159c1490
2020-06-07T01:32:43.606293+02:00 server kernel: [51901.089287] RBP: ffff99a7159c1200 R08: ffff99ab67a60c60 R09: 000000000002eb00
2020-06-07T01:32:43.606294+02:00 server kernel: [51901.089291] R10: ffffb248c0087dc0 R11: 00000000000000c6 R12: 0000000000000000
2020-06-07T01:32:43.606295+02:00 server kernel: [51901.089294] R13: 0000000000000103 R14: ffff99aae4934238 R15: ffff99ab31902000
2020-06-07T01:32:43.606296+02:00 server kernel: [51901.089299] FS:  0000000000000000(0000) GS:ffff99ab67a40000(0000) knlGS:0000000000000000
2020-06-07T01:32:43.606297+02:00 server kernel: [51901.089303] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
2020-06-07T01:32:43.606303+02:00 server kernel: [51901.089305] CR2: 00000000008e0000 CR3: 00000004df60a000 CR4: 00000000000026e0
2020-06-07T01:32:43.606304+02:00 server kernel: [51901.089307] Call Trace:
2020-06-07T01:32:43.606305+02:00 server kernel: [51901.089315]  __sk_destruct+0x10d/0x1d0
2020-06-07T01:32:43.606306+02:00 server kernel: [51901.089319]  inet_release+0x34/0x60
2020-06-07T01:32:43.606307+02:00 server kernel: [51901.089325]  __sock_release+0x81/0xb0
2020-06-07T01:32:43.606308+02:00 server kernel: [51901.089358]  svc_sock_free+0x38/0x60 [sunrpc]
2020-06-07T01:32:43.606308+02:00 server kernel: [51901.089374]  svc_xprt_put+0x99/0xe0 [sunrpc]
2020-06-07T01:32:43.606310+02:00 server kernel: [51901.089389]  svc_recv+0x9c0/0xa40 [sunrpc]
2020-06-07T01:32:43.606310+02:00 server kernel: [51901.089410]  ? nfsd_destroy+0x60/0x60 [nfsd]
2020-06-07T01:32:43.606311+02:00 server kernel: [51901.089417]  nfsd+0xd1/0x150 [nfsd]
2020-06-07T01:32:43.606312+02:00 server kernel: [51901.089420]  kthread+0x10d/0x130
2020-06-07T01:32:43.606313+02:00 server kernel: [51901.089423]  ? kthread_park+0x90/0x90
2020-06-07T01:32:43.606314+02:00 server kernel: [51901.089426]  ret_from_fork+0x35/0x40

A vSphere 5.5 host accesses this linux server with nfs v3 for backup purposes (a Veeam backup server want to store a new backup here).

The kernel is tainted due to vboxdrv. The OS is openSUSE Leap 15.1, with the kernel and Virtualbox replaced with uptodate versions from proper rpm packages (built on that very vSphere host in a OBS server VM..).

I used to be subscribed to this ML, but that subscription has been lost 04/09, thus I cannot reply properly to the general prot. fault thread, started 05/12 from syzbot with Bruce looking into it.

It seems somewhat related.

Interestingly, we're using a couple of NFS v4 mounts for subsets of home here, and mount /work and other shares from various Tumbleweed systems with NFS v4 here without any undesired effects.

Since the kernel upgrade, every time, this Veeam thing triggers these v3 mounts, the crash happens. I've disabled this backup target for now until the problem is resolved, because it effectively prevents further nfs accesses to this server, and blocks our desktops until the server is rebooted.

A cursory look into 5.6.{15,16} changelogs seems to imply, that this issue is still pending.

Let me know, if I can provide any further info's.

Thanks,
Pete

#1 lore raw
Quoted 82 lines · CDT Hans-Peter Jansen

Hi,

after upgrading the kernel from 5.6.11 to 5.6.14, we suffer from regular crashes of nfsd here:

2020-06-07T01:32:43.600306+02:00 server rpc.mountd[2664]: authenticated
mount request from 192.168.3.16:303 for /work (/work)
2020-06-07T01:32:43.602594+02:00 server rpc.mountd[2664]: authenticated
mount request from 192.168.3.16:304 for /work/vmware (/work)
2020-06-07T01:32:43.602971+02:00 server rpc.mountd[2664]: authenticated
mount request from 192.168.3.16:305 for /work/vSphere (/work)
2020-06-07T01:32:43.606276+02:00 server kernel: [51901.089211] general
protection fault, probably for non-canonical address 0xb9159d506ba40000:
0000 [#1] SMP PTI 2020-06-07T01:32:43.606284+02:00 server kernel:
[51901.089226] CPU: 1 PID: 3190 Comm: nfsd Tainted: G           O     
5.6.14-lp151.2-default #1 openSUSE Tumbleweed (unreleased)
2020-06-07T01:32:43.606286+02:00 server kernel: [51901.089234] Hardware
name: System manufacturer System Product Name/P7F-E, BIOS 0906   
09/20/2010 2020-06-07T01:32:43.606287+02:00 server kernel: [51901.089247]
RIP: 0010:cgroup_sk_free+0x26/0x80 2020-06-07T01:32:43.606288+02:00 server
kernel: [51901.089257] Code: 00 00 00 00 66 66 66 66 90 53 48 8b 07 48 c7
c3 30 72 07 b6 a8 01 75 07 48 85 c0 48 0f 45 d8 48 8b 83 18 09 00 00 a8 03
75 1a <65> 48 ff 08 f6 43 7c 01 74 02 5b c3 48 8b 43 18 a8 03 75 26 65 48
2020-06-07T01:32:43.606290+02:00 server kernel: [51901.089276] RSP:
0018:ffffb248c21e7e10 EFLAGS: 00010246 2020-06-07T01:32:43.606291+02:00
server kernel: [51901.089280] RAX: b91603a504000000 RBX: ffff99ab141a0000
RCX: 0000000000000021 2020-06-07T01:32:43.606292+02:00 server kernel:
[51901.089284] RDX: ffffffffb6135ec4 RSI: 0000000000010080 RDI:
ffff99a7159c1490 2020-06-07T01:32:43.606293+02:00 server kernel:
[51901.089287] RBP: ffff99a7159c1200 R08: ffff99ab67a60c60 R09:
000000000002eb00 2020-06-07T01:32:43.606294+02:00 server kernel:
[51901.089291] R10: ffffb248c0087dc0 R11: 00000000000000c6 R12:
0000000000000000 2020-06-07T01:32:43.606295+02:00 server kernel:
[51901.089294] R13: 0000000000000103 R14: ffff99aae4934238 R15:
ffff99ab31902000 2020-06-07T01:32:43.606296+02:00 server kernel:
[51901.089299] FS:  0000000000000000(0000) GS:ffff99ab67a40000(0000)
knlGS:0000000000000000 2020-06-07T01:32:43.606297+02:00 server kernel:
[51901.089303] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
2020-06-07T01:32:43.606303+02:00 server kernel: [51901.089305] CR2:
00000000008e0000 CR3: 00000004df60a000 CR4: 00000000000026e0
2020-06-07T01:32:43.606304+02:00 server kernel: [51901.089307] Call Trace:
2020-06-07T01:32:43.606305+02:00 server kernel: [51901.089315] 
__sk_destruct+0x10d/0x1d0 2020-06-07T01:32:43.606306+02:00 server kernel:
[51901.089319]  inet_release+0x34/0x60 2020-06-07T01:32:43.606307+02:00
server kernel: [51901.089325]  __sock_release+0x81/0xb0
2020-06-07T01:32:43.606308+02:00 server kernel: [51901.089358] 
svc_sock_free+0x38/0x60 [sunrpc] 2020-06-07T01:32:43.606308+02:00 server
kernel: [51901.089374]  svc_xprt_put+0x99/0xe0 [sunrpc]
2020-06-07T01:32:43.606310+02:00 server kernel: [51901.089389] 
svc_recv+0x9c0/0xa40 [sunrpc] 2020-06-07T01:32:43.606310+02:00 server
kernel: [51901.089410]  ? nfsd_destroy+0x60/0x60 [nfsd]
2020-06-07T01:32:43.606311+02:00 server kernel: [51901.089417] 
nfsd+0xd1/0x150 [nfsd] 2020-06-07T01:32:43.606312+02:00 server kernel:
[51901.089420]  kthread+0x10d/0x130 2020-06-07T01:32:43.606313+02:00 server
kernel: [51901.089423]  ? kthread_park+0x90/0x90
2020-06-07T01:32:43.606314+02:00 server kernel: [51901.089426] 

ret_from_fork+0x35/0x40

A vSphere 5.5 host accesses this linux server with nfs v3 for backup purposes (a Veeam backup server want to store a new backup here).

The kernel is tainted due to vboxdrv. The OS is openSUSE Leap 15.1, with the kernel and Virtualbox replaced with uptodate versions from proper rpm packages (built on that very vSphere host in a OBS server VM..).

I used to be subscribed to this ML, but that subscription has been lost 04/09, thus I cannot reply properly to the general prot. fault thread, started 05/12 from syzbot with Bruce looking into it.

It seems somewhat related.

Interestingly, we're using a couple of NFS v4 mounts for subsets of home here, and mount /work and other shares from various Tumbleweed systems with NFS v4 here without any undesired effects.

Since the kernel upgrade, every time, this Veeam thing triggers these v3 mounts, the crash happens. I've disabled this backup target for now until the problem is resolved, because it effectively prevents further nfs accesses to this server, and blocks our desktops until the server is rebooted.

Quoted 7 lines

A cursory look into 5.6.{15,16} changelogs seems to imply, that this issue is still pending.

Let me know, if I can provide any further info's.

Thanks,
Pete

I see similar issues in Fedora kernels 5.6.14 through 5.6.16
https://bugzilla.redhat.com/show_bug.cgi?id=1839287

On the client I mount /home with sec=krb5p, and /mnt/koji with sec=krb5

Signature · 2 lines
Anthony - https://messinet.com
F9B6 560E 68EA 037D 8C3D  D1C9 FF31 3BDB D9D8 99B6
#2 lore raw

Am Sonntag, 7. Juni 2020, 18:01:55 CEST schrieb Anthony Joseph Messina:

Quoted 20 lines · CDT Hans-Peter Jansen
Quoted 17 lines

Hi,

after upgrading the kernel from 5.6.11 to 5.6.14, we suffer from regular crashes of nfsd here:

2020-06-07T01:32:43.600306+02:00 server rpc.mountd[2664]: authenticated
mount request from 192.168.3.16:303 for /work (/work)
2020-06-07T01:32:43.602594+02:00 server rpc.mountd[2664]: authenticated
mount request from 192.168.3.16:304 for /work/vmware (/work)
2020-06-07T01:32:43.602971+02:00 server rpc.mountd[2664]: authenticated
mount request from 192.168.3.16:305 for /work/vSphere (/work)
2020-06-07T01:32:43.606276+02:00 server kernel: [51901.089211] general
protection fault, probably for non-canonical address 0xb9159d506ba40000:
0000 [#1] SMP PTI 2020-06-07T01:32:43.606284+02:00 server kernel:
[51901.089226] CPU: 1 PID: 3190 Comm: nfsd Tainted: G           O
5.6.14-lp151.2-default #1 openSUSE Tumbleweed (unreleased)
2020-06-07T01:32:43.606286+02:00 server kernel: [51901.089234] Hardware
Quoted 1 line

name: System manufacturer System Product Name/P7F-E, BIOS 0906

Quoted 4 lines

I see similar issues in Fedora kernels 5.6.14 through 5.6.16
https://bugzilla.redhat.com/show_bug.cgi?id=1839287

On the client I mount /home with sec=krb5p, and /mnt/koji with sec=krb5

Thanks for confirmation.

Apart from the hassle with server reboots, this issue has some DOS potential, I'm afraid.

Cheers,
Pete

#3 lore raw
Quoted 28 lines · Hans-Peter Jansen

Am Sonntag, 7. Juni 2020, 18:01:55 CEST schrieb Anthony Joseph Messina:

Quoted 23 lines · CDT Hans-Peter Jansen
Quoted 17 lines

Hi,

after upgrading the kernel from 5.6.11 to 5.6.14, we suffer from regular crashes of nfsd here:

2020-06-07T01:32:43.600306+02:00 server rpc.mountd[2664]: authenticated
mount request from 192.168.3.16:303 for /work (/work)
2020-06-07T01:32:43.602594+02:00 server rpc.mountd[2664]: authenticated
mount request from 192.168.3.16:304 for /work/vmware (/work)
2020-06-07T01:32:43.602971+02:00 server rpc.mountd[2664]: authenticated
mount request from 192.168.3.16:305 for /work/vSphere (/work)
2020-06-07T01:32:43.606276+02:00 server kernel: [51901.089211] general
protection fault, probably for non-canonical address 0xb9159d506ba40000:
0000 [#1] SMP PTI 2020-06-07T01:32:43.606284+02:00 server kernel:
[51901.089226] CPU: 1 PID: 3190 Comm: nfsd Tainted: G           O
5.6.14-lp151.2-default #1 openSUSE Tumbleweed (unreleased)
2020-06-07T01:32:43.606286+02:00 server kernel: [51901.089234] Hardware
Quoted 1 line

name: System manufacturer System Product Name/P7F-E, BIOS 0906

I see similar issues in Fedora kernels 5.6.14 through 5.6.16
https://bugzilla.redhat.com/show_bug.cgi?id=1839287

Quoted 6 lines
Quoted 1 line

On the client I mount /home with sec=krb5p, and /mnt/koji with sec=krb5

Thanks for confirmation.

Apart from the hassle with server reboots, this issue has some DOS potential, I'm afraid.

If you have a reproducer (even a partial one) then bisecting between a known good kernel and v5.6.14 (or 16) would be helpful.

--
Chuck Lever

#4 lore raw

Am Montag, 8. Juni 2020, 17:28:53 CEST schrieb Chuck Lever:

Quoted 28 lines
Quoted 28 lines · Hans-Peter Jansen

Am Sonntag, 7. Juni 2020, 18:01:55 CEST schrieb Anthony Joseph Messina:

Quoted 24 lines · CDT Hans-Peter Jansen
Quoted 17 lines

Hi,

after upgrading the kernel from 5.6.11 to 5.6.14, we suffer from regular crashes of nfsd here:

2020-06-07T01:32:43.600306+02:00 server rpc.mountd[2664]: authenticated
mount request from 192.168.3.16:303 for /work (/work)
2020-06-07T01:32:43.602594+02:00 server rpc.mountd[2664]: authenticated
mount request from 192.168.3.16:304 for /work/vmware (/work)
2020-06-07T01:32:43.602971+02:00 server rpc.mountd[2664]: authenticated
mount request from 192.168.3.16:305 for /work/vSphere (/work)
2020-06-07T01:32:43.606276+02:00 server kernel: [51901.089211] general
protection fault, probably for non-canonical address 0xb9159d506ba40000:
0000 [#1] SMP PTI 2020-06-07T01:32:43.606284+02:00 server kernel:
[51901.089226] CPU: 1 PID: 3190 Comm: nfsd Tainted: G           O
5.6.14-lp151.2-default #1 openSUSE Tumbleweed (unreleased)
2020-06-07T01:32:43.606286+02:00 server kernel: [51901.089234] Hardware
Quoted 1 line

name: System manufacturer System Product Name/P7F-E, BIOS 0906

I see similar issues in Fedora kernels 5.6.14 through 5.6.16
https://bugzilla.redhat.com/show_bug.cgi?id=1839287

On the client I mount /home with sec=krb5p, and /mnt/koji with sec=krb5

Quoted 7 lines
Quoted 4 lines

Thanks for confirmation.

Apart from the hassle with server reboots, this issue has some DOS potential, I'm afraid.

If you have a reproducer (even a partial one) then bisecting between a known good kernel and v5.6.14 (or 16) would be helpful.

I would love to bisect, but this is my primary production machine, that needs to be up as much as possible. Apart from that, I'm about to leave the site for a week and been severely time constrained for the next couple of weeks..

Sorry.

Anthony?
--
Pete

#5 lore raw
Quoted 43 lines · CDT Hans-Peter Jansen

Am Montag, 8. Juni 2020, 17:28:53 CEST schrieb Chuck Lever:

Quoted 34 lines
Quoted 34 lines · Hans-Peter Jansen

Am Sonntag, 7. Juni 2020, 18:01:55 CEST schrieb Anthony Joseph Messina:

Quoted 30 lines · CDT Hans-Peter Jansen
Quoted 22 lines

Hi,

after upgrading the kernel from 5.6.11 to 5.6.14, we suffer from regular
crashes of nfsd here:

2020-06-07T01:32:43.600306+02:00 server rpc.mountd[2664]:
authenticated
mount request from 192.168.3.16:303 for /work (/work)
2020-06-07T01:32:43.602594+02:00 server rpc.mountd[2664]:
authenticated
mount request from 192.168.3.16:304 for /work/vmware (/work)
2020-06-07T01:32:43.602971+02:00 server rpc.mountd[2664]:
authenticated
mount request from 192.168.3.16:305 for /work/vSphere (/work)
2020-06-07T01:32:43.606276+02:00 server kernel: [51901.089211] general
protection fault, probably for non-canonical address
0xb9159d506ba40000:
0000 [#1] SMP PTI 2020-06-07T01:32:43.606284+02:00 server kernel:
[51901.089226] CPU: 1 PID: 3190 Comm: nfsd Tainted: G           O
5.6.14-lp151.2-default #1 openSUSE Tumbleweed (unreleased)
2020-06-07T01:32:43.606286+02:00 server kernel: [51901.089234]
Quoted 2 lines

Hardware
name: System manufacturer System Product Name/P7F-E, BIOS 0906

I see similar issues in Fedora kernels 5.6.14 through 5.6.16
https://bugzilla.redhat.com/show_bug.cgi?id=1839287

On the client I mount /home with sec=krb5p, and /mnt/koji with sec=krb5

Quoted 7 lines
Quoted 4 lines

Thanks for confirmation.

Apart from the hassle with server reboots, this issue has some DOS potential, I'm afraid.

If you have a reproducer (even a partial one) then bisecting between a known good kernel and v5.6.14 (or 16) would be helpful.

Quoted 8 lines

I would love to bisect, but this is my primary production machine, that needs to be up as much as possible. Apart from that, I'm about to leave the site for a week and been severely time constrained for the next couple of weeks..

Sorry.

Anthony?

Unfortunately, this is also my main workstation and I have no experience building custom kernels. The diff in net/sunrpc between v5.6.13 and v5.6.14 is relatively small, though it may not point to the root issue. I'm typically only able to "follow along" code like this to spot issues, being a nurse, not a kernel programmer.

Thank you for your help. -A

Signature · 2 lines
Anthony - https://messinet.com
F9B6 560E 68EA 037D 8C3D  D1C9 FF31 3BDB D9D8 99B6
 net/sunrpc/auth_gss/auth_gss.c        | 12 ++++------
 net/sunrpc/auth_gss/gss_krb5_crypto.c |  8 +++----
 net/sunrpc/auth_gss/gss_krb5_wrap.c   | 44 +++++++++++++++++++++++------------
 net/sunrpc/auth_gss/gss_mech_switch.c |  3 ++-
 net/sunrpc/auth_gss/svcauth_gss.c     | 10 +++-----
 net/sunrpc/clnt.c                     |  5 ++++
 net/sunrpc/xdr.c                      | 41 ++++++++++++++++++++++++++++++++
 net/sunrpc/xprtrdma/backchannel.c     |  2 +-
 net/sunrpc/xprtrdma/frwr_ops.c        | 14 +++++++----
 net/sunrpc/xprtrdma/transport.c       |  2 +-
 net/sunrpc/xprtrdma/verbs.c           | 15 +++++-------
 net/sunrpc/xprtrdma/xprt_rdma.h       |  5 ++--
 12 files changed, 108 insertions(+), 53 deletions(-)
net/sunrpc/auth_gss/auth_gss.c +5 -7
index 2dc740acb3bf..a7ad150fd4ee 100644
@@ -2030,7 +2030,6 @@ gss_unwrap_resp_priv(struct rpc_task *task, struct rpc_cred *cred,
 	struct xdr_buf *rcv_buf = &rqstp->rq_rcv_buf;
 	struct kvec *head = rqstp->rq_rcv_buf.head;
 	struct rpc_auth *auth = cred->cr_auth;
-	unsigned int savedlen = rcv_buf->len;
 	u32 offset, opaque_len, maj_stat;
 	__be32 *p;
 
@@ -2041,9 +2040,9 @@ gss_unwrap_resp_priv(struct rpc_task *task, struct rpc_cred *cred,
 	offset = (u8 *)(p) - (u8 *)head->iov_base;
 	if (offset + opaque_len > rcv_buf->len)
 		goto unwrap_failed;
-	rcv_buf->len = offset + opaque_len;
 
-	maj_stat = gss_unwrap(ctx->gc_gss_ctx, offset, rcv_buf);
+	maj_stat = gss_unwrap(ctx->gc_gss_ctx, offset,
+			      offset + opaque_len, rcv_buf);
 	if (maj_stat == GSS_S_CONTEXT_EXPIRED)
 		clear_bit(RPCAUTH_CRED_UPTODATE, &cred->cr_flags);
 	if (maj_stat != GSS_S_COMPLETE)
@@ -2057,10 +2056,9 @@ gss_unwrap_resp_priv(struct rpc_task *task, struct rpc_cred *cred,
 	 */
 	xdr_init_decode(xdr, rcv_buf, p, rqstp);
 
-	auth->au_rslack = auth->au_verfsize + 2 +
-			  XDR_QUADLEN(savedlen - rcv_buf->len);
-	auth->au_ralign = auth->au_verfsize + 2 +
-			  XDR_QUADLEN(savedlen - rcv_buf->len);
+	auth->au_rslack = auth->au_verfsize + 2 + ctx->gc_gss_ctx->slack;
+	auth->au_ralign = auth->au_verfsize + 2 + ctx->gc_gss_ctx->align;
+
 	return 0;
 unwrap_failed:
 	trace_rpcgss_unwrap_failed(task);
net/sunrpc/auth_gss/gss_krb5_crypto.c +4 -4
index 6f2d30d7b766..e7180da1fc6a 100644
@@ -851,8 +851,8 @@ gss_krb5_aes_encrypt(struct krb5_ctx *kctx, u32 offset,
 }
 
 u32
-gss_krb5_aes_decrypt(struct krb5_ctx *kctx, u32 offset, struct xdr_buf *buf,
-		     u32 *headskip, u32 *tailskip)
+gss_krb5_aes_decrypt(struct krb5_ctx *kctx, u32 offset, u32 len,
+		     struct xdr_buf *buf, u32 *headskip, u32 *tailskip)
 {
 	struct xdr_buf subbuf;
 	u32 ret = 0;
@@ -881,7 +881,7 @@ gss_krb5_aes_decrypt(struct krb5_ctx *kctx, u32 offset, struct xdr_buf *buf,
 
 	/* create a segment skipping the header and leaving out the checksum */
 	xdr_buf_subsegment(buf, &subbuf, offset + GSS_KRB5_TOK_HDR_LEN,
-				    (buf->len - offset - GSS_KRB5_TOK_HDR_LEN -
+				    (len - offset - GSS_KRB5_TOK_HDR_LEN -
 				     kctx->gk5e->cksumlength));
 
 	nblocks = (subbuf.len + blocksize - 1) / blocksize;
@@ -926,7 +926,7 @@ gss_krb5_aes_decrypt(struct krb5_ctx *kctx, u32 offset, struct xdr_buf *buf,
 		goto out_err;
 
 	/* Get the packet's hmac value */
-	ret = read_bytes_from_xdr_buf(buf, buf->len - kctx->gk5e->cksumlength,
+	ret = read_bytes_from_xdr_buf(buf, len - kctx->gk5e->cksumlength,
 				      pkt_hmac, kctx->gk5e->cksumlength);
 	if (ret)
 		goto out_err;
net/sunrpc/auth_gss/gss_krb5_wrap.c +29 -15

120 lines not on this page · raw message

net/sunrpc/auth_gss/gss_mech_switch.c +2 -1
index db550bfc2642..69316ab1b9fa 100644
@@ -411,10 +411,11 @@ gss_wrap(struct gss_ctx	*ctx_id,
 u32
 gss_unwrap(struct gss_ctx	*ctx_id,
 	   int			offset,
+	   int			len,
 	   struct xdr_buf	*buf)
 {
 	return ctx_id->mech_type->gm_ops
-		->gss_unwrap(ctx_id, offset, buf);
+		->gss_unwrap(ctx_id, offset, len, buf);
 }
 
 
net/sunrpc/auth_gss/svcauth_gss.c +3 -7
index 65b67b257302..322fd48887f9 100644
@@ -900,7 +900,7 @@ unwrap_integ_data(struct svc_rqst *rqstp, struct xdr_buf *buf, u32 seq, struct g
 	if (svc_getnl(&buf->head[0]) != seq)
 		goto out;
 	/* trim off the mic and padding at the end before returning */
-	buf->len -= 4 + round_up_to_quad(mic.len);
+	xdr_buf_trim(buf, round_up_to_quad(mic.len) + 4);
 	stat = 0;
 out:
 	kfree(mic.data);
@@ -928,7 +928,7 @@ static int
 unwrap_priv_data(struct svc_rqst *rqstp, struct xdr_buf *buf, u32 seq, struct gss_ctx *ctx)
 {
 	u32 priv_len, maj_stat;
-	int pad, saved_len, remaining_len, offset;
+	int pad, remaining_len, offset;
 
 	clear_bit(RQ_SPLICE_OK, &rqstp->rq_flags);
 
@@ -948,12 +948,8 @@ unwrap_priv_data(struct svc_rqst *rqstp, struct xdr_buf *buf, u32 seq, struct gs
 	buf->len -= pad;
 	fix_priv_head(buf, pad);
 
-	/* Maybe it would be better to give gss_unwrap a length parameter: */
-	saved_len = buf->len;
-	buf->len = priv_len;
-	maj_stat = gss_unwrap(ctx, 0, buf);
+	maj_stat = gss_unwrap(ctx, 0, priv_len, buf);
 	pad = priv_len - buf->len;
-	buf->len = saved_len;
 	buf->len -= pad;
 	/* The upper layers assume the buffer is aligned on 4-byte boundaries.
 	 * In the krb5p case, at least, the data ends up offset, so we need to
net/sunrpc/clnt.c +5 -0
index 7324b21f923e..3ceaefb2f0bc 100644
@@ -2416,6 +2416,11 @@ rpc_check_timeout(struct rpc_task *task)
 {
 	struct rpc_clnt	*clnt = task->tk_client;
 
+	if (RPC_SIGNALLED(task)) {
+		rpc_call_rpcerror(task, -ERESTARTSYS);
+		return;
+	}
+
 	if (xprt_adjust_timeout(task->tk_rqstp) == 0)
 		return;
 
net/sunrpc/xdr.c +41 -0

47 lines not on this page · raw message

net/sunrpc/xprtrdma/backchannel.c +1 -1
index 1a0ae0c61353..4b43910a6ed2 100644
@@ -115,7 +115,7 @@ int xprt_rdma_bc_send_reply(struct rpc_rqst *rqst)
 	if (rc < 0)
 		goto failed_marshal;
 
-	if (rpcrdma_ep_post(&r_xprt->rx_ia, &r_xprt->rx_ep, req))
+	if (rpcrdma_post_sends(r_xprt, req))
 		goto drop_connection;
 	return 0;
 
net/sunrpc/xprtrdma/frwr_ops.c +9 -5
index 125297c9aa3e..79059d48f52b 100644
@@ -372,18 +372,22 @@ static void frwr_wc_fastreg(struct ib_cq *cq, struct ib_wc *wc)
 }
 
 /**
- * frwr_send - post Send WR containing the RPC Call message
- * @ia: interface adapter
- * @req: Prepared RPC Call
+ * frwr_send - post Send WRs containing the RPC Call message
+ * @r_xprt: controlling transport instance
+ * @req: prepared RPC Call
  *
  * For FRWR, chain any FastReg WRs to the Send WR. Only a
  * single ib_post_send call is needed to register memory
  * and then post the Send WR.
  *
- * Returns the result of ib_post_send.
+ * Returns the return code from ib_post_send.
+ *
+ * Caller must hold the transport send lock to ensure that the
+ * pointers to the transport's rdma_cm_id and QP are stable.
  */
-int frwr_send(struct rpcrdma_ia *ia, struct rpcrdma_req *req)
+int frwr_send(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req)
 {
+	struct rpcrdma_ia *ia = &r_xprt->rx_ia;
 	struct ib_send_wr *post_wr;
 	struct rpcrdma_mr *mr;
 
net/sunrpc/xprtrdma/transport.c +1 -1
index 3cfeba68ee9a..46e7949788e1 100644
@@ -694,7 +694,7 @@ xprt_rdma_send_request(struct rpc_rqst *rqst)
 		goto drop_connection;
 	rqst->rq_xtime = ktime_get();
 
-	if (rpcrdma_ep_post(&r_xprt->rx_ia, &r_xprt->rx_ep, req))
+	if (rpcrdma_post_sends(r_xprt, req))
 		goto drop_connection;
 
 	rqst->rq_xmit_bytes_sent += rqst->rq_snd_buf.len;
net/sunrpc/xprtrdma/verbs.c +6 -9
index 353f61ac8d51..a48b99f3682c 100644
@@ -1502,20 +1502,17 @@ static void rpcrdma_regbuf_free(struct rpcrdma_regbuf *rb)
 }
 
 /**
- * rpcrdma_ep_post - Post WRs to a transport's Send Queue
- * @ia: transport's device information
- * @ep: transport's RDMA endpoint information
+ * rpcrdma_post_sends - Post WRs to a transport's Send Queue
+ * @r_xprt: controlling transport instance
  * @req: rpcrdma_req containing the Send WR to post
  *
  * Returns 0 if the post was successful, otherwise -ENOTCONN
  * is returned.
  */
-int
-rpcrdma_ep_post(struct rpcrdma_ia *ia,
-		struct rpcrdma_ep *ep,
-		struct rpcrdma_req *req)
+int rpcrdma_post_sends(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req)
 {
 	struct ib_send_wr *send_wr = &req->rl_wr;
+	struct rpcrdma_ep *ep = &r_xprt->rx_ep;
 	int rc;
 
 	if (!ep->rep_send_count || kref_read(&req->rl_kref) > 1) {
@@ -1526,8 +1523,8 @@ rpcrdma_ep_post(struct rpcrdma_ia *ia,
 		--ep->rep_send_count;
 	}
 
-	rc = frwr_send(ia, req);
-	trace_xprtrdma_post_send(req, rc);
+	trace_xprtrdma_post_send(req);
+	rc = frwr_send(r_xprt, req);
 	if (rc)
 		return -ENOTCONN;
 	return 0;
net/sunrpc/xprtrdma/xprt_rdma.h +2 -3
index 37d5080c250b..600574a0d838 100644
@@ -469,8 +469,7 @@ void rpcrdma_ep_destroy(struct rpcrdma_xprt *r_xprt);
 int rpcrdma_ep_connect(struct rpcrdma_ep *, struct rpcrdma_ia *);
 void rpcrdma_ep_disconnect(struct rpcrdma_ep *, struct rpcrdma_ia *);
 
-int rpcrdma_ep_post(struct rpcrdma_ia *, struct rpcrdma_ep *,
-				struct rpcrdma_req *);
+int rpcrdma_post_sends(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req);
 void rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, bool temp);
 
 /*
@@ -544,7 +543,7 @@ struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt,
 				struct rpcrdma_mr_seg *seg,
 				int nsegs, bool writing, __be32 xid,
 				struct rpcrdma_mr *mr);
-int frwr_send(struct rpcrdma_ia *ia, struct rpcrdma_req *req);
+int frwr_send(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req);
 void frwr_reminv(struct rpcrdma_rep *rep, struct list_head *mrs);
 void frwr_unmap_sync(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req);
 void frwr_unmap_async(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req);
#6 lore raw
In reply to Hans-Peter Jansen
Quoted 48 lines · Hans-Peter Jansen

Hi,

after upgrading the kernel from 5.6.11 to 5.6.14, we suffer from regular crashes of nfsd here:

2020-06-07T01:32:43.600306+02:00 server rpc.mountd[2664]: authenticated mount request from 192.168.3.16:303 for /work (/work)
2020-06-07T01:32:43.602594+02:00 server rpc.mountd[2664]: authenticated mount request from 192.168.3.16:304 for /work/vmware (/work)
2020-06-07T01:32:43.602971+02:00 server rpc.mountd[2664]: authenticated mount request from 192.168.3.16:305 for /work/vSphere (/work)
2020-06-07T01:32:43.606276+02:00 server kernel: [51901.089211] general protection fault, probably for non-canonical address 0xb9159d506ba40000: 0000 [#1] SMP PTI
2020-06-07T01:32:43.606284+02:00 server kernel: [51901.089226] CPU: 1 PID: 3190 Comm: nfsd Tainted: G           O      5.6.14-lp151.2-default #1 openSUSE Tumbleweed (unreleased)
2020-06-07T01:32:43.606286+02:00 server kernel: [51901.089234] Hardware name: System manufacturer System Product Name/P7F-E, BIOS 0906    09/20/2010
2020-06-07T01:32:43.606287+02:00 server kernel: [51901.089247] RIP: 0010:cgroup_sk_free+0x26/0x80
2020-06-07T01:32:43.606288+02:00 server kernel: [51901.089257] Code: 00 00 00 00 66 66 66 66 90 53 48 8b 07 48 c7 c3 30 72 07 b6 a8 01 75 07 48 85 c0 48 0f 45 d8 48 8b 83 18 09 00 00 a8 03 
75 1a <65> 48 ff 08 f6 43 7c 01 74 02 5b c3 48 8b 43 18 a8 03 75 26 65 48
2020-06-07T01:32:43.606290+02:00 server kernel: [51901.089276] RSP: 0018:ffffb248c21e7e10 EFLAGS: 00010246
2020-06-07T01:32:43.606291+02:00 server kernel: [51901.089280] RAX: b91603a504000000 RBX: ffff99ab141a0000 RCX: 0000000000000021
2020-06-07T01:32:43.606292+02:00 server kernel: [51901.089284] RDX: ffffffffb6135ec4 RSI: 0000000000010080 RDI: ffff99a7159c1490
2020-06-07T01:32:43.606293+02:00 server kernel: [51901.089287] RBP: ffff99a7159c1200 R08: ffff99ab67a60c60 R09: 000000000002eb00
2020-06-07T01:32:43.606294+02:00 server kernel: [51901.089291] R10: ffffb248c0087dc0 R11: 00000000000000c6 R12: 0000000000000000
2020-06-07T01:32:43.606295+02:00 server kernel: [51901.089294] R13: 0000000000000103 R14: ffff99aae4934238 R15: ffff99ab31902000
2020-06-07T01:32:43.606296+02:00 server kernel: [51901.089299] FS:  0000000000000000(0000) GS:ffff99ab67a40000(0000) knlGS:0000000000000000
2020-06-07T01:32:43.606297+02:00 server kernel: [51901.089303] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
2020-06-07T01:32:43.606303+02:00 server kernel: [51901.089305] CR2: 00000000008e0000 CR3: 00000004df60a000 CR4: 00000000000026e0
2020-06-07T01:32:43.606304+02:00 server kernel: [51901.089307] Call Trace:
2020-06-07T01:32:43.606305+02:00 server kernel: [51901.089315]  __sk_destruct+0x10d/0x1d0
2020-06-07T01:32:43.606306+02:00 server kernel: [51901.089319]  inet_release+0x34/0x60
2020-06-07T01:32:43.606307+02:00 server kernel: [51901.089325]  __sock_release+0x81/0xb0
2020-06-07T01:32:43.606308+02:00 server kernel: [51901.089358]  svc_sock_free+0x38/0x60 [sunrpc]
2020-06-07T01:32:43.606308+02:00 server kernel: [51901.089374]  svc_xprt_put+0x99/0xe0 [sunrpc]
2020-06-07T01:32:43.606310+02:00 server kernel: [51901.089389]  svc_recv+0x9c0/0xa40 [sunrpc]
2020-06-07T01:32:43.606310+02:00 server kernel: [51901.089410]  ? nfsd_destroy+0x60/0x60 [nfsd]
2020-06-07T01:32:43.606311+02:00 server kernel: [51901.089417]  nfsd+0xd1/0x150 [nfsd]
2020-06-07T01:32:43.606312+02:00 server kernel: [51901.089420]  kthread+0x10d/0x130
2020-06-07T01:32:43.606313+02:00 server kernel: [51901.089423]  ? kthread_park+0x90/0x90
2020-06-07T01:32:43.606314+02:00 server kernel: [51901.089426]  ret_from_fork+0x35/0x40

A vSphere 5.5 host accesses this linux server with nfs v3 for backup purposes (a Veeam backup server want to store a new backup here).

The kernel is tainted due to vboxdrv. The OS is openSUSE Leap 15.1, with the kernel and Virtualbox replaced with uptodate versions from proper rpm packages (built on that very vSphere host in a OBS server VM..).

Quoted 5 lines

I used to be subscribed to this ML, but that subscription has been lost 04/09, thus I cannot reply properly to the general prot. fault thread, started 05/12 from syzbot with Bruce looking into it.

It seems somewhat related.

Your backtrace doesn't look anything like the syzbot crashes Bruce is looking at, and there are no fs/nfsd/ changes between v5.6.11 and v5.6.14. His crashes appear to be related entirely to the order of destruction of net namespaces and NFS server data structures -- nothing at the socket layer.

The net/sunrpc/ changes in that commit range have nothing to do with socket allocation. However, this:

   [51901.089247] RIP: 0010:cgroup_sk_free+0x26/0x80

suggests something else. There is a cgroup/sk related change in that commit range:

e2d928d5ee43 ("netprio_cgroup: Fix unlimited memory leak of v2 cgroups")

I'm not sure how to help you further, since you are not available to test this theory for a few weeks. The best I can suggest for others is to stick with v5.6.11-based kernels until someone with a reproducer can bisect between .11 and .14 to confirm the theory.

Quoted 10 lines

Interestingly, we're using a couple of NFS v4 mounts for subsets of home here, and mount /work and other shares from various Tumbleweed systems with NFS v4 here without any undesired effects.

Since the kernel upgrade, every time, this Veeam thing triggers these v3 mounts, the crash happens. I've disabled this backup target for now until the problem is resolved, because it effectively prevents further nfs accesses to this server, and blocks our desktops until the server is rebooted.

Quoted 4 lines

A cursory look into 5.6.{15,16} changelogs seems to imply, that this issue is still pending.

Let me know, if I can provide any further info's.

--
Chuck Lever