xprtrdam: Don't treat a call as bcall when bc_serv is NULL
Subsystems (3)
PATCH 1/1 xprtrdam: Don't treat a call as bcall when bc_serv is NULL
When rdma server returns a fault reply, rpcrdma may treats it as a bcall. As using NFSv3, a bc server is never exist. rpcrdma_bc_receive_call will meets NULL pointer as,
[ 226.057890] BUG: unable to handle kernel NULL pointer dereference at 00000000000000c8 ... [ 226.058704] RIP: 0010:_raw_spin_lock+0xc/0x20 ... [ 226.059732] Call Trace: [ 226.059878] rpcrdma_bc_receive_call+0x138/0x327 [rpcrdma] [ 226.060011] __ib_process_cq+0x89/0x170 [ib_core] [ 226.060092] ib_cq_poll_work+0x26/0x80 [ib_core] [ 226.060257] process_one_work+0x1a7/0x360 [ 226.060367] ? create_worker+0x1a0/0x1a0 [ 226.060440] worker_thread+0x30/0x390 [ 226.060500] ? create_worker+0x1a0/0x1a0 [ 226.060574] kthread+0x116/0x130 [ 226.060661] ? kthread_flush_work_fn+0x10/0x10 [ 226.060724] ret_from_fork+0x35/0x40
...
--- net/sunrpc/xprtrdma/rpc_rdma.c | 5 +++++ 1 file changed, 5 insertions(+)
net/sunrpc/xprtrdma/rpc_rdma.c +5 -0
@@ -1121,9 +1121,14 @@ static bool rpcrdma_is_bcall(struct rpcrdma_xprt *r_xprt, struct rpcrdma_rep *rep) #if defined(CONFIG_SUNRPC_BACKCHANNEL) { + struct rpc_xprt *xprt = &r_xprt->rx_xprt;struct xdr_stream *xdr = &rep->rr_stream;__be32 *p; +/* no bc service, not a bcall. */ +if (xprt->bc_serv ==NULL ) +return false; +if (rep->rr_proc != rdma_msg)return false;
Signature · 1 line
2.27.0
Quoted 38 lines · Kinglong Mee
When rdma server returns a fault reply, rpcrdma may treats it as a bcall. As using NFSv3, a bc server is never exist.
rpcrdma_bc_receive_callwill meets NULL pointer as,[ 226.057890] BUG: unable to handle kernel NULL pointer dereference at 00000000000000c8 ... [ 226.058704] RIP: 0010:_raw_spin_lock+0xc/0x20 ... [ 226.059732] Call Trace: [ 226.059878] rpcrdma_bc_receive_call+0x138/0x327 [rpcrdma] [ 226.060011] __ib_process_cq+0x89/0x170 [ib_core] [ 226.060092] ib_cq_poll_work+0x26/0x80 [ib_core] [ 226.060257] process_one_work+0x1a7/0x360 [ 226.060367] ? create_worker+0x1a0/0x1a0 [ 226.060440] worker_thread+0x30/0x390 [ 226.060500] ? create_worker+0x1a0/0x1a0 [ 226.060574] kthread+0x116/0x130 [ 226.060661] ? kthread_flush_work_fn+0x10/0x10 [ 226.060724] ret_from_fork+0x35/0x40...
Signed-off-by Kinglong Mee <[email protected]>---
net/sunrpc/xprtrdma/rpc_rdma.c| 5 +++++1 file changed, 5 insertions(+)net/sunrpc/xprtrdma/rpc_rdma.c +0
-0@@ -1121,9 +1121,14 @@ static bool rpcrdma_is_bcall(struct rpcrdma_xprt *r_xprt, struct rpcrdma_rep *rep) #if defined(CONFIG_SUNRPC_BACKCHANNEL) {
net/sunrpc/xprtrdma/rpc_rdma.c static bool · line 1133
Quoted 10 lines
net/sunrpc/xprtrdma/rpc_rdma.c +5
-0+struct rpc_xprt *xprt = &r_xprt->rx_xprt;struct xdr_stream *xdr = &rep->rr_stream;__be32 *p; +/* no bc service, not a bcall. */ +if (xprt->bc_serv ==NULL ) +return false; +if (rep->rr_proc != rdma_msg)return false;
I'm not sure what you mean above by "fault reply".
The check here for whether the RPC/RDMA procedure is an RDMA_MSG is supposed to be enough to avoid any further processing of an RDMA_ERR type procedure.
What kind of fault has occurred? Can you share with us the actual RPC/RDMA transport header that triggers the BUG?
--
Chuck Lever
Hi Chuck,
Quoted 51 lines · Chuck Lever III
Quoted 38 lines · Kinglong Mee
When rdma server returns a fault reply, rpcrdma may treats it as a bcall. As using NFSv3, a bc server is never exist.
rpcrdma_bc_receive_callwill meets NULL pointer as,[ 226.057890] BUG: unable to handle kernel NULL pointer dereference at 00000000000000c8 ... [ 226.058704] RIP: 0010:_raw_spin_lock+0xc/0x20 ... [ 226.059732] Call Trace: [ 226.059878] rpcrdma_bc_receive_call+0x138/0x327 [rpcrdma] [ 226.060011] __ib_process_cq+0x89/0x170 [ib_core] [ 226.060092] ib_cq_poll_work+0x26/0x80 [ib_core] [ 226.060257] process_one_work+0x1a7/0x360 [ 226.060367] ? create_worker+0x1a0/0x1a0 [ 226.060440] worker_thread+0x30/0x390 [ 226.060500] ? create_worker+0x1a0/0x1a0 [ 226.060574] kthread+0x116/0x130 [ 226.060661] ? kthread_flush_work_fn+0x10/0x10 [ 226.060724] ret_from_fork+0x35/0x40...
Signed-off-by Kinglong Mee <[email protected]>---
net/sunrpc/xprtrdma/rpc_rdma.c| 5 +++++1 file changed, 5 insertions(+)net/sunrpc/xprtrdma/rpc_rdma.c +0
-0@@ -1121,9 +1121,14 @@ static bool rpcrdma_is_bcall(struct rpcrdma_xprt *r_xprt, struct rpcrdma_rep *rep) #if defined(CONFIG_SUNRPC_BACKCHANNEL) {net/sunrpc/xprtrdma/rpc_rdma.c static bool · line 1133
Quoted 10 lines
net/sunrpc/xprtrdma/rpc_rdma.c +5
-0+struct rpc_xprt *xprt = &r_xprt->rx_xprt;struct xdr_stream *xdr = &rep->rr_stream;__be32 *p; +/* no bc service, not a bcall. */ +if (xprt->bc_serv ==NULL ) +return false; +if (rep->rr_proc != rdma_msg)return false;I'm not sure what you mean above by "fault reply".
Quoted 6 lines
The check here for whether the RPC/RDMA procedure is an
RDMA_MSGis supposed to be enough to avoid any further processing of anRDMA_ERRtype procedure.What kind of fault has occurred? Can you share with us the actual RPC/RDMA transport header that triggers the BUG?
I have a nfs rdma server, but it handles drc reply wrongly sometimes, it returns a bad format reply to nfs client.
Nfs rdma client treats the bad format reply as a bcall, and
Thanks,
Kinglong Mee
Quoted 60 lines · Kinglong Mee
Hi Chuck,
Quoted 40 lines · Chuck Lever III
Quoted 40 lines · Kinglong Mee
When rdma server returns a fault reply, rpcrdma may treats it as a bcall. As using NFSv3, a bc server is never exist.
rpcrdma_bc_receive_callwill meets NULL pointer as,[ 226.057890] BUG: unable to handle kernel NULL pointer dereference at 00000000000000c8 ... [ 226.058704] RIP: 0010:_raw_spin_lock+0xc/0x20 ... [ 226.059732] Call Trace: [ 226.059878] rpcrdma_bc_receive_call+0x138/0x327 [rpcrdma] [ 226.060011] __ib_process_cq+0x89/0x170 [ib_core] [ 226.060092] ib_cq_poll_work+0x26/0x80 [ib_core] [ 226.060257] process_one_work+0x1a7/0x360 [ 226.060367] ? create_worker+0x1a0/0x1a0 [ 226.060440] worker_thread+0x30/0x390 [ 226.060500] ? create_worker+0x1a0/0x1a0 [ 226.060574] kthread+0x116/0x130 [ 226.060661] ? kthread_flush_work_fn+0x10/0x10 [ 226.060724] ret_from_fork+0x35/0x40...
Signed-off-by Kinglong Mee <[email protected]>---
net/sunrpc/xprtrdma/rpc_rdma.c| 5 +++++1 file changed, 5 insertions(+)net/sunrpc/xprtrdma/rpc_rdma.c +1
-0@@ -1121,9 +1121,14 @@ static bool rpcrdma_is_bcall(struct rpcrdma_xprt *r_xprt, struct rpcrdma_rep *rep) #if defined(CONFIG_SUNRPC_BACKCHANNEL) { + struct rpc_xprt *xprt = &r_xprt->rx_xprt;struct xdr_stream *xdr = &rep->rr_stream;__be32 *p;Quoted 12 lines
net/sunrpc/xprtrdma/rpc_rdma.c static bool · line 1133
Quoted 6 lines
net/sunrpc/xprtrdma/rpc_rdma.c +4
-0+/* no bc service, not a bcall. */ +if (xprt->bc_serv ==NULL ) +return false; +if (rep->rr_proc != rdma_msg)return false;I'm not sure what you mean above by "fault reply".
The check here for whether the RPC/RDMA procedure is anRDMA_MSGis supposed to be enough to avoid any further processing of anRDMA_ERRtype procedure.
What kind of fault has occurred? Can you share with us the actual RPC/RDMA transport header that triggers the BUG?
Quoted 3 lines
I have a nfs rdma server, but it handles drc reply wrongly sometimes, it returns a bad format reply to nfs client.
Nfs rdma client treats the bad format reply as a bcall, and
Doesn't this test return false:
1144 if (*p != cpu_to_be32(RPC_CALL))
1145 return false;
But OK: a malicious NFSv3 server can trigger a client crash. That's a bug.
This is an additional conditional in a hot path. Would it make sense to move the new test to just after 1145?
Even then, it could be a bcall, the client simply isn't prepared to process it. So "/* no bc service */" by itself would be a more accurate comment.
--
Chuck Lever
Quoted 60 lines · Chuck Lever III
Quoted 60 lines · Kinglong Mee
Hi Chuck,
Quoted 52 lines · Chuck Lever III
Quoted 36 lines · Kinglong Mee
When rdma server returns a fault reply, rpcrdma may treats it as a bcall. As using NFSv3, a bc server is never exist.
rpcrdma_bc_receive_callwill meets NULL pointer as,[ 226.057890] BUG: unable to handle kernel NULL pointer dereference at 00000000000000c8 ... [ 226.058704] RIP: 0010:_raw_spin_lock+0xc/0x20 ... [ 226.059732] Call Trace: [ 226.059878] rpcrdma_bc_receive_call+0x138/0x327 [rpcrdma] [ 226.060011] __ib_process_cq+0x89/0x170 [ib_core] [ 226.060092] ib_cq_poll_work+0x26/0x80 [ib_core] [ 226.060257] process_one_work+0x1a7/0x360 [ 226.060367] ? create_worker+0x1a0/0x1a0 [ 226.060440] worker_thread+0x30/0x390 [ 226.060500] ? create_worker+0x1a0/0x1a0 [ 226.060574] kthread+0x116/0x130 [ 226.060661] ? kthread_flush_work_fn+0x10/0x10 [ 226.060724] ret_from_fork+0x35/0x40...
Signed-off-by Kinglong Mee <[email protected]>---
net/sunrpc/xprtrdma/rpc_rdma.c| 5 +++++1 file changed, 5 insertions(+)net/sunrpc/xprtrdma/rpc_rdma.c +0
-0@@ -1121,9 +1121,14 @@ static bool rpcrdma_is_bcall(struct rpcrdma_xprt *r_xprt, struct rpcrdma_rep *rep) #if defined(CONFIG_SUNRPC_BACKCHANNEL) {net/sunrpc/xprtrdma/rpc_rdma.c static bool · line 1133
Quoted 10 lines
net/sunrpc/xprtrdma/rpc_rdma.c +5
-0+struct rpc_xprt *xprt = &r_xprt->rx_xprt;struct xdr_stream *xdr = &rep->rr_stream;__be32 *p; +/* no bc service, not a bcall. */ +if (xprt->bc_serv ==NULL ) +return false; +if (rep->rr_proc != rdma_msg)return false;I'm not sure what you mean above by "fault reply".
The check here for whether the RPC/RDMA procedure is anRDMA_MSGis supposed to be enough to avoid any further processing of anRDMA_ERRtype procedure.
What kind of fault has occurred? Can you share with us the actual RPC/RDMA transport header that triggers the BUG?
Quoted 8 lines
Quoted 3 lines
I have a nfs rdma server, but it handles drc reply wrongly sometimes, it returns a bad format reply to nfs client.
Nfs rdma client treats the bad format reply as a bcall, andDoesn't this test return false:
1144 if (*p !=
cpu_to_be32(RPC_CALL))
1145 return false;
No, it doesn't return false here.
The debug message at rpcrdma_bc_receive_call are,
[56579.837169] RPC: rpcrdma_bc_receive_call: callback XID 00000001, length=20 [56579.837174] RPC: rpcrdma_bc_receive_call: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 04
Quoted 9 lines
But OK: a malicious NFSv3 server can trigger a client crash. That's a bug.
This is an additional conditional in a hot path. Would it make sense to move the new test to just after 1145? > Even then, it could be a bcall, the client simply isn't prepared to process it. So "/* no bc service */" by itself would be a more accurate comment.
Okay, I will update it and send a v2 patch after moving the checking after 1145.
Thanks,
Kinglong Mee
PATCH v2 1/1 xprtrdma: treat all calls not a bcall when bc_serv is NULL
In reply to Chuck Lever IIIWhen a rdma server returns a fault format reply, nfs v3 client may treats it as a bcall when bc service is not exist.
The debug message at rpcrdma_bc_receive_call are,
[56579.837169] RPC: rpcrdma_bc_receive_call: callback XID 00000001, length=20 [56579.837174] RPC: rpcrdma_bc_receive_call: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 04
After that, rpcrdma_bc_receive_call will meets NULL pointer as,
[ 226.057890] BUG: unable to handle kernel NULL pointer dereference at 00000000000000c8 ... [ 226.058704] RIP: 0010:_raw_spin_lock+0xc/0x20 ... [ 226.059732] Call Trace: [ 226.059878] rpcrdma_bc_receive_call+0x138/0x327 [rpcrdma] [ 226.060011] __ib_process_cq+0x89/0x170 [ib_core] [ 226.060092] ib_cq_poll_work+0x26/0x80 [ib_core] [ 226.060257] process_one_work+0x1a7/0x360 [ 226.060367] ? create_worker+0x1a0/0x1a0 [ 226.060440] worker_thread+0x30/0x390 [ 226.060500] ? create_worker+0x1a0/0x1a0 [ 226.060574] kthread+0x116/0x130 [ 226.060661] ? kthread_flush_work_fn+0x10/0x10 [ 226.060724] ret_from_fork+0x35/0x40
...
--- net/sunrpc/xprtrdma/rpc_rdma.c | 5 +++++ 1 file changed, 5 insertions(+)
net/sunrpc/xprtrdma/rpc_rdma.c +5 -0
@@ -1121,6 +1121,7 @@ static bool rpcrdma_is_bcall(struct rpcrdma_xprt *r_xprt, struct rpcrdma_rep *rep) #if defined(CONFIG_SUNRPC_BACKCHANNEL) { + struct rpc_xprt *xprt = &r_xprt->rx_xprt;struct xdr_stream *xdr = &rep->rr_stream;__be32 *p;
@@ -1144,6 +1145,10 @@ rpcrdma_is_bcall(struct rpcrdma_xprt *r_xprt, struct rpcrdma_rep *rep)if (*p !=cpu_to_be32 (RPC_CALL ))return false; +/* No bc service. */ +if (xprt->bc_serv ==NULL ) +return false; +/* Now that we are sure this is a backchannel call, * advance to the RPC header. */
Signature · 1 line
2.27.0
Re: [PATCH v2] xprtrdma: treat all calls not a bcall when bc_serv is NULL
Quoted 29 lines · Kinglong Mee
When a rdma server returns a fault format reply, nfs v3 client may treats it as a bcall when bc service is not exist.
The debug message at
rpcrdma_bc_receive_callare,[56579.837169] RPC: rpcrdma_bc_receive_call: callback XID 00000001, length=20 [56579.837174] RPC: rpcrdma_bc_receive_call: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 04After that,
rpcrdma_bc_receive_callwill meets NULL pointer as,[ 226.057890] BUG: unable to handle kernel NULL pointer dereference at 00000000000000c8 ... [ 226.058704] RIP: 0010:_raw_spin_lock+0xc/0x20 ... [ 226.059732] Call Trace: [ 226.059878] rpcrdma_bc_receive_call+0x138/0x327 [rpcrdma] [ 226.060011] __ib_process_cq+0x89/0x170 [ib_core] [ 226.060092] ib_cq_poll_work+0x26/0x80 [ib_core] [ 226.060257] process_one_work+0x1a7/0x360 [ 226.060367] ? create_worker+0x1a0/0x1a0 [ 226.060440] worker_thread+0x30/0x390 [ 226.060500] ? create_worker+0x1a0/0x1a0 [ 226.060574] kthread+0x116/0x130 [ 226.060661] ? kthread_flush_work_fn+0x10/0x10 [ 226.060724] ret_from_fork+0x35/0x40
Quoted 3 lines
...
Signed-off-by Kinglong Mee <[email protected]>
Patch is good, Reviewed-by: Chuck Lever <[email protected]>
Commentary (no changes to v2 needed):
The description still suggests we are adapting the client to work around a broken server. I don't feel this is the right reason to accept this change. Instead: we really don't want the client to be vulnerable to bad input for any reason. After this fix is applied, bad RPC messages are eventually dropped rather than processed, which is the desired behavior.
Anna, I recommend adding Cc: stable for this fix.
Kinglong, please ensure that client Receive resources are not leaked in this case. If they are, send along an additional patch; if not, let us know and we can close this issue. Thank you!
Quoted 30 lines
---
net/sunrpc/xprtrdma/rpc_rdma.c| 5 +++++1 file changed, 5 insertions(+)net/sunrpc/xprtrdma/rpc_rdma.c +5
-0@@ -1121,6 +1121,7 @@ static bool rpcrdma_is_bcall(struct rpcrdma_xprt *r_xprt, struct rpcrdma_rep *rep) #if defined(CONFIG_SUNRPC_BACKCHANNEL) { + struct rpc_xprt *xprt = &r_xprt->rx_xprt;struct xdr_stream *xdr = &rep->rr_stream;__be32 *p;@@ -1144,6 +1145,10 @@ rpcrdma_is_bcall(struct rpcrdma_xprt *r_xprt, struct rpcrdma_rep *rep)if (*p !=cpu_to_be32 (RPC_CALL ))return false; +/* No bc service. */ +if (xprt->bc_serv ==NULL ) +return false; +/* Now that we are sure this is a backchannel call, * advance to the RPC header. */ Signature · 1 line
2.27.0
--
Chuck Lever
Re: [PATCH v2] xprtrdma: treat all calls not a bcall when bc_serv is NULL
Quoted 44 lines · Chuck Lever III
Quoted 29 lines · Kinglong Mee
When a rdma server returns a fault format reply, nfs v3 client may treats it as a bcall when bc service is not exist.
The debug message at
rpcrdma_bc_receive_callare,[56579.837169] RPC: rpcrdma_bc_receive_call: callback XID 00000001, length=20 [56579.837174] RPC: rpcrdma_bc_receive_call: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 04After that,
rpcrdma_bc_receive_callwill meets NULL pointer as,[ 226.057890] BUG: unable to handle kernel NULL pointer dereference at 00000000000000c8 ... [ 226.058704] RIP: 0010:_raw_spin_lock+0xc/0x20 ... [ 226.059732] Call Trace: [ 226.059878] rpcrdma_bc_receive_call+0x138/0x327 [rpcrdma] [ 226.060011] __ib_process_cq+0x89/0x170 [ib_core] [ 226.060092] ib_cq_poll_work+0x26/0x80 [ib_core] [ 226.060257] process_one_work+0x1a7/0x360 [ 226.060367] ? create_worker+0x1a0/0x1a0 [ 226.060440] worker_thread+0x30/0x390 [ 226.060500] ? create_worker+0x1a0/0x1a0 [ 226.060574] kthread+0x116/0x130 [ 226.060661] ? kthread_flush_work_fn+0x10/0x10 [ 226.060724] ret_from_fork+0x35/0x40Quoted 3 lines
...
Signed-off-by Kinglong Mee <[email protected]>Patch is good, Reviewed-by: Chuck Lever <[email protected]>
Commentary (no changes to v2 needed):
The description still suggests we are adapting the client to work around a broken server. I don't feel this is the right reason to accept this change. Instead: we really don't want the client to be vulnerable to bad input for any reason. After this fix is applied, bad RPC messages are eventually dropped rather than processed, which is the desired behavior.
Quoted 5 lines
Anna, I recommend adding Cc: stable for this fix.
Kinglong, please ensure that client Receive resources are not leaked in this case. If they are, send along an additional patch; if not, let us know and we can close this issue.
I have double check of the code.
I think client Receive resources are not leaked here. We can close it.
Thanks,
Kinglong Mee
Quoted 38 lines
Quoted 32 lines
---
net/sunrpc/xprtrdma/rpc_rdma.c| 5 +++++1 file changed, 5 insertions(+)net/sunrpc/xprtrdma/rpc_rdma.c +5
-0@@ -1121,6 +1121,7 @@ static bool rpcrdma_is_bcall(struct rpcrdma_xprt *r_xprt, struct rpcrdma_rep *rep) #if defined(CONFIG_SUNRPC_BACKCHANNEL) { + struct rpc_xprt *xprt = &r_xprt->rx_xprt;struct xdr_stream *xdr = &rep->rr_stream;__be32 *p;@@ -1144,6 +1145,10 @@ rpcrdma_is_bcall(struct rpcrdma_xprt *r_xprt, struct rpcrdma_rep *rep)if (*p !=cpu_to_be32 (RPC_CALL ))return false; +/* No bc service. */ +if (xprt->bc_serv ==NULL ) +return false; +/* Now that we are sure this is a backchannel call, * advance to the RPC header. */ Signature · 1 line
2.27.0--
Chuck Lever