Skip to contentSkip to conversation

xprtrdam: Don't treat a call as bcall when bc_serv is NULL

PATCH In review Kinglong Mee 2 participants 8 messages May 21, 2022 – May 23, 2022

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 xprtrdam: Don't treat a call as bcall when bc_serv is NULL

When rdma server returns a fault reply, rpcrdma may treats it as a bcall. As using NFSv3, a bc server is never exist. rpcrdma_bc_receive_call will meets NULL pointer as,

[  226.057890] BUG: unable to handle kernel NULL pointer dereference at 00000000000000c8
...
[  226.058704] RIP: 0010:_raw_spin_lock+0xc/0x20
...
[  226.059732] Call Trace:
[  226.059878]  rpcrdma_bc_receive_call+0x138/0x327 [rpcrdma]
[  226.060011]  __ib_process_cq+0x89/0x170 [ib_core]
[  226.060092]  ib_cq_poll_work+0x26/0x80 [ib_core]
[  226.060257]  process_one_work+0x1a7/0x360
[  226.060367]  ? create_worker+0x1a0/0x1a0
[  226.060440]  worker_thread+0x30/0x390
[  226.060500]  ? create_worker+0x1a0/0x1a0
[  226.060574]  kthread+0x116/0x130
[  226.060661]  ? kthread_flush_work_fn+0x10/0x10
[  226.060724]  ret_from_fork+0x35/0x40

...

---
 net/sunrpc/xprtrdma/rpc_rdma.c | 5 +++++
 1 file changed, 5 insertions(+)
net/sunrpc/xprtrdma/rpc_rdma.c +5 -0
index 281ddb87ac8d..9486bb98eb2f 100644
@@ -1121,9 +1121,14 @@ static bool
 rpcrdma_is_bcall(struct rpcrdma_xprt *r_xprt, struct rpcrdma_rep *rep)
 #if defined(CONFIG_SUNRPC_BACKCHANNEL)
 {
+	struct rpc_xprt *xprt = &r_xprt->rx_xprt;
 	struct xdr_stream *xdr = &rep->rr_stream;
 	__be32 *p;
 
+	/* no bc service, not a bcall. */
+	if (xprt->bc_serv == NULL)
+		return false;
+
 	if (rep->rr_proc != rdma_msg)
 		return false;
 
Signature · 1 line
2.27.0
#1 lore raw
Quoted 38 lines · Kinglong Mee

When rdma server returns a fault reply, rpcrdma may treats it as a bcall. As using NFSv3, a bc server is never exist. rpcrdma_bc_receive_call will meets NULL pointer as,

[  226.057890] BUG: unable to handle kernel NULL pointer dereference at 00000000000000c8
...
[  226.058704] RIP: 0010:_raw_spin_lock+0xc/0x20
...
[  226.059732] Call Trace:
[  226.059878]  rpcrdma_bc_receive_call+0x138/0x327 [rpcrdma]
[  226.060011]  __ib_process_cq+0x89/0x170 [ib_core]
[  226.060092]  ib_cq_poll_work+0x26/0x80 [ib_core]
[  226.060257]  process_one_work+0x1a7/0x360
[  226.060367]  ? create_worker+0x1a0/0x1a0
[  226.060440]  worker_thread+0x30/0x390
[  226.060500]  ? create_worker+0x1a0/0x1a0
[  226.060574]  kthread+0x116/0x130
[  226.060661]  ? kthread_flush_work_fn+0x10/0x10
[  226.060724]  ret_from_fork+0x35/0x40

...

---
net/sunrpc/xprtrdma/rpc_rdma.c | 5 +++++

1 file changed, 5 insertions(+)
net/sunrpc/xprtrdma/rpc_rdma.c +0 -0
index 281ddb87ac8d..9486bb98eb2f 100644
@@ -1121,9 +1121,14 @@ static bool
 rpcrdma_is_bcall(struct rpcrdma_xprt *r_xprt, struct rpcrdma_rep *rep)
 #if defined(CONFIG_SUNRPC_BACKCHANNEL)
 {

net/sunrpc/xprtrdma/rpc_rdma.c static bool · line 1133

Quoted 10 lines
net/sunrpc/xprtrdma/rpc_rdma.c +5 -0
+	struct rpc_xprt *xprt = &r_xprt->rx_xprt;
 	struct xdr_stream *xdr = &rep->rr_stream;
 	__be32 *p;
 
+	/* no bc service, not a bcall. */
+	if (xprt->bc_serv == NULL)
+		return false;
+
 	if (rep->rr_proc != rdma_msg)
 		return false;

I'm not sure what you mean above by "fault reply".

The check here for whether the RPC/RDMA procedure is an RDMA_MSG is supposed to be enough to avoid any further processing of an RDMA_ERR type procedure.

What kind of fault has occurred? Can you share with us the actual RPC/RDMA transport header that triggers the BUG?

--
Chuck Lever

#2 lore raw

Hi Chuck,

Quoted 51 lines · Chuck Lever III
Quoted 38 lines · Kinglong Mee

When rdma server returns a fault reply, rpcrdma may treats it as a bcall. As using NFSv3, a bc server is never exist. rpcrdma_bc_receive_call will meets NULL pointer as,

[  226.057890] BUG: unable to handle kernel NULL pointer dereference at 00000000000000c8
...
[  226.058704] RIP: 0010:_raw_spin_lock+0xc/0x20
...
[  226.059732] Call Trace:
[  226.059878]  rpcrdma_bc_receive_call+0x138/0x327 [rpcrdma]
[  226.060011]  __ib_process_cq+0x89/0x170 [ib_core]
[  226.060092]  ib_cq_poll_work+0x26/0x80 [ib_core]
[  226.060257]  process_one_work+0x1a7/0x360
[  226.060367]  ? create_worker+0x1a0/0x1a0
[  226.060440]  worker_thread+0x30/0x390
[  226.060500]  ? create_worker+0x1a0/0x1a0
[  226.060574]  kthread+0x116/0x130
[  226.060661]  ? kthread_flush_work_fn+0x10/0x10
[  226.060724]  ret_from_fork+0x35/0x40

...

---
net/sunrpc/xprtrdma/rpc_rdma.c | 5 +++++

1 file changed, 5 insertions(+)
net/sunrpc/xprtrdma/rpc_rdma.c +0 -0
index 281ddb87ac8d..9486bb98eb2f 100644
@@ -1121,9 +1121,14 @@ static bool
 rpcrdma_is_bcall(struct rpcrdma_xprt *r_xprt, struct rpcrdma_rep *rep)
 #if defined(CONFIG_SUNRPC_BACKCHANNEL)
 {

net/sunrpc/xprtrdma/rpc_rdma.c static bool · line 1133

Quoted 10 lines
net/sunrpc/xprtrdma/rpc_rdma.c +5 -0
+	struct rpc_xprt *xprt = &r_xprt->rx_xprt;
 	struct xdr_stream *xdr = &rep->rr_stream;
 	__be32 *p;
 
+	/* no bc service, not a bcall. */
+	if (xprt->bc_serv == NULL)
+		return false;
+
 	if (rep->rr_proc != rdma_msg)
 		return false;

I'm not sure what you mean above by "fault reply".

Quoted 6 lines

The check here for whether the RPC/RDMA procedure is an RDMA_MSG is supposed to be enough to avoid any further processing of an RDMA_ERR type procedure.

What kind of fault has occurred? Can you share with us the actual RPC/RDMA transport header that triggers the BUG?

I have a nfs rdma server, but it handles drc reply wrongly sometimes, it returns a bad format reply to nfs client.
Nfs rdma client treats the bad format reply as a bcall, and

Thanks,
Kinglong Mee

#3 lore raw
Quoted 60 lines · Kinglong Mee

Hi Chuck,

Quoted 40 lines · Chuck Lever III
Quoted 40 lines · Kinglong Mee

When rdma server returns a fault reply, rpcrdma may treats it as a bcall. As using NFSv3, a bc server is never exist. rpcrdma_bc_receive_call will meets NULL pointer as,

[  226.057890] BUG: unable to handle kernel NULL pointer dereference at 00000000000000c8
...
[  226.058704] RIP: 0010:_raw_spin_lock+0xc/0x20
...
[  226.059732] Call Trace:
[  226.059878]  rpcrdma_bc_receive_call+0x138/0x327 [rpcrdma]
[  226.060011]  __ib_process_cq+0x89/0x170 [ib_core]
[  226.060092]  ib_cq_poll_work+0x26/0x80 [ib_core]
[  226.060257]  process_one_work+0x1a7/0x360
[  226.060367]  ? create_worker+0x1a0/0x1a0
[  226.060440]  worker_thread+0x30/0x390
[  226.060500]  ? create_worker+0x1a0/0x1a0
[  226.060574]  kthread+0x116/0x130
[  226.060661]  ? kthread_flush_work_fn+0x10/0x10
[  226.060724]  ret_from_fork+0x35/0x40

...

---
net/sunrpc/xprtrdma/rpc_rdma.c | 5 +++++

1 file changed, 5 insertions(+)
net/sunrpc/xprtrdma/rpc_rdma.c +1 -0
index 281ddb87ac8d..9486bb98eb2f 100644
@@ -1121,9 +1121,14 @@ static bool
 rpcrdma_is_bcall(struct rpcrdma_xprt *r_xprt, struct rpcrdma_rep *rep)
 #if defined(CONFIG_SUNRPC_BACKCHANNEL)
 {
+	struct rpc_xprt *xprt = &r_xprt->rx_xprt;
 	struct xdr_stream *xdr = &rep->rr_stream;
 	__be32 *p;
 
Quoted 12 lines

net/sunrpc/xprtrdma/rpc_rdma.c static bool · line 1133

Quoted 6 lines
net/sunrpc/xprtrdma/rpc_rdma.c +4 -0
+	/* no bc service, not a bcall. */
+	if (xprt->bc_serv == NULL)
+		return false;
+
 	if (rep->rr_proc != rdma_msg)
 		return false;

I'm not sure what you mean above by "fault reply".
The check here for whether the RPC/RDMA procedure is an RDMA_MSG is supposed to be enough to avoid any further processing of an RDMA_ERR type procedure.
What kind of fault has occurred? Can you share with us the actual RPC/RDMA transport header that triggers the BUG?

Quoted 3 lines

I have a nfs rdma server, but it handles drc reply wrongly sometimes, it returns a bad format reply to nfs client.
Nfs rdma client treats the bad format reply as a bcall, and

Doesn't this test return false:

1144 if (*p != cpu_to_be32(RPC_CALL))
1145 return false;

But OK: a malicious NFSv3 server can trigger a client crash. That's a bug.

This is an additional conditional in a hot path. Would it make sense to move the new test to just after 1145?

Even then, it could be a bcall, the client simply isn't prepared to process it. So "/* no bc service */" by itself would be a more accurate comment.

--
Chuck Lever

#4 lore raw
Quoted 60 lines · Chuck Lever III
Quoted 60 lines · Kinglong Mee

Hi Chuck,

Quoted 52 lines · Chuck Lever III
Quoted 36 lines · Kinglong Mee

When rdma server returns a fault reply, rpcrdma may treats it as a bcall. As using NFSv3, a bc server is never exist. rpcrdma_bc_receive_call will meets NULL pointer as,

[  226.057890] BUG: unable to handle kernel NULL pointer dereference at 00000000000000c8
...
[  226.058704] RIP: 0010:_raw_spin_lock+0xc/0x20
...
[  226.059732] Call Trace:
[  226.059878]  rpcrdma_bc_receive_call+0x138/0x327 [rpcrdma]
[  226.060011]  __ib_process_cq+0x89/0x170 [ib_core]
[  226.060092]  ib_cq_poll_work+0x26/0x80 [ib_core]
[  226.060257]  process_one_work+0x1a7/0x360
[  226.060367]  ? create_worker+0x1a0/0x1a0
[  226.060440]  worker_thread+0x30/0x390
[  226.060500]  ? create_worker+0x1a0/0x1a0
[  226.060574]  kthread+0x116/0x130
[  226.060661]  ? kthread_flush_work_fn+0x10/0x10
[  226.060724]  ret_from_fork+0x35/0x40

...

---
net/sunrpc/xprtrdma/rpc_rdma.c | 5 +++++

1 file changed, 5 insertions(+)
net/sunrpc/xprtrdma/rpc_rdma.c +0 -0
index 281ddb87ac8d..9486bb98eb2f 100644
@@ -1121,9 +1121,14 @@ static bool
 rpcrdma_is_bcall(struct rpcrdma_xprt *r_xprt, struct rpcrdma_rep *rep)
 #if defined(CONFIG_SUNRPC_BACKCHANNEL)
 {

net/sunrpc/xprtrdma/rpc_rdma.c static bool · line 1133

Quoted 10 lines
net/sunrpc/xprtrdma/rpc_rdma.c +5 -0
+	struct rpc_xprt *xprt = &r_xprt->rx_xprt;
 	struct xdr_stream *xdr = &rep->rr_stream;
 	__be32 *p;
 
+	/* no bc service, not a bcall. */
+	if (xprt->bc_serv == NULL)
+		return false;
+
 	if (rep->rr_proc != rdma_msg)
 		return false;

I'm not sure what you mean above by "fault reply".
The check here for whether the RPC/RDMA procedure is an RDMA_MSG is supposed to be enough to avoid any further processing of an RDMA_ERR type procedure.
What kind of fault has occurred? Can you share with us the actual RPC/RDMA transport header that triggers the BUG?

Quoted 8 lines
Quoted 3 lines

I have a nfs rdma server, but it handles drc reply wrongly sometimes, it returns a bad format reply to nfs client.
Nfs rdma client treats the bad format reply as a bcall, and

Doesn't this test return false:

1144 if (*p != cpu_to_be32(RPC_CALL))
1145 return false;

No, it doesn't return false here.

The debug message at rpcrdma_bc_receive_call are,

[56579.837169] RPC:       rpcrdma_bc_receive_call: callback XID 00000001, length=20
[56579.837174] RPC:       rpcrdma_bc_receive_call: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 04
Quoted 9 lines

But OK: a malicious NFSv3 server can trigger a client crash. That's a bug.

This is an additional conditional in a hot path. Would it make sense to move the new test to just after 1145? > Even then, it could be a bcall, the client simply isn't prepared to process it. So "/* no bc service */" by itself would be a more accurate comment.

Okay, I will update it and send a v2 patch after moving the checking after 1145.

Thanks,
Kinglong Mee

#5 lore raw

PATCH v2 1/1 xprtrdma: treat all calls not a bcall when bc_serv is NULL

In reply to Chuck Lever III

When a rdma server returns a fault format reply, nfs v3 client may treats it as a bcall when bc service is not exist.

The debug message at rpcrdma_bc_receive_call are,

[56579.837169] RPC:       rpcrdma_bc_receive_call: callback XID 00000001, length=20
[56579.837174] RPC:       rpcrdma_bc_receive_call: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 04

After that, rpcrdma_bc_receive_call will meets NULL pointer as,

[  226.057890] BUG: unable to handle kernel NULL pointer dereference at 00000000000000c8
...
[  226.058704] RIP: 0010:_raw_spin_lock+0xc/0x20
...
[  226.059732] Call Trace:
[  226.059878]  rpcrdma_bc_receive_call+0x138/0x327 [rpcrdma]
[  226.060011]  __ib_process_cq+0x89/0x170 [ib_core]
[  226.060092]  ib_cq_poll_work+0x26/0x80 [ib_core]
[  226.060257]  process_one_work+0x1a7/0x360
[  226.060367]  ? create_worker+0x1a0/0x1a0
[  226.060440]  worker_thread+0x30/0x390
[  226.060500]  ? create_worker+0x1a0/0x1a0
[  226.060574]  kthread+0x116/0x130
[  226.060661]  ? kthread_flush_work_fn+0x10/0x10
[  226.060724]  ret_from_fork+0x35/0x40

...

---
 net/sunrpc/xprtrdma/rpc_rdma.c | 5 +++++
 1 file changed, 5 insertions(+)
net/sunrpc/xprtrdma/rpc_rdma.c +5 -0
index 281ddb87ac8d..190a4de239c8 100644
@@ -1121,6 +1121,7 @@ static bool
 rpcrdma_is_bcall(struct rpcrdma_xprt *r_xprt, struct rpcrdma_rep *rep)
 #if defined(CONFIG_SUNRPC_BACKCHANNEL)
 {
+	struct rpc_xprt *xprt = &r_xprt->rx_xprt;
 	struct xdr_stream *xdr = &rep->rr_stream;
 	__be32 *p;
 
@@ -1144,6 +1145,10 @@ rpcrdma_is_bcall(struct rpcrdma_xprt *r_xprt, struct rpcrdma_rep *rep)
 	if (*p != cpu_to_be32(RPC_CALL))
 		return false;
 
+	/* No bc service. */
+	if (xprt->bc_serv == NULL)
+		return false;
+
 	/* Now that we are sure this is a backchannel call,
 	 * advance to the RPC header.
 	 */
Signature · 1 line
2.27.0
#6 lore raw

Re: [PATCH v2] xprtrdma: treat all calls not a bcall when bc_serv is NULL

Quoted 29 lines · Kinglong Mee

When a rdma server returns a fault format reply, nfs v3 client may treats it as a bcall when bc service is not exist.

The debug message at rpcrdma_bc_receive_call are,

[56579.837169] RPC:       rpcrdma_bc_receive_call: callback XID 00000001, length=20
[56579.837174] RPC:       rpcrdma_bc_receive_call: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 04

After that, rpcrdma_bc_receive_call will meets NULL pointer as,

[  226.057890] BUG: unable to handle kernel NULL pointer dereference at 00000000000000c8
...
[  226.058704] RIP: 0010:_raw_spin_lock+0xc/0x20
...
[  226.059732] Call Trace:
[  226.059878]  rpcrdma_bc_receive_call+0x138/0x327 [rpcrdma]
[  226.060011]  __ib_process_cq+0x89/0x170 [ib_core]
[  226.060092]  ib_cq_poll_work+0x26/0x80 [ib_core]
[  226.060257]  process_one_work+0x1a7/0x360
[  226.060367]  ? create_worker+0x1a0/0x1a0
[  226.060440]  worker_thread+0x30/0x390
[  226.060500]  ? create_worker+0x1a0/0x1a0
[  226.060574]  kthread+0x116/0x130
[  226.060661]  ? kthread_flush_work_fn+0x10/0x10
[  226.060724]  ret_from_fork+0x35/0x40
Quoted 3 lines

...

Patch is good, Reviewed-by: Chuck Lever <[email protected]>

Commentary (no changes to v2 needed):

The description still suggests we are adapting the client to work around a broken server. I don't feel this is the right reason to accept this change. Instead: we really don't want the client to be vulnerable to bad input for any reason. After this fix is applied, bad RPC messages are eventually dropped rather than processed, which is the desired behavior.

Anna, I recommend adding Cc: stable for this fix.

Kinglong, please ensure that client Receive resources are not leaked in this case. If they are, send along an additional patch; if not, let us know and we can close this issue. Thank you!

Quoted 30 lines

---
net/sunrpc/xprtrdma/rpc_rdma.c | 5 +++++

1 file changed, 5 insertions(+)
net/sunrpc/xprtrdma/rpc_rdma.c +5 -0
index 281ddb87ac8d..190a4de239c8 100644
@@ -1121,6 +1121,7 @@ static bool
 rpcrdma_is_bcall(struct rpcrdma_xprt *r_xprt, struct rpcrdma_rep *rep)
 #if defined(CONFIG_SUNRPC_BACKCHANNEL)
 {
+	struct rpc_xprt *xprt = &r_xprt->rx_xprt;
 	struct xdr_stream *xdr = &rep->rr_stream;
 	__be32 *p;
 
@@ -1144,6 +1145,10 @@ rpcrdma_is_bcall(struct rpcrdma_xprt *r_xprt, struct rpcrdma_rep *rep)
 	if (*p != cpu_to_be32(RPC_CALL))
 		return false;
 
+	/* No bc service. */
+	if (xprt->bc_serv == NULL)
+		return false;
+
 	/* Now that we are sure this is a backchannel call,
 	 * advance to the RPC header.
 	 */
Signature · 1 line
2.27.0

--
Chuck Lever

#7 lore raw

Re: [PATCH v2] xprtrdma: treat all calls not a bcall when bc_serv is NULL

Quoted 44 lines · Chuck Lever III
Quoted 29 lines · Kinglong Mee

When a rdma server returns a fault format reply, nfs v3 client may treats it as a bcall when bc service is not exist.

The debug message at rpcrdma_bc_receive_call are,

[56579.837169] RPC:       rpcrdma_bc_receive_call: callback XID 00000001, length=20
[56579.837174] RPC:       rpcrdma_bc_receive_call: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 04

After that, rpcrdma_bc_receive_call will meets NULL pointer as,

[  226.057890] BUG: unable to handle kernel NULL pointer dereference at 00000000000000c8
...
[  226.058704] RIP: 0010:_raw_spin_lock+0xc/0x20
...
[  226.059732] Call Trace:
[  226.059878]  rpcrdma_bc_receive_call+0x138/0x327 [rpcrdma]
[  226.060011]  __ib_process_cq+0x89/0x170 [ib_core]
[  226.060092]  ib_cq_poll_work+0x26/0x80 [ib_core]
[  226.060257]  process_one_work+0x1a7/0x360
[  226.060367]  ? create_worker+0x1a0/0x1a0
[  226.060440]  worker_thread+0x30/0x390
[  226.060500]  ? create_worker+0x1a0/0x1a0
[  226.060574]  kthread+0x116/0x130
[  226.060661]  ? kthread_flush_work_fn+0x10/0x10
[  226.060724]  ret_from_fork+0x35/0x40
Quoted 3 lines

...

Patch is good, Reviewed-by: Chuck Lever <[email protected]>

Commentary (no changes to v2 needed):

The description still suggests we are adapting the client to work around a broken server. I don't feel this is the right reason to accept this change. Instead: we really don't want the client to be vulnerable to bad input for any reason. After this fix is applied, bad RPC messages are eventually dropped rather than processed, which is the desired behavior.

Quoted 5 lines

Anna, I recommend adding Cc: stable for this fix.

Kinglong, please ensure that client Receive resources are not leaked in this case. If they are, send along an additional patch; if not, let us know and we can close this issue.

I have double check of the code.
I think client Receive resources are not leaked here. We can close it.

Thanks,
Kinglong Mee

Quoted 38 lines
Quoted 32 lines

---
net/sunrpc/xprtrdma/rpc_rdma.c | 5 +++++

1 file changed, 5 insertions(+)
net/sunrpc/xprtrdma/rpc_rdma.c +5 -0
index 281ddb87ac8d..190a4de239c8 100644
@@ -1121,6 +1121,7 @@ static bool
 rpcrdma_is_bcall(struct rpcrdma_xprt *r_xprt, struct rpcrdma_rep *rep)
 #if defined(CONFIG_SUNRPC_BACKCHANNEL)
 {
+	struct rpc_xprt *xprt = &r_xprt->rx_xprt;
 	struct xdr_stream *xdr = &rep->rr_stream;
 	__be32 *p;
 
@@ -1144,6 +1145,10 @@ rpcrdma_is_bcall(struct rpcrdma_xprt *r_xprt, struct rpcrdma_rep *rep)
 	if (*p != cpu_to_be32(RPC_CALL))
 		return false;
 
+	/* No bc service. */
+	if (xprt->bc_serv == NULL)
+		return false;
+
 	/* Now that we are sure this is a backchannel call,
 	 * advance to the RPC header.
 	 */
Signature · 1 line
2.27.0

--
Chuck Lever