svcrdma: Advertise the correct max payload

PATCH Superseded Steve Wise 3 participants 7 messages Sep 22, 2014

Subsystems (4)

Networking NFS client NFS server InfiniBand

lore

Share & tools

lore

Steve Wise

author

2014-09-22 18:36 UTC #0 lore raw

PATCH 1/1 svcrdma: Advertise the correct max payload

Svcrdma currently advertises 1MB, which is too large. The correct value is the max scatter-gather allowed in an NFSRDMA IO chunk * the host page size. This bug is usually benign because the Linux X64 NFSRDMA client correctly limits the payload size to the correct value (64*4096 = 256KB). But if the Linux client is PPC64 with a 64KB page size, then the client will indeed use a payload size that will overflow the server.

Signed-off-by Steve Wise <[email protected]>

---

 net/sunrpc/xprtrdma/svc_rdma_transport.c |    2 +-
 net/sunrpc/xprtrdma/xprt_rdma.h          |    2 ++
 2 files changed, 3 insertions(+), 1 deletions(-)

net/sunrpc/xprtrdma/svc_rdma_transport.c +1 -1

index 374feb4..4e61880 100644
@@ -91,7 +91,7 @@ struct svc_xprt_class svc_rdma_class = {
 	.xcl_name = "rdma",
 	.xcl_owner = THIS_MODULE,
 	.xcl_ops = &svc_rdma_ops,
-	.xcl_max_payload = RPCSVC_MAXPAYLOAD_TCP,
+	.xcl_max_payload = RPCSVC_MAXPAYLOAD_RDMA,
 	.xcl_ident = XPRT_TRANSPORT_RDMA,
 };

net/sunrpc/xprtrdma/xprt_rdma.h +2 -0

index c419498..467a77c 100644
@@ -392,4 +392,6 @@ extern struct kmem_cache *svc_rdma_ctxt_cachep;
 /* Workqueue created in svc_rdma.c */
 extern struct workqueue_struct *svc_rdma_wq;
 
+#define RPCSVC_MAXPAYLOAD_RDMA	(RPCRDMA_MAX_DATA_SEGS << PAGE_SHIFT)
+
 #endif				/* _LINUX_SUNRPC_XPRT_RDMA_H */

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

J. Bruce Fields

2014-09-22 18:39 UTC #1 lore raw

Quoted 32 lines · Steve Wise

Svcrdma currently advertises 1MB, which is too large. The correct value is the max scatter-gather allowed in an NFSRDMA IO chunk * the host page size. This bug is usually benign because the Linux X64 NFSRDMA client correctly limits the payload size to the correct value (64*4096 = 256KB). But if the Linux client is PPC64 with a 64KB page size, then the client will indeed use a payload size that will overflow the server.
Signed-off-by Steve Wise <[email protected]>
---
 net/sunrpc/xprtrdma/svc_rdma_transport.c |    2 +-
 net/sunrpc/xprtrdma/xprt_rdma.h          |    2 ++
 2 files changed, 3 insertions(+), 1 deletions(-)
net/sunrpc/xprtrdma/svc_rdma_transport.c +1 -1
index 374feb4..4e61880 100644
@@ -91,7 +91,7 @@ struct svc_xprt_class svc_rdma_class = {
 	.xcl_name = "rdma",
 	.xcl_owner = THIS_MODULE,
 	.xcl_ops = &svc_rdma_ops,
-	.xcl_max_payload = RPCSVC_MAXPAYLOAD_TCP,
+	.xcl_max_payload = RPCSVC_MAXPAYLOAD_RDMA,
 	.xcl_ident = XPRT_TRANSPORT_RDMA,
 };
 
net/sunrpc/xprtrdma/xprt_rdma.h +0 -0
index c419498..467a77c 100644

net/sunrpc/xprtrdma/xprt_rdma.h extern struct kmem_cache *svc_rdma_ctxt_cachep; · line 395

Quoted 4 lines

net/sunrpc/xprtrdma/xprt_rdma.h +1 -0

@@ -392,4 +392,6 @@ extern struct kmem_cache *svc_rdma_ctxt_cachep;
 /* Workqueue created in svc_rdma.c */
 extern struct workqueue_struct *svc_rdma_wq;
 
+#define RPCSVC_MAXPAYLOAD_RDMA	(RPCRDMA_MAX_DATA_SEGS << PAGE_SHIFT)

Do you want to define this as the minimum of this and RPCSVC_MAXPAYLOAD_TCP, in case RPCRDMA_MAX_DATA_SEGS gets increased some day?

--b.

net/sunrpc/xprtrdma/xprt_rdma.h extern struct kmem_cache *svc_rdma_ctxt_cachep; · line 397

Quoted 3 lines

net/sunrpc/xprtrdma/xprt_rdma.h +1 -0
+
 #endif				/* _LINUX_SUNRPC_XPRT_RDMA_H */

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Steve Wise

author

2014-09-22 18:42 UTC #2 lore raw

Quoted 36 lines · J. Bruce Fields

Quoted 35 lines · Steve Wise
Svcrdma currently advertises 1MB, which is too large. The correct value is the max scatter-gather allowed in an NFSRDMA IO chunk * the host page size. This bug is usually benign because the Linux X64 NFSRDMA client correctly limits the payload size to the correct value (64*4096 = 256KB). But if the Linux client is PPC64 with a 64KB page size, then the client will indeed use a payload size that will overflow the server.
Signed-off-by Steve Wise <[email protected]>
---
  net/sunrpc/xprtrdma/svc_rdma_transport.c |    2 +-
  net/sunrpc/xprtrdma/xprt_rdma.h          |    2 ++
  2 files changed, 3 insertions(+), 1 deletions(-)
net/sunrpc/xprtrdma/svc_rdma_transport.c +1 -1
index 374feb4..4e61880 100644
@@ -91,7 +91,7 @@ struct svc_xprt_class svc_rdma_class = {
  	.xcl_name = "rdma",
  	.xcl_owner = THIS_MODULE,
  	.xcl_ops = &svc_rdma_ops,
-	.xcl_max_payload = RPCSVC_MAXPAYLOAD_TCP,
+	.xcl_max_payload = RPCSVC_MAXPAYLOAD_RDMA,
  	.xcl_ident = XPRT_TRANSPORT_RDMA,
  };
 
net/sunrpc/xprtrdma/xprt_rdma.h +0 -0
index c419498..467a77c 100644
@@ -392,4 +392,6 @@ extern struct kmem_cache *svc_rdma_ctxt_cachep;
  /* Workqueue created in svc_rdma.c */
  extern struct workqueue_struct *svc_rdma_wq;
 

Quoted 4 lines

net/sunrpc/xprtrdma/xprt_rdma.h extern struct kmem_cache *svc_rdma_ctxt_cachep; · line 395
Quoted 1 line
net/sunrpc/xprtrdma/xprt_rdma.h +1 -0
+#define RPCSVC_MAXPAYLOAD_RDMA	(RPCRDMA_MAX_DATA_SEGS << PAGE_SHIFT)
Do you want to define this as the minimum of this and RPCSVC_MAXPAYLOAD_TCP, in case RPCRDMA_MAX_DATA_SEGS gets increased some day?

Why would it need to be limited by MAXPAYLOAD_TCP?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

J. Bruce Fields

2014-09-22 18:55 UTC #3 lore raw

Quoted 40 lines · Steve Wise

Quoted 30 lines · J. Bruce Fields
Quoted 29 lines · Steve Wise
Svcrdma currently advertises 1MB, which is too large. The correct value is the max scatter-gather allowed in an NFSRDMA IO chunk * the host page size. This bug is usually benign because the Linux X64 NFSRDMA client correctly limits the payload size to the correct value (64*4096 = 256KB). But if the Linux client is PPC64 with a 64KB page size, then the client will indeed use a payload size that will overflow the server.
Signed-off-by Steve Wise <[email protected]>
---
 net/sunrpc/xprtrdma/svc_rdma_transport.c |    2 +-
 net/sunrpc/xprtrdma/xprt_rdma.h          |    2 ++
 2 files changed, 3 insertions(+), 1 deletions(-)
net/sunrpc/xprtrdma/svc_rdma_transport.c +1 -1
index 374feb4..4e61880 100644
@@ -91,7 +91,7 @@ struct svc_xprt_class svc_rdma_class = {
 	.xcl_name = "rdma",
 	.xcl_owner = THIS_MODULE,
 	.xcl_ops = &svc_rdma_ops,
-	.xcl_max_payload = RPCSVC_MAXPAYLOAD_TCP,
+	.xcl_max_payload = RPCSVC_MAXPAYLOAD_RDMA,
 	.xcl_ident = XPRT_TRANSPORT_RDMA,
 };
net/sunrpc/xprtrdma/xprt_rdma.h +0 -1
index c419498..467a77c 100644
--- a/net/sunrpc/xprtrdma/xprt_rdma.h
Quoted 8 lines
net/sunrpc/xprtrdma/xprt_rdma.h extern struct kmem_cache *svc_rdma_ctxt_cachep; · line 394
Quoted 5 lines
+++ b/net/sunrpc/xprtrdma/xprt_rdma.h
net/sunrpc/xprtrdma/xprt_rdma.h +1 -0
@@ -392,4 +392,6 @@ extern struct kmem_cache *svc_rdma_ctxt_cachep;
 /* Workqueue created in svc_rdma.c */
 extern struct workqueue_struct *svc_rdma_wq;
+#define RPCSVC_MAXPAYLOAD_RDMA	(RPCRDMA_MAX_DATA_SEGS << PAGE_SHIFT)
Do you want to define this as the minimum of this and RPCSVC_MAXPAYLOAD_TCP, in case RPCRDMA_MAX_DATA_SEGS gets increased some day?

Quoted 1 line

Why would it need to be limited by MAXPAYLOAD_TCP?

Because you're also limited by the size of the rq_pages array, which is determined by RPCSVC_MAXPAGES, calculated from RPCSVC_MAXPAYLOAD.

(Actually you probably want RPCSVC_MAXPAYLOAD, not MAXPAYLOAD_TCP.)

--b.

Steve Wise

author

2014-09-22 19:12 UTC #4 lore raw

Quoted 40 lines · J. Bruce Fields

Quoted 39 lines · Steve Wise
Quoted 38 lines · J. Bruce Fields
Quoted 31 lines · Steve Wise
Svcrdma currently advertises 1MB, which is too large. The correct value is the max scatter-gather allowed in an NFSRDMA IO chunk * the host page size. This bug is usually benign because the Linux X64 NFSRDMA client correctly limits the payload size to the correct value (64*4096 = 256KB). But if the Linux client is PPC64 with a 64KB page size, then the client will indeed use a payload size that will overflow the server.
Signed-off-by Steve Wise <[email protected]>
---
  net/sunrpc/xprtrdma/svc_rdma_transport.c |    2 +-
  net/sunrpc/xprtrdma/xprt_rdma.h          |    2 ++
  2 files changed, 3 insertions(+), 1 deletions(-)
net/sunrpc/xprtrdma/svc_rdma_transport.c +1 -1
index 374feb4..4e61880 100644
@@ -91,7 +91,7 @@ struct svc_xprt_class svc_rdma_class = {
  	.xcl_name = "rdma",
  	.xcl_owner = THIS_MODULE,
  	.xcl_ops = &svc_rdma_ops,
-	.xcl_max_payload = RPCSVC_MAXPAYLOAD_TCP,
+	.xcl_max_payload = RPCSVC_MAXPAYLOAD_RDMA,
  	.xcl_ident = XPRT_TRANSPORT_RDMA,
  };
net/sunrpc/xprtrdma/xprt_rdma.h +0 -0
index c419498..467a77c 100644
net/sunrpc/xprtrdma/xprt_rdma.h extern struct kmem_cache *svc_rdma_ctxt_cachep; · line 394
Quoted 3 lines
net/sunrpc/xprtrdma/xprt_rdma.h +1 -0
@@ -392,4 +392,6 @@ extern struct kmem_cache *svc_rdma_ctxt_cachep;
  /* Workqueue created in svc_rdma.c */
  extern struct workqueue_struct *svc_rdma_wq;
+#define RPCSVC_MAXPAYLOAD_RDMA	(RPCRDMA_MAX_DATA_SEGS << PAGE_SHIFT)
Do you want to define this as the minimum of this and RPCSVC_MAXPAYLOAD_TCP, in case RPCRDMA_MAX_DATA_SEGS gets increased some day?

Quoted 7 lines

Quoted 1 line
Why would it need to be limited by MAXPAYLOAD_TCP?
Because you're also limited by the size of the rq_pages array, which is determined by RPCSVC_MAXPAGES, calculated from RPCSVC_MAXPAYLOAD.
(Actually you probably want RPCSVC_MAXPAYLOAD, not MAXPAYLOAD_TCP.)

I see. I agree.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Chuck Lever

2014-09-22 19:16 UTC #5 lore raw

In reply to J. Bruce Fields

Quoted 41 lines · J. Bruce Fields

Quoted 40 lines · Steve Wise
Quoted 38 lines · J. Bruce Fields
Quoted 31 lines · Steve Wise
Svcrdma currently advertises 1MB, which is too large. The correct value is the max scatter-gather allowed in an NFSRDMA IO chunk * the host page size. This bug is usually benign because the Linux X64 NFSRDMA client correctly limits the payload size to the correct value (64*4096 = 256KB). But if the Linux client is PPC64 with a 64KB page size, then the client will indeed use a payload size that will overflow the server.
Signed-off-by Steve Wise <[email protected]>
---
net/sunrpc/xprtrdma/svc_rdma_transport.c |    2 +-
net/sunrpc/xprtrdma/xprt_rdma.h          |    2 ++
2 files changed, 3 insertions(+), 1 deletions(-)
net/sunrpc/xprtrdma/svc_rdma_transport.c +1 -1
index 374feb4..4e61880 100644
@@ -91,7 +91,7 @@ struct svc_xprt_class svc_rdma_class = {
 	.xcl_name = "rdma",
 	.xcl_owner = THIS_MODULE,
 	.xcl_ops = &svc_rdma_ops,
-	.xcl_max_payload = RPCSVC_MAXPAYLOAD_TCP,
+	.xcl_max_payload = RPCSVC_MAXPAYLOAD_RDMA,
 	.xcl_ident = XPRT_TRANSPORT_RDMA,
 };
net/sunrpc/xprtrdma/xprt_rdma.h +0 -0
index c419498..467a77c 100644
net/sunrpc/xprtrdma/xprt_rdma.h extern struct kmem_cache *svc_rdma_ctxt_cachep; · line 394
Quoted 3 lines
net/sunrpc/xprtrdma/xprt_rdma.h +1 -0
@@ -392,4 +392,6 @@ extern struct kmem_cache *svc_rdma_ctxt_cachep;
 /* Workqueue created in svc_rdma.c */
 extern struct workqueue_struct *svc_rdma_wq;
+#define RPCSVC_MAXPAYLOAD_RDMA	(RPCRDMA_MAX_DATA_SEGS << PAGE_SHIFT)
Do you want to define this as the minimum of this and RPCSVC_MAXPAYLOAD_TCP, in case RPCRDMA_MAX_DATA_SEGS gets increased some day?

Quoted 6 lines

Quoted 1 line
Why would it need to be limited by MAXPAYLOAD_TCP?
Because you're also limited by the size of the rq_pages array, which is determined by RPCSVC_MAXPAGES, calculated from RPCSVC_MAXPAYLOAD.
(Actually you probably want RPCSVC_MAXPAYLOAD, not MAXPAYLOAD_TCP.)

Agree with this last bit.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Chuck Lever

2014-09-22 18:47 UTC #6 lore raw

In reply to J. Bruce Fields

Quoted 36 lines · J. Bruce Fields

Quoted 35 lines · Steve Wise
Svcrdma currently advertises 1MB, which is too large. The correct value is the max scatter-gather allowed in an NFSRDMA IO chunk * the host page size. This bug is usually benign because the Linux X64 NFSRDMA client correctly limits the payload size to the correct value (64*4096 = 256KB). But if the Linux client is PPC64 with a 64KB page size, then the client will indeed use a payload size that will overflow the server.
Signed-off-by Steve Wise <[email protected]>
---
net/sunrpc/xprtrdma/svc_rdma_transport.c |    2 +-
net/sunrpc/xprtrdma/xprt_rdma.h          |    2 ++
2 files changed, 3 insertions(+), 1 deletions(-)
net/sunrpc/xprtrdma/svc_rdma_transport.c +1 -1
index 374feb4..4e61880 100644
@@ -91,7 +91,7 @@ struct svc_xprt_class svc_rdma_class = {
 	.xcl_name = "rdma",
 	.xcl_owner = THIS_MODULE,
 	.xcl_ops = &svc_rdma_ops,
-	.xcl_max_payload = RPCSVC_MAXPAYLOAD_TCP,
+	.xcl_max_payload = RPCSVC_MAXPAYLOAD_RDMA,
 	.xcl_ident = XPRT_TRANSPORT_RDMA,
 };
 
net/sunrpc/xprtrdma/xprt_rdma.h +0 -0
index c419498..467a77c 100644
@@ -392,4 +392,6 @@ extern struct kmem_cache *svc_rdma_ctxt_cachep;
 /* Workqueue created in svc_rdma.c */
 extern struct workqueue_struct *svc_rdma_wq;
 

Quoted 5 lines

net/sunrpc/xprtrdma/xprt_rdma.h extern struct kmem_cache *svc_rdma_ctxt_cachep; · line 395
Quoted 1 line
net/sunrpc/xprtrdma/xprt_rdma.h +1 -0
+#define RPCSVC_MAXPAYLOAD_RDMA	(RPCRDMA_MAX_DATA_SEGS << PAGE_SHIFT)
Do you want to define this as the minimum of this and RPCSVC_MAXPAYLOAD_TCP, in case RPCRDMA_MAX_DATA_SEGS gets increased some day?

Hey Bruce-

I don’t see the _TCP and _RDMA settings are related.

Quoted 9 lines

--b.
net/sunrpc/xprtrdma/xprt_rdma.h extern struct kmem_cache *svc_rdma_ctxt_cachep; · line 397
Quoted 3 lines
net/sunrpc/xprtrdma/xprt_rdma.h +1 -0
+
 #endif				/* _LINUX_SUNRPC_XPRT_RDMA_H */
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html