Skip to contentSkip to conversation

svcrdma: fix bounce buffers for non-zero page offsets

PATCH In review Dan Aloni 3 participants 6 messages Oct 2, 2020 – Oct 15, 2020

Subsystems (4)
lore
Share & tools
lore

Dan Aloni

author
#0 lore raw

PATCH 1/1 svcrdma: fix bounce buffers for non-zero page offsets

This was discovered using O_DIRECT and small unaligned file offsets at the client side.

Fixes e248aa7be86 ("svcrdma: Remove max_sge check at connect time")
---
 net/sunrpc/xprtrdma/svc_rdma_sendto.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/sunrpc/xprtrdma/svc_rdma_sendto.c +1 -1
index 7b94d971feb3..c991eb1fd4e3 100644
@@ -638,7 +638,7 @@ static int svc_rdma_pull_up_reply_msg(struct svcxprt_rdma *rdma,
 		while (remaining) {
 			len = min_t(u32, PAGE_SIZE - pageoff, remaining);
 
-			memcpy(dst, page_address(*ppages), len);
+			memcpy(dst, page_address(*ppages) + pageoff, len);
 			remaining -= len;
 			dst += len;
 			pageoff = 0;
Signature · 1 line
2.26.2
#1 lore raw

Hi Dan-

Quoted 16 lines · Dan Aloni

This was discovered using O_DIRECT and small unaligned file offsets at the client side.

Fixes e248aa7be86 ("svcrdma: Remove max_sge check at connect time")

---
net/sunrpc/xprtrdma/svc_rdma_sendto.c | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)
net/sunrpc/xprtrdma/svc_rdma_sendto.c +0 -0
index 7b94d971feb3..c991eb1fd4e3 100644

net/sunrpc/xprtrdma/svc_rdma_sendto.c static int svc_rdma_pull_up_reply_msg(struct svcxprt_rdma *rdma, · line 641

Quoted 5 lines
net/sunrpc/xprtrdma/svc_rdma_sendto.c +1 -1
@@ -638,7 +638,7 @@ static int svc_rdma_pull_up_reply_msg(struct svcxprt_rdma *rdma,
 		while (remaining) {
 			len = min_t(u32, PAGE_SIZE - pageoff, remaining);
 
-			memcpy(dst, page_address(*ppages), len);
+			memcpy(dst, page_address(*ppages) + pageoff, len);

I'm assuming the only relevant place that sets xdr->page_base is nfsd_splice_actor() ?

Quoted 6 lines
net/sunrpc/xprtrdma/svc_rdma_sendto.c +0 -0
 			remaining -= len;
 			dst += len;
 			pageoff = 0;
Signature · 1 line
2.26.2

--
Chuck Lever

author
#2 lore raw
Quoted 21 lines · Chuck Lever

Hi Dan-

Quoted 19 lines · Dan Aloni

This was discovered using O_DIRECT and small unaligned file offsets at the client side.

Fixes e248aa7be86 ("svcrdma: Remove max_sge check at connect time")

---
net/sunrpc/xprtrdma/svc_rdma_sendto.c | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)
net/sunrpc/xprtrdma/svc_rdma_sendto.c +0 -0
index 7b94d971feb3..c991eb1fd4e3 100644
@@ -638,7 +638,7 @@ static int svc_rdma_pull_up_reply_msg(struct svcxprt_rdma *rdma,
 		while (remaining) {
 			len = min_t(u32, PAGE_SIZE - pageoff, remaining);
 
Quoted 5 lines

net/sunrpc/xprtrdma/svc_rdma_sendto.c static int svc_rdma_pull_up_reply_msg(struct svcxprt_rdma *rdma, · line 641

Quoted 2 lines
net/sunrpc/xprtrdma/svc_rdma_sendto.c +1 -1
-			memcpy(dst, page_address(*ppages), len);
+			memcpy(dst, page_address(*ppages) + pageoff, len);

I'm assuming the only relevant place that sets xdr->page_base is nfsd_splice_actor() ?

Yes, and traces at the server indeed indicate that splicing happened. This works for both tmpfs and ext4 as host FSes.

Signature · 1 line
Dan Aloni
#3 lore raw
Quoted 26 lines · Dan Aloni
Quoted 21 lines · Chuck Lever

Hi Dan-

Quoted 19 lines · Dan Aloni

This was discovered using O_DIRECT and small unaligned file offsets at the client side.

Fixes e248aa7be86 ("svcrdma: Remove max_sge check at connect time")

---
net/sunrpc/xprtrdma/svc_rdma_sendto.c | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)
net/sunrpc/xprtrdma/svc_rdma_sendto.c +0 -0
index 7b94d971feb3..c991eb1fd4e3 100644
@@ -638,7 +638,7 @@ static int svc_rdma_pull_up_reply_msg(struct svcxprt_rdma *rdma,
 		while (remaining) {
 			len = min_t(u32, PAGE_SIZE - pageoff, remaining);
 
Quoted 8 lines
Quoted 5 lines

net/sunrpc/xprtrdma/svc_rdma_sendto.c static int svc_rdma_pull_up_reply_msg(struct svcxprt_rdma *rdma, · line 641

Quoted 2 lines
net/sunrpc/xprtrdma/svc_rdma_sendto.c +1 -1
-			memcpy(dst, page_address(*ppages), len);
+			memcpy(dst, page_address(*ppages) + pageoff, len);

I'm assuming the only relevant place that sets xdr->page_base is nfsd_splice_actor() ?

Yes, and traces at the server indeed indicate that splicing happened. This works for both tmpfs and ext4 as host FSes.

Seems plausible, thanks. I'll throw it into my test environment.

--
Chuck Lever

author
#4 lore raw

PATCH v2 1/1 svcrdma: fix bounce buffers for unaligned offsets and multiple pages

This was discovered using O_DIRECT at the client side, with small unaligned file offsets or IOs that span multiple file pages.

Fixes e248aa7be86 ("svcrdma: Remove max_sge check at connect time")
---
 net/sunrpc/xprtrdma/svc_rdma_sendto.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Extended testing found another issue with the loop.

net/sunrpc/xprtrdma/svc_rdma_sendto.c +2 -1
index 7b94d971feb3..c3d588b149aa 100644
@@ -638,10 +638,11 @@ static int svc_rdma_pull_up_reply_msg(struct svcxprt_rdma *rdma,
 		while (remaining) {
 			len = min_t(u32, PAGE_SIZE - pageoff, remaining);
 
-			memcpy(dst, page_address(*ppages), len);
+			memcpy(dst, page_address(*ppages) + pageoff, len);
 			remaining -= len;
 			dst += len;
 			pageoff = 0;
+			ppages++;
 		}
 	}
 
Signature · 1 line
2.26.2
#5 lore raw

Re: [PATCH v2] svcrdma: fix bounce buffers for unaligned offsets and multiple pages

Thanks, applying for 5.10.--b.

Quoted 30 lines · Dan Aloni

This was discovered using O_DIRECT at the client side, with small unaligned file offsets or IOs that span multiple file pages.

Fixes e248aa7be86 ("svcrdma: Remove max_sge check at connect time")
---
 net/sunrpc/xprtrdma/svc_rdma_sendto.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Extended testing found another issue with the loop.

net/sunrpc/xprtrdma/svc_rdma_sendto.c +2 -1
index 7b94d971feb3..c3d588b149aa 100644
@@ -638,10 +638,11 @@ static int svc_rdma_pull_up_reply_msg(struct svcxprt_rdma *rdma,
 		while (remaining) {
 			len = min_t(u32, PAGE_SIZE - pageoff, remaining);
 
-			memcpy(dst, page_address(*ppages), len);
+			memcpy(dst, page_address(*ppages) + pageoff, len);
 			remaining -= len;
 			dst += len;
 			pageoff = 0;
+			ppages++;
 		}
 	}
 
Signature · 1 line
2.26.2