Skip to contentSkip to conversation

NFS: Fix handling of reply page vector

PATCH In review Chuck Lever 3 participants 6 messages Apr 8, 2019

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 NFS: Fix handling of reply page vector

NFSv4 GETACL and FS_LOCATIONS requests stopped working in v5.1-rc.

These two need the extra padding to be added directly to the reply length.

Fixes 02ef04e432ba ("NFS: Account for XDR pad of buf->pages")
---
 fs/nfs/nfs4xdr.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
fs/nfs/nfs4xdr.c +2 -2
index cfcabc3..6024461 100644
@@ -2589,7 +2589,7 @@ static void nfs4_xdr_enc_getacl(struct rpc_rqst *req, struct xdr_stream *xdr,
 			ARRAY_SIZE(nfs4_acl_bitmap), &hdr);
 
 	rpc_prepare_reply_pages(req, args->acl_pages, 0,
-				args->acl_len, replen);
+				args->acl_len, replen + 1);
 	encode_nops(&hdr);
 }
 
@@ -2811,7 +2811,7 @@ static void nfs4_xdr_enc_fs_locations(struct rpc_rqst *req,
 	}
 
 	rpc_prepare_reply_pages(req, (struct page **)&args->page, 0,
-				PAGE_SIZE, replen);
+				PAGE_SIZE, replen + 1);
 	encode_nops(&hdr);
 }
 
#1 lore raw

Hi Chuck,

Tested ACL and FS_LOCATION and it works.

Quoted 36 lines · Chuck Lever

NFSv4 GETACL and FS_LOCATIONS requests stopped working in v5.1-rc.

These two need the extra padding to be added directly to the reply length.

Fixes 02ef04e432ba ("NFS: Account for XDR pad of buf->pages")
---
 fs/nfs/nfs4xdr.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
fs/nfs/nfs4xdr.c +2 -2
index cfcabc3..6024461 100644
@@ -2589,7 +2589,7 @@ static void nfs4_xdr_enc_getacl(struct rpc_rqst *req, struct xdr_stream *xdr,
                        ARRAY_SIZE(nfs4_acl_bitmap), &hdr);
 
        rpc_prepare_reply_pages(req, args->acl_pages, 0,
-                               args->acl_len, replen);
+                               args->acl_len, replen + 1);
        encode_nops(&hdr);
 }
 
@@ -2811,7 +2811,7 @@ static void nfs4_xdr_enc_fs_locations(struct rpc_rqst *req,
        }
 
        rpc_prepare_reply_pages(req, (struct page **)&args->page, 0,
-                               PAGE_SIZE, replen);
+                               PAGE_SIZE, replen + 1);
        encode_nops(&hdr);
 }
 
#2 lore raw
In reply to Chuck Lever
Quoted 28 lines · Chuck Lever

NFSv4 GETACL and FS_LOCATIONS requests stopped working in v5.1-rc.

These two need the extra padding to be added directly to the reply length.

Fixes 02ef04e432ba ("NFS: Account for XDR pad of buf->pages")
---
 fs/nfs/nfs4xdr.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
fs/nfs/nfs4xdr.c +1 -1
index cfcabc3..6024461 100644
@@ -2589,7 +2589,7 @@ static void nfs4_xdr_enc_getacl(struct rpc_rqst
 *req, struct xdr_stream *xdr,
 			ARRAY_SIZE(nfs4_acl_bitmap), &hdr);
 
 	rpc_prepare_reply_pages(req, args->acl_pages, 0,
-				args->acl_len, replen);
+				args->acl_len, replen + 1);
 	encode_nops(&hdr);
 }

fs/nfs/nfs4xdr.c static void nfs4_xdr_enc_fs_locations(struct · line 2817

Quoted 8 lines
fs/nfs/nfs4xdr.c +1 -1
@@ -2811,7 +2811,7 @@ static void nfs4_xdr_enc_fs_locations(struct
 rpc_rqst *req,
 	}
 
 	rpc_prepare_reply_pages(req, (struct page **)&args->page, 0,
-				PAGE_SIZE, replen);
+				PAGE_SIZE, replen + 1);
 	encode_nops(&hdr);
 }

I'm having trouble with the math here. Why are we pre-emptively subtracting a word from the tail? The header constants are always 4-bit aligned because they are calculated as a word count, so I'm not understanding why we need commit 02ef04e432ba at all.

Can you please explain, Chuck?

Signature · 3 lines
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
#3 lore raw
Quoted 47 lines · Trond Myklebust
Quoted 28 lines · Chuck Lever

NFSv4 GETACL and FS_LOCATIONS requests stopped working in v5.1-rc.

These two need the extra padding to be added directly to the reply length.

Fixes 02ef04e432ba ("NFS: Account for XDR pad of buf->pages")

---
fs/nfs/nfs4xdr.c | 4 ++--

1 file changed, 2 insertions(+), 2 deletions(-)
fs/nfs/nfs4xdr.c +1 -1
index cfcabc3..6024461 100644
@@ -2589,7 +2589,7 @@ static void nfs4_xdr_enc_getacl(struct rpc_rqst
 *req, struct xdr_stream *xdr,
 			ARRAY_SIZE(nfs4_acl_bitmap), &hdr);
 
 	rpc_prepare_reply_pages(req, args->acl_pages, 0,
-				args->acl_len, replen);
+				args->acl_len, replen + 1);
 	encode_nops(&hdr);
 }

fs/nfs/nfs4xdr.c static void nfs4_xdr_enc_fs_locations(struct · line 2817

Quoted 8 lines
fs/nfs/nfs4xdr.c +1 -1
@@ -2811,7 +2811,7 @@ static void nfs4_xdr_enc_fs_locations(struct
 rpc_rqst *req,
 	}
 
 	rpc_prepare_reply_pages(req, (struct page **)&args->page, 0,
-				PAGE_SIZE, replen);
+				PAGE_SIZE, replen + 1);
 	encode_nops(&hdr);
 }

I'm having trouble with the math here. Why are we pre-emptively subtracting a word from the tail? The header constants are always 4-bit aligned because they are calculated as a word count, so I'm not understanding why we need commit 02ef04e432ba at all.

Quoted 1 line

Can you please explain, Chuck?

The goal is to allocate a reply buffer that is just large enough to fit the expected reply, and ensure that the variable-length payload will start exactly where the xdr_buf's pages begin.

In cases where the payload length is not aligned to four bytes, an extra quad has to be allocated in the tail. So, the total reply length is increased by one quad so there is enough space for the XDR pad bytes in the tail.

--
Chuck Lever

#4 lore raw
Quoted 53 lines · Chuck Lever
Quoted 51 lines

On Apr 8, 2019, at 4:38 PM, Trond Myklebust <
[email protected]> wrote:

Quoted 37 lines · Chuck Lever

NFSv4 GETACL and FS_LOCATIONS requests stopped working in v5.1- rc.

These two need the extra padding to be added directly to the reply
length.

Fixes 02ef04e432ba ("NFS: Account for XDR pad of buf->pages")

---
fs/nfs/nfs4xdr.c | 4 ++--

1 file changed, 2 insertions(+), 2 deletions(-)
fs/nfs/nfs4xdr.c +0 -0
index cfcabc3..6024461 100644
@@ -2589,7 +2589,7 @@ static void nfs4_xdr_enc_getacl(struct

rpc_rqst
*req, struct xdr_stream *xdr,

			ARRAY_SIZE(nfs4_acl_bitmap), &hdr);

	rpc_prepare_reply_pages(req, args->acl_pages, 0,
fs/nfs/nfs4xdr.c +1 -1
-				args->acl_len, replen);
+				args->acl_len, replen + 1);
	encode_nops(&hdr);
}
fs/nfs/nfs4xdr.c +0 -0
@@ -2811,7 +2811,7 @@ static void

nfs4_xdr_enc_fs_locations(struct
rpc_rqst *req,

	}

	rpc_prepare_reply_pages(req, (struct page **)&args->page, 0,
fs/nfs/nfs4xdr.c +1 -1
-				PAGE_SIZE, replen);
+				PAGE_SIZE, replen + 1);
Quoted 3 lines
	encode_nops(&hdr);
}

I'm having trouble with the math here. Why are we pre-emptively subtracting a word from the tail? The header constants are always 4-bit
aligned because they are calculated as a word count, so I'm not understanding why we need commit 02ef04e432ba at all.

Quoted 1 line

Can you please explain, Chuck?

Quoted 9 lines

The goal is to allocate a reply buffer that is just large enough to fit the expected reply, and ensure that the variable-length payload will start exactly where the xdr_buf's pages begin.

In cases where the payload length is not aligned to four bytes, an extra quad has to be allocated in the tail. So, the total reply length is increased by one quad so there is enough space for the XDR pad bytes in the tail.

Right, but we should never hit that problem because the proc->p_arglen and proc->p_replen are always in units of 32-bit words.

IOW: the functions that allocate memory, will always do so in full words, hence it should not be necessary for xdr_inline_pages() to adjust that allocation.

The one thing that we _might_ want to do if we're to do anything at all is to perhaps adjust tail->iov_base by (xdr->page_len & 3) bytes to ensure that we have word-aligned data in the tail.

i.e. capture the padding in the remaining bytes in that first word, so that xdr_read_pages() sets word aligned values for xdr->p and xdr->end.

Signature · 3 lines
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
#5 lore raw
Quoted 62 lines · Trond Myklebust
Quoted 57 lines · Chuck Lever
Quoted 51 lines

On Apr 8, 2019, at 4:38 PM, Trond Myklebust <
[email protected]> wrote:

Quoted 37 lines · Chuck Lever

NFSv4 GETACL and FS_LOCATIONS requests stopped working in v5.1- rc.

These two need the extra padding to be added directly to the reply
length.

Fixes 02ef04e432ba ("NFS: Account for XDR pad of buf->pages")

---
fs/nfs/nfs4xdr.c | 4 ++--

1 file changed, 2 insertions(+), 2 deletions(-)
fs/nfs/nfs4xdr.c +0 -0
index cfcabc3..6024461 100644
@@ -2589,7 +2589,7 @@ static void nfs4_xdr_enc_getacl(struct

rpc_rqst
*req, struct xdr_stream *xdr,

			ARRAY_SIZE(nfs4_acl_bitmap), &hdr);

	rpc_prepare_reply_pages(req, args->acl_pages, 0,
fs/nfs/nfs4xdr.c +1 -1
-				args->acl_len, replen);
+				args->acl_len, replen + 1);
	encode_nops(&hdr);
}
fs/nfs/nfs4xdr.c +0 -0
@@ -2811,7 +2811,7 @@ static void

nfs4_xdr_enc_fs_locations(struct
rpc_rqst *req,

	}

	rpc_prepare_reply_pages(req, (struct page **)&args->page, 0,
fs/nfs/nfs4xdr.c +1 -1
-				PAGE_SIZE, replen);
+				PAGE_SIZE, replen + 1);
Quoted 3 lines
	encode_nops(&hdr);
}

I'm having trouble with the math here. Why are we pre-emptively subtracting a word from the tail? The header constants are always 4-bit
aligned because they are calculated as a word count, so I'm not understanding why we need commit 02ef04e432ba at all.

Quoted 1 line

Can you please explain, Chuck?

The goal is to allocate a reply buffer that is just large enough to fit the expected reply, and ensure that the variable-length payload will start exactly where the xdr_buf's pages begin.

Quoted 7 lines
Quoted 5 lines

In cases where the payload length is not aligned to four bytes, an extra quad has to be allocated in the tail. So, the total reply length is increased by one quad so there is enough space for the XDR pad bytes in the tail.

Right, but we should never hit that problem because the proc->p_arglen and proc->p_replen are always in units of 32-bit words.

But buf->pages aren't. The point is to accommodate XDR padding of the data payload, which goes into the page cache. You want just the data to go into the page cache, and the padding to land in the tail.

Quoted 10 lines

IOW: the functions that allocate memory, will always do so in full words, hence it should not be necessary for xdr_inline_pages() to adjust that allocation.

The one thing that we _might_ want to do if we're to do anything at all is to perhaps adjust tail->iov_base by (xdr->page_len & 3) bytes to ensure that we have word-aligned data in the tail.

i.e. capture the padding in the remaining bytes in that first word, so that xdr_read_pages() sets word aligned values for xdr->p and xdr->end.

I'm in favor of hiding the details of that in the generic XDR code, and I probably should have taken that step in 02ef04e432ba.

So I think we should add a quad to the tail whenever the expected size of the data payload (ie, payload that goes into the page cache) is going to be non-word aligned. Otherwise the tail doesn't need that space. In those cases, the tail can be eliminated.

The reason to do that is that then the RDMA transport does not need to register a 4-byte memory region for the XDR padding.

--
Chuck Lever