Skip to contentSkip to conversation

ceph: misc fix size truncate for fscrypt

PATCH Superseded [email protected] 2 participants 10 messages Apr 7, 2022 Ceph client

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover ceph: misc fix size truncate for fscrypt

From: Xiubo Li <[email protected]>

Xiubo Li (2):
ceph: flush small range instead of the whole map for truncate
ceph: fix coherency issue when truncating file size for fscrypt

 fs/ceph/inode.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)
Signature · 1 line
2.27.0
#1 lore raw

PATCH 1/2 ceph: flush small range instead of the whole map for truncate

From: Xiubo Li <[email protected]>

---
 fs/ceph/inode.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
fs/ceph/inode.c +6 -2
index 45ca4e598ef0..f4059d73edd5 100644
@@ -2275,8 +2275,12 @@ static int fill_fscrypt_truncate(struct inode *inode,
 	     ceph_cap_string(issued));
 
 	/* Try to writeback the dirty pagecaches */
-	if (issued & (CEPH_CAP_FILE_BUFFER))
-		filemap_write_and_wait(inode->i_mapping);
+	if (issued & (CEPH_CAP_FILE_BUFFER)) {
+		ret = filemap_write_and_wait_range(inode->i_mapping,
+						   orig_pos, LLONG_MAX);
+		if (ret < 0)
+			goto out;
+	}
 
 	page = __page_cache_alloc(GFP_KERNEL);
 	if (page == NULL) {
Signature · 1 line
2.27.0
#2 lore raw

Re: [PATCH 1/2] ceph: flush small range instead of the whole map for truncate

Quoted 15 lines · On Thu, 2022-04-07 at 22:41 +0800, [email protected] wrote:

From: Xiubo Li <[email protected]>

---
 fs/ceph/inode.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
fs/ceph/inode.c +0 -0
index 45ca4e598ef0..f4059d73edd5 100644
@@ -2275,8 +2275,12 @@ static int fill_fscrypt_truncate(struct inode *inode,
 	     ceph_cap_string(issued));
 
 	/* Try to writeback the dirty pagecaches */

fs/ceph/inode.c static int fill_fscrypt_truncate(struct inode *inode, · line 2285

Quoted 10 lines
fs/ceph/inode.c +6 -2
-	if (issued & (CEPH_CAP_FILE_BUFFER))
-		filemap_write_and_wait(inode->i_mapping);
+	if (issued & (CEPH_CAP_FILE_BUFFER)) {
+		ret = filemap_write_and_wait_range(inode->i_mapping,
+						   orig_pos, LLONG_MAX);
+		if (ret < 0)
+			goto out;
+	}
 
 

Not much point in writing back blocks we're just going to truncate away anyhow. Maybe this should be writing with this range?

orig_pos, orig_pos + CEPH_FSCRYPT_BLOCK_SIZE - 1

Quoted 3 lines
fs/ceph/inode.c +0 -0
 	page = __page_cache_alloc(GFP_KERNEL);

if (page == NULL) {

Signature · 1 line
Jeff Layton <[email protected]>
author
#3 lore raw

Re: [PATCH 1/2] ceph: flush small range instead of the whole map for truncate

Quoted 30 lines · Jeff Layton
Quoted 15 lines · On Thu, 2022-04-07 at 22:41 +0800, [email protected] wrote:

From: Xiubo Li <[email protected]>

---
  fs/ceph/inode.c | 8 ++++++--
  1 file changed, 6 insertions(+), 2 deletions(-)
fs/ceph/inode.c +0 -0
index 45ca4e598ef0..f4059d73edd5 100644
@@ -2275,8 +2275,12 @@ static int fill_fscrypt_truncate(struct inode *inode,
  	     ceph_cap_string(issued));
 
  	/* Try to writeback the dirty pagecaches */

fs/ceph/inode.c static int fill_fscrypt_truncate(struct inode *inode, · line 2285

Quoted 10 lines
fs/ceph/inode.c +6 -2
-	if (issued & (CEPH_CAP_FILE_BUFFER))
-		filemap_write_and_wait(inode->i_mapping);
+	if (issued & (CEPH_CAP_FILE_BUFFER)) {
+		ret = filemap_write_and_wait_range(inode->i_mapping,
+						   orig_pos, LLONG_MAX);
+		if (ret < 0)
+			goto out;
+	}
 
 

Not much point in writing back blocks we're just going to truncate away anyhow. Maybe this should be writing with this range?

Quoted 1 line

orig_pos, orig_pos + CEPH_FSCRYPT_BLOCK_SIZE - 1

We need to make sure the last block is not buffered in pagecache, because we will aways sync read that from Rados.

Yeah, this looks much better.

I will fix and test it again.

Thanks

Quoted 3 lines
Quoted 3 lines
fs/ceph/inode.c +0 -0
  	page = __page_cache_alloc(GFP_KERNEL);
  	if (page == NULL) {
#4 lore raw

PATCH 2/2 ceph: fix coherency issue when truncating file size for fscrypt

In reply to [email protected]

From: Xiubo Li <[email protected]>

When truncating the file size the MDS will help update the last encrypted block, and during this we need to make sure the client won't fill the pagecaches.

---
 fs/ceph/inode.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
fs/ceph/inode.c +6 -1
index f4059d73edd5..cc1829ab497d 100644
@@ -2647,9 +2647,12 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c
 		req->r_num_caps = 1;
 		req->r_stamp = attr->ia_ctime;
 		if (fill_fscrypt) {
+			filemap_invalidate_lock(inode->i_mapping);
 			err = fill_fscrypt_truncate(inode, req, attr);
-			if (err)
+			if (err) {
+				filemap_invalidate_unlock(inode->i_mapping);
 				goto out;
+			}
 		}
 
 		/*
@@ -2660,6 +2663,8 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c
 		 * it.
 		 */
 		err = ceph_mdsc_do_request(mdsc, NULL, req);
+		if (fill_fscrypt)
+			filemap_invalidate_unlock(inode->i_mapping);
 		if (err == -EAGAIN && truncate_retry--) {
 			dout("setattr %p result=%d (%s locally, %d remote), retry it!\n",
 			     inode, err, ceph_cap_string(dirtied), mask);
Signature · 1 line
2.27.0
#5 lore raw

Re: [PATCH 2/2] ceph: fix coherency issue when truncating file size for fscrypt

Quoted 30 lines · On Thu, 2022-04-07 at 22:41 +0800, [email protected] wrote:

From: Xiubo Li <[email protected]>

When truncating the file size the MDS will help update the last encrypted block, and during this we need to make sure the client won't fill the pagecaches.

---
 fs/ceph/inode.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
fs/ceph/inode.c +4 -1
index f4059d73edd5..cc1829ab497d 100644
@@ -2647,9 +2647,12 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c
 		req->r_num_caps = 1;
 		req->r_stamp = attr->ia_ctime;
 		if (fill_fscrypt) {
+			filemap_invalidate_lock(inode->i_mapping);
 			err = fill_fscrypt_truncate(inode, req, attr);
-			if (err)
+			if (err) {
+				filemap_invalidate_unlock(inode->i_mapping);
 				goto out;
+			}
 		}
 
 		/*

fs/ceph/inode.c int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c · line 2670

Quoted 8 lines
fs/ceph/inode.c +2 -0
@@ -2660,6 +2663,8 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c
 		 * it.
 		 */
 		err = ceph_mdsc_do_request(mdsc, NULL, req);
+		if (fill_fscrypt)
+			filemap_invalidate_unlock(inode->i_mapping);
 		if (err == -EAGAIN && truncate_retry--) {
 			dout("setattr %p result=%d (%s locally, %d remote), retry it!\n",
 			     inode, err, ceph_cap_string(dirtied), mask);

Looks reasonable. Is there any reason we shouldn't do this in the non- encrypted case too? I suppose it doesn't make as much difference in that case.

I'll plan to pull this and the other patch into the wip-fscrypt branch. Should I just fold them into your earlier patches?

Signature · 1 line
Jeff Layton <[email protected]>
#6 lore raw

Re: [PATCH 2/2] ceph: fix coherency issue when truncating file size for fscrypt

Quoted 44 lines · Jeff Layton
Quoted 30 lines · On Thu, 2022-04-07 at 22:41 +0800, [email protected] wrote:

From: Xiubo Li <[email protected]>

When truncating the file size the MDS will help update the last encrypted block, and during this we need to make sure the client won't fill the pagecaches.

---
 fs/ceph/inode.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
fs/ceph/inode.c +4 -1
index f4059d73edd5..cc1829ab497d 100644
@@ -2647,9 +2647,12 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c
 		req->r_num_caps = 1;
 		req->r_stamp = attr->ia_ctime;
 		if (fill_fscrypt) {
+			filemap_invalidate_lock(inode->i_mapping);
 			err = fill_fscrypt_truncate(inode, req, attr);
-			if (err)
+			if (err) {
+				filemap_invalidate_unlock(inode->i_mapping);
 				goto out;
+			}
 		}
 
 		/*

fs/ceph/inode.c int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c · line 2670

Quoted 8 lines
fs/ceph/inode.c +2 -0
@@ -2660,6 +2663,8 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c
 		 * it.
 		 */
 		err = ceph_mdsc_do_request(mdsc, NULL, req);
+		if (fill_fscrypt)
+			filemap_invalidate_unlock(inode->i_mapping);
 		if (err == -EAGAIN && truncate_retry--) {
 			dout("setattr %p result=%d (%s locally, %d remote), retry it!\n",
 			     inode, err, ceph_cap_string(dirtied), mask);

Looks reasonable. Is there any reason we shouldn't do this in the non- encrypted case too? I suppose it doesn't make as much difference in that case.

Quoted 2 lines

I'll plan to pull this and the other patch into the wip-fscrypt branch. Should I just fold them into your earlier patches?

OTOH...do we really need this? I'm not sure I understand the race you're trying to prevent. Can you lay it out for me?

Thanks,

Signature · 1 line
Jeff Layton <[email protected]>
author
#7 lore raw

Re: [PATCH 2/2] ceph: fix coherency issue when truncating file size for fscrypt

Quoted 30 lines · Jeff Layton
Quoted 29 lines · Jeff Layton
Quoted 28 lines · On Thu, 2022-04-07 at 22:41 +0800, [email protected] wrote:

From: Xiubo Li <[email protected]>

When truncating the file size the MDS will help update the last encrypted block, and during this we need to make sure the client won't fill the pagecaches.

---
  fs/ceph/inode.c | 7 ++++++-
  1 file changed, 6 insertions(+), 1 deletion(-)
fs/ceph/inode.c +4 -1
index f4059d73edd5..cc1829ab497d 100644
@@ -2647,9 +2647,12 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c
  		req->r_num_caps = 1;
  		req->r_stamp = attr->ia_ctime;
  		if (fill_fscrypt) {
+			filemap_invalidate_lock(inode->i_mapping);
  			err = fill_fscrypt_truncate(inode, req, attr);
-			if (err)
+			if (err) {
+				filemap_invalidate_unlock(inode->i_mapping);
  				goto out;
+			}
  		}
 
Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
fs/ceph/inode.c +0 -0
  		/*
@@ -2660,6 +2663,8 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c
  		 * it.
  		 */
  		err = ceph_mdsc_do_request(mdsc, NULL, req);
Quoted 8 lines

fs/ceph/inode.c int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c · line 2670

Quoted 5 lines
fs/ceph/inode.c +2 -0
+		if (fill_fscrypt)
+			filemap_invalidate_unlock(inode->i_mapping);
  		if (err == -EAGAIN && truncate_retry--) {
  			dout("setattr %p result=%d (%s locally, %d remote), retry it!\n",
  			     inode, err, ceph_cap_string(dirtied), mask);

Looks reasonable. Is there any reason we shouldn't do this in the non- encrypted case too? I suppose it doesn't make as much difference in that case.

We only need this in encrypted case, which will do the RMW for the last block.

Quoted 2 lines
Quoted 2 lines

I'll plan to pull this and the other patch into the wip-fscrypt branch. Should I just fold them into your earlier patches?

Yeah, certainly.

Quoted 2 lines

OTOH...do we really need this? I'm not sure I understand the race you're trying to prevent. Can you lay it out for me?

I am thinking during the RMW for the last block, the page fault still could happen because the page fault function doesn't prevent that.

And we should prevent it during the RMW is going on.

-- Xiubo

Quoted 2 lines

Thanks,

#8 lore raw

Re: [PATCH 2/2] ceph: fix coherency issue when truncating file size for fscrypt

Quoted 51 lines · Xiubo Li
Quoted 30 lines · Jeff Layton
Quoted 29 lines · Jeff Layton
Quoted 28 lines · On Thu, 2022-04-07 at 22:41 +0800, [email protected] wrote:

From: Xiubo Li <[email protected]>

When truncating the file size the MDS will help update the last encrypted block, and during this we need to make sure the client won't fill the pagecaches.

---
  fs/ceph/inode.c | 7 ++++++-
  1 file changed, 6 insertions(+), 1 deletion(-)
fs/ceph/inode.c +4 -1
index f4059d73edd5..cc1829ab497d 100644
@@ -2647,9 +2647,12 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c
  		req->r_num_caps = 1;
  		req->r_stamp = attr->ia_ctime;
  		if (fill_fscrypt) {
+			filemap_invalidate_lock(inode->i_mapping);
  			err = fill_fscrypt_truncate(inode, req, attr);
-			if (err)
+			if (err) {
+				filemap_invalidate_unlock(inode->i_mapping);
  				goto out;
+			}
  		}
 
Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
fs/ceph/inode.c +0 -0
  		/*
@@ -2660,6 +2663,8 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c
  		 * it.
  		 */
  		err = ceph_mdsc_do_request(mdsc, NULL, req);
Quoted 8 lines

fs/ceph/inode.c int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c · line 2670

Quoted 5 lines
fs/ceph/inode.c +2 -0
+		if (fill_fscrypt)
+			filemap_invalidate_unlock(inode->i_mapping);
  		if (err == -EAGAIN && truncate_retry--) {
  			dout("setattr %p result=%d (%s locally, %d remote), retry it!\n",
  			     inode, err, ceph_cap_string(dirtied), mask);

Looks reasonable. Is there any reason we shouldn't do this in the non- encrypted case too? I suppose it doesn't make as much difference in that case.

We only need this in encrypted case, which will do the RMW for the last block.

Quoted 2 lines
Quoted 2 lines

I'll plan to pull this and the other patch into the wip-fscrypt branch. Should I just fold them into your earlier patches?

Quoted 9 lines

Yeah, certainly.

Quoted 2 lines

OTOH...do we really need this? I'm not sure I understand the race you're trying to prevent. Can you lay it out for me?

I am thinking during the RMW for the last block, the page fault still could happen because the page fault function doesn't prevent that.

And we should prevent it during the RMW is going on.

Right, but the RMW is being done using an anonymous page, and at this point in the process we haven't really touched the pagecache yet. That doesn't happen until __ceph_do_pending_vmtruncate.

Most of the callers for filemap_invalidate_lock/_unlock are in the hole punching codepaths, and not so much in truncate. What outcome are you trying to prevent with this? Can you lay out the potential race and why it would be harmful?

Signature · 1 line
Jeff Layton <[email protected]>
author
#9 lore raw

Re: [PATCH 2/2] ceph: fix coherency issue when truncating file size for fscrypt

Quoted 59 lines · Jeff Layton
Quoted 50 lines · Xiubo Li
Quoted 30 lines · Jeff Layton
Quoted 29 lines · Jeff Layton
Quoted 28 lines · On Thu, 2022-04-07 at 22:41 +0800, [email protected] wrote:

From: Xiubo Li <[email protected]>

When truncating the file size the MDS will help update the last encrypted block, and during this we need to make sure the client won't fill the pagecaches.

---
   fs/ceph/inode.c | 7 ++++++-
   1 file changed, 6 insertions(+), 1 deletion(-)
fs/ceph/inode.c +4 -1
index f4059d73edd5..cc1829ab497d 100644
@@ -2647,9 +2647,12 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c
   		req->r_num_caps = 1;
   		req->r_stamp = attr->ia_ctime;
   		if (fill_fscrypt) {
+			filemap_invalidate_lock(inode->i_mapping);
   			err = fill_fscrypt_truncate(inode, req, attr);
-			if (err)
+			if (err) {
+				filemap_invalidate_unlock(inode->i_mapping);
   				goto out;
+			}
   		}
 
Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
fs/ceph/inode.c +0 -0
   		/*
@@ -2660,6 +2663,8 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c
   		 * it.
   		 */
   		err = ceph_mdsc_do_request(mdsc, NULL, req);
Quoted 8 lines

fs/ceph/inode.c int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c · line 2670

Quoted 5 lines
fs/ceph/inode.c +2 -0
+		if (fill_fscrypt)
+			filemap_invalidate_unlock(inode->i_mapping);
   		if (err == -EAGAIN && truncate_retry--) {
   			dout("setattr %p result=%d (%s locally, %d remote), retry it!\n",
   			     inode, err, ceph_cap_string(dirtied), mask);

Looks reasonable. Is there any reason we shouldn't do this in the non- encrypted case too? I suppose it doesn't make as much difference in that case.

We only need this in encrypted case, which will do the RMW for the last block.

Quoted 2 lines
Quoted 2 lines

I'll plan to pull this and the other patch into the wip-fscrypt branch. Should I just fold them into your earlier patches?

Quoted 8 lines

Yeah, certainly.

Quoted 2 lines

OTOH...do we really need this? I'm not sure I understand the race you're trying to prevent. Can you lay it out for me?

I am thinking during the RMW for the last block, the page fault still could happen because the page fault function doesn't prevent that.

And we should prevent it during the RMW is going on.

Quoted 8 lines

Right, but the RMW is being done using an anonymous page, and at this point in the process we haven't really touched the pagecache yet. That doesn't happen until __ceph_do_pending_vmtruncate.

Most of the callers for filemap_invalidate_lock/_unlock are in the hole punching codepaths, and not so much in truncate. What outcome are you trying to prevent with this? Can you lay out the potential race and why it would be harmful?

Yeah, here I forgot to invalidate the mapping. After writing the dirty pagecache back we should invalidate the mapping and drop the related page too.

It should be:

filemap_invalidate_lock(inode->i_mapping);

write pagecache back;

invalidate the mapping and drop the pages;

do the RMW;

filemap_invalidate_unlock(inode->i_mapping);

As you mentioned in another mail, other processes could do the map read at the same time, and we should make sure that when we are truncating the size, we should block map read to continue and just trigger a page fault and the page fault should wait our truncate size finish ?

-- Xiubo