ceph: misc fix size truncate for fscrypt
cover ceph: misc fix size truncate for fscrypt
From: Xiubo Li <[email protected]>
Xiubo Li (2):
ceph: flush small range instead of the whole map for truncate
ceph: fix coherency issue when truncating file size for fscrypt
fs/ceph/inode.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-)
Signature · 1 line
2.27.0
PATCH 1/2 ceph: flush small range instead of the whole map for truncate
From: Xiubo Li <[email protected]>
--- fs/ceph/inode.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
fs/ceph/inode.c +6 -2
@@ -2275,8 +2275,12 @@ static int fill_fscrypt_truncate(struct inode *inode, ceph_cap_string(issued));/* Try to writeback the dirty pagecaches */ -if (issued & (CEPH_CAP_FILE_BUFFER )) -filemap_write_and_wait (inode->i_mapping); +if (issued & (CEPH_CAP_FILE_BUFFER )) { + ret =filemap_write_and_wait_range (inode->i_mapping, + orig_pos,LLONG_MAX ); +if (ret <0 ) + goto out; + } page =__page_cache_alloc (GFP_KERNEL );if (page ==NULL ) {
Signature · 1 line
2.27.0
Re: [PATCH 1/2] ceph: flush small range instead of the whole map for truncate
Quoted 15 lines · On Thu, 2022-04-07 at 22:41 +0800, [email protected] wrote:
From: Xiubo Li <[email protected]>
Signed-off-by Xiubo Li <[email protected]>--- fs/ceph/inode.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)fs/ceph/inode.c +0
-0@@ -2275,8 +2275,12 @@ static int fill_fscrypt_truncate(struct inode *inode, ceph_cap_string(issued));/* Try to writeback the dirty pagecaches */
fs/ceph/inode.c static int fill_fscrypt_truncate(struct inode *inode, · line 2285
Quoted 10 lines
fs/ceph/inode.c +6
-2-if (issued & (CEPH_CAP_FILE_BUFFER )) -filemap_write_and_wait (inode->i_mapping); +if (issued & (CEPH_CAP_FILE_BUFFER )) { + ret =filemap_write_and_wait_range (inode->i_mapping, + orig_pos,LLONG_MAX ); +if (ret <0 ) + goto out; + }
Not much point in writing back blocks we're just going to truncate away anyhow. Maybe this should be writing with this range?
orig_pos, orig_pos + CEPH_FSCRYPT_BLOCK_SIZE - 1
Quoted 3 lines
fs/ceph/inode.c +0
-0page =__page_cache_alloc (GFP_KERNEL );if (page == NULL) {
Signature · 1 line
Jeff Layton <[email protected]>
Re: [PATCH 1/2] ceph: flush small range instead of the whole map for truncate
Quoted 30 lines · Jeff Layton
Quoted 15 lines · On Thu, 2022-04-07 at 22:41 +0800, [email protected] wrote:
From: Xiubo Li <[email protected]>
Signed-off-by Xiubo Li <[email protected]>--- fs/ceph/inode.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)fs/ceph/inode.c +0
-0@@ -2275,8 +2275,12 @@ static int fill_fscrypt_truncate(struct inode *inode, ceph_cap_string(issued));/* Try to writeback the dirty pagecaches */ fs/ceph/inode.c static int fill_fscrypt_truncate(struct inode *inode, · line 2285
Quoted 10 lines
fs/ceph/inode.c +6
-2-if (issued & (CEPH_CAP_FILE_BUFFER )) -filemap_write_and_wait (inode->i_mapping); +if (issued & (CEPH_CAP_FILE_BUFFER )) { + ret =filemap_write_and_wait_range (inode->i_mapping, + orig_pos,LLONG_MAX ); +if (ret <0 ) + goto out; + }Not much point in writing back blocks we're just going to truncate away anyhow. Maybe this should be writing with this range?
Quoted 1 line
orig_pos,orig_pos+CEPH_FSCRYPT_BLOCK_SIZE- 1
We need to make sure the last block is not buffered in pagecache, because we will aways sync read that from Rados.
Yeah, this looks much better.
I will fix and test it again.
Thanks
Quoted 3 lines
Quoted 3 lines
fs/ceph/inode.c +0
-0page =__page_cache_alloc (GFP_KERNEL );if (page == NULL) {
PATCH 2/2 ceph: fix coherency issue when truncating file size for fscrypt
In reply to [email protected]From: Xiubo Li <[email protected]>
When truncating the file size the MDS will help update the last encrypted block, and during this we need to make sure the client won't fill the pagecaches.
--- fs/ceph/inode.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
fs/ceph/inode.c +6 -1
@@ -2647,9 +2647,12 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c req->r_num_caps =1 ; req->r_stamp = attr->ia_ctime;if (fill_fscrypt) { +filemap_invalidate_lock (inode->i_mapping); err =fill_fscrypt_truncate (inode, req, attr); -if (err) +if (err) { +filemap_invalidate_unlock (inode->i_mapping); goto out; + } }/*
@@ -2660,6 +2663,8 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c* it. */ err =ceph_mdsc_do_request (mdsc,NULL , req); +if (fill_fscrypt) +filemap_invalidate_unlock (inode->i_mapping);if (err == -EAGAIN && truncate_retry--) {dout ("setattr %p result=%d (%s locally, %d remote), retry it!\n" , inode, err,ceph_cap_string (dirtied), mask);
Signature · 1 line
2.27.0
Re: [PATCH 2/2] ceph: fix coherency issue when truncating file size for fscrypt
Quoted 30 lines · On Thu, 2022-04-07 at 22:41 +0800, [email protected] wrote:
From: Xiubo Li <[email protected]>
When truncating the file size the MDS will help update the last encrypted block, and during this we need to make sure the client won't fill the pagecaches.
Signed-off-by Xiubo Li <[email protected]>--- fs/ceph/inode.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)fs/ceph/inode.c +4
-1@@ -2647,9 +2647,12 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c req->r_num_caps =1 ; req->r_stamp = attr->ia_ctime;if (fill_fscrypt) { +filemap_invalidate_lock (inode->i_mapping); err =fill_fscrypt_truncate (inode, req, attr); -if (err) +if (err) { +filemap_invalidate_unlock (inode->i_mapping); goto out; + } }/*
fs/ceph/inode.c int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c · line 2670
Quoted 8 lines
fs/ceph/inode.c +2
-0@@ -2660,6 +2663,8 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c* it. */ err =ceph_mdsc_do_request (mdsc,NULL , req); +if (fill_fscrypt) +filemap_invalidate_unlock (inode->i_mapping);if (err == -EAGAIN && truncate_retry--) {dout ("setattr %p result=%d (%s locally, %d remote), retry it!\n" , inode, err,ceph_cap_string (dirtied), mask);
Looks reasonable. Is there any reason we shouldn't do this in the non- encrypted case too? I suppose it doesn't make as much difference in that case.
I'll plan to pull this and the other patch into the wip-fscrypt branch. Should I just fold them into your earlier patches?
Signature · 1 line
Jeff Layton <[email protected]>
Re: [PATCH 2/2] ceph: fix coherency issue when truncating file size for fscrypt
Quoted 44 lines · Jeff Layton
Quoted 30 lines · On Thu, 2022-04-07 at 22:41 +0800, [email protected] wrote:
From: Xiubo Li <[email protected]>
When truncating the file size the MDS will help update the last encrypted block, and during this we need to make sure the client won't fill the pagecaches.
Signed-off-by Xiubo Li <[email protected]>--- fs/ceph/inode.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)fs/ceph/inode.c +4
-1@@ -2647,9 +2647,12 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c req->r_num_caps =1 ; req->r_stamp = attr->ia_ctime;if (fill_fscrypt) { +filemap_invalidate_lock (inode->i_mapping); err =fill_fscrypt_truncate (inode, req, attr); -if (err) +if (err) { +filemap_invalidate_unlock (inode->i_mapping); goto out; + } }/* fs/ceph/inode.c int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c · line 2670
Quoted 8 lines
fs/ceph/inode.c +2
-0@@ -2660,6 +2663,8 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c* it. */ err =ceph_mdsc_do_request (mdsc,NULL , req); +if (fill_fscrypt) +filemap_invalidate_unlock (inode->i_mapping);if (err == -EAGAIN && truncate_retry--) {dout ("setattr %p result=%d (%s locally, %d remote), retry it!\n" , inode, err,ceph_cap_string (dirtied), mask);Looks reasonable. Is there any reason we shouldn't do this in the non- encrypted case too? I suppose it doesn't make as much difference in that case.
Quoted 2 lines
I'll plan to pull this and the other patch into the wip-fscrypt branch. Should I just fold them into your earlier patches?
OTOH...do we really need this? I'm not sure I understand the race you're trying to prevent. Can you lay it out for me?
Thanks,
Signature · 1 line
Jeff Layton <[email protected]>
Re: [PATCH 2/2] ceph: fix coherency issue when truncating file size for fscrypt
Quoted 30 lines · Jeff Layton
Quoted 29 lines · Jeff Layton
Quoted 28 lines · On Thu, 2022-04-07 at 22:41 +0800, [email protected] wrote:
From: Xiubo Li <[email protected]>
When truncating the file size the MDS will help update the last encrypted block, and during this we need to make sure the client won't fill the pagecaches.
Signed-off-by Xiubo Li <[email protected]>--- fs/ceph/inode.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)fs/ceph/inode.c +4
-1@@ -2647,9 +2647,12 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c req->r_num_caps =1 ; req->r_stamp = attr->ia_ctime;if (fill_fscrypt) { +filemap_invalidate_lock (inode->i_mapping); err =fill_fscrypt_truncate (inode, req, attr); -if (err) +if (err) { +filemap_invalidate_unlock (inode->i_mapping); goto out; + } }
Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
fs/ceph/inode.c +0
-0/* @@ -2660,6 +2663,8 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c* it. */ err =ceph_mdsc_do_request (mdsc,NULL , req);Quoted 8 lines
fs/ceph/inode.c int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c · line 2670
Quoted 5 lines
fs/ceph/inode.c +2
-0+if (fill_fscrypt) +filemap_invalidate_unlock (inode->i_mapping);if (err == -EAGAIN && truncate_retry--) {dout ("setattr %p result=%d (%s locally, %d remote), retry it!\n" , inode, err,ceph_cap_string (dirtied), mask);Looks reasonable. Is there any reason we shouldn't do this in the non- encrypted case too? I suppose it doesn't make as much difference in that case.
We only need this in encrypted case, which will do the RMW for the last block.
Quoted 2 lines
Quoted 2 lines
I'll plan to pull this and the other patch into the wip-fscrypt branch. Should I just fold them into your earlier patches?
Yeah, certainly.
Quoted 2 lines
OTOH...do we really need this? I'm not sure I understand the race you're trying to prevent. Can you lay it out for me?
I am thinking during the RMW for the last block, the page fault still could happen because the page fault function doesn't prevent that.
And we should prevent it during the RMW is going on.
-- Xiubo
Quoted 2 lines
Thanks,
Re: [PATCH 2/2] ceph: fix coherency issue when truncating file size for fscrypt
Quoted 51 lines · Xiubo Li
Quoted 30 lines · Jeff Layton
Quoted 29 lines · Jeff Layton
Quoted 28 lines · On Thu, 2022-04-07 at 22:41 +0800, [email protected] wrote:
From: Xiubo Li <[email protected]>
When truncating the file size the MDS will help update the last encrypted block, and during this we need to make sure the client won't fill the pagecaches.
Signed-off-by Xiubo Li <[email protected]>--- fs/ceph/inode.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)fs/ceph/inode.c +4
-1@@ -2647,9 +2647,12 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c req->r_num_caps =1 ; req->r_stamp = attr->ia_ctime;if (fill_fscrypt) { +filemap_invalidate_lock (inode->i_mapping); err =fill_fscrypt_truncate (inode, req, attr); -if (err) +if (err) { +filemap_invalidate_unlock (inode->i_mapping); goto out; + } }Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
fs/ceph/inode.c +0
-0/* @@ -2660,6 +2663,8 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c* it. */ err =ceph_mdsc_do_request (mdsc,NULL , req);Quoted 8 lines
fs/ceph/inode.c int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c · line 2670
Quoted 5 lines
fs/ceph/inode.c +2
-0+if (fill_fscrypt) +filemap_invalidate_unlock (inode->i_mapping);if (err == -EAGAIN && truncate_retry--) {dout ("setattr %p result=%d (%s locally, %d remote), retry it!\n" , inode, err,ceph_cap_string (dirtied), mask);Looks reasonable. Is there any reason we shouldn't do this in the non- encrypted case too? I suppose it doesn't make as much difference in that case.
We only need this in encrypted case, which will do the RMW for the last block.
Quoted 2 lines
Quoted 2 lines
I'll plan to pull this and the other patch into the wip-fscrypt branch. Should I just fold them into your earlier patches?
Quoted 9 lines
Yeah, certainly.
Quoted 2 lines
OTOH...do we really need this? I'm not sure I understand the race you're trying to prevent. Can you lay it out for me?
I am thinking during the RMW for the last block, the page fault still could happen because the page fault function doesn't prevent that.
And we should prevent it during the RMW is going on.
Right, but the RMW is being done using an anonymous page, and at this point in the process we haven't really touched the pagecache yet. That doesn't happen until __ceph_do_pending_vmtruncate.
Most of the callers for filemap_invalidate_lock/_unlock are in the hole punching codepaths, and not so much in truncate. What outcome are you trying to prevent with this? Can you lay out the potential race and why it would be harmful?
Signature · 1 line
Jeff Layton <[email protected]>
Re: [PATCH 2/2] ceph: fix coherency issue when truncating file size for fscrypt
Quoted 59 lines · Jeff Layton
Quoted 50 lines · Xiubo Li
Quoted 30 lines · Jeff Layton
Quoted 29 lines · Jeff Layton
Quoted 28 lines · On Thu, 2022-04-07 at 22:41 +0800, [email protected] wrote:
From: Xiubo Li <[email protected]>
When truncating the file size the MDS will help update the last encrypted block, and during this we need to make sure the client won't fill the pagecaches.
Signed-off-by Xiubo Li <[email protected]>--- fs/ceph/inode.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)fs/ceph/inode.c +4
-1@@ -2647,9 +2647,12 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c req->r_num_caps =1 ; req->r_stamp = attr->ia_ctime;if (fill_fscrypt) { +filemap_invalidate_lock (inode->i_mapping); err =fill_fscrypt_truncate (inode, req, attr); -if (err) +if (err) { +filemap_invalidate_unlock (inode->i_mapping); goto out; + } }Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
fs/ceph/inode.c +0
-0/* @@ -2660,6 +2663,8 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c* it. */ err =ceph_mdsc_do_request (mdsc,NULL , req);Quoted 8 lines
fs/ceph/inode.c int __ceph_setattr(struct inode *inode, struct iattr *attr, struct ceph_iattr *c · line 2670
Quoted 5 lines
fs/ceph/inode.c +2
-0+if (fill_fscrypt) +filemap_invalidate_unlock (inode->i_mapping);if (err == -EAGAIN && truncate_retry--) {dout ("setattr %p result=%d (%s locally, %d remote), retry it!\n" , inode, err,ceph_cap_string (dirtied), mask);Looks reasonable. Is there any reason we shouldn't do this in the non- encrypted case too? I suppose it doesn't make as much difference in that case.
We only need this in encrypted case, which will do the RMW for the last block.
Quoted 2 lines
Quoted 2 lines
I'll plan to pull this and the other patch into the wip-fscrypt branch. Should I just fold them into your earlier patches?
Quoted 8 lines
Yeah, certainly.
Quoted 2 lines
OTOH...do we really need this? I'm not sure I understand the race you're trying to prevent. Can you lay it out for me?
I am thinking during the RMW for the last block, the page fault still could happen because the page fault function doesn't prevent that.
And we should prevent it during the RMW is going on.
Quoted 8 lines
Right, but the RMW is being done using an anonymous page, and at this point in the process we haven't really touched the pagecache yet. That doesn't happen until
__ceph_do_pending_vmtruncate.Most of the callers for filemap_invalidate_lock/_unlock are in the hole punching codepaths, and not so much in truncate. What outcome are you trying to prevent with this? Can you lay out the potential race and why it would be harmful?
Yeah, here I forgot to invalidate the mapping. After writing the dirty pagecache back we should invalidate the mapping and drop the related page too.
It should be:
filemap_invalidate_lock(inode->i_mapping);
write pagecache back;
invalidate the mapping and drop the pages;
do the RMW;
filemap_invalidate_unlock(inode->i_mapping);
As you mentioned in another mail, other processes could do the map read at the same time, and we should make sure that when we are truncating the size, we should block map read to continue and just trigger a page fault and the page fault should wait our truncate size finish ?
-- Xiubo