Skip to contentSkip to conversation

ceph: invalidate pages when doing direct/sync writes

PATCH Superseded Luís Henriques 3 participants 5 messages Apr 7, 2022 Ceph client

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 ceph: invalidate pages when doing direct/sync writes

When doing a direct/sync write, we need to invalidate the page cache in the range being written to. If we don't do this, the cache will include invalid data as we just did a write that avoided the page cache.

---
 fs/ceph/file.c | 9 +++++++++
 1 file changed, 9 insertions(+)

Ok, here's a new attempt. After discussion in this thread and on IRC, I think this is the right fix. generic/647 now passes with and without encryption. Thanks!

Changes since v2:
- Invalidation needs to be done after a write

Changes since v1:
- Replaced truncate_inode_pages_range() by invalidate_inode_pages2_range
- Call fscache_invalidate with FSCACHE_INVAL_DIO_WRITE if we're doing DIO

fs/ceph/file.c +9 -0
index 5072570c2203..63e67eb60310 100644
@@ -1938,6 +1938,15 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos,
 			break;
 		}
 		ceph_clear_error_write(ci);
+		ret = invalidate_inode_pages2_range(
+				inode->i_mapping,
+				pos >> PAGE_SHIFT,
+				(pos + len - 1) >> PAGE_SHIFT);
+		if (ret < 0) {
+			dout("invalidate_inode_pages2_range returned %d\n",
+			     ret);
+			ret = 0;
+		}
 		pos += len;
 		written += len;
 		dout("sync_write written %d\n", written);
#1 lore raw
Quoted 30 lines · Luís Henriques

When doing a direct/sync write, we need to invalidate the page cache in the range being written to. If we don't do this, the cache will include invalid data as we just did a write that avoided the page cache.

---
 fs/ceph/file.c | 9 +++++++++
 1 file changed, 9 insertions(+)

Ok, here's a new attempt. After discussion in this thread and on IRC, I think this is the right fix. generic/647 now passes with and without encryption. Thanks!

Changes since v2:
- Invalidation needs to be done after a write

Changes since v1:
- Replaced truncate_inode_pages_range() by invalidate_inode_pages2_range
- Call fscache_invalidate with FSCACHE_INVAL_DIO_WRITE if we're doing DIO

fs/ceph/file.c +2 -0
index 5072570c2203..63e67eb60310 100644
@@ -1938,6 +1938,15 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos,
 			break;
 		}
 		ceph_clear_error_write(ci);
+		ret = invalidate_inode_pages2_range(
+				inode->i_mapping,

fs/ceph/file.c ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos, · line 1952

Quoted 10 lines
fs/ceph/file.c +7 -0
+				pos >> PAGE_SHIFT,
+				(pos + len - 1) >> PAGE_SHIFT);
+		if (ret < 0) {
+			dout("invalidate_inode_pages2_range returned %d\n",
+			     ret);
+			ret = 0;
+		}
 		pos += len;
 		written += len;
 		dout("sync_write written %d\n", written);

Looks good. I suspect we can also remove the invalidate_indode_pages2_range call earlier in this function too. I may roll that into this patch.

I'll give this an xfstests run with fscrypt enabled and see how it does.

Thanks!

Signature · 1 line
Jeff Layton <[email protected]>
#2 lore raw
Quoted 37 lines · Jeff Layton
Quoted 36 lines · Luís Henriques

When doing a direct/sync write, we need to invalidate the page cache in the range being written to. If we don't do this, the cache will include invalid data as we just did a write that avoided the page cache.

---
 fs/ceph/file.c | 9 +++++++++
 1 file changed, 9 insertions(+)

Ok, here's a new attempt. After discussion in this thread and on IRC, I think this is the right fix. generic/647 now passes with and without encryption. Thanks!

Changes since v2:
- Invalidation needs to be done after a write

Changes since v1:
- Replaced truncate_inode_pages_range() by invalidate_inode_pages2_range
- Call fscache_invalidate with FSCACHE_INVAL_DIO_WRITE if we're doing DIO

fs/ceph/file.c +8 -0
index 5072570c2203..63e67eb60310 100644
@@ -1938,6 +1938,15 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos,
 			break;
 		}
 		ceph_clear_error_write(ci);
+		ret = invalidate_inode_pages2_range(
+				inode->i_mapping,
+				pos >> PAGE_SHIFT,
+				(pos + len - 1) >> PAGE_SHIFT);
+		if (ret < 0) {
+			dout("invalidate_inode_pages2_range returned %d\n",
+			     ret);
+			ret = 0;
Quoted 8 lines

fs/ceph/file.c ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos, · line 1952

Quoted 4 lines
fs/ceph/file.c +1 -0
+		}
 		pos += len;
 		written += len;
 		dout("sync_write written %d\n", written);

Looks good. I suspect we can also remove the invalidate_indode_pages2_range call earlier in this function too. I may roll that into this patch.

Right, that occurred to me as well but I wasn't really sure that would be safe.

Quoted 1 line

I'll give this an xfstests run with fscrypt enabled and see how it does.

I'll do the same here, I just run a few tests on a vstart cluster. But I'll definitely give it some more testing.

Cheers,

Signature · 1 line
Luís
Reviewed-by on v3 1/1
#3 lore raw
In reply to Luís Henriques
Quoted 31 lines · Luís Henriques

When doing a direct/sync write, we need to invalidate the page cache in the range being written to. If we don't do this, the cache will include invalid data as we just did a write that avoided the page cache.

---
  fs/ceph/file.c | 9 +++++++++
  1 file changed, 9 insertions(+)

Ok, here's a new attempt. After discussion in this thread and on IRC, I think this is the right fix. generic/647 now passes with and without encryption. Thanks!

Changes since v2:
- Invalidation needs to be done after a write

Changes since v1:
- Replaced truncate_inode_pages_range() by invalidate_inode_pages2_range
- Call fscache_invalidate with FSCACHE_INVAL_DIO_WRITE if we're doing DIO

fs/ceph/file.c +2 -0
index 5072570c2203..63e67eb60310 100644
@@ -1938,6 +1938,15 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos,
  			break;
  		}
  		ceph_clear_error_write(ci);
+		ret = invalidate_inode_pages2_range(
+				inode->i_mapping,

fs/ceph/file.c ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos, · line 1952

Quoted 10 lines
fs/ceph/file.c +7 -0
+				pos >> PAGE_SHIFT,
+				(pos + len - 1) >> PAGE_SHIFT);
+		if (ret < 0) {
+			dout("invalidate_inode_pages2_range returned %d\n",
+			     ret);
+			ret = 0;
+		}
  		pos += len;
  		written += len;
  		dout("sync_write written %d\n", written);

LGTM.

Maybe it worth adding a comment to explain why we need this and where the mapping come from ?

#4 lore raw
Quoted 44 lines · Xiubo Li
Quoted 31 lines · Luís Henriques

When doing a direct/sync write, we need to invalidate the page cache in the range being written to. If we don't do this, the cache will include invalid data as we just did a write that avoided the page cache.

---
  fs/ceph/file.c | 9 +++++++++
  1 file changed, 9 insertions(+)

Ok, here's a new attempt. After discussion in this thread and on IRC, I think this is the right fix. generic/647 now passes with and without encryption. Thanks!

Changes since v2:
- Invalidation needs to be done after a write

Changes since v1:
- Replaced truncate_inode_pages_range() by invalidate_inode_pages2_range
- Call fscache_invalidate with FSCACHE_INVAL_DIO_WRITE if we're doing DIO

fs/ceph/file.c +2 -0
index 5072570c2203..63e67eb60310 100644
@@ -1938,6 +1938,15 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos,
  			break;
  		}
  		ceph_clear_error_write(ci);
+		ret = invalidate_inode_pages2_range(
+				inode->i_mapping,

fs/ceph/file.c ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos, · line 1952

Quoted 10 lines
fs/ceph/file.c +7 -0
+				pos >> PAGE_SHIFT,
+				(pos + len - 1) >> PAGE_SHIFT);
+		if (ret < 0) {
+			dout("invalidate_inode_pages2_range returned %d\n",
+			     ret);
+			ret = 0;
+		}
  		pos += len;
  		written += len;
  		dout("sync_write written %d\n", written);

LGTM.

Quoted 5 lines

Maybe it worth adding a comment to explain why we need this and where the mapping come from ?

Sure, I'll send out v4 with an extra comment. Thanks.

Cheers,

Signature · 1 line
Luís