Skip to contentSkip to conversation

implement containerized syncfs for overlayfs

RFC PATCH Superseded Chengguang Xu 4 participants 23 messages Oct 25, 2020 – Nov 6, 2020

Subsystems (2)

v2 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

v2 cover implement containerized syncfs for overlayfs

Current syncfs(2) syscall on overlayfs just calls sync_filesystem() on upper_sb to synchronize whole dirty inodes in upper filesystem regardless of the overlay ownership of the inode. In the use case of container, when multiple containers using the same underlying upper filesystem, it has some shortcomings as below.

(1) Performance
Synchronization is probably heavy because it actually syncs unnecessary inodes for target overlayfs.

(2) Interference
Unplanned synchronization will probably impact IO performance of unrelated container processes on the other overlayfs.

This series try to implement containerized syncfs for overlayfs so that only sync target dirty upper inodes which are belong to specific overlayfs instance. By doing this, it is able to reduce cost of synchronization and will not seriously impact IO performance of unrelated processes.

v1->v2:
- Mark overlayfs' inode dirty itself instead of adding notification mechanism to vfs inode.

Chengguang Xu (8):
ovl: setup overlayfs' private bdi
ovl: implement ->writepages operation
ovl: implement overlayfs' ->evict_inode operation
ovl: mark overlayfs' inode dirty on modification
ovl: mark overlayfs' inode dirty on shared writable mmap
ovl: implement overlayfs' ->write_inode operation
ovl: cache dirty overlayfs' inode
ovl: implement containerized syncfs for overlayfs

 fs/overlayfs/file.c      |  4 +++
 fs/overlayfs/inode.c     | 27 +++++++++++++++++++
 fs/overlayfs/overlayfs.h |  4 +++
 fs/overlayfs/super.c     | 57 +++++++++++++++++++++++++++++++++++++---
 fs/overlayfs/util.c      | 14 ++++++++++
 5 files changed, 102 insertions(+), 4 deletions(-)
Signature · 1 line
2.26.2
#1 lore raw

RFC PATCH v2 1/8 ovl: setup overlayfs' private bdi

Setup overlayfs' private bdi so that we can collect overlayfs' own dirty inodes.

---
 fs/overlayfs/super.c | 4 ++++
 1 file changed, 4 insertions(+)
fs/overlayfs/super.c +4 -0
index 290983bcfbb3..d1e546abce87 100644
@@ -1869,6 +1869,10 @@ static int ovl_fill_super(struct super_block *sb, void *data, int silent)
 	if (!ofs)
 		goto out;
 
+	err = super_setup_bdi(sb);
+	if (err)
+		goto out_err;
+
 	ofs->creator_cred = cred = prepare_creds();
 	if (!cred)
 		goto out_err;
Signature · 1 line
2.26.2
#2 lore raw

RFC PATCH v2 2/8 ovl: implement ->writepages operation

In reply to Chengguang Xu

Implement overlayfs' ->writepages operation so that we can sync dirty data/metadata to upper filesystem.

---
 fs/overlayfs/inode.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
fs/overlayfs/inode.c +26 -0
index b584dca845ba..f27fc5be34df 100644
@@ -11,6 +11,7 @@
 #include <linux/posix_acl.h>
 #include <linux/ratelimit.h>
 #include <linux/fiemap.h>
+#include <linux/writeback.h>
 #include "overlayfs.h"
 
 
@@ -516,7 +517,32 @@ static const struct inode_operations ovl_special_inode_operations = {
 	.update_time	= ovl_update_time,
 };
 
+static int ovl_writepages(struct address_space *mapping,
+			  struct writeback_control *wbc)
+{
+	struct inode *upper_inode = ovl_inode_upper(mapping->host);
+	struct ovl_fs *ofs =  mapping->host->i_sb->s_fs_info;
+	struct writeback_control tmp_wbc = *wbc;
+
+	if (!ovl_should_sync(ofs))
+		return 0;
+
+	/*
+	 * for sync(2) writeback, it has a separate external IO
+	 * completion path by checking PAGECACHE_TAG_WRITEBACK
+	 * in pagecache, we have to set for_sync to 0 in thie case,
+	 * let writeback waits completion after syncing individual
+	 * dirty inode, because we haven't implemented overlayfs'
+	 * own pagecache yet.
+	 */
+	if (wbc->for_sync && (wbc->sync_mode == WB_SYNC_ALL))
+		tmp_wbc.for_sync = 0;
+
+	return sync_inode(upper_inode, &tmp_wbc);
+}
+
 static const struct address_space_operations ovl_aops = {
+	.writepages		= ovl_writepages,
 	/* For O_DIRECT dentry_open() checks f_mapping->a_ops->direct_IO */
 	.direct_IO		= noop_direct_IO,
 };
Signature · 1 line
2.26.2
reviewer
#3 lore raw

Re: [RFC PATCH v2 2/8] ovl: implement ->writepages operation

Quoted 34 lines · Chengguang Xu

Implement overlayfs' ->writepages operation so that we can sync dirty data/metadata to upper filesystem.

---
 fs/overlayfs/inode.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
fs/overlayfs/inode.c +11 -0
index b584dca845ba..f27fc5be34df 100644
@@ -11,6 +11,7 @@
 #include <linux/posix_acl.h>
 #include <linux/ratelimit.h>
 #include <linux/fiemap.h>
+#include <linux/writeback.h>
 #include "overlayfs.h"
 
 
@@ -516,7 +517,32 @@ static const struct inode_operations ovl_special_inode_operations = {
 	.update_time	= ovl_update_time,
 };
 
+static int ovl_writepages(struct address_space *mapping,
+			  struct writeback_control *wbc)
+{
+	struct inode *upper_inode = ovl_inode_upper(mapping->host);
+	struct ovl_fs *ofs =  mapping->host->i_sb->s_fs_info;
+	struct writeback_control tmp_wbc = *wbc;
+
+	if (!ovl_should_sync(ofs))
+		return 0;
+

fs/overlayfs/inode.c static const struct inode_operations ovl_special_inode_operations = { · line 539

Quoted 10 lines
fs/overlayfs/inode.c +10 -0
+	/*
+	 * for sync(2) writeback, it has a separate external IO
+	 * completion path by checking PAGECACHE_TAG_WRITEBACK
+	 * in pagecache, we have to set for_sync to 0 in thie case,
+	 * let writeback waits completion after syncing individual
+	 * dirty inode, because we haven't implemented overlayfs'
+	 * own pagecache yet.
+	 */
+	if (wbc->for_sync && (wbc->sync_mode == WB_SYNC_ALL))
+		tmp_wbc.for_sync = 0;

This looks really hacky as it closely depends on the internal details of writeback implementation. I'd be more open to say export wait_sb_inodes() for overlayfs use... Because that's what I gather you need in your overlayfs ->syncfs() implementation.

Honza

Quoted 13 lines
fs/overlayfs/inode.c +5 -0
+
+	return sync_inode(upper_inode, &tmp_wbc);
+}
+
 static const struct address_space_operations ovl_aops = {
+	.writepages		= ovl_writepages,
 	/* For O_DIRECT dentry_open() checks f_mapping->a_ops->direct_IO */
 	.direct_IO		= noop_direct_IO,
 };
Signature · 1 line
2.26.2
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#4 lore raw

Re: [RFC PATCH v2 2/8] ovl: implement ->writepages operation

---- 在 星期二, 2020-11-03 01:17:41 Jan Kara <[email protected]> 撰写 ----

Quoted 42 lines · Chengguang Xu
Quoted 41 lines

Implement overlayfs' ->writepages operation so that we can sync dirty data/metadata to upper filesystem.

---
 fs/overlayfs/inode.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
fs/overlayfs/inode.c +18 -0
index b584dca845ba..f27fc5be34df 100644
@@ -11,6 +11,7 @@
 #include <linux/posix_acl.h>
 #include <linux/ratelimit.h>
 #include <linux/fiemap.h>
+#include <linux/writeback.h>
 #include "overlayfs.h"
 
 
@@ -516,7 +517,32 @@ static const struct inode_operations ovl_special_inode_operations = {
     .update_time    = ovl_update_time,
 };
 
+static int ovl_writepages(struct address_space *mapping,
+              struct writeback_control *wbc)
+{
+    struct inode *upper_inode = ovl_inode_upper(mapping->host);
+    struct ovl_fs *ofs =  mapping->host->i_sb->s_fs_info;
+    struct writeback_control tmp_wbc = *wbc;
+
+    if (!ovl_should_sync(ofs))
+        return 0;
+
+    /*
+     * for sync(2) writeback, it has a separate external IO
+     * completion path by checking PAGECACHE_TAG_WRITEBACK
+     * in pagecache, we have to set for_sync to 0 in thie case,
+     * let writeback waits completion after syncing individual
+     * dirty inode, because we haven't implemented overlayfs'
+     * own pagecache yet.
Quoted 9 lines

fs/overlayfs/inode.c static const struct inode_operations ovl_special_inode_operations = { · line 539

Quoted 3 lines
fs/overlayfs/inode.c +3 -0
+     */
+    if (wbc->for_sync && (wbc->sync_mode == WB_SYNC_ALL))
+        tmp_wbc.for_sync = 0;

This looks really hacky as it closely depends on the internal details of writeback implementation. I'd be more open to say export wait_sb_inodes() for overlayfs use... Because that's what I gather you need in your overlayfs ->syncfs() implementation.

Does that mean we gather synced overlay's inode into a new waiting list(overlay's) and do the waiting behavior in overlay's ->syncfs() ?

Thanks,
Chengguang

reviewer
#5 lore raw

Re: [RFC PATCH v2 2/8] ovl: implement ->writepages operation

Quoted 53 lines · Chengguang Xu

---- 在 星期二, 2020-11-03 01:17:41 Jan Kara <[email protected]> 撰写 ----

Quoted 42 lines · Chengguang Xu
Quoted 41 lines

Implement overlayfs' ->writepages operation so that we can sync dirty data/metadata to upper filesystem.

---
 fs/overlayfs/inode.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
fs/overlayfs/inode.c +18 -0
index b584dca845ba..f27fc5be34df 100644
@@ -11,6 +11,7 @@
 #include <linux/posix_acl.h>
 #include <linux/ratelimit.h>
 #include <linux/fiemap.h>
+#include <linux/writeback.h>
 #include "overlayfs.h"
 
 
@@ -516,7 +517,32 @@ static const struct inode_operations ovl_special_inode_operations = {
     .update_time    = ovl_update_time,
 };
 
+static int ovl_writepages(struct address_space *mapping,
+              struct writeback_control *wbc)
+{
+    struct inode *upper_inode = ovl_inode_upper(mapping->host);
+    struct ovl_fs *ofs =  mapping->host->i_sb->s_fs_info;
+    struct writeback_control tmp_wbc = *wbc;
+
+    if (!ovl_should_sync(ofs))
+        return 0;
+
+    /*
+     * for sync(2) writeback, it has a separate external IO
+     * completion path by checking PAGECACHE_TAG_WRITEBACK
+     * in pagecache, we have to set for_sync to 0 in thie case,
+     * let writeback waits completion after syncing individual
+     * dirty inode, because we haven't implemented overlayfs'
+     * own pagecache yet.
Quoted 9 lines

fs/overlayfs/inode.c static const struct inode_operations ovl_special_inode_operations = { · line 539

Quoted 3 lines
fs/overlayfs/inode.c +3 -0
+     */
+    if (wbc->for_sync && (wbc->sync_mode == WB_SYNC_ALL))
+        tmp_wbc.for_sync = 0;

This looks really hacky as it closely depends on the internal details of writeback implementation. I'd be more open to say export wait_sb_inodes() for overlayfs use... Because that's what I gather you need in your overlayfs ->syncfs() implementation.

Quoted 2 lines

Does that mean we gather synced overlay's inode into a new waiting list(overlay's) and do the waiting behavior in overlay's ->syncfs() ?

My idea was that you'd just use the standard writeback logic which ends up gathering upper_sb inodes in the upper_sb->s_inodes_wb and then wait for them in overlay's ->syncfs(). Maybe we'll end up waiting for more inodes than strictly necessary but it shouldn't be too bad I'd say...

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#6 lore raw

Re: [RFC PATCH v2 2/8] ovl: implement ->writepages operation

---- 在 星期四, 2020-11-05 21:55:06 Jan Kara <[email protected]> 撰写 ----

Quoted 54 lines · Chengguang Xu
Quoted 53 lines

---- 在 星期二, 2020-11-03 01:17:41 Jan Kara <[email protected]> 撰写 ----

Quoted 51 lines · Chengguang Xu
Quoted 34 lines

Implement overlayfs' ->writepages operation so that we can sync dirty data/metadata to upper filesystem.

---
 fs/overlayfs/inode.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
fs/overlayfs/inode.c +11 -0
index b584dca845ba..f27fc5be34df 100644
@@ -11,6 +11,7 @@
 #include <linux/posix_acl.h>
 #include <linux/ratelimit.h>
 #include <linux/fiemap.h>
+#include <linux/writeback.h>
 #include "overlayfs.h"
 
 
@@ -516,7 +517,32 @@ static const struct inode_operations ovl_special_inode_operations = {
     .update_time    = ovl_update_time,
 };
 
+static int ovl_writepages(struct address_space *mapping,
+              struct writeback_control *wbc)
+{
+    struct inode *upper_inode = ovl_inode_upper(mapping->host);
+    struct ovl_fs *ofs =  mapping->host->i_sb->s_fs_info;
+    struct writeback_control tmp_wbc = *wbc;
+
+    if (!ovl_should_sync(ofs))
+        return 0;
+

fs/overlayfs/inode.c static const struct inode_operations ovl_special_inode_operations = { · line 539

Quoted 10 lines
fs/overlayfs/inode.c +10 -0
+    /*
+     * for sync(2) writeback, it has a separate external IO
+     * completion path by checking PAGECACHE_TAG_WRITEBACK
+     * in pagecache, we have to set for_sync to 0 in thie case,
+     * let writeback waits completion after syncing individual
+     * dirty inode, because we haven't implemented overlayfs'
+     * own pagecache yet.
+     */
+    if (wbc->for_sync && (wbc->sync_mode == WB_SYNC_ALL))
+        tmp_wbc.for_sync = 0;

This looks really hacky as it closely depends on the internal details of writeback implementation. I'd be more open to say export wait_sb_inodes() for overlayfs use... Because that's what I gather you need in your overlayfs ->syncfs() implementation.

Quoted 8 lines
Quoted 2 lines

Does that mean we gather synced overlay's inode into a new waiting list(overlay's) and do the waiting behavior in overlay's ->syncfs() ?

My idea was that you'd just use the standard writeback logic which ends up gathering upper_sb inodes in the upper_sb->s_inodes_wb and then wait for them in overlay's ->syncfs(). Maybe we'll end up waiting for more inodes than strictly necessary but it shouldn't be too bad I'd say...

Yeah, I agree with you, I'll modify in next version.

Thanks,
Chengguang

#7 lore raw

RFC PATCH v2 3/8 ovl: implement overlayfs' ->evict_inode operation

In reply to Chengguang Xu

Implement overlayfs' ->evict_inode operation,
so that we can clear all inode dirty flags.

---
 fs/overlayfs/super.c | 7 +++++++
 1 file changed, 7 insertions(+)
fs/overlayfs/super.c +7 -0
index d1e546abce87..5d7e12b8bd1f 100644
@@ -390,11 +390,18 @@ static int ovl_remount(struct super_block *sb, int *flags, char *data)
 	return ret;
 }
 
+static void ovl_evict_inode(struct inode *inode)
+{
+	inode->i_state &= ~I_DIRTY_ALL;
+	clear_inode(inode);
+}
+
 static const struct super_operations ovl_super_operations = {
 	.alloc_inode	= ovl_alloc_inode,
 	.free_inode	= ovl_free_inode,
 	.destroy_inode	= ovl_destroy_inode,
 	.drop_inode	= generic_delete_inode,
+	.evict_inode	= ovl_evict_inode,
 	.put_super	= ovl_put_super,
 	.sync_fs	= ovl_sync_fs,
 	.statfs		= ovl_statfs,
Signature · 1 line
2.26.2
#8 lore raw

RFC PATCH v2 4/8 ovl: mark overlayfs' inode dirty on modification

In reply to Chengguang Xu

Mark overlayfs' inode dirty on modification so that we can recognize target inodes during syncfs.

---
 fs/overlayfs/inode.c     |  1 +
 fs/overlayfs/overlayfs.h |  4 ++++
 fs/overlayfs/util.c      | 14 ++++++++++++++
 3 files changed, 19 insertions(+)
fs/overlayfs/inode.c +1 -0
index f27fc5be34df..426f70ca5d5e 100644
@@ -468,6 +468,7 @@ int ovl_update_time(struct inode *inode, struct timespec64 *ts, int flags)
 		if (upperpath.dentry) {
 			touch_atime(&upperpath);
 			inode->i_atime = d_inode(upperpath.dentry)->i_atime;
+			ovl_mark_inode_dirty(inode);
 		}
 	}
 	return 0;
fs/overlayfs/overlayfs.h +4 -0
index f8880aa2ba0e..eaf1d5b05d8e 100644
@@ -247,6 +247,7 @@ static inline bool ovl_open_flags_need_copy_up(int flags)
 }
 
 /* util.c */
+void ovl_mark_inode_dirty(struct inode *inode);
 int ovl_want_write(struct dentry *dentry);
 void ovl_drop_write(struct dentry *dentry);
 struct dentry *ovl_workdir(struct dentry *dentry);
@@ -472,6 +473,9 @@ static inline void ovl_copyattr(struct inode *from, struct inode *to)
 	to->i_mtime = from->i_mtime;
 	to->i_ctime = from->i_ctime;
 	i_size_write(to, i_size_read(from));
+
+	if (ovl_inode_upper(to) && from->i_state & I_DIRTY_ALL)
+		ovl_mark_inode_dirty(to);
 }
 
 static inline void ovl_copyflags(struct inode *from, struct inode *to)
fs/overlayfs/util.c +14 -0
index 23f475627d07..a6f59df744ae 100644
@@ -950,3 +950,17 @@ char *ovl_get_redirect_xattr(struct ovl_fs *ofs, struct dentry *dentry,
 	kfree(buf);
 	return ERR_PTR(res);
 }
+
+/*
+ * We intentionally add I_DIRTY_SYNC flag regardless dirty flag
+ * of upper inode so that we have chance to invoke ->write_inode
+ * to re-dirty overlayfs' inode during writeback process.
+ */
+void ovl_mark_inode_dirty(struct inode *inode)
+{
+	struct inode *upper = ovl_inode_upper(inode);
+	unsigned long iflag = I_DIRTY_SYNC;
+
+	iflag |= upper->i_state & I_DIRTY_ALL;
+	__mark_inode_dirty(inode, iflag);
+}
Signature · 1 line
2.26.2
#9 lore raw

RFC PATCH v2 5/8 ovl: mark overlayfs' inode dirty on shared writable mmap

In reply to Chengguang Xu

Overlayfs cannot be notified when mmapped area gets dirty, so we need to proactively mark inode dirty in ->mmap operation.

---
 fs/overlayfs/file.c | 4 ++++
 1 file changed, 4 insertions(+)
fs/overlayfs/file.c +4 -0
index efccb7c1f9bc..cd6fcdfd81a9 100644
@@ -486,6 +486,10 @@ static int ovl_mmap(struct file *file, struct vm_area_struct *vma)
 		/* Drop reference count from new vm_file value */
 		fput(realfile);
 	} else {
+		if (vma->vm_flags & (VM_SHARED|VM_MAYSHARE) &&
+		    vma->vm_flags & (VM_WRITE|VM_MAYWRITE))
+			ovl_mark_inode_dirty(file_inode(file));
+
 		/* Drop reference count from previous vm_file value */
 		fput(file);
 	}
Signature · 1 line
2.26.2
reviewer
#10 lore raw

Re: [RFC PATCH v2 5/8] ovl: mark overlayfs' inode dirty on shared writable mmap

Quoted 13 lines · Chengguang Xu

Overlayfs cannot be notified when mmapped area gets dirty, so we need to proactively mark inode dirty in ->mmap operation.

---
 fs/overlayfs/file.c | 4 ++++
 1 file changed, 4 insertions(+)
fs/overlayfs/file.c +0 -0
index efccb7c1f9bc..cd6fcdfd81a9 100644

fs/overlayfs/file.c static int ovl_mmap(struct file *file, struct vm_area_struct *vma) · line 492

Quoted 7 lines
fs/overlayfs/file.c +4 -0
@@ -486,6 +486,10 @@ static int ovl_mmap(struct file *file, struct vm_area_struct *vma)
 		/* Drop reference count from new vm_file value */
 		fput(realfile);
 	} else {
+		if (vma->vm_flags & (VM_SHARED|VM_MAYSHARE) &&
+		    vma->vm_flags & (VM_WRITE|VM_MAYWRITE))
+			ovl_mark_inode_dirty(file_inode(file));
+

But does this work reliably? I mean once writeback runs, your inode (as well as upper inode) is cleaned. Then a page fault comes so file has dirty pages again and would need flushing but overlayfs inode stays clean? Am I missing something?

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#11 lore raw

Re: [RFC PATCH v2 5/8] ovl: mark overlayfs' inode dirty on shared writable mmap

---- 在 星期二, 2020-11-03 01:30:52 Jan Kara <[email protected]> 撰写 ----

Quoted 18 lines · Chengguang Xu
Quoted 17 lines

Overlayfs cannot be notified when mmapped area gets dirty, so we need to proactively mark inode dirty in ->mmap operation.

---
 fs/overlayfs/file.c | 4 ++++
 1 file changed, 4 insertions(+)
fs/overlayfs/file.c +1 -0
index efccb7c1f9bc..cd6fcdfd81a9 100644
@@ -486,6 +486,10 @@ static int ovl_mmap(struct file *file, struct vm_area_struct *vma)
         /* Drop reference count from new vm_file value */
         fput(realfile);
     } else {
+        if (vma->vm_flags & (VM_SHARED|VM_MAYSHARE) &&
Quoted 9 lines

fs/overlayfs/file.c static int ovl_mmap(struct file *file, struct vm_area_struct *vma) · line 492

Quoted 3 lines
fs/overlayfs/file.c +3 -0
+            vma->vm_flags & (VM_WRITE|VM_MAYWRITE))
+            ovl_mark_inode_dirty(file_inode(file));
+

But does this work reliably? I mean once writeback runs, your inode (as well as upper inode) is cleaned. Then a page fault comes so file has dirty pages again and would need flushing but overlayfs inode stays clean? Am I missing something?

Yeah, this is key point of this approach, in order to fix the issue I explicitly set I_DIRTY_SYNC flag in ovl_mark_inode_dirty(), so what i mean is during writeback we will call into ->write_inode() by this flag(I_DIRTY_SYNC) and at that place we get chance to check mapping and re-dirty overlay's inode. The code logic like below in ovl_write_inode().

    if (mapping_writably_mapped(upper->i_mapping) ||
         mapping_tagged(upper->i_mapping, PAGECACHE_TAG_WRITEBACK))
                 iflag |= I_DIRTY_PAGES; 

Thanks,
Chengguang

reviewer
#12 lore raw

Re: [RFC PATCH v2 5/8] ovl: mark overlayfs' inode dirty on shared writable mmap

Quoted 35 lines · Chengguang Xu

---- 在 星期二, 2020-11-03 01:30:52 Jan Kara <[email protected]> 撰写 ----

Quoted 18 lines · Chengguang Xu
Quoted 17 lines

Overlayfs cannot be notified when mmapped area gets dirty, so we need to proactively mark inode dirty in ->mmap operation.

---
 fs/overlayfs/file.c | 4 ++++
 1 file changed, 4 insertions(+)
fs/overlayfs/file.c +1 -0
index efccb7c1f9bc..cd6fcdfd81a9 100644
@@ -486,6 +486,10 @@ static int ovl_mmap(struct file *file, struct vm_area_struct *vma)
         /* Drop reference count from new vm_file value */
         fput(realfile);
     } else {
+        if (vma->vm_flags & (VM_SHARED|VM_MAYSHARE) &&
Quoted 9 lines

fs/overlayfs/file.c static int ovl_mmap(struct file *file, struct vm_area_struct *vma) · line 492

Quoted 3 lines
fs/overlayfs/file.c +3 -0
+            vma->vm_flags & (VM_WRITE|VM_MAYWRITE))
+            ovl_mark_inode_dirty(file_inode(file));
+

But does this work reliably? I mean once writeback runs, your inode (as well as upper inode) is cleaned. Then a page fault comes so file has dirty pages again and would need flushing but overlayfs inode stays clean? Am I missing something?

Yeah, this is key point of this approach, in order to fix the issue I explicitly set I_DIRTY_SYNC flag in ovl_mark_inode_dirty(), so what i mean is during writeback we will call into ->write_inode() by this flag(I_DIRTY_SYNC) and at that place we get chance to check mapping and re-dirty overlay's inode. The code logic like below in ovl_write_inode().

Quoted 3 lines
    if (mapping_writably_mapped(upper->i_mapping) ||
         mapping_tagged(upper->i_mapping, PAGECACHE_TAG_WRITEBACK))
                 iflag |= I_DIRTY_PAGES; 

OK, but suppose the upper mapping is clean at this moment (upper inode has been fully written out for whatever reason, but it is still mapped) so your overlayfs inode becomes clean as well. Then I don't see a mechanism which would make your overlayfs inode dirty again when a write to mmap happens, set_page_dirty() will end up marking upper inode with I_DIRTY_PAGES flag.

Note that ovl_mmap() gets called only at mmap(2) syscall time but then pages get faulted in, dirtied, cleaned fully at discretion of the mm / writeback subsystem.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
maintainer
#13 lore raw

Re: [RFC PATCH v2 5/8] ovl: mark overlayfs' inode dirty on shared writable mmap

Quoted 47 lines · Jan Kara
Quoted 35 lines · Chengguang Xu

---- 在 星期二, 2020-11-03 01:30:52 Jan Kara <[email protected]> 撰写 ----

Quoted 18 lines · Chengguang Xu
Quoted 17 lines

Overlayfs cannot be notified when mmapped area gets dirty, so we need to proactively mark inode dirty in ->mmap operation.

---
 fs/overlayfs/file.c | 4 ++++
 1 file changed, 4 insertions(+)
fs/overlayfs/file.c +1 -0
index efccb7c1f9bc..cd6fcdfd81a9 100644
@@ -486,6 +486,10 @@ static int ovl_mmap(struct file *file, struct vm_area_struct *vma)
         /* Drop reference count from new vm_file value */
         fput(realfile);
     } else {
+        if (vma->vm_flags & (VM_SHARED|VM_MAYSHARE) &&
Quoted 9 lines

fs/overlayfs/file.c static int ovl_mmap(struct file *file, struct vm_area_struct *vma) · line 492

Quoted 3 lines
fs/overlayfs/file.c +3 -0
+            vma->vm_flags & (VM_WRITE|VM_MAYWRITE))
+            ovl_mark_inode_dirty(file_inode(file));
+

But does this work reliably? I mean once writeback runs, your inode (as well as upper inode) is cleaned. Then a page fault comes so file has dirty pages again and would need flushing but overlayfs inode stays clean? Am I missing something?

Yeah, this is key point of this approach, in order to fix the issue I explicitly set I_DIRTY_SYNC flag in ovl_mark_inode_dirty(), so what i mean is during writeback we will call into ->write_inode() by this flag(I_DIRTY_SYNC) and at that place we get chance to check mapping and re-dirty overlay's inode. The code logic like below in ovl_write_inode().

Quoted 3 lines
    if (mapping_writably_mapped(upper->i_mapping) ||
         mapping_tagged(upper->i_mapping, PAGECACHE_TAG_WRITEBACK))
                 iflag |= I_DIRTY_PAGES;

OK, but suppose the upper mapping is clean at this moment (upper inode has been fully written out for whatever reason, but it is still mapped) so your overlayfs inode becomes clean as well. Then I don't see a mechanism which would make your overlayfs inode dirty again when a write to mmap happens, set_page_dirty() will end up marking upper inode with I_DIRTY_PAGES flag.

Quoted 4 lines

Note that ovl_mmap() gets called only at mmap(2) syscall time but then pages get faulted in, dirtied, cleaned fully at discretion of the mm / writeback subsystem.

Perhaps I will add some background.

What I suggested was to maintain a "suspect list" in addition to the dirty ovl inodes.

ovl inode is added to the suspect list on mmap (writable) and removed from the suspect list on release() flush() or on sync_fs() if real inode is no longer writably mapped.

There was another variant where ovl inode is added to suspect list on open for write and removed from suspect list on release() flush() or sync_fs() if real inode is not inode_is_open_for_write().

In both cases the list will have inodes whose real is not dirty, but in both cases
the list shouldn't be terribly large to traverse on sync_fs().

Chengguang tried to implement the idea without an actual list by re-dirtying the "suspect" inodes on every write_inode(), but I personally have no idea if his idea works.

I think we can resort to using an actual suspect list if you say that it cannot work like this?

Thanks,
Amir.

reviewer
#14 lore raw

Re: [RFC PATCH v2 5/8] ovl: mark overlayfs' inode dirty on shared writable mmap

Quoted 70 lines · Amir Goldstein
Quoted 47 lines · Jan Kara
Quoted 35 lines · Chengguang Xu

---- 在 星期二, 2020-11-03 01:30:52 Jan Kara <[email protected]> 撰写 ----

Quoted 18 lines · Chengguang Xu
Quoted 17 lines

Overlayfs cannot be notified when mmapped area gets dirty, so we need to proactively mark inode dirty in ->mmap operation.

---
 fs/overlayfs/file.c | 4 ++++
 1 file changed, 4 insertions(+)
fs/overlayfs/file.c +1 -0
index efccb7c1f9bc..cd6fcdfd81a9 100644
@@ -486,6 +486,10 @@ static int ovl_mmap(struct file *file, struct vm_area_struct *vma)
         /* Drop reference count from new vm_file value */
         fput(realfile);
     } else {
+        if (vma->vm_flags & (VM_SHARED|VM_MAYSHARE) &&
Quoted 9 lines

fs/overlayfs/file.c static int ovl_mmap(struct file *file, struct vm_area_struct *vma) · line 492

Quoted 3 lines
fs/overlayfs/file.c +3 -0
+            vma->vm_flags & (VM_WRITE|VM_MAYWRITE))
+            ovl_mark_inode_dirty(file_inode(file));
+

But does this work reliably? I mean once writeback runs, your inode (as well as upper inode) is cleaned. Then a page fault comes so file has dirty pages again and would need flushing but overlayfs inode stays clean? Am I missing something?

Yeah, this is key point of this approach, in order to fix the issue I explicitly set I_DIRTY_SYNC flag in ovl_mark_inode_dirty(), so what i mean is during writeback we will call into ->write_inode() by this flag(I_DIRTY_SYNC) and at that place we get chance to check mapping and re-dirty overlay's inode. The code logic like below in ovl_write_inode().

Quoted 3 lines
    if (mapping_writably_mapped(upper->i_mapping) ||
         mapping_tagged(upper->i_mapping, PAGECACHE_TAG_WRITEBACK))
                 iflag |= I_DIRTY_PAGES;

OK, but suppose the upper mapping is clean at this moment (upper inode has been fully written out for whatever reason, but it is still mapped) so your overlayfs inode becomes clean as well. Then I don't see a mechanism which would make your overlayfs inode dirty again when a write to mmap happens, set_page_dirty() will end up marking upper inode with I_DIRTY_PAGES flag.

Quoted 4 lines

Note that ovl_mmap() gets called only at mmap(2) syscall time but then pages get faulted in, dirtied, cleaned fully at discretion of the mm / writeback subsystem.

Perhaps I will add some background.

What I suggested was to maintain a "suspect list" in addition to the dirty ovl inodes.

ovl inode is added to the suspect list on mmap (writable) and removed from the suspect list on release() flush() or on sync_fs() if real inode is no longer writably mapped.

There was another variant where ovl inode is added to suspect list on open for write and removed from suspect list on release() flush() or sync_fs() if real inode is not inode_is_open_for_write().

In both cases the list will have inodes whose real is not dirty, but in both cases
the list shouldn't be terribly large to traverse on sync_fs().

Quoted 6 lines

Chengguang tried to implement the idea without an actual list by re-dirtying the "suspect" inodes on every write_inode(), but I personally have no idea if his idea works.

I think we can resort to using an actual suspect list if you say that it cannot work like this?

Yeah, the suspect list (i.e., additional list of inodes to check on sync) you describe should work fine. Also the "keep suspect inode dirty" idea of Chengguang could work fine but we'd have to use something like inode_is_open_for_write() or inode_is_writeably_mapped() (which would need to be implemented but it should be easy vma_interval_tree_foreach() walk checking each found VMA for vma->vm_flags & VM_WRITE) for checking whether inode should be redirtied or not.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#15 lore raw

Re: [RFC PATCH v2 5/8] ovl: mark overlayfs' inode dirty on shared writable mmap

---- 在 星期四, 2020-11-05 23:54:34 Jan Kara <[email protected]> 撰写 ----

Quoted 75 lines · Amir Goldstein
Quoted 74 lines · Jan Kara
Quoted 47 lines · Chengguang Xu
Quoted 35 lines

---- 在 星期二, 2020-11-03 01:30:52 Jan Kara <[email protected]> 撰写 ----

Quoted 18 lines · Chengguang Xu
Quoted 17 lines

Overlayfs cannot be notified when mmapped area gets dirty, so we need to proactively mark inode dirty in ->mmap operation.

---
 fs/overlayfs/file.c | 4 ++++
 1 file changed, 4 insertions(+)
fs/overlayfs/file.c +1 -0
index efccb7c1f9bc..cd6fcdfd81a9 100644
@@ -486,6 +486,10 @@ static int ovl_mmap(struct file *file, struct vm_area_struct *vma)
         /* Drop reference count from new vm_file value */
         fput(realfile);
     } else {
+        if (vma->vm_flags & (VM_SHARED|VM_MAYSHARE) &&
Quoted 9 lines

fs/overlayfs/file.c static int ovl_mmap(struct file *file, struct vm_area_struct *vma) · line 492

Quoted 3 lines
fs/overlayfs/file.c +3 -0
+            vma->vm_flags & (VM_WRITE|VM_MAYWRITE))
+            ovl_mark_inode_dirty(file_inode(file));
+

But does this work reliably? I mean once writeback runs, your inode (as well as upper inode) is cleaned. Then a page fault comes so file has dirty pages again and would need flushing but overlayfs inode stays clean? Am I missing something?

Yeah, this is key point of this approach, in order to fix the issue I explicitly set I_DIRTY_SYNC flag in ovl_mark_inode_dirty(), so what i mean is during writeback we will call into ->write_inode() by this flag(I_DIRTY_SYNC) and at that place we get chance to check mapping and re-dirty overlay's inode. The code logic like below in ovl_write_inode().

Quoted 3 lines
    if (mapping_writably_mapped(upper->i_mapping) ||
         mapping_tagged(upper->i_mapping, PAGECACHE_TAG_WRITEBACK))
                 iflag |= I_DIRTY_PAGES;

OK, but suppose the upper mapping is clean at this moment (upper inode has been fully written out for whatever reason, but it is still mapped) so your overlayfs inode becomes clean as well. Then I don't see a mechanism which would make your overlayfs inode dirty again when a write to mmap happens, set_page_dirty() will end up marking upper inode with I_DIRTY_PAGES flag.

Quoted 4 lines

Note that ovl_mmap() gets called only at mmap(2) syscall time but then pages get faulted in, dirtied, cleaned fully at discretion of the mm / writeback subsystem.

Perhaps I will add some background.

What I suggested was to maintain a "suspect list" in addition to the dirty ovl inodes.

ovl inode is added to the suspect list on mmap (writable) and removed from the suspect list on release() flush() or on sync_fs() if real inode is no longer writably mapped.

There was another variant where ovl inode is added to suspect list on open for write and removed from suspect list on release() flush() or sync_fs() if real inode is not inode_is_open_for_write().

In both cases the list will have inodes whose real is not dirty, but in both cases
the list shouldn't be terribly large to traverse on sync_fs().

Chengguang tried to implement the idea without an actual list by re-dirtying the "suspect" inodes on every write_inode(), but I personally have no idea if his idea works.

Quoted 5 lines
Quoted 2 lines

I think we can resort to using an actual suspect list if you say that it cannot work like this?

Yeah, the suspect list (i.e., additional list of inodes to check on sync) you describe should work fine.

I think this solution still has the problem we have met in below thread[1] The main problem is the state combination of clean overlayfs' inode && dirty upper inode.

[1] https://www.spinics.net/lists/linux-unionfs/msg07448.html

Quoted 7 lines

Also the "keep suspect inode dirty" idea
of Chengguang could work fine but we'd have to use something like inode_is_open_for_write() or inode_is_writeably_mapped() (which would need to be implemented but it should be easy vma_interval_tree_foreach() walk checking each found VMA for vma->vm_flags & VM_WRITE) for checking whether inode should be redirtied or not.

I'm curious that isn't it enough to check i_mmap_writable by mapping_writably_mapped() ? Am I missing something?

Thanks,
Chengguang

reviewer
#16 lore raw

Re: [RFC PATCH v2 5/8] ovl: mark overlayfs' inode dirty on shared writable mmap

Quoted 76 lines · Chengguang Xu

---- 在 星期四, 2020-11-05 23:54:34 Jan Kara <[email protected]> 撰写 ----

Quoted 75 lines · Amir Goldstein
Quoted 74 lines · Jan Kara
Quoted 47 lines · Chengguang Xu
Quoted 35 lines

---- 在 星期二, 2020-11-03 01:30:52 Jan Kara <[email protected]> 撰写 ----

Quoted 18 lines · Chengguang Xu
Quoted 17 lines

Overlayfs cannot be notified when mmapped area gets dirty, so we need to proactively mark inode dirty in ->mmap operation.

---
 fs/overlayfs/file.c | 4 ++++
 1 file changed, 4 insertions(+)
fs/overlayfs/file.c +1 -0
index efccb7c1f9bc..cd6fcdfd81a9 100644
@@ -486,6 +486,10 @@ static int ovl_mmap(struct file *file, struct vm_area_struct *vma)
         /* Drop reference count from new vm_file value */
         fput(realfile);
     } else {
+        if (vma->vm_flags & (VM_SHARED|VM_MAYSHARE) &&
Quoted 9 lines

fs/overlayfs/file.c static int ovl_mmap(struct file *file, struct vm_area_struct *vma) · line 492

Quoted 3 lines
fs/overlayfs/file.c +3 -0
+            vma->vm_flags & (VM_WRITE|VM_MAYWRITE))
+            ovl_mark_inode_dirty(file_inode(file));
+

But does this work reliably? I mean once writeback runs, your inode (as well as upper inode) is cleaned. Then a page fault comes so file has dirty pages again and would need flushing but overlayfs inode stays clean? Am I missing something?

Yeah, this is key point of this approach, in order to fix the issue I explicitly set I_DIRTY_SYNC flag in ovl_mark_inode_dirty(), so what i mean is during writeback we will call into ->write_inode() by this flag(I_DIRTY_SYNC) and at that place we get chance to check mapping and re-dirty overlay's inode. The code logic like below in ovl_write_inode().

Quoted 3 lines
    if (mapping_writably_mapped(upper->i_mapping) ||
         mapping_tagged(upper->i_mapping, PAGECACHE_TAG_WRITEBACK))
                 iflag |= I_DIRTY_PAGES;

OK, but suppose the upper mapping is clean at this moment (upper inode has been fully written out for whatever reason, but it is still mapped) so your overlayfs inode becomes clean as well. Then I don't see a mechanism which would make your overlayfs inode dirty again when a write to mmap happens, set_page_dirty() will end up marking upper inode with I_DIRTY_PAGES flag.

Quoted 4 lines

Note that ovl_mmap() gets called only at mmap(2) syscall time but then pages get faulted in, dirtied, cleaned fully at discretion of the mm / writeback subsystem.

Perhaps I will add some background.

What I suggested was to maintain a "suspect list" in addition to the dirty ovl inodes.

ovl inode is added to the suspect list on mmap (writable) and removed from the suspect list on release() flush() or on sync_fs() if real inode is no longer writably mapped.

There was another variant where ovl inode is added to suspect list on open for write and removed from suspect list on release() flush() or sync_fs() if real inode is not inode_is_open_for_write().

In both cases the list will have inodes whose real is not dirty, but in both cases
the list shouldn't be terribly large to traverse on sync_fs().

Chengguang tried to implement the idea without an actual list by re-dirtying the "suspect" inodes on every write_inode(), but I personally have no idea if his idea works.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

I think we can resort to using an actual suspect list if you say that it cannot work like this?

Yeah, the suspect list (i.e., additional list of inodes to check on sync) you describe should work fine.

I think this solution still has the problem we have met in below thread[1] The main problem is the state combination of clean overlayfs' inode && dirty upper inode.

But I think the scheme Amir proposed and I detailed in my previous email should prevent that state. Because while the inode is mapped, it will be kept in the dirty list. So which scenario do you think would lead to clean overlayfs inode and dirty upper inode?

Quoted 10 lines

[1] https://www.spinics.net/lists/linux-unionfs/msg07448.html

Quoted 7 lines

Also the "keep suspect inode dirty" idea
of Chengguang could work fine but we'd have to use something like inode_is_open_for_write() or inode_is_writeably_mapped() (which would need to be implemented but it should be easy vma_interval_tree_foreach() walk checking each found VMA for vma->vm_flags & VM_WRITE) for checking whether inode should be redirtied or not.

Quoted 2 lines

I'm curious that isn't it enough to check i_mmap_writable by mapping_writably_mapped() ? Am I missing something?

What is i_mmap_writeable? I've grepped the tree and didn't find anything like that...

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#17 lore raw

Re: [RFC PATCH v2 5/8] ovl: mark overlayfs' inode dirty on shared writable mmap

---- 在 星期五, 2020-11-06 16:50:23 Jan Kara <[email protected]> 撰写 ----

Quoted 83 lines · Chengguang Xu
Quoted 82 lines

---- 在 星期四, 2020-11-05 23:54:34 Jan Kara <[email protected]> 撰写 ----

Quoted 80 lines · Amir Goldstein
Quoted 70 lines · Jan Kara
Quoted 47 lines · Chengguang Xu
Quoted 35 lines

---- 在 星期二, 2020-11-03 01:30:52 Jan Kara <[email protected]> 撰写 ----

Quoted 18 lines · Chengguang Xu
Quoted 17 lines

Overlayfs cannot be notified when mmapped area gets dirty, so we need to proactively mark inode dirty in ->mmap operation.

---
 fs/overlayfs/file.c | 4 ++++
 1 file changed, 4 insertions(+)
fs/overlayfs/file.c +1 -0
index efccb7c1f9bc..cd6fcdfd81a9 100644
@@ -486,6 +486,10 @@ static int ovl_mmap(struct file *file, struct vm_area_struct *vma)
         /* Drop reference count from new vm_file value */
         fput(realfile);
     } else {
+        if (vma->vm_flags & (VM_SHARED|VM_MAYSHARE) &&
Quoted 9 lines

fs/overlayfs/file.c static int ovl_mmap(struct file *file, struct vm_area_struct *vma) · line 492

Quoted 3 lines
fs/overlayfs/file.c +3 -0
+            vma->vm_flags & (VM_WRITE|VM_MAYWRITE))
+            ovl_mark_inode_dirty(file_inode(file));
+

But does this work reliably? I mean once writeback runs, your inode (as well as upper inode) is cleaned. Then a page fault comes so file has dirty pages again and would need flushing but overlayfs inode stays clean? Am I missing something?

Yeah, this is key point of this approach, in order to fix the issue I explicitly set I_DIRTY_SYNC flag in ovl_mark_inode_dirty(), so what i mean is during writeback we will call into ->write_inode() by this flag(I_DIRTY_SYNC) and at that place we get chance to check mapping and re-dirty overlay's inode. The code logic like below in ovl_write_inode().

Quoted 3 lines
    if (mapping_writably_mapped(upper->i_mapping) ||
         mapping_tagged(upper->i_mapping, PAGECACHE_TAG_WRITEBACK))
                 iflag |= I_DIRTY_PAGES;

OK, but suppose the upper mapping is clean at this moment (upper inode has been fully written out for whatever reason, but it is still mapped) so your overlayfs inode becomes clean as well. Then I don't see a mechanism which would make your overlayfs inode dirty again when a write to mmap happens, set_page_dirty() will end up marking upper inode with I_DIRTY_PAGES flag.

Quoted 4 lines

Note that ovl_mmap() gets called only at mmap(2) syscall time but then pages get faulted in, dirtied, cleaned fully at discretion of the mm / writeback subsystem.

Perhaps I will add some background.

What I suggested was to maintain a "suspect list" in addition to the dirty ovl inodes.

ovl inode is added to the suspect list on mmap (writable) and removed from the suspect list on release() flush() or on sync_fs() if real inode is no longer writably mapped.

There was another variant where ovl inode is added to suspect list on open for write and removed from suspect list on release() flush() or sync_fs() if real inode is not inode_is_open_for_write().

In both cases the list will have inodes whose real is not dirty, but in both cases
the list shouldn't be terribly large to traverse on sync_fs().

Quoted 6 lines

Chengguang tried to implement the idea without an actual list by re-dirtying the "suspect" inodes on every write_inode(), but I personally have no idea if his idea works.

I think we can resort to using an actual suspect list if you say that it cannot work like this?

Yeah, the suspect list (i.e., additional list of inodes to check on sync) you describe should work fine.

Quoted 7 lines
Quoted 2 lines

I think this solution still has the problem we have met in below thread[1] The main problem is the state combination of clean overlayfs' inode && dirty upper inode.

But I think the scheme Amir proposed and I detailed in my previous email should prevent that state. Because while the inode is mapped, it will be kept in the dirty list. So which scenario do you think would lead to clean overlayfs inode and dirty upper inode?

If keeping in the dirty list means making overlayfs inode dirty, then I think we don't need extra list for that, vfs itself has writeback list and the solution will be exactly the same as mine(re-dirty) . Right?

Quoted 11 lines
Quoted 11 lines

[1] https://www.spinics.net/lists/linux-unionfs/msg07448.html

Quoted 7 lines

Also the "keep suspect inode dirty" idea
of Chengguang could work fine but we'd have to use something like inode_is_open_for_write() or inode_is_writeably_mapped() (which would need to be implemented but it should be easy vma_interval_tree_foreach() walk checking each found VMA for vma->vm_flags & VM_WRITE) for checking whether inode should be redirtied or not.

Quoted 6 lines
Quoted 2 lines

I'm curious that isn't it enough to check i_mmap_writable by mapping_writably_mapped() ? Am I missing something?

What is i_mmap_writeable? I've grepped the tree and didn't find anything like that...

Maybe spelling mistake? The reason I check this is I'm afraid of the permission change of vma by mprotect(2).

Thanks,
Chenguang

#18 lore raw

RFC PATCH v2 6/8 ovl: implement overlayfs' ->write_inode operation

In reply to Chengguang Xu

Implement overlayfs' ->write_inode to sync dirty data and redirty overlayfs' inode if necessary.

---
 fs/overlayfs/super.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
fs/overlayfs/super.c +26 -0
index 5d7e12b8bd1f..960e79e7a8b5 100644
@@ -15,6 +15,7 @@
 #include <linux/seq_file.h>
 #include <linux/posix_acl_xattr.h>
 #include <linux/exportfs.h>
+#include <linux/writeback.h>
 #include "overlayfs.h"
 
 MODULE_AUTHOR("Miklos Szeredi <[email protected]>");
@@ -396,11 +397,36 @@ static void ovl_evict_inode (struct inode *inode)
 	clear_inode(inode);
 }
 
+static int ovl_write_inode(struct inode *inode,
+			   struct writeback_control *wbc)
+{
+	struct inode *upper = ovl_inode_upper(inode);
+	unsigned long iflag = 0;
+	int ret = 0;
+
+	if (!upper)
+		return 0;
+
+	if (upper->i_sb->s_op->write_inode)
+		ret = upper->i_sb->s_op->write_inode(inode, wbc);
+
+	iflag |= upper->i_state & I_DIRTY_ALL;
+	if (mapping_writably_mapped(upper->i_mapping) ||
+	    mapping_tagged(upper->i_mapping, PAGECACHE_TAG_WRITEBACK))
+		iflag |= I_DIRTY_PAGES;
+
+	if (iflag)
+		ovl_mark_inode_dirty(inode);
+
+	return ret;
+}
+
 static const struct super_operations ovl_super_operations = {
 	.alloc_inode	= ovl_alloc_inode,
 	.free_inode	= ovl_free_inode,
 	.destroy_inode	= ovl_destroy_inode,
 	.drop_inode	= generic_delete_inode,
+	.write_inode	= ovl_write_inode,
 	.evict_inode	= ovl_evict_inode,
 	.put_super	= ovl_put_super,
 	.sync_fs	= ovl_sync_fs,
Signature · 1 line
2.26.2
#19 lore raw

RFC PATCH v2 7/8 ovl: cache dirty overlayfs' inode

In reply to Chengguang Xu

Now drop overlayfs' inode will sync dirty data,
so we change to only drop clean inode.

---
 fs/overlayfs/super.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
fs/overlayfs/super.c +11 -1
index 960e79e7a8b5..1d04117fb6ad 100644
@@ -421,11 +421,21 @@ static int ovl_write_inode(struct inode *inode,
 	return ret;
 }
 
+int ovl_drop_inode(struct inode *inode)
+{
+	struct inode *upper = ovl_inode_upper(inode);
+
+	if (!upper || !(inode->i_state & I_DIRTY_ALL))
+		return 1;
+
+	return generic_drop_inode(inode);
+}
+
 static const struct super_operations ovl_super_operations = {
 	.alloc_inode	= ovl_alloc_inode,
 	.free_inode	= ovl_free_inode,
 	.destroy_inode	= ovl_destroy_inode,
-	.drop_inode	= generic_delete_inode,
+	.drop_inode	= ovl_drop_inode,
 	.write_inode	= ovl_write_inode,
 	.evict_inode	= ovl_evict_inode,
 	.put_super	= ovl_put_super,
Signature · 1 line
2.26.2
#20 lore raw

RFC PATCH v2 8/8 ovl: implement containerized syncfs for overlayfs

In reply to Chengguang Xu

Now overlayfs can only sync dirty inode during
syncfs, so remove unnecessary sync_filesystem()
on upper file system.

---
 fs/overlayfs/super.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
fs/overlayfs/super.c +5 -3
index 1d04117fb6ad..df33e8c8f1d0 100644
@@ -271,8 +271,7 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
 	 * Not called for sync(2) call or an emergency sync (SB_I_SKIP_SYNC).
 	 * All the super blocks will be iterated, including upper_sb.
 	 *
-	 * If this is a syncfs(2) call, then we do need to call
-	 * sync_filesystem() on upper_sb, but enough if we do it when being
+	 * If this is a syncfs(2) call, it will be enough we do it when being
 	 * called with wait == 1.
 	 */
 	if (!wait)
@@ -281,7 +280,10 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
 	upper_sb = ovl_upper_mnt(ofs)->mnt_sb;
 
 	down_read(&upper_sb->s_umount);
-	ret = sync_filesystem(upper_sb);
+	if (upper_sb->s_op->sync_fs)
+		ret = upper_sb->s_op->sync_fs(upper_sb, wait);
+	if (!ret)
+		ret = sync_blockdev(upper_sb->s_bdev);
 	up_read(&upper_sb->s_umount);
 
 	return ret;
Signature · 1 line
2.26.2
maintainer
#21 lore raw
In reply to Chengguang Xu
Quoted 11 lines · Chengguang Xu

Current syncfs(2) syscall on overlayfs just calls sync_filesystem() on upper_sb to synchronize whole dirty inodes in upper filesystem regardless of the overlay ownership of the inode. In the use case of container, when multiple containers using the same underlying upper filesystem, it has some shortcomings as below.

(1) Performance
Synchronization is probably heavy because it actually syncs unnecessary inodes for target overlayfs.

Quoted 8 lines

(2) Interference
Unplanned synchronization will probably impact IO performance of unrelated container processes on the other overlayfs.

This series try to implement containerized syncfs for overlayfs so that only sync target dirty upper inodes which are belong to specific overlayfs instance. By doing this, it is able to reduce cost of synchronization and will not seriously impact IO performance of unrelated processes.

Series looks good at first glance. Still need to do an in-depth review.

In the meantime can you post some numbers showing the performance improvements?

Thanks,
Miklos

#22 lore raw

---- 在 星期五, 2020-10-30 23:46:00 Miklos Szeredi <[email protected]> 撰写 ----

Quoted 16 lines · Chengguang Xu
Quoted 15 lines

Current syncfs(2) syscall on overlayfs just calls sync_filesystem() on upper_sb to synchronize whole dirty inodes in upper filesystem regardless of the overlay ownership of the inode. In the use case of container, when multiple containers using the same underlying upper filesystem, it has some shortcomings as below.

(1) Performance
Synchronization is probably heavy because it actually syncs unnecessary inodes for target overlayfs.

(2) Interference
Unplanned synchronization will probably impact IO performance of unrelated container processes on the other overlayfs.

Quoted 9 lines
Quoted 4 lines

This series try to implement containerized syncfs for overlayfs so that only sync target dirty upper inodes which are belong to specific overlayfs instance. By doing this, it is able to reduce cost of synchronization and will not seriously impact IO performance of unrelated processes.

Series looks good at first glance. Still need to do an in-depth review.

In the meantime can you post some numbers showing the performance improvements?

Actually, the performance improvement depends on workload and the number of overlay instances.

I did a very simple test on my dev box (intel NUC ) and the result seems quite obvious.

I made two overlay instances and untar a kernel source to an overlay then compare

umount time(sys part) of two instances.

------------------------------------------
test env:
Intel(R) Core(TM) i7-8809G CPU @ 3.10GHz 8core
32GB memory
Samsung NVMe SSD 970 pro

test script:

[root@localhost ovl]# cat do.sh
#!/bin/sh

rm -rf upper/* &> /dev/null
echo 3 > /proc/sys/vm/drop_caches

mount -t overlay overlay -o lowerdir=lower,workdir=work,upperdir=upper merged mount -t overlay overlay -o lowerdir=lower2,workdir=work2,upperdir=upper2 merged2

tar Jxvf linux-5.9.tar.xz -C merged &> /dev/null

time umount merged2
time umount merged

---------------------------------
test result:

[root@localhost ovl]# seq 3 | while read line ; do sh -x do.sh ; done
+ rm -rf upper/linux-5.9
+ echo 3
+ mount -t overlay overlay -o lowerdir=lower,workdir=work,upperdir=upper merged
+ mount -t overlay overlay -o lowerdir=lower2,workdir=work2,upperdir=upper2 merged2
+ tar Jxvf linux-5.9.tar.xz -C merged
+ umount merged2

real    0m0.437s
user    0m0.001s
sys     0m0.001s

+ umount merged

real    0m2.107s
user    0m0.002s
sys     0m0.090s

+ rm -rf upper/linux-5.9
+ echo 3
+ mount -t overlay overlay -o lowerdir=lower,workdir=work,upperdir=upper merged
+ mount -t overlay overlay -o lowerdir=lower2,workdir=work2,upperdir=upper2 merged2
+ tar Jxvf linux-5.9.tar.xz -C merged
+ umount merged2

real    0m0.443s
user    0m0.002s
sys     0m0.001s

+ umount merged

real    0m2.032s
user    0m0.001s
sys     0m0.105s

+ rm -rf upper/linux-5.9
+ echo 3
+ mount -t overlay overlay -o lowerdir=lower,workdir=work,upperdir=upper merged
+ mount -t overlay overlay -o lowerdir=lower2,workdir=work2,upperdir=upper2 merged2
+ tar Jxvf linux-5.9.tar.xz -C merged
+ umount merged2

real    0m0.263s
user    0m0.001s
sys     0m0.001s

+ umount merged

real    0m2.094s
user    0m0.000s
sys     0m0.083s

[root@localhost o

Thanks,
Chengguang