Skip to contentSkip to conversation

implement containerized syncfs for overlayfs

RFC PATCH Superseded Chengguang Xu 3 participants 24 messages Nov 13, 2020 – Apr 14, 2021

Subsystems (3)

v4 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

v4 cover implement containerized syncfs for overlayfs

Current syncfs(2) syscall on overlayfs just calls sync_filesystem() on upper_sb to synchronize whole dirty inodes in upper filesystem regardless of the overlay ownership of the inode. In the use case of container, when multiple containers using the same underlying upper filesystem, it has some shortcomings as below.

(1) Performance
Synchronization is probably heavy because it actually syncs unnecessary inodes for target overlayfs.

(2) Interference
Unplanned synchronization will probably impact IO performance of unrelated container processes on the other overlayfs.

This series try to implement containerized syncfs for overlayfs so that only sync target dirty upper inodes which are belong to specific overlayfs instance. By doing this, it is able to reduce cost of synchronization and will not seriously impact IO performance of unrelated processes.

v1->v2:
- Mark overlayfs' inode dirty itself instead of adding notification mechanism to vfs inode.

v2->v3:
- Introduce overlayfs' extra syncfs wait list to wait target upper inodes in ->sync_fs.

v3->v4:
- Using wait_sb_inodes() to wait syncing upper inodes.
- Mark overlay inode dirty only when having upper inode and VM_SHARED flag in ovl_mmap().
- Check upper i_state after checking upper mmap state
in ovl_write_inode.

Chengguang Xu (9):
ovl: setup overlayfs' private bdi
ovl: implement ->writepages operation
ovl: implement overlayfs' ->evict_inode operation
ovl: mark overlayfs' inode dirty on modification
ovl: mark overlayfs' inode dirty on shared mmap
ovl: implement overlayfs' ->write_inode operation
ovl: cache dirty overlayfs' inode
fs: export wait_sb_inodes()
ovl: implement containerized syncfs for overlayfs

 fs/fs-writeback.c         |  3 +-
 fs/overlayfs/file.c       |  3 ++
 fs/overlayfs/inode.c      | 15 ++++++++++
 fs/overlayfs/overlayfs.h  |  4 +++
 fs/overlayfs/super.c      | 63 ++++++++++++++++++++++++++++++++++++---
 fs/overlayfs/util.c       | 14 +++++++++
 include/linux/writeback.h |  1 +
 7 files changed, 98 insertions(+), 5 deletions(-)
Signature · 1 line
2.26.2
#1 lore raw

RFC PATCH v4 1/9 ovl: setup overlayfs' private bdi

Setup overlayfs' private bdi so that we can collect overlayfs' own dirty inodes.

---
 fs/overlayfs/super.c | 4 ++++
 1 file changed, 4 insertions(+)
fs/overlayfs/super.c +4 -0
index 290983bcfbb3..d1e546abce87 100644
@@ -1869,6 +1869,10 @@ static int ovl_fill_super(struct super_block *sb, void *data, int silent)
 	if (!ofs)
 		goto out;
 
+	err = super_setup_bdi(sb);
+	if (err)
+		goto out_err;
+
 	ofs->creator_cred = cred = prepare_creds();
 	if (!cred)
 		goto out_err;
Signature · 1 line
2.26.2
#2 lore raw

RFC PATCH v4 2/9 ovl: implement ->writepages operation

In reply to Chengguang Xu

Implement overlayfs' ->writepages operation so that we can sync dirty data/metadata to upper filesystem.

---
 fs/overlayfs/inode.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
fs/overlayfs/inode.c +14 -0
index b584dca845ba..8cfa75e86f56 100644
@@ -11,6 +11,7 @@
 #include <linux/posix_acl.h>
 #include <linux/ratelimit.h>
 #include <linux/fiemap.h>
+#include <linux/writeback.h>
 #include "overlayfs.h"
 
 
@@ -516,7 +517,20 @@ static const struct inode_operations ovl_special_inode_operations = {
 	.update_time	= ovl_update_time,
 };
 
+static int ovl_writepages(struct address_space *mapping,
+			  struct writeback_control *wbc)
+{
+	struct inode *inode = mapping->host;
+	struct ovl_fs *ofs = inode->i_sb->s_fs_info;
+	struct inode *upper = ovl_inode_upper(inode);
+
+	if (!ovl_should_sync(ofs))
+		return 0;
+	return sync_inode(upper, wbc);
+}
+
 static const struct address_space_operations ovl_aops = {
+	.writepages		= ovl_writepages,
 	/* For O_DIRECT dentry_open() checks f_mapping->a_ops->direct_IO */
 	.direct_IO		= noop_direct_IO,
 };
Signature · 1 line
2.26.2
#3 lore raw

RFC PATCH v4 3/9 ovl: implement overlayfs' ->evict_inode operation

In reply to Chengguang Xu

Implement overlayfs' ->evict_inode operation,
so that we can clear all inode dirty flags.

---
 fs/overlayfs/super.c | 7 +++++++
 1 file changed, 7 insertions(+)
fs/overlayfs/super.c +7 -0
index d1e546abce87..883172ac8a12 100644
@@ -390,11 +390,18 @@ static int ovl_remount(struct super_block *sb, int *flags, char *data)
 	return ret;
 }
 
+static void ovl_evict_inode(struct inode *inode)
+{
+	inode->i_state &= ~I_DIRTY_ALL;
+	clear_inode(inode);
+}
+
 static const struct super_operations ovl_super_operations = {
 	.alloc_inode	= ovl_alloc_inode,
 	.free_inode	= ovl_free_inode,
 	.destroy_inode	= ovl_destroy_inode,
 	.drop_inode	= generic_delete_inode,
+	.evict_inode	= ovl_evict_inode,
 	.put_super	= ovl_put_super,
 	.sync_fs	= ovl_sync_fs,
 	.statfs		= ovl_statfs,
Signature · 1 line
2.26.2
#4 lore raw

RFC PATCH v4 4/9 ovl: mark overlayfs' inode dirty on modification

In reply to Chengguang Xu

Mark overlayfs' inode dirty on modification so that we can recognize target inodes during syncfs.

---
 fs/overlayfs/inode.c     |  1 +
 fs/overlayfs/overlayfs.h |  4 ++++
 fs/overlayfs/util.c      | 14 ++++++++++++++
 3 files changed, 19 insertions(+)
fs/overlayfs/inode.c +1 -0
index 8cfa75e86f56..342693657ab0 100644
@@ -468,6 +468,7 @@ int ovl_update_time(struct inode *inode, struct timespec64 *ts, int flags)
 		if (upperpath.dentry) {
 			touch_atime(&upperpath);
 			inode->i_atime = d_inode(upperpath.dentry)->i_atime;
+			ovl_mark_inode_dirty(inode);
 		}
 	}
 	return 0;
fs/overlayfs/overlayfs.h +4 -0
index f8880aa2ba0e..eaf1d5b05d8e 100644
@@ -247,6 +247,7 @@ static inline bool ovl_open_flags_need_copy_up(int flags)
 }
 
 /* util.c */
+void ovl_mark_inode_dirty(struct inode *inode);
 int ovl_want_write(struct dentry *dentry);
 void ovl_drop_write(struct dentry *dentry);
 struct dentry *ovl_workdir(struct dentry *dentry);
@@ -472,6 +473,9 @@ static inline void ovl_copyattr(struct inode *from, struct inode *to)
 	to->i_mtime = from->i_mtime;
 	to->i_ctime = from->i_ctime;
 	i_size_write(to, i_size_read(from));
+
+	if (ovl_inode_upper(to) && from->i_state & I_DIRTY_ALL)
+		ovl_mark_inode_dirty(to);
 }
 
 static inline void ovl_copyflags(struct inode *from, struct inode *to)
fs/overlayfs/util.c +14 -0
index 23f475627d07..a6f59df744ae 100644
@@ -950,3 +950,17 @@ char *ovl_get_redirect_xattr(struct ovl_fs *ofs, struct dentry *dentry,
 	kfree(buf);
 	return ERR_PTR(res);
 }
+
+/*
+ * We intentionally add I_DIRTY_SYNC flag regardless dirty flag
+ * of upper inode so that we have chance to invoke ->write_inode
+ * to re-dirty overlayfs' inode during writeback process.
+ */
+void ovl_mark_inode_dirty(struct inode *inode)
+{
+	struct inode *upper = ovl_inode_upper(inode);
+	unsigned long iflag = I_DIRTY_SYNC;
+
+	iflag |= upper->i_state & I_DIRTY_ALL;
+	__mark_inode_dirty(inode, iflag);
+}
Signature · 1 line
2.26.2
maintainer
#5 lore raw

Re: [RFC PATCH v4 4/9] ovl: mark overlayfs' inode dirty on modification

Quoted 36 lines · Chengguang Xu

Mark overlayfs' inode dirty on modification so that we can recognize target inodes during syncfs.

---
 fs/overlayfs/inode.c     |  1 +
 fs/overlayfs/overlayfs.h |  4 ++++
 fs/overlayfs/util.c      | 14 ++++++++++++++
 3 files changed, 19 insertions(+)
fs/overlayfs/inode.c +1 -0
index 8cfa75e86f56..342693657ab0 100644
@@ -468,6 +468,7 @@ int ovl_update_time(struct inode *inode, struct timespec64 *ts, int flags)
                if (upperpath.dentry) {
                        touch_atime(&upperpath);
                        inode->i_atime = d_inode(upperpath.dentry)->i_atime;
+                       ovl_mark_inode_dirty(inode);
                }
        }
        return 0;
fs/overlayfs/overlayfs.h +1 -0
index f8880aa2ba0e..eaf1d5b05d8e 100644
@@ -247,6 +247,7 @@ static inline bool ovl_open_flags_need_copy_up(int flags)
 }
 
 /* util.c */
+void ovl_mark_inode_dirty(struct inode *inode);
 int ovl_want_write(struct dentry *dentry);
 void ovl_drop_write(struct dentry *dentry);
 struct dentry *ovl_workdir(struct dentry *dentry);

fs/overlayfs/overlayfs.h static inline void ovl_copyattr(struct inode *from, struct inode *to) · line 479

Quoted 7 lines
fs/overlayfs/overlayfs.h +3 -0
@@ -472,6 +473,9 @@ static inline void ovl_copyattr(struct inode *from, struct inode *to)
        to->i_mtime = from->i_mtime;
        to->i_ctime = from->i_ctime;
        i_size_write(to, i_size_read(from));
+
+       if (ovl_inode_upper(to) && from->i_state & I_DIRTY_ALL)
+               ovl_mark_inode_dirty(to);
 }

Okay, ovl_copyattr() certainly seems a good place to copy dirtyness as well.

What I'm fearing is that it does not cover all the places where underlying inode can be dirtied. This really needs an audit of all filesystem modifying operations.

Quoted 28 lines
fs/overlayfs/overlayfs.h +0 -0
 static inline void ovl_copyflags(struct inode *from, struct inode *to)
fs/overlayfs/util.c +14 -1
index 23f475627d07..a6f59df744ae 100644
@@ -950,3 +950,17 @@ char *ovl_get_redirect_xattr(struct ovl_fs *ofs, struct dentry *dentry,
        kfree(buf);
        return ERR_PTR(res);
 }
+
+/*
+ * We intentionally add I_DIRTY_SYNC flag regardless dirty flag
+ * of upper inode so that we have chance to invoke ->write_inode
+ * to re-dirty overlayfs' inode during writeback process.
+ */
+void ovl_mark_inode_dirty(struct inode *inode)
+{
+       struct inode *upper = ovl_inode_upper(inode);
+       unsigned long iflag = I_DIRTY_SYNC;
+
+       iflag |= upper->i_state & I_DIRTY_ALL;
+       __mark_inode_dirty(inode, iflag);
+}
--

2.26.2

#6 lore raw

Re: [RFC PATCH v4 4/9] ovl: mark overlayfs' inode dirty on modification

---- 在 星期五, 2021-04-09 21:45:28 Miklos Szeredi <[email protected]> 撰写 ----

Quoted 47 lines · Chengguang Xu
Quoted 36 lines

Mark overlayfs' inode dirty on modification so that we can recognize target inodes during syncfs.

---
 fs/overlayfs/inode.c     |  1 +
 fs/overlayfs/overlayfs.h |  4 ++++
 fs/overlayfs/util.c      | 14 ++++++++++++++
 3 files changed, 19 insertions(+)
fs/overlayfs/inode.c +1 -0
index 8cfa75e86f56..342693657ab0 100644
@@ -468,6 +468,7 @@ int ovl_update_time(struct inode *inode, struct timespec64 *ts, int flags)
                if (upperpath.dentry) {
                        touch_atime(&upperpath);
                        inode->i_atime = d_inode(upperpath.dentry)->i_atime;
+                       ovl_mark_inode_dirty(inode);
                }
        }
        return 0;
fs/overlayfs/overlayfs.h +1 -0
index f8880aa2ba0e..eaf1d5b05d8e 100644
@@ -247,6 +247,7 @@ static inline bool ovl_open_flags_need_copy_up(int flags)
 }
 
 /* util.c */
+void ovl_mark_inode_dirty(struct inode *inode);
 int ovl_want_write(struct dentry *dentry);
 void ovl_drop_write(struct dentry *dentry);
 struct dentry *ovl_workdir(struct dentry *dentry);

fs/overlayfs/overlayfs.h static inline void ovl_copyattr(struct inode *from, struct inode *to) · line 479

Quoted 7 lines
fs/overlayfs/overlayfs.h +3 -0
@@ -472,6 +473,9 @@ static inline void ovl_copyattr(struct inode *from, struct inode *to)
        to->i_mtime = from->i_mtime;
        to->i_ctime = from->i_ctime;
        i_size_write(to, i_size_read(from));
+
+       if (ovl_inode_upper(to) && from->i_state & I_DIRTY_ALL)
+               ovl_mark_inode_dirty(to);
 }

Okay, ovl_copyattr() certainly seems a good place to copy dirtyness as well.

Quoted 3 lines

What I'm fearing is that it does not cover all the places where underlying inode can be dirtied. This really needs an audit of all filesystem modifying operations.

You are right. Let me fix it in next version.

Thanks,
Chengguang

Quoted 29 lines
Quoted 28 lines
fs/overlayfs/overlayfs.h +0 -0
 static inline void ovl_copyflags(struct inode *from, struct inode *to)
fs/overlayfs/util.c +14 -1
index 23f475627d07..a6f59df744ae 100644
@@ -950,3 +950,17 @@ char *ovl_get_redirect_xattr(struct ovl_fs *ofs, struct dentry *dentry,
        kfree(buf);
        return ERR_PTR(res);
 }
+
+/*
+ * We intentionally add I_DIRTY_SYNC flag regardless dirty flag
+ * of upper inode so that we have chance to invoke ->write_inode
+ * to re-dirty overlayfs' inode during writeback process.
+ */
+void ovl_mark_inode_dirty(struct inode *inode)
+{
+       struct inode *upper = ovl_inode_upper(inode);
+       unsigned long iflag = I_DIRTY_SYNC;
+
+       iflag |= upper->i_state & I_DIRTY_ALL;
+       __mark_inode_dirty(inode, iflag);
+}
--

2.26.2

#7 lore raw

RFC PATCH v4 5/9 ovl: mark overlayfs' inode dirty on shared mmap

In reply to Chengguang Xu

Overlayfs cannot be notified when mmapped area gets dirty, so we need to proactively mark inode dirty in ->mmap operation.

---
 fs/overlayfs/file.c | 3 +++
 1 file changed, 3 insertions(+)
fs/overlayfs/file.c +3 -0
index efccb7c1f9bc..3f9de5343513 100644
@@ -486,6 +486,9 @@ static int ovl_mmap(struct file *file, struct vm_area_struct *vma)
 		/* Drop reference count from new vm_file value */
 		fput(realfile);
 	} else {
+		if (ovl_inode_upper(file_inode(file)) &&
+		    vma->vm_flags & VM_SHARED)
+			ovl_mark_inode_dirty(file_inode(file));
 		/* Drop reference count from previous vm_file value */
 		fput(file);
 	}
Signature · 1 line
2.26.2
#8 lore raw

RFC PATCH v4 6/9 ovl: implement overlayfs' ->write_inode operation

In reply to Chengguang Xu

Implement overlayfs' ->write_inode to sync dirty data and redirty overlayfs' inode if necessary.

---
 fs/overlayfs/super.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
fs/overlayfs/super.c +30 -0
index 883172ac8a12..82e001b97f38 100644
@@ -390,6 +390,35 @@ static int ovl_remount(struct super_block *sb, int *flags, char *data)
 	return ret;
 }
 
+static int ovl_write_inode(struct inode *inode,
+			   struct writeback_control *wbc)
+{
+	struct ovl_fs *ofs = inode->i_sb->s_fs_info;
+	struct inode *upper = ovl_inode_upper(inode);
+	unsigned long iflag = 0;
+	int ret = 0;
+
+	if (!upper)
+		return 0;
+
+	if (!ovl_should_sync(ofs))
+		return 0;
+
+	if (upper->i_sb->s_op->write_inode)
+		ret = upper->i_sb->s_op->write_inode(inode, wbc);
+
+	if (mapping_writably_mapped(upper->i_mapping) ||
+	    mapping_tagged(upper->i_mapping, PAGECACHE_TAG_WRITEBACK))
+		iflag |= I_DIRTY_PAGES;
+
+	iflag |= upper->i_state & I_DIRTY_ALL;
+
+	if (iflag)
+		ovl_mark_inode_dirty(inode);
+
+	return ret;
+}
+
 static void ovl_evict_inode(struct inode *inode)
 {
 	inode->i_state &= ~I_DIRTY_ALL;
@@ -402,6 +431,7 @@ static const struct super_operations ovl_super_operations = {
 	.destroy_inode	= ovl_destroy_inode,
 	.drop_inode	= generic_delete_inode,
 	.evict_inode	= ovl_evict_inode,
+	.write_inode	= ovl_write_inode,
 	.put_super	= ovl_put_super,
 	.sync_fs	= ovl_sync_fs,
 	.statfs		= ovl_statfs,
Signature · 1 line
2.26.2
maintainer
#9 lore raw

Re: [RFC PATCH v4 6/9] ovl: implement overlayfs' ->write_inode operation

Quoted 29 lines · Chengguang Xu

Implement overlayfs' ->write_inode to sync dirty data and redirty overlayfs' inode if necessary.

---
 fs/overlayfs/super.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
fs/overlayfs/super.c +12 -0
index 883172ac8a12..82e001b97f38 100644
@@ -390,6 +390,35 @@ static int ovl_remount(struct super_block *sb, int *flags, char *data)
        return ret;
 }
 
+static int ovl_write_inode(struct inode *inode,
+                          struct writeback_control *wbc)
+{
+       struct ovl_fs *ofs = inode->i_sb->s_fs_info;
+       struct inode *upper = ovl_inode_upper(inode);
+       unsigned long iflag = 0;
+       int ret = 0;
+
+       if (!upper)
+               return 0;
+
+       if (!ovl_should_sync(ofs))

fs/overlayfs/super.c static int ovl_remount(struct super_block *sb, int *flags, char *data) · line 414

Quoted 10 lines
fs/overlayfs/super.c +10 -0
+               return 0;
+
+       if (upper->i_sb->s_op->write_inode)
+               ret = upper->i_sb->s_op->write_inode(inode, wbc);
+
+       if (mapping_writably_mapped(upper->i_mapping) ||
+           mapping_tagged(upper->i_mapping, PAGECACHE_TAG_WRITEBACK))
+               iflag |= I_DIRTY_PAGES;
+
+       iflag |= upper->i_state & I_DIRTY_ALL;

How is I_DIRTY_SYNC added/removed from the overlay inode?

Thanks,
Miklos

#10 lore raw

Re: [RFC PATCH v4 6/9] ovl: implement overlayfs' ->write_inode operation

---- 在 星期五, 2021-04-09 21:49:07 Miklos Szeredi <[email protected]> 撰写 ----

Quoted 34 lines · Chengguang Xu
Quoted 33 lines

Implement overlayfs' ->write_inode to sync dirty data and redirty overlayfs' inode if necessary.

---
 fs/overlayfs/super.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
fs/overlayfs/super.c +16 -0
index 883172ac8a12..82e001b97f38 100644
@@ -390,6 +390,35 @@ static int ovl_remount(struct super_block *sb, int *flags, char *data)
        return ret;
 }
 
+static int ovl_write_inode(struct inode *inode,
+                          struct writeback_control *wbc)
+{
+       struct ovl_fs *ofs = inode->i_sb->s_fs_info;
+       struct inode *upper = ovl_inode_upper(inode);
+       unsigned long iflag = 0;
+       int ret = 0;
+
+       if (!upper)
+               return 0;
+
+       if (!ovl_should_sync(ofs))
+               return 0;
+
+       if (upper->i_sb->s_op->write_inode)
+               ret = upper->i_sb->s_op->write_inode(inode, wbc);
Quoted 9 lines

fs/overlayfs/super.c static int ovl_remount(struct super_block *sb, int *flags, char *data) · line 414

Quoted 6 lines
fs/overlayfs/super.c +6 -0
+
+       if (mapping_writably_mapped(upper->i_mapping) ||
+           mapping_tagged(upper->i_mapping, PAGECACHE_TAG_WRITEBACK))
+               iflag |= I_DIRTY_PAGES;
+
+       iflag |= upper->i_state & I_DIRTY_ALL;

How is I_DIRTY_SYNC added/removed from the overlay inode?

generally, I_DIRTY_SYNC is added to overlay inode when the operation dirties upper inode, I'll check all those places and call ovl_mark_inode_dirty() to do it. After writeback if upper inode becomes clean status, we will remove I_DIRTY_SYNC from overlay inode.

One exception is mmaped file, it will always keep I_DIRTY_SYNC until to be evicted.

Thanks,
Chengguang

#11 lore raw

RFC PATCH v4 7/9 ovl: cache dirty overlayfs' inode

In reply to Chengguang Xu

Now drop overlayfs' inode will sync dirty data,
so we change to only drop clean inode.

---
 fs/overlayfs/super.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
fs/overlayfs/super.c +10 -1
index 82e001b97f38..982b3954b47c 100644
@@ -425,11 +425,20 @@ static void ovl_evict_inode(struct inode *inode)
 	clear_inode(inode);
 }
 
+static int ovl_drop_inode(struct inode *inode)
+{
+	struct inode *upper = ovl_inode_upper(inode);
+
+	if (!upper || !(inode->i_state & I_DIRTY_ALL))
+		return 1;
+	return generic_drop_inode(inode);
+}
+
 static const struct super_operations ovl_super_operations = {
 	.alloc_inode	= ovl_alloc_inode,
 	.free_inode	= ovl_free_inode,
 	.destroy_inode	= ovl_destroy_inode,
-	.drop_inode	= generic_delete_inode,
+	.drop_inode	= ovl_drop_inode,
 	.evict_inode	= ovl_evict_inode,
 	.write_inode	= ovl_write_inode,
 	.put_super	= ovl_put_super,
Signature · 1 line
2.26.2
maintainer
#12 lore raw

Re: [RFC PATCH v4 7/9] ovl: cache dirty overlayfs' inode

Quoted 3 lines · Chengguang Xu

Now drop overlayfs' inode will sync dirty data,
so we change to only drop clean inode.

I don't understand what happens here. Please add more explanation.

Thanks,
Miklos

#13 lore raw

Re: [RFC PATCH v4 7/9] ovl: cache dirty overlayfs' inode

---- 在 星期五, 2021-04-09 21:50:35 Miklos Szeredi <[email protected]> 撰写 ----

Quoted 6 lines · Chengguang Xu
Quoted 3 lines

Now drop overlayfs' inode will sync dirty data,
so we change to only drop clean inode.

I don't understand what happens here. Please add more explanation.

In iput_final(), clean overlayfs inode will directly drop as the same as before, dirty overlayfs inode will keep in the cache to wait writeback to sync dirty data and then add to lru list to wait reclaim.

The purpose of doing this is to keep compatible behavior with original one, because without this series, dropping overlayfs inode will not trigger syncing underlying dirty inode.

Thanks,
Chengguang

maintainer
#14 lore raw

Re: [RFC PATCH v4 7/9] ovl: cache dirty overlayfs' inode

Quoted 9 lines · Chengguang Xu

---- 在 星期五, 2021-04-09 21:50:35 Miklos Szeredi <[email protected]> 撰写 ----

Quoted 6 lines · Chengguang Xu
Quoted 3 lines

Now drop overlayfs' inode will sync dirty data,
so we change to only drop clean inode.

I don't understand what happens here. Please add more explanation.

Quoted 7 lines

In iput_final(), clean overlayfs inode will directly drop as the same as before, dirty overlayfs inode will keep in the cache to wait writeback to sync dirty data and then add to lru list to wait reclaim.

The purpose of doing this is to keep compatible behavior with original one, because without this series, dropping overlayfs inode will not trigger syncing underlying dirty inode.

I get it now. Can you please update the patch header with this description?

Thanks,
Miklos

#15 lore raw

Re: [RFC PATCH v4 7/9] ovl: cache dirty overlayfs' inode

---- 在 星期二, 2021-04-13 16:43:27 Miklos Szeredi <[email protected]> 撰写 ----

Quoted 14 lines · Chengguang Xu
Quoted 13 lines

---- 在 星期五, 2021-04-09 21:50:35 Miklos Szeredi <[email protected]> 撰写 ----

Quoted 6 lines · Chengguang Xu
Quoted 3 lines

Now drop overlayfs' inode will sync dirty data,
so we change to only drop clean inode.

I don't understand what happens here. Please add more explanation.

In iput_final(), clean overlayfs inode will directly drop as the same as before, dirty overlayfs inode will keep in the cache to wait writeback to sync dirty data and then add to lru list to wait reclaim.

Quoted 6 lines
Quoted 3 lines

The purpose of doing this is to keep compatible behavior with original one, because without this series, dropping overlayfs inode will not trigger syncing underlying dirty inode.

I get it now. Can you please update the patch header with this description?

No problem, I'll update commit log in next version.

Thanks,
Chengguang

#16 lore raw

RFC PATCH v4 8/9 fs: export wait_sb_inodes()

In reply to Chengguang Xu

In order to wait syncing upper inodes we need to call wait_sb_inodes() in overlayfs' ->sync_fs.

---
 fs/fs-writeback.c         | 3 ++-
 include/linux/writeback.h | 1 +
 2 files changed, 3 insertions(+), 1 deletion(-)
fs/fs-writeback.c +2 -1
index 149227160ff0..55dafdc72661 100644
@@ -2365,7 +2365,7 @@ EXPORT_SYMBOL(__mark_inode_dirty);
  * completed by the time we have gained the lock and waited for all IO that is
  * in progress regardless of the order callers are granted the lock.
  */
-static void wait_sb_inodes(struct super_block *sb)
+void wait_sb_inodes(struct super_block *sb)
 {
 	LIST_HEAD(sync_list);
 
@@ -2449,6 +2449,7 @@ static void wait_sb_inodes(struct super_block *sb)
 	rcu_read_unlock();
 	mutex_unlock(&sb->s_sync_lock);
 }
+EXPORT_SYMBOL(wait_sb_inodes);
 
 static void __writeback_inodes_sb_nr(struct super_block *sb, unsigned long nr,
 				     enum wb_reason reason, bool skip_if_busy)
include/linux/writeback.h +1 -0
index 8e5c5bb16e2d..53e826af33b6 100644
@@ -198,6 +198,7 @@ void wakeup_flusher_threads_bdi(struct backing_dev_info *bdi,
 				enum wb_reason reason);
 void inode_wait_for_writeback(struct inode *inode);
 void inode_io_list_del(struct inode *inode);
+void wait_sb_inodes(struct super_block *sb);
 
 /* writeback.h requires fs.h; it, too, is not included from here. */
 static inline void wait_on_inode(struct inode *inode)
Signature · 1 line
2.26.2
#17 lore raw

RFC PATCH v4 9/9 ovl: implement containerized syncfs for overlayfs

In reply to Chengguang Xu

Now overlayfs can only sync dirty inode during syncfs, so remove unnecessary sync_filesystem() on upper file system.

---
 fs/overlayfs/super.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
fs/overlayfs/super.c +8 -3
index 982b3954b47c..58507f1cd583 100644
@@ -15,6 +15,8 @@
 #include <linux/seq_file.h>
 #include <linux/posix_acl_xattr.h>
 #include <linux/exportfs.h>
+#include <linux/blkdev.h>
+#include <linux/writeback.h>
 #include "overlayfs.h"
 
 MODULE_AUTHOR("Miklos Szeredi <[email protected]>");
@@ -270,8 +272,7 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
 	 * Not called for sync(2) call or an emergency sync (SB_I_SKIP_SYNC).
 	 * All the super blocks will be iterated, including upper_sb.
 	 *
-	 * If this is a syncfs(2) call, then we do need to call
-	 * sync_filesystem() on upper_sb, but enough if we do it when being
+	 * if this is a syncfs(2) call, it will be enough we do it when being
 	 * called with wait == 1.
 	 */
 	if (!wait)
@@ -280,7 +281,11 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
 	upper_sb = ovl_upper_mnt(ofs)->mnt_sb;
 
 	down_read(&upper_sb->s_umount);
-	ret = sync_filesystem(upper_sb);
+	wait_sb_inodes(upper_sb);
+	if (upper_sb->s_op->sync_fs)
+		ret = upper_sb->s_op->sync_fs(upper_sb, wait);
+	if (!ret)
+		ret = sync_blockdev(upper_sb->s_bdev);
 	up_read(&upper_sb->s_umount);
 
 	return ret;
Signature · 1 line
2.26.2
maintainer
#18 lore raw

Re: [RFC PATCH v4 9/9] ovl: implement containerized syncfs for overlayfs

Quoted 34 lines · Chengguang Xu

Now overlayfs can only sync dirty inode during syncfs, so remove unnecessary sync_filesystem() on upper file system.

---
 fs/overlayfs/super.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
fs/overlayfs/super.c +3 -2
index 982b3954b47c..58507f1cd583 100644
@@ -15,6 +15,8 @@
 #include <linux/seq_file.h>
 #include <linux/posix_acl_xattr.h>
 #include <linux/exportfs.h>
+#include <linux/blkdev.h>
+#include <linux/writeback.h>
 #include "overlayfs.h"
 
 MODULE_AUTHOR("Miklos Szeredi <[email protected]>");
@@ -270,8 +272,7 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
         * Not called for sync(2) call or an emergency sync (SB_I_SKIP_SYNC).
         * All the super blocks will be iterated, including upper_sb.
         *
-        * If this is a syncfs(2) call, then we do need to call
-        * sync_filesystem() on upper_sb, but enough if we do it when being
+        * if this is a syncfs(2) call, it will be enough we do it when being
         * called with wait == 1.
         */
        if (!wait)

fs/overlayfs/super.c static int ovl_sync_fs(struct super_block *sb, int wait) · line 288

Quoted 9 lines
fs/overlayfs/super.c +5 -1
@@ -280,7 +281,11 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
        upper_sb = ovl_upper_mnt(ofs)->mnt_sb;
 
        down_read(&upper_sb->s_umount);
-       ret = sync_filesystem(upper_sb);
+       wait_sb_inodes(upper_sb);
+       if (upper_sb->s_op->sync_fs)
+               ret = upper_sb->s_op->sync_fs(upper_sb, wait);
+       if (!ret)
+               ret = sync_blockdev(upper_sb->s_bdev);

Should this instead be __sync_blockdev(..., wait)?

Thanks,
Miklos

#19 lore raw

Re: [RFC PATCH v4 9/9] ovl: implement containerized syncfs for overlayfs

---- 在 星期五, 2021-04-09 21:51:26 Miklos Szeredi <[email protected]> 撰写 ----

Quoted 37 lines · Chengguang Xu
Quoted 36 lines

Now overlayfs can only sync dirty inode during syncfs, so remove unnecessary sync_filesystem() on upper file system.

---
 fs/overlayfs/super.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
fs/overlayfs/super.c +3 -2
index 982b3954b47c..58507f1cd583 100644
@@ -15,6 +15,8 @@
 #include <linux/seq_file.h>
 #include <linux/posix_acl_xattr.h>
 #include <linux/exportfs.h>
+#include <linux/blkdev.h>
+#include <linux/writeback.h>
 #include "overlayfs.h"
 
 MODULE_AUTHOR("Miklos Szeredi <[email protected]>");
@@ -270,8 +272,7 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
         * Not called for sync(2) call or an emergency sync (SB_I_SKIP_SYNC).
         * All the super blocks will be iterated, including upper_sb.
         *
-        * If this is a syncfs(2) call, then we do need to call
-        * sync_filesystem() on upper_sb, but enough if we do it when being
+        * if this is a syncfs(2) call, it will be enough we do it when being
         * called with wait == 1.
         */
        if (!wait)
@@ -280,7 +281,11 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
        upper_sb = ovl_upper_mnt(ofs)->mnt_sb;
 
Quoted 9 lines

fs/overlayfs/super.c static int ovl_sync_fs(struct super_block *sb, int wait) · line 288

Quoted 7 lines
fs/overlayfs/super.c +5 -1
        down_read(&upper_sb->s_umount);
-       ret = sync_filesystem(upper_sb);
+       wait_sb_inodes(upper_sb);
+       if (upper_sb->s_op->sync_fs)
+               ret = upper_sb->s_op->sync_fs(upper_sb, wait);
+       if (!ret)
+               ret = sync_blockdev(upper_sb->s_bdev);

Should this instead be __sync_blockdev(..., wait)?

I don't remember why we skipped the case of (wait == 0) here, just guess it's not worth to export internal function __sync_blockdev() to modules, do you prefer to call __sync_blockdev() and handle both nowait and wait cases?

Thanks,
Chengguang

maintainer
#20 lore raw

Re: [RFC PATCH v4 9/9] ovl: implement containerized syncfs for overlayfs

Quoted 49 lines · Chengguang Xu

---- 在 星期五, 2021-04-09 21:51:26 Miklos Szeredi <[email protected]> 撰写 ----

Quoted 37 lines · Chengguang Xu
Quoted 36 lines

Now overlayfs can only sync dirty inode during syncfs, so remove unnecessary sync_filesystem() on upper file system.

---
 fs/overlayfs/super.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
fs/overlayfs/super.c +3 -2
index 982b3954b47c..58507f1cd583 100644
@@ -15,6 +15,8 @@
 #include <linux/seq_file.h>
 #include <linux/posix_acl_xattr.h>
 #include <linux/exportfs.h>
+#include <linux/blkdev.h>
+#include <linux/writeback.h>
 #include "overlayfs.h"
 
 MODULE_AUTHOR("Miklos Szeredi <[email protected]>");
@@ -270,8 +272,7 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
         * Not called for sync(2) call or an emergency sync (SB_I_SKIP_SYNC).
         * All the super blocks will be iterated, including upper_sb.
         *
-        * If this is a syncfs(2) call, then we do need to call
-        * sync_filesystem() on upper_sb, but enough if we do it when being
+        * if this is a syncfs(2) call, it will be enough we do it when being
         * called with wait == 1.
         */
        if (!wait)
@@ -280,7 +281,11 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
        upper_sb = ovl_upper_mnt(ofs)->mnt_sb;
 
Quoted 9 lines

fs/overlayfs/super.c static int ovl_sync_fs(struct super_block *sb, int wait) · line 288

Quoted 7 lines
fs/overlayfs/super.c +5 -1
        down_read(&upper_sb->s_umount);
-       ret = sync_filesystem(upper_sb);
+       wait_sb_inodes(upper_sb);
+       if (upper_sb->s_op->sync_fs)
+               ret = upper_sb->s_op->sync_fs(upper_sb, wait);
+       if (!ret)
+               ret = sync_blockdev(upper_sb->s_bdev);

Should this instead be __sync_blockdev(..., wait)?

Quoted 3 lines

I don't remember why we skipped the case of (wait == 0) here, just guess it's not worth to export internal function __sync_blockdev() to modules, do you prefer to call __sync_blockdev() and handle both nowait and wait cases?

Possibly it would make most sense to export the "->sync_fs() + __sync_blockdev()" part of __sync_filesystem() as a separate helper.

Thanks,
Miklos

#21 lore raw
In reply to Chengguang Xu

---- 在 星期五, 2020-11-13 14:55:46 Chengguang Xu <[email protected]> 撰写 ----

Quoted 10 lines

Current syncfs(2) syscall on overlayfs just calls sync_filesystem() on upper_sb to synchronize whole dirty inodes in upper filesystem regardless of the overlay ownership of the inode. In the use case of container, when multiple containers using the same underlying upper filesystem, it has some shortcomings as below.

(1) Performance
Synchronization is probably heavy because it actually syncs unnecessary inodes for target overlayfs.

Quoted 8 lines

(2) Interference
Unplanned synchronization will probably impact IO performance of unrelated container processes on the other overlayfs.

This series try to implement containerized syncfs for overlayfs so that only sync target dirty upper inodes which are belong to specific overlayfs instance. By doing this, it is able to reduce cost of synchronization and will not seriously impact IO performance of unrelated processes.

Hi Miklos,

I think this version has addressed all previous issues and comments from Jack and Amir. Have you got time to review this patch series?

Thanks,
Chengguang

maintainer
#22 lore raw
Quoted 21 lines · Chengguang Xu

---- 在 星期五, 2020-11-13 14:55:46 Chengguang Xu <[email protected]> 撰写 ----

Quoted 10 lines

Current syncfs(2) syscall on overlayfs just calls sync_filesystem() on upper_sb to synchronize whole dirty inodes in upper filesystem regardless of the overlay ownership of the inode. In the use case of container, when multiple containers using the same underlying upper filesystem, it has some shortcomings as below.

(1) Performance
Synchronization is probably heavy because it actually syncs unnecessary inodes for target overlayfs.

Quoted 8 lines

(2) Interference
Unplanned synchronization will probably impact IO performance of unrelated container processes on the other overlayfs.

This series try to implement containerized syncfs for overlayfs so that only sync target dirty upper inodes which are belong to specific overlayfs instance. By doing this, it is able to reduce cost of synchronization and will not seriously impact IO performance of unrelated processes.

Quoted 4 lines

Hi Miklos,

I think this version has addressed all previous issues and comments from Jack and Amir. Have you got time to review this patch series?

Hopefully yes.

I'm really keen to finish off the unprivileged overlay patches first. Will test and post a new version shortly.

Thanks,
Miklos

maintainer Acked-by
#23 lore raw
In reply to Chengguang Xu
Quoted 20 lines · Chengguang Xu

---- 在 星期五, 2020-11-13 14:55:46 Chengguang Xu <[email protected]> 撰写 ----

Quoted 10 lines

Current syncfs(2) syscall on overlayfs just calls sync_filesystem() on upper_sb to synchronize whole dirty inodes in upper filesystem regardless of the overlay ownership of the inode. In the use case of container, when multiple containers using the same underlying upper filesystem, it has some shortcomings as below.

(1) Performance
Synchronization is probably heavy because it actually syncs unnecessary inodes for target overlayfs.

Quoted 8 lines

(2) Interference
Unplanned synchronization will probably impact IO performance of unrelated container processes on the other overlayfs.

This series try to implement containerized syncfs for overlayfs so that only sync target dirty upper inodes which are belong to specific overlayfs instance. By doing this, it is able to reduce cost of synchronization and will not seriously impact IO performance of unrelated processes.

Quoted 4 lines

Hi Miklos,

I think this version has addressed all previous issues and comments from Jack and Amir. Have you got time to review this patch series?

Yes, the patches now look good to me. Feel free to add:

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR