implement containerized syncfs for overlayfs
Subsystems (2)
v2 cover implement containerized syncfs for overlayfs
Current syncfs(2) syscall on overlayfs just calls sync_filesystem() on upper_sb to synchronize whole dirty inodes in upper filesystem regardless of the overlay ownership of the inode. In the use case of container, when multiple containers using the same underlying upper filesystem, it has some shortcomings as below.
(1) Performance
Synchronization is probably heavy because it actually syncs unnecessary inodes for target overlayfs.
(2) Interference
Unplanned synchronization will probably impact IO performance of unrelated container processes on the other overlayfs.
This series try to implement containerized syncfs for overlayfs so that only sync target dirty upper inodes which are belong to specific overlayfs instance. By doing this, it is able to reduce cost of synchronization and will not seriously impact IO performance of unrelated processes.
v1->v2:
- Mark overlayfs' inode dirty itself instead of adding notification mechanism to vfs inode.
Chengguang Xu (8):
ovl: setup overlayfs' private bdi
ovl: implement ->writepages operation
ovl: implement overlayfs' ->evict_inode operation
ovl: mark overlayfs' inode dirty on modification
ovl: mark overlayfs' inode dirty on shared writable mmap
ovl: implement overlayfs' ->write_inode operation
ovl: cache dirty overlayfs' inode
ovl: implement containerized syncfs for overlayfs
fs/overlayfs/file.c | 4 +++ fs/overlayfs/inode.c | 27 +++++++++++++++++++ fs/overlayfs/overlayfs.h | 4 +++ fs/overlayfs/super.c | 57 +++++++++++++++++++++++++++++++++++++--- fs/overlayfs/util.c | 14 ++++++++++ 5 files changed, 102 insertions(+), 4 deletions(-)
Signature · 1 line
2.26.2
RFC PATCH v2 1/8 ovl: setup overlayfs' private bdi
Setup overlayfs' private bdi so that we can collect overlayfs' own dirty inodes.
--- fs/overlayfs/super.c | 4 ++++ 1 file changed, 4 insertions(+)
fs/overlayfs/super.c +4 -0
@@ -1869,6 +1869,10 @@ static int ovl_fill_super(struct super_block *sb, void *data, int silent)if (!ofs) goto out; + err =super_setup_bdi (sb); +if (err) + goto out_err; + ofs->creator_cred = cred =prepare_creds ();if (!cred) goto out_err;
Signature · 1 line
2.26.2
RFC PATCH v2 2/8 ovl: implement ->writepages operation
In reply to Chengguang XuImplement overlayfs' ->writepages operation so that we can sync dirty data/metadata to upper filesystem.
--- fs/overlayfs/inode.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)
fs/overlayfs/inode.c +26 -0
@@ -11,6 +11,7 @@#include <linux/posix_acl.h> #include <linux/ratelimit.h> #include <linux/fiemap.h> +#include <linux/writeback.h> #include "overlayfs.h"
@@ -516,7 +517,32 @@ static const struct inode_operations ovl_special_inode_operations = { .update_time = ovl_update_time, }; +static int ovl_writepages (struct address_space *mapping, +struct writeback_control *wbc) +{ +struct inode *upper_inode =ovl_inode_upper (mapping->host); +struct ovl_fs *ofs = mapping->host->i_sb->s_fs_info; +struct writeback_control tmp_wbc = *wbc; + +if (!ovl_should_sync (ofs)) +return 0 ; + +/* +* for sync(2) writeback, it has a separate external IO +* completion path by checking PAGECACHE_TAG_WRITEBACK +* in pagecache, we have to set for_sync to 0 in thie case, +* let writeback waits completion after syncing individual +* dirty inode, because we haven't implemented overlayfs' +* own pagecache yet. +*/ +if (wbc->for_sync && (wbc->sync_mode ==WB_SYNC_ALL )) + tmp_wbc.for_sync =0 ; + +return sync_inode (upper_inode, &tmp_wbc); +} +static const struct address_space_operations ovl_aops = { + .writepages = ovl_writepages,/* For O_DIRECT dentry_open() checks f_mapping->a_ops->direct_IO */ .direct_IO = noop_direct_IO, };
Signature · 1 line
2.26.2
Jan Kara
reviewerRe: [RFC PATCH v2 2/8] ovl: implement ->writepages operation
Quoted 34 lines · Chengguang Xu
Implement overlayfs' ->writepages operation so that we can sync dirty data/metadata to upper filesystem.
Signed-off-by Chengguang Xu <[email protected]>--- fs/overlayfs/inode.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)fs/overlayfs/inode.c +11
-0@@ -11,6 +11,7 @@#include <linux/posix_acl.h> #include <linux/ratelimit.h> #include <linux/fiemap.h> +#include <linux/writeback.h> #include "overlayfs.h" @@ -516,7 +517,32 @@ static const struct inode_operations ovl_special_inode_operations = { .update_time = ovl_update_time, }; +static int ovl_writepages (struct address_space *mapping, +struct writeback_control *wbc) +{ +struct inode *upper_inode =ovl_inode_upper (mapping->host); +struct ovl_fs *ofs = mapping->host->i_sb->s_fs_info; +struct writeback_control tmp_wbc = *wbc; + +if (!ovl_should_sync (ofs)) +return 0 ; +
fs/overlayfs/inode.c static const struct inode_operations ovl_special_inode_operations = { · line 539
Quoted 10 lines
fs/overlayfs/inode.c +10
-0+/* +* for sync(2) writeback, it has a separate external IO +* completion path by checking PAGECACHE_TAG_WRITEBACK +* in pagecache, we have to set for_sync to 0 in thie case, +* let writeback waits completion after syncing individual +* dirty inode, because we haven't implemented overlayfs' +* own pagecache yet. +*/ +if (wbc->for_sync && (wbc->sync_mode ==WB_SYNC_ALL )) + tmp_wbc.for_sync =0 ;
This looks really hacky as it closely depends on the internal details of writeback implementation. I'd be more open to say export wait_sb_inodes() for overlayfs use... Because that's what I gather you need in your overlayfs ->syncfs() implementation.
Honza
Quoted 13 lines
fs/overlayfs/inode.c +5
-0+ +return sync_inode (upper_inode, &tmp_wbc); +} +static const struct address_space_operations ovl_aops = { + .writepages = ovl_writepages,/* For O_DIRECT dentry_open() checks f_mapping->a_ops->direct_IO */ .direct_IO = noop_direct_IO, };Signature · 1 line
2.26.2
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [RFC PATCH v2 2/8] ovl: implement ->writepages operation
---- 在 星期二, 2020-11-03 01:17:41 Jan Kara <[email protected]> 撰写 ----
Quoted 42 lines · Chengguang Xu
Quoted 41 lines
Implement overlayfs' ->writepages operation so that we can sync dirty data/metadata to upper filesystem.
Signed-off-by Chengguang Xu <[email protected]>--- fs/overlayfs/inode.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)fs/overlayfs/inode.c +18
-0@@ -11,6 +11,7 @@#include <linux/posix_acl.h> #include <linux/ratelimit.h> #include <linux/fiemap.h> +#include <linux/writeback.h> #include "overlayfs.h" @@ -516,7 +517,32 @@ static const struct inode_operations ovl_special_inode_operations = { .update_time = ovl_update_time, }; +static int ovl_writepages (struct address_space *mapping, +struct writeback_control *wbc) +{ +struct inode *upper_inode =ovl_inode_upper (mapping->host); +struct ovl_fs *ofs = mapping->host->i_sb->s_fs_info; +struct writeback_control tmp_wbc = *wbc; + +if (!ovl_should_sync (ofs)) +return 0 ; + +/* +* for sync(2) writeback, it has a separate external IO +* completion path by checking PAGECACHE_TAG_WRITEBACK +* in pagecache, we have to set for_sync to 0 in thie case, +* let writeback waits completion after syncing individual +* dirty inode, because we haven't implemented overlayfs' +* own pagecache yet.
Quoted 9 lines
fs/overlayfs/inode.c static const struct inode_operations ovl_special_inode_operations = { · line 539
Quoted 3 lines
fs/overlayfs/inode.c +3
-0+*/ +if (wbc->for_sync && (wbc->sync_mode ==WB_SYNC_ALL )) + tmp_wbc.for_sync =0 ;This looks really hacky as it closely depends on the internal details of writeback implementation. I'd be more open to say export
wait_sb_inodes()for overlayfs use... Because that's what I gather you need in your overlayfs ->syncfs() implementation.
Does that mean we gather synced overlay's inode into a new waiting list(overlay's) and do the waiting behavior in overlay's ->syncfs() ?
Thanks,
Chengguang
Jan Kara
reviewerRe: [RFC PATCH v2 2/8] ovl: implement ->writepages operation
Quoted 53 lines · Chengguang Xu
---- 在 星期二, 2020-11-03 01:17:41 Jan Kara <[email protected]> 撰写 ----
Quoted 42 lines · Chengguang Xu
Quoted 41 lines
Implement overlayfs' ->writepages operation so that we can sync dirty data/metadata to upper filesystem.
Signed-off-by Chengguang Xu <[email protected]>--- fs/overlayfs/inode.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)fs/overlayfs/inode.c +18
-0@@ -11,6 +11,7 @@#include <linux/posix_acl.h> #include <linux/ratelimit.h> #include <linux/fiemap.h> +#include <linux/writeback.h> #include "overlayfs.h" @@ -516,7 +517,32 @@ static const struct inode_operations ovl_special_inode_operations = { .update_time = ovl_update_time, }; +static int ovl_writepages (struct address_space *mapping, +struct writeback_control *wbc) +{ +struct inode *upper_inode =ovl_inode_upper (mapping->host); +struct ovl_fs *ofs = mapping->host->i_sb->s_fs_info; +struct writeback_control tmp_wbc = *wbc; + +if (!ovl_should_sync (ofs)) +return 0 ; + +/* +* for sync(2) writeback, it has a separate external IO +* completion path by checking PAGECACHE_TAG_WRITEBACK +* in pagecache, we have to set for_sync to 0 in thie case, +* let writeback waits completion after syncing individual +* dirty inode, because we haven't implemented overlayfs' +* own pagecache yet. Quoted 9 lines
fs/overlayfs/inode.c static const struct inode_operations ovl_special_inode_operations = { · line 539
Quoted 3 lines
fs/overlayfs/inode.c +3
-0+*/ +if (wbc->for_sync && (wbc->sync_mode ==WB_SYNC_ALL )) + tmp_wbc.for_sync =0 ;This looks really hacky as it closely depends on the internal details of writeback implementation. I'd be more open to say export
wait_sb_inodes()for overlayfs use... Because that's what I gather you need in your overlayfs ->syncfs() implementation.
Quoted 2 lines
Does that mean we gather synced overlay's inode into a new waiting list(overlay's) and do the waiting behavior in overlay's ->syncfs() ?
My idea was that you'd just use the standard writeback logic which ends up gathering upper_sb inodes in the upper_sb->s_inodes_wb and then wait for them in overlay's ->syncfs(). Maybe we'll end up waiting for more inodes than strictly necessary but it shouldn't be too bad I'd say...
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [RFC PATCH v2 2/8] ovl: implement ->writepages operation
---- 在 星期四, 2020-11-05 21:55:06 Jan Kara <[email protected]> 撰写 ----
Quoted 54 lines · Chengguang Xu
Quoted 53 lines
---- 在 星期二, 2020-11-03 01:17:41 Jan Kara <[email protected]> 撰写 ----
Quoted 51 lines · Chengguang Xu
Quoted 34 lines
Implement overlayfs' ->writepages operation so that we can sync dirty data/metadata to upper filesystem.
Signed-off-by Chengguang Xu <[email protected]>--- fs/overlayfs/inode.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)fs/overlayfs/inode.c +11
-0@@ -11,6 +11,7 @@#include <linux/posix_acl.h> #include <linux/ratelimit.h> #include <linux/fiemap.h> +#include <linux/writeback.h> #include "overlayfs.h" @@ -516,7 +517,32 @@ static const struct inode_operations ovl_special_inode_operations = { .update_time = ovl_update_time, }; +static int ovl_writepages (struct address_space *mapping, +struct writeback_control *wbc) +{ +struct inode *upper_inode =ovl_inode_upper (mapping->host); +struct ovl_fs *ofs = mapping->host->i_sb->s_fs_info; +struct writeback_control tmp_wbc = *wbc; + +if (!ovl_should_sync (ofs)) +return 0 ; +fs/overlayfs/inode.c static const struct inode_operations ovl_special_inode_operations = { · line 539
Quoted 10 lines
fs/overlayfs/inode.c +10
-0+/* +* for sync(2) writeback, it has a separate external IO +* completion path by checking PAGECACHE_TAG_WRITEBACK +* in pagecache, we have to set for_sync to 0 in thie case, +* let writeback waits completion after syncing individual +* dirty inode, because we haven't implemented overlayfs' +* own pagecache yet. +*/ +if (wbc->for_sync && (wbc->sync_mode ==WB_SYNC_ALL )) + tmp_wbc.for_sync =0 ;This looks really hacky as it closely depends on the internal details of writeback implementation. I'd be more open to say export
wait_sb_inodes()for overlayfs use... Because that's what I gather you need in your overlayfs ->syncfs() implementation.
Quoted 8 lines
Quoted 2 lines
Does that mean we gather synced overlay's inode into a new waiting list(overlay's) and do the waiting behavior in overlay's ->syncfs() ?
My idea was that you'd just use the standard writeback logic which ends up gathering
upper_sbinodes in theupper_sb->s_inodes_wband then wait for them in overlay's ->syncfs(). Maybe we'll end up waiting for more inodes than strictly necessary but it shouldn't be too bad I'd say...
Yeah, I agree with you, I'll modify in next version.
Thanks,
Chengguang
RFC PATCH v2 3/8 ovl: implement overlayfs' ->evict_inode operation
In reply to Chengguang XuImplement overlayfs' ->evict_inode operation,
so that we can clear all inode dirty flags.
--- fs/overlayfs/super.c | 7 +++++++ 1 file changed, 7 insertions(+)
fs/overlayfs/super.c +7 -0
@@ -390,11 +390,18 @@ static int ovl_remount(struct super_block *sb, int *flags, char *data)return ret; } +static void ovl_evict_inode (struct inode *inode) +{ + inode->i_state &= ~I_DIRTY_ALL ; +clear_inode (inode); +} +static const struct super_operations ovl_super_operations = { .alloc_inode = ovl_alloc_inode, .free_inode = ovl_free_inode, .destroy_inode = ovl_destroy_inode, .drop_inode = generic_delete_inode, + .evict_inode = ovl_evict_inode, .put_super = ovl_put_super, .sync_fs = ovl_sync_fs, .statfs = ovl_statfs,
Signature · 1 line
2.26.2
RFC PATCH v2 4/8 ovl: mark overlayfs' inode dirty on modification
In reply to Chengguang XuMark overlayfs' inode dirty on modification so that we can recognize target inodes during syncfs.
--- fs/overlayfs/inode.c | 1 + fs/overlayfs/overlayfs.h | 4 ++++ fs/overlayfs/util.c | 14 ++++++++++++++ 3 files changed, 19 insertions(+)
fs/overlayfs/inode.c +1 -0
@@ -468,6 +468,7 @@ int ovl_update_time(struct inode *inode, struct timespec64 *ts, int flags)if (upperpath.dentry) {touch_atime (&upperpath); inode->i_atime =d_inode (upperpath.dentry)->i_atime; +ovl_mark_inode_dirty (inode); } }return 0 ;
fs/overlayfs/overlayfs.h +4 -0
@@ -247,6 +247,7 @@ static inline bool ovl_open_flags_need_copy_up(int flags) }/* util.c */ +void ovl_mark_inode_dirty (struct inode *inode);int ovl_want_write (struct dentry *dentry);void ovl_drop_write (struct dentry *dentry);struct dentry *ovl_workdir (struct dentry *dentry);
@@ -472,6 +473,9 @@ static inline void ovl_copyattr(struct inode *from, struct inode *to) to->i_mtime = from->i_mtime; to->i_ctime = from->i_ctime;i_size_write (to,i_size_read (from)); + +if (ovl_inode_upper (to) && from->i_state &I_DIRTY_ALL ) +ovl_mark_inode_dirty (to); }static inline void ovl_copyflags (struct inode *from,struct inode *to)
fs/overlayfs/util.c +14 -0
@@ -950,3 +950,17 @@ char *ovl_get_redirect_xattr(struct ovl_fs *ofs, struct dentry *dentry,kfree (buf);return ERR_PTR (res); } + +/* +* We intentionally add I_DIRTY_SYNC flag regardless dirty flag +* of upper inode so that we have chance to invoke ->write_inode +* to re-dirty overlayfs' inode during writeback process. +*/ +void ovl_mark_inode_dirty (struct inode *inode) +{ +struct inode *upper =ovl_inode_upper (inode); +unsigned long iflag =I_DIRTY_SYNC ; + + iflag |= upper->i_state &I_DIRTY_ALL ; +__mark_inode_dirty (inode, iflag); +}
Signature · 1 line
2.26.2
RFC PATCH v2 5/8 ovl: mark overlayfs' inode dirty on shared writable mmap
In reply to Chengguang XuOverlayfs cannot be notified when mmapped area gets dirty, so we need to proactively mark inode dirty in ->mmap operation.
--- fs/overlayfs/file.c | 4 ++++ 1 file changed, 4 insertions(+)
fs/overlayfs/file.c +4 -0
@@ -486,6 +486,10 @@ static int ovl_mmap(struct file *file, struct vm_area_struct *vma)/* Drop reference count from new vm_file value */ fput (realfile); }else { +if (vma->vm_flags & (VM_SHARED |VM_MAYSHARE ) && + vma->vm_flags & (VM_WRITE |VM_MAYWRITE )) +ovl_mark_inode_dirty (file_inode (file)); +/* Drop reference count from previous vm_file value */ fput (file); }
Signature · 1 line
2.26.2
Jan Kara
reviewerRe: [RFC PATCH v2 5/8] ovl: mark overlayfs' inode dirty on shared writable mmap
Quoted 13 lines · Chengguang Xu
Overlayfs cannot be notified when mmapped area gets dirty, so we need to proactively mark inode dirty in ->mmap operation.
Signed-off-by Chengguang Xu <[email protected]>--- fs/overlayfs/file.c | 4 ++++ 1 file changed, 4 insertions(+)fs/overlayfs/file.c +0
-0
fs/overlayfs/file.c static int ovl_mmap(struct file *file, struct vm_area_struct *vma) · line 492
Quoted 7 lines
fs/overlayfs/file.c +4
-0@@ -486,6 +486,10 @@ static int ovl_mmap(struct file *file, struct vm_area_struct *vma)/* Drop reference count from new vm_file value */ fput (realfile); }else { +if (vma->vm_flags & (VM_SHARED |VM_MAYSHARE ) && + vma->vm_flags & (VM_WRITE |VM_MAYWRITE )) +ovl_mark_inode_dirty (file_inode (file)); +
But does this work reliably? I mean once writeback runs, your inode (as well as upper inode) is cleaned. Then a page fault comes so file has dirty pages again and would need flushing but overlayfs inode stays clean? Am I missing something?
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [RFC PATCH v2 5/8] ovl: mark overlayfs' inode dirty on shared writable mmap
---- 在 星期二, 2020-11-03 01:30:52 Jan Kara <[email protected]> 撰写 ----
Quoted 18 lines · Chengguang Xu
Quoted 17 lines
Overlayfs cannot be notified when mmapped area gets dirty, so we need to proactively mark inode dirty in ->mmap operation.
Signed-off-by Chengguang Xu <[email protected]>--- fs/overlayfs/file.c | 4 ++++ 1 file changed, 4 insertions(+)fs/overlayfs/file.c +1
-0@@ -486,6 +486,10 @@ static int ovl_mmap(struct file *file, struct vm_area_struct *vma)/* Drop reference count from new vm_file value */ fput (realfile); }else { + if (vma->vm_flags & (VM_SHARED|VM_MAYSHARE) &&
Quoted 9 lines
fs/overlayfs/file.c static int ovl_mmap(struct file *file, struct vm_area_struct *vma) · line 492
Quoted 3 lines
fs/overlayfs/file.c +3
-0+ vma->vm_flags & (VM_WRITE|VM_MAYWRITE)) +ovl_mark_inode_dirty (file_inode (file)); +But does this work reliably? I mean once writeback runs, your inode (as well as upper inode) is cleaned. Then a page fault comes so file has dirty pages again and would need flushing but overlayfs inode stays clean? Am I missing something?
Yeah, this is key point of this approach, in order to fix the issue I explicitly set I_DIRTY_SYNC flag in ovl_mark_inode_dirty(), so what i mean is during writeback we will call into ->write_inode() by this flag(I_DIRTY_SYNC) and at that place we get chance to check mapping and re-dirty overlay's inode. The code logic like below in ovl_write_inode().
if (mapping_writably_mapped(upper->i_mapping) ||
mapping_tagged(upper->i_mapping, PAGECACHE_TAG_WRITEBACK))
iflag |= I_DIRTY_PAGES; Thanks,
Chengguang
Jan Kara
reviewerRe: [RFC PATCH v2 5/8] ovl: mark overlayfs' inode dirty on shared writable mmap
Quoted 35 lines · Chengguang Xu
---- 在 星期二, 2020-11-03 01:30:52 Jan Kara <[email protected]> 撰写 ----
Quoted 18 lines · Chengguang Xu
Quoted 17 lines
Overlayfs cannot be notified when mmapped area gets dirty, so we need to proactively mark inode dirty in ->mmap operation.
Signed-off-by Chengguang Xu <[email protected]>--- fs/overlayfs/file.c | 4 ++++ 1 file changed, 4 insertions(+)fs/overlayfs/file.c +1
-0@@ -486,6 +486,10 @@ static int ovl_mmap(struct file *file, struct vm_area_struct *vma)/* Drop reference count from new vm_file value */ fput (realfile); }else { + if (vma->vm_flags & (VM_SHARED|VM_MAYSHARE) &&Quoted 9 lines
fs/overlayfs/file.c static int ovl_mmap(struct file *file, struct vm_area_struct *vma) · line 492
Quoted 3 lines
fs/overlayfs/file.c +3
-0+ vma->vm_flags & (VM_WRITE|VM_MAYWRITE)) +ovl_mark_inode_dirty (file_inode (file)); +But does this work reliably? I mean once writeback runs, your inode (as well as upper inode) is cleaned. Then a page fault comes so file has dirty pages again and would need flushing but overlayfs inode stays clean? Am I missing something?
Yeah, this is key point of this approach, in order to fix the issue I explicitly set
I_DIRTY_SYNCflag inovl_mark_inode_dirty(), so what i mean is during writeback we will call into ->write_inode() by thisflag(I_DIRTY_SYNC)and at that place we get chance to check mapping and re-dirty overlay's inode. The code logic like below inovl_write_inode().
Quoted 3 lines
if (mapping_writably_mapped(upper->i_mapping) ||
mapping_tagged(upper->i_mapping, PAGECACHE_TAG_WRITEBACK))
iflag |= I_DIRTY_PAGES; OK, but suppose the upper mapping is clean at this moment (upper inode has been fully written out for whatever reason, but it is still mapped) so your overlayfs inode becomes clean as well. Then I don't see a mechanism which would make your overlayfs inode dirty again when a write to mmap happens, set_page_dirty() will end up marking upper inode with I_DIRTY_PAGES flag.
Note that ovl_mmap() gets called only at mmap(2) syscall time but then pages get faulted in, dirtied, cleaned fully at discretion of the mm / writeback subsystem.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Amir Goldstein
maintainerRe: [RFC PATCH v2 5/8] ovl: mark overlayfs' inode dirty on shared writable mmap
Quoted 47 lines · Jan Kara
Quoted 35 lines · Chengguang Xu
---- 在 星期二, 2020-11-03 01:30:52 Jan Kara <[email protected]> 撰写 ----
Quoted 18 lines · Chengguang Xu
Quoted 17 lines
Overlayfs cannot be notified when mmapped area gets dirty, so we need to proactively mark inode dirty in ->mmap operation.
Signed-off-by Chengguang Xu <[email protected]>--- fs/overlayfs/file.c | 4 ++++ 1 file changed, 4 insertions(+)fs/overlayfs/file.c +1
-0@@ -486,6 +486,10 @@ static int ovl_mmap(struct file *file, struct vm_area_struct *vma)/* Drop reference count from new vm_file value */ fput (realfile); }else { + if (vma->vm_flags & (VM_SHARED|VM_MAYSHARE) &&Quoted 9 lines
fs/overlayfs/file.c static int ovl_mmap(struct file *file, struct vm_area_struct *vma) · line 492
Quoted 3 lines
fs/overlayfs/file.c +3
-0+ vma->vm_flags & (VM_WRITE|VM_MAYWRITE)) +ovl_mark_inode_dirty (file_inode (file)); +But does this work reliably? I mean once writeback runs, your inode (as well as upper inode) is cleaned. Then a page fault comes so file has dirty pages again and would need flushing but overlayfs inode stays clean? Am I missing something?
Yeah, this is key point of this approach, in order to fix the issue I explicitly set
I_DIRTY_SYNCflag inovl_mark_inode_dirty(), so what i mean is during writeback we will call into ->write_inode() by thisflag(I_DIRTY_SYNC)and at that place we get chance to check mapping and re-dirty overlay's inode. The code logic like below inovl_write_inode().Quoted 3 lines
if (mapping_writably_mapped(upper->i_mapping) || mapping_tagged(upper->i_mapping, PAGECACHE_TAG_WRITEBACK)) iflag |= I_DIRTY_PAGES;OK, but suppose the upper mapping is clean at this moment (upper inode has been fully written out for whatever reason, but it is still mapped) so your overlayfs inode becomes clean as well. Then I don't see a mechanism which would make your overlayfs inode dirty again when a write to mmap happens,
set_page_dirty()will end up marking upper inode withI_DIRTY_PAGESflag.
Quoted 4 lines
Note that
ovl_mmap()gets called only at mmap(2) syscall time but then pages get faulted in, dirtied, cleaned fully at discretion of the mm / writeback subsystem.
Perhaps I will add some background.
What I suggested was to maintain a "suspect list" in addition to the dirty ovl inodes.
ovl inode is added to the suspect list on mmap (writable) and removed from the suspect list on release() flush() or on sync_fs() if real inode is no longer writably mapped.
There was another variant where ovl inode is added to suspect list on open for write and removed from suspect list on release() flush() or sync_fs() if real inode is not inode_is_open_for_write().
In both cases the list will have inodes whose real is not dirty, but in both cases
the list shouldn't be terribly large to traverse on sync_fs().
Chengguang tried to implement the idea without an actual list by re-dirtying the "suspect" inodes on every write_inode(), but I personally have no idea if his idea works.
I think we can resort to using an actual suspect list if you say that it cannot work like this?
Thanks,
Amir.
Jan Kara
reviewerRe: [RFC PATCH v2 5/8] ovl: mark overlayfs' inode dirty on shared writable mmap
Quoted 70 lines · Amir Goldstein
Quoted 47 lines · Jan Kara
Quoted 35 lines · Chengguang Xu
---- 在 星期二, 2020-11-03 01:30:52 Jan Kara <[email protected]> 撰写 ----
Quoted 18 lines · Chengguang Xu
Quoted 17 lines
Overlayfs cannot be notified when mmapped area gets dirty, so we need to proactively mark inode dirty in ->mmap operation.
Signed-off-by Chengguang Xu <[email protected]>--- fs/overlayfs/file.c | 4 ++++ 1 file changed, 4 insertions(+)fs/overlayfs/file.c +1
-0@@ -486,6 +486,10 @@ static int ovl_mmap(struct file *file, struct vm_area_struct *vma)/* Drop reference count from new vm_file value */ fput (realfile); }else { + if (vma->vm_flags & (VM_SHARED|VM_MAYSHARE) &&Quoted 9 lines
fs/overlayfs/file.c static int ovl_mmap(struct file *file, struct vm_area_struct *vma) · line 492
Quoted 3 lines
fs/overlayfs/file.c +3
-0+ vma->vm_flags & (VM_WRITE|VM_MAYWRITE)) +ovl_mark_inode_dirty (file_inode (file)); +But does this work reliably? I mean once writeback runs, your inode (as well as upper inode) is cleaned. Then a page fault comes so file has dirty pages again and would need flushing but overlayfs inode stays clean? Am I missing something?
Yeah, this is key point of this approach, in order to fix the issue I explicitly set
I_DIRTY_SYNCflag inovl_mark_inode_dirty(), so what i mean is during writeback we will call into ->write_inode() by thisflag(I_DIRTY_SYNC)and at that place we get chance to check mapping and re-dirty overlay's inode. The code logic like below inovl_write_inode().Quoted 3 lines
if (mapping_writably_mapped(upper->i_mapping) || mapping_tagged(upper->i_mapping, PAGECACHE_TAG_WRITEBACK)) iflag |= I_DIRTY_PAGES;OK, but suppose the upper mapping is clean at this moment (upper inode has been fully written out for whatever reason, but it is still mapped) so your overlayfs inode becomes clean as well. Then I don't see a mechanism which would make your overlayfs inode dirty again when a write to mmap happens,
set_page_dirty()will end up marking upper inode withI_DIRTY_PAGESflag.Quoted 4 lines
Note that
ovl_mmap()gets called only at mmap(2) syscall time but then pages get faulted in, dirtied, cleaned fully at discretion of the mm / writeback subsystem.Perhaps I will add some background.
What I suggested was to maintain a "suspect list" in addition to the dirty ovl inodes.
ovl inode is added to the suspect list on mmap (writable) and removed from the suspect list on
release()flush()or onsync_fs()if real inode is no longer writably mapped.There was another variant where ovl inode is added to suspect list on open for write and removed from suspect list on
release()flush()orsync_fs()if real inode is notinode_is_open_for_write().In both cases the list will have inodes whose real is not dirty, but in both cases
the list shouldn't be terribly large to traverse onsync_fs().
Quoted 6 lines
Chengguang tried to implement the idea without an actual list by re-dirtying the "suspect" inodes on every
write_inode(), but I personally have no idea if his idea works.I think we can resort to using an actual suspect list if you say that it cannot work like this?
Yeah, the suspect list (i.e., additional list of inodes to check on sync) you describe should work fine. Also the "keep suspect inode dirty" idea of Chengguang could work fine but we'd have to use something like inode_is_open_for_write() or inode_is_writeably_mapped() (which would need to be implemented but it should be easy vma_interval_tree_foreach() walk checking each found VMA for vma->vm_flags & VM_WRITE) for checking whether inode should be redirtied or not.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [RFC PATCH v2 5/8] ovl: mark overlayfs' inode dirty on shared writable mmap
---- 在 星期四, 2020-11-05 23:54:34 Jan Kara <[email protected]> 撰写 ----
Quoted 75 lines · Amir Goldstein
Quoted 74 lines · Jan Kara
Quoted 47 lines · Chengguang Xu
Quoted 35 lines
---- 在 星期二, 2020-11-03 01:30:52 Jan Kara <[email protected]> 撰写 ----
Quoted 18 lines · Chengguang Xu
Quoted 17 lines
Overlayfs cannot be notified when mmapped area gets dirty, so we need to proactively mark inode dirty in ->mmap operation.
Signed-off-by Chengguang Xu <[email protected]>--- fs/overlayfs/file.c | 4 ++++ 1 file changed, 4 insertions(+)fs/overlayfs/file.c +1
-0@@ -486,6 +486,10 @@ static int ovl_mmap(struct file *file, struct vm_area_struct *vma)/* Drop reference count from new vm_file value */ fput (realfile); }else { + if (vma->vm_flags & (VM_SHARED|VM_MAYSHARE) &&Quoted 9 lines
fs/overlayfs/file.c static int ovl_mmap(struct file *file, struct vm_area_struct *vma) · line 492
Quoted 3 lines
fs/overlayfs/file.c +3
-0+ vma->vm_flags & (VM_WRITE|VM_MAYWRITE)) +ovl_mark_inode_dirty (file_inode (file)); +But does this work reliably? I mean once writeback runs, your inode (as well as upper inode) is cleaned. Then a page fault comes so file has dirty pages again and would need flushing but overlayfs inode stays clean? Am I missing something?
Yeah, this is key point of this approach, in order to fix the issue I explicitly set
I_DIRTY_SYNCflag inovl_mark_inode_dirty(), so what i mean is during writeback we will call into ->write_inode() by thisflag(I_DIRTY_SYNC)and at that place we get chance to check mapping and re-dirty overlay's inode. The code logic like below inovl_write_inode().Quoted 3 lines
if (mapping_writably_mapped(upper->i_mapping) || mapping_tagged(upper->i_mapping, PAGECACHE_TAG_WRITEBACK)) iflag |= I_DIRTY_PAGES;OK, but suppose the upper mapping is clean at this moment (upper inode has been fully written out for whatever reason, but it is still mapped) so your overlayfs inode becomes clean as well. Then I don't see a mechanism which would make your overlayfs inode dirty again when a write to mmap happens,
set_page_dirty()will end up marking upper inode withI_DIRTY_PAGESflag.Quoted 4 lines
Note that
ovl_mmap()gets called only at mmap(2) syscall time but then pages get faulted in, dirtied, cleaned fully at discretion of the mm / writeback subsystem.Perhaps I will add some background.
What I suggested was to maintain a "suspect list" in addition to the dirty ovl inodes.
ovl inode is added to the suspect list on mmap (writable) and removed from the suspect list on
release()flush()or onsync_fs()if real inode is no longer writably mapped.There was another variant where ovl inode is added to suspect list on open for write and removed from suspect list on
release()flush()orsync_fs()if real inode is notinode_is_open_for_write().In both cases the list will have inodes whose real is not dirty, but in both cases
the list shouldn't be terribly large to traverse onsync_fs().Chengguang tried to implement the idea without an actual list by re-dirtying the "suspect" inodes on every
write_inode(), but I personally have no idea if his idea works.
Quoted 5 lines
Quoted 2 lines
I think we can resort to using an actual suspect list if you say that it cannot work like this?
Yeah, the suspect list (i.e., additional list of inodes to check on sync) you describe should work fine.
I think this solution still has the problem we have met in below thread[1] The main problem is the state combination of clean overlayfs' inode && dirty upper inode.
[1] https://www.spinics.net/lists/linux-unionfs/msg07448.html
Quoted 7 lines
Also the "keep suspect inode dirty" idea
of Chengguang could work fine but we'd have to use something likeinode_is_open_for_write()orinode_is_writeably_mapped()(which would need to be implemented but it should be easyvma_interval_tree_foreach()walk checking each found VMA forvma->vm_flags&VM_WRITE) for checking whether inode should be redirtied or not.
I'm curious that isn't it enough to check i_mmap_writable by mapping_writably_mapped() ? Am I missing something?
Thanks,
Chengguang
Jan Kara
reviewerRe: [RFC PATCH v2 5/8] ovl: mark overlayfs' inode dirty on shared writable mmap
Quoted 76 lines · Chengguang Xu
---- 在 星期四, 2020-11-05 23:54:34 Jan Kara <[email protected]> 撰写 ----
Quoted 75 lines · Amir Goldstein
Quoted 74 lines · Jan Kara
Quoted 47 lines · Chengguang Xu
Quoted 35 lines
---- 在 星期二, 2020-11-03 01:30:52 Jan Kara <[email protected]> 撰写 ----
Quoted 18 lines · Chengguang Xu
Quoted 17 lines
Overlayfs cannot be notified when mmapped area gets dirty, so we need to proactively mark inode dirty in ->mmap operation.
Signed-off-by Chengguang Xu <[email protected]>--- fs/overlayfs/file.c | 4 ++++ 1 file changed, 4 insertions(+)fs/overlayfs/file.c +1
-0@@ -486,6 +486,10 @@ static int ovl_mmap(struct file *file, struct vm_area_struct *vma)/* Drop reference count from new vm_file value */ fput (realfile); }else { + if (vma->vm_flags & (VM_SHARED|VM_MAYSHARE) &&Quoted 9 lines
fs/overlayfs/file.c static int ovl_mmap(struct file *file, struct vm_area_struct *vma) · line 492
Quoted 3 lines
fs/overlayfs/file.c +3
-0+ vma->vm_flags & (VM_WRITE|VM_MAYWRITE)) +ovl_mark_inode_dirty (file_inode (file)); +But does this work reliably? I mean once writeback runs, your inode (as well as upper inode) is cleaned. Then a page fault comes so file has dirty pages again and would need flushing but overlayfs inode stays clean? Am I missing something?
Yeah, this is key point of this approach, in order to fix the issue I explicitly set
I_DIRTY_SYNCflag inovl_mark_inode_dirty(), so what i mean is during writeback we will call into ->write_inode() by thisflag(I_DIRTY_SYNC)and at that place we get chance to check mapping and re-dirty overlay's inode. The code logic like below inovl_write_inode().Quoted 3 lines
if (mapping_writably_mapped(upper->i_mapping) || mapping_tagged(upper->i_mapping, PAGECACHE_TAG_WRITEBACK)) iflag |= I_DIRTY_PAGES;OK, but suppose the upper mapping is clean at this moment (upper inode has been fully written out for whatever reason, but it is still mapped) so your overlayfs inode becomes clean as well. Then I don't see a mechanism which would make your overlayfs inode dirty again when a write to mmap happens,
set_page_dirty()will end up marking upper inode withI_DIRTY_PAGESflag.Quoted 4 lines
Note that
ovl_mmap()gets called only at mmap(2) syscall time but then pages get faulted in, dirtied, cleaned fully at discretion of the mm / writeback subsystem.Perhaps I will add some background.
What I suggested was to maintain a "suspect list" in addition to the dirty ovl inodes.
ovl inode is added to the suspect list on mmap (writable) and removed from the suspect list on
release()flush()or onsync_fs()if real inode is no longer writably mapped.There was another variant where ovl inode is added to suspect list on open for write and removed from suspect list on
release()flush()orsync_fs()if real inode is notinode_is_open_for_write().In both cases the list will have inodes whose real is not dirty, but in both cases
the list shouldn't be terribly large to traverse onsync_fs().Chengguang tried to implement the idea without an actual list by re-dirtying the "suspect" inodes on every
write_inode(), but I personally have no idea if his idea works.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
I think we can resort to using an actual suspect list if you say that it cannot work like this?
Yeah, the suspect list (i.e., additional list of inodes to check on sync) you describe should work fine.
I think this solution still has the problem we have met in below thread[1] The main problem is the state combination of clean overlayfs' inode && dirty upper inode.
But I think the scheme Amir proposed and I detailed in my previous email should prevent that state. Because while the inode is mapped, it will be kept in the dirty list. So which scenario do you think would lead to clean overlayfs inode and dirty upper inode?
Quoted 10 lines
[1] https://www.spinics.net/lists/linux-unionfs/msg07448.html
Quoted 7 lines
Also the "keep suspect inode dirty" idea
of Chengguang could work fine but we'd have to use something likeinode_is_open_for_write()orinode_is_writeably_mapped()(which would need to be implemented but it should be easyvma_interval_tree_foreach()walk checking each found VMA forvma->vm_flags&VM_WRITE) for checking whether inode should be redirtied or not.
Quoted 2 lines
I'm curious that isn't it enough to check
i_mmap_writablebymapping_writably_mapped()? Am I missing something?
What is i_mmap_writeable? I've grepped the tree and didn't find anything like that...
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [RFC PATCH v2 5/8] ovl: mark overlayfs' inode dirty on shared writable mmap
---- 在 星期五, 2020-11-06 16:50:23 Jan Kara <[email protected]> 撰写 ----
Quoted 83 lines · Chengguang Xu
Quoted 82 lines
---- 在 星期四, 2020-11-05 23:54:34 Jan Kara <[email protected]> 撰写 ----
Quoted 80 lines · Amir Goldstein
Quoted 70 lines · Jan Kara
Quoted 47 lines · Chengguang Xu
Quoted 35 lines
---- 在 星期二, 2020-11-03 01:30:52 Jan Kara <[email protected]> 撰写 ----
Quoted 18 lines · Chengguang Xu
Quoted 17 lines
Overlayfs cannot be notified when mmapped area gets dirty, so we need to proactively mark inode dirty in ->mmap operation.
Signed-off-by Chengguang Xu <[email protected]>--- fs/overlayfs/file.c | 4 ++++ 1 file changed, 4 insertions(+)fs/overlayfs/file.c +1
-0@@ -486,6 +486,10 @@ static int ovl_mmap(struct file *file, struct vm_area_struct *vma)/* Drop reference count from new vm_file value */ fput (realfile); }else { + if (vma->vm_flags & (VM_SHARED|VM_MAYSHARE) &&Quoted 9 lines
fs/overlayfs/file.c static int ovl_mmap(struct file *file, struct vm_area_struct *vma) · line 492
Quoted 3 lines
fs/overlayfs/file.c +3
-0+ vma->vm_flags & (VM_WRITE|VM_MAYWRITE)) +ovl_mark_inode_dirty (file_inode (file)); +But does this work reliably? I mean once writeback runs, your inode (as well as upper inode) is cleaned. Then a page fault comes so file has dirty pages again and would need flushing but overlayfs inode stays clean? Am I missing something?
Yeah, this is key point of this approach, in order to fix the issue I explicitly set
I_DIRTY_SYNCflag inovl_mark_inode_dirty(), so what i mean is during writeback we will call into ->write_inode() by thisflag(I_DIRTY_SYNC)and at that place we get chance to check mapping and re-dirty overlay's inode. The code logic like below inovl_write_inode().Quoted 3 lines
if (mapping_writably_mapped(upper->i_mapping) || mapping_tagged(upper->i_mapping, PAGECACHE_TAG_WRITEBACK)) iflag |= I_DIRTY_PAGES;OK, but suppose the upper mapping is clean at this moment (upper inode has been fully written out for whatever reason, but it is still mapped) so your overlayfs inode becomes clean as well. Then I don't see a mechanism which would make your overlayfs inode dirty again when a write to mmap happens,
set_page_dirty()will end up marking upper inode withI_DIRTY_PAGESflag.Quoted 4 lines
Note that
ovl_mmap()gets called only at mmap(2) syscall time but then pages get faulted in, dirtied, cleaned fully at discretion of the mm / writeback subsystem.Perhaps I will add some background.
What I suggested was to maintain a "suspect list" in addition to the dirty ovl inodes.
ovl inode is added to the suspect list on mmap (writable) and removed from the suspect list on
release()flush()or onsync_fs()if real inode is no longer writably mapped.There was another variant where ovl inode is added to suspect list on open for write and removed from suspect list on
release()flush()orsync_fs()if real inode is notinode_is_open_for_write().In both cases the list will have inodes whose real is not dirty, but in both cases
the list shouldn't be terribly large to traverse onsync_fs().Quoted 6 lines
Chengguang tried to implement the idea without an actual list by re-dirtying the "suspect" inodes on every
write_inode(), but I personally have no idea if his idea works.I think we can resort to using an actual suspect list if you say that it cannot work like this?
Yeah, the suspect list (i.e., additional list of inodes to check on sync) you describe should work fine.
Quoted 7 lines
Quoted 2 lines
I think this solution still has the problem we have met in below thread[1] The main problem is the state combination of clean overlayfs' inode && dirty upper inode.
But I think the scheme Amir proposed and I detailed in my previous email should prevent that state. Because while the inode is mapped, it will be kept in the dirty list. So which scenario do you think would lead to clean overlayfs inode and dirty upper inode?
If keeping in the dirty list means making overlayfs inode dirty, then I think we don't need extra list for that, vfs itself has writeback list and the solution will be exactly the same as mine(re-dirty) . Right?
Quoted 11 lines
Quoted 11 lines
[1] https://www.spinics.net/lists/linux-unionfs/msg07448.html
Quoted 7 lines
Also the "keep suspect inode dirty" idea
of Chengguang could work fine but we'd have to use something likeinode_is_open_for_write()orinode_is_writeably_mapped()(which would need to be implemented but it should be easyvma_interval_tree_foreach()walk checking each found VMA forvma->vm_flags&VM_WRITE) for checking whether inode should be redirtied or not.
Quoted 6 lines
Quoted 2 lines
I'm curious that isn't it enough to check
i_mmap_writablebymapping_writably_mapped()? Am I missing something?What is
i_mmap_writeable? I've grepped the tree and didn't find anything like that...
Maybe spelling mistake? The reason I check this is I'm afraid of the permission change of vma by mprotect(2).
Thanks,
Chenguang
RFC PATCH v2 6/8 ovl: implement overlayfs' ->write_inode operation
In reply to Chengguang XuImplement overlayfs' ->write_inode to sync dirty data and redirty overlayfs' inode if necessary.
--- fs/overlayfs/super.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)
fs/overlayfs/super.c +26 -0
@@ -15,6 +15,7 @@#include <linux/seq_file.h> #include <linux/posix_acl_xattr.h> #include <linux/exportfs.h> +#include <linux/writeback.h> #include "overlayfs.h" MODULE_AUTHOR ("Miklos Szeredi <[email protected]>" );
@@ -396,11 +397,36 @@ static void ovl_evict_inode (struct inode *inode)clear_inode (inode); } +static int ovl_write_inode (struct inode *inode, +struct writeback_control *wbc) +{ +struct inode *upper =ovl_inode_upper (inode); +unsigned long iflag =0 ; +int ret =0 ; + +if (!upper) +return 0 ; + +if (upper->i_sb->s_op->write_inode) + ret = upper->i_sb->s_op->write_inode (inode, wbc); + + iflag |= upper->i_state &I_DIRTY_ALL ; +if (mapping_writably_mapped (upper->i_mapping) || +mapping_tagged (upper->i_mapping,PAGECACHE_TAG_WRITEBACK )) + iflag |=I_DIRTY_PAGES ; + +if (iflag) +ovl_mark_inode_dirty (inode); + +return ret; +} +static const struct super_operations ovl_super_operations = { .alloc_inode = ovl_alloc_inode, .free_inode = ovl_free_inode, .destroy_inode = ovl_destroy_inode, .drop_inode = generic_delete_inode, + .write_inode = ovl_write_inode, .evict_inode = ovl_evict_inode, .put_super = ovl_put_super, .sync_fs = ovl_sync_fs,
Signature · 1 line
2.26.2
RFC PATCH v2 7/8 ovl: cache dirty overlayfs' inode
In reply to Chengguang XuNow drop overlayfs' inode will sync dirty data,
so we change to only drop clean inode.
--- fs/overlayfs/super.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
fs/overlayfs/super.c +11 -1
@@ -421,11 +421,21 @@ static int ovl_write_inode(struct inode *inode,return ret; } +int ovl_drop_inode (struct inode *inode) +{ +struct inode *upper =ovl_inode_upper (inode); + +if (!upper || !(inode->i_state &I_DIRTY_ALL )) +return 1 ; + +return generic_drop_inode (inode); +} +static const struct super_operations ovl_super_operations = { .alloc_inode = ovl_alloc_inode, .free_inode = ovl_free_inode, .destroy_inode = ovl_destroy_inode, - .drop_inode = generic_delete_inode, + .drop_inode = ovl_drop_inode, .write_inode = ovl_write_inode, .evict_inode = ovl_evict_inode, .put_super = ovl_put_super,
Signature · 1 line
2.26.2
RFC PATCH v2 8/8 ovl: implement containerized syncfs for overlayfs
In reply to Chengguang XuNow overlayfs can only sync dirty inode during
syncfs, so remove unnecessary sync_filesystem()
on upper file system.
--- fs/overlayfs/super.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
fs/overlayfs/super.c +5 -3
@@ -271,8 +271,7 @@ static int ovl_sync_fs(struct super_block *sb, int wait)* Not called for sync(2) call or an emergency sync (SB_I_SKIP_SYNC). * All the super blocks will be iterated, including upper_sb. * -* If this is a syncfs(2) call, then we do need to call -* sync_filesystem() on upper_sb, but enough if we do it when being +* If this is a syncfs(2) call, it will be enough we do it when being * called with wait == 1. */ if (!wait)
@@ -281,7 +280,10 @@ static int ovl_sync_fs(struct super_block *sb, int wait) upper_sb =ovl_upper_mnt (ofs)->mnt_sb;down_read (&upper_sb->s_umount); - ret =sync_filesystem (upper_sb); +if (upper_sb->s_op->sync_fs) + ret = upper_sb->s_op->sync_fs (upper_sb, wait); +if (!ret) + ret =sync_blockdev (upper_sb->s_bdev);up_read (&upper_sb->s_umount);return ret;
Signature · 1 line
2.26.2
Miklos Szeredi
maintainerQuoted 11 lines · Chengguang Xu
Current syncfs(2) syscall on overlayfs just calls
sync_filesystem()onupper_sbto synchronize whole dirty inodes in upper filesystem regardless of the overlay ownership of the inode. In the use case of container, when multiple containers using the same underlying upper filesystem, it has some shortcomings as below.(1) Performance
Synchronization is probably heavy because it actually syncs unnecessary inodes for target overlayfs.
Quoted 8 lines
(2) Interference
Unplanned synchronization will probably impact IO performance of unrelated container processes on the other overlayfs.This series try to implement containerized syncfs for overlayfs so that only sync target dirty upper inodes which are belong to specific overlayfs instance. By doing this, it is able to reduce cost of synchronization and will not seriously impact IO performance of unrelated processes.
Series looks good at first glance. Still need to do an in-depth review.
In the meantime can you post some numbers showing the performance improvements?
Thanks,
Miklos
---- 在 星期五, 2020-10-30 23:46:00 Miklos Szeredi <[email protected]> 撰写 ----
Quoted 16 lines · Chengguang Xu
Quoted 15 lines
Current syncfs(2) syscall on overlayfs just calls
sync_filesystem()onupper_sbto synchronize whole dirty inodes in upper filesystem regardless of the overlay ownership of the inode. In the use case of container, when multiple containers using the same underlying upper filesystem, it has some shortcomings as below.(1) Performance
Synchronization is probably heavy because it actually syncs unnecessary inodes for target overlayfs.(2) Interference
Unplanned synchronization will probably impact IO performance of unrelated container processes on the other overlayfs.
Quoted 9 lines
Quoted 4 lines
This series try to implement containerized syncfs for overlayfs so that only sync target dirty upper inodes which are belong to specific overlayfs instance. By doing this, it is able to reduce cost of synchronization and will not seriously impact IO performance of unrelated processes.
Series looks good at first glance. Still need to do an in-depth review.
In the meantime can you post some numbers showing the performance improvements?
Actually, the performance improvement depends on workload and the number of overlay instances.
I did a very simple test on my dev box (intel NUC ) and the result seems quite obvious.
I made two overlay instances and untar a kernel source to an overlay then compare
umount time(sys part) of two instances.
------------------------------------------
test env:
Intel(R) Core(TM) i7-8809G CPU @ 3.10GHz 8core
32GB memory
Samsung NVMe SSD 970 pro
test script:
[root@localhost ovl]# cat do.sh
#!/bin/sh
rm -rf upper/* &> /dev/null
echo 3 > /proc/sys/vm/drop_caches
mount -t overlay overlay -o lowerdir=lower,workdir=work,upperdir=upper merged mount -t overlay overlay -o lowerdir=lower2,workdir=work2,upperdir=upper2 merged2
tar Jxvf linux-5.9.tar.xz -C merged &> /dev/null
time umount merged2
time umount merged
---------------------------------
test result:
[root@localhost ovl]# seq 3 | while read line ; do sh -x do.sh ; done
+ rm -rf upper/linux-5.9
+ echo 3
+ mount -t overlay overlay -o lowerdir=lower,workdir=work,upperdir=upper merged
+ mount -t overlay overlay -o lowerdir=lower2,workdir=work2,upperdir=upper2 merged2
+ tar Jxvf linux-5.9.tar.xz -C merged
+ umount merged2
real 0m0.437s user 0m0.001s sys 0m0.001s
+ umount merged
real 0m2.107s user 0m0.002s sys 0m0.090s
+ rm -rf upper/linux-5.9
+ echo 3
+ mount -t overlay overlay -o lowerdir=lower,workdir=work,upperdir=upper merged
+ mount -t overlay overlay -o lowerdir=lower2,workdir=work2,upperdir=upper2 merged2
+ tar Jxvf linux-5.9.tar.xz -C merged
+ umount merged2
real 0m0.443s user 0m0.002s sys 0m0.001s
+ umount merged
real 0m2.032s user 0m0.001s sys 0m0.105s
+ rm -rf upper/linux-5.9
+ echo 3
+ mount -t overlay overlay -o lowerdir=lower,workdir=work,upperdir=upper merged
+ mount -t overlay overlay -o lowerdir=lower2,workdir=work2,upperdir=upper2 merged2
+ tar Jxvf linux-5.9.tar.xz -C merged
+ umount merged2
real 0m0.263s user 0m0.001s sys 0m0.001s
+ umount merged
real 0m2.094s user 0m0.000s sys 0m0.083s
[root@localhost o
Thanks,
Chengguang