Skip to contentSkip to conversation

ovl: skip overlayfs superblocks at global sync

PATCH Applied Konstantin Khlebnikov 4 participants 9 messages Apr 9, 2020 – Apr 21, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ovl: skip overlayfs superblocks at global sync

Stacked filesystems like overlayfs has no own writeback, but they have to forward syncfs() requests to backend for keeping data integrity.

During global sync() each overlayfs instance calls method ->sync_fs() for backend although it itself is in global list of superblocks too. As a result one syscall sync() could write one superblock several times and send multiple disk barriers.

This patch adds flag SB_I_SKIP_SYNC into sb->sb_iflags to avoid that.

---
 fs/overlayfs/super.c |    5 +++--
 fs/sync.c            |    3 ++-
 include/linux/fs.h   |    2 ++
 3 files changed, 7 insertions(+), 3 deletions(-)
fs/overlayfs/super.c +3 -2
index 732ad5495c92..59df13d16280 100644
@@ -261,8 +261,8 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
 		return 0;
 
 	/*
-	 * If this is a sync(2) call or an emergency sync, all the super blocks
-	 * will be iterated, including upper_sb, so no need to do anything.
+	 * Not called for sync(2) call or an emergency sync (SB_I_SKIP_SYNC).
+	 * All the super blocks will be iterated, including upper_sb.
 	 *
 	 * If this is a syncfs(2) call, then we do need to call
 	 * sync_filesystem() on upper_sb, but enough if we do it when being
@@ -1818,6 +1818,7 @@ static int ovl_fill_super(struct super_block *sb, void *data, int silent)
 	sb->s_xattr = ovl_xattr_handlers;
 	sb->s_fs_info = ofs;
 	sb->s_flags |= SB_POSIXACL;
+	sb->s_iflags |= SB_I_SKIP_SYNC;
 
 	err = -ENOMEM;
 	root_dentry = ovl_get_root(sb, upperpath.dentry, oe);
fs/sync.c +2 -1
index 4d1ff010bc5a..16c2630ee4bf 100644
@@ -76,7 +76,8 @@ static void sync_inodes_one_sb(struct super_block *sb, void *arg)
 
 static void sync_fs_one_sb(struct super_block *sb, void *arg)
 {
-	if (!sb_rdonly(sb) && sb->s_op->sync_fs)
+	if (!sb_rdonly(sb) && !(sb->s_iflags & SB_I_SKIP_SYNC) &&
+	    sb->s_op->sync_fs)
 		sb->s_op->sync_fs(sb, *(int *)arg);
 }
 
include/linux/fs.h +2 -0
index 4f6f59b4f22a..f186a966a36c 100644
@@ -1409,6 +1409,8 @@ extern int send_sigurg(struct fown_struct *fown);
 #define SB_I_IMA_UNVERIFIABLE_SIGNATURE	0x00000020
 #define SB_I_UNTRUSTED_MOUNTER		0x00000040
 
+#define SB_I_SKIP_SYNC	0x00000100	/* Skip superblock at global sync */
+
 /* Possible states of 'frozen' field */
 enum {
 	SB_UNFROZEN = 0,		/* FS is unfrozen */
maintainer Reviewed-by on v1 1/1
#1 lore raw
Quoted 9 lines · Konstantin Khlebnikov

Stacked filesystems like overlayfs has no own writeback, but they have to forward syncfs() requests to backend for keeping data integrity.

During global sync() each overlayfs instance calls method ->sync_fs() for backend although it itself is in global list of superblocks too. As a result one syscall sync() could write one superblock several times and send multiple disk barriers.

Quoted 5 lines

This patch adds flag SB_I_SKIP_SYNC into sb->sb_iflags to avoid that.

---

Seems reasonable.
You may add:

+CC: containers list

This bring up old memories.
I posted this way back to fix handling of emergency_remount() in the presence of loop mounted fs:
https://lore.kernel.org/linux-ext4/CAA2m6vfatWKS1CQFpaRbii2AXiZFvQUjVvYhGxWTSpz+2rxDyg@mail.gmail.com/

But seems to me that emergency_sync() and sync(2) are equally broken for this use case.

I wonder if anyone cares enough about resilience of loop mounted fs to try and change the iterate_* functions to iterate supers/bdevs in reverse order...

Thanks,
Amir.

#2 lore raw
Quoted 17 lines · Amir Goldstein
Quoted 9 lines · Konstantin Khlebnikov

Stacked filesystems like overlayfs has no own writeback, but they have to forward syncfs() requests to backend for keeping data integrity.

During global sync() each overlayfs instance calls method ->sync_fs() for backend although it itself is in global list of superblocks too. As a result one syscall sync() could write one superblock several times and send multiple disk barriers.

Quoted 5 lines

This patch adds flag SB_I_SKIP_SYNC into sb->sb_iflags to avoid that.

---

Quoted 5 lines

Seems reasonable.
You may add:

+CC: containers list

Thanks

Quoted 11 lines

This bring up old memories.
I posted this way back to fix handling of emergency_remount() in the presence of loop mounted fs:
https://lore.kernel.org/linux-ext4/CAA2m6vfatWKS1CQFpaRbii2AXiZFvQUjVvYhGxWTSpz+2rxDyg@mail.gmail.com/

But seems to me that emergency_sync() and sync(2) are equally broken for this use case.

I wonder if anyone cares enough about resilience of loop mounted fs to try and change the iterate_* functions to iterate supers/bdevs in reverse order...

Now I see reason behind "sync; sync; sync; reboot" =)

Order old -> new allows to not miss new items if list modifies. Might be important for some users.

bdev iteration seems already reversed: inode_sb_list_add adds to the head

Quoted 4 lines

Thanks,
Amir.

maintainer
#3 lore raw
Quoted 36 lines · Konstantin Khlebnikov
Quoted 17 lines · Amir Goldstein
Quoted 9 lines · Konstantin Khlebnikov

Stacked filesystems like overlayfs has no own writeback, but they have to forward syncfs() requests to backend for keeping data integrity.

During global sync() each overlayfs instance calls method ->sync_fs() for backend although it itself is in global list of superblocks too. As a result one syscall sync() could write one superblock several times and send multiple disk barriers.

Quoted 5 lines

This patch adds flag SB_I_SKIP_SYNC into sb->sb_iflags to avoid that.

---

Quoted 5 lines

Seems reasonable.
You may add:

+CC: containers list

Thanks

Quoted 9 lines

This bring up old memories.
I posted this way back to fix handling of emergency_remount() in the presence of loop mounted fs:
https://lore.kernel.org/linux-ext4/CAA2m6vfatWKS1CQFpaRbii2AXiZFvQUjVvYhGxWTSpz+2rxDyg@mail.gmail.com/

But seems to me that emergency_sync() and sync(2) are equally broken for this use case.

Quoted 8 lines
Quoted 2 lines

I wonder if anyone cares enough about resilience of loop mounted fs to try and change the iterate_* functions to iterate supers/bdevs in reverse order...

Now I see reason behind "sync; sync; sync; reboot" =)

Order old -> new allows to not miss new items if list modifies. Might be important for some users.

That's not the reason I suggested reverse order.
The reason is that with loop mounted fs, the correct order of flushing is:
1. sync loop mounted fs inodes => writes to loop image file
2. sync loop mounted fs sb => fsyncs the loop image file
3. sync the loop image host fs sb

With forward sb iteration order, #3 happens before #1, so the loop mounted fs changes are not really being made durable by a single sync(2) call.

Quoted 2 lines

bdev iteration seems already reversed: inode_sb_list_add adds to the head

I think bdev iteration order will not make a difference in this case. flushing /dev/loopX will not be needed and it happens too late anyway.

Thanks,
Amir.

#4 lore raw
Quoted 53 lines · Amir Goldstein
Quoted 36 lines · Konstantin Khlebnikov
Quoted 17 lines · Amir Goldstein
Quoted 9 lines · Konstantin Khlebnikov

Stacked filesystems like overlayfs has no own writeback, but they have to forward syncfs() requests to backend for keeping data integrity.

During global sync() each overlayfs instance calls method ->sync_fs() for backend although it itself is in global list of superblocks too. As a result one syscall sync() could write one superblock several times and send multiple disk barriers.

Quoted 5 lines

This patch adds flag SB_I_SKIP_SYNC into sb->sb_iflags to avoid that.

---

Quoted 5 lines

Seems reasonable.
You may add:

+CC: containers list

Thanks

Quoted 9 lines

This bring up old memories.
I posted this way back to fix handling of emergency_remount() in the presence of loop mounted fs:
https://lore.kernel.org/linux-ext4/CAA2m6vfatWKS1CQFpaRbii2AXiZFvQUjVvYhGxWTSpz+2rxDyg@mail.gmail.com/

But seems to me that emergency_sync() and sync(2) are equally broken for this use case.

Quoted 8 lines
Quoted 2 lines

I wonder if anyone cares enough about resilience of loop mounted fs to try and change the iterate_* functions to iterate supers/bdevs in reverse order...

Now I see reason behind "sync; sync; sync; reboot" =)

Order old -> new allows to not miss new items if list modifies. Might be important for some users.

That's not the reason I suggested reverse order.
The reason is that with loop mounted fs, the correct order of flushing is:
1. sync loop mounted fs inodes => writes to loop image file
2. sync loop mounted fs sb => fsyncs the loop image file
3. sync the loop image host fs sb

Quoted 3 lines

With forward sb iteration order, #3 happens before #1, so the loop mounted fs changes are not really being made durable by a single sync(2) call.

If fs in loop mounted with barriers then sync_fs will issue REQ_OP_FLUSH to loop device and trigger fsync() for image file. Sync() might write something twice but data should be safe. Without barriers this scenario is broken for sure.

Emergency remount R/O is other thing. It really needs reverse order.

Quoted 11 lines
Quoted 3 lines

bdev iteration seems already reversed: inode_sb_list_add adds to the head

I think bdev iteration order will not make a difference in this case. flushing /dev/loopX will not be needed and it happens too late anyway.

Thanks,
Amir.

maintainer
#5 lore raw
Quoted 61 lines · Konstantin Khlebnikov
Quoted 53 lines · Amir Goldstein
Quoted 36 lines · Konstantin Khlebnikov
Quoted 17 lines · Amir Goldstein
Quoted 9 lines · Konstantin Khlebnikov

Stacked filesystems like overlayfs has no own writeback, but they have to forward syncfs() requests to backend for keeping data integrity.

During global sync() each overlayfs instance calls method ->sync_fs() for backend although it itself is in global list of superblocks too. As a result one syscall sync() could write one superblock several times and send multiple disk barriers.

Quoted 5 lines

This patch adds flag SB_I_SKIP_SYNC into sb->sb_iflags to avoid that.

---

Quoted 5 lines

Seems reasonable.
You may add:

+CC: containers list

Thanks

Quoted 9 lines

This bring up old memories.
I posted this way back to fix handling of emergency_remount() in the presence of loop mounted fs:
https://lore.kernel.org/linux-ext4/CAA2m6vfatWKS1CQFpaRbii2AXiZFvQUjVvYhGxWTSpz+2rxDyg@mail.gmail.com/

But seems to me that emergency_sync() and sync(2) are equally broken for this use case.

Quoted 8 lines
Quoted 2 lines

I wonder if anyone cares enough about resilience of loop mounted fs to try and change the iterate_* functions to iterate supers/bdevs in reverse order...

Now I see reason behind "sync; sync; sync; reboot" =)

Order old -> new allows to not miss new items if list modifies. Might be important for some users.

That's not the reason I suggested reverse order.
The reason is that with loop mounted fs, the correct order of flushing is:
1. sync loop mounted fs inodes => writes to loop image file
2. sync loop mounted fs sb => fsyncs the loop image file
3. sync the loop image host fs sb

Quoted 3 lines

With forward sb iteration order, #3 happens before #1, so the loop mounted fs changes are not really being made durable by a single sync(2) call.

Quoted 7 lines

If fs in loop mounted with barriers then sync_fs will issue REQ_OP_FLUSH to loop device and trigger fsync() for image file. Sync() might write something twice but data should be safe. Without barriers this scenario is broken for sure.

Emergency remount R/O is other thing. It really needs reverse order.

Correct. There is no problem with durability.
Although for some filesystems it would be more efficient to first write and fsync the loop images and then sync_fs().
I can potentially result in less overall disk barriers.

Thanks,
Amir.

#6 lore raw
In reply to Konstantin Khlebnikov
Quoted 9 lines · Konstantin Khlebnikov

Stacked filesystems like overlayfs has no own writeback, but they have to forward syncfs() requests to backend for keeping data integrity.

During global sync() each overlayfs instance calls method ->sync_fs() for backend although it itself is in global list of superblocks too. As a result one syscall sync() could write one superblock several times and send multiple disk barriers.

This patch adds flag SB_I_SKIP_SYNC into sb->sb_iflags to avoid that.

Why wouldn't you just remove the ->sync_fs method from overlay?

I mean, if you don't need the filesystem to do anything special for one specific data integrity sync_fs call, you don't need it for any of them, yes?

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
maintainer
#7 lore raw
Quoted 10 lines · Dave Chinner
Quoted 8 lines · Konstantin Khlebnikov

Stacked filesystems like overlayfs has no own writeback, but they have to forward syncfs() requests to backend for keeping data integrity.

During global sync() each overlayfs instance calls method ->sync_fs() for backend although it itself is in global list of superblocks too. As a result one syscall sync() could write one superblock several times and send multiple disk barriers.

Quoted 8 lines
Quoted 1 line

This patch adds flag SB_I_SKIP_SYNC into sb->sb_iflags to avoid that.

Why wouldn't you just remove the ->sync_fs method from overlay?

I mean, if you don't need the filesystem to do anything special for one specific data integrity sync_fs call, you don't need it for any of them, yes?

No, but I understand the confusion.

Say you have 1000 overlay sb's all of them using upper directories from a single xfs sb (quite common for containers).

syncfs(2) of each overlay, must call sync_fs of xfs (see ovl_sync_fs) sync(2) will call xfs sync_fs anyway, so there is no point in calling ovl_sync_fs => xfs sync_fs 1000 more times.

Thanks,
Amir.

maintainer Applied
#8 lore raw
In reply to Konstantin Khlebnikov
Quoted 9 lines · Konstantin Khlebnikov

Stacked filesystems like overlayfs has no own writeback, but they have to forward syncfs() requests to backend for keeping data integrity.

During global sync() each overlayfs instance calls method ->sync_fs() for backend although it itself is in global list of superblocks too. As a result one syscall sync() could write one superblock several times and send multiple disk barriers.

Quoted 4 lines

This patch adds flag SB_I_SKIP_SYNC into sb->sb_iflags to avoid that.

Thanks, applied.

Miklos