Skip to contentSkip to conversation

ovl: lazy copy up of data on first write

RFC PATCH In review Amir Goldstein 2 participants 7 messages Jan 18, 2019 Overlay filesystem

lore
Share & tools
lore

Amir Goldstein

author maintainer
#0 lore raw

RFC PATCH 1/1 ovl: lazy copy up of data on first write

When metacopy feature is enabled, copy up only metadata when opening a file O_RDWR and defer copy up of data until the first write operation.

On read operations of lazy copy up file, the lower file is opened O_RDONLY on every vfs call until the first write operation.

XXX: mmap() of a lazy opened file will map the meta inode, so this still needs to be fixed.

---

Hi Miklos,

Here's a draft for lazy copy up of data.
Sending it out for your initial feedback.

When opening file for write, we always cache a real file with upper inode, but that inode may be a meta copy.

This leaves ovl_mmap() doing the wrong thing (mapping the meta copy).

One way do deal with this would be to implement a_ops as you suggested and copy up data on page write fault.

Do you have another idea how to make use of this RFC patch without implementing a_aops? Or maybe there is no reason to bother?

Perhaps we could install ovl_a_aop only for files open for write as first stage? I think that would be easy enough for me to do. I still don't have the full picture in my head for dealing with LOWER->LOWER_SHARED transition.

This RFC patch BTW passes unionmount and the overlay/quick xfstests with metacopy=on/off.

The one test that does fail is the metacopy test overlay/060 because it fails the expectation to find a data copied up file after open for write. Changing the test as follows makes it pass:

# Trigger data copy up and check absence of metacopy xattr. mount_overlay $_lowerdir
- $XFS_IO_PROG -c "open -a $SCRATCH_MNT/$_target"
+ $XFS_IO_PROG -c "falloc 0 $_size" $SCRATCH_MNT/$_target
echo "check properties of data copied up file" check_file_size_contents $SCRATCH_MNT/$_target $_size "$_data"

Thoughts?

Thanks,
Amir.

 fs/overlayfs/copy_up.c   |  5 +---
 fs/overlayfs/file.c      | 63 +++++++++++++++++++++++++++++++++-------
 fs/overlayfs/overlayfs.h | 10 ++++++-
 fs/overlayfs/util.c      |  4 +--
 4 files changed, 64 insertions(+), 18 deletions(-)
fs/overlayfs/copy_up.c +1 -4
index 48119b23375b..df495f14bce0 100644
@@ -731,10 +731,7 @@ static bool ovl_need_meta_copy_up(struct dentry *dentry, umode_t mode,
 	if (!S_ISREG(mode))
 		return false;
 
-	if (flags && ((OPEN_FMODE(flags) & FMODE_WRITE) || (flags & O_TRUNC)))
-		return false;
-
-	return true;
+	return !ovl_open_flags_need_data_copy_up(flags);
 }
 
 /* Copy up data of an inode which was copied up metadata only in the past. */
fs/overlayfs/file.c +52 -11

134 lines not on this page · raw message

fs/overlayfs/overlayfs.h +9 -1
index 5e45cb3630a0..d3a7d8936f21 100644
@@ -195,14 +195,22 @@ static inline struct dentry *ovl_do_tmpfile(struct dentry *dentry, umode_t mode)
 	return ret;
 }
 
-static inline bool ovl_open_flags_need_copy_up(int flags)
+static inline bool ovl_open_flags_need_data_copy_up(int flags)
 {
 	if (!flags)
 		return false;
 
+	/* Either O_RDWR or O_WRONLY will trigger data copy up */
 	return ((OPEN_FMODE(flags) & FMODE_WRITE) || (flags & O_TRUNC));
 }
 
+static inline bool ovl_open_flags_need_copy_up(int flags)
+{
+	/* O_RDWR|O_WRONLY together will trigger meta copy up */
+	return (flags & O_ACCMODE) == (O_RDWR | O_WRONLY) ||
+		ovl_open_flags_need_data_copy_up(flags);
+}
+
 /* util.c */
 int ovl_want_write(struct dentry *dentry);
 void ovl_drop_write(struct dentry *dentry);
fs/overlayfs/util.c +2 -2
index 7c01327b1852..df6d13185f40 100644
@@ -367,7 +367,7 @@ void ovl_set_upperdata(struct inode *inode)
 /* Caller should hold ovl_inode->lock */
 bool ovl_dentry_needs_data_copy_up_locked(struct dentry *dentry, int flags)
 {
-	if (!ovl_open_flags_need_copy_up(flags))
+	if (!ovl_open_flags_need_data_copy_up(flags))
 		return false;
 
 	return !ovl_test_flag(OVL_UPPERDATA, d_inode(dentry));
@@ -375,7 +375,7 @@ bool ovl_dentry_needs_data_copy_up_locked(struct dentry *dentry, int flags)
 
 bool ovl_dentry_needs_data_copy_up(struct dentry *dentry, int flags)
 {
-	if (!ovl_open_flags_need_copy_up(flags))
+	if (!ovl_open_flags_need_data_copy_up(flags))
 		return false;
 
 	return !ovl_has_upperdata(d_inode(dentry));
Signature · 1 line
2.17.1
#1 lore raw
Quoted 4 lines · Amir Goldstein

When metacopy feature is enabled, copy up only metadata when opening a file O_RDWR and defer copy up of data until the first write operation.

Amir,

What's the primary use case of lazy data copy up. Are there users who open file O_RDWR but never write to it. Or we want to transfer latency of copy up from open to write.

Should this behavior be an opt in with another mount option (and not be enabled automatically with metacopy=on).

Thanks
Vivek

Quoted 270 lines

On read operations of lazy copy up file, the lower file is opened O_RDONLY on every vfs call until the first write operation.

XXX: mmap() of a lazy opened file will map the meta inode, so this still needs to be fixed.

---

Hi Miklos,

Here's a draft for lazy copy up of data.
Sending it out for your initial feedback.

When opening file for write, we always cache a real file with upper inode, but that inode may be a meta copy.

This leaves ovl_mmap() doing the wrong thing (mapping the meta copy).

One way do deal with this would be to implement a_ops as you suggested and copy up data on page write fault.

Do you have another idea how to make use of this RFC patch without implementing a_aops? Or maybe there is no reason to bother?

Perhaps we could install ovl_a_aop only for files open for write as first stage? I think that would be easy enough for me to do. I still don't have the full picture in my head for dealing with LOWER->LOWER_SHARED transition.

This RFC patch BTW passes unionmount and the overlay/quick xfstests with metacopy=on/off.

The one test that does fail is the metacopy test overlay/060 because it fails the expectation to find a data copied up file after open for write. Changing the test as follows makes it pass:

# Trigger data copy up and check absence of metacopy xattr. mount_overlay $_lowerdir

(quoted diff) +1 -1
-       $XFS_IO_PROG -c "open -a $SCRATCH_MNT/$_target"
+       $XFS_IO_PROG -c "falloc 0 $_size" $SCRATCH_MNT/$_target
        echo "check properties of data copied up file"
        check_file_size_contents $SCRATCH_MNT/$_target $_size "$_data"

Thoughts?

Thanks,
Amir.

 fs/overlayfs/copy_up.c   |  5 +---
 fs/overlayfs/file.c      | 63 +++++++++++++++++++++++++++++++++-------
 fs/overlayfs/overlayfs.h | 10 ++++++-
 fs/overlayfs/util.c      |  4 +--
 4 files changed, 64 insertions(+), 18 deletions(-)
fs/overlayfs/copy_up.c +1 -4
index 48119b23375b..df495f14bce0 100644
@@ -731,10 +731,7 @@ static bool ovl_need_meta_copy_up(struct dentry *dentry, umode_t mode,
 	if (!S_ISREG(mode))
 		return false;
 
-	if (flags && ((OPEN_FMODE(flags) & FMODE_WRITE) || (flags & O_TRUNC)))
-		return false;
-
-	return true;
+	return !ovl_open_flags_need_data_copy_up(flags);
 }
 
 /* Copy up data of an inode which was copied up metadata only in the past. */
fs/overlayfs/file.c +52 -11
index 84dd957efa24..e15b16e386f4 100644
@@ -29,10 +29,15 @@ static struct file *ovl_open_realfile(const struct file *file,
 	struct inode *inode = file_inode(file);
 	struct file *realfile;
 	const struct cred *old_cred;
+	unsigned int flags = file->f_flags | O_NOATIME;
+
+	/* Open lower file O_RDONLY if O_RDWR file data wasn't copied up yet */
+	if (S_ISREG(inode->i_mode) && realinode != ovl_inode_upper(inode))
+		flags &= ~(O_RDWR | O_WRONLY);
 
 	old_cred = ovl_override_creds(inode->i_sb);
-	realfile = open_with_fake_path(&file->f_path, file->f_flags | O_NOATIME,
-				       realinode, current_cred());
+	realfile = open_with_fake_path(&file->f_path, flags, realinode,
+				       current_cred());
 	revert_creds(old_cred);
 
 	pr_debug("open(%p[%pD2/%c], 0%o) -> (%p, 0%o)\n",
@@ -80,21 +85,41 @@ static int ovl_change_flags(struct file *file, unsigned int flags)
 	return 0;
 }
 
+/*
+ * If file was opened for write with lazy copy of data, the first operation
+ * to call this function with @rdonly == false will trigger copy up of data.
+ * fdatasync() and fadvise(POSIX_FADV_DONTNEED) are "rdonly" ops in the sense
+ * that they do not dirty pages. If pages are already dirty then file data
+ * has already been copied up and those operations will get the upper file
+ * and write its inode's dirty pages.
+ */
 static int ovl_real_fdget_meta(const struct file *file, struct fd *real,
-			       bool allow_meta)
+			       bool allow_meta, bool rdonly)
 {
+	struct dentry *dentry = file_dentry(file);
 	struct inode *inode = file_inode(file);
 	struct inode *realinode;
+	int err;
 
 	real->flags = 0;
 	real->file = file->private_data;
 
+	if (!rdonly) {
+		/* Maybe copy up data on first write op */
+		err = ovl_open_maybe_copy_up(dentry, file->f_flags);
+		if (err)
+			return err;
+	}
+
 	if (allow_meta)
 		realinode = ovl_inode_real(inode);
 	else
 		realinode = ovl_inode_realdata(inode);
 
-	/* Has it been copied up since we'd opened it? */
+	/*
+	 * Has it been copied up since we'd opened it O_RDONLY?
+	 * or not yet data copied up since we'd opened it O_RDWR?
+	 */
 	if (unlikely(file_inode(real->file) != realinode)) {
 		real->flags = FDPUT_FPUT;
 		real->file = ovl_open_realfile(file, realinode);
@@ -111,23 +136,39 @@ static int ovl_real_fdget_meta(const struct file *file, struct fd *real,
 
 static int ovl_real_fdget(const struct file *file, struct fd *real)
 {
-	return ovl_real_fdget_meta(file, real, false);
+	return ovl_real_fdget_meta(file, real, false, false);
+}
+
+static int ovl_real_fdget_rdonly(const struct file *file, struct fd *real)
+{
+	return ovl_real_fdget_meta(file, real, false, true);
 }
 
 static int ovl_open(struct inode *inode, struct file *file)
 {
 	struct dentry *dentry = file_dentry(file);
 	struct file *realfile;
+	int copyup_flags = file->f_flags;
 	int err;
 
-	err = ovl_open_maybe_copy_up(dentry, file->f_flags);
+	/* (O_RDWR | O_WRONLY) signals that we want meta copy up */
+	if ((copyup_flags & O_ACCMODE) == O_RDWR)
+		copyup_flags |= O_WRONLY;
+
+	/* Allow to copy up meta and defer copy up data to first writable op */
+	err = ovl_open_maybe_copy_up(dentry, copyup_flags);
 	if (err)
 		return err;
 
 	/* No longer need these flags, so don't pass them on to underlying fs */
 	file->f_flags &= ~(O_CREAT | O_EXCL | O_NOCTTY | O_TRUNC);
 
-	realfile = ovl_open_realfile(file, ovl_inode_realdata(inode));
+	/*
+	 * For lazy copy up of data, we cache the upper meta file in
+	 * anticipation for copy up of data on first write. Opening O_RDWR
+	 * and only reading is not going to be efficient in that case.
+	 */
+	realfile = ovl_open_realfile(file, ovl_inode_real(inode));
 	if (IS_ERR(realfile))
 		return PTR_ERR(realfile);
 
@@ -201,7 +242,7 @@ static ssize_t ovl_read_iter(struct kiocb *iocb, struct iov_iter *iter)
 	if (!iov_iter_count(iter))
 		return 0;
 
-	ret = ovl_real_fdget(file, &real);
+	ret = ovl_real_fdget_rdonly(file, &real);
 	if (ret)
 		return ret;
 
@@ -263,7 +304,7 @@ static int ovl_fsync(struct file *file, loff_t start, loff_t end, int datasync)
 	const struct cred *old_cred;
 	int ret;
 
-	ret = ovl_real_fdget_meta(file, &real, !datasync);
+	ret = ovl_real_fdget_meta(file, &real, !datasync, true);
 	if (ret)
 		return ret;
 
@@ -339,7 +380,7 @@ static int ovl_fadvise(struct file *file, loff_t offset, loff_t len, int advice)
 	const struct cred *old_cred;
 	int ret;
 
-	ret = ovl_real_fdget(file, &real);
+	ret = ovl_real_fdget_rdonly(file, &real);
 	if (ret)
 		return ret;
 
@@ -447,7 +488,7 @@ static loff_t ovl_copyfile(struct file *file_in, loff_t pos_in,
 	if (ret)
 		return ret;
 
-	ret = ovl_real_fdget(file_in, &real_in);
+	ret = ovl_real_fdget_rdonly(file_in, &real_in);
 	if (ret) {
 		fdput(real_out);
 		return ret;
fs/overlayfs/overlayfs.h +9 -1
index 5e45cb3630a0..d3a7d8936f21 100644
@@ -195,14 +195,22 @@ static inline struct dentry *ovl_do_tmpfile(struct dentry *dentry, umode_t mode)
 	return ret;
 }
 
-static inline bool ovl_open_flags_need_copy_up(int flags)
+static inline bool ovl_open_flags_need_data_copy_up(int flags)
 {
 	if (!flags)
 		return false;
 
+	/* Either O_RDWR or O_WRONLY will trigger data copy up */
 	return ((OPEN_FMODE(flags) & FMODE_WRITE) || (flags & O_TRUNC));
 }
 
+static inline bool ovl_open_flags_need_copy_up(int flags)
+{
+	/* O_RDWR|O_WRONLY together will trigger meta copy up */
+	return (flags & O_ACCMODE) == (O_RDWR | O_WRONLY) ||
+		ovl_open_flags_need_data_copy_up(flags);
+}
+
 /* util.c */
 int ovl_want_write(struct dentry *dentry);
 void ovl_drop_write(struct dentry *dentry);
fs/overlayfs/util.c +2 -2
index 7c01327b1852..df6d13185f40 100644
@@ -367,7 +367,7 @@ void ovl_set_upperdata(struct inode *inode)
 /* Caller should hold ovl_inode->lock */
 bool ovl_dentry_needs_data_copy_up_locked(struct dentry *dentry, int flags)
 {
-	if (!ovl_open_flags_need_copy_up(flags))
+	if (!ovl_open_flags_need_data_copy_up(flags))
 		return false;
 
 	return !ovl_test_flag(OVL_UPPERDATA, d_inode(dentry));
@@ -375,7 +375,7 @@ bool ovl_dentry_needs_data_copy_up_locked(struct dentry *dentry, int flags)
 
 bool ovl_dentry_needs_data_copy_up(struct dentry *dentry, int flags)
 {
-	if (!ovl_open_flags_need_copy_up(flags))
+	if (!ovl_open_flags_need_data_copy_up(flags))
 		return false;
 
 	return !ovl_has_upperdata(d_inode(dentry));
Signature · 1 line
2.17.1
author maintainer
#2 lore raw
Quoted 11 lines · Vivek Goyal
Quoted 4 lines · Amir Goldstein

When metacopy feature is enabled, copy up only metadata when opening a file O_RDWR and defer copy up of data until the first write operation.

Amir,

What's the primary use case of lazy data copy up. Are there users who open file O_RDWR but never write to it.

I don't know of all the cases, but AFAIK, MSOffice apps (over network share and maybe also LibreOffice) open the document file O_RDWR just to check if editing is allowed, but but they re-open the file read-only, because the document file is never written to. A temp file is written and moved over it. So copy up of an MSOffice document file data is never beneficial.

Quoted 1 line

Or we want to transfer latency of copy up from open to write.

Indeed. Chengguang reported that in their use case, the latency at open time is an issue.

Quoted 4 lines

Should this behavior be an opt in with another mount option (and not be enabled automatically with metacopy=on).

I can't think of one good reason for user to opt-in for copy up data on open. Or on a use case where latency on open is desired over latency on write. Can you?

There is no contract with metacopy=on that states when a file is copied up as meta and when it is copied up as data.
If there was a contract, it would probably state that only data modification operations SHOULD copy up data and open is not a data modification operation (it was just the last opportunity to do anything before stacked file ops).

Thanks,
Amir.

#3 lore raw
Quoted 25 lines · Amir Goldstein
Quoted 11 lines · Vivek Goyal
Quoted 4 lines · Amir Goldstein

When metacopy feature is enabled, copy up only metadata when opening a file O_RDWR and defer copy up of data until the first write operation.

Amir,

What's the primary use case of lazy data copy up. Are there users who open file O_RDWR but never write to it.

I don't know of all the cases, but AFAIK, MSOffice apps (over network share and maybe also LibreOffice) open the document file O_RDWR just to check if editing is allowed, but but they re-open the file read-only, because the document file is never written to. A temp file is written and moved over it. So copy up of an MSOffice document file data is never beneficial.

Quoted 1 line

Or we want to transfer latency of copy up from open to write.

Indeed. Chengguang reported that in their use case, the latency at open time is an issue.

Quoted 1 line
Quoted 7 lines
Quoted 3 lines

Should this behavior be an opt in with another mount option (and not be enabled automatically with metacopy=on).

I can't think of one good reason for user to opt-in for copy up data on open. Or on a use case where latency on open is desired over latency on write. Can you?

I can't. I am wondering why are we changing default. Why is it a better default to do copy up on first WRITE and not during open time. In his emails, Chengguang, just said their applications will benefit from it without giving further details. So if their application is not writing, they could easily solve this problem by opening file O_RDONLY as well?

Or is it the case that they don't know in advance whether file will actually be written or not. So they open O_RDWR and after that it could be written or not written?

Thanks
Vivek

author maintainer
#4 lore raw
Quoted 31 lines · Vivek Goyal
Quoted 29 lines · Amir Goldstein
Quoted 11 lines · Vivek Goyal
Quoted 4 lines · Amir Goldstein

When metacopy feature is enabled, copy up only metadata when opening a file O_RDWR and defer copy up of data until the first write operation.

Amir,

What's the primary use case of lazy data copy up. Are there users who open file O_RDWR but never write to it.

I don't know of all the cases, but AFAIK, MSOffice apps (over network share and maybe also LibreOffice) open the document file O_RDWR just to check if editing is allowed, but but they re-open the file read-only, because the document file is never written to. A temp file is written and moved over it. So copy up of an MSOffice document file data is never beneficial.

Quoted 1 line

Or we want to transfer latency of copy up from open to write.

Indeed. Chengguang reported that in their use case, the latency at open time is an issue.

Quoted 4 lines

Should this behavior be an opt in with another mount option (and not be enabled automatically with metacopy=on).

Quoted 6 lines
Quoted 3 lines

I can't think of one good reason for user to opt-in for copy up data on open. Or on a use case where latency on open is desired over latency on write. Can you?

I can't. I am wondering why are we changing default. Why is it a better default to do copy up on first WRITE and not during open time. In his

Oh there are very good reasons to prefer latency on WRITE over latency on open.

One reason is that traditionally, open is not an operation where long latency are expected, so apps may behave badly with long latency on open.

Another reason, if application is latency sensitive, it can use aio, but there is no async open, or is there??

Quoted 4 lines

emails, Chengguang, just said their applications will benefit from it without giving further details. So if their application is not writing, they could easily solve this problem by opening file O_RDONLY as well?

My understanding is that latency on open() is an issue and latency on first write is not an issue, which is something I am familiar with from other apps, but I will let Chengguang elaborate.

Thanks,
Amir.

#5 lore raw
In reply to Amir Goldstein
Quoted 25 lines · Amir Goldstein
Quoted 11 lines · Vivek Goyal
Quoted 4 lines · Amir Goldstein

When metacopy feature is enabled, copy up only metadata when opening a file O_RDWR and defer copy up of data until the first write operation.

Amir,

What's the primary use case of lazy data copy up. Are there users who open file O_RDWR but never write to it.

I don't know of all the cases, but AFAIK, MSOffice apps (over network share and maybe also LibreOffice) open the document file O_RDWR just to check if editing is allowed, but but they re-open the file read-only, because the document file is never written to. A temp file is written and moved over it. So copy up of an MSOffice document file data is never beneficial.

Quoted 1 line

Or we want to transfer latency of copy up from open to write.

Indeed. Chengguang reported that in their use case, the latency at open time is an issue.

Quoted 1 line
Quoted 7 lines
Quoted 3 lines

Should this behavior be an opt in with another mount option (and not be enabled automatically with metacopy=on).

I can't think of one good reason for user to opt-in for copy up data on open. Or on a use case where latency on open is desired over latency on write. Can you?

IIUC, now if I open a file O_RDWR and only issue reads to file and never issue writes, then for every read, I will open a lower file, finish read and close fd. This is slower path. What's the performance penalty? If this is significant, then it might make sense to opt-in for lazy copy up behavior (instead of relying that application will open file O_RDONLY if they never issue writes to it).

Thanks
Vivek

author maintainer
#6 lore raw
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

Should this behavior be an opt in with another mount option (and not be enabled automatically with metacopy=on).

Quoted 7 lines
Quoted 3 lines

I can't think of one good reason for user to opt-in for copy up data on open. Or on a use case where latency on open is desired over latency on write. Can you?

IIUC, now if I open a file O_RDWR and only issue reads to file and never issue writes, then for every read, I will open a lower file, finish read and close fd. This is slower path. What's the performance penalty? If

That is correct, but as you rightfully noted, that is probably a corner case and not the common behavior.
It is not something that we cannot fix if we find that it is important. We can choose to cache the lower file and we can also choose to cache both upper and lower real files.

Quoted 4 lines

this is significant, then it might make sense to opt-in for lazy copy up behavior (instead of relying that application will open file O_RDONLY if they never issue writes to it).

Maybe. My instinct is that this case where complication the user with more configurations is not worth it and we can find a good enough solution for all cases.

Let's see what Miklos has to say.

Thanks,
Amir.