Skip to contentSkip to conversation

VFS changes for NFSv4.2 "inter" server-to-server COPY op

PATCH NAK Olga Kornievskaia 8 participants 22 messages Mar 2, 2017 – Sep 1, 2018 1 message not in the archive

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover VFS changes for NFSv4.2 "inter" server-to-server COPY op

One patch adds a check for validity of arguments.

In another patch is crucial to the existance of NFSv4.2 inter SSC COPY. In order to implement the "inter" SSC COPY that involves two files from different superblock, it requires removal of the cross-devices check in VFS. Given that in the future the "cp" command will be written to use copy_file_range(), it would be important for "inter" SSC COPY to use the same interface.

This patch has first been met with an objection from Christoph saying that "no other VFS operation is supported between mountpoints" and another objection that since been addressed that "no xfstest coverage". We argue that no other VFS operation needed such support and now there exists one.

An alternative approach might be to define a new file system function (eg., bool allow_cross_device_op() that a filesystem can define and NFS will define it and return true. And based on the existence of that function and its result the check could be relaxed)?

If that check were to be removed, it would be beneficial to check the the superblocks for the clone operation so that it's not called unnecessarily.

Anna Schumaker (1):
fs: Don't copy beyond the end of the file

Andy Adamson (1):
VFS permit cross device vfs_copy_file_range

Olga Kornievskaia (1):
VFS don't try clone if superblocks are different

 fs/read_write.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
Signature · 1 line
1.8.3.1
#1 lore raw

PATCH 1/3 fs: Don't copy beyond the end of the file

From: Anna Schumaker <[email protected]>

---
 fs/read_write.c | 3 +++
 1 file changed, 3 insertions(+)
fs/read_write.c +3 -0
index 5816d4c..1d9e305 100644
@@ -1526,6 +1526,9 @@ ssize_t vfs_copy_file_range(struct file *file_in, loff_t pos_in,
 	if (unlikely(ret))
 		return ret;
 
+	if (pos_in >= i_size_read(inode_in))
+		return -EINVAL;
+
 	if (!(file_in->f_mode & FMODE_READ) ||
 	    !(file_out->f_mode & FMODE_WRITE) ||
 	    (file_out->f_flags & O_APPEND))
Signature · 1 line
1.8.3.1
#2 lore raw

Re: [PATCH v1 1/3] fs: Don't copy beyond the end of the file

Quoted 12 lines · Olga Kornievskaia

From: Anna Schumaker <[email protected]>

---
 fs/read_write.c | 3 +++
 1 file changed, 3 insertions(+)
fs/read_write.c +0 -0
index 5816d4c..1d9e305 100644

fs/read_write.c ssize_t vfs_copy_file_range(struct file *file_in, loff_t pos_in, · line 1531

Quoted 6 lines
fs/read_write.c +3 -0
@@ -1526,6 +1526,9 @@ ssize_t vfs_copy_file_range(struct file *file_in, loff_t pos_in,
 	if (unlikely(ret))
 		return ret;
 
+	if (pos_in >= i_size_read(inode_in))
+		return -EINVAL;
+

This ought to go in vfs_clone_file_prep_inodes.

(He says, noticing that btrfs never got updated to use that validator...)

--D

Quoted 6 lines
fs/read_write.c +0 -0
 	if (!(file_in->f_mode & FMODE_READ) ||
 	    !(file_out->f_mode & FMODE_WRITE) ||
 	    (file_out->f_flags & O_APPEND))
Signature · 1 line
1.8.3.1
#3 lore raw

Re: [PATCH v1 1/3] fs: Don't copy beyond the end of the file

Quoted 19 lines · Darrick J. Wong
Quoted 15 lines · Olga Kornievskaia

From: Anna Schumaker <[email protected]>

---
fs/read_write.c | 3 +++

1 file changed, 3 insertions(+)
fs/read_write.c +0 -0
index 5816d4c..1d9e305 100644
@@ -1526,6 +1526,9 @@ ssize_t vfs_copy_file_range(struct file *file_in, loff_t pos_in,
 	if (unlikely(ret))
 		return ret;
 
Quoted 7 lines

fs/read_write.c ssize_t vfs_copy_file_range(struct file *file_in, loff_t pos_in, · line 1531

Quoted 3 lines
fs/read_write.c +3 -0
+	if (pos_in >= i_size_read(inode_in))
+		return -EINVAL;
+

This ought to go in vfs_clone_file_prep_inodes.

(He says, noticing that btrfs never got updated to use that validator…)

I apologize I’m not fully understanding the suggestion here. How btrfs is related to the check that I’m suggesting for the copy_file_range(). I don’t see how it would fix the problem for the copy_file_range().

Is the suggestion that NFS’s clone implementation is suppose to call vfs_clone_file_prep_inodes() where the check would be added and thus because vfs_copy_file_range() first decides to call clone() instead of copy() then that check would be met?

Quoted 9 lines

--D

Quoted 6 lines
fs/read_write.c +0 -0
 	if (!(file_in->f_mode & FMODE_READ) ||
 	    !(file_out->f_mode & FMODE_WRITE) ||
 	    (file_out->f_flags & O_APPEND))
Signature · 1 line
1.8.3.1
#4 lore raw

Re: [PATCH v1 1/3] fs: Don't copy beyond the end of the file

Quoted 25 lines · Olga Kornievskaia
Quoted 25 lines · Darrick J. Wong
Quoted 12 lines · Olga Kornievskaia

From: Anna Schumaker <[email protected]>

---
fs/read_write.c | 3 +++

1 file changed, 3 insertions(+)
fs/read_write.c +0 -0
index 5816d4c..1d9e305 100644

fs/read_write.c ssize_t vfs_copy_file_range(struct file *file_in, loff_t pos_in, · line 1531

Quoted 6 lines
fs/read_write.c +3 -0
@@ -1526,6 +1526,9 @@ ssize_t vfs_copy_file_range(struct file *file_in, loff_t pos_in,
 	if (unlikely(ret))
 		return ret;
 
+	if (pos_in >= i_size_read(inode_in))
+		return -EINVAL;
+

This ought to go in vfs_clone_file_prep_inodes.

Quoted 5 lines
Quoted 1 line

(He says, noticing that btrfs never got updated to use that validator…)

I apologize I’m not fully understanding the suggestion here. How btrfs is related to the check that I’m suggesting for the copy_file_range(). I don’t see how it would fix the problem for the copy_file_range().

Never mind, I misread vfs_copy as vfs_clone and thought you were talking about something else. :(

Sorry about the noise.

--D

Quoted 15 lines

Is the suggestion that NFS’s clone implementation is suppose to call vfs_clone_file_prep_inodes() where the check would be added and thus because vfs_copy_file_range() first decides to call clone() instead of copy() then that check would be met?

Quoted 9 lines

--D

Quoted 6 lines
fs/read_write.c +0 -0
 	if (!(file_in->f_mode & FMODE_READ) ||
 	    !(file_out->f_mode & FMODE_WRITE) ||
 	    (file_out->f_flags & O_APPEND))
Signature · 1 line
1.8.3.1
#5 lore raw

Re: [PATCH v1 1/3] fs: Don't copy beyond the end of the file

Quoted 4 lines · Darrick J. Wong

Never mind, I misread vfs_copy as vfs_clone and thought you were talking about something else. :(

Sorry about the noise.

And either way we'll need test coverage for copy_file_range before we do any more changes to it. I'm sick and tired of this doctoring around without having any test coverage. I've asked for it again and again and we've made very little progress despite having the code in the tree for almost a year and a half.

#6 lore raw

Re: [PATCH v1 1/3] fs: Don't copy beyond the end of the file

Quoted 7 lines · Christoph Hellwig
Quoted 3 lines · Darrick J. Wong

Never mind, I misread vfs_copy as vfs_clone and thought you were talking about something else. :(

Quoted 7 lines
Quoted 1 line

Sorry about the noise.

And either way we'll need test coverage for copy_file_range before we do any more changes to it. I'm sick and tired of this doctoring around without having any test coverage. I've asked for it again and again and we've made very little progress despite having the code in the tree for almost a year and a half.

Anna has added test cases to the xfstest suite. Is there something that’s missing?

#7 lore raw

Re: [PATCH v1 1/3] fs: Don't copy beyond the end of the file

Quoted 1 line · Olga Kornievskaia

Anna has added test cases to the xfstest suite. Is there something that’s missing?

No, we still don't have that. Anna did one attempt (or maybe two) but didn't seem to have time to follow up the review comments. Maybe you can give it a spin to make sure we at least have basic test coverage, including the cases mentioned in the man page like this.

#8 lore raw

Re: [PATCH v1 1/3] fs: Don't copy beyond the end of the file

Quoted 10 lines · Christoph Hellwig
Quoted 1 line · Olga Kornievskaia

Anna has added test cases to the xfstest suite. Is there something that’s missing?

No, we still don't have that. Anna did one attempt (or maybe two) but didn't seem to have time to follow up the review comments. Maybe you can give it a spin to make sure we at least have basic test coverage, including the cases mentioned in the man page like this.

I’m reading some mail archives and I see that Anna submitted patches for xfstests. The comment was to re-work it as a part of xfs_io command. She posted the patches for that. According to the announcement I see here: http://oss.sgi.com/archives/xfs/2016-08/msg00221.html it says it’s including the new copy_file_range command in xfs_io. I don’t run these tests so I don’t know but it look to me that the tests are in.

Can you please what I’m getting wrong?

Thank you.

#9 lore raw

Re: [PATCH v1 1/3] fs: Don't copy beyond the end of the file

Quoted 2 lines · Olga Kornievskaia

I’m reading some mail archives and I see that Anna submitted patches for xfstests. The comment was to re-work it as a part of xfs_io command. She posted the patches for that. According to the announcement I see here: http://oss.sgi.com/archives/xfs/2016-08/msg00221.html it says it’s including the new copy_file_range command in xfs_io. I don’t run these tests so I don’t know but it look to me that the tests are in.

That's xfsprogs and the xfs_io command. Now that this step is done someone (e.g. Anna or you) needs to send the patches to xfstests that use this command.

#10 lore raw

PATCH 2/3 VFS permit cross device vfs_copy_file_range

In reply to Olga Kornievskaia

Allow nfs_copy_file_range to copy across devices.
NFSv4.2 inter server to server copy always copies across devices, and NFSv4.2 intra server to server copy can copy across devices on the same server.

If a file system's fileoperations copy_file_range operation prohibits cross-device copies, fall back to do_splice_direct. This is needed for nfsd_copy_file_range() which is called by the inter server to server destination server acting as an NFS client, and reading the file from the source server.

---
 fs/read_write.c | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)
fs/read_write.c +1 -5
index 1d9e305..75084cd 100644
@@ -1534,10 +1534,6 @@ ssize_t vfs_copy_file_range(struct file *file_in, loff_t pos_in,
 	    (file_out->f_flags & O_APPEND))
 		return -EBADF;
 
-	/* this could be relaxed once a method supports cross-fs copies */
-	if (inode_in->i_sb != inode_out->i_sb)
-		return -EXDEV;
-
 	if (len == 0)
 		return 0;
 
@@ -1559,7 +1555,7 @@ ssize_t vfs_copy_file_range(struct file *file_in, loff_t pos_in,
 	if (file_out->f_op->copy_file_range) {
 		ret = file_out->f_op->copy_file_range(file_in, pos_in, file_out,
 						      pos_out, len, flags);
-		if (ret != -EOPNOTSUPP)
+		if (ret != -EOPNOTSUPP && ret != -EXDEV)
 			goto done;
 	}
 
Signature · 1 line
1.8.3.1
#11 lore raw

Re: [PATCH v1 2/3] VFS permit cross device vfs_copy_file_range

Quoted 10 lines · Olga Kornievskaia

Allow nfs_copy_file_range to copy across devices.
NFSv4.2 inter server to server copy always copies across devices, and NFSv4.2 intra server to server copy can copy across devices on the same server.

If a file system's fileoperations copy_file_range operation prohibits cross-device copies, fall back to do_splice_direct. This is needed for nfsd_copy_file_range() which is called by the inter server to server destination server acting as an NFS client, and reading the file from the source server.

NAK, we really should not do operations between different superblocks.

#12 lore raw

Re: [PATCH v1 2/3] VFS permit cross device vfs_copy_file_range

Quoted 9 lines · Christoph Hellwig
Quoted 5 lines · Olga Kornievskaia

Allow nfs_copy_file_range to copy across devices.
NFSv4.2 inter server to server copy always copies across devices, and NFSv4.2 intra server to server copy can copy across devices on the same server.

Quoted 7 lines
Quoted 5 lines

If a file system's fileoperations copy_file_range operation prohibits cross-device copies, fall back to do_splice_direct. This is needed for nfsd_copy_file_range() which is called by the inter server to server destination server acting as an NFS client, and reading the file from the source server.

NAK, we really should not do operations between different superblocks.

Can you provide some reasoning as to why? What would it break? The reasoning for including one is to allow for a file system to achieve better performance which seems like a feature that would be of great benefit.

#13 lore raw

Re: [PATCH v1 2/3] VFS permit cross device vfs_copy_file_range

Quoted 18 lines · Olga Kornievskaia
Quoted 16 lines · Christoph Hellwig
Quoted 10 lines · Olga Kornievskaia

Allow nfs_copy_file_range to copy across devices.
NFSv4.2 inter server to server copy always copies across devices, and NFSv4.2 intra server to server copy can copy across devices on the same server.

If a file system's fileoperations copy_file_range operation prohibits cross-device copies, fall back to do_splice_direct. This is needed for nfsd_copy_file_range() which is called by the inter server to server destination server acting as an NFS client, and reading the file from the source server.

Quoted 3 lines
Quoted 1 line

NAK, we really should not do operations between different superblocks.

Can you provide some reasoning as to why? What would it break? The reasoning for including one is to allow for a file system to achieve better performance which seems like a feature that would be of great benefit.

Christoph, could you please elaborate on your objection.

Al, could you weigh in with regards to if and what would it take to allow for copy_file_range() to allow for copies between different superblocks.

We would appreciate the feedback of how can we enable this performance feature to be useful to users.

Thank you.

#14 lore raw

Re: [PATCH v1 2/3] VFS permit cross device vfs_copy_file_range

In reply to Olga Kornievskaia
Quoted 8 lines · Olga Kornievskaia
Quoted 3 lines · Christoph Hellwig

NAK, we really should not do operations between different superblocks.

Can you provide some reasoning as to why? What would it break? The reasoning for including one is to allow for a file system to achieve better performance which seems like a feature that would be of great benefit. --

Yes, this has come up a few times. What's going on?:

- There was an explanation, and I missed it.
- The explanation is complicated and Christoph hasn't had time to write it up.
- Christoph has a strong suspicion there are issues without being sure exactly where they are.
- Something else?

I mean, the whole enterprise is dead in the water without this, as far as I can tell, so I'd really like to understand the answer one way or the other.

--b.

#15 lore raw

Re: [PATCH v1 2/3] VFS permit cross device vfs_copy_file_range

Quoted 10 lines · J. Bruce Fields
Quoted 8 lines · Olga Kornievskaia
Quoted 3 lines · Christoph Hellwig

NAK, we really should not do operations between different superblocks.

Can you provide some reasoning as to why? What would it break? The reasoning for including one is to allow for a file system to achieve better performance which seems like a feature that would be of great benefit. --

Quoted 8 lines

Yes, this has come up a few times. What's going on?:

- There was an explanation, and I missed it.
- The explanation is complicated and Christoph hasn't had time to write it up.
- Christoph has a strong suspicion there are issues without being sure exactly where they are.
- Something else?

As far as I can tell from talking at LSF:

- file system operations crossing superblocks is unusual. No specific known issue. Also not sure I understand why splice isn't precedent.
- implementation (with server acting as a client, long-running process, etc.) will be complicated and ugly. Sure, I guess we'll see how complicated and weigh that against any advantages. (Though I'm curious about case e.g. of btrfs clone--can't it easily clone across filesystem boundaries in some cases when they share storage?)

Anyway. I'll read the patches. Probably not this week.

--b.

A message that is not in the archive (sent off-list or not yet received): [email protected]

#17 lore raw

Re: [PATCH v1 2/3] VFS permit cross device vfs_copy_file_range

Quoted 4 lines · Olga Kornievskaia

Thank you for the update. I guess I don’t see how the proposed NFS implementation is complicated and ugly (but I’m biased). I’ll try to give you some performance number. My 1 data point (1gb) inter copy showed 30% improvement (how can that be ignored).

That would be useful, thanks--if it comes with some details about the setup.

I'm not so curious about percent improvement, as how to predict the performance on a given network.

If server-to-server copy looks like it's normally able to use close to the available bandwidth between the two servers, and if a traditional read-write-copy loop is similarly able to use the available bandwidth, then I can figure out whether server-to-server copy will help on my setup.

--b.

#18 lore raw

Re: [PATCH v1 2/3] VFS permit cross device vfs_copy_file_range

Quoted 8 lines · J. Bruce Fields
Quoted 4 lines · Olga Kornievskaia

Thank you for the update. I guess I don’t see how the proposed NFS implementation is complicated and ugly (but I’m biased). I’ll try to give you some performance number. My 1 data point (1gb) inter copy showed 30% improvement (how can that be ignored).

That would be useful, thanks--if it comes with some details about the setup.

What I have available to me are two laptops that I run my VMs on. It is not a setup that is representative of a real setup. I think this setup can only provide percent improvement numbers.

Andy was suggesting that perhaps the performance lab at Redhat would be able to do some testing of the patches for some real world performance?

Quoted 14 lines

I'm not so curious about percent improvement, as how to predict the performance on a given network.

If server-to-server copy looks like it's normally able to use close to the available bandwidth between the two servers, and if a traditional read-write-copy loop is similarly able to use the available bandwidth, then I can figure out whether server-to-server copy will help on my setup.

--b.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#19 lore raw

Re: [PATCH v1 2/3] VFS permit cross device vfs_copy_file_range

In reply to Olga Kornievskaia
Quoted 5 lines · Olga Kornievskaia

Allow nfs_copy_file_range to copy across devices.
NFSv4.2 inter server to server copy always copies across devices, and NFSv4.2 intra server to server copy can copy across devices on the same server.

Quoted 7 lines

If a file system's fileoperations copy_file_range operation prohibits cross-device copies, fall back to do_splice_direct. This is needed for nfsd_copy_file_range() which is called by the inter server to server destination server acting as an NFS client, and reading the file from the source server.

What has happened to the patch? We unwittingly used copy_file_range in the glibc test suite, without realizing that it does not support cross-device copies.

Thanks,
Florian

#20 lore raw

Re: [PATCH v1 2/3] VFS permit cross device vfs_copy_file_range

Quoted 13 lines · Florian Weimer
Quoted 11 lines · Olga Kornievskaia

Allow nfs_copy_file_range to copy across devices.
NFSv4.2 inter server to server copy always copies across devices, and NFSv4.2 intra server to server copy can copy across devices on the same server.

If a file system's fileoperations copy_file_range operation prohibits cross-device copies, fall back to do_splice_direct. This is needed for nfsd_copy_file_range() which is called by the inter server to server destination server acting as an NFS client, and reading the file from the source server.

Quoted 5 lines
Quoted 1 line

What has happened to the patch? We unwittingly used copy_file_range in the glibc test suite, without realizing that it does not support cross-device copies.

I'm still planning to fight for the patch to be included. As per maintainers request, I separated out the async copy patches out and hopefully that will be going into 4.20. I'm working on the "inter" copy offload functionality which does require this patch. I will start submitting and pushing this patch along with the rest of the patches.

Are you interested in the copy_file_range() functionality that does support cross-device copies? If so can you tell me how are you using it? It would be also helpful to watch for the submission of the patch and argue in its favor.

Thank you.

Quoted 3 lines

Thanks,
Florian

#21 lore raw

Re: [PATCH v1 2/3] VFS permit cross device vfs_copy_file_range

Quoted 27 lines · Olga Kornievskaia
Quoted 13 lines · Florian Weimer
Quoted 11 lines · Olga Kornievskaia

Allow nfs_copy_file_range to copy across devices.
NFSv4.2 inter server to server copy always copies across devices, and NFSv4.2 intra server to server copy can copy across devices on the same server.

If a file system's fileoperations copy_file_range operation prohibits cross-device copies, fall back to do_splice_direct. This is needed for nfsd_copy_file_range() which is called by the inter server to server destination server acting as an NFS client, and reading the file from the source server.

Quoted 5 lines
Quoted 1 line

What has happened to the patch? We unwittingly used copy_file_range in the glibc test suite, without realizing that it does not support cross-device copies.

I'm still planning to fight for the patch to be included. As per maintainers request, I separated out the async copy patches out and hopefully that will be going into 4.20. I'm working on the "inter" copy offload functionality which does require this patch. I will start submitting and pushing this patch along with the rest of the patches.

Quoted 4 lines

Are you interested in the copy_file_range() functionality that does support cross-device copies? If so can you tell me how are you using it? It would be also helpful to watch for the submission of the patch and argue in its favor.

SMB3 clients and server (Windows, Macs, Samba and probably most every modern NAS out there) support the SMB3 "CopyChunk" operation used by the Linux client.

I would expect that one of the most useful cases is cross device For example you have two mounts to your server or server cluster \\server\share1 is mounted to /mnt1
and
\\server\backup is mounted to /mnt2

and you want to do copy_file or copy_file_range of various files from /mnt1 to /mnt2

As long as they are both the same file system type, why not let the file system tell you whether it can do the copy.

Given that this is 10x to 100x faster than the alternative and this is a common case (and there are many 100s of millions of SMB3 server capable systems out there which already support copychunk (and thus would support copy file) - why would we want to restrict it.

It is much saner to let the file system tell the VFS if it can't support the cross device copy.

---- and to make it even more obvious why this patch matters --- Virtualization clients (and various Windows and NetApp server) support another copy offload mechanism (not just CopyChunk) ie via ODX which would support cross server not just cross share
copy. Enabling this would be a BIG incentive for finishing up
ODX copy support in Linux (cifs.ko using SMB3). This is
fairly widely supported by servers.

Signature · 3 lines
Thanks,

Steve
#22 lore raw

PATCH 3/3 VFS don't try clone if superblocks are different

In reply to Olga Kornievskaia

Only try clone_file_range if superblocks are the same.

---
 fs/read_write.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
fs/read_write.c +2 -1
index 75084cd..00161bd 100644
@@ -1543,7 +1543,8 @@ ssize_t vfs_copy_file_range(struct file *file_in, loff_t pos_in,
 	 * Try cloning first, this is supported by more file systems, and
 	 * more efficient if both clone and copy are supported (e.g. NFS).
 	 */
-	if (file_in->f_op->clone_file_range) {
+	if (inode_in->i_sb == inode_out->i_sb &&
+			file_in->f_op->clone_file_range) {
 		ret = file_in->f_op->clone_file_range(file_in, pos_in,
 				file_out, pos_out, len);
 		if (ret == 0) {
Signature · 1 line
1.8.3.1