Skip to contentSkip to conversation

sys_copy_range() rough draft

RFC In review Zach Brown 6 participants 16 messages May 14, 2013 – May 21, 2013

Subsystems (7)
x86 VFS Btrfs file system NFS client +3 more subsystems
lore
Share & tools
lore

Zach Brown

author
#0 lore raw

cover sys_copy_range() rough draft

We've been talking about implementing some form of bulk data copy offloading for a while now. BTRFS and OCFS2 implement forms of copy offloading with ioctls, NFS 4.2 will include a byte-granular COPY operation, and the SCSI XCOPY command is being implemented now that Windows can issue it.

In the past we've discussed promoting the ocfs2 reflink ioctl into a system call that would create a new file and implicitly copy the source data into the new file:
https://lkml.org/lkml/2009/9/14/481

These draft patches take the simpler approach of only copying data between existing files. The patches 1) make a system call out of the btrfs CLONE_RANGE ioctl, 2) implement the btrfs .copy_range method with the ioctl's guts, 3) implement the nfs .copy_range by sending a COPY op, and 4) serve the COPY op in nfsd by calling the .copy_range method again.

The nfs patch is an untested hack. I'm happy to beat it in to shape but I'll need some guidance.

I'd like strong review feedback on the interfaces, here are some possible topics:

a) Hopefully being able to specify a portion of the data to copy will avoid *huge* syscall latencies and the motivation for new async semantics.

b) The BTRFS ioctl and nfs COPY let you specify a count of 0 to copy from the start offset to the end of the file. Does anyone have a strong feeling about this? I'm leaning towards not bothering with it in the syscall interface.

c) I chose to return partial progess in the ssize_t return code. This limits the length of the range and the size_t count argument can be too large and return errors, much like other io syscalls. This seemed less awful than some extra argument with a pointer to a status value.

d) I'm dreading mentioning a vector of ranges to copy in one syscall because I don't want to think about overlaping ranges and file systems that use range locks -- xfs for now, but more if Jan gets his way. I'd rather that we get some experience with this simpler syscall before taking on that headache.

I'm sure I'm forgetting some other details.

I'm going to keep hacking away at this. My next step is to get ext4 supporting .copy_range, probably with a quick hack to copy the contents of bios. Hopefully that'll give enough time to also integrate review feedback.

Thoughts?

- z

author
#1 lore raw

RFC 1/4 vfs: add copy_range syscall and vfs entry point

This adds a syscall and vfs entry point for clone_range which offloads data copying between existing files.

The syscall is a thin wrapper around the vfs entry point. Its arguments are inspired by sys_splice().

The behaviour of the vfs helper is derived from the current btrfs CLONE_RANGE ioctl.

---
 fs/Makefile                       |   2 +-
 fs/copy_range.c                   | 127 ++++++++++++++++++++++++++++++++++++++
 include/linux/fs.h                |   3 +
 include/uapi/asm-generic/unistd.h |   4 +-
 kernel/sys_ni.c                   |   1 +
 5 files changed, 135 insertions(+), 2 deletions(-)
 create mode 100644 fs/copy_range.c
fs/Makefile +1 -1
index 4fe6df3..1be83b3 100644
@@ -11,7 +11,7 @@ obj-y :=	open.o read_write.o file_table.o super.o \
 		attr.o bad_inode.o file.o filesystems.o namespace.o \
 		seq_file.o xattr.o libfs.o fs-writeback.o \
 		pnode.o splice.o sync.o utimes.o \
-		stack.o fs_struct.o statfs.o
+		stack.o fs_struct.o statfs.o copy_range.o
 
 ifeq ($(CONFIG_BLOCK),y)
 obj-y +=	buffer.o bio.o block_dev.o direct-io.o mpage.o ioprio.o
fs/copy_range.c new file +127 -0

127 lines not on this page · raw message

include/linux/fs.h +3 -0
index 43db02e..6214893 100644
@@ -1543,6 +1543,7 @@ struct file_operations {
 	long (*fallocate)(struct file *file, int mode, loff_t offset,
 			  loff_t len);
 	int (*show_fdinfo)(struct seq_file *m, struct file *f);
+	ssize_t (*copy_range)(struct file *, loff_t, struct file *, loff_t, size_t);
 };
 
 struct inode_operations {
@@ -1588,6 +1589,8 @@ extern ssize_t vfs_readv(struct file *, const struct iovec __user *,
 		unsigned long, loff_t *);
 extern ssize_t vfs_writev(struct file *, const struct iovec __user *,
 		unsigned long, loff_t *);
+extern ssize_t vfs_copy_range(struct file *, loff_t , struct file *, loff_t,
+		size_t);
 
 struct super_operations {
    	struct inode *(*alloc_inode)(struct super_block *sb);
include/uapi/asm-generic/unistd.h +3 -1
index 0cc74c4..3935d1c 100644
@@ -692,9 +692,11 @@ __SC_COMP(__NR_process_vm_writev, sys_process_vm_writev, \
 __SYSCALL(__NR_kcmp, sys_kcmp)
 #define __NR_finit_module 273
 __SYSCALL(__NR_finit_module, sys_finit_module)
+#define __NR_copy_range 274
+__SYSCALL(__NR_copy_range, sys_copy_range)
 
 #undef __NR_syscalls
-#define __NR_syscalls 274
+#define __NR_syscalls 275
 
 /*
  * All syscalls below here should go away really,
kernel/sys_ni.c +1 -0
index 7078052..af7808a 100644
@@ -151,6 +151,7 @@ cond_syscall(sys_process_vm_readv);
 cond_syscall(sys_process_vm_writev);
 cond_syscall(compat_sys_process_vm_readv);
 cond_syscall(compat_sys_process_vm_writev);
+cond_syscall(sys_copy_range);
 
 /* arch-specific weak syscall entries */
 cond_syscall(sys_pciconfig_read);
Signature · 1 line
1.7.11.7
#2 lore raw

Re: [RFC v0 1/4] vfs: add copy_range syscall and vfs entry point

Quoted 5 lines · Zach Brown

This adds a syscall and vfs entry point for clone_range which offloads data copying between existing files.

The syscall is a thin wrapper around the vfs entry point. Its arguments are inspired by sys_splice().

Why introduce a new syscall instead of extending sys_splice?

Perhaps adding a new SPLICE_F_COPYRANGE flag to avoid compatibility problems with userspace which may expect ESPIPE when given two non-pipes would be useful.

If the user doesn't need a out offset, then sendfile() should also be able to transparently utilize COPY/CLONE_RANGE, too.

author
#3 lore raw

Re: [RFC v0 1/4] vfs: add copy_range syscall and vfs entry point

Quoted 1 line · Eric Wong

Why introduce a new syscall instead of extending sys_splice?

Personally, I think it's ugly to have different operations use the same syscall just because their arguments match.

But that preference aside, sure, if the consensus is that we'd rather use the splice() entry point then I can duck tape the pieces together to make it work.

Quoted 2 lines

If the user doesn't need a out offset, then sendfile() should also be able to transparently utilize COPY/CLONE_RANGE, too.

Perhaps, yeah.

- z

#4 lore raw

Re: [RFC v0 1/4] vfs: add copy_range syscall and vfs entry point

Quoted 4 lines · Zach Brown
Quoted 1 line · Eric Wong

Why introduce a new syscall instead of extending sys_splice?

Personally, I think it's ugly to have different operations use the same syscall just because their arguments match.

I agree with Zach - having a system call called "splice" do copy offloads is not intuitive.

This is a very reasonable name for something that battled its way through several standards bodies (for NFS and SCSI :)), so we should give it a reasonable name

thanks!

Ric

Quoted 10 lines

But that preference aside, sure, if the consensus is that we'd rather use the splice() entry point then I can duck tape the pieces together to make it work.

Quoted 2 lines

If the user doesn't need a out offset, then sendfile() should also be able to transparently utilize COPY/CLONE_RANGE, too.

Perhaps, yeah.

- z

#5 lore raw

Re: [RFC v0 1/4] vfs: add copy_range syscall and vfs entry point

In reply to Zach Brown
Quoted 5 lines · Zach Brown
Quoted 1 line · Eric Wong

Why introduce a new syscall instead of extending sys_splice?

Personally, I think it's ugly to have different operations use the same syscall just because their arguments match.

Fair enough. I think adding a (currently unused) flags parameter would make sense for future-proofing.

If this were extended to sockets/pipes, peek/nonblock/waitall flags may be added.

Basically, something like splice/tee, but without needing to create/manage a pipe in userspace or needing extra syscalls/looping.

author
#6 lore raw

Re: [RFC v0 1/4] vfs: add copy_range syscall and vfs entry point

Quoted 9 lines · Eric Wong
Quoted 5 lines · Zach Brown
Quoted 1 line · Eric Wong

Why introduce a new syscall instead of extending sys_splice?

Personally, I think it's ugly to have different operations use the same syscall just because their arguments match.

Fair enough. I think adding a (currently unused) flags parameter would make sense for future-proofing.

Yeah, that seems reasonble.

- z

author
#7 lore raw

RFC 2/4 x86: add sys_copy_range to syscall tables

In reply to Zach Brown

Add sys_copy_range to the x86 syscall tables. Happily, it doesn't require compat helpers.

---
 arch/x86/syscalls/syscall_32.tbl | 1 +
 arch/x86/syscalls/syscall_64.tbl | 1 +
 2 files changed, 2 insertions(+)
arch/x86/syscalls/syscall_32.tbl +1 -0
index aabfb83..d75d1b5 100644
@@ -357,3 +357,4 @@
 348	i386	process_vm_writev	sys_process_vm_writev		compat_sys_process_vm_writev
 349	i386	kcmp			sys_kcmp
 350	i386	finit_module		sys_finit_module
+351	i386	copy_range		sys_copy_range
arch/x86/syscalls/syscall_64.tbl +1 -0
index 38ae65d..dc9d766 100644
@@ -320,6 +320,7 @@
 311	64	process_vm_writev	sys_process_vm_writev
 312	common	kcmp			sys_kcmp
 313	common	finit_module		sys_finit_module
+314	common	copy_range		sys_copy_range
 
 #
 # x32-specific system call numbers start at 512 to avoid cache impact
Signature · 1 line
1.7.11.7
author
#8 lore raw

RFC 3/4 btrfs: add .copy_range file operation

In reply to Zach Brown

This rearranges the existing COPY_RANGE ioctl implementation so that the .copy_range file operation can call the core loop that copies file data extent items.

The extent copying loop is lifted up into its own function. It retains the core btrfs error checks that should be shared between the CLONE_RANGE ioctl and copy_range syscall.

---
 fs/btrfs/ctree.h |   3 ++
 fs/btrfs/file.c  |   1 +
 fs/btrfs/ioctl.c | 122 +++++++++++++++++++++++++++++++++----------------------
 3 files changed, 77 insertions(+), 49 deletions(-)
fs/btrfs/ctree.h +3 -0
index 63c328a..bf9555c 100644
@@ -3612,6 +3612,9 @@ int btrfs_dirty_pages(struct btrfs_root *root, struct inode *inode,
 		      struct page **pages, size_t num_pages,
 		      loff_t pos, size_t write_bytes,
 		      struct extent_state **cached);
+ssize_t btrfs_copy_range(struct file *file_in, loff_t pos_in,
+			 struct file *file_out, loff_t pos_out,
+			 size_t count);
 
 /* tree-defrag.c */
 int btrfs_defrag_leaves(struct btrfs_trans_handle *trans,
fs/btrfs/file.c +1 -0
index 4205ba7..d75cc07 100644
@@ -2460,6 +2460,7 @@ const struct file_operations btrfs_file_operations = {
 #ifdef CONFIG_COMPAT
 	.compat_ioctl	= btrfs_ioctl,
 #endif
+	.copy_range	= btrfs_copy_range,
 };
 
 void btrfs_auto_defrag_exit(void)
fs/btrfs/ioctl.c +73 -49

167 lines not on this page · raw message

Signature · 1 line
1.7.11.7
author
#9 lore raw

RFC 4/4 nfs, nfsd: rough sys_copy_range and COPY support

In reply to Zach Brown

This crude patch illustrates the simplest plumbing involved in supporting sys_call_range with the NFS COPY operation that's pending in the 4.2 draft spec.

The patch is based on a previous prototype that used the COPY op to implement sys_copyfileat which created a new file (based on the ocfs2 reflink ioctl). By contrast, this copies file contents between existing files.

There's still a lot of implementation and testing to do, but this can get discussion going.

---
 fs/nfs/file.c           |  25 +++++++++
 fs/nfs/nfs4proc.c       |  72 ++++++++++++++++++++++++++
 fs/nfs/nfs4xdr.c        | 132 ++++++++++++++++++++++++++++++++++++++++++++++++
 fs/nfsd/nfs4proc.c      |  35 +++++++++++++
 fs/nfsd/nfs4xdr.c       |  43 ++++++++++++++++
 fs/nfsd/vfs.c           |  41 +++++++++++++++
 fs/nfsd/vfs.h           |   3 ++
 fs/nfsd/xdr4.h          |  21 ++++++++
 include/linux/nfs4.h    |   6 ++-
 include/linux/nfs_xdr.h |  24 +++++++++
 10 files changed, 401 insertions(+), 1 deletion(-)
fs/nfs/file.c +25 -0
index a87a44f..7d7bedf 100644
@@ -917,6 +917,30 @@ int nfs_setlease(struct file *file, long arg, struct file_lock **fl)
 }
 EXPORT_SYMBOL_GPL(nfs_setlease);
 
+ssize_t nfs_copy_range(struct file *file_in, loff_t pos_in,
+		       struct file *file_out, loff_t pos_out,
+		       size_t count)
+{
+	struct dentry *dentry_in = file_in->f_path.dentry;
+	struct dentry *dentry_out = file_out->f_path.dentry;
+	struct inode *inode_in = dentry_in->d_inode;
+	struct inode *inode_out = dentry_out->d_inode;
+	loff_t ret;
+
+	dprintk("NFS copy_range(%s/%s@%llu, %s/%s@%llu, %zd)\n",
+		dentry_in->d_parent->d_name.name, dentry_in->d_name.name,
+		(unsigned long long)pos_in,
+		dentry_out->d_parent->d_name.name, dentry_out->d_name.name,
+		(unsigned long long)pos_out, count);
+
+	if (NFS_PROTO(inode_in)->copy == NULL)
+		ret = -EOPNOTSUPP;
+	else
+		ret = NFS_PROTO(inode_in)->copy(inode_in, inode_out, NULL,
+						0, count, pos_in, pos_out);
+	return ret;
+}
+
 const struct file_operations nfs_file_operations = {
 	.llseek		= nfs_file_llseek,
 	.read		= do_sync_read,
@@ -934,5 +958,6 @@ const struct file_operations nfs_file_operations = {
 	.splice_write	= nfs_file_splice_write,
 	.check_flags	= nfs_check_flags,
 	.setlease	= nfs_setlease,
+	.copy_range	= nfs_copy_range,
 };
 EXPORT_SYMBOL_GPL(nfs_file_operations);
fs/nfs/nfs4proc.c +72 -0

84 lines not on this page · raw message

fs/nfs/nfs4xdr.c +132 -0

174 lines not on this page · raw message

fs/nfsd/nfs4proc.c +35 -0
index 27d74a2..2f62ebb 100644
@@ -986,6 +986,37 @@ nfsd4_write(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	return status;
 }
 
+/*
+ * XXX:
+ *  - do something with stateids :)
+ *  - implement callback results and OFFLOAD_ABORT
+ *  - inter-server copies?
+ */
+static __be32
+nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
+		struct nfsd4_copy *copy)
+{
+	__be32 status;
+
+	/* only support copying data to an existing file */
+	if (copy->ca_destinationlen)
+		return nfserr_inval;
+
+	if (!cstate->current_fh.fh_dentry || !cstate->save_fh.fh_dentry)
+		return nfserr_nofilehandle;
+
+	status = nfsd_copy_range(rqstp, &cstate->save_fh, copy->ca_src_offset,
+				 &cstate->current_fh, copy->ca_dst_offset,
+				 copy->ca_count);
+	if (status == nfs_ok)
+		copy->u.cr_bytes_copied = copy->ca_count;
+
+	/* don't support async callbacks yet */
+	copy->u.ok.cr_callback_id_length = 0;
+
+	return status;
+}
+
 /* This routine never returns NFS_OK!  If there are no other errors, it
  * will return NFSERR_SAME or NFSERR_NOT_SAME depending on whether the
  * attributes matched.  VERIFY is implemented by mapping NFSERR_SAME
@@ -1798,6 +1829,10 @@ static struct nfsd4_operation nfsd4_ops[] = {
 		.op_get_currentstateid = (stateid_getter)nfsd4_get_freestateid,
 		.op_rsize_bop = (nfsd4op_rsize)nfsd4_only_status_rsize,
 	},
+	[OP_COPY] = {
+		.op_func = (nfsd4op_func)nfsd4_copy,
+		.op_name = "OP_COPY",
+	},
 };
 
 #ifdef NFSD_DEBUG
fs/nfsd/nfs4xdr.c +43 -0

67 lines not on this page · raw message

fs/nfsd/vfs.c +41 -0

53 lines not on this page · raw message

fs/nfsd/vfs.h +3 -0
index 5b58941..bbc9483 100644
@@ -85,6 +85,9 @@ __be32		nfsd_symlink(struct svc_rqst *, struct svc_fh *,
 				struct svc_fh *res, struct iattr *);
 __be32		nfsd_link(struct svc_rqst *, struct svc_fh *,
 				char *, int, struct svc_fh *);
+__be32		nfsd_copy_range(struct svc_rqst *,
+				struct svc_fh *, u64,
+				struct svc_fh *, u64, u64);
 __be32		nfsd_rename(struct svc_rqst *,
 				struct svc_fh *, char *, int,
 				struct svc_fh *, char *, int);
fs/nfsd/xdr4.h +21 -0
index 3b271d2..95fd1c3 100644
@@ -426,6 +426,26 @@ struct nfsd4_reclaim_complete {
 	u32 rca_one_fs;
 };
 
+struct nfsd4_copy {
+	/* request */
+	u64             ca_src_offset;
+	u64             ca_dst_offset;
+	u64             ca_count;
+	u32             ca_flags;
+	u32             ca_destinationlen;
+	char *          ca_destination;
+
+	/* response */
+	union {
+		struct {
+			u32			cr_callback_id_length;
+			stateid_t *             cr_callback_id;
+		} ok;
+		u64	cr_bytes_copied;
+	} u;
+
+};
+
 struct nfsd4_op {
 	int					opnum;
 	__be32					status;
@@ -471,6 +491,7 @@ struct nfsd4_op {
 		struct nfsd4_reclaim_complete	reclaim_complete;
 		struct nfsd4_test_stateid	test_stateid;
 		struct nfsd4_free_stateid	free_stateid;
+		struct nfsd4_copy		copy;
 	} u;
 	struct nfs4_replay *			replay;
 };
include/linux/nfs4.h +5 -1
index 7b8fc73..6be484e 100644
@@ -100,6 +100,7 @@ enum nfs_opnum4 {
 	OP_WANT_DELEGATION = 56,
 	OP_DESTROY_CLIENTID = 57,
 	OP_RECLAIM_COMPLETE = 58,
+	OP_COPY = 59,
 
 	OP_ILLEGAL = 10044,
 };
@@ -108,7 +109,7 @@ enum nfs_opnum4 {
 Needs to be updated if more operations are defined in future.*/
 
 #define FIRST_NFS4_OP	OP_ACCESS
-#define LAST_NFS4_OP 	OP_RECLAIM_COMPLETE
+#define LAST_NFS4_OP	OP_COPY
 
 enum nfsstat4 {
 	NFS4_OK = 0,
@@ -456,6 +457,9 @@ enum {
 	NFSPROC4_CLNT_GETDEVICELIST,
 	NFSPROC4_CLNT_BIND_CONN_TO_SESSION,
 	NFSPROC4_CLNT_DESTROY_CLIENTID,
+
+	/* nfs42 */
+	NFSPROC4_CLNT_COPY,
 };
 
 /* nfs41 types */
include/linux/nfs_xdr.h +24 -0
index 104b62f..2256e31 100644
@@ -1184,6 +1184,28 @@ struct nfs41_free_stateid_res {
 	unsigned int			status;
 };
 
+struct nfs_copy_args {
+	struct nfs_fh			*fh;
+	struct nfs_fh			*dir_fh;
+	u32				*bitmask;
+	__u64				src_offset;
+	__u64				dst_offset;
+	__u64				count;
+	__u32				flags;
+	const struct qstr		*destination;
+	struct nfs4_sequence_args	seq_args;
+};
+
+struct nfs_copy_res {
+	struct nfs_fh			*fh;
+	struct nfs_fattr		*fattr;
+	__u32				callback_id_length;
+	nfs4_stateid			*callback_id;
+	__u64				bytes_copied;
+	const struct nfs_server		*server;
+	struct nfs4_sequence_res	seq_res;
+};
+
 #else
 
 struct pnfs_ds_commit_info {
@@ -1433,6 +1455,8 @@ struct nfs_rpc_ops {
 	struct nfs_server *(*create_server)(struct nfs_mount_info *, struct nfs_subversion *);
 	struct nfs_server *(*clone_server)(struct nfs_server *, struct nfs_fh *,
 					   struct nfs_fattr *, rpc_authflavor_t);
+	loff_t  (*copy)    (struct inode *, struct inode *, struct qstr *,
+				int, loff_t, loff_t, loff_t);
 };
 
 /*
Signature · 6 lines
1.7.11.7

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#10 lore raw

Re: [RFC v0 4/4] nfs, nfsd: rough sys_copy_range and COPY support

Quoted 11 lines · Zach Brown

This crude patch illustrates the simplest plumbing involved in supporting sys_call_range with the NFS COPY operation that's pending in the 4.2 draft spec.

The patch is based on a previous prototype that used the COPY op to implement sys_copyfileat which created a new file (based on the ocfs2 reflink ioctl). By contrast, this copies file contents between existing files.

There's still a lot of implementation and testing to do, but this can get discussion going.

I'm using:

	git://github.com/loghyr/NFSv4.2

as my reference for the draft protocol.

On a quick skim, one thing this is missing before it complies is a client implementation of CB_OFFLOAD: "If a client desires an intra-server file copy, then it MUST support the COPY and CB_OFFLOAD operations."

The server doesn't have to implement CB_OFFLOAD, though, so we should ditch these todo's:

Quoted 6 lines

+/*
+ * XXX:

+ *  - do something with stateids :)
+ *  - implement callback results and OFFLOAD_ABORT
+ *  - inter-server copies?

+ */

...

fs/nfsd/nfs4proc.c nfsd4_write(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, · line 1014

Quoted 1 line
fs/nfsd/nfs4proc.c +1 -0
+	/* don't support async callbacks yet */

...

lest someone go try to implement them for no reason. (Stranger things have happened.)

Nits, possibly to ignore for now:

Quoted 9 lines
fs/nfsd/nfs4proc.c +5 -0
+	copy->u.ok.cr_callback_id_length = 0;
+
+	return status;
+}
+
 /* This routine never returns NFS_OK!  If there are no other errors, it
  * will return NFSERR_SAME or NFSERR_NOT_SAME depending on whether the
  * attributes matched.  VERIFY is implemented by mapping NFSERR_SAME

fs/nfsd/nfs4proc.c static struct nfsd4_operation nfsd4_ops[] = { · line 1835

Quoted 7 lines
fs/nfsd/nfs4proc.c +4 -0
@@ -1798,6 +1829,10 @@ static struct nfsd4_operation nfsd4_ops[] = {
 		.op_get_currentstateid = (stateid_getter)nfsd4_get_freestateid,
 		.op_rsize_bop = (nfsd4op_rsize)nfsd4_only_status_rsize,
 	},
+	[OP_COPY] = {
+		.op_func = (nfsd4op_func)nfsd4_copy,
+		.op_name = "OP_COPY",
+	},

There's some more boilerplate to fill in (see other ops).

fs/nfsd/nfs4xdr.c static nfsd4_dec nfsd41_dec_ops[] = { · line 1580

Quoted 9 lines
fs/nfsd/nfs4xdr.c +2 -0
+static __be32
 nfsd4_decode_noop(struct nfsd4_compoundargs *argp, void *p)
 {
 	return nfs_ok;
@@ -1557,6 +1577,7 @@ static nfsd4_dec nfsd41_dec_ops[] = {
 	[OP_WANT_DELEGATION]	= (nfsd4_dec)nfsd4_decode_notsupp,
 	[OP_DESTROY_CLIENTID]	= (nfsd4_dec)nfsd4_decode_destroy_clientid,
 	[OP_RECLAIM_COMPLETE]	= (nfsd4_dec)nfsd4_decode_reclaim_complete,
+	[OP_COPY]               = (nfsd4_dec)nfsd4_decode_copy,

And this should be made 4.2-specific.

Quoted 64 lines
fs/nfsd/nfs4xdr.c +22 -0
 };
 
 struct nfsd4_minorversion_ops {
@@ -3394,6 +3415,27 @@ nfsd4_encode_test_stateid(struct nfsd4_compoundres *resp, __be32 nfserr,
 }
 
 static __be32
+nfsd4_encode_copy(struct nfsd4_compoundres *resp, __be32 nfserr,
+			struct nfsd4_copy *copy)
+{
+	__be32 *p;
+
+	if (!nfserr) {
+		RESERVE_SPACE(4);
+		WRITE32(copy->u.ok.cr_callback_id_length);
+		ADJUST_ARGS();
+		if (copy->u.ok.cr_callback_id_length == 1)
+			nfsd4_encode_stateid(resp, copy->u.ok.cr_callback_id);
+	} else {
+		RESERVE_SPACE(8);
+		WRITE64(copy->u.cr_bytes_copied);
+		ADJUST_ARGS();
+	}
+
+	return nfserr;
+}
+
+static __be32
 nfsd4_encode_noop(struct nfsd4_compoundres *resp, __be32 nfserr, void *p)
 {
 	return nfserr;
@@ -3465,6 +3507,7 @@ static nfsd4_enc nfsd4_enc_ops[] = {
 	[OP_WANT_DELEGATION]	= (nfsd4_enc)nfsd4_encode_noop,
 	[OP_DESTROY_CLIENTID]	= (nfsd4_enc)nfsd4_encode_noop,
 	[OP_RECLAIM_COMPLETE]	= (nfsd4_enc)nfsd4_encode_noop,
+	[OP_COPY]		= (nfsd4_enc)nfsd4_encode_copy,
 };
 
 /*
fs/nfsd/vfs.c +10 -0
index 84ce601..0c1b427 100644
@@ -28,6 +28,8 @@
 #include <asm/uaccess.h>
 #include <linux/exportfs.h>
 #include <linux/writeback.h>
+#include <linux/fs_struct.h>
+#include <linux/kmod.h>
 
 #ifdef CONFIG_NFSD_V3
 #include "xdr3.h"
@@ -621,6 +623,45 @@ int nfsd4_is_junction(struct dentry *dentry)
 		return 0;
 	return 1;
 }
+
+__be32
+nfsd_copy_range(struct svc_rqst *rqstp, struct svc_fh *fhp_in, u64 pos_in,
+		struct svc_fh *fhp_out, u64 pos_out, u64 count)
+{
+	struct file *filp_in = NULL;
+	struct file *filp_out = NULL;
+	int err;

fs/nfsd/vfs.c int nfsd4_is_junction(struct dentry *dentry) · line 643

Quoted 10 lines
fs/nfsd/vfs.c +10 -0
+
+	/* XXX verify pos and count within sane limits? */
+
+	err = nfsd_open(rqstp, fhp_in, S_IFREG, NFSD_MAY_READ, &filp_in);
+	if (err)
+		goto out;
+
+	err = nfsd_open(rqstp, fhp_out, S_IFREG, NFSD_MAY_WRITE, &filp_out);
+	if (err)
+		goto out;

Looking at the xdr... the COPY operation takes stateid's, which nfsd can use to look up files, so the opens shouldn't be required.

--b.

Quoted 157 lines
fs/nfsd/vfs.c +21 -0
+
+	err = vfs_copy_range(filp_in, pos_in, filp_out, pos_out, count);
+	/* fall back if .copy_range isn't supported */
+
+	if (!err && EX_ISSYNC(fhp_out->fh_export))
+		err = vfs_fsync_range(filp_out, pos_out, pos_out + count-1, 0);
+
+out:
+	if (filp_in)
+		nfsd_close(filp_in);
+	if (filp_out)
+		nfsd_close(filp_out);
+
+	if (err < 0)
+		err = nfserrno(err);
+	else
+		err = 0;
+
+	return err;
+}
+
 #endif /* defined(CONFIG_NFSD_V4) */
 
 #ifdef CONFIG_NFSD_V3
fs/nfsd/vfs.h +3 -0
index 5b58941..bbc9483 100644
@@ -85,6 +85,9 @@ __be32		nfsd_symlink(struct svc_rqst *, struct svc_fh *,
 				struct svc_fh *res, struct iattr *);
 __be32		nfsd_link(struct svc_rqst *, struct svc_fh *,
 				char *, int, struct svc_fh *);
+__be32		nfsd_copy_range(struct svc_rqst *,
+				struct svc_fh *, u64,
+				struct svc_fh *, u64, u64);
 __be32		nfsd_rename(struct svc_rqst *,
 				struct svc_fh *, char *, int,
 				struct svc_fh *, char *, int);
fs/nfsd/xdr4.h +21 -0
index 3b271d2..95fd1c3 100644
@@ -426,6 +426,26 @@ struct nfsd4_reclaim_complete {
 	u32 rca_one_fs;
 };
 
+struct nfsd4_copy {
+	/* request */
+	u64             ca_src_offset;
+	u64             ca_dst_offset;
+	u64             ca_count;
+	u32             ca_flags;
+	u32             ca_destinationlen;
+	char *          ca_destination;
+
+	/* response */
+	union {
+		struct {
+			u32			cr_callback_id_length;
+			stateid_t *             cr_callback_id;
+		} ok;
+		u64	cr_bytes_copied;
+	} u;
+
+};
+
 struct nfsd4_op {
 	int					opnum;
 	__be32					status;
@@ -471,6 +491,7 @@ struct nfsd4_op {
 		struct nfsd4_reclaim_complete	reclaim_complete;
 		struct nfsd4_test_stateid	test_stateid;
 		struct nfsd4_free_stateid	free_stateid;
+		struct nfsd4_copy		copy;
 	} u;
 	struct nfs4_replay *			replay;
 };
include/linux/nfs4.h +5 -1
index 7b8fc73..6be484e 100644
@@ -100,6 +100,7 @@ enum nfs_opnum4 {
 	OP_WANT_DELEGATION = 56,
 	OP_DESTROY_CLIENTID = 57,
 	OP_RECLAIM_COMPLETE = 58,
+	OP_COPY = 59,
 
 	OP_ILLEGAL = 10044,
 };
@@ -108,7 +109,7 @@ enum nfs_opnum4 {
 Needs to be updated if more operations are defined in future.*/
 
 #define FIRST_NFS4_OP	OP_ACCESS
-#define LAST_NFS4_OP 	OP_RECLAIM_COMPLETE
+#define LAST_NFS4_OP	OP_COPY
 
 enum nfsstat4 {
 	NFS4_OK = 0,
@@ -456,6 +457,9 @@ enum {
 	NFSPROC4_CLNT_GETDEVICELIST,
 	NFSPROC4_CLNT_BIND_CONN_TO_SESSION,
 	NFSPROC4_CLNT_DESTROY_CLIENTID,
+
+	/* nfs42 */
+	NFSPROC4_CLNT_COPY,
 };
 
 /* nfs41 types */
include/linux/nfs_xdr.h +24 -0
index 104b62f..2256e31 100644
@@ -1184,6 +1184,28 @@ struct nfs41_free_stateid_res {
 	unsigned int			status;
 };
 
+struct nfs_copy_args {
+	struct nfs_fh			*fh;
+	struct nfs_fh			*dir_fh;
+	u32				*bitmask;
+	__u64				src_offset;
+	__u64				dst_offset;
+	__u64				count;
+	__u32				flags;
+	const struct qstr		*destination;
+	struct nfs4_sequence_args	seq_args;
+};
+
+struct nfs_copy_res {
+	struct nfs_fh			*fh;
+	struct nfs_fattr		*fattr;
+	__u32				callback_id_length;
+	nfs4_stateid			*callback_id;
+	__u64				bytes_copied;
+	const struct nfs_server		*server;
+	struct nfs4_sequence_res	seq_res;
+};
+
 #else
 
 struct pnfs_ds_commit_info {
@@ -1433,6 +1455,8 @@ struct nfs_rpc_ops {
 	struct nfs_server *(*create_server)(struct nfs_mount_info *, struct nfs_subversion *);
 	struct nfs_server *(*clone_server)(struct nfs_server *, struct nfs_fh *,
 					   struct nfs_fattr *, rpc_authflavor_t);
+	loff_t  (*copy)    (struct inode *, struct inode *, struct qstr *,
+				int, loff_t, loff_t, loff_t);
 };
 
 /*
Signature · 6 lines
1.7.11.7

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#11 lore raw

Re: [RFC v0 4/4] nfs, nfsd: rough sys_copy_range and COPY support

Quoted 15 lines · J. Bruce Fields
Quoted 11 lines · Zach Brown

This crude patch illustrates the simplest plumbing involved in supporting sys_call_range with the NFS COPY operation that's pending in the 4.2 draft spec.

The patch is based on a previous prototype that used the COPY op to implement sys_copyfileat which created a new file (based on the ocfs2 reflink ioctl). By contrast, this copies file contents between existing files.

There's still a lot of implementation and testing to do, but this can get discussion going.

I'm using:

Quoted 8 lines
	git://github.com/loghyr/NFSv4.2

as my reference for the draft protocol.

On a quick skim, one thing this is missing before it complies is a client implementation of CB_OFFLOAD: "If a client desires an intra-server file copy, then it MUST support the COPY and CB_OFFLOAD operations."

Note that Bryan is currently working on updating the NFS implementation to match the draft protocol.

Signature · 10 lines
Trond Myklebust
Linux NFS client maintainer

NetApp
[email protected]
www.netapp.com
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#12 lore raw

Re: [RFC v0 4/4] nfs, nfsd: rough sys_copy_range and COPY support

Quoted 20 lines · Trond
Quoted 19 lines · J. Bruce Fields
Quoted 11 lines · Zach Brown

This crude patch illustrates the simplest plumbing involved in supporting sys_call_range with the NFS COPY operation that's pending in the 4.2 draft spec.

The patch is based on a previous prototype that used the COPY op to implement sys_copyfileat which created a new file (based on the ocfs2 reflink ioctl). By contrast, this copies file contents between existing files.

There's still a lot of implementation and testing to do, but this can get discussion going.

I'm using:

	git://github.com/loghyr/NFSv4.2

as my reference for the draft protocol.

Quoted 7 lines
Quoted 4 lines

On a quick skim, one thing this is missing before it complies is a client implementation of CB_OFFLOAD: "If a client desires an intra-server file copy, then it MUST support the COPY and CB_OFFLOAD operations."

Note that Bryan is currently working on updating the NFS implementation to match the draft protocol.

OK, good.--b.

#13 lore raw
In reply to Zach Brown
Quoted 33 lines · Zach Brown

We've been talking about implementing some form of bulk data copy offloading for a while now. BTRFS and OCFS2 implement forms of copy offloading with ioctls, NFS 4.2 will include a byte-granular COPY operation, and the SCSI XCOPY command is being implemented now that Windows can issue it.

In the past we've discussed promoting the ocfs2 reflink ioctl into a system call that would create a new file and implicitly copy the source data into the new file:
https://lkml.org/lkml/2009/9/14/481

These draft patches take the simpler approach of only copying data between existing files. The patches 1) make a system call out of the btrfs CLONE_RANGE ioctl, 2) implement the btrfs .copy_range method with the ioctl's guts, 3) implement the nfs .copy_range by sending a COPY op, and 4) serve the COPY op in nfsd by calling the .copy_range method again.

The nfs patch is an untested hack. I'm happy to beat it in to shape but I'll need some guidance.

I'd like strong review feedback on the interfaces, here are some possible topics:

a) Hopefully being able to specify a portion of the data to copy will avoid *huge* syscall latencies and the motivation for new async semantics.

b) The BTRFS ioctl and nfs COPY let you specify a count of 0 to copy from the start offset to the end of the file. Does anyone have a strong feeling about this? I'm leaning towards not bothering with it in the syscall interface.

Quoted 8 lines

c) I chose to return partial progess in the ssize_t return code. This limits the length of the range and the size_t count argument can be too large and return errors, much like other io syscalls. This seemed less awful than some extra argument with a pointer to a status value.

d) I'm dreading mentioning a vector of ranges to copy in one syscall because I don't want to think about overlaping ranges and file systems that use range locks -- xfs for now, but more if Jan gets his way.

XFS doesn't use range locks (yet).

Quoted 9 lines

I'd rather that we get some experience with this simpler syscall before taking on that headache.

I'm sure I'm forgetting some other details.

I'm going to keep hacking away at this. My next step is to get ext4 supporting .copy_range, probably with a quick hack to copy the contents of bios. Hopefully that'll give enough time to also integrate review feedback.

Wouldn't the easiest "support all filesystems" hack just be to add a destination offset parameter to do_splice_direct() and call that when the filesystem doesn't supply a ->copy_range method? i.e. use the mechanisms we already have for copying from one file to another via the page cache as efficiently as possible?

Cheers,

Dave.

Signature · 6 lines
Dave Chinner
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
author
#14 lore raw
Quoted 11 lines · Dave Chinner
Quoted 4 lines · Zach Brown

I'm going to keep hacking away at this. My next step is to get ext4 supporting .copy_range, probably with a quick hack to copy the contents of bios. Hopefully that'll give enough time to also integrate review feedback.

Wouldn't the easiest "support all filesystems" hack just be to add a destination offset parameter to do_splice_direct() and call that when the filesystem doesn't supply a ->copy_range method? i.e. use the mechanisms we already have for copying from one file to another via the page cache as efficiently as possible?

Probably; and this in-kernel buffered fallback is particularly desirable for nfsd when the exported fs doesn't provide .copy_range. Having nfsd service the COPY op is still a significant win over having the client move the data backand forth over the wire.

But in that quote above I was talking about implementing .copy_range in ext4 as though it could use XCOPY today. I'd like to get a feel for how bad it's going to be to juggle the bio XCOPY IO with unwritten extent conversion, RMW with overlapping existing blocks, i_size advancing, etc. (It's so much like O_DIRECT that I'm already crying a little.)

- z

#15 lore raw
Quoted 13 lines · Zach Brown
Quoted 11 lines · Dave Chinner
Quoted 4 lines · Zach Brown

I'm going to keep hacking away at this. My next step is to get ext4 supporting .copy_range, probably with a quick hack to copy the contents of bios. Hopefully that'll give enough time to also integrate review feedback.

Wouldn't the easiest "support all filesystems" hack just be to add a destination offset parameter to do_splice_direct() and call that when the filesystem doesn't supply a ->copy_range method? i.e. use the mechanisms we already have for copying from one file to another via the page cache as efficiently as possible?

Quoted 4 lines

Probably; and this in-kernel buffered fallback is particularly desirable for nfsd when the exported fs doesn't provide .copy_range. Having nfsd service the COPY op is still a significant win over having the client move the data backand forth over the wire.

Sure. That's kind of what I was thinking to make it easy to test and have widespread support up front.

Quoted 5 lines

But in that quote above I was talking about implementing .copy_range in ext4 as though it could use XCOPY today. I'd like to get a feel for how bad it's going to be to juggle the bio XCOPY IO with unwritten extent conversion, RMW with overlapping existing blocks, i_size advancing, etc. (It's so much like O_DIRECT that I'm already crying a little.)

Toss anything that is hard back to the page cache path. Overlapping blocks, partial blocks and so can be handled by the slow path without making the offload path complex.

Make the offload do the simple stuff fast - the mapping and completion callbacks should be no different to the direct IO bits we have now, and if you only handle filesystem block aligned ranges in the offload (rather than sector alignment) most of the grot that DIO code has to handle goes away....

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]