copy offload support in Linux - new system call needed?

We had an active thread a couple of years back that came out of the reflink work and, at the time, there seemed to be moderately positive support for adding a new system call that would fit this use case (Joel Becker's copyfile()).

Can we resurrect this effort? Is copyfile() still a good way to go, or should we look at other hooks?

Quoted 7 lines · Ric Wheeler

We had an active thread a couple of years back that came out of the reflink work and, at the time, there seemed to be moderately positive support for adding a new system call that would fit this use case (Joel Becker's copyfile()).

Can we resurrect this effort? Is copyfile() still a good way to go, or should we look at other hooks?

copyfile(2) is probably a good way to go, provided that we do _not_ go baroque as it had happened the last time syscall had been discussed.

IOW, to hell with progress reports, etc. - just a fastpath kind of thing, in the same kind of relationship to cp(1) as rename(2) is to mv(1). If it works - fine, if not - caller has to be ready to deal with handling cross-device case anyway.

Back at LinuxCon Prague, we talked about the new NFS and SCSI commands that let us offload copy operations to a storage device (like an NFS server or storage array).

This got new life in the virtual machine world where you might want to clone bulky guest files or ranges of blocks and was driven through the standards bodies by vmware, microsoft and some of the major storage vendors. Windows8 has this functionality fully coded and integrated in the GUI, I assume vmware also uses it and there are some vendors who announced support at the SNIA SDC conference.

We had an active thread a couple of years back that came out of the reflink work and, at the time, there seemed to be moderately positive support for adding a new system call that would fit this use case (Joel Becker's copyfile()).

Can we resurrect this effort? Is copyfile() still a good way to go, or should we look at other hooks?

Quoted 11 lines · Ric Wheeler

Back at LinuxCon Prague, we talked about the new NFS and SCSI commands that let us offload copy operations to a storage device (like an NFS server or storage array).

This got new life in the virtual machine world where you might want to clone bulky guest files or ranges of blocks and was driven through the standards bodies by vmware, microsoft and some of the major storage vendors. Windows8 has this functionality fully coded and integrated in the GUI, I assume vmware also uses it and there are some vendors who announced support at the SNIA SDC conference.

Quoted 7 lines

We had an active thread a couple of years back that came out of the reflink work and, at the time, there seemed to be moderately positive support for adding a new system call that would fit this use case (Joel Becker's copyfile()).

Can we resurrect this effort? Is copyfile() still a good way to go, or should we look at other hooks?

Windows uses a COPYCHUNK call, which specifies the following parameters:

Definition of a copy "chunk":

hyper source_off;
hyper target_off;
uint32 length;

and an array of these chunks which is passed
into their kernel.

This is what we have to implement in Samba.

Jeremy.

Quoted 12 lines · Al Viro

Quoted 7 lines · Ric Wheeler

We had an active thread a couple of years back that came out of the reflink work and, at the time, there seemed to be moderately positive support for adding a new system call that would fit this use case (Joel Becker's copyfile()).

Can we resurrect this effort? Is copyfile() still a good way to go, or should we look at other hooks?

copyfile(2) is probably a good way to go, provided that we do _not_ go baroque as it had happened the last time syscall had been discussed.

Quoted 4 lines

IOW, to hell with progress reports, etc. - just a fastpath kind of thing, in the same kind of relationship to cp(1) as rename(2) is to mv(1). If it works - fine, if not - caller has to be ready to deal with handling cross-device case anyway.

I think that this approach makes a lot of sense. Most of the devices/targets that support the copy offload, will do it in very reasonable amounts of time.

Let me see if I can dig up some of the presentations from the NetApp guys who presented overviews or the specifications from the IETF and T10....

Ric

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 13 lines · Ric Wheeler

Quoted 12 lines · Al Viro

Quoted 7 lines · Ric Wheeler

We had an active thread a couple of years back that came out of the reflink work and, at the time, there seemed to be moderately positive support for adding a new system call that would fit this use case (Joel Becker's copyfile()).

Can we resurrect this effort? Is copyfile() still a good way to go, or should we look at other hooks?

copyfile(2) is probably a good way to go, provided that we do _not_ go baroque as it had happened the last time syscall had been discussed.

Quoted 8 lines

Quoted 4 lines

IOW, to hell with progress reports, etc. - just a fastpath kind of thing, in the same kind of relationship to cp(1) as rename(2) is to mv(1). If it works - fine, if not - caller has to be ready to deal with handling cross-device case anyway.

I think that this approach makes a lot of sense. Most of the devices/targets that support the copy offload, will do it in very reasonable amounts of time.

The current NFSv4.2 draft rolls both the "fast" and "slow" cases into one operation:

	http://tools.ietf.org/html/draft-ietf-nfsv4-minorversion2-06#section-2

Perhaps we should ask for separate operations for the two cases. (Or at least a "please don't bother if this is going to take 8 hours" flag....)

Quoted 28 lines · J. Bruce Fields

Quoted 13 lines · Ric Wheeler

Quoted 12 lines · Al Viro

Quoted 7 lines · Ric Wheeler

We had an active thread a couple of years back that came out of the reflink work and, at the time, there seemed to be moderately positive support for adding a new system call that would fit this use case (Joel Becker's copyfile()).

Can we resurrect this effort? Is copyfile() still a good way to go, or should we look at other hooks?

copyfile(2) is probably a good way to go, provided that we do _not_ go baroque as it had happened the last time syscall had been discussed.

Quoted 8 lines

Quoted 4 lines

IOW, to hell with progress reports, etc. - just a fastpath kind of thing, in the same kind of relationship to cp(1) as rename(2) is to mv(1). If it works - fine, if not - caller has to be ready to deal with handling cross-device case anyway.

I think that this approach makes a lot of sense. Most of the devices/targets that support the copy offload, will do it in very reasonable amounts of time.

The current NFSv4.2 draft rolls both the "fast" and "slow" cases into one operation:

	http://tools.ietf.org/html/draft-ietf-nfsv4-minorversion2-06#section-2

Quoted 2 lines

Perhaps we should ask for separate operations for the two cases. (Or at least a "please don't bother if this is going to take 8 hours" flag....)

How would the server know? I suggest we deal with this by adding an ioctl() to allow the application to poll for progress: I'm assuming now that we don't expect more than 1 copyfile() system call at a time per file descriptor...

Quoted 32 lines · Trond Myklebust

Quoted 30 lines · J. Bruce Fields

Quoted 13 lines · Ric Wheeler

Quoted 12 lines · Al Viro

Quoted 7 lines · Ric Wheeler

We had an active thread a couple of years back that came out of the reflink work and, at the time, there seemed to be moderately positive support for adding a new system call that would fit this use case (Joel Becker's copyfile()).

Can we resurrect this effort? Is copyfile() still a good way to go, or should we look at other hooks?

copyfile(2) is probably a good way to go, provided that we do _not_ go baroque as it had happened the last time syscall had been discussed.

Quoted 8 lines

Quoted 4 lines

IOW, to hell with progress reports, etc. - just a fastpath kind of thing, in the same kind of relationship to cp(1) as rename(2) is to mv(1). If it works - fine, if not - caller has to be ready to deal with handling cross-device case anyway.

I think that this approach makes a lot of sense. Most of the devices/targets that support the copy offload, will do it in very reasonable amounts of time.

The current NFSv4.2 draft rolls both the "fast" and "slow" cases into one operation:

	http://tools.ietf.org/html/draft-ietf-nfsv4-minorversion2-06#section-2

Perhaps we should ask for separate operations for the two cases. (Or at least a "please don't bother if this is going to take 8 hours" flag....)

Quoted 4 lines

How would the server know? I suggest we deal with this by adding an ioctl() to allow the application to poll for progress: I'm assuming now that we don't expect more than 1 copyfile() system call at a time per file descriptor...

If we're using this to copy VM image files, I could easily imagine wanting to clone multiple copies of the VM in parallel.

Quoted 29 lines · Trond Myklebust

Quoted 28 lines · J. Bruce Fields

Quoted 13 lines · Ric Wheeler

Quoted 12 lines · Al Viro

Quoted 7 lines · Ric Wheeler

We had an active thread a couple of years back that came out of the reflink work and, at the time, there seemed to be moderately positive support for adding a new system call that would fit this use case (Joel Becker's copyfile()).

Can we resurrect this effort? Is copyfile() still a good way to go, or should we look at other hooks?

copyfile(2) is probably a good way to go, provided that we do _not_ go baroque as it had happened the last time syscall had been discussed.

Quoted 8 lines

Quoted 4 lines

IOW, to hell with progress reports, etc. - just a fastpath kind of thing, in the same kind of relationship to cp(1) as rename(2) is to mv(1). If it works - fine, if not - caller has to be ready to deal with handling cross-device case anyway.

I think that this approach makes a lot of sense. Most of the devices/targets that support the copy offload, will do it in very reasonable amounts of time.

The current NFSv4.2 draft rolls both the "fast" and "slow" cases into one operation:

	http://tools.ietf.org/html/draft-ietf-nfsv4-minorversion2-06#section-2

Quoted 7 lines

Quoted 2 lines

Perhaps we should ask for separate operations for the two cases. (Or at least a "please don't bother if this is going to take 8 hours" flag....)

How would the server know? I suggest we deal with this by adding an ioctl() to allow the application to poll for progress: I'm assuming now that we don't expect more than 1 copyfile() system call at a time per file descriptor...

If we're using this to copy VM image files, I could easily imagine wanting to clone multiple copies of the VM in parallel.

-chris

Quoted 41 lines · On Thu, 15 Dec 2011 10:52:13 -0500 Chris Mason…

Quoted 29 lines · Trond Myklebust

Quoted 28 lines · J. Bruce Fields

Quoted 13 lines · Ric Wheeler

Quoted 12 lines · Al Viro

Quoted 7 lines · Ric Wheeler

We had an active thread a couple of years back that came out of the reflink work and, at the time, there seemed to be moderately positive support for adding a new system call that would fit this use case (Joel Becker's copyfile()).

Can we resurrect this effort? Is copyfile() still a good way to go, or should we look at other hooks?

copyfile(2) is probably a good way to go, provided that we do _not_ go baroque as it had happened the last time syscall had been discussed.

Quoted 8 lines

Quoted 4 lines

IOW, to hell with progress reports, etc. - just a fastpath kind of thing, in the same kind of relationship to cp(1) as rename(2) is to mv(1). If it works - fine, if not - caller has to be ready to deal with handling cross-device case anyway.

I think that this approach makes a lot of sense. Most of the devices/targets that support the copy offload, will do it in very reasonable amounts of time.

The current NFSv4.2 draft rolls both the "fast" and "slow" cases into one operation:

	http://tools.ietf.org/html/draft-ietf-nfsv4-minorversion2-06#section-2

Quoted 7 lines

Quoted 2 lines

Perhaps we should ask for separate operations for the two cases. (Or at least a "please don't bother if this is going to take 8 hours" flag....)

How would the server know? I suggest we deal with this by adding an ioctl() to allow the application to poll for progress: I'm assuming now that we don't expect more than 1 copyfile() system call at a time per file descriptor...

If we're using this to copy VM image files, I could easily imagine wanting to clone multiple copies of the VM in parallel.

Quoted 2 lines

-chris

Not really a problem is it? Just dup() the fd before you issue the copyfile()? Or even simpler, just do periodic stat() on the destination file if you want a progress report.

Regardless, I like the simple approach that Al is suggesting here.

How would the server know? I suggest we deal with this by adding an ioctl() to allow the application to poll for progress: I'm assuming now

Quoted 2 lines

How would the server know? I suggest we deal with this by adding an ioctl() to allow the application to poll for progress: I'm assuming now

Why not support something like the async-iocb?

Quoted 51 lines · Jeff Layton

Quoted 38 lines · On Thu, 15 Dec 2011 10:52:13 -0500 Chris Mason…

Quoted 29 lines · Trond Myklebust

Quoted 28 lines · J. Bruce Fields

Quoted 13 lines · Ric Wheeler

Quoted 12 lines · Al Viro

Quoted 7 lines · Ric Wheeler

We had an active thread a couple of years back that came out of the reflink work and, at the time, there seemed to be moderately positive support for adding a new system call that would fit this use case (Joel Becker's copyfile()).

Can we resurrect this effort? Is copyfile() still a good way to go, or should we look at other hooks?

copyfile(2) is probably a good way to go, provided that we do _not_ go baroque as it had happened the last time syscall had been discussed.

Quoted 8 lines

Quoted 4 lines

IOW, to hell with progress reports, etc. - just a fastpath kind of thing, in the same kind of relationship to cp(1) as rename(2) is to mv(1). If it works - fine, if not - caller has to be ready to deal with handling cross-device case anyway.

I think that this approach makes a lot of sense. Most of the devices/targets that support the copy offload, will do it in very reasonable amounts of time.

The current NFSv4.2 draft rolls both the "fast" and "slow" cases into one operation:

	http://tools.ietf.org/html/draft-ietf-nfsv4-minorversion2-06#section-2

Quoted 7 lines

Quoted 2 lines

Perhaps we should ask for separate operations for the two cases. (Or at least a "please don't bother if this is going to take 8 hours" flag....)

How would the server know? I suggest we deal with this by adding an ioctl() to allow the application to poll for progress: I'm assuming now that we don't expect more than 1 copyfile() system call at a time per file descriptor...

Quoted 5 lines

If we're using this to copy VM image files, I could easily imagine wanting to clone multiple copies of the VM in parallel.

-chris

Not really a problem is it? Just dup() the fd before you issue the copyfile()? Or even simpler, just do periodic stat() on the destination file if you want a progress report.

Quoted 1 line

Regardless, I like the simple approach that Al is suggesting here.

Periodic stat() isn't good enough if you are copying subranges of a file. Part of the application here (as I understood it) is to initialise specific disk volumes on existing VM images when doing thin provisioning. In that case, the reported image size won't ever change...

Quoted 52 lines · On Thu, 15 Dec 2011 11:06:16 -0500 Trond Myklebust…

Quoted 51 lines · Jeff Layton

Quoted 38 lines · On Thu, 15 Dec 2011 10:52:13 -0500 Chris Mason <[email protected]> wrote:

Quoted 29 lines · Trond Myklebust

Quoted 28 lines · J. Bruce Fields

Quoted 13 lines · Ric Wheeler

Quoted 12 lines · Al Viro

Quoted 7 lines · Ric Wheeler

We had an active thread a couple of years back that came out of the reflink work and, at the time, there seemed to be moderately positive support for adding a new system call that would fit this use case (Joel Becker's copyfile()).

Can we resurrect this effort? Is copyfile() still a good way to go, or should we look at other hooks?

copyfile(2) is probably a good way to go, provided that we do _not_ go baroque as it had happened the last time syscall had been discussed.

Quoted 8 lines

Quoted 4 lines

IOW, to hell with progress reports, etc. - just a fastpath kind of thing, in the same kind of relationship to cp(1) as rename(2) is to mv(1). If it works - fine, if not - caller has to be ready to deal with handling cross-device case anyway.

I think that this approach makes a lot of sense. Most of the devices/targets that support the copy offload, will do it in very reasonable amounts of time.

The current NFSv4.2 draft rolls both the "fast" and "slow" cases into one operation:

	http://tools.ietf.org/html/draft-ietf-nfsv4-minorversion2-06#section-2

Quoted 7 lines

Quoted 2 lines

Perhaps we should ask for separate operations for the two cases. (Or at least a "please don't bother if this is going to take 8 hours" flag....)

How would the server know? I suggest we deal with this by adding an ioctl() to allow the application to poll for progress: I'm assuming now that we don't expect more than 1 copyfile() system call at a time per file descriptor...

Quoted 5 lines

If we're using this to copy VM image files, I could easily imagine wanting to clone multiple copies of the VM in parallel.

-chris

Not really a problem is it? Just dup() the fd before you issue the copyfile()? Or even simpler, just do periodic stat() on the destination file if you want a progress report.

Quoted 7 lines

Quoted 1 line

Regardless, I like the simple approach that Al is suggesting here.

Periodic stat() isn't good enough if you are copying subranges of a file. Part of the application here (as I understood it) is to initialise specific disk volumes on existing VM images when doing thin provisioning. In that case, the reported image size won't ever change...

If they were sparse files then st_blocks would presumably change, but that's not necessarily going to be the case. So, ok stat() is out for this...

What's the use-case for these sorts of progress reports anyway? Progress meters in GUI apps?

Either way, I think adding as simple an interface as possible to begin with makes sense. If you want to add progress reports or other doohickeys later, then that can be done in a separate set of patches...

Quoted 2 lines

Why not support something like the async-iocb?

You could, but that would tie copyfile() to the aio interface which was one of the things that I believe Al was opposed to when we discussed this at LSF/MM-2010.

Quoted 7 lines

Quoted 2 lines

Why not support something like the async-iocb?

You could, but that would tie copyfile() to the aio interface which was one of the things that I believe Al was opposed to when we discussed this at LSF/MM-2010.

virtualization vendors who support this offload do it at a layer above the guest-OS(Intra-LUN(tm) locking or whatever fancy locking). So I think 'copyfile' is going to be appealing to application-developers more than the hypervisor-vendors.

So let's think about it from end-users perspective:
Won't everyone replicate code to check - 'Am I done'? It will just make application folks write more (ugly)code. Because you would then have to maintain another queue/etc to check for this operation.

We can just support full-copy. Partial copies can be returned as failure.

Quoted 6 lines · Chetan

Quoted 5 lines

Quoted 1 line

Why not support something like the async-iocb?

You could, but that would tie copyfile() to the aio interface which was one of the things that I believe Al was opposed to when we discussed this at LSF/MM-2010.

virtualization vendors who support this offload do it at a layer above the guest-OS(Intra-LUN(tm) locking or whatever fancy locking). So I think 'copyfile' is going to be appealing to application-developers more than the hypervisor-vendors.

The application is thin provisioning, not the 'cp' command. When virtualisation vendors do support this, it will mainly be as part of their image management toolkits, not the hypervisor.

Quoted 3 lines

So let's think about it from end-users perspective:
Won't everyone replicate code to check - 'Am I done'? It will just make application folks write more (ugly)code. Because you would then have to maintain another queue/etc to check for this operation.

'Am I done' is easy: copyfile() returns with the number of bytes that have been copied.

'Is my copyfile() syscall making progress' is the question that needs answering.

Quoted 1 line

We can just support full-copy. Partial copies can be returned as failure.

Then you have to check the entire range on error instead of just resuming the copy from where it stopped.

What I would like to see is a way to make sure that we can interrupt any long running command & also make sure that our timeouts (for SCSI specifically) are not too aggressive.

I think that hypervisor vendors will be very interested in this feature which
would explain why vmware was active in drafting both the NFS and T10

Quoted 26 lines · J. Bruce Fields

Quoted 13 lines · Ric Wheeler

Quoted 12 lines · Al Viro

Quoted 7 lines · Ric Wheeler

We had an active thread a couple of years back that came out of the reflink work and, at the time, there seemed to be moderately positive support for adding a new system call that would fit this use case (Joel Becker's copyfile()).

Can we resurrect this effort? Is copyfile() still a good way to go, or should we look at other hooks?

copyfile(2) is probably a good way to go, provided that we do _not_ go baroque as it had happened the last time syscall had been discussed.

Quoted 8 lines

Quoted 4 lines

IOW, to hell with progress reports, etc. - just a fastpath kind of thing, in the same kind of relationship to cp(1) as rename(2) is to mv(1). If it works - fine, if not - caller has to be ready to deal with handling cross-device case anyway.

I think that this approach makes a lot of sense. Most of the devices/targets that support the copy offload, will do it in very reasonable amounts of time.

The current NFSv4.2 draft rolls both the "fast" and "slow" cases into one operation:

Quoted 4 lines

	http://tools.ietf.org/html/draft-ietf-nfsv4-minorversion2-06#section-2

Perhaps we should ask for separate operations for the two cases. (Or at least a "please don't bother if this is going to take 8 hours" flag....)

How would the server know?

Quoted 3 lines

I think that hypervisor vendors will be very interested in this feature which
would explain why vmware was active in drafting both the NFS and T10

Specs are the only way to convince storage-target-vendors ;). Otherwise target-stack will need to implement multiple custom-CDB-handlers for different front-end APIs(which is ugly).

Chetan

Quoted 9 lines · Al Viro

Quoted 7 lines · Ric Wheeler

We had an active thread a couple of years back that came out of the reflink work and, at the time, there seemed to be moderately positive support for adding a new system call that would fit this use case (Joel Becker's copyfile()).

Can we resurrect this effort? Is copyfile() still a good way to go, or should we look at other hooks?

Quoted 7 lines

copyfile(2) is probably a good way to go, provided that we do _not_ go baroque as it had happened the last time syscall had been discussed.

IOW, to hell with progress reports, etc. - just a fastpath kind of thing, in the same kind of relationship to cp(1) as rename(2) is to mv(1). If it works - fine, if not - caller has to be ready to deal with handling cross-device case anyway.

I think that this approach makes a lot of sense. Most of the devices/targets that support the copy offload, will do it in very reasonable amounts of time.

Let me see if I can dig up some of the presentations from the NetApp guys who presented overviews or the specifications from the IETF and T10....

Quoted 28 lines · Jeremy Allison

Quoted 11 lines · Ric Wheeler

Back at LinuxCon Prague, we talked about the new NFS and SCSI commands that let us offload copy operations to a storage device (like an NFS server or storage array).

This got new life in the virtual machine world where you might want to clone bulky guest files or ranges of blocks and was driven through the standards bodies by vmware, microsoft and some of the major storage vendors. Windows8 has this functionality fully coded and integrated in the GUI, I assume vmware also uses it and there are some vendors who announced support at the SNIA SDC conference.

Quoted 7 lines

We had an active thread a couple of years back that came out of the reflink work and, at the time, there seemed to be moderately positive support for adding a new system call that would fit this use case (Joel Becker's copyfile()).

Can we resurrect this effort? Is copyfile() still a good way to go, or should we look at other hooks?

Windows uses a COPYCHUNK call, which specifies the following parameters:

Definition of a copy "chunk":

hyper source_off;
hyper target_off;
uint32 length;

Quoted 6 lines

and an array of these chunks which is passed
into their kernel.

This is what we have to implement in Samba.

Jeremy.

This is a public pointer to the draft NFS proposal:

http://tools.ietf.org/id/draft-lentini-nfsv4-server-side-copy-06.txt

The T10 site has some click through that I was not too happy about agreeing to. NetApp (Fred Knight) had some nice presentations that he presented about how SCSI does this in two different ways...

Quoted 3 lines

Can we resurrect this effort? Is copyfile() still a good way to go, or should we look at other hooks?

Windows uses a COPYCHUNK call, which specifies the following parameters:

Definition of a copy "chunk":

hyper source_off;
hyper target_off;
uint32 length;

and an array of these chunks which is passed
into their kernel.

This is what we have to implement in Samba.

Quoted 13 lines · Jeremy Allison

Quoted 3 lines

Can we resurrect this effort? Is copyfile() still a good way to go, or should we look at other hooks?

Windows uses a COPYCHUNK call, which specifies the following parameters:

Definition of a copy "chunk":

hyper source_off;
hyper target_off;
uint32 length;

Quoted 5 lines

and an array of these chunks which is passed
into their kernel.

This is what we have to implement in Samba.

Could we do this by (re-)allowing sendfile() between two files?

Quoted 13 lines · Jeremy Allison

Quoted 3 lines

Can we resurrect this effort? Is copyfile() still a good way to go, or should we look at other hooks?

Windows uses a COPYCHUNK call, which specifies the following parameters:

Definition of a copy "chunk":

hyper source_off;
hyper target_off;
uint32 length;

Quoted 5 lines

and an array of these chunks which is passed
into their kernel.

This is what we have to implement in Samba.

Could we do this by (re-)allowing sendfile() between two files?

That was my immediate thought, but sendfile has plumbing that is page cache based and we require completely different infrastructure and semantics for an array offload.

e.g. for an array offload, we have to flush the source file page cache first so that the data being copied is known to be on disk, then invalidate the destination page cache if overwriting or extend and pre-allocate blocks if not. Then we have to map both files and hand that off to the array.

Then there's a whole bunch of tricky questions about what the state of the destination file should look like while the copy is in progress, whether the source file should be allowed to change (e.g. it can't be truncated and have blocks freed and then reused by other files half way through the copy offload operation), and so on.

sendfile() has well known, fixed semantics that we can't change to suit what is needed for an offload operation that could potentially take hours to complete. Hence I think an new syscall is the way to go....