Skip to contentSkip to conversation

OT: why no file copy() libc/syscall ??

DISCUSSION Ihar 'Philips' Filipau 5 participants 7 messages Nov 11, 2003 7 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw
Quoted 7 lines · Florian Weimer
Quoted 4 lines · Andreas Dilger
Quoted 3 lines

This is fast turning into a creeping horror of aggregation. I defy anybody to create an API to cover all the options mentioned so far and *not* have it look like the process_clone horror we so roundly derided a few weeks ago.

Quoted 5 lines
Quoted 1 line

int sys_copy(int fd_src, int fd_dst)

Doesn't work. You have to set the security attributes while you open fd_dst.

int new_fd = sys_copy( int src_fd ); /* cloned copy, out of any fs */
fchmod( new_fd, XXX_WHAT_EVER ); /* do the job. */
...
flink(new_fd, "/some/path/some/file/name"); /* commit to fs */ close(new_fd); /* bye-bye */

I beleive this can be more useful. Not only in naive tries to replace
cp(1) with kernel ;-)

Signature · 5 lines
Ihar 'Philips' Filipau  / with best regards from Saarbruecken.
--                                                           _ _ _
 "... and for $64000 question, could you get yourself       |_|*|_|
   vaguely familiar with the notion of on-topic posting?"   |_|_|*|
                                -- Al Viro @ LKML           |*|*|*|
#7 lore raw
Quoted 13 lines · Ihar 'Philips' Filipau
Quoted 12 lines · Florian Weimer
Quoted 7 lines · Andreas Dilger
Quoted 5 lines

This is fast turning into a creeping horror of aggregation. I defy anybody
to create an API to cover all the options mentioned so far and *not* have it
look like the process_clone horror we so roundly derided a few weeks ago.

int sys_copy(int fd_src, int fd_dst)

Quoted 7 lines
Quoted 2 lines

Doesn't work. You have to set the security attributes while you open fd_dst.

int new_fd = sys_copy( int src_fd ); /* cloned copy, out of any fs */
fchmod( new_fd, XXX_WHAT_EVER ); /* do the job. */
...
flink(new_fd, "/some/path/some/file/name"); /* commit to fs */

The associate open file descriptor with a new path system call (flink here) has already been rejected for solid security reasons.

Quoted 4 lines

close(new_fd); /* bye-bye */

I beleive this can be more useful. Not only in naive tries to replace
cp(1) with kernel ;-)

Eliminating the flink and using file descriptors you wind up with something like:

	in_fd = open(oldpath, O_RDONLY);
	fstat(in_fd, statbuf);
	out_fd = open(newpath, O_WRONLY|flags, statbuf->st_mode);
	sendfile(out_fd, in_fd, 0, statbuf->st_size);
	close(out_fd);
	close(in_fd);

So if you can do it with open file descriptors why do you need a new system call?

Signature · 5 lines
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		[email protected]

		Remember Cernan and Schmitt

A message that is not in the archive (sent off-list or not yet received): [email protected]

#9 lore raw
Quoted 6 lines · Rogier Wolff

On Mon, Nov 10, 2003 at 08:05:11PM -0500, Albert Cahalan wrote:

long copy_fd_to_fd (int src, int dst, int len)

The kernel then becomes something

Quoted 8 lines
	if (islocalfile (src) && issocket (dst)) 
		/* Call the old sendfile */ 
		return sendfile (....);

	if (isCIFS (src), isCIFS(dst))
		/* Tell remote host to copy the file. */
		return CIFS_copy_file (....); 

B.S.

Quoted 5 lines

But alas, last time Linus didn't agree with me and decided we should do something like "sendfile", which is IMHO just a special case of this one.

I will reply on behalf of Linus: "Send patch!"

I beleive you are not developer - so you even cannot estimate what
you are proposing.

This kind of patch will never be accepted.

Just try to imagine: 20 file systems, so 20*20 == 400 ifs?

So I beleive you will get more more positive responses, If you will
start improveing vfs, e.g. adding generic routines for optimized move of file from one file system to another, with API which allow it to extrapolate nicely to networked file systems.
Since right now there is no way to pass file from one fs to another -
so basicly this thread is already, well, over ;-)

Quoted 8 lines

If we implement this in kernel (at first just the copy_fd_fd and the default implementation), then we can get "cp" to use this, and then suddenly whenever we upgrade the kernel, cp can use the newly optimized copying mechanism. (e.g. whenever we manage to specify a socket as the destination, cp would suddenly start to use "sendfile"!!)

Silly. cp is least frequent application I use.
And cvs I beleive already uses sendfile().
So all your /arguments/ go directly into /dev/null, since if file is
not in cvs - you know - it just doesn't exist ;-)))

Quoted 3 lines

Roger.

Signature · 5 lines
Ihar 'Philips' Filipau  / with best regards from Saarbruecken.
--                                                           _ _ _
 "... and for $64000 question, could you get yourself       |_|*|_|
   vaguely familiar with the notion of on-topic posting?"   |_|_|*|
                                -- Al Viro @ LKML           |*|*|*|
#10 lore raw
Quoted 10 lines · Ihar 'Philips' Filipau
Quoted 4 lines · Rogier Wolff

But alas, last time Linus didn't agree with me and decided we should do something like "sendfile", which is IMHO just a special case of this one.

I will reply on behalf of Linus: "Send patch!"

I beleive you are not developer - so you even cannot estimate what
you are proposing.

Wrong.

Quoted 1 line

This kind of patch will never be accepted.

Yes. As I said: Linus doesn't agree with me. I don't sleep less from knowing that. Feel free to disagree with me as well.

Quoted 1 line

Just try to imagine: 20 file systems, so 20*20 == 400 ifs?

Right! And: Wrong!

The idea is that the default will make sure that the kernel handles the call. It's just as efficient as the userspace implementation.

But currently we have decided that the extra efficiency of

	"local file -> socket" 

matters enough to us that we want to optimize that case. Fine. So now we have "sendfile". This is currently implemented as a special systemcall. I.e. one of those 400 cases you mentioned.

But I expect that only a few cases will be important enough that we care to optimize their implementation.

If we end up with 400 ifs, because we CAN optimize each and every case by itself, and we find that important enough to actually implement, then of course the "string of ifs" is a nice candidate to optimize again.

Quoted 4 lines

So I beleive you will get more more positive responses, If you will
start improveing vfs, e.g. adding generic routines for optimized move of file from one file system to another, with API which allow it to extrapolate nicely to networked file systems.

Once my proposed "copy_fd_to_fd" is in place, the road is open towards just leaving the current special case that detects: "src uses pagecache dst is a socket" and then calls the current sendfile implementation.

Quoted 1 line

Silly. cp is least frequent application I use.

Yeah. So? You reject a general idea just because you don't use the application that I used as an example in my proposal.

Quoted 1 line

And cvs I beleive already uses sendfile().

Fine. For compatibilty we'll leave "sendfile" in place. But if somehow someone builds a filesystem which cannot use the pagecache, then "sendfile" will fail. Or if somehow we manage to get the socket hooked up to something else (*). Either CVS needs to handle that case internally, or it will fail. In the first case, that causes extra code in lots of applications that want to continue to work, in the latter case, it's bad.

Roger.

(*) Suppose I manage to stop and restart an application. The "restart" program might need to "sit between" the original application and its filedescriptors. So now, what used to be a socket suddenly becomes a pipe. It'd be nice if things would continue to work. Everything is a file remember?

Signature · 3 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
**** "Linux is like a wigwam -  no windows, no gates, apache inside!" ****
#11 lore raw
Quoted 9 lines · Rogier Wolff

Fine. For compatibilty we'll leave "sendfile" in place. But if somehow someone builds a filesystem which cannot use the pagecache, then "sendfile" will fail. Or if somehow we manage to get the socket hooked up to something else (*). Either CVS needs to handle that case internally, or it will fail. In the first case, that causes extra code in lots of applications that want to continue to work, in the latter case, it's bad.

I beleive - if you really want to have something like this - you need
to go to e.g. nfs/coda/smbfs developers and talk with them: how it can be implemented in this situations.

Implement it with ioctl() - to really see make it sense or it just
complicates things enourmously. Actually given networked file systems could be just NOT capable of this kind of operation at all.

Insisting on new syscall is silly: syscall is interface - it has
nothing to do with functionality. Ocasionally syscalls are used to access functionality ;-) So start from functionality first. Syscall (or whatever interface will fit better) can be implemented in 15 minutes any time after functionality is in place.

Signature · 5 lines
Ihar 'Philips' Filipau  / with best regards from Saarbruecken.
--                                                           _ _ _
 "... and for $64000 question, could you get yourself       |_|*|_|
   vaguely familiar with the notion of on-topic posting?"   |_|_|*|
                                -- Al Viro @ LKML           |*|*|*|
#12 lore raw
In reply to Rogier Wolff
Quoted 3 lines · Rogier Wolff

Fine. For compatibilty we'll leave "sendfile" in place. But if somehow someone builds a filesystem which cannot use the pagecache, then "sendfile" will fail. Or if somehow we manage to get the socket hooked

...

Quoted 5 lines

(*) Suppose I manage to stop and restart an application. The "restart" program might need to "sit between" the original application and its filedescriptors. So now, what used to be a socket suddenly becomes a pipe. It'd be nice if things would continue to work. Everything is a file remember?

man sendfile(2)

NOTES
...
Applications may wish to fall back to read/write in the case where sendfile() fails with EINVAL or ENOSYS.

So we get something in a userspace library (libc?) that does copyfile(whatever, whereever) and uses a few kernel primitives like open/close/sendfile and the appropriate fallback code to a read/write loop whenever the sendfile doesn't work.

It works now, and it will work better when sendfile becomes more versatile, and the sky is the limit once the underlying filesystem can provide it's own optimized implementation for instance when both fd's refer to objects within the same (remote) filesystem.

Similarily, we might at some point be able to optimize sendfile between two sockets by pushing the connection off to a router somewhere in the network completely bypassing the local NIC.

Jan

#13 lore raw
Quoted 3 lines · Jan Harkes

Similarily, we might at some point be able to optimize sendfile between two sockets by pushing the connection off to a router somewhere in the network completely bypassing the local NIC.

Security can of worms there.. :)