OT: why no file copy() libc/syscall ??
A message that is not in the archive (sent off-list or not yet received): [email protected]
sendfile(2) - ?
Quoted 21 lines · Davide Rossetti
it may be orribly RTFM... but writing a simple framework I realized there is no libc/POSIX/whoknows
copy(const char* dest_file_name, const char* src_file_name)What is the technical reason???
I understand that there may be little space for kernel side optimizations in this area but anyway I'm surprised I have to write
< the bits to clone the metadata of
src_file_nameon openingdest_file_name> const int BUFSIZE = 1<<12;
char buffer[BUFSIZE];
int nrb;
while((nrb =read(infd, buffer, BUFSIZE)!= -1) {
ret =write(outfd, buffer, nrb);
if(ret != nrb) {...}
}instead of something similar to:
sys_fscopy(...)regards
Signature · 5 lines
Ihar 'Philips' Filipau / with best regards from Saarbruecken.
-- _ _ _
"... and for $64000 question, could you get yourself |_|*|_|
vaguely familiar with the notion of on-topic posting?" |_|_|*|
-- Al Viro @ LKML |*|*|*|Quoted 1 line · Ihar 'Philips' Filipau
sendfile(2) - ?
I don't think that is what he was referring to.. The sample code is strictly user mode file->file copying.
Quoted 6 lines
Quoted 5 lines · Davide Rossetti
it may be orribly RTFM... but writing a simple framework I realized there is no libc/POSIX/whoknows
copy(const char* dest_file_name, const char* src_file_name)What is the technical reason???
It isn't an application for the kernel.
Quoted 13 lines
Quoted 13 lines
I understand that there may be little space for kernel side optimizations in this area but anyway I'm surprised I have to write
< the bits to clone the metadata of
src_file_nameon openingdest_file_name>
const int BUFSIZE = 1<<12;
char buffer[BUFSIZE];
int nrb;
while((nrb =read(infd, buffer, BUFSIZE)!= -1) {
ret =write(outfd, buffer, nrb);
if(ret != nrb) {...}
}
Quoted 2 lines
Quoted 2 lines
instead of something similar to:
sys_fscopy(...)
It is too simple to implement in user mode.
There are some other issues too:
The security context of the output depends on the user process. If it is a privileged process (ie, may change the context of the result) then the user process has to setup that context before the file is copied.
There are also some issues with mandatory security controls. If it is copied in kernel mode, then the previous labels could be automatically carried over to the resulting file... But that may not be what you want (and frequently, it isn't).
Now back to the copy.. You don't have to use a read/write loop- mmap is faster. And this is the other reason for not doing it in Kernel mode. Buffer management of this type is much easier in user space since the copy procedure doesn't have to deal with memory limitations, cache flushes page faulting of processes unrelated to the copy, but is related to cache pressure.
Quoted 2 lines · Jesse Pollard
Now back to the copy.. You don't have to use a read/write loop- mmap is faster. And this is the other reason for not doing it in Kernel mode.
Actually, last I checked, read/write was actually faster. Linus explained why a month or two ago.
Signature · 2 lines
Daniel Jacobowitz MontaVista Software Debian GNU/Linux Developer
Quoted 6 lines · Daniel Jacobowitz
Quoted 2 lines · Jesse Pollard
Now back to the copy.. You don't have to use a read/write loop- mmap is faster. And this is the other reason for not doing it in Kernel mode.
Actually, last I checked, read/write was actually faster. Linus explained why a month or two ago.
It would also not break on large files...
Signature · 6 lines
................................................................ : [email protected] : And I see the elder races, : :.........................: putrid forms of man : : Jakob Østergaard : See him rise and claim the earth, : : OZ9ABN : his downfall is at hand. : :.........................:............{Konkhra}...............:
Quoted 3 lines · Jesse Pollard
Quoted 1 line
Quoted 1 line
sys_fscopy(...)It is too simple to implement in user mode.
Is it? Please explain the simple steps which cp(1) should take in order to observe that it is being asked to duplicate a file on a file system such as CIFS (or NFSv4?) which allows the client to issue a 'copy file' command over the network without actually transferring the data twice, and to invoke such a command.
Signature · 1 line
dwmw2
Quoted 10 lines · David Woodhouse
Quoted 3 lines · Jesse Pollard
Quoted 1 line
Quoted 1 line
sys_fscopy(...)It is too simple to implement in user mode.
Is it? Please explain the simple steps which cp(1) should take in order to observe that it is being asked to duplicate a file on a file system such as CIFS (or NFSv4?) which allows the client to issue a 'copy file' command over the network without actually transferring the data twice, and to invoke such a command.
Ah. That is an optimization question, not a question of kernel/user mode.
Since the error checking for source and destination both include doing a stat and statfs, the device information (and FS info) can both be retrieved.
And mmap doesn't require data transfer "twice" (local copy). Since that copy only pagefaults (though read/write may be faster for some files - I thought that was true for small files that fit in cache, and large files faster via mmap and depends on the page size; and the tradeoff would be system dependant).
And since both source and destination may be remote you do get to decide based on source and destination devices: if they are the same, and one on a remote node, then BOTH will be on the remote, then you get to use the CIFS/NFS file copy. (check the doc on "stat/statfs" for additional info).
I don't believe it works when source and destination are on DIFFERENT remote nodes, though.
Strictly up to the implementation of cp/mv.
Though you will loose portability of cp/mv. (Of course, you also loose it with a syscall for file copy too; as well as the MUCH more complicated implementation/security checks).
Quoted 4 lines · Jesse Pollard
Quoted 1 line · Ihar 'Philips' Filipau
sendfile(2) - ?I don't think that is what he was referring to.. The sample code is strictly user mode
file->filecopying.
Quoted 8 lines
Quoted 6 lines
Quoted 5 lines · Davide Rossetti
it may be orribly RTFM... but writing a simple framework I realized there is no libc/POSIX/whoknows
copy(const char* dest_file_name, const char* src_file_name)What is the technical reason???
It isn't an application for the kernel.
Maybe I was misunderstood... I'm asking why the libc/iso/ansi/posix engineer did not add the spec a user-mode API to do copy file to file ???
if there was such a standard _user_ API, we could talk about user/kernel implementation issues... but my question is more "primitive" somehow :)
Quoted 30 lines
Quoted 13 lines
Quoted 13 lines
I understand that there may be little space for kernel side optimizations in this area but anyway I'm surprised I have to write
< the bits to clone the metadata of
src_file_nameon openingdest_file_name>
const int BUFSIZE = 1<<12;
char buffer[BUFSIZE];
int nrb;
while((nrb =read(infd, buffer, BUFSIZE)!= -1) {
ret =write(outfd, buffer, nrb);
if(ret != nrb) {...}
}Quoted 2 lines
Quoted 2 lines
instead of something similar to:
sys_fscopy(...)It is too simple to implement in user mode.
There are some other issues too:
The security context of the output depends on the user process. If it is a privileged process (ie, may change the context of the result) then the user process has to setup that context before the file is copied.
There are also some issues with mandatory security controls. If it is copied in kernel mode, then the previous labels could be automatically carried over to the resulting file... But that may not be what you want (and frequently, it isn't).
Quoted 6 lines
Now back to the copy.. You don't have to use a read/write loop- mmap is faster. And this is the other reason for not doing it in Kernel mode. Buffer management of this type is much easier in user space since the copy procedure doesn't have to deal with memory limitations, cache flushes page faulting of processes unrelated to the copy, but is related to cache pressure.
ok... so I have to code a framework routine which auto-benchmarks (at either runtime or configure time) and uses at least 2 implementations, one using read/write and another mmap(), as I know for sure that on different Unices they perform differently... ah.. and the day we add sys_sendfile(fd,fd) (if it is not there yet) I have to add yet another implementation... and doing file copies of gigabyte sized files with mmap() on 32bit archs isn't so trivial, you have to do windowing I guess...
seems scary at least ;)
<joke>
it seems similar to saying that we do not need a rename() Posix/XOpen/etc API as we can do:
rename(to, from) {
link(to, from); // make hardlink
unlink(from); // remove original
}
</joke>
regards
Signature · 3 lines
______/ Rossetti Davide INFN - Roma I - APE group \______________ pho +390649914507/412 web: http://apegate.roma1.infn.it/~rossetti fax +390649914423 email: [email protected]
Quoted 2 lines · davide.rossetti
Maybe I was misunderstood... I'm asking why the libc/iso/ansi/posix engineer did not add the spec a user-mode API to do copy file to file ???
Because there was no prior art.
Andreas.
Signature · 4 lines
Andreas Schwab, SuSE Labs, [email protected] SuSE Linux AG, Deutschherrnstr. 15-19, D-90429 Nürnberg Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different."
Quoted 6 lines · Andreas Schwab
Quoted 2 lines · davide.rossetti
Maybe I was misunderstood... I'm asking why the libc/iso/ansi/posix engineer did not add the spec a user-mode API to do copy file to file ???
Because there was no prior art.
:) but late revisions of specs are really recent!!!
folks are talking about implementing all sort of stuff (web servers, parallel filesystems, ...) (partly) in kernel mode and no one cares of (maybe accelerated) fs copies ???
Signature · 3 lines
______/ Rossetti Davide INFN - Roma I - APE group \______________ pho +390649914507/412 web: http://apegate.roma1.infn.it/~rossetti fax +390649914423 email: [email protected]