What is the most efficient way to copy a bunch of files nowadays?
After having read-only mucked with the headers of a bunch of files[1] I've selected a subset of 2500 or so with sizes going from 2K to 85M which I want to copy to another directory in the same local filesystem. What is the "best" (CPU usage, fragmentation, wall clock time, system responsiveness during and after the copy) way to copy these files? read+write, mmap+write, read+mmap, mmap+mmap+memcpy, something else? That's with recent kernels, of course.
OG.
[1] Say rpms from a random distribution and its updates for instance
Quoted 6 lines
After having read-only mucked with the headers of a bunch of files[1] I've selected a subset of 2500 or so with sizes going from 2K to 85M which I want to copy to another directory in the same local filesystem. What is the "best" (
CPU usage
nice
Quoted 1 line
fragmentation
What? You can't really influence it.
Quoted 1 line
wall clock time
Using a filesystem with aggressive caching.
Quoted 1 line
system responsiveness during and after the copy)
See one above. Set /proc/sys/vm/dirty_ratio high to have the flush taking place late, or low to have it "early+often".
Quoted 2 lines
way to copy these files? read+write, mmap+write, read+mmap, mmap+mmap+memcpy,
It all boils down to the same: read the file and write it back. So all your methods cost equally much. With splice() instead of read/write however, you may save some cycles.
Quoted 1 line
something else? That's with recent kernels, of course.
Jan Engelhardt