Skip to contentSkip to conversation

What is the most efficient way to copy a bunch of files nowadays?

DISCUSSION Olivier Galibert 2 participants 2 messages Apr 11, 2006 – Apr 13, 2006

lore
Share & tools
lore
#0 lore raw

After having read-only mucked with the headers of a bunch of files[1] I've selected a subset of 2500 or so with sizes going from 2K to 85M which I want to copy to another directory in the same local filesystem. What is the "best" (CPU usage, fragmentation, wall clock time, system responsiveness during and after the copy) way to copy these files? read+write, mmap+write, read+mmap, mmap+mmap+memcpy, something else? That's with recent kernels, of course.

OG.

[1] Say rpms from a random distribution and its updates for instance

#1 lore raw
Quoted 6 lines

After having read-only mucked with the headers of a bunch of files[1] I've selected a subset of 2500 or so with sizes going from 2K to 85M which I want to copy to another directory in the same local filesystem. What is the "best" (

CPU usage

nice

Quoted 1 line

fragmentation

What? You can't really influence it.

Quoted 1 line

wall clock time

Using a filesystem with aggressive caching.

Quoted 1 line

system responsiveness during and after the copy)

See one above. Set /proc/sys/vm/dirty_ratio high to have the flush taking place late, or low to have it "early+often".

Quoted 2 lines

way to copy these files? read+write, mmap+write, read+mmap, mmap+mmap+memcpy,

It all boils down to the same: read the file and write it back. So all your methods cost equally much. With splice() instead of read/write however, you may save some cycles.

Quoted 1 line

something else? That's with recent kernels, of course.

Jan Engelhardt