*Really* bad I/O latency with md raid5+dm-crypt+lvm
Quoted 6 lines
Summary: I was hoping to use a layered storage setup, namely lvm on dm-crypt on md raid5 for a new box I'm setting up, but that isn't looking so good since a single heavyish writer will monopolise any and all I/O on the "device". F. ex. while cp'ing a few GB of data from an external disk to the array it takes ~10sec to run ls and ~2min to start aptitude. Clueless attempts at a diagnosis below.
Did you try running strace to see where ls pauses?
Did you try running latencytop (and generally, top/htop while doing your tests)?
(...)
Quoted 7 lines
Anyway, as soon as I copy something to the array or create a larger (upwards of a few hundred MiB) tar archive the box becomes utterly unresponsive until that job is finished. Even on the local console the completion time for a simple ls or cat is of the order of tens of seconds, just forget about launching emacs.
Now I know that people have been ranting about desktop responsiveness for a while but that was very much an abstract thing for me until now.
I think the above (big latency when doing some bigger IO) is a general Linux problem.
I see similar behaviour on quite powerful hardware, i.e. Core i7, 8 GB RAM, 2x HDD in a software RAID-1 array (no dm-crypt), when tarring something big, or writing dd if=/dev/zero of=/home/me/bigfile - doing ls in another terminal or just starting top can take up to a minute.
Quite interestingly, background RAID synchronization have almost no effect on latency.
(...)
Quoted 5 lines
According to openssl speed aes-256-cbc the CPUs encryption speed is ~113 MiB/s (single core, est. for 512b blocks). Obviously the array is much faster than that. I can't find the benchmarks ATM but the numbers seemed plausible for 70 MiB/s (optimistic est. for sequential access) disks at the time.
You can find some dm-crypt benchmarks i.e. here:
http://blog.wpkg.org/2009/04/23/cipher-benchmark-for-dm-crypt-luks/
Obviously, they will not match your hardware.
Also note that dm-crypt is not "SMP-ready", so whatever hardware you have, it will only use once CPU - this may seriously limit the performance, depending on your usage and hardware.
Signature · 1 line
Tomasz Chmielewski
Quoted 23 lines · Tomasz Chmielewski
Quoted 6 lines
Summary: I was hoping to use a layered storage setup, namely lvm on dm-crypt on md raid5 for a new box I'm setting up, but that isn't looking so good since a single heavyish writer will monopolise any and all I/O on the "device". F. ex. while cp'ing a few GB of data from an external disk to the array it takes ~10sec to run ls and ~2min to start aptitude. Clueless attempts at a diagnosis below.
Did you try running strace to see where ls pauses?
Did you try running latencytop (and generally, top/htop while doing your tests)?
(...)
Quoted 7 lines
Anyway, as soon as I copy something to the array or create a larger (upwards of a few hundred MiB) tar archive the box becomes utterly unresponsive until that job is finished. Even on the local console the completion time for a simple ls or cat is of the order of tens of seconds, just forget about launching emacs.
Now I know that people have been ranting about desktop responsiveness for a while but that was very much an abstract thing for me until now.
Quoted 2 lines
I think the above (big latency when doing some bigger IO) is a general Linux problem.
It would be interesting to test latest -rc. Though it may prove to be unrelated. the symptoms sound very much like a recent thread wrt writers starving readers.
-Mike