md-raid5+lvm+dm-crypt+kvm: one streaming write starves all reads
Hi again!
The very short version is that writing a larger stream of data to an encrypted volume starves all reads on *all* encrypted volumes. Doesn't matter if they reside on different disks or even different controllers.
Example: Alice is watching a video (on a windows client using VLC to access the file on a samba share, in case it matters). Then Bob starts backing up his /home of multiple gigs using tar, at which point the video turns into a slideshow. And that's putting it mildly ...
Since the unencrypted volumes are fine it seem dm-crypt is the culprit but it might as well be any other layer in our storage setup or, more likely, a combination.
Anyway, on the lowest level are a md-raid1 and two -raid5 arrays, grouped into lvm VGs raid1 and raid5 respectively. Some of the LVs on that are encrypted, some aren't. These LVs in turn serve as virtual disks for a handful of kvm boxes (via virtio). OS is Debian 5.02 (lenny) using the 2.6.30 kernel from backports.org, running on an Intel C2Q with 8GB of RAM.
I thought 2.6.30 would split the crypto work between the four cores, but it doesn't. Performance maxes out just over 100MiB/s, which is close to the crypto performance of one core. (openssl speed aes-256-cbc gives me ~115MiB/s for one core, ~440MiB/s for all four.) That's probably kvm's fault, since from the pov of the kernel doing the I/O a whole virtual machine is just one process ... and all parallelization, I/O scheduling etc. seems to go out of the window.
One of the VMs is our file server and it's doing fine except that writes nearly drown out reads, see above. I've tried messing with the I/O schedulers on the host and guest side, no change. One write stream is enough to bring the whole disk subsystem to its knees. 15 secs to execute /bin/ls as soon as one dd if=/dev/zero of=test bs=4M is running ...
All I can think of at this point is that the disks are "too fast" for the CPU. The VM sends down write requests continously, all of which take cycles to encrypt before they even hit the I/O scheduling queue. Now along comes a lonely read that has to wait ages for decryption since the crypt processes are so busy with writes. Since the arrays can write 2-3x faster than the CPU can encrypt, the (I/O scheduling) queue never comes into action ...
Any ideas on how I could diagnose the problem are very much appreciated, I'll gladly answer any request for further info.
Thanks,
Chris
This sounds more like a general I/O flushing problem. I uses to use a (slow) MO drive for backup and had this all the time after the kernel developers removed easy tuning of the filesystem flush paramaters. (Which I still think was arrogant and stupid.) With earlier flush I did not run into emergency flushes. With the default paramaters I always ran into emergency flushes and the whole machine would lock up for minutes at a time. The lock-up is gone now, likely due to different locking granularity, but I still miss my manually tuned filesystem on occasion, when I run into exactly the starvation you describe.
It is possible that tuning the paramaters can be done again by now, I have not looked into it for some time.
I think the idea was that early writing has a bit lower performance, but is easier to interrupt.
So, possibly no connection to dm-crypt, just the
buffer-cache having badly tuned parameters for
this type of operation.
Arno
Quoted 57 lines · Christian Pernegger
Hi again!
The very short version is that writing a larger stream of data to an encrypted volume starves all reads on *all* encrypted volumes. Doesn't matter if they reside on different disks or even different controllers.
Example: Alice is watching a video (on a windows client using VLC to access the file on a samba share, in case it matters). Then Bob starts backing up his /home of multiple gigs using tar, at which point the video turns into a slideshow. And that's putting it mildly ...
Since the unencrypted volumes are fine it seem dm-crypt is the culprit but it might as well be any other layer in our storage setup or, more likely, a combination.
Anyway, on the lowest level are a md-raid1 and two -raid5 arrays, grouped into lvm VGs raid1 and raid5 respectively. Some of the LVs on that are encrypted, some aren't. These LVs in turn serve as virtual disks for a handful of kvm boxes (via virtio). OS is Debian 5.02 (lenny) using the 2.6.30 kernel from backports.org, running on an Intel C2Q with 8GB of RAM.
I thought 2.6.30 would split the crypto work between the four cores, but it doesn't. Performance maxes out just over 100MiB/s, which is close to the crypto performance of one core. (openssl speed aes-256-cbc gives me ~115MiB/s for one core, ~440MiB/s for all four.) That's probably kvm's fault, since from the pov of the kernel doing the I/O a whole virtual machine is just one process ... and all parallelization, I/O scheduling etc. seems to go out of the window.
One of the VMs is our file server and it's doing fine except that writes nearly drown out reads, see above. I've tried messing with the I/O schedulers on the host and guest side, no change. One write stream is enough to bring the whole disk subsystem to its knees. 15 secs to execute
/bin/lsas soon as one dd if=/dev/zero of=test bs=4M is running ...All I can think of at this point is that the disks are "too fast" for the CPU. The VM sends down write requests continously, all of which take cycles to encrypt before they even hit the I/O scheduling queue. Now along comes a lonely read that has to wait ages for decryption since the crypt processes are so busy with writes. Since the arrays can write 2-3x faster than the CPU can encrypt, the (I/O scheduling) queue never comes into action ...
Any ideas on how I could diagnose the problem are very much appreciated, I'll gladly answer any request for further info.
Thanks,
Chris
_______________________________________________
dm-crypt mailing list
[email protected]
http://www.saout.de/mailman/listinfo/dm-crypt
Signature · 7 lines
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: [email protected] GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F ---- Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans If it's in the news, don't worry about it. The very definition of "news" is "something that hardly ever happens." -- Bruce Schneier
Quoted 3 lines
So, possibly no connection to dm-crypt, just the
buffer-cache having badly tuned parameters for
this type of operation.
Quite possible. Normally the block devices kvm provides for guests are backed by the host's cache, which results in native I/O throughput (100MiB/s read and write on encrypted partitions) in the guests but has a major starvation problem when dm-crypt is part of the mix. Forcing kvm to go directly to the disks bypassing the host's cache (using drive=...,cache=none) fixes the starvation issue completely but limits throughput to an appaling 25MB/s read and 15MB/s write ...
Any idea why this setup would behave as it does? How can I check if flushes really are the problem?
Thanks,
Chris
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 2 lines
Hmm. So without dm-crypt no problem? The main effect of dm-crypt is to make the disk appear slower, because of the encryption.
Unfortunately it's not *just* slower. Yes, the arrays would be 2x-3x as fast without encryption, but that's not the point. It's that I/O bandwidth isn't shared fairly at all. As if all crypt requests went into one big FIFO ... (But then, maybe they do.) The stream with the most/biggest requests then gets all the bandwidth.
Quoted 1 line
You could try this with a dummy-cipher that does not take encryption time.
Excellent idea, what's the cipher string for that? I couldn't find anything obvious in /proc/crypto or under /lib/modules/.../kernel/crypto/.
Thanks,
Chris
Quoted 8 lines · Christian Pernegger
Quoted 2 lines
Hmm. So without dm-crypt no problem? The main effect of dm-crypt is to make the disk appear slower, because of the encryption.
Unfortunately it's not *just* slower. Yes, the arrays would be 2x-3x as fast without encryption, but that's not the point. It's that I/O bandwidth isn't shared fairly at all. As if all crypt requests went into one big FIFO ... (But then, maybe they do.) The stream with the most/biggest requests then gets all the bandwidth.
Yes. And that may be some s"synergetic" effect triggered by the slowness. At least that is one possible explanation.
Quoted 5 lines
Quoted 1 line
You could try this with a dummy-cipher that does not take encryption time.
Excellent idea, what's the cipher string for that? I couldn't find anything obvious in
/proc/cryptoor under/lib/modules/.../kernel/crypto/.
I have not tried this myself, but hit candidates are "ciper_null" and "ecb(ciper_null)". I have both (kernel 2.6.30.5)
Arno
Signature · 7 lines
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: [email protected] GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F ---- Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans If it's in the news, don't worry about it. The very definition of "news" is "something that hardly ever happens." -- Bruce Schneier
Quoted 2 lines
Yes. And that may be some s"synergetic" effect triggered by the slowness. At least that is one possible explanation.
Agreed.
Quoted 5 lines
You could try this with a dummy-cipher that does not take encryption time.
I have not tried this myself, but hit candidates are "
ciper_null" and "ecb(ciper_null)". I have both (kernel 2.6.30.5)
They're there all right, must have overlooked them. However I haven't managed to construct a cipher spec that'll work with cryptsetup. Maybe because it takes no key ...
Various tunings of the host's page cache (under /proc/sys/vm/) to make flushes less bursty didn't help much.
In the meantime I've set up a dedicated kvm guest for these tests, with one encrypted and one unencrypted scratch volume. Maybe some systematic benchmarking will point somewhere.
Seems a lot of people are having this problem, since ~2.6.24 at least, and it seems to get worse the slower the CPUs are, the faster the disks are. Chosing a faster cipher seems to help.
Chris
Quoted 7 lines · Christian Pernegger
Quoted 2 lines
Yes. And that may be some s"synergetic" effect triggered by the slowness. At least that is one possible explanation.
Agreed.
Quoted 2 lines
You could try this with a dummy-cipher that does not take encryption time.
Quoted 7 lines
Quoted 3 lines
I have not tried this myself, but hit candidates are "
ciper_null" and "ecb(ciper_null)". I have both (kernel 2.6.30.5)They're there all right, must have overlooked them. However I haven't managed to construct a cipher spec that'll work with cryptsetup. Maybe because it takes no key ...
Possibly. I do not know how to do this either.
Quoted 2 lines
Various tunings of the host's page cache (under
/proc/sys/vm/) to make flushes less bursty didn't help much.
A pity. Have you tried to set dirty_writeback_centisecs to something very low, e.g. 10? If I understand this correctly that would cause regular flushes to start after 1 sec.
Quoted 3 lines
In the meantime I've set up a dedicated kvm guest for these tests, with one encrypted and one unencrypted scratch volume. Maybe some systematic benchmarking will point somewhere.
Good approach. You can also try different speed ciphers. 2fish should be faster than AES.
Quoted 3 lines
Seems a lot of people are having this problem, since ~2.6.24 at least, and it seems to get worse the slower the CPUs are, the faster the disks are. Chosing a faster cipher seems to help.
That would be consistent with a backlog before the cipher step that eventually leads to an emergency flush. One possible solution would be to slow down writes before the buffer becomes full. However I don't think there are provisions for that in the kernel.
Arno
Quoted 6 lines
Chris
_______________________________________________
dm-crypt mailing list
[email protected]
http://www.saout.de/mailman/listinfo/dm-crypt
Signature · 7 lines
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: [email protected] GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F ---- Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans If it's in the news, don't worry about it. The very definition of "news" is "something that hardly ever happens." -- Bruce Schneier