Bad performance with software RAID5 and LUKS encryption
Hi,
I have set up a Linux software RAID5 on three hard drives and want to encrypt it with cryptsetup/LUKS. My tests showed that the encryption leads to a massive performance decrease that I cannot explain.
The RAID5 is able to write 187 MB/s [1] without encryption. With encryption on top of it, write speed is down to about 40 MB/s.
The RAID has a chunk size of 512K and a write intent bitmap. I used -c aes-xts-plain -s 512 --align-payload=2048 as the parameters for cryptsetup luksFormat, so the payload should be aligned to 2048 blocks of 512 bytes (i.e., 1MB), if I understood this correctly. cryptsetup luksDump shows a payload offset of 4096. So I think the alignment is correct and fits to the RAID chunk size.
The CPU should not be the bottleneck here, as it has hardware support for AES (aesni_intel). If I write on another drive (an SSD with LVM) that is also encrypted, I do have a write speed of about 150 MB/s. top shows that the CPU usage is indeed very low, only the RAID5 xor takes 14%.
I also tried putting a filesystem (ext4) directly on the unencrypted RAID so see if the layering is problem. The filesystem decreases the performance a little bit as expected, but by far not that much (write speed varying, but > 100 MB/s).
Summary:
Disks + RAID5: good
Disks + RAID5 + ext4: good
Disks + RAID5 + encryption: bad
SSD + encryption + LVM + ext4: good
The read performance is not affected by the encryption, it is 207 MB/s without and 205 MB/s with encryption (also showing that CPU power is not the problem).
Is it possible to improve the write performance of the encrypted RAID? What could be the reason of the bad performance?
Software versions:
Ubuntu 11.04
Linux 2.6.38
cryptsetup 1.1.3
Help / ideas / etc. would be very much appreciated.
Greetings,
Philipp
[1] All speed measurements were done with several runs of dd if=/dev/zero of=DEV bs=100M count=100
Quoted 2 lines · Philipp Wendler
[1] All speed measurements were done with several runs of dd if=/dev/zero of=DEV bs=100M count=100
Did you consider using another block size? I have noticed that this can have considerable impact under the conditions I was testing once. Do you see any changes if you try, e.g.
dd if=/dev/zero of=DEV bs=4K count=2500000
Jens
Hi,
Am 05.07.2011 15:30, schrieb Jens Wahnes:
Quoted 6 lines
Quoted 2 lines · Philipp Wendler
[1] All speed measurements were done with several runs of dd if=/dev/zero of=DEV bs=100M count=100
Did you consider using another block size?
So far I tested with a large block size only, because I was expecting a bad performance for small block sizes due to the raid5 anyway.
But I have now run a series of tests with 4K, 1M and 10M. Block size indeed affects the performance (although not as much as I would have expected), but the difference for with/without encryption still remains as large as before:
without encryption: 150-180 MB/s
with encryption: 30-40 MB/s
Greetings, Philipp
Hello,
Am 05.07.2011 08:45, schrieb Philipp Wendler:
Quoted 6 lines
I have set up a Linux software RAID5 on three hard drives and want to encrypt it with cryptsetup/LUKS. My tests showed that the encryption leads to a massive performance decrease that I cannot explain.
The RAID5 is able to write 187 MB/s [1] without encryption. With encryption on top of it, write speed is down to about 40 MB/s.
Sorry for answering to a mail in this very old thread,
but it seems I finally found a solution,
and I know that there are some other people interested in the solution as well (so if you got this email directly from me, I BCC'ed you because you contacted me about this).
If you want to read up the full story, here's the link:
http://www.saout.de/pipermail/dm-crypt/2011-July/001773.html
Today I read about the stripe_cache_size setting of md RAID, and tried it out. With the default value of 256, the performance is slow as described. With a value of 4096, I get a performance increase from about 40-50 MB/s to 123 MB/s. For values >= 8192, I get 140 MB/s out of it.
Background: The stripe cache stores recently written blocks. If data is written continuously, it might happen that during a first write only a part of one stripe is written. This means, the RAID code has to read the complete stripe from disk, update it, and write it completely again. If a second write comes in for another part of the same stripe, all this would have to be done again. Now, if the cache is used and still contains the data written by the first write, the read that was necessary before the second write can be omitted.
Now it seems that dm-crypt always writes with small block size to the underlying disk, even when I write with a big block size. Could this be true?
Could this perhaps be improved? While I have found a solution for me, this could probably solve performance problems for many people.
Furthermore, dm-crypt write is still slower than unencrypted write, although for reads the performance of encrypted and unencrypted are the same. So I guess the small block size still has a performance penalty (probably when first writing to a stripe and it is not yet in cache).
My current setup is Ubuntu 12.04 (Linux 3.2).
Nothing else has changed compared to when I first asked about this.
Greetings, Philipp
Hi,
thanks for the info. I think I may add a "performance" section to the FAQ, and this could be one of the items. Do you have the reference where you found the info on the stripe_cache_size?
Arno
Quoted 52 lines · Philipp Wendler
Hello,
Am 05.07.2011 08:45, schrieb Philipp Wendler:
Quoted 6 lines
I have set up a Linux software RAID5 on three hard drives and want to encrypt it with cryptsetup/LUKS. My tests showed that the encryption leads to a massive performance decrease that I cannot explain.
The RAID5 is able to write 187 MB/s [1] without encryption. With encryption on top of it, write speed is down to about 40 MB/s.
Sorry for answering to a mail in this very old thread,
but it seems I finally found a solution,
and I know that there are some other people interested in the solution as well (so if you got this email directly from me, I BCC'ed you because you contacted me about this).
If you want to read up the full story, here's the link:
http://www.saout.de/pipermail/dm-crypt/2011-July/001773.htmlToday I read about the
stripe_cache_sizesetting of md RAID, and tried it out. With the default value of 256, the performance is slow as described. With a value of 4096, I get a performance increase from about 40-50 MB/s to 123 MB/s. For values >= 8192, I get 140 MB/s out of it.Background: The stripe cache stores recently written blocks. If data is written continuously, it might happen that during a first write only a part of one stripe is written. This means, the RAID code has to read the complete stripe from disk, update it, and write it completely again. If a second write comes in for another part of the same stripe, all this would have to be done again. Now, if the cache is used and still contains the data written by the first write, the read that was necessary before the second write can be omitted.
Now it seems that dm-crypt always writes with small block size to the underlying disk, even when I write with a big block size. Could this be true?
Could this perhaps be improved? While I have found a solution for me, this could probably solve performance problems for many people.Furthermore, dm-crypt write is still slower than unencrypted write, although for reads the performance of encrypted and unencrypted are the same. So I guess the small block size still has a performance penalty (probably when first writing to a stripe and it is not yet in cache).
My current setup is Ubuntu 12.04 (Linux 3.2).
Nothing else has changed compared to when I first asked about this.Greetings, Philipp
_______________________________________________
dm-crypt mailing list
[email protected]
http://www.saout.de/mailman/listinfo/dm-crypt
Signature · 6 lines
Arno Wagner, Dr. sc. techn., Dipl. Inform., Email: [email protected] GnuPG: ID: CB5D9718 FP: 12D6 C03B 1B30 33BB 13CF B774 E35C 5FA1 CB5D 9718 ---- One of the painful things about our time is that those who feel certainty are stupid, and those with any imagination and understanding are filled with doubt and indecision. -- Bertrand Russell
Hi,
Am 05.02.2013 21:45, schrieb Arno Wagner:
Quoted 4 lines
thanks for the info. I think I may add a "performance" section to the FAQ, and this could be one of the items. Do you have the reference where you found the info on the
stripe_cache_size?
Actually, I first read about the concept of a stripe cache in this btrfs announcement post:
http://article.gmane.org/gmane.comp.file-systems.btrfs/23006
Then I googled it and found some usages:
http://h3x.no/2011/07/09/tuning-ubuntu-mdadm-raid56
http://ubuntuforums.org/showthread.php?t=1494846
Unfortunately, I didn't find a real citable source.
The kernel documentation (Documentation/md.txt) also only contains 3 lines saying nothing about what it actually is.
Greetings, Philipp
Quoted 54 lines · Philipp Wendler
Quoted 52 lines
Hello,
Am 05.07.2011 08:45, schrieb Philipp Wendler:
Quoted 6 lines
I have set up a Linux software RAID5 on three hard drives and want to encrypt it with cryptsetup/LUKS. My tests showed that the encryption leads to a massive performance decrease that I cannot explain.
The RAID5 is able to write 187 MB/s [1] without encryption. With encryption on top of it, write speed is down to about 40 MB/s.
Sorry for answering to a mail in this very old thread,
but it seems I finally found a solution,
and I know that there are some other people interested in the solution as well (so if you got this email directly from me, I BCC'ed you because you contacted me about this).
If you want to read up the full story, here's the link:
http://www.saout.de/pipermail/dm-crypt/2011-July/001773.htmlToday I read about the
stripe_cache_sizesetting of md RAID, and tried it out. With the default value of 256, the performance is slow as described. With a value of 4096, I get a performance increase from about 40-50 MB/s to 123 MB/s. For values >= 8192, I get 140 MB/s out of it.Background: The stripe cache stores recently written blocks. If data is written continuously, it might happen that during a first write only a part of one stripe is written. This means, the RAID code has to read the complete stripe from disk, update it, and write it completely again. If a second write comes in for another part of the same stripe, all this would have to be done again. Now, if the cache is used and still contains the data written by the first write, the read that was necessary before the second write can be omitted.
Now it seems that dm-crypt always writes with small block size to the underlying disk, even when I write with a big block size. Could this be true?
Could this perhaps be improved? While I have found a solution for me, this could probably solve performance problems for many people.Furthermore, dm-crypt write is still slower than unencrypted write, although for reads the performance of encrypted and unencrypted are the same. So I guess the small block size still has a performance penalty (probably when first writing to a stripe and it is not yet in cache).
My current setup is Ubuntu 12.04 (Linux 3.2).
Nothing else has changed compared to when I first asked about this.Greetings, Philipp
_______________________________________________
dm-crypt mailing list
[email protected]
http://www.saout.de/mailman/listinfo/dm-crypt
Hi Philipp,
thanks, if that explains it reasonably well, I will just paraphrase. But first I have to understand it myself ;-)
Arno
Quoted 81 lines · Philipp Wendler
Hi,
Am 05.02.2013 21:45, schrieb Arno Wagner:
Quoted 4 lines
thanks for the info. I think I may add a "performance" section to the FAQ, and this could be one of the items. Do you have the reference where you found the info on the
stripe_cache_size?Actually, I first read about the concept of a stripe cache in this btrfs announcement post:
http://article.gmane.org/gmane.comp.file-systems.btrfs/23006
Then I googled it and found some usages:
http://h3x.no/2011/07/09/tuning-ubuntu-mdadm-raid56
http://ubuntuforums.org/showthread.php?t=1494846Unfortunately, I didn't find a real citable source.
The kernel documentation (Documentation/md.txt) also only contains 3 lines saying nothing about what it actually is.Greetings, Philipp
Quoted 45 lines · Philipp Wendler
Quoted 44 lines
Hello,
Am 05.07.2011 08:45, schrieb Philipp Wendler:
Quoted 6 lines
I have set up a Linux software RAID5 on three hard drives and want to encrypt it with cryptsetup/LUKS. My tests showed that the encryption leads to a massive performance decrease that I cannot explain.
The RAID5 is able to write 187 MB/s [1] without encryption. With encryption on top of it, write speed is down to about 40 MB/s.
Sorry for answering to a mail in this very old thread,
but it seems I finally found a solution,
and I know that there are some other people interested in the solution as well (so if you got this email directly from me, I BCC'ed you because you contacted me about this).
If you want to read up the full story, here's the link:
http://www.saout.de/pipermail/dm-crypt/2011-July/001773.htmlToday I read about the
stripe_cache_sizesetting of md RAID, and tried it out. With the default value of 256, the performance is slow as described. With a value of 4096, I get a performance increase from about 40-50 MB/s to 123 MB/s. For values >= 8192, I get 140 MB/s out of it.Background: The stripe cache stores recently written blocks. If data is written continuously, it might happen that during a first write only a part of one stripe is written. This means, the RAID code has to read the complete stripe from disk, update it, and write it completely again. If a second write comes in for another part of the same stripe, all this would have to be done again. Now, if the cache is used and still contains the data written by the first write, the read that was necessary before the second write can be omitted.
Now it seems that dm-crypt always writes with small block size to the underlying disk, even when I write with a big block size. Could this be true?
Could this perhaps be improved? While I have found a solution for me, this could probably solve performance problems for many people.Furthermore, dm-crypt write is still slower than unencrypted write, although for reads the performance of encrypted and unencrypted are the same. So I guess the small block size still has a performance penalty (probably when first writing to a stripe and it is not yet in cache).
Quoted 9 lines
Quoted 8 lines
My current setup is Ubuntu 12.04 (Linux 3.2).
Nothing else has changed compared to when I first asked about this.Greetings, Philipp
_______________________________________________
dm-crypt mailing list
[email protected]
http://www.saout.de/mailman/listinfo/dm-crypt_______________________________________________
dm-crypt mailing list
[email protected]
http://www.saout.de/mailman/listinfo/dm-crypt
Signature · 6 lines
Arno Wagner, Dr. sc. techn., Dipl. Inform., Email: [email protected] GnuPG: ID: CB5D9718 FP: 12D6 C03B 1B30 33BB 13CF B774 E35C 5FA1 CB5D 9718 ---- One of the painful things about our time is that those who feel certainty are stupid, and those with any imagination and understanding are filled with doubt and indecision. -- Bertrand Russell