dm-crypt on raid 5 - having all disks on single controller makes dm-crypt slower
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
(please CC me, I'm not subscribed to the list)
Hello,
I've discovered some strange behaviour on my Debian system, with dm-crypt on raid 5 showing slow performance and I am seeking ways to remedy that.
My setup:
Gigabyte EX-58UDR3 motherboeard with an Intel Corei7 920 (no AES-NI) Intel X58 chipset
Debian sid/unstable
kernels tried: 3.12.9, 3.13.4 (Debian kernels)
4 x 3 TB SATA disks on a Intel X58 AHCI controller.
That controller has 6 SATA ports and connects directly to the chipsets South Bridge, which connects to the North Bridge via a very fast DMI link. Second PCIe-SATA controller on motherboard (ACHI driver, probably JMicron chipset) - 2 SATA ports.
On that disk I have a RAID5 array (source media are the 4th partition on each disk) called /dev/md/luks. On top of that RAID5 I've created a LUKS volume. That LUKS volume is a PV for LVM.
4 disks => RAID5 => LUKS Volume => LVM PV
Doing tests with hdparm -t, the RAID5 arrays shows ca. 520MB/s read speed. Each single disk has about 170MB/s with hdparm, so that is ok.
However, hdparm -t /dev/mapper/encrypted_raid5 gets me only ca. 150MB/s. Performing dd tests on the mounted LVM filesystems gets me about 160MB/s.
Now I connected two of the four disks to the second SATA controller
hdparm for those disks dropped to ca. 130MB/s, and hdparm -t for the RAID5 array dropped to ca. 400MB/s. That was to be expected, as that second controller seems somewhat slower.
And now comes the funny part:
Now I did a hdparm -t /dev/mapper/encrypted_raid5
and that got me ca. 300MB/s
Running top whilst doing the hdparm tests showed me that, when all 4 disks were on the Intel X58 AHCI controller, only 1 kworker thread was running on a single CPU core, utilizing 100% CPU time.
When 2 of those disks were connected to the second SATA controller, 2 kworker threads were running on two different CPU cores.
It seems that when the disks sit on two different controllers, dm-crypt is using 2 kworker threads to handle the data, but uses only one kworker thread when all disks sit on a single controller.
My guess is that one single cpu core can decrypt/encrypt the LUKS volume at about 150MB/s, which seems to be reasonable for that kind of CPU. Thus, dm-crypt using more than one kworker threads leads to doubling of the decryption speed, even if the two disks are slightly slower when connected to the second controller.
I'd like dm-crypt to use as many threads as possible (e.g. one kworker for each disk in the RAID5 array). That would give me reasonable speeds for the encrypted LUKS volume, close to that of the unencrypted RAID5 volume.
I've experimented with the RAID5 arrays group_thread_cnt sysfs setting. That is set to zero by default, disabling multithreaded RAID5. However, enabling that and setting it to 2 or 4 did not change anything speed-wise.
I haven't tried to fiddle with the raid5's or luks_volume's rq_affinity setting, yet. Would that help?
Is there a way to enforce dm-crypt using more than one kworker thread to encrypt/decrypt data for a crypt-on-raid5 setup?
Regards,
Dominik
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQIcBAEBCgAGBQJTDND9AAoJEH57oErWeAO1IHgP/jghJGdRS/mWZ0SU3goeWSB3 0Bii9xRb0Eo1tPS+TTXRad308pCL8BXloNdgLJXL9T1P/TLOaX0e7QxVRCC3j6Uj siBZ7rMBp60bHQu6JNUyXPwssgQhyRhL7Web2tHm7RQYDUoNaVdIjL0p4KvCRAX6 4RE2FzMEze/DgumZtXYgSlBBxVb9Za87Y/iwH+SJQ87kloKPkD7kgX2hByYJ5D7h NswCns1slj5KYyOSqikcoJM1Z9Sd1eJjMhRJ37/hH5ERMiP7m+LMO07ZsxtPCdE0 Fyi1J6Ra28YzKJPTcaBSa2oy+zeauennCIgLbA2px+7t3zEoaUiMSg4WrFbrV8i4 x1WWiHHrZV23dza2Hgy4BcE1lXAGxEKCuOSZr0Js/b20DfCeoV84oc4GIlMo3ZL4 LztRJ4yrtOUehH9KUwTqFLFk3r9xWDMC3ZlrbyyoxVAB5Nr+w1FnAzQRnvC7DOHb wU2lj9zWloNT8hUsCwtdpLxQWSj9iWw1j70xuqSUGIP3SLBgBQdUfpZP0Xwwe0Bi 577yt6CbQcaqGgjhtlgetwL9e8Vj8rMXbKZUKbyimimwmHEAJ++pdOWnBTdz7NOQ IKXP29mbI7+uIrzcgQvH6o5NS6psxdIcuxjfgYoR8CLPcNB87xKDcMXF9hU8JkZA fVE8Z2X5LPDksJrs3eb4
=5Mww
-----END PGP SIGNATURE-----
"C. Dominik Bódi":
Quoted 23 lines
(please CC me, I'm not subscribed to the list)
Hello,
I've discovered some strange behaviour on my Debian system, with dm-crypt on raid 5 showing slow performance and I am seeking ways to remedy that.
My setup:
Gigabyte EX-58UDR3 motherboeard with an Intel Corei7 920 (no AES-NI) Intel X58 chipset
Debian sid/unstable
kernels tried: 3.12.9, 3.13.4 (Debian kernels)
4 x 3 TB SATA disks on a Intel X58 AHCI controller.
That controller has 6 SATA ports and connects directly to the chipsets South Bridge, which connects to the North Bridge via a very fast DMI link. Second PCIe-SATA controller on motherboard (ACHI driver, probably JMicron chipset) - 2 SATA ports.On that disk I have a RAID5 array (source media are the 4th partition on each disk) called
/dev/md/luks. On top of that RAID5 I've created a LUKS volume. That LUKS volume is a PV for LVM.
Quoted 7 lines
4 disks => RAID5 => LUKS Volume => LVM PV
Doing tests with hdparm -t, the RAID5 arrays shows ca. 520MB/s read speed. Each single disk has about 170MB/s with hdparm, so that is ok.
However, hdparm -t
/dev/mapper/encrypted_raid5gets me only ca. 150MB/s. Performing dd tests on the mounted LVM filesystems gets me about 160MB/s.
*only* 150 ?
there's lots of numbers here but maybe we should work backwards and KIS ask yourself what do you want to achieve ?
when you know the results you want and/or need (2 different things) you can work backwards & figure out out how to get there
do you *need* to have a certain read/write speed for your application or desired conclusion in mind ?
do you have a recent version of cryptsetup with the benchmark option ? that can give you an idea of in memory performance and depends on your CPU & chosen ciphers/hashes
i didn't quite get if your RAID is a hardware or software array ?
Quoted 15 lines
Now I connected two of the four disks to the second SATA controller
hdparm for those disks dropped to ca. 130MB/s, and hdparm -t for the RAID5 array dropped to ca. 400MB/s. That was to be expected, as that second controller seems somewhat slower.
And now comes the funny part:
Now I did a hdparm -t/dev/mapper/encrypted_raid5
and that got me ca. 300MB/sRunning top whilst doing the hdparm tests showed me that, when all 4 disks were on the Intel X58 AHCI controller, only 1 kworker thread was running on a single CPU core, utilizing 100% CPU time.
Quoted 6 lines
When 2 of those disks were connected to the second SATA controller, 2 kworker threads were running on two different CPU cores.
It seems that when the disks sit on two different controllers, dm-crypt is using 2 kworker threads to handle the data, but uses only one kworker thread when all disks sit on a single controller.
i'll leave that to the experts ;-)
Quoted 29 lines
My guess is that one single cpu core can decrypt/encrypt the LUKS volume at about 150MB/s, which seems to be reasonable for that kind of CPU. Thus, dm-crypt using more than one kworker threads leads to doubling of the decryption speed, even if the two disks are slightly slower when connected to the second controller.
I'd like dm-crypt to use as many threads as possible (e.g. one kworker for each disk in the RAID5 array). That would give me reasonable speeds for the encrypted LUKS volume, close to that of the unencrypted RAID5 volume.
I've experimented with the RAID5 arrays
group_thread_cntsysfs setting. That is set to zero by default, disabling multithreaded RAID5. However, enabling that and setting it to 2 or 4 did not change anything speed-wise.I haven't tried to fiddle with the raid5's or luks_volume's
rq_affinitysetting, yet. Would that help?Is there a way to enforce dm-crypt using more than one kworker thread to encrypt/decrypt data for a crypt-on-raid5 setup?
Regards,
Dominik
_______________________________________________
dm-crypt mailing list
[email protected]
http://www.saout.de/mailman/listinfo/dm-crypt
Am 26.02.2014 00:58, schrieb [email protected]:
Quoted 7 lines
*only* 150 ?
there's lots of numbers here but maybe we should work backwards and KIS ask yourself what do you want to achieve ?
when you know the results you want and/or need (2 different things) you can work backwards & figure out out how to get there
do you *need* to have a certain read/write speed for your application or desired conclusion in mind ?
Well, yes, I'd like to have as much throughput as possible. The cpu is a quad core with HT, so could dm-crypt not utilize 4 threads when there are 4 disks in the raid5 array.
An the thing is:
4 disks on controller A => dm-crypt 150MB/s
2 disk on controller A, 2 disks on controller B => dm-crypt 300MB/s
I don't understand that behaviour. Why does it make a difference for dm-crypt if the disks sit on different controllers or not? Why should I waste speed when dm-crypt could easily utilize 4 worker threads and max out the raid5 performance? Why doesn't it do so automatically. Is there any means to make dm-crypt use 4 threads even if all 4 drives sit on a single controller?
I know the simple answer would be to reverse the stack, encrypt each raid drive singularly and then put the raid5 array on top of the encrypted drives. That makes it really difficult to unlock the drives remotely (via dropbear in the initramfs), though.
Quoted 3 lines
do you have a recent version of cryptsetup with the benchmark option ? that can give you an idea of in memory performance and depends on your CPU & chosen ciphers/hashes
I'm using aes-xts-plain64 with 256 key size. The benchmark shows me a throughput of 175 MB/s. I reckon the benchmark only runs a single thread, so that seems to be ok.
Quoted 1 line
i didn't quite get if your RAID is a hardware or software array ?
Hmmm, sorry for that oversight. Yes, it is a software raid5 array set up with mdadm.
Quoted 1 line
i'll leave that to the experts ;-)
Yes, indeed, I was looking for an answer from someone that knows the dm-crypt kernel code quite well.
Regards,
Dominik
My best guess here is this:
There is one work queue (IO QUEUE) per controller. Which in turn means that dm-crypt is fed by two queues and as such decides to have one kworker instance per queue. I might be completely wrong though. (Actually there's still mdraid, which seems to have accordingly 1/2 queues).
Regards
-Sven
Quoted 24 lines · C. Dominik Bódi
Am 26.02.2014 00:58, schrieb [email protected]:
Quoted 7 lines
*only* 150 ?
there's lots of numbers here but maybe we should work backwards and KIS ask yourself what do you want to achieve ?
when you know the results you want and/or need (2 different things) you can work backwards & figure out out how to get there
do you *need* to have a certain read/write speed for your application or desired conclusion in mind ?Well, yes, I'd like to have as much throughput as possible. The cpu is a quad core with HT, so could dm-crypt not utilize 4 threads when there are 4 disks in the raid5 array.
An the thing is:
4 disks on controller A => dm-crypt 150MB/s
2 disk on controller A, 2 disks on controller B => dm-crypt 300MB/sI don't understand that behaviour. Why does it make a difference for dm-crypt if the disks sit on different controllers or not? Why should I waste speed when dm-crypt could easily utilize 4 worker threads and max out the raid5 performance? Why doesn't it do so automatically. Is there any means to make dm-crypt use 4 threads even if all 4 drives sit on a single controller?
Just on a meta-layer: This is a known issue, but it seems the fix is not quite clear. If you are really concerned, put the encryption below the RAID, but that has other issues.
Arno
Quoted 43 lines · Sven Eschenberg
My best guess here is this:
There is one work queue (IO QUEUE) per controller. Which in turn means that dm-crypt is fed by two queues and as such decides to have one kworker instance per queue. I might be completely wrong though. (Actually there's still mdraid, which seems to have accordingly 1/2 queues).
Regards
-Sven
Quoted 17 lines · C. Dominik Bódi
Am 26.02.2014 00:58, schrieb [email protected]:
Quoted 7 lines
*only* 150 ?
there's lots of numbers here but maybe we should work backwards and KIS ask yourself what do you want to achieve ?
when you know the results you want and/or need (2 different things) you can work backwards & figure out out how to get there
do you *need* to have a certain read/write speed for your application or desired conclusion in mind ?Well, yes, I'd like to have as much throughput as possible. The cpu is a quad core with HT, so could dm-crypt not utilize 4 threads when there are 4 disks in the raid5 array.
An the thing is:
4 disks on controller A => dm-crypt 150MB/s
2 disk on controller A, 2 disks on controller B => dm-crypt 300MB/sQuoted 7 lines
I don't understand that behaviour. Why does it make a difference for dm-crypt if the disks sit on different controllers or not? Why should I waste speed when dm-crypt could easily utilize 4 worker threads and max out the raid5 performance? Why doesn't it do so automatically. Is there any means to make dm-crypt use 4 threads even if all 4 drives sit on a single controller?
_______________________________________________
dm-crypt mailing list
[email protected]
http://www.saout.de/mailman/listinfo/dm-crypt
Signature · 4 lines
Arno Wagner, Dr. sc. techn., Dipl. Inform., Email: [email protected] GnuPG: ID: CB5D9718 FP: 12D6 C03B 1B30 33BB 13CF B774 E35C 5FA1 CB5D 9718 ---- A good decision is based on knowledge and not on numbers. - Plato