dm-crypt multithreaded yet?
Hi all.
Sorry for what is probably a very tiresome topic here, but just trying to find out what the current status is...
I thought I knew the answer to this question (dm-crypt multithreaded yet? no) but I had read in several places that dm-crypt had gained multithreaded capabilities. I'm still skeptical though. I certainly don't see it on Fedora 17's 3.3.4-5.fc17.x86_64 kernel.
Sadly, I don't have an AES-NI capable CPU, but I do have 4 (real, 8 with HT enabled) cores and dm-crypting on my SSD reduces its performance down from about 260MB/s (native) to 95MB/s. :-( I would like to think I could get more performance out of it if more of the (usually available anyway) CPU could be thrown at it.
Is this something being worked on currently?
I have taken note of
http://people.redhat.com/mpatocka/patches/kernel/dm-crypt-paralelizace/current/
but that work seems to be going back to Feb/2012. And TBH, I'm not entirely clear if that work is aimed at multi-threaded dm-crypt or not.
Cheers,
b.
Am 08.08.2012 16:29, schrieb Brian J. Murrell:
Quoted 10 lines
I thought I knew the answer to this question (dm-crypt multithreaded yet? no) but I had read in several places that dm-crypt had gained multithreaded capabilities. I'm still skeptical though. I certainly don't see it on Fedora 17's 3.3.4-5.fc17.x86_64 kernel.
Sadly, I don't have an AES-NI capable CPU, but I do have 4 (real, 8 with HT enabled) cores and dm-crypting on my SSD reduces its performance down from about 260MB/s (native) to 95MB/s. :-( I would like to think I could get more performance out of it if more of the (usually available anyway) CPU could be thrown at it.
This is in 2.6.38 and later, I think:
The commit message also explains why you don't notice it.
Quoted 1 line · Thomas Bächler
This is in 2.6.38 and later, I think:
No, it is not. This is something different.
In current kernel is per-cpu changes but it will not use parallel cpu cores if it is submitted from the same cpu.
Mikulas' patches tries to create parallel threads and run operations on all cores (plus some other changes as sorting).
Milan
Am 08.08.2012 16:29, schrieb Brian J. Murrell:
Quoted 10 lines
I thought I knew the answer to this question (dm-crypt multithreaded yet? no) but I had read in several places that dm-crypt had gained multithreaded capabilities. I'm still skeptical though. I certainly don't see it on Fedora 17's 3.3.4-5.fc17.x86_64 kernel.
Sadly, I don't have an AES-NI capable CPU, but I do have 4 (real, 8 with HT enabled) cores and dm-crypting on my SSD reduces its performance down from about 260MB/s (native) to 95MB/s. :-( I would like to think I could get more performance out of it if more of the (usually available anyway) CPU could be thrown at it.
If that helps, my 500MB/s SSD is giving me 270MB/s when reading an encrypted file (i.e. going through the block and the filesystem layer) on my laptop:
Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ Quoted 4 lines · Marc MERLIN
If that helps, my 500MB/s SSD is giving me 270MB/s when reading an encrypted file (i.e. going through the block and the filesystem layer) on my laptop:
Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz
That would be AES-NI giving you that throughput. Unfortunately not all of us have AES-NI (and nor can we just throw away an entire laptop just to get it) but do have lots of idle cores sitting around that could help out.
b.
Quoted 1 line · Brian J. Murrell
Is this something being worked on currently?
Sort of. I have no idea what are upstream DM priorities now.
Anyway, I have patches rebased for 3.5/3.6-rc here
http://mbroz.fedorapeople.org/dm-crypt/
If you can test it and prove it helps for your use case, its would be nice. Just take it as experimental, there can be bugs still.
Quoted 2 lines
but that work seems to be going back to Feb/2012. And TBH, I'm not entirely clear if that work is aimed at multi-threaded dm-crypt or not.
Should be. Search archive of this list as well.
Milan
Quoted 3 lines · Milan Broz
Anyway, I have patches rebased for 3.5/3.6-rc here
http://mbroz.fedorapeople.org/dm-crypt/
Unfortunately the files in http://mbroz.fedorapeople.org/dm-crypt/3.6-rc
all return 403 trying to fetch them.
Quoted 2 lines
If you can test it and prove it helps for your use case, its would be nice. Just take it as experimental, there can be bugs still.
I was going to try this on an FC18 kernel. Are you using these patches by any chance?
Cheers,
b.
Quoted 7 lines · Brian J. Murrell
Quoted 3 lines · Milan Broz
Anyway, I have patches rebased for 3.5/3.6-rc here
http://mbroz.fedorapeople.org/dm-crypt/Unfortunately the files in http://mbroz.fedorapeople.org/dm-crypt/3.6-rc
all return 403 trying to fetch them.
No idea why it turned world non-readable, fixed.
Quoted 5 lines
Quoted 2 lines
If you can test it and prove it helps for your use case, its would be nice. Just take it as experimental, there can be bugs still.
I was going to try this on an FC18 kernel. Are you using these patches by any chance?
But I am afraid I did not find any real situation where these patches really helps (AND not cause problems with different pattern). See also discussion on dm-devel list, I still think generic kernel workqueues should be used to provide such parallelization and we should not duplicate code in dm-crypt...
My suggestion is that using AES-NI extension helps much more with the current upstream code than anything else (for AES, obviously).
Milan
Quoted 2 lines · Milan Broz
My suggestion is that using AES-NI extension helps much more with the current upstream code than anything else (for AES, obviously).
And not to forget the optimized modules in the kernel. Here's an example with a WD Black Caviar using serpent encryption on a test machine.
This is stock cryptsetup. Wouldn't have had more speed without encryption..
[htd@test2 ~]$ cat /proc/cpuinfo
model name : Intel(R) Core(TM) i5 CPU M 450 @ 2.40GHz
[root@test2 ~]# hdparm -i /dev/sda | grep Model
Model=WDC WD7500BPKT-00PK4T0, FwRev=01.01A01, SerialNo=WD-XXXXXXXXX
[root@test2 ~]# cryptsetup luksDump /dev/sda1 LUKS header information for /dev/sda1
Version: 1 Cipher name: serpent Cipher mode: xts-plain64:whirlpool Hash spec: whirlpool
Payload offset: 4096
MK bits: 512
[....]
[root@test2 mapper]# hdparm -t /dev/mapper/luks-xxxxxxxxx
/dev/mapper/luks-xxxxxxxxxxxx:
Timing buffered disk reads: 386 MB in 3.01 seconds = 128.27 MB/sec
Modules:
serpent_sse2_x86_64 50364 6 serpent_generic 25563 1 serpent_sse2_x86_64 lrw 13145 1 serpent_sse2_x86_64 glue_helper 13172 1 serpent_sse2_x86_64 ablk_helper 13269 1 serpent_sse2_x86_64 xts 12871 1 serpent_sse2_x86_64 gf128mul 14333 2 lrw,xts dm_crypt 22893 3
Quoted 2 lines · Milan Broz
No idea why it turned world non-readable, fixed.
Thanks.
Quoted 2 lines
But I am afraid I did not find any real situation where these patches really helps
If successive block I/Os from a sequential I/O can all get queued up by different CPU (cores) I can't imagine why that wouldn't help.
Quoted 3 lines
See also discussion on dm-devel list, I still think generic kernel workqueues should be used to provide such parallelization and we should not duplicate code in dm-crypt...
Well, definitely, if the parallelization work is duplicating existing kernel functionality, that needs to be factored out. But regardless...
Quoted 2 lines
My suggestion is that using AES-NI extension helps much more with the current upstream code than anything else (for AES, obviously).
But of course. I am sure anyone that has a CPU with AES-NI instructions wouldn't care so much (at all even) about multi-core dm-crypt. But unfortunately not everyone has AES-NI capable processors, yet they might have 4 cores (8 if HT enabled) available, with 3 (7, again if HT is enabled) all sitting around doing nothing while one is grunting away with all of the workload.
I wish I could just trade in my laptop for one with AES-NI, but other than the lack of AES-NI it's far better than what I would get in return for trading it in. Kind of like trading in one's Ferrari just because it doesn't have a leather steering-wheel for a Fiat that does have a leather steering wheel. :-/
Cheers,
b.