aes-xts-plain with aes_x86_64 makes my SSD 5x slower than my encrypted HD
I got a new Samsumg 830 512GB SSD which is supposed to be very high performance.
The raw device seems fast enough on a quick hdparm test:/dev/sda4:
Timing cached reads: 14258 MB in 2.00 seconds = 7136.70 MB/sec
Timing buffered disk reads: 1392 MB in 3.00 seconds = 463.45 MB/sec <<<<
which is 4x faster than my non encrypted spinning disk, as expected.
But once I encrypt it, it drops to 5 times slower than my 1TB spinning disk in the same laptop:
gandalfthegreat:~# hdparm -tT /dev/mapper/ssdcrypt /dev/mapper/ssdcrypt: Timing cached reads: 15412 MB in 2.00 seconds = 7715.37 MB/sec Timing buffered disk reads: 70 MB in 3.06 seconds = 22.91 MB/sec <<<<
gandalfthegreat:~# hdparm -tT /dev/mapper/cryptroot (spinning disk) /dev/mapper/cryptroot: Timing cached reads: 16222 MB in 2.00 seconds = 8121.03 MB/sec Timing buffered disk reads: 308 MB in 3.01 seconds = 102.24 MB/sec <<<<
I used aes-xts-plain as recommended on
http://www.mayrhofer.eu.org/ssd-linux-benchmark
gandalfthegreat:~# cryptsetup status /dev/mapper/ssdcrypt /dev/mapper/ssdcrypt is active. type: LUKS1 cipher: aes-xts-plain keysize: 256 bits device: /dev/sda4 offset: 4096 sectors size: 926308752 sectors mode: read/write
I tried
cryptsetup luksFormat --align-payload=8192
the first time, so my offset was 8K, but that did not make a difference in speed.
gandalfthegreat:~# lsmod |grep -e aes aesni_intel 50443 66 cryptd 14517 18 ghash_clmulni_intel,aesni_intel aes_x86_64 16796 1 aesni_intel
Kernel: 3.4.4-amd64
gandalfthegreat:~# cryptsetup --version cryptsetup 1.4.3
I know that SSDs are weird and all, but getting a raw device speed of a mere 23MB/sec down from 463MB/s and compared to 102MB/s for a similarly spinning drive, is a problem, is it not?
Any suggestions would be appreciated.
Thanks,
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/Quoted 7 lines · Marc MERLIN
I know that SSDs are weird and all, but getting a raw device speed of a
mere 23MB/sec down from 463MB/s and compared to 102MB/s for a similarly
spinning drive, is a problem, is it not?Any suggestions would be appreciated.
I'm using Debian sid (so still at 3.2 kernel), currently using a 256G Samsung SSD. What I get is:
root@scapa:~# hdparm -t /dev/sda /dev/mapper/scapa_crypt
/dev/sda:
Timing buffered disk reads: 690 MB in 3.01 seconds = 229.47 MB/sec
/dev/mapper/scapa_crypt:
Timing buffered disk reads: 590 MB in 3.00 seconds = 196.43 MB/sec
root@scapa:~# cryptsetup status /dev/mapper/scapa_crypt /dev/mapper/scapa_crypt is active and is in use. type: LUKS1 cipher: aes-xts-plain keysize: 256 bits device: /dev/sda2 offset: 4096 sectors size: 499587760 sectors mode: read/write flags: discards
Regards,
Signature · 1 line
Yves-Alexis
Quoted 1 line · Marc MERLIN
Timing buffered disk reads: 70 MB in 3.06 seconds = 22.91 MB/sec <<<<
I don't know why reading speed is that slow in your case, especially as you are using AES-NI, which should give you the highest speed available. Maybe others here on the list have a suggestion. Probably, you should provide some more information.
Otherwise, on the newer Intel i3/i5/i7, twofish-3way is faster than AES. You could try to re-format your drive with twofish-xts-plain64 and adding twofish_common, twofish_x86_64 and twofish_x86_64_3way to your initram (as long as your kernel is built with these enabled).
Quoted 5 lines · Yves-Alexis Perez
Quoted 1 line
Any suggestions would be appreciated.
I'm using Debian sid (so still at 3.2 kernel), currently using a 256G Samsung SSD. What I get is:
Quoted 7 lines
root@scapa:~# hdparm -t /dev/sda /dev/mapper/scapa_crypt
/dev/sda:
Timing buffered disk reads: 690 MB in 3.01 seconds = 229.47 MB/sec
/dev/mapper/scapa_crypt:
Timing buffered disk reads: 590 MB in 3.00 seconds = 196.43 MB/sec
Right, that's more what I would expect.
Quoted 4 lines · Heinz Diehl
I don't know why reading speed is that slow in your case, especially as you are using AES-NI, which should give you the highest speed available. Maybe others here on the list have a suggestion. Probably, you should provide some more information.
I'm happy to if you tell me what else I can give.
Note how my test showed that I had the same aes-xts-plain on the hard drive in the same laptop, and that one runs at 100MB/s, which is pretty much its native speed.
Just to make sure that I'm not CPU limited, I tried
gandalfthegreat:~# dd if=/dev/sda4 of=/dev/null bs=1M count=1024 1073741824 bytes (1.1 GB) copied, 2.23959 s, 479 MB/s
gandalfthegreat:~# dd if=/dev/mapper/ssdcrypt of=/dev/null bs=1M count=1024 1073741824 bytes (1.1 GB) copied, 44.3302 s, 24.2 MB/s
atop shows dd isn't really pegging a single core:
THR SYSCPU USRCPU RDDSK WRDSK ST EXC S CPUNR CPU CMD 1 0.60s 0.01s 226.2M 0K -- - D 3 6% dd
Quoted 4 lines
Otherwise, on the newer Intel i3/i5/i7, twofish-3way is faster than AES. You could try to re-format your drive with twofish-xts-plain64 and adding
twofish_common,twofish_x86_64andtwofish_x86_64_3wayto your initram (as long as your kernel is built with these enabled).
Thanks for the hint.
Since it's a lenovo T530 with a recent CPU, I tried it:
gandalfthegreat:~# cryptsetup -c twofish-xts-plain64 -s 256 luksFormat /dev/sda4
Result is the same:
gandalfthegreat:~# hdparm -tT /dev/mapper/ssdcrypt
/dev/mapper/ssdcrypt:
Timing cached reads: 16614 MB in 2.00 seconds = 8317.50 MB/sec
Timing buffered disk reads: 68 MB in 3.08 seconds = 22.09 MB/sec
Grumble.
It shouldn't be a hardware problem since I do see 400MB/s before encryption.
I'm a bit lost here. I'll try other kernels just in case, but that shouldn't make a difference.
Thanks for the answers,
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ Quoted 5 lines · Marc MERLIN
Quoted 4 lines · Yves-Alexis Perez
Quoted 1 line
Any suggestions would be appreciated.
I'm using Debian sid (so still at 3.2 kernel), currently using a 256G Samsung SSD. What I get is:
SID? That would be "unstable", whit possible assorted problems.
[...]
Quoted 6 lines
gandalfthegreat:~# dd if=/dev/mapper/ssdcrypt of=/dev/null bs=1M count=1024 1073741824 bytes (1.1 GB) copied, 44.3302 s, 24.2 MB/satop shows dd isn't really pegging a single core:
THR SYSCPU USRCPU RDDSK WRDSK ST EXC S CPUNR CPU CMD 1 0.60s 0.01s 226.2M 0K -- - D 3 6% dd
It would not, as AES-NI (AFAIK) does need very little CPU assistance. AES-NI may be the problem though. Can you try with the normal AES module? I think unloading the AES-NI module may be enough for that, but I am not sure.
Maybe AES-NI needs very long for something it needs to do each sector. Google("aes-ni slow") found at least some indications that aes-ni may still have problems.
[...]
Quoted 4 lines
It shouldn't be a hardware problem since I do see 400MB/s before encryption.
I'm a bit lost here. I'll try other kernels just in case, but that shouldn't make a difference.
It could. I remember that some time ago, quite a few people had issues with slow crypto due to some problems in the device-mapper layers. You might have hit that.
Here is one benchmark from me, 1GB luks-file via /dev/loop, residing on a 3-way RAID1 with 2 HDDs as write-mostly and one older SSD. Kernel is 3.3.8, self-compiled on top of Debian squeeze, pure software AES on AMD quad-core. (Yes, I know this is complicated, but apparently works well, see below ;-)
/dev/mapper/x1:
Timing buffered disk reads: 612 MB in 3.00 seconds = 203.80 MB/sec
And the raw SSD:/dev/sdd:
Timing buffered disk reads: 618 MB in 3.01 seconds = 205.56 MB/sec
Arno
Signature · 6 lines
Arno Wagner, Dr. sc. techn., Dipl. Inform., Email: [email protected] GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F ---- One of the painful things about our time is that those who feel certainty are stupid, and those with any imagination and understanding are filled with doubt and indecision. -- Bertrand Russell
Quoted 8 lines · Arno Wagner
Quoted 5 lines · Marc MERLIN
Quoted 4 lines · Yves-Alexis Perez
Quoted 1 line
Any suggestions would be appreciated.
I'm using Debian sid (so still at 3.2 kernel), currently using a 256G Samsung SSD. What I get is:
SID? That would be "unstable", whit possible assorted problems.
*I* am running SID, not the original reporter. And I have pretty decent speed, thank you :)
Quoted 9 lines
[...]
Quoted 6 lines
gandalfthegreat:~# dd if=/dev/mapper/ssdcrypt of=/dev/null bs=1M count=1024 1073741824 bytes (1.1 GB) copied, 44.3302 s, 24.2 MB/satop shows dd isn't really pegging a single core:
THR SYSCPU USRCPU RDDSK WRDSK ST EXC S CPUNR CPU CMD 1 0.60s 0.01s 226.2M 0K -- - D 3 6% dd
Quoted 8 lines
It would not, as AES-NI (AFAIK) does need very little CPU assistance. AES-NI may be the problem though. Can you try with the normal AES module? I think unloading the AES-NI module may be enough for that, but I am not sure.
Maybe AES-NI needs very long for something it needs to do each sector.
Google("aes-ni slow")found at least some indications that aes-ni may still have problems.
And I do use aes-ni too.
Signature · 1 line
Yves-Alexis
Quoted 17 lines · Arno Wagner
Quoted 5 lines · Marc MERLIN
Quoted 4 lines · Yves-Alexis Perez
Quoted 1 line
Any suggestions would be appreciated.
I'm using Debian sid (so still at 3.2 kernel), currently using a 256G Samsung SSD. What I get is:
SID? That would be "unstable", whit possible assorted problems.
[...]
Quoted 6 lines
gandalfthegreat:~# dd if=/dev/mapper/ssdcrypt of=/dev/null bs=1M count=1024 1073741824 bytes (1.1 GB) copied, 44.3302 s, 24.2 MB/satop shows dd isn't really pegging a single core:
THR SYSCPU USRCPU RDDSK WRDSK ST EXC S CPUNR CPU CMD 1 0.60s 0.01s 226.2M 0K -- - D 3 6% dd
Quoted 8 lines
It would not, as AES-NI (AFAIK) does need very little CPU assistance. AES-NI may be the problem though. Can you try with the normal AES module? I think unloading the AES-NI module may be enough for that, but I am not sure.
Maybe AES-NI needs very long for something it needs to do each sector.
Google("aes-ni slow")found at least some indications that aes-ni may still have problems.
It was worth a shot, thanks for the suggestion.
gandalfthegreat:~# lsmod | grep aes aes_x86_64 16796 34 gandalfthegreat:~# hdparm -t -T /dev/mapper/cryptroot
/dev/mapper/cryptroot:
Timing cached reads: 15802 MB in 2.00 seconds = 7909.98 MB/sec
Timing buffered disk reads: 68 MB in 3.04 seconds = 22.39 MB/sec
gandalfthegreat:~#
Unfortunately, speed is exactly the same.
with aes_x86_64:
gandalfthegreat:/var/local/VirtualBox VMs/w2k_virtual# dd if=test of=/dev/null 4192640+0 records in
4192640+0 records out
2146631680 bytes (2.1 GB) copied, 12.4848 s, 172 MB/s
I then rebooted with aesni_intel and got this:
gandalfthegreat:/var/local/VirtualBox VMs/w2k_virtual# dd if=test of=/dev/null 4192640+0 records in
4192640+0 records out
2146631680 bytes (2.1 GB) copied, 8.42506 s, 255 MB/s
So those speeds are expected/believable, but I still don't know why hdparm is so slow on /dev/mapper/cryptroot.
Mmmh....
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/Quoted 9 lines · Marc MERLIN
Quoted 8 lines · Arno Wagner
Quoted 5 lines · Marc MERLIN
Quoted 4 lines · Yves-Alexis Perez
Quoted 1 line
Any suggestions would be appreciated.
I'm using Debian sid (so still at 3.2 kernel), currently using a 256G Samsung SSD. What I get is:
SID? That would be "unstable", whit possible assorted problems.
Ok, sorry for my confusion, what kernel/distro are you running?
[...]
Quoted 23 lines
Quoted 3 lines
Maybe AES-NI needs very long for something it needs to do each sector.
Google("aes-ni slow")found at least some indications that aes-ni may still have problems.It was worth a shot, thanks for the suggestion.
gandalfthegreat:~# lsmod | grep aes aes_x86_64 16796 34 gandalfthegreat:~# hdparm -t -T /dev/mapper/cryptroot
/dev/mapper/cryptroot:
Timing cached reads: 15802 MB in 2.00 seconds = 7909.98 MB/sec
Timing buffered disk reads: 68 MB in 3.04 seconds = 22.39 MB/sec
gandalfthegreat:~#Unfortunately, speed is exactly the same.
with
aes_x86_64:
gandalfthegreat:/var/local/VirtualBox VMs/w2k_virtual# dd if=test of=/dev/null 4192640+0 records in
4192640+0 records out
2146631680 bytes (2.1 GB) copied, 12.4848 s, 172 MB/s
Quoted 5 lines
I then rebooted with
aesni_inteland got this:
gandalfthegreat:/var/local/VirtualBox VMs/w2k_virtual# dd if=test of=/dev/null 4192640+0 records in
4192640+0 records out
2146631680 bytes (2.1 GB) copied, 8.42506 s, 255 MB/s
That does not seem to be it.
Quoted 4 lines
So those speeds are expected/believable, but I still don't know why hdparm is so slow on
/dev/mapper/cryptroot.Mmmh....
Indeed.
Arno
Signature · 6 lines
Arno Wagner, Dr. sc. techn., Dipl. Inform., Email: [email protected] GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F ---- One of the painful things about our time is that those who feel certainty are stupid, and those with any imagination and understanding are filled with doubt and indecision. -- Bertrand Russell
Quoted 2 lines
Quoted 2 lines
So those speeds are expected/believable, but I still don't know why hdparm is so slow on
/dev/mapper/cryptroot.
First, hdparm basically run 2M read in direct-io mode for test, all IOs are sumbitted from one process.
AES-NI helps a lot and because it is prioritised in crypto api, you usually using it without any additional config if hw supports it. (Also I see some patches whit run XTS blocks in parallel on crypto api list.)
But that's not the problem you are seeing now. (I will explain it on recent kernel code, note that RHEL5/6 have older code and dmcrypt behaves slightly differently when processing requests.)
Current in-kernel dmcrypt code tries to keep IO processing on CPU which submitted request. It means, that if one process generates all requests, all requests are usually processed by one core where it is this process scheduled not using other CPU cores.
(Saying usually because page cache can submit request on different cpu.)
So no wonder that you get slow operation - in dd/hdparm usually only one cpu is processing the request. If CPU is fast enough, no problem. If not you will see slowdown. AES-NI will speed up this on that cpu core, but will not help run request in parallel on multi-core systems.
In real fs use case, more applications submitting io requests, you will get much better throughput. Also if not using directio, page cache can help and fs are usually more clever too (writes are usually better in performance - try it).
I do not like this dmcrypt mode a we tried to fix it. There is a bunch of patches from Mikulas Patocka which switches parallelization to use all available
cpus (if not limited by paramater).
In my tests it improved performance in some cases but not in all situations
(there were some slow downs which scares me).
(You can see patches here http://people.redhat.com/mpatocka/patches/kernel/dm-crypt-paralelizace/)
Unfortunately discussion stopped and device-mapper maintainer forgot about it.
Well, your mails apparently caused some head-ups here, so I'll try to return to this. (Will post them to dm-devel directly this time.)
I think fast SSD case is exactly situation where it should help. We eill need some testers though :)
(I have currently no fast SSD available for testing.)
Milan
Quoted 5 lines · Milan Broz
So no wonder that you get slow operation - in dd/hdparm usually only one cpu is processing the request. If CPU is fast enough, no problem. If not you will see slowdown. AES-NI will speed up this on that cpu core,
but will not help run request in parallel on multi-core systems.
Then I should see slow operations too, since I'm doing exactly the same test. My guess is that is a kernel issue somewhere, but maybe we should try a common ground (say, a grml live or some fedora live) and report back?
Regards,
Signature · 1 line
Yves-Alexis
Quoted 11 lines · Yves-Alexis Perez
Quoted 5 lines · Milan Broz
So no wonder that you get slow operation - in dd/hdparm usually only one cpu is processing the request. If CPU is fast enough, no problem. If not you will see slowdown. AES-NI will speed up this on that cpu core,
but will not help run request in parallel on multi-core systems.Then I should see slow operations too, since I'm doing exactly the same test. My guess is that is a kernel issue somewhere, but maybe we should try a common ground (say, a grml live or some fedora live) and report back?
Well, yes. I explained dmcrypt part, there can be other problem. E.g. alignment (bit it _seems_ correct from mails).
Reading again, 23MB/s is really slow, so yes there must be something else.
Common distro env is nice but be sure you have proper crypto modules available. Also do not use Fedora rawhide, it has kernel compiled with debug tools not usable for testing.
You can try start with this:
1) (this should be not problem but better check it) Check alignment of partitions. Is it aligned to SSD page size? (Aligning to 1MiB is always correct ;-)
Paste fdisk -l -u /dev/sda
2) try to switch io scheduler to "noop" or "deadline" (paste lsblk -t)
You can do it online for sda (again, check with lsblk -t):
echo "noop">/sys/block/sda/queue/scheduler
Also you can try to increase queue size.
(Hard core version is to run blktrace and check if request are not split unnecessarily :)
3) Let's test cipher_null (no encryption just fake-copy) (you need cryptsetup 1.4.3 for this test).
create test LUKS device with null cipher: cryptsetup luksFormat -c null
Repeat tests now - is the problem the same? (please send output). (For dmcrypt device speed should be only slighly slower.)
please note: cipher null means no encryption, just use dmcrypt layer, so do not use for valid data :-)
4) which aes module are you using? check lsmod, check /proc/crypto
you should use either aes-ni or optimized modules (x86_64 etc)
Milan
p.s.
sorry for removing cc on first reply, my mistake.
I remember a test which I conducted after I received my Intel SDD 510 120 GB about 1 year ago. I don't remember all the details, but the essence was that write speed decreased dramatically if the write process wasn't aligned.
I did:
dd if=/dev/zero of=/dev/sda bs=4k
I got around 400 Mb/sec, which was expected. But when I did:
dd if=/dev/zero of=/dev/sda bs=5k
I got only around 40 Mb/sec
When you write in 5k blocks, the SSD has to do a lot of overhead because of the Read-Modify-Write operations. I guess this might also be the case if you write in 4k blocks, but don't align them to the SSD's "native" blocks. So I would check for any misalignment.
Regards,
André
Quoted 3 lines · Arno Wagner
Quoted 1 line
Quoted 1 line
SID? That would be "unstable", whit possible assorted problems.
Ok, sorry for my confusion, what kernel/distro are you running?
Debian testing with pieces of unstable :)
That gives me cryptsetup 1.4.3
(but debian unstable is often not more unstable than your released fedora core or ubuntu in my experience)
Hi Milan,
Thanks for the answer and all your questions.
Quoted 3 lines · Milan Broz
AES-NI helps a lot and because it is prioritised in crypto api, you usually using it without any additional config if hw supports it. (Also I see some patches whit run XTS blocks in parallel on crypto api list.)
Yes, the modules all worked perfectly and the correct one was prioritized.
Quoted 4 lines
So no wonder that you get slow operation - in dd/hdparm usually only one cpu is processing the request. If CPU is fast enough, no problem. If not you will see slowdown. AES-NI will speed up this on that cpu core, but will not help run request in parallel on multi-core systems.
Obviously, now that I've already verified that dd is slow while filesystem operations are almost 10x faster, you're obviously onto something here. But, I'm confused, why does atop show that dd is only using 6% CPU?
Oooh, it's not that my CPU is fully used, it's just that my CPU is able to decrypt as quickly as the data is coming in for a 100MB/s hard drive, but not a 500MB/s SSD and however scheduling works if the data is coming in faster than the CPU can decode those pages.
(editted: actually no, using 'null' encryptino still gives 25MB/s).
Quoted 11 lines
I do not like this dmcrypt mode a we tried to fix it. There is a bunch of patches from Mikulas Patocka which switches parallelization to use all available
cpus (if not limited by paramater).
In my tests it improved performance in some cases but not in all situations
(there were some slow downs which scares me).
(You can see patches here http://people.redhat.com/mpatocka/patches/kernel/dm-crypt-paralelizace/)Unfortunately discussion stopped and device-mapper maintainer forgot about it.
Well, your mails apparently caused some head-ups here, so I'll try to return to this. (Will post them to dm-devel directly this time.)
I appreciate your answer and your looking into this. Since I run recent self compiled kernel.org kernels, I can test patches as long as they're reasonably certain not to turn my data into garbage :)
(I have backups, but it just too me too long to rebuild my laptop after my last SSD crash).
Quoted 3 lines · Milan Broz
Common distro env is nice but be sure you have proper crypto modules available. Also do not use Fedora rawhide, it has kernel compiled with debug tools not usable for testing.
Mmmh, I have one possible thing. I have a preempt kernel. Could that be it?
http://marc.merlins.org/tmp/config-3.4.4-amd64-preempt.txt
Quoted 6 lines
You can try start with this:
1) (this should be not problem but better check it) Check alignment of partitions. Is it aligned to SSD page size? (Aligning to 1MiB is always correct ;-)
Paste fdisk -l -u/dev/sda
Disk /dev/sda: 512.1 GB, 512110190592 bytes
255 heads, 63 sectors/track, 62260 cylinders, total 1000215216 sectors Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x09aaf50a
Device Boot Start End Blocks Id System /dev/sda1 * 2048 502047 250000 83 Linux /dev/sda2 502048 52930847 26214400 83 Linux /dev/sda3 52930848 73902367 10485760 82 Linux swap / Solaris /dev/sda4 73902368 1000215215 463156424 83 Linux
I also used:
cryptsetup luksFormat --align-payload=8192
Quoted 2 lines
2) try to switch io scheduler to "noop" or "deadline" (paste lsblk -t)
I tried both noop and deadline (never used cfq) and it didn't help.
gandalfthegreat:~# lsblk -t NAME ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED RQ-SIZE sda 0 512 0 512 512 0 deadline 128 ├─sda1 0 512 0 512 512 0 deadline 128 ├─sda2 0 512 0 512 512 0 deadline 128 ├─sda3 0 512 0 512 512 0 deadline 128 └─sda4 0 512 0 512 512 0 deadline 128 └─cryptroot (dm-0) 0 512 0 512 512 0 128
But just to make sure, I tried cfg, noop, and deadline and it didn't make a difference.
Quoted 1 line
Also you can try to increase queue size.
Not sure which one it is:
gandalfthegreat:/sys/block/sda/queue# grep . * add_random:1 discard_granularity:512 discard_max_bytes:2147450880 discard_zeroes_data:0 hw_sector_size:512 iostats:1 logical_block_size:512 max_hw_sectors_kb:32767 max_integrity_segments:0 max_sectors_kb:512 max_segments:168 max_segment_size:65536 minimum_io_size:512 nomerges:0 nr_requests:128 optimal_io_size:0 physical_block_size:512 read_ahead_kb:128 rotational:0 rq_affinity:1 scheduler:[noop] deadline cfq
Quoted 2 lines
(Hard core version is to run blktrace and check if request are not split unnecessarily :)
I'm not too sure how to read the output, but there it is:
http://marc.merlins.org/tmp/blktrace.txt
Generated with:
gandalfthegreat:~# reset_cache ; dd if=/dev/mapper/cryptroot of=/dev/null bs=1M count=10; killall blktrace 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.514707 s, 20.4 MB/s
Quoted 4 lines
3) Let's test
cipher_null(no encryption just fake-copy) (you need cryptsetup 1.4.3 for this test).create test LUKS device with null cipher: cryptsetup luksFormat -c null
gandalfthegreat:~# cryptsetup luksFormat --align-payload=8192 -c null /dev/sda2 Are you sure? (Type uppercase yes): YES Enter LUKS passphrase: Verify passphrase: gandalfthegreat:~# cryptsetup luksOpen /dev/sda2 test Enter passphrase for /dev/sda2: gandalfthegreat:~# hdparm -t -T /dev/mapper/test /dev/mapper/test: Timing cached reads: 12932 MB in 2.00 seconds = 6471.89 MB/sec Timing buffered disk reads: 76 MB in 3.04 seconds = 25.01 MB/sec gandalfthegreat:~#
So it's not related to the kind of encryption.
Quoted 2 lines
please note: cipher null means no encryption, just use dmcrypt layer, so do not use for valid data :-)
Yes, I figured :)
Quoted 3 lines
4) which aes module are you using? check lsmod, check
/proc/cryptoyou should use either aes-ni or optimized modules (
x86_64etc)
Yep, I tried using both (aes-ni is default) and result was the same.
I'll try rebuilding a non preempt kernel just in case.
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ Quoted 10 lines · Marc MERLIN
Quoted 3 lines · Arno Wagner
Quoted 1 line
Quoted 1 line
SID? That would be "unstable", whit possible assorted problems.
Ok, sorry for my confusion, what kernel/distro are you running?
Debian testing with pieces of unstable :)
That gives me cryptsetup 1.4.3
(but debian unstable is often not more unstable than your released fedora core or ubuntu in my experience)
[…]
Quoted 1 line
I'll try rebuilding a non preempt kernel just in case.
Maybe try with Debian kernel first? It's just an apt-get install away and will give you a setup identical as mine.
Regards,
Signature · 1 line
Yves-Alexis
Quoted 6 lines · Milan Broz
Quoted 2 lines
Mmmh, I have one possible thing. I have a preempt kernel. Could that be it?
http://marc.merlins.org/tmp/config-3.4.4-amd64-preempt.txtCan you send me your kernel .config then? Preempt should not be problem. Which kernel version? which architecture? Any additional patches over mainline code?
I just sent you my config, it was in the URL above :) No patches, kernel 3.4.4 from kernel.org, see above.
I tried without preempt and with volprempt and duplicated the same slow speeds.
Quoted 7 lines
Quoted 7 lines
Disk
/dev/sda: 512.1 GB, 512110190592 bytes
255 heads, 63 sectors/track, 62260 cylinders, total 1000215216 sectors Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier:0x09aaf50a
Quoted 8 lines
Quoted 5 lines
Device Boot Start End Blocks Id System /dev/sda1 * 2048 502047 250000 83 Linux /dev/sda2 502048 52930847 26214400 83 Linux /dev/sda3 52930848 73902367 10485760 82 Linux swap / Solaris /dev/sda4 73902368 1000215215 463156424 83 LinuxHm. sda1 is apparently aligned. But I think
other partitions are not aligned properly.
Really?
I used fdisk -H32 -S32 /dev/sda as recomended on
http://www.void.gr/kargig/blog/2012/01/11/linux-ssd-partition-alignment-tips/
Quoted 6 lines
No idea which block size/page size your SSD internaly use, but to be safe, let's assume ideal is 256KiB (= 512 * 512 byte sectors). 73902368 seems not to be multiple of 512...
Well, your sda2 configuration is created with +256MB, which is SI unit, so not multiple of 1024 but 1000!
Oh my, I can't believe they changed that (been using linux since 1993 back then +1MB in fdisk was a real MB).
Quoted 7 lines
It should be created with +256M, so you end with:
Device Boot Start End Blocks Id System /dev/sda1 2048 526335 262144 83 Linux
/dev/sda2526336 ...But I think this is not the core problem but you should try it to fix. (I hope I did not mixed up numbers above :)
It would be a problem for erase blocks if it is wrong, but not for reading, especially when I can do filesystem reads at 260MB/s on the filesystem created on the encrypted device.
But I'm confused, I thought
fdisk -H32 -S32
would protect me from mis-alignment and fdisk/util-linux 2.20.1 was smart enough to use proper boundaries on its own on SSDs anyway.
Quoted 10 lines
... Thinking about it, we can test it without partition change (for reads).
This test is perhaps useless but maybe someone can find it useful later.
Let's map properly aligned device to sda4 (read only). Result will be just garbage, but it is just for speed testing.
For mapping above, properly aligned (1MiB alignment = 2048 sectors) should start on sector 7390412. This is +1760 sector shift for sda4.
Quoted 7 lines
Map 1GiB device there:
# dmsetup create test_plain -r --table "0 2097152 linear /dev/sda4 1760"Map null cryptsetup over it
# echo "password" | cryptsetup create -c null test_crypt /dev/mapper/test_plainNow repeat read dd test for
/dev/mapper/test_plainand/dev/mapper/test_crypt
Same thing:
gandalfthegreat:~# dmsetup create test_plain -r --table "0 2097152 linear /dev/sda4 1760" gandalfthegreat:~# echo "password" | cryptsetup create -c null test_crypt /dev/mapper/test_plain gandalfthegreat:~# dmsetup status test_crypt 0 2097152 crypt gandalfthegreat:~# cryptsetup status test_crypt /dev/mapper/test_crypt is active. type: PLAIN cipher: cipher_null-ecb keysize: 0 bits device: /dev/mapper/test_plain offset: 0 sectors size: 2097152 sectors mode: read/write
gandalfthegreat:~# hdparm -t -T /dev/mapper/test_plain /dev/mapper/test_plain: Timing cached reads: 13978 MB in 2.00 seconds = 6997.30 MB/sec Timing buffered disk reads: 958 MB in 3.00 seconds = 319.00 MB/sec
gandalfthegreat:~# hdparm -t -T /dev/mapper/test_crypt /dev/mapper/test_crypt: Timing cached reads: 15434 MB in 2.00 seconds = 7725.50 MB/sec Timing buffered disk reads: 76 MB in 3.03 seconds = 25.05 MB/sec gandalfthegreat:~#
Quoted 4 lines
Quoted 2 lines
I also used:
cryptsetup luksFormat --align-payload=8192It will not help if underlying partition is misaligned.
That's correct of course :)
Now I'm going to have to do some more reading to see whether I'm really misaligned or not. I had done all the relevant reading on SSDs to make sure I did use proper alignment.
I wonder why fdisk -H32 -S32 failed me.
Quoted 3 lines
I think this one, try to increase it to 8192 or so...
Quoted 1 line
nr_requests:128
Didn't help.
gandalfthegreat:/sys/block/sda/queue# echo 8192 > nr_requests gandalfthegreat:/sys/block/sda/queue# hdparm -t -T /dev/mapper/cryptroot
/dev/mapper/cryptroot:
Timing cached reads: 14970 MB in 2.00 seconds = 7492.34 MB/sec
Timing buffered disk reads: 72 MB in 3.08 seconds = 23.35 MB/sec
gandalfthegreat:/sys/block/sda/queue#
Thanks for the other suggestions. Hopefully we'll nail this somehow :)
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ Quoted 10 lines · Marc MERLIN
Quoted 6 lines · Milan Broz
Quoted 2 lines
Mmmh, I have one possible thing. I have a preempt kernel. Could that be it?
http://marc.merlins.org/tmp/config-3.4.4-amd64-preempt.txtCan you send me your kernel .config then? Preempt should not be problem. Which kernel version? which architecture? Any additional patches over mainline code?
I just sent you my config, it was in the URL above :) No patches, kernel 3.4.4 from kernel.org, see above.
Ehm... sorry, I completely missed that. Thanks.
Quoted 3 lines
Really?
I used fdisk -H32 -S32/dev/sdaas recomended on
http://www.void.gr/kargig/blog/2012/01/11/linux-ssd-partition-alignment-tips/
Do not use -H32 -S32. It is crazy and obsolete way how to align it... Someone is wrong in the internet seems http://xkcd.com/386/ ;-)
Disk driver should set topology parameters which fdisk uses. But for your case all is set to 512 bytes...
Whatever, there was a bug in fdisk, fixed now thanks to your report :)
https://github.com/karelzak/util-linux/commit/c0175e6185ac81843cbad33cbea75abd033f0e66
Quoted 1 line
Thanks for the other suggestions. Hopefully we'll nail this somehow :)
Well, please try some default distro compiled kernel if you can reproduce it there.
Milan
Quoted 5 lines · Milan Broz
Quoted 4 lines · Marc MERLIN
Quoted 3 lines · Milan Broz
Quoted 2 lines
Mmmh, I have one possible thing. I have a preempt kernel. Could that be it?
http://marc.merlins.org/tmp/config-3.4.4-amd64-preempt.txt
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Can you send me your kernel .config then? Preempt should not be problem. Which kernel version? which architecture? Any additional patches over mainline code?
I just sent you my config, it was in the URL above :) No patches, kernel 3.4.4 from kernel.org, see above.
Ehm... sorry, I completely missed that. Thanks.
Mmmh, so I installed "standard" linux-image-3.2.0-3-amd64 from debian. And....
nothing changed :(
/dev/mapper/cryptroot:
Timing cached reads: 19642 MB in 2.00 seconds = 9833.88 MB/sec
Timing buffered disk reads: 72 MB in 3.05 seconds = 23.59 MB/sec
Did you find anything more useful here:
http://marc.merlins.org/tmp/blktrace.txt
Or can I take another blktrace that helps?
Quoted 6 lines
Quoted 3 lines
Really?
I used fdisk -H32 -S32/dev/sdaas recomended on
http://www.void.gr/kargig/blog/2012/01/11/linux-ssd-partition-alignment-tips/Do not use -H32 -S32. It is crazy and obsolete way how to align it... Someone is wrong in the internet seems http://xkcd.com/386/ ;-)
Yes, I know the cartoon :)
Mmmh, so I'll have to reformat everything so that all my partition start numbers are multiple of 512.
Maybe I can get parted to move at least partition #4 without losing all my data. I'll try that.
However is using
cryptsetup luksFormat --align-payload=8192
still the right thing for me?
(with the understanding that alignment shouldn't really an issue for reads, but for writes)
Quoted 5 lines
Disk driver should set topology parameters which fdisk uses. But for your case all is set to 512 bytes...
Whatever, there was a bug in fdisk, fixed now thanks to your report :)
https://github.com/karelzak/util-linux/commit/c0175e6185ac81843cbad33cbea75abd033f0e66
Cool, thanks for that.
Ok, so I repartitioned my first 3 partitions, which I could do without losing data:
Disk /dev/sda: 512.1 GB, 512110190592 bytes
255 heads, 63 sectors/track, 62260 cylinders, total 1000215216 sectors Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x09aaf50a
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 502271 250112 83 Linux /dev/sda2 502272 52930559 26214144 83 Linux /dev/sda3 52930560 73902079 10485760 82 Linux swap / Solaris /dev/sda4 73902368 1000215215 463156424 83 Linux
OMG, that actually helped:
gandalfthegreat:~# echo test | cryptsetup create -c null test_crypt /dev/sda2 gandalfthegreat:~# hdparm -t -T /dev/mapper/test_crypt /dev/mapper/test_crypt: Timing cached reads: 18186 MB in 2.00 seconds = 9103.83 MB/sec Timing buffered disk reads: 524 MB in 3.04 seconds = 172.63 MB/sec
But with LUKS, it falls apart:
gandalfthegreat:~# cryptsetup luksFormat --align-payload=8192 -c aes-xts-plain -s 256 /dev/sda2 (...) gandalfthegreat:~# cryptsetup luksOpen --allow-discards /dev/sda2 test (...) gandalfthegreat:~# hdparm -t -T /dev/mapper/test /dev/mapper/test: Timing cached reads: 17436 MB in 2.00 seconds = 8728.61 MB/sec Timing buffered disk reads: 74 MB in 3.03 seconds = 24.44 MB/sec
Grumble.
So
1) alignement has some effect and I'm not sure how to get luksFormat aligned right
2) Even in the better case above, 172.63 MB/sec is too slow. I was getting faster speeds by dding a file from potentially the encrypted filesystem.
If blktrace doesn't show anything useful, is there anything else I can capture?
Thanks,
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ Quoted 5 lines · Marc MERLIN
Timing buffered disk reads: 72 MB in 3.05 seconds = 23.59 MB/sec
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes
Please correct me if I should be wrong, but your drive should report 512/4096 here, so it lies about the real blocksize it uses (4k). This raises the question if you have created your filesystem on top of the encrypted partition with e.g. "-b 4096".
Quoted 3 lines · Marc MERLIN
Mmmh, so I installed "standard" linux-image-3.2.0-3-amd64 from debian. And....
nothing changed :(
Well. I found SSD where I can reproduce something similar. Seems page cache messes something with readahead...
# lsblk /dev/sdc -t NAME ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED RQ-SIZE sdc 0 512 0 512 512 0 noop 128 └─sdc_null_crypt (dm-0) 0 512 0 512 512 0 128
- Disable readahead
# blockdev --setra 0 /dev/mapper/sdc_null_crypt
# hdparm -t /dev/mapper/sdc_null_crypt Timing buffered disk reads: 106 MB in 3.01 seconds = 35.18 MB/sec
(pretty bad)
But with direct-io it still works:
# hdparm --direct -t /dev/mapper/sdc_null_crypt Timing O_DIRECT disk reads: 698 MB in 3.00 seconds = 232.44 MB/sec
(now with read-ahead set to 4M)
# blockdev --setra 8192 /dev/mapper/sdc_null_crypt # hdparm -t /dev/mapper/sdc_null_crypt Timing buffered disk reads: 664 MB in 3.00 seconds = 221.29 MB/sec
So. Can you please try to increase readahead (and also run it with direct-io)? Just to check if it is the same problem or not...
Thanks,
Milan
Quoted 2 lines · Milan Broz
- Disable readahead
# blockdev --setra 0 /dev/mapper/sdc_null_crypt
This seems not only to be a problem with SSD drives, but also with conventional harddisks. I've never seen something like this, because I have had "blockdev --setra 8192 /dev/sda" in my rc.local for ages, because it's well known that it speeds up all disks significantly.
Here's a short test. The partition is encrypted with 256-bit twofish-xts-plain64:sha256.
[root@wildsau ~]# hdparm -i /dev/sda
/dev/sda:
Model=Hitachi HTS723232L9A360, FwRev=FC4OC30F, SerialNo=090119FC1400NEGNHSTD
Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs } RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=4 BuffType=DualPortCache, BuffSize=15058kB, MaxMultSect=16, MultSect=16 CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=625142448 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120} PIO modes: pio0 pio1 pio2 pio3 pio4
DMA modes: mdma0 mdma1 mdma2
UDMA modes: udma0 udma1 udma2 udma3 udma4 udma5 *udma6 AdvancedPM=yes: mode=0x80 (128) WriteCache=enabled
Drive conforms to: unknown: ATA/ATAPI-2,3,4,5,6,7
With "blockdev --setra 8192" og "hdparm -tT":
Timing buffered disk reads: 234 MB in 3.01 seconds = 77.69 MB/sec
With "blockdev --setra 0" og "hdparm -tT":
Timing cached reads: 4670 MB in 2.00 seconds = 2336.29 MB/sec
Timing buffered disk reads: 14 MB in 3.06 seconds = 4.57 MB/sec
~~~~~~~~~~With "blockdev --setra 0" og "hdparm --direct -tT":
Timing O_DIRECT cached reads: 250 MB in 2.01 seconds = 124.45 MB/sec
Timing O_DIRECT disk reads: 238 MB in 3.02 seconds = 78.80 MB/sec
On 24.07.2012, Heinz Diehl wrote:
[....]
And that's obviously not all. The problem occurs also with _unencrypted_ partitions/devices, here's the proof (/dev/sda1 is a 500M ext2 partition mounted on /boot; using the same Hitachi rotational drive as in my previous test):
[root@wildsau /]# blockdev --setra 8192 /dev/sda1 [root@wildsau /]# hdparm -t /dev/sda1
Timing buffered disk reads: 236 MB in 3.00 seconds = 78.57 MB/sec
[root@wildsau /]# blockdev --setra 0 /dev/sda1
[root@wildsau /]# hdparm -t /dev/sda1
Timing buffered disk reads: 26 MB in 3.05 seconds = 8.51 MB/sec
[root@wildsau /]# blockdev --setra 0 /dev/sda1 [root@wildsau /]# hdparm --direct -t /dev/sda1
Timing O_DIRECT disk reads: 236 MB in 3.02 seconds = 78.09 MB/sec
All this is on F17, with a vanilla 3.5 kernel from kernel.org (with just 3 own small patches wich do not touch code involved here).
Quoted 4 lines · Heinz Diehl
On 24.07.2012, Heinz Diehl wrote:
[....]
Quoted 8 lines
And that's obviously not all. The problem occurs also with _unencrypted_ partitions/devices, here's the proof (
/dev/sda1is a 500M ext2 partition mounted on /boot; using the same Hitachi rotational drive as in my previous test):[root@wildsau /]# blockdev --setra 8192
/dev/sda1[root@wildsau /]# hdparm -t/dev/sda1
Timing buffered disk reads: 236 MB in 3.00 seconds = 78.57 MB/sec
Funny you'd say that. I checked mine (with 3.4.4)
By default, it comes with:
gandalfthegreat:~# blockdev --report /dev/sda2 RO RA SSZ BSZ StartSec Size Device rw 256 512 4096 502272 26843283456 /dev/sda2
gandalfthegreat:~# blockdev --setra 0 /dev/sda2 gandalfthegreat:~# hdparm -t /dev/sda2 /dev/sda2: Timing buffered disk reads: 2 MB in 4.81 seconds = 425.77 kB/sec
gandalfthegreat:~# hdparm --direct -t /dev/sda2 /dev/sda2: Timing O_DIRECT disk reads: 432 MB in 3.01 seconds = 143.68 MB/sec
So I know it doens't make sense, but apparently it's SSD/machine sensitive and for me the default of 256 is enough for the block device, but not enough for a dm-crypt'ed device.
I just checked that my other dmcrypt device I just created:
gandalfthegreat:~# blockdev --report /dev/mapper/test RO RA SSZ BSZ StartSec Size Device rw 256 512 4096 0 26839089152 /dev/mapper/test
So I do get 256 RA consistently and it's enough for the raw block device, but not enough for a dm-crypted device.
gandalfthegreat:~# hdparm -t /dev/mapper/test /dev/mapper/test: Timing buffered disk reads: 66 MB in 3.05 seconds = 21.62 MB/sec
By the way, just for run, I tried
cryptsetup luksFormat --align-payload=8192 -c twofish-xts-plain64 /dev/sda2 instead of
cryptsetup luksFormat --align-payload=8192 -c aes-xts-plain /dev/sda2 since it was suggested that it might be faster on recent laptops.
Well, on a Lenovo 530 with
model name : Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz
I get 177 MB/s for twofish-xts-plain64 and 281 MB/s for aes-xts-plain For comparison, I got 406 MB/s for null dmcrypt.
So it sounds like for my SSD, aes-xts-plain is the fastest with aesni_intel once I've run blockdev --settra 8192.
I tried --settra 81920 (10x more) and it raised aes-xts-plain from 281MB/s to 298MB/s and null dmcrypt 406MB/s to 446MB/s.
Hope this helps.
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ Quoted 3 lines · Marc MERLIN
So I know it doens't make sense, but apparently it's SSD/machine sensitive and for me the default of 256 is enough for the block device, but not enough for a dm-crypt'ed device.
I guess this has nothing to do with what kind of drive you have, but the real blocksize of the device. Conventional drives with 512/512 are fine with the default, but advanced format and SSD-drives which run with a 4k blocksize need more..
Quoted 4 lines · Heinz Diehl
Please correct me if I should be wrong, but your drive should report 512/4096 here, so it lies about the real blocksize it uses (4k). This raises the question if you have created your filesystem on top of the encrypted partition with e.g. "-b 4096".
I'm using btrfs, which defaults to 4K blocks. Also, I was I seeing 270MB/s reading a big file with btrfs on top of cryptroot.
Quoted 3 lines · Milan Broz
Seems I am running out of ideas :)
(I just read the mails again and I think I am missing something obvious. Whatever, I will return to it later.)
I wanted to command you for not giving up, you definitely went the extra mile :)
Quoted 2 lines
btw why is in blktrace process generating IO in log [bash] and not [dd] or [hdparm] ?
I noticed that. I used dd that time so that I could force an entire GB of data, and I was also surprised that it showed up in bash when indeed I did:
gandalfthegreat:~# dd if=/dev/mapper/cryptroot of=/dev/null bs=1M count=1024
Quoted 3 lines
Apparently you generated it with dd. (But maybe just old blktrace), I have # blktrace --version
blktrace version 2.0.0
Same version here.
Quoted 2 lines · Milan Broz
So. Can you please try to increase readahead (and also run it with direct-io)? Just to check if it is the same problem or not...
So, direct IO is faster but not as fast as it should, and .... readahead fixes it!
gandalfthegreat:~# hdparm --direct -t /dev/mapper/cryptroot /dev/mapper/cryptroot: Timing O_DIRECT disk reads: 242 MB in 3.02 seconds = 80.26 MB/sec
gandalfthegreat:~# dd if=/dev/mapper/cryptroot of=/dev/null bs=1M count=1024 iflag=direct 1073741824 bytes (1.1 GB) copied, 18.4517 s, 58.2 MB/s
gandalfthegreat:~# blockdev --setra 8192 /dev/mapper/cryptroot gandalfthegreat:~# hdparm --direct -t /dev/mapper/cryptroot /dev/mapper/cryptroot: Timing O_DIRECT disk reads: 256 MB in 3.01 seconds = 85.10 MB/sec
gandalfthegreat:~# dd if=/dev/mapper/cryptroot of=/dev/null bs=1M count=1024 iflag=direct 1073741824 bytes (1.1 GB) copied, 16.7627 s, 64.1 MB/s
But non dirct IO is now fast:
gandalfthegreat:~# hdparm -t /dev/mapper/cryptroot /dev/mapper/cryptroot: Timing buffered disk reads: 784 MB in 3.00 seconds = 261.08 MB/sec
gandalfthegreat:~# dd if=/dev/mapper/cryptroot of=/dev/null bs=1M count=1024 1073741824 bytes (1.1 GB) copied, 3.82117 s, 281 MB/s
A big thank you helping me track this down.
Obviously for now I'll make sure I have this in my initscripts, and while your direct IO is much faster than mine (weird), it looks like you can reproduce this too in the non direct IO case at least. Hopefully that'll help fix the kernel.
Feel free to ping me if there are kernel patches you'd like me to try before submission.
Thanks again,
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ Quoted 9 lines · Marc MERLIN
Quoted 4 lines · Heinz Diehl
Please correct me if I should be wrong, but your drive should report 512/4096 here, so it lies about the real blocksize it uses (4k). This raises the question if you have created your filesystem on top of the encrypted partition with e.g. "-b 4096".
I'm using btrfs, which defaults to 4K blocks. Also, I was I seeing 270MB/s reading a big file with btrfs on top of cryptroot.
Quoted 7 lines
Quoted 3 lines · Milan Broz
Seems I am running out of ideas :)
(I just read the mails again and I think I am missing something obvious. Whatever, I will return to it later.)I wanted to command you for not giving up, you definitely went the extra mile :)
:)
So seems elevator completely misbehaves for SSD in some situations.
I have no time to check it today but this must be fixed. Read-ahead is just stupid workaround...
# echo "0">/sys/block/sdc/queue/rotational # hdparm -t /dev/mapper/sdc_null_crypt Timing buffered disk reads: 220 MB in 3.01 seconds = 73.07 MB/sec
# echo "1">/sys/block/sdc/queue/rotational # hdparm -t /dev/mapper/sdc_null_crypt Timing buffered disk reads: 652 MB in 3.01 seconds = 216.75 MB/sec
This SSD is quicker if set to rotational mode!
(So it merges requests in fact.)
Milan
Quoted 7 lines · Milan Broz
I have no time to check it today but this must be fixed. Read-ahead is just stupid workaround...
# echo "0">/sys/block/sdc/queue/rotational # hdparm -t /dev/mapper/sdc_null_crypt Timing buffered disk reads: 220 MB in 3.01 seconds = 73.07 MB/sec
Quoted 6 lines
# echo "1">/sys/block/sdc/queue/rotational # hdparm -t /dev/mapper/sdc_null_crypt Timing buffered disk reads: 652 MB in 3.01 seconds = 216.75 MB/secThis SSD is quicker if set to rotational mode!
(So it merges requests in fact.)
That didn't help for me, strangely.
gandalfthegreat:~# blockdev --setra 256 /dev/mapper/cryptroot gandalfthegreat:~# reset_cache gandalfthegreat:~# dd if=/dev/mapper/cryptroot of=/dev/null bs=1M count=1024 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 43.4968 s, 24.7 MB/s gandalfthegreat:~# dmsetup ls | grep cryptroot cryptroot (254:0) gandalfthegreat:~# echo "1">/sys/block/dm-0/queue/rotational gandalfthegreat:~# reset_cache gandalfthegreat:~# dd if=/dev/mapper/cryptroot of=/dev/null bs=1M count=1024 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 42.9933 s, 25.0 MB/s
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ Quoted 2 lines · Milan Broz
Disk driver should set topology parameters which fdisk uses. But for your case all is set to 512 bytes...
In case of a 4k alignment, I use "fdisk -c -u /dev/sda". The first partition will start at 2048, and all gets properly aligned as long as I'm using "+G" for defining the partitions. The last partition on a disk can be misaligned if you choose to use "the rest". Since fdisk displays the sectors in an inclusive way, the endsize fdisk is showing ("p") +1 should be divisible by 8. You can see it the other way round and say (endsize MOD 8 = 7).
It's the same thing with Western Digital and Seagate drives which use "advanced format". As far as I know, proper alignment is not the whole thing: some drives doesn't report that they are using the new blocksize, and the OS can therefore not know (reporting 512/512 logical/physical blocksize, instead of 512/4096). So the filesystem which is created on top of these partitions has to be created using the 4096 blocksize: "mkfs.ext4 -b 4096 /dev/sdx", e.g.
I don't know how this is handled via the dmcrypt layer..
Quoted 3 lines · Marc MERLIN
On Mon, Jul 23, 2012 at 10:14:07AM +0200, Arno Wagner wrote:
(editted: actually no, using 'null' encryptino still gives 25MB/s).
Ok, the we can forget about aes-ni etc, it is problem somewhere else.
Quoted 2 lines
Mmmh, I have one possible thing. I have a preempt kernel. Could that be it?
http://marc.merlins.org/tmp/config-3.4.4-amd64-preempt.txt
Can you send me your kernel .config then? Preempt should not be problem. Which kernel version? which architecture? Any additional patches over mainline code?
Quoted 9 lines
Quoted 1 line
Paste fdisk -l -u
/dev/sdaDisk
/dev/sda: 512.1 GB, 512110190592 bytes
255 heads, 63 sectors/track, 62260 cylinders, total 1000215216 sectors Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier:0x09aaf50a
Quoted 5 lines
Device Boot Start End Blocks Id System /dev/sda1 * 2048 502047 250000 83 Linux /dev/sda2 502048 52930847 26214400 83 Linux /dev/sda3 52930848 73902367 10485760 82 Linux swap / Solaris /dev/sda4 73902368 1000215215 463156424 83 Linux
Hm. sda1 is apparently aligned. But I think
other partitions are not aligned properly.
No idea which block size/page size your SSD internaly use, but to be safe, let's assume ideal is 256KiB (= 512 * 512 byte sectors). 73902368 seems not to be multiple of 512...
Well, your sda2 configuration is created with +256MB, which is SI unit, so not multiple of 1024 but 1000!
It should be created with +256M, so you end with:
Device Boot Start End Blocks Id System /dev/sda1 2048 526335 262144 83 Linux
/dev/sda2 526336 ...
But I think this is not the core problem but you should try it to fix. (I hope I did not mixed up numbers above :)
... Thinking about it, we can test it without partition change (for reads).
This test is perhaps useless but maybe someone can find it useful later.
Let's map properly aligned device to sda4 (read only). Result will be just garbage, but it is just for speed testing.
For mapping above, properly aligned (1MiB alignment = 2048 sectors) should start on sector 7390412. This is +1760 sector shift for sda4.
Map 1GiB device there:
# dmsetup create test_plain -r --table "0 2097152 linear /dev/sda4 1760"
Map null cryptsetup over it
# echo "password" | cryptsetup create -c null test_crypt /dev/mapper/test_plain
Now repeat read dd test for /dev/mapper/test_plain and /dev/mapper/test_crypt
Still the same slow down?
Remove devices
# dmsetup remove test_crypt # dmsetup remove test_plain
(Btw if you use real cipher and zero device instead of underlying disk, you can measure encryption throughput:
# dmsetup create test_plain --table "0 2097152 zero" # echo "password" | cryptsetup create -c aes-xts-plain64 -s 256 test_crypt /dev/mapper/test_plain ... very useful to prove that your cpu encryption is slow... but we proved that it is not the case here, so just for the archive :-)
Quoted 2 lines
I also used:
cryptsetup luksFormat --align-payload=8192
It will not help if underlying partition is misaligned.
Quoted 3 lines
Quoted 1 line
2) try to switch io scheduler to "noop" or "deadline"
But just to make sure, I tried cfg, noop, and deadline and it didn't make a difference.
ok
Quoted 3 lines
Quoted 1 line
Also you can try to increase queue size.
Not sure which one it is:
I think this one, try to increase it to 8192 or so...
Quoted 1 line
nr_requests:128
Milan
I remember a test which I conducted after I received my Intel SDD 510 120 GB about 1 year ago. I don't remember all the details, but the essence was that write speed decreased dramatically if the write process wasn't aligned.
I did:
dd if=/dev/zero of=/dev/sda bs=4k
I got around 400 Mb/sec, which was expected. But when I did:
dd if=/dev/zero of=/dev/sda bs=5k
I got only around 40 Mb/sec
When you write in 5k blocks, the SSD has to do a lot of overhead because of the Read-Modify-Write operations. I guess this might also be the case if you write in 4k blocks, but don't align them to the SSD's "native" blocks. So I would check for any misalignment.
Regards,
André
Quoted 4 lines · Yves-Alexis Perez
Quoted 1 line
I'll try rebuilding a non preempt kernel just in case.
Maybe try with Debian kernel first? It's just an apt-get install away and will give you a setup identical as mine.
I have a custom initramfs, it was easier to compile a new kernel than to try and get the debian one to boot my system :)
But for reference, what kernel package are you using?
Quoted 11 lines · André Gall
I remember a test which I conducted after I received my Intel SDD 510 120 GB about 1 year ago. I don't remember all the details, but the essence was that write speed decreased dramatically if the write process wasn't aligned.
I did:
dd if=/dev/zero of=/dev/sda bs=4k
I got around 400 Mb/sec, which was expected. But when I did:dd if=/dev/zero of=/dev/sda bs=5k
I got only around 40 Mb/sec
Yes, write alignment and write amplification are definitely issues on SSDs, but note how I was only doing read tests :)
See
http://www.void.gr/kargig/blog/2012/01/11/linux-ssd-partition-alignment-tips/
Cheers,
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ Quoted 4 lines · Marc MERLIN
Quoted 3 lines · Yves-Alexis Perez
Quoted 1 line
I'll try rebuilding a non preempt kernel just in case.
Maybe try with Debian kernel first? It's just an apt-get install
Quoted 8 lines
away
Quoted 1 line
and will give you a setup identical as mine.
I have a custom initramfs, it was easier to compile a new kernel than to try
and get the debian one to boot my system :)But for reference, what kernel package are you using?
Linux version 3.2.0-3-amd64 (Debian 3.2.21-3) ([email protected]) (gcc version 4.6.3 (Debian 4.6.3-8) ) #1 SMP Thu Jun 28 09:07:26 UTC 2012
And I'll upgrade to 3.2.23-1 just know.
Signature · 1 line
Yves-Alexis
Quoted 6 lines · Milan Broz
I do not like this dmcrypt mode a we tried to fix it. There is a bunch of patches from Mikulas Patocka which switches parallelization to use all available
cpus (if not limited by paramater).
In my tests it improved performance in some cases but not in all situations
(there were some slow downs which scares me).
(You can see patches here http://people.redhat.com/mpatocka/patches/kernel/dm-crypt-paralelizace/)
This is definitely the way to go, in the age of multicore-systems. Some of the patches are already included in linux-3.5, and the rest needs to be rebased on top of it. I'd like to try the whole series on a quadcore testing machine, but I'm not familiar with the code in most of the patches, and only one single wrong merge could lead to wrong conclusions.
Quoted 9 lines · Heinz Diehl
Quoted 6 lines · Milan Broz
I do not like this dmcrypt mode a we tried to fix it. There is a bunch of patches from Mikulas Patocka which switches parallelization to use all available
cpus (if not limited by paramater).
In my tests it improved performance in some cases but not in all situations
(there were some slow downs which scares me).
(You can see patches here http://people.redhat.com/mpatocka/patches/kernel/dm-crypt-paralelizace/)
Quoted 6 lines
This is definitely the way to go, in the age of multicore-systems. Some of the patches are already included in linux-3.5, and the rest needs to be rebased on top of it. I'd like to try the whole series on a quadcore testing machine, but I'm not familiar with the code in most of the patches, and only one single wrong merge could lead to wrong conclusions.
You can use my working repo, it can be applied cleanly on top of 3.5
http://mbroz.fedorapeople.org/dm-crypt/parallel/
But I run just very simple test, there can be mistakes.
(There are still a lot of questions and maybe some ideas will not be implemented this way, Anyway, if you can prove it helps in your config, let me know.)
Milan
Quoted 10 lines · Marc MERLIN
Result is the same:
gandalfthegreat:~# hdparm -tT /dev/mapper/ssdcrypt
/dev/mapper/ssdcrypt:
Timing cached reads: 16614 MB in 2.00 seconds = 8317.50 MB/sec
Timing buffered disk reads: 68 MB in 3.08 seconds = 22.09 MB/secGrumble.
It shouldn't be a hardware problem since I do see 400MB/s before encryption.
Ok, I'm not sure I understand. I mounted /dev/mapper/ssdcrypt (btrfs) and when going through the filesystem I get much higher speeds than using dd on the raw block device.
But... when I run
The test below shows that I can access the encrypted SSD 10x faster by talking to it through a btrfs filesystem than doing a dd read from the device node.
So I suppose I could just ignore the dd device node thing, however not really. Let's take a directory with some source files inside:
gandalfthegreat:/var/local# find src | wc -l 15261 gandalfthegreat:/var/local# echo 3 > /proc/sys/vm/drop_caches; time du -sh src 514M src real 0m5.865s
So on an encrypted spinning disk, it takes 6 seconds.
On my SSD in the same machine with the same encryption and the same btrfs filesystem, I get 5 times slower:
gandalfthegreat:/mnt/btrfs_pool1/var/local# time du -sh src 514M src real 0m24.937s
Incidently 5x is also the speed difference between my encrypted HD and encrypted SSD with dd.
Now, why du is 5x slower and dd of a file from the filesystem is 2.5x faster, I have no idea :-/
See below:
1) drop caches
gandalfthegreat:/mnt/btrfs_pool1/var/local/VirtualBox VMs/w2k_virtual# echo 3 > /proc/sys/vm/drop_caches
gandalfthegreat:/mnt/btrfs_pool1/var/local/VirtualBox VMs/w2k_virtual# dd if=w2k-s001.vmdk of=/dev/null 2146631680 bytes (2.1 GB) copied, 8.03898 s, 267 MB/s
-> 267MB/s reading from the file through the encrypted filesystem. That's good.
For comparison
gandalfthegreat:/mnt/mnt2# dd if=w2k-s001.vmdk of=/dev/null 2146631680 bytes (2.1 GB) copied, 4.33393 s, 495 MB/s -> almost 500MB/s reading through another unencrypted filesystem on the same SSD
gandalfthegreat:/mnt/btrfs_pool1/var/local/VirtualBox VMs/w2k_virtual# dd if=/dev/mapper/ssdcrypt of=/dev/null bs=1M count=1000 1048576000 bytes (1.0 GB) copied, 45.1234 s, 23.2 MB/s
-> 23MB/s reading from the block device that my FS is mounted from. WTF?
gandalfthegreat:/mnt/btrfs_pool1/var/local/VirtualBox VMs/w2k_virtual# echo 3 > /proc/sys/vm/drop_caches; dd if=w2k-s001.vmdk of=test 2146631680 bytes (2.1 GB) copied, 17.9129 s, 120 MB/s
-> 120MB/s copying a file from the SSD to itself. That's not bad.
gandalfthegreat:/mnt/btrfs_pool1/var/local/VirtualBox VMs/w2k_virtual# echo 3 > /proc/sys/vm/drop_caches; dd if=test of=/dev/null 2146631680 bytes (2.1 GB) copied, 8.4907 s, 253 MB/s
-> reading the new copied file still shows 253MB/s, good.
gandalfthegreat:/mnt/btrfs_pool1/var/local/VirtualBox VMs/w2k_virtual# dd if=test of=/dev/null 2146631680 bytes (2.1 GB) copied, 2.11001 s, 1.0 GB/s
-> reading without dropping the cache shows 1GB/s
I'm very lost now, any idea what's going on?
Thanks,
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/Quoted 18 lines · Marc MERLIN
I got a new Samsumg 830 512GB SSD which is supposed to be very high performance.
The raw device seems fast enough on a quick hdparm test:/dev/sda4:
Timing cached reads: 14258 MB in 2.00 seconds = 7136.70 MB/sec
Timing buffered disk reads: 1392 MB in 3.00 seconds = 463.45 MB/sec <<<<which is 4x faster than my non encrypted spinning disk, as expected.
But once I encrypt it, it drops to 5 times slower than my 1TB spinning disk in the same laptop:
gandalfthegreat:~# hdparm -tT /dev/mapper/ssdcrypt /dev/mapper/ssdcrypt: Timing cached reads: 15412 MB in 2.00 seconds = 7715.37 MB/sec Timing buffered disk reads: 70 MB in 3.06 seconds = 22.91 MB/sec <<<<
Quoted 5 lines
gandalfthegreat:~# hdparm -tT /dev/mapper/cryptroot (spinning disk) /dev/mapper/cryptroot: Timing cached reads: 16222 MB in 2.00 seconds = 8121.03 MB/sec Timing buffered disk reads: 308 MB in 3.01 seconds = 102.24 MB/sec <<<<
Hello,
I didn't read the whole thread, but are you aware that many/most SSDs use internal processors for compression, deduplication, etc ..
so if you write encrypted data to the SSD, it's not able to do it's internal magic, and thus you get a lot worse performance compared to non-encrypted data.
So did you try benchmarking with *random* data *without* encryption?
Also always first write to the disk, and only read after it has been already written to.
-- Pasi
Quoted 2 lines · Pasi Kärkkäinen
I didn't read the whole thread, but are you aware that many/most SSDs use internal processors for compression, deduplication, etc ..
Yes
Quoted 2 lines
so if you write encrypted data to the SSD, it's not able to do it's internal magic, and thus you get a lot worse performance compared to non-encrypted data.
Only on some controllers like sandforce, the Samsung 830 wasn't supposed to be affected
Quoted 2 lines
So did you try benchmarking with *random* data *without* encryption?
Also always first write to the disk, and only read after it has been already written to.
Yes, both were parts of my tests.
But, I owed everyone an update, which I just finished typing:
http://marc.merlins.org/perso/linux/post_2012-08-15_The-tale-of-SSDs_-Crucial-C300-early-Death_-Samsung-830-extreme-random-IO-slowness_-and-settling-with-OCZ-Vertex-4.html
Basically, the samsung 830 just sucks. I got 2 of them, they both utterly sucked. There is no excuse for an SSD being several times slower than a slow hard drive on _READs_ (not even talking about writes).
I'm not sure how I could have gotten 2 bad drives from Samsung in 2 different shipments, so I'm afraid the entire line may be bad. At least, it was for me after extensive benchmarking, and even using their own windows benchmarking tool.
In the end, I got a OCZ Vertex 4 and it's superfast as per the benchmarks I posted in the link above.
Cheers,
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 7 lines · André Gall
Hi Marc,
thank you very much for your detailed investigation. I wanted to buy the Samsung SSD 830 in the next few days to replace my old SSD. I chose this drive explicitely because of it's "supposed" performance with encrypted data according to the benchmarks of incompressible data all around the internet. Seems like I should rethink this decision :)
That's also why I bought it, and obviously it didn't work out.
I seem to remember that the OCZ Vertex 4 also does ok with data that doesn't compress.
Quoted 5 lines
Quoted 2 lines
Samsung's old Benchmark registered so low on their own Random IO test that the bar graph showing speed was a single line at the '0' mark.
You don't happen to have a screenshot of this? This would make it much easier for people to understand there's something wrong!
Sure thing. Page updated:
http://marc.merlins.org/perso/linux/post_2012-08-15_The-tale-of-SSDs_-Crucial-C300-early-Death_-Samsung-830-extreme-random-IO-slowness_-and-settling-with-OCZ-Vertex-4.html
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/