slow read performance, but fast writes?
Hi,
i've got a really strange problem with my setup, and i'm not quite sure why.
First of all, the setup. Got 6x 2TB Samsung hdds, each one encrypted with dm_crypt, on top of the encrypted block devices, there's a raid6. Normally you would assume, that writes are slower than reads, since dm-crypt has to encrypt 2 additional parity stripes (and md raid6 has to calculate the 2 parity stripes).
Let's take a look on the write performance first. Here's an average of 10 seconds iostat while running "dd if=/dev/zero of=/daten/testfile bs=1M"
avg-cpu: %user %nice %system %iowait %steal %idle
0.12 0.05 94.15 2.17 0.00 3.50
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 9.30 15200.00 1.80 2972.70 44.40 73472.40 49.43 15.53 5.21 201.72 5.09 0.26 76.68 sdd 6.30 14495.00 1.60 3759.60 31.60 73756.00 39.24 2.91 0.77 133.44 0.71 0.19 73.08 sdf 3.10 14519.30 1.70 3650.00 19.20 73429.20 40.23 2.86 0.78 137.06 0.72 0.20 72.52 sde 6.10 14240.10 1.30 3964.20 29.60 73572.40 37.12 2.80 0.70 173.38 0.65 0.18 71.70 sdg 3.10 14859.40 1.10 3306.50 16.80 73410.00 44.40 2.72 0.82 3.45 0.82 0.19 63.06 sdc 0.20 14276.20 1.00 3909.40 4.80 73450.00 37.57 2.65 0.67 41.80 0.66 0.17 65.29 md3 0.00 0.00 0.00 4609.60 0.00 294870.40
127.94 0.00 0.00 0.00 0.00 0.00 0.00
Works as expected. Since this machine is powered by an quadcore, all this crypto and raid6 threads are nicely balanced over all 4 cores. Sequential Write averages out at about 300MB/s.
So the CPU is capable of encrypting 6 times 72MB per second (while calculating parity for raid6).
Now for the Reads.
10 seconds iostat while running "dd if=/dev/md3 of=/dev/null bs=1M"
avg-cpu: %user %nice %system %iowait %steal %idle
0.15 0.05 37.61 19.90 0.00 42.29
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 5700.20 0.00 1141.90 0.00 27507.20 0.00
48.18 6.37 5.52 5.52 0.00 0.42 48.27
sdd 5420.60 0.00 1419.80 0.00 27462.40 0.00
38.68 4.47 3.14 3.14 0.00 0.30 42.12
sdf 5646.90 0.00 1198.10 0.00 27418.00 0.00
45.77 5.50 4.57 4.57 0.00 0.36 42.91
sde 5837.10 0.00 1006.40 0.00 27507.20 0.00
54.66 6.69 6.61 6.61 0.00 0.49 49.45
sdg 5671.10 0.00 1171.30 0.00 27401.20 0.00
46.79 4.65 3.95 3.95 0.00 0.34 39.99
sdc 5994.20 0.00 851.30 0.00 27485.60 0.00
64.57 7.11 8.27 8.27 0.00 0.57 48.44
md3 0.00 0.00 40927.60 0.00 163710.40 0.00
8.00 0.00 0.00 0.00 0.00 0.00 0.00
42% idle and almost 20% iowait?
I've already tried everything i thought of. changed kernel CONFIG_HZ, tried a tickless Kernel, enabled/disabled ncq, changed bios from ahci to ide mode, tuned read ahead for harddisks, crypto-devices and raid array (with different combinations).
Same thing happens if i start badblocks on all 6 crypto devices. Througput drops to 29MB/s on each drive. When i overclock the CPU (AMD 910e) about 20%, the read rate increases at exactly 20%.
Any advice would be greatly appreciated.
thanks and best regards, Peter
The RAID6 calculations should not matter much. For example here is the RAID6 bootup-benchmark from my elderly Athlon 64 X2 5600+:
...
raid6: using algorithm sse2x2 (4867 MB/s)
What I suspect for your problem on reading is an I/O chip or attachment with asymmetrical performance. This can happen, e.g. when the buffering is asymetrical or one direction has a long switchover time when going to/from different targets while the other does not. Reads have to read one stripe in turn from each disk, (with head movement, rotational latency, etc.) while writes go to 1. OS buffer and 2. disk buffer.
So writes are in principle faster on any Unix, as they get buffered by the kernel and the disks, while reads do not. Writes used to be slower because disk heads would need to be positioned a lot better, but this difference has mostly vanished in modern drives.
Still, I suspect a hardware problem in the contoller(s) that the disks attach to or their attachment to the system.
Arno
Quoted 78 lines · Peter Merhaut
Hi,
i've got a really strange problem with my setup, and i'm not quite sure why.
First of all, the setup. Got 6x 2TB Samsung hdds, each one encrypted withdm_crypt, on top of the encrypted block devices, there's a raid6. Normally you would assume, that writes are slower than reads, since dm-crypt has to encrypt 2 additional parity stripes (and md raid6 has to calculate the 2 parity stripes).
Let's take a look on the write performance first. Here's an average of 10 seconds iostat while running "dd if=/dev/zero of=/daten/testfile bs=1M"avg-cpu: %user %nice %system %iowait %steal %idle
0.12 0.05 94.15 2.17 0.00 3.50Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
avgrq-sz avgqu-sz awaitr_awaitw_awaitsvctm %utilsdb 9.30 15200.00 1.80 2972.70 44.40 73472.40 49.43 15.53 5.21 201.72 5.09 0.26 76.68 sdd 6.30 14495.00 1.60 3759.60 31.60 73756.00 39.24 2.91 0.77 133.44 0.71 0.19 73.08 sdf 3.10 14519.30 1.70 3650.00 19.20 73429.20 40.23 2.86 0.78 137.06 0.72 0.20 72.52 sde 6.10 14240.10 1.30 3964.20 29.60 73572.40 37.12 2.80 0.70 173.38 0.65 0.18 71.70 sdg 3.10 14859.40 1.10 3306.50 16.80 73410.00 44.40 2.72 0.82 3.45 0.82 0.19 63.06 sdc 0.20 14276.20 1.00 3909.40 4.80 73450.00 37.57 2.65 0.67 41.80 0.66 0.17 65.29 md3 0.00 0.00 0.00 4609.60 0.00 294870.40127.94 0.00 0.00 0.00 0.00 0.00 0.00
Works as expected. Since this machine is powered by an quadcore, all this crypto and raid6 threads are nicely balanced over all 4 cores. Sequential Write averages out at about 300MB/s.
So the CPU is capable of encrypting 6 times 72MB per second (while calculating parity for raid6).Now for the Reads.
10 seconds iostat while running "dd if=/dev/md3 of=/dev/null bs=1M"
avg-cpu: %user %nice %system %iowait %steal %idle
0.15 0.05 37.61 19.90 0.00 42.29Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
avgrq-sz avgqu-sz awaitr_awaitw_awaitsvctm %util
sdb 5700.20 0.00 1141.90 0.00 27507.20 0.00
48.18 6.37 5.52 5.52 0.00 0.42 48.27
sdd 5420.60 0.00 1419.80 0.00 27462.40 0.00
38.68 4.47 3.14 3.14 0.00 0.30 42.12
sdf 5646.90 0.00 1198.10 0.00 27418.00 0.00
45.77 5.50 4.57 4.57 0.00 0.36 42.91
sde 5837.10 0.00 1006.40 0.00 27507.20 0.00
54.66 6.69 6.61 6.61 0.00 0.49 49.45
sdg 5671.10 0.00 1171.30 0.00 27401.20 0.00
46.79 4.65 3.95 3.95 0.00 0.34 39.99
sdc 5994.20 0.00 851.30 0.00 27485.60 0.00
64.57 7.11 8.27 8.27 0.00 0.57 48.44
md3 0.00 0.00 40927.60 0.00 163710.40 0.00
8.00 0.00 0.00 0.00 0.00 0.00 0.0042% idle and almost 20% iowait?
I've already tried everything i thought of. changed kernelCONFIG_HZ, tried a tickless Kernel, enabled/disabled ncq, changed bios from ahci to ide mode, tuned read ahead for harddisks, crypto-devices and raid array (with different combinations).
Same thing happens if i start badblocks on all 6 crypto devices. Througput drops to 29MB/s on each drive. When i overclock the CPU (AMD 910e) about 20%, the read rate increases at exactly 20%.
Any advice would be greatly appreciated.thanks and best regards, Peter
_______________________________________________
dm-crypt mailing list
[email protected]
http://www.saout.de/mailman/listinfo/dm-crypt
Signature · 7 lines
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: [email protected] GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F ---- Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans If it's in the news, don't worry about it. The very definition of "news" is "something that hardly ever happens." -- Bruce Schneier
Quoted 1 line · Peter Merhaut
i've got a really strange problem with my setup, and i'm not quite sure why.
It is known problem, current implementation depends on how are IO submitted and writes and reads differs this way (I think page cache interferes here). Reads are slower, because parallel operations are timed differently causing these waits.
If you use some more realistic IO pattern than dd, you can get different output though. (dd is NOT good performance test. But everyone uses that:-)
We have some ideas how to improve parallel processing inside dm-crypt but other things had higher priority still.
(You can see this problem even measured against zero virtual devices, so read/write caching is not the real problem here.)
Milan
that's what i suspected, so i ran badblocks on all 6 hdds simultaneously, getting 100-110MB/s each(!).
so the io subsystem is capable of ~600MB/s read, with very low iowait. when i start badblocks on all 6 crypto devices. the read speed drops to 29MB/s with high iowait.
when i start it on just one crypto device, i got 100MB/s. start a second badblocks, the speed on both drops to 80MB/s.
so the problem must be with dm_crypt or the way dm_crypt interacts with the underlying storage. maybe a problem/limitation when 6 crypto threads are running simultaneously?
Am 09/14/2011 05:04 PM, schrieb Arno Wagner:
Quoted 106 lines
The RAID6 calculations should not matter much. For example here is the RAID6 bootup-benchmark from my elderly Athlon 64 X2 5600+:
...
raid6: using algorithm sse2x2 (4867 MB/s)What I suspect for your problem on reading is an I/O chip or attachment with asymmetrical performance. This can happen, e.g. when the buffering is asymetrical or one direction has a long switchover time when going to/from different targets while the other does not. Reads have to read one stripe in turn from each disk, (with head movement, rotational latency, etc.) while writes go to 1. OS buffer and 2. disk buffer.
So writes are in principle faster on any Unix, as they get buffered by the kernel and the disks, while reads do not. Writes used to be slower because disk heads would need to be positioned a lot better, but this difference has mostly vanished in modern drives.
Still, I suspect a hardware problem in the contoller(s) that the disks attach to or their attachment to the system.
Arno
Quoted 77 lines · Peter Merhaut
Hi,
i've got a really strange problem with my setup, and i'm not quite sure why.
First of all, the setup. Got 6x 2TB Samsung hdds, each one encrypted withdm_crypt, on top of the encrypted block devices, there's a raid6. Normally you would assume, that writes are slower than reads, since dm-crypt has to encrypt 2 additional parity stripes (and md raid6 has to calculate the 2 parity stripes).
Let's take a look on the write performance first. Here's an average of 10 seconds iostat while running "dd if=/dev/zero of=/daten/testfile bs=1M"avg-cpu: %user %nice %system %iowait %steal %idle
0.12 0.05 94.15 2.17 0.00 3.50Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
avgrq-sz avgqu-sz awaitr_awaitw_awaitsvctm %utilsdb 9.30 15200.00 1.80 2972.70 44.40 73472.40 49.43 15.53 5.21 201.72 5.09 0.26 76.68 sdd 6.30 14495.00 1.60 3759.60 31.60 73756.00 39.24 2.91 0.77 133.44 0.71 0.19 73.08 sdf 3.10 14519.30 1.70 3650.00 19.20 73429.20 40.23 2.86 0.78 137.06 0.72 0.20 72.52 sde 6.10 14240.10 1.30 3964.20 29.60 73572.40 37.12 2.80 0.70 173.38 0.65 0.18 71.70 sdg 3.10 14859.40 1.10 3306.50 16.80 73410.00 44.40 2.72 0.82 3.45 0.82 0.19 63.06 sdc 0.20 14276.20 1.00 3909.40 4.80 73450.00 37.57 2.65 0.67 41.80 0.66 0.17 65.29 md3 0.00 0.00 0.00 4609.60 0.00 294870.40127.94 0.00 0.00 0.00 0.00 0.00 0.00
Works as expected. Since this machine is powered by an quadcore, all this crypto and raid6 threads are nicely balanced over all 4 cores. Sequential Write averages out at about 300MB/s.
So the CPU is capable of encrypting 6 times 72MB per second (while calculating parity for raid6).Now for the Reads.
10 seconds iostat while running "dd if=/dev/md3 of=/dev/null bs=1M"
avg-cpu: %user %nice %system %iowait %steal %idle
0.15 0.05 37.61 19.90 0.00 42.29Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
avgrq-sz avgqu-sz awaitr_awaitw_awaitsvctm %util
sdb 5700.20 0.00 1141.90 0.00 27507.20 0.00
48.18 6.37 5.52 5.52 0.00 0.42 48.27
sdd 5420.60 0.00 1419.80 0.00 27462.40 0.00
38.68 4.47 3.14 3.14 0.00 0.30 42.12
sdf 5646.90 0.00 1198.10 0.00 27418.00 0.00
45.77 5.50 4.57 4.57 0.00 0.36 42.91
sde 5837.10 0.00 1006.40 0.00 27507.20 0.00
54.66 6.69 6.61 6.61 0.00 0.49 49.45
sdg 5671.10 0.00 1171.30 0.00 27401.20 0.00
46.79 4.65 3.95 3.95 0.00 0.34 39.99
sdc 5994.20 0.00 851.30 0.00 27485.60 0.00
64.57 7.11 8.27 8.27 0.00 0.57 48.44
md3 0.00 0.00 40927.60 0.00 163710.40 0.00
8.00 0.00 0.00 0.00 0.00 0.00 0.0042% idle and almost 20% iowait?
I've already tried everything i thought of. changed kernelCONFIG_HZ, tried a tickless Kernel, enabled/disabled ncq, changed bios from ahci to ide mode, tuned read ahead for harddisks, crypto-devices and raid array (with different combinations).
Same thing happens if i start badblocks on all 6 crypto devices. Througput drops to 29MB/s on each drive. When i overclock the CPU (AMD 910e) about 20%, the read rate increases at exactly 20%.
Any advice would be greatly appreciated.thanks and best regards, Peter
_______________________________________________
dm-crypt mailing list
[email protected]
http://www.saout.de/mailman/listinfo/dm-crypt
Hi
how does that solve my issue with e.g. the extremely slow badblocks performance on the crypto-devices? (absolutely NO md involved!)
best regards, Peter
Am 09/15/2011 11:49 AM, schrieb Nikolay Kichukov:
Quoted 143 lines
Hi,
this query is best suited for md-raid mailing list as what you measure:dd if=/dev/md3 of=/dev/null bs=1M
does not involve any decryption. You are reading directly from the md device and there is no matter what is actually there.
Cheers,
-NikQuoted 6 lines · Peter Merhaut
that's what i suspected, so i ran badblocks on all 6 hdds simultaneously, getting 100-110MB/s each(!).
so the io subsystem is capable of ~600MB/s read, with very low iowait. when i start badblocks on all 6 crypto devices. the read speed drops to 29MB/s with high iowait.Quoted 39 lines
when i start it on just one crypto device, i got 100MB/s. start a second badblocks, the speed on both drops to 80MB/s.
so the problem must be with
dm_cryptor the waydm_cryptinteracts with the underlying storage. maybe a problem/limitation when 6 crypto threads are running simultaneously?Am 09/14/2011 05:04 PM, schrieb Arno Wagner:
Quoted 31 lines
The RAID6 calculations should not matter much. For example here is the RAID6 bootup-benchmark from my elderly Athlon 64 X2 5600+:
...
raid6: using algorithm sse2x2 (4867 MB/s)What I suspect for your problem on reading is an I/O chip or attachment with asymmetrical performance. This can happen, e.g. when the buffering is asymetrical or one direction has a long switchover time when going to/from different targets while the other does not. Reads have to read one stripe in turn from each disk, (with head movement, rotational latency, etc.) while writes go to 1. OS buffer and 2. disk buffer.
So writes are in principle faster on any Unix, as they get buffered by the kernel and the disks, while reads do not. Writes used to be slower because disk heads would need to be positioned a lot better, but this difference has mostly vanished in modern drives.
Still, I suspect a hardware problem in the contoller(s) that the disks attach to or their attachment to the system.
Arno
Quoted 3 lines · Peter Merhaut
Hi,
i've got a really strange problem with my setup, and i'm not quite
sure
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
why.
First of all, the setup. Got 6x 2TB Samsung hdds, each one encrypted withdm_crypt, on top of the encrypted block devices, there's a raid6. Normally you would assume, that writes are slower than reads, since dm-crypt has to encrypt 2 additional parity stripes (and md raid6has to
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
calculate the 2 parity stripes).
Let's take a look on the write performance first. Here's an average of 10 seconds iostat while running "dd if=/dev/zeroof=/daten/testfile bs=1M"
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
avg-cpu: %user %nice %system %iowait %steal %idle
0.12 0.05 94.15 2.17 0.00 3.50Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
avgrq-sz avgqu-sz awaitr_awaitw_awaitsvctm %utilsdb 9.30 15200.00 1.80 2972.70 44.40 73472.40 49.43 15.53 5.21 201.72 5.09 0.26 76.68 sdd 6.30 14495.00 1.60 3759.60 31.60 73756.00 39.24 2.91 0.77 133.44 0.71 0.19 73.08 sdf 3.10 14519.30 1.70 3650.00 19.20 73429.20 40.23 2.86 0.78 137.06 0.72 0.20 72.52 sde 6.10 14240.10 1.30 3964.20 29.60 73572.40 37.12 2.80 0.70 173.38 0.65 0.18 71.70 sdg 3.10 14859.40 1.10 3306.50 16.80 73410.00 44.40 2.72 0.82 3.45 0.82 0.19 63.06 sdc 0.20 14276.20 1.00 3909.40 4.80 73450.00 37.57 2.65 0.67 41.80 0.66 0.17 65.29 md3 0.00 0.00 0.00 4609.60 0.00 294870.40127.94 0.00 0.00 0.00 0.00 0.00 0.00
Works as expected. Since this machine is powered by an quadcore, all this crypto and raid6 threads are nicely balanced over all 4 cores. Sequential Write averages out at about 300MB/s.
So the CPU is capable of encrypting 6 times 72MB per second (while calculating parity for raid6).Now for the Reads.
10 seconds iostat while running "dd if=/dev/md3 of=/dev/null bs=1M"
avg-cpu: %user %nice %system %iowait %steal %idle
0.15 0.05 37.61 19.90 0.00 42.29Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
avgrq-sz avgqu-sz awaitr_awaitw_awaitsvctm %util
sdb 5700.20 0.00 1141.90 0.00 27507.20 0.00
48.18 6.37 5.52 5.52 0.00 0.42 48.27
sdd 5420.60 0.00 1419.80 0.00 27462.40 0.00
38.68 4.47 3.14 3.14 0.00 0.30 42.12
sdf 5646.90 0.00 1198.10 0.00 27418.00 0.00Quoted 8 lines
Quoted 8 lines
45.77 5.50 4.57 4.57 0.00 0.36 42.91
sde 5837.10 0.00 1006.40 0.00 27507.20 0.00
54.66 6.69 6.61 6.61 0.00 0.49 49.45
sdg 5671.10 0.00 1171.30 0.00 27401.20 0.00
46.79 4.65 3.95 3.95 0.00 0.34 39.99
sdc 5994.20 0.00 851.30 0.00 27485.60 0.00
64.57 7.11 8.27 8.27 0.00 0.57 48.44
md3 0.00 0.00 40927.60 0.00 163710.400.00
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
8.00 0.00 0.00 0.00 0.00 0.00 0.00
42% idle and almost 20% iowait?
I've already tried everything i thought of. changed kernelCONFIG_HZ, tried a tickless Kernel, enabled/disabled ncq, changed bios fromahci to
Quoted 1 line
Quoted 1 line
Quoted 1 line
ide mode, tuned read ahead for harddisks, crypto-devices and raid
array
Quoted 17 lines
Quoted 6 lines
Quoted 6 lines
(with different combinations).
Same thing happens if i start badblocks on all 6 crypto devices. Througput drops to 29MB/s on each drive. When i overclock the CPU (AMD 910e) about 20%, the read rate increases at exactly 20%. Any advice would be greatly appreciated.Quoted 6 lines
Quoted 5 lines
thanks and best regards, Peter
_______________________________________________
dm-crypt mailing list
[email protected]
http://www.saout.de/mailman/listinfo/dm-crypt_______________________________________________
dm-crypt mailing list
[email protected]
http://www.saout.de/mailman/listinfo/dm-crypt
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi,
this query is best suited for md-raid mailing list as what you measure:
dd if=/dev/md3 of=/dev/null bs=1M
does not involve any decryption. You are reading directly from the md device and there is no matter what is actually there.
Cheers,
- -Nik
Quoted 121 lines · Peter Merhaut
that's what i suspected, so i ran badblocks on all 6 hdds simultaneously, getting 100-110MB/s each(!).
so the io subsystem is capable of ~600MB/s read, with very low iowait. when i start badblocks on all 6 crypto devices. the read speed drops to 29MB/s with high iowait.when i start it on just one crypto device, i got 100MB/s. start a second badblocks, the speed on both drops to 80MB/s.
so the problem must be with
dm_cryptor the waydm_cryptinteracts with the underlying storage. maybe a problem/limitation when 6 crypto threads are running simultaneously?Am 09/14/2011 05:04 PM, schrieb Arno Wagner:
Quoted 21 lines
The RAID6 calculations should not matter much. For example here is the RAID6 bootup-benchmark from my elderly Athlon 64 X2 5600+:
...
raid6: using algorithm sse2x2 (4867 MB/s)What I suspect for your problem on reading is an I/O chip or attachment with asymmetrical performance. This can happen, e.g. when the buffering is asymetrical or one direction has a long switchover time when going to/from different targets while the other does not. Reads have to read one stripe in turn from each disk, (with head movement, rotational latency, etc.) while writes go to 1. OS buffer and 2. disk buffer.
So writes are in principle faster on any Unix, as they get buffered by the kernel and the disks, while reads do not. Writes used to be slower because disk heads would need to be positioned a lot better, but this difference has mostly vanished in modern drives.
Quoted 85 lines
Still, I suspect a hardware problem in the contoller(s) that the disks attach to or their attachment to the system.
Arno
Quoted 77 lines · Peter Merhaut
Hi,
i've got a really strange problem with my setup, and i'm not quite sure why.
First of all, the setup. Got 6x 2TB Samsung hdds, each one encrypted withdm_crypt, on top of the encrypted block devices, there's a raid6. Normally you would assume, that writes are slower than reads, since dm-crypt has to encrypt 2 additional parity stripes (and md raid6 has to calculate the 2 parity stripes).
Let's take a look on the write performance first. Here's an average of 10 seconds iostat while running "dd if=/dev/zero of=/daten/testfile bs=1M"avg-cpu: %user %nice %system %iowait %steal %idle
0.12 0.05 94.15 2.17 0.00 3.50Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
avgrq-sz avgqu-sz awaitr_awaitw_awaitsvctm %utilsdb 9.30 15200.00 1.80 2972.70 44.40 73472.40 49.43 15.53 5.21 201.72 5.09 0.26 76.68 sdd 6.30 14495.00 1.60 3759.60 31.60 73756.00 39.24 2.91 0.77 133.44 0.71 0.19 73.08 sdf 3.10 14519.30 1.70 3650.00 19.20 73429.20 40.23 2.86 0.78 137.06 0.72 0.20 72.52 sde 6.10 14240.10 1.30 3964.20 29.60 73572.40 37.12 2.80 0.70 173.38 0.65 0.18 71.70 sdg 3.10 14859.40 1.10 3306.50 16.80 73410.00 44.40 2.72 0.82 3.45 0.82 0.19 63.06 sdc 0.20 14276.20 1.00 3909.40 4.80 73450.00 37.57 2.65 0.67 41.80 0.66 0.17 65.29 md3 0.00 0.00 0.00 4609.60 0.00 294870.40127.94 0.00 0.00 0.00 0.00 0.00 0.00
Works as expected. Since this machine is powered by an quadcore, all this crypto and raid6 threads are nicely balanced over all 4 cores. Sequential Write averages out at about 300MB/s.
So the CPU is capable of encrypting 6 times 72MB per second (while calculating parity for raid6).Now for the Reads.
10 seconds iostat while running "dd if=/dev/md3 of=/dev/null bs=1M"
avg-cpu: %user %nice %system %iowait %steal %idle
0.15 0.05 37.61 19.90 0.00 42.29Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
avgrq-sz avgqu-sz awaitr_awaitw_awaitsvctm %util
sdb 5700.20 0.00 1141.90 0.00 27507.20 0.00
48.18 6.37 5.52 5.52 0.00 0.42 48.27
sdd 5420.60 0.00 1419.80 0.00 27462.40 0.00
38.68 4.47 3.14 3.14 0.00 0.30 42.12
sdf 5646.90 0.00 1198.10 0.00 27418.00 0.00
45.77 5.50 4.57 4.57 0.00 0.36 42.91
sde 5837.10 0.00 1006.40 0.00 27507.20 0.00
54.66 6.69 6.61 6.61 0.00 0.49 49.45
sdg 5671.10 0.00 1171.30 0.00 27401.20 0.00
46.79 4.65 3.95 3.95 0.00 0.34 39.99
sdc 5994.20 0.00 851.30 0.00 27485.60 0.00
64.57 7.11 8.27 8.27 0.00 0.57 48.44
md3 0.00 0.00 40927.60 0.00 163710.40 0.00
8.00 0.00 0.00 0.00 0.00 0.00 0.0042% idle and almost 20% iowait?
I've already tried everything i thought of. changed kernelCONFIG_HZ, tried a tickless Kernel, enabled/disabled ncq, changed bios from ahci to ide mode, tuned read ahead for harddisks, crypto-devices and raid array (with different combinations).
Same thing happens if i start badblocks on all 6 crypto devices. Througput drops to 29MB/s on each drive. When i overclock the CPU (AMD 910e) about 20%, the read rate increases at exactly 20%.
Any advice would be greatly appreciated.thanks and best regards, Peter
_______________________________________________
dm-crypt mailing list
[email protected]
http://www.saout.de/mailman/listinfo/dm-crypt
Quoted 4 lines
_______________________________________________
dm-crypt mailing list
[email protected]
http://www.saout.de/mailman/listinfo/dm-crypt
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQEcBAEBAgAGBQJOccpBAAoJEDFLYVOGGjgXOMAH/24/+1Q1hIbo0bPfIyDdGums SREqEsEz+ty/eOpaoKyjnFM0Ec0RF6ktQ5eVCQndje0SZp0R7ViQueooa8Y5GPBc 4brlPx7hJJ2MD9qsv7u5QOhT5BzClFaD891efN5O11VYuz25aioVWw6aMJlXfu4Y Mc4lVfgpyUwMXnLijtZYXxucGcJlrdGeEq7TcZ38DmM+hhhWzYfAWIlC5YNW7Jvs TvNUbIRe6KjHvYwyaG82n+9OgqGkcv303iNLEzqTypyVu7oBtPU1C9B7+3QEwwXn 857kyuxXmq2f1BtXJzleXknng2x3YgyqT0wmN3CK1jBVMmz87ONrkryFLBUtgCU= =jjZC
-----END PGP SIGNATURE-----
Quoted 6 lines · Nikolay Kichukov
Hi,
this query is best suited for md-raid mailing list as what you measure:dd if=/dev/md3 of=/dev/null bs=1M
does not involve any decryption. You are reading directly from the md device and there is no matter what is actually there.
heh. I read that as crypt device and it is MD directly, sorry for confusion. Seems my brain is still in different timezone :)
yes, this test has nothing to do with crypt.
Anyway, it is interesting that the problem is there even without crypt, perhaps you should ask on linux-raid list then.
for badblock - I guess it just submits all IO from one cpu,
so only one core is used for encryption in fact. There applies the same what I wrote before.
Milan
Yeah but why is it using one cpu for encryption? I got 6 encrypted block devices. When writing to them, all cpu cores are used for encryption, therefor it's fast. When reading, it seems to use 1 cpu for all 6 crypto threads?
Am 09/15/2011 12:45 PM, schrieb Milan Broz:
Quoted 19 lines · Nikolay Kichukov
Quoted 6 lines
Hi,
this query is best suited for md-raid mailing list as what you measure:dd if=/dev/md3 of=/dev/null bs=1M
does not involve any decryption. You are reading directly from the md device and there is no matter what is actually there.
heh. I read that as crypt device and it is MD directly, sorry for confusion. Seems my brain is still in different timezone :)
yes, this test has nothing to do with crypt.
Anyway, it is interesting that the problem is there even without crypt, perhaps you should ask on linux-raid list then.
for badblock - I guess it just submits all IO from one cpu,
so only one core is used for encryption in fact. There applies the same what I wrote before.Milan