Skip to contentSkip to conversation

dm-crypt and huge performance penalty

DISCUSSION Dmitry Nezhevenko 2 participants 3 messages Sep 20, 2010 – Sep 23, 2010

lore
Share & tools
lore
#0 lore raw

Hi,

I've just purchased three HDD drives and tried to setup mdadm RAID5 + dm-crypt.

What I've found is that dm-crypt reduces storage I/O performance very much while CPU is not bottleneck.

Firstly I've tried to setup raid5 and then dm-crypt on top of it. After discovering issue I've tried just raid5 and confirmed that mdadm itself works as expected.

And the last idea is to setup three dm-crypt partitions and then raid5 on top of independently encrypted drives.

So here are my results:

           |  mdadm   | mdadm+dm-crypt | 3 x dm-crypt + mdadm |
Seq. read  | 168 Mb/s | 57 Mb/s        | 119 Mb/s             |
Seq. write | 80 Mb/s  | 36 Mb/s        | 64.4 Mb/s            |

Also here are some results using both dd and bonnie as benchmark:
http://pastebin.com/hwwkRRTd

I'm using 3 x WD1002FAEX-00Z3A0 HDD. SATA controller is just embedded to m/b:

00:1f.2 IDE interface: Intel Corporation 5 Series/3400 Series Chipset 4 port SATA IDE Controller (rev 06)
00:1f.5 IDE interface: Intel Corporation 5 Series/3400 Series Chipset 2 port SATA IDE Controller (rev 06)

dd tests were done using simple dd:
if=/dev/resulting-device of=/dev/zero bs=1m count=512

For Bonnie I've used 50GB reiserfs3 partition on top of resulting device.

Are these numbers expected? They looks really strange for me.

Signature · 1 line
WBR, Dmitry
#1 lore raw
Quoted 4 lines · Dmitry Nezhevenko

Firstly I've tried to setup raid5 and then dm-crypt on top of it. After discovering issue I've tried just raid5 and confirmed that mdadm itself works as expected.

Quoted 8 lines

And the last idea is to setup three dm-crypt partitions and then raid5 on top of independently encrypted drives.

So here are my results:

          |  mdadm   | mdadm+dm-crypt | 3 x dm-crypt + mdadm |
Seq. read  | 168 Mb/s | 57 Mb/s        | 119 Mb/s             |
Seq. write | 80 Mb/s  | 36 Mb/s        | 64.4 Mb/s            |

dm-crypt will only use one core per device, so this is expected behavior (and I believe an issue that is being worked on). From these numbers it looks like you have a dual core or dual cpu system?

Signature · 1 line
Ryan Castellucci http://ryanc.org/
#2 lore raw
Quoted 10 lines · Ryan Castellucci
Quoted 9 lines · Dmitry Nezhevenko

Firstly I've tried to setup raid5 and then dm-crypt on top of it. After discovering issue I've tried just raid5 and confirmed that mdadm itself works as expected.

And the last idea is to setup three dm-crypt partitions and then raid5 on top of independently encrypted drives.

So here are my results:

Quoted 8 lines
Quoted 3 lines
          |  mdadm   | mdadm+dm-crypt | 3 x dm-crypt + mdadm |
Seq. read  | 168 Mb/s | 57 Mb/s        | 119 Mb/s             |
Seq. write | 80 Mb/s  | 36 Mb/s        | 64.4 Mb/s            |

dm-crypt will only use one core per device, so this is expected behavior (and I believe an issue that is being worked on). From these numbers it looks like you have a dual core or dual cpu system?

Hi, Yes, The machine is home NAS with inexpensive Intel Core i3 CPU that has two real cores and HT.

Signature · 1 line
WBR, Dmitry