Skip to contentSkip to conversation

dm-crypt / Software Raid5 issues on Ubuntu Lucid 64bit

dm-crypt BOBA FETT 2 participants 3 messages Sep 3, 2010

lore
Share & tools
lore

BOBA FETT

author
#0 lore raw

Hey folks,

I am switching from loop-aes to dm_crypt. I am trying to use it on a fresh Ubuntu 64bit server installation with my 6x750gb Software raid5 array. Array is clean and freshly created, I encrypted it using

cryptsetup -y -c aes -h ripemd160 -s 256 create STUFF /dev/md0

Format went fine, but when I try to copy back large amount of data to the encrypted array (about 900gig) my copy thread always hangs itself at some point during the process. I can't access the array anymore, though the raid itself is fine, no drives are kicked out or anything. Reboot gives me back access and the array is still ok.
When the problem occures I see the following trace in log:

Aug 29 02:54:24 liberty24 kernel: [221558.020638] Pid: 400, comm: md0_raid5 Not tainted 2.6.32-24-server #41-Ubuntu Unknow
Aug 29 02:54:24 liberty24 kernel: [221558.020638] RIP: 0010:[<ffffffff814353b8>]  [<ffffffff814353b8>] clone_endio+0x38/0xe0
Aug 29 02:54:24 liberty24 kernel: [221558.020638] RSP: 0018:ffff880069531c40  EFLAGS: 00010246
Aug 29 02:54:24 liberty24 kernel: [221558.020638] RAX: ffffffffa023f760 RBX: 0000000000000000 RCX: 0100000000000081
Aug 29 02:54:24 liberty24 kernel: [221558.020638] RDX: 0000000000000017 RSI: 0000000000000000 RDI: ffffc9001247e040
Aug 29 02:54:24 liberty24 kernel: [221558.035928] RBP: ffff880069531c70 R08: 0000000000000000 R09: 010000000000282c
Aug 29 02:54:24 liberty24 kernel: [221558.035928] R10: 0000000000000000 R11: 0000000000000001 R12: ffff88000ca0ed80
Aug 29 02:54:24 liberty24 kernel: [221558.035928] R13: ffff88000df85ed0 R14: ffff8800695ca600 R15: ffbf8800116cb348
Aug 29 02:54:24 liberty24 kernel: [221558.035928] FS:  00007fb75e7f4700(0000) GS:ffff880001c40000(0000) knlGS:0000000000000000
Aug 29 02:54:24 liberty24 kernel: [221558.035928] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Aug 29 02:54:24 liberty24 kernel: [221558.035928] CR2: 00007f2d10190000 CR3: 0000000053ae8000 CR4: 00000000000006e0
Aug 29 02:54:24 liberty24 kernel: [221558.035928] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 29 02:54:24 liberty24 kernel: [221558.035928] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 29 02:54:24 liberty24 kernel: [221558.035928] Process md0_raid5 (pid: 400, threadinfo ffff880069530000, task ffff88006a08dbc0)
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  ffff88001be85ed8 0000000000000000 0000000000000000 ffff88000ca0ed80
Aug 29 02:54:24 liberty24 kernel: [221558.035928] <0> ffff8800695ca600 ffff88001be85ed8 ffff880069531c80 ffffffff8117159d
Aug 29 02:54:24 liberty24 kernel: [221558.035928] <0> ffff880069531cb0 ffffffffa023d8c9 ffff880069531ca0 ffff88004d8312b0
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffff8117159d>] bio_endio+0x1d/0x40
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffffa023d8c9>] crypt_dec_pending+0x69/0x90 [dm_crypt]
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffffa023daa8>] crypt_endio+0x68/0x150 [dm_crypt]
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffff8117159d>] bio_endio+0x1d/0x40
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffffa0200add>] handle_stripe5+0x46d/0x9a0 [raid456]
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffffa0201028>] handle_stripe+0x18/0x30 [raid456]
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffffa0201432>] raid5d+0x202/0x320 [raid456]
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffff814298dc>] md_thread+0x5c/0x130
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffff81085090>] ? autoremove_wake_function+0x0/0x40
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffff81429880>] ? md_thread+0x0/0x130
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffff81084d16>] kthread+0x96/0xa0
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffff810141ea>] child_rip+0xa/0x20
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffff81084c80>] ? kthread+0x0/0xa0
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffff810141e0>] ? child_rip+0x0/0x20
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  RSP <ffff880069531c40>
Aug 29 02:54:24 liberty24 kernel: [221558.133125] ---[ end trace 29f7fd9a7bbb5a00 ]---

Using the array uncrypted, data copy to it works just fine, copied over the full 900 GB now 6 times without a hickup.

Can anybody help me with this?

thanks

Thorsten

#1 lore raw

Had those self-same issues (well, crypted RAID-array io-locking on me during samba-operations on it), and moving to kernel 2.6.34 fixed it for me. Never figured out what was really causing those problems.

Quoted 90 lines · On Fri, 03 Sep 2010 15:22:38 +0200 BOBA FETT <[email protected]> wrote:

Hey folks,

I am switching from loop-aes to dm_crypt. I am trying to use it on a fresh Ubuntu 64bit server installation with my 6x750gb Software raid5 array.
Array is clean and freshly created, I encrypted it using

cryptsetup -y -c aes -h ripemd160 -s 256 create STUFF /dev/md0

Format went fine, but when I try to copy back large amount of data to the encrypted array (about 900gig) my copy thread always hangs itself at some point during the process. I can't access the array anymore, though the raid itself is fine, no drives are kicked out or anything. Reboot gives me back access and the array is still ok.
When the problem occures I see the following trace in log:

Aug 29 02:54:24 liberty24 kernel: [221558.020638] Pid: 400, comm: 
md0_raid5 Not tainted 2.6.32-24-server #41-Ubuntu Unknow
Aug 29 02:54:24 liberty24 kernel: [221558.020638] RIP: 
0010:[<ffffffff814353b8>]  [<ffffffff814353b8>] clone_endio+0x38/0xe0
Aug 29 02:54:24 liberty24 kernel: [221558.020638] RSP: 
0018:ffff880069531c40  EFLAGS: 00010246
Aug 29 02:54:24 liberty24 kernel: [221558.020638] RAX: ffffffffa023f760 
RBX: 0000000000000000 RCX: 0100000000000081
Aug 29 02:54:24 liberty24 kernel: [221558.020638] RDX: 0000000000000017 
RSI: 0000000000000000 RDI: ffffc9001247e040
Aug 29 02:54:24 liberty24 kernel: [221558.035928] RBP: ffff880069531c70 
R08: 0000000000000000 R09: 010000000000282c
Aug 29 02:54:24 liberty24 kernel: [221558.035928] R10: 0000000000000000 
R11: 0000000000000001 R12: ffff88000ca0ed80
Aug 29 02:54:24 liberty24 kernel: [221558.035928] R13: ffff88000df85ed0 
R14: ffff8800695ca600 R15: ffbf8800116cb348
Aug 29 02:54:24 liberty24 kernel: [221558.035928] FS:  
00007fb75e7f4700(0000) GS:ffff880001c40000(0000) knlGS:0000000000000000
Aug 29 02:54:24 liberty24 kernel: [221558.035928] CS:  0010 DS: 0018 ES: 
0018 CR0: 000000008005003b
Aug 29 02:54:24 liberty24 kernel: [221558.035928] CR2: 00007f2d10190000 
CR3: 0000000053ae8000 CR4: 00000000000006e0
Aug 29 02:54:24 liberty24 kernel: [221558.035928] DR0: 0000000000000000 
DR1: 0000000000000000 DR2: 0000000000000000
Aug 29 02:54:24 liberty24 kernel: [221558.035928] DR3: 0000000000000000 
DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 29 02:54:24 liberty24 kernel: [221558.035928] Process md0_raid5 
(pid: 400, threadinfo ffff880069530000, task ffff88006a08dbc0)
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  ffff88001be85ed8 
0000000000000000 0000000000000000 ffff88000ca0ed80
Aug 29 02:54:24 liberty24 kernel: [221558.035928] <0> ffff8800695ca600 
ffff88001be85ed8 ffff880069531c80 ffffffff8117159d
Aug 29 02:54:24 liberty24 kernel: [221558.035928] <0> ffff880069531cb0 
ffffffffa023d8c9 ffff880069531ca0 ffff88004d8312b0
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffff8117159d>] 
bio_endio+0x1d/0x40
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffffa023d8c9>] 
crypt_dec_pending+0x69/0x90 [dm_crypt]
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffffa023daa8>] 
crypt_endio+0x68/0x150 [dm_crypt]
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffff8117159d>] 
bio_endio+0x1d/0x40
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffffa0200add>] 
handle_stripe5+0x46d/0x9a0 [raid456]
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffffa0201028>] 
handle_stripe+0x18/0x30 [raid456]
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffffa0201432>] 
raid5d+0x202/0x320 [raid456]
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffff814298dc>] 
md_thread+0x5c/0x130
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffff81085090>] 
? autoremove_wake_function+0x0/0x40
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffff81429880>] 
? md_thread+0x0/0x130
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffff81084d16>] 
kthread+0x96/0xa0
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffff810141ea>] 
child_rip+0xa/0x20
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffff81084c80>] 
? kthread+0x0/0xa0
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffff810141e0>] 
? child_rip+0x0/0x20
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  RSP <ffff880069531c40>
Aug 29 02:54:24 liberty24 kernel: [221558.133125] ---[ end trace 

29f7fd9a7bbb5a00 ]---

Using the array uncrypted, data copy to it works just fine, copied over the full 900 GB now 6 times without a hickup.

Can anybody help me with this?

thanks

Thorsten

author
#2 lore raw

Hey Rick,

thx for the quick reply. Alright, I am gonna try a newer kernel next then. I am running the default up2date Lucid kernel right now which is 2.6.32-24-server.

Take care

Thorsten

Quoted 100 lines · Rick Moritz

Had those self-same issues (well, crypted RAID-array io-locking on me during samba-operations on it), and moving to kernel 2.6.34 fixed it for me. Never figured out what was really causing those problems.

Quoted 82 lines · On Fri, 03 Sep 2010 15:22:38 +0200 BOBA FETT<[email protected]> wrote:

Hey folks,

I am switching from loop-aes to dm_crypt. I am trying to use it on a fresh Ubuntu 64bit server installation with my 6x750gb Software raid5 array.
Array is clean and freshly created, I encrypted it using

cryptsetup -y -c aes -h ripemd160 -s 256 create STUFF /dev/md0

Format went fine, but when I try to copy back large amount of data to the encrypted array (about 900gig) my copy thread always hangs itself at some point during the process. I can't access the array anymore, though the raid itself is fine, no drives are kicked out or anything. Reboot gives me back access and the array is still ok.
When the problem occures I see the following trace in log:

Aug 29 02:54:24 liberty24 kernel: [221558.020638] Pid: 400, comm:
md0_raid5 Not tainted 2.6.32-24-server #41-Ubuntu Unknow
Aug 29 02:54:24 liberty24 kernel: [221558.020638] RIP:
0010:[<ffffffff814353b8>]  [<ffffffff814353b8>] clone_endio+0x38/0xe0
Aug 29 02:54:24 liberty24 kernel: [221558.020638] RSP:
0018:ffff880069531c40  EFLAGS: 00010246
Aug 29 02:54:24 liberty24 kernel: [221558.020638] RAX: ffffffffa023f760
RBX: 0000000000000000 RCX: 0100000000000081
Aug 29 02:54:24 liberty24 kernel: [221558.020638] RDX: 0000000000000017
RSI: 0000000000000000 RDI: ffffc9001247e040
Aug 29 02:54:24 liberty24 kernel: [221558.035928] RBP: ffff880069531c70
R08: 0000000000000000 R09: 010000000000282c
Aug 29 02:54:24 liberty24 kernel: [221558.035928] R10: 0000000000000000
R11: 0000000000000001 R12: ffff88000ca0ed80
Aug 29 02:54:24 liberty24 kernel: [221558.035928] R13: ffff88000df85ed0
R14: ffff8800695ca600 R15: ffbf8800116cb348
Aug 29 02:54:24 liberty24 kernel: [221558.035928] FS:
00007fb75e7f4700(0000) GS:ffff880001c40000(0000) knlGS:0000000000000000
Aug 29 02:54:24 liberty24 kernel: [221558.035928] CS:  0010 DS: 0018 ES:
0018 CR0: 000000008005003b
Aug 29 02:54:24 liberty24 kernel: [221558.035928] CR2: 00007f2d10190000
CR3: 0000000053ae8000 CR4: 00000000000006e0
Aug 29 02:54:24 liberty24 kernel: [221558.035928] DR0: 0000000000000000
DR1: 0000000000000000 DR2: 0000000000000000
Aug 29 02:54:24 liberty24 kernel: [221558.035928] DR3: 0000000000000000
DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 29 02:54:24 liberty24 kernel: [221558.035928] Process md0_raid5
(pid: 400, threadinfo ffff880069530000, task ffff88006a08dbc0)
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  ffff88001be85ed8
0000000000000000 0000000000000000 ffff88000ca0ed80
Aug 29 02:54:24 liberty24 kernel: [221558.035928]<0>  ffff8800695ca600
ffff88001be85ed8 ffff880069531c80 ffffffff8117159d
Aug 29 02:54:24 liberty24 kernel: [221558.035928]<0>  ffff880069531cb0
ffffffffa023d8c9 ffff880069531ca0 ffff88004d8312b0
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffff8117159d>]
bio_endio+0x1d/0x40
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffffa023d8c9>]
crypt_dec_pending+0x69/0x90 [dm_crypt]
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffffa023daa8>]
crypt_endio+0x68/0x150 [dm_crypt]
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffff8117159d>]
bio_endio+0x1d/0x40
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffffa0200add>]
handle_stripe5+0x46d/0x9a0 [raid456]
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffffa0201028>]
handle_stripe+0x18/0x30 [raid456]
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffffa0201432>]
raid5d+0x202/0x320 [raid456]
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffff814298dc>]
md_thread+0x5c/0x130
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffff81085090>]
? autoremove_wake_function+0x0/0x40
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffff81429880>]
? md_thread+0x0/0x130
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffff81084d16>]
kthread+0x96/0xa0
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffff810141ea>]
child_rip+0xa/0x20
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffff81084c80>]
? kthread+0x0/0xa0
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  [<ffffffff810141e0>]
? child_rip+0x0/0x20
Aug 29 02:54:24 liberty24 kernel: [221558.035928]  RSP<ffff880069531c40>
Aug 29 02:54:24 liberty24 kernel: [221558.133125] ---[ end trace

29f7fd9a7bbb5a00 ]---

Quoted 8 lines

Using the array uncrypted, data copy to it works just fine, copied over the full 900 GB now 6 times without a hickup.

Can anybody help me with this?

thanks

Thorsten

_______________________________________________
dm-crypt mailing list
[email protected]
http://www.saout.de/mailman/listinfo/dm-crypt