raid5 crash
I am running kernel 2.6.9-gentoo-r10 on an Athlon XP 2400+ computer with a SiI 3114 SATA controller hosting 4 WD2500JD-00G drives. I have combined these drives into a raid5 array using software raid, but unfortunately the array is not stable. I have tried several filesystems (ext3, reiserfs, xfs), but after copying several gigabytes of data into the array (using scp) and then trying to read them back (using rsync to compare over the network) always results in data corruption. Here is the output from 'dmesg':
kernel BUG at drivers/md/raid5.c:813!
invalid operand: 0000 [#1]
Modules linked in: sata_sil libata sbp2 ohci1394 ieee1394 usb_storage ehci_hcd usbcore CPU: 0 EIP: 0060:[<c039cdd2>] Not tainted VLI EFLAGS: 00010006 (2.6.9-gentoo-r10)
EIP is at add_stripe_bio+0x1c2/0x200
eax: 00045168 ebx: d3974b00 ecx: d3974980 edx: 00000000
esi: 00045140 edi: 00000000 ebp: e33200a4 esp: f0a05ac4
ds: 007b es: 007b ss: 0068
Process rsync (pid: 32092, threadinfo=f0a04000 task=f6c10020)
Stack: 00000000 00000296 00000140 e3320028 00045140 00000000 d3974980 c039e092
e3320028 d3974980 00000000 00000000 00000000 f0a05b1c de3e1ae0 00045158
00000000 00000003 00000004 de3e1ae0 dfe90e00 00000000 00000003 f7d85088
Call Trace:
[<c039e092>] make_request+0x122/0x200
[<c032dc6f>] generic_make_request+0x15f/0x1e0
[<c011d590>] autoremove_wake_function+0x0/0x60
[<c032dd4d>] submit_bio+0x5d/0x100
[<c0172d43>] mpage_bio_submit+0x23/0x40
[<c0173170>] do_mpage_readpage+0x2d0/0x480
[<c012367d>] __do_softirq+0x7d/0x90
[<c02b54df>] radix_tree_node_alloc+0x1f/0x60
[<c02b5762>] radix_tree_insert+0xe2/0x100
[<c0136e54>] add_to_page_cache+0x54/0x80
[<c017346b>] mpage_readpages+0x14b/0x180
[<c018f1f0>] reiserfs_get_block+0x0/0x1450
[<c013ddf4>] read_pages+0x134/0x140
[<c018f1f0>] reiserfs_get_block+0x0/0x1450
[<c013b390>] __alloc_pages+0x1d0/0x370
[<c01081c5>] do_IRQ+0xc5/0xe0
[<c013e04f>] do_page_cache_readahead+0xcf/0x130
[<c013e19f>] page_cache_readahead+0xef/0x1e0
[<c013765c>] do_generic_mapping_read+0x11c/0x4d0
[<c0137cae>] __generic_file_aio_read+0x1be/0x1f0
[<c0137a10>] file_read_actor+0x0/0xe0
[<c0137e1a>] generic_file_read+0xba/0xe0
[<c011ac24>] do_page_fault+0x194/0x591
[<c011d590>] autoremove_wake_function+0x0/0x60
[<c0126f6b>] update_wall_time+0xb/0x40
[<c012739f>] do_timer+0xdf/0xf0
[<c015270c>] vfs_read+0xbc/0x170
[<c012367d>] __do_softirq+0x7d/0x90
[<c0152a71>] sys_read+0x51/0x80
[<c010603b>] syscall_call+0x7/0xb
Code: 72 08 0f ba a8 90 00 00 00 02 83 c4 0c 5b 5e 5f 5d c3 89 cb e9 cd fe ff ff 8b 5d 00 e9 c5 fe ff ff 77 08 39 f0 0f 86 94 fe ff ff <0f> 0b 2d 0370 92 44 c0 e9 87 fe ff ff 0f 87 a8 fe ff ff 39 f0
Any idea whether this is a kernel bug or a hardware problem? Please CC any replies to me.
Sincerely,
Signature · 1 line
Kristian
The sii 3114 is a RAID controller by itself.
Is in not conflicting somewhere (like running software RAID5 and at the same time hardware RAID X?)
Quoted 63 lines · you
I am running kernel 2.6.9-gentoo-r10 on an Athlon XP 2400+ computer with a SiI 3114 SATA controller hosting 4 WD2500JD-00G drives. I have combined these drives into a raid5 array using software raid, but unfortunately the array is not stable. I have tried several filesystems (ext3, reiserfs, xfs), but after copying several gigabytes of data into the array (using scp) and then trying to read them back (using rsync to compare over the network) always results in data corruption. Here is the output from 'dmesg':
kernel BUG at
drivers/md/raid5.c:813!
invalid operand: 0000 [#1]Modules linked in: sata_sil libata sbp2 ohci1394 ieee1394 usb_storage ehci_hcd usbcore CPU: 0 EIP: 0060:[<c039cdd2>] Not tainted VLI EFLAGS: 00010006 (2.6.9-gentoo-r10)EIP is at
add_stripe_bio+0x1c2/0x200
eax: 00045168 ebx:d3974b00ecx:d3974980edx: 00000000
esi: 00045140 edi: 00000000 ebp:e33200a4esp:f0a05ac4
ds: 007b es: 007b ss: 0068
Process rsync (pid: 32092, threadinfo=f0a04000 task=f6c10020)Stack: 00000000 00000296 00000140 e3320028 00045140 00000000 d3974980 c039e092 e3320028 d3974980 00000000 00000000 00000000 f0a05b1c de3e1ae0 00045158 00000000 00000003 00000004 de3e1ae0 dfe90e00 00000000 00000003 f7d85088 Call Trace: [<c039e092>] make_request+0x122/0x200 [<c032dc6f>] generic_make_request+0x15f/0x1e0 [<c011d590>] autoremove_wake_function+0x0/0x60 [<c032dd4d>] submit_bio+0x5d/0x100 [<c0172d43>] mpage_bio_submit+0x23/0x40 [<c0173170>] do_mpage_readpage+0x2d0/0x480 [<c012367d>] __do_softirq+0x7d/0x90 [<c02b54df>] radix_tree_node_alloc+0x1f/0x60 [<c02b5762>] radix_tree_insert+0xe2/0x100 [<c0136e54>] add_to_page_cache+0x54/0x80 [<c017346b>] mpage_readpages+0x14b/0x180 [<c018f1f0>] reiserfs_get_block+0x0/0x1450 [<c013ddf4>] read_pages+0x134/0x140 [<c018f1f0>] reiserfs_get_block+0x0/0x1450 [<c013b390>] __alloc_pages+0x1d0/0x370 [<c01081c5>] do_IRQ+0xc5/0xe0 [<c013e04f>] do_page_cache_readahead+0xcf/0x130 [<c013e19f>] page_cache_readahead+0xef/0x1e0 [<c013765c>] do_generic_mapping_read+0x11c/0x4d0 [<c0137cae>] __generic_file_aio_read+0x1be/0x1f0 [<c0137a10>] file_read_actor+0x0/0xe0 [<c0137e1a>] generic_file_read+0xba/0xe0 [<c011ac24>] do_page_fault+0x194/0x591 [<c011d590>] autoremove_wake_function+0x0/0x60 [<c0126f6b>] update_wall_time+0xb/0x40 [<c012739f>] do_timer+0xdf/0xf0 [<c015270c>] vfs_read+0xbc/0x170 [<c012367d>] __do_softirq+0x7d/0x90 [<c0152a71>] sys_read+0x51/0x80 [<c010603b>] syscall_call+0x7/0xb Code: 72 08 0f ba a8 90 00 00 00 02 83 c4 0c 5b 5e 5f 5d c3 89 cb e9 cd feff ff 8b 5d 00 e9 c5 fe ff ff 77 08 39 f0 0f 86 94 fe ff ff <0f> 0b 2d 0370 92 44 c0 e9 87 fe ff ff 0f 87 a8 fe ff ff 39 f0
Any idea whether this is a kernel bug or a hardware problem? Please CC any replies to me.
Sincerely,
Quoted 3 lines
The sii 3114 is a RAID controller by itself.
Is in not conflicting somewhere (like running software RAID5 and at the same time hardware RAID X?)
No. The SiI 3114 is only being used as an SATA controller; I have not configured any hardware raid.
Sincerely,
Signature · 1 line
Kristian
Quoted 9 lines · On Wednesday December 22, [email protected] wrote:
I am running kernel 2.6.9-gentoo-r10 on an Athlon XP 2400+ computer with a SiI 3114 SATA controller hosting 4 WD2500JD-00G drives. I have combined these drives into a raid5 array using software raid, but unfortunately the array is not stable. I have tried several filesystems (ext3, reiserfs, xfs), but after copying several gigabytes of data into the array (using scp) and then trying to read them back (using rsync to compare over the network) always results in data corruption. Here is the output from 'dmesg':
kernel BUG at
drivers/md/raid5.c:813!
This BUG happens when there are two outstanding read (or write) requests for the same piece of storage (more accurately, two "bio"s that overlap).
raid5 cannot currently handle this situation.
Most filesystems would never make requests like this.
I note that you are using reiserfs in this case. It is possible that reiserfs with tail-packing enabled could do this.
I doubt very much that this would happen with ext3. I don't know about xfs, but I doubt it would happen their either.
When using some other filesystem, what sort of data corruption are you getting?
NeilBrown
Quoted 5 lines · Neil Brown
Quoted 3 lines · On Wednesday December 22, [email protected] wrote:
I am running kernel 2.6.9-gentoo-r10 on an Athlon XP 2400+ computer with a SiI 3114 SATA controller hosting 4 WD2500JD-00G drives. I have combined these drives into a raid5 array using software raid, but unfortunately the array is not stable. I have tried several filesystems (ext3, reiserfs, xfs), but after copying several gigabytes of data into the array (using scp) and then trying to read them back (using rsync to compare over the network) always results in data corruption. Here is the output from 'dmesg':
kernel BUG at
drivers/md/raid5.c:813!
Have you a bios option called ext-p2p discard time? Try setting it higher. I posted another thread about sii3112 at lkml about this issue...
Prakash
Quoted 4 lines
I doubt very much that this would happen with ext3. I don't know about xfs, but I doubt it would happen their either.
When using some other filesystem, what sort of data corruption are you getting?
This is with ext3:
kernel BUG at drivers/md/raid5.c:813!
invalid operand: 0000 [#1]
Modules linked in: sata_sil libata sbp2 ohci1394 ieee1394 usb_storage ehci_hcd usbcore CPU: 0 EIP: 0060:[<c039cdd2>] Not tainted VLI EFLAGS: 00010016 (2.6.9-gentoo-r10)
EIP is at add_stripe_bio+0x1c2/0x200
eax: 493c4b40 ebx: cbc5eaa0 ecx: e8257da0 edx: 00000000
esi: 493c4b18 edi: 00000000 ebp: f58958b0 esp: f5995a98
ds: 007b es: 007b ss: 0068
Process rsync (pid: 8803, threadinfo=f5994000 task=d841aaa0)
Stack: c01551f7 f7ddf200 00000118 f58957c8 493c4b18 00000000 e8257da0 c039e092
f58957c8 e8257da0 00000001 00000000 00000000 f5995af0 f5fde2e0 493c4b30
00000000 00000003 00000004 f5fde2e0 dfe90e00 00000001 00000000 f7d84088
Call Trace:
[<c01551f7>] __getblk+0x37/0x70
[<c039e092>] make_request+0x122/0x200
[<c032dc6f>] generic_make_request+0x15f/0x1e0
[<c032dd4d>] submit_bio+0x5d/0x100
[<c01b5ec4>] ext3_get_block+0x64/0xb0
[<c0172d43>] mpage_bio_submit+0x23/0x40
[<c0173170>] do_mpage_readpage+0x2d0/0x480
[<c0332b08>] as_next_request+0x38/0x50
[<c032a706>] elv_next_request+0x16/0x110
[<c02b4c1e>] kobject_put+0x1e/0x30
[<c02b4bf0>] kobject_release+0x0/0x10
[<c02b54df>] radix_tree_node_alloc+0x1f/0x60
[<c02b5762>] radix_tree_insert+0xe2/0x100
[<c0136e54>] add_to_page_cache+0x54/0x80
[<c017346b>] mpage_readpages+0x14b/0x180
[<c01b5e60>] ext3_get_block+0x0/0xb0
[<c013ddf4>] read_pages+0x134/0x140
[<c01b5e60>] ext3_get_block+0x0/0xb0
[<c013b390>] __alloc_pages+0x1d0/0x370
[<c013e04f>] do_page_cache_readahead+0xcf/0x130
[<c013e234>] page_cache_readahead+0x184/0x1e0
[<c013765c>] do_generic_mapping_read+0x11c/0x4d0
[<c0137cae>] __generic_file_aio_read+0x1be/0x1f0
[<c0137a10>] file_read_actor+0x0/0xe0
[<c02b4c1e>] kobject_put+0x1e/0x30
[<c0137d3a>] generic_file_aio_read+0x5a/0x80
[<c015261e>] do_sync_read+0xbe/0xf0
[<f92cdc66>] ata_qc_complete+0x46/0xe0 [libata]
[<c011d590>] autoremove_wake_function+0x0/0x60
[<c036b9aa>] scsi_finish_command+0x7a/0xc0
[<c011bb9f>] recalc_task_prio+0x8f/0x190
[<c015270c>] vfs_read+0xbc/0x170
[<c040e363>] schedule+0x2a3/0x470
[<c0152a71>] sys_read+0x51/0x80
[<c010603b>] syscall_call+0x7/0xb
Code: 72 08 0f ba a8 90 00 00 00 02 83 c4 0c 5b 5e 5f 5d c3 89 cb e9 cd fe ff ff 8b 5d 00 e9 c5 fe ff ff 77 08 39 f0 0f 86 94 fe ff ff <0f> 0b 2d 0370 92 44 c0 e9 87 fe ff ff 0f 87 a8 fe ff ff 39 f0
So apparently it can happen with ext3 as well.
Signature · 1 line
Kristian
Re: raid5 crash (possible VM problem???)
Quoted 8 lines · On Thursday December 23, [email protected] wrote:
Quoted 4 lines
I doubt very much that this would happen with ext3. I don't know about xfs, but I doubt it would happen their either.
When using some other filesystem, what sort of data corruption are you getting?This is with ext3:
kernel BUG at
drivers/md/raid5.c:813!
This is very suspect...
This BUG is triggered if raid5 receives a read request while there is already an outstanding read request that overlaps the new one. i.e. this->sector < that->sector and
this->sector + this->size > that->sector
I admit that my understanding of ext3 isn't perfect, but I would be VERY surprised if ext3 would ever submit overlapping requests.
I guess it could happen if the filesystem were corrupted (and two files claimed to own the same block) but I doubt that is the case here.
I suspect there is a problem somewhere else.. in the VM maybe??
You could try this patch. It might hide the real problem, or it might cause it to manifest in some other way.
It changes the raid5 behaviour so that if an overlap is found, it doesn't BUG, but instead waits a little while and tries again.
NeilBrown
### Diffstat output
./drivers/md/raid5.c | 18 ++++++++++++++---- 1 files changed, 14 insertions(+), 4 deletions(-)
drivers/md/raid5.c renamed from drivers/md/raid5.c~current~ +14 -4
@@ -232,6 +232,7 @@ static struct stripe_head *__find_stripe }static void unplug_slaves (mddev_t *mddev); +static void raid5_unplug_device (request_queue_t *q);static struct stripe_head *get_active_stripe (raid5_conf_t *conf,sector_t sector,int pd_idx,int noblock)
@@ -793,7 +794,7 @@ static void compute_parity(struct stripe* toread/towrite point to the first in a chain. * The bi_next chain must be in order. */ -static void add_stripe_bio (struct stripe_head *sh,struct bio *bi,int dd_idx,int forwrite) +static int add_stripe_bio (struct stripe_head *sh,struct bio *bi,int dd_idx,int forwrite) {struct bio **bip;raid5_conf_t *conf = sh->raid_conf;
@@ -810,10 +811,10 @@ static void add_stripe_bio (struct stripelse bip = &sh->dev[dd_idx].toread;while (*bip && (*bip)->bi_sector < bi->bi_sector) { -BUG_ON ((*bip)->bi_sector + ((*bip)->bi_size >>9 ) > bi->bi_sector); +if ((*bip)->bi_sector + ((*bip)->bi_size >>9 ) > bi->bi_sector) +return 0 ;/* cannot add just now due to overlap */ bip = & (*bip)->bi_next; } -/* FIXME do I need to worry about overlapping bion */ if (*bip && bi->bi_next && (*bip) != bi->bi_next)BUG ();if (*bip)
@@ -840,6 +841,7 @@ static void add_stripe_bio (struct stripif (sector >= sh->dev[dd_idx].sector +STRIPE_SECTORS )set_bit (R5_OVERWRITE , &sh->dev[dd_idx].flags); } +return 1 ; }
@@ -1413,7 +1415,15 @@ static int make_request (request_queue_t sh =get_active_stripe (conf, new_sector, pd_idx, (bi->bi_rw&RWA_MASK ));if (sh) { -add_stripe_bio (sh, bi, dd_idx, (bi->bi_rw&RW_MASK )); +while (!add_stripe_bio (sh, bi, dd_idx, (bi->bi_rw&RW_MASK ))) { +/* add failed due to overlap. Flush everything +* and wait a while +* FIXME - overlapping requests should be handled better +*/ +raid5_unplug_device (mddev->queue); +set_current_state (TASK_UNINTERRUPTIBLE ); +schedule_timeout (1 ); + }raid5_plug_device (conf);handle_stripe (sh);
Re: raid5 crash (possible VM problem???)
Quoted 2 lines
You could try this patch. It might hide the real problem, or it might cause it to manifest in some other way.
I've applied the patch, but I now get another error; this might not be related to raid5, however, I have tested the individual SATA disks without getting any errors. This only happens when combining them into a raid5 array (other raid levels not tested).
ReiserFS: md3: journal params: device md3, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30
ReiserFS: md3: checking transaction log (md3)
ReiserFS: md3: Using r5 hash to sort names
attempt to access beyond end of device
md3: rw=0, want=18446744063991695384, limit=1465175040
attempt to access beyond end of device
md3: rw=0, want=18446744062355374704, limit=1465175040
attempt to access beyond end of device
md3: rw=0, want=4913837584, limit=1465175040
attempt to access beyond end of device
md3: rw=0, want=18446744071656162744, limit=1465175040
I have 4 250GB SATA disk combined into one raid5 volume (kernel 2.6.10), and this error happens after copying a few gigabytes of data into the volume and then trying to read them back.
Signature · 1 line
Kristian
Hi,
Quoted 5 lines · Neil Brown
Quoted 1 line
kernel BUG at
drivers/md/raid5.c:813!This BUG happens when there are two outstanding read (or write) requests for the same piece of storage (more accurately, two "bio"s that overlap).
Ouch.
Quoted 2 lines
raid5 cannot currently handle this situation. Most filesystems would never make requests like this.
I don't think there's anything to stop me doing two O_DIRECT read()s from the same bit of a file at the same time. As far as I can see, this should be easily triggered by user-space.
And if you get corruption in a filesystem such that two files share the same block, then this possibility arises again. That can happen due to corruption in an indirect block (ext2/3) or in the reiserfs tree; or more commonly due to a bitmap block getting corrupted, resulting in the same block being allocated twice.
This is a situation we really need to handle. ext3 goes to great lengths to make sure that if such cases happen, the worst that results should be the filesystem taking itself readonly cleanly.
It's really bad behaviour for a fault such as a bad IDE cable to be able to oops the entire kernel.
Quoted 1 line
I doubt very much that this would happen with ext3.
It certainly shouldn't do so for buffered IO if things are running fine, but as soon as you get corrupt data, or start using O_DIRECT, it's possible.
Cheers,
Stephen