SW RAID5 + high memory support freezes 2.6.3 kernel

Hello,
issue http://www.spinics.net/lists/lvm/msg10322.html could be still present in the current 2.6.3 kernel. I am able to repeat the conditions to halt the 2.6.3 kernel (using mkfs.ext3 on RAID device):

Feb 23 02:52:12 psilocybus kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000008
Feb 23 02:52:12 psilocybus kernel: printing eip:
Feb 23 02:52:12 psilocybus kernel: f9885205
Feb 23 02:52:12 psilocybus kernel: *pde = 00000000
Feb 23 02:52:12 psilocybus kernel: Oops: 0000 [#1]
Feb 23 02:52:12 psilocybus kernel: CPU:    0
Feb 23 02:52:12 psilocybus kernel: EIP:    0060:[<f9885205>]    Tainted: PF 
Feb 23 02:52:12 psilocybus kernel: EFLAGS: 00010246
Feb 23 02:52:12 psilocybus kernel: EIP is at make_request+0x5/0x210 [raid5]
Feb 23 02:52:12 psilocybus kernel: eax: f10e2f2b   ebx: f6631800   ecx: f7fe9040   edx: 00001000
Feb 23 02:52:12 psilocybus kernel: esi: 00000008   edi: c644dbc0   ebp: 025c0e08   esp: c1b8bcd0
Feb 23 02:52:12 psilocybus kernel: ds: 007b   es: 007b   ss: 0068
Feb 23 02:52:12 psilocybus kernel: Process pdflush (pid: 6, threadinfo=c1b8a000 task=c1b8d900)
Feb 23 02:52:12 psilocybus kernel: Stack: c021aea6 f6631800 c644dbc0 e79694f0 f9857b01 c19f94a8 00000001 ffb28000 
Feb 23 02:52:12 psilocybus kernel: db022000 00001000 025c0e10 0000000c 00000001 00000001 00000000 025c0e10 
Feb 23 02:52:12 psilocybus kernel: f66e4000 00000000 c644dd40 c644dbc0 f9857c03 c644dbc0 c644dbc0 c644dd40 
Feb 23 02:52:12 psilocybus kernel: Call Trace:
Feb 23 02:52:12 psilocybus kernel: [<c021aea6>] generic_make_request+0x106/0x180
Feb 23 02:52:12 psilocybus kernel: [<f9857b01>] loop_transfer_bio+0xc1/0x120 [loop]
Feb 23 02:52:12 psilocybus kernel: [<f9857c03>] loop_make_request+0xa3/0x150 [loop]
Feb 23 02:52:12 psilocybus kernel: [<c021aea6>] generic_make_request+0x106/0x180
Feb 23 02:52:12 psilocybus kernel: [<c015bb2b>] bio_alloc+0xcb/0x1a0
Feb 23 02:52:12 psilocybus kernel: [<c021af5d>] submit_bio+0x3d/0x70
Feb 23 02:52:12 psilocybus kernel: [<c0159a10>] __block_write_full_page+0x1d0/0x3d0
Feb 23 02:52:12 psilocybus kernel: [<c018fd13>] ext3_journal_dirty_data+0x23/0x60
Feb 23 02:52:12 psilocybus kernel: [<c015b244>] block_write_full_page+0xe4/0xf0
Feb 23 02:52:12 psilocybus kernel: [<c015df90>] blkdev_get_block+0x0/0x50
Feb 23 02:52:12 psilocybus kernel: [<c015e0ef>] blkdev_writepage+0x1f/0x30
Feb 23 02:52:12 psilocybus kernel: [<c015df90>] blkdev_get_block+0x0/0x50
Feb 23 02:52:12 psilocybus kernel: [<c017840e>] mpage_writepages+0x20e/0x300
Feb 23 02:52:12 psilocybus kernel: [<c017840e>] mpage_writepages+0x20e/0x300
Feb 23 02:52:12 psilocybus kernel: [<c015e0d0>] blkdev_writepage+0x0/0x30
Feb 23 02:52:12 psilocybus kernel: [<c015f31f>] generic_writepages+0x1f/0x23
Feb 23 02:52:12 psilocybus kernel: [<c013fdee>] do_writepages+0x1e/0x40
Feb 23 02:52:12 psilocybus kernel: [<c0176b54>] __sync_single_inode+0xd4/0x230
Feb 23 02:52:12 psilocybus kernel: [<c0176efe>] sync_sb_inodes+0x19e/0x260
Feb 23 02:52:12 psilocybus kernel: [<c017700d>] writeback_inodes+0x4d/0xa0
Feb 23 02:52:12 psilocybus kernel: [<c013fb2b>] background_writeout+0x7b/0xc0
Feb 23 02:52:12 psilocybus kernel: [<c0140212>] __pdflush+0xd2/0x1d0
Feb 23 02:52:12 psilocybus kernel: [<c0140310>] pdflush+0x0/0x20
Feb 23 02:52:12 psilocybus kernel: [<c014031f>] pdflush+0xf/0x20
Feb 23 02:52:12 psilocybus kernel: [<c013fab0>] background_writeout+0x0/0xc0
Feb 23 02:52:12 psilocybus kernel: [<c0109284>] kernel_thread_helper+0x0/0xc
Feb 23 02:52:12 psilocybus kernel: [<c0109289>] kernel_thread_helper+0x5/0xc
Feb 23 02:52:12 psilocybus kernel: 
Feb 23 02:52:12 psilocybus kernel: Code: a7 c0 09 b6 d3 db 66 06 71 67 67 d7 32 47 2a 92 23 22 ee b1 
Feb 23 02:52:12 psilocybus kernel: int3: 0000 [#2]
Feb 23 02:52:12 psilocybus kernel: CPU:    0
Feb 23 02:52:12 psilocybus kernel: EIP:    0060:[<f9885147>]    Tainted: PF 
Feb 23 02:52:12 psilocybus kernel: EFLAGS: 00000217
Feb 23 02:52:12 psilocybus kernel: EIP is at raid5_unplug_device+0x7/0xc0 [raid5]
Feb 23 02:52:12 psilocybus kernel: eax: f6631765   ebx: 00000212   ecx: c1bd0a10   edx: c1bd0a00
Feb 23 02:52:12 psilocybus kernel: esi: f66318f4   edi: f66318f0   ebp: c1ba2000   esp: c1ba3f6c
Feb 23 02:52:12 psilocybus kernel: ds: 007b   es: 007b   ss: 0068
Feb 23 02:52:12 psilocybus kernel: Process kblock0 f25f3ee0 f5e326c0 00000001 00000000 
Feb 23 02:52:12 psilocybus kernel: f7dc15c0 00d8cfe8 ffffe000 f25f2000 f9892378 f7dc15c0 00d8cfe8 00000001 
Feb 23 02:52:12 psilocybus kernel: f7dc1664 00004000 00d8cf00 00000008 0000011d 0df83a00 f98981bc c011b25e 
Feb 23 02:52:12 psilocybus kernel: Call Trace:
Feb 23 02:52:12 psilocybus kernel: [<f9892378>] md_do_sync+0x228/0x780 [md]
Feb 23 02:52:12 psilocybus kernel: [<c011b25e>] recalc_task_prio+0x8e/0x1b0
Feb 23 02:52:12 psilocybus kernel: [<f98912a6>] md_thread+0xc6/0x1a0 [md]

So, RAID5 works for me only without high memory support.

Pavol

_____________________________________________________________________________
[[email protected]] [http://trip.sk/wilder/] [talker: ttt.sk 5678] [ICQ: 133403556]

To be fair, your subject should say that
SW RAID5 + high memory + loop device freezes 2.6.3 kernel

                          ^^^^^^^^^^^^^^

Would you be able to try the same tes without using "loop" in the middle and see what happens?

The trace you sent:

seems to suggest that %esi is being dereferenced at make_request+0x5, but when I disassemble my raid5.o, it doesn't.

I tried disassembling the code:

but that just produced nonsense.

Could you
gdb raid5.o
disassemble make_request

and send me the output please.

NeilBrown

hm, yes. And the loop code which was involved here was removed from the kernel last week, so it's a bit academic.

Retest on current 2.6.3-bk or 2.6.3-mm3 please.

I tried 2.6.3-bk4 and seems it works without problems :)
I'm going to copy about 300 GB data to this cryptoloop/RAID5 and report the eventual bugs....

Pavol

_____________________________________________________________________________
[[email protected]] [http://trip.sk/wilder/] [talker: ttt.sk 5678] [ICQ: 133403556]

Pavol Luptak <[email protected]> writes:

[...]

same here with 2.6.3-mm3 + some dm patches i need.

S.

Neil,

Currently, the device size calculation is not correct when hot-adding devices to arrays with non-persistent superblocks. Device size is always calculated as if there were a physical superblock on every device. The attached simple change to hot_add_disk() fixes the problem.

Thanks,
Paul

@@ -2365,7 +2365,12 @@ static int hot_add_disk(mddev_t * mddev,
 		return -EINVAL;
 	}
 
-	rdev->sb_offset = calc_dev_sboffset(rdev->bdev);
+	if (mddev->persistent)
+		rdev->sb_offset = calc_dev_sboffset(rdev->bdev);
+	else
+		rdev->sb_offset = rdev->bdev->bd_inode->i_size
+							>> BLOCK_SIZE_BITS;
+
 	size = calc_dev_size(rdev, mddev->chunk_size);
 	rdev->size = size;
 

Hmm.. I had always assumed that non-persistent superblocks only worked for linear and raid0. I'm not sure I would trust any other configuration.

Are you seriously using raid1 with non-persistent superblocks? How do you ensure reliable re-assembly after a device failure followed by shutdown?

NeilBrown

Yes, and this works fine in 2.4, as well. We opted for non-persistent superblocks in order to support creation of raid1 mirrors over partitions that already had filesystems or other data present (and yes, we calculate the size of the new md device to make sure the existing data will fit).

There is a user-level clustering framework that sets up, monitors, and takes down the mirror. Disk failures and other events are handled by this framework.

--
Paul

The attached patch makes sure that resync/recovery get aborted (and status recorded correctly) when there are no spare drives left (due to the failure of a spare during a resync). Previously, if a spare failed during a resync, the resync would continue until completed (and would appear to be successful).

Also, there was an erroneous usage of master_bio->bi_bdev field (which is not always properly set) that has been changed to the simpler comparison vs. r1_bio->read_disk (as is done in raid1_end_request). I think Neil has already addressed this issue in another patch that has been pushed to Andrew...

Thanks,
Paul

@@ -818,6 +818,8 @@ static void sync_request_write(mddev_t *
 		put_buf(r1_bio);
 		return;
 	}
+	/* assume failure until we find a drive to write this to */
+	clear_bit(R1BIO_Uptodate, &r1_bio->state);
 
 	spin_lock_irq(&conf->device_lock);
 	for (i = 0; i < disks ; i++) {

@@ -825,7 +827,7 @@ static void sync_request_write(mddev_t *
 		if (!conf->mirrors[i].rdev || 
 		    conf->mirrors[i].rdev->faulty)
 			continue;
-		if (conf->mirrors[i].rdev->bdev == bio->bi_bdev)
+		if (i == r1_bio->read_disk)
 			/*
 			 * we read from here, no need to write
 			 */

@@ -838,6 +840,8 @@ static void sync_request_write(mddev_t *
 			continue;
 		atomic_inc(&conf->mirrors[i].rdev->nr_pending);
 		r1_bio->write_bios[i] = bio;
+		/* we found a drive to write to */
+		set_bit(R1BIO_Uptodate, &r1_bio->state);
 	}
 	spin_unlock_irq(&conf->device_lock);
 

@@ -859,7 +863,8 @@ static void sync_request_write(mddev_t *
 	}
 
 	if (atomic_dec_and_test(&r1_bio->remaining)) {
-		md_done_sync(mddev, r1_bio->master_bio->bi_size >> 9, 1);
+		md_done_sync(mddev, r1_bio->master_bio->bi_size >> 9,
+				test_bit(R1BIO_Uptodate, &r1_bio->state));
 		put_buf(r1_bio);
 	}
 }

Hi,
could you clarify exactly what the problem is.

As far as I can see (without looking very deeply) the only problem is that the resync process continues on when there is no point doing so, thus wasting time.
At the end of the process, any devices that failed should still be marked failed and so the array will be insync, but might be degraded
- but there is no risk to data.

Am I missing something?

(I"m not saying I don't like the patch - it looks like a good idea - I just want to be sure I understand it).

NeilBrown

Yes, that's right. The pointless continuance of the resync is the problem. The bad device does get marked as "failed", so there is no problem with that.

--
Paul