bio too big device md1 (16 > 8)
Hi,
i'm using 2.6.21-rc5-git9 +
http://www.kernel.org/pub/linux/kernel/people/agk/patches/2.6/editing/dm-merge-max_hw_sector.patch
( i've been testing with and without it, and first encountered it on 2.6.18-debian )
I've setup a raid1 array md1 (it was created in a degraded mode using the debian installer)
(md0 is also a small raid1 array created in degraded mode, but i did not have any issue with it)
md1 hold a lvm physical volume holding a vg and several lvs
mdadm -D /dev/md1:/dev/md1:
Version : 00.90.03
Creation Time : Sun Mar 25 16:34:42 2007
Raid Level : raid1
Array Size : 290607744 (277.15 GiB 297.58 GB)
Device Size : 290607744 (277.15 GiB 297.58 GB) Raid Devices : 2
Total Devices : 1
Preferred Minor : 1
Persistence : Superblock is persistent
Update Time : Tue Apr 3 01:37:23 2007
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0
UUID : af8d2807:e573935d:04be1e12:bc7defbb
Events : 0.422096
Number Major Minor RaidDevice State
0 3 3 0 active sync /dev/hda3
1 0 0 1 removed
the problem i'm encountering is when i add /dev/md2 to /dev/md1.
mdadm -D /dev/md2/dev/md2:
Version : 00.90.03
Creation Time : Sun Apr 1 15:06:43 2007
Raid Level : linear
Array Size : 290607808 (277.15 GiB 297.58 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 2
Persistence : Superblock is persistent
Update Time : Sun Apr 1 15:06:43 2007
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Rounding : 64K
UUID : 887ecdeb:5f205eb6:4cd470d6:4cbda83c (local to host odo)
Events : 0.1
Number Major Minor RaidDevice State
0 34 4 0 active sync /dev/hdg4
1 57 2 1 active sync /dev/hdk2
2 91 3 2 active sync /dev/hds3
3 89 2 3 active sync /dev/hdo2
I use mdadm --manage --add /dev/md1 /dev/md2
when I do so here is what happen:
md: bind<md2>
RAID1 conf printout:
--- wd:1 rd:2
disk 0, wo:0, o:1, dev:hda3
disk 1, wo:1, o:1, dev:md2
md: syncing RAID array md1
md: minimum _guaranteed_ reconstruction speed: 1000 KB/sec/disc.
md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reconstruction.
md: using 128k window, over a total of 290607744 blocks.
bio too big device md1 (16 > 8)
Device dm-7, XFS metadata write error block 0x243ec0 in dm-7
bio too big device md1 (16 > 8)
I/O error in filesystem ("dm-8") meta-data dev dm-8 block 0x1b5b6550 ("xfs_trans_read_buf") error 5 buf count 8192
bio too big device md1 (16 > 8)
I/O error in filesystem ("dm-8") meta-data dev dm-8 block 0x1fb3b00 ("xfs_trans_read_buf") error 5 buf count 8192
every filesystems on md1 get corrupted.
I manually fail md2 then reboot and so i can boot the fs again. (but md1 is still degraded)
Any idea ?
I can provide more information if needed. (the only weird thing is /dev/hdo that doesn't seem to be lba48-ready, but i guess that shouldn't be a geometry issue.)
This is difficult.
Summary of problem is:
Filesystem on LVM on md/raid1
Add an md/linear to the md/raid1 and fs dies with 'bio too big'.
Where to begin....
The fs level builds bios to send down to the device, and it queries the device to find out how big the bio can be.
There are two ways it can query the device
1/ it can look at the max_sectors field in the queue structure. 2/ it can call the merge_bvec_fn function in the queue structure.
It actually does both. max_sectors sets an over-all maximum, and merge_bvec_fn - if defined - can ACK or NAK each page being added.
It is currently very awkward for a stacked device to respond to a merge_bvec_fn by calling the merge_bvec_fn of the underlying devices. So they don't.
I'm not sure what dm does, but md simply sets max_sectors to 1 page if there is a merge_bvec_fn in an underlying device.
So in this particular case, as md/linear defines a merge_bvec_fn, as soon as the md/linear array is added to the md/raid1 array, the max_sectors of the md/raid1 is set to 1 page.
This works ok when a filesystem is on an md/raid1 as the filesystem check max_sectors every time before building the bio.
However in your case, dm (aka LVM) is in there too.
dm doesn't know that md/raid1 has just changed max_sectors and there is no convenient way for it to find out. So when the filesystem tries to get the max_sectors for the dm device, it gets the value that dm set up when it was created, which was somewhat larger than one page. When the request gets down to the raid1 layer, it caused a problem.
You can probably make your array work by adding the md/linear array to the md/raid1 *before* enabling the LVM device on top of it. However that isn't really a good long-term solution.
We really need better ways for stacked devices to communicate.
Alasdair Kergon (dm developer) has some patches to make it practical to recursively call merge_bvec_fn so they might be part of the solution, but more discussion is needed before that becomes a reality.
Sorry I cannot be more helpful at this stage.
NeilBrown
Quoted 11 lines · Neil Brown
This is difficult.
Summary of problem is:
Filesystem on LVM on md/raid1
Add an md/linear to the md/raid1 and fs dies with 'bio too big'.[...]
You can probably make your array work by adding the md/linear array to the md/raid1 *before* enabling the LVM device on top of it. However that isn't really a good long-term solution.
Thanks for the explanations !
That's the kind of solution i did use. But it seems I achieved to add the md/linear array to the md/raid1 after lvm was enabled, I only have to not to use an initrd.
the kernel I'm using at the moment is a 2.6.18-4-vserver-xen (debian one) plus the following patches:
- dm-io-fix-bi_max_vecs.patch
- dm-merge-max_hw_sector.patch
also i'm using #define RESYNC_BLOCK_SIZE PAGE_SIZE rather than (64*1024) in raid1.c.
Not sure if that could explain the fact i was able to add the md/linear array to the md/raid1 after enabling lvm. (still i had not to use an initrd)
When I have more time I'll try to reproduce this on a test machine.
Quoted 5 lines
We really need better ways for stacked devices to communicate.
Alasdair Kergon (dm developer) has some patches to make it practical to recursively call
merge_bvec_fnso they might be part of the solution, but more discussion is needed before that becomes a reality.
Ok, i'd be curious to test them (if they're usable and publicly available)
Quoted 1 line
Sorry I cannot be more helpful at this stage.
Thanks. (at least i understand(-ish) what's going on) ;-)