Buffer I/O error with mirrored lvm on top of files on Btrfs
Hello,
I tried to test LVM Raid1 robustness with following setup:
fallocate -l 128M file1
fallocate -l 128M file2
losetup /dev/loop0 file1
losetup /dev/loop1 file2
pvcreate /dev/loop{0,1}
vgcreate testvg /dev/loop{0,1}
lvcreate -L 100M --mirrorlog core -m 1 -n testlv testvg
So far so good. But suddenly on attempt to use this volume:
mkfs.ext4 /dev/testvg/testlv
<uninteresting messages>
Superblock backups stored on blocks:
8193, 24577, 40961, 57345, 73729
Allocating group tables: done
Warning: could not read block 0: Attempt to read block from filesystem resulted in short read
Warning: could not erase sector 0: Attempt to write block to filesystem resulted in short write Writing inode tables: doneext2fs_update_bb_inode: Attempt to read block from filesystem resulted in short read while setting bad block inode
And following error messages in dmesg appears:
[211114.749967] device-mapper: raid1: All sides of mirror have failed. [211114.754559] Buffer I/O error on device dm-4, logical block 0 [211114.755097] Buffer I/O error on device dm-4, logical block 72 [211114.758176] Buffer I/O error on device dm-4, logical block 25584 [211114.758184] Buffer I/O error on device dm-4, logical block 25584 [211114.758192] Buffer I/O error on device dm-4, logical block 25598 [211114.758195] Buffer I/O error on device dm-4, logical block 25598 [211114.758201] Buffer I/O error on device dm-4, logical block 0 [211114.758204] Buffer I/O error on device dm-4, logical block 0 [211114.758209] Buffer I/O error on device dm-4, logical block 1
With a couple of more experiments I found that:
1) This setup works well if underlying filesystem is ext4 or FAT32, but fails if underlying FS is Btrfs or ntfs-3g;
2) It works well everywhere, if LV is not mirrored.
It seems to me that there is obscure problem somewhere.
I concocted a script to help with reproducing this problem. It constructs volume group from two test files in current directory:
----------------------
#!/bin/sh
#PV size has to specified in megabytes, or loopback files creation with dd will break PVSIZE=128M
LVSIZE=100M
FILES='loopback1 loopback2'
LBDEVICES=''
VGNAME='lvmlbtest'
LVNAME='mirror'
#####################
# Test
#####################
for f in $FILES; do
fallocate -l $PVSIZE $f || dd if=/dev/zero of=$f bs=$((1024*1024))
count=${PVSIZE%M} || exit 1
losetup -v -f $f
LBDEVICES="$LBDEVICES $(losetup -a|grep $f|cut -d: -f1|tail -n 1)"
done
pvcreate $LBDEVICES
vgcreate $VGNAME $LBDEVICES
lvcreate -L $LVSIZE --mirrorlog core -m 1 -n $LVNAME $VGNAME
LVPATH=$(lvs --noheadings -o lv_path|grep $VGNAME|grep $LVNAME)
read -p 'LV created. Press enter to proceed with mkfs ' sleepvar mkfs.ext4 $LVPATH
#####################
# Cleanup
#####################
read -p 'Press enter to delete test VG ' sleepvar
vgremove -f $VGNAME
sync
for LB in $LBDEVICES; do
losetup -v -d $LB
done
for f in $FILES; do
rm -v $f
done