Skip to contentSkip to conversation

RFH: ext3 on EVMS on SW-RAID1 problem

DISCUSSION Philipp Matthias Hahn 2 participants 2 messages Apr 27, 2005 – Apr 28, 2005

lore
Share & tools
lore
#0 lore raw

Hello and help!

One of our university fileservers shows strange problems since last friday. Syslog show the following messages:

	attempt to access beyond end of device
	dm-8: rw=0, want=8589934592, limit=262142

The strange thing: If I mount a disk-image of that volume via loop, everything works fine!

The server was running Debian sarge with an unpatched 2.6.11.6 than, but is running an 2.6.11.7 now and still shows the same problem. EVMS is version 2.5.2-1 and DevMapper is version 1.01.00-4.

moradin:/var/tmp# dd if=/dev/evms/bsp2005 of=/var/tmp/bsp2005.e3
262142+0 records in
262142+0 records out
134216704 bytes transferred in 5.012082 seconds (26778633 bytes/sec)
moradin:/var/tmp# mount -o loop /var/tmp/bsp2005.e3 /mnt
moradin:/var/tmp# stat /mnt/i386-gnu-linux/tools/lib/gcc-lib/mips-linux/3.3.2/include/stddef.h
  File: `/mnt/i386-gnu-linux/tools/lib/gcc-lib/mips-linux/3.3.2/include/stddef.h'
  Size: 12695           Blocks: 28         IO Block: 4096   regular file
Device: 700h/1792d      Inode: 24840       Links: 1
Access: (0640/-rw-r-----)  Uid: ( 1000/  pmhahn)   Gid: (19992/ bsp2005)
Access: 2005-04-27 13:06:16.000000000 +0200
Modify: 2005-04-13 15:21:57.000000000 +0200
Change: 2005-04-22 08:35:09.000000000 +0200
moradin:/var/tmp# md5sum /mnt/i386-gnu-linux/tools/lib/gcc-lib/mips-linux/3.3.2/include/stddef.h
3a5f8185367677ce39f9f8d2a72a2705  /mnt/i386-gnu-linux/tools/lib/gcc-lib/mips-linux/3.3.2/include/stddef.h
moradin:~# umount /mnt
moradin:~# mount /dev/evms/bsp2005 /mnt
moradin:~# stat /mnt/i386-gnu-linux/tools/lib/gcc-lib/mips-linux/3.3.2/include/stddef.h
  File: `/mnt/i386-gnu-linux/tools/lib/gcc-lib/mips-linux/3.3.2/include/stddef.h'
  Size: 12695           Blocks: 28         IO Block: 4096   regular file
Device: fd08h/64776d    Inode: 24840       Links: 1
Access: (0640/-rw-r-----)  Uid: ( 1000/  pmhahn)   Gid: (19992/ bsp2005)
Access: 2005-04-27 13:06:16.000000000 +0200
Modify: 2005-04-13 15:21:57.000000000 +0200
Change: 2005-04-22 08:35:09.000000000 +0200
moradin:~# md5sum /mnt/i386-gnu-linux/tools/lib/gcc-lib/mips-linux/3.3.2/include/stddef.h
error processing /mnt/i386-gnu-linux/tools/lib/gcc-lib/mips-linux/3.3.2/include/stddef.h: failed in buffer_read(fd): mdfile: Input/output error
moradin:~# umount /mnt

bsp2005 is an ext3-filesystem, from which a snapshot bsp2005_snap is created. They both live in a lvm-region, which is based on a Software-RAID1 using two partitions of two SCSI discs:

	 lvm/svs/bsp2005#origin#
	   lvm/svs/bsp2005
	     md/md0
	       sda4
		 sda
	       sdb4
		 sdb

Is something wrong with this setup or is it a known problem? Since the same solution was working last year without problems, I'm very confused about this strange error behaviour.

BYtE
Philipp

Signature · 3 lines
  / /  (_)__  __ ____  __ Philipp Hahn
 / /__/ / _ \/ // /\ \/ /
/____/_/_//_/\_,_/ /_/\_\ [email protected]
#1 lore raw
Quoted 10 lines · Philipp Matthias Hahn

One of our university fileservers shows strange problems since last friday. Syslog show the following messages:

	attempt to access beyond end of device
	dm-8: rw=0, want=8589934592, limit=262142

The strange thing: If I mount a disk-image of that volume via loop, everything works fine!

The server was running Debian sarge with an unpatched 2.6.11.6 than, but is running an 2.6.11.7 now and still shows the same problem. EVMS is version 2.5.2-1 and DevMapper is version 1.01.00-4.

I see a similar problem under recent Fedora Core 3 kernels with LVM2. It appears when I create a snapshot of a volume. See Red Hat's Bugzilla:

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=152162

Exact steps I used to reproduce the problem (which also results in file corruption, even when reading from the non-snapshot volume). I used a scratch partition, /dev/sda8:

########################################################################

# create the software RAID as a 2 device mirror with 1 missing
mdadm -C -l 1 -n 2 /dev/md0 /dev/sda8 missing
# create the LVM setup
pvcreate /dev/md0
vgcreate lvtest /dev/md0
lvcreate -L100m -n test lvtest
# make a filesystem and put some data on it
mke2fs -j /dev/lvtest/test
mount /dev/lvtest/test /mnt
cp --preserve=all -r /boot/* /mnt/
umount /mnt
blockdev --flushbufs /dev/lvtest/test
# now mount it, create a snapshot, and see the result
mount /dev/lvtest/test /mnt
lvcreate -s -L10m -n snap /dev/lvtest/test
diff -ur /boot /mnt
########################################################################

The output I got from diff was:

diff: /mnt/System.map-2.6.10-1.766_FC3: Input/output error

and I got a bunch of messages like:

attempt to access beyond end of device
dm-4: rw=0, want=8300006146, limit=204800
Buffer I/O error on device dm-4, logical block 4150003072

from the kernel. These only seem to appear sometimes - other times I get file corruption (although the corruption appears to be block-aligned).

If I then do:

########################################################################
lvremove /dev/lvtest/snap
umount /mnt
blockdev --flushbufs /dev/lvtest/test
mount /dev/lvtest/test /mnt
diff -ur /boot /mnt
########################################################################

It compares with no errors.

Signature · 3 lines
Chris Adams <[email protected]>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.