lvm - lvm_map access beyond end of device

Hello,

Vital info:

    Linux-2.2.18
    LVM version 0.9  by Heinz Mauelshagen  (13/11/2000)
    gcc version 2.95.2
    GDT7563RN Firmware 2.27.04-R03F

After about 27 days of uptime one of our Linux machines that is being used as a Samba, Netatalk, and FTP server; the main data mount went into a read-only state. The machine has a GDT7563RN ICP Vortex RAID card with 6-SCSI channels. Five channels are being used with nine 18GB disks on each channel. The RAID controller takes those 45 disks and makes three RAID-5 units with two hot-spares. We are using LVM to take those three RAID disks and striping those to one big block device. We are mounting that 656GB disk as an EXT2 mount on /data. During testing we banged the h*ll out of it with multiple bonnie tests and never found a problem. But during normal use we got events in our syslog (attached), also note the RAID controller showed no events. After a umount of the mounted filesystem, a reboot then a fsck the system worked fine. There were no bad reports from the fsck. Yes, I said no bad reports from fsck, it did take 3.5 hours though. What's causing this?

Inline attachments syslog, dmesg-boot.

syslog:

Mar  8 01:15:47 files kernel: lvm - lvm_map access beyond end of device;
*rsector: 2451636592 or size: 8 wrong for minor:  0
Mar  8 01:15:47 files kernel: Bad lvm_map in ll_rw_block
Mar  8 01:15:47 files kernel: EXT2-fs error (device lvm(58,0)):
ext2_readdir: bad entry in directory #43122713: rec_len is smaller than
minimal - offset=0, inode=0, rec_len=0, name_len=0
Mar  8 01:15:47 files kernel: Remounting filesystem read-only
Mar  8 01:15:48 files kernel: lvm - lvm_map access beyond end of device;
*rsector: 2596938272 or size: 8 wrong for minor:  0
Mar  8 01:15:48 files kernel: Bad lvm_map in ll_rw_block
Mar  8 01:15:48 files kernel: EXT2-fs error (device lvm(58,0)):
ext2_readdir: bad entry in directory #43122714: rec_len is smaller than
minimal - offset=0, inode=0, rec_len=0, name_len=0
Mar  8 01:15:48 files kernel: Remounting filesystem read-only

dmesg-boot:
1 1 89

 0d 000 00  1    0    0   0   0    0    0    00
 0e 000 00  0    0    0   0   0    1    1    91
 0f 000 00  0    0    0   0   0    1    1    99
 10 0FF 0F  1    1    0   1   0    1    1    A1
 11 0FF 0F  1    1    0   1   0    1    1    A9
 12 000 00  1    0    0   0   0    0    0    00
 13 0FF 0F  1    1    0   1   0    1    1    B1
 14 000 00  1    0    0   0   0    0    0    00
 15 0FF 0F  1    1    0   1   0    1    1    B9
 16 000 00  1    0    0   0   0    0    0    00
 17 0FF 0F  1    1    0   1   0    1    1    C1

IRQ to pin mappings:
IRQ0 -> 2
IRQ1 -> 1
IRQ3 -> 3
IRQ4 -> 4
IRQ6 -> 6
IRQ7 -> 7
IRQ8 -> 8
IRQ12 -> 12
IRQ13 -> 13
IRQ14 -> 14
IRQ15 -> 15
IRQ16 -> 16
IRQ17 -> 17
IRQ19 -> 19
IRQ21 -> 21
IRQ23 -> 23 .................................... done. checking TSC synchronization across CPUs: passed.
PCI: PCI BIOS revision 2.10 entry at 0xfdab0
PCI: Using configuration type 1
PCI: Probing PCI hardware
PCI->APIC IRQ transform: (B0,I12,P0) -> 19 PCI->APIC IRQ transform: (B0,I12,P0) -> 19 PCI->APIC IRQ transform: (B0,I13,P0) -> 17 PCI->APIC IRQ transform: (B0,I14,P0) -> 21 PCI->APIC IRQ transform: (B0,I16,P0) -> 16 PCI->APIC IRQ transform: (B0,I16,P1) -> 21 PCI->APIC IRQ transform: (B0,I18,P3) -> 21 PCI->APIC IRQ transform: (B2,I7,P0) -> 23
Linux NET4.0 for Linux 2.2
Based upon Swansea University Computer Society NET3.039
NET4: Unix domain sockets 1.0 for Linux NET4.0.
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP, IGMP
TCP: Hash tables configured (ehash 524288 bhash 65536) Starting kswapd v 1.5
Detected PS/2 Mouse Port.
pty: 256 Unix98 ptys configured
Real Time Clock Driver v1.09
RAM disk driver initialized: 16 RAM disks of 4096K size
loop: registered device at major 7
PIIX4: IDE controller on PCI bus 00 dev 91
PIIX4: not 100% native mode: will probe irqs later

    ide0: BM-DMA at 0x3060-0x3067, BIOS settings: hda:DMA, hdb:pio
    ide1: BM-DMA at 0x3068-0x306f, BIOS settings: hdc:pio, hdd:pio

Floppy drive(s): fd0 is 1.44M
FDC 0 is a National Semiconductor PC87306
LVM version 0.9 by Heinz Mauelshagen (13/11/2000)
lvm
-- Driver successfully initialized
Configuring GDT-PCI HA at 0/13 IRQ 17
scsi0 : GDT7563RN
scsi : 1 host.
Vendor: ICP Model: Host Drive #00 Rev:
Type: Direct-Access ANSI SCSI revision: 02
Detected scsi disk sda at scsi0, channel 0, id 0, lun 0
Vendor: ICP Model: Host Drive #15 Rev:
Type: Direct-Access ANSI SCSI revision: 02
Detected scsi disk sdb at scsi0, channel 0, id 15, lun 0
Vendor: ICP Model: Host Drive #29 Rev:
Type: Direct-Access ANSI SCSI revision: 02
Detected scsi disk sdc at scsi0, channel 0, id 29, lun 0 scsi : detected 3 SCSI disks total.
SCSI device sda: hdwr sector= 512 bytes. Sectors= 501790275 [245014 MB] [245.0 GB]
SCSI device sdb: hdwr sector= 512 bytes. Sectors= 465949260 [227514 MB] [227.5 GB]
SCSI device sdc: hdwr sector= 512 bytes. Sectors= 465949260 [227514 MB] [227.5 GB]
Partition check:
sda: sda1 sda2 sda3
sdb: sdb1
sdc: sdc1
VFS: Mounted root (ext2 filesystem) readonly.
Freeing unused kernel memory: 68k freed
Adding Swap: 2097136k swap-space (priority -1) eepro100.c:v1.09j-t 9/29/99 Donald Becker
http://cesdis.gsfc.nasa.gov/linux/drivers/eepro100.html
eepro100.c: $Revision: 1.20.2.10 $ 2000/05/31 Modified by Andrey V. Savochkin <[email protected]> and others
eepro100.c: VA Linux custom, Dragan Stancevic <[email protected]> 2000/11/15
eth0: Intel PCI EtherExpress Pro100 82557, 00:D0:B7:88:55:39, IRQ 21.
Receiver lock-up bug exists -- enabling work-around.
Board assembly 000000-000, Physical connectors present: RJ45 Primary interface chip i82555 PHY #1.
Forcing 100Mbs full-duplex operation.
General self-test: passed.
Serial sub-system self-test: passed.
Internal registers self-test: passed.
ROM checksum self-test: passed (0x04f4518b).
NET4: AppleTalk 0.18 for Linux NET4.0
(scsi1) <Adaptec AIC-7896/7 Ultra2 SCSI host adapter> found at PCI 0/12/0
(scsi1) Wide Channel A, SCSI ID=7, 32/255 SCBs
(scsi1) Downloading sequencer code... 392 instructions downloaded (scsi2) <Adaptec AIC-7896/7 Ultra2 SCSI host adapter> found at PCI 0/12/1
(scsi2) Wide Channel B, SCSI ID=7, 32/255 SCBs
(scsi2) Downloading sequencer code... 392 instructions downloaded (scsi3) <Adaptec AIC-7899 Ultra 160/m SCSI host adapter> found at PCI 0/16/0
(scsi3) Wide Channel A, SCSI ID=7, 32/255 SCBs
(scsi3) Downloading sequencer code... 392 instructions downloaded (scsi4) <Adaptec AIC-7899 Ultra 160/m SCSI host adapter> found at PCI 0/16/1
(scsi4) Wide Channel B, SCSI ID=7, 32/255 SCBs
(scsi4) Downloading sequencer code... 392 instructions downloaded scsi1 : Adaptec AHA274x/284x/294x (EISA/VLB/PCI-Fast SCSI) 5.1.31/3.2.4
<Adaptec AIC-7896/7 Ultra2 SCSI host adapter>
scsi2 : Adaptec AHA274x/284x/294x (EISA/VLB/PCI-Fast SCSI) 5.1.31/3.2.4
<Adaptec AIC-7896/7 Ultra2 SCSI host adapter>
scsi3 : Adaptec AHA274x/284x/294x (EISA/VLB/PCI-Fast SCSI) 5.1.31/3.2.4
<Adaptec AIC-7899 Ultra 160/m SCSI host adapter>
scsi4 : Adaptec AHA274x/284x/294x (EISA/VLB/PCI-Fast SCSI) 5.1.31/3.2.4
<Adaptec AIC-7899 Ultra 160/m SCSI host adapter>
scsi : 5 hosts.
(scsi2:0:6:0) Synchronous at 20.0 Mbyte/sec, offset 16.

  Vendor: TOSHIBA   Model: CD-ROM XM-6401TA  Rev: 1001
  Type:   CD-ROM                             ANSI SCSI revision: 02
  Vendor: EXABYTE   Model: Exabyte EZ17      Rev: 1.07
  Type:   Medium Changer                     ANSI SCSI revision: 02
(scsi3:0:1:0) Synchronous at 80.0 Mbyte/sec, offset 31.
  Vendor: EXABYTE   Model: Mammoth2          Rev: v02j
  Type:   Sequential-Access                  ANSI SCSI revision: 02
  Vendor: EXABYTE   Model: EXB-210           Rev: 1.07
  Type:   Medium Changer                     ANSI SCSI revision: 02

(scsi4:0:1:0) Synchronous at 80.0 Mbyte/sec, offset 31.
Vendor: EXABYTE Model: Mammoth2 Rev: v02p
Type: Sequential-Access ANSI SCSI revision: 02
Detected scsi generic sg4 at scsi3, channel 0, id 0, lun 0 Detected scsi generic sg6 at scsi4, channel 0, id 0, lun 0
st: bufsize 32768, wrt 30720, max buffers 5, s/g segs 16. Detected scsi tape st0 at scsi3, channel 0, id 1, lun 0 Detected scsi tape st1 at scsi4, channel 0, id 1, lun 0

[snip]

As an aside, other than this problem, how do you find ext2 on such a large filesystem? So far this is the largest ext2 filesystem I've seen. Are you looking into ext3 at all to avoid fsck on crash? Granted, in this case you still needed to do the whole fsck but in other cases ext3 would help.

Is this the only output in your syslog, a representative sample, a small part of a huge number of messages? How often do you have this problem, or is it the first time?

Can you verify that inodes 43122713 and 43122714 are actually valid directories or at least valid inode numbers? dumpe2fs -h should be enough to tell you the number of inodes in the filesystem.

According to my calculations for your 656 GB 4kB block filesystem:

sector 2451636592 / 8 sectors/block = 306454574
sector 2596938272 / 8 sectors/block = 324617284

656 GB * 1024 MB/GB * 256 blocks/MB = 171966464 blocks

So LVM is correct in refusing to map these blocks. The real question is where do the block numbers come from? The "bad entry" messages are simply a result of LVM not filling in the buffer for ext2, and it is all zeros.

The first point where we could have a bogus block number would be in inode_getblk(), where we access inode->u.ext2_i.i_data[0] (the block number here is not checked), but you say that e2fsck reported all was well.

The below patch for 2.2.18 verifies the block numbers read from disk before reading them. It would at least help narrow down the problem here.

Cheers, Andreas
========================================================================

@@ -228,13 +228,20 @@
 					  int create, int new_block, int * err)
 {
 	u32 * p;
-	int tmp, goal = 0;
+	u32 tmp, goal = 0;
 	struct buffer_head * result;
 	int blocks = inode->i_sb->s_blocksize / 512;
 
 	p = inode->u.ext2_i.i_data + nr;
 repeat:
 	tmp = *p;
+	if (tmp > le32_to_cpu(EXT2_SB(inode->i_sb)->s_es->s_blocks_count)) {
+		ext2_warning(inode->i_sb, __FUNCTION__,
+			     "inode #%u has bad direct block[%d] = %u\n",
+			     inode->i_ino, nr, tmp);
+		*err = -EIO;
+		return NULL;
+	}
 	if (tmp) {
 		struct buffer_head * result;
 		result = getblk (inode->i_dev, le32_to_cpu(tmp), inode->i_sb->s_blocksize);

@@ -293,7 +301,7 @@
 					  int create, int blocksize, 
 					  int new_block, int * err)
 {
-	int tmp, goal = 0;
+	u32 tmp, goal = 0;
 	u32 * p;
 	struct buffer_head * result;
 	int blocks = inode->i_sb->s_blocksize / 512;

@@ -311,6 +319,13 @@
 	p = (u32 *) bh->b_data + nr;
 repeat:
 	tmp = le32_to_cpu(*p);
+	if (tmp > le32_to_cpu(EXT2_SB(inode->i_sb)->s_es->s_blocks_count)) {
+		ext2_warning(inode->i_sb, __FUNCTION__,
+			     "inode #%u has bad indirect block[%u][%d] = %u\n",
+			     inode->i_ino, bh->b_blocknr, nr, tmp);
+		*err = -EIO;
+		return NULL;
+	}
 	if (tmp) {
 		result = getblk (bh->b_dev, tmp, blocksize);
 		if (tmp == le32_to_cpu(*p)) {

Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

EXT2 is nice and snappy. This is our first crash... but the fsck was fun to watch ;) We might switch if there is a stable alternative. I've used EXT3 on my personal system and it worked fine, and it was really sweet to move from EXT2<->EXT3. Do you recommend it for a production system? Why isn't it in 2.2.18 OR 2.4.x?

This was a grep kernel of the days syslog file. I did take a look and it was the only thing relevant.

# dumpe2fs -h /dev/vg0/lv0
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          1e473942-d8f1-4e48-905f-2f872f7e4424
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      filetype sparse_super
Filesystem state:         not clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              87375872
Block count:              174723072
Reserved block count:     0
Free blocks:              163429202
Free inodes:              86687944
First block:              0
Block size:               4096
Fragment size:            4096
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         16384
Inode blocks per group:   512
Last mount time:          Thu Mar  8 12:28:00 2001
Last write time:          Thu Mar  8 22:42:13 2001
Mount count:              1
Maximum mount count:      20
Last checked:             Thu Mar  8 12:27:58 2001
Check interval:           15552000 (6 months)
Next check after:         Tue Sep  4 13:27:58 2001
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               128

I'll see if I can add them in next reboot... which is tricky business with a 24x7 machine in production. I'll insert this into 2.2.19 when it comes out.