Attempt to Access Beyond End of Device

Hi,

We've set up Debian GNU/Linux 4.0 "Etch" on an IBM x3400 machine with two 73.4GB SAS hard drives in hardware RAID 1 with a battery-backed cache. We are using the stock Debian 2.6.18-4-686 kernel.

The filesystems were created using the following optimization:

        -l size=32768b,version=2 -n 64k

The filesystems are mounted using the following optimization:

        logbufs=8,logbsize=256k

Today, we were unable to mount the "large" (not really at < 70GB) /var (/dev/sda8), getting the following error:

Attempt to access beyond end of device
sda: rw=0, want=143139140, limit=143134720
I/O error in filesystem ("sda8") meta-data dev sda8 block 0x75cd27f
("xf:read_buf") errors buf count 512
XFS: size check 2 failed
Mount: /dev/sda8: can't read superblock

An attempted repair also fails:

        # xfs_repair /dev/sda8
        Phase 1 - find and verify superblock...
        Attempt to access beyond end of device
        Sda: rw=0, want=143139140, limit=143134720
        Xfs_repair: read failed.      Input/output error

The partition table looks okay:

        #Partition table of /dev/sda
        /dev/sda1 : start=          63,size        9637, Id=83, bootable
        /dev/sda2 : start=       96390,size   143042760, Id=5
        /dev/sda3 : start=           0,size           0, Id=0
        /dev/sda4 : start=           0,size           0, Id=0   
        /dev/sda5 : start=       96453,size     3903732, Id=82
        /dev/sda6 : start=     4000248,size     7807527, Id=83
        /dev/sda7 : start=    11807838,size     7807527, Id=83
        /dev/sda8 : start=    19615428,size   123523722, Id=83

The machine doesn't have valuable data, yet, so a simple reinstall should help get it back up. However I'm more concerned about what could cause this. It's the first time for me to use a version 2 log, 64k directories (?) and 256k in-memory log buffers. Are any of these to blame?

We're also looking for generic filesystem tweaks for a PostgreSQL + Apache server, which this will be (with more load direct to PostgreSQL than to/through Apache). Are the above choices for mkfs.xfs and mount well-made?

Please advise.

Thank you very much.

Federico Sevilla III
F S 3 Consulting Inc.
http://www.fs3.ph

I believe the logbsize should be in bytes, so 262144, it may also take 256k though I have not tried it.

As far as the creation, that's a good question, I use SW raid so XFS auto-tunes it for my hardware.

That looks mighty strange, what kind of raid card?

Justin.

It accepts 256k. The initial mount succeeded, and the problem did not surface until after some reboot (ie: not the first reboot, either, that went fine).

This is the built-in Adaptec AACRAID of the IBM x3400, doing a pretty simplistic RAID 1.

Federico Sevilla III
F S 3 Consulting Inc.
http://www.fs3.ph

Have you checked the following page?

http://linuxmafia.com/faq/Hardware/sata.html

Does the error occur if you use a different filesystem?

Justin.

I checked it now (thanks!) and I believe this is what we've got (or similar) as we use AACRAID:

Adaptec AAR 2400, 2410, 2410SA, 2120S, 2200S, 2810SA (8-port), 21610SA (16-port) series PCI cards — real hardware RAID, using the slightly anemic Intel IOP302/303 I/O co-processor chips. Use "aacraid" driver. (Should not be confused with the Adaptec 2400A ATA RAID host adapter, for which one uses the dpt_i2o driver, that card being a legacy of Adaptec's buying DPT — nor with the low-end Adaptec AAR 12x0 series, which please see.) Faster at random I/O than the 3Ware cards. Optional battery is available for the card's cache, for more reliable operation in the event of power loss, etc. (Card disables the drive's write cache.)

We also have the optional battery mentioned.

I haven't tried any other filesystem, yet. I'm looking for clues as to what this could be pointing to, first... and emails like yours are definitely helping, thank you very much.

Cheers!

Federico Sevilla III
F S 3 Consulting Inc.
http://www.fs3.ph

can you send along the results of:

	# xfs_db -r -c "sb 0" -c "print" /dev/sda8?

and

	# cat /proc/partitions

... and does the hardware raid have a funky sector size?

-Eric

Will do first thing tomorrow when I get back access to the machine, and will revert to the list.

Please advise as to how best to usually find this out. For whatever it's worth, it doesn't allow us to pick a stripe width for RAID 1 (as we probably shouldn't be able to specify one, anyway). So whatever we're using is our only choice as far as the hardware RAID 1 goes. (We could probably use software RAID 1, though, if push comes to shove.)

Thank you very much for your time.

Federico Sevilla III
F S 3 Consulting Inc.
http://www.fs3.ph

Dunno :)

FWIW, the size check that is failing is simply looking at the fs geometry and trying to go out and read the last sector.

-Eric

Hi,

Quoting Bhagi rathi <[email protected]>:

It wasn't an XFS-centric problem, after all. Running cfdisk on the system would likewise fail because the partition had been set beyond the disk boundary. I'm not sure about how this happened. Either the controller reported a large size to Debian during installation, or Debian mis-read the reported size on installation. At any rate, redoing the entire installation (including RAID setup) with the same procedure resulted in the correct (ie: smaller) disk boundaries, so the server is back up with the new size.

Previously, we were using version=1 logs with size=32768b, and then mounted using logbufs=8,logbsize=32768.

Now, we are using version=2 logs with size=32768b, and then mounted using logbufs=8,logbsize=256k.

If I understand your advise correctly, I should not mount with logbsize > 32k, or I should create the filesystem using version=2 logs with size=256k. Is this understanding correct?

Are there generic optimization suggestions for I/O-intensive servers in general, as far as on-disk and in-memory log buffer sizes are concerned?

Please advise.

Thank you very much.

Federico Sevilla III
F S 3 Consulting Inc.
http://www.fs3.ph

Federico Sevilla III wrote:
"It wasn't an XFS-centric problem, after all"

Adding Federico's comment... Some Adadptec Controller firmware versions will pass incorrect device parameters to the linux kernel. The kernel log output misled me to believe is was a corrupt filesystem or partition map, however it was just an adaptec bug.

Adaptac has a fix procedure for this at: http://ask.adaptec.com/scripts/adaptec_tic.cfg/php.exe/enduser/std_adp.php?p_faqid=16914

Below is the my kernel's dmesg output. After following Adaptec's procedure, the errors went away and I was able to mount the filesystem and see all of my existing data.
-----
mount: /dev/sda: can't read superblock
Mount Error at /dev/sdj. Is filesystem realtime? Need a File System?
attempt to access beyond end of device
sda: rw=0, want=YYYY, limit=XXXX
I/O error in filesystem ("sda") meta-data dev sdj block 0xZZZZ
("xfs_read_buf") error 5 buf count 512
XFS: size check 2 failed
-----

Regards,

Michael Willis

-- Eric Sandeen wrote :

can you send along the results of:

	# xfs_db -r -c "sb 0" -c "print" /dev/sda8?

and

	# cat /proc/partitions

... and does the hardware raid have a funky sector size?

-Eric

--
This message was sent on behalf of [email protected] at openSubscriber.com
http://www.opensubscriber.com/message/[email protected]/7550412.html

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Thanks for the followup; we've seen this a couple times, and it has always stumped me.

-Eric

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs