XFS repair on / in a hosted environment

I have a virtual server hosted in a XEN environment. Kernel is 2.6.16.13-XenU and I can't change it. :( Host provider is not responsive to getting us a newer kernel.

Anyway, during my nightly rsync backup, I started seeing the following:

Jul 18 12:51:01 hosted kernel: 0x0: 00 00 00 00 01 00 00 00 23 00 00 00
00 00 00 00
Jul 18 12:51:01 hosted kernel: Filesystem "sda1": XFS internal error
xfs_da_do_buf(2) at line 2221 of file fs/xfs/xfs_da_btree.c.  Caller
0xc02288fd
Jul 18 12:51:01 hosted kernel:  [<c0237b71>] xfs_corruption_error+0xbb/0xc6
Jul 18 12:51:01 hosted kernel:  [<c02288fd>] xfs_da_read_buf+0x25/0x2a
Jul 18 12:51:01 hosted kernel:  [<c025e4e5>] xfs_buf_read_flags+0x1f/0x82
Jul 18 12:51:01 hosted kernel:  [<c025b1aa>] kmem_zone_alloc+0x43/0x88
Jul 18 12:51:01 hosted kernel:  [<c02287bf>] xfs_da_do_buf+0x5df/0x6a9
Jul 18 12:51:01 hosted kernel:  [<c02288fd>] xfs_da_read_buf+0x25/0x2a
Jul 18 12:51:01 hosted kernel:  [<c0169d88>] dput+0x1b/0x11c
Jul 18 12:51:01 hosted kernel:  [<c02288fd>] xfs_da_read_buf+0x25/0x2a
Jul 18 12:51:01 hosted kernel:  [<c022cc12>]
xfs_dir2_block_getdents+0xe0/0x2e4
Jul 18 12:51:01 hosted kernel:  [<c022cc12>]
xfs_dir2_block_getdents+0xe0/0x2e4
Jul 18 12:51:01 hosted kernel:  [<c013e7ee>]
get_page_from_freelist+0xa8/0x34b
Jul 18 12:51:01 hosted kernel:  [<c0219919>] xfs_bmap_last_offset+0xd5/0xe8
Jul 18 12:51:01 hosted kernel:  [<c022c537>]
xfs_dir2_put_dirent64_direct+0x0/0x8b
Jul 18 12:51:01 hosted kernel:  [<c022c17e>] xfs_dir2_isblock+0x1c/0x65
Jul 18 12:51:01 hosted kernel:  [<c022c537>]
xfs_dir2_put_dirent64_direct+0x0/0x8b
Jul 18 12:51:01 hosted kernel:  [<c022c452>] xfs_dir2_getdents+0xb0/0xe4
Jul 18 12:51:01 hosted kernel:  [<c022c537>]
xfs_dir2_put_dirent64_direct+0x0/0x8b
Jul 18 12:51:01 hosted kernel:  [<c02559d4>] xfs_readdir+0x3f/0x51
Jul 18 12:51:01 hosted kernel:  [<c025ea5e>] linvfs_readdir+0xfd/0x1d2
Jul 18 12:51:01 hosted kernel:  [<c0165d46>] vfs_readdir+0x42/0x6f
Jul 18 12:51:01 hosted kernel:  [<c0165e59>] filldir64+0x0/0xc6
Jul 18 12:51:01 hosted kernel:  [<c0165f82>] sys_getdents64+0x63/0xa9
Jul 18 12:51:01 hosted kernel:  [<c0165e59>] filldir64+0x0/0xc6
Jul 18 12:51:01 hosted kernel:  [<c0104869>] syscall_call+0x7/0xb

I've narrowed this down to a single directory which cannot be removed:

/usr/src/linux-2.6.16-xen/sound/pci/au88x0

any attempt to get the contents of the directory results in an error and the trace:

hosted:/usr/src/linux-2.6.16-xen/sound/pci# ls -l au88x0/
ls: reading directory au88x0/: Unknown error 990

Everything is on / so I can't easily get it unmounted to run xfs_repair.

I try booting into single user mode, mount -o remount ,ro /. xfs_repair -d /dev/sda1 still complains about a mounted writable partition.

===

On boot, mount shows:

/dev/sda1 on / type xfs (rw)

# mount -o remount,ro /

after the remount, mount still shows / as rw

# mount
/dev/sda1 on / type xfs (rw)

but attempts to modify it result in:

hosted:~# touch foo
touch: cannot touch `foo': Read-only file system

Ok, so try xfs_repair:

hosted:~# xfs_repair -d /dev/sda1
xfs_repair: /dev/sda1 contains a mounted and writable filesystem

fatal error -- couldn't initialize XFS library

xfs_repair thinks we are writable.

I tried doing this as early as I could in the boot process by putting a script in

/etc/rcS.d/S00fixroot

but that is not early enough.

Any thoughts?

-Rupa

.....

what does /proc/mounts tell you (rather than /etc/mtab)?

If /proc/mounts says ro, then try upgrading your xfsprogs package. and then retrying the repair....

Cheers,

Dave.

Dave Chinner
Principal Engineer
SGI Australian Software Group

/proc/mounts shows ro.

What version is new enough? I'm running debian sarge, so have:

# xfs_repair -V
xfs_repair version 2.9.0

ftp://oss.sgi.com/projects/xfs/cmd_tars/

only shows a xfs_progs up to 2.8.21-1.

Thanks,

-Rupa

I built the latest xfsprogs from cvs. Still no go. This is perhaps something strange with the Xen environment?

hosted:/# mount
/dev/sda1 on / type xfs (rw)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
none on /dev/shm type tmpfs (rw)

hosted:/# cat /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / xfs ro 0 0
tmpfs /lib/init/rw tmpfs rw,nosuid 0 0
proc /proc proc rw,nosuid,nodev,noexec 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,nosuid,noexec 0 0
none /dev/shm tmpfs rw 0 0

mount and /proc/mounts don't agree. Thre is no /dev/root. I tried the mount -o remount,ro on both / and /dev/sda1. No good, only /dev/root is being changed to ro.

<boggle>

Ok, created a /dev/root. xfs_repair works as expected. Now to figure out what got dumped into lost+found.

-Rupa

The above /etc/mtab vs /proc/mounts problem was the reason I mentioned you should get a recent xfsprogs - we changed it to look at /proc/mounts rather than /etc/mtab for this reason....

That sounds like an init script or initrd setup problem....

Oh, good to know you made it past the first (high) hurdle.

Cheers,

Dave.

Dave Chinner
Principal Engineer
SGI Australian Software Group