Should xfs_repair take this long?

I am trying to restore a corrupt xfs partition. It is 6TB total, it
is an LVM of two 3TB fiber channel SAN volumes. The host is running RHEL4, 2.6.9-42.0.2.ELsmp, and the version of xfsprogs is xfsprogs-2.6.13-2. The host has four threaded AMD Opterons, 4GB of RAM and 2GB of swap located on an internal SCSI disk. It is unclear how the xfs partition was damaged, but it reports a bad superblock and will not mount. I am running this command;

xfs_repair -o assume_xfs /dev/mapper/vg0-hladata3

This command has been running for two days now. There is cpu
activity and i/o activity on the physical SAN. There is some swapping but not an unusual amount and swapon -s shows only a small amount in use. I have seen information implying xfs_repair needs a large amount of memory to work well, otherwise it will take a long time. My question is, giving my setup, is there an estimate of how long I should wait before expecting a result? Should I add swap space? Is there anything else I should do?

thanks in advance for any help.

Thomas Walker

The terminal shows a lot of "." dots running across the screen
quickly, and every few hours it says this;

.....................................................found candidate secondary superblock... unable to verify superblock, continuing...
found candidate secondary superblock...
unable to verify superblock, continuing...

Thomas Walker

Le Thu, 15 Mar 2007 07:27:08 -0400
Thomas Walker <[email protected]> écrivait:

Is there any output from xfs_repair ? This doesn't sound good. I've run xfs_repair on some badly corrupted fs up to 13 TB, and it never took more than a couple of minutes.

----------------------------------------
Emmanuel Florac     |   Intellique
----------------------------------------

Le Thu, 15 Mar 2007 11:08:00 -0400
Thomas Walker <[email protected]> écrivait:

The repair _MAY_ return, unfortunately...

Yes, let it go now...

It should finish if you're testing the right device, and the LV is properly assembled. If it reach the end of the device and find nothing, you should restart LVM first to check that your PV/VG/LV are correctly set up, and retry.
If it doesn't work after that, backup will be your last friend.

----------------------------------------
Emmanuel Florac     |   Intellique
----------------------------------------

I checked the status of the LVM before starting xfs_repair. I can't
promise it's all in order, but at least the various pvdisplay, vgdisplay, lvdisplay, etc all came back normal. So... I'll just wait a few days and hope that xfs_repair comes back with something eventually.

thanks for at least taking an interest, too bad that there's nothing
we can really do about it.

Thomas Walker

The primary superblock is not good, and it's trying to find a valid secondary superblock. Doesn't sound promising so far - reapir can't start until a valid superblok is found....

Can you dump the first sector of the device the fielsystem is on:

# dd if=/dev/mapper/vg0-hladata3 bs=512 count=1 iflag=direct 2> /dev/null | od -Ax -x

So we can see if that really holds a primary XFS superblock?

Cheers,

Dave.

Dave Chinner
Principal Engineer
SGI Australian Software Group

I see in other posts that "parted" is sometimes a culprit in these
problems. Indeed, in my case I did use "parted" to create a gpt partition table on these xfs volumes and I was asked my parted to use the secondary signature.

--- snip ---

i also remember something about parted (maybe...) finding a backup gpt signature at the end of a disk, and "helpfully" copying it over the front end if so. This was a bug. sgi guys do you remember?

But for this one has to invoke parted, and commit the operations done, am I right?

if I recall, even invoking parted could do this.
--- snip ---

So maybe I got bit the same way. parted may be overwritten something
at the head of the volume. Is there any way to repair the super block though? It seems that everyone agrees xfs can't do anything until it has a super block somewhere and I don't seem to have one. If there's no way to repair, then what about recovery? I see mention of possibly doing an xfs dump to another disk, reformat the original volume, and then xfs restore back. Is there any online procedure for how to do that if it applies to me here?

Thomas Walker

Doesn't look like partition blocks at the start of each volume, though.

That's beacuse repair can't work out where things are supposed to be without a superblock to tell it critical information. Manually trying to find and repair a superblock is a hit and miss affair - at this point we don't even know if the primary superblocks have been overwritten or whether something else is wrong with LVM...

In a word: backups.

You need to be able to mount the filesystem to dump it, so until you can run repair there's no simple recovery option.

If the lvm config is correct and repair cannot find a valid secondary superblock, then you really need to start doing dangerous things to try to recover. i'd suggest taking a copy of the lvm volumes before doing anything else.

Then, find a secondary superblock in the volume (first 4 bytes of the sector are "XFSB" in hex) and copy that sector to block zero of the filesystem. If repair still won't do it's stuff, then you need to use xfs_db to modify that superblock until it does. Then when repair runs, you get to look in lost+found and try to work out what all the broken bits are.....

Cheers,

Dave.

Dave Chinner
Principal Engineer
SGI Australian Software Group