Skip to contentSkip to conversation

RAID hiccup, LVM information wiped, XFS FS gone, how to proceed?

DISCUSSION Emmanuel Florac 2 participants 3 messages Feb 11, 2019

lore
Share & tools
lore
#0 lore raw

Hi everyone,

I have a lost filesystem at hand: the RAID controller had a hiccup (or someone tripped on the cord, or did some other mistake, I have no way to know because the system is in some other country anyway).

The configuration was quite simple: RAID array -> pv -> vg -> lv -> XFS.

I've hexedit'ed the /dev/sdb and it looks like the first few kB have been wiped out: no LVM signature, no VG UUID... However after the first few blocks, it looks very similar to another disk>vg>lv configuration which I hexedit side by side (there are series of 00 with regularly placed "FIB" values interspersed with other values, then after a while what is obviously file content) so my guess is that the data is probably still there (better be, there are 80 TB missing).

xfs_repair can't seem to be able to do anything on the raw disk, alas. Any ideas? I don't even know why I can't seem to find the XFSB signature on disk: when hexediting (for instance) /dev/vg0/somelv, I see it starting with XFSB; but editing /dev/sdc (which is hosting the above mentioned lv), it's not there.

(I know, I could simply restore the vgcfgbackup if it existed... please let's suppose this command doesn't exist).

Signature · 6 lines
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<[email protected]>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------
#1 lore raw

Le Mon, 11 Feb 2019 17:23:59 +0100 Emmanuel Florac <[email protected]> écrivait:

Quoted 3 lines

(I know, I could simply restore the vgcfgbackup if it existed... please let's suppose this command doesn't exist).

Hum, actually I have a backup of the vg configuration but... it looks like actually 7 or 8 kB were erased from the disk. So no dice, I must recreate the PV first but wouldn't it wipe everything?

Signature · 6 lines
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<[email protected]>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------
#2 lore raw
Quoted 11 lines · Emmanuel Florac

Le Mon, 11 Feb 2019 17:23:59 +0100 Emmanuel Florac <[email protected]> écrivait:

Quoted 3 lines

(I know, I could simply restore the vgcfgbackup if it existed... please let's suppose this command doesn't exist).

Hum, actually I have a backup of the vg configuration but... it looks like actually 7 or 8 kB were erased from the disk. So no dice, I must recreate the PV first but wouldn't it wipe everything?

You're right to be concerned. First you need to make sure the raid is assembled correctly, and functioning correctly. Is there a battery backed cache? There should be raid logs that can be extract to show at least what the hardware raid thinks happened; and if it did a cache flush upon power being restored, the log should indicate that as well. If it's possible the cache was full and the contents lost, that's also useful information.

Anyway, get the raid sorted out first. Only then can you move on to LVM checking and recovery options, from the LVM list:

[email protected]

And only once LVM is healthy and working correctly can you move on to XFS checks and if advised, repair. Just be really deliberate, don't go stomping around with repairs and writes to disk that aren't deliberate, or you risk user induced data loss. Basically, I advise doing only checks and posting the results, and only doing repairs recommended on the applicable list by developers or expert users.

All the lists will want information not provided in your first post, example:
http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

--
Chris Murphy