Skip to contentSkip to conversation

Lost 1 of 9 PVs and ext3 filesys needs recovery but --partial activates read-only

linux-lvm InDatcom 1 participant 1 message Nov 6, 2006

lore
Share & tools
lore

InDatcom

author
#0 lore raw

Hello,

We have a server which has 9 HDs, each of them is a PV on a single VG. A few days ago 1 of the HDs (/dev/hdk) died suddenly, hardware dead, bios won't detect it anymore so nothing more to do hardware-wise. Right now we don't have another disk to replace it with and do a pvcreate -u / vgcfgrestore process.

So we wanted to try a --partial mount, we were able to activate the VG and LV, but it seems the ext3 filesystem has inconsistency errors and the journal needs to be corrected but the VG is activated read-only so we can't do this.

The question is:

Is there a way to activate a VG in --partial mode in read/write so we can run fsck.ext3 /dev/vg/lv ??

I can't find any mention of this either on man pages, mailinglist, google, etc. Someone on the mailinglist even mentioned this, that he didn't understand why --partial is done read-only.

We tried: lvchange -ay -P /dev/vg/lv and this is what we get:
Partial mode. Incomplete volume groups will be activated read-only.

Either that or a way to remove that bad PV from the VG metadata, I've read a couple of mails about using vgreduce --removemissing, the man page is a little confusing, I almost ran that command but I tried it on a test box first and yes, what it does is REMOVE the LV that has a missing PV, if you read the man quickly it leads you to believe that it will remove the bad PVs from your VG and lets you activate/mount it back on, but NOO, it kills the LV to get the VG consistency back on. But that is exactly what I would like to do, remove the bad PV from the LV, knowing that I will have a big (300+gb) hole on the VG so I can backup and redo the filesystem.

Here's our steps:

root@localhost backup]# pvscan -P
Partial mode. Incomplete volume groups will be activated read-only. /dev/ioerror: read failed after 0 of 4096 at 375809572864: Input/output
error
/dev/ioerror: read failed after 0 of 4096 at 0: Input/output error /dev/ioerror: read failed after 0 of 4096 at 0: Input/output error Couldn't find device with uuid 'j05492-Dloo-RxMp-aIcJ-Y7dW-XA6L-mhuBtd'. /dev/ioerror: read failed after 0 of 4096 at 0: Input/output error Couldn't find device with uuid 'j05492-Dloo-RxMp-aIcJ-Y7dW-XA6L-mhuBtd'. /dev/ioerror: read failed after 0 of 4096 at 0: Input/output error Couldn't find device with uuid 'j05492-Dloo-RxMp-aIcJ-Y7dW-XA6L-mhuBtd'.

  PV /dev/hda3        VG pop   lvm2 [225.92 GB / 0    free]
  PV /dev/hdb1        VG pop   lvm2 [298.09 GB / 0    free]
  PV /dev/hde1        VG pop   lvm2 [298.09 GB / 0    free]
  PV /dev/hdf1        VG pop   lvm2 [298.09 GB / 0    free]
  PV /dev/hdg1        VG pop   lvm2 [279.46 GB / 0    free]
  PV /dev/hdh1        VG pop   lvm2 [279.46 GB / 0    free]
  PV /dev/hdj1        VG pop   lvm2 [298.09 GB / 0    free]
  PV unknown device   VG pop   lvm2 [298.09 GB / 0    free]
  PV /dev/hdl1        VG pop   lvm2 [298.09 GB / 0    free]

Total: 9 [2.51 TB] / in use: 9 [2.51 TB] / in no VG: 0 [0 ]

[root@localhost backup]# vgscan -P
Partial mode. Incomplete volume groups will be activated read-only. Reading all physical volumes. This may take a while... /dev/ioerror: read failed after 0 of 4096 at 375809572864: Input/output
error
/dev/ioerror: read failed after 0 of 4096 at 0: Input/output error /dev/ioerror: read failed after 0 of 4096 at 0: Input/output error Couldn't find device with uuid 'j05492-Dloo-RxMp-aIcJ-Y7dW-XA6L-mhuBtd'. /dev/ioerror: read failed after 0 of 4096 at 0: Input/output error Couldn't find device with uuid 'j05492-Dloo-RxMp-aIcJ-Y7dW-XA6L-mhuBtd'. Found volume group "pop" using metadata type lvm2

[root@localhost backup]# vgchange -ay -P
Partial mode. Incomplete volume groups will be activated read-only. /dev/ioerror: read failed after 0 of 4096 at 0: Input/output error /dev/ioerror: read failed after 0 of 4096 at 375809572864: Input/output
error
/dev/ioerror: read failed after 0 of 4096 at 0: Input/output error Couldn't find device with uuid 'j05492-Dloo-RxMp-aIcJ-Y7dW-XA6L-mhuBtd'. /dev/ioerror: read failed after 0 of 4096 at 0: Input/output error Couldn't find device with uuid 'j05492-Dloo-RxMp-aIcJ-Y7dW-XA6L-mhuBtd'. 1 logical volume(s) in volume group "pop" now active

After this LV is active and is mountable, BUT:

[root@localhost /]# mount -t ext3 /dev/pop/data /backup/
mount: block device /dev/pop/data is write-protected, mounting read-only
mount: wrong fs type, bad option, bad superblock on /dev/pop/data,
missing codepage or other error
In some cases useful info is found in syslog - try dmesg | tail or so

[root@localhost /]# dmesg|tail
EXT3-fs: INFO: recovery required on readonly filesystem.
EXT3-fs: write access unavailable, cannot proceed.

[root@localhost /]# fsck.ext3 /dev/pop/data e2fsck 1.38 (30-Jun-2005)
/dev/pop/data: recovering journal
fsck.ext3: unable to set superblock flags on /dev/pop/data

Need read/write permissions to fix this and then possibly be able to recover some of the data.

Any help appreciated.

Ismael Sanchez