Dead device and cleaning up LVM
I have a volgroup that was created on top of an iSCSI mounted block device (target was stgt on RHEL 5.3 and initiator is the default iSCSI initiator in RHEL 4.7).
Through a goof the target went offline and I tried shutting down the initiator software before disabling the volume group on the initiator with vgchange -an. Now I'm left with a bit of a mess including iSCSI initiator modules that won't completely unload. I'm trying to avoid a reboot and we have been theorizing that we somehow need to tell LVM to "let go" of the remaining ghosts of the iSCSI device before things can go back to normal.
The /dev/sdXX device is gone, but how can I convice LVM to quit trying to use /dev/dm-2? Can I edit the /etc/lvm/cache/.cache file and remove entries there?
I'm seeing the following stuff in dmesg:
scsi7 (0:1): rejecting I/O to dead device
And all the various pvscan/vgscan/lvscan tools show something similar to the following:
/dev/dm-2: read failed after 0 of 4096 at 4398038056960: Input/output error /dev/dm-2: read failed after 0 of 4096 at 4398038114304: Input/output error /dev/dm-2: read failed after 0 of 4096 at 0: Input/output error /dev/dm-2: read failed after 0 of 4096 at 4096: Input/output error /dev/dm-2: read failed after 0 of 4096 at 0: Input/output error
Anyone know how I can remove this device both from the system and from LVM safely? I can reboot, but saving that as a last resort if possible.
Also, any best practices or suggestions for using iSCSI backed block devices as volgroup members would be appreciated. I'm worried about LVM not recognizing the volume group of physical volumes correctly on reboots...
Thanks,
Ray
Quoted 3 lines · Ray Van Dolson
Anyone know how I can remove this device both from the system and from LVM safely? I can reboot, but saving that as a last resort if possible.
You don't mention the name of your VG/LV but you should be able to remove the active lvm devices with "dmsetup remove", e.g.:
dmsetup remove $VG-$LV
For each LV in each VG that was using the iSCSI devices.
Quoted 4 lines
Also, any best practices or suggestions for using iSCSI backed block devices as volgroup members would be appreciated. I'm worried about LVM not recognizing the volume group of physical volumes correctly on reboots...
You might need to label any file system mounts in /etc/fstab with "_netdev" and enable the netfs service to have the VGs automatically activated/mounted at boot time.
Regards,
Bryn.
Quoted 11 lines · Bryn M. Reeves
Quoted 3 lines · Ray Van Dolson
Anyone know how I can remove this device both from the system and from LVM safely? I can reboot, but saving that as a last resort if possible.
You don't mention the name of your VG/LV but you should be able to remove the active lvm devices with "dmsetup remove", e.g.:
dmsetup remove $VG-$LV
For each LV in each VG that was using the iSCSI devices.
Beautiful. This did the trick! I can now also unload the iSCSI kernel modules correctly on the initiator.
Thanks,
Ray
Quoted 9 lines · Ray Van Dolson
Quoted 8 lines · Bryn M. Reeves
Quoted 3 lines · Ray Van Dolson
Anyone know how I can remove this device both from the system and from LVM safely? I can reboot, but saving that as a last resort if possible.
You don't mention the name of your VG/LV but you should be able to remove the active lvm devices with "dmsetup remove", e.g.:
Quoted 7 lines
Quoted 3 lines
dmsetup remove $VG-$LV
For each LV in each VG that was using the iSCSI devices.
Beautiful. This did the trick! I can now also unload the iSCSI kernel modules correctly on the initiator.
Also, just an FYI in case anyone else runs across this thread... to get my filesystem back (pvscan didn't detect the LVM at all):
# pvcreate --uuid 3HpAVg-fRz0-HWXp-EFxK-khVO-IvN6-J4a6w1 /dev/sdb (Found the old UUID by examining the /etc/lvm/backup directory) # vgcfgrestore FTP1 # vgchange -ay FTP1 # fsck.ext3 /dev/FTP1/LogVol00 # mount -a
Back in action!
Ray
Quoted 11 lines · Ray Van Dolson
Quoted 10 lines · Ray Van Dolson
Quoted 9 lines · Bryn M. Reeves
Quoted 3 lines · Ray Van Dolson
Anyone know how I can remove this device both from the system and from LVM safely? I can reboot, but saving that as a last resort if possible.
You don't mention the name of your VG/LV but you should be able to remove the active lvm devices with "dmsetup remove", e.g.:
dmsetup remove $VG-$LV
For each LV in each VG that was using the iSCSI devices.
Quoted 8 lines
Quoted 3 lines
Beautiful. This did the trick! I can now also unload the iSCSI kernel modules correctly on the initiator.
Also, just an FYI in case anyone else runs across this thread... to get my filesystem back (pvscan didn't detect the LVM at all):
# pvcreate --uuid 3HpAVg-fRz0-HWXp-EFxK-khVO-IvN6-J4a6w1 /dev/sdb
If you also specify "--restorefile=" and the path to the current metadata backup in /etc/lvm, pvcreate will ensure that the new PV can accommodate the VG you want to restore to it. It'll almost certainly always work without this if you always use the default options but by specifying --restorefile you can be completely sure.
Quoted 7 lines
(Found the old UUID by examining the
/etc/lvm/backupdirectory)# vgcfgrestore FTP1 # vgchange -ay FTP1 # fsck.ext3 /dev/FTP1/LogVol00 # mount -aBack in action!
Glad to hear it!
Regards,
Bryn.
Quoted 4 lines
You don't mention the name of your VG/LV but you should be able to remove the active lvm devices with "dmsetup remove", e.g.:
dmsetup remove $VG-$LV
IIUC you can also use
lvchange -an /dev/<VG>/<LV>
at leas that's what I do when an LVM volume becomes unavailable (e.g., because the corresponding USB drive was disconnected). I guess "vgchange -an" might work as well.
Stefan