Recovering "broken" disk ( 17th )
Folks,
I have had a disk go bad on me, causing me to lose one PV.
I seem to have retrieved the partition using ddrescue, but it also seems to be missing some label information, because pvscan doesn't see it.
Using hexdump, I see the string " LVM2 " at 0x1004, but nothing before that. The whole phrase is:
0x01000 16 d6 8e db 20 4c 56 4d 32 20 78 5b 35 41 25 72
I find what appears to be an LVM2 configuration section at 0x1200, and so I was able to read the UUID that this PV should have.
On another machine, I dumped a PV partition, and find "LABLEONE" at 0x200, with the same " LVM2 " at 0x01000.
I was concerned that my dump was offset, but the comparison to the "good" one suggests that that isn't the problem, but just the missing "LABLEONE" and related information at 0x0200.
If I do a "pvcreate --uuid xxxx" would this fix that recovered partition so that pvscan and friends can work properly, and I can finally boot that machine?
Thank you,
Brian
_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
You will need a lvm backup file for the pvcreate --uuid I believe (there may be some option to get around needing the backup file).
That will put the header back on if you either have an lvm backup and/or archive file, you might also need a vgcfgrestore afterwards depending on if anything else is missing.
I have never done it, but it looks possible to make a lvm backup file by reading it directly off the disk with dd, so that you will have a file that pvcreate is ok with, that is if there is no way to force it without a backup file.
But, this should get the pv back showing up with whatever sectors that you successfully recovered.
Quoted 35 lines · Brian McCullough
Folks,
I have had a disk go bad on me, causing me to lose one PV.
I seem to have retrieved the partition using ddrescue, but it also seems to be missing some label information, because pvscan doesn't see it.
Using hexdump, I see the string " LVM2 " at
0x1004, but nothing before that. The whole phrase is:
0x0100016 d6 8e db 20 4c 56 4d 32 20 78 5b 35 41 25 72I find what appears to be an LVM2 configuration section at
0x1200, and so I was able to read the UUID that this PV should have.On another machine, I dumped a PV partition, and find "LABLEONE" at
0x200, with the same " LVM2 " at0x01000.I was concerned that my dump was offset, but the comparison to the "good" one suggests that that isn't the problem, but just the missing "LABLEONE" and related information at
0x0200.If I do a "pvcreate --uuid xxxx" would this fix that recovered partition so that pvscan and friends can work properly, and I can finally boot that machine?
Thank you,
Brian
Quoted 7 lines
_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Quoted 7 lines · Roger Heflin
You will need a lvm backup file for the pvcreate --uuid I believe (there may be some option to get around needing the backup file).
That will put the header back on if you either have an lvm backup and/or archive file, you might also need a vgcfgrestore afterwards depending on if anything else is missing.
Quoted 7 lines
I have never done it, but it looks possible to make a lvm backup file by reading it directly off the disk with dd, so that you will have a file that pvcreate is ok with, that is if there is no way to force it without a backup file.
But, this should get the pv back showing up with whatever sectors that you successfully recovered.
Thank you, Roger.
I might be able to do that on the original machine, which won't boot at the moment because of this failure, but I have been using a working machine so far to use ddrestore etc.
Yes, as you said, I was able to use dd to copy off the vgconfig data into a file. So you think that I might be able to use that as the reference in pvcreate? That makes some sense.
I started this out asking ( but not clearly ) whether the data on the disk appeared to be in the correct place and whether what was there was correct, but I think that I answered that for myself with the comparison to the working PV partition.
Thanks again,
Brian
_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
I would edit the vgconfig you dd'ed with an editor and make sure it looks reasonable for what you think you had.
When you do the pvcreate --uuid it won't use anything except the uuid info so the rest may not need to be exactly right, if you have to do a vgcfgrestore to get it to read the rest of the info will be used.
I have seen some weird disk controller failures that appeared to zero out the first bit of the disk (enough to get the partition table, grub, and the pv header depending on where the first partition starts).
You will need to reinstall grub if this was the bootable disk, since there were 384 bytes of grub in the sector with the partition table that you know are missing.
Quoted 35 lines · Brian McCullough
Quoted 5 lines · Roger Heflin
You will need a lvm backup file for the pvcreate --uuid I believe (there may be some option to get around needing the backup file).
That will put the header back on if you either have an lvm backup and/or archive file, you might also need a vgcfgrestore afterwards depending on
if
Quoted 4 lines
anything else is missing.
I have never done it, but it looks possible to make a lvm backup file by reading it directly off the disk with dd, so that you will have a file
that
Quoted 4 lines
pvcreate is ok with, that is if there is no way to force it without a backup file.
But, this should get the pv back showing up with whatever sectors that
you
Quoted 1 line
successfully recovered.
Thank you, Roger.
I might be able to do that on the original machine, which won't boot at the moment because of this failure, but I have been using a working machine so far to use ddrestore etc.
Yes, as you said, I was able to use dd to copy off the vgconfig data into a file. So you think that I might be able to use that as the reference in pvcreate? That makes some sense.
I started this out asking ( but not clearly ) whether the data on the disk appeared to be in the correct place and whether what was there was correct, but I think that I answered that for myself with the comparison to the working PV partition.
Quoted 10 lines
Thanks again,
Brian_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Quoted 2 lines · Roger Heflin
I would edit the vgconfig you dd'ed with an editor and make sure it looks reasonable for what you think you had.
It turns out, comparing the information that I pulled off of the drive with what I find in /etc/lvm/backup, that the first part of the vgconfig information is missing. As I said in one of my messages, the information that I retrieved from the disk starts at 0x1200. I don't know whether that is correct or not. It does not appear to be a proper "backup" file, which I think it should be.
I rebooted ( partially ) the machine and copied the vgconfig backup file from that, but am somewhat concerned, because I don't seem to be able to match the UUIDs. The one that I seem to see in the vgconfig data that I pulled off of the drive vs what I got out of /etc/lvm/backup. Maybe I am just mis-reading it. I will continue my research for a bit.
Quoted 3 lines
When you do the pvcreate --uuid it won't use anything except the uuid info so the rest may not need to be exactly right, if you have to do a vgcfgrestore to get it to read the rest of the info will be used.
Oh, thank you. I did see that things got somewhat different on the
target drive when I did "pvcreate --uuid --restorefile." I got paranoid when I saw that, and re-copied the ddrestore file back to the target
drive before I did anything else. Should I do "pvcreate --uuid
--norestorefile," instead? Then, once it is back in the machine, do the pvscan and vgcfgrestore, and expect good things?
Quoted 3 lines
I have seen some weird disk controller failures that appeared to zero out the first bit of the disk (enough to get the partition table, grub, and the pv header depending on where the first partition starts).
I APPEAR to have a partition table, containing an NTFS partition, an LVM partiton ( the one that I am concentrating on ) and a Linux partion. I would have thought that it was all LVM, but my memory could easily be wrong.
Quoted 3 lines
You will need to reinstall grub if this was the bootable disk, since there were 384 bytes of grub in the sector with the partition table that you know are missing.
Fortunately, this is all data, nothing to do with the boot sequence, except that the machine will not boot with the missing PV.
Thank you,
Brian
_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
is the pv in the root device vg? if not changing fstab to not mount the
missing fs(es) should get it bootable. I have a practice of putting
",nofail" on all non-root filesystems (ie defaults,nofail) since priority #1 is getting the machine up and on the network after a reboot such that it can be ssh'ed to and fixed as needed.
If it is not the root device then on the root device there should be several prior archive copy in /etc/lvm/archive/<vgname>*, and maybe some copies in /etc/lvm/backup.
In the vg backup file there will be a bunch of uuids, you want the specific pv uuid and not the vg/lv uuids. Each pv has a uuid and each lv has a uuid and the vg has a uuid.
Quoted 57 lines · Brian McCullough
Quoted 2 lines · Roger Heflin
I would edit the vgconfig you dd'ed with an editor and make sure it looks reasonable for what you think you had.
It turns out, comparing the information that I pulled off of the drive with what I find in
/etc/lvm/backup, that the first part of the vgconfig information is missing. As I said in one of my messages, the information that I retrieved from the disk starts at0x1200. I don't know whether that is correct or not. It does not appear to be a proper "backup" file, which I think it should be.I rebooted ( partially ) the machine and copied the vgconfig backup file from that, but am somewhat concerned, because I don't seem to be able to match the UUIDs. The one that I seem to see in the vgconfig data that I pulled off of the drive vs what I got out of
/etc/lvm/backup. Maybe I am just mis-reading it. I will continue my research for a bit.Quoted 1 line
When you do the pvcreate --uuid it won't use anything except the uuid
info
Quoted 2 lines
so the rest may not need to be exactly right, if you have to do a vgcfgrestore to get it to read the rest of the info will be used.
Oh, thank you. I did see that things got somewhat different on the
target drive when I did "pvcreate --uuid --restorefile." I got paranoid when I saw that, and re-copied the ddrestore file back to the target
drive before I did anything else. Should I do "pvcreate --uuid
--norestorefile," instead? Then, once it is back in the machine, do the pvscan and vgcfgrestore, and expect good things?Quoted 2 lines
I have seen some weird disk controller failures that appeared to zero out the first bit of the disk (enough to get the partition table, grub, and
the
Quoted 1 line
pv header depending on where the first partition starts).
I APPEAR to have a partition table, containing an NTFS partition, an LVM partiton ( the one that I am concentrating on ) and a Linux partion. I would have thought that it was all LVM, but my memory could easily be wrong.
Quoted 1 line
You will need to reinstall grub if this was the bootable disk, since
there
Quoted 1 line
were 384 bytes of grub in the sector with the partition table that you
know
Quoted 1 line
are missing.
Fortunately, this is all data, nothing to do with the boot sequence, except that the machine will not boot with the missing PV.
Quoted 10 lines
Thank you,
Brian_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
replying to my last email.
do the pvcreate -uuid and then do a pvs/lvs/vgs and see if the vg/lv's look
like that are there. if so do a vgchange -ay <vgname> and then test
mounting the fs.
And if with the fs either commented out and/or ,nofail the normal os boots up work from there as you should have backup files.
Quoted 10 lines · Roger Heflin
is the pv in the root device vg? if not changing fstab to not mount the
missing fs(es) should get it bootable. I have a practice of putting
",nofail" on all non-root filesystems (ie defaults,nofail) since priority #1 is getting the machine up and on the network after a reboot such that it can be ssh'ed to and fixed as needed.If it is not the root device then on the root device there should be several prior archive copy in /etc/lvm/archive/<vgname>*, and maybe some copies in
/etc/lvm/backup.
Quoted 76 lines
In the vg backup file there will be a bunch of uuids, you want the specific pv uuid and not the vg/lv uuids. Each pv has a uuid and each lv has a uuid and the vg has a uuid.
Quoted 69 lines · Brian McCullough
Quoted 1 line · Roger Heflin
I would edit the vgconfig you dd'ed with an editor and make sure it
looks
Quoted 1 line
reasonable for what you think you had.
It turns out, comparing the information that I pulled off of the drive with what I find in
/etc/lvm/backup, that the first part of the vgconfig information is missing. As I said in one of my messages, the information that I retrieved from the disk starts at0x1200. I don't know whether that is correct or not. It does not appear to be a proper "backup" file, which I think it should be.I rebooted ( partially ) the machine and copied the vgconfig backup file from that, but am somewhat concerned, because I don't seem to be able to match the UUIDs. The one that I seem to see in the vgconfig data that I pulled off of the drive vs what I got out of
/etc/lvm/backup. Maybe I am just mis-reading it. I will continue my research for a bit.Quoted 1 line
When you do the pvcreate --uuid it won't use anything except the uuid
info
Quoted 2 lines
so the rest may not need to be exactly right, if you have to do a vgcfgrestore to get it to read the rest of the info will be used.
Oh, thank you. I did see that things got somewhat different on the
target drive when I did "pvcreate --uuid --restorefile." I got paranoid when I saw that, and re-copied the ddrestore file back to the target
drive before I did anything else. Should I do "pvcreate --uuid
--norestorefile," instead? Then, once it is back in the machine, do the pvscan and vgcfgrestore, and expect good things?Quoted 1 line
I have seen some weird disk controller failures that appeared to zero
out
Quoted 1 line
the first bit of the disk (enough to get the partition table, grub, and
the
Quoted 1 line
pv header depending on where the first partition starts).
I APPEAR to have a partition table, containing an NTFS partition, an LVM partiton ( the one that I am concentrating on ) and a Linux partion. I would have thought that it was all LVM, but my memory could easily be wrong.
Quoted 1 line
You will need to reinstall grub if this was the bootable disk, since
there
Quoted 1 line
were 384 bytes of grub in the sector with the partition table that you
know
Quoted 1 line
are missing.
Fortunately, this is all data, nothing to do with the boot sequence, except that the machine will not boot with the missing PV.
Thank you,
Brian_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Quoted 1 line · Roger Heflin
replying to my last email.
Thank you for your help, Roger. I will answer with more detail when the sun comes around to my side of the planet, but will say that this is a "secondary" VG, so your suggestion of changing fstab is very attractive.
B
_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/