Skip to contentSkip to conversation

file-based locking initialization failed after pvresize

linux-lvm Christian Lahti 4 participants 8 messages May 30, 2013 – May 31, 2013 5 messages not in the archive

lore
Share & tools
lore
#0 lore raw

Hello all:

I have a VMWare virtual CentOS 6 server with the following disk layout:

/dev/sda 15GB
/dev/sdb 100GB

Standard CentOS LVM setup

on sda1
/dev/vg_centos6/lv_root mounted on /
/dev/vg_centos6/lv_swap

on sdb1
/dev/vg_centos6/lv_var mounted on /var

I was starting to run low on disk space for /var, so I shut down the machine gracefully, extended the disk in VMWare to 250G and rebooted in single user mode. I then ran the following two commands:

pvresize --setphysicalvolumesize 250G /dev/sdb1
lvresize -L250G /dev/vg_centos6/lv_var

I then rebooted expecting the /var to now be 250G instead of 100G. What I got was:

No such file or directory trying to open /dev/vg_centos6/lv_var

After dropping to a maintenance shell I could not run any pv* or lv* commands, always resulting in the error:

File-based locking initialization failed

After Googling forever I found some lv* and pv* commands take --ignorelockingfailure as a parameter, so now I can "see" the pv and lv information but I do not know how to recover this partition:

pvdisplay --ignorelockingfailure /dev/sdb1
File-based locking initialization failed
--- Physical volume ---
PV Name /dev/sdb1
VG Name vg_centos6
PV Size 250 GiB / not useable 3.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 63999
Free PE 25599
Allocated PE 38400

lvdisplay --ignorelockingfailure /dev/vg_centos6/lv_var File-based locking initialization failed
--- Logical volume ---
LV Path /dev/vg_centos6/lv_var
LV Name lv_var
VG Name vg_centos6
LV Write Access read/write
LV Status suspended

# open 0
LV Size 150.00 GiB
Current LE 38400
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:2

I cannot mount this, I cannot fsck, /dev/mapper/vg_centos6/lv_var does not even get created. This is a critical server, can anyone please tell me how to reverse this or at least recover the data?

Thanks!

/Christian

#1 lore raw

Hello all:

I have a VMWare virtual CentOS 6 server with the following disk layout:

/dev/sda 15GB
/dev/sdb 100GB

Standard CentOS LVM setup

on sda1
/dev/vg_centos6/lv_root mounted on /
/dev/vg_centos6/lv_swap

on sdb1
/dev/vg_centos6/lv_var mounted on /var

I was starting to run low on disk space for /var, so I shut down the machine gracefully, extended the disk in VMWare to 250G and rebooted in single user mode. I then ran the following two commands:

pvresize --setphysicalvolumesize 250G /dev/sdb1
lvresize -L250G /dev/vg_centos6/lv_var

I then rebooted expecting the /var to now be 250G instead of 100G. What I got was:

No such file or directory trying to open /dev/vg_centos6/lv_var

After dropping to a maintenance shell I could not run any pv* or lv* commands, always resulting in the error:

File-based locking initialization failed

After Googling forever I found some lv* and pv* commands take --ignorelockingfailure as a parameter, so now I can "see" the pv and lv information but I do not know how to recover this partition:

pvdisplay --ignorelockingfailure /dev/sdb1
File-based locking initialization failed
--- Physical volume ---
PV Name /dev/sdb1
VG Name vg_centos6
PV Size 250 GiB / not useable 3.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 63999
Free PE 25599
Allocated PE 38400

lvdisplay --ignorelockingfailure /dev/vg_centos6/lv_var File-based locking initialization failed
--- Logical volume ---
LV Path /dev/vg_centos6/lv_var
LV Name lv_var
VG Name vg_centos6
LV Write Access read/write
LV Status suspended

# open 0
LV Size 150.00 GiB
Current LE 38400
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:2

I cannot mount this, I cannot fsck, /dev/mapper/vg_centos6/lv_var does not even get created. This is a critical server, can anyone please tell me how to reverse this or at least recover the data?

Thanks!

/Christian

#4 lore raw
In reply to a message not in the archive

Sorry forgot to copy the list.

---------- Forwarded message ----------
From: Christian Lahti <[email protected]>
Date: Thu, May 30, 2013 at 8:36 PM
Subject: Re: [linux-lvm] file-based locking initialization failed after pvresize
To: matthew patton <[email protected]>

Thank you for your reply, I realize that I must have done something rather bone-headed and I am usually pretty clued. That being said I am still in the same pickle I was in to begin with, something constructive would be fantastically appreciated. If there were proper documentation *anywhere* google-able I would not have bothered the mailing list. I am pretty sure the data must be in tact since this was done in single user mode and no writes to the disk/volume have occurred since the two commands were run, I have /etc/lvm in tact with information prior to and after the event. If someone would be so kind as to please let me know exactly what information they might need, I will do my level best with minimum "back and forth" to provide such information to get out of said pickle. And I furthermore promise not to do it again.

/Christian

Quoted 12 lines · matthew patton

What's with people and "extending" disks? DO NOT DO THAT!!!! But if you do, then you can't forget to do all the rest of the necessary steps. You have to fix the partition table. And you have to make sure your PVresize doesn't run off the end of the media. Is VMware's 250G really 250G when probed by PVscan? Doubt it very much.

Next time don't partition disks for starters. And "grow" filesystems by ADDing disks and adding them to the volume group and then resize2fs. Just because Windows was written for dummies and lets you get away with what you did, doesn't mean you should attempt it in Linux if you don't know what you're doing.

#5 lore raw

Dne 31.5.2013 05:37, Christian Lahti napsal(a):

Quoted 9 lines

Sorry forgot to copy the list.

---------- Forwarded message ----------
From: *Christian Lahti* <[email protected] <mailto:[email protected]>>
Date: Thu, May 30, 2013 at 8:36 PM
Subject: Re: [linux-lvm] file-based locking initialization failed after pvresize
To: matthew patton <[email protected] <mailto:[email protected]>>

Quoted 11 lines

Thank you for your reply, I realize that I must have done something rather bone-headed and I am usually pretty clued. That being said I am still in the same pickle I was in to begin with, something constructive would be fantastically appreciated. If there were proper documentation *anywhere* google-able I would not have bothered the mailing list. I am pretty sure the data must be in tact since this was done in single user mode and no writes to the disk/volume have occurred since the two commands were run, I have /etc/lvm in tact with information prior to and after the event. If someone would be so kind as to please let me know exactly what information they might need, I will do my level best with minimum "back and forth" to provide such information to get out of said pickle. And I furthermore promise not to do it again.

I'm mostly convinced you've probably already made some irreparable damage to your disk data - since the story is usually - Data are intact - I've done only this and that :) - but anyway it's probably better trying to resolve this interactively on #lvm irc channel.

The easiest way is to restore to the previous version of your metadata,
just before you've started your pvresize command (which must have been loudly objecting to your command line options)

Also next time - when the tool is giving you WARNINGS - and you have no idea, what it means - it's not a sign to go forward, but to STOP immediately...

Regards

Zdenek

#6 lore raw
In reply to Christian Lahti

Hi,

Did you try booting off another system (e.g. live usb ) and restoring the LVM meta data ? If your root filesystem is still bootable, you can find a backup of the LVM metadata in /etc/lvm/backup, and then start over your data resizing procedure. Look for the vgcfgrestore command.

The right procedure to resize a LV after growing a PV is :
- adjust the partition table on sdb with fdisk, the simple way is deleting the partition table and creating a new one with the main partition having the same start offset (_very_ important)
- THEN pvresize the device (no need to add any --setphysicalwhatever argument)
- then lvresize, but if you want to be safe always check that the size you set is larger than the current volume size otherwise you can damage your data pretty bad. A good way to do this is to add a "+" in the lvresize command : lvresize -L +100G /device adds 100G to volume.
- then run a filesystem resize command, depending on the filesystem used (xfs_growfs for XFS, resize2fs for ext4)

Anyway, If you are going to use full disks with LVM, it's always easier to not even bother creating a partition table : use the whole device with LVM (/dev/sdX) instead of a partition (/dev/sdX1).

HTH,

Gabriel

Quoted 26 lines · Christian Lahti

Sorry forgot to copy the list.

---------- Forwarded message ----------
From: Christian Lahti <[email protected]>
Date: Thu, May 30, 2013 at 8:36 PM
Subject: Re: [linux-lvm] file-based locking initialization failed after pvresize
To: matthew patton <[email protected]>

Thank you for your reply, I realize that I must have done something rather bone-headed and I am usually pretty clued. That being said I am still in the same pickle I was in to begin with, something constructive would be fantastically appreciated. If there were proper documentation *anywhere* google-able I would not have bothered the mailing list. I am pretty sure the data must be in tact since this was done in single user mode and no writes to the disk/volume have occurred since the two commands were run, I have /etc/lvm in tact with information prior to and after the event. If someone would be so kind as to please let me know exactly what information they might need, I will do my level best with minimum "back and forth" to provide such information to get out of said pickle. And I furthermore promise not to do it again.

/Christian

On Thu, May 30, 2013 at 3:14 PM, matthew patton <[email protected]> wrote:
What's with people and "extending" disks? DO NOT DO THAT!!!! But if you do, then you can't forget to do all the rest of the necessary steps. You have to fix the partition table. And you have to make sure your PVresize doesn't run off the end of the media. Is VMware's 250G really 250G when probed by PVscan? Doubt it very much.

Next time don't partition disks for starters. And "grow" filesystems by ADDing disks and adding them to the volume group and then resize2fs. Just because Windows was written for dummies and lets you get away with what you did, doesn't mean you should attempt it in Linux if you don't know what you're doing.

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#11 lore raw
In reply to a message not in the archive

I am a linux expert

all the problems associated with the LVM
Linux systems are reliable.
LVM is not a problem, but your system and kernel
clean kernel.org occurs only in linux.
this system are:
kernel.org, debian, knopix,
and those that benefit from the source.

systems:
Ubuntu, CentOS, mac.OS, and all VirtualMachine type XEN. not equipped with LVM
only virtual vLVM
This is your fault,
because your system is part of the
IBM, AIX kernel

AIX kernel is not free, so dumb access to their source.

who would like to have a stable system

Contact [email protected]
https://plus.google.com/105547044162966304899

linux free

#7 lore raw
In reply to Gabriel Barazer

Dne 31.5.2013 11:04, Gabriel Barazer napsal(a):

Quoted 7 lines

Hi,

Did you try booting off another system (e.g. live usb ) and restoring the LVM meta data ? If your root filesystem is still bootable, you can find a backup of the LVM metadata in /etc/lvm/backup, and then start over your data resizing procedure. Look for the vgcfgrestore command.

Quoted 11 lines

The right procedure to resize a LV after growing a PV is :
- adjust the partition table on sdb with fdisk, the simple way is deleting the partition table and creating a new one with the main partition having the same start offset (_very_ important)
- THEN pvresize the device (no need to add any --setphysicalwhatever argument)
- then lvresize, but if you want to be safe always check that the size you set is larger than the current volume size otherwise you can damage your data pretty bad. A good way to do this is to add a "+" in the lvresize command :
lvresize -L +100G /device adds 100G to volume.
- then run a filesystem resize command, depending on the filesystem used (xfs_growfs for XFS, resize2fs for ext4)

For couple years lvresize has the option '-r' so it knows how to resize common filesystems (via fsadm script) - so it keeps the right order of commands (very important in case you are reducing size)

Zdenek

#12 lore raw
In reply to Christian Lahti

I was assisted on the #lvm IRC channel by kabi_ (thank you), the solution for others who might find themselves in the same boat:

* look in /etc/lvm/archive and find the file that has the information just prior to the destructive act
* in this text file is a UUID for the volume that has been corrupted
* boot into rescue disk (or single user mode if the corrupted disk is not your boot volume) and run the following
pvcreate --config 'global{locking_type=0}' --uuid <UUID> --restorefile /etc/lvm/backup/<file containing previous info> /dev/<device>
vgcfgrestore -f /etc/lvm/archive/<file containing previous info> <volume group name>

Thanks again!

/Christian

Quoted 67 lines · Christian Lahti

Hello all:

I have a VMWare virtual CentOS 6 server with the following disk layout:

/dev/sda 15GB
/dev/sdb 100GB

Standard CentOS LVM setup

on sda1
/dev/vg_centos6/lv_root mounted on /
/dev/vg_centos6/lv_swap

on sdb1
/dev/vg_centos6/lv_var mounted on /var

I was starting to run low on disk space for /var, so I shut down the machine gracefully, extended the disk in VMWare to 250G and rebooted in single user mode. I then ran the following two commands:

pvresize --setphysicalvolumesize 250G /dev/sdb1
lvresize -L250G /dev/vg_centos6/lv_var

I then rebooted expecting the /var to now be 250G instead of 100G. What I got was:

No such file or directory trying to open /dev/vg_centos6/lv_var

After dropping to a maintenance shell I could not run any pv* or lv* commands, always resulting in the error:

File-based locking initialization failed

After Googling forever I found some lv* and pv* commands take --ignorelockingfailure as a parameter, so now I can "see" the pv and lv information but I do not know how to recover this partition:

pvdisplay --ignorelockingfailure /dev/sdb1
File-based locking initialization failed
--- Physical volume ---
PV Name /dev/sdb1
VG Name vg_centos6
PV Size 250 GiB / not useable 3.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 63999
Free PE 25599
Allocated PE 38400

lvdisplay --ignorelockingfailure /dev/vg_centos6/lv_var File-based locking initialization failed
--- Logical volume ---
LV Path /dev/vg_centos6/lv_var
LV Name lv_var
VG Name vg_centos6
LV Write Access read/write
LV Status suspended

# open 0
LV Size 150.00 GiB
Current LE 38400
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:2

I cannot mount this, I cannot fsck, /dev/mapper/vg_centos6/lv_var does not even get created. This is a critical server, can anyone please tell me how to reverse this or at least recover the data?

Thanks!

/Christian