Skip to contentSkip to conversation

No device found for PV but is actually fine

linux-lvm Michal Svoboda 3 participants 6 messages May 8, 2014 – May 13, 2014

lore
Share & tools
lore

Michal Svoboda

author
#0 lore raw

Hi all,

I have a weird issue:

# pvs
 No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
 PV         VG    Fmt  Attr PSize   PFree
 /dev/sdc   data2 lvm2 a--  298.09g    0 
 /dev/sdd1  data  lvm2 a--  298.09g    0 
# pvscan 
  No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
  No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
  PV /dev/sdc    VG data2   lvm2 [298.09 GiB / 0    free]
  PV /dev/sdd1   VG data    lvm2 [298.09 GiB / 0    free]
  Total: 2 [596.18 GiB] / in use: 2 [596.18 GiB] / in no VG: 0 [0   ]
# pvscan --cache
  Found duplicate PV JszGc1S16sbsFQweAlaj8Wd6y4BeVqcN: using /dev/sde2 not /dev/md0
  Found duplicate PV JszGc1S16sbsFQweAlaj8Wd6y4BeVqcN: using /dev/sdf2 not /dev/sde2

The VG on the "missing" PV is called "system", but:

# vgdisplay /dev/system
  No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
  --- Volume group ---
  VG Name               system
  System ID             
  Format                lvm2
  Metadata Areas        0
  Metadata Sequence No  28
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                8
  Open LV               8
  Max PV                0
  Cur PV                1
  Act PV                0
  VG Size               232.88 GiB
  PE Size               4.00 MiB
  Total PE              59616
  Alloc PE / Size       33792 / 132.00 GiB
  Free  PE / Size       25824 / 100.88 GiB
  VG UUID               fQoJ8p-BI3D-BcEH-BiQX-JJqf-p6TT-6KIoRt

Actually all LVs on the "system" VG are present and accounted for. The LVs contain the root file system, and some virtual machine drives, all of which are running *fine* (and the system in fact boots up from the "missing" PV :). But I can't resize:

# lvextend -L +16G /dev/system/vm-monitor
  No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
  Cannot change VG system while PVs are missing.
  Consider vgreduce --removemissing.

The "system" VG is situated on a md raid1. I think it all happened after I replaced all the drives of that md with mdadm --replace with bigger disks. But I can't be sure, because the system in fact works and except the inability to resize there are no problems.

I have googled around a bit, but to no avail. Can someone help me diagnose and repair this?

Version info:

# pvs --version
  LVM version:     2.02.105(2) (2014-01-20)
  Library version: 1.02.84 (2014-01-20)
  Driver version:  4.27.0
# cat /proc/version 
Linux version 3.14.0-4-ARCH (nobody@var-lib-archbuild-extra-x86_64-thomas) (gcc version 4.8.2 20140206 (prerelease) (GCC) ) #1 SMP PREEMPT Wed Apr 9 21:11:25 CEST 2014

Thanks,
Michal Svoboda

#1 lore raw

Hi Michael,

Check your pvs output to see if any of your pvs have the 'm'
attribute. This means the device has been marked as missing in the metadata.

You can remove this attribute by running the vgextend command.

    # vgextend --restoremissing /dev/device

I usually see this happen on virtual machines. Is this setup running on a virt?

Jack Waterworth
Senior Technical Support Engineer
Red Hat Global Support Services North America
919.754.4625

Quoted 83 lines · Michal Svoboda

Hi all,

I have a weird issue:

# pvs
  No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
  PV         VG    Fmt  Attr PSize   PFree
  /dev/sdc   data2 lvm2 a--  298.09g    0
  /dev/sdd1  data  lvm2 a--  298.09g    0
# pvscan
   No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
   No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
   PV /dev/sdc    VG data2   lvm2 [298.09 GiB / 0    free]
   PV /dev/sdd1   VG data    lvm2 [298.09 GiB / 0    free]
   Total: 2 [596.18 GiB] / in use: 2 [596.18 GiB] / in no VG: 0 [0   ]
# pvscan --cache
   Found duplicate PV JszGc1S16sbsFQweAlaj8Wd6y4BeVqcN: using /dev/sde2 not /dev/md0
   Found duplicate PV JszGc1S16sbsFQweAlaj8Wd6y4BeVqcN: using /dev/sdf2 not /dev/sde2

The VG on the "missing" PV is called "system", but:

# vgdisplay /dev/system
   No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
   --- Volume group ---
   VG Name               system
   System ID
   Format                lvm2
   Metadata Areas        0
   Metadata Sequence No  28
   VG Access             read/write
   VG Status             resizable
   MAX LV                0
   Cur LV                8
   Open LV               8
   Max PV                0
   Cur PV                1
   Act PV                0
   VG Size               232.88 GiB
   PE Size               4.00 MiB
   Total PE              59616
   Alloc PE / Size       33792 / 132.00 GiB
   Free  PE / Size       25824 / 100.88 GiB
   VG UUID               fQoJ8p-BI3D-BcEH-BiQX-JJqf-p6TT-6KIoRt

Actually all LVs on the "system" VG are present and accounted for. The LVs contain the root file system, and some virtual machine drives, all of which are running *fine* (and the system in fact boots up from the "missing" PV :). But I can't resize:

# lvextend -L +16G /dev/system/vm-monitor
   No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
   Cannot change VG system while PVs are missing.
   Consider vgreduce --removemissing.

The "system" VG is situated on a md raid1. I think it all happened after I replaced all the drives of that md with mdadm --replace with bigger disks. But I can't be sure, because the system in fact works and except the inability to resize there are no problems.

I have googled around a bit, but to no avail. Can someone help me diagnose and repair this?

Version info:

# pvs --version
   LVM version:     2.02.105(2) (2014-01-20)
   Library version: 1.02.84 (2014-01-20)
   Driver version:  4.27.0
# cat /proc/version
Linux version 3.14.0-4-ARCH (nobody@var-lib-archbuild-extra-x86_64-thomas) (gcc version 4.8.2 20140206 (prerelease) (GCC) ) #1 SMP PREEMPT Wed Apr 9 21:11:25 CEST 2014

Thanks,
Michal Svoboda

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#2 lore raw
In reply to Michal Svoboda
Quoted 17 lines · Michal Svoboda

Hi all,

I have a weird issue:

# pvs
  No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
  PV         VG    Fmt  Attr PSize   PFree
  /dev/sdc   data2 lvm2 a--  298.09g    0
  /dev/sdd1  data  lvm2 a--  298.09g    0
# pvscan
   No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
   No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
   PV /dev/sdc    VG data2   lvm2 [298.09 GiB / 0    free]
   PV /dev/sdd1   VG data    lvm2 [298.09 GiB / 0    free]
   Total: 2 [596.18 GiB] / in use: 2 [596.18 GiB] / in no VG: 0 [0   ]
Quoted 3 lines
# pvscan --cache
   Found duplicate PV JszGc1S16sbsFQweAlaj8Wd6y4BeVqcN: using /dev/sde2 not /dev/md0
   Found duplicate PV JszGc1S16sbsFQweAlaj8Wd6y4BeVqcN: using /dev/sdf2 not /dev/sde2

Peter, is this an error in LVM filtering or is manual filter modification required?

Michal(-e), I suggest modifying global_filter to accept only the devices you want in the VG.

-- Martian

Quoted 64 lines

The VG on the "missing" PV is called "system", but:

# vgdisplay /dev/system
   No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
   --- Volume group ---
   VG Name               system
   System ID
   Format                lvm2
   Metadata Areas        0
   Metadata Sequence No  28
   VG Access             read/write
   VG Status             resizable
   MAX LV                0
   Cur LV                8
   Open LV               8
   Max PV                0
   Cur PV                1
   Act PV                0
   VG Size               232.88 GiB
   PE Size               4.00 MiB
   Total PE              59616
   Alloc PE / Size       33792 / 132.00 GiB
   Free  PE / Size       25824 / 100.88 GiB
   VG UUID               fQoJ8p-BI3D-BcEH-BiQX-JJqf-p6TT-6KIoRt

Actually all LVs on the "system" VG are present and accounted for. The LVs contain the root file system, and some virtual machine drives, all of which are running *fine* (and the system in fact boots up from the "missing" PV :). But I can't resize:

# lvextend -L +16G /dev/system/vm-monitor
   No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
   Cannot change VG system while PVs are missing.
   Consider vgreduce --removemissing.

The "system" VG is situated on a md raid1. I think it all happened after I replaced all the drives of that md with mdadm --replace with bigger disks. But I can't be sure, because the system in fact works and except the inability to resize there are no problems.

I have googled around a bit, but to no avail. Can someone help me diagnose and repair this?

Version info:

# pvs --version
   LVM version:     2.02.105(2) (2014-01-20)
   Library version: 1.02.84 (2014-01-20)
   Driver version:  4.27.0
# cat /proc/version
Linux version 3.14.0-4-ARCH (nobody@var-lib-archbuild-extra-x86_64-thomas) (gcc version 4.8.2 20140206 (prerelease) (GCC) ) #1 SMP PREEMPT Wed Apr 9 21:11:25 CEST 2014

Thanks,
Michal Svoboda

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

author
#3 lore raw
Quoted 2 lines · Marian Csontos

Michal(-e), I suggest modifying global_filter to accept only the devices you want in the VG.

Hi,

I've tried playing with filter and global_filter in lvm.conf, but it did not help - as if the conf file was ignored. I tried blacklisting and whitelisting. Does something need to be restarted/rebooted for these settings to take effect?

I then did some more experiments and eventually solved the issue.

First I tried pvscan --cache /dev/md0, at which point md0 was "accepted" as existing PV, and the "system" VG would show up in vgdisplay. I was even able to resize one of the LVs. However, after running pvscan --cache with no extra agruments, I would receive the duplicate PV errors as in previous e-mail and the "system" VG would disappear from vgdisplay. :-o

Finally I solved the issue by changing initcpio hook from "mdadm_udev" to "mdadm". This is arch-linux specific thing that changes stuff that is exported to the initial ramdisk used for boot. User-wise, that change would stop using udev auto-magic for assembling md arrays, but use /etc/mdadm.conf directly instead. I don't know the implementation details.

After that I receive *no more errors* about either missing or duplicate PV and all VGs/LVs are displayed by the respective display commands. So for me issue is solved but if anyone wants to get to the bottom I am open to do some more experiments.

Michal Svoboda

#4 lore raw
Quoted 10 lines · Michal Svoboda
Quoted 2 lines · Marian Csontos

Michal(-e), I suggest modifying global_filter to accept only the devices you want in the VG.

Hi,

I've tried playing with filter and global_filter in lvm.conf, but it did not help - as if the conf file was ignored. I tried blacklisting and whitelisting. Does something need to be restarted/rebooted for these settings to take effect?

Oh yes, you would need to restart lvmetad (on Fedora/RHEL/CentOS systemctl restart lvm2-lvmetad.service, not sure it is the same on Arch.)

You may want to scrub the RAID as there may be data not written to both legs.

IMO it would be the best to take over udev rules and systemd units from Fedora as Peter spends lot of time to keep them up to date and working.

I have seen some threads on Arch forum yesterday related to boot problems with LVM on mdadm after upgrading from lvm2-2.02.105 to .106 - I think people there will appreciate if you add a comment...

-- Martian

Quoted 29 lines

I then did some more experiments and eventually solved the issue.

First I tried pvscan --cache /dev/md0, at which point md0 was "accepted" as existing PV, and the "system" VG would show up in vgdisplay. I was even able to resize one of the LVs. However, after running pvscan --cache with no extra agruments, I would receive the duplicate PV errors as in previous e-mail and the "system" VG would disappear from vgdisplay. :-o

Finally I solved the issue by changing initcpio hook from "mdadm_udev" to "mdadm". This is arch-linux specific thing that changes stuff that is exported to the initial ramdisk used for boot. User-wise, that change would stop using udev auto-magic for assembling md arrays, but use /etc/mdadm.conf directly instead. I don't know the implementation details.

After that I receive *no more errors* about either missing or duplicate PV and all VGs/LVs are displayed by the respective display commands. So for me issue is solved but if anyone wants to get to the bottom I am open to do some more experiments.

Michal Svoboda

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

author
#5 lore raw
Quoted 3 lines · Marian Csontos

Oh yes, you would need to restart lvmetad (on Fedora/RHEL/CentOS systemctl restart lvm2-lvmetad.service, not sure it is the same on Arch.)

Thanks - yeah it's the same.

Quoted 2 lines

You may want to scrub the RAID as there may be data not written to both legs.

Thanks for this suggestion too. I did that, found some small amount of mismatch blocks. Don't know if they are linked to this problem but better safe than sorry.

Michal Svoboda