Unable to remove a pvmove LV

The problem appeared after I've had a problem with this controller:

0000:00:0e.0 RAID bus controller: Silicon Image, Inc. (formerly CMD Technology Inc) PCI0680 Ultra ATA-133 Host Controller (rev 02)

I've added two more disks to this controller and to the volume group and messages similar to this:

Oct 26 20:28:21 serv0 kernel: hda: dma_intr: status=0x7f { DriveReady DeviceFault SeekComplete DataRequest CorrectedError Index Error }
Oct 26 20:28:21 serv0 kernel: hda: dma_intr: error=0x00 { }
Oct 26 20:28:21 serv0 kernel: hda: DMA disabled
Oct 26 20:28:21 serv0 kernel: hdb: DMA disabled
Oct 26 20:28:21 serv0 kernel: ide0: reset: success

began to appiear in the logs. I've tried removing the new disks but this LV became invincible. The machine is running Debian sarge. Details follow:

serv0:/# dpkg -s lvm2 |grep Version
Version: 2.01.04-5
serv0:/# uname -a
Linux serv0 2.6.8-2-k7 #1 Thu May 19 18:03:29 JST 2005 i686 GNU/Linux
serv0:/# lvdisplay
  --- Logical volume ---
  LV Name                /dev/s0_data/data
  VG Name                s0_data
  LV UUID                Wzs9Lu-5IUj-DK88-UVnZ-QJpj-gHuD-3a01jq
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                1.02 TB
  Current LE             4164
  Segments               23
  Allocation             inherit
  Read ahead sectors     0
  Block device           254:0

--- Logical volume ---

  LV Name                /dev/s0_data/pvmove0
  VG Name                s0_data
  LV UUID                H8As4Y-m61V-krSa-lGTd-pNRh-Ef1e-UStXSC
  LV Write Access        read/write
  LV Status              NOT available
  LV Size                46.50 GB
  Current LE             186
  Segments               1
  Allocation             contiguous
  Read ahead sectors     0

serv0:/# lvremove -v -f -d /dev/s0_data/pvmove0
    Using logical volume(s) on command line
  Can't remove locked LV pvmove0
serv0:/# pvmove --abort -v
    Finding all volume groups
    Finding volume group "s0_data"
serv0:/# lvdisplay
  --- Logical volume ---
  LV Name                /dev/s0_data/data
  VG Name                s0_data
  LV UUID                Wzs9Lu-5IUj-DK88-UVnZ-QJpj-gHuD-3a01jq
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                1.02 TB
  Current LE             4164
  Segments               23
  Allocation             inherit
  Read ahead sectors     0
  Block device           254:0

--- Logical volume ---

  LV Name                /dev/s0_data/pvmove0
  VG Name                s0_data
  LV UUID                H8As4Y-m61V-krSa-lGTd-pNRh-Ef1e-UStXSC
  LV Write Access        read/write
  LV Status              NOT available
  LV Size                46.50 GB
  Current LE             186
  Segments               1
  Allocation             contiguous
  Read ahead sectors     0

serv0:/#

Additionally the pvmove process was killed several times by the OOM killer, since in my opinion it was leaking memory. The PVs are 50 GBs each, and at some time at over 90 % the pvmove process was eating all the memory(512 MBs). Surprisingly after trying to resume some minutes later I've found the move had completed successfully.

Cheers,
Delian

The userspace process just manage and monitor the pvmove - the actual work is done by a kernel thread.

So killing the userspace processes doesn't stop the pvmove. Only pvmove --abort is capable of telling the kernel to abandon it.

Alasdair

[email protected]

The bug in the userspace is quite nasty though. Has it been reported and/or fixed in the recent versions ?

Additionally, would you suggest a way to remove the stale LV ? Ofcourse as a last step I might go creating a new VG and moving the data to it, but moving a terrabyte is not s.t. I would like to do often.

On the other side, the stale LV is not causing any problems ..

Cheers,
Delian

Another way to remove a misbehaving pvmove logical volume is to restore to a metadata config state previous to the pvmove command. The command to do this is vgcfgrestore. This is only possible if you haven't modified the volume group or the logical volume in a way that might cause you to lose something important. For example if you created a new LV and put a filesystem on it *after* a failed or uncleaned pvmove, you CANNOT restore to the config before the pvmove or you'll lose your new LV.

DISCLAIMER: I don't know what happens if you (accidentally?) try to restore metadata that results in removing an active, in use volume group or logical volume. Does anyone have any information about how dangerous this is? Is there a check for active VGs or LVs when restoring previous metadata with vgcfgrestore?

For example, assuming a volume group named "vg0", to see the metadata config history for vg0 use:

vgcfgrestore -l vg0

Look for the desired config before the pvmove command. Let's say the pvmove happened after the vg0_00009.vg metadata backup was stored, then restore that config with:

vgcfgrestore -v --file /etc/lvm/archive/vg0_00009.vg vg0

Be careful what you restore and look at any recent changes carefully before using vgcfgrestore.

Regards,
Randy

..:.::::
Randall Jones     GST      NASA Goddard Space Flight Center
HPC Visualization Support       http://hpcvis.gsfc.nasa.gov
Scientific Visualization Studio    http://svs.gsfc.nasa.gov
[email protected]      Code 610.3      301-286-2239

Unfortunately I've removed several PVs from it to leave the problematic cotroller with two HDDs only(which was proven to work).

Cheers,
Delian