Strange pvmove problem

Hi!

I'm trying to move PEs away from two PVs (because I want to remove the disks from the system), but pvmove always fails:

# pvmove /dev/hdd1 /dev/sda3
    Wiping cache of LVM-capable devices
    Finding volume group "LVM"
    Archiving volume group "LVM" metadata (seqno 51).
    Creating logical volume pvmove0
    Moving 3468 extents of logical volume LVM/data
    Found volume group "LVM"
    Updating volume group metadata
    Creating volume group backup "/etc/lvm/backup/LVM" (seqno 52).
    Found volume group "LVM"
    Found volume group "LVM"
    Suspending LVM-data (253:1)
    Found volume group "LVM"
    Creating LVM-pvmove0
    Loading LVM-pvmove0 table
  device-mapper: reload ioctl failed: Invalid argument
  ABORTING: Temporary mirror activation failed.  Run pvmove --abort.
    Found volume group "LVM"
    Loading LVM-pvmove0 table
  device-mapper: reload ioctl failed: Invalid argument
    Loading LVM-data table
  device-mapper: reload ioctl failed: Invalid argument

After that the system remains in an unstable state. All [vg|pv|lv]* commands hang and the LV becomes unaccessible too.

After rebooting and excecuting "pvmove" without any arguments the VG becomes accessible again and I'm back where I started (without any PE moved). Trying to move data from another PV makes no difference.

The system's running with Debian Testing and Kernel 2.6.16.16.

Can anyone please help me finding out whats causing this?

ciao
Dariush

# pvdisplay
File descriptor 3 left open
File descriptor 4 left open
File descriptor 6 left open
File descriptor 8 left open
  --- Physical volume ---
  PV Name               /dev/hda3
  VG Name               LVM
  PV Size               129.48 GB / not usable 0
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              33147
  Free PE               33147
  Allocated PE          0
  PV UUID               uc4kn4-xMQ7-M4vw-1thl-OcCF-LIV9-aeRTn0

--- Physical volume ---

  PV Name               /dev/hdd1
  VG Name               LVM
  PV Size               37.27 GB / not usable 0
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              9541
  Free PE               6073
  Allocated PE          3468
  PV UUID               tNzm5d-ClTt-0vJ6-ZQA9-0eYQ-IwKy-tbxV6o

--- Physical volume ---

  PV Name               /dev/sda3
  VG Name               LVM
  PV Size               214.19 GB / not usable 0
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              54833
  Free PE               54734
  Allocated PE          99
  PV UUID               CfA2kz-5Gt8-Zshc-3Eou-fscj-1Lfn-xOEQSY

--- Physical volume ---

  PV Name               /dev/sdb1
  VG Name               LVM
  PV Size               232.88 GB / not usable 0
  Allocatable           yes (but full)
  PE Size (KByte)       4096
  Total PE              59618
  Free PE               0
  Allocated PE          59618
  PV UUID               L6S26w-zr8S-t0RN-si42-zWEd-rJA5-Ms0JIb

# vgdisplay
File descriptor 3 left open
File descriptor 4 left open
File descriptor 6 left open
File descriptor 8 left open
  --- Volume group ---
  VG Name               LVM
  System ID
  Format                lvm2
  Metadata Areas        4
  Metadata Sequence No  55
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                4
  Act PV                4
  VG Size               613.82 GB
  PE Size               4.00 MB
  Total PE              157139
  Alloc PE / Size       63185 / 246.82 GB
  Free  PE / Size       93954 / 367.01 GB
  VG UUID               GoVzA8-HPSg-FNhr-6fM3-Fo94-k2IG-462F9E

# lvdisplay
File descriptor 3 left open
File descriptor 4 left open
File descriptor 6 left open
File descriptor 8 left open
  --- Logical volume ---
  LV Name                /dev/LVM/data
  VG Name                LVM
  LV UUID                zscrIa-BwvW-rktv-3L5P-QP67-BkfG-ri3NZL
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                246.82 GB
  Current LE             63185
  Segments               3
  Allocation             inherit
  Read ahead sectors     0
  Block device           253:1

I had some problems with pvmove some time ago, if the PV to empty contained parts from several different LVs. But this was due to some old bug with a 2.6.12 or 13 kernel. May be it's interesting to have a look into /etc/lvm/backup/LVM and the associated backups, to understand whats going on.

To overcome the problem of hanging LVM I shut down my server and performed a "pvmove --abort" in single user mode while the LVM was not activated. This restored it to the state before.

Dieter.

Uhm, this is the file created right before I started pvmove:

<SNIP>
# Generated by LVM2: Fri Jul 28 10:15:24 2006

contents = "Text Format Volume Group"
version = 1

description = "Created *before* executing 'pvmove -v /dev/hdd1 /dev/hda3'"

creation_host = "palomar" # Linux palomar 2.6.16.16 #2 PREEMPT Sun

May 14 13:56:04 CEST 2006 i686

creation_time = 1154074524 # Fri Jul 28 10:15:24 2006

LVM {
        id = "GoVzA8-HPSg-FNhr-6fM3-Fo94-k2IG-462F9E"
        seqno = 39
        status = ["RESIZEABLE", "READ", "WRITE"]
        extent_size = 8192              # 4 Megabytes
        max_lv = 0
        max_pv = 0

        physical_volumes {

                pv0 {
                        id = "uc4kn4-xMQ7-M4vw-1thl-OcCF-LIV9-aeRTn0"
                        device = "/dev/hda3"    # Hint only

                        status = ["ALLOCATABLE"]
                        pe_start = 384
                        pe_count = 33147        # 129.48 Gigabytes
                }

                pv1 {
                        id = "tNzm5d-ClTt-0vJ6-ZQA9-0eYQ-IwKy-tbxV6o"
                        device = "/dev/hdd1"    # Hint only

                        status = ["ALLOCATABLE"]
                        pe_start = 384
                        pe_count = 9541 # 37.2695 Gigabytes
                }

                pv2 {
                        id = "CfA2kz-5Gt8-Zshc-3Eou-fscj-1Lfn-xOEQSY"
                        device = "/dev/sda3"    # Hint only

                        status = ["ALLOCATABLE"]
                        pe_start = 384
                        pe_count = 54833        # 214.191 Gigabytes
                }

                pv3 {
                        id = "L6S26w-zr8S-t0RN-si42-zWEd-rJA5-Ms0JIb"
                        device = "/dev/sdb1"    # Hint only

                        status = ["ALLOCATABLE"]
                        pe_start = 384
                        pe_count = 59618        # 232.883 Gigabytes
                }
        }

        logical_volumes {

                data {
                        id = "zscrIa-BwvW-rktv-3L5P-QP67-BkfG-ri3NZL"
                        status = ["READ", "WRITE", "VISIBLE"]
                        segment_count = 3

                        segment1 {
                                start_extent = 0
                                extent_count = 59618    # 232.883 Gigabytes

                                type = "striped"
                                stripe_count = 1        # linear

                                stripes = [
                                        "pv3", 0
                                ]
                        }
                        segment2 {
                                start_extent = 59618
                                extent_count = 3468     # 13.5469 Gigabytes

                                type = "striped"
                                stripe_count = 1        # linear

                                stripes = [
                                        "pv1", 6073
                                ]
                        }
                        segment3 {
                                start_extent = 63086
                                extent_count = 99       # 396 Megabytes

                                type = "striped"
                                stripe_count = 1        # linear

                                stripes = [
                                        "pv2", 54734
                                ]
                        }
                }
        }
}

</SNIP>

I diff'ed it againt more recent backups and against /etc/lvm/backup/LVM. They are pretty much the same (except the "seqno" attribute).

I'd be already happy if I could free /dev/sda3 (which is _nearly_ unused). Then I could create a new VG and move the files around by hand (like in the old days, when we had to use partitions). But even thats not possible since all disks big enough to hold the data are bound by this (damaged?) VG.

ciao
Dariush

....

seems the pvmove was not even started because of a problems with the underlaying device-mapper:

sorry, no idea :-(

any messages from syslog / dmesg?
does it work while LV not active (i.E. boot from rescue CD)?

Dieter.

Jupp, these:

device-mapper: unrecognised sync argument to mirror log: 2
device-mapper: dm-mirror: Error creating mirror dirty log
device-mapper: error adding target to table
device-mapper: unrecognised sync argument to mirror log: 2
device-mapper: dm-mirror: Error creating mirror dirty log
device-mapper: error adding target to table
device-mapper: device 253:1 too small for target
device-mapper: error adding target to table

Nope. :(

ciao
Dariush

You've got a kernel/userspace version mismatch on device-mapper and lvm2.

Check your config an upgrade.

Heinz

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Red Hat GmbH
Consulting Development Engineer                   Am Sonnenhang 11
Storage Development                               56242 Marienrachdorf
                                                  Germany
[email protected]                            PHONE +49  171 7803392
                                                  FAX   +49 2626 924446

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Ah, I thought with using Debian Testing I would have been using bleeding edge versions allready. Obviously I was wrong.

Thanks for the hint!

ciao
Dariush

I was able to workaround the problem by creating a new LV (which I placed exactly on those PVs which I intended to keep) and then moving all files to the the new LV by hand. This way I didn't have to use pvmove.

Anyway, thanks for your help!

ciao
Dariush