mirror and snapshot incompatible

According to lvcreate, a snapshot of a mirrored volume cannot be created ("Snapshots and mirrors may not yet be mixed"). However, devmapper does not seem to have the same limitation when used directly (not through lvm).

Is there a current reason for lvcreate to enforce this rather harsh limitation?

What are the risks I'm taking by manually creating a dm-snapshot from a dm-mirror device? (of course I'm also replacing the dm-mirror device with a snapshot-origin of the actual dm-mirror device).

If there is an actual reason for preventing this, what is the suggested way to consistently backup a mirror volume? Is there a way to temporarily take one of the mirror "legs" (mimage devices) offline in order to back it up, and resync it later?

Yoav

It is possible to do all the things you are asking through device-mapper... but there is no easy way to do it. (and LVM does not yet have the capabilities to manage this yet either.)

brassow

I know how to do it through device-mapper. I'm just worried that maybe there's a reason why lvm prevents it. Are you aware of such reason, or is it safe?

hello,
I can't make the clustered LVM running. it can't start the built-in
clustered locking mechanism (using cman).

lvm is compiled with following options:

./configure --with-clvmd=cman --with-cluster=shared --libdir=/usr/lib64/ --enable-dmeventd --enable-cmdlib

does anyone have some experiences that could help me make it working? thanks, steve

when the clvmd starts it reports this:

CLVMD[1772a6d0]: Oct 29 13:16:57 CLVMD started
CLVMD[1772a6d0]: Oct 29 13:16:57 Connected to CMAN
CLVMD[1772a6d0]: Oct 29 13:16:57 CMAN initialisation complete CLVMD[1772a6d0]: Oct 29 13:16:57 DLM initialisation complete CLVMD[1772a6d0]: Oct 29 13:16:57 Cluster ready, doing some more initialisation CLVMD[1772a6d0]: Oct 29 13:16:57 starting LVM thread
CLVMD[41d0c940]: Oct 29 13:16:57 LVM thread function started CLVMD[1772a6d0]: Oct 29 13:16:57 clvmd ready for work CLVMD[1772a6d0]: Oct 29 13:16:57 Using timeout of 60 seconds
File descriptor 3 (/dev/tty) leaked on lvm invocation. Parent PID 13867: clvmd File descriptor 5 (/dev/pts/3) leaked on lvm invocation. Parent PID 13867: clvmd File descriptor 7 (pipe:[12892]) leaked on lvm invocation. Parent PID 13867: clvmd File descriptor 8 (/dev/zero) leaked on lvm invocation. Parent PID 13867: clvmd
WARNING: Locking disabled. Be careful! This could corrupt your metadata.
CLVMD[41d0c940]: Oct 29 13:16:57 LVM thread waiting for work

one more report:

# vgscan
File descriptor 3 (/dev/tty) leaked on vgscan invocation. Parent PID 3389: bash
File descriptor 5 (/dev/pts/1) leaked on vgscan invocation. Parent PID 3389: bash
File descriptor 7 (pipe:[9066]) leaked on vgscan invocation. Parent PID 3389: bash
  Unknown locking type requested.
  Locking type 3 initialisation failed.

steve

Eurosoftware s.r.o.
[email protected]
+420 379 307 379
+420 724 554 104

If you've built using with-cluster=shared then the locking type in lvm.conf should be 2 and not 3, provided you have remembered to install the shared library that gets built.

I recommend you don't build it shared (the default), and use locking type 3

Chrissie

ok, I have recompiled it with --with-cluster=internal, but still seeing:

WARNING: Locking disabled. Be careful! This could corrupt your metadata.

in lvm.conf I have locking_type = 3, complete configure line was:

./configure --with-clvmd=cman --with-cluster=internal --libdir=/usr/lib64/ --enable-dmeventd --enable-cmdlib

cluster was compiled with following options:

./configure --disable_kernel_check --enable_xen --libdir=/usr/lib64/ --without_gnbd --without_kernel_modules --without_gfs --without_gfs2

running on kernel 2.6.25

any suggestion?

thanks steve

Eurosoftware s.r.o.
[email protected]
+420 379 307 379
+420 724 554 104

Is it during clvmd initialization only?

clvmd tries to initialize locks for possible already activated volumes (mostly safety/force restart check, clvmd must start before clustered LVs are activated anyway)

It simply run lvs command (forcing nolocking for this command) to check which volumes are activated.
The log message is misleading here...

After clvmd initialization, it should work as expected.

Milan

yes. where else could it occur?

the LVs are imho not activated but CLVMd still complains about disabled locking:

xen01:/etc/init.d # lvscan
File descriptor 3 (/dev/tty) leaked on lvscan invocation. Parent PID 2637: bash File descriptor 5 (/dev/pts/5) leaked on lvscan invocation. Parent PID 2637: bash File descriptor 7 (pipe:[721181]) leaked on lvscan invocation. Parent PID 2637: bash
inactive '/dev/xen/test' [3,00 GB] inherit
inactive '/dev/xen/test2' [3,00 GB] inherit
xen01:/etc/init.d # lvs
File descriptor 3 (/dev/tty) leaked on lvs invocation. Parent PID 2637: bash File descriptor 5 (/dev/pts/5) leaked on lvs invocation. Parent PID 2637: bash File descriptor 7 (pipe:[721181]) leaked on lvs invocation. Parent PID 2637: bash

  LV    VG   Attr   LSize Origin Snap%  Move Log Copy%  Convert
  test  xen  -wi--- 3,00G
  test2 xen  -wi--- 3,00G

this is the CLVMd log written during those command invocation:

xen01:/usr/local/src/LVM2.2.02.41 # clvmd -d 2

CLVMD[c7bbc6d0]: Nov  3 19:49:56 CLVMD started
CLVMD[c7bbc6d0]: Nov  3 19:49:56 Connected to CMAN
CLVMD[c7bbc6d0]: Nov  3 19:49:56 CMAN initialisation complete
CLVMD[c7bbc6d0]: Nov  3 19:49:57 DLM initialisation complete
CLVMD[c7bbc6d0]: Nov  3 19:49:57 Cluster ready, doing some more initialisation
CLVMD[c7bbc6d0]: Nov  3 19:49:57 starting LVM thread
CLVMD[426e3940]: Nov  3 19:49:57 LVM thread function started
CLVMD[c7bbc6d0]: Nov  3 19:49:57 clvmd ready for work
CLVMD[c7bbc6d0]: Nov  3 19:49:57 Using timeout of 60 seconds

File descriptor 3 (/dev/tty) leaked on lvm invocation. Parent PID 13833: clvmd File descriptor 5 (/dev/pts/3) leaked on lvm invocation. Parent PID 13833: clvmd File descriptor 7 (pipe:[712876]) leaked on lvm invocation. Parent PID 13833: clvmd File descriptor 8 (/dev/zero) leaked on lvm invocation. Parent PID 13833: clvmd
WARNING: Locking disabled. Be careful! This could corrupt your metadata.

CLVMD[426e3940]: Nov  3 19:49:57 LVM thread waiting for work
CLVMD[c7bbc6d0]: Nov  3 19:50:09 Got new connection on fd 12
CLVMD[c7bbc6d0]: Nov  3 19:50:09 Read on local socket 12, len = 26
CLVMD[c7bbc6d0]: Nov  3 19:50:09 creating pipe, [13, 14]
CLVMD[c7bbc6d0]: Nov  3 19:50:09 Creating pre&post thread
CLVMD[40f3e940]: Nov  3 19:50:09 in sub thread: client = 0x592d60
CLVMD[40f3e940]: Nov  3 19:50:09 Sub thread ready for work.
CLVMD[40f3e940]: Nov  3 19:50:09 doing PRE command LOCK_VG 'V_xen' at 1 (client=0x592d60)
CLVMD[40f3e940]: Nov  3 19:50:09 sync_lock: 'V_xen' mode:3 flags=0
CLVMD[40f3e940]: Nov  3 19:50:09 sync_lock: returning lkid 5d0001
CLVMD[40f3e940]: Nov  3 19:50:09 Writing status 0 down pipe 14
CLVMD[40f3e940]: Nov  3 19:50:09 Waiting to do post command - state = 0
CLVMD[c7bbc6d0]: Nov  3 19:50:09 Created pre&post thread, state = 0
CLVMD[c7bbc6d0]: Nov  3 19:50:09 read on PIPE 13: 4 bytes: status: 0
CLVMD[c7bbc6d0]: Nov  3 19:50:09 background routine status was 0, sock_client=0x592d60
CLVMD[c7bbc6d0]: Nov  3 19:50:09 distribute command: XID = 0
CLVMD[c7bbc6d0]: Nov  3 19:50:09 add_to_lvmqueue: cmd=0x593200. client=0x592d60, msg=0x592ee0, len=26, csid=(nil), xid=0
CLVMD[426e3940]: Nov  3 19:50:09 process_work_item: local
CLVMD[426e3940]: Nov  3 19:50:09 process_local_command: LOCK_VG (0x33) msg=0x593240, msglen =26, client=0x592d60
CLVMD[426e3940]: Nov  3 19:50:09 Dropping metadata for VG xen
CLVMD[426e3940]: Nov  3 19:50:09 Reply from node xen01.es.gk-software.com: 0 bytes
CLVMD[426e3940]: Nov  3 19:50:09 Got 1 replies, expecting: 1
CLVMD[426e3940]: Nov  3 19:50:09 LVM thread waiting for work
CLVMD[40f3e940]: Nov  3 19:50:09 Got post command condition...
CLVMD[40f3e940]: Nov  3 19:50:09 Waiting for next pre command
CLVMD[c7bbc6d0]: Nov  3 19:50:09 read on PIPE 13: 4 bytes: status: 0
CLVMD[c7bbc6d0]: Nov  3 19:50:09 background routine status was 0, sock_client=0x592d60
CLVMD[c7bbc6d0]: Nov  3 19:50:09 Send local reply
CLVMD[c7bbc6d0]: Nov  3 19:50:09 Read on local socket 12, len = 26
CLVMD[40f3e940]: Nov  3 19:50:09 Got pre command condition...
CLVMD[40f3e940]: Nov  3 19:50:09 doing PRE command LOCK_VG 'V_xen' at 6 (client=0x592d60)
CLVMD[40f3e940]: Nov  3 19:50:09 sync_unlock: 'V_xen' lkid:5d0001
CLVMD[40f3e940]: Nov  3 19:50:09 Writing status 0 down pipe 14
CLVMD[40f3e940]: Nov  3 19:50:09 Waiting to do post command - state = 0
CLVMD[c7bbc6d0]: Nov  3 19:50:09 read on PIPE 13: 4 bytes: status: 0
CLVMD[c7bbc6d0]: Nov  3 19:50:09 background routine status was 0, sock_client=0x592d60
CLVMD[c7bbc6d0]: Nov  3 19:50:09 distribute command: XID = 1
CLVMD[c7bbc6d0]: Nov  3 19:50:09 add_to_lvmqueue: cmd=0x593200. client=0x592d60, msg=0x592ee0, len=26, csid=(nil), xid=1
CLVMD[426e3940]: Nov  3 19:50:09 process_work_item: local
CLVMD[426e3940]: Nov  3 19:50:09 process_local_command: LOCK_VG (0x33) msg=0x5931d0, msglen =26, client=0x592d60
CLVMD[426e3940]: Nov  3 19:50:09 Dropping metadata for VG xen
CLVMD[426e3940]: Nov  3 19:50:09 Reply from node xen01.es.gk-software.com: 0 bytes
CLVMD[426e3940]: Nov  3 19:50:09 Got 1 replies, expecting: 1
CLVMD[426e3940]: Nov  3 19:50:09 LVM thread waiting for work
CLVMD[40f3e940]: Nov  3 19:50:09 Got post command condition...
CLVMD[40f3e940]: Nov  3 19:50:09 Waiting for next pre command
CLVMD[c7bbc6d0]: Nov  3 19:50:09 read on PIPE 13: 4 bytes: status: 0
CLVMD[c7bbc6d0]: Nov  3 19:50:09 background routine status was 0, sock_client=0x592d60
CLVMD[c7bbc6d0]: Nov  3 19:50:09 Send local reply
CLVMD[c7bbc6d0]: Nov  3 19:50:09 Read on local socket 12, len = 0
CLVMD[c7bbc6d0]: Nov  3 19:50:09 EOF on local socket: inprogress=0
CLVMD[c7bbc6d0]: Nov  3 19:50:09 Waiting for child thread
CLVMD[40f3e940]: Nov  3 19:50:09 Got pre command condition...
CLVMD[40f3e940]: Nov  3 19:50:09 Subthread finished
CLVMD[c7bbc6d0]: Nov  3 19:50:09 Joined child thread
CLVMD[c7bbc6d0]: Nov  3 19:50:09 ret == 0, errno = 2. removing client
CLVMD[c7bbc6d0]: Nov  3 19:50:09 add_to_lvmqueue: cmd=0x592f10. client=0x592d60, msg=(nil), len=0, csid=(nil), xid=1
CLVMD[426e3940]: Nov  3 19:50:09 process_work_item: free fd 12
CLVMD[426e3940]: Nov  3 19:50:09 LVM thread waiting for work
CLVMD[c7bbc6d0]: Nov  3 19:50:12 Got new connection on fd 12
CLVMD[c7bbc6d0]: Nov  3 19:50:12 Read on local socket 12, len = 26
CLVMD[c7bbc6d0]: Nov  3 19:50:12 creating pipe, [13, 14]
CLVMD[c7bbc6d0]: Nov  3 19:50:12 Creating pre&post thread
CLVMD[40f3e940]: Nov  3 19:50:12 in sub thread: client = 0x592d60
CLVMD[40f3e940]: Nov  3 19:50:12 Sub thread ready for work.
CLVMD[40f3e940]: Nov  3 19:50:12 doing PRE command LOCK_VG 'V_xen' at 1 (client=0x592d60)
CLVMD[40f3e940]: Nov  3 19:50:12 sync_lock: 'V_xen' mode:3 flags=0
CLVMD[40f3e940]: Nov  3 19:50:12 sync_lock: returning lkid 38f0001
CLVMD[40f3e940]: Nov  3 19:50:12 Writing status 0 down pipe 14
CLVMD[40f3e940]: Nov  3 19:50:12 Waiting to do post command - state = 0
CLVMD[c7bbc6d0]: Nov  3 19:50:12 Created pre&post thread, state = 0
CLVMD[c7bbc6d0]: Nov  3 19:50:12 read on PIPE 13: 4 bytes: status: 0
CLVMD[c7bbc6d0]: Nov  3 19:50:12 background routine status was 0, sock_client=0x592d60
CLVMD[c7bbc6d0]: Nov  3 19:50:12 distribute command: XID = 2
CLVMD[c7bbc6d0]: Nov  3 19:50:12 add_to_lvmqueue: cmd=0x592f10. client=0x592d60, msg=0x592ee0, len=26, csid=(nil), xid=2
CLVMD[426e3940]: Nov  3 19:50:12 process_work_item: local
CLVMD[426e3940]: Nov  3 19:50:12 process_local_command: LOCK_VG (0x33) msg=0x592f80, msglen =26, client=0x592d60
CLVMD[426e3940]: Nov  3 19:50:12 Dropping metadata for VG xen
CLVMD[426e3940]: Nov  3 19:50:12 Reply from node xen01.es.gk-software.com: 0 bytes
CLVMD[426e3940]: Nov  3 19:50:12 Got 1 replies, expecting: 1
CLVMD[426e3940]: Nov  3 19:50:12 LVM thread waiting for work
CLVMD[40f3e940]: Nov  3 19:50:12 Got post command condition...
CLVMD[40f3e940]: Nov  3 19:50:12 Waiting for next pre command
CLVMD[c7bbc6d0]: Nov  3 19:50:12 read on PIPE 13: 4 bytes: status: 0
CLVMD[c7bbc6d0]: Nov  3 19:50:12 background routine status was 0, sock_client=0x592d60
CLVMD[c7bbc6d0]: Nov  3 19:50:12 Send local reply
CLVMD[c7bbc6d0]: Nov  3 19:50:12 Read on local socket 12, len = 26
CLVMD[40f3e940]: Nov  3 19:50:12 Got pre command condition...
CLVMD[40f3e940]: Nov  3 19:50:12 doing PRE command LOCK_VG 'V_xen' at 6 (client=0x592d60)
CLVMD[40f3e940]: Nov  3 19:50:12 sync_unlock: 'V_xen' lkid:38f0001
CLVMD[40f3e940]: Nov  3 19:50:12 Writing status 0 down pipe 14
CLVMD[40f3e940]: Nov  3 19:50:12 Waiting to do post command - state = 0
CLVMD[c7bbc6d0]: Nov  3 19:50:12 read on PIPE 13: 4 bytes: status: 0
CLVMD[c7bbc6d0]: Nov  3 19:50:12 background routine status was 0, sock_client=0x592d60
CLVMD[c7bbc6d0]: Nov  3 19:50:12 distribute command: XID = 3
CLVMD[c7bbc6d0]: Nov  3 19:50:12 add_to_lvmqueue: cmd=0x592f10. client=0x592d60, msg=0x592ee0, len=26, csid=(nil), xid=3
CLVMD[426e3940]: Nov  3 19:50:12 process_work_item: local
CLVMD[426e3940]: Nov  3 19:50:12 process_local_command: LOCK_VG (0x33) msg=0x592f50, msglen =26, client=0x592d60
CLVMD[426e3940]: Nov  3 19:50:12 Dropping metadata for VG xen
CLVMD[426e3940]: Nov  3 19:50:12 Reply from node xen01.es.gk-software.com: 0 bytes
CLVMD[426e3940]: Nov  3 19:50:12 Got 1 replies, expecting: 1
CLVMD[426e3940]: Nov  3 19:50:12 LVM thread waiting for work
CLVMD[40f3e940]: Nov  3 19:50:12 Got post command condition...
CLVMD[40f3e940]: Nov  3 19:50:12 Waiting for next pre command
CLVMD[c7bbc6d0]: Nov  3 19:50:12 read on PIPE 13: 4 bytes: status: 0
CLVMD[c7bbc6d0]: Nov  3 19:50:12 background routine status was 0, sock_client=0x592d60
CLVMD[c7bbc6d0]: Nov  3 19:50:12 Send local reply
CLVMD[c7bbc6d0]: Nov  3 19:50:12 Read on local socket 12, len = 0
CLVMD[c7bbc6d0]: Nov  3 19:50:12 EOF on local socket: inprogress=0
CLVMD[c7bbc6d0]: Nov  3 19:50:12 Waiting for child thread
CLVMD[40f3e940]: Nov  3 19:50:12 Got pre command condition...
CLVMD[40f3e940]: Nov  3 19:50:12 Subthread finished
CLVMD[c7bbc6d0]: Nov  3 19:50:12 Joined child thread
CLVMD[c7bbc6d0]: Nov  3 19:50:12 ret == 0, errno = 9. removing client
CLVMD[c7bbc6d0]: Nov  3 19:50:12 add_to_lvmqueue: cmd=0x592f10. client=0x592d60, msg=(nil), len=0, csid=(nil), xid=3
CLVMD[426e3940]: Nov  3 19:50:12 process_work_item: free fd 12
CLVMD[426e3940]: Nov  3 19:50:12 LVM thread waiting for work

bye stepan

Eurosoftware s.r.o.
[email protected]
+420 379 307 379
+420 724 554 104

I have looked into the sources:

1) the clvmd starts the thread lvm_thread_fn() from clvmd.c
2) it calls the init_lvm() function from lvm-functions.c
3) it calls the get_initial_state() function from lvm-functions.c
4) in get_initial_state is following call:

FILE *lvs = popen

            ("lvm lvs  --config 'log{command_names=0 prefix=\"\"}' 
--nolocking --noheadings -o vg_uuid,lv_uuid,lv_attr,vg_attr",
             "r");

-> it calls lvm command with the --nolocking option - this is passed to all following steps:

5) the lvm commands calls lvm_run_command() function from lvmcmdline.c
6) it calls init_locking() function from locking.c
7) init_locking() than print the "locking disabled" warning, because the --nolocking option

SO:
a) is it alright, that the "lvm lvs" command is internally called with --nolocking option?
b) does it mean, that locking works alright for the rest of clvmd operations but the get_initial_state() call? in this case the debug message is very confusing.

bye stepan

Eurosoftware s.r.o.
[email protected]
+420 379 307 379
+420 724 554 104

Check there isn't another line in lvm.conf that puts the locking_type back to 1. clvmd prints that line out at startup if it detects that locking_type is not set to 3, or it is set to 2 and the library name is wrong, or locking_type is other than 2 or 3.

Chrissie

So the calling application (bash) forgot to close some descriptors (running lvm in mc shell usually produces this:-)

That's just warning, lvm close all descriptors, you can ignore these messages. (and if the parent process is clvmd, it is bug in clvmd and we should fix it :)

Locking type 2 and 3 are cluster locking
Type 2 is for external locking library, 3 is internal (compiled-in)

For type 3 you should use --with-cluster=internal in configure. (you can still switch type 2 with this setting, but if you do not use own locking library, it is probably not needed).

Milan
--
[email protected]

No, there isn't a reason LVM prevents it, other than we haven't gotten to it yet. That being said... because we haven't gotten to it yet also means there probably hasn't been a whole lot of testing of such a configuration. Therefore, it's difficult to quantify the safety.

brassow

For production environment I guess I'll switch to md+lvm for mirroring so that I'll be able to safely snapshot the mirrored fs. I'll keep playing with devmapper mirror combined with snapshot (without lvm) and report if it misbehaves. So far it hasn't, but I only did some limited experiments. I hope some future version of lvm will add support for this combination.

Yoav

There's another subject I'd like to raise here - snapshot cloning. Since we use snapshots for copying our (large) oracle databases for development and crash-test(especially crash test:) purposes, we needed to come up with a way to clone snapshots.

By the way - it's a cool way to copy databases and it's being used in solaris/zfs. For devel/(crash)testing it is great, because at little storage expense you can very quickly copy very large databases (we actually create snaps of live oracle databases(no archivelogs) and it works;) I wonder why it's so rarely mentioned here - everybody seem to use them as temporary backup snapshots only...

I haven't found a way to do it via lvm, but two ways without it. Both assume than origin (vg00/test) and cloned snapshot(vg00/snap) are not mounted or written to in any way.

I. the crude but simple way:

1) lvcreate -s -L orig_snap_size -n snapclone vg00/snap Create a cloning target

2) dd if=/dev/mapper/vg00-snap-cow of=/dev/mapper/vg00-snapclone-cow bs=1M count=percent_used
Copy the cow table from old to new snapshot. I assume, that since snapshot cow tables are filled up incrementally,it'll be sufficient to copy just the meaningful beginning instead of all useless data. This makes cloning a 30GB snapshot that is only 5% full very fast. Correct me if I'm wrong on this.

3) lvchange -an vg00/test; lvchange -ay vg00/test
Make lvm reload new data for the cloned snapshot. I haven't found other way to do it so far, so if you know any please let me know.

Done;)

The biggest drawback of this is than lvchange takes off line all other snapshots as well (so all cloned DB's would have to be closed), and this is evil;)

II. More elegant way. I've found a perl scipt posted on this list that was used to restore the origin from a snapshot. It simply reads cow table offsets and creates a set of dd commands to copy them from cow to the origin. However replacing the origin with another just-made snapshot of it creates a clone of the first one.

The drawback is that it uses dd's, so it launches enormous amount of processes and creates a lot of scattered reads/writes for an actually continuous area. And this'll probably be slow. But this can probably be optimized.

However both origin and snapshot still have to be unmounted there's no need to reload cow tables, so when the script is done, the clone is ready to use.

III. Finally - the ultimate way ;) It's just an idea - how I imagine it could work. Create clones on live snaps/lvs in a similar way pvmove works.

1) replace the cow-lv with a mirror target, that will create a copy of the cow table and still allow it to be used during the operation. If possible, mirror just enough from the beginnig, so that big snapshots with low % usage clone quickly.
2) when it's done, suspend the origin and cloned snap to ensure data cosistency
3) separate the mirrors
4) add a new snapshot entry with cow table on the just-mirrored lv.
5) resume normal operation

done:)

Am I wrong and it's not that simple? I don't know lvm/devmapper internals yet... However, if it's the right way to go, I might try to write something like this when I have some time to spare.

I think it's simpler than making snapshots of snapshots (from implementation's point of view)

What do you think about it?
Martin

Zfs has a send/receive with an incremental mode for remote cloning. Is there anything like that for lvm?

  Les Mikesell
   [email protected]