Problems with snapshot rollback
Hi,
we are trying to use lvm to be able to use snapshots. Basically what we need to do is to take a snapshot of the data of our mysql DBs, perform some data modification statements on them and then rollback to the initial point. We need to repeat the process several times, for testing purposes. However, something is not working or, at least, something is behaving in a really weird way. Not sure if it's a bug or something we're not considering.
Here's what we actually do:
We have an Amazon server running ubuntu 12.04 64bits and we have three EBS disks 1TB each. We built a LVM with those 3 disks and mounted there the mysql data.
This is how it looks with lvdisplay:
--- Logical volume ---
LV Name /dev/vg0/mysql VG Name vg0 LV UUID DOMf6L-Q25i-nU1F-LdKj-K4Be-arpB-oLJGQB LV Write Access read/write LV Status available # open 1 LV Size 800.00 GiB Current LE 204800 Segments 2 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 252:1
Then, we create the Snapshot in order to be able to get back to this point later:
# CREATE SNAPSHOT
# stop mysql in a clean way echo "set global innodb_fast_shutdown=0" | mysql service mysql stop # create the snapshot SNNAME=snmysql$(date +%Y%m%d%H%M%S) echo $SNNAME > /tmp/snname.log lvcreate -l100%FREE -s -n $SNNAME /dev/vg0/mysql
Now, lvdisplay adds this to the output:
--- Logical volume ---
LV Name /dev/vg0/snmysql20121220132053 VG Name vg0 LV UUID lfOmbb-jC6D-r1fz-EvC7-ieMD-mL1S-lVxcRl LV Write Access read/write LV snapshot status active destination for /dev/vg0/mysql LV Status available # open 0 LV Size 800.00 GiB Current LE 204800 COW-table size 779.86 GiB COW-table LE 199644 Allocated to snapshot 0.00% Snapshot chunk size 4.00 KiB Segments 3 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 252:2
At this point, we perform a set of some simple data modifications with a bash script which is quite simple:
service mysql start
seq 10 | while read i; do
echo "DROP DATABASE IF EXISTS db$i; CREATE DATABASE db$i" | mysql -uroot seq 100 | while read j; do
echo "CREATE TABLE t$j (i integer not null auto_increment primary
key)" | mysql -uroot db$i
echo "INSERT INTO t$j values (1); INSERT INTO t$j values (2);" |
mysql -uroot db$i
done
done
Everything looks ok so far.
Then, we try to rollback the changes by "merging" the snapshot we had made:
# ROLLBACK SN
echo "set global innodb_fast_shutdown=0" | mysql
service mysql stop
umount /mnt/mysql
# get the snapshot name we had used SNNAME=$(cat /tmp/snname.log) lvconvert --merge /dev/vg0/$SNNAME
And here's where we start getting weird behaviours. The first time it usually runs ok:
Merging of volume snmysql20121220132053 started.
mysql: Merged: 0.0%
mysql: Merged: 0.0%
Merge of snapshot into logical volume mysql has finished.
Logical volume "snmysql20121220132053" successfully removed
That's great and everything looks and works perfectly.
But, if we repeat the procedure (i.e. make the snapshot, modify data, rollback to snapshot), at some point (it could be the second, third or the Nth iteration) the lvconvert command starts beahving this way:
lvconvert --merge /dev/vg0/$SNNAME
Merging of volume snmysql20121220133552 started.
mysql: Merged: 0.0%
mysql: Merged: 100.0%
mysql: Merged: 100.0%
mysql: Merged: 100.0%
mysql: Merged: 100.0%
mysql: Merged: 100.0%
mysql: Merged: 100.0%
mysql: Merged: 100.0%
mysql: Merged: 100.0%
And it gets at that point repeating that 100.0% output forever. We can break it but, apparentely we can't break it as soon as it gets to 100% but a while later. Actually, we think that we could detect the moment by monitoring the situation using lvs. As long as lvs output looks like this, the status seems to be incomplete and we should not break it:
File descriptor 4 (pipe:[327994]) leaked on lvs invocation. Parent PID 32243: sh
/dev/dm-2: read failed after 0 of 4096 at 0: Input/output error
LV VG Attr LSize Origin Snap% Move Log Copy% Convert apt vg0 -wi-ao 100.00g mysql vg0 Owi-a- 800.00g 0.00
See the "Input/output error" legend. After some seconds, the lvs output looks like this:
File descriptor 4 (pipe:[325418]) leaked on lvs invocation. Parent PID 32413: sh
LV VG Attr LSize Origin Snap% Move Log Copy% Convert apt vg0 -wi-ao 100.00g mysql vg0 -wi-a- 800.00g
At this point, it seems to be safe to break the lvconvert --merge process.
Is this a right behaviour? Is this expectable? Would it be ok to work that way, I mean, just break the lvconvert process once the lvs output looks ok? Wouldn't it be a dirty way to work with it? Are we missing something?
Thanks in advance.
If you *intend* to rollback, have you considered doing the test in the snapshot itself (which can be written to) and not bothering with a later merge?
But for any unpredictable problems with devices, first thing to rule out is unintended interference from udev.
Alasdair