LVM snapshot setup for mysql-zrm backups problem
I'm not sure where is the best place to ask this so, since I am using LVM, I am asking here in the hope that someone can at least direct me to the appropriate place to ask this. This is my first time setting up LVM snapshots for mysql-zrm backups.
CentOS-6.6
MySQL-5.6.22
MySQL-zrm-3.0 (EPEL).
Short version:
I am unable to tell if the MySQL-zrm LVM snapshots are indeed taking place, or if the backup is being taken directly from the main database LVs. The log file info isn't giving me enough info to tell one way or another.
Some more info:
I am setting up a database server with all of the database files on LVM LVs and am using mysql-zrm for logical backups using LVM snapshots to reduce the database lock times.
I have followed all of the various setup instructions very closely, and am using the default "lvm-snapshot.pl" plugin. However, the only log file lines mentioning snapshot are the following:
mysql:backup:INFO: Running snapshot verification using command /usr/share/mysql-zrm/plugins/lvm-snapshot.pl --action verify-config 2>/home/tmp/xH8KKpsF2O
There are no other log entries following this one that seem to have anything to do with snapshots. There are no errors. The backups succeed. Both "--verbose" and "--noquiet" are enabled.
Here are the salient entries from the mysql-zrm.conf file:
backup-level=0
backup-mode=logical
lvm-snapshot=20G snapshot-plugin="/usr/share/mysql-zrm/plugins/lvm-snapshot.pl" backup-type=regular
destination=/home/mysql-zrm
retention-policy=4W
compress=1
Any info or help would be much appreciated!
Thanks,
--Phil
Signature · 2 lines
Philip Amadeo Saeli [email protected]
Quoted 1 line · Philip Amadeo Saeli
I'm not sure where is the best place to ask this so, since I am using
MySQL list, perhaps?
What does lvs say? Is there any snapshot of the LV where your database lives? Copy if to backup destination and try to check the content - I presume there is a tool to check the backup...
Just a word of warning:
if the script is using old-style snapshots, you want to copy and delete it ASAP, or you will see severely degraded performance, and much worse once you have multiple snapshots in the system.
Also once the space allocated for snapshot gets full, the snapshot becomes invalid, so hurry up!
-- Martian
Quoted 48 lines
LVM, I am asking here in the hope that someone can at least direct me to the appropriate place to ask this. This is my first time setting up LVM snapshots for mysql-zrm backups.
CentOS-6.6
MySQL-5.6.22
MySQL-zrm-3.0 (EPEL).Short version:
I am unable to tell if the MySQL-zrm LVM snapshots are indeed taking place, or if the backup is being taken directly from the main database LVs. The log file info isn't giving me enough info to tell one way or another.
Some more info:
I am setting up a database server with all of the database files on LVM LVs and am using mysql-zrm for logical backups using LVM snapshots to reduce the database lock times.
I have followed all of the various setup instructions very closely, and am using the default "lvm-snapshot.pl" plugin. However, the only log file lines mentioning snapshot are the following:
mysql:backup:INFO: Running snapshot verification using command /usr/share/mysql-zrm/plugins/lvm-snapshot.pl --action verify-config 2>/home/tmp/xH8KKpsF2OThere are no other log entries following this one that seem to have anything to do with snapshots. There are no errors. The backups succeed. Both "--verbose" and "--noquiet" are enabled.
Here are the salient entries from the mysql-zrm.conf file:
backup-level=0
backup-mode=logical
lvm-snapshot=20G snapshot-plugin="/usr/share/mysql-zrm/plugins/lvm-snapshot.pl" backup-type=regular
destination=/home/mysql-zrm
retention-policy=4W
compress=1Any info or help would be much appreciated!
Thanks,
--Phil
Thanks for the response. More info and questions following.
Quoted 5 lines · Marian Csontos
On 01/28/2015 06:46 AM, Philip Amadeo Saeli wrote:
What does
lvssay? Is there any snapshot of the LV where your database lives? Copy if to backup destination and try to check the content - I presume there is a tool to check the backup...
The backups appear to be OK. However, my suspicion is that they're being made from the main database LVs rather than from a freshly-made snapshot.
I am able to manually make a snapshot LV and it looks OK. I have also upped the LVM log level so I can better monitor any LVM errors or warnings.
However, I have another question:
When I run any of the various LVM cmds with the "-v" flag, I get a "DEGRADED MODE. Incomplete RAID LVs will be processed." message in the output. I find this surprising since none of the LVM vols on that system are RAID vols. They are simply single PV vols created for the sole purpose of enabling snapshots for consistent point-in-time database backups. Any ideas why it is printing that?
Quoted 5 lines
Just a word of warning:
if the script is using old-style snapshots, you want to copy and delete it ASAP, or you will see severely degraded performance, and much worse once you have multiple snapshots in the system.
What do you mean by "old-style" snapshots? Does that mean non-thin snapshots? Or something else? What type of degraded performance? Any references?
Note that the snapshots I'm using are ephemeral and are to exist only long enough for the actual database backup operation to complete.
Quoted 4 lines
Also once the space allocated for snapshot gets full, the snapshot becomes invalid, so hurry up!
-- Martian
Thanks!
--Phil
Quoted 46 lines
Quoted 46 lines
the appropriate place to ask this. This is my first time setting up LVM snapshots for mysql-zrm backups.
CentOS-6.6
MySQL-5.6.22
MySQL-zrm-3.0 (EPEL).Short version:
I am unable to tell if the MySQL-zrm LVM snapshots are indeed taking place, or if the backup is being taken directly from the main database LVs. The log file info isn't giving me enough info to tell one way or another.
Some more info:
I am setting up a database server with all of the database files on LVM LVs and am using mysql-zrm for logical backups using LVM snapshots to reduce the database lock times.
I have followed all of the various setup instructions very closely, and am using the default "lvm-snapshot.pl" plugin. However, the only log file lines mentioning snapshot are the following:
mysql:backup:INFO: Running snapshot verification using command /usr/share/mysql-zrm/plugins/lvm-snapshot.pl --action verify-config 2>/home/tmp/xH8KKpsF2OThere are no other log entries following this one that seem to have anything to do with snapshots. There are no errors. The backups succeed. Both "--verbose" and "--noquiet" are enabled.
Here are the salient entries from the mysql-zrm.conf file:
backup-level=0
backup-mode=logical
lvm-snapshot=20G snapshot-plugin="/usr/share/mysql-zrm/plugins/lvm-snapshot.pl" backup-type=regular
destination=/home/mysql-zrm
retention-policy=4W
compress=1Any info or help would be much appreciated!
Thanks,
--Phil
Signature · 2 lines
Philip Amadeo Saeli [email protected]
Quoted 5 lines · Philip Amadeo Saeli
Thanks for the response. More info and questions following.
Quoted 1 line · Marian Csontos
On 01/28/2015 06:46 AM, Philip Amadeo Saeli wrote:
Quoted 7 lines
Quoted 3 lines
What does
lvssay? Is there any snapshot of the LV where your database lives? Copy if to backup destination and try to check the content - I presume there is a tool to check the backup...The backups appear to be OK. However, my suspicion is that they're being made from the main database LVs rather than from a freshly-made snapshot.
That's exactly question for mysql-list.
You could monitor the snapshot with for example iotop or something to see if it is being read from but...
Quoted 5 lines
I am able to manually make a snapshot LV and it looks OK. I have also upped the LVM log level so I can better monitor any LVM errors or warnings.
Quoted 8 lines
However, I have another question:
When I run any of the various LVM cmds with the "-v" flag, I get a "DEGRADED MODE. Incomplete RAID LVs will be processed." message in the output. I find this surprising since none of the LVM vols on that system are RAID vols. They are simply single PV vols created for the sole purpose of enabling snapshots for consistent point-in-time database backups. Any ideas why it is printing that?
Post the output of the command with -vvvv, please. And may be lvmdump.
Quoted 9 lines
Quoted 6 lines
Just a word of warning:
if the script is using old-style snapshots, you want to copy and delete it ASAP, or you will see severely degraded performance, and much worse once you have multiple snapshots in the system.
What do you mean by "old-style" snapshots? Does that mean non-thin snapshots? Or something else?
Yes, non-thin snapshots.
Quoted 2 lines
What type of degraded performance? Any
references?
For example I were unable to squeeze more than few hundred IOPS on rather high-end hardware from old-snapshots (but it was on RHEL6 kernel missing few recent performance optimizations.)
And the more snapshots of the origin LV you have the write performance is divided by the number of them as the original value must be written to each of them. (And there is no optimization for that. So remember to remove the snapshots early.)
HTH
Marian
Quoted 60 lines
Note that the snapshots I'm using are ephemeral and are to exist only long enough for the actual database backup operation to complete.
Quoted 4 lines
Also once the space allocated for snapshot gets full, the snapshot becomes invalid, so hurry up!
-- Martian
Thanks!
--Phil
Quoted 46 lines
Quoted 46 lines
the appropriate place to ask this. This is my first time setting up LVM snapshots for mysql-zrm backups.
CentOS-6.6
MySQL-5.6.22
MySQL-zrm-3.0 (EPEL).Short version:
I am unable to tell if the MySQL-zrm LVM snapshots are indeed taking place, or if the backup is being taken directly from the main database LVs. The log file info isn't giving me enough info to tell one way or another.
Some more info:
I am setting up a database server with all of the database files on LVM LVs and am using mysql-zrm for logical backups using LVM snapshots to reduce the database lock times.
I have followed all of the various setup instructions very closely, and am using the default "lvm-snapshot.pl" plugin. However, the only log file lines mentioning snapshot are the following:
mysql:backup:INFO: Running snapshot verification using command /usr/share/mysql-zrm/plugins/lvm-snapshot.pl --action verify-config 2>/home/tmp/xH8KKpsF2OThere are no other log entries following this one that seem to have anything to do with snapshots. There are no errors. The backups succeed. Both "--verbose" and "--noquiet" are enabled.
Here are the salient entries from the mysql-zrm.conf file:
backup-level=0
backup-mode=logical
lvm-snapshot=20G snapshot-plugin="/usr/share/mysql-zrm/plugins/lvm-snapshot.pl" backup-type=regular
destination=/home/mysql-zrm
retention-policy=4W
compress=1Any info or help would be much appreciated!
Thanks,
--Phil