LVM snapshot limits
I'm trying to create 45 LVM snapshots on one of my dev machines. I have about 7TB of disk to work with, but I keep having wierd issues. I can't seem to make more than 30 snapshots. At about 15-20 snapshots they stop being created reliably and as I get to 30 snapshots I stop getting snapshots and start getting COW table sized logical volumes that are in no way snapshots. Some of these snapshots cause kernel panics on RHEL 5 when I try to remove them.
If anyone wants to take a look, I have included a script that reproduces the issue. It works on both RHEL 5 and Ubuntu 7.04. The script only requires a few gigabytes of storage, so I don't think the problem has to do with my large volume of storage.
I filed a support ticket with redhat, but I figured I'd float it here to see if anyone has any suggestions/experience.
Shaun Kruger
Quoted 7 lines · Shaun Kruger
I'm trying to create 45 LVM snapshots on one of my dev machines. I have about 7TB of disk to work with, but I keep having wierd issues. I can't seem to make more than 30 snapshots. At about 15-20 snapshots they stop being created reliably and as I get to 30 snapshots I stop getting snapshots and start getting COW table sized logical volumes that are in no way snapshots. Some of these snapshots cause kernel panics on RHEL 5 when I try to remove them.
Hi,
both problems are already known and will be fixed in RHEL 5.1 release. (See Red Hat Bugzilla bugs 211516 and 211525).
(Upstream kernel is already fixed.)
Anyway, multiple snapshots from the same origin can cause considerable performance decrease (there is no shared area for snapshots in current implementation, so write to origin will cause new writes to every snapshot copy-on-write area).
Milan
--
[email protected]
Thanks for the information. I'll post that to my redhat support ticket and see if they can help me with a more immediate fix.
Quoted 4 lines
Anyway, multiple snapshots from the same origin can cause considerable performance decrease (there is no shared area for snapshots in current implementation, so write to origin will cause new writes to every snapshot copy-on-write area).
This copy on write issue had already occoured to us. We were going to roll out with the concept of a persistent or nonpersistent database instance. The nonpersistent ones were going to be removed before refreshing the data on the main database source. There is another script that recreates all the developer instances after the update is complete.
Shaun
Quoted 4 lines
Hi,
both problems are already known and will be fixed in RHEL 5.1 release. (See Red Hat Bugzilla bugs 211516 and 211525).
(Upstream kernel is already fixed.)
Thanks again for your help on this issue. Between your response and redhat support I was able to quickly get a working kernel.
There is however one issue that probably needs attention. If I create snapshots until I get an error (about 200 snaps), remove them, and repeat the creation/removal multiple times it is possible to destroy the LVM metadata. When this happens pvscan shows that there is no VG associated with the physical volume. Restarting results in full loss of data. I was able to reproduce this twice.
While it is not impacting my use of LVM snapshots this is probably a bug that should be looked into. It was quite worrying when my volume group disappeared.
I can provide my scripts and detailed instructions on reproducing the issue for anyone who is interested in looking into it.
Shaun Kruger
Quoted 7 lines · Shaun Kruger
There is however one issue that probably needs attention. If I create snapshots until I get an error (about 200 snaps), remove them, and repeat the creation/removal multiple times it is possible to destroy the LVM metadata. When this happens pvscan shows that there is no VG associated with the physical volume. Restarting results in full loss of data. I was able to reproduce this twice.
Quoted 6 lines
While it is not impacting my use of LVM snapshots this is probably a bug that should be looked into. It was quite worrying when my volume group disappeared.
I can provide my scripts and detailed instructions on reproducing the issue for anyone who is interested in looking into it.
Please create bug on bugzilla.redhat.com (lvm2 component) (or report this through Red Hat support)
and attach your scripts there with description of problem, thanks.
Is the problem caused by overfilled metadata area on disk ? (Will increasing metadata areas by "pvcreate --metadatasize ... " help ?)
Milan
--
[email protected]
Quoted 4 lines · Milan Broz
Quoted 4 lines
There is however one issue that probably needs attention. If I create snapshots until I get an error (about 200 snaps), remove them, and repeat the creation/removal multiple times it is possible to destroy the LVM metadata. When this happens pvscan shows that there is no VG
Cool! Great torture test! BwaHaHaHa...
Signature · 4 lines
Stuart D. Gathman <[email protected]> Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial.