Lvm Strange Problem
Hi all,
i have some trouble at work caused by strange thing happend at our lvm volumes. I hope that some of you have any idea, because i'am out of any good one.
We use an EMC Storage system with iscsi volume on different xen hosts. These gentoo hardened xen hosts have an lv on that iscsi target. Now there are many machines running for more than 400 days. Some days ago the datacenter moved over to a new building and all the machines were turned off.
We powered them up, and about 50% of the more than 30 servers on the iscsi were timestamped 23. feb 2010.
For now it seems that most of the freezed systems were snapshoted last year.
So, any hint would be useful
Thx
Joerg
Quoted 4 lines · J�rg Stephan
We powered them up, and about 50% of the more than 30 servers on the iscsi were timestamped 23. feb 2010.
For now it seems that most of the freezed systems were snapshoted last year.
It seems pretty clear that all your servers have both a snapshot and a live LV. At power up, they search for their disk and randomly find either the live or the snapshot first. You need to take the snapshots out of the search path.
Signature · 4 lines
Stuart D. Gathman <[email protected]> Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial.
Am 11.10.2010 14:40, schrieb Stuart D. Gathman:
Quoted 6 lines · J�rg Stephan
Quoted 5 lines
We powered them up, and about 50% of the more than 30 servers on the iscsi were timestamped 23. feb 2010.
Quoted 8 lines
Quoted 2 lines
For now it seems that most of the freezed systems were snapshoted last year.
It seems pretty clear that all your servers have both a snapshot and a live LV. At power up, they search for their disk and randomly find either the live or the snapshot first. You need to take the snapshots out of the search path.
Hi again,
well, mostly the snapshot was removed some days after they were made. Also the snapshots had of course different names, how could it be that they were used? And even if this, why are all machines with the failure bound to the same date?
Greets
J�rg
Quoted 11 lines · Stuart D. Gathman
Quoted 4 lines · J�rg Stephan
We powered them up, and about 50% of the more than 30 servers on the iscsi were timestamped 23. feb 2010.
For now it seems that most of the freezed systems were snapshoted last year.
It seems pretty clear that all your servers have both a snapshot and a live LV. At power up, they search for their disk and randomly find either the live or the snapshot first. You need to take the snapshots out of the search path.
Alternatively, you might change the UUID of the contained VG or filesystem when taking a snapshot.
Signature · 4 lines
Stuart D. Gathman <[email protected]> Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial.
Quoted 9 lines · J�rg Stephan
Quoted 5 lines
It seems pretty clear that all your servers have both a snapshot and a live LV. At power up, they search for their disk and randomly find either the live or the snapshot first. You need to take the snapshots out of the search path.
well, mostly the snapshot was removed some days after they were made. Also the snapshots had of course different names, how could it be that they were used? And even if this, why are all machines with the failure bound to the same date?
I believe your problem is with your SAN, not LVM (you did mention using iSCSI). I'm talking about SAN snapshots, not LVM snapshots (although the SAN server could very well use LVM underneath). The SAN volumes have no names, just LUNs. At boot, they are searched for a matching UUID, just like directly attached physical volumes. When taking a SAN snapshot (clone), the UUID of the clone is the same as the original, whether a filesystem or VG is on the SAN LUN.
Signature · 4 lines
Stuart D. Gathman <[email protected]> Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial.
Quoted 10 lines
I believe your problem is with your SAN, not LVM (you did mention using iSCSI). I'm talking about SAN snapshots, not LVM snapshots (although the SAN server could very well use LVM underneath). The SAN volumes have no names, just LUNs. At boot, they are searched for a matching UUID, just like directly attached physical volumes. When taking a SAN snapshot (clone), the UUID of the clone is the same as the original, whether a filesystem or VG is on the SAN LUN.
Okay, but we dont use SAN snapshots, we use an lvcreate --snapshot on the client. So the storage just dont know about them.
What happens is, we lv snapshot a volume, mount it copy the data to the new volume we create after snapshooting the source, remove the old snapshot and turning of the new machine. It rans about half an year, gets stoped, and after it turning on again the lvm data on the iscsi volume is last cahnged on the 23 februar 2010, even if the snapshot was made much earlier.
Greets
Quoted 11 lines · J�rg Stephan
Quoted 7 lines
I believe your problem is with your SAN, not LVM (you did mention using iSCSI). I'm talking about SAN snapshots, not LVM snapshots (although the SAN server could very well use LVM underneath). The SAN volumes have no names, just LUNs. At boot, they are searched for a matching UUID, just like directly attached physical volumes. When taking a SAN snapshot (clone), the UUID of the clone is the same as the original, whether a filesystem or VG is on the SAN LUN.
Okay, but we dont use SAN snapshots, we use an lvcreate --snapshot on the client. So the storage just dont know about them.
Quoted 6 lines
What happens is, we lv snapshot a volume, mount it copy the data to the new volume we create after snapshooting the source, remove the old snapshot and turning of the new machine. It rans about half an year, gets stoped, and after it turning on again the lvm data on the iscsi volume is last cahnged on the 23 februar 2010, even if the snapshot was made much earlier.
If the contents of (50% of) your SAN disks have reverted, then your issue is with SAN (iSCSI is a SAN protocol), not with LVM.
Signature · 4 lines
Stuart D. Gathman <[email protected]> Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial.