vgchange -a y /dev/System fails
It fails and says:
parameter error setting up snapshot copy on write exception
So yes, I have snapshots in there, but I can't delete the snapshots if the volume isn't active, and I can't activate the volume because of a snapshot.
To make things worse, /dev/System contains /usr and /var.
The snapshots were created with 1.0.1rc4 on 2.4.14 and I now run 2.4.17 with 1.0.1rc4-ish
What's the magic override I need to use to make the volume active again?
Thanks
Marc
Signature · 4 lines
Microsoft is to operating systems & security ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | Finger [email protected] for PGP keyMarc,
I am afraid you need to figure out the last metadata backup in /etc/lvmconf/ which *doesn't* contain the flaky snapshot(s).
You didn't mention the name of you VG so let's say it was 'A'.
You can do that with "vgcfgrestore -f /etc/lvmconf/A.conf.1.old -n A -ll". Replace the '1' in the metadata backup filename above with 2, 3 and so on (you should have a default of up to 9).
After you found one (say #3), remember all you PVs belonging to that VG and run
pvcreate -yff /dev/AllYourPVsBelongingToA
for pv in /dev/AllYourPVsBelongingToA
do
vgcfgrestore -f /etc/lvmconf/A.conf.3.old -n A $pv
done
vgscan
vgchange -ay A
That should be it (crossing fingers ;-)
Quoted 26 lines · Marc MERLIN
It fails and says:
parameter error setting up snapshot copy on write exceptionSo yes, I have snapshots in there, but I can't delete the snapshots if the volume isn't active, and I can't activate the volume because of a snapshot.
To make things worse,
/dev/Systemcontains /usr and /var.The snapshots were created with 1.0.1rc4 on 2.4.14 and I now run 2.4.17 with 1.0.1rc4-ish
What's the magic override I need to use to make the volume active again?
Thanks
MarcSignature · 10 lines
Microsoft is to operating systems & security .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | Finger [email protected] for PGP key _______________________________________________ linux-lvm mailing list [email protected] http://lists.sistina.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
Signature · 15 lines
Regards,
Heinz -- The LVM Guy --
*** Software bugs are stupid.
Nevertheless it needs not so stupid people to solve them ***
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Heinz Mauelshagen Sistina Software Inc.
Senior Consultant/Developer Am Sonnenhang 11
56242 Marienrachdorf
Germany
[email protected] +49 2626 141200
FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-[Adding a Cc to lvm-devel, since I apparently hit a bug in the code. Please remove or edit as appropriate]
Quoted 21 lines · Heinz J . Mauelshagen
Quoted 10 lines
It fails and says:
parameter error setting up snapshot copy on write exceptionSo yes, I have snapshots in there, but I can't delete the snapshots if the volume isn't active, and I can't activate the volume because of a snapshot.
To make things worse,
/dev/Systemcontains /usr and /var.The snapshots were created with 1.0.1rc4 on 2.4.14 and I now run 2.4.17 with 1.0.1rc4-ish
I am afraid you need to figure out the last metadata backup in
/etc/lvmconf/which *doesn't* contain the flaky snapshot(s).You didn't mention the name of you VG so let's say it was 'A'.
You can do that with "vgcfgrestore -f
/etc/lvmconf/A.conf.1.old-n A -ll". Replace the '1' in the metadata backup filename above with 2, 3 and so on (you should have a default of up to 9).
After you found one (say #3), remember all you PVs belonging to that VG and run
Quoted 7 lines
pvcreate -yff
/dev/AllYourPVsBelongingToA
for pv in/dev/AllYourPVsBelongingToA
do
vgcfgrestore -f/etc/lvmconf/A.conf.3.old-n A $pv
done
vgscan
vgchange -ay A
Unfortunately, that doesn't work because I had a cron job that was creating daily snapshots (in an attempt to emulate ONTAP's snapshot's feature), so my 10 backups contain snapshot LVs
If the backups were text, I could edit them and remove the snapshot info in there, but with a binary backup, I don't know how to do that.
If I know how big my /var and /usr partitions were and in which order I created them (vgcfgrestore -ll shows this anyway), can I recreate the VG on top of itself without losing the data in the the LVs?
Is there another way to recover from this?
More importantly, is this a known bug? Can I give info to help fix it?
Marc
Signature · 4 lines
Microsoft is to operating systems & security ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | Finger [email protected] for PGP keyQuoted 33 lines · Marc MERLIN
[Adding a Cc to lvm-devel, since I apparently hit a bug in the code. Please remove or edit as appropriate]
Quoted 21 lines · Heinz J . Mauelshagen
Quoted 10 lines
It fails and says:
parameter error setting up snapshot copy on write exceptionSo yes, I have snapshots in there, but I can't delete the snapshots if the volume isn't active, and I can't activate the volume because of a snapshot.
To make things worse,
/dev/Systemcontains /usr and /var.The snapshots were created with 1.0.1rc4 on 2.4.14 and I now run 2.4.17 with 1.0.1rc4-ish
I am afraid you need to figure out the last metadata backup in
/etc/lvmconf/which *doesn't* contain the flaky snapshot(s).You didn't mention the name of you VG so let's say it was 'A'.
You can do that with "vgcfgrestore -f
/etc/lvmconf/A.conf.1.old-n A -ll". Replace the '1' in the metadata backup filename above with 2, 3 and so on (you should have a default of up to 9).
After you found one (say #3), remember all you PVs belonging to that VG and runQuoted 7 lines
pvcreate -yff
/dev/AllYourPVsBelongingToA
for pv in/dev/AllYourPVsBelongingToA
do
vgcfgrestore -f/etc/lvmconf/A.conf.3.old-n A $pv
done
vgscan
vgchange -ay A
Quoted 3 lines
Unfortunately, that doesn't work because I had a cron job that was creating daily snapshots (in an attempt to emulate ONTAP's snapshot's feature), so my 10 backups contain snapshot LVs
Well, in this case you might still have a consistent backup with snapshots which you can use like mentioned above.
Quoted 10 lines
If the backups were text, I could edit them and remove the snapshot info in there, but with a binary backup, I don't know how to do that.
If I know how big my /var and /usr partitions were and in which order I created them (vgcfgrestore -ll shows this anyway), can I recreate the VG on top of itself without losing the data in the the LVs?
Is there another way to recover from this?More importantly, is this a known bug? Can I give info to help fix it?
We get ocassional problem reports like yours.
Any syslog or console output which is related to a snapshot flaw would help.
Quoted 13 lines
Marc
Signature · 10 lines
Microsoft is to operating systems & security .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | Finger [email protected] for PGP key _______________________________________________ linux-lvm mailing list [email protected] http://lists.sistina.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
Signature · 15 lines
Regards,
Heinz -- The LVM Guy --
*** Software bugs are stupid.
Nevertheless it needs not so stupid people to solve them ***
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Heinz Mauelshagen Sistina Software Inc.
Senior Consultant/Developer Am Sonnenhang 11
56242 Marienrachdorf
Germany
[email protected] +49 2626 141200
FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-Quoted 6 lines · Heinz J . Mauelshagen
Quoted 3 lines
Unfortunately, that doesn't work because I had a cron job that was creating daily snapshots (in an attempt to emulate ONTAP's snapshot's feature), so my 10 backups contain snapshot LVs
Well, in this case you might still have a consistent backup with snapshots which you can use like mentioned above.
I tried a few and none seemed to work, but I restored one from before I made snapshots and recovered from this.
I now have my system back (/usr, /var)
As for /home, as soon as I tried to access it (e2fsck, tune2fs -O ^has_journal, or whatever), the process hung in "wait_o" according to ps.
I tried to restore the VG to its state before I added snapshots, but that only made it go away :-(
mechastreisand:/etc/lvmconf# vgchange -a n /dev/BigAssLVMRaid vgchange -- volume group "BigAssLVMRaid" successfully deactivated
mechastreisand:/etc/lvmconf# vgcfgrestore -f /etc/lvmconf/GoodBigAssLVMRaid.conf -n BigAssLVMRaid -ll --- Volume group --- VG Name BigAssLVMRaid VG Access read/write VG Status NOT available/resizable VG # 1 MAX LV 255 Cur LV 1 Open LV 0 MAX LV Size 1.00 TB Max PV 255 Cur PV 9 Act PV 9 VG Size 868.69 GB PE Size 64.00 MB Total PE 13899 Alloc PE / Size 13120 / 820.00 GB Free PE / Size 779 / 48.69 GB VG UUID VgCZzh-ldUN-Ix1n-mZL3-EWsj-ATke-7Y4RQj
--- Logical volume ---
LV Name /dev/BigAssLVMRaid/BigAssLVMPartition VG Name BigAssLVMRaid LV Write Access read/write LV Status NOT available LV # 1 # open 0 LV Size 820.00 GB Current LE 13120 Allocated LE 13120 Allocation next free Read ahead sectors 32 Block device 58:0
--- Physical volume ---
PV Name /dev/rd/disc0/part7 VG Name BigAssLVMRaid PV Size 72.31 GB / NOT usable 121.73 MB [LVM: 128.00 KB] PV# 1 PV Status NOT available Allocatable yes (but full) Cur LV 1 PE Size (KByte) 65536 Total PE 1155 Free PE 0 Allocated PE 1155 PV UUID Eriv40-q8V1-jgaS-ZXKh-8xwm-Wq7V-H9tVrl (...) mechastreisand:/etc/lvmconf# vgcfgrestore -f /etc/lvmconf/GoodBigAssLVMRaid.conf -n BigAssLVMRaid -ll | grep PV | grep rd | sed "s/.*\/dev/\/dev/" /dev/rd/disc0/part7 /dev/rd/disc1/part1 /dev/rd/disc2/part1 /dev/rd/c1d0p1 /dev/rd/c1d1p1 /dev/rd/c2d0p1 /dev/rd/c2d1p1 /dev/rd/c3d0p1 /dev/rd/c3d1p1 mechastreisand:/etc/lvmconf# vgcfgrestore -f /etc/lvmconf/GoodBigAssLVMRaid.conf -n BigAssLVMRaid -ll | grep PV | grep rd | sed "s/.*\/dev/\/dev/" > /tmp/partlist mechastreisand:/etc/lvmconf# pvcreate -yff `cat /tmp/partlist ` pvcreate -- WARNING: forcing physical volume creation on "/dev/rd/disc0/part7" of volume group "BigAssLVMRaid" pvcreate -- removing lvmtab entry pvcreate -- physical volume "/dev/rd/disc0/part7" successfully created pvcreate -- WARNING: forcing physical volume creation on "/dev/rd/disc1/part1" of volume group "BigAssLVMRaid" pvcreate -- physical volume "/dev/rd/disc1/part1" successfully created pvcreate -- WARNING: forcing physical volume creation on "/dev/rd/disc2/part1" of volume group "BigAssLVMRaid" pvcreate -- physical volume "/dev/rd/disc2/part1" successfully created pvcreate -- WARNING: forcing physical volume creation on "/dev/rd/c1d0p1" of volume group "BigAssLVMRaid" pvcreate -- physical volume "/dev/rd/c1d0p1" successfully created pvcreate -- WARNING: forcing physical volume creation on "/dev/rd/c1d1p1" of volume group "BigAssLVMRaid" pvcreate -- physical volume "/dev/rd/c1d1p1" successfully created pvcreate -- WARNING: forcing physical volume creation on "/dev/rd/c2d0p1" of volume group "BigAssLVMRaid" pvcreate -- physical volume "/dev/rd/c2d0p1" successfully created pvcreate -- WARNING: forcing physical volume creation on "/dev/rd/c2d1p1" of volume group "BigAssLVMRaid" pvcreate -- physical volume "/dev/rd/c2d1p1" successfully created pvcreate -- WARNING: forcing physical volume creation on "/dev/rd/c3d0p1" of volume group "BigAssLVMRaid" pvcreate -- physical volume "/dev/rd/c3d0p1" successfully created pvcreate -- WARNING: forcing physical volume creation on "/dev/rd/c3d1p1" of volume group "BigAssLVMRaid" pvcreate -- physical volume "/dev/rd/c3d1p1" successfully created
mechastreisand:/etc/lvmconf# for pv in `cat /tmp/partlist `
Quoted 3 lines
do
vgcfgrestore -f/etc/lvmconf/GoodBigAssLVMRaid.conf-n BigAssLVMRaid $pv done
vgcfgrestore -- VGDA for "BigAssLVMRaid" successfully restored to physical volume "/dev/rd/disc0/part7" vgcfgrestore -- you may not have an actual backup of restored volume group "BigAssLVMRaid"
vgcfgrestore -- VGDA for "BigAssLVMRaid" successfully restored to physical volume "/dev/rd/disc1/part1" vgcfgrestore -- you may not have an actual backup of restored volume group "BigAssLVMRaid"
vgcfgrestore -- VGDA for "BigAssLVMRaid" successfully restored to physical volume "/dev/rd/disc2/part1" vgcfgrestore -- you may not have an actual backup of restored volume group "BigAssLVMRaid"
vgcfgrestore -- VGDA for "BigAssLVMRaid" successfully restored to physical volume "/dev/rd/c1d0p1" vgcfgrestore -- you may not have an actual backup of restored volume group "BigAssLVMRaid"
vgcfgrestore -- VGDA for "BigAssLVMRaid" successfully restored to physical volume "/dev/rd/c1d1p1" vgcfgrestore -- you may not have an actual backup of restored volume group "BigAssLVMRaid"
vgcfgrestore -- VGDA for "BigAssLVMRaid" successfully restored to physical volume "/dev/rd/c2d0p1" vgcfgrestore -- you may not have an actual backup of restored volume group "BigAssLVMRaid"
vgcfgrestore -- VGDA for "BigAssLVMRaid" successfully restored to physical volume "/dev/rd/c2d1p1" vgcfgrestore -- you may not have an actual backup of restored volume group "BigAssLVMRaid"
vgcfgrestore -- VGDA for "BigAssLVMRaid" successfully restored to physical volume "/dev/rd/c3d0p1" vgcfgrestore -- you may not have an actual backup of restored volume group "BigAssLVMRaid"
vgcfgrestore -- VGDA for "BigAssLVMRaid" successfully restored to physical volume "/dev/rd/c3d1p1" vgcfgrestore -- you may not have an actual backup of restored volume group "BigAssLVMRaid"
mechastreisand:/etc/lvmconf# vgscan vgscan -- reading all physical volumes (this may take a while...) vgscan -- found active volume group "System" vgscan -- "/etc/lvmtab" and "/etc/lvmtab.d" successfully created vgscan -- WARNING: This program does not do a VGDA backup of your volume group
-> My BigAssLVMRaid VG is gone :-(
Did I just lose my data?
Marc
Signature · 4 lines
Microsoft is to operating systems & security ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | Finger [email protected] for PGP keyQuoted 19 lines · Marc MERLIN
Quoted 6 lines · Heinz J . Mauelshagen
Quoted 3 lines
Unfortunately, that doesn't work because I had a cron job that was creating daily snapshots (in an attempt to emulate ONTAP's snapshot's feature), so my 10 backups contain snapshot LVs
Well, in this case you might still have a consistent backup with snapshots which you can use like mentioned above.
I tried a few and none seemed to work, but I restored one from before I made snapshots and recovered from this.
I now have my system back (/usr, /var)As for /home, as soon as I tried to access it (e2fsck, tune2fs -O ^has_journal, or whatever), the process hung in "
wait_o" according to ps.I tried to restore the VG to its state before I added snapshots, but that only made it go away :-(
mechastreisand:/etc/lvmconf# vgchange -a n /dev/BigAssLVMRaid
<SNIP>
Quoted 3 lines
-> My BigAssLVMRaid VG is gone :-(
Did I just lose my data?
If you really tried all your metadata backups unsuccessfully, I am afraid so :-( Some users reported snapshot problems with the recent 2.4.x kernels. They are not 100% reliable due to VM issues being sorted out by the kernel community hopefully which will hopefully be integrated in 2.4.18.
Quoted 13 lines
Marc
Signature · 10 lines
Microsoft is to operating systems & security .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | Finger [email protected] for PGP key _______________________________________________ linux-lvm mailing list [email protected] http://lists.sistina.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
Signature · 15 lines
Regards,
Heinz -- The LVM Guy --
*** Software bugs are stupid.
Nevertheless it needs not so stupid people to solve them ***
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Heinz Mauelshagen Sistina Software Inc.
Senior Consultant/Developer Am Sonnenhang 11
56242 Marienrachdorf
Germany
[email protected] +49 2626 141200
FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-Quoted 6 lines · Heinz J . Mauelshagen
Quoted 3 lines
-> My BigAssLVMRaid VG is gone :-(
Did I just lose my data?
If you really tried all your metadata backups unsuccessfully, I am afraid so :-(
I tried other ones, but every time, I'm getting:
vgcfgrestore -- VGDA for "BigAssLVMRaid" successfully restored to physical volume "/dev/rd/disc0/part7"
vgcfgrestore -- you may not have an actual backup of restored volume group "BigAssLVMRaid"
Am I supposed to give a special flag to vgcfgrestore, or maybe pvcreate? (I did not use the default physicalextentsize)
As a reminder, I am currently running this command:
FILE=/etc/lvmconf/BigAssLVMRaid.conf.2.old; vgcfgrestore -f $FILE -n BigAssLVMRaid -ll; read a; pvcreate -yff cat /tmp/partlist; for pv in cat /tmp/partlist ; do vgcfgrestore -f $FILE -n BigAssLVMRaid $pv; done; vgscan
Short of all this, can I re-create the VG the same way I did the first time, and hope to get my filesytem again if I recreate the same LVs? (like I can do with software raid)
This VG was working fine before I restored the other one (active and all)
Quoted 3 lines
Some users reported snapshot problems with the recent 2.4.x kernels. They are not 100% reliable due to VM issues being sorted out by the kernel community hopefully which will hopefully be integrated in 2.4.18.
Ok. I guess I'll know to steer clear from snapshots in the meantime, but if LVM doesn't even allow me to recover a VG that hasn't been overwriten with random data, I don't know how much data I'm willing to trust to it...
Either way, thanks for your help.
Marc
Signature · 4 lines
Microsoft is to operating systems & security ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | Finger [email protected] for PGP keyOn Fri, Feb 01, 2002 at 01:09:28PM -0800, Marc MERLIN wrote:
I wanted to thank Heinz publically for going beyond the call of duty, and helping until the problem was fixed.
I thought I'd give a few lines to explain the resolution for the benefit of other list members and the archives.
Quoted 5 lines · Heinz J . Mauelshagen
Quoted 4 lines
Quoted 3 lines
-> My BigAssLVMRaid VG is gone :-(
Did I just lose my data?
Quoted 8 lines
Quoted 2 lines
If you really tried all your metadata backups unsuccessfully, I am afraid so :-(
I tried other ones, but every time, I'm getting:
vgcfgrestore -- VGDA for "BigAssLVMRaid" successfully restored to physical volume "/dev/rd/disc0/part7"
vgcfgrestore -- you may not have an actual backup of restored volume group "BigAssLVMRaid"
The reason why vgchange -a -y didn't pick up my VG is due to a small bug in vgcfgrestore, which only restored the UUIDs of 8 out of my 9 PVs (quoted from memory, sorry if I got this slightly wrong).
Either way, the fix it to upgrade to a newer version of the lvm tools (from CVS or the next snapshot)
Quoted 1 line
This VG was working fine before I restored the other one (active and all)
The reason why I tried to restore it in the first place is that any kind of partition access on that VG would hang the machine.
The problem turned out to be the crappy VM in 2.4.17, and not LVM.
Actually 2.4.14 also had a few issues fscking the 800G partition, and froze when fsck was going to write the changes to the partition (and its 4 snapshots)
It probably would have completed had I deleted the snapshots, but I ended up going back to 2.4.7, and after an 8H fsck, the partition was mountable again.
Again, thanks to Heinz for his invaluable help.
Marc
Signature · 4 lines
Microsoft is to operating systems & security ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | Finger [email protected] for PGP keyA little while ago I had the same problems - the only way i could get around it was using 2.4.16-xfs. I was using 2.4.17-xfs+lvm (1.0.1) and when a snapshot failed I couldn't remove it and when I rebooted the system would hang. By having a 2.4.16-xfs kernel with lvm-1.0.1rc4(ish) it would fix the problems on the fly during bootup and all was happy. YMMV At the time I did ask the mailling list why 2.4.16-xfs was more robust than 2.4.17-xfs+lvm; however, I never recieved an answer.
Heady
Quoted 16 lines · you
It fails and says:
parameter error setting up snapshot copy on write exceptionSo yes, I have snapshots in there, but I can't delete the snapshots if the volume isn't active, and I can't activate the volume because of a snapshot.
To make things worse,
/dev/Systemcontains /usr and /var.The snapshots were created with 1.0.1rc4 on 2.4.14 and I now run 2.4.17 with 1.0.1rc4-ish
What's the magic override I need to use to make the volume active again?
Thanks
Marc
Signature · 3 lines
Adrian Head (Public Key available on request.)
Quoted 7 lines · Adrian Head
A little while ago I had the same problems - the only way i could get around it was using 2.4.16-xfs. I was using 2.4.17-xfs+lvm (1.0.1) and when a snapshot failed I couldn't remove it and when I rebooted the system would hang. By having a 2.4.16-xfs kernel with lvm-1.0.1rc4(ish) it would fix the problems on the fly during bootup and all was happy. YMMV At the time I did ask the mailling list why 2.4.16-xfs was more robust than 2.4.17-xfs+lvm; however, I never recieved an answer.
Well, as linux-kernel reports there's still problems with 2.4 up to .17 in the vm subsystem which show up on heavier load or on long running systems. Andrea Arcangeli has some fixes but I don't know if they already have been ported to 2.4.18-pre jet.
Quoted 31 lines
Heady
Quoted 16 lines · you
It fails and says:
parameter error setting up snapshot copy on write exceptionSo yes, I have snapshots in there, but I can't delete the snapshots if the volume isn't active, and I can't activate the volume because of a snapshot.
To make things worse,
/dev/Systemcontains /usr and /var.The snapshots were created with 1.0.1rc4 on 2.4.14 and I now run 2.4.17 with 1.0.1rc4-ish
What's the magic override I need to use to make the volume active again?
Thanks
MarcSignature · 9 lines
Adrian Head (Public Key available on request.) _______________________________________________ linux-lvm mailing list [email protected] http://lists.sistina.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
Signature · 15 lines
Regards,
Heinz -- The LVM Guy --
*** Software bugs are stupid.
Nevertheless it needs not so stupid people to solve them ***
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Heinz Mauelshagen Sistina Software Inc.
Senior Consultant/Developer Am Sonnenhang 11
56242 Marienrachdorf
Germany
[email protected] +49 2626 141200
FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-