Skip to contentSkip to conversation

Corrupted filesystem, possible after livemigration with iSCSI storagebackend.

DISCUSSION Espen Berg 3 participants 3 messages Mar 15, 2010 KVM

lore
Share & tools
lore

Espen Berg

author
#0 lore raw

In our KVM system we have two iSCSI backends (master/slave configuration) with failover and two KVM hosts supporting live migration.

The iSCSI volumes are shared by the host as a block device in KVM, and the volumes are available on both frontends. After a reboot one of the KVMs where not able to start again due to file system corruption. We use XFS and have problems to understand what caused the corruption.

We have ruled out the iSCSI backend as both the master and slave data where consistent at the time.

Anyone else had similar problems? What is the recommended way to share an iSCSI drive among the two host machines?

Should XFS be ok as a file system for live migration? I'm not able to find any documentation stating that a clustered file system (GFS2 etc.) is recommended. Are there any concurrent writes on the two host machines during a livemigtation?

<disk type='block' device='disk'>

      <driver name='qemu'/>
      <source dev='/dev/disk/by-path/ip-<ip>:3260-iscsi-test2-lun-0'/>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' unit='0'/>

</disk>

#virsh version
Compiled against library: libvir 0.7.6
Using library: libvir 0.7.6
Using API: QEMU 0.7.6
Running hypervisor: QEMU 0.11.0

#uname -a
Linux vm01 2.6.32-bpo.2-amd64 #1 SMP Fri Feb 12 16:50:27 UTC 2010 x86_64 GNU/Linux

Regards
Espen

#1 lore raw
Quoted 19 lines · Espen Berg

In our KVM system we have two iSCSI backends (master/slave configuration) with failover and two KVM hosts supporting live migration.

The iSCSI volumes are shared by the host as a block device in KVM, and the volumes are available on both frontends. After a reboot one of the KVMs where not able to start again due to file system corruption. We use XFS and have problems to understand what caused the corruption.

We have ruled out the iSCSI backend as both the master and slave data where consistent at the time.

Anyone else had similar problems? What is the recommended way to share an iSCSI drive among the two host machines?

Should XFS be ok as a file system for live migration? I'm not able to find any documentation stating that a clustered file system (GFS2 etc.) is recommended. Are there any concurrent writes on the two host machines during a livemigtation?

Quoted 6 lines

<disk type='block' device='disk'>
<driver name='qemu'/>
<source dev='/dev/disk/by-path/ip-<ip>:3260-iscsi-test2-lun-0'/> <target dev='sda' bus='scsi'/>
<address type='drive' controller='0' bus='0' unit='0'/> </disk>

You need to use cache=off if you've got one iscsi drive mounted on two separate physical machines.

The additional layer of caching will result in inconsistency because iSCSI doesn't have a mechanism to provide cache coherence between two nodes.

Regards,

Anthony Liguori

Quoted 19 lines

#virsh version
Compiled against library: libvir 0.7.6
Using library: libvir 0.7.6
Using API: QEMU 0.7.6
Running hypervisor: QEMU 0.11.0

#uname -a
Linux vm01 2.6.32-bpo.2-amd64 #1 SMP Fri Feb 12 16:50:27 UTC 2010 x86_64 GNU/Linux

Regards
Espen

Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#2 lore raw
Quoted 27 lines · Anthony Liguori
Quoted 19 lines · Espen Berg

In our KVM system we have two iSCSI backends (master/slave configuration) with failover and two KVM hosts supporting live migration.

The iSCSI volumes are shared by the host as a block device in KVM, and the volumes are available on both frontends. After a reboot one of the KVMs where not able to start again due to file system corruption. We use XFS and have problems to understand what caused the corruption.

We have ruled out the iSCSI backend as both the master and slave data where consistent at the time.

Anyone else had similar problems? What is the recommended way to share an iSCSI drive among the two host machines?

Should XFS be ok as a file system for live migration? I'm not able to find any documentation stating that a clustered file system (GFS2 etc.) is recommended. Are there any concurrent writes on the two host machines during a livemigtation?

Quoted 6 lines

<disk type='block' device='disk'>
<driver name='qemu'/>
<source dev='/dev/disk/by-path/ip-<ip>:3260-iscsi-test2-lun-0'/> <target dev='sda' bus='scsi'/>
<address type='drive' controller='0' bus='0' unit='0'/> </disk>

Quoted 2 lines

You need to use cache=off if you've got one iscsi drive mounted on two separate physical machines.

FYI, this can be done by changing the disk XML driver

<driver name='qemu'/>

to be

<driver name='qemu' cache='none'/>

Regards,
Daniel

Signature · 4 lines
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|