fix handling of VIR_DOMAIN_EVENT_STARTED

Hi all,

I am using fence-virt.
When using a serial listener, fence_virtd does not receive a request from VM which started after fence_virtd start.

Step 1) Start fence_virtd.

         # fence_virtd -d5 -F
Step 2) And then start VM.
         # virsh start srv-a1
Step 3) A request sent from the VM,
         # fence_virt -D/dev/ttyS1 -oXXX
        but fence_virtd not receive it.

Below, x3650f is a host, srv-XX is a VM.

[root at x3650f ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.1 (Santiago) [root at x3650f ~]#
[root at x3650f ~]# rpm -qa | egrep '^libvirt|^qemu' | sort libvirt-0.8.7-18.el6.x86_64 libvirt-client-0.8.7-18.el6.x86_64 libvirt-devel-0.8.7-18.el6.x86_64 libvirt-java-0.4.7-1.el6.noarch libvirt-java-devel-0.4.7-1.el6.noarch libvirt-python-0.8.7-18.el6.x86_64 libvirt-qpid-0.2.22-6.el6.x86_64 qemu-img-0.12.1.2-2.160.el6.x86_64 qemu-kvm-0.12.1.2-2.160.el6.x86_64
[root at x3650f ~]#

And I used following fence-virt.
http://fence-virt.git.sourceforge.net/git/gitweb.cgi?p=fence-virt/fence-virt;a=commit;h=53a2b293d12991beb6c2927f78586257bff5173a

[root at x3650f ~]# tar xfz fence-virt-53a2b29.tar.gz && cd fence-virt-53a2b29 && ./build [root at x3650f ~]# cd && mkdir plugins && cd plugins
[root at x3650f plugins]# ln -s /root/fence-virt-53a2b29/server/serial.so . [root at x3650f plugins]# ln -s /root/fence-virt-53a2b29/server/libvirt.so . [root at x3650f plugins]# ls -l
lrwxrwxrwx 1 root root 42 Sep 13 10:28 libvirt.so -> /root/fence-virt-53a2b29/server/libvirt.so lrwxrwxrwx 1 root root 41 Sep 13 10:28 serial.so -> /root/fence-virt-53a2b29/server/serial.so [root at x3650f plugins]#

I attached /etc/fence_virt.conf and /etc/libvirt/qemu/srv-a1.xml, please refer to it.

Step 1)
Start fence_virtd.

[root at x3650f ~]# ./fence-virt-53a2b29/server/fence_virtd -d5 -F Background mode disabled
Debugging threshold is now 5
(snip)
Backend plugin: libvirt
Listener plugin: serial
Searching /root/plugins for plugins...
Searching for plugins in /root/plugins
Loading plugin from /root/plugins/serial.so
Failed to map backend_plugin_version
Registered listener plugin serial 0.4
Loading plugin from /root/plugins/libvirt.so
Registered backend plugin libvirt 0.1
2 plugins found
Available backends:

      libvirt 0.1
  Available listeners:
      serial 0.4

Debugging threshold is now 5
Using qemu:///system
Debugging threshold is now 5
Got /var/lib/libvirt/qemu for uri
Got serial for mode
Libvirt event listener starting
* Socket path: /var/lib/libvirt/qemu
* Mode: Serial
myEventAddHandleFunc:128: Add handle 6 1 0x37558aef80 0x7f68cc000920 (nil) myEventAddTimeoutFunc:160: Adding Timeout -1 0x37558a7f90 0x7f68cc000920 event_thread:448 :: Registering domain event cbs myEventUpdateHandleFunc:140: Updated Handle 0 0 myEventUpdateHandleFunc:140: Updated Handle 0 1 myEventUpdateHandleFunc:140: Updated Handle 0 0 myEventUpdateHandleFunc:140: Updated Handle 0 1
<- Step 2) I started VM here.
myEventUpdateHandleFunc:140: Updated Handle 0 0 myEventUpdateHandleFunc:140: Updated Handle 0 1
Registered 621d8c61-1070-7aab-6158-3889d68470ab on 9
<- Step 3) I run fence_virt here.
(snip)

Step 3)
A request sent from the VM, but fence_virtd not receive it.

[root at srv-a1 ~]# ./fence-virt-53a2b29/client/fence_virt -D/dev/ttyS1 -olist Unknown response (255)
[root at srv-a1 ~]#

I attached the patch for this problem.

* Modified to send a SIGHUP to the primary thread from the secondary thread detects the start of the VM, and
the primary thread is modified so as to select() the requests from the VM which started.
* Also, when four VM(s) are started simultaneously, poll() <in event_thread()@server/virt-serial.c> could not handle all the boot.
Therefore, I revised it to call domainStarted()/domainStopped() in myDomainEventCallback1().

Regards,
Kazunori INOUE
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: fix-handling-VIR_DOMAIN_EVENT_STARTED.patch
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20110913/d11987f2/attachment.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: fence_virt.conf
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20110913/d11987f2/attachment.conf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: srv-a1.xml
Type: text/xml
Size: 2530 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20110913/d11987f2/attachment.xml>

That's certainly one way of doing it. I might have used 'pthread_cond_singal'.

You're using pthread_kill, so what I suspect happens is 4 SIGHUPs are coming in very quickly, causing more modification to be required.

I'll look at this a bit more with Marek.

-- Lon

I am sorry for the delay.

I have a fix based on yours that uses a pipe to wake up select() rather than pthread_kill; I'll attach it here. This ends up being a bit lower impact, as well; it doesn't require changing the main program - only the serial/vmchannel plugin.

Apart from that, the patch here is largely the same.

Let me know what you think at your convenience.

-- Lon
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: 0001-Fix-serial-domain-handling.patch
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20111025/d7b5bd33/attachment.ksh>

Hi, Lon

Thank you for your reply.
I tested this patch, and confirmed that the problem was solved. (It was good also when four VM (s) was started simultaneously.)

Many thanks!

Hi,

Could you please merge?

Regards,
Kazunori INOUE

On 09/13/2011 04:44 AM, Kazunori INOUE wrote:

[all late and wrong]

Merged in to 0.3.0

-- Lon