backend plugin for monitoring a host's status

PATCH New Kazunori INOUE 3 participants 6 messages Oct 5, 2011 – Oct 31, 2011

lore

Share & tools

lore

Kazunori INOUE

author

2011-10-05 08:56 UTC #0 lore raw

PATCH 1/1 backend plugin for monitoring a host's status

Hi all,

I think that the communication function of fence-virt is flexible, so I want to use it more effectively.

Therefore I made backend plugin for a guest to get the host's status using the communication facility of fence-virt,
and I changed to allow specifying one more backend (for fencing, and replying the host's status).

I created the backend "pm-monitor" which has met the following configurations / requirements.
- Both hosts and VMs, cluster (Pacemaker) have been configured.

Here's an overview of function. Please refer to attached 'overview.png'. (*) pingd resource notifies the status of connection with a specific
host to pacemaker, and pacemaker manages the result.
(1) resource (vm-client) which requires the host's status is executed.
(2) vm-client requests 'host_status (result of pingd)' to the host with fence_virt.
(3) use the serial listener,
(4) fence_virtd (pm-monitor backend) gets the 'result of pingd' from pacemaker and answers it after conversion.
- the conversion rule is set in /etc/pm-monitor.conf

Here's a description of the attached files.
* add_general_backend.patch
- add the server/pm-fence.c
- change the configure.in and server/Makefile.in
* overview.png
- figure of the overview.
* vm-client
- Resource Agent for VM.
- I'm going to post this RA to the appropriate community
(https://github.com/ClusterLabs/resource-agents).
* fence_virt.conf
- sample configuration.
* pm-monitor.conf
- sample configuration for pm-monitor.so

* host.cli, mon-host.txt
- sample configuration file of a host cluster, and mon-host.txt is an output of the crm_mon command.
* VM.cli, mon-VM.txt
- sample configuration file of a VM cluster, and mon-VM.txt is an output of the crm_mon command.

Best Regards
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: add_general_backend.patch
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20111005/746fa8b5/attachment.ksh>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: overview.PNG
Type: image/png
Size: 23810 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20111005/746fa8b5/attachment.png>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: vm-client
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20111005/746fa8b5/attachment-0001.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: fence_virt.conf
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20111005/746fa8b5/attachment.conf>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: pm-monitor.conf
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20111005/746fa8b5/attachment-0001.conf>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: host.cli
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20111005/746fa8b5/attachment-0002.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: VM.cli
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20111005/746fa8b5/attachment-0003.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: mon-host.txt
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20111005/746fa8b5/attachment.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: mon-VM.txt
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20111005/746fa8b5/attachment-0001.txt>

Andrew Beekhof

2011-10-12 01:40 UTC #1 lore raw

PATCH 1/1 backend plugin for monitoring a host's status

Was making host connectivity information available to the guests the goal here or just something to demonstrate the concept? Interesting idea :-)

2011/10/5 Kazunori INOUE <[email protected]>:

Quoted 49 lines

Hi all,
I think that the communication function of fence-virt is flexible, so I want to use it more effectively.
Therefore I made backend plugin for a guest to get the host's status using the communication facility of fence-virt,
and I changed to allow specifying one more backend (for fencing, and replying the host's status).
I created the backend "pm-monitor" which has met the following configurations / requirements.
- Both hosts and VMs, cluster (Pacemaker) have been configured.
Here's an overview of function. Please refer to attached 'overview.png'. (*) pingd resource notifies the status of connection with a specific ? ?host to pacemaker, and pacemaker manages the result.
(1) resource (vm-client) which requires the host's status is executed.
(2) vm-client requests 'host_status (result of pingd)' to the host ? ?with fence_virt.
(3) use the serial listener,
(4) fence_virtd (pm-monitor backend) gets the 'result of pingd' from ? ?pacemaker and answers it after conversion.
? ?- the conversion rule is set in /etc/pm-monitor.conf
Here's a description of the attached files.
* add_general_backend.patch
?- add the server/pm-fence.c
?- change the configure.in and server/Makefile.in
* overview.png
?- figure of the overview.
* vm-client
?- Resource Agent for VM.
?- I'm going to post this RA to the appropriate community ? ?(https://github.com/ClusterLabs/resource-agents).
* fence_virt.conf
?- sample configuration.
* pm-monitor.conf
?- sample configuration for pm-monitor.so
* host.cli, mon-host.txt
?- sample configuration file of a host cluster, ? ?and mon-host.txt is an output of the crm_mon command.
* VM.cli, mon-VM.txt
?- sample configuration file of a VM cluster,
? ?and mon-VM.txt is an output of the crm_mon command.
Best Regards

Kazunori INOUE

author

2011-10-13 08:52 UTC #2 lore raw

PATCH 1/1 backend plugin for monitoring a host's status

Hi, Andrew

We (NTT) will participate in the HA Mini-Conference, please let we explain at that time.

Best Regards

Quoted 53 lines · Andrew Beekhof

Was making host connectivity information available to the guests the goal here or just something to demonstrate the concept? Interesting idea :-)
2011/10/5 Kazunori INOUE<[email protected]>:
Quoted 48 lines
Hi all,
I think that the communication function of fence-virt is flexible, so I want to use it more effectively.
Therefore I made backend plugin for a guest to get the host's status using the communication facility of fence-virt,
and I changed to allow specifying one more backend (for fencing, and replying the host's status).
I created the backend "pm-monitor" which has met the following configurations / requirements.
- Both hosts and VMs, cluster (Pacemaker) have been configured.
Here's an overview of function. Please refer to attached 'overview.png'. (*) pingd resource notifies the status of connection with a specific
host to pacemaker, and pacemaker manages the result.
(1) resource (vm-client) which requires the host's status is executed.
(2) vm-client requests 'host_status (result of pingd)' to the host with fence_virt.
(3) use the serial listener,
(4) fence_virtd (pm-monitor backend) gets the 'result of pingd' from pacemaker and answers it after conversion.
- the conversion rule is set in /etc/pm-monitor.conf
Here's a description of the attached files.
* add_general_backend.patch
- add the server/pm-fence.c
- change the configure.in and server/Makefile.in
* overview.png
- figure of the overview.
* vm-client
- Resource Agent for VM.
- I'm going to post this RA to the appropriate community
(https://github.com/ClusterLabs/resource-agents).
* fence_virt.conf
- sample configuration.
* pm-monitor.conf
- sample configuration for pm-monitor.so
* host.cli, mon-host.txt
- sample configuration file of a host cluster, and mon-host.txt is an output of the crm_mon command.
* VM.cli, mon-VM.txt
- sample configuration file of a VM cluster, and mon-VM.txt is an output of the crm_mon command.
Best Regards

Andrew Beekhof

2011-10-13 23:18 UTC #3 lore raw

PATCH 1/1 backend plugin for monitoring a host's status

Quoted 4 lines · Kazunori INOUE

Hi, Andrew
We (NTT) will participate in the HA Mini-Conference, please let we explain at that time.

Sounds good :-)

Quoted 60 lines

Best Regards
Quoted 55 lines · Andrew Beekhof
Was making host connectivity information available to the guests the goal here or just something to demonstrate the concept? Interesting idea :-)
2011/10/5 Kazunori INOUE<[email protected]>:
Quoted 49 lines
Hi all,
I think that the communication function of fence-virt is flexible, so I want to use it more effectively.
Therefore I made backend plugin for a guest to get the host's status using the communication facility of fence-virt,
and I changed to allow specifying one more backend (for fencing, and replying the host's status).
I created the backend "pm-monitor" which has met the following configurations / requirements.
- Both hosts and VMs, cluster (Pacemaker) have been configured.
Here's an overview of function. Please refer to attached 'overview.png'. (*) pingd resource notifies the status of connection with a specific ? ?host to pacemaker, and pacemaker manages the result.
(1) resource (vm-client) which requires the host's status is executed.
(2) vm-client requests 'host_status (result of pingd)' to the host ? ?with fence_virt.
(3) use the serial listener,
(4) fence_virtd (pm-monitor backend) gets the 'result of pingd' from ? ?pacemaker and answers it after conversion.
? ?- the conversion rule is set in /etc/pm-monitor.conf
Here's a description of the attached files.
* add_general_backend.patch
?- add the server/pm-fence.c
?- change the configure.in and server/Makefile.in
* overview.png
?- figure of the overview.
* vm-client
?- Resource Agent for VM.
?- I'm going to post this RA to the appropriate community ? ?(https://github.com/ClusterLabs/resource-agents).
* fence_virt.conf
?- sample configuration.
* pm-monitor.conf
?- sample configuration for pm-monitor.so
* host.cli, mon-host.txt
?- sample configuration file of a host cluster, ? ?and mon-host.txt is an output of the crm_mon command.
* VM.cli, mon-VM.txt
?- sample configuration file of a VM cluster,
? ?and mon-VM.txt is an output of the crm_mon command.
Best Regards

Lon Hohberger

2011-10-26 01:35 UTC #4 lore raw

PATCH 1/1 backend plugin for monitoring a host's status

In reply to Kazunori INOUE

Quoted 14 lines · Kazunori INOUE

Hi all,
I think that the communication function of fence-virt is flexible, so I want to use it more effectively.
Therefore I made backend plugin for a guest to get the host's status using the communication facility of fence-virt,
and I changed to allow specifying one more backend (for fencing, and replying the host's status).
I created the backend "pm-monitor" which has met the following configurations / requirements.
- Both hosts and VMs, cluster (Pacemaker) have been configured.

Quoted 10 lines

Here's an overview of function. Please refer to attached 'overview.png'. (*) pingd resource notifies the status of connection with a specific
host to pacemaker, and pacemaker manages the result.
(1) resource (vm-client) which requires the host's status is executed.
(2) vm-client requests 'host_status (result of pingd)' to the host with fence_virt.
(3) use the serial listener,
(4) fence_virtd (pm-monitor backend) gets the 'result of pingd' from pacemaker and answers it after conversion.
- the conversion rule is set in /etc/pm-monitor.conf

Originally, the devstatus callback was supposed to be what provided the answer to the question "is my fencing [device|host] operational?" - but your patch is more than that, it looks like a generalized way to do arbitrary request/response pacemaker resource monitoring from within VMs.

That kind of monitoring is interesting.

Quoted 4 lines

Here's a description of the attached files.
* add_general_backend.patch
- add the server/pm-fence.c
- change the configure.in and server/Makefile.in

I think I understand how it operates; your explanation is very detailed.

* I don't quite understand all of the benefits. Presumably, one uses the serial configuration to prevent guests/hosts from sharing the same networks - i.e., it's designed for environments where guests and hosts are not allowed to talk over the network to each other. So, presumably, we're using this to indirectly monitor an IP on the host network, using fence-virt as a bridge. What I don't understand is the benefit to the virtual machine cluster in doing this.

* Do you see other uses besides monitoring pingd sets?

* It may be that this new project may be a better "backbone" for this sort of general request/response than fence-virt; it's a much more general communications medium for guest<->host communication:

http://git.fedorahosted.org/git/?p=vios-proxy.git

(although it is written in C++ ;) ).

Notes about the patch itself:

* no support for multicast listener (this looks intentional - is it?)

* this will break on-wire compatibility; we need to be very careful here.

* some duplicate build system changes with the other patch (not a big deal)

* [not directly related to your patch] the vmchannel/serial support in fence-virt probably should be replaced with more recent technology in libvirt

-- Lon

Kazunori INOUE

author

2011-10-31 08:19 UTC #5 lore raw

PATCH 1/1 backend plugin for monitoring a host's status

Hi, Lon

Quoted 48 lines · Lon Hohberger

Quoted 14 lines · Kazunori INOUE
Hi all,
I think that the communication function of fence-virt is flexible, so I want to use it more effectively.
Therefore I made backend plugin for a guest to get the host's status using the communication facility of fence-virt,
and I changed to allow specifying one more backend (for fencing, and replying the host's status).
I created the backend "pm-monitor" which has met the following configurations / requirements.
- Both hosts and VMs, cluster (Pacemaker) have been configured.
Quoted 10 lines
Here's an overview of function. Please refer to attached 'overview.png'. (*) pingd resource notifies the status of connection with a specific
host to pacemaker, and pacemaker manages the result.
(1) resource (vm-client) which requires the host's status is executed.
(2) vm-client requests 'host_status (result of pingd)' to the host with fence_virt.
(3) use the serial listener,
(4) fence_virtd (pm-monitor backend) gets the 'result of pingd' from pacemaker and answers it after conversion.
- the conversion rule is set in /etc/pm-monitor.conf
Originally, the devstatus callback was supposed to be what provided the answer to the question "is my fencing [device|host] operational?" - but your patch is more than that, it looks like a generalized way to do arbitrary request/response pacemaker resource monitoring from within VMs.
That kind of monitoring is interesting.
Quoted 4 lines
Here's a description of the attached files.
* add_general_backend.patch
- add the server/pm-fence.c
- change the configure.in and server/Makefile.in
I think I understand how it operates; your explanation is very detailed.
* I don't quite understand all of the benefits. Presumably, one uses the serial configuration to prevent guests/hosts from sharing the same networks - i.e., it's designed for environments where guests and hosts are not allowed to talk over the network to each other. So, presumably, we're using this to indirectly monitor an IP on the host network, using fence-virt as a bridge. What I don't understand is the benefit to the virtual machine cluster in doing this.

Quoted 2 lines

* Do you see other uses besides monitoring pingd sets?

Yes, VM can get the status of host's hardware (status of disk, status of CPU, etc.) if the following RAs are running on the host.
- HealthCPU
- SysInfo
- diskd (disk monitor RA - published only in Japanese community.
http://hg.sourceforge.jp/view/linux-ha/pm_diskd/)

Since such RA writes status of hardware into the CIB(*), vm-client can specify the attribute and can get it's value.
(*) example of the CIB

    # cibadmin -Q
    <cib>
      <status>
        <node_state ...>
          <transient_attributes ...>
            <instance_attributes ...>
              <nvpair id="..." name="default_ping_set" value="100"/>
              <nvpair id="..." name="#health-cpu" value="green"/>
              <nvpair id="..." name="diskd" value="normal"/>
                :

Our final purpose is to get the status of host's hardware who cannot perform acquisition directly from a guest by this way.

Quoted 8 lines

* It may be that this new project may be a better "backbone" for this sort of general request/response than fence-virt; it's a much more general communications medium for guest<->host communication:
http://git.fedorahosted.org/git/?p=vios-proxy.git
(although it is written in C++ ;) ).

OK, thank you. We will also see it.

Quoted 6 lines

Notes about the patch itself:
* no support for multicast listener (this looks intentional -
    is it?)

Yes. This patch is only support for serial listener.

Best Regards,
Kazunori INOUE

Quoted 11 lines

* this will break on-wire compatibility; we need to be very careful here.
* some duplicate build system changes with the other patch (not a big deal)
* [not directly related to your patch] the vmchannel/serial support in fence-virt probably should be replaced with more recent technology in libvirt
-- Lon