Introduce Advanced Watch Dog module
v3 cover Introduce Advanced Watch Dog module
From: Zhang Chen <[email protected]>
Advanced Watch Dog is an universal monitoring module on VMM side, it can be used to detect network down(VMM to guest, VMM to VMM, VMM to another remote server) and do previously set operation. Current AWD patch just accept any input as the signal to refresh the watchdog timer, and we can also make a certain interactive protocol here. For the output user can pre-write
some command or some messages in the AWD opt-script. We noticed that there is no way
for VMM communicate directly, maybe some people think we don't need such things(up layer
software like openstack can handle it). But we engaged with real customer found that in some cases,they need a lightweight and efficient mechanism to solve some practical problems(openstack is too heavy).
for example: When it detects lost connection with the paired node,it will send message to admin, notify another VMM, send qmp command to qemu do some operation like restart the VM, build VMM heartbeat system, etc.
It make user have basic VM/Host network monitoring tools and basic false tolerance and recovery solution.
Demo usage(for COLO heartbeat service):
In primary node:
-chardev socket,id=h1,host=3.3.3.3,port=9009,server,nowait
-chardev socket,id=heartbeat0,host=3.3.3.3,port=4445
-object iothread,id=iothread2
-object advanced-watchdog,id=heart1,server=on,awd_node=h1,notification_node=heartbeat0,opt_script=colo_opt_script_path,iothread=iothread1,pulse_interval=1000,timeout=5000
In secondary node:
-monitor tcp::4445,server,nowait
-chardev socket,id=h1,host=3.3.3.3,port=9009,reconnect=1
-chardev socket,id=heart1,host=3.3.3.8,port=4445
-object iothread,id=iothread1
-object advanced-watchdog,id=heart1,server=off,awd_node=h1,notification_node=heart1,opt_script=colo_secondary_opt_script,iothread=iothread1,timeout=10000
V3:
- Rebased on Qemu 4.2.0-rc1 code.
- Fix commit message issue.
V2:
- Addressed Philippe comments add configure selector for AWD.
Initial:
- Initial version.
Zhang Chen (4):
net/awd.c: Introduce Advanced Watch Dog module framework net/awd.c: Initailize input/output chardev net/awd.c: Load advanced watch dog worker thread job vl.c: Make Advanced Watch Dog delayed initialization
configure | 9 + net/Makefile.objs | 1 + net/awd.c | 491 ++++++++++++++++++++++++++++++++++++++++++++++ qemu-options.hx | 6 + vl.c | 7 + 5 files changed, 514 insertions(+) create mode 100644 net/awd.c
Signature · 1 line
2.17.1
PATCH v3 1/4 net/awd.c: Introduce Advanced Watch Dog module framework
From: Zhang Chen <[email protected]>
This patch introduce a new module named Advanced Watch Dog, and defined the input and output parameter. AWD use standard chardev as the way of communicationg with the outside world.
If you want to use it, please add "--enable-awd" when configure qemu.
Demo command:
-object advanced-watchdog,id=heart1,server=on,awd_node=h1,notification_node=heartbeat0,opt_script=opt_script_path,iothread=iothread1,pulse_interval=1000,timeout=5000
--- configure | 9 ++ net/Makefile.objs | 1 + net/awd.c | 261 ++++++++++++++++++++++++++++++++++++++++++++++ qemu-options.hx | 6 ++ 4 files changed, 277 insertions(+) create mode 100644 net/awd.c
configure +9 -0
@@ -383,6 +383,7 @@ vhost_scsi="" vhost_vsock="" vhost_user="" vhost_user_fs="" +awd="no" kvm="no" hax="no" hvf="no"
@@ -1304,6 +1305,10 @@ for opt do ;; --enable-vhost-user-fs) vhost_user_fs="yes" ;; + --disable-awd) awd="no" + ;; + --enable-awd) awd="yes" + ;; --disable-opengl) opengl="no" ;; --enable-opengl) opengl="yes"
@@ -1780,6 +1785,7 @@ disabled with --disable-FEATURE, default is enabled if available: vhost-crypto vhost-user-crypto backend support vhost-kernel vhost kernel backend support vhost-user vhost-user backend support + awd Advanced Watch Dog support spice spice rbd rados block device (rbd) libiscsi iscsi support
@@ -7043,6 +7049,9 @@ fi if test "$vhost_user" = "yes" ; then echo "CONFIG_VHOST_USER=y" >> $config_host_mak fi +if test "$awd" = "yes" ; then + echo "CONFIG_AWD=y" >> $config_host_mak +fi if test "$vhost_user_fs" = "yes" ; then echo "CONFIG_VHOST_USER_FS=y" >> $config_host_mak fi
net/Makefile.objs +1 -0
@@ -19,6 +19,7 @@ common-obj-y += colo-compare.o common-obj-y += colo.o common-obj-y += filter-rewriter.o common-obj-y += filter-replay.o +common-obj-$(CONFIG_AWD) += awd.o tap-obj-$(CONFIG_LINUX) = tap-linux.o tap-obj-$(CONFIG_BSD) = tap-bsd.o
net/awd.c new file +261 -0
261 lines not on this page · raw message
qemu-options.hx +6 -0
@@ -4589,6 +4589,12 @@ Dump the network traffic on netdev @var{dev} to the file specified by The file format is libpcap, so it can be analyzed with tools such as tcpdump or Wireshark. +@item -object advanced-watchdog,id=@var{id},awd_node=@var{chardevid},notification_node=@var{chardevid},server=@var{server},iothread=@var{id}[,pulse_interval=@var{time_ms},timeout=@var{time_ms}] + +Advanced Watch Dog is an universal monitoring module on VMM side, it can be used to detect network down(VMM to guest, VMM to VMM, VMM to another remote server) and do previously set operation. +for example: send message to admin, notify another VMM, send qmp command to qemu do some operation like restart the VM, build VMM heartbeat system, etc. +It make user have basic VM/Host network monitoring tools and basic false tolerance and recovery solution. + @item -object colo-compare,id=@var{id},primary_in=@var{chardevid},secondary_in=@var{chardevid},outdev=@var{chardevid},iothread=@var{id}[,vnet_hdr_support][,notify_dev=@var{id}] Colo-compare gets packet from primary_in@var{chardevid} and secondary_in@var{chardevid}, than compare primary packet with
Signature · 1 line
2.17.1
PATCH v3 2/4 net/awd.c: Initailize input/output chardev
In reply to Zhang ChenFrom: Zhang Chen <[email protected]>
Find and check the chardev awd_node and notification_node, The awd_node used for keep connect with outside(like VM client/other host/Remote server), and the notification_node used for do some operation when disconnect event occur.
--- net/awd.c | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+)
net/awd.c +37 -0
@@ -42,6 +42,8 @@ typedef struct AwdState {char *opt_script;uint32_t pulse_interval;uint32_t timeout; +CharBackend chr_awd_node; +CharBackend chr_notification_node;IOThread *iothread; } AwdState;
@@ -175,9 +177,30 @@ out:error_propagate (errp, local_err); } +static int find_and_check_chardev (Chardev **chr, +char *chr_name, +Error **errp) +{ + *chr =qemu_chr_find (chr_name); +if (*chr ==NULL ) { +error_setg (errp,"Device '%s' not found" , + chr_name); +return 1 ; + } + +if (!qemu_chr_has_feature (*chr,QEMU_CHAR_FEATURE_RECONNECTABLE )) { +error_setg (errp,"chardev \"%s\" is not reconnectable" , + chr_name); +return 1 ; + } + +return 0 ; +} +static void awd_complete (UserCreatable *uc,Error **errp) {AwdState *s =AWD (uc); +Chardev *chr;if (!s->awd_node || !s->iothread || !s->notification_node || !s->opt_script) {
@@ -187,6 +210,20 @@ static void awd_complete(UserCreatable *uc, Error **errp)return ; } +if (find_and_check_chardev (&chr, s->awd_node, errp) || + !qemu_chr_fe_init (&s->chr_awd_node, chr, errp)) { +error_setg (errp,"advanced-watchdog can't find chardev awd_node: %s" , + s->awd_node); +return ; + } + +if (find_and_check_chardev (&chr, s->notification_node, errp) || + !qemu_chr_fe_init (&s->chr_notification_node, chr, errp)) { +error_setg (errp,"advanced-watchdog can't find " +"chardev notification_node: %s" , s->notification_node); +return ; + } +return ; }
Signature · 1 line
2.17.1
PATCH v3 3/4 net/awd.c: Load advanced watch dog worker thread job
In reply to Zhang ChenFrom: Zhang Chen <[email protected]>
This patch load pulse_timer and timeout_timer in the new iothread. The pulse timer will send pulse info to awd_node, and the timeout timer will check the reply pulse from awd_node. If timeout occur, it will send opt_script's data to the notification_node.
--- net/awd.c | 193 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 193 insertions(+)
net/awd.c +193 -0
@@ -40,17 +40,137 @@ typedef struct AwdState {char *awd_node;char *notification_node;char *opt_script; +char *opt_script_data;uint32_t pulse_interval;uint32_t timeout;CharBackend chr_awd_node;CharBackend chr_notification_node; +SocketReadState awd_rs; + +QEMUTimer *pulse_timer; +QEMUTimer *timeout_timer;IOThread *iothread; +GMainContext *worker_context; } AwdState;typedef struct AwdClass {ObjectClass parent_class; }AwdClass ; +static int awd_chr_send (AwdState *s, +const uint8_t *buf, +uint32_t size) +{ +int ret =0 ; +uint32_t len =htonl (size); + +if (!size) { +return 0 ; + } + + ret =qemu_chr_fe_write_all (&s->chr_awd_node, (uint8_t *)&len, +sizeof (len)); +if (ret !=sizeof (len)) { + goto err; + } + + ret =qemu_chr_fe_write_all (&s->chr_awd_node, (uint8_t *)buf, + size); +if (ret != size) { + goto err; + } + +return 0 ; + +err: +return ret <0 ? ret : -EIO ; +} + +static int awd_chr_can_read (void *opaque) +{ +return AWD_READ_LEN_MAX ; +} + +static void awd_node_in (void *opaque,const uint8_t *buf,int size) +{ +AwdState *s =AWD (opaque); +int ret; + + ret =net_fill_rstate (&s->awd_rs, buf, size); +if (ret ==-1 ) { +qemu_chr_fe_set_handlers (&s->chr_awd_node,NULL ,NULL ,NULL ,NULL , +NULL ,NULL , true); +error_report ("advanced-watchdog get pulse error" ); + } +} + +static void awd_send_pulse (void *opaque) +{ +AwdState *s = opaque; +char buf[] ="advanced-watchdog pulse" ; + +awd_chr_send (s, (uint8_t *)buf,sizeof (buf)); +} + +static void awd_regular_pulse (void *opaque) +{ +AwdState *s = opaque; + +awd_send_pulse (s); +timer_mod (s->pulse_timer,qemu_clock_get_ms (QEMU_CLOCK_VIRTUAL ) + + s->pulse_interval); +} + +static void awd_timeout (void *opaque) +{ +AwdState *s = opaque; +int ret =0 ; + + ret =qemu_chr_fe_write_all (&s->chr_notification_node, + (uint8_t *)s->opt_script_data, +strlen (s->opt_script_data)); +if (ret) { +error_report ("advanced-watchdog notification failure" ); + } +} + +static void awd_timer_init (AwdState *s) +{ +AioContext *ctx =iothread_get_aio_context (s->iothread); + + s->timeout_timer =aio_timer_new (ctx,QEMU_CLOCK_VIRTUAL ,SCALE_MS , + awd_timeout, s); + + s->pulse_timer =aio_timer_new (ctx,QEMU_CLOCK_VIRTUAL ,SCALE_MS , + awd_regular_pulse, s); + +if (!s->pulse_interval) { + s->pulse_interval =AWD_PULSE_INTERVAL_DEFAULT ; + } + +if (!s->timeout) { + s->timeout =AWD_TIMEOUT_DEFAULT ; + } + +timer_mod (s->pulse_timer,qemu_clock_get_ms (QEMU_CLOCK_VIRTUAL ) + + s->pulse_interval); +} + +static void awd_timer_del (AwdState *s) +{ +if (s->pulse_timer) { +timer_del (s->pulse_timer); +timer_free (s->pulse_timer); + s->pulse_timer =NULL ; + } + +if (s->timeout_timer) { +timer_del (s->timeout_timer); +timer_free (s->timeout_timer); + s->timeout_timer =NULL ; + } + } +static char *awd_get_node (Object *obj,Error **errp) {AwdState *s =AWD (obj);
@@ -177,6 +297,22 @@ out:error_propagate (errp, local_err); } +static void awd_rs_finalize (SocketReadState *awd_rs) +{ +AwdState *s =container_of (awd_rs, AwdState, awd_rs); + +if (!s->server) { +char buf[] ="advanced-watchdog reply pulse" ; + +awd_chr_send (s, (uint8_t *)buf,sizeof (buf)); + } + +timer_mod (s->timeout_timer,qemu_clock_get_ms (QEMU_CLOCK_VIRTUAL ) + + s->timeout); + +error_report ("advanced-watchdog got message : %s" , awd_rs->buf); +} +static int find_and_check_chardev (Chardev **chr,char *chr_name,Error **errp)
@@ -197,6 +333,46 @@ static int find_and_check_chardev(Chardev **chr,return 0 ; } +static void awd_iothread (AwdState *s) +{ +object_ref (OBJECT (s->iothread)); + s->worker_context =iothread_get_g_main_context (s->iothread); + +qemu_chr_fe_set_handlers (&s->chr_awd_node, awd_chr_can_read, + awd_node_in,NULL ,NULL , + s, s->worker_context, true); + +awd_timer_init (s); +} + +static int get_opt_script_data (AwdState *s) +{ +FILE *opt_fd; +long fsize; + + opt_fd =fopen (s->opt_script,"r" ); +if (opt_fd ==NULL ) { +error_report ("advanced-watchdog can't open " +"opt_script: %s" , s->opt_script); +return -1 ; + } + +fseek (opt_fd,0 ,SEEK_END ); + fsize =ftell (opt_fd); +fseek (opt_fd,0 ,SEEK_SET ); + s->opt_script_data =malloc (fsize +1 ); + +if (!fread (s->opt_script_data,1 , fsize, opt_fd)) { +error_report ("advanced-watchdog can't read " +"opt_script: %s" , s->opt_script); +return -1 ; + } + +fclose (opt_fd); + +return 0 ; +} +static void awd_complete (UserCreatable *uc,Error **errp) {AwdState *s =AWD (uc);
@@ -224,6 +400,16 @@ static void awd_complete(UserCreatable *uc, Error **errp)return ; } +if (get_opt_script_data (s)) { +error_setg (errp,"advanced-watchdog can't get " +"opt script data: %s" , s->opt_script); +return ; + } + +net_socket_rs_init (&s->awd_rs, awd_rs_finalize, false); + +awd_iothread (s); +return ; }
@@ -272,6 +458,13 @@ static void awd_finalize(Object *obj) {AwdState *s =AWD (obj); +qemu_chr_fe_deinit (&s->chr_awd_node, false); +qemu_chr_fe_deinit (&s->chr_notification_node, false); + +if (s->iothread) { +awd_timer_del (s); + } +g_free (s->awd_node);g_free (s->notification_node); }
Signature · 1 line
2.17.1
PATCH v3 4/4 vl.c: Make Advanced Watch Dog delayed initialization
In reply to Zhang ChenFrom: Zhang Chen <[email protected]>
Advanced Watch Dog module needs chardev socket
to initialize properly before running.
--- vl.c | 7 +++++++ 1 file changed, 7 insertions(+)
vl.c +7 -0
@@ -2689,6 +2689,13 @@ static bool object_create_initial(const char *type, QemuOpts *opts)return false; } +/* +* Reason: Advanced Watch Dog property "chardev". +*/ +if (g_str_equal (type,"advanced-watchdog" )) { +return false; + } +/* Memory allocation by backends needs to be done * after configure_accelerator() (due to the tcg_enabled() * checks at memory_region_init_*()).
Signature · 1 line
2.17.1
Hi All~
No news for a long time.
Please give me more comments about this series.
Thanks
Zhang Chen
Quoted 53 lines · Chen
From: Zhang Chen <[email protected]>
Advanced Watch Dog is an universal monitoring module on VMM side, it can be used to detect network
down(VMM to guest, VMM to VMM, VMM to another remote server)and do previously set operation. Current AWD patch just accept any input as the signal to refresh the watchdog timer, and we can also make a certain interactive protocol here. For the output user can pre-write
some command or some messages in the AWD opt-script. We noticed that there is no way
for VMM communicate directly, maybe some people think we don't need such things(up layer
software like openstack can handle it). But we engaged with real customer found that in some cases,they need a lightweight and efficient mechanism to solve some practical problems(openstack is too heavy).
for example: When it detects lost connection with the paired node,it will send message to admin, notify another VMM, send qmp command to qemu do some operation like restart the VM, build VMM heartbeat system, etc.
It make user have basic VM/Host network monitoring tools and basic false tolerance and recovery solution.Demo usage(for COLO heartbeat service):
In primary node:
(quoted diff) +0
-4-chardev socket,id=h1,host=3.3.3.3,port=9009,server,nowait -chardev socket,id=heartbeat0,host=3.3.3.3,port=4445 -object iothread,id=iothread2 -object advanced-watchdog,id=heart1,server=on,awd_node=h1,notification_node=heartbeat0,opt_script=colo_opt_script_path,iothread=iothread1,pulse_interval=1000,timeout=5000In secondary node:
(quoted diff) +0
-5-monitor tcp::4445,server,nowait -chardev socket,id=h1,host=3.3.3.3,port=9009,reconnect=1 -chardev socket,id=heart1,host=3.3.3.8,port=4445 -object iothread,id=iothread1 -object advanced-watchdog,id=heart1,server=off,awd_node=h1,notification_node=heart1,opt_script=colo_secondary_opt_script,iothread=iothread1,timeout=10000V3:
- Rebased on Qemu 4.2.0-rc1 code.
- Fix commit message issue.V2:
- Addressed Philippe comments add configure selector for AWD.Initial:
- Initial version.Zhang Chen (4):
net/awd.c: Introduce Advanced Watch Dog module frameworknet/awd.c: Initailize input/output chardevnet/awd.c: Load advanced watch dog worker thread jobvl.c: Make Advanced Watch Dog delayed initializationconfigure | 9 + net/Makefile.objs | 1 + net/awd.c | 491 ++++++++++++++++++++++++++++++++++++++++++++++ qemu-options.hx | 6 + vl.c | 7 + 5 files changed, 514 insertions(+)create mode 100644
net/awd.c
Quoted 5 lines · Chen
Hi All~
No news for a long time.
Please give me more comments about this series.
Sorry, people were probably busy with the QEMU release candidates.
Even before looking at the code, the series is completely missing documentation on how to use it and on the chardev protocol. The documentation should go in docs/ and should be written as restructuredText.
The qemu-options.hx patches also lack documentation about the properties accepted by the new object.
In particular:
Quoted 5 lines
Quoted 5 lines
(quoted diff) +0
-4-chardev socket,id=h1,host=3.3.3.3,port=9009,server,nowait -chardev socket,id=heartbeat0,host=3.3.3.3,port=4445 -object iothread,id=iothread2 -objectadvanced-watchdog,id=heart1,server=on,awd_node=h1,notification_node=heartbeat0,opt_script=colo_opt_script_path,iothread=iothread1,pulse_interval=1000,timeout=5000
What are the two sockets for, and what should be in colo_opt_script_path?
Quoted 9 lines
Quoted 9 lines
In secondary node:
(quoted diff) +0
-5-monitor tcp::4445,server,nowait -chardev socket,id=h1,host=3.3.3.3,port=9009,reconnect=1 -chardev socket,id=heart1,host=3.3.3.8,port=4445 -object iothread,id=iothread1 -objectadvanced-watchdog,id=heart1,server=off,awd_node=h1,notification_node=heart1,opt_script=colo_secondary_opt_script,iothread=iothread1,timeout=10000
Same here.
Paolo
Quoted 6 lines · Paolo Bonzini
Quoted 5 lines · Chen
Hi All~
No news for a long time.
Please give me more comments about this series.
Quoted 8 lines
Sorry, people were probably busy with the QEMU release candidates.
Even before looking at the code, the series is completely missing documentation on how to use it and on the chardev protocol. The documentation should go in docs/ and should be written as restructuredText.
The qemu-options.hx patches also lack documentation about the properties accepted by the new object.
OK, I will add documentation in docs/ and qemu-options.hx in next version.
For the chardev protocol part, current implementation just use plaintext that make AWD easy to connect with other user defined node, I am not very familiar with this part, do you have any suggestions here? maybe use some general protocol is better? or Jason have any suggestions?
Quoted 9 lines
In particular:
Quoted 5 lines
Quoted 5 lines
(quoted diff) +0
-4-chardev socket,id=h1,host=3.3.3.3,port=9009,server,nowait -chardev socket,id=heartbeat0,host=3.3.3.3,port=4445 -object iothread,id=iothread2 -objectadvanced-watchdog,id=heart1,server=on,awd_node=h1,notification_node=heartbeat0,opt_script=colo_opt_script_path,iothread=iothread1,pulse_interval=1000,timeout=5000
What are the two sockets for, and what should be in
colo_opt_script_path?
Anything user want to send when timeout, for example:
If timeout is detected, AWD send quit command to Qemu.
colo_opt_script_path=/tmp/qemu-qmp-quit.script
------------------------------------
qemu-qmp-quit.script:
{ "execute": "quit" }
------------------------------------
Thanks
Zhang Chen
Quoted 13 lines
Quoted 9 lines
Quoted 9 lines
In secondary node:
(quoted diff) +0
-5-monitor tcp::4445,server,nowait -chardev socket,id=h1,host=3.3.3.3,port=9009,reconnect=1 -chardev socket,id=heart1,host=3.3.3.8,port=4445 -object iothread,id=iothread1 -objectadvanced-watchdog,id=heart1,server=off,awd_node=h1,notification_node=heart1,opt_script=colo_secondary_opt_script,iothread=iothread1,timeout=10000
Same here.
Paolo