Hyper-V: add tracing to VMBus module and trace messages/events
v3 cover Hyper-V: add tracing to VMBus module and trace messages/events
Changes since v2:
- Use DEFINE_EVENT in PATCH1/2 to avoid compile warnings [Stephen Hemminger]
- Add PATCH17 tracing channel events [Stephen Hemminger]
Messages between guest and host are used in Hyper-V as control flow. To simplify debugging various issues which are often hard to reproduce add tracepoints to all message senders and handlers. This is not a performance critical path and tracing overhead should be negligible.
The example usage and output is:
Enable all tracing events:
# echo 1 > /sys/kernel/debug/tracing/events/hyperv/enable
Do something which causes messages to be sent between host and guest, e.g. hot remove a VMBus device.
Check events:
# cat /sys/kernel/debug/tracing/trace
# tracer: nop
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
<idle>-0 [011] ..s. 122.981583: vmbus_on_msg_dpc: msgtype=1
kworker/11:7-1506 [011] .... 122.981597: vmbus_on_message: msgtype=1
kworker/11:7-1506 [011] .... 122.981598: vmbus_onoffer: child_relid 0x10, monitorid 0x2, is_dedicated 1, connection_id 0x10010, if_type f8615163-df3e-46c5-913f-f2d2f965ed0e, if_instance 6676e078-e4b3-44da-8a7d-12eafb577d31, chn_flags 0x0, mmio_megabytes 0, sub_channel_index 0
kworker/11:7-1506 [011] .... 122.982130: vmbus_establish_gpadl_header: sending child_relid 0x10, gpadl 0xe1e34, range_buflen 2056 rangecount 1, ret 0
kworker/11:7-1506 [011] .... 122.982133: vmbus_establish_gpadl_body: sending msgnumber 0, gpadl 0xe1e34, ret 0
kworker/11:7-1506 [011] .... 122.982136: vmbus_establish_gpadl_body: sending msgnumber 0, gpadl 0xe1e34, ret 0
kworker/11:7-1506 [011] .... 122.982137: vmbus_establish_gpadl_body: sending msgnumber 0, gpadl 0xe1e34, ret 0
kworker/11:7-1506 [011] .... 122.982139: vmbus_establish_gpadl_body: sending msgnumber 0, gpadl 0xe1e34, ret 0
kworker/11:7-1506 [011] .... 122.982141: vmbus_establish_gpadl_body: sending msgnumber 0, gpadl 0xe1e34, ret 0
kworker/11:7-1506 [011] .... 122.982142: vmbus_establish_gpadl_body: sending msgnumber 0, gpadl 0xe1e34, ret 0
kworker/11:7-1506 [011] .... 122.982144: vmbus_establish_gpadl_body: sending msgnumber 0, gpadl 0xe1e34, ret 0
kworker/11:7-1506 [011] .... 122.982146: vmbus_establish_gpadl_body: sending msgnumber 0, gpadl 0xe1e34, ret 0
kworker/11:7-1506 [011] .... 122.982148: vmbus_establish_gpadl_body: sending msgnumber 0, gpadl 0xe1e34, ret 0
<idle>-0 [011] ..s. 122.982336: vmbus_on_msg_dpc: msgtype=10
<idle>-0 [011] ..s. 122.982337: vmbus_ongpadl_created: child_relid 0x10, gpadl 0xe1e34, creation_status 0
kworker/11:7-1506 [011] .... 122.982351: vmbus_open: sending child_relid 0x10, openid 16, gpadlhandle 0xe1e34, target_vp 0xb, offset 0x80, ret 0
kworker/3:1-214 [003] .... 123.015007: vmbus_setevent: relid 0x5
<idle>-0 [011] ..s. 123.029467: vmbus_on_msg_dpc: msgtype=6
<idle>-0 [011] ..s. 123.029470: vmbus_onopen_result: child_relid 0x10, openid 16, status 0
kworker/11:7-1506 [011] .... 123.029492: vmbus_setevent: relid 0x10
<idle>-0 [011] d.h. 123.029533: vmbus_chan_sched: relid 0x10
kworker/11:7-1506 [011] .... 123.029539: vmbus_setevent: relid 0x10CHANNELMSG_UNLOAD/CHANNELMSG_UNLOAD_RESPONSE are not traced as these are mostly used on crash.
Vitaly Kuznetsov (17):
hyper-v: trace vmbus_on_msg_dpc()
hyper-v: trace vmbus_on_message()
hyper-v: trace vmbus_onoffer()
hyper-v: trace vmbus_onoffer_rescind()
hyper-v: trace vmbus_onopen_result()
hyper-v: trace vmbus_ongpadl_created()
hyper-v: trace vmbus_ongpadl_torndown()
hyper-v: trace vmbus_onversion_response()
hyper-v: trace vmbus_request_offers()
hyper-v: trace vmbus_open()
hyper-v: trace vmbus_close_internal()
hyper-v: trace vmbus_establish_gpadl()
hyper-v: trace vmbus_teardown_gpadl()
hyper-v: trace vmbus_negotiate_version()
hyper-v: trace vmbus_release_relid()
hyper-v: trace vmbus_send_tl_connect_request()
hyper-v: trace channel events
drivers/hv/Makefile | 4 +- drivers/hv/channel.c | 21 ++- drivers/hv/channel_mgmt.c | 26 +++- drivers/hv/connection.c | 5 + drivers/hv/hv_trace.c | 4 + drivers/hv/hv_trace.h | 327 ++++++++++++++++++++++++++++++++++++++++++++++ drivers/hv/hyperv_vmbus.h | 2 + drivers/hv/vmbus_drv.c | 4 + 8 files changed, 388 insertions(+), 5 deletions(-) create mode 100644 drivers/hv/hv_trace.c create mode 100644 drivers/hv/hv_trace.h
Signature · 1 line
2.13.6
PATCH v3 1/17 hyper-v: trace vmbus_on_msg_dpc()
Add tracing subsystem to Hyper-V VMBus module and add tracepoint to vmbus_on_msg_dpc() which is called when we receive a message from host.
--- drivers/hv/Makefile | 4 +++- drivers/hv/hv_trace.c | 4 ++++ drivers/hv/hv_trace.h | 29 +++++++++++++++++++++++++++++ drivers/hv/hyperv_vmbus.h | 2 ++ drivers/hv/vmbus_drv.c | 2 ++ 5 files changed, 40 insertions(+), 1 deletion(-) create mode 100644 drivers/hv/hv_trace.c create mode 100644 drivers/hv/hv_trace.h
drivers/hv/Makefile +3 -1
@@ -2,7 +2,9 @@ obj-$(CONFIG_HYPERV) += hv_vmbus.o obj-$(CONFIG_HYPERV_UTILS) += hv_utils.o obj-$(CONFIG_HYPERV_BALLOON) += hv_balloon.o +CFLAGS_hv_trace.o = -I$(src) + hv_vmbus-y := vmbus_drv.o \ hv.o connection.o channel.o \ - channel_mgmt.o ring_buffer.o + channel_mgmt.o ring_buffer.o hv_trace.o hv_utils-y := hv_util.o hv_kvp.o hv_snapshot.o hv_fcopy.o hv_utils_transport.o
drivers/hv/hv_trace.c new file +4 -0
@@ -0,0 +1,4 @@ +#include "hyperv_vmbus.h" + +#define CREATE_TRACE_POINTS +#include "hv_trace.h"
drivers/hv/hv_trace.h new file +29 -0
@@ -0,0 +1,29 @@ +#undef TRACE_SYSTEM +#define TRACE_SYSTEM hyperv + +#if !defined(_HV_TRACE_H) || defined(TRACE_HEADER_MULTI_READ ) +#define _HV_TRACE_H + +#include <linux/tracepoint.h> + +DECLARE_EVENT_CLASS (vmbus_hdr_msg, + TP_PROTO(const struct vmbus_channel_message_header *hdr), +TP_ARGS (hdr), + TP_STRUCT__entry(__field(unsigned int, msgtype)), + TP_fast_assign(__entry->msgtype = hdr->msgtype;), +TP_printk ("msgtype=%u" , __entry->msgtype) +); + +DEFINE_EVENT (vmbus_hdr_msg, vmbus_on_msg_dpc, + TP_PROTO(const struct vmbus_channel_message_header *hdr), +TP_ARGS (hdr) +); + +#undef TRACE_INCLUDE_PATH +#define TRACE_INCLUDE_PATH . +#undef TRACE_INCLUDE_FILE +#define TRACE_INCLUDE_FILE hv_trace +#endif /* _HV_TRACE_H */ + +/* This part must be outside protection */ +#include <trace/define_trace.h>
drivers/hv/hyperv_vmbus.h +2 -0
@@ -31,6 +31,8 @@#include <linux/hyperv.h> #include <linux/interrupt.h> +#include "hv_trace.h" +/* * Timeout for services such as KVP and fcopy. */
drivers/hv/vmbus_drv.c +2 -0
@@ -834,6 +834,8 @@ void vmbus_on_msg_dpc(unsigned long data) hdr = (struct vmbus_channel_message_header *)msg->u.payload; +trace_vmbus_on_msg_dpc (hdr); +if (hdr->msgtype >=CHANNELMSG_COUNT ) {WARN_ONCE (1 ,"unknown msgtype=%d\n" , hdr->msgtype); goto msg_handled;
Signature · 1 line
2.13.6
PATCH v3 2/17 hyper-v: trace vmbus_on_message()
In reply to Vitaly KuznetsovAdd tracepoint to vmbus_on_message() which is called when we start processing a blocking from work context.
--- drivers/hv/channel_mgmt.c | 2 ++ drivers/hv/hv_trace.h | 5 +++++ 2 files changed, 7 insertions(+)
drivers/hv/channel_mgmt.c +2 -0
@@ -1170,6 +1170,8 @@ void vmbus_onmessage(void *context) hdr = (struct vmbus_channel_message_header *)msg->u.payload; size = msg->header.payload_size; +trace_vmbus_on_message (hdr); +if (hdr->msgtype >=CHANNELMSG_COUNT ) {pr_err ("Received invalid channel message type %d size %d\n" , hdr->msgtype, size);
drivers/hv/hv_trace.h +5 -0
@@ -19,6 +19,11 @@ DEFINE_EVENT(vmbus_hdr_msg, vmbus_on_msg_dpc,TP_ARGS (hdr) ); +DEFINE_EVENT (vmbus_hdr_msg, vmbus_on_message, + TP_PROTO(const struct vmbus_channel_message_header *hdr), +TP_ARGS (hdr) +); +#undef TRACE_INCLUDE_PATH#define TRACE_INCLUDE_PATH .#undef TRACE_INCLUDE_FILE
Signature · 1 line
2.13.6
PATCH v3 3/17 hyper-v: trace vmbus_onoffer()
In reply to Vitaly KuznetsovAdd tracepoint to CHANNELMSG_OFFERCHANNEL handler.
--- drivers/hv/channel_mgmt.c | 2 ++ drivers/hv/hv_trace.h | 37 +++++++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+)
drivers/hv/channel_mgmt.c +2 -0
@@ -797,6 +797,8 @@ static void vmbus_onoffer(struct vmbus_channel_message_header *hdr) offer = (struct vmbus_channel_offer_channel *)hdr; +trace_vmbus_onoffer (offer); +/* Allocate the channel object and save this offer. */ newchannel =alloc_channel ();if (!newchannel) {
drivers/hv/hv_trace.h +37 -0
@@ -24,6 +24,43 @@ DEFINE_EVENT(vmbus_hdr_msg, vmbus_on_message,TP_ARGS (hdr) ); +TRACE_EVENT (vmbus_onoffer, + TP_PROTO(const struct vmbus_channel_offer_channel *offer), +TP_ARGS (offer), +TP_STRUCT__entry ( + __field(u32, child_relid) + __field(u8, monitorid) + __field(u16, is_ddc_int) + __field(u32, connection_id) + __array(char, if_type, 16) + __array(char, if_instance, 16) + __field(u16, chn_flags) + __field(u16, mmio_mb) +__field (u16, sub_idx) + ), + TP_fast_assign(__entry->child_relid = offer->child_relid; + __entry->monitorid = offer->monitorid; + __entry->is_ddc_int = offer->is_dedicated_interrupt; + __entry->connection_id = offer->connection_id; + memcpy(__entry->if_type, + &offer->offer.if_type.b, 16); + memcpy(__entry->if_instance, + &offer->offer.if_instance.b, 16); + __entry->chn_flags = offer->offer.chn_flags; + __entry->mmio_mb = offer->offer.mmio_megabytes; + __entry->sub_idx = offer->offer.sub_channel_index; + ), +TP_printk ("child_relid 0x%x, monitorid 0x%x, is_dedicated %d, " +"connection_id 0x%x, if_type %pUl, if_instance %pUl, " +"chn_flags 0x%x, mmio_megabytes %d, sub_channel_index %d" , + __entry->child_relid, __entry->monitorid, + __entry->is_ddc_int, __entry->connection_id, + __entry->if_type, __entry->if_instance, + __entry->chn_flags, __entry->mmio_mb, + __entry->sub_idx + ) + ); +#undef TRACE_INCLUDE_PATH#define TRACE_INCLUDE_PATH .#undef TRACE_INCLUDE_FILE
Signature · 1 line
2.13.6
PATCH v3 4/17 hyper-v: trace vmbus_onoffer_rescind()
In reply to Vitaly KuznetsovAdd tracepoint to CHANNELMSG_RESCIND_CHANNELOFFER handler.
--- drivers/hv/channel_mgmt.c | 2 ++ drivers/hv/hv_trace.h | 8 ++++++++ 2 files changed, 10 insertions(+)
drivers/hv/channel_mgmt.c +2 -0
@@ -841,6 +841,8 @@ static void vmbus_onoffer_rescind(struct vmbus_channel_message_header *hdr) rescind = (struct vmbus_channel_rescind_offer *)hdr; +trace_vmbus_onoffer_rescind (rescind); +/* * The offer msg and the corresponding rescind msg * from the host are guranteed to be ordered -
drivers/hv/hv_trace.h +8 -0
@@ -61,6 +61,14 @@ TRACE_EVENT(vmbus_onoffer, ) ); +TRACE_EVENT (vmbus_onoffer_rescind, + TP_PROTO(const struct vmbus_channel_rescind_offer *offer), +TP_ARGS (offer), +TP_STRUCT__entry (__field (u32, child_relid)), +TP_fast_assign (__entry->child_relid = offer->child_relid), +TP_printk ("child_relid 0x%x" , __entry->child_relid) + ); +#undef TRACE_INCLUDE_PATH#define TRACE_INCLUDE_PATH .#undef TRACE_INCLUDE_FILE
Signature · 1 line
2.13.6
PATCH v3 5/17 hyper-v: trace vmbus_onopen_result()
In reply to Vitaly KuznetsovAdd tracepoint to CHANNELMSG_OPENCHANNEL_RESULT handler.
--- drivers/hv/channel_mgmt.c | 2 ++ drivers/hv/hv_trace.h | 17 +++++++++++++++++ 2 files changed, 19 insertions(+)
drivers/hv/channel_mgmt.c +2 -0
@@ -980,6 +980,8 @@ static void vmbus_onopen_result(struct vmbus_channel_message_header *hdr) result = (struct vmbus_channel_open_result *)hdr; +trace_vmbus_onopen_result (result); +/* * Find the open msg, copy the result and signal/unblock the wait event */
drivers/hv/hv_trace.h +17 -0
@@ -69,6 +69,23 @@ TRACE_EVENT(vmbus_onoffer_rescind,TP_printk ("child_relid 0x%x" , __entry->child_relid) ); +TRACE_EVENT (vmbus_onopen_result, + TP_PROTO(const struct vmbus_channel_open_result *result), +TP_ARGS (result), +TP_STRUCT__entry ( + __field(u32, child_relid) + __field(u32, openid) +__field (u32, status) + ), + TP_fast_assign(__entry->child_relid = result->child_relid; + __entry->openid = result->openid; + __entry->status = result->status; + ), +TP_printk ("child_relid 0x%x, openid %d, status %d" , + __entry->child_relid, __entry->openid, __entry->status + ) + ); +#undef TRACE_INCLUDE_PATH#define TRACE_INCLUDE_PATH .#undef TRACE_INCLUDE_FILE
Signature · 1 line
2.13.6
PATCH v3 6/17 hyper-v: trace vmbus_ongpadl_created()
In reply to Vitaly KuznetsovAdd tracepoint to CHANNELMSG_GPADL_CREATED handler.
--- drivers/hv/channel_mgmt.c | 2 ++ drivers/hv/hv_trace.h | 17 +++++++++++++++++ 2 files changed, 19 insertions(+)
drivers/hv/channel_mgmt.c +2 -0
@@ -1026,6 +1026,8 @@ static void vmbus_ongpadl_created(struct vmbus_channel_message_header *hdr) gpadlcreated = (struct vmbus_channel_gpadl_created *)hdr; +trace_vmbus_ongpadl_created (gpadlcreated); +/* * Find the establish msg, copy the result and signal/unblock the wait * event
drivers/hv/hv_trace.h +17 -0
@@ -86,6 +86,23 @@ TRACE_EVENT(vmbus_onopen_result, ) ); +TRACE_EVENT (vmbus_ongpadl_created, + TP_PROTO(const struct vmbus_channel_gpadl_created *gpadlcreated), +TP_ARGS (gpadlcreated), +TP_STRUCT__entry ( + __field(u32, child_relid) + __field(u32, gpadl) +__field (u32, status) + ), + TP_fast_assign(__entry->child_relid = gpadlcreated->child_relid; + __entry->gpadl = gpadlcreated->gpadl; + __entry->status = gpadlcreated->creation_status; + ), +TP_printk ("child_relid 0x%x, gpadl 0x%x, creation_status %d" , + __entry->child_relid, __entry->gpadl, __entry->status + ) + ); +#undef TRACE_INCLUDE_PATH#define TRACE_INCLUDE_PATH .#undef TRACE_INCLUDE_FILE
Signature · 1 line
2.13.6
PATCH v3 7/17 hyper-v: trace vmbus_ongpadl_torndown()
In reply to Vitaly KuznetsovAdd tracepoint to CHANNELMSG_GPADL_TORNDOWN handler.
--- drivers/hv/channel_mgmt.c | 2 ++ drivers/hv/hv_trace.h | 8 ++++++++ 2 files changed, 10 insertions(+)
drivers/hv/channel_mgmt.c +2 -0
@@ -1076,6 +1076,8 @@ static void vmbus_ongpadl_torndown( gpadl_torndown = (struct vmbus_channel_gpadl_torndown *)hdr; +trace_vmbus_ongpadl_torndown (gpadl_torndown); +/* * Find the open msg, copy the result and signal/unblock the wait event */
drivers/hv/hv_trace.h +8 -0
@@ -103,6 +103,14 @@ TRACE_EVENT(vmbus_ongpadl_created, ) ); +TRACE_EVENT (vmbus_ongpadl_torndown, + TP_PROTO(const struct vmbus_channel_gpadl_torndown *gpadltorndown), +TP_ARGS (gpadltorndown), +TP_STRUCT__entry (__field (u32, gpadl)), +TP_fast_assign (__entry->gpadl = gpadltorndown->gpadl), +TP_printk ("gpadl 0x%x" , __entry->gpadl) + ); +#undef TRACE_INCLUDE_PATH#define TRACE_INCLUDE_PATH .#undef TRACE_INCLUDE_FILE
Signature · 1 line
2.13.6
PATCH v3 8/17 hyper-v: trace vmbus_onversion_response()
In reply to Vitaly KuznetsovAdd tracepoint to CHANNELMSG_VERSION_RESPONSE handler.
--- drivers/hv/channel_mgmt.c | 3 +++ drivers/hv/hv_trace.h | 11 +++++++++++ 2 files changed, 14 insertions(+)
drivers/hv/channel_mgmt.c +3 -0
@@ -1121,6 +1121,9 @@ static void vmbus_onversion_response(unsigned long flags; version_response = (struct vmbus_channel_version_response *)hdr; + +trace_vmbus_onversion_response (version_response); +spin_lock_irqsave (&vmbus_connection.channelmsg_lock, flags); list_for_each_entry(msginfo, &vmbus_connection.chn_msg_list,
drivers/hv/hv_trace.h +11 -0
@@ -111,6 +111,17 @@ TRACE_EVENT(vmbus_ongpadl_torndown,TP_printk ("gpadl 0x%x" , __entry->gpadl) ); +TRACE_EVENT (vmbus_onversion_response, + TP_PROTO(const struct vmbus_channel_version_response *response), +TP_ARGS (response), +TP_STRUCT__entry ( +__field (u8, ver) + ), + TP_fast_assign(__entry->ver = response->version_supported; + ), +TP_printk ("version_supported %d" , __entry->ver) + ); +#undef TRACE_INCLUDE_PATH#define TRACE_INCLUDE_PATH .#undef TRACE_INCLUDE_FILE
Signature · 1 line
2.13.6
PATCH v3 9/17 hyper-v: trace vmbus_request_offers()
In reply to Vitaly KuznetsovAdd tracepoint to CHANNELMSG_REQUESTOFFERS sender.
--- drivers/hv/channel_mgmt.c | 4 +++- drivers/hv/hv_trace.h | 8 ++++++++ 2 files changed, 11 insertions(+), 1 deletion(-)
drivers/hv/channel_mgmt.c +3 -1
@@ -1218,9 +1218,11 @@ int vmbus_request_offers(void) msg->msgtype =CHANNELMSG_REQUESTOFFERS ; - ret =vmbus_post_msg (msg,sizeof (struct vmbus_channel_message_header ), true); + +trace_vmbus_request_offers (ret); +if (ret !=0 ) {pr_err ("Unable to request offers - %d\n" , ret);
drivers/hv/hv_trace.h +8 -0
@@ -122,6 +122,14 @@ TRACE_EVENT(vmbus_onversion_response,TP_printk ("version_supported %d" , __entry->ver) ); +TRACE_EVENT (vmbus_request_offers, + TP_PROTO(int ret), +TP_ARGS (ret), +TP_STRUCT__entry (__field (int, ret)), +TP_fast_assign (__entry->ret = ret), +TP_printk ("sending ret %d" , __entry->ret) + ); +#undef TRACE_INCLUDE_PATH#define TRACE_INCLUDE_PATH .#undef TRACE_INCLUDE_FILE
Signature · 1 line
2.13.6
PATCH v3 10/17 hyper-v: trace vmbus_open()
In reply to Vitaly KuznetsovAdd tracepoint to CHANNELMSG_OPENCHANNEL sender.
--- drivers/hv/channel.c | 2 ++ drivers/hv/hv_trace.h | 27 +++++++++++++++++++++++++++ 2 files changed, 29 insertions(+)
drivers/hv/channel.c +2 -0
@@ -185,6 +185,8 @@ int vmbus_open(struct vmbus_channel *newchannel, u32 send_ringbuffer_size, ret =vmbus_post_msg (open_msg,sizeof (struct vmbus_channel_open_channel ), true); +trace_vmbus_open (open_msg, ret); +if (ret !=0 ) { err = ret; goto error_clean_msglist;
drivers/hv/hv_trace.h +27 -0
@@ -130,6 +130,33 @@ TRACE_EVENT(vmbus_request_offers,TP_printk ("sending ret %d" , __entry->ret) ); +TRACE_EVENT (vmbus_open, + TP_PROTO(const struct vmbus_channel_open_channel *msg, int ret), +TP_ARGS (msg, ret), +TP_STRUCT__entry ( + __field(u32, child_relid) + __field(u32, openid) + __field(u32, gpadlhandle) + __field(u32, target_vp) + __field(u32, offset) +__field (int, ret) + ), +TP_fast_assign ( + __entry->child_relid = msg->child_relid; + __entry->openid = msg->openid; + __entry->gpadlhandle = msg->ringbuffer_gpadlhandle; + __entry->target_vp = msg->target_vp; + __entry->offset = msg->downstream_ringbuffer_pageoffset; + __entry->ret = ret; + ), +TP_printk ("sending child_relid 0x%x, openid %d, " +"gpadlhandle 0x%x, target_vp 0x%x, offset 0x%x, ret %d" , + __entry->child_relid, __entry->openid, + __entry->gpadlhandle, __entry->target_vp, + __entry->offset, __entry->ret + ) + ); +#undef TRACE_INCLUDE_PATH#define TRACE_INCLUDE_PATH .#undef TRACE_INCLUDE_FILE
Signature · 1 line
2.13.6
PATCH v3 11/17 hyper-v: trace vmbus_close_internal()
In reply to Vitaly KuznetsovAdd tracepoint to CHANNELMSG_CLOSECHANNEL sender.
--- drivers/hv/channel.c | 2 ++ drivers/hv/hv_trace.h | 15 +++++++++++++++ 2 files changed, 17 insertions(+)
drivers/hv/channel.c +2 -0
@@ -591,6 +591,8 @@ static int vmbus_close_internal(struct vmbus_channel *channel) ret =vmbus_post_msg (msg,sizeof (struct vmbus_channel_close_channel ), true); +trace_vmbus_close_internal (msg, ret); +if (ret) {pr_err ("Close failed: close post msg return is %d\n" , ret);/*
drivers/hv/hv_trace.h +15 -0
@@ -157,6 +157,21 @@ TRACE_EVENT(vmbus_open, ) ); +TRACE_EVENT (vmbus_close_internal, + TP_PROTO(const struct vmbus_channel_close_channel *msg, int ret), +TP_ARGS (msg, ret), +TP_STRUCT__entry ( + __field(u32, child_relid) +__field (int, ret) + ), +TP_fast_assign ( + __entry->child_relid = msg->child_relid; + __entry->ret = ret; + ), +TP_printk ("sending child_relid 0x%x, ret %d" , __entry->child_relid, + __entry->ret) + ); +#undef TRACE_INCLUDE_PATH#define TRACE_INCLUDE_PATH .#undef TRACE_INCLUDE_FILE
Signature · 1 line
2.13.6
PATCH v3 12/17 hyper-v: trace vmbus_establish_gpadl()
In reply to Vitaly KuznetsovAdd tracepoint to CHANNELMSG_GPADL_HEADER/CHANNELMSG_GPADL_BODY sender.
--- drivers/hv/channel.c | 6 ++++++ drivers/hv/hv_trace.h | 42 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+)
drivers/hv/channel.c +6 -0
@@ -435,6 +435,9 @@ int vmbus_establish_gpadl(struct vmbus_channel *channel, void *kbuffer, ret =vmbus_post_msg (gpadlmsg, msginfo->msgsize -sizeof (*msginfo), true); + +trace_vmbus_establish_gpadl_header (gpadlmsg, ret); +if (ret !=0 ) goto cleanup;
@@ -450,6 +453,9 @@ int vmbus_establish_gpadl(struct vmbus_channel *channel, void *kbuffer, ret =vmbus_post_msg (gpadl_body, submsginfo->msgsize -sizeof (*submsginfo), true); + +trace_vmbus_establish_gpadl_body (gpadl_body, ret); +if (ret !=0 ) goto cleanup;
drivers/hv/hv_trace.h +42 -0
48 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH v3 13/17 hyper-v: trace vmbus_teardown_gpadl()
In reply to Vitaly KuznetsovAdd tracepoint to CHANNELMSG_GPADL_TEARDOWN sender.
--- drivers/hv/channel.c | 2 ++ drivers/hv/hv_trace.h | 18 ++++++++++++++++++ 2 files changed, 20 insertions(+)
drivers/hv/channel.c +2 -0
@@ -519,6 +519,8 @@ int vmbus_teardown_gpadl(struct vmbus_channel *channel, u32 gpadl_handle) ret =vmbus_post_msg (msg,sizeof (struct vmbus_channel_gpadl_teardown ), true); +trace_vmbus_teardown_gpadl (msg, ret); +if (ret) goto post_msg_err;
drivers/hv/hv_trace.h +18 -0
@@ -214,6 +214,24 @@ TRACE_EVENT(vmbus_establish_gpadl_body, ) ); +TRACE_EVENT (vmbus_teardown_gpadl, + TP_PROTO(const struct vmbus_channel_gpadl_teardown *msg, int ret), +TP_ARGS (msg, ret), +TP_STRUCT__entry ( + __field(u32, child_relid) + __field(u32, gpadl) +__field (int, ret) + ), +TP_fast_assign ( + __entry->child_relid = msg->child_relid; + __entry->gpadl = msg->gpadl; + __entry->ret = ret; + ), +TP_printk ("sending child_relid 0x%x, gpadl 0x%x, ret %d" , + __entry->child_relid, __entry->gpadl, __entry->ret + ) + ); +#undef TRACE_INCLUDE_PATH#define TRACE_INCLUDE_PATH .#undef TRACE_INCLUDE_FILE
Signature · 1 line
2.13.6
PATCH v3 14/17 hyper-v: trace vmbus_negotiate_version()
In reply to Vitaly KuznetsovAdd tracepoint to CHANNELMSG_INITIATE_CONTACT sender.
--- drivers/hv/connection.c | 3 +++ drivers/hv/hv_trace.h | 26 ++++++++++++++++++++++++++ 2 files changed, 29 insertions(+)
drivers/hv/connection.c +3 -0
@@ -117,6 +117,9 @@ static int vmbus_negotiate_version(struct vmbus_channel_msginfo *msginfo, ret =vmbus_post_msg (msg,sizeof (struct vmbus_channel_initiate_contact ), true); + +trace_vmbus_negotiate_version (msg, ret); +if (ret !=0 ) {spin_lock_irqsave (&vmbus_connection.channelmsg_lock, flags);list_del (&msginfo->msglistentry);
drivers/hv/hv_trace.h +26 -0
@@ -232,6 +232,32 @@ TRACE_EVENT(vmbus_teardown_gpadl, ) ); +TRACE_EVENT (vmbus_negotiate_version, + TP_PROTO(const struct vmbus_channel_initiate_contact *msg, int ret), +TP_ARGS (msg, ret), +TP_STRUCT__entry ( + __field(u32, ver) + __field(u32, target_vcpu) + __field(int, ret) + __field(u64, int_page) + __field(u64, mon_page1) +__field (u64, mon_page2) + ), +TP_fast_assign ( + __entry->ver = msg->vmbus_version_requested; + __entry->target_vcpu = msg->target_vcpu; + __entry->int_page = msg->interrupt_page; + __entry->mon_page1 = msg->monitor_page1; + __entry->mon_page2 = msg->monitor_page2; + __entry->ret = ret; + ), +TP_printk ("sending vmbus_version_requested %d, target_vcpu 0x%x, " +"pages %llx:%llx:%llx, ret %d" , + __entry->ver, __entry->target_vcpu, __entry->int_page, + __entry->mon_page1, __entry->mon_page2, __entry->ret + ) + ); +#undef TRACE_INCLUDE_PATH#define TRACE_INCLUDE_PATH .#undef TRACE_INCLUDE_FILE
Signature · 1 line
2.13.6
PATCH v3 15/17 hyper-v: trace vmbus_release_relid()
In reply to Vitaly KuznetsovAdd tracepoint to CHANNELMSG_RELID_RELEASED sender.
--- drivers/hv/channel_mgmt.c | 7 +++++-- drivers/hv/hv_trace.h | 16 ++++++++++++++++ 2 files changed, 21 insertions(+), 2 deletions(-)
drivers/hv/channel_mgmt.c +5 -2
@@ -373,12 +373,15 @@ static void percpu_channel_deq(void *arg)static void vmbus_release_relid (u32 relid) {struct vmbus_channel_relid_released msg; +int ret;memset (&msg,0 ,sizeof (struct vmbus_channel_relid_released )); msg.child_relid = relid; msg.header.msgtype =CHANNELMSG_RELID_RELEASED ; -vmbus_post_msg (&msg,sizeof (struct vmbus_channel_relid_released ), - true); + ret =vmbus_post_msg (&msg,sizeof (struct vmbus_channel_relid_released ), + true); + +trace_vmbus_release_relid (&msg, ret); }void hv_process_channel_removal (struct vmbus_channel *channel,u32 relid)
drivers/hv/hv_trace.h +16 -0
@@ -258,6 +258,22 @@ TRACE_EVENT(vmbus_negotiate_version, ) ); +TRACE_EVENT (vmbus_release_relid, + TP_PROTO(const struct vmbus_channel_relid_released *msg, int ret), +TP_ARGS (msg, ret), +TP_STRUCT__entry ( + __field(u32, child_relid) +__field (int, ret) + ), +TP_fast_assign ( + __entry->child_relid = msg->child_relid; + __entry->ret = ret; + ), +TP_printk ("sending child_relid 0x%x, ret %d" , + __entry->child_relid, __entry->ret + ) + ); +#undef TRACE_INCLUDE_PATH#define TRACE_INCLUDE_PATH .#undef TRACE_INCLUDE_FILE
Signature · 1 line
2.13.6
PATCH v3 16/17 hyper-v: trace vmbus_send_tl_connect_request()
In reply to Vitaly KuznetsovAdd tracepoint to CHANNELMSG_TL_CONNECT_REQUEST sender.
--- drivers/hv/channel.c | 7 ++++++- drivers/hv/hv_trace.h | 20 ++++++++++++++++++++ 2 files changed, 26 insertions(+), 1 deletion(-)
drivers/hv/channel.c +6 -1
@@ -236,13 +236,18 @@ int vmbus_send_tl_connect_request(const uuid_le *shv_guest_servie_id, const uuid_le *shv_host_servie_id) {struct vmbus_channel_tl_connect_request conn_msg; +int ret;memset (&conn_msg,0 ,sizeof (conn_msg)); conn_msg.header.msgtype =CHANNELMSG_TL_CONNECT_REQUEST ; conn_msg.guest_endpoint_id = *shv_guest_servie_id; conn_msg.host_service_id = *shv_host_servie_id; -return vmbus_post_msg (&conn_msg,sizeof (conn_msg), true); + ret =vmbus_post_msg (&conn_msg,sizeof (conn_msg), true); + +trace_vmbus_send_tl_connect_request (&conn_msg, ret); + +return ret; }EXPORT_SYMBOL_GPL (vmbus_send_tl_connect_request);
drivers/hv/hv_trace.h +20 -0
@@ -274,6 +274,26 @@ TRACE_EVENT(vmbus_release_relid, ) ); +TRACE_EVENT (vmbus_send_tl_connect_request, + TP_PROTO(const struct vmbus_channel_tl_connect_request *msg, + int ret), +TP_ARGS (msg, ret), +TP_STRUCT__entry ( + __array(char, guest_id, 16) + __array(char, host_id, 16) +__field (int, ret) + ), +TP_fast_assign ( + memcpy(__entry->guest_id, &msg->guest_endpoint_id.b, 16); + memcpy(__entry->host_id, &msg->host_service_id.b, 16); + __entry->ret = ret; + ), +TP_printk ("sending guest_endpoint_id %pUl, host_service_id %pUl, " +"ret %d" , + __entry->guest_id, __entry->host_id, __entry->ret + ) + ); +#undef TRACE_INCLUDE_PATH#define TRACE_INCLUDE_PATH .#undef TRACE_INCLUDE_FILE
Signature · 1 line
2.13.6
PATCH v3 17/17 hyper-v: trace channel events
In reply to Vitaly KuznetsovAdded an additional set of trace points for when channel gets notified or signals host.
--- drivers/hv/channel.c | 2 ++ drivers/hv/connection.c | 2 ++ drivers/hv/hv_trace.h | 23 +++++++++++++++++++++++ drivers/hv/vmbus_drv.c | 2 ++ 4 files changed, 29 insertions(+)
drivers/hv/channel.c +2 -0
@@ -43,6 +43,8 @@ void vmbus_setevent(struct vmbus_channel *channel) {struct hv_monitor_page *monitorpage; +trace_vmbus_setevent (channel); +/* * For channels marked as in "low latency" mode * bypass the monitor page mechanism.
drivers/hv/connection.c +2 -0
@@ -322,6 +322,8 @@ void vmbus_on_event(unsigned long data)struct vmbus_channel *channel = (void *) data;unsigned long time_limit = jiffies +2 ; +trace_vmbus_on_event (channel); + do { void (*callback_fn)(void *);
drivers/hv/hv_trace.h +23 -0
@@ -294,6 +294,29 @@ TRACE_EVENT(vmbus_send_tl_connect_request, ) ); +DECLARE_EVENT_CLASS (vmbus_channel, + TP_PROTO(const struct vmbus_channel *channel), +TP_ARGS (channel), +TP_STRUCT__entry (__field (u32, relid)), +TP_fast_assign (__entry->relid = channel->offermsg.child_relid), +TP_printk ("relid 0x%x" , __entry->relid) +); + +DEFINE_EVENT (vmbus_channel, vmbus_chan_sched, + TP_PROTO(const struct vmbus_channel *channel), +TP_ARGS (channel) +); + +DEFINE_EVENT (vmbus_channel, vmbus_setevent, + TP_PROTO(const struct vmbus_channel *channel), +TP_ARGS (channel) +); + +DEFINE_EVENT (vmbus_channel, vmbus_on_event, + TP_PROTO(const struct vmbus_channel *channel), +TP_ARGS (channel) +); +#undef TRACE_INCLUDE_PATH#define TRACE_INCLUDE_PATH .#undef TRACE_INCLUDE_FILE
drivers/hv/vmbus_drv.c +2 -0
@@ -945,6 +945,8 @@ static void vmbus_chan_sched(struct hv_per_cpu_context *hv_cpu)if (channel->rescind)continue ; +trace_vmbus_chan_sched (channel); +switch (channel->callback_mode) {case HV_CALL_ISR :vmbus_channel_isr (channel);
Signature · 1 line
2.13.6
Quoted 74 lines · On Thu, 5 Oct 2017 16:50:27 +0200 Vitaly Kuznetsov <[email protected]> wrote:
Changes since v2:
- UseDEFINE_EVENTin PATCH1/2 to avoid compile warnings [Stephen Hemminger]
- Add PATCH17 tracing channel events [Stephen Hemminger]Messages between guest and host are used in Hyper-V as control flow. To simplify debugging various issues which are often hard to reproduce add tracepoints to all message senders and handlers. This is not a performance critical path and tracing overhead should be negligible.
The example usage and output is:
Enable all tracing events:
# echo 1 > /sys/kernel/debug/tracing/events/hyperv/enableDo something which causes messages to be sent between host and guest, e.g. hot remove a VMBus device.
Check events:
# cat /sys/kernel/debug/tracing/trace# tracer: nop
#
# _-----=> irqs-off
# / _----=> need-resched# | / _---=> hardirq/softirq # || / _--=> preempt-depth # ||| / delay # TASK-PID CPU# |||| TIMESTAMP FUNCTION # | | | |||| | | <idle>-0 [011] ..s. 122.981583: vmbus_on_msg_dpc: msgtype=1 kworker/11:7-1506 [011] .... 122.981597: vmbus_on_message: msgtype=1 kworker/11:7-1506 [011] .... 122.981598: vmbus_onoffer: child_relid 0x10, monitorid 0x2, is_dedicated 1, connection_id 0x10010, if_type f8615163-df3e-46c5-913f-f2d2f965ed0e, if_instance 6676e078-e4b3-44da-8a7d-12eafb577d31, chn_flags 0x0, mmio_megabytes 0, sub_channel_index 0 kworker/11:7-1506 [011] .... 122.982130: vmbus_establish_gpadl_header: sending child_relid 0x10, gpadl 0xe1e34, range_buflen 2056 rangecount 1, ret 0 kworker/11:7-1506 [011] .... 122.982133: vmbus_establish_gpadl_body: sending msgnumber 0, gpadl 0xe1e34, ret 0 kworker/11:7-1506 [011] .... 122.982136: vmbus_establish_gpadl_body: sending msgnumber 0, gpadl 0xe1e34, ret 0 kworker/11:7-1506 [011] .... 122.982137: vmbus_establish_gpadl_body: sending msgnumber 0, gpadl 0xe1e34, ret 0 kworker/11:7-1506 [011] .... 122.982139: vmbus_establish_gpadl_body: sending msgnumber 0, gpadl 0xe1e34, ret 0 kworker/11:7-1506 [011] .... 122.982141: vmbus_establish_gpadl_body: sending msgnumber 0, gpadl 0xe1e34, ret 0 kworker/11:7-1506 [011] .... 122.982142: vmbus_establish_gpadl_body: sending msgnumber 0, gpadl 0xe1e34, ret 0 kworker/11:7-1506 [011] .... 122.982144: vmbus_establish_gpadl_body: sending msgnumber 0, gpadl 0xe1e34, ret 0 kworker/11:7-1506 [011] .... 122.982146: vmbus_establish_gpadl_body: sending msgnumber 0, gpadl 0xe1e34, ret 0 kworker/11:7-1506 [011] .... 122.982148: vmbus_establish_gpadl_body: sending msgnumber 0, gpadl 0xe1e34, ret 0 <idle>-0 [011] ..s. 122.982336: vmbus_on_msg_dpc: msgtype=10 <idle>-0 [011] ..s. 122.982337: vmbus_ongpadl_created: child_relid 0x10, gpadl 0xe1e34, creation_status 0 kworker/11:7-1506 [011] .... 122.982351: vmbus_open: sending child_relid 0x10, openid 16, gpadlhandle 0xe1e34, target_vp 0xb, offset 0x80, ret 0 kworker/3:1-214 [003] .... 123.015007: vmbus_setevent: relid 0x5 <idle>-0 [011] ..s. 123.029467: vmbus_on_msg_dpc: msgtype=6 <idle>-0 [011] ..s. 123.029470: vmbus_onopen_result: child_relid 0x10, openid 16, status 0 kworker/11:7-1506 [011] .... 123.029492: vmbus_setevent: relid 0x10 <idle>-0 [011] d.h. 123.029533: vmbus_chan_sched: relid 0x10 kworker/11:7-1506 [011] .... 123.029539: vmbus_setevent: relid 0x10CHANNELMSG_UNLOAD/CHANNELMSG_UNLOAD_RESPONSE are not traced as these are mostly used on crash.
Vitaly Kuznetsov (17):
hyper-v: tracevmbus_on_msg_dpc()
hyper-v: tracevmbus_on_message()
hyper-v: tracevmbus_onoffer()
hyper-v: tracevmbus_onoffer_rescind()
hyper-v: tracevmbus_onopen_result()
hyper-v: tracevmbus_ongpadl_created()
hyper-v: tracevmbus_ongpadl_torndown()
hyper-v: tracevmbus_onversion_response()
hyper-v: tracevmbus_request_offers()
hyper-v: tracevmbus_open()
hyper-v: tracevmbus_close_internal()
hyper-v: tracevmbus_establish_gpadl()
hyper-v: tracevmbus_teardown_gpadl()
hyper-v: tracevmbus_negotiate_version()
hyper-v: tracevmbus_release_relid()
hyper-v: tracevmbus_send_tl_connect_request()
hyper-v: trace channel events
Quoted 12 lines
drivers/hv/Makefile | 4 +- drivers/hv/channel.c | 21 ++- drivers/hv/channel_mgmt.c | 26 +++- drivers/hv/connection.c | 5 + drivers/hv/hv_trace.c | 4 + drivers/hv/hv_trace.h | 327 ++++++++++++++++++++++++++++++++++++++++++++++ drivers/hv/hyperv_vmbus.h | 2 + drivers/hv/vmbus_drv.c | 4 + 8 files changed, 388 insertions(+), 5 deletions(-) create mode 100644 drivers/hv/hv_trace.c create mode 100644 drivers/hv/hv_trace.h
Looks good.