kvm/vmx: Output TSC offset

Add a tracepoint write_tsc_offset for tracing TSC offset change. We want to merge ftrace's trace data of guest OSs and the host OS using TSC for timestamp in chronological order. We need "TSC offset" values for each guest when merge those because the TSC value on a guest is always the host TSC plus guest's TSC offset. If we get the TSC offset values, we can calculate the host TSC value for each guest events from the TSC offset and the event TSC value. The host TSC values of the guest events are used when we want to merge trace data of guests and the host in chronological order. (Note: the trace_clock of both the host and the guest must be set x86-tsc in this case)

TSC offset is stored in the VMCS by vmx_write_tsc_offset() or vmx_adjust_tsc_offset(). KVM executes the former function when a guest boots. The latter function is executed when kvm clock is updated. Only host can read TSC offset value from VMCS, so a host needs to output TSC offset value when TSC offset is changed.

Since the TSC offset is not often changed, it could be overwritten by other frequent events while tracing. To avoid that, I recommend to use a special instance for getting this event:

1. set a instance before booting a guest

 # cd /sys/kernel/debug/tracing/instances
 # mkdir tsc_offset
 # cd tsc_offset
 # echo x86-tsc > trace_clock
 # echo 1 > events/kvm/kvm_write_tsc_offset/enable

2. boot a guest

Signed-off-by Yoshihiro YUNOMAE <[email protected]>

Cc Marcelo Tosatti <[email protected]>

Cc Gleb Natapov <[email protected]>

Cc Thomas Gleixner <[email protected]>

Cc Ingo Molnar <[email protected]>

Cc "H. Peter Anvin" <[email protected]>

---
 arch/x86/kvm/trace.h |   18 ++++++++++++++++++
 arch/x86/kvm/vmx.c   |    3 +++
 arch/x86/kvm/x86.c   |    1 +
 3 files changed, 22 insertions(+)

arch/x86/kvm/trace.h +8 -0

index fe5e00e..9c22e39 100644
@@ -815,6 +815,24 @@ TRACE_EVENT(kvm_track_tsc,
 		  __print_symbolic(__entry->host_clock, host_clocks))
 );
 
+TRACE_EVENT(kvm_write_tsc_offset,
+	TP_PROTO(__u64 previous_tsc_offset, __u64 next_tsc_offset),
+	TP_ARGS(previous_tsc_offset, next_tsc_offset),
+
+	TP_STRUCT__entry(
+		__field(	__u64,	previous_tsc_offset		)
+		__field(	__u64,	next_tsc_offset			)
+	),

Quoted 58 lines · Yoshihiro YUNOMAE

Add a tracepoint write_tsc_offset for tracing TSC offset change. We want to merge ftrace's trace data of guest OSs and the host OS using TSC for timestamp in chronological order. We need "TSC offset" values for each guest when merge those because the TSC value on a guest is always the host TSC plus guest's TSC offset. If we get the TSC offset values, we can calculate the host TSC value for each guest events from the TSC offset and the event TSC value. The host TSC values of the guest events are used when we want to merge trace data of guests and the host in chronological order. (Note: the trace_clock of both the host and the guest must be set x86-tsc in this case)

TSC offset is stored in the VMCS by vmx_write_tsc_offset() or vmx_adjust_tsc_offset(). KVM executes the former function when a guest boots. The latter function is executed when kvm clock is updated. Only host can read TSC offset value from VMCS, so a host needs to output TSC offset value when TSC offset is changed.

Since the TSC offset is not often changed, it could be overwritten by other frequent events while tracing. To avoid that, I recommend to use a special instance for getting this event:

1. set a instance before booting a guest

 # cd /sys/kernel/debug/tracing/instances
 # mkdir tsc_offset
 # cd tsc_offset
 # echo x86-tsc > trace_clock
 # echo 1 > events/kvm/kvm_write_tsc_offset/enable

2. boot a guest

Signed-off-by Yoshihiro YUNOMAE <[email protected]>

Cc Marcelo Tosatti <[email protected]>

Cc Gleb Natapov <[email protected]>

Cc Thomas Gleixner <[email protected]>

Cc Ingo Molnar <[email protected]>

Cc "H. Peter Anvin" <[email protected]>

---
 arch/x86/kvm/trace.h |   18 ++++++++++++++++++
 arch/x86/kvm/vmx.c   |    3 +++
 arch/x86/kvm/x86.c   |    1 +
 3 files changed, 22 insertions(+)

arch/x86/kvm/trace.h +8 -0

index fe5e00e..9c22e39 100644
@@ -815,6 +815,24 @@ TRACE_EVENT(kvm_track_tsc,
 		  __print_symbolic(__entry->host_clock, host_clocks))
 );
 
+TRACE_EVENT(kvm_write_tsc_offset,
+	TP_PROTO(__u64 previous_tsc_offset, __u64 next_tsc_offset),
+	TP_ARGS(previous_tsc_offset, next_tsc_offset),
+
+	TP_STRUCT__entry(
+		__field(	__u64,	previous_tsc_offset		)
+		__field(	__u64,	next_tsc_offset			)
+	),

arch/x86/kvm/trace.h TRACE_EVENT(kvm_track_tsc, · line 835

Quoted 10 lines

arch/x86/kvm/trace.h +10 -0

+
+	TP_fast_assign(
+		__entry->previous_tsc_offset	= previous_tsc_offset;
+		__entry->next_tsc_offset	= next_tsc_offset;
+	),
+
+	TP_printk("previous=%llu next=%llu",
+		  __entry->previous_tsc_offset, __entry->next_tsc_offset)
+);
+

Yoshihiro YUNOMAE,

1) Why is previous_tsc_offset necessary?

2) The TSC offset traces should include vcpu number, so that its possible to correlate traces of SMP guests (the tool should use the individual vcpu tsc offsets when converting guests trace).

3) Please add traces for svm.c.

Quoted 72 lines · Marcelo Tosatti

Quoted 58 lines · Yoshihiro YUNOMAE

Add a tracepoint write_tsc_offset for tracing TSC offset change. We want to merge ftrace's trace data of guest OSs and the host OS using TSC for timestamp in chronological order. We need "TSC offset" values for each guest when merge those because the TSC value on a guest is always the host TSC plus guest's TSC offset. If we get the TSC offset values, we can calculate the host TSC value for each guest events from the TSC offset and the event TSC value. The host TSC values of the guest events are used when we want to merge trace data of guests and the host in chronological order. (Note: the trace_clock of both the host and the guest must be set x86-tsc in this case)

TSC offset is stored in the VMCS by vmx_write_tsc_offset() or vmx_adjust_tsc_offset(). KVM executes the former function when a guest boots. The latter function is executed when kvm clock is updated. Only host can read TSC offset value from VMCS, so a host needs to output TSC offset value when TSC offset is changed.

Since the TSC offset is not often changed, it could be overwritten by other frequent events while tracing. To avoid that, I recommend to use a special instance for getting this event:

1. set a instance before booting a guest

  # cd /sys/kernel/debug/tracing/instances
  # mkdir tsc_offset
  # cd tsc_offset
  # echo x86-tsc > trace_clock
  # echo 1 > events/kvm/kvm_write_tsc_offset/enable

2. boot a guest

Signed-off-by Yoshihiro YUNOMAE <[email protected]>

Cc Marcelo Tosatti <[email protected]>

Cc Gleb Natapov <[email protected]>

Cc Thomas Gleixner <[email protected]>

Cc Ingo Molnar <[email protected]>

Cc "H. Peter Anvin" <[email protected]>

---
  arch/x86/kvm/trace.h |   18 ++++++++++++++++++
  arch/x86/kvm/vmx.c   |    3 +++
  arch/x86/kvm/x86.c   |    1 +
  3 files changed, 22 insertions(+)

arch/x86/kvm/trace.h +8 -0

index fe5e00e..9c22e39 100644
@@ -815,6 +815,24 @@ TRACE_EVENT(kvm_track_tsc,
  		  __print_symbolic(__entry->host_clock, host_clocks))
  );
 
+TRACE_EVENT(kvm_write_tsc_offset,
+	TP_PROTO(__u64 previous_tsc_offset, __u64 next_tsc_offset),
+	TP_ARGS(previous_tsc_offset, next_tsc_offset),
+
+	TP_STRUCT__entry(
+		__field(	__u64,	previous_tsc_offset		)
+		__field(	__u64,	next_tsc_offset			)
+	),

arch/x86/kvm/trace.h TRACE_EVENT(kvm_track_tsc, · line 835

Quoted 10 lines

arch/x86/kvm/trace.h +10 -0

+
+	TP_fast_assign(
+		__entry->previous_tsc_offset	= previous_tsc_offset;
+		__entry->next_tsc_offset	= next_tsc_offset;
+	),
+
+	TP_printk("previous=%llu next=%llu",
+		  __entry->previous_tsc_offset, __entry->next_tsc_offset)
+);
+

Yoshihiro YUNOMAE,

Quoted 1 line

1) Why is previous_tsc_offset necessary?

Quoted 4 lines

2) The TSC offset traces should include vcpu number, so that its possible to correlate traces of SMP guests (the tool should use the individual vcpu tsc offsets when converting guests trace).

Why PID is not enough? No other trace, except kvm_entry, outputs vcpu id.

Quoted 1 line

3) Please add traces for svm.c.

Quoted 73 lines · Gleb Natapov

Quoted 72 lines · Marcelo Tosatti

Quoted 58 lines · Yoshihiro YUNOMAE

Add a tracepoint write_tsc_offset for tracing TSC offset change. We want to merge ftrace's trace data of guest OSs and the host OS using TSC for timestamp in chronological order. We need "TSC offset" values for each guest when merge those because the TSC value on a guest is always the host TSC plus guest's TSC offset. If we get the TSC offset values, we can calculate the host TSC value for each guest events from the TSC offset and the event TSC value. The host TSC values of the guest events are used when we want to merge trace data of guests and the host in chronological order. (Note: the trace_clock of both the host and the guest must be set x86-tsc in this case)

TSC offset is stored in the VMCS by vmx_write_tsc_offset() or vmx_adjust_tsc_offset(). KVM executes the former function when a guest boots. The latter function is executed when kvm clock is updated. Only host can read TSC offset value from VMCS, so a host needs to output TSC offset value when TSC offset is changed.

Since the TSC offset is not often changed, it could be overwritten by other frequent events while tracing. To avoid that, I recommend to use a special instance for getting this event:

1. set a instance before booting a guest

 # cd /sys/kernel/debug/tracing/instances
 # mkdir tsc_offset
 # cd tsc_offset
 # echo x86-tsc > trace_clock
 # echo 1 > events/kvm/kvm_write_tsc_offset/enable

2. boot a guest

Signed-off-by Yoshihiro YUNOMAE <[email protected]>

Cc Marcelo Tosatti <[email protected]>

Cc Gleb Natapov <[email protected]>

Cc Thomas Gleixner <[email protected]>

Cc Ingo Molnar <[email protected]>

Cc "H. Peter Anvin" <[email protected]>

---
 arch/x86/kvm/trace.h |   18 ++++++++++++++++++
 arch/x86/kvm/vmx.c   |    3 +++
 arch/x86/kvm/x86.c   |    1 +
 3 files changed, 22 insertions(+)

arch/x86/kvm/trace.h +8 -0

index fe5e00e..9c22e39 100644
@@ -815,6 +815,24 @@ TRACE_EVENT(kvm_track_tsc,
 		  __print_symbolic(__entry->host_clock, host_clocks))
 );
 
+TRACE_EVENT(kvm_write_tsc_offset,
+	TP_PROTO(__u64 previous_tsc_offset, __u64 next_tsc_offset),
+	TP_ARGS(previous_tsc_offset, next_tsc_offset),
+
+	TP_STRUCT__entry(
+		__field(	__u64,	previous_tsc_offset		)
+		__field(	__u64,	next_tsc_offset			)
+	),

arch/x86/kvm/trace.h TRACE_EVENT(kvm_track_tsc, · line 835

Quoted 10 lines

arch/x86/kvm/trace.h +10 -0

+
+	TP_fast_assign(
+		__entry->previous_tsc_offset	= previous_tsc_offset;
+		__entry->next_tsc_offset	= next_tsc_offset;
+	),
+
+	TP_printk("previous=%llu next=%llu",
+		  __entry->previous_tsc_offset, __entry->next_tsc_offset)
+);
+

Yoshihiro YUNOMAE,

Quoted 1 line

Quoted 1 line

1) Why is previous_tsc_offset necessary?

I was considering the situations where we did not enable kvm_write_tsc_offset event before booting a guest or where we did not use multiple buffers. Here, we will need another new I/F to get current TSC offset of a given VCPU. For example, if kvm_write_tsc_offset is not included in the host's trace data, we get the current TSC offset from the new I/F and apply it to all guest events. On the other hand, if kvm_write_tsc_offset event appears more than once, we apply the previous offset to guest events before the first TSC offset change.

Since we support only for using multiple buffers now, we don't need to record previous TSC offset at this time. But I'm conscious that we have to change the format of kvm_write_tsc_offset event when we support those situations.

Quoted 5 lines

Quoted 4 lines

2) The TSC offset traces should include vcpu number, so that its possible to correlate traces of SMP guests (the tool should use the individual vcpu tsc offsets when converting guests trace).

Why PID is not enough? No other trace, except kvm_entry, outputs vcpu id.

As Gleb mentioned, a tool can understand TSC offset for each vcpu from PID and vcpu number of kvm_entry. IMO, that is indirect way, so I would be better off including vcpu number.

Quoted 1 line

Quoted 1 line

3) Please add traces for svm.c.

Sure, I'll add the tracepoint for SVM.

Quoted 73 lines · Gleb Natapov

Quoted 72 lines · Marcelo Tosatti

Quoted 58 lines · Yoshihiro YUNOMAE

Add a tracepoint write_tsc_offset for tracing TSC offset change. We want to merge ftrace's trace data of guest OSs and the host OS using TSC for timestamp in chronological order. We need "TSC offset" values for each guest when merge those because the TSC value on a guest is always the host TSC plus guest's TSC offset. If we get the TSC offset values, we can calculate the host TSC value for each guest events from the TSC offset and the event TSC value. The host TSC values of the guest events are used when we want to merge trace data of guests and the host in chronological order. (Note: the trace_clock of both the host and the guest must be set x86-tsc in this case)

TSC offset is stored in the VMCS by vmx_write_tsc_offset() or vmx_adjust_tsc_offset(). KVM executes the former function when a guest boots. The latter function is executed when kvm clock is updated. Only host can read TSC offset value from VMCS, so a host needs to output TSC offset value when TSC offset is changed.

Since the TSC offset is not often changed, it could be overwritten by other frequent events while tracing. To avoid that, I recommend to use a special instance for getting this event:

1. set a instance before booting a guest

 # cd /sys/kernel/debug/tracing/instances
 # mkdir tsc_offset
 # cd tsc_offset
 # echo x86-tsc > trace_clock
 # echo 1 > events/kvm/kvm_write_tsc_offset/enable

2. boot a guest

Signed-off-by Yoshihiro YUNOMAE <[email protected]>

Cc Marcelo Tosatti <[email protected]>

Cc Gleb Natapov <[email protected]>

Cc Thomas Gleixner <[email protected]>

Cc Ingo Molnar <[email protected]>

Cc "H. Peter Anvin" <[email protected]>

---
 arch/x86/kvm/trace.h |   18 ++++++++++++++++++
 arch/x86/kvm/vmx.c   |    3 +++
 arch/x86/kvm/x86.c   |    1 +
 3 files changed, 22 insertions(+)

arch/x86/kvm/trace.h +8 -0

index fe5e00e..9c22e39 100644
@@ -815,6 +815,24 @@ TRACE_EVENT(kvm_track_tsc,
 		  __print_symbolic(__entry->host_clock, host_clocks))
 );
 
+TRACE_EVENT(kvm_write_tsc_offset,
+	TP_PROTO(__u64 previous_tsc_offset, __u64 next_tsc_offset),
+	TP_ARGS(previous_tsc_offset, next_tsc_offset),
+
+	TP_STRUCT__entry(
+		__field(	__u64,	previous_tsc_offset		)
+		__field(	__u64,	next_tsc_offset			)
+	),

arch/x86/kvm/trace.h TRACE_EVENT(kvm_track_tsc, · line 835

Quoted 10 lines

arch/x86/kvm/trace.h +10 -0

+
+	TP_fast_assign(
+		__entry->previous_tsc_offset	= previous_tsc_offset;
+		__entry->next_tsc_offset	= next_tsc_offset;
+	),
+
+	TP_printk("previous=%llu next=%llu",
+		  __entry->previous_tsc_offset, __entry->next_tsc_offset)
+);
+

Yoshihiro YUNOMAE,

Quoted 1 line

Quoted 1 line

1) Why is previous_tsc_offset necessary?

I was considering the situations where we did not enable kvm_write_tsc_offset event before booting a guest or where we did not use multiple buffers. Here, we will need another new I/F to get current TSC offset of a given VCPU. For example, if kvm_write_tsc_offset is not included in the host's trace data, we get the current TSC offset from the new I/F and apply it to all guest events. On the other hand, if kvm_write_tsc_offset event appears more than once, we apply the previous offset to guest events before the first TSC offset change.

Since we support only for using multiple buffers now, we don't need to record previous TSC offset at this time. But I'm conscious that we have to change the format of kvm_write_tsc_offset event when we support those situations.

Quoted 4 lines

Quoted 4 lines

2) The TSC offset traces should include vcpu number, so that its possible to correlate traces of SMP guests (the tool should use the individual vcpu tsc offsets when converting guests trace).

Quoted 1 line

Why PID is not enough? No other trace, except kvm_entry, outputs vcpu id.

As Gleb mentioned, a tool can understand TSC offset for each vcpu from PID and vcpu number of kvm_entry. IMO, that is indirect way, so I would be better off including vcpu number.

Quoted 1 line

Quoted 1 line

3) Please add traces for svm.c.

Sure, I'll add the tracepoint for SVM.

Thanks,

Yoshihiro YUNOMAE

Signature · 4 lines

Yoshihiro YUNOMAE
Software Platform Research Dept. Linux Technology Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]

Quoted 96 lines · Yoshihiro YUNOMAE

Quoted 73 lines · Gleb Natapov

Quoted 72 lines · Marcelo Tosatti

Quoted 58 lines · Yoshihiro YUNOMAE

Add a tracepoint write_tsc_offset for tracing TSC offset change. We want to merge ftrace's trace data of guest OSs and the host OS using TSC for timestamp in chronological order. We need "TSC offset" values for each guest when merge those because the TSC value on a guest is always the host TSC plus guest's TSC offset. If we get the TSC offset values, we can calculate the host TSC value for each guest events from the TSC offset and the event TSC value. The host TSC values of the guest events are used when we want to merge trace data of guests and the host in chronological order. (Note: the trace_clock of both the host and the guest must be set x86-tsc in this case)

TSC offset is stored in the VMCS by vmx_write_tsc_offset() or vmx_adjust_tsc_offset(). KVM executes the former function when a guest boots. The latter function is executed when kvm clock is updated. Only host can read TSC offset value from VMCS, so a host needs to output TSC offset value when TSC offset is changed.

Since the TSC offset is not often changed, it could be overwritten by other frequent events while tracing. To avoid that, I recommend to use a special instance for getting this event:

1. set a instance before booting a guest

  # cd /sys/kernel/debug/tracing/instances
  # mkdir tsc_offset
  # cd tsc_offset
  # echo x86-tsc > trace_clock
  # echo 1 > events/kvm/kvm_write_tsc_offset/enable

2. boot a guest

Signed-off-by Yoshihiro YUNOMAE <[email protected]>

Cc Marcelo Tosatti <[email protected]>

Cc Gleb Natapov <[email protected]>

Cc Thomas Gleixner <[email protected]>

Cc Ingo Molnar <[email protected]>

Cc "H. Peter Anvin" <[email protected]>

---
  arch/x86/kvm/trace.h |   18 ++++++++++++++++++
  arch/x86/kvm/vmx.c   |    3 +++
  arch/x86/kvm/x86.c   |    1 +
  3 files changed, 22 insertions(+)

arch/x86/kvm/trace.h +8 -0

index fe5e00e..9c22e39 100644
@@ -815,6 +815,24 @@ TRACE_EVENT(kvm_track_tsc,
  		  __print_symbolic(__entry->host_clock, host_clocks))
  );
 
+TRACE_EVENT(kvm_write_tsc_offset,
+	TP_PROTO(__u64 previous_tsc_offset, __u64 next_tsc_offset),
+	TP_ARGS(previous_tsc_offset, next_tsc_offset),
+
+	TP_STRUCT__entry(
+		__field(	__u64,	previous_tsc_offset		)
+		__field(	__u64,	next_tsc_offset			)
+	),

arch/x86/kvm/trace.h TRACE_EVENT(kvm_track_tsc, · line 835

Quoted 10 lines

arch/x86/kvm/trace.h +10 -0

+
+	TP_fast_assign(
+		__entry->previous_tsc_offset	= previous_tsc_offset;
+		__entry->next_tsc_offset	= next_tsc_offset;
+	),
+
+	TP_printk("previous=%llu next=%llu",
+		  __entry->previous_tsc_offset, __entry->next_tsc_offset)
+);
+

Yoshihiro YUNOMAE,

Quoted 1 line

Quoted 1 line

1) Why is previous_tsc_offset necessary?

I was considering the situations where we did not enable kvm_write_tsc_offset event before booting a guest or where we did not use multiple buffers. Here, we will need another new I/F to get current TSC offset of a given VCPU. For example, if kvm_write_tsc_offset is not included in the host's trace data, we get the current TSC offset from the new I/F and apply it to all guest events. On the other hand, if kvm_write_tsc_offset event appears more than once, we apply the previous offset to guest events before the first TSC offset change.

Since we support only for using multiple buffers now, we don't need to record previous TSC offset at this time. But I'm conscious that we have to change the format of kvm_write_tsc_offset event when we support those situations.

Quoted 5 lines

Quoted 4 lines

2) The TSC offset traces should include vcpu number, so that its possible to correlate traces of SMP guests (the tool should use the individual vcpu tsc offsets when converting guests trace).

Why PID is not enough? No other trace, except kvm_entry, outputs vcpu id.

Quoted 4 lines

As Gleb mentioned, a tool can understand TSC offset for each vcpu from PID and vcpu number of kvm_entry. IMO, that is indirect way, so I would be better off including vcpu number.

But doesn't the tool operates on vcpu's PID for all other events. I mean to figure out what vcpu an event belongs too during merge. Why tsc offset event is different?

Hi Gleb,

Quoted 101 lines · Gleb Natapov

Quoted 100 lines · Yoshihiro YUNOMAE

Quoted 73 lines · Gleb Natapov

Quoted 72 lines · Marcelo Tosatti

Quoted 58 lines · Yoshihiro YUNOMAE

Add a tracepoint write_tsc_offset for tracing TSC offset change. We want to merge ftrace's trace data of guest OSs and the host OS using TSC for timestamp in chronological order. We need "TSC offset" values for each guest when merge those because the TSC value on a guest is always the host TSC plus guest's TSC offset. If we get the TSC offset values, we can calculate the host TSC value for each guest events from the TSC offset and the event TSC value. The host TSC values of the guest events are used when we want to merge trace data of guests and the host in chronological order. (Note: the trace_clock of both the host and the guest must be set x86-tsc in this case)

TSC offset is stored in the VMCS by vmx_write_tsc_offset() or vmx_adjust_tsc_offset(). KVM executes the former function when a guest boots. The latter function is executed when kvm clock is updated. Only host can read TSC offset value from VMCS, so a host needs to output TSC offset value when TSC offset is changed.

Since the TSC offset is not often changed, it could be overwritten by other frequent events while tracing. To avoid that, I recommend to use a special instance for getting this event:

1. set a instance before booting a guest

 # cd /sys/kernel/debug/tracing/instances
 # mkdir tsc_offset
 # cd tsc_offset
 # echo x86-tsc > trace_clock
 # echo 1 > events/kvm/kvm_write_tsc_offset/enable

2. boot a guest

Signed-off-by Yoshihiro YUNOMAE <[email protected]>

Cc Marcelo Tosatti <[email protected]>

Cc Gleb Natapov <[email protected]>

Cc Thomas Gleixner <[email protected]>

Cc Ingo Molnar <[email protected]>

Cc "H. Peter Anvin" <[email protected]>

---
 arch/x86/kvm/trace.h |   18 ++++++++++++++++++
 arch/x86/kvm/vmx.c   |    3 +++
 arch/x86/kvm/x86.c   |    1 +
 3 files changed, 22 insertions(+)

arch/x86/kvm/trace.h +8 -0

index fe5e00e..9c22e39 100644
@@ -815,6 +815,24 @@ TRACE_EVENT(kvm_track_tsc,
 		  __print_symbolic(__entry->host_clock, host_clocks))
 );
 
+TRACE_EVENT(kvm_write_tsc_offset,
+	TP_PROTO(__u64 previous_tsc_offset, __u64 next_tsc_offset),
+	TP_ARGS(previous_tsc_offset, next_tsc_offset),
+
+	TP_STRUCT__entry(
+		__field(	__u64,	previous_tsc_offset		)
+		__field(	__u64,	next_tsc_offset			)
+	),

arch/x86/kvm/trace.h TRACE_EVENT(kvm_track_tsc, · line 835

Quoted 10 lines

arch/x86/kvm/trace.h +10 -0

+
+	TP_fast_assign(
+		__entry->previous_tsc_offset	= previous_tsc_offset;
+		__entry->next_tsc_offset	= next_tsc_offset;
+	),
+
+	TP_printk("previous=%llu next=%llu",
+		  __entry->previous_tsc_offset, __entry->next_tsc_offset)
+);
+

Yoshihiro YUNOMAE,

Quoted 1 line

Quoted 1 line

1) Why is previous_tsc_offset necessary?

I was considering the situations where we did not enable kvm_write_tsc_offset event before booting a guest or where we did not use multiple buffers. Here, we will need another new I/F to get current TSC offset of a given VCPU. For example, if kvm_write_tsc_offset is not included in the host's trace data, we get the current TSC offset from the new I/F and apply it to all guest events. On the other hand, if kvm_write_tsc_offset event appears more than once, we apply the previous offset to guest events before the first TSC offset change.

Since we support only for using multiple buffers now, we don't need to record previous TSC offset at this time. But I'm conscious that we have to change the format of kvm_write_tsc_offset event when we support those situations.

Quoted 5 lines

Quoted 4 lines

2) The TSC offset traces should include vcpu number, so that its possible to correlate traces of SMP guests (the tool should use the individual vcpu tsc offsets when converting guests trace).

Why PID is not enough? No other trace, except kvm_entry, outputs vcpu id.

As Gleb mentioned, a tool can understand TSC offset for each vcpu from PID and vcpu number of kvm_entry. IMO, that is indirect way, so I would be better off including vcpu number.

Quoted 3 lines

But doesn't the tool operates on vcpu's PID for all other events. I mean to figure out what vcpu an event belongs too during merge. Why tsc offset event is different?

In vcpu_load()@virt/kvm/kvm_main.c, it seems that PID of the vcpu thread can be changed. Are you familiar with this situation?

If the situation can be occurred, outputting vcpu number is better, I think. If not occurred, as you say, we will be able to merge those data without vcpu number in write_tsc_offset event.

However, when we
focus on output data of the write_tsc_offset event, it is difficult to directly understand contents of the data if vcpu number information is not included. So, including the information is useful, I think.

Quoted 108 lines · Yoshihiro YUNOMAE

Hi Gleb,

Quoted 104 lines · Gleb Natapov

Quoted 94 lines · Yoshihiro YUNOMAE

Quoted 73 lines · Gleb Natapov

Quoted 72 lines · Marcelo Tosatti

Quoted 58 lines · Yoshihiro YUNOMAE

Add a tracepoint write_tsc_offset for tracing TSC offset change. We want to merge ftrace's trace data of guest OSs and the host OS using TSC for timestamp in chronological order. We need "TSC offset" values for each guest when merge those because the TSC value on a guest is always the host TSC plus guest's TSC offset. If we get the TSC offset values, we can calculate the host TSC value for each guest events from the TSC offset and the event TSC value. The host TSC values of the guest events are used when we want to merge trace data of guests and the host in chronological order. (Note: the trace_clock of both the host and the guest must be set x86-tsc in this case)

TSC offset is stored in the VMCS by vmx_write_tsc_offset() or vmx_adjust_tsc_offset(). KVM executes the former function when a guest boots. The latter function is executed when kvm clock is updated. Only host can read TSC offset value from VMCS, so a host needs to output TSC offset value when TSC offset is changed.

Since the TSC offset is not often changed, it could be overwritten by other frequent events while tracing. To avoid that, I recommend to use a special instance for getting this event:

1. set a instance before booting a guest

  # cd /sys/kernel/debug/tracing/instances
  # mkdir tsc_offset
  # cd tsc_offset
  # echo x86-tsc > trace_clock
  # echo 1 > events/kvm/kvm_write_tsc_offset/enable

2. boot a guest

Signed-off-by Yoshihiro YUNOMAE <[email protected]>

Cc Marcelo Tosatti <[email protected]>

Cc Gleb Natapov <[email protected]>

Cc Thomas Gleixner <[email protected]>

Cc Ingo Molnar <[email protected]>

Cc "H. Peter Anvin" <[email protected]>

---
  arch/x86/kvm/trace.h |   18 ++++++++++++++++++
  arch/x86/kvm/vmx.c   |    3 +++
  arch/x86/kvm/x86.c   |    1 +
  3 files changed, 22 insertions(+)

arch/x86/kvm/trace.h +8 -0

index fe5e00e..9c22e39 100644
@@ -815,6 +815,24 @@ TRACE_EVENT(kvm_track_tsc,
  		  __print_symbolic(__entry->host_clock, host_clocks))
  );
 
+TRACE_EVENT(kvm_write_tsc_offset,
+	TP_PROTO(__u64 previous_tsc_offset, __u64 next_tsc_offset),
+	TP_ARGS(previous_tsc_offset, next_tsc_offset),
+
+	TP_STRUCT__entry(
+		__field(	__u64,	previous_tsc_offset		)
+		__field(	__u64,	next_tsc_offset			)
+	),

arch/x86/kvm/trace.h TRACE_EVENT(kvm_track_tsc, · line 835

Quoted 10 lines

arch/x86/kvm/trace.h +10 -0

+
+	TP_fast_assign(
+		__entry->previous_tsc_offset	= previous_tsc_offset;
+		__entry->next_tsc_offset	= next_tsc_offset;
+	),
+
+	TP_printk("previous=%llu next=%llu",
+		  __entry->previous_tsc_offset, __entry->next_tsc_offset)
+);
+

Yoshihiro YUNOMAE,

Quoted 1 line

Quoted 1 line

1) Why is previous_tsc_offset necessary?

I was considering the situations where we did not enable kvm_write_tsc_offset event before booting a guest or where we did not use multiple buffers. Here, we will need another new I/F to get current TSC offset of a given VCPU. For example, if kvm_write_tsc_offset is not included in the host's trace data, we get the current TSC offset from the new I/F and apply it to all guest events. On the other hand, if kvm_write_tsc_offset event appears more than once, we apply the previous offset to guest events before the first TSC offset change.

Since we support only for using multiple buffers now, we don't need to record previous TSC offset at this time. But I'm conscious that we have to change the format of kvm_write_tsc_offset event when we support those situations.

Quoted 4 lines

Quoted 4 lines

2) The TSC offset traces should include vcpu number, so that its possible to correlate traces of SMP guests (the tool should use the individual vcpu tsc offsets when converting guests trace).

Quoted 6 lines

Quoted 1 line

Why PID is not enough? No other trace, except kvm_entry, outputs vcpu id.

As Gleb mentioned, a tool can understand TSC offset for each vcpu from PID and vcpu number of kvm_entry. IMO, that is indirect way, so I would be better off including vcpu number.

But doesn't the tool operates on vcpu's PID for all other events. I mean to figure out what vcpu an event belongs too during merge. Why tsc offset event is different?

Quoted 2 lines

In vcpu_load()@virt/kvm/kvm_main.c, it seems that PID of the vcpu thread can be changed. Are you familiar with this situation?

Recommended way of using KVM API is to have dedicated thread per vcpu and this is how all known userspace implementations use it, but having one thread drive several vcpus (not simultaneously obviously) also works, but not recommended.

Quoted 3 lines

If the situation can be occurred, outputting vcpu number is better, I think. If not occurred, as you say, we will be able to merge those data without vcpu number in write_tsc_offset event.

The thing is that all other traces that you want to merge do not contain vcpu number, only pid, so if the situation occurs how do you merge the data?

Quoted 5 lines

However, when we
focus on output data of the write_tsc_offset event, it is difficult to directly understand contents of the data if vcpu number information is not included. So, including the information is useful, I think.

How your tool does it now?

Quoted 104 lines · Yoshihiro YUNOMAE

Hi Gleb,

Quoted 101 lines · Gleb Natapov

Quoted 100 lines · Yoshihiro YUNOMAE

Quoted 73 lines · Gleb Natapov

Quoted 72 lines · Marcelo Tosatti

Quoted 58 lines · Yoshihiro YUNOMAE

Add a tracepoint write_tsc_offset for tracing TSC offset change. We want to merge ftrace's trace data of guest OSs and the host OS using TSC for timestamp in chronological order. We need "TSC offset" values for each guest when merge those because the TSC value on a guest is always the host TSC plus guest's TSC offset. If we get the TSC offset values, we can calculate the host TSC value for each guest events from the TSC offset and the event TSC value. The host TSC values of the guest events are used when we want to merge trace data of guests and the host in chronological order. (Note: the trace_clock of both the host and the guest must be set x86-tsc in this case)

TSC offset is stored in the VMCS by vmx_write_tsc_offset() or vmx_adjust_tsc_offset(). KVM executes the former function when a guest boots. The latter function is executed when kvm clock is updated. Only host can read TSC offset value from VMCS, so a host needs to output TSC offset value when TSC offset is changed.

Since the TSC offset is not often changed, it could be overwritten by other frequent events while tracing. To avoid that, I recommend to use a special instance for getting this event:

1. set a instance before booting a guest

 # cd /sys/kernel/debug/tracing/instances
 # mkdir tsc_offset
 # cd tsc_offset
 # echo x86-tsc > trace_clock
 # echo 1 > events/kvm/kvm_write_tsc_offset/enable

2. boot a guest

Signed-off-by Yoshihiro YUNOMAE <[email protected]>

Cc Marcelo Tosatti <[email protected]>

Cc Gleb Natapov <[email protected]>

Cc Thomas Gleixner <[email protected]>

Cc Ingo Molnar <[email protected]>

Cc "H. Peter Anvin" <[email protected]>

---
 arch/x86/kvm/trace.h |   18 ++++++++++++++++++
 arch/x86/kvm/vmx.c   |    3 +++
 arch/x86/kvm/x86.c   |    1 +
 3 files changed, 22 insertions(+)

arch/x86/kvm/trace.h +8 -0

index fe5e00e..9c22e39 100644
@@ -815,6 +815,24 @@ TRACE_EVENT(kvm_track_tsc,
 		  __print_symbolic(__entry->host_clock, host_clocks))
 );
 
+TRACE_EVENT(kvm_write_tsc_offset,
+	TP_PROTO(__u64 previous_tsc_offset, __u64 next_tsc_offset),
+	TP_ARGS(previous_tsc_offset, next_tsc_offset),
+
+	TP_STRUCT__entry(
+		__field(	__u64,	previous_tsc_offset		)
+		__field(	__u64,	next_tsc_offset			)
+	),

arch/x86/kvm/trace.h TRACE_EVENT(kvm_track_tsc, · line 835

Quoted 10 lines

arch/x86/kvm/trace.h +10 -0

+
+	TP_fast_assign(
+		__entry->previous_tsc_offset	= previous_tsc_offset;
+		__entry->next_tsc_offset	= next_tsc_offset;
+	),
+
+	TP_printk("previous=%llu next=%llu",
+		  __entry->previous_tsc_offset, __entry->next_tsc_offset)
+);
+

Yoshihiro YUNOMAE,

Quoted 1 line

Quoted 1 line

1) Why is previous_tsc_offset necessary?

I was considering the situations where we did not enable kvm_write_tsc_offset event before booting a guest or where we did not use multiple buffers. Here, we will need another new I/F to get current TSC offset of a given VCPU. For example, if kvm_write_tsc_offset is not included in the host's trace data, we get the current TSC offset from the new I/F and apply it to all guest events. On the other hand, if kvm_write_tsc_offset event appears more than once, we apply the previous offset to guest events before the first TSC offset change.

Since we support only for using multiple buffers now, we don't need to record previous TSC offset at this time. But I'm conscious that we have to change the format of kvm_write_tsc_offset event when we support those situations.

Quoted 5 lines

Quoted 4 lines

2) The TSC offset traces should include vcpu number, so that its possible to correlate traces of SMP guests (the tool should use the individual vcpu tsc offsets when converting guests trace).

Why PID is not enough? No other trace, except kvm_entry, outputs vcpu id.

As Gleb mentioned, a tool can understand TSC offset for each vcpu from PID and vcpu number of kvm_entry. IMO, that is indirect way, so I would be better off including vcpu number.

Quoted 6 lines

Quoted 3 lines

But doesn't the tool operates on vcpu's PID for all other events. I mean to figure out what vcpu an event belongs too during merge. Why tsc offset event is different?

In vcpu_load()@virt/kvm/kvm_main.c, it seems that PID of the vcpu thread can be changed. Are you familiar with this situation?

Recommended way of using KVM API is to have dedicated thread per vcpu and this is how all known userspace implementations use it, but having one thread drive several vcpus (not simultaneously obviously) also works, but not recommended.

Quoted 3 lines

If the situation can be occurred, outputting vcpu number is better, I think. If not occurred, as you say, we will be able to merge those data without vcpu number in write_tsc_offset event.

The thing is that all other traces that you want to merge do not contain vcpu number, only pid, so if the situation occurs how do you merge the data?

Quoted 5 lines

However, when we
focus on output data of the write_tsc_offset event, it is difficult to directly understand contents of the data if vcpu number information is not included. So, including the information is useful, I think.

How your tool does it now?

Quoted 119 lines · Gleb Natapov

Quoted 104 lines · Yoshihiro YUNOMAE

Hi Gleb,

Quoted 101 lines · Gleb Natapov

Quoted 100 lines · Yoshihiro YUNOMAE

Quoted 73 lines · Gleb Natapov

Quoted 72 lines · Marcelo Tosatti

Quoted 58 lines · Yoshihiro YUNOMAE

Add a tracepoint write_tsc_offset for tracing TSC offset change. We want to merge ftrace's trace data of guest OSs and the host OS using TSC for timestamp in chronological order. We need "TSC offset" values for each guest when merge those because the TSC value on a guest is always the host TSC plus guest's TSC offset. If we get the TSC offset values, we can calculate the host TSC value for each guest events from the TSC offset and the event TSC value. The host TSC values of the guest events are used when we want to merge trace data of guests and the host in chronological order. (Note: the trace_clock of both the host and the guest must be set x86-tsc in this case)

TSC offset is stored in the VMCS by vmx_write_tsc_offset() or vmx_adjust_tsc_offset(). KVM executes the former function when a guest boots. The latter function is executed when kvm clock is updated. Only host can read TSC offset value from VMCS, so a host needs to output TSC offset value when TSC offset is changed.

Since the TSC offset is not often changed, it could be overwritten by other frequent events while tracing. To avoid that, I recommend to use a special instance for getting this event:

1. set a instance before booting a guest

 # cd /sys/kernel/debug/tracing/instances
 # mkdir tsc_offset
 # cd tsc_offset
 # echo x86-tsc > trace_clock
 # echo 1 > events/kvm/kvm_write_tsc_offset/enable

2. boot a guest

Signed-off-by Yoshihiro YUNOMAE <[email protected]>

Cc Marcelo Tosatti <[email protected]>

Cc Gleb Natapov <[email protected]>

Cc Thomas Gleixner <[email protected]>

Cc Ingo Molnar <[email protected]>

Cc "H. Peter Anvin" <[email protected]>

---
 arch/x86/kvm/trace.h |   18 ++++++++++++++++++
 arch/x86/kvm/vmx.c   |    3 +++
 arch/x86/kvm/x86.c   |    1 +
 3 files changed, 22 insertions(+)

arch/x86/kvm/trace.h +8 -0

index fe5e00e..9c22e39 100644
@@ -815,6 +815,24 @@ TRACE_EVENT(kvm_track_tsc,
 		  __print_symbolic(__entry->host_clock, host_clocks))
 );
 
+TRACE_EVENT(kvm_write_tsc_offset,
+	TP_PROTO(__u64 previous_tsc_offset, __u64 next_tsc_offset),
+	TP_ARGS(previous_tsc_offset, next_tsc_offset),
+
+	TP_STRUCT__entry(
+		__field(	__u64,	previous_tsc_offset		)
+		__field(	__u64,	next_tsc_offset			)
+	),

arch/x86/kvm/trace.h TRACE_EVENT(kvm_track_tsc, · line 835

Quoted 10 lines

arch/x86/kvm/trace.h +10 -0

+
+	TP_fast_assign(
+		__entry->previous_tsc_offset	= previous_tsc_offset;
+		__entry->next_tsc_offset	= next_tsc_offset;
+	),
+
+	TP_printk("previous=%llu next=%llu",
+		  __entry->previous_tsc_offset, __entry->next_tsc_offset)
+);
+

Yoshihiro YUNOMAE,

Quoted 1 line

Quoted 1 line

1) Why is previous_tsc_offset necessary?

I was considering the situations where we did not enable kvm_write_tsc_offset event before booting a guest or where we did not use multiple buffers. Here, we will need another new I/F to get current TSC offset of a given VCPU. For example, if kvm_write_tsc_offset is not included in the host's trace data, we get the current TSC offset from the new I/F and apply it to all guest events. On the other hand, if kvm_write_tsc_offset event appears more than once, we apply the previous offset to guest events before the first TSC offset change.

Since we support only for using multiple buffers now, we don't need to record previous TSC offset at this time. But I'm conscious that we have to change the format of kvm_write_tsc_offset event when we support those situations.

Quoted 5 lines

Quoted 4 lines

2) The TSC offset traces should include vcpu number, so that its possible to correlate traces of SMP guests (the tool should use the individual vcpu tsc offsets when converting guests trace).

Why PID is not enough? No other trace, except kvm_entry, outputs vcpu id.

As Gleb mentioned, a tool can understand TSC offset for each vcpu from PID and vcpu number of kvm_entry. IMO, that is indirect way, so I would be better off including vcpu number.

Quoted 6 lines

Quoted 3 lines

But doesn't the tool operates on vcpu's PID for all other events. I mean to figure out what vcpu an event belongs too during merge. Why tsc offset event is different?

In vcpu_load()@virt/kvm/kvm_main.c, it seems that PID of the vcpu thread can be changed. Are you familiar with this situation?

Recommended way of using KVM API is to have dedicated thread per vcpu and this is how all known userspace implementations use it, but having one thread drive several vcpus (not simultaneously obviously) also works, but not recommended.

Quoted 3 lines

If the situation can be occurred, outputting vcpu number is better, I think. If not occurred, as you say, we will be able to merge those data without vcpu number in write_tsc_offset event.

Quoted 3 lines

The thing is that all other traces that you want to merge do not contain vcpu number, only pid, so if the situation occurs how do you merge the data?

Guest traces contain vcpu number and not pid (because guest is unaware of host PID).

Quoted 6 lines

Quoted 5 lines

However, when we
focus on output data of the write_tsc_offset event, it is difficult to directly understand contents of the data if vcpu number information is not included. So, including the information is useful, I think.

How your tool does it now?

It merges guest trace with host trace (by converting the TSC timestamp in the guest trace to host TSC using tsc_offset information).

By not recording vcpu ID in the tsc_offset trace, it is necessary to supply the tool with PID<->VCPU_id tuples for translation (so its an additional step required, and it makes trace merge impossible if the information is not available).

Quoted 3 lines

Guest traces contain vcpu number and not pid (because guest is unaware of host PID).

No, guest trace is just a regular ftrace done inside a guest. It contains guest's PIDs which is useless for host.

I do not know how exactly guest traces are transfered to a host, if each vcpu buffer is transfered separately host can figure out what trace entry belong to which vcpu based on what buffer the trace is in. But the information about what buffer belongs to which vcpu id should be transfered to a host somehow too.

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

However, when we
focus on output data of the write_tsc_offset event, it is difficult to directly understand contents of the data if vcpu number information is not included. So, including the information is useful, I think.

Quoted 5 lines

Quoted 1 line

How your tool does it now?

It merges guest trace with host trace (by converting the TSC timestamp in the guest trace to host TSC using tsc_offset information).

I mean how it does it now without vcpu id. The answer is that it works for only one vcpu now.

Quoted 4 lines

By not recording vcpu ID in the tsc_offset trace, it is necessary to supply the tool with PID<->VCPU_id tuples for translation (so its an additional step required, and it makes trace merge impossible if the information is not available).

The tool needs PID<->VCPU_id tuples to do the merging of any trace entry. Without that it does not know how to interpret entry timestamps (which offset to use). Apparently it will get this information from vmentry trace point. What is so special about tsc_offset tracing that it needs to contain vcpuid by itself.

Quoted 5 lines · Gleb Natapov

Quoted 3 lines

Guest traces contain vcpu number and not pid (because guest is unaware of host PID).

No, guest trace is just a regular ftrace done inside a guest. It contains guest's PIDs which is useless for host.

# tracer: nop
#
# entries-in-buffer/entries-written: 5333/5333 #P:4
#
# _-----=> irqs-off
# / _----=> need-resched

#                            | / _---=> hardirq/softirq
#                            || / _--=> preempt-depth
#                            ||| /     delay
#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
#              | |       |   ||||       |         |

Traces contain CPU ID.

Quoted 11 lines

I do not know how exactly guest traces are transfered to a host, if each vcpu buffer is transfered separately host can figure out what trace entry belong to which vcpu based on what buffer the trace is in. But the information about what buffer belongs to which vcpu id should be transfered to a host somehow too.

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

However, when we
focus on output data of the write_tsc_offset event, it is difficult to directly understand contents of the data if vcpu number information is not included. So, including the information is useful, I think.

Quoted 7 lines

Quoted 5 lines

Quoted 1 line

How your tool does it now?

It merges guest trace with host trace (by converting the TSC timestamp in the guest trace to host TSC using tsc_offset information).

I mean how it does it now without vcpu id. The answer is that it works for only one vcpu now.

Yes.

Quoted 9 lines

Quoted 4 lines

By not recording vcpu ID in the tsc_offset trace, it is necessary to supply the tool with PID<->VCPU_id tuples for translation (so its an additional step required, and it makes trace merge impossible if the information is not available).

The tool needs PID<->VCPU_id tuples to do the merging of any trace entry. Without that it does not know how to interpret entry timestamps (which offset to use). Apparently it will get this information from vmentry trace point. What is so special about tsc_offset tracing that it needs to contain vcpuid by itself.

If the tsc_offset tracepoint contains vcpu ID, its possible to lookup guest trace entry (which contains CPU ID), and match on that.

Without that, PID<->VCPU_id tuples are necessary. Yes?

Quoted 13 lines · Marcelo Tosatti

Quoted 5 lines · Gleb Natapov

Quoted 3 lines

Guest traces contain vcpu number and not pid (because guest is unaware of host PID).

No, guest trace is just a regular ftrace done inside a guest. It contains guest's PIDs which is useless for host.

# tracer: nop
#
# entries-in-buffer/entries-written: 5333/5333 #P:4
#
# _-----=> irqs-off
# / _----=> need-resched

Quoted 8 lines

#                            | / _---=> hardirq/softirq
#                            || / _--=> preempt-depth
#                            ||| /     delay
#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
#              | |       |   ||||       |         |

Traces contain CPU ID.

Doh, yes it does, but this is not vcpu_id. vcpu_id is seen as apic id in a guest, so additional step is needed to map between numbers that you see in the trace and vcpu_id. This is easy to do by looking at /proc/cpuinfo of a guest.

Quoted 34 lines

Quoted 11 lines

I do not know how exactly guest traces are transfered to a host, if each vcpu buffer is transfered separately host can figure out what trace entry belong to which vcpu based on what buffer the trace is in. But the information about what buffer belongs to which vcpu id should be transfered to a host somehow too.

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

However, when we
focus on output data of the write_tsc_offset event, it is difficult to directly understand contents of the data if vcpu number information is not included. So, including the information is useful, I think.

Quoted 7 lines

Quoted 5 lines

Quoted 1 line

How your tool does it now?

It merges guest trace with host trace (by converting the TSC timestamp in the guest trace to host TSC using tsc_offset information).

I mean how it does it now without vcpu id. The answer is that it works for only one vcpu now.

Yes.

Quoted 9 lines

Quoted 4 lines

By not recording vcpu ID in the tsc_offset trace, it is necessary to supply the tool with PID<->VCPU_id tuples for translation (so its an additional step required, and it makes trace merge impossible if the information is not available).

The tool needs PID<->VCPU_id tuples to do the merging of any trace entry. Without that it does not know how to interpret entry timestamps (which offset to use). Apparently it will get this information from vmentry trace point. What is so special about tsc_offset tracing that it needs to contain vcpuid by itself.

If the tsc_offset tracepoint contains vcpu ID, its possible to lookup guest trace entry (which contains CPU ID), and match on that.

Quoted 1 line

Without that, PID<->VCPU_id tuples are necessary. Yes?

Ah, I think I see it now. For some reason I assumed that merge is done for each vcpu separately, so you need to separate host events per vcpu too, but this is not the case, host and guest event are merged based on tsc timestamp only. In this case I see the merits of having vcpu id in tsc_offset change trace point. It will make things easier a bit.

Quoted 72 lines · Marcelo Tosatti

Quoted 58 lines · Yoshihiro YUNOMAE

Add a tracepoint write_tsc_offset for tracing TSC offset change. We want to merge ftrace's trace data of guest OSs and the host OS using TSC for timestamp in chronological order. We need "TSC offset" values for each guest when merge those because the TSC value on a guest is always the host TSC plus guest's TSC offset. If we get the TSC offset values, we can calculate the host TSC value for each guest events from the TSC offset and the event TSC value. The host TSC values of the guest events are used when we want to merge trace data of guests and the host in chronological order. (Note: the trace_clock of both the host and the guest must be set x86-tsc in this case)

TSC offset is stored in the VMCS by vmx_write_tsc_offset() or vmx_adjust_tsc_offset(). KVM executes the former function when a guest boots. The latter function is executed when kvm clock is updated. Only host can read TSC offset value from VMCS, so a host needs to output TSC offset value when TSC offset is changed.

Since the TSC offset is not often changed, it could be overwritten by other frequent events while tracing. To avoid that, I recommend to use a special instance for getting this event:

1. set a instance before booting a guest

 # cd /sys/kernel/debug/tracing/instances
 # mkdir tsc_offset
 # cd tsc_offset
 # echo x86-tsc > trace_clock
 # echo 1 > events/kvm/kvm_write_tsc_offset/enable

2. boot a guest

Signed-off-by Yoshihiro YUNOMAE <[email protected]>

Cc Marcelo Tosatti <[email protected]>

Cc Gleb Natapov <[email protected]>

Cc Thomas Gleixner <[email protected]>

Cc Ingo Molnar <[email protected]>

Cc "H. Peter Anvin" <[email protected]>

---
 arch/x86/kvm/trace.h |   18 ++++++++++++++++++
 arch/x86/kvm/vmx.c   |    3 +++
 arch/x86/kvm/x86.c   |    1 +
 3 files changed, 22 insertions(+)

arch/x86/kvm/trace.h +8 -0

index fe5e00e..9c22e39 100644
@@ -815,6 +815,24 @@ TRACE_EVENT(kvm_track_tsc,
 		  __print_symbolic(__entry->host_clock, host_clocks))
 );
 
+TRACE_EVENT(kvm_write_tsc_offset,
+	TP_PROTO(__u64 previous_tsc_offset, __u64 next_tsc_offset),
+	TP_ARGS(previous_tsc_offset, next_tsc_offset),
+
+	TP_STRUCT__entry(
+		__field(	__u64,	previous_tsc_offset		)
+		__field(	__u64,	next_tsc_offset			)
+	),

arch/x86/kvm/trace.h TRACE_EVENT(kvm_track_tsc, · line 835

Quoted 10 lines

arch/x86/kvm/trace.h +10 -0

+
+	TP_fast_assign(
+		__entry->previous_tsc_offset	= previous_tsc_offset;
+		__entry->next_tsc_offset	= next_tsc_offset;
+	),
+
+	TP_printk("previous=%llu next=%llu",
+		  __entry->previous_tsc_offset, __entry->next_tsc_offset)
+);
+

Yoshihiro YUNOMAE,

Quoted 6 lines

1) Why is previous_tsc_offset necessary?

2) The TSC offset traces should include vcpu number, so that its possible to correlate traces of SMP guests (the tool should use the individual vcpu tsc offsets when converting guests trace).

Why PID is not enough? No other trace, except kvm_entry, outputs vcpu id.