Tracing KVM with Systemtap

Hi all,

I am a developer of Systemtap. I am looking into tracing KVM (the kernel part and QEMU) and also the KVM guests with Systemtap. I googled and found references to Xenprobes and xdt+dtrace, and I was wondering if someone is working on the dynamic tracing interface for KVM?

I've read the KVM kernel code and I think some expensive operations (things that need to be trapped back to the host kernel - eg. loading of control registers on x86/x64) can be interesting spots for adding an SDT (static marker), and I/O operations performed for the guests can be useful information to collect.

I know that KVM guests run like a userspace process and thus techniques for tracing Xen might be overkilled, and also gdb can be used to trace KVM guests. However, are that anything special I need to be aware of before I go further into the development of the Systemtap KVM probes?

(Opinions / Suggestions / Criticisms welcome!)

Hi Rayson,
For the KVM kernel module Linux trace events are already used. For example, see arch/x86/kvm/trace.h and check out /sys/kernel/debug/tracing/events/kvm/*. There is a set of useful static trace points for vm_exit/vm_enter, pio, mmio, etc.

For the KVM guest there is perf-kvm(1). This allows perf(1) to look up addresses inside the guest (kernel only?). It produces system-wide performance profiles including guests. Perhaps someone can comment on perf-kvm's full feature set and limitations?

For QEMU userspace Prerna Saxena and I are proposing a static tracing patchset. It abstracts the trace backend (SystemTap, LTTng UST, DTrace, etc) from the actual tracepoints so that portability can be achieved. There is a built-in trace backend that has a basic feature set but isn't as fancy as SystemTap. I have implemented LTTng Userspace Tracer support, perhaps you'd like to add SystemTap/DTrace support with sdt.h?

http://www.mail-archive.com/[email protected]/msg41323.html
http://repo.or.cz/w/qemu/stefanha.git/shortlog/refs/heads/tracing_v3

Stefan
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 11 lines · Stefan Hajnoczi

Hi Rayson,
For the KVM kernel module Linux trace events are already used.  For example, see arch/x86/kvm/trace.h and check out /sys/kernel/debug/tracing/events/kvm/*.  There is a set of useful static trace points for vm_exit/vm_enter, pio, mmio, etc.

For the KVM guest there is perf-kvm(1).  This allows perf(1) to look up addresses inside the guest (kernel only?).  It produces system-wide performance profiles including guests.  Perhaps someone can comment on perf-kvm's full feature set and limitations?

Quoted 7 lines

For QEMU userspace Prerna Saxena and I are proposing a static tracing patchset.  It abstracts the trace backend (SystemTap, LTTng UST, DTrace, etc) from the actual tracepoints so that portability can be achieved.  There is a built-in trace backend that has a basic feature set but isn't as fancy as SystemTap.  I have implemented LTTng Userspace Tracer support, perhaps you'd like to add SystemTap/DTrace support with sdt.h?

Thanks Stefan for the reply!

I've looked at the tracing additions in QEMU, including the "Simple trace backend" (simpletrace.c) and the "tracetool" script, and I think the SystemTap version can be implemented in a straightforward way.

One thing I was wondering, there seems to be not a lot of probes (except the examples?) in the QEMU code, are we expected to see more probes in the next release, or this work will be a long-term project that will not be added to the official QEMU code in the near future?

(I believe if we can get the tracing framework integrated, then specific probes can be added on-demand -- but of course that is just my own opinion :-D )

Right now there are few pre-defined probes ("trace events" in QEMU tracing speak). As I develop I try to be mindful of new ones I create and whether they would be generally useful. I intend to contribute more probes and hope others will too!

Prerna is also looking at adding useful probes.

Stefan

Quoted 4 lines · Stefan Hajnoczi

Right now there are few pre-defined probes ("trace events" in QEMU tracing speak).  As I develop I try to be mindful of new ones I create and whether they would be generally useful.  I intend to contribute more probes and hope others will too!

I am still looking at/hacking the QEMU code. I have looked at the following places in the code that I think can be useful to have statistics gathered:

net.c qemu_deliver_packet(), etc - network statistics

<CPU Arch>/op_helper.c global_cpu_lock(), tlb_fill() - lock & unlock, and TLB refill statistics

balloon.c, hw/virtio-balloon.c - ballooning information.

I will see what other probes are useful for the end users. Also, are there developer documentations for KVM? (I googled but found a lot of presentations about KVM but not a lot of info about the internals.)

Quoted 1 line

net.c qemu_deliver_packet(), etc - network statistics

Yes.

Quoted 2 lines

<CPU Arch>/op_helper.c global_cpu_lock(), tlb_fill() - lock & unlock, and TLB refill statistics

These are not relevant to KVM, they are only used when running with KVM disabled (TCG mode).

Quoted 1 line

balloon.c, hw/virtio-balloon.c - ballooning information.

Prerna added a balloon event which is in qemu.git trace-events. Does that one do what you need?

Quoted 3 lines

I will see what other probes are useful for the end users. Also, are there developer documentations for KVM? (I googled but found a lot of presentations about KVM but not a lot of info about the internals.)

Not really. I suggest grabbing the source and following vl.c:main() to the main KVM execution code.

Stefan
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 6 lines · Stefan Hajnoczi

Quoted 3 lines

I will see what other probes are useful for the end users. Also, are there developer documentations for KVM? (I googled but found a lot of presentations about KVM but not a lot of info about the internals.)

Not really.  I suggest grabbing the source and following vl.c:main() to the main KVM execution code.

I was looking for the hardware interfacing code earlier this morning -- QEMU has the hardware specific directories (e.g. target-i386/ , target-ppc/ ), and I was trying to understand the execution environment when the host and guest are running on the same architecture.

I believe cpu_gen_code() and other related functions are what I should dig into...

KVM does not generate code. Almost all the "emulation" code in the source tree is part of the Tiny Code Generator (TCG) used when KVM is not enabled (e.g. to emulate an ARM board on an x86-64 host).

If you follow the life-cycle in vl.c it will take you through cpus.c and into kvm-all.c:kvm_cpu_exec(). Note that the details differ slightly between qemu.git and qemu-kvm.git, and I have described qemu.git.

Stefan
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 3 lines · Stefan Hajnoczi

KVM does not generate code.  Almost all the "emulation" code in the source tree is part of the Tiny Code Generator (TCG) used when KVM is not enabled (e.g. to emulate an ARM board on an x86-64 host).

Thanks, that's what I thought too. Otherwise it would be really slow to run KVM :)

But if KVM is not used, and QEMU host & guest are running on the same architecture, is TCG off? (Hmm, I guess I can find that answer myself by reading the code).

Stefan, are you accepting patches? If so, I will create a patch with the Systemtap framework & other probes.

Quoted 8 lines · Rayson Ho

I added a probe point for a conference demo, are you interested in adding it in?
void qemu_send_packet(VLANClientState *vc, const uint8_t *buf, int
size)
{

+     trace_qemu_send_packet(buf, size);
     qemu_send_packet_async(vc, buf, size, NULL);

}

Cool, I recently talked to Prerna Saxena about improving tracing coverage. The net subsystem is one of the areas where we need to add trace events.

Please include the VLANClientState *vc pointer in the trace so sent packets can be correlated to their network clients.

Want to send a patch to qemu-devel?

Stefan

Please review the probe addition in QEMU for the network module, note that I added probes for send & receive Ethernet frames for rtl8139 so far but not other network devices. Also for receive, the probe would only fire when a frame is successfully received, so any errors are ignored.

=========================================================================

hw/rtl8139.c +1 -0

index a22530c..df04f59 100644
@@ -53,6 +53,7 @@
  #include "net.h"
  #include "loader.h"
  #include "sysemu.h"
+#include "trace.h"
 
  /* debug RTL8139 card */
  //#define DEBUG_RTL8139 1

hw/rtl8139.c +2 -0

@@ -1742,6 +1745,8 @@ static uint32_t rtl8139_RxConfig_read(RTL8139State *s)
 
  static void rtl8139_transfer_frame(RTL8139State *s, const uint8_t *buf, int size, int do_interrupt)
  {
+    trace_rtl8139_transfer_frame(s, buf, size, do_interrupt);
+
     if (!size)
     {
         DEBUG_PRINT(("RTL8139: +++ empty ethernet frame\n"));

net.c +1 -0

index 9ba5be2..ba74f74 100644
@@ -36,6 +36,7 @@
  #include "qemu-common.h"
  #include "qemu_socket.h"
  #include "hw/qdev.h"
+#include "trace.h"
 
  static QTAILQ_HEAD(, VLANState) vlans;
  static QTAILQ_HEAD(, VLANClientState) non_vlan_clients;