Tracing KVM with LTTng
Hello,
I'm currently doing my master degree on tracing and measuring with LTTng the impact of virtualization on a physical host.
One focus of this work is to be able to represent in a graphical way the state of a machine where virtual machines are running.
I have started to connect the already defined TRACE_EVENTs with LTTng probes and adapted LTTV to represent a new state : VIRT.
In the following screenshots, I captured a trace of 4 seconds on the physical machine. My machine has 8 CPUs, a KVM VM was started with 4 VCPUs. At the beginning, the VM was idle, after 1 second I started an infinite loop and an other 1 second later.
On [1] you can see the impact of the loops on the kvm threads during the 4 seconds.
On [2] you can see a closer look of the state of the kvm threads (blue means syscall, red means running in VM mode (vm_entry), dark yellow means waiting for CPU).
On [3] you can see the relationship between the kvm threads and other processes on the system (here especially kondemand and ksmd), each white dot meaning there are events not displayed because the zoom level is to high On [4] you can see the resource view, which makes clear the relationship between physical resources, the interrupts and the VM
On [5] you can see the statistics view on a specific physical CPU and on the whole trace.
And finally on [6] you can see an other test with a ping flood.
In the next days, I will send my patches to the official LTTng git. My next step is to synchronise the traces collected from the host with the traces collected from the guest (by finding an efficient way to share the TSC_OFFSET) to have some infos of what is happening during the time the VM has the control.
The reason I post these screenshots now is that I will be at Linuxcon next week, and I would really appreciate some feedbacks and ideas for future improvements from the KVM community.
So if you are interested, contact me directly and if you are there we'll try to meet.
Thanks,
Julien
[1] http://ju.klipix.org/lttng/lttng-kvm-busy.png
[2] http://ju.klipix.org/lttng/lttng-kvm.png
[3] http://ju.klipix.org/lttng/lttng-kvm-ksmd.png
[4] http://ju.klipix.org/lttng/lttng-kvm-ressource-view.png
[5] http://ju.klipix.org/lttng/lttng-kvm-stats.png
[6] http://ju.klipix.org/lttng/lttng-kvm-ping.png
Quoted 3 lines · Julien Desfossez
On [2] you can see a closer look of the state of the kvm threads (blue means syscall, red means running in VM mode (
vm_entry), dark yellow means waiting for CPU).
This is a great visualization. It shows the state of the entire system in a clear way and makes the wait times easier to understand.
Quoted 5 lines
In the next days, I will send my patches to the official LTTng git. My next step is to synchronise the traces collected from the host with the traces collected from the guest (by finding an efficient way to share the
TSC_OFFSET) to have some infos of what is happening during the time the VM has the control.
I'd like to try this patch out, will be checking the LTTng mailing list.
Quoted 5 lines
The reason I post these screenshots now is that I will be at Linuxcon next week, and I would really appreciate some feedbacks and ideas for future improvements from the KVM community.
So if you are interested, contact me directly and if you are there we'll try to meet.
I'd like to find out more about what you're doing. Unfortunately I will not be at LinuxCon/KVM Forum this year.
Prerna Saxena and I have been working on static trace events for QEMU userspace. Here is the commit to add LTTng Userspace Tracer support:
http://repo.or.cz/w/qemu/stefanha.git/commitdiff/5560c202f4eeeec5cc37692d35e53f784c74d65c
Using the tracing branch above you can place static trace events in QEMU userspace and collect the trace with LTTng. See the trace-events file for sample trace event definitions.
Stefan