View for virtual machine monitoring
Hello,
We are currently working on a new view in Eclipse's TMF plugin (Tracing and Monitoring Framework) specific to virtual machine analysis. This view requires kernel traces from the host and from each guest with a set of specific tracepoints activated. The traces are then merged together and analysed in a way that the real state of each system can be rebuilt, while taking into account all the interactions between the different systems.
The main purpose of this view is to easily point out latency problems due to resource sharing. For now, we only consider CPU time, but more resources (such as memory allocation, disks...) will be added.
Two screenshots are attached. The first one shows the virtual machines and the state of their respective virtual CPUs. The second screenshot gives in-depth information about one of the virtual CPUs, showing only the threads that interacted with this vCPU and their state during the time of the trace. We think that this approach of showing information across the layers (OS, KVM, guest OS, and eventually JVM...) can be helpful to investigate latency-related problems specific to virtual machines.
Legend:
Green: user mode
Blue: kernel mode
Yellow: process blocked
Purple: vCPU preempted
Grey: vCPU idle
For the sake of our experience, we pinned vCPU0 of VM1 and vCPU0 of VM2 on the same physical CPU, and ran a CPU-intensive workload for one second one each one of them. We generated our traces using the low-overhead LTTng tracer. We can clearly see that during that second, both of the virtual CPUs are fighting over the same physical CPU.
We seek any thoughts or suggestions on the effectiveness of this view or on our approach. Any real life problems waiting for investigation are also welcome.
Mohamad Gebai
A message that is not in the archive (sent off-list or not yet received): 1373394774.51dc57569fe05-AbLIzcVVHEexwLxtslwksLDks+cytr/[email protected]
Hello Mohamad!
Your work looks very interesting. I have been forced to be away from it for a couple months now due to other work priorities, but I have been building something quite similar myself.
----- Original Message -----
Quoted 8 lines
Hello,
We are currently working on a new view in Eclipse's TMF plugin (Tracing and Monitoring Framework) specific to virtual machine analysis. This view requires
kernel traces from the host and from each guest with a set of specific tracepoints activated. The traces are then merged together and analysed in a way that the real state of each system can be rebuilt, while taking into account all the interactions between the different systems.
I assume you are using LTTng for Linux, are you using it for KVM as well?
I assume then that you are using CTF formatted traces?
Are you using TMF's CTF parser?
Quoted 5 lines
The main purpose of this view is to easily point out latency problems due to resource sharing. For now, we only consider CPU time, but more resources (such
as memory allocation, disks...) will be added.
Quoted 9 lines
Two screenshots are attached. The first one shows the virtual machines and the
state of their respective virtual CPUs. The second screenshot gives in-depth information about one of the virtual CPUs, showing only the threads that interacted with this vCPU and their state during the time of the trace. We think that this approach of showing information across the layers (OS, KVM, guest OS, and eventually JVM...) can be helpful to investigate latency-related
problems specific to virtual machines.
I agree!
Quoted 16 lines
Legend:
Green: user mode
Blue: kernel mode
Yellow: process blocked
Purple: vCPU preempted
Grey: vCPU idleFor the sake of our experience, we pinned vCPU0 of VM1 and vCPU0 of VM2 on the
same physical CPU, and ran a CPU-intensive workload for one second one each one
of them. We generated our traces using the low-overhead LTTng tracer. We can clearly see that during that second, both of the virtual CPUs are fighting over
the same physical CPU.
Quoted 2 lines
We seek any thoughts or suggestions on the effectiveness of this view or on our approach. Any real life problems waiting for investigation are also welcome.
I am interested to know how you setup your view under the hood. Did you build from the code base that was already there with the ControlFlowView (which is what I did), and then using the TMF state system infrastructure to model state of the various elements you wish to display?
If you look through the history on this list you will see some links that I posted to the prototype that I was working with on github as well as some screenshots. I went with an approach of trying to make the view a generic display of hierarchical state of objects vs time, and then pluggable code that understands the event schema, iterating the events and updating the view. I like the idea of having a view that can be data driven and so it is then fairly straight forward to plug in any sort of state vs. time in context display. My work is incomplete, it still lacks a number of features that I intend to add including the ability to have multiple instances of the view open at the same time, all syncronized and/or a single view that aggregates the contents of many different traces. It loo
ks as though you are already doing that, though I can't help but wonder how you defined the hierarchy when different levels in the hierarchy have different traces.
best regards,
Aaron Spear
A message that is not in the archive (sent off-list or not yet received): 77342852.21701313.1373397836659.JavaMail.root-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org
Quoted 6 lines
Hello Mohamad!
Your work looks very interesting. I have been forced to be away from it for a couple months now due to other work priorities, but I have been building something quite similar myself.
Hello,
Thank you for your elaborated answer!
Quoted 7 lines
----- Original Message -----
Quoted 6 lines
Hello,
We are currently working on a new view in Eclipse's TMF plugin (Tracing and Monitoring Framework) specific to virtual machine analysis. This view requires
kernel traces from the host and from each guest with a set of specific tracepoints activated. The traces are then merged together and analysed in
Quoted 5 lines
a
Quoted 2 lines
way that the real state of each system can be rebuilt, while taking into account all the interactions between the different systems.
I assume you are using LTTng for Linux, are you using it for KVM as well?
Yes, I am using the tracepoints of KVM that are already in the kernel, and generating the traces using LTTng. Actually I had disabled a feature in this view before taking the screenshots, which showed when the CPU was in VMX root mode to handle a VMEXIT. This information was built from KVM's tracepoints and showed the overhead caused by virtualization.
Quoted 2 lines
I assume then that you are using CTF formatted traces?
Yes.
Quoted 1 line
Are you using TMF's CTF parser?
Yes, I am using the CTF parser from TMF.
Quoted 17 lines
Quoted 1 line
The main purpose of this view is to easily point out latency problems due
to
Quoted 7 lines
resource sharing. For now, we only consider CPU time, but more resources (such
as memory allocation, disks...) will be added.Two screenshots are attached. The first one shows the virtual machines and the
state of their respective virtual CPUs. The second screenshot givesin-depth
Quoted 6 lines
information about one of the virtual CPUs, showing only the threads that interacted with this vCPU and their state during the time of the trace. We think that this approach of showing information across the layers (OS, KVM, guest OS, and eventually JVM...) can be helpful to investigate latency-related
problems specific to virtual machines.
Quoted 7 lines
I agree!
I am interested to know how you setup your view under the hood. Did you build from the code base that was already there with the ControlFlowView (which is what I did), and then using the TMF
state system infrastructure to model state of the various elements you wish to display?
Actually I used the ResourceView as a base for my view since I have different data types in the same view (VMs, vCPUs, threads and eventually interrupts).
Quoted 3 lines
If you look through the history on this list you will see some links that I posted to the prototype that I was working with on github as well as some screenshots.
Yes I have seen your previous work in TMF.
Quoted 11 lines
I went with an approach of trying to make the view a generic display of hierarchical state of objects vs time, and then pluggable code that understands the event schema, iterating the events and updating the view. I like the idea of having a view that can be data driven and so it is then fairly straight forward to plug in any sort of state vs. time in context display. My work is incomplete, it still lacks a number of features that I intend to add including the ability to have multiple instances of the view open at the same time, all syncronized and/or a single view that aggregates the contents of many different traces. It looks as though you are already doing that, though I can't help but wonder how you defined the hierarchy when different levels in the hierarchy have different traces.
How it works for now is that we have to create an Experiment with a predefined type (VM analysis type), and add to it all of the traces we want to analyse. At this step, we also have to specify which traces were recorded on the host, and which ones were recorded on the guests.
Thanks to Genevieve Bastien's experimental work, we were able to merge all of the traces together as one single trace owned by the Experiment. Then the events are handled one by one, and depending on if they came from a host, or from a guest, the state system is modified accordingly.
I will very soon post a link for my experimental branch for this view if you want to take a closer look at the code underneath.
Thanks again for your reponse,
Mohamad.
A message that is not in the archive (sent off-list or not yet received): 1373401690.51dc725a2c350-AbLIzcVVHEexwLxtslwksLDks+cytr/[email protected]
Quoted 12 lines
How it works for now is that we have to create an Experiment with a predefined
type (VM analysis type), and add to it all of the traces we want to analyse. At
this step, we also have to specify which traces were recorded on the host, and
which ones were recorded on the guests.
Thanks to Genevieve Bastien's experimental work, we were able to merge all of the traces together as one single trace owned by the Experiment. Then the events are handled one by one, and depending on if they came from a host, or from a guest, the state system is modified accordingly.
I attached a screenshot showing the state system built for this particular Experiment.
Mohamad.
Hi Mohamad,
Quite impressive!
Small detail: I don't know if you use the exact same colors as the Control Flow View, but the color you use for "vCPU preempted" seems similar to the one used for the Interrupted (IRQ) state. You should make sure they can be differentiated if shown side-by-side. In any case, it's easy to tweak ;)
There is still some designing/reviewing to do on the required concepts, like experiment types and state systems for experiments. But this is a very good example of what it will be possible to do once those features are integrated (hopefully in the coming months!)
Cheers,
Alexandre
Quoted 38 lines · Mohamad Gebai
Hello,
We are currently working on a new view in Eclipse's TMF plugin (Tracing and Monitoring Framework) specific to virtual machine analysis. This view requires kernel traces from the host and from each guest with a set of specific tracepoints activated. The traces are then merged together and analysed in a way that the real state of each system can be rebuilt, while taking into account all the interactions between the different systems.The main purpose of this view is to easily point out latency problems due to resource sharing. For now, we only consider CPU time, but more resources (such as memory allocation, disks...) will be added.
Two screenshots are attached. The first one shows the virtual machines and the state of their respective virtual CPUs. The second screenshot gives in-depth information about one of the virtual CPUs, showing only the threads that interacted with this vCPU and their state during the time of the trace. We think that this approach of showing information across the layers (OS, KVM, guest OS, and eventually JVM...) can be helpful to investigate latency-related problems specific to virtual machines.
Legend:
Green: user mode
Blue: kernel mode
Yellow: process blocked
Purple: vCPU preempted
Grey: vCPU idleFor the sake of our experience, we pinned vCPU0 of VM1 and vCPU0 of VM2 on the same physical CPU, and ran a CPU-intensive workload for one second one each one of them. We generated our traces using the low-overhead LTTng tracer. We can clearly see that during that second, both of the virtual CPUs are fighting over the same physical CPU.
We seek any thoughts or suggestions on the effectiveness of this view or on our approach. Any real life problems waiting for investigation are also welcome.
Mohamad Gebai
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 3 lines
Hi Mohamad,
Quite impressive!
Hi Alex,
Thank you!
Quoted 5 lines
Small detail: I don't know if you use the exact same colors as the Control Flow View, but the color you use for "vCPU preempted" seems similar to the one used for the Interrupted (IRQ) state. You should make sure they can be differentiated if shown side-by-side. In any case, it's easy to tweak ;)
Actually I started from the Resource View. I agree that the colors are similar, I hadn't noticed because I don't handle interrupts events for now. Thank you for the comment!
Quoted 4 lines
There is still some designing/reviewing to do on the required concepts, like experiment types and state systems for experiments. But this is a very good example of what it will be possible to do once those features are integrated (hopefully in the coming months!)
I agree!
Quoted 1 line
Cheers,
Mohamad.