Skip to contentSkip to conversation

View for virtual machine monitoring

DISCUSSION Mohamad Gebai 3 participants 6 messages Jul 9, 2013 4 messages not in the archive

lore
Share & tools
lore
#0 lore raw

Hello,
We are currently working on a new view in Eclipse's TMF plugin (Tracing and Monitoring Framework) specific to virtual machine analysis. This view requires kernel traces from the host and from each guest with a set of specific tracepoints activated. The traces are then merged together and analysed in a way that the real state of each system can be rebuilt, while taking into account all the interactions between the different systems.

The main purpose of this view is to easily point out latency problems due to resource sharing. For now, we only consider CPU time, but more resources (such as memory allocation, disks...) will be added.

Two screenshots are attached. The first one shows the virtual machines and the state of their respective virtual CPUs. The second screenshot gives in-depth information about one of the virtual CPUs, showing only the threads that interacted with this vCPU and their state during the time of the trace. We think that this approach of showing information across the layers (OS, KVM, guest OS, and eventually JVM...) can be helpful to investigate latency-related problems specific to virtual machines.

Legend:
Green: user mode
Blue: kernel mode
Yellow: process blocked
Purple: vCPU preempted
Grey: vCPU idle

For the sake of our experience, we pinned vCPU0 of VM1 and vCPU0 of VM2 on the same physical CPU, and ran a CPU-intensive workload for one second one each one of them. We generated our traces using the low-overhead LTTng tracer. We can clearly see that during that second, both of the virtual CPUs are fighting over the same physical CPU.

We seek any thoughts or suggestions on the effectiveness of this view or on our approach. Any real life problems waiting for investigation are also welcome.

Mohamad Gebai

A message that is not in the archive (sent off-list or not yet received): 1373394774.51dc57569fe05-AbLIzcVVHEexwLxtslwksLDks+cytr/[email protected]

#2 lore raw

Hello Mohamad!

Your work looks very interesting. I have been forced to be away from it for a couple months now due to other work priorities, but I have been building something quite similar myself.

----- Original Message -----

Quoted 8 lines

Hello,
We are currently working on a new view in Eclipse's TMF plugin (Tracing and Monitoring Framework) specific to virtual machine analysis. This view requires
kernel traces from the host and from each guest with a set of specific tracepoints activated. The traces are then merged together and analysed in a way that the real state of each system can be rebuilt, while taking into account all the interactions between the different systems.

I assume you are using LTTng for Linux, are you using it for KVM as well?

I assume then that you are using CTF formatted traces?

Are you using TMF's CTF parser?

Quoted 5 lines

The main purpose of this view is to easily point out latency problems due to resource sharing. For now, we only consider CPU time, but more resources (such
as memory allocation, disks...) will be added.

Quoted 9 lines

Two screenshots are attached. The first one shows the virtual machines and the
state of their respective virtual CPUs. The second screenshot gives in-depth information about one of the virtual CPUs, showing only the threads that interacted with this vCPU and their state during the time of the trace. We think that this approach of showing information across the layers (OS, KVM, guest OS, and eventually JVM...) can be helpful to investigate latency-related
problems specific to virtual machines.

I agree!

Quoted 16 lines

Legend:
Green: user mode
Blue: kernel mode
Yellow: process blocked
Purple: vCPU preempted
Grey: vCPU idle

For the sake of our experience, we pinned vCPU0 of VM1 and vCPU0 of VM2 on the
same physical CPU, and ran a CPU-intensive workload for one second one each one
of them. We generated our traces using the low-overhead LTTng tracer. We can clearly see that during that second, both of the virtual CPUs are fighting over
the same physical CPU.

Quoted 2 lines

We seek any thoughts or suggestions on the effectiveness of this view or on our approach. Any real life problems waiting for investigation are also welcome.

I am interested to know how you setup your view under the hood. Did you build from the code base that was already there with the ControlFlowView (which is what I did), and then using the TMF state system infrastructure to model state of the various elements you wish to display?

If you look through the history on this list you will see some links that I posted to the prototype that I was working with on github as well as some screenshots. I went with an approach of trying to make the view a generic display of hierarchical state of objects vs time, and then pluggable code that understands the event schema, iterating the events and updating the view. I like the idea of having a view that can be data driven and so it is then fairly straight forward to plug in any sort of state vs. time in context display. My work is incomplete, it still lacks a number of features that I intend to add including the ability to have multiple instances of the view open at the same time, all syncronized and/or a single view that aggregates the contents of many different traces. It loo
ks as though you are already doing that, though I can't help but wonder how you defined the hierarchy when different levels in the hierarchy have different traces.

best regards,
Aaron Spear

A message that is not in the archive (sent off-list or not yet received): 77342852.21701313.1373397836659.JavaMail.root-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org

#4 lore raw
Quoted 6 lines

Hello Mohamad!

Your work looks very interesting. I have been forced to be away from it for a couple months now due to other work priorities, but I have been building something quite similar myself.

Hello,
Thank you for your elaborated answer!

Quoted 7 lines

----- Original Message -----

Quoted 6 lines

Hello,
We are currently working on a new view in Eclipse's TMF plugin (Tracing and Monitoring Framework) specific to virtual machine analysis. This view requires
kernel traces from the host and from each guest with a set of specific tracepoints activated. The traces are then merged together and analysed in

Quoted 5 lines

a

Quoted 2 lines

way that the real state of each system can be rebuilt, while taking into account all the interactions between the different systems.

I assume you are using LTTng for Linux, are you using it for KVM as well?

Yes, I am using the tracepoints of KVM that are already in the kernel, and generating the traces using LTTng. Actually I had disabled a feature in this view before taking the screenshots, which showed when the CPU was in VMX root mode to handle a VMEXIT. This information was built from KVM's tracepoints and showed the overhead caused by virtualization.

Quoted 2 lines

I assume then that you are using CTF formatted traces?

Yes.

Quoted 1 line

Are you using TMF's CTF parser?

Yes, I am using the CTF parser from TMF.

Quoted 17 lines
Quoted 1 line

The main purpose of this view is to easily point out latency problems due

to

Quoted 7 lines

resource sharing. For now, we only consider CPU time, but more resources (such
as memory allocation, disks...) will be added.

Two screenshots are attached. The first one shows the virtual machines and the
state of their respective virtual CPUs. The second screenshot gives

in-depth

Quoted 6 lines

information about one of the virtual CPUs, showing only the threads that interacted with this vCPU and their state during the time of the trace. We think that this approach of showing information across the layers (OS, KVM, guest OS, and eventually JVM...) can be helpful to investigate latency-related
problems specific to virtual machines.

Quoted 7 lines

I agree!

I am interested to know how you setup your view under the hood. Did you build from the code base that was already there with the ControlFlowView (which is what I did), and then using the TMF
state system infrastructure to model state of the various elements you wish to display?

Actually I used the ResourceView as a base for my view since I have different data types in the same view (VMs, vCPUs, threads and eventually interrupts).

Quoted 3 lines

If you look through the history on this list you will see some links that I posted to the prototype that I was working with on github as well as some screenshots.

Yes I have seen your previous work in TMF.

Quoted 11 lines

I went with an approach of trying to make the view a generic display of hierarchical state of objects vs time, and then pluggable code that understands the event schema, iterating the events and updating the view. I like the idea of having a view that can be data driven and so it is then fairly straight forward to plug in any sort of state vs. time in context display. My work is incomplete, it still lacks a number of features that I intend to add including the ability to have multiple instances of the view open at the same time, all syncronized and/or a single view that aggregates the contents of many different traces. It looks as though you are already doing that, though I can't help but wonder how you defined the hierarchy when different levels in the hierarchy have different traces.

How it works for now is that we have to create an Experiment with a predefined type (VM analysis type), and add to it all of the traces we want to analyse. At this step, we also have to specify which traces were recorded on the host, and which ones were recorded on the guests.
Thanks to Genevieve Bastien's experimental work, we were able to merge all of the traces together as one single trace owned by the Experiment. Then the events are handled one by one, and depending on if they came from a host, or from a guest, the state system is modified accordingly.
I will very soon post a link for my experimental branch for this view if you want to take a closer look at the code underneath.

Thanks again for your reponse,
Mohamad.

A message that is not in the archive (sent off-list or not yet received): 1373401690.51dc725a2c350-AbLIzcVVHEexwLxtslwksLDks+cytr/[email protected]

#6 lore raw
Quoted 12 lines

How it works for now is that we have to create an Experiment with a predefined
type (VM analysis type), and add to it all of the traces we want to analyse. At
this step, we also have to specify which traces were recorded on the host, and
which ones were recorded on the guests.
Thanks to Genevieve Bastien's experimental work, we were able to merge all of the traces together as one single trace owned by the Experiment. Then the events are handled one by one, and depending on if they came from a host, or from a guest, the state system is modified accordingly.

I attached a screenshot showing the state system built for this particular Experiment.

Mohamad.

#7 lore raw
In reply to a message not in the archive

Hi Mohamad,

Quite impressive!

Small detail: I don't know if you use the exact same colors as the Control Flow View, but the color you use for "vCPU preempted" seems similar to the one used for the Interrupted (IRQ) state. You should make sure they can be differentiated if shown side-by-side. In any case, it's easy to tweak ;)

There is still some designing/reviewing to do on the required concepts, like experiment types and state systems for experiments. But this is a very good example of what it will be possible to do once those features are integrated (hopefully in the coming months!)

Cheers,
Alexandre

Quoted 38 lines · Mohamad Gebai

Hello,
We are currently working on a new view in Eclipse's TMF plugin (Tracing and Monitoring Framework) specific to virtual machine analysis. This view requires kernel traces from the host and from each guest with a set of specific tracepoints activated. The traces are then merged together and analysed in a way that the real state of each system can be rebuilt, while taking into account all the interactions between the different systems.

The main purpose of this view is to easily point out latency problems due to resource sharing. For now, we only consider CPU time, but more resources (such as memory allocation, disks...) will be added.

Two screenshots are attached. The first one shows the virtual machines and the state of their respective virtual CPUs. The second screenshot gives in-depth information about one of the virtual CPUs, showing only the threads that interacted with this vCPU and their state during the time of the trace. We think that this approach of showing information across the layers (OS, KVM, guest OS, and eventually JVM...) can be helpful to investigate latency-related problems specific to virtual machines.

Legend:
Green: user mode
Blue: kernel mode
Yellow: process blocked
Purple: vCPU preempted
Grey: vCPU idle

For the sake of our experience, we pinned vCPU0 of VM1 and vCPU0 of VM2 on the same physical CPU, and ran a CPU-intensive workload for one second one each one of them. We generated our traces using the low-overhead LTTng tracer. We can clearly see that during that second, both of the virtual CPUs are fighting over the same physical CPU.

We seek any thoughts or suggestions on the effectiveness of this view or on our approach. Any real life problems waiting for investigation are also welcome.

Mohamad Gebai

A message that is not in the archive (sent off-list or not yet received): [email protected]

#9 lore raw
Quoted 3 lines

Hi Mohamad,

Quite impressive!

Hi Alex,
Thank you!

Quoted 5 lines

Small detail: I don't know if you use the exact same colors as the Control Flow View, but the color you use for "vCPU preempted" seems similar to the one used for the Interrupted (IRQ) state. You should make sure they can be differentiated if shown side-by-side. In any case, it's easy to tweak ;)

Actually I started from the Resource View. I agree that the colors are similar, I hadn't noticed because I don't handle interrupts events for now. Thank you for the comment!

Quoted 4 lines

There is still some designing/reviewing to do on the required concepts, like experiment types and state systems for experiments. But this is a very good example of what it will be possible to do once those features are integrated (hopefully in the coming months!)

I agree!

Quoted 1 line

Cheers,

Mohamad.