Shared value between host and guests
Hi,
I'm tracing kvm guests and their host and an issue we have is to identify which host process corresponds to which guest trace so we can follow the state of the virtual cpu on the host.
For our needs, we fixed this problem by adding a kernel module on host and guest where hosts and guests make an hypercall received by the other and sending a UID between them. By tracing those hypercalls and tracking the uid value, we can know which thread on the host corresponds to which guest machine.
But we would like to achieve this without having to add an additional module for tracing purposes. I'm wondering if there's already a value somewhere in the kernel that is uniquely shared between the host process/threads and the corresponding guest. And if not, would it be possible/desirable to add one? For instance, the bootid of the guest could be sent to the host process at the start and stored in some system file. The trace could then retrieve it and the analysis can figure out who's who all the traces.
Fyi, this is the kind of virtual machine analyses we can obtain by tracing guest and host:
http://archive.eclipse.org/tracecompass.incubator/doc/org.eclipse.tracecompass.incubator.virtual.machine.analysis.doc.user/Virtual-Machine-Analysis-User-Documentation.html
We have more, but it is not yet documented unfortunately.
Thanks,
Geneviève
Quoted 3 lines · Genevieve Bastien
I'm tracing kvm guests and their host and an issue we have is to identify which host process corresponds to which guest trace so we can follow the state of the virtual cpu on the host.
There are uevents for the VM lifecycle that include the PID. Please see Linux commit 286de8f6ac9202f1c9012784639156c6ec386eb8 & fdeaf7e3eb37c6dbc4b4ac97dbe1945d239eb788.
Did you check if these are appropriate for your use case?
Stefan
Quoted 9 lines · Stefan Hajnoczi
Quoted 3 lines · Genevieve Bastien
I'm tracing kvm guests and their host and an issue we have is to identify which host process corresponds to which guest trace so we can follow the state of the virtual cpu on the host.
There are uevents for the VM lifecycle that include the PID. Please see Linux commit
286de8f6ac9202f1c9012784639156c6ec386eb8&fdeaf7e3eb37c6dbc4b4ac97dbe1945d239eb788.Did you check if these are appropriate for your use case?
Thanks for the hint, I didn't know about those. But the uevents are only for the host right? There is no specific value in debugfs for guest.
But I'll see if I can use some udev rules to trigger <some-kvm-tracing-script> to achieve what I need.
Thanks,
Geneviève
Quoted 5 lines · Geneviève Bastien
Quoted 4 lines · Stefan Hajnoczi
Quoted 3 lines · Genevieve Bastien
I'm tracing kvm guests and their host and an issue we have is to identify which host process corresponds to which guest trace so we can follow the state of the virtual cpu on the host.
Quoted 7 lines
Quoted 5 lines
There are uevents for the VM lifecycle that include the PID. Please see Linux commit
286de8f6ac9202f1c9012784639156c6ec386eb8&fdeaf7e3eb37c6dbc4b4ac97dbe1945d239eb788.Did you check if these are appropriate for your use case?
Thanks for the hint, I didn't know about those. But the uevents are only for the host right? There is no specific value in debugfs for guest.
I'm not sure I understand the question.
Uevent are emitted on the host. Given the PID field, it should be possible to correlate them to a specific guest (e.g. by looking at information from the user or VM management tools, or simply by looking at /proc/$PID/cmdline for the QEMU -name argument that can be used to identify guests on the host side).
Stefan
Quoted 5 lines · Stefan Hajnoczi
Quoted 2 lines · Geneviève Bastien
Thanks for the hint, I didn't know about those. But the uevents are only for the host right? There is no specific value in debugfs for guest.
I'm not sure I understand the question.
Quoted 7 lines
Uevent are emitted on the host. Given the PID field, it should be possible to correlate them to a specific guest (e.g. by looking at information from the user or VM management tools, or simply by looking at
/proc/$PID/cmdlinefor the QEMU -name argument that can be used to identify guests on the host side).Stefan
With the information from the host and those uevents, we can indeed easily figure out in our traces which processes are associated with a guest and get its command line and a lot more information.
The missing piece is in the guest trace. For example, we have lttng traces taken on the guest and the host. We may have multiple guests and multiple hosts to trace. It's easy to know which traces are from hosts from the events, but which are from guests? And which guest trace goes with which of the kvm processes on which host? The guest trace contains no information that can be linked to the host.
I was thinking that the guest could do a hypercall to the host after bootup to share some unique information, for instance its bootid, that the host could store somewhere. That information would be available in the traces so that we can easily associate the guest with its host process and the states of its virtual CPUs to that of the corresponding threads, and much more.
I hope this describes our need a little better.
Thanks,
Geneviève (tahini on irc)
Quoted 24 lines · Geneviève Bastien
Quoted 5 lines · Stefan Hajnoczi
Quoted 2 lines · Geneviève Bastien
Thanks for the hint, I didn't know about those. But the uevents are only for the host right? There is no specific value in debugfs for guest.
I'm not sure I understand the question.
Quoted 7 lines
Uevent are emitted on the host. Given the PID field, it should be possible to correlate them to a specific guest (e.g. by looking at information from the user or VM management tools, or simply by looking at
/proc/$PID/cmdlinefor the QEMU -name argument that can be used to identify guests on the host side).Stefan
With the information from the host and those uevents, we can indeed easily figure out in our traces which processes are associated with a guest and get its command line and a lot more information.
The missing piece is in the guest trace. For example, we have lttng traces taken on the guest and the host. We may have multiple guests and multiple hosts to trace. It's easy to know which traces are from hosts from the events, but which are from guests? And which guest trace goes with which of the kvm processes on which host? The guest trace contains no information that can be linked to the host.
Quoted 8 lines
I was thinking that the guest could do a hypercall to the host after bootup to share some unique information, for instance its bootid, that the host could store somewhere. That information would be available in the traces so that we can easily associate the guest with its host process and the states of its virtual CPUs to that of the corresponding threads, and much more.
I hope this describes our need a little better.
The QEMU -uuid <uuid> option makes a UUID available to the guest via SMBIOS and fw_cfg on x86. Inside the guest you can print it like this:
# dmidecode -s system-uuid 01020304-0506-0708-090A-0B0C0D0E0F10
Maybe you can base the guest trace filename off the UUID:
guest-01020304-0506-0708-090A-0B0C0D0E0F10-trace-001.dat
On the host you can either find the UUIDs in the libvirt domain XML:
# virsh dump my-domain
<domain ...>
<uuid>0102030405060708090a0b0c0d0e0f10</uuid>
...Or you can use the kvm.ko uevent to find the QEMU PID and then check /proc/$PID/cmdline for the uuid. The UUID can also be fetched via the query-uuid QMP command if you don't want to search /proc/$PID/cmdline for -uuid <uuid>.
Maybe you can base the host trace filename off the UUID too:
host-01020304-0506-0708-090A-0B0C0D0E0F10-trace-001.dat
Then the ask of correlating traces becomes pretty easy for post-processing scripts since they can look at the filenames.
Stefan
2018-01-26 22:03 GMT+08:00 Stefan Hajnoczi <[email protected]>:
Quoted 44 lines · Geneviève Bastien
Quoted 24 lines · Stefan Hajnoczi
Quoted 5 lines · Geneviève Bastien
Quoted 2 lines
Thanks for the hint, I didn't know about those. But the uevents are only for the host right? There is no specific value in debugfs for guest.
I'm not sure I understand the question.
Quoted 7 lines
Uevent are emitted on the host. Given the PID field, it should be possible to correlate them to a specific guest (e.g. by looking at information from the user or VM management tools, or simply by looking at
/proc/$PID/cmdlinefor the QEMU -name argument that can be used to identify guests on the host side).Stefan
With the information from the host and those uevents, we can indeed easily figure out in our traces which processes are associated with a guest and get its command line and a lot more information.
The missing piece is in the guest trace. For example, we have lttng traces taken on the guest and the host. We may have multiple guests and multiple hosts to trace. It's easy to know which traces are from hosts from the events, but which are from guests? And which guest trace goes with which of the kvm processes on which host? The guest trace contains no information that can be linked to the host.
Quoted 8 lines
I was thinking that the guest could do a hypercall to the host after bootup to share some unique information, for instance its bootid, that the host could store somewhere. That information would be available in the traces so that we can easily associate the guest with its host process and the states of its virtual CPUs to that of the corresponding threads, and much more.
I hope this describes our need a little better.
The QEMU -uuid <uuid> option makes a UUID available to the guest via SMBIOS and
fw_cfgon x86. Inside the guest you can print it like this:# dmidecode -s system-uuid 01020304-0506-0708-090A-0B0C0D0E0F10Maybe you can base the guest trace filename off the UUID:
guest-01020304-0506-0708-090A-0B0C0D0E0F10-trace-001.dat
Quoted 8 lines
On the host you can either find the UUIDs in the libvirt domain XML:
# virsh dump my-domain <domain ...> <uuid>0102030405060708090a0b0c0d0e0f10</uuid> ...Or you can use the kvm.ko uevent to find the QEMU PID and then check
How to use the kvm.ko uevent to find the QEMU PID?
Regards,
Wanpeng Li
Quoted 47 lines · Wanpeng Li
2018-01-26 22:03 GMT+08:00 Stefan Hajnoczi <[email protected]>:
Quoted 46 lines · Geneviève Bastien
Quoted 24 lines · Stefan Hajnoczi
Quoted 5 lines · Geneviève Bastien
Quoted 2 lines
Thanks for the hint, I didn't know about those. But the uevents are only for the host right? There is no specific value in debugfs for guest.
I'm not sure I understand the question.
Quoted 7 lines
Uevent are emitted on the host. Given the PID field, it should be possible to correlate them to a specific guest (e.g. by looking at information from the user or VM management tools, or simply by looking at
/proc/$PID/cmdlinefor the QEMU -name argument that can be used to identify guests on the host side).Stefan
With the information from the host and those uevents, we can indeed easily figure out in our traces which processes are associated with a guest and get its command line and a lot more information.
The missing piece is in the guest trace. For example, we have lttng traces taken on the guest and the host. We may have multiple guests and multiple hosts to trace. It's easy to know which traces are from hosts from the events, but which are from guests? And which guest trace goes with which of the kvm processes on which host? The guest trace contains no information that can be linked to the host.
Quoted 8 lines
I was thinking that the guest could do a hypercall to the host after bootup to share some unique information, for instance its bootid, that the host could store somewhere. That information would be available in the traces so that we can easily associate the guest with its host process and the states of its virtual CPUs to that of the corresponding threads, and much more.
I hope this describes our need a little better.
The QEMU -uuid <uuid> option makes a UUID available to the guest via SMBIOS and
fw_cfgon x86. Inside the guest you can print it like this:# dmidecode -s system-uuid 01020304-0506-0708-090A-0B0C0D0E0F10Maybe you can base the guest trace filename off the UUID:
guest-01020304-0506-0708-090A-0B0C0D0E0F10-trace-001.dat
On the host you can either find the UUIDs in the libvirt domain XML:
Quoted 8 lines
Quoted 6 lines
# virsh dump my-domain <domain ...> <uuid>0102030405060708090a0b0c0d0e0f10</uuid> ...Or you can use the kvm.ko uevent to find the QEMU PID and then check
How to use the kvm.ko uevent to find the QEMU PID?
Add a udev rule on the host for kvm.ko EVENT=create/destroy events. These events have a PID=<pid> field that a script can use to record the QEMU instance that is associated with this guest.
Please see virt/kvm/kvm_main.c:kvm_uevent_notify_change().
Stefan
RFC: lttng-module: how to track the link between host and guest traces
In reply to Stefan HajnocziHi all,
Here included is a conversation I had on the kvm mailing list regarding how to link traces from host and guests when tracing virtual environments. Following Stefan's last suggestions, here is a suggestion on what can be done in lttng-modules, it's not intrusive and does not require modifications to the kernel. Let me know what you think:
* If we want to trace guests _and_ host and linek them together, we require to set the VM's -uuid parameter. I think virsh does that by default anyway.
- We can then obtain this uuid in lttng via dmi_get_system_info(DMI_PRODUCT_UUID). We can add it to a statedump event, or in the 'env' section of the metadata (which might be easier to manager and does not require activating an additional event).
* Guest statedump: on host, we can access the guests by parsing the struct dentry* kvm_debugfs_dir for entries with the <pid>-xx format. We can then parse the /proc/<pid>/cmdline to obtain the uuid and associate it to the guest.
* Guest lifetime during the trace: We can add a lttng-module that adds a udev rule to intercept VM creation/destruction. We can trace those events, adding the pid/uuid in the fields.
What do you think of this approach?
Thanks,
Geneviève
-------- Forwarded Message --------
Subject: Re: Shared value between host and guests
Date: Fri, 26 Jan 2018 14:03:48 +0000
From: Stefan Hajnoczi <[email protected]>
To: Geneviève Bastien <[email protected]>
CC: [email protected]
Quoted 24 lines · Geneviève Bastien
Quoted 5 lines · Stefan Hajnoczi
Quoted 2 lines · Geneviève Bastien
Thanks for the hint, I didn't know about those. But the uevents are only for the host right? There is no specific value in debugfs for guest.
I'm not sure I understand the question.
Quoted 7 lines
Uevent are emitted on the host. Given the PID field, it should be possible to correlate them to a specific guest (e.g. by looking at information from the user or VM management tools, or simply by looking at
/proc/$PID/cmdlinefor the QEMU -name argument that can be used to identify guests on the host side).Stefan
With the information from the host and those uevents, we can indeed easily figure out in our traces which processes are associated with a guest and get its command line and a lot more information.
The missing piece is in the guest trace. For example, we have lttng traces taken on the guest and the host. We may have multiple guests and multiple hosts to trace. It's easy to know which traces are from hosts from the events, but which are from guests? And which guest trace goes with which of the kvm processes on which host? The guest trace contains no information that can be linked to the host.
Quoted 8 lines
I was thinking that the guest could do a hypercall to the host after bootup to share some unique information, for instance its bootid, that the host could store somewhere. That information would be available in the traces so that we can easily associate the guest with its host process and the states of its virtual CPUs to that of the corresponding threads, and much more.
I hope this describes our need a little better.
The QEMU -uuid <uuid> option makes a UUID available to the guest via SMBIOS and fw_cfg on x86. Inside the guest you can print it like this:
# dmidecode -s system-uuid 01020304-0506-0708-090A-0B0C0D0E0F10
Maybe you can base the guest trace filename off the UUID:
guest-01020304-0506-0708-090A-0B0C0D0E0F10-trace-001.dat
On the host you can either find the UUIDs in the libvirt domain XML:
# virsh dump my-domain
<domain ...>
<uuid>0102030405060708090a0b0c0d0e0f10</uuid>
...Or you can use the kvm.ko uevent to find the QEMU PID and then check /proc/$PID/cmdline for the uuid. The UUID can also be fetched via the query-uuid QMP command if you don't want to search /proc/$PID/cmdline for -uuid <uuid>.
Maybe you can base the host trace filename off the UUID too:
host-01020304-0506-0708-090A-0B0C0D0E0F10-trace-001.dat
Then the ask of correlating traces becomes pretty easy for post-processing scripts since they can look at the filenames.
Stefan
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Quoted 23 lines · Stefan Hajnoczi
The QEMU -uuid <uuid> option makes a UUID available to the guest via SMBIOS and
fw_cfgon x86. Inside the guest you can print it like this:# dmidecode -s system-uuid 01020304-0506-0708-090A-0B0C0D0E0F10Maybe you can base the guest trace filename off the UUID:
guest-01020304-0506-0708-090A-0B0C0D0E0F10-trace-001.dat
On the host you can either find the UUIDs in the libvirt domain XML:
# virsh dump my-domain <domain ...> <uuid>0102030405060708090a0b0c0d0e0f10</uuid> ...Or you can use the kvm.ko uevent to find the QEMU PID and then check
/proc/$PID/cmdlinefor the uuid. The UUID can also be fetched via the query-uuid QMP command if you don't want to search/proc/$PID/cmdlinefor -uuid <uuid>.
Quoted 8 lines
Maybe you can base the host trace filename off the UUID too:
host-01020304-0506-0708-090A-0B0C0D0E0F10-trace-001.dat
Then the ask of correlating traces becomes pretty easy for post-processing scripts since they can look at the filenames.
Stefan
I think this approach would work. Requiring users to set a uuid to their VM if they plan on tracing it doesn't seem too bad a requirement.
Thanks for your help,
Geneviève