Add some tracepoints for clarification of the cause of troubles

This patch enables us to know RunState transition. It will be userful for investigation when the trouble occured in special event such like live migration, shutdown, suspend, and so on.

Signed-off-by Kazuya Saito <[email protected]>

---
 trace-events |    1 +
 vl.c         |    2 +-
 2 files changed, 2 insertions(+), 1 deletions(-)

trace-events +1 -0

index d6a847d..1e352a7 100644
@@ -474,6 +474,7 @@ scsi_request_sense(int target, int lun, int tag) "target %d lun %d tag %d"
 # vl.c
 vm_state_notify(int running, int reason) "running %d reason %d"
 load_file(const char *name, const char *path) "name %s location %s"
+runstate_set(int new_state) "new state %d"
 
 # block/qcow2.c
 qcow2_writev_start_req(void *co, int64_t sector, int nb_sectors) "co %p sector %" PRIx64 " nb_sectors %d"

vl.c +0 -0

index a621aec..4bd3671 100644

This patch enables us to know RunState transition. It will be userful for investigation when the trouble occured in special event such like live migration, shutdown, suspend, and so on.

Signed-off-by Kazuya Saito <[email protected]>

---
 trace-events |    1 +
 vl.c         |    2 +-
 2 files changed, 2 insertions(+), 1 deletions(-)

This patch adds tracepoints at ioctl to kvm. Tracing these ioctl is useful for clarification whether the cause of troubles is qemu or kvm.

Signed-off-by Kazuya Saito <[email protected]>

---
 kvm-all.c    |    4 ++++
 trace-events |    5 +++++
 2 files changed, 9 insertions(+), 0 deletions(-)

kvm-all.c +4 -0

index 4decfdc..da15623 100644
@@ -33,6 +33,7 @@
 #include "exec/memory.h"
 #include "exec/address-spaces.h"
 #include "qemu/event_notifier.h"
+#include "trace.h"
 
 /* This check must be after config-host.h is included */
 #ifdef CONFIG_EVENTFD

@@ -1636,6 +1637,7 @@ int kvm_ioctl(KVMState *s, int type, ...)
     arg = va_arg(ap, void *);
     va_end(ap);
 
+    trace_kvm_ioctl(type);
     ret = ioctl(s->fd, type, arg);
     if (ret == -1) {
         ret = -errno;

@@ -1653,6 +1655,7 @@ int kvm_vm_ioctl(KVMState *s, int type, ...)
     arg = va_arg(ap, void *);
     va_end(ap);
 
+    trace_kvm_vm_ioctl(type);
     ret = ioctl(s->vmfd, type, arg);
     if (ret == -1) {
         ret = -errno;

@@ -1670,6 +1673,7 @@ int kvm_vcpu_ioctl(CPUState *cpu, int type, ...)
     arg = va_arg(ap, void *);
     va_end(ap);
 
+    trace_kvm_vcpu_ioctl(type);
     ret = ioctl(cpu->kvm_fd, type, arg);
     if (ret == -1) {
         ret = -errno;

trace-events +0 -0

index 1e352a7..1af7060 100644

This patch enable us to know exit reason of KVM_RUN. It will help us know where the trouble is caused.

Signed-off-by Kazuya Saito <[email protected]>

---
 kvm-all.c    |    1 +
 trace-events |    1 +
 2 files changed, 2 insertions(+), 0 deletions(-)

kvm-all.c +1 -0

index da15623..29fca13 100644
@@ -1576,6 +1576,7 @@ int kvm_cpu_exec(CPUArchState *env)
             abort();
         }
 
+        trace_kvm_run_exit(run->exit_reason);
         switch (run->exit_reason) {
         case KVM_EXIT_IO:
             DPRINTF("handle_io\n");

trace-events +0 -0

index 1af7060..c691ce4 100644

This patch adds tracepoints at creating and removing virtual devices. It is useful for investigation of trouble related to virtual devices.

Signed-off-by Kazuya Saito <[email protected]>

---
 hw/qdev.c    |    3 +++
 trace-events |    4 ++++
 2 files changed, 7 insertions(+), 0 deletions(-)

hw/qdev.c +3 -0

index 0b20280..0fda23e 100644
@@ -30,6 +30,7 @@
 #include "qapi/error.h"
 #include "qapi/qmp/qerror.h"
 #include "qapi/visitor.h"
+#include "trace.h"
 
 int qdev_hotplug = 0;
 static bool qdev_hot_added = false;

@@ -124,6 +125,7 @@ DeviceState *qdev_create(BusState *bus, const char *name)
         }
     }
 
+    trace_qdev_create(dev, dev->parent_bus);
     return dev;
 }
 

@@ -268,6 +270,7 @@ void qdev_init_nofail(DeviceState *dev)
 /* Unlink device from bus and free the structure.  */
 void qdev_free(DeviceState *dev)
 {
+    trace_qdev_free(dev, dev->parent_bus);
     object_unparent(OBJECT(dev));
 }
 

trace-events +4 -0

index c691ce4..235b978 100644
@@ -1102,3 +1102,7 @@ kvm_ioctl(int type) "type %d"
 kvm_vm_ioctl(int type) "type %d"
 kvm_vcpu_ioctl(int type) "type %d"
 kvm_run_exit(uint32_t reason) "reason %d"
+
+# qdev.c
+qdev_create(void *dev, void *bus) "dev %p, bus %p"
+qdev_free(void *dev, void *bus) "dev %p, bus %p"

Quoted 5 lines

This patch adds tracepoints at creating and removing virtual devices. It is useful for investigation of trouble related to virtual devices.

Signed-off-by Kazuya Saito <[email protected]>

I would prefer not to do this. I had previously posted a patch to remove qdev_free() in favor of using the QOM function object_unparent() directly, which adding stuff to qdev_free() would interfere with. And you should rather add a tracepoint to object_new() or better to object_initialize() than into the legacy qdev_create() - which doesn't cover qdev_try_create() btw. Either way, adding new tracepoints with the legacy "qdev" in the name is ugly.

Regards,
Andreas

P.S. Your patches arrived in HTML format, please check your workflow.

This series adds tracepoints for helping us clarify the cause of troubles. Virtualization on Linux is composed of some components such as qemu, kvm, libvirt, and so on. So it is very important to clarify firstly and swiftly the cause of troubles is on what component of them. Although qemu has useful information of this because it stands among kvm, libvirt and guest, it doesn't output the information by trace or log system.
These patches add tracepoints which lead to reduce the time of the clarification. We'd like to add the tracepoints as the first set because, based on our experience, we've found out they must be useful for an investigation in the future. Without those tracepoints, we had a really hard time investigating a problem since the problem's reproducibility was quite low and there was no clue in the dump of qemu.

Kazuya Saito (5):
vl: add runstate_set tracepoint
kvm-all: add kvm_ioctl, kvm_vm_ioctl, kvm_vcpu_ioctl tracepoints
kvm-all: add kvm_run_exit tracepoint
qdev: add qdev_{create,free} tracepoints
qdev-monitor: add device_add tracepoint

Il 22/03/2013 09:25, Kazuya Saito ha scritto:

Quoted 15 lines

This series adds tracepoints for helping us clarify the cause of troubles. Virtualization on Linux is composed of some components such as qemu, kvm, libvirt, and so on. So it is very important to clarify firstly and swiftly the cause of troubles is on what component of them. Although qemu has useful information of this because it stands among kvm, libvirt and guest, it doesn't output the information by trace or log system.
These patches add tracepoints which lead to reduce the time of the clarification. We'd like to add the tracepoints as the first set because, based on our experience, we've found out they must be useful for an investigation in the future. Without those tracepoints, we had a really hard time investigating a problem since the problem's reproducibility was quite low and there was no clue in the dump of qemu.

Quoted 6 lines

Kazuya Saito (5):
vl: add runstate_set tracepoint
kvm-all: add kvm_ioctl, kvm_vm_ioctl, kvm_vcpu_ioctl tracepoints
kvm-all: add kvm_run_exit tracepoint
qdev: add qdev_{create,free} tracepoints
qdev-monitor: add device_add tracepoint

I'm not sure 4-5 are that useful, but the first 3 patches are definitely good stuff.

Quoted 3 lines

Quoted 3 lines

I'm not sure 4-5 are that useful, but the first 3 patches are definitely good stuff.

Thanks. I'll modify the patch you pointed out about CPU number and re-post it.
I'd like to add tracepoints to the virtual device creation/deletion part. I had an issue that when a Windows guest OS booted up, it didn't have a virtual NIC device which it should have had. It took us time to figure out which the guest OS or the QEMU had the issue. If I have had the tracepoints in the virtual device creation/deletion should have eased the situation much better.

Il 26/03/2013 08:13, Kazuya Saito ha scritto:

Quoted 11 lines

Quoted 3 lines

Quoted 3 lines

I'm not sure 4-5 are that useful, but the first 3 patches are definitely good stuff.

Thanks. I'll modify the patch you pointed out about CPU number and re-post it.
I'd like to add tracepoints to the virtual device creation/deletion part. I had an issue that when a Windows guest OS booted up, it didn't have a virtual NIC device which it should have had. It took us time to figure out which the guest OS or the QEMU had the issue. If I have had the tracepoints in the virtual device creation/deletion should have eased the situation much better.

Wouldn't you get the same information from the command line?

Quoted 2 lines

Wouldn't you get the same information from the command line?

I think the information you said is different from what I meant. The information I wanted to know is whether QEMU creates/deletes a device successfully or not.

We cannot get it from the command line.
I was sure I specified a NIC device for the guest, but the Windows guest didn't have it when it booted. There were two possibilities. One was QEMU failed to create the NIC device, and the other was QEMU created it successfully and the Windows guest failed to detect it. Since QEMU output limited message at the moment, it was difficult to prove that it was not QEMU's issue. I took a coredump of the QEMU and proved the QEMU must have a valid structures for the NIC device. I believe the tracepoints could have allowed me to figure out where the issue existed a lot faster and easier.

Quoted 5 lines

Quoted 1 line

Wouldn't you get the same information from the command line?

I think the information you said is different from what I meant. The information I wanted to know is whether QEMU creates/deletes a device successfully or not.

Failing to create a device will always exit QEMU. If you cannot assume that, you're really in debugging territory and your tools should be the "info" monitor commands (info qtree, info pci) or gdb...

Quoted 10 lines

We cannot get it from the command line.
I was sure I specified a NIC device for the guest, but the Windows guest didn't have it when it booted. There were two possibilities. One was QEMU failed to create the NIC device, and the other was QEMU created it successfully and the Windows guest failed to detect it. Since QEMU output limited message at the moment, it was difficult to prove that it was not QEMU's issue. I took a coredump of the QEMU and proved the QEMU must have a valid structures for the NIC device. I believe the tracepoints could have allowed me to figure out where the issue existed a lot faster and easier.

Had you tried "info qtree" or "info pci"?

I didn't try these commands. I tried them just now and it seems to solve the problem. So I think no need to add the tracepoints I posted. Thank you for your kindness.

Il 28/03/2013 06:53, Kazuya Saito ha scritto:

Quoted 3 lines

I didn't try these commands. I tried them just now and it seems to solve the problem. So I think no need to add the tracepoints I posted. Thank you for your kindness.

Thanks to you!

Anthony or Stefan, can you pick up patch 1
(http://article.gmane.org/gmane.comp.emulators.qemu/201873/raw)?

Kazuya, please resubmit patches 2 and 3. Label them [PATCH uq/master] so that the KVM maintainers see them, and CC [email protected] too.

Paolo

Quoted 5 lines · Paolo Bonzini

Il 28/03/2013 06:53, Kazuya Saito ha scritto:

Quoted 3 lines

I didn't try these commands. I tried them just now and it seems to solve the problem. So I think no need to add the tracepoints I posted. Thank you for your kindness.

Quoted 4 lines

Thanks to you!

Anthony or Stefan, can you pick up patch 1
(http://article.gmane.org/gmane.comp.emulators.qemu/201873/raw)?

Stefan, please take this through the tracing tree.