Skip to contentSkip to conversation

trace: add ability to do simple printf logging via systemtap

PATCH Applied Daniel P. Berrangé 2 participants 4 messages Jan 17, 2019 – Jan 18, 2019 Kbuild

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 trace: add ability to do simple printf logging via systemtap

The dtrace systemtap trace backend for QEMU is very powerful but it is also somewhat unfriendly to users who aren't familiar with systemtap, or who don't need its power right now.

stap -e "....some strange script...."

The 'log' backend for QEMU by comparison is very crude but incredibly easy to use:

 $ qemu -d trace:qio* ...some args...
 [email protected]:qio_channel_socket_new Socket new ioc=0x563a8a39d400
 [email protected]:qio_task_new Task new task=0x563a891d0570 source=0x563a8a39d400 func=0x563a86f1e6c0 opaque=0x563a89078000
 [email protected]:qio_task_thread_start Task thread start task=0x563a891d0570 worker=0x563a86f1ce50 opaque=0x563a891d9d90
 [email protected]:qio_task_thread_run Task thread run task=0x563a891d0570
 [email protected]:qio_channel_socket_connect_sync Socket connect sync ioc=0x563a8a39d400 addr=0x563a891d9d90
 [email protected]:qio_channel_socket_connect_fail Socket connect fail ioc=0x563a8a39d400

This commit introduces a way to do simple printf style logging of probe points using systemtap. In particular it creates another set of tapsets, one per emulator:

/usr/share/systemtap/tapset/qemu-*-log.stp

These pre-define probe functions which simply call printf() on their arguments. The printf() format string is taken from the normal trace-events files, with a little munging to the format specifiers to cope with systemtap's more restrictive syntax.

With this you can now do

 $ stap -e 'probe qemu.system.x86_64.log.qio*{}'
 22806@1547735341399856820 qio_channel_socket_new Socket new ioc=0x56135d1d7c00
 22806@1547735341399862570 qio_task_new Task new task=0x56135cd66eb0 source=0x56135d1d7c00 func=0x56135af746c0 opaque=0x56135bf06400
 22806@1547735341399865943 qio_task_thread_start Task thread start task=0x56135cd66eb0 worker=0x56135af72e50 opaque=0x56135c071d70
 22806@1547735341399976816 qio_task_thread_run Task thread run task=0x56135cd66eb0

We go one step further though and introduce a 'qemu-trace-stap' tool to make this even easier

 $ qemu-trace-stap run qemu-system-x86_64 qio*
 22806@1547735341399856820 qio_channel_socket_new Socket new ioc=0x56135d1d7c00
 22806@1547735341399862570 qio_task_new Task new task=0x56135cd66eb0 source=0x56135d1d7c00 func=0x56135af746c0 opaque=0x56135bf06400
 22806@1547735341399865943 qio_task_thread_start Task thread start task=0x56135cd66eb0 worker=0x56135af72e50 opaque=0x56135c071d70
 22806@1547735341399976816 qio_task_thread_run Task thread run task=0x56135cd66eb0

This tool is clever in that it will automatically change the SYSTEMTAP_TAPSET env variable to point to the directory containing the right set of probes for the QEMU binary path you give it. This is useful if you have QEMU installed in /usr but are trying to test and trace a binary in /home/berrange/usr/qemu-git. In that case you'd do

 $ qemu-trace-stap run /home/berrange/usr/qemu-git/bin/qemu-system-x86_64 qio*

And it'll make sure /home/berrange/usr/qemu-git/share/systemtap/tapset is used for the trace session

The 'qemu-trace-stap' script takes a verbose arg so you can understand what it is running

 $ qemu-trace-stap run /home/berrange/usr/qemu-git/bin/qemu-system-x86_64 qio*
 Using tapset dir '/home/berrange/usr/qemu-git/share/systemtap/tapset' for binary '/home/berrange/usr/qemu-git/bin/qemu-system-x86_64'
 Compiling script 'probe qemu.system.x86_64.log.qio* {}'
 Running script, <Ctrl>-c to quit
 ...trace output...

It can enable multiple probes at once

 $ qemu-trace-stap run qemu-system-x86_64 qio* qcrypto* buffer*

Finally if you can't remember what probes are valid it can tell you

 $ qemu-trace-stap list qemu-system-x86_64
 ahci_check_irq
 ahci_cmd_done
 ahci_dma_prepare_buf
 ahci_dma_prepare_buf_fail
 ahci_dma_rw_buf
 ahci_irq_lower
 ...snip...

Or list just those matching a prefix pattern

 $ qemu-trace-stap list -v qemu-system-x86_64 qio*
 Using tapset dir '/home/berrange/usr/qemu-git/share/systemtap/tapset' for binary '/home/berrange/usr/qemu-git/bin/qemu-system-x86_64'
 Listing probes with name 'qemu.system.x86_64.log.qio*'
 qio_channel_command_abort
 qio_channel_command_new_pid
 qio_channel_command_new_spawn
 qio_channel_command_wait
 qio_channel_file_new_fd
 ...snip...
---
 Makefile                             |   3 +
 Makefile.target                      |  11 +-
 scripts/qemu-trace-stap              | 159 +++++++++++++++++++++++++++
 scripts/tracetool/format/log_stap.py |  75 +++++++++++++
 4 files changed, 247 insertions(+), 1 deletion(-)
 create mode 100755 scripts/qemu-trace-stap
 create mode 100644 scripts/tracetool/format/log_stap.py
Makefile +3 -0
index a9ac16d94e..f29c1fbe43 100644
@@ -735,6 +735,9 @@ endif
 ifneq ($(HELPERS-y),)
 	$(call install-prog,$(HELPERS-y),$(DESTDIR)$(libexecdir))
 endif
+ifdef CONFIG_TRACE_SYSTEMTAP
+	$(INSTALL_PROG) "scripts/qemu-trace-stap" $(DESTDIR)$(bindir)
+endif
 ifneq ($(BLOBS),)
 	set -e; for x in $(BLOBS); do \
 		$(INSTALL_DATA) $(SRC_PATH)/pc-bios/$$x "$(DESTDIR)$(qemu_datadir)"; \
Makefile.target +10 -1
index 44ec4b630c..cb85eb450c 100644
@@ -45,7 +45,7 @@ config-target.h: config-target.h-timestamp
 config-target.h-timestamp: config-target.mak
 
 ifdef CONFIG_TRACE_SYSTEMTAP
-stap: $(QEMU_PROG).stp-installed $(QEMU_PROG).stp $(QEMU_PROG)-simpletrace.stp
+stap: $(QEMU_PROG).stp-installed $(QEMU_PROG).stp $(QEMU_PROG)-simpletrace.stp $(QEMU_PROG)-log.stp
 
 ifdef CONFIG_USER_ONLY
 TARGET_TYPE=user
@@ -84,6 +84,14 @@ $(QEMU_PROG)-simpletrace.stp: $(BUILD_DIR)/trace-events-all $(tracetool-y)
 		--probe-prefix=qemu.$(TARGET_TYPE).$(TARGET_NAME) \
 		$< > $@,"GEN","$(TARGET_DIR)$(QEMU_PROG)-simpletrace.stp")
 
+$(QEMU_PROG)-log.stp: $(BUILD_DIR)/trace-events-all $(tracetool-y)
+	$(call quiet-command,$(TRACETOOL) \
+		--group=all \
+		--format=log-stap \
+		--backends=$(TRACE_BACKENDS) \
+		--probe-prefix=qemu.$(TARGET_TYPE).$(TARGET_NAME) \
+		$< > $@,"GEN","$(TARGET_DIR)$(QEMU_PROG)-log.stp")
+
 else
 stap:
 endif
@@ -224,6 +232,7 @@ ifdef CONFIG_TRACE_SYSTEMTAP
 	$(INSTALL_DIR) "$(DESTDIR)$(qemu_datadir)/../systemtap/tapset"
 	$(INSTALL_DATA) $(QEMU_PROG).stp-installed "$(DESTDIR)$(qemu_datadir)/../systemtap/tapset/$(QEMU_PROG).stp"
 	$(INSTALL_DATA) $(QEMU_PROG)-simpletrace.stp "$(DESTDIR)$(qemu_datadir)/../systemtap/tapset/$(QEMU_PROG)-simpletrace.stp"
+	$(INSTALL_DATA) $(QEMU_PROG)-log.stp "$(DESTDIR)$(qemu_datadir)/../systemtap/tapset/$(QEMU_PROG)-log.stp"
 endif
 
 GENERATED_FILES += config-target.h
scripts/qemu-trace-stap new file +159 -0

159 lines not on this page · raw message

scripts/tracetool/format/log_stap.py new file +75 -0

75 lines not on this page · raw message

Signature · 1 line
2.20.1
#2 lore raw
Quoted 4 lines · Stefan Hajnoczi

On Thu, Jan 17, 2019 at 02:52:07PM +0000, Daniel P. Berrangé wrote:

Nice improvement, thanks! Do you want to send a docs/devel/tracing.txt follow-up patch to document this feature?

Ok, will do.

Quoted 5 lines

Thanks, applied to my tracing tree:
https://github.com/stefanha/qemu/commits/tracing

Stefan

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#3 lore raw
In reply to Daniel P. Berrangé

On Thu, Jan 17, 2019 at 02:52:07PM +0000, Daniel P. Berrangé wrote:

Dequeued. I encountered two issues during testing:

1. s/binary/file/ in qemu-trace-stap's which() function:

scripts/qemu-trace-stap +1 -1
index b2eb41fbb0..3b4ec5e9b9 100755
@@ -37,7 +37,7 @@ def which(file):
         if os.path.exists(os.path.join(path, file)):
                 return os.path.join(path, file)
 
-    print("Unable to find '%s' in $PATH" % binary)
+    print("Unable to find '%s' in $PATH" % file)
     sys.exit(1)
 
 

2. The tpm_crb_mmio_read trace event's TARGET_FMT_plx format specifier macro isn't handled. stap sees a literal "TARGET_FMT_plx" string and reports that the number of format string arguments is incorrect:

semantic error: Wrong number of args to formatted print operator: identifier 'printf' at :9189:5
source: printf("%d@%d tpm_crb_mmio_read CRB read 0x TARGET_FMT_plx len:%u val: 0x%x\n", pid(), gettimeofday_ns(), addr, size, val)