trace: make systemtap easier to use for simple logging
v4 cover trace: make systemtap easier to use for simple logging
This is a followup to
v1: https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg04173.html
v2: https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg04759.html
v3: https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg05477.html
Changed in v4:
- Fix license to be GPLv2-or-later (Eric)
- Misc typo fixes (Eric)
Changed in v3:
- g_strndup the QXL log_buf before printing to avoid a race with guest removing the nul terminator (Stefan)
- Add filtering of PIDs with "-p PID" arg to qemu-trace-stap
- Add a man page for qemu-trace-stap (Eric)
- Remove examples in docs/devel/tracing.txt and just refer people to the above man page.
- Use strerror(errno) instead of raw errno in vfio traces (Eric/Alex)
- Quote probe wildcards to be shell glob safe (Eric)
- Mention explicitly that it is launched separately from QEMU (Eric)
Changed in v2:
- Fix safety of QXL logging
- Handle format specifier macros is a more reliable manner
- Fix trace-events files missing newline
- Remove use of %m formats
Daniel P. Berrangé (4):
display: ensure qxl log_buf is a nul terminated string
trace: enforce that every trace-events file has a final newline
trace: forbid use of %m in trace event format strings
trace: add ability to do simple printf logging via systemtap
MAINTAINERS | 1 + Makefile | 10 ++ Makefile.target | 11 +- docs/devel/tracing.txt | 4 + hw/display/qxl.c | 14 ++- hw/display/trace-events | 2 +- hw/gpio/trace-events | 2 +- hw/vfio/pci.c | 2 +- hw/vfio/trace-events | 2 +- scripts/qemu-trace-stap | 175 +++++++++++++++++++++++++++ scripts/qemu-trace-stap.texi | 140 +++++++++++++++++++++ scripts/tracetool/__init__.py | 6 + scripts/tracetool/format/log_stap.py | 127 +++++++++++++++++++ 13 files changed, 487 insertions(+), 9 deletions(-) create mode 100755 scripts/qemu-trace-stap create mode 100644 scripts/qemu-trace-stap.texi create mode 100644 scripts/tracetool/format/log_stap.py
Signature · 1 line
2.20.1
PATCH v4 1/4 display: ensure qxl log_buf is a nul terminated string
The QXL_IO_LOG command allows the guest to send log messages to the host via a buffer in the QXLRam struct. QEMU prints these to the console if the qxl 'guestdebug' option is set to non-zero. It will also feed them to the trace subsystem if any backends are built-in.
In both cases the log_buf data will get treated as being as a nul terminated string, by the printf '%s' format specifier and / or other code reading the buffer.
QEMU does nothing to guarantee that the log_buf really is nul terminated, so there is potential for out of bounds array access.
This would affect any QEMU which has the log, syslog or ftrace trace backends built into QEMU. It can only be triggered if the 'qxl_io_log' trace event is enabled, however, so they are not vulnerable without specific administrative action to enable this.
It would also affect QEMU if the 'guestdebug' parameter is set to a non-zero value, which again is not the default and requires explicit admin opt-in.
--- hw/display/qxl.c | 14 ++++++++++---- hw/display/trace-events | 2 +- 2 files changed, 11 insertions(+), 5 deletions(-)
hw/display/qxl.c +10 -4
@@ -1763,10 +1763,16 @@ async_common:qxl_set_mode (d, val,0 );break ;case QXL_IO_LOG : -trace_qxl_io_log (d->id, d->ram->log_buf); -if (d->guestdebug) { -fprintf (stderr,"qxl/guest-%d: %" PRId64": %s" , d->id, -qemu_clock_get_ns (QEMU_CLOCK_VIRTUAL ), d->ram->log_buf); +if (TRACE_QXL_IO_LOG_ENABLED || d->guestdebug) { +/* We cannot trust the guest to NUL terminate d->ram->log_buf */ +char *log_buf =g_strndup ((const char *)d->ram->log_buf, +sizeof (d->ram->log_buf)); +trace_qxl_io_log (d->id, log_buf); +if (d->guestdebug) { +fprintf (stderr,"qxl/guest-%d: %" PRId64": %s" , d->id, +qemu_clock_get_ns (QEMU_CLOCK_VIRTUAL ), log_buf); + } +g_free (log_buf); }break ;case QXL_IO_RESET :
hw/display/trace-events +1 -1
@@ -72,7 +72,7 @@ qxl_interface_update_area_complete_rest(int qid, uint32_t num_updated_rects) "%d qxl_interface_update_area_complete_overflow(int qid, int max) "%d max=%d" qxl_interface_update_area_complete_schedule_bh(int qid, uint32_t num_dirty) "%d #dirty=%d" qxl_io_destroy_primary_ignored(int qid, const char *mode) "%d %s" -qxl_io_log(int qid, const uint8_t *log_buf) "%d %s" +qxl_io_log(int qid, const char *log_buf) "%d %s" qxl_io_read_unexpected(int qid) "%d" qxl_io_unexpected_vga_mode(int qid, uint64_t addr, uint64_t val, const char *desc) "%d 0x%"PRIx64"=%"PRIu64" (%s)" qxl_io_write(int qid, const char *mode, uint64_t addr, const char *aname, uint64_t val, unsigned size, int async) "%d %s addr=%"PRIu64 " (%s) val=%"PRIu64" size=%u async=%d"
Signature · 1 line
2.20.1
PATCH v4 2/4 trace: enforce that every trace-events file has a final newline
In reply to Daniel P. BerrangéWhen generating the trace-events-all file, the build system simply concatenates all the individual trace-events files. If any one of those files does not have a final newline, the printf format string will have the contents of the first line of the next file appended to it, which is usually a '#' comment.
--- hw/gpio/trace-events | 2 +- scripts/tracetool/__init__.py | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-)
hw/gpio/trace-events +1 -1
@@ -4,4 +4,4 @@ nrf51_gpio_read(uint64_t offset, uint64_t r) "offset 0x%" PRIx64 " value 0x%" PRIx64 nrf51_gpio_write(uint64_t offset, uint64_t value) "offset 0x%" PRIx64 " value 0x%" PRIx64 nrf51_gpio_set(int64_t line, int64_t value) "line %" PRIi64 " value %" PRIi64 -nrf51_gpio_update_output_irq(int64_t line, int64_t value) "line %" PRIi64 " value %" PRIi64 +nrf51_gpio_update_output_irq(int64_t line, int64_t value) "line %" PRIi64 " value %" PRIi64
scripts/tracetool/__init__.py +2 -0
@@ -350,6 +350,8 @@ def read_events(fobj, fname): events = [] for lineno, line in enumerate(fobj, 1): + if line[-1] != '\n': + raise ValueError("%s does not end with a new line" % fname) if not line.strip(): continue if line.lstrip().startswith('#'):
Signature · 1 line
2.20.1
PATCH v4 3/4 trace: forbid use of %m in trace event format strings
In reply to Daniel P. BerrangéThe '%m' format instructs glibc's printf()/syslog() implementation to insert the contents of strerror(errno). Since this is a glibc extension it should generally be avoided in QEMU due to need for portability to a variety of platforms.
Even though vfio is Linux-only code that could otherwise use "%m", it must still be avoided in trace-events files because several of the backends do not use the format string and so this error information is invisible to them.
The errno string value should be given as an explicit trace argument instead, making it accessible to all backends. This also allows it to work correctly with future patches that use the format string with systemtap's simple printf code.
--- hw/vfio/pci.c | 2 +- hw/vfio/trace-events | 2 +- scripts/tracetool/__init__.py | 4 ++++ 3 files changed, 6 insertions(+), 2 deletions(-)
hw/vfio/pci.c +1 -1
@@ -2581,7 +2581,7 @@ static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) ret =ioctl (vdev->vbasedev.fd,VFIO_DEVICE_GET_IRQ_INFO , &irq_info);if (ret) {/* This can fail for an old kernel or legacy PCI dev */ -trace_vfio_populate_device_get_irq_info_failure (); +trace_vfio_populate_device_get_irq_info_failure (strerror (errno)); }else if (irq_info.count ==1 ) { vdev->pci_aer = true; }else {
hw/vfio/trace-events +1 -1
@@ -37,7 +37,7 @@ vfio_pci_hot_reset_has_dep_devices(const char *name) "%s: hot reset dependent de vfio_pci_hot_reset_dep_devices(int domain, int bus, int slot, int function, int group_id) "\t%04x:%02x:%02x.%x group %d" vfio_pci_hot_reset_result(const char *name, const char *result) "%s hot reset: %s" vfio_populate_device_config(const char *name, unsigned long size, unsigned long offset, unsigned long flags) "Device %s config:\n size: 0x%lx, offset: 0x%lx, flags: 0x%lx" -vfio_populate_device_get_irq_info_failure(void) "VFIO_DEVICE_GET_IRQ_INFO failure: %m" +vfio_populate_device_get_irq_info_failure(const char *errstr) "VFIO_DEVICE_GET_IRQ_INFO failure: %s" vfio_realize(const char *name, int group_id) " (%s) group %d" vfio_mdev(const char *name, bool is_mdev) " (%s) is_mdev %d" vfio_add_ext_cap_dropped(const char *name, uint16_t cap, uint16_t offset) "%s 0x%x@0x%x"
scripts/tracetool/__init__.py +4 -0
@@ -274,6 +274,10 @@ class Event(object): props = groups["props"].split() fmt = groups["fmt"] fmt_trans = groups["fmt_trans"] + if fmt.find("%m") != -1 or fmt_trans.find("%m") != -1: + raise ValueError("Event format '%m' is forbidden, pass the error " + "as an explicit trace argument") + if len(fmt_trans) > 0: fmt = [fmt_trans, fmt] args = Arguments.build(groups["args"])
Signature · 1 line
2.20.1
PATCH v4 4/4 trace: add ability to do simple printf logging via systemtap
In reply to Daniel P. BerrangéThe dtrace systemtap trace backend for QEMU is very powerful but it is also somewhat unfriendly to users who aren't familiar with systemtap, or who don't need its power right now.
stap -e "....some strange script...."
The 'log' backend for QEMU by comparison is very crude but incredibly easy to use:
$ qemu -d trace:qio* ...some args... [email protected]:qio_channel_socket_new Socket new ioc=0x563a8a39d400 [email protected]:qio_task_new Task new task=0x563a891d0570 source=0x563a8a39d400 func=0x563a86f1e6c0 opaque=0x563a89078000 [email protected]:qio_task_thread_start Task thread start task=0x563a891d0570 worker=0x563a86f1ce50 opaque=0x563a891d9d90 [email protected]:qio_task_thread_run Task thread run task=0x563a891d0570 [email protected]:qio_channel_socket_connect_sync Socket connect sync ioc=0x563a8a39d400 addr=0x563a891d9d90 [email protected]:qio_channel_socket_connect_fail Socket connect fail ioc=0x563a8a39d400
This commit introduces a way to do simple printf style logging of probe points using systemtap. In particular it creates another set of tapsets, one per emulator:
/usr/share/systemtap/tapset/qemu-*-log.stp
These pre-define probe functions which simply call printf() on their arguments. The printf() format string is taken from the normal trace-events files, with a little munging to the format specifiers to cope with systemtap's more restrictive syntax.
With this you can now do
$ stap -e 'probe qemu.system.x86_64.log.qio*{}'
22806@1547735341399856820 qio_channel_socket_new Socket new ioc=0x56135d1d7c00
22806@1547735341399862570 qio_task_new Task new task=0x56135cd66eb0 source=0x56135d1d7c00 func=0x56135af746c0 opaque=0x56135bf06400
22806@1547735341399865943 qio_task_thread_start Task thread start task=0x56135cd66eb0 worker=0x56135af72e50 opaque=0x56135c071d70
22806@1547735341399976816 qio_task_thread_run Task thread run task=0x56135cd66eb0We go one step further though and introduce a 'qemu-trace-stap' tool to make this even easier
$ qemu-trace-stap run qemu-system-x86_64 'qio*' 22806@1547735341399856820 qio_channel_socket_new Socket new ioc=0x56135d1d7c00 22806@1547735341399862570 qio_task_new Task new task=0x56135cd66eb0 source=0x56135d1d7c00 func=0x56135af746c0 opaque=0x56135bf06400 22806@1547735341399865943 qio_task_thread_start Task thread start task=0x56135cd66eb0 worker=0x56135af72e50 opaque=0x56135c071d70 22806@1547735341399976816 qio_task_thread_run Task thread run task=0x56135cd66eb0
This tool is clever in that it will automatically change the SYSTEMTAP_TAPSET env variable to point to the directory containing the right set of probes for the QEMU binary path you give it. This is useful if you have QEMU installed in /usr but are trying to test and trace a binary in /home/berrange/usr/qemu-git. In that case you'd do
$ qemu-trace-stap run /home/berrange/usr/qemu-git/bin/qemu-system-x86_64 'qio*'
And it'll make sure /home/berrange/usr/qemu-git/share/systemtap/tapset is used for the trace session
The 'qemu-trace-stap' script takes a verbose arg so you can understand what it is running
$ qemu-trace-stap run /home/berrange/usr/qemu-git/bin/qemu-system-x86_64 'qio*'
Using tapset dir '/home/berrange/usr/qemu-git/share/systemtap/tapset' for binary '/home/berrange/usr/qemu-git/bin/qemu-system-x86_64'
Compiling script 'probe qemu.system.x86_64.log.qio* {}'
Running script, <Ctrl>-c to quit
...trace output...It can enable multiple probes at once
$ qemu-trace-stap run qemu-system-x86_64 'qio*' 'qcrypto*' 'buffer*'
By default it monitors all existing running processes and all future launched proceses. This can be restricted to a specific PID using the --pid arg
$ qemu-trace-stap run --pid 2532 qemu-system-x86_64 'qio*'
Finally if you can't remember what probes are valid it can tell you
$ qemu-trace-stap list qemu-system-x86_64 ahci_check_irq ahci_cmd_done ahci_dma_prepare_buf ahci_dma_prepare_buf_fail ahci_dma_rw_buf ahci_irq_lower ...snip...
Or list just those matching a prefix pattern
$ qemu-trace-stap list -v qemu-system-x86_64 'qio*' Using tapset dir '/home/berrange/usr/qemu-git/share/systemtap/tapset' for binary '/home/berrange/usr/qemu-git/bin/qemu-system-x86_64' Listing probes with name 'qemu.system.x86_64.log.qio*' qio_channel_command_abort qio_channel_command_new_pid qio_channel_command_new_spawn qio_channel_command_wait qio_channel_file_new_fd ...snip...
--- MAINTAINERS | 1 + Makefile | 10 ++ Makefile.target | 11 +- docs/devel/tracing.txt | 4 + scripts/qemu-trace-stap | 175 +++++++++++++++++++++++++++ scripts/qemu-trace-stap.texi | 140 +++++++++++++++++++++ scripts/tracetool/format/log_stap.py | 127 +++++++++++++++++++ 7 files changed, 467 insertions(+), 1 deletion(-) create mode 100755 scripts/qemu-trace-stap create mode 100644 scripts/qemu-trace-stap.texi create mode 100644 scripts/tracetool/format/log_stap.py
MAINTAINERS +1 -0
@@ -2017,6 +2017,7 @@ F: trace-events F: qemu-option-trace.texi F: scripts/tracetool.py F: scripts/tracetool/ +F: scripts/qemu-trace-stap* F: docs/devel/tracing.txt T: git https://github.com/stefanha/qemu.git tracing
Makefile +10 -0
@@ -305,6 +305,9 @@ DOCS+=docs/qemu-cpu-models.7 ifdef CONFIG_VIRTFS DOCS+=fsdev/virtfs-proxy-helper.1 endif +ifdef CONFIG_TRACE_SYSTEMTAP +DOCS+=scripts/qemu-trace-stap.1 +endif else DOCS= endif
@@ -700,6 +703,9 @@ ifneq ($(TOOLS),) $(INSTALL_DIR) "$(DESTDIR)$(mandir)/man8" $(INSTALL_DATA) qemu-nbd.8 "$(DESTDIR)$(mandir)/man8" endif +ifdef CONFIG_TRACE_SYSTEMTAP + $(INSTALL_DATA) scripts/qemu-trace-stap.1 "$(DESTDIR)$(mandir)/man1" +endif ifneq (,$(findstring qemu-ga,$(TOOLS))) $(INSTALL_DATA) qemu-ga.8 "$(DESTDIR)$(mandir)/man8" $(INSTALL_DATA) docs/interop/qemu-ga-ref.html "$(DESTDIR)$(qemu_docdir)"
@@ -738,6 +744,9 @@ endif ifneq ($(HELPERS-y),) $(call install-prog,$(HELPERS-y),$(DESTDIR)$(libexecdir)) endif +ifdef CONFIG_TRACE_SYSTEMTAP + $(INSTALL_PROG) "scripts/qemu-trace-stap" $(DESTDIR)$(bindir) +endif ifneq ($(BLOBS),) set -e; for x in $(BLOBS); do \ $(INSTALL_DATA) $(SRC_PATH)/pc-bios/$$x "$(DESTDIR)$(qemu_datadir)"; \
@@ -838,6 +847,7 @@ qemu-nbd.8: qemu-nbd.texi qemu-option-trace.texi qemu-ga.8: qemu-ga.texi docs/qemu-block-drivers.7: docs/qemu-block-drivers.texi docs/qemu-cpu-models.7: docs/qemu-cpu-models.texi +scripts/qemu-trace-stap.1: scripts/qemu-trace-stap.texi html: qemu-doc.html docs/interop/qemu-qmp-ref.html docs/interop/qemu-ga-ref.html info: qemu-doc.info docs/interop/qemu-qmp-ref.info docs/interop/qemu-ga-ref.info
Makefile.target +10 -1
@@ -45,7 +45,7 @@ config-target.h: config-target.h-timestamp config-target.h-timestamp: config-target.mak ifdef CONFIG_TRACE_SYSTEMTAP -stap: $(QEMU_PROG).stp-installed $(QEMU_PROG).stp $(QEMU_PROG)-simpletrace.stp +stap: $(QEMU_PROG).stp-installed $(QEMU_PROG).stp $(QEMU_PROG)-simpletrace.stp $(QEMU_PROG)-log.stp ifdef CONFIG_USER_ONLY TARGET_TYPE=user
@@ -84,6 +84,14 @@ $(QEMU_PROG)-simpletrace.stp: $(BUILD_DIR)/trace-events-all $(tracetool-y) --probe-prefix=qemu.$(TARGET_TYPE).$(TARGET_NAME) \ $< > $@,"GEN","$(TARGET_DIR)$(QEMU_PROG)-simpletrace.stp") +$(QEMU_PROG)-log.stp: $(BUILD_DIR)/trace-events-all $(tracetool-y) + $(call quiet-command,$(TRACETOOL) \ + --group=all \ + --format=log-stap \ + --backends=$(TRACE_BACKENDS) \ + --probe-prefix=qemu.$(TARGET_TYPE).$(TARGET_NAME) \ + $< > $@,"GEN","$(TARGET_DIR)$(QEMU_PROG)-log.stp") + else stap: endif
@@ -227,6 +235,7 @@ ifdef CONFIG_TRACE_SYSTEMTAP $(INSTALL_DIR) "$(DESTDIR)$(qemu_datadir)/../systemtap/tapset" $(INSTALL_DATA) $(QEMU_PROG).stp-installed "$(DESTDIR)$(qemu_datadir)/../systemtap/tapset/$(QEMU_PROG).stp" $(INSTALL_DATA) $(QEMU_PROG)-simpletrace.stp "$(DESTDIR)$(qemu_datadir)/../systemtap/tapset/$(QEMU_PROG)-simpletrace.stp" + $(INSTALL_DATA) $(QEMU_PROG)-log.stp "$(DESTDIR)$(qemu_datadir)/../systemtap/tapset/$(QEMU_PROG)-log.stp" endif GENERATED_FILES += config-target.h
docs/devel/tracing.txt +4 -0
@@ -317,6 +317,10 @@ probes: --target-name x86_64 \ <trace-events-all >qemu.stp +To facilitate simple usage of systemtap where there merely needs to be printf +logging of certain probes, a helper script "qemu-trace-stap" is provided. +Consult its manual page for guidance on its usage. + == Trace event properties == Each event in the "trace-events-all" file can be prefixed with a space-separated
scripts/qemu-trace-stap new file +175 -0
175 lines not on this page · raw message
scripts/qemu-trace-stap.texi new file +140 -0
140 lines not on this page · raw message
scripts/tracetool/format/log_stap.py new file +127 -0
127 lines not on this page · raw message
Signature · 1 line
2.20.1
Stefan Hajnoczi
AppliedQuoted 45 lines · Daniel P. Berrangé
This is a followup to
v1: https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg04173.html
v2: https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg04759.html
v3: https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg05477.htmlChanged in v4:
- Fix license to be GPLv2-or-later (Eric)
- Misc typo fixes (Eric)Changed in v3:
-
g_strndupthe QXLlog_bufbefore printing to avoid a race with guest removing the nul terminator (Stefan)
- Add filtering of PIDs with "-p PID" arg to qemu-trace-stap
- Add a man page for qemu-trace-stap (Eric)
- Remove examples indocs/devel/tracing.txtand just refer people to the above man page.
- Use strerror(errno) instead of raw errno in vfio traces (Eric/Alex)
- Quote probe wildcards to be shell glob safe (Eric)
- Mention explicitly that it is launched separately from QEMU (Eric)Changed in v2:
- Fix safety of QXL logging
- Handle format specifier macros is a more reliable manner
- Fix trace-events files missing newline
- Remove use of %m formatsDaniel P. Berrangé (4):
display: ensure qxllog_bufis a nul terminated string
trace: enforce that every trace-events file has a final newline
trace: forbid use of %m in trace event format strings
trace: add ability to do simple printf logging via systemtapMAINTAINERS | 1 + Makefile | 10 ++ Makefile.target | 11 +- docs/devel/tracing.txt | 4 + hw/display/qxl.c | 14 ++- hw/display/trace-events | 2 +- hw/gpio/trace-events | 2 +- hw/vfio/pci.c | 2 +- hw/vfio/trace-events | 2 +-
Quoted 12 lines
scripts/qemu-trace-stap | 175 +++++++++++++++++++++++++++ scripts/qemu-trace-stap.texi | 140 +++++++++++++++++++++ scripts/tracetool/__init__.py | 6 + scripts/tracetool/format/log_stap.py | 127 +++++++++++++++++++ 13 files changed, 487 insertions(+), 9 deletions(-) create mode 100755 scripts/qemu-trace-stap create mode 100644 scripts/qemu-trace-stap.texi create mode 100644 scripts/tracetool/format/log_stap.pySignature · 1 line
2.20.1
Thanks, applied to my tracing tree:
https://github.com/stefanha/qemu/commits/tracing
Stefan