Skip to contentSkip to conversation

trace: make systemtap easier to use for simple logging

PATCH Superseded Daniel P. Berrangé 6 participants 25 messages Jan 18, 2019 – Oct 18, 2019 Kbuild

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v2 cover trace: make systemtap easier to use for simple logging

This is a followup to

v1: https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg04173.html

Changed in v2:

- Fix safety of QXL logging
- Handle format specifier macros is a more reliable manner
- Fix trace-events files missing newline
- Remove use of %m formats

Daniel P. Berrangé (4):
display: ensure qxl log_buf is a nul terminated string
trace: enforce that every trace-events file has a final newline
trace: forbid use of %m in trace event format strings
trace: add ability to do simple printf logging via systemtap

 Makefile                             |   3 +
 Makefile.target                      |  11 +-
 docs/devel/tracing.txt               |  32 +++++
 hw/display/qxl.c                     |   3 +-
 hw/display/trace-events              |   2 +-
 hw/gpio/trace-events                 |   2 +-
 hw/vfio/pci.c                        |   2 +-
 hw/vfio/trace-events                 |   2 +-
 scripts/qemu-trace-stap              | 167 +++++++++++++++++++++++++++
 scripts/tracetool/__init__.py        |   6 +
 scripts/tracetool/format/log_stap.py | 127 ++++++++++++++++++++
 11 files changed, 351 insertions(+), 6 deletions(-)
 create mode 100755 scripts/qemu-trace-stap
 create mode 100644 scripts/tracetool/format/log_stap.py
Signature · 1 line
2.20.1
#1 lore raw

PATCH v2 1/4 display: ensure qxl log_buf is a nul terminated string

The QXL_IO_LOG command allows the guest to send log messages to the host via a buffer in the QXLRam struct. QEMU prints these to the console if the qxl 'guestdebug' option is set to non-zero. It will also feed them to the trace subsystem if any backends are built-in.

In both cases the log_buf data will get treated as being as a nul terminated string, by the printf '%s' format specifier and / or other code reading the buffer.

QEMU does nothing to guarantee that the log_buf really is nul terminated, so there is potential for out of bounds array access.

This would affect any QEMU which has the log, syslog or ftrace trace backends built into QEMU. It can only be triggered if the 'qxl_io_log' trace event is enabled, however, so they are not vulnerable without specific administrative action to enable this.

It would also affect QEMU if the 'guestdebug' parameter is set to a non-zero value, which again is not the default and requires explicit admin opt-in.

---
 hw/display/qxl.c        | 3 ++-
 hw/display/trace-events | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)
hw/display/qxl.c +2 -1
index 8e9a65e75b..eefdf4baac 100644
@@ -1763,7 +1763,8 @@ async_common:
         qxl_set_mode(d, val, 0);
         break;
     case QXL_IO_LOG:
-        trace_qxl_io_log(d->id, d->ram->log_buf);
+        d->ram->log_buf[sizeof(d->ram->log_buf) - 1] = '\0';
+        trace_qxl_io_log(d->id, (const char *)d->ram->log_buf);
         if (d->guestdebug) {
             fprintf(stderr, "qxl/guest-%d: %" PRId64 ": %s", d->id,
                     qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL), d->ram->log_buf);
hw/display/trace-events +1 -1
index 5a48c6cb6a..387c6b8931 100644
@@ -72,7 +72,7 @@ qxl_interface_update_area_complete_rest(int qid, uint32_t num_updated_rects) "%d
 qxl_interface_update_area_complete_overflow(int qid, int max) "%d max=%d"
 qxl_interface_update_area_complete_schedule_bh(int qid, uint32_t num_dirty) "%d #dirty=%d"
 qxl_io_destroy_primary_ignored(int qid, const char *mode) "%d %s"
-qxl_io_log(int qid, const uint8_t *log_buf) "%d %s"
+qxl_io_log(int qid, const char *log_buf) "%d %s"
 qxl_io_read_unexpected(int qid) "%d"
 qxl_io_unexpected_vga_mode(int qid, uint64_t addr, uint64_t val, const char *desc) "%d 0x%"PRIx64"=%"PRIu64" (%s)"
 qxl_io_write(int qid, const char *mode, uint64_t addr, const char *aname, uint64_t val, unsigned size, int async) "%d %s addr=%"PRIu64 " (%s) val=%"PRIu64" size=%u async=%d"
Signature · 1 line
2.20.1
#7 lore raw

PATCH v2 2/4 trace: enforce that every trace-events file has a final newline

In reply to Daniel P. Berrangé

When generating the trace-events-all file, the build system simply concatenates all the individual trace-events files. If any one of those files does not have a final newline, the printf format string will have the contents of the first line of the next file appended to it, which is usually a '#' comment.

---
 hw/gpio/trace-events          | 2 +-
 scripts/tracetool/__init__.py | 2 ++
 2 files changed, 3 insertions(+), 1 deletion(-)
hw/gpio/trace-events +1 -1
index cb41a89756..5d4dd200c2 100644
@@ -4,4 +4,4 @@
 nrf51_gpio_read(uint64_t offset, uint64_t r) "offset 0x%" PRIx64 " value 0x%" PRIx64
 nrf51_gpio_write(uint64_t offset, uint64_t value) "offset 0x%" PRIx64 " value 0x%" PRIx64
 nrf51_gpio_set(int64_t line, int64_t value) "line %" PRIi64 " value %" PRIi64
-nrf51_gpio_update_output_irq(int64_t line, int64_t value) "line %" PRIi64 " value %" PRIi64
\ No newline at end of file
+nrf51_gpio_update_output_irq(int64_t line, int64_t value) "line %" PRIi64 " value %" PRIi64
scripts/tracetool/__init__.py +2 -0
index 0e3c9e146c..3478ac93ab 100644
@@ -350,6 +350,8 @@ def read_events(fobj, fname):
 
     events = []
     for lineno, line in enumerate(fobj, 1):
+        if line[-1] != '\n':
+            raise ValueError("%s does not end with a new line" % fname)
         if not line.strip():
             continue
         if line.lstrip().startswith('#'):
Signature · 1 line
2.20.1
#11 lore raw

PATCH v2 3/4 trace: forbid use of %m in trace event format strings

In reply to Daniel P. Berrangé

The '%m' format specifier instructs glibc's printf() implementation to insert the contents of strerror(errno). This is not something that should ever be used in trace-events files because several of the backends do not use the format string and so this error information is invisible to them. The errno value should be given as an explicit trace argument instead. Use of '%m' should also be avoided as it is not portable to all QEMU build targets.

---
 hw/vfio/pci.c                 | 2 +-
 hw/vfio/trace-events          | 2 +-
 scripts/tracetool/__init__.py | 4 ++++
 3 files changed, 6 insertions(+), 2 deletions(-)
hw/vfio/pci.c +1 -1
index c0cb1ec289..85f1908cfe 100644
@@ -2581,7 +2581,7 @@ static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp)
     ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info);
     if (ret) {
         /* This can fail for an old kernel or legacy PCI dev */
-        trace_vfio_populate_device_get_irq_info_failure();
+        trace_vfio_populate_device_get_irq_info_failure(errno);
     } else if (irq_info.count == 1) {
         vdev->pci_aer = true;
     } else {
hw/vfio/trace-events +1 -1
index a85e8662ea..6d412afc83 100644
@@ -37,7 +37,7 @@ vfio_pci_hot_reset_has_dep_devices(const char *name) "%s: hot reset dependent de
 vfio_pci_hot_reset_dep_devices(int domain, int bus, int slot, int function, int group_id) "\t%04x:%02x:%02x.%x group %d"
 vfio_pci_hot_reset_result(const char *name, const char *result) "%s hot reset: %s"
 vfio_populate_device_config(const char *name, unsigned long size, unsigned long offset, unsigned long flags) "Device %s config:\n  size: 0x%lx, offset: 0x%lx, flags: 0x%lx"
-vfio_populate_device_get_irq_info_failure(void) "VFIO_DEVICE_GET_IRQ_INFO failure: %m"
+vfio_populate_device_get_irq_info_failure(int err) "VFIO_DEVICE_GET_IRQ_INFO failure: %d"
 vfio_realize(const char *name, int group_id) " (%s) group %d"
 vfio_mdev(const char *name, bool is_mdev) " (%s) is_mdev %d"
 vfio_add_ext_cap_dropped(const char *name, uint16_t cap, uint16_t offset) "%s 0x%x@0x%x"
scripts/tracetool/__init__.py +4 -0
index 3478ac93ab..6fca674936 100644
@@ -274,6 +274,10 @@ class Event(object):
         props = groups["props"].split()
         fmt = groups["fmt"]
         fmt_trans = groups["fmt_trans"]
+        if fmt.find("%m") != -1 or fmt_trans.find("%m") != -1:
+            raise ValueError("Event format '%m' is forbidden, pass the error "
+                             "as an explicit trace argument")
+
         if len(fmt_trans) > 0:
             fmt = [fmt_trans, fmt]
         args = Arguments.build(groups["args"])
Signature · 1 line
2.20.1
#20 lore raw

PATCH v2 4/4 trace: add ability to do simple printf logging via systemtap

In reply to Daniel P. Berrangé

The dtrace systemtap trace backend for QEMU is very powerful but it is also somewhat unfriendly to users who aren't familiar with systemtap, or who don't need its power right now.

stap -e "....some strange script...."

The 'log' backend for QEMU by comparison is very crude but incredibly easy to use:

 $ qemu -d trace:qio* ...some args...
 [email protected]:qio_channel_socket_new Socket new ioc=0x563a8a39d400
 [email protected]:qio_task_new Task new task=0x563a891d0570 source=0x563a8a39d400 func=0x563a86f1e6c0 opaque=0x563a89078000
 [email protected]:qio_task_thread_start Task thread start task=0x563a891d0570 worker=0x563a86f1ce50 opaque=0x563a891d9d90
 [email protected]:qio_task_thread_run Task thread run task=0x563a891d0570
 [email protected]:qio_channel_socket_connect_sync Socket connect sync ioc=0x563a8a39d400 addr=0x563a891d9d90
 [email protected]:qio_channel_socket_connect_fail Socket connect fail ioc=0x563a8a39d400

This commit introduces a way to do simple printf style logging of probe points using systemtap. In particular it creates another set of tapsets, one per emulator:

/usr/share/systemtap/tapset/qemu-*-log.stp

These pre-define probe functions which simply call printf() on their arguments. The printf() format string is taken from the normal trace-events files, with a little munging to the format specifiers to cope with systemtap's more restrictive syntax.

With this you can now do

 $ stap -e 'probe qemu.system.x86_64.log.qio*{}'
 22806@1547735341399856820 qio_channel_socket_new Socket new ioc=0x56135d1d7c00
 22806@1547735341399862570 qio_task_new Task new task=0x56135cd66eb0 source=0x56135d1d7c00 func=0x56135af746c0 opaque=0x56135bf06400
 22806@1547735341399865943 qio_task_thread_start Task thread start task=0x56135cd66eb0 worker=0x56135af72e50 opaque=0x56135c071d70
 22806@1547735341399976816 qio_task_thread_run Task thread run task=0x56135cd66eb0

We go one step further though and introduce a 'qemu-trace-stap' tool to make this even easier

 $ qemu-trace-stap run qemu-system-x86_64 qio*
 22806@1547735341399856820 qio_channel_socket_new Socket new ioc=0x56135d1d7c00
 22806@1547735341399862570 qio_task_new Task new task=0x56135cd66eb0 source=0x56135d1d7c00 func=0x56135af746c0 opaque=0x56135bf06400
 22806@1547735341399865943 qio_task_thread_start Task thread start task=0x56135cd66eb0 worker=0x56135af72e50 opaque=0x56135c071d70
 22806@1547735341399976816 qio_task_thread_run Task thread run task=0x56135cd66eb0

This tool is clever in that it will automatically change the SYSTEMTAP_TAPSET env variable to point to the directory containing the right set of probes for the QEMU binary path you give it. This is useful if you have QEMU installed in /usr but are trying to test and trace a binary in /home/berrange/usr/qemu-git. In that case you'd do

 $ qemu-trace-stap run /home/berrange/usr/qemu-git/bin/qemu-system-x86_64 qio*

And it'll make sure /home/berrange/usr/qemu-git/share/systemtap/tapset is used for the trace session

The 'qemu-trace-stap' script takes a verbose arg so you can understand what it is running

 $ qemu-trace-stap run /home/berrange/usr/qemu-git/bin/qemu-system-x86_64 qio*
 Using tapset dir '/home/berrange/usr/qemu-git/share/systemtap/tapset' for binary '/home/berrange/usr/qemu-git/bin/qemu-system-x86_64'
 Compiling script 'probe qemu.system.x86_64.log.qio* {}'
 Running script, <Ctrl>-c to quit
 ...trace output...

It can enable multiple probes at once

 $ qemu-trace-stap run qemu-system-x86_64 qio* qcrypto* buffer*

Finally if you can't remember what probes are valid it can tell you

 $ qemu-trace-stap list qemu-system-x86_64
 ahci_check_irq
 ahci_cmd_done
 ahci_dma_prepare_buf
 ahci_dma_prepare_buf_fail
 ahci_dma_rw_buf
 ahci_irq_lower
 ...snip...

Or list just those matching a prefix pattern

 $ qemu-trace-stap list -v qemu-system-x86_64 qio*
 Using tapset dir '/home/berrange/usr/qemu-git/share/systemtap/tapset' for binary '/home/berrange/usr/qemu-git/bin/qemu-system-x86_64'
 Listing probes with name 'qemu.system.x86_64.log.qio*'
 qio_channel_command_abort
 qio_channel_command_new_pid
 qio_channel_command_new_spawn
 qio_channel_command_wait
 qio_channel_file_new_fd
 ...snip...
---
 Makefile                             |   3 +
 Makefile.target                      |  11 +-
 docs/devel/tracing.txt               |  32 +++++
 scripts/qemu-trace-stap              | 167 +++++++++++++++++++++++++++
 scripts/tracetool/format/log_stap.py | 127 ++++++++++++++++++++
 5 files changed, 339 insertions(+), 1 deletion(-)
 create mode 100755 scripts/qemu-trace-stap
 create mode 100644 scripts/tracetool/format/log_stap.py
Makefile +3 -0
index dccba1dca2..22e481e7ba 100644
@@ -736,6 +736,9 @@ endif
 ifneq ($(HELPERS-y),)
 	$(call install-prog,$(HELPERS-y),$(DESTDIR)$(libexecdir))
 endif
+ifdef CONFIG_TRACE_SYSTEMTAP
+	$(INSTALL_PROG) "scripts/qemu-trace-stap" $(DESTDIR)$(bindir)
+endif
 ifneq ($(BLOBS),)
 	set -e; for x in $(BLOBS); do \
 		$(INSTALL_DATA) $(SRC_PATH)/pc-bios/$$x "$(DESTDIR)$(qemu_datadir)"; \
Makefile.target +10 -1
index 39f72e81be..401dc1ea6f 100644
@@ -45,7 +45,7 @@ config-target.h: config-target.h-timestamp
 config-target.h-timestamp: config-target.mak
 
 ifdef CONFIG_TRACE_SYSTEMTAP
-stap: $(QEMU_PROG).stp-installed $(QEMU_PROG).stp $(QEMU_PROG)-simpletrace.stp
+stap: $(QEMU_PROG).stp-installed $(QEMU_PROG).stp $(QEMU_PROG)-simpletrace.stp $(QEMU_PROG)-log.stp
 
 ifdef CONFIG_USER_ONLY
 TARGET_TYPE=user
@@ -84,6 +84,14 @@ $(QEMU_PROG)-simpletrace.stp: $(BUILD_DIR)/trace-events-all $(tracetool-y)
 		--probe-prefix=qemu.$(TARGET_TYPE).$(TARGET_NAME) \
 		$< > $@,"GEN","$(TARGET_DIR)$(QEMU_PROG)-simpletrace.stp")
 
+$(QEMU_PROG)-log.stp: $(BUILD_DIR)/trace-events-all $(tracetool-y)
+	$(call quiet-command,$(TRACETOOL) \
+		--group=all \
+		--format=log-stap \
+		--backends=$(TRACE_BACKENDS) \
+		--probe-prefix=qemu.$(TARGET_TYPE).$(TARGET_NAME) \
+		$< > $@,"GEN","$(TARGET_DIR)$(QEMU_PROG)-log.stp")
+
 else
 stap:
 endif
@@ -227,6 +235,7 @@ ifdef CONFIG_TRACE_SYSTEMTAP
 	$(INSTALL_DIR) "$(DESTDIR)$(qemu_datadir)/../systemtap/tapset"
 	$(INSTALL_DATA) $(QEMU_PROG).stp-installed "$(DESTDIR)$(qemu_datadir)/../systemtap/tapset/$(QEMU_PROG).stp"
 	$(INSTALL_DATA) $(QEMU_PROG)-simpletrace.stp "$(DESTDIR)$(qemu_datadir)/../systemtap/tapset/$(QEMU_PROG)-simpletrace.stp"
+	$(INSTALL_DATA) $(QEMU_PROG)-log.stp "$(DESTDIR)$(qemu_datadir)/../systemtap/tapset/$(QEMU_PROG)-log.stp"
 endif
 
 GENERATED_FILES += config-target.h
docs/devel/tracing.txt +32 -0
index bc52f12485..53327cd35b 100644
@@ -317,6 +317,38 @@ probes:
                          --target-name x86_64 \
                          <trace-events-all >qemu.stp
 
+To facilitate simple usage of systemtap where there merely needs to be printf
+logging of certain probes, a helper script "qemu-trace-stap" is provided
+
+This can list all probes available for a given QEMU emulator binary:
+
+     $ qemu-trace-stap list qemu-system-x86_64
+     ahci_check_irq
+     ahci_cmd_done
+     ahci_dma_prepare_buf
+     ahci_dma_prepare_buf_fail
+     ahci_dma_rw_buf
+     ahci_irq_lower
+     ...snip...
+
+Optionally restricted to a pattern prefix:
+
+     $ qemu-trace-stap list qemu-system-x86_64 qio*
+     qio_channel_command_abort
+     qio_channel_command_new_pid
+     qio_channel_command_new_spawn
+     qio_channel_command_wait
+     qio_channel_file_new_fd
+     ...snip...
+
+It can then run a simple systemtap script which prints each requested probe
+
+     $ qemu-trace-stap run qemu-system-x86_64 qio*
+     22806@1547735341399856820 qio_channel_socket_new Socket new ioc=0x56135d1d7c00
+     22806@1547735341399862570 qio_task_new Task new task=0x56135cd66eb0 source=0x56135d1d7c00 func=0x56135af746c0 opaque=0x56135bf06400
+     22806@1547735341399865943 qio_task_thread_start Task thread start task=0x56135cd66eb0 worker=0x56135af72e50 opaque=0x56135c071d70
+     22806@1547735341399976816 qio_task_thread_run Task thread run task=0x56135cd66eb0
+
 == Trace event properties ==
 
 Each event in the "trace-events-all" file can be prefixed with a space-separated
scripts/qemu-trace-stap new file +167 -0

167 lines not on this page · raw message

scripts/tracetool/format/log_stap.py new file +127 -0

127 lines not on this page · raw message

Signature · 1 line
2.20.1
Reviewed-by on v2 1/4
#2 lore raw

Re: [Qemu-devel] [PATCH v2 1/4] display: ensure qxl log_buf is a nul terminated string

In reply to Daniel P. Berrangé
Quoted 21 lines · Daniel P. Berrangé

The QXL_IO_LOG command allows the guest to send log messages to the host via a buffer in the QXLRam struct. QEMU prints these to the console if the qxl 'guestdebug' option is set to non-zero. It will also feed them to the trace subsystem if any backends are built-in.

In both cases the log_buf data will get treated as being as a nul terminated string, by the printf '%s' format specifier and / or other code reading the buffer.

QEMU does nothing to guarantee that the log_buf really is nul terminated, so there is potential for out of bounds array access.

This would affect any QEMU which has the log, syslog or ftrace trace backends built into QEMU. It can only be triggered if the 'qxl_io_log' trace event is enabled, however, so they are not vulnerable without specific administrative action to enable this.

It would also affect QEMU if the 'guestdebug' parameter is set to a non-zero value, which again is not the default and requires explicit admin opt-in.

Quoted 5 lines
---
 hw/display/qxl.c        | 3 ++-
 hw/display/trace-events | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)

hw/display/qxl.c async_common: · line 1766

Quoted 7 lines

+++ b/hw/display/qxl.c

hw/display/qxl.c +1 -1
@@ -1763,7 +1763,8 @@ async_common:
         qxl_set_mode(d, val, 0);
         break;
     case QXL_IO_LOG:
-        trace_qxl_io_log(d->id, d->ram->log_buf);
+        d->ram->log_buf[sizeof(d->ram->log_buf) - 1] = '\0';

May lose a character from the log, but the improved safety is worth it.

Quoted 18 lines
hw/display/qxl.c +1 -0
+        trace_qxl_io_log(d->id, (const char *)d->ram->log_buf);
         if (d->guestdebug) {
             fprintf(stderr, "qxl/guest-%d: %" PRId64 ": %s", d->id,
                     qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL), d->ram->log_buf);
hw/display/trace-events +1 -1
index 5a48c6cb6a..387c6b8931 100644
@@ -72,7 +72,7 @@ qxl_interface_update_area_complete_rest(int qid, uint32_t num_updated_rects) "%d
 qxl_interface_update_area_complete_overflow(int qid, int max) "%d max=%d"
 qxl_interface_update_area_complete_schedule_bh(int qid, uint32_t num_dirty) "%d #dirty=%d"
 qxl_io_destroy_primary_ignored(int qid, const char *mode) "%d %s"
-qxl_io_log(int qid, const uint8_t *log_buf) "%d %s"
+qxl_io_log(int qid, const char *log_buf) "%d %s"
 qxl_io_read_unexpected(int qid) "%d"
 qxl_io_unexpected_vga_mode(int qid, uint64_t addr, uint64_t val, const char *desc) "%d 0x%"PRIx64"=%"PRIu64" (%s)"
 qxl_io_write(int qid, const char *mode, uint64_t addr, const char *aname, uint64_t val, unsigned size, int async) "%d %s addr=%"PRIu64 " (%s) val=%"PRIu64" size=%u async=%d"
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
Reviewed-by on v2 2/4
#8 lore raw

Re: [Qemu-devel] [PATCH v2 2/4] trace: enforce that every trace-events file has a final newline

In reply to Daniel P. Berrangé
Quoted 6 lines · Daniel P. Berrangé

When generating the trace-events-all file, the build system simply concatenates all the individual trace-events files. If any one of those files does not have a final newline, the printf format string will have the contents of the first line of the next file appended to it, which is usually a '#' comment.

Quoted 6 lines
---
 hw/gpio/trace-events          | 2 +-
 scripts/tracetool/__init__.py | 2 ++
 2 files changed, 3 insertions(+), 1 deletion(-)

Does checkpatch and/or patchew flag what are intended to be text files but which lack a trailing newline? If not, how hard would it be to get them to do so?

scripts/tracetool/__init__.py

Quoted 12 lines
hw/gpio/trace-events +1 -1
index cb41a89756..5d4dd200c2 100644
@@ -4,4 +4,4 @@
 nrf51_gpio_read(uint64_t offset, uint64_t r) "offset 0x%" PRIx64 " value 0x%" PRIx64
 nrf51_gpio_write(uint64_t offset, uint64_t value) "offset 0x%" PRIx64 " value 0x%" PRIx64
 nrf51_gpio_set(int64_t line, int64_t value) "line %" PRIi64 " value %" PRIi64
-nrf51_gpio_update_output_irq(int64_t line, int64_t value) "line %" PRIi64 " value %" PRIi64
\ No newline at end of file
+nrf51_gpio_update_output_irq(int64_t line, int64_t value) "line %" PRIi64 " value %" PRIi64
scripts/tracetool/__init__.py +0 -0

Fixes the problem file...

scripts/tracetool/__init__.py def read_events(fobj, fname): · line 354

Quoted 9 lines

index 0e3c9e146c..3478ac93ab 100644

scripts/tracetool/__init__.py +2 -0
@@ -350,6 +350,8 @@ def read_events(fobj, fname):
 
     events = []
     for lineno, line in enumerate(fobj, 1):
+        if line[-1] != '\n':
+            raise ValueError("%s does not end with a new line" % fname)

and ensures that no future trace-events files will have the problem, whether or not checkpatch flags other such files.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#3 lore raw

Re: [Qemu-devel] [PATCH v2 1/4] display: ensure qxl log_buf is a nul terminated string

In reply to Eric Blake
Quoted 30 lines · Eric Blake
Quoted 21 lines · Daniel P. Berrangé

The QXL_IO_LOG command allows the guest to send log messages to the host via a buffer in the QXLRam struct. QEMU prints these to the console if the qxl 'guestdebug' option is set to non-zero. It will also feed them to the trace subsystem if any backends are built-in.

In both cases the log_buf data will get treated as being as a nul terminated string, by the printf '%s' format specifier and / or other code reading the buffer.

QEMU does nothing to guarantee that the log_buf really is nul terminated, so there is potential for out of bounds array access.

This would affect any QEMU which has the log, syslog or ftrace trace backends built into QEMU. It can only be triggered if the 'qxl_io_log' trace event is enabled, however, so they are not vulnerable without specific administrative action to enable this.

It would also affect QEMU if the 'guestdebug' parameter is set to a non-zero value, which again is not the default and requires explicit admin opt-in.

Quoted 5 lines
---
 hw/display/qxl.c        | 3 ++-
 hw/display/trace-events | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)
Quoted 9 lines

hw/display/qxl.c async_common: · line 1766

Quoted 7 lines

+++ b/hw/display/qxl.c

hw/display/qxl.c +1 -1
@@ -1763,7 +1763,8 @@ async_common:
         qxl_set_mode(d, val, 0);
         break;
     case QXL_IO_LOG:
-        trace_qxl_io_log(d->id, d->ram->log_buf);
+        d->ram->log_buf[sizeof(d->ram->log_buf) - 1] = '\0';

May lose a character from the log, but the improved safety is worth it.

In practice it won't. The Xorg QXL driver and Windows XDDM DOD driver don't use this feature at all. The Windows QXL miniport driver uses snprintf when populating the buffer so will already have truncated if it was going to overflow.

Quoted 11 lines
Quoted 11 lines
hw/display/qxl.c +1 -0
+        trace_qxl_io_log(d->id, (const char *)d->ram->log_buf);
         if (d->guestdebug) {
             fprintf(stderr, "qxl/guest-%d: %" PRId64 ": %s", d->id,
                     qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL), d->ram->log_buf);
hw/display/trace-events +0 -0
index 5a48c6cb6a..387c6b8931 100644
@@ -72,7 +72,7 @@ qxl_interface_update_area_complete_rest(int qid, uint32_t num_updated_rects) "%d
 qxl_interface_update_area_complete_overflow(int qid, int max) "%d max=%d"
Quoted 14 lines
Quoted 8 lines
hw/display/trace-events +1 -1
 qxl_interface_update_area_complete_schedule_bh(int qid, uint32_t num_dirty) "%d #dirty=%d"
 qxl_io_destroy_primary_ignored(int qid, const char *mode) "%d %s"
-qxl_io_log(int qid, const uint8_t *log_buf) "%d %s"
+qxl_io_log(int qid, const char *log_buf) "%d %s"
 qxl_io_read_unexpected(int qid) "%d"
 qxl_io_unexpected_vga_mode(int qid, uint64_t addr, uint64_t val, const char *desc) "%d 0x%"PRIx64"=%"PRIu64" (%s)"
 qxl_io_write(int qid, const char *mode, uint64_t addr, const char *aname, uint64_t val, unsigned size, int async) "%d %s addr=%"PRIu64 " (%s) val=%"PRIu64" size=%u async=%d"
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#12 lore raw

Re: [Qemu-devel] [PATCH v2 3/4] trace: forbid use of %m in trace event format strings

In reply to Daniel P. Berrangé
Quoted 2 lines · Daniel P. Berrangé

The '%m' format specifier instructs glibc's printf() implementation to insert the contents of strerror(errno).

That is a glibc-only extension in printf(), but mandatory (and supported in ALL platforms) in syslog(). However, you are correct that:

Quoted 6 lines

This is not something that
should ever be used in trace-events files because several of the backends do not use the format string and so this error information is invisible to them. The errno value should be given as an explicit trace argument instead. Use of '%m' should also be avoided as it is not portable to all QEMU build targets.

If all of our traces are compiled to syslog(), it would be portable, but that's not the case.

What's more, using %m is subject to race conditions - the more code you have between when the format string containing %m is given, and the point where the actual error message is emitted, the higher the probability that some of the intermediate code could have stomped on errno in the meantime, rendering the logged message incorrect. And forcing logging wrappers to save/restore the current state of errno, just in case they might encounter %m, can get wasteful.

Should checkpatch be taught to flag %m as suspicious?

However, tracing the errno value is NOT what %m did; I'd rather see...

Quoted 13 lines
---
 hw/vfio/pci.c                 | 2 +-
 hw/vfio/trace-events          | 2 +-
 scripts/tracetool/__init__.py | 4 ++++
 3 files changed, 6 insertions(+), 2 deletions(-)
hw/vfio/pci.c +0 -0
index c0cb1ec289..85f1908cfe 100644

hw/vfio/pci.c static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) · line 2584

Quoted 5 lines
hw/vfio/pci.c +1 -1
@@ -2581,7 +2581,7 @@ static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp)
     ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info);
     if (ret) {
         /* This can fail for an old kernel or legacy PCI dev */
-        trace_vfio_populate_device_get_irq_info_failure();
+        trace_vfio_populate_device_get_irq_info_failure(errno);

trace_vfio_populate_device_get_irq_info_failure(strerror(errno))

hw/vfio/trace-events vfio_pci_hot_reset_has_dep_devices(const char *name) "%s: hot reset dependent de · line 40

Quoted 13 lines
hw/vfio/pci.c +0 -0
     } else if (irq_info.count == 1) {
         vdev->pci_aer = true;
     } else {
hw/vfio/trace-events +1 -1
index a85e8662ea..6d412afc83 100644
@@ -37,7 +37,7 @@ vfio_pci_hot_reset_has_dep_devices(const char *name) "%s: hot reset dependent de
 vfio_pci_hot_reset_dep_devices(int domain, int bus, int slot, int function, int group_id) "\t%04x:%02x:%02x.%x group %d"
 vfio_pci_hot_reset_result(const char *name, const char *result) "%s hot reset: %s"
 vfio_populate_device_config(const char *name, unsigned long size, unsigned long offset, unsigned long flags) "Device %s config:\n  size: 0x%lx, offset: 0x%lx, flags: 0x%lx"
-vfio_populate_device_get_irq_info_failure(void) "VFIO_DEVICE_GET_IRQ_INFO failure: %m"
+vfio_populate_device_get_irq_info_failure(int err) "VFIO_DEVICE_GET_IRQ_INFO failure: %d"

vfio_populate_device_get_irq_info_failur(const char *err) "VFIO_DEVICE_GET_IRQ_INFO failure: %s"

...if we are trying to match the original intent.

Quoted 8 lines
hw/vfio/trace-events +0 -0
 vfio_realize(const char *name, int group_id) " (%s) group %d"
 vfio_mdev(const char *name, bool is_mdev) " (%s) is_mdev %d"
 vfio_add_ext_cap_dropped(const char *name, uint16_t cap, uint16_t offset) "%s 0x%x@0x%x"
scripts/tracetool/__init__.py +0 -0
index 3478ac93ab..6fca674936 100644

scripts/tracetool/__init__.py class Event(object): · line 279

Quoted 6 lines
scripts/tracetool/__init__.py +3 -0
@@ -274,6 +274,10 @@ class Event(object):
         props = groups["props"].split()
         fmt = groups["fmt"]
         fmt_trans = groups["fmt_trans"]
+        if fmt.find("%m") != -1 or fmt_trans.find("%m") != -1:
+            raise ValueError("Event format '%m' is forbidden, pass the error "
+                             "as an explicit trace argument")

Whether or not checkpatch decides to flag %m as suspicious in the overall code base, I like that you are forbidding it in trace files.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#21 lore raw

Re: [Qemu-devel] [PATCH v2 4/4] trace: add ability to do simple printf logging via systemtap

In reply to Daniel P. Berrangé
Quoted 4 lines · Daniel P. Berrangé

The dtrace systemtap trace backend for QEMU is very powerful but it is also somewhat unfriendly to users who aren't familiar with systemtap, or who don't need its power right now.

Quoted 8 lines

stap -e "....some strange script...."

We go one step further though and introduce a 'qemu-trace-stap' tool to make this even easier

 $ qemu-trace-stap run qemu-system-x86_64 qio*
 22806@1547735341399856820 qio_channel_socket_new Socket new ioc=0x56135d1d7c00

One thing I did not immediately get on the first read, and therefore which may be worth mentioning in the docs portion of the patch:

Am I correct that stap is always run as a separate process from the process(es) being traced, and that you can start the two in either order (qemu first, then stap; or stap first, then qemu), where the tracing is effectively output as long as both processes are running and trace points being hit in the qemu process?

Quoted 7 lines
---
 Makefile                             |   3 +
 Makefile.target                      |  11 +-
 docs/devel/tracing.txt               |  32 +++++
 scripts/qemu-trace-stap              | 167 +++++++++++++++++++++++++++
 scripts/tracetool/format/log_stap.py | 127 ++++++++++++++++++++

Overall, I'm impressed with the concept, but my familiarity with stap and python is weak enough that I'm not leaving a formal R-b. Still, I'll give it a shot at looking for anything that stands out as I read through.

Quoted 9 lines
 5 files changed, 339 insertions(+), 1 deletion(-)
 create mode 100755 scripts/qemu-trace-stap
 create mode 100644 scripts/tracetool/format/log_stap.py
Makefile +0 -0
index dccba1dca2..22e481e7ba 100644

Makefile endif · line 741

Quoted 6 lines
Makefile +3 -0
@@ -736,6 +736,9 @@ endif
 ifneq ($(HELPERS-y),)
 	$(call install-prog,$(HELPERS-y),$(DESTDIR)$(libexecdir))
 endif
+ifdef CONFIG_TRACE_SYSTEMTAP
+	$(INSTALL_PROG) "scripts/qemu-trace-stap" $(DESTDIR)$(bindir)
+endif

No man page being installed? If we're planning on this binary being available on a similar par with qemu-io, it deserves some installed documentation.

Makefile probes: · line 322

Quoted 8 lines
Makefile +4 -0
+++ b/docs/devel/tracing.txt
@@ -317,6 +317,38 @@ probes:
                          --target-name x86_64 \
                          <trace-events-all >qemu.stp
 
+To facilitate simple usage of systemtap where there merely needs to be printf
+logging of certain probes, a helper script "qemu-trace-stap" is provided
+

Trailing '.' would look better

Quoted 12 lines
Makefile +12 -0
+This can list all probes available for a given QEMU emulator binary:
+
+     $ qemu-trace-stap list qemu-system-x86_64
+     ahci_check_irq
+     ahci_cmd_done
+     ahci_dma_prepare_buf
+     ahci_dma_prepare_buf_fail
+     ahci_dma_rw_buf
+     ahci_irq_lower
+     ...snip...
+
+Optionally restricted to a pattern prefix:

Makefile probes: · line 344

Quoted 10 lines
Makefile +10 -0
+
+     $ qemu-trace-stap list qemu-system-x86_64 qio*
+     qio_channel_command_abort
+     qio_channel_command_new_pid
+     qio_channel_command_new_spawn
+     qio_channel_command_wait
+     qio_channel_file_new_fd
+     ...snip...
+
+It can then run a simple systemtap script which prints each requested probe

Trailing ':' would look better

Quoted 19 lines
Makefile +17 -0
+
+     $ qemu-trace-stap run qemu-system-x86_64 qio*
+     22806@1547735341399856820 qio_channel_socket_new Socket new ioc=0x56135d1d7c00
+     22806@1547735341399862570 qio_task_new Task new task=0x56135cd66eb0 source=0x56135d1d7c00 func=0x56135af746c0 opaque=0x56135bf06400
+     22806@1547735341399865943 qio_task_thread_start Task thread start task=0x56135cd66eb0 worker=0x56135af72e50 opaque=0x56135c071d70
+     22806@1547735341399976816 qio_task_thread_run Task thread run task=0x56135cd66eb0
+
 
+++ b/scripts/qemu-trace-stap
 
+
+def main():
+    parser = argparse.ArgumentParser(description="QEMU SystemTap trace tool")
+
+    subparser = parser.add_subparsers(help="commands")
+    subparser.required = True
+    subparser.dest = "command"
+
+    runparser = subparser.add_parser("run", help="Run a trace session",

Makefile probes:

Quoted 10 lines
Makefile +10 -0
+                                     formatter_class=argparse.RawDescriptionHelpFormatter,
+                                     epilog="""
+
+To watch all trace points on the qemu-system-x86_64 binary:
+
+   %(argv0)s run qemu-system-x86_64
+
+To only watch the trace points matching the qio* and qcrypto* patterns
+
+   %(argv0)s run qemu-system-x86_64 qio* qcrypto*

This output is not copy-pastable - it includes shell globs which might inadvertantly expand based on the contents of $PWD. Should you amend the output to be properly quoted for reuse in shell commands?

Quoted 9 lines

+""" % {"argv0": sys.argv[0]})

Makefile +8 -0
+    runparser.set_defaults(func=cmd_run)
+    runparser.add_argument("-v", "--verbose", help="Print verbose progress info",
+                           action='store_true')
+    runparser.add_argument("binary", help="QEMU system or user emulator binary")
+    runparser.add_argument("probes", help="Probe names or wildcards",
+                           nargs=argparse.REMAINDER)
+
+    listparser = subparser.add_parser("list", help="List probe points",

Makefile

Quoted 10 lines
Makefile +10 -0
+                                      formatter_class=argparse.RawDescriptionHelpFormatter,
+                                      epilog="""
+
+To list all trace points on the qemu-system-x86_64 binary:
+
+   %(argv0)s list qemu-system-x86_64
+
+To only list the trace points matching the qio* and qcrypto* patterns
+
+   %(argv0)s list qemu-system-x86_64 qio* qcrypto*

and again

Quoted 45 lines

+""" % {"argv0": sys.argv[0]})

Makefile +14 -0
+    listparser.set_defaults(func=cmd_list)
+    listparser.add_argument("-v", "--verbose", help="Print verbose progress info",
+                            action='store_true')
+    listparser.add_argument("binary", help="QEMU system or user emulator binary")
+    listparser.add_argument("probes", help="Probe names or wildcards",
+                            nargs=argparse.REMAINDER)
+
+    args = parser.parse_args()
+
+    args.func(args)
+    sys.exit(0)
+
+if __name__ == '__main__':
+    main()
scripts/tracetool/format/log_stap.py new file +24 -0
new file mode 100644
index 0000000000..3ccbc09d61
@@ -0,0 +1,127 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+"""
+Generate .stp file that printfs log messages (DTrace with SystemTAP only).
+"""
+
+__author__     = "Daniel P. Berrange <[email protected]>"
+__copyright__  = "Copyright (C) 2014-2019, Red Hat, Inc."
+__license__    = "GPL version 2 or (at your option) any later version"
+
+__maintainer__ = "Daniel Berrange"
+__email__      = "[email protected]"
+
+import re
+
+from tracetool import out
+from tracetool.backend.dtrace import binary, probeprefix
+from tracetool.backend.simple import is_string
+from tracetool.format.stap import stap_escape
+
+def global_var_name(name):
+    return probeprefix().replace(".", "_") + "_" + name
+

scripts/tracetool/format/log_stap.py line 34

Quoted 10 lines
scripts/tracetool/format/log_stap.py new file +10 -0
+STATE_SKIP = 0
+STATE_LITERAL = 1
+STATE_MACRO = 2
+
+def c_macro_to_format(macro):
+    if macro.startswith("PRI"):
+        return macro[3]
+
+    if macro == "TARGET_FMT_plx":
+        return "%016x"

Do we really want to permit TARGET_FMT in our trace-events files, or should that be fixed in hw/tpm/trace-events? See commit 73ff0610.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#13 lore raw

Re: [Qemu-devel] [PATCH v2 3/4] trace: forbid use of %m in trace event format strings

In reply to Eric Blake
Quoted 13 lines · On Fri, 18 Jan 2019 11:50:39 -0600 Eric Blake <[email protected]> wrote:
Quoted 2 lines · Daniel P. Berrangé

The '%m' format specifier instructs glibc's printf() implementation to insert the contents of strerror(errno).

That is a glibc-only extension in printf(), but mandatory (and supported in ALL platforms) in syslog(). However, you are correct that:

Quoted 6 lines

This is not something that
should ever be used in trace-events files because several of the backends do not use the format string and so this error information is invisible to them. The errno value should be given as an explicit trace argument instead. Use of '%m' should also be avoided as it is not portable to all QEMU build targets.

It's portable to all QEMU build targets that support vfio, but we can remove it if we want it out of the trace system, which of course does support other targets.

Quoted 23 lines

If all of our traces are compiled to syslog(), it would be portable, but that's not the case.

What's more, using %m is subject to race conditions - the more code you have between when the format string containing %m is given, and the point where the actual error message is emitted, the higher the probability that some of the intermediate code could have stomped on errno in the meantime, rendering the logged message incorrect. And forcing logging wrappers to save/restore the current state of errno, just in case they might encounter %m, can get wasteful.

Should checkpatch be taught to flag %m as suspicious?

However, tracing the errno value is NOT what %m did; I'd rather see...

Quoted 8 lines
---
 hw/vfio/pci.c                 | 2 +-
 hw/vfio/trace-events          | 2 +-
 scripts/tracetool/__init__.py | 4 ++++
 3 files changed, 6 insertions(+), 2 deletions(-)
Quoted 12 lines

hw/vfio/pci.c static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) · line 2584

Quoted 10 lines
hw/vfio/pci.c +1 -1
index c0cb1ec289..85f1908cfe 100644
@@ -2581,7 +2581,7 @@ static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp)
     ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info);
     if (ret) {
         /* This can fail for an old kernel or legacy PCI dev */
-        trace_vfio_populate_device_get_irq_info_failure();
+        trace_vfio_populate_device_get_irq_info_failure(errno);  

trace_vfio_populate_device_get_irq_info_failure(strerror(errno))

Yep, I agree, the intent of the original code is not maintained. Thanks,

Alex

Quoted 37 lines

hw/vfio/trace-events vfio_pci_hot_reset_has_dep_devices(const char *name) "%s: hot reset dependent de · line 40

Quoted 13 lines
hw/vfio/pci.c +0 -0
     } else if (irq_info.count == 1) {
         vdev->pci_aer = true;
     } else {
hw/vfio/trace-events +1 -1
index a85e8662ea..6d412afc83 100644
@@ -37,7 +37,7 @@ vfio_pci_hot_reset_has_dep_devices(const char *name) "%s: hot reset dependent de
 vfio_pci_hot_reset_dep_devices(int domain, int bus, int slot, int function, int group_id) "\t%04x:%02x:%02x.%x group %d"
 vfio_pci_hot_reset_result(const char *name, const char *result) "%s hot reset: %s"
 vfio_populate_device_config(const char *name, unsigned long size, unsigned long offset, unsigned long flags) "Device %s config:\n  size: 0x%lx, offset: 0x%lx, flags: 0x%lx"
-vfio_populate_device_get_irq_info_failure(void) "VFIO_DEVICE_GET_IRQ_INFO failure: %m"
+vfio_populate_device_get_irq_info_failure(int err) "VFIO_DEVICE_GET_IRQ_INFO failure: %d"  

vfio_populate_device_get_irq_info_failur(const char *err) "VFIO_DEVICE_GET_IRQ_INFO failure: %s"

...if we are trying to match the original intent.

Quoted 8 lines
hw/vfio/trace-events +0 -0
 vfio_realize(const char *name, int group_id) " (%s) group %d"
 vfio_mdev(const char *name, bool is_mdev) " (%s) is_mdev %d"
 vfio_add_ext_cap_dropped(const char *name, uint16_t cap, uint16_t offset) "%s 0x%x@0x%x"
scripts/tracetool/__init__.py +0 -0
index 3478ac93ab..6fca674936 100644

scripts/tracetool/__init__.py class Event(object): · line 279

Quoted 6 lines
scripts/tracetool/__init__.py +3 -0
@@ -274,6 +274,10 @@ class Event(object):
         props = groups["props"].split()
         fmt = groups["fmt"]
         fmt_trans = groups["fmt_trans"]
+        if fmt.find("%m") != -1 or fmt_trans.find("%m") != -1:
+            raise ValueError("Event format '%m' is forbidden, pass the error "
+                             "as an explicit trace argument")  

Whether or not checkpatch decides to flag %m as suspicious in the overall code base, I like that you are forbidding it in trace files.

Reviewed-by on v2 1/4
#4 lore raw

Re: [Qemu-devel] [PATCH v2 1/4] display: ensure qxl log_buf is a nul terminated string

In reply to Daniel P. Berrangé

hw/display/qxl.c async_common: · line 1768

Quoted 5 lines

case QXL_IO_LOG:

hw/display/qxl.c +2 -1
-        trace_qxl_io_log(d->id, d->ram->log_buf);
+        d->ram->log_buf[sizeof(d->ram->log_buf) - 1] = '\0';
+        trace_qxl_io_log(d->id, (const char *)d->ram->log_buf);
         if (d->guestdebug) {
#5 lore raw

Re: [Qemu-devel] [PATCH v2 1/4] display: ensure qxl log_buf is a nul terminated string

In reply to Daniel P. Berrangé

hw/display/qxl.c async_common: · line 1767

Quoted 11 lines · Daniel P. Berrangé
hw/display/qxl.c +2 -1
index 8e9a65e75b..eefdf4baac 100644
@@ -1763,7 +1763,8 @@ async_common:
         qxl_set_mode(d, val, 0);
         break;
     case QXL_IO_LOG:
-        trace_qxl_io_log(d->id, d->ram->log_buf);
+        d->ram->log_buf[sizeof(d->ram->log_buf) - 1] = '\0';
+        trace_qxl_io_log(d->id, (const char *)d->ram->log_buf);

This is a PCI BAR shared with the guest? Then NUL termination is subject to races with vcpu threads that modify log_buf[] while we access it.

The safe way to do this is to copy in log_buf[] and then NUL-terminate the local copy.

Reviewed-by on v2 2/4
#10 lore raw

Re: [Qemu-devel] [PATCH v2 2/4] trace: enforce that every trace-events file has a final newline

In reply to Daniel P. Berrangé
Quoted 11 lines · Daniel P. Berrangé

When generating the trace-events-all file, the build system simply concatenates all the individual trace-events files. If any one of those files does not have a final newline, the printf format string will have the contents of the first line of the next file appended to it, which is usually a '#' comment.

---
 hw/gpio/trace-events          | 2 +-
 scripts/tracetool/__init__.py | 2 ++
 2 files changed, 3 insertions(+), 1 deletion(-)
#22 lore raw

Re: [Qemu-devel] [PATCH v2 4/4] trace: add ability to do simple printf logging via systemtap

In reply to Eric Blake
Quoted 10 lines · Eric Blake

scripts/tracetool/format/log_stap.py line 34

Quoted 6 lines · Daniel P. Berrangé
scripts/tracetool/format/log_stap.py new file +6 -0
+def c_macro_to_format(macro):
+    if macro.startswith("PRI"):
+        return macro[3]
+
+    if macro == "TARGET_FMT_plx":
+        return "%016x"

Do we really want to permit TARGET_FMT in our trace-events files, or should that be fixed in hw/tpm/trace-events? See commit 73ff0610.

TARGET_FMT_plx is constant across all QEMU targets. I think supporting it is nice since it makes writing trace events more natural/convenient.

Stefan

#15 lore raw

Re: [Qemu-devel] [PATCH v2 3/4] trace: forbid use of %m in trace event format strings

In reply to Daniel P. Berrangé
Quoted 10 lines · Daniel P. Berrangé
Quoted 9 lines

hw/vfio/pci.c static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) · line 2584

Quoted 7 lines

+++ b/hw/vfio/pci.c

hw/vfio/pci.c +1 -1
@@ -2581,7 +2581,7 @@ static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp)
     ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info);
     if (ret) {
         /* This can fail for an old kernel or legacy PCI dev */
-        trace_vfio_populate_device_get_irq_info_failure();
+        trace_vfio_populate_device_get_irq_info_failure(errno);

trace_vfio_populate_device_get_irq_info_failure(strerror(errno))

Quoted 3 lines

The caveat is that 'strerror' is not required to be thread safe, however, given that this is Linux only code I guess we can assume the glibc impl which fortunately is thread safe.

If we are going to worry about thread-safety of strerror(), we have a LOT of code to scrub (we are using it rather liberally throughout the code base).

Quoted 4 lines

On this point though, does anyone know of any platforms we support[1], or are likely to support in future, where 'strerror' is *not* thread safe ?

I'm not coming up with one, and I think the problem is independent of this series (if we DO have a problem, it's a series all its own to eradicate the use of strerror() in favor of something safer, either picking strerror_l() or dealing with the glibc vs. BSD differences in strerror_r()).

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#16 lore raw

Re: [Qemu-devel] [PATCH v2 3/4] trace: forbid use of %m in trace event format strings

Quoted 12 lines · Eric Blake
Quoted 10 lines · Daniel P. Berrangé
Quoted 9 lines

hw/vfio/pci.c static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) · line 2584

Quoted 7 lines

+++ b/hw/vfio/pci.c

hw/vfio/pci.c +1 -1
@@ -2581,7 +2581,7 @@ static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp)
     ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info);
     if (ret) {
         /* This can fail for an old kernel or legacy PCI dev */
-        trace_vfio_populate_device_get_irq_info_failure();
+        trace_vfio_populate_device_get_irq_info_failure(errno);

trace_vfio_populate_device_get_irq_info_failure(strerror(errno))

Quoted 7 lines
Quoted 3 lines

The caveat is that 'strerror' is not required to be thread safe, however, given that this is Linux only code I guess we can assume the glibc impl which fortunately is thread safe.

If we are going to worry about thread-safety of strerror(), we have a LOT of code to scrub (we are using it rather liberally throughout the code base).

Yes, indeed we do and it was something that always worried me a little (as well as a few other non-reentrant APIs we used previously).

Quoted 9 lines
Quoted 3 lines

On this point though, does anyone know of any platforms we support[1], or are likely to support in future, where 'strerror' is *not* thread safe ?

I'm not coming up with one, and I think the problem is independent of this series (if we DO have a problem, it's a series all its own to eradicate the use of strerror() in favor of something safer, either picking strerror_l() or dealing with the glibc vs. BSD differences in strerror_r()).

Agree that its not really something for this series - this just made me think of it again.

We went through the scrubbing in libvirt to use the sane, but still tedious to call, variant of strerror_r() many years ago. With luck though it is a worry that can be confined the dustbin of ancient UNIX history....unless someone can point to evidence to the contrary ?

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#17 lore raw

Re: [Qemu-devel] [PATCH v2 3/4] trace: forbid use of %m in trace event format strings

Quoted 5 lines · Daniel P. Berrangé
Quoted 5 lines
Quoted 4 lines

On this point though, does anyone know of any platforms we support[1], or are likely to support in future, where 'strerror' is *not* thread safe ?

Quoted 8 lines
Quoted 5 lines

I'm not coming up with one, and I think the problem is independent of this series (if we DO have a problem, it's a series all its own to eradicate the use of strerror() in favor of something safer, either picking strerror_l() or dealing with the glibc vs. BSD differences in strerror_r()).

Agree that its not really something for this series - this just made me think of it again.

Shoot - FreeBSD strerror() is not threadsafe:
https://github.com/freebsd/freebsd/blob/master/lib/libc/string/strerror.c#L119

char *
strerror(int num)

{
	static char ebuf[NL_TEXTMAX];

	if (strerror_r(num, ebuf, sizeof(ebuf)) != 0)
		errno = EINVAL;
	return (ebuf);
}
Quoted 5 lines

We went through the scrubbing in libvirt to use the sane, but still tedious to call, variant of strerror_r() many years ago. With luck though it is a worry that can be confined the dustbin of ancient UNIX history....unless someone can point to evidence to the contrary ?

libvirt has it easy - they let gnulib do all the work of futzing around with getting a working strerror() despite platform bugs and despite glibc's insistence on a non-POSIX signature if _GNU_SOURCE is defined. We'll have to do a bit more legwork.

That said, I've added it to:
https://wiki.qemu.org/Contribute/BiteSizedTasks#Error_checking

if someone wants to do the grunt work.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#6 lore raw

Re: [Qemu-devel] [PATCH v2 1/4] display: ensure qxl log_buf is a nul terminated string

In reply to Stefan Hajnoczi
Quoted 16 lines · Stefan Hajnoczi

hw/display/qxl.c async_common: · line 1767

Quoted 11 lines · Daniel P. Berrangé
hw/display/qxl.c +2 -1
index 8e9a65e75b..eefdf4baac 100644
@@ -1763,7 +1763,8 @@ async_common:
         qxl_set_mode(d, val, 0);
         break;
     case QXL_IO_LOG:
-        trace_qxl_io_log(d->id, d->ram->log_buf);
+        d->ram->log_buf[sizeof(d->ram->log_buf) - 1] = '\0';
+        trace_qxl_io_log(d->id, (const char *)d->ram->log_buf);

This is a PCI BAR shared with the guest? Then NUL termination is subject to races with vcpu threads that modify log_buf[] while we access it.

Doh, yes, it is racy.

Quoted 2 lines

The safe way to do this is to copy in log_buf[] and then NUL-terminate the local copy.

log_buf is 4k in size, which I don't really want to allocate on the stack. Using malloc would impose a perf penalty on logging but not sure if that's significant enough to worry about. Alternatively we could just drop the tracepoint, given that you can already use the 'guestdebug' option to get these fprintf'd to stderr.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#9 lore raw

Re: [Qemu-devel] [PATCH v2 2/4] trace: enforce that every trace-events file has a final newline

In reply to Eric Blake
Quoted 14 lines · Eric Blake
Quoted 6 lines · Daniel P. Berrangé

When generating the trace-events-all file, the build system simply concatenates all the individual trace-events files. If any one of those files does not have a final newline, the printf format string will have the contents of the first line of the next file appended to it, which is usually a '#' comment.

Quoted 6 lines
---
 hw/gpio/trace-events          | 2 +-
 scripts/tracetool/__init__.py | 2 ++
 2 files changed, 3 insertions(+), 1 deletion(-)
Quoted 5 lines

Does checkpatch and/or patchew flag what are intended to be text files but which lack a trailing newline? If not, how hard would it be to get them to do so?

it probably should check for missing newlines at end of file, but I'm going to let someone else tackle that due to time pressures on what was supposed to be a quick enhancement :-)

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#24 lore raw

Re: [Qemu-devel] [PATCH v2 4/4] trace: add ability to do simple printf logging via systemtap

In reply to Eric Blake
Quoted 14 lines · Eric Blake
Quoted 4 lines · Daniel P. Berrangé

The dtrace systemtap trace backend for QEMU is very powerful but it is also somewhat unfriendly to users who aren't familiar with systemtap, or who don't need its power right now.

Quoted 8 lines

stap -e "....some strange script...."

We go one step further though and introduce a 'qemu-trace-stap' tool to make this even easier

 $ qemu-trace-stap run qemu-system-x86_64 qio*
 22806@1547735341399856820 qio_channel_socket_new Socket new ioc=0x56135d1d7c00
Quoted 8 lines

One thing I did not immediately get on the first read, and therefore which may be worth mentioning in the docs portion of the patch:

Am I correct that stap is always run as a separate process from the process(es) being traced, and that you can start the two in either order (qemu first, then stap; or stap first, then qemu), where the tracing is effectively output as long as both processes are running and trace points being hit in the qemu process?

If you're using the stap binary directly you have a choice of it working against any existing process on the system, or you can have it launch the process on your behalf and target just that one process.

For our wrapper script I'm only targetting the former, since I think the common case is that someone/something else has already run the QEMU process. I could however easily add a "-p PID" option too which would allow the trace to be tailored to a specific QEMU process, instead of all running QEMU processes.

And yes, there's no ordering dependancy - unless it is critical that you see traces from QEMU's early initialization of course, in which case you'd launch the tracing first.

Quoted 4 lines
Quoted 4 lines
 5 files changed, 339 insertions(+), 1 deletion(-)
 create mode 100755 scripts/qemu-trace-stap
 create mode 100644 scripts/tracetool/format/log_stap.py
Quoted 15 lines

Makefile endif · line 741

Quoted 11 lines
Makefile +3 -0
index dccba1dca2..22e481e7ba 100644
@@ -736,6 +736,9 @@ endif
 ifneq ($(HELPERS-y),)
 	$(call install-prog,$(HELPERS-y),$(DESTDIR)$(libexecdir))
 endif
+ifdef CONFIG_TRACE_SYSTEMTAP
+	$(INSTALL_PROG) "scripts/qemu-trace-stap" $(DESTDIR)$(bindir)
+endif

No man page being installed? If we're planning on this binary being available on a similar par with qemu-io, it deserves some installed documentation.

Heh, no good deed / quick enhancement goes unpunished :-P Yes, I'll do a proper job of this and write a man page too :-)

Quoted 11 lines

Makefile endif

Quoted 7 lines
Makefile +7 -0
+To watch all trace points on the qemu-system-x86_64 binary:
+
+   %(argv0)s run qemu-system-x86_64
+
+To only watch the trace points matching the qio* and qcrypto* patterns
+
+   %(argv0)s run qemu-system-x86_64 qio* qcrypto*

This output is not copy-pastable - it includes shell globs which might inadvertantly expand based on the contents of $PWD. Should you amend the output to be properly quoted for reuse in shell commands?

Yes, that is true

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#23 lore raw

Re: [Qemu-devel] [PATCH v2 4/4] trace: add ability to do simple printf logging via systemtap

In reply to Stefan Hajnoczi
Quoted 12 lines · Stefan Hajnoczi
Quoted 10 lines · Eric Blake

scripts/tracetool/format/log_stap.py line 34

Quoted 6 lines · Daniel P. Berrangé
scripts/tracetool/format/log_stap.py new file +6 -0
+def c_macro_to_format(macro):
+    if macro.startswith("PRI"):
+        return macro[3]
+
+    if macro == "TARGET_FMT_plx":
+        return "%016x"

Do we really want to permit TARGET_FMT in our trace-events files, or should that be fixed in hw/tpm/trace-events? See commit 73ff0610.

Quoted 2 lines

TARGET_FMT_plx is constant across all QEMU targets. I think supporting it is nice since it makes writing trace events more natural/convenient.

Yep, I'm fine with supporting custom macros beyond PRI* in trace-events, provided that they can be subsituted out with a simple formats code for stap.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#14 lore raw

Re: [Qemu-devel] [PATCH v2 3/4] trace: forbid use of %m in trace event format strings

In reply to Eric Blake
Quoted 37 lines · Eric Blake
Quoted 2 lines · Daniel P. Berrangé

The '%m' format specifier instructs glibc's printf() implementation to insert the contents of strerror(errno).

That is a glibc-only extension in printf(), but mandatory (and supported in ALL platforms) in syslog(). However, you are correct that:

Quoted 6 lines

This is not something that
should ever be used in trace-events files because several of the backends do not use the format string and so this error information is invisible to them. The errno value should be given as an explicit trace argument instead. Use of '%m' should also be avoided as it is not portable to all QEMU build targets.

If all of our traces are compiled to syslog(), it would be portable, but that's not the case.

What's more, using %m is subject to race conditions - the more code you have between when the format string containing %m is given, and the point where the actual error message is emitted, the higher the probability that some of the intermediate code could have stomped on errno in the meantime, rendering the logged message incorrect. And forcing logging wrappers to save/restore the current state of errno, just in case they might encounter %m, can get wasteful.

Should checkpatch be taught to flag %m as suspicious?

However, tracing the errno value is NOT what %m did; I'd rather see...

Quoted 8 lines
---
 hw/vfio/pci.c                 | 2 +-
 hw/vfio/trace-events          | 2 +-
 scripts/tracetool/__init__.py | 4 ++++
 3 files changed, 6 insertions(+), 2 deletions(-)
Quoted 12 lines

hw/vfio/pci.c static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) · line 2584

Quoted 10 lines
hw/vfio/pci.c +1 -1
index c0cb1ec289..85f1908cfe 100644
@@ -2581,7 +2581,7 @@ static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp)
     ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info);
     if (ret) {
         /* This can fail for an old kernel or legacy PCI dev */
-        trace_vfio_populate_device_get_irq_info_failure();
+        trace_vfio_populate_device_get_irq_info_failure(errno);

trace_vfio_populate_device_get_irq_info_failure(strerror(errno))

The caveat is that 'strerror' is not required to be thread safe, however, given that this is Linux only code I guess we can assume the glibc impl which fortunately is thread safe.

On this point though, does anyone know of any platforms we support[1], or are likely to support in future, where 'strerror' is *not* thread safe ?

I see rumours that Windows strerror() is not thread safe but not found a concrete source for that.

[1] https://qemu.weilnetz.de/doc/qemu-doc.html#Supported-build-platforms

Quoted 6 lines
Quoted 6 lines
scripts/tracetool/__init__.py +0 -0
index 3478ac93ab..6fca674936 100644
@@ -274,6 +274,10 @@ class Event(object):
         props = groups["props"].split()
Quoted 8 lines

scripts/tracetool/__init__.py class Event(object): · line 279

Quoted 5 lines
scripts/tracetool/__init__.py +3 -0
         fmt = groups["fmt"]
         fmt_trans = groups["fmt_trans"]
+        if fmt.find("%m") != -1 or fmt_trans.find("%m") != -1:
+            raise ValueError("Event format '%m' is forbidden, pass the error "
+                             "as an explicit trace argument")

Whether or not checkpatch decides to flag %m as suspicious in the overall code base, I like that you are forbidding it in trace files.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#18 lore raw

Re: [Qemu-devel] [PATCH v2 3/4] trace: forbid use of %m in trace event format strings

In reply to Eric Blake
Quoted 40 lines · Eric Blake
Quoted 5 lines · Daniel P. Berrangé
Quoted 5 lines
Quoted 4 lines

On this point though, does anyone know of any platforms we support[1], or are likely to support in future, where 'strerror' is *not* thread safe ?

Quoted 8 lines
Quoted 5 lines

I'm not coming up with one, and I think the problem is independent of this series (if we DO have a problem, it's a series all its own to eradicate the use of strerror() in favor of something safer, either picking strerror_l() or dealing with the glibc vs. BSD differences in strerror_r()).

Agree that its not really something for this series - this just made me think of it again.

Shoot - FreeBSD strerror() is not threadsafe:
https://github.com/freebsd/freebsd/blob/master/lib/libc/string/strerror.c#L119

char *
strerror(int num)

{
	static char ebuf[NL_TEXTMAX];

	if (strerror_r(num, ebuf, sizeof(ebuf)) != 0)
		errno = EINVAL;
	return (ebuf);
}
Quoted 5 lines

We went through the scrubbing in libvirt to use the sane, but still tedious to call, variant of strerror_r() many years ago. With luck though it is a worry that can be confined the dustbin of ancient UNIX history....unless someone can point to evidence to the contrary ?

libvirt has it easy - they let gnulib do all the work of futzing around with getting a working strerror() despite platform bugs and despite glibc's insistence on a non-POSIX signature if _GNU_SOURCE is defined. We'll have to do a bit more legwork.

Quoted 4 lines

That said, I've added it to:
https://wiki.qemu.org/Contribute/BiteSizedTasks#Error_checking

if someone wants to do the grunt work.

I think we should change that task to switch to g_strerror() from glib instead ... as far as I can see, this is a proper wrapper around strerror_r(), so we don't have to deal with the implementation oddities of strerror_r() in QEMU.

Thomas

#19 lore raw

Re: [Qemu-devel] [PATCH v2 3/4] trace: forbid use of %m in trace event format strings

Quoted 44 lines · Thomas Huth
Quoted 43 lines · Eric Blake
Quoted 5 lines · Daniel P. Berrangé
Quoted 5 lines
Quoted 4 lines

On this point though, does anyone know of any platforms we support[1], or are likely to support in future, where 'strerror' is *not* thread safe ?

Quoted 8 lines
Quoted 5 lines

I'm not coming up with one, and I think the problem is independent of this series (if we DO have a problem, it's a series all its own to eradicate the use of strerror() in favor of something safer, either picking strerror_l() or dealing with the glibc vs. BSD differences in strerror_r()).

Agree that its not really something for this series - this just made me think of it again.

Shoot - FreeBSD strerror() is not threadsafe:
https://github.com/freebsd/freebsd/blob/master/lib/libc/string/strerror.c#L119

char *
strerror(int num)

{
	static char ebuf[NL_TEXTMAX];

	if (strerror_r(num, ebuf, sizeof(ebuf)) != 0)
		errno = EINVAL;
	return (ebuf);
}
Quoted 5 lines

We went through the scrubbing in libvirt to use the sane, but still tedious to call, variant of strerror_r() many years ago. With luck though it is a worry that can be confined the dustbin of ancient UNIX history....unless someone can point to evidence to the contrary ?

libvirt has it easy - they let gnulib do all the work of futzing around with getting a working strerror() despite platform bugs and despite glibc's insistence on a non-POSIX signature if _GNU_SOURCE is defined. We'll have to do a bit more legwork.

That said, I've added it to:
https://wiki.qemu.org/Contribute/BiteSizedTasks#Error_checking

Quoted 6 lines
Quoted 1 line

if someone wants to do the grunt work.

I think we should change that task to switch to g_strerror() from glib instead ... as far as I can see, this is a proper wrapper around strerror_r(), so we don't have to deal with the implementation oddities of strerror_r() in QEMU.

Yeah, I think using g_strerror() makes sense. We've just adopted that in libvirt precisely to avoid these platform portability oddities and the really unpleasant API calling convention of strerror_r().

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|