trace: make systemtap easier to use for simple logging
v2 cover trace: make systemtap easier to use for simple logging
This is a followup to
v1: https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg04173.html
Changed in v2:
- Fix safety of QXL logging
- Handle format specifier macros is a more reliable manner
- Fix trace-events files missing newline
- Remove use of %m formats
Daniel P. Berrangé (4):
display: ensure qxl log_buf is a nul terminated string
trace: enforce that every trace-events file has a final newline
trace: forbid use of %m in trace event format strings
trace: add ability to do simple printf logging via systemtap
Makefile | 3 + Makefile.target | 11 +- docs/devel/tracing.txt | 32 +++++ hw/display/qxl.c | 3 +- hw/display/trace-events | 2 +- hw/gpio/trace-events | 2 +- hw/vfio/pci.c | 2 +- hw/vfio/trace-events | 2 +- scripts/qemu-trace-stap | 167 +++++++++++++++++++++++++++ scripts/tracetool/__init__.py | 6 + scripts/tracetool/format/log_stap.py | 127 ++++++++++++++++++++ 11 files changed, 351 insertions(+), 6 deletions(-) create mode 100755 scripts/qemu-trace-stap create mode 100644 scripts/tracetool/format/log_stap.py
Signature · 1 line
2.20.1
PATCH v2 1/4 display: ensure qxl log_buf is a nul terminated string
The QXL_IO_LOG command allows the guest to send log messages to the host via a buffer in the QXLRam struct. QEMU prints these to the console if the qxl 'guestdebug' option is set to non-zero. It will also feed them to the trace subsystem if any backends are built-in.
In both cases the log_buf data will get treated as being as a nul terminated string, by the printf '%s' format specifier and / or other code reading the buffer.
QEMU does nothing to guarantee that the log_buf really is nul terminated, so there is potential for out of bounds array access.
This would affect any QEMU which has the log, syslog or ftrace trace backends built into QEMU. It can only be triggered if the 'qxl_io_log' trace event is enabled, however, so they are not vulnerable without specific administrative action to enable this.
It would also affect QEMU if the 'guestdebug' parameter is set to a non-zero value, which again is not the default and requires explicit admin opt-in.
--- hw/display/qxl.c | 3 ++- hw/display/trace-events | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)
hw/display/qxl.c +2 -1
@@ -1763,7 +1763,8 @@ async_common:qxl_set_mode (d, val,0 );break ;case QXL_IO_LOG : -trace_qxl_io_log (d->id, d->ram->log_buf); + d->ram->log_buf[sizeof (d->ram->log_buf) -1 ] ='\0' ; +trace_qxl_io_log (d->id, (const char *)d->ram->log_buf);if (d->guestdebug) {fprintf (stderr,"qxl/guest-%d: %" PRId64": %s" , d->id,qemu_clock_get_ns (QEMU_CLOCK_VIRTUAL ), d->ram->log_buf);
hw/display/trace-events +1 -1
@@ -72,7 +72,7 @@ qxl_interface_update_area_complete_rest(int qid, uint32_t num_updated_rects) "%d qxl_interface_update_area_complete_overflow(int qid, int max) "%d max=%d" qxl_interface_update_area_complete_schedule_bh(int qid, uint32_t num_dirty) "%d #dirty=%d" qxl_io_destroy_primary_ignored(int qid, const char *mode) "%d %s" -qxl_io_log(int qid, const uint8_t *log_buf) "%d %s" +qxl_io_log(int qid, const char *log_buf) "%d %s" qxl_io_read_unexpected(int qid) "%d" qxl_io_unexpected_vga_mode(int qid, uint64_t addr, uint64_t val, const char *desc) "%d 0x%"PRIx64"=%"PRIu64" (%s)" qxl_io_write(int qid, const char *mode, uint64_t addr, const char *aname, uint64_t val, unsigned size, int async) "%d %s addr=%"PRIu64 " (%s) val=%"PRIu64" size=%u async=%d"
Signature · 1 line
2.20.1
PATCH v2 2/4 trace: enforce that every trace-events file has a final newline
In reply to Daniel P. BerrangéWhen generating the trace-events-all file, the build system simply concatenates all the individual trace-events files. If any one of those files does not have a final newline, the printf format string will have the contents of the first line of the next file appended to it, which is usually a '#' comment.
--- hw/gpio/trace-events | 2 +- scripts/tracetool/__init__.py | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-)
hw/gpio/trace-events +1 -1
@@ -4,4 +4,4 @@ nrf51_gpio_read(uint64_t offset, uint64_t r) "offset 0x%" PRIx64 " value 0x%" PRIx64 nrf51_gpio_write(uint64_t offset, uint64_t value) "offset 0x%" PRIx64 " value 0x%" PRIx64 nrf51_gpio_set(int64_t line, int64_t value) "line %" PRIi64 " value %" PRIi64 -nrf51_gpio_update_output_irq(int64_t line, int64_t value) "line %" PRIi64 " value %" PRIi64 +nrf51_gpio_update_output_irq(int64_t line, int64_t value) "line %" PRIi64 " value %" PRIi64
scripts/tracetool/__init__.py +2 -0
@@ -350,6 +350,8 @@ def read_events(fobj, fname): events = [] for lineno, line in enumerate(fobj, 1): + if line[-1] != '\n': + raise ValueError("%s does not end with a new line" % fname) if not line.strip(): continue if line.lstrip().startswith('#'):
Signature · 1 line
2.20.1
PATCH v2 3/4 trace: forbid use of %m in trace event format strings
In reply to Daniel P. BerrangéThe '%m' format specifier instructs glibc's printf() implementation to insert the contents of strerror(errno). This is not something that should ever be used in trace-events files because several of the backends do not use the format string and so this error information is invisible to them. The errno value should be given as an explicit trace argument instead. Use of '%m' should also be avoided as it is not portable to all QEMU build targets.
--- hw/vfio/pci.c | 2 +- hw/vfio/trace-events | 2 +- scripts/tracetool/__init__.py | 4 ++++ 3 files changed, 6 insertions(+), 2 deletions(-)
hw/vfio/pci.c +1 -1
@@ -2581,7 +2581,7 @@ static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) ret =ioctl (vdev->vbasedev.fd,VFIO_DEVICE_GET_IRQ_INFO , &irq_info);if (ret) {/* This can fail for an old kernel or legacy PCI dev */ -trace_vfio_populate_device_get_irq_info_failure (); +trace_vfio_populate_device_get_irq_info_failure (errno); }else if (irq_info.count ==1 ) { vdev->pci_aer = true; }else {
hw/vfio/trace-events +1 -1
@@ -37,7 +37,7 @@ vfio_pci_hot_reset_has_dep_devices(const char *name) "%s: hot reset dependent de vfio_pci_hot_reset_dep_devices(int domain, int bus, int slot, int function, int group_id) "\t%04x:%02x:%02x.%x group %d" vfio_pci_hot_reset_result(const char *name, const char *result) "%s hot reset: %s" vfio_populate_device_config(const char *name, unsigned long size, unsigned long offset, unsigned long flags) "Device %s config:\n size: 0x%lx, offset: 0x%lx, flags: 0x%lx" -vfio_populate_device_get_irq_info_failure(void) "VFIO_DEVICE_GET_IRQ_INFO failure: %m" +vfio_populate_device_get_irq_info_failure(int err) "VFIO_DEVICE_GET_IRQ_INFO failure: %d" vfio_realize(const char *name, int group_id) " (%s) group %d" vfio_mdev(const char *name, bool is_mdev) " (%s) is_mdev %d" vfio_add_ext_cap_dropped(const char *name, uint16_t cap, uint16_t offset) "%s 0x%x@0x%x"
scripts/tracetool/__init__.py +4 -0
@@ -274,6 +274,10 @@ class Event(object): props = groups["props"].split() fmt = groups["fmt"] fmt_trans = groups["fmt_trans"] + if fmt.find("%m") != -1 or fmt_trans.find("%m") != -1: + raise ValueError("Event format '%m' is forbidden, pass the error " + "as an explicit trace argument") + if len(fmt_trans) > 0: fmt = [fmt_trans, fmt] args = Arguments.build(groups["args"])
Signature · 1 line
2.20.1
PATCH v2 4/4 trace: add ability to do simple printf logging via systemtap
In reply to Daniel P. BerrangéThe dtrace systemtap trace backend for QEMU is very powerful but it is also somewhat unfriendly to users who aren't familiar with systemtap, or who don't need its power right now.
stap -e "....some strange script...."
The 'log' backend for QEMU by comparison is very crude but incredibly easy to use:
$ qemu -d trace:qio* ...some args... [email protected]:qio_channel_socket_new Socket new ioc=0x563a8a39d400 [email protected]:qio_task_new Task new task=0x563a891d0570 source=0x563a8a39d400 func=0x563a86f1e6c0 opaque=0x563a89078000 [email protected]:qio_task_thread_start Task thread start task=0x563a891d0570 worker=0x563a86f1ce50 opaque=0x563a891d9d90 [email protected]:qio_task_thread_run Task thread run task=0x563a891d0570 [email protected]:qio_channel_socket_connect_sync Socket connect sync ioc=0x563a8a39d400 addr=0x563a891d9d90 [email protected]:qio_channel_socket_connect_fail Socket connect fail ioc=0x563a8a39d400
This commit introduces a way to do simple printf style logging of probe points using systemtap. In particular it creates another set of tapsets, one per emulator:
/usr/share/systemtap/tapset/qemu-*-log.stp
These pre-define probe functions which simply call printf() on their arguments. The printf() format string is taken from the normal trace-events files, with a little munging to the format specifiers to cope with systemtap's more restrictive syntax.
With this you can now do
$ stap -e 'probe qemu.system.x86_64.log.qio*{}'
22806@1547735341399856820 qio_channel_socket_new Socket new ioc=0x56135d1d7c00
22806@1547735341399862570 qio_task_new Task new task=0x56135cd66eb0 source=0x56135d1d7c00 func=0x56135af746c0 opaque=0x56135bf06400
22806@1547735341399865943 qio_task_thread_start Task thread start task=0x56135cd66eb0 worker=0x56135af72e50 opaque=0x56135c071d70
22806@1547735341399976816 qio_task_thread_run Task thread run task=0x56135cd66eb0We go one step further though and introduce a 'qemu-trace-stap' tool to make this even easier
$ qemu-trace-stap run qemu-system-x86_64 qio* 22806@1547735341399856820 qio_channel_socket_new Socket new ioc=0x56135d1d7c00 22806@1547735341399862570 qio_task_new Task new task=0x56135cd66eb0 source=0x56135d1d7c00 func=0x56135af746c0 opaque=0x56135bf06400 22806@1547735341399865943 qio_task_thread_start Task thread start task=0x56135cd66eb0 worker=0x56135af72e50 opaque=0x56135c071d70 22806@1547735341399976816 qio_task_thread_run Task thread run task=0x56135cd66eb0
This tool is clever in that it will automatically change the SYSTEMTAP_TAPSET env variable to point to the directory containing the right set of probes for the QEMU binary path you give it. This is useful if you have QEMU installed in /usr but are trying to test and trace a binary in /home/berrange/usr/qemu-git. In that case you'd do
$ qemu-trace-stap run /home/berrange/usr/qemu-git/bin/qemu-system-x86_64 qio*
And it'll make sure /home/berrange/usr/qemu-git/share/systemtap/tapset is used for the trace session
The 'qemu-trace-stap' script takes a verbose arg so you can understand what it is running
$ qemu-trace-stap run /home/berrange/usr/qemu-git/bin/qemu-system-x86_64 qio*
Using tapset dir '/home/berrange/usr/qemu-git/share/systemtap/tapset' for binary '/home/berrange/usr/qemu-git/bin/qemu-system-x86_64'
Compiling script 'probe qemu.system.x86_64.log.qio* {}'
Running script, <Ctrl>-c to quit
...trace output...It can enable multiple probes at once
$ qemu-trace-stap run qemu-system-x86_64 qio* qcrypto* buffer*
Finally if you can't remember what probes are valid it can tell you
$ qemu-trace-stap list qemu-system-x86_64 ahci_check_irq ahci_cmd_done ahci_dma_prepare_buf ahci_dma_prepare_buf_fail ahci_dma_rw_buf ahci_irq_lower ...snip...
Or list just those matching a prefix pattern
$ qemu-trace-stap list -v qemu-system-x86_64 qio* Using tapset dir '/home/berrange/usr/qemu-git/share/systemtap/tapset' for binary '/home/berrange/usr/qemu-git/bin/qemu-system-x86_64' Listing probes with name 'qemu.system.x86_64.log.qio*' qio_channel_command_abort qio_channel_command_new_pid qio_channel_command_new_spawn qio_channel_command_wait qio_channel_file_new_fd ...snip...
--- Makefile | 3 + Makefile.target | 11 +- docs/devel/tracing.txt | 32 +++++ scripts/qemu-trace-stap | 167 +++++++++++++++++++++++++++ scripts/tracetool/format/log_stap.py | 127 ++++++++++++++++++++ 5 files changed, 339 insertions(+), 1 deletion(-) create mode 100755 scripts/qemu-trace-stap create mode 100644 scripts/tracetool/format/log_stap.py
Makefile +3 -0
@@ -736,6 +736,9 @@ endif ifneq ($(HELPERS-y),) $(call install-prog,$(HELPERS-y),$(DESTDIR)$(libexecdir)) endif +ifdef CONFIG_TRACE_SYSTEMTAP + $(INSTALL_PROG) "scripts/qemu-trace-stap" $(DESTDIR)$(bindir) +endif ifneq ($(BLOBS),) set -e; for x in $(BLOBS); do \ $(INSTALL_DATA) $(SRC_PATH)/pc-bios/$$x "$(DESTDIR)$(qemu_datadir)"; \
Makefile.target +10 -1
@@ -45,7 +45,7 @@ config-target.h: config-target.h-timestamp config-target.h-timestamp: config-target.mak ifdef CONFIG_TRACE_SYSTEMTAP -stap: $(QEMU_PROG).stp-installed $(QEMU_PROG).stp $(QEMU_PROG)-simpletrace.stp +stap: $(QEMU_PROG).stp-installed $(QEMU_PROG).stp $(QEMU_PROG)-simpletrace.stp $(QEMU_PROG)-log.stp ifdef CONFIG_USER_ONLY TARGET_TYPE=user
@@ -84,6 +84,14 @@ $(QEMU_PROG)-simpletrace.stp: $(BUILD_DIR)/trace-events-all $(tracetool-y) --probe-prefix=qemu.$(TARGET_TYPE).$(TARGET_NAME) \ $< > $@,"GEN","$(TARGET_DIR)$(QEMU_PROG)-simpletrace.stp") +$(QEMU_PROG)-log.stp: $(BUILD_DIR)/trace-events-all $(tracetool-y) + $(call quiet-command,$(TRACETOOL) \ + --group=all \ + --format=log-stap \ + --backends=$(TRACE_BACKENDS) \ + --probe-prefix=qemu.$(TARGET_TYPE).$(TARGET_NAME) \ + $< > $@,"GEN","$(TARGET_DIR)$(QEMU_PROG)-log.stp") + else stap: endif
@@ -227,6 +235,7 @@ ifdef CONFIG_TRACE_SYSTEMTAP $(INSTALL_DIR) "$(DESTDIR)$(qemu_datadir)/../systemtap/tapset" $(INSTALL_DATA) $(QEMU_PROG).stp-installed "$(DESTDIR)$(qemu_datadir)/../systemtap/tapset/$(QEMU_PROG).stp" $(INSTALL_DATA) $(QEMU_PROG)-simpletrace.stp "$(DESTDIR)$(qemu_datadir)/../systemtap/tapset/$(QEMU_PROG)-simpletrace.stp" + $(INSTALL_DATA) $(QEMU_PROG)-log.stp "$(DESTDIR)$(qemu_datadir)/../systemtap/tapset/$(QEMU_PROG)-log.stp" endif GENERATED_FILES += config-target.h
docs/devel/tracing.txt +32 -0
@@ -317,6 +317,38 @@ probes: --target-name x86_64 \ <trace-events-all >qemu.stp +To facilitate simple usage of systemtap where there merely needs to be printf +logging of certain probes, a helper script "qemu-trace-stap" is provided + +This can list all probes available for a given QEMU emulator binary: + + $ qemu-trace-stap list qemu-system-x86_64 + ahci_check_irq + ahci_cmd_done + ahci_dma_prepare_buf + ahci_dma_prepare_buf_fail + ahci_dma_rw_buf + ahci_irq_lower + ...snip... + +Optionally restricted to a pattern prefix: + + $ qemu-trace-stap list qemu-system-x86_64 qio* + qio_channel_command_abort + qio_channel_command_new_pid + qio_channel_command_new_spawn + qio_channel_command_wait + qio_channel_file_new_fd + ...snip... + +It can then run a simple systemtap script which prints each requested probe + + $ qemu-trace-stap run qemu-system-x86_64 qio* + 22806@1547735341399856820 qio_channel_socket_new Socket new ioc=0x56135d1d7c00 + 22806@1547735341399862570 qio_task_new Task new task=0x56135cd66eb0 source=0x56135d1d7c00 func=0x56135af746c0 opaque=0x56135bf06400 + 22806@1547735341399865943 qio_task_thread_start Task thread start task=0x56135cd66eb0 worker=0x56135af72e50 opaque=0x56135c071d70 + 22806@1547735341399976816 qio_task_thread_run Task thread run task=0x56135cd66eb0 + == Trace event properties == Each event in the "trace-events-all" file can be prefixed with a space-separated
scripts/qemu-trace-stap new file +167 -0
167 lines not on this page · raw message
scripts/tracetool/format/log_stap.py new file +127 -0
127 lines not on this page · raw message
Signature · 1 line
2.20.1
Re: [Qemu-devel] [PATCH v2 1/4] display: ensure qxl log_buf is a nul terminated string
In reply to Daniel P. BerrangéQuoted 21 lines · Daniel P. Berrangé
The
QXL_IO_LOGcommand allows the guest to send log messages to the host via a buffer in the QXLRam struct. QEMU prints these to the console if the qxl 'guestdebug' option is set to non-zero. It will also feed them to the trace subsystem if any backends are built-in.In both cases the
log_bufdata will get treated as being as a nul terminated string, by the printf '%s' format specifier and / or other code reading the buffer.QEMU does nothing to guarantee that the
log_bufreally is nul terminated, so there is potential for out of bounds array access.This would affect any QEMU which has the log, syslog or ftrace trace backends built into QEMU. It can only be triggered if the '
qxl_io_log' trace event is enabled, however, so they are not vulnerable without specific administrative action to enable this.It would also affect QEMU if the 'guestdebug' parameter is set to a non-zero value, which again is not the default and requires explicit admin opt-in.
Quoted 5 lines
Signed-off-by Daniel P. Berrangé <[email protected]>--- hw/display/qxl.c | 3 ++- hw/display/trace-events | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)
hw/display/qxl.c async_common: · line 1766
Quoted 7 lines
+++
b/hw/display/qxl.chw/display/qxl.c +1
-1@@ -1763,7 +1763,8 @@ async_common:qxl_set_mode (d, val,0 );break ;case QXL_IO_LOG : -trace_qxl_io_log (d->id, d->ram->log_buf); + d->ram->log_buf[sizeof (d->ram->log_buf) -1 ] ='\0' ;
May lose a character from the log, but the improved safety is worth it.
Quoted 18 lines
hw/display/qxl.c +1
-0+trace_qxl_io_log (d->id, (const char *)d->ram->log_buf);if (d->guestdebug) {fprintf (stderr,"qxl/guest-%d: %" PRId64": %s" , d->id,qemu_clock_get_ns (QEMU_CLOCK_VIRTUAL ), d->ram->log_buf);hw/display/trace-events +1
-1@@ -72,7 +72,7 @@ qxl_interface_update_area_complete_rest(int qid, uint32_t num_updated_rects) "%d qxl_interface_update_area_complete_overflow(int qid, int max) "%d max=%d" qxl_interface_update_area_complete_schedule_bh(int qid, uint32_t num_dirty) "%d #dirty=%d" qxl_io_destroy_primary_ignored(int qid, const char *mode) "%d %s" -qxl_io_log(int qid, const uint8_t *log_buf) "%d %s" +qxl_io_log(int qid, const char *log_buf) "%d %s" qxl_io_read_unexpected(int qid) "%d" qxl_io_unexpected_vga_mode(int qid, uint64_t addr, uint64_t val, const char *desc) "%d 0x%"PRIx64"=%"PRIu64" (%s)" qxl_io_write(int qid, const char *mode, uint64_t addr, const char *aname, uint64_t val, unsigned size, int async) "%d %s addr=%"PRIu64 " (%s) val=%"PRIu64" size=%u async=%d"
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [Qemu-devel] [PATCH v2 2/4] trace: enforce that every trace-events file has a final newline
In reply to Daniel P. BerrangéQuoted 6 lines · Daniel P. Berrangé
When generating the trace-events-all file, the build system simply concatenates all the individual trace-events files. If any one of those files does not have a final newline, the printf format string will have the contents of the first line of the next file appended to it, which is usually a '#' comment.
Quoted 6 lines
Signed-off-by Daniel P. Berrangé <[email protected]>--- hw/gpio/trace-events | 2 +- scripts/tracetool/__init__.py | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-)
Does checkpatch and/or patchew flag what are intended to be text files but which lack a trailing newline? If not, how hard would it be to get them to do so?
scripts/tracetool/__init__.py
Quoted 12 lines
hw/gpio/trace-events +1
-1@@ -4,4 +4,4 @@ nrf51_gpio_read(uint64_t offset, uint64_t r) "offset 0x%" PRIx64 " value 0x%" PRIx64 nrf51_gpio_write(uint64_t offset, uint64_t value) "offset 0x%" PRIx64 " value 0x%" PRIx64 nrf51_gpio_set(int64_t line, int64_t value) "line %" PRIi64 " value %" PRIi64 -nrf51_gpio_update_output_irq(int64_t line, int64_t value) "line %" PRIi64 " value %" PRIi64 +nrf51_gpio_update_output_irq(int64_t line, int64_t value) "line %" PRIi64 " value %" PRIi64scripts/tracetool/__init__.py +0
-0
Fixes the problem file...
scripts/tracetool/__init__.py def read_events(fobj, fname): · line 354
Quoted 9 lines
index
0e3c9e146c..3478ac93ab100644scripts/tracetool/__init__.py +2
-0@@ -350,6 +350,8 @@ def read_events(fobj, fname): events = [] for lineno, line in enumerate(fobj, 1): + if line[-1] != '\n': + raise ValueError("%s does not end with a new line" % fname)
and ensures that no future trace-events files will have the problem, whether or not checkpatch flags other such files.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [Qemu-devel] [PATCH v2 1/4] display: ensure qxl log_buf is a nul terminated string
In reply to Eric BlakeQuoted 30 lines · Eric Blake
Quoted 21 lines · Daniel P. Berrangé
The
QXL_IO_LOGcommand allows the guest to send log messages to the host via a buffer in the QXLRam struct. QEMU prints these to the console if the qxl 'guestdebug' option is set to non-zero. It will also feed them to the trace subsystem if any backends are built-in.In both cases the
log_bufdata will get treated as being as a nul terminated string, by the printf '%s' format specifier and / or other code reading the buffer.QEMU does nothing to guarantee that the
log_bufreally is nul terminated, so there is potential for out of bounds array access.This would affect any QEMU which has the log, syslog or ftrace trace backends built into QEMU. It can only be triggered if the '
qxl_io_log' trace event is enabled, however, so they are not vulnerable without specific administrative action to enable this.It would also affect QEMU if the 'guestdebug' parameter is set to a non-zero value, which again is not the default and requires explicit admin opt-in.
Quoted 5 lines
Signed-off-by Daniel P. Berrangé <[email protected]>--- hw/display/qxl.c | 3 ++- hw/display/trace-events | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)Reviewed-by Eric Blake <[email protected]>
Quoted 9 lines
hw/display/qxl.c async_common: · line 1766
Quoted 7 lines
+++
b/hw/display/qxl.chw/display/qxl.c +1
-1@@ -1763,7 +1763,8 @@ async_common:qxl_set_mode (d, val,0 );break ;case QXL_IO_LOG : -trace_qxl_io_log (d->id, d->ram->log_buf); + d->ram->log_buf[sizeof (d->ram->log_buf) -1 ] ='\0' ;May lose a character from the log, but the improved safety is worth it.
In practice it won't. The Xorg QXL driver and Windows XDDM DOD driver don't use this feature at all. The Windows QXL miniport driver uses snprintf when populating the buffer so will already have truncated if it was going to overflow.
Quoted 11 lines
Quoted 11 lines
hw/display/qxl.c +1
-0+trace_qxl_io_log (d->id, (const char *)d->ram->log_buf);if (d->guestdebug) {fprintf (stderr,"qxl/guest-%d: %" PRId64": %s" , d->id,qemu_clock_get_ns (QEMU_CLOCK_VIRTUAL ), d->ram->log_buf);hw/display/trace-events +0
-0@@ -72,7 +72,7 @@ qxl_interface_update_area_complete_rest(int qid, uint32_t num_updated_rects) "%d qxl_interface_update_area_complete_overflow(int qid, int max) "%d max=%d"
Quoted 14 lines
Quoted 8 lines
hw/display/trace-events +1
-1qxl_interface_update_area_complete_schedule_bh(int qid, uint32_t num_dirty) "%d #dirty=%d" qxl_io_destroy_primary_ignored(int qid, const char *mode) "%d %s" -qxl_io_log(int qid, const uint8_t *log_buf) "%d %s" +qxl_io_log(int qid, const char *log_buf) "%d %s" qxl_io_read_unexpected(int qid) "%d" qxl_io_unexpected_vga_mode(int qid, uint64_t addr, uint64_t val, const char *desc) "%d 0x%"PRIx64"=%"PRIu64" (%s)" qxl_io_write(int qid, const char *mode, uint64_t addr, const char *aname, uint64_t val, unsigned size, int async) "%d %s addr=%"PRIu64 " (%s) val=%"PRIu64" size=%u async=%d"Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [Qemu-devel] [PATCH v2 3/4] trace: forbid use of %m in trace event format strings
In reply to Daniel P. BerrangéQuoted 2 lines · Daniel P. Berrangé
The '%m' format specifier instructs glibc's
printf()implementation to insert the contents of strerror(errno).
That is a glibc-only extension in printf(), but mandatory (and supported in ALL platforms) in syslog(). However, you are correct that:
Quoted 6 lines
This is not something that
should ever be used in trace-events files because several of the backends do not use the format string and so this error information is invisible to them. The errno value should be given as an explicit trace argument instead. Use of '%m' should also be avoided as it is not portable to all QEMU build targets.
If all of our traces are compiled to syslog(), it would be portable, but that's not the case.
What's more, using %m is subject to race conditions - the more code you have between when the format string containing %m is given, and the point where the actual error message is emitted, the higher the probability that some of the intermediate code could have stomped on errno in the meantime, rendering the logged message incorrect. And forcing logging wrappers to save/restore the current state of errno, just in case they might encounter %m, can get wasteful.
Should checkpatch be taught to flag %m as suspicious?
However, tracing the errno value is NOT what %m did; I'd rather see...
Quoted 13 lines
Signed-off-by Daniel P. Berrangé <[email protected]>--- hw/vfio/pci.c | 2 +- hw/vfio/trace-events | 2 +- scripts/tracetool/__init__.py | 4 ++++ 3 files changed, 6 insertions(+), 2 deletions(-)hw/vfio/pci.c +0
-0
hw/vfio/pci.c static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) · line 2584
Quoted 5 lines
hw/vfio/pci.c +1
-1@@ -2581,7 +2581,7 @@ static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) ret =ioctl (vdev->vbasedev.fd,VFIO_DEVICE_GET_IRQ_INFO , &irq_info);if (ret) {/* This can fail for an old kernel or legacy PCI dev */ -trace_vfio_populate_device_get_irq_info_failure (); +trace_vfio_populate_device_get_irq_info_failure (errno);
trace_vfio_populate_device_get_irq_info_failure(strerror(errno))
hw/vfio/trace-events vfio_pci_hot_reset_has_dep_devices(const char *name) "%s: hot reset dependent de · line 40
Quoted 13 lines
hw/vfio/pci.c +0
-0}else if (irq_info.count ==1 ) { vdev->pci_aer = true; }else {hw/vfio/trace-events +1
-1@@ -37,7 +37,7 @@ vfio_pci_hot_reset_has_dep_devices(const char *name) "%s: hot reset dependent de vfio_pci_hot_reset_dep_devices(int domain, int bus, int slot, int function, int group_id) "\t%04x:%02x:%02x.%x group %d" vfio_pci_hot_reset_result(const char *name, const char *result) "%s hot reset: %s" vfio_populate_device_config(const char *name, unsigned long size, unsigned long offset, unsigned long flags) "Device %s config:\n size: 0x%lx, offset: 0x%lx, flags: 0x%lx" -vfio_populate_device_get_irq_info_failure(void) "VFIO_DEVICE_GET_IRQ_INFO failure: %m" +vfio_populate_device_get_irq_info_failure(int err) "VFIO_DEVICE_GET_IRQ_INFO failure: %d"
vfio_populate_device_get_irq_info_failur(const char *err) "VFIO_DEVICE_GET_IRQ_INFO failure: %s"
...if we are trying to match the original intent.
Quoted 8 lines
hw/vfio/trace-events +0
-0vfio_realize(const char *name, int group_id) " (%s) group %d" vfio_mdev(const char *name, bool is_mdev) " (%s) is_mdev %d" vfio_add_ext_cap_dropped(const char *name, uint16_t cap, uint16_t offset) "%s 0x%x@0x%x"scripts/tracetool/__init__.py +0
-0
scripts/tracetool/__init__.py class Event(object): · line 279
Quoted 6 lines
scripts/tracetool/__init__.py +3
-0@@ -274,6 +274,10 @@ class Event(object): props = groups["props"].split() fmt = groups["fmt"] fmt_trans = groups["fmt_trans"] + if fmt.find("%m") != -1 or fmt_trans.find("%m") != -1: + raise ValueError("Event format '%m' is forbidden, pass the error " + "as an explicit trace argument")
Whether or not checkpatch decides to flag %m as suspicious in the overall code base, I like that you are forbidding it in trace files.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [Qemu-devel] [PATCH v2 4/4] trace: add ability to do simple printf logging via systemtap
In reply to Daniel P. BerrangéQuoted 4 lines · Daniel P. Berrangé
The dtrace systemtap trace backend for QEMU is very powerful but it is also somewhat unfriendly to users who aren't familiar with systemtap, or who don't need its power right now.
Quoted 8 lines
stap -e "....some strange script...."
We go one step further though and introduce a 'qemu-trace-stap' tool to make this even easier
$ qemu-trace-stap run qemu-system-x86_64 qio* 22806@1547735341399856820 qio_channel_socket_new Socket new ioc=0x56135d1d7c00
One thing I did not immediately get on the first read, and therefore which may be worth mentioning in the docs portion of the patch:
Am I correct that stap is always run as a separate process from the process(es) being traced, and that you can start the two in either order (qemu first, then stap; or stap first, then qemu), where the tracing is effectively output as long as both processes are running and trace points being hit in the qemu process?
Quoted 7 lines
Signed-off-by Daniel P. Berrangé <[email protected]>--- Makefile | 3 + Makefile.target | 11 +- docs/devel/tracing.txt | 32 +++++ scripts/qemu-trace-stap | 167 +++++++++++++++++++++++++++ scripts/tracetool/format/log_stap.py | 127 ++++++++++++++++++++
Overall, I'm impressed with the concept, but my familiarity with stap and python is weak enough that I'm not leaving a formal R-b. Still, I'll give it a shot at looking for anything that stands out as I read through.
Quoted 9 lines
5 files changed, 339 insertions(+), 1 deletion(-) create mode 100755 scripts/qemu-trace-stap create mode 100644 scripts/tracetool/format/log_stap.pyMakefile +0
-0
Makefile endif · line 741
Quoted 6 lines
Makefile +3
-0@@ -736,6 +736,9 @@ endif ifneq ($(HELPERS-y),) $(call install-prog,$(HELPERS-y),$(DESTDIR)$(libexecdir)) endif +ifdef CONFIG_TRACE_SYSTEMTAP + $(INSTALL_PROG) "scripts/qemu-trace-stap" $(DESTDIR)$(bindir) +endif
No man page being installed? If we're planning on this binary being available on a similar par with qemu-io, it deserves some installed documentation.
Makefile probes: · line 322
Quoted 8 lines
Makefile +4
-0+++ b/docs/devel/tracing.txt@@ -317,6 +317,38 @@ probes: --target-name x86_64 \ <trace-events-all >qemu.stp +To facilitate simple usage of systemtap where there merely needs to be printf +logging of certain probes, a helper script "qemu-trace-stap" is provided +
Trailing '.' would look better
Quoted 12 lines
Makefile +12
-0+This can list all probes available for a given QEMU emulator binary: + + $ qemu-trace-stap list qemu-system-x86_64 + ahci_check_irq + ahci_cmd_done + ahci_dma_prepare_buf + ahci_dma_prepare_buf_fail + ahci_dma_rw_buf + ahci_irq_lower + ...snip... + +Optionally restricted to a pattern prefix:
Makefile probes: · line 344
Quoted 10 lines
Makefile +10
-0+ + $ qemu-trace-stap list qemu-system-x86_64 qio* + qio_channel_command_abort + qio_channel_command_new_pid + qio_channel_command_new_spawn + qio_channel_command_wait + qio_channel_file_new_fd + ...snip... + +It can then run a simple systemtap script which prints each requested probe
Trailing ':' would look better
Quoted 19 lines
Makefile +17
-0+ + $ qemu-trace-stap run qemu-system-x86_64 qio* + 22806@1547735341399856820 qio_channel_socket_new Socket new ioc=0x56135d1d7c00 + 22806@1547735341399862570 qio_task_new Task new task=0x56135cd66eb0 source=0x56135d1d7c00 func=0x56135af746c0 opaque=0x56135bf06400 + 22806@1547735341399865943 qio_task_thread_start Task thread start task=0x56135cd66eb0 worker=0x56135af72e50 opaque=0x56135c071d70 + 22806@1547735341399976816 qio_task_thread_run Task thread run task=0x56135cd66eb0 + +++ b/scripts/qemu-trace-stap + +def main(): + parser = argparse.ArgumentParser(description="QEMU SystemTap trace tool") + + subparser = parser.add_subparsers(help="commands") + subparser.required = True + subparser.dest = "command" + + runparser = subparser.add_parser("run", help="Run a trace session",
Makefile probes:
Quoted 10 lines
Makefile +10
-0+ formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=""" + +To watch all trace points on the qemu-system-x86_64 binary: + + %(argv0)s run qemu-system-x86_64 + +To only watch the trace points matching the qio* and qcrypto* patterns + + %(argv0)s run qemu-system-x86_64 qio* qcrypto*
This output is not copy-pastable - it includes shell globs which might inadvertantly expand based on the contents of $PWD. Should you amend the output to be properly quoted for reuse in shell commands?
Quoted 9 lines
+""" % {"argv0": sys.argv[0]})
Makefile +8
-0+ runparser.set_defaults(func=cmd_run) + runparser.add_argument("-v", "--verbose", help="Print verbose progress info", + action='store_true') + runparser.add_argument("binary", help="QEMU system or user emulator binary") + runparser.add_argument("probes", help="Probe names or wildcards", + nargs=argparse.REMAINDER) + + listparser = subparser.add_parser("list", help="List probe points",
Makefile
Quoted 10 lines
Makefile +10
-0+ formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=""" + +To list all trace points on the qemu-system-x86_64 binary: + + %(argv0)s list qemu-system-x86_64 + +To only list the trace points matching the qio* and qcrypto* patterns + + %(argv0)s list qemu-system-x86_64 qio* qcrypto*
and again
Quoted 45 lines
+""" % {"argv0": sys.argv[0]})
Makefile +14
-0+ listparser.set_defaults(func=cmd_list) + listparser.add_argument("-v", "--verbose", help="Print verbose progress info", + action='store_true') + listparser.add_argument("binary", help="QEMU system or user emulator binary") + listparser.add_argument("probes", help="Probe names or wildcards", + nargs=argparse.REMAINDER) + + args = parser.parse_args() + + args.func(args) + sys.exit(0) + +if __name__ == '__main__': + main()scripts/tracetool/format/log_stap.py new file +24
-0@@ -0,0 +1,127 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- + +""" +Generate .stp file that printfs log messages (DTrace with SystemTAP only). +""" + +__author__ = "Daniel P. Berrange <[email protected]>" +__copyright__ = "Copyright (C) 2014-2019, Red Hat, Inc." +__license__ = "GPL version 2 or (at your option) any later version" + +__maintainer__ = "Daniel Berrange" +__email__ = "[email protected]" + +import re + +from tracetool import out +from tracetool.backend.dtrace import binary, probeprefix +from tracetool.backend.simple import is_string +from tracetool.format.stap import stap_escape + +def global_var_name(name): + return probeprefix().replace(".", "_") + "_" + name +
scripts/tracetool/format/log_stap.py line 34
Quoted 10 lines
scripts/tracetool/format/log_stap.py new file +10
-0+STATE_SKIP = 0 +STATE_LITERAL = 1 +STATE_MACRO = 2 + +def c_macro_to_format(macro): + if macro.startswith("PRI"): + return macro[3] + + if macro == "TARGET_FMT_plx": + return "%016x"
Do we really want to permit TARGET_FMT in our trace-events files, or should that be fixed in hw/tpm/trace-events? See commit 73ff0610.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [Qemu-devel] [PATCH v2 3/4] trace: forbid use of %m in trace event format strings
In reply to Eric BlakeQuoted 13 lines · On Fri, 18 Jan 2019 11:50:39 -0600 Eric Blake <[email protected]> wrote:
Quoted 2 lines · Daniel P. Berrangé
The '%m' format specifier instructs glibc's
printf()implementation to insert the contents of strerror(errno).That is a glibc-only extension in
printf(), but mandatory (and supported in ALL platforms) insyslog(). However, you are correct that:Quoted 6 lines
This is not something that
should ever be used in trace-events files because several of the backends do not use the format string and so this error information is invisible to them. The errno value should be given as an explicit trace argument instead. Use of '%m' should also be avoided as it is not portable to all QEMU build targets.
It's portable to all QEMU build targets that support vfio, but we can remove it if we want it out of the trace system, which of course does support other targets.
Quoted 23 lines
If all of our traces are compiled to
syslog(), it would be portable, but that's not the case.What's more, using %m is subject to race conditions - the more code you have between when the format string containing %m is given, and the point where the actual error message is emitted, the higher the probability that some of the intermediate code could have stomped on errno in the meantime, rendering the logged message incorrect. And forcing logging wrappers to save/restore the current state of errno, just in case they might encounter %m, can get wasteful.
Should checkpatch be taught to flag %m as suspicious?
However, tracing the errno value is NOT what %m did; I'd rather see...
Quoted 8 lines
Signed-off-by Daniel P. Berrangé <[email protected]>--- hw/vfio/pci.c | 2 +- hw/vfio/trace-events | 2 +- scripts/tracetool/__init__.py | 4 ++++ 3 files changed, 6 insertions(+), 2 deletions(-)
Quoted 12 lines
hw/vfio/pci.c static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) · line 2584
Quoted 10 lines
hw/vfio/pci.c +1
-1@@ -2581,7 +2581,7 @@ static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) ret =ioctl (vdev->vbasedev.fd,VFIO_DEVICE_GET_IRQ_INFO , &irq_info);if (ret) {/* This can fail for an old kernel or legacy PCI dev */ -trace_vfio_populate_device_get_irq_info_failure (); +trace_vfio_populate_device_get_irq_info_failure (errno);trace_vfio_populate_device_get_irq_info_failure(strerror(errno))
Yep, I agree, the intent of the original code is not maintained. Thanks,
Alex
Quoted 37 lines
hw/vfio/trace-events vfio_pci_hot_reset_has_dep_devices(const char *name) "%s: hot reset dependent de · line 40
Quoted 13 lines
hw/vfio/pci.c +0
-0}else if (irq_info.count ==1 ) { vdev->pci_aer = true; }else {hw/vfio/trace-events +1
-1@@ -37,7 +37,7 @@ vfio_pci_hot_reset_has_dep_devices(const char *name) "%s: hot reset dependent de vfio_pci_hot_reset_dep_devices(int domain, int bus, int slot, int function, int group_id) "\t%04x:%02x:%02x.%x group %d" vfio_pci_hot_reset_result(const char *name, const char *result) "%s hot reset: %s" vfio_populate_device_config(const char *name, unsigned long size, unsigned long offset, unsigned long flags) "Device %s config:\n size: 0x%lx, offset: 0x%lx, flags: 0x%lx" -vfio_populate_device_get_irq_info_failure(void) "VFIO_DEVICE_GET_IRQ_INFO failure: %m" +vfio_populate_device_get_irq_info_failure(int err) "VFIO_DEVICE_GET_IRQ_INFO failure: %d"
vfio_populate_device_get_irq_info_failur(const char *err)"VFIO_DEVICE_GET_IRQ_INFOfailure: %s"...if we are trying to match the original intent.
Quoted 8 lines
hw/vfio/trace-events +0
-0vfio_realize(const char *name, int group_id) " (%s) group %d" vfio_mdev(const char *name, bool is_mdev) " (%s) is_mdev %d" vfio_add_ext_cap_dropped(const char *name, uint16_t cap, uint16_t offset) "%s 0x%x@0x%x"scripts/tracetool/__init__.py +0
-0scripts/tracetool/__init__.py class Event(object): · line 279
Quoted 6 lines
scripts/tracetool/__init__.py +3
-0@@ -274,6 +274,10 @@ class Event(object): props = groups["props"].split() fmt = groups["fmt"] fmt_trans = groups["fmt_trans"] + if fmt.find("%m") != -1 or fmt_trans.find("%m") != -1: + raise ValueError("Event format '%m' is forbidden, pass the error " + "as an explicit trace argument")Whether or not checkpatch decides to flag %m as suspicious in the overall code base, I like that you are forbidding it in trace files.
Re: [Qemu-devel] [PATCH v2 1/4] display: ensure qxl log_buf is a nul terminated string
In reply to Daniel P. Berrangéhw/display/qxl.c async_common: · line 1768
Quoted 5 lines
case
QXL_IO_LOG:hw/display/qxl.c +2
-1-trace_qxl_io_log (d->id, d->ram->log_buf); + d->ram->log_buf[sizeof (d->ram->log_buf) -1 ] ='\0' ; +trace_qxl_io_log (d->id, (const char *)d->ram->log_buf);if (d->guestdebug) {
Re: [Qemu-devel] [PATCH v2 1/4] display: ensure qxl log_buf is a nul terminated string
In reply to Daniel P. Berrangéhw/display/qxl.c async_common: · line 1767
Quoted 11 lines · Daniel P. Berrangé
hw/display/qxl.c +2
-1@@ -1763,7 +1763,8 @@ async_common:qxl_set_mode (d, val,0 );break ;case QXL_IO_LOG : -trace_qxl_io_log (d->id, d->ram->log_buf); + d->ram->log_buf[sizeof (d->ram->log_buf) -1 ] ='\0' ; +trace_qxl_io_log (d->id, (const char *)d->ram->log_buf);
This is a PCI BAR shared with the guest? Then NUL termination is subject to races with vcpu threads that modify log_buf[] while we access it.
The safe way to do this is to copy in log_buf[] and then NUL-terminate the local copy.
Re: [Qemu-devel] [PATCH v2 2/4] trace: enforce that every trace-events file has a final newline
In reply to Daniel P. BerrangéQuoted 11 lines · Daniel P. Berrangé
When generating the trace-events-all file, the build system simply concatenates all the individual trace-events files. If any one of those files does not have a final newline, the printf format string will have the contents of the first line of the next file appended to it, which is usually a '#' comment.
Signed-off-by Daniel P. Berrangé <[email protected]>--- hw/gpio/trace-events | 2 +- scripts/tracetool/__init__.py | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-)
Re: [Qemu-devel] [PATCH v2 4/4] trace: add ability to do simple printf logging via systemtap
In reply to Eric BlakeQuoted 10 lines · Eric Blake
scripts/tracetool/format/log_stap.py line 34
Quoted 6 lines · Daniel P. Berrangé
scripts/tracetool/format/log_stap.py new file +6
-0+def c_macro_to_format(macro): + if macro.startswith("PRI"): + return macro[3] + + if macro == "TARGET_FMT_plx": + return "%016x"Do we really want to permit
TARGET_FMTin our trace-events files, or should that be fixed in hw/tpm/trace-events? See commit73ff0610.
TARGET_FMT_plx is constant across all QEMU targets. I think supporting it is nice since it makes writing trace events more natural/convenient.
Stefan
Re: [Qemu-devel] [PATCH v2 3/4] trace: forbid use of %m in trace event format strings
In reply to Daniel P. BerrangéQuoted 10 lines · Daniel P. Berrangé
Quoted 9 lines
hw/vfio/pci.c static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) · line 2584
Quoted 7 lines
+++
b/hw/vfio/pci.chw/vfio/pci.c +1
-1@@ -2581,7 +2581,7 @@ static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) ret =ioctl (vdev->vbasedev.fd,VFIO_DEVICE_GET_IRQ_INFO , &irq_info);if (ret) {/* This can fail for an old kernel or legacy PCI dev */ -trace_vfio_populate_device_get_irq_info_failure (); +trace_vfio_populate_device_get_irq_info_failure (errno);trace_vfio_populate_device_get_irq_info_failure(strerror(errno))
Quoted 3 lines
The caveat is that 'strerror' is not required to be thread safe, however, given that this is Linux only code I guess we can assume the glibc impl which fortunately is thread safe.
If we are going to worry about thread-safety of strerror(), we have a LOT of code to scrub (we are using it rather liberally throughout the code base).
Quoted 4 lines
On this point though, does anyone know of any platforms we support[1], or are likely to support in future, where 'strerror' is *not* thread safe ?
I'm not coming up with one, and I think the problem is independent of this series (if we DO have a problem, it's a series all its own to eradicate the use of strerror() in favor of something safer, either picking strerror_l() or dealing with the glibc vs. BSD differences in strerror_r()).
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [Qemu-devel] [PATCH v2 3/4] trace: forbid use of %m in trace event format strings
Quoted 12 lines · Eric Blake
Quoted 10 lines · Daniel P. Berrangé
Quoted 9 lines
hw/vfio/pci.c static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) · line 2584
Quoted 7 lines
+++
b/hw/vfio/pci.chw/vfio/pci.c +1
-1@@ -2581,7 +2581,7 @@ static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) ret =ioctl (vdev->vbasedev.fd,VFIO_DEVICE_GET_IRQ_INFO , &irq_info);if (ret) {/* This can fail for an old kernel or legacy PCI dev */ -trace_vfio_populate_device_get_irq_info_failure (); +trace_vfio_populate_device_get_irq_info_failure (errno);trace_vfio_populate_device_get_irq_info_failure(strerror(errno))
Quoted 7 lines
Quoted 3 lines
The caveat is that 'strerror' is not required to be thread safe, however, given that this is Linux only code I guess we can assume the glibc impl which fortunately is thread safe.
If we are going to worry about thread-safety of
strerror(), we have a LOT of code to scrub (we are using it rather liberally throughout the code base).
Yes, indeed we do and it was something that always worried me a little (as well as a few other non-reentrant APIs we used previously).
Quoted 9 lines
Quoted 3 lines
On this point though, does anyone know of any platforms we support[1], or are likely to support in future, where 'strerror' is *not* thread safe ?
I'm not coming up with one, and I think the problem is independent of this series (if we DO have a problem, it's a series all its own to eradicate the use of
strerror()in favor of something safer, either pickingstrerror_l()or dealing with the glibc vs. BSD differences instrerror_r()).
Agree that its not really something for this series - this just made me think of it again.
We went through the scrubbing in libvirt to use the sane, but still tedious to call, variant of strerror_r() many years ago. With luck though it is a worry that can be confined the dustbin of ancient UNIX history....unless someone can point to evidence to the contrary ?
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [Qemu-devel] [PATCH v2 3/4] trace: forbid use of %m in trace event format strings
Quoted 5 lines · Daniel P. Berrangé
Quoted 5 lines
Quoted 4 lines
On this point though, does anyone know of any platforms we support[1], or are likely to support in future, where 'strerror' is *not* thread safe ?
Quoted 8 lines
Quoted 5 lines
I'm not coming up with one, and I think the problem is independent of this series (if we DO have a problem, it's a series all its own to eradicate the use of
strerror()in favor of something safer, either pickingstrerror_l()or dealing with the glibc vs. BSD differences instrerror_r()).Agree that its not really something for this series - this just made me think of it again.
Shoot - FreeBSD strerror() is not threadsafe:
https://github.com/freebsd/freebsd/blob/master/lib/libc/string/strerror.c#L119
char *
strerror(int num)
{
static char ebuf[NL_TEXTMAX];
if (strerror_r(num, ebuf, sizeof(ebuf)) != 0)
errno = EINVAL;
return (ebuf);
}Quoted 5 lines
We went through the scrubbing in libvirt to use the sane, but still tedious to call, variant of
strerror_r()many years ago. With luck though it is a worry that can be confined the dustbin of ancient UNIX history....unless someone can point to evidence to the contrary ?
libvirt has it easy - they let gnulib do all the work of futzing around with getting a working strerror() despite platform bugs and despite glibc's insistence on a non-POSIX signature if _GNU_SOURCE is defined. We'll have to do a bit more legwork.
That said, I've added it to:
https://wiki.qemu.org/Contribute/BiteSizedTasks#Error_checking
if someone wants to do the grunt work.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [Qemu-devel] [PATCH v2 1/4] display: ensure qxl log_buf is a nul terminated string
In reply to Stefan HajnocziQuoted 16 lines · Stefan Hajnoczi
hw/display/qxl.c async_common: · line 1767
Quoted 11 lines · Daniel P. Berrangé
hw/display/qxl.c +2
-1@@ -1763,7 +1763,8 @@ async_common:qxl_set_mode (d, val,0 );break ;case QXL_IO_LOG : -trace_qxl_io_log (d->id, d->ram->log_buf); + d->ram->log_buf[sizeof (d->ram->log_buf) -1 ] ='\0' ; +trace_qxl_io_log (d->id, (const char *)d->ram->log_buf);This is a PCI BAR shared with the guest? Then NUL termination is subject to races with vcpu threads that modify log_buf[] while we access it.
Doh, yes, it is racy.
Quoted 2 lines
The safe way to do this is to copy in log_buf[] and then NUL-terminate the local copy.
log_buf is 4k in size, which I don't really want to allocate on the stack. Using malloc would impose a perf penalty on logging but not sure if that's significant enough to worry about. Alternatively we could just drop the tracepoint, given that you can already use the 'guestdebug' option to get these fprintf'd to stderr.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [Qemu-devel] [PATCH v2 2/4] trace: enforce that every trace-events file has a final newline
In reply to Eric BlakeQuoted 14 lines · Eric Blake
Quoted 6 lines · Daniel P. Berrangé
When generating the trace-events-all file, the build system simply concatenates all the individual trace-events files. If any one of those files does not have a final newline, the printf format string will have the contents of the first line of the next file appended to it, which is usually a '#' comment.
Quoted 6 lines
Signed-off-by Daniel P. Berrangé <[email protected]>--- hw/gpio/trace-events | 2 +- scripts/tracetool/__init__.py | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-)
Quoted 5 lines
Reviewed-by Eric Blake <[email protected]>Does checkpatch and/or patchew flag what are intended to be text files but which lack a trailing newline? If not, how hard would it be to get them to do so?
it probably should check for missing newlines at end of file, but I'm going to let someone else tackle that due to time pressures on what was supposed to be a quick enhancement :-)
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [Qemu-devel] [PATCH v2 4/4] trace: add ability to do simple printf logging via systemtap
In reply to Eric BlakeQuoted 14 lines · Eric Blake
Quoted 4 lines · Daniel P. Berrangé
The dtrace systemtap trace backend for QEMU is very powerful but it is also somewhat unfriendly to users who aren't familiar with systemtap, or who don't need its power right now.
Quoted 8 lines
stap -e "....some strange script...."
We go one step further though and introduce a 'qemu-trace-stap' tool to make this even easier
$ qemu-trace-stap run qemu-system-x86_64 qio* 22806@1547735341399856820 qio_channel_socket_new Socket new ioc=0x56135d1d7c00
Quoted 8 lines
One thing I did not immediately get on the first read, and therefore which may be worth mentioning in the docs portion of the patch:
Am I correct that stap is always run as a separate process from the process(es) being traced, and that you can start the two in either order (qemu first, then stap; or stap first, then qemu), where the tracing is effectively output as long as both processes are running and trace points being hit in the qemu process?
If you're using the stap binary directly you have a choice of it working against any existing process on the system, or you can have it launch the process on your behalf and target just that one process.
For our wrapper script I'm only targetting the former, since I think the common case is that someone/something else has already run the QEMU process. I could however easily add a "-p PID" option too which would allow the trace to be tailored to a specific QEMU process, instead of all running QEMU processes.
And yes, there's no ordering dependancy - unless it is critical that you see traces from QEMU's early initialization of course, in which case you'd launch the tracing first.
Quoted 4 lines
Quoted 4 lines
5 files changed, 339 insertions(+), 1 deletion(-) create mode 100755 scripts/qemu-trace-stap create mode 100644 scripts/tracetool/format/log_stap.py
Quoted 15 lines
Makefile endif · line 741
Quoted 11 lines
Makefile +3
-0@@ -736,6 +736,9 @@ endif ifneq ($(HELPERS-y),) $(call install-prog,$(HELPERS-y),$(DESTDIR)$(libexecdir)) endif +ifdef CONFIG_TRACE_SYSTEMTAP + $(INSTALL_PROG) "scripts/qemu-trace-stap" $(DESTDIR)$(bindir) +endifNo man page being installed? If we're planning on this binary being available on a similar par with qemu-io, it deserves some installed documentation.
Heh, no good deed / quick enhancement goes unpunished :-P Yes, I'll do a proper job of this and write a man page too :-)
Quoted 11 lines
Makefile endif
Quoted 7 lines
Makefile +7
-0+To watch all trace points on the qemu-system-x86_64 binary: + + %(argv0)s run qemu-system-x86_64 + +To only watch the trace points matching the qio* and qcrypto* patterns + + %(argv0)s run qemu-system-x86_64 qio* qcrypto*This output is not copy-pastable - it includes shell globs which might inadvertantly expand based on the contents of $PWD. Should you amend the output to be properly quoted for reuse in shell commands?
Yes, that is true
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [Qemu-devel] [PATCH v2 4/4] trace: add ability to do simple printf logging via systemtap
In reply to Stefan HajnocziQuoted 12 lines · Stefan Hajnoczi
Quoted 10 lines · Eric Blake
scripts/tracetool/format/log_stap.py line 34
Quoted 6 lines · Daniel P. Berrangé
scripts/tracetool/format/log_stap.py new file +6
-0+def c_macro_to_format(macro): + if macro.startswith("PRI"): + return macro[3] + + if macro == "TARGET_FMT_plx": + return "%016x"Do we really want to permit
TARGET_FMTin our trace-events files, or should that be fixed in hw/tpm/trace-events? See commit73ff0610.
Quoted 2 lines
TARGET_FMT_plxis constant across all QEMU targets. I think supporting it is nice since it makes writing trace events more natural/convenient.
Yep, I'm fine with supporting custom macros beyond PRI* in trace-events, provided that they can be subsituted out with a simple formats code for stap.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [Qemu-devel] [PATCH v2 3/4] trace: forbid use of %m in trace event format strings
In reply to Eric BlakeQuoted 37 lines · Eric Blake
Quoted 2 lines · Daniel P. Berrangé
The '%m' format specifier instructs glibc's
printf()implementation to insert the contents of strerror(errno).That is a glibc-only extension in
printf(), but mandatory (and supported in ALL platforms) insyslog(). However, you are correct that:Quoted 6 lines
This is not something that
should ever be used in trace-events files because several of the backends do not use the format string and so this error information is invisible to them. The errno value should be given as an explicit trace argument instead. Use of '%m' should also be avoided as it is not portable to all QEMU build targets.If all of our traces are compiled to
syslog(), it would be portable, but that's not the case.What's more, using %m is subject to race conditions - the more code you have between when the format string containing %m is given, and the point where the actual error message is emitted, the higher the probability that some of the intermediate code could have stomped on errno in the meantime, rendering the logged message incorrect. And forcing logging wrappers to save/restore the current state of errno, just in case they might encounter %m, can get wasteful.
Should checkpatch be taught to flag %m as suspicious?
However, tracing the errno value is NOT what %m did; I'd rather see...
Quoted 8 lines
Signed-off-by Daniel P. Berrangé <[email protected]>--- hw/vfio/pci.c | 2 +- hw/vfio/trace-events | 2 +- scripts/tracetool/__init__.py | 4 ++++ 3 files changed, 6 insertions(+), 2 deletions(-)
Quoted 12 lines
hw/vfio/pci.c static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) · line 2584
Quoted 10 lines
hw/vfio/pci.c +1
-1@@ -2581,7 +2581,7 @@ static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) ret =ioctl (vdev->vbasedev.fd,VFIO_DEVICE_GET_IRQ_INFO , &irq_info);if (ret) {/* This can fail for an old kernel or legacy PCI dev */ -trace_vfio_populate_device_get_irq_info_failure (); +trace_vfio_populate_device_get_irq_info_failure (errno);trace_vfio_populate_device_get_irq_info_failure(strerror(errno))
The caveat is that 'strerror' is not required to be thread safe, however, given that this is Linux only code I guess we can assume the glibc impl which fortunately is thread safe.
On this point though, does anyone know of any platforms we support[1], or are likely to support in future, where 'strerror' is *not* thread safe ?
I see rumours that Windows strerror() is not thread safe but not found a concrete source for that.
[1] https://qemu.weilnetz.de/doc/qemu-doc.html#Supported-build-platforms
Quoted 6 lines
Quoted 6 lines
scripts/tracetool/__init__.py +0
-0@@ -274,6 +274,10 @@ class Event(object): props = groups["props"].split()
Quoted 8 lines
scripts/tracetool/__init__.py class Event(object): · line 279
Quoted 5 lines
scripts/tracetool/__init__.py +3
-0fmt = groups["fmt"] fmt_trans = groups["fmt_trans"] + if fmt.find("%m") != -1 or fmt_trans.find("%m") != -1: + raise ValueError("Event format '%m' is forbidden, pass the error " + "as an explicit trace argument")Whether or not checkpatch decides to flag %m as suspicious in the overall code base, I like that you are forbidding it in trace files.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [Qemu-devel] [PATCH v2 3/4] trace: forbid use of %m in trace event format strings
In reply to Eric BlakeQuoted 40 lines · Eric Blake
Quoted 5 lines · Daniel P. Berrangé
Quoted 5 lines
Quoted 4 lines
On this point though, does anyone know of any platforms we support[1], or are likely to support in future, where 'strerror' is *not* thread safe ?
Quoted 8 lines
Quoted 5 lines
I'm not coming up with one, and I think the problem is independent of this series (if we DO have a problem, it's a series all its own to eradicate the use of
strerror()in favor of something safer, either pickingstrerror_l()or dealing with the glibc vs. BSD differences instrerror_r()).Agree that its not really something for this series - this just made me think of it again.
Shoot - FreeBSD
strerror()is not threadsafe:
https://github.com/freebsd/freebsd/blob/master/lib/libc/string/strerror.c#L119char *
strerror(int num){ static char ebuf[NL_TEXTMAX]; if (strerror_r(num, ebuf, sizeof(ebuf)) != 0) errno = EINVAL; return (ebuf); }Quoted 5 lines
We went through the scrubbing in libvirt to use the sane, but still tedious to call, variant of
strerror_r()many years ago. With luck though it is a worry that can be confined the dustbin of ancient UNIX history....unless someone can point to evidence to the contrary ?libvirt has it easy - they let gnulib do all the work of futzing around with getting a working
strerror()despite platform bugs and despite glibc's insistence on a non-POSIX signature if_GNU_SOURCEis defined. We'll have to do a bit more legwork.
Quoted 4 lines
That said, I've added it to:
https://wiki.qemu.org/Contribute/BiteSizedTasks#Error_checkingif someone wants to do the grunt work.
I think we should change that task to switch to g_strerror() from glib instead ... as far as I can see, this is a proper wrapper around strerror_r(), so we don't have to deal with the implementation oddities of strerror_r() in QEMU.
Thomas
Re: [Qemu-devel] [PATCH v2 3/4] trace: forbid use of %m in trace event format strings
Quoted 44 lines · Thomas Huth
Quoted 43 lines · Eric Blake
Quoted 5 lines · Daniel P. Berrangé
Quoted 5 lines
Quoted 4 lines
On this point though, does anyone know of any platforms we support[1], or are likely to support in future, where 'strerror' is *not* thread safe ?
Quoted 8 lines
Quoted 5 lines
I'm not coming up with one, and I think the problem is independent of this series (if we DO have a problem, it's a series all its own to eradicate the use of
strerror()in favor of something safer, either pickingstrerror_l()or dealing with the glibc vs. BSD differences instrerror_r()).Agree that its not really something for this series - this just made me think of it again.
Shoot - FreeBSD
strerror()is not threadsafe:
https://github.com/freebsd/freebsd/blob/master/lib/libc/string/strerror.c#L119char *
strerror(int num){ static char ebuf[NL_TEXTMAX]; if (strerror_r(num, ebuf, sizeof(ebuf)) != 0) errno = EINVAL; return (ebuf); }Quoted 5 lines
We went through the scrubbing in libvirt to use the sane, but still tedious to call, variant of
strerror_r()many years ago. With luck though it is a worry that can be confined the dustbin of ancient UNIX history....unless someone can point to evidence to the contrary ?libvirt has it easy - they let gnulib do all the work of futzing around with getting a working
strerror()despite platform bugs and despite glibc's insistence on a non-POSIX signature if_GNU_SOURCEis defined. We'll have to do a bit more legwork.That said, I've added it to:
https://wiki.qemu.org/Contribute/BiteSizedTasks#Error_checking
Quoted 6 lines
Quoted 1 line
if someone wants to do the grunt work.
I think we should change that task to switch to
g_strerror()from glib instead ... as far as I can see, this is a proper wrapper aroundstrerror_r(), so we don't have to deal with the implementation oddities ofstrerror_r()in QEMU.
Yeah, I think using g_strerror() makes sense. We've just adopted that in libvirt precisely to avoid these platform portability oddities and the really unpleasant API calling convention of strerror_r().
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|