Add VM kernel tracing over vsockets and FIFOs
v5 cover Add VM kernel tracing over vsockets and FIFOs
This patchset adds support for tracing guest kernels to trace-cmd.
Changes in v5:
- add FIFOs transport for tracing data
- fixed a bug in tracecmd_msg_wait_close
Changes in v4:
- detect and use splice(2) on vsock sockets if possible
- switch port numbers to unsigned int
- trace-cmd record --date flag is now set for all guests if provided by the user
- removed grow_cap and exponential buffer size growth
Changes in v3:
- addressed Steven's feedback
- detect and disable guest tracing if <linux/vm_sockets.h> is not available
- the --date flag is now treated as global for all guest instances
- fixed a bug that caused --date to be ignored for host tracing data when tracing guests
Changes in v2:
- rebased on top of protocol V3
- fixed system clock timestamps with the --date flag
Slavomir Kaslev (10):
trace-cmd: Add tracecmd_create_recorder_virt function
trace-cmd: Add TRACE_REQ and TRACE_RESP messages
trace-cmd: Add buffer instance flags for tracing in guest and agent
context
trace-cmd: Add VM kernel tracing over vsockets transport
trace-cmd: Use splice(2) for vsockets if available
trace-cmd: Add trace-cmd setup-guest command
trace-cmd: Try to autodetect number of guest CPUs in setup-guest if
not specified
trace-cmd: Make setup-guest auto attach FIFOs to the guest VM config
trace-cmd: Set both input and output to non-blocking when recording is
stopped
trace-cmd: Add VM tracing over FIFOs transport
Steven Rostedt (VMware) (1):
trace-cmd: Detect if vsockets are available
Makefile | 7 + include/trace-cmd/trace-cmd.h | 17 +- lib/trace-cmd/trace-recorder.c | 69 ++- tracecmd/Makefile | 7 +- tracecmd/include/trace-local.h | 26 ++ tracecmd/trace-agent.c | 257 +++++++++++ tracecmd/trace-cmd.c | 4 + tracecmd/trace-msg.c | 222 +++++++++- tracecmd/trace-record.c | 759 +++++++++++++++++++++++++++++++-- tracecmd/trace-setup-guest.c | 243 +++++++++++ tracecmd/trace-usage.c | 21 +- 11 files changed, 1557 insertions(+), 75 deletions(-) create mode 100644 tracecmd/trace-agent.c create mode 100644 tracecmd/trace-setup-guest.c
--
2.19.1
RFC PATCH v5 1/11 trace-cmd: Detect if vsockets are available
From: "Steven Rostedt (VMware)" <[email protected]>
Detect and define VSOCK if vsockets are available on the system. This macro is used to disable VM remote tracing features on older kernels.
--- Makefile | 7 +++++++ 1 file changed, 7 insertions(+)
Makefile +7 -0
@@ -204,6 +204,13 @@ CFLAGS ?= -g -Wall CPPFLAGS ?= LDFLAGS ?= +VSOCK_DEFINED := $(shell if (echo "\#include <linux/vm_sockets.h>" | $(CC) -E - >/dev/null 2>&1) ; then echo 1; else echo 0 ; fi) + +export VSOCK_DEFINED +ifeq ($(VSOCK_DEFINED), 1) +CFLAGS += -DVSOCK +endif + export CFLAGS export INCLUDES
Signature · 1 line
2.19.1
RFC PATCH v5 2/11 trace-cmd: Add tracecmd_create_recorder_virt function
In reply to Slavomir KaslevAdd tracecmd_create_recorder_virt function that will be used for tracing VM guests.
--- include/trace-cmd/trace-cmd.h | 1 + lib/trace-cmd/trace-recorder.c | 53 ++++++++++++++++++++++++---------- 2 files changed, 39 insertions(+), 15 deletions(-)
include/trace-cmd/trace-cmd.h +1 -0
@@ -277,6 +277,7 @@ enum {void tracecmd_free_recorder (struct tracecmd_recorder *recorder);struct tracecmd_recorder *tracecmd_create_recorder (const char *file,int cpu,unsigned flags);struct tracecmd_recorder *tracecmd_create_recorder_fd (int fd,int cpu,unsigned flags); +struct tracecmd_recorder *tracecmd_create_recorder_virt (const char *file,int cpu,unsigned flags,int trace_fd);struct tracecmd_recorder *tracecmd_create_recorder_maxkb (const char *file,int cpu,unsigned flags,int maxkb);struct tracecmd_recorder *tracecmd_create_buffer_recorder_fd (int fd,int cpu,unsigned flags,const char *buffer);struct tracecmd_recorder *tracecmd_create_buffer_recorder (const char *file,int cpu,unsigned flags,const char *buffer);
lib/trace-cmd/trace-recorder.c +38 -15
81 lines not on this page · raw message
Signature · 1 line
2.19.1
RFC PATCH v5 3/11 trace-cmd: Add TRACE_REQ and TRACE_RESP messages
In reply to Slavomir KaslevAdd TRACE_REQ and TRACE_RESP messages which are used for initiating guest VM tracing.
--- include/trace-cmd/trace-cmd.h | 16 ++- tracecmd/trace-msg.c | 213 +++++++++++++++++++++++++++++++++- 2 files changed, 224 insertions(+), 5 deletions(-)
include/trace-cmd/trace-cmd.h +15 -1
@@ -326,7 +326,7 @@ int tracecmd_msg_send_init_data(struct tracecmd_msg_handle *msg_handle,int tracecmd_msg_data_send (struct tracecmd_msg_handle *msg_handle,const char *buf,int size);int tracecmd_msg_finish_sending_data (struct tracecmd_msg_handle *msg_handle); -void tracecmd_msg_send_close_msg (struct tracecmd_msg_handle *msg_handle); +int tracecmd_msg_send_close_msg (struct tracecmd_msg_handle *msg_handle);/* for server */ int tracecmd_msg_initial_setting (struct tracecmd_msg_handle *msg_handle);
@@ -337,6 +337,20 @@ int tracecmd_msg_collect_data(struct tracecmd_msg_handle *msg_handle, int ofd);bool tracecmd_msg_done (struct tracecmd_msg_handle *msg_handle);void tracecmd_msg_set_done (struct tracecmd_msg_handle *msg_handle); +int tracecmd_msg_send_trace_req (struct tracecmd_msg_handle *msg_handle, +int argc,char **argv); +int tracecmd_msg_recv_trace_req (struct tracecmd_msg_handle *msg_handle, +int *argc,char ***argv); + +int tracecmd_msg_send_trace_resp (struct tracecmd_msg_handle *msg_handle, +int nr_cpus,int page_size, +unsigned int *ports); +int tracecmd_msg_recv_trace_resp (struct tracecmd_msg_handle *msg_handle, +int *nr_cpus,int *page_size, +unsigned int **ports); + +int tracecmd_msg_wait_close (struct tracecmd_msg_handle *msg_handle); +/* --- Plugin handling --- */ extern struct tep_plugin_option trace_ftrace_options[];
tracecmd/trace-msg.c +209 -4
256 lines not on this page · raw message
Signature · 1 line
2.19.1
RFC PATCH v5 4/11 trace-cmd: Add buffer instance flags for tracing in guest and agent context
In reply to Slavomir KaslevAdd BUFFER_FL_GUEST and BUFFER_FL_AGENT flags to differentiate when trace-record.c is being called to trace guest or the VM tracing agent.
Also disable functions talking to the local tracefs when called in recording guest instances context.
--- tracecmd/include/trace-local.h | 2 ++ tracecmd/trace-record.c | 55 ++++++++++++++++++++++++++++++++-- 2 files changed, 55 insertions(+), 2 deletions(-)
tracecmd/include/trace-local.h +2 -0
@@ -149,6 +149,8 @@ char *strstrip(char *str);enum buffer_instance_flags {BUFFER_FL_KEEP =1 <<0 ,BUFFER_FL_PROFILE =1 <<1 , +BUFFER_FL_GUEST =1 <<2 , +BUFFER_FL_AGENT =1 <<3 , };struct func_list {
tracecmd/trace-record.c +53 -2
169 lines not on this page · raw message
Signature · 1 line
2.19.1
RFC PATCH v5 6/11 trace-cmd: Use splice(2) for vsockets if available
In reply to Slavomir KaslevDetect if splice(2) reading is supported for vsockets (Linux 4.20 and later) and use it, or fallback to read/write-ing otherwise.
--- lib/trace-cmd/trace-recorder.c | 6 ++-- tracecmd/trace-record.c | 56 ++++++++++++++++++++++++++++++++-- 2 files changed, 56 insertions(+), 6 deletions(-)
lib/trace-cmd/trace-recorder.c +2 -4
@@ -380,10 +380,8 @@ static long splice_data(struct tracecmd_recorder *recorder) ret =splice (recorder->brass[0 ],NULL , recorder->fd,NULL , read, recorder->fd_flags);if (ret <0 ) { -if (errno !=EAGAIN && errno !=EINTR ) { -warning ("recorder error in splice output" ); -return -1 ; - } +if (errno !=EAGAIN && errno !=EINTR ) +return ret;return total_read; }else update_fd (recorder, ret);
tracecmd/trace-record.c +54 -2
79 lines not on this page · raw message
Signature · 1 line
2.19.1
RFC PATCH v5 5/11 trace-cmd: Add VM kernel tracing over vsockets transport
In reply to Slavomir KaslevThis patch adds VM tracing over vsockets. The new trace-cmd agent command needs run on each guest we want to trace:
you@guest2 # trace-cmd agent
Then trace-cmd record on the host can collect data from both the host and several guests simultaneously:
you@host $ trace-cmd record -A guest1 -e irq -e sched \
-A guest2 -p function -e all
--- tracecmd/Makefile | 6 +- tracecmd/include/trace-local.h | 16 + tracecmd/trace-agent.c | 225 +++++++++++++ tracecmd/trace-cmd.c | 3 + tracecmd/trace-msg.c | 1 + tracecmd/trace-record.c | 594 ++++++++++++++++++++++++++++++--- tracecmd/trace-usage.c | 13 +- 7 files changed, 811 insertions(+), 47 deletions(-) create mode 100644 tracecmd/trace-agent.c
tracecmd/Makefile +5 -1
@@ -33,13 +33,17 @@ TRACE_CMD_OBJS += trace-output.o TRACE_CMD_OBJS += trace-usage.o TRACE_CMD_OBJS += trace-msg.o +ifeq ($(VSOCK_DEFINED), 1) +TRACE_CMD_OBJS += trace-agent.o +endif + ALL_OBJS := $(TRACE_CMD_OBJS:%.o=$(bdir)/%.o) all_objs := $(sort $(ALL_OBJS)) all_deps := $(all_objs:$(bdir)/%.o=$(bdir)/.%.d) CONFIG_INCLUDES = -CONFIG_LIBS = +CONFIG_LIBS = -lrt CONFIG_FLAGS = all: $(TARGETS)
tracecmd/include/trace-local.h +16 -0
@@ -12,6 +12,8 @@#include "trace-cmd.h" #include "event-utils.h" +#define TRACE_AGENT_DEFAULT_PORT 823 +extern int debug;extern int quiet;
@@ -64,6 +66,8 @@ void trace_split(int argc, char **argv);void trace_listen (int argc,char **argv); +void trace_agent (int argc,char **argv); +void trace_restore (int argc,char **argv);void trace_clear (int argc,char **argv);
@@ -88,6 +92,10 @@ void trace_list(int argc, char **argv);void trace_usage (int argc,char **argv); +int trace_record_agent (struct tracecmd_msg_handle *msg_handle, +int cpus,int *fds, +int argc,char **argv); +struct hook_list ; void trace_init_profile(struct tracecmd_input *handle, struct hook_list *hooks,
@@ -176,6 +184,7 @@ struct buffer_instance {struct func_list *notrace_funcs;const char *clock; +unsigned int *client_ports;struct trace_seq *s_save;struct trace_seq *s_print;
@@ -190,6 +199,13 @@ struct buffer_instance {int tracing_on_fd;int buffer_size;int cpu_count; + +int argc; +char **argv; + +int cid; +int port; +int *fds; };extern struct buffer_instance top_instance;
tracecmd/trace-agent.c new file +225 -0
225 lines not on this page · raw message
tracecmd/trace-cmd.c +3 -0
@@ -83,6 +83,9 @@ struct command commands[] = { {"hist", trace_hist}, {"mem", trace_mem}, {"listen", trace_listen}, +#ifdef VSOCK + {"agent", trace_agent}, +#endif {"split", trace_split}, {"restore", trace_restore}, {"stack", trace_stack},
tracecmd/trace-msg.c +1 -0
@@ -86,6 +86,7 @@ struct tracecmd_msg_trace_req { } __attribute__((packed));struct tracecmd_msg_trace_resp { +be32 flags;be32 cpus;be32 page_size; } __attribute__((packed));
tracecmd/trace-record.c +549 -45
840 lines not on this page · raw message
tracecmd/trace-usage.c +12 -1
@@ -231,11 +231,22 @@ static struct usage_help usage_help[] = {"listen on a network socket for trace clients" ," %s listen -p port[-D][-o file][-d dir][-l logfile]\n" " Creates a socket to listen for clients.\n" -" -D create it in daemon mode.\n" +" -p port number to listen on.\n" +" -D run in daemon mode.\n" " -o file name to use for clients.\n" " -d directory to store client files.\n" " -l logfile to write messages to.\n" }, +#ifdef VSOCK + { +"agent" , +"listen on a vsocket for trace clients" , +" %s agent -p port[-D]\n" +" Creates a vsocket to listen for clients.\n" +" -p port number to listen on.\n" +" -D run in daemon mode.\n" + }, +#endif {"list" , "list the available events, plugins or options",
Signature · 1 line
2.19.1
RFC PATCH v5 7/11 trace-cmd: Add `trace-cmd setup-guest` command
In reply to Slavomir KaslevAdd trace-cmd setup-guest command that creates the necessary FIFOs for tracing a guest over FIFOs instead of vsockets.
--- tracecmd/Makefile | 1 + tracecmd/include/trace-local.h | 6 ++ tracecmd/trace-cmd.c | 1 + tracecmd/trace-setup-guest.c | 181 +++++++++++++++++++++++++++++++++ tracecmd/trace-usage.c | 8 ++ 5 files changed, 197 insertions(+) create mode 100644 tracecmd/trace-setup-guest.c
tracecmd/Makefile +1 -0
@@ -35,6 +35,7 @@ TRACE_CMD_OBJS += trace-msg.o ifeq ($(VSOCK_DEFINED), 1) TRACE_CMD_OBJS += trace-agent.o +TRACE_CMD_OBJS += trace-setup-guest.o endif ALL_OBJS := $(TRACE_CMD_OBJS:%.o=$(bdir)/%.o)
tracecmd/include/trace-local.h +6 -0
@@ -14,6 +14,10 @@#define TRACE_AGENT_DEFAULT_PORT 823 +#define GUEST_PIPE_NAME "trace-pipe-cpu" +#define GUEST_DIR_FMT "/var/lib/trace-cmd/virt/%s" +#define GUEST_FIFO_FMT GUEST_DIR_FMT "/" GUEST_PIPE_NAME "%d" +extern int debug;extern int quiet;
@@ -68,6 +72,8 @@ void trace_listen(int argc, char **argv);void trace_agent (int argc,char **argv); +void trace_setup_guest (int argc,char **argv); +void trace_restore (int argc,char **argv);void trace_clear (int argc,char **argv);
tracecmd/trace-cmd.c +1 -0
@@ -85,6 +85,7 @@ struct command commands[] = { {"listen", trace_listen}, #ifdef VSOCK {"agent", trace_agent}, + {"setup-guest", trace_setup_guest}, #endif {"split", trace_split}, {"restore", trace_restore},
tracecmd/trace-setup-guest.c new file +181 -0
181 lines not on this page · raw message
tracecmd/trace-usage.c +8 -0
@@ -246,6 +246,14 @@ static struct usage_help usage_help[] = { " -p port number to listen on.\n" " -D run in daemon mode.\n" }, + { + "setup-guest", + "create FIFOs for tracing guest VMs", + " %s setup-guest -c cpus[-p perm][-g group] guest\n" + " -c number of guest virtual CPUs\n" + " -p FIFOs permissions (default: 0660)\n" + " -g FIFOs group owner\n" + }, #endif { "list",
Signature · 1 line
2.19.1
RFC PATCH v5 8/11 trace-cmd: Try to autodetect number of guest CPUs in setup-guest if not specified
In reply to Slavomir KaslevWhen no number of guest CPUs is provided with the -c flag to trace-cmd setup-guest, try to autodetect it using virsh for libvirt managed guests.
--- tracecmd/trace-setup-guest.c | 21 +++++++++++++++++++++ tracecmd/trace-usage.c | 2 +- 2 files changed, 22 insertions(+), 1 deletion(-)
tracecmd/trace-setup-guest.c +21 -0
@@ -100,6 +100,24 @@ static int make_guest_fifos(const char *guest, int nr_cpus, mode_t mode)return ret; } +static int get_guest_cpu_count (const char *guest) +{ +const char *cmd_fmt ="virsh vcpucount --maximum '%s' 2>/dev/null" ; +int nr_cpus =-1 ; +char cmd[1024 ]; +FILE *f; + +snprintf (cmd,sizeof (cmd), cmd_fmt, guest); + f =popen (cmd,"r" ); +if (!f) +return -1 ; + +fscanf (f,"%d" , &nr_cpus); +pclose (f); + +return nr_cpus; +} +static void do_setup_guest (const char *guest,int nr_cpus,mode_t mode,gid_t gid) {gid_t save_egid;
@@ -174,6 +192,9 @@ void trace_setup_guest(int argc, char **argv) guest = argv[optind+1 ]; +if (nr_cpus <=0 ) + nr_cpus =get_guest_cpu_count (guest); +if (nr_cpus <=0 )pdie ("invalid number of cpus for guest %s" , guest);
tracecmd/trace-usage.c +1 -1
@@ -249,7 +249,7 @@ static struct usage_help usage_help[] = { {"setup-guest" ,"create FIFOs for tracing guest VMs" , -" %s setup-guest -c cpus[-p perm][-g group] guest\n" +" %s setup-guest [-c cpus][-p perm][-g group] guest\n" " -c number of guest virtual CPUs\n" " -p FIFOs permissions (default: 0660)\n" " -g FIFOs group owner\n"
Signature · 1 line
2.19.1
RFC PATCH v5 9/11 trace-cmd: Make setup-guest auto attach FIFOs to the guest VM config
In reply to Slavomir KaslevThis patch tries to attach the newly created FIFOs for guest tracing as virtio serial devices to libvirt managed guests.
--- tracecmd/trace-setup-guest.c | 41 ++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+)
tracecmd/trace-setup-guest.c +41 -0
@@ -118,6 +118,43 @@ static int get_guest_cpu_count(const char *guest)return nr_cpus; } +static int attach_guest_fifos (const char *guest,int nr_cpus) +{ +const char *cmd_fmt = +"virsh attach-device --config '%s' '%s' >/dev/null 2>/dev/null" ; +const char *xml_fmt = +"<channel type='pipe'>\n" +" <source path='%s'/>\n" +" <target type='virtio' name='%s%d'/>\n" +"</channel>" ; +char tmp_path[PATH_MAX ], path[PATH_MAX ]; +char cmd[PATH_MAX ], xml[PATH_MAX ]; +int i, fd, ret =0 ; + +strcpy (tmp_path,"/tmp/pipexmlXXXXXX" ); + fd =mkstemp (tmp_path); +if (fd <0 ) +return fd; + +for (i =0 ; i < nr_cpus; i++) { +snprintf (path,sizeof (path),GUEST_FIFO_FMT , guest, i); +snprintf (xml,sizeof (xml), xml_fmt, path,GUEST_PIPE_NAME , i); +pwrite (fd, xml,strlen (xml),0 ); + +snprintf (cmd,sizeof (cmd), cmd_fmt, guest, tmp_path); + errno =0 ; +if (system (cmd) !=0 ) { + ret =-1 ; +break ; + } + } + +close (fd); +unlink (tmp_path); + +return ret; +} +static void do_setup_guest (const char *guest,int nr_cpus,mode_t mode,gid_t gid) {gid_t save_egid;
@@ -136,6 +173,10 @@ static void do_setup_guest(const char *guest, int nr_cpus, mode_t mode, gid_t giif (ret <0 )pdie ("failed to create FIFOs for %s" , guest); + ret =attach_guest_fifos (guest, nr_cpus); +if (ret <0 ) +warning ("failed to attach FIFOs to %s" , guest); + ret =setegid (save_egid);if (ret <0 )pdie ("failed to restore effective group ID" );
Signature · 1 line
2.19.1
RFC PATCH v5 10/11 trace-cmd: Set both input and output to non-blocking when recording is stopped
In reply to Slavomir KaslevWhen tracing VMs over FIFOs both the agent and the host recording process might get blocked doing IO over a FIFO when the other side is gone. In this case we won't get EPIPE error (as in the vsockets case) since the FIFOs will always be opened by the hypervisor.
--- lib/trace-cmd/trace-recorder.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
lib/trace-cmd/trace-recorder.c +8 -2
@@ -40,6 +40,7 @@ struct tracecmd_recorder {int pages;int count;unsigned fd_flags; +unsigned trace_fd_flags;unsigned flags; };
@@ -121,6 +122,8 @@ tracecmd_create_buffer_recorder_fd2(int fd, int fd2, int cpu, unsigned flags,if (!(recorder->flags &TRACECMD_RECORD_BLOCK )) recorder->fd_flags |=2 ;/* and NON_BLOCK */ + recorder->trace_fd_flags =1 ;/* SPLICE_F_MOVE */ +/* Init to know what to free and release */ recorder->trace_fd =-1 ; recorder->brass[0 ] =-1 ;
@@ -366,7 +369,7 @@ static long splice_data(struct tracecmd_recorder *recorder)long ret; read =splice (recorder->trace_fd,NULL , recorder->brass[1 ],NULL , - recorder->pipe_size,1 /* SPLICE_F_MOVE */ ); + recorder->pipe_size, recorder->trace_fd_flags);if (read <0 ) {if (errno !=EAGAIN && errno !=EINTR ) {warning ("recorder error in splice input" );
@@ -434,9 +437,12 @@ static void set_nonblock(struct tracecmd_recorder *recorder)/* Do not block on reads for flushing */ flags =fcntl (recorder->trace_fd,F_GETFL );fcntl (recorder->trace_fd,F_SETFL , flags |O_NONBLOCK ); + recorder->trace_fd_flags |=2 ;/* SPLICE_F_NONBLOCK */ /* Do not block on streams for write */ - recorder->fd_flags |=2 ;/* NON_BLOCK */ + flags =fcntl (recorder->fd,F_GETFL ); +fcntl (recorder->fd,F_SETFL , flags |O_NONBLOCK ); + recorder->fd_flags |=2 ;/* SPLICE_F_NONBLOCK */ }long tracecmd_flush_recording (struct tracecmd_recorder *recorder)
Signature · 1 line
2.19.1
RFC PATCH v5 11/11 trace-cmd: Add VM tracing over FIFOs transport
In reply to Slavomir KaslevFIFOs offer ~3x the throughput of vsockets. This patch adds support for using FIFOs to stream tracing data back to the host when tracing VMs.
--- include/trace-cmd/trace-cmd.h | 8 ++-- tracecmd/include/trace-local.h | 4 +- tracecmd/trace-agent.c | 40 ++++++++++++++-- tracecmd/trace-msg.c | 26 +++++++---- tracecmd/trace-record.c | 84 ++++++++++++++++++++++++++++------ 5 files changed, 131 insertions(+), 31 deletions(-)
include/trace-cmd/trace-cmd.h +4 -4
@@ -338,16 +338,16 @@ bool tracecmd_msg_done(struct tracecmd_msg_handle *msg_handle);void tracecmd_msg_set_done (struct tracecmd_msg_handle *msg_handle);int tracecmd_msg_send_trace_req (struct tracecmd_msg_handle *msg_handle, -int argc,char **argv); +int argc,char **argv,bool use_fifos);int tracecmd_msg_recv_trace_req (struct tracecmd_msg_handle *msg_handle, -int *argc,char ***argv); +int *argc,char ***argv,bool *use_fifos);int tracecmd_msg_send_trace_resp (struct tracecmd_msg_handle *msg_handle,int nr_cpus,int page_size, -unsigned int *ports); +unsigned int *ports,bool use_fifos);int tracecmd_msg_recv_trace_resp (struct tracecmd_msg_handle *msg_handle,int *nr_cpus,int *page_size, -unsigned int **ports); +unsigned int **ports,bool *use_fifos);int tracecmd_msg_wait_close (struct tracecmd_msg_handle *msg_handle);
tracecmd/include/trace-local.h +3 -1
@@ -17,6 +17,7 @@#define GUEST_PIPE_NAME "trace-pipe-cpu"#define GUEST_DIR_FMT "/var/lib/trace-cmd/virt/%s"#define GUEST_FIFO_FMT GUEST_DIR_FMT "/" GUEST_PIPE_NAME "%d" +#define VIRTIO_FIFO_FMT "/dev/virtio-ports/" GUEST_PIPE_NAME "%d"extern int debug;extern int quiet;
@@ -100,7 +101,7 @@ void trace_usage(int argc, char **argv);int trace_record_agent (struct tracecmd_msg_handle *msg_handle,int cpus,int *fds, -int argc,char **argv); +int argc,char **argv,bool use_fifos);struct hook_list ;
@@ -212,6 +213,7 @@ struct buffer_instance {int cid;int port;int *fds; +bool use_fifos; };extern struct buffer_instance top_instance;
tracecmd/trace-agent.c +36 -4
@@ -83,12 +83,39 @@ static void make_vsocks(int nr, int *fds, unsigned int *ports) } } +static int open_agent_fifos (int nr_cpus,int *fds) +{ +char path[PATH_MAX ]; +int i, fd, ret =0 ; + +for (i =0 ; i < nr_cpus; i++) { +snprintf (path,sizeof (path),VIRTIO_FIFO_FMT , i); + fd =open (path,O_WRONLY ); +if (fd <0 ) { + ret =-1 ; +break ; + } + + fds[i] = fd; + } + +if (!ret) +return ret; + +/* We failed to open all FIFOs so clean up and return error */ +while (--i >=0 ) +close (fds[i]); + +return ret; +} +static void agent_handle (int sd,int nr_cpus,int page_size) {struct tracecmd_msg_handle *msg_handle;unsigned int *ports;char **argv =NULL ;int argc =0 ; +bool use_fifos;int *fds;int ret;
@@ -101,17 +128,22 @@ static void agent_handle(int sd, int nr_cpus, int page_size)if (!msg_handle)die ("Failed to allocate message handle" ); - ret =tracecmd_msg_recv_trace_req (msg_handle, &argc, &argv); + ret =tracecmd_msg_recv_trace_req (msg_handle, &argc, &argv, &use_fifos);if (ret <0 )die ("Failed to receive trace request" ); -make_vsocks (nr_cpus, fds, ports); +if (use_fifos &&open_agent_fifos (nr_cpus, fds)) + use_fifos = false; + +if (!use_fifos) +make_vsocks (nr_cpus, fds, ports); - ret =tracecmd_msg_send_trace_resp (msg_handle, nr_cpus, page_size, ports); + ret =tracecmd_msg_send_trace_resp (msg_handle, nr_cpus, page_size, + ports, use_fifos);if (ret <0 )die ("Failed to send trace response" ); -trace_record_agent (msg_handle, nr_cpus, fds, argc, argv); +trace_record_agent (msg_handle, nr_cpus, fds, argc, argv, use_fifos);free (argv[0 ]);free (argv);
tracecmd/trace-msg.c +17 -9
@@ -175,6 +175,10 @@ static int msg_write(int fd, struct tracecmd_msg *msg)return __do_write_check (fd, msg->buf, data_size); } +enum msg_trace_flags { +MSG_TRACE_USE_FIFOS =1 <<0 , +}; +enum msg_opt_command {MSGOPT_USETCP =1 , };
@@ -732,7 +736,7 @@ error:return ret; } -static int make_trace_req (struct tracecmd_msg *msg,int argc,char **argv) +static int make_trace_req (struct tracecmd_msg *msg,int argc,char **argv,bool use_fifos) {size_t args_size =0 ;char *p;
@@ -742,6 +746,7 @@ static int make_trace_req(struct tracecmd_msg *msg, int argc, char **argv) args_size +=strlen (argv[i]) +1 ; msg->hdr.size =htonl (ntohl (msg->hdr.size) + args_size); + msg->trace_req.flags = use_fifos ?htonl (MSG_TRACE_USE_FIFOS ) :htonl (0 ); msg->trace_req.argc =htonl (argc); msg->buf =calloc (args_size,1 );if (!msg->buf)
@@ -755,13 +760,13 @@ static int make_trace_req(struct tracecmd_msg *msg, int argc, char **argv) }int tracecmd_msg_send_trace_req (struct tracecmd_msg_handle *msg_handle, -int argc,char **argv) +int argc,char **argv,bool use_fifos) {struct tracecmd_msg msg;int ret;tracecmd_msg_init (MSG_TRACE_REQ , &msg); - ret =make_trace_req (&msg, argc, argv); + ret =make_trace_req (&msg, argc, argv, use_fifos);if (ret <0 )return ret;
@@ -774,7 +779,7 @@ int tracecmd_msg_send_trace_req(struct tracecmd_msg_handle *msg_handle,* free(argv); */ int tracecmd_msg_recv_trace_req (struct tracecmd_msg_handle *msg_handle, -int *argc,char ***argv) +int *argc,char ***argv,bool *use_fifos) {struct tracecmd_msg msg;char *p, *buf_end, **args;
@@ -819,6 +824,7 @@ int tracecmd_msg_recv_trace_req(struct tracecmd_msg_handle *msg_handle, msg.buf =NULL ; *argc = nr_args; *argv = args; + *use_fifos =ntohl (msg.trace_req.flags) &MSG_TRACE_USE_FIFOS ; ret =0 ; out:
@@ -826,13 +832,14 @@ out:return ret; } -static int make_trace_resp (struct tracecmd_msg *msg, -int page_size,int nr_cpus,unsigned int *ports) +static int make_trace_resp (struct tracecmd_msg *msg,int page_size,int nr_cpus, +unsigned int *ports,bool use_fifos) {int ports_size = nr_cpus *sizeof (*msg->port_array);int i; msg->hdr.size =htonl (ntohl (msg->hdr.size) + ports_size); + msg->trace_resp.flags = use_fifos ?htonl (MSG_TRACE_USE_FIFOS ) :htonl (0 ); msg->trace_resp.cpus =htonl (nr_cpus); msg->trace_resp.page_size =htonl (page_size);
@@ -848,13 +855,13 @@ static int make_trace_resp(struct tracecmd_msg *msg,int tracecmd_msg_send_trace_resp (struct tracecmd_msg_handle *msg_handle,int nr_cpus,int page_size, -unsigned int *ports) +unsigned int *ports,bool use_fifos) {struct tracecmd_msg msg;int ret;tracecmd_msg_init (MSG_TRACE_RESP , &msg); - ret =make_trace_resp (&msg, page_size, nr_cpus, ports); + ret =make_trace_resp (&msg, page_size, nr_cpus, ports, use_fifos);if (ret <0 )return ret;
@@ -863,7 +870,7 @@ int tracecmd_msg_send_trace_resp(struct tracecmd_msg_handle *msg_handle,int tracecmd_msg_recv_trace_resp (struct tracecmd_msg_handle *msg_handle,int *nr_cpus,int *page_size, -unsigned int **ports) +unsigned int **ports,bool *use_fifos) {struct tracecmd_msg msg;size_t buf_len;
@@ -885,6 +892,7 @@ int tracecmd_msg_recv_trace_resp(struct tracecmd_msg_handle *msg_handle, goto out; } + *use_fifos =ntohl (msg.trace_resp.flags) &MSG_TRACE_USE_FIFOS ; *nr_cpus =ntohl (msg.trace_resp.cpus); *page_size =ntohl (msg.trace_resp.page_size); *ports =calloc (*nr_cpus,sizeof (**ports));
tracecmd/trace-record.c +71 -13
161 lines not on this page · raw message
Signature · 1 line
2.19.1
Quoted 1 line · On Mon, 4 Feb 2019 08:58:37 +0200 Slavomir Kaslev <[email protected]> wrote:
This patchset adds support for tracing guest kernels to trace-cmd.
IIRC, you mentioned I should ignore these because you had another series coming? Let me know if that's not what you said.
Also, I looked at the splice code wrt. the buffer percentage. The splice code will return if either the file descriptor has O_NONBLOCK or the splice command has SPLICE_F_NONBLOCK set.
Or do you think that we need to do something in case the flag changes? To make everything non-block, we need to send a single to the code, which should cause the code to start again.
Once the trace_fd is set to O_NONBLOCK the splice fd should return immediately when there's nothing it can read left.
-- Steve
Quoted 5 lines
Changes in v5:
- add FIFOs transport for tracing data
- fixed a bug intracecmd_msg_wait_close