trace-cmd: Support the feature recording trace data of guests on the host
Subsystems (2)
v3 cover trace-cmd: Support the feature recording trace data of guests on the host
Hi Steven,
This is a v3 patch set for realizing a part of "Integrated trace" feature which is a trace merging system for a virtualization environment. Currently, trace-cmd does not have following features yet:
a) Server and client for a virtualization environment
b) Structured message platform between guests and host
c) Agent feature of a client
d) Merge feature of trace data of multiple guests and host in chronological order
This patch set supports above a) and b) features.
<overall view>
+------------+ +------------+
Guest | a), c) | | a), c) | client/agent
^ +------------+ +------------+
| ^ ^ ^ ^
============|===|=================|===|===========
| v b)v v b)v
v +----------------------------------+
Host | a) | server
+----------------------------------+
||output || ||
\/ \/ \/
/--------+ /--------+ /--------+
| 010101 | | 101010 | | 100101 | binary data
| 010100 | | 010100 | | 110011 |
+--------+ +--------+ +--------+
\ /
\-----------------------------------/
|| d)
\/
/-----------------------------------+
| (guest1) 123456: sched_switch... | text data
| (guest2) 123458: kmem_free... |
| (host) 123500: kvm_exit (guest1)|
| (host) 123510: kvm_entry(guest1)|
| (guest1) 123550: sched_switch... |
+-----------------------------------+a) Server and client for a virtualization environment
trace-cmd has listen mode for network, but using network will be a high cost
operation for inducing a lot of memory copying. From kernel-3.6, the virtio-console driver supports splice_write and ftrace supports "steal" for fops. So, guest clients of trace-cmd can send trace data without copying memory by using splice(2). If guest clients use virtio-serial, the server also needs to support virtio-serial I/F.
b) Structured message platform between guests and a host Currently, a server(clients) sends unstructured character string to
clients(server), so clients(server) must parse the unstructured messages. Since it is hard to add complex contents in the protocol, structured binary message trace-msg is introduced as the communication protocol.
c) Agent feature of a client
Current trace-cmd client can operate only as "record" mode, so the client
will send trace data to the server immediately. However, when an user tries to collect trace data of multiple guests on a host, the user must log in to each guest. This is hard to use, I think. So, trace-cmd client had better support agent mode which receives a message from the server.
d) Merge feature of trace data of multiple guests and a host in chronological order
Current trace-cmd has a merge feature for multiple machines whose times are
synchronized by NTP. When we use the feature, we execute "trace-cmd record" with --date option on each machine, and then we run "trace-cmd report" with -i option for each file.
However, there are cases that times of those machines cannot be synchronized.
For example, although multiple users can run guests on virtualization environments (e.g. multi-tenant cloud hosting), there are no guarantee that they use the same NTP server. Moreover, even if the times are synchronized, trace data cannot exactly be merged because the NTP-synchronized time granularity may not be enough fine for sorting guest-host switching events.
So, I'm considering that trace data use x86-tsc as timestamp in order to merge
trace data. By using x86-tsc, we can merge trace data even if time of those machines is not synchronized when CPU has the invariant TSC feature or the constant TSC feature. And the precision will be enough for understanding operations of guests and host. However, TSC values on a guest are not equal to the values on the host because
TSC_guest = TSC_host + TSC_offset.
This series actually doesn't support TSC offset, but I'd like to add such
feature to fix host/guest clock difference in the other series. TSC offset values can be gotten as write_tsc_offset trace event from kernel-3.11. (see https://lkml.org/lkml/2013/6/12/72)
For a), this patch introduces "virt-server" and "record --virt" modes for achieving low-overhead communication of trace data of guests. "virt-server" is a server mode for collecting trace data of guests. On the other hand, "record --virt" mode is a guest client for sending trace data of the guest. Although these functions are similar to "listen" and "record -N" modes each, these do not use network but use virtio-serial for low-overhead communication.
For b), this patch series introduce specific message protocol in order to handle communication messages with 8 commands. When we extend any messages, using structured message will be easier than using unstructured message.
<How to use>
1. Run virt-server on a host
# trace-cmd virt-server --dom guest1 -c 2
2. Set up of virtio-serial pipe of guest1 on the host Add the following tags to domain XML files.
# virsh edit guest1
<channel type='unix'>
<source mode='connect' path='/tmp/trace-cmd/virt/agent-ctl-path'/>
<target type='virtio' name='agent-ctl-path'/>
</channel>
<channel type='pipe'>
<source path='/tmp/trace-cmd/virt/guest1/trace-path-cpu0'/>
<target type='virtio' name='trace-path-cpu0'/>
</channel>
<channel type='pipe'>
<source path='/tmp/trace-cmd/virt/guest1/trace-path-cpu1'/>
<target type='virtio' name='trace-path-cpu1'/>
</channel>3. Boot the guest
# virsh start guest1
4. Run the guest1's client(see trace-cmd-record(1) with the *--virt* option)
# trace-cmd record -e sched* --virt
If you want to boot another guest sends trace-data via virtio-serial, you will manually make the guest domain directory and trace data I/Fs.
- Make guest domain directory on the host
# mkdir -p /tmp/trace-cmd/virt/<DOMAIN> # chmod 710 /tmp/trace-cmd/virt/<DOMAIN> # chgrp qemu /tmp/trace-cmd/virt/<DOMAIN>
- Make FIFO on the host
# mkfifo /tmp/trace-cmd/virt/<DOMAIN>/trace-path-cpu{0,1,...,X}.{in,out}<Result>
I measured CPU usage outputted by top command on a guest when client sends trace data. Client means "record -N"(NW) or "record --virt"(virtio-serial).
NW virtio-serial(splice)
client(fedora19) ~2.9[%] ~1.7[%]
<Future work>
- Add an agent mode based on "record --virt"
- Add a merging feature of trace data of guests and host to "report"
Changes in V2:
[1/5] Add a comment in open_udp()
[2/5] Regacy protocol support in order to keep backward compatibility
Changes in V3:
[2/6] Change the license of trace-msg.c to LGPL v2.1 [4/6] Change _nw/_NW to _net/_NET
[5/6] Change _nw/_NW to _net/_NET
[6/6] Add this patch based on Steven's review
(https://lkml.org/lkml/2013/10/14/618)
Thank you,
---
Yoshihiro YUNOMAE (6):
[CLEANUP] trace-cmd: Split out binding a port and fork reader from open_udp()
trace-cmd: Apply the trace-msg protocol for communication between a server and clients
trace-cmd: Use poll(2) to wait for a message
trace-cmd: Add virt-server mode for a virtualization environment
trace-cmd: Add --virt option for record mode
trace-cmd: Add --dom option which makes a domain directory to virt-server
Documentation/trace-cmd-record.1.txt | 11 Documentation/trace-cmd-virt-server.1.txt | 116 ++++ Makefile | 2 trace-cmd.c | 3 trace-cmd.h | 14 trace-listen.c | 727 ++++++++++++++++++++---- trace-msg.c | 874 +++++++++++++++++++++++++++++ trace-msg.h | 31 + trace-output.c | 4 trace-record.c | 146 ++++- trace-recorder.c | 54 +- trace-usage.c | 10 12 files changed, 1825 insertions(+), 167 deletions(-) create mode 100644 Documentation/trace-cmd-virt-server.1.txt create mode 100644 trace-msg.c create mode 100644 trace-msg.h
Signature · 4 lines
Yoshihiro YUNOMAE Software Platform Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: [email protected]
PATCH v3 1/6 trace-cmd: Split out binding a port and fork reader from open_udp()
Split out binding a port and fork reader from open_udp() for avoiding duplicate codes between listen mode and virt-server mode.
Changes in V2: Add a comment in open_udp()
--- trace-listen.c | 38 ++++++++++++++++++++++++++++++-------- 1 file changed, 30 insertions(+), 8 deletions(-)
trace-listen.c +30 -8
@@ -228,13 +228,12 @@ static void process_udp_child(int sfd, const char *host, const char *port,#define START_PORT_SEARCH 1500#define MAX_PORT_SEARCH 6000 -static int open_udp (const char *node,const char *port,int *pid, -int cpu,int pagesize,int start_port) +static int udp_bind_a_port (int start_port,int *sfd) {struct addrinfo hints;struct addrinfo *result, *rp; -int sfd, s;char buf[BUFSIZ ]; +int s;int num_port = start_port; again:
@@ -250,15 +249,15 @@ static int open_udp(const char *node, const char *port, int *pid,pdie ("getaddrinfo: error opening udp socket" );for (rp = result; rp !=NULL ; rp = rp->ai_next) { - sfd =socket (rp->ai_family, rp->ai_socktype, - rp->ai_protocol); -if (sfd <0 ) + *sfd =socket (rp->ai_family, rp->ai_socktype, + rp->ai_protocol); +if (*sfd <0 )continue ; -if (bind (sfd, rp->ai_addr, rp->ai_addrlen) ==0 ) +if (bind (*sfd, rp->ai_addr, rp->ai_addrlen) ==0 )break ; -close (sfd); +close (*sfd); }if (rp ==NULL ) {
@@ -270,6 +269,12 @@ static int open_udp(const char *node, const char *port, int *pid,freeaddrinfo (result); +return num_port; +} + +static void fork_udp_reader (int sfd,const char *node,const char *port, +int *pid,int cpu,int pagesize) +{ *pid =fork ();if (*pid <0 )
@@ -279,6 +284,23 @@ static int open_udp(const char *node, const char *port, int *pid,process_udp_child (sfd, node, port, cpu, pagesize);close (sfd); +} + +static int open_udp (const char *node,const char *port,int *pid, +int cpu,int pagesize,int start_port) +{ +int sfd; +int num_port; + +/* +* udp_bind_a_port() currently does not return an error, but if that +* changes in the future, we have a check for it now. +*/ + num_port =udp_bind_a_port (start_port, &sfd); +if (num_port <0 ) +return num_port; + +fork_udp_reader (sfd, node, port, pid, cpu, pagesize);return num_port; }
PATCH v3 2/6 trace-cmd: Apply the trace-msg protocol for communication between a server and clients
In reply to Yoshihiro YUNOMAEApply trace-msg protocol for communication between a server and clients.
Currently, trace-listen(server) and trace-record -N(client) operate as follows:
<server> <client>
listen to socket fd
connect to socket fd
accept the client
send "tracecmd"
+------------> receive "tracecmd"
check "tracecmd"
send cpus
receive cpus <------------+
print "cpus=XXX"
send pagesize
|
receive pagesize <--------+
print "pagesize=XXX"
send option
|
receive option <----------+
understand option
send port_array
+------------> receive port_array
understand port_array
send meta data
receive meta data <-------+
record meta data
(snip)
read block
--- start sending trace data on child processes ---
--- When client finishes sending trace data ---
close(socket fd)
read size = 0
close(socket fd)All messages are unstructured character strings, so server(client) using the protocol must parse the unstructured messages. Since it is hard to add complex contents in the protocol, structured binary message trace-msg is introduced as the communication protocol.
By applying this patch, server and client operate as follows:
<server> <client>
listen to socket fd
connect to socket fd
accept the client
send "tracecmd"
+------------> receive "tracecmd"
check "tracecmd"
send "V2\0<MAGIC_NUMBER>\00" as the v2 protocol
receive "V2" <------------+
check "V2"
read "<MAGIC_NUMBER>\00"
send "V2"
+---------------> receive "V2"
check "V2"
send cpus,pagesize,option(MSG_TINIT)
receive MSG_TINIT <-------+
print "cpus=XXX"
print "pagesize=XXX"
understand option
send port_array
+--MSG_RINIT-> receive MSG_RINIT
understand port_array
send meta data(MSG_SENDMETA)
receive MSG_SENDMETA <----+
record meta data
(snip)
send a message to finish sending meta data
| (MSG_FINMETA)
receive MSG_FINMETA <-----+
read block
--- start sending trace data on child processes ---
--- When client finishes sending trace data ---
send MSG_CLOSE
receive MSG_CLOSE <-------+
close(socket fd) close(socket fd)By introducing the v2 protocol, after the client checks "tracecmd", the client will send "V2\0<MAGIC_NUMBER>\00\0". This complex message is used when the new client tries to connect to the old server. The new client wants to check whether the reply message from the server is "V2" or not. However, the old server does not respond to the client before receiving cpu numbers, page size, and options. Each message is separated with "\0" in the old server, so the client send "V2" as cpu numbers, "<MAGIC_NUMBER>" as page size, and "0" as no options. On the other hands, the old server will understand the messages as cpus=0, pagesize=<MAGIC_NUMBER>, and options=0, and then the server will send the message "\0" as port numbers. Then, the message which the client receives is not "V2" but "\0", so the client will reconnect to the old server as the v1 protocol.
Changes in V2: Regacy porotocol support in order to keep backward compatibility Changes in V3: Change the license of trace-msg.c to LGPL v2.1
--- Makefile | 2 trace-cmd.h | 11 + trace-listen.c | 133 +++++++---- trace-msg.c | 683 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ trace-msg.h | 27 ++ trace-output.c | 4 trace-record.c | 86 ++++++- 7 files changed, 880 insertions(+), 66 deletions(-) create mode 100644 trace-msg.c create mode 100644 trace-msg.h
Makefile +1 -1
@@ -314,7 +314,7 @@ KERNEL_SHARK_OBJS = $(TRACE_VIEW_OBJS) $(TRACE_GRAPH_OBJS) $(TRACE_GUI_OBJS) \ PEVENT_LIB_OBJS = event-parse.o trace-seq.o parse-filter.o parse-utils.o TCMD_LIB_OBJS = $(PEVENT_LIB_OBJS) trace-util.o trace-input.o trace-ftrace.o \ trace-output.o trace-recorder.o trace-restore.o trace-usage.o \ - trace-blk-hack.o kbuffer-parse.o + trace-blk-hack.o kbuffer-parse.o trace-msg.o PLUGIN_OBJS = plugin_hrtimer.o plugin_kmem.o plugin_sched_switch.o \ plugin_mac80211.o plugin_jbd2.o plugin_function.o plugin_kvm.o \
trace-cmd.h +11 -0
@@ -248,6 +248,17 @@ void tracecmd_stop_recording(struct tracecmd_recorder *recorder);void tracecmd_stat_cpu (struct trace_seq *s,int cpu);long tracecmd_flush_recording (struct tracecmd_recorder *recorder); +/* for clients */ +int tracecmd_msg_send_init_data (int fd); +int tracecmd_msg_metadata_send (int fd,char *buf,int size); +int tracecmd_msg_finish_sending_metadata (int fd); +void tracecmd_msg_send_close_msg (); + +/* for server */ +int tracecmd_msg_initial_setting (int fd,int *cpus,int *pagesize); +int tracecmd_msg_send_port_array (int fd,int total_cpus,int *ports); +int tracecmd_msg_collect_metadata (int ifd,int ofd); +/* --- Plugin handling --- */ extern struct plugin_option trace_ftrace_options[];
trace-listen.c +82 -51
199 lines not on this page · raw message
trace-msg.c new file +683 -0
683 lines not on this page · raw message
trace-msg.h new file +27 -0
@@ -0,0 +1,27 @@ +#ifndef _TRACE_MSG_H_ +#define _TRACE_MSG_H_ + +#include <stdbool.h> + +#define UDP_MAX_PACKET (65536 - 20) +#define V2_MAGIC "677768\0" + +#define V1_PROTOCOL 1 +#define V2_PROTOCOL 2 + +/* for both client and server */ +extern bool use_tcp; +extern int cpu_count; + +/* for client */ +extern unsigned int page_size; +extern int *client_ports; +extern bool send_metadata; + +/* for server */ +extern bool done; + +void plog (const char *fmt, ...); +void pdie (const char *fmt, ...); + +#endif /* _TRACE_MSG_H_ */
trace-output.c +4 -0
@@ -36,6 +36,7 @@#include <glob.h> #include "trace-cmd-local.h" +#include "trace-msg.h" #include "version.h" /* We can't depend on the host size for size_t, all must be 64 bit */
@@ -80,6 +81,9 @@ struct list_event_system {static stsize_t do_write_check (struct tracecmd_output *handle,void *data,tsize_t size) { +if (send_metadata) +return tracecmd_msg_metadata_send (handle->fd, data, size); +return __do_write_check (handle->fd, data, size); }
trace-record.c +72 -14
161 lines not on this page · raw message
PATCH v3 3/6 trace-cmd: Use poll(2) to wait for a message
In reply to Yoshihiro YUNOMAEUse poll(2) to wait for a message. If a client/server cannot send a message for any reasons, the current server/client will wait in a blocking read operation. So, we use poll(2) for avoiding remaining in a blocking state.
--- trace-msg.c | 42 ++++++++++++++++++++++++++++++++++++------ 1 file changed, 36 insertions(+), 6 deletions(-)
trace-msg.c +36 -6
@@ -396,6 +396,27 @@ error:return -ENOMSG ; } +#define MSG_WAIT_MSEC 5000 + +/* +* A return value of 0 indicates time-out +*/ +static int tracecmd_msg_recv_wait (int fd,char *buf,struct tracecmd_msg **msg) +{ +struct pollfd pfd; +int ret; + + pfd.fd = fd; + pfd.events =POLLIN ; + ret =poll (&pfd,1 ,MSG_WAIT_MSEC ); +if (ret <0 ) { +return -errno; + }else if (ret ==0 ) +return -ETIMEDOUT ; + +return tracecmd_msg_recv (fd, buf); +} +static void *tracecmd_msg_buf_access (struct tracecmd_msg *msg,int offset) {return (void *)msg + offset;
@@ -407,9 +428,12 @@ static int tracecmd_msg_wait_for_msg(int fd, struct tracecmd_msg **msg)u32 cmd;int ret; - ret =tracecmd_msg_recv (fd, msg_tmp); -if (ret <0 ) + ret =tracecmd_msg_recv_wait (fd, msg_tmp, msg); +if (ret <0 ) { +if (ret == -ETIMEDOUT ) +warning ("Connection timed out\n" );return ret; + } *msg = (struct tracecmd_msg *)msg_tmp; cmd =ntohl ((*msg)->cmd);
@@ -487,9 +511,12 @@ int tracecmd_msg_initial_setting(int fd, int *cpus, int *pagesize)u32 size =0 ;u32 cmd; - ret =tracecmd_msg_recv (fd, buf); -if (ret <0 ) + ret =tracecmd_msg_recv_wait (fd, buf, &msg); +if (ret <0 ) { +if (ret == -ETIMEDOUT ) +warning ("Connection timed out\n" );return ret; + } msg = (struct tracecmd_msg *)buf; cmd =ntohl (msg->cmd);
@@ -625,9 +652,12 @@ int tracecmd_msg_collect_metadata(int ifd, int ofd)int ret; do { - ret = tracecmd_msg_recv(ifd, buf); + ret = tracecmd_msg_recv_wait(ifd, buf, &msg); if (ret < 0) { - warning("reading client"); + if (ret == -ETIMEDOUT) + warning("Connection timed out\n"); + else + warning("reading client"); return ret; }
PATCH v3 4/6 trace-cmd: Add virt-server mode for a virtualization environment
In reply to Yoshihiro YUNOMAEAdd the virt-server mode for a virtualization environment based on the listen mode for networking. This mode works like client/server mode over TCP/UDP, but it uses virtio-serial channel instead of IP network. Using networking for collecting trace data of guests is generally high overhead caused by processing of the network stack.
We use virtio-serial for collecting trace data of guests. virtio-serial is a simple communication path between the guest and the host. Moreover, since virtio-serial and ftrace can use splice(2), memory copying is not occurred on the guests. Therefore, total overhead for collecting trace data of the guests will be reduced. The implementation of clients will be shown in another patch.
virt-server uses two kinds of virtio-serial I/Fs:
(1) agent-ctl-path(UNIX domain socket)
=> control path of an agent trace-cmd each guest
(2) trace-path-cpuX(named pipe)
=> trace data path each vcpu
Those I/Fs must be defined as below paths:
(1) /tmp/trace-cmd/virt/agent-ctl-path
(2) /tmp/trace-cmd/virt/<guest domain>/trace-path-cpuX
If we run virt-server, agent-ctl-path I/F is automatically created because virt-server operates as a server mode of UNIX domain socket. However, trace-path-cpuX is not automatically created because we need to separate trace data for each guests.
When the client uses virtio-serial, the client must notify the server of the connection. This is because a virtio-serial I/F on the guest is a just character device. In other words, the server cannot understand whether the client exists or not even if the client opens the I/F. So, the server using virtio-serial waits for the connection message MSG_TCONNECT from the client.
The server and the client operate as follows:
<server> <client>
wait for MSG_TCONNECT
open virtio-serial I/F
send MSG_TCONNECT
receive MSG_TCONNECT <----+
send MSG_RCONNECT
+---------------> receive MSG_RCONNECT
check "tracecmd-V2"
send cpus,pagesize,option(MSG_TINIT)
receive MSG_TINIT <-------+
print "cpus=XXX"
print "pagesize=XXX"
understand option
send port_array
+--MSG_RINIT-> receive MSG_RINIT
understand port_array
send meta data(MSG_SENDMETA)
receive MSG_SENDMETA <----+
record meta data
(snip)
send a message to finish sending meta data
| (MSG_FINMETA)
receive MSG_FINMETA <-----+
read block
--- start sending trace data on child processes ---
--- When client finishes sending trace data ---
send MSG_CLOSE
receive MSG_CLOSE <-------+
close(socket fd) close(socket fd)
<How to set up>
1. Run virt-server on a host before booting guests
# trace-cmd virt-server
2. Make guest domain directory
# mkdir -p /tmp/trace-cmd/virt/<domain> # chmod 710 /tmp/trace-cmd/virt/<domain> # chgrp qemu /tmp/trace-cmd/virt/<domain>
3. Make FIFO on the host
# mkfifo /tmp/trace-cmd/virt/<domain>/trace-path-cpu{0,1,...,X}.{in,out}4. Set up of virtio-serial pipe of a guest on the host Add the following tags to domain XML files.
# virsh edit <domain>
<channel type='unix'>
<source mode='connect' path='/tmp/trace-cmd/virt/agent-ctl-path'/>
<target type='virtio' name='agent-ctl-path'/>
</channel>
<channel type='pipe'>
<source path='/tmp/trace-cmd/virt/<domain>/trace-path-cpu0'/>
<target type='virtio' name='trace-path-cpu0'/>
</channel>
... (cpu1, cpu2, ...)5. Boot the guest
# virsh start <domain>
6. Check I/F of virtio-serial on the guest
# ls /dev/virtio-ports
...
agent-ctl-path
...
trace-path-cpu0
...Next, the user will run trace-cmd with record --virt options or other options for virtualization on the guest.
This patch adds only minimum features of virt-server as follows:
<Features>
- virt-server subcommand
- Create I/F directory(/tmp/trace-cmd/virt/)
- Use named pipe I/Fs of virtio-serial for trace data paths
- Use UNIX domain socket for connecting clients on guests
- Use splice(2) for collecting trace data of guests
<Restrictions>
- Use libvirt when we boot guests
Changes in V3: Change _nw/_NW to _net/_NET
--- Documentation/trace-cmd-virt-server.1.txt | 89 ++++++ trace-cmd.c | 3 trace-cmd.h | 2 trace-listen.c | 439 ++++++++++++++++++++++++----- trace-msg.c | 105 ++++++- trace-recorder.c | 54 +++- trace-usage.c | 10 + 7 files changed, 602 insertions(+), 100 deletions(-) create mode 100644 Documentation/trace-cmd-virt-server.1.txt
Documentation/trace-cmd-virt-server.1.txt new file +89 -0
@@ -0,0 +1,89 @@ +TRACE-CMD-VIRT-SERVER(1) +======================== + +NAME +---- +trace-cmd-virt-server - listen for incoming connection to record tracing of + guests' clients + +SYNOPSIS +-------- +*trace-cmd virt-server ['OPTIONS'] + +DESCRIPTION +----------- +The trace-cmd(1) virt-server sets up UNIX domain socket I/F for communicating +with guests' clients that run 'trace-cmd-record(1)' with the *--virt* option. +When a connection is made, and the guest's client sends data, it will create a +file called 'trace.DOMAIN.dat'. Where DOMAIN is the name of the guest named +by libvirt. + +OPTIONS +------- +*-D*:: + This options causes trace-cmd listen to go into a daemon mode and run in + the background. + +*-d* 'dir':: + This option specifies a directory to write the data files into. + +*-o* 'filename':: + This option overrides the default 'trace' in the 'trace.DOMAIN.dat' that + is created when guest's client connects. + +*-l* 'filename':: + This option writes the output messages to a log file instead of standard output. + +SET UP +------ +Here, an example is written as follows: + +1. Run virt-server on a host + # trace-cmd virt-server + +2. Make guest domain directory + # mkdir -p /tmp/trace-cmd/virt/<DOMAIN> + # chmod 710 /tmp/trace-cmd/virt/<DOMAIN> + # chgrp qemu /tmp/trace-cmd/virt/<DOMAIN> + +3. Make FIFO on the host + # mkfifo /tmp/trace-cmd/virt/<DOMAIN>/trace-path-cpu{0,1,...,X}.{in,out} + +4. Set up of virtio-serial pipe of a guest on the host + Add the following tags to domain XML files. + # virsh edit <guest domain> + <channel type='unix'> + <source mode='connect' path='/tmp/trace-cmd/virt/agent-ctl-path'/> + <target type='virtio' name='agent-ctl-path'/> + </channel> + <channel type='pipe'> + <source path='/tmp/trace-cmd/virt/<DOMAIN>/trace-path-cpu0'/> + <target type='virtio' name='trace-path-cpu0'/> + </channel> + ... (cpu1, cpu2, ...) + +5. Boot the guest + # virsh start <DOMAIN> + +6. Run the guest's client(see trace-cmd-record(1) with the *--virt* option) + # trace-cmd record -e sched* --virt + +SEE ALSO +-------- +trace-cmd(1), trace-cmd-record(1), trace-cmd-report(1), trace-cmd-start(1), +trace-cmd-stop(1), trace-cmd-extract(1), trace-cmd-reset(1), +trace-cmd-split(1), trace-cmd-list(1) + +AUTHOR +------ +Written by Yoshihiro YUNOMAE, <[email protected]> + +RESOURCES +--------- +git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git + +COPYING +------- +Copyright \(C) 2013 Hitachi, Ltd. Free use of this software is granted under +the terms of the GNU Public License (GPL). +
trace-cmd.c +2 -1
@@ -219,7 +219,8 @@ int main (int argc, char **argv) }else if (strcmp (argv[1 ],"mem" ) ==0 ) {trace_mem (argc, argv);exit (0 ); - }else if (strcmp (argv[1 ],"listen" ) ==0 ) { + }else if (strcmp (argv[1 ],"listen" ) ==0 || +strcmp (argv[1 ],"virt-server" ) ==0 ) {trace_listen (argc, argv);exit (0 ); }else if (strcmp (argv[1 ],"split" ) ==0 ) {
trace-cmd.h +2 -0
@@ -242,6 +242,7 @@ struct tracecmd_recorder *tracecmd_create_recorder_maxkb(const char *file, int cstruct tracecmd_recorder *tracecmd_create_buffer_recorder_fd (int fd,int cpu,unsigned flags,const char *buffer);struct tracecmd_recorder *tracecmd_create_buffer_recorder (const char *file,int cpu,unsigned flags,const char *buffer);struct tracecmd_recorder *tracecmd_create_buffer_recorder_maxkb (const char *file,int cpu,unsigned flags,const char *buffer,int maxkb); +struct tracecmd_recorder *tracecmd_create_recorder_virt (const char *file,int cpu,int trace_fd);int tracecmd_start_recording (struct tracecmd_recorder *recorder,unsigned long sleep);void tracecmd_stop_recording (struct tracecmd_recorder *recorder);
@@ -255,6 +256,7 @@ int tracecmd_msg_finish_sending_metadata(int fd);void tracecmd_msg_send_close_msg ();/* for server */ +int tracecmd_msg_set_connection (int fd,const char *domain);int tracecmd_msg_initial_setting (int fd,int *cpus,int *pagesize);int tracecmd_msg_send_port_array (int fd,int total_cpus,int *ports);int tracecmd_msg_collect_metadata (int ifd,int ofd);
trace-listen.c +362 -77
674 lines not on this page · raw message
trace-msg.c +98 -7
179 lines not on this page · raw message
trace-recorder.c +39 -15
72 lines not on this page · raw message
trace-usage.c +10 -0
@@ -153,6 +153,16 @@ static struct usage_help usage_help[] = {" -l logfile to write messages to.\n" }, { +"virt-server" , +"listen on a virtio-serial for trace clients" , +" %s virt-server [-o file][-d dir][-l logfile]\n" +" Creates a socket to listen for clients.\n" +" -D create it in daemon mode.\n" +" -o file name to use for clients.\n" +" -d diretory to store client files.\n" +" -l logfile to write messages to.\n" + }, + {"list" ,"list the available events, plugins or options" ," %s list [-e][-t][-o][-f [regex]]\n"
PATCH v3 5/6 trace-cmd: Add --virt option for record mode
In reply to Yoshihiro YUNOMAEAdd --virt option for record mode for a virtualization environment. If we use this option on a guest, we can send trace data in low-overhead. This is because guests can send trace data to a host without copying the data by using splice(2).
The format is:
trace-cmd record --virt -e sched*
<Note>
The client using virtio-serial does not wait for the connection message "tracecmd" from the server. The client sends the connection message MSG_TCONNECT first.
<Restriction>
This feature can use from kernel-3.6 which supports splice_read for ftrace and splice_write for virtio-serial.
Changes in V3: Change _nw/_NW to _net/_NET
--- Documentation/trace-cmd-record.1.txt | 11 ++++- trace-cmd.h | 3 + trace-msg.c | 80 ++++++++++++++++++++++++++++++++-- trace-msg.h | 4 ++ trace-record.c | 70 ++++++++++++++++++++++++++++-- 5 files changed, 156 insertions(+), 12 deletions(-)
Documentation/trace-cmd-record.1.txt +10 -1
@@ -240,6 +240,15 @@ OPTIONS timestamp to gettimeofday which will allow wall time output from the timestamps reading the created 'trace.dat' file. +*--virt*:: + This option is usded on a guest in a virtualization environment. If a host + is running "trace-cmd virt-server", this option is used to have the data + sent to the host with virtio-serial like *-N* option. (see also + trace-cmd-virt-server(1)) + + Note: This option is not supported with latency tracer plugins: + wakeup, wakeup_rt, irqsoff, preemptoff and preemptirqsoff + EXAMPLES --------
@@ -302,7 +311,7 @@ SEE ALSO -------- trace-cmd(1), trace-cmd-report(1), trace-cmd-start(1), trace-cmd-stop(1), trace-cmd-extract(1), trace-cmd-reset(1), trace-cmd-split(1), -trace-cmd-list(1), trace-cmd-listen(1) +trace-cmd-list(1), trace-cmd-listen(1), trace-cmd-virt-server(1) AUTHOR ------
trace-cmd.h +2 -1
@@ -250,7 +250,8 @@ void tracecmd_stat_cpu(struct trace_seq *s, int cpu);long tracecmd_flush_recording (struct tracecmd_recorder *recorder);/* for clients */ -int tracecmd_msg_send_init_data (int fd); +int tracecmd_msg_connect_to_server (int fd); +int tracecmd_msg_send_init_data_net (int fd);int tracecmd_msg_metadata_send (int fd,char *buf,int size);int tracecmd_msg_finish_sending_metadata (int fd);void tracecmd_msg_send_close_msg ();
trace-msg.c +75 -5
149 lines not on this page · raw message
trace-msg.h +4 -0
@@ -2,6 +2,9 @@#define _TRACE_MSG_H_#include <stdbool.h> +#define VIRTIO_PORTS "/dev/virtio-ports/" +#define AGENT_CTL_PATH VIRTIO_PORTS "agent-ctl-path" +#define TRACE_PATH_CPU VIRTIO_PORTS "trace-path-cpu%d"#define UDP_MAX_PACKET (65536 - 20)#define V2_MAGIC "677768\0"
@@ -17,6 +20,7 @@ extern int cpu_count;extern unsigned int page_size;extern int *client_ports;extern bool send_metadata; +extern int *virt_sfds;/* for server */ extern bool done;
trace-record.c +65 -5
176 lines not on this page · raw message
PATCH v3 6/6 trace-cmd: Add --dom option which makes a domain directory to virt-server
In reply to Yoshihiro YUNOMAEAdd --dom option which makes a domain directory to virt-server. When a user already knows domain name of a guest before running virt-server, trace-cmd should automatically set up I/Fs of the guest. By adding --dom option, trace-cmd creates a domain directory with 0710 and qemu group.
This patch adds additional options for --dom as follows:
-m <permission>
This option changes the permission of domain directory. If you don't use this option, the default permission is 0710.
-g <group>
This option changes group of domain directory. If you don't use this option, the default group is qemu.
-c <cpu>
This option creates trace data I/Fs(trace-path-cpu*.{in,out}) for each CPU of 'domain'. If you don't use this option, those files are not created.
Here, an example you use this option is written as follows:
- trace-cmd creates a guest1 directory with trace data I/Fs of 2 CPUs.
# trace-cmd virt-server --dom guest1 -c 2
- trace-cmd creates guest2 and guest3 directories
# trace-cmd virt-server --dom guest2 -c 3 --dom guest3 -c 1
Changes in V3: Add this patch
--- Documentation/trace-cmd-virt-server.1.txt | 57 ++++++++--- trace-listen.c | 151 ++++++++++++++++++++++++++--- 2 files changed, 177 insertions(+), 31 deletions(-)
Documentation/trace-cmd-virt-server.1.txt +42 -15
@@ -34,40 +34,67 @@ OPTIONS *-l* 'filename':: This option writes the output messages to a log file instead of standard output. +*--dom* 'domain':: + This option makes a directory for the 'domain'. You can use additional options + *-m*, *-g*, *-c* after this option for the 'domain'. If you don't use these + additional options, the directory is made as 0710 and qemu group and + trace data I/Fs(trace-path-cpu*.{in,out}) are not created. + +*-m* 'permission':: + This option changes the permission of 'domain' directory. If you don't use + this option, the default permission is 0710. + +*-g* 'group':: + This option changes group of 'domain' directory. If you don't use this option, + the default group is qemu. + +*-c* 'cpu':: + This option creates trace data I/Fs(trace-path-cpu*.{in,out}) for each CPU + of 'domain'. If you don't use this option, those files are not created. + SET UP ------ Here, an example is written as follows: 1. Run virt-server on a host - # trace-cmd virt-server + # trace-cmd virt-server --dom guest1 -c 2 -2. Make guest domain directory - # mkdir -p /tmp/trace-cmd/virt/<DOMAIN> - # chmod 710 /tmp/trace-cmd/virt/<DOMAIN> - # chgrp qemu /tmp/trace-cmd/virt/<DOMAIN> - -3. Make FIFO on the host - # mkfifo /tmp/trace-cmd/virt/<DOMAIN>/trace-path-cpu{0,1,...,X}.{in,out} + <NOTE> Set up two guests(guest1, guest2): + # trace-cmd virt-server --dom guest1 -c 2 --dom guest2 -c 1 -4. Set up of virtio-serial pipe of a guest on the host +2. Set up of virtio-serial pipe of guest1 on the host Add the following tags to domain XML files. - # virsh edit <guest domain> + # virsh edit guest1 <channel type='unix'> <source mode='connect' path='/tmp/trace-cmd/virt/agent-ctl-path'/> <target type='virtio' name='agent-ctl-path'/> </channel> <channel type='pipe'> - <source path='/tmp/trace-cmd/virt/<DOMAIN>/trace-path-cpu0'/> + <source path='/tmp/trace-cmd/virt/guest1/trace-path-cpu0'/> <target type='virtio' name='trace-path-cpu0'/> </channel> - ... (cpu1, cpu2, ...) + <channel type='pipe'> + <source path='/tmp/trace-cmd/virt/guest1/trace-path-cpu1'/> + <target type='virtio' name='trace-path-cpu1'/> + </channel> -5. Boot the guest - # virsh start <DOMAIN> +3. Boot the guest + # virsh start guest1 -6. Run the guest's client(see trace-cmd-record(1) with the *--virt* option) +4. Run the guest1's client(see trace-cmd-record(1) with the *--virt* option) # trace-cmd record -e sched* --virt +If you want to boot another guest sends trace-data via virtio-serial, +you will manually make the guest domain directory and trace data I/Fs. + +- Make guest domain directory on the host + # mkdir -p /tmp/trace-cmd/virt/<DOMAIN> + # chmod 710 /tmp/trace-cmd/virt/<DOMAIN> + # chgrp qemu /tmp/trace-cmd/virt/<DOMAIN> + +- Make FIFO on the host + # mkfifo /tmp/trace-cmd/virt/<DOMAIN>/trace-path-cpu{0,1,...,X}.{in,out} + SEE ALSO -------- trace-cmd(1), trace-cmd-record(1), trace-cmd-report(1), trace-cmd-start(1),
trace-listen.c +135 -16
212 lines not on this page · raw message