Skip to contentSkip to conversation

Timestamp synchronization of host - guest tracing session

PATCH Superseded Tzvetomir Stoyanov (VMware) 2 participants 18 messages Mar 2, 2020 – Mar 4, 2020 Documentation

v21 of 34 Superseded Latest is v34 →

lore
Share & tools
lore
#0 lore raw

v21 cover Timestamp synchronization of host - guest tracing session

Basic infrastructure for host - guest timestamp synchronization and a POC implementation of PTP-like algorithm.

[
v21 changes:
- Rebased on top of latest master.
- Remove these two patches from the set, as they are already merged:
trace-cmd: Add new library API for local CPU count
trace-cmd: Add support for negative time offsets in trace.dat file
- Added more descriptive warning message when fails to extract Time Shift information from the trace.dat file.
- Added a warning message when fails to obtain vcpu - pid mappings.
- Handle the case with sparse VCPU numbers in VCPU - PID mapping array.
- Fixed broken recording over FIFOs.
v20 changes:
- Rebased on top of latest master.
- Removed the VCPUS_MAX hardcoded limit, reimplemented the cpu_pid[] array to be dynamically allocated.
- Added a warning when reading of TRACECMD_OPTION_TIME_SHIFT option fails, due to unexpected option's size.
- Improved loading of TRACECMD_OPTION_GUEST option data, as Steven suggested.
v19 changes:
- Rebased on top of latest master. The librtacefs is already merged, this allows to drop few patches from the set and use librtacefs APIs instead.
- Reimplemented all new trace.dat options to be in binary format, instead of text. Leverage the new "trace-cmd dump" sub command to track what is written in the file.
- Addressed Steven's comments.

v18 changes: addressed Steven Rostdet comments:
- Replaced semaphores with pthread mutexes.
- Made bitmask with time sync protocols unlimited, so we can support more than 32 protocols. That required to redesign the trace request packet format.
- A lot of small fixes.

v17 changes:
- Implemented new PTP logic for calculating the clocks offset, using histogram of all PTP samples. It gives better results than the logic with the fastest response time, so set the new one as default in the POC patch.

v16 changes:
- Fixed compilation in case no VSOCK is available (Thanks to Slavomir Kaslev)
- Fixed a typo in trace-cmd-record.1.txt (Thanks to Slavomir Kaslev)
- Added forgotten file in the patch "trace-cmd: Add new library APIs for ftrace instances." - trace-instance.c
- Fixed few compilation warnings related to TSYNC_DEBUG code.
- Removed a blank line at the end of "tsync_readme" file.

v15 changes:
- Removed the patch for "--proc-map" from the series, as it should not be part of it.

v14 changes:
- Bring back the PTP-like algorithm and removed the ftrace event based logic.
- Reimplemented the PTP-like algorithm to use raw ftrace markers, instead of clock_gettime() API.
- Refactored the logic to be algorithm independent and plugin friendly.
- Implemented continuous timestamps synchronization, while the trace is running.
- Moved logic from trace-cmd application to libtracecmd, as new library APIs.
- Implemented new trace id functionality.
- Implemented new guest section in host trace.dat file.

v13 changes:
- Remove few patches from the set, as they were merged.
- Rebased to the latest master, Slavomirs patchest "Add VM kernel tracing over vsockets and FIFOs" got merged!

v12 changes:
- Rebased on top of Slavomir's v13 "Add VM kernel tracing over vsockets and FIFOs"

v11 changes:
- Rebased on top of Slavomir's v10 "Add VM kernel tracing over vsockets and FIFOs"
- Addressed Slavomir's commnents from version 10 of the patch series.

v10 changes:
- Fixed broken compilation, call to timestamp_correction_calc() in timestamp_correct was smashed.
- Replaced deprecated tep_data_event_from_type() API with tep_find_event().
- Fixed a warning on assignment const to non const.

v9 changes:
- Fixed implementation of binary search algorithm in timestamp_correct()

v8 changes:
- Added rmdir() call in tracecmd_remove_instance(), to completely remove the instance. However, there is an issue with deleting the instances using rmdir(), which is investigated.
- Few changes in read_qemu_guests_pids(), timestamp_correct(), tsync_offset_load()
tracecmd_clock_context_new() and find_raw_events() suggested by Slavomir.

v7 changes:
- Added warning messages in case time synchronization cannot be negotiated or fails.
- Few optimizations and checks in read_qemu_guests_pids(), tsync_offset_load(), and find_raw_events(), suggested by Slavomir Kaslev.
- Reworked timestamp_correct() to not use static variables.
- Check TRACECMD_OPTION_TIME_SHIFT before reading time sync samples from the trace.dat file

v6 changes:
- Refactored tracecmd_msg_snd_time_sync() and tracecmd_msg_rcv_time_sync() functions:
removed any time sync calculations logic as separate functions in trace-timesync.c file
- Defined TSYNC_PROBE, TSYNC_REQ and TSYNC_RESP messages, in order to make the time sync protocol comprehensible.
- Addressed Steven Rostedt comments.
- Addressed Slavomir Kaslev commnets.

v5 changes:
- Rebased to Slavomir's v8 "Add VM kernel tracing over vsockets and FIFOs" patch series.
- Implemented an algorithm for time drift correction.
- Addressed Slavomir's commnets.
- Refactored the code: moved all time sync specific implementation in trace-timesync.c
- Isolated all hardcoded event specific stuff in a structure, so it could be easily moved to external plugins.
- Added a check for VSOCK support: do not perform vsock dependent time synchronisation in case there is no VSOCK support.

v4 changes:
- Removed the implementation of PTP-like algorithm. The current logic relies on matching time stamps of kvm_exit/virtio_transport_recv_pkt events on host to virtio_transport_alloc_pkt/vp_notify events on guest.
- Rebased to Slavomir's v7 "Add VM kernel tracing over vsockets and FIFOs" patch series.
- Decreased the time synch probes from 5000 to 300.
- Addressed Steven Rostedt comments.
- Code cleanup.

v3 changes:
- Removed any magic constants, used in the PTP-like algorithm, as Slavomir Kaslev suggested.
- Implemented new algorithm, based on mapping kvm_exit events in host context to vsock_send events in guest context, suggested by Steven Rostedt.

v2 changes:
- Addressed Steven Rostedt comments.
- Modified PTP-like timestamps sync algorithm to gain more accuracy, with the help of Yordan Karadzhov and Slavomir Kaslev.
]

Tzvetomir Stoyanov (4):
trace-cmd: Find and store pids of tasks, which run virtual CPUs of
given VM
trace-cmd: Implement new API tracecmd_add_option_v()
trace-cmd: Add definitions of htonll() and ntohll()
trace-cmd: Implement new option in trace.dat file:
TRACECMD_OPTION_TIME_SHIFT

Tzvetomir Stoyanov (VMware) (9):
trace-cmd: Add new API to generate a unique ID of the tracing session
trace-cmd: Store the session tracing ID in the trace.dat file
trace-cmd: Exchange tracing IDs between host and guest
trace-cmd: Add guest information in host's trace.dat file
trace-cmd: Add host trace clock as guest trace argument
trace-cmd: Refactor few trace-cmd internal functions.
trace-cmd: Basic infrastructure for host - guest timestamp
synchronization
trace-cmd: [POC] PTP-like algorithm for host - guest timestamp
synchronization
trace-cmd: Debug scripts for PTP-like algorithm for host - guest
timestamp synchronization

 Documentation/trace-cmd-record.1.txt      |   7 +
 include/trace-cmd/trace-cmd.h             |  79 ++-
 include/traceevent/event-parse.h          |   1 +
 lib/trace-cmd/Makefile                    |   2 +
 lib/trace-cmd/include/trace-cmd-local.h   |  45 +-
 lib/trace-cmd/include/trace-tsync-local.h |  38 ++
 lib/trace-cmd/include/trace-write-local.h |  43 ++
 lib/trace-cmd/trace-input.c               | 332 ++++++++++-
 lib/trace-cmd/trace-msg.c                 | 402 +++++++++++--
 lib/trace-cmd/trace-output.c              | 119 +++-
 lib/trace-cmd/trace-timesync-ptp.c        | 651 ++++++++++++++++++++++
 lib/trace-cmd/trace-timesync.c            | 506 +++++++++++++++++
 lib/trace-cmd/trace-util.c                |  37 ++
 scripts/debug/tsync_hist.py               |  57 ++
 scripts/debug/tsync_readme                |  12 +
 scripts/debug/tsync_res.py                |  46 ++
 tracecmd/Makefile                         |   3 +-
 tracecmd/include/trace-local.h            |  24 +-
 tracecmd/trace-agent.c                    |  57 +-
 tracecmd/trace-dump.c                     | 122 ++++
 tracecmd/trace-record.c                   | 263 ++++++++-
 tracecmd/trace-tsync.c                    | 272 +++++++++
 tracecmd/trace-usage.c                    |   4 +
 23 files changed, 2981 insertions(+), 141 deletions(-)
 create mode 100644 lib/trace-cmd/include/trace-tsync-local.h
 create mode 100644 lib/trace-cmd/include/trace-write-local.h
 create mode 100644 lib/trace-cmd/trace-timesync-ptp.c
 create mode 100644 lib/trace-cmd/trace-timesync.c
 create mode 100644 scripts/debug/tsync_hist.py
 create mode 100644 scripts/debug/tsync_readme
 create mode 100644 scripts/debug/tsync_res.py
 create mode 100644 tracecmd/trace-tsync.c
Signature · 1 line
2.24.1
#1 lore raw

PATCH v21 1/13 trace-cmd: Find and store pids of tasks, which run virtual CPUs of given VM

From: Tzvetomir Stoyanov <[email protected]>

In order to match host and guest events, a mapping between guest VCPU and the host task, running this VCPU is needed. Extended existing struct guest to hold such mapping and added logic in read_qemu_guests() function to initialize it. Implemented a new internal API, get_guest_vcpu_pid(), to retrieve VCPU-task mapping for given VM.

---
 tracecmd/include/trace-local.h |  2 +
 tracecmd/trace-record.c        | 84 ++++++++++++++++++++++++++++++++++
 2 files changed, 86 insertions(+)
tracecmd/include/trace-local.h +2 -0
index 29f27793..a5cf0640 100644
@@ -247,6 +247,8 @@ void update_first_instance(struct buffer_instance *instance, int topt);
 
 void show_instance_file(struct buffer_instance *instance, const char *name);
 
+int get_guest_vcpu_pid(unsigned int guest_cid, unsigned int guest_vcpu);
+
 /* moved from trace-cmd.h */
 void tracecmd_create_top_instance(char *name);
 void tracecmd_remove_instances(void);
tracecmd/trace-record.c +84 -0

113 lines not on this page · raw message

Signature · 1 line
2.24.1
#2 lore raw

Re: [PATCH v21 01/13] trace-cmd: Find and store pids of tasks, which run virtual CPUs of given VM

tracecmd/trace-record.c static char *get_qemu_guest_name(char *arg) · line 3085

Quoted 10 lines · On Mon, 2 Mar 2020 12:13:52 +0200 "Tzvetomir Stoyanov (VMware)"…
tracecmd/trace-record.c +10 -0
+static int read_qemu_guests_pids(char *guest_task, struct guest *guest)
+{
+	struct dirent *entry;
+	char path[PATH_MAX];
+	char *buf = NULL;
+	size_t n = 0;
+	long vcpu;
+	long pid;
+	DIR *dir;
+	FILE *f;
	int ret = 0;

tracecmd/trace-record.c static char *get_qemu_guest_name(char *arg) · line 3092

Quoted 7 lines
tracecmd/trace-record.c +7 -0
+
+	snprintf(path, sizeof(path), "/proc/%s/task", guest_task);
+	dir = opendir(path);
+	if (!dir)
+		return -1;
+
+	while ((entry = readdir(dir))) {
	while (!ret && (entry = readdir(dir)) {
Quoted 8 lines
tracecmd/trace-record.c +8 -0
+		if (!(entry->d_type == DT_DIR && is_digits(entry->d_name)))
+			continue;
+
+		snprintf(path, sizeof(path), "/proc/%s/task/%s/comm",
+			 guest_task, entry->d_name);
+		f = fopen(path, "r");
+		if (!f)
+			continue;

tracecmd/trace-record.c static char *get_qemu_guest_name(char *arg) · line 3110

Quoted 10 lines
tracecmd/trace-record.c +10 -0
+
+		if (getline(&buf, &n, f) >= 0 &&
+		    strncmp(buf, "CPU ", 4) == 0) {
+			vcpu = strtol(buf + 4, NULL, 10);
+			pid = strtol(entry->d_name, NULL, 10);
+			if (vcpu < INT_MAX && pid < INT_MAX &&
+			    vcpu >= 0 && pid >= 0) {
+				if (set_vcpu_pid_mapping(guest, vcpu, pid))
+					return -1;
+			}

Perhaps this should error on else. Also, you don't want a return here, but instead, a break.

				if (...) {
					ret = -1;
					break;
				}

tracecmd/trace-record.c static char *get_qemu_guest_name(char *arg) · line 3116

Quoted 6 lines
tracecmd/trace-record.c +6 -0
+		}
+
+		fclose(f);
+	}
+	free(buf);
+	return 0;
	return ret;

Otherwise you don't close the file descriptor and free buf.

-- Steve

Quoted 39 lines
tracecmd/trace-record.c +22 -0
+}
+
 static void read_qemu_guests(void)
 {
 	static bool initialized;
@@ -3115,6 +3179,10 @@ static void read_qemu_guests(void)
 		if (!is_qemu)
 			goto next;
 
+		if (read_qemu_guests_pids(entry->d_name, &guest))
+			warning("Failed to retrieve VPCU - PID mapping for guest %s",
+					guest.name ? guest.name : "Unknown");
+
 		guests = realloc(guests, (guests_len + 1) * sizeof(*guests));
 		if (!guests)
 			die("Can not allocate guest buffer");
@@ -3160,6 +3228,22 @@ static char *parse_guest_name(char *guest, int *cid, int *port)
 	return guest;
 }
 
+int get_guest_vcpu_pid(unsigned int guest_cid, unsigned int guest_vcpu)
+{
+	int i;
+
+	if (!guests)
+		return -1;
+
+	for (i = 0; i < guests_len; i++) {
+		if (guests[i].cpu_pid < 0 || guest_vcpu >= guests[i].cpu_max)
+			continue;
+		if (guest_cid == guests[i].cid)
+			return guests[i].cpu_pid[guest_vcpu];
+	}
+	return -1;
+}
+
 static void set_prio(int prio)
 {
 	struct sched_param sp;
#3 lore raw

Re: [PATCH v21 01/13] trace-cmd: Find and store pids of tasks, which run virtual CPUs of given VM

In reply to Tzvetomir Stoyanov (VMware)

tracecmd/trace-record.c struct guest { · line 3056

Quoted 12 lines · On Mon, 2 Mar 2020 12:13:52 +0200 "Tzvetomir Stoyanov (VMware)"…
tracecmd/trace-record.c +12 -0
+static int set_vcpu_pid_mapping(struct guest *guest, int cpu, int pid)
+{
+	int *cpu_pid;
+	int i;
+
+	if (cpu >= guest->cpu_max) {
+		cpu_pid = realloc(guest->cpu_pid, (cpu + 1) * sizeof(int));
+		if (!cpu_pid)
+			return -1;
+		/* Handle sparse CPU numbers */
+		for (i = guest->cpu_max; i < cpu; i++)
+			guest->cpu_pid[i] = -1;

That needs to be: cpu_pid[i] = -1; as realloc() may have freed guest->cpu_pid here.

-- Steve

Quoted 6 lines
tracecmd/trace-record.c +6 -0
+		guest->cpu_max = cpu + 1;
+		guest->cpu_pid = cpu_pid;
+	}
+	guest->cpu_pid[cpu] = pid;
+	return 0;
+}
#4 lore raw

PATCH v21 2/13 trace-cmd: Implement new API tracecmd_add_option_v()

In reply to Tzvetomir Stoyanov (VMware)

From: Tzvetomir Stoyanov <[email protected]>

A new tracecmd API tracecmd_add_option_v() is introduced. It adds new option in trace.dat, similar to tracecmd_add_option(), but the option's data is passed as list of buffers. The standard struct iovec is used as input parameter, containing the option's data buffers.

---
 include/trace-cmd/trace-cmd.h    |   5 ++
 include/traceevent/event-parse.h |   1 +
 lib/trace-cmd/trace-output.c     | 117 +++++++++++++++++++++++++++----
 3 files changed, 108 insertions(+), 15 deletions(-)
include/trace-cmd/trace-cmd.h +5 -0
index d1b4a60f..b6393a89 100644
@@ -278,11 +278,16 @@ struct tracecmd_output *tracecmd_create_init_file_override(const char *output_fi
 struct tracecmd_option *tracecmd_add_option(struct tracecmd_output *handle,
 					    unsigned short id, int size,
 					    const void *data);
+struct tracecmd_option *
+tracecmd_add_option_v(struct tracecmd_output *handle,
+		      unsigned short id, const struct iovec *vector, int count);
+
 struct tracecmd_option *tracecmd_add_buffer_option(struct tracecmd_output *handle,
 						   const char *name, int cpus);
 
 int tracecmd_write_cpus(struct tracecmd_output *handle, int cpus);
 int tracecmd_write_options(struct tracecmd_output *handle);
+int tracecmd_append_options(struct tracecmd_output *handle);
 int tracecmd_update_option(struct tracecmd_output *handle,
 			   struct tracecmd_option *option, int size,
 			   const void *data);
include/traceevent/event-parse.h +1 -0
index 52bafa54..cfab6005 100644
@@ -11,6 +11,7 @@
 #include <stdio.h>
 #include <regex.h>
 #include <string.h>
+#include <sys/uio.h>
 
 #include "traceevent/trace-seq.h"
 
lib/trace-cmd/trace-output.c +102 -15

161 lines not on this page · raw message

Signature · 1 line
2.24.1
#5 lore raw

PATCH v21 3/13 trace-cmd: Add new API to generate a unique ID of the tracing session

In reply to Tzvetomir Stoyanov (VMware)

A new libtracecmd API is implemented:

	unsigned long long tracecmd_generate_traceid(void);

At trace-cmd init phase a unique ID, used for the current tracing session.

---
 include/trace-cmd/trace-cmd.h  |  1 +
 lib/trace-cmd/trace-util.c     | 37 ++++++++++++++++++++++++++++++++++
 tracecmd/include/trace-local.h |  1 +
 tracecmd/trace-record.c        |  1 +
 4 files changed, 40 insertions(+)
include/trace-cmd/trace-cmd.h +1 -0
index b6393a89..7519e82a 100644
@@ -415,6 +415,7 @@ void tracecmd_plog_error(const char *fmt, ...);
 int tracecmd_set_logfile(char *logfile);
 
 /* --- System --- */
+unsigned long long tracecmd_generate_traceid(void);
 int tracecmd_count_cpus(void);
 
 /* --- Hack! --- */
lib/trace-cmd/trace-util.c +37 -0
index b8841393..04dc804c 100644
@@ -18,6 +18,8 @@
 #include <sys/mount.h>
 #include <sys/types.h>
 #include <sys/stat.h>
+#include <sys/sysinfo.h>
+#include <time.h>
 
 #include "trace-cmd.h"
 #include "event-utils.h"
@@ -510,3 +512,38 @@ int tracecmd_count_cpus(void)
 
 	return cpus;
 }
+
+#define FNV_64_PRIME 0x100000001b3ULL
+/*
+ * tracecmd_generate_traceid - Generate a unique ID, used to identify
+ *			       the current tracing session
+ *
+ * Returns unique ID
+ */
+unsigned long long tracecmd_generate_traceid(void)
+{
+	unsigned long long hash = 0;
+	unsigned char *ustr;
+	struct sysinfo sinfo;
+	struct timespec ts;
+	char *str = NULL;
+
+	clock_gettime(CLOCK_MONOTONIC_RAW, &ts);
+	sysinfo(&sinfo);
+	asprintf(&str, "%ld %ld %ld %ld %ld %ld %ld %ld %d",
+		 ts.tv_sec, ts.tv_nsec,
+		 sinfo.loads[0], sinfo.loads[1], sinfo.loads[2],
+		 sinfo.freeram, sinfo.sharedram, sinfo.freeswap,
+		 sinfo.procs);
+	if (!str)
+		return 0;
+	ustr = (unsigned char *)str;
+	hash = 0;
+	while (*ustr) {
+		hash ^= (unsigned long long)*ustr++;
+		hash *= FNV_64_PRIME;
+	}
+
+	free(str);
+	return hash;
+}
tracecmd/include/trace-local.h +1 -0
index a5cf0640..6fe22bb0 100644
@@ -185,6 +185,7 @@ struct pid_addr_maps {
 struct buffer_instance {
 	struct buffer_instance	*next;
 	struct tracefs_instance	*tracefs;
+	unsigned long long	trace_id;
 	char			*cpumask;
 	struct event_list	*events;
 	struct event_list	**event_next;
tracecmd/trace-record.c +1 -0
index 3cbc832c..ef05bbc6 100644
@@ -5246,6 +5246,7 @@ void init_top_instance(void)
 		top_instance.tracefs = tracefs_instance_alloc(NULL);
 	top_instance.cpu_count = tracecmd_count_cpus();
 	top_instance.flags = BUFFER_FL_KEEP;
+	top_instance.trace_id = tracecmd_generate_traceid();
 	init_instance(&top_instance);
 }
 
Signature · 1 line
2.24.1
#6 lore raw

PATCH v21 4/13 trace-cmd: Store the session tracing ID in the trace.dat file

In reply to Tzvetomir Stoyanov (VMware)

The ID of the current tracing session is stored in the trace.dat file. A new file option is introduced:
TRACECMD_OPTION_TRACEID
The data is stored in the file as unsigned long long integer
A new libtracecmd API is introduced, to get the ID when the trace.dat file is read:

	unsigned long long tracecmd_get_traceid(struct tracecmd_input *handle);
---
 include/trace-cmd/trace-cmd.h |  2 ++
 lib/trace-cmd/trace-input.c   | 16 ++++++++++++++++
 tracecmd/trace-dump.c         | 11 +++++++++++
 tracecmd/trace-record.c       |  9 +++++++++
 4 files changed, 38 insertions(+)
include/trace-cmd/trace-cmd.h +2 -0
index 7519e82a..6075a37e 100644
@@ -109,6 +109,7 @@ enum {
 	TRACECMD_OPTION_CPUCOUNT,
 	TRACECMD_OPTION_VERSION,
 	TRACECMD_OPTION_PROCMAPS,
+	TRACECMD_OPTION_TRACEID,
 };
 
 enum {
@@ -151,6 +152,7 @@ int tracecmd_copy_headers(struct tracecmd_input *handle, int fd);
 void tracecmd_set_flag(struct tracecmd_input *handle, int flag);
 void tracecmd_clear_flag(struct tracecmd_input *handle, int flag);
 unsigned long tracecmd_get_flags(struct tracecmd_input *handle);
+unsigned long long tracecmd_get_traceid(struct tracecmd_input *handle);
 
 void tracecmd_parse_trace_clock(struct tracecmd_input *handle, char *file, int size);
 
lib/trace-cmd/trace-input.c +16 -0
index 209c1c98..8b5405fb 100644
@@ -82,6 +82,7 @@ struct tracecmd_input {
 	struct tep_plugin_list	*plugin_list;
 	struct tracecmd_input	*parent;
 	unsigned long		flags;
+	unsigned long long	trace_id;
 	int			fd;
 	int			long_size;
 	int			page_size;
@@ -2319,6 +2320,10 @@ static int handle_options(struct tracecmd_input *handle)
 			if (buf[size-1] == '\0')
 				trace_pid_map_load(handle, buf);
 			break;
+		case TRACECMD_OPTION_TRACEID:
+			handle->trace_id = tep_read_number(handle->pevent,
+							   &cpus, 8);
+			break;
 		default:
 			warning("unknown option %d", option);
 			break;
@@ -3422,3 +3427,14 @@ void tracecmd_set_show_data_func(struct tracecmd_input *handle,
 {
 	handle->show_data_func = func;
 }
+
+/**
+ * tracecmd_get_traceid - get the trace id of the session
+ * @handle: input handle for the trace.dat file
+ *
+ * Returns the trace id, written in the trace file
+ */
+unsigned long long tracecmd_get_traceid(struct tracecmd_input *handle)
+{
+	return handle->trace_id;
+}
tracecmd/trace-dump.c +11 -0
index 415e9138..2dfeaa0f 100644
@@ -358,6 +358,14 @@ static void dump_option_int(int fd, int size, char *desc)
 	do_print(OPTIONS, "%d\n", val);
 }
 
+static void dump_option_xlong(int fd, int size, char *desc)
+{
+	long long val;
+
+	do_print(OPTIONS, "\t\t[Option %s, %d bytes]\n", desc, size);
+	read_file_number(fd, &val, size);
+	do_print(OPTIONS, "0x%llX\n", val);
+}
 static void dump_options(int fd)
 {
 	unsigned short option;
@@ -408,6 +416,9 @@ static void dump_options(int fd)
 		case TRACECMD_OPTION_PROCMAPS:
 			dump_option_string(fd, size, "PROCMAPS");
 			break;
+		case TRACECMD_OPTION_TRACEID:
+			dump_option_xlong(fd, size, "TRACEID");
+			break;
 		default:
 			do_print(OPTIONS, " %d %d\t[Unknown option, size - skipping]\n",
 				 option, size);
tracecmd/trace-record.c +9 -0
index ef05bbc6..c3f47a9a 100644
@@ -3902,6 +3902,13 @@ add_pid_maps(struct tracecmd_output *handle, struct buffer_instance *instance)
 	trace_seq_destroy(&s);
 }
 
+static void
+add_trace_id(struct tracecmd_output *handle, struct buffer_instance *instance)
+{
+	tracecmd_add_option(handle, TRACECMD_OPTION_TRACEID,
+			    sizeof(long long), &instance->trace_id);
+}
+
 static void
 add_buffer_stat(struct tracecmd_output *handle, struct buffer_instance *instance)
 {
@@ -4007,6 +4014,8 @@ static void add_options(struct tracecmd_output *handle, struct common_record_con
 	add_option_hooks(handle);
 	add_uname(handle);
 	add_version(handle);
+	if (!no_top_instance())
+		add_trace_id(handle, &top_instance);
 }
 
 static void write_guest_file(struct buffer_instance *instance)
Signature · 1 line
2.24.1
#7 lore raw

PATCH v21 5/13 trace-cmd: Add definitions of htonll() and ntohll()

In reply to Tzvetomir Stoyanov (VMware)

From: Tzvetomir Stoyanov <[email protected]>

Reorganized libtracecmd internal headers:
- Renamed trace-cmd-local.h to trace-write-local.h
This internal header implements static __do_write_check() function, used in trace-input.c, trace-output.c and trace-msg.c files. The header cannot be included in other files, which do not call __do_write_check(). The new name trace-write-local.h reflects more closely the purpose of the file.
- Added trace-cmd-local.h file, to share code inside libtracecmd.
- Added definitions of htonll() and ntohll(), if not already defined, in
trace-cmd-local.h

---
 lib/trace-cmd/include/trace-cmd-local.h   | 45 ++++++-----------------
 lib/trace-cmd/include/trace-write-local.h | 43 ++++++++++++++++++++++
 lib/trace-cmd/trace-input.c               |  2 +-
 lib/trace-cmd/trace-msg.c                 |  2 +-
 lib/trace-cmd/trace-output.c              |  2 +
 5 files changed, 58 insertions(+), 36 deletions(-)
 create mode 100644 lib/trace-cmd/include/trace-write-local.h
lib/trace-cmd/include/trace-cmd-local.h +11 -34
index 09574dbd..95dce66c 100644
@@ -6,10 +6,8 @@
 #ifndef _TRACE_CMD_LOCAL_H
 #define _TRACE_CMD_LOCAL_H
 
-/* Local for trace-input.c and trace-output.c */
-
-#include "trace-cmd.h"
-#include "event-utils.h"
+/* Can be overridden */
+void warning(const char *fmt, ...);
 
 /* trace.dat file format version */
 #define FILE_VERSION 6
@@ -18,36 +16,15 @@
 #define STR(x)	_STR(x)
 #define FILE_VERSION_STRING STR(FILE_VERSION)
 
-static ssize_t __do_write(int fd, const void *data, size_t size)
-{
-	ssize_t tot = 0;
-	ssize_t w;
-
-	do {
-		w = write(fd, data + tot, size - tot);
-		tot += w;
-
-		if (!w)
-			break;
-		if (w < 0)
-			return w;
-	} while (tot != size);
-
-	return tot;
-}
-
-static ssize_t
-__do_write_check(int fd, const void *data, size_t size)
-{
-	ssize_t ret;
-
-	ret = __do_write(fd, data, size);
-	if (ret < 0)
-		return ret;
-	if (ret != size)
-		return -1;
+#ifndef htonll
+# if __BYTE_ORDER == __LITTLE_ENDIAN
+#define htonll(x) __bswap_64(x)
+#define ntohll(x) __bswap_64(x)
+#else
+#define htonll(x) (x)
+#define ntohll(x) (x)
+#endif
+#endif
 
-	return 0;
-}
 
 #endif /* _TRACE_CMD_LOCAL_H */
lib/trace-cmd/include/trace-write-local.h new file +43 -0

43 lines not on this page · raw message

lib/trace-cmd/trace-input.c +1 -1
index 8b5405fb..d1d010de 100644
@@ -15,7 +15,7 @@
 
 #include <linux/time64.h>
 
-#include "trace-cmd-local.h"
+#include "trace-write-local.h"
 #include "trace-local.h"
 #include "kbuffer.h"
 #include "list.h"
lib/trace-cmd/trace-msg.c +1 -1
index 9c8a6908..85a2bae6 100644
@@ -22,7 +22,7 @@
 #include <sys/types.h>
 #include <linux/types.h>
 
-#include "trace-cmd-local.h"
+#include "trace-write-local.h"
 #include "trace-local.h"
 #include "trace-msg.h"
 
lib/trace-cmd/trace-output.c +2 -0
index acdd2d57..4a9a857d 100644
@@ -21,7 +21,9 @@
 #include <glob.h>
 
 #include "tracefs.h"
+#include "trace-cmd.h"
 #include "trace-cmd-local.h"
+#include "trace-write-local.h"
 #include "list.h"
 #include "trace-msg.h"
 
Signature · 1 line
2.24.1
#8 lore raw

PATCH v21 6/13 trace-cmd: Exchange tracing IDs between host and guest

In reply to Tzvetomir Stoyanov (VMware)

Extend the trace request and trace reply messages, to include also the IDs of host and guest tracing sessions. Those IDs are used to unambiguously match the tracing sessions, when reading trace.dat files.

---
 include/trace-cmd/trace-cmd.h  | 12 ++++++++----
 lib/trace-cmd/trace-msg.c      | 31 ++++++++++++++++++++++---------
 tracecmd/include/trace-local.h |  3 ++-
 tracecmd/trace-agent.c         | 10 +++++++---
 tracecmd/trace-record.c        | 10 +++++++---
 5 files changed, 46 insertions(+), 20 deletions(-)
include/trace-cmd/trace-cmd.h +8 -4
index 6075a37e..55abd489 100644
@@ -370,16 +370,20 @@ bool tracecmd_msg_done(struct tracecmd_msg_handle *msg_handle);
 void tracecmd_msg_set_done(struct tracecmd_msg_handle *msg_handle);
 
 int tracecmd_msg_send_trace_req(struct tracecmd_msg_handle *msg_handle,
-				int argc, char **argv, bool use_fifos);
+				int argc, char **argv, bool use_fifos,
+				unsigned long long trace_id);
 int tracecmd_msg_recv_trace_req(struct tracecmd_msg_handle *msg_handle,
-				int *argc, char ***argv, bool *use_fifos);
+				int *argc, char ***argv, bool *use_fifos,
+				unsigned long long *trace_id);
 
 int tracecmd_msg_send_trace_resp(struct tracecmd_msg_handle *msg_handle,
 				 int nr_cpus, int page_size,
-				 unsigned int *ports, bool use_fifos);
+				 unsigned int *ports, bool use_fifos,
+				 unsigned long long trace_id);
 int tracecmd_msg_recv_trace_resp(struct tracecmd_msg_handle *msg_handle,
 				 int *nr_cpus, int *page_size,
-				 unsigned int **ports, bool *use_fifos);
+				 unsigned int **ports, bool *use_fifos,
+				 unsigned long long *trace_id);
 
 /* --- Plugin handling --- */
 extern struct tep_plugin_option trace_ftrace_options[];
lib/trace-cmd/trace-msg.c +22 -9
index 85a2bae6..d6a68ac3 100644
@@ -23,6 +23,7 @@
 #include <linux/types.h>
 
 #include "trace-write-local.h"
+#include "trace-cmd-local.h"
 #include "trace-local.h"
 #include "trace-msg.h"
 
@@ -63,12 +64,14 @@ struct tracecmd_msg_rinit {
 struct tracecmd_msg_trace_req {
 	be32 flags;
 	be32 argc;
+	u64 trace_id;
 } __attribute__((packed));
 
 struct tracecmd_msg_trace_resp {
 	be32 flags;
 	be32 cpus;
 	be32 page_size;
+	u64 trace_id;
 } __attribute__((packed));
 
 struct tracecmd_msg_header {
@@ -811,7 +814,8 @@ int tracecmd_msg_wait_close_resp(struct tracecmd_msg_handle *msg_handle)
 	return tracecmd_msg_wait_for_cmd(msg_handle, MSG_CLOSE_RESP);
 }
 
-static int make_trace_req(struct tracecmd_msg *msg, int argc, char **argv, bool use_fifos)
+static int make_trace_req(struct tracecmd_msg *msg, int argc, char **argv,
+			  bool use_fifos, unsigned long long trace_id)
 {
 	size_t args_size = 0;
 	char *p;
@@ -823,6 +827,7 @@ static int make_trace_req(struct tracecmd_msg *msg, int argc, char **argv, bool
 	msg->hdr.size = htonl(ntohl(msg->hdr.size) + args_size);
 	msg->trace_req.flags = use_fifos ? htonl(MSG_TRACE_USE_FIFOS) : htonl(0);
 	msg->trace_req.argc = htonl(argc);
+	msg->trace_req.trace_id = htonll(trace_id);
 	msg->buf = calloc(args_size, 1);
 	if (!msg->buf)
 		return -ENOMEM;
@@ -835,13 +840,14 @@ static int make_trace_req(struct tracecmd_msg *msg, int argc, char **argv, bool
 }
 
 int tracecmd_msg_send_trace_req(struct tracecmd_msg_handle *msg_handle,
-				int argc, char **argv, bool use_fifos)
+				int argc, char **argv, bool use_fifos,
+				unsigned long long trace_id)
 {
 	struct tracecmd_msg msg;
 	int ret;
 
 	tracecmd_msg_init(MSG_TRACE_REQ, &msg);
-	ret = make_trace_req(&msg, argc, argv, use_fifos);
+	ret = make_trace_req(&msg, argc, argv, use_fifos, trace_id);
 	if (ret < 0)
 		return ret;
 
@@ -854,7 +860,8 @@ int tracecmd_msg_send_trace_req(struct tracecmd_msg_handle *msg_handle,
   *     free(argv);
   */
 int tracecmd_msg_recv_trace_req(struct tracecmd_msg_handle *msg_handle,
-				int *argc, char ***argv, bool *use_fifos)
+				int *argc, char ***argv, bool *use_fifos,
+				unsigned long long *trace_id)
 {
 	struct tracecmd_msg msg;
 	char *p, *buf_end, **args;
@@ -901,7 +908,7 @@ int tracecmd_msg_recv_trace_req(struct tracecmd_msg_handle *msg_handle,
 	*argc = nr_args;
 	*argv = args;
 	*use_fifos = ntohl(msg.trace_req.flags) & MSG_TRACE_USE_FIFOS;
-
+	*trace_id = ntohll(msg.trace_req.trace_id);
 	/*
 	 * On success we're passing msg.buf to the caller through argv[0] so we
 	 * reset it here before calling msg_free().
@@ -921,7 +928,8 @@ out:
 }
 
 static int make_trace_resp(struct tracecmd_msg *msg, int page_size, int nr_cpus,
-			   unsigned int *ports, bool use_fifos)
+			   unsigned int *ports, bool use_fifos,
+			   unsigned long long trace_id)
 {
 	int data_size;
 
@@ -935,19 +943,22 @@ static int make_trace_resp(struct tracecmd_msg *msg, int page_size, int nr_cpus,
 	msg->trace_resp.flags = use_fifos ? htonl(MSG_TRACE_USE_FIFOS) : htonl(0);
 	msg->trace_resp.cpus = htonl(nr_cpus);
 	msg->trace_resp.page_size = htonl(page_size);
+	msg->trace_resp.trace_id = htonll(trace_id);
 
 	return 0;
 }
 
 int tracecmd_msg_send_trace_resp(struct tracecmd_msg_handle *msg_handle,
 				 int nr_cpus, int page_size,
-				 unsigned int *ports, bool use_fifos)
+				 unsigned int *ports, bool use_fifos,
+				 unsigned long long trace_id)
 {
 	struct tracecmd_msg msg;
 	int ret;
 
 	tracecmd_msg_init(MSG_TRACE_RESP, &msg);
-	ret = make_trace_resp(&msg, page_size, nr_cpus, ports, use_fifos);
+	ret = make_trace_resp(&msg, page_size, nr_cpus, ports,
+			      use_fifos, trace_id);
 	if (ret < 0)
 		return ret;
 
@@ -956,7 +967,8 @@ int tracecmd_msg_send_trace_resp(struct tracecmd_msg_handle *msg_handle,
 
 int tracecmd_msg_recv_trace_resp(struct tracecmd_msg_handle *msg_handle,
 				 int *nr_cpus, int *page_size,
-				 unsigned int **ports, bool *use_fifos)
+				 unsigned int **ports, bool *use_fifos,
+				 unsigned long long *trace_id)
 {
 	struct tracecmd_msg msg;
 	char *p, *buf_end;
@@ -981,6 +993,7 @@ int tracecmd_msg_recv_trace_resp(struct tracecmd_msg_handle *msg_handle,
 	*use_fifos = ntohl(msg.trace_resp.flags) & MSG_TRACE_USE_FIFOS;
 	*nr_cpus = ntohl(msg.trace_resp.cpus);
 	*page_size = ntohl(msg.trace_resp.page_size);
+	*trace_id = ntohll(msg.trace_resp.trace_id);
 	*ports = calloc(*nr_cpus, sizeof(**ports));
 	if (!*ports) {
 		ret = -ENOMEM;
tracecmd/include/trace-local.h +2 -1
index 6fe22bb0..4f2b5541 100644
@@ -101,7 +101,8 @@ void trace_dump(int argc, char **argv);
 
 int trace_record_agent(struct tracecmd_msg_handle *msg_handle,
 		       int cpus, int *fds,
-		       int argc, char **argv, bool use_fifos);
+		       int argc, char **argv, bool use_fifos,
+		       unsigned long long trace_id);
 
 struct hook_list;
 
tracecmd/trace-agent.c +7 -3
index 1c6e0a3a..52d27195 100644
@@ -128,6 +128,7 @@ cleanup:
 static void agent_handle(int sd, int nr_cpus, int page_size)
 {
 	struct tracecmd_msg_handle *msg_handle;
+	unsigned long long trace_id;
 	unsigned int *ports;
 	char **argv = NULL;
 	int argc = 0;
@@ -144,7 +145,8 @@ static void agent_handle(int sd, int nr_cpus, int page_size)
 	if (!msg_handle)
 		die("Failed to allocate message handle");
 
-	ret = tracecmd_msg_recv_trace_req(msg_handle, &argc, &argv, &use_fifos);
+	ret = tracecmd_msg_recv_trace_req(msg_handle, &argc, &argv,
+					  &use_fifos, &trace_id);
 	if (ret < 0)
 		die("Failed to receive trace request");
 
@@ -154,12 +156,14 @@ static void agent_handle(int sd, int nr_cpus, int page_size)
 	if (!use_fifos)
 		make_vsocks(nr_cpus, fds, ports);
 
+	trace_id = tracecmd_generate_traceid();
 	ret = tracecmd_msg_send_trace_resp(msg_handle, nr_cpus, page_size,
-					   ports, use_fifos);
+					   ports, use_fifos, trace_id);
 	if (ret < 0)
 		die("Failed to send trace response");
 
-	trace_record_agent(msg_handle, nr_cpus, fds, argc, argv, use_fifos);
+	trace_record_agent(msg_handle, nr_cpus, fds, argc, argv,
+			   use_fifos, trace_id);
 
 	free(argv[0]);
 	free(argv);
tracecmd/trace-record.c +7 -3
index c3f47a9a..b55f407c 100644
@@ -3693,12 +3693,14 @@ static void connect_to_agent(struct buffer_instance *instance)
 		die("Failed to allocate message handle");
 
 	ret = tracecmd_msg_send_trace_req(msg_handle, instance->argc,
-					  instance->argv, use_fifos);
+					  instance->argv, use_fifos,
+					  top_instance.trace_id);
 	if (ret < 0)
 		die("Failed to send trace request");
 
 	ret = tracecmd_msg_recv_trace_resp(msg_handle, &nr_cpus, &page_size,
-					   &ports, &use_fifos);
+					   &ports, &use_fifos,
+					   &instance->trace_id);
 	if (ret < 0)
 		die("Failed to receive trace response");
 
@@ -6280,7 +6282,8 @@ void trace_record(int argc, char **argv)
 int trace_record_agent(struct tracecmd_msg_handle *msg_handle,
 		       int cpus, int *fds,
 		       int argc, char **argv,
-		       bool use_fifos)
+		       bool use_fifos,
+		       unsigned long long trace_id)
 {
 	struct common_record_context ctx;
 	char **argv_plus;
@@ -6310,6 +6313,7 @@ int trace_record_agent(struct tracecmd_msg_handle *msg_handle,
 	ctx.instance->flags |= BUFFER_FL_AGENT;
 	ctx.instance->msg_handle = msg_handle;
 	msg_handle->version = V3_PROTOCOL;
+	top_instance.trace_id = trace_id;
 	record_trace(argc, argv, &ctx);
 
 	free(argv_plus);
Signature · 1 line
2.24.1
#9 lore raw

PATCH v21 7/13 trace-cmd: Implement new option in trace.dat file: TRACECMD_OPTION_TIME_SHIFT

In reply to Tzvetomir Stoyanov (VMware)

From: Tzvetomir Stoyanov <[email protected]>

The TRACECMD_OPTION_TIME_SHIFT is used when synchronizing trace time stamps between two trace.dat files. It contains multiple long long (time, offset) pairs, describing time stamps _offset_, measured in the given local _time_. The content of the option buffer is:
8 bytes - long long integer, ID of the tracing session
4 bytes - integer, count of timestamp offsets
long long array of size _count_, local time in which the offset is measured long long array of size _count_, offset of the time stamps

---
 include/trace-cmd/trace-cmd.h |   3 +
 lib/trace-cmd/trace-input.c   | 186 +++++++++++++++++++++++++++++++++-
 tracecmd/trace-dump.c         |  49 +++++++++
 3 files changed, 236 insertions(+), 2 deletions(-)
include/trace-cmd/trace-cmd.h +3 -0
index 55abd489..ac46637e 100644
@@ -110,6 +110,7 @@ enum {
 	TRACECMD_OPTION_VERSION,
 	TRACECMD_OPTION_PROCMAPS,
 	TRACECMD_OPTION_TRACEID,
+	TRACECMD_OPTION_TIME_SHIFT,
 };
 
 enum {
@@ -153,6 +154,8 @@ void tracecmd_set_flag(struct tracecmd_input *handle, int flag);
 void tracecmd_clear_flag(struct tracecmd_input *handle, int flag);
 unsigned long tracecmd_get_flags(struct tracecmd_input *handle);
 unsigned long long tracecmd_get_traceid(struct tracecmd_input *handle);
+unsigned long long tracecmd_get_tsync_peer(struct tracecmd_input *handle);
+int tracecmd_enable_tsync(struct tracecmd_input *handle, bool enable);
 
 void tracecmd_parse_trace_clock(struct tracecmd_input *handle, char *file, int size);
 
lib/trace-cmd/trace-input.c +184 -2

249 lines not on this page · raw message

tracecmd/trace-dump.c +49 -0

61 lines not on this page · raw message

Signature · 1 line
2.24.1
#10 lore raw

PATCH v21 8/13 trace-cmd: Add guest information in host's trace.dat file

In reply to Tzvetomir Stoyanov (VMware)

New trace.dat option is introduced: TRACECMD_OPTION_GUEST. Written in the host's trace.dat file, it contains information about guests, traced at the same time: guest trace ID, number of VCPUs and PIDs of the host tasks, running those VCPU. The data is stored in the file as:
Guest name, null terminated string
long long (8 bytes) trace-id
int (4 bytes) number of guest CPUs
array of size number of guest CPUs:
int (4 bytes) Guest CPU id
int (4 bytes) Host PID, running the guest CPU

---
 include/trace-cmd/trace-cmd.h |   5 ++
 lib/trace-cmd/trace-input.c   | 128 ++++++++++++++++++++++++++++++++++
 tracecmd/trace-dump.c         |  62 ++++++++++++++++
 tracecmd/trace-record.c       |  61 ++++++++++++++++
 4 files changed, 256 insertions(+)
include/trace-cmd/trace-cmd.h +5 -0
index ac46637e..0375f500 100644
@@ -111,6 +111,7 @@ enum {
 	TRACECMD_OPTION_PROCMAPS,
 	TRACECMD_OPTION_TRACEID,
 	TRACECMD_OPTION_TIME_SHIFT,
+	TRACECMD_OPTION_GUEST,
 };
 
 enum {
@@ -154,6 +155,10 @@ void tracecmd_set_flag(struct tracecmd_input *handle, int flag);
 void tracecmd_clear_flag(struct tracecmd_input *handle, int flag);
 unsigned long tracecmd_get_flags(struct tracecmd_input *handle);
 unsigned long long tracecmd_get_traceid(struct tracecmd_input *handle);
+int tracecmd_get_guest_cpumap(struct tracecmd_input *handle,
+			      unsigned long long trace_id,
+			      char **name,
+			      int *vcpu_count, int **cpu_pid);
 unsigned long long tracecmd_get_tsync_peer(struct tracecmd_input *handle);
 int tracecmd_enable_tsync(struct tracecmd_input *handle, bool enable);
 
lib/trace-cmd/trace-input.c +128 -0

164 lines not on this page · raw message

tracecmd/trace-dump.c +62 -0

74 lines not on this page · raw message

tracecmd/trace-record.c +61 -0

79 lines not on this page · raw message

Signature · 1 line
2.24.1
#11 lore raw

PATCH v21 9/13 trace-cmd: Add host trace clock as guest trace argument

In reply to Tzvetomir Stoyanov (VMware)

When tracing host and guest machines, both should use the same tracing clock for event timestamps. If a clock is specified as host tracing argument, with option "-C clock_name", the same is injected as guest tracing argument. If the user wants to use different tracing clocks, it can specify it using "-C clock_name" as guest tracing argument. In that case, the one specified by the user has higher priority.

---
 tracecmd/include/trace-local.h |  1 +
 tracecmd/trace-record.c        | 21 +++++++++++++++++++++
 2 files changed, 22 insertions(+)
tracecmd/include/trace-local.h +1 -0
index 4f2b5541..dc934f28 100644
@@ -167,6 +167,7 @@ enum buffer_instance_flags {
 	BUFFER_FL_PROFILE	= 1 << 1,
 	BUFFER_FL_GUEST		= 1 << 2,
 	BUFFER_FL_AGENT		= 1 << 3,
+	BUFFER_FL_HAS_CLOCK	= 1 << 4,
 };
 
 struct func_list {
tracecmd/trace-record.c +21 -0
index 15cae9be..e0b18044 100644
@@ -5598,6 +5598,8 @@ static void parse_record_options(int argc,
 	char *sav;
 	int name_counter = 0;
 	int neg_event = 0;
+	struct buffer_instance *instance;
+	bool guest_sync_set = false;
 
 	init_common_record_context(ctx, curr_cmd);
 
@@ -5641,6 +5643,8 @@ static void parse_record_options(int argc,
 		 */
 		if (c != 'B' && c != 'A' && is_guest(ctx->instance)) {
 			add_arg(ctx->instance, c, opts, long_options, optarg);
+			if (c == 'C')
+				ctx->instance->flags |= BUFFER_FL_HAS_CLOCK;
 			continue;
 		}
 
@@ -5755,6 +5759,9 @@ static void parse_record_options(int argc,
 			break;
 		case 'C':
 			ctx->instance->clock = optarg;
+			ctx->instance->flags |= BUFFER_FL_HAS_CLOCK;
+			if (is_top_instance(ctx->instance))
+				guest_sync_set = true;
 			break;
 		case 'v':
 			neg_event = 1;
@@ -5979,6 +5986,20 @@ static void parse_record_options(int argc,
 				add_argv(instance, "--date", true);
 		}
 	}
+	if (guest_sync_set) {
+	/* If -C is specified, prepend clock to all guest VM flags */
+		for_all_instances(instance) {
+			if (top_instance.clock) {
+				if (is_guest(instance) &&
+				    !(instance->flags & BUFFER_FL_HAS_CLOCK)) {
+					add_argv(instance,
+						 (char *)top_instance.clock,
+						 true);
+					add_argv(instance, "-C", true);
+				}
+			}
+		}
+	}
 
 	if (!ctx->filtered && ctx->instance->filter_mod)
 		add_func(&ctx->instance->filter_funcs,
Signature · 1 line
2.24.1
#12 lore raw

PATCH v21 10/13 trace-cmd: Refactor few trace-cmd internal functions.

In reply to Tzvetomir Stoyanov (VMware)

In order to reuse code inside trace-cmd application context, few functions are made non static and "trace_" is prepended to their names:

	int trace_make_vsock(unsigned int port);
	int trace_get_vsock_port(int sd, unsigned int *port);
	int trace_open_vsock(unsigned int cid, unsigned int port);
	char *trace_get_guest_file(const char *file, const char *guest);
---
 tracecmd/include/trace-local.h |  6 ++++++
 tracecmd/trace-agent.c         | 10 +++++-----
 tracecmd/trace-record.c        | 18 +++++++++---------
 3 files changed, 20 insertions(+), 14 deletions(-)
tracecmd/include/trace-local.h +6 -0
index dc934f28..95d0026b 100644
@@ -263,6 +263,12 @@ void tracecmd_disable_tracing(void);
 void tracecmd_enable_tracing(void);
 void tracecmd_stat_cpu(struct trace_seq *s, int cpu);
 
+int trace_make_vsock(unsigned int port);
+int trace_get_vsock_port(int sd, unsigned int *port);
+int trace_open_vsock(unsigned int cid, unsigned int port);
+
+char *trace_get_guest_file(const char *file, const char *guest);
+
 /* No longer in event-utils.h */
 void __noreturn die(const char *fmt, ...); /* Can be overriden */
 void *malloc_or_die(unsigned int size); /* Can be overridden */
tracecmd/trace-agent.c +5 -5
index 52d27195..a8ef0852 100644
@@ -40,7 +40,7 @@ static int get_local_cid(unsigned int *cid)
 	return ret;
 }
 
-static int make_vsock(unsigned int port)
+int trace_make_vsock(unsigned int port)
 {
 	struct sockaddr_vm addr = {
 		.svm_family = AF_VSOCK,
@@ -64,7 +64,7 @@ static int make_vsock(unsigned int port)
 	return sd;
 }
 
-static int get_vsock_port(int sd, unsigned int *port)
+int trace_get_vsock_port(int sd, unsigned int *port)
 {
 	struct sockaddr_vm addr;
 	socklen_t addr_len = sizeof(addr);
@@ -87,11 +87,11 @@ static void make_vsocks(int nr, int *fds, unsigned int *ports)
 	int i, fd, ret;
 
 	for (i = 0; i < nr; i++) {
-		fd = make_vsock(VMADDR_PORT_ANY);
+		fd = trace_make_vsock(VMADDR_PORT_ANY);
 		if (fd < 0)
 			die("Failed to open vsocket");
 
-		ret = get_vsock_port(fd, &port);
+		ret = trace_get_vsock_port(fd, &port);
 		if (ret < 0)
 			die("Failed to get vsocket address");
 
@@ -210,7 +210,7 @@ static void agent_serve(unsigned int port)
 	nr_cpus = tracecmd_count_cpus();
 	page_size = getpagesize();
 
-	sd = make_vsock(port);
+	sd = trace_make_vsock(port);
 	if (sd < 0)
 		die("Failed to open vsocket");
 
tracecmd/trace-record.c +9 -9
index e0b18044..174a9bdd 100644
@@ -537,7 +537,7 @@ static char *get_temp_file(struct buffer_instance *instance, int cpu)
 	return file;
 }
 
-static char *get_guest_file(const char *file, const char *guest)
+char *trace_get_guest_file(const char *file, const char *guest)
 {
 	const char *p;
 	char *out = NULL;
@@ -2930,7 +2930,7 @@ static int connect_port(const char *host, unsigned int port)
 }
 
 #ifdef VSOCK
-static int open_vsock(unsigned int cid, unsigned int port)
+int trace_open_vsock(unsigned int cid, unsigned int port)
 {
 	struct sockaddr_vm addr = {
 		.svm_family = AF_VSOCK,
@@ -2993,7 +2993,7 @@ static bool can_splice_read_vsock(void)
 }
 
 #else
-static inline int open_vsock(unsigned int cid, unsigned int port)
+int trace_open_vsock(unsigned int cid, unsigned int port)
 {
 	die("vsock is not supported");
 	return -1;
@@ -3304,7 +3304,7 @@ create_recorder_instance(struct buffer_instance *instance, const char *file, int
 		if (instance->use_fifos)
 			fd = instance->fds[cpu];
 		else
-			fd = open_vsock(instance->cid, instance->client_ports[cpu]);
+			fd = trace_open_vsock(instance->cid, instance->client_ports[cpu]);
 		if (fd < 0)
 			die("Failed to connect to agent");
 
@@ -3696,7 +3696,7 @@ static void connect_to_agent(struct buffer_instance *instance)
 		use_fifos = nr_fifos > 0;
 	}
 
-	sd = open_vsock(instance->cid, instance->port);
+	sd = trace_open_vsock(instance->cid, instance->port);
 	if (sd < 0)
 		die("Failed to connect to vsocket @%u:%u",
 		    instance->cid, instance->port);
@@ -3747,8 +3747,8 @@ static void setup_guest(struct buffer_instance *instance)
 	int fd;
 
 	/* Create a place to store the guest meta data */
-	file = get_guest_file(output_file,
-			      tracefs_instance_get_name(instance->tracefs));
+	file = trace_get_guest_file(output_file,
+				    tracefs_instance_get_name(instance->tracefs));
 	if (!file)
 		die("Failed to allocate memory");
 
@@ -4084,8 +4084,8 @@ static void write_guest_file(struct buffer_instance *instance)
 	char **temp_files;
 	int i, fd;
 
-	file = get_guest_file(output_file,
-			      tracefs_instance_get_name(instance->tracefs));
+	file = trace_get_guest_file(output_file,
+				    tracefs_instance_get_name(instance->tracefs));
 	if (!file)
 		die("Failed to allocate memory");
 
Signature · 1 line
2.24.1
#13 lore raw

PATCH v21 11/13 trace-cmd: Basic infrastructure for host - guest timestamp synchronization

In reply to Tzvetomir Stoyanov (VMware)

The infrastructure for host - guest timestamp synchronization is divided in two parts:
- logic in libtracecmd
- logic in trace-cmd application

The libtracecmd is extended with new trace message, MSG_TIME_SYNC, used to exchange time synch information between host and guest. The trace request and response messages are extended to handle the time synch negotiation. The logic is implemented in trace-timesync.c file:
- Register / unregister time sync protocols.
- Choosing what protocol will be used for a tracing session.
- Init common context, used by all protocols - a ftrace instance, vsockets.
- Invoke protocol routines to perform time synchronization.
- Store calculated offsets in an array.

The trace-cmd application is extended to perform timestamp synchronization.
The main logic is in trace-tsync.c file
- Negotiate what time synch algorithm will be used for the tracing session.
- Run pthreads, to do continuous timestamp synchronization, during the trace.
- Store calculated offsets in guest's trace.dat file, using TRACECMD_OPTION_TIME_SHIFT option
- A new trace-cmd option is added, to control the timestamp synchronization:
--tsync-interval: set the time sync loop interval, in ms. The default
value is 0 - timestamp synchronization is performed twice, before and after the trace. If a negative number is specified, the timestamp synchronization is disabled.

---
 Documentation/trace-cmd-record.1.txt      |   7 +
 include/trace-cmd/trace-cmd.h             |  58 ++-
 lib/trace-cmd/Makefile                    |   1 +
 lib/trace-cmd/include/trace-tsync-local.h |  38 ++
 lib/trace-cmd/trace-msg.c                 | 395 ++++++++++++++---
 lib/trace-cmd/trace-timesync.c            | 498 ++++++++++++++++++++++
 tracecmd/Makefile                         |   3 +-
 tracecmd/include/trace-local.h            |  11 +
 tracecmd/trace-agent.c                    |  39 +-
 tracecmd/trace-record.c                   |  40 +-
 tracecmd/trace-tsync.c                    | 272 ++++++++++++
 tracecmd/trace-usage.c                    |   4 +
 12 files changed, 1299 insertions(+), 67 deletions(-)
 create mode 100644 lib/trace-cmd/include/trace-tsync-local.h
 create mode 100644 lib/trace-cmd/trace-timesync.c
 create mode 100644 tracecmd/trace-tsync.c
Documentation/trace-cmd-record.1.txt +7 -0
index 0d75e43d..f5ecad5c 100644
@@ -338,6 +338,13 @@ OPTIONS
     the offset will also be in nanoseconds even if the displayed units are
     in microseconds.
 
+*--tsync-interval*::
+    Set the loop interval, in ms, for timestamps synchronization with guests:
+        If a negative number is specified, timestamps synchronization is disabled
+        If 0 is specified, no loop is performed - timestamps offset is calculated only twice,"
+        at the beginning and at the end of the trace.
+        Timestamps synchronization with guests works only if there is support for VSOCK.\n"
+
 *--stderr*::
     Have output go to stderr instead of stdout, but the output of the command
     executed will not be changed. This is useful if you want to monitor the
include/trace-cmd/trace-cmd.h +54 -4

73 lines not on this page · raw message

lib/trace-cmd/Makefile +1 -0
index 29c36ca1..ab7440ac 100644
@@ -16,6 +16,7 @@ OBJS += trace-util.o
 OBJS += trace-filter-hash.o
 OBJS += trace-msg.o
 OBJS += trace-plugin.o
+OBJS += trace-timesync.o
 
 # Additional util objects
 OBJS += trace-blk-hack.o
lib/trace-cmd/include/trace-tsync-local.h new file +38 -0
new file mode 100644
index 00000000..1de9d5e5
@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: LGPL-2.1 */
+/*
+ * Copyright (C) 2019, VMware, Tzvetomir Stoyanov <[email protected]>
+ *
+ */
+#ifndef _TRACE_TSYNC_LOCAL_H
+#define _TRACE_TSYNC_LOCAL_H
+
+#include <stdbool.h>
+
+struct clock_sync_context {
+	void				*proto_data;	/* time sync protocol specific data */
+	bool				is_server;	/* server side time sync role */
+	struct tracefs_instance		*instance;	/* ftrace buffer, used for time sync events */
+
+	/* Arrays with calculated time offsets at given time */
+	int				sync_size;	/* Allocated size of sync_ts and sync_offsets */
+	int				sync_count;	/* Number of elements in sync_ts and sync_offsets */
+	long long			*sync_ts;
+	long long			*sync_offsets;
+
+	/* Identifiers of local and remote time sync peers: cid and port */
+	unsigned int			local_cid;
+	unsigned int			local_port;
+	unsigned int			remote_cid;
+	unsigned int			remote_port;
+};
+
+int tracecmd_tsync_proto_register(unsigned int proto_id, int weight,
+				int (*init)(struct tracecmd_time_sync *),
+				int (*free)(struct tracecmd_time_sync *),
+				int (*calc)(struct tracecmd_time_sync *,
+					    long long *, long long *));
+int tracecmd_tsync_proto_unregister(unsigned int proto_id);
+
+int ptp_clock_sync_register(void);
+
+#endif /* _TRACE_TSYNC_LOCAL_H */
lib/trace-cmd/trace-msg.c +340 -55

529 lines not on this page · raw message

lib/trace-cmd/trace-timesync.c new file +498 -0

498 lines not on this page · raw message

tracecmd/Makefile +2 -1
index 0976341c..17e67600 100644
@@ -31,6 +31,7 @@ TRACE_CMD_OBJS += trace-show.o
 TRACE_CMD_OBJS += trace-list.o
 TRACE_CMD_OBJS += trace-usage.o
 TRACE_CMD_OBJS += trace-dump.o
+TRACE_CMD_OBJS += trace-tsync.o
 
 ifeq ($(VSOCK_DEFINED), 1)
 TRACE_CMD_OBJS += trace-agent.o
@@ -43,7 +44,7 @@ all_objs := $(sort $(ALL_OBJS))
 all_deps := $(all_objs:$(bdir)/%.o=$(bdir)/.%.d)
 
 CONFIG_INCLUDES =
-CONFIG_LIBS	= -lrt
+CONFIG_LIBS	= -lrt -lpthread
 CONFIG_FLAGS	=
 
 all: $(TARGETS)
tracecmd/include/trace-local.h +11 -0
index 95d0026b..49c52b17 100644
@@ -229,6 +229,10 @@ struct buffer_instance {
 	unsigned int		port;
 	int			*fds;
 	bool			use_fifos;
+
+	pthread_t		tsync_thread;
+	bool			tsync_thread_running;
+	struct tracecmd_time_sync tsync;
 };
 
 void init_top_instance(void);
@@ -263,6 +267,13 @@ void tracecmd_disable_tracing(void);
 void tracecmd_enable_tracing(void);
 void tracecmd_stat_cpu(struct trace_seq *s, int cpu);
 
+int tracecmd_host_tsync(struct buffer_instance *instance,
+			 unsigned int tsync_port);
+void tracecmd_host_tsync_complete(struct buffer_instance *instance);
+unsigned int tracecmd_guest_tsync(char *tsync_protos,
+				  unsigned int tsync_protos_size, char *clock,
+				  unsigned int *tsync_port, pthread_t *thr_id);
+
 int trace_make_vsock(unsigned int port);
 int trace_get_vsock_port(int sd, unsigned int *port);
 int trace_open_vsock(unsigned int cid, unsigned int port);
tracecmd/trace-agent.c +36 -3
index a8ef0852..b5816966 100644
@@ -19,6 +19,7 @@
 #include <sys/wait.h>
 #include <unistd.h>
 #include <linux/vm_sockets.h>
+#include <pthread.h>
 
 #include "trace-local.h"
 #include "trace-msg.h"
@@ -125,11 +126,30 @@ cleanup:
 	return ret;
 }
 
+static char *get_clock(int argc, char **argv)
+{
+	int i;
+
+	if (!argc || !argv)
+		return NULL;
+
+	for (i = 0; i < argc - 1; i++) {
+		if (!strcmp("-C", argv[i]))
+			return argv[i+1];
+	}
+	return NULL;
+}
+
 static void agent_handle(int sd, int nr_cpus, int page_size)
 {
 	struct tracecmd_msg_handle *msg_handle;
+	unsigned int tsync_protos_size = 0;
+	unsigned int tsync_proto = 0;
 	unsigned long long trace_id;
+	unsigned int tsync_port = 0;
+	char *tsync_protos = NULL;
 	unsigned int *ports;
+	pthread_t sync_thr;
 	char **argv = NULL;
 	int argc = 0;
 	bool use_fifos;
@@ -146,7 +166,8 @@ static void agent_handle(int sd, int nr_cpus, int page_size)
 		die("Failed to allocate message handle");
 
 	ret = tracecmd_msg_recv_trace_req(msg_handle, &argc, &argv,
-					  &use_fifos, &trace_id);
+					  &use_fifos, &trace_id,
+					  &tsync_protos, &tsync_protos_size);
 	if (ret < 0)
 		die("Failed to receive trace request");
 
@@ -155,16 +176,28 @@ static void agent_handle(int sd, int nr_cpus, int page_size)
 
 	if (!use_fifos)
 		make_vsocks(nr_cpus, fds, ports);
-
+	if (tsync_protos) {
+		tsync_proto = tracecmd_guest_tsync(tsync_protos,
+						   tsync_protos_size,
+						   get_clock(argc, argv),
+						   &tsync_port, &sync_thr);
+		if (!tsync_proto)
+			warning("Failed to negotiate timestamps synchronization with the host");
+	}
 	trace_id = tracecmd_generate_traceid();
 	ret = tracecmd_msg_send_trace_resp(msg_handle, nr_cpus, page_size,
-					   ports, use_fifos, trace_id);
+					   ports, use_fifos, trace_id,
+					   tsync_proto, tsync_port);
 	if (ret < 0)
 		die("Failed to send trace response");
 
 	trace_record_agent(msg_handle, nr_cpus, fds, argc, argv,
 			   use_fifos, trace_id);
 
+	if (tsync_proto)
+		pthread_join(sync_thr, NULL);
+
+	free(tsync_protos);
 	free(argv[0]);
 	free(argv);
 	free(ports);
tracecmd/trace-record.c +36 -4
index 174a9bdd..0a287f24 100644
@@ -672,6 +672,11 @@ static void tell_guests_to_stop(void)
 			tracecmd_msg_send_close_msg(instance->msg_handle);
 	}
 
+	for_all_instances(instance) {
+		if (is_guest(instance))
+			tracecmd_host_tsync_complete(instance);
+	}
+
 	/* Wait for guests to acknowledge */
 	for_all_instances(instance) {
 		if (is_guest(instance)) {
@@ -3685,6 +3690,10 @@ static void connect_to_agent(struct buffer_instance *instance)
 {
 	struct tracecmd_msg_handle *msg_handle;
 	int sd, ret, nr_fifos, nr_cpus, page_size;
+	unsigned int tsync_protos_reply = 0;
+	unsigned int tsync_port = 0;
+	char *protos = NULL;
+	int protos_count = 0;
 	unsigned int *ports;
 	int i, *fds = NULL;
 	bool use_fifos = false;
@@ -3705,17 +3714,32 @@ static void connect_to_agent(struct buffer_instance *instance)
 	if (!msg_handle)
 		die("Failed to allocate message handle");
 
+	if (instance->tsync.loop_interval >= 0)
+		tracecmd_tsync_proto_getall(&protos, &protos_count);
+
 	ret = tracecmd_msg_send_trace_req(msg_handle, instance->argc,
 					  instance->argv, use_fifos,
-					  top_instance.trace_id);
+					  top_instance.trace_id,
+					  protos, protos_count);
 	if (ret < 0)
 		die("Failed to send trace request");
 
+	free(protos);
+
 	ret = tracecmd_msg_recv_trace_resp(msg_handle, &nr_cpus, &page_size,
 					   &ports, &use_fifos,
-					   &instance->trace_id);
+					   &instance->trace_id,
+					   &tsync_protos_reply, &tsync_port);
 	if (ret < 0)
-		die("Failed to receive trace response");
+		die("Failed to receive trace response %d", ret);
+
+	if (protos_count && tsync_protos_reply) {
+		if (tsync_proto_is_supported(tsync_protos_reply)) {
+			instance->tsync.sync_proto = tsync_protos_reply;
+			tracecmd_host_tsync(instance, tsync_port);
+		} else
+			warning("Failed to negotiate timestamps synchronization with the guest");
+	}
 
 	if (use_fifos) {
 		if (nr_cpus != nr_fifos) {
@@ -3763,7 +3787,8 @@ static void setup_guest(struct buffer_instance *instance)
 	close(fd);
 }
 
-static void setup_agent(struct buffer_instance *instance, struct common_record_context *ctx)
+static void setup_agent(struct buffer_instance *instance,
+			struct common_record_context *ctx)
 {
 	struct tracecmd_output *network_handle;
 
@@ -5323,6 +5348,7 @@ void init_top_instance(void)
 }
 
 enum {
+	OPT_tsyncinterval	= 242,
 	OPT_user		= 243,
 	OPT_procmap		= 244,
 	OPT_quiet		= 245,
@@ -5626,6 +5652,7 @@ static void parse_record_options(int argc,
 			{"proc-map", no_argument, NULL, OPT_procmap},
 			{"user", required_argument, NULL, OPT_user},
 			{"module", required_argument, NULL, OPT_module},
+			{"tsync-interval", required_argument, NULL, OPT_tsyncinterval},
 			{NULL, 0, NULL, 0}
 		};
 
@@ -5968,6 +5995,10 @@ static void parse_record_options(int argc,
 			ctx->instance->filter_mod = optarg;
 			ctx->filtered = 0;
 			break;
+		case OPT_tsyncinterval:
+			top_instance.tsync.loop_interval = atoi(optarg);
+			guest_sync_set = true;
+			break;
 		case OPT_quiet:
 		case 'q':
 			quiet = true;
@@ -5998,6 +6029,7 @@ static void parse_record_options(int argc,
 					add_argv(instance, "-C", true);
 				}
 			}
+			instance->tsync.loop_interval = top_instance.tsync.loop_interval;
 		}
 	}
 
tracecmd/trace-tsync.c new file +272 -0

272 lines not on this page · raw message

tracecmd/trace-usage.c +4 -0
index 502a4102..d43ca490 100644
@@ -60,6 +60,10 @@ static struct usage_help usage_help[] = {
 		"          --no-filter include trace-cmd threads in the trace\n"
 		"          --proc-map save the traced processes address map into the trace.dat file\n"
 		"          --user execute the specified [command ...] as given user\n"
+		"          --tsync-interval set the loop interval, in ms, for timestamps synchronization with guests:"
+		"               If a negative number is specified, timestamps synchronization is disabled"
+		"               If 0 is specified, no loop is performed - timestamps offset is calculated only twice,"
+		"                                                         at the beginnig and at the end of the trace\n"
 	},
 	{
 		"start",
Signature · 1 line
2.24.1
#14 lore raw

Re: [PATCH v21 11/13] trace-cmd: Basic infrastructure for host - guest timestamp synchronization

lib/trace-cmd/trace-msg.c static int make_trace_resp(struct tracecmd_msg *msg, int page_size, int nr_cpus, · line 1218

Quoted 6 lines · On Mon, 2 Mar 2020 12:14:02 +0200 "Tzvetomir Stoyanov (VMware)"…
lib/trace-cmd/trace-msg.c +1 -1
@@ -940,7 +1215,11 @@ static int make_trace_resp(struct tracecmd_msg *msg, int page_size, int nr_cpus,
 	write_uints(msg->buf, data_size, ports, nr_cpus);
 
 	msg->hdr.size = htonl(ntohl(msg->hdr.size) + data_size);
-	msg->trace_resp.flags = use_fifos ? htonl(MSG_TRACE_USE_FIFOS) : htonl(0);
+	msg->trace_req.flags = use_fifos ? MSG_TRACE_USE_FIFOS : 0;

Is there a reason to use "msg->trace_req" instead of "msg->trace_resp"?

-- Steve

Quoted 7 lines
lib/trace-cmd/trace-msg.c +4 -0
+	msg->trace_resp.flags = htonl(msg->trace_resp.flags);
+	msg->trace_resp.tsync_proto = htonl(tsync_proto);
+	msg->trace_resp.tsync_port = htonl(tsync_port);
+
 	msg->trace_resp.cpus = htonl(nr_cpus);
 	msg->trace_resp.page_size = htonl(page_size);
 	msg->trace_resp.trace_id = htonll(trace_id);
#15 lore raw

Re: [PATCH v21 11/13] trace-cmd: Basic infrastructure for host - guest timestamp synchronization

Quoted 4 lines · Steven Rostedt

On Mon, 2 Mar 2020 12:14:02 +0200
"Tzvetomir Stoyanov (VMware)" <[email protected]> wrote:

Quoted 9 lines

lib/trace-cmd/trace-msg.c static int make_trace_resp(struct tracecmd_msg *msg, int page_size, int nr_cpus, · line 1218

Quoted 6 lines
lib/trace-cmd/trace-msg.c +1 -1
@@ -940,7 +1215,11 @@ static int make_trace_resp(struct tracecmd_msg *msg, int page_size, int nr_cpus,
      write_uints(msg->buf, data_size, ports, nr_cpus);
 
      msg->hdr.size = htonl(ntohl(msg->hdr.size) + data_size);
-     msg->trace_resp.flags = use_fifos ? htonl(MSG_TRACE_USE_FIFOS) : htonl(0);
+     msg->trace_req.flags = use_fifos ? MSG_TRACE_USE_FIFOS : 0;

Is there a reason to use "msg->trace_req" instead of "msg->trace_resp"?

No, it looks like a copy - paste error from a wrong place.

Quoted 9 lines

-- Steve

Quoted 7 lines
lib/trace-cmd/trace-msg.c +4 -0
+     msg->trace_resp.flags = htonl(msg->trace_resp.flags);
+     msg->trace_resp.tsync_proto = htonl(tsync_proto);
+     msg->trace_resp.tsync_port = htonl(tsync_port);
+
      msg->trace_resp.cpus = htonl(nr_cpus);
      msg->trace_resp.page_size = htonl(page_size);
      msg->trace_resp.trace_id = htonll(trace_id);
Signature · 2 lines
Tzvetomir (Ceco) Stoyanov
VMware Open Source Technology Center
#16 lore raw

PATCH v21 12/13 trace-cmd: [POC] PTP-like algorithm for host - guest timestamp synchronization

In reply to Tzvetomir Stoyanov (VMware)

PTP protocol is designed for synchronizing clocks of machines in a local network. The same approach can be used for host - guest timestamp synchronization. This implementation uses ftrace raw markers to track trace timestamps of PTP events. The patch is a POC, two different algorithms for PTP calculations are proposed:
- Choosing the sample with the fastest response time for calculating the clocks offset.
- Calculating the clocks offset using the average of all PTP samples.

The implementation can be tuned using those parameters:
- #define FASTEST_RESPONSE - is defined, the sample with the fastest response time is used for calculating the clocks offset. Otherwise the histogram of all samples is used.
- #define PTP_SYNC_LOOP 340 - defines the number of samples, used for one calculation.
- --tsync-interval - a trace-cmd argument, choose the intervals between offset calculations, performed continuously during the trace.
- #define TSYNC_DEBUG - if defined, a debug information is collected and stored in files, in the guest machine:
s-cid*.txt - For each offset calculation: host and guest clocks and calculated offset. res-cid*.txt - For each tracing session: all calculated clock offsets.

---
 include/trace-cmd/trace-cmd.h      |   1 +
 lib/trace-cmd/Makefile             |   1 +
 lib/trace-cmd/trace-timesync-ptp.c | 651 +++++++++++++++++++++++++++++
 lib/trace-cmd/trace-timesync.c     |   8 +
 tracecmd/trace-agent.c             |   2 +
 tracecmd/trace-record.c            |  23 +-
 6 files changed, 678 insertions(+), 8 deletions(-)
 create mode 100644 lib/trace-cmd/trace-timesync-ptp.c
include/trace-cmd/trace-cmd.h +1 -0
index 42b47e2f..b5207248 100644
@@ -414,6 +414,7 @@ int tracecmd_msg_recv_time_sync(struct tracecmd_msg_handle *msg_handle,
 
 enum{
 	TRACECMD_TIME_SYNC_PROTO_NONE	= 0,
+	TRACECMD_TIME_SYNC_PROTO_PTP	= 1,
 };
 enum{
 	TRACECMD_TIME_SYNC_CMD_PROBE	= 1,
lib/trace-cmd/Makefile +1 -0
index ab7440ac..79672529 100644
@@ -17,6 +17,7 @@ OBJS += trace-filter-hash.o
 OBJS += trace-msg.o
 OBJS += trace-plugin.o
 OBJS += trace-timesync.o
+OBJS += trace-timesync-ptp.o
 
 # Additional util objects
 OBJS += trace-blk-hack.o
lib/trace-cmd/trace-timesync-ptp.c new file +651 -0

651 lines not on this page · raw message

lib/trace-cmd/trace-timesync.c +8 -0
index 0534b14f..68d53127 100644
@@ -46,6 +46,14 @@ static struct tsync_proto *tsync_proto_find(unsigned int proto_id)
 	return NULL;
 }
 
+/**
+ * tracecmd_tsync_init - Initialize the global, per task, time sync data.
+ */
+void tracecmd_tsync_init(void)
+{
+	ptp_clock_sync_register();
+}
+
 int tracecmd_tsync_proto_register(unsigned int proto_id, int weight,
 				int (*init)(struct tracecmd_time_sync *),
 				int (*free)(struct tracecmd_time_sync *),
tracecmd/trace-agent.c +2 -0
index b5816966..2fd09dc3 100644
@@ -247,6 +247,8 @@ static void agent_serve(unsigned int port)
 	if (sd < 0)
 		die("Failed to open vsocket");
 
+	tracecmd_tsync_init();
+
 	if (!get_local_cid(&cid))
 		printf("listening on @%u:%u\n", cid, port);
 
tracecmd/trace-record.c +15 -8
index 0a287f24..8045494c 100644
@@ -6129,10 +6129,6 @@ static bool has_local_instances(void)
 	return false;
 }
 
-/*
- * This function contains common code for the following commands:
- * record, start, stream, profile.
- */
 static void record_trace(int argc, char **argv,
 			 struct common_record_context *ctx)
 {
@@ -6266,12 +6262,23 @@ static void record_trace(int argc, char **argv,
 	finalize_record_trace(ctx);
 }
 
+/*
+ * This function contains common code for the following commands:
+ * record, start, stream, profile.
+ */
+static void record_trace_command(int argc, char **argv,
+				 struct common_record_context *ctx)
+{
+	tracecmd_tsync_init();
+	record_trace(argc, argv, ctx);
+}
+
 void trace_start(int argc, char **argv)
 {
 	struct common_record_context ctx;
 
 	parse_record_options(argc, argv, CMD_start, &ctx);
-	record_trace(argc, argv, &ctx);
+	record_trace_command(argc, argv, &ctx);
 	exit(0);
 }
 
@@ -6351,7 +6358,7 @@ void trace_stream(int argc, char **argv)
 	struct common_record_context ctx;
 
 	parse_record_options(argc, argv, CMD_stream, &ctx);
-	record_trace(argc, argv, &ctx);
+	record_trace_command(argc, argv, &ctx);
 	exit(0);
 }
 
@@ -6370,7 +6377,7 @@ void trace_profile(int argc, char **argv)
 	if (!buffer_instances)
 		top_instance.flags |= BUFFER_FL_PROFILE;
 
-	record_trace(argc, argv, &ctx);
+	record_trace_command(argc, argv, &ctx);
 	do_trace_profile();
 	exit(0);
 }
@@ -6389,7 +6396,7 @@ void trace_record(int argc, char **argv)
 	struct common_record_context ctx;
 
 	parse_record_options(argc, argv, CMD_record, &ctx);
-	record_trace(argc, argv, &ctx);
+	record_trace_command(argc, argv, &ctx);
 	exit(0);
 }
 
Signature · 1 line
2.24.1
#17 lore raw

PATCH v21 13/13 trace-cmd: Debug scripts for PTP-like algorithm for host - guest timestamp synchronization

In reply to Tzvetomir Stoyanov (VMware)

These scripts can be used to visualise debug files, written when the PTP-like algorithm is compiled with TSYNC_DEBUG defined. The files are located in the guest machine:
s-cid*.txt - For each offset calculation: host and guest clocks and calculated offset. res-cid*.txt - For each tracing session: all calculated clock offsets.

tsync_hist.py plots a histogram, using data from a s-cid*.txt file:

	"python tsync_hist.py s-cid2_1.txt"

tsync_res.py plots a line, using data from res-cid*.txt file:
"python tsync_res.py res-cid2.txt"

---
 scripts/debug/tsync_hist.py | 57 +++++++++++++++++++++++++++++++++++++
 scripts/debug/tsync_readme  | 12 ++++++++
 scripts/debug/tsync_res.py  | 46 ++++++++++++++++++++++++++++++
 3 files changed, 115 insertions(+)
 create mode 100644 scripts/debug/tsync_hist.py
 create mode 100644 scripts/debug/tsync_readme
 create mode 100644 scripts/debug/tsync_res.py
scripts/debug/tsync_hist.py new file +57 -0
new file mode 100644
index 00000000..819d1e8f
@@ -0,0 +1,57 @@
+# SPDX-License-Identifier: GPL-2.0
+#
+# Copyright (C) 2019, VMware Inc, Tzvetomir Stoyanov <[email protected]>
+# Copyright (C) 2019, VMware Inc, Yordan Karadzhov <[email protected]>
+
+
+import matplotlib.pyplot as plt
+import matplotlib.lines as mlines
+import numpy as np
+import sys
+
+def newline(p1, p2):
+    ax = plt.gca()
+    xmin, xmax = ax.get_xbound()
+
+    if(p2[0] == p1[0]):
+        xmin = xmax = p1[0]
+        ymin, ymax = ax.get_ybound()
+    else:
+        ymax = p1[1]+(p2[1]-p1[1])/(p2[0]-p1[0])*(xmax-p1[0])
+        ymin = p1[1]+(p2[1]-p1[1])/(p2[0]-p1[0])*(xmin-p1[0])
+
+    l = mlines.Line2D([xmin,xmax], [ymin,ymax], color='red')
+    ax.add_line(l)
+    return l
+
+
+data = np.loadtxt(fname = sys.argv[1])
+selected_ts  = data[-1, 1]
+selected_ofs = data[-1, 0]
+data = data[:-1,:]
+
+x = data[:, 1] - data[:, 0]
+
+mean = x.mean()
+std = x.std()
+
+num_bins = 500
+min = x.min() #+ .4 * (x.max() - x.min())
+max = x.max() #- .4 * (x.max() - x.min())
+bins = np.linspace(min, max, num_bins, endpoint = False, dtype=int)
+
+fig, ax = plt.subplots()
+
+# the histogram of the data
+n, bins, patches = ax.hist(x, bins, histtype=u'step');
+
+ax.set_xlabel('clock offset [$\mu$s]')
+ax.set_ylabel('entries')
+ax.set_title("$\sigma$=%i" % std)
+
+x1, y1 = [selected_ofs, min], [selected_ofs, max]
+newline(x1, y1)
+
+# Tweak spacing to prevent clipping of ylabel
+fig.tight_layout()
+plt.show()
scripts/debug/tsync_readme new file +12 -0
new file mode 100644
index 00000000..f3ebb25d
@@ -0,0 +1,12 @@
+PTP-like algorithm debug
+========================
+
+tsync_*.py scripts can be used to visualise debug files, written when the PTP-like algorithm
+is compiled with TSYNC_DEBUG defined. The files are located in the guest machine:
+    s-cid*.txt - For each offset calculation: host and guest clocks and calculated offset.
+    res-cid*.txt - For each tracing session: all calculated clock offsets.
+
+tsync_hist.py plots a histogram, using data from a s-cid*.txt file:
+	"python tsync_hist.py s-cid2_1.txt"
+tsync_res.py plots a line, using data from res-cid*.txt file:
+	"python tsync_res.py res-cid2.txt"
scripts/debug/tsync_res.py new file +46 -0

46 lines not on this page · raw message

Signature · 1 line
2.24.1