Skip to contentSkip to conversation

Timestamp synchronization of host - guest tracing session

PATCH Superseded Tzvetomir Stoyanov (VMware) 3 participants 31 messages Dec 13, 2019 – Jan 31, 2020 Documentation

v18 of 34 Superseded Latest is v34 →

lore
Share & tools
lore
#0 lore raw

v18 cover Timestamp synchronization of host - guest tracing session

Basic infrastructure for host - guest timestamp synchronization and a POC implementation of PTP-like algorithm.

[
v18 changes: addressed Steven Rostdet comments:
- Replaced semaphores with pthread mutexes.
- Made bitmask with time sync protocols unlimited, so we can support more than 32 protocols. That required to redesign the trace request packet format.
- A lot of small fixes.

v17 changes:
- Implemented new PTP logic for calculating the clocks offset, using histogram of all PTP samples. It gives better results than the logic with the fastest response time, so set the new one as default in the POC patch.

v16 changes:
- Fixed compilation in case no VSOCK is available (Thanks to Slavomir Kaslev)
- Fixed a typo in trace-cmd-record.1.txt (Thanks to Slavomir Kaslev)
- Added forgotten file in the patch "trace-cmd: Add new library APIs for ftrace instances." - trace-instance.c
- Fixed few compilation warnings related to TSYNC_DEBUG code.
- Removed a blank line at the end of "tsync_readme" file.

v15 changes:
- Removed the patch for "--proc-map" from the series, as it should not be part of it.

v14 changes:
- Bring back the PTP-like algorithm and removed the ftrace event based logic.
- Reimplemented the PTP-like algorithm to use raw ftrace markers, instead of clock_gettime() API.
- Refactored the logic to be algorithm independent and plugin friendly.
- Implemented continuous timestamps synchronization, while the trace is running.
- Moved logic from trace-cmd application to libtracecmd, as new library APIs.
- Implemented new trace id functionality.
- Implemented new guest section in host trace.dat file.

v13 changes:
- Remove few patches from the set, as they were merged.
- Rebased to the latest master, Slavomirs patchest "Add VM kernel tracing over vsockets and FIFOs" got merged!

v12 changes:
- Rebased on top of Slavomir's v13 "Add VM kernel tracing over vsockets and FIFOs"

v11 changes:
- Rebased on top of Slavomir's v10 "Add VM kernel tracing over vsockets and FIFOs"
- Addressed Slavomir's commnents from version 10 of the patch series.

v10 changes:
- Fixed broken compilation, call to timestamp_correction_calc() in timestamp_correct was smashed.
- Replaced deprecated tep_data_event_from_type() API with tep_find_event().
- Fixed a warning on assignment const to non const.

v9 changes:
- Fixed implementation of binary search algorithm in timestamp_correct()

v8 changes:
- Added rmdir() call in tracecmd_remove_instance(), to completely remove the instance. However, there is an issue with deleting the instances using rmdir(), which is investigated.
- Few changes in read_qemu_guests_pids(), timestamp_correct(), tsync_offset_load()
tracecmd_clock_context_new() and find_raw_events() suggested by Slavomir.

v7 changes:
- Added warning messages in case time synchronization cannot be negotiated or fails.
- Few optimizations and checks in read_qemu_guests_pids(), tsync_offset_load(), and find_raw_events(), suggested by Slavomir Kaslev.
- Reworked timestamp_correct() to not use static variables.
- Check TRACECMD_OPTION_TIME_SHIFT before reading time sync samples from the trace.dat file

v6 changes:
- Refactored tracecmd_msg_snd_time_sync() and tracecmd_msg_rcv_time_sync() functions:
removed any time sync calculations logic as separate functions in trace-timesync.c file
- Defined TSYNC_PROBE, TSYNC_REQ and TSYNC_RESP messages, in order to make the time sync protocol comprehensible.
- Addressed Steven Rostedt comments.
- Addressed Slavomir Kaslev commnets.

v5 changes:
- Rebased to Slavomir's v8 "Add VM kernel tracing over vsockets and FIFOs" patch series.
- Implemented an algorithm for time drift correction.
- Addressed Slavomir's commnets.
- Refactored the code: moved all time sync specific implementation in trace-timesync.c
- Isolated all hardcoded event specific stuff in a structure, so it could be easily moved to external plugins.
- Added a check for VSOCK support: do not perform vsock dependent time synchronisation in case there is no VSOCK support.

v4 changes:
- Removed the implementation of PTP-like algorithm. The current logic relies on matching time stamps of kvm_exit/virtio_transport_recv_pkt events on host to virtio_transport_alloc_pkt/vp_notify events on guest.
- Rebased to Slavomir's v7 "Add VM kernel tracing over vsockets and FIFOs" patch series.
- Decreased the time synch probes from 5000 to 300.
- Addressed Steven Rostedt comments.
- Code cleanup.

v3 changes:
- Removed any magic constants, used in the PTP-like algorithm, as Slavomir Kaslev suggested.
- Implemented new algorithm, based on mapping kvm_exit events in host context to vsock_send events in guest context, suggested by Steven Rostedt.

v2 changes:
- Addressed Steven Rostedt comments.
- Modified PTP-like timestamps sync algorithm to gain more accuracy, with the help of Yordan Karadzhov and Slavomir Kaslev.
]

Tzvetomir Stoyanov (7):
trace-cmd: Implement new lib API: tracecmd_local_events_system()
trace-cmd: Add support for negative time offsets in trace.dat file
trace-cmd: Add implementations of htonll() and ntohll()
trace-cmd: Add new library APIs for ftrace instances.
trace-cmd: Find and store pids of tasks, which run virtual CPUs of
given VM
trace-cmd: Implement new API tracecmd_add_option_v()
trace-cmd: Implement new option in trace.dat file:
TRACECMD_OPTION_TIME_SHIFT

Tzvetomir Stoyanov (VMware) (11):
trace-cmd: Add new library API for local CPU count
trace-cmd: Add new library API for reading ftrace buffers
trace-cmd: Add new API to generate a unique ID of the tracing session
trace-cmd: Store the session tracing ID in the trace.dat file
trace-cmd: Exchange tracing IDs between host and guest
trace-cmd: Add guest information in host's trace.dat file
trace-cmd: Add host trace clock as guest trace argument
trace-cmd: Refactor few trace-cmd internal functions.
trace-cmd: Basic infrastructure for host - guest timestamp
synchronization
trace-cmd: [POC] PTP-like algorithm for host - guest timestamp
synchronization
trace-cmd: Debug scripts for PTP-like algorithm for host - guest
timestamp synchronization

 Documentation/trace-cmd-record.1.txt      |   7 +
 include/trace-cmd/trace-cmd.h             | 121 +++-
 include/trace-cmd/trace-msg.h             |  10 +
 include/traceevent/event-parse.h          |   1 +
 lib/trace-cmd/Makefile                    |   3 +
 lib/trace-cmd/include/trace-cmd-local.h   |  33 +-
 lib/trace-cmd/include/trace-tsync-local.h |  38 ++
 lib/trace-cmd/trace-input.c               | 323 ++++++++++-
 lib/trace-cmd/trace-instance.c            | 335 +++++++++++
 lib/trace-cmd/trace-msg.c                 | 403 ++++++++++++--
 lib/trace-cmd/trace-output.c              | 117 +++-
 lib/trace-cmd/trace-timesync-ptp.c        | 649 ++++++++++++++++++++++
 lib/trace-cmd/trace-timesync.c            | 504 +++++++++++++++++
 lib/trace-cmd/trace-util.c                | 374 +++++++++++--
 scripts/debug/tsync_hist.py               |  57 ++
 scripts/debug/tsync_readme                |  12 +
 scripts/debug/tsync_res.py                |  46 ++
 tracecmd/Makefile                         |   3 +-
 tracecmd/include/trace-local.h            |  33 +-
 tracecmd/trace-agent.c                    |  58 +-
 tracecmd/trace-list.c                     |   2 +-
 tracecmd/trace-profile.c                  |   2 +-
 tracecmd/trace-read.c                     |   4 +-
 tracecmd/trace-record.c                   | 612 ++++++++++----------
 tracecmd/trace-show.c                     |   2 +
 tracecmd/trace-stat.c                     |  24 +-
 tracecmd/trace-tsync.c                    | 271 +++++++++
 tracecmd/trace-usage.c                    |   4 +
 28 files changed, 3563 insertions(+), 485 deletions(-)
 create mode 100644 lib/trace-cmd/include/trace-tsync-local.h
 create mode 100644 lib/trace-cmd/trace-instance.c
 create mode 100644 lib/trace-cmd/trace-timesync-ptp.c
 create mode 100644 lib/trace-cmd/trace-timesync.c
 create mode 100644 scripts/debug/tsync_hist.py
 create mode 100644 scripts/debug/tsync_readme
 create mode 100644 scripts/debug/tsync_res.py
 create mode 100644 tracecmd/trace-tsync.c
Signature · 1 line
2.23.0
#1 lore raw

PATCH v18 1/18 trace-cmd: Implement new lib API: tracecmd_local_events_system()

From: Tzvetomir Stoyanov <[email protected]>

The new tracecmd lib API tracecmd_local_events_system() creates a tep handler and initializes it with the events of the specified subsystems. It can be used instead of tracecmd_local_events(), when not all events are going to be parsed using that tep handler.

---
 include/trace-cmd/trace-cmd.h |   4 +-
 lib/trace-cmd/trace-util.c    | 102 +++++++++++++++++++++++-----------
 2 files changed, 72 insertions(+), 34 deletions(-)
include/trace-cmd/trace-cmd.h +3 -1
index 13afce7..9363a26 100644
@@ -27,8 +27,10 @@ extern int tracecmd_disable_plugins;
 char **tracecmd_event_systems(const char *tracing_dir);
 char **tracecmd_system_events(const char *tracing_dir, const char *system);
 struct tep_handle *tracecmd_local_events(const char *tracing_dir);
+struct tep_handle *tracecmd_local_events_system(const char *tracing_dir,
+						const char * const *sys_names);
 int tracecmd_fill_local_events(const char *tracing_dir,
-			       struct tep_handle *pevent, int *parsing_failures);
+			       struct tep_handle *tep, int *parsing_failures);
 char **tracecmd_local_plugins(const char *tracing_dir);
 
 char **tracecmd_add_list(char **list, const char *name, int len);
lib/trace-cmd/trace-util.c +69 -33

135 lines not on this page · raw message

Signature · 1 line
2.23.0
#2 lore raw

PATCH v18 2/18 trace-cmd: Add support for negative time offsets in trace.dat file

In reply to Tzvetomir Stoyanov (VMware)

From: Tzvetomir Stoyanov <[email protected]>

When synchronizing timestamps between different machines, there are cases when the time offset is negative. This patch changes the way time offset is written and read from trace.dat file - as signed decimal, instead of hex.

---
 include/trace-cmd/trace-cmd.h | 2 +-
 lib/trace-cmd/trace-input.c   | 6 +++---
 tracecmd/trace-read.c         | 4 ++--
 3 files changed, 6 insertions(+), 6 deletions(-)
include/trace-cmd/trace-cmd.h +1 -1
index 9363a26..7f9cb73 100644
@@ -138,7 +138,7 @@ const char *tracecmd_buffer_instance_name(struct tracecmd_input *handle, int ind
 struct tracecmd_input *tracecmd_buffer_instance_handle(struct tracecmd_input *handle, int indx);
 int tracecmd_is_buffer_instance(struct tracecmd_input *handle);
 
-void tracecmd_set_ts_offset(struct tracecmd_input *handle, unsigned long long offset);
+void tracecmd_set_ts_offset(struct tracecmd_input *handle, long long offset);
 void tracecmd_set_ts2secs(struct tracecmd_input *handle, unsigned long long hz);
 
 void tracecmd_print_events(struct tracecmd_input *handle, const char *regex);
lib/trace-cmd/trace-input.c +3 -3
index 3b187e3..91362f3 100644
@@ -90,7 +90,7 @@ struct tracecmd_input {
 	bool			read_page;
 	bool			use_pipe;
 	struct cpu_data 	*cpu_data;
-	unsigned long long	ts_offset;
+	long long		ts_offset;
 	double			ts2secs;
 	char *			cpustats;
 	char *			uname;
@@ -2127,7 +2127,7 @@ static int init_cpu(struct tracecmd_input *handle, int cpu)
 }
 
 void tracecmd_set_ts_offset(struct tracecmd_input *handle,
-			    unsigned long long offset)
+			    long long offset)
 {
 	handle->ts_offset = offset;
 }
@@ -2305,7 +2305,7 @@ tracecmd_search_task_map(struct tracecmd_input *handle,
 
 static int handle_options(struct tracecmd_input *handle)
 {
-	unsigned long long offset;
+	long long offset;
 	unsigned short option;
 	unsigned int size;
 	char *cpustats = NULL;
tracecmd/trace-read.c +2 -2
index c0d640d..4ce2e98 100644
@@ -58,7 +58,7 @@ static struct list_head handle_list;
 struct input_files {
 	struct list_head	list;
 	const char		*file;
-	unsigned long long	tsoffset;
+	long long		tsoffset;
 	unsigned long long	ts2secs;
 };
 static struct list_head input_files;
@@ -1466,7 +1466,7 @@ void trace_report (int argc, char **argv)
 	struct input_files *inputs;
 	struct handle_list *handles;
 	enum output_type otype;
-	unsigned long long tsoffset = 0;
+	long long tsoffset = 0;
 	unsigned long long ts2secs = 0;
 	unsigned long long ts2sc;
 	int show_stat = 0;
Signature · 1 line
2.23.0
#3 lore raw

PATCH v18 3/18 trace-cmd: Add implementations of htonll() and ntohll()

In reply to Tzvetomir Stoyanov (VMware)

From: Tzvetomir Stoyanov <[email protected]>

Implementations of htonll() and ntohll() are added as macros, if they are not already defined.

---
 include/trace-cmd/trace-msg.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
include/trace-cmd/trace-msg.h +10 -0
index aab8a69..654ca43 100644
@@ -12,4 +12,14 @@
 
 extern unsigned int page_size;
 
+#ifndef htonll
+# if __BYTE_ORDER == __LITTLE_ENDIAN
+#define htonll(x) __bswap_64(x)
+#define ntohll(x) __bswap_64(x)
+#else
+#define htonll(x) (x)
+#define ntohll(x) (x)
+#endif
+#endif
+
 #endif /* _TRACE_MSG_H_ */
Signature · 1 line
2.23.0
#4 lore raw

Re: [PATCH v18 03/18] trace-cmd: Add implementations of htonll() and ntohll()

[ Replying this time to the latest version. Same comment though ]

Quoted 5 lines · On Fri, 13 Dec 2019 17:30:14 +0200 "Tzvetomir Stoyanov (VMware)"…

From: Tzvetomir Stoyanov <[email protected]>

Implementations of htonll() and ntohll() are added as macros, if they are not already defined.

I'm somewhat nervous about this being in a global header, as it is mostly generic. I wonder if we should prefix these with "tracecmd_" just to avoid any name collisions.

We could update this later when we make trace-cmd into a library.

-- Steve

Quoted 24 lines
---
 include/trace-cmd/trace-msg.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
include/trace-cmd/trace-msg.h +10 -0
index aab8a69..654ca43 100644
@@ -12,4 +12,14 @@
 
 extern unsigned int page_size;
 
+#ifndef htonll
+# if __BYTE_ORDER == __LITTLE_ENDIAN
+#define htonll(x) __bswap_64(x)
+#define ntohll(x) __bswap_64(x)
+#else
+#define htonll(x) (x)
+#define ntohll(x) (x)
+#endif
+#endif
+
 #endif /* _TRACE_MSG_H_ */
#5 lore raw

PATCH v18 4/18 trace-cmd: Add new library APIs for ftrace instances.

In reply to Tzvetomir Stoyanov (VMware)

From: Tzvetomir Stoyanov <[email protected]>

In order to reuse the code, the functionality related to ftrace instances is moved from trace-cmd application to libtracecmd. The following new library APIs are introduced:

library structure, representing a ftrace instance:

	struct tracecmd_instance;

APIs for creating and deleting ftrace instances:

	struct tracecmd_instance *tracecmd_alloc_instance(const char *name);
	void tracecmd_free_instance(struct tracecmd_instance *instance);
	int tracecmd_make_instance(struct tracecmd_instance *instance);
	int tracecmd_remove_instance(struct tracecmd_instance *instance);

APIs for getting and setting ftrace instance parameters:

        char *tracecmd_get_instance_name(struct tracecmd_instance *instance);
        int tracecmd_alloc_instance_clock(struct tracecmd_instance *instance, const char *clock);
        char *tracecmd_get_instance_clock(struct tracecmd_instance *instance);

APIs for reading and writing ftrace files, instance aware:

	char *tracecmd_get_instance_file(struct tracecmd_instance *instance, const char *file);
	char *tracecmd_get_instance_dir(struct tracecmd_instance *instance);
	int tracecmd_write_instance_file(struct tracecmd_instance *instance,
	                                 const char *file, const char *str,
        	                         const char *type);
	int tracecmd_write_file(const char *file, const char *str, const char *type);
	char *tracecmd_read_instance_file(struct tracecmd_instance *instance,
        	                          char *file, int *psize);

API for setting ftrace clock, instance aware:

	int tracecmd_set_clock(struct tracecmd_instance *instance, char **old_clock);
---
 include/trace-cmd/trace-cmd.h           |  24 ++
 lib/trace-cmd/Makefile                  |   1 +
 lib/trace-cmd/include/trace-cmd-local.h |  33 +--
 lib/trace-cmd/trace-instance.c          | 335 +++++++++++++++++++++++
 lib/trace-cmd/trace-util.c              |  71 ++++-
 tracecmd/include/trace-local.h          |   6 +-
 tracecmd/trace-list.c                   |   2 +-
 tracecmd/trace-record.c                 | 343 +++++++-----------------
 tracecmd/trace-show.c                   |   2 +
 tracecmd/trace-stat.c                   |  20 +-
 10 files changed, 544 insertions(+), 293 deletions(-)
 create mode 100644 lib/trace-cmd/trace-instance.c
include/trace-cmd/trace-cmd.h +24 -0
index 7f9cb73..32182bc 100644
@@ -356,6 +356,30 @@ int tracecmd_msg_send_trace_resp(struct tracecmd_msg_handle *msg_handle,
 int tracecmd_msg_recv_trace_resp(struct tracecmd_msg_handle *msg_handle,
 				 int *nr_cpus, int *page_size,
 				 unsigned int **ports, bool *use_fifos);
+/* --- ftrace instances --- */
+
+struct tracecmd_instance;
+
+struct tracecmd_instance *tracecmd_alloc_instance(const char *name);
+void tracecmd_free_instance(struct tracecmd_instance *instance);
+int tracecmd_make_instance(struct tracecmd_instance *instance);
+int tracecmd_remove_instance(struct tracecmd_instance *instance);
+char *
+tracecmd_get_instance_file(struct tracecmd_instance *instance, const char *file);
+char *tracecmd_get_instance_dir(struct tracecmd_instance *instance);
+int tracecmd_write_instance_file(struct tracecmd_instance *instance,
+				 const char *file, const char *str,
+				 const char *type);
+
+int tracecmd_write_file(const char *file, const char *str, const char *type);
+char *tracecmd_read_instance_file(struct tracecmd_instance *instance,
+				  char *file, int *psize);
+char *tracecmd_get_instance_name(struct tracecmd_instance *instance);
+int tracecmd_alloc_instance_clock(struct tracecmd_instance *instance,
+				  const char *clock);
+char *tracecmd_get_instance_clock(struct tracecmd_instance *instance);
+int tracecmd_set_clock(struct tracecmd_instance *instance, char **old_clock);
+
 
 /* --- Plugin handling --- */
 extern struct tep_plugin_option trace_ftrace_options[];
lib/trace-cmd/Makefile +1 -0
index 3b4b5aa..18c7013 100644
@@ -13,6 +13,7 @@ OBJS += trace-input.o
 OBJS += trace-output.o
 OBJS += trace-recorder.o
 OBJS += trace-util.o
+OBJS += trace-instance.o
 OBJS += trace-filter-hash.o
 OBJS += trace-msg.o
 
lib/trace-cmd/include/trace-cmd-local.h +2 -31
index 09574db..eef4d39 100644
@@ -18,36 +18,7 @@
 #define STR(x)	_STR(x)
 #define FILE_VERSION_STRING STR(FILE_VERSION)
 
-static ssize_t __do_write(int fd, const void *data, size_t size)
-{
-	ssize_t tot = 0;
-	ssize_t w;
-
-	do {
-		w = write(fd, data + tot, size - tot);
-		tot += w;
-
-		if (!w)
-			break;
-		if (w < 0)
-			return w;
-	} while (tot != size);
-
-	return tot;
-}
-
-static ssize_t
-__do_write_check(int fd, const void *data, size_t size)
-{
-	ssize_t ret;
-
-	ret = __do_write(fd, data, size);
-	if (ret < 0)
-		return ret;
-	if (ret != size)
-		return -1;
-
-	return 0;
-}
+ssize_t __do_write_check(int fd, const void *data, size_t size);
+void __noreturn die(const char *fmt, ...); /* Can be overriden */
 
 #endif /* _TRACE_CMD_LOCAL_H */
lib/trace-cmd/trace-instance.c new file +335 -0

335 lines not on this page · raw message

lib/trace-cmd/trace-util.c +66 -5

86 lines not on this page · raw message

tracecmd/include/trace-local.h +3 -3
index fedc0b7..3a05e9c 100644
@@ -181,7 +181,7 @@ struct pid_addr_maps {
 
 struct buffer_instance {
 	struct buffer_instance	*next;
-	const char		*name;
+	struct tracecmd_instance	*ftrace;
 	char			*cpumask;
 	struct event_list	*events;
 	struct event_list	**event_next;
@@ -195,7 +195,6 @@ struct buffer_instance {
 	struct func_list	*filter_funcs;
 	struct func_list	*notrace_funcs;
 
-	const char		*clock;
 	unsigned int		*client_ports;
 
 	struct trace_seq	*s_save;
@@ -225,6 +224,8 @@ struct buffer_instance {
 	bool			use_fifos;
 };
 
+void init_top_instance(void);
+
 extern struct buffer_instance top_instance;
 extern struct buffer_instance *buffer_instances;
 extern struct buffer_instance *first_instance;
@@ -238,7 +239,6 @@ extern struct buffer_instance *first_instance;
 
 struct buffer_instance *create_instance(const char *name);
 void add_instance(struct buffer_instance *instance, int cpu_count);
-char *get_instance_file(struct buffer_instance *instance, const char *file);
 void update_first_instance(struct buffer_instance *instance, int topt);
 
 void show_instance_file(struct buffer_instance *instance, const char *name);
tracecmd/trace-list.c +1 -1
index 65099a5..e0782ed 100644
@@ -33,7 +33,7 @@ void show_instance_file(struct buffer_instance *instance, const char *name)
 {
 	char *path;
 
-	path = get_instance_file(instance, name);
+	path = tracecmd_get_instance_file(instance->ftrace, name);
 	dump_file_content(path);
 	tracecmd_put_tracing_file(path);
 }
tracecmd/trace-record.c +99 -244

734 lines not on this page · raw message

tracecmd/trace-show.c +2 -0
index 96bfe77..ca0f4cf 100644
@@ -51,6 +51,8 @@ void trace_show(int argc, char **argv)
 		{NULL, 0, NULL, 0}
 	};
 
+	init_top_instance();
+
 	while ((c = getopt_long(argc-1, argv+1, "B:c:fsp",
 				long_options, &option_index)) >= 0) {
 		switch (c) {
tracecmd/trace-stat.c +11 -9
index 538f4ad..5cc9d28 100644
@@ -30,7 +30,7 @@ static int get_instance_file_fd(struct buffer_instance *instance,
 	char *path;
 	int fd;
 
-	path = get_instance_file(instance, file);
+	path = tracecmd_get_instance_file(instance->ftrace, file);
 	fd = open(path, O_RDONLY);
 	tracecmd_put_tracing_file(path);
 
@@ -347,7 +347,7 @@ static void report_events(struct buffer_instance *instance)
 
 	free(str);
 
-	path = get_instance_file(instance, "events");
+	path = tracecmd_get_instance_file(instance->ftrace, "events");
 	if (!path)
 		die("malloc");
 
@@ -436,7 +436,7 @@ static void report_event_filters(struct buffer_instance *instance)
 	enum event_iter_type type;
 	enum event_process processed = PROCESSED_NONE;
 
-	path = get_instance_file(instance, "events");
+	path = tracecmd_get_instance_file(instance->ftrace, "events");
 	if (!path)
 		die("malloc");
 
@@ -509,7 +509,7 @@ static void report_event_triggers(struct buffer_instance *instance)
 	enum event_iter_type type;
 	enum event_process processed = PROCESSED_NONE;
 
-	path = get_instance_file(instance, "events");
+	path = tracecmd_get_instance_file(instance->ftrace, "events");
 	if (!path)
 		die("malloc");
 
@@ -598,7 +598,7 @@ static void report_graph_funcs(struct buffer_instance *instance)
 {
 	char *path;
 
-	path = get_instance_file(instance, "set_graph_function");
+	path = tracecmd_get_instance_file(instance->ftrace, "set_graph_function");
 	if (!path)
 		die("malloc");
 
@@ -606,7 +606,7 @@ static void report_graph_funcs(struct buffer_instance *instance)
 	
 	tracecmd_put_tracing_file(path);
 
-	path = get_instance_file(instance, "set_graph_notrace");
+	path = tracecmd_get_instance_file(instance->ftrace, "set_graph_notrace");
 	if (!path)
 		die("malloc");
 
@@ -619,7 +619,7 @@ static void report_ftrace_filters(struct buffer_instance *instance)
 {
 	char *path;
 
-	path = get_instance_file(instance, "set_ftrace_filter");
+	path = tracecmd_get_instance_file(instance->ftrace, "set_ftrace_filter");
 	if (!path)
 		die("malloc");
 
@@ -627,7 +627,7 @@ static void report_ftrace_filters(struct buffer_instance *instance)
 	
 	tracecmd_put_tracing_file(path);
 
-	path = get_instance_file(instance, "set_ftrace_notrace");
+	path = tracecmd_get_instance_file(instance->ftrace, "set_ftrace_notrace");
 	if (!path)
 		die("malloc");
 
@@ -857,7 +857,7 @@ static void stat_instance(struct buffer_instance *instance)
 	if (instance != &top_instance) {
 		if (instance != first_instance)
 			printf("---------------\n");
-		printf("Instance: %s\n", instance->name);
+		printf("Instance: %s\n", tracecmd_get_instance_name(instance->ftrace));
 	}
 
 	report_plugin(instance);
@@ -882,6 +882,8 @@ void trace_stat (int argc, char **argv)
 	int status;
 	int c;
 
+	init_top_instance();
+
 	for (;;) {
 		c = getopt(argc-1, argv+1, "tB:");
 		if (c == -1)
Signature · 1 line
2.23.0
#6 lore raw

Re: [PATCH v18 04/18] trace-cmd: Add new library APIs for ftrace instances.

Quoted 26 lines · On Fri, 13 Dec 2019 17:30:15 +0200 "Tzvetomir Stoyanov (VMware)"…
lib/trace-cmd/trace-instance.c new file +26 -0
+/**
+ * tracecmd_get_instance_file - return the path to a instance file
+ * @instance: buffer instance for the file, can be NULL for the top instance
+ * @file: name of file to return
+ *
+ * Returns the path name of the @file for the given @instance
+ *
+ * Must use tracecmd_put_tracing_file() to free the returned string
+ */
+char *
+tracecmd_get_instance_file(struct tracecmd_instance *instance, const char *file)
+{
+	char *path;
+	char *buf;
+	int ret;
+
+	if (instance && instance->name) {
+		ret = asprintf(&buf, "instances/%s/%s", instance->name, file);
+		if (ret < 0)
+			die("Failed to allocate name for %s/%s", instance->name, file);
+		path = tracecmd_get_tracing_file(buf);
+		free(buf);
+	} else
+		path = tracecmd_get_tracing_file(file);
+
+	return path;

lib/trace-cmd/trace-instance.c line 150

Quoted 10 lines
lib/trace-cmd/trace-instance.c new file +10 -0
+}
+
+/**
+ * tracecmd_get_instance_file - return the path to a instance file
+ * @instance: buffer instance for the file, can be NULL for the top instance
+ * @file: name of file to return
+ *
+ * Returns the path name of the @file for the given @instance
+ *
+ * Must use tracecmd_put_tracing_file() to free the returned string

The above is a cut and paste of tracecmd_get_instance_file, not of the instance_dir.

Quoted 22 lines

+ */

lib/trace-cmd/trace-instance.c new file +21 -0
+char *tracecmd_get_instance_dir(struct tracecmd_instance *instance)
+{
+	char *buf;
+	char *path;
+	int ret;
+
+	if (instance->name) {
+		ret = asprintf(&buf, "instances/%s", instance->name);
+		if (ret < 0)
+			die("Failed to allocate for instance %s", instance->name);
+		path = tracecmd_get_tracing_file(buf);
+		free(buf);
+	} else
+		path = tracecmd_find_tracing_dir();
+
+	return path;
+}
+
+/**
+ * tracecmd_write_instance_file - Write in trace file of specific instance
+ * @instance: buffer instance for the file, can be NULL for the top instance

lib/trace-cmd/trace-instance.c line 182

Quoted 10 lines
lib/trace-cmd/trace-instance.c new file +10 -0
+ * @file: name of the file
+ * @str: Nul terminated string, that will be written in the file
+ * @type: Nul terminated string, describing the current write operation
+ *	  Used for logging purposes
+ *
+ * Returns the number of written bytes, or -1 in case of an error
+ */
+int tracecmd_write_instance_file(struct tracecmd_instance *instance,
+				 const char *file, const char *str,
+				 const char *type)

I think we should get rid of the type and move the logic that needs it to the callers.

Quoted 14 lines
lib/trace-cmd/trace-instance.c new file +14 -0
+{
+	struct stat st;
+	char *path;
+	int ret;
+
+	path = tracecmd_get_instance_file(instance, file);
+	ret = stat(path, &st);
+	if (ret == 0)
+		ret = tracecmd_write_file(path, str, type);
+	tracecmd_put_tracing_file(path);
+
+	return ret;
+}
+

[..]

Quoted 19 lines
lib/trace-cmd/trace-util.c +19 -0
+/*
+ * tracecmd_write_file - Write in trace file
+ * @file: Full name of the trace file.
+ * @str: Null terminated string, that will be written in the file.
+ * @type: Null terminated string, describing the current write operation.
+ *	  Used for logging purposes.
+ *
+ * Returns the number of written bytes, or -1 in case of an error
+ */
+int tracecmd_write_file(const char *file, const char *str, const char *type)
+{
+	char buf[BUFSIZ];
+	int ret;
+	int fd;
+
+	fd = open(file, O_WRONLY | O_TRUNC);
+	if (fd < 0)
+		die("opening to '%s'", file);
+	ret = write(fd, str, strlen(str));

lib/trace-cmd/trace-util.c int tracecmd_stack_tracer_status(int *status) · line 1143

Quoted 10 lines
lib/trace-cmd/trace-util.c +10 -0
+	close(fd);
+	if (ret < 0 && type) {
+		/* write failed */
+		fd = open(file, O_RDONLY);
+		if (fd < 0)
+			die("writing to '%s'", file);
+		/* the filter has the error */
+		while ((ret = read(fd, buf, BUFSIZ)) > 0)
+			fprintf(stderr, "%.*s", ret, buf);
+		die("Failed %s of %s\n", type, file);

Yeah, the above needs to be moved to the upper layers. It even has a comment about "the filter has the error". That's reference to the filter file of trace events, which is the only file that really does that.

-- Steve

Quoted 5 lines
lib/trace-cmd/trace-util.c +4 -0
+		close(fd);
+	}
+	return ret;
+}
#7 lore raw

Re: [PATCH v18 04/18] trace-cmd: Add new library APIs for ftrace instances.

Quoted 133 lines · Steven Rostedt
Quoted 26 lines · On Fri, 13 Dec 2019 17:30:15 +0200 "Tzvetomir Stoyanov (VMware)"…
lib/trace-cmd/trace-instance.c new file +26 -0
+/**
+ * tracecmd_get_instance_file - return the path to a instance file
+ * @instance: buffer instance for the file, can be NULL for the top instance
+ * @file: name of file to return
+ *
+ * Returns the path name of the @file for the given @instance
+ *
+ * Must use tracecmd_put_tracing_file() to free the returned string
+ */
+char *
+tracecmd_get_instance_file(struct tracecmd_instance *instance, const char *file)
+{
+     char *path;
+     char *buf;
+     int ret;
+
+     if (instance && instance->name) {
+             ret = asprintf(&buf, "instances/%s/%s", instance->name, file);
+             if (ret < 0)
+                     die("Failed to allocate name for %s/%s", instance->name, file);
+             path = tracecmd_get_tracing_file(buf);
+             free(buf);
+     } else
+             path = tracecmd_get_tracing_file(file);
+
+     return path;

lib/trace-cmd/trace-instance.c line 150

Quoted 10 lines
lib/trace-cmd/trace-instance.c new file +10 -0
+}
+
+/**
+ * tracecmd_get_instance_file - return the path to a instance file
+ * @instance: buffer instance for the file, can be NULL for the top instance
+ * @file: name of file to return
+ *
+ * Returns the path name of the @file for the given @instance
+ *
+ * Must use tracecmd_put_tracing_file() to free the returned string

The above is a cut and paste of tracecmd_get_instance_file, not of the instance_dir.

Quoted 22 lines

+ */

lib/trace-cmd/trace-instance.c new file +21 -0
+char *tracecmd_get_instance_dir(struct tracecmd_instance *instance)
+{
+     char *buf;
+     char *path;
+     int ret;
+
+     if (instance->name) {
+             ret = asprintf(&buf, "instances/%s", instance->name);
+             if (ret < 0)
+                     die("Failed to allocate for instance %s", instance->name);
+             path = tracecmd_get_tracing_file(buf);
+             free(buf);
+     } else
+             path = tracecmd_find_tracing_dir();
+
+     return path;
+}
+
+/**
+ * tracecmd_write_instance_file - Write in trace file of specific instance
+ * @instance: buffer instance for the file, can be NULL for the top instance

lib/trace-cmd/trace-instance.c line 182

Quoted 10 lines
lib/trace-cmd/trace-instance.c new file +10 -0
+ * @file: name of the file
+ * @str: Nul terminated string, that will be written in the file
+ * @type: Nul terminated string, describing the current write operation
+ *     Used for logging purposes
+ *
+ * Returns the number of written bytes, or -1 in case of an error
+ */
+int tracecmd_write_instance_file(struct tracecmd_instance *instance,
+                              const char *file, const char *str,
+                              const char *type)

I think we should get rid of the type and move the logic that needs it to the callers.

Quoted 14 lines
lib/trace-cmd/trace-instance.c new file +14 -0
+{
+     struct stat st;
+     char *path;
+     int ret;
+
+     path = tracecmd_get_instance_file(instance, file);
+     ret = stat(path, &st);
+     if (ret == 0)
+             ret = tracecmd_write_file(path, str, type);
+     tracecmd_put_tracing_file(path);
+
+     return ret;
+}
+

[..]

Quoted 19 lines
lib/trace-cmd/trace-util.c +19 -0
+/*
+ * tracecmd_write_file - Write in trace file
+ * @file: Full name of the trace file.
+ * @str: Null terminated string, that will be written in the file.
+ * @type: Null terminated string, describing the current write operation.
+ *     Used for logging purposes.
+ *
+ * Returns the number of written bytes, or -1 in case of an error
+ */
+int tracecmd_write_file(const char *file, const char *str, const char *type)
+{
+     char buf[BUFSIZ];
+     int ret;
+     int fd;
+
+     fd = open(file, O_WRONLY | O_TRUNC);
+     if (fd < 0)
+             die("opening to '%s'", file);
+     ret = write(fd, str, strlen(str));

lib/trace-cmd/trace-util.c int tracecmd_stack_tracer_status(int *status) · line 1143

Quoted 10 lines
lib/trace-cmd/trace-util.c +10 -0
+     close(fd);
+     if (ret < 0 && type) {
+             /* write failed */
+             fd = open(file, O_RDONLY);
+             if (fd < 0)
+                     die("writing to '%s'", file);
+             /* the filter has the error */
+             while ((ret = read(fd, buf, BUFSIZ)) > 0)
+                     fprintf(stderr, "%.*s", ret, buf);
+             die("Failed %s of %s\n", type, file);

Yeah, the above needs to be moved to the upper layers. It even has a comment about "the filter has the error". That's reference to the filter file of trace events, which is the only file that really does that.

Quoted 2 lines

-- Steve

These new instance APIs are going to be part of the new libtracefs, I'll remove this patch
from the series, when the libtracefs code is merged.

Quoted 5 lines
Quoted 5 lines
lib/trace-cmd/trace-util.c +4 -0
+             close(fd);
+     }
+     return ret;
+}
Signature · 2 lines
Tzvetomir (Ceco) Stoyanov
VMware Open Source Technology Center
#8 lore raw

PATCH v18 5/18 trace-cmd: Add new library API for local CPU count

In reply to Tzvetomir Stoyanov (VMware)

In order to reuse the code, the function detecting the number of local CPU is moved from trace-cmd application to libtracecmd. The following new library API is introduced:

	int tracecmd_count_cpus(void);
---
 include/trace-cmd/trace-cmd.h  |  3 +++
 lib/trace-cmd/trace-util.c     | 49 ++++++++++++++++++++++++++++++++++
 tracecmd/include/trace-local.h |  2 --
 tracecmd/trace-agent.c         |  2 +-
 tracecmd/trace-profile.c       |  2 +-
 tracecmd/trace-record.c        | 46 +++----------------------------
 tracecmd/trace-stat.c          |  4 +--
 7 files changed, 59 insertions(+), 49 deletions(-)
include/trace-cmd/trace-cmd.h +3 -0
index 32182bc..f9ad823 100644
@@ -427,6 +427,9 @@ void tracecmd_plog(const char *fmt, ...);
 void tracecmd_plog_error(const char *fmt, ...);
 int tracecmd_set_logfile(char *logfile);
 
+/* --- System --- */
+int tracecmd_count_cpus(void);
+
 /* --- Hack! --- */
 int tracecmd_blk_hack(struct tracecmd_input *handle);
 
lib/trace-cmd/trace-util.c +49 -0

55 lines not on this page · raw message

tracecmd/include/trace-local.h +0 -2
index 3a05e9c..a51f0d0 100644
@@ -243,8 +243,6 @@ void update_first_instance(struct buffer_instance *instance, int topt);
 
 void show_instance_file(struct buffer_instance *instance, const char *name);
 
-int count_cpus(void);
-
 /* moved from trace-cmd.h */
 void tracecmd_create_top_instance(char *name);
 void tracecmd_remove_instances(void);
tracecmd/trace-agent.c +1 -1
index 93e3edf..1c6e0a3 100644
@@ -203,7 +203,7 @@ static void agent_serve(unsigned int port)
 
 	signal(SIGCHLD, handle_sigchld);
 
-	nr_cpus = count_cpus();
+	nr_cpus = tracecmd_count_cpus();
 	page_size = getpagesize();
 
 	sd = make_vsock(port);
tracecmd/trace-profile.c +1 -1
index 231e381..cfae2a2 100644
@@ -1314,7 +1314,7 @@ void trace_init_profile(struct tracecmd_input *handle, struct hook_list *hook,
 	 * system.
 	 */
 	if (!h->cpus)
-		h->cpus = count_cpus();
+		h->cpus = tracecmd_count_cpus();
 
 	list_head_init(&h->migrate_starts);
 	h->cpu_starts = malloc(sizeof(*h->cpu_starts) * h->cpus);
tracecmd/trace-record.c +3 -43

70 lines not on this page · raw message

tracecmd/trace-stat.c +2 -2
index 5cc9d28..bb6e40b 100644
@@ -737,7 +737,7 @@ static void report_cpumask(struct buffer_instance *instance)
 	cont = strstrip(str);
 
 	/* check to make sure all CPUs on this machine are set */
-	cpus = count_cpus();
+	cpus = tracecmd_count_cpus();
 
 	for (i = strlen(cont) - 1; i >= 0 && cpus > 0; i--) {
 		if (cont[i] == ',')
@@ -896,7 +896,7 @@ void trace_stat (int argc, char **argv)
 			instance = create_instance(optarg);
 			if (!instance)
 				die("Failed to create instance");
-			add_instance(instance, count_cpus());
+			add_instance(instance, tracecmd_count_cpus());
 			/* top instance requires direct access */
 			if (!topt && is_top_instance(first_instance))
 				first_instance = instance;
Signature · 1 line
2.23.0
#9 lore raw

PATCH v18 6/18 trace-cmd: Add new library API for reading ftrace buffers

In reply to Tzvetomir Stoyanov (VMware)

A new libtracecmd API is introduced:

int tracecmd_iterate_raw_events(struct tep_handle *tep,
struct tracecmd_instance *instance, int (*callback)(struct tep_event *,
struct tep_record *, int, void *context),
void *callback_context);

It reads events from trace_pipe_raw, per cpu ftrace buffer, and calls a user callback for each of them. The API is instance aware.
The new API can be used to search for a specific event, or sequence of events, in a ftrace instance.

---
 include/trace-cmd/trace-cmd.h |   8 +++
 lib/trace-cmd/trace-util.c    | 115 ++++++++++++++++++++++++++++++++++
 2 files changed, 123 insertions(+)
include/trace-cmd/trace-cmd.h +8 -0
index f9ad823..c0b9f4b 100644
@@ -380,6 +380,14 @@ int tracecmd_alloc_instance_clock(struct tracecmd_instance *instance,
 char *tracecmd_get_instance_clock(struct tracecmd_instance *instance);
 int tracecmd_set_clock(struct tracecmd_instance *instance, char **old_clock);
 
+int tracecmd_iterate_raw_events(struct tep_handle *tep,
+				struct tracecmd_instance *instance,
+				   int (*callback)(struct tep_event *,
+						   struct tep_record *,
+						   int, void *context),
+				void *callback_context);
+
+
 
 /* --- Plugin handling --- */
 extern struct tep_plugin_option trace_ftrace_options[];
lib/trace-cmd/trace-util.c +115 -0

118 lines not on this page · raw message

Signature · 1 line
2.23.0
#10 lore raw

PATCH v18 7/18 trace-cmd: Find and store pids of tasks, which run virtual CPUs of given VM

In reply to Tzvetomir Stoyanov (VMware)

From: Tzvetomir Stoyanov <[email protected]>

In order to match host and guest events, a mapping between guest VCPU and the host task, running this VCPU is needed. Extended existing struct guest to hold such mapping and added logic in read_qemu_guests() function to initialize it. Implemented a new internal API, get_guest_vcpu_pid(), to retrieve VCPU-task mapping for given VM.

---
 tracecmd/include/trace-local.h |  2 ++
 tracecmd/trace-record.c        | 54 ++++++++++++++++++++++++++++++++++
 2 files changed, 56 insertions(+)
tracecmd/include/trace-local.h +2 -0
index a51f0d0..51abef1 100644
@@ -243,6 +243,8 @@ void update_first_instance(struct buffer_instance *instance, int topt);
 
 void show_instance_file(struct buffer_instance *instance, const char *name);
 
+int get_guest_vcpu_pid(unsigned int guest_cid, unsigned int guest_vcpu);
+
 /* moved from trace-cmd.h */
 void tracecmd_create_top_instance(char *name);
 void tracecmd_remove_instances(void);
tracecmd/trace-record.c +54 -0

82 lines not on this page · raw message

Signature · 1 line
2.23.0
#11 lore raw

PATCH v18 8/18 trace-cmd: Implement new API tracecmd_add_option_v()

In reply to Tzvetomir Stoyanov (VMware)

From: Tzvetomir Stoyanov <[email protected]>

A new tracecmd API tracecmd_add_option_v() is introduced. It adds new option in trace.dat, similar to tracecmd_add_option(), but the option's data is passed as list of buffers. The standard struct iovec is used as input parameter, containing the option's data buffers.

---
 include/trace-cmd/trace-cmd.h    |   5 ++
 include/traceevent/event-parse.h |   1 +
 lib/trace-cmd/trace-output.c     | 117 +++++++++++++++++++++++++++----
 3 files changed, 108 insertions(+), 15 deletions(-)
include/trace-cmd/trace-cmd.h +5 -0
index c0b9f4b..9a2b7db 100644
@@ -261,11 +261,16 @@ struct tracecmd_output *tracecmd_create_init_file_override(const char *output_fi
 struct tracecmd_option *tracecmd_add_option(struct tracecmd_output *handle,
 					    unsigned short id, int size,
 					    const void *data);
+struct tracecmd_option *
+tracecmd_add_option_v(struct tracecmd_output *handle,
+		      unsigned short id, const struct iovec *vector, int count);
+
 struct tracecmd_option *tracecmd_add_buffer_option(struct tracecmd_output *handle,
 						   const char *name, int cpus);
 
 int tracecmd_write_cpus(struct tracecmd_output *handle, int cpus);
 int tracecmd_write_options(struct tracecmd_output *handle);
+int tracecmd_append_options(struct tracecmd_output *handle);
 int tracecmd_update_option(struct tracecmd_output *handle,
 			   struct tracecmd_option *option, int size,
 			   const void *data);
include/traceevent/event-parse.h +1 -0
index a77af4c..30e06a7 100644
@@ -11,6 +11,7 @@
 #include <stdio.h>
 #include <regex.h>
 #include <string.h>
+#include <sys/uio.h>
 
 #include "traceevent/trace-seq.h"
 
lib/trace-cmd/trace-output.c +102 -15

161 lines not on this page · raw message

Signature · 1 line
2.23.0
#12 lore raw

PATCH v18 9/18 trace-cmd: Add new API to generate a unique ID of the tracing session

In reply to Tzvetomir Stoyanov (VMware)

A new libtracecmd API is implemented:

	unsigned long long tracecmd_generate_traceid(void);

At trace-cmd init phase a unique ID, used for the current tracing session.

---
 include/trace-cmd/trace-cmd.h  |  1 +
 lib/trace-cmd/trace-util.c     | 37 ++++++++++++++++++++++++++++++++++
 tracecmd/include/trace-local.h |  1 +
 tracecmd/trace-record.c        |  1 +
 4 files changed, 40 insertions(+)
include/trace-cmd/trace-cmd.h +1 -0
index 9a2b7db..ad69313 100644
@@ -441,6 +441,7 @@ void tracecmd_plog_error(const char *fmt, ...);
 int tracecmd_set_logfile(char *logfile);
 
 /* --- System --- */
+unsigned long long tracecmd_generate_traceid(void);
 int tracecmd_count_cpus(void);
 
 /* --- Hack! --- */
lib/trace-cmd/trace-util.c +37 -0
index 75ec5a3..fabd558 100644
@@ -18,6 +18,8 @@
 #include <sys/mount.h>
 #include <sys/types.h>
 #include <sys/stat.h>
+#include <sys/sysinfo.h>
+#include <time.h>
 
 #include "trace-cmd.h"
 #include "event-utils.h"
@@ -1309,3 +1311,38 @@ error:
 	tracecmd_put_tracing_file(path);
 	return ret;
 }
+
+#define FNV_64_PRIME 0x100000001b3ULL
+/*
+ * tracecmd_generate_traceid - Generate a unique ID, used to identify
+ *			       the current tracing session
+ *
+ * Returns unique ID
+ */
+unsigned long long tracecmd_generate_traceid(void)
+{
+	unsigned long long hash = 0;
+	unsigned char *ustr;
+	struct sysinfo sinfo;
+	struct timespec ts;
+	char *str = NULL;
+
+	clock_gettime(CLOCK_MONOTONIC_RAW, &ts);
+	sysinfo(&sinfo);
+	asprintf(&str, "%ld %ld %ld %ld %ld %ld %ld %ld %d",
+		 ts.tv_sec, ts.tv_nsec,
+		 sinfo.loads[0], sinfo.loads[1], sinfo.loads[2],
+		 sinfo.freeram, sinfo.sharedram, sinfo.freeswap,
+		 sinfo.procs);
+	if (!str)
+		return 0;
+	ustr = (unsigned char *)str;
+	hash = 0;
+	while (*ustr) {
+		hash ^= (unsigned long long)*ustr++;
+		hash *= FNV_64_PRIME;
+	}
+
+	free(str);
+	return hash;
+}
tracecmd/include/trace-local.h +1 -0
index 51abef1..6203e0d 100644
@@ -182,6 +182,7 @@ struct pid_addr_maps {
 struct buffer_instance {
 	struct buffer_instance	*next;
 	struct tracecmd_instance	*ftrace;
+	unsigned long long	trace_id;
 	char			*cpumask;
 	struct event_list	*events;
 	struct event_list	**event_next;
tracecmd/trace-record.c +1 -0
index 7a3d687..7089f7b 100644
@@ -5061,6 +5061,7 @@ void init_top_instance(void)
 		top_instance.ftrace = tracecmd_alloc_instance(NULL);
 	top_instance.cpu_count = tracecmd_count_cpus();
 	top_instance.flags = BUFFER_FL_KEEP;
+	top_instance.trace_id = tracecmd_generate_traceid();
 	tracecmd_init_instance(&top_instance);
 }
 
Signature · 1 line
2.23.0
#13 lore raw

PATCH v18 10/18 trace-cmd: Store the session tracing ID in the trace.dat file

In reply to Tzvetomir Stoyanov (VMware)

The ID of the current tracing session is stored in the trace.dat file. A new file option is introduced:
TRACECMD_OPTION_TRACEID
The data is stored in the file as NULL terminated string:

	"\nTraceID: %llu\n"
A new libtracecmd API is introduced, to get the ID when the trace.dat file is read:
	unsigned long long tracecmd_get_traceid(struct tracecmd_input *handle);
---
 include/trace-cmd/trace-cmd.h |  2 ++
 lib/trace-cmd/trace-input.c   | 25 +++++++++++++++++++++++++
 tracecmd/trace-record.c       | 15 +++++++++++++++
 3 files changed, 42 insertions(+)
include/trace-cmd/trace-cmd.h +2 -0
index ad69313..6cc2913 100644
@@ -86,6 +86,7 @@ enum {
 	TRACECMD_OPTION_CPUCOUNT,
 	TRACECMD_OPTION_VERSION,
 	TRACECMD_OPTION_PROCMAPS,
+	TRACECMD_OPTION_TRACEID,
 };
 
 enum {
@@ -128,6 +129,7 @@ int tracecmd_copy_headers(struct tracecmd_input *handle, int fd);
 void tracecmd_set_flag(struct tracecmd_input *handle, int flag);
 void tracecmd_clear_flag(struct tracecmd_input *handle, int flag);
 unsigned long tracecmd_get_flags(struct tracecmd_input *handle);
+unsigned long long tracecmd_get_traceid(struct tracecmd_input *handle);
 
 void tracecmd_parse_trace_clock(struct tracecmd_input *handle, char *file, int size);
 
lib/trace-cmd/trace-input.c +25 -0
index 91362f3..40d9825 100644
@@ -79,6 +79,7 @@ struct tracecmd_input {
 	struct tep_plugin_list	*plugin_list;
 	struct tracecmd_input	*parent;
 	unsigned long		flags;
+	unsigned long long	trace_id;
 	int			fd;
 	int			long_size;
 	int			page_size;
@@ -2169,6 +2170,15 @@ static void procmap_free(struct pid_addr_maps *maps)
 	free(maps);
 }
 
+static int trace_traceid_load(struct tracecmd_input *handle, char *buf)
+{
+
+	if (sscanf(buf, "%*s %llu", &handle->trace_id) == 1)
+		return 0;
+
+	return -1;
+}
+
 #define STR_PROCMAP_LINE_MAX	(PATH_MAX+22)
 static int trace_pid_map_load(struct tracecmd_input *handle, char *buf)
 {
@@ -2406,6 +2416,10 @@ static int handle_options(struct tracecmd_input *handle)
 			if (buf[size-1] == '\0')
 				trace_pid_map_load(handle, buf);
 			break;
+		case TRACECMD_OPTION_TRACEID:
+			if (buf[size-1] == '\0')
+				trace_traceid_load(handle, buf);
+			break;
 		default:
 			warning("unknown option %d", option);
 			break;
@@ -3507,3 +3521,14 @@ void tracecmd_set_show_data_func(struct tracecmd_input *handle,
 {
 	handle->show_data_func = func;
 }
+
+/**
+ * tracecmd_get_traceid - get the trace id of the session
+ * @handle: input handle for the trace.dat file
+ *
+ * Returns the trace id, written in the trace file
+ */
+unsigned long long tracecmd_get_traceid(struct tracecmd_input *handle)
+{
+	return handle->trace_id;
+}
tracecmd/trace-record.c +15 -0
index 7089f7b..b65aeb2 100644
@@ -3714,6 +3714,19 @@ add_pid_maps(struct tracecmd_output *handle, struct buffer_instance *instance)
 	trace_seq_destroy(&s);
 }
 
+static void
+add_trace_id(struct tracecmd_output *handle, struct buffer_instance *instance)
+{
+	char *str = NULL;
+	int len;
+
+	len = asprintf(&str, "\nTraceID: %llu\n", instance->trace_id);
+	if (len < 1)
+		return;
+	tracecmd_add_option(handle, TRACECMD_OPTION_TRACEID, len + 1, str);
+	free(str);
+}
+
 static void
 add_buffer_stat(struct tracecmd_output *handle, struct buffer_instance *instance)
 {
@@ -3817,6 +3830,8 @@ static void add_options(struct tracecmd_output *handle, struct common_record_con
 	add_option_hooks(handle);
 	add_uname(handle);
 	add_version(handle);
+	if (!no_top_instance())
+		add_trace_id(handle, &top_instance);
 }
 
 static void write_guest_file(struct buffer_instance *instance)
Signature · 1 line
2.23.0
#14 lore raw

Re: [PATCH v18 10/18] trace-cmd: Store the session tracing ID in the trace.dat file

Quoted 5 lines · On Fri, 13 Dec 2019 17:30:21 +0200 "Tzvetomir Stoyanov (VMware)"…

The ID of the current tracing session is stored in the trace.dat file. A new file option is introduced:
TRACECMD_OPTION_TRACEID
The data is stored in the file as NULL terminated string:

	"\nTraceID: %llu\n"

If the trace ID is just a 64 bit number, why is it stored in the file as a string?

-- Steve

Quoted 117 lines

A new libtracecmd API is introduced, to get the ID when the trace.dat file is read:

	unsigned long long tracecmd_get_traceid(struct tracecmd_input *handle);
---
 include/trace-cmd/trace-cmd.h |  2 ++
 lib/trace-cmd/trace-input.c   | 25 +++++++++++++++++++++++++
 tracecmd/trace-record.c       | 15 +++++++++++++++
 3 files changed, 42 insertions(+)
include/trace-cmd/trace-cmd.h +2 -0
index ad69313..6cc2913 100644
@@ -86,6 +86,7 @@ enum {
 	TRACECMD_OPTION_CPUCOUNT,
 	TRACECMD_OPTION_VERSION,
 	TRACECMD_OPTION_PROCMAPS,
+	TRACECMD_OPTION_TRACEID,
 };
 
 enum {
@@ -128,6 +129,7 @@ int tracecmd_copy_headers(struct tracecmd_input *handle, int fd);
 void tracecmd_set_flag(struct tracecmd_input *handle, int flag);
 void tracecmd_clear_flag(struct tracecmd_input *handle, int flag);
 unsigned long tracecmd_get_flags(struct tracecmd_input *handle);
+unsigned long long tracecmd_get_traceid(struct tracecmd_input *handle);
 
 void tracecmd_parse_trace_clock(struct tracecmd_input *handle, char *file, int size);
 
lib/trace-cmd/trace-input.c +25 -0
index 91362f3..40d9825 100644
@@ -79,6 +79,7 @@ struct tracecmd_input {
 	struct tep_plugin_list	*plugin_list;
 	struct tracecmd_input	*parent;
 	unsigned long		flags;
+	unsigned long long	trace_id;
 	int			fd;
 	int			long_size;
 	int			page_size;
@@ -2169,6 +2170,15 @@ static void procmap_free(struct pid_addr_maps *maps)
 	free(maps);
 }
 
+static int trace_traceid_load(struct tracecmd_input *handle, char *buf)
+{
+
+	if (sscanf(buf, "%*s %llu", &handle->trace_id) == 1)
+		return 0;
+
+	return -1;
+}
+
 #define STR_PROCMAP_LINE_MAX	(PATH_MAX+22)
 static int trace_pid_map_load(struct tracecmd_input *handle, char *buf)
 {
@@ -2406,6 +2416,10 @@ static int handle_options(struct tracecmd_input *handle)
 			if (buf[size-1] == '\0')
 				trace_pid_map_load(handle, buf);
 			break;
+		case TRACECMD_OPTION_TRACEID:
+			if (buf[size-1] == '\0')
+				trace_traceid_load(handle, buf);
+			break;
 		default:
 			warning("unknown option %d", option);
 			break;
@@ -3507,3 +3521,14 @@ void tracecmd_set_show_data_func(struct tracecmd_input *handle,
 {
 	handle->show_data_func = func;
 }
+
+/**
+ * tracecmd_get_traceid - get the trace id of the session
+ * @handle: input handle for the trace.dat file
+ *
+ * Returns the trace id, written in the trace file
+ */
+unsigned long long tracecmd_get_traceid(struct tracecmd_input *handle)
+{
+	return handle->trace_id;
+}
tracecmd/trace-record.c +15 -0
index 7089f7b..b65aeb2 100644
@@ -3714,6 +3714,19 @@ add_pid_maps(struct tracecmd_output *handle, struct buffer_instance *instance)
 	trace_seq_destroy(&s);
 }
 
+static void
+add_trace_id(struct tracecmd_output *handle, struct buffer_instance *instance)
+{
+	char *str = NULL;
+	int len;
+
+	len = asprintf(&str, "\nTraceID: %llu\n", instance->trace_id);
+	if (len < 1)
+		return;
+	tracecmd_add_option(handle, TRACECMD_OPTION_TRACEID, len + 1, str);
+	free(str);
+}
+
 static void
 add_buffer_stat(struct tracecmd_output *handle, struct buffer_instance *instance)
 {
@@ -3817,6 +3830,8 @@ static void add_options(struct tracecmd_output *handle, struct common_record_con
 	add_option_hooks(handle);
 	add_uname(handle);
 	add_version(handle);
+	if (!no_top_instance())
+		add_trace_id(handle, &top_instance);
 }
 
 static void write_guest_file(struct buffer_instance *instance)
#15 lore raw

Re: [PATCH v18 10/18] trace-cmd: Store the session tracing ID in the trace.dat file

Quoted 4 lines · Steven Rostedt

On Fri, 13 Dec 2019 17:30:21 +0200
"Tzvetomir Stoyanov (VMware)" <[email protected]> wrote:

Quoted 9 lines
Quoted 5 lines

The ID of the current tracing session is stored in the trace.dat file. A new file option is introduced:
TRACECMD_OPTION_TRACEID
The data is stored in the file as NULL terminated string:

      "\nTraceID: %llu\n"

If the trace ID is just a 64 bit number, why is it stored in the file as a string?

The only reason for this is to avoid endian long long conversions, and to simplify reading of trace.dat file, for debug purposes.

Quoted 121 lines

-- Steve

Quoted 117 lines

A new libtracecmd API is introduced, to get the ID when the trace.dat file is read:

      unsigned long long tracecmd_get_traceid(struct tracecmd_input *handle);
---
 include/trace-cmd/trace-cmd.h |  2 ++
 lib/trace-cmd/trace-input.c   | 25 +++++++++++++++++++++++++
 tracecmd/trace-record.c       | 15 +++++++++++++++
 3 files changed, 42 insertions(+)
include/trace-cmd/trace-cmd.h +2 -0
index ad69313..6cc2913 100644
@@ -86,6 +86,7 @@ enum {
      TRACECMD_OPTION_CPUCOUNT,
      TRACECMD_OPTION_VERSION,
      TRACECMD_OPTION_PROCMAPS,
+     TRACECMD_OPTION_TRACEID,
 };
 
 enum {
@@ -128,6 +129,7 @@ int tracecmd_copy_headers(struct tracecmd_input *handle, int fd);
 void tracecmd_set_flag(struct tracecmd_input *handle, int flag);
 void tracecmd_clear_flag(struct tracecmd_input *handle, int flag);
 unsigned long tracecmd_get_flags(struct tracecmd_input *handle);
+unsigned long long tracecmd_get_traceid(struct tracecmd_input *handle);
 
 void tracecmd_parse_trace_clock(struct tracecmd_input *handle, char *file, int size);
 
lib/trace-cmd/trace-input.c +25 -0
index 91362f3..40d9825 100644
@@ -79,6 +79,7 @@ struct tracecmd_input {
      struct tep_plugin_list  *plugin_list;
      struct tracecmd_input   *parent;
      unsigned long           flags;
+     unsigned long long      trace_id;
      int                     fd;
      int                     long_size;
      int                     page_size;
@@ -2169,6 +2170,15 @@ static void procmap_free(struct pid_addr_maps *maps)
      free(maps);
 }
 
+static int trace_traceid_load(struct tracecmd_input *handle, char *buf)
+{
+
+     if (sscanf(buf, "%*s %llu", &handle->trace_id) == 1)
+             return 0;
+
+     return -1;
+}
+
 #define STR_PROCMAP_LINE_MAX (PATH_MAX+22)
 static int trace_pid_map_load(struct tracecmd_input *handle, char *buf)
 {
@@ -2406,6 +2416,10 @@ static int handle_options(struct tracecmd_input *handle)
                      if (buf[size-1] == '\0')
                              trace_pid_map_load(handle, buf);
                      break;
+             case TRACECMD_OPTION_TRACEID:
+                     if (buf[size-1] == '\0')
+                             trace_traceid_load(handle, buf);
+                     break;
              default:
                      warning("unknown option %d", option);
                      break;
@@ -3507,3 +3521,14 @@ void tracecmd_set_show_data_func(struct tracecmd_input *handle,
 {
      handle->show_data_func = func;
 }
+
+/**
+ * tracecmd_get_traceid - get the trace id of the session
+ * @handle: input handle for the trace.dat file
+ *
+ * Returns the trace id, written in the trace file
+ */
+unsigned long long tracecmd_get_traceid(struct tracecmd_input *handle)
+{
+     return handle->trace_id;
+}
tracecmd/trace-record.c +15 -0
index 7089f7b..b65aeb2 100644
@@ -3714,6 +3714,19 @@ add_pid_maps(struct tracecmd_output *handle, struct buffer_instance *instance)
      trace_seq_destroy(&s);
 }
 
+static void
+add_trace_id(struct tracecmd_output *handle, struct buffer_instance *instance)
+{
+     char *str = NULL;
+     int len;
+
+     len = asprintf(&str, "\nTraceID: %llu\n", instance->trace_id);
+     if (len < 1)
+             return;
+     tracecmd_add_option(handle, TRACECMD_OPTION_TRACEID, len + 1, str);
+     free(str);
+}
+
 static void
 add_buffer_stat(struct tracecmd_output *handle, struct buffer_instance *instance)
 {
@@ -3817,6 +3830,8 @@ static void add_options(struct tracecmd_output *handle, struct common_record_con
      add_option_hooks(handle);
      add_uname(handle);
      add_version(handle);
+     if (!no_top_instance())
+             add_trace_id(handle, &top_instance);
 }
 
 static void write_guest_file(struct buffer_instance *instance)
Signature · 2 lines
Tzvetomir (Ceco) Stoyanov
VMware Open Source Technology Center
#16 lore raw

PATCH v18 11/18 trace-cmd: Exchange tracing IDs between host and guest

In reply to Tzvetomir Stoyanov (VMware)

Extend the trace request and trace reply messages, to include also the IDs of host and guest tracing sessions. Those IDs are used to unambiguously match the tracing sessions, when reading trace.dat files.

---
 include/trace-cmd/trace-cmd.h  | 12 ++++++++----
 lib/trace-cmd/trace-msg.c      | 31 +++++++++++++++++++++++--------
 tracecmd/include/trace-local.h |  4 +++-
 tracecmd/trace-agent.c         | 12 +++++++-----
 tracecmd/trace-record.c        | 12 +++++++++---
 5 files changed, 50 insertions(+), 21 deletions(-)
include/trace-cmd/trace-cmd.h +8 -4
index 6cc2913..23415e4 100644
@@ -353,16 +353,20 @@ bool tracecmd_msg_done(struct tracecmd_msg_handle *msg_handle);
 void tracecmd_msg_set_done(struct tracecmd_msg_handle *msg_handle);
 
 int tracecmd_msg_send_trace_req(struct tracecmd_msg_handle *msg_handle,
-				int argc, char **argv, bool use_fifos);
+				int argc, char **argv, bool use_fifos,
+				unsigned long long trace_id);
 int tracecmd_msg_recv_trace_req(struct tracecmd_msg_handle *msg_handle,
-				int *argc, char ***argv, bool *use_fifos);
+				int *argc, char ***argv, bool *use_fifos,
+				unsigned long long *trace_id);
 
 int tracecmd_msg_send_trace_resp(struct tracecmd_msg_handle *msg_handle,
 				 int nr_cpus, int page_size,
-				 unsigned int *ports, bool use_fifos);
+				 unsigned int *ports, bool use_fifos,
+				 unsigned long long trace_id);
 int tracecmd_msg_recv_trace_resp(struct tracecmd_msg_handle *msg_handle,
 				 int *nr_cpus, int *page_size,
-				 unsigned int **ports, bool *use_fifos);
+				 unsigned int **ports, bool *use_fifos,
+				 unsigned long long *trace_id);
 /* --- ftrace instances --- */
 
 struct tracecmd_instance;
lib/trace-cmd/trace-msg.c +23 -8
index 9c8a690..a34297f 100644
@@ -63,12 +63,14 @@ struct tracecmd_msg_rinit {
 struct tracecmd_msg_trace_req {
 	be32 flags;
 	be32 argc;
+	u64 trace_id;
 } __attribute__((packed));
 
 struct tracecmd_msg_trace_resp {
 	be32 flags;
 	be32 cpus;
 	be32 page_size;
+	u64 trace_id;
 } __attribute__((packed));
 
 struct tracecmd_msg_header {
@@ -811,7 +813,8 @@ int tracecmd_msg_wait_close_resp(struct tracecmd_msg_handle *msg_handle)
 	return tracecmd_msg_wait_for_cmd(msg_handle, MSG_CLOSE_RESP);
 }
 
-static int make_trace_req(struct tracecmd_msg *msg, int argc, char **argv, bool use_fifos)
+static int make_trace_req(struct tracecmd_msg *msg, int argc, char **argv,
+			  bool use_fifos, unsigned long long trace_id)
 {
 	size_t args_size = 0;
 	char *p;
@@ -823,6 +826,7 @@ static int make_trace_req(struct tracecmd_msg *msg, int argc, char **argv, bool
 	msg->hdr.size = htonl(ntohl(msg->hdr.size) + args_size);
 	msg->trace_req.flags = use_fifos ? htonl(MSG_TRACE_USE_FIFOS) : htonl(0);
 	msg->trace_req.argc = htonl(argc);
+	msg->trace_req.trace_id = htonll(trace_id);
 	msg->buf = calloc(args_size, 1);
 	if (!msg->buf)
 		return -ENOMEM;
@@ -835,13 +839,15 @@ static int make_trace_req(struct tracecmd_msg *msg, int argc, char **argv, bool
 }
 
 int tracecmd_msg_send_trace_req(struct tracecmd_msg_handle *msg_handle,
-				int argc, char **argv, bool use_fifos)
+				int argc, char **argv, bool use_fifos,
+				unsigned long long trace_id)
 {
 	struct tracecmd_msg msg;
 	int ret;
 
 	tracecmd_msg_init(MSG_TRACE_REQ, &msg);
-	ret = make_trace_req(&msg, argc, argv, use_fifos);
+	ret = make_trace_req(&msg, argc, argv,
+			     use_fifos, trace_id);
 	if (ret < 0)
 		return ret;
 
@@ -854,7 +860,8 @@ int tracecmd_msg_send_trace_req(struct tracecmd_msg_handle *msg_handle,
   *     free(argv);
   */
 int tracecmd_msg_recv_trace_req(struct tracecmd_msg_handle *msg_handle,
-				int *argc, char ***argv, bool *use_fifos)
+				int *argc, char ***argv, bool *use_fifos,
+				unsigned long long *trace_id)
 {
 	struct tracecmd_msg msg;
 	char *p, *buf_end, **args;
@@ -901,6 +908,7 @@ int tracecmd_msg_recv_trace_req(struct tracecmd_msg_handle *msg_handle,
 	*argc = nr_args;
 	*argv = args;
 	*use_fifos = ntohl(msg.trace_req.flags) & MSG_TRACE_USE_FIFOS;
+	*trace_id = ntohll(msg.trace_req.trace_id);
 
 	/*
 	 * On success we're passing msg.buf to the caller through argv[0] so we
@@ -921,7 +929,8 @@ out:
 }
 
 static int make_trace_resp(struct tracecmd_msg *msg, int page_size, int nr_cpus,
-			   unsigned int *ports, bool use_fifos)
+			   unsigned int *ports, bool use_fifos,
+			   unsigned long long trace_id)
 {
 	int data_size;
 
@@ -935,19 +944,22 @@ static int make_trace_resp(struct tracecmd_msg *msg, int page_size, int nr_cpus,
 	msg->trace_resp.flags = use_fifos ? htonl(MSG_TRACE_USE_FIFOS) : htonl(0);
 	msg->trace_resp.cpus = htonl(nr_cpus);
 	msg->trace_resp.page_size = htonl(page_size);
+	msg->trace_resp.trace_id = htonll(trace_id);
 
 	return 0;
 }
 
 int tracecmd_msg_send_trace_resp(struct tracecmd_msg_handle *msg_handle,
 				 int nr_cpus, int page_size,
-				 unsigned int *ports, bool use_fifos)
+				 unsigned int *ports, bool use_fifos,
+				 unsigned long long trace_id)
 {
 	struct tracecmd_msg msg;
 	int ret;
 
 	tracecmd_msg_init(MSG_TRACE_RESP, &msg);
-	ret = make_trace_resp(&msg, page_size, nr_cpus, ports, use_fifos);
+	ret = make_trace_resp(&msg, page_size, nr_cpus, ports,
+			      use_fifos, trace_id);
 	if (ret < 0)
 		return ret;
 
@@ -956,13 +968,15 @@ int tracecmd_msg_send_trace_resp(struct tracecmd_msg_handle *msg_handle,
 
 int tracecmd_msg_recv_trace_resp(struct tracecmd_msg_handle *msg_handle,
 				 int *nr_cpus, int *page_size,
-				 unsigned int **ports, bool *use_fifos)
+				 unsigned int **ports, bool *use_fifos,
+				 unsigned long long *trace_id)
 {
 	struct tracecmd_msg msg;
 	char *p, *buf_end;
 	ssize_t buf_len;
 	int i, ret;
 
+	memset(&msg, 0, sizeof(msg));
 	ret = tracecmd_msg_recv(msg_handle->fd, &msg);
 	if (ret < 0)
 		return ret;
@@ -981,6 +995,7 @@ int tracecmd_msg_recv_trace_resp(struct tracecmd_msg_handle *msg_handle,
 	*use_fifos = ntohl(msg.trace_resp.flags) & MSG_TRACE_USE_FIFOS;
 	*nr_cpus = ntohl(msg.trace_resp.cpus);
 	*page_size = ntohl(msg.trace_resp.page_size);
+	*trace_id = ntohll(msg.trace_resp.trace_id);
 	*ports = calloc(*nr_cpus, sizeof(**ports));
 	if (!*ports) {
 		ret = -ENOMEM;
tracecmd/include/trace-local.h +3 -1
index 6203e0d..90933cc 100644
@@ -98,7 +98,9 @@ void trace_usage(int argc, char **argv);
 
 int trace_record_agent(struct tracecmd_msg_handle *msg_handle,
 		       int cpus, int *fds,
-		       int argc, char **argv, bool use_fifos);
+		       int argc, char **argv,
+		       bool use_fifos,
+		       unsigned long long trace_id);
 
 struct hook_list;
 
tracecmd/trace-agent.c +7 -5
index 1c6e0a3..cc330b7 100644
@@ -128,6 +128,7 @@ cleanup:
 static void agent_handle(int sd, int nr_cpus, int page_size)
 {
 	struct tracecmd_msg_handle *msg_handle;
+	unsigned long long trace_id;
 	unsigned int *ports;
 	char **argv = NULL;
 	int argc = 0;
@@ -144,7 +145,8 @@ static void agent_handle(int sd, int nr_cpus, int page_size)
 	if (!msg_handle)
 		die("Failed to allocate message handle");
 
-	ret = tracecmd_msg_recv_trace_req(msg_handle, &argc, &argv, &use_fifos);
+	ret = tracecmd_msg_recv_trace_req(msg_handle, &argc, &argv,
+					  &use_fifos, &trace_id);
 	if (ret < 0)
 		die("Failed to receive trace request");
 
@@ -153,13 +155,13 @@ static void agent_handle(int sd, int nr_cpus, int page_size)
 
 	if (!use_fifos)
 		make_vsocks(nr_cpus, fds, ports);
-
+	trace_id = tracecmd_generate_traceid();
 	ret = tracecmd_msg_send_trace_resp(msg_handle, nr_cpus, page_size,
-					   ports, use_fifos);
+					   ports, use_fifos, trace_id);
 	if (ret < 0)
 		die("Failed to send trace response");
-
-	trace_record_agent(msg_handle, nr_cpus, fds, argc, argv, use_fifos);
+	trace_record_agent(msg_handle, nr_cpus, fds, argc, argv,
+			   use_fifos, trace_id);
 
 	free(argv[0]);
 	free(argv);
tracecmd/trace-record.c +9 -3
index b65aeb2..4bd9b7a 100644
@@ -3505,12 +3505,16 @@ static void connect_to_agent(struct buffer_instance *instance)
 		die("Failed to allocate message handle");
 
 	ret = tracecmd_msg_send_trace_req(msg_handle, instance->argc,
-					  instance->argv, use_fifos);
+					  instance->argv, use_fifos,
+					  top_instance.trace_id);
+
 	if (ret < 0)
 		die("Failed to send trace request");
 
 	ret = tracecmd_msg_recv_trace_resp(msg_handle, &nr_cpus, &page_size,
-					   &ports, &use_fifos);
+					   &ports, &use_fifos,
+					   &instance->trace_id);
+
 	if (ret < 0)
 		die("Failed to receive trace response");
 
@@ -6116,7 +6120,8 @@ void trace_record(int argc, char **argv)
 int trace_record_agent(struct tracecmd_msg_handle *msg_handle,
 		       int cpus, int *fds,
 		       int argc, char **argv,
-		       bool use_fifos)
+		       bool use_fifos,
+		       unsigned long long trace_id)
 {
 	struct common_record_context ctx;
 	char **argv_plus;
@@ -6146,6 +6151,7 @@ int trace_record_agent(struct tracecmd_msg_handle *msg_handle,
 	ctx.instance->flags |= BUFFER_FL_AGENT;
 	ctx.instance->msg_handle = msg_handle;
 	msg_handle->version = V3_PROTOCOL;
+	top_instance.trace_id = trace_id;
 	record_trace(argc, argv, &ctx);
 
 	free(argv_plus);
Signature · 1 line
2.23.0
#17 lore raw

PATCH v18 12/18 trace-cmd: Implement new option in trace.dat file: TRACECMD_OPTION_TIME_SHIFT

In reply to Tzvetomir Stoyanov (VMware)

From: Tzvetomir Stoyanov <[email protected]>

The TRACECMD_OPTION_TIME_SHIFT is used when synchronizing trace time stamps between two trace.dat files. It contains multiple long long (time, offset) pairs, describing time stamps _offset_, measured in the given local _time_. The content of the option buffer is:
8 bytes - long long integer, ID of the tracing session
4 bytes - integer, count of timestamp offsets
long long array of size _count_, local time in which the offset is measured long long array of size _count_, offset of the time stamps

---
 include/trace-cmd/trace-cmd.h |   3 +
 lib/trace-cmd/trace-input.c   | 182 +++++++++++++++++++++++++++++++++-
 2 files changed, 183 insertions(+), 2 deletions(-)
include/trace-cmd/trace-cmd.h +3 -0
index 23415e4..b06a0c3 100644
@@ -87,6 +87,7 @@ enum {
 	TRACECMD_OPTION_VERSION,
 	TRACECMD_OPTION_PROCMAPS,
 	TRACECMD_OPTION_TRACEID,
+	TRACECMD_OPTION_TIME_SHIFT,
 };
 
 enum {
@@ -130,6 +131,8 @@ void tracecmd_set_flag(struct tracecmd_input *handle, int flag);
 void tracecmd_clear_flag(struct tracecmd_input *handle, int flag);
 unsigned long tracecmd_get_flags(struct tracecmd_input *handle);
 unsigned long long tracecmd_get_traceid(struct tracecmd_input *handle);
+unsigned long long tracecmd_get_tsync_peer(struct tracecmd_input *handle);
+int tracecmd_enable_tsync(struct tracecmd_input *handle, bool enable);
 
 void tracecmd_parse_trace_clock(struct tracecmd_input *handle, char *file, int size);
 
lib/trace-cmd/trace-input.c +180 -2

239 lines not on this page · raw message

Signature · 1 line
2.23.0
#18 lore raw

Re: [PATCH v18 12/18] trace-cmd: Implement new option in trace.dat file: TRACECMD_OPTION_TIME_SHIFT

Quoted 30 lines · On Fri, 13 Dec 2019 17:30:23 +0200 "Tzvetomir Stoyanov (VMware)"…
lib/trace-cmd/trace-input.c +22 -0
@@ -2370,6 +2489,28 @@ static int handle_options(struct tracecmd_input *handle)
 			offset = strtoll(buf, NULL, 0);
 			handle->ts_offset += offset;
 			break;
+		case TRACECMD_OPTION_TIME_SHIFT:
+			/*
+			 * long long int (8 bytes) trace session ID
+			 * int (4 bytes) count of timestamp offsets.
+			 * long long array of size [count] of times,
+			 *      when the offsets were calculated.
+			 * long long array of size [count] of timestamp offsets.
+			 */
+			if (size < 12 || handle->flags & TRACECMD_FL_IGNORE_DATE)
+				break;
+			handle->host.trace_id = tep_read_number(handle->pevent,
+								buf, 8);
+			handle->host.ts_samples_count = tep_read_number(handle->pevent,
+									buf + 8, 4);
+			samples_size = (8 * handle->host.ts_samples_count);
+			if (size != (12 + (2 * samples_size)))
+				break;
+			handle->host.ts_samples = malloc(2 * samples_size);
+			if (!handle->host.ts_samples)
+				return -ENOMEM;
+			tsync_offset_load(handle, buf + 12);
+			break;
 		case TRACECMD_OPTION_CPUSTAT:
 			buf[size-1] = '\n';
 			cpustats = realloc(cpustats, cpustats_size + size + 1);

lib/trace-cmd/trace-input.c void tracecmd_close(struct tracecmd_input *handle) · line 3222

Quoted 4 lines
lib/trace-cmd/trace-input.c +1 -0
@@ -3078,6 +3219,8 @@ void tracecmd_close(struct tracecmd_input *handle)
 	trace_pid_map_free(handle->pid_maps);
 	handle->pid_maps = NULL;
 
+	trace_tsync_offset_free(&handle->host);

You need to add in tracecmd_buffer_instance_handle() something like:

	memset(&new_handle->host, 0, sizeof(new_handle->host));

or make a copy of it. Because if the host created sub buffers along with the top level, you get multiple frees on the host.ts_samples, as the new_handle will have a pointer to the original handle.

This is the same bug fixed by ca4b997b6505df3ce for trace_clock.

-- Steve

Quoted 4 lines
lib/trace-cmd/trace-input.c +1 -0
+
 	if (handle->flags & TRACECMD_FL_BUFFER_INSTANCE)
 		tracecmd_close(handle->parent);
 	else {
#19 lore raw

PATCH v18 13/18 trace-cmd: Add guest information in host's trace.dat file

In reply to Tzvetomir Stoyanov (VMware)

New trace.dat option is introduced: TRACECMD_OPTION_GUEST. Written in the host's trace.dat file, it contains information about guests, traced at the same time: guest trace ID, number of VCPUs and PIDs of the host tasks, running those VCPU. The data is stored in the file as NULL terminated string:

	"Guest %s %llu %d\n" -> guest name, number of VCPUs
	"%d %d\n" -> VCPU, PID of host task
	.....
	"%d %d\n" -> VCPU, PID of host task
---
 include/trace-cmd/trace-cmd.h |   5 ++
 lib/trace-cmd/trace-input.c   | 110 ++++++++++++++++++++++++++++++++++
 tracecmd/trace-record.c       |  46 ++++++++++++++
 3 files changed, 161 insertions(+)
include/trace-cmd/trace-cmd.h +5 -0
index b06a0c3..da8eeaf 100644
@@ -88,6 +88,7 @@ enum {
 	TRACECMD_OPTION_PROCMAPS,
 	TRACECMD_OPTION_TRACEID,
 	TRACECMD_OPTION_TIME_SHIFT,
+	TRACECMD_OPTION_GUEST,
 };
 
 enum {
@@ -131,6 +132,10 @@ void tracecmd_set_flag(struct tracecmd_input *handle, int flag);
 void tracecmd_clear_flag(struct tracecmd_input *handle, int flag);
 unsigned long tracecmd_get_flags(struct tracecmd_input *handle);
 unsigned long long tracecmd_get_traceid(struct tracecmd_input *handle);
+int tracecmd_get_guest_cpumap(struct tracecmd_input *handle,
+			      unsigned long long trace_id,
+			      char **name,
+			      int *vcpu_count, int **cpu_pid);
 unsigned long long tracecmd_get_tsync_peer(struct tracecmd_input *handle);
 int tracecmd_enable_tsync(struct tracecmd_input *handle, bool enable);
 
lib/trace-cmd/trace-input.c +110 -0

146 lines not on this page · raw message

tracecmd/trace-record.c +46 -0

64 lines not on this page · raw message

Signature · 1 line
2.23.0
#20 lore raw

Re: [PATCH v18 13/18] trace-cmd: Add guest information in host's trace.dat file

Quoted 9 lines · On Fri, 13 Dec 2019 17:30:24 +0200 "Tzvetomir Stoyanov (VMware)"…

New trace.dat option is introduced: TRACECMD_OPTION_GUEST. Written in the host's trace.dat file, it contains information about guests, traced at the same time: guest trace ID, number of VCPUs and PIDs of the host tasks, running those VCPU. The data is stored in the file as NULL terminated string:

	"Guest %s %llu %d\n" -> guest name, number of VCPUs
	"%d %d\n" -> VCPU, PID of host task
	.....
	"%d %d\n" -> VCPU, PID of host task

Why is this a string, and not a structure?

-- Steve

#21 lore raw

Re: [PATCH v18 13/18] trace-cmd: Add guest information in host's trace.dat file

Quoted 14 lines · Steven Rostedt
Quoted 9 lines · On Fri, 13 Dec 2019 17:30:24 +0200 "Tzvetomir Stoyanov (VMware)"…

New trace.dat option is introduced: TRACECMD_OPTION_GUEST. Written in the host's trace.dat file, it contains information about guests, traced at the same time: guest trace ID, number of VCPUs and PIDs of the host tasks, running those VCPU. The data is stored in the file as NULL terminated string:

      "Guest %s %llu %d\n" -> guest name, number of VCPUs
      "%d %d\n" -> VCPU, PID of host task
      .....
      "%d %d\n" -> VCPU, PID of host task
Quoted 1 line

Why is this a string, and not a structure?

To avoid endian long long conversions, and to simplify reading of trace.dat file, for debug purposes.
I would suggest to keep it text, for two reasons:
- this information is not huge, the binary format will not reduce the
size of the trace.dat file significantly.
- it is very useful to be possible to check what guest information is
stored in the file, using simple cat

We could think of implementing some stand alone trace.dat file reader, to extract and display that meta data information.

Quoted 2 lines

-- Steve

Signature · 2 lines
Tzvetomir (Ceco) Stoyanov
VMware Open Source Technology Center
#22 lore raw

Re: [PATCH v18 13/18] trace-cmd: Add guest information in host's trace.dat file

Quoted 2 lines · On Mon, 6 Jan 2020 16:43:03 +0200 Tzvetomir Stoyanov <[email protected]>…

We could think of implementing some stand alone trace.dat file reader, to extract and display that meta data information.

This is the right thing to do. I've actually started doing this here and there, but mostly via the trace-cmd report options.

But we should have a full parser of the trace.dat file. Could be an option for trace-cmd. "trace-cmd dump"?

-- Steve

#23 lore raw

PATCH v18 14/18 trace-cmd: Add host trace clock as guest trace argument

In reply to Tzvetomir Stoyanov (VMware)

When tracing host and guest machines, both should use the same tracing clock for event timestamps. If a clock is specified as host tracing argument, with option "-C clock_name", the same is injected as guest tracing argument. If the user wants to use different tracing clocks, it can specify it using "-C clock_name" as guest tracing argument. In that case, the one specified by the user has higher priority.

---
 tracecmd/include/trace-local.h |  1 +
 tracecmd/trace-record.c        | 24 ++++++++++++++++++++++--
 2 files changed, 23 insertions(+), 2 deletions(-)
tracecmd/include/trace-local.h +1 -0
index 90933cc..4c8d6cb 100644
@@ -165,6 +165,7 @@ enum buffer_instance_flags {
 	BUFFER_FL_PROFILE	= 1 << 1,
 	BUFFER_FL_GUEST		= 1 << 2,
 	BUFFER_FL_AGENT		= 1 << 3,
+	BUFFER_FL_HAS_CLOCK	= 1 << 4,
 };
 
 struct func_list {
tracecmd/trace-record.c +22 -2
index e4e2d63..51b40ff 100644
@@ -5405,6 +5405,8 @@ static void parse_record_options(int argc,
 	char *sav;
 	int name_counter = 0;
 	int neg_event = 0;
+	struct buffer_instance *instance;
+	bool guest_sync_set = false;
 
 	init_common_record_context(ctx, curr_cmd);
 
@@ -5448,6 +5450,8 @@ static void parse_record_options(int argc,
 		 */
 		if (c != 'B' && c != 'A' && is_guest(ctx->instance)) {
 			add_arg(ctx->instance, c, opts, long_options, optarg);
+			if (c == 'C')
+				ctx->instance->flags |= BUFFER_FL_HAS_CLOCK;
 			continue;
 		}
 
@@ -5562,6 +5566,9 @@ static void parse_record_options(int argc,
 			break;
 		case 'C':
 			tracecmd_alloc_instance_clock(ctx->instance->ftrace, optarg);
+			ctx->instance->flags |= BUFFER_FL_HAS_CLOCK;
+			if (is_top_instance(ctx->instance))
+				guest_sync_set = true;
 			break;
 		case 'v':
 			neg_event = 1;
@@ -5779,14 +5786,27 @@ static void parse_record_options(int argc,
 
 	/* If --date is specified, prepend it to all guest VM flags */
 	if (ctx->date) {
-		struct buffer_instance *instance;
-
 		for_all_instances(instance) {
 			if (is_guest(instance))
 				add_argv(instance, "--date", true);
 		}
 	}
 
+	if (guest_sync_set) {
+		/* If -C is specified, prepend clock to all guest VM flags */
+		for_all_instances(instance) {
+			if (tracecmd_get_instance_clock(top_instance.ftrace)) {
+				if (is_guest(instance) &&
+				    !(instance->flags & BUFFER_FL_HAS_CLOCK)) {
+					add_argv(instance,
+						 tracecmd_get_instance_clock(top_instance.ftrace),
+						 true);
+					add_argv(instance, "-C", true);
+				}
+			}
+		}
+	}
+
 	if (!ctx->filtered && ctx->instance->filter_mod)
 		add_func(&ctx->instance->filter_funcs,
 			 ctx->instance->filter_mod, "*");
Signature · 1 line
2.23.0
#24 lore raw

PATCH v18 15/18 trace-cmd: Refactor few trace-cmd internal functions.

In reply to Tzvetomir Stoyanov (VMware)

In order to reuse code inside trace-cmd application context, few functions are made non static and "trace_" is prepended to their names:

	int trace_make_vsock(unsigned int port);
	int trace_get_vsock_port(int sd, unsigned int *port);
	int trace_open_vsock(unsigned int cid, unsigned int port);
	char *trace_get_guest_file(const char *file, const char *guest);
---
 tracecmd/include/trace-local.h |  8 ++++++++
 tracecmd/trace-agent.c         | 10 +++++-----
 tracecmd/trace-record.c        | 16 ++++++++--------
 3 files changed, 21 insertions(+), 13 deletions(-)
tracecmd/include/trace-local.h +8 -0
index 4c8d6cb..d3e96c2 100644
@@ -19,6 +19,8 @@
 #define GUEST_FIFO_FMT		GUEST_DIR_FMT "/" GUEST_PIPE_NAME "%d"
 #define VIRTIO_FIFO_FMT		"/dev/virtio-ports/" GUEST_PIPE_NAME "%d"
 
+#define TRACE_FILENAME		"trace.dat"
+
 /* fix stupid glib guint64 typecasts and printf formats */
 typedef unsigned long long u64;
 
@@ -260,6 +262,12 @@ void tracecmd_disable_tracing(void);
 void tracecmd_enable_tracing(void);
 void tracecmd_stat_cpu(struct trace_seq *s, int cpu);
 
+int trace_make_vsock(unsigned int port);
+int trace_get_vsock_port(int sd, unsigned int *port);
+int trace_open_vsock(unsigned int cid, unsigned int port);
+
+char *trace_get_guest_file(const char *file, const char *guest);
+
 /* No longer in event-utils.h */
 void __noreturn die(const char *fmt, ...); /* Can be overriden */
 void *malloc_or_die(unsigned int size); /* Can be overridden */
tracecmd/trace-agent.c +5 -5
index cc330b7..db58064 100644
@@ -40,7 +40,7 @@ static int get_local_cid(unsigned int *cid)
 	return ret;
 }
 
-static int make_vsock(unsigned int port)
+int trace_make_vsock(unsigned int port)
 {
 	struct sockaddr_vm addr = {
 		.svm_family = AF_VSOCK,
@@ -64,7 +64,7 @@ static int make_vsock(unsigned int port)
 	return sd;
 }
 
-static int get_vsock_port(int sd, unsigned int *port)
+int trace_get_vsock_port(int sd, unsigned int *port)
 {
 	struct sockaddr_vm addr;
 	socklen_t addr_len = sizeof(addr);
@@ -87,11 +87,11 @@ static void make_vsocks(int nr, int *fds, unsigned int *ports)
 	int i, fd, ret;
 
 	for (i = 0; i < nr; i++) {
-		fd = make_vsock(VMADDR_PORT_ANY);
+		fd = trace_make_vsock(VMADDR_PORT_ANY);
 		if (fd < 0)
 			die("Failed to open vsocket");
 
-		ret = get_vsock_port(fd, &port);
+		ret = trace_get_vsock_port(fd, &port);
 		if (ret < 0)
 			die("Failed to get vsocket address");
 
@@ -208,7 +208,7 @@ static void agent_serve(unsigned int port)
 	nr_cpus = tracecmd_count_cpus();
 	page_size = getpagesize();
 
-	sd = make_vsock(port);
+	sd = trace_make_vsock(port);
 	if (sd < 0)
 		die("Failed to open vsocket");
 
tracecmd/trace-record.c +8 -8
index 51b40ff..e84b714 100644
@@ -68,7 +68,7 @@ static int rt_prio;
 
 static int keep;
 
-static const char *output_file = "trace.dat";
+static const char *output_file = TRACE_FILENAME;
 
 static int latency;
 static int sleep_time = 1000;
@@ -536,7 +536,7 @@ static char *get_temp_file(struct buffer_instance *instance, int cpu)
 	return file;
 }
 
-static char *get_guest_file(const char *file, const char *guest)
+char *trace_get_guest_file(const char *file, const char *guest)
 {
 	const char *p;
 	char *out = NULL;
@@ -2772,7 +2772,7 @@ static int connect_port(const char *host, unsigned int port)
 }
 
 #ifdef VSOCK
-static int open_vsock(unsigned int cid, unsigned int port)
+int trace_open_vsock(unsigned int cid, unsigned int port)
 {
 	struct sockaddr_vm addr = {
 		.svm_family = AF_VSOCK,
@@ -2835,7 +2835,7 @@ static bool can_splice_read_vsock(void)
 }
 
 #else
-static inline int open_vsock(unsigned int cid, unsigned int port)
+int trace_open_vsock(unsigned int cid, unsigned int port)
 {
 	die("vsock is not supported");
 	return -1;
@@ -3117,7 +3117,7 @@ create_recorder_instance(struct buffer_instance *instance, const char *file, int
 		if (instance->use_fifos)
 			fd = instance->fds[cpu];
 		else
-			fd = open_vsock(instance->cid, instance->client_ports[cpu]);
+			fd = trace_open_vsock(instance->cid, instance->client_ports[cpu]);
 		if (fd < 0)
 			die("Failed to connect to agent");
 
@@ -3508,7 +3508,7 @@ static void connect_to_agent(struct buffer_instance *instance)
 		use_fifos = nr_fifos > 0;
 	}
 
-	sd = open_vsock(instance->cid, instance->port);
+	sd = trace_open_vsock(instance->cid, instance->port);
 	if (sd < 0)
 		die("Failed to connect to vsocket @%u:%u",
 		    instance->cid, instance->port);
@@ -3562,7 +3562,7 @@ static void setup_guest(struct buffer_instance *instance)
 	int fd;
 
 	/* Create a place to store the guest meta data */
-	file = get_guest_file(output_file, tracecmd_get_instance_name(instance->ftrace));
+	file = trace_get_guest_file(output_file, tracecmd_get_instance_name(instance->ftrace));
 	if (!file)
 		die("Failed to allocate memory");
 
@@ -3887,7 +3887,7 @@ static void write_guest_file(struct buffer_instance *instance)
 	char **temp_files;
 	int i, fd;
 
-	file = get_guest_file(output_file, tracecmd_get_instance_name(instance->ftrace));
+	file = trace_get_guest_file(output_file, tracecmd_get_instance_name(instance->ftrace));
 	if (!file)
 		die("Failed to allocate memory");
 
Signature · 1 line
2.23.0
#25 lore raw

PATCH v18 16/18 trace-cmd: Basic infrastructure for host - guest timestamp synchronization

In reply to Tzvetomir Stoyanov (VMware)

The infrastructure for host - guest timestamp synchronization is divided in two parts:
- logic in libtracecmd
- logic in trace-cmd application

The libtracecmd is extended with new trace message, MSG_TIME_SYNC, used to exchange time synch information between host and guest. The trace request and response messages are extended to handle the time synch negotiation. The logic is implemented in trace-timesync.c file:
- Register / unregister time sync protocols.
- Choosing what protocol will be used for a tracing session.
- Init common context, used by all protocols - a ftrace instance, vsockets.
- Invoke protocol routines to perform time synchronization.
- Store calculated offsets in an array.

The trace-cmd application is extended to perform timestamp synchronization.
The main logic is in trace-tsync.c file
- Negotiate what time synch algorithm will be used for the tracing session.
- Run pthreads, to do continuous timestamp synchronization, during the trace.
- Store calculated offsets in guest's trace.dat file, using TRACECMD_OPTION_TIME_SHIFT option
- A new trace-cmd option is added, to control the timestamp synchronization:
--tsync-interval: set the time sync loop interval, in ms. The default
value is 0 - timestamp synchronization is performed twice, before and after the trace. If a negative number is specified, the timestamp synchronization is disabled.

---
 Documentation/trace-cmd-record.1.txt      |   7 +
 include/trace-cmd/trace-cmd.h             |  59 ++-
 lib/trace-cmd/Makefile                    |   1 +
 lib/trace-cmd/include/trace-tsync-local.h |  38 ++
 lib/trace-cmd/trace-msg.c                 | 394 ++++++++++++++---
 lib/trace-cmd/trace-timesync.c            | 496 ++++++++++++++++++++++
 tracecmd/Makefile                         |   3 +-
 tracecmd/include/trace-local.h            |  11 +
 tracecmd/trace-agent.c                    |  38 +-
 tracecmd/trace-record.c                   |  42 +-
 tracecmd/trace-tsync.c                    | 271 ++++++++++++
 tracecmd/trace-usage.c                    |   4 +
 12 files changed, 1297 insertions(+), 67 deletions(-)
 create mode 100644 lib/trace-cmd/include/trace-tsync-local.h
 create mode 100644 lib/trace-cmd/trace-timesync.c
 create mode 100644 tracecmd/trace-tsync.c
Documentation/trace-cmd-record.1.txt +7 -0
index 0d75e43..f5ecad5 100644
@@ -338,6 +338,13 @@ OPTIONS
     the offset will also be in nanoseconds even if the displayed units are
     in microseconds.
 
+*--tsync-interval*::
+    Set the loop interval, in ms, for timestamps synchronization with guests:
+        If a negative number is specified, timestamps synchronization is disabled
+        If 0 is specified, no loop is performed - timestamps offset is calculated only twice,"
+        at the beginning and at the end of the trace.
+        Timestamps synchronization with guests works only if there is support for VSOCK.\n"
+
 *--stderr*::
     Have output go to stderr instead of stdout, but the output of the command
     executed will not be changed. This is useful if you want to monitor the
include/trace-cmd/trace-cmd.h +55 -4

80 lines not on this page · raw message

lib/trace-cmd/Makefile +1 -0
index 18c7013..6221b36 100644
@@ -16,6 +16,7 @@ OBJS += trace-util.o
 OBJS += trace-instance.o
 OBJS += trace-filter-hash.o
 OBJS += trace-msg.o
+OBJS += trace-timesync.o
 
 # Additional util objects
 OBJS += trace-blk-hack.o
lib/trace-cmd/include/trace-tsync-local.h new file +38 -0
new file mode 100644
index 0000000..511c838
@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: LGPL-2.1 */
+/*
+ * Copyright (C) 2019, VMware, Tzvetomir Stoyanov <[email protected]>
+ *
+ */
+#ifndef _TRACE_TSYNC_LOCAL_H
+#define _TRACE_TSYNC_LOCAL_H
+
+#include <stdbool.h>
+
+struct clock_sync_context {
+	void				*proto_data;	/* time sync protocol specific data */
+	bool				server;		/* server or guest time sync role */
+	struct tracecmd_instance	*vinst;		/* ftrace buffer, used for time sync events */
+
+	/* Arrays with calculated time offsets at given time */
+	int				sync_size;	/* Allocated size of sync_ts and sync_offsets */
+	int				sync_count;	/* Number of elements in sync_ts and sync_offsets */
+	long long			*sync_ts;
+	long long			*sync_offsets;
+
+	/* Identifiers of local and remote time sync peers: cid and port */
+	unsigned int			local_cid;
+	unsigned int			local_port;
+	unsigned int			remote_cid;
+	unsigned int			remote_port;
+};
+
+int tracecmd_tsync_proto_register(unsigned int proto_id, int weight,
+				int (*init)(struct tracecmd_time_sync *),
+				int (*free)(struct tracecmd_time_sync *),
+				int (*calc)(struct tracecmd_time_sync *,
+					    long long *, long long *));
+int tracecmd_tsync_proto_unregister(unsigned int proto_id);
+
+int ptp_clock_sync_register(void);
+
+#endif /* _TRACE_TSYNC_LOCAL_H */
lib/trace-cmd/trace-msg.c +339 -55

533 lines not on this page · raw message

lib/trace-cmd/trace-timesync.c new file +496 -0

496 lines not on this page · raw message

tracecmd/Makefile +2 -1
index 29a623b..908dab2 100644
@@ -30,6 +30,7 @@ TRACE_CMD_OBJS += trace-check-events.o
 TRACE_CMD_OBJS += trace-show.o
 TRACE_CMD_OBJS += trace-list.o
 TRACE_CMD_OBJS += trace-usage.o
+TRACE_CMD_OBJS += trace-tsync.o
 
 ifeq ($(VSOCK_DEFINED), 1)
 TRACE_CMD_OBJS += trace-agent.o
@@ -42,7 +43,7 @@ all_objs := $(sort $(ALL_OBJS))
 all_deps := $(all_objs:$(bdir)/%.o=$(bdir)/.%.d)
 
 CONFIG_INCLUDES =
-CONFIG_LIBS	= -lrt
+CONFIG_LIBS	= -lrt -lpthread
 CONFIG_FLAGS	=
 
 all: $(TARGETS)
tracecmd/include/trace-local.h +11 -0
index d3e96c2..4a8125b 100644
@@ -228,6 +228,10 @@ struct buffer_instance {
 	unsigned int		port;
 	int			*fds;
 	bool			use_fifos;
+
+	pthread_t		tsync_thread;
+	bool			tsync_thread_running;
+	struct tracecmd_time_sync tsync;
 };
 
 void init_top_instance(void);
@@ -262,6 +266,13 @@ void tracecmd_disable_tracing(void);
 void tracecmd_enable_tracing(void);
 void tracecmd_stat_cpu(struct trace_seq *s, int cpu);
 
+int tracecmd_host_tsync(struct buffer_instance *instance,
+			 unsigned int tsync_port);
+void tracecmd_host_tsync_complete(struct buffer_instance *instance);
+unsigned int tracecmd_guest_tsync(char *tsync_protos,
+				  unsigned int tsync_protos_size, char *clock,
+				  unsigned int *tsync_port, pthread_t *thr_id);
+
 int trace_make_vsock(unsigned int port);
 int trace_get_vsock_port(int sd, unsigned int *port);
 int trace_open_vsock(unsigned int cid, unsigned int port);
tracecmd/trace-agent.c +36 -2
index db58064..d58e5e5 100644
@@ -19,6 +19,7 @@
 #include <sys/wait.h>
 #include <unistd.h>
 #include <linux/vm_sockets.h>
+#include <pthread.h>
 
 #include "trace-local.h"
 #include "trace-msg.h"
@@ -125,11 +126,30 @@ cleanup:
 	return ret;
 }
 
+static char *get_clock(int argc, char **argv)
+{
+	int i;
+
+	if (!argc || !argv)
+		return NULL;
+
+	for (i = 0; i < argc - 1; i++) {
+		if (!strcmp("-C", argv[i]))
+			return argv[i+1];
+	}
+	return NULL;
+}
+
 static void agent_handle(int sd, int nr_cpus, int page_size)
 {
 	struct tracecmd_msg_handle *msg_handle;
+	unsigned int tsync_protos_size = 0;
+	unsigned int tsync_proto = 0;
 	unsigned long long trace_id;
+	unsigned int tsync_port = 0;
+	char *tsync_protos = NULL;
 	unsigned int *ports;
+	pthread_t sync_thr;
 	char **argv = NULL;
 	int argc = 0;
 	bool use_fifos;
@@ -146,7 +166,8 @@ static void agent_handle(int sd, int nr_cpus, int page_size)
 		die("Failed to allocate message handle");
 
 	ret = tracecmd_msg_recv_trace_req(msg_handle, &argc, &argv,
-					  &use_fifos, &trace_id);
+					  &use_fifos, &trace_id,
+					  &tsync_protos, &tsync_protos_size);
 	if (ret < 0)
 		die("Failed to receive trace request");
 
@@ -155,14 +176,27 @@ static void agent_handle(int sd, int nr_cpus, int page_size)
 
 	if (!use_fifos)
 		make_vsocks(nr_cpus, fds, ports);
+	if (tsync_protos) {
+		tsync_proto = tracecmd_guest_tsync(tsync_protos,
+						   tsync_protos_size,
+						   get_clock(argc, argv),
+						   &tsync_port, &sync_thr);
+		if (!tsync_proto)
+			warning("Failed to negotiate timestamps synchronization with the host");
+	}
 	trace_id = tracecmd_generate_traceid();
 	ret = tracecmd_msg_send_trace_resp(msg_handle, nr_cpus, page_size,
-					   ports, use_fifos, trace_id);
+					   ports, use_fifos, trace_id,
+					   tsync_proto, tsync_port);
 	if (ret < 0)
 		die("Failed to send trace response");
 	trace_record_agent(msg_handle, nr_cpus, fds, argc, argv,
 			   use_fifos, trace_id);
 
+	if (tsync_proto)
+		pthread_join(sync_thr, NULL);
+
+	free(tsync_protos);
 	free(argv[0]);
 	free(argv);
 	free(ports);
tracecmd/trace-record.c +37 -5

99 lines not on this page · raw message

tracecmd/trace-tsync.c new file +271 -0

271 lines not on this page · raw message

tracecmd/trace-usage.c +4 -0
index 05ec021..9fa61e1 100644
@@ -60,6 +60,10 @@ static struct usage_help usage_help[] = {
 		"          --no-filter include trace-cmd threads in the trace\n"
 		"          --proc-map save the traced processes address map into the trace.dat file\n"
 		"          --user execute the specified [command ...] as given user\n"
+		"          --tsync-interval set the loop interval, in ms, for timestamps synchronization with guests:"
+		"               If a negative number is specified, timestamps synchronization is disabled"
+		"               If 0 is specified, no loop is performed - timestamps offset is calculated only twice,"
+		"                                                         at the beginnig and at the end of the trace\n"
 	},
 	{
 		"start",
Signature · 1 line
2.23.0
#26 lore raw

Re: [PATCH v18 16/18] trace-cmd: Basic infrastructure for host - guest timestamp synchronization

Quoted 170 lines · On Fri, 13 Dec 2019 17:30:27 +0200 "Tzvetomir Stoyanov (VMware)"…

The infrastructure for host - guest timestamp synchronization is divided in two parts:
- logic in libtracecmd
- logic in trace-cmd application

The libtracecmd is extended with new trace message, MSG_TIME_SYNC, used to exchange time synch information between host and guest. The trace request and response messages are extended to handle the time synch negotiation. The logic is implemented in trace-timesync.c file:
- Register / unregister time sync protocols.
- Choosing what protocol will be used for a tracing session.
- Init common context, used by all protocols - a ftrace instance, vsockets.
- Invoke protocol routines to perform time synchronization.
- Store calculated offsets in an array.

The trace-cmd application is extended to perform timestamp synchronization.
The main logic is in trace-tsync.c file
- Negotiate what time synch algorithm will be used for the tracing session.
- Run pthreads, to do continuous timestamp synchronization, during the trace.
- Store calculated offsets in guest's trace.dat file, using TRACECMD_OPTION_TIME_SHIFT option
- A new trace-cmd option is added, to control the timestamp synchronization:
--tsync-interval: set the time sync loop interval, in ms. The default
value is 0 - timestamp synchronization is performed twice, before and after the trace. If a negative number is specified, the timestamp synchronization is disabled.

---
 Documentation/trace-cmd-record.1.txt      |   7 +
 include/trace-cmd/trace-cmd.h             |  59 ++-
 lib/trace-cmd/Makefile                    |   1 +
 lib/trace-cmd/include/trace-tsync-local.h |  38 ++
 lib/trace-cmd/trace-msg.c                 | 394 ++++++++++++++---
 lib/trace-cmd/trace-timesync.c            | 496 ++++++++++++++++++++++
 tracecmd/Makefile                         |   3 +-
 tracecmd/include/trace-local.h            |  11 +
 tracecmd/trace-agent.c                    |  38 +-
 tracecmd/trace-record.c                   |  42 +-
 tracecmd/trace-tsync.c                    | 271 ++++++++++++
 tracecmd/trace-usage.c                    |   4 +
 12 files changed, 1297 insertions(+), 67 deletions(-)
 create mode 100644 lib/trace-cmd/include/trace-tsync-local.h
 create mode 100644 lib/trace-cmd/trace-timesync.c
 create mode 100644 tracecmd/trace-tsync.c
Documentation/trace-cmd-record.1.txt +7 -0
index 0d75e43..f5ecad5 100644
@@ -338,6 +338,13 @@ OPTIONS
     the offset will also be in nanoseconds even if the displayed units are
     in microseconds.
 
+*--tsync-interval*::
+    Set the loop interval, in ms, for timestamps synchronization with guests:
+        If a negative number is specified, timestamps synchronization is disabled
+        If 0 is specified, no loop is performed - timestamps offset is calculated only twice,"
+        at the beginning and at the end of the trace.
+        Timestamps synchronization with guests works only if there is support for VSOCK.\n"
+
 *--stderr*::
     Have output go to stderr instead of stdout, but the output of the command
     executed will not be changed. This is useful if you want to monitor the
include/trace-cmd/trace-cmd.h +55 -4
index da8eeaf..bf52785 100644
@@ -362,19 +362,38 @@ void tracecmd_msg_set_done(struct tracecmd_msg_handle *msg_handle);
 
 int tracecmd_msg_send_trace_req(struct tracecmd_msg_handle *msg_handle,
 				int argc, char **argv, bool use_fifos,
-				unsigned long long trace_id);
+				unsigned long long trace_id,
+				char *tsync_protos,
+				int tsync_protos_size);
+
 int tracecmd_msg_recv_trace_req(struct tracecmd_msg_handle *msg_handle,
 				int *argc, char ***argv, bool *use_fifos,
-				unsigned long long *trace_id);
+				unsigned long long *trace_id,
+				char **tsync_protos,
+				unsigned int *tsync_protos_size);
 
 int tracecmd_msg_send_trace_resp(struct tracecmd_msg_handle *msg_handle,
 				 int nr_cpus, int page_size,
 				 unsigned int *ports, bool use_fifos,
-				 unsigned long long trace_id);
+				 unsigned long long trace_id,
+				 unsigned int tsync_proto,
+				 unsigned int tsync_port);
 int tracecmd_msg_recv_trace_resp(struct tracecmd_msg_handle *msg_handle,
 				 int *nr_cpus, int *page_size,
 				 unsigned int **ports, bool *use_fifos,
-				 unsigned long long *trace_id);
+				 unsigned long long *trace_id,
+				 unsigned int *tsync_proto,
+				 unsigned int *tsync_port);
+
+int tracecmd_msg_send_time_sync(struct tracecmd_msg_handle *msg_handle,
+				unsigned int sync_protocol,
+				unsigned int sync_msg_id,
+				unsigned int payload_size, char *payload);
+int tracecmd_msg_recv_time_sync(struct tracecmd_msg_handle *msg_handle,
+				unsigned int *sync_protocol,
+				unsigned int *sync_msg_id,
+				unsigned int *payload_size, char **payload);
+
 /* --- ftrace instances --- */
 
 struct tracecmd_instance;
@@ -407,6 +426,38 @@ int tracecmd_iterate_raw_events(struct tep_handle *tep,
 				void *callback_context);
 
 
+/* --- Timestamp synchronization --- */
+
+enum{
+	TRACECMD_TIME_SYNC_PROTO_NONE	= 0,
+};
+enum{
+	TRACECMD_TIME_SYNC_CMD_PROBE	= 1,
+	TRACECMD_TIME_SYNC_CMD_STOP	= 2,
+};
+
+#define TRACECMD_TIME_SYNC_PROTO_PTP_WEIGHT	10
+
+struct tracecmd_time_sync {
+	unsigned int			sync_proto;
+	int				loop_interval;
+	pthread_mutex_t			lock;
+	pthread_cond_t			cond;
+	char				*clock_str;
+	struct tracecmd_msg_handle	*msg_handle;
+	void				*context;
+};
+
+void tracecmd_tsync_init(void);
+int tracecmd_tsync_proto_getall(char **proto_mask, int *words);
+unsigned int tracecmd_tsync_proto_select(char *proto_mask, int words);
+bool tsync_proto_is_supported(unsigned int proto_id);
+void tracecmd_tsync_with_host(struct tracecmd_time_sync *tsync);
+void tracecmd_tsync_with_guest(struct tracecmd_time_sync *tsync);
+int tracecmd_tsync_get_offsets(struct tracecmd_time_sync *tsync,
+				int *count,
+				long long **ts, long long **offsets);
+void tracecmd_tsync_free(struct tracecmd_time_sync *tsync);
 
 /* --- Plugin handling --- */
 extern struct tep_plugin_option trace_ftrace_options[];
lib/trace-cmd/Makefile +1 -0
index 18c7013..6221b36 100644
@@ -16,6 +16,7 @@ OBJS += trace-util.o
 OBJS += trace-instance.o
 OBJS += trace-filter-hash.o
 OBJS += trace-msg.o
+OBJS += trace-timesync.o
 
 # Additional util objects
 OBJS += trace-blk-hack.o
lib/trace-cmd/include/trace-tsync-local.h new file +3 -0
new file mode 100644
index 0000000..511c838
@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: LGPL-2.1 */
+/*
+ * Copyright (C) 2019, VMware, Tzvetomir Stoyanov <[email protected]>

lib/trace-cmd/include/trace-tsync-local.h line 13

Quoted 10 lines
lib/trace-cmd/include/trace-tsync-local.h new file +10 -0
+ *
+ */
+#ifndef _TRACE_TSYNC_LOCAL_H
+#define _TRACE_TSYNC_LOCAL_H
+
+#include <stdbool.h>
+
+struct clock_sync_context {
+	void				*proto_data;	/* time sync protocol specific data */
+	bool				server;		/* server or guest time sync role */

Let's call this "is_server" as it is a boolean value. "server" looks like it should be a pointer or something.

lib/trace-cmd/include/trace-tsync-local.h line 14

Quoted 1 line
lib/trace-cmd/include/trace-tsync-local.h new file +1 -0
+	struct tracecmd_instance	*vinst;		/* ftrace buffer, used for time sync events */

As this is part of a structure and not a global that would interfere with other name spaces, I think it should be called "instance", as I had no idea what "vinst" was when looking at it below. Let's be consistent and call struct tracecmd_instance "instance" when we can. That way we know exactly what it is by looking at the name.

Quoted 110 lines
lib/trace-cmd/include/trace-tsync-local.h new file +24 -0
+
+	/* Arrays with calculated time offsets at given time */
+	int				sync_size;	/* Allocated size of sync_ts and sync_offsets */
+	int				sync_count;	/* Number of elements in sync_ts and sync_offsets */
+	long long			*sync_ts;
+	long long			*sync_offsets;
+
+	/* Identifiers of local and remote time sync peers: cid and port */
+	unsigned int			local_cid;
+	unsigned int			local_port;
+	unsigned int			remote_cid;
+	unsigned int			remote_port;
+};
+
+int tracecmd_tsync_proto_register(unsigned int proto_id, int weight,
+				int (*init)(struct tracecmd_time_sync *),
+				int (*free)(struct tracecmd_time_sync *),
+				int (*calc)(struct tracecmd_time_sync *,
+					    long long *, long long *));
+int tracecmd_tsync_proto_unregister(unsigned int proto_id);
+
+int ptp_clock_sync_register(void);
+
+#endif /* _TRACE_TSYNC_LOCAL_H */
lib/trace-cmd/trace-msg.c +35 -2
index a34297f..551b9a0 100644
@@ -28,6 +28,8 @@
 
 typedef __u32 u32;
 typedef __be32 be32;
+typedef __u64 u64;
+typedef __s64 s64;
 
 static inline void dprint(const char *fmt, ...)
 {
@@ -60,9 +62,20 @@ struct tracecmd_msg_rinit {
 	be32 cpus;
 } __attribute__((packed));
 
+#define TRACE_REQ_PARAM_SIZE  (2 * sizeof(int))
+enum trace_req_params {
+	TRACE_REQUEST_ARGS,
+	TRACE_REQUEST_TSYNC_PROTOS,
+};
+
+struct tracecmd_msg_trace_req_param {
+	int id;
+	int length;
+	char *value;
+};
+
 struct tracecmd_msg_trace_req {
 	be32 flags;
-	be32 argc;
 	u64 trace_id;
 } __attribute__((packed));
 
@@ -71,6 +84,13 @@ struct tracecmd_msg_trace_resp {
 	be32 cpus;
 	be32 page_size;
 	u64 trace_id;
+	be32 tsync_proto;
+	be32 tsync_port;
+} __attribute__((packed));
+
+struct tracecmd_msg_tsync {
+	be32 sync_protocol;
+	be32 sync_msg_id;
 } __attribute__((packed));
 
 struct tracecmd_msg_header {
@@ -88,7 +108,8 @@ struct tracecmd_msg_header {
 	C(NOT_SUPP,	5,	0),					\
 	C(TRACE_REQ,	6,	sizeof(struct tracecmd_msg_trace_req)),	\
 	C(TRACE_RESP,	7,	sizeof(struct tracecmd_msg_trace_resp)),\
-	C(CLOSE_RESP,	8,	0),
+	C(CLOSE_RESP,	8,	0),					\
+	C(TIME_SYNC,	9,	sizeof(struct tracecmd_msg_tsync)),
 
 #undef C
 #define C(a,b,c)	MSG_##a = b
@@ -122,6 +143,7 @@ struct tracecmd_msg {
 		struct tracecmd_msg_rinit	rinit;
 		struct tracecmd_msg_trace_req	trace_req;
 		struct tracecmd_msg_trace_resp	trace_resp;
+		struct tracecmd_msg_tsync	tsync;
 	};
 	char					*buf;
 } __attribute__((packed));
@@ -264,6 +286,17 @@ static int tracecmd_msg_send(int fd, struct tracecmd_msg *msg)
 	return ret;
 }
 
+static int msg_send_nofree(int fd, struct tracecmd_msg *msg)
+{
+	int ret = 0;
+
+	ret = msg_write(fd, msg);
+	if (ret < 0)
+		ret = -ECOMM;
+
+	return ret;
+}
+
 static int msg_read(int fd, void *buf, u32 size, int *n)
 {
 	ssize_t r;

lib/trace-cmd/trace-msg.c int tracecmd_msg_wait_close_resp(struct tracecmd_msg_handle *msg_handle) · line 852

Quoted 9 lines
lib/trace-cmd/trace-msg.c +4 -2
@@ -813,82 +846,142 @@ int tracecmd_msg_wait_close_resp(struct tracecmd_msg_handle *msg_handle)
 	return tracecmd_msg_wait_for_cmd(msg_handle, MSG_CLOSE_RESP);
 }
 
-static int make_trace_req(struct tracecmd_msg *msg, int argc, char **argv,
-			  bool use_fifos, unsigned long long trace_id)
+static int make_trace_req_protos(char **buf, int *size,
+				 int protos_size, char *tsync_protos)
+{
+	size_t buf_size = 0;

Why initialize buf_size to zero, when you set it below in the first instruction?

Quoted 17 lines
lib/trace-cmd/trace-msg.c +17 -0
+	char *nbuf;
+	char *p;
+
+	buf_size = TRACE_REQ_PARAM_SIZE + protos_size;
+	nbuf = realloc(*buf, *size + buf_size);
+	if (!nbuf)
+		return -1;
+
+	p = nbuf + *size;
+	memset(p, 0, buf_size);
+
+	*(unsigned int *)p = htonl(TRACE_REQUEST_TSYNC_PROTOS);
+	p += sizeof(int);
+	*(unsigned int *)p = htonl(protos_size);
+	p += sizeof(int);
+
+	memcpy(p, tsync_protos, protos_size);

lib/trace-cmd/trace-msg.c int tracecmd_msg_wait_close_resp(struct tracecmd_msg_handle *msg_handle) · line 879

Quoted 10 lines
lib/trace-cmd/trace-msg.c +8 -0
+
+	*size += buf_size;
+	*buf = nbuf;
+	return 0;
+}
+
+static int make_trace_req_args(char **buf, int *size, int argc, char **argv)
 {
 	size_t args_size = 0;
+	size_t buf_size = 0;

Same here. args_size and buf_size are not used before they are assigned.

Quoted 39 lines
lib/trace-cmd/trace-msg.c +21 -8
+	char *nbuf;
 	char *p;
 	int i;
 
+	args_size = sizeof(int);
 	for (i = 0; i < argc; i++)
 		args_size += strlen(argv[i]) + 1;
 
-	msg->hdr.size = htonl(ntohl(msg->hdr.size) + args_size);
-	msg->trace_req.flags = use_fifos ? htonl(MSG_TRACE_USE_FIFOS) : htonl(0);
-	msg->trace_req.argc = htonl(argc);
-	msg->trace_req.trace_id = htonll(trace_id);
-	msg->buf = calloc(args_size, 1);
-	if (!msg->buf)
-		return -ENOMEM;
+	buf_size = TRACE_REQ_PARAM_SIZE + args_size;
+	nbuf = realloc(*buf, *size + buf_size);
+	if (!nbuf)
+		return -1;
+
+	p = nbuf + *size;
+	memset(p, 0, buf_size);
+
+	*(unsigned int *)p = htonl(TRACE_REQUEST_ARGS);
+	p += sizeof(int);
+	*(unsigned int *)p = htonl(args_size);
+	p += sizeof(int);
 
-	p = msg->buf;
+	*(unsigned int *)p = htonl(argc);
+	p += sizeof(int);
 	for (i = 0; i < argc; i++)
 		p = stpcpy(p, argv[i]) + 1;
 
+	*size += buf_size;
+	*buf = nbuf;
+	return 0;
+}
+

[..]

lib/trace-cmd/trace-msg.c

Quoted 3 lines
lib/trace-cmd/trace-msg.c +3 -0
+
+
+#define PROTO_MASK_SIZE (sizeof(char))

I'm thinking that declaring a mask the size of 1 is a bit overkill.

lib/trace-cmd/trace-msg.c

Quoted 11 lines
lib/trace-cmd/trace-msg.c +11 -0
+/**
+ * tracecmd_tsync_proto_select - Select time sync protocol, to be used for
+ *		timestamp synchronization with a peer
+ *
+ * @proto_mask: bitmask array of time sync protocols, supported by the peer
+ * @words: size of the @protos array
+ *
+ * Retuns Id of a time sync protocol, that can be used with the peer, or 0
+ *	  in case there is no match with supported protocols
+ */
+unsigned int tracecmd_tsync_proto_select(char *proto_mask, int words)

Probably should just call it "length", as words usually means "int" or "long".

lib/trace-cmd/trace-msg.c

Quoted 9 lines
lib/trace-cmd/trace-msg.c +9 -0
+{
+	struct tsync_proto *selected = NULL;
+	struct tsync_proto *proto;
+	int word;
+	int id;
+
+	for (word = 0; word < words; word++) {
+		for (proto = tsync_proto_list; proto; proto = proto->next) {
+			if (proto->proto_id < word * PROTO_MASK_SIZE)

Then we can remove the multiplier here.

Also, if the tsync_proto_list is sorted, we could optimize this. But that's an optimization for another day, as we currently only have a single proto.

Quoted 22 lines
lib/trace-cmd/trace-msg.c +22 -0
+				continue;
+
+			id = proto->proto_id - word * PROTO_MASK_SIZE;
+			if (id >= PROTO_MASK_SIZE)
+				continue;
+
+			if ((1 << id) & proto_mask[word]) {
+				if (selected) {
+					if (selected->weight < proto->weight)
+						selected = proto;
+				} else
+					selected = proto;
+			}
+		}
+	}
+
+	if (selected)
+		return selected->proto_id;
+
+	return 0;
+}
+

[..]

lib/trace-cmd/trace-msg.c

Quoted 4 lines
lib/trace-cmd/trace-msg.c +4 -0
+static struct tracecmd_instance *
+clock_synch_create_instance(const char *clock, unsigned int cid)
+{
+	struct tracecmd_instance *vinst;

Let's call this "instance".

Quoted 15 lines
lib/trace-cmd/trace-msg.c +15 -0
+	char inst_name[256];
+
+	snprintf(inst_name, 256, "clock_synch-%d", cid);
+
+	vinst = tracecmd_alloc_instance(inst_name);
+	if (!vinst)
+		return NULL;
+
+	tracecmd_make_instance(vinst);
+	tracecmd_write_instance_file(vinst, "trace", "\0", NULL);
+	tracecmd_alloc_instance_clock(vinst, clock);
+	tracecmd_set_clock(vinst, NULL);
+	return vinst;
+}
+

[..]

Quoted 22 lines
lib/trace-cmd/trace-msg.c +22 -0
+/**
+ * tracecmd_tsync_free - Free time sync context, allocated by
+ *		tracecmd_tsync_with_host() or tracecmd_tsync_with_guest() APIs
+ *
+ * @tsync: Pointer to time sync context
+ *
+ */
+void tracecmd_tsync_free(struct tracecmd_time_sync *tsync)
+{
+	struct clock_sync_context *tsync_context;
+	struct tsync_proto *proto;
+
+	if (!tsync->context)
+		return;
+	tsync_context = (struct clock_sync_context *)tsync->context;
+
+	proto = tsync_proto_find(tsync->sync_proto);
+	if (proto && proto->clock_sync_free)
+		proto->clock_sync_free(tsync);
+
+	clock_synch_delete_instance(tsync_context->vinst);
+	tsync_context->vinst = NULL;

lib/trace-cmd/trace-msg.c

Quoted 10 lines
lib/trace-cmd/trace-msg.c +10 -0
+
+	free(tsync_context->sync_ts);
+	free(tsync_context->sync_offsets);
+	tsync_context->sync_ts = NULL;
+	tsync_context->sync_offsets = NULL;
+	tsync_context->sync_count = 0;
+	tsync_context->sync_size = 0;
+	pthread_mutex_destroy(&tsync->lock);
+	pthread_cond_destroy(&tsync->cond);
+	free(tsync->clock_str);

I would think we would want a free(tsync) here. As the name of the function suggests.

Quoted 180 lines
lib/trace-cmd/trace-msg.c +176 -0
+}
+
+int tracecmd_tsync_send(struct tracecmd_time_sync *tsync,
+				  struct tsync_proto *proto)
+{
+	long long timestamp = 0;
+	long long offset = 0;
+	int ret;
+
+	ret = proto->clock_sync_calc(tsync, &offset, &timestamp);
+
+	return ret;
+}
+
 
 
 
+++ b/tracecmd/trace-tsync.c
@@ -0,0 +1,271 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2019, VMware, Tzvetomir Stoyanov <[email protected]>
+ *
+ */
+#include <stdlib.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <sys/socket.h>
+#include <unistd.h>
+#include <linux/vm_sockets.h>
+#include <pthread.h>
+
+#include "trace-local.h"
+#include "trace-msg.h"
+
+static int get_first_cpu(cpu_set_t **pin_mask, size_t *m_size)
+{
+	int cpus = tracecmd_count_cpus();
+	cpu_set_t *cpu_mask;
+	int mask_size;
+	int i;
+
+	cpu_mask = CPU_ALLOC(cpus);
+	*pin_mask = CPU_ALLOC(cpus);
+	if (!cpu_mask || !*pin_mask || 1)
+		goto error;
+
+	mask_size = CPU_ALLOC_SIZE(cpus);
+	CPU_ZERO_S(mask_size, cpu_mask);
+	CPU_ZERO_S(mask_size, *pin_mask);
+
+	if (sched_getaffinity(0, mask_size, cpu_mask) == -1)
+		goto error;
+
+	for (i = 0; i < cpus; i++) {
+		if (CPU_ISSET_S(i, mask_size, cpu_mask)) {
+			CPU_SET_S(i, mask_size, *pin_mask);
+			break;
+		}
+	}
+
+	if (CPU_COUNT_S(mask_size, *pin_mask) < 1)
+		goto error;
+
+	CPU_FREE(cpu_mask);
+	*m_size = mask_size;
+	return 0;
+
+error:
+	if (cpu_mask)
+		CPU_FREE(cpu_mask);
+	if (*pin_mask)
+		CPU_FREE(*pin_mask);
+	*pin_mask = NULL;
+	*m_size = 0;
+	return -1;
+}
+
+static void *tsync_host_thread(void *data)
+{
+	struct tracecmd_time_sync *tsync = NULL;
+
+	tsync = (struct tracecmd_time_sync *)data;
+
+	tracecmd_tsync_with_guest(tsync);
+
+	tracecmd_msg_handle_close(tsync->msg_handle);
+	tsync->msg_handle = NULL;
+
+	pthread_exit(0);
+}
+
+int tracecmd_host_tsync(struct buffer_instance *instance,
+			 unsigned int tsync_port)
+{
+	struct tracecmd_msg_handle *msg_handle = NULL;
+	cpu_set_t *pin_mask = NULL;
+	pthread_attr_t attrib;
+	size_t mask_size = 0;
+	int ret;
+	int fd;
+
+	if (!instance->tsync.sync_proto)
+		return -1;
+
+	fd = trace_open_vsock(instance->cid, tsync_port);
+	if (fd < 0) {
+		ret = -1;
+		goto out;
+	}
+	msg_handle = tracecmd_msg_handle_alloc(fd, 0);
+	if (!msg_handle) {
+		ret = -1;
+		goto out;
+	}
+
+	instance->tsync.msg_handle = msg_handle;
+	if (tracecmd_get_instance_clock(top_instance.ftrace))
+		instance->tsync.clock_str = strdup(tracecmd_get_instance_clock(top_instance.ftrace));
+	pthread_mutex_init(&instance->tsync.lock, NULL);
+	pthread_cond_init(&instance->tsync.cond, NULL);
+
+	pthread_attr_init(&attrib);
+	pthread_attr_setdetachstate(&attrib, PTHREAD_CREATE_JOINABLE);
+	if (!get_first_cpu(&pin_mask, &mask_size))
+		pthread_attr_setaffinity_np(&attrib, mask_size, pin_mask);
+
+	ret = pthread_create(&instance->tsync_thread, &attrib,
+			     tsync_host_thread, &instance->tsync);
+	if (!ret)
+		instance->tsync_thread_running = true;
+	if (pin_mask)
+		CPU_FREE(pin_mask);
+	pthread_attr_destroy(&attrib);
+
+out:
+	if (ret) {
+		if (msg_handle)
+			tracecmd_msg_handle_close(msg_handle);
+	}
+
+	return ret;
+}
+
+static void write_guest_time_shift(struct buffer_instance *instance)
+{
+	struct tracecmd_output *handle;
+	struct iovec vector[4];
+	long long *offsets;
+	long long *ts;
+	char *file;
+	int count;
+	int ret;
+	int fd;
+
+	ret = tracecmd_tsync_get_offsets(&instance->tsync, &count, &ts, &offsets);
+	if (ret < 0 || !count || !ts || !offsets)
+		return;
+
+	file = trace_get_guest_file(TRACE_FILENAME,
+			      tracecmd_get_instance_name(instance->ftrace));
+	fd = open(file, O_RDWR);
+	if (fd < 0)
+		die("error opening %s", file);
+	free(file);
+	handle = tracecmd_get_output_handle_fd(fd);
+	vector[0].iov_len = 8;
+	vector[0].iov_base = &top_instance.trace_id;
+	vector[1].iov_len = 4;
+	vector[1].iov_base = &count;
+	vector[2].iov_len = 8 * count;
+	vector[2].iov_base = ts;
+	vector[3].iov_len = 8 * count;
+	vector[3].iov_base = offsets;
+	tracecmd_add_option_v(handle, TRACECMD_OPTION_TIME_SHIFT, vector, 4);
+	tracecmd_append_options(handle);
+	tracecmd_output_close(handle);
+#ifdef TSYNC_DEBUG
+	if (count > 1)
+		printf("Got %d timestamp synch samples for guest %s in %lld ns trace\n\r",

lib/trace-cmd/trace-msg.c line 171

Quoted 10 lines
lib/trace-cmd/trace-msg.c +10 -0
+			count, tracecmd_get_instance_name(instance->ftrace),
+			ts[count - 1] - ts[0]);
+#endif
+}
+
+void tracecmd_host_tsync_complete(struct buffer_instance *instance)
+{
+	if (!instance->tsync_thread_running)
+		return;
+

Probably should add a comment to what is going on here ;-)

Quoted 54 lines
lib/trace-cmd/trace-msg.c +54 -0
+	pthread_mutex_lock(&instance->tsync.lock);
+	pthread_cond_signal(&instance->tsync.cond);
+	pthread_mutex_unlock(&instance->tsync.lock);
+	pthread_join(instance->tsync_thread, NULL);
+	write_guest_time_shift(instance);
+	tracecmd_tsync_free(&instance->tsync);
+}
+
+static void *tsync_agent_thread(void *data)
+{
+	struct tracecmd_time_sync *tsync = NULL;
+	int sd;
+
+	tsync = (struct tracecmd_time_sync *)data;
+
+	while (true) {
+		sd = accept(tsync->msg_handle->fd, NULL, NULL);
+		if (sd < 0) {
+			if (errno == EINTR)
+				continue;
+			goto out;
+		}
+		break;
+	}
+	close(tsync->msg_handle->fd);
+	tsync->msg_handle->fd = sd;
+
+	tracecmd_tsync_with_host(tsync);
+
+out:
+	tracecmd_msg_handle_close(tsync->msg_handle);
+	tracecmd_tsync_free(tsync);
+	free(tsync);
+	close(sd);
+
+	pthread_exit(0);
+}
+
+unsigned int tracecmd_guest_tsync(char *tsync_protos,
+				  unsigned int tsync_protos_size, char *clock,
+				  unsigned int *tsync_port, pthread_t *thr_id)
+{
+	struct tracecmd_time_sync *tsync = NULL;
+	cpu_set_t *pin_mask = NULL;
+	pthread_attr_t attrib;
+	size_t mask_size = 0;
+	unsigned int proto;
+	int ret;
+	int fd;
+
+	fd = -1;
+	proto = tracecmd_tsync_proto_select(tsync_protos, tsync_protos_size);
+	if (!proto)
+		return 0;

lib/trace-cmd/trace-msg.c line 235

Quoted 10 lines
lib/trace-cmd/trace-msg.c +10 -0
+#ifdef VSOCK
+	fd = trace_make_vsock(VMADDR_PORT_ANY);
+	if (fd < 0)
+		goto error;
+
+	ret = trace_get_vsock_port(fd, tsync_port);
+	if (ret < 0)
+		goto error;
+#else
+	return 0;

If we have no synchronization support, shouldn't this return an error?

Quoted 20 lines
lib/trace-cmd/trace-msg.c +20 -0
+#endif
+
+	tsync = calloc(1, sizeof(struct tracecmd_time_sync));
+	tsync->msg_handle = tracecmd_msg_handle_alloc(fd, 0);
+	if (clock)
+		tsync->clock_str = strdup(clock);
+
+	pthread_attr_init(&attrib);
+	tsync->sync_proto = proto;
+	pthread_attr_setdetachstate(&attrib, PTHREAD_CREATE_JOINABLE);
+	if (!get_first_cpu(&pin_mask, &mask_size))
+		pthread_attr_setaffinity_np(&attrib, mask_size, pin_mask);
+
+	ret = pthread_create(thr_id, &attrib, tsync_agent_thread, tsync);
+
+	if (pin_mask)
+		CPU_FREE(pin_mask);
+	pthread_attr_destroy(&attrib);
+
+	if (ret)

lib/trace-cmd/trace-msg.c line 265

Quoted 10 lines
lib/trace-cmd/trace-msg.c +10 -0
+		goto error;
+
+	return proto;
+
+error:
+	if (tsync) {
+		if (tsync->msg_handle)
+			tracecmd_msg_handle_close(tsync->msg_handle);
+		if (tsync->clock_str)
+			free(tsync->clock_str);

You don't need the if statement. free() can take a NULL pointer.

Anyway, it's looking good. I like a lot of what you did. Especially with the helpers you created.

I still need to take a deeper look at this and the following patches. I'm thinking that this patch could possibly be broken up into two or three patches. One patch to create the protocol, another one or two that use the protocol.

Although we are very close to getting this in, I've been thinking more that we should release 2.9, without the time sync. We may need to just up the protocol when we implement the synchronization.

I'll look more at this on Monday.

Cheers!

-- Steve

Quoted 21 lines
lib/trace-cmd/trace-msg.c +6 -0
+		free(tsync);
+	}
+	if (fd > 0)
+		close(fd);
+	return 0;
+}
tracecmd/trace-usage.c +4 -0
index 05ec021..9fa61e1 100644
@@ -60,6 +60,10 @@ static struct usage_help usage_help[] = {
 		"          --no-filter include trace-cmd threads in the trace\n"
 		"          --proc-map save the traced processes address map into the trace.dat file\n"
 		"          --user execute the specified [command ...] as given user\n"
+		"          --tsync-interval set the loop interval, in ms, for timestamps synchronization with guests:"
+		"               If a negative number is specified, timestamps synchronization is disabled"
+		"               If 0 is specified, no loop is performed - timestamps offset is calculated only twice,"
+		"                                                         at the beginnig and at the end of the trace\n"
 	},
 	{
 		"start",
#27 lore raw

Re: [PATCH v18 16/18] trace-cmd: Basic infrastructure for host - guest timestamp synchronization

Quoted 1 line · Steven Rostedt

[ .. ]

Quoted 4 lines
Quoted 1 line

+#define PROTO_MASK_SIZE (sizeof(char))

I'm thinking that declaring a mask the size of 1 is a bit overkill.

This define is a leftover from the previous implementation of this bitmask, where
the size was 4 bytes. It was very easy to switch to 1 byte using the define, that's
why I would prefer to keep it - in case we decide to change the size again. The code below is written with the assumption that the size could be more than 1 byte.

[ .. ]

Quoted 21 lines
Quoted 21 lines

+ */

lib/trace-cmd/trace-timesync.c new file +20 -0
+void tracecmd_tsync_free(struct tracecmd_time_sync *tsync)
+{
+     struct clock_sync_context *tsync_context;
+     struct tsync_proto *proto;
+
+     if (!tsync->context)
+             return;
+     tsync_context = (struct clock_sync_context *)tsync->context;
+
+     proto = tsync_proto_find(tsync->sync_proto);
+     if (proto && proto->clock_sync_free)
+             proto->clock_sync_free(tsync);
+
+     clock_synch_delete_instance(tsync_context->vinst);
+     tsync_context->vinst = NULL;
+
+     free(tsync_context->sync_ts);
+     free(tsync_context->sync_offsets);
+     tsync_context->sync_ts = NULL;
+     tsync_context->sync_offsets = NULL;
Quoted 9 lines

lib/trace-cmd/trace-timesync.c line 330

Quoted 5 lines
lib/trace-cmd/trace-timesync.c new file +5 -0
+     tsync_context->sync_count = 0;
+     tsync_context->sync_size = 0;
+     pthread_mutex_destroy(&tsync->lock);
+     pthread_cond_destroy(&tsync->cond);
+     free(tsync->clock_str);

I would think we would want a free(tsync) here. As the name of the function suggests.

There is no API to allocate it, that's why this function does not free the memory. There is one use case where this memory is not allocated, and free(tsync) will not work for it.

[ .. ]

Quoted 21 lines
Quoted 21 lines
tracecmd/trace-tsync.c new file +21 -0
+
+unsigned int tracecmd_guest_tsync(char *tsync_protos,
+                               unsigned int tsync_protos_size, char *clock,
+                               unsigned int *tsync_port, pthread_t *thr_id)
+{
+     struct tracecmd_time_sync *tsync = NULL;
+     cpu_set_t *pin_mask = NULL;
+     pthread_attr_t attrib;
+     size_t mask_size = 0;
+     unsigned int proto;
+     int ret;
+     int fd;
+
+     fd = -1;
+     proto = tracecmd_tsync_proto_select(tsync_protos, tsync_protos_size);
+     if (!proto)
+             return 0;
+#ifdef VSOCK
+     fd = trace_make_vsock(VMADDR_PORT_ANY);
+     if (fd < 0)
+             goto error;
Quoted 8 lines

tracecmd/trace-tsync.c line 235

Quoted 6 lines
tracecmd/trace-tsync.c new file +6 -0
+
+     ret = trace_get_vsock_port(fd, tsync_port);
+     if (ret < 0)
+             goto error;
+#else
+     return 0;

If we have no synchronization support, shouldn't this return an error?

This function returns the id of negotiated time sync protocol. 0 means the negotiation was not successful, no protocol is selected. When the caller receives 0, it means we have no synchronization with the peer.

Quoted 1 line

[ .. ]

Quoted 15 lines

You don't need the if statement. free() can take a NULL pointer.

Anyway, it's looking good. I like a lot of what you did. Especially with the helpers you created.

I still need to take a deeper look at this and the following patches. I'm thinking that this patch could possibly be broken up into two or three patches. One patch to create the protocol, another one or two that use the protocol.

Although we are very close to getting this in, I've been thinking more that we should release 2.9, without the time sync. We may need to just up the protocol when we implement the synchronization.

Quoted 6 lines

I'll look more at this on Monday.

Cheers!

-- Steve

Thanks, Steven!
I'll send the next version, addressing your comments.

Quoted 24 lines
Quoted 23 lines
tracecmd/trace-tsync.c new file +6 -0
+             free(tsync);
+     }
+     if (fd > 0)
+             close(fd);
+     return 0;
+}
tracecmd/trace-usage.c +4 -0
index 05ec021..9fa61e1 100644
@@ -60,6 +60,10 @@ static struct usage_help usage_help[] = {
              "          --no-filter include trace-cmd threads in the trace\n"
              "          --proc-map save the traced processes address map into the trace.dat file\n"
              "          --user execute the specified [command ...] as given user\n"
+             "          --tsync-interval set the loop interval, in ms, for timestamps synchronization with guests:"
+             "               If a negative number is specified, timestamps synchronization is disabled"
+             "               If 0 is specified, no loop is performed - timestamps offset is calculated only twice,"
+             "                                                         at the beginnig and at the end of the trace\n"
      },
      {
              "start",
Signature · 2 lines
Tzvetomir (Ceco) Stoyanov
VMware Open Source Technology Center
#28 lore raw

Re: [PATCH v18 16/18] trace-cmd: Basic infrastructure for host - guest timestamp synchronization

Quoted 7 lines · On Fri, 31 Jan 2020 09:53:41 +0000 Tzvetomir Stoyanov <[email protected]>…
Quoted 1 line · Steven Rostedt

[ .. ]

Quoted 4 lines
Quoted 1 line

+#define PROTO_MASK_SIZE (sizeof(char))

I'm thinking that declaring a mask the size of 1 is a bit overkill.

Quoted 7 lines

This define is a leftover from the previous implementation of this bitmask, where
the size was 4 bytes. It was very easy to switch to 1 byte using the define, that's
why I would prefer to keep it - in case we decide to change the size again. The code below is written with the assumption that the size could be more than 1 byte.

OK, that makes more sense.

Quoted 32 lines

[ .. ]

Quoted 21 lines
Quoted 21 lines

+ */

lib/trace-cmd/trace-timesync.c new file +20 -0
+void tracecmd_tsync_free(struct tracecmd_time_sync *tsync)
+{
+     struct clock_sync_context *tsync_context;
+     struct tsync_proto *proto;
+
+     if (!tsync->context)
+             return;
+     tsync_context = (struct clock_sync_context *)tsync->context;
+
+     proto = tsync_proto_find(tsync->sync_proto);
+     if (proto && proto->clock_sync_free)
+             proto->clock_sync_free(tsync);
+
+     clock_synch_delete_instance(tsync_context->vinst);
+     tsync_context->vinst = NULL;
+
+     free(tsync_context->sync_ts);
+     free(tsync_context->sync_offsets);
+     tsync_context->sync_ts = NULL;
+     tsync_context->sync_offsets = NULL;
Quoted 9 lines

lib/trace-cmd/trace-timesync.c line 330

Quoted 5 lines
lib/trace-cmd/trace-timesync.c new file +5 -0
+     tsync_context->sync_count = 0;
+     tsync_context->sync_size = 0;
+     pthread_mutex_destroy(&tsync->lock);
+     pthread_cond_destroy(&tsync->cond);
+     free(tsync->clock_str);  

I would think we would want a free(tsync) here. As the name of the function suggests.

Quoted 3 lines

There is no API to allocate it, that's why this function does not free the memory. There is one use case where this memory is not allocated, and free(tsync) will not work for it.

Then perhaps we should call it: tracecmd_tsync_cleanup(), as that's what it is doing. _free() usually means you are actually freeing what you pass in.

Quoted 32 lines

[ .. ]

Quoted 21 lines
Quoted 21 lines
tracecmd/trace-tsync.c new file +21 -0
+
+unsigned int tracecmd_guest_tsync(char *tsync_protos,
+                               unsigned int tsync_protos_size, char *clock,
+                               unsigned int *tsync_port, pthread_t *thr_id)
+{
+     struct tracecmd_time_sync *tsync = NULL;
+     cpu_set_t *pin_mask = NULL;
+     pthread_attr_t attrib;
+     size_t mask_size = 0;
+     unsigned int proto;
+     int ret;
+     int fd;
+
+     fd = -1;
+     proto = tracecmd_tsync_proto_select(tsync_protos, tsync_protos_size);
+     if (!proto)
+             return 0;
+#ifdef VSOCK
+     fd = trace_make_vsock(VMADDR_PORT_ANY);
+     if (fd < 0)
+             goto error;
Quoted 8 lines

tracecmd/trace-tsync.c line 235

Quoted 6 lines
tracecmd/trace-tsync.c new file +6 -0
+
+     ret = trace_get_vsock_port(fd, tsync_port);
+     if (ret < 0)
+             goto error;
+#else
+     return 0;  

If we have no synchronization support, shouldn't this return an error?

Quoted 3 lines

This function returns the id of negotiated time sync protocol. 0 means the negotiation was not successful, no protocol is selected. When the caller receives 0, it means we have no synchronization with the peer.

OK, so this function definitely needs a "kernel doc" header explaining this.

-- Steve

#29 lore raw

PATCH v18 17/18 trace-cmd: [POC] PTP-like algorithm for host - guest timestamp synchronization

In reply to Tzvetomir Stoyanov (VMware)

PTP protocol is designed for synchronizing clocks of machines in a local network. The same approach can be used for host - guest timestamp synchronization. This implementation uses ftrace raw markers to track trace timestamps of PTP events. The patch is a POC, two different algorithms for PTP calculations are proposed:
- Choosing the sample with the fastest response time for calculating the clocks offset.
- Calculating the clocks offset using the average of all PTP samples.

The implementation can be tuned using those parameters:
- #define FASTEST_RESPONSE - is defined, the sample with the fastest response time is used for calculating the clocks offset. Otherwise the histogram of all samples is used.
- #define PTP_SYNC_LOOP 340 - defines the number of samples, used for one calculation.
- --tsync-interval - a trace-cmd argument, choose the intervals between offset calculations, performed continuously during the trace.
- #define TSYNC_DEBUG - if defined, a debug information is collected and stored in files, in the guest machine:
s-cid*.txt - For each offset calculation: host and guest clocks and calculated offset. res-cid*.txt - For each tracing session: all calculated clock offsets.

---
 include/trace-cmd/trace-cmd.h      |   1 +
 lib/trace-cmd/Makefile             |   1 +
 lib/trace-cmd/trace-timesync-ptp.c | 649 +++++++++++++++++++++++++++++
 lib/trace-cmd/trace-timesync.c     |   8 +
 tracecmd/trace-agent.c             |   2 +
 tracecmd/trace-record.c            |  25 +-
 6 files changed, 678 insertions(+), 8 deletions(-)
 create mode 100644 lib/trace-cmd/trace-timesync-ptp.c
include/trace-cmd/trace-cmd.h +1 -0
index bf52785..45f7eae 100644
@@ -430,6 +430,7 @@ int tracecmd_iterate_raw_events(struct tep_handle *tep,
 
 enum{
 	TRACECMD_TIME_SYNC_PROTO_NONE	= 0,
+	TRACECMD_TIME_SYNC_PROTO_PTP	= 1,
 };
 enum{
 	TRACECMD_TIME_SYNC_CMD_PROBE	= 1,
lib/trace-cmd/Makefile +1 -0
index 6221b36..a2b9d08 100644
@@ -17,6 +17,7 @@ OBJS += trace-instance.o
 OBJS += trace-filter-hash.o
 OBJS += trace-msg.o
 OBJS += trace-timesync.o
+OBJS += trace-timesync-ptp.o
 
 # Additional util objects
 OBJS += trace-blk-hack.o
lib/trace-cmd/trace-timesync-ptp.c new file +649 -0

649 lines not on this page · raw message

lib/trace-cmd/trace-timesync.c +8 -0
index 6f23a63..d5620cf 100644
@@ -45,6 +45,14 @@ static struct tsync_proto *tsync_proto_find(unsigned int proto_id)
 	return NULL;
 }
 
+/**
+ * tracecmd_tsync_init - Initialize the global, per task, time sync data.
+ */
+void tracecmd_tsync_init(void)
+{
+	ptp_clock_sync_register();
+}
+
 int tracecmd_tsync_proto_register(unsigned int proto_id, int weight,
 				int (*init)(struct tracecmd_time_sync *),
 				int (*free)(struct tracecmd_time_sync *),
tracecmd/trace-agent.c +2 -0
index d58e5e5..37c7438 100644
@@ -246,6 +246,8 @@ static void agent_serve(unsigned int port)
 	if (sd < 0)
 		die("Failed to open vsocket");
 
+	tracecmd_tsync_init();
+
 	if (!get_local_cid(&cid))
 		printf("listening on @%u:%u\n", cid, port);
 
tracecmd/trace-record.c +17 -8
index 6654b5b..ba14969 100644
@@ -5952,10 +5952,7 @@ static void set_clock(struct buffer_instance *instance)
 		tracecmd_put_tracing_file(path);
 	}
 }
-/*
- * This function contains common code for the following commands:
- * record, start, stream, profile.
- */
+
 static void record_trace(int argc, char **argv,
 			 struct common_record_context *ctx)
 {
@@ -6088,12 +6085,24 @@ static void record_trace(int argc, char **argv,
 	finalize_record_trace(ctx);
 }
 
+/*
+ * This function contains common code for the following commands:
+ * record, start, stream, profile.
+ */
+static void record_trace_command(int argc, char **argv,
+				 struct common_record_context *ctx)
+{
+	tracecmd_tsync_init();
+	record_trace(argc, argv, ctx);
+}
+
+
 void trace_start(int argc, char **argv)
 {
 	struct common_record_context ctx;
 
 	parse_record_options(argc, argv, CMD_start, &ctx);
-	record_trace(argc, argv, &ctx);
+	record_trace_command(argc, argv, &ctx);
 	exit(0);
 }
 
@@ -6173,7 +6182,7 @@ void trace_stream(int argc, char **argv)
 	struct common_record_context ctx;
 
 	parse_record_options(argc, argv, CMD_stream, &ctx);
-	record_trace(argc, argv, &ctx);
+	record_trace_command(argc, argv, &ctx);
 	exit(0);
 }
 
@@ -6192,7 +6201,7 @@ void trace_profile(int argc, char **argv)
 	if (!buffer_instances)
 		top_instance.flags |= BUFFER_FL_PROFILE;
 
-	record_trace(argc, argv, &ctx);
+	record_trace_command(argc, argv, &ctx);
 	do_trace_profile();
 	exit(0);
 }
@@ -6211,7 +6220,7 @@ void trace_record(int argc, char **argv)
 	struct common_record_context ctx;
 
 	parse_record_options(argc, argv, CMD_record, &ctx);
-	record_trace(argc, argv, &ctx);
+	record_trace_command(argc, argv, &ctx);
 	exit(0);
 }
 
Signature · 1 line
2.23.0
#30 lore raw

PATCH v18 18/18 trace-cmd: Debug scripts for PTP-like algorithm for host - guest timestamp synchronization

In reply to Tzvetomir Stoyanov (VMware)

These scripts can be used to visualise debug files, written when the PTP-like algorithm is compiled with TSYNC_DEBUG defined. The files are located in the guest machine:
s-cid*.txt - For each offset calculation: host and guest clocks and calculated offset. res-cid*.txt - For each tracing session: all calculated clock offsets.

tsync_hist.py plots a histogram, using data from a s-cid*.txt file:

	"python tsync_hist.py s-cid2_1.txt"

tsync_res.py plots a line, using data from res-cid*.txt file:
"python tsync_res.py res-cid2.txt"

---
 scripts/debug/tsync_hist.py | 57 +++++++++++++++++++++++++++++++++++++
 scripts/debug/tsync_readme  | 12 ++++++++
 scripts/debug/tsync_res.py  | 46 ++++++++++++++++++++++++++++++
 3 files changed, 115 insertions(+)
 create mode 100644 scripts/debug/tsync_hist.py
 create mode 100644 scripts/debug/tsync_readme
 create mode 100644 scripts/debug/tsync_res.py
scripts/debug/tsync_hist.py new file +57 -0
new file mode 100644
index 0000000..819d1e8
@@ -0,0 +1,57 @@
+# SPDX-License-Identifier: GPL-2.0
+#
+# Copyright (C) 2019, VMware Inc, Tzvetomir Stoyanov <[email protected]>
+# Copyright (C) 2019, VMware Inc, Yordan Karadzhov <[email protected]>
+
+
+import matplotlib.pyplot as plt
+import matplotlib.lines as mlines
+import numpy as np
+import sys
+
+def newline(p1, p2):
+    ax = plt.gca()
+    xmin, xmax = ax.get_xbound()
+
+    if(p2[0] == p1[0]):
+        xmin = xmax = p1[0]
+        ymin, ymax = ax.get_ybound()
+    else:
+        ymax = p1[1]+(p2[1]-p1[1])/(p2[0]-p1[0])*(xmax-p1[0])
+        ymin = p1[1]+(p2[1]-p1[1])/(p2[0]-p1[0])*(xmin-p1[0])
+
+    l = mlines.Line2D([xmin,xmax], [ymin,ymax], color='red')
+    ax.add_line(l)
+    return l
+
+
+data = np.loadtxt(fname = sys.argv[1])
+selected_ts  = data[-1, 1]
+selected_ofs = data[-1, 0]
+data = data[:-1,:]
+
+x = data[:, 1] - data[:, 0]
+
+mean = x.mean()
+std = x.std()
+
+num_bins = 500
+min = x.min() #+ .4 * (x.max() - x.min())
+max = x.max() #- .4 * (x.max() - x.min())
+bins = np.linspace(min, max, num_bins, endpoint = False, dtype=int)
+
+fig, ax = plt.subplots()
+
+# the histogram of the data
+n, bins, patches = ax.hist(x, bins, histtype=u'step');
+
+ax.set_xlabel('clock offset [$\mu$s]')
+ax.set_ylabel('entries')
+ax.set_title("$\sigma$=%i" % std)
+
+x1, y1 = [selected_ofs, min], [selected_ofs, max]
+newline(x1, y1)
+
+# Tweak spacing to prevent clipping of ylabel
+fig.tight_layout()
+plt.show()
scripts/debug/tsync_readme new file +12 -0
new file mode 100644
index 0000000..f3ebb25
@@ -0,0 +1,12 @@
+PTP-like algorithm debug
+========================
+
+tsync_*.py scripts can be used to visualise debug files, written when the PTP-like algorithm
+is compiled with TSYNC_DEBUG defined. The files are located in the guest machine:
+    s-cid*.txt - For each offset calculation: host and guest clocks and calculated offset.
+    res-cid*.txt - For each tracing session: all calculated clock offsets.
+
+tsync_hist.py plots a histogram, using data from a s-cid*.txt file:
+	"python tsync_hist.py s-cid2_1.txt"
+tsync_res.py plots a line, using data from res-cid*.txt file:
+	"python tsync_res.py res-cid2.txt"
scripts/debug/tsync_res.py new file +46 -0

46 lines not on this page · raw message

Signature · 1 line
2.23.0