Timestamp synchronization of host - guest tracing session
v17 cover Timestamp synchronization of host - guest tracing session
Basic infrastructure for host - guest timestamp synchronization and a POC implementation of PTP-like algorithm.
[
v17 changes:
- Implemented new PTP logic for calculating the clocks offset, using histogram of all PTP samples. It gives better results than the logic with the fastest response time, so set the new one as default in the POC patch.
v16 changes:
- Fixed compilation in case no VSOCK is available (Thanks to Slavomir Kaslev)
- Fixed a typo in trace-cmd-record.1.txt (Thanks to Slavomir Kaslev)
- Added forgotten file in the patch "trace-cmd: Add new library APIs for ftrace instances." - trace-instance.c
- Fixed few compilation warnings related to TSYNC_DEBUG code.
- Removed a blank line at the end of "tsync_readme" file.
v15 changes:
- Removed the patch for "--proc-map" from the series, as it should not be part of it.
v14 changes:
- Bring back the PTP-like algorithm and removed the ftrace event based logic.
- Reimplemented the PTP-like algorithm to use raw ftrace markers, instead of clock_gettime() API.
- Refactored the logic to be algorithm independent and plugin friendly.
- Implemented continuous timestamps synchronization, while the trace is running.
- Moved logic from trace-cmd application to libtracecmd, as new library APIs.
- Implemented new trace id functionality.
- Implemented new guest section in host trace.dat file.
v13 changes:
- Remove few patches from the set, as they were merged.
- Rebased to the latest master, Slavomirs patchest "Add VM kernel tracing over vsockets and FIFOs" got merged!
v12 changes:
- Rebased on top of Slavomir's v13 "Add VM kernel tracing over vsockets and FIFOs"
v11 changes:
- Rebased on top of Slavomir's v10 "Add VM kernel tracing over vsockets and FIFOs"
- Addressed Slavomir's commnents from version 10 of the patch series.
v10 changes:
- Fixed broken compilation, call to timestamp_correction_calc() in timestamp_correct was smashed.
- Replaced deprecated tep_data_event_from_type() API with tep_find_event().
- Fixed a warning on assignment const to non const.
v9 changes:
- Fixed implementation of binary search algorithm in timestamp_correct()
v8 changes:
- Added rmdir() call in tracecmd_remove_instance(), to completely remove the instance. However, there is an issue with deleting the instances using rmdir(), which is investigated.
- Few changes in read_qemu_guests_pids(), timestamp_correct(), tsync_offset_load()
tracecmd_clock_context_new() and find_raw_events() suggested by Slavomir.
v7 changes:
- Added warning messages in case time synchronization cannot be negotiated or fails.
- Few optimizations and checks in read_qemu_guests_pids(), tsync_offset_load(), and find_raw_events(), suggested by Slavomir Kaslev.
- Reworked timestamp_correct() to not use static variables.
- Check TRACECMD_OPTION_TIME_SHIFT before reading time sync samples from the trace.dat file
v6 changes:
- Refactored tracecmd_msg_snd_time_sync() and tracecmd_msg_rcv_time_sync() functions:
removed any time sync calculations logic as separate functions in trace-timesync.c file
- Defined TSYNC_PROBE, TSYNC_REQ and TSYNC_RESP messages, in order to make the time sync protocol comprehensible.
- Addressed Steven Rostedt comments.
- Addressed Slavomir Kaslev commnets.
v5 changes:
- Rebased to Slavomir's v8 "Add VM kernel tracing over vsockets and FIFOs" patch series.
- Implemented an algorithm for time drift correction.
- Addressed Slavomir's commnets.
- Refactored the code: moved all time sync specific implementation in trace-timesync.c
- Isolated all hardcoded event specific stuff in a structure, so it could be easily moved to external plugins.
- Added a check for VSOCK support: do not perform vsock dependent time synchronisation in case there is no VSOCK support.
v4 changes:
- Removed the implementation of PTP-like algorithm. The current logic relies on matching time stamps of kvm_exit/virtio_transport_recv_pkt events on host to virtio_transport_alloc_pkt/vp_notify events on guest.
- Rebased to Slavomir's v7 "Add VM kernel tracing over vsockets and FIFOs" patch series.
- Decreased the time synch probes from 5000 to 300.
- Addressed Steven Rostedt comments.
- Code cleanup.
v3 changes:
- Removed any magic constants, used in the PTP-like algorithm, as Slavomir Kaslev suggested.
- Implemented new algorithm, based on mapping kvm_exit events in host context to vsock_send events in guest context, suggested by Steven Rostedt.
v2 changes:
- Addressed Steven Rostedt comments.
- Modified PTP-like timestamps sync algorithm to gain more accuracy, with the help of Yordan Karadzhov and Slavomir Kaslev.
]
Tzvetomir Stoyanov (7):
trace-cmd: Implement new lib API: tracecmd_local_events_system()
trace-cmd: Add support for negative time offsets in trace.dat file
trace-cmd: Add implementations of htonll() and ntohll()
trace-cmd: Add new library APIs for ftrace instances.
trace-cmd: Find and store pids of tasks, which run virtual CPUs of
given VM
trace-cmd: Implement new API tracecmd_add_option_v()
trace-cmd: Implement new option in trace.dat file:
TRACECMD_OPTION_TIME_SHIFT
Tzvetomir Stoyanov (VMware) (11):
trace-cmd: Add new library API for local CPU count
trace-cmd: Add new library API for reading ftrace buffers
trace-cmd: Add new API to generate a unique ID of the tracing session
trace-cmd: Store the session tracing ID in the trace.dat file
trace-cmd: Exchange tracing IDs between host and guest
trace-cmd: Add guest information in host's trace.dat file
trace-cmd: Add host trace clock as guest trace argument
trace-cmd: Refactor few trace-cmd internal functions.
trace-cmd: Basic infrastructure for host - guest timestamp
synchronization
trace-cmd: [POC] PTP-like algorithm for host - guest timestamp
synchronization
trace-cmd: Debug scripts for PTP-like algorithm for host - guest
timestamp synchronization
Documentation/trace-cmd-record.1.txt | 6 + include/trace-cmd/trace-cmd.h | 116 +++- include/trace-cmd/trace-msg.h | 10 + include/traceevent/event-parse.h | 1 + lib/trace-cmd/Makefile | 3 + lib/trace-cmd/include/trace-cmd-local.h | 33 +- lib/trace-cmd/include/trace-tsync-local.h | 36 ++ lib/trace-cmd/trace-input.c | 314 ++++++++++- lib/trace-cmd/trace-instance.c | 265 +++++++++ lib/trace-cmd/trace-msg.c | 162 +++++- lib/trace-cmd/trace-output.c | 116 +++- lib/trace-cmd/trace-timesync-ptp.c | 649 ++++++++++++++++++++++ lib/trace-cmd/trace-timesync.c | 469 ++++++++++++++++ lib/trace-cmd/trace-util.c | 358 ++++++++++-- scripts/debug/tsync_hist.py | 57 ++ scripts/debug/tsync_readme | 12 + scripts/debug/tsync_res.py | 46 ++ tracecmd/Makefile | 3 +- tracecmd/include/trace-local.h | 33 +- tracecmd/trace-agent.c | 48 +- tracecmd/trace-list.c | 2 +- tracecmd/trace-profile.c | 2 +- tracecmd/trace-read.c | 4 +- tracecmd/trace-record.c | 600 ++++++++++---------- tracecmd/trace-show.c | 2 + tracecmd/trace-stat.c | 24 +- tracecmd/trace-tsync.c | 268 +++++++++ tracecmd/trace-usage.c | 4 + 28 files changed, 3199 insertions(+), 444 deletions(-) create mode 100644 lib/trace-cmd/include/trace-tsync-local.h create mode 100644 lib/trace-cmd/trace-instance.c create mode 100644 lib/trace-cmd/trace-timesync-ptp.c create mode 100644 lib/trace-cmd/trace-timesync.c create mode 100644 scripts/debug/tsync_hist.py create mode 100644 scripts/debug/tsync_readme create mode 100644 scripts/debug/tsync_res.py create mode 100644 tracecmd/trace-tsync.c
Signature · 1 line
2.23.0
PATCH v17 1/18 trace-cmd: Implement new lib API: tracecmd_local_events_system()
From: Tzvetomir Stoyanov <[email protected]>
The new tracecmd lib API tracecmd_local_events_system() creates a tep handler and initializes it with the events of the specified subsystems.
--- include/trace-cmd/trace-cmd.h | 4 +- lib/trace-cmd/trace-util.c | 102 +++++++++++++++++++++++----------- 2 files changed, 72 insertions(+), 34 deletions(-)
include/trace-cmd/trace-cmd.h +3 -1
@@ -27,8 +27,10 @@ extern int tracecmd_disable_plugins;char **tracecmd_event_systems (const char *tracing_dir);char **tracecmd_system_events (const char *tracing_dir,const char *system);struct tep_handle *tracecmd_local_events (const char *tracing_dir); +struct tep_handle *tracecmd_local_events_system (const char *tracing_dir, +const char *const *sys_names);int tracecmd_fill_local_events (const char *tracing_dir, -struct tep_handle *pevent,int *parsing_failures); +struct tep_handle *tep,int *parsing_failures);char **tracecmd_local_plugins (const char *tracing_dir);char **tracecmd_add_list (char **list,const char *name,int len);
lib/trace-cmd/trace-util.c +69 -33
135 lines not on this page · raw message
Signature · 1 line
2.23.0
PATCH v17 2/18 trace-cmd: Add support for negative time offsets in trace.dat file
In reply to Tzvetomir Stoyanov (VMware)From: Tzvetomir Stoyanov <[email protected]>
When synchronizing timestamps between different machines, there are cases when the time offset is negative. This patch changes the way time offset is written and read from trace.dat file - as signed decimal, instead of hex.
--- include/trace-cmd/trace-cmd.h | 2 +- lib/trace-cmd/trace-input.c | 6 +++--- tracecmd/trace-read.c | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-)
include/trace-cmd/trace-cmd.h +1 -1
@@ -138,7 +138,7 @@ const char *tracecmd_buffer_instance_name(struct tracecmd_input *handle, int indstruct tracecmd_input *tracecmd_buffer_instance_handle (struct tracecmd_input *handle,int indx);int tracecmd_is_buffer_instance (struct tracecmd_input *handle); -void tracecmd_set_ts_offset (struct tracecmd_input *handle,unsigned long long offset); +void tracecmd_set_ts_offset (struct tracecmd_input *handle,long long offset);void tracecmd_set_ts2secs (struct tracecmd_input *handle,unsigned long long hz);void tracecmd_print_events (struct tracecmd_input *handle,const char *regex);
lib/trace-cmd/trace-input.c +3 -3
@@ -90,7 +90,7 @@ struct tracecmd_input {bool read_page;bool use_pipe;struct cpu_data *cpu_data; -unsigned long long ts_offset; +long long ts_offset;double ts2secs;char * cpustats;char * uname;
@@ -2127,7 +2127,7 @@ static int init_cpu(struct tracecmd_input *handle, int cpu) }void tracecmd_set_ts_offset (struct tracecmd_input *handle, -unsigned long long offset) +long long offset) { handle->ts_offset = offset; }
@@ -2305,7 +2305,7 @@ tracecmd_search_task_map(struct tracecmd_input *handle,static int handle_options (struct tracecmd_input *handle) { -unsigned long long offset; +long long offset;unsigned short option;unsigned int size;char *cpustats =NULL ;
tracecmd/trace-read.c +2 -2
@@ -58,7 +58,7 @@ static struct list_head handle_list;struct input_files {struct list_head list;const char *file; -unsigned long long tsoffset; +long long tsoffset;unsigned long long ts2secs; };static struct list_head input_files;
@@ -1466,7 +1466,7 @@ void trace_report (int argc, char **argv)struct input_files *inputs;struct handle_list *handles;enum output_type otype; -unsigned long long tsoffset =0 ; +long long tsoffset =0 ;unsigned long long ts2secs =0 ;unsigned long long ts2sc;int show_stat =0 ;
Signature · 1 line
2.23.0
PATCH v17 3/18 trace-cmd: Add implementations of htonll() and ntohll()
In reply to Tzvetomir Stoyanov (VMware)From: Tzvetomir Stoyanov <[email protected]>
Implementations of htonll() and ntohll() are added as macros, if they are not already defined.
--- include/trace-cmd/trace-msg.h | 10 ++++++++++ 1 file changed, 10 insertions(+)
include/trace-cmd/trace-msg.h +10 -0
@@ -12,4 +12,14 @@extern unsigned int page_size; +#ifndef htonll +# if __BYTE_ORDER == __LITTLE_ENDIAN +#define htonll (x) __bswap_64(x) +#define ntohll (x) __bswap_64(x) +#else +#define htonll (x) (x) +#define ntohll (x) (x) +#endif +#endif +#endif /* _TRACE_MSG_H_ */
Signature · 1 line
2.23.0
Re: [PATCH v17 03/18] trace-cmd: Add implementations of htonll() and ntohll()
Quoted 4 lines · On Tue, 3 Dec 2019 12:35:07 +0200 "Tzvetomir Stoyanov (VMware)"…
From: Tzvetomir Stoyanov <[email protected]>
Implementations of
htonll()andntohll()are added as macros, if they are not already defined.
I'm somewhat nervous about this being in a global header, as it is mostly generic. I wonder if we should prefix these with "tracecmd_" just to avoid any name collisions.
We could update this later when we make trace-cmd into a library.
-- Steve
Quoted 25 lines
Signed-off-by Tzvetomir Stoyanov <[email protected]>--- include/trace-cmd/trace-msg.h | 10 ++++++++++ 1 file changed, 10 insertions(+)include/trace-cmd/trace-msg.h +10
-0@@ -12,4 +12,14 @@extern unsigned int page_size; +#ifndef htonll +# if __BYTE_ORDER == __LITTLE_ENDIAN +#define htonll (x) __bswap_64(x) +#define ntohll (x) __bswap_64(x) +#else +#define htonll (x) (x) +#define ntohll (x) (x) +#endif +#endif +#endif /* _TRACE_MSG_H_ */
Re: [PATCH v17 03/18] trace-cmd: Add implementations of htonll() and ntohll()
Quoted 6 lines · Steven Rostedt
Quoted 2 lines · On Tue, 3 Dec 2019 12:35:07 +0200 "Tzvetomir Stoyanov (VMware)"…
From: Tzvetomir Stoyanov <[email protected]>
Quoted 9 lines
Quoted 2 lines
Implementations of
htonll()andntohll()are added as macros, if they are not already defined.I'm somewhat nervous about this being in a global header, as it is mostly generic. I wonder if we should prefix these with "
tracecmd_" just to avoid any name collisions.We could update this later when we make trace-cmd into a library.
I can move them to trace-msg.c file, as these are used only there. Note that a lot of clean-ups are required for libtracecmd.
Quoted 29 lines
-- Steve
Quoted 25 lines
Signed-off-by Tzvetomir Stoyanov <[email protected]>--- include/trace-cmd/trace-msg.h | 10 ++++++++++ 1 file changed, 10 insertions(+)include/trace-cmd/trace-msg.h +10
-0@@ -12,4 +12,14 @@extern unsigned int page_size; +#ifndef htonll +# if __BYTE_ORDER == __LITTLE_ENDIAN +#define htonll (x) __bswap_64(x) +#define ntohll (x) __bswap_64(x) +#else +#define htonll (x) (x) +#define ntohll (x) (x) +#endif +#endif +#endif /* _TRACE_MSG_H_ */
Signature · 2 lines
Tzvetomir (Ceco) Stoyanov VMware Open Source Technology Center
PATCH v17 4/18 trace-cmd: Add new library APIs for ftrace instances.
In reply to Tzvetomir Stoyanov (VMware)From: Tzvetomir Stoyanov <[email protected]>
In order to reuse the code, the functionality related to ftrace instances is moved from trace-cmd application to libtracecmd. The following new library APIs are introduced:
library structure, representing a ftrace instance:
struct tracecmd_instance {
char *name;
char *clock;
};APIs for creating and deleting ftrace instances:
struct tracecmd_instance *tracecmd_create_instance(const char *name); void tracecmd_free_instance(struct tracecmd_instance *instance); int tracecmd_make_instance(struct tracecmd_instance *instance); void tracecmd_remove_instance(struct tracecmd_instance *instance);
APIs for reading and writing ftrace files, instance aware:
char *tracecmd_get_instance_file(struct tracecmd_instance *instance, const char *file);
char *tracecmd_get_instance_dir(struct tracecmd_instance *instance);
int tracecmd_write_instance_file(struct tracecmd_instance *instance,
const char *file, const char *str,
const char *type);
int tracecmd_write_file(const char *file, const char *str, const char *type);
char *tracecmd_read_instance_file(struct tracecmd_instance *instance,
char *file, int *psize);API for setting ftrace clock, instance aware:
void tracecmd_set_clock(struct tracecmd_instance *instance, char **old_clock);
--- include/trace-cmd/trace-cmd.h | 24 ++ lib/trace-cmd/Makefile | 1 + lib/trace-cmd/include/trace-cmd-local.h | 33 +-- lib/trace-cmd/trace-instance.c | 265 ++++++++++++++++++ lib/trace-cmd/trace-util.c | 71 ++++- tracecmd/include/trace-local.h | 6 +- tracecmd/trace-list.c | 2 +- tracecmd/trace-record.c | 341 +++++++----------------- tracecmd/trace-show.c | 2 + tracecmd/trace-stat.c | 20 +- 10 files changed, 472 insertions(+), 293 deletions(-) create mode 100644 lib/trace-cmd/trace-instance.c
include/trace-cmd/trace-cmd.h +24 -0
@@ -356,6 +356,30 @@ int tracecmd_msg_send_trace_resp(struct tracecmd_msg_handle *msg_handle,int tracecmd_msg_recv_trace_resp (struct tracecmd_msg_handle *msg_handle,int *nr_cpus,int *page_size,unsigned int **ports,bool *use_fifos); +/* --- ftrace instances --- */ + +struct tracecmd_instance { +char *name; +char *clock; +}; + +struct tracecmd_instance *tracecmd_create_instance (const char *name); +void tracecmd_free_instance (struct tracecmd_instance *instance); +int tracecmd_make_instance (struct tracecmd_instance *instance); +void tracecmd_remove_instance (struct tracecmd_instance *instance); +char * +tracecmd_get_instance_file (struct tracecmd_instance *instance,const char *file); +char *tracecmd_get_instance_dir (struct tracecmd_instance *instance); +int tracecmd_write_instance_file (struct tracecmd_instance *instance, +const char *file,const char *str, +const char *type); + +int tracecmd_write_file (const char *file,const char *str,const char *type); +char *tracecmd_read_instance_file (struct tracecmd_instance *instance, +char *file,int *psize); + +void tracecmd_set_clock (struct tracecmd_instance *instance,char **old_clock); +/* --- Plugin handling --- */ extern struct tep_plugin_option trace_ftrace_options[];
lib/trace-cmd/Makefile +1 -0
@@ -13,6 +13,7 @@ OBJS += trace-input.o OBJS += trace-output.o OBJS += trace-recorder.o OBJS += trace-util.o +OBJS += trace-instance.o OBJS += trace-filter-hash.o OBJS += trace-msg.o
lib/trace-cmd/include/trace-cmd-local.h +2 -31
@@ -18,36 +18,7 @@#define STR (x) _STR(x)#define FILE_VERSION_STRING STR(FILE_VERSION) -static ssize_t __do_write (int fd,const void *data,size_t size) -{ -ssize_t tot =0 ; -ssize_t w; - -do { - w =write (fd, data + tot, size - tot); - tot += w; - -if (!w) -break ; -if (w <0 ) -return w; - }while (tot != size); - -return tot; -} - -static ssize_t -__do_write_check (int fd,const void *data,size_t size) -{ -ssize_t ret; - - ret =__do_write (fd, data, size); -if (ret <0 ) -return ret; -if (ret != size) -return -1 ; - -return 0 ; -} +ssize_t __do_write_check (int fd,const void *data,size_t size); +void __noreturn die (const char *fmt, ...);/* Can be overriden */ #endif /* _TRACE_CMD_LOCAL_H */
lib/trace-cmd/trace-instance.c new file +265 -0
265 lines not on this page · raw message
lib/trace-cmd/trace-util.c +66 -5
86 lines not on this page · raw message
tracecmd/include/trace-local.h +3 -3
@@ -181,7 +181,7 @@ struct pid_addr_maps {struct buffer_instance {struct buffer_instance *next; -const char *name; +struct tracecmd_instance *ftrace;char *cpumask;struct event_list *events;struct event_list **event_next;
@@ -195,7 +195,6 @@ struct buffer_instance {struct func_list *filter_funcs;struct func_list *notrace_funcs; -const char *clock;unsigned int *client_ports;struct trace_seq *s_save;
@@ -225,6 +224,8 @@ struct buffer_instance {bool use_fifos; }; +void init_top_instance (void ); +extern struct buffer_instance top_instance;extern struct buffer_instance *buffer_instances;extern struct buffer_instance *first_instance;
@@ -238,7 +239,6 @@ extern struct buffer_instance *first_instance;struct buffer_instance *create_instance (const char *name);void add_instance (struct buffer_instance *instance,int cpu_count); -char *get_instance_file (struct buffer_instance *instance,const char *file);void update_first_instance (struct buffer_instance *instance,int topt);void show_instance_file (struct buffer_instance *instance,const char *name);
tracecmd/trace-list.c +1 -1
@@ -33,7 +33,7 @@ void show_instance_file(struct buffer_instance *instance, const char *name) {char *path; - path =get_instance_file (instance, name); + path =tracecmd_get_instance_file (instance->ftrace, name);dump_file_content (path);tracecmd_put_tracing_file (path); }
tracecmd/trace-record.c +97 -244
732 lines not on this page · raw message
tracecmd/trace-show.c +2 -0
@@ -51,6 +51,8 @@ void trace_show(int argc, char **argv) {NULL, 0, NULL, 0} }; +init_top_instance (); +while ((c =getopt_long (argc-1 , argv+1 ,"B:c:fsp" , long_options, &option_index)) >=0 ) {switch (c) {
tracecmd/trace-stat.c +11 -9
@@ -30,7 +30,7 @@ static int get_instance_file_fd(struct buffer_instance *instance,char *path;int fd; - path =get_instance_file (instance, file); + path =tracecmd_get_instance_file (instance->ftrace, file); fd =open (path,O_RDONLY );tracecmd_put_tracing_file (path);
@@ -347,7 +347,7 @@ static void report_events(struct buffer_instance *instance)free (str); - path =get_instance_file (instance,"events" ); + path =tracecmd_get_instance_file (instance->ftrace,"events" );if (!path)die ("malloc" );
@@ -436,7 +436,7 @@ static void report_event_filters(struct buffer_instance *instance)enum event_iter_type type;enum event_process processed =PROCESSED_NONE ; - path =get_instance_file (instance,"events" ); + path =tracecmd_get_instance_file (instance->ftrace,"events" );if (!path)die ("malloc" );
@@ -509,7 +509,7 @@ static void report_event_triggers(struct buffer_instance *instance)enum event_iter_type type;enum event_process processed =PROCESSED_NONE ; - path =get_instance_file (instance,"events" ); + path =tracecmd_get_instance_file (instance->ftrace,"events" );if (!path)die ("malloc" );
@@ -598,7 +598,7 @@ static void report_graph_funcs(struct buffer_instance *instance) {char *path; - path =get_instance_file (instance,"set_graph_function" ); + path =tracecmd_get_instance_file (instance->ftrace,"set_graph_function" );if (!path)die ("malloc" );
@@ -606,7 +606,7 @@ static void report_graph_funcs(struct buffer_instance *instance)tracecmd_put_tracing_file (path); - path =get_instance_file (instance,"set_graph_notrace" ); + path =tracecmd_get_instance_file (instance->ftrace,"set_graph_notrace" );if (!path)die ("malloc" );
@@ -619,7 +619,7 @@ static void report_ftrace_filters(struct buffer_instance *instance) {char *path; - path =get_instance_file (instance,"set_ftrace_filter" ); + path =tracecmd_get_instance_file (instance->ftrace,"set_ftrace_filter" );if (!path)die ("malloc" );
@@ -627,7 +627,7 @@ static void report_ftrace_filters(struct buffer_instance *instance)tracecmd_put_tracing_file (path); - path =get_instance_file (instance,"set_ftrace_notrace" ); + path =tracecmd_get_instance_file (instance->ftrace,"set_ftrace_notrace" );if (!path)die ("malloc" );
@@ -857,7 +857,7 @@ static void stat_instance(struct buffer_instance *instance)if (instance != &top_instance) {if (instance != first_instance)printf ("---------------\n" ); -printf ("Instance: %s\n" , instance->name); +printf ("Instance: %s\n" , instance->ftrace->name); }report_plugin (instance);
@@ -882,6 +882,8 @@ void trace_stat (int argc, char **argv)int status;int c; +init_top_instance (); +for (;;) { c =getopt (argc-1 , argv+1 ,"tB:" );if (c ==-1 )
Signature · 1 line
2.23.0
Re: [PATCH v17 04/18] trace-cmd: Add new library APIs for ftrace instances.
Quoted 6 lines · On Tue, 3 Dec 2019 12:35:08 +0200 "Tzvetomir Stoyanov (VMware)"…
From: Tzvetomir Stoyanov <[email protected]>
In order to reuse the code, the functionality related to ftrace instances is moved from trace-cmd application to libtracecmd. The following new library APIs are introduced:
Quoted 8 lines
library structure, representing a ftrace instance:
struct tracecmd_instance { char *name; char *clock; };APIs for creating and deleting ftrace instances:
struct tracecmd_instance *tracecmd_create_instance(const char *name);
It looks to me that tracecmd_create_instance() allocates the instance, and tracecmd_free_instance() frees the memory. Let's rename this to tracecmd_alloc_instance(), otherwise it becomes ambiguous to actually creating the instance on file. When I first saw this, I thought this would create the instance
e.g. mkdir /sys/kernel/tracing/instances/foo
Which looks to be what tracecmd_make_instance() does.
Quoted 37 lines
void tracecmd_free_instance(struct tracecmd_instance *instance); int tracecmd_make_instance(struct tracecmd_instance *instance); void tracecmd_remove_instance(struct tracecmd_instance *instance);APIs for reading and writing ftrace files, instance aware:
char *tracecmd_get_instance_file(struct tracecmd_instance *instance, const char *file); char *tracecmd_get_instance_dir(struct tracecmd_instance *instance); int tracecmd_write_instance_file(struct tracecmd_instance *instance, const char *file, const char *str, const char *type); int tracecmd_write_file(const char *file, const char *str, const char *type); char *tracecmd_read_instance_file(struct tracecmd_instance *instance, char *file, int *psize);API for setting ftrace clock, instance aware:
void tracecmd_set_clock(struct tracecmd_instance *instance, char **old_clock);Signed-off-by Tzvetomir Stoyanov (VMware) <[email protected]>--- include/trace-cmd/trace-cmd.h | 24 ++ lib/trace-cmd/Makefile | 1 + lib/trace-cmd/include/trace-cmd-local.h | 33 +-- lib/trace-cmd/trace-instance.c | 265 ++++++++++++++++++ lib/trace-cmd/trace-util.c | 71 ++++- tracecmd/include/trace-local.h | 6 +- tracecmd/trace-list.c | 2 +- tracecmd/trace-record.c | 341 +++++++----------------- tracecmd/trace-show.c | 2 + tracecmd/trace-stat.c | 20 +- 10 files changed, 472 insertions(+), 293 deletions(-) create mode 100644 lib/trace-cmd/trace-instance.cinclude/trace-cmd/trace-cmd.h +0
-0
include/trace-cmd/trace-cmd.h int tracecmd_msg_send_trace_resp(struct tracecmd_msg_handle *msg_handle, · line 364
Quoted 9 lines
include/trace-cmd/trace-cmd.h +6
-0@@ -356,6 +356,30 @@ int tracecmd_msg_send_trace_resp(struct tracecmd_msg_handle *msg_handle,int tracecmd_msg_recv_trace_resp (struct tracecmd_msg_handle *msg_handle,int *nr_cpus,int *page_size,unsigned int **ports,bool *use_fifos); +/* --- ftrace instances --- */ + +struct tracecmd_instance { +char *name; +char *clock; +};
I wonder if we should keep this as a private structure. That is, only declare the struct in the public header, but define the structure in a private one. This will help with keeping people from using it directly, and allowing for updating the structure without having to update the library version.
Quoted 109 lines
include/trace-cmd/trace-cmd.h +18
-0+ +struct tracecmd_instance *tracecmd_create_instance (const char *name); +void tracecmd_free_instance (struct tracecmd_instance *instance); +int tracecmd_make_instance (struct tracecmd_instance *instance); +void tracecmd_remove_instance (struct tracecmd_instance *instance); +char * +tracecmd_get_instance_file (struct tracecmd_instance *instance,const char *file); +char *tracecmd_get_instance_dir (struct tracecmd_instance *instance); +int tracecmd_write_instance_file (struct tracecmd_instance *instance, +const char *file,const char *str, +const char *type); + +int tracecmd_write_file (const char *file,const char *str,const char *type); +char *tracecmd_read_instance_file (struct tracecmd_instance *instance, +char *file,int *psize); + +void tracecmd_set_clock (struct tracecmd_instance *instance,char **old_clock); +/* --- Plugin handling --- */ extern struct tep_plugin_option trace_ftrace_options[];lib/trace-cmd/Makefile +1
-0@@ -13,6 +13,7 @@ OBJS += trace-input.o OBJS += trace-output.o OBJS += trace-recorder.o OBJS += trace-util.o +OBJS += trace-instance.o OBJS += trace-filter-hash.o OBJS += trace-msg.olib/trace-cmd/include/trace-cmd-local.h +2
-31@@ -18,36 +18,7 @@#define STR (x) _STR(x)#define FILE_VERSION_STRING STR(FILE_VERSION) -static ssize_t __do_write (int fd,const void *data,size_t size) -{ -ssize_t tot =0 ; -ssize_t w; - -do { - w =write (fd, data + tot, size - tot); - tot += w; - -if (!w) -break ; -if (w <0 ) -return w; - }while (tot != size); - -return tot; -} - -static ssize_t -__do_write_check (int fd,const void *data,size_t size) -{ -ssize_t ret; - - ret =__do_write (fd, data, size); -if (ret <0 ) -return ret; -if (ret != size) -return -1 ; - -return 0 ; -} +ssize_t __do_write_check (int fd,const void *data,size_t size); +void __noreturn die (const char *fmt, ...);/* Can be overriden */ #endif /* _TRACE_CMD_LOCAL_H */ lib/trace-cmd/trace-instance.c new file +27
-0@@ -0,0 +1,265 @@ +// SPDX-License-Identifier: LGPL-2.1 +/* +* Copyright (C) 2019, VMware, Tzvetomir Stoyanov [email protected]> +* +*/ + +#include <stdio.h> +#include <stdlib.h> +#include <fcntl.h> +#include <time.h> +#include <poll.h> +#include <unistd.h> +#include <errno.h> +#include <sys/stat.h> + +#include "trace-cmd.h" +#include "trace-cmd-local.h" + +/** +* tracecmd_put_tracing_file - Free tracing file / dir, created by +* tracecmd_get_instance_dir() or tracecmd_get_instance_file() +* APIs. +*@name: The name of the tracing file or dir +*/ +void tracecmd_put_tracing_file (char *name) +{ +free (name);
lib/trace-cmd/trace-instance.c line 37
Quoted 10 lines
lib/trace-cmd/trace-instance.c new file +10
-0+} + +/** +* tracecmd_create_instance - allocate a new ftrace instance +* @name: The name of the instance (instance will point to this) +* +* Returns a newly allocated instance. Note that @name will not be +* copied, and the instance buffer will point to the string itself. +*/ +struct tracecmd_instance *tracecmd_create_instance (const char *name)
Again, better name would be "_alloc_instance"
Quoted 49 lines
lib/trace-cmd/trace-instance.c new file +49
-0+{ +struct tracecmd_instance *instance; + + instance =malloc (sizeof (*instance)); +if (!instance) +return NULL ; +memset (instance,0 ,sizeof (*instance)); +if (name) + instance->name =strdup (name); + +return instance; +} + +/** +* tracecmd_free_instance - Free an instance struct, previously allocated by +* tracecmd_create_instance(). +*@instance: Pointer to the instance to be freed +* +*/ +void tracecmd_free_instance (struct tracecmd_instance *instance) +{ +if (!instance) +return ; + +free (instance->name); +free (instance); +} + +/** +* tracecmd_make_instance - Create a new ftrace instance +* @instance: Pointer to the instance to be created +* +* Returns -1 in case of an erro, or 0 otherwise. +*/ +int tracecmd_make_instance (struct tracecmd_instance *instance) +{ +struct stat st; +char *path; +int ret; + + path =tracecmd_get_instance_dir (instance); + ret =stat (path, &st); +if (ret <0 ) { + ret =mkdir (path,0777 ); +if (ret <0 ) +return ret; + + }else + ret =1 ;
lib/trace-cmd/trace-instance.c line 96
Quoted 10 lines
lib/trace-cmd/trace-instance.c new file +10
-0+tracecmd_put_tracing_file (path); +return ret; +} + +/** +* tracecmd_remove_instance - Remove a ftrace instance +* @instance: Pointer to the instance to be removed +* +*/ +void tracecmd_remove_instance (struct tracecmd_instance *instance)
Should return an int.
lib/trace-cmd/trace-instance.c line 101
Quoted 5 lines
lib/trace-cmd/trace-instance.c new file +5
-0+{ +char *path; + + path =tracecmd_get_instance_dir (instance); +rmdir (path);
We should check the return status of rmdir, and return that.
lib/trace-cmd/trace-instance.c line 106
Quoted 5 lines
lib/trace-cmd/trace-instance.c new file +5
-0+tracecmd_put_tracing_file (path); +} + +/** +* tracecmd_get_instance_file - return the path to a instance file.
Note, none of the one line descriptions should end in a period. There's a few sprinkled around.
Quoted 49 lines
lib/trace-cmd/trace-instance.c new file +49
-0+* @instance: buffer instance for the file, can be NULL for the top instance +* @file: name of file to return +* +* Returns the path name of the @file for the given @instance. +* +* Must use tracecmd_put_tracing_file() to free the returned string. +*/ +char * +tracecmd_get_instance_file (struct tracecmd_instance *instance,const char *file) +{ +char *path; +char *buf; +int ret; + +if (instance && instance->name) { + ret =asprintf (&buf,"instances/%s/%s" , instance->name, file); +if (ret <0 ) +die ("Failed to allocate name for %s/%s" , instance->name, file); + path =tracecmd_get_tracing_file (buf); +free (buf); + }else + path =tracecmd_get_tracing_file (file); + +return path; +} + +/** +* tracecmd_get_instance_file - return the path to a instance file. +* @instance: buffer instance for the file, can be NULL for the top instance +* @file: name of file to return +* +* Returns the path name of the @file for the given @instance. +* +* Must use tracecmd_put_tracing_file() to free the returned string. +*/ +char *tracecmd_get_instance_dir (struct tracecmd_instance *instance) +{ +char *buf; +char *path; +int ret; + +if (instance->name) { + ret =asprintf (&buf,"instances/%s" , instance->name); +if (ret <0 ) +die ("Failed to allocate for instance %s" , instance->name); + path =tracecmd_get_tracing_file (buf); +free (buf); + }else + path =tracecmd_find_tracing_dir ();
lib/trace-cmd/trace-instance.c line 165
Quoted 10 lines
lib/trace-cmd/trace-instance.c new file +10
-0+ +return path; +} + +/** +* tracecmd_write_instance_file - Write in trace file of specific instance. +* @instance: buffer instance for the file, can be NULL for the top instance +* @file: name of the file +* @str: Null terminated string, that will be written in the file. +* @type: Null terminated string, describing the current write operation.
BTW, the correct term is "nul terminated string", as NULL is a pointer, and "nul" is '\0'.
Quoted 18 lines
lib/trace-cmd/trace-instance.c new file +18
-0+* Used for logging purposes. +* +* Returns the number of written bytes, or -1 in case of an error +*/ +int tracecmd_write_instance_file (struct tracecmd_instance *instance, +const char *file,const char *str, +const char *type) +{ +struct stat st; +char *path; +int ret; + + path =tracecmd_get_instance_file (instance, file); + ret =stat (path, &st); +if (ret ==0 ) + ret =tracecmd_write_file (path, str, type); +tracecmd_put_tracing_file (path); +
lib/trace-cmd/trace-instance.c line 193
Quoted 10 lines
lib/trace-cmd/trace-instance.c new file +10
-0+return ret; +} + +/** +* tracecmd_read_instance_file - Read from a trace file of specific instance. +* @instance: buffer instance for the file, can be NULL for the top instance +* @file: name of the file +* @psize: Returns the number of bytes read. +* +* Returns a pointer to a NULL terminated string, read from the file, or NULL in
Again, 'a nul terminated string'
lib/trace-cmd/trace-instance.c line 194
Quoted 1 line
lib/trace-cmd/trace-instance.c new file +1
-0+* case of an error.
We should also state that the returned string needs to be freed via free().
Quoted 8 lines
lib/trace-cmd/trace-instance.c new file +8
-0+*/ +char *tracecmd_read_instance_file (struct tracecmd_instance *instance, +char *file,int *psize) +{ +char buffer[BUFSIZ ]; +int size =0 ; +char *path; +char *buf;
lib/trace-cmd/trace-instance.c line 212
Quoted 10 lines
lib/trace-cmd/trace-instance.c new file +10
-0+int fd; +int r; + + path =tracecmd_get_instance_file (instance, file); + fd =open (path,O_RDONLY ); +tracecmd_put_tracing_file (path); +if (fd <0 ) { +warning ("File %s not found" , file); +return NULL ; + }
Probably should add an empty line here.
lib/trace-cmd/trace-instance.c line 220
Quoted 8 lines
lib/trace-cmd/trace-instance.c new file +8
-0+ do { + r = read(fd, buffer, BUFSIZ); + if (r <= 0) + continue; + if (size) + buf = realloc(buf, size+r+1); + else + buf = malloc(r+1);
I know this is only cut and pasted from what the original was, but we should probably (either in this patch, or perhaps a separate one) fix the above. As realloc(NULL, x) is equivalent to malloc(x), we can just have:
buf = realloc(buf, size+r+1);
lib/trace-cmd/trace-instance.c line 222
Quoted 2 lines
lib/trace-cmd/trace-instance.c new file +2
-0+if (!buf) +die ("Failed to allocate instance file buffer" );
As this is becoming a library function, we should remove "die()", as that is only allowed in the application. Not library calls.
That would also mean we need to clean up anything we allocated on failure, and that the realloc would need to be:
new_buf = realloc(buf, size + r + 1);
if (!new_buf) {
free(buf);
return NULL;All the functions that are being moved into the library needs to have their die() calls removed. This is OK to do in a separate patch.
Preferably right after this patch (so reviewers know it's happening).
Quoted 4 lines
lib/trace-cmd/trace-instance.c new file +4
-0+memcpy (buf+size, buffer, r); + size += r; + }while (r); +
lib/trace-cmd/trace-instance.c line 236
Quoted 10 lines
lib/trace-cmd/trace-instance.c new file +10
-0+ buf[size] ='\0' ; +if (psize) + *psize = size; +return buf; +} + +/** +* tracecmd_set_clock - Set the clock of ftrace event's timestamps, per instance. +* @instance: Pointer to ftrace instance, containing the desired clock. +* @old_clock: Optional, return the newly allocated string with the old clock.
This looks to be just a copy of the old code and forced into a library function. As a library function, it should return an "int" and be passed the clock name directly. Not via the instance->clock.
-- Steve
Quoted 30 lines
lib/trace-cmd/trace-instance.c new file +29
-0+* +*/ +void tracecmd_set_clock (struct tracecmd_instance *instance,char **old_clock) +{ +char *content; +char *str; + +if (!instance->clock) +return ; + +/* The current clock is in brackets, reset it when we are done */ + content =tracecmd_read_instance_file (instance,"trace_clock" ,NULL ); + +/* check if first clock is set */ +if (*content =='[' ) + str =strtok (content+1 ,"]" ); +else { + str =strtok (content,"[" ); +if (!str) +die ("Can not find clock in trace_clock" ); + str =strtok (NULL ,"]" ); + } +if (old_clock) + *old_clock =strdup (str); + +free (content); +tracecmd_write_instance_file (instance, +"trace_clock" , instance->clock,"clock" ); +}
Re: [PATCH v17 04/18] trace-cmd: Add new library APIs for ftrace instances.
Quoted 1 line · Steven Rostedt
[ ... ]
Quoted 32 lines
lib/trace-cmd/trace-instance.c line 220
Quoted 8 lines
lib/trace-cmd/trace-instance.c new file +8
-0+ do { + r = read(fd, buffer, BUFSIZ); + if (r <= 0) + continue; + if (size) + buf = realloc(buf, size+r+1); + else + buf = malloc(r+1);I know this is only cut and pasted from what the original was, but we should probably (either in this patch, or perhaps a separate one) fix the above. As
realloc(NULL, x)is equivalent to malloc(x), we can just have:buf = realloc(buf, size+r+1);lib/trace-cmd/trace-instance.c line 222
Quoted 2 lines
lib/trace-cmd/trace-instance.c new file +2
-0+if (!buf) +die ("Failed to allocate instance file buffer" );As this is becoming a library function, we should remove "
die()", as that is only allowed in the application. Not library calls.That would also mean we need to clean up anything we allocated on failure, and that the realloc would need to be:
new_buf = realloc(buf, size + r + 1); if (!new_buf) { free(buf); return NULL;
Quoted 5 lines
All the functions that are being moved into the library needs to have their
die()calls removed. This is OK to do in a separate patch.Preferably right after this patch (so reviewers know it's happening).
I fixed most of the comments, except those die() call. There are a lot of things that
should be fixed in the existing libtracecmd code, in order to become a real library.
Also, I started to work on a new library providing APIs for interaction with files
from tracefs, libtraceftrace (this is only a work name, we should think about more suitable one).
I would suggest changes from this patch set to be considered only as part of the time stamp sync
feature. There will be at least two library patch sets - for cleaning up the existing libtracecmd code and
for the new ftrace library.
Quoted 11 lines
Quoted 11 lines
lib/trace-cmd/trace-instance.c new file +10
-0+memcpy (buf+size, buffer, r); + size += r; + }while (r); + + buf[size] ='\0' ; +if (psize) + *psize = size; +return buf; +} +
Quoted 8 lines
Quoted 4 lines
+/**
+ *tracecmd_set_clock- Set the clock of ftrace event's timestamps, per instance.
+ * @instance: Pointer to ftrace instance, containing the desired clock.
+ * @old_clock: Optional, return the newly allocated string with the old clock.This looks to be just a copy of the old code and forced into a library function. As a library function, it should return an "int" and be passed the clock name directly. Not via the
instance->clock.
The idea of this API is to apply the clock, already set to the instance. This follows
the existing logic of configuring the instance clock, that's why I put the "char *clock"
in the new "struct tracecmd_instance". We can remove the clock from the structure
and pass it as an argument to tracecmd_set_clock(), so the "struct tracecmd_instance"
will hold only the instance name ?
Quoted 25 lines
-- Steve
Quoted 22 lines
+ *
+ */lib/trace-cmd/trace-instance.c new file +20
-0+void tracecmd_set_clock (struct tracecmd_instance *instance,char **old_clock) +{ +char *content; +char *str; + +if (!instance->clock) +return ; + +/* The current clock is in brackets, reset it when we are done */ + content =tracecmd_read_instance_file (instance,"trace_clock" ,NULL ); + +/* check if first clock is set */ +if (*content =='[' ) + str =strtok (content+1 ,"]" ); +else { + str =strtok (content,"[" ); +if (!str) +die ("Can not find clock in trace_clock" ); + str =strtok (NULL ,"]" ); + }
Quoted 8 lines
lib/trace-cmd/trace-instance.c line 265
Quoted 8 lines
lib/trace-cmd/trace-instance.c new file +7
-0+if (old_clock) + *old_clock =strdup (str); + +free (content); +tracecmd_write_instance_file (instance, +"trace_clock" , instance->clock,"clock" ); +}
Thanks !
Signature · 2 lines
Tzvetomir (Ceco) Stoyanov VMware Open Source Technology Center
PATCH v17 5/18 trace-cmd: Add new library API for local CPU count
In reply to Tzvetomir Stoyanov (VMware)In order to reuse the code, the function detecting the number of local CPU is moved from trace-cmd application to libtracecmd. The following new library API is introduced:
int tracecmd_count_cpus(void);
--- include/trace-cmd/trace-cmd.h | 3 +++ lib/trace-cmd/trace-util.c | 40 +++++++++++++++++++++++++++++ tracecmd/include/trace-local.h | 2 -- tracecmd/trace-agent.c | 2 +- tracecmd/trace-profile.c | 2 +- tracecmd/trace-record.c | 46 +++------------------------------- tracecmd/trace-stat.c | 4 +-- 7 files changed, 50 insertions(+), 49 deletions(-)
include/trace-cmd/trace-cmd.h +3 -0
@@ -427,6 +427,9 @@ void tracecmd_plog(const char *fmt, ...);void tracecmd_plog_error (const char *fmt, ...);int tracecmd_set_logfile (char *logfile); +/* --- System --- */ +int tracecmd_count_cpus (void ); +/* --- Hack! --- */ int tracecmd_blk_hack (struct tracecmd_input *handle);
lib/trace-cmd/trace-util.c +40 -0
@@ -1112,6 +1112,46 @@ int tracecmd_stack_tracer_status(int *status)return 1 ;/* full success */ } +int tracecmd_count_cpus (void ) +{ +FILE *fp; +char buf[1024 ]; +int cpus =0 ; +char *pbuf; +size_t *pn; +size_t n; +int r; + + cpus =sysconf (_SC_NPROCESSORS_CONF); +if (cpus >0 ) +return cpus; + +warning ("sysconf could not determine number of CPUS" ); + +/* Do the hack to figure out # of CPUS */ + n =1024 ; + pn = &n; + pbuf = buf; + + fp =fopen ("/proc/cpuinfo" ,"r" ); +if (!fp) +die ("Can not read cpuinfo" ); + +while ((r =getline (&pbuf, pn, fp)) >=0 ) { +char *p; + +if (strncmp (buf,"processor" ,9 ) !=0 ) +continue ; +for (p = buf+9 ;isspace (*p); p++) + ; +if (*p ==':' ) + cpus++; + } +fclose (fp); + +return cpus; +} +/* * tracecmd_write_file - Write in trace file * @file: Full name of the trace file.
tracecmd/include/trace-local.h +0 -2
@@ -243,8 +243,6 @@ void update_first_instance(struct buffer_instance *instance, int topt);void show_instance_file (struct buffer_instance *instance,const char *name); -int count_cpus (void ); -/* moved from trace-cmd.h */ void tracecmd_create_top_instance (char *name);void tracecmd_remove_instances (void );
tracecmd/trace-agent.c +1 -1
@@ -203,7 +203,7 @@ static void agent_serve(unsigned int port)signal (SIGCHLD , handle_sigchld); - nr_cpus =count_cpus (); + nr_cpus =tracecmd_count_cpus (); page_size =getpagesize (); sd =make_vsock (port);
tracecmd/trace-profile.c +1 -1
@@ -1314,7 +1314,7 @@ void trace_init_profile(struct tracecmd_input *handle, struct hook_list *hook,* system. */ if (!h->cpus) - h->cpus =count_cpus (); + h->cpus =tracecmd_count_cpus ();list_head_init (&h->migrate_starts); h->cpu_starts =malloc (sizeof (*h->cpu_starts) * h->cpus);
tracecmd/trace-record.c +3 -43
70 lines not on this page · raw message
tracecmd/trace-stat.c +2 -2
@@ -737,7 +737,7 @@ static void report_cpumask(struct buffer_instance *instance) cont =strstrip (str);/* check to make sure all CPUs on this machine are set */ - cpus =count_cpus (); + cpus =tracecmd_count_cpus ();for (i =strlen (cont) -1 ; i >=0 && cpus >0 ; i--) {if (cont[i] ==',' )
@@ -896,7 +896,7 @@ void trace_stat (int argc, char **argv) instance =create_instance (optarg);if (!instance)die ("Failed to create instance" ); -add_instance (instance,count_cpus ()); +add_instance (instance,tracecmd_count_cpus ());/* top instance requires direct access */ if (!topt &&is_top_instance (first_instance)) first_instance = instance;
Signature · 1 line
2.23.0
Re: [PATCH v17 05/18] trace-cmd: Add new library API for local CPU count
Quoted 43 lines · On Tue, 3 Dec 2019 12:35:09 +0200 "Tzvetomir Stoyanov (VMware)"…
In order to reuse the code, the function detecting the number of local CPU is moved from trace-cmd application to libtracecmd. The following new library API is introduced:
int tracecmd_count_cpus(void);Signed-off-by Tzvetomir Stoyanov (VMware) <[email protected]>--- include/trace-cmd/trace-cmd.h | 3 +++ lib/trace-cmd/trace-util.c | 40 +++++++++++++++++++++++++++++ tracecmd/include/trace-local.h | 2 -- tracecmd/trace-agent.c | 2 +- tracecmd/trace-profile.c | 2 +- tracecmd/trace-record.c | 46 +++------------------------------- tracecmd/trace-stat.c | 4 +-- 7 files changed, 50 insertions(+), 49 deletions(-)include/trace-cmd/trace-cmd.h +3
-0@@ -427,6 +427,9 @@ void tracecmd_plog(const char *fmt, ...);void tracecmd_plog_error (const char *fmt, ...);int tracecmd_set_logfile (char *logfile); +/* --- System --- */ +int tracecmd_count_cpus (void ); +/* --- Hack! --- */ int tracecmd_blk_hack (struct tracecmd_input *handle);lib/trace-cmd/trace-util.c +5
-0@@ -1112,6 +1112,46 @@ int tracecmd_stack_tracer_status(int *status)return 1 ;/* full success */ } +int tracecmd_count_cpus (void ) +{ +FILE *fp; +char buf[1024 ]; +int cpus =0 ;
lib/trace-cmd/trace-util.c int tracecmd_stack_tracer_status(int *status) · line 1129
Quoted 10 lines
lib/trace-cmd/trace-util.c +10
-0+char *pbuf; +size_t *pn; +size_t n; +int r; + + cpus =sysconf (_SC_NPROCESSORS_CONF); +if (cpus >0 ) +return cpus; + +warning ("sysconf could not determine number of CPUS" );
As this is now a library function, probably should have a static variable here:
static int once;
if (!once) {
once++;
warning("...");
}No need to do more than one warning.
lib/trace-cmd/trace-util.c int tracecmd_stack_tracer_status(int *status) · line 1138
Quoted 9 lines
lib/trace-cmd/trace-util.c +9
-0+ +/* Do the hack to figure out # of CPUS */ + n =1024 ; + pn = &n; + pbuf = buf; + + fp =fopen ("/proc/cpuinfo" ,"r" ); +if (!fp) +die ("Can not read cpuinfo" );
Need to get rid of the die(). Could be in a separate patch.
Will also need to have the call sites check to see if the returned number is negative, and fail there.
-- Steve
Quoted 20 lines
lib/trace-cmd/trace-util.c +16
-0+ +while ((r =getline (&pbuf, pn, fp)) >=0 ) { +char *p; + +if (strncmp (buf,"processor" ,9 ) !=0 ) +continue ; +for (p = buf+9 ;isspace (*p); p++) + ; +if (*p ==':' ) + cpus++; + } +fclose (fp); + +return cpus; +} +/* * tracecmd_write_file - Write in trace file * @file: Full name of the trace file. tracecmd/include/trace-local.h +0
-0
PATCH v17 6/18 trace-cmd: Add new library API for reading ftrace buffers
In reply to Tzvetomir Stoyanov (VMware)A new libtracecmd API is introduced:
int tracecmd_iterate_raw_events(struct tep_handle *tep,
struct tracecmd_instance *instance, int (*callback)(struct tep_event *,
struct tep_record *, int, void *context),
void *callback_context);
It reads events from trace_pipe_raw, per cpu ftrace buffer, and calls a user callback for each of them. The API is instance aware.
--- include/trace-cmd/trace-cmd.h | 8 +++ lib/trace-cmd/trace-util.c | 114 ++++++++++++++++++++++++++++++++++ 2 files changed, 122 insertions(+)
include/trace-cmd/trace-cmd.h +8 -0
@@ -380,6 +380,14 @@ char *tracecmd_read_instance_file(struct tracecmd_instance *instance,void tracecmd_set_clock (struct tracecmd_instance *instance,char **old_clock); +int tracecmd_iterate_raw_events (struct tep_handle *tep, +struct tracecmd_instance *instance, +int (*callback)(struct tep_event *, +struct tep_record *, +int ,void *context), +void *callback_context); + +/* --- Plugin handling --- */ extern struct tep_plugin_option trace_ftrace_options[];
lib/trace-cmd/trace-util.c +114 -0
117 lines not on this page · raw message
Signature · 1 line
2.23.0
Re: [PATCH v17 06/18] trace-cmd: Add new library API for reading ftrace buffers
Quoted 11 lines · On Tue, 3 Dec 2019 12:35:10 +0200 "Tzvetomir Stoyanov (VMware)"…
A new libtracecmd API is introduced:
int tracecmd_iterate_raw_events(struct
tep_handle*tep,
structtracecmd_instance*instance, int (*callback)(structtep_event*,
structtep_record*, int, void *context),
void*callback_context);It reads events from
trace_pipe_raw, per cpu ftrace buffer, and calls a user callback for each of them. The API is instance aware.
Hmm, would be good to know what this would be used for.
Quoted 52 lines
Signed-off-by Tzvetomir Stoyanov (VMware) <[email protected]>--- include/trace-cmd/trace-cmd.h | 8 +++ lib/trace-cmd/trace-util.c | 114 ++++++++++++++++++++++++++++++++++ 2 files changed, 122 insertions(+)include/trace-cmd/trace-cmd.h +8
-0@@ -380,6 +380,14 @@ char *tracecmd_read_instance_file(struct tracecmd_instance *instance,void tracecmd_set_clock (struct tracecmd_instance *instance,char **old_clock); +int tracecmd_iterate_raw_events (struct tep_handle *tep, +struct tracecmd_instance *instance, +int (*callback)(struct tep_event *, +struct tep_record *, +int ,void *context), +void *callback_context); + +/* --- Plugin handling --- */ extern struct tep_plugin_option trace_ftrace_options[];lib/trace-cmd/trace-util.c +18
-0@@ -1185,3 +1185,117 @@ int tracecmd_write_file(const char *file, const char *str, const char *type) }return ret; } + +static int +get_events_in_page (struct tep_handle *tep,void *page, +int size,int cpu, +int (*callback)(struct tep_event *, +struct tep_record *, +int ,void *), +void *callback_context) +{ +struct tep_record *last_record =NULL ; +struct tep_event *event =NULL ; +struct tep_record *record; +int id, cnt =0 ; + +if (size <=0 ) +return 0 ; + +while (true) {
lib/trace-cmd/trace-util.c int tracecmd_write_file(const char *file, const char *str, const char *type) · line 1215
Quoted 10 lines
lib/trace-cmd/trace-util.c +10
-0+ event =NULL ; + record =tracecmd_read_page_record (tep, page, size, + last_record); +if (!record) +break ; +free_record (last_record); + id =tep_data_type (tep, record); + event =tep_find_event (tep, id); +if (event && callback) { +if (callback (event, record, cpu, callback_context))
Hmm, need to be documented that breaking out of the loop will likely call future reads to lose events. The records on the page after the record that causes the break will be lost from future reads.
-- Steve
Quoted 86 lines
lib/trace-cmd/trace-util.c +86
-0+break ; + } + last_record = record; + } +free_record (last_record); + +return cnt; +} + +/* +* tracecmd_iterate_raw_events - Iterate through events in trace_pipe_raw +* per CPU trace files +* @tep: a handle to the trace event parser context +* @instance: ftrace instance, can be NULL for the top instance +* @callback: A user function, called for each record from the file. +* @callback_context: A custom context, passed to the user callback function +* +* If the @callback returns non-zero, the iteration stops. +* +* Returns -1 in case of an error, or 0 otherwise. +*/ +int tracecmd_iterate_raw_events (struct tep_handle *tep, +struct tracecmd_instance *instance, +int (*callback)(struct tep_event *, +struct tep_record *, +int ,void *), +void *callback_context) +{ +unsigned int p_size; +struct dirent *dent; +char file[PATH_MAX ]; +void *page =NULL ; +struct stat st; +char *path; +DIR *dir; +int ret; +int cpu; +int fd; +int r; + + p_size =getpagesize (); + path =tracecmd_get_instance_file (instance,"per_cpu" ); +if (!path) +return -1 ; + dir =opendir (path); +if (!dir) { + ret =-1 ; + goto error; + } + page =malloc (p_size); +if (!page) { + ret =-1 ; + goto error; + } +while ((dent =readdir (dir))) { +const char *name = dent->d_name; + +if (strlen (name) <4 ||strncmp (name,"cpu" ,3 ) !=0 ) +continue ; + cpu =atoi (name +3 ); +sprintf (file,"%s/%s" , path, name); + ret =stat (file, &st); +if (ret <0 || !S_ISDIR (st.st_mode)) +continue ; + +sprintf (file,"%s/%s/trace_pipe_raw" , path, name); + fd =open (file,O_RDONLY |O_NONBLOCK ); +if (fd <0 ) +continue ; +do { + r =read (fd, page, p_size); +if (r >0 ) +get_events_in_page (tep, page, r, cpu, + callback, callback_context); + }while (r >0 ); +close (fd); + } + ret =0 ; + +error: +if (dir) +closedir (dir); +free (page); +tracecmd_put_tracing_file (path); +return ret; +}
PATCH v17 7/18 trace-cmd: Find and store pids of tasks, which run virtual CPUs of given VM
In reply to Tzvetomir Stoyanov (VMware)From: Tzvetomir Stoyanov <[email protected]>
In order to match host and guest events, a mapping between guest VCPU and the host task, running this VCPU is needed. Extended existing struct guest to hold such mapping and added logic in read_qemu_guests() function to initialize it. Implemented a new internal API, get_guest_vcpu_pid(), to retrieve VCPU-task mapping for given VM.
--- tracecmd/include/trace-local.h | 2 ++ tracecmd/trace-record.c | 57 ++++++++++++++++++++++++++++++++++ 2 files changed, 59 insertions(+)
tracecmd/include/trace-local.h +2 -0
@@ -243,6 +243,8 @@ void update_first_instance(struct buffer_instance *instance, int topt);void show_instance_file (struct buffer_instance *instance,const char *name); +int get_guest_vcpu_pid (unsigned int guest_cid,unsigned int guest_vcpu); +/* moved from trace-cmd.h */ void tracecmd_create_top_instance (char *name);void tracecmd_remove_instances (void );
tracecmd/trace-record.c +57 -0
85 lines not on this page · raw message
Signature · 1 line
2.23.0
Re: [PATCH v17 07/18] trace-cmd: Find and store pids of tasks, which run virtual CPUs of given VM
Quoted 73 lines · On Tue, 3 Dec 2019 12:35:11 +0200 "Tzvetomir Stoyanov (VMware)"…
From: Tzvetomir Stoyanov <[email protected]>
In order to match host and guest events, a mapping between guest VCPU and the host task, running this VCPU is needed. Extended existing struct guest to hold such mapping and added logic in
read_qemu_guests()function to initialize it. Implemented a new internal API,get_guest_vcpu_pid(), to retrieve VCPU-task mapping for given VM.Signed-off-by Tzvetomir Stoyanov <[email protected]>--- tracecmd/include/trace-local.h | 2 ++ tracecmd/trace-record.c | 57 ++++++++++++++++++++++++++++++++++ 2 files changed, 59 insertions(+)tracecmd/include/trace-local.h +2
-0@@ -243,6 +243,8 @@ void update_first_instance(struct buffer_instance *instance, int topt);void show_instance_file (struct buffer_instance *instance,const char *name); +int get_guest_vcpu_pid (unsigned int guest_cid,unsigned int guest_vcpu); +/* moved from trace-cmd.h */ void tracecmd_create_top_instance (char *name);void tracecmd_remove_instances (void );tracecmd/trace-record.c +27
-0@@ -2873,10 +2873,12 @@ static bool is_digits(const char *s)return true; } +#define VCPUS_MAX 256struct guest {char *name;int cid;int pid; +int cpu_pid[VCPUS_MAX ]; };static struct guest *guests;@@ -2894,6 +2896,46 @@ static char *get_qemu_guest_name(char *arg)return arg; } +static void read_qemu_guests_pids (char *guest_task,struct guest *guest) +{ +struct dirent *entry; +char path[PATH_MAX ]; +char *buf =NULL ; +size_t n =0 ; +int vcpu; +DIR *dir; +FILE *f; + +snprintf (path,sizeof (path),"/proc/%s/task" , guest_task); + dir =opendir (path); +if (!dir) +return ; + +while ((entry =readdir (dir))) { +if (!(entry->d_type ==DT_DIR &&is_digits (entry->d_name))) +continue ; + +snprintf (path,sizeof (path),"/proc/%s/task/%s/comm" , + guest_task, entry->d_name); + f =fopen (path,"r" ); +if (!f) +continue ; +if (getline (&buf, &n, f) <0 )
tracecmd/trace-record.c static char *get_qemu_guest_name(char *arg) · line 2933
Quoted 10 lines
tracecmd/trace-record.c +10
-0+ goto next; +if (strncmp (buf,"CPU " ,4 ) !=0 ) + goto next; + + vcpu =atoi (buf+4 ); +if (!(vcpu >=0 && vcpu <VCPUS_MAX )) + goto next; + guest->cpu_pid[vcpu] =atoi (entry->d_name); + +next:
I don't usually mind goto's but the above could be simply written as:
if (getline(&buf, &n, f) >= 0 &&
strncmp(buf, "CPU ", 4) == 0) {
vcpu = atoi(buf + 4);
if (vcpu >= 0 && vcpu < VCPUS_MAX)
guest->cpu_pid[vcpu] = atoi(entry->d_name);
}-- Steve
Quoted 37 lines
tracecmd/trace-record.c +20
-0+fclose (f); + } +free (buf); +} +static void read_qemu_guests (void ) {static bool initialized;@@ -2957,6 +2999,8 @@ static void read_qemu_guests(void)if (!is_qemu) goto next; +read_qemu_guests_pids (entry->d_name, &guest); + guests =realloc (guests, (guests_len +1 ) *sizeof (*guests));if (!guests)die ("Can not allocate guest buffer" );@@ -3002,6 +3046,19 @@ static char *parse_guest_name(char *guest, int *cid, int *port)return guest; } +int get_guest_vcpu_pid (unsigned int guest_cid,unsigned int guest_vcpu) +{ +int i; + +if (!guests || guest_vcpu >=VCPUS_MAX ) +return -1 ; + +for (i =0 ; i < guests_len; i++) +if (guest_cid == guests[i].cid) +return guests[i].cpu_pid[guest_vcpu]; +return -1 ; +} +static void set_prio (int prio) {struct sched_param sp;
PATCH v17 8/18 trace-cmd: Implement new API tracecmd_add_option_v()
In reply to Tzvetomir Stoyanov (VMware)From: Tzvetomir Stoyanov <[email protected]>
A new tracecmd API tracecmd_add_option_v() is introduced. It adds new option in trace.dat, similar to tracecmd_add_option(), but the option's data is passed as list of buffers. The standard struct iovec is used as input parameter, containing the option's data buffers.
--- include/trace-cmd/trace-cmd.h | 5 ++ include/traceevent/event-parse.h | 1 + lib/trace-cmd/trace-output.c | 116 +++++++++++++++++++++++++++---- 3 files changed, 108 insertions(+), 14 deletions(-)
include/trace-cmd/trace-cmd.h +5 -0
@@ -261,11 +261,16 @@ struct tracecmd_output *tracecmd_create_init_file_override(const char *output_fistruct tracecmd_option *tracecmd_add_option (struct tracecmd_output *handle,unsigned short id,int size,const void *data); +struct tracecmd_option * +tracecmd_add_option_v (struct tracecmd_output *handle, +unsigned short id,const struct iovec *vector,int count); +struct tracecmd_option *tracecmd_add_buffer_option (struct tracecmd_output *handle,const char *name,int cpus);int tracecmd_write_cpus (struct tracecmd_output *handle,int cpus);int tracecmd_write_options (struct tracecmd_output *handle); +int tracecmd_append_options (struct tracecmd_output *handle);int tracecmd_update_option (struct tracecmd_output *handle,struct tracecmd_option *option,int size,const void *data);
include/traceevent/event-parse.h +1 -0
@@ -11,6 +11,7 @@#include <stdio.h> #include <regex.h> #include <string.h> +#include <sys/uio.h> #include "traceevent/trace-seq.h"
lib/trace-cmd/trace-output.c +102 -14
161 lines not on this page · raw message
Signature · 1 line
2.23.0
Re: [PATCH v17 08/18] trace-cmd: Implement new API tracecmd_add_option_v()
Quoted 41 lines · On Tue, 3 Dec 2019 12:35:12 +0200 "Tzvetomir Stoyanov (VMware)"…
lib/trace-cmd/trace-output.c +37
-0@@ -1046,6 +1083,57 @@ int tracecmd_write_options(struct tracecmd_output *handle)return 0 ; } +int tracecmd_append_options (struct tracecmd_output *handle) +{ +struct tracecmd_option *options; +unsigned short option; +unsigned short endian2; +unsigned int endian4; +off_t offset; +int r; + +/* If already written, ignore */ +if (handle->options_written) +return 0 ; + +if (lseek64 (handle->fd,0 ,SEEK_END ) == (off_t)-1 ) +return -1 ; + offset =lseek64 (handle->fd,-2 ,SEEK_CUR ); +if (offset == (off_t)-1 ) +return -1 ; + + r =pread (handle->fd, &option,2 , offset); +if (r !=2 || option !=TRACECMD_OPTION_DONE ) +return -1 ; + +list_for_each_entry (options, &handle->options, list) { + endian2 =convert_endian_2 (handle, options->id); +if (do_write_check (handle, &endian2,2 )) +return -1 ; + + endian4 =convert_endian_4 (handle, options->size); +if (do_write_check (handle, &endian4,4 )) +return -1 ; + +/* Save the data location in case it needs to be updated */ + options->offset =lseek64 (handle->fd,0 ,SEEK_CUR ); + +if (do_write_check (handle, options->data, + options->size))
lib/trace-cmd/trace-output.c int tracecmd_write_options(struct tracecmd_output *handle) · line 1132
Quoted 10 lines
lib/trace-cmd/trace-output.c +10
-0+return -1 ; + } + + option =TRACECMD_OPTION_DONE ; + +if (do_write_check (handle, &option,2 )) +return -1 ; + + handle->options_written =1 ; + handle->options_written =1 ;
I think I'm seeing double.
-- Steve
Quoted 7 lines
lib/trace-cmd/trace-output.c +4
-0+ +return 0 ; +} +int tracecmd_update_option (struct tracecmd_output *handle,struct tracecmd_option *option,int size,const void *data)
PATCH v17 9/18 trace-cmd: Add new API to generate a unique ID of the tracing session
In reply to Tzvetomir Stoyanov (VMware)A new libtracecmd API is implemented:
unsigned long long tracecmd_generate_traceid(void);
At trace-cmd init phase a unique ID, used for the current tracing session.
--- include/trace-cmd/trace-cmd.h | 1 + lib/trace-cmd/trace-util.c | 31 +++++++++++++++++++++++++++++++ tracecmd/include/trace-local.h | 1 + tracecmd/trace-record.c | 1 + 4 files changed, 34 insertions(+)
include/trace-cmd/trace-cmd.h +1 -0
@@ -441,6 +441,7 @@ void tracecmd_plog_error(const char *fmt, ...);int tracecmd_set_logfile (char *logfile);/* --- System --- */ +unsigned long long tracecmd_generate_traceid (void );int tracecmd_count_cpus (void );/* --- Hack! --- */
lib/trace-cmd/trace-util.c +31 -0
@@ -18,6 +18,8 @@#include <sys/mount.h> #include <sys/types.h> #include <sys/stat.h> +#include <sys/sysinfo.h> +#include <time.h> #include "trace-cmd.h" #include "event-utils.h"
@@ -1299,3 +1301,32 @@ error:tracecmd_put_tracing_file (path);return ret; } + +#define FNV_64_PRIME 0x100000001b3ULL +unsigned long long tracecmd_generate_traceid (void ) +{ +unsigned long long hash =0 ; +unsigned char *ustr; +struct sysinfo sinfo; +struct timespec ts; +char *str =NULL ; + +clock_gettime (CLOCK_MONOTONIC_RAW , &ts); +sysinfo (&sinfo); +asprintf (&str,"%ld %ld %ld %ld %ld %ld %ld %ld %d" , + ts.tv_sec, ts.tv_nsec, + sinfo.loads[0 ], sinfo.loads[1 ], sinfo.loads[2 ], + sinfo.freeram, sinfo.sharedram, sinfo.freeswap, + sinfo.procs); +if (!str) +return 0 ; + ustr = (unsigned char *)str; + hash =0 ; +while (*ustr) { + hash ^= (unsigned long long )*ustr++; + hash *=FNV_64_PRIME ; + } + +free (str); +return hash; +}
tracecmd/include/trace-local.h +1 -0
@@ -182,6 +182,7 @@ struct pid_addr_maps {struct buffer_instance {struct buffer_instance *next;struct tracecmd_instance *ftrace; +unsigned long long trace_id;char *cpumask;struct event_list *events;struct event_list **event_next;
tracecmd/trace-record.c +1 -0
@@ -5062,6 +5062,7 @@ void init_top_instance(void) top_instance.ftrace =tracecmd_create_instance (NULL ); top_instance.cpu_count =tracecmd_count_cpus (); top_instance.flags =BUFFER_FL_KEEP ; + top_instance.trace_id =tracecmd_generate_traceid ();tracecmd_init_instance (&top_instance); }
Signature · 1 line
2.23.0
PATCH v17 10/18 trace-cmd: Store the session tracing ID in the trace.dat file
In reply to Tzvetomir Stoyanov (VMware)The ID of the current tracing session is stored in the trace.dat file. A new file option is introduced:
TRACECMD_OPTION_TRACEID
The data is stored in the file as NULL terminated string:
"\nTraceID: %llu\n" A new libtracecmd API is introduced, to get the ID when the trace.dat file is read: unsigned long long tracecmd_get_traceid(struct tracecmd_input *handle);
--- include/trace-cmd/trace-cmd.h | 2 ++ lib/trace-cmd/trace-input.c | 25 +++++++++++++++++++++++++ tracecmd/trace-record.c | 15 +++++++++++++++ 3 files changed, 42 insertions(+)
include/trace-cmd/trace-cmd.h +2 -0
@@ -86,6 +86,7 @@ enum {TRACECMD_OPTION_CPUCOUNT ,TRACECMD_OPTION_VERSION ,TRACECMD_OPTION_PROCMAPS , + TRACECMD_OPTION_TRACEID, }; enum {
@@ -128,6 +129,7 @@ int tracecmd_copy_headers(struct tracecmd_input *handle, int fd);void tracecmd_set_flag (struct tracecmd_input *handle,int flag);void tracecmd_clear_flag (struct tracecmd_input *handle,int flag);unsigned long tracecmd_get_flags (struct tracecmd_input *handle); +unsigned long long tracecmd_get_traceid (struct tracecmd_input *handle);void tracecmd_parse_trace_clock (struct tracecmd_input *handle,char *file,int size);
lib/trace-cmd/trace-input.c +25 -0
@@ -79,6 +79,7 @@ struct tracecmd_input {struct tep_plugin_list *plugin_list;struct tracecmd_input *parent;unsigned long flags; +unsigned long long trace_id;int fd;int long_size;int page_size;
@@ -2169,6 +2170,15 @@ static void procmap_free(struct pid_addr_maps *maps)free (maps); } +static int trace_traceid_load (struct tracecmd_input *handle,char *buf) +{ + +if (sscanf (buf,"%*s %llu" , &handle->trace_id) ==1 ) +return 0 ; + +return -1 ; +} +#define STR_PROCMAP_LINE_MAX (PATH_MAX+22)static int trace_pid_map_load (struct tracecmd_input *handle,char *buf) {
@@ -2406,6 +2416,10 @@ static int handle_options(struct tracecmd_input *handle)if (buf[size-1 ] =='\0' )trace_pid_map_load (handle, buf);break ; +case TRACECMD_OPTION_TRACEID : +if (buf[size-1 ] =='\0' ) +trace_traceid_load (handle, buf); +break ;default :warning ("unknown option %d" , option);break ;
@@ -3507,3 +3521,14 @@ void tracecmd_set_show_data_func(struct tracecmd_input *handle, { handle->show_data_func = func; } + +/** +* tracecmd_get_traceid - get the trace id of the session +* @handle: input handle for the trace.dat file +* +* Returns the trace id, written in the trace file +*/ +unsigned long long tracecmd_get_traceid (struct tracecmd_input *handle) +{ +return handle->trace_id; +}
tracecmd/trace-record.c +15 -0
@@ -3715,6 +3715,19 @@ add_pid_maps(struct tracecmd_output *handle, struct buffer_instance *instance)trace_seq_destroy (&s); } +static void +add_trace_id (struct tracecmd_output *handle,struct buffer_instance *instance) +{ +char *str =NULL ; +int len; + + len =asprintf (&str,"\nTraceID: %llu\n" , instance->trace_id); +if (len <1 ) +return ; +tracecmd_add_option (handle,TRACECMD_OPTION_TRACEID , len +1 , str); +free (str); +} +static void add_buffer_stat (struct tracecmd_output *handle,struct buffer_instance *instance) {
@@ -3818,6 +3831,8 @@ static void add_options(struct tracecmd_output *handle, struct common_record_conadd_option_hooks (handle);add_uname (handle);add_version (handle); +if (!no_top_instance ()) +add_trace_id (handle, &top_instance); }static void write_guest_file (struct buffer_instance *instance)
Signature · 1 line
2.23.0
PATCH v17 11/18 trace-cmd: Exchange tracing IDs between host and guest
In reply to Tzvetomir Stoyanov (VMware)Extend the trace request and trace reply messages, to include also the IDs of host and guest tracing sessions. Those IDs are used to unambiguously match the tracing sessions, when reading trace.dat files.
--- include/trace-cmd/trace-cmd.h | 12 ++++++++---- lib/trace-cmd/trace-msg.c | 31 +++++++++++++++++++++++-------- tracecmd/include/trace-local.h | 5 +++-- tracecmd/trace-agent.c | 12 +++++++----- tracecmd/trace-record.c | 12 +++++++++--- 5 files changed, 50 insertions(+), 22 deletions(-)
include/trace-cmd/trace-cmd.h +8 -4
@@ -353,16 +353,20 @@ bool tracecmd_msg_done(struct tracecmd_msg_handle *msg_handle);void tracecmd_msg_set_done (struct tracecmd_msg_handle *msg_handle);int tracecmd_msg_send_trace_req (struct tracecmd_msg_handle *msg_handle, -int argc,char **argv,bool use_fifos); +int argc,char **argv,bool use_fifos, +unsigned long long trace_id);int tracecmd_msg_recv_trace_req (struct tracecmd_msg_handle *msg_handle, -int *argc,char ***argv,bool *use_fifos); +int *argc,char ***argv,bool *use_fifos, +unsigned long long *trace_id);int tracecmd_msg_send_trace_resp (struct tracecmd_msg_handle *msg_handle,int nr_cpus,int page_size, -unsigned int *ports,bool use_fifos); +unsigned int *ports,bool use_fifos, +unsigned long long trace_id);int tracecmd_msg_recv_trace_resp (struct tracecmd_msg_handle *msg_handle,int *nr_cpus,int *page_size, -unsigned int **ports,bool *use_fifos); +unsigned int **ports,bool *use_fifos, +unsigned long long *trace_id);/* --- ftrace instances --- */ struct tracecmd_instance {
lib/trace-cmd/trace-msg.c +23 -8
@@ -63,12 +63,14 @@ struct tracecmd_msg_rinit {struct tracecmd_msg_trace_req {be32 flags;be32 argc; +u64 trace_id; } __attribute__((packed));struct tracecmd_msg_trace_resp {be32 flags;be32 cpus;be32 page_size; +u64 trace_id; } __attribute__((packed));struct tracecmd_msg_header {
@@ -811,7 +813,8 @@ int tracecmd_msg_wait_close_resp(struct tracecmd_msg_handle *msg_handle)return tracecmd_msg_wait_for_cmd (msg_handle,MSG_CLOSE_RESP ); } -static int make_trace_req (struct tracecmd_msg *msg,int argc,char **argv,bool use_fifos) +static int make_trace_req (struct tracecmd_msg *msg,int argc,char **argv, +bool use_fifos,unsigned long long trace_id) {size_t args_size =0 ;char *p;
@@ -823,6 +826,7 @@ static int make_trace_req(struct tracecmd_msg *msg, int argc, char **argv, bool msg->hdr.size =htonl (ntohl (msg->hdr.size) + args_size); msg->trace_req.flags = use_fifos ?htonl (MSG_TRACE_USE_FIFOS ) :htonl (0 ); msg->trace_req.argc =htonl (argc); + msg->trace_req.trace_id =htonll (trace_id); msg->buf =calloc (args_size,1 );if (!msg->buf)return -ENOMEM ;
@@ -835,13 +839,15 @@ static int make_trace_req(struct tracecmd_msg *msg, int argc, char **argv, bool }int tracecmd_msg_send_trace_req (struct tracecmd_msg_handle *msg_handle, -int argc,char **argv,bool use_fifos) +int argc,char **argv,bool use_fifos, +unsigned long long trace_id) {struct tracecmd_msg msg;int ret;tracecmd_msg_init (MSG_TRACE_REQ , &msg); - ret =make_trace_req (&msg, argc, argv, use_fifos); + ret =make_trace_req (&msg, argc, argv, + use_fifos, trace_id);if (ret <0 )return ret;
@@ -854,7 +860,8 @@ int tracecmd_msg_send_trace_req(struct tracecmd_msg_handle *msg_handle,* free(argv); */ int tracecmd_msg_recv_trace_req (struct tracecmd_msg_handle *msg_handle, -int *argc,char ***argv,bool *use_fifos) +int *argc,char ***argv,bool *use_fifos, +unsigned long long *trace_id) {struct tracecmd_msg msg;char *p, *buf_end, **args;
@@ -901,6 +908,7 @@ int tracecmd_msg_recv_trace_req(struct tracecmd_msg_handle *msg_handle, *argc = nr_args; *argv = args; *use_fifos =ntohl (msg.trace_req.flags) &MSG_TRACE_USE_FIFOS ; + *trace_id =ntohll (msg.trace_req.trace_id);/* * On success we're passing msg.buf to the caller through argv[0] so we
@@ -921,7 +929,8 @@ out: }static int make_trace_resp (struct tracecmd_msg *msg,int page_size,int nr_cpus, -unsigned int *ports,bool use_fifos) +unsigned int *ports,bool use_fifos, +unsigned long long trace_id) {int data_size;
@@ -935,19 +944,22 @@ static int make_trace_resp(struct tracecmd_msg *msg, int page_size, int nr_cpus, msg->trace_resp.flags = use_fifos ?htonl (MSG_TRACE_USE_FIFOS ) :htonl (0 ); msg->trace_resp.cpus =htonl (nr_cpus); msg->trace_resp.page_size =htonl (page_size); + msg->trace_resp.trace_id =htonll (trace_id);return 0 ; }int tracecmd_msg_send_trace_resp (struct tracecmd_msg_handle *msg_handle,int nr_cpus,int page_size, -unsigned int *ports,bool use_fifos) +unsigned int *ports,bool use_fifos, +unsigned long long trace_id) {struct tracecmd_msg msg;int ret;tracecmd_msg_init (MSG_TRACE_RESP , &msg); - ret =make_trace_resp (&msg, page_size, nr_cpus, ports, use_fifos); + ret =make_trace_resp (&msg, page_size, nr_cpus, ports, + use_fifos, trace_id);if (ret <0 )return ret;
@@ -956,13 +968,15 @@ int tracecmd_msg_send_trace_resp(struct tracecmd_msg_handle *msg_handle,int tracecmd_msg_recv_trace_resp (struct tracecmd_msg_handle *msg_handle,int *nr_cpus,int *page_size, -unsigned int **ports,bool *use_fifos) +unsigned int **ports,bool *use_fifos, +unsigned long long *trace_id) {struct tracecmd_msg msg;char *p, *buf_end;ssize_t buf_len;int i, ret; +memset (&msg,0 ,sizeof (msg)); ret =tracecmd_msg_recv (msg_handle->fd, &msg);if (ret <0 )return ret;
@@ -981,6 +995,7 @@ int tracecmd_msg_recv_trace_resp(struct tracecmd_msg_handle *msg_handle, *use_fifos =ntohl (msg.trace_resp.flags) &MSG_TRACE_USE_FIFOS ; *nr_cpus =ntohl (msg.trace_resp.cpus); *page_size =ntohl (msg.trace_resp.page_size); + *trace_id =ntohll (msg.trace_resp.trace_id); *ports =calloc (*nr_cpus,sizeof (**ports));if (!*ports) { ret = -ENOMEM ;
tracecmd/include/trace-local.h +3 -2
@@ -98,8 +98,9 @@ void trace_usage(int argc, char **argv);int trace_record_agent (struct tracecmd_msg_handle *msg_handle,int cpus,int *fds, -int argc,char **argv,bool use_fifos); - +int argc,char **argv, +bool use_fifos, +unsigned long long trace_id);struct hook_list ; void trace_init_profile(struct tracecmd_input *handle, struct hook_list *hooks,
tracecmd/trace-agent.c +7 -5
@@ -128,6 +128,7 @@ cleanup:static void agent_handle (int sd,int nr_cpus,int page_size) {struct tracecmd_msg_handle *msg_handle; +unsigned long long trace_id;unsigned int *ports;char **argv =NULL ;int argc =0 ;
@@ -144,7 +145,8 @@ static void agent_handle(int sd, int nr_cpus, int page_size)if (!msg_handle)die ("Failed to allocate message handle" ); - ret =tracecmd_msg_recv_trace_req (msg_handle, &argc, &argv, &use_fifos); + ret =tracecmd_msg_recv_trace_req (msg_handle, &argc, &argv, + &use_fifos, &trace_id);if (ret <0 )die ("Failed to receive trace request" );
@@ -153,13 +155,13 @@ static void agent_handle(int sd, int nr_cpus, int page_size)if (!use_fifos)make_vsocks (nr_cpus, fds, ports); - + trace_id =tracecmd_generate_traceid (); ret =tracecmd_msg_send_trace_resp (msg_handle, nr_cpus, page_size, - ports, use_fifos); + ports, use_fifos, trace_id);if (ret <0 )die ("Failed to send trace response" ); - -trace_record_agent (msg_handle, nr_cpus, fds, argc, argv, use_fifos); +trace_record_agent (msg_handle, nr_cpus, fds, argc, argv, + use_fifos, trace_id);free (argv[0 ]);free (argv);
tracecmd/trace-record.c +9 -3
@@ -3507,12 +3507,16 @@ static void connect_to_agent(struct buffer_instance *instance)die ("Failed to allocate message handle" ); ret =tracecmd_msg_send_trace_req (msg_handle, instance->argc, - instance->argv, use_fifos); + instance->argv, use_fifos, + top_instance.trace_id); +if (ret <0 )die ("Failed to send trace request" ); ret =tracecmd_msg_recv_trace_resp (msg_handle, &nr_cpus, &page_size, - &ports, &use_fifos); + &ports, &use_fifos, + &instance->trace_id); +if (ret <0 )die ("Failed to receive trace response" );
@@ -6117,7 +6121,8 @@ void trace_record(int argc, char **argv)int trace_record_agent (struct tracecmd_msg_handle *msg_handle,int cpus,int *fds,int argc,char **argv, -bool use_fifos) +bool use_fifos, +unsigned long long trace_id) {struct common_record_context ctx;char **argv_plus;
@@ -6147,6 +6152,7 @@ int trace_record_agent(struct tracecmd_msg_handle *msg_handle, ctx.instance->flags |=BUFFER_FL_AGENT ; ctx.instance->msg_handle = msg_handle; msg_handle->version =V3_PROTOCOL ; + top_instance.trace_id = trace_id;record_trace (argc, argv, &ctx);free (argv_plus);
Signature · 1 line
2.23.0
Re: [PATCH v17 11/18] trace-cmd: Exchange tracing IDs between host and guest
tracecmd/include/trace-local.h void trace_usage(int argc, char **argv); · line 103
Quoted 13 lines · On Tue, 3 Dec 2019 12:35:15 +0200 "Tzvetomir Stoyanov (VMware)"…
tracecmd/include/trace-local.h +3
-2@@ -98,8 +98,9 @@ void trace_usage(int argc, char **argv);int trace_record_agent (struct tracecmd_msg_handle *msg_handle,int cpus,int *fds, -int argc,char **argv,bool use_fifos); - +int argc,char **argv, +bool use_fifos, +unsigned long long trace_id);
I'd keep the empty line between the prototype and the struct declaration below.
-- Steve
Quoted 3 lines
tracecmd/include/trace-local.h +0
-0struct hook_list ; void trace_init_profile(struct tracecmd_input *handle, struct hook_list *hooks,
PATCH v17 12/18 trace-cmd: Implement new option in trace.dat file: TRACECMD_OPTION_TIME_SHIFT
In reply to Tzvetomir Stoyanov (VMware)From: Tzvetomir Stoyanov <[email protected]>
The TRACECMD_OPTION_TIME_SHIFT is used when synchronizing trace time stamps between two trace.dat files. It contains multiple long long (time, offset) pairs, describing time stamps _offset_, measured in the given local _time_. The content of the option buffer is:
8 bytes - long long integer, ID of the tracing session
4 bytes - integer, count of timestamp offsets
long long array of size _count_, local time in which the offset is measured long long array of size _count_, offset of the time stamps
--- include/trace-cmd/trace-cmd.h | 6 ++ lib/trace-cmd/trace-input.c | 170 +++++++++++++++++++++++++++++++++- 2 files changed, 174 insertions(+), 2 deletions(-)
include/trace-cmd/trace-cmd.h +6 -0
@@ -87,6 +87,7 @@ enum {TRACECMD_OPTION_VERSION ,TRACECMD_OPTION_PROCMAPS ,TRACECMD_OPTION_TRACEID , + TRACECMD_OPTION_TIME_SHIFT, }; enum {
@@ -130,6 +131,11 @@ void tracecmd_set_flag(struct tracecmd_input *handle, int flag);void tracecmd_clear_flag (struct tracecmd_input *handle,int flag);unsigned long tracecmd_get_flags (struct tracecmd_input *handle);unsigned long long tracecmd_get_traceid (struct tracecmd_input *handle); +int tracecmd_get_guest_cpumap (struct tracecmd_input *handle, +unsigned long long trace_id, +char *name, +int *vcpu_count,int *cpu_pid); +unsigned long long tracecmd_get_tsync_peer (struct tracecmd_input *handle);void tracecmd_parse_trace_clock (struct tracecmd_input *handle,char *file,int size);
lib/trace-cmd/trace-input.c +168 -2
227 lines not on this page · raw message
Signature · 1 line
2.23.0
Re: [PATCH v17 12/18] trace-cmd: Implement new option in trace.dat file: TRACECMD_OPTION_TIME_SHIFT
Quoted 73 lines · On Tue, 3 Dec 2019 12:35:16 +0200 "Tzvetomir Stoyanov (VMware)"…
From: Tzvetomir Stoyanov <[email protected]>
The
TRACECMD_OPTION_TIME_SHIFTis used when synchronizing trace time stamps between two trace.dat files. It contains multiple long long (time, offset) pairs, describing time stamps _offset_, measured in the given local _time_. The content of the option buffer is:
8 bytes - long long integer, ID of the tracing session
4 bytes - integer, count of timestamp offsets
long long array of size _count_, local time in which the offset is measured long long array of size _count_, offset of the time stampsSigned-off-by Tzvetomir Stoyanov <[email protected]>--- include/trace-cmd/trace-cmd.h | 6 ++ lib/trace-cmd/trace-input.c | 170 +++++++++++++++++++++++++++++++++- 2 files changed, 174 insertions(+), 2 deletions(-)include/trace-cmd/trace-cmd.h +6
-0@@ -87,6 +87,7 @@ enum {TRACECMD_OPTION_VERSION ,TRACECMD_OPTION_PROCMAPS ,TRACECMD_OPTION_TRACEID , + TRACECMD_OPTION_TIME_SHIFT, }; enum {@@ -130,6 +131,11 @@ void tracecmd_set_flag(struct tracecmd_input *handle, int flag);void tracecmd_clear_flag (struct tracecmd_input *handle,int flag);unsigned long tracecmd_get_flags (struct tracecmd_input *handle);unsigned long long tracecmd_get_traceid (struct tracecmd_input *handle); +int tracecmd_get_guest_cpumap (struct tracecmd_input *handle, +unsigned long long trace_id, +char *name, +int *vcpu_count,int *cpu_pid); +unsigned long long tracecmd_get_tsync_peer (struct tracecmd_input *handle);void tracecmd_parse_trace_clock (struct tracecmd_input *handle,char *file,int size);lib/trace-cmd/trace-input.c +13
-0@@ -74,6 +74,18 @@ struct input_buffer_instance {size_t offset; }; +struct ts_offset_sample { +long long time; +long long offset; +}; + +struct host_trace_info { +bool sync_enable; +unsigned long long trace_id; +int ts_samples_count; +struct ts_offset_sample *ts_samples; +}; +struct tracecmd_input {struct tep_handle *pevent;struct tep_plugin_list *plugin_list;@@ -92,6 +104,7 @@ struct tracecmd_input {bool use_pipe;struct cpu_data *cpu_data;long long ts_offset; +struct host_trace_info host;double ts2secs;char * cpustats;char * uname;
lib/trace-cmd/trace-input.c static void free_next(struct tracecmd_input *handle, int cpu) · line 1094
Quoted 10 lines
lib/trace-cmd/trace-input.c +7
-0@@ -1072,6 +1085,67 @@ static void free_next(struct tracecmd_input *handle, int cpu)free_record (record); } +static inline unsigned long long +timestamp_correction_calc (unsigned long long ts,struct ts_offset_sample *min, +struct ts_offset_sample *max) +{ +long long tscor = min->offset + + (((((long long )ts) - min->time)* + (max->offset-min->offset))/(max->time-min->time));
When every I see a division like this, I'd like to think we should add:
long long offset = ((long long)ts - min->time) * (max->offset - min->offset); long long delta = max->time - min->time; long long tscor = min->offset + (offset + delta / 2) / delta;
This handles rounding of delta instead of truncating. Also, it's best to have spaces around operators, as it's hard to see where the variable stops and the operation beings with max->time-min->time.
lib/trace-cmd/trace-input.c static void free_next(struct tracecmd_input *handle, int cpu) · line 1097
Quoted 3 lines
lib/trace-cmd/trace-input.c +3
-0+ +if (tscor <0 ) +return ts -llabs (tscor);
I'm curious to why the test and using llabs, instead of just returning:
ts + tscor ?
Is there a difference between that when tscor is negative and using llabs?
Quoted 5 lines
lib/trace-cmd/trace-input.c +5
-0+ +return ts + tscor; +} + +static unsigned long long timestamp_correct(unsigned long long ts,
lib/trace-cmd/trace-input.c static void free_next(struct tracecmd_input *handle, int cpu) · line 1112
Quoted 10 lines
lib/trace-cmd/trace-input.c +10
-0+ struct tracecmd_input *handle) +{ +struct host_trace_info *host = &handle->host; +int min, mid, max; + +if (handle->ts_offset) +return ts + handle->ts_offset; + +if (!host->sync_enable || !host->ts_samples_count || !host->ts_samples) +return ts;
Hmm, perhaps we should make host->sync_enable false when host->ts_samples_count or host->ts_samples are NULL and remove the extra checks? (slight optimization)
Quoted 21 lines
lib/trace-cmd/trace-input.c +21
-0+ +/* We have one sample, nothing to calc here */ +if (host->ts_samples_count ==1 ) +return ts + host->ts_samples[0 ].offset; + +/* We have two samples, nothing to search here */ +if (host->ts_samples_count ==2 ) +return timestamp_correction_calc (ts, &host->ts_samples[0 ], + &host->ts_samples[1 ]); + +/* We have more than two samples */ +if (ts <= host->ts_samples[0 ].time) +return timestamp_correction_calc (ts, + &host->ts_samples[0 ], + &host->ts_samples[1 ]); +else if (ts >= host->ts_samples[host->ts_samples_count-1 ].time) +return timestamp_correction_calc (ts, + &host->ts_samples[host->ts_samples_count-2 ], + &host->ts_samples[host->ts_samples_count-1 ]); + min =0 ; + max = host->ts_samples_count-1 ;
lib/trace-cmd/trace-input.c static void free_next(struct tracecmd_input *handle, int cpu) · line 1143
Quoted 10 lines
lib/trace-cmd/trace-input.c +10
-0+ mid = (min + max)/2 ; +while (min <= max) { +if (ts < host->ts_samples[mid].time) + max = mid -1 ; +else if (ts > host->ts_samples[mid].time) + min = mid +1 ; +else +break ; + mid = (min + max)/2 ; + }
Hmm, probably should libc's bsearch() instead of open coding a binary search.
Quoted 31 lines
lib/trace-cmd/trace-input.c +8
-2+ +return timestamp_correction_calc (ts, &host->ts_samples[mid], + &host->ts_samples[mid+1 ]); +} +/* * Page is mapped, now read in the page header info. */ @@ -1093,7 +1167,7 @@ static int update_page_info(struct tracecmd_input *handle, int cpu) kbuffer_subbuffer_size(kbuf));return -1 ; } - handle->cpu_data[cpu].timestamp =kbuffer_timestamp (kbuf) + handle->ts_offset; + handle->cpu_data[cpu].timestamp =timestamp_correct (kbuffer_timestamp (kbuf), handle);if (handle->ts2secs) handle->cpu_data[cpu].timestamp *= handle->ts2secs;@@ -1820,7 +1894,7 @@ read_again: goto read_again; } - handle->cpu_data[cpu].timestamp = ts + handle->ts_offset; + handle->cpu_data[cpu].timestamp =timestamp_correct (ts, handle);if (handle->ts2secs) { handle->cpu_data[cpu].timestamp *= handle->ts2secs;@@ -2143,6 +2217,47 @@ void tracecmd_set_ts2secs(struct tracecmd_input *handle, handle->use_trace_clock = false; } +static int tsync_offset_cmp (const void *a,const void *b)
lib/trace-cmd/trace-input.c void tracecmd_set_ts2secs(struct tracecmd_input *handle, · line 2230
Quoted 10 lines
lib/trace-cmd/trace-input.c +10
-0+{ +struct ts_offset_sample *ts_a = (struct ts_offset_sample *)a; +struct ts_offset_sample *ts_b = (struct ts_offset_sample *)b; + +if (ts_a->time > ts_b->time) +return 1 ; +if (ts_a->time < ts_b->time) +return -1 ; +return 0 ; +}
We have the compare function for bsearch() here.
Quoted 34 lines
lib/trace-cmd/trace-input.c +30
-0+ +static void tsync_offset_load (struct tracecmd_input *handle,char *buf) +{ +struct host_trace_info *host = &handle->host; +long long *buf8 = (long long *)buf; +int i, j; + +for (i =0 ; i < host->ts_samples_count; i++) { + host->ts_samples[i].time =tep_read_number (handle->pevent, + buf8 + i,8 ); + host->ts_samples[i].offset =tep_read_number (handle->pevent, + buf8 + host->ts_samples_count+i,8 ); + } +qsort (host->ts_samples, host->ts_samples_count, +sizeof (struct ts_offset_sample ), tsync_offset_cmp); +/* Filter possible samples with equal time */ +for (i =0 , j =0 ; i < host->ts_samples_count; i++) { +if (i ==0 || host->ts_samples[i].time != host->ts_samples[i-1 ].time) + host->ts_samples[j++] = host->ts_samples[i]; + } + host->ts_samples_count = j; + host->sync_enable = true; +} + +static void trace_tsync_offset_free (struct host_trace_info *host) +{ +free (host->ts_samples); + host->ts_samples =NULL ; +} +static int trace_pid_map_cmp (const void *a,const void *b) {struct tracecmd_proc_addr_map *m_a = (struct tracecmd_proc_addr_map *)a;
lib/trace-cmd/trace-input.c static int handle_options(struct tracecmd_input *handle) · line 2441
Quoted 4 lines
lib/trace-cmd/trace-input.c +1
-0@@ -2323,6 +2438,7 @@ static int handle_options(struct tracecmd_input *handle)struct input_buffer_instance *buffer;struct hook_list *hook;char *buf; +int sampes_size;
"sampes"? Is this short for "samples"? Just use "samples_size" then.
Quoted 59 lines
lib/trace-cmd/trace-input.c +38
-0int cpus;/* By default, use usecs, unless told otherwise */ @@ -2370,6 +2486,28 @@ static int handle_options(struct tracecmd_input *handle) offset =strtoll (buf,NULL ,0 ); handle->ts_offset += offset;break ; +case TRACECMD_OPTION_TIME_SHIFT : +/* +* long long int (8 bytes) trace session ID +* int (4 bytes) count of timestamp offsets. +* long long array of size [count] of times, +* when the offsets were calculated. +* long long array of size [count] of timestamp offsets. +*/ +if (size <12 || handle->flags &TRACECMD_FL_IGNORE_DATE ) +break ; + handle->host.trace_id =tep_read_number (handle->pevent, + buf,8 ); + handle->host.ts_samples_count =tep_read_number (handle->pevent, + buf +8 ,4 ); + sampes_size = (8 * handle->host.ts_samples_count); +if (size != (12 + (2 * sampes_size))) +break ; + handle->host.ts_samples =malloc (2 * sampes_size); +if (!handle->host.ts_samples) +return -ENOMEM ; +tsync_offset_load (handle, buf +12 ); +break ;case TRACECMD_OPTION_CPUSTAT : buf[size-1 ] ='\n' ; cpustats =realloc (cpustats, cpustats_size + size +1 );@@ -3078,6 +3216,8 @@ void tracecmd_close(struct tracecmd_input *handle)trace_pid_map_free (handle->pid_maps); handle->pid_maps =NULL ; +trace_tsync_offset_free (&handle->host); +if (handle->flags &TRACECMD_FL_BUFFER_INSTANCE )tracecmd_close (handle->parent);else {@@ -3532,3 +3672,29 @@ unsigned long long tracecmd_get_traceid(struct tracecmd_input *handle) {return handle->trace_id; } + +/** +* tracecmd_get_tsync_peer - get the trace session id of the peer host +* @handle: input handle for the trace.dat file +* +* Returns the trace id of the peer host, written in the trace file +* +* This information is stored in guest trace.dat file +*/ +unsigned long long tracecmd_get_tsync_peer (struct tracecmd_input *handle) +{ +return handle->host.trace_id; +} +
lib/trace-cmd/trace-input.c unsigned long long tracecmd_get_traceid(struct tracecmd_input *handle) · line 3698
Quoted 10 lines
lib/trace-cmd/trace-input.c +10
-0+/** +* tracecmd_enable_tsync - enable / disable the timestamps correction +* @handle: input handle for the trace.dat file +* @enable: enable / disable the timestamps correction +* +* Enables or disables timestamps correction on file load, using the array of +* recorded time offsets +*/ +void tracecmd_enable_tsync (struct tracecmd_input *handle,bool enable) +{
Perhaps here we should check if samples are allocated already, and only allow it to be enabled if they are.
-- Steve
Quoted 2 lines
lib/trace-cmd/trace-input.c +2
-0+ handle->host.sync_enable = enable; +}
Re: [PATCH v17 12/18] trace-cmd: Implement new option in trace.dat file: TRACECMD_OPTION_TIME_SHIFT
On Thu, Dec 5, 2019 at 2:46 AM Steven Rostedt <[email protected]> wrote:
[ ... ]
Quoted 24 lines
lib/trace-cmd/trace-input.c static void free_next(struct tracecmd_input *handle, int cpu) · line 1094
Quoted 7 lines
lib/trace-cmd/trace-input.c +7
-0+static inline unsigned long long +timestamp_correction_calc (unsigned long long ts,struct ts_offset_sample *min, +struct ts_offset_sample *max) +{ +long long tscor = min->offset + + (((((long long )ts) - min->time)* + (max->offset-min->offset))/(max->time-min->time));When every I see a division like this, I'd like to think we should add:
long long offset = ((long long)ts - min->time) * (max->offset - min->offset); long long delta = max->time - min->time; long long tscor = min->offset + (offset + delta / 2) / delta;This handles rounding of delta instead of truncating. Also, it's best to have spaces around operators, as it's hard to see where the variable stops and the operation beings with max->time-min->time.
lib/trace-cmd/trace-input.c static void free_next(struct tracecmd_input *handle, int cpu) · line 1097
Quoted 3 lines
lib/trace-cmd/trace-input.c +3
-0+ +if (tscor <0 ) +return ts -llabs (tscor);
Quoted 7 lines
I'm curious to why the test and using llabs, instead of just returning:
ts + tscor ?
Is there a difference between that when tscor is negative and using llabs?
The only reason to use llabs() is mixed signed - unsigned calculation. As time correction tscor can be negative, will it be converted to unsigned in "ts + tscor" ?
Quoted 46 lines
Quoted 5 lines
lib/trace-cmd/trace-input.c +5
-0+ +return ts + tscor; +} + +static unsigned long long timestamp_correct(unsigned long long ts,lib/trace-cmd/trace-input.c static void free_next(struct tracecmd_input *handle, int cpu) · line 1112
Quoted 10 lines
lib/trace-cmd/trace-input.c +10
-0+ struct tracecmd_input *handle) +{ +struct host_trace_info *host = &handle->host; +int min, mid, max; + +if (handle->ts_offset) +return ts + handle->ts_offset; + +if (!host->sync_enable || !host->ts_samples_count || !host->ts_samples) +return ts;Hmm, perhaps we should make
host->sync_enablefalse whenhost->ts_samples_countorhost->ts_samplesare NULL and remove the extra checks? (slight optimization)Quoted 26 lines
lib/trace-cmd/trace-input.c +26
-0+ +/* We have one sample, nothing to calc here */ +if (host->ts_samples_count ==1 ) +return ts + host->ts_samples[0 ].offset; + +/* We have two samples, nothing to search here */ +if (host->ts_samples_count ==2 ) +return timestamp_correction_calc (ts, &host->ts_samples[0 ], + &host->ts_samples[1 ]); + +/* We have more than two samples */ +if (ts <= host->ts_samples[0 ].time) +return timestamp_correction_calc (ts, + &host->ts_samples[0 ], + &host->ts_samples[1 ]); +else if (ts >= host->ts_samples[host->ts_samples_count-1 ].time) +return timestamp_correction_calc (ts, + &host->ts_samples[host->ts_samples_count-2 ], + &host->ts_samples[host->ts_samples_count-1 ]); + min =0 ; + max = host->ts_samples_count-1 ; + mid = (min + max)/2 ; +while (min <= max) { +if (ts < host->ts_samples[mid].time) + max = mid -1 ; +else if (ts > host->ts_samples[mid].time)
Quoted 9 lines
lib/trace-cmd/trace-input.c static void free_next(struct tracecmd_input *handle, int cpu) · line 1143
Quoted 5 lines
lib/trace-cmd/trace-input.c +5
-0+ min = mid +1 ; + else + break; + mid = (min + max)/2 ; + }Hmm, probably should libc's
bsearch()instead of open coding a binary search.
I do not use bsearch() as there is no exact match it that search - only the interval in which the timestamp fits should be found. May be there is a way to use bsearch() in this scenario, I can think about it.
[ ...]
Quoted 16 lines
lib/trace-cmd/trace-input.c void tracecmd_set_ts2secs(struct tracecmd_input *handle, · line 2230
Quoted 13 lines
}
lib/trace-cmd/trace-input.c +11
-0+static int tsync_offset_cmp (const void *a,const void *b) +{ +struct ts_offset_sample *ts_a = (struct ts_offset_sample *)a; +struct ts_offset_sample *ts_b = (struct ts_offset_sample *)b; + +if (ts_a->time > ts_b->time) +return 1 ; +if (ts_a->time < ts_b->time) +return -1 ; +return 0 ; +}We have the compare function for
bsearch()here.
We can use it only for exact match, cannot use the same function for finding the interval.
[ ... ]
Quoted 11 lines
lib/trace-cmd/trace-input.c static int handle_options(struct tracecmd_input *handle) · line 2441
Quoted 8 lines
static int
trace_pid_map_cmp(const void *a, const void *b){ struct tracecmd_proc_addr_map *m_a = (struct tracecmd_proc_addr_map *)a;lib/trace-cmd/trace-input.c +1
-0@@ -2323,6 +2438,7 @@ static int handle_options(struct tracecmd_input *handle)struct input_buffer_instance *buffer;struct hook_list *hook;char *buf; +int sampes_size;"sampes"? Is this short for "samples"? Just use "
samples_size" then.
It's a typo, fixed it :)
Quoted 74 lines
Quoted 60 lines
lib/trace-cmd/trace-input.c +38
-0int cpus;/* By default, use usecs, unless told otherwise */ @@ -2370,6 +2486,28 @@ static int handle_options(struct tracecmd_input *handle) offset =strtoll (buf,NULL ,0 ); handle->ts_offset += offset;break ; +case TRACECMD_OPTION_TIME_SHIFT : +/* +* long long int (8 bytes) trace session ID +* int (4 bytes) count of timestamp offsets. +* long long array of size [count] of times, +* when the offsets were calculated. +* long long array of size [count] of timestamp offsets. +*/ +if (size <12 || handle->flags &TRACECMD_FL_IGNORE_DATE ) +break ; + handle->host.trace_id =tep_read_number (handle->pevent, + buf,8 ); + handle->host.ts_samples_count =tep_read_number (handle->pevent, + buf +8 ,4 ); + sampes_size = (8 * handle->host.ts_samples_count); +if (size != (12 + (2 * sampes_size))) +break ; + handle->host.ts_samples =malloc (2 * sampes_size); +if (!handle->host.ts_samples) +return -ENOMEM ; +tsync_offset_load (handle, buf +12 ); +break ;case TRACECMD_OPTION_CPUSTAT : buf[size-1 ] ='\n' ; cpustats =realloc (cpustats, cpustats_size + size +1 );@@ -3078,6 +3216,8 @@ void tracecmd_close(struct tracecmd_input *handle)trace_pid_map_free (handle->pid_maps); handle->pid_maps =NULL ; +trace_tsync_offset_free (&handle->host); +if (handle->flags &TRACECMD_FL_BUFFER_INSTANCE )tracecmd_close (handle->parent);else {@@ -3532,3 +3672,29 @@ unsigned long long tracecmd_get_traceid(struct tracecmd_input *handle) {return handle->trace_id; } + +/** +* tracecmd_get_tsync_peer - get the trace session id of the peer host +* @handle: input handle for the trace.dat file +* +* Returns the trace id of the peer host, written in the trace file +* +* This information is stored in guest trace.dat file +*/ +unsigned long long tracecmd_get_tsync_peer (struct tracecmd_input *handle) +{ +return handle->host.trace_id; +} +lib/trace-cmd/trace-input.c unsigned long long tracecmd_get_traceid(struct tracecmd_input *handle) · line 3698
Quoted 10 lines
lib/trace-cmd/trace-input.c +10
-0+/** +* tracecmd_enable_tsync - enable / disable the timestamps correction +* @handle: input handle for the trace.dat file +* @enable: enable / disable the timestamps correction +* +* Enables or disables timestamps correction on file load, using the array of +* recorded time offsets +*/ +void tracecmd_enable_tsync (struct tracecmd_input *handle,bool enable) +{Perhaps here we should check if samples are allocated already, and only allow it to be enabled if they are.
Quoted 6 lines
-- Steve
Quoted 2 lines
lib/trace-cmd/trace-input.c +2
-0+ handle->host.sync_enable = enable; +}
Thanks !
Signature · 2 lines
Tzvetomir (Ceco) Stoyanov VMware Open Source Technology Center
PATCH v17 13/18 trace-cmd: Add guest information in host's trace.dat file
In reply to Tzvetomir Stoyanov (VMware)New trace.dat option is introduced: TRACECMD_OPTION_GUEST. Written in the host's trace.dat file, it contains information about guests, traced at the same time: guest trace ID, number of VCPUs and PIDs of the host tasks, running those VCPU. The data is stored in the file as NULL terminated string:
"Guest %s %llu %d\n" -> guest name, number of VCPUs "%d %d\n" -> VCPU, PID of host task ..... "%d %d\n" -> VCPU, PID of host task
--- include/trace-cmd/trace-cmd.h | 1 + lib/trace-cmd/trace-input.c | 113 ++++++++++++++++++++++++++++++++++ tracecmd/trace-record.c | 45 ++++++++++++++ 3 files changed, 159 insertions(+)
include/trace-cmd/trace-cmd.h +1 -0
@@ -88,6 +88,7 @@ enum {TRACECMD_OPTION_PROCMAPS ,TRACECMD_OPTION_TRACEID ,TRACECMD_OPTION_TIME_SHIFT , + TRACECMD_OPTION_GUEST, }; enum {
lib/trace-cmd/trace-input.c +113 -0
149 lines not on this page · raw message
tracecmd/trace-record.c +45 -0
63 lines not on this page · raw message
Signature · 1 line
2.23.0
Re: [PATCH v17 13/18] trace-cmd: Add guest information in host's trace.dat file
Quoted 102 lines · On Tue, 3 Dec 2019 12:35:17 +0200 "Tzvetomir Stoyanov (VMware)"…
New trace.dat option is introduced:
TRACECMD_OPTION_GUEST. Written in the host's trace.dat file, it contains information about guests, traced at the same time: guest trace ID, number of VCPUs and PIDs of the host tasks, running those VCPU. The data is stored in the file as NULL terminated string:"Guest %s %llu %d\n" -> guest name, number of VCPUs "%d %d\n" -> VCPU, PID of host task ..... "%d %d\n" -> VCPU, PID of host taskSigned-off-by Tzvetomir Stoyanov (VMware) <[email protected]>--- include/trace-cmd/trace-cmd.h | 1 + lib/trace-cmd/trace-input.c | 113 ++++++++++++++++++++++++++++++++++ tracecmd/trace-record.c | 45 ++++++++++++++ 3 files changed, 159 insertions(+)include/trace-cmd/trace-cmd.h +1
-0@@ -88,6 +88,7 @@ enum {TRACECMD_OPTION_PROCMAPS ,TRACECMD_OPTION_TRACEID ,TRACECMD_OPTION_TIME_SHIFT , + TRACECMD_OPTION_GUEST, }; enum {lib/trace-cmd/trace-input.c +51
-0@@ -79,6 +79,14 @@ struct ts_offset_sample {long long offset; }; +struct guest_trace_info { +struct guest_trace_info *next; +char *name; +unsigned long long trace_id; +int vcpu_count; +int *cpu_pid; +}; +struct host_trace_info {bool sync_enable;unsigned long long trace_id;@@ -112,6 +120,7 @@ struct tracecmd_input {char * trace_clock;struct input_buffer_instance *buffers;int parsing_failures; +struct guest_trace_info *guest;struct tracecmd_ftrace finfo;@@ -2294,6 +2303,73 @@ static int trace_traceid_load(struct tracecmd_input *handle, char *buf)return -1 ; } +static void trace_guests_free (struct tracecmd_input *handle) +{ +struct guest_trace_info *guest; + +while (handle->guest) { + guest = handle->guest; + handle->guest = handle->guest->next; +free (guest->name); +free (guest->cpu_pid); +free (guest); + } +} + +static int trace_guest_load (struct tracecmd_input *handle,char *buf) +{ +struct guest_trace_info *guest =NULL ; +unsigned long long tid; +char *line; +int cpu, pid; + + line =strchr (buf,'\n' ); +if (!line) + goto error; + *line ='\0' ; + + guest =calloc (1 ,sizeof (struct guest_trace_info )); +if (!guest) + goto error; + +if (sscanf (buf,"%*s %ms %llu %d" , &guest->name, &tid, &cpu) !=3 ) + goto error; + guest->trace_id = tid; + guest->vcpu_count = cpu; + + guest->cpu_pid =calloc (guest->vcpu_count,sizeof (int )); +if (!guest->cpu_pid) + goto error; + + buf = line +1 ; + line =strchr (buf,'\n' ); +while (line) { + *line ='\0' ;
lib/trace-cmd/trace-input.c static int trace_traceid_load(struct tracecmd_input *handle, char *buf) · line 2357
Quoted 10 lines
lib/trace-cmd/trace-input.c +10
-0+if (sscanf (buf,"%d %d" , &cpu, &pid) !=2 ) + goto error; + +if (cpu < guest->vcpu_count) + guest->cpu_pid[cpu] = pid; + buf = line +1 ; +if (!*buf) +break ; + + line =strchr (buf,'\n' );
Hmm, strtok_r() seems more appropriate here than strchr(), then you don't need to deal with the *line = '\0'.
Quoted 38 lines
lib/trace-cmd/trace-input.c +20
-0+ } + + guest->next = handle->guest; + handle->guest = guest; +return 0 ; + +error: +if (guest) { +free (guest->cpu_pid); +free (guest->name); +free (guest); + } +return -1 ; +} +#define STR_PROCMAP_LINE_MAX (PATH_MAX+22)static int trace_pid_map_load (struct tracecmd_input *handle,char *buf) {@@ -2558,6 +2634,10 @@ static int handle_options(struct tracecmd_input *handle)if (buf[size-1 ] =='\0' )trace_traceid_load (handle, buf);break ; +case TRACECMD_OPTION_GUEST : +if (buf[size-1 ] =='\0' ) +trace_guest_load (handle, buf); +break ;default :warning ("unknown option %d" , option);break ;@@ -3217,6 +3297,7 @@ void tracecmd_close(struct tracecmd_input *handle) handle->pid_maps =NULL ;trace_tsync_offset_free (&handle->host); +trace_guests_free (handle);if (handle->flags &TRACECMD_FL_BUFFER_INSTANCE )tracecmd_close (handle->parent);
lib/trace-cmd/trace-input.c unsigned long long tracecmd_get_traceid(struct tracecmd_input *handle) · line 3760
Quoted 7 lines
lib/trace-cmd/trace-input.c +4
-0@@ -3673,6 +3754,38 @@ unsigned long long tracecmd_get_traceid(struct tracecmd_input *handle)return handle->trace_id; } +/** +* tracecmd_get_guest_cpumap - get the mapping of guest VCPU to host process +* @handle: input handle for the trace.dat file +* @trace_id: ID of the guest tracing session
@name, @vcpu_count, and @cpu_pid should still be up in the above section.
lib/trace-cmd/trace-input.c unsigned long long tracecmd_get_traceid(struct tracecmd_input *handle) · line 3770
Quoted 10 lines
lib/trace-cmd/trace-input.c +10
-0+* +* Returns @name of the guest, number of VPUs (@vcpu_count) +* and array @cpu_pid with size @vcpu_count. Array index is VCPU id, array +* content is PID of the host process, running this VCPU. +* +* This information is stored in host trace.dat file +*/ +int tracecmd_get_guest_cpumap(struct tracecmd_input *handle, + unsigned long long trace_id, + char *name,
name isn't returned.
lib/trace-cmd/trace-input.c unsigned long long tracecmd_get_traceid(struct tracecmd_input *handle) · line 3783
Quoted 13 lines
lib/trace-cmd/trace-input.c +13
-0+ int *vcpu_count, int *cpu_pid) +{ + struct guest_trace_info *guest = handle->guest; + +while (guest) { +if (guest->trace_id == trace_id) +break ; + guest = guest->next; + } +if (!guest) +return -1 ; + + name = guest->name;
The above doesn't do anything, and will probably be optimized out by the compiler.
-- Steve
Quoted 78 lines
lib/trace-cmd/trace-input.c +5
-0+ *vcpu_count = guest->vcpu_count; + cpu_pid = guest->cpu_pid; +return 0 ; +} +/** * tracecmd_get_tsync_peer - get the trace session id of the peer host * @handle: input handle for the trace.dat file tracecmd/trace-record.c +45
-0@@ -2884,6 +2884,19 @@ struct guest {static struct guest *guests;static size_t guests_len; +static struct guest *get_guest_info (unsigned int guest_cid) +{ +int i; + +if (!guests) +return NULL ; + +for (i =0 ; i < guests_len; i++) +if (guest_cid == guests[i].cid) +return guests + i; +return NULL ; +} +static char *get_qemu_guest_name (char *arg) {char *tok, *end = arg;@@ -3689,6 +3702,33 @@ static void append_buffer(struct tracecmd_output *handle, } } +static void +add_guest_info (struct tracecmd_output *handle,struct buffer_instance *instance) +{ +struct guest *guest =get_guest_info (instance->cid); +struct trace_seq s; +int i; + +if (!guest) +return ; +for (i =0 ; i <VCPUS_MAX ; i++) +if (!guest->cpu_pid[i]) +break ; + +trace_seq_init (&s); + +trace_seq_printf (&s,"Guest %s %llu %d\n" , + guest->name, instance->trace_id, i); +for (i =0 ; i <VCPUS_MAX ; i++) { +if (!guest->cpu_pid[i]) +break ; +trace_seq_printf (&s,"%d %d\n" , i, guest->cpu_pid[i]); + } +trace_seq_terminate (&s); +tracecmd_add_option (handle,TRACECMD_OPTION_GUEST , + s.len +1 , s.buffer); +trace_seq_destroy (&s); +}static void add_pid_maps (struct tracecmd_output *handle,struct buffer_instance *instance)@@ -3976,6 +4016,11 @@ static void record_data(struct common_record_context *ctx)add_pid_maps (handle, instance); } +for_all_instances (instance) { +if (is_guest (instance)) +add_guest_info (handle, instance); + } +tracecmd_append_cpu_data (handle, local_cpu_count, temp_files);for (i =0 ; i < max_cpu_count; i++)
PATCH v17 14/18 trace-cmd: Add host trace clock as guest trace argument
In reply to Tzvetomir Stoyanov (VMware)When tracing host and guest machines, both should use the same tracing clock for event timestamps. If a clock is specified as host tracing argument, with option "-C clock_name", the same is injected as guest tracing argument. If the user wants to use different tracing clocks, it can specify it using "-C clock_name" as guest tracing argument. In that case, the one specified by the user has higher priority.
--- tracecmd/trace-record.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-)
tracecmd/trace-record.c +17 -2
@@ -5405,6 +5405,8 @@ static void parse_record_options(int argc,char *sav;int name_counter =0 ;int neg_event =0 ; +struct buffer_instance *instance; +bool guest_config = false;init_common_record_context (ctx, curr_cmd);
@@ -5562,6 +5564,7 @@ static void parse_record_options(int argc,break ;case 'C' : ctx->instance->ftrace->clock = optarg; + guest_config = true;break ;case 'v' : neg_event =1 ;
@@ -5779,14 +5782,26 @@ static void parse_record_options(int argc,/* If --date is specified, prepend it to all guest VM flags */ if (ctx->date) { -struct buffer_instance *instance; -for_all_instances (instance) {if (is_guest (instance))add_argv (instance,"--date" , true); } } +if (guest_config) { +/* If -C is specified, prepend clock to all guest VM flags */ +for_all_instances (instance) { +if (top_instance.ftrace->clock) { +if (is_guest (instance)) { +add_argv (instance, + (char *)top_instance.ftrace->clock, + true); +add_argv (instance,"-C" , true); + } + } + } + } +if (!ctx->filtered && ctx->instance->filter_mod)add_func (&ctx->instance->filter_funcs, ctx->instance->filter_mod,"*" );
Signature · 1 line
2.23.0
Re: [PATCH v17 14/18] trace-cmd: Add host trace clock as guest trace argument
Quoted 18 lines · On Tue, 3 Dec 2019 12:35:18 +0200 "Tzvetomir Stoyanov (VMware)"…
When tracing host and guest machines, both should use the same tracing clock for event timestamps. If a clock is specified as host tracing argument, with option "-C
clock_name", the same is injected as guest tracing argument. If the user wants to use different tracing clocks, it can specify it using "-Cclock_name" as guest tracing argument. In that case, the one specified by the user has higher priority.Signed-off-by Tzvetomir Stoyanov (VMware) <[email protected]>--- tracecmd/trace-record.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-)tracecmd/trace-record.c +0
-0
tracecmd/trace-record.c static void parse_record_options(int argc, · line 5409
Quoted 5 lines
tracecmd/trace-record.c +2
-0@@ -5405,6 +5405,8 @@ static void parse_record_options(int argc,char *sav;int name_counter =0 ;int neg_event =0 ; +struct buffer_instance *instance; +bool guest_config = false;
Should have a different name, like "clock_set", as "guest_config" doesn't let us know what this is about.
Quoted 18 lines
tracecmd/trace-record.c +1
-2init_common_record_context (ctx, curr_cmd);@@ -5562,6 +5564,7 @@ static void parse_record_options(int argc,break ;case 'C' : ctx->instance->ftrace->clock = optarg; + guest_config = true;break ;case 'v' : neg_event =1 ;@@ -5779,14 +5782,26 @@ static void parse_record_options(int argc,/* If --date is specified, prepend it to all guest VM flags */ if (ctx->date) { -struct buffer_instance *instance; -for_all_instances (instance) {
tracecmd/trace-record.c static void parse_record_options(int argc, · line 5795
Quoted 10 lines
tracecmd/trace-record.c +5
-0if (is_guest (instance))add_argv (instance,"--date" , true); } } +if (guest_config) { +/* If -C is specified, prepend clock to all guest VM flags */ +for_all_instances (instance) { +if (top_instance.ftrace->clock) { +if (is_guest (instance)) {
We should only append this, if the guest didn't have a clock set already. As the change log seems to say, if the user states a "-C clock" for the guest, that should take precedence over the host clock set. That is, a user may specifically state that they are using a different clock. If we have frequency and offset set, it should still work with different clocks.
-- Steve
Quoted 12 lines
tracecmd/trace-record.c +9
-0+add_argv (instance, + (char *)top_instance.ftrace->clock, + true); +add_argv (instance,"-C" , true); + } + } + } + } +if (!ctx->filtered && ctx->instance->filter_mod)add_func (&ctx->instance->filter_funcs, ctx->instance->filter_mod,"*" );
Re: [PATCH v17 14/18] trace-cmd: Add host trace clock as guest trace argument
Quoted 1 line · Steven Rostedt
[ ... ]
Quoted 5 lines
Quoted 5 lines
tracecmd/trace-record.c +0
-0@@ -5405,6 +5405,8 @@ static void parse_record_options(int argc,
Quoted 8 lines
tracecmd/trace-record.c static void parse_record_options(int argc, · line 5409
Quoted 5 lines
tracecmd/trace-record.c +2
-0char *sav;int name_counter =0 ;int neg_event =0 ; +struct buffer_instance *instance; +bool guest_config = false;Should have a different name, like "
clock_set", as "guest_config" doesn't let us know what this is about.
I use the same flag in patch 16 from the same series, with the same purpose - to apply host config to the guest. That's why decided to call it "guest_config".
Quoted 25 lines
Quoted 25 lines
tracecmd/trace-record.c +1
-2init_common_record_context (ctx, curr_cmd);@@ -5562,6 +5564,7 @@ static void parse_record_options(int argc,break ;case 'C' : ctx->instance->ftrace->clock = optarg; + guest_config = true;break ;case 'v' : neg_event =1 ;@@ -5779,14 +5782,26 @@ static void parse_record_options(int argc,/* If --date is specified, prepend it to all guest VM flags */ if (ctx->date) { -struct buffer_instance *instance; -for_all_instances (instance) {if (is_guest (instance))add_argv (instance,"--date" , true); } }
Quoted 12 lines
tracecmd/trace-record.c static void parse_record_options(int argc, · line 5795
Quoted 5 lines
tracecmd/trace-record.c +5
-0+if (guest_config) { +/* If -C is specified, prepend clock to all guest VM flags */ +for_all_instances (instance) { +if (top_instance.ftrace->clock) { +if (is_guest (instance)) {We should only append this, if the guest didn't have a clock set already. As the change log seems to say, if the user states a "-C clock" for the guest, that should take precedence over the host clock set. That is, a user may specifically state that they are using a different clock. If we have frequency and offset set, it should still work with different clocks.
The guest config string is not parsed in the host context, that's why the host doesn't know if a guest has an explicit "-C clock" argument.
I can parse the guest config here, but this will complicate the implementation. Using the current approach still guarantees that the user specified config has higher
priority than injected one - add_argv() API prepends to the beginning of the string, so
user arguments are always after the injected one. When guest parses the string, in case of
duplicated "-C clock" arguments, the last one wins.
Quoted 17 lines
-- Steve
Quoted 12 lines
tracecmd/trace-record.c +9
-0+add_argv (instance, + (char *)top_instance.ftrace->clock, + true); +add_argv (instance,"-C" , true); + } + } + } + } +if (!ctx->filtered && ctx->instance->filter_mod)add_func (&ctx->instance->filter_funcs, ctx->instance->filter_mod,"*" );
Thanks !
Signature · 2 lines
Tzvetomir (Ceco) Stoyanov VMware Open Source Technology Center
Re: [PATCH v17 14/18] trace-cmd: Add host trace clock as guest trace argument
Quoted 13 lines · On Tue, 10 Dec 2019 10:49:43 +0200 Tzvetomir Stoyanov <[email protected]>…
Quoted 6 lines
tracecmd/trace-record.c static void parse_record_options(int argc, · line 5795
Quoted 5 lines
tracecmd/trace-record.c +5
-0+if (guest_config) { +/* If -C is specified, prepend clock to all guest VM flags */ +for_all_instances (instance) { +if (top_instance.ftrace->clock) { +if (is_guest (instance)) {Quoted 6 lines
We should only append this, if the guest didn't have a clock set already. As the change log seems to say, if the user states a "-C clock" for the guest, that should take precedence over the host clock set. That is, a user may specifically state that they are using a different clock. If we have frequency and offset set, it should still work with different clocks.
Quoted 10 lines
The guest config string is not parsed in the host context, that's why the host doesn't know if a guest has an explicit "-C clock" argument.
I can parse the guest config here, but this will complicate the implementation. Using the current approach still guarantees that the user specified config has higher
priority than injected one -add_argv()API prepends to the beginning of the string, so
user arguments are always after the injected one. When guest parses the string, in case of
duplicated "-C clock" arguments, the last one wins.
I'm confused. I'm looking at this:
for (;;) {
[..]
switch (c) {
[..]
case 'A':
[..]
ctx->instance->flags |= BUFFER_FL_GUEST;
[..]
case 'C':
ctx->instance->ftrace->clock = optarg;
guest_config = true;
break;
[..]
}
[..]
}
[..]
if (guest_config) {
/* If -C is specified, prepend clock to all guest VM flags */
for_all_instances(instance) {
if (top_instance.ftrace->clock) {Why can't we have here:
if (top_instance.ftrace->clock && !instance->ftrace->clock)
If the guest instance was given a -C, I would think we don't want to add another -C to pass to that guest?
-- Steve
if (is_guest(instance)) {
add_argv(instance,
(char *)top_instance.ftrace->clock,
true);
add_argv(instance, "-C", true);
}
}
}
}Re: [PATCH v17 14/18] trace-cmd: Add host trace clock as guest trace argument
Quoted 53 lines · Steven Rostedt
Quoted 13 lines · On Tue, 10 Dec 2019 10:49:43 +0200 Tzvetomir Stoyanov <[email protected]>…
Quoted 6 lines
tracecmd/trace-record.c static void parse_record_options(int argc, · line 5795
Quoted 5 lines
tracecmd/trace-record.c +5
-0+if (guest_config) { +/* If -C is specified, prepend clock to all guest VM flags */ +for_all_instances (instance) { +if (top_instance.ftrace->clock) { +if (is_guest (instance)) {Quoted 6 lines
We should only append this, if the guest didn't have a clock set already. As the change log seems to say, if the user states a "-C clock" for the guest, that should take precedence over the host clock set. That is, a user may specifically state that they are using a different clock. If we have frequency and offset set, it should still work with different clocks.
Quoted 10 lines
The guest config string is not parsed in the host context, that's why the host doesn't know if a guest has an explicit "-C clock" argument.
I can parse the guest config here, but this will complicate the implementation. Using the current approach still guarantees that the user specified config has higher
priority than injected one -add_argv()API prepends to the beginning of the string, so
user arguments are always after the injected one. When guest parses the string, in case of
duplicated "-C clock" arguments, the last one wins.I'm confused. I'm looking at this:
for (;;) { [..] switch (c) { [..] case 'A': [..] ctx->instance->flags |= BUFFER_FL_GUEST; [..] case 'C': ctx->instance->ftrace->clock = optarg; guest_config = true; break; [..] } [..] } [..] if (guest_config) { /* If -C is specified, prepend clock to all guest VM flags */ for_all_instances(instance) { if (top_instance.ftrace->clock) {
Quoted 7 lines
Why can't we have here:
if (
top_instance.ftrace->clock&& !instance->ftrace->clock)If the guest instance was given a -C, I would think we don't want to add another -C to pass to that guest?
In case of a guest ("-A" option), the logic skips the switch(), so the guest args are not parsed.
There is a check, right before the switch() :
/*
* If the current instance is to record a guest, then save
* all the arguments for this instance.
*/
if (c != 'B' && c != 'A' && is_guest(ctx->instance)) {
add_arg(ctx->instance, c, opts, long_options, optarg);
continue;
}I can put inside that if() a check for "-C" guest argument, but it will look like a hack.
The confusion is that guest_config is set to true for any "-C" host argument, including those for instances, but only the one from the top instance is used to inject
guest clock arg.
Quoted 11 lines
-- Steve
if (is_guest(instance)) { add_argv(instance, (char *)top_instance.ftrace->clock, true); add_argv(instance, "-C", true); } } } }
Signature · 2 lines
Tzvetomir (Ceco) Stoyanov VMware Open Source Technology Center
Re: [PATCH v17 14/18] trace-cmd: Add host trace clock as guest trace argument
Quoted 3 lines · On Wed, 11 Dec 2019 10:21:44 +0200 Tzvetomir Stoyanov <[email protected]>…
In case of a guest ("-A" option), the logic skips the
switch(), so the guest args are not parsed.
There is a check, right before theswitch():
Ah, thanks, I forgot about that.
Quoted 10 lines
/*
* If the current instance is to record a guest, then save
* all the arguments for this instance.
*/
if (c != 'B' && c != 'A' && is_guest(ctx->instance)) {
add_arg(ctx->instance, c, opts, long_options, optarg);
continue;
}Quoted 2 lines
I can put inside that
if()a check for "-C" guest argument, but it will look like a hack.
I disagree. It's no more of a "hack" than appending a "-C" to the arguments. I think it's the right solution. We can simply add:
if (c == 'C') ctx->instance->flags |= BUFFER_FL_HAS_CLOCK;
Then we could test if that flag is set for the instance below.
-- Steve
Quoted 21 lines
The confusion is that
guest_configis set to true for any "-C" host argument, including those for instances, but only the one from the top instance is used to inject
guest clock arg.Quoted 11 lines
-- Steve
if (is_guest(instance)) { add_argv(instance, (char *)top_instance.ftrace->clock, true); add_argv(instance, "-C", true); } } } }
PATCH v17 15/18 trace-cmd: Refactor few trace-cmd internal functions.
In reply to Tzvetomir Stoyanov (VMware)In order to reuse code inside trace-cmd application context, few functions are made non static:
int make_vsock(unsigned int port); int get_vsock_port(int sd, unsigned int *port); int open_vsock(unsigned int cid, unsigned int port); char *get_guest_file(const char *file, const char *guest);
--- tracecmd/include/trace-local.h | 8 ++++++++ tracecmd/trace-agent.c | 4 ++-- tracecmd/trace-record.c | 8 ++++---- 3 files changed, 14 insertions(+), 6 deletions(-)
tracecmd/include/trace-local.h +8 -0
@@ -19,6 +19,8 @@#define GUEST_FIFO_FMT GUEST_DIR_FMT "/" GUEST_PIPE_NAME "%d"#define VIRTIO_FIFO_FMT "/dev/virtio-ports/" GUEST_PIPE_NAME "%d" +#define TRACE_FILENAME "trace.dat" +/* fix stupid glib guint64 typecasts and printf formats */ typedef unsigned long long u64 ;
@@ -258,6 +260,12 @@ void tracecmd_disable_tracing(void);void tracecmd_enable_tracing (void );void tracecmd_stat_cpu (struct trace_seq *s,int cpu); +int make_vsock (unsigned int port); +int get_vsock_port (int sd,unsigned int *port); +int open_vsock (unsigned int cid,unsigned int port); + +char *get_guest_file (const char *file,const char *guest); +/* No longer in event-utils.h */ void __noreturn die (const char *fmt, ...);/* Can be overriden */ void *malloc_or_die (unsigned int size);/* Can be overridden */
tracecmd/trace-agent.c +2 -2
@@ -40,7 +40,7 @@ static int get_local_cid(unsigned int *cid)return ret; } -static int make_vsock (unsigned int port) +int make_vsock (unsigned int port) {struct sockaddr_vm addr = { .svm_family =AF_VSOCK ,
@@ -64,7 +64,7 @@ static int make_vsock(unsigned int port)return sd; } -static int get_vsock_port (int sd,unsigned int *port) +int get_vsock_port (int sd,unsigned int *port) {struct sockaddr_vm addr;socklen_t addr_len =sizeof (addr);
tracecmd/trace-record.c +4 -4
@@ -68,7 +68,7 @@ static int rt_prio;static int keep; -static const char *output_file ="trace.dat" ; +static const char *output_file =TRACE_FILENAME ;static int latency;static int sleep_time =1000 ;
@@ -536,7 +536,7 @@ static char *get_temp_file(struct buffer_instance *instance, int cpu)return file; } -static char *get_guest_file (const char *file,const char *guest) +char *get_guest_file (const char *file,const char *guest) {const char *p;char *out =NULL ;
@@ -2772,7 +2772,7 @@ static int connect_port(const char *host, unsigned int port) } #ifdef VSOCK -static int open_vsock(unsigned int cid, unsigned int port) +int open_vsock(unsigned int cid, unsigned int port) { struct sockaddr_vm addr = { .svm_family = AF_VSOCK,
@@ -2835,7 +2835,7 @@ static bool can_splice_read_vsock(void) }#else -static inline int open_vsock (unsigned int cid,unsigned int port) +int open_vsock (unsigned int cid,unsigned int port) {die ("vsock is not supported" );return -1 ;
Signature · 1 line
2.23.0
Re: [PATCH v17 15/18] trace-cmd: Refactor few trace-cmd internal functions.
Quoted 6 lines · On Tue, 3 Dec 2019 12:35:19 +0200 "Tzvetomir Stoyanov (VMware)"…
In order to reuse code inside trace-cmd application context, few functions are made non static:
int make_vsock(unsigned int port); int get_vsock_port(int sd, unsigned int *port); int open_vsock(unsigned int cid, unsigned int port); char *get_guest_file(const char *file, const char *guest);
We should probably rename them to have a "trace_" prefix. This way we know what functions are not static, but "trace_" means it's local, where as "tracecmd_" means its exported.
-- Steve
Quoted 3 lines
Signed-off-by Tzvetomir Stoyanov (VMware) <[email protected]>---
PATCH v17 16/18 trace-cmd: Basic infrastructure for host - guest timestamp synchronization
In reply to Tzvetomir Stoyanov (VMware)The infrastructure for host - guest timestamp synchronization is divided in two parts:
- logic in libtracecmd
- logic in trace-cmd application
The libtracecmd is extended with new trace message, MSG_TIME_SYNC, used to exchange time synch information between host and guest. The trace request and response messages are extended to handle the time synch negotiation. The logic is implemented in trace-timesync.c file:
- Register / unregister time sync protocols.
- Choosing what protocol will be used for a tracing session.
- Init common context, used by all protocols - a ftrace instance, vsockets.
- Invoke protocol routines to perform time synchronization.
- Store calculated offsets in an array.
The trace-cmd application is extended to perform timestamp synchronization.
The main logic is in trace-tsync.c file
- Negotiate what time synch algorithm will be used for the tracing session.
- Run pthreads, to do continuous timestamp synchronization, during the trace.
- Store calculated offsets in guest's trace.dat file, using TRACECMD_OPTION_TIME_SHIFT option
- A new trace-cmd option is added, to control the timestamp synchronization:
--tsync-interval: set the time sync loop interval, in ms. The default
value is 0 - timestamp synchronization is performed twice, before and after the trace. If a negative number is specified, the timestamp synchronization is disabled.
--- Documentation/trace-cmd-record.1.txt | 6 + include/trace-cmd/trace-cmd.h | 54 ++- lib/trace-cmd/Makefile | 1 + lib/trace-cmd/include/trace-tsync-local.h | 36 ++ lib/trace-cmd/trace-msg.c | 147 ++++++- lib/trace-cmd/trace-timesync.c | 461 ++++++++++++++++++++++ tracecmd/Makefile | 3 +- tracecmd/include/trace-local.h | 11 + tracecmd/trace-agent.c | 34 +- tracecmd/trace-record.c | 37 +- tracecmd/trace-tsync.c | 268 +++++++++++++ tracecmd/trace-usage.c | 4 + 12 files changed, 1035 insertions(+), 27 deletions(-) create mode 100644 lib/trace-cmd/include/trace-tsync-local.h create mode 100644 lib/trace-cmd/trace-timesync.c create mode 100644 tracecmd/trace-tsync.c
Documentation/trace-cmd-record.1.txt +6 -0
@@ -338,6 +338,12 @@ OPTIONS the offset will also be in nanoseconds even if the displayed units are in microseconds. +*--tsync-interval*:: + Set the loop interval, in ms, for timestamps synchronization with guests: + If a negative number is specified, timestamps synchronization is disabled + If 0 is specified, no loop is performed - timestamps offset is calculated only twice," + at the beginning and at the end of the trace\n" + *--stderr*:: Have output go to stderr instead of stdout, but the output of the command executed will not be changed. This is useful if you want to monitor the
include/trace-cmd/trace-cmd.h +50 -4
81 lines not on this page · raw message
lib/trace-cmd/Makefile +1 -0
@@ -16,6 +16,7 @@ OBJS += trace-util.o OBJS += trace-instance.o OBJS += trace-filter-hash.o OBJS += trace-msg.o +OBJS += trace-timesync.o # Additional util objects OBJS += trace-blk-hack.o
lib/trace-cmd/include/trace-tsync-local.h new file +36 -0
@@ -0,0 +1,36 @@ +/* SPDX-License-Identifier: LGPL-2.1 */ +/* +* Copyright (C) 2019, VMware, Tzvetomir Stoyanov <[email protected]> +* +*/ +#ifndef _TRACE_TSYNC_LOCAL_H +#define _TRACE_TSYNC_LOCAL_H + +#include <stdbool.h> + +struct clock_sync_context { +void *proto_data; +bool server; +struct tracecmd_instance *vinst; + +int sync_size; +int sync_count; +long long *sync_ts; +long long *sync_offsets; + +unsigned int local_cid; +unsigned int local_port; +unsigned int remote_cid; +unsigned int remote_port; +}; + +int tracecmd_tsync_proto_register (unsigned int proto_id,int weight, +int (*init)(struct tracecmd_time_sync *), +int (*free)(struct tracecmd_time_sync *), +int (*calc)(struct tracecmd_time_sync *, +long long *,long long *)); +int tracecmd_tsync_proto_unregister (unsigned int proto_id); + +int ptp_clock_sync_register (void ); + +#endif /* _TRACE_TSYNC_LOCAL_H */
lib/trace-cmd/trace-msg.c +131 -16
259 lines not on this page · raw message
lib/trace-cmd/trace-timesync.c new file +461 -0
461 lines not on this page · raw message
tracecmd/Makefile +2 -1
@@ -30,6 +30,7 @@ TRACE_CMD_OBJS += trace-check-events.o TRACE_CMD_OBJS += trace-show.o TRACE_CMD_OBJS += trace-list.o TRACE_CMD_OBJS += trace-usage.o +TRACE_CMD_OBJS += trace-tsync.o ifeq ($(VSOCK_DEFINED), 1) TRACE_CMD_OBJS += trace-agent.o
@@ -42,7 +43,7 @@ all_objs := $(sort $(ALL_OBJS)) all_deps := $(all_objs:$(bdir)/%.o=$(bdir)/.%.d) CONFIG_INCLUDES = -CONFIG_LIBS = -lrt +CONFIG_LIBS = -lrt -lpthread CONFIG_FLAGS = all: $(TARGETS)
tracecmd/include/trace-local.h +11 -0
@@ -226,6 +226,10 @@ struct buffer_instance {unsigned int port;int *fds;bool use_fifos; + +pthread_t tsync_thread; +bool tsync_thread_running; +struct tracecmd_time_sync tsync; };void init_top_instance (void );
@@ -260,6 +264,13 @@ void tracecmd_disable_tracing(void);void tracecmd_enable_tracing (void );void tracecmd_stat_cpu (struct trace_seq *s,int cpu); +int tracecmd_host_tsync (struct buffer_instance *instance, +unsigned int tsync_port); +void tracecmd_host_tsync_complete (struct buffer_instance *instance); +unsigned int tracecmd_guest_tsync (unsigned int tsync_protos,char *clock, +unsigned int *tsync_port, +pthread_t *thr_id); +int make_vsock (unsigned int port);int get_vsock_port (int sd,unsigned int *port);int open_vsock (unsigned int cid,unsigned int port);
tracecmd/trace-agent.c +32 -2
@@ -19,6 +19,7 @@#include <sys/wait.h> #include <unistd.h> #include <linux/vm_sockets.h> +#include <pthread.h> #include "trace-local.h" #include "trace-msg.h"
@@ -125,11 +126,28 @@ cleanup:return ret; } +static char *get_clock (int argc,char **argv) +{ +int i; + +if (!argc || !argv) +return NULL ; + +for (i =0 ; i < argc -1 ; i++) { +if (!strcmp ("-C" , argv[i])) +return argv[i+1 ]; + } +return NULL ; +} +static void agent_handle (int sd,int nr_cpus,int page_size) {struct tracecmd_msg_handle *msg_handle;unsigned long long trace_id; +unsigned int tsync_protos =0 ; +unsigned int tsync_port =0 ;unsigned int *ports; +pthread_t sync_thr;char **argv =NULL ;int argc =0 ;bool use_fifos;
@@ -146,7 +164,8 @@ static void agent_handle(int sd, int nr_cpus, int page_size)die ("Failed to allocate message handle" ); ret =tracecmd_msg_recv_trace_req (msg_handle, &argc, &argv, - &use_fifos, &trace_id); + &use_fifos, &trace_id, + &tsync_protos);if (ret <0 )die ("Failed to receive trace request" );
@@ -155,14 +174,25 @@ static void agent_handle(int sd, int nr_cpus, int page_size)if (!use_fifos)make_vsocks (nr_cpus, fds, ports); +if (tsync_protos) { + tsync_protos =tracecmd_guest_tsync (tsync_protos, +get_clock (argc, argv), + &tsync_port, &sync_thr); +if (!tsync_protos) +warning ("Failed to negotiate timestamps synchronization with the host" ); + } trace_id =tracecmd_generate_traceid (); ret =tracecmd_msg_send_trace_resp (msg_handle, nr_cpus, page_size, - ports, use_fifos, trace_id); + ports, use_fifos, trace_id, + tsync_protos, tsync_port);if (ret <0 )die ("Failed to send trace response" );trace_record_agent (msg_handle, nr_cpus, fds, argc, argv, use_fifos, trace_id); +if (tsync_protos) +pthread_join (sync_thr,NULL ); +free (argv[0 ]);free (argv);free (ports);
tracecmd/trace-record.c +33 -4
@@ -670,6 +670,11 @@ static void tell_guests_to_stop(void)tracecmd_msg_send_close_msg (instance->msg_handle); } +for_all_instances (instance) { +if (is_guest (instance)) +tracecmd_host_tsync_complete (instance); + } +/* Wait for guests to acknowledge */ for_all_instances (instance) {if (is_guest (instance)) {
@@ -3501,6 +3506,8 @@ static void connect_to_agent(struct buffer_instance *instance) {struct tracecmd_msg_handle *msg_handle;int sd, ret, nr_fifos, nr_cpus, page_size; +unsigned int tsync_protos_reply =0 ; +unsigned int tsync_port =0 ;unsigned int *ports;int i, *fds =NULL ;bool use_fifos = false;
@@ -3519,19 +3526,32 @@ static void connect_to_agent(struct buffer_instance *instance)if (!msg_handle)die ("Failed to allocate message handle" ); +if (instance->tsync.loop_interval >=0 ) + instance->tsync.sync_proto =tracecmd_tsync_proto_getall (); + ret =tracecmd_msg_send_trace_req (msg_handle, instance->argc, instance->argv, use_fifos, - top_instance.trace_id); + top_instance.trace_id, + instance->tsync.sync_proto);if (ret <0 )die ("Failed to send trace request" ); ret =tracecmd_msg_recv_trace_resp (msg_handle, &nr_cpus, &page_size, &ports, &use_fifos, - &instance->trace_id); + &instance->trace_id, + &tsync_protos_reply, &tsync_port);if (ret <0 ) -die ("Failed to receive trace response" ); +die ("Failed to receive trace response %d" , ret); + +if (instance->tsync.sync_proto) { +if (instance->tsync.sync_proto & tsync_protos_reply) { + instance->tsync.sync_proto = tsync_protos_reply; +tracecmd_host_tsync (instance, tsync_port); + }else +warning ("Failed to negotiate timestamps synchronization with the guest" ); + }if (use_fifos) {if (nr_cpus != nr_fifos) {
@@ -3578,7 +3598,9 @@ static void setup_guest(struct buffer_instance *instance)close (fd); } -static void setup_agent (struct buffer_instance *instance,struct common_record_context *ctx) +static void setup_agent (struct buffer_instance *instance, +struct common_record_context *ctx) + {struct tracecmd_output *network_handle;
@@ -5131,6 +5153,7 @@ void init_top_instance(void) }enum { + OPT_tsyncinterval =242 , OPT_user =243 , OPT_procmap =244 , OPT_quiet =245 ,
@@ -5433,6 +5456,7 @@ static void parse_record_options(int argc, {"proc-map", no_argument, NULL, OPT_procmap}, {"user", required_argument, NULL, OPT_user}, {"module", required_argument, NULL, OPT_module}, + {"tsync-interval", required_argument, NULL, OPT_tsyncinterval}, {NULL, 0, NULL, 0} };
@@ -5771,6 +5795,10 @@ static void parse_record_options(int argc, ctx->instance->filter_mod = optarg; ctx->filtered =0 ;break ; +case OPT_tsyncinterval: + top_instance.tsync.loop_interval =atoi (optarg); + guest_config = true; +break ;case OPT_quiet:case 'q' : quiet = true;
@@ -5799,6 +5827,7 @@ static void parse_record_options(int argc,add_argv (instance,"-C" , true); } } + instance->tsync.loop_interval = top_instance.tsync.loop_interval; } }
tracecmd/trace-tsync.c new file +268 -0
268 lines not on this page · raw message
tracecmd/trace-usage.c +4 -0
@@ -60,6 +60,10 @@ static struct usage_help usage_help[] = {" --no-filter include trace-cmd threads in the trace\n" " --proc-map save the traced processes address map into the trace.dat file\n" " --user execute the specified [command ...] as given user\n" +" --tsync-interval set the loop interval, in ms, for timestamps synchronization with guests:" +" If a negative number is specified, timestamps synchronization is disabled" +" If 0 is specified, no loop is performed - timestamps offset is calculated only twice," +" at the beginnig and at the end of the trace\n" }, { "start",
Signature · 1 line
2.23.0
Re: [PATCH v17 16/18] trace-cmd: Basic infrastructure for host - guest timestamp synchronization
include/trace-cmd/trace-cmd.h int tracecmd_iterate_raw_events(struct tep_handle *tep, · line 441
Quoted 4 lines · On Tue, 3 Dec 2019 12:35:20 +0200 "Tzvetomir Stoyanov (VMware)"…
include/trace-cmd/trace-cmd.h +4
-0+struct tracecmd_time_sync { +unsigned int sync_proto; +int loop_interval; +sem_t sem;
Ug, semaphores are a horrible construct. I believe you can do the same thing here with pthread_cond variables. I attached a small program that uses pthread_cond and pthread_mutex to do interval times like this.
Note, another thing is, you can just have a global "end" variable that can be set and checked (as I did in my program here).
-- Steve
include/trace-cmd/trace-cmd.h int tracecmd_iterate_raw_events(struct tep_handle *tep, · line 446
Quoted 5 lines
include/trace-cmd/trace-cmd.h +5
-0+char *clock_str; +struct tracecmd_msg_handle *msg_handle; +void *context; +}; +
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <stdarg.h>
#include <errno.h>
#include <stdbool.h>
#include <pthread.h>
#define gettid() syscall(__NR_gettid)
static bool end;
static void __vdie(const char *fmt, va_list ap, int err)
{
int ret = errno;
if (err && errno)
perror("bmp-read");
else
ret = -1;
fprintf(stderr, " ");
vfprintf(stderr, fmt, ap);
fprintf(stderr, "\n");
exit(ret);
}void pdie(const char *fmt, ...)
{
va_list ap;
va_start(ap, fmt);
__vdie(fmt, ap, 1);
va_end(ap);
}#define NR_THREADS 8
static int nr_threads = NR_THREADS;
static pthread_t *threads;
static pthread_barrier_t threads_started;
struct t_info {
int id;
int interval;
pthread_mutex_t lock;
pthread_cond_t cond;
};static void *thread_func(void *arg)
{
struct t_info *t = arg;
pthread_mutex_t *this_mutex = &t->lock;
pthread_cond_t *this_cond = &t->cond;
struct timespec ts;
int cnt = 0;
pthread_barrier_wait(&threads_started);
while (true) {
pthread_mutex_lock(this_mutex);
printf("task %d running %d\n", t->id, ++cnt);
if (end)
break;
clock_gettime(CLOCK_REALTIME, &ts);
ts.tv_sec += t->id + 1;
pthread_cond_timedwait(this_cond, this_mutex, &ts);
pthread_mutex_unlock(this_mutex);
}
pthread_mutex_unlock(this_mutex);
printf("Task %d finished\n", t->id);
return NULL;
}int main (int argc, char **argv)
{
struct t_info *infos;
int ret;
int i;
infos = calloc(nr_threads, sizeof(*infos));
if (!infos)
pdie("calloc");
threads = calloc(nr_threads, sizeof(*threads));
if (!threads)
pdie("calloc");
pthread_barrier_init(&threads_started, NULL, nr_threads + 1);
for (i = 0; i < nr_threads; i++) {
infos[i].id = i;
if (pthread_mutex_init(&infos[i].lock, NULL))
pdie("pthread_mutex_init");
ret = pthread_create(&threads[i], NULL, thread_func, &infos[i]);
if (ret < 0)
pdie("creating thread %d", i);
}
pthread_barrier_wait(&threads_started);
sleep(30);
end = true;
for (i = 0; i < nr_threads; i++) {
pthread_mutex_lock(&infos[i].lock);
pthread_cond_signal(&infos[i].cond);
pthread_mutex_unlock(&infos[i].lock);
pthread_join(threads[i], NULL);
}
return 0;
}Re: [PATCH v17 16/18] trace-cmd: Basic infrastructure for host - guest timestamp synchronization
In reply to Tzvetomir Stoyanov (VMware)Quoted 50 lines · On Tue, 3 Dec 2019 12:35:20 +0200 "Tzvetomir Stoyanov (VMware)"…
The infrastructure for host - guest timestamp synchronization is divided in two parts:
- logic in libtracecmd
- logic in trace-cmd applicationThe libtracecmd is extended with new trace message,
MSG_TIME_SYNC, used to exchange time synch information between host and guest. The trace request and response messages are extended to handle the time synch negotiation. The logic is implemented intrace-timesync.cfile:
- Register / unregister time sync protocols.
- Choosing what protocol will be used for a tracing session.
- Init common context, used by all protocols - a ftrace instance, vsockets.
- Invoke protocol routines to perform time synchronization.
- Store calculated offsets in an array.The trace-cmd application is extended to perform timestamp synchronization.
The main logic is intrace-tsync.cfile
- Negotiate what time synch algorithm will be used for the tracing session.
- Run pthreads, to do continuous timestamp synchronization, during the trace.
- Store calculated offsets in guest's trace.dat file, usingTRACECMD_OPTION_TIME_SHIFToption
- A new trace-cmd option is added, to control the timestamp synchronization:
--tsync-interval: set the time sync loop interval, in ms. The default
value is 0 - timestamp synchronization is performed twice, before and after the trace. If a negative number is specified, the timestamp synchronization is disabled.Signed-off-by Tzvetomir Stoyanov (VMware) <[email protected]>--- Documentation/trace-cmd-record.1.txt | 6 + include/trace-cmd/trace-cmd.h | 54 ++- lib/trace-cmd/Makefile | 1 + lib/trace-cmd/include/trace-tsync-local.h | 36 ++ lib/trace-cmd/trace-msg.c | 147 ++++++- lib/trace-cmd/trace-timesync.c | 461 ++++++++++++++++++++++ tracecmd/Makefile | 3 +- tracecmd/include/trace-local.h | 11 + tracecmd/trace-agent.c | 34 +- tracecmd/trace-record.c | 37 +- tracecmd/trace-tsync.c | 268 +++++++++++++ tracecmd/trace-usage.c | 4 + 12 files changed, 1035 insertions(+), 27 deletions(-) create mode 100644 lib/trace-cmd/include/trace-tsync-local.h create mode 100644 lib/trace-cmd/trace-timesync.c create mode 100644 tracecmd/trace-tsync.cDocumentation/trace-cmd-record.1.txt +0
-0
Documentation/trace-cmd-record.1.txt OPTIONS · line 345
Quoted 8 lines
Documentation/trace-cmd-record.1.txt +5
-0@@ -338,6 +338,12 @@ OPTIONS the offset will also be in nanoseconds even if the displayed units are in microseconds. +*--tsync-interval*:: + Set the loop interval, in ms, for timestamps synchronization with guests: + If a negative number is specified, timestamps synchronization is disabled + If 0 is specified, no loop is performed - timestamps offset is calculated only twice," + at the beginning and at the end of the trace\n"
We should probably state that it currently only supports VSOCK
Quoted 22 lines
Documentation/trace-cmd-record.1.txt +1
-0+ *--stderr*:: Have output go to stderr instead of stdout, but the output of the command executed will not be changed. This is useful if you want to monitor theinclude/trace-cmd/trace-cmd.h +6
-0/* --- ftrace instances --- */ struct tracecmd_instance {@@ -406,6 +423,35 @@ int tracecmd_iterate_raw_events(struct tep_handle *tep, void *callback_context); +/* --- Timestamp synchronization --- */ + +enum { +TRACECMD_TIME_SYNC_PROTO_NONE =0 , +}; +enum {
include/trace-cmd/trace-cmd.h int tracecmd_iterate_raw_events(struct tep_handle *tep, · line 441
Quoted 10 lines
include/trace-cmd/trace-cmd.h +10
-0+TRACECMD_TIME_SYNC_CMD_PROBE =1 , + TRACECMD_TIME_SYNC_CMD_STOP = 2, +}; + +#define TRACECMD_TIME_SYNC_PROTO_PTP_WEIGHT 10 + +struct tracecmd_time_sync { +unsigned int sync_proto; +int loop_interval; +sem_t sem;
I already mentioned what I think about semaphores.
Quoted 34 lines
include/trace-cmd/trace-cmd.h +13
-0+char *clock_str; +struct tracecmd_msg_handle *msg_handle; +void *context; +}; + +unsigned int tracecmd_tsync_proto_getall (void ); +unsigned int tracecmd_tsync_proto_select (unsigned int protos); +void tracecmd_tsync_with_host (struct tracecmd_time_sync *tsync); +void tracecmd_tsync_with_guest (struct tracecmd_time_sync *tsync); +int tracecmd_tsync_get_offsets (struct tracecmd_time_sync *tsync, +int *count, +long long **ts,long long **offsets); +void tracecmd_tsync_free (struct tracecmd_time_sync *tsync);/* --- Plugin handling --- */ extern struct tep_plugin_option trace_ftrace_options[];lib/trace-cmd/Makefile +1
-0@@ -16,6 +16,7 @@ OBJS += trace-util.o OBJS += trace-instance.o OBJS += trace-filter-hash.o OBJS += trace-msg.o +OBJS += trace-timesync.o # Additional util objects OBJS += trace-blk-hack.olib/trace-cmd/include/trace-tsync-local.h new file +0
-0
lib/trace-cmd/include/trace-tsync-local.h line 10
Quoted 10 lines
lib/trace-cmd/include/trace-tsync-local.h new file +10
-0@@ -0,0 +1,36 @@ +/* SPDX-License-Identifier: LGPL-2.1 */ +/* +* Copyright (C) 2019, VMware, Tzvetomir Stoyanov <[email protected]> +* +*/ +#ifndef _TRACE_TSYNC_LOCAL_H +#define _TRACE_TSYNC_LOCAL_H + +#include <stdbool.h> +
We should have a comment here that explains what the fields of the structure are.
Quoted 80 lines
lib/trace-cmd/include/trace-tsync-local.h new file +26
-0+struct clock_sync_context { +void *proto_data; +bool server; +struct tracecmd_instance *vinst; + +int sync_size; +int sync_count; +long long *sync_ts; +long long *sync_offsets; + +unsigned int local_cid; +unsigned int local_port; +unsigned int remote_cid; +unsigned int remote_port; +}; + +int tracecmd_tsync_proto_register (unsigned int proto_id,int weight, +int (*init)(struct tracecmd_time_sync *), +int (*free)(struct tracecmd_time_sync *), +int (*calc)(struct tracecmd_time_sync *, +long long *,long long *)); +int tracecmd_tsync_proto_unregister (unsigned int proto_id); + +int ptp_clock_sync_register (void ); + +#endif /* _TRACE_TSYNC_LOCAL_H */ lib/trace-cmd/trace-msg.c +13
-1@@ -28,6 +28,8 @@typedef __u32 u32 ;typedef __be32 be32 ; +typedef __u64 u64 ; +typedef __s64 s64 ;static inline void dprint (const char *fmt, ...) {@@ -64,6 +66,7 @@ struct tracecmd_msg_trace_req {be32 flags;be32 argc;u64 trace_id; +be32 tsync_protos; } __attribute__((packed));struct tracecmd_msg_trace_resp {@@ -71,6 +74,13 @@ struct tracecmd_msg_trace_resp {be32 cpus;be32 page_size;u64 trace_id; +be32 tsync_proto; +be32 tsync_port; +} __attribute__((packed)); + +struct tracecmd_msg_tsync { +be32 sync_protocol; +be32 sync_msg_id; } __attribute__((packed));struct tracecmd_msg_header {@@ -88,7 +98,8 @@ struct tracecmd_msg_header {C (NOT_SUPP ,5 ,0 ), \C (TRACE_REQ ,6 ,sizeof (struct tracecmd_msg_trace_req )), \C (TRACE_RESP ,7 ,sizeof (struct tracecmd_msg_trace_resp )),\ -C (CLOSE_RESP ,8 ,0 ), +C (CLOSE_RESP ,8 ,0 ), \ +C (TIME_SYNC ,9 ,sizeof (struct tracecmd_msg_tsync )), #undef C #define C(a,b,c) MSG_##a = b@@ -122,6 +133,7 @@ struct tracecmd_msg {struct tracecmd_msg_rinit rinit;struct tracecmd_msg_trace_req trace_req;struct tracecmd_msg_trace_resp trace_resp; +struct tracecmd_msg_tsync tsync; };char *buf; } __attribute__((packed));
lib/trace-cmd/trace-msg.c static int tracecmd_msg_send(int fd, struct tracecmd_msg *msg) · line 279
Quoted 4 lines
lib/trace-cmd/trace-msg.c +1
-0@@ -264,6 +276,17 @@ static int tracecmd_msg_send(int fd, struct tracecmd_msg *msg)return ret; } +static int tracecmd_msg_send_nofree (int fd,struct tracecmd_msg *msg)
We probably shouldn't have a static function that starts with "tracecmd_".
Quoted 24 lines
lib/trace-cmd/trace-msg.c +12
-1+{ +int ret =0 ; + + ret =msg_write (fd, msg); +if (ret <0 ) + ret = -ECOMM ; + +return ret; +} +static int msg_read (int fd,void *buf,u32 size,int *n) {ssize_t r;@@ -814,7 +837,8 @@ int tracecmd_msg_wait_close_resp(struct tracecmd_msg_handle *msg_handle) }static int make_trace_req (struct tracecmd_msg *msg,int argc,char **argv, -bool use_fifos,unsigned long long trace_id) +bool use_fifos,unsigned long long trace_id, +unsigned int tsync_protos) {size_t args_size =0 ;char *p;
lib/trace-cmd/trace-msg.c static int make_trace_req(struct tracecmd_msg *msg, int argc, char **argv, · line 853
Quoted 7 lines
lib/trace-cmd/trace-msg.c +3
-1@@ -824,7 +848,11 @@ static int make_trace_req(struct tracecmd_msg *msg, int argc, char **argv, args_size +=strlen (argv[i]) +1 ; msg->hdr.size =htonl (ntohl (msg->hdr.size) + args_size); - msg->trace_req.flags = use_fifos ?htonl (MSG_TRACE_USE_FIFOS ) :htonl (0 ); + msg->trace_req.flags =0 ; +if (use_fifos) + msg->trace_req.flags |=MSG_TRACE_USE_FIFOS ;
Why the change to flags here?
Quoted 53 lines
lib/trace-cmd/trace-msg.c +13
-8+ msg->trace_req.flags =htonl (msg->trace_req.flags); + msg->trace_req.tsync_protos =htonl (tsync_protos); msg->trace_req.argc =htonl (argc); msg->trace_req.trace_id =htonll (trace_id); msg->buf =calloc (args_size,1 );@@ -840,28 +868,30 @@ static int make_trace_req(struct tracecmd_msg *msg, int argc, char **argv,int tracecmd_msg_send_trace_req (struct tracecmd_msg_handle *msg_handle,int argc,char **argv,bool use_fifos, -unsigned long long trace_id) +unsigned long long trace_id, +unsigned int tsync_protos) {struct tracecmd_msg msg;int ret;tracecmd_msg_init (MSG_TRACE_REQ , &msg); ret =make_trace_req (&msg, argc, argv, - use_fifos, trace_id); + use_fifos, trace_id, tsync_protos);if (ret <0 )return ret;return tracecmd_msg_send (msg_handle->fd, &msg); } -/* -* NOTE: On success, the returned `argv` should be freed with: -* free(argv[0]); -* free(argv); -*/ +/* +* NOTE: On success, the returned `argv` should be freed with: +* free(argv[0]); +* free(argv); +*/ int tracecmd_msg_recv_trace_req (struct tracecmd_msg_handle *msg_handle,int *argc,char ***argv,bool *use_fifos, -unsigned long long *trace_id) +unsigned long long *trace_id, +unsigned int *tsync_protos) {struct tracecmd_msg msg;char *p, *buf_end, **args;@@ -908,6 +938,7 @@ int tracecmd_msg_recv_trace_req(struct tracecmd_msg_handle *msg_handle, *argc = nr_args; *argv = args; *use_fifos =ntohl (msg.trace_req.flags) &MSG_TRACE_USE_FIFOS ; + *tsync_protos =ntohl (msg.trace_req.tsync_protos); *trace_id =ntohll (msg.trace_req.trace_id);/*
lib/trace-cmd/trace-msg.c out: · line 961
Quoted 3 lines
lib/trace-cmd/trace-msg.c +0
-0@@ -928,9 +959,81 @@ out:return ret; }
Should add a kerneldoc comment that describes this function. If for anything else, to describe the parameters.
Quoted 6 lines
lib/trace-cmd/trace-msg.c +6
-0+int tracecmd_msg_send_time_sync (struct tracecmd_msg_handle *msg_handle, +unsigned int sync_protocol, +unsigned int sync_msg_id, +unsigned int payload_size,char *payload) +{ +struct tracecmd_msg msg;
lib/trace-cmd/trace-msg.c out: · line 977
Quoted 10 lines
lib/trace-cmd/trace-msg.c +10
-0+ +tracecmd_msg_init (MSG_TIME_SYNC , &msg); + msg.tsync.sync_protocol =htonl (sync_protocol); + msg.tsync.sync_msg_id =htonl (sync_msg_id); + msg.hdr.size =htonl (ntohl (msg.hdr.size) + payload_size); + + msg.buf = payload; +return tracecmd_msg_send_nofree (msg_handle->fd, &msg); +} +
Should add a kerneldoc comment for this function too.
Quoted 23 lines
lib/trace-cmd/trace-msg.c +23
-0+int tracecmd_msg_recv_time_sync (struct tracecmd_msg_handle *msg_handle, +unsigned int *sync_protocol, +unsigned int *sync_msg_id, +unsigned int *payload_size,char **payload) +{ +struct tracecmd_msg msg; +int ret =-1 ; +int buf_size; + +memset (&msg,0 ,sizeof (msg)); + ret =tracecmd_msg_recv (msg_handle->fd, &msg); +if (ret <0 ) + goto out; + +if (ntohl (msg.hdr.cmd) !=MSG_TIME_SYNC ) { + ret = -EOPNOTSUPP ; + goto out; + } + +if (sync_protocol) + *sync_protocol =ntohl (msg.tsync.sync_protocol); +if (sync_msg_id) + *sync_msg_id =ntohl (msg.tsync.sync_msg_id);
lib/trace-cmd/trace-msg.c out: · line 1010
Quoted 10 lines
lib/trace-cmd/trace-msg.c +10
-0+ + buf_size =msg_buf_len (&msg); +if (buf_size <0 ) { + ret = -EINVAL ; + goto out; + } + +if (buf_size && payload && payload_size) { +if (*payload_size) { +if (*payload_size != buf_size || *payload ==NULL ) {
This patch doesn't yet have a use case for payload_size, so I don't know how its used. But I'm not sure we should make it have to equal buf_size, perhaps we should have it be:
*payload_size < buf_size
As it should be OK to over allocate, but not under allocate.
Quoted 111 lines
lib/trace-cmd/trace-msg.c +40
-5+ ret = -ENOMEM ; + goto out; + } +memcpy (*payload, msg.buf, buf_size); + goto out; + } + + *payload =malloc (buf_size); +if (*payload ==NULL ) { + ret = -ENOMEM ; + goto out; + } + *payload_size = buf_size; +memcpy (*payload, msg.buf, buf_size); + } + +out: +msg_free (&msg); +return ret; +} +static int make_trace_resp (struct tracecmd_msg *msg,int page_size,int nr_cpus,unsigned int *ports,bool use_fifos, -unsigned long long trace_id) +unsigned long long trace_id, +unsigned int tsync_proto, +unsigned int tsync_port) {int data_size;@@ -941,7 +1044,13 @@ static int make_trace_resp(struct tracecmd_msg *msg, int page_size, int nr_cpus,write_uints (msg->buf, data_size, ports, nr_cpus); msg->hdr.size =htonl (ntohl (msg->hdr.size) + data_size); - msg->trace_resp.flags = use_fifos ?htonl (MSG_TRACE_USE_FIFOS ) :htonl (0 ); + msg->trace_resp.flags =0 ; +if (use_fifos) + msg->trace_resp.flags |=MSG_TRACE_USE_FIFOS ; + msg->trace_resp.flags =htonl (msg->trace_resp.flags); + msg->trace_resp.tsync_proto =htonl (tsync_proto); + msg->trace_resp.tsync_port =htonl (tsync_port); + msg->trace_resp.cpus =htonl (nr_cpus); msg->trace_resp.page_size =htonl (page_size); msg->trace_resp.trace_id =htonll (trace_id);@@ -952,14 +1061,16 @@ static int make_trace_resp(struct tracecmd_msg *msg, int page_size, int nr_cpus,int tracecmd_msg_send_trace_resp (struct tracecmd_msg_handle *msg_handle,int nr_cpus,int page_size,unsigned int *ports,bool use_fifos, -unsigned long long trace_id) +unsigned long long trace_id, +unsigned int tsync_proto, +unsigned int tsync_port) {struct tracecmd_msg msg;int ret;tracecmd_msg_init (MSG_TRACE_RESP , &msg); ret =make_trace_resp (&msg, page_size, nr_cpus, ports, - use_fifos, trace_id); + use_fifos, trace_id, tsync_proto, tsync_port);if (ret <0 )return ret;@@ -969,7 +1080,9 @@ int tracecmd_msg_send_trace_resp(struct tracecmd_msg_handle *msg_handle,int tracecmd_msg_recv_trace_resp (struct tracecmd_msg_handle *msg_handle,int *nr_cpus,int *page_size,unsigned int **ports,bool *use_fifos, -unsigned long long *trace_id) +unsigned long long *trace_id, +unsigned int *tsync_proto, +unsigned int *tsync_port) {struct tracecmd_msg msg;char *p, *buf_end;@@ -996,6 +1109,8 @@ int tracecmd_msg_recv_trace_resp(struct tracecmd_msg_handle *msg_handle, *nr_cpus =ntohl (msg.trace_resp.cpus); *page_size =ntohl (msg.trace_resp.page_size); *trace_id =ntohll (msg.trace_resp.trace_id); + *tsync_proto =ntohl (msg.trace_resp.tsync_proto); + *tsync_port =ntohl (msg.trace_resp.tsync_port); *ports =calloc (*nr_cpus,sizeof (**ports));if (!*ports) { ret = -ENOMEM ;lib/trace-cmd/trace-timesync.c new file +21
-0@@ -0,0 +1,461 @@ +// SPDX-License-Identifier: LGPL-2.1 +/* +* Copyright (C) 2019, VMware, Tzvetomir Stoyanov <[email protected]> +* +*/ + +#include <fcntl.h> +#include <stdlib.h> +#include <unistd.h> +#include <arpa/inet.h> +#include <linux/vm_sockets.h> +#include <linux/limits.h> +#include <sys/types.h> +#include <sys/stat.h> +#include <time.h> +#include <dirent.h> +#include <errno.h> +#include "trace-cmd.h" +#include "trace-cmd-local.h" +#include "trace-tsync-local.h" +
lib/trace-cmd/trace-timesync.c line 31
Quoted 10 lines
lib/trace-cmd/trace-timesync.c new file +10
-0+struct tsync_proto { +struct tsync_proto *next; +unsigned int proto_id; +int weight; + +int (*clock_sync_init)(struct tracecmd_time_sync *clock_context); +int (*clock_sync_free)(struct tracecmd_time_sync *clock_context); +int (*clock_sync_calc)(struct tracecmd_time_sync *clock_context, +long long *offset,long long *timestamp); +}static * tsync_proto_list;
Just a nit, but it's usually cleaner and easier to read to separate the struct declaration and the variables. That is:
struct tsync_proto {
[..]
};static struct tsync_proto *tsync_proto_list;
lib/trace-cmd/trace-timesync.c line 33
Quoted 2 lines
lib/trace-cmd/trace-timesync.c new file +2
-0+ +static struct tsync_proto *tsync_proto_get (unsigned int proto_id)
I'd rename this to tsync_find_proto(), as to me, "_get" usually means that a reference count is incremented and it should be paired with a "_put" somewhere.
lib/trace-cmd/trace-timesync.c line 41
Quoted 8 lines
lib/trace-cmd/trace-timesync.c new file +8
-0+{ +struct tsync_proto *proto = tsync_proto_list; + +while (proto) { +if (proto->proto_id == proto_id) +return proto; + proto = proto->next; + }
Another nit, but I usually prefer for loops for this ;-)
for (proto = tsync_proto_list; proto; proto = proto->next) {
if (proto->proto_id == proto_id)
return proto;
}Quoted 38 lines
lib/trace-cmd/trace-timesync.c new file +38
-0+return NULL ; +} + +int tracecmd_tsync_proto_register (unsigned int proto_id,int weight, +int (*init)(struct tracecmd_time_sync *), +int (*free)(struct tracecmd_time_sync *), +int (*calc)(struct tracecmd_time_sync *, +long long *,long long *)) +{ +struct tsync_proto *proto; + +if (tsync_proto_get (proto_id)) +return -1 ; + proto =calloc (1 ,sizeof (struct tsync_proto )); +if (!proto) +return -1 ; + proto->proto_id = proto_id; + proto->weight = weight; + proto->clock_sync_init = init; + proto->clock_sync_free = free; + proto->clock_sync_calc = calc; + + proto->next = tsync_proto_list; + tsync_proto_list = proto; +return 0 ; +} + +int tracecmd_tsync_proto_unregister (unsigned int proto_id) +{ +struct tsync_proto *proto = tsync_proto_list; +struct tsync_proto *pdel; + +if (tsync_proto_list && tsync_proto_list->proto_id == proto_id) { + pdel = tsync_proto_list; + tsync_proto_list = pdel->next; +free (pdel); +return 0 ; + }
lib/trace-cmd/trace-timesync.c line 89
Quoted 10 lines
lib/trace-cmd/trace-timesync.c new file +10
-0+ +while (proto->next) { +if (proto->next->proto_id == proto_id) { + pdel = proto->next; + proto->next = pdel->next; +free (pdel); +return 0 ; + } + proto = proto->next; + }
The above could be written as one loop:
struct tsync_proto **last = &tsync_proto_list;
for (; *last; last = &(*last)->next) {
if ((*last)->proto_id = proto_id) {
struct tsync_proto *proto = *last;
*last = proto->next;
free(proto);
return 0;
}
}Quoted 27 lines
lib/trace-cmd/trace-timesync.c new file +27
-0+ +return -1 ; +} + +/** +* tracecmd_tsync_get_offsets - Return the calculated time offsests +* +* @tsync: Pointer to time sync context. +* @count: Returns the number of calculated time offsets. +* @ts: Array of size @count containing timestamps of callculated offsets +* @offsets: array of size @count, containing offsets for each timestamp. +* +* Retuns -1 in case of an error, or 0 otherwise +*/ +int tracecmd_tsync_get_offsets (struct tracecmd_time_sync *tsync, +int *count, +long long **ts,long long **offsets) +{ +struct clock_sync_context *tsync_context; + +if (!tsync || !tsync->context) +return -1 ; + tsync_context = (struct clock_sync_context *)tsync->context; +if (count) + *count = tsync_context->sync_count; +if (ts) + *ts = tsync_context->sync_ts;
lib/trace-cmd/trace-timesync.c line 126
Quoted 10 lines
lib/trace-cmd/trace-timesync.c new file +10
-0+if (offsets) + *offsets = tsync_context->sync_offsets; +return 0 ; +} + +/** +* tracecmd_tsync_proto_select - Select time sync protocol, to be used for +* timestamp synchronization with a peer. +* +* @protos: Bitmask of time sync protocols, supported by the peer.
Hmm, so we are limiting the number of protocols that we will ever support to 32? This makes me nervous (I always think of the famous quote of "Who will ever need more than 64K of RAM?")
To be safe, we should probably make this an array:
int tracecmd_tsync_proto_select(unsigned int *protos, int words)
The proto_id should be a bit, not a mask, that is:
int word;
for (word = 0; word < words; word++) {
for (proto = tsync_proto_list; proto; proto = proto->next) {
if (proto->proto_id < word * 32)
continue;
id = proto->proto_id - word * 32;
if (id >= 32)
continue;
if ((1 << id) & protos[word]) {
if (selected)
[...]While we only have a single word, callers would just do:
unsigned int protos; ret = tracecmd_tsync_proto_select(&protos, 1);
At least this will never limit us to just 32 protocols, even though we will likely never need more. But we are safe if we ever do ;-)
Quoted 22 lines
lib/trace-cmd/trace-timesync.c new file +22
-0+* +* Retuns Id of a time sync protocol, that can be used with the peer, or 0 +* in case there is no match with supported protocols. +*/ +unsigned int tracecmd_tsync_proto_select (unsigned int protos) +{ +struct tsync_proto *proto = tsync_proto_list; +struct tsync_proto *selected =NULL ; + +while (proto) { +if (proto->proto_id & protos) { +if (selected) { +if (selected->weight < proto->weight) + selected = proto; + }else + selected = proto; + } + proto = proto->next; + } + +if (selected) +return selected->proto_id;
lib/trace-cmd/trace-timesync.c line 158
Quoted 10 lines
lib/trace-cmd/trace-timesync.c new file +10
-0+ +return 0 ; +} + +/** +* tracecmd_tsync_proto_getall - Returns bitmask of all supported +* time sync protocols. +* +*/ +unsigned int tracecmd_tsync_proto_getall (void )
With the bitmask I suggest above, this would need to pass back as allocate mask.
Quoted 520 lines
lib/trace-cmd/trace-timesync.c new file +303
-0+{ +struct tsync_proto *proto = tsync_proto_list; +unsigned int protos =0 ; + +while (proto) { + protos |= proto->proto_id; + proto = proto->next; + } + +return protos; +} + +static int get_vsocket_params (int fd,unsigned int *lcid,unsigned int *lport, +unsigned int *rcid,unsigned int *rport) +{ +struct sockaddr_vm addr; +socklen_t addr_len =sizeof (addr); + +memset (&addr,0 ,sizeof (addr)); +if (getsockname (fd, (struct sockaddr *)&addr, &addr_len)) +return -1 ; +if (addr.svm_family !=AF_VSOCK ) +return -1 ; + *lport = addr.svm_port; + *lcid = addr.svm_cid; + +memset (&addr,0 ,sizeof (addr)); + addr_len =sizeof (addr); +if (getpeername (fd, (struct sockaddr *)&addr, &addr_len)) +return -1 ; +if (addr.svm_family !=AF_VSOCK ) +return -1 ; + *rport = addr.svm_port; + *rcid = addr.svm_cid; + +return 0 ; +} + +static struct tracecmd_instance * +clock_synch_create_instance (const char *clock,unsigned int cid) +{ +struct tracecmd_instance *vinst; +char inst_name[256 ]; + +snprintf (inst_name,256 ,"clock_synch-%d" , cid); + + vinst =tracecmd_create_instance (inst_name); +if (!vinst) +return NULL ; + +tracecmd_make_instance (vinst); +tracecmd_write_instance_file (vinst,"trace" ,"\0" ,NULL ); +if (clock) + vinst->clock =strdup (clock); +tracecmd_set_clock (vinst,NULL ); +return vinst; +} + +static void +clock_synch_delete_instance (struct tracecmd_instance *inst) +{ +if (!inst) +return ; +tracecmd_remove_instance (inst); +free (inst->clock); +tracecmd_free_instance (inst); +} + +static int clock_context_init (struct tracecmd_time_sync *tsync,bool server) +{ +struct clock_sync_context *clock =NULL ; +struct tsync_proto *protocol; + +if (tsync->context) +return 0 ; + + protocol =tsync_proto_get (tsync->sync_proto); +if (!protocol) +return -1 ; + + clock =calloc (1 ,sizeof (struct clock_sync_context )); +if (!clock) +return -1 ; + + clock->server = server; +if (get_vsocket_params (tsync->msg_handle->fd, &clock->local_cid, + &clock->local_port, &clock->remote_cid, + &clock->remote_port)) + goto error; + + clock->vinst =clock_synch_create_instance (tsync->clock_str, + clock->remote_cid); +if (!clock->vinst) + goto error; + + tsync->context = clock; +if (protocol->clock_sync_init && protocol->clock_sync_init (tsync) <0 ) + goto error; + +return 0 ; +error: + tsync->context =NULL ; +free (clock); +return -1 ; +} + +/** +* tracecmd_tsync_free - Free time sync context, allocated by +* tracecmd_tsync_with_host() or tracecmd_tsync_with_guest() APIs +* +* @tsync: Pointer to time sync context. +* +*/ +void tracecmd_tsync_free (struct tracecmd_time_sync *tsync) +{ +struct clock_sync_context *tsync_context; +struct tsync_proto *proto; + +if (!tsync->context) +return ; + tsync_context = (struct clock_sync_context *)tsync->context; + + proto =tsync_proto_get (tsync->sync_proto); +if (proto && proto->clock_sync_free) + proto->clock_sync_free (tsync); + +clock_synch_delete_instance (tsync_context->vinst); + tsync_context->vinst =NULL ; + +free (tsync_context->sync_ts); +free (tsync_context->sync_offsets); + tsync_context->sync_ts =NULL ; + tsync_context->sync_offsets =NULL ; + tsync_context->sync_count =0 ; + tsync_context->sync_size =0 ; +sem_destroy (&tsync->sem); +free (tsync->clock_str); +} + +static int tracecmd_tsync_send (struct tracecmd_time_sync *tsync, +struct tsync_proto *proto) +{ +long long timestamp =0 ; +long long offset =0 ; +int ret; + + ret = proto->clock_sync_calc (tsync, &offset, ×tamp); + +return ret; +} + +/** +* tracecmd_tsync_with_host - Synchronize timestamps with host +* +* @tsync: Pointer to time sync context. +* +* This API is supposed to be called in guest context. It waits for a time +* sync request from the host and replies with a time sample, until time sync +* stop command is received. +* +*/ +void tracecmd_tsync_with_host (struct tracecmd_time_sync *tsync) +{ +struct tsync_proto *proto; +unsigned int protocol; +unsigned int command; +int ret; + + proto =tsync_proto_get (tsync->sync_proto); +if (!proto || !proto->clock_sync_calc) +return ; + +clock_context_init (tsync, true); +if (!tsync->context) +return ; + +while (true) { + ret =tracecmd_msg_recv_time_sync (tsync->msg_handle, + &protocol, &command, +NULL ,NULL ); + +if (ret || + protocol !=TRACECMD_TIME_SYNC_PROTO_NONE || + command !=TRACECMD_TIME_SYNC_CMD_PROBE ) +break ; + ret =tracecmd_tsync_send (tsync, proto); +if (ret) +break ; + } +} + +static int tracecmd_tsync_get (struct tracecmd_time_sync *tsync, +struct tsync_proto *proto, +int array_step) +{ +struct clock_sync_context *clock; +long long *sync_offsets =NULL ; +long long *sync_ts =NULL ; +long long timestamp =0 ; +long long offset =0 ; +int ret; + + ret = proto->clock_sync_calc (tsync, &offset, ×tamp); +if (ret) { +warning ("Failed to synchronize timestamps with guest" ); +return -1 ; + } +if (!offset || !timestamp) +return 0 ; + clock = tsync->context; +if (clock->sync_count >= clock->sync_size) { + sync_ts =realloc (clock->sync_ts, + (clock->sync_size + array_step) *sizeof (long long )); + sync_offsets =realloc (clock->sync_offsets, + (clock->sync_size + array_step) *sizeof (long long )); +if (!sync_ts || !sync_offsets) { +free (sync_ts); +free (sync_offsets); +return -1 ; + } + clock->sync_size += array_step; + clock->sync_ts = sync_ts; + clock->sync_offsets = sync_offsets; + } + + clock->sync_ts[clock->sync_count] = timestamp; + clock->sync_offsets[clock->sync_count] = offset; + clock->sync_count++; + +return 0 ; +} + +#define TIMER_SEC_NANO 1000000000LL +static inline void get_ts_loop_delay (struct timespec *timeout,int delay_ms) +{ +memset (timeout,0 ,sizeof (struct timespec )); +clock_gettime (CLOCK_REALTIME , timeout); + + timeout->tv_nsec += ((unsigned long long )delay_ms *1000000LL ); + +if (timeout->tv_nsec >=TIMER_SEC_NANO ) { + timeout->tv_sec += timeout->tv_nsec /TIMER_SEC_NANO ; + timeout->tv_nsec %=TIMER_SEC_NANO ; + } +} + +#define CLOCK_TS_ARRAY 5 +/** +* tracecmd_tsync_with_guest - Synchronize timestamps with guest +* +* @tsync: Pointer to time sync context. +* +* This API is supposed to be called in host context, in a separate thread. +* It loops infinite, until the timesync semaphore is released. +* +*/ +void tracecmd_tsync_with_guest (struct tracecmd_time_sync *tsync) +{ +int ts_array_size =CLOCK_TS_ARRAY ; +struct tsync_proto *proto; +struct timespec timeout; +bool end = false; +int ret; + + proto =tsync_proto_get (tsync->sync_proto); +if (!proto || !proto->clock_sync_calc) +return ; + +clock_context_init (tsync, false); +if (!tsync->context) +return ; + +if (tsync->loop_interval >0 && + tsync->loop_interval < (CLOCK_TS_ARRAY *1000 )) + ts_array_size = (CLOCK_TS_ARRAY *1000 ) / tsync->loop_interval; + +while (true) { + ret =tracecmd_msg_send_time_sync (tsync->msg_handle, +TRACECMD_TIME_SYNC_PROTO_NONE , +TRACECMD_TIME_SYNC_CMD_PROBE , +0 ,NULL ); + ret =tracecmd_tsync_get (tsync, proto, ts_array_size); +if (ret || end) +break ; +if (tsync->loop_interval >0 ) { +get_ts_loop_delay (&timeout, tsync->loop_interval); + ret =sem_timedwait (&tsync->sem, &timeout); +if (ret <0 ) { +if (errno !=ETIMEDOUT && errno !=EINTR ) +break ; + }else if (!ret) + end = true; + }else { +sem_wait (&tsync->sem); + end = true; + } + }; + +tracecmd_msg_send_time_sync (tsync->msg_handle, +TRACECMD_TIME_SYNC_PROTO_NONE , +TRACECMD_TIME_SYNC_CMD_STOP , +0 ,NULL ); +}tracecmd/Makefile +2
-1@@ -30,6 +30,7 @@ TRACE_CMD_OBJS += trace-check-events.o TRACE_CMD_OBJS += trace-show.o TRACE_CMD_OBJS += trace-list.o TRACE_CMD_OBJS += trace-usage.o +TRACE_CMD_OBJS += trace-tsync.o ifeq ($(VSOCK_DEFINED), 1) TRACE_CMD_OBJS += trace-agent.o@@ -42,7 +43,7 @@ all_objs := $(sort $(ALL_OBJS)) all_deps := $(all_objs:$(bdir)/%.o=$(bdir)/.%.d) CONFIG_INCLUDES = -CONFIG_LIBS = -lrt +CONFIG_LIBS = -lrt -lpthread CONFIG_FLAGS = all: $(TARGETS)tracecmd/include/trace-local.h +11
-0@@ -226,6 +226,10 @@ struct buffer_instance {unsigned int port;int *fds;bool use_fifos; + +pthread_t tsync_thread; +bool tsync_thread_running; +struct tracecmd_time_sync tsync; };void init_top_instance (void );@@ -260,6 +264,13 @@ void tracecmd_disable_tracing(void);void tracecmd_enable_tracing (void );void tracecmd_stat_cpu (struct trace_seq *s,int cpu); +int tracecmd_host_tsync (struct buffer_instance *instance, +unsigned int tsync_port); +void tracecmd_host_tsync_complete (struct buffer_instance *instance); +unsigned int tracecmd_guest_tsync (unsigned int tsync_protos,char *clock, +unsigned int *tsync_port, +pthread_t *thr_id); +int make_vsock (unsigned int port);int get_vsock_port (int sd,unsigned int *port);int open_vsock (unsigned int cid,unsigned int port);tracecmd/trace-agent.c +32
-2@@ -19,6 +19,7 @@#include <sys/wait.h> #include <unistd.h> #include <linux/vm_sockets.h> +#include <pthread.h> #include "trace-local.h" #include "trace-msg.h" @@ -125,11 +126,28 @@ cleanup:return ret; } +static char *get_clock (int argc,char **argv) +{ +int i; + +if (!argc || !argv) +return NULL ; + +for (i =0 ; i < argc -1 ; i++) { +if (!strcmp ("-C" , argv[i])) +return argv[i+1 ]; + } +return NULL ; +} +static void agent_handle (int sd,int nr_cpus,int page_size) {struct tracecmd_msg_handle *msg_handle;unsigned long long trace_id; +unsigned int tsync_protos =0 ; +unsigned int tsync_port =0 ;unsigned int *ports; +pthread_t sync_thr;char **argv =NULL ;int argc =0 ;bool use_fifos;@@ -146,7 +164,8 @@ static void agent_handle(int sd, int nr_cpus, int page_size)die ("Failed to allocate message handle" ); ret =tracecmd_msg_recv_trace_req (msg_handle, &argc, &argv, - &use_fifos, &trace_id); + &use_fifos, &trace_id, + &tsync_protos);if (ret <0 )die ("Failed to receive trace request" );@@ -155,14 +174,25 @@ static void agent_handle(int sd, int nr_cpus, int page_size)if (!use_fifos)make_vsocks (nr_cpus, fds, ports); +if (tsync_protos) { + tsync_protos =tracecmd_guest_tsync (tsync_protos, +get_clock (argc, argv), + &tsync_port, &sync_thr); +if (!tsync_protos) +warning ("Failed to negotiate timestamps synchronization with the host" ); + } trace_id =tracecmd_generate_traceid (); ret =tracecmd_msg_send_trace_resp (msg_handle, nr_cpus, page_size, - ports, use_fifos, trace_id); + ports, use_fifos, trace_id, + tsync_protos, tsync_port);if (ret <0 )die ("Failed to send trace response" );trace_record_agent (msg_handle, nr_cpus, fds, argc, argv, use_fifos, trace_id); +if (tsync_protos) +pthread_join (sync_thr,NULL ); +free (argv[0 ]);free (argv);free (ports);tracecmd/trace-record.c +28
-4@@ -670,6 +670,11 @@ static void tell_guests_to_stop(void)tracecmd_msg_send_close_msg (instance->msg_handle); } +for_all_instances (instance) { +if (is_guest (instance)) +tracecmd_host_tsync_complete (instance); + } +/* Wait for guests to acknowledge */ for_all_instances (instance) {if (is_guest (instance)) {@@ -3501,6 +3506,8 @@ static void connect_to_agent(struct buffer_instance *instance) {struct tracecmd_msg_handle *msg_handle;int sd, ret, nr_fifos, nr_cpus, page_size; +unsigned int tsync_protos_reply =0 ; +unsigned int tsync_port =0 ;unsigned int *ports;int i, *fds =NULL ;bool use_fifos = false;@@ -3519,19 +3526,32 @@ static void connect_to_agent(struct buffer_instance *instance)if (!msg_handle)die ("Failed to allocate message handle" ); +if (instance->tsync.loop_interval >=0 ) + instance->tsync.sync_proto =tracecmd_tsync_proto_getall (); + ret =tracecmd_msg_send_trace_req (msg_handle, instance->argc, instance->argv, use_fifos, - top_instance.trace_id); + top_instance.trace_id, + instance->tsync.sync_proto);if (ret <0 )die ("Failed to send trace request" ); ret =tracecmd_msg_recv_trace_resp (msg_handle, &nr_cpus, &page_size, &ports, &use_fifos, - &instance->trace_id); + &instance->trace_id, + &tsync_protos_reply, &tsync_port);if (ret <0 ) -die ("Failed to receive trace response" ); +die ("Failed to receive trace response %d" , ret); + +if (instance->tsync.sync_proto) { +if (instance->tsync.sync_proto & tsync_protos_reply) { + instance->tsync.sync_proto = tsync_protos_reply; +tracecmd_host_tsync (instance, tsync_port); + }else +warning ("Failed to negotiate timestamps synchronization with the guest" ); + }if (use_fifos) {if (nr_cpus != nr_fifos) {@@ -3578,7 +3598,9 @@ static void setup_guest(struct buffer_instance *instance)close (fd); } -static void setup_agent (struct buffer_instance *instance,struct common_record_context *ctx) +static void setup_agent (struct buffer_instance *instance, +struct common_record_context *ctx) + {struct tracecmd_output *network_handle;@@ -5131,6 +5153,7 @@ void init_top_instance(void) }enum { + OPT_tsyncinterval =242 , OPT_user =243 , OPT_procmap =244 , OPT_quiet =245 ,@@ -5433,6 +5456,7 @@ static void parse_record_options(int argc, {"proc-map", no_argument, NULL, OPT_procmap}, {"user", required_argument, NULL, OPT_user}, {"module", required_argument, NULL, OPT_module}, + {"tsync-interval", required_argument, NULL, OPT_tsyncinterval}, {NULL, 0, NULL, 0} };
tracecmd/trace-record.c static void parse_record_options(int argc, · line 5800
Quoted 6 lines
tracecmd/trace-record.c +3
-0@@ -5771,6 +5795,10 @@ static void parse_record_options(int argc, ctx->instance->filter_mod = optarg; ctx->filtered =0 ;break ; +case OPT_tsyncinterval: + top_instance.tsync.loop_interval =atoi (optarg); + guest_config = true;
I still don't like the name of the variable. "guest_config" makes it sound like this is a guest config, or we set a guest config. Where as you are just going to do something to the guests when this gets set. It's still rather confusing.
Still would be better with "clock_option_set" or something like that. It's more descriptive of why it is true. Or perhaps something like "guest_sync_set", where we are saying that we need to do something for synchronizing guests.
Quoted 231 lines
tracecmd/trace-record.c +2
-0+break ;case OPT_quiet:case 'q' : quiet = true;@@ -5799,6 +5827,7 @@ static void parse_record_options(int argc,add_argv (instance,"-C" , true); } } + instance->tsync.loop_interval = top_instance.tsync.loop_interval; } }tracecmd/trace-tsync.c new file +213
-0@@ -0,0 +1,268 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Copyright (C) 2019, VMware, Tzvetomir Stoyanov <[email protected]> +* +*/ +#include <stdlib.h> +#include <errno.h> +#include <fcntl.h> +#include <sys/socket.h> +#include <unistd.h> +#include <linux/vm_sockets.h> +#include <pthread.h> + +#include "trace-local.h" +#include "trace-msg.h" + +static int get_first_cpu (cpu_set_t **pin_mask,size_t *m_size) +{ +int cpus =tracecmd_count_cpus (); +cpu_set_t *cpu_mask; +int mask_size; +int i; + + cpu_mask =CPU_ALLOC (cpus); + *pin_mask =CPU_ALLOC (cpus); +if (!cpu_mask || !*pin_mask ||1 ) + goto error; + + mask_size =CPU_ALLOC_SIZE (cpus); +CPU_ZERO_S (mask_size, cpu_mask); +CPU_ZERO_S (mask_size, *pin_mask); + +if (sched_getaffinity (0 , mask_size, cpu_mask) ==-1 ) + goto error; + +for (i =0 ; i < cpus; i++) { +if (CPU_ISSET_S (i, mask_size, cpu_mask)) { +CPU_SET_S (i, mask_size, *pin_mask); +break ; + } + } + +if (CPU_COUNT_S (mask_size, *pin_mask) <1 ) + goto error; + +CPU_FREE (cpu_mask); + *m_size = mask_size; +return 0 ; + +error: +if (cpu_mask) +CPU_FREE (cpu_mask); +if (*pin_mask) +CPU_FREE (*pin_mask); + *pin_mask =NULL ; + *m_size =0 ; +return -1 ; +} + +static void *tsync_host_thread (void *data) +{ +struct tracecmd_time_sync *tsync =NULL ; + + tsync = (struct tracecmd_time_sync *)data; + +tracecmd_tsync_with_guest (tsync); + +tracecmd_msg_handle_close (tsync->msg_handle); + tsync->msg_handle =NULL ; + +pthread_exit (0 ); +} + +int tracecmd_host_tsync (struct buffer_instance *instance, +unsigned int tsync_port) +{ +struct tracecmd_msg_handle *msg_handle =NULL ; +cpu_set_t *pin_mask =NULL ; +pthread_attr_t attrib; +size_t mask_size =0 ; +int ret; +int fd; + +if (!instance->tsync.sync_proto) +return -1 ; + + fd =open_vsock (instance->cid, tsync_port); +if (fd <0 ) { + ret =-1 ; + goto out; + } + msg_handle =tracecmd_msg_handle_alloc (fd,0 ); +if (!msg_handle) { + ret =-1 ; + goto out; + } + + instance->tsync.msg_handle = msg_handle; +if (top_instance.ftrace->clock) + instance->tsync.clock_str =strdup (top_instance.ftrace->clock); + ret =sem_init (&instance->tsync.sem,0 ,0 ); +if (ret) + goto out; + +pthread_attr_init (&attrib); +pthread_attr_setdetachstate (&attrib,PTHREAD_CREATE_JOINABLE ); +if (!get_first_cpu (&pin_mask, &mask_size)) +pthread_attr_setaffinity_np (&attrib, mask_size, pin_mask); + + ret =pthread_create (&instance->tsync_thread, &attrib, + tsync_host_thread, &instance->tsync); +if (!ret) + instance->tsync_thread_running = true; +if (pin_mask) +CPU_FREE (pin_mask); +pthread_attr_destroy (&attrib); + +out: +if (ret) { +if (msg_handle) +tracecmd_msg_handle_close (msg_handle); + } + +return ret; +} + +static void write_guest_time_shift (struct buffer_instance *instance) +{ +struct tracecmd_output *handle; +struct iovec vector[4 ]; +long long *offsets; +long long *ts; +char *file; +int count; +int ret; +int fd; + + ret =tracecmd_tsync_get_offsets (&instance->tsync, &count, &ts, &offsets); +if (ret <0 || !count || !ts || !offsets) +return ; + + file =get_guest_file (TRACE_FILENAME , instance->ftrace->name); + fd =open (file,O_RDWR ); +if (fd <0 ) +die ("error opening %s" , file); +free (file); + handle =tracecmd_get_output_handle_fd (fd); + vector[0 ].iov_len =8 ; + vector[0 ].iov_base = &top_instance.trace_id; + vector[1 ].iov_len =4 ; + vector[1 ].iov_base = &count; + vector[2 ].iov_len =8 * count; + vector[2 ].iov_base = ts; + vector[3 ].iov_len =8 * count; + vector[3 ].iov_base = offsets; +tracecmd_add_option_v (handle,TRACECMD_OPTION_TIME_SHIFT , vector,4 ); +tracecmd_append_options (handle); +tracecmd_output_close (handle); +#ifdef TSYNC_DEBUG +if (count >1 ) +printf ("Got %d timestamp synch samples for guest %s in %lld ns trace\n\r" , + count, instance->ftrace->name, ts[count -1 ] - ts[0 ]); +#endif +} + +void tracecmd_host_tsync_complete (struct buffer_instance *instance) +{ +if (!instance->tsync_thread_running) +return ; + +sem_post (&instance->tsync.sem); +pthread_join (instance->tsync_thread,NULL ); +write_guest_time_shift (instance); +tracecmd_tsync_free (&instance->tsync); +} + +static void *tsync_agent_thread (void *data) +{ +struct tracecmd_time_sync *tsync =NULL ; +int sd; + + tsync = (struct tracecmd_time_sync *)data; + +while (true) { + sd =accept (tsync->msg_handle->fd,NULL ,NULL ); +if (sd <0 ) { +if (errno ==EINTR ) +continue ; + goto out; + } +break ; + } +close (tsync->msg_handle->fd); + tsync->msg_handle->fd = sd; + +tracecmd_tsync_with_host (tsync); + +out: +tracecmd_msg_handle_close (tsync->msg_handle); +tracecmd_tsync_free (tsync); +free (tsync); +close (sd); + +pthread_exit (0 ); +} + +unsigned int tracecmd_guest_tsync (unsigned int tsync_protos,char *clock, +unsigned int *tsync_port, +pthread_t *thr_id) +{ +struct tracecmd_time_sync *tsync =NULL ; +cpu_set_t *pin_mask =NULL ; +pthread_attr_t attrib;
tracecmd/trace-tsync.c line 223
Quoted 10 lines
tracecmd/trace-tsync.c new file +10
-0+size_t mask_size =0 ; +unsigned int proto; +int ret; +int fd; + + fd =-1 ; + proto =tracecmd_tsync_proto_select (tsync_protos); +if (!proto) +return 0 ; +#ifdef VSOCK
Instead of this, perhaps we should not have this file get compiled if VSOCK is not defined?
-- Steve
Quoted 60 lines
tracecmd/trace-tsync.c new file +45
-0+ fd =make_vsock (VMADDR_PORT_ANY ); +if (fd <0 ) + goto error; + + ret =get_vsock_port (fd, tsync_port); +if (ret <0 ) + goto error; +#else +return 0 ; +#endif + + tsync =calloc (1 ,sizeof (struct tracecmd_time_sync )); + tsync->msg_handle =tracecmd_msg_handle_alloc (fd,0 ); +if (clock) + tsync->clock_str =strdup (clock); + +pthread_attr_init (&attrib); + tsync->sync_proto = proto; +pthread_attr_setdetachstate (&attrib,PTHREAD_CREATE_JOINABLE ); +if (!get_first_cpu (&pin_mask, &mask_size)) +pthread_attr_setaffinity_np (&attrib, mask_size, pin_mask); + + ret =pthread_create (thr_id, &attrib, tsync_agent_thread, tsync); + +if (pin_mask) +CPU_FREE (pin_mask); +pthread_attr_destroy (&attrib); + +if (ret) + goto error; + +return proto; + +error: +if (tsync) { +if (tsync->msg_handle) +tracecmd_msg_handle_close (tsync->msg_handle); +if (tsync->clock_str) +free (tsync->clock_str); +free (tsync); + } +if (fd >0 ) +close (fd); +return 0 ; +}tracecmd/trace-usage.c +4
-0@@ -60,6 +60,10 @@ static struct usage_help usage_help[] = {" --no-filter include trace-cmd threads in the trace\n" " --proc-map save the traced processes address map into the trace.dat file\n" " --user execute the specified [command ...] as given user\n" +" --tsync-interval set the loop interval, in ms, for timestamps synchronization with guests:" +" If a negative number is specified, timestamps synchronization is disabled" +" If 0 is specified, no loop is performed - timestamps offset is calculated only twice," +" at the beginnig and at the end of the trace\n" }, { "start",
Re: [PATCH v17 16/18] trace-cmd: Basic infrastructure for host - guest timestamp synchronization
Thanks, Steven!
I will prepare the next version addressing the comments.
Quoted 1 line · Steven Rostedt
[ ... ]
Quoted 5 lines
Quoted 2 lines
+static int
tracecmd_msg_send_nofree(int fd, struct tracecmd_msg *msg)We probably shouldn't have a static function that starts with "
tracecmd_".
I changed this one, but noticed few other static "tracecmd_" functions in the legacy code, will rename them in a separate patch.
Quoted 1 line
[ ... ]
Quoted 10 lines
Quoted 10 lines
static int make_trace_req(struct
tracecmd_msg*msg, int argc, char **argv,lib/trace-cmd/trace-msg.c +2
-1- bool use_fifos, unsigned long long trace_id) + bool use_fifos, unsigned long long trace_id, + unsigned int tsync_protos) { size_t args_size = 0;char *p;@@ -824,7 +848,11 @@ static int make_trace_req(struct tracecmd_msg *msg, int argc, char **argv, args_size +=strlen (argv[i]) +1 ;
Quoted 7 lines
lib/trace-cmd/trace-msg.c static int make_trace_req(struct tracecmd_msg *msg, int argc, char **argv, · line 853
Quoted 5 lines
lib/trace-cmd/trace-msg.c +3
-1msg->hdr.size =htonl (ntohl (msg->hdr.size) + args_size); - msg->trace_req.flags = use_fifos ?htonl (MSG_TRACE_USE_FIFOS ) :htonl (0 ); + msg->trace_req.flags =0 ; +if (use_fifos) + msg->trace_req.flags |=MSG_TRACE_USE_FIFOS ;Why the change to flags here?
Fixed, It was a leftover from a previous version of this patch, where flags were extended with time sync specific one.
However, I removed that time sync specific flag but forgot about this change.
Quoted 1 line
[ ... ]
Quoted 33 lines
lib/trace-cmd/trace-timesync.c line 126
Quoted 6 lines
lib/trace-cmd/trace-timesync.c new file +6
-0+ +/** +* tracecmd_tsync_proto_select - Select time sync protocol, to be used for +* timestamp synchronization with a peer. +* +* @protos: Bitmask of time sync protocols, supported by the peer. Hmm, so we are limiting the number of protocols that we will ever support to 32? This makes me nervous (I always think of the famous quote of "Who will ever need more than 64K of RAM?")
To be safe, we should probably make this an array:
int
tracecmd_tsync_proto_select(unsigned int *protos, int words)The
proto_idshould be a bit, not a mask, that is:int word; for (word = 0; word < words; word++) { for (proto = tsync_proto_list; proto; proto = proto->next) { if (proto->proto_id < word * 32) continue; id = proto->proto_id - word * 32; if (id >= 32) continue; if ((1 << id) & protos[word]) { if (selected) [...]
Quoted 9 lines
While we only have a single word, callers would just do:
unsigned int protos; ret = tracecmd_tsync_proto_select(&protos, 1);At least this will never limit us to just 32 protocols, even though we will likely never need more. But we are safe if we ever do ;-)
I changed this and tracecmd_tsync_proto_getall() to work with unlimited bitmasks,
but in order to support more than 32 protocols the trace request packet must be changed. In the current patch I added field
be32 tsync_protos;
in struct tracecmd_msg_trace_req, which is bitmask for all time sync protocols, supported
by the host. Defining this as a fixed size bitmap simplifies a lot the protocol. We can add a
dynamic bitmask as part of the trace request packet payload, but that payload is already used
for the trace arguments. We can extend the payload to hold both dynamic bitmask and trace arguments,
but I think it will complicate a lot the packet handling and the benefit will be only the posibility to support
more than 32 time sync protocols.
Quoted 1 line
Quoted 1 line
+ *
[ ... ]
Signature · 2 lines
Tzvetomir (Ceco) Stoyanov VMware Open Source Technology Center
Re: [PATCH v17 16/18] trace-cmd: Basic infrastructure for host - guest timestamp synchronization
Quoted 7 lines · On Thu, 12 Dec 2019 14:34:39 +0200 Tzvetomir Stoyanov <[email protected]>…
Thanks, Steven!
I will prepare the next version addressing the comments.Quoted 1 line · Steven Rostedt
[ ... ]
Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
+static int
tracecmd_msg_send_nofree(int fd, struct tracecmd_msg *msg)We probably shouldn't have a static function that starts with "
tracecmd_".I changed this one, but noticed few other static "
tracecmd_" functions in the legacy code, will rename them in a separate patch.
Yeah, those should probably be changed too.
Quoted 50 lines
Quoted 1 line
[ ... ]
Quoted 33 lines
lib/trace-cmd/trace-timesync.c line 126
Quoted 6 lines
lib/trace-cmd/trace-timesync.c new file +6
-0+ +/** +* tracecmd_tsync_proto_select - Select time sync protocol, to be used for +* timestamp synchronization with a peer. +* +* @protos: Bitmask of time sync protocols, supported by the peer. Hmm, so we are limiting the number of protocols that we will ever support to 32? This makes me nervous (I always think of the famous quote of "Who will ever need more than 64K of RAM?")
To be safe, we should probably make this an array:
int
tracecmd_tsync_proto_select(unsigned int *protos, int words)The
proto_idshould be a bit, not a mask, that is:int word; for (word = 0; word < words; word++) { for (proto = tsync_proto_list; proto; proto = proto->next) { if (proto->proto_id < word * 32) continue; id = proto->proto_id - word * 32; if (id >= 32) continue; if ((1 << id) & protos[word]) { if (selected) [...]Quoted 9 lines
While we only have a single word, callers would just do:
unsigned int protos; ret = tracecmd_tsync_proto_select(&protos, 1);At least this will never limit us to just 32 protocols, even though we will likely never need more. But we are safe if we ever do ;-)
I changed this and
tracecmd_tsync_proto_getall()to work with unlimited bitmasks,
but in order to support more than 32 protocols the trace request packet must be changed. In the current patch I added fieldbe32 tsync_protos;
Quoted 11 lines
in struct
tracecmd_msg_trace_req, which is bitmask for all time sync protocols, supported
by the host. Defining this as a fixed size bitmap simplifies a lot the protocol. We can add a
dynamic bitmask as part of the trace request packet payload, but that payload is already used
for the trace arguments. We can extend the payload to hold both dynamic bitmask and trace arguments,
but I think it will complicate a lot the packet handling and the benefit will be only the posibility to support
more than 32 time sync protocols.
Perhaps instead, we can add an option to which 32bit word the protocols are used. That is, we pass an index of what the tsync_protos are, and the response can pick one of them, and even say how many more words it has.
That is, I don't think we should change the protocol to pass the entire bitmask. Just part of it, and if we find a match, we use that. That should keep it from being too complicated, right?
-- Steve
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
+ *
[ ... ]
Re: [PATCH v17 16/18] trace-cmd: Basic infrastructure for host - guest timestamp synchronization
In reply to Steven RostedtForgot to comment on that in the previous reply :
Quoted 1 line · Steven Rostedt
[ ... ]
Quoted 13 lines
Quoted 13 lines
tracecmd/trace-tsync.c new file +13
-0+ +unsigned int tracecmd_guest_tsync (unsigned int tsync_protos,char *clock, +unsigned int *tsync_port, +pthread_t *thr_id) +{ +struct tracecmd_time_sync *tsync =NULL ; +cpu_set_t *pin_mask =NULL ; +pthread_attr_t attrib; +size_t mask_size =0 ; +unsigned int proto; +int ret; +int fd; +
Quoted 8 lines
tracecmd/trace-tsync.c line 223
Quoted 5 lines
tracecmd/trace-tsync.c new file +5
-0+ fd =-1 ; + proto =tracecmd_tsync_proto_select (tsync_protos); +if (!proto) +return 0 ; +#ifdef VSOCK Instead of this, perhaps we should not have this file get compiled if VSOCK is not defined?
I would recommend to isolate VSOCK defines inside the time synchronization code. The caller might not be aware of time synchronization internal dependencies. The current implementation relies on vsoks, but we may change this in the future and
this should not affect the callers.
Quoted 65 lines
-- Steve
Quoted 60 lines
tracecmd/trace-tsync.c new file +45
-0+ fd =make_vsock (VMADDR_PORT_ANY ); +if (fd <0 ) + goto error; + + ret =get_vsock_port (fd, tsync_port); +if (ret <0 ) + goto error; +#else +return 0 ; +#endif + + tsync =calloc (1 ,sizeof (struct tracecmd_time_sync )); + tsync->msg_handle =tracecmd_msg_handle_alloc (fd,0 ); +if (clock) + tsync->clock_str =strdup (clock); + +pthread_attr_init (&attrib); + tsync->sync_proto = proto; +pthread_attr_setdetachstate (&attrib,PTHREAD_CREATE_JOINABLE ); +if (!get_first_cpu (&pin_mask, &mask_size)) +pthread_attr_setaffinity_np (&attrib, mask_size, pin_mask); + + ret =pthread_create (thr_id, &attrib, tsync_agent_thread, tsync); + +if (pin_mask) +CPU_FREE (pin_mask); +pthread_attr_destroy (&attrib); + +if (ret) + goto error; + +return proto; + +error: +if (tsync) { +if (tsync->msg_handle) +tracecmd_msg_handle_close (tsync->msg_handle); +if (tsync->clock_str) +free (tsync->clock_str); +free (tsync); + } +if (fd >0 ) +close (fd); +return 0 ; +}tracecmd/trace-usage.c +4
-0@@ -60,6 +60,10 @@ static struct usage_help usage_help[] = {" --no-filter include trace-cmd threads in the trace\n" " --proc-map save the traced processes address map into the trace.dat file\n" " --user execute the specified [command ...] as given user\n" +" --tsync-interval set the loop interval, in ms, for timestamps synchronization with guests:" +" If a negative number is specified, timestamps synchronization is disabled" +" If 0 is specified, no loop is performed - timestamps offset is calculated only twice," +" at the beginnig and at the end of the trace\n" }, { "start",
Signature · 2 lines
Tzvetomir (Ceco) Stoyanov VMware Open Source Technology Center
PATCH v17 17/18 trace-cmd: [POC] PTP-like algorithm for host - guest timestamp synchronization
In reply to Tzvetomir Stoyanov (VMware)PTP protocol is designed for synchronizing clocks of machines in a local network. The same approach can be used for host - guest timestamp synchronization. This implementation uses ftrace raw markers to track trace timestamps of PTP events. The patch is a POC, two different algorithms for PTP calculations are proposed:
- Choosing the sample with the fastest response time for calculating the clocks offset.
- Calculating the clocks offset using the average of all PTP samples.
The implementation can be tuned using those parameters:
- #define FASTEST_RESPONSE - is defined, the sample with the fastest response time is used for calculating the clocks offset. Otherwise the histogram of all samples is used.
- #define PTP_SYNC_LOOP 340 - defines the number of samples, used for one calculation.
- --tsync-interval - a trace-cmd argument, choose the intervals between offset calculations, performed continuously during the trace.
- #define TSYNC_DEBUG - if defined, a debug information is collected and stored in files, in the guest machine:
s-cid*.txt - For each offset calculation: host and guest clocks and calculated offset. res-cid*.txt - For each tracing session: all calculated clock offsets.
--- include/trace-cmd/trace-cmd.h | 2 + lib/trace-cmd/Makefile | 1 + lib/trace-cmd/trace-timesync-ptp.c | 649 +++++++++++++++++++++++++++++ lib/trace-cmd/trace-timesync.c | 8 + tracecmd/trace-agent.c | 2 + tracecmd/trace-record.c | 25 +- 6 files changed, 679 insertions(+), 8 deletions(-) create mode 100644 lib/trace-cmd/trace-timesync-ptp.c
include/trace-cmd/trace-cmd.h +2 -0
@@ -427,6 +427,7 @@ int tracecmd_iterate_raw_events(struct tep_handle *tep,enum {TRACECMD_TIME_SYNC_PROTO_NONE =0 , +TRACECMD_TIME_SYNC_PROTO_PTP =1 <<0 , };enum {TRACECMD_TIME_SYNC_CMD_PROBE =1 ,
@@ -444,6 +445,7 @@ struct tracecmd_time_sync {void *context; }; +void tracecmd_tsync_init (void );unsigned int tracecmd_tsync_proto_getall (void );unsigned int tracecmd_tsync_proto_select (unsigned int protos);void tracecmd_tsync_with_host (struct tracecmd_time_sync *tsync);
lib/trace-cmd/Makefile +1 -0
@@ -17,6 +17,7 @@ OBJS += trace-instance.o OBJS += trace-filter-hash.o OBJS += trace-msg.o OBJS += trace-timesync.o +OBJS += trace-timesync-ptp.o # Additional util objects OBJS += trace-blk-hack.o
lib/trace-cmd/trace-timesync-ptp.c new file +649 -0
649 lines not on this page · raw message
lib/trace-cmd/trace-timesync.c +8 -0
@@ -42,6 +42,14 @@ static struct tsync_proto *tsync_proto_get(unsigned int proto_id)return NULL ; } +/** +* tracecmd_tsync_init - Initialize the global, per task, time sync data. +*/ +void tracecmd_tsync_init (void ) +{ +ptp_clock_sync_register (); +} + int tracecmd_tsync_proto_register(unsigned int proto_id, int weight, int (*init)(struct tracecmd_time_sync *), int (*free)(struct tracecmd_time_sync *),
tracecmd/trace-agent.c +2 -0
@@ -242,6 +242,8 @@ static void agent_serve(unsigned int port)if (sd <0 )die ("Failed to open vsocket" ); +tracecmd_tsync_init (); +if (!get_local_cid (&cid))printf ("listening on @%u:%u\n" , cid, port);
tracecmd/trace-record.c +17 -8
@@ -5944,10 +5944,7 @@ static void set_clock(struct buffer_instance *instance)tracecmd_put_tracing_file (path); } } -/* -* This function contains common code for the following commands: -* record, start, stream, profile. -*/ +static void record_trace (int argc,char **argv,struct common_record_context *ctx) {
@@ -6080,12 +6077,24 @@ static void record_trace(int argc, char **argv,finalize_record_trace (ctx); } +/* +* This function contains common code for the following commands: +* record, start, stream, profile. +*/ +static void record_trace_command (int argc,char **argv, +struct common_record_context *ctx) +{ +tracecmd_tsync_init (); +record_trace (argc, argv, ctx); +} + +void trace_start (int argc,char **argv) {struct common_record_context ctx;parse_record_options (argc, argv, CMD_start, &ctx); -record_trace (argc, argv, &ctx); +record_trace_command (argc, argv, &ctx);exit (0 ); }
@@ -6165,7 +6174,7 @@ void trace_stream(int argc, char **argv)struct common_record_context ctx;parse_record_options (argc, argv, CMD_stream, &ctx); -record_trace (argc, argv, &ctx); +record_trace_command (argc, argv, &ctx);exit (0 ); }
@@ -6184,7 +6193,7 @@ void trace_profile(int argc, char **argv)if (!buffer_instances) top_instance.flags |=BUFFER_FL_PROFILE ; -record_trace (argc, argv, &ctx); +record_trace_command (argc, argv, &ctx);do_trace_profile ();exit (0 ); }
@@ -6203,7 +6212,7 @@ void trace_record(int argc, char **argv)struct common_record_context ctx;parse_record_options (argc, argv, CMD_record, &ctx); -record_trace (argc, argv, &ctx); +record_trace_command (argc, argv, &ctx);exit (0 ); }
Signature · 1 line
2.23.0
PATCH v17 18/18 trace-cmd: Debug scripts for PTP-like algorithm for host - guest timestamp synchronization
In reply to Tzvetomir Stoyanov (VMware)These scripts can be used to visualise debug files, written when the PTP-like algorithm is compiled with TSYNC_DEBUG defined. The files are located in the guest machine:
s-cid*.txt - For each offset calculation: host and guest clocks and calculated offset. res-cid*.txt - For each tracing session: all calculated clock offsets.
tsync_hist.py plots a histogram, using data from a s-cid*.txt file:
"python tsync_hist.py s-cid2_1.txt"
tsync_res.py plots a line, using data from res-cid*.txt file:
"python tsync_res.py res-cid2.txt"
--- scripts/debug/tsync_hist.py | 57 +++++++++++++++++++++++++++++++++++++ scripts/debug/tsync_readme | 12 ++++++++ scripts/debug/tsync_res.py | 46 ++++++++++++++++++++++++++++++ 3 files changed, 115 insertions(+) create mode 100644 scripts/debug/tsync_hist.py create mode 100644 scripts/debug/tsync_readme create mode 100644 scripts/debug/tsync_res.py
scripts/debug/tsync_hist.py new file +57 -0
@@ -0,0 +1,57 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Copyright (C) 2019, VMware Inc, Tzvetomir Stoyanov <[email protected]> +# Copyright (C) 2019, VMware Inc, Yordan Karadzhov <[email protected]> + + +import matplotlib.pyplot as plt +import matplotlib.lines as mlines +import numpy as np +import sys + +def newline(p1, p2): + ax = plt.gca() + xmin, xmax = ax.get_xbound() + + if(p2[0] == p1[0]): + xmin = xmax = p1[0] + ymin, ymax = ax.get_ybound() + else: + ymax = p1[1]+(p2[1]-p1[1])/(p2[0]-p1[0])*(xmax-p1[0]) + ymin = p1[1]+(p2[1]-p1[1])/(p2[0]-p1[0])*(xmin-p1[0]) + + l = mlines.Line2D([xmin,xmax], [ymin,ymax], color='red') + ax.add_line(l) + return l + + +data = np.loadtxt(fname = sys.argv[1]) +selected_ts = data[-1, 1] +selected_ofs = data[-1, 0] +data = data[:-1,:] + +x = data[:, 1] - data[:, 0] + +mean = x.mean() +std = x.std() + +num_bins = 500 +min = x.min() #+ .4 * (x.max() - x.min()) +max = x.max() #- .4 * (x.max() - x.min()) +bins = np.linspace(min, max, num_bins, endpoint = False, dtype=int) + +fig, ax = plt.subplots() + +# the histogram of the data +n, bins, patches = ax.hist(x, bins, histtype=u'step'); + +ax.set_xlabel('clock offset [$\mu$s]') +ax.set_ylabel('entries') +ax.set_title("$\sigma$=%i" % std) + +x1, y1 = [selected_ofs, min], [selected_ofs, max] +newline(x1, y1) + +# Tweak spacing to prevent clipping of ylabel +fig.tight_layout() +plt.show()
scripts/debug/tsync_readme new file +12 -0
@@ -0,0 +1,12 @@ +PTP-like algorithm debug +======================== + +tsync_*.py scripts can be used to visualise debug files, written when the PTP-like algorithm +is compiled with TSYNC_DEBUG defined. The files are located in the guest machine: + s-cid*.txt - For each offset calculation: host and guest clocks and calculated offset. + res-cid*.txt - For each tracing session: all calculated clock offsets. + +tsync_hist.py plots a histogram, using data from a s-cid*.txt file: + "python tsync_hist.py s-cid2_1.txt" +tsync_res.py plots a line, using data from res-cid*.txt file: + "python tsync_res.py res-cid2.txt"
scripts/debug/tsync_res.py new file +46 -0
46 lines not on this page · raw message
Signature · 1 line
2.23.0