Skip to contentSkip to conversation

trace-cmd: rework of the pid detection of vcpus

PATCH In review Vineeth Pillai 2 participants 8 messages May 4, 2022 – Jul 12, 2022

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 trace-cmd: rework of the pid detection of vcpus

The current detection of vcpu pid is based on the assumption that vcpuid is monotonically increasing starting from 0 to max_cpus - 1. But the crosvm uses the apic id as the vcpuid if topology is exposed to the guest. And apicid can be non-contiguous.

This patch gets the max apicid from guest and then uses it to detect the vcpu pids. Even this approach is not fool proof an it assumes that apicid monotonically increases. If we encounter a scenario where it is not the case, we might need to implement a mechanism to get a map of cpuid to vcpu from the guest and then use that to detect the pid.

---
 .../include/private/trace-cmd-private.h       |  5 +-
 lib/trace-cmd/trace-msg.c                     | 12 ++--
 lib/trace-cmd/trace-util.c                    | 49 ++++++++++++--
 tracecmd/include/trace-local.h                |  1 +
 tracecmd/trace-agent.c                        | 11 ++--
 tracecmd/trace-record.c                       | 65 +++++++++++--------
 6 files changed, 98 insertions(+), 45 deletions(-)
lib/trace-cmd/include/private/trace-cmd-private.h +3 -2
index 766e0a7..49612b9 100644
@@ -434,12 +434,12 @@ int tracecmd_msg_recv_trace_req(struct tracecmd_msg_handle *msg_handle,
 				struct tracecmd_tsync_protos **protos);
 
 int tracecmd_msg_send_trace_resp(struct tracecmd_msg_handle *msg_handle,
-				 int nr_cpus, int page_size,
+				 int nr_cpus, int max_apicid, int page_size,
 				 unsigned int *ports, bool use_fifos,
 				 unsigned long long trace_id,
 				 const char *tsync_proto, unsigned int tsync_port);
 int tracecmd_msg_recv_trace_resp(struct tracecmd_msg_handle *msg_handle,
-				 int *nr_cpus, int *page_size,
+				 int *nr_cpus, int *max_apicid, int *page_size,
 				 unsigned int **ports, bool *use_fifos,
 				 unsigned long long *trace_id,
 				 char **tsync_proto,
@@ -595,6 +595,7 @@ int tracecmd_set_logfile(char *logfile);
 /* --- System --- */
 unsigned long long tracecmd_generate_traceid(void);
 int tracecmd_count_cpus(void);
+int tracecmd_count_cpus_apicid(int *max_apicid);
 
 /* --- Hack! --- */
 int tracecmd_blk_hack(struct tracecmd_input *handle);
lib/trace-cmd/trace-msg.c +8 -4
index 39465ad..1bb56a0 100644
@@ -73,6 +73,7 @@ struct tracecmd_msg_trace_req {
 struct tracecmd_msg_trace_resp {
 	be32 flags;
 	be32 cpus;
+	be32 max_apicid;
 	be32 page_size;
 	u64 trace_id;
 	char tsync_proto_name[TRACECMD_TSYNC_PNAME_LENGTH];
@@ -1295,7 +1296,8 @@ out:
 	return ret;
 }
 
-static int make_trace_resp(struct tracecmd_msg *msg, int page_size, int nr_cpus,
+static int make_trace_resp(struct tracecmd_msg *msg, int page_size,
+			   int nr_cpus, int max_apicid,
 			   unsigned int *ports, bool use_fifos,
 			   unsigned long long trace_id,
 			   const char *tsync_proto,
@@ -1319,6 +1321,7 @@ static int make_trace_resp(struct tracecmd_msg *msg, int page_size, int nr_cpus,
 	msg->trace_resp.tsync_port = htonl(tsync_port);
 
 	msg->trace_resp.cpus = htonl(nr_cpus);
+	msg->trace_resp.max_apicid = htonl(max_apicid);
 	msg->trace_resp.page_size = htonl(page_size);
 	msg->trace_resp.trace_id = htonll(trace_id);
 
@@ -1326,7 +1329,7 @@ static int make_trace_resp(struct tracecmd_msg *msg, int page_size, int nr_cpus,
 }
 
 int tracecmd_msg_send_trace_resp(struct tracecmd_msg_handle *msg_handle,
-				 int nr_cpus, int page_size,
+				 int nr_cpus, int max_apicid, int page_size,
 				 unsigned int *ports, bool use_fifos,
 				 unsigned long long trace_id,
 				 const char *tsync_proto, unsigned int tsync_port)
@@ -1335,7 +1338,7 @@ int tracecmd_msg_send_trace_resp(struct tracecmd_msg_handle *msg_handle,
 	int ret;
 
 	tracecmd_msg_init(MSG_TRACE_RESP, &msg);
-	ret = make_trace_resp(&msg, page_size, nr_cpus, ports,
+	ret = make_trace_resp(&msg, page_size, nr_cpus, max_apicid, ports,
 			      use_fifos, trace_id, tsync_proto, tsync_port);
 	if (ret < 0)
 		return ret;
@@ -1344,7 +1347,7 @@ int tracecmd_msg_send_trace_resp(struct tracecmd_msg_handle *msg_handle,
 }
 
 int tracecmd_msg_recv_trace_resp(struct tracecmd_msg_handle *msg_handle,
-				 int *nr_cpus, int *page_size,
+				 int *nr_cpus, int *max_apicid, int *page_size,
 				 unsigned int **ports, bool *use_fifos,
 				 unsigned long long *trace_id,
 				 char **tsync_proto,
@@ -1371,6 +1374,7 @@ int tracecmd_msg_recv_trace_resp(struct tracecmd_msg_handle *msg_handle,
 	}
 
 	*use_fifos = ntohl(msg.trace_resp.flags) & MSG_TRACE_USE_FIFOS;
+	*max_apicid = ntohl(msg.trace_resp.max_apicid);
 	*nr_cpus = ntohl(msg.trace_resp.cpus);
 	*page_size = ntohl(msg.trace_resp.page_size);
 	*trace_id = ntohll(msg.trace_resp.trace_id);
lib/trace-cmd/trace-util.c +42 -7

70 lines not on this page · raw message

tracecmd/include/trace-local.h +1 -0
index e3fec13..13064db 100644
@@ -278,6 +278,7 @@ struct buffer_instance {
 	int			tracing_on_fd;
 	int			buffer_size;
 	int			cpu_count;
+	int			max_apicid;
 
 	int			argc;
 	char			**argv;
tracecmd/trace-agent.c +6 -5
index f0723a6..81d57ec 100644
@@ -122,7 +122,8 @@ static void trace_print_connection(int fd, const char *network)
 		tracecmd_debug("Could not print connection fd:%d\n", fd);
 }
 
-static void agent_handle(int sd, int nr_cpus, int page_size, const char *network)
+static void agent_handle(int sd, int nr_cpus, int max_apicid,
+			 int page_size, const char *network)
 {
 	struct tracecmd_tsync_protos *tsync_protos = NULL;
 	struct tracecmd_time_sync *tsync = NULL;
@@ -197,7 +198,7 @@ static void agent_handle(int sd, int nr_cpus, int page_size, const char *network
 		}
 	}
 	trace_id = tracecmd_generate_traceid();
-	ret = tracecmd_msg_send_trace_resp(msg_handle, nr_cpus, page_size,
+	ret = tracecmd_msg_send_trace_resp(msg_handle, nr_cpus, max_apicid, page_size,
 					   ports, use_fifos, trace_id,
 					   tsync_proto, tsync_port);
 	if (ret < 0)
@@ -255,7 +256,7 @@ static void agent_serve(unsigned int port, bool do_daemon, const char *network)
 	struct sockaddr *addr = NULL;
 	socklen_t *addr_len_p = NULL;
 	socklen_t addr_len = sizeof(net_addr);
-	int sd, cd, nr_cpus;
+	int sd, cd, nr_cpus, max_apicid;
 	unsigned int cid;
 	pid_t pid;
 
@@ -266,7 +267,7 @@ static void agent_serve(unsigned int port, bool do_daemon, const char *network)
 		addr_len_p = &addr_len;
 	}
 
-	nr_cpus = tracecmd_count_cpus();
+	nr_cpus = tracecmd_count_cpus_apicid(&max_apicid);
 	page_size = getpagesize();
 
 	if (network) {
@@ -311,7 +312,7 @@ static void agent_serve(unsigned int port, bool do_daemon, const char *network)
 		if (pid == 0) {
 			close(sd);
 			signal(SIGCHLD, SIG_DFL);
-			agent_handle(cd, nr_cpus, page_size, network);
+			agent_handle(cd, nr_cpus, max_apicid, page_size, network);
 		}
 		if (pid > 0)
 			handler_pid = pid;
tracecmd/trace-record.c +38 -27

144 lines not on this page · raw message

Signature · 1 line
2.36.0.464.gb9c8b46e94-goog
#1 lore raw
Quoted 12 lines · On Wed, 4 May 2022 01:02:42 +0000 Vineeth Pillai <[email protected]> wrote:

/* --- System --- */
unsigned long long tracecmd_generate_traceid(void); int tracecmd_count_cpus(void);

lib/trace-cmd/include/private/trace-cmd-private.h +1 -0
+int tracecmd_count_cpus_apicid(int *max_apicid);
 
 /* --- Hack! --- */
 int tracecmd_blk_hack(struct tracecmd_input *handle);
lib/trace-cmd/trace-msg.c +0 -0
index 39465ad..1bb56a0 100644

lib/trace-cmd/trace-msg.c struct tracecmd_msg_trace_req { · line 79

Quoted 7 lines
lib/trace-cmd/trace-msg.c +1 -0
@@ -73,6 +73,7 @@ struct tracecmd_msg_trace_req {
 struct tracecmd_msg_trace_resp {
 	be32 flags;
 	be32 cpus;
+	be32 max_apicid;
 	be32 page_size;
 	u64 trace_id;
 	char tsync_proto_name[TRACECMD_TSYNC_PNAME_LENGTH];

Unfortunately, the above will break the protocol for released instances of trace-cmd that is already out there. One requirement I have is that if two instances of trace-cmd use to work (on host and guest) that if you upgrade one of them, what use to work still does.

We need to figure out another way to handle this :-/

Quoted 12 lines

static void stop_mapping_vcpus(struct buffer_instance *instance,

lib/trace-cmd/trace-msg.c +1 -1
@@ -3961,21 +3959,19 @@ static void stop_mapping_vcpus(struct buffer_instance *instance,
 	struct trace_mapping tmap = { };
 	struct tep_handle *tep;
 	const char *systems[] = { "kvm", NULL };
-	int i;
 
 	if (!guest->instance)
 		return;
 
 	tmap.pids = guest->task_pids;
+	tmap.nr_cpus = 0;

lib/trace-cmd/trace-msg.c static void stop_mapping_vcpus(struct buffer_instance *instance, · line 3977

Quoted 10 lines
lib/trace-cmd/trace-msg.c +2 -3
 	tmap.max_cpus = instance->cpu_count;
+	tmap.max_apicid = instance->max_apicid;
 
-	tmap.map = malloc(sizeof(*tmap.map) * tmap.max_cpus);
+	tmap.map = calloc((tmap.max_apicid + 1), sizeof(*tmap.map));
 	if (!tmap.map)
 		return;
 
-	for (i = 0; i < tmap.max_cpus; i++)
-		tmap.map[i] = -1;

Note, we wrote this with -1, because we envisioned working with hypervisors that may not be Linux, and that PID = 0 may be legitimate.

lib/trace-cmd/trace-msg.c static void stop_mapping_vcpus(struct buffer_instance *instance, · line 3977

Quoted 4 lines
lib/trace-cmd/trace-msg.c +0 -1
-
 	tracefs_instance_file_write(guest->instance, "events/kvm/kvm_entry/enable", "0");
 
 	tep = tracefs_local_events_system(NULL, systems);

We'll have to investigate a solution for this further.

-- Steve

#2 lore raw
Quoted 6 lines · Steven Rostedt
Quoted 5 lines
+     be32 max_apicid;
      be32 page_size;
      u64 trace_id;
      char tsync_proto_name[TRACECMD_TSYNC_PNAME_LENGTH];
Quoted 7 lines

Unfortunately, the above will break the protocol for released instances of trace-cmd that is already out there. One requirement I have is that if two instances of trace-cmd use to work (on host and guest) that if you upgrade one of them, what use to work still does.

We need to figure out another way to handle this :-/

Ahh ok, understood..
I think we should also think about versioning the guest/host protocols because this kind of situation may arise in future where we would need to redesign or upgrade the protocol and trace-cmd should then be able to show meaningful message to user to upgrade the binary. We might also be hitting protocol bugs as well down the lane and versioning would become necessary.

Quoted 6 lines

tracecmd/trace-record.c static void stop_mapping_vcpus(struct buffer_instance *instance, · line 3977

Quoted 2 lines
tracecmd/trace-record.c +0 -2
-     for (i = 0; i < tmap.max_cpus; i++)
-             tmap.map[i] = -1;

Note, we wrote this with -1, because we envisioned working with hypervisors that may not be Linux, and that PID = 0 may be legitimate.

Thanks for the clarification.

Quoted 1 line

We'll have to investigate a solution for this further.

Sure, I will have a look at this again and see if/how we can fix this without touching existing protocol. I was thinking about adding the pid details to debugfs and then reading it from there. Its not optimal as we would need a supporting kernel, but it would be accurate if we have kernel support and we need not rely on trace messages to get the pids.

Thanks,
Vineeth

#3 lore raw
Quoted 12 lines · On Mon, 23 May 2022 09:39:45 -0400 Vineeth Pillai <[email protected]> wrote:
Quoted 11 lines · Steven Rostedt
Quoted 5 lines
 
+     be32 max_apicid;
      be32 page_size;
      u64 trace_id;
      char tsync_proto_name[TRACECMD_TSYNC_PNAME_LENGTH];  

Unfortunately, the above will break the protocol for released instances of trace-cmd that is already out there. One requirement I have is that if two instances of trace-cmd use to work (on host and guest) that if you upgrade one of them, what use to work still does.

Quoted 4 lines
Quoted 2 lines

We need to figure out another way to handle this :-/

Ahh ok, understood..
I think we should also think about versioning the guest/host protocols

We actually have that. We are currently at version 3. If this becomes too much of an issue, we can always move to v4. But to do that would require a lot more thinking, as I like to support all versions, and I don't want to have a lot of them. Thus, if we need to update to v4. It would likely be a entirely new protocol to fix and extend the issues of v3.

Quoted 7 lines
Quoted 1 line

We'll have to investigate a solution for this further.

Sure, I will have a look at this again and see if/how we can fix this without touching existing protocol. I was thinking about adding the pid details to debugfs and then reading it from there. Its not optimal as we would need a supporting kernel, but it would be accurate if we have kernel support and we need not rely on trace messages to get the pids.

That would be useful too, in more than just trace-cmd I believe.

-- Steve

#4 lore raw
Quoted 9 lines · Steven Rostedt
Quoted 2 lines

I think we should also think about versioning the guest/host protocols

We actually have that. We are currently at version 3. If this becomes too much of an issue, we can always move to v4. But to do that would require a lot more thinking, as I like to support all versions, and I don't want to have a lot of them. Thus, if we need to update to v4. It would likely be a entirely new protocol to fix and extend the issues of v3.

Thanks for the details.. Yeah, doesn't make sense to have a version change for this.

Quoted 8 lines
Quoted 5 lines

Sure, I will have a look at this again and see if/how we can fix this without touching existing protocol. I was thinking about adding the pid details to debugfs and then reading it from there. Its not optimal as we would need a supporting kernel, but it would be accurate if we have kernel support and we need not rely on trace messages to get the pids.

That would be useful too, in more than just trace-cmd I believe.

Posted a patch upstream for exposing pid in debugfs:
https://www.spinics.net/lists/kvm/msg278146.html

I shall work on trace-cmd changes as well once the above patch is finalized..

Thanks,
Vineeth

#5 lore raw
Quoted 4 lines · On Tue, 24 May 2022 11:35:40 -0400 Vineeth Pillai <[email protected]> wrote:
Quoted 3 lines · Steven Rostedt
Quoted 2 lines

I think we should also think about versioning the guest/host protocols

Quoted 8 lines
Quoted 6 lines

We actually have that. We are currently at version 3. If this becomes too much of an issue, we can always move to v4. But to do that would require a lot more thinking, as I like to support all versions, and I don't want to have a lot of them. Thus, if we need to update to v4. It would likely be a entirely new protocol to fix and extend the issues of v3.

Thanks for the details.. Yeah, doesn't make sense to have a version change for this.

But there may be other reasons to update to v4 ;-)

Quoted 11 lines
Quoted 9 lines
Quoted 6 lines

Sure, I will have a look at this again and see if/how we can fix this without touching existing protocol. I was thinking about adding the pid details to debugfs and then reading it from there. Its not optimal as we would need a supporting kernel, but it would be accurate if we have kernel support and we need not rely on trace messages to get the pids.

That would be useful too, in more than just trace-cmd I believe.

Posted a patch upstream for exposing pid in debugfs:
https://www.spinics.net/lists/kvm/msg278146.html

Saw that.

Quoted 3 lines

I shall work on trace-cmd changes as well once the above patch is finalized..

Thanks,

-- Steve

#6 lore raw
In reply to Vineeth Pillai
Quoted 5 lines · On Wed, 4 May 2022 01:02:42 +0000 Vineeth Pillai <[email protected]> wrote:

The current detection of vcpu pid is based on the assumption that vcpuid is monotonically increasing starting from 0 to max_cpus - 1. But the crosvm uses the apic id as the vcpuid if topology is exposed to the guest. And apicid can be non-contiguous.

Quoted 7 lines

This patch gets the max apicid from guest and then uses it to detect the vcpu pids. Even this approach is not fool proof an it assumes that apicid monotonically increases. If we encounter a scenario where it is not the case, we might need to implement a mechanism to get a map of cpuid to vcpu from the guest and then use that to detect the pid.

If you haven't noticed I have patches that are a workaround that fixes this issue if the conditions are "right". (I Cc'd you) [1]

The work around is to not depend on the number in debugfs/kvm/vcpuX to match the CPU of the guest, but instead to match the ordering of the CPUs of the guest. It also requires that the guest's CPUs are consecutive and non sparse. That is, if you have vcpu0, vcpu1, vcpu8 and vcpu9 on the host, then the guest needs to have CPU 0-3 with the mapping of:

vcpu0 -> CPU 0
vcpu1 -> CPU 1
vcpu8 -> CPU 2
vcpu9 -> CPU 3

and then it works. Of course if this is not true, then it breaks again and will require some more communication between the host and the guest.

[1] https://lore.kernel.org/all/[email protected]/

#7 lore raw

Hi Steven,

Quoted 3 lines · Steven Rostedt

If you haven't noticed I have patches that are a workaround that fixes this issue if the conditions are "right". (I Cc'd you) [1]

I was on vacation and just combing through the mails :-). I shall have a look at the patch soon..

Quoted 7 lines

The work around is to not depend on the number in debugfs/kvm/vcpuX to match the CPU of the guest, but instead to match the ordering of the CPUs of the guest. It also requires that the guest's CPUs are consecutive and non sparse. That is, if you have vcpu0, vcpu1, vcpu8 and vcpu9 on the host, then the guest needs to have CPU 0-3 with the mapping of:

Quoted 7 lines

vcpu0 -> CPU 0
vcpu1 -> CPU 1
vcpu8 -> CPU 2
vcpu9 -> CPU 3

and then it works. Of course if this is not true, then it breaks again and will require some more communication between the host and the guest.

This makes sense.

I vaguely remember seeing non-increasing patterns in some server machines, but that's really rare I guess. When the debugfs changes for exposing pid are widely available, we can have that also as an option.

Thanks for the patches. I shall have a look at the patches soon.

~Vineeth