Skip to contentSkip to conversation

perf: Correlating user process data to samples

RFC PATCH In review Beau Belgrave 6 participants 13 messages Apr 12, 2024 – Apr 18, 2024

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover perf: Correlating user process data to samples

In the Open Telemetry profiling SIG [1], we are trying to find a way to grab a tracing association quickly on a per-sample basis. The team at Elastic has a bespoke way to do this [2], however, I'd like to see a more general way to achieve this. The folks I've been talking with seem open to the idea of just having a TLS value for this we could capture upon each sample. We could then just state, Open Telemetry SDKs should have a TLS value for span correlation. However, we need a way to sample the TLS or other value(s) when a sampling event is generated. This is supported today on Windows via EventActivityIdControl() [3]. Since Open Telemetry works on both Windows and Linux, ideally we can do something as efficient for Linux based workloads.

This series is to explore how it would be best possible to collect supporting data from a user process when a profile sample is collected. Having a value stored in TLS makes a lot of sense for this however there are other ways to explore. Whatever is chosen, kernel samples taken in process context should be able to get this supporting data. In these patches on X64 the fsbase and gsbase are used for this.

An option to explore suggested by Mathieu Desnoyers is to utilize rseq for processes to register a value location that can be included when profiling if desired. This would allow a tighter contract between user processes and a profiler. It would allow better labeling/categorizing the correlation values.

An idea flow would look like this:
User Task Profile
do_work(); sample() -> IP + No activity
...
set_activity(123);
...
do_work(); sample() -> IP + activity (123)
...
set_activity(124);
...
do_work(); sample() -> IP + activity (124)

Ideally, the set_activity() method would not be a syscall. It needs to be very cheap as this should not bottleneck work. Ideally this is just a memcpy of 16-20 bytes as it is on Windows via EventActivityIdControl() using EVENT_ACTIVITY_CTRL_SET_ID.

For those not aware, Open Telemetry allows collecting data from multiple machines and show where time was spent. The tracing context is already available for logs, but not for profiling samples. The idea is to show where slowdowns occur and have profile samples to explain why they slowed down. This must be possible without having to track context switches to do this correlation. This is because the profiling rates are typically 20hz - 1Khz, while the context switching rates are much higher. We do not want to have to consume high context switch rates just to know a correlation for a 20hz signal. Often these 20hz signals are always enabled in some environments.

Regardless if TLS, rseq, or other source is used I believe we will need a way for perf_events to include it within a sample. The changes in this series show how it could be done with TLS. There is some factoring work under perf to make it easier to add more dump types using the existing ABI. This is mostly to make the patches clearer, certainly the refactor parts could get dropped and we could have duplicated/specialized paths.

1. https://opentelemetry.io/blog/2024/profiling/
2. https://www.elastic.co/blog/continuous-profiling-distributed-tracing-correlation
3. https://learn.microsoft.com/en-us/windows/win32/api/evntprov/nf-evntprov-eventactivityidcontrol

Beau Belgrave (4):
perf/core: Introduce perf_prepare_dump_data()
perf: Introduce PERF_SAMPLE_TLS_USER sample type perf/core: Factor perf_output_sample_udump() perf/x86/core: Add tls dump support

 arch/Kconfig                      |   7 ++
 arch/x86/Kconfig                  |   1 +
 arch/x86/events/core.c            |  14 +++
 arch/x86/include/asm/perf_event.h |   5 +
 include/linux/perf_event.h        |   7 ++
 include/uapi/linux/perf_event.h   |   5 +-
 kernel/events/core.c              | 166 +++++++++++++++++++++++-------
 kernel/events/internal.h          |  16 +++
 8 files changed, 180 insertions(+), 41 deletions(-)

base-commit: fec50db7033ea478773b159e0e2efb135270e3b7

Signature · 1 line
2.34.1
#1 lore raw

RFC PATCH 1/4 perf/core: Introduce perf_prepare_dump_data()

Factor out perf_prepare_dump_data() so that the same logic is used for dumping stack data as other types, such as TLS.

Slightly refactor perf_sample_ustack_size() to perf_sample_dump_size(). Move reg checks up into perf_ustack_task_size() since the task size must now be calculated before preparing dump data.

---
 kernel/events/core.c | 79 ++++++++++++++++++++++++++------------------
 1 file changed, 47 insertions(+), 32 deletions(-)
kernel/events/core.c +47 -32
index 724e6d7e128f..07de5cc2aa25 100644
@@ -6912,7 +6912,13 @@ static void perf_sample_regs_intr(struct perf_regs *regs_intr,
  */
 static u64 perf_ustack_task_size(struct pt_regs *regs)
 {
-	unsigned long addr = perf_user_stack_pointer(regs);
+	unsigned long addr;
+
+	/* No regs, no stack pointer, no dump. */
+	if (!regs)
+		return 0;
+
+	addr = perf_user_stack_pointer(regs);
 
 	if (!addr || addr >= TASK_SIZE)
 		return 0;
@@ -6921,42 +6927,35 @@ static u64 perf_ustack_task_size(struct pt_regs *regs)
 }
 
 static u16
-perf_sample_ustack_size(u16 stack_size, u16 header_size,
-			struct pt_regs *regs)
+perf_sample_dump_size(u16 dump_size, u16 header_size, u64 task_size)
 {
-	u64 task_size;
-
-	/* No regs, no stack pointer, no dump. */
-	if (!regs)
-		return 0;
-
 	/*
-	 * Check if we fit in with the requested stack size into the:
+	 * Check if we fit in with the requested dump size into the:
 	 * - TASK_SIZE
 	 *   If we don't, we limit the size to the TASK_SIZE.
 	 *
 	 * - remaining sample size
-	 *   If we don't, we customize the stack size to
+	 *   If we don't, we customize the dump size to
 	 *   fit in to the remaining sample size.
 	 */
 
-	task_size  = min((u64) USHRT_MAX, perf_ustack_task_size(regs));
-	stack_size = min(stack_size, (u16) task_size);
+	task_size  = min((u64) USHRT_MAX, task_size);
+	dump_size = min(dump_size, (u16) task_size);
 
 	/* Current header size plus static size and dynamic size. */
 	header_size += 2 * sizeof(u64);
 
-	/* Do we fit in with the current stack dump size? */
-	if ((u16) (header_size + stack_size) < header_size) {
+	/* Do we fit in with the current dump size? */
+	if ((u16) (header_size + dump_size) < header_size) {
 		/*
 		 * If we overflow the maximum size for the sample,
-		 * we customize the stack dump size to fit in.
+		 * we customize the dump size to fit in.
 		 */
-		stack_size = USHRT_MAX - header_size - sizeof(u64);
-		stack_size = round_up(stack_size, sizeof(u64));
+		dump_size = USHRT_MAX - header_size - sizeof(u64);
+		dump_size = round_up(dump_size, sizeof(u64));
 	}
 
-	return stack_size;
+	return dump_size;
 }
 
 static void
@@ -7648,6 +7647,32 @@ static __always_inline u64 __cond_set(u64 flags, u64 s, u64 d)
 	return d * !!(flags & s);
 }
 
+static inline u16
+perf_prepare_dump_data(struct perf_sample_data *data,
+		       struct perf_event *event,
+		       struct pt_regs *regs,
+		       u16 dump_size,
+		       u64 task_size)
+{
+	u16 header_size = perf_sample_data_size(data, event);
+	u16 size = sizeof(u64);
+
+	dump_size = perf_sample_dump_size(dump_size, header_size,
+					  task_size);
+
+	/*
+	 * If there is something to dump, add space for the dump
+	 * itself and for the field that tells the dynamic size,
+	 * which is how many have been actually dumped.
+	 */
+	if (dump_size)
+		size += sizeof(u64) + dump_size;
+
+	data->dyn_size += size;
+
+	return dump_size;
+}
+
 void perf_prepare_sample(struct perf_sample_data *data,
 			 struct perf_event *event,
 			 struct pt_regs *regs)
@@ -7725,22 +7750,12 @@ void perf_prepare_sample(struct perf_sample_data *data,
 		 * up the rest of the sample size.
 		 */
 		u16 stack_size = event->attr.sample_stack_user;
-		u16 header_size = perf_sample_data_size(data, event);
-		u16 size = sizeof(u64);
-
-		stack_size = perf_sample_ustack_size(stack_size, header_size,
-						     data->regs_user.regs);
+		u64 task_size = perf_ustack_task_size(regs);
 
-		/*
-		 * If there is something to dump, add space for the dump
-		 * itself and for the field that tells the dynamic size,
-		 * which is how many have been actually dumped.
-		 */
-		if (stack_size)
-			size += sizeof(u64) + stack_size;
+		stack_size = perf_prepare_dump_data(data, event, regs,
+						    stack_size, task_size);
 
 		data->stack_user_size = stack_size;
-		data->dyn_size += size;
 		data->sample_flags |= PERF_SAMPLE_STACK_USER;
 	}
 
Signature · 1 line
2.34.1
#2 lore raw

RFC PATCH 2/4 perf: Introduce PERF_SAMPLE_TLS_USER sample type

In reply to Beau Belgrave

When samples are generated, there is no way via the perf_event ABI to fetch per-thread data. This data is very useful in tracing scenarios that involve correlation IDs, such as OpenTelemetry. They are also useful for tracking per-thread performance details directly within a cooperating user process.

The newly establish OpenTelemetry profiling group requires a way to get tracing correlations on both Linux and Windows. On Windows this correlation is on a per-thread basis directly via ETW. On Linux we need a fast mechanism to store these details and TLS seems like the best option, see links for more details.

Add a new sample type (PERF_SAMPLE_TLS_USER) that fetches TLS data up to X bytes per-sample. Use the existing PERF_SAMPLE_STACK_USER ABI for outputting data out to consumers. Store requested data size by the user in the previously reserved u16 (__reserved_2) within perf_event_attr.

Add tls_addr and tls_user_size to perf_sample_data and calculate them during sample preparation. This allows the output side to know if truncation is going to occur and not having to re-fetch the TLS value from the user process a second time.

Add CONFIG_HAVE_PERF_USER_TLS_DUMP so that architectures can specify if they have a TLS specific register (or other logic) that can be used for dumping. This does not yet enable any architecture to do TLS dump, it simply makes it possible by allowing a arch defined method named arch_perf_user_tls_pointer().

Add perf_tls struct that arch_perf_user_tls_pointer() utilizes to set TLS details of the address and size (for 32bit on 64bit compat cases).

---
 arch/Kconfig                    |   7 +++
 include/linux/perf_event.h      |   7 +++
 include/uapi/linux/perf_event.h |   5 +-
 kernel/events/core.c            | 105 +++++++++++++++++++++++++++++++-
 kernel/events/internal.h        |  16 +++++
 5 files changed, 137 insertions(+), 3 deletions(-)
arch/Kconfig +7 -0
index 9f066785bb71..6afaf5f46e2f 100644
@@ -430,6 +430,13 @@ config HAVE_PERF_USER_STACK_DUMP
 	  access to the user stack pointer which is not unified across
 	  architectures.
 
+config HAVE_PERF_USER_TLS_DUMP
+	bool
+	help
+	  Support user tls dumps for perf event samples. This needs
+	  access to the user tls pointer which is not unified across
+	  architectures.
+
 config HAVE_ARCH_JUMP_LABEL
 	bool
 
include/linux/perf_event.h +7 -0
index d2a15c0c6f8a..7fac81929eed 100644
@@ -1202,8 +1202,15 @@ struct perf_sample_data {
 	u64				data_page_size;
 	u64				code_page_size;
 	u64				aux_size;
+	u64				tls_addr;
+	u64				tls_user_size;
 } ____cacheline_aligned;
 
+struct perf_tls {
+	unsigned long base; /* Base address for TLS */
+	unsigned long size; /* Size of base address */
+};
+
 /* default value for data source */
 #define PERF_MEM_NA (PERF_MEM_S(OP, NA)   |\
 		    PERF_MEM_S(LVL, NA)   |\
include/uapi/linux/perf_event.h +3 -2
index 3a64499b0f5d..b62669cfe581 100644
@@ -162,8 +162,9 @@ enum perf_event_sample_format {
 	PERF_SAMPLE_DATA_PAGE_SIZE		= 1U << 22,
 	PERF_SAMPLE_CODE_PAGE_SIZE		= 1U << 23,
 	PERF_SAMPLE_WEIGHT_STRUCT		= 1U << 24,
+	PERF_SAMPLE_TLS_USER			= 1U << 25,
 
-	PERF_SAMPLE_MAX = 1U << 25,		/* non-ABI */
+	PERF_SAMPLE_MAX = 1U << 26,		/* non-ABI */
 };
 
 #define PERF_SAMPLE_WEIGHT_TYPE	(PERF_SAMPLE_WEIGHT | PERF_SAMPLE_WEIGHT_STRUCT)
@@ -509,7 +510,7 @@ struct perf_event_attr {
 	 */
 	__u32	aux_watermark;
 	__u16	sample_max_stack;
-	__u16	__reserved_2;
+	__u16	sample_tls_user; /* Size of TLS data to dump on samples */
 	__u32	aux_sample_size;
 	__u32	__reserved_3;
 
kernel/events/core.c +104 -1

141 lines not on this page · raw message

kernel/events/internal.h +16 -0
index 5150d5f84c03..b42747b1eb04 100644
@@ -243,4 +243,20 @@ static inline bool arch_perf_have_user_stack_dump(void)
 #define perf_user_stack_pointer(regs) 0
 #endif /* CONFIG_HAVE_PERF_USER_STACK_DUMP */
 
+#ifdef CONFIG_HAVE_PERF_USER_TLS_DUMP
+static inline bool arch_perf_have_user_tls_dump(void)
+{
+	return true;
+}
+
+#define perf_user_tls_pointer(tls) arch_perf_user_tls_pointer(tls)
+#else
+static inline bool arch_perf_have_user_tls_dump(void)
+{
+	return false;
+}
+
+#define perf_user_tls_pointer(tls) memset(tls, 0, sizeof(*tls))
+#endif /* CONFIG_HAVE_PERF_USER_TLS_DUMP */
+
 #endif /* _KERNEL_EVENTS_INTERNAL_H */
Signature · 1 line
2.34.1
#3 lore raw

RFC PATCH 3/4 perf/core: Factor perf_output_sample_udump()

In reply to Beau Belgrave

We now have two user dump sources (stack and tls). Both are doing the same logic to ensure the user dump ABI output is properly handled. The only difference is one gets the address within the method, and the other is passed the address.

Add perf_output_sample_udump() and utilize it for both stack and tls sample dumps. The sp register is now fetched outside of this method and passed to it. This allows both stack and tls to utilize the same code.

---
 kernel/events/core.c | 68 +++++++++++++-------------------------------
 1 file changed, 19 insertions(+), 49 deletions(-)
kernel/events/core.c +19 -49
index f848bf4be9bd..6b3cf5afdd32 100644
@@ -6998,47 +6998,10 @@ perf_sample_dump_size(u16 dump_size, u16 header_size, u64 task_size)
 }
 
 static void
-perf_output_sample_ustack(struct perf_output_handle *handle, u64 dump_size,
-			  struct pt_regs *regs)
-{
-	/* Case of a kernel thread, nothing to dump */
-	if (!regs) {
-		u64 size = 0;
-		perf_output_put(handle, size);
-	} else {
-		unsigned long sp;
-		unsigned int rem;
-		u64 dyn_size;
-
-		/*
-		 * We dump:
-		 * static size
-		 *   - the size requested by user or the best one we can fit
-		 *     in to the sample max size
-		 * data
-		 *   - user stack dump data
-		 * dynamic size
-		 *   - the actual dumped size
-		 */
-
-		/* Static size. */
-		perf_output_put(handle, dump_size);
-
-		/* Data. */
-		sp = perf_user_stack_pointer(regs);
-		rem = __output_copy_user(handle, (void *) sp, dump_size);
-		dyn_size = dump_size - rem;
-
-		perf_output_skip(handle, rem);
-
-		/* Dynamic size. */
-		perf_output_put(handle, dyn_size);
-	}
-}
-
-static void
-perf_output_sample_utls(struct perf_output_handle *handle, u64 addr,
-			u64 dump_size, struct pt_regs *regs)
+perf_output_sample_udump(struct perf_output_handle *handle,
+			 unsigned long addr,
+			 u64 dump_size,
+			 struct pt_regs *regs)
 {
 	/* Case of a kernel thread, nothing to dump */
 	if (!regs) {
@@ -7054,7 +7017,7 @@ perf_output_sample_utls(struct perf_output_handle *handle, u64 addr,
 		 *   - the size requested by user or the best one we can fit
 		 *     in to the sample max size
 		 * data
-		 *   - user tls dump data
+		 *   - user dump data
 		 * dynamic size
 		 *   - the actual dumped size
 		 */
@@ -7507,9 +7470,16 @@ void perf_output_sample(struct perf_output_handle *handle,
 	}
 
 	if (sample_type & PERF_SAMPLE_STACK_USER) {
-		perf_output_sample_ustack(handle,
-					  data->stack_user_size,
-					  data->regs_user.regs);
+		struct pt_regs *regs = data->regs_user.regs;
+		unsigned long sp = 0;
+
+		if (regs)
+			sp = perf_user_stack_pointer(regs);
+
+		perf_output_sample_udump(handle,
+					 sp,
+					 data->stack_user_size,
+					 regs);
 	}
 
 	if (sample_type & PERF_SAMPLE_WEIGHT_TYPE)
@@ -7551,10 +7521,10 @@ void perf_output_sample(struct perf_output_handle *handle,
 		perf_output_put(handle, data->code_page_size);
 
 	if (sample_type & PERF_SAMPLE_TLS_USER) {
-		perf_output_sample_utls(handle,
-					data->tls_addr,
-					data->tls_user_size,
-					data->regs_user.regs);
+		perf_output_sample_udump(handle,
+					 data->tls_addr,
+					 data->tls_user_size,
+					 data->regs_user.regs);
 	}
 
 	if (sample_type & PERF_SAMPLE_AUX) {
Signature · 1 line
2.34.1
#4 lore raw

RFC PATCH 4/4 perf/x86/core: Add tls dump support

In reply to Beau Belgrave

Now that perf supports TLS dumps, x86-64 can provide the details for how to get TLS data for user threads.

Enable HAVE_PERF_USER_TLS_DUMP Kconfig only for x86-64. I do not have access to x86 to validate 32-bit.

Utilize mmap_is_ia32() to determine 32/64 bit threads. Use fsbase for 64-bit and gsbase for 32-bit with appropriate size.

---
 arch/x86/Kconfig                  |  1 +
 arch/x86/events/core.c            | 14 ++++++++++++++
 arch/x86/include/asm/perf_event.h |  5 +++++
 3 files changed, 20 insertions(+)
arch/x86/Kconfig +1 -0
index 4fff6ed46e90..8d46ec8ded0c 100644
@@ -263,6 +263,7 @@ config X86
 	select HAVE_PCI
 	select HAVE_PERF_REGS
 	select HAVE_PERF_USER_STACK_DUMP
+	select HAVE_PERF_USER_TLS_DUMP		if X86_64
 	select MMU_GATHER_RCU_TABLE_FREE	if PARAVIRT
 	select MMU_GATHER_MERGE_VMAS
 	select HAVE_POSIX_CPU_TIMERS_TASK_WORK
arch/x86/events/core.c +14 -0
index 09050641ce5d..3f851db4c591 100644
@@ -41,6 +41,7 @@
 #include <asm/desc.h>
 #include <asm/ldt.h>
 #include <asm/unwind.h>
+#include <asm/elf.h>
 
 #include "perf_event.h"
 
@@ -3002,3 +3003,16 @@ u64 perf_get_hw_event_config(int hw_event)
 	return 0;
 }
 EXPORT_SYMBOL_GPL(perf_get_hw_event_config);
+
+#ifdef CONFIG_X86_64
+void arch_perf_user_tls_pointer(struct perf_tls *tls)
+{
+	if (!mmap_is_ia32()) {
+		tls->base = current->thread.fsbase;
+		tls->size = sizeof(u64);
+	} else {
+		tls->base = current->thread.gsbase;
+		tls->size = sizeof(u32);
+	}
+}
+#endif
arch/x86/include/asm/perf_event.h +5 -0
index 3736b8a46c04..d0f65e572c20 100644
@@ -628,4 +628,9 @@ static __always_inline void perf_lopwr_cb(bool lopwr_in)
 
 #define arch_perf_out_copy_user copy_from_user_nmi
 
+#ifdef CONFIG_HAVE_PERF_USER_TLS_DUMP
+struct perf_tls;
+extern void arch_perf_user_tls_pointer(struct perf_tls *tls);
+#endif
+
 #endif /* _ASM_X86_PERF_EVENT_H */
Signature · 1 line
2.34.1
reviewer
#5 lore raw
In reply to Beau Belgrave
Quoted 6 lines · Beau Belgrave

In the Open Telemetry profiling SIG [1], we are trying to find a way to grab a tracing association quickly on a per-sample basis. The team at Elastic has a bespoke way to do this [2], however, I'd like to see a more general way to achieve this. The folks I've been talking with seem open to the idea of just having a TLS value for this we could capture

Presumably TLS == Thread Local Storage.

Quoted 14 lines

upon each sample. We could then just state, Open Telemetry SDKs should have a TLS value for span correlation. However, we need a way to sample the TLS or other value(s) when a sampling event is generated. This is supported today on Windows via EventActivityIdControl() [3]. Since Open Telemetry works on both Windows and Linux, ideally we can do something as efficient for Linux based workloads.

This series is to explore how it would be best possible to collect supporting data from a user process when a profile sample is collected. Having a value stored in TLS makes a lot of sense for this however there are other ways to explore. Whatever is chosen, kernel samples taken in process context should be able to get this supporting data. In these patches on X64 the fsbase and gsbase are used for this.

Quoted 5 lines

An option to explore suggested by Mathieu Desnoyers is to utilize rseq for processes to register a value location that can be included when profiling if desired. This would allow a tighter contract between user processes and a profiler. It would allow better labeling/categorizing the correlation values.

It is hard to understand this idea. Are you saying stash a cookie in TLS for samples to capture to indicate an activity? Restartable sequences are about preemption on a CPU not of a thread, so at least my intuition is that they feel different. You could stash information like this today by changing the thread name which generates comm events. I've wondered about having similar information in some form of reserved for profiling stack slot, for example, to stash a pointer to the name of a function being interpreted. Snapshotting all of a stack is bad performance wise and for security. A stack slot would be able to deal with nesting.

Quoted 28 lines

An idea flow would look like this:
User Task Profile
do_work(); sample() -> IP + No activity
...
set_activity(123);
...
do_work(); sample() -> IP + activity (123)
...
set_activity(124);
...
do_work(); sample() -> IP + activity (124)

Ideally, the set_activity() method would not be a syscall. It needs to be very cheap as this should not bottleneck work. Ideally this is just a memcpy of 16-20 bytes as it is on Windows via EventActivityIdControl() using EVENT_ACTIVITY_CTRL_SET_ID.

For those not aware, Open Telemetry allows collecting data from multiple machines and show where time was spent. The tracing context is already available for logs, but not for profiling samples. The idea is to show where slowdowns occur and have profile samples to explain why they slowed down. This must be possible without having to track context switches to do this correlation. This is because the profiling rates are typically 20hz - 1Khz, while the context switching rates are much higher. We do not want to have to consume high context switch rates just to know a correlation for a 20hz signal. Often these 20hz signals are always enabled in some environments.

Quoted 6 lines

Regardless if TLS, rseq, or other source is used I believe we will need a way for perf_events to include it within a sample. The changes in this series show how it could be done with TLS. There is some factoring work under perf to make it easier to add more dump types using the existing ABI. This is mostly to make the patches clearer, certainly the refactor parts could get dropped and we could have duplicated/specialized paths.

fs and gs may be used for more than just the C runtime's TLS. For example, they may be used by emulators or managed runtimes. I'm not clear why this specific case couldn't be handled through BPF.

Thanks,
Ian

Quoted 25 lines

1. https://opentelemetry.io/blog/2024/profiling/
2. https://www.elastic.co/blog/continuous-profiling-distributed-tracing-correlation
3. https://learn.microsoft.com/en-us/windows/win32/api/evntprov/nf-evntprov-eventactivityidcontrol

Beau Belgrave (4):
perf/core: Introduce perf_prepare_dump_data()
perf: Introduce PERF_SAMPLE_TLS_USER sample type perf/core: Factor perf_output_sample_udump() perf/x86/core: Add tls dump support

 arch/Kconfig                      |   7 ++
 arch/x86/Kconfig                  |   1 +
 arch/x86/events/core.c            |  14 +++
 arch/x86/include/asm/perf_event.h |   5 +
 include/linux/perf_event.h        |   7 ++
 include/uapi/linux/perf_event.h   |   5 +-
 kernel/events/core.c              | 166 +++++++++++++++++++++++-------
 kernel/events/internal.h          |  16 +++
 8 files changed, 180 insertions(+), 41 deletions(-)

base-commit: fec50db7033ea478773b159e0e2efb135270e3b7 --
2.34.1

#6 lore raw
Quoted 10 lines · Ian Rogers
Quoted 6 lines · Beau Belgrave

In the Open Telemetry profiling SIG [1], we are trying to find a way to grab a tracing association quickly on a per-sample basis. The team at Elastic has a bespoke way to do this [2], however, I'd like to see a more general way to achieve this. The folks I've been talking with seem open to the idea of just having a TLS value for this we could capture

Presumably TLS == Thread Local Storage.

Yes, the initial idea is to use thread local storage (TLS). It seems to be the fastest option to save a per-thread value that changes at a fast rate.

Quoted 20 lines
Quoted 14 lines

upon each sample. We could then just state, Open Telemetry SDKs should have a TLS value for span correlation. However, we need a way to sample the TLS or other value(s) when a sampling event is generated. This is supported today on Windows via EventActivityIdControl() [3]. Since Open Telemetry works on both Windows and Linux, ideally we can do something as efficient for Linux based workloads.

This series is to explore how it would be best possible to collect supporting data from a user process when a profile sample is collected. Having a value stored in TLS makes a lot of sense for this however there are other ways to explore. Whatever is chosen, kernel samples taken in process context should be able to get this supporting data. In these patches on X64 the fsbase and gsbase are used for this.

Quoted 5 lines

An option to explore suggested by Mathieu Desnoyers is to utilize rseq for processes to register a value location that can be included when profiling if desired. This would allow a tighter contract between user processes and a profiler. It would allow better labeling/categorizing the correlation values.

Quoted 11 lines

It is hard to understand this idea. Are you saying stash a cookie in TLS for samples to capture to indicate an activity? Restartable sequences are about preemption on a CPU not of a thread, so at least my intuition is that they feel different. You could stash information like this today by changing the thread name which generates comm events. I've wondered about having similar information in some form of reserved for profiling stack slot, for example, to stash a pointer to the name of a function being interpreted. Snapshotting all of a stack is bad performance wise and for security. A stack slot would be able to deal with nesting.

You are getting the idea. A slot or tag for a thread would be great! I'm not a fan of overriding the thread comm name (as that already has a use). TLS would be fine, if we could also pass an offset + size + type.

Maybe a stack slot that just points to parts of TLS? That way you could have a set of slots that don't require much memory and selectively copy them out of TLS (or where ever those slots point to in user memory).

When I was talking to Mathieu about this, it seems that rseq already had a place to potentially put these slots. I'm unsure though how the per thread aspects would work.

Mathieu, can you post your ideas here about that?

Quoted 35 lines
Quoted 28 lines

An idea flow would look like this:
User Task Profile
do_work(); sample() -> IP + No activity
...
set_activity(123);
...
do_work(); sample() -> IP + activity (123)
...
set_activity(124);
...
do_work(); sample() -> IP + activity (124)

Ideally, the set_activity() method would not be a syscall. It needs to be very cheap as this should not bottleneck work. Ideally this is just a memcpy of 16-20 bytes as it is on Windows via EventActivityIdControl() using EVENT_ACTIVITY_CTRL_SET_ID.

For those not aware, Open Telemetry allows collecting data from multiple machines and show where time was spent. The tracing context is already available for logs, but not for profiling samples. The idea is to show where slowdowns occur and have profile samples to explain why they slowed down. This must be possible without having to track context switches to do this correlation. This is because the profiling rates are typically 20hz - 1Khz, while the context switching rates are much higher. We do not want to have to consume high context switch rates just to know a correlation for a 20hz signal. Often these 20hz signals are always enabled in some environments.

Quoted 6 lines

Regardless if TLS, rseq, or other source is used I believe we will need a way for perf_events to include it within a sample. The changes in this series show how it could be done with TLS. There is some factoring work under perf to make it easier to add more dump types using the existing ABI. This is mostly to make the patches clearer, certainly the refactor parts could get dropped and we could have duplicated/specialized paths.

Quoted 4 lines

fs and gs may be used for more than just the C runtime's TLS. For example, they may be used by emulators or managed runtimes. I'm not clear why this specific case couldn't be handled through BPF.

Agree about the fs/gs possibly being used for other things. If we had a stack slot we could avoid the confusion and have tighter couplings.

You can do this in eBPF (see [2]). However, it's very clunky and depends on specific SDKs per-language/runtime. We ourselves don't run our profilers with anything other than CAP_PERFMON, and also have environments without BPF enabled due to various reasons. It'd be great if we could get this data directly from perf. At the very least, I'd love to get a standardized way to attribute thread values accessible to other performance systems (like eBPF and perf).

Thanks,
-Beau

Quoted 28 lines

Thanks,
Ian

Quoted 25 lines

1. https://opentelemetry.io/blog/2024/profiling/
2. https://www.elastic.co/blog/continuous-profiling-distributed-tracing-correlation
3. https://learn.microsoft.com/en-us/windows/win32/api/evntprov/nf-evntprov-eventactivityidcontrol

Beau Belgrave (4):
perf/core: Introduce perf_prepare_dump_data()
perf: Introduce PERF_SAMPLE_TLS_USER sample type perf/core: Factor perf_output_sample_udump() perf/x86/core: Add tls dump support

 arch/Kconfig                      |   7 ++
 arch/x86/Kconfig                  |   1 +
 arch/x86/events/core.c            |  14 +++
 arch/x86/include/asm/perf_event.h |   5 +
 include/linux/perf_event.h        |   7 ++
 include/uapi/linux/perf_event.h   |   5 +-
 kernel/events/core.c              | 166 +++++++++++++++++++++++-------
 kernel/events/internal.h          |  16 +++
 8 files changed, 180 insertions(+), 41 deletions(-)

base-commit: fec50db7033ea478773b159e0e2efb135270e3b7 --
2.34.1

#7 lore raw
Quoted 56 lines · Beau Belgrave
Quoted 10 lines · Ian Rogers
Quoted 6 lines · Beau Belgrave

In the Open Telemetry profiling SIG [1], we are trying to find a way to grab a tracing association quickly on a per-sample basis. The team at Elastic has a bespoke way to do this [2], however, I'd like to see a more general way to achieve this. The folks I've been talking with seem open to the idea of just having a TLS value for this we could capture

Presumably TLS == Thread Local Storage.

Yes, the initial idea is to use thread local storage (TLS). It seems to be the fastest option to save a per-thread value that changes at a fast rate.

Quoted 20 lines
Quoted 14 lines

upon each sample. We could then just state, Open Telemetry SDKs should have a TLS value for span correlation. However, we need a way to sample the TLS or other value(s) when a sampling event is generated. This is supported today on Windows via EventActivityIdControl() [3]. Since Open Telemetry works on both Windows and Linux, ideally we can do something as efficient for Linux based workloads.

This series is to explore how it would be best possible to collect supporting data from a user process when a profile sample is collected. Having a value stored in TLS makes a lot of sense for this however there are other ways to explore. Whatever is chosen, kernel samples taken in process context should be able to get this supporting data. In these patches on X64 the fsbase and gsbase are used for this.

Quoted 5 lines

An option to explore suggested by Mathieu Desnoyers is to utilize rseq for processes to register a value location that can be included when profiling if desired. This would allow a tighter contract between user processes and a profiler. It would allow better labeling/categorizing the correlation values.

Quoted 11 lines

It is hard to understand this idea. Are you saying stash a cookie in TLS for samples to capture to indicate an activity? Restartable sequences are about preemption on a CPU not of a thread, so at least my intuition is that they feel different. You could stash information like this today by changing the thread name which generates comm events. I've wondered about having similar information in some form of reserved for profiling stack slot, for example, to stash a pointer to the name of a function being interpreted. Snapshotting all of a stack is bad performance wise and for security. A stack slot would be able to deal with nesting.

You are getting the idea. A slot or tag for a thread would be great! I'm not a fan of overriding the thread comm name (as that already has a use). TLS would be fine, if we could also pass an offset + size + type.

Maybe a stack slot that just points to parts of TLS? That way you could have a set of slots that don't require much memory and selectively copy them out of TLS (or where ever those slots point to in user memory).

Quoted 5 lines

When I was talking to Mathieu about this, it seems that rseq already had a place to potentially put these slots. I'm unsure though how the per thread aspects would work.

Mathieu, can you post your ideas here about that?

Sure. I'll try to summarize my thoughts here. By all means, let me know if I'm missing important pieces of the puzzle.

First of all, here is my understanding of what information we want to share between userspace and kernel.

A 128-bit activity ID identifies "uniquely" (as far as a 128-bit random UUID allows) a portion of the dependency chain involved in doing some work (e.g. answer a HTTP request) across one or many participating hosts.

Activity IDs have a parent/child relationship: a parent activity ID can create children activity IDs.

For instance, if one host has the service "dispatch", another host has a "web server", and a third host has a SQL database, we should be able to follow the chain of activities needed to answer a web query by following those activity IDs, linking them together through parent/child relationships. This usually requires the communication protocols to convey those activity IDs across hosts.

The reason why this information must be provided from userspace is because it's userspace that knows where to find those activity IDs within its application-layer communication protocols.

With tracing, taking a full trace of the activity ID spans begin/end from all hosts allow reconstructing the activity IDs parent/child relationships, so we typically only need to extract information about activity ID span begin/end with parent/child info to a tracer.

Using activity IDs from a kernel profiler is trickier, because we do not have access to the complete span begin/end trace to reconstruct the activity ID parent/child relationship. This is where I suspect we'd want to introduce a notion of "activity ID stack", so a profiler could reconstruct the currently active stack of activity IDs for the current thread by walking that stack.

This profiling could be triggered either from an interrupt (sampling use-case), which would then walk the user-space data on return-to-userspace as noted by Peter, or it could also be triggered from a system call return to userspace. This latter option would make it possible to correlate system calls with their associated activity ID stacks.

The basic scenario is simple enough: a thread pushes a new current activity ID (starts a span), possibly nests other spans, and ends them. It all happens neatly within a single thread.

More advanced scenarios require more thoughts:

- non-blocking communication, where a thread can hop between different requests. Technically, it should be able to swap its current activity ID stack as it swaps handled requests.

- green threads (userspace scheduler): the userspace scheduler should be able to swap the activity ID stack of the current thread when swapping between user level threads.

- task "posting" (e.g. work queues types of work dispatch):
the activity ID stacks should probably be handed over with the work item, and set as current activity ID stack by the worker thread.

- exceptions should be able to restore the activity ID stack from a previously saved state.

- Interpreters, JITs. Not sure about the constraints there, we may need support from the runtimes.

Those activity IDs are frequently updated, so going through a system call each time would be a non-starter. This is where thinking in terms of sharing a per-thread data structure (populated by user-space, read by the kernel) becomes relevant.

A few words about how the rseq(2) system call could help: the main building block of rseq is a per-thread "struct rseq" ABI, which is registered by libc on thread creation, and is guaranteed to be accessible at an offset from the thread pointer in userspace, and can be accessed using the task struct "rseq" pointer from the kernel (in contexts that can handle page faults).

Since Linux v6.3 the rseq structure becomes extensible, and we're working with GNU libc to add support for this extension scheme [1]. So even though the primary use-case for rseq was to enable per-cpu data structures in user-space, it can be used for other purposes where shared per-thread data is needed between kernel and userspace.

We could envision adding a new field to struct rseq which would contain the top level pointer to the current "activity ID stack". The layout of this stack would have to be defined as a kernel ABI. We'd want to support push/pop of activity IDs from that stack, as well as moving all of or portions of the activity ID stack somewhere else, as well as saving/recovering the stack from a saved state to accommodate the "advanced" scenarios described above (and probably other scenarios I'm missing).

rseq(2) could also play a role in letting the kernel expose a seed to be used for generation of random activity IDs through yet another new struct rseq field if this happens to be relevant. It could either be a seed, or just a generation counter to be used to check whether the seed needs to be regenerated after sleep/hibernate/fork/clone [2].

I'm hoping all this makes some sense, or at least highlights holes in my understanding. Feedback is welcome!

Thanks,

Mathieu

[1] https://sourceware.org/pipermail/libc-alpha/2024-March/155587.html
[2] https://sourceware.org/pipermail/libc-alpha/2024-March/155577.html

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com
maintainer
#8 lore raw
In reply to Beau Belgrave
Quoted 11 lines · Beau Belgrave

An idea flow would look like this:
User Task Profile
do_work(); sample() -> IP + No activity
...
set_activity(123);
...
do_work(); sample() -> IP + activity (123)
...
set_activity(124);
...
do_work(); sample() -> IP + activity (124)

This, start with this, because until I saw this, I was utterly confused as to what the heck you were on about.

I started by thinking we already have TID in samples so you can already associate back to user processes and got increasingly confused the further I went.

What you seem to want to do however is have some task-state included so you can see what the thread is doing.

Anyway, since we typically run stuff from NMI context, accessing user data is 'interesting'. As such I would really like to make this work depend on the call-graph rework that pushes all the user access bits into return-to-user.

#9 lore raw
Quoted 15 lines · Peter Zijlstra
Quoted 11 lines · Beau Belgrave

An idea flow would look like this:
User Task Profile
do_work(); sample() -> IP + No activity
...
set_activity(123);
...
do_work(); sample() -> IP + activity (123)
...
set_activity(124);
...
do_work(); sample() -> IP + activity (124)

Quoted 3 lines

This, start with this, because until I saw this, I was utterly confused as to what the heck you were on about.

Will do.

Quoted 7 lines

I started by thinking we already have TID in samples so you can already associate back to user processes and got increasingly confused the further I went.

What you seem to want to do however is have some task-state included so you can see what the thread is doing.

Yeah, there is typically an external context (not on the machine) that wants to be tied to each sample. The context could be a simple integer, UUID, or something else entirely. For OTel, this is a 16-byte array [1].

Quoted 4 lines

Anyway, since we typically run stuff from NMI context, accessing user data is 'interesting'. As such I would really like to make this work depend on the call-graph rework that pushes all the user access bits into return-to-user.

Cool, I assume that's the SFRAME work? Are there pointers to work I could look at and think about what a rebase looks like? Or do you have someone in mind I should work with for this?

Thanks,
-Beau

1. https://www.w3.org/TR/trace-context/#version-format

maintainer
#10 lore raw
Quoted 8 lines · Beau Belgrave
Quoted 4 lines

Anyway, since we typically run stuff from NMI context, accessing user data is 'interesting'. As such I would really like to make this work depend on the call-graph rework that pushes all the user access bits into return-to-user.

Cool, I assume that's the SFRAME work? Are there pointers to work I could look at and think about what a rebase looks like? Or do you have someone in mind I should work with for this?

I've been offline for a little while and still need to catch up with things myself.

Josh was working on that when I dropped off IIRC, I'm not entirely sure where things are at currently (and there is no way I can ever hope to process the backlog).

Anybody know where we are with that?

maintainer
#11 lore raw
Quoted 11 lines · On Sat, 13 Apr 2024 12:53:38 +0200 Peter Zijlstra <[email protected]> wrote:
Quoted 8 lines · Beau Belgrave
Quoted 4 lines

Anyway, since we typically run stuff from NMI context, accessing user data is 'interesting'. As such I would really like to make this work depend on the call-graph rework that pushes all the user access bits into return-to-user.

Cool, I assume that's the SFRAME work? Are there pointers to work I could look at and think about what a rebase looks like? Or do you have someone in mind I should work with for this?

Quoted 8 lines

I've been offline for a little while and still need to catch up with things myself.

Josh was working on that when I dropped off IIRC, I'm not entirely sure where things are at currently (and there is no way I can ever hope to process the backlog).

Anybody know where we are with that?

It's still very much on my RADAR, but with layoffs and such, my priorities have unfortunately changed. I'm hoping to start helping out in the near future though (in a month or two).

Josh was working on it, but I think he got pulled off onto other priorities too :-p

-- Steve

#12 lore raw
Quoted 21 lines · Steven Rostedt
Quoted 18 lines · On Sat, 13 Apr 2024 12:53:38 +0200 Peter Zijlstra <[email protected]> wrote:
Quoted 8 lines · Beau Belgrave
Quoted 4 lines

Anyway, since we typically run stuff from NMI context, accessing user data is 'interesting'. As such I would really like to make this work depend on the call-graph rework that pushes all the user access bits into return-to-user.

Cool, I assume that's the SFRAME work? Are there pointers to work I could look at and think about what a rebase looks like? Or do you have someone in mind I should work with for this?

I've been offline for a little while and still need to catch up with things myself.

Josh was working on that when I dropped off IIRC, I'm not entirely sure where things are at currently (and there is no way I can ever hope to process the backlog).

Quoted 8 lines
Quoted 1 line

Anybody know where we are with that?

It's still very much on my RADAR, but with layoffs and such, my priorities have unfortunately changed. I'm hoping to start helping out in the near future though (in a month or two).

Josh was working on it, but I think he got pulled off onto other priorities too :-p

Yeah, this is still a priority for me and I hope to get back to it over the next few weeks (crosses fingers).

Signature · 1 line
Josh