Skip to contentSkip to conversation

trace,smp: Add tracepoints around remotelly called functions

RFC PATCH Superseded Leonardo Bras 3 participants 5 messages May 10, 2023 – May 15, 2023 Tracing

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

RFC PATCH v3 1/1 trace,smp: Add tracepoints around remotelly called functions

When running RT workloads in isolated CPUs, many cases of deadline misses are caused by remote CPU requests such as smp_call_function*().

For those cases, having the names of those functions running around the deadline miss moment could help (a lot) finding a target for the next improvements.

Add tracepoints for acquiring the function name & csd before entry and after returning from the remote-cpu requested function.

Also, add tracepoints on the remote cpus requesting them.

---

Changes since RFCv2:
- Fixed some spacing issues and trace calls

Changes since RFCv1:
- Implemented trace_csd_queue_cpu() as suggested by Valentin Schneider
- Using EVENT_CLASS in order to avoid duplication
- Introduced new helper: csd_do_func()
- Name change from smp_call_function_* to csd_function_*
- Rebased on top of torvalds/master

 include/trace/events/smp.h | 72 ++++++++++++++++++++++++++++++++++++++
 kernel/smp.c               | 41 +++++++++++++---------
 2 files changed, 96 insertions(+), 17 deletions(-)
 create mode 100644 include/trace/events/smp.h
include/trace/events/smp.h new file +72 -0
new file mode 100644
index 000000000000..c304318a0203
@@ -0,0 +1,72 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM smp
+
+#if !defined(_TRACE_SMP_H) || defined(TRACE_HEADER_MULTI_READ)
+#define _TRACE_SMP_H
+
+#include <linux/tracepoint.h>
+
+TRACE_EVENT(csd_queue_cpu,
+
+	TP_PROTO(const unsigned int cpu,
+		 unsigned long callsite,
+		 smp_call_func_t func,
+		 call_single_data_t *csd),
+
+	TP_ARGS(cpu, callsite, func, csd),
+
+	TP_STRUCT__entry(
+		__field(unsigned int, cpu)
+		__field(void *, callsite)
+		__field(void *, func)
+		__field(void *, csd)
+	),
+
+	TP_fast_assign(
+		__entry->cpu = cpu;
+		__entry->callsite = (void *)callsite;
+		__entry->func = func;
+		__entry->csd  = csd;
+	),
+
+	TP_printk("cpu=%u callsite=%pS func=%pS csd=%p",
+		  __entry->cpu, __entry->callsite, __entry->func, __entry->csd)
+);
+
+/*
+ * Tracepoints for a function which is called as an effect of smp_call_function.*
+ */
+DECLARE_EVENT_CLASS(csd_function,
+
+	TP_PROTO(smp_call_func_t func, call_single_data_t *csd),
+
+	TP_ARGS(func, csd),
+
+	TP_STRUCT__entry(
+		__field(void *,	func)
+		__field(void *,	csd)
+	),
+
+	TP_fast_assign(
+		__entry->func	= func;
+		__entry->csd	= csd;
+	),
+
+	TP_printk("function %ps, csd = %p", __entry->func, __entry->csd)
+);
+
+DEFINE_EVENT(csd_function, csd_function_entry,
+	TP_PROTO(smp_call_func_t func, call_single_data_t *csd),
+	TP_ARGS(func, csd)
+);
+
+DEFINE_EVENT(csd_function, csd_function_exit,
+	TP_PROTO(smp_call_func_t func, call_single_data_t *csd),
+	TP_ARGS(func, csd)
+);
+
+#endif /* _TRACE_SMP_H */
+
+/* This part must be outside protection */
+#include <trace/define_trace.h>
kernel/smp.c +24 -17

114 lines not on this page · raw message

Signature · 1 line
2.40.1
#1 lore raw
kernel test robot noticed the following build warnings:

Hi Leonardo,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build warnings:

[auto build test WARNING on linus/master]
[also build test WARNING on v6.4-rc1 next-20230510]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Leonardo-Bras/trace-smp-Add-tracepoints-around-remotelly-called-functions/20230511-070554
base:   linus/master
patch link:    https://lore.kernel.org/r/20230510230128.150384-1-leobras%40redhat.com
patch subject: [RFC PATCH v3 1/1] trace,smp: Add tracepoints around remotelly called functions
config: riscv-buildonly-randconfig-r004-20230509 (https://download.01.org/0day-ci/archive/20230511/[email protected]/config)
compiler: clang version 17.0.0 (https://github.com/llvm/llvm-project b0fb98227c90adf2536c9ad644a74d5e92961111)

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# install riscv cross compiling tool for clang build
# apt-get install binutils-riscv64-linux-gnu
# https://github.com/intel-lab-lkp/linux/commit/fcef1c202a1635b9c7244f06bbad605562b83d43
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Leonardo-Bras/trace-smp-Add-tracepoints-around-remotelly-called-functions/20230511-070554 git checkout fcef1c202a1635b9c7244f06bbad605562b83d43
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=riscv olddefconfig COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=riscv SHELL=/bin/bash

If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <[email protected]>
| Link: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

>> kernel/smp.c:389:27: warning: passing 8-byte aligned argument to 32-byte aligned parameter 3 of 'csd_do_func' may result in an unaligned pointer access [-Walign-mismatch]
                   csd_do_func(func, info, csd);
                                           ^

1 warning generated.

vim +/csd_do_func +389 kernel/smp.c

369
370 /*
371 * Insert a previously allocated call_single_data_t element
372 * for execution on the given CPU. data must already have
373 * ->func, ->info, and ->flags set.
374 */
375 static int generic_exec_single(int cpu, struct __call_single_data *csd)
376 {
377 if (cpu == smp_processor_id()) {
378 smp_call_func_t func = csd->func;
379 void *info = csd->info;
380 unsigned long flags;
381
382 /*
383 * We can unlock early even for the synchronous on-stack case,
384 * since we're doing this from the same CPU..
385 */
386 csd_lock_record(csd);
387 csd_unlock(csd);
388 local_irq_save(flags);

 > 389			csd_do_func(func, info, csd);

390 csd_lock_record(NULL);
391 local_irq_restore(flags);
392 return 0;
393 }
394
395 if ((unsigned)cpu >= nr_cpu_ids || !cpu_online(cpu)) {
396 csd_unlock(csd);
397 return -ENXIO;
398 }
399
400 __smp_call_single_queue(cpu, &csd->node.llist);
401
402 return 0;
403 }
404

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests
#2 lore raw
Quoted 35 lines · kernel test robot

Hi Leonardo,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build warnings:

[auto build test WARNING on linus/master]
[also build test WARNING on v6.4-rc1 next-20230510]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Leonardo-Bras/trace-smp-Add-tracepoints-around-remotelly-called-functions/20230511-070554
base:   linus/master
patch link:    https://lore.kernel.org/r/20230510230128.150384-1-leobras%40redhat.com
patch subject: [RFC PATCH v3 1/1] trace,smp: Add tracepoints around remotelly called functions
config: riscv-buildonly-randconfig-r004-20230509 (https://download.01.org/0day-ci/archive/20230511/[email protected]/config)
compiler: clang version 17.0.0 (https://github.com/llvm/llvm-project b0fb98227c90adf2536c9ad644a74d5e92961111)

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# install riscv cross compiling tool for clang build
# apt-get install binutils-riscv64-linux-gnu
# https://github.com/intel-lab-lkp/linux/commit/fcef1c202a1635b9c7244f06bbad605562b83d43
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Leonardo-Bras/trace-smp-Add-tracepoints-around-remotelly-called-functions/20230511-070554 git checkout fcef1c202a1635b9c7244f06bbad605562b83d43
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=riscv olddefconfig COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=riscv SHELL=/bin/bash

If you fix the issue, kindly add following tag where applicable

Quoted 2 lines
Quoted 6 lines

All warnings (new ones prefixed by >>):

Quoted 1 line
Quoted 1 line

kernel/smp.c:389:27: warning: passing 8-byte aligned argument to 32-byte aligned parameter 3 of 'csd_do_func' may result in an unaligned pointer access [-Walign-mismatch]

                   csd_do_func(func, info, csd);
                                           ^

1 warning generated.

Oh, ok, it makes sense.

On generic_exec_single(), the second parameter is a struct __call_single_data, while csd_do_func() csd argument is a call_single_data_t.

In smp.h we have:

/* Use __aligned() to avoid to use 2 cache lines for 1 csd */ typedef struct __call_single_data call_single_data_t

	__aligned(sizeof(struct __call_single_data));

Which makes call_single_data_t 32-byte aligned, while struct __call_single_data is 8-byte aligned in 64BIT.

Changing like this should fix the warning:
-csd_do_func(smp_call_func_t func, void *info, call_single_data_t *csd) +csd_do_func(smp_call_func_t func, void *info, struct __call_single_data *csd)

I will send this on v4 shortly.

Quoted 41 lines

vim +/csd_do_func +389 kernel/smp.c

   369	
   370	/*
   371	 * Insert a previously allocated call_single_data_t element
   372	 * for execution on the given CPU. data must already have
   373	 * ->func, ->info, and ->flags set.
   374	 */
   375	static int generic_exec_single(int cpu, struct __call_single_data *csd)
   376	{
   377		if (cpu == smp_processor_id()) {
   378			smp_call_func_t func = csd->func;
   379			void *info = csd->info;
   380			unsigned long flags;
   381	
   382			/*
   383			 * We can unlock early even for the synchronous on-stack case,
   384			 * since we're doing this from the same CPU..
   385			 */
   386			csd_lock_record(csd);
   387			csd_unlock(csd);
   388			local_irq_save(flags);
Quoted 1 line

389 csd_do_func(func, info, csd);

   390			csd_lock_record(NULL);
   391			local_irq_restore(flags);
   392			return 0;
   393		}
   394	
   395		if ((unsigned)cpu >= nr_cpu_ids || !cpu_online(cpu)) {
   396			csd_unlock(csd);
   397			return -ENXIO;
   398		}
   399	
   400		__smp_call_single_queue(cpu, &csd->node.llist);
   401	
   402		return 0;
   403	}
   404	
#3 lore raw
In reply to Leonardo Bras
Quoted 11 lines · Leonardo Bras

When running RT workloads in isolated CPUs, many cases of deadline misses are caused by remote CPU requests such as smp_call_function*().

For those cases, having the names of those functions running around the deadline miss moment could help (a lot) finding a target for the next improvements.

Add tracepoints for acquiring the function name & csd before entry and after returning from the remote-cpu requested function.

Also, add tracepoints on the remote cpus requesting them.

Per the v1 thread, I got some context as to why we want this, but this changelog is quite sparse on the subject. How about:

"""
The recently added ipi_send_{cpu,cpumask} tracepoints allow finding sources of IPIs targeting CPUs running latency-sensitive applications.

For NOHZ_FULL CPUs, all IPIs are interference, and those tracepoints are sufficient to find them and work on getting rid of them. In some setups however, not *all* IPIs are to be suppressed, but long-running IPI callbacks can still be problematic.

Add a pair of tracepoints to mark the start and end of processing a CSD IPI callback, similar to what exists for softirq, workqueue or timer callbacks. """

And you can probably split the csd_queue_cpu TP into a separate commit, with something along the lines of:

"""
Add a tracepoint for when a CSD is queued to a remote CPU's call_single_queue. This allows finding exactly which CPU queued a given CSD when looking at a csd_function_{entry,exit} event, and also enables us to accurately measure IPI delivery time with e.g. a synthetic event:

  $ echo 'hist:keys=cpu,csd.hex:ts=common_timestamp.usecs' >\
      /sys/kernel/tracing/events/smp/csd_queue_cpu/trigger
  $ echo 'csd_latency unsigned int dst_cpu; unsigned long csd; u64 time' >\
      /sys/kernel/tracing/synthetic_events
  $ echo \
  'hist:keys=common_cpu,csd.hex:'\
  'time=common_timestamp.usecs-$ts:'\
  'onmatch(smp.csd_queue_cpu).trace(csd_latency,common_cpu,csd,$time)' >\
      /sys/kernel/tracing/events/smp/csd_function_entry/trigger
  $ trace-cmd record -e 'synthetic:csd_latency' hackbench
  $ trace-cmd report
  <...>-467   [001]    21.824263: csd_queue_cpu:        cpu=0 callsite=try_to_wake_up+0x2ea func=sched_ttwu_pending csd=0xffff8880076148b8
  <...>-467   [001]    21.824280: ipi_send_cpu:         cpu=0 callsite=try_to_wake_up+0x2ea callback=generic_smp_call_function_single_interrupt+0x0
  <...>-489   [000]    21.824299: csd_function_entry:   func=sched_ttwu_pending csd=0xffff8880076148b8
  <...>-489   [000]    21.824320: csd_latency:          dst_cpu=0, csd=18446612682193848504, time=36
"""
Quoted 2 lines

Overall I like the CSD vs IPI split, it's a saner approach than logging it all as IPIs, even if it does generate more events as we have to emit an event for every csd queued (i.e. no _cpumask() variant is possible).

Some nitpicks below, and one other thing: are we happy with the smp event namespace, and with the new TP names? Should the namespace be csd instead, to match the trace_<namespace>_* nomenclature?

Quoted 46 lines

---

Changes since RFCv2:
- Fixed some spacing issues and trace calls

Changes since RFCv1:
- Implemented trace_csd_queue_cpu() as suggested by Valentin Schneider
- Using EVENT_CLASS in order to avoid duplication
- Introduced new helper: csd_do_func()
- Name change from smp_call_function_* to csd_function_*
- Rebased on top of torvalds/master

 include/trace/events/smp.h | 72 ++++++++++++++++++++++++++++++++++++++
 kernel/smp.c               | 41 +++++++++++++---------
 2 files changed, 96 insertions(+), 17 deletions(-)
 create mode 100644 include/trace/events/smp.h
include/trace/events/smp.h new file +23 -0
new file mode 100644
index 000000000000..c304318a0203
@@ -0,0 +1,72 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM smp
+
+#if !defined(_TRACE_SMP_H) || defined(TRACE_HEADER_MULTI_READ)
+#define _TRACE_SMP_H
+
+#include <linux/tracepoint.h>
+
+TRACE_EVENT(csd_queue_cpu,
+
+	TP_PROTO(const unsigned int cpu,
+		 unsigned long callsite,
+		 smp_call_func_t func,
+		 call_single_data_t *csd),
+
+	TP_ARGS(cpu, callsite, func, csd),
+
+	TP_STRUCT__entry(
+		__field(unsigned int, cpu)
+		__field(void *, callsite)
+		__field(void *, func)
+		__field(void *, csd)

include/trace/events/smp.h line 33

Quoted 10 lines
include/trace/events/smp.h new file +10 -0
+	),
+
+	TP_fast_assign(
+		__entry->cpu = cpu;
+		__entry->callsite = (void *)callsite;
+		__entry->func = func;
+		__entry->csd  = csd;
+	),
+
+	TP_printk("cpu=%u callsite=%pS func=%pS csd=%p",
                                       ^^^^^^^^

This can be func=%ps, offsets should always be 0 so not very useful (I know I put %pS for the IPI TPs, that's a force of habit...)

Quoted 14 lines
include/trace/events/smp.h new file +13 -0
+		  __entry->cpu, __entry->callsite, __entry->func, __entry->csd)
+);
 
+
+/*
+ * Tracepoints for a function which is called as an effect of smp_call_function.*
+ */
+DECLARE_EVENT_CLASS(csd_function,
+
+	TP_PROTO(smp_call_func_t func, call_single_data_t *csd),
+
+	TP_ARGS(func, csd),
+
+	TP_STRUCT__entry(

include/trace/events/smp.h line 56

Quoted 10 lines
include/trace/events/smp.h new file +10 -0
+		__field(void *,	func)
+		__field(void *,	csd)
+	),
+
+	TP_fast_assign(
+		__entry->func	= func;
+		__entry->csd	= csd;
+	),
+
+	TP_printk("function %ps, csd = %p", __entry->func, __entry->csd)

To match the style of the other TPs, that should be:

        TP_printk("func=%ps csd=%p", __entry->func, __entry->csd)
#4 lore raw
Quoted 48 lines · Valentin Schneider
Quoted 11 lines · Leonardo Bras

When running RT workloads in isolated CPUs, many cases of deadline misses are caused by remote CPU requests such as smp_call_function*().

For those cases, having the names of those functions running around the deadline miss moment could help (a lot) finding a target for the next improvements.

Add tracepoints for acquiring the function name & csd before entry and after returning from the remote-cpu requested function.

Also, add tracepoints on the remote cpus requesting them.

Per the v1 thread, I got some context as to why we want this, but this changelog is quite sparse on the subject. How about:

"""
The recently added ipi_send_{cpu,cpumask} tracepoints allow finding sources of IPIs targeting CPUs running latency-sensitive applications.

For NOHZ_FULL CPUs, all IPIs are interference, and those tracepoints are sufficient to find them and work on getting rid of them. In some setups however, not *all* IPIs are to be suppressed, but long-running IPI callbacks can still be problematic.

Add a pair of tracepoints to mark the start and end of processing a CSD IPI callback, similar to what exists for softirq, workqueue or timer callbacks. """

And you can probably split the csd_queue_cpu TP into a separate commit, with something along the lines of:

"""
Add a tracepoint for when a CSD is queued to a remote CPU's call_single_queue. This allows finding exactly which CPU queued a given CSD when looking at a csd_function_{entry,exit} event, and also enables us to accurately measure IPI delivery time with e.g. a synthetic event:

  $ echo 'hist:keys=cpu,csd.hex:ts=common_timestamp.usecs' >\
      /sys/kernel/tracing/events/smp/csd_queue_cpu/trigger
  $ echo 'csd_latency unsigned int dst_cpu; unsigned long csd; u64 time' >\
      /sys/kernel/tracing/synthetic_events
  $ echo \
  'hist:keys=common_cpu,csd.hex:'\
  'time=common_timestamp.usecs-$ts:'\
  'onmatch(smp.csd_queue_cpu).trace(csd_latency,common_cpu,csd,$time)' >\
      /sys/kernel/tracing/events/smp/csd_function_entry/trigger
Quoted 7 lines
  $ trace-cmd record -e 'synthetic:csd_latency' hackbench
  $ trace-cmd report
  <...>-467   [001]    21.824263: csd_queue_cpu:        cpu=0 callsite=try_to_wake_up+0x2ea func=sched_ttwu_pending csd=0xffff8880076148b8
  <...>-467   [001]    21.824280: ipi_send_cpu:         cpu=0 callsite=try_to_wake_up+0x2ea callback=generic_smp_call_function_single_interrupt+0x0
  <...>-489   [000]    21.824299: csd_function_entry:   func=sched_ttwu_pending csd=0xffff8880076148b8
  <...>-489   [000]    21.824320: csd_latency:          dst_cpu=0, csd=18446612682193848504, time=36
"""

Sure, I will split the patch and add the commit msg as suggested.

Quoted 11 lines
Quoted 3 lines

Overall I like the CSD vs IPI split, it's a saner approach than logging it all as IPIs, even if it does generate more events as we have to emit an event for every csd queued (i.e. no _cpumask() variant is possible).

Some nitpicks below, and one other thing: are we happy with the smp event namespace, and with the new TP names? Should the namespace be csd instead, to match the trace_<namespace>_* nomenclature?

I selected "smp.h" filename because those tracepoints are getting used on smp.c.

I think keeping a generic name is good for any future tracepoints people end up designing for smp.c, but I am open to discussion.

Quoted 58 lines
Quoted 47 lines

---

Changes since RFCv2:
- Fixed some spacing issues and trace calls

Changes since RFCv1:
- Implemented trace_csd_queue_cpu() as suggested by Valentin Schneider
- Using EVENT_CLASS in order to avoid duplication
- Introduced new helper: csd_do_func()
- Name change from smp_call_function_* to csd_function_*
- Rebased on top of torvalds/master

 include/trace/events/smp.h | 72 ++++++++++++++++++++++++++++++++++++++
 kernel/smp.c               | 41 +++++++++++++---------
 2 files changed, 96 insertions(+), 17 deletions(-)
 create mode 100644 include/trace/events/smp.h
include/trace/events/smp.h new file +23 -0
new file mode 100644
index 000000000000..c304318a0203
@@ -0,0 +1,72 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM smp
+
+#if !defined(_TRACE_SMP_H) || defined(TRACE_HEADER_MULTI_READ)
+#define _TRACE_SMP_H
+
+#include <linux/tracepoint.h>
+
+TRACE_EVENT(csd_queue_cpu,
+
+	TP_PROTO(const unsigned int cpu,
+		 unsigned long callsite,
+		 smp_call_func_t func,
+		 call_single_data_t *csd),
+
+	TP_ARGS(cpu, callsite, func, csd),
+
+	TP_STRUCT__entry(
+		__field(unsigned int, cpu)
+		__field(void *, callsite)
+		__field(void *, func)
+		__field(void *, csd)

include/trace/events/smp.h line 33

Quoted 10 lines
include/trace/events/smp.h new file +10 -0
+	),
+
+	TP_fast_assign(
+		__entry->cpu = cpu;
+		__entry->callsite = (void *)callsite;
+		__entry->func = func;
+		__entry->csd  = csd;
+	),
+
+	TP_printk("cpu=%u callsite=%pS func=%pS csd=%p",
                                       ^^^^^^^^
Quoted 3 lines

This can be func=%ps, offsets should always be 0 so not very useful (I know I put %pS for the IPI TPs, that's a force of habit...)

I agree, let's do as suggested.

Quoted 27 lines
Quoted 14 lines
include/trace/events/smp.h new file +13 -0
+		  __entry->cpu, __entry->callsite, __entry->func, __entry->csd)
+);
 
+
+/*
+ * Tracepoints for a function which is called as an effect of smp_call_function.*
+ */
+DECLARE_EVENT_CLASS(csd_function,
+
+	TP_PROTO(smp_call_func_t func, call_single_data_t *csd),
+
+	TP_ARGS(func, csd),
+
+	TP_STRUCT__entry(

include/trace/events/smp.h line 56

Quoted 10 lines
include/trace/events/smp.h new file +10 -0
+		__field(void *,	func)
+		__field(void *,	csd)
+	),
+
+	TP_fast_assign(
+		__entry->func	= func;
+		__entry->csd	= csd;
+	),
+
+	TP_printk("function %ps, csd = %p", __entry->func, __entry->csd)

To match the style of the other TPs, that should be:

Quoted 2 lines
        TP_printk("func=%ps csd=%p", __entry->func, __entry->csd)

Done!

Thanks for reviewing!
Leo