trace,smp: Add tracepoints around remotelly called functions
RFC PATCH v3 1/1 trace,smp: Add tracepoints around remotelly called functions
When running RT workloads in isolated CPUs, many cases of deadline misses are caused by remote CPU requests such as smp_call_function*().
For those cases, having the names of those functions running around the deadline miss moment could help (a lot) finding a target for the next improvements.
Add tracepoints for acquiring the function name & csd before entry and after returning from the remote-cpu requested function.
Also, add tracepoints on the remote cpus requesting them.
---
Changes since RFCv2:
- Fixed some spacing issues and trace calls
Changes since RFCv1:
- Implemented trace_csd_queue_cpu() as suggested by Valentin Schneider
- Using EVENT_CLASS in order to avoid duplication
- Introduced new helper: csd_do_func()
- Name change from smp_call_function_* to csd_function_*
- Rebased on top of torvalds/master
include/trace/events/smp.h | 72 ++++++++++++++++++++++++++++++++++++++ kernel/smp.c | 41 +++++++++++++--------- 2 files changed, 96 insertions(+), 17 deletions(-) create mode 100644 include/trace/events/smp.h
include/trace/events/smp.h new file +72 -0
@@ -0,0 +1,72 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#undef TRACE_SYSTEM +#define TRACE_SYSTEM smp + +#if !defined(_TRACE_SMP_H) || defined(TRACE_HEADER_MULTI_READ ) +#define _TRACE_SMP_H + +#include <linux/tracepoint.h> + +TRACE_EVENT (csd_queue_cpu, + + TP_PROTO(const unsigned int cpu, + unsigned long callsite, + smp_call_func_t func, + call_single_data_t *csd), + +TP_ARGS (cpu, callsite, func, csd), + +TP_STRUCT__entry ( + __field(unsigned int, cpu) + __field(void *, callsite) + __field(void *, func) + __field(void *, csd) + ), + +TP_fast_assign ( + __entry->cpu = cpu; + __entry->callsite = (void *)callsite; + __entry->func = func; + __entry->csd = csd; + ), + +TP_printk ("cpu=%u callsite=%pS func=%pS csd=%p" , + __entry->cpu, __entry->callsite, __entry->func, __entry->csd) +); + +/* +* Tracepoints for a function which is called as an effect of smp_call_function.* +*/ +DECLARE_EVENT_CLASS (csd_function, + + TP_PROTO(smp_call_func_t func, call_single_data_t *csd), + +TP_ARGS (func, csd), + +TP_STRUCT__entry ( + __field(void *, func) + __field(void *, csd) + ), + +TP_fast_assign ( + __entry->func = func; + __entry->csd = csd; + ), + +TP_printk ("function %ps, csd = %p" , __entry->func, __entry->csd) +); + +DEFINE_EVENT (csd_function, csd_function_entry, + TP_PROTO(smp_call_func_t func, call_single_data_t *csd), +TP_ARGS (func, csd) +); + +DEFINE_EVENT (csd_function, csd_function_exit, + TP_PROTO(smp_call_func_t func, call_single_data_t *csd), +TP_ARGS (func, csd) +); + +#endif /* _TRACE_SMP_H */ + +/* This part must be outside protection */ +#include <trace/define_trace.h>
kernel/smp.c +24 -17
114 lines not on this page · raw message
Signature · 1 line
2.40.1
kernel test robot
build botkernel test robot noticed the following build warnings:
Hi Leonardo,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build warnings:
[auto build test WARNING on linus/master]
[also build test WARNING on v6.4-rc1 next-20230510]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Leonardo-Bras/trace-smp-Add-tracepoints-around-remotelly-called-functions/20230511-070554 base: linus/master patch link: https://lore.kernel.org/r/20230510230128.150384-1-leobras%40redhat.com patch subject: [RFC PATCH v3 1/1] trace,smp: Add tracepoints around remotelly called functions config: riscv-buildonly-randconfig-r004-20230509 (https://download.01.org/0day-ci/archive/20230511/[email protected]/config) compiler: clang version 17.0.0 (https://github.com/llvm/llvm-project b0fb98227c90adf2536c9ad644a74d5e92961111)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# install riscv cross compiling tool for clang build
# apt-get install binutils-riscv64-linux-gnu
# https://github.com/intel-lab-lkp/linux/commit/fcef1c202a1635b9c7244f06bbad605562b83d43
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Leonardo-Bras/trace-smp-Add-tracepoints-around-remotelly-called-functions/20230511-070554 git checkout fcef1c202a1635b9c7244f06bbad605562b83d43
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=riscv olddefconfig COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=riscv SHELL=/bin/bash
If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <[email protected]>
| Link: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
>> kernel/smp.c:389:27: warning: passing 8-byte aligned argument to 32-byte aligned parameter 3 of 'csd_do_func' may result in an unaligned pointer access [-Walign-mismatch]
csd_do_func(func, info, csd);
^1 warning generated.
vim +/csd_do_func +389 kernel/smp.c
369
370 /*
371 * Insert a previously allocated call_single_data_t element
372 * for execution on the given CPU. data must already have
373 * ->func, ->info, and ->flags set.
374 */
375 static int generic_exec_single(int cpu, struct __call_single_data *csd)
376 {
377 if (cpu == smp_processor_id()) {
378 smp_call_func_t func = csd->func;
379 void *info = csd->info;
380 unsigned long flags;
381
382 /*
383 * We can unlock early even for the synchronous on-stack case,
384 * since we're doing this from the same CPU..
385 */
386 csd_lock_record(csd);
387 csd_unlock(csd);
388 local_irq_save(flags);
> 389 csd_do_func(func, info, csd);
390 csd_lock_record(NULL);
391 local_irq_restore(flags);
392 return 0;
393 }
394
395 if ((unsigned)cpu >= nr_cpu_ids || !cpu_online(cpu)) {
396 csd_unlock(csd);
397 return -ENXIO;
398 }
399
400 __smp_call_single_queue(cpu, &csd->node.llist);
401
402 return 0;
403 }
404
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests
Quoted 35 lines · kernel test robot
Hi Leonardo,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build warnings:
[auto build test WARNING on linus/master]
[also build test WARNING on v6.4-rc1 next-20230510]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]url: https://github.com/intel-lab-lkp/linux/commits/Leonardo-Bras/trace-smp-Add-tracepoints-around-remotelly-called-functions/20230511-070554 base: linus/master patch link: https://lore.kernel.org/r/20230510230128.150384-1-leobras%40redhat.com patch subject: [RFC PATCH v3 1/1] trace,smp: Add tracepoints around remotelly called functions config: riscv-buildonly-randconfig-r004-20230509 (https://download.01.org/0day-ci/archive/20230511/[email protected]/config) compiler: clang version 17.0.0 (https://github.com/llvm/llvm-project b0fb98227c90adf2536c9ad644a74d5e92961111)reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# install riscv cross compiling tool for clang build
# apt-get install binutils-riscv64-linux-gnu
# https://github.com/intel-lab-lkp/linux/commit/fcef1c202a1635b9c7244f06bbad605562b83d43
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Leonardo-Bras/trace-smp-Add-tracepoints-around-remotelly-called-functions/20230511-070554 git checkoutfcef1c202a1635b9c7244f06bbad605562b83d43
# save the config file
mkdirbuild_dir&& cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=riscv olddefconfig COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=riscv SHELL=/bin/bashIf you fix the issue, kindly add following tag where applicable
Quoted 2 lines
Quoted 6 lines
All warnings (new ones prefixed by >>):
Quoted 1 line
Quoted 1 line
kernel/smp.c:389:27: warning: passing 8-byte aligned argument to 32-byte aligned parameter 3 of 'csd_do_func' may result in an unaligned pointer access [-Walign-mismatch]csd_do_func(func, info, csd); ^1 warning generated.
Oh, ok, it makes sense.
On generic_exec_single(), the second parameter is a struct __call_single_data, while csd_do_func() csd argument is a call_single_data_t.
In smp.h we have:
/* Use __aligned() to avoid to use 2 cache lines for 1 csd */ typedef struct __call_single_data call_single_data_t
__aligned(sizeof(struct __call_single_data));
Which makes call_single_data_t 32-byte aligned, while struct __call_single_data is 8-byte aligned in 64BIT.
Changing like this should fix the warning:
-csd_do_func(smp_call_func_t func, void *info, call_single_data_t *csd) +csd_do_func(smp_call_func_t func, void *info, struct __call_single_data *csd)
I will send this on v4 shortly.
Quoted 41 lines
vim +/csd_do_func +389
kernel/smp.c369 370 /* 371 * Insert a previously allocated call_single_data_t element 372 * for execution on the given CPU. data must already have 373 * ->func, ->info, and ->flags set. 374 */ 375 static int generic_exec_single(int cpu, struct __call_single_data *csd) 376 { 377 if (cpu == smp_processor_id()) { 378 smp_call_func_t func = csd->func; 379 void *info = csd->info; 380 unsigned long flags; 381 382 /* 383 * We can unlock early even for the synchronous on-stack case, 384 * since we're doing this from the same CPU.. 385 */ 386 csd_lock_record(csd); 387 csd_unlock(csd); 388 local_irq_save(flags);Quoted 1 line
389
csd_do_func(func, info, csd);390 csd_lock_record(NULL); 391 local_irq_restore(flags); 392 return 0; 393 } 394 395 if ((unsigned)cpu >= nr_cpu_ids || !cpu_online(cpu)) { 396 csd_unlock(csd); 397 return -ENXIO; 398 } 399 400 __smp_call_single_queue(cpu, &csd->node.llist); 401 402 return 0; 403 } 404
Quoted 11 lines · Leonardo Bras
When running RT workloads in isolated CPUs, many cases of deadline misses are caused by remote CPU requests such as smp_call_function*().
For those cases, having the names of those functions running around the deadline miss moment could help (a lot) finding a target for the next improvements.
Add tracepoints for acquiring the function name & csd before entry and after returning from the remote-cpu requested function.
Also, add tracepoints on the remote cpus requesting them.
Per the v1 thread, I got some context as to why we want this, but this changelog is quite sparse on the subject. How about:
"""
The recently added ipi_send_{cpu,cpumask} tracepoints allow finding sources of IPIs targeting CPUs running latency-sensitive applications.
For NOHZ_FULL CPUs, all IPIs are interference, and those tracepoints are sufficient to find them and work on getting rid of them. In some setups however, not *all* IPIs are to be suppressed, but long-running IPI callbacks can still be problematic.
Add a pair of tracepoints to mark the start and end of processing a CSD IPI callback, similar to what exists for softirq, workqueue or timer callbacks. """
And you can probably split the csd_queue_cpu TP into a separate commit, with something along the lines of:
"""
Add a tracepoint for when a CSD is queued to a remote CPU's call_single_queue. This allows finding exactly which CPU queued a given CSD when looking at a csd_function_{entry,exit} event, and also enables us to accurately measure IPI delivery time with e.g. a synthetic event:
$ echo 'hist:keys=cpu,csd.hex:ts=common_timestamp.usecs' >\
/sys/kernel/tracing/events/smp/csd_queue_cpu/trigger
$ echo 'csd_latency unsigned int dst_cpu; unsigned long csd; u64 time' >\
/sys/kernel/tracing/synthetic_events
$ echo \
'hist:keys=common_cpu,csd.hex:'\
'time=common_timestamp.usecs-$ts:'\
'onmatch(smp.csd_queue_cpu).trace(csd_latency,common_cpu,csd,$time)' >\
/sys/kernel/tracing/events/smp/csd_function_entry/trigger$ trace-cmd record -e 'synthetic:csd_latency' hackbench $ trace-cmd report <...>-467 [001] 21.824263: csd_queue_cpu: cpu=0 callsite=try_to_wake_up+0x2ea func=sched_ttwu_pending csd=0xffff8880076148b8 <...>-467 [001] 21.824280: ipi_send_cpu: cpu=0 callsite=try_to_wake_up+0x2ea callback=generic_smp_call_function_single_interrupt+0x0 <...>-489 [000] 21.824299: csd_function_entry: func=sched_ttwu_pending csd=0xffff8880076148b8 <...>-489 [000] 21.824320: csd_latency: dst_cpu=0, csd=18446612682193848504, time=36 """
Quoted 2 lines
Signed-off-by Leonardo Bras <[email protected]>
Overall I like the CSD vs IPI split, it's a saner approach than logging it all as IPIs, even if it does generate more events as we have to emit an event for every csd queued (i.e. no _cpumask() variant is possible).
Some nitpicks below, and one other thing: are we happy with the smp event namespace, and with the new TP names? Should the namespace be csd instead, to match the trace_<namespace>_* nomenclature?
Quoted 46 lines
---
Changes since RFCv2:
- Fixed some spacing issues and trace callsChanges since RFCv1:
- Implementedtrace_csd_queue_cpu()as suggested by Valentin Schneider
- UsingEVENT_CLASSin order to avoid duplication
- Introduced new helper:csd_do_func()
- Name change from smp_call_function_* to csd_function_*
- Rebased on top of torvalds/masterinclude/trace/events/smp.h | 72 ++++++++++++++++++++++++++++++++++++++ kernel/smp.c | 41 +++++++++++++--------- 2 files changed, 96 insertions(+), 17 deletions(-) create mode 100644 include/trace/events/smp.hinclude/trace/events/smp.h new file +23
-0@@ -0,0 +1,72 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#undef TRACE_SYSTEM +#define TRACE_SYSTEM smp + +#if !defined(_TRACE_SMP_H) || defined(TRACE_HEADER_MULTI_READ ) +#define _TRACE_SMP_H + +#include <linux/tracepoint.h> + +TRACE_EVENT(csd_queue_cpu, + + TP_PROTO(const unsigned int cpu, + unsigned long callsite, + smp_call_func_t func, + call_single_data_t *csd), + + TP_ARGS(cpu, callsite, func, csd), + + TP_STRUCT__entry( + __field(unsigned int, cpu) + __field(void *, callsite) + __field(void *, func) + __field(void *, csd)
include/trace/events/smp.h line 33
Quoted 10 lines
include/trace/events/smp.h new file +10
-0+ ), + + TP_fast_assign( + __entry->cpu = cpu; + __entry->callsite = (void *)callsite; + __entry->func = func; + __entry->csd = csd; + ), + + TP_printk("cpu=%u callsite=%pS func=%pS csd=%p",
^^^^^^^^
This can be func=%ps, offsets should always be 0 so not very useful (I know I put %pS for the IPI TPs, that's a force of habit...)
Quoted 14 lines
include/trace/events/smp.h new file +13
-0+ __entry->cpu, __entry->callsite, __entry->func, __entry->csd) +); + +/* +* Tracepoints for a function which is called as an effect of smp_call_function.* +*/ +DECLARE_EVENT_CLASS(csd_function, + + TP_PROTO(smp_call_func_t func, call_single_data_t *csd), + + TP_ARGS(func, csd), + + TP_STRUCT__entry(
include/trace/events/smp.h line 56
Quoted 10 lines
include/trace/events/smp.h new file +10
-0+ __field(void *, func) +__field (void *, csd) + ), + +TP_fast_assign ( + __entry->func = func; + __entry->csd = csd; + ), + +TP_printk ("function %ps, csd = %p" , __entry->func, __entry->csd)
To match the style of the other TPs, that should be:
TP_printk("func=%ps csd=%p", __entry->func, __entry->csd)Quoted 48 lines · Valentin Schneider
Quoted 11 lines · Leonardo Bras
When running RT workloads in isolated CPUs, many cases of deadline misses are caused by remote CPU requests such as smp_call_function*().
For those cases, having the names of those functions running around the deadline miss moment could help (a lot) finding a target for the next improvements.
Add tracepoints for acquiring the function name & csd before entry and after returning from the remote-cpu requested function.
Also, add tracepoints on the remote cpus requesting them.
Per the v1 thread, I got some context as to why we want this, but this changelog is quite sparse on the subject. How about:
"""
The recently added ipi_send_{cpu,cpumask} tracepoints allow finding sources of IPIs targeting CPUs running latency-sensitive applications.For
NOHZ_FULLCPUs, all IPIs are interference, and those tracepoints are sufficient to find them and work on getting rid of them. In some setups however, not *all* IPIs are to be suppressed, but long-running IPI callbacks can still be problematic.Add a pair of tracepoints to mark the start and end of processing a CSD IPI callback, similar to what exists for softirq, workqueue or timer callbacks. """
And you can probably split the
csd_queue_cpuTP into a separate commit, with something along the lines of:"""
Add a tracepoint for when a CSD is queued to a remote CPU'scall_single_queue. This allows finding exactly which CPU queued a given CSD when looking at a csd_function_{entry,exit} event, and also enables us to accurately measure IPI delivery time with e.g. a synthetic event:$ echo 'hist:keys=cpu,csd.hex:ts=common_timestamp.usecs' >\ /sys/kernel/tracing/events/smp/csd_queue_cpu/trigger $ echo 'csd_latency unsigned int dst_cpu; unsigned long csd; u64 time' >\ /sys/kernel/tracing/synthetic_events $ echo \ 'hist:keys=common_cpu,csd.hex:'\ 'time=common_timestamp.usecs-$ts:'\ 'onmatch(smp.csd_queue_cpu).trace(csd_latency,common_cpu,csd,$time)' >\ /sys/kernel/tracing/events/smp/csd_function_entry/trigger
Quoted 7 lines
$ trace-cmd record -e 'synthetic:csd_latency' hackbench $ trace-cmd report <...>-467 [001] 21.824263: csd_queue_cpu: cpu=0 callsite=try_to_wake_up+0x2ea func=sched_ttwu_pending csd=0xffff8880076148b8 <...>-467 [001] 21.824280: ipi_send_cpu: cpu=0 callsite=try_to_wake_up+0x2ea callback=generic_smp_call_function_single_interrupt+0x0 <...>-489 [000] 21.824299: csd_function_entry: func=sched_ttwu_pending csd=0xffff8880076148b8 <...>-489 [000] 21.824320: csd_latency: dst_cpu=0, csd=18446612682193848504, time=36 """
Sure, I will split the patch and add the commit msg as suggested.
Quoted 11 lines
Quoted 3 lines
Signed-off-by Leonardo Bras <[email protected]>Overall I like the CSD vs IPI split, it's a saner approach than logging it all as IPIs, even if it does generate more events as we have to emit an event for every csd queued (i.e. no
_cpumask()variant is possible).Some nitpicks below, and one other thing: are we happy with the smp event namespace, and with the new TP names? Should the namespace be csd instead, to match the trace_<namespace>_* nomenclature?
I selected "smp.h" filename because those tracepoints are getting used on smp.c.
I think keeping a generic name is good for any future tracepoints people end up designing for smp.c, but I am open to discussion.
Quoted 58 lines
Quoted 47 lines
---
Changes since RFCv2:
- Fixed some spacing issues and trace callsChanges since RFCv1:
- Implementedtrace_csd_queue_cpu()as suggested by Valentin Schneider
- UsingEVENT_CLASSin order to avoid duplication
- Introduced new helper:csd_do_func()
- Name change from smp_call_function_* to csd_function_*
- Rebased on top of torvalds/masterinclude/trace/events/smp.h | 72 ++++++++++++++++++++++++++++++++++++++ kernel/smp.c | 41 +++++++++++++--------- 2 files changed, 96 insertions(+), 17 deletions(-) create mode 100644 include/trace/events/smp.hinclude/trace/events/smp.h new file +23
-0@@ -0,0 +1,72 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#undef TRACE_SYSTEM +#define TRACE_SYSTEM smp + +#if !defined(_TRACE_SMP_H) || defined(TRACE_HEADER_MULTI_READ ) +#define _TRACE_SMP_H + +#include <linux/tracepoint.h> + +TRACE_EVENT(csd_queue_cpu, + + TP_PROTO(const unsigned int cpu, + unsigned long callsite, + smp_call_func_t func, + call_single_data_t *csd), + + TP_ARGS(cpu, callsite, func, csd), + + TP_STRUCT__entry( + __field(unsigned int, cpu) + __field(void *, callsite) + __field(void *, func) + __field(void *, csd)include/trace/events/smp.h line 33
Quoted 10 lines
include/trace/events/smp.h new file +10
-0+ ), + + TP_fast_assign( + __entry->cpu = cpu; + __entry->callsite = (void *)callsite; + __entry->func = func; + __entry->csd = csd; + ), + + TP_printk("cpu=%u callsite=%pS func=%pS csd=%p",^^^^^^^^
Quoted 3 lines
This can be func=%ps, offsets should always be 0 so not very useful (I know I put %pS for the IPI TPs, that's a force of habit...)
I agree, let's do as suggested.
Quoted 27 lines
Quoted 14 lines
include/trace/events/smp.h new file +13
-0+ __entry->cpu, __entry->callsite, __entry->func, __entry->csd) +); + +/* +* Tracepoints for a function which is called as an effect of smp_call_function.* +*/ +DECLARE_EVENT_CLASS(csd_function, + + TP_PROTO(smp_call_func_t func, call_single_data_t *csd), + + TP_ARGS(func, csd), + + TP_STRUCT__entry(include/trace/events/smp.h line 56
Quoted 10 lines
include/trace/events/smp.h new file +10
-0+ __field(void *, func) +__field (void *, csd) + ), + +TP_fast_assign ( + __entry->func = func; + __entry->csd = csd; + ), + +TP_printk ("function %ps, csd = %p" , __entry->func, __entry->csd)To match the style of the other TPs, that should be:
Quoted 2 lines
TP_printk("func=%ps csd=%p", __entry->func, __entry->csd)Done!
Thanks for reviewing!
Leo