tracing: make tracer_init_tracefs initcall asynchronous
Subsystems (2)
v3 cover tracing: make tracer_init_tracefs initcall asynchronous
Move trace_eval_init() to subsys_initcall to make it start earlier.
And to avoid tracer_init_tracefs being blocked by trace_event_sem which trace_eval_init() hold [1], queue tracer_init_tracefs() to eval_map_wq to let the two works being executed sequentially.
It can speed up the initialization of kernel as result of making tracer_init_tracefs asynchronous.
On my arm64 platform, it reduce ~20ms of 125ms which total time do_initcalls spend.
Mark-PK Tsai (2):
tracing: Avoid adding tracer option before update_tracer_options
tracing: make tracer_init_tracefs initcall asynchronous
kernel/trace/trace.c | 38 ++++++++++++++++++++++++++++---------- 1 file changed, 28 insertions(+), 10 deletions(-)
Signature · 1 line
2.18.0
PATCH v3 1/2 tracing: Avoid adding tracer option before update_tracer_options
To prepare for support asynchronous tracer_init_tracefs initcall, avoid calling create_trace_option_files before __update_tracer_options. Otherwise, create_trace_option_files will show warning because some tracers in trace_types list are already in tr->topts.
For example, hwlat_tracer call register_tracer in late_initcall, and global_trace.dir is already created in tracing_init_dentry, hwlat_tracer will be put into tr->topts.
Then if the __update_tracer_options is executed after hwlat_tracer registered, create_trace_option_files find that hwlat_tracer is already in tr->topts.
--- kernel/trace/trace.c | 7 +++++++ 1 file changed, 7 insertions(+)
kernel/trace/trace.c +7 -0
@@ -6317,12 +6317,18 @@ static void tracing_set_nop(struct trace_array *tr) tr->current_trace = &nop_trace; } +static bool tracer_options_updated; +static void add_tracer_options (struct trace_array *tr,struct tracer *t) {/* Only enable if the directory has been created already. */ if (!tr->dir)return ; +/* Only create trace option files after update_tracer_options finish */ +if (!tracer_options_updated) +return ; +create_trace_option_files (tr, t); }
@@ -9133,6 +9139,7 @@ static void update_tracer_options(struct trace_array *tr) {mutex_lock (&trace_types_lock);__update_tracer_options (tr); + tracer_options_updated = true;mutex_unlock (&trace_types_lock); }
Signature · 1 line
2.18.0
Steven Rostedt
maintainerRe: [PATCH v3 1/2] tracing: Avoid adding tracer option before update_tracer_options
Quoted 12 lines · On Wed, 23 Mar 2022 23:22:56 +0800 Mark-PK Tsai <[email protected]>…
To prepare for support asynchronous
tracer_init_tracefsinitcall, avoid callingcreate_trace_option_filesbefore__update_tracer_options. Otherwise,create_trace_option_fileswill show warning because some tracers intrace_typeslist are already intr->topts.For example,
hwlat_tracercallregister_tracerinlate_initcall, and global_trace.dir is already created intracing_init_dentry,hwlat_tracerwill be put intotr->topts.
Then if the__update_tracer_optionsis executed afterhwlat_tracerregistered,create_trace_option_filesfind thathwlat_traceris already intr->topts.
Quoted 3 lines
Before this patch:
# ls /sys/kernel/tracing/options annotate funcgraph-duration hex stacktrace bin funcgraph-irqs irq-info sym-addr blk_cgname funcgraph-overhead latency-format sym-offset blk_cgroup funcgraph-overrun markers sym-userobj blk_classic funcgraph-proc overwrite test_nop_accept block funcgraph-tail pause-on-trace test_nop_refuse context-info func-no-repeats printk-msg-only trace_printk disable_on_free func_stack_trace print-parent userstacktrace display-graph function-fork raw verbose event-fork function-trace record-cmd funcgraph-abstime graph-time record-tgid funcgraph-cpu hash-ptr sleep-time
After this patch:
# ls /sys/kernel/tracing/options annotate disable_on_free irq-info raw trace_printk bin display-graph latency-format record-cmd userstacktrace blk_cgname event-fork markers record-tgid verbose blk_cgroup function-fork overwrite stacktrace blk_classic function-trace pause-on-trace sym-addr block hash-ptr printk-msg-only sym-offset context-info hex print-parent sym-userobj
Not good.
Quoted 28 lines
--- kernel/trace/trace.c | 7 +++++++ 1 file changed, 7 insertions(+)kernel/trace/trace.c +6
-0@@ -6317,12 +6317,18 @@ static void tracing_set_nop(struct trace_array *tr) tr->current_trace = &nop_trace; } +static bool tracer_options_updated; +static void add_tracer_options (struct trace_array *tr,struct tracer *t) {/* Only enable if the directory has been created already. */ if (!tr->dir)return ; +/* Only create trace option files after update_tracer_options finish */ +if (!tracer_options_updated) +return ; +create_trace_option_files (tr, t); }
kernel/trace/trace.c static void update_tracer_options(struct trace_array *tr) · line 9142
Quoted 4 lines
kernel/trace/trace.c +1
-0@@ -9133,6 +9139,7 @@ static void update_tracer_options(struct trace_array *tr) {mutex_lock (&trace_types_lock);__update_tracer_options (tr); + tracer_options_updated = true;
I think you want to set this before the __update, as doing it after just makes the update a nop.
-- Steve
kernel/trace/trace.c static void update_tracer_options(struct trace_array *tr) · line 9145
Quoted 3 lines
kernel/trace/trace.c +0
-0mutex_unlock (&trace_types_lock); }
_______________________________________________
Linux-mediatek mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-mediatek
Re: [PATCH v3 1/2] tracing: Avoid adding tracer option before update_tracer_options
Quoted 44 lines
Quoted 12 lines
To prepare for support asynchronous
tracer_init_tracefsinitcall, avoid callingcreate_trace_option_filesbefore__update_tracer_options. Otherwise,create_trace_option_fileswill show warning because some tracers intrace_typeslist are already intr->topts.For example,
hwlat_tracercallregister_tracerinlate_initcall, and global_trace.dir is already created intracing_init_dentry,hwlat_tracerwill be put intotr->topts.
Then if the__update_tracer_optionsis executed afterhwlat_tracerregistered,create_trace_option_filesfind thathwlat_traceris already intr->topts.Quoted 3 lines
Before this patch:
# ls /sys/kernel/tracing/options annotate funcgraph-duration hex stacktrace bin funcgraph-irqs irq-info sym-addr blk_cgname funcgraph-overhead latency-format sym-offset blk_cgroup funcgraph-overrun markers sym-userobj blk_classic funcgraph-proc overwrite test_nop_accept block funcgraph-tail pause-on-trace test_nop_refuse context-info func-no-repeats printk-msg-only trace_printk disable_on_free func_stack_trace print-parent userstacktrace display-graph function-fork raw verbose event-fork function-trace record-cmd funcgraph-abstime graph-time record-tgid funcgraph-cpu hash-ptr sleep-timeAfter this patch:
# ls /sys/kernel/tracing/options annotate disable_on_free irq-info raw trace_printk bin display-graph latency-format record-cmd userstacktrace blk_cgname event-fork markers record-tgid verbose blk_cgroup function-fork overwrite stacktrace blk_classic function-trace pause-on-trace sym-addr block hash-ptr printk-msg-only sym-offset context-info hex print-parent sym-userobj
Quoted 1 line
Not good.
Calling __update_tracer_options() when tracer_options_updated=false have no effect because it return directly in add_tracer_options().
I will fix it in v4 as you suggest in the below comment.
Quoted 28 lines
Quoted 28 lines
--- kernel/trace/trace.c | 7 +++++++ 1 file changed, 7 insertions(+)kernel/trace/trace.c +6
-0@@ -6317,12 +6317,18 @@ static void tracing_set_nop(struct trace_array *tr) tr->current_trace = &nop_trace; } +static bool tracer_options_updated; +static void add_tracer_options (struct trace_array *tr,struct tracer *t) {/* Only enable if the directory has been created already. */ if (!tr->dir)return ; +/* Only create trace option files after update_tracer_options finish */ +if (!tracer_options_updated) +return ; +create_trace_option_files (tr, t); }
Quoted 8 lines
kernel/trace/trace.c static void update_tracer_options(struct trace_array *tr) · line 9142
Quoted 5 lines
kernel/trace/trace.c +1
-0@@ -9133,6 +9139,7 @@ static void update_tracer_options(struct trace_array *tr) {mutex_lock (&trace_types_lock);__update_tracer_options (tr); + tracer_options_updated = true;I think you want to set this before the
__update, as doing it after just makes the update a nop.
You are right.
I will update in v4, thanks!
Quoted 7 lines
-- Steve
Quoted 3 lines
kernel/trace/trace.c +0
-0mutex_unlock (&trace_types_lock); }
PATCH v3 2/2 tracing: make tracer_init_tracefs initcall asynchronous
In reply to Mark-PK TsaiMove trace_eval_init() to subsys_initcall to make it start earlier.
And to avoid tracer_init_tracefs being blocked by trace_event_sem which trace_eval_init() hold [1], queue tracer_init_tracefs() to eval_map_wq to let the two works being executed sequentially.
It can speed up the initialization of kernel as result of making tracer_init_tracefs asynchronous.
On my arm64 platform, it reduce ~20ms of 125ms which total time do_initcalls spend.
[1]: https://lore.kernel.org/r/[email protected]
--- kernel/trace/trace.c | 31 +++++++++++++++++++++---------- 1 file changed, 21 insertions(+), 10 deletions(-)
kernel/trace/trace.c +21 -10
@@ -9571,6 +9571,7 @@ extern struct trace_eval_map *__stop_ftrace_eval_maps[];static struct workqueue_struct *eval_map_wq__initdata ;static struct work_struct eval_map_work__initdata ; +static struct work_struct tracerfs_init_work__initdata ;static void __init eval_map_work_func (struct work_struct *work) {
@@ -9596,6 +9597,8 @@ static int __init trace_eval_init(void)return 0 ; } +subsys_initcall (trace_eval_init); +static int __init trace_eval_sync (void ) {/* Make sure the eval map updates are finished */
@@ -9678,15 +9681,8 @@ static struct notifier_block trace_module_nb = { };#endif /* CONFIG_MODULES */ -static __init int tracer_init_tracefs (void ) +static __init void tracer_init_tracefs_work_func (struct work_struct *work) { -int ret; - -trace_access_lock_init (); - - ret =tracing_init_dentry (); -if (ret) -return 0 ;event_trace_init ();
@@ -9708,8 +9704,6 @@ static __init int tracer_init_tracefs(void)trace_create_file ("saved_tgids" ,TRACE_MODE_READ ,NULL ,NULL , &tracing_saved_tgids_fops); -trace_eval_init (); -trace_create_eval_file (NULL );#ifdef CONFIG_MODULES
@@ -9724,6 +9718,23 @@ static __init int tracer_init_tracefs(void)create_trace_instances (NULL );update_tracer_options (&global_trace); +} + +static __init int tracer_init_tracefs (void ) +{ +int ret; + +trace_access_lock_init (); + + ret =tracing_init_dentry (); +if (ret) +return 0 ; + +INIT_WORK (&tracerfs_init_work, tracer_init_tracefs_work_func); +if (!eval_map_wq) +tracer_init_tracefs_work_func (&tracerfs_init_work); +else +queue_work (eval_map_wq, &tracerfs_init_work);return 0 ; }
Signature · 1 line
2.18.0
Steven Rostedt
maintainerkernel/trace/trace.c static __init int tracer_init_tracefs(void) · line 9735
Quoted 13 lines · On Wed, 23 Mar 2022 23:22:57 +0800 Mark-PK Tsai <[email protected]>…
kernel/trace/trace.c +13
-0+static __init int tracer_init_tracefs (void ) +{ +int ret; + +trace_access_lock_init (); + + ret =tracing_init_dentry (); +if (ret) +return 0 ; + +INIT_WORK (&tracerfs_init_work, tracer_init_tracefs_work_func); +if (!eval_map_wq) +tracer_init_tracefs_work_func (&tracerfs_init_work);
Why go through the bother of doing the INIT_WORK if eval_map_wq is not created? Just do:
if (eval_map_wq) {
INIT_WORK(&tracerfs_init_work, tracer_init_tracefs_work_func);
queue_work(eval_map_wq, &tracerfs_init_work);
} else {
tracer_init_tracefs_work_func(NULL);
}But that's just a nit anyway.
-- Steve
kernel/trace/trace.c static __init int tracer_init_tracefs(void) · line 9740
Quoted 5 lines
kernel/trace/trace.c +2
-0+else +queue_work (eval_map_wq, &tracerfs_init_work);return 0 ; }
_______________________________________________
Linux-mediatek mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-mediatek
Quoted 18 lines
Quoted 4 lines
kernel/trace/trace.c +3
-0+static __init int tracer_init_tracefs (void ) +{ +int ret;kernel/trace/trace.c static __init int tracer_init_tracefs(void) · line 9735
Quoted 10 lines
kernel/trace/trace.c +10
-0+ +trace_access_lock_init (); + + ret =tracing_init_dentry (); +if (ret) +return 0 ; + +INIT_WORK (&tracerfs_init_work, tracer_init_tracefs_work_func); +if (!eval_map_wq) +tracer_init_tracefs_work_func (&tracerfs_init_work);Why go through the bother of doing the
INIT_WORKifeval_map_wqis not created? Just do:
Quoted 6 lines
if (eval_map_wq) {
INIT_WORK(&tracerfs_init_work, tracer_init_tracefs_work_func);
queue_work(eval_map_wq, &tracerfs_init_work);
} else {
tracer_init_tracefs_work_func(NULL);
}Got it, I will update it in v4.
Thanks!
Quoted 12 lines
But that's just a nit anyway.
-- Steve
Quoted 5 lines
kernel/trace/trace.c +2
-0+else +queue_work (eval_map_wq, &tracerfs_init_work);return 0 ; }
v3 cover tracing: make tracer_init_tracefs initcall asynchronous
In reply to Mark-PK TsaiHi Steve,
Could you please help to review this patch?
Sorry to disturb you, I just want to make sure it has not been forgotten.
Quoted 22 lines
Move
trace_eval_init()tosubsys_initcallto make it start earlier.
And to avoidtracer_init_tracefsbeing blocked bytrace_event_semwhichtrace_eval_init()hold [1], queuetracer_init_tracefs()toeval_map_wqto let the two works being executed sequentially.It can speed up the initialization of kernel as result of making
tracer_init_tracefsasynchronous.On my arm64 platform, it reduce ~20ms of 125ms which total time
do_initcallsspend.Mark-PK Tsai (2):
tracing: Avoid adding tracer option beforeupdate_tracer_options
tracing: maketracer_init_tracefsinitcall asynchronouskernel/trace/trace.c | 38 ++++++++++++++++++++++++++++---------- 1 file changed, 28 insertions(+), 10 deletions(-)Signature · 1 line
2.18.0
Steven Rostedt
maintainerQuoted 3 lines · On Fri, 22 Apr 2022 17:27:51 +0800 Mark-PK Tsai <[email protected]>…
Could you please help to review this patch?
Sorry to disturb you, I just want to make sure it has not been forgotten.
No problem. Sorry for not getting to it, but my queue is rather deep :-p
Hopefully I can get to it by next week. Feel free to remind me again in a week in case you do not hear back from me. But it is in my local patchwork, but so is a lot of other patches I need to review :-/
-- Steve