kgdb segv in the latest tip due to perf ctx changes
hi,
while starting kgdb early debug on latest tip tree, I got SIGSEGV inside kernel in following location:
fetch_bp_busy_slots (bp=0xffff880075f69000) atkernel/hw_breakpoint.c:137
137 struct task_struct *tsk = bp->ctx->task;
the segv is caused by the kgdb code assuming the event has ctx field defined, which is not true at the the time by the following backtrace:
(gdb) bt
#0 fetch_bp_busy_slots (bp=0xffff880075f69000) at kernel/hw_breakpoint.c:137
#1 __reserve_bp_slot (bp=0xffff880075f69000) at kernel/hw_breakpoint.c:314
#2 0xffffffff810ac0c5 in reserve_bp_slot (bp=0xffff880075f69000) at kernel/hw_breakpoint.c:336
#3 0xffffffff810ac0f6 in register_perf_hw_breakpoint (bp=0xffff880075f69000) at kernel/hw_breakpoint.c:412
#4 0xffffffff810ac165 in hw_breakpoint_event_init (bp=0xffff880075f69000) at kernel/hw_breakpoint.c:580
#5 0xffffffff810a3da6 in perf_init_event (event=0xffff880075f69000) at kernel/perf_event.c:5268
#6 0xffffffff810aa658 in perf_event_alloc (attr=0xffff88007899ddf0, cpu=0, group_leader=0xffff880075f69000, parent_event=0x0,
overflow_handler=0) at kernel/perf_event.c:5356#7 0xffffffff810aae6c in perf_event_create_kernel_counter (attr=<value optimized out>, cpu=0, task=0x0,
overflow_handler=<value optimized out>) at kernel/perf_event.c:5752
#8 0xffffffff810ab8b5 in register_wide_hw_breakpoint (attr=0xffff88007899ddf0, triggered=0) at kernel/hw_breakpoint.c:518 #9 0xffffffff81025e2a in kgdb_arch_late () at arch/x86/kernel/kgdb.c:646
#10 0xffffffff8107be62 in kgdb_register_callbacks (new_dbg_io_ops=0xffffffff81a3de00) at kernel/debug/debug_core.c:802 #11 kgdb_register_io_module (new_dbg_io_ops=0xffffffff81a3de00) at kernel/debug/debug_core.c:906
#12 0xffffffff812662bc in configure_kgdboc () at drivers/serial/kgdboc.c:137
#13 0xffffffff81ae805b in init_kgdboc () at drivers/serial/kgdboc.c:159
#14 0xffffffff81abd61e in do_one_initcall (fn=0xffffffff81ae8047 <init_kgdboc>) at init/main.c:750
#15 0xffffffff81abd828 in do_initcalls (unused=<value optimized out>) at init/main.c:780
#16 do_basic_setup (unused=<value optimized out>) at init/main.c:801
#17 kernel_init (unused=<value optimized out>) at init/main.c:893
#18 0xffffffff810033d4 in ?? () at arch/x86/kernel/entry_64.S:1156
#19 0x0000000000000000 in ?? ()
I found out it's due to foolowing commit, that's causing the init code to be called without the ctx field being defined...
commit c3f00c70276d8ae82578c8b773e2db657f69a478
Author: Peter Zijlstra <[email protected]> Date: Wed Aug 18 14:37:15 2010 +0200
I'm not sure if kgdb is wrong assuming the filed exists or perf code not defining the ctx field at that time..
I can reproduce easily, let me know if I can help
wbr,
jirka
Jiri,
Can you try this simple patch which is attached?
Quoted 4 lines · Jiri Olsa
while starting kgdb early debug on latest tip tree, I got SIGSEGV inside kernel in following location:
[clip]
Quoted 9 lines
I found out it's due to foolowing commit, that's causing the init code to be called without the ctx field being defined...
commit
c3f00c70276d8ae82578c8b773e2db657f69a478Author: Peter Zijlstra <[email protected]> Date: Wed Aug 18 14:37:15 2010 +0200
I took a look at the tip core, and the ctx parameter is no longer passed into the perf_event_alloc() from perf_event_create_kernel_counter() kgdb no longer gets it filled in for free.
The reality is that kgdb never had a true context or a way to mark the hw breakpoint as a kernel only context for the hw breakpoint reservations. The patch is only a short term fix perhaps until on of the perf guys explains the right way to use it. :-)
Thanks,
Jason.
Quoted 1 line
From
17f3febd001a26aee9a75c61152b60b7e0ae1ea9Mon Sep 17 00:00:00 2001
From: Jason Wessel <[email protected]>
Date: Fri, 24 Sep 2010 15:21:11 -0500
Subject: [PATCH] Fix null dereference when using early kgdb
--- kernel/hw_breakpoint.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/hw_breakpoint.c +1 -1
@@ -213,7 +213,7 @@ toggle_bp_slot(struct perf_event *bp, bool enable, enum bp_type_idx type, int weight) {int cpu = bp->cpu; -struct task_struct *tsk = bp->ctx->task; +struct task_struct *tsk = bp->ctx ? bp->ctx->task :NULL ;/* Pinned counter cpu profiling */ if (!tsk) {
Signature · 1 line
1.6.3.3
Quoted 22 lines · Jason Wessel
Jiri,
Can you try this simple patch which is attached?
Quoted 4 lines · Jiri Olsa
while starting kgdb early debug on latest tip tree, I got SIGSEGV inside kernel in following location:
[clip]
Quoted 9 lines
I found out it's due to foolowing commit, that's causing the init code to be called without the ctx field being defined...
commit
c3f00c70276d8ae82578c8b773e2db657f69a478Author: Peter Zijlstra <[email protected]> Date: Wed Aug 18 14:37:15 2010 +0200
Quoted 8 lines
I took a look at the tip core, and the ctx parameter is no longer passed into the
perf_event_alloc()fromperf_event_create_kernel_counter()kgdb no longer gets it filled in for free.The reality is that kgdb never had a true context or a way to mark the hw breakpoint as a kernel only context for the hw breakpoint reservations. The patch is only a short term fix perhaps until on of the perf guys explains the right way to use it. :-)
Argh, yes, its using the ctx rather early.. we cannot have a context before we've initialized the event, and here it looks like hw_breakpoint wants to use the context to initialize the event, chick, egg, etc..
Frederic, anything we can do about that?
Quoted 17 lines
differences between files attachment (0001-Fix-null-dereference-when-using-early-kgdb.patch) From
17f3febd001a26aee9a75c61152b60b7e0ae1ea9Mon Sep 17 00:00:00 2001
From: Jason Wessel <[email protected]>
Date: Fri, 24 Sep 2010 15:21:11 -0500
Subject: [PATCH] Fix null dereference when using early kgdbSigned-off-by Jason Wessel <[email protected]>--- kernel/hw_breakpoint.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)kernel/hw_breakpoint.c +0
-0
kernel/hw_breakpoint.c toggle_bp_slot(struct perf_event *bp, bool enable, enum bp_type_idx type, · line 219
Quoted 8 lines
kernel/hw_breakpoint.c +1
-1@@ -213,7 +213,7 @@ toggle_bp_slot(struct perf_event *bp, bool enable, enum bp_type_idx type, int weight) {int cpu = bp->cpu; -struct task_struct *tsk = bp->ctx->task; +struct task_struct *tsk = bp->ctx ? bp->ctx->task :NULL ;/* Pinned counter cpu profiling */ if (!tsk) {
That'll probably screw over some accounting, not sure what tsk is used for there.
Quoted 32 lines · Peter Zijlstra
Quoted 22 lines · Jason Wessel
Jiri,
Can you try this simple patch which is attached?
Quoted 4 lines · Jiri Olsa
while starting kgdb early debug on latest tip tree, I got SIGSEGV inside kernel in following location:
[clip]
Quoted 9 lines
I found out it's due to foolowing commit, that's causing the init code to be called without the ctx field being defined...
commit
c3f00c70276d8ae82578c8b773e2db657f69a478Author: Peter Zijlstra <[email protected]> Date: Wed Aug 18 14:37:15 2010 +0200Quoted 8 lines
I took a look at the tip core, and the ctx parameter is no longer passed into the
perf_event_alloc()fromperf_event_create_kernel_counter()kgdb no longer gets it filled in for free.The reality is that kgdb never had a true context or a way to mark the hw breakpoint as a kernel only context for the hw breakpoint reservations. The patch is only a short term fix perhaps until on of the perf guys explains the right way to use it. :-)
Quoted 5 lines
Argh, yes, its using the ctx rather early.. we cannot have a context before we've initialized the event, and here it looks like
hw_breakpointwants to use the context to initialize the event, chick, egg, etc..Frederic, anything we can do about that?
Jason's patch is partially good, it just lacks one place to handle. Jiri, can you test that?
kernel/hw_breakpoint.c +2 -2
@@ -134,7 +134,7 @@ fetch_bp_busy_slots(struct bp_busy_slots *slots, struct perf_event *bp, enum bp_type_idx type) {int cpu = bp->cpu; -struct task_struct *tsk = bp->ctx->task; +struct task_struct *tsk = bp->ctx ? bp->ctx->task :NULL ;if (cpu >=0 ) { slots->pinned =per_cpu (nr_cpu_bp_pinned[type], cpu);
@@ -213,7 +213,7 @@ toggle_bp_slot(struct perf_event *bp, bool enable, enum bp_type_idx type, int weight) {int cpu = bp->cpu; -struct task_struct *tsk = bp->ctx->task; +struct task_struct *tsk = bp->ctx ? bp->ctx->task :NULL ;/* Pinned counter cpu profiling */ if (!tsk) {
Quoted 23 lines
Quoted 23 lines
differences between files attachment (0001-Fix-null-dereference-when-using-early-kgdb.patch) From
17f3febd001a26aee9a75c61152b60b7e0ae1ea9Mon Sep 17 00:00:00 2001
From: Jason Wessel <[email protected]>
Date: Fri, 24 Sep 2010 15:21:11 -0500
Subject: [PATCH] Fix null dereference when using early kgdbSigned-off-by Jason Wessel <[email protected]>--- kernel/hw_breakpoint.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)kernel/hw_breakpoint.c +1
-1@@ -213,7 +213,7 @@ toggle_bp_slot(struct perf_event *bp, bool enable, enum bp_type_idx type, int weight) {int cpu = bp->cpu; -struct task_struct *tsk = bp->ctx->task; +struct task_struct *tsk = bp->ctx ? bp->ctx->task :NULL ;
Quoted 5 lines
kernel/hw_breakpoint.c toggle_bp_slot(struct perf_event *bp, bool enable, enum bp_type_idx type, · line 219
Quoted 2 lines
kernel/hw_breakpoint.c +0
-0/* Pinned counter cpu profiling */ if (!tsk) {That'll probably screw over some accounting, not sure what tsk is used for there.
Nope it's ok. tsk is used to know if we are dealing with a task/cpu bound breakpoint or a cpu wide bound one.
If tsk ends up being NULL, it will think it's a cpu wide bound breakpoint, which it is in the case of kgdb breakpoints.
Jiri Olsa
Tested-byQuoted 37 lines · Frederic Weisbecker
Quoted 36 lines · Peter Zijlstra
Quoted 22 lines · Jason Wessel
Jiri,
Can you try this simple patch which is attached?
Quoted 4 lines · Jiri Olsa
while starting kgdb early debug on latest tip tree, I got SIGSEGV inside kernel in following location:
[clip]
Quoted 9 lines
I found out it's due to foolowing commit, that's causing the init code to be called without the ctx field being defined...
commit
c3f00c70276d8ae82578c8b773e2db657f69a478Author: Peter Zijlstra <[email protected]> Date: Wed Aug 18 14:37:15 2010 +0200Quoted 8 lines
I took a look at the tip core, and the ctx parameter is no longer passed into the
perf_event_alloc()fromperf_event_create_kernel_counter()kgdb no longer gets it filled in for free.The reality is that kgdb never had a true context or a way to mark the hw breakpoint as a kernel only context for the hw breakpoint reservations. The patch is only a short term fix perhaps until on of the perf guys explains the right way to use it. :-)
Argh, yes, its using the ctx rather early.. we cannot have a context before we've initialized the event, and here it looks like
hw_breakpointwants to use the context to initialize the event, chick, egg, etc..
Quoted 6 lines
Quoted 1 line
Frederic, anything we can do about that?
Jason's patch is partially good, it just lacks one place to handle. Jiri, can you test that?
it works ok
feel free to add
thanks,
jirka
Quoted 45 lines · Jiri Olsa
Quoted 37 lines · Frederic Weisbecker
Quoted 36 lines · Peter Zijlstra
Quoted 22 lines · Jason Wessel
Jiri,
Can you try this simple patch which is attached?
Quoted 4 lines · Jiri Olsa
while starting kgdb early debug on latest tip tree, I got SIGSEGV inside kernel in following location:
[clip]
Quoted 9 lines
I found out it's due to foolowing commit, that's causing the init code to be called without the ctx field being defined...
commit
c3f00c70276d8ae82578c8b773e2db657f69a478Author: Peter Zijlstra <[email protected]> Date: Wed Aug 18 14:37:15 2010 +0200Quoted 8 lines
I took a look at the tip core, and the ctx parameter is no longer passed into the
perf_event_alloc()fromperf_event_create_kernel_counter()kgdb no longer gets it filled in for free.The reality is that kgdb never had a true context or a way to mark the hw breakpoint as a kernel only context for the hw breakpoint reservations. The patch is only a short term fix perhaps until on of the perf guys explains the right way to use it. :-)
Argh, yes, its using the ctx rather early.. we cannot have a context before we've initialized the event, and here it looks like
hw_breakpointwants to use the context to initialize the event, chick, egg, etc..Quoted 6 lines
Quoted 1 line
Frederic, anything we can do about that?
Jason's patch is partially good, it just lacks one place to handle. Jiri, can you test that?
Quoted 7 lines
it works ok
feel free to add
Tested-by Jiri Olsa <[email protected]>thanks,
jirka
Actually I'm just realizing that, after the recent perf changes, it does not only concern the early kgdb needs but every breakpoints now that the context is found/allocated after the event gets initialized.
So I need to do a good bunch of changes in the breakpoints to fix that.
I'll come back with something.
Thanks.
Quoted 8 lines · Frederic Weisbecker
Actually I'm just realizing that, after the recent perf changes, it does not only concern the early kgdb needs but every breakpoints now that the context is found/allocated after the event gets initialized.
So I need to do a good bunch of changes in the breakpoints to fix that.
I'll come back with something.
Right, disregard my email from a minute ago then ;-)
Quoted 21 lines · Frederic Weisbecker
Quoted 1 line
Frederic, anything we can do about that?
Jason's patch is partially good, it just lacks one place to handle. Jiri, can you test that?
kernel/hw_breakpoint.c +1
-1@@ -134,7 +134,7 @@ fetch_bp_busy_slots(struct bp_busy_slots *slots, struct perf_event *bp, enum bp_type_idx type) {int cpu = bp->cpu; -struct task_struct *tsk = bp->ctx->task; +struct task_struct *tsk = bp->ctx ? bp->ctx->task :NULL ;if (cpu >=0 ) { slots->pinned =per_cpu (nr_cpu_bp_pinned[type], cpu);
kernel/hw_breakpoint.c toggle_bp_slot(struct perf_event *bp, bool enable, enum bp_type_idx type, · line 219
Quoted 8 lines
kernel/hw_breakpoint.c +1
-1@@ -213,7 +213,7 @@ toggle_bp_slot(struct perf_event *bp, bool enable, enum bp_type_idx type, int weight) {int cpu = bp->cpu; -struct task_struct *tsk = bp->ctx->task; +struct task_struct *tsk = bp->ctx ? bp->ctx->task :NULL ;/* Pinned counter cpu profiling */ if (!tsk) {
That's identical to writing *tsk = NULL;
You seem to be missing the detail that perf_event->ctx will _always_ be NULL during pmu::event_init()
Quoted 9 lines
Quoted 2 lines
That'll probably screw over some accounting, not sure what tsk is used for there.
Nope it's ok. tsk is used to know if we are dealing with a task/cpu bound breakpoint or a cpu wide bound one.
If tsk ends up being NULL, it will think it's a cpu wide bound breakpoint, which it is in the case of kgdb breakpoints.
See above, there's currently no way to know that in pmu::event_init().