perf: fix kernel panic when parsing user space CS saved in pt_regs
PATCH 1/1 perf: fix kernel panic when parsing user space CS saved in pt_regs
From: Zhang Yanmin <[email protected]>
We hit a kernel panic when running perf to collect some performance data. kenel is x86_64 and user space apps are 32bit.
[ 71.965351, 1] [ Binder_2] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004 [ 71.965360, 1] [ Binder_2] IP: [<ffffffff82012091>] get_segment_base+0x71/0xc0 [ 71.965367, 1] [ Binder_2] PGD 6c65f067 PUD 0 [ 71.965375, 1] [ Binder_2] Oops: 0000 [#1] PREEMPT SMP [ 71.965413, 1] [ Binder_2] Modules linked in: ddrgx snd_merr_dpcm_wm8958 snd_intel_sst snd_soc_sst_platform snd_soc_wm8994 snd_soc_wm_hubs lm3559 imx1x5 atomisp_css2401a0_v21 libmsrlisthelper rmi4 bcm_bt_lpm videobuf_vmalloc videobuf_core fps_throttle hdmi_audio pn544(O) tngdisp bcm4335(O) cfg80211 [ 71.965420, 1] [ Binder_2] CPU: 1 PID: 304 Comm: Binder_2 Tainted: G W O 3.10.20-263902-g184bfbc-dirty #14 [ 71.965426, 1] [ Binder_2] task: ffff8800764dc300 ti: ffff88006c6e8000 task.ti: ffff88006c6e8000 [ 71.965439, 1] [ Binder_2] RIP: 0010:[<ffffffff82012091>] [<ffffffæf82012091>] get_segment_base+0x71/0xc0 [ 71.965<44, 1] [ Binder_2] RSP: 0018:ffff^X8007ea87b98 EFLAGS: 00010092 [ 71.965447, 1] [ !Binder_2] RAX: 0000000000000024 RBX: 0000000000000000 RCX: 0000000000000000 [ 71.965450, 1] [ Binder_2] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000009 [ 71.965454, 1] [ Binder_2] RBP: ffff88007ea87ba8 R08: ffffffff83143b3c R09: ffffffff831848a8 [ 71.965458, 1] [ Binder_2] R10: 0000000000000000 R11: 00000000001bf2d8 R12: 0000000000000000 [ 71.965462, 1] [ Binder_2] R13: ffff88006c6e9fd8 R14: ffff88006c6e9f58 R15: ffff8800764dc300 [ 71.965468, 1_ [ Binder_2] FS: 0000000000000000(0000) GS:ffff88007ea80000(006b) knlGS:00000000f704add0 [ 71.965472, 1] [ Binder_2] CS: 0010 DS: 002b ES: 002b CR0: 0000000080050033 [ 71.965476, 1] [ Binder_2] CR2: 0000000000000004 CR3: 0000000076588000 CR4: 00000^P00001007e0 [ 71.965480, 1] [ Binder_2] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 71.965485, 1] [ Binder_2] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 71.966141, 1] [ Binder_2] Stack: [ 71.966152, 1] [ Binder_2] ffff88005f266c00 0000000000000000 ffff88007ea87c18 ffffffff82013cac [ 71.966161, 1] [ Binder_2] ffff88007ea87d58 00000016fe4704a0 00000000000001a7 ffff88007ea87ef8 [ 71.966171, 1] [ Binder_6] ffff88005f266c00 ffff88007ea87ef8 ffff8800!e07b400 ffff88005f266c00 [ 71.966173, 1] [ Binder_2] Call Trace: [ 71.966179, 1] [ Binder_2] <NMI> [ 71.966190, 1] [ Binder_2] [<ffffffff82013cac>] perf_callchain_user+0x15c/0x240 [ 71.966202, 1] [ Binder_2] [<ffffffff82160754>] perf_callchain+0x134/0x180 [ 71.966210, 1] [ Binder_2] [<fffff&ff820e0787>] ? local_clock+0x47/0x60 [ 71.966221, 1] [ Binder_2] [<ffffffff8215d49b>] perf_prepare_sample+0x1bb/0x240 [ 71.966231, 1] [ Binder_2] [<ffffffff8215d667>] __perf_event_overflow+0x147/0x230 [ 71.966241, 1] [ Binder_2] [<ffffffff82012f68>] ? x86_perf_event_set_period+0xd8/0x150 [ 71.966252, 1] [ Binder_2] [<ffffffff8215df24>] perf_event_overflow+0x14/0x20 [ 71.966260, 1] [ Binder_2] [<ffffffff820194d2>] intel_pmu_handle_irq+0x1c2/0x270 [ 71.966270, 1] [ Binder_2] [<ffffffff828b5d60>] ? call_softirq+0x30/0x30 [ 71.966284, 1] [ Binder_2]$ [<ffffffff828aff01>] perf_event_nmi_handler+0x21/0x30 [ 71.966293, 1] [ Binder_2] [<ffffffff828af5b9>] nmi_handle.isr!.1+0x59/0x=0 [ 71.966303, 1] [` Binder_2] [<ffffffff828af6d8>] default_do_nmi+0x58/0x240 [ 71.966312, 1] [ " Binder_2] [<ffffffff828af978>] do_nmi+0xb8/0xf0 [ 71.966321, 1] [ Binder_2] [|ffffffgf828aebe7>] end_repeat_nmi+0x1e/0x2e [ 71.966332, 1] [ Binder_2] [<ffffffff828b5d60>] ? call_softirq+0x30/0x30 [ 71.966341, 1] [ Binder_2] [<ffffffff828b5d60>] ? call_softirq+0x30/0x30 [ 71.966350, 1] [ Binder_2] [<fFffffff828b5d60>] ? call_softirq+0x30/0x30
Basically, ia32 uses sysenter to start system calls.
sysexit_from_sys_call=>trace_hardirqs_on_thunk. Before calling, sysexit_from_sys_call already pops up pt_regs, then trace_hardirqs_on_thunk would reuse pt_regs space. If perf NMI happens here, perf might use a bad pt_regs.
The patch fixes it by moving the calling to trace_hardirqs_on_thunk ahead of the stack popup.
--- arch/x86/ia32/ia32entry.S | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/ia32/ia32entry.S +1 -1
@@ -167,6 +167,7 @@ sysenter_dispatch: testl $_TIF_ALLWORK_MASK,TI_flags+THREAD_INFO(%rsp,RIP-ARGOFFSET) jnz sysexit_audit sysexit_from_sys_call: + TRACE_IRQS_ON andl $~TS_COMPAT,TI_status+THREAD_INFO(%rsp,RIP-ARGOFFSET) /* clear IF, that popfq doesn't enable interrupts early */ andl $~0x200,EFLAGS-R11(%rsp)
@@ -181,7 +182,6 @@ sysexit_from_sys_call: /*CFI_RESTORE rflags*/ popq_cfi %rcx /* User %esp */ CFI_REGISTER rsp,rcx - TRACE_IRQS_ON ENABLE_INTERRUPTS_SYSEXIT32 #ifdef CONFIG_AUDITSYSCALL
Signature · 1 line
1.8.3.2
Quoted 5 lines · Liu ShuoX
From: Zhang Yanmin <[email protected]>
We hit a kernel panic when running perf to collect some performance data. kenel is
x86_64and user space apps are 32bit.
Quoted 8 lines
[ 71.965351, 1] [ Binder_2] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004 [ 71.965360, 1] [ Binder_2] IP: [<ffffffff82012091>] get_segment_base+0x71/0xc0 [ 71.965367, 1] [ Binder_2] PGD 6c65f067 PUD 0 [ 71.965375, 1] [ Binder_2] Oops: 0000 [#1] PREEMPT SMP [ 71.965413, 1] [ Binder_2] Modules linked in: ddrgx snd_merr_dpcm_wm8958 snd_intel_sst snd_soc_sst_platform snd_soc_wm8994 snd_soc_wm_hubs lm3559 imx1x5 atomisp_css2401a0_v21 libmsrlisthelper rmi4 bcm_bt_lpm videobuf_vmalloc videobuf_core fps_throttle hdmi_audio pn544(O) tngdisp bcm4335(O) cfg80211 [ 71.965420, 1] [ Binder_2] CPU: 1 PID: 304 Comm: Binder_2 Tainted: G W O 3.10.20-263902-g184bfbc-dirty #14 [ 71.965426, 1] [ Binder_2] task: ffff8800764dc300 ti: ffff88006c6e8000 task.ti: ffff88006c6e8000 [ 71.965439, 1] [ Binder_2] RIP: 0010:[<ffffffff82012091>] [<ffffffæf82012091>] get_segment_base+0x71/0xc0
^
Quoted 1 line
[ 71.965<44, 1] [
Binder_2] RSP: 0018:ffff^X8007ea87b98 EFLAGS: 00010092
^ ^
Quoted 1 line
[ 71.965447, 1] [ !Binder_2] RAX: 0000000000000024 RBX: 0000000000000000 RCX: 0000000000000000
^
Quoted 5 lines
[ 71.965450, 1] [ Binder_2] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000009 [ 71.965454, 1] [ Binder_2] RBP: ffff88007ea87ba8 R08: ffffffff83143b3c R09: ffffffff831848a8 [ 71.965458, 1] [ Binder_2] R10: 0000000000000000 R11: 00000000001bf2d8 R12: 0000000000000000 [ 71.965462, 1] [ Binder_2] R13: ffff88006c6e9fd8 R14: ffff88006c6e9f58 R15: ffff8800764dc300 [ 71.965468, 1_ [ Binder_2] FS: 0000000000000000(0000) GS:ffff88007ea80000(006b) knlGS:00000000f704add0
^
Are you suffering some serious corruption?
Quoted 10 lines
Basically, ia32 uses sysenter to start system calls.
sysexit_from_sys_call=>trace_hardirqs_on_thunk. Before calling,
sysexit_from_sys_callalready pops uppt_regs, thentrace_hardirqs_on_thunkwould reusept_regsspace. If perf NMI happens here, perf might use a badpt_regs.The patch fixes it by moving the calling to
trace_hardirqs_on_thunkahead of the stack popup.Change-Id: I6c4fc46b009ea056f2321ce5b8f54cf8769a7bdd
No idea what that is, but it needs to go.
I'll leave the actual patch to hpa, this isn't something I'm too familiar with.
Quoted 26 lines
Signed-off-by Zhang Yanmin <[email protected]>--- arch/x86/ia32/ia32entry.S | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/x86/ia32/ia32entry.S +1
-0@@ -167,6 +167,7 @@ sysenter_dispatch: testl $_TIF_ALLWORK_MASK,TI_flags+THREAD_INFO(%rsp,RIP-ARGOFFSET) jnz sysexit_audit sysexit_from_sys_call: + TRACE_IRQS_ON andl $~TS_COMPAT,TI_status+THREAD_INFO(%rsp,RIP-ARGOFFSET) /* clear IF, that popfq doesn't enable interrupts early */ andl $~0x200,EFLAGS-R11(%rsp) @@ -181,7 +182,6 @@ sysexit_from_sys_call:arch/x86/ia32/ia32entry.S +0
-1/*CFI_RESTORE rflags*/ popq_cfi %rcx /* User %esp */ CFI_REGISTER rsp,rcx - TRACE_IRQS_ON ENABLE_INTERRUPTS_SYSEXIT32 #ifdef CONFIG_AUDITSYSCALLSignature · 1 line
1.8.3.2
Quoted 19 lines · Peter Zijlstra
Quoted 5 lines · Liu ShuoX
From: Zhang Yanmin <[email protected]>
We hit a kernel panic when running perf to collect some performance data. kenel is
x86_64and user space apps are 32bit.Quoted 8 lines
[ 71.965351, 1] [ Binder_2] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004 [ 71.965360, 1] [ Binder_2] IP: [<ffffffff82012091>] get_segment_base+0x71/0xc0 [ 71.965367, 1] [ Binder_2] PGD 6c65f067 PUD 0 [ 71.965375, 1] [ Binder_2] Oops: 0000 [#1] PREEMPT SMP [ 71.965413, 1] [ Binder_2] Modules linked in: ddrgx snd_merr_dpcm_wm8958 snd_intel_sst snd_soc_sst_platform snd_soc_wm8994 snd_soc_wm_hubs lm3559 imx1x5 atomisp_css2401a0_v21 libmsrlisthelper rmi4 bcm_bt_lpm videobuf_vmalloc videobuf_core fps_throttle hdmi_audio pn544(O) tngdisp bcm4335(O) cfg80211 [ 71.965420, 1] [ Binder_2] CPU: 1 PID: 304 Comm: Binder_2 Tainted: G W O 3.10.20-263902-g184bfbc-dirty #14 [ 71.965426, 1] [ Binder_2] task: ffff8800764dc300 ti: ffff88006c6e8000 task.ti: ffff88006c6e8000 [ 71.965439, 1] [ Binder_2] RIP: 0010:[<ffffffff82012091>] [<ffffff?f82012091>] get_segment_base+0x71/0xc0^Quoted 1 line
[ 71.965<44, 1] [
Binder_2] RSP: 0018:ffff^X8007ea87b98 EFLAGS: 00010092^ ^
Quoted 1 line
[ 71.965447, 1] [ !Binder_2] RAX: 0000000000000024 RBX: 0000000000000000 RCX: 0000000000000000
^
Quoted 8 lines
Quoted 5 lines
[ 71.965450, 1] [ Binder_2] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000009 [ 71.965454, 1] [ Binder_2] RBP: ffff88007ea87ba8 R08: ffffffff83143b3c R09: ffffffff831848a8 [ 71.965458, 1] [ Binder_2] R10: 0000000000000000 R11: 00000000001bf2d8 R12: 0000000000000000 [ 71.965462, 1] [ Binder_2] R13: ffff88006c6e9fd8 R14: ffff88006c6e9f58 R15: ffff8800764dc300 [ 71.965468, 1_ [ Binder_2] FS: 0000000000000000(0000) GS:ffff88007ea80000(006b) knlGS:00000000f704add0^Are you suffering some serious corruption?
The log captured by pstore after rebooting, so there are some corruption. Please ignore those.
Quoted 10 lines
Quoted 10 lines
Basically, ia32 uses sysenter to start system calls.
sysexit_from_sys_call=>trace_hardirqs_on_thunk. Before calling,
sysexit_from_sys_callalready pops uppt_regs, thentrace_hardirqs_on_thunkwould reusept_regsspace. If perf NMI happens here, perf might use a badpt_regs.The patch fixes it by moving the calling to
trace_hardirqs_on_thunkahead of the stack popup.
Quoted 6 lines
Quoted 1 line
Change-Id: I6c4fc46b009ea056f2321ce5b8f54cf8769a7bdd
No idea what that is, but it needs to go.
I'll leave the actual patch to hpa, this isn't something I'm too familiar with.
Thanks. I will send out v2. We changed the code in v2.
PATCH v2 1/1 perf: fix kernel panic when parsing user space CS saved in pt_regs
From: Zhang Yanmin <[email protected]>
ChangeLog V2: Before sysexit, perf NMI might arrive. There is
still a race. Here we change rsp to keep it pointing to pt_regs->orig_ax.
In addition, after sti, before sysexit, an irq might arrives. That causes more chances for perf NMI to jump in.
We hit a kernel panic when running perf to collect some performance data. kenel is x86_64 and user space apps are 32bit.
[71.965351, 1] [Binder_2] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004 [71.965360, 1] [Binder_2] IP: [<ffffffff82012091>] get_segment_base+0x71/0xc0 [71.965367, 1] [Binder_2] PGD 6c65f067 PUD 0
[71.965375, 1] [Binder_2] Oops: 0000 [#1] PREEMPT SMP
[71.965413, 1] [Binder_2] Modules linked in: <...>
[71.965420, 1] [Binder_2] CPU: 1 PID: 304 Comm: Binder_2 Tainted: G W O 3.10.20-263902-g184bfbc-dirty #14
[71.965426, 1] [Binder_2] task: ffff8800764dc300 ti: ffff88006c6e8000 task.ti: ffff88006c6e8000 [71.965439, 1] [Binder_2] RIP: 0010:[<ffffffff82012091>] [<ffffffæf82012091>] get_segment_base+0x71/0xc0 [71.965<44, 1] [Binder_2] RSP: 0018:ffff^X8007ea87b98 EFLAGS: 00010092
[71.965447, 1] [Binder_2] RAX: 0000000000000024 RBX: 0000000000000000 RCX: 0000000000000000 [71.965450, 1] [Binder_2] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000009 [71.965454, 1] [Binder_2] RBP: ffff88007ea87ba8 R08: ffffffff83143b3c R09: ffffffff831848a8 [71.965458, 1] [Binder_2] R10: 0000000000000000 R11: 00000000001bf2d8 R12: 0000000000000000 [71.965462, 1] [Binder_2] R13: ffff88006c6e9fd8 R14: ffff88006c6e9f58 R15: ffff8800764dc300 [71.965468, 1_ [Binder_2] FS: 0000000000000000(0000) GS:ffff88007ea80000(006b) knlGS:00000000f704add0 [71.965472, 1] [Binder_2] CS: 0010 DS: 002b ES: 002b CR0: 0000000080050033
[71.965476, 1] [Binder_2] CR2: 0000000000000004 CR3: 0000000076588000 CR4: 00000^P00001007e0 [71.965480, 1] [Binder_2] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [71.965485, 1] [Binder_2] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [71.966141, 1] [Binder_2] Stack:
[71.966152, 1] [Binder_2] ffff88005f266c00 0000000000000000 ffff88007ea87c18 ffffffff82013cac [71.966161, 1] [Binder_2] ffff88007ea87d58 00000016fe4704a0 00000000000001a7 ffff88007ea87ef8 [71.966171, 1] [Binder_6] ffff88005f266c00 ffff88007ea87ef8 ffff8800!e07b400 ffff88005f266c00 [71.966173, 1] [Binder_2] Call Trace: [71.966179, 1] [Binder_2] <NMI> [71.966190, 1] [Binder_2] [<ffffffff82013cac>] perf_callchain_user+0x15c/0x240 [71.966202, 1] [Binder_2] [<ffffffff82160754>] perf_callchain+0x134/0x180 [71.966210, 1] [Binder_2] [<fffff&ff820e0787>] ? local_clock+0x47/0x60 [71.966221, 1] [Binder_2] [<ffffffff8215d49b>] perf_prepare_sample+0x1bb/0x240 [71.966231, 1] [Binder_2] [<ffffffff8215d667>] __perf_event_overflow+0x147/0x230 [71.966241, 1] [Binder_2] [<ffffffff82012f68>] ? x86_perf_event_set_period+0xd8/0x150 [71.966252, 1] [Binder_2] [<ffffffff8215df24>] perf_event_overflow+0x14/0x20 [71.966260, 1] [Binder_2] [<ffffffff820194d2>] intel_pmu_handle_irq+0x1c2/0x270 [71.966270, 1] [Binder_2] [<ffffffff828b5d60>] ? call_softirq+0x30/0x30 [71.966284, 1] [Binder_2]$ [<ffffffff828aff01>] perf_event_nmi_handler+0x21/0x30 [71.966293, 1] [Binder_2] [<ffffffff828af5b9>] nmi_handle.isr!.1+0x59/0x=0 [71.966303, 1] [Binder_2] [<ffffffff828af6d8>] default_do_nmi+0x58/0x240 [71.966312, 1] [Binder_2] [<ffffffff828af978>] do_nmi+0xb8/0xf0 [71.966321, 1] [Binder_2] [|ffffffgf828aebe7>] end_repeat_nmi+0x1e/0x2e [71.966332, 1] [Binder_2] [<ffffffff828b5d60>] ? call_softirq+0x30/0x30 [71.966341, 1] [Binder_2] [<ffffffff828b5d60>] ? call_softirq+0x30/0x30 [71.966350, 1] [Binder_2] [<fFffffff828b5d60>] ? call_softirq+0x30/0x30
Basically, ia32 uses sysenter to start system calls.
sysexit_from_sys_call=>trace_hardirqs_on_thunk. Before calling, sysexit_from_sys_call already pops up pt_regs, then trace_hardirqs_on_thunk would reuse pt_regs space. If perf NMI happens here, perf might use a bad pt_regs.
The patch fixes it by keeping rsp pointing under pt_regs->ip.
--- arch/x86/ia32/ia32entry.S | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
arch/x86/ia32/ia32entry.S +6 -5
@@ -172,15 +172,16 @@ sysexit_from_sys_call: andl $~0x200,EFLAGS-R11(%rsp) movl RIP-R11(%rsp),%edx /* User %eip */ CFI_REGISTER rip,rdx - RESTORE_ARGS 0,24,0,0,0,0 - xorq %r8,%r8 + RESTORE_ARGS 0,0,0,0,0,0 + movq 24(%rsp),%r8 /* rflags */ + movq 32(%rsp),%rcx /* User %esp */ xorq %r9,%r9 xorq %r10,%r10 xorq %r11,%r11 - popfq_cfi + pushq_cfi %r8 /*CFI_RESTORE rflags*/ - popq_cfi %rcx /* User %esp */ - CFI_REGISTER rsp,rcx + popfq_cfi + xorq %r8,%r8 TRACE_IRQS_ON ENABLE_INTERRUPTS_SYSEXIT32
Signature · 1 line
1.8.3.2
PATCH v3 1/1 perf: fix kernel panic when parsing user space CS saved in pt_regs
From: Zhang Yanmin <[email protected]>
ChangeLog V3: Keep rsp pointing to pt_regs before sysexit.
ChangeLog V2: Before sysexit, perf NMI might arrive. There is
still a race. Here we change rsp to keep it pointing to pt_regs->orig_ax.
In addition, after sti, before sysexit, an irq might arrives. That causes more chances for perf NMI to jump in.
We hit a kernel panic when running perf to collect some performance data. kenel is x86_64 and user space apps are 32bit.
BUG: unable to handle kernel NULL pointer dereference at 0000000000000004 IP: [<ffffffff82012091>] get_segment_base+0x71/0xc0 PGD 6c65f067 PUD 0 Oops: 0000 [#1] PREEMPT SMP Modules linked in: <...> CPU: 1 PID: 304 Comm: Binder_2 Tainted: G W O 3.10.20-263902-g184bfbc-dirty #14 task: ffff8800764dc300 ti: ffff88006c6e8000 task.ti: ffff88006c6e8000 RIP: 0010:[<ffffffff82012091>] [<ffffffæf82012091>] get_segment_base+0x71/0xc0 RSP: 0018:ffff^X8007ea87b98 EFLAGS: 00010092 RAX: 0000000000000024 RBX: 0000000000000000 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000009 RBP: ffff88007ea87ba8 R08: ffffffff83143b3c R09: ffffffff831848a8 R10: 0000000000000000 R11: 00000000001bf2d8 R12: 0000000000000000 R13: ffff88006c6e9fd8 R14: ffff88006c6e9f58 R15: ffff8800764dc300 FS: 0000000000000000(0000) GS:ffff88007ea80000(006b) knlGS:00000000f704add0 CS: 0010 DS: 002b ES: 002b CR0: 0000000080050033 CR2: 0000000000000004 CR3: 0000000076588000 CR4: 00000^P00001007e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Stack: ffff88005f266c00 0000000000000000 ffff88007ea87c18 ffffffff82013cac ffff88007ea87d58 00000016fe4704a0 00000000000001a7 ffff88007ea87ef8 ffff88005f266c00 ffff88007ea87ef8 ffff8800!e07b400 ffff88005f266c00 Call Trace: <NMI> [<ffffffff82013cac>] perf_callchain_user+0x15c/0x240 [<ffffffff82160754>] perf_callchain+0x134/0x180 [<fffff&ff820e0787>] ? local_clock+0x47/0x60 [<ffffffff8215d49b>] perf_prepare_sample+0x1bb/0x240 [<ffffffff8215d667>] __perf_event_overflow+0x147/0x230 [<ffffffff82012f68>] ? x86_perf_event_set_period+0xd8/0x150 [<ffffffff8215df24>] perf_event_overflow+0x14/0x20 [<ffffffff820194d2>] intel_pmu_handle_irq+0x1c2/0x270 [<ffffffff828b5d60>] ? call_softirq+0x30/0x30 [<ffffffff828aff01>] perf_event_nmi_handler+0x21/0x30 [<ffffffff828af5b9>] nmi_handle.isr!.1+0x59/0x=0 [<ffffffff828af6d8>] default_do_nmi+0x58/0x240 [<ffffffff828af978>] do_nmi+0xb8/0xf0 [|ffffffgf828aebe7>] end_repeat_nmi+0x1e/0x2e [<ffffffff828b5d60>] ? call_softirq+0x30/0x30 [<ffffffff828b5d60>] ? call_softirq+0x30/0x30 [<fFffffff828b5d60>] ? call_softirq+0x30/0x30
Basically, ia32 uses sysenter to start system calls.
sysexit_from_sys_call=>trace_hardirqs_on_thunk. Before calling, sysexit_from_sys_call already pops up pt_regs, then trace_hardirqs_on_thunk would reuse pt_regs space. If perf NMI happens here, perf might use a bad pt_regs.
The patch fixes it by keeping rsp pointing to pt_regs.
--- arch/x86/ia32/ia32entry.S | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
arch/x86/ia32/ia32entry.S +6 -5
@@ -172,15 +172,16 @@ sysexit_from_sys_call: andl $~0x200,EFLAGS-R11(%rsp) movl RIP-R11(%rsp),%edx /* User %eip */ CFI_REGISTER rip,rdx - RESTORE_ARGS 0,24,0,0,0,0 - xorq %r8,%r8 + RESTORE_ARGS 0,-ARG_SKIP,0,0,0,0 + movq EFLAGS-R11(%rsp),%r8 /* rflags */ + movq RSP-R11(%rsp),%rcx /* User %esp */ xorq %r9,%r9 xorq %r10,%r10 xorq %r11,%r11 - popfq_cfi + pushq_cfi %r8 /*CFI_RESTORE rflags*/ - popq_cfi %rcx /* User %esp */ - CFI_REGISTER rsp,rcx + popfq_cfi + xorq %r8,%r8 TRACE_IRQS_ON ENABLE_INTERRUPTS_SYSEXIT32
Signature · 1 line
1.8.3.2
Quoted 25 lines · Liu ShuoX
Quoted 24 lines · Peter Zijlstra
Quoted 5 lines · Liu ShuoX
From: Zhang Yanmin <[email protected]>
We hit a kernel panic when running perf to collect some performance data. kenel is
x86_64and user space apps are 32bit.Quoted 8 lines
[ 71.965351, 1] [ Binder_2] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004 [ 71.965360, 1] [ Binder_2] IP: [<ffffffff82012091>] get_segment_base+0x71/0xc0 [ 71.965367, 1] [ Binder_2] PGD 6c65f067 PUD 0 [ 71.965375, 1] [ Binder_2] Oops: 0000 [#1] PREEMPT SMP [ 71.965413, 1] [ Binder_2] Modules linked in: ddrgx snd_merr_dpcm_wm8958 snd_intel_sst snd_soc_sst_platform snd_soc_wm8994 snd_soc_wm_hubs lm3559 imx1x5 atomisp_css2401a0_v21 libmsrlisthelper rmi4 bcm_bt_lpm videobuf_vmalloc videobuf_core fps_throttle hdmi_audio pn544(O) tngdisp bcm4335(O) cfg80211 [ 71.965420, 1] [ Binder_2] CPU: 1 PID: 304 Comm: Binder_2 Tainted: G W O 3.10.20-263902-g184bfbc-dirty #14 [ 71.965426, 1] [ Binder_2] task: ffff8800764dc300 ti: ffff88006c6e8000 task.ti: ffff88006c6e8000 [ 71.965439, 1] [ Binder_2] RIP: 0010:[<ffffffff82012091>] [<ffffff?f82012091>] get_segment_base+0x71/0xc0^Quoted 1 line
[ 71.965<44, 1] [
Binder_2] RSP: 0018:ffff^X8007ea87b98 EFLAGS: 00010092^ ^
Quoted 1 line
[ 71.965447, 1] [ !Binder_2] RAX: 0000000000000024 RBX: 0000000000000000 RCX: 0000000000000000
^Quoted 5 lines
[ 71.965450, 1] [ Binder_2] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000009 [ 71.965454, 1] [ Binder_2] RBP: ffff88007ea87ba8 R08: ffffffff83143b3c R09: ffffffff831848a8 [ 71.965458, 1] [ Binder_2] R10: 0000000000000000 R11: 00000000001bf2d8 R12: 0000000000000000 [ 71.965462, 1] [ Binder_2] R13: ffff88006c6e9fd8 R14: ffff88006c6e9f58 R15: ffff8800764dc300 [ 71.965468, 1_ [ Binder_2] FS: 0000000000000000(0000) GS:ffff88007ea80000(006b) knlGS:00000000f704add0
Quoted 5 lines
Quoted 3 lines
^Are you suffering some serious corruption?
The log captured by pstore after rebooting, so there are some corruption. Please ignore those.
Why does pstore cause corruption? I thought that stuff was supposed to be 'good' ?
Quoted 26 lines · Peter Zijlstra
Quoted 25 lines · Liu ShuoX
Quoted 24 lines · Peter Zijlstra
Quoted 5 lines · Liu ShuoX
From: Zhang Yanmin <[email protected]>
We hit a kernel panic when running perf to collect some performance data. kenel is
x86_64and user space apps are 32bit.Quoted 8 lines
[ 71.965351, 1] [ Binder_2] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004 [ 71.965360, 1] [ Binder_2] IP: [<ffffffff82012091>] get_segment_base+0x71/0xc0 [ 71.965367, 1] [ Binder_2] PGD 6c65f067 PUD 0 [ 71.965375, 1] [ Binder_2] Oops: 0000 [#1] PREEMPT SMP [ 71.965413, 1] [ Binder_2] Modules linked in: ddrgx snd_merr_dpcm_wm8958 snd_intel_sst snd_soc_sst_platform snd_soc_wm8994 snd_soc_wm_hubs lm3559 imx1x5 atomisp_css2401a0_v21 libmsrlisthelper rmi4 bcm_bt_lpm videobuf_vmalloc videobuf_core fps_throttle hdmi_audio pn544(O) tngdisp bcm4335(O) cfg80211 [ 71.965420, 1] [ Binder_2] CPU: 1 PID: 304 Comm: Binder_2 Tainted: G W O 3.10.20-263902-g184bfbc-dirty #14 [ 71.965426, 1] [ Binder_2] task: ffff8800764dc300 ti: ffff88006c6e8000 task.ti: ffff88006c6e8000 [ 71.965439, 1] [ Binder_2] RIP: 0010:[<ffffffff82012091>] [<ffffff?f82012091>] get_segment_base+0x71/0xc0^Quoted 1 line
[ 71.965<44, 1] [
Binder_2] RSP: 0018:ffff^X8007ea87b98 EFLAGS: 00010092^ ^
Quoted 1 line
[ 71.965447, 1] [ !Binder_2] RAX: 0000000000000024 RBX: 0000000000000000 RCX: 0000000000000000
^Quoted 5 lines
[ 71.965450, 1] [ Binder_2] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000009 [ 71.965454, 1] [ Binder_2] RBP: ffff88007ea87ba8 R08: ffffffff83143b3c R09: ffffffff831848a8 [ 71.965458, 1] [ Binder_2] R10: 0000000000000000 R11: 00000000001bf2d8 R12: 0000000000000000 [ 71.965462, 1] [ Binder_2] R13: ffff88006c6e9fd8 R14: ffff88006c6e9f58 R15: ffff8800764dc300 [ 71.965468, 1_ [ Binder_2] FS: 0000000000000000(0000) GS:ffff88007ea80000(006b) knlGS:00000000f704add0
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
^Are you suffering some serious corruption?
The log captured by pstore after rebooting, so there are some corruption. Please ignore those.
Why does pstore cause corruption? I thought that stuff was supposed to be 'good' ?
pstore is good if the board is reset by WarmReset as memory content is kept across rebooting. If it's a ColdReset, memory might lose some or all contents. My board uses Coldreset, which is a very fast ColdReset. Most memory content can be kept. But sometimes, some data has little change.
Yanmin
Quoted 11 lines · Yanmin
Quoted 2 lines · Peter Zijlstra
Why does pstore cause corruption? I thought that stuff was supposed to be 'good' ?
pstore is good if the board is reset by WarmReset as memory content is kept across rebooting.
If it's a ColdReset, memory might lose some or all contents. My board uses Coldreset, which is a very
fast ColdReset. Most memory content can be kept. But sometimes, some data has little change.
Oh, I thought pstore was persistent across cold resets or even power outages. My bad then.
Quoted 4 lines · Peter Zijlstra
Quoted 3 lines · Yanmin
Quoted 2 lines · Peter Zijlstra
Why does pstore cause corruption? I thought that stuff was supposed to be 'good' ?
Quoted 8 lines
Quoted 6 lines
pstore is good if the board is reset by WarmReset as memory content is kept across rebooting.
If it's a ColdReset, memory might lose some or all contents. My board uses Coldreset, which is a very
fast ColdReset. Most memory content can be kept. But sometimes, some data has little change.Oh, I thought pstore was persistent across cold resets or even power outages. My bad then.
Sorry for misleading you. It depends on backend. Pstore can use RAM or other storage, for example, EFI-provided backend. If using EFI backend, pstore can keep persistent across cold reset or even power outrages.
If using RAM, usually pstore can keep persistent at WarmReset. Since RAM access (even at non-cache mode) is fast and more flexible, in fact, we often use RAM as backend of pstore.
Quoted 7 lines · Yanmin
Sorry for misleading you. It depends on backend. Pstore can use RAM or other storage, for example, EFI-provided backend. If using EFI backend, pstore can keep persistent across cold reset or even power outrages.
If using RAM, usually pstore can keep persistent at WarmReset. Since RAM access (even at non-cache mode) is fast and more flexible, in fact, we often use RAM as backend of pstore.
Funny definition of persistent you end up with in that case... :-) But yes, that does explain.