Skip to contentSkip to conversation

Random shadow stack pointer corruption

DISCUSSION Yu-cheng Yu 5 participants 9 messages Jul 18, 2020 – Jul 29, 2020

lore
Share & tools
lore
#0 lore raw

Hi,

My shadow stack tests start to have random shadow stack pointer corruption after v5.7 (excluding). The symptom looks like some locking issue or the kernel is confused about which CPU a task is on. In later tip/master, this can be triggered by creating two tasks and each does continuous pthread_create()/pthread_join(). If the kernel has max_cpus=1, the issue goes away. I also checked XSAVES/XRSTORS, but this does not seem to be an issue coming from there.

The tests I run take a long time to complete, and some commit points in bisect do not show failures right away. However, the issue can be more easily triggered after the point of:

d77290507ab2 x86/entry/32: Convert IRET exception to IDTENTRY_SW

Can anyone help me find places to look at?

Thanks,
Yu-cheng

#1 lore raw
Quoted 10 lines · Yu-cheng Yu

Hi,

My shadow stack tests start to have random shadow stack pointer corruption after v5.7 (excluding). The symptom looks like some locking issue or the kernel is confused about which CPU a task is on. In later tip/master, this can be triggered by creating two tasks and each does continuous pthread_create()/pthread_join(). If the kernel has max_cpus=1, the issue goes away. I also checked XSAVES/XRSTORS, but this does not seem to be an issue coming from there.

What do you mean "shadow stack pointer corruption"? Is SSP itself corrupt while running in the kernel? Is one of the MSRs getting corrupted? Is the memory to which the shadow stack points getting corrupted? Is the CPU rejecting an attempt to change SSP?

--Andy

Quoted 12 lines

The tests I run take a long time to complete, and some commit points in bisect do not show failures right away. However, the issue can be more easily triggered after the point of:

d77290507ab2 x86/entry/32: Convert IRET exception to IDTENTRY_SW

Can anyone help me find places to look at?

Thanks,
Yu-cheng

#2 lore raw
Quoted 11 lines · Andy Lutomirski
Quoted 9 lines · Yu-cheng Yu

Hi,

My shadow stack tests start to have random shadow stack pointer corruption after v5.7 (excluding). The symptom looks like some locking issue or the kernel is confused about which CPU a task is on. In later tip/master, this can be triggered by creating two tasks and each does continuous pthread_create()/pthread_join(). If the kernel has max_cpus=1, the issue goes away. I also checked XSAVES/XRSTORS, but this does not seem to be an issue coming from there.

Quoted 4 lines

What do you mean "shadow stack pointer corruption"? Is SSP itself corrupt while running in the kernel? Is one of the MSRs getting corrupted? Is the memory to which the shadow stack points getting corrupted? Is the CPU rejecting an attempt to change SSP?

What I see is, a new thread after ret_from_fork() and iret back to ring-3, its shadow stack pointer (MSR_IA32_PL3_SSP) is corrupted.

#3 lore raw
Quoted 13 lines · Yu-cheng Yu
Quoted 11 lines · Andy Lutomirski
Quoted 9 lines · Yu-cheng Yu

Hi,

My shadow stack tests start to have random shadow stack pointer corruption after v5.7 (excluding). The symptom looks like some locking issue or the kernel is confused about which CPU a task is on. In later tip/master, this can be triggered by creating two tasks and each does continuous pthread_create()/pthread_join(). If the kernel has max_cpus=1, the issue goes away. I also checked XSAVES/XRSTORS, but this does not seem to be an issue coming from there.

Quoted 10 lines
Quoted 4 lines

What do you mean "shadow stack pointer corruption"? Is SSP itself corrupt while running in the kernel? Is one of the MSRs getting corrupted? Is the memory to which the shadow stack points getting corrupted? Is the CPU rejecting an attempt to change SSP?

What I see is, a new thread after ret_from_fork() and iret back to ring-3, its shadow stack pointer (MSR_IA32_PL3_SSP) is corrupted.

This is going to be impossible to diagnose, given that the upstream kernel doesn't know about these MSRs at all. If you point to a git tree, maybe I can spot the issue.

#4 lore raw
In reply to Yu-cheng Yu
Quoted 12 lines · Yu-cheng Yu
Quoted 11 lines · Andy Lutomirski
Quoted 9 lines · Yu-cheng Yu

Hi,

My shadow stack tests start to have random shadow stack pointer corruption after v5.7 (excluding). The symptom looks like some locking issue or the kernel is confused about which CPU a task is on. In later tip/master, this can be triggered by creating two tasks and each does continuous pthread_create()/pthread_join(). If the kernel has max_cpus=1, the issue goes away. I also checked XSAVES/XRSTORS, but this does not seem to be an issue coming from there.

Quoted 7 lines
Quoted 4 lines

What do you mean "shadow stack pointer corruption"? Is SSP itself corrupt while running in the kernel? Is one of the MSRs getting corrupted? Is the memory to which the shadow stack points getting corrupted? Is the CPU rejecting an attempt to change SSP?

What I see is, a new thread after ret_from_fork() and iret back to ring-3, its shadow stack pointer (MSR_IA32_PL3_SSP) is corrupted.

Does corrupt mean random? Or is it a valid stack address, just not for _this_ thread? Or NULL? Or is it a kernel address? Have you tried tracing *ALL* the WRMSR's and XRSTOR's that write to the MSR?

#5 lore raw
Quoted 19 lines · Dave Hansen
Quoted 17 lines · Yu-cheng Yu
Quoted 15 lines · Andy Lutomirski
Quoted 9 lines · Yu-cheng Yu

Hi,

My shadow stack tests start to have random shadow stack pointer corruption after v5.7 (excluding). The symptom looks like some locking issue or the kernel is confused about which CPU a task is on. In later tip/master, this can be triggered by creating two tasks and each does continuous pthread_create()/pthread_join(). If the kernel has max_cpus=1, the issue goes away. I also checked XSAVES/XRSTORS, but this does not seem to be an issue coming from there.

What do you mean "shadow stack pointer corruption"? Is SSP itself corrupt while running in the kernel? Is one of the MSRs getting corrupted? Is the memory to which the shadow stack points getting corrupted? Is the CPU rejecting an attempt to change SSP?

Quoted 6 lines
Quoted 2 lines

What I see is, a new thread after ret_from_fork() and iret back to ring-3, its shadow stack pointer (MSR_IA32_PL3_SSP) is corrupted.

Does corrupt mean random? Or is it a valid stack address, just not for _this_ thread? Or NULL? Or is it a kernel address? Have you tried tracing *ALL* the WRMSR's and XRSTOR's that write to the MSR?

Another data point. When memory corruption happened, there was no core dump at all. We verified that core dump was enabled and we did get core dump for other programs.

Signature · 1 line
H.J.
#6 lore raw
In reply to Dave Hansen
Quoted 18 lines · Dave Hansen
Quoted 17 lines · Yu-cheng Yu
Quoted 15 lines · Andy Lutomirski
Quoted 9 lines · Yu-cheng Yu

Hi,

My shadow stack tests start to have random shadow stack pointer corruption after v5.7 (excluding). The symptom looks like some locking issue or the kernel is confused about which CPU a task is on. In later tip/master, this can be triggered by creating two tasks and each does continuous pthread_create()/pthread_join(). If the kernel has max_cpus=1, the issue goes away. I also checked XSAVES/XRSTORS, but this does not seem to be an issue coming from there.

What do you mean "shadow stack pointer corruption"? Is SSP itself corrupt while running in the kernel? Is one of the MSRs getting corrupted? Is the memory to which the shadow stack points getting corrupted? Is the CPU rejecting an attempt to change SSP?

Quoted 6 lines
Quoted 2 lines

What I see is, a new thread after ret_from_fork() and iret back to ring-3, its shadow stack pointer (MSR_IA32_PL3_SSP) is corrupted.

Does corrupt mean random? Or is it a valid stack address, just not for _this_ thread? Or NULL? Or is it a kernel address? Have you tried tracing *ALL* the WRMSR's and XRSTOR's that write to the MSR?

When a shadow stack address is changed, the address appears to be other task's. I traced all WRMSR's and XRSTOR's. I also verified there have not been any XRSTORS from a wrong buffer. When rc6 is tagged, I will re-base, test, and share current patches.

#7 lore raw
Quoted 23 lines · Yu-cheng Yu
Quoted 21 lines · Dave Hansen
Quoted 19 lines · Yu-cheng Yu
Quoted 11 lines · Andy Lutomirski
Quoted 9 lines · Yu-cheng Yu

Hi,

My shadow stack tests start to have random shadow stack pointer corruption after v5.7 (excluding). The symptom looks like some locking issue or the kernel is confused about which CPU a task is on. In later tip/master, this can be triggered by creating two tasks and each does continuous pthread_create()/pthread_join(). If the kernel has max_cpus=1, the issue goes away. I also checked XSAVES/XRSTORS, but this does not seem to be an issue coming from there.

Quoted 4 lines

What do you mean "shadow stack pointer corruption"? Is SSP itself corrupt while running in the kernel? Is one of the MSRs getting corrupted? Is the memory to which the shadow stack points getting corrupted? Is the CPU rejecting an attempt to change SSP?

What I see is, a new thread after ret_from_fork() and iret back to ring-3, its shadow stack pointer (MSR_IA32_PL3_SSP) is corrupted.

Quoted 9 lines
Quoted 3 lines

Does corrupt mean random? Or is it a valid stack address, just not for _this_ thread? Or NULL? Or is it a kernel address? Have you tried tracing *ALL* the WRMSR's and XRSTOR's that write to the MSR?

When a shadow stack address is changed, the address appears to be other task's. I traced all WRMSR's and XRSTOR's. I also verified there have not been any XRSTORS from a wrong buffer. When rc6 is tagged, I will re-base, test, and share current patches.

We have identified that

ommit 91eeafea1e4b7c95cc4f38af186d7d48fceef89a

Author: Thomas Gleixner <[email protected]>
Date:   Thu May 21 22:05:28 2020 +0200

x86/entry: Switch page fault exception to IDTENTRY_RAW

Convert page fault exceptions to IDTENTRY_RAW:

- Implement the C entry point with DEFINE_IDTENTRY_RAW
- Add the CR2 read into the exception handler
- Add the idtentry_enter/exit_cond_rcu() invocations in in the regular page fault handler and in the async PF part.
- Emit the ASM stub with DECLARE_IDTENTRY_RAW
- Remove the ASM idtentry in 64-bit
- Remove the CR2 read from 64-bit
- Remove the open coded ASM entry code in 32-bit
- Fix up the XEN/PV code
- Remove the old prototypes

No functional change.

triggered the shadow stack corruption when the process returned from syscall.
SSP MSR somehow was changed between setting SSP MSR and IRET. Could
there be a page fault between setting SSP MSR and IRET?

Signature · 1 line
H.J.
#8 lore raw
Quoted 60 lines · H.J. Lu
Quoted 23 lines · Yu-cheng Yu
Quoted 23 lines · Dave Hansen
Quoted 19 lines · Yu-cheng Yu
Quoted 11 lines · Andy Lutomirski
Quoted 9 lines · Yu-cheng Yu

Hi,

My shadow stack tests start to have random shadow stack pointer corruption after v5.7 (excluding). The symptom looks like some locking issue or the kernel is confused about which CPU a task is on. In later tip/master, this can be triggered by creating two tasks and each does continuous pthread_create()/pthread_join(). If the kernel has max_cpus=1, the issue goes away. I also checked XSAVES/XRSTORS, but this does not seem to be an issue coming from there.

Quoted 4 lines

What do you mean "shadow stack pointer corruption"? Is SSP itself corrupt while running in the kernel? Is one of the MSRs getting corrupted? Is the memory to which the shadow stack points getting corrupted? Is the CPU rejecting an attempt to change SSP?

What I see is, a new thread after ret_from_fork() and iret back to ring-3, its shadow stack pointer (MSR_IA32_PL3_SSP) is corrupted.

Quoted 9 lines
Quoted 3 lines

Does corrupt mean random? Or is it a valid stack address, just not for _this_ thread? Or NULL? Or is it a kernel address? Have you tried tracing *ALL* the WRMSR's and XRSTOR's that write to the MSR?

When a shadow stack address is changed, the address appears to be other task's. I traced all WRMSR's and XRSTOR's. I also verified there have not been any XRSTORS from a wrong buffer. When rc6 is tagged, I will re-base, test, and share current patches.

We have identified that

ommit 91eeafea1e4b7c95cc4f38af186d7d48fceef89a

Author: Thomas Gleixner <[email protected]>
Date:   Thu May 21 22:05:28 2020 +0200

x86/entry: Switch page fault exception to IDTENTRY_RAW

Convert page fault exceptions to IDTENTRY_RAW:

- Implement the C entry point with DEFINE_IDTENTRY_RAW
- Add the CR2 read into the exception handler
- Add the idtentry_enter/exit_cond_rcu() invocations in in the regular page fault handler and in the async PF part.
- Emit the ASM stub with DECLARE_IDTENTRY_RAW
- Remove the ASM idtentry in 64-bit
- Remove the CR2 read from 64-bit
- Remove the open coded ASM entry code in 32-bit
- Fix up the XEN/PV code
- Remove the old prototypes

Quoted 5 lines

No functional change.

triggered the shadow stack corruption when the process returned from syscall.
SSP MSR somehow was changed between setting SSP MSR and IRET. Could
there be a page fault between setting SSP MSR and IRET?

Not upstream because there’s no SSP MSR.

Quoted 3 lines
Signature · 2 lines
-- 
H.J.