riscv: fix race when vmap stack overflow
PATCH 1/1 riscv: fix race when vmap stack overflow
Currently, when detecting vmap stack overflow, riscv firstly switches to the so called shadow stack, then use this shadow stack to call the get_overflow_stack() to get the overflow stack. However, there's a race here if two or more harts use the same shadow stack at the same time.
To solve this race, we introduce spin_shadow_stack atomic var, which will make the shadow stack usage serialized.
--- arch/riscv/kernel/entry.S | 4 ++++ arch/riscv/kernel/traps.c | 4 ++++ 2 files changed, 8 insertions(+)
arch/riscv/kernel/entry.S +4 -0
@@ -404,6 +404,10 @@ handle_syscall_trace_exit: #ifdef CONFIG_VMAP_STACK handle_kernel_stack_overflow: +1: la sp, spin_shadow_stack + amoswap.w sp, sp, (sp) + bnez sp, 1b + la sp, shadow_stack addi sp, sp, SHADOW_OVERFLOW_STACK_SIZE
arch/riscv/kernel/traps.c +4 -0
@@ -221,11 +221,15 @@ asmlinkage unsigned long get_overflow_stack(void)OVERFLOW_STACK_SIZE ; } +atomic_t spin_shadow_stack; + asmlinkage void handle_bad_stack(struct pt_regs *regs) {unsigned long tsk_stk = (unsigned long )current->stack;unsigned long ovf_stk = (unsigned long )this_cpu_ptr (overflow_stack); +atomic_set_release (&spin_shadow_stack,0 ); +console_verbose ();pr_emerg ("Insufficient stack space to handle exception!\n" );
Signature · 1 line
2.37.2
Quoted 56 lines · Jisheng Zhang
Currently, when detecting vmap stack overflow, riscv firstly switches to the so called shadow stack, then use this shadow stack to call the
get_overflow_stack()to get the overflow stack. However, there's a race here if two or more harts use the same shadow stack at the same time.To solve this race, we introduce
spin_shadow_stackatomic var, which will make the shadow stack usage serialized.Fixes 31da94c25aea ("riscv: add VMAP_STACK overflow detection")Signed-off-by Jisheng Zhang <[email protected]>Suggested-by Guo Ren <[email protected]>--- arch/riscv/kernel/entry.S | 4 ++++ arch/riscv/kernel/traps.c | 4 ++++ 2 files changed, 8 insertions(+)arch/riscv/kernel/entry.S +4
-0@@ -404,6 +404,10 @@ handle_syscall_trace_exit: #ifdef CONFIG_VMAP_STACK handle_kernel_stack_overflow: +1: la sp, spin_shadow_stack + amoswap.w sp, sp, (sp) + bnez sp, 1b + la sp, shadow_stack addi sp, sp, SHADOW_OVERFLOW_STACK_SIZEarch/riscv/kernel/traps.c +4
-1@@ -221,11 +221,15 @@ asmlinkage unsigned long get_overflow_stack(void)OVERFLOW_STACK_SIZE ; } +atomic_t spin_shadow_stack; + asmlinkage void handle_bad_stack(struct pt_regs *regs) {unsigned long tsk_stk = (unsigned long )current->stack;unsigned long ovf_stk = (unsigned long )this_cpu_ptr (overflow_stack); +atomic_set_release (&spin_shadow_stack,0 ); +console_verbose ();pr_emerg ("Insufficient stack space to handle exception!\n" ); --2.37.2
Signature · 2 lines
Best Regards Guo Ren
Quoted 23 lines · Jisheng Zhang
Currently, when detecting vmap stack overflow, riscv firstly switches to the so called shadow stack, then use this shadow stack to call the
get_overflow_stack()to get the overflow stack. However, there's a race here if two or more harts use the same shadow stack at the same time.To solve this race, we introduce
spin_shadow_stackatomic var, which will make the shadow stack usage serialized.Fixes 31da94c25aea ("riscv: add VMAP_STACK overflow detection")Signed-off-by Jisheng Zhang <[email protected]>Suggested-by Guo Ren <[email protected]>--- arch/riscv/kernel/entry.S | 4 ++++ arch/riscv/kernel/traps.c | 4 ++++ 2 files changed, 8 insertions(+)arch/riscv/kernel/entry.S +0
-0
arch/riscv/kernel/entry.S handle_syscall_trace_exit: · line 408
Quoted 5 lines
arch/riscv/kernel/entry.S +2
-0@@ -404,6 +404,10 @@ handle_syscall_trace_exit: #ifdef CONFIG_VMAP_STACK handle_kernel_stack_overflow: +1: la sp, spin_shadow_stack + amoswap.w sp, sp, (sp)
If CONFIG_64BIT=y, it would be broken. Because we only hold 32bit of the sp, and the next loop would get the wrong sp value of &spin_shadow_stack.
Here is the correction.
-----
arch/riscv/include/asm/asm.h +1 -0
@@ -23,6 +23,7 @@#define REG_L __REG_SEL(ld, lw)#define REG_S __REG_SEL(sd, sw)#define REG_SC __REG_SEL(sc.d, sc.w) +#define REG_AMOSWAP __REG_SEL(amoswap.d, amoswap.w)#define REG_ASM __REG_SEL(.dword, .word)#define SZREG __REG_SEL(8, 4)#define LGREG __REG_SEL(3, 2)
arch/riscv/kernel/entry.S +2 -0
@@ -404,6 +404,10 @@ handle_syscall_trace_exit: #ifdef CONFIG_VMAP_STACK handle_kernel_stack_overflow: +1: la sp, spin_shadow_stack + /* Reuse the address as the spin value, so they must be all
XLEN's width. */
+ REG_AMOSWAP sp, sp, (sp)
+ bnez sp, 1b
+
la sp, shadow_stack
addi sp, sp, SHADOW_OVERFLOW_STACK_SIZE
arch/riscv/kernel/traps.c +4 -0
@@ -221,11 +221,15 @@ asmlinkage unsigned long get_overflow_stack(void)OVERFLOW_STACK_SIZE ; } +unsigned long spin_shadow_stack =0 ; + asmlinkage void handle_bad_stack(struct pt_regs *regs) {unsigned long tsk_stk = (unsigned long )current->stack;unsigned long ovf_stk = (unsigned long )this_cpu_ptr (overflow_stack); +smp_store_release (&spin_shadow_stack,0 ); +console_verbose ();pr_emerg ("Insufficient stack space to handle exception!\n" );
Quoted 26 lines
arch/riscv/kernel/entry.S +2
-0+ bnez sp, 1b + la sp, shadow_stack addi sp, sp, SHADOW_OVERFLOW_STACK_SIZEarch/riscv/kernel/traps.c +4
-1@@ -221,11 +221,15 @@ asmlinkage unsigned long get_overflow_stack(void)OVERFLOW_STACK_SIZE ; } +atomic_t spin_shadow_stack; + asmlinkage void handle_bad_stack(struct pt_regs *regs) {unsigned long tsk_stk = (unsigned long )current->stack;unsigned long ovf_stk = (unsigned long )this_cpu_ptr (overflow_stack); +atomic_set_release (&spin_shadow_stack,0 ); +console_verbose ();pr_emerg ("Insufficient stack space to handle exception!\n" ); --
Quoted 2 lines
2.37.2
--
Best Regards
Guo Ren
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 25 lines · Guo Ren
Quoted 24 lines · Jisheng Zhang
Currently, when detecting vmap stack overflow, riscv firstly switches to the so called shadow stack, then use this shadow stack to call the
get_overflow_stack()to get the overflow stack. However, there's a race here if two or more harts use the same shadow stack at the same time.To solve this race, we introduce
spin_shadow_stackatomic var, which will make the shadow stack usage serialized.Fixes 31da94c25aea ("riscv: add VMAP_STACK overflow detection")Signed-off-by Jisheng Zhang <[email protected]>Suggested-by Guo Ren <[email protected]>--- arch/riscv/kernel/entry.S | 4 ++++ arch/riscv/kernel/traps.c | 4 ++++ 2 files changed, 8 insertions(+)arch/riscv/kernel/entry.S +0
-0@@ -404,6 +404,10 @@ handle_syscall_trace_exit:
Quoted 7 lines
arch/riscv/kernel/entry.S handle_syscall_trace_exit:
Quoted 4 lines
arch/riscv/kernel/entry.S +2
-0#ifdef CONFIG_VMAP_STACK handle_kernel_stack_overflow: +1: la sp, spin_shadow_stack + amoswap.w sp, sp, (sp)If
CONFIG_64BIT=y, it would be broken. Because we only hold 32bit of the sp, and the next loop would get the wrong sp value of&spin_shadow_stack.
Hi Guo,
Don't worry about it. the spin_shadow_stack is just a flag used for "spin", if hart is allowed to used the shadow_stack, we load its address in next instruction by "la sp, shadow_stack". But I agree with use unsigned int instead of atomic_t, and use smp_store_release directly. V2 has been sent out, could you please review it?
Thanks
Hi Jisheng,
Quoted 39 lines · Jisheng Zhang
Currently, when detecting vmap stack overflow, riscv firstly switches to the so called shadow stack, then use this shadow stack to call the
get_overflow_stack()to get the overflow stack. However, there's a race here if two or more harts use the same shadow stack at the same time.To solve this race, we introduce
spin_shadow_stackatomic var, which will make the shadow stack usage serialized.Fixes 31da94c25aea ("riscv: add VMAP_STACK overflow detection")Signed-off-by Jisheng Zhang <[email protected]>Suggested-by Guo Ren <[email protected]>--- arch/riscv/kernel/entry.S | 4 ++++ arch/riscv/kernel/traps.c | 4 ++++ 2 files changed, 8 insertions(+)arch/riscv/kernel/entry.S +4
-0@@ -404,6 +404,10 @@ handle_syscall_trace_exit: #ifdef CONFIG_VMAP_STACK handle_kernel_stack_overflow: +1: la sp, spin_shadow_stack + amoswap.w sp, sp, (sp) + bnez sp, 1b + la sp, shadow_stack addi sp, sp, SHADOW_OVERFLOW_STACK_SIZEarch/riscv/kernel/traps.c +0
-0@@ -221,11 +221,15 @@ asmlinkage unsigned long get_overflow_stack(void)OVERFLOW_STACK_SIZE ; }
arch/riscv/kernel/traps.c asmlinkage unsigned long get_overflow_stack(void) · line 232
Quoted 10 lines
arch/riscv/kernel/traps.c +4
-0+atomic_t spin_shadow_stack; + asmlinkage void handle_bad_stack(struct pt_regs *regs) {unsigned long tsk_stk = (unsigned long )current->stack;unsigned long ovf_stk = (unsigned long )this_cpu_ptr (overflow_stack); +atomic_set_release (&spin_shadow_stack,0 ); +
Have not really looked the details: should there be a matching acquire?
Andrea
Quoted 6 lines
arch/riscv/kernel/traps.c +0
-0console_verbose ();pr_emerg ("Insufficient stack space to handle exception!\n" );Signature · 1 line
2.37.2
Quoted 35 lines · Jisheng Zhang
Quoted 25 lines · Guo Ren
Quoted 24 lines · Jisheng Zhang
Currently, when detecting vmap stack overflow, riscv firstly switches to the so called shadow stack, then use this shadow stack to call the
get_overflow_stack()to get the overflow stack. However, there's a race here if two or more harts use the same shadow stack at the same time.To solve this race, we introduce
spin_shadow_stackatomic var, which will make the shadow stack usage serialized.Fixes 31da94c25aea ("riscv: add VMAP_STACK overflow detection")Signed-off-by Jisheng Zhang <[email protected]>Suggested-by Guo Ren <[email protected]>--- arch/riscv/kernel/entry.S | 4 ++++ arch/riscv/kernel/traps.c | 4 ++++ 2 files changed, 8 insertions(+)arch/riscv/kernel/entry.S +0
-0@@ -404,6 +404,10 @@ handle_syscall_trace_exit:Quoted 7 lines
arch/riscv/kernel/entry.S handle_syscall_trace_exit:
Quoted 4 lines
arch/riscv/kernel/entry.S +2
-0#ifdef CONFIG_VMAP_STACK handle_kernel_stack_overflow: +1: la sp, spin_shadow_stack + amoswap.w sp, sp, (sp)If
CONFIG_64BIT=y, it would be broken. Because we only hold 32bit of the sp, and the next loop would get the wrong sp value of&spin_shadow_stack.
Quoted 5 lines
Hi Guo,
Don't worry about it. the
spin_shadow_stackis just a flag used for "spin", if hart is allowed to used theshadow_stack, we load its address in next instruction by "la sp,shadow_stack".
Haha, yes, my brain is at fault :)
Quoted 3 lines
But I agree with use unsigned int instead of
atomic_t, and usesmp_store_releasedirectly. V2 has been sent out, could you please review it?
Okay
Quoted 2 lines
Thanks
Signature · 2 lines
Best Regards Guo Ren
Quoted 51 lines · Andrea Parri
Hi Jisheng,
Quoted 47 lines · Jisheng Zhang
Currently, when detecting vmap stack overflow, riscv firstly switches to the so called shadow stack, then use this shadow stack to call the
get_overflow_stack()to get the overflow stack. However, there's a race here if two or more harts use the same shadow stack at the same time.To solve this race, we introduce
spin_shadow_stackatomic var, which will make the shadow stack usage serialized.Fixes 31da94c25aea ("riscv: add VMAP_STACK overflow detection")Signed-off-by Jisheng Zhang <[email protected]>Suggested-by Guo Ren <[email protected]>--- arch/riscv/kernel/entry.S | 4 ++++ arch/riscv/kernel/traps.c | 4 ++++ 2 files changed, 8 insertions(+)arch/riscv/kernel/entry.S +4
-0@@ -404,6 +404,10 @@ handle_syscall_trace_exit: #ifdef CONFIG_VMAP_STACK handle_kernel_stack_overflow: +1: la sp, spin_shadow_stack + amoswap.w sp, sp, (sp) + bnez sp, 1b + la sp, shadow_stack addi sp, sp, SHADOW_OVERFLOW_STACK_SIZEarch/riscv/kernel/traps.c +2
-0@@ -221,11 +221,15 @@ asmlinkage unsigned long get_overflow_stack(void)OVERFLOW_STACK_SIZE ; } +atomic_t spin_shadow_stack; + asmlinkage void handle_bad_stack(struct pt_regs *regs) {unsigned long tsk_stk = (unsigned long )current->stack;unsigned long ovf_stk = (unsigned long )this_cpu_ptr (overflow_stack);
Quoted 4 lines
arch/riscv/kernel/traps.c asmlinkage unsigned long get_overflow_stack(void) · line 232
Quoted 2 lines
arch/riscv/kernel/traps.c +2
-0+atomic_set_release (&spin_shadow_stack,0 ); +Have not really looked the details: should there be a matching acquire?
I use atomic_set_release here, because I need earlier memory operations finished to make sure the sp is ready then set the spin flag.
The following memory operations order is not important, because we just care about sp value.
Also, we use relax amoswap before, because sp has naturelly dependency. But giving them RCsc is okay here, because we don't care about performance here.
eg:
handle_kernel_stack_overflow:
+1: la sp, spin_shadow_stack + amoswap.w.aqrl sp, sp, (sp) + bnez sp, 1b
+
....
+ smp_store_release(&spin_shadow_stack, 0);
+ smp_mb();
Quoted 10 lines
Andrea
Quoted 6 lines
arch/riscv/kernel/traps.c +0
-1console_verbose ();pr_emerg ("Insufficient stack space to handle exception!\n" ); --2.37.2
--
Best Regards
Guo Ren
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
+
atomic_set_release(&spin_shadow_stack, 0);Have not really looked the details: should there be a matching acquire?
I use
atomic_set_releasehere, because I need earlier memory operations finished to make sure the sp is ready then set the spin flag.
Quoted 6 lines
The following memory operations order is not important, because we just care about sp value.
Also, we use relax amoswap before, because sp has naturelly dependency. But giving them RCsc is okay here, because we don't care about performance here.
Thanks for the clarification.
I'm not really suggesting to add unneeded synchronization, even more so in local/private constructs as in this case. It just felt odd to see the release without a pairing acquire, so I asked. ;-)
Thanks,
Andrea
Review
Quoted 9 lines
eg:
handle_kernel_stack_overflow:
+1: la sp,spin_shadow_stack(quoted diff) +5
-0+ amoswap.w.aqrl sp, sp, (sp) + bnez sp, 1b + + smp_store_release(&spin_shadow_stack, 0); + smp_mb();
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 15 lines · Andrea Parri
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
+
atomic_set_release(&spin_shadow_stack, 0);Have not really looked the details: should there be a matching acquire?
I use
atomic_set_releasehere, because I need earlier memory operations finished to make sure the sp is ready then set the spin flag.Quoted 6 lines
The following memory operations order is not important, because we just care about sp value.
Also, we use relax amoswap before, because sp has naturelly dependency. But giving them RCsc is okay here, because we don't care about performance here.
Quoted 20 lines
Thanks for the clarification.
I'm not really suggesting to add unneeded synchronization, even more so in local/private constructs as in this case. It just felt odd to see the release without a pairing acquire, so I asked. ;-)
Thanks,
AndreaQuoted 9 lines
eg:
handle_kernel_stack_overflow:
+1: la sp,spin_shadow_stack(quoted diff) +5
-0+ amoswap.w.aqrl sp, sp, (sp) + bnez sp, 1b + + smp_store_release(&spin_shadow_stack, 0); + smp_mb();
Hi Andrea, Zhang, Guo:
Consider this implementation:) smp_store_mb(&spin_shadow_stack, 0);
Thanks,
Tong.
Quoted 6 lines
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv.
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Hi Tong,
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
I use
atomic_set_releasehere, because I need earlier memory operations finished to make sure the sp is ready then set the spin flag.Consider this implementation:) smp_store_mb(&spin_shadow_stack, 0);
smp_store_mb() has "WRITE_ONCE(); smp_mb()" semantics; so it doesn't guarantee that the store to spin_shadow_stack is ordered after program -order earlier memory accesses.
Andrea
Quoted 8 lines · Andrea Parri
Hi Tong,
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
I use
atomic_set_releasehere, because I need earlier memory operations finished to make sure the sp is ready then set the spin flag.Consider this implementation:)
Quoted 8 lines
Quoted 1 line
smp_store_mb(&spin_shadow_stack, 0);
smp_store_mb()has "WRITE_ONCE();smp_mb()" semantics; so it doesn't guarantee that the store tospin_shadow_stackis ordered after program -order earlier memory accesses.Andrea
.
Hi Andrea:
IIUC, the earlier memory access amoswap.aqrl, here .aqrl guarantee it.
But anyway, consider we don't care about performance here, using smp_store_release()(add barrier()) surely right.
Thanks,
Tong.
Quoted 16 lines · Andrea Parri
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
+
atomic_set_release(&spin_shadow_stack, 0);Have not really looked the details: should there be a matching acquire?
I use
atomic_set_releasehere, because I need earlier memory operations finished to make sure the sp is ready then set the spin flag.Quoted 6 lines
The following memory operations order is not important, because we just care about sp value.
Also, we use relax amoswap before, because sp has naturelly dependency. But giving them RCsc is okay here, because we don't care about performance here.
Quoted 5 lines
Thanks for the clarification.
I'm not really suggesting to add unneeded synchronization, even more so in local/private constructs as in this case. It just felt odd to see the release without a pairing acquire, so I asked. ;-)
Okay, let's keep:
handle_kernel_stack_overflow:
+1: la sp, spin_shadow_stack + amoswap.w sp, sp, (sp) + bnez sp, 1b
+
....
+ smp_store_release(&spin_shadow_stack, 0);
Quoted 14 lines
Thanks,
AndreaQuoted 9 lines
eg:
handle_kernel_stack_overflow:
+1: la sp,spin_shadow_stack(quoted diff) +5
-0+ amoswap.w.aqrl sp, sp, (sp) + bnez sp, 1b + + smp_store_release(&spin_shadow_stack, 0); + smp_mb();
Signature · 2 lines
Best Regards Guo Ren
Quoted 18 lines · Tong Tiangen
Quoted 14 lines · Andrea Parri
Hi Tong,
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
I use
atomic_set_releasehere, because I need earlier memory operations finished to make sure the sp is ready then set the spin flag.Consider this implementation:)
smp_store_mb(&spin_shadow_stack, 0);
smp_store_mb()has "WRITE_ONCE();smp_mb()" semantics; so it doesn't guarantee that the store tospin_shadow_stackis ordered after program -order earlier memory accesses.
Quoted 8 lines
Quoted 2 lines
Andrea
.Hi Andrea:
IIUC, the earlier memory access amoswap.aqrl, here .aqrl guarantee it. But anyway, consider we don't care about performance here, using smp_store_release()(add
barrier()) surely right.
We use smp_store_release() is for:
//load per-cpu overflow stack
REG_L sp, -8(sp)Not amoswap.
Actually, amoswap.aqrl guarantees nothing because all instructions depend on the sp register.
Quoted 3 lines
Thanks,
Tong.
Signature · 2 lines
Best Regards Guo Ren
Quoted 22 lines · Guo Ren
Quoted 21 lines · Tong Tiangen
Quoted 16 lines · Andrea Parri
Hi Tong,
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
I use
atomic_set_releasehere, because I need earlier memory operations finished to make sure the sp is ready then set the spin flag.Consider this implementation:)
smp_store_mb(&spin_shadow_stack, 0);
smp_store_mb()has "WRITE_ONCE();smp_mb()" semantics; so it doesn't guarantee that the store tospin_shadow_stackis ordered after program -order earlier memory accesses.Andrea
.
Quoted 8 lines
Quoted 5 lines
Hi Andrea:
IIUC, the earlier memory access amoswap.aqrl, here .aqrl guarantee it. But anyway, consider we don't care about performance here, using smp_store_release()(add
barrier()) surely right.We use
smp_store_release()is for://load per-cpu overflow stack REG_L sp, -8(sp)
Oh, missing that, you're right. This guarantee the spin flag is set after the sp is used.
Quoted 5 lines
Not amoswap.
Actually, amoswap.aqrl guarantees nothing because all instructions depend on the sp register.
right.
Thanks,
Tong.
Quoted 7 lines
Quoted 4 lines
Thanks,
Tong.