stackleak: allow to specify arch specific stackleak poison function
Heiko Carstens
maintainercover stackleak: allow to specify arch specific stackleak poison function
Factor out the code that fills the stack with the stackleak poison value in order to allow architectures to provide a faster implementation.
Use this to provide an s390 specific implementation which can fill the stack with the poison value much faster (factor of ~10 compared to the current version).
Note that the s390 stackleak support is currently only available via linux-next (as of today), and the s390 kernel tree at kernel.org[1]. Therefore, if there are no objections, I'd like to add these two patches to the s390 tree, so they can go upstream via the next merge window together with the s390 support.
Thanks,
Heiko
Heiko Carstens (2):
stackleak: allow to specify arch specific stackleak poison function s390/stackleak: provide fast __stackleak_poison() implementation
arch/s390/include/asm/processor.h | 35 +++++++++++++++++++++++++++++++ kernel/stackleak.c | 17 +++++++++++---- 2 files changed, 48 insertions(+), 4 deletions(-)
Signature · 1 line
2.37.2
Heiko Carstens
maintainerPATCH 1/2 stackleak: allow to specify arch specific stackleak poison function
Factor out the code that fills the stack with the stackleak poison value in order to allow architectures to provide a faster implementation.
--- kernel/stackleak.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-)
kernel/stackleak.c +13 -4
@@ -70,6 +70,18 @@ late_initcall(stackleak_sysctls_init);#define skip_erasing () false#endif /* CONFIG_STACKLEAK_RUNTIME_DISABLE */ +#ifndef __stackleak_poison +static __always_inline void __stackleak_poison(unsigned long erase_low, +unsigned long erase_high, +unsigned long poison) +{ +while (erase_low < erase_high) { + *(unsigned long *)erase_low = poison; + erase_low +=sizeof (unsigned long ); + } +} +#endif + static __always_inline void __stackleak_erase(bool on_task_stack) {const unsigned long task_stack_low =stackleak_task_low_bound (current);
@@ -101,10 +113,7 @@ static __always_inline void __stackleak_erase(bool on_task_stack)else erase_high = task_stack_high; -while (erase_low < erase_high) { - *(unsigned long *)erase_low =STACKLEAK_POISON ; - erase_low +=sizeof (unsigned long ); - } +__stackleak_poison (erase_low, erase_high,STACKLEAK_POISON );/* Reset the 'lowest_stack' value for the next syscall */ current->lowest_stack = task_stack_high;
Signature · 1 line
2.37.2
Quoted 5 lines · Heiko Carstens
Factor out the code that fills the stack with the stackleak poison value in order to allow architectures to provide a faster implementation.
Acked-by Vasily Gorbik <[email protected]>Signed-off-by Heiko Carstens <[email protected]>
As on patch 2, it might be nicer to have a noinstr-safe memset64() and use that directly, but I don't have strong feelings either way, and I'll defer to Kees's judgement:
Thanks,
Mark.
Quoted 42 lines
--- kernel/stackleak.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-)kernel/stackleak.c +13
-4@@ -70,6 +70,18 @@ late_initcall(stackleak_sysctls_init);#define skip_erasing () false#endif /* CONFIG_STACKLEAK_RUNTIME_DISABLE */ +#ifndef __stackleak_poison +static __always_inline void __stackleak_poison(unsigned long erase_low, +unsigned long erase_high, +unsigned long poison) +{ +while (erase_low < erase_high) { + *(unsigned long *)erase_low = poison; + erase_low +=sizeof (unsigned long ); + } +} +#endif + static __always_inline void __stackleak_erase(bool on_task_stack) {const unsigned long task_stack_low =stackleak_task_low_bound (current);@@ -101,10 +113,7 @@ static __always_inline void __stackleak_erase(bool on_task_stack)else erase_high = task_stack_high; -while (erase_low < erase_high) { - *(unsigned long *)erase_low =STACKLEAK_POISON ; - erase_low +=sizeof (unsigned long ); - } +__stackleak_poison (erase_low, erase_high,STACKLEAK_POISON );/* Reset the 'lowest_stack' value for the next syscall */ current->lowest_stack = task_stack_high;Signature · 1 line
2.37.2
Heiko Carstens
maintainerQuoted 10 lines · Mark Rutland
Quoted 5 lines · Heiko Carstens
Factor out the code that fills the stack with the stackleak poison value in order to allow architectures to provide a faster implementation.
Acked-by Vasily Gorbik <[email protected]>Signed-off-by Heiko Carstens <[email protected]>As on patch 2, it might be nicer to have a noinstr-safe
memset64()and use that directly, but I don't have strong feelings either way, and I'll defer to Kees's judgement:
Wouldn't that enforce that memset64() wouldn't be allowed to have an own stackframe, since otherwise it would write poison values to it, since we have
if (on_task_stack) erase_high = current_stack_pointer;
in __stackleak_erase()?
That was actually my motiviation to make this s390 optimization an always inline asm.
Besides that this wouldn't be a problem for at least s390, since memset64() is an asm function which comes whithout the need for a stackframe, but on the other hand this would add a quite subtle requirement to memset64(), if I'm not mistaken.
Quoted 12 lines · Heiko Carstens
Quoted 10 lines · Mark Rutland
Quoted 5 lines · Heiko Carstens
Factor out the code that fills the stack with the stackleak poison value in order to allow architectures to provide a faster implementation.
Acked-by Vasily Gorbik <[email protected]>Signed-off-by Heiko Carstens <[email protected]>As on patch 2, it might be nicer to have a noinstr-safe
memset64()and use that directly, but I don't have strong feelings either way, and I'll defer to Kees's judgement:
Quoted 8 lines
Wouldn't that enforce that
memset64()wouldn't be allowed to have an own stackframe, since otherwise it would write poison values to it, since we haveif (on_task_stack) erase_high = current_stack_pointer;in
__stackleak_erase()?
Yes, sorry -- I was implicitly assuming that a noinstr-safe version would be __always_inline.
Quoted 7 lines
That was actually my motiviation to make this s390 optimization an always inline asm.
Besides that this wouldn't be a problem for at least s390, since
memset64()is an asm function which comes whithout the need for a stackframe, but on the other hand this would add a quite subtle requirement tomemset64(), if I'm not mistaken.
That's a fair enough justification, I think. Thanks for the details!
Thanks,
Mark.
Heiko Carstens
maintainerPATCH 2/2 s390/stackleak: provide fast __stackleak_poison() implementation
In reply to Heiko CarstensProvide an s390 specific __stackleak_poison() implementation which is faster than the generic variant.
For the original implementation with an enforced 4kb stackframe for the getpid() system call the system call overhead increases by a factor of 3 if the stackleak feature is enabled. Using the s390 mvc based variant this is reduced to an increase of 25% instead.
This is within the expected area, since the mvc based implementation is more or less a memset64() variant which comes with similar results. See commit 0b77d6701cf8 ("s390: implement memset16, memset32 & memset64").
--- arch/s390/include/asm/processor.h | 35 +++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+)
arch/s390/include/asm/processor.h +35 -0
@@ -118,6 +118,41 @@ unsigned long vdso_size(void);#define HAVE_ARCH_PICK_MMAP_LAYOUT +#define __stackleak_poison __stackleak_poison +static __always_inline void __stackleak_poison(unsigned long erase_low, +unsigned long erase_high, +unsigned long poison) +{ +unsigned long tmp, count; + + count = erase_high - erase_low; +if (!count) +return ; + asmvolatile ( +" cghi %[count],8\n" +" je 2f\n" +" aghi %[count],-(8+1)\n" +" srlg %[tmp],%[count],8\n" +" ltgr %[tmp],%[tmp]\n" +" jz 1f\n" +"0: stg %[poison],0(%[addr])\n" +" mvc 8(256-8,%[addr]),0(%[addr])\n" +" la %[addr],256(%[addr])\n" +" brctg %[tmp],0b\n" +"1: stg %[poison],0(%[addr])\n" +" larl %[tmp],3f\n" +" ex %[count],0(%[tmp])\n" +" j 4f\n" +"2: stg %[poison],0(%[addr])\n" +" j 4f\n" +"3: mvc 8(1,%[addr]),0(%[addr])\n" +"4:\n" + : [addr]"+&a" (erase_low), [count]"+&d" (count), [tmp]"=&a" (tmp) + : [poison]"d" (poison) + :"memory" ,"cc" + ); +} +/* * Thread structure */
Signature · 1 line
2.37.2
Re: [PATCH 2/2] s390/stackleak: provide fast __stackleak_poison() implementation
Quoted 11 lines · Heiko Carstens
Provide an s390 specific
__stackleak_poison()implementation which is faster than the generic variant.For the original implementation with an enforced 4kb stackframe for the
getpid()system call the system call overhead increases by a factor of 3 if the stackleak feature is enabled. Using the s390 mvc based variant this is reduced to an increase of 25% instead.This is within the expected area, since the mvc based implementation is more or less a
memset64()variant which comes with similar results. See commit0b77d6701cf8("s390: implement memset16, memset32 & memset64").
With that in mind, could we use memset64() directly (if we made it noninstr-safe)?
Mark.
Quoted 56 lines
Reviewed-by Vasily Gorbik <[email protected]>Signed-off-by Heiko Carstens <[email protected]>--- arch/s390/include/asm/processor.h | 35 +++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+)arch/s390/include/asm/processor.h +35
-0@@ -118,6 +118,41 @@ unsigned long vdso_size(void);#define HAVE_ARCH_PICK_MMAP_LAYOUT +#define __stackleak_poison __stackleak_poison +static __always_inline void __stackleak_poison(unsigned long erase_low, +unsigned long erase_high, +unsigned long poison) +{ +unsigned long tmp, count; + + count = erase_high - erase_low; +if (!count) +return ; + asmvolatile ( +" cghi %[count],8\n" +" je 2f\n" +" aghi %[count],-(8+1)\n" +" srlg %[tmp],%[count],8\n" +" ltgr %[tmp],%[tmp]\n" +" jz 1f\n" +"0: stg %[poison],0(%[addr])\n" +" mvc 8(256-8,%[addr]),0(%[addr])\n" +" la %[addr],256(%[addr])\n" +" brctg %[tmp],0b\n" +"1: stg %[poison],0(%[addr])\n" +" larl %[tmp],3f\n" +" ex %[count],0(%[tmp])\n" +" j 4f\n" +"2: stg %[poison],0(%[addr])\n" +" j 4f\n" +"3: mvc 8(1,%[addr]),0(%[addr])\n" +"4:\n" + : [addr]"+&a" (erase_low), [count]"+&d" (count), [tmp]"=&a" (tmp) + : [poison]"d" (poison) + :"memory" ,"cc" + ); +} +/* * Thread structure */ Signature · 1 line
2.37.2
Heiko Carstens
maintainerQuoted 18 lines · Heiko Carstens
Factor out the code that fills the stack with the stackleak poison value in order to allow architectures to provide a faster implementation.
Use this to provide an s390 specific implementation which can fill the stack with the poison value much faster (factor of ~10 compared to the current version).
Note that the s390 stackleak support is currently only available via linux-next (as of today), and the s390 kernel tree at kernel.org[1]. Therefore, if there are no objections, I'd like to add these two patches to the s390 tree, so they can go upstream via the next merge window together with the s390 support.
Thanks,
Heiko
Quoted 7 lines
Heiko Carstens (2):
stackleak: allow to specify arch specific stackleak poison function s390/stackleak: provide fast__stackleak_poison()implementationarch/s390/include/asm/processor.h | 35 +++++++++++++++++++++++++++++++ kernel/stackleak.c | 17 +++++++++++---- 2 files changed, 48 insertions(+), 4 deletions(-)
Given that this series seems to be straight forward, and Mark already gave his Ack we're going to put these two patches on the s390 git tree, even though there was no response from Kees yet.
If there will be any complaints I'm sure we can easily solve that.