Skip to contentSkip to conversation

powerpc barrier_nospec

RFC PATCH In review Michal Suchanek 7 participants 30 messages Mar 13, 2018 – Mar 23, 2018 PowerPC

lore
Share & tools
lore

Michal Suchanek

author
#0 lore raw

cover powerpc barrier_nospec

Hello,

this is patchset adding barrier_nospec on powerpc. It is based on the out-of-tree gmb() patch and the existing rfi patches.

I do not have the tests for the Spectre/Meltdown issues available so this is untested.

Feedback on the general approach as well as actual effectivity is welcome.

Thanks

Michal

Michal Suchanek (8):
powerpc: Add barrier_nospec
powerpc: Use barrier_nospec in copy_from_user powerpc/64: Use barrier_nospec in syscall entry powerpc/64s: Add support for ori barrier_nospec powerpc/64: Patch barrier_nospec in modules powerpc/64: barrier_nospec: Add debugfs trigger powerpc/64s: barrier_nospec: Add hcall triggerr powerpc/64: barrier_nospec: Add commandline trigger

 arch/powerpc/include/asm/barrier.h        |  9 ++++
 arch/powerpc/include/asm/feature-fixups.h |  9 ++++
 arch/powerpc/include/asm/setup.h          | 11 +++++
 arch/powerpc/include/asm/uaccess.h        | 11 ++++-
 arch/powerpc/kernel/entry_64.S            |  3 ++
 arch/powerpc/kernel/module.c              |  6 +++
 arch/powerpc/kernel/setup_64.c            | 72 +++++++++++++++++++++++++++++++
 arch/powerpc/kernel/vmlinux.lds.S         |  7 +++
 arch/powerpc/lib/feature-fixups.c         | 38 ++++++++++++++++
 arch/powerpc/platforms/pseries/setup.c    | 38 ++++++++++------
 10 files changed, 190 insertions(+), 14 deletions(-)
Signature · 1 line
2.13.6
author
#1 lore raw

RFC PATCH 1/8 powerpc: Add barrier_nospec

Copypasta from original gmb() and rfi implementation

Signed-off-by Michal Suchanek <[email protected]>
---
 arch/powerpc/include/asm/barrier.h | 9 +++++++++
 1 file changed, 9 insertions(+)
arch/powerpc/include/asm/barrier.h +9 -0
index 10daa1d56e0a..8e47b3abe405 100644
@@ -75,6 +75,15 @@ do {									\
 	___p1;								\
 })
 
+/* TODO: add patching so this can be disabled */
+/* Prevent speculative execution past this barrier. */
+#define barrier_nospec_asm ori 31,31,0
+#ifdef __ASSEMBLY__
+#define barrier_nospec barrier_nospec_asm
+#else
+#define barrier_nospec() __asm__ __volatile__ (stringify_in_c(barrier_nospec_asm) : : :)
+#endif
+
 #include <asm-generic/barrier.h>
 
 #endif /* _ASM_POWERPC_BARRIER_H */
Signature · 1 line
2.13.6
#2 lore raw

Re: [PATCH RFC 1/8] powerpc: Add barrier_nospec

Quoted 1 line · Michal Suchanek

Copypasta from original gmb() and rfi implementation

Actual real changelogs would be very welcome. Seeing as I've not seen these mythical RFI patches, this leaves one quite puzzled :-)

author
#3 lore raw

cover powerpc barrier_nospec

Yes, it is good idea to add some commit messages.

Also I rebased the patches on top v3 of series

Setup RFI flush after PowerVM LPM migration

Thanks

Michal

Michal Suchanek (9):
powerpc: Add barrier_nospec
powerpc: Use barrier_nospec in copy_from_user powerpc/64: Use barrier_nospec in syscall entry powerpc/64s: Use barrier_nospec in RFI_FLUSH_SLOT powerpc/64s: Add support for ori barrier_nospec patching powerpc/64: Patch barrier_nospec in modules powerpc/64: barrier_nospec: Add debugfs trigger powerpc/64s: barrier_nospec: Add hcall triggerr powerpc/64: barrier_nospec: Add commandline trigger

 arch/powerpc/include/asm/barrier.h        |  9 ++++
 arch/powerpc/include/asm/exception-64s.h  |  2 +-
 arch/powerpc/include/asm/feature-fixups.h |  9 ++++
 arch/powerpc/include/asm/setup.h          | 11 ++++
 arch/powerpc/include/asm/uaccess.h        | 11 +++-
 arch/powerpc/kernel/entry_64.S            |  3 ++
 arch/powerpc/kernel/module.c              |  6 +++
 arch/powerpc/kernel/setup_64.c            | 87 +++++++++++++++++++++++++++++++
 arch/powerpc/kernel/vmlinux.lds.S         |  7 +++
 arch/powerpc/lib/feature-fixups.c         | 47 ++++++++++++++---
 arch/powerpc/platforms/pseries/mobility.c |  2 +-
 arch/powerpc/platforms/pseries/pseries.h  |  2 +-
 arch/powerpc/platforms/pseries/setup.c    | 37 +++++++++----
 13 files changed, 213 insertions(+), 20 deletions(-)
Signature · 1 line
2.13.6
author
#4 lore raw

RFC PATCH 1/9 powerpc: Add barrier_nospec

When the firmware supports it an otherwise useless combination of ORI instruction arguments is interpreted as speculation barrier. Implement barrier_nospec using this instruction.

Based on the out-of-tree gmb() implementation.

Signed-off-by Michal Suchanek <[email protected]>
---
 arch/powerpc/include/asm/barrier.h | 9 +++++++++
 1 file changed, 9 insertions(+)
arch/powerpc/include/asm/barrier.h +9 -0
index 10daa1d56e0a..8e47b3abe405 100644
@@ -75,6 +75,15 @@ do {									\
 	___p1;								\
 })
 
+/* TODO: add patching so this can be disabled */
+/* Prevent speculative execution past this barrier. */
+#define barrier_nospec_asm ori 31,31,0
+#ifdef __ASSEMBLY__
+#define barrier_nospec barrier_nospec_asm
+#else
+#define barrier_nospec() __asm__ __volatile__ (stringify_in_c(barrier_nospec_asm) : : :)
+#endif
+
 #include <asm-generic/barrier.h>
 
 #endif /* _ASM_POWERPC_BARRIER_H */
Signature · 1 line
2.13.6
author
#5 lore raw

RFC PATCH 2/9 powerpc: Use barrier_nospec in copy_from_user

In reply to Michal Suchanek

This is based on x86 patch doing the same.

Signed-off-by Michal Suchanek <[email protected]>
---
 arch/powerpc/include/asm/uaccess.h | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
arch/powerpc/include/asm/uaccess.h +10 -1
index 51bfeb8777f0..af9b0e731f46 100644
@@ -248,6 +248,7 @@ do {								\
 	__chk_user_ptr(ptr);					\
 	if (!is_kernel_addr((unsigned long)__gu_addr))		\
 		might_fault();					\
+	barrier_nospec();					\
 	__get_user_size(__gu_val, __gu_addr, (size), __gu_err);	\
 	(x) = (__typeof__(*(ptr)))__gu_val;			\
 	__gu_err;						\

@@ -258,8 +259,10 @@ do {								\
 	long __gu_err = -EFAULT;					\
 	unsigned long  __gu_val = 0;					\
 	const __typeof__(*(ptr)) __user *__gu_addr = (ptr);		\
+	int can_access = access_ok(VERIFY_READ, __gu_addr, (size));	\
 	might_fault();							\
-	if (access_ok(VERIFY_READ, __gu_addr, (size)))			\
+	barrier_nospec();						\
+	if (can_access)							\
 		__get_user_size(__gu_val, __gu_addr, (size), __gu_err);	\
 	(x) = (__force __typeof__(*(ptr)))__gu_val;				\
 	__gu_err;							\

@@ -271,6 +274,7 @@ do {								\
 	unsigned long __gu_val;					\
 	const __typeof__(*(ptr)) __user *__gu_addr = (ptr);	\
 	__chk_user_ptr(ptr);					\
+	barrier_nospec();					\
 	__get_user_size(__gu_val, __gu_addr, (size), __gu_err);	\
 	(x) = (__force __typeof__(*(ptr)))__gu_val;			\
 	__gu_err;						\

@@ -298,15 +302,19 @@ static inline unsigned long raw_copy_from_user(void *to,
 
 		switch (n) {
 		case 1:
+			barrier_nospec();
 			__get_user_size(*(u8 *)to, from, 1, ret);
 			break;
 		case 2:
+			barrier_nospec();
 			__get_user_size(*(u16 *)to, from, 2, ret);
 			break;
 		case 4:
+			barrier_nospec();
 			__get_user_size(*(u32 *)to, from, 4, ret);
 			break;
 		case 8:
+			barrier_nospec();
 			__get_user_size(*(u64 *)to, from, 8, ret);
 			break;
 		}

@@ -314,6 +322,7 @@ static inline unsigned long raw_copy_from_user(void *to,
 			return 0;
 	}
 
+	barrier_nospec();
 	return __copy_tofrom_user((__force void __user *)to, from, n);
 }
Signature · 1 line
2.13.6
#6 lore raw

Re: [PATCH RFC rebase 2/9] powerpc: Use barrier_nospec in copy_from_user

Quoted 8 lines · Michal Suchanek

This is based on x86 patch doing the same.

Signed-off-by Michal Suchanek <[email protected]>

---

arch/powerpc/include/asm/uaccess.h +0 -0
@@ -258,8 +259,10 @@ do {                                                               \
        long __gu_err = -EFAULT;                                        \

arch/powerpc/include/asm/uaccess.h do { \ · line 268

Quoted 10 lines
arch/powerpc/include/asm/uaccess.h +3 -1
        unsigned long  __gu_val = 0;                                    \
        const __typeof__(*(ptr)) __user *__gu_addr = (ptr);             \
+       int can_access = access_ok(VERIFY_READ, __gu_addr, (size));     \
        might_fault();                                                  \
-       if (access_ok(VERIFY_READ, __gu_addr, (size)))                  \
+       barrier_nospec();                                               \
+       if (can_access)                                                 \
                __get_user_size(__gu_val, __gu_addr, (size), __gu_err); \
        (x) = (__force __typeof__(*(ptr)))__gu_val;                             \
        __gu_err;                                                       \

Is the above really correct? The barrier is *before* the conditional branch that might be mis-predicted.

I don't know how the ppc barrier works, but that sounds completely bogus.

Linus

reviewer
#7 lore raw

Re: [PATCH RFC rebase 2/9] powerpc: Use barrier_nospec in copy_from_user

Quoted 23 lines · Linus Torvalds
Quoted 8 lines · Michal Suchanek

This is based on x86 patch doing the same.

Signed-off-by Michal Suchanek <[email protected]>

---

arch/powerpc/include/asm/uaccess.h +0 -0
@@ -258,8 +259,10 @@ do {                                                               \
        long __gu_err = -EFAULT;                                        \

arch/powerpc/include/asm/uaccess.h do { \ · line 268

Quoted 10 lines
arch/powerpc/include/asm/uaccess.h +3 -1
        unsigned long  __gu_val = 0;                                    \
        const __typeof__(*(ptr)) __user *__gu_addr = (ptr);             \
+       int can_access = access_ok(VERIFY_READ, __gu_addr, (size));     \
        might_fault();                                                  \
-       if (access_ok(VERIFY_READ, __gu_addr, (size)))                  \
+       barrier_nospec();                                               \
+       if (can_access)                                                 \
                __get_user_size(__gu_val, __gu_addr, (size), __gu_err); \
        (x) = (__force __typeof__(*(ptr)))__gu_val;                             \
        __gu_err;                                                       \

Is the above really correct? The barrier is *before* the conditional branch that might be mis-predicted.

Quoted 1 line

I don't know how the ppc barrier works, but that sounds completely bogus.

Yeah it should be after the branch.

I don't have a formal spec for the barrier yet, it should be defined in a hopefully soon to be released revision of the ISA.

But the gist is it will stall execution until any older branches are no longer speculating.

It doesn't order any two arbitrary instructions, such as a comparison and a branch, which I suspect is how Michal was interpreting it.

cheers

author
#8 lore raw

RFC PATCH 3/9 powerpc/64: Use barrier_nospec in syscall entry

In reply to Michal Suchanek

On powerpc syscall entry is done in assembly so patch in an explicit barrier_nospec.

Signed-off-by Michal Suchanek <[email protected]>
---
 arch/powerpc/kernel/entry_64.S | 3 +++
 1 file changed, 3 insertions(+)
arch/powerpc/kernel/entry_64.S +3 -0
index 2cb5109a7ea3..7bfc4cf48af2 100644
@@ -36,6 +36,7 @@
 #include <asm/context_tracking.h>
 #include <asm/tm.h>
 #include <asm/ppc-opcode.h>
+#include <asm/barrier.h>
 #include <asm/export.h>
 #ifdef CONFIG_PPC_BOOK3S
 #include <asm/exception-64s.h>

@@ -159,6 +160,7 @@ system_call:			/* label this so stack traces look sane */
 	andi.	r11,r10,_TIF_SYSCALL_DOTRACE
 	bne	.Lsyscall_dotrace		/* does not return */
 	cmpldi	0,r0,NR_syscalls
+	barrier_nospec
 	bge-	.Lsyscall_enosys
 
 .Lsyscall:

@@ -319,6 +321,7 @@ END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
 	ld	r10,TI_FLAGS(r10)
 
 	cmpldi	r0,NR_syscalls
+	barrier_nospec
 	blt+	.Lsyscall
 
 	/* Return code is already in r3 thanks to do_syscall_trace_enter() */
Signature · 1 line
2.13.6
reviewer
#9 lore raw

Re: [PATCH RFC rebase 3/9] powerpc/64: Use barrier_nospec in syscall entry

Quoted 2 lines · On Thu, 15 Mar 2018 20:15:52 +0100 Michal Suchanek <[email protected]> wrote:

On powerpc syscall entry is done in assembly so patch in an explicit barrier_nospec.

Same comment as Linus for this -- the barriers are before the branch here, so is it possible the branch instruction can be speculative while the index is used to load the syscall table?

Thanks,
Nick

Quoted 34 lines
Signed-off-by Michal Suchanek <[email protected]>
---
 arch/powerpc/kernel/entry_64.S | 3 +++
 1 file changed, 3 insertions(+)
arch/powerpc/kernel/entry_64.S +3 -0
index 2cb5109a7ea3..7bfc4cf48af2 100644
@@ -36,6 +36,7 @@
 #include <asm/context_tracking.h>
 #include <asm/tm.h>
 #include <asm/ppc-opcode.h>
+#include <asm/barrier.h>
 #include <asm/export.h>
 #ifdef CONFIG_PPC_BOOK3S
 #include <asm/exception-64s.h>

@@ -159,6 +160,7 @@ system_call:			/* label this so stack traces look sane */
 	andi.	r11,r10,_TIF_SYSCALL_DOTRACE
 	bne	.Lsyscall_dotrace		/* does not return */
 	cmpldi	0,r0,NR_syscalls
+	barrier_nospec
 	bge-	.Lsyscall_enosys
 
 .Lsyscall:

@@ -319,6 +321,7 @@ END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
 	ld	r10,TI_FLAGS(r10)
 
 	cmpldi	r0,NR_syscalls
+	barrier_nospec
 	blt+	.Lsyscall
 
 	/* Return code is already in r3 thanks to do_syscall_trace_enter() */
author
#10 lore raw

Re: [PATCH RFC rebase 3/9] powerpc/64: Use barrier_nospec in syscall entry

Hello,

Quoted 9 lines · On Fri, 16 Mar 2018 15:18:23 +1000 Nicholas Piggin <[email protected]> wrote:
Quoted 2 lines · On Thu, 15 Mar 2018 20:15:52 +0100 Michal Suchanek <[email protected]> wrote:

On powerpc syscall entry is done in assembly so patch in an explicit barrier_nospec.

Same comment as Linus for this -- the barriers are before the branch here, so is it possible the branch instruction can be speculative while the index is used to load the syscall table?

As far as I understand barriers they separate code before the barrier and code after the barrier.

So inserting barrier_nospec after cmpldi means that the result of the cmpldi has to be known before any instruction following barrier_nospec that depends on the result can be executed.

In many cases it is useful to put the barrier after a branch. It allows the compiler to speculate on the computed value at compile time and if it is constrained optimize out the branch. It may also result in the need to include many barriers and less readable code.

However, you have probably knowledge of the powerpc implementation of the barrier so if the semantic is actually different then please enlighten me.

Thanks

Michal

reviewer
#11 lore raw

Re: [PATCH RFC rebase 3/9] powerpc/64: Use barrier_nospec in syscall entry

Quoted 22 lines · On Fri, 16 Mar 2018 10:15:49 +0100 Michal Suchánek <[email protected]> wrote:

Hello,

Quoted 9 lines · On Fri, 16 Mar 2018 15:18:23 +1000 Nicholas Piggin <[email protected]> wrote:
Quoted 2 lines · On Thu, 15 Mar 2018 20:15:52 +0100 Michal Suchanek <[email protected]> wrote:

On powerpc syscall entry is done in assembly so patch in an explicit barrier_nospec.

Same comment as Linus for this -- the barriers are before the branch here, so is it possible the branch instruction can be speculative while the index is used to load the syscall table?

As far as I understand barriers they separate code before the barrier and code after the barrier.

So inserting barrier_nospec after cmpldi means that the result of the cmpldi has to be known before any instruction following barrier_nospec that depends on the result can be executed.

Quoted 8 lines

In many cases it is useful to put the barrier after a branch. It allows the compiler to speculate on the computed value at compile time and if it is constrained optimize out the branch. It may also result in the need to include many barriers and less readable code.

However, you have probably knowledge of the powerpc implementation of the barrier so if the semantic is actually different then please enlighten me.

I actually don't. I'm assuming we should be able to say that no previous instruction is speculative when a subsequent one is executed.

But the branch instruction itself that is speculated, not the compare.

Usually even if all sources are ready, the pipeline may take in some cycles after a branch, before that branch can finish executing and squash speculation if it was wrong. Perhaps there is only a couple of cycles of instructions that get a chance to reach execution units and disturb any caches, but still there could be some window and I don't think we would have architectural gurantees on that.

I'll try to ask around and see if there's any documentation we can give you yet.

Thanks,
Nick

reviewer
#12 lore raw

Re: [PATCH RFC rebase 3/9] powerpc/64: Use barrier_nospec in syscall entry

In reply to Michal Suchánek

Hi Michal,

Thanks for working on this series in the absence of any documentation.

Quoted 13 lines · Michal Suchánek
Quoted 9 lines · On Fri, 16 Mar 2018 15:18:23 +1000 Nicholas Piggin <[email protected]> wrote:
Quoted 2 lines · On Thu, 15 Mar 2018 20:15:52 +0100 Michal Suchanek <[email protected]> wrote:

On powerpc syscall entry is done in assembly so patch in an explicit barrier_nospec.

Same comment as Linus for this -- the barriers are before the branch here, so is it possible the branch instruction can be speculative while the index is used to load the syscall table?

Quoted 6 lines

As far as I understand barriers they separate code before the barrier and code after the barrier.

So inserting barrier_nospec after cmpldi means that the result of the cmpldi has to be known before any instruction following barrier_nospec that depends on the result can be executed.

That would make sense, but I don't think that's how the barrier's been defined.

I don't have a formal spec for it (yet), but what I do have indicates it only orders older branches vs future instructions.

Quoted 3 lines

However, you have probably knowledge of the powerpc implementation of the barrier so if the semantic is actually different then please enlighten me.

We have some knowledge, but only some :)

It's not necessarily implemented the same way on each chip revision, so it's not entirely clear what the formal semantics will be vs what we are seeing in current implementations. But I think it's safe to say it should always go after the branch that might be speculatively executed.

Will try and get some better documentation for you.

cheers

#13 lore raw

Re: [PATCH RFC rebase 3/9] powerpc/64: Use barrier_nospec in syscall entry

In reply to Michal Suchánek
Quoted 3 lines · Michal Suchánek

As far as I understand barriers they separate code before the barrier and code after the barrier.

Almost certainly not. Even if you were to do an expensive serialization before the branch, the branch will still predict after the serialization.

The thing is, it doesn't make sense to insert a barrier before a conditional branch for Spectre mitigation.

The problem is not that the data isn't ready for the branch - the problem is that the branch is predicted _regardless_ of the data.

Sure, some micro-architecture might not predict branches at all if they have a stable conditional, so a barrier _can_ make sense.

But fundamentally, good branch prediction - in order to be optimal - has to happen before instructions have even been parsed, much less things like "stable conditional register state" having been decided on. You'll want to do I$ prefetching etc.

So the problem is that even if the data is ready, the branch will be predicted according to some unrelated historical data, and a barrier to make the branch conditional be stable is pointless.

A barrier *after* the branch, making sure that you don't actually start executing instructions past it (even if you might have predicted and fetched stuff past it) *if* you have mis-predicted the previous branch, is what a sane architecture would specify.

Of course, on x86, we mostly tried to avoid branch prediction being the critical problem and having to have barriers by just making it an address generation dependency instead. That should presumably work on powerpc too, since address generation is part of the memory ordering definition. But obviously a microarchitecture *could* end up speculating and just redoing even for memory ordering, and maybe the ppc architects prefer the barrier since they are already used to crazy and not very well architected barriers elsewhere.

Linus

author
#14 lore raw

RFC PATCH 4/9 powerpc/64s: Use barrier_nospec in RFI_FLUSH_SLOT

In reply to Michal Suchanek

The RFI flush support patches the speculation barrier into RFI_FLUSH_SLOT as part of the RFI flush. Use separate barrier_nospec instead.

Signed-off-by Michal Suchanek <[email protected]>
---
 arch/powerpc/include/asm/exception-64s.h | 2 +-
 arch/powerpc/lib/feature-fixups.c        | 9 +++------
 2 files changed, 4 insertions(+), 7 deletions(-)
arch/powerpc/include/asm/exception-64s.h +1 -1
index 471b2274fbeb..bb5a3052b29b 100644
@@ -81,9 +81,9 @@
  * L1-D cache when returning to userspace or a guest.
  */
 #define RFI_FLUSH_SLOT							\
+	barrier_nospec_asm;						\
 	RFI_FLUSH_FIXUP_SECTION;					\
 	nop;								\
-	nop;								\
 	nop
 
 #define RFI_TO_KERNEL							\
arch/powerpc/lib/feature-fixups.c +3 -6
index 35f80ab7cbd8..4cc2f0c5c863 100644
@@ -119,7 +119,7 @@ void do_feature_fixups(unsigned long value, void *fixup_start, void *fixup_end)
 #ifdef CONFIG_PPC_BOOK3S_64
 void do_rfi_flush_fixups(enum l1d_flush_type types)
 {
-	unsigned int instrs[3], *dest;
+	unsigned int instrs[2], *dest;
 	long *start, *end;
 	int i;
 

@@ -128,15 +128,13 @@ void do_rfi_flush_fixups(enum l1d_flush_type types)
 
 	instrs[0] = 0x60000000; /* nop */
 	instrs[1] = 0x60000000; /* nop */
-	instrs[2] = 0x60000000; /* nop */
 
 	if (types & L1D_FLUSH_FALLBACK)
-		/* b .+16 to fallback flush */
-		instrs[0] = 0x48000010;
+		/* b .+12 to fallback flush */
+		instrs[0] = 0x4800000c;
 
 	i = 0;
 	if (types & L1D_FLUSH_ORI) {
-		instrs[i++] = 0x63ff0000; /* ori 31,31,0 speculation barrier */
 		instrs[i++] = 0x63de0000; /* ori 30,30,0 L1d flush*/
 	}
 

@@ -150,7 +148,6 @@ void do_rfi_flush_fixups(enum l1d_flush_type types)
 
 		patch_instruction(dest, instrs[0]);
 		patch_instruction(dest + 1, instrs[1]);
-		patch_instruction(dest + 2, instrs[2]);
 	}
 
 	printk(KERN_DEBUG "rfi-flush: patched %d locations (%s flush)\n", i,
Signature · 1 line
2.13.6
author
#15 lore raw

RFC PATCH 5/9 powerpc/64s: Add support for ori barrier_nospec patching

In reply to Michal Suchanek

Based on the RFI patching. This is required to be able to disable the speculation barrier.

Only one barrier type is supported and it does nothing when the firmware does not enable it. Also re-patching modules is not supported So the only meaningful thing that can be done is patching out the speculation barrier at boot when the user says it is not wanted.

Signed-off-by Michal Suchanek <[email protected]>
---
 arch/powerpc/include/asm/barrier.h        |  4 ++--
 arch/powerpc/include/asm/feature-fixups.h |  9 +++++++++
 arch/powerpc/include/asm/setup.h          |  8 ++++++++
 arch/powerpc/kernel/setup_64.c            | 30 ++++++++++++++++++++++++++++++
 arch/powerpc/kernel/vmlinux.lds.S         |  7 +++++++
 arch/powerpc/lib/feature-fixups.c         | 27 +++++++++++++++++++++++++++
 6 files changed, 83 insertions(+), 2 deletions(-)
arch/powerpc/include/asm/barrier.h +2 -2
index 8e47b3abe405..4079a95e84c2 100644
@@ -75,9 +75,9 @@ do {									\
 	___p1;								\
 })
 
-/* TODO: add patching so this can be disabled */
 /* Prevent speculative execution past this barrier. */
-#define barrier_nospec_asm ori 31,31,0
+#define barrier_nospec_asm SPEC_BARRIER_FIXUP_SECTION;			\
+				nop
 #ifdef __ASSEMBLY__
 #define barrier_nospec barrier_nospec_asm
 #else
arch/powerpc/include/asm/feature-fixups.h +9 -0
index 1e82eb3caabd..9d3382618ffd 100644
@@ -195,11 +195,20 @@ label##3:					       	\
 	FTR_ENTRY_OFFSET 951b-952b;			\
 	.popsection;
 
+#define SPEC_BARRIER_FIXUP_SECTION			\
+953:							\
+	.pushsection __spec_barrier_fixup,"a";		\
+	.align 2;					\
+954:							\
+	FTR_ENTRY_OFFSET 953b-954b;			\
+	.popsection;
+
 
 #ifndef __ASSEMBLY__
 #include <linux/types.h>
 
 extern long __start___rfi_flush_fixup, __stop___rfi_flush_fixup;
+extern long __start___spec_barrier_fixup, __stop___spec_barrier_fixup;
 
 void apply_feature_fixups(void);
 void setup_feature_keys(void);
arch/powerpc/include/asm/setup.h +8 -0
index bbcdf929be54..c7e9e66c2a38 100644
@@ -49,8 +49,16 @@ enum l1d_flush_type {
 	L1D_FLUSH_MTTRIG	= 0x8,
 };
 
+/* These are bit flags */
+enum spec_barrier_type {
+	SPEC_BARRIER_NONE	= 0x1,
+	SPEC_BARRIER_ORI	= 0x2,
+};
+
 void setup_rfi_flush(enum l1d_flush_type, bool enable);
 void do_rfi_flush_fixups(enum l1d_flush_type types);
+void setup_barrier_nospec(enum spec_barrier_type, bool enable);
+void do_barrier_nospec_fixups(enum spec_barrier_type type);
 
 #endif /* !__ASSEMBLY__ */
arch/powerpc/kernel/setup_64.c +30 -0
index 4ec4a27b36a9..767240074cad 100644
@@ -815,6 +815,10 @@ static enum l1d_flush_type enabled_flush_types;
 static void *l1d_flush_fallback_area;
 static bool no_rfi_flush;
 bool rfi_flush;
+enum spec_barrier_type powerpc_barrier_nospec;
+static enum spec_barrier_type barrier_nospec_type;
+static bool no_nospec;
+bool barrier_nospec_enabled;
 
 static int __init handle_no_rfi_flush(char *p)
 {

@@ -900,6 +904,32 @@ void setup_rfi_flush(enum l1d_flush_type types, bool enable)
 		rfi_flush_enable(enable);
 }
 
+void barrier_nospec_enable(bool enable)
+{
+	barrier_nospec_enabled = enable;
+
+	if (enable) {
+		powerpc_barrier_nospec = barrier_nospec_type;
+		do_barrier_nospec_fixups(powerpc_barrier_nospec);
+		on_each_cpu(do_nothing, NULL, 1);
+	} else {
+		powerpc_barrier_nospec = SPEC_BARRIER_NONE;
+		do_barrier_nospec_fixups(powerpc_barrier_nospec);
+	}
+}
+
+void setup_barrier_nospec(enum spec_barrier_type type, bool enable)
+{
+	/*
+	 * Only one barrier type is supported and it does nothing when the
+	 * firmware does not enable it. So the only meaningful thing to do
+	 * here is check the user preference.
+	 */
+	barrier_nospec_type = SPEC_BARRIER_ORI;
+
+	barrier_nospec_enable(!no_nospec && enable);
+}
+
 #ifdef CONFIG_DEBUG_FS
 static int rfi_flush_set(void *data, u64 val)
 {
arch/powerpc/kernel/vmlinux.lds.S +7 -0
index c8af90ff49f0..744b58ff77f1 100644
@@ -139,6 +139,13 @@ SECTIONS
 		*(__rfi_flush_fixup)
 		__stop___rfi_flush_fixup = .;
 	}
+
+	. = ALIGN(8);
+	__spec_barrier_fixup : AT(ADDR(__spec_barrier_fixup) - LOAD_OFFSET) {
+		__start___spec_barrier_fixup = .;
+		*(__spec_barrier_fixup)
+		__stop___spec_barrier_fixup = .;
+	}
 #endif
 
 	EXCEPTION_TABLE(0)
arch/powerpc/lib/feature-fixups.c +27 -0
index 4cc2f0c5c863..dfeb7feeccef 100644
@@ -159,6 +159,33 @@ void do_rfi_flush_fixups(enum l1d_flush_type types)
 		(types &  L1D_FLUSH_MTTRIG)     ? "mttrig type"
 						: "unknown");
 }
+
+void do_barrier_nospec_fixups(enum spec_barrier_type type)
+{
+	unsigned int instr, *dest;
+	long *start, *end;
+	int i;
+
+	start = PTRRELOC(&__start___spec_barrier_fixup),
+	end = PTRRELOC(&__stop___spec_barrier_fixup);
+
+	instr = 0x60000000; /* nop */
+
+	if (type == SPEC_BARRIER_ORI) {
+		pr_info("barrier_nospec: using ORI speculation barrier\n");
+		instr = 0x63ff0000; /* ori 31,31,0 speculation barrier */
+	}
+
+	for (i = 0; start < end; start++, i++) {
+		dest = (void *)start + *start;
+
+		pr_devel("patching dest %lx\n", (unsigned long)dest);
+		patch_instruction(dest, instr);
+	}
+
+	printk(KERN_DEBUG "barrier-nospec: patched %d locations\n", i);
+}
+
 #endif /* CONFIG_PPC_BOOK3S_64 */
 
 void do_lwsync_fixups(unsigned long value, void *fixup_start, void *fixup_end)
Signature · 1 line
2.13.6
author
#16 lore raw

RFC PATCH 6/9 powerpc/64: Patch barrier_nospec in modules

In reply to Michal Suchanek

Note that unlike RFI which is patched only in kernel the nospec state reflects settings at the time the module was loaded.

Iterating all modules and re-patching every time the settings change is not implemented.

Based on lwsync patching.

Signed-off-by Michal Suchanek <[email protected]>
---
 arch/powerpc/include/asm/setup.h  |  5 ++++-
 arch/powerpc/kernel/module.c      |  6 ++++++
 arch/powerpc/kernel/setup_64.c    |  4 ++--
 arch/powerpc/lib/feature-fixups.c | 17 ++++++++++++++---
 4 files changed, 26 insertions(+), 6 deletions(-)
arch/powerpc/include/asm/setup.h +4 -1
index c7e9e66c2a38..92520d2483b8 100644
@@ -58,7 +58,10 @@ enum spec_barrier_type {
 void setup_rfi_flush(enum l1d_flush_type, bool enable);
 void do_rfi_flush_fixups(enum l1d_flush_type types);
 void setup_barrier_nospec(enum spec_barrier_type, bool enable);
-void do_barrier_nospec_fixups(enum spec_barrier_type type);
+void do_barrier_nospec_fixups_kernel(enum spec_barrier_type type);
+void do_barrier_nospec_fixups(enum spec_barrier_type type,
+			      void *start, void *end);
+extern enum spec_barrier_type powerpc_barrier_nospec;
 
 #endif /* !__ASSEMBLY__ */
arch/powerpc/kernel/module.c +6 -0
index 3f7ba0f5bf29..7b6d0ec06a21 100644
@@ -72,6 +72,12 @@ int module_finalize(const Elf_Ehdr *hdr,
 		do_feature_fixups(powerpc_firmware_features,
 				  (void *)sect->sh_addr,
 				  (void *)sect->sh_addr + sect->sh_size);
+
+	sect = find_section(hdr, sechdrs, "__spec_barrier_fixup");
+	if (sect != NULL)
+		do_barrier_nospec_fixups(powerpc_barrier_nospec,
+				  (void *)sect->sh_addr,
+				  (void *)sect->sh_addr + sect->sh_size);
 #endif
 
 	sect = find_section(hdr, sechdrs, "__lwsync_fixup");
arch/powerpc/kernel/setup_64.c +2 -2
index 767240074cad..f60e0e3b5ad2 100644
@@ -910,11 +910,11 @@ void barrier_nospec_enable(bool enable)
 
 	if (enable) {
 		powerpc_barrier_nospec = barrier_nospec_type;
-		do_barrier_nospec_fixups(powerpc_barrier_nospec);
+		do_barrier_nospec_fixups_kernel(powerpc_barrier_nospec);
 		on_each_cpu(do_nothing, NULL, 1);
 	} else {
 		powerpc_barrier_nospec = SPEC_BARRIER_NONE;
-		do_barrier_nospec_fixups(powerpc_barrier_nospec);
+		do_barrier_nospec_fixups_kernel(powerpc_barrier_nospec);
 	}
 }
arch/powerpc/lib/feature-fixups.c +14 -3
index dfeb7feeccef..a529ac6b2a5d 100644
@@ -160,14 +160,15 @@ void do_rfi_flush_fixups(enum l1d_flush_type types)
 						: "unknown");
 }
 
-void do_barrier_nospec_fixups(enum spec_barrier_type type)
+void do_barrier_nospec_fixups(enum spec_barrier_type type,
+			      void *fixup_start, void *fixup_end)
 {
 	unsigned int instr, *dest;
 	long *start, *end;
 	int i;
 
-	start = PTRRELOC(&__start___spec_barrier_fixup),
-	end = PTRRELOC(&__stop___spec_barrier_fixup);
+	start = fixup_start;
+	end = fixup_end;
 
 	instr = 0x60000000; /* nop */
 

@@ -186,6 +187,16 @@ void do_barrier_nospec_fixups(enum spec_barrier_type type)
 	printk(KERN_DEBUG "barrier-nospec: patched %d locations\n", i);
 }
 
+void do_barrier_nospec_fixups_kernel(enum spec_barrier_type type)
+{
+	void *start, *end;
+
+	start = PTRRELOC(&__start___spec_barrier_fixup),
+	end = PTRRELOC(&__stop___spec_barrier_fixup);
+
+	do_barrier_nospec_fixups(type, start, end);
+}
+
 #endif /* CONFIG_PPC_BOOK3S_64 */
 
 void do_lwsync_fixups(unsigned long value, void *fixup_start, void *fixup_end)
Signature · 1 line
2.13.6
author
#17 lore raw

RFC PATCH 7/9 powerpc/64: barrier_nospec: Add debugfs trigger

In reply to Michal Suchanek

Copypasta from rfi implementation

Signed-off-by Michal Suchanek <[email protected]>
---
 arch/powerpc/kernel/setup_64.c | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)
arch/powerpc/kernel/setup_64.c +35 -0
index f60e0e3b5ad2..f6678a7b6114 100644
@@ -963,6 +963,41 @@ static __init int rfi_flush_debugfs_init(void)
 	return 0;
 }
 device_initcall(rfi_flush_debugfs_init);
+
+static int barrier_nospec_set(void *data, u64 val)
+{
+	switch (val) {
+	case 0:
+	case 1:
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	if (!!val == !!barrier_nospec_enabled)
+		return 0;
+
+	barrier_nospec_enable(!!val);
+
+	return 0;
+}
+
+static int barrier_nospec_get(void *data, u64 *val)
+{
+	*val = barrier_nospec_enabled ? 1 : 0;
+	return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(fops_barrier_nospec,
+			barrier_nospec_get, barrier_nospec_set, "%llu\n");
+
+static __init int barrier_nospec_debugfs_init(void)
+{
+	debugfs_create_file("barrier_nospec", 0600, powerpc_debugfs_root, NULL,
+			    &fops_barrier_nospec);
+	return 0;
+}
+device_initcall(barrier_nospec_debugfs_init);
 #endif
 
 ssize_t cpu_show_meltdown(struct device *dev, struct device_attribute *attr, char *buf)
Signature · 1 line
2.13.6
author
#18 lore raw

RFC PATCH 8/9 powerpc/64s: barrier_nospec: Add hcall triggerr

In reply to Michal Suchanek

Adapted from the RFI implementation

Signed-off-by Michal Suchanek <[email protected]>
---
 arch/powerpc/platforms/pseries/mobility.c |  2 +-
 arch/powerpc/platforms/pseries/pseries.h  |  2 +-
 arch/powerpc/platforms/pseries/setup.c    | 37 ++++++++++++++++++++++---------
 3 files changed, 29 insertions(+), 12 deletions(-)
arch/powerpc/platforms/pseries/mobility.c +1 -1
index 8a8033a249c7..9d506be1580e 100644
@@ -349,7 +349,7 @@ void post_mobility_fixup(void)
 			"failed: %d\n", rc);
 
 	/* Possibly switch to a new RFI flush type */
-	pseries_setup_rfi_flush();
+	pseries_setup_rfi_nospec();
 
 	return;
 }
arch/powerpc/platforms/pseries/pseries.h +1 -1
index 27cdcb69fd18..d49670c67686 100644
@@ -100,6 +100,6 @@ static inline unsigned long cmo_get_page_size(void)
 
 int dlpar_workqueue_init(void);
 
-void pseries_setup_rfi_flush(void);
+void pseries_setup_rfi_nospec(void);
 
 #endif /* _PSERIES_PSERIES_H */
arch/powerpc/platforms/pseries/setup.c +27 -10
index 9877c3dfcdc8..4b899a4db6dd 100644
@@ -459,30 +459,47 @@ static void __init find_and_init_phbs(void)
 	of_pci_check_probe_only();
 }
 
-void pseries_setup_rfi_flush(void)
+void pseries_setup_rfi_nospec(void)
 {
 	struct h_cpu_char_result result;
-	enum l1d_flush_type types;
-	bool enable;
+	enum l1d_flush_type flush_types;
+	enum spec_barrier_type barrier_type;
+	bool flush_enable;
+	bool barrier_enable;
 	long rc;
 
 	/* Enable by default */
-	enable = true;
-	types = L1D_FLUSH_FALLBACK;
+	flush_enable = true;
+	flush_types = L1D_FLUSH_FALLBACK;
+	barrier_enable = true;
+	/* no fallback available if the firmware does not tell us */
+	barrier_type = SPEC_BARRIER_NONE;
 
 	rc = plpar_get_cpu_characteristics(&result);
 	if (rc == H_SUCCESS) {
 		if (result.character & H_CPU_CHAR_L1D_FLUSH_TRIG2)
-			types |= L1D_FLUSH_MTTRIG;
+			flush_types |= L1D_FLUSH_MTTRIG;
 		if (result.character & H_CPU_CHAR_L1D_FLUSH_ORI30)
-			types |= L1D_FLUSH_ORI;
+			flush_types |= L1D_FLUSH_ORI;
+		if (result.character & H_CPU_CHAR_SPEC_BAR_ORI31)
+			barrier_type |= SPEC_BARRIER_ORI;
 
 		if ((!(result.behaviour & H_CPU_BEHAV_L1D_FLUSH_PR)) ||
 		    (!(result.behaviour & H_CPU_BEHAV_FAVOUR_SECURITY)))
-			enable = false;
+			flush_enable = false;
+		/*
+		 * Do not check H_CPU_BEHAV_BNDS_CHK_SPEC_BAR - the ORI does
+		 * nothing anyway when not supported.
+		 */
+		if ((!(result.behaviour & H_CPU_BEHAV_FAVOUR_SECURITY)))
+			barrier_enable = false;
+	} else {
+		/* Default to fallback if case hcall is not available */
+		flush_types = L1D_FLUSH_FALLBACK;
 	}
 
-	setup_rfi_flush(types, enable);
+	setup_barrier_nospec(barrier_type, barrier_enable);
+	setup_rfi_flush(flush_types, flush_enable);
 }
 
 #ifdef CONFIG_PCI_IOV

@@ -658,7 +675,7 @@ static void __init pSeries_setup_arch(void)
 
 	fwnmi_init();
 
-	pseries_setup_rfi_flush();
+	pseries_setup_rfi_nospec();
 
 	/* By default, only probe PCI (can be overridden by rtas_pci) */
 	pci_add_flags(PCI_PROBE_ONLY);
Signature · 1 line
2.13.6
author
#19 lore raw

RFC PATCH 9/9 powerpc/64: barrier_nospec: Add commandline trigger

In reply to Michal Suchanek

Add commandline options spectre_v2 and nospectre_v2

These are named same as similar x86 options regardless of actual effect to not require platform-specific configuration.

Supported options:
nospectre_v2 or spectre_v2=off - speculation barrier not used spectre_v2=on or spectre_v2=auto - speculation barrier used

Changing the settings after boot is not supported and VM migration may change requirements so auto is same as on.

Based on s390 implementation

Signed-off-by Michal Suchanek <[email protected]>
---
 arch/powerpc/kernel/setup_64.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
arch/powerpc/kernel/setup_64.c +22 -0
index f6678a7b6114..c74e656265df 100644
@@ -840,6 +840,28 @@ static int __init handle_no_pti(char *p)
 }
 early_param("nopti", handle_no_pti);
 
+static int __init nospectre_v2_setup_early(char *str)
+{
+	no_nospec = true;
+	return 0;
+}
+early_param("nospectre_v2", nospectre_v2_setup_early);
+
+static int __init spectre_v2_setup_early(char *str)
+{
+	if (str && !strncmp(str, "on", 2))
+		no_nospec = false;
+
+	if (str && !strncmp(str, "off", 3))
+		no_nospec = true;
+
+	if (str && !strncmp(str, "auto", 4))
+		no_nospec = false;
+
+	return 0;
+}
+early_param("spectre_v2", spectre_v2_setup_early);
+
 static void do_nothing(void *unused)
 {
 	/*
Signature · 1 line
2.13.6
#20 lore raw

Re: [PATCH RFC rebase 9/9] powerpc/64: barrier_nospec: Add commandline trigger

Quoted 8 lines · Michal Suchanek

Add commandline options spectre_v2 and nospectre_v2

These are named same as similar x86 options regardless of actual effect to not require platform-specific configuration.

Supported options:
nospectre_v2 or spectre_v2=off - speculation barrier not used spectre_v2=on or spectre_v2=auto - speculation barrier used

Why the barrier is enabled only for spectre variant 2 mitigations? It can be used as well for variant 1 mitigations. In fact I am not sure that the places where it is used fall under spectre 2 at all.

Quoted 34 lines

Changing the settings after boot is not supported and VM migration may change requirements so auto is same as on.

Based on s390 implementation

Signed-off-by Michal Suchanek <[email protected]>
---
 arch/powerpc/kernel/setup_64.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
arch/powerpc/kernel/setup_64.c +15 -0
index f6678a7b6114..c74e656265df 100644
@@ -840,6 +840,28 @@ static int __init handle_no_pti(char *p)
 }
 early_param("nopti", handle_no_pti);
 
+static int __init nospectre_v2_setup_early(char *str)
+{
+	no_nospec = true;
+	return 0;
+}
+early_param("nospectre_v2", nospectre_v2_setup_early);
+
+static int __init spectre_v2_setup_early(char *str)
+{
+	if (str && !strncmp(str, "on", 2))
+		no_nospec = false;
+
+	if (str && !strncmp(str, "off", 3))
+		no_nospec = true;
+

arch/powerpc/kernel/setup_64.c static int __init handle_no_pti(char *p) · line 867

Quoted 10 lines
arch/powerpc/kernel/setup_64.c +7 -0
+	if (str && !strncmp(str, "auto", 4))
+		no_nospec = false;
+
+	return 0;
+}
+early_param("spectre_v2", spectre_v2_setup_early);
+
 static void do_nothing(void *unused)
 {
 	/*

Thanks,

Diana

author
#22 lore raw
Quoted 6 lines · On Fri, 16 Mar 2018 09:08:10 +0100 Greg Kroah-Hartman…
Quoted 1 line · Michal Suchanek

Yes, it is good idea to add some commit messages.

Always a good idea :)

Also, any reason you are not tagging these for the stable release(s)?

Oh, right. cc:stable would be a good idea as well.

Thanks

Michal

author
#23 lore raw

RFC PATCH 2/8 powerpc: Use barrier_nospec in copy_from_user

In reply to Michal Suchanek

Coopypasta from x86.

Signed-off-by Michal Suchanek <[email protected]>
---
 arch/powerpc/include/asm/uaccess.h | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
arch/powerpc/include/asm/uaccess.h +10 -1
index 51bfeb8777f0..af9b0e731f46 100644
@@ -248,6 +248,7 @@ do {								\
 	__chk_user_ptr(ptr);					\
 	if (!is_kernel_addr((unsigned long)__gu_addr))		\
 		might_fault();					\
+	barrier_nospec();					\
 	__get_user_size(__gu_val, __gu_addr, (size), __gu_err);	\
 	(x) = (__typeof__(*(ptr)))__gu_val;			\
 	__gu_err;						\

@@ -258,8 +259,10 @@ do {								\
 	long __gu_err = -EFAULT;					\
 	unsigned long  __gu_val = 0;					\
 	const __typeof__(*(ptr)) __user *__gu_addr = (ptr);		\
+	int can_access = access_ok(VERIFY_READ, __gu_addr, (size));	\
 	might_fault();							\
-	if (access_ok(VERIFY_READ, __gu_addr, (size)))			\
+	barrier_nospec();						\
+	if (can_access)							\
 		__get_user_size(__gu_val, __gu_addr, (size), __gu_err);	\
 	(x) = (__force __typeof__(*(ptr)))__gu_val;				\
 	__gu_err;							\

@@ -271,6 +274,7 @@ do {								\
 	unsigned long __gu_val;					\
 	const __typeof__(*(ptr)) __user *__gu_addr = (ptr);	\
 	__chk_user_ptr(ptr);					\
+	barrier_nospec();					\
 	__get_user_size(__gu_val, __gu_addr, (size), __gu_err);	\
 	(x) = (__force __typeof__(*(ptr)))__gu_val;			\
 	__gu_err;						\

@@ -298,15 +302,19 @@ static inline unsigned long raw_copy_from_user(void *to,
 
 		switch (n) {
 		case 1:
+			barrier_nospec();
 			__get_user_size(*(u8 *)to, from, 1, ret);
 			break;
 		case 2:
+			barrier_nospec();
 			__get_user_size(*(u16 *)to, from, 2, ret);
 			break;
 		case 4:
+			barrier_nospec();
 			__get_user_size(*(u32 *)to, from, 4, ret);
 			break;
 		case 8:
+			barrier_nospec();
 			__get_user_size(*(u64 *)to, from, 8, ret);
 			break;
 		}

@@ -314,6 +322,7 @@ static inline unsigned long raw_copy_from_user(void *to,
 			return 0;
 	}
 
+	barrier_nospec();
 	return __copy_tofrom_user((__force void __user *)to, from, n);
 }
Signature · 1 line
2.13.6
author
#24 lore raw

RFC PATCH 3/8 powerpc/64: Use barrier_nospec in syscall entry

In reply to Michal Suchanek
Signed-off-by Michal Suchanek <[email protected]>
---
 arch/powerpc/kernel/entry_64.S | 3 +++
 1 file changed, 3 insertions(+)
arch/powerpc/kernel/entry_64.S +3 -0
index 2cb5109a7ea3..7bfc4cf48af2 100644
@@ -36,6 +36,7 @@
 #include <asm/context_tracking.h>
 #include <asm/tm.h>
 #include <asm/ppc-opcode.h>
+#include <asm/barrier.h>
 #include <asm/export.h>
 #ifdef CONFIG_PPC_BOOK3S
 #include <asm/exception-64s.h>

@@ -159,6 +160,7 @@ system_call:			/* label this so stack traces look sane */
 	andi.	r11,r10,_TIF_SYSCALL_DOTRACE
 	bne	.Lsyscall_dotrace		/* does not return */
 	cmpldi	0,r0,NR_syscalls
+	barrier_nospec
 	bge-	.Lsyscall_enosys
 
 .Lsyscall:

@@ -319,6 +321,7 @@ END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
 	ld	r10,TI_FLAGS(r10)
 
 	cmpldi	r0,NR_syscalls
+	barrier_nospec
 	blt+	.Lsyscall
 
 	/* Return code is already in r3 thanks to do_syscall_trace_enter() */
Signature · 1 line
2.13.6
author
#25 lore raw

RFC PATCH 4/8 powerpc/64s: Add support for ori barrier_nospec

In reply to Michal Suchanek

Copypasta from rfi implementation

Signed-off-by Michal Suchanek <[email protected]>
---
 arch/powerpc/include/asm/barrier.h        |  4 ++--
 arch/powerpc/include/asm/feature-fixups.h |  9 +++++++++
 arch/powerpc/include/asm/setup.h          |  8 ++++++++
 arch/powerpc/kernel/setup_64.c            | 29 +++++++++++++++++++++++++++++
 arch/powerpc/kernel/vmlinux.lds.S         |  7 +++++++
 arch/powerpc/lib/feature-fixups.c         | 27 +++++++++++++++++++++++++++
 6 files changed, 82 insertions(+), 2 deletions(-)
arch/powerpc/include/asm/barrier.h +2 -2
index 8e47b3abe405..4079a95e84c2 100644
@@ -75,9 +75,9 @@ do {									\
 	___p1;								\
 })
 
-/* TODO: add patching so this can be disabled */
 /* Prevent speculative execution past this barrier. */
-#define barrier_nospec_asm ori 31,31,0
+#define barrier_nospec_asm SPEC_BARRIER_FIXUP_SECTION;			\
+				nop
 #ifdef __ASSEMBLY__
 #define barrier_nospec barrier_nospec_asm
 #else
arch/powerpc/include/asm/feature-fixups.h +9 -0
index 1e82eb3caabd..9d3382618ffd 100644
@@ -195,11 +195,20 @@ label##3:					       	\
 	FTR_ENTRY_OFFSET 951b-952b;			\
 	.popsection;
 
+#define SPEC_BARRIER_FIXUP_SECTION			\
+953:							\
+	.pushsection __spec_barrier_fixup,"a";		\
+	.align 2;					\
+954:							\
+	FTR_ENTRY_OFFSET 953b-954b;			\
+	.popsection;
+
 
 #ifndef __ASSEMBLY__
 #include <linux/types.h>
 
 extern long __start___rfi_flush_fixup, __stop___rfi_flush_fixup;
+extern long __start___spec_barrier_fixup, __stop___spec_barrier_fixup;
 
 void apply_feature_fixups(void);
 void setup_feature_keys(void);
arch/powerpc/include/asm/setup.h +8 -0
index 469b7fdc9be4..486d02e4a310 100644
@@ -49,8 +49,16 @@ enum l1d_flush_type {
 	L1D_FLUSH_MTTRIG	= 0x8,
 };
 
+/* These are bit flags */
+enum spec_barrier_type {
+	SPEC_BARRIER_NONE	= 0x1,
+	SPEC_BARRIER_ORI	= 0x2,
+};
+
 void __init setup_rfi_flush(enum l1d_flush_type, bool enable);
 void do_rfi_flush_fixups(enum l1d_flush_type types);
+void __init setup_barrier_nospec(enum spec_barrier_type, bool enable);
+void do_barrier_nospec_fixups(enum spec_barrier_type type);
 
 #endif /* !__ASSEMBLY__ */
arch/powerpc/kernel/setup_64.c +29 -0
index c388cc3357fa..09f21a954bfc 100644
@@ -815,6 +815,10 @@ static enum l1d_flush_type enabled_flush_types;
 static void *l1d_flush_fallback_area;
 static bool no_rfi_flush;
 bool rfi_flush;
+enum spec_barrier_type powerpc_barrier_nospec;
+static enum spec_barrier_type barrier_nospec_type;
+static bool no_nospec;
+bool barrier_nospec_enabled;
 
 static int __init handle_no_rfi_flush(char *p)
 {

@@ -899,6 +903,31 @@ void __init setup_rfi_flush(enum l1d_flush_type types, bool enable)
 		rfi_flush_enable(enable);
 }
 
+void barrier_nospec_enable(bool enable)
+{
+	barrier_nospec_enabled = enable;
+
+	if (enable) {
+		powerpc_barrier_nospec = barrier_nospec_type;
+		do_barrier_nospec_fixups(powerpc_barrier_nospec);
+		on_each_cpu(do_nothing, NULL, 1);
+	} else {
+		powerpc_barrier_nospec = SPEC_BARRIER_NONE;
+		do_barrier_nospec_fixups(powerpc_barrier_nospec);
+	}
+}
+
+void __init setup_barrier_nospec(enum spec_barrier_type type, bool enable)
+{
+	if (type & SPEC_BARRIER_ORI)
+		pr_info("barrier_nospec: Using ori type flush\n");
+
+	barrier_nospec_type = type;
+
+	if (!no_nospec)
+		barrier_nospec_enable(enable);
+}
+
 #ifdef CONFIG_DEBUG_FS
 static int rfi_flush_set(void *data, u64 val)
 {
arch/powerpc/kernel/vmlinux.lds.S +7 -0
index c8af90ff49f0..744b58ff77f1 100644
@@ -139,6 +139,13 @@ SECTIONS
 		*(__rfi_flush_fixup)
 		__stop___rfi_flush_fixup = .;
 	}
+
+	. = ALIGN(8);
+	__spec_barrier_fixup : AT(ADDR(__spec_barrier_fixup) - LOAD_OFFSET) {
+		__start___spec_barrier_fixup = .;
+		*(__spec_barrier_fixup)
+		__stop___spec_barrier_fixup = .;
+	}
 #endif
 
 	EXCEPTION_TABLE(0)
arch/powerpc/lib/feature-fixups.c +27 -0
index 73697c4e3468..000e153184ad 100644
@@ -155,6 +155,33 @@ void do_rfi_flush_fixups(enum l1d_flush_type types)
 
 	printk(KERN_DEBUG "rfi-flush: patched %d locations\n", i);
 }
+
+void do_barrier_nospec_fixups(enum spec_barrier_type type)
+{
+	unsigned int instr, *dest;
+	long *start, *end;
+	int i;
+
+	start = PTRRELOC(&__start___spec_barrier_fixup),
+	end = PTRRELOC(&__stop___spec_barrier_fixup);
+
+	instr = 0x60000000; /* nop */
+
+	if (type == SPEC_BARRIER_ORI) {
+		pr_info("barrier_nospec: using ORI speculation barrier\n");
+		instr = 0x63ff0000; /* ori 31,31,0 speculation barrier */
+	}
+
+	for (i = 0; start < end; start++, i++) {
+		dest = (void *)start + *start;
+
+		pr_devel("patching dest %lx\n", (unsigned long)dest);
+		patch_instruction(dest, instr);
+	}
+
+	printk(KERN_DEBUG "barrier-nospec: patched %d locations\n", i);
+}
+
 #endif /* CONFIG_PPC_BOOK3S_64 */
 
 void do_lwsync_fixups(unsigned long value, void *fixup_start, void *fixup_end)
Signature · 1 line
2.13.6
author
#26 lore raw

RFC PATCH 5/8 powerpc/64: Patch barrier_nospec in modules

In reply to Michal Suchanek

Copypasta from lwsync patching.

Note that unlike RFI which is patched only in kernel the nospec state reflects settings at the time the module was loaded.

Iterating all modules and re-patching every time the settings change is not implemented.

Signed-off-by Michal Suchanek <[email protected]>
---
 arch/powerpc/include/asm/setup.h  |  5 ++++-
 arch/powerpc/kernel/module.c      |  6 ++++++
 arch/powerpc/kernel/setup_64.c    |  4 ++--
 arch/powerpc/lib/feature-fixups.c | 17 ++++++++++++++---
 4 files changed, 26 insertions(+), 6 deletions(-)
arch/powerpc/include/asm/setup.h +4 -1
index 486d02e4a310..7e3a41248810 100644
@@ -58,7 +58,10 @@ enum spec_barrier_type {
 void __init setup_rfi_flush(enum l1d_flush_type, bool enable);
 void do_rfi_flush_fixups(enum l1d_flush_type types);
 void __init setup_barrier_nospec(enum spec_barrier_type, bool enable);
-void do_barrier_nospec_fixups(enum spec_barrier_type type);
+void do_barrier_nospec_fixups_kernel(enum spec_barrier_type type);
+void do_barrier_nospec_fixups(enum spec_barrier_type type,
+			      void *start, void *end);
+extern enum spec_barrier_type powerpc_barrier_nospec;
 
 #endif /* !__ASSEMBLY__ */
arch/powerpc/kernel/module.c +6 -0
index 3f7ba0f5bf29..7b6d0ec06a21 100644
@@ -72,6 +72,12 @@ int module_finalize(const Elf_Ehdr *hdr,
 		do_feature_fixups(powerpc_firmware_features,
 				  (void *)sect->sh_addr,
 				  (void *)sect->sh_addr + sect->sh_size);
+
+	sect = find_section(hdr, sechdrs, "__spec_barrier_fixup");
+	if (sect != NULL)
+		do_barrier_nospec_fixups(powerpc_barrier_nospec,
+				  (void *)sect->sh_addr,
+				  (void *)sect->sh_addr + sect->sh_size);
 #endif
 
 	sect = find_section(hdr, sechdrs, "__lwsync_fixup");
arch/powerpc/kernel/setup_64.c +2 -2
index 09f21a954bfc..d1d9f047161e 100644
@@ -909,11 +909,11 @@ void barrier_nospec_enable(bool enable)
 
 	if (enable) {
 		powerpc_barrier_nospec = barrier_nospec_type;
-		do_barrier_nospec_fixups(powerpc_barrier_nospec);
+		do_barrier_nospec_fixups_kernel(powerpc_barrier_nospec);
 		on_each_cpu(do_nothing, NULL, 1);
 	} else {
 		powerpc_barrier_nospec = SPEC_BARRIER_NONE;
-		do_barrier_nospec_fixups(powerpc_barrier_nospec);
+		do_barrier_nospec_fixups_kernel(powerpc_barrier_nospec);
 	}
 }
arch/powerpc/lib/feature-fixups.c +14 -3
index 000e153184ad..b59ebc2215e8 100644
@@ -156,14 +156,15 @@ void do_rfi_flush_fixups(enum l1d_flush_type types)
 	printk(KERN_DEBUG "rfi-flush: patched %d locations\n", i);
 }
 
-void do_barrier_nospec_fixups(enum spec_barrier_type type)
+void do_barrier_nospec_fixups(enum spec_barrier_type type,
+			      void *fixup_start, void *fixup_end)
 {
 	unsigned int instr, *dest;
 	long *start, *end;
 	int i;
 
-	start = PTRRELOC(&__start___spec_barrier_fixup),
-	end = PTRRELOC(&__stop___spec_barrier_fixup);
+	start = fixup_start;
+	end = fixup_end;
 
 	instr = 0x60000000; /* nop */
 

@@ -182,6 +183,16 @@ void do_barrier_nospec_fixups(enum spec_barrier_type type)
 	printk(KERN_DEBUG "barrier-nospec: patched %d locations\n", i);
 }
 
+void do_barrier_nospec_fixups_kernel(enum spec_barrier_type type)
+{
+	void *start, *end;
+
+	start = PTRRELOC(&__start___spec_barrier_fixup),
+	end = PTRRELOC(&__stop___spec_barrier_fixup);
+
+	do_barrier_nospec_fixups(type, start, end);
+}
+
 #endif /* CONFIG_PPC_BOOK3S_64 */
 
 void do_lwsync_fixups(unsigned long value, void *fixup_start, void *fixup_end)
Signature · 1 line
2.13.6
author
#27 lore raw

RFC PATCH 6/8 powerpc/64: barrier_nospec: Add debugfs trigger

In reply to Michal Suchanek

Copypasta from rfi implementation

Signed-off-by Michal Suchanek <[email protected]>
---
 arch/powerpc/kernel/setup_64.c | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)
arch/powerpc/kernel/setup_64.c +35 -0
index d1d9f047161e..4b67b7b877d9 100644
@@ -955,6 +955,41 @@ static __init int rfi_flush_debugfs_init(void)
 	return 0;
 }
 device_initcall(rfi_flush_debugfs_init);
+
+static int barrier_nospec_set(void *data, u64 val)
+{
+	switch (val) {
+	case 0:
+	case 1:
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	if (!!val == !!barrier_nospec_enabled)
+		return 0;
+
+	barrier_nospec_enable(!!val);
+
+	return 0;
+}
+
+static int barrier_nospec_get(void *data, u64 *val)
+{
+	*val = barrier_nospec_enabled ? 1 : 0;
+	return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(fops_barrier_nospec,
+			barrier_nospec_get, barrier_nospec_set, "%llu\n");
+
+static __init int barrier_nospec_debugfs_init(void)
+{
+	debugfs_create_file("barrier_nospec", 0600, powerpc_debugfs_root, NULL,
+			    &fops_barrier_nospec);
+	return 0;
+}
+device_initcall(barrier_nospec_debugfs_init);
 #endif
 
 ssize_t cpu_show_meltdown(struct device *dev, struct device_attribute *attr, char *buf)
Signature · 1 line
2.13.6
author
#28 lore raw

RFC PATCH 7/8 powerpc/64s: barrier_nospec: Add hcall triggerr

In reply to Michal Suchanek

Copypasta from rfi implementation

Signed-off-by Michal Suchanek <[email protected]>
---
 arch/powerpc/platforms/pseries/setup.c | 38 ++++++++++++++++++++++------------
 1 file changed, 25 insertions(+), 13 deletions(-)
arch/powerpc/platforms/pseries/setup.c +25 -13
index 1a527625acf7..b779ddb8e250 100644
@@ -459,38 +459,50 @@ static void __init find_and_init_phbs(void)
 	of_pci_check_probe_only();
 }
 
-static void pseries_setup_rfi_flush(void)
+static void pseries_setup_rfi_nospec(void)
 {
 	struct h_cpu_char_result result;
-	enum l1d_flush_type types;
-	bool enable;
+	enum l1d_flush_type flush_types;
+	enum spec_barrier_type barrier_type;
+	bool flush_enable;
+	bool barrier_enable;
 	long rc;
 
 	/* Enable by default */
-	enable = true;
+	flush_enable = true;
+	barrier_enable = true;
+	/* no fallback if the firmware does not tell us */
+	barrier_type = SPEC_BARRIER_NONE;
 
 	rc = plpar_get_cpu_characteristics(&result);
 	if (rc == H_SUCCESS) {
-		types = L1D_FLUSH_NONE;
+		flush_types = L1D_FLUSH_NONE;
 
 		if (result.character & H_CPU_CHAR_L1D_FLUSH_TRIG2)
-			types |= L1D_FLUSH_MTTRIG;
+			flush_types |= L1D_FLUSH_MTTRIG;
 		if (result.character & H_CPU_CHAR_L1D_FLUSH_ORI30)
-			types |= L1D_FLUSH_ORI;
+			flush_types |= L1D_FLUSH_ORI;
+		if (result.character & H_CPU_CHAR_SPEC_BAR_ORI31)
+			barrier_type |= SPEC_BARRIER_ORI;
 
 		/* Use fallback if nothing set in hcall */
-		if (types == L1D_FLUSH_NONE)
-			types = L1D_FLUSH_FALLBACK;
+		if (flush_types == L1D_FLUSH_NONE)
+			flush_types = L1D_FLUSH_FALLBACK;
 
 		if ((!(result.behaviour & H_CPU_BEHAV_L1D_FLUSH_PR)) ||
 		    (!(result.behaviour & H_CPU_BEHAV_FAVOUR_SECURITY)))
-			enable = false;
+			flush_enable = false;
+
+		if ((!(result.behaviour & H_CPU_BEHAV_BNDS_CHK_SPEC_BAR)) ||
+		    (!(result.behaviour & H_CPU_BEHAV_FAVOUR_SECURITY)))
+			barrier_enable = false;
 	} else {
 		/* Default to fallback if case hcall is not available */
-		types = L1D_FLUSH_FALLBACK;
+		flush_types = L1D_FLUSH_FALLBACK;
 	}
 
-	setup_rfi_flush(types, enable);
+	setup_barrier_nospec(barrier_type, barrier_enable);
+	setup_rfi_flush(flush_types, flush_enable);
 }
 
 #ifdef CONFIG_PCI_IOV

@@ -666,7 +678,7 @@ static void __init pSeries_setup_arch(void)
 
 	fwnmi_init();
 
-	pseries_setup_rfi_flush();
+	pseries_setup_rfi_nospec();
 
 	/* By default, only probe PCI (can be overridden by rtas_pci) */
 	pci_add_flags(PCI_PROBE_ONLY);
Signature · 1 line
2.13.6
author
#29 lore raw

RFC PATCH 8/8 powerpc/64: barrier_nospec: Add commandline trigger

In reply to Michal Suchanek

Copypasta from rfi implementation

Signed-off-by Michal Suchanek <[email protected]>
---
 arch/powerpc/kernel/setup_64.c | 8 ++++++++
 1 file changed, 8 insertions(+)
arch/powerpc/kernel/setup_64.c +8 -0
index 4b67b7b877d9..257f0e6be107 100644
@@ -840,6 +840,14 @@ static int __init handle_no_pti(char *p)
 }
 early_param("nopti", handle_no_pti);
 
+static int __init handle_no_nospec(char *p)
+{
+	pr_info("barrier_nospec: disabled on command line.");
+	no_nospec = true;
+	return 0;
+}
+early_param("no_nospec", handle_no_nospec);
+
 static void do_nothing(void *unused)
 {
 	/*
Signature · 1 line
2.13.6