powerpc/64s: Fix compiler store ordering to SLB shadow area
Nicholas Piggin
reviewerPATCH 1/1 powerpc/64s: Fix compiler store ordering to SLB shadow area
The stores to update the SLB shadow area must be made as they appear in the C code, so that the hypervisor does not see an entry with mismatched vsid and esid. Use WRITE_ONCE for this.
GCC has been observed to elide the first store to esid in the update, which means that if the hypervisor interrupts the guest after storing to vsid, it could see an entry with old esid and new vsid, which may possibly result in memory corruption.
--- arch/powerpc/mm/slb.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
arch/powerpc/mm/slb.c +4 -4
@@ -63,14 +63,14 @@ static inline void slb_shadow_update(unsigned long ea, int ssize,* updating it. No write barriers are needed here, provided * we only update the current CPU's SLB shadow buffer. */ - p->save_area[index].esid =0 ; - p->save_area[index].vsid =cpu_to_be64 (mk_vsid_data (ea, ssize, flags)); - p->save_area[index].esid =cpu_to_be64 (mk_esid_data (ea, ssize, index)); +WRITE_ONCE (p->save_area[index].esid,0 ); +WRITE_ONCE (p->save_area[index].vsid,cpu_to_be64 (mk_vsid_data (ea, ssize, flags))); +WRITE_ONCE (p->save_area[index].esid,cpu_to_be64 (mk_esid_data (ea, ssize, index))); }static inline void slb_shadow_clear (enum slb_index index) { -get_slb_shadow ()->save_area[index].esid =0 ; +WRITE_ONCE (get_slb_shadow ()->save_area[index].esid,0 ); } static inline void create_shadowed_slbe(unsigned long ea, int ssize,
Signature · 1 line
2.17.0
Michael Ellerman
reviewerQuoted 19 lines · Nicholas Piggin
The stores to update the SLB shadow area must be made as they appear in the C code, so that the hypervisor does not see an entry with mismatched vsid and esid. Use
WRITE_ONCEfor this.GCC has been observed to elide the first store to esid in the update, which means that if the hypervisor interrupts the guest after storing to vsid, it could see an entry with old esid and new vsid, which may possibly result in memory corruption.
Signed-off-by Nicholas Piggin <[email protected]>--- arch/powerpc/mm/slb.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)arch/powerpc/mm/slb.c +0
-0
arch/powerpc/mm/slb.c static inline void slb_shadow_update(unsigned long ea, int ssize, · line 68
Quoted 9 lines
arch/powerpc/mm/slb.c +3
-3@@ -63,14 +63,14 @@ static inline void slb_shadow_update(unsigned long ea, int ssize,* updating it. No write barriers are needed here, provided * we only update the current CPU's SLB shadow buffer. */ - p->save_area[index].esid =0 ; - p->save_area[index].vsid =cpu_to_be64 (mk_vsid_data (ea, ssize, flags)); - p->save_area[index].esid =cpu_to_be64 (mk_esid_data (ea, ssize, index)); +WRITE_ONCE (p->save_area[index].esid,0 ); +WRITE_ONCE (p->save_area[index].vsid,cpu_to_be64 (mk_vsid_data (ea, ssize, flags))); +WRITE_ONCE (p->save_area[index].esid,cpu_to_be64 (mk_esid_data (ea, ssize, index)));
What's the code-gen for that look like? I suspect it's terrible?
Should we just do it in inline-asm I wonder?
cheers
Nicholas Piggin
reviewerQuoted 16 lines · On Fri, 01 Jun 2018 00:22:21 +1000 Michael Ellerman <[email protected]> wrote:
Quoted 14 lines · Nicholas Piggin
The stores to update the SLB shadow area must be made as they appear in the C code, so that the hypervisor does not see an entry with mismatched vsid and esid. Use
WRITE_ONCEfor this.GCC has been observed to elide the first store to esid in the update, which means that if the hypervisor interrupts the guest after storing to vsid, it could see an entry with old esid and new vsid, which may possibly result in memory corruption.
Signed-off-by Nicholas Piggin <[email protected]>--- arch/powerpc/mm/slb.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
Quoted 16 lines
Quoted 5 lines
arch/powerpc/mm/slb.c +0
-0arch/powerpc/mm/slb.c static inline void slb_shadow_update(unsigned long ea, int ssize, · line 68
Quoted 9 lines
arch/powerpc/mm/slb.c +3
-3@@ -63,14 +63,14 @@ static inline void slb_shadow_update(unsigned long ea, int ssize,* updating it. No write barriers are needed here, provided * we only update the current CPU's SLB shadow buffer. */ - p->save_area[index].esid =0 ; - p->save_area[index].vsid =cpu_to_be64 (mk_vsid_data (ea, ssize, flags)); - p->save_area[index].esid =cpu_to_be64 (mk_esid_data (ea, ssize, index)); +WRITE_ONCE (p->save_area[index].esid,0 ); +WRITE_ONCE (p->save_area[index].vsid,cpu_to_be64 (mk_vsid_data (ea, ssize, flags))); +WRITE_ONCE (p->save_area[index].esid,cpu_to_be64 (mk_esid_data (ea, ssize, index)));What's the code-gen for that look like? I suspect it's terrible?
Yeah it's not great.
Quoted 2 lines
Should we just do it in inline-asm I wonder?
There should be no fundamental correctness reason why we can't store to a volatile with a byteswap store. The other option we could do is add a compiler barrier() between each store. The reason I didn't is that in theory we don't need to invalidate all memory contents here, but in practice probably the end result code generation would be better.
Thanks,
Nick
Michael Ellerman
reviewerQuoted 34 lines · Nicholas Piggin
Quoted 15 lines · On Fri, 01 Jun 2018 00:22:21 +1000 Michael Ellerman <[email protected]> wrote:
Quoted 14 lines · Nicholas Piggin
The stores to update the SLB shadow area must be made as they appear in the C code, so that the hypervisor does not see an entry with mismatched vsid and esid. Use
WRITE_ONCEfor this.GCC has been observed to elide the first store to esid in the update, which means that if the hypervisor interrupts the guest after storing to vsid, it could see an entry with old esid and new vsid, which may possibly result in memory corruption.
Signed-off-by Nicholas Piggin <[email protected]>--- arch/powerpc/mm/slb.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)Quoted 16 lines
Quoted 5 lines
arch/powerpc/mm/slb.c +0
-0arch/powerpc/mm/slb.c static inline void slb_shadow_update(unsigned long ea, int ssize, · line 68
Quoted 9 lines
arch/powerpc/mm/slb.c +3
-3@@ -63,14 +63,14 @@ static inline void slb_shadow_update(unsigned long ea, int ssize,* updating it. No write barriers are needed here, provided * we only update the current CPU's SLB shadow buffer. */ - p->save_area[index].esid =0 ; - p->save_area[index].vsid =cpu_to_be64 (mk_vsid_data (ea, ssize, flags)); - p->save_area[index].esid =cpu_to_be64 (mk_esid_data (ea, ssize, index)); +WRITE_ONCE (p->save_area[index].esid,0 ); +WRITE_ONCE (p->save_area[index].vsid,cpu_to_be64 (mk_vsid_data (ea, ssize, flags))); +WRITE_ONCE (p->save_area[index].esid,cpu_to_be64 (mk_esid_data (ea, ssize, index)));What's the code-gen for that look like? I suspect it's terrible?
Quoted 1 line
Yeah it's not great.
Actually with GCC 7 the WRITE_ONCE() doesn't make it any worse.
Which is a little suspicious. But it is doing the first store:
li r10,0 # r10 = 0 ld r29,56(r13) # r29 = paca->slb_shadow_ptr rldicr r8,r31,4,59 # r8 = index rldicr r9,r9,32,31 add r29,r29,r8 # r29 = r29 + index oris r9,r9,65535 std r10,16(r29) # esid = r10 = 0
So I'll just merge this as-is.
cheers
Quoted 12 lines · Nicholas Piggin
Quoted 9 lines · On Fri, 01 Jun 2018 00:22:21 +1000 Michael Ellerman <[email protected]> wrote:
arch/powerpc/mm/slb.c static inline void slb_shadow_update(unsigned long ea, int ssize, · line 68
Quoted 6 lines · Nicholas Piggin
arch/powerpc/mm/slb.c +3
-3- p->save_area[index].esid =0 ; - p->save_area[index].vsid =cpu_to_be64 (mk_vsid_data (ea, ssize, flags)); - p->save_area[index].esid =cpu_to_be64 (mk_esid_data (ea, ssize, index)); +WRITE_ONCE (p->save_area[index].esid,0 ); +WRITE_ONCE (p->save_area[index].vsid,cpu_to_be64 (mk_vsid_data (ea, ssize, flags))); +WRITE_ONCE (p->save_area[index].esid,cpu_to_be64 (mk_esid_data (ea, ssize, index)));What's the code-gen for that look like? I suspect it's terrible?
Quoted 4 lines
Yeah it's not great.
Quoted 2 lines
Should we just do it in inline-asm I wonder?
That is my recommendation: that will work for all compiler versions.
Quoted 2 lines
There should be no fundamental correctness reason why we can't store to a volatile with a byteswap store.
There are may operations that are *not* correct to merge into a volatile memory access, and which are fine is different for every arch. GCC simply disallows combining anything into any volatile memory by default. This is kind of fine because volatile already means "I want this to go slow", in common cases ;-)
I'll see what I can do to make the byteswap load/stores work with volatile (for powerpc).
Quoted 5 lines
The other option we could do is
add a compilerbarrier()between each store. The reason I didn't is that in theory we don't need to invalidate all memory contents here, but in practice probably the end result code generation would be better.
Something like
p->save_area[index].esid = 0;
asm("" : : "m"(p->save_area[index].esid));
p->save_area[index].vsid = cpu_to_be64(mk_vsid_data(ea, ssize, flags));
asm("" : : "m"(p->save_area[index].vsid));
p->save_area[index].esid = cpu_to_be64(mk_esid_data(ea, ssize, index));should do the trick (and once again for the second write to esid, if you want to be sure it is not optimised away).
Segher
Michael Ellerman
AppliedQuoted 10 lines · Nicholas Piggin
The stores to update the SLB shadow area must be made as they appear in the C code, so that the hypervisor does not see an entry with mismatched vsid and esid. Use
WRITE_ONCEfor this.GCC has been observed to elide the first store to esid in the update, which means that if the hypervisor interrupts the guest after storing to vsid, it could see an entry with old esid and new vsid, which may possibly result in memory corruption.
Signed-off-by Nicholas Piggin <[email protected]>
Applied to powerpc next, thanks.
https://git.kernel.org/powerpc/c/926bc2f100c24d4842b3064b5af44a
cheers