x86/special_insn: reverse __force_order logic
Subsystems (2)
PATCH 1/1 x86/special_insn: reverse __force_order logic
From: Nadav Amit <[email protected]>
The __force_order logic seems to be inverted. __force_order is supposedly used to manipulate the compiler to use its memory dependencies analysis to enforce orders between CR writes and reads. Therefore, the memory should behave as a "CR": when the CR is read, the memory should be "read" by the inline assembly, and __force_order should be an output. When the CR is written, the memory should be "written".
This change should allow to remove the "volatile" qualifier from CR reads at a later patch.
While at it, remove the extra new-line from the inline assembly, as it only confuses GCC when it estimates the cost of the inline assembly.
---
Unless I misunderstand the logic, __force_order should also be used by rdpkru() and wrpkru() which do not have dependency on __force_order. I also did not understand why native_write_cr0() has R/W dependency on __force_order, and why native_write_cr4() no longer has any dependency on __force_order.
--- arch/x86/include/asm/special_insns.h | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
arch/x86/include/asm/special_insns.h +7 -7
@@ -24,32 +24,32 @@ void native_write_cr0(unsigned long val);static inline unsigned long native_read_cr0 (void ) {unsigned long val; - asmvolatile ("mov %%cr0,%0\n\t" :"=r" (val),"=m" (__force_order)); + asmvolatile ("mov %%cr0,%0" :"=r" (val) :"m" (__force_order));return val; }static __always_inline unsigned long native_read_cr2 (void ) {unsigned long val; - asmvolatile ("mov %%cr2,%0\n\t" :"=r" (val),"=m" (__force_order)); + asmvolatile ("mov %%cr2,%0" :"=r" (val) :"m" (__force_order));return val; } static __always_inline void native_write_cr2(unsigned long val) { - asmvolatile ("mov %0,%%cr2" : :"r" (val),"m" (__force_order)); + asmvolatile ("mov %1,%%cr2" :"=m" (__force_order) :"r" (val)); }static inline unsigned long __native_read_cr3 (void ) {unsigned long val; - asmvolatile ("mov %%cr3,%0\n\t" :"=r" (val),"=m" (__force_order)); + asmvolatile ("mov %%cr3,%0" :"=r" (val) :"m" (__force_order));return val; }static inline void native_write_cr3 (unsigned long val) { - asmvolatile ("mov %0,%%cr3" : :"r" (val),"m" (__force_order)); + asmvolatile ("mov %1,%%cr3" :"=m" (__force_order) :"r" (val)); }static inline unsigned long native_read_cr4 (void )
@@ -64,10 +64,10 @@ static inline unsigned long native_read_cr4(void) asm volatile("1: mov %%cr4, %0\n" "2:\n" _ASM_EXTABLE(1b, 2b) - : "=r" (val), "=m" (__force_order) : "0" (0)); + : "=r" (val) : "m" (__force_order), "0" (0)); #else/* CR4 always exists on x86_64. */ - asmvolatile ("mov %%cr4,%0\n\t" :"=r" (val),"=m" (__force_order)); + asmvolatile ("mov %%cr4,%0" :"=r" (val) :"m" (__force_order));#endif return val; }
Signature · 1 line
2.25.1
[email protected]
reviewerQuoted 84 lines · Nadav Amit
From: Nadav Amit <[email protected]>
The
__force_orderlogic seems to be inverted.__force_orderis supposedly used to manipulate the compiler to use its memory dependencies analysis to enforce orders between CR writes and reads. Therefore, the memory should behave as a "CR": when the CR is read, the memory should be "read" by the inline assembly, and__force_ordershould
be an output. When the CR is written, the memory should be "written".This change should allow to remove the "volatile" qualifier from CR reads at a later patch.
While at it, remove the extra new-line from the inline assembly, as it only confuses GCC when it estimates the cost of the inline assembly.
Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Borislav Petkov <[email protected]>Cc "H. Peter Anvin" <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Kees Cook <[email protected]>Signed-off-by Nadav Amit <[email protected]>---
Unless I misunderstand the logic,
__force_ordershould also be used byrdpkru()andwrpkru()which do not have dependency on__force_order. I also did not understand whynative_write_cr0()has R/W dependency on__force_order, and whynative_write_cr4()no longer has any dependency on__force_order.
---arch/x86/include/asm/special_insns.h| 14 +++++++-------1 file changed, 7 insertions(+), 7 deletions(-)arch/x86/include/asm/special_insns.h +5
-5@@ -24,32 +24,32 @@ void native_write_cr0(unsigned long val);static inline unsigned long native_read_cr0 (void ) {unsigned long val; - asmvolatile ("mov %%cr0,%0\n\t" :"=r" (val),"=m" (__force_order)); + asmvolatile ("mov %%cr0,%0" :"=r" (val) :"m" (__force_order));return val; }static __always_inline unsigned long native_read_cr2 (void ) {unsigned long val; - asmvolatile ("mov %%cr2,%0\n\t" :"=r" (val),"=m" (__force_order)); + asmvolatile ("mov %%cr2,%0" :"=r" (val) :"m" (__force_order));return val; } static __always_inline void native_write_cr2(unsigned long val) { - asmvolatile ("mov %0,%%cr2" : :"r" (val),"m" (__force_order)); + asmvolatile ("mov %1,%%cr2" :"=m" (__force_order) :"r" (val)); }static inline unsigned long __native_read_cr3 (void ) {unsigned long val; - asmvolatile ("mov %%cr3,%0\n\t" :"=r" (val),"=m" (__force_order)); + asmvolatile ("mov %%cr3,%0" :"=r" (val) :"m" (__force_order));return val; }static inline void native_write_cr3 (unsigned long val) { - asmvolatile ("mov %0,%%cr3" : :"r" (val),"m" (__force_order)); + asmvolatile ("mov %1,%%cr3" :"=m" (__force_order) :"r" (val)); }static inline unsigned long native_read_cr4 (void )@@ -64,10 +64,10 @@ static inline unsigned long native_read_cr4(void) asmvolatile ("1: mov %%cr4, %0\n" "2:\n"
arch/x86/include/asm/special_insns.h static inline unsigned long native_read_cr4(void) · line 73
Quoted 10 lines
arch/x86/include/asm/special_insns.h +2
-2_ASM_EXTABLE(1b, 2b) - : "=r" (val), "=m" (__force_order) : "0" (0)); + : "=r" (val) : "m" (__force_order), "0" (0)); #else/* CR4 always exists on x86_64. */ - asmvolatile ("mov %%cr4,%0\n\t" :"=r" (val),"=m" (__force_order)); + asmvolatile ("mov %%cr4,%0" :"=r" (val) :"m" (__force_order));#endif return val; }
This seems reasonable to me, and I fully agree with you that the logic seems to be just plain wrong (and unnoticed since all volatile operations are strictly ordered anyway), but you better not remove the volatile from cr2 read... unlike the other CRs that one is written by hardware and so genuinely is volatile.
However, I do believe "=m" is at least theoretically wrong. You are only writing *part* of the total state represented by this token (you can think of it as equivalent to a bitop), so it should be "+m".
Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Quoted 5 lines · Nadav Amit
Unless I misunderstand the logic,
__force_ordershould also be used byrdpkru()andwrpkru()which do not have dependency on__force_order. I also did not understand whynative_write_cr0()has R/W dependency on__force_order, and whynative_write_cr4()no longer has any dependency on__force_order.
There was a fairly large thread about this thing here:
https://lkml.kernel.org/r/[email protected]
I didn't keep up, but I think the general concensus was that it's indeed a bit naf.
Quoted 10 lines · On Sep 2, 2020, at 5:54 AM, [email protected] wrote:
Quoted 5 lines · Nadav Amit
Unless I misunderstand the logic,
__force_ordershould also be used byrdpkru()andwrpkru()which do not have dependency on__force_order. I also did not understand whynative_write_cr0()has R/W dependency on__force_order, and whynative_write_cr4()no longer has any dependency on__force_order.
Quoted 6 lines
There was a fairly large thread about this thing here:
I didn't keep up, but I think the general concensus was that it's indeed a bit naf.
Thanks for pointer. I did not see the discussion, and embarrassingly, I have also never figured out how to reply on lkml emails without registering to lkml.
Anyhow, indeed, the patch that Arvind provided seems to address this issue.
Quoted 3 lines · Nadav Amit
Thanks for pointer. I did not see the discussion, and embarrassingly, I have also never figured out how to reply on lkml emails without registering to lkml.
The lore.kernel.org thing I pointed you to allows you to download an mbox with the email in (see the very bottom of the page). Use your favorite MUA to open it and reply :-)
Quoted 11 lines · On Sep 2, 2020, at 9:56 AM, [email protected] wrote:
Quoted 3 lines · Nadav Amit
Thanks for pointer. I did not see the discussion, and embarrassingly, I have also never figured out how to reply on lkml emails without registering to lkml.
The lore.kernel.org thing I pointed you to allows you to download an mbox with the email in (see the very bottom of the page). Use your favorite MUA to open it and reply :-)
Thanks!