x86/asm: Force native_apic_mem_read to use mov
PATCH 1/1 x86/asm: Force native_apic_mem_read to use mov
Previously, when compiled with clang, native_apic_mem_read gets inlined into __xapic_wait_icr_idle and optimized to a testl instruction. When run in a VM with SEV-ES enabled, it attempts to emulate this instruction, but the emulator does not support it. Instead, use inline assembly to force native_apic_mem_read to use the mov instruction which is supported by the emulator.
--- arch/x86/include/asm/apic.h | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
arch/x86/include/asm/apic.h +12 -1
@@ -109,7 +109,18 @@ static inline void native_apic_mem_write(u32 reg, u32 v)static inline u32 native_apic_mem_read (u32 reg) { -return *((volatile u32 *)(APIC_BASE + reg)); +volatile u32 *addr = (volatile u32 *)(APIC_BASE + reg); +u32 out; + +/* +* Functionally, what we want to do is simply return *addr. However, +* this accesses an MMIO which may need to be emulated in some cases. +* The emulator doesn't necessarily support all instructions, so we +* force the read from addr to use a mov instruction. +*/ + asm_inline("movl %1, %0" : "=r"(out) : "m"(*addr)); + + return out; }extern void native_apic_wait_icr_idle (void );
Signature · 1 line
2.37.1.559.g78731f0fdb-goog
Quoted 25 lines · Adam Dunlap
Previously, when compiled with clang,
native_apic_mem_readgets inlined into__xapic_wait_icr_idleand optimized to a testl instruction. When run in a VM with SEV-ES enabled, it attempts to emulate this instruction, but the emulator does not support it. Instead, use inline assembly to forcenative_apic_mem_readto use the mov instruction which is supported by the emulator.Signed-off-by Adam Dunlap <[email protected]>Reviewed-by Marc Orr <[email protected]>Reviewed-by Jacob Xu <[email protected]>--- arch/x86/include/asm/apic.h | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)arch/x86/include/asm/apic.h +2
-1@@ -109,7 +109,18 @@ static inline void native_apic_mem_write(u32 reg, u32 v)static inline u32 native_apic_mem_read (u32 reg) { -return *((volatile u32 *)(APIC_BASE + reg)); +volatile u32 *addr = (volatile u32 *)(APIC_BASE + reg); +u32 out;
arch/x86/include/asm/apic.h static inline void native_apic_mem_write(u32 reg, u32 v) · line 123
Quoted 10 lines
arch/x86/include/asm/apic.h +10
-0+ +/* +* Functionally, what we want to do is simply return *addr. However, +* this accesses an MMIO which may need to be emulated in some cases. +* The emulator doesn't necessarily support all instructions, so we +* force the read from addr to use a mov instruction. +*/ + asm_inline("movl %1, %0" : "=r"(out) : "m"(*addr)); + + return out;
Can't this just be:
return readl((void __iomem *)(APIC_BASE + reg));
H. Peter Anvin
reviewerQuoted 39 lines · Sean Christopherson
Quoted 25 lines · Adam Dunlap
Previously, when compiled with clang,
native_apic_mem_readgets inlined into__xapic_wait_icr_idleand optimized to a testl instruction. When run in a VM with SEV-ES enabled, it attempts to emulate this instruction, but the emulator does not support it. Instead, use inline assembly to forcenative_apic_mem_readto use the mov instruction which is supported by the emulator.Signed-off-by Adam Dunlap <[email protected]>Reviewed-by Marc Orr <[email protected]>Reviewed-by Jacob Xu <[email protected]>--- arch/x86/include/asm/apic.h | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)arch/x86/include/asm/apic.h +2
-1@@ -109,7 +109,18 @@ static inline void native_apic_mem_write(u32 reg, u32 v)static inline u32 native_apic_mem_read (u32 reg) { -return *((volatile u32 *)(APIC_BASE + reg)); +volatile u32 *addr = (volatile u32 *)(APIC_BASE + reg); +u32 out;arch/x86/include/asm/apic.h static inline void native_apic_mem_write(u32 reg, u32 v) · line 123
Quoted 10 lines
arch/x86/include/asm/apic.h +10
-0+ +/* +* Functionally, what we want to do is simply return *addr. However, +* this accesses an MMIO which may need to be emulated in some cases. +* The emulator doesn't necessarily support all instructions, so we +* force the read from addr to use a mov instruction. +*/ + asm_inline("movl %1, %0" : "=r"(out) : "m"(*addr)); + + return out;Can't this just be:
Quoted 1 line
return readl((void __iomem *)(APIC_BASE + reg));
The very point of the patch is to force a specific instruction sequence.
Quoted 40 lines · H. Peter Anvin
Quoted 39 lines · Sean Christopherson
Quoted 25 lines · Adam Dunlap
Previously, when compiled with clang,
native_apic_mem_readgets inlined into__xapic_wait_icr_idleand optimized to a testl instruction. When run in a VM with SEV-ES enabled, it attempts to emulate this instruction, but the emulator does not support it. Instead, use inline assembly to forcenative_apic_mem_readto use the mov instruction which is supported by the emulator.Signed-off-by Adam Dunlap <[email protected]>Reviewed-by Marc Orr <[email protected]>Reviewed-by Jacob Xu <[email protected]>--- arch/x86/include/asm/apic.h | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)arch/x86/include/asm/apic.h +2
-1@@ -109,7 +109,18 @@ static inline void native_apic_mem_write(u32 reg, u32 v)static inline u32 native_apic_mem_read (u32 reg) { -return *((volatile u32 *)(APIC_BASE + reg)); +volatile u32 *addr = (volatile u32 *)(APIC_BASE + reg); +u32 out;arch/x86/include/asm/apic.h static inline void native_apic_mem_write(u32 reg, u32 v) · line 123
Quoted 10 lines
arch/x86/include/asm/apic.h +10
-0+ +/* +* Functionally, what we want to do is simply return *addr. However, +* this accesses an MMIO which may need to be emulated in some cases. +* The emulator doesn't necessarily support all instructions, so we +* force the read from addr to use a mov instruction. +*/ + asm_inline("movl %1, %0" : "=r"(out) : "m"(*addr)); + + return out;Can't this just be:
Quoted 3 lines
Quoted 1 line
return readl((void __iomem *)(APIC_BASE + reg));The very point of the patch is to force a specific instruction sequence.
Yes, and that specific emulator-friendly instruction also has to be forced for all of the core x86 read/write MMIO helpers. And it's also possible for MMIO read/write to be enlightened to skip the MOV and go straight to #VMGEXIT, i.e. the xAPIC code shouldn't assume MOV is the best/only option (ignoring the handling of the P54C erratum in the write path).
H. Peter Anvin
reviewerQuoted 43 lines · Sean Christopherson
Quoted 42 lines · H. Peter Anvin
Quoted 40 lines · Sean Christopherson
Quoted 25 lines · Adam Dunlap
Previously, when compiled with clang,
native_apic_mem_readgets inlined into__xapic_wait_icr_idleand optimized to a testl instruction. When run in a VM with SEV-ES enabled, it attempts to emulate this instruction, but the emulator does not support it. Instead, use inline assembly to forcenative_apic_mem_readto use the mov instruction which is supported by the emulator.Signed-off-by Adam Dunlap <[email protected]>Reviewed-by Marc Orr <[email protected]>Reviewed-by Jacob Xu <[email protected]>--- arch/x86/include/asm/apic.h | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)arch/x86/include/asm/apic.h +2
-1@@ -109,7 +109,18 @@ static inline void native_apic_mem_write(u32 reg, u32 v)static inline u32 native_apic_mem_read (u32 reg) { -return *((volatile u32 *)(APIC_BASE + reg)); +volatile u32 *addr = (volatile u32 *)(APIC_BASE + reg); +u32 out;arch/x86/include/asm/apic.h static inline void native_apic_mem_write(u32 reg, u32 v) · line 123
Quoted 10 lines
arch/x86/include/asm/apic.h +10
-0+ +/* +* Functionally, what we want to do is simply return *addr. However, +* this accesses an MMIO which may need to be emulated in some cases. +* The emulator doesn't necessarily support all instructions, so we +* force the read from addr to use a mov instruction. +*/ + asm_inline("movl %1, %0" : "=r"(out) : "m"(*addr)); + + return out;Can't this just be:
return readl((void __iomem *)(APIC_BASE + reg));
Quoted 7 lines
Quoted 1 line
The very point of the patch is to force a specific instruction sequence.
Yes, and that specific emulator-friendly instruction also has to be forced for all of the core x86 read/write MMIO helpers. And it's also possible for MMIO read/write to be enlightened to skip the MOV and go straight to #VMGEXIT, i.e. the xAPIC code shouldn't assume MOV is the best/only option (ignoring the handling of the P54C erratum in the write path).
That's not reasonable... but xAPIC is "special" enough.
Quoted 47 lines · H. Peter Anvin
Quoted 45 lines · Sean Christopherson
Quoted 43 lines · H. Peter Anvin
Quoted 39 lines · Sean Christopherson
Quoted 25 lines · Adam Dunlap
Previously, when compiled with clang,
native_apic_mem_readgets inlined into__xapic_wait_icr_idleand optimized to a testl instruction. When run in a VM with SEV-ES enabled, it attempts to emulate this instruction, but the emulator does not support it. Instead, use inline assembly to forcenative_apic_mem_readto use the mov instruction which is supported by the emulator.Signed-off-by Adam Dunlap <[email protected]>Reviewed-by Marc Orr <[email protected]>Reviewed-by Jacob Xu <[email protected]>--- arch/x86/include/asm/apic.h | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)arch/x86/include/asm/apic.h +2
-1@@ -109,7 +109,18 @@ static inline void native_apic_mem_write(u32 reg, u32 v)static inline u32 native_apic_mem_read (u32 reg) { -return *((volatile u32 *)(APIC_BASE + reg)); +volatile u32 *addr = (volatile u32 *)(APIC_BASE + reg); +u32 out;arch/x86/include/asm/apic.h static inline void native_apic_mem_write(u32 reg, u32 v) · line 123
Quoted 10 lines
arch/x86/include/asm/apic.h +10
-0+ +/* +* Functionally, what we want to do is simply return *addr. However, +* this accesses an MMIO which may need to be emulated in some cases. +* The emulator doesn't necessarily support all instructions, so we +* force the read from addr to use a mov instruction. +*/ + asm_inline("movl %1, %0" : "=r"(out) : "m"(*addr)); + + return out;Can't this just be:
Quoted 1 line
return readl((void __iomem *)(APIC_BASE + reg));The very point of the patch is to force a specific instruction sequence.
Quoted 7 lines
Quoted 5 lines
Yes, and that specific emulator-friendly instruction also has to be forced for all of the core x86 read/write MMIO helpers. And it's also possible for MMIO read/write to be enlightened to skip the MOV and go straight to #VMGEXIT, i.e. the xAPIC code shouldn't assume MOV is the best/only option (ignoring the handling of the P54C erratum in the write path).
That's not reasonable... but xAPIC is "special" enough.
Thanks for your responses. I think for now it makes sense to use the readl function because
I haven't seen it require the ax register so can't verify the result. I will send out a modified
patch using readl shortly.
PATCH v2 1/1 x86/asm: Force native_apic_mem_read to use mov
Previously, when compiled with clang, native_apic_mem_read gets inlined into __xapic_wait_icr_idle and optimized to a testl instruction. When run in a VM with SEV-ES enabled, it attempts to emulate this instruction, but the emulator does not support it. Instead, use inline assembly to force native_apic_mem_read to use the mov instruction which is supported by the emulator.
---
V1 -> V2: Replaced asm with readl function which does the same thing
arch/x86/include/asm/apic.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/include/asm/apic.h +2 -1
@@ -12,6 +12,7 @@#include <asm/mpspec.h> #include <asm/msr.h> #include <asm/hardirq.h> +#include <asm/io.h> #define ARCH_APICTIMER_STOPS_ON_C3 1
@@ -109,7 +110,7 @@ static inline void native_apic_mem_write(u32 reg, u32 v)static inline u32 native_apic_mem_read (u32 reg) { -return *((volatile u32 *)(APIC_BASE + reg)); +return readl ((void __iomem *)(APIC_BASE + reg)); }extern void native_apic_wait_icr_idle (void );
Signature · 1 line
2.37.1.559.g78731f0fdb-goog
PATCH RESEND v2 1/1 x86/asm: Force native_apic_mem_read to use mov
Previously, when compiled with clang, native_apic_mem_read gets inlined into __xapic_wait_icr_idle and optimized to a testl instruction. When run in a VM with SEV-ES enabled, it attempts to emulate this instruction, but the emulator does not support it. Instead, use inline assembly to force native_apic_mem_read to use the mov instruction which is supported by the emulator.
---
V1 -> V2: Replaced asm with readl function which does the same thing
arch/x86/include/asm/apic.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/include/asm/apic.h +2 -1
@@ -12,6 +12,7 @@#include <asm/mpspec.h> #include <asm/msr.h> #include <asm/hardirq.h> +#include <asm/io.h> #define ARCH_APICTIMER_STOPS_ON_C3 1
@@ -109,7 +110,7 @@ static inline void native_apic_mem_write(u32 reg, u32 v)static inline u32 native_apic_mem_read (u32 reg) { -return *((volatile u32 *)(APIC_BASE + reg)); +return readl ((void __iomem *)(APIC_BASE + reg)); }extern void native_apic_wait_icr_idle (void );
Signature · 1 line
2.37.1.559.g78731f0fdb-goog
Quoted 7 lines · Adam Dunlap
Previously, when compiled with clang,
native_apic_mem_readgets inlined into__xapic_wait_icr_idleand optimized to a testl instruction. When run in a VM with SEV-ES enabled, it attempts to emulate this instruction, but the emulator does not support it. Instead, use inline assembly to forcenative_apic_mem_readto use the mov instruction which is supported by the emulator.
This seems to be an issue with the SEV-ES in guest #VC handler's "emulator" right?
If that's the case I think this should be fixed in the #VC handler instead of fixing the code that is failing to be emulated. What if there are other places where a testl is used and our tests have not caught them.
Quoted 10 lines · Peter Gonda
Quoted 7 lines · Adam Dunlap
Previously, when compiled with clang,
native_apic_mem_readgets inlined into__xapic_wait_icr_idleand optimized to a testl instruction. When run in a VM with SEV-ES enabled, it attempts to emulate this instruction, but the emulator does not support it. Instead, use inline assembly to forcenative_apic_mem_readto use the mov instruction which is supported by the emulator.
Quoted 7 lines
This seems to be an issue with the SEV-ES in guest #VC handler's "emulator" right?
If that's the case I think this should be fixed in the #VC handler instead of fixing the code that is failing to be emulated. What if there are other places where a testl is used and our tests have not caught them.
That was my initial reaction too. But we spoke w/ Tom offline (before sending this) and my understanding was that we really should be using MOV for MMIO. I've cc'd Tom so he can speak to this directly though.
Quoted 15 lines · Marc Orr
Quoted 13 lines · Peter Gonda
Quoted 7 lines · Adam Dunlap
Previously, when compiled with clang,
native_apic_mem_readgets inlined into__xapic_wait_icr_idleand optimized to a testl instruction. When run in a VM with SEV-ES enabled, it attempts to emulate this instruction, but the emulator does not support it. Instead, use inline assembly to forcenative_apic_mem_readto use the mov instruction which is supported by the emulator.This seems to be an issue with the SEV-ES in guest #VC handler's "emulator" right?
Quoted 8 lines
Quoted 4 lines
If that's the case I think this should be fixed in the #VC handler instead of fixing the code that is failing to be emulated. What if there are other places where a testl is used and our tests have not caught them.
That was my initial reaction too. But we spoke w/ Tom offline (before sending this) and my understanding was that we really should be using MOV for MMIO. I've cc'd Tom so he can speak to this directly though.
Actually cc'ing Tom :-).
Quoted 16 lines · Marc Orr
Quoted 15 lines · Marc Orr
Quoted 13 lines · Peter Gonda
Quoted 7 lines · Adam Dunlap
Previously, when compiled with clang,
native_apic_mem_readgets inlined into__xapic_wait_icr_idleand optimized to a testl instruction. When run in a VM with SEV-ES enabled, it attempts to emulate this instruction, but the emulator does not support it. Instead, use inline assembly to forcenative_apic_mem_readto use the mov instruction which is supported by the emulator.This seems to be an issue with the SEV-ES in guest #VC handler's "emulator" right?
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
If that's the case I think this should be fixed in the #VC handler instead of fixing the code that is failing to be emulated. What if there are other places where a testl is used and our tests have not caught them.
That was my initial reaction too. But we spoke w/ Tom offline (before sending this) and my understanding was that we really should be using MOV for MMIO. I've cc'd Tom so he can speak to this directly though.
I did finally find the section in our PPR that references using the MOV instruction, but it was for MMIO configuration space, not general MMIO operations.
So, yes, the #VC handler could be extended to handle a TEST instruction to fix this. My worry would be if the compiler decided to use a different instruction in the future. I see that the native_apic_mem_write() is using assembler to perform its operation, it just seemed right that the native_apic_mem_read() could do the same.
Thanks,
Tom
Quoted 2 lines
Actually cc'ing Tom :-).
Quoted 9 lines · Peter Gonda
Quoted 6 lines · Adam Dunlap
Previously, when compiled with clang,
native_apic_mem_readgets inlined into__xapic_wait_icr_idleand optimized to a testl instruction. When run in a VM with SEV-ES enabled, it attempts to emulate this instruction, but the emulator does not support it. Instead, use inline assembly to forcenative_apic_mem_readto use the mov instruction which is supported by the emulator.This seems to be an issue with the SEV-ES in guest #VC handler's "emulator" right?
No.
It's not just an SEV-ES thing. It's a problem for TDX and _probably_ a problem for normal virtualization where it's a host-side issue. Kirill wrote a lot of great background information in here:
So, the question is not "should we extend the MMIO instruction decoders to handle one more instruction?". It is "should we extend the MMIO decoders to handle *ALL* memory read instructions?"
That's an even more emphatic "NO".
readl() seems to be the right thing to do. Also, Dear TDX, SEV and virt
folks: please look for more of these. They're going to bite you sooner or later. You should have caught this one before now.
Quoted 17 lines · Dave Hansen
Quoted 9 lines · Peter Gonda
Quoted 6 lines · Adam Dunlap
Previously, when compiled with clang,
native_apic_mem_readgets inlined into__xapic_wait_icr_idleand optimized to a testl instruction. When run in a VM with SEV-ES enabled, it attempts to emulate this instruction, but the emulator does not support it. Instead, use inline assembly to forcenative_apic_mem_readto use the mov instruction which is supported by the emulator.This seems to be an issue with the SEV-ES in guest #VC handler's "emulator" right?
No.
It's not just an SEV-ES thing. It's a problem for TDX and _probably_ a problem for normal virtualization where it's a host-side issue. Kirill wrote a lot of great background information in here:
Quoted 7 lines
So, the question is not "should we extend the MMIO instruction decoders to handle one more instruction?". It is "should we extend the MMIO decoders to handle *ALL* memory read instructions?"
That's an even more emphatic "NO".
+1, keep the guest-side decoding as simple as possible.
Quoted 3 lines
readl()seems to be the right thing to do. Also, Dear TDX, SEV and virt
folks: please look for more of these. They're going to bite you sooner or later. You should have caught this one before now.
Quoted 21 lines · Sean Christopherson
Quoted 19 lines · Dave Hansen
Quoted 9 lines · Peter Gonda
Quoted 6 lines · Adam Dunlap
Previously, when compiled with clang,
native_apic_mem_readgets inlined into__xapic_wait_icr_idleand optimized to a testl instruction. When run in a VM with SEV-ES enabled, it attempts to emulate this instruction, but the emulator does not support it. Instead, use inline assembly to forcenative_apic_mem_readto use the mov instruction which is supported by the emulator.This seems to be an issue with the SEV-ES in guest #VC handler's "emulator" right?
No.
It's not just an SEV-ES thing. It's a problem for TDX and _probably_ a problem for normal virtualization where it's a host-side issue. Kirill wrote a lot of great background information in here:
Quoted 11 lines
Quoted 5 lines
So, the question is not "should we extend the MMIO instruction decoders to handle one more instruction?". It is "should we extend the MMIO decoders to handle *ALL* memory read instructions?"
That's an even more emphatic "NO".
+1, keep the guest-side decoding as simple as possible.
Quoted 3 lines
readl()seems to be the right thing to do. Also, Dear TDX, SEV and virt
folks: please look for more of these. They're going to bite you sooner or later. You should have caught this one before now.
Thanks for the context here Dave. Fixing the offending MMIO instruction here makes sense.
A message that is not in the archive (sent off-list or not yet received): CAMBK9=YB=8EQymDUda300qPFAL1=7dzC61c0pshrWEC5ibrUfQ@mail.gmail.com
[resent with plain text]
Thanks for all the responses. Is the consensus that we should use the readl function here or instead use inline assembly directly as in the patch I originally sent out:
asm_inline("movl %1, %0" : "=r"(out) : "m"(*addr));
? The readl function has this exact same code, I'm just not sure which version fits better stylistically.
Quoted 9 lines · Adam Dunlap
Thanks for all the responses. Is the consensus that we should use the readl function here or instead use inline assembly directly as in the patch I originally sent out:
asm_inline("movl %1, %0" : "=r"(out) : "m"(*addr));? The readl function has this exact same code, I'm just not sure which version fits better stylistically.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 11 lines · H. Peter Anvin
Quoted 8 lines · Adam Dunlap
Thanks for all the responses. Is the consensus that we should use the readl function here or instead use inline assembly directly as in the patch I originally sent out:
asm_inline("movl %1, %0" : "=r"(out) : "m"(*addr));? The readl function has this exact same code, I'm just not sure which version fits better stylistically.
Is mov with an arbitrary addressing mode still acceptable for whatever is causing this problem?
The acceptable forms of MOV are covered by insn_decode_mmio() in arch/x86/lib/insn-eval.c.
Thanks,
Tom
Quoted 9 lines · Tom Lendacky
Quoted 7 lines · H. Peter Anvin
Quoted 6 lines · Adam Dunlap
Thanks for all the responses. Is the consensus that we should use the readl function here or instead use inline assembly directly as in the patch I originally sent out:
asm_inline("movl %1, %0" : "=r"(out) : "m"(*addr));
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
? The readl function has this exact same code, I'm just not sure which version fits better stylistically.
Is mov with an arbitrary addressing mode still acceptable for whatever is causing this problem?
The acceptable forms of MOV are covered by
insn_decode_mmio()inarch/x86/lib/insn-eval.c.
Is this blocked on an item? There seems to be consensus that this patch fixes a bug and is taking the right high-level approach (i.e., change the guest code to avoid triggering a sequence that isn't supported under CVM exception-based emulation). Without something like this, we weren't able to build the kernel w/ CLANG when it is configured to run under SEV-ES.
We sent out two versions of the patch. One that does the mov directly
[1] and a second that calls readl [2]. Is one of these two patches acceptable? Or do we need to follow up on something?
[1] https://lore.kernel.org/lkml/[email protected]/T/
[2] https://lore.kernel.org/all/[email protected]/
Thanks,
Marc
Quoted 7 lines
Is this blocked on an item? There seems to be consensus that this patch fixes a bug and is taking the right high-level approach (i.e., change the guest code to avoid triggering a sequence that isn't supported under CVM exception-based emulation). Without something like this, we weren't able to build the kernel w/ CLANG when it is configured to run under SEV-ES.
Quoted 6 lines
We sent out two versions of the patch. One that does the mov directly
[1] and a second that calls readl [2]. Is one of these two patches acceptable? Or do we need to follow up on something?[1] https://lore.kernel.org/lkml/[email protected]/T/
[2] https://lore.kernel.org/all/[email protected]/
Signal-boosting this thread: is this blocked on any item?
We are still running into this issue (SEV-ES guest unexpectedly requests termination minutes after booting) and applying this patch seems to fix it.
Thanks,
Sid
Quoted 7 lines · Sidharth Telang
Quoted 6 lines
Is this blocked on an item? There seems to be consensus that this patch fixes a bug and is taking the right high-level approach (i.e., change the guest code to avoid triggering a sequence that isn't supported under CVM exception-based emulation). Without something like this, we weren't able to build the kernel w/ CLANG when it is configured to run under SEV-ES.
Quoted 8 lines
Quoted 6 lines
We sent out two versions of the patch. One that does the mov directly
[1] and a second that calls readl [2]. Is one of these two patches acceptable? Or do we need to follow up on something?[1] https://lore.kernel.org/lkml/[email protected]/T/
[2] https://lore.kernel.org/all/[email protected]/Signal-boosting this thread: is this blocked on any item?
Yes, it's blocked on you sending a well-described patch.
You sent out two patches which both received a lot of discussion and induced a lot of confusion. Can you please take all the knowledge from this thread and send a third patch that has a proper changelog incorporating all that knowledge?
Which approach should that patch have? Whatever one is as close to what native_apic_mem_write() does as possible.
H. Peter Anvin
reviewerQuoted 28 lines · Adam Dunlap
Previously, when compiled with clang,
native_apic_mem_readgets inlined into__xapic_wait_icr_idleand optimized to a testl instruction. When run in a VM with SEV-ES enabled, it attempts to emulate this instruction, but the emulator does not support it. Instead, use inline assembly to forcenative_apic_mem_readto use the mov instruction which is supported by the emulator.Signed-off-by Adam Dunlap <[email protected]>Reviewed-by Marc Orr <[email protected]>Reviewed-by Jacob Xu <[email protected]>---
arch/x86/include/asm/apic.h| 13 ++++++++++++-1 file changed, 12 insertions(+), 1 deletion(-)arch/x86/include/asm/apic.h +5
-1@@ -109,7 +109,18 @@ static inline void native_apic_mem_write(u32 reg, u32 v)static inline u32 native_apic_mem_read (u32 reg) { -return *((volatile u32 *)(APIC_BASE + reg)); +volatile u32 *addr = (volatile u32 *)(APIC_BASE + reg); +u32 out; + +/* +* Functionally, what we want to do is simply return *addr. However,
arch/x86/include/asm/apic.h static inline void native_apic_mem_write(u32 reg, u32 v) · line 126
Quoted 10 lines
arch/x86/include/asm/apic.h +7
-0+* this accesses an MMIO which may need to be emulated in some cases. +* The emulator doesn't necessarily support all instructions, so we +* force the read from addr to use a mov instruction. +*/ + asm_inline("movl %1, %0" : "=r"(out) : "m"(*addr)); + + return out; }extern void native_apic_wait_icr_idle (void );
As I recall, there are some variants which only support using the ax register, so if you are going to do this might as well go all the way and force a specific instruction with specific registers, like mov (%rdx),%rax (by analogy with the I/O registers.)