clflush instruction emulation

PATCH In review Wei Huang 2 participants 6 messages Apr 15, 2010

Share & tools

Wei Huang

author

2010-04-15 16:38 UTC #0 lore raw

PATCH 1/1 clflush instruction emulation

We recently found that FreeBSD 8.0 guest failed to install and boot on Xen. The reason was that FreeBSD detected clflush feature and invoked this instruction to flush MMIO space. This caused a page fault; but x86_emulate.c failed to emulate this instruction (not supported). As a result, a page fault was detected inside FreeBSD. A similar issue was reported earlier.

http://lists.xensource.com/archives/html/xen-devel/2010-03/msg00362.html

I created a patch which enables clflush emulation. I have verified that this patch solves FreeBSD issue. This patch returns immediately without doing anything. Note that we don't have to intercept clflush in SVM. So the only time we have page_fault for clflush instruction is when guest VM flushes MMIO space. In this case, do we need to send this command over to QEMU? I didn't do it in this patch anyway.

Best,
-Wei

xen/arch/x86/x86_emulate/x86_emulate.c +7 -1

diff -r 12a610b600b0 xen/arch/x86/x86_emulate/x86_emulate.c
@@ -227,7 +227,8 @@
     DstMem|SrcReg|ModRM, DstMem|SrcReg|ModRM, 0, 0,
     /* 0xA8 - 0xAF */
     ImplicitOps, ImplicitOps, 0, DstBitBase|SrcReg|ModRM,
-    DstMem|SrcReg|ModRM, DstMem|SrcReg|ModRM, 0, DstReg|SrcMem|ModRM,
+    DstMem|SrcReg|ModRM, DstMem|SrcReg|ModRM, DstMem|SrcReg|ModRM, 
+    DstReg|SrcMem|ModRM,
     /* 0xB0 - 0xB7 */
     ByteOp|DstMem|SrcReg|ModRM, DstMem|SrcReg|ModRM,
     DstReg|SrcMem|ModRM|Mov, DstBitBase|SrcReg|ModRM,

@@ -3948,6 +3949,11 @@
         src.val = x86_seg_gs;
         goto pop_seg;
 
+    case 0xae: /* clflush mem8 */
+        /* we don't need to do anything here */
+        rc = X86EMUL_OKAY;
+        break;
+
     case 0xb0 ... 0xb1: /* cmpxchg */
         /* Save real source value, then compare EAX against destination. */
         src.orig_val = src.val;

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Keir Fraser

2010-04-15 16:59 UTC #1 lore raw

Quoted 6 lines · Wei Huang

I created a patch which enables clflush emulation. I have verified that this patch solves FreeBSD issue. This patch returns immediately without doing anything. Note that we don't have to intercept clflush in SVM. So the only time we have page_fault for clflush instruction is when guest VM flushes MMIO space. In this case, do we need to send this command over to QEMU? I didn't do it in this patch anyway.

I don't think we need to anything since all guest reads/writes to the mmio space will get intercepted to qemu, and hence qemu's view of the space is coherent with the guest with no need for special handling of CLFLUSH.

Still the emulator should call out and let the caller decide what to do. We could safely turn CLFLUSH into WBINVD (i.e., call the existing wbinvd hook). Do you know if these CLFLUSH emulations happen often or only very rarely? That might help us decide how smart we need to be about emulating CLFLUSH (i.e., add a hook specifically for clflush, separately from wbinvd).

-- Keir

Wei Huang

author

2010-04-15 17:25 UTC #2 lore raw

WBINVD is a bit heavy since it flushes all caches. I saw 1170 times of CLFLUSH during FreeBSD booting. After guest boots, 0 was observed. 1170 is not huge though.

-Wei

Quoted 26 lines · Keir Fraser

Quoted 7 lines · Wei Huang
I created a patch which enables clflush emulation. I have verified that this patch solves FreeBSD issue. This patch returns immediately without doing anything. Note that we don't have to intercept clflush in SVM. So the only time we have page_fault for clflush instruction is when guest VM flushes MMIO space. In this case, do we need to send this command over to QEMU? I didn't do it in this patch anyway.
I don't think we need to anything since all guest reads/writes to the mmio space will get intercepted to qemu, and hence qemu's view of the space is coherent with the guest with no need for special handling of CLFLUSH.
Still the emulator should call out and let the caller decide what to do. We could safely turn CLFLUSH into WBINVD (i.e., call the existing wbinvd hook). Do you know if these CLFLUSH emulations happen often or only very rarely? That might help us decide how smart we need to be about emulating CLFLUSH (i.e., add a hook specifically for clflush, separately from wbinvd).
-- Keir

Keir Fraser

2010-04-15 17:32 UTC #3 lore raw

Our WBINVD emulation doesn't do anything unless the guest has some devices passed through to it. I think that will do for now then.

Thanks,
Keir

Quoted 35 lines · Wei Huang

WBINVD is a bit heavy since it flushes all caches. I saw 1170 times of CLFLUSH during FreeBSD booting. After guest boots, 0 was observed. 1170 is not huge though.
-Wei
Quoted 26 lines · Keir Fraser
Quoted 7 lines · Wei Huang
I created a patch which enables clflush emulation. I have verified that this patch solves FreeBSD issue. This patch returns immediately without doing anything. Note that we don't have to intercept clflush in SVM. So the only time we have page_fault for clflush instruction is when guest VM flushes MMIO space. In this case, do we need to send this command over to QEMU? I didn't do it in this patch anyway.
I don't think we need to anything since all guest reads/writes to the mmio space will get intercepted to qemu, and hence qemu's view of the space is coherent with the guest with no need for special handling of CLFLUSH.
Still the emulator should call out and let the caller decide what to do. We could safely turn CLFLUSH into WBINVD (i.e., call the existing wbinvd hook). Do you know if these CLFLUSH emulations happen often or only very rarely? That might help us decide how smart we need to be about emulating CLFLUSH (i.e., add a hook specifically for clflush, separately from wbinvd).
-- Keir

Keir Fraser

2010-04-15 17:49 UTC #4 lore raw

See what you think of xen-unstable:21189. It is also a bit more precise in its decoding of CLFLUSH, as a member of Grp15.

-- Keir

Quoted 50 lines · Keir Fraser

Our WBINVD emulation doesn't do anything unless the guest has some devices passed through to it. I think that will do for now then.
Thanks,
Keir
Quoted 4 lines · Wei Huang
WBINVD is a bit heavy since it flushes all caches. I saw 1170 times of CLFLUSH during FreeBSD booting. After guest boots, 0 was observed. 1170 is not huge though.
Quoted 31 lines
-Wei
Quoted 26 lines · Keir Fraser
Quoted 7 lines · Wei Huang
I created a patch which enables clflush emulation. I have verified that this patch solves FreeBSD issue. This patch returns immediately without doing anything. Note that we don't have to intercept clflush in SVM. So the only time we have page_fault for clflush instruction is when guest VM flushes MMIO space. In this case, do we need to send this command over to QEMU? I didn't do it in this patch anyway.
I don't think we need to anything since all guest reads/writes to the mmio space will get intercepted to qemu, and hence qemu's view of the space is coherent with the guest with no need for special handling of CLFLUSH.
Still the emulator should call out and let the caller decide what to do. We could safely turn CLFLUSH into WBINVD (i.e., call the existing wbinvd hook). Do you know if these CLFLUSH emulations happen often or only very rarely? That might help us decide how smart we need to be about emulating CLFLUSH (i.e., add a hook specifically for clflush, separately from wbinvd).
-- Keir
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Huang2, Wei

author

2010-04-15 18:43 UTC #5 lore raw

I can confirm that 21189 fixed FreeBSD issue.

Thanks,
-Wei

-----Original Message-----
From: Keir Fraser [mailto:[email protected]]
Sent: Thursday, April 15, 2010 12:50 PM
To: Huang2, Wei
Cc: '[email protected]'
Subject: Re: [Xen-devel] [PATCH] clflush instruction emulation

See what you think of xen-unstable:21189. It is also a bit more precise in its decoding of CLFLUSH, as a member of Grp15.

-- Keir

Quoted 50 lines · Keir Fraser

Our WBINVD emulation doesn't do anything unless the guest has some devices passed through to it. I think that will do for now then.
Thanks,
Keir
Quoted 4 lines · Wei Huang
WBINVD is a bit heavy since it flushes all caches. I saw 1170 times of CLFLUSH during FreeBSD booting. After guest boots, 0 was observed. 1170 is not huge though.
Quoted 31 lines
-Wei
Quoted 26 lines · Keir Fraser
Quoted 7 lines · Wei Huang
I created a patch which enables clflush emulation. I have verified that this patch solves FreeBSD issue. This patch returns immediately without doing anything. Note that we don't have to intercept clflush in SVM. So the only time we have page_fault for clflush instruction is when guest VM flushes MMIO space. In this case, do we need to send this command over to QEMU? I didn't do it in this patch anyway.
I don't think we need to anything since all guest reads/writes to the mmio space will get intercepted to qemu, and hence qemu's view of the space is coherent with the guest with no need for special handling of CLFLUSH.
Still the emulator should call out and let the caller decide what to do. We could safely turn CLFLUSH into WBINVD (i.e., call the existing wbinvd hook). Do you know if these CLFLUSH emulations happen often or only very rarely? That might help us decide how smart we need to be about emulating CLFLUSH (i.e., add a hook specifically for clflush, separately from wbinvd).
-- Keir
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel