Add support for cmpxchg16b emulation
v4 cover Add support for cmpxchg16b emulation
This series adds 128-bit compare-exchange support needed for KVM to emulate guest cmpxchg16b instructions.
The AMD IOMMU driver updates 256-bit device table entries with two 128-bit cmpxchg operations. For hardware-accelerated vIOMMU, QEMU traps those DTE accesses to install nested page tables [1]. KVM must emulate guest cmpxchg16b on the trapped MMIO regions; without that, DTE access emulation fails.
Patch 1: extends the x86 user CMPXCHG helpers with __try_cmpxchg128_user_asm() (cmpxchg16b on x86-64), wired into unsafe_try_cmpxchg_user().
Patch 2: extends KVM CMPXCHG8B emulation to 16-byte operands (REX.W=1) and uses the new helper for atomic 16-byte compare-exchange on guest RAM via emulator_cmpxchg_emulated().
You can find the KUT to test this series at [2].
[1] https://github.com/AMDESE/qemu-iommu/blob/wip/for_iommufd_hw_queue-v8_amd_viommu_20260106/hw/i386/amd_viommu.c#L517
[2] https://lore.kernel.org/kvm/[email protected]/T/#u
Changes since v3:
https://lore.kernel.org/all/[email protected]/
Patch1: Split the old value across RAX and RDX using "+a" and "+b" instead of
using "+A" [Dave]
Patch 2: Use __c instead of ctx in cmpxchg macro [Sashiko]
Changes since v2:
https://lore.kernel.org/all/[email protected]/
Patch 1: Trigger build failure if __try_cmpxchg128_user_asm is called on 32 bit
machines [Sean]
Sairaj Kodilkar (2):
x86/uaccess: Extend CMPXCHG user helpers to 128-bit operands
KVM: x86: Add support for cmpxchg16b emulation
arch/x86/include/asm/uaccess.h | 64 +++++++++++++++++++++++++++++++++- arch/x86/kvm/emulate.c | 50 +++++++++++++++++--------- arch/x86/kvm/kvm_emulate.h | 6 ++++ arch/x86/kvm/x86.c | 9 ++++- 4 files changed, 111 insertions(+), 18 deletions(-)
base-commit: 45c13f3f9e3bb15fd89ff2864c6f627a3b4b4229
Signature · 1 line
2.34.1
PATCH v4 1/2 x86/uaccess: Extend CMPXCHG user helpers to 128-bit operands
Extend the existing user CMPXCHG helpers to support 16-byte operands on x86-64, using LOCK_PREFIX "cmpxchg16b". This mirrors the existing __try_cmpxchg64_user_asm() / cmpxchg8b path provided for 32-bit kernels, where KVM needs an atomic compare-exchange wider than the generic cmpxchg helper can provide.
On 32-bit kernels, stub the helper to generate build failure because cmpxchg16b requires 64-bit GPRs and is not available.
KVM uses this to atomically emulate guest cmpxchg16b on guest RAM mapped via userspace addresses.
--- arch/x86/include/asm/uaccess.h | 64 +++++++++++++++++++++++++++++++++- 1 file changed, 63 insertions(+), 1 deletion(-)
arch/x86/include/asm/uaccess.h +63 -1
@@ -407,6 +407,27 @@ do { \if (unlikely (!success)) \ *_old = __old; \ likely(success); }) +#else // !CONFIG_X86_32 +#define __try_cmpxchg128_user_asm (_ptr, _pold, _new, label) ({ \ + bool success; \ + __typeof__(_ptr) _old = (__typeof__(_ptr))(_pold); \ + u64 __old_low = (u64)*_old; \ + u64 __old_high = (u64)(*_old >> 64); \ + __typeof__(*(_ptr)) __new = (_new); \ + asm_goto_output("\n" \ + "1: " LOCK_PREFIX "cmpxchg16b %[ptr]\n" \ + _ASM_EXTABLE_UA(1b, %l[label]) \ + : "=@ccz" (success), \ + "+a" (__old_low), \ + "+d" (__old_high), \ + [ptr] "+m" (*_ptr) \ + : "b" ((u64)__new), \ + "c" ((u64)((u128)__new >> 64)) \ + : "memory" \ + : label); \ + if (unlikely(!success)) \ + *_old = ((u128)__old_high << 64) | __old_low; \ + likely(success); })#endif // CONFIG_X86_32 #else // !CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT #define __try_cmpxchg_user_asm (itype, ltype, _ptr, _pold, _new, label) ({ \
@@ -463,6 +484,32 @@ do { \if (unlikely (!__result)) \ *_old = __old; \ likely(__result); }) +#else //!CONFIG_X86_32 +#define __try_cmpxchg128_user_asm (_ptr, _pold, _new, label) ({ \ + int __result; \ + __typeof__(_ptr) _old = (__typeof__(_ptr))(_pold); \ + u64 __old_low = (u64)*_old; \ + u64 __old_high = (u64)(*_old >> 64); \ + __typeof__(*(_ptr)) __new = (_new); \ + asm volatile("\n" \ + "1: " LOCK_PREFIX "cmpxchg16b %[ptr]\n" \ + "mov $0, %[result]\n\t" \ + "setz %b[result]\n" \ + "2:\n" \ + _ASM_EXTABLE_TYPE_REG(1b, 2b, EX_TYPE_EFAULT_REG, \ + %[result]) \ + : [result] "=q" (__result), \ + "+a" (__old_low), \ + "+d" (__old_high), \ + [ptr] "+m" (*_ptr) \ + : "b" ((u64)__new), \ + "c" ((u64)((u128)__new >> 64)) \ + : "memory", "cc"); \ + if (unlikely(__result < 0)) \ + goto label; \ + if (unlikely(!__result)) \ + *_old = ((u128)__old_high << 64) | __old_low; \ + likely(__result); })#endif // CONFIG_X86_32 #endif // CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT
@@ -551,11 +598,18 @@ do { \extern void __try_cmpxchg_user_wrong_size (void ); -#ifndef CONFIG_X86_32 +#if defined(CONFIG_X86_32 ) || !defined(X86_FEATURE_CX16 ) +/* +* Always fail on 32 bit arch or 64 arch without 128 bit cmpxchg support +*/ +#define __try_cmpxchg128_user_asm (_ptr, _pold, _new, label) ({ BUILD_BUG_ON(1); 0; }) +#else #define __try_cmpxchg64_user_asm (_ptr, _oldp, _nval, _label) \ __try_cmpxchg_user_asm("q", "r", (_ptr), (_oldp), (_nval), _label) +#endif +/* * Force the pointer to u<size> to match the size expected by the asm helper. * clang/LLVM compiles all cases and only discards the unused paths after
@@ -580,6 +634,14 @@ extern void __try_cmpxchg_user_wrong_size(void); case 8: __ret = __try_cmpxchg64_user_asm((__force u64 *)(_ptr), (_oldp),\ (_nval), _label); \break ; \ +case 16 : \ +if (boot_cpu_has (X86_FEATURE_CX16 )) \ + __ret =__try_cmpxchg128_user_asm ( \ + (__force u128 *)(_ptr), \ + (_oldp), (_nval), _label); \ +else \ + __ret =0 ; \ +break ; \default :__try_cmpxchg_user_wrong_size (); \ } \ __ret; })
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [PATCH v4 1/2] x86/uaccess: Extend CMPXCHG user helpers to 128-bit operands
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] When the host CPU lacks CX16 support, the 16-byte fallback returns a mismatch without updating the…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] When the host CPU lacks CX16 support, the 16-byte fallback returns a mismatch without updating the old value pointer, violating the cmpxchg API contract and causing infinite retry loops. --
commit 5f4dae7c1efaa8622accc402392f8d8a532481b4
Author: Sairaj Kodilkar <[email protected]>
x86/uaccess: Extend CMPXCHG user helpers to 128-bit operands
This commit extends the user CMPXCHG helpers to support 16-byte operands on x86-64 using cmpxchg16b. KVM uses this to atomically emulate guest cmpxchg16b on guest RAM mapped via userspace addresses.
Quoted 3 lines
arch/x86/include/asm/uaccess.h +0
-0
[ ... ]
arch/x86/include/asm/uaccess.h extern void __try_cmpxchg_user_wrong_size(void); · line 643
Quoted 11 lines
arch/x86/include/asm/uaccess.h +7
-0@@ -580,6 +634,14 @@ extern void __try_cmpxchg_user_wrong_size(void); case 8: __ret = __try_cmpxchg64_user_asm((__force u64 *)(_ptr), (_oldp),\ (_nval), _label); \break ; \ +case 16 : \ +if (boot_cpu_has (X86_FEATURE_CX16 )) \ + __ret =__try_cmpxchg128_user_asm ( \ + (__force u128 *)(_ptr), \ + (_oldp), (_nval), _label); \ +else \ + __ret =0 ; \
[Severity: High]
Does setting __ret to 0 here on hardware without CX16 support violate the cmpxchg API contract?
When boot_cpu_has(X86_FEATURE_CX16) is false, it returns 0 (mismatch) but fails to fetch the actual memory value and write it into _oldp.
If a guest executes cmpxchg16b on an MMIO address (forcing KVM emulation), KVM's emulator decodes it and invokes unsafe_try_cmpxchg_user. Because it returns 0 without updating the pointer, x86_emulate_insn skips writeback_registers() and returns EMULATION_OK.
The guest registers remain unchanged while EFLAGS shows failure (ZF=0).
Can this cause a denial of service where the guest endlessly retries the instruction with the same stale registers?
Should this instead use goto _label; to cleanly abort the unsupported operation, which would cause KVM to properly abort the emulation?
Quoted 4 lines
arch/x86/include/asm/uaccess.h +1
-0+break ; \default :__try_cmpxchg_user_wrong_size (); \ } \ __ret; })
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Re: [PATCH v4 1/2] x86/uaccess: Extend CMPXCHG user helpers to 128-bit operands
Quoted 41 lines · On 9/8/2026 1:49 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] When the host CPU lacks CX16 support, the 16-byte fallback returns a mismatch without updating the old value pointer, violating the cmpxchg API contract and causing infinite retry loops. --commit
5f4dae7c1efaa8622accc402392f8d8a532481b4
Author: Sairaj Kodilkar <[email protected]>x86/uaccess: Extend CMPXCHG user helpers to 128-bit operands
This commit extends the user CMPXCHG helpers to support 16-byte operands on x86-64 using cmpxchg16b. KVM uses this to atomically emulate guest cmpxchg16b on guest RAM mapped via userspace addresses.
Quoted 3 lines
arch/x86/include/asm/uaccess.h +0
-0[ ... ]
arch/x86/include/asm/uaccess.h extern void __try_cmpxchg_user_wrong_size(void); · line 643
Quoted 11 lines
arch/x86/include/asm/uaccess.h +7
-0@@ -580,6 +634,14 @@ extern void __try_cmpxchg_user_wrong_size(void); case 8: __ret = __try_cmpxchg64_user_asm((__force u64 *)(_ptr), (_oldp),\ (_nval), _label); \break ; \ +case 16 : \ +if (boot_cpu_has (X86_FEATURE_CX16 )) \ + __ret =__try_cmpxchg128_user_asm ( \ + (__force u128 *)(_ptr), \ + (_oldp), (_nval), _label); \ +else \ + __ret =0 ; \[Severity: High]
Does setting__retto 0 here on hardware without CX16 support violate the cmpxchg API contract?When
boot_cpu_has(X86_FEATURE_CX16)is false, it returns 0 (mismatch) but fails to fetch the actual memory value and write it into_oldp.If a guest executes cmpxchg16b on an MMIO address (forcing KVM emulation), KVM's emulator decodes it and invokes
unsafe_try_cmpxchg_user. Because it returns 0 without updating the pointer,x86_emulate_insnskipswriteback_registers()and returnsEMULATION_OK.
Quoted 8 lines
The guest registers remain unchanged while EFLAGS shows failure (ZF=0).
Can this cause a denial of service where the guest endlessly retries the instruction with the same stale registers?
Should this instead use goto
_label; to cleanly abort the unsupported operation, which would cause KVM to properly abort the emulation?
The caller __try_cmpxchg_user() flips the return value, so 0 means failure from unsafe_try_cmpxchg_user(). Hence writeback_registers() will fail with X86EMUL_CMPXCHG_FAILED which will halt the guest.
Thanks
Sairaj
Quoted 5 lines
Quoted 4 lines
arch/x86/include/asm/uaccess.h +1
-0+break ; \default :__try_cmpxchg_user_wrong_size (); \ } \ __ret; })
PATCH v4 2/2 KVM: x86: Add support for cmpxchg16b emulation
In reply to Sairaj KodilkarAMD and Intel both provides support for 128 bit cmpxchg operands using cmpxchg8b/cmpxchg16b instructions (opcode 0FC7). However, kvm does not support emulating cmpxchg16b (i.e when destination memory is 128 bit and REX.W = 1) which causes emulation failure when QEMU guest performs a cmpxchg16b on a memory region setup as a IO.
This failure is seen on the AMD IOMMU driver which writes 256-bit device table entries with two 128-bit cmpxchg operations. For guests using hardware-accelerated vIOMMU, QEMU traps device table accesses to set up nested page tables (see [1]). Without 128-bit cmpxchg emulation, KVM cannot handle these traps and DTE access emulation fails.
Hence extend cmpxchg8b to perform cmpxchg16b when the destination memory is 128 bit.
--- arch/x86/include/asm/uaccess.h | 2 +- arch/x86/kvm/emulate.c | 50 +++++++++++++++++++++++----------- arch/x86/kvm/kvm_emulate.h | 6 ++++ arch/x86/kvm/x86.c | 7 ++++- 4 files changed, 47 insertions(+), 18 deletions(-)
arch/x86/include/asm/uaccess.h +1 -1
@@ -598,7 +598,7 @@ do { \extern void __try_cmpxchg_user_wrong_size (void ); -#if defined(CONFIG_X86_32 ) || !defined(X86_FEATURE_CX16 ) +#ifdef CONFIG_X86_32 /* * Always fail on 32 bit arch or 64 arch without 128 bit cmpxchg support */
arch/x86/kvm/emulate.c +34 -16
66 lines not on this page · raw message
arch/x86/kvm/kvm_emulate.h +6 -0
@@ -263,6 +263,9 @@ struct operand {union {unsigned long orig_val;u64 orig_val64; +#ifdef CONFIG_X86_64 +u128 orig_val128; +#endif };union {unsigned long *reg;
@@ -276,6 +279,9 @@ struct operand {union {unsigned long val;u64 val64; +#ifdef CONFIG_X86_64 +u128 val128; +#endif char valptr[sizeof (avx256_t)];sse128_t vec_val;avx256_t vec_val2;
arch/x86/kvm/x86.c +6 -1
@@ -5271,7 +5271,7 @@ static int emulator_cmpxchg_emulated(struct x86_emulate_ctxt *ctxt,int r;/* guests cmpxchg8b have to be emulated atomically */ -if (bytes >8 || (bytes & (bytes -1 ))) +if (bytes >2 *sizeof (unsigned long ) || (bytes & (bytes -1 ))) goto emul_write; gpa =kvm_mmu_gva_to_gpa_write (vcpu, addr,NULL );
@@ -5311,6 +5311,11 @@ static int emulator_cmpxchg_emulated(struct x86_emulate_ctxt *ctxt,case 8 : r =emulator_try_cmpxchg_user (u64, hva, old, new);break ; +#ifdef CONFIG_X86_64 +case 16 : + r =emulator_try_cmpxchg_user (u128, hva, old, new); +break ; +#endif default :BUG (); }
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [PATCH v4 2/2] KVM: x86: Add support for cmpxchg16b emulation
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Missing guest CPUID check for X86_FEATURE_CX16 allows guest to invoke CMPXCHG16B emulation…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Missing guest CPUID check for X86_FEATURE_CX16 allows guest to invoke CMPXCHG16B emulation regardless of guest capabilities. --
commit 17ba312dad40f9de38e63207a6f7c861db26c57a
Author: Sairaj Kodilkar <[email protected]>
Subject: KVM: x86: Add support for cmpxchg16b emulation
This commit extends cmpxchg8b emulation in KVM to support the cmpxchg16b instruction when the destination memory is 128-bit and REX.W is set. This addresses an emulation failure where guests, such as the AMD IOMMU driver, use 128-bit cmpxchg operations on MMIO regions.
Quoted 4 lines
arch/x86/kvm/emulate.c +0
-0
[ ... ]
Quoted 1 line
arch/x86/kvm/emulate.c +0
-0@@ -2184,24 +2184,36 @@ static int em_call_near_abs(struct x86_emulate_ctxt *ctxt)
[ ... ]
Quoted 2 lines
static int
em_cmpxchg8b(struct x86_emulate_ctxt *ctxt){
[ ... ]
Quoted 10 lines
arch/x86/kvm/emulate.c +1
-7-if (ctxt->dst.bytes ==16 ) +if (WARN_ON_ONCE (8 + !!(ctxt->rex_bits &REX_W ) *8 != ctxt->dst.bytes))return X86EMUL_UNHANDLEABLE ; -if (((u32) (old >>0 ) != (u32 )reg_read (ctxt,VCPU_REGS_RAX )) || - ((u32) (old >>32 ) != (u32 )reg_read (ctxt,VCPU_REGS_RDX ))) { - *reg_write (ctxt,VCPU_REGS_RAX ) = (u32) (old >>0 ); - *reg_write (ctxt,VCPU_REGS_RDX ) = (u32) (old >>32 ); - ctxt->eflags &= ~X86_EFLAGS_ZF ; - }else {
arch/x86/kvm/emulate.c static int em_call_near_abs(struct x86_emulate_ctxt *ctxt) · line 2215
Quoted 10 lines
arch/x86/kvm/emulate.c +5
-5- ctxt->dst.val64 = ((u64 )reg_read (ctxt,VCPU_REGS_RCX ) <<32 ) | - (u32 )reg_read (ctxt,VCPU_REGS_RBX ); - - ctxt->eflags |=X86_EFLAGS_ZF ; - } +if (!(ctxt->rex_bits &REX_W )) +em_cmpxchg8b_16b (ctxt,32 ,64 ); +#ifdef CONFIG_X86_64 + else + em_cmpxchg8b_16b(ctxt, 64, 128);
[Severity: Medium]
Does this unconditionally allow a guest to invoke cmpxchg16b emulation without checking if the guest CPUID actually supports X86_FEATURE_CX16?
If a hypervisor intentionally hides the CX16 CPUID feature to mimic older hardware, it seems a guest could still execute a 128-bit cmpxchg targeting an MMIO address, trapping to KVM and triggering this unconditional emulation path in em_cmpxchg8b().
This appears to violate KVM guidelines requiring guest-visible features to be discoverable and adhere to the guest CPUID configuration, which can break live migration capability negotiation. Could we add a capability check for X86_FEATURE_CX16 before performing the 128-bit emulation?
Quoted 3 lines
arch/x86/kvm/emulate.c +1
-0+#endif return X86EMUL_CONTINUE ; }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2