target/i386: emulate LAM in TCG
cover target/i386: emulate LAM in TCG
This series emulates LAM in TCG for x86_64 system mode and reports it in "-cpu max". linux-user support will follow in a separate series.
Patch 1 keeps the CR3 LAM bits out of the PML5 table address. Patch 2 makes the large-page check in tlb_flush_range_locked() compare under the flush mask, as agreed in [2]. Patch 3 is the LAM emulation, and patch 4 adds a system-mode test.
The design follows the RFC thread [1]:
- On a TLB miss, get_physical_address() masks the address and runs the canonical check and the page walk on the result. The entry is installed under the tagged address, so each tag of a page has its own entry. INVLPG and INVLPGA flush every tagged alias with tlb_flush_page_bits_by_mmuidx(), and a change of CR4.LAM_SUP flushes the TLB.
- Instruction fetches are not masked. If masking changes the address, an entry filled by a data access is not executable, and an entry filled by a fetch is not readable or writable.
- Exceptions do not report the tagged address, which answers Richard's question [3]. On #PF, CR2 receives the masked address (SDM Vol. 3A, section 4.4.3), and #GP reports no address.
- gdbstub, monitor, and qemu_plugin_translate_vaddr() translate a tagged address as a data access does.
Known gaps:
- Data breakpoints on an untagged address do not fire for accesses through a tagged alias.
- With LAM_U48 and 5-level paging, gdbstub, monitor, and qemu_plugin_translate_vaddr() translate an instruction address at or above 2^47 as a data address, not as a fetch does.
[1] https://lists.nongnu.org/archive/html/qemu-devel/2026-10/msg01840.html
[2] https://lists.nongnu.org/archive/html/qemu-devel/2026-11/msg01599.html
[3] https://lists.nongnu.org/archive/html/qemu-devel/2026-10/msg05620.html
Tim (4):
target/i386: mask the PML5 table address taken from CR3 accel/tcg: compare large pages under the flush mask target/i386: emulate LAM in TCG tests/tcg/x86_64: add a LAM system test
accel/tcg/cputlb.c | 4 +- target/i386/cpu.c | 2 +- target/i386/cpu.h | 46 ++ target/i386/helper.c | 8 +- target/i386/tcg/system/excp_helper.c | 33 +- target/i386/tcg/system/misc_helper.c | 24 +- target/i386/tcg/system/svm_helper.c | 3 +- tests/tcg/x86_64/system/boot.S | 14 + tests/tcg/x86_64/system/lam.c | 797 +++++++++++++++++++++++++++ tests/tcg/x86_64/system/meson.build | 15 + 10 files changed, 934 insertions(+), 12 deletions(-) create mode 100644 tests/tcg/x86_64/system/lam.c
Signature · 1 line
2.47.3
PATCH 1/4 target/i386: mask the PML5 table address taken from CR3
The 5-level page walks in mmu_translate() and x86_cpu_translate_for_debug() take the PML5 table address from CR3 with ~0xfff, which keeps CR3 bits 63:52. The 4-level walks already use PG_ADDRESS_MASK. Use PG_ADDRESS_MASK for the 5-level walks as well, so that CR3 control bits above the physical address width, such as the LAM bits 61 and 62, do not become part of the table address.
--- target/i386/helper.c | 2 +- target/i386/tcg/system/excp_helper.c | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-)
target/i386/helper.c +1 -1
@@ -297,7 +297,7 @@ bool x86_cpu_translate_for_debug(CPUState *cs, vaddr addr, }if (la57) { - pml5e_addr = ((env->cr[3 ] & ~0xfff ) + + pml5e_addr = ((env->cr[3 ] &PG_ADDRESS_MASK ) + (((addr >>48 ) &0x1ff ) <<3 )) & a20_mask; pml5e =x86_ldq_phys (cs, pml5e_addr);if (!(pml5e &PG_PRESENT_MASK )) {
target/i386/tcg/system/excp_helper.c +2 -1
@@ -173,7 +173,8 @@ static bool mmu_translate(CPUX86State *env, const TranslateParams *in,/* * Page table level 5 */ - pte_addr = (in->cr3 & ~0xfff ) + (((addr >>48 ) &0x1ff ) <<3 ); + pte_addr = (in->cr3 &PG_ADDRESS_MASK ) + + (((addr >>48 ) &0x1ff ) <<3 );if (!ptw_translate (&pte_trans, pte_addr)) {return false; }
Signature · 1 line
2.47.3
PATCH 2/4 accel/tcg: compare large pages under the flush mask
In reply to Timtlb_flush_range_locked() flushes entries whose address matches the range under @mask, the low @bits of the address. Its large page check, however, compares the range end with the recorded large page in all bits. If the large page was entered under an address that differs from the flushed range above @bits, the check fails, and the large page is not flushed.
Compare under @mask as well. This matters for the x86 LAM emulation in the next patch, which installs TLB entries under tagged addresses and flushes every tagged alias on INVLPG of the untagged address with tlb_flush_page_bits_by_mmuidx().
--- accel/tcg/cputlb.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
accel/tcg/cputlb.c +3 -1
@@ -691,8 +691,10 @@ static void tlb_flush_range_locked(CPUState *cpu, int midx,* Check if we need to flush due to large pages. * Because large_page_mask contains all 1's from the msb, * we only need to test the end of the range. +* Compare under @mask, ignoring bits above @bits. */ -if (((addr + len -1 ) & d->large_page_mask) == d->large_page_addr) { +if (((addr + len -1 ) & d->large_page_mask & mask) == + (d->large_page_addr & mask)) {tlb_debug ("forcing full flush midx %d (" "%016" VADDR_PRIx"/%016" VADDR_PRIx")\n" , midx, d->large_page_addr, d->large_page_mask);
Signature · 1 line
2.47.3
PATCH 3/4 target/i386: emulate LAM in TCG
In reply to TimLinear Address Masking (LAM) lets software keep metadata in bits 62:57 or 62:48 of 64-bit data addresses. CR3.LAM_U57 and CR3.LAM_U48 enable it for user addresses, and CR4.LAM_SUP for supervisor addresses. Implement it in TCG and add it to TCG_7_1_EAX_FEATURES.
get_physical_address() masks the address and runs the canonical check and the page walk on the result. Since masking keeps bit 63, this is the canonical check that the SDM specifies for LAM. CR2 receives the masked address on #PF. The TLB entry is installed under the tagged address, which is the same way target/arm does it for top-byte-ignore. Instruction fetches are not masked, so if masking changes the address, an entry filled by a data access is not executable and one filled by a fetch is not readable or writable.
Since each tag of a page has its own entry, INVLPG and INVLPGA flush every tagged alias with tlb_flush_page_bits_by_mmuidx(), and a change of CR4.LAM_SUP flushes the TLB.
MOV to CR3 and VMRUN accept the CR3 LAM bits if CPUID reports LAM. Masking checks CPUID too, since RSM and gdbstub load CR3 without checking reserved bits.
x86_cpu_translate_for_debug() masks addresses as for a data access, for gdbstub, monitor, and qemu_plugin_translate_vaddr(). With LAM_U48 and 5-level paging, it therefore translates an instruction address at or above 2^47 differently from a fetch.
--- target/i386/cpu.c | 2 +- target/i386/cpu.h | 46 ++++++++++++++++++++++++++++ target/i386/helper.c | 6 +++- target/i386/tcg/system/excp_helper.c | 30 ++++++++++++++++-- target/i386/tcg/system/misc_helper.c | 24 +++++++++++++-- target/i386/tcg/system/svm_helper.c | 3 +- 6 files changed, 102 insertions(+), 9 deletions(-)
target/i386/cpu.c +1 -1
@@ -1006,7 +1006,7 @@ void x86_cpu_vendor_words2str(char *dst, uint32_t vendor1,#define TCG_7_0_EDX_FEATURES (CPUID_7_0_EDX_FSRM | CPUID_7_0_EDX_KERNEL_FEATURES)#define TCG_7_1_EAX_FEATURES (CPUID_7_1_EAX_FZRM | CPUID_7_1_EAX_FSRS | \ - CPUID_7_1_EAX_FSRC | CPUID_7_1_EAX_CMPCCXADD) + CPUID_7_1_EAX_FSRC | CPUID_7_1_EAX_CMPCCXADD | CPUID_7_1_EAX_LAM)#define TCG_7_1_ECX_FEATURES 0#define TCG_7_1_EDX_FEATURES CPUID_7_1_EDX_APXF#define TCG_7_2_EDX_FEATURES 0
target/i386/cpu.h +46 -0
58 lines not on this page · raw message
target/i386/helper.c +5 -1
@@ -204,7 +204,8 @@ void cpu_x86_update_cr4(CPUX86State *env, uint32_t new_cr4)#endif if ((new_cr4 ^ env->cr[4 ]) & (CR4_PGE_MASK |CR4_PAE_MASK |CR4_PSE_MASK | -CR4_SMEP_MASK |CR4_SMAP_MASK |CR4_LA57_MASK )) { +CR4_SMEP_MASK |CR4_SMAP_MASK |CR4_LA57_MASK | +CR4_LAM_SUP_MASK )) {tlb_flush (env_cpu (env)); }
@@ -290,6 +291,9 @@ bool x86_cpu_translate_for_debug(CPUState *cs, vaddr addr,uint64_t pml4e_addr, pml4e;int32_t sext; +/* LAM: translate as for a data access */ + addr =x86_lam_mask_addr (env, addr); +/* test virtual address sign extension */ sext = la57 ? (int64_t )addr >>56 : (int64_t )addr >>47 ;if (sext !=0 && sext !=-1 ) {
target/i386/tcg/system/excp_helper.c +27 -3
@@ -581,15 +581,35 @@ static bool get_physical_address(CPUX86State *env, vaddr addr, }if (likely (env->cr[0 ] &CR0_PG_MASK || use_stage2)) { +int lam_prot_mask =0 ; + in.cr3 = env->cr[3 ]; in.mmu_idx = mmu_idx; in.ptw_idx = use_stage2 ?MMU_NESTED_IDX :MMU_PHYS_IDX ; in.pg_mode =get_pg_mode (env);if (in.pg_mode &PG_MODE_LMA ) { -/* test virtual address sign extension */ +vaddr masked =x86_lam_mask_addr (env, addr);int shift = in.pg_mode &PG_MODE_LA57 ?56 :47 ; -int64_t sext = (int64_t )addr >> shift; +int64_t sext; + +/* +* LAM masks data addresses but not fetches. Both install +* the entry under the unmasked address, so if masking +* changes it, keep only the permissions of this access. +*/ +if (masked != addr) { +if (access_type ==MMU_INST_FETCH ) { + lam_prot_mask =PAGE_READ |PAGE_WRITE ; + }else { + lam_prot_mask =PAGE_EXEC ; + addr = masked; + in.addr = addr; + } + } + +/* test virtual address sign extension */ + sext = (int64_t )addr >> shift;if (sext !=0 && sext !=-1 ) { *err = (TranslateFault ){ .exception_index = EXCP0D_GPF,
@@ -599,7 +619,11 @@ static bool get_physical_address(CPUX86State *env, vaddr addr,return false; } } -return mmu_translate (env, &in, out, err, ra); +if (!mmu_translate (env, &in, out, err, ra)) { +return false; + } + out->prot &= ~lam_prot_mask; +return true; }break ; }
target/i386/tcg/system/misc_helper.c +22 -2
@@ -90,8 +90,7 @@ void HELPER(write_crN)(CPUX86State *env, int reg, target_ulong t0)cpu_x86_update_cr0 (env, t0);break ;case 3 : -if ((env->efer &MSR_EFER_LMA ) && - (t0 & ((~0ULL ) <<env_archcpu (env)->phys_bits))) { +if ((env->efer &MSR_EFER_LMA ) && (t0 &cr3_reserved_bits (env))) {cpu_vmexit (env,SVM_EXIT_ERR ,0 ,GETPC ()); }if (!(env->efer &MSR_EFER_LMA )) {
@@ -504,6 +503,27 @@ void HELPER(rdmsr)(CPUX86State *env)void HELPER (flush_page )(CPUX86State *env,target_ulong addr) { +#ifdef TARGET_X86_64 +if (env->hflags &HF_LMA_MASK ) { +int shift = env->cr[4 ] &CR4_LA57_MASK ?56 :47 ; +int64_t sext = (int64_t )addr >> shift; +unsigned lam_bits; + +/* no LAM for the operand: non-canonical is a no-op */ +if (sext !=0 && sext !=-1 ) { +return ; + } + +/* TLB entries are per tag: flush every tagged alias */ + lam_bits =x86_lam_addr_bits (env, addr); +if (lam_bits <64 ) { +tlb_flush_page_bits_by_mmuidx (env_cpu (env), addr, +MAKE_64BIT_MASK (0 ,NB_MMU_MODES ), + lam_bits); +return ; + } + } +#endif tlb_flush_page (env_cpu (env), addr); }
target/i386/tcg/system/svm_helper.c +1 -2
@@ -315,8 +315,7 @@ void HELPER(vmrun)(CPUX86State *env, int aflag, int next_eip_addend)cpu_vmexit (env,SVM_EXIT_ERR ,0 ,GETPC ()); } new_cr3 = x86_ldq_phys(cs, env->vm_vmcb + offsetof(struct vmcb, save.cr3)); -if ((env->efer &MSR_EFER_LMA ) && - (new_cr3 & ((~0ULL ) << cpu->phys_bits))) { +if ((env->efer &MSR_EFER_LMA ) && (new_cr3 &cr3_reserved_bits (env))) {cpu_vmexit (env,SVM_EXIT_ERR ,0 ,GETPC ()); } new_cr4 = x86_ldq_phys(cs, env->vm_vmcb + offsetof(struct vmcb, save.cr4));
Signature · 1 line
2.47.3
PATCH 4/4 tests/tcg/x86_64: add a LAM system test
In reply to TimTest LAM_U57, LAM_U48, and LAM_SUP at CPL 0: tagged aliases, the canonical checks on the masked address, CR2 on #PF, TLB flushes on CR3 and CR4.LAM_SUP writes and on INVLPG of an untagged 4 KiB or 2 MiB page, helpers and atomics, and instruction fetches through tagged addresses.
The test runs with 4-level paging and with 5-level paging, for which boot.S gains an LA57 option, each with and without LAM in CPUID. Each run has its own build, which fails unless CPUID reports LAM exactly when expected. The identity map uses 4 KiB pages, which leaves the 2 MiB test page as the only large page in the TLB.
--- tests/tcg/x86_64/system/boot.S | 14 + tests/tcg/x86_64/system/lam.c | 797 ++++++++++++++++++++++++++++ tests/tcg/x86_64/system/meson.build | 15 + 3 files changed, 826 insertions(+) create mode 100644 tests/tcg/x86_64/system/lam.c
tests/tcg/x86_64/system/boot.S +14 -0
@@ -91,6 +91,10 @@ _start: /* Enable PAE mode (bit 5). */ mov %cr4, %eax btsl $5, %eax +#ifdef LA57 + /* Enable 5-level paging (bit 12). */ + btsl $12, %eax +#endif mov %eax, %cr4 #define MSR_EFER 0xc0000080 /* extended feature register */
@@ -102,7 +106,11 @@ _start: wrmsr /* Enable paging */ +#ifdef LA57 + mov $.Lpml5, %ecx +#else mov $.Lpml4, %ecx +#endif mov %ecx, %cr3 mov %cr0, %eax
@@ -272,3 +280,9 @@ i = 0 .align 4096 .Lpml4: .quad .Lpdp + 7 /* 0-512 GB */ + +#ifdef LA57 +.align 4096 +.Lpml5: + .quad .Lpml4 + 7 +#endif
tests/tcg/x86_64/system/lam.c new file +797 -0
797 lines not on this page · raw message
tests/tcg/x86_64/system/meson.build +15 -0
@@ -48,6 +48,21 @@ tests += { } } +# The LAM test takes exceptions on the current stack, without CR4.OSFXSR. +lam_cflags = cflags + ['-mno-red-zone', '-mgeneral-regs-only'] +foreach paging, extra: {'': [], '-la57': ['-DLA57']} + foreach suffix, cpu: {'': 'max', '-off': 'max,lam=off'} + tests += { + 'lam.c': { + 'exe_name': 'lam' + paging + suffix, + 'cflags': lam_cflags + extra + + ['-DEXPECT_LAM=' + (suffix == '' ? '1' : '0')], + 'qemu_args': ['-cpu', cpu] + qemu_def_args, + } + } + endforeach +endforeach + if 'qemu-system-x86_64' in emulators tcg_tests += { 'x86_64-softmmu': {
Signature · 1 line
2.47.3