Support x2APIC mode with TCG accelerator
v4 cover Support x2APIC mode with TCG accelerator
Hi everyone,
This series implements x2APIC mode in userspace local APIC and the RDMSR/WRMSR helper to access x2APIC registers in x2APIC mode. Intel iommu and AMD iommu are adjusted to support x2APIC interrupt remapping. With this series, we can now boot Linux kernel into x2APIC mode with TCG accelerator using either Intel or AMD iommu.
Testing to boot my own built Linux 6.3.0-rc2, the kernel successfully boot with enabled x2APIC and can enumerate CPU with APIC ID 257
Using Intel IOMMU
qemu/build/qemu-system-x86_64 \
-smp 2,maxcpus=260 \
-cpu qemu64,x2apic=on \
-machine q35 \
-device intel-iommu,intremap=on,eim=on \
-device qemu64-x86_64-cpu,x2apic=on,core-id=257,socket-id=0,thread-id=0 \
-m 2G \
-kernel $KERNEL_DIR \
-append "nokaslr console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \
-drive file=$IMAGE_DIR,format=raw \
-nographic \
-s
Using AMD IOMMU
qemu/build/qemu-system-x86_64 \
-smp 2,maxcpus=260 \
-cpu qemu64,x2apic=on \
-machine q35 \
-device amd-iommu,intremap=on,xtsup=on \
-device qemu64-x86_64-cpu,x2apic=on,core-id=257,socket-id=0,thread-id=0 \
-m 2G \
-kernel $KERNEL_DIR \
-append "nokaslr console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \
-drive file=$IMAGE_DIR,format=raw \
-nographic \
-s
Testing the emulated userspace APIC with kvm-unit-tests, disable test device with this patch
lib/x86/fwcfg.c +1 -0
@@ -27,6 +27,7 @@ static void read_cfg_override(void)if ((str =getenv ("TEST_DEVICE" ))) no_test_device = !atol (str); + no_test_device = true;if ((str =getenv ("MEMLIMIT" ))) fw_override[FW_CFG_MAX_RAM ] =atol (str) *1024 *1024 ;
~ env QEMU=/home/minh/Desktop/oss/qemu/build/qemu-system-x86_64 ACCEL=tcg \
./run_tests.sh -v -g apic
TESTNAME=apic-split TIMEOUT=90s ACCEL=tcg ./x86/run x86/apic.flat -smp 2 -cpu qemu64,+x2apic,+tsc-deadline -machine kernel_irqchip=split FAIL apic-split (54 tests, 8 unexpected failures, 1 skipped) TESTNAME=ioapic-split TIMEOUT=90s ACCEL=tcg ./x86/run x86/ioapic.flat -smp 1 -cpu qemu64 -machine kernel_irqchip=split PASS ioapic-split (19 tests) TESTNAME=x2apic TIMEOUT=30 ACCEL=tcg ./x86/run x86/apic.flat -smp 2 -cpu qemu64,+x2apic,+tsc-deadline FAIL x2apic (54 tests, 8 unexpected failures, 1 skipped) TESTNAME=xapic TIMEOUT=60 ACCEL=tcg ./x86/run x86/apic.flat -smp 2 -cpu qemu64,-x2apic,+tsc-deadline -machine pit=off FAIL xapic (43 tests, 6 unexpected failures, 2 skipped)
FAIL: apic_disable: *0xfee00030: 50014
FAIL: apic_disable: *0xfee00080: f0
FAIL: apic_disable: *0xfee00030: 50014
FAIL: apic_disable: *0xfee00080: f0
FAIL: apicbase: relocate apic
These errors are because we don't disable MMIO region when switching to x2APIC and don't support relocate MMIO region yet. This is a problem because, MMIO region is the same for all CPUs, in order to support these we need to figure out how to allocate and manage different MMIO regions for each CPUs. This can be an improvement in the future.
FAIL: nmi-after-sti
FAIL: multiple nmi
These errors are in the way we handle CPU_INTERRUPT_NMI in core TCG.
FAIL: TMCCT should stay at zero
This error is related to APIC timer which should be addressed in separate patch.
Version 4 changes,
- Patch 5:
+ Instead of replacing IVHD type 0x10 with type 0x11, export both types for backward compatibility with old guest operating system
+ Flip the xtsup feature check condition in amdvi_int_remap_ga for readability
Version 3 changes,
- Patch 2:
+ Allow APIC ID > 255 only when x2APIC feature is supported on CPU
+ Make physical destination mode IPI which has destination id 0xffffffff a broadcast to xAPIC CPUs
+ Make cluster address 0xf in cluster model of xAPIC logical destination mode a broadcast to all clusters
+ Create new extended_log_dest to store APIC_LDR information in x2APIC instead of extending log_dest for backward compatibility in vmstate
Version 2 changes,
- Add support for APIC ID larger than 255
- Adjust AMD iommu for x2APIC suuport
- Reorganize and split patch 1,2 into patch 1,2,3 in version 2
Thanks,
Quang Minh.
Bui Quang Minh (5):
i386/tcg: implement x2APIC registers MSR access
apic: add support for x2APIC mode
apic, i386/tcg: add x2apic transitions
intel_iommu: allow Extended Interrupt Mode when using userspace APIC
amd_iommu: report x2APIC support to the operating system
hw/i386/acpi-build.c | 127 +++++---- hw/i386/amd_iommu.c | 21 +- hw/i386/amd_iommu.h | 16 +- hw/i386/intel_iommu.c | 11 - hw/i386/x86.c | 8 +- hw/intc/apic.c | 395 +++++++++++++++++++++------ hw/intc/apic_common.c | 16 +- hw/intc/trace-events | 4 +- include/hw/i386/apic.h | 6 +- include/hw/i386/apic_internal.h | 7 +- target/i386/cpu-sysemu.c | 18 +- target/i386/cpu.c | 5 +- target/i386/cpu.h | 9 + target/i386/tcg/sysemu/misc_helper.c | 31 +++ 14 files changed, 500 insertions(+), 174 deletions(-)
Signature · 1 line
2.25.1
PATCH v4 1/5 i386/tcg: implement x2APIC registers MSR access
This commit refactors apic_mem_read/write to support both MMIO access in xAPIC and MSR access in x2APIC.
--- hw/intc/apic.c | 79 ++++++++++++++++++---------- hw/intc/trace-events | 4 +- include/hw/i386/apic.h | 3 ++ target/i386/cpu.h | 3 ++ target/i386/tcg/sysemu/misc_helper.c | 27 ++++++++++ 5 files changed, 86 insertions(+), 30 deletions(-)
hw/intc/apic.c +51 -28
@@ -288,6 +288,13 @@ void apic_deliver_irq(uint8_t dest, uint8_t dest_mode, uint8_t delivery_mode,apic_bus_deliver (deliver_bitmask, delivery_mode, vector_num, trigger_mode); } +bool is_x2apic_mode (DeviceState *dev) +{ +APICCommonState *s =APIC (dev); + +return s->apicbase &MSR_IA32_APICBASE_EXTD ; +} +static void apic_set_base (APICCommonState *s,uint64_t val) { s->apicbase = (val & 0xfffff000) |
@@ -636,16 +643,11 @@ static void apic_timer(void *opaque)apic_timer_update (s, s->next_time); } -static uint64_t apic_mem_read (void *opaque,hwaddr addr,unsigned size) +uint64_t apic_register_read (int index) {DeviceState *dev;APICCommonState *s; -uint32_t val; -int index; - -if (size <4 ) { -return 0 ; - } +uint64_t val; dev =cpu_get_current_apic ();if (!dev) {
@@ -653,7 +655,6 @@ static uint64_t apic_mem_read(void *opaque, hwaddr addr, unsigned size) } s =APIC (dev); - index = (addr >>4 ) &0xff ;switch (index) {case 0x02 :/* id */ val = s->id <<24 ;
@@ -720,7 +721,23 @@ static uint64_t apic_mem_read(void *opaque, hwaddr addr, unsigned size) val =0 ;break ; } -trace_apic_mem_readl (addr, val); + +trace_apic_register_read (index, val); +return val; +} + +static uint64_t apic_mem_read (void *opaque,hwaddr addr,unsigned size) +{ +uint32_t val; +int index; + +if (size <4 ) { +return 0 ; + } + + index = (addr >>4 ) &0xff ; + val = (uint32_t )apic_register_read (index); +return val; }
@@ -737,27 +754,10 @@ static void apic_send_msi(MSIMessage *msi)apic_deliver_irq (dest, dest_mode, delivery, vector, trigger_mode); } -static void apic_mem_write (void *opaque,hwaddr addr,uint64_t val, -unsigned size) +void apic_register_write (int index,uint64_t val) {DeviceState *dev;APICCommonState *s; -int index = (addr >>4 ) &0xff ; - -if (size <4 ) { -return ; - } - -if (addr >0xfff || !index) { -/* MSI and MMIO APIC are at the same memory location, -* but actually not on the global bus: MSI is on PCI bus -* APIC is connected directly to the CPU. -* Mapping them on the global bus happens to work because -* MSI registers are reserved in APIC MMIO and vice versa. */ -MSIMessage msi = { .address = addr, .data = val }; -apic_send_msi (&msi); -return ; - } dev =cpu_get_current_apic ();if (!dev) {
@@ -765,7 +765,7 @@ static void apic_mem_write(void *opaque, hwaddr addr, uint64_t val, } s =APIC (dev); -trace_apic_mem_writel (addr, val); +trace_apic_register_write (index, val);switch (index) {case 0x02 :
@@ -843,6 +843,29 @@ static void apic_mem_write(void *opaque, hwaddr addr, uint64_t val, } } +static void apic_mem_write (void *opaque,hwaddr addr,uint64_t val, +unsigned size) +{ +int index = (addr >>4 ) &0xff ; + +if (size <4 ) { +return ; + } + +if (addr >0xfff || !index) { +/* MSI and MMIO APIC are at the same memory location, +* but actually not on the global bus: MSI is on PCI bus +* APIC is connected directly to the CPU. +* Mapping them on the global bus happens to work because +* MSI registers are reserved in APIC MMIO and vice versa. */ +MSIMessage msi = { .address = addr, .data = val }; +apic_send_msi (&msi); +return ; + } + +apic_register_write (index, val); +} +static void apic_pre_save (APICCommonState *s) {apic_sync_vapic (s,SYNC_FROM_VAPIC );
hw/intc/trace-events +2 -2
@@ -14,8 +14,8 @@ cpu_get_apic_base(uint64_t val) "0x%016"PRIx64 # apic.c apic_local_deliver(int vector, uint32_t lvt) "vector %d delivery mode %d" apic_deliver_irq(uint8_t dest, uint8_t dest_mode, uint8_t delivery_mode, uint8_t vector_num, uint8_t trigger_mode) "dest %d dest_mode %d delivery_mode %d vector %d trigger_mode %d" -apic_mem_readl(uint64_t addr, uint32_t val) "0x%"PRIx64" = 0x%08x" -apic_mem_writel(uint64_t addr, uint32_t val) "0x%"PRIx64" = 0x%08x" +apic_register_read(uint8_t reg, uint64_t val) "register 0x%02x = 0x%"PRIx64 +apic_register_write(uint8_t reg, uint64_t val) "register 0x%02x = 0x%"PRIx64 # ioapic.c ioapic_set_remote_irr(int n) "set remote irr for pin %d"
include/hw/i386/apic.h +3 -0
@@ -18,6 +18,9 @@ void apic_sipi(DeviceState *s);void apic_poll_irq (DeviceState *d);void apic_designate_bsp (DeviceState *d,bool bsp);int apic_get_highest_priority_irr (DeviceState *dev); +uint64_t apic_register_read (int index); +void apic_register_write (int index,uint64_t val); +bool is_x2apic_mode (DeviceState *d);/* pc.c */ DeviceState *cpu_get_current_apic (void );
target/i386/cpu.h +3 -0
@@ -545,6 +545,9 @@ typedef enum X86Seg {#define MSR_IA32_VMX_TRUE_ENTRY_CTLS 0x00000490#define MSR_IA32_VMX_VMFUNC 0x00000491 +#define MSR_APIC_START 0x00000800 +#define MSR_APIC_END 0x000008ff +#define XSTATE_FP_BIT 0#define XSTATE_SSE_BIT 1#define XSTATE_YMM_BIT 2
target/i386/tcg/sysemu/misc_helper.c +27 -0
@@ -25,6 +25,7 @@#include "exec/address-spaces.h" #include "exec/exec-all.h" #include "tcg/helper-tcg.h" +#include "hw/i386/apic.h" void helper_outb (CPUX86State *env,uint32_t port,uint32_t data) {
@@ -289,6 +290,19 @@ void helper_wrmsr(CPUX86State *env) env->msr_bndcfgs = val;cpu_sync_bndcs_hflags (env);break ; + case MSR_APIC_START ... MSR_APIC_END: { +int index = (uint32_t )env->regs[R_ECX ] -MSR_APIC_START ; + +if (!is_x2apic_mode (env_archcpu (env)->apic_state)) { + goto error; + } + +qemu_mutex_lock_iothread (); +apic_register_write (index, val); +qemu_mutex_unlock_iothread (); + +break ; + }default :if ((uint32_t )env->regs[R_ECX ] >=MSR_MC0_CTL && (uint32_t)env->regs[R_ECX] < MSR_MC0_CTL +
@@ -455,6 +469,19 @@ void helper_rdmsr(CPUX86State *env) val = (cs->nr_threads * cs->nr_cores) | (cs->nr_cores <<16 );break ; } + case MSR_APIC_START ... MSR_APIC_END: { +int index = (uint32_t )env->regs[R_ECX ] -MSR_APIC_START ; + +if (!is_x2apic_mode (env_archcpu (env)->apic_state)) { +raise_exception_ra (env,EXCP0D_GPF ,GETPC ()); + } + +qemu_mutex_lock_iothread (); + val =apic_register_read (index); +qemu_mutex_unlock_iothread (); + +break ; + }default :if ((uint32_t )env->regs[R_ECX ] >=MSR_MC0_CTL && (uint32_t)env->regs[R_ECX] < MSR_MC0_CTL +
Signature · 1 line
2.25.1
PATCH v4 2/5 apic: add support for x2APIC mode
In reply to Bui Quang MinhThis commit extends the APIC ID to 32-bit long and remove the 255 max APIC ID limit in userspace APIC. The array that manages local APICs is now dynamically allocated based on the max APIC ID of created x86 machine. Also, new x2APIC IPI destination determination scheme, self IPI and x2APIC mode register access are supported.
--- hw/i386/x86.c | 8 +- hw/intc/apic.c | 266 ++++++++++++++++++++++++-------- hw/intc/apic_common.c | 9 ++ include/hw/i386/apic.h | 3 +- include/hw/i386/apic_internal.h | 7 +- target/i386/cpu-sysemu.c | 8 +- 6 files changed, 231 insertions(+), 70 deletions(-)
hw/i386/x86.c +6 -2
@@ -132,11 +132,11 @@ void x86_cpus_init(X86MachineState *x86ms, int default_cpu_version)* Can we support APIC ID 255 or higher? * * Under Xen: yes. -* With userspace emulated lapic: no +* With userspace emulated lapic: checked later in apic_common_set_id. * With KVM's in-kernel lapic: only if X2APIC API is enabled. */ if (x86ms->apic_id_limit >255 && !xen_enabled () && - (!kvm_irqchip_in_kernel () || !kvm_enable_x2apic ())) { +kvm_irqchip_in_kernel () && !kvm_enable_x2apic ()) {error_report ("current -smp configuration requires kernel " "irqchip and X2APIC API support." );exit (EXIT_FAILURE );
@@ -146,6 +146,10 @@ void x86_cpus_init(X86MachineState *x86ms, int default_cpu_version)kvm_set_max_apic_id (x86ms->apic_id_limit); } +if (!kvm_irqchip_in_kernel ()) { +apic_set_max_apic_id (x86ms->apic_id_limit); + } + possible_cpus = mc->possible_cpu_arch_ids (ms);for (i =0 ; i < ms->smp.cpus; i++) {x86_cpu_new (x86ms, possible_cpus->cpus[i].arch_id, &error_fatal);
hw/intc/apic.c +203 -63
430 lines not on this page · raw message
hw/intc/apic_common.c +9 -0
@@ -284,6 +284,10 @@ static void apic_common_realize(DeviceState *dev, Error **errp) }vmstate_register_with_alias_id (NULL , instance_id, &vmstate_apic_common, s,-1 ,0 ,NULL ); + +/* APIC LDR in x2APIC mode */ + s->extended_log_dest = ((s->initial_apic_id &0xffff0 ) <<16 ) | + (1 << (s->initial_apic_id &0xf )); }static void apic_common_unrealize (DeviceState *dev)
@@ -424,6 +428,11 @@ static void apic_common_set_id(Object *obj, Visitor *v, const char *name,return ; } +if (value >=255 && !cpu_has_x2apic_feature (&s->cpu->env)) { +error_setg (errp,"APIC ID %d requires x2APIC feature in CPU" , value); +return ; + } + s->initial_apic_id = value; s->id = (uint8_t )value; }
include/hw/i386/apic.h +1 -2
@@ -3,8 +3,7 @@/* apic.c */ -void apic_deliver_irq (uint8_t dest,uint8_t dest_mode,uint8_t delivery_mode, -uint8_t vector_num,uint8_t trigger_mode); +void apic_set_max_apic_id (uint32_t max_apic_id);int apic_accept_pic_intr (DeviceState *s);void apic_deliver_pic_intr (DeviceState *s,int level);void apic_deliver_nmi (DeviceState *d);
include/hw/i386/apic_internal.h +5 -2
@@ -46,8 +46,10 @@#define APIC_DM_EXTINT 7/* APIC destination mode */ -#define APIC_DESTMODE_FLAT 0xf -#define APIC_DESTMODE_CLUSTER 1 +#define APIC_DESTMODE_PHYSICAL 0 +#define APIC_DESTMODE_LOGICAL 1 +#define APIC_DESTMODE_LOGICAL_FLAT 0xf +#define APIC_DESTMODE_LOGICAL_CLUSTER 0#define APIC_TRIGGER_EDGE 0#define APIC_TRIGGER_LEVEL 1
@@ -187,6 +189,7 @@ struct APICCommonState {DeviceState *vapic;hwaddr vapic_paddr;/* note: persistence via kvmvapic */ bool legacy_instance_id; +uint32_t extended_log_dest; }; typedef struct VAPICState {
target/i386/cpu-sysemu.c +7 -1
@@ -281,11 +281,17 @@ void x86_cpu_apic_create(X86CPU *cpu, Error **errp) OBJECT(cpu->apic_state));object_unref (OBJECT (cpu->apic_state)); -qdev_prop_set_uint32 (cpu->apic_state,"id" , cpu->apic_id);/* TODO: convert to link<> */ apic =APIC_COMMON (cpu->apic_state); apic->cpu = cpu; apic->apicbase =APIC_DEFAULT_ADDRESS |MSR_IA32_APICBASE_ENABLE ; + +/* +* apic_common_set_id needs to check if the CPU has x2APIC +* feature in case APIC ID >= 255, so we need to set apic->cpu +* before setting APIC ID +*/ +qdev_prop_set_uint32 (cpu->apic_state,"id" , cpu->apic_id); }void x86_cpu_apic_realize (X86CPU *cpu,Error **errp)
Signature · 1 line
2.25.1
PATCH v4 3/5 apic, i386/tcg: add x2apic transitions
In reply to Bui Quang MinhThis commit adds support for x2APIC transitions when writing to MSR_IA32_APICBASE register and finally adds CPUID_EXT_X2APIC to TCG_EXT_FEATURES.
--- hw/intc/apic.c | 50 ++++++++++++++++++++++++++++ hw/intc/apic_common.c | 7 ++-- target/i386/cpu-sysemu.c | 10 ++++++ target/i386/cpu.c | 5 +-- target/i386/cpu.h | 6 ++++ target/i386/tcg/sysemu/misc_helper.c | 4 +++ 6 files changed, 75 insertions(+), 7 deletions(-)
hw/intc/apic.c +50 -0
@@ -309,8 +309,41 @@ bool is_x2apic_mode(DeviceState *dev)return s->apicbase &MSR_IA32_APICBASE_EXTD ; } +static void apic_set_base_check (APICCommonState *s,uint64_t val) +{ +/* Enable x2apic when x2apic is not supported by CPU */ +if (!cpu_has_x2apic_feature (&s->cpu->env) && + val &MSR_IA32_APICBASE_EXTD ) +raise_exception_ra (&s->cpu->env,EXCP0D_GPF ,GETPC ()); + +/* +* Transition into invalid state +* (s->apicbase & MSR_IA32_APICBASE_ENABLE == 0) && +* (s->apicbase & MSR_IA32_APICBASE_EXTD) == 1 +*/ +if (!(val &MSR_IA32_APICBASE_ENABLE ) && + (val &MSR_IA32_APICBASE_EXTD )) +raise_exception_ra (&s->cpu->env,EXCP0D_GPF ,GETPC ()); + +/* Invalid transition from disabled mode to x2APIC */ +if (!(s->apicbase &MSR_IA32_APICBASE_ENABLE ) && + !(s->apicbase &MSR_IA32_APICBASE_EXTD ) && + (val &MSR_IA32_APICBASE_ENABLE ) && + (val &MSR_IA32_APICBASE_EXTD )) +raise_exception_ra (&s->cpu->env,EXCP0D_GPF ,GETPC ()); + +/* Invalid transition from x2APIC to xAPIC */ +if ((s->apicbase &MSR_IA32_APICBASE_ENABLE ) && + (s->apicbase &MSR_IA32_APICBASE_EXTD ) && + (val &MSR_IA32_APICBASE_ENABLE ) && + !(val &MSR_IA32_APICBASE_EXTD )) +raise_exception_ra (&s->cpu->env,EXCP0D_GPF ,GETPC ()); +} +static void apic_set_base (APICCommonState *s,uint64_t val) { +apic_set_base_check (s, val); + s->apicbase = (val &0xfffff000 ) | (s->apicbase & (MSR_IA32_APICBASE_BSP |MSR_IA32_APICBASE_ENABLE ));/* if disabled, cannot be enabled again */
@@ -319,6 +352,23 @@ static void apic_set_base(APICCommonState *s, uint64_t val)cpu_clear_apic_feature (&s->cpu->env); s->spurious_vec &= ~APIC_SV_ENABLE ; } + +/* Transition from disabled mode to xAPIC */ +if (!(s->apicbase &MSR_IA32_APICBASE_ENABLE ) && + (val &MSR_IA32_APICBASE_ENABLE )) { + s->apicbase |=MSR_IA32_APICBASE_ENABLE ; +cpu_set_apic_feature (&s->cpu->env); + } + +/* Transition from xAPIC to x2APIC */ +if (cpu_has_x2apic_feature (&s->cpu->env) && + !(s->apicbase &MSR_IA32_APICBASE_EXTD ) && + (val &MSR_IA32_APICBASE_EXTD )) { + s->apicbase |=MSR_IA32_APICBASE_EXTD ; + + s->log_dest = ((s->initial_apic_id &0xffff0 ) <<16 ) | + (1 << (s->initial_apic_id &0xf )); + } }static void apic_set_tpr (APICCommonState *s,uint8_t val)
hw/intc/apic_common.c +2 -5
@@ -43,11 +43,8 @@ void cpu_set_apic_base(DeviceState *dev, uint64_t val)if (dev) {APICCommonState *s =APIC_COMMON (dev);APICCommonClass *info =APIC_COMMON_GET_CLASS (s); -/* switching to x2APIC, reset possibly modified xAPIC ID */ -if (!(s->apicbase &MSR_IA32_APICBASE_EXTD ) && - (val &MSR_IA32_APICBASE_EXTD )) { - s->id = s->initial_apic_id; - } +/* Reset possibly modified xAPIC ID */ + s->id = s->initial_apic_id; info->set_base (s, val); } }
target/i386/cpu-sysemu.c +10 -0
@@ -235,6 +235,16 @@ void cpu_clear_apic_feature(CPUX86State *env) env->features[FEAT_1_EDX ] &= ~CPUID_APIC ; } +void cpu_set_apic_feature (CPUX86State *env) +{ + env->features[FEAT_1_EDX ] |=CPUID_APIC ; +} + +bool cpu_has_x2apic_feature (CPUX86State *env) +{ +return env->features[FEAT_1_ECX ] &CPUID_EXT_X2APIC ; +} +bool cpu_is_bsp (X86CPU *cpu) {return cpu_get_apic_base (cpu->apic_state) &MSR_IA32_APICBASE_BSP ;
target/i386/cpu.c +3 -2
@@ -629,12 +629,13 @@ void x86_cpu_vendor_words2str(char *dst, uint32_t vendor1, CPUID_EXT_XSAVE | /* CPUID_EXT_OSXSAVE is dynamic */ \ CPUID_EXT_MOVBE | CPUID_EXT_AES | CPUID_EXT_HYPERVISOR | \ CPUID_EXT_RDRAND | CPUID_EXT_AVX | CPUID_EXT_F16C | \ - CPUID_EXT_FMA) + CPUID_EXT_FMA | CPUID_EXT_X2APIC)/* missing: CPUID_EXT_DTES64, CPUID_EXT_DSCPL, CPUID_EXT_VMX, CPUID_EXT_SMX, CPUID_EXT_EST, CPUID_EXT_TM2, CPUID_EXT_CID, CPUID_EXT_XTPR, CPUID_EXT_PDCM, CPUID_EXT_PCID, CPUID_EXT_DCA, -CPUID_EXT_X2APIC, CPUID_EXT_TSC_DEADLINE_TIMER */ +CPUID_EXT_TSC_DEADLINE_TIMER +*/ #ifdef TARGET_X86_64 #define TCG_EXT2_X86_64_FEATURES (CPUID_EXT2_SYSCALL | CPUID_EXT2_LM)
target/i386/cpu.h +6 -0
@@ -379,6 +379,10 @@ typedef enum X86Seg {#define MSR_IA32_APICBASE_ENABLE (1<<11)#define MSR_IA32_APICBASE_EXTD (1 << 10)#define MSR_IA32_APICBASE_BASE (0xfffffU<<12) +#define MSR_IA32_APICBASE_RESERVED \ + (~(uint64_t)(MSR_IA32_APICBASE_BSP | MSR_IA32_APICBASE_ENABLE \ + | MSR_IA32_APICBASE_EXTD | MSR_IA32_APICBASE_BASE)) +#define MSR_IA32_FEATURE_CONTROL 0x0000003a#define MSR_TSC_ADJUST 0x0000003b#define MSR_IA32_SPEC_CTRL 0x48
@@ -2190,8 +2194,10 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count, uint32_t *eax, uint32_t *ebx, uint32_t *ecx, uint32_t *edx);void cpu_clear_apic_feature (CPUX86State *env); +void cpu_set_apic_feature (CPUX86State *env);void host_cpuid (uint32_t function,uint32_t count,uint32_t *eax,uint32_t *ebx,uint32_t *ecx,uint32_t *edx); +bool cpu_has_x2apic_feature (CPUX86State *env);/* helper.c */ void x86_cpu_set_a20 (X86CPU *cpu,int a20_state);
target/i386/tcg/sysemu/misc_helper.c +4 -0
@@ -159,6 +159,10 @@ void helper_wrmsr(CPUX86State *env) env->sysenter_eip = val;break ;case MSR_IA32_APICBASE : +if (val &MSR_IA32_APICBASE_RESERVED ) { + goto error; + } +cpu_set_apic_base (env_archcpu (env)->apic_state, val);break ;case MSR_EFER :
Signature · 1 line
2.25.1
PATCH v4 4/5 intel_iommu: allow Extended Interrupt Mode when using userspace APIC
In reply to Bui Quang MinhAs userspace APIC now supports x2APIC, intel interrupt remapping hardware can be set to EIM mode when userspace local APIC is used.
--- hw/i386/intel_iommu.c | 11 ----------- 1 file changed, 11 deletions(-)
hw/i386/intel_iommu.c +0 -11
@@ -4046,17 +4046,6 @@ static bool vtd_decide_config(IntelIOMMUState *s, Error **errp) &&x86_iommu_ir_supported (x86_iommu) ?ON_OFF_AUTO_ON :ON_OFF_AUTO_OFF ; } -if (s->intr_eim ==ON_OFF_AUTO_ON && !s->buggy_eim) { -if (!kvm_irqchip_is_split ()) { -error_setg (errp,"eim=on requires accel=kvm,kernel-irqchip=split" ); -return false; - } -if (!kvm_enable_x2apic ()) { -error_setg (errp,"eim=on requires support on the KVM side" -"(X2APIC_API, first shipped in v4.7)" ); -return false; - } - }/* Currently only address widths supported are 39 and 48 bits */ if ((s->aw_bits != VTD_HOST_AW_39BIT) &&
Signature · 1 line
2.25.1
PATCH v4 5/5 amd_iommu: report x2APIC support to the operating system
In reply to Bui Quang MinhThis commit adds XTSup configuration to let user choose to whether enable this feature or not. When XTSup is enabled, additional bytes in IRTE with enabled guest virtual VAPIC are used to support 32-bit destination id.
Additionally, this commit exports IVHD type 0x11 besides the old IVHD type 0x10 in ACPI table. IVHD type 0x10 does not report full set of IOMMU features only the legacy ones, so operating system (e.g. Linux) may only detects x2APIC support if IVHD type 0x11 is available. The IVHD type 0x10 is kept so that old operating system that only parses type 0x10 can detect the IOMMU device.
--- hw/i386/acpi-build.c | 127 ++++++++++++++++++++++++++----------------- hw/i386/amd_iommu.c | 21 ++++++- hw/i386/amd_iommu.h | 16 ++++-- 3 files changed, 108 insertions(+), 56 deletions(-)
hw/i386/acpi-build.c +78 -49
@@ -2339,30 +2339,23 @@ static void build_amd_iommu(GArray *table_data, BIOSLinker *linker, const char *oem_id, const char *oem_table_id) { -int ivhd_table_len =24 ;AMDVIState *s =AMD_IOMMU_DEVICE (x86_iommu_get_default ());GArray *ivhd_blob =g_array_new (false, true,1 );AcpiTable table = { .sig ="IVRS" , .rev =1 , .oem_id = oem_id, .oem_table_id = oem_table_id }; +uint64_t feature_report;acpi_table_begin (&table, table_data);/* IVinfo - IO virtualization information common to all * IOMMU units in a system */ -build_append_int_noprefix (table_data,40UL <<8 /* PASize */ ,4 ); +build_append_int_noprefix (table_data, + (1UL <<0 ) |/* EFRSup */ + (40UL <<8 ),/* PASize */ +4 );/* reserved */ build_append_int_noprefix (table_data,0 ,8 ); -/* IVHD definition - type 10h */ -build_append_int_noprefix (table_data,0x10 ,1 ); -/* virtualization flags */ -build_append_int_noprefix (table_data, - (1UL <<0 ) |/* HtTunEn */ - (1UL <<4 ) |/* iotblSup */ - (1UL <<6 ) |/* PrefSup */ - (1UL <<7 ),/* PPRSup */ -1 ); -/* * A PCI bus walk, for each PCI host bridge, is necessary to create a * complete set of IVHD entries. Do this into a separate blob so that we
@@ -2382,56 +2375,92 @@ build_amd_iommu(GArray *table_data, BIOSLinker *linker, const char *oem_id,build_append_int_noprefix (ivhd_blob,0x0000001 ,4 ); } - ivhd_table_len += ivhd_blob->len; -/* * When interrupt remapping is supported, we add a special IVHD device -* for type IO-APIC. -*/ -if (x86_iommu_ir_supported (x86_iommu_get_default ())) { - ivhd_table_len +=8 ; - } - -/* IVHD length */ -build_append_int_noprefix (table_data, ivhd_table_len,2 ); -/* DeviceID */ -build_append_int_noprefix (table_data, -object_property_get_int (OBJECT (&s->pci),"addr" , - &error_abort),2 ); -/* Capability offset */ -build_append_int_noprefix (table_data, s->pci.capab_offset,2 ); -/* IOMMU base address */ -build_append_int_noprefix (table_data, s->mmio.addr,8 ); -/* PCI Segment Group */ -build_append_int_noprefix (table_data,0 ,2 ); -/* IOMMU info */ -build_append_int_noprefix (table_data,0 ,2 ); -/* IOMMU Feature Reporting */ -build_append_int_noprefix (table_data, - (48UL <<30 ) |/* HATS */ - (48UL <<28 ) |/* GATS */ - (1UL <<2 ) |/* GTSup */ - (1UL <<6 ),/* GASup */ -4 ); - -/* IVHD entries as found above */ -g_array_append_vals (table_data, ivhd_blob->data, ivhd_blob->len); -g_array_free (ivhd_blob, TRUE); - -/* -* Add a special IVHD device type. +* for type IO-APIC * Refer to spec - Table 95: IVHD device entry type codes * * Linux IOMMU driver checks for the special IVHD device (type IO-APIC). * See Linux kernel commit 'c2ff5cf5294bcbd7fa50f7d860e90a66db7e5059' */ if (x86_iommu_ir_supported (x86_iommu_get_default ())) { -build_append_int_noprefix (table_data, +build_append_int_noprefix (ivhd_blob, (0x1ull <<56 ) |/* type IOAPIC */ (IOAPIC_SB_DEVID <<40 ) |/* IOAPIC devid */ 0x48 ,/* special device */ 8 ); } + +/* IVHD definition - type 10h */ +build_append_int_noprefix (table_data,0x10 ,1 ); +/* virtualization flags */ +build_append_int_noprefix (table_data, + (1UL <<0 ) |/* HtTunEn */ + (1UL <<4 ) |/* iotblSup */ + (1UL <<6 ) |/* PrefSup */ + (1UL <<7 ),/* PPRSup */ +1 ); + +/* IVHD length */ +build_append_int_noprefix (table_data, ivhd_blob->len +24 ,2 ); +/* DeviceID */ +build_append_int_noprefix (table_data, +object_property_get_int (OBJECT (&s->pci),"addr" , + &error_abort),2 ); +/* Capability offset */ +build_append_int_noprefix (table_data, s->pci.capab_offset,2 ); +/* IOMMU base address */ +build_append_int_noprefix (table_data, s->mmio.addr,8 ); +/* PCI Segment Group */ +build_append_int_noprefix (table_data,0 ,2 ); +/* IOMMU info */ +build_append_int_noprefix (table_data,0 ,2 ); +/* IOMMU Feature Reporting */ + feature_report = (48UL <<30 ) |/* HATS */ + (48UL <<28 ) |/* GATS */ + (1UL <<2 ) |/* GTSup */ + (1UL <<6 );/* GASup */ +if (s->xtsup) { + feature_report |= (1UL <<0 );/* XTSup */ + } +build_append_int_noprefix (table_data, feature_report,4 ); + +/* IVHD entries as found above */ +g_array_append_vals (table_data, ivhd_blob->data, ivhd_blob->len); + +/* IVHD definition - type 11h */ +build_append_int_noprefix (table_data,0x11 ,1 ); +/* virtualization flags */ +build_append_int_noprefix (table_data, + (1UL <<0 ) |/* HtTunEn */ + (1UL <<4 ),/* iotblSup */ +1 ); + +/* IVHD length */ +build_append_int_noprefix (table_data, ivhd_blob->len +40 ,2 ); +/* DeviceID */ +build_append_int_noprefix (table_data, +object_property_get_int (OBJECT (&s->pci),"addr" , + &error_abort),2 ); +/* Capability offset */ +build_append_int_noprefix (table_data, s->pci.capab_offset,2 ); +/* IOMMU base address */ +build_append_int_noprefix (table_data, s->mmio.addr,8 ); +/* PCI Segment Group */ +build_append_int_noprefix (table_data,0 ,2 ); +/* IOMMU info */ +build_append_int_noprefix (table_data,0 ,2 ); +/* IOMMU Attributes */ +build_append_int_noprefix (table_data,0 ,4 ); +/* EFR Register Image */ +build_append_int_noprefix (table_data, s->efr_reg,8 ); +/* EFR Register Image 2 */ +build_append_int_noprefix (table_data,0 ,8 ); + +/* IVHD entries as found above */ +g_array_append_vals (table_data, ivhd_blob->data, ivhd_blob->len); + +g_array_free (ivhd_blob, TRUE);acpi_table_end (linker, &table); }
hw/i386/amd_iommu.c +19 -2
@@ -31,6 +31,7 @@#include "hw/i386/apic_internal.h" #include "trace.h" #include "hw/i386/apic-msidef.h" +#include "hw/qdev-properties.h" /* used AMD-Vi MMIO registers */ const char *amdvi_mmio_low[] = {
@@ -1155,7 +1156,12 @@ static int amdvi_int_remap_ga(AMDVIState *iommu, irq->vector = irte.hi.fields.vector; irq->dest_mode = irte.lo.fields_remap.dm; irq->redir_hint = irte.lo.fields_remap.rq_eoi; - irq->dest = irte.lo.fields_remap.destination; +if (iommu->xtsup) { + irq->dest = irte.lo.fields_remap.destination | + (irte.hi.fields.destination_hi <<24 ); + }else { + irq->dest = irte.lo.fields_remap.destination &0xff ; + }return 0 ; }
@@ -1503,10 +1509,15 @@ static void amdvi_init(AMDVIState *s) s->enabled = false; s->ats_enabled = false; s->cmdbuf_enabled = false; + s->efr_reg =AMDVI_DEFAULT_EXT_FEATURES ; + +if (s->xtsup) { + s->efr_reg |=AMDVI_FEATURE_XT ; + }/* reset MMIO */ memset (s->mmior,0 ,AMDVI_MMIO_SIZE ); -amdvi_set_quad (s,AMDVI_MMIO_EXT_FEATURES ,AMDVI_EXT_FEATURES , +amdvi_set_quad (s,AMDVI_MMIO_EXT_FEATURES , s->efr_reg,0xffffffffffffffef ,0 );amdvi_set_quad (s,AMDVI_MMIO_STATUS ,0 ,0x98 ,0x67 ); }
@@ -1591,6 +1602,11 @@ static void amdvi_sysbus_realize(DeviceState *dev, Error **errp)amdvi_init (s); } +static Property amdvi_properties[] = { +DEFINE_PROP_BOOL ("xtsup" , AMDVIState, xtsup, false), +DEFINE_PROP_END_OF_LIST (), +}; +static const VMStateDescription vmstate_amdvi_sysbus = { .name ="amd-iommu" , .unmigratable =1
@@ -1617,6 +1633,7 @@ static void amdvi_sysbus_class_init(ObjectClass *klass, void *data) dc->user_creatable = true;set_bit (DEVICE_CATEGORY_MISC , dc->categories); dc->desc ="AMD IOMMU (AMD-Vi) DMA Remapping device" ; +device_class_set_props (dc, amdvi_properties); }static const TypeInfo amdvi_sysbus = {
hw/i386/amd_iommu.h +11 -5
@@ -154,6 +154,7 @@#define AMDVI_FEATURE_PREFETCH (1ULL << 0)/* page prefetch */ #define AMDVI_FEATURE_PPR (1ULL << 1)/* PPR Support */ +#define AMDVI_FEATURE_XT (1ULL << 2)/* x2APIC Support */ #define AMDVI_FEATURE_GT (1ULL << 4)/* Guest Translation */ #define AMDVI_FEATURE_IA (1ULL << 6)/* inval all support */ #define AMDVI_FEATURE_GA (1ULL << 7)/* guest VAPIC support */
@@ -173,8 +174,9 @@#define AMDVI_IOTLB_MAX_SIZE 1024#define AMDVI_DEVID_SHIFT 36 -/* extended feature support */ -#define AMDVI_EXT_FEATURES (AMDVI_FEATURE_PREFETCH | AMDVI_FEATURE_PPR | \ +/* default extended feature */ +#define AMDVI_DEFAULT_EXT_FEATURES \ + (AMDVI_FEATURE_PREFETCH | AMDVI_FEATURE_PPR | \ AMDVI_FEATURE_IA | AMDVI_FEATURE_GT | AMDVI_FEATURE_HE | \ AMDVI_GATS_MODE | AMDVI_HATS_MODE | AMDVI_FEATURE_GA)
@@ -278,8 +280,8 @@ union irte_ga_lo { dm:1, /* ------ */ guest_mode:1, - destination:8, - rsvd_1:48; + destination:24, + rsvd_1:32; } fields_remap; };
@@ -287,7 +289,8 @@ union irte_ga_hi {uint64_t val;struct {uint64_t vector:8 , - rsvd_2:56 ; + rsvd_2:48 , + destination_hi:8 ; } fields; };
@@ -366,6 +369,9 @@ struct AMDVIState {/* Interrupt remapping */ bool ga_enabled; +bool xtsup; + + uint64_t efr_reg; /* extended feature register */ };#endif
Signature · 1 line
2.25.1
Re: [PATCH v4 5/5] amd_iommu: report x2APIC support to the operating system
Quoted 297 lines · Bui Quang Minh
This commit adds XTSup configuration to let user choose to whether enable this feature or not. When XTSup is enabled, additional bytes in IRTE with enabled guest virtual VAPIC are used to support 32-bit destination id.
Additionally, this commit exports IVHD type
0x11besides the old IVHD type0x10in ACPI table. IVHD type0x10does not report full set of IOMMU features only the legacy ones, so operating system (e.g. Linux) may only detects x2APIC support if IVHD type0x11is available. The IVHD type0x10is kept so that old operating system that only parses type0x10can detect the IOMMU device.Signed-off-by Bui Quang Minh <[email protected]>--- hw/i386/acpi-build.c | 127 ++++++++++++++++++++++++++----------------- hw/i386/amd_iommu.c | 21 ++++++- hw/i386/amd_iommu.h | 16 ++++-- 3 files changed, 108 insertions(+), 56 deletions(-)hw/i386/acpi-build.c +78
-49@@ -2339,30 +2339,23 @@ static void build_amd_iommu(GArray *table_data, BIOSLinker *linker, const char *oem_id, const char *oem_table_id) { -int ivhd_table_len =24 ;AMDVIState *s =AMD_IOMMU_DEVICE (x86_iommu_get_default ());GArray *ivhd_blob =g_array_new (false, true,1 );AcpiTable table = { .sig ="IVRS" , .rev =1 , .oem_id = oem_id, .oem_table_id = oem_table_id }; +uint64_t feature_report;acpi_table_begin (&table, table_data);/* IVinfo - IO virtualization information common to all * IOMMU units in a system */ -build_append_int_noprefix (table_data,40UL <<8 /* PASize */ ,4 ); +build_append_int_noprefix (table_data, + (1UL <<0 ) |/* EFRSup */ + (40UL <<8 ),/* PASize */ +4 );/* reserved */ build_append_int_noprefix (table_data,0 ,8 ); -/* IVHD definition - type 10h */ -build_append_int_noprefix (table_data,0x10 ,1 ); -/* virtualization flags */ -build_append_int_noprefix (table_data, - (1UL <<0 ) |/* HtTunEn */ - (1UL <<4 ) |/* iotblSup */ - (1UL <<6 ) |/* PrefSup */ - (1UL <<7 ),/* PPRSup */ -1 ); -/* * A PCI bus walk, for each PCI host bridge, is necessary to create a * complete set of IVHD entries. Do this into a separate blob so that we @@ -2382,56 +2375,92 @@ build_amd_iommu(GArray *table_data, BIOSLinker *linker, const char *oem_id,build_append_int_noprefix (ivhd_blob,0x0000001 ,4 ); } - ivhd_table_len += ivhd_blob->len; -/* * When interrupt remapping is supported, we add a special IVHD device -* for type IO-APIC. -*/ -if (x86_iommu_ir_supported (x86_iommu_get_default ())) { - ivhd_table_len +=8 ; - } - -/* IVHD length */ -build_append_int_noprefix (table_data, ivhd_table_len,2 ); -/* DeviceID */ -build_append_int_noprefix (table_data, -object_property_get_int (OBJECT (&s->pci),"addr" , - &error_abort),2 ); -/* Capability offset */ -build_append_int_noprefix (table_data, s->pci.capab_offset,2 ); -/* IOMMU base address */ -build_append_int_noprefix (table_data, s->mmio.addr,8 ); -/* PCI Segment Group */ -build_append_int_noprefix (table_data,0 ,2 ); -/* IOMMU info */ -build_append_int_noprefix (table_data,0 ,2 ); -/* IOMMU Feature Reporting */ -build_append_int_noprefix (table_data, - (48UL <<30 ) |/* HATS */ - (48UL <<28 ) |/* GATS */ - (1UL <<2 ) |/* GTSup */ - (1UL <<6 ),/* GASup */ -4 ); - -/* IVHD entries as found above */ -g_array_append_vals (table_data, ivhd_blob->data, ivhd_blob->len); -g_array_free (ivhd_blob, TRUE); - -/* -* Add a special IVHD device type. +* for type IO-APIC * Refer to spec - Table 95: IVHD device entry type codes * * Linux IOMMU driver checks for the special IVHD device (type IO-APIC). * See Linux kernel commit 'c2ff5cf5294bcbd7fa50f7d860e90a66db7e5059' */ if (x86_iommu_ir_supported (x86_iommu_get_default ())) { -build_append_int_noprefix (table_data, +build_append_int_noprefix (ivhd_blob, (0x1ull <<56 ) |/* type IOAPIC */ (IOAPIC_SB_DEVID <<40 ) |/* IOAPIC devid */ 0x48 ,/* special device */ 8 ); } + +/* IVHD definition - type 10h */ +build_append_int_noprefix (table_data,0x10 ,1 ); +/* virtualization flags */ +build_append_int_noprefix (table_data, + (1UL <<0 ) |/* HtTunEn */ + (1UL <<4 ) |/* iotblSup */ + (1UL <<6 ) |/* PrefSup */ + (1UL <<7 ),/* PPRSup */ +1 ); + +/* IVHD length */ +build_append_int_noprefix (table_data, ivhd_blob->len +24 ,2 ); +/* DeviceID */ +build_append_int_noprefix (table_data, +object_property_get_int (OBJECT (&s->pci),"addr" , + &error_abort),2 ); +/* Capability offset */ +build_append_int_noprefix (table_data, s->pci.capab_offset,2 ); +/* IOMMU base address */ +build_append_int_noprefix (table_data, s->mmio.addr,8 ); +/* PCI Segment Group */ +build_append_int_noprefix (table_data,0 ,2 ); +/* IOMMU info */ +build_append_int_noprefix (table_data,0 ,2 ); +/* IOMMU Feature Reporting */ + feature_report = (48UL <<30 ) |/* HATS */ + (48UL <<28 ) |/* GATS */ + (1UL <<2 ) |/* GTSup */ + (1UL <<6 );/* GASup */ +if (s->xtsup) { + feature_report |= (1UL <<0 );/* XTSup */ + } +build_append_int_noprefix (table_data, feature_report,4 ); + +/* IVHD entries as found above */ +g_array_append_vals (table_data, ivhd_blob->data, ivhd_blob->len); + +/* IVHD definition - type 11h */ +build_append_int_noprefix (table_data,0x11 ,1 ); +/* virtualization flags */ +build_append_int_noprefix (table_data, + (1UL <<0 ) |/* HtTunEn */ + (1UL <<4 ),/* iotblSup */ +1 ); + +/* IVHD length */ +build_append_int_noprefix (table_data, ivhd_blob->len +40 ,2 ); +/* DeviceID */ +build_append_int_noprefix (table_data, +object_property_get_int (OBJECT (&s->pci),"addr" , + &error_abort),2 ); +/* Capability offset */ +build_append_int_noprefix (table_data, s->pci.capab_offset,2 ); +/* IOMMU base address */ +build_append_int_noprefix (table_data, s->mmio.addr,8 ); +/* PCI Segment Group */ +build_append_int_noprefix (table_data,0 ,2 ); +/* IOMMU info */ +build_append_int_noprefix (table_data,0 ,2 ); +/* IOMMU Attributes */ +build_append_int_noprefix (table_data,0 ,4 ); +/* EFR Register Image */ +build_append_int_noprefix (table_data, s->efr_reg,8 ); +/* EFR Register Image 2 */ +build_append_int_noprefix (table_data,0 ,8 ); + +/* IVHD entries as found above */ +g_array_append_vals (table_data, ivhd_blob->data, ivhd_blob->len); + +g_array_free (ivhd_blob, TRUE);acpi_table_end (linker, &table); }hw/i386/amd_iommu.c +19
-2@@ -31,6 +31,7 @@#include "hw/i386/apic_internal.h" #include "trace.h" #include "hw/i386/apic-msidef.h" +#include "hw/qdev-properties.h" /* used AMD-Vi MMIO registers */ const char *amdvi_mmio_low[] = {@@ -1155,7 +1156,12 @@ static int amdvi_int_remap_ga(AMDVIState *iommu, irq->vector = irte.hi.fields.vector; irq->dest_mode = irte.lo.fields_remap.dm; irq->redir_hint = irte.lo.fields_remap.rq_eoi; - irq->dest = irte.lo.fields_remap.destination; +if (iommu->xtsup) { + irq->dest = irte.lo.fields_remap.destination | + (irte.hi.fields.destination_hi <<24 ); + }else { + irq->dest = irte.lo.fields_remap.destination &0xff ; + }return 0 ; }@@ -1503,10 +1509,15 @@ static void amdvi_init(AMDVIState *s) s->enabled = false; s->ats_enabled = false; s->cmdbuf_enabled = false; + s->efr_reg =AMDVI_DEFAULT_EXT_FEATURES ; + +if (s->xtsup) { + s->efr_reg |=AMDVI_FEATURE_XT ; + }/* reset MMIO */ memset (s->mmior,0 ,AMDVI_MMIO_SIZE ); -amdvi_set_quad (s,AMDVI_MMIO_EXT_FEATURES ,AMDVI_EXT_FEATURES , +amdvi_set_quad (s,AMDVI_MMIO_EXT_FEATURES , s->efr_reg,0xffffffffffffffef ,0 );amdvi_set_quad (s,AMDVI_MMIO_STATUS ,0 ,0x98 ,0x67 ); }@@ -1591,6 +1602,11 @@ static void amdvi_sysbus_realize(DeviceState *dev, Error **errp)amdvi_init (s); } +static Property amdvi_properties[] = { +DEFINE_PROP_BOOL ("xtsup" , AMDVIState, xtsup, false), +DEFINE_PROP_END_OF_LIST (), +}; +static const VMStateDescription vmstate_amdvi_sysbus = { .name ="amd-iommu" , .unmigratable =1 @@ -1617,6 +1633,7 @@ static void amdvi_sysbus_class_init(ObjectClass *klass, void *data) dc->user_creatable = true;set_bit (DEVICE_CATEGORY_MISC , dc->categories); dc->desc ="AMD IOMMU (AMD-Vi) DMA Remapping device" ; +device_class_set_props (dc, amdvi_properties); }static const TypeInfo amdvi_sysbus = {hw/i386/amd_iommu.h +8
-5@@ -154,6 +154,7 @@#define AMDVI_FEATURE_PREFETCH (1ULL << 0)/* page prefetch */ #define AMDVI_FEATURE_PPR (1ULL << 1)/* PPR Support */ +#define AMDVI_FEATURE_XT (1ULL << 2)/* x2APIC Support */ #define AMDVI_FEATURE_GT (1ULL << 4)/* Guest Translation */ #define AMDVI_FEATURE_IA (1ULL << 6)/* inval all support */ #define AMDVI_FEATURE_GA (1ULL << 7)/* guest VAPIC support */ @@ -173,8 +174,9 @@#define AMDVI_IOTLB_MAX_SIZE 1024#define AMDVI_DEVID_SHIFT 36 -/* extended feature support */ -#define AMDVI_EXT_FEATURES (AMDVI_FEATURE_PREFETCH | AMDVI_FEATURE_PPR | \ +/* default extended feature */ +#define AMDVI_DEFAULT_EXT_FEATURES \ + (AMDVI_FEATURE_PREFETCH | AMDVI_FEATURE_PPR | \ AMDVI_FEATURE_IA | AMDVI_FEATURE_GT | AMDVI_FEATURE_HE | \ AMDVI_GATS_MODE | AMDVI_HATS_MODE | AMDVI_FEATURE_GA)@@ -278,8 +280,8 @@ union irte_ga_lo { dm:1, /* ------ */ guest_mode:1, - destination:8, - rsvd_1:48; + destination:24, + rsvd_1:32; } fields_remap; };@@ -287,7 +289,8 @@ union irte_ga_hi {uint64_t val;struct {uint64_t vector:8 , - rsvd_2:56 ; + rsvd_2:48 , + destination_hi:8 ; } fields; };
hw/i386/amd_iommu.h struct AMDVIState { · line 375
Quoted 7 lines
hw/i386/amd_iommu.h +3
-0@@ -366,6 +369,9 @@ struct AMDVIState {/* Interrupt remapping */ bool ga_enabled; +bool xtsup; + +uint64_t efr_reg;/* extended feature register */ };
It would be cleaner to have efr_reg as a function and just call it when needed.
With that addressed:
Quoted 3 lines
hw/i386/amd_iommu.h +0
-0#endif Signature · 1 line
2.25.1
Re: [PATCH v4 5/5] amd_iommu: report x2APIC support to the operating system
Quoted 309 lines · Michael S. Tsirkin
Quoted 297 lines · Bui Quang Minh
This commit adds XTSup configuration to let user choose to whether enable this feature or not. When XTSup is enabled, additional bytes in IRTE with enabled guest virtual VAPIC are used to support 32-bit destination id.
Additionally, this commit exports IVHD type
0x11besides the old IVHD type0x10in ACPI table. IVHD type0x10does not report full set of IOMMU features only the legacy ones, so operating system (e.g. Linux) may only detects x2APIC support if IVHD type0x11is available. The IVHD type0x10is kept so that old operating system that only parses type0x10can detect the IOMMU device.Signed-off-by Bui Quang Minh <[email protected]>--- hw/i386/acpi-build.c | 127 ++++++++++++++++++++++++++----------------- hw/i386/amd_iommu.c | 21 ++++++- hw/i386/amd_iommu.h | 16 ++++-- 3 files changed, 108 insertions(+), 56 deletions(-)hw/i386/acpi-build.c +78
-49@@ -2339,30 +2339,23 @@ static void build_amd_iommu(GArray *table_data, BIOSLinker *linker, const char *oem_id, const char *oem_table_id) { -int ivhd_table_len =24 ;AMDVIState *s =AMD_IOMMU_DEVICE (x86_iommu_get_default ());GArray *ivhd_blob =g_array_new (false, true,1 );AcpiTable table = { .sig ="IVRS" , .rev =1 , .oem_id = oem_id, .oem_table_id = oem_table_id }; +uint64_t feature_report;acpi_table_begin (&table, table_data);/* IVinfo - IO virtualization information common to all * IOMMU units in a system */ -build_append_int_noprefix (table_data,40UL <<8 /* PASize */ ,4 ); +build_append_int_noprefix (table_data, + (1UL <<0 ) |/* EFRSup */ + (40UL <<8 ),/* PASize */ +4 );/* reserved */ build_append_int_noprefix (table_data,0 ,8 ); -/* IVHD definition - type 10h */ -build_append_int_noprefix (table_data,0x10 ,1 ); -/* virtualization flags */ -build_append_int_noprefix (table_data, - (1UL <<0 ) |/* HtTunEn */ - (1UL <<4 ) |/* iotblSup */ - (1UL <<6 ) |/* PrefSup */ - (1UL <<7 ),/* PPRSup */ -1 ); -/* * A PCI bus walk, for each PCI host bridge, is necessary to create a * complete set of IVHD entries. Do this into a separate blob so that we @@ -2382,56 +2375,92 @@ build_amd_iommu(GArray *table_data, BIOSLinker *linker, const char *oem_id,build_append_int_noprefix (ivhd_blob,0x0000001 ,4 ); } - ivhd_table_len += ivhd_blob->len; -/* * When interrupt remapping is supported, we add a special IVHD device -* for type IO-APIC. -*/ -if (x86_iommu_ir_supported (x86_iommu_get_default ())) { - ivhd_table_len +=8 ; - } - -/* IVHD length */ -build_append_int_noprefix (table_data, ivhd_table_len,2 ); -/* DeviceID */ -build_append_int_noprefix (table_data, -object_property_get_int (OBJECT (&s->pci),"addr" , - &error_abort),2 ); -/* Capability offset */ -build_append_int_noprefix (table_data, s->pci.capab_offset,2 ); -/* IOMMU base address */ -build_append_int_noprefix (table_data, s->mmio.addr,8 ); -/* PCI Segment Group */ -build_append_int_noprefix (table_data,0 ,2 ); -/* IOMMU info */ -build_append_int_noprefix (table_data,0 ,2 ); -/* IOMMU Feature Reporting */ -build_append_int_noprefix (table_data, - (48UL <<30 ) |/* HATS */ - (48UL <<28 ) |/* GATS */ - (1UL <<2 ) |/* GTSup */ - (1UL <<6 ),/* GASup */ -4 ); - -/* IVHD entries as found above */ -g_array_append_vals (table_data, ivhd_blob->data, ivhd_blob->len); -g_array_free (ivhd_blob, TRUE); - -/* -* Add a special IVHD device type. +* for type IO-APIC * Refer to spec - Table 95: IVHD device entry type codes * * Linux IOMMU driver checks for the special IVHD device (type IO-APIC). * See Linux kernel commit 'c2ff5cf5294bcbd7fa50f7d860e90a66db7e5059' */ if (x86_iommu_ir_supported (x86_iommu_get_default ())) { -build_append_int_noprefix (table_data, +build_append_int_noprefix (ivhd_blob, (0x1ull <<56 ) |/* type IOAPIC */ (IOAPIC_SB_DEVID <<40 ) |/* IOAPIC devid */ 0x48 ,/* special device */ 8 ); } + +/* IVHD definition - type 10h */ +build_append_int_noprefix (table_data,0x10 ,1 ); +/* virtualization flags */ +build_append_int_noprefix (table_data, + (1UL <<0 ) |/* HtTunEn */ + (1UL <<4 ) |/* iotblSup */ + (1UL <<6 ) |/* PrefSup */ + (1UL <<7 ),/* PPRSup */ +1 ); + +/* IVHD length */ +build_append_int_noprefix (table_data, ivhd_blob->len +24 ,2 ); +/* DeviceID */ +build_append_int_noprefix (table_data, +object_property_get_int (OBJECT (&s->pci),"addr" , + &error_abort),2 ); +/* Capability offset */ +build_append_int_noprefix (table_data, s->pci.capab_offset,2 ); +/* IOMMU base address */ +build_append_int_noprefix (table_data, s->mmio.addr,8 ); +/* PCI Segment Group */ +build_append_int_noprefix (table_data,0 ,2 ); +/* IOMMU info */ +build_append_int_noprefix (table_data,0 ,2 ); +/* IOMMU Feature Reporting */ + feature_report = (48UL <<30 ) |/* HATS */ + (48UL <<28 ) |/* GATS */ + (1UL <<2 ) |/* GTSup */ + (1UL <<6 );/* GASup */ +if (s->xtsup) { + feature_report |= (1UL <<0 );/* XTSup */ + } +build_append_int_noprefix (table_data, feature_report,4 ); + +/* IVHD entries as found above */ +g_array_append_vals (table_data, ivhd_blob->data, ivhd_blob->len); + +/* IVHD definition - type 11h */ +build_append_int_noprefix (table_data,0x11 ,1 ); +/* virtualization flags */ +build_append_int_noprefix (table_data, + (1UL <<0 ) |/* HtTunEn */ + (1UL <<4 ),/* iotblSup */ +1 ); + +/* IVHD length */ +build_append_int_noprefix (table_data, ivhd_blob->len +40 ,2 ); +/* DeviceID */ +build_append_int_noprefix (table_data, +object_property_get_int (OBJECT (&s->pci),"addr" , + &error_abort),2 ); +/* Capability offset */ +build_append_int_noprefix (table_data, s->pci.capab_offset,2 ); +/* IOMMU base address */ +build_append_int_noprefix (table_data, s->mmio.addr,8 ); +/* PCI Segment Group */ +build_append_int_noprefix (table_data,0 ,2 ); +/* IOMMU info */ +build_append_int_noprefix (table_data,0 ,2 ); +/* IOMMU Attributes */ +build_append_int_noprefix (table_data,0 ,4 ); +/* EFR Register Image */ +build_append_int_noprefix (table_data, s->efr_reg,8 ); +/* EFR Register Image 2 */ +build_append_int_noprefix (table_data,0 ,8 ); + +/* IVHD entries as found above */ +g_array_append_vals (table_data, ivhd_blob->data, ivhd_blob->len); + +g_array_free (ivhd_blob, TRUE);acpi_table_end (linker, &table); }hw/i386/amd_iommu.c +19
-2@@ -31,6 +31,7 @@#include "hw/i386/apic_internal.h" #include "trace.h" #include "hw/i386/apic-msidef.h" +#include "hw/qdev-properties.h" /* used AMD-Vi MMIO registers */ const char *amdvi_mmio_low[] = {@@ -1155,7 +1156,12 @@ static int amdvi_int_remap_ga(AMDVIState *iommu, irq->vector = irte.hi.fields.vector; irq->dest_mode = irte.lo.fields_remap.dm; irq->redir_hint = irte.lo.fields_remap.rq_eoi; - irq->dest = irte.lo.fields_remap.destination; +if (iommu->xtsup) { + irq->dest = irte.lo.fields_remap.destination | + (irte.hi.fields.destination_hi <<24 ); + }else { + irq->dest = irte.lo.fields_remap.destination &0xff ; + }return 0 ; }@@ -1503,10 +1509,15 @@ static void amdvi_init(AMDVIState *s) s->enabled = false; s->ats_enabled = false; s->cmdbuf_enabled = false; + s->efr_reg =AMDVI_DEFAULT_EXT_FEATURES ; + +if (s->xtsup) { + s->efr_reg |=AMDVI_FEATURE_XT ; + }/* reset MMIO */ memset (s->mmior,0 ,AMDVI_MMIO_SIZE ); -amdvi_set_quad (s,AMDVI_MMIO_EXT_FEATURES ,AMDVI_EXT_FEATURES , +amdvi_set_quad (s,AMDVI_MMIO_EXT_FEATURES , s->efr_reg,0xffffffffffffffef ,0 );amdvi_set_quad (s,AMDVI_MMIO_STATUS ,0 ,0x98 ,0x67 ); }@@ -1591,6 +1602,11 @@ static void amdvi_sysbus_realize(DeviceState *dev, Error **errp)amdvi_init (s); } +static Property amdvi_properties[] = { +DEFINE_PROP_BOOL ("xtsup" , AMDVIState, xtsup, false), +DEFINE_PROP_END_OF_LIST (), +}; +static const VMStateDescription vmstate_amdvi_sysbus = { .name ="amd-iommu" , .unmigratable =1 @@ -1617,6 +1633,7 @@ static void amdvi_sysbus_class_init(ObjectClass *klass, void *data) dc->user_creatable = true;set_bit (DEVICE_CATEGORY_MISC , dc->categories); dc->desc ="AMD IOMMU (AMD-Vi) DMA Remapping device" ; +device_class_set_props (dc, amdvi_properties); }static const TypeInfo amdvi_sysbus = {hw/i386/amd_iommu.h +8
-5@@ -154,6 +154,7 @@#define AMDVI_FEATURE_PREFETCH (1ULL << 0)/* page prefetch */ #define AMDVI_FEATURE_PPR (1ULL << 1)/* PPR Support */ +#define AMDVI_FEATURE_XT (1ULL << 2)/* x2APIC Support */ #define AMDVI_FEATURE_GT (1ULL << 4)/* Guest Translation */ #define AMDVI_FEATURE_IA (1ULL << 6)/* inval all support */ #define AMDVI_FEATURE_GA (1ULL << 7)/* guest VAPIC support */ @@ -173,8 +174,9 @@#define AMDVI_IOTLB_MAX_SIZE 1024#define AMDVI_DEVID_SHIFT 36 -/* extended feature support */ -#define AMDVI_EXT_FEATURES (AMDVI_FEATURE_PREFETCH | AMDVI_FEATURE_PPR | \ +/* default extended feature */ +#define AMDVI_DEFAULT_EXT_FEATURES \ + (AMDVI_FEATURE_PREFETCH | AMDVI_FEATURE_PPR | \ AMDVI_FEATURE_IA | AMDVI_FEATURE_GT | AMDVI_FEATURE_HE | \ AMDVI_GATS_MODE | AMDVI_HATS_MODE | AMDVI_FEATURE_GA)@@ -278,8 +280,8 @@ union irte_ga_lo { dm:1, /* ------ */ guest_mode:1, - destination:8, - rsvd_1:48; + destination:24, + rsvd_1:32; } fields_remap; };@@ -287,7 +289,8 @@ union irte_ga_hi {uint64_t val;struct {uint64_t vector:8 , - rsvd_2:56 ; + rsvd_2:48 , + destination_hi:8 ; } fields; };hw/i386/amd_iommu.h struct AMDVIState { · line 375
Quoted 7 lines
hw/i386/amd_iommu.h +3
-0@@ -366,6 +369,9 @@ struct AMDVIState {/* Interrupt remapping */ bool ga_enabled; +bool xtsup; + +uint64_t efr_reg;/* extended feature register */ };It would be cleaner to have
efr_regas a function and just call it when needed.
Quoted 3 lines
With that addressed:
Reviewed-by Michael S. Tsirkin <[email protected]>
You mean function like this
uint64_t amdvi_extended_feature_register(AMDVIState *s)
{
uint64_t feature = AMDVI_DEFAULT_EXT_FEATURES;
if (s->xtsup)
feature |= AMDVI_FEATURE_XT;
return feature;
}Quoted 4 lines
Quoted 3 lines
hw/i386/amd_iommu.h +0
-0#endif Signature · 1 line
2.25.1
Re: [PATCH v4 5/5] amd_iommu: report x2APIC support to the operating system
Quoted 301 lines · Bui Quang Minh
Quoted 294 lines · Michael S. Tsirkin
Quoted 283 lines · Bui Quang Minh
This commit adds XTSup configuration to let user choose to whether enable this feature or not. When XTSup is enabled, additional bytes in IRTE with enabled guest virtual VAPIC are used to support 32-bit destination id.
Additionally, this commit exports IVHD type
0x11besides the old IVHD type0x10in ACPI table. IVHD type0x10does not report full set of IOMMU features only the legacy ones, so operating system (e.g. Linux) may only detects x2APIC support if IVHD type0x11is available. The IVHD type0x10is kept so that old operating system that only parses type0x10can detect the IOMMU device.Signed-off-by Bui Quang Minh <[email protected]>--- hw/i386/acpi-build.c | 127 ++++++++++++++++++++++++++----------------- hw/i386/amd_iommu.c | 21 ++++++- hw/i386/amd_iommu.h | 16 ++++-- 3 files changed, 108 insertions(+), 56 deletions(-)hw/i386/acpi-build.c +78
-49@@ -2339,30 +2339,23 @@ static void build_amd_iommu(GArray *table_data, BIOSLinker *linker, const char *oem_id, const char *oem_table_id) { -int ivhd_table_len =24 ;AMDVIState *s =AMD_IOMMU_DEVICE (x86_iommu_get_default ());GArray *ivhd_blob =g_array_new (false, true,1 );AcpiTable table = { .sig ="IVRS" , .rev =1 , .oem_id = oem_id, .oem_table_id = oem_table_id }; +uint64_t feature_report;acpi_table_begin (&table, table_data);/* IVinfo - IO virtualization information common to all * IOMMU units in a system */ -build_append_int_noprefix (table_data,40UL <<8 /* PASize */ ,4 ); +build_append_int_noprefix (table_data, + (1UL <<0 ) |/* EFRSup */ + (40UL <<8 ),/* PASize */ +4 );/* reserved */ build_append_int_noprefix (table_data,0 ,8 ); -/* IVHD definition - type 10h */ -build_append_int_noprefix (table_data,0x10 ,1 ); -/* virtualization flags */ -build_append_int_noprefix (table_data, - (1UL <<0 ) |/* HtTunEn */ - (1UL <<4 ) |/* iotblSup */ - (1UL <<6 ) |/* PrefSup */ - (1UL <<7 ),/* PPRSup */ -1 ); -/* * A PCI bus walk, for each PCI host bridge, is necessary to create a * complete set of IVHD entries. Do this into a separate blob so that we @@ -2382,56 +2375,92 @@ build_amd_iommu(GArray *table_data, BIOSLinker *linker, const char *oem_id,build_append_int_noprefix (ivhd_blob,0x0000001 ,4 ); } - ivhd_table_len += ivhd_blob->len; -/* * When interrupt remapping is supported, we add a special IVHD device -* for type IO-APIC. -*/ -if (x86_iommu_ir_supported (x86_iommu_get_default ())) { - ivhd_table_len +=8 ; - } - -/* IVHD length */ -build_append_int_noprefix (table_data, ivhd_table_len,2 ); -/* DeviceID */ -build_append_int_noprefix (table_data, -object_property_get_int (OBJECT (&s->pci),"addr" , - &error_abort),2 ); -/* Capability offset */ -build_append_int_noprefix (table_data, s->pci.capab_offset,2 ); -/* IOMMU base address */ -build_append_int_noprefix (table_data, s->mmio.addr,8 ); -/* PCI Segment Group */ -build_append_int_noprefix (table_data,0 ,2 ); -/* IOMMU info */ -build_append_int_noprefix (table_data,0 ,2 ); -/* IOMMU Feature Reporting */ -build_append_int_noprefix (table_data, - (48UL <<30 ) |/* HATS */ - (48UL <<28 ) |/* GATS */ - (1UL <<2 ) |/* GTSup */ - (1UL <<6 ),/* GASup */ -4 ); - -/* IVHD entries as found above */ -g_array_append_vals (table_data, ivhd_blob->data, ivhd_blob->len); -g_array_free (ivhd_blob, TRUE); - -/* -* Add a special IVHD device type. +* for type IO-APIC * Refer to spec - Table 95: IVHD device entry type codes * * Linux IOMMU driver checks for the special IVHD device (type IO-APIC). * See Linux kernel commit 'c2ff5cf5294bcbd7fa50f7d860e90a66db7e5059' */ if (x86_iommu_ir_supported (x86_iommu_get_default ())) { -build_append_int_noprefix (table_data, +build_append_int_noprefix (ivhd_blob, (0x1ull <<56 ) |/* type IOAPIC */ (IOAPIC_SB_DEVID <<40 ) |/* IOAPIC devid */ 0x48 ,/* special device */ 8 ); } + +/* IVHD definition - type 10h */ +build_append_int_noprefix (table_data,0x10 ,1 ); +/* virtualization flags */ +build_append_int_noprefix (table_data, + (1UL <<0 ) |/* HtTunEn */ + (1UL <<4 ) |/* iotblSup */ + (1UL <<6 ) |/* PrefSup */ + (1UL <<7 ),/* PPRSup */ +1 ); + +/* IVHD length */ +build_append_int_noprefix (table_data, ivhd_blob->len +24 ,2 ); +/* DeviceID */ +build_append_int_noprefix (table_data, +object_property_get_int (OBJECT (&s->pci),"addr" , + &error_abort),2 ); +/* Capability offset */ +build_append_int_noprefix (table_data, s->pci.capab_offset,2 ); +/* IOMMU base address */ +build_append_int_noprefix (table_data, s->mmio.addr,8 ); +/* PCI Segment Group */ +build_append_int_noprefix (table_data,0 ,2 ); +/* IOMMU info */ +build_append_int_noprefix (table_data,0 ,2 ); +/* IOMMU Feature Reporting */ + feature_report = (48UL <<30 ) |/* HATS */ + (48UL <<28 ) |/* GATS */ + (1UL <<2 ) |/* GTSup */ + (1UL <<6 );/* GASup */ +if (s->xtsup) { + feature_report |= (1UL <<0 );/* XTSup */ + } +build_append_int_noprefix (table_data, feature_report,4 ); + +/* IVHD entries as found above */ +g_array_append_vals (table_data, ivhd_blob->data, ivhd_blob->len); + +/* IVHD definition - type 11h */ +build_append_int_noprefix (table_data,0x11 ,1 ); +/* virtualization flags */ +build_append_int_noprefix (table_data, + (1UL <<0 ) |/* HtTunEn */ + (1UL <<4 ),/* iotblSup */ +1 ); + +/* IVHD length */ +build_append_int_noprefix (table_data, ivhd_blob->len +40 ,2 ); +/* DeviceID */ +build_append_int_noprefix (table_data, +object_property_get_int (OBJECT (&s->pci),"addr" , + &error_abort),2 ); +/* Capability offset */ +build_append_int_noprefix (table_data, s->pci.capab_offset,2 ); +/* IOMMU base address */ +build_append_int_noprefix (table_data, s->mmio.addr,8 ); +/* PCI Segment Group */ +build_append_int_noprefix (table_data,0 ,2 ); +/* IOMMU info */ +build_append_int_noprefix (table_data,0 ,2 ); +/* IOMMU Attributes */ +build_append_int_noprefix (table_data,0 ,4 ); +/* EFR Register Image */ +build_append_int_noprefix (table_data, s->efr_reg,8 ); +/* EFR Register Image 2 */ +build_append_int_noprefix (table_data,0 ,8 ); + +/* IVHD entries as found above */ +g_array_append_vals (table_data, ivhd_blob->data, ivhd_blob->len); + +g_array_free (ivhd_blob, TRUE);acpi_table_end (linker, &table); }hw/i386/amd_iommu.c +19
-2@@ -31,6 +31,7 @@#include "hw/i386/apic_internal.h" #include "trace.h" #include "hw/i386/apic-msidef.h" +#include "hw/qdev-properties.h" /* used AMD-Vi MMIO registers */ const char *amdvi_mmio_low[] = {@@ -1155,7 +1156,12 @@ static int amdvi_int_remap_ga(AMDVIState *iommu, irq->vector = irte.hi.fields.vector; irq->dest_mode = irte.lo.fields_remap.dm; irq->redir_hint = irte.lo.fields_remap.rq_eoi; - irq->dest = irte.lo.fields_remap.destination; +if (iommu->xtsup) { + irq->dest = irte.lo.fields_remap.destination | + (irte.hi.fields.destination_hi <<24 ); + }else { + irq->dest = irte.lo.fields_remap.destination &0xff ; + }return 0 ; }@@ -1503,10 +1509,15 @@ static void amdvi_init(AMDVIState *s) s->enabled = false; s->ats_enabled = false; s->cmdbuf_enabled = false; + s->efr_reg =AMDVI_DEFAULT_EXT_FEATURES ; + +if (s->xtsup) { + s->efr_reg |=AMDVI_FEATURE_XT ; + }/* reset MMIO */ memset (s->mmior,0 ,AMDVI_MMIO_SIZE ); -amdvi_set_quad (s,AMDVI_MMIO_EXT_FEATURES ,AMDVI_EXT_FEATURES , +amdvi_set_quad (s,AMDVI_MMIO_EXT_FEATURES , s->efr_reg,0xffffffffffffffef ,0 );amdvi_set_quad (s,AMDVI_MMIO_STATUS ,0 ,0x98 ,0x67 ); }@@ -1591,6 +1602,11 @@ static void amdvi_sysbus_realize(DeviceState *dev, Error **errp)amdvi_init (s); } +static Property amdvi_properties[] = { +DEFINE_PROP_BOOL ("xtsup" , AMDVIState, xtsup, false), +DEFINE_PROP_END_OF_LIST (), +}; +static const VMStateDescription vmstate_amdvi_sysbus = { .name ="amd-iommu" , .unmigratable =1 @@ -1617,6 +1633,7 @@ static void amdvi_sysbus_class_init(ObjectClass *klass, void *data) dc->user_creatable = true;set_bit (DEVICE_CATEGORY_MISC , dc->categories); dc->desc ="AMD IOMMU (AMD-Vi) DMA Remapping device" ; +device_class_set_props (dc, amdvi_properties); }static const TypeInfo amdvi_sysbus = {hw/i386/amd_iommu.h +8
-5@@ -154,6 +154,7 @@#define AMDVI_FEATURE_PREFETCH (1ULL << 0)/* page prefetch */ #define AMDVI_FEATURE_PPR (1ULL << 1)/* PPR Support */ +#define AMDVI_FEATURE_XT (1ULL << 2)/* x2APIC Support */ #define AMDVI_FEATURE_GT (1ULL << 4)/* Guest Translation */ #define AMDVI_FEATURE_IA (1ULL << 6)/* inval all support */ #define AMDVI_FEATURE_GA (1ULL << 7)/* guest VAPIC support */ @@ -173,8 +174,9 @@#define AMDVI_IOTLB_MAX_SIZE 1024#define AMDVI_DEVID_SHIFT 36 -/* extended feature support */ -#define AMDVI_EXT_FEATURES (AMDVI_FEATURE_PREFETCH | AMDVI_FEATURE_PPR | \ +/* default extended feature */ +#define AMDVI_DEFAULT_EXT_FEATURES \ + (AMDVI_FEATURE_PREFETCH | AMDVI_FEATURE_PPR | \ AMDVI_FEATURE_IA | AMDVI_FEATURE_GT | AMDVI_FEATURE_HE | \ AMDVI_GATS_MODE | AMDVI_HATS_MODE | AMDVI_FEATURE_GA)@@ -278,8 +280,8 @@ union irte_ga_lo { dm:1, /* ------ */ guest_mode:1, - destination:8, - rsvd_1:48; + destination:24, + rsvd_1:32; } fields_remap; };@@ -287,7 +289,8 @@ union irte_ga_hi {uint64_t val;struct {uint64_t vector:8 , - rsvd_2:56 ; + rsvd_2:48 , + destination_hi:8 ; } fields; };hw/i386/amd_iommu.h struct AMDVIState { · line 374
Quoted 6 lines
hw/i386/amd_iommu.h +3
-0@@ -366,6 +369,9 @@ struct AMDVIState {/* Interrupt remapping */ bool ga_enabled; +bool xtsup; + +uint64_t efr_reg;/* extended feature register */ };It would be cleaner to have
efr_regas a function and just call it when needed.Quoted 3 lines
With that addressed:
Reviewed-by Michael S. Tsirkin <[email protected]>You mean function like this
Quoted 8 lines
uint64_tamdvi_extended_feature_register(AMDVIState *s){ uint64_t feature = AMDVI_DEFAULT_EXT_FEATURES; if (s->xtsup) feature |= AMDVI_FEATURE_XT; return feature; }
Yes exactly. going to do that?
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
hw/i386/amd_iommu.h +0
-0#endif Signature · 1 line
2.25.1
Re: [PATCH v4 5/5] amd_iommu: report x2APIC support to the operating system
Quoted 308 lines · Michael S. Tsirkin
Quoted 307 lines · Bui Quang Minh
Quoted 294 lines · Michael S. Tsirkin
Quoted 283 lines · Bui Quang Minh
This commit adds XTSup configuration to let user choose to whether enable this feature or not. When XTSup is enabled, additional bytes in IRTE with enabled guest virtual VAPIC are used to support 32-bit destination id.
Additionally, this commit exports IVHD type
0x11besides the old IVHD type0x10in ACPI table. IVHD type0x10does not report full set of IOMMU features only the legacy ones, so operating system (e.g. Linux) may only detects x2APIC support if IVHD type0x11is available. The IVHD type0x10is kept so that old operating system that only parses type0x10can detect the IOMMU device.Signed-off-by Bui Quang Minh <[email protected]>--- hw/i386/acpi-build.c | 127 ++++++++++++++++++++++++++----------------- hw/i386/amd_iommu.c | 21 ++++++- hw/i386/amd_iommu.h | 16 ++++-- 3 files changed, 108 insertions(+), 56 deletions(-)hw/i386/acpi-build.c +78
-49@@ -2339,30 +2339,23 @@ static void build_amd_iommu(GArray *table_data, BIOSLinker *linker, const char *oem_id, const char *oem_table_id) { -int ivhd_table_len =24 ;AMDVIState *s =AMD_IOMMU_DEVICE (x86_iommu_get_default ());GArray *ivhd_blob =g_array_new (false, true,1 );AcpiTable table = { .sig ="IVRS" , .rev =1 , .oem_id = oem_id, .oem_table_id = oem_table_id }; +uint64_t feature_report;acpi_table_begin (&table, table_data);/* IVinfo - IO virtualization information common to all * IOMMU units in a system */ -build_append_int_noprefix (table_data,40UL <<8 /* PASize */ ,4 ); +build_append_int_noprefix (table_data, + (1UL <<0 ) |/* EFRSup */ + (40UL <<8 ),/* PASize */ +4 );/* reserved */ build_append_int_noprefix (table_data,0 ,8 ); -/* IVHD definition - type 10h */ -build_append_int_noprefix (table_data,0x10 ,1 ); -/* virtualization flags */ -build_append_int_noprefix (table_data, - (1UL <<0 ) |/* HtTunEn */ - (1UL <<4 ) |/* iotblSup */ - (1UL <<6 ) |/* PrefSup */ - (1UL <<7 ),/* PPRSup */ -1 ); -/* * A PCI bus walk, for each PCI host bridge, is necessary to create a * complete set of IVHD entries. Do this into a separate blob so that we @@ -2382,56 +2375,92 @@ build_amd_iommu(GArray *table_data, BIOSLinker *linker, const char *oem_id,build_append_int_noprefix (ivhd_blob,0x0000001 ,4 ); } - ivhd_table_len += ivhd_blob->len; -/* * When interrupt remapping is supported, we add a special IVHD device -* for type IO-APIC. -*/ -if (x86_iommu_ir_supported (x86_iommu_get_default ())) { - ivhd_table_len +=8 ; - } - -/* IVHD length */ -build_append_int_noprefix (table_data, ivhd_table_len,2 ); -/* DeviceID */ -build_append_int_noprefix (table_data, -object_property_get_int (OBJECT (&s->pci),"addr" , - &error_abort),2 ); -/* Capability offset */ -build_append_int_noprefix (table_data, s->pci.capab_offset,2 ); -/* IOMMU base address */ -build_append_int_noprefix (table_data, s->mmio.addr,8 ); -/* PCI Segment Group */ -build_append_int_noprefix (table_data,0 ,2 ); -/* IOMMU info */ -build_append_int_noprefix (table_data,0 ,2 ); -/* IOMMU Feature Reporting */ -build_append_int_noprefix (table_data, - (48UL <<30 ) |/* HATS */ - (48UL <<28 ) |/* GATS */ - (1UL <<2 ) |/* GTSup */ - (1UL <<6 ),/* GASup */ -4 ); - -/* IVHD entries as found above */ -g_array_append_vals (table_data, ivhd_blob->data, ivhd_blob->len); -g_array_free (ivhd_blob, TRUE); - -/* -* Add a special IVHD device type. +* for type IO-APIC * Refer to spec - Table 95: IVHD device entry type codes * * Linux IOMMU driver checks for the special IVHD device (type IO-APIC). * See Linux kernel commit 'c2ff5cf5294bcbd7fa50f7d860e90a66db7e5059' */ if (x86_iommu_ir_supported (x86_iommu_get_default ())) { -build_append_int_noprefix (table_data, +build_append_int_noprefix (ivhd_blob, (0x1ull <<56 ) |/* type IOAPIC */ (IOAPIC_SB_DEVID <<40 ) |/* IOAPIC devid */ 0x48 ,/* special device */ 8 ); } + +/* IVHD definition - type 10h */ +build_append_int_noprefix (table_data,0x10 ,1 ); +/* virtualization flags */ +build_append_int_noprefix (table_data, + (1UL <<0 ) |/* HtTunEn */ + (1UL <<4 ) |/* iotblSup */ + (1UL <<6 ) |/* PrefSup */ + (1UL <<7 ),/* PPRSup */ +1 ); + +/* IVHD length */ +build_append_int_noprefix (table_data, ivhd_blob->len +24 ,2 ); +/* DeviceID */ +build_append_int_noprefix (table_data, +object_property_get_int (OBJECT (&s->pci),"addr" , + &error_abort),2 ); +/* Capability offset */ +build_append_int_noprefix (table_data, s->pci.capab_offset,2 ); +/* IOMMU base address */ +build_append_int_noprefix (table_data, s->mmio.addr,8 ); +/* PCI Segment Group */ +build_append_int_noprefix (table_data,0 ,2 ); +/* IOMMU info */ +build_append_int_noprefix (table_data,0 ,2 ); +/* IOMMU Feature Reporting */ + feature_report = (48UL <<30 ) |/* HATS */ + (48UL <<28 ) |/* GATS */ + (1UL <<2 ) |/* GTSup */ + (1UL <<6 );/* GASup */ +if (s->xtsup) { + feature_report |= (1UL <<0 );/* XTSup */ + } +build_append_int_noprefix (table_data, feature_report,4 ); + +/* IVHD entries as found above */ +g_array_append_vals (table_data, ivhd_blob->data, ivhd_blob->len); + +/* IVHD definition - type 11h */ +build_append_int_noprefix (table_data,0x11 ,1 ); +/* virtualization flags */ +build_append_int_noprefix (table_data, + (1UL <<0 ) |/* HtTunEn */ + (1UL <<4 ),/* iotblSup */ +1 ); + +/* IVHD length */ +build_append_int_noprefix (table_data, ivhd_blob->len +40 ,2 ); +/* DeviceID */ +build_append_int_noprefix (table_data, +object_property_get_int (OBJECT (&s->pci),"addr" , + &error_abort),2 ); +/* Capability offset */ +build_append_int_noprefix (table_data, s->pci.capab_offset,2 ); +/* IOMMU base address */ +build_append_int_noprefix (table_data, s->mmio.addr,8 ); +/* PCI Segment Group */ +build_append_int_noprefix (table_data,0 ,2 ); +/* IOMMU info */ +build_append_int_noprefix (table_data,0 ,2 ); +/* IOMMU Attributes */ +build_append_int_noprefix (table_data,0 ,4 ); +/* EFR Register Image */ +build_append_int_noprefix (table_data, s->efr_reg,8 ); +/* EFR Register Image 2 */ +build_append_int_noprefix (table_data,0 ,8 ); + +/* IVHD entries as found above */ +g_array_append_vals (table_data, ivhd_blob->data, ivhd_blob->len); + +g_array_free (ivhd_blob, TRUE);acpi_table_end (linker, &table); }hw/i386/amd_iommu.c +19
-2@@ -31,6 +31,7 @@#include "hw/i386/apic_internal.h" #include "trace.h" #include "hw/i386/apic-msidef.h" +#include "hw/qdev-properties.h" /* used AMD-Vi MMIO registers */ const char *amdvi_mmio_low[] = {@@ -1155,7 +1156,12 @@ static int amdvi_int_remap_ga(AMDVIState *iommu, irq->vector = irte.hi.fields.vector; irq->dest_mode = irte.lo.fields_remap.dm; irq->redir_hint = irte.lo.fields_remap.rq_eoi; - irq->dest = irte.lo.fields_remap.destination; +if (iommu->xtsup) { + irq->dest = irte.lo.fields_remap.destination | + (irte.hi.fields.destination_hi <<24 ); + }else { + irq->dest = irte.lo.fields_remap.destination &0xff ; + }return 0 ; }@@ -1503,10 +1509,15 @@ static void amdvi_init(AMDVIState *s) s->enabled = false; s->ats_enabled = false; s->cmdbuf_enabled = false; + s->efr_reg =AMDVI_DEFAULT_EXT_FEATURES ; + +if (s->xtsup) { + s->efr_reg |=AMDVI_FEATURE_XT ; + }/* reset MMIO */ memset (s->mmior,0 ,AMDVI_MMIO_SIZE ); -amdvi_set_quad (s,AMDVI_MMIO_EXT_FEATURES ,AMDVI_EXT_FEATURES , +amdvi_set_quad (s,AMDVI_MMIO_EXT_FEATURES , s->efr_reg,0xffffffffffffffef ,0 );amdvi_set_quad (s,AMDVI_MMIO_STATUS ,0 ,0x98 ,0x67 ); }@@ -1591,6 +1602,11 @@ static void amdvi_sysbus_realize(DeviceState *dev, Error **errp)amdvi_init (s); } +static Property amdvi_properties[] = { +DEFINE_PROP_BOOL ("xtsup" , AMDVIState, xtsup, false), +DEFINE_PROP_END_OF_LIST (), +}; +static const VMStateDescription vmstate_amdvi_sysbus = { .name ="amd-iommu" , .unmigratable =1 @@ -1617,6 +1633,7 @@ static void amdvi_sysbus_class_init(ObjectClass *klass, void *data) dc->user_creatable = true;set_bit (DEVICE_CATEGORY_MISC , dc->categories); dc->desc ="AMD IOMMU (AMD-Vi) DMA Remapping device" ; +device_class_set_props (dc, amdvi_properties); }static const TypeInfo amdvi_sysbus = {hw/i386/amd_iommu.h +8
-5@@ -154,6 +154,7 @@#define AMDVI_FEATURE_PREFETCH (1ULL << 0)/* page prefetch */ #define AMDVI_FEATURE_PPR (1ULL << 1)/* PPR Support */ +#define AMDVI_FEATURE_XT (1ULL << 2)/* x2APIC Support */ #define AMDVI_FEATURE_GT (1ULL << 4)/* Guest Translation */ #define AMDVI_FEATURE_IA (1ULL << 6)/* inval all support */ #define AMDVI_FEATURE_GA (1ULL << 7)/* guest VAPIC support */ @@ -173,8 +174,9 @@#define AMDVI_IOTLB_MAX_SIZE 1024#define AMDVI_DEVID_SHIFT 36 -/* extended feature support */ -#define AMDVI_EXT_FEATURES (AMDVI_FEATURE_PREFETCH | AMDVI_FEATURE_PPR | \ +/* default extended feature */ +#define AMDVI_DEFAULT_EXT_FEATURES \ + (AMDVI_FEATURE_PREFETCH | AMDVI_FEATURE_PPR | \ AMDVI_FEATURE_IA | AMDVI_FEATURE_GT | AMDVI_FEATURE_HE | \ AMDVI_GATS_MODE | AMDVI_HATS_MODE | AMDVI_FEATURE_GA)@@ -278,8 +280,8 @@ union irte_ga_lo { dm:1, /* ------ */ guest_mode:1, - destination:8, - rsvd_1:48; + destination:24, + rsvd_1:32; } fields_remap; };@@ -287,7 +289,8 @@ union irte_ga_hi {uint64_t val;struct {uint64_t vector:8 , - rsvd_2:56 ; + rsvd_2:48 , + destination_hi:8 ; } fields; };hw/i386/amd_iommu.h struct AMDVIState { · line 374
Quoted 6 lines
hw/i386/amd_iommu.h +3
-0@@ -366,6 +369,9 @@ struct AMDVIState {/* Interrupt remapping */ bool ga_enabled; +bool xtsup; + +uint64_t efr_reg;/* extended feature register */ };It would be cleaner to have
efr_regas a function and just call it when needed.Quoted 3 lines
With that addressed:
Reviewed-by Michael S. Tsirkin <[email protected]>You mean function like this
uint64_tamdvi_extended_feature_register(AMDVIState *s){ uint64_t feature = AMDVI_DEFAULT_EXT_FEATURES; if (s->xtsup) feature |= AMDVI_FEATURE_XT;
Quoted 4 lines
Quoted 2 lines
return feature; }Yes exactly. going to do that?
Okay, I'll do it.
Quoted 76 lines · Bui Quang Minh
Hi everyone,
This series implements x2APIC mode in userspace local APIC and the RDMSR/WRMSR helper to access x2APIC registers in x2APIC mode. Intel iommu and AMD iommu are adjusted to support x2APIC interrupt remapping. With this series, we can now boot Linux kernel into x2APIC mode with TCG accelerator using either Intel or AMD iommu.
Testing to boot my own built Linux 6.3.0-rc2, the kernel successfully boot with enabled x2APIC and can enumerate CPU with APIC ID 257 Using Intel IOMMU
qemu/build/qemu-system-x86_64 \
-smp 2,maxcpus=260 \
-cpu qemu64,x2apic=on \
-machine q35 \
-device intel-iommu,intremap=on,eim=on \
-device qemu64-x86_64-cpu,x2apic=on,core-id=257,socket-id=0,thread-id=0 \
-m 2G \
-kernel $KERNEL_DIR \
-append "nokaslr console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \
-drive file=$IMAGE_DIR,format=raw \
-nographic \
-sUsing AMD IOMMU
qemu/build/qemu-system-x86_64 \
-smp 2,maxcpus=260 \
-cpu qemu64,x2apic=on \
-machine q35 \
-device amd-iommu,intremap=on,xtsup=on \
-device qemu64-x86_64-cpu,x2apic=on,core-id=257,socket-id=0,thread-id=0 \
-m 2G \
-kernel $KERNEL_DIR \
-append "nokaslr console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \
-drive file=$IMAGE_DIR,format=raw \
-nographic \
-sTesting the emulated userspace APIC with kvm-unit-tests, disable test device with this patch
lib/x86/fwcfg.c +1
-0@@ -27,6 +27,7 @@ static void read_cfg_override(void)if ((str =getenv ("TEST_DEVICE" ))) no_test_device = !atol (str); + no_test_device = true;if ((str =getenv ("MEMLIMIT" ))) fw_override[FW_CFG_MAX_RAM ] =atol (str) *1024 *1024 ;~ env QEMU=/home/minh/Desktop/oss/qemu/build/qemu-system-x86_64 ACCEL=tcg \
./run_tests.sh -v -g apicTESTNAME=apic-split TIMEOUT=90s ACCEL=tcg ./x86/run x86/apic.flat -smp 2 -cpu qemu64,+x2apic,+tsc-deadline -machine kernel_irqchip=split FAIL apic-split (54 tests, 8 unexpected failures, 1 skipped) TESTNAME=ioapic-split TIMEOUT=90s ACCEL=tcg ./x86/run x86/ioapic.flat -smp 1 -cpu qemu64 -machine kernel_irqchip=split PASS ioapic-split (19 tests) TESTNAME=x2apic TIMEOUT=30 ACCEL=tcg ./x86/run x86/apic.flat -smp 2 -cpu qemu64,+x2apic,+tsc-deadline FAIL x2apic (54 tests, 8 unexpected failures, 1 skipped) TESTNAME=xapic TIMEOUT=60 ACCEL=tcg ./x86/run x86/apic.flat -smp 2 -cpu qemu64,-x2apic,+tsc-deadline -machine pit=off FAIL xapic (43 tests, 6 unexpected failures, 2 skipped)
FAIL:
apic_disable: *0xfee00030: 50014
FAIL:apic_disable: *0xfee00080: f0
FAIL:apic_disable: *0xfee00030: 50014
FAIL:apic_disable: *0xfee00080: f0
FAIL: apicbase: relocate apic
Quoted 5 lines
These errors are because we don't disable MMIO region when switching to x2APIC and don't support relocate MMIO region yet. This is a problem because, MMIO region is the same for all CPUs, in order to support these we need to figure out how to allocate and manage different MMIO regions for each CPUs.
Oh interesting point.
Paolo what do you say? Can memory core support something like this?
Quoted 62 lines
This can be an improvement in the future.
FAIL: nmi-after-sti
FAIL: multiple nmiThese errors are in the way we handle
CPU_INTERRUPT_NMIin core TCG.FAIL: TMCCT should stay at zero
This error is related to APIC timer which should be addressed in separate patch.
Version 4 changes,
- Patch 5:
+ Instead of replacing IVHD type0x10with type0x11, export both types for backward compatibility with old guest operating system
+ Flip the xtsup feature check condition inamdvi_int_remap_gafor readabilityVersion 3 changes,
- Patch 2:
+ Allow APIC ID > 255 only when x2APIC feature is supported on CPU
+ Make physical destination mode IPI which has destination id0xffffffffa broadcast to xAPIC CPUs
+ Make cluster address0xfin cluster model of xAPIC logical destination mode a broadcast to all clusters
+ Create newextended_log_destto storeAPIC_LDRinformation in x2APIC instead of extendinglog_destfor backward compatibility in vmstateVersion 2 changes,
- Add support for APIC ID larger than 255
- Adjust AMD iommu for x2APIC suuport
- Reorganize and split patch 1,2 into patch 1,2,3 in version 2Thanks,
Quang Minh.Bui Quang Minh (5):
i386/tcg: implement x2APIC registers MSR access
apic: add support for x2APIC mode
apic, i386/tcg: add x2apic transitions
intel_iommu: allow Extended Interrupt Mode when using userspace APIC
amd_iommu: report x2APIC support to the operating systemhw/i386/acpi-build.c | 127 +++++---- hw/i386/amd_iommu.c | 21 +- hw/i386/amd_iommu.h | 16 +- hw/i386/intel_iommu.c | 11 - hw/i386/x86.c | 8 +- hw/intc/apic.c | 395 +++++++++++++++++++++------ hw/intc/apic_common.c | 16 +- hw/intc/trace-events | 4 +- include/hw/i386/apic.h | 6 +- include/hw/i386/apic_internal.h | 7 +- target/i386/cpu-sysemu.c | 18 +- target/i386/cpu.c | 5 +- target/i386/cpu.h | 9 + target/i386/tcg/sysemu/misc_helper.c | 31 +++ 14 files changed, 500 insertions(+), 174 deletions(-)Signature · 1 line
2.25.1
Quoted 86 lines · Bui Quang Minh
Hi everyone,
This series implements x2APIC mode in userspace local APIC and the RDMSR/WRMSR helper to access x2APIC registers in x2APIC mode. Intel iommu and AMD iommu are adjusted to support x2APIC interrupt remapping. With this series, we can now boot Linux kernel into x2APIC mode with TCG accelerator using either Intel or AMD iommu.
Testing to boot my own built Linux 6.3.0-rc2, the kernel successfully boot with enabled x2APIC and can enumerate CPU with APIC ID 257
Using Intel IOMMU
qemu/build/qemu-system-x86_64 \
-smp 2,maxcpus=260 \
-cpu qemu64,x2apic=on \
-machine q35 \
-device intel-iommu,intremap=on,eim=on \
-device qemu64-x86_64-cpu,x2apic=on,core-id=257,socket-id=0,thread-id=0 \
-m 2G \
-kernel $KERNEL_DIR \
-append "nokaslr console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \
-drive file=$IMAGE_DIR,format=raw \
-nographic \
-sUsing AMD IOMMU
qemu/build/qemu-system-x86_64 \
-smp 2,maxcpus=260 \
-cpu qemu64,x2apic=on \
-machine q35 \
-device amd-iommu,intremap=on,xtsup=on \
-device qemu64-x86_64-cpu,x2apic=on,core-id=257,socket-id=0,thread-id=0 \
-m 2G \
-kernel $KERNEL_DIR \
-append "nokaslr console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \
-drive file=$IMAGE_DIR,format=raw \
-nographic \
-sTesting the emulated userspace APIC with kvm-unit-tests, disable test device with this patch
lib/x86/fwcfg.c +1
-0@@ -27,6 +27,7 @@ static void read_cfg_override(void)if ((str =getenv ("TEST_DEVICE" ))) no_test_device = !atol (str); + no_test_device = true;if ((str =getenv ("MEMLIMIT" ))) fw_override[FW_CFG_MAX_RAM ] =atol (str) *1024 *1024 ;~ env QEMU=/home/minh/Desktop/oss/qemu/build/qemu-system-x86_64 ACCEL=tcg \
./run_tests.sh -v -g apicTESTNAME=apic-split TIMEOUT=90s ACCEL=tcg ./x86/run x86/apic.flat -smp 2 -cpu qemu64,+x2apic,+tsc-deadline -machine kernel_irqchip=split FAIL apic-split (54 tests, 8 unexpected failures, 1 skipped) TESTNAME=ioapic-split TIMEOUT=90s ACCEL=tcg ./x86/run x86/ioapic.flat -smp 1 -cpu qemu64 -machine kernel_irqchip=split PASS ioapic-split (19 tests) TESTNAME=x2apic TIMEOUT=30 ACCEL=tcg ./x86/run x86/apic.flat -smp 2 -cpu qemu64,+x2apic,+tsc-deadline FAIL x2apic (54 tests, 8 unexpected failures, 1 skipped) TESTNAME=xapic TIMEOUT=60 ACCEL=tcg ./x86/run x86/apic.flat -smp 2 -cpu qemu64,-x2apic,+tsc-deadline -machine pit=off FAIL xapic (43 tests, 6 unexpected failures, 2 skipped)
FAIL:
apic_disable: *0xfee00030: 50014
FAIL:apic_disable: *0xfee00080: f0
FAIL:apic_disable: *0xfee00030: 50014
FAIL:apic_disable: *0xfee00080: f0
FAIL: apicbase: relocate apicThese errors are because we don't disable MMIO region when switching to x2APIC and don't support relocate MMIO region yet. This is a problem because, MMIO region is the same for all CPUs, in order to support these we need to figure out how to allocate and manage different MMIO regions for each CPUs. This can be an improvement in the future.
FAIL: nmi-after-sti
FAIL: multiple nmi
Quoted 6 lines
These errors are in the way we handle
CPU_INTERRUPT_NMIin core TCG.FAIL: TMCCT should stay at zero
This error is related to APIC timer which should be addressed in separate patch.
So, I'm not sure how safe it is to merge as is - are we creating a way for users to shoot themselves in the foot? Would be better to just fix these issues before we merge.
Quoted 50 lines
Version 4 changes,
- Patch 5:
+ Instead of replacing IVHD type0x10with type0x11, export both types for backward compatibility with old guest operating system
+ Flip the xtsup feature check condition inamdvi_int_remap_gafor readabilityVersion 3 changes,
- Patch 2:
+ Allow APIC ID > 255 only when x2APIC feature is supported on CPU
+ Make physical destination mode IPI which has destination id0xffffffffa broadcast to xAPIC CPUs
+ Make cluster address0xfin cluster model of xAPIC logical destination mode a broadcast to all clusters
+ Create newextended_log_destto storeAPIC_LDRinformation in x2APIC instead of extendinglog_destfor backward compatibility in vmstateVersion 2 changes,
- Add support for APIC ID larger than 255
- Adjust AMD iommu for x2APIC suuport
- Reorganize and split patch 1,2 into patch 1,2,3 in version 2Thanks,
Quang Minh.Bui Quang Minh (5):
i386/tcg: implement x2APIC registers MSR access
apic: add support for x2APIC mode
apic, i386/tcg: add x2apic transitions
intel_iommu: allow Extended Interrupt Mode when using userspace APIC
amd_iommu: report x2APIC support to the operating systemhw/i386/acpi-build.c | 127 +++++---- hw/i386/amd_iommu.c | 21 +- hw/i386/amd_iommu.h | 16 +- hw/i386/intel_iommu.c | 11 - hw/i386/x86.c | 8 +- hw/intc/apic.c | 395 +++++++++++++++++++++------ hw/intc/apic_common.c | 16 +- hw/intc/trace-events | 4 +- include/hw/i386/apic.h | 6 +- include/hw/i386/apic_internal.h | 7 +- target/i386/cpu-sysemu.c | 18 +- target/i386/cpu.c | 5 +- target/i386/cpu.h | 9 + target/i386/tcg/sysemu/misc_helper.c | 31 +++ 14 files changed, 500 insertions(+), 174 deletions(-)Signature · 1 line
2.25.1
Quoted 89 lines · Michael S. Tsirkin
Quoted 88 lines · Bui Quang Minh
Hi everyone,
This series implements x2APIC mode in userspace local APIC and the RDMSR/WRMSR helper to access x2APIC registers in x2APIC mode. Intel iommu and AMD iommu are adjusted to support x2APIC interrupt remapping. With this series, we can now boot Linux kernel into x2APIC mode with TCG accelerator using either Intel or AMD iommu.
Testing to boot my own built Linux 6.3.0-rc2, the kernel successfully boot with enabled x2APIC and can enumerate CPU with APIC ID 257
Using Intel IOMMU
qemu/build/qemu-system-x86_64 \ -smp 2,maxcpus=260 \ -cpu qemu64,x2apic=on \ -machine q35 \ -device intel-iommu,intremap=on,eim=on \ -device qemu64-x86_64-cpu,x2apic=on,core-id=257,socket-id=0,thread-id=0 \ -m 2G \ -kernel $KERNEL_DIR \ -append "nokaslr console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \ -drive file=$IMAGE_DIR,format=raw \ -nographic \ -sUsing AMD IOMMU
qemu/build/qemu-system-x86_64 \ -smp 2,maxcpus=260 \ -cpu qemu64,x2apic=on \ -machine q35 \ -device amd-iommu,intremap=on,xtsup=on \ -device qemu64-x86_64-cpu,x2apic=on,core-id=257,socket-id=0,thread-id=0 \ -m 2G \ -kernel $KERNEL_DIR \ -append "nokaslr console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \ -drive file=$IMAGE_DIR,format=raw \ -nographic \ -sTesting the emulated userspace APIC with kvm-unit-tests, disable test device with this patch
lib/x86/fwcfg.c +1
-0@@ -27,6 +27,7 @@ static void read_cfg_override(void)if ((str =getenv ("TEST_DEVICE" ))) no_test_device = !atol (str); + no_test_device = true;if ((str =getenv ("MEMLIMIT" ))) fw_override[FW_CFG_MAX_RAM ] =atol (str) *1024 *1024 ;~ env QEMU=/home/minh/Desktop/oss/qemu/build/qemu-system-x86_64 ACCEL=tcg \
./run_tests.sh -v -g apicTESTNAME=apic-split TIMEOUT=90s ACCEL=tcg ./x86/run x86/apic.flat -smp 2 -cpu qemu64,+x2apic,+tsc-deadline -machine kernel_irqchip=split FAIL apic-split (54 tests, 8 unexpected failures, 1 skipped) TESTNAME=ioapic-split TIMEOUT=90s ACCEL=tcg ./x86/run x86/ioapic.flat -smp 1 -cpu qemu64 -machine kernel_irqchip=split PASS ioapic-split (19 tests) TESTNAME=x2apic TIMEOUT=30 ACCEL=tcg ./x86/run x86/apic.flat -smp 2 -cpu qemu64,+x2apic,+tsc-deadline FAIL x2apic (54 tests, 8 unexpected failures, 1 skipped) TESTNAME=xapic TIMEOUT=60 ACCEL=tcg ./x86/run x86/apic.flat -smp 2 -cpu qemu64,-x2apic,+tsc-deadline -machine pit=off FAIL xapic (43 tests, 6 unexpected failures, 2 skipped)
FAIL: apic_disable: *0xfee00030: 50014 FAIL: apic_disable: *0xfee00080: f0 FAIL: apic_disable: *0xfee00030: 50014 FAIL: apic_disable: *0xfee00080: f0 FAIL: apicbase: relocate apicThese errors are because we don't disable MMIO region when switching to x2APIC and don't support relocate MMIO region yet. This is a problem because, MMIO region is the same for all CPUs, in order to support these we need to figure out how to allocate and manage different MMIO regions for each CPUs. This can be an improvement in the future.
FAIL: nmi-after-sti
FAIL: multiple nmiThese errors are in the way we handle
CPU_INTERRUPT_NMIin core TCG.
Quoted 8 lines
Quoted 4 lines
FAIL: TMCCT should stay at zero
This error is related to APIC timer which should be addressed in separate patch.
So, I'm not sure how safe it is to merge as is - are we creating a way for users to shoot themselves in the foot? Would be better to just fix these issues before we merge.
I may be biased as I am the author and willing this to be merged. These issues are not created by this series, the current APIC implementation fails those tests too.
Currently, I only have some ideas to fix the timer failed test case. The timer failed test case is that when the timer is in one-shot mode, the initial count is non-zero. If the current count reaches 0, a timer interrupt is delivered and no more interrupt. At this time, if we write to timer LVT to change to periodic mode, according to Intel SDM Section 11.5.4 APIC timer
"Changing the mode of the APIC timer (from one-shot to periodic or vice versa) by writing to the timer LVT entry does not start the timer. To start the timer, it is necessary to write to the initial- count register as described above"
So it is expected that periodic mode does not start until initial count is written. However, in our current implementation, we see the initial is non-zero, so we start the new periodic timer. The solution might be not calling apic_timer_update when timer LVT written anymore and make the apic_get_current_count returns 0 in the above case.
In my opinion, this series is expected to add the x2APIC feature, so if the user wants to use it, they need to enable this CPU feature flag. Otherwise, with their old configuration, it is expected to behave the same.
Quoted 51 lines
Quoted 50 lines
Version 4 changes,
- Patch 5:
+ Instead of replacing IVHD type0x10with type0x11, export both types for backward compatibility with old guest operating system
+ Flip the xtsup feature check condition inamdvi_int_remap_gafor readabilityVersion 3 changes,
- Patch 2:
+ Allow APIC ID > 255 only when x2APIC feature is supported on CPU
+ Make physical destination mode IPI which has destination id0xffffffffa broadcast to xAPIC CPUs
+ Make cluster address0xfin cluster model of xAPIC logical destination mode a broadcast to all clusters
+ Create newextended_log_destto storeAPIC_LDRinformation in x2APIC instead of extendinglog_destfor backward compatibility in vmstateVersion 2 changes,
- Add support for APIC ID larger than 255
- Adjust AMD iommu for x2APIC suuport
- Reorganize and split patch 1,2 into patch 1,2,3 in version 2Thanks,
Quang Minh.Bui Quang Minh (5):
i386/tcg: implement x2APIC registers MSR access
apic: add support for x2APIC mode
apic, i386/tcg: add x2apic transitions
intel_iommu: allow Extended Interrupt Mode when using userspace APIC
amd_iommu: report x2APIC support to the operating systemhw/i386/acpi-build.c | 127 +++++---- hw/i386/amd_iommu.c | 21 +- hw/i386/amd_iommu.h | 16 +- hw/i386/intel_iommu.c | 11 - hw/i386/x86.c | 8 +- hw/intc/apic.c | 395 +++++++++++++++++++++------ hw/intc/apic_common.c | 16 +- hw/intc/trace-events | 4 +- include/hw/i386/apic.h | 6 +- include/hw/i386/apic_internal.h | 7 +- target/i386/cpu-sysemu.c | 18 +- target/i386/cpu.c | 5 +- target/i386/cpu.h | 9 + target/i386/tcg/sysemu/misc_helper.c | 31 +++ 14 files changed, 500 insertions(+), 174 deletions(-)Signature · 1 line
2.25.1