Skip to contentSkip to conversation

Support x2APIC mode with TCG accelerator

PATCH Superseded Bui Quang Minh 2 participants 14 messages Dec 25, 2023 – Jan 7, 2024 Arm/Amlogic Meson SoC support

v11 of 12 Superseded Latest is v12 →

lore
Share & tools
lore
#0 lore raw

v11 cover Support x2APIC mode with TCG accelerator

Hi everyone,

This series implements x2APIC mode in userspace local APIC and the RDMSR/WRMSR helper to access x2APIC registers in x2APIC mode. Intel iommu and AMD iommu are adjusted to support x2APIC interrupt remapping. With this series, we can now boot Linux kernel into x2APIC mode with TCG accelerator using either Intel or AMD iommu.

Testing to boot my own built Linux 6.3.0-rc2, the kernel successfully boot with enabled x2APIC and can enumerate CPU with APIC ID 257

Using Intel IOMMU

qemu/build/qemu-system-x86_64 \
-smp 2,maxcpus=260 \
-cpu qemu64,x2apic=on \
-machine q35 \
-device intel-iommu,intremap=on,eim=on \
-device qemu64-x86_64-cpu,x2apic=on,core-id=257,socket-id=0,thread-id=0 \
-m 2G \
-kernel $KERNEL_DIR \
-append "nokaslr console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \
-drive file=$IMAGE_DIR,format=raw \
-nographic \
-s

Using AMD IOMMU

qemu/build/qemu-system-x86_64 \
-smp 2,maxcpus=260 \
-cpu qemu64,x2apic=on \
-machine q35 \
-device amd-iommu,intremap=on,xtsup=on \
-device qemu64-x86_64-cpu,x2apic=on,core-id=257,socket-id=0,thread-id=0 \
-m 2G \
-kernel $KERNEL_DIR \
-append "nokaslr console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \
-drive file=$IMAGE_DIR,format=raw \
-nographic \
-s

Testing the emulated userspace APIC with kvm-unit-tests, disable test device with this patch

lib/x86/fwcfg.c +1 -0
index 1734afb..f56fe1c 100644
@@ -27,6 +27,7 @@ static void read_cfg_override(void)
 
        if ((str = getenv("TEST_DEVICE")))
                no_test_device = !atol(str);
+       no_test_device = true;
 
        if ((str = getenv("MEMLIMIT")))
                fw_override[FW_CFG_MAX_RAM] = atol(str) * 1024 * 1024;

~ env QEMU=/home/minh/Desktop/oss/qemu/build/qemu-system-x86_64 ACCEL=tcg \
./run_tests.sh -v -g apic

TESTNAME=apic-split TIMEOUT=90s ACCEL=tcg ./x86/run x86/apic.flat -smp 2 -cpu qemu64,+x2apic,+tsc-deadline -machine kernel_irqchip=split FAIL apic-split (54 tests, 8 unexpected failures, 1 skipped) TESTNAME=ioapic-split TIMEOUT=90s ACCEL=tcg ./x86/run x86/ioapic.flat -smp 1 -cpu qemu64 -machine kernel_irqchip=split PASS ioapic-split (19 tests) TESTNAME=x2apic TIMEOUT=30 ACCEL=tcg ./x86/run x86/apic.flat -smp 2 -cpu qemu64,+x2apic,+tsc-deadline FAIL x2apic (54 tests, 8 unexpected failures, 1 skipped) TESTNAME=xapic TIMEOUT=60 ACCEL=tcg ./x86/run x86/apic.flat -smp 2 -cpu qemu64,-x2apic,+tsc-deadline -machine pit=off FAIL xapic (43 tests, 6 unexpected failures, 2 skipped)

FAIL: apic_disable: *0xfee00030: 50014
FAIL: apic_disable: *0xfee00080: f0
FAIL: apic_disable: *0xfee00030: 50014
FAIL: apic_disable: *0xfee00080: f0
FAIL: apicbase: relocate apic

These errors are because we don't disable MMIO region when switching to x2APIC and don't support relocate MMIO region yet. This is a problem because, MMIO region is the same for all CPUs, in order to support these we need to figure out how to allocate and manage different MMIO regions for each CPUs. This can be an improvement in the future.

FAIL: nmi-after-sti
FAIL: multiple nmi

These errors are in the way we handle CPU_INTERRUPT_NMI in core TCG.

FAIL: TMCCT should stay at zero

This error is related to APIC timer which should be addressed in separate patch.

Version 11 changes,
- Patch 2:
+ Rebase to master and fix conflict with commit c04cfb4596 (hw/i386: fix short-circuit logic with non-optimizing builds)

Version 10 changes,
- Patch 2:
+ Fix null pointer dereference due to uninitialized local_apics when using machine none
- Patch 5, 7:
+ These patches are added to follow the bios-tables-test instructions to commit the new changed IVRS.ivrs binary file

Version 9 changes,
- Patch 1:
+ Create apic_msr_read/write which is a small wrapper around apic_register_read/write that have additional x2apic mode check
- Patch 2:
+ Remove raise_exception_ra which is is TCG specific. Instead, return -1 and let the accelerator raise the appropriate exception
+ Refactor apic_get_delivery_bitmask a little bit to reduce line length
+ Move cpu_has_x2apic_feature and cpu_set_apic_feature from patch 3 to patch 2 so that patch 2 can be compiled without patch 3
- Patch 3:
+ set_base in APICCommonClass now returns an int to indicate error
+ Remove raise_exception_ra in apic_set base which is is TCG specific. Instead, return -1 and let the accelerator raise the appropriate exception

Version 8 changes,
- Patch 2, 4:
+ Rebase to master and resolve conflicts in these 2 patches

Version 7 changes,
- Patch 4:
+ If eim=on, keep checking if kvm x2APIC is enabled when kernel-irqchip is split

Version 6 changes,
- Patch 5:
+ Make all places use the amdvi_extended_feature_register to get extended feature register

Version 5 changes,
- Patch 3:
+ Rebase to master and fix conflict
- Patch 5:
+ Create a helper function to get amdvi extended feature register instead of storing it in AMDVIState

Version 4 changes,
- Patch 5:
+ Instead of replacing IVHD type 0x10 with type 0x11, export both types for backward compatibility with old guest operating system
+ Flip the xtsup feature check condition in amdvi_int_remap_ga for readability

Version 3 changes,
- Patch 2:
+ Allow APIC ID > 255 only when x2APIC feature is supported on CPU
+ Make physical destination mode IPI which has destination id 0xffffffff a broadcast to xAPIC CPUs
+ Make cluster address 0xf in cluster model of xAPIC logical destination mode a broadcast to all clusters
+ Create new extended_log_dest to store APIC_LDR information in x2APIC instead of extending log_dest for backward compatibility in vmstate

Version 2 changes,
- Add support for APIC ID larger than 255
- Adjust AMD iommu for x2APIC support
- Reorganize and split patch 1,2 into patch 1,2,3 in version 2

Thanks,
Quang Minh.

Bui Quang Minh (7):
i386/tcg: implement x2APIC registers MSR access
apic: add support for x2APIC mode
apic, i386/tcg: add x2apic transitions
intel_iommu: allow Extended Interrupt Mode when using userspace APIC
test: bios-tables-test: prepare IVRS change in ACPI table
amd_iommu: report x2APIC support to the operating system
test: bios-tables-test: add IVRS changed binary

 hw/i386/acpi-build.c                 | 129 +++++---
 hw/i386/amd_iommu.c                  |  29 +-
 hw/i386/amd_iommu.h                  |  16 +-
 hw/i386/intel_iommu.c                |   6 +-
 hw/i386/kvm/apic.c                   |   3 +-
 hw/i386/x86.c                        |   6 +-
 hw/i386/xen/xen_apic.c               |   3 +-
 hw/intc/apic.c                       | 473 +++++++++++++++++++++------
 hw/intc/apic_common.c                |  22 +-
 hw/intc/trace-events                 |   4 +-
 include/hw/i386/apic.h               |   8 +-
 include/hw/i386/apic_internal.h      |   9 +-
 target/i386/cpu-sysemu.c             |  18 +-
 target/i386/cpu.c                    |   9 +-
 target/i386/cpu.h                    |   9 +
 target/i386/tcg/sysemu/misc_helper.c |  41 ++-
 target/i386/whpx/whpx-apic.c         |   3 +-
 tests/data/acpi/q35/IVRS.ivrs        | Bin 104 -> 176 bytes
 18 files changed, 600 insertions(+), 188 deletions(-)
Signature · 1 line
2.25.1
#1 lore raw

PATCH v11 1/7 i386/tcg: implement x2APIC registers MSR access

This commit creates apic_register_read/write which are used by both apic_mem_read/write for MMIO access and apic_msr_read/write for MSR access.

The apic_msr_read/write returns -1 on error, accelerator can use this to raise the appropriate exception.

---
 hw/intc/apic.c                       | 122 ++++++++++++++++++++-------
 hw/intc/trace-events                 |   4 +-
 include/hw/i386/apic.h               |   3 +
 target/i386/cpu.h                    |   3 +
 target/i386/tcg/sysemu/misc_helper.c |  27 ++++++
 5 files changed, 127 insertions(+), 32 deletions(-)
hw/intc/apic.c +92 -30
index ac3d47d231..7a349c0723 100644
@@ -288,6 +288,13 @@ void apic_deliver_irq(uint8_t dest, uint8_t dest_mode, uint8_t delivery_mode,
     apic_bus_deliver(deliver_bitmask, delivery_mode, vector_num, trigger_mode);
 }
 
+bool is_x2apic_mode(DeviceState *dev)
+{
+    APICCommonState *s = APIC(dev);
+
+    return s->apicbase & MSR_IA32_APICBASE_EXTD;
+}
+
 static void apic_set_base(APICCommonState *s, uint64_t val)
 {
     s->apicbase = (val & 0xfffff000) |
@@ -636,24 +643,19 @@ static void apic_timer(void *opaque)
     apic_timer_update(s, s->next_time);
 }
 
-static uint64_t apic_mem_read(void *opaque, hwaddr addr, unsigned size)
+static int apic_register_read(int index, uint64_t *value)
 {
     DeviceState *dev;
     APICCommonState *s;
     uint32_t val;
-    int index;
-
-    if (size < 4) {
-        return 0;
-    }
+    int ret = 0;
 
     dev = cpu_get_current_apic();
     if (!dev) {
-        return 0;
+        return -1;
     }
     s = APIC(dev);
 
-    index = (addr >> 4) & 0xff;
     switch(index) {
     case 0x02: /* id */
         val = s->id << 24;
@@ -718,12 +720,46 @@ static uint64_t apic_mem_read(void *opaque, hwaddr addr, unsigned size)
     default:
         s->esr |= APIC_ESR_ILLEGAL_ADDRESS;
         val = 0;
+        ret = -1;
         break;
     }
-    trace_apic_mem_readl(addr, val);
+
+    trace_apic_register_read(index, val);
+    *value = val;
+    return ret;
+}
+
+static uint64_t apic_mem_read(void *opaque, hwaddr addr, unsigned size)
+{
+    uint64_t val;
+    int index;
+
+    if (size < 4) {
+        return 0;
+    }
+
+    index = (addr >> 4) & 0xff;
+    apic_register_read(index, &val);
+
     return val;
 }
 
+int apic_msr_read(int index, uint64_t *val)
+{
+    DeviceState *dev;
+
+    dev = cpu_get_current_apic();
+    if (!dev) {
+        return -1;
+    }
+
+    if (!is_x2apic_mode(dev)) {
+        return -1;
+    }
+
+    return apic_register_read(index, val);
+}
+
 static void apic_send_msi(MSIMessage *msi)
 {
     uint64_t addr = msi->address;
@@ -737,35 +773,18 @@ static void apic_send_msi(MSIMessage *msi)
     apic_deliver_irq(dest, dest_mode, delivery, vector, trigger_mode);
 }
 
-static void apic_mem_write(void *opaque, hwaddr addr, uint64_t val,
-                           unsigned size)
+static int apic_register_write(int index, uint64_t val)
 {
     DeviceState *dev;
     APICCommonState *s;
-    int index = (addr >> 4) & 0xff;
-
-    if (size < 4) {
-        return;
-    }
-
-    if (addr > 0xfff || !index) {
-        /* MSI and MMIO APIC are at the same memory location,
-         * but actually not on the global bus: MSI is on PCI bus
-         * APIC is connected directly to the CPU.
-         * Mapping them on the global bus happens to work because
-         * MSI registers are reserved in APIC MMIO and vice versa. */
-        MSIMessage msi = { .address = addr, .data = val };
-        apic_send_msi(&msi);
-        return;
-    }
 
     dev = cpu_get_current_apic();
     if (!dev) {
-        return;
+        return -1;
     }
     s = APIC(dev);
 
-    trace_apic_mem_writel(addr, val);
+    trace_apic_register_write(index, val);
 
     switch(index) {
     case 0x02:
@@ -839,8 +858,51 @@ static void apic_mem_write(void *opaque, hwaddr addr, uint64_t val,
         break;
     default:
         s->esr |= APIC_ESR_ILLEGAL_ADDRESS;
-        break;
+        return -1;
     }
+
+    return 0;
+}
+
+static void apic_mem_write(void *opaque, hwaddr addr, uint64_t val,
+                           unsigned size)
+{
+    int index = (addr >> 4) & 0xff;
+
+    if (size < 4) {
+        return;
+    }
+
+    if (addr > 0xfff || !index) {
+        /*
+         * MSI and MMIO APIC are at the same memory location,
+         * but actually not on the global bus: MSI is on PCI bus
+         * APIC is connected directly to the CPU.
+         * Mapping them on the global bus happens to work because
+         * MSI registers are reserved in APIC MMIO and vice versa.
+         */
+        MSIMessage msi = { .address = addr, .data = val };
+        apic_send_msi(&msi);
+        return;
+    }
+
+    apic_register_write(index, val);
+}
+
+int apic_msr_write(int index, uint64_t val)
+{
+    DeviceState *dev;
+
+    dev = cpu_get_current_apic();
+    if (!dev) {
+        return -1;
+    }
+
+    if (!is_x2apic_mode(dev)) {
+        return -1;
+    }
+
+    return apic_register_write(index, val);
 }
 
 static void apic_pre_save(APICCommonState *s)
hw/intc/trace-events +2 -2
index 36ff71f947..1ef29d0256 100644
@@ -14,8 +14,8 @@ cpu_get_apic_base(uint64_t val) "0x%016"PRIx64
 # apic.c
 apic_local_deliver(int vector, uint32_t lvt) "vector %d delivery mode %d"
 apic_deliver_irq(uint8_t dest, uint8_t dest_mode, uint8_t delivery_mode, uint8_t vector_num, uint8_t trigger_mode) "dest %d dest_mode %d delivery_mode %d vector %d trigger_mode %d"
-apic_mem_readl(uint64_t addr, uint32_t val)  "0x%"PRIx64" = 0x%08x"
-apic_mem_writel(uint64_t addr, uint32_t val) "0x%"PRIx64" = 0x%08x"
+apic_register_read(uint8_t reg, uint64_t val) "register 0x%02x = 0x%"PRIx64
+apic_register_write(uint8_t reg, uint64_t val) "register 0x%02x = 0x%"PRIx64
 
 # ioapic.c
 ioapic_set_remote_irr(int n) "set remote irr for pin %d"
include/hw/i386/apic.h +3 -0
index bdc15a7a73..ddea4213db 100644
@@ -18,6 +18,9 @@ void apic_sipi(DeviceState *s);
 void apic_poll_irq(DeviceState *d);
 void apic_designate_bsp(DeviceState *d, bool bsp);
 int apic_get_highest_priority_irr(DeviceState *dev);
+int apic_msr_read(int index, uint64_t *val);
+int apic_msr_write(int index, uint64_t val);
+bool is_x2apic_mode(DeviceState *d);
 
 /* pc.c */
 DeviceState *cpu_get_current_apic(void);
target/i386/cpu.h +3 -0
index ef987f344c..1064a2c67d 100644
@@ -545,6 +545,9 @@ typedef enum X86Seg {
 #define MSR_IA32_VMX_TRUE_ENTRY_CTLS     0x00000490
 #define MSR_IA32_VMX_VMFUNC             0x00000491
 
+#define MSR_APIC_START                  0x00000800
+#define MSR_APIC_END                    0x000008ff
+
 #define XSTATE_FP_BIT                   0
 #define XSTATE_SSE_BIT                  1
 #define XSTATE_YMM_BIT                  2
target/i386/tcg/sysemu/misc_helper.c +27 -0
index e1528b7f80..6fccdb3dca 100644
@@ -25,6 +25,7 @@
 #include "exec/address-spaces.h"
 #include "exec/exec-all.h"
 #include "tcg/helper-tcg.h"
+#include "hw/i386/apic.h"
 
 void helper_outb(CPUX86State *env, uint32_t port, uint32_t data)
 {
@@ -289,6 +290,19 @@ void helper_wrmsr(CPUX86State *env)
         env->msr_bndcfgs = val;
         cpu_sync_bndcs_hflags(env);
         break;
+    case MSR_APIC_START ... MSR_APIC_END: {
+        int ret;
+        int index = (uint32_t)env->regs[R_ECX] - MSR_APIC_START;
+
+        qemu_mutex_lock_iothread();
+        ret = apic_msr_write(index, val);
+        qemu_mutex_unlock_iothread();
+        if (ret < 0) {
+            goto error;
+        }
+
+        break;
+    }
     default:
         if ((uint32_t)env->regs[R_ECX] >= MSR_MC0_CTL
             && (uint32_t)env->regs[R_ECX] < MSR_MC0_CTL +
@@ -455,6 +469,19 @@ void helper_rdmsr(CPUX86State *env)
         val = (cs->nr_threads * cs->nr_cores) | (cs->nr_cores << 16);
         break;
     }
+    case MSR_APIC_START ... MSR_APIC_END: {
+        int ret;
+        int index = (uint32_t)env->regs[R_ECX] - MSR_APIC_START;
+
+        qemu_mutex_lock_iothread();
+        ret = apic_msr_read(index, &val);
+        qemu_mutex_unlock_iothread();
+        if (ret < 0) {
+            raise_exception_err_ra(env, EXCP0D_GPF, 0, GETPC());
+        }
+
+        break;
+    }
     default:
         if ((uint32_t)env->regs[R_ECX] >= MSR_MC0_CTL
             && (uint32_t)env->regs[R_ECX] < MSR_MC0_CTL +
Signature · 1 line
2.25.1
#2 lore raw

PATCH v11 2/7 apic: add support for x2APIC mode

In reply to Bui Quang Minh

This commit extends the APIC ID to 32-bit long and remove the 255 max APIC ID limit in userspace APIC. The array that manages local APICs is now dynamically allocated based on the max APIC ID of created x86 machine. Also, new x2APIC IPI destination determination scheme, self IPI and x2APIC mode register access are supported.

---
 hw/i386/x86.c                   |   6 +-
 hw/intc/apic.c                  | 289 ++++++++++++++++++++++++--------
 hw/intc/apic_common.c           |   9 +
 include/hw/i386/apic.h          |   3 +-
 include/hw/i386/apic_internal.h |   7 +-
 target/i386/cpu-sysemu.c        |  18 +-
 target/i386/cpu.h               |   2 +
 7 files changed, 259 insertions(+), 75 deletions(-)
hw/i386/x86.c +5 -1
index 2b6291ad8d..3d1bdd334e 100644
@@ -137,7 +137,7 @@ void x86_cpus_init(X86MachineState *x86ms, int default_cpu_version)
      * a literal `0` in configurations where kvm_* aren't defined)
      */
     if (kvm_enabled() && x86ms->apic_id_limit > 255 &&
-        (!kvm_irqchip_in_kernel() || !kvm_enable_x2apic())) {
+        kvm_irqchip_in_kernel() && !kvm_enable_x2apic()) {
         error_report("current -smp configuration requires kernel "
                      "irqchip and X2APIC API support.");
         exit(EXIT_FAILURE);
@@ -147,6 +147,10 @@ void x86_cpus_init(X86MachineState *x86ms, int default_cpu_version)
         kvm_set_max_apic_id(x86ms->apic_id_limit);
     }
 
+    if (!kvm_irqchip_in_kernel()) {
+        apic_set_max_apic_id(x86ms->apic_id_limit);
+    }
+
     possible_cpus = mc->possible_cpu_arch_ids(ms);
     for (i = 0; i < ms->smp.cpus; i++) {
         x86_cpu_new(x86ms, possible_cpus->cpus[i].arch_id, &error_fatal);
hw/intc/apic.c +220 -69

450 lines not on this page · raw message

hw/intc/apic_common.c +9 -0
index bccb4241c2..4bc3d2f149 100644
@@ -287,6 +287,10 @@ static void apic_common_realize(DeviceState *dev, Error **errp)
     }
     vmstate_register_with_alias_id(NULL, instance_id, &vmstate_apic_common,
                                    s, -1, 0, NULL);
+
+    /* APIC LDR in x2APIC mode */
+    s->extended_log_dest = ((s->initial_apic_id & 0xffff0) << 16) |
+                            (1 << (s->initial_apic_id & 0xf));
 }
 
 static void apic_common_unrealize(DeviceState *dev)
@@ -427,6 +431,11 @@ static void apic_common_set_id(Object *obj, Visitor *v, const char *name,
         return;
     }
 
+    if (value >= 255 && !cpu_has_x2apic_feature(&s->cpu->env)) {
+        error_setg(errp, "APIC ID %d requires x2APIC feature in CPU", value);
+        return;
+    }
+
     s->initial_apic_id = value;
     s->id = (uint8_t)value;
 }
include/hw/i386/apic.h +1 -2
index ddea4213db..c8ca41ab44 100644
@@ -3,8 +3,7 @@
 
 
 /* apic.c */
-void apic_deliver_irq(uint8_t dest, uint8_t dest_mode, uint8_t delivery_mode,
-                      uint8_t vector_num, uint8_t trigger_mode);
+void apic_set_max_apic_id(uint32_t max_apic_id);
 int apic_accept_pic_intr(DeviceState *s);
 void apic_deliver_pic_intr(DeviceState *s, int level);
 void apic_deliver_nmi(DeviceState *d);
include/hw/i386/apic_internal.h +5 -2
index 5f2ba24bfc..e796e6cae3 100644
@@ -46,8 +46,10 @@
 #define APIC_DM_EXTINT                  7
 
 /* APIC destination mode */
-#define APIC_DESTMODE_FLAT              0xf
-#define APIC_DESTMODE_CLUSTER           1
+#define APIC_DESTMODE_PHYSICAL          0
+#define APIC_DESTMODE_LOGICAL           1
+#define APIC_DESTMODE_LOGICAL_FLAT      0xf
+#define APIC_DESTMODE_LOGICAL_CLUSTER   0
 
 #define APIC_TRIGGER_EDGE               0
 #define APIC_TRIGGER_LEVEL              1
@@ -187,6 +189,7 @@ struct APICCommonState {
     DeviceState *vapic;
     hwaddr vapic_paddr; /* note: persistence via kvmvapic */
     bool legacy_instance_id;
+    uint32_t extended_log_dest;
 };
 
 typedef struct VAPICState {
target/i386/cpu-sysemu.c +17 -1
index 2375e48178..7422096737 100644
@@ -235,6 +235,16 @@ void cpu_clear_apic_feature(CPUX86State *env)
     env->features[FEAT_1_EDX] &= ~CPUID_APIC;
 }
 
+void cpu_set_apic_feature(CPUX86State *env)
+{
+    env->features[FEAT_1_EDX] |= CPUID_APIC;
+}
+
+bool cpu_has_x2apic_feature(CPUX86State *env)
+{
+    return env->features[FEAT_1_ECX] & CPUID_EXT_X2APIC;
+}
+
 bool cpu_is_bsp(X86CPU *cpu)
 {
     return cpu_get_apic_base(cpu->apic_state) & MSR_IA32_APICBASE_BSP;
@@ -281,11 +291,17 @@ void x86_cpu_apic_create(X86CPU *cpu, Error **errp)
                               OBJECT(cpu->apic_state));
     object_unref(OBJECT(cpu->apic_state));
 
-    qdev_prop_set_uint32(cpu->apic_state, "id", cpu->apic_id);
     /* TODO: convert to link<> */
     apic = APIC_COMMON(cpu->apic_state);
     apic->cpu = cpu;
     apic->apicbase = APIC_DEFAULT_ADDRESS | MSR_IA32_APICBASE_ENABLE;
+
+    /*
+     * apic_common_set_id needs to check if the CPU has x2APIC
+     * feature in case APIC ID >= 255, so we need to set apic->cpu
+     * before setting APIC ID
+     */
+    qdev_prop_set_uint32(cpu->apic_state, "id", cpu->apic_id);
 }
 
 void x86_cpu_apic_realize(X86CPU *cpu, Error **errp)
target/i386/cpu.h +2 -0
index 1064a2c67d..0fd4cab0af 100644
@@ -2238,8 +2238,10 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,
                    uint32_t *eax, uint32_t *ebx,
                    uint32_t *ecx, uint32_t *edx);
 void cpu_clear_apic_feature(CPUX86State *env);
+void cpu_set_apic_feature(CPUX86State *env);
 void host_cpuid(uint32_t function, uint32_t count,
                 uint32_t *eax, uint32_t *ebx, uint32_t *ecx, uint32_t *edx);
+bool cpu_has_x2apic_feature(CPUX86State *env);
 
 /* helper.c */
 void x86_cpu_set_a20(X86CPU *cpu, int a20_state);
Signature · 1 line
2.25.1
#3 lore raw

PATCH v11 3/7 apic, i386/tcg: add x2apic transitions

In reply to Bui Quang Minh

This commit adds support for x2APIC transitions when writing to MSR_IA32_APICBASE register and finally adds CPUID_EXT_X2APIC to TCG_EXT_FEATURES.

The set_base in APICCommonClass now returns an integer to indicate error in execution. apic_set_base return -1 on invalid APIC state transition, accelerator can use this to raise appropriate exception.

---
 hw/i386/kvm/apic.c                   |  3 +-
 hw/i386/xen/xen_apic.c               |  3 +-
 hw/intc/apic.c                       | 62 +++++++++++++++++++++++++++-
 hw/intc/apic_common.c                | 13 +++---
 include/hw/i386/apic.h               |  2 +-
 include/hw/i386/apic_internal.h      |  2 +-
 target/i386/cpu.c                    |  9 ++--
 target/i386/cpu.h                    |  4 ++
 target/i386/tcg/sysemu/misc_helper.c | 14 ++++++-
 target/i386/whpx/whpx-apic.c         |  3 +-
 10 files changed, 96 insertions(+), 19 deletions(-)
hw/i386/kvm/apic.c +2 -1
index 1e89ca0899..a72c28e8a7 100644
@@ -95,9 +95,10 @@ void kvm_get_apic_state(DeviceState *dev, struct kvm_lapic_state *kapic)
     apic_next_timer(s, s->initial_count_load_time);
 }
 
-static void kvm_apic_set_base(APICCommonState *s, uint64_t val)
+static int kvm_apic_set_base(APICCommonState *s, uint64_t val)
 {
     s->apicbase = val;
+    return 0;
 }
 
 static void kvm_apic_set_tpr(APICCommonState *s, uint8_t val)
hw/i386/xen/xen_apic.c +2 -1
index 7c7a60b166..101e16a766 100644
@@ -49,8 +49,9 @@ static void xen_apic_realize(DeviceState *dev, Error **errp)
     msi_nonbroken = true;
 }
 
-static void xen_apic_set_base(APICCommonState *s, uint64_t val)
+static int xen_apic_set_base(APICCommonState *s, uint64_t val)
 {
+    return 0;
 }
 
 static void xen_apic_set_tpr(APICCommonState *s, uint8_t val)
hw/intc/apic.c +61 -1

75 lines not on this page · raw message

hw/intc/apic_common.c +6 -7
index 4bc3d2f149..b13a7b0457 100644
@@ -35,20 +35,19 @@
 
 bool apic_report_tpr_access;
 
-void cpu_set_apic_base(DeviceState *dev, uint64_t val)
+int cpu_set_apic_base(DeviceState *dev, uint64_t val)
 {
     trace_cpu_set_apic_base(val);
 
     if (dev) {
         APICCommonState *s = APIC_COMMON(dev);
         APICCommonClass *info = APIC_COMMON_GET_CLASS(s);
-        /* switching to x2APIC, reset possibly modified xAPIC ID */
-        if (!(s->apicbase & MSR_IA32_APICBASE_EXTD) &&
-            (val & MSR_IA32_APICBASE_EXTD)) {
-            s->id = s->initial_apic_id;
-        }
-        info->set_base(s, val);
+        /* Reset possibly modified xAPIC ID */
+        s->id = s->initial_apic_id;
+        return info->set_base(s, val);
     }
+
+    return 0;
 }
 
 uint64_t cpu_get_apic_base(DeviceState *dev)
include/hw/i386/apic.h +1 -1
index c8ca41ab44..f6e7489f2d 100644
@@ -8,7 +8,7 @@ int apic_accept_pic_intr(DeviceState *s);
 void apic_deliver_pic_intr(DeviceState *s, int level);
 void apic_deliver_nmi(DeviceState *d);
 int apic_get_interrupt(DeviceState *s);
-void cpu_set_apic_base(DeviceState *s, uint64_t val);
+int cpu_set_apic_base(DeviceState *s, uint64_t val);
 uint64_t cpu_get_apic_base(DeviceState *s);
 void cpu_set_apic_tpr(DeviceState *s, uint8_t val);
 uint8_t cpu_get_apic_tpr(DeviceState *s);
include/hw/i386/apic_internal.h +1 -1
index e796e6cae3..d6e85833da 100644
@@ -137,7 +137,7 @@ struct APICCommonClass {
 
     DeviceRealize realize;
     DeviceUnrealize unrealize;
-    void (*set_base)(APICCommonState *s, uint64_t val);
+    int (*set_base)(APICCommonState *s, uint64_t val);
     void (*set_tpr)(APICCommonState *s, uint8_t val);
     uint8_t (*get_tpr)(APICCommonState *s);
     void (*enable_tpr_reporting)(APICCommonState *s, bool enable);
target/i386/cpu.c +5 -4
index 95d5f16cd5..b38fc68dfa 100644
@@ -631,8 +631,8 @@ void x86_cpu_vendor_words2str(char *dst, uint32_t vendor1,
  * in CPL=3; remove them if they are ever implemented for system emulation.
  */
 #if defined CONFIG_USER_ONLY
-#define CPUID_EXT_KERNEL_FEATURES (CPUID_EXT_PCID | CPUID_EXT_TSC_DEADLINE_TIMER | \
-                                 CPUID_EXT_X2APIC)
+#define CPUID_EXT_KERNEL_FEATURES \
+          (CPUID_EXT_PCID | CPUID_EXT_TSC_DEADLINE_TIMER)
 #else
 #define CPUID_EXT_KERNEL_FEATURES 0
 #endif
@@ -642,12 +642,13 @@ void x86_cpu_vendor_words2str(char *dst, uint32_t vendor1,
           CPUID_EXT_XSAVE | /* CPUID_EXT_OSXSAVE is dynamic */   \
           CPUID_EXT_MOVBE | CPUID_EXT_AES | CPUID_EXT_HYPERVISOR | \
           CPUID_EXT_RDRAND | CPUID_EXT_AVX | CPUID_EXT_F16C | \
-          CPUID_EXT_FMA | CPUID_EXT_KERNEL_FEATURES)
+          CPUID_EXT_FMA | CPUID_EXT_X2APIC | CPUID_EXT_KERNEL_FEATURES)
           /* missing:
           CPUID_EXT_DTES64, CPUID_EXT_DSCPL, CPUID_EXT_VMX, CPUID_EXT_SMX,
           CPUID_EXT_EST, CPUID_EXT_TM2, CPUID_EXT_CID,
           CPUID_EXT_XTPR, CPUID_EXT_PDCM, CPUID_EXT_PCID, CPUID_EXT_DCA,
-          CPUID_EXT_X2APIC, CPUID_EXT_TSC_DEADLINE_TIMER */
+          CPUID_EXT_TSC_DEADLINE_TIMER
+          */
 
 #ifdef TARGET_X86_64
 #define TCG_EXT2_X86_64_FEATURES CPUID_EXT2_LM
target/i386/cpu.h +4 -0
index 0fd4cab0af..820fa7edf8 100644
@@ -379,6 +379,10 @@ typedef enum X86Seg {
 #define MSR_IA32_APICBASE_ENABLE        (1<<11)
 #define MSR_IA32_APICBASE_EXTD          (1 << 10)
 #define MSR_IA32_APICBASE_BASE          (0xfffffU<<12)
+#define MSR_IA32_APICBASE_RESERVED \
+        (~(uint64_t)(MSR_IA32_APICBASE_BSP | MSR_IA32_APICBASE_ENABLE \
+                     | MSR_IA32_APICBASE_EXTD | MSR_IA32_APICBASE_BASE))
+
 #define MSR_IA32_FEATURE_CONTROL        0x0000003a
 #define MSR_TSC_ADJUST                  0x0000003b
 #define MSR_IA32_SPEC_CTRL              0x48
target/i386/tcg/sysemu/misc_helper.c +12 -2
index 6fccdb3dca..29260d657d 100644
@@ -158,9 +158,19 @@ void helper_wrmsr(CPUX86State *env)
     case MSR_IA32_SYSENTER_EIP:
         env->sysenter_eip = val;
         break;
-    case MSR_IA32_APICBASE:
-        cpu_set_apic_base(env_archcpu(env)->apic_state, val);
+    case MSR_IA32_APICBASE: {
+        int ret;
+
+        if (val & MSR_IA32_APICBASE_RESERVED) {
+            goto error;
+        }
+
+        ret = cpu_set_apic_base(env_archcpu(env)->apic_state, val);
+        if (ret < 0) {
+            goto error;
+        }
         break;
+    }
     case MSR_EFER:
         {
             uint64_t update_mask;
target/i386/whpx/whpx-apic.c +2 -1
index 8710e37567..7e14ded978 100644
@@ -90,9 +90,10 @@ static void whpx_get_apic_state(APICCommonState *s,
     apic_next_timer(s, s->initial_count_load_time);
 }
 
-static void whpx_apic_set_base(APICCommonState *s, uint64_t val)
+static int whpx_apic_set_base(APICCommonState *s, uint64_t val)
 {
     s->apicbase = val;
+    return 0;
 }
 
 static void whpx_put_apic_base(CPUState *cpu, uint64_t val)
Signature · 1 line
2.25.1
#4 lore raw

PATCH v11 4/7 intel_iommu: allow Extended Interrupt Mode when using userspace APIC

In reply to Bui Quang Minh

As userspace APIC now supports x2APIC, intel interrupt remapping hardware can be set to EIM mode when userspace local APIC is used.

---
 hw/i386/intel_iommu.c | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)
hw/i386/intel_iommu.c +1 -5
index 5085a6fee3..cb6ce4a646 100644
@@ -4124,11 +4124,7 @@ static bool vtd_decide_config(IntelIOMMUState *s, Error **errp)
                                               ON_OFF_AUTO_ON : ON_OFF_AUTO_OFF;
     }
     if (s->intr_eim == ON_OFF_AUTO_ON && !s->buggy_eim) {
-        if (!kvm_irqchip_is_split()) {
-            error_setg(errp, "eim=on requires accel=kvm,kernel-irqchip=split");
-            return false;
-        }
-        if (kvm_enabled() && !kvm_enable_x2apic()) {
+        if (kvm_irqchip_is_split() && !kvm_enable_x2apic()) {
             error_setg(errp, "eim=on requires support on the KVM side"
                              "(X2APIC_API, first shipped in v4.7)");
             return false;
Signature · 1 line
2.25.1
#5 lore raw

PATCH v11 5/7 test: bios-tables-test: prepare IVRS change in ACPI table

In reply to Bui Quang Minh

Following the instructions in bios-tables-test, this lists that IVRS.ivrs in ACPI table will be changed to add new IVHD type 0x11.

---
 tests/qtest/bios-tables-test-allowed-diff.h | 1 +
 1 file changed, 1 insertion(+)
tests/qtest/bios-tables-test-allowed-diff.h +1 -0
index dfb8523c8b..ac420db6b7 100644
@@ -1 +1,2 @@
 /* List of comma-separated changed AML files to ignore */
+"tests/data/acpi/q35/IVRS.ivrs",
Signature · 1 line
2.25.1
#6 lore raw

PATCH v11 6/7 amd_iommu: report x2APIC support to the operating system

In reply to Bui Quang Minh

This commit adds XTSup configuration to let user choose to whether enable this feature or not. When XTSup is enabled, additional bytes in IRTE with enabled guest virtual VAPIC are used to support 32-bit destination id.

Additionally, this commit exports IVHD type 0x11 besides the old IVHD type 0x10 in ACPI table. IVHD type 0x10 does not report full set of IOMMU features only the legacy ones, so operating system (e.g. Linux) may only detects x2APIC support if IVHD type 0x11 is available. The IVHD type 0x10 is kept so that old operating system that only parses type 0x10 can detect the IOMMU device.

---
 hw/i386/acpi-build.c | 129 +++++++++++++++++++++++++++----------------
 hw/i386/amd_iommu.c  |  29 +++++++++-
 hw/i386/amd_iommu.h  |  16 ++++--
 3 files changed, 117 insertions(+), 57 deletions(-)
hw/i386/acpi-build.c +80 -49
index 80db183b78..f17fb63be3 100644
@@ -2333,30 +2333,23 @@ static void
 build_amd_iommu(GArray *table_data, BIOSLinker *linker, const char *oem_id,
                 const char *oem_table_id)
 {
-    int ivhd_table_len = 24;
     AMDVIState *s = AMD_IOMMU_DEVICE(x86_iommu_get_default());
     GArray *ivhd_blob = g_array_new(false, true, 1);
     AcpiTable table = { .sig = "IVRS", .rev = 1, .oem_id = oem_id,
                         .oem_table_id = oem_table_id };
+    uint64_t feature_report;
 
     acpi_table_begin(&table, table_data);
     /* IVinfo - IO virtualization information common to all
      * IOMMU units in a system
      */
-    build_append_int_noprefix(table_data, 40UL << 8/* PASize */, 4);
+    build_append_int_noprefix(table_data,
+                             (1UL << 0) | /* EFRSup */
+                             (40UL << 8), /* PASize */
+                             4);
     /* reserved */
     build_append_int_noprefix(table_data, 0, 8);
 
-    /* IVHD definition - type 10h */
-    build_append_int_noprefix(table_data, 0x10, 1);
-    /* virtualization flags */
-    build_append_int_noprefix(table_data,
-                             (1UL << 0) | /* HtTunEn      */
-                             (1UL << 4) | /* iotblSup     */
-                             (1UL << 6) | /* PrefSup      */
-                             (1UL << 7),  /* PPRSup       */
-                             1);
-
     /*
      * A PCI bus walk, for each PCI host bridge, is necessary to create a
      * complete set of IVHD entries.  Do this into a separate blob so that we
@@ -2376,56 +2369,94 @@ build_amd_iommu(GArray *table_data, BIOSLinker *linker, const char *oem_id,
         build_append_int_noprefix(ivhd_blob, 0x0000001, 4);
     }
 
-    ivhd_table_len += ivhd_blob->len;
-
     /*
      * When interrupt remapping is supported, we add a special IVHD device
-     * for type IO-APIC.
-     */
-    if (x86_iommu_ir_supported(x86_iommu_get_default())) {
-        ivhd_table_len += 8;
-    }
-
-    /* IVHD length */
-    build_append_int_noprefix(table_data, ivhd_table_len, 2);
-    /* DeviceID */
-    build_append_int_noprefix(table_data,
-                              object_property_get_int(OBJECT(&s->pci), "addr",
-                                                      &error_abort), 2);
-    /* Capability offset */
-    build_append_int_noprefix(table_data, s->pci.capab_offset, 2);
-    /* IOMMU base address */
-    build_append_int_noprefix(table_data, s->mmio.addr, 8);
-    /* PCI Segment Group */
-    build_append_int_noprefix(table_data, 0, 2);
-    /* IOMMU info */
-    build_append_int_noprefix(table_data, 0, 2);
-    /* IOMMU Feature Reporting */
-    build_append_int_noprefix(table_data,
-                             (48UL << 30) | /* HATS   */
-                             (48UL << 28) | /* GATS   */
-                             (1UL << 2)   | /* GTSup  */
-                             (1UL << 6),    /* GASup  */
-                             4);
-
-    /* IVHD entries as found above */
-    g_array_append_vals(table_data, ivhd_blob->data, ivhd_blob->len);
-    g_array_free(ivhd_blob, TRUE);
-
-    /*
-     * Add a special IVHD device type.
+     * for type IO-APIC
      * Refer to spec - Table 95: IVHD device entry type codes
      *
      * Linux IOMMU driver checks for the special IVHD device (type IO-APIC).
      * See Linux kernel commit 'c2ff5cf5294bcbd7fa50f7d860e90a66db7e5059'
      */
     if (x86_iommu_ir_supported(x86_iommu_get_default())) {
-        build_append_int_noprefix(table_data,
+        build_append_int_noprefix(ivhd_blob,
                                  (0x1ull << 56) |           /* type IOAPIC */
                                  (IOAPIC_SB_DEVID << 40) |  /* IOAPIC devid */
                                  0x48,                      /* special device */
                                  8);
     }
+
+    /* IVHD definition - type 10h */
+    build_append_int_noprefix(table_data, 0x10, 1);
+    /* virtualization flags */
+    build_append_int_noprefix(table_data,
+                             (1UL << 0) | /* HtTunEn      */
+                             (1UL << 4) | /* iotblSup     */
+                             (1UL << 6) | /* PrefSup      */
+                             (1UL << 7),  /* PPRSup       */
+                             1);
+
+    /* IVHD length */
+    build_append_int_noprefix(table_data, ivhd_blob->len + 24, 2);
+    /* DeviceID */
+    build_append_int_noprefix(table_data,
+                              object_property_get_int(OBJECT(&s->pci), "addr",
+                                                      &error_abort), 2);
+    /* Capability offset */
+    build_append_int_noprefix(table_data, s->pci.capab_offset, 2);
+    /* IOMMU base address */
+    build_append_int_noprefix(table_data, s->mmio.addr, 8);
+    /* PCI Segment Group */
+    build_append_int_noprefix(table_data, 0, 2);
+    /* IOMMU info */
+    build_append_int_noprefix(table_data, 0, 2);
+    /* IOMMU Feature Reporting */
+    feature_report = (48UL << 30) | /* HATS   */
+                     (48UL << 28) | /* GATS   */
+                     (1UL << 2)   | /* GTSup  */
+                     (1UL << 6);    /* GASup  */
+    if (s->xtsup) {
+        feature_report |= (1UL << 0); /* XTSup */
+    }
+    build_append_int_noprefix(table_data, feature_report, 4);
+
+    /* IVHD entries as found above */
+    g_array_append_vals(table_data, ivhd_blob->data, ivhd_blob->len);
+
+   /* IVHD definition - type 11h */
+    build_append_int_noprefix(table_data, 0x11, 1);
+    /* virtualization flags */
+    build_append_int_noprefix(table_data,
+                             (1UL << 0) | /* HtTunEn      */
+                             (1UL << 4),  /* iotblSup     */
+                             1);
+
+    /* IVHD length */
+    build_append_int_noprefix(table_data, ivhd_blob->len + 40, 2);
+    /* DeviceID */
+    build_append_int_noprefix(table_data,
+                              object_property_get_int(OBJECT(&s->pci), "addr",
+                                                      &error_abort), 2);
+    /* Capability offset */
+    build_append_int_noprefix(table_data, s->pci.capab_offset, 2);
+    /* IOMMU base address */
+    build_append_int_noprefix(table_data, s->mmio.addr, 8);
+    /* PCI Segment Group */
+    build_append_int_noprefix(table_data, 0, 2);
+    /* IOMMU info */
+    build_append_int_noprefix(table_data, 0, 2);
+    /* IOMMU Attributes */
+    build_append_int_noprefix(table_data, 0, 4);
+    /* EFR Register Image */
+    build_append_int_noprefix(table_data,
+                              amdvi_extended_feature_register(s),
+                              8);
+    /* EFR Register Image 2 */
+    build_append_int_noprefix(table_data, 0, 8);
+
+    /* IVHD entries as found above */
+    g_array_append_vals(table_data, ivhd_blob->data, ivhd_blob->len);
+
+    g_array_free(ivhd_blob, TRUE);
     acpi_table_end(linker, &table);
 }
 
hw/i386/amd_iommu.c +26 -3
index 4203144da9..7329553ad3 100644
@@ -31,6 +31,7 @@
 #include "hw/i386/apic_internal.h"
 #include "trace.h"
 #include "hw/i386/apic-msidef.h"
+#include "hw/qdev-properties.h"
 
 /* used AMD-Vi MMIO registers */
 const char *amdvi_mmio_low[] = {
@@ -74,6 +75,16 @@ typedef struct AMDVIIOTLBEntry {
     uint64_t page_mask;         /* physical page size  */
 } AMDVIIOTLBEntry;
 
+uint64_t amdvi_extended_feature_register(AMDVIState *s)
+{
+    uint64_t feature = AMDVI_DEFAULT_EXT_FEATURES;
+    if (s->xtsup) {
+        feature |= AMDVI_FEATURE_XT;
+    }
+
+    return feature;
+}
+
 /* configure MMIO registers at startup/reset */
 static void amdvi_set_quad(AMDVIState *s, hwaddr addr, uint64_t val,
                            uint64_t romask, uint64_t w1cmask)
@@ -1155,7 +1166,12 @@ static int amdvi_int_remap_ga(AMDVIState *iommu,
     irq->vector = irte.hi.fields.vector;
     irq->dest_mode = irte.lo.fields_remap.dm;
     irq->redir_hint = irte.lo.fields_remap.rq_eoi;
-    irq->dest = irte.lo.fields_remap.destination;
+    if (iommu->xtsup) {
+        irq->dest = irte.lo.fields_remap.destination |
+                    (irte.hi.fields.destination_hi << 24);
+    } else {
+        irq->dest = irte.lo.fields_remap.destination & 0xff;
+    }
 
     return 0;
 }
@@ -1505,8 +1521,9 @@ static void amdvi_init(AMDVIState *s)
 
     /* reset MMIO */
     memset(s->mmior, 0, AMDVI_MMIO_SIZE);
-    amdvi_set_quad(s, AMDVI_MMIO_EXT_FEATURES, AMDVI_EXT_FEATURES,
-            0xffffffffffffffef, 0);
+    amdvi_set_quad(s, AMDVI_MMIO_EXT_FEATURES,
+                   amdvi_extended_feature_register(s),
+                   0xffffffffffffffef, 0);
     amdvi_set_quad(s, AMDVI_MMIO_STATUS, 0, 0x98, 0x67);
 }
 
@@ -1589,6 +1606,11 @@ static void amdvi_sysbus_realize(DeviceState *dev, Error **errp)
     amdvi_init(s);
 }
 
+static Property amdvi_properties[] = {
+    DEFINE_PROP_BOOL("xtsup", AMDVIState, xtsup, false),
+    DEFINE_PROP_END_OF_LIST(),
+};
+
 static const VMStateDescription vmstate_amdvi_sysbus = {
     .name = "amd-iommu",
     .unmigratable = 1
@@ -1615,6 +1637,7 @@ static void amdvi_sysbus_class_init(ObjectClass *klass, void *data)
     dc->user_creatable = true;
     set_bit(DEVICE_CATEGORY_MISC, dc->categories);
     dc->desc = "AMD IOMMU (AMD-Vi) DMA Remapping device";
+    device_class_set_props(dc, amdvi_properties);
 }
 
 static const TypeInfo amdvi_sysbus = {
hw/i386/amd_iommu.h +11 -5
index c5065a3e27..73619fe9ea 100644
@@ -154,6 +154,7 @@
 
 #define AMDVI_FEATURE_PREFETCH            (1ULL << 0) /* page prefetch       */
 #define AMDVI_FEATURE_PPR                 (1ULL << 1) /* PPR Support         */
+#define AMDVI_FEATURE_XT                  (1ULL << 2) /* x2APIC Support      */
 #define AMDVI_FEATURE_GT                  (1ULL << 4) /* Guest Translation   */
 #define AMDVI_FEATURE_IA                  (1ULL << 6) /* inval all support   */
 #define AMDVI_FEATURE_GA                  (1ULL << 7) /* guest VAPIC support */
@@ -173,8 +174,9 @@
 #define AMDVI_IOTLB_MAX_SIZE 1024
 #define AMDVI_DEVID_SHIFT    36
 
-/* extended feature support */
-#define AMDVI_EXT_FEATURES (AMDVI_FEATURE_PREFETCH | AMDVI_FEATURE_PPR | \
+/* default extended feature */
+#define AMDVI_DEFAULT_EXT_FEATURES \
+        (AMDVI_FEATURE_PREFETCH | AMDVI_FEATURE_PPR | \
         AMDVI_FEATURE_IA | AMDVI_FEATURE_GT | AMDVI_FEATURE_HE | \
         AMDVI_GATS_MODE | AMDVI_HATS_MODE | AMDVI_FEATURE_GA)
 
@@ -276,8 +278,8 @@ union irte_ga_lo {
                 dm:1,
                 /* ------ */
                 guest_mode:1,
-                destination:8,
-                rsvd_1:48;
+                destination:24,
+                rsvd_1:32;
   } fields_remap;
 };
 
@@ -285,7 +287,8 @@ union irte_ga_hi {
   uint64_t val;
   struct {
       uint64_t  vector:8,
-                rsvd_2:56;
+                rsvd_2:48,
+                destination_hi:8;
   } fields;
 };
 
@@ -364,6 +367,9 @@ struct AMDVIState {
 
     /* Interrupt remapping */
     bool ga_enabled;
+    bool xtsup;
 };
 
+uint64_t amdvi_extended_feature_register(AMDVIState *s);
+
 #endif
Signature · 1 line
2.25.1
#7 lore raw

PATCH v11 7/7 test: bios-tables-test: add IVRS changed binary

In reply to Bui Quang Minh

Following the instructions in bios-tables-test, this adds the changed IVRS.ivrs binary.

New IVRS differs in length, checksum, it enables EFRSup in Virtualization Info and adds IVHD type 0x11 with the same device entries as in IVHD type 0x10.

ASL diff:

/*
* Intel ACPI Component Architecture
* AML/ASL+ Disassembler version 20230628 (64-bit version)
* Copyright (c) 2000 - 2023 Intel Corporation
*
- * Disassembly of tests/data/acpi/q35/IVRS.ivrs, Wed Nov 8 21:39:58 2023
+ * Disassembly of /tmp/aml-2ODND2, Wed Nov 8 21:39:58 2023
*
* ACPI Data Table [IVRS]
*
* Format: [HexOffset DecimalOffset ByteLength] FieldName : FieldValue (in hex) */

[000h 0000 004h] Signature : "IVRS" [I/O Virtualization Reporting Structure]
-[004h 0004 004h] Table Length : 00000068
+[004h 0004 004h] Table Length : 000000B0

 [008h 0008 001h]                    Revision : 01
-[009h 0009 001h]                    Checksum : 43
+[009h 0009 001h]                    Checksum : 74

[00Ah 0010 006h] Oem ID : "BOCHS "
[010h 0016 008h] Oem Table ID : "BXPC "
[018h 0024 004h] Oem Revision : 00000001
[01Ch 0028 004h] Asl Compiler ID : "BXPC"
[020h 0032 004h] Asl Compiler Revision : 00000001

-[024h 0036 004h] Virtualization Info : 00002800
+[024h 0036 004h] Virtualization Info : 00002801
[028h 0040 008h] Reserved : 0000000000000000

[030h 0048 001h] Subtable Type : 10 [Hardware Definition Block (IVHD)]
[031h 0049 001h] Flags (decoded below) : D1
HtTunEn : 1
PassPW : 0
ResPassPW : 0
Isoc Control : 0
Iotlb Support : 1
Coherent : 0
Prefetch Support : 1
PPR Support : 1
[032h 0050 002h] Length : 0038
[034h 0052 002h] DeviceId : 0010
[036h 0054 002h] Capability Offset : 0040
[038h 0056 008h] Base Address : 00000000FED80000
@@ -108,25 +108,129 @@
LINT1 Pass : 0

[060h 0096 001h] Subtable Type : 48 [Device Entry: Special Device]
[061h 0097 002h] Device ID : 0000
[063h 0099 001h] Data Setting (decoded below) : 00
INITPass : 0 EIntPass : 0
NMIPass : 0
Reserved : 0
System MGMT : 0
LINT0 Pass : 0 LINT1 Pass : 0
[064h 0100 001h] Handle : 00
[065h 0101 002h] Source Used Device ID : 00A0
[067h 0103 001h] Variety : 01

-Raw Table Data: Length 104 (0x68)
+[068h 0104 001h] Subtable Type : 11 [Hardware Definition Block (IVHD)]
+[069h 0105 001h] Flags (decoded below) : 11
+ HtTunEn : 1
+ PassPW : 0
+ ResPassPW : 0
+ Isoc Control : 0
+ Iotlb Support : 1
+ Coherent : 0
+ Prefetch Support : 0
+ PPR Support : 0
+[06Ah 0106 002h] Length : 0048
+[06Ch 0108 002h] DeviceId : 0010
+[06Eh 0110 002h] Capability Offset : 0040
+[070h 0112 008h] Base Address : 00000000FED80000
+[078h 0120 002h] PCI Segment Group : 0000
+[07Ah 0122 002h] Virtualization Info : 0000
+[07Ch 0124 004h] Attributes : 00000000
+[080h 0128 008h] EFR Image : 00000000000029D3
+[088h 0136 008h] Reserved : 0000000000000000
+
+[090h 0144 001h] Subtable Type : 02 [Device Entry: Select One Device]
+[091h 0145 002h] Device ID : 0000
+[093h 0147 001h] Data Setting (decoded below) : 00
+ INITPass : 0
+ EIntPass : 0
+ NMIPass : 0
+ Reserved : 0
+ System MGMT : 0
+ LINT0 Pass : 0
+ LINT1 Pass : 0
+
+[094h 0148 001h] Subtable Type : 02 [Device Entry: Select One Device]
+[095h 0149 002h] Device ID : 0008
+[097h 0151 001h] Data Setting (decoded below) : 00
+ INITPass : 0
+ EIntPass : 0
+ NMIPass : 0
+ Reserved : 0
+ System MGMT : 0
+ LINT0 Pass : 0
+ LINT1 Pass : 0
+
+[098h 0152 001h] Subtable Type : 02 [Device Entry: Select One Device]
+[099h 0153 002h] Device ID : 0010
+[09Bh 0155 001h] Data Setting (decoded below) : 00
+ INITPass : 0
+ EIntPass : 0
+ NMIPass : 0
+ Reserved : 0
+ System MGMT : 0
+ LINT0 Pass : 0
+ LINT1 Pass : 0
+
+[09Ch 0156 001h] Subtable Type : 02 [Device Entry: Select One Device]
+[09Dh 0157 002h] Device ID : 00F8
+[09Fh 0159 001h] Data Setting (decoded below) : 00
+ INITPass : 0
+ EIntPass : 0
+ NMIPass : 0
+ Reserved : 0
+ System MGMT : 0
+ LINT0 Pass : 0
+ LINT1 Pass : 0
+
+[0A0h 0160 001h] Subtable Type : 02 [Device Entry: Select One Device]
+[0A1h 0161 002h] Device ID : 00FA
+[0A3h 0163 001h] Data Setting (decoded below) : 00
+ INITPass : 0
+ EIntPass : 0
+ NMIPass : 0
+ Reserved : 0
+ System MGMT : 0
+ LINT0 Pass : 0
+ LINT1 Pass : 0
+
+[0A4h 0164 001h] Subtable Type : 02 [Device Entry: Select One Device]
+[0A5h 0165 002h] Device ID : 00FB
+[0A7h 0167 001h] Data Setting (decoded below) : 00
+ INITPass : 0
+ EIntPass : 0
+ NMIPass : 0
+ Reserved : 0
+ System MGMT : 0
+ LINT0 Pass : 0
+ LINT1 Pass : 0
+
+[0A8h 0168 001h] Subtable Type : 48 [Device Entry: Special Device]
+[0A9h 0169 002h] Device ID : 0000
+[0ABh 0171 001h] Data Setting (decoded below) : 00
+ INITPass : 0
+ EIntPass : 0
+ NMIPass : 0
+ Reserved : 0
+ System MGMT : 0
+ LINT0 Pass : 0
+ LINT1 Pass : 0
+[0ACh 0172 001h] Handle : 00
+[0ADh 0173 002h] Source Used Device ID : 00A0
+[0AFh 0175 001h] Variety : 01
+
+Raw Table Data: Length 176 (0xB0)

-    0000: 49 56 52 53 68 00 00 00 01 43 42 4F 43 48 53 20  // IVRSh....CBOCHS
+    0000: 49 56 52 53 B0 00 00 00 01 74 42 4F 43 48 53 20  // IVRS.....tBOCHS
     0010: 42 58 50 43 20 20 20 20 01 00 00 00 42 58 50 43  // BXPC    ....BXPC
-    0020: 01 00 00 00 00 28 00 00 00 00 00 00 00 00 00 00  // .....(..........
+    0020: 01 00 00 00 01 28 00 00 00 00 00 00 00 00 00 00  // .....(..........
     0030: 10 D1 38 00 10 00 40 00 00 00 D8 FE 00 00 00 00  // ..8...@.........
     0040: 00 00 00 00 44 00 00 00 02 00 00 00 02 08 00 00  // ....D...........
     0050: 02 10 00 00 02 F8 00 00 02 FA 00 00 02 FB 00 00  // ................
-    0060: 48 00 00 00 00 A0 00 01                          // H.......
+    0060: 48 00 00 00 00 A0 00 01 11 11 48 00 10 00 40 00  // H.........H...@.
+    0070: 00 00 D8 FE 00 00 00 00 00 00 00 00 00 00 00 00  // ................
+    0080: D3 29 00 00 00 00 00 00 00 00 00 00 00 00 00 00  // .)..............
+    0090: 02 00 00 00 02 08 00 00 02 10 00 00 02 F8 00 00  // ................
+    00A0: 02 FA 00 00 02 FB 00 00 48 00 00 00 00 A0 00 01  // ........H.......
---
 tests/data/acpi/q35/IVRS.ivrs               | Bin 104 -> 176 bytes
 tests/qtest/bios-tables-test-allowed-diff.h |   1 -
 2 files changed, 1 deletion(-)
tests/data/acpi/q35/IVRS.ivrs binary +0 -0
index 17611202e53a32f7da8e4925d6955b384670b8b1..7f9e91aabc0b7777a7efc9f219587a4f91f0edb1 100644
GIT binary patch
tests/qtest/bios-tables-test-allowed-diff.h +0 -1
index ac420db6b7..dfb8523c8b 100644
@@ -1,2 +1 @@
 /* List of comma-separated changed AML files to ignore */
-"tests/data/acpi/q35/IVRS.ivrs",
Signature · 1 line
2.25.1
#8 lore raw
In reply to Bui Quang Minh
Quoted 41 lines · Bui Quang Minh

Hi everyone,

This series implements x2APIC mode in userspace local APIC and the RDMSR/WRMSR helper to access x2APIC registers in x2APIC mode. Intel iommu and AMD iommu are adjusted to support x2APIC interrupt remapping. With this series, we can now boot Linux kernel into x2APIC mode with TCG accelerator using either Intel or AMD iommu.

Testing to boot my own built Linux 6.3.0-rc2, the kernel successfully boot with enabled x2APIC and can enumerate CPU with APIC ID 257

Using Intel IOMMU

qemu/build/qemu-system-x86_64 \
-smp 2,maxcpus=260 \
-cpu qemu64,x2apic=on \
-machine q35 \
-device intel-iommu,intremap=on,eim=on \
-device qemu64-x86_64-cpu,x2apic=on,core-id=257,socket-id=0,thread-id=0 \
-m 2G \
-kernel $KERNEL_DIR \
-append "nokaslr console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \
-drive file=$IMAGE_DIR,format=raw \
-nographic \
-s

Using AMD IOMMU

qemu/build/qemu-system-x86_64 \
-smp 2,maxcpus=260 \
-cpu qemu64,x2apic=on \
-machine q35 \
-device amd-iommu,intremap=on,xtsup=on \
-device qemu64-x86_64-cpu,x2apic=on,core-id=257,socket-id=0,thread-id=0 \
-m 2G \
-kernel $KERNEL_DIR \
-append "nokaslr console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \
-drive file=$IMAGE_DIR,format=raw \
-nographic \
-s

Quoted 2 lines

Testing the emulated userspace APIC with kvm-unit-tests, disable test device with this patch

Seems to break build for windows/amd64
https://gitlab.com/mstredhat/qemu/-/pipelines/1118886361/failures

Quoted 155 lines
lib/x86/fwcfg.c +1 -0
index 1734afb..f56fe1c 100644
@@ -27,6 +27,7 @@ static void read_cfg_override(void)
 
        if ((str = getenv("TEST_DEVICE")))
                no_test_device = !atol(str);
+       no_test_device = true;
 
        if ((str = getenv("MEMLIMIT")))
                fw_override[FW_CFG_MAX_RAM] = atol(str) * 1024 * 1024;

~ env QEMU=/home/minh/Desktop/oss/qemu/build/qemu-system-x86_64 ACCEL=tcg \
./run_tests.sh -v -g apic

TESTNAME=apic-split TIMEOUT=90s ACCEL=tcg ./x86/run x86/apic.flat -smp 2 -cpu qemu64,+x2apic,+tsc-deadline -machine kernel_irqchip=split FAIL apic-split (54 tests, 8 unexpected failures, 1 skipped) TESTNAME=ioapic-split TIMEOUT=90s ACCEL=tcg ./x86/run x86/ioapic.flat -smp 1 -cpu qemu64 -machine kernel_irqchip=split PASS ioapic-split (19 tests) TESTNAME=x2apic TIMEOUT=30 ACCEL=tcg ./x86/run x86/apic.flat -smp 2 -cpu qemu64,+x2apic,+tsc-deadline FAIL x2apic (54 tests, 8 unexpected failures, 1 skipped) TESTNAME=xapic TIMEOUT=60 ACCEL=tcg ./x86/run x86/apic.flat -smp 2 -cpu qemu64,-x2apic,+tsc-deadline -machine pit=off FAIL xapic (43 tests, 6 unexpected failures, 2 skipped)

FAIL: apic_disable: *0xfee00030: 50014
FAIL: apic_disable: *0xfee00080: f0
FAIL: apic_disable: *0xfee00030: 50014
FAIL: apic_disable: *0xfee00080: f0
FAIL: apicbase: relocate apic

These errors are because we don't disable MMIO region when switching to x2APIC and don't support relocate MMIO region yet. This is a problem because, MMIO region is the same for all CPUs, in order to support these we need to figure out how to allocate and manage different MMIO regions for each CPUs. This can be an improvement in the future.

FAIL: nmi-after-sti
FAIL: multiple nmi

These errors are in the way we handle CPU_INTERRUPT_NMI in core TCG.

FAIL: TMCCT should stay at zero

This error is related to APIC timer which should be addressed in separate patch.

Version 11 changes,
- Patch 2:
+ Rebase to master and fix conflict with commit c04cfb4596 (hw/i386: fix short-circuit logic with non-optimizing builds)

Version 10 changes,
- Patch 2:
+ Fix null pointer dereference due to uninitialized local_apics when using machine none
- Patch 5, 7:
+ These patches are added to follow the bios-tables-test instructions to commit the new changed IVRS.ivrs binary file

Version 9 changes,
- Patch 1:
+ Create apic_msr_read/write which is a small wrapper around apic_register_read/write that have additional x2apic mode check
- Patch 2:
+ Remove raise_exception_ra which is is TCG specific. Instead, return -1 and let the accelerator raise the appropriate exception
+ Refactor apic_get_delivery_bitmask a little bit to reduce line length
+ Move cpu_has_x2apic_feature and cpu_set_apic_feature from patch 3 to patch 2 so that patch 2 can be compiled without patch 3
- Patch 3:
+ set_base in APICCommonClass now returns an int to indicate error
+ Remove raise_exception_ra in apic_set base which is is TCG specific. Instead, return -1 and let the accelerator raise the appropriate exception

Version 8 changes,
- Patch 2, 4:
+ Rebase to master and resolve conflicts in these 2 patches

Version 7 changes,
- Patch 4:
+ If eim=on, keep checking if kvm x2APIC is enabled when kernel-irqchip is split

Version 6 changes,
- Patch 5:
+ Make all places use the amdvi_extended_feature_register to get extended feature register

Version 5 changes,
- Patch 3:
+ Rebase to master and fix conflict
- Patch 5:
+ Create a helper function to get amdvi extended feature register instead of storing it in AMDVIState

Version 4 changes,
- Patch 5:
+ Instead of replacing IVHD type 0x10 with type 0x11, export both types for backward compatibility with old guest operating system
+ Flip the xtsup feature check condition in amdvi_int_remap_ga for readability

Version 3 changes,
- Patch 2:
+ Allow APIC ID > 255 only when x2APIC feature is supported on CPU
+ Make physical destination mode IPI which has destination id 0xffffffff a broadcast to xAPIC CPUs
+ Make cluster address 0xf in cluster model of xAPIC logical destination mode a broadcast to all clusters
+ Create new extended_log_dest to store APIC_LDR information in x2APIC instead of extending log_dest for backward compatibility in vmstate

Version 2 changes,
- Add support for APIC ID larger than 255
- Adjust AMD iommu for x2APIC support
- Reorganize and split patch 1,2 into patch 1,2,3 in version 2

Thanks,
Quang Minh.

Bui Quang Minh (7):
i386/tcg: implement x2APIC registers MSR access
apic: add support for x2APIC mode
apic, i386/tcg: add x2apic transitions
intel_iommu: allow Extended Interrupt Mode when using userspace APIC
test: bios-tables-test: prepare IVRS change in ACPI table
amd_iommu: report x2APIC support to the operating system
test: bios-tables-test: add IVRS changed binary

 hw/i386/acpi-build.c                 | 129 +++++---
 hw/i386/amd_iommu.c                  |  29 +-
 hw/i386/amd_iommu.h                  |  16 +-
 hw/i386/intel_iommu.c                |   6 +-
 hw/i386/kvm/apic.c                   |   3 +-
 hw/i386/x86.c                        |   6 +-
 hw/i386/xen/xen_apic.c               |   3 +-
 hw/intc/apic.c                       | 473 +++++++++++++++++++++------
 hw/intc/apic_common.c                |  22 +-
 hw/intc/trace-events                 |   4 +-
 include/hw/i386/apic.h               |   8 +-
 include/hw/i386/apic_internal.h      |   9 +-
 target/i386/cpu-sysemu.c             |  18 +-
 target/i386/cpu.c                    |   9 +-
 target/i386/cpu.h                    |   9 +
 target/i386/tcg/sysemu/misc_helper.c |  41 ++-
 target/i386/whpx/whpx-apic.c         |   3 +-
 tests/data/acpi/q35/IVRS.ivrs        | Bin 104 -> 176 bytes
 18 files changed, 600 insertions(+), 188 deletions(-)
Signature · 1 line
2.25.1
#9 lore raw
Quoted 42 lines · Michael S. Tsirkin
Quoted 41 lines · Bui Quang Minh

Hi everyone,

This series implements x2APIC mode in userspace local APIC and the RDMSR/WRMSR helper to access x2APIC registers in x2APIC mode. Intel iommu and AMD iommu are adjusted to support x2APIC interrupt remapping. With this series, we can now boot Linux kernel into x2APIC mode with TCG accelerator using either Intel or AMD iommu.

Testing to boot my own built Linux 6.3.0-rc2, the kernel successfully boot with enabled x2APIC and can enumerate CPU with APIC ID 257

Using Intel IOMMU

qemu/build/qemu-system-x86_64 \
   -smp 2,maxcpus=260 \
   -cpu qemu64,x2apic=on \
   -machine q35 \
   -device intel-iommu,intremap=on,eim=on \
   -device qemu64-x86_64-cpu,x2apic=on,core-id=257,socket-id=0,thread-id=0 \
   -m 2G \
   -kernel $KERNEL_DIR \
   -append "nokaslr console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \
   -drive file=$IMAGE_DIR,format=raw \
   -nographic \
   -s

Using AMD IOMMU

qemu/build/qemu-system-x86_64 \
   -smp 2,maxcpus=260 \
   -cpu qemu64,x2apic=on \
   -machine q35 \
   -device amd-iommu,intremap=on,xtsup=on \
   -device qemu64-x86_64-cpu,x2apic=on,core-id=257,socket-id=0,thread-id=0 \
   -m 2G \
   -kernel $KERNEL_DIR \
   -append "nokaslr console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \
   -drive file=$IMAGE_DIR,format=raw \
   -nographic \
   -s
Quoted 5 lines
Quoted 2 lines

Testing the emulated userspace APIC with kvm-unit-tests, disable test device with this patch

Seems to break build for windows/amd64
https://gitlab.com/mstredhat/qemu/-/pipelines/1118886361/failures

The failure is because when CONFIG_AMD_IOMMU=n, amd_iommu.c is not built so the linker cannot find the definition of amdvi_extended_feature_register (amdvi_extended_feature_register is used in acpi-build.c). I create a stub to solve this problem and it passes all CI tests. I will squash the following changes into patch 6. What do you think about this?

hw/i386/amd_iommu_stub.c new file +26 -0
new file mode 100644
index 0000000000..d62a3732e6
@@ -0,0 +1,26 @@
+/*
+ * Stubs for AMD IOMMU emulation
+ *
+ * Copyright (C) 2023 Bui Quang Minh <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include "qemu/osdep.h"
+#include "amd_iommu.h"
+
+uint64_t amdvi_extended_feature_register(AMDVIState *s)
+{
+    return AMDVI_DEFAULT_EXT_FEATURES;
+}
hw/i386/meson.build +2 -1
index 369c6bf823..d38637b046 100644
@@ -9,7 +9,8 @@ i386_ss.add(files(
 
 i386_ss.add(when: 'CONFIG_X86_IOMMU', if_true: files('x86-iommu.c'),
                                       if_false: files('x86-iommu-stub.c'))
-i386_ss.add(when: 'CONFIG_AMD_IOMMU', if_true: files('amd_iommu.c'))
+i386_ss.add(when: 'CONFIG_AMD_IOMMU', if_true: files('amd_iommu.c'),
+                                      if_false: files('amd_iommu_stub.c'))
 i386_ss.add(when: 'CONFIG_I440FX', if_true: files('pc_piix.c'))
 i386_ss.add(when: 'CONFIG_MICROVM', if_true: files('microvm.c', 

'acpi-microvm.c', 'microvm-dt.c'))
i386_ss.add(when: 'CONFIG_Q35', if_true: files('pc_q35.c'))

Thanks,
Quang Minh.

#10 lore raw

hw/i386/amd_iommu_stub.c

Quoted 56 lines · Bui Quang Minh
Quoted 42 lines · Michael S. Tsirkin
Quoted 41 lines · Bui Quang Minh

Hi everyone,

This series implements x2APIC mode in userspace local APIC and the RDMSR/WRMSR helper to access x2APIC registers in x2APIC mode. Intel iommu and AMD iommu are adjusted to support x2APIC interrupt remapping. With this series, we can now boot Linux kernel into x2APIC mode with TCG accelerator using either Intel or AMD iommu.

Testing to boot my own built Linux 6.3.0-rc2, the kernel successfully boot with enabled x2APIC and can enumerate CPU with APIC ID 257

Using Intel IOMMU

qemu/build/qemu-system-x86_64 \
   -smp 2,maxcpus=260 \
   -cpu qemu64,x2apic=on \
   -machine q35 \
   -device intel-iommu,intremap=on,eim=on \
   -device qemu64-x86_64-cpu,x2apic=on,core-id=257,socket-id=0,thread-id=0 \
   -m 2G \
   -kernel $KERNEL_DIR \
   -append "nokaslr console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \
   -drive file=$IMAGE_DIR,format=raw \
   -nographic \
   -s

Using AMD IOMMU

qemu/build/qemu-system-x86_64 \
   -smp 2,maxcpus=260 \
   -cpu qemu64,x2apic=on \
   -machine q35 \
   -device amd-iommu,intremap=on,xtsup=on \
   -device qemu64-x86_64-cpu,x2apic=on,core-id=257,socket-id=0,thread-id=0 \
   -m 2G \
   -kernel $KERNEL_DIR \
   -append "nokaslr console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \
   -drive file=$IMAGE_DIR,format=raw \
   -nographic \
   -s
Quoted 5 lines
Quoted 2 lines

Testing the emulated userspace APIC with kvm-unit-tests, disable test device with this patch

Seems to break build for windows/amd64
https://gitlab.com/mstredhat/qemu/-/pipelines/1118886361/failures

The failure is because when CONFIG_AMD_IOMMU=n, amd_iommu.c is not built so the linker cannot find the definition of amdvi_extended_feature_register (amdvi_extended_feature_register is used in acpi-build.c). I create a stub to solve this problem and it passes all CI tests. I will squash the following changes into patch 6. What do you think about this?

hw/i386/amd_iommu_stub.c +0 -0

I'd probably call it amd_iommu-stub.c even if the mix of _ and - looks weird, but this is how all other stubs are called.
Document this in the commit log too please.

Quoted 50 lines

new file mode 100644
index 0000000000..d62a3732e6

hw/i386/amd_iommu_stub.c new file +26 -0
@@ -0,0 +1,26 @@
+/*
+ * Stubs for AMD IOMMU emulation
+ *
+ * Copyright (C) 2023 Bui Quang Minh <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include "qemu/osdep.h"
+#include "amd_iommu.h"
+
+uint64_t amdvi_extended_feature_register(AMDVIState *s)
+{
+    return AMDVI_DEFAULT_EXT_FEATURES;
+}
hw/i386/meson.build +2 -1
index 369c6bf823..d38637b046 100644
@@ -9,7 +9,8 @@ i386_ss.add(files(
 
 i386_ss.add(when: 'CONFIG_X86_IOMMU', if_true: files('x86-iommu.c'),
                                       if_false: files('x86-iommu-stub.c'))
-i386_ss.add(when: 'CONFIG_AMD_IOMMU', if_true: files('amd_iommu.c'))
+i386_ss.add(when: 'CONFIG_AMD_IOMMU', if_true: files('amd_iommu.c'),
+                                      if_false: files('amd_iommu_stub.c'))
 i386_ss.add(when: 'CONFIG_I440FX', if_true: files('pc_piix.c'))
 i386_ss.add(when: 'CONFIG_MICROVM', if_true: files('microvm.c',
 'acpi-microvm.c', 'microvm-dt.c'))

i386_ss.add(when: 'CONFIG_Q35', if_true: files('pc_q35.c'))

Thanks,
Quang Minh.

#11 lore raw
In reply to Michael S. Tsirkin
Quoted 42 lines · Michael S. Tsirkin
Quoted 41 lines · Bui Quang Minh

Hi everyone,

This series implements x2APIC mode in userspace local APIC and the RDMSR/WRMSR helper to access x2APIC registers in x2APIC mode. Intel iommu and AMD iommu are adjusted to support x2APIC interrupt remapping. With this series, we can now boot Linux kernel into x2APIC mode with TCG accelerator using either Intel or AMD iommu.

Testing to boot my own built Linux 6.3.0-rc2, the kernel successfully boot with enabled x2APIC and can enumerate CPU with APIC ID 257

Using Intel IOMMU

qemu/build/qemu-system-x86_64 \
   -smp 2,maxcpus=260 \
   -cpu qemu64,x2apic=on \
   -machine q35 \
   -device intel-iommu,intremap=on,eim=on \
   -device qemu64-x86_64-cpu,x2apic=on,core-id=257,socket-id=0,thread-id=0 \
   -m 2G \
   -kernel $KERNEL_DIR \
   -append "nokaslr console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \
   -drive file=$IMAGE_DIR,format=raw \
   -nographic \
   -s

Using AMD IOMMU

qemu/build/qemu-system-x86_64 \
   -smp 2,maxcpus=260 \
   -cpu qemu64,x2apic=on \
   -machine q35 \
   -device amd-iommu,intremap=on,xtsup=on \
   -device qemu64-x86_64-cpu,x2apic=on,core-id=257,socket-id=0,thread-id=0 \
   -m 2G \
   -kernel $KERNEL_DIR \
   -append "nokaslr console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \
   -drive file=$IMAGE_DIR,format=raw \
   -nographic \
   -s
Quoted 5 lines
Quoted 2 lines

Testing the emulated userspace APIC with kvm-unit-tests, disable test device with this patch

Seems to break build for windows/amd64
https://gitlab.com/mstredhat/qemu/-/pipelines/1118886361/failures

I saw the CI test "pages" failed too. On my CI, most of the time, it failed with

$ htags -anT --tree-view=filetree -m qemu_init -t "Welcome to the QEMU sourcecode"
00:24
htags: Negative exec line limit = -8099

It only succeeded once. I could not reproduce locally. Do you have any ideas what the problem is?

Thanks,
Quang Minh.

#12 lore raw
Quoted 49 lines · Bui Quang Minh
Quoted 42 lines · Michael S. Tsirkin
Quoted 41 lines · Bui Quang Minh

Hi everyone,

This series implements x2APIC mode in userspace local APIC and the RDMSR/WRMSR helper to access x2APIC registers in x2APIC mode. Intel iommu and AMD iommu are adjusted to support x2APIC interrupt remapping. With this series, we can now boot Linux kernel into x2APIC mode with TCG accelerator using either Intel or AMD iommu.

Testing to boot my own built Linux 6.3.0-rc2, the kernel successfully boot with enabled x2APIC and can enumerate CPU with APIC ID 257

Using Intel IOMMU

qemu/build/qemu-system-x86_64 \
   -smp 2,maxcpus=260 \
   -cpu qemu64,x2apic=on \
   -machine q35 \
   -device intel-iommu,intremap=on,eim=on \
   -device qemu64-x86_64-cpu,x2apic=on,core-id=257,socket-id=0,thread-id=0 \
   -m 2G \
   -kernel $KERNEL_DIR \
   -append "nokaslr console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \
   -drive file=$IMAGE_DIR,format=raw \
   -nographic \
   -s

Using AMD IOMMU

qemu/build/qemu-system-x86_64 \
   -smp 2,maxcpus=260 \
   -cpu qemu64,x2apic=on \
   -machine q35 \
   -device amd-iommu,intremap=on,xtsup=on \
   -device qemu64-x86_64-cpu,x2apic=on,core-id=257,socket-id=0,thread-id=0 \
   -m 2G \
   -kernel $KERNEL_DIR \
   -append "nokaslr console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \
   -drive file=$IMAGE_DIR,format=raw \
   -nographic \
   -s

Quoted 5 lines
Quoted 2 lines

Testing the emulated userspace APIC with kvm-unit-tests, disable test device with this patch

Seems to break build for windows/amd64
https://gitlab.com/mstredhat/qemu/-/pipelines/1118886361/failures

Quoted 7 lines

I saw the CI test "pages" failed too. On my CI, most of the time, it failed with

$ htags -anT --tree-view=filetree -m qemu_init -t "Welcome to the QEMU sourcecode"
00:24
htags: Negative exec line limit = -8099

It only succeeded once. I could not reproduce locally. Do you have any ideas what the problem is?

I think I briefly understand why pages test fails. Internally, htags call global to parse the output of gtags. When executing command, it expects the size of argv and env to 20*1024 (https://github.com/harai/gnu-global/blob/f86ba74d867385353815f8656c4a6cf4029c1f0b/libutil/xargs.c#L92-L105). The failed test case only happens when the last commit is patch 7 (test: bios-tables-test: add IVRS changed binary) which has a very long commit message (around 9000 bytes). By default, Gitlab appends some environment variables to the runner and one of them is CI_COMMIT_MESSAGE which contains the long commit message. So it exceeds the limit of 20*1024 bytes and fails.

In my opinion, this failed test is not so critical and seems unrelated to the series so I skip this failed test. I will post the new version to fix the windows/amd64 build soon.

Thanks,
Quang Minh.

#13 lore raw
Quoted 71 lines · Bui Quang Minh
Quoted 62 lines · Bui Quang Minh
Quoted 52 lines · Michael S. Tsirkin
Quoted 51 lines · Bui Quang Minh

Hi everyone,

This series implements x2APIC mode in userspace local APIC and the RDMSR/WRMSR helper to access x2APIC registers in x2APIC mode. Intel iommu
and AMD iommu are adjusted to support x2APIC interrupt remapping. With this
series, we can now boot Linux kernel into x2APIC mode with TCG accelerator
using either Intel or AMD iommu.

Testing to boot my own built Linux 6.3.0-rc2, the kernel successfully boot
with enabled x2APIC and can enumerate CPU with APIC ID 257

Using Intel IOMMU

qemu/build/qemu-system-x86_64 \
   -smp 2,maxcpus=260 \
   -cpu qemu64,x2apic=on \
   -machine q35 \
   -device intel-iommu,intremap=on,eim=on \
   -device qemu64-x86_64-cpu,x2apic=on,core-id=257,socket-id=0,thread-id=0 \
   -m 2G \
   -kernel $KERNEL_DIR \
   -append "nokaslr console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \
   -drive file=$IMAGE_DIR,format=raw \
   -nographic \
   -s

Using AMD IOMMU

qemu/build/qemu-system-x86_64 \
   -smp 2,maxcpus=260 \
   -cpu qemu64,x2apic=on \
   -machine q35 \
   -device amd-iommu,intremap=on,xtsup=on \
   -device qemu64-x86_64-cpu,x2apic=on,core-id=257,socket-id=0,thread-id=0 \
   -m 2G \
   -kernel $KERNEL_DIR \
   -append "nokaslr console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \
   -drive file=$IMAGE_DIR,format=raw \
   -nographic \
   -s

Quoted 5 lines
Quoted 2 lines

Testing the emulated userspace APIC with kvm-unit-tests, disable test device with this patch

Seems to break build for windows/amd64
https://gitlab.com/mstredhat/qemu/-/pipelines/1118886361/failures

I saw the CI test "pages" failed too. On my CI, most of the time, it failed with

Quoted 7 lines
$ htags -anT --tree-view=filetree -m qemu_init -t "Welcome to the QEMU
sourcecode"
00:24
htags: Negative exec line limit = -8099

It only succeeded once. I could not reproduce locally. Do you have any ideas what the problem is?

Quoted 8 lines

I think I briefly understand why pages test fails. Internally, htags call
global to parse the output of gtags. When executing command, it expects the
size of argv and env to 20*1024 (https://github.com/harai/gnu-global/blob/f86ba74d867385353815f8656c4a6cf4029c1f0b/libutil/xargs.c#L92-L105).
The failed test case only happens when the last commit is patch 7 (test:
bios-tables-test: add IVRS changed binary) which has a very long commit
message (around 9000 bytes). By default, Gitlab appends some environment
variables to the runner and one of them is CI_COMMIT_MESSAGE which contains
the long commit message. So it exceeds the limit of 20*1024 bytes and fails.

Thanks for debugging this!
So I think we should clear CI_COMMIT_MESSAGE when invoking htags, right?

Quoted 2 lines

In my opinion, this failed test is not so critical and seems unrelated to the series so I skip this failed test.

Yes ok to ignore. But even better if we also add a workaround.

Quoted 5 lines

I will post the new version to fix
the windows/amd64 build soon.

Thanks,
Quang Minh.