Skip to content

Support x2APIC mode with TCG accelerator

Bui Quang Minh qemu-devel 14 versions – New

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v12 Latest7 patches 0 repliesNew
  2. v117 patches 6 repliesSuperseded
  3. v107 patches 1 replySuperseded
Earlier versions · 11
VersionPostedPatchesRepliesStatus
v95 patches9 repliesSuperseded
v85 patches7 repliesSuperseded
v75 patches1 replySuperseded
v65 patches5 repliesSuperseded
v55 patches1 replySuperseded
v45 patches7 repliesSuperseded
v3 posting 3/35 patches9 repliesSuperseded
v3 posting 2/35 patches0 repliesSuperseded
v3 posting 1/30 of 5 patches2 repliesSuperseded
v25 patches17 repliesSuperseded
v14 patches17 repliesSuperseded

Patches in v12 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/7i386/tcg: implement x2APIC registers MSR accesshw/intc/apic.c hw/intc/trace-events include/hw/i386/apic.h +2 more+127 -320 replies
2/7apic: add support for x2APIC modehw/i386/x86.c hw/intc/apic.c hw/intc/apic_common.c +4 more+259 -750 replies
3/7apic, i386/tcg: add x2apic transitionshw/i386/kvm/apic.c hw/i386/xen/xen_apic.c hw/intc/apic.c +7 more+96 -190 replies
4/7intel_iommu: allow Extended Interrupt Mode when using userspace APIChw/i386/intel_iommu.c+1 -50 replies
5/7test: bios-tables-test: prepare IVRS change in ACPI tabletests/qtest/bios-tables-test-allowed-diff.h+1 -00 replies
6/7amd_iommu: report x2APIC support to the operating systemhw/i386/acpi-build.c hw/i386/amd_iommu-stub.c hw/i386/amd_iommu.c +2 more+145 -580 replies
7/7test: bios-tables-test: add IVRS changed binarytests/data/acpi/q35/IVRS.ivrs tests/qtest/bios-tables-test-allowed-diff.h+0 -10 replies

Reviews trailers per patch

1 of 7 patches has a Reviewed-by or Acked-by.

PatchPeter Xu
1/7
2/7
3/7
4/7A-b
5/7
6/7
7/7

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v12 thread

Cover letter [PATCH v12 0/7] Support x2APIC mode with TCG accelerator

Hi everyone,

This series implements x2APIC mode in userspace local APIC and the RDMSR/WRMSR helper to access x2APIC registers in x2APIC mode. Intel iommu and AMD iommu are adjusted to support x2APIC interrupt remapping. With this series, we can now boot Linux kernel into x2APIC mode with TCG accelerator using either Intel or AMD iommu.

Testing to boot my own built Linux 6.3.0-rc2 and 6.7.0-rc6, the kernel successfully boot with enabled x2APIC and can enumerate CPU with APIC ID 257

Using Intel IOMMU

qemu/build/qemu-system-x86_64 \
-smp 2,maxcpus=260 \
-cpu qemu64,x2apic=on \
-machine q35 \
-device intel-iommu,intremap=on,eim=on \
-device qemu64-x86_64-cpu,x2apic=on,core-id=257,socket-id=0,thread-id=0 \
-m 2G \
-kernel $KERNEL_DIR \
-append "nokaslr console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \
-drive file=$IMAGE_DIR,format=raw \
-nographic \
-s

Using AMD IOMMU

qemu/build/qemu-system-x86_64 \
-smp 2,maxcpus=260 \
-cpu qemu64,x2apic=on \
-machine q35 \
-device amd-iommu,intremap=on,xtsup=on \
-device qemu64-x86_64-cpu,x2apic=on,core-id=257,socket-id=0,thread-id=0 \
-m 2G \
-kernel $KERNEL_DIR \
-append "nokaslr console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \
-drive file=$IMAGE_DIR,format=raw \
-nographic \
-s

Testing the emulated userspace APIC with kvm-unit-tests, disable test device with this patch

lib/x86/fwcfg.c +1 -0
index 1734afb..f56fe1c 100644
@@ -27,6 +27,7 @@ static void read_cfg_override(void)
 
        if ((str = getenv("TEST_DEVICE")))
                no_test_device = !atol(str);
+       no_test_device = true;
 
        if ((str = getenv("MEMLIMIT")))
                fw_override[FW_CFG_MAX_RAM] = atol(str) * 1024 * 1024;

~ env QEMU=/home/minh/Desktop/oss/qemu/build/qemu-system-x86_64 ACCEL=tcg \
./run_tests.sh -v -g apic

TESTNAME=apic-split TIMEOUT=90s ACCEL=tcg ./x86/run x86/apic.flat -smp 2 -cpu qemu64,+x2apic,+tsc-deadline -machine kernel_irqchip=split FAIL apic-split (54 tests, 8 unexpected failures, 1 skipped) TESTNAME=ioapic-split TIMEOUT=90s ACCEL=tcg ./x86/run x86/ioapic.flat -smp 1 -cpu qemu64 -machine kernel_irqchip=split PASS ioapic-split (19 tests) TESTNAME=x2apic TIMEOUT=30 ACCEL=tcg ./x86/run x86/apic.flat -smp 2 -cpu qemu64,+x2apic,+tsc-deadline FAIL x2apic (54 tests, 8 unexpected failures, 1 skipped) TESTNAME=xapic TIMEOUT=60 ACCEL=tcg ./x86/run x86/apic.flat -smp 2 -cpu qemu64,-x2apic,+tsc-deadline -machine pit=off FAIL xapic (43 tests, 6 unexpected failures, 2 skipped)

FAIL: apic_disable: *0xfee00030: 50014
FAIL: apic_disable: *0xfee00080: f0
FAIL: apic_disable: *0xfee00030: 50014
FAIL: apic_disable: *0xfee00080: f0
FAIL: apicbase: relocate apic

These errors are because we don't disable MMIO region when switching to x2APIC and don't support relocate MMIO region yet. This is a problem because, MMIO region is the same for all CPUs, in order to support these we need to figure out how to allocate and manage different MMIO regions for each CPUs. This can be an improvement in the future.

FAIL: nmi-after-sti
FAIL: multiple nmi

These errors are in the way we handle CPU_INTERRUPT_NMI in core TCG.

FAIL: TMCCT should stay at zero

This error is related to APIC timer which should be addressed in separate patch.

Version 12 changes,
- Patch 1:
+ Rename qemu_mutex_lock_iothread to bql_lock
- Patch 2:
+ Correct the logical x2APIC ID (which is contained in logical destination register) calculation
- Patch 6:
+ Add amd_iommu-stub.c to provide the definition for amdvi_extended_feature_register when CONFIG_AMD_IOMMU=n

Version 11 changes,
- Patch 2:
+ Rebase to master and fix conflict with commit c04cfb4596 (hw/i386: fix short-circuit logic with non-optimizing builds)

Version 10 changes,
- Patch 2:
+ Fix null pointer dereference due to uninitialized local_apics when using machine none
- Patch 5, 7:
+ These patches are added to follow the bios-tables-test instructions to commit the new changed IVRS.ivrs binary file

Version 9 changes,
- Patch 1:
+ Create apic_msr_read/write which is a small wrapper around apic_register_read/write that have additional x2apic mode check
- Patch 2:
+ Remove raise_exception_ra which is is TCG specific. Instead, return -1 and let the accelerator raise the appropriate exception
+ Refactor apic_get_delivery_bitmask a little bit to reduce line length
+ Move cpu_has_x2apic_feature and cpu_set_apic_feature from patch 3 to patch 2 so that patch 2 can be compiled without patch 3
- Patch 3:
+ set_base in APICCommonClass now returns an int to indicate error
+ Remove raise_exception_ra in apic_set base which is is TCG specific. Instead, return -1 and let the accelerator raise the appropriate exception

Version 8 changes,
- Patch 2, 4:
+ Rebase to master and resolve conflicts in these 2 patches

Version 7 changes,
- Patch 4:
+ If eim=on, keep checking if kvm x2APIC is enabled when kernel-irqchip is split

Version 6 changes,
- Patch 5:
+ Make all places use the amdvi_extended_feature_register to get extended feature register

Version 5 changes,
- Patch 3:
+ Rebase to master and fix conflict
- Patch 5:
+ Create a helper function to get amdvi extended feature register instead of storing it in AMDVIState

Version 4 changes,
- Patch 5:
+ Instead of replacing IVHD type 0x10 with type 0x11, export both types for backward compatibility with old guest operating system
+ Flip the xtsup feature check condition in amdvi_int_remap_ga for readability

Version 3 changes,
- Patch 2:
+ Allow APIC ID > 255 only when x2APIC feature is supported on CPU
+ Make physical destination mode IPI which has destination id 0xffffffff a broadcast to xAPIC CPUs
+ Make cluster address 0xf in cluster model of xAPIC logical destination mode a broadcast to all clusters
+ Create new extended_log_dest to store APIC_LDR information in x2APIC instead of extending log_dest for backward compatibility in vmstate

Version 2 changes,
- Add support for APIC ID larger than 255
- Adjust AMD iommu for x2APIC support
- Reorganize and split patch 1,2 into patch 1,2,3 in version 2

Thanks,
Quang Minh.

Bui Quang Minh (7):
i386/tcg: implement x2APIC registers MSR access
apic: add support for x2APIC mode
apic, i386/tcg: add x2apic transitions
intel_iommu: allow Extended Interrupt Mode when using userspace APIC
test: bios-tables-test: prepare IVRS change in ACPI table
amd_iommu: report x2APIC support to the operating system
test: bios-tables-test: add IVRS changed binary

 hw/i386/acpi-build.c                 | 129 +++++---
 hw/i386/amd_iommu-stub.c             |  26 ++
 hw/i386/amd_iommu.c                  |  29 +-
 hw/i386/amd_iommu.h                  |  16 +-
 hw/i386/intel_iommu.c                |   6 +-
 hw/i386/kvm/apic.c                   |   3 +-
 hw/i386/meson.build                  |   3 +-
 hw/i386/x86.c                        |   6 +-
 hw/i386/xen/xen_apic.c               |   3 +-
 hw/intc/apic.c                       | 473 +++++++++++++++++++++------
 hw/intc/apic_common.c                |  22 +-
 hw/intc/trace-events                 |   4 +-
 include/hw/i386/apic.h               |   8 +-
 include/hw/i386/apic_internal.h      |   9 +-
 target/i386/cpu-sysemu.c             |  18 +-
 target/i386/cpu.c                    |   9 +-
 target/i386/cpu.h                    |   9 +
 target/i386/tcg/sysemu/misc_helper.c |  41 ++-
 target/i386/whpx/whpx-apic.c         |   3 +-
 tests/data/acpi/q35/IVRS.ivrs        | Bin 104 -> 176 bytes
 20 files changed, 628 insertions(+), 189 deletions(-)
 create mode 100644 hw/i386/amd_iommu-stub.c
Signature · 1 line
2.25.1