Skip to contentSkip to conversation

KVM/ARM: Fix >256 vcpus

RFC Superseded Eric Auger 3 participants 11 messages Sep 11, 2019 – Sep 12, 2019

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Eric Auger

author
#0 lore raw

v2 cover KVM/ARM: Fix >256 vcpus

Since 4.18, KVM/ARM exposes a KVM_MAX_VCPUS equal to 512. However it was reported [1] that a VM with more than 256 vcpus cannot be launched. 5.4 is about to fix the situation with 2 patches:
- one upgrade of the KVM_IRQ_LINE API [2] supporting a vcpu id encoded on 12 bits,
- the reduction of KVM IO devices consumed by each GICv3 redistributor [3]

This series uses the new KVM_IRQ_LINE API and also checks the associated capability (KVM_CAP_ARM_IRQ_LINE_LAYOUT_2) in machvirt.

Without the series, as soon as the -smp arguments exceeds 256, QEMU exits with "kvm_set_irq: Invalid argument".

Best Regards

Eric

References:
[1] Can we boot a 512U kvm guest?

    https://patchwork.kernel.org/patch/11091501/
[2] [PATCH] KVM: arm/arm64: vgic: Allow more than 256 vcpus for KVM_IRQ_LINE
    https://patchwork.kernel.org/patch/11099609/
[3] [PATCH] KVM: arm/arm64: vgic: Use a single IO device per redistributor
    https://patchwork.kernel.org/patch/11112141/

This series can be found at:
https://github.com/eauger/qemu/tree/v4.1.0-256fix-rfc-v2

History:
v1 -> v2:
- New layout set for kvm_arm_gic_set_irq and
arm_cpu_kvm_set_irq through kvm_arm_set_irq
- Introduced kvm_arm_irq_line_layout_mismatch()

Eric Auger (3):
linux headers: update for KVM_CAP_ARM_IRQ_LINE_LAYOUT_2 intc/arm_gic: Support IRQ injection for more than 256 vpus
virt: Check KVM_CAP_ARM_IRQ_LINE_LAYOUT_2 for smp_cpus > 256

 hw/arm/virt.c                                |  4 +++
 hw/intc/arm_gic_kvm.c                        |  7 ++--
 include/standard-headers/asm-x86/bootparam.h |  2 ++
 include/standard-headers/asm-x86/kvm_para.h  |  1 +
 include/standard-headers/linux/ethtool.h     |  2 ++
 include/standard-headers/linux/pci_regs.h    |  4 +++
 include/standard-headers/linux/virtio_ids.h  |  1 +
 include/standard-headers/linux/virtio_pmem.h |  6 ++--
 linux-headers/asm-arm/kvm.h                  | 16 ++++++++-
 linux-headers/asm-arm/unistd-common.h        |  2 ++
 linux-headers/asm-arm64/kvm.h                | 21 ++++++++++-
 linux-headers/asm-generic/mman-common.h      | 15 ++++----
 linux-headers/asm-generic/mman.h             | 10 +++---
 linux-headers/asm-generic/unistd.h           |  8 ++++-
 linux-headers/asm-mips/unistd_n32.h          |  1 +
 linux-headers/asm-mips/unistd_n64.h          |  1 +
 linux-headers/asm-mips/unistd_o32.h          |  1 +
 linux-headers/asm-powerpc/mman.h             |  6 +---
 linux-headers/asm-powerpc/unistd_32.h        |  2 ++
 linux-headers/asm-powerpc/unistd_64.h        |  2 ++
 linux-headers/asm-s390/unistd_32.h           |  2 ++
 linux-headers/asm-s390/unistd_64.h           |  2 ++
 linux-headers/asm-x86/kvm.h                  | 28 +++++++++++----
 linux-headers/asm-x86/unistd_32.h            |  2 ++
 linux-headers/asm-x86/unistd_64.h            |  2 ++
 linux-headers/asm-x86/unistd_x32.h           |  2 ++
 linux-headers/linux/kvm.h                    | 12 +++++--
 linux-headers/linux/psp-sev.h                |  5 +--
 target/arm/cpu.c                             | 10 +++---
 target/arm/kvm.c                             | 37 ++++++++++++++++++++
 target/arm/kvm_arm.h                         | 16 +++++++++
 31 files changed, 182 insertions(+), 48 deletions(-)
Signature · 1 line
2.20.1
author
#1 lore raw

RFC v2 1/3 linux headers: update for KVM_CAP_ARM_IRQ_LINE_LAYOUT_2

Temporary header update against 5.3-rc5 + Marc's patch:
"[PATCH] KVM: arm/arm64: vgic: Allow more than 256 vcpus for KVM_IRQ_LINE"

---
 include/standard-headers/asm-x86/bootparam.h |  2 ++
 include/standard-headers/asm-x86/kvm_para.h  |  1 +
 include/standard-headers/linux/ethtool.h     |  2 ++
 include/standard-headers/linux/pci_regs.h    |  4 +++
 include/standard-headers/linux/virtio_ids.h  |  1 +
 include/standard-headers/linux/virtio_pmem.h |  6 ++---
 linux-headers/asm-arm/kvm.h                  | 16 ++++++++++-
 linux-headers/asm-arm/unistd-common.h        |  2 ++
 linux-headers/asm-arm64/kvm.h                | 21 ++++++++++++++-
 linux-headers/asm-generic/mman-common.h      | 15 ++++++-----
 linux-headers/asm-generic/mman.h             | 10 +++----
 linux-headers/asm-generic/unistd.h           |  8 +++++-
 linux-headers/asm-mips/unistd_n32.h          |  1 +
 linux-headers/asm-mips/unistd_n64.h          |  1 +
 linux-headers/asm-mips/unistd_o32.h          |  1 +
 linux-headers/asm-powerpc/mman.h             |  6 +----
 linux-headers/asm-powerpc/unistd_32.h        |  2 ++
 linux-headers/asm-powerpc/unistd_64.h        |  2 ++
 linux-headers/asm-s390/unistd_32.h           |  2 ++
 linux-headers/asm-s390/unistd_64.h           |  2 ++
 linux-headers/asm-x86/kvm.h                  | 28 +++++++++++++++-----
 linux-headers/asm-x86/unistd_32.h            |  2 ++
 linux-headers/asm-x86/unistd_64.h            |  2 ++
 linux-headers/asm-x86/unistd_x32.h           |  2 ++
 linux-headers/linux/kvm.h                    | 12 ++++++---
 linux-headers/linux/psp-sev.h                |  5 +---
 26 files changed, 119 insertions(+), 37 deletions(-)
include/standard-headers/asm-x86/bootparam.h +2 -0
index 67d4f0119f..a6f7cf535e 100644
@@ -29,6 +29,8 @@
 #define XLF_EFI_HANDOVER_32		(1<<2)
 #define XLF_EFI_HANDOVER_64		(1<<3)
 #define XLF_EFI_KEXEC			(1<<4)
+#define XLF_5LEVEL			(1<<5)
+#define XLF_5LEVEL_ENABLED		(1<<6)
 
 
 #endif /* _ASM_X86_BOOTPARAM_H */
include/standard-headers/asm-x86/kvm_para.h +1 -0
index e1715143fd..90604a8fb7 100644
@@ -30,6 +30,7 @@
 #define KVM_FEATURE_ASYNC_PF_VMEXIT	10
 #define KVM_FEATURE_PV_SEND_IPI	11
 #define KVM_FEATURE_POLL_CONTROL	12
+#define KVM_FEATURE_PV_SCHED_YIELD	13
 
 #define KVM_HINTS_REALTIME      0
 
include/standard-headers/linux/ethtool.h +2 -0
index 9b9919a8f6..16d0eeea86 100644
@@ -1483,6 +1483,8 @@ enum ethtool_link_mode_bit_indices {
 	ETHTOOL_LINK_MODE_200000baseLR4_ER4_FR4_Full_BIT = 64,
 	ETHTOOL_LINK_MODE_200000baseDR4_Full_BIT	 = 65,
 	ETHTOOL_LINK_MODE_200000baseCR4_Full_BIT	 = 66,
+	ETHTOOL_LINK_MODE_100baseT1_Full_BIT		 = 67,
+	ETHTOOL_LINK_MODE_1000baseT1_Full_BIT		 = 68,
 
 	/* must be last entry */
 	__ETHTOOL_LINK_MODE_MASK_NBITS
include/standard-headers/linux/pci_regs.h +4 -0
index 27164769d1..f28e562d7c 100644
@@ -528,6 +528,7 @@
 #define  PCI_EXP_LNKCAP_SLS_5_0GB 0x00000002 /* LNKCAP2 SLS Vector bit 1 */
 #define  PCI_EXP_LNKCAP_SLS_8_0GB 0x00000003 /* LNKCAP2 SLS Vector bit 2 */
 #define  PCI_EXP_LNKCAP_SLS_16_0GB 0x00000004 /* LNKCAP2 SLS Vector bit 3 */
+#define  PCI_EXP_LNKCAP_SLS_32_0GB 0x00000005 /* LNKCAP2 SLS Vector bit 4 */
 #define  PCI_EXP_LNKCAP_MLW	0x000003f0 /* Maximum Link Width */
 #define  PCI_EXP_LNKCAP_ASPMS	0x00000c00 /* ASPM Support */
 #define  PCI_EXP_LNKCAP_L0SEL	0x00007000 /* L0s Exit Latency */
@@ -556,6 +557,7 @@
 #define  PCI_EXP_LNKSTA_CLS_5_0GB 0x0002 /* Current Link Speed 5.0GT/s */
 #define  PCI_EXP_LNKSTA_CLS_8_0GB 0x0003 /* Current Link Speed 8.0GT/s */
 #define  PCI_EXP_LNKSTA_CLS_16_0GB 0x0004 /* Current Link Speed 16.0GT/s */
+#define  PCI_EXP_LNKSTA_CLS_32_0GB 0x0005 /* Current Link Speed 32.0GT/s */
 #define  PCI_EXP_LNKSTA_NLW	0x03f0	/* Negotiated Link Width */
 #define  PCI_EXP_LNKSTA_NLW_X1	0x0010	/* Current Link Width x1 */
 #define  PCI_EXP_LNKSTA_NLW_X2	0x0020	/* Current Link Width x2 */
@@ -661,6 +663,7 @@
 #define  PCI_EXP_LNKCAP2_SLS_5_0GB	0x00000004 /* Supported Speed 5GT/s */
 #define  PCI_EXP_LNKCAP2_SLS_8_0GB	0x00000008 /* Supported Speed 8GT/s */
 #define  PCI_EXP_LNKCAP2_SLS_16_0GB	0x00000010 /* Supported Speed 16GT/s */
+#define  PCI_EXP_LNKCAP2_SLS_32_0GB	0x00000020 /* Supported Speed 32GT/s */
 #define  PCI_EXP_LNKCAP2_CROSSLINK	0x00000100 /* Crosslink supported */
 #define PCI_EXP_LNKCTL2		48	/* Link Control 2 */
 #define  PCI_EXP_LNKCTL2_TLS		0x000f
@@ -668,6 +671,7 @@
 #define  PCI_EXP_LNKCTL2_TLS_5_0GT	0x0002 /* Supported Speed 5GT/s */
 #define  PCI_EXP_LNKCTL2_TLS_8_0GT	0x0003 /* Supported Speed 8GT/s */
 #define  PCI_EXP_LNKCTL2_TLS_16_0GT	0x0004 /* Supported Speed 16GT/s */
+#define  PCI_EXP_LNKCTL2_TLS_32_0GT	0x0005 /* Supported Speed 32GT/s */
 #define PCI_EXP_LNKSTA2		50	/* Link Status 2 */
 #define PCI_CAP_EXP_ENDPOINT_SIZEOF_V2	52	/* v2 endpoints with link end here */
 #define PCI_EXP_SLTCAP2		52	/* Slot Capabilities 2 */
include/standard-headers/linux/virtio_ids.h +1 -0
index 32b2f94d1f..348fd0176f 100644
@@ -43,6 +43,7 @@
 #define VIRTIO_ID_INPUT        18 /* virtio input */
 #define VIRTIO_ID_VSOCK        19 /* virtio vsock transport */
 #define VIRTIO_ID_CRYPTO       20 /* virtio crypto */
+#define VIRTIO_ID_IOMMU        23 /* virtio IOMMU */
 #define VIRTIO_ID_PMEM         27 /* virtio pmem */
 
 #endif /* _LINUX_VIRTIO_IDS_H */
include/standard-headers/linux/virtio_pmem.h +3 -3
index 7e3d43b121..fc029de798 100644
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/* SPDX-License-Identifier: (GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause */
 /*
  * Definitions for virtio-pmem devices.
  *
@@ -7,8 +7,8 @@
  * Author(s): Pankaj Gupta <[email protected]>
  */
 
-#ifndef _UAPI_LINUX_VIRTIO_PMEM_H
-#define _UAPI_LINUX_VIRTIO_PMEM_H
+#ifndef _LINUX_VIRTIO_PMEM_H
+#define _LINUX_VIRTIO_PMEM_H
 
 #include "standard-headers/linux/types.h"
 #include "standard-headers/linux/virtio_ids.h"
linux-headers/asm-arm/kvm.h +15 -1
index e1f8b74558..9d379d3372 100644
@@ -214,6 +214,18 @@ struct kvm_vcpu_events {
 #define KVM_REG_ARM_FW_REG(r)		(KVM_REG_ARM | KVM_REG_SIZE_U64 | \
 					 KVM_REG_ARM_FW | ((r) & 0xffff))
 #define KVM_REG_ARM_PSCI_VERSION	KVM_REG_ARM_FW_REG(0)
+#define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_1	KVM_REG_ARM_FW_REG(1)
+	/* Higher values mean better protection. */
+#define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_1_NOT_AVAIL		0
+#define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_1_AVAIL		1
+#define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_1_NOT_REQUIRED	2
+#define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2	KVM_REG_ARM_FW_REG(2)
+	/* Higher values mean better protection. */
+#define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2_NOT_AVAIL		0
+#define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2_UNKNOWN		1
+#define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2_AVAIL		2
+#define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2_NOT_REQUIRED	3
+#define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2_ENABLED	(1U << 4)
 
 /* Device Control API: ARM VGIC */
 #define KVM_DEV_ARM_VGIC_GRP_ADDR	0
@@ -254,8 +266,10 @@ struct kvm_vcpu_events {
 #define   KVM_DEV_ARM_ITS_CTRL_RESET		4
 
 /* KVM_IRQ_LINE irq field index values */
+#define KVM_ARM_IRQ_VCPU2_SHIFT		28
+#define KVM_ARM_IRQ_VCPU2_MASK		0xf
 #define KVM_ARM_IRQ_TYPE_SHIFT		24
-#define KVM_ARM_IRQ_TYPE_MASK		0xff
+#define KVM_ARM_IRQ_TYPE_MASK		0xf
 #define KVM_ARM_IRQ_VCPU_SHIFT		16
 #define KVM_ARM_IRQ_VCPU_MASK		0xff
 #define KVM_ARM_IRQ_NUM_SHIFT		0
linux-headers/asm-arm/unistd-common.h +2 -0
index 27a9b6da27..eb5d361b11 100644
@@ -388,5 +388,7 @@
 #define __NR_fsconfig (__NR_SYSCALL_BASE + 431)
 #define __NR_fsmount (__NR_SYSCALL_BASE + 432)
 #define __NR_fspick (__NR_SYSCALL_BASE + 433)
+#define __NR_pidfd_open (__NR_SYSCALL_BASE + 434)
+#define __NR_clone3 (__NR_SYSCALL_BASE + 435)
 
 #endif /* _ASM_ARM_UNISTD_COMMON_H */
linux-headers/asm-arm64/kvm.h +20 -1
index 2431ec35a9..0ce6e49f3a 100644
@@ -229,6 +229,16 @@ struct kvm_vcpu_events {
 #define KVM_REG_ARM_FW_REG(r)		(KVM_REG_ARM64 | KVM_REG_SIZE_U64 | \
 					 KVM_REG_ARM_FW | ((r) & 0xffff))
 #define KVM_REG_ARM_PSCI_VERSION	KVM_REG_ARM_FW_REG(0)
+#define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_1	KVM_REG_ARM_FW_REG(1)
+#define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_1_NOT_AVAIL		0
+#define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_1_AVAIL		1
+#define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_1_NOT_REQUIRED	2
+#define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2	KVM_REG_ARM_FW_REG(2)
+#define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2_NOT_AVAIL		0
+#define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2_UNKNOWN		1
+#define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2_AVAIL		2
+#define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2_NOT_REQUIRED	3
+#define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2_ENABLED     	(1U << 4)
 
 /* SVE registers */
 #define KVM_REG_ARM64_SVE		(0x15 << KVM_REG_ARM_COPROC_SHIFT)
@@ -260,6 +270,13 @@ struct kvm_vcpu_events {
 	 KVM_REG_SIZE_U256 |						\
 	 ((i) & (KVM_ARM64_SVE_MAX_SLICES - 1)))
 
+/*
+ * Register values for KVM_REG_ARM64_SVE_ZREG(), KVM_REG_ARM64_SVE_PREG() and
+ * KVM_REG_ARM64_SVE_FFR() are represented in memory in an endianness-
+ * invariant layout which differs from the layout used for the FPSIMD
+ * V-registers on big-endian systems: see sigcontext.h for more explanation.
+ */
+
 #define KVM_ARM64_SVE_VQ_MIN __SVE_VQ_MIN
 #define KVM_ARM64_SVE_VQ_MAX __SVE_VQ_MAX
 
@@ -308,8 +325,10 @@ struct kvm_vcpu_events {
 #define   KVM_ARM_VCPU_TIMER_IRQ_PTIMER		1
 
 /* KVM_IRQ_LINE irq field index values */
+#define KVM_ARM_IRQ_VCPU2_SHIFT		28
+#define KVM_ARM_IRQ_VCPU2_MASK		0xf
 #define KVM_ARM_IRQ_TYPE_SHIFT		24
-#define KVM_ARM_IRQ_TYPE_MASK		0xff
+#define KVM_ARM_IRQ_TYPE_MASK		0xf
 #define KVM_ARM_IRQ_VCPU_SHIFT		16
 #define KVM_ARM_IRQ_VCPU_MASK		0xff
 #define KVM_ARM_IRQ_NUM_SHIFT		0
linux-headers/asm-generic/mman-common.h +9 -6
index abd238d0f7..63b1f506ea 100644
@@ -19,15 +19,18 @@
 #define MAP_TYPE	0x0f		/* Mask for type of mapping */
 #define MAP_FIXED	0x10		/* Interpret addr exactly */
 #define MAP_ANONYMOUS	0x20		/* don't use a file */
-#ifdef CONFIG_MMAP_ALLOW_UNINITIALIZED
-# define MAP_UNINITIALIZED 0x4000000	/* For anonymous mmap, memory could be uninitialized */
-#else
-# define MAP_UNINITIALIZED 0x0		/* Don't support this flag */
-#endif
 
-/* 0x0100 - 0x80000 flags are defined in asm-generic/mman.h */
+/* 0x0100 - 0x4000 flags are defined in asm-generic/mman.h */
+#define MAP_POPULATE		0x008000	/* populate (prefault) pagetables */
+#define MAP_NONBLOCK		0x010000	/* do not block on IO */
+#define MAP_STACK		0x020000	/* give out an address that is best suited for process/thread stacks */
+#define MAP_HUGETLB		0x040000	/* create a huge page mapping */
+#define MAP_SYNC		0x080000 /* perform synchronous page faults for the mapping */
 #define MAP_FIXED_NOREPLACE	0x100000	/* MAP_FIXED which doesn't unmap underlying mapping */
 
+#define MAP_UNINITIALIZED 0x4000000	/* For anonymous mmap, memory could be
+					 * uninitialized */
+
 /*
  * Flags for mlock
  */
linux-headers/asm-generic/mman.h +4 -6
index 653687d977..57e8195d0b 100644
@@ -9,13 +9,11 @@
 #define MAP_EXECUTABLE	0x1000		/* mark it as an executable */
 #define MAP_LOCKED	0x2000		/* pages are locked */
 #define MAP_NORESERVE	0x4000		/* don't check for reservations */
-#define MAP_POPULATE	0x8000		/* populate (prefault) pagetables */
-#define MAP_NONBLOCK	0x10000		/* do not block on IO */
-#define MAP_STACK	0x20000		/* give out an address that is best suited for process/thread stacks */
-#define MAP_HUGETLB	0x40000		/* create a huge page mapping */
-#define MAP_SYNC	0x80000		/* perform synchronous page faults for the mapping */
 
-/* Bits [26:31] are reserved, see mman-common.h for MAP_HUGETLB usage */
+/*
+ * Bits [26:31] are reserved, see asm-generic/hugetlb_encode.h
+ * for MAP_HUGETLB usage
+ */
 
 #define MCL_CURRENT	1		/* lock all current mappings */
 #define MCL_FUTURE	2		/* lock all future mappings */
linux-headers/asm-generic/unistd.h +7 -1
index a87904daf1..1be0e798e3 100644
@@ -844,9 +844,15 @@ __SYSCALL(__NR_fsconfig, sys_fsconfig)
 __SYSCALL(__NR_fsmount, sys_fsmount)
 #define __NR_fspick 433
 __SYSCALL(__NR_fspick, sys_fspick)
+#define __NR_pidfd_open 434
+__SYSCALL(__NR_pidfd_open, sys_pidfd_open)
+#ifdef __ARCH_WANT_SYS_CLONE3
+#define __NR_clone3 435
+__SYSCALL(__NR_clone3, sys_clone3)
+#endif
 
 #undef __NR_syscalls
-#define __NR_syscalls 434
+#define __NR_syscalls 436
 
 /*
  * 32 bit systems traditionally used different
linux-headers/asm-mips/unistd_n32.h +1 -0
index fb988de900..7dffe8e34e 100644
@@ -363,6 +363,7 @@
 #define __NR_fsconfig	(__NR_Linux + 431)
 #define __NR_fsmount	(__NR_Linux + 432)
 #define __NR_fspick	(__NR_Linux + 433)
+#define __NR_pidfd_open	(__NR_Linux + 434)
 
 
 #endif /* _ASM_MIPS_UNISTD_N32_H */
linux-headers/asm-mips/unistd_n64.h +1 -0
index 17359163c9..f4592d6fc5 100644
@@ -339,6 +339,7 @@
 #define __NR_fsconfig	(__NR_Linux + 431)
 #define __NR_fsmount	(__NR_Linux + 432)
 #define __NR_fspick	(__NR_Linux + 433)
+#define __NR_pidfd_open	(__NR_Linux + 434)
 
 
 #endif /* _ASM_MIPS_UNISTD_N64_H */
linux-headers/asm-mips/unistd_o32.h +1 -0
index 83c8d8fb83..04c6728352 100644
@@ -409,6 +409,7 @@
 #define __NR_fsconfig	(__NR_Linux + 431)
 #define __NR_fsmount	(__NR_Linux + 432)
 #define __NR_fspick	(__NR_Linux + 433)
+#define __NR_pidfd_open	(__NR_Linux + 434)
 
 
 #endif /* _ASM_MIPS_UNISTD_O32_H */
linux-headers/asm-powerpc/mman.h +1 -5
index 1c2b3fca05..8db7c2a3be 100644
@@ -21,15 +21,11 @@
 #define MAP_DENYWRITE	0x0800		/* ETXTBSY */
 #define MAP_EXECUTABLE	0x1000		/* mark it as an executable */
 
+
 #define MCL_CURRENT     0x2000          /* lock all currently mapped pages */
 #define MCL_FUTURE      0x4000          /* lock all additions to address space */
 #define MCL_ONFAULT	0x8000		/* lock all pages that are faulted in */
 
-#define MAP_POPULATE	0x8000		/* populate (prefault) pagetables */
-#define MAP_NONBLOCK	0x10000		/* do not block on IO */
-#define MAP_STACK	0x20000		/* give out an address that is best suited for process/thread stacks */
-#define MAP_HUGETLB	0x40000		/* create a huge page mapping */
-
 /* Override any generic PKEY permission defines */
 #define PKEY_DISABLE_EXECUTE   0x4
 #undef PKEY_ACCESS_MASK
linux-headers/asm-powerpc/unistd_32.h +2 -0
index 04cb2d3e61..5584cc1b4f 100644
@@ -416,6 +416,8 @@
 #define __NR_fsconfig	431
 #define __NR_fsmount	432
 #define __NR_fspick	433
+#define __NR_pidfd_open	434
+#define __NR_clone3	435
 
 
 #endif /* _ASM_POWERPC_UNISTD_32_H */
linux-headers/asm-powerpc/unistd_64.h +2 -0
index b1e6921490..251bcff77e 100644
@@ -388,6 +388,8 @@
 #define __NR_fsconfig	431
 #define __NR_fsmount	432
 #define __NR_fspick	433
+#define __NR_pidfd_open	434
+#define __NR_clone3	435
 
 
 #endif /* _ASM_POWERPC_UNISTD_64_H */
linux-headers/asm-s390/unistd_32.h +2 -0
index 941853f3e9..7cce3ee296 100644
@@ -406,5 +406,7 @@
 #define __NR_fsconfig 431
 #define __NR_fsmount 432
 #define __NR_fspick 433
+#define __NR_pidfd_open 434
+#define __NR_clone3 435
 
 #endif /* _ASM_S390_UNISTD_32_H */
linux-headers/asm-s390/unistd_64.h +2 -0
index 90271d7f82..2371ff1e7a 100644
@@ -354,5 +354,7 @@
 #define __NR_fsconfig 431
 #define __NR_fsmount 432
 #define __NR_fspick 433
+#define __NR_pidfd_open 434
+#define __NR_clone3 435
 
 #endif /* _ASM_S390_UNISTD_64_H */
linux-headers/asm-x86/kvm.h +21 -7
index 6e7dd792e4..503d3f42da 100644
@@ -378,23 +378,24 @@ struct kvm_sync_regs {
 	struct kvm_vcpu_events events;
 };
 
-#define KVM_X86_QUIRK_LINT0_REENABLED	(1 << 0)
-#define KVM_X86_QUIRK_CD_NW_CLEARED	(1 << 1)
-#define KVM_X86_QUIRK_LAPIC_MMIO_HOLE	(1 << 2)
-#define KVM_X86_QUIRK_OUT_7E_INC_RIP	(1 << 3)
+#define KVM_X86_QUIRK_LINT0_REENABLED	   (1 << 0)
+#define KVM_X86_QUIRK_CD_NW_CLEARED	   (1 << 1)
+#define KVM_X86_QUIRK_LAPIC_MMIO_HOLE	   (1 << 2)
+#define KVM_X86_QUIRK_OUT_7E_INC_RIP	   (1 << 3)
+#define KVM_X86_QUIRK_MISC_ENABLE_NO_MWAIT (1 << 4)
 
 #define KVM_STATE_NESTED_FORMAT_VMX	0
-#define KVM_STATE_NESTED_FORMAT_SVM	1
+#define KVM_STATE_NESTED_FORMAT_SVM	1	/* unused */
 
 #define KVM_STATE_NESTED_GUEST_MODE	0x00000001
 #define KVM_STATE_NESTED_RUN_PENDING	0x00000002
 #define KVM_STATE_NESTED_EVMCS		0x00000004
 
-#define KVM_STATE_NESTED_VMX_VMCS_SIZE	0x1000
-
 #define KVM_STATE_NESTED_SMM_GUEST_MODE	0x00000001
 #define KVM_STATE_NESTED_SMM_VMXON	0x00000002
 
+#define KVM_STATE_NESTED_VMX_VMCS_SIZE	0x1000
+
 struct kvm_vmx_nested_state_data {
 	__u8 vmcs12[KVM_STATE_NESTED_VMX_VMCS_SIZE];
 	__u8 shadow_vmcs12[KVM_STATE_NESTED_VMX_VMCS_SIZE];
@@ -432,4 +433,17 @@ struct kvm_nested_state {
 	} data;
 };
 
+/* for KVM_CAP_PMU_EVENT_FILTER */
+struct kvm_pmu_event_filter {
+	__u32 action;
+	__u32 nevents;
+	__u32 fixed_counter_bitmap;
+	__u32 flags;
+	__u32 pad[4];
+	__u64 events[0];
+};
+
+#define KVM_PMU_EVENT_ALLOW 0
+#define KVM_PMU_EVENT_DENY 1
+
 #endif /* _ASM_X86_KVM_H */
linux-headers/asm-x86/unistd_32.h +2 -0
index 57bb48854c..e8ebec1cdc 100644
@@ -424,5 +424,7 @@
 #define __NR_fsconfig 431
 #define __NR_fsmount 432
 #define __NR_fspick 433
+#define __NR_pidfd_open 434
+#define __NR_clone3 435
 
 #endif /* _ASM_X86_UNISTD_32_H */
linux-headers/asm-x86/unistd_64.h +2 -0
index fe6aa0688a..a2f863d549 100644
@@ -346,5 +346,7 @@
 #define __NR_fsconfig 431
 #define __NR_fsmount 432
 #define __NR_fspick 433
+#define __NR_pidfd_open 434
+#define __NR_clone3 435
 
 #endif /* _ASM_X86_UNISTD_64_H */
linux-headers/asm-x86/unistd_x32.h +2 -0
index 09cca49ba7..4cdc67d848 100644
@@ -299,6 +299,8 @@
 #define __NR_fsconfig (__X32_SYSCALL_BIT + 431)
 #define __NR_fsmount (__X32_SYSCALL_BIT + 432)
 #define __NR_fspick (__X32_SYSCALL_BIT + 433)
+#define __NR_pidfd_open (__X32_SYSCALL_BIT + 434)
+#define __NR_clone3 (__X32_SYSCALL_BIT + 435)
 #define __NR_rt_sigaction (__X32_SYSCALL_BIT + 512)
 #define __NR_rt_sigreturn (__X32_SYSCALL_BIT + 513)
 #define __NR_ioctl (__X32_SYSCALL_BIT + 514)
linux-headers/linux/kvm.h +9 -3
index c8423e760c..a6a65ace2c 100644
@@ -116,7 +116,7 @@ struct kvm_irq_level {
 	 * ACPI gsi notion of irq.
 	 * For IA-64 (APIC model) IOAPIC0: irq 0-23; IOAPIC1: irq 24-47..
 	 * For X86 (standard AT mode) PIC0/1: irq 0-15. IOAPIC0: 0-23..
-	 * For ARM: See Documentation/virtual/kvm/api.txt
+	 * For ARM: See Documentation/virt/kvm/api.txt
 	 */
 	union {
 		__u32 irq;
@@ -696,9 +696,11 @@ struct kvm_ioeventfd {
 #define KVM_X86_DISABLE_EXITS_MWAIT          (1 << 0)
 #define KVM_X86_DISABLE_EXITS_HLT            (1 << 1)
 #define KVM_X86_DISABLE_EXITS_PAUSE          (1 << 2)
+#define KVM_X86_DISABLE_EXITS_CSTATE         (1 << 3)
 #define KVM_X86_DISABLE_VALID_EXITS          (KVM_X86_DISABLE_EXITS_MWAIT | \
                                               KVM_X86_DISABLE_EXITS_HLT | \
-                                              KVM_X86_DISABLE_EXITS_PAUSE)
+                                              KVM_X86_DISABLE_EXITS_PAUSE | \
+                                              KVM_X86_DISABLE_EXITS_CSTATE)
 
 /* for KVM_ENABLE_CAP */
 struct kvm_enable_cap {
@@ -993,6 +995,8 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_ARM_SVE 170
 #define KVM_CAP_ARM_PTRAUTH_ADDRESS 171
 #define KVM_CAP_ARM_PTRAUTH_GENERIC 172
+#define KVM_CAP_PMU_EVENT_FILTER 173
+#define KVM_CAP_ARM_IRQ_LINE_LAYOUT_2 174
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
@@ -1083,7 +1087,7 @@ struct kvm_xen_hvm_config {
  *
  * KVM_IRQFD_FLAG_RESAMPLE indicates resamplefd is valid and specifies
  * the irqfd to operate in resampling mode for level triggered interrupt
- * emulation.  See Documentation/virtual/kvm/api.txt.
+ * emulation.  See Documentation/virt/kvm/api.txt.
  */
 #define KVM_IRQFD_FLAG_RESAMPLE (1 << 1)
 
@@ -1327,6 +1331,8 @@ struct kvm_s390_ucas_mapping {
 #define KVM_PPC_GET_RMMU_INFO	  _IOW(KVMIO,  0xb0, struct kvm_ppc_rmmu_info)
 /* Available with KVM_CAP_PPC_GET_CPU_CHAR */
 #define KVM_PPC_GET_CPU_CHAR	  _IOR(KVMIO,  0xb1, struct kvm_ppc_cpu_char)
+/* Available with KVM_CAP_PMU_EVENT_FILTER */
+#define KVM_SET_PMU_EVENT_FILTER  _IOW(KVMIO,  0xb2, struct kvm_pmu_event_filter)
 
 /* ioctl for vm fd */
 #define KVM_CREATE_DEVICE	  _IOWR(KVMIO,  0xe0, struct kvm_create_device)
linux-headers/linux/psp-sev.h +1 -4
index 36bbe17d8f..34c39690c0 100644
@@ -1,3 +1,4 @@
+/* SPDX-License-Identifier: GPL-2.0-only WITH Linux-syscall-note */
 /*
  * Userspace interface for AMD Secure Encrypted Virtualization (SEV)
  * platform management commands.
@@ -7,10 +8,6 @@
  * Author: Brijesh Singh <[email protected]>
  *
  * SEV API specification is available at: https://developer.amd.com/sev/
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
  */
 
 #ifndef __PSP_SEV_USER_H__
Signature · 1 line
2.20.1
author
#2 lore raw

RFC v2 2/3 intc/arm_gic: Support IRQ injection for more than 256 vpus

In reply to Eric Auger

Host kernels that expose the KVM_CAP_ARM_IRQ_LINE_LAYOUT_2 capability allow injection of interrupts along with vcpu ids larger than 255. Let's encode the vpcu id on 12 bits according to the upgraded KVM_IRQ_LINE ABI when needed.

Given that we have two callsites that need to assemble the value for kvm_set_irq(), a new helper routine, kvm_arm_set_irq is introduced.

Without that patch qemu exits with "kvm_set_irq: Invalid argument" message.

---
 hw/intc/arm_gic_kvm.c |  7 ++-----
 target/arm/cpu.c      | 10 ++++------
 target/arm/kvm.c      | 16 ++++++++++++++++
 target/arm/kvm_arm.h  |  1 +
 4 files changed, 23 insertions(+), 11 deletions(-)
hw/intc/arm_gic_kvm.c +2 -5
index b56fda144f..9deb15e7e6 100644
@@ -55,7 +55,7 @@ void kvm_arm_gic_set_irq(uint32_t num_irq, int irq, int level)
      * has separate fields in the irq number for type,
      * CPU number and interrupt number.
      */
-    int kvm_irq, irqtype, cpu;
+    int irqtype, cpu;
 
     if (irq < (num_irq - GIC_INTERNAL)) {
         /* External interrupt. The kernel numbers these like the GIC
@@ -72,10 +72,7 @@ void kvm_arm_gic_set_irq(uint32_t num_irq, int irq, int level)
         cpu = irq / GIC_INTERNAL;
         irq %= GIC_INTERNAL;
     }
-    kvm_irq = (irqtype << KVM_ARM_IRQ_TYPE_SHIFT)
-        | (cpu << KVM_ARM_IRQ_VCPU_SHIFT) | irq;
-
-    kvm_set_irq(kvm_state, kvm_irq, !!level);
+    kvm_arm_set_irq(cpu, irqtype, irq, !!level);
 }
 
 static void kvm_arm_gicv2_set_irq(void *opaque, int irq, int level)
target/arm/cpu.c +4 -6
index 2399c14471..13813fb213 100644
@@ -576,16 +576,16 @@ static void arm_cpu_kvm_set_irq(void *opaque, int irq, int level)
     ARMCPU *cpu = opaque;
     CPUARMState *env = &cpu->env;
     CPUState *cs = CPU(cpu);
-    int kvm_irq = KVM_ARM_IRQ_TYPE_CPU << KVM_ARM_IRQ_TYPE_SHIFT;
     uint32_t linestate_bit;
+    int irq_id;
 
     switch (irq) {
     case ARM_CPU_IRQ:
-        kvm_irq |= KVM_ARM_IRQ_CPU_IRQ;
+        irq_id = KVM_ARM_IRQ_CPU_IRQ;
         linestate_bit = CPU_INTERRUPT_HARD;
         break;
     case ARM_CPU_FIQ:
-        kvm_irq |= KVM_ARM_IRQ_CPU_FIQ;
+        irq_id = KVM_ARM_IRQ_CPU_FIQ;
         linestate_bit = CPU_INTERRUPT_FIQ;
         break;
     default:
@@ -597,9 +597,7 @@ static void arm_cpu_kvm_set_irq(void *opaque, int irq, int level)
     } else {
         env->irq_line_state &= ~linestate_bit;
     }
-
-    kvm_irq |= cs->cpu_index << KVM_ARM_IRQ_VCPU_SHIFT;
-    kvm_set_irq(kvm_state, kvm_irq, level ? 1 : 0);
+    kvm_arm_set_irq(cs->cpu_index, KVM_ARM_IRQ_TYPE_CPU, irq_id, !!level);
 #endif
 }
 
target/arm/kvm.c +16 -0
index b2eaa50b8d..6cdfa2204f 100644
@@ -744,6 +744,22 @@ int kvm_arm_vgic_probe(void)
     }
 }
 
+int kvm_arm_set_irq(int cpu, int irqtype, int irq, int level)
+{
+    int kvm_irq = 0;
+
+    kvm_irq = (irqtype << KVM_ARM_IRQ_TYPE_SHIFT) | irq;
+
+    if (cpu != 0) {
+        int cpu_idx2 = cpu / 256;
+        int cpu_idx1 = cpu % 256;
+
+        kvm_irq |= (cpu_idx1 << KVM_ARM_IRQ_VCPU_SHIFT) |
+           ((cpu_idx2 & KVM_ARM_IRQ_VCPU2_MASK) << KVM_ARM_IRQ_VCPU2_SHIFT);
+    }
+    return kvm_set_irq(kvm_state, kvm_irq, !!level);
+}
+
 int kvm_arch_fixup_msi_route(struct kvm_irq_routing_entry *route,
                              uint64_t address, uint32_t data, PCIDevice *dev)
 {
target/arm/kvm_arm.h +1 -0
index b3106c8600..b4e19457a0 100644
@@ -253,6 +253,7 @@ int kvm_arm_vgic_probe(void);
 
 void kvm_arm_pmu_set_irq(CPUState *cs, int irq);
 void kvm_arm_pmu_init(CPUState *cs);
+int kvm_arm_set_irq(int cpu, int irqtype, int irq, int level);
 
 #else
 
Signature · 1 line
2.20.1
#3 lore raw

Re: [Qemu-arm] [Qemu-devel] [RFC v2 2/3] intc/arm_gic: Support IRQ injection for more than 256 vpus

Quoted 86 lines · Eric Auger

Host kernels that expose the KVM_CAP_ARM_IRQ_LINE_LAYOUT_2 capability allow injection of interrupts along with vcpu ids larger than 255. Let's encode the vpcu id on 12 bits according to the upgraded KVM_IRQ_LINE ABI when needed.

Given that we have two callsites that need to assemble the value for kvm_set_irq(), a new helper routine, kvm_arm_set_irq is introduced.

Without that patch qemu exits with "kvm_set_irq: Invalid argument" message.

---
 hw/intc/arm_gic_kvm.c |  7 ++-----
 target/arm/cpu.c      | 10 ++++------
 target/arm/kvm.c      | 16 ++++++++++++++++
 target/arm/kvm_arm.h  |  1 +
 4 files changed, 23 insertions(+), 11 deletions(-)
hw/intc/arm_gic_kvm.c +2 -5
index b56fda144f..9deb15e7e6 100644
@@ -55,7 +55,7 @@ void kvm_arm_gic_set_irq(uint32_t num_irq, int irq, int level)
      * has separate fields in the irq number for type,
      * CPU number and interrupt number.
      */
-    int kvm_irq, irqtype, cpu;
+    int irqtype, cpu;
 
     if (irq < (num_irq - GIC_INTERNAL)) {
         /* External interrupt. The kernel numbers these like the GIC
@@ -72,10 +72,7 @@ void kvm_arm_gic_set_irq(uint32_t num_irq, int irq, int level)
         cpu = irq / GIC_INTERNAL;
         irq %= GIC_INTERNAL;
     }
-    kvm_irq = (irqtype << KVM_ARM_IRQ_TYPE_SHIFT)
-        | (cpu << KVM_ARM_IRQ_VCPU_SHIFT) | irq;
-
-    kvm_set_irq(kvm_state, kvm_irq, !!level);
+    kvm_arm_set_irq(cpu, irqtype, irq, !!level);
 }
 
 static void kvm_arm_gicv2_set_irq(void *opaque, int irq, int level)
target/arm/cpu.c +4 -6
index 2399c14471..13813fb213 100644
@@ -576,16 +576,16 @@ static void arm_cpu_kvm_set_irq(void *opaque, int irq, int level)
     ARMCPU *cpu = opaque;
     CPUARMState *env = &cpu->env;
     CPUState *cs = CPU(cpu);
-    int kvm_irq = KVM_ARM_IRQ_TYPE_CPU << KVM_ARM_IRQ_TYPE_SHIFT;
     uint32_t linestate_bit;
+    int irq_id;
 
     switch (irq) {
     case ARM_CPU_IRQ:
-        kvm_irq |= KVM_ARM_IRQ_CPU_IRQ;
+        irq_id = KVM_ARM_IRQ_CPU_IRQ;
         linestate_bit = CPU_INTERRUPT_HARD;
         break;
     case ARM_CPU_FIQ:
-        kvm_irq |= KVM_ARM_IRQ_CPU_FIQ;
+        irq_id = KVM_ARM_IRQ_CPU_FIQ;
         linestate_bit = CPU_INTERRUPT_FIQ;
         break;
     default:
@@ -597,9 +597,7 @@ static void arm_cpu_kvm_set_irq(void *opaque, int irq, int level)
     } else {
         env->irq_line_state &= ~linestate_bit;
     }
-
-    kvm_irq |= cs->cpu_index << KVM_ARM_IRQ_VCPU_SHIFT;
-    kvm_set_irq(kvm_state, kvm_irq, level ? 1 : 0);
+    kvm_arm_set_irq(cs->cpu_index, KVM_ARM_IRQ_TYPE_CPU, irq_id, !!level);
 #endif
 }
 
target/arm/kvm.c +0 -0
index b2eaa50b8d..6cdfa2204f 100644

target/arm/kvm.c int kvm_arm_vgic_probe(void) · line 749

Quoted 6 lines
target/arm/kvm.c +3 -0
@@ -744,6 +744,22 @@ int kvm_arm_vgic_probe(void)
     }
 }
 
+int kvm_arm_set_irq(int cpu, int irqtype, int irq, int level)
+{
+    int kvm_irq = 0;

No need to init to zero, and could just immediately init with the line below instead.

target/arm/kvm.c int kvm_arm_vgic_probe(void) · line 753

Quoted 4 lines
target/arm/kvm.c +4 -0
+
+    kvm_irq = (irqtype << KVM_ARM_IRQ_TYPE_SHIFT) | irq;
+
+    if (cpu != 0) {

No need for this 'if'

target/arm/kvm.c int kvm_arm_vgic_probe(void) · line 758

Quoted 5 lines
target/arm/kvm.c +5 -0
+        int cpu_idx2 = cpu / 256;
+        int cpu_idx1 = cpu % 256;
+
+        kvm_irq |= (cpu_idx1 << KVM_ARM_IRQ_VCPU_SHIFT) |
+           ((cpu_idx2 & KVM_ARM_IRQ_VCPU2_MASK) << KVM_ARM_IRQ_VCPU2_SHIFT);

Masking should be unnecessary as the only way it'll do anything is if we have vcpus >= 4096, which I imagine will never happen or will be guarded against happening elsewhere. Silently masking doesn't look right anyway, so I'd either add an assert(cpu_idx2 < 16) and drop the masking or just drop the masking.

Quoted 23 lines
target/arm/kvm.c +4 -0
+    }
+    return kvm_set_irq(kvm_state, kvm_irq, !!level);
+}
+
 int kvm_arch_fixup_msi_route(struct kvm_irq_routing_entry *route,
                              uint64_t address, uint32_t data, PCIDevice *dev)
 {
target/arm/kvm_arm.h +1 -0
index b3106c8600..b4e19457a0 100644
@@ -253,6 +253,7 @@ int kvm_arm_vgic_probe(void);
 
 void kvm_arm_pmu_set_irq(CPUState *cs, int irq);
 void kvm_arm_pmu_init(CPUState *cs);
+int kvm_arm_set_irq(int cpu, int irqtype, int irq, int level);
 
 #else
 
Signature · 1 line
2.20.1

Thanks,
drew

author
#4 lore raw

Re: [Qemu-arm] [Qemu-devel] [RFC v2 2/3] intc/arm_gic: Support IRQ injection for more than 256 vpus

Hi Drew,

Quoted 110 lines · Andrew Jones
Quoted 86 lines · Eric Auger

Host kernels that expose the KVM_CAP_ARM_IRQ_LINE_LAYOUT_2 capability allow injection of interrupts along with vcpu ids larger than 255. Let's encode the vpcu id on 12 bits according to the upgraded KVM_IRQ_LINE ABI when needed.

Given that we have two callsites that need to assemble the value for kvm_set_irq(), a new helper routine, kvm_arm_set_irq is introduced.

Without that patch qemu exits with "kvm_set_irq: Invalid argument" message.

---
 hw/intc/arm_gic_kvm.c |  7 ++-----
 target/arm/cpu.c      | 10 ++++------
 target/arm/kvm.c      | 16 ++++++++++++++++
 target/arm/kvm_arm.h  |  1 +
 4 files changed, 23 insertions(+), 11 deletions(-)
hw/intc/arm_gic_kvm.c +2 -5
index b56fda144f..9deb15e7e6 100644
@@ -55,7 +55,7 @@ void kvm_arm_gic_set_irq(uint32_t num_irq, int irq, int level)
      * has separate fields in the irq number for type,
      * CPU number and interrupt number.
      */
-    int kvm_irq, irqtype, cpu;
+    int irqtype, cpu;
 
     if (irq < (num_irq - GIC_INTERNAL)) {
         /* External interrupt. The kernel numbers these like the GIC
@@ -72,10 +72,7 @@ void kvm_arm_gic_set_irq(uint32_t num_irq, int irq, int level)
         cpu = irq / GIC_INTERNAL;
         irq %= GIC_INTERNAL;
     }
-    kvm_irq = (irqtype << KVM_ARM_IRQ_TYPE_SHIFT)
-        | (cpu << KVM_ARM_IRQ_VCPU_SHIFT) | irq;
-
-    kvm_set_irq(kvm_state, kvm_irq, !!level);
+    kvm_arm_set_irq(cpu, irqtype, irq, !!level);
 }
 
 static void kvm_arm_gicv2_set_irq(void *opaque, int irq, int level)
target/arm/cpu.c +4 -6
index 2399c14471..13813fb213 100644
@@ -576,16 +576,16 @@ static void arm_cpu_kvm_set_irq(void *opaque, int irq, int level)
     ARMCPU *cpu = opaque;
     CPUARMState *env = &cpu->env;
     CPUState *cs = CPU(cpu);
-    int kvm_irq = KVM_ARM_IRQ_TYPE_CPU << KVM_ARM_IRQ_TYPE_SHIFT;
     uint32_t linestate_bit;
+    int irq_id;
 
     switch (irq) {
     case ARM_CPU_IRQ:
-        kvm_irq |= KVM_ARM_IRQ_CPU_IRQ;
+        irq_id = KVM_ARM_IRQ_CPU_IRQ;
         linestate_bit = CPU_INTERRUPT_HARD;
         break;
     case ARM_CPU_FIQ:
-        kvm_irq |= KVM_ARM_IRQ_CPU_FIQ;
+        irq_id = KVM_ARM_IRQ_CPU_FIQ;
         linestate_bit = CPU_INTERRUPT_FIQ;
         break;
     default:
@@ -597,9 +597,7 @@ static void arm_cpu_kvm_set_irq(void *opaque, int irq, int level)
     } else {
         env->irq_line_state &= ~linestate_bit;
     }
-
-    kvm_irq |= cs->cpu_index << KVM_ARM_IRQ_VCPU_SHIFT;
-    kvm_set_irq(kvm_state, kvm_irq, level ? 1 : 0);
+    kvm_arm_set_irq(cs->cpu_index, KVM_ARM_IRQ_TYPE_CPU, irq_id, !!level);
 #endif
 }
 
target/arm/kvm.c +0 -0
index b2eaa50b8d..6cdfa2204f 100644

target/arm/kvm.c int kvm_arm_vgic_probe(void) · line 749

Quoted 6 lines
target/arm/kvm.c +3 -0
@@ -744,6 +744,22 @@ int kvm_arm_vgic_probe(void)
     }
 }
 
+int kvm_arm_set_irq(int cpu, int irqtype, int irq, int level)
+{
+    int kvm_irq = 0;

No need to init to zero, and could just immediately init with the line below instead.

target/arm/kvm.c int kvm_arm_vgic_probe(void) · line 753

Quoted 4 lines
target/arm/kvm.c +4 -0
+
+    kvm_irq = (irqtype << KVM_ARM_IRQ_TYPE_SHIFT) | irq;
+
+    if (cpu != 0) {

No need for this 'if'

target/arm/kvm.c int kvm_arm_vgic_probe(void) · line 758

Quoted 5 lines
target/arm/kvm.c +5 -0
+        int cpu_idx2 = cpu / 256;
+        int cpu_idx1 = cpu % 256;
+
+        kvm_irq |= (cpu_idx1 << KVM_ARM_IRQ_VCPU_SHIFT) |
+           ((cpu_idx2 & KVM_ARM_IRQ_VCPU2_MASK) << KVM_ARM_IRQ_VCPU2_SHIFT);
Quoted 5 lines

Masking should be unnecessary as the only way it'll do anything is if we have vcpus >= 4096, which I imagine will never happen or will be guarded against happening elsewhere. Silently masking doesn't look right anyway, so I'd either add an assert(cpu_idx2 < 16) and drop the masking or just drop the masking.

All that makes sense.

Thank you for the review.

Eric

Quoted 28 lines
Quoted 24 lines
target/arm/kvm.c +4 -0
+    }
+    return kvm_set_irq(kvm_state, kvm_irq, !!level);
+}
+
 int kvm_arch_fixup_msi_route(struct kvm_irq_routing_entry *route,
                              uint64_t address, uint32_t data, PCIDevice *dev)
 {
target/arm/kvm_arm.h +1 -0
index b3106c8600..b4e19457a0 100644
@@ -253,6 +253,7 @@ int kvm_arm_vgic_probe(void);
 
 void kvm_arm_pmu_set_irq(CPUState *cs, int irq);
 void kvm_arm_pmu_init(CPUState *cs);
+int kvm_arm_set_irq(int cpu, int irqtype, int irq, int level);
 
 #else
 
Signature · 1 line
2.20.1

Thanks,
drew

author
#5 lore raw

RFC v2 3/3 virt: Check KVM_CAP_ARM_IRQ_LINE_LAYOUT_2 for smp_cpus > 256

In reply to Eric Auger

Host kernel within [4.18, 5.3] report an erroneous KVM_MAX_VCPUS=512 for ARM. The actual capability to instantiate more than 256 vcpus was fixed in 5.4 with the upgrade of the KVM_IRQ_LINE ABI to support vcpu id encoded on 12 bits instead of 8 and a redistributor consuming a single KVM IO device instead of 2.

So let's check this capability when attempting to use more than 256 vcpus.

---
 hw/arm/virt.c        |  4 ++++
 target/arm/kvm.c     | 21 +++++++++++++++++++++
 target/arm/kvm_arm.h | 15 +++++++++++++++
 3 files changed, 40 insertions(+)
hw/arm/virt.c +4 -0
index 0d1629ccb3..465e3140f7 100644
@@ -1575,6 +1575,10 @@ static void machvirt_init(MachineState *machine)
         virt_max_cpus = GIC_NCPU;
     }
 
+    if (kvm_arm_irq_line_layout_mismatch(MACHINE(vms), max_cpus)) {
+        exit(1);
+    }
+
     if (max_cpus > virt_max_cpus) {
         error_report("Number of SMP CPUs requested (%d) exceeds max CPUs "
                      "supported by machine 'mach-virt' (%d)",
target/arm/kvm.c +21 -0
index 6cdfa2204f..b601e2f35a 100644
@@ -171,6 +171,27 @@ bool kvm_arm_pmu_supported(CPUState *cpu)
     return kvm_check_extension(s, KVM_CAP_ARM_PMU_V3);
 }
 
+bool kvm_arm_irq_line_layout_mismatch(MachineState *ms, int vcpus)
+{
+    KVMState *s;
+    bool ret;
+
+    if (!kvm_enabled()) {
+        return false;
+    }
+
+    s = KVM_STATE(ms->accelerator);
+
+    ret = vcpus > 256 &&
+          !kvm_check_extension(s, KVM_CAP_ARM_IRQ_LINE_LAYOUT_2);
+
+    if (ret) {
+        error_report("Using more than 256 vcpus requires a host kernel "
+                     "with KVM_CAP_ARM_IRQ_LINE_LAYOUT_2");
+    }
+    return ret;
+}
+
 int kvm_arm_get_max_vm_ipa_size(MachineState *ms)
 {
     KVMState *s = KVM_STATE(ms->accelerator);
target/arm/kvm_arm.h +15 -0
index b4e19457a0..d893d950d8 100644
@@ -233,6 +233,16 @@ bool kvm_arm_pmu_supported(CPUState *cs);
  */
 int kvm_arm_get_max_vm_ipa_size(MachineState *ms);
 
+/**
+ * kvm_arm_irq_line_layout_mismatch - Returns whether the number of vcpus
+ * exceeds the limit imposed by the legacy KVM_IRQ_LINE ARM layout
+ * (without the vcpu2_index field).
+ *
+ * @ms: Machine state handle
+ * @vcpus: number of vcpus
+ */
+bool kvm_arm_irq_line_layout_mismatch(MachineState *ms, int vcpus);
+
 /**
  * kvm_arm_sync_mpstate_to_kvm
  * @cpu: ARMCPU
@@ -281,6 +291,11 @@ static inline int kvm_arm_get_max_vm_ipa_size(MachineState *ms)
     return -ENOENT;
 }
 
+static inline bool kvm_arm_irq_line_layout_mismatch(MachineState *ms, int vcpus)
+{
+    return false;
+}
+
 static inline int kvm_arm_vgic_probe(void)
 {
     return 0;
Signature · 1 line
2.20.1
Reviewed-by on v2 3/3
#6 lore raw

Re: [Qemu-arm] [Qemu-devel] [RFC v2 3/3] virt: Check KVM_CAP_ARM_IRQ_LINE_LAYOUT_2 for smp_cpus > 256

Quoted 100 lines · Eric Auger

Host kernel within [4.18, 5.3] report an erroneous KVM_MAX_VCPUS=512 for ARM. The actual capability to instantiate more than 256 vcpus was fixed in 5.4 with the upgrade of the KVM_IRQ_LINE ABI to support vcpu id encoded on 12 bits instead of 8 and a redistributor consuming a single KVM IO device instead of 2.

So let's check this capability when attempting to use more than 256 vcpus.

---
 hw/arm/virt.c        |  4 ++++
 target/arm/kvm.c     | 21 +++++++++++++++++++++
 target/arm/kvm_arm.h | 15 +++++++++++++++
 3 files changed, 40 insertions(+)
hw/arm/virt.c +4 -0
index 0d1629ccb3..465e3140f7 100644
@@ -1575,6 +1575,10 @@ static void machvirt_init(MachineState *machine)
         virt_max_cpus = GIC_NCPU;
     }
 
+    if (kvm_arm_irq_line_layout_mismatch(MACHINE(vms), max_cpus)) {
+        exit(1);
+    }
+
     if (max_cpus > virt_max_cpus) {
         error_report("Number of SMP CPUs requested (%d) exceeds max CPUs "
                      "supported by machine 'mach-virt' (%d)",
target/arm/kvm.c +21 -0
index 6cdfa2204f..b601e2f35a 100644
@@ -171,6 +171,27 @@ bool kvm_arm_pmu_supported(CPUState *cpu)
     return kvm_check_extension(s, KVM_CAP_ARM_PMU_V3);
 }
 
+bool kvm_arm_irq_line_layout_mismatch(MachineState *ms, int vcpus)
+{
+    KVMState *s;
+    bool ret;
+
+    if (!kvm_enabled()) {
+        return false;
+    }
+
+    s = KVM_STATE(ms->accelerator);
+
+    ret = vcpus > 256 &&
+          !kvm_check_extension(s, KVM_CAP_ARM_IRQ_LINE_LAYOUT_2);
+
+    if (ret) {
+        error_report("Using more than 256 vcpus requires a host kernel "
+                     "with KVM_CAP_ARM_IRQ_LINE_LAYOUT_2");
+    }
+    return ret;
+}
+
 int kvm_arm_get_max_vm_ipa_size(MachineState *ms)
 {
     KVMState *s = KVM_STATE(ms->accelerator);
target/arm/kvm_arm.h +15 -0
index b4e19457a0..d893d950d8 100644
@@ -233,6 +233,16 @@ bool kvm_arm_pmu_supported(CPUState *cs);
  */
 int kvm_arm_get_max_vm_ipa_size(MachineState *ms);
 
+/**
+ * kvm_arm_irq_line_layout_mismatch - Returns whether the number of vcpus
+ * exceeds the limit imposed by the legacy KVM_IRQ_LINE ARM layout
+ * (without the vcpu2_index field).
+ *
+ * @ms: Machine state handle
+ * @vcpus: number of vcpus
+ */
+bool kvm_arm_irq_line_layout_mismatch(MachineState *ms, int vcpus);
+
 /**
  * kvm_arm_sync_mpstate_to_kvm
  * @cpu: ARMCPU
@@ -281,6 +291,11 @@ static inline int kvm_arm_get_max_vm_ipa_size(MachineState *ms)
     return -ENOENT;
 }
 
+static inline bool kvm_arm_irq_line_layout_mismatch(MachineState *ms, int vcpus)
+{
+    return false;
+}
+
 static inline int kvm_arm_vgic_probe(void)
 {
     return 0;
Signature · 1 line
2.20.1
#7 lore raw

Re: [Qemu-arm] [RFC v2 3/3] virt: Check KVM_CAP_ARM_IRQ_LINE_LAYOUT_2 for smp_cpus > 256

In reply to Eric Auger
Quoted 22 lines · Eric Auger

Host kernel within [4.18, 5.3] report an erroneous KVM_MAX_VCPUS=512 for ARM. The actual capability to instantiate more than 256 vcpus was fixed in 5.4 with the upgrade of the KVM_IRQ_LINE ABI to support vcpu id encoded on 12 bits instead of 8 and a redistributor consuming a single KVM IO device instead of 2.

So let's check this capability when attempting to use more than 256 vcpus.

---
 hw/arm/virt.c        |  4 ++++
 target/arm/kvm.c     | 21 +++++++++++++++++++++
 target/arm/kvm_arm.h | 15 +++++++++++++++
 3 files changed, 40 insertions(+)
hw/arm/virt.c +0 -0
index 0d1629ccb3..465e3140f7 100644

hw/arm/virt.c static void machvirt_init(MachineState *machine) · line 1581

Quoted 7 lines
hw/arm/virt.c +4 -0
@@ -1575,6 +1575,10 @@ static void machvirt_init(MachineState *machine)
         virt_max_cpus = GIC_NCPU;
     }
 
+    if (kvm_arm_irq_line_layout_mismatch(MACHINE(vms), max_cpus)) {
+        exit(1);
+    }
+

Is there really no place to put this check in common code?

thanks
-- PMM

author
#8 lore raw

Re: [Qemu-arm] [Qemu-devel] [RFC v2 3/3] virt: Check KVM_CAP_ARM_IRQ_LINE_LAYOUT_2 for smp_cpus > 256

Hi Peter,

Quoted 26 lines · Peter Maydell
Quoted 25 lines · Eric Auger

Host kernel within [4.18, 5.3] report an erroneous KVM_MAX_VCPUS=512 for ARM. The actual capability to instantiate more than 256 vcpus was fixed in 5.4 with the upgrade of the KVM_IRQ_LINE ABI to support vcpu id encoded on 12 bits instead of 8 and a redistributor consuming a single KVM IO device instead of 2.

So let's check this capability when attempting to use more than 256 vcpus.

---
 hw/arm/virt.c        |  4 ++++
 target/arm/kvm.c     | 21 +++++++++++++++++++++
 target/arm/kvm_arm.h | 15 +++++++++++++++
 3 files changed, 40 insertions(+)
hw/arm/virt.c +0 -0
index 0d1629ccb3..465e3140f7 100644
@@ -1575,6 +1575,10 @@ static void machvirt_init(MachineState *machine)
         virt_max_cpus = GIC_NCPU;
     }
 
Quoted 6 lines

hw/arm/virt.c static void machvirt_init(MachineState *machine) · line 1581

Quoted 4 lines
hw/arm/virt.c +4 -0
+    if (kvm_arm_irq_line_layout_mismatch(MACHINE(vms), max_cpus)) {
+        exit(1);
+    }
+

Is there really no place to put this check in common code?

Not sure what you mean by common code here? Do you mean in a common code for ARM machines (I don't think we have any atm) or directly in kvm_init(). I did not want to pollute this latter with this ARM specific fix.

Thanks

Eric

Quoted 4 lines

thanks
-- PMM

#9 lore raw

Re: [Qemu-arm] [Qemu-devel] [RFC v2 3/3] virt: Check KVM_CAP_ARM_IRQ_LINE_LAYOUT_2 for smp_cpus > 256

Quoted 10 lines · Auger Eric

Hi Peter,

Quoted 1 line · Peter Maydell

Is there really no place to put this check in common code?

Not sure what you mean by common code here? Do you mean in a common code for ARM machines (I don't think we have any atm) or directly in kvm_init(). I did not want to pollute this latter with this ARM specific fix.

I'd just rather we didn't have to have the same "if ..." check in every arm board that supports KVM.

If kvm_init() happens at a point where we have enough info to make the check, then you can put the check in kvm_arch_init(), which is the architecture-specific hook that kvm_init() calls.

thanks
-- PMM

author
#10 lore raw

Re: [Qemu-devel] [RFC v2 3/3] virt: Check KVM_CAP_ARM_IRQ_LINE_LAYOUT_2 for smp_cpus > 256

Hi Peter,

Quoted 12 lines · Peter Maydell
Quoted 10 lines · Auger Eric

Hi Peter,

Quoted 1 line · Peter Maydell

Is there really no place to put this check in common code?

Not sure what you mean by common code here? Do you mean in a common code for ARM machines (I don't think we have any atm) or directly in kvm_init(). I did not want to pollute this latter with this ARM specific fix.

Quoted 6 lines

I'd just rather we didn't have to have the same "if ..." check in every arm board that supports KVM.

If kvm_init() happens at a point where we have enough info to make the check, then you can put the check in kvm_arch_init(), which is the architecture-specific hook that kvm_init() calls.

OK Thank you for the hint. It should be OK.

Thanks!

Eric

Quoted 4 lines

thanks
-- PMM