Skip to contentSkip to conversation

vGICv3 support

PATCH Superseded Pavel Fedin 3 participants 20 messages Aug 26, 2015 – Sep 8, 2015

v12 of 14 Superseded Latest is v14 →

lore
Share & tools
lore
#0 lore raw

v12 cover vGICv3 support

This series introduces support for GICv3 by KVM. Software emulation is currently not supported.

V11 => V12
- Do not set timer PPI CPU mask in device tree for GICv3
- Fixed small styling issues in hw/arm/virt.c
- Completely untied kvm_gic_access() from GIC and turned into kvm_device_access(). Attribute mapping is now handled in gicd/gicc wrapper macros.

v11 => v10
- Fixed minor issues with checkpatch and comments, reported by Eric Auger
- Make reusable kvm_gic_supports_attr(), moved to kvm-all.c and renamed as kvm_device_check_attr(). Useful for future live migration.

v9 => v10
- Renamed "gicversion" option to "gic-version" (was forgotten in v9)
- Data pointer in kvm_gic_access() is now void * because in case of vGICv3 this function is expected to operate on 64-bit registers too (GICD_IROUTER for instance)

v8 => v9
- Removed all limitations on CPU and IRQ number from the base class
- Added back missing properties, interface is now the same as in GICv2
- Refactored reusable parts of vGICv2 code, decreased number of changes
- Removed GIC type check from kvm_arch_irqchip_create(), no more need to specify GIC type early
- Fixed up all commit messages / logs
- Removed 'nvic' field assignment in virt machine (was forgotten in v8)
- CPU number limitation for 'virt' machine now comes from memory map (how many redistributors can be placed). With current layout it appears to be 126.

v7 => v8
- Removed all unused SW emulation code
- Removed unnecessary attributes from common class
- Set "unmigratable" flag for GICv3 device
- Removed unnecessary conditions from kvm_arm_gicv3_realize()
- Fixed GIC type setting in vexpress model, was done in wrong place
- Fixed condition style in hw/intc/Makefile.objs
- Cleaned up virt machine memory map

v6 => v7
- Wrap own GIC type definitions on top of KVM ones. Fixed build on non-ARM-Linux hosts

v5 => v6
- Fixed various checkpatch.pl style warnings
- Removed TODO in gicv3_init_irqs_and_mmio(), relevant memory API patch included
- gicv3_init_irqs_and_mmio() now takes 3 arguments instead of 4. It is more convenient to pass MMIO descriptors as array

v4 => v5
- Do not reintroduce several constants shared with GICv2, reuse them instead.
- Added gicv3_init_irqs_and_mmio() in base class, to be used by both software emulation and KVM code. Avoids code duplication.
- Do not add NULL msi-parent phandle to PCI device in the FDT
- Removed a couple of stale things from virt.c

v3 => v4
- Fixed stupid build breakage in patch 0002
- Rebased on top of current master, patch 0003 adjusted according to kvm_irqchip_create() changes
- Added assertion against uninitialized kernel_irqchip_type
- Removed kernel_irqchip_type initialization from models which do not use KVM vGIC

v2 => v3
- Removed some unrelated and unnecessary changes from virt machine, occasionally slipped in; some of them caused qemu to crash on ARM32.
- Fixed build for ARM32; vGICv3 code requires definitions which are present only in ARM64 kernel

v1 => v2
- Base class included, taken from the series by Shlomo Pongratz:
http://lists.nongnu.org/archive/html/qemu-devel/2015-06/msg01512.html
The code is refactored as little as possible in order to simplify further addition of software emulation:
- Minor fixes in code style and comments, according to old reviews
- Removed REV_V3 definition because it's currently not used, and it does not add any meaning to number 3.
- Removed reserved regions for MBI and ITS (except for 'virt' machine memory map). These should go to separate classes when implemented.
- Improved commit messages
- vGIC patches restructured
- Use 'gicversion' option instead of virt-v3 machine

Pavel Fedin (4):
intc/gic: Extract some reusable vGIC code
arm_kvm: Do not assume particular GIC type in
kvm_arch_irqchip_create()
hw/intc: Initial implementation of vGICv3 hw/arm/virt: Add gic-version option to virt machine

Shlomo Pongratz (1):
hw/intc: Implement GIC-500 base class

 hw/arm/virt.c                      | 118 +++++++++++++++++++++++------
 hw/intc/Makefile.objs              |   2 +
 hw/intc/arm_gic_kvm.c              |  84 ++++-----------------
 hw/intc/arm_gicv3_common.c         | 140 ++++++++++++++++++++++++++++++++++
 hw/intc/arm_gicv3_kvm.c            | 149 +++++++++++++++++++++++++++++++++++++
 hw/intc/vgic_common.h              |  49 ++++++++++++
 include/hw/arm/virt.h              |   5 +-
 include/hw/intc/arm_gicv3_common.h |  68 +++++++++++++++++
 include/sysemu/kvm.h               |  26 +++++++
 kvm-all.c                          |  33 ++++++++
 target-arm/kvm.c                   |  10 +--
 target-arm/kvm_arm.h               |  10 +++
 target-arm/machine.c               |  18 +++++
 13 files changed, 611 insertions(+), 101 deletions(-)
 create mode 100644 hw/intc/arm_gicv3_common.c
 create mode 100644 hw/intc/arm_gicv3_kvm.c
 create mode 100644 hw/intc/vgic_common.h
 create mode 100644 include/hw/intc/arm_gicv3_common.h
Signature · 1 line
1.9.5.msysgit.0
#7 lore raw

PATCH v12 3/5 arm_kvm: Do not assume particular GIC type in kvm_arch_irqchip_create()

This allows to use different GIC types from v2. There are no kernels which could advertise KVM_CAP_DEVICE_CTRL without the actual ability to create GIC with it.

---
 target-arm/kvm.c | 10 +---------
 1 file changed, 1 insertion(+), 9 deletions(-)
target-arm/kvm.c +1 -9
index b278542..22383c5 100644
@@ -585,18 +585,10 @@ void kvm_arch_init_irq_routing(KVMState *s)
 
 int kvm_arch_irqchip_create(KVMState *s)
 {
-    int ret;
-
     /* If we can create the VGIC using the newer device control API, we
      * let the device do this when it initializes itself, otherwise we
      * fall back to the old API */
-
-    ret = kvm_create_device(s, KVM_DEV_TYPE_ARM_VGIC_V2, true);
-    if (ret == 0) {
-        return 1;
-    }
-
-    return 0;
+    return kvm_check_extension(s, KVM_CAP_DEVICE_CTRL);
 }
 
 int kvm_arch_fixup_msi_route(struct kvm_irq_routing_entry *route,
Signature · 1 line
1.9.5.msysgit.0
#1 lore raw

PATCH v12 1/5 hw/intc: Implement GIC-500 base class

In reply to Pavel Fedin

From: Shlomo Pongratz <[email protected]>

This class is to be used by both software and KVM implementations of GICv3

Currently it is mostly a placeholder, but in future it is supposed to hold qemu's representation of GICv3 state, which is necessary for migration.

The interface of this class is fully compatible with GICv2 one. This is done in order to simplify integration with existing code.

---
 hw/intc/Makefile.objs              |   1 +
 hw/intc/arm_gicv3_common.c         | 140 +++++++++++++++++++++++++++++++++++++
 include/hw/intc/arm_gicv3_common.h |  68 ++++++++++++++++++
 3 files changed, 209 insertions(+)
 create mode 100644 hw/intc/arm_gicv3_common.c
 create mode 100644 include/hw/intc/arm_gicv3_common.h
hw/intc/Makefile.objs +1 -0
index 092d8a8..1317e5a 100644
@@ -12,6 +12,7 @@ common-obj-$(CONFIG_IOAPIC) += ioapic_common.o
 common-obj-$(CONFIG_ARM_GIC) += arm_gic_common.o
 common-obj-$(CONFIG_ARM_GIC) += arm_gic.o
 common-obj-$(CONFIG_ARM_GIC) += arm_gicv2m.o
+common-obj-$(CONFIG_ARM_GIC) += arm_gicv3_common.o
 common-obj-$(CONFIG_OPENPIC) += openpic.o
 
 obj-$(CONFIG_APIC) += apic.o apic_common.o
hw/intc/arm_gicv3_common.c new file +140 -0

140 lines not on this page · raw message

include/hw/intc/arm_gicv3_common.h new file +68 -0

68 lines not on this page · raw message

Signature · 1 line
1.9.5.msysgit.0
#3 lore raw

PATCH v12 2/5 intc/gic: Extract some reusable vGIC code

In reply to Pavel Fedin

Some functions previously used only by vGICv2 are useful also for vGICv3 implementation. Untie them from GICState and make accessible from within other modules:
- kvm_arm_gic_set_irq()
- kvm_gic_supports_attr() - moved to common code and renamed to kvm_device_check_attr()
- kvm_gic_access() - turned into GIC-independent kvm_device_access(). Data pointer changed to void * because some GICv3 registers are 64-bit wide
- kvm_gicd_access()
- kvm_gicc_access() - actually GICv2-specific, but changed to keep the code style unified with kvm_gicd_access()

Some of these changes are not used right now, but they will be helpful for implementing live migration.

Actually kvm_dist_get() and kvm_dist_put() could also be made reusable, but they would require two extra parameters (s->dev_fd and s->num_cpu) as well as lots of typecasts of 's' to DeviceState * and back to GICState *. This makes the code very ugly so i decided to stop at this point. I tried also an approach with making a base class for all possible GICs, but it would contain only three variables (dev_fd, cpu_num and irq_num), and accessing them through the rest of the code would be again tedious (either ugly casts or qemu-style separate object pointer). So i disliked it too.

---
 hw/intc/arm_gic_kvm.c | 84 ++++++++++-----------------------------------------
 hw/intc/vgic_common.h | 49 ++++++++++++++++++++++++++++++
 include/sysemu/kvm.h  | 26 ++++++++++++++++
 kvm-all.c             | 33 ++++++++++++++++++++
 4 files changed, 124 insertions(+), 68 deletions(-)
 create mode 100644 hw/intc/vgic_common.h
hw/intc/arm_gic_kvm.c +16 -68
index e5d0f67..e129c90 100644
@@ -23,6 +23,7 @@
 #include "sysemu/kvm.h"
 #include "kvm_arm.h"
 #include "gic_internal.h"
+#include "vgic_common.h"
 
 //#define DEBUG_GIC_KVM
 
@@ -52,7 +53,7 @@ typedef struct KVMARMGICClass {
     void (*parent_reset)(DeviceState *dev);
 } KVMARMGICClass;
 
-static void kvm_arm_gic_set_irq(void *opaque, int irq, int level)
+void kvm_arm_gic_set_irq(uint32_t num_irq, int irq, int level)
 {
     /* Meaning of the 'irq' parameter:
      *  [0..N-1] : external interrupts
@@ -63,10 +64,9 @@ static void kvm_arm_gic_set_irq(void *opaque, int irq, int level)
      * has separate fields in the irq number for type,
      * CPU number and interrupt number.
      */
-    GICState *s = (GICState *)opaque;
     int kvm_irq, irqtype, cpu;
 
-    if (irq < (s->num_irq - GIC_INTERNAL)) {
+    if (irq < (num_irq - GIC_INTERNAL)) {
         /* External interrupt. The kernel numbers these like the GIC
          * hardware, with external interrupt IDs starting after the
          * internal ones.
@@ -77,7 +77,7 @@ static void kvm_arm_gic_set_irq(void *opaque, int irq, int level)
     } else {
         /* Internal interrupt: decode into (cpu, interrupt id) */
         irqtype = KVM_ARM_IRQ_TYPE_PPI;
-        irq -= (s->num_irq - GIC_INTERNAL);
+        irq -= (num_irq - GIC_INTERNAL);
         cpu = irq / GIC_INTERNAL;
         irq %= GIC_INTERNAL;
     }
@@ -87,69 +87,16 @@ static void kvm_arm_gic_set_irq(void *opaque, int irq, int level)
     kvm_set_irq(kvm_state, kvm_irq, !!level);
 }
 
-static bool kvm_arm_gic_can_save_restore(GICState *s)
-{
-    return s->dev_fd >= 0;
-}
-
-static bool kvm_gic_supports_attr(GICState *s, int group, int attrnum)
-{
-    struct kvm_device_attr attr = {
-        .group = group,
-        .attr = attrnum,
-        .flags = 0,
-    };
-
-    if (s->dev_fd == -1) {
-        return false;
-    }
-
-    return kvm_device_ioctl(s->dev_fd, KVM_HAS_DEVICE_ATTR, &attr) == 0;
-}
-
-static void kvm_gic_access(GICState *s, int group, int offset,
-                                   int cpu, uint32_t *val, bool write)
+static void kvm_arm_gicv2_set_irq(void *opaque, int irq, int level)
 {
-    struct kvm_device_attr attr;
-    int type;
-    int err;
-
-    cpu = cpu & 0xff;
-
-    attr.flags = 0;
-    attr.group = group;
-    attr.attr = (((uint64_t)cpu << KVM_DEV_ARM_VGIC_CPUID_SHIFT) &
-                 KVM_DEV_ARM_VGIC_CPUID_MASK) |
-                (((uint64_t)offset << KVM_DEV_ARM_VGIC_OFFSET_SHIFT) &
-                 KVM_DEV_ARM_VGIC_OFFSET_MASK);
-    attr.addr = (uintptr_t)val;
-
-    if (write) {
-        type = KVM_SET_DEVICE_ATTR;
-    } else {
-        type = KVM_GET_DEVICE_ATTR;
-    }
-
-    err = kvm_device_ioctl(s->dev_fd, type, &attr);
-    if (err < 0) {
-        fprintf(stderr, "KVM_{SET/GET}_DEVICE_ATTR failed: %s\n",
-                strerror(-err));
-        abort();
-    }
-}
+    GICState *s = (GICState *)opaque;
 
-static void kvm_gicd_access(GICState *s, int offset, int cpu,
-                            uint32_t *val, bool write)
-{
-    kvm_gic_access(s, KVM_DEV_ARM_VGIC_GRP_DIST_REGS,
-                   offset, cpu, val, write);
+    kvm_arm_gic_set_irq(s->num_irq, irq, level);
 }
 
-static void kvm_gicc_access(GICState *s, int offset, int cpu,
-                            uint32_t *val, bool write)
+static bool kvm_arm_gic_can_save_restore(GICState *s)
 {
-    kvm_gic_access(s, KVM_DEV_ARM_VGIC_GRP_CPU_REGS,
-                   offset, cpu, val, write);
+    return s->dev_fd >= 0;
 }
 
 #define for_each_irq_reg(_ctr, _max_irq, _field_width) \
@@ -559,7 +506,7 @@ static void kvm_arm_gic_realize(DeviceState *dev, Error **errp)
         return;
     }
 
-    gic_init_irqs_and_mmio(s, kvm_arm_gic_set_irq, NULL);
+    gic_init_irqs_and_mmio(s, kvm_arm_gicv2_set_irq, NULL);
 
     for (i = 0; i < s->num_irq - GIC_INTERNAL; i++) {
         qemu_irq irq = qdev_get_gpio_in(dev, i);
@@ -576,16 +523,17 @@ static void kvm_arm_gic_realize(DeviceState *dev, Error **errp)
         return;
     }
 
-    if (kvm_gic_supports_attr(s, KVM_DEV_ARM_VGIC_GRP_NR_IRQS, 0)) {
+    if (kvm_device_check_attr(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_NR_IRQS, 0)) {
         uint32_t numirqs = s->num_irq;
-        kvm_gic_access(s, KVM_DEV_ARM_VGIC_GRP_NR_IRQS, 0, 0, &numirqs, 1);
+        kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_NR_IRQS, 0,
+                          &numirqs, true);
     }
 
     /* Tell the kernel to complete VGIC initialization now */
-    if (kvm_gic_supports_attr(s, KVM_DEV_ARM_VGIC_GRP_CTRL,
+    if (kvm_device_check_attr(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_CTRL,
                               KVM_DEV_ARM_VGIC_CTRL_INIT)) {
-        kvm_gic_access(s, KVM_DEV_ARM_VGIC_GRP_CTRL,
-                          KVM_DEV_ARM_VGIC_CTRL_INIT, 0, 0, 1);
+        kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_CTRL,
+                          KVM_DEV_ARM_VGIC_CTRL_INIT, NULL, true);
     }
 
     /* Distributor */
hw/intc/vgic_common.h new file +49 -0

49 lines not on this page · raw message

include/sysemu/kvm.h +26 -0
index 983e99e..2a58b4d 100644
@@ -240,6 +240,32 @@ int kvm_device_ioctl(int fd, int type, ...);
 int kvm_vm_check_attr(KVMState *s, uint32_t group, uint64_t attr);
 
 /**
+ * kvm_device_check_attr - check for existence of a specific device attribute
+ * @fd: The device file descriptor
+ * @group: the group
+ * @attr: the attribute of that group to query for
+ *
+ * Returns: 1 if the attribute exists
+ *          0 if the attribute either does not exist or if the vm device
+ *            interface is unavailable
+ */
+int kvm_device_check_attr(int fd, uint32_t group, uint64_t attr);
+
+/**
+ * kvm_device_access - set or get value of a specific vm attribute
+ * @fd: The device file descriptor
+ * @group: the group
+ * @attr: the attribute of that group to set or get
+ * @val: pointer to a storage area for the value
+ * @write: true for set and false for get operation
+ *
+ * This function is not allowed to fail. Use kvm_device_check_attr()
+ * in order to check for the availability of optional attributes.
+ */
+void kvm_device_access(int fd, int group, uint64_t attr,
+                       void *val, bool write);
+
+/**
  * kvm_create_device - create a KVM device for the device control API
  * @KVMState: The KVMState pointer
  * @type: The KVM device type (see Documentation/virtual/kvm/devices in the
kvm-all.c +33 -0
index 06e06f2..a902fb8 100644
@@ -24,6 +24,7 @@
 #include "qemu/atomic.h"
 #include "qemu/option.h"
 #include "qemu/config-file.h"
+#include "qemu/error-report.h"
 #include "hw/hw.h"
 #include "hw/pci/msi.h"
 #include "hw/s390x/adapter.h"
@@ -2003,6 +2004,38 @@ int kvm_vm_check_attr(KVMState *s, uint32_t group, uint64_t attr)
     return ret ? 0 : 1;
 }
 
+int kvm_device_check_attr(int dev_fd, uint32_t group, uint64_t attr)
+{
+    struct kvm_device_attr attribute = {
+        .group = group,
+        .attr = attr,
+        .flags = 0,
+    };
+
+    return kvm_device_ioctl(dev_fd, KVM_HAS_DEVICE_ATTR, &attribute) ? 0 : 1;
+}
+
+void kvm_device_access(int fd, int group, uint64_t attr,
+                       void *val, bool write)
+{
+    struct kvm_device_attr kvmattr;
+    int err;
+
+    kvmattr.flags = 0;
+    kvmattr.group = group;
+    kvmattr.attr = attr;
+    kvmattr.addr = (uintptr_t)val;
+
+    err = kvm_device_ioctl(fd,
+                           write ? KVM_SET_DEVICE_ATTR : KVM_GET_DEVICE_ATTR,
+                           &kvmattr);
+    if (err < 0) {
+        error_report("KVM_%s_DEVICE_ATTR failed: %s",
+                     write ? "SET" : "GET", strerror(-err));
+        abort();
+    }
+}
+
 int kvm_has_sync_mmu(void)
 {
     return kvm_check_extension(kvm_state, KVM_CAP_SYNC_MMU);
Signature · 1 line
1.9.5.msysgit.0
#9 lore raw

PATCH v12 4/5 hw/intc: Initial implementation of vGICv3

In reply to Pavel Fedin

This is the initial version of KVM-accelerated GICv3 support. State load and save are not yet supported, live migration is not possible.

In order to get correct class name in a simpler way, gicv3_class_name() function is implemented, similar to gic_class_name().

---
 hw/intc/Makefile.objs   |   1 +
 hw/intc/arm_gicv3_kvm.c | 149 ++++++++++++++++++++++++++++++++++++++++++++++++
 target-arm/kvm_arm.h    |  10 ++++
 target-arm/machine.c    |  18 ++++++
 4 files changed, 178 insertions(+)
 create mode 100644 hw/intc/arm_gicv3_kvm.c
hw/intc/Makefile.objs +1 -0
index 1317e5a..004b0c2 100644
@@ -17,6 +17,7 @@ common-obj-$(CONFIG_OPENPIC) += openpic.o
 
 obj-$(CONFIG_APIC) += apic.o apic_common.o
 obj-$(CONFIG_ARM_GIC_KVM) += arm_gic_kvm.o
+obj-$(call land,$(CONFIG_ARM_GIC_KVM),$(TARGET_AARCH64)) += arm_gicv3_kvm.o
 obj-$(CONFIG_STELLARIS) += armv7m_nvic.o
 obj-$(CONFIG_EXYNOS4) += exynos4210_gic.o exynos4210_combiner.o
 obj-$(CONFIG_GRLIB) += grlib_irqmp.o
hw/intc/arm_gicv3_kvm.c new file +149 -0

149 lines not on this page · raw message

target-arm/kvm_arm.h +10 -0
index b3e0ab7..b6f5b69 100644
@@ -196,4 +196,14 @@ static inline const char *gic_class_name(void)
     return kvm_irqchip_in_kernel() ? "kvm-arm-gic" : "arm_gic";
 }
 
+/**
+ * gicv3_class_name
+ *
+ * Return name of GICv3 class to use depending on whether KVM acceleration is
+ * in use. May throw an error if the chosen implementation is not available.
+ *
+ * Returns: class name to use
+ */
+const char *gicv3_class_name(void);
+
 #endif
target-arm/machine.c +18 -0
index 32adfe7..36a0d15 100644
@@ -1,5 +1,6 @@
 #include "hw/hw.h"
 #include "hw/boards.h"
+#include "qemu/error-report.h"
 #include "sysemu/kvm.h"
 #include "kvm_arm.h"
 #include "internals.h"
@@ -328,3 +329,20 @@ const VMStateDescription vmstate_arm_cpu = {
         NULL
     }
 };
+
+const char *gicv3_class_name(void)
+{
+    if (kvm_irqchip_in_kernel()) {
+#ifdef TARGET_AARCH64
+        return "kvm-arm-gicv3";
+#else
+        error_report("KVM GICv3 acceleration is not supported on this "
+                     "platform\n");
+#endif
+    } else {
+        /* TODO: Software emulation is not implemented yet */
+        error_report("KVM is currently required for GICv3 emulation\n");
+    }
+
+    exit(1);
+}
Signature · 1 line
1.9.5.msysgit.0
#10 lore raw

PATCH v12 5/5 hw/arm/virt: Add gic-version option to virt machine

In reply to Pavel Fedin

Add gic_version to VirtMachineState, set it to value of the option and pass it around where necessary. Instantiate devices and fdt nodes according to the choice.

max_cpus for virt machine increased to 126 (calculated from redistributor space available in the memory map). GICv2 compatibility check happens inside arm_gic_common_realize().

ITS regions are added to the memory map too, however currently they are not used, just reserved.

---
 hw/arm/virt.c         | 118 ++++++++++++++++++++++++++++++++++++++++----------
 include/hw/arm/virt.h |   5 ++-
 2 files changed, 99 insertions(+), 24 deletions(-)
hw/arm/virt.c +95 -23
index d5a8417..7044cb9 100644
@@ -50,6 +50,7 @@
 #include "hw/arm/fdt.h"
 #include "hw/intc/arm_gic_common.h"
 #include "kvm_arm.h"
+#include "qapi/visitor.h"
 
 /* Number of external interrupt lines to configure the GIC with */
 #define NUM_IRQS 256
@@ -79,6 +80,7 @@ typedef struct {
 typedef struct {
     MachineState parent;
     bool secure;
+    int32_t gic_version;
 } VirtMachineState;
 
 #define TYPE_VIRT_MACHINE   "virt"
@@ -109,6 +111,10 @@ static const MemMapEntry a15memmap[] = {
     [VIRT_GIC_DIST] =           { 0x08000000, 0x00010000 },
     [VIRT_GIC_CPU] =            { 0x08010000, 0x00010000 },
     [VIRT_GIC_V2M] =            { 0x08020000, 0x00001000 },
+    [VIRT_ITS_CONTROL] =        { 0x08020000, 0x00010000 },
+    [VIRT_ITS_TRANSLATION] =    { 0x08030000, 0x00010000 },
+    /* This redistributor space allows up to 2*64kB*126 CPUs */
+    [VIRT_GIC_REDIST] =         { 0x08040000, 0x00FC0000 },
     [VIRT_UART] =               { 0x09000000, 0x00001000 },
     [VIRT_RTC] =                { 0x09010000, 0x00001000 },
     [VIRT_FW_CFG] =             { 0x09020000, 0x0000000a },
@@ -251,7 +257,7 @@ static void fdt_add_psci_node(const VirtBoardInfo *vbi)
     qemu_fdt_setprop_cell(fdt, "/psci", "migrate", migrate_fn);
 }
 
-static void fdt_add_timer_nodes(const VirtBoardInfo *vbi)
+static void fdt_add_timer_nodes(const VirtBoardInfo *vbi, int gictype)
 {
     /* Note that on A15 h/w these interrupts are level-triggered,
      * but for the GIC implementation provided by both QEMU and KVM
@@ -260,8 +266,11 @@ static void fdt_add_timer_nodes(const VirtBoardInfo *vbi)
     ARMCPU *armcpu;
     uint32_t irqflags = GIC_FDT_IRQ_FLAGS_EDGE_LO_HI;
 
-    irqflags = deposit32(irqflags, GIC_FDT_IRQ_PPI_CPU_START,
-                         GIC_FDT_IRQ_PPI_CPU_WIDTH, (1 << vbi->smp_cpus) - 1);
+    if (gictype == 2) {
+        irqflags = deposit32(irqflags, GIC_FDT_IRQ_PPI_CPU_START,
+                             GIC_FDT_IRQ_PPI_CPU_WIDTH,
+                             (1 << vbi->smp_cpus) - 1);
+    }
 
     qemu_fdt_add_subnode(vbi->fdt, "/timer");
 
@@ -285,6 +294,18 @@ static void fdt_add_cpu_nodes(const VirtBoardInfo *vbi)
 {
     int cpu;
 
+    /*
+     * From Documentation/devicetree/bindings/arm/cpus.txt
+     *  On ARM v8 64-bit systems value should be set to 2,
+     *  that corresponds to the MPIDR_EL1 register size.
+     *  If MPIDR_EL1[63:32] value is equal to 0 on all CPUs
+     *  in the system, #address-cells can be set to 1, since
+     *  MPIDR_EL1[63:32] bits are not used for CPUs
+     *  identification.
+     *
+     *  Now GIC500 doesn't support affinities 2 & 3 so currently
+     *  #address-cells can stay 1 until future GIC
+     */
     qemu_fdt_add_subnode(vbi->fdt, "/cpus");
     qemu_fdt_setprop_cell(vbi->fdt, "/cpus", "#address-cells", 0x1);
     qemu_fdt_setprop_cell(vbi->fdt, "/cpus", "#size-cells", 0x0);
@@ -321,25 +342,36 @@ static void fdt_add_v2m_gic_node(VirtBoardInfo *vbi)
     qemu_fdt_setprop_cell(vbi->fdt, "/intc/v2m", "phandle", vbi->v2m_phandle);
 }
 
-static void fdt_add_gic_node(VirtBoardInfo *vbi)
+static void fdt_add_gic_node(VirtBoardInfo *vbi, int type)
 {
     vbi->gic_phandle = qemu_fdt_alloc_phandle(vbi->fdt);
     qemu_fdt_setprop_cell(vbi->fdt, "/", "interrupt-parent", vbi->gic_phandle);
 
     qemu_fdt_add_subnode(vbi->fdt, "/intc");
-    /* 'cortex-a15-gic' means 'GIC v2' */
-    qemu_fdt_setprop_string(vbi->fdt, "/intc", "compatible",
-                            "arm,cortex-a15-gic");
     qemu_fdt_setprop_cell(vbi->fdt, "/intc", "#interrupt-cells", 3);
     qemu_fdt_setprop(vbi->fdt, "/intc", "interrupt-controller", NULL, 0);
-    qemu_fdt_setprop_sized_cells(vbi->fdt, "/intc", "reg",
-                                     2, vbi->memmap[VIRT_GIC_DIST].base,
-                                     2, vbi->memmap[VIRT_GIC_DIST].size,
-                                     2, vbi->memmap[VIRT_GIC_CPU].base,
-                                     2, vbi->memmap[VIRT_GIC_CPU].size);
     qemu_fdt_setprop_cell(vbi->fdt, "/intc", "#address-cells", 0x2);
     qemu_fdt_setprop_cell(vbi->fdt, "/intc", "#size-cells", 0x2);
     qemu_fdt_setprop(vbi->fdt, "/intc", "ranges", NULL, 0);
+    if (type == 3) {
+        qemu_fdt_setprop_string(vbi->fdt, "/intc", "compatible",
+                                "arm,gic-v3");
+        qemu_fdt_setprop_sized_cells(vbi->fdt, "/intc", "reg",
+                                     2, vbi->memmap[VIRT_GIC_DIST].base,
+                                     2, vbi->memmap[VIRT_GIC_DIST].size,
+                                     2, vbi->memmap[VIRT_GIC_REDIST].base,
+                                     2, vbi->memmap[VIRT_GIC_REDIST].size);
+    } else {
+        /* 'cortex-a15-gic' means 'GIC v2' */
+        qemu_fdt_setprop_string(vbi->fdt, "/intc", "compatible",
+                                "arm,cortex-a15-gic");
+        qemu_fdt_setprop_sized_cells(vbi->fdt, "/intc", "reg",
+                                      2, vbi->memmap[VIRT_GIC_DIST].base,
+                                      2, vbi->memmap[VIRT_GIC_DIST].size,
+                                      2, vbi->memmap[VIRT_GIC_CPU].base,
+                                      2, vbi->memmap[VIRT_GIC_CPU].size);
+    }
+
     qemu_fdt_setprop_cell(vbi->fdt, "/intc", "phandle", vbi->gic_phandle);
 }
 
@@ -362,18 +394,18 @@ static void create_v2m(VirtBoardInfo *vbi, qemu_irq *pic)
     fdt_add_v2m_gic_node(vbi);
 }
 
-static void create_gic(VirtBoardInfo *vbi, qemu_irq *pic)
+static void create_gic(VirtBoardInfo *vbi, qemu_irq *pic, int type)
 {
-    /* We create a standalone GIC v2 */
+    /* We create a standalone GIC */
     DeviceState *gicdev;
     SysBusDevice *gicbusdev;
     const char *gictype;
     int i;
 
-    gictype = gic_class_name();
+    gictype = (type == 3) ? gicv3_class_name() : gic_class_name();
 
     gicdev = qdev_create(NULL, gictype);
-    qdev_prop_set_uint32(gicdev, "revision", 2);
+    qdev_prop_set_uint32(gicdev, "revision", type);
     qdev_prop_set_uint32(gicdev, "num-cpu", smp_cpus);
     /* Note that the num-irq property counts both internal and external
      * interrupts; there are always 32 of the former (mandated by GIC spec).
@@ -382,7 +414,11 @@ static void create_gic(VirtBoardInfo *vbi, qemu_irq *pic)
     qdev_init_nofail(gicdev);
     gicbusdev = SYS_BUS_DEVICE(gicdev);
     sysbus_mmio_map(gicbusdev, 0, vbi->memmap[VIRT_GIC_DIST].base);
-    sysbus_mmio_map(gicbusdev, 1, vbi->memmap[VIRT_GIC_CPU].base);
+    if (type == 3) {
+        sysbus_mmio_map(gicbusdev, 1, vbi->memmap[VIRT_GIC_REDIST].base);
+    } else {
+        sysbus_mmio_map(gicbusdev, 1, vbi->memmap[VIRT_GIC_CPU].base);
+    }
 
     /* Wire the outputs from each CPU's generic timer to the
      * appropriate GIC PPI inputs, and the GIC's IRQ output to
@@ -417,9 +453,11 @@ static void create_gic(VirtBoardInfo *vbi, qemu_irq *pic)
         pic[i] = qdev_get_gpio_in(gicdev, i);
     }
 
-    fdt_add_gic_node(vbi);
+    fdt_add_gic_node(vbi, type);
 
-    create_v2m(vbi, pic);
+    if (type == 2) {
+        create_v2m(vbi, pic);
+    }
 }
 
 static void create_uart(const VirtBoardInfo *vbi, qemu_irq *pic)
@@ -723,7 +761,10 @@ static void create_pcie(const VirtBoardInfo *vbi, qemu_irq *pic)
     qemu_fdt_setprop_cells(vbi->fdt, nodename, "bus-range", 0,
                            nr_pcie_buses - 1);
 
-    qemu_fdt_setprop_cells(vbi->fdt, nodename, "msi-parent", vbi->v2m_phandle);
+    if (vbi->v2m_phandle) {
+        qemu_fdt_setprop_cells(vbi->fdt, nodename, "msi-parent",
+                               vbi->v2m_phandle);
+    }
 
     qemu_fdt_setprop_sized_cells(vbi->fdt, nodename, "reg",
                                  2, base_ecam, 2, size_ecam);
@@ -873,7 +914,7 @@ static void machvirt_init(MachineState *machine)
         object_property_set_bool(cpuobj, true, "realized", NULL);
     }
     g_strfreev(cpustr);
-    fdt_add_timer_nodes(vbi);
+    fdt_add_timer_nodes(vbi, vms->gic_version);
     fdt_add_cpu_nodes(vbi);
     fdt_add_psci_node(vbi);
 
@@ -883,7 +924,7 @@ static void machvirt_init(MachineState *machine)
 
     create_flash(vbi);
 
-    create_gic(vbi, pic);
+    create_gic(vbi, pic, vms->gic_version);
 
     create_uart(vbi, pic);
 
@@ -941,6 +982,26 @@ static void virt_set_secure(Object *obj, bool value, Error **errp)
     vms->secure = value;
 }
 
+static void virt_get_gic_version(Object *obj, Visitor *v, void *opaque,
+                                 const char *name, Error **errp)
+{
+    VirtMachineState *vms = VIRT_MACHINE(obj);
+
+    visit_type_int32(v, &vms->gic_version, name, errp);
+}
+
+static void virt_set_gic_version(Object *obj, Visitor *v, void *opaque,
+                                 const char *name, Error **errp)
+{
+    VirtMachineState *vms = VIRT_MACHINE(obj);
+
+    visit_type_int32(v, &vms->gic_version, name, errp);
+    if (vms->gic_version != 2 && vms->gic_version != 3) {
+        error_report("Only GICv2 and GICv3 supported currently\n");
+        exit(1);
+    }
+}
+
 static void virt_instance_init(Object *obj)
 {
     VirtMachineState *vms = VIRT_MACHINE(obj);
@@ -953,6 +1014,14 @@ static void virt_instance_init(Object *obj)
                                     "Set on/off to enable/disable the ARM "
                                     "Security Extensions (TrustZone)",
                                     NULL);
+
+    /* Default GIC type is v2 */
+    vms->gic_version = 2;
+    object_property_add(obj, "gic-version", "int", virt_get_gic_version,
+                        virt_set_gic_version, NULL, NULL, NULL);
+    object_property_set_description(obj, "gic-version",
+                                    "Set GIC version. "
+                                    "Valid values are 2 and 3", NULL);
 }
 
 static void virt_class_init(ObjectClass *oc, void *data)
@@ -962,7 +1031,10 @@ static void virt_class_init(ObjectClass *oc, void *data)
     mc->name = TYPE_VIRT_MACHINE;
     mc->desc = "ARM Virtual Machine",
     mc->init = machvirt_init;
-    mc->max_cpus = 8;
+    /* Our maximum number of CPUs depends on how many redistributors
+     * we can fit into memory map
+     */
+    mc->max_cpus = a15memmap[VIRT_GIC_REDIST].size / 0x20000;
     mc->has_dynamic_sysbus = true;
     mc->block_default_type = IF_VIRTIO;
     mc->no_cdrom = 1;
include/hw/arm/virt.h +4 -1
index d22fd8e..6c5a9c9 100644
@@ -46,6 +46,10 @@ enum {
     VIRT_CPUPERIPHS,
     VIRT_GIC_DIST,
     VIRT_GIC_CPU,
+    VIRT_GIC_V2M,
+    VIRT_ITS_CONTROL,
+    VIRT_ITS_TRANSLATION,
+    VIRT_GIC_REDIST,
     VIRT_UART,
     VIRT_MMIO,
     VIRT_RTC,
@@ -54,7 +58,6 @@ enum {
     VIRT_PCIE_MMIO,
     VIRT_PCIE_PIO,
     VIRT_PCIE_ECAM,
-    VIRT_GIC_V2M,
     VIRT_PLATFORM_BUS,
 };
 
Signature · 1 line
1.9.5.msysgit.0
Reviewed-by on v12 1/5
#2 lore raw

Re: [Qemu-devel] [PATCH v12 1/5] hw/intc: Implement GIC-500 base class

In reply to Pavel Fedin
Quoted 17 lines · Pavel Fedin

From: Shlomo Pongratz <[email protected]>

This class is to be used by both software and KVM implementations of GICv3

Currently it is mostly a placeholder, but in future it is supposed to hold qemu's representation of GICv3 state, which is necessary for migration.

The interface of this class is fully compatible with GICv2 one. This is done in order to simplify integration with existing code.

+++ b/hw/intc/arm_gicv3_common.c

hw/intc/arm_gicv3_common.c line 7

Quoted 7 lines
hw/intc/arm_gicv3_common.c new file +7 -0
@@ -0,0 +1,140 @@
+/*
+ * ARM GICv3 support - common bits of emulated and KVM kernel model
+ *
+ * Copyright (c) 2012 Linaro Limited
+ * Copyright (c) 2015 Huawei.
+ * Written by Peter Maydell
+ * Extended to 64 cores by Shlomo Pongratz

You might want to reword that line, the 64 cores bit isn't quite right any more.

include/hw/intc/arm_gicv3_common.h line 48

Quoted 1 line
include/hw/intc/arm_gicv3_common.h new file +1 -0
+#define TYPE_ARM_GICV3_COMMON "arm_gicv3_common"

We prefer '-' in type names, not '_' (see
http://wiki.qemu.org/QOMConventions). Obviously we have a bunch of older devices in the tree that don't follow that guideline...

Otherwise

thanks
-- PMM

#4 lore raw

Re: [Qemu-devel] [PATCH v12 2/5] intc/gic: Extract some reusable vGIC code

In reply to Pavel Fedin
Quoted 25 lines · Pavel Fedin

Some functions previously used only by vGICv2 are useful also for vGICv3 implementation. Untie them from GICState and make accessible from within other modules:
- kvm_arm_gic_set_irq()
- kvm_gic_supports_attr() - moved to common code and renamed to kvm_device_check_attr()
- kvm_gic_access() - turned into GIC-independent kvm_device_access(). Data pointer changed to void * because some GICv3 registers are 64-bit wide
- kvm_gicd_access()
- kvm_gicc_access() - actually GICv2-specific, but changed to keep the code style unified with kvm_gicd_access()

Some of these changes are not used right now, but they will be helpful for implementing live migration.

Actually kvm_dist_get() and kvm_dist_put() could also be made reusable, but they would require two extra parameters (s->dev_fd and s->num_cpu) as well as lots of typecasts of 's' to DeviceState * and back to GICState *. This makes the code very ugly so i decided to stop at this point. I tried also an approach with making a base class for all possible GICs, but it would contain only three variables (dev_fd, cpu_num and irq_num), and accessing them through the rest of the code would be again tedious (either ugly casts or qemu-style separate object pointer). So i disliked it too.

Quoted 2 lines

Minor issues only:

Quoted 9 lines
hw/intc/arm_gic_kvm.c +0 -9
-static bool kvm_arm_gic_can_save_restore(GICState *s)
-{
-    return s->dev_fd >= 0;
-}
-
-static bool kvm_gic_supports_attr(GICState *s, int group, int attrnum)
-{
-    struct kvm_device_attr attr = {
-        .group = group,

hw/intc/arm_gic_kvm.c static void kvm_arm_gic_set_irq(void *opaque, int irq, int level) · line 108

Quoted 10 lines
hw/intc/arm_gic_kvm.c +0 -10
-        .attr = attrnum,
-        .flags = 0,
-    };
-
-    if (s->dev_fd == -1) {
-        return false;
-    }
-
-    return kvm_device_ioctl(s->dev_fd, KVM_HAS_DEVICE_ATTR, &attr) == 0;
-}

This function copes with being handed a -1 filedescriptor (as will happen for old kernels with vGICv2 support but no irqchip device API support), but your new kvm_device_check_attr() doesn't. You either need to support it in that new function, or make the calling code avoid calling it for the -1 case.

Quoted 7 lines
hw/intc/vgic_common.h new file +7 -0
+
+/**
+ * kvm_arm_gic_set_irq - Send an IRQ to the in-kernel vGIC
+ * @num_irq: Total number of IRQs configured for the GIC instance
+ * @irq: qemu internal IRQ line number:
+ *  [0..N-1] : external interrupts
+ *  [N..N+31] : PPI (internal) interrupts for CPU 0

hw/intc/vgic_common.h line 39

Quoted 10 lines
hw/intc/vgic_common.h new file +10 -0
+ *  [N+32..N+63] : PPI (internal interrupts for CPU 1
+ * @level: level of the IRQ line.
+ */
+void kvm_arm_gic_set_irq(uint32_t num_irq, int irq, int level);
+
+#define KVM_VGIC_ATTR(offset, cpu) \
+    ((((uint64_t)(cpu) << KVM_DEV_ARM_VGIC_CPUID_SHIFT) & \
+      KVM_DEV_ARM_VGIC_CPUID_MASK) | \
+     (((uint64_t)(offset) << KVM_DEV_ARM_VGIC_OFFSET_SHIFT) & \
+      KVM_DEV_ARM_VGIC_OFFSET_MASK))

This is assuming that GICv3 will use the same masks/shifts for register access that GICv2 does at the moment, right? I'm not sure that's necessarily the case, especially since the v2 mask limits us to 256 CPUs maximum.

I guess we can fix that up when we need to, though.

hw/intc/vgic_common.h line 47

Quoted 8 lines
hw/intc/vgic_common.h new file +8 -0
+
+#define kvm_gicd_access(s, offset, cpu, val, write) \
+    kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_DIST_REGS, \
+                      KVM_VGIC_ATTR(offset, cpu), val, write)
+
+#define kvm_gicc_access(s, offset, cpu, val, write) \
+    kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_CPU_REGS, \
+                      KVM_VGIC_ATTR(offset, cpu), val, write)

Can you make these two static inline functions, not #defines, please? Doc comments would be nice too.

thanks
-- PMM

Reviewed-by on v12 3/5
#8 lore raw

Re: [Qemu-devel] [PATCH v12 3/5] arm_kvm: Do not assume particular GIC type in kvm_arch_irqchip_create()

In reply to Pavel Fedin
Quoted 3 lines · Pavel Fedin

This allows to use different GIC types from v2. There are no kernels which could advertise KVM_CAP_DEVICE_CTRL without the actual ability to create GIC with it.

Minor grammar nit, since you're respinning this series anyway:
"allows us to use".

Quoted 4 lines

thanks
-- PMM

#11 lore raw

Re: [Qemu-devel] [PATCH v12 5/5] hw/arm/virt: Add gic-version option to virt machine

In reply to Pavel Fedin
Quoted 8 lines · Pavel Fedin

Add gic_version to VirtMachineState, set it to value of the option and pass it around where necessary. Instantiate devices and fdt nodes according to the choice.

max_cpus for virt machine increased to 126 (calculated from redistributor space available in the memory map). GICv2 compatibility check happens inside arm_gic_common_realize().

Quoted 5 lines

ITS regions are added to the memory map too, however currently they are not used, just reserved.

We also need to update the ACPI table generation code, otherwise we will be instantiating a system with a GICv3 and claiming in the ACPI tables that it is a GICv2.

Other than that, I think my remaining review comments on this patch should be pretty straightforward to address.

(I also haven't tried running this patchset yet, but I should be able to get it going on a fast model tomorrow I hope.)

Quoted 26 lines
---
 hw/arm/virt.c         | 118 ++++++++++++++++++++++++++++++++++++++++----------
 include/hw/arm/virt.h |   5 ++-
 2 files changed, 99 insertions(+), 24 deletions(-)
hw/arm/virt.c +2 -0
index d5a8417..7044cb9 100644
@@ -50,6 +50,7 @@
 #include "hw/arm/fdt.h"
 #include "hw/intc/arm_gic_common.h"
 #include "kvm_arm.h"
+#include "qapi/visitor.h"
 
 /* Number of external interrupt lines to configure the GIC with */
 #define NUM_IRQS 256
@@ -79,6 +80,7 @@ typedef struct {
 typedef struct {
     MachineState parent;
     bool secure;
+    int32_t gic_version;
 } VirtMachineState;
 
 #define TYPE_VIRT_MACHINE   "virt"

hw/arm/virt.c static const MemMapEntry a15memmap[] = { · line 115

Quoted 5 lines
hw/arm/virt.c +2 -0
@@ -109,6 +111,10 @@ static const MemMapEntry a15memmap[] = {
     [VIRT_GIC_DIST] =           { 0x08000000, 0x00010000 },
     [VIRT_GIC_CPU] =            { 0x08010000, 0x00010000 },
     [VIRT_GIC_V2M] =            { 0x08020000, 0x00001000 },
+    [VIRT_ITS_CONTROL] =        { 0x08020000, 0x00010000 },
+    [VIRT_ITS_TRANSLATION] =    { 0x08030000, 0x00010000 },

Why have separate ITS_CONTROL and ITS_TRANSLATION entries, given that the ITS pages are architecturally required to be contiguous and the dt binding doesn't require you to specify the two addresses separately?

hw/arm/virt.c static const MemMapEntry a15memmap[] = { · line 120

Quoted 5 lines
hw/arm/virt.c +2 -0
+    /* This redistributor space allows up to 2*64kB*126 CPUs */
+    [VIRT_GIC_REDIST] =         { 0x08040000, 0x00FC0000 },
     [VIRT_UART] =               { 0x09000000, 0x00001000 },
     [VIRT_RTC] =                { 0x09010000, 0x00001000 },
     [VIRT_FW_CFG] =             { 0x09020000, 0x0000000a },

Hmm, redistributor space. The good news here is that the architecture (and the dt binding) lets us specify multiple blocks of memory space for redistributors, so we can always raise the CPU limit by adding an extra block of redistributors somewhere else in the memory map (ie above 4GB somewhere) without breaking backwards compatibility. The bad news is that the KVM API for the VGIC only currently supports a single contiguous block of redistributors, so we'll need to extend that somehow when we get to that point.

I think we need to leave enough space for all of GICC/GICV/GICH (that's 2 pages for GICC, 2 for GICV, 1 for GICH). They're optional in a GICv3, but we may want them for emulation later on and if we haven't left ourselves enough space we'll be a bit stuck.

Quoted 6 lines
hw/arm/virt.c +2 -0
@@ -285,6 +294,18 @@ static void fdt_add_cpu_nodes(const VirtBoardInfo *vbi)
 {
     int cpu;
 
+    /*
+     * From Documentation/devicetree/bindings/arm/cpus.txt

hw/arm/virt.c static void fdt_add_cpu_nodes(const VirtBoardInfo *vbi) · line 308

Quoted 10 lines
hw/arm/virt.c +10 -0
+     *  On ARM v8 64-bit systems value should be set to 2,
+     *  that corresponds to the MPIDR_EL1 register size.
+     *  If MPIDR_EL1[63:32] value is equal to 0 on all CPUs
+     *  in the system, #address-cells can be set to 1, since
+     *  MPIDR_EL1[63:32] bits are not used for CPUs
+     *  identification.
+     *
+     *  Now GIC500 doesn't support affinities 2 & 3 so currently
+     *  #address-cells can stay 1 until future GIC
+     */

This change is true even without the GICv3 additions. I think it ought to go in its own patch, together with a check so we can exit with an error if armcpu->mp_affinity is too large to fit in a single dt cell.

Quoted 5 lines
hw/arm/virt.c +0 -0
     qemu_fdt_add_subnode(vbi->fdt, "/cpus");
     qemu_fdt_setprop_cell(vbi->fdt, "/cpus", "#address-cells", 0x1);
     qemu_fdt_setprop_cell(vbi->fdt, "/cpus", "#size-cells", 0x0);
@@ -953,6 +1014,14 @@ static void virt_instance_init(Object *obj)
                                     "Set on/off to enable/disable the ARM "

hw/arm/virt.c static void virt_instance_init(Object *obj) · line 1024

Quoted 10 lines
hw/arm/virt.c +8 -0
                                     "Security Extensions (TrustZone)",
                                     NULL);
+
+    /* Default GIC type is v2 */
+    vms->gic_version = 2;
+    object_property_add(obj, "gic-version", "int", virt_get_gic_version,
+                        virt_set_gic_version, NULL, NULL, NULL);
+    object_property_set_description(obj, "gic-version",
+                                    "Set GIC version. "
+                                    "Valid values are 2 and 3", NULL);

I think we should also allow 'gic-version=host" to use whatever the best VGIC the host kernel supports is (by analogy with -cpu host).

(I'm also starting to think we want a more general board option for "do the best you can for everything" that works for setting both CPU and GIC, and works for both KVM and TCG, but we can add that later.)

thanks
-- PMM

#5 lore raw

Re: [Qemu-devel] [PATCH v12 2/5] intc/gic: Extract some reusable vGIC code

In reply to Peter Maydell

Hello!

Quoted 10 lines

hw/intc/vgic_common.h line 47

Quoted 8 lines
hw/intc/vgic_common.h new file +8 -0
+
+#define kvm_gicd_access(s, offset, cpu, val, write) \
+    kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_DIST_REGS, \
+                      KVM_VGIC_ATTR(offset, cpu), val, write)
+
+#define kvm_gicc_access(s, offset, cpu, val, write) \
+    kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_CPU_REGS, \
+                      KVM_VGIC_ATTR(offset, cpu), val, write)

Can you make these two static inline functions, not #defines, please?

I can, of course, and i did in previous versions, but you disliked it because inline functions would require dev_fd instead of 's', because 's' is different in GICv2 and GICv3. Macro approach allowed to work around this difference. Or do you mean you strictly don't want them to be reusable?

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia

#12 lore raw

Re: [Qemu-devel] [PATCH v12 5/5] hw/arm/virt: Add gic-version option to virt machine

In reply to Peter Maydell

Hello!

Quoted 4 lines

Why have separate ITS_CONTROL and ITS_TRANSLATION entries, given that the ITS pages are architecturally required to be contiguous and the dt binding doesn't require you to specify the two addresses separately?

It is more convenient for ITS implementation. I already have the code for it, and for ITS device it is useful to have two separate sysbus-mmio regions, one for control and one for translation. Because control area is fed to KVM and translation area is handled by qemu in userspace, because it has to catch writes and inject them as KVM_SIGNAL_MSI ioctl's. So, i have to map two regions, therefore i need two bases. Sizes are just added together when i create DT binding. Well, i could do the opposite, store them as a single region in memory map, and just add the appropriate offset when mapping the second sysbus-mmio region. It's the same amount of arithmetic, actually. What would you prefer?

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia

#6 lore raw

Re: [Qemu-devel] [PATCH v12 2/5] intc/gic: Extract some reusable vGIC code

In reply to Pavel Fedin
Quoted 13 lines · Pavel Fedin

Hello!

Quoted 10 lines

hw/intc/vgic_common.h line 47

Quoted 8 lines
hw/intc/vgic_common.h new file +8 -0
+
+#define kvm_gicd_access(s, offset, cpu, val, write) \
+    kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_DIST_REGS, \
+                      KVM_VGIC_ATTR(offset, cpu), val, write)
+
+#define kvm_gicc_access(s, offset, cpu, val, write) \
+    kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_CPU_REGS, \
+                      KVM_VGIC_ATTR(offset, cpu), val, write)

Can you make these two static inline functions, not #defines, please?

Quoted 5 lines

I can, of course, and i did in previous versions, but you disliked it
because inline functions would require dev_fd instead of 's', because 's' is different in GICv2 and GICv3. Macro approach allowed to work around this difference. Or do you mean you strictly don't want them to be reusable?

Oh, right. In that case, yes, I don't think they're worth trying to share between the two devices.

-- PMM

#13 lore raw

Re: [Qemu-devel] [PATCH v12 5/5] hw/arm/virt: Add gic-version option to virt machine

In reply to Pavel Fedin
Quoted 7 lines · Pavel Fedin

Hello!

Quoted 4 lines

Why have separate ITS_CONTROL and ITS_TRANSLATION entries, given that the ITS pages are architecturally required to be contiguous and the dt binding doesn't require you to specify the two addresses separately?

Quoted 8 lines

It is more convenient for ITS implementation. I already have
the code for it, and for ITS device it is useful to have two separate sysbus-mmio regions, one for control and one for translation. Because control area is fed to KVM and translation area is handled by qemu in userspace, because it has to catch writes and inject them as KVM_SIGNAL_MSI ioctl's. So, i have to map two regions, therefore i need two bases. Sizes are just added together when i create DT binding.

This is an internal detail of the device's implementation, so you shouldn't expose it to users of the device. Create a single 128K container memory region, and map your two 64K regions into it. Then just expose the one container region from the GIC device.

thanks
-- PMM

#16 lore raw

Re: [Qemu-devel] [PATCH v12 5/5] hw/arm/virt: Add gic-version option to virt machine

In reply to Peter Maydell

Hello!

Quoted 4 lines

I think we need to leave enough space for all of GICC/GICV/GICH (that's 2 pages for GICC, 2 for GICV, 1 for GICH). They're optional in a GICv3, but we may want them for emulation later on and if we haven't left ourselves enough space we'll be a bit stuck.

Do we really need this? Are we going to have a model with HYP mode, inside of which we could run another model? BTW, our GICv2 implementation also doesn't assume that it can have GICV/GICH. Additionally, according to GIC-500 arch manual, GICC_DIR with affinity routing enabled has an offset of 0x10000, and it's 17 pages instead of 2.
Do we want to waste our precious address space?
But, well, i have calculated that we would have 124 maximum CPUs instead of 126. So - your final word on this?

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia

#17 lore raw

Re: [Qemu-devel] [PATCH v12 5/5] hw/arm/virt: Add gic-version option to virt machine

Quoted 10 lines · Pavel Fedin

Hello!

Quoted 4 lines

I think we need to leave enough space for all of GICC/GICV/GICH (that's 2 pages for GICC, 2 for GICV, 1 for GICH). They're optional in a GICv3, but we may want them for emulation later on and if we haven't left ourselves enough space we'll be a bit stuck.

Do we really need this? Are we going to have a model with HYP mode,
inside of which we could run another model? BTW, our GICv2 implementation also doesn't assume that it can have GICV/GICH.

Yes, I think that implementation of Hyp mode is very likely in the future -- Xilinx want it, for instance.

Quoted 3 lines

Additionally, according to GIC-500 arch manual, GICC_DIR with
affinity routing enabled has an offset of 0x10000, and it's 17 pages instead of 2.

A register at offset 0x10000 means 2 64K pages, doesn't it? Can you give a more exact reference to the bit of the manual you mean?

Quoted 3 lines

Do we want to waste our precious address space?
But, well, i have calculated that we would have 124 maximum CPUs
instead of 126. So - your final word on this?

Two fewer CPUs is not a big deal: anybody who finds 124 CPUs a restrictive limit would probably also find 126 CPUs too few. We can just add support for split redistributor ranges in KVM before we get to the point where it's a problem.

thanks
-- PMM

#18 lore raw

Re: [Qemu-devel] [PATCH v12 5/5] hw/arm/virt: Add gic-version option to virt machine

Hello!

Quoted 2 lines

Yes, I think that implementation of Hyp mode is very likely in the future -- Xilinx want it, for instance.

Ok. Will add it.

Quoted 3 lines

A register at offset 0x10000 means 2 64K pages, doesn't it? Can you give a more exact reference to the bit of the manual you mean?

I have mistaken 4K for page. Just ignore this :)
My manual is IHI0069A "ARM® Generic Interrupt Controller Architecture Specification GIC architecture version 3.0 and version 4.0". And i'm really not very careful in reading it...

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia

#15 lore raw

Re: [Qemu-devel] [PATCH v12 5/5] hw/arm/virt: Add gic-version option to virt machine

In reply to Diana Craciun

Hello!

Quoted 2 lines

Do you plan posting the ITS patches on the list? I am interested in reviewing and testing them.

Yes, i do. However i am waiting for prerequisites (base vGICv3 in qemu and ITS emulation in kernel to get in). I don't like to post patch-on-patch-on-patch because i'm in doubt anyone would be bothered, neither it's useful.
But, ok, since you request it, i'll post it after vGICv3 v13, which will address latest comments.
I could also share my userspace implementation of the ITS. However it will not work because Shlomo's TCG GICv3 does not have LPIs. It can be interesting for any of you if you want to continue this work. It is very preliminary but i got it to the point where it injects an LPI into the GIC. I wrote it when there was no in-kernel ITS patches, i supposed to implement raw LPIs in kernel and ITS in userspace, similar to GICv2m.

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia

#14 lore raw

Re: [Qemu-devel] [PATCH v12 5/5] hw/arm/virt: Add gic-version option to virt machine

In reply to Pavel Fedin

Hi Pavel,

Quoted 1 line · Pavel Fedin

It is more convenient for ITS implementation. I already have the code for it, and for ITS device it is useful to have two separate sysbus-mmio regions, one for control and one for translation.

Do you plan posting the ITS patches on the list? I am interested in reviewing and testing them.

Thanks,
Diana

#19 lore raw

Re: [Qemu-devel] [PATCH v12 5/5] hw/arm/virt: Add gic-version option to virt machine

In reply to Peter Maydell

Hello!

Quoted 3 lines

We also need to update the ACPI table generation code, otherwise we will be instantiating a system with a GICv3 and claiming in the ACPI tables that it is a GICv2.

I have done everything except this piece. Where can i get any documentation on how to describe GICv3 in MADT?

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia