Skip to contentSkip to conversation

hvf: arm: Add support for GICv3

PATCH Superseded Alexander Graf 4 participants 8 messages Dec 19, 2022 – Mar 10, 2023

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 hvf: arm: Add support for GICv3

We currently only support GICv2 emulation. To also support GICv3, we will need to pass a few system registers into their respective handler functions.

This patch adds support for HVF to call into the TCG callbacks for GICv3 system register handlers. This is safe because the GICv3 TCG code is generic as long as we limit ourselves to EL0 and EL1 - which are the only modes supported by HVF.

To make sure nobody trips over that, we also annotate callbacks that don't work in HVF mode, such as EL state change hooks.

With GICv3 support in place, we can run with more than 8 vCPUs.

---
 hw/intc/arm_gicv3_cpuif.c   |   8 +-
 target/arm/hvf/hvf.c        | 151 ++++++++++++++++++++++++++++++++++++
 target/arm/hvf/trace-events |   2 +
 3 files changed, 160 insertions(+), 1 deletion(-)
hw/intc/arm_gicv3_cpuif.c +7 -1
index b17b29288c..b4e387268c 100644
@@ -21,6 +21,7 @@
 #include "hw/irq.h"
 #include "cpu.h"
 #include "target/arm/cpregs.h"
+#include "sysemu/tcg.h"
 
 /*
  * Special case return value from hppvi_index(); must be larger than
@@ -2810,6 +2811,8 @@ void gicv3_init_cpuif(GICv3State *s)
          * which case we'd get the wrong value.
          * So instead we define the regs with no ri->opaque info, and
          * get back to the GICv3CPUState from the CPUARMState.
+         *
+         * These CP regs callbacks can be called from either TCG or HVF code.
          */
         define_arm_cp_regs(cpu, gicv3_cpuif_reginfo);
 
@@ -2905,6 +2908,9 @@ void gicv3_init_cpuif(GICv3State *s)
                 define_arm_cp_regs(cpu, gicv3_cpuif_ich_apxr23_reginfo);
             }
         }
-        arm_register_el_change_hook(cpu, gicv3_cpuif_el_change_hook, cs);
+        if (tcg_enabled()) {
+            /* We can only trap EL changes with TCG for now */
+            arm_register_el_change_hook(cpu, gicv3_cpuif_el_change_hook, cs);
+        }
     }
 }
target/arm/hvf/hvf.c +151 -0

181 lines not on this page · raw message

target/arm/hvf/trace-events +2 -0
index 820e8e0297..4fbbe4b45e 100644
@@ -9,3 +9,5 @@ hvf_unknown_hvc(uint64_t x0) "unknown HVC! 0x%016"PRIx64
 hvf_unknown_smc(uint64_t x0) "unknown SMC! 0x%016"PRIx64
 hvf_exit(uint64_t syndrome, uint32_t ec, uint64_t pc) "exit: 0x%"PRIx64" [ec=0x%x pc=0x%"PRIx64"]"
 hvf_psci_call(uint64_t x0, uint64_t x1, uint64_t x2, uint64_t x3, uint32_t cpuid) "PSCI Call x0=0x%016"PRIx64" x1=0x%016"PRIx64" x2=0x%016"PRIx64" x3=0x%016"PRIx64" cpu=0x%x"
+hvf_vgic_write(const char *name, uint64_t val) "vgic write to %s [val=0x%016"PRIx64"]"
+hvf_vgic_read(const char *name, uint64_t val) "vgic read from %s [val=0x%016"PRIx64"]"
Signature · 1 line
2.37.1 (Apple Git-137.1)
#1 lore raw
Quoted 1 line · Alexander Graf

We currently only support GICv2 emulation.

Before looking into it, I think it's worth finalizing the GIC version in the hvf case - only v2 is allowed and fail early if user selects the unsupported versions. Currently finalize_gic_version() does not deal with hvf at all.

#2 lore raw

Hi Zenghui,

Quoted 7 lines · Zenghui Yu
Quoted 1 line · Alexander Graf

We currently only support GICv2 emulation.

Before looking into it, I think it's worth finalizing the GIC version in the hvf case - only v2 is allowed and fail early if user selects the unsupported versions. Currently finalize_gic_version() does not deal with hvf at all.

Currently finalize_gic_version() treats HVF the same as TCG, which is incorrect. However, with this patch applied, they happen to match.

I don't think it's worth changing the finalize_gic_version() implementation to reflect the gicv2 only state for HVF. Instead, let's rather get GICv3 support in and then add explicit handling for HVF on top.

Alex

#3 lore raw
Quoted 12 lines · Alexander Graf

Hi Zenghui,

Quoted 7 lines · Zenghui Yu
Quoted 1 line · Alexander Graf

We currently only support GICv2 emulation.

Before looking into it, I think it's worth finalizing the GIC version in the hvf case - only v2 is allowed and fail early if user selects the unsupported versions. Currently finalize_gic_version() does not deal with hvf at all.

Quoted 3 lines

Currently finalize_gic_version() treats HVF the same as TCG, which is incorrect. However, with this patch applied, they happen to match.

I don't think it's worth changing the finalize_gic_version() implementation to reflect the gicv2 only state for HVF. Instead, let's rather get GICv3 support in and then add explicit handling for HVF on top.

I'm fine with it. I was just a bit confused when the guest got stuck with a (blindly selected) gic-version but *no* error_report like "Hey, please do not try GIC version other than v2" was printed. Apparently I'm not familiar with how things work on my M1. ;-)

I'll have a try with this patch. Thanks.

#4 lore raw
In reply to Alexander Graf
Quoted 43 lines · Alexander Graf

We currently only support GICv2 emulation. To also support GICv3, we will need to pass a few system registers into their respective handler functions.

This patch adds support for HVF to call into the TCG callbacks for GICv3 system register handlers. This is safe because the GICv3 TCG code is generic as long as we limit ourselves to EL0 and EL1 - which are the only modes supported by HVF.

To make sure nobody trips over that, we also annotate callbacks that don't work in HVF mode, such as EL state change hooks.

With GICv3 support in place, we can run with more than 8 vCPUs.

---
 hw/intc/arm_gicv3_cpuif.c   |   8 +-
 target/arm/hvf/hvf.c        | 151 ++++++++++++++++++++++++++++++++++++
 target/arm/hvf/trace-events |   2 +
 3 files changed, 160 insertions(+), 1 deletion(-)
hw/intc/arm_gicv3_cpuif.c +3 -0
index b17b29288c..b4e387268c 100644
@@ -21,6 +21,7 @@
 #include "hw/irq.h"
 #include "cpu.h"
 #include "target/arm/cpregs.h"
+#include "sysemu/tcg.h"
 
 /*
  * Special case return value from hppvi_index(); must be larger than
@@ -2810,6 +2811,8 @@ void gicv3_init_cpuif(GICv3State *s)
          * which case we'd get the wrong value.
          * So instead we define the regs with no ri->opaque info, and
          * get back to the GICv3CPUState from the CPUARMState.
+         *
+         * These CP regs callbacks can be called from either TCG or HVF code.
          */
         define_arm_cp_regs(cpu, gicv3_cpuif_reginfo);
 

hw/intc/arm_gicv3_cpuif.c void gicv3_init_cpuif(GICv3State *s) · line 2912

Quoted 6 lines
hw/intc/arm_gicv3_cpuif.c +2 -1
@@ -2905,6 +2908,9 @@ void gicv3_init_cpuif(GICv3State *s)
                 define_arm_cp_regs(cpu, gicv3_cpuif_ich_apxr23_reginfo);
             }
         }
-        arm_register_el_change_hook(cpu, gicv3_cpuif_el_change_hook, cs);
+        if (tcg_enabled()) {
+            /* We can only trap EL changes with TCG for now */

We could expand this a bit:

We can only trap EL changes with TCG. However the GIC interrupt state only changes on EL changes involving EL2 or EL3, so for the non-TCG case this is OK, as EL2 and EL3 can't exist.

and assert:
assert(!arm_feature(&cpu->env, ARM_FEATURE_EL2)); assert(!arm_feature(&cpu->env, ARM_FEATURE_EL3));

hw/intc/arm_gicv3_cpuif.c void gicv3_init_cpuif(GICv3State *s)

Quoted 8 lines
hw/intc/arm_gicv3_cpuif.c +8 -0
+static uint32_t hvf_reg2cp_reg(uint32_t reg)
+{
+    return ENCODE_AA64_CP_REG(CP_REG_ARM64_SYSREG_CP,
+                              (reg >> 10) & 0xf,
+                              (reg >> 1) & 0xf,
+                              (reg >> 20) & 0x3,
+                              (reg >> 14) & 0x7,
+                              (reg >> 17) & 0x7);

This file has #defines for these shift and mask constants (SYSREG_OP0_SHIFT etc).

Quoted 15 lines
hw/intc/arm_gicv3_cpuif.c +15 -0
+}
+
+static bool hvf_sysreg_read_cp(CPUState *cpu, uint32_t reg, uint64_t *val)
+{
+    ARMCPU *arm_cpu = ARM_CPU(cpu);
+    CPUARMState *env = &arm_cpu->env;
+    const ARMCPRegInfo *ri;
+
+    ri = get_arm_cp_reginfo(arm_cpu->cp_regs, hvf_reg2cp_reg(reg));
+    if (ri) {
+        if (ri->accessfn) {
+            if (ri->accessfn(env, ri, true) != CP_ACCESS_OK) {
+                return false;
+            }
+        }

hw/intc/arm_gicv3_cpuif.c

Quoted 10 lines
hw/intc/arm_gicv3_cpuif.c +10 -0
+        if (ri->type & ARM_CP_CONST) {
+            *val = ri->resetvalue;
+        } else if (ri->readfn) {
+            *val = ri->readfn(env, ri);
+        } else {
+            *val = CPREG_FIELD64(env, ri);
+        }
+        trace_hvf_vgic_read(ri->name, *val);
+        return true;
+    }

Can we get here for attempts by EL0 to access EL1-only sysregs, or does hvf send the exception to EL1 without trapping out to us? If we can get here for EL0 accesses we need to check against ri->access as well as ri->accessfn.

Otherwise looks OK.

thanks

-- PMM

#5 lore raw
Quoted 51 lines · Peter Maydell
Quoted 42 lines · Alexander Graf

We currently only support GICv2 emulation. To also support GICv3, we will need to pass a few system registers into their respective handler functions.

This patch adds support for HVF to call into the TCG callbacks for GICv3 system register handlers. This is safe because the GICv3 TCG code is generic as long as we limit ourselves to EL0 and EL1 - which are the only modes supported by HVF.

To make sure nobody trips over that, we also annotate callbacks that don't work in HVF mode, such as EL state change hooks.

With GICv3 support in place, we can run with more than 8 vCPUs.

---
  hw/intc/arm_gicv3_cpuif.c   |   8 +-
  target/arm/hvf/hvf.c        | 151 ++++++++++++++++++++++++++++++++++++
  target/arm/hvf/trace-events |   2 +
  3 files changed, 160 insertions(+), 1 deletion(-)
hw/intc/arm_gicv3_cpuif.c +3 -0
index b17b29288c..b4e387268c 100644
@@ -21,6 +21,7 @@
  #include "hw/irq.h"
  #include "cpu.h"
  #include "target/arm/cpregs.h"
+#include "sysemu/tcg.h"
 
  /*
   * Special case return value from hppvi_index(); must be larger than
@@ -2810,6 +2811,8 @@ void gicv3_init_cpuif(GICv3State *s)
           * which case we'd get the wrong value.
           * So instead we define the regs with no ri->opaque info, and
           * get back to the GICv3CPUState from the CPUARMState.
+         *
+         * These CP regs callbacks can be called from either TCG or HVF code.
           */
          define_arm_cp_regs(cpu, gicv3_cpuif_reginfo);
 

hw/intc/arm_gicv3_cpuif.c void gicv3_init_cpuif(GICv3State *s) · line 2912

Quoted 6 lines
hw/intc/arm_gicv3_cpuif.c +2 -1
@@ -2905,6 +2908,9 @@ void gicv3_init_cpuif(GICv3State *s)
                  define_arm_cp_regs(cpu, gicv3_cpuif_ich_apxr23_reginfo);
              }
          }
-        arm_register_el_change_hook(cpu, gicv3_cpuif_el_change_hook, cs);
+        if (tcg_enabled()) {
+            /* We can only trap EL changes with TCG for now */

We could expand this a bit:

Quoted 7 lines

We can only trap EL changes with TCG. However the GIC interrupt state only changes on EL changes involving EL2 or EL3, so for the non-TCG case this is OK, as EL2 and EL3 can't exist.

and assert:
assert(!arm_feature(&cpu->env, ARM_FEATURE_EL2)); assert(!arm_feature(&cpu->env, ARM_FEATURE_EL3));

Good idea! Let me add that.

Quoted 11 lines

hw/intc/arm_gicv3_cpuif.c void gicv3_init_cpuif(GICv3State *s)

Quoted 9 lines
hw/intc/arm_gicv3_cpuif.c +8 -0
+static uint32_t hvf_reg2cp_reg(uint32_t reg)
+{
+    return ENCODE_AA64_CP_REG(CP_REG_ARM64_SYSREG_CP,
+                              (reg >> 10) & 0xf,
+                              (reg >> 1) & 0xf,
+                              (reg >> 20) & 0x3,
+                              (reg >> 14) & 0x7,
+                              (reg >> 17) & 0x7);

This file has #defines for these shift and mask constants (SYSREG_OP0_SHIFT etc).

Ugh, thanks for catching that!

Quoted 22 lines
Quoted 22 lines
hw/intc/arm_gicv3_cpuif.c +21 -0
+}
+
+static bool hvf_sysreg_read_cp(CPUState *cpu, uint32_t reg, uint64_t *val)
+{
+    ARMCPU *arm_cpu = ARM_CPU(cpu);
+    CPUARMState *env = &arm_cpu->env;
+    const ARMCPRegInfo *ri;
+
+    ri = get_arm_cp_reginfo(arm_cpu->cp_regs, hvf_reg2cp_reg(reg));
+    if (ri) {
+        if (ri->accessfn) {
+            if (ri->accessfn(env, ri, true) != CP_ACCESS_OK) {
+                return false;
+            }
+        }
+        if (ri->type & ARM_CP_CONST) {
+            *val = ri->resetvalue;
+        } else if (ri->readfn) {
+            *val = ri->readfn(env, ri);
+        } else {
+            *val = CPREG_FIELD64(env, ri);
Quoted 8 lines

hw/intc/arm_gicv3_cpuif.c

Quoted 4 lines
hw/intc/arm_gicv3_cpuif.c +4 -0
+        }
+        trace_hvf_vgic_read(ri->name, *val);
+        return true;
+    }

Can we get here for attempts by EL0 to access EL1-only sysregs, or does hvf send the exception to EL1 without trapping out to us? If we can get here for EL0 accesses we need to check against ri->access as well as ri->accessfn.

I just validated, GICv3 EL1 registers trap to EL1 inside the guest:

$ cat a.S
.global start
.global _main
_main:
start:
        mrs x0, ICC_AP0R0_EL1
        mov x0, #0x1234
        msr ICC_AP0R0_EL1, x0
        mov x0, #0
        ret
$ gcc -nostdlib a.S
$ gdb ./a.out
(gdb) r
Program received signal SIGILL, Illegal instruction.
0x00000000004000d4 in start ()
(gdb) x/i $pc
=> 0x4000d4 <start>:        mrs     x0, icc_ap0r0_el1

So no need to check ri->access :)

Alex

Tested-by on v1 1/1
#6 lore raw
In reply to Alexander Graf
Quoted 9 lines · Alexander Graf

We currently only support GICv2 emulation. To also support GICv3, we will need to pass a few system registers into their respective handler functions.

This patch adds support for HVF to call into the TCG callbacks for GICv3 system register handlers. This is safe because the GICv3 TCG code is generic as long as we limit ourselves to EL0 and EL1 - which are the only modes supported by HVF.

Quoted 7 lines

To make sure nobody trips over that, we also annotate callbacks that don't work in HVF mode, such as EL state change hooks.

With GICv3 support in place, we can run with more than 8 vCPUs.

---

#7 lore raw
Quoted 14 lines · Joelle van Dyne
Quoted 12 lines · Alexander Graf

We currently only support GICv2 emulation. To also support GICv3, we will need to pass a few system registers into their respective handler functions.

This patch adds support for HVF to call into the TCG callbacks for GICv3 system register handlers. This is safe because the GICv3 TCG code is generic as long as we limit ourselves to EL0 and EL1 - which are the only modes supported by HVF.

To make sure nobody trips over that, we also annotate callbacks that don't work in HVF mode, such as EL state change hooks.

Quoted 6 lines
Quoted 4 lines

With GICv3 support in place, we can run with more than 8 vCPUs.

---

Just FYI, this patch is already upstream (commit a2260983c6553).

-- PMM