Skip to contentSkip to conversation

hvf: Implement Apple Silicon Support

PATCH Superseded Alexander Graf 4 participants 32 messages Sep 15, 2021 – Sep 22, 2021 Arm/Amlogic Meson SoC support

v11 of 12 Superseded Latest is v12 →

lore
Share & tools
lore
#0 lore raw

v11 cover hvf: Implement Apple Silicon Support

Now that Apple Silicon is widely available, people are obviously excited to try and run virtualized workloads on them, such as Linux and Windows.

This patch set implements a fully functional version to get the ball going on that. With this applied, I can successfully run both Linux and Windows as guests. I am not aware of any limitations specific to Hypervisor.framework apart from:

- gdbstub debugging (breakpoints)
- missing GICv3 support
- Windows will not work due to UDEF SMC implementation

To use hvf support, please make sure to run -M virt,highmem=off to fit in M1's physical address space limits and use -cpu host.

Enjoy!

Alex

v1 -> v2:

- New patch: hvf: Actually set SIG_IPI mask
- New patch: hvf: Introduce hvf vcpu struct
- New patch: hvf: arm: Mark CPU as dirty on reset
- Removed patch: hw/arm/virt: Disable highmem when on hypervisor.framework
- Removed patch: arm: Synchronize CPU on PSCI on
- Fix build on 32bit arm
- Merge vcpu kick function patch into ARM enablement
- Implement WFI handling (allows vCPUs to sleep)
- Synchronize system registers (fixes OVMF crashes and reboot)
- Don't always call cpu_synchronize_state()
- Use more fine grained iothread locking
- Populate aa64mmfr0 from hardware
- Make safe to ctrl-C entitlement application

v2 -> v3:

- Removed patch: hvf: Actually set SIG_IPI mask
- New patch: hvf: arm: Add support for GICv3
- New patch: hvf: arm: Implement -cpu host
- Advance PC on SMC
- Use cp list interface for sysreg syncs
- Do not set current_cpu
- Fix sysreg isread mask
- Move sysreg handling to functions
- Remove WFI logic again
- Revert to global iothread locking

v3 -> v4:

- Removed patch: hvf: arm: Mark CPU as dirty on reset
- New patch: hvf: Simplify post reset/init/loadvm hooks
- Remove i386-softmmu target (meson.build for hvf target)
- Combine both if statements (PSCI)
- Use hv.h instead of Hypervisor.h for 10.15 compat
- Remove manual inclusion of Hypervisor.h in common .c files
- No longer include Hypervisor.h in arm hvf .c files
- Remove unused exe_full variable
- Reuse exe_name variable

v4 -> v5:

- Use g_free() on destroy

v5 -> v6:

- Switch SYSREG() macro order to the same as asm intrinsics

v6 -> v7:

- Already merged: hvf: Add hypervisor entitlement to output binaries
- Already merged: hvf: x86: Remove unused definitions
- Patch split: hvf: Move common code out

    -> hvf: Move assert_hvf_ok() into common directory
    -> hvf: Move vcpu thread functions into common directory
    -> hvf: Move cpu functions into common directory
    -> hvf: Move hvf internal definitions into common header
    -> hvf: Make hvf_set_phys_mem() static
    -> hvf: Remove use of hv_uvaddr_t and hv_gpaddr_t
    -> hvf: Split out common code on vcpu init and destroy
    -> hvf: Use cpu_synchronize_state()
    -> hvf: Make synchronize functions static
    -> hvf: Remove hvf-accel-ops.h

- New patch: hvf: arm: Implement PSCI handling
- New patch: arm: Enable Windows 10 trusted SMCCC boot call
- New patch: hvf: arm: Handle Windows 10 SMC call
- Removed patch: "arm: Set PSCI to 0.2 for HVF" (included above)
- Removed patch: "hvf: arm: Add support for GICv3" (deferred to later)
- Remove osdep.h include from hvf_int.h
- Synchronize SIMD registers as well
- Prepend 0x for hex values
- Convert DPRINTF to trace points
- Use main event loop (fixes gdbstub issues)
- Remove PSCI support, inject UDEF on HVC/SMC
- Change vtimer logic to look at ctl.istatus for vtimer mask sync
- Add kick callback again (fixes remote CPU notification)
- Move function define to own header
- Do not propagate SVE features for HVF
- Remove stray whitespace change
- Verify that EL0 and EL1 do not allow AArch32 mode
- Only probe host CPU features once
- Move WFI into function
- Improve comment wording
- Simplify HVF matching logic in meson build file

v7 -> v8:

- checkpatch fixes
- Do not advance for HVC, PC is already updated by hvf (fixes Linux boot)

v8 -> v9:

- [Merged] hvf: Move assert_hvf_ok() into common directory
- [Merged] hvf: Move vcpu thread functions into common directory
- [Merged] hvf: Move cpu functions into common directory
- [Merged] hvf: Move hvf internal definitions into common header
- [Merged] hvf: Make hvf_set_phys_mem() static
- [Merged] hvf: Remove use of hv_uvaddr_t and hv_gpaddr_t
- [Merged] hvf: Split out common code on vcpu init and destroy
- [Merged] hvf: Use cpu_synchronize_state()
- [Merged] hvf: Make synchronize functions static
- [Merged] hvf: Remove hvf-accel-ops.h
- [Merged] hvf: Introduce hvf vcpu struct
- [Merged] hvf: Simplify post reset/init/loadvm hooks
- [Dropped] arm: Enable Windows 10 trusted SMCCC boot call
- [Dropped] hvf: arm: Handle Windows 10 SMC call
- [New] arm: Move PMC register definitions to cpu.h
- [New] hvf: Add execute to dirty log permission bitmap
- [New] hvf: Introduce hvf_arch_init() callback
- [New] hvf: arm: Implement PSCI handling
- [New] hvf: arm: Add rudimentary PMC support
- [New] arm: tcg: Adhere to SMCCC 1.3 section 5.2
- [New] hvf: arm: Adhere to SMCCC 1.3 section 5.2
- Make kick function non-weak
- Use arm_cpu_do_interrupt()
- Remove CNTPCT_EL0 write case
- Inject UDEF on invalid sysreg access
- Add support for OS locking sysregs
- Remove PMCCNTR_EL0 handling
- Print PC on unhandled sysreg trace
- Sync SP (x31) based on SP_EL0/SP_EL1
- Fix SPSR_EL1 mapping
- Only sync known sysregs, assert when syncing fails
- Improve error message on unhandled ec
- Move vtimer sync to post-exit (fixes disable corner case from kvm-unit-tests)
- Add vtimer offset, migration and pause logic
- Flush registers only after EXCP checkers (fixes PSCI on race)
- Remove Windows specifics and just comply with SMCCC spec
- Zero-initialize host_isar
- Use M1 SCTLR reset value
- Add support for cntv offsets
- Improve code readability
- Use new hvf_raise_exception() prototype
- Make cpu_off function void
- Add comment about return value, use -1 for "not found"
- Remove cpu_synchronize_state() when halted

v9 -> v10:

- [Dropped] hvf: arm: Adhere to SMCCC 1.3 section 5.2
- Only handle PSCI calls for the current conduit
- Return true/false
- Return -1 in x0 on unknown SMC/HVC calls
- Move to target/arm/internals.h
- Fail -cpu host class creation gracefully
- Adjust error message on -cpu host realize failure
- Extend SCTLR comment that hvf returns 0 as default value
- Return true/false
- Report errors lazily
- Fix comment

v10 -> v11:

- Treat SMC as UDEF. A follow-up patch set will try to change behavior consistently in TCG as well as HVF.

Alexander Graf (9):
arm: Move PMC register definitions to cpu.h
hvf: Add execute to dirty log permission bitmap
hvf: Introduce hvf_arch_init() callback
hvf: Add Apple Silicon support
hvf: arm: Implement -cpu host
hvf: arm: Implement PSCI handling
arm: Add Hypervisor.framework build target
hvf: arm: Add rudimentary PMC support
arm: tcg: Adhere to SMCCC 1.3 section 5.2

Peter Collingbourne (1):
arm/hvf: Add a WFI handler

 MAINTAINERS                 |    5 +
 accel/hvf/hvf-accel-ops.c   |   21 +-
 include/sysemu/hvf_int.h    |   12 +-
 meson.build                 |    8 +
 target/arm/cpu.c            |   17 +-
 target/arm/cpu.h            |    2 +
 target/arm/helper.c         |   44 --
 target/arm/hvf/hvf.c        | 1268 +++++++++++++++++++++++++++++++++++
 target/arm/hvf/meson.build  |    3 +
 target/arm/hvf/trace-events |   11 +
 target/arm/hvf_arm.h        |   19 +
 target/arm/internals.h      |   44 ++
 target/arm/kvm_arm.h        |    2 -
 target/arm/meson.build      |    2 +
 target/arm/psci.c           |   35 +-
 target/i386/hvf/hvf.c       |   10 +
 16 files changed, 1415 insertions(+), 88 deletions(-)
 create mode 100644 target/arm/hvf/hvf.c
 create mode 100644 target/arm/hvf/meson.build
 create mode 100644 target/arm/hvf/trace-events
 create mode 100644 target/arm/hvf_arm.h
Signature · 1 line
2.30.1 (Apple Git-130)
#1 lore raw

PATCH v11 1/10 arm: Move PMC register definitions to cpu.h

We will need PMC register definitions in accel specific code later. Move all constant definitions to common arm headers so we can reuse them.

---

v9 -> v10:

- Move to target/arm/internals.h

---
 target/arm/helper.c    | 44 ------------------------------------------
 target/arm/internals.h | 44 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 44 insertions(+), 44 deletions(-)
target/arm/helper.c +0 -44
index b210da2bc2..21ee94ec2e 100644
@@ -1114,50 +1114,6 @@ static const ARMCPRegInfo v6_cp_reginfo[] = {
     REGINFO_SENTINEL
 };
 
-/* Definitions for the PMU registers */
-#define PMCRN_MASK  0xf800
-#define PMCRN_SHIFT 11
-#define PMCRLC  0x40
-#define PMCRDP  0x20
-#define PMCRX   0x10
-#define PMCRD   0x8
-#define PMCRC   0x4
-#define PMCRP   0x2
-#define PMCRE   0x1
-/*
- * Mask of PMCR bits writeable by guest (not including WO bits like C, P,
- * which can be written as 1 to trigger behaviour but which stay RAZ).
- */
-#define PMCR_WRITEABLE_MASK (PMCRLC | PMCRDP | PMCRX | PMCRD | PMCRE)
-
-#define PMXEVTYPER_P          0x80000000
-#define PMXEVTYPER_U          0x40000000
-#define PMXEVTYPER_NSK        0x20000000
-#define PMXEVTYPER_NSU        0x10000000
-#define PMXEVTYPER_NSH        0x08000000
-#define PMXEVTYPER_M          0x04000000
-#define PMXEVTYPER_MT         0x02000000
-#define PMXEVTYPER_EVTCOUNT   0x0000ffff
-#define PMXEVTYPER_MASK       (PMXEVTYPER_P | PMXEVTYPER_U | PMXEVTYPER_NSK | \
-                               PMXEVTYPER_NSU | PMXEVTYPER_NSH | \
-                               PMXEVTYPER_M | PMXEVTYPER_MT | \
-                               PMXEVTYPER_EVTCOUNT)
-
-#define PMCCFILTR             0xf8000000
-#define PMCCFILTR_M           PMXEVTYPER_M
-#define PMCCFILTR_EL0         (PMCCFILTR | PMCCFILTR_M)
-
-static inline uint32_t pmu_num_counters(CPUARMState *env)
-{
-  return (env->cp15.c9_pmcr & PMCRN_MASK) >> PMCRN_SHIFT;
-}
-
-/* Bits allowed to be set/cleared for PMCNTEN* and PMINTEN* */
-static inline uint64_t pmu_counter_mask(CPUARMState *env)
-{
-  return (1 << 31) | ((1 << pmu_num_counters(env)) - 1);
-}
-
 typedef struct pm_event {
     uint16_t number; /* PMEVTYPER.evtCount is 16 bits wide */
     /* If the event is supported on this CPU (used to generate PMCEID[01]) */
target/arm/internals.h +44 -0

48 lines not on this page · raw message

Signature · 1 line
2.30.1 (Apple Git-130)
Reviewed-by on v11 1/10
#2 lore raw

Re: [PATCH v11 01/10] arm: Move PMC register definitions to cpu.h

Quoted 6 lines · Alexander Graf

We will need PMC register definitions in accel specific code later. Move all constant definitions to common arm headers so we can reuse them.

Subject should say internals.h.

Otherwise

PS: your mail server has been sending me delivery failure messages because it seems to be subscribed to a blocklist that has blocked one of gmail's outgoing mail servers:

554 5.7.1 Service unavailable; Client host [209.85.128.51] blocked using dnsbl.sorbs.net; Currently Sending Spam See:
http://www.sorbs.net/lookup.shtml?209.85.128.51

-- PMM

#3 lore raw

PATCH v11 2/10 hvf: Add execute to dirty log permission bitmap

In reply to Alexander Graf

Hvf's permission bitmap during and after dirty logging does not include the HV_MEMORY_EXEC permission. At least on Apple Silicon, this leads to instruction faults once dirty logging was enabled.

Add the bit to make it work properly.

---
 accel/hvf/hvf-accel-ops.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
accel/hvf/hvf-accel-ops.c +2 -2
index d1691be989..71cc2fa70f 100644
@@ -239,12 +239,12 @@ static void hvf_set_dirty_tracking(MemoryRegionSection *section, bool on)
     if (on) {
         slot->flags |= HVF_SLOT_LOG;
         hv_vm_protect((uintptr_t)slot->start, (size_t)slot->size,
-                      HV_MEMORY_READ);
+                      HV_MEMORY_READ | HV_MEMORY_EXEC);
     /* stop tracking region*/
     } else {
         slot->flags &= ~HVF_SLOT_LOG;
         hv_vm_protect((uintptr_t)slot->start, (size_t)slot->size,
-                      HV_MEMORY_READ | HV_MEMORY_WRITE);
+                      HV_MEMORY_READ | HV_MEMORY_WRITE | HV_MEMORY_EXEC);
     }
 }
 
Signature · 1 line
2.30.1 (Apple Git-130)
Reviewed-by on v11 2/10
#4 lore raw

Re: [PATCH v11 02/10] hvf: Add execute to dirty log permission bitmap

Quoted 20 lines · Alexander Graf

Hvf's permission bitmap during and after dirty logging does not include the HV_MEMORY_EXEC permission. At least on Apple Silicon, this leads to instruction faults once dirty logging was enabled.

Add the bit to make it work properly.

---
 accel/hvf/hvf-accel-ops.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
accel/hvf/hvf-accel-ops.c +0 -0
index d1691be989..71cc2fa70f 100644
@@ -239,12 +239,12 @@ static void hvf_set_dirty_tracking(MemoryRegionSection *section, bool on)
     if (on) {
         slot->flags |= HVF_SLOT_LOG;
         hv_vm_protect((uintptr_t)slot->start, (size_t)slot->size,

accel/hvf/hvf-accel-ops.c static void hvf_set_dirty_tracking(MemoryRegionSection *section, bool on) · line 249

Quoted 10 lines
accel/hvf/hvf-accel-ops.c +2 -2
-                      HV_MEMORY_READ);
+                      HV_MEMORY_READ | HV_MEMORY_EXEC);
     /* stop tracking region*/
     } else {
         slot->flags &= ~HVF_SLOT_LOG;
         hv_vm_protect((uintptr_t)slot->start, (size_t)slot->size,
-                      HV_MEMORY_READ | HV_MEMORY_WRITE);
+                      HV_MEMORY_READ | HV_MEMORY_WRITE | HV_MEMORY_EXEC);
     }
 }

Makes sense -- this matches the premissions we set initially for memory regions in hvf_set_phys_mem().

Should we change also the hv_vm_protect() call in target/i386/hvf/hvf.c:ept_emulation_fault(), for consistency ?

thanks
-- PMM

#5 lore raw

Re: [PATCH v11 02/10] hvf: Add execute to dirty log permission bitmap

Quoted 33 lines · Peter Maydell
Quoted 19 lines · Alexander Graf

Hvf's permission bitmap during and after dirty logging does not include the HV_MEMORY_EXEC permission. At least on Apple Silicon, this leads to instruction faults once dirty logging was enabled.

Add the bit to make it work properly.

---
 accel/hvf/hvf-accel-ops.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
accel/hvf/hvf-accel-ops.c +0 -0
index d1691be989..71cc2fa70f 100644
@@ -239,12 +239,12 @@ static void hvf_set_dirty_tracking(MemoryRegionSection *section, bool on)
     if (on) {
         slot->flags |= HVF_SLOT_LOG;
         hv_vm_protect((uintptr_t)slot->start, (size_t)slot->size,

accel/hvf/hvf-accel-ops.c static void hvf_set_dirty_tracking(MemoryRegionSection *section, bool on) · line 249

Quoted 10 lines
accel/hvf/hvf-accel-ops.c +2 -2
-                      HV_MEMORY_READ);
+                      HV_MEMORY_READ | HV_MEMORY_EXEC);
     /* stop tracking region*/
     } else {
         slot->flags &= ~HVF_SLOT_LOG;
         hv_vm_protect((uintptr_t)slot->start, (size_t)slot->size,
-                      HV_MEMORY_READ | HV_MEMORY_WRITE);
+                      HV_MEMORY_READ | HV_MEMORY_WRITE | HV_MEMORY_EXEC);
     }
 }

Makes sense -- this matches the premissions we set initially for memory regions in hvf_set_phys_mem().

Quoted 4 lines

Should we change also the hv_vm_protect() call in target/i386/hvf/hvf.c:ept_emulation_fault(), for consistency ?

The x86 hvf code seems to deal just fine with !X mappings, so I'd leave it as is as part of the arm enablement series - especially because I have limited testing capabilities for x86 hvf.

Roman, Cameron, would you like to pick up the ept_emulation_fault() part?

Alex

#6 lore raw

Re: [PATCH v11 02/10] hvf: Add execute to dirty log permission bitmap

Quoted 38 lines · Alexander Graf
Quoted 35 lines · Peter Maydell
Quoted 19 lines · Alexander Graf

Hvf's permission bitmap during and after dirty logging does not include the HV_MEMORY_EXEC permission. At least on Apple Silicon, this leads to instruction faults once dirty logging was enabled.

Add the bit to make it work properly.

---
 accel/hvf/hvf-accel-ops.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
accel/hvf/hvf-accel-ops.c +0 -0
index d1691be989..71cc2fa70f 100644
@@ -239,12 +239,12 @@ static void hvf_set_dirty_tracking(MemoryRegionSection *section, bool on)
     if (on) {
         slot->flags |= HVF_SLOT_LOG;
         hv_vm_protect((uintptr_t)slot->start, (size_t)slot->size,

accel/hvf/hvf-accel-ops.c static void hvf_set_dirty_tracking(MemoryRegionSection *section, bool on) · line 249

Quoted 10 lines
accel/hvf/hvf-accel-ops.c +2 -2
-                      HV_MEMORY_READ);
+                      HV_MEMORY_READ | HV_MEMORY_EXEC);
     /* stop tracking region*/
     } else {
         slot->flags &= ~HVF_SLOT_LOG;
         hv_vm_protect((uintptr_t)slot->start, (size_t)slot->size,
-                      HV_MEMORY_READ | HV_MEMORY_WRITE);
+                      HV_MEMORY_READ | HV_MEMORY_WRITE | HV_MEMORY_EXEC);
     }
 }

Makes sense -- this matches the premissions we set initially for memory regions in hvf_set_phys_mem().

Quoted 7 lines
Quoted 2 lines

Should we change also the hv_vm_protect() call in target/i386/hvf/hvf.c:ept_emulation_fault(), for consistency ?

The x86 hvf code seems to deal just fine with !X mappings, so I'd leave it as is as part of the arm enablement series - especially because I have limited testing capabilities for x86 hvf.

Yeah, I should have been clearer -- that would be best as a separate x86-specific patch.

-- PMM

#7 lore raw

PATCH v11 3/10 hvf: Introduce hvf_arch_init() callback

In reply to Alexander Graf

We will need to install a migration helper for the ARM hvf backend. Let's introduce an arch callback for the overall hvf init chain to do so.

---
 accel/hvf/hvf-accel-ops.c | 3 ++-
 include/sysemu/hvf_int.h  | 1 +
 target/i386/hvf/hvf.c     | 5 +++++
 3 files changed, 8 insertions(+), 1 deletion(-)
accel/hvf/hvf-accel-ops.c +2 -1
index 71cc2fa70f..65d431868f 100644
@@ -324,7 +324,8 @@ static int hvf_accel_init(MachineState *ms)
 
     hvf_state = s;
     memory_listener_register(&hvf_memory_listener, &address_space_memory);
-    return 0;
+
+    return hvf_arch_init();
 }
 
 static void hvf_accel_class_init(ObjectClass *oc, void *data)
include/sysemu/hvf_int.h +1 -0
index 8b66a4e7d0..0466106d16 100644
@@ -48,6 +48,7 @@ struct hvf_vcpu_state {
 };
 
 void assert_hvf_ok(hv_return_t ret);
+int hvf_arch_init(void);
 int hvf_arch_init_vcpu(CPUState *cpu);
 void hvf_arch_vcpu_destroy(CPUState *cpu);
 int hvf_vcpu_exec(CPUState *);
target/i386/hvf/hvf.c +5 -0
index 79ba4ed93a..abef24a9c8 100644
@@ -206,6 +206,11 @@ static inline bool apic_bus_freq_is_known(CPUX86State *env)
     return env->apic_bus_freq != 0;
 }
 
+int hvf_arch_init(void)
+{
+    return 0;
+}
+
 int hvf_arch_init_vcpu(CPUState *cpu)
 {
     X86CPU *x86cpu = X86_CPU(cpu);
Signature · 1 line
2.30.1 (Apple Git-130)
Reviewed-by on v11 3/10
#8 lore raw

Re: [PATCH v11 03/10] hvf: Introduce hvf_arch_init() callback

Quoted 17 lines · Alexander Graf

We will need to install a migration helper for the ARM hvf backend. Let's introduce an arch callback for the overall hvf init chain to do so.

---
 accel/hvf/hvf-accel-ops.c | 3 ++-
 include/sysemu/hvf_int.h  | 1 +
 target/i386/hvf/hvf.c     | 5 +++++
 3 files changed, 8 insertions(+), 1 deletion(-)
accel/hvf/hvf-accel-ops.c +0 -0
index 71cc2fa70f..65d431868f 100644

accel/hvf/hvf-accel-ops.c static int hvf_accel_init(MachineState *ms) · line 329

Quoted 7 lines
accel/hvf/hvf-accel-ops.c +2 -1
@@ -324,7 +324,8 @@ static int hvf_accel_init(MachineState *ms)
 
     hvf_state = s;
     memory_listener_register(&hvf_memory_listener, &address_space_memory);
-    return 0;
+
+    return hvf_arch_init();
 }

If the arch-init fails this will leave various things that hvf_accel_init() just set up not cleaned up. But kvm's init function doesn't seem to bother to do that cleanup either, and if we return failure vl.c is going to call exit(), so I guess it's OK.

thanks
-- PMM

#9 lore raw

PATCH v11 4/10 hvf: Add Apple Silicon support

In reply to Alexander Graf

With Apple Silicon available to the masses, it's a good time to add support for driving its virtualization extensions from QEMU.

This patch adds all necessary architecture specific code to get basic VMs working, including save/restore.

Known limitations:

- WFI handling is missing (follows in later patch)
- No watchpoint/breakpoint support

---

v1 -> v2:

- Merge vcpu kick function patch
- Implement WFI handling (allows vCPUs to sleep)
- Synchronize system registers (fixes OVMF crashes and reboot)
- Don't always call cpu_synchronize_state()
- Use more fine grained iothread locking
- Populate aa64mmfr0 from hardware

v2 -> v3:

- Advance PC on SMC
- Use cp list interface for sysreg syncs
- Do not set current_cpu
- Fix sysreg isread mask
- Move sysreg handling to functions
- Remove WFI logic again
- Revert to global iothread locking
- Use Hypervisor.h on arm, hv.h does not contain aarch64 definitions

v3 -> v4:

- No longer include Hypervisor.h

v5 -> v6:

- Swap sysreg definition order. This way we're in line with asm outputs.

v6 -> v7:

- Remove osdep.h include from hvf_int.h
- Synchronize SIMD registers as well
- Prepend 0x for hex values
- Convert DPRINTF to trace points
- Use main event loop (fixes gdbstub issues)
- Remove PSCI support, inject UDEF on HVC/SMC
- Change vtimer logic to look at ctl.istatus for vtimer mask sync
- Add kick callback again (fixes remote CPU notification)

v7 -> v8:

- Fix checkpatch errors

v8 -> v9:

- Make kick function non-weak
- Use arm_cpu_do_interrupt()
- Remove CNTPCT_EL0 write case
- Inject UDEF on invalid sysreg access
- Add support for OS locking sysregs
- Remove PMCCNTR_EL0 handling
- Print PC on unhandled sysreg trace
- Sync SP (x31) based on SP_EL0/SP_EL1
- Fix SPSR_EL1 mapping
- Only sync known sysregs, assert when syncing fails
- Improve error message on unhandled ec
- Move vtimer sync to post-exit (fixes disable corner case from kvm-unit-tests)
- Add vtimer offset, migration and pause logic
- Flush registers only after EXCP checkers (fixes PSCI on race)

v9 -> v10:

- Return -1 in x0 on unknown SMC/HVC calls

v10 -> v11:

- Treat SMC as UDEF. A follow-up patch set will try to change behavior consistently in TCG as well as HVF.

---
 MAINTAINERS                 |   5 +
 accel/hvf/hvf-accel-ops.c   |   9 +
 include/sysemu/hvf_int.h    |  10 +-
 meson.build                 |   1 +
 target/arm/hvf/hvf.c        | 794 ++++++++++++++++++++++++++++++++++++
 target/arm/hvf/trace-events |  10 +
 target/i386/hvf/hvf.c       |   5 +
 7 files changed, 833 insertions(+), 1 deletion(-)
 create mode 100644 target/arm/hvf/hvf.c
 create mode 100644 target/arm/hvf/trace-events
MAINTAINERS +5 -0
index 6c20634d63..d7915ec128 100644
@@ -433,6 +433,11 @@ F: accel/accel-*.c
 F: accel/Makefile.objs
 F: accel/stubs/Makefile.objs
 
+Apple Silicon HVF CPUs
+M: Alexander Graf <[email protected]>
+S: Maintained
+F: target/arm/hvf/
+
 X86 HVF CPUs
 M: Cameron Esfahani <[email protected]>
 M: Roman Bolshakov <[email protected]>
accel/hvf/hvf-accel-ops.c +9 -0
index 65d431868f..4f75927a8e 100644
@@ -60,6 +60,10 @@
 
 HVFState *hvf_state;
 
+#ifdef __aarch64__
+#define HV_VM_DEFAULT NULL
+#endif
+
 /* Memory slots */
 
 hvf_slot *hvf_find_overlap_slot(uint64_t start, uint64_t size)
@@ -376,7 +380,11 @@ static int hvf_init_vcpu(CPUState *cpu)
     pthread_sigmask(SIG_BLOCK, NULL, &set);
     sigdelset(&set, SIG_IPI);
 
+#ifdef __aarch64__
+    r = hv_vcpu_create(&cpu->hvf->fd, (hv_vcpu_exit_t **)&cpu->hvf->exit, NULL);
+#else
     r = hv_vcpu_create((hv_vcpuid_t *)&cpu->hvf->fd, HV_VCPU_DEFAULT);
+#endif
     cpu->vcpu_dirty = 1;
     assert_hvf_ok(r);
 
@@ -452,6 +460,7 @@ static void hvf_accel_ops_class_init(ObjectClass *oc, void *data)
     AccelOpsClass *ops = ACCEL_OPS_CLASS(oc);
 
     ops->create_vcpu_thread = hvf_start_vcpu_thread;
+    ops->kick_vcpu_thread = hvf_kick_vcpu_thread;
 
     ops->synchronize_post_reset = hvf_cpu_synchronize_post_reset;
     ops->synchronize_post_init = hvf_cpu_synchronize_post_init;
include/sysemu/hvf_int.h +9 -1
index 0466106d16..7c245c7b11 100644
@@ -11,7 +11,11 @@
 #ifndef HVF_INT_H
 #define HVF_INT_H
 
+#ifdef __aarch64__
+#include <Hypervisor/Hypervisor.h>
+#else
 #include <Hypervisor/hv.h>
+#endif
 
 /* hvf_slot flags */
 #define HVF_SLOT_LOG (1 << 0)
@@ -40,11 +44,14 @@ struct HVFState {
     int num_slots;
 
     hvf_vcpu_caps *hvf_caps;
+    uint64_t vtimer_offset;
 };
 extern HVFState *hvf_state;
 
 struct hvf_vcpu_state {
-    int fd;
+    uint64_t fd;
+    void *exit;
+    bool vtimer_masked;
 };
 
 void assert_hvf_ok(hv_return_t ret);
@@ -55,5 +62,6 @@ int hvf_vcpu_exec(CPUState *);
 hvf_slot *hvf_find_overlap_slot(uint64_t, uint64_t);
 int hvf_put_registers(CPUState *);
 int hvf_get_registers(CPUState *);
+void hvf_kick_vcpu_thread(CPUState *cpu);
 
 #endif
meson.build +1 -0
index 2711cbb789..baa28d7d62 100644
@@ -2169,6 +2169,7 @@ if have_system or have_user
     'accel/tcg',
     'hw/core',
     'target/arm',
+    'target/arm/hvf',
     'target/hppa',
     'target/i386',
     'target/i386/kvm',
target/arm/hvf/hvf.c new file +794 -0

794 lines not on this page · raw message

target/arm/hvf/trace-events new file +10 -0
new file mode 100644
index 0000000000..e972bdd9ce
@@ -0,0 +1,10 @@
+hvf_unhandled_sysreg_read(uint64_t pc, uint32_t reg, uint32_t op0, uint32_t op1, uint32_t crn, uint32_t crm, uint32_t op2) "unhandled sysreg read at pc=0x%"PRIx64": 0x%08x (op0=%d op1=%d crn=%d crm=%d op2=%d)"
+hvf_unhandled_sysreg_write(uint64_t pc, uint32_t reg, uint32_t op0, uint32_t op1, uint32_t crn, uint32_t crm, uint32_t op2) "unhandled sysreg write at pc=0x%"PRIx64": 0x%08x (op0=%d op1=%d crn=%d crm=%d op2=%d)"
+hvf_inject_fiq(void) "injecting FIQ"
+hvf_inject_irq(void) "injecting IRQ"
+hvf_data_abort(uint64_t pc, uint64_t va, uint64_t pa, bool isv, bool iswrite, bool s1ptw, uint32_t len, uint32_t srt) "data abort: [pc=0x%"PRIx64" va=0x%016"PRIx64" pa=0x%016"PRIx64" isv=%d iswrite=%d s1ptw=%d len=%d srt=%d]"
+hvf_sysreg_read(uint32_t reg, uint32_t op0, uint32_t op1, uint32_t crn, uint32_t crm, uint32_t op2, uint64_t val) "sysreg read 0x%08x (op0=%d op1=%d crn=%d crm=%d op2=%d) = 0x%016"PRIx64
+hvf_sysreg_write(uint32_t reg, uint32_t op0, uint32_t op1, uint32_t crn, uint32_t crm, uint32_t op2, uint64_t val) "sysreg write 0x%08x (op0=%d op1=%d crn=%d crm=%d op2=%d, val=0x%016"PRIx64")"
+hvf_unknown_hvf(uint64_t x0) "unknown HVC! 0x%016"PRIx64
+hvf_unknown_smc(uint64_t x0) "unknown SMC! 0x%016"PRIx64
+hvf_exit(uint64_t syndrome, uint32_t ec, uint64_t pc) "exit: 0x%"PRIx64" [ec=0x%x pc=0x%"PRIx64"]"
target/i386/hvf/hvf.c +5 -0
index abef24a9c8..33a4e74980 100644
@@ -206,6 +206,11 @@ static inline bool apic_bus_freq_is_known(CPUX86State *env)
     return env->apic_bus_freq != 0;
 }
 
+void hvf_kick_vcpu_thread(CPUState *cpu)
+{
+    cpus_kick_thread(cpu);
+}
+
 int hvf_arch_init(void)
 {
     return 0;
Signature · 1 line
2.30.1 (Apple Git-130)
Reviewed-by on v11 4/10
#10 lore raw

Re: [PATCH v11 04/10] hvf: Add Apple Silicon support

Quoted 9 lines · Alexander Graf

With Apple Silicon available to the masses, it's a good time to add support for driving its virtualization extensions from QEMU.

This patch adds all necessary architecture specific code to get basic VMs working, including save/restore.

Known limitations:

Quoted 8 lines

- WFI handling is missing (follows in later patch)
- No watchpoint/breakpoint support

---

thanks
-- PMM

#11 lore raw

PATCH v11 5/10 arm/hvf: Add a WFI handler

In reply to Alexander Graf

From: Peter Collingbourne <[email protected]>

Sleep on WFI until the VTIMER is due but allow ourselves to be woken up on IPI.

In this implementation IPI is blocked on the CPU thread at startup and pselect() is used to atomically unblock the signal and begin sleeping. The signal is sent unconditionally so there's no need to worry about races between actually sleeping and the "we think we're sleeping" state. It may lead to an extra wakeup but that's better than missing it entirely.

[agraf: Remove unused 'set' variable, always advance PC on WFX trap,
support vm stop / continue operations and cntv offsets]

---

v6 -> v7:

- Move WFI into function
- Improve comment wording

v8 -> v9:

- Add support for cntv offsets
- Improve code readability

---
 accel/hvf/hvf-accel-ops.c |  5 ++-
 include/sysemu/hvf_int.h  |  1 +
 target/arm/hvf/hvf.c      | 76 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 79 insertions(+), 3 deletions(-)
accel/hvf/hvf-accel-ops.c +2 -3
index 4f75927a8e..93976f4ece 100644
@@ -370,15 +370,14 @@ static int hvf_init_vcpu(CPUState *cpu)
     cpu->hvf = g_malloc0(sizeof(*cpu->hvf));
 
     /* init cpu signals */
-    sigset_t set;
     struct sigaction sigact;
 
     memset(&sigact, 0, sizeof(sigact));
     sigact.sa_handler = dummy_signal;
     sigaction(SIG_IPI, &sigact, NULL);
 
-    pthread_sigmask(SIG_BLOCK, NULL, &set);
-    sigdelset(&set, SIG_IPI);
+    pthread_sigmask(SIG_BLOCK, NULL, &cpu->hvf->unblock_ipi_mask);
+    sigdelset(&cpu->hvf->unblock_ipi_mask, SIG_IPI);
 
 #ifdef __aarch64__
     r = hv_vcpu_create(&cpu->hvf->fd, (hv_vcpu_exit_t **)&cpu->hvf->exit, NULL);
include/sysemu/hvf_int.h +1 -0
index 7c245c7b11..6545f7cd61 100644
@@ -52,6 +52,7 @@ struct hvf_vcpu_state {
     uint64_t fd;
     void *exit;
     bool vtimer_masked;
+    sigset_t unblock_ipi_mask;
 };
 
 void assert_hvf_ok(hv_return_t ret);
target/arm/hvf/hvf.c +76 -0

100 lines not on this page · raw message

Signature · 1 line
2.30.1 (Apple Git-130)
#12 lore raw

Re: [PATCH v11 05/10] arm/hvf: Add a WFI handler

Quoted 34 lines · Alexander Graf

From: Peter Collingbourne <[email protected]>

Sleep on WFI until the VTIMER is due but allow ourselves to be woken up on IPI.

In this implementation IPI is blocked on the CPU thread at startup and pselect() is used to atomically unblock the signal and begin sleeping. The signal is sent unconditionally so there's no need to worry about races between actually sleeping and the "we think we're sleeping" state. It may lead to an extra wakeup but that's better than missing it entirely.

[agraf: Remove unused 'set' variable, always advance PC on WFX trap,
support vm stop / continue operations and cntv offsets]

---

target/arm/hvf/hvf.c +1 -0
index 8fe008dab5..49f265cc08 100644
@@ -2,6 +2,7 @@
  * QEMU Hypervisor.framework support for Apple Silicon
 
  * Copyright 2020 Alexander Graf <[email protected]>
+ * Copyright 2020 Google LLC
  *
  * This work is licensed under the terms of the GNU GPL, version 2 or later.
  * See the COPYING file in the top-level directory.

target/arm/hvf/hvf.c int hvf_arch_init_vcpu(CPUState *cpu) · line 494

Quoted 4 lines
target/arm/hvf/hvf.c +1 -0
@@ -490,6 +491,7 @@ int hvf_arch_init_vcpu(CPUState *cpu)
 
 void hvf_kick_vcpu_thread(CPUState *cpu)
 {
+    cpus_kick_thread(cpu);

Doesn't this belong to the previous patch?

Quoted 27 lines
target/arm/hvf/hvf.c +24 -0
     hv_vcpus_exit(&cpu->hvf->fd, 1);
 }
 
+static void hvf_wfi(CPUState *cpu)
+{
+    ARMCPU *arm_cpu = ARM_CPU(cpu);
+    hv_return_t r;
+    uint64_t ctl;
+    uint64_t cval;
+    int64_t ticks_to_sleep;
+    uint64_t seconds;
+    uint64_t nanos;
+
+    if (cpu->interrupt_request & (CPU_INTERRUPT_HARD | CPU_INTERRUPT_FIQ)) {
+        /* Interrupt pending, no need to wait */
+        return;
+    }
+
+    r = hv_vcpu_get_sys_reg(cpu->hvf->fd, HV_SYS_REG_CNTV_CTL_EL0, &ctl);
+    assert_hvf_ok(r);
+
+    if (!(ctl & 1) || (ctl & 2)) {
+        /* Timer disabled or masked, just wait for an IPI. */
+        hvf_wait_for_ipi(cpu, NULL);
+        return;
+    }
+

target/arm/hvf/hvf.c static uint64_t hvf_vtimer_val_raw(void) · line 668

Quoted 10 lines
target/arm/hvf/hvf.c +10 -0
+    r = hv_vcpu_get_sys_reg(cpu->hvf->fd, HV_SYS_REG_CNTV_CVAL_EL0, &cval);
+    assert_hvf_ok(r);
+
+    ticks_to_sleep = cval - hvf_vtimer_val();
+    if (ticks_to_sleep < 0) {
+        return;
+    }
+
+    nanos = ticks_to_sleep * gt_cntfrq_period_ns(arm_cpu);
+    seconds = nanos / NANOSECONDS_PER_SECOND;

muldiv64()?

target/arm/hvf/hvf.c static uint64_t hvf_vtimer_val_raw(void) · line 680

Quoted 12 lines
target/arm/hvf/hvf.c +12 -0
+    nanos -= (seconds * NANOSECONDS_PER_SECOND);
+
+    /*
+     * Don't sleep for less than the time a context switch would take,
+     * so that we can satisfy fast timer requests on the same CPU.
+     * Measurements on M1 show the sweet spot to be ~2ms.
+     */
+    if (!seconds && nanos < (2 * SCALE_MS)) {
+        return;
+    }
+
+    struct timespec ts = { seconds, nanos };

QEMU style still declares variables at top of function/block.

Quoted 2 lines
target/arm/hvf/hvf.c +2 -0
+    hvf_wait_for_ipi(cpu, &ts);
+}
#13 lore raw

Re: [PATCH v11 05/10] arm/hvf: Add a WFI handler

Quoted 35 lines · Philippe Mathieu-Daudé
Quoted 34 lines · Alexander Graf

From: Peter Collingbourne <[email protected]>

Sleep on WFI until the VTIMER is due but allow ourselves to be woken up on IPI.

In this implementation IPI is blocked on the CPU thread at startup and pselect() is used to atomically unblock the signal and begin sleeping. The signal is sent unconditionally so there's no need to worry about races between actually sleeping and the "we think we're sleeping" state. It may lead to an extra wakeup but that's better than missing it entirely.

[agraf: Remove unused 'set' variable, always advance PC on WFX trap,
support vm stop / continue operations and cntv offsets]

---

target/arm/hvf/hvf.c +1 -0
index 8fe008dab5..49f265cc08 100644
@@ -2,6 +2,7 @@
  * QEMU Hypervisor.framework support for Apple Silicon
 
  * Copyright 2020 Alexander Graf <[email protected]>
+ * Copyright 2020 Google LLC
  *
  * This work is licensed under the terms of the GNU GPL, version 2 or later.
  * See the COPYING file in the top-level directory.
@@ -490,6 +491,7 @@ int hvf_arch_init_vcpu(CPUState *cpu)
 
Quoted 4 lines

target/arm/hvf/hvf.c int hvf_arch_init_vcpu(CPUState *cpu) · line 494

Quoted 3 lines
target/arm/hvf/hvf.c +1 -0
 void hvf_kick_vcpu_thread(CPUState *cpu)
 {
+    cpus_kick_thread(cpu);

Doesn't this belong to the previous patch?

Until this patch, we're never running outside guest context on the vCPU thread, so hv_vcpus_exit() is enough to kick us out :).

Thanks a lot for the review!

Alex

Reviewed-by on v11 5/10
#14 lore raw

Re: [PATCH v11 05/10] arm/hvf: Add a WFI handler

In reply to Alexander Graf
Quoted 13 lines · Alexander Graf

From: Peter Collingbourne <[email protected]>

Sleep on WFI until the VTIMER is due but allow ourselves to be woken up on IPI.

In this implementation IPI is blocked on the CPU thread at startup and pselect() is used to atomically unblock the signal and begin sleeping. The signal is sent unconditionally so there's no need to worry about races between actually sleeping and the "we think we're sleeping" state. It may lead to an extra wakeup but that's better than missing it entirely.

Quoted 8 lines

[agraf: Remove unused 'set' variable, always advance PC on WFX trap,
support vm stop / continue operations and cntv offsets]

---

Other than the points Philippe raises,

thanks
-- PMM

#15 lore raw

PATCH v11 6/10 hvf: arm: Implement -cpu host

In reply to Alexander Graf

Now that we have working system register sync, we push more target CPU properties into the virtual machine. That might be useful in some situations, but is not the typical case that users want.

So let's add a -cpu host option that allows them to explicitly pass all CPU capabilities of their host CPU into the guest.

---

v6 -> v7:

- Move function define to own header
- Do not propagate SVE features for HVF
- Remove stray whitespace change
- Verify that EL0 and EL1 do not allow AArch32 mode
- Only probe host CPU features once

v8 -> v9:

- Zero-initialize host_isar
- Use M1 SCTLR reset value

v9 -> v10:

- Fail -cpu host class creation gracefully
- Adjust error message on -cpu host realize failure
- Extend SCTLR comment that hvf returns 0 as default value
- Return true/false
- Report errors lazily

---
 target/arm/cpu.c     | 13 +++++--
 target/arm/cpu.h     |  2 +
 target/arm/hvf/hvf.c | 91 ++++++++++++++++++++++++++++++++++++++++++++
 target/arm/hvf_arm.h | 19 +++++++++
 target/arm/kvm_arm.h |  2 -
 5 files changed, 121 insertions(+), 6 deletions(-)
 create mode 100644 target/arm/hvf_arm.h
target/arm/cpu.c +9 -4
index ba0741b20e..016df2e1e7 100644
@@ -39,6 +39,7 @@
 #include "sysemu/tcg.h"
 #include "sysemu/hw_accel.h"
 #include "kvm_arm.h"
+#include "hvf_arm.h"
 #include "disas/capstone.h"
 #include "fpu/softfloat.h"
 
@@ -1400,8 +1401,8 @@ static void arm_cpu_realizefn(DeviceState *dev, Error **errp)
      * this is the first point where we can report it.
      */
     if (cpu->host_cpu_probe_failed) {
-        if (!kvm_enabled()) {
-            error_setg(errp, "The 'host' CPU type can only be used with KVM");
+        if (!kvm_enabled() && !hvf_enabled()) {
+            error_setg(errp, "The 'host' CPU type can only be used with KVM or HVF");
         } else {
             error_setg(errp, "Failed to retrieve host CPU features");
         }
@@ -2061,15 +2062,19 @@ static void arm_cpu_class_init(ObjectClass *oc, void *data)
 #endif /* CONFIG_TCG */
 }
 
-#ifdef CONFIG_KVM
+#if defined(CONFIG_KVM) || defined(CONFIG_HVF)
 static void arm_host_initfn(Object *obj)
 {
     ARMCPU *cpu = ARM_CPU(obj);
 
+#ifdef CONFIG_KVM
     kvm_arm_set_cpu_features_from_host(cpu);
     if (arm_feature(&cpu->env, ARM_FEATURE_AARCH64)) {
         aarch64_add_sve_properties(obj);
     }
+#else
+    hvf_arm_set_cpu_features_from_host(cpu);
+#endif
     arm_cpu_post_init(obj);
 }
 
@@ -2129,7 +2134,7 @@ static void arm_cpu_register_types(void)
 {
     type_register_static(&arm_cpu_type_info);
 
-#ifdef CONFIG_KVM
+#if defined(CONFIG_KVM) || defined(CONFIG_HVF)
     type_register_static(&host_arm_cpu_type_info);
 #endif
 }
target/arm/cpu.h +2 -0
index 09d9027734..3ed03391a8 100644
@@ -3015,6 +3015,8 @@ bool write_cpustate_to_list(ARMCPU *cpu, bool kvm_sync);
 #define ARM_CPU_TYPE_NAME(name) (name ARM_CPU_TYPE_SUFFIX)
 #define CPU_RESOLVING_TYPE TYPE_ARM_CPU
 
+#define TYPE_ARM_HOST_CPU "host-" TYPE_ARM_CPU
+
 #define cpu_signal_handler cpu_arm_signal_handler
 #define cpu_list arm_cpu_list
 
target/arm/hvf/hvf.c +91 -0

109 lines not on this page · raw message

target/arm/hvf_arm.h new file +19 -0
new file mode 100644
index 0000000000..603074a331
@@ -0,0 +1,19 @@
+/*
+ * QEMU Hypervisor.framework (HVF) support -- ARM specifics
+ *
+ * Copyright (c) 2021 Alexander Graf
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef QEMU_HVF_ARM_H
+#define QEMU_HVF_ARM_H
+
+#include "qemu/accel.h"
+#include "cpu.h"
+
+void hvf_arm_set_cpu_features_from_host(struct ARMCPU *cpu);
+
+#endif
target/arm/kvm_arm.h +0 -2
index 0613454975..b7f78b5215 100644
@@ -214,8 +214,6 @@ bool kvm_arm_create_scratch_host_vcpu(const uint32_t *cpus_to_try,
  */
 void kvm_arm_destroy_scratch_host_vcpu(int *fdarray);
 
-#define TYPE_ARM_HOST_CPU "host-" TYPE_ARM_CPU
-
 /**
  * ARMHostCPUFeatures: information about the host CPU (identified
  * by asking the host kernel)
Signature · 1 line
2.30.1 (Apple Git-130)
Reviewed-by on v11 6/10
#16 lore raw

Re: [PATCH v11 06/10] hvf: arm: Implement -cpu host

Quoted 13 lines · Alexander Graf

Now that we have working system register sync, we push more target CPU properties into the virtual machine. That might be useful in some situations, but is not the typical case that users want.

So let's add a -cpu host option that allows them to explicitly pass all CPU capabilities of their host CPU into the guest.

target/arm/hvf/hvf.c static uint64_t hvf_get_reg(CPUState *cpu, int rt) · line 479

Quoted 7 lines
target/arm/hvf/hvf.c +7 -0
+    /*
+     * A scratch vCPU returns SCTLR 0, so let's fill our default with the M1
+     * boot SCTLR from https://github.com/AsahiLinux/m1n1/issues/97
+     */
+    ahcf->reset_sctlr = 0x30100180;
+    /* OVMF chokes on boot if SPAN is not set, so default it to on */
+    ahcf->reset_sctlr |= 0x00800000;

Isn't that just an OVMF bug ? If you want this then you need to convince me why this isn't just a workaround for a buggy guest.

Otherwise

thanks
-- PMM

#17 lore raw

Re: [PATCH v11 06/10] hvf: arm: Implement -cpu host

Quoted 19 lines · Peter Maydell
Quoted 11 lines · Alexander Graf

Now that we have working system register sync, we push more target CPU properties into the virtual machine. That might be useful in some situations, but is not the typical case that users want.

So let's add a -cpu host option that allows them to explicitly pass all CPU capabilities of their host CPU into the guest.

target/arm/hvf/hvf.c static uint64_t hvf_get_reg(CPUState *cpu, int rt) · line 479

Quoted 7 lines
target/arm/hvf/hvf.c +7 -0
+    /*
+     * A scratch vCPU returns SCTLR 0, so let's fill our default with the M1
+     * boot SCTLR from https://github.com/AsahiLinux/m1n1/issues/97
+     */
+    ahcf->reset_sctlr = 0x30100180;
+    /* OVMF chokes on boot if SPAN is not set, so default it to on */
+    ahcf->reset_sctlr |= 0x00800000;
Quoted 2 lines

Isn't that just an OVMF bug ? If you want this then you need to convince me why this isn't just a workaround for a buggy guest.

I couldn't find anything in the ARMv8 spec that explicitly says "If you support PAN, SCTLR.SPAN should be 1 by default". It is RES1 for CPUs that do not implement PAN. Beware that for SPAN, "1" means disabled and "0" means enabled.

So by making it 0 we are effectively enabling PAN support on behalf of the firmware, which sort-of goes against the whole idea of being backwards compatible, no? I'd say that OVMF behaves just fine by assuming PAN support is not enabled unless it does it explicitly itself.

To make sure we get full agreement, I've added Ard to CC as well though :)

Alex

#18 lore raw

Re: [PATCH v11 06/10] hvf: arm: Implement -cpu host

Quoted 11 lines · Alexander Graf
Quoted 8 lines · Peter Maydell
Quoted 7 lines · Alexander Graf

Now that we have working system register sync, we push more target CPU properties into the virtual machine. That might be useful in some situations, but is not the typical case that users want.

So let's add a -cpu host option that allows them to explicitly pass all CPU capabilities of their host CPU into the guest.

Quoted 7 lines
Quoted 7 lines

target/arm/hvf/hvf.c static uint64_t hvf_get_reg(CPUState *cpu, int rt) · line 475

Quoted 7 lines
target/arm/hvf/hvf.c +3 -0
+    /*
+     * A scratch vCPU returns SCTLR 0, so let's fill our default with the M1
+     * boot SCTLR from https://github.com/AsahiLinux/m1n1/issues/97

Side note: SCTLR_EL1 is a 64-bit register, do you have anything that prints the full 64-bits to confirm that [63:32] are indeed all 0?

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
target/arm/hvf/hvf.c +4 -0
+     */
+    ahcf->reset_sctlr = 0x30100180;
+    /* OVMF chokes on boot if SPAN is not set, so default it to on */
+    ahcf->reset_sctlr |= 0x00800000;
Quoted 8 lines
Quoted 2 lines

Isn't that just an OVMF bug ? If you want this then you need to convince me why this isn't just a workaround for a buggy guest.

I couldn't find anything in the ARMv8 spec that explicitly says "If you support PAN, SCTLR.SPAN should be 1 by default". It is RES1 for CPUs that do not implement PAN. Beware that for SPAN, "1" means disabled and "0" means enabled.

It's UNKNOWN on reset. So unless OVMF is relying on whatever is launching it to set SCTLR correctly (ie there is some part of the "firmware-to-OVMF" contract it is relying on) then it seems to me that it's OVMF's job to initialize it to what it needs. (Lots of SCTLR is like that.)

Linux does this here:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/kernel/head.S?h=v5.15-rc1#n485
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/include/asm/sysreg.h?h=v5.15-rc1#n695
because the INIT_SCTLR_EL1_MMU_OFF constant includes forcing all "this kernel expects these to be RES0/RES1 because that's all the architectural features we know about at this time" bits to their RESn values.

But we can probably construct an argument for why having it set makes sense, yes.

-- PMM

#19 lore raw

Re: [PATCH v11 06/10] hvf: arm: Implement -cpu host

Quoted 51 lines · Peter Maydell
Quoted 11 lines · Alexander Graf
Quoted 8 lines · Peter Maydell
Quoted 7 lines · Alexander Graf

Now that we have working system register sync, we push more target CPU properties into the virtual machine. That might be useful in some situations, but is not the typical case that users want.

So let's add a -cpu host option that allows them to explicitly pass all CPU capabilities of their host CPU into the guest.

Quoted 7 lines
Quoted 7 lines

target/arm/hvf/hvf.c static uint64_t hvf_get_reg(CPUState *cpu, int rt) · line 475

Quoted 7 lines
target/arm/hvf/hvf.c +3 -0
+    /*
+     * A scratch vCPU returns SCTLR 0, so let's fill our default with the M1
+     * boot SCTLR from https://github.com/AsahiLinux/m1n1/issues/97

Side note: SCTLR_EL1 is a 64-bit register, do you have anything that prints the full 64-bits to confirm that [63:32] are indeed all 0?

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
target/arm/hvf/hvf.c +4 -0
+     */
+    ahcf->reset_sctlr = 0x30100180;
+    /* OVMF chokes on boot if SPAN is not set, so default it to on */
+    ahcf->reset_sctlr |= 0x00800000;
Quoted 8 lines
Quoted 2 lines

Isn't that just an OVMF bug ? If you want this then you need to convince me why this isn't just a workaround for a buggy guest.

I couldn't find anything in the ARMv8 spec that explicitly says "If you support PAN, SCTLR.SPAN should be 1 by default". It is RES1 for CPUs that do not implement PAN. Beware that for SPAN, "1" means disabled and "0" means enabled.

It's UNKNOWN on reset. So unless OVMF is relying on whatever is launching it to set SCTLR correctly (ie there is some part of the "firmware-to-OVMF" contract it is relying on) then it seems to me that it's OVMF's job to initialize it to what it needs. (Lots of SCTLR is like that.)

Linux does this here:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/kernel/head.S?h=v5.15-rc1#n485
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/include/asm/sysreg.h?h=v5.15-rc1#n695
because the INIT_SCTLR_EL1_MMU_OFF constant includes forcing all "this kernel expects these to be RES0/RES1 because that's all the architectural features we know about at this time" bits to their RESn values.

Quoted 3 lines

But we can probably construct an argument for why having it set makes sense, yes.

I'd argue that compliance with the architecture means that the software should not clear RES1 bits, but I don't think we can blame it for not touching bits that were in in invalid state upon entry.

#20 lore raw

Re: [PATCH v11 06/10] hvf: arm: Implement -cpu host

Quoted 2 lines · Ard Biesheuvel

I'd argue that compliance with the architecture means that the software should not clear RES1 bits

Architecturally, RES1 means that "software
* Must not rely on the bit reading as 1.
* Must use an SBOP policy to write to the bit."
(SBOP=="should be 1 or preserved", ie you can preserve the existing value, as in "read register, change some bits, write back", or you can write a 1.)

Quoted 2 lines

but I don't think we can blame it
for not touching bits that were in in invalid state upon entry.

SCTLR_EL1.SPAN == 0 is perfectly valid for a CPU that supports the PAN feature. It's just not the value OVMF wants, so OVMF should be setting it to what it does want. Also, as the first thing to run after reset (ie firmware) OVMF absolutely is responsible for dealing with system registers which have UNKNOWN values out of reset.

Put another way, if you want to argue that this is an "invalid state" you need to point to the specification that defines the valid state that OVMF should see when it is handed control.

-- PMM

#21 lore raw

Re: [PATCH v11 06/10] hvf: arm: Implement -cpu host

Quoted 11 lines · Peter Maydell
Quoted 2 lines · Ard Biesheuvel

I'd argue that compliance with the architecture means that the software should not clear RES1 bits

Architecturally, RES1 means that "software
* Must not rely on the bit reading as 1.
* Must use an SBOP policy to write to the bit."
(SBOP=="should be 1 or preserved", ie you can preserve the existing value, as in "read register, change some bits, write back", or you can write a 1.)

OVMF preserves the bit, and does not reason or care about its value. So in this sense, it is compliant.

Quoted 10 lines
Quoted 2 lines

but I don't think we can blame it
for not touching bits that were in in invalid state upon entry.

SCTLR_EL1.SPAN == 0 is perfectly valid for a CPU that supports the PAN feature. It's just not the value OVMF wants, so OVMF should be setting it to what it does want. Also, as the first thing to run after reset (ie firmware) OVMF absolutely is responsible for dealing with system registers which have UNKNOWN values out of reset.

Fair enough. But I'd still suggest fixing this at both ends.

Quoted 5 lines

Put another way, if you want to argue that this is an "invalid state" you need to point to the specification that defines the valid state that OVMF should see when it is handed control.

-- PMM

#22 lore raw

Re: [PATCH v11 06/10] hvf: arm: Implement -cpu host

Quoted 14 lines · Ard Biesheuvel
Quoted 11 lines · Peter Maydell
Quoted 2 lines · Ard Biesheuvel

I'd argue that compliance with the architecture means that the software should not clear RES1 bits

Architecturally, RES1 means that "software
* Must not rely on the bit reading as 1.
* Must use an SBOP policy to write to the bit."
(SBOP=="should be 1 or preserved", ie you can preserve the existing value, as in "read register, change some bits, write back", or you can write a 1.)

Quoted 2 lines

OVMF preserves the bit, and does not reason or care about its value. So in this sense, it is compliant.

Hmm. Alex, can you give more details about what fails here ?

Quoted 11 lines
Quoted 10 lines
Quoted 2 lines

but I don't think we can blame it
for not touching bits that were in in invalid state upon entry.

SCTLR_EL1.SPAN == 0 is perfectly valid for a CPU that supports the PAN feature. It's just not the value OVMF wants, so OVMF should be setting it to what it does want. Also, as the first thing to run after reset (ie firmware) OVMF absolutely is responsible for dealing with system registers which have UNKNOWN values out of reset.

Quoted 1 line

Fair enough. But I'd still suggest fixing this at both ends.

Yes, the version of this code that we committed sets SPAN to 1. (This argument is mostly about what the comment justifying that value should say :-))

-- PMM

#23 lore raw

Re: [PATCH v11 06/10] hvf: arm: Implement -cpu host

Quoted 16 lines · Peter Maydell
Quoted 14 lines · Ard Biesheuvel
Quoted 11 lines · Peter Maydell
Quoted 2 lines · Ard Biesheuvel

I'd argue that compliance with the architecture means that the software should not clear RES1 bits

Architecturally, RES1 means that "software
* Must not rely on the bit reading as 1.
* Must use an SBOP policy to write to the bit."
(SBOP=="should be 1 or preserved", ie you can preserve the existing value, as in "read register, change some bits, write back", or you can write a 1.)

Quoted 5 lines
Quoted 2 lines

OVMF preserves the bit, and does not reason or care about its value. So in this sense, it is compliant.

Hmm. Alex, can you give more details about what fails here ?

It seems that EDK2 ends up setting EL0 r/o or r/w permissions on some mappings, even though it never runs anything at EL0. So any execution that gets initiated via the timer interrupt causing a EL1->EL1 IRQ exception will run with PAN enabled and lose access to those mappings.

So it seems like a definite bug that is unrelated to reset state of the registers and assumptions related to that.

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 2 lines

but I don't think we can blame it
for not touching bits that were in in invalid state upon entry.

SCTLR_EL1.SPAN == 0 is perfectly valid for a CPU that supports the PAN feature. It's just not the value OVMF wants, so OVMF should be setting it to what it does want. Also, as the first thing to run after reset (ie firmware) OVMF absolutely is responsible for dealing with system registers which have UNKNOWN values out of reset.

Quoted 6 lines
Quoted 1 line

Fair enough. But I'd still suggest fixing this at both ends.

Yes, the version of this code that we committed sets SPAN to 1. (This argument is mostly about what the comment justifying that value should say :-))

OK, that makes sense. But I'd like to get EDK2 fixed as well, obviously.

#24 lore raw

Re: [PATCH v11 06/10] hvf: arm: Implement -cpu host

In reply to Peter Maydell
Quoted 15 lines · Peter Maydell
Quoted 14 lines · Alexander Graf
Quoted 12 lines · Peter Maydell
Quoted 11 lines · Alexander Graf

Now that we have working system register sync, we push more target CPU properties into the virtual machine. That might be useful in some situations, but is not the typical case that users want.

So let's add a -cpu host option that allows them to explicitly pass all CPU capabilities of their host CPU into the guest.

Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

target/arm/hvf/hvf.c static uint64_t hvf_get_reg(CPUState *cpu, int rt) · line 475

Quoted 3 lines
target/arm/hvf/hvf.c +3 -0
+    /*
+     * A scratch vCPU returns SCTLR 0, so let's fill our default with the M1
+     * boot SCTLR from https://github.com/AsahiLinux/m1n1/issues/97

Side note: SCTLR_EL1 is a 64-bit register, do you have anything that prints the full 64-bits to confirm that [63:32] are indeed all 0?

Yes, m1n1 prints the full 64bit value:
https://github.com/AsahiLinux/m1n1/blob/main/src/memory.c#L459

That said, I'm not sure we really have to model the guest's reset SCTLR in EL1 to be identical to the host's reset SCTLR in EL2. I think it's a great start, but as long as there is no spec that indicates what SCTLR should be in EL1, we can make our own rules IMHO.

Alex

#25 lore raw

PATCH v11 7/10 hvf: arm: Implement PSCI handling

In reply to Alexander Graf

We need to handle PSCI calls. Most of the TCG code works for us, but we can simplify it to only handle aa64 mode and we need to handle SUSPEND differently.

This patch takes the TCG code as template and duplicates it in HVF.

To tell the guest that we support PSCI 0.2 now, update the check in arm_cpu_initfn() as well.

---

v6 -> v7:

- This patch integrates "arm: Set PSCI to 0.2 for HVF"

v7 -> v8:

- Do not advance for HVC, PC is already updated by hvf
- Fix checkpatch error

v8 -> v9:

- Use new hvf_raise_exception() prototype
- Make cpu_off function void
- Add comment about return value, use -1 for "not found"
- Remove cpu_synchronize_state() when halted

v9 -> v10:

- Only handle PSCI calls for the current conduit
- Return true/false

v10 -> v11:

- Inject UDEF on SMC. Will be changed in TCG+HVF in a later patch set.

---
 target/arm/cpu.c            |   4 +-
 target/arm/hvf/hvf.c        | 138 ++++++++++++++++++++++++++++++++++--
 target/arm/hvf/trace-events |   1 +
 3 files changed, 136 insertions(+), 7 deletions(-)
target/arm/cpu.c +2 -2
index 016df2e1e7..e44c301527 100644
@@ -1096,8 +1096,8 @@ static void arm_cpu_initfn(Object *obj)
     cpu->psci_version = 1; /* By default assume PSCI v0.1 */
     cpu->kvm_target = QEMU_KVM_ARM_TARGET_NONE;
 
-    if (tcg_enabled()) {
-        cpu->psci_version = 2; /* TCG implements PSCI 0.2 */
+    if (tcg_enabled() || hvf_enabled()) {
+        cpu->psci_version = 2; /* TCG and HVF implement PSCI 0.2 */
     }
 }
 
target/arm/hvf/hvf.c +133 -5

165 lines not on this page · raw message

target/arm/hvf/trace-events +1 -0
index e972bdd9ce..cf4fb68f79 100644
@@ -8,3 +8,4 @@ hvf_sysreg_write(uint32_t reg, uint32_t op0, uint32_t op1, uint32_t crn, uint32_
 hvf_unknown_hvf(uint64_t x0) "unknown HVC! 0x%016"PRIx64
 hvf_unknown_smc(uint64_t x0) "unknown SMC! 0x%016"PRIx64
 hvf_exit(uint64_t syndrome, uint32_t ec, uint64_t pc) "exit: 0x%"PRIx64" [ec=0x%x pc=0x%"PRIx64"]"
+hvf_psci_call(uint64_t x0, uint64_t x1, uint64_t x2, uint64_t x3, uint32_t cpuid) "PSCI Call x0=0x%016"PRIx64" x1=0x%016"PRIx64" x2=0x%016"PRIx64" x3=0x%016"PRIx64" cpu=0x%x"
Signature · 1 line
2.30.1 (Apple Git-130)
Reviewed-by on v11 7/10
#26 lore raw

Re: [PATCH v11 07/10] hvf: arm: Implement PSCI handling

Quoted 13 lines · Alexander Graf

We need to handle PSCI calls. Most of the TCG code works for us, but we can simplify it to only handle aa64 mode and we need to handle SUSPEND differently.

This patch takes the TCG code as template and duplicates it in HVF.

To tell the guest that we support PSCI 0.2 now, update the check in arm_cpu_initfn() as well.

target/arm/hvf/hvf.c static void hvf_raise_exception(CPUState *cpu, uint32_t excp, · line 671

Quoted 7 lines
target/arm/hvf/hvf.c +7 -0
+    case QEMU_PSCI_0_2_FN_SYSTEM_RESET:
+        qemu_system_reset_request(SHUTDOWN_CAUSE_GUEST_RESET);
+        /* QEMU reset and shutdown are async requests, but PSCI
+         * mandates that we never return from the reset/shutdown
+         * call, so power the CPU off now so it doesn't execute
+         * anything further.
+         */

QEMU coding standard wants the opening "/*" of a block comment on its own line.

Quoted 21 lines
target/arm/hvf/hvf.c +9 -5
@@ -898,14 +1011,29 @@ int hvf_vcpu_exec(CPUState *cpu)
         break;
     case EC_AA64_HVC:
         cpu_synchronize_state(cpu);
-        trace_hvf_unknown_hvf(env->xregs[0]);
-        /* SMCCC 1.3 section 5.2 says every unknown SMCCC call returns -1 */
-        env->xregs[0] = -1;
+        if ((arm_cpu->psci_conduit != QEMU_PSCI_CONDUIT_HVC) ||
+            !hvf_handle_psci_call(cpu)) {
+            trace_hvf_unknown_hvf(env->xregs[0]);
+            /* SMCCC 1.3 section 5.2 says every unknown SMCCC call returns -1 */
+            env->xregs[0] = -1;
+        }
         break;
     case EC_AA64_SMC:
         cpu_synchronize_state(cpu);
-        trace_hvf_unknown_smc(env->xregs[0]);
-        hvf_raise_exception(cpu, EXCP_UDEF, syn_uncategorized());
+
+        if (arm_cpu->psci_conduit == QEMU_PSCI_CONDUIT_SMC) {
+            advance_pc = true;

target/arm/hvf/hvf.c int hvf_vcpu_exec(CPUState *cpu) · line 1035

Quoted 10 lines
target/arm/hvf/hvf.c +10 -0
+
+            if (!hvf_handle_psci_call(cpu)) {
+                trace_hvf_unknown_smc(env->xregs[0]);
+                /* SMCCC 1.3 section 5.2 says every unknown SMCCC call returns -1 */
+                env->xregs[0] = -1;
+            }
+        } else {
+            trace_hvf_unknown_smc(env->xregs[0]);
+            hvf_raise_exception(cpu, EXCP_UDEF, syn_uncategorized());
+        }

We seem to have ended up with an odd asymmetry in the way the code for HVC and SMC has been structured.

Otherwise

thanks
-- PMM

#27 lore raw

PATCH v11 8/10 arm: Add Hypervisor.framework build target

In reply to Alexander Graf

Now that we have all logic in place that we need to handle Hypervisor.framework on Apple Silicon systems, let's add CONFIG_HVF for aarch64 as well so that we can build it.

---

v1 -> v2:

- Fix build on 32bit arm

v3 -> v4:

- Remove i386-softmmu target

v6 -> v7:

- Simplify HVF matching logic in meson build file

---
 meson.build                | 7 +++++++
 target/arm/hvf/meson.build | 3 +++
 target/arm/meson.build     | 2 ++
 3 files changed, 12 insertions(+)
 create mode 100644 target/arm/hvf/meson.build
meson.build +7 -0
index baa28d7d62..15ef4d3c41 100644
@@ -77,6 +77,13 @@ else
 endif
 
 accelerator_targets = { 'CONFIG_KVM': kvm_targets }
+
+if cpu in ['aarch64']
+  accelerator_targets += {
+    'CONFIG_HVF': ['aarch64-softmmu']
+  }
+endif
+
 if cpu in ['x86', 'x86_64', 'arm', 'aarch64']
   # i386 emulator provides xenpv machine type for multiple architectures
   accelerator_targets += {
target/arm/hvf/meson.build new file +3 -0
new file mode 100644
index 0000000000..855e6cce5a
@@ -0,0 +1,3 @@
+arm_softmmu_ss.add(when: [hvf, 'CONFIG_HVF'], if_true: files(
+  'hvf.c',
+))
target/arm/meson.build +2 -0
index 25a02bf276..50f152214a 100644
@@ -60,5 +60,7 @@ arm_softmmu_ss.add(files(
   'psci.c',
 ))
 
+subdir('hvf')
+
 target_arch += {'arm': arm_ss}
 target_softmmu_arch += {'arm': arm_softmmu_ss}
Signature · 1 line
2.30.1 (Apple Git-130)
#28 lore raw

PATCH v11 9/10 hvf: arm: Add rudimentary PMC support

In reply to Alexander Graf

We can expose cycle counters on the PMU easily. To be as compatible as possible, let's do so, but make sure we don't expose any other architectural counters that we can not model yet.

This allows OSs to work that require PMU support.

---
 target/arm/hvf/hvf.c | 179 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 179 insertions(+)
target/arm/hvf/hvf.c +179 -0
index 829ff3ff3b..0a4b03c11b 100644
@@ -42,6 +42,18 @@
 #define SYSREG_OSLSR_EL1      SYSREG(2, 0, 1, 1, 4)
 #define SYSREG_OSDLR_EL1      SYSREG(2, 0, 1, 3, 4)
 #define SYSREG_CNTPCT_EL0     SYSREG(3, 3, 14, 0, 1)
+#define SYSREG_PMCR_EL0       SYSREG(3, 3, 9, 12, 0)
+#define SYSREG_PMUSERENR_EL0  SYSREG(3, 3, 9, 14, 0)
+#define SYSREG_PMCNTENSET_EL0 SYSREG(3, 3, 9, 12, 1)
+#define SYSREG_PMCNTENCLR_EL0 SYSREG(3, 3, 9, 12, 2)
+#define SYSREG_PMINTENCLR_EL1 SYSREG(3, 0, 9, 14, 2)
+#define SYSREG_PMOVSCLR_EL0   SYSREG(3, 3, 9, 12, 3)
+#define SYSREG_PMSWINC_EL0    SYSREG(3, 3, 9, 12, 4)
+#define SYSREG_PMSELR_EL0     SYSREG(3, 3, 9, 12, 5)
+#define SYSREG_PMCEID0_EL0    SYSREG(3, 3, 9, 12, 6)
+#define SYSREG_PMCEID1_EL0    SYSREG(3, 3, 9, 12, 7)
+#define SYSREG_PMCCNTR_EL0    SYSREG(3, 3, 9, 13, 0)
+#define SYSREG_PMCCFILTR_EL0  SYSREG(3, 3, 14, 15, 7)
 
 #define WFX_IS_WFE (1 << 0)
 
@@ -723,6 +735,40 @@ static int hvf_sysreg_read(CPUState *cpu, uint32_t reg, uint32_t rt)
         val = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL) /
               gt_cntfrq_period_ns(arm_cpu);
         break;
+    case SYSREG_PMCR_EL0:
+        val = env->cp15.c9_pmcr;
+        break;
+    case SYSREG_PMCCNTR_EL0:
+        pmu_op_start(env);
+        val = env->cp15.c15_ccnt;
+        pmu_op_finish(env);
+        break;
+    case SYSREG_PMCNTENCLR_EL0:
+        val = env->cp15.c9_pmcnten;
+        break;
+    case SYSREG_PMOVSCLR_EL0:
+        val = env->cp15.c9_pmovsr;
+        break;
+    case SYSREG_PMSELR_EL0:
+        val = env->cp15.c9_pmselr;
+        break;
+    case SYSREG_PMINTENCLR_EL1:
+        val = env->cp15.c9_pminten;
+        break;
+    case SYSREG_PMCCFILTR_EL0:
+        val = env->cp15.pmccfiltr_el0;
+        break;
+    case SYSREG_PMCNTENSET_EL0:
+        val = env->cp15.c9_pmcnten;
+        break;
+    case SYSREG_PMUSERENR_EL0:
+        val = env->cp15.c9_pmuserenr;
+        break;
+    case SYSREG_PMCEID0_EL0:
+    case SYSREG_PMCEID1_EL0:
+        /* We can't really count anything yet, declare all events invalid */
+        val = 0;
+        break;
     case SYSREG_OSLSR_EL1:
         val = env->cp15.oslsr_el1;
         break;
@@ -753,6 +799,82 @@ static int hvf_sysreg_read(CPUState *cpu, uint32_t reg, uint32_t rt)
     return 0;
 }
 
+static void pmu_update_irq(CPUARMState *env)
+{
+    ARMCPU *cpu = env_archcpu(env);
+    qemu_set_irq(cpu->pmu_interrupt, (env->cp15.c9_pmcr & PMCRE) &&
+            (env->cp15.c9_pminten & env->cp15.c9_pmovsr));
+}
+
+static bool pmu_event_supported(uint16_t number)
+{
+    return false;
+}
+
+/* Returns true if the counter (pass 31 for PMCCNTR) should count events using
+ * the current EL, security state, and register configuration.
+ */
+static bool pmu_counter_enabled(CPUARMState *env, uint8_t counter)
+{
+    uint64_t filter;
+    bool enabled, filtered = true;
+    int el = arm_current_el(env);
+
+    enabled = (env->cp15.c9_pmcr & PMCRE) &&
+              (env->cp15.c9_pmcnten & (1 << counter));
+
+    if (counter == 31) {
+        filter = env->cp15.pmccfiltr_el0;
+    } else {
+        filter = env->cp15.c14_pmevtyper[counter];
+    }
+
+    if (el == 0) {
+        filtered = filter & PMXEVTYPER_U;
+    } else if (el == 1) {
+        filtered = filter & PMXEVTYPER_P;
+    }
+
+    if (counter != 31) {
+        /*
+         * If not checking PMCCNTR, ensure the counter is setup to an event we
+         * support
+         */
+        uint16_t event = filter & PMXEVTYPER_EVTCOUNT;
+        if (!pmu_event_supported(event)) {
+            return false;
+        }
+    }
+
+    return enabled && !filtered;
+}
+
+static void pmswinc_write(CPUARMState *env, uint64_t value)
+{
+    unsigned int i;
+    for (i = 0; i < pmu_num_counters(env); i++) {
+        /* Increment a counter's count iff: */
+        if ((value & (1 << i)) && /* counter's bit is set */
+                /* counter is enabled and not filtered */
+                pmu_counter_enabled(env, i) &&
+                /* counter is SW_INCR */
+                (env->cp15.c14_pmevtyper[i] & PMXEVTYPER_EVTCOUNT) == 0x0) {
+            /*
+             * Detect if this write causes an overflow since we can't predict
+             * PMSWINC overflows like we can for other events
+             */
+            uint32_t new_pmswinc = env->cp15.c14_pmevcntr[i] + 1;
+
+            if (env->cp15.c14_pmevcntr[i] & ~new_pmswinc & INT32_MIN) {
+                env->cp15.c9_pmovsr |= (1 << i);
+                pmu_update_irq(env);
+            }
+
+            env->cp15.c14_pmevcntr[i] = new_pmswinc;
+        }
+    }
+}
+
 static int hvf_sysreg_write(CPUState *cpu, uint32_t reg, uint64_t val)
 {
     ARMCPU *arm_cpu = ARM_CPU(cpu);
@@ -767,6 +889,63 @@ static int hvf_sysreg_write(CPUState *cpu, uint32_t reg, uint64_t val)
                            val);
 
     switch (reg) {
+    case SYSREG_PMCCNTR_EL0:
+        pmu_op_start(env);
+        env->cp15.c15_ccnt = val;
+        pmu_op_finish(env);
+        break;
+    case SYSREG_PMCR_EL0:
+        pmu_op_start(env);
+
+        if (val & PMCRC) {
+            /* The counter has been reset */
+            env->cp15.c15_ccnt = 0;
+        }
+
+        if (val & PMCRP) {
+            unsigned int i;
+            for (i = 0; i < pmu_num_counters(env); i++) {
+                env->cp15.c14_pmevcntr[i] = 0;
+            }
+        }
+
+        env->cp15.c9_pmcr &= ~PMCR_WRITEABLE_MASK;
+        env->cp15.c9_pmcr |= (val & PMCR_WRITEABLE_MASK);
+
+        pmu_op_finish(env);
+        break;
+    case SYSREG_PMUSERENR_EL0:
+        env->cp15.c9_pmuserenr = val & 0xf;
+        break;
+    case SYSREG_PMCNTENSET_EL0:
+        env->cp15.c9_pmcnten |= (val & pmu_counter_mask(env));
+        break;
+    case SYSREG_PMCNTENCLR_EL0:
+        env->cp15.c9_pmcnten &= ~(val & pmu_counter_mask(env));
+        break;
+    case SYSREG_PMINTENCLR_EL1:
+        pmu_op_start(env);
+        env->cp15.c9_pminten |= val;
+        pmu_op_finish(env);
+        break;
+    case SYSREG_PMOVSCLR_EL0:
+        pmu_op_start(env);
+        env->cp15.c9_pmovsr &= ~val;
+        pmu_op_finish(env);
+        break;
+    case SYSREG_PMSWINC_EL0:
+        pmu_op_start(env);
+        pmswinc_write(env, val);
+        pmu_op_finish(env);
+        break;
+    case SYSREG_PMSELR_EL0:
+        env->cp15.c9_pmselr = val & 0x1f;
+        break;
+    case SYSREG_PMCCFILTR_EL0:
+        pmu_op_start(env);
+        env->cp15.pmccfiltr_el0 = val & PMCCFILTR_EL0;
+        pmu_op_finish(env);
+        break;
     case SYSREG_OSLAR_EL1:
         env->cp15.oslsr_el1 = val & 1;
         break;
Signature · 1 line
2.30.1 (Apple Git-130)
Reviewed-by on v11 9/10
#29 lore raw

Re: [PATCH v11 09/10] hvf: arm: Add rudimentary PMC support

Quoted 11 lines · Alexander Graf

We can expose cycle counters on the PMU easily. To be as compatible as possible, let's do so, but make sure we don't expose any other architectural counters that we can not model yet.

This allows OSs to work that require PMU support.

---
 target/arm/hvf/hvf.c | 179 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 179 insertions(+)

thanks
-- PMM

#30 lore raw

PATCH v11 10/10 arm: tcg: Adhere to SMCCC 1.3 section 5.2

In reply to Alexander Graf

The SMCCC 1.3 spec section 5.2 says

The Unknown SMC Function Identifier is a sign-extended value of (-1) that is returned in the R0, W0 or X0 registers. An implementation must return this error code when it receives:

* An SMC or HVC call with an unknown Function Identifier
* An SMC or HVC call for a removed Function Identifier
* An SMC64/HVC64 call from AArch32 state

To comply with these statements, let's always return -1 when we encounter an unknown HVC or SMC call.

---

v8 -> v9:

- Remove Windows specifics and just comply with SMCCC spec

v9 -> v10:

- Fix comment

---
 target/arm/psci.c | 35 ++++++-----------------------------
 1 file changed, 6 insertions(+), 29 deletions(-)
target/arm/psci.c +6 -29
index 6709e28013..b279c0b9a4 100644
@@ -27,15 +27,13 @@
 
 bool arm_is_psci_call(ARMCPU *cpu, int excp_type)
 {
-    /* Return true if the r0/x0 value indicates a PSCI call and
-     * the exception type matches the configured PSCI conduit. This is
-     * called before the SMC/HVC instruction is executed, to decide whether
-     * we should treat it as a PSCI call or with the architecturally
+    /*
+     * Return true if the exception type matches the configured PSCI conduit.
+     * This is called before the SMC/HVC instruction is executed, to decide
+     * whether we should treat it as a PSCI call or with the architecturally
      * defined behaviour for an SMC or HVC (which might be UNDEF or trap
      * to EL2 or to EL3).
      */
-    CPUARMState *env = &cpu->env;
-    uint64_t param = is_a64(env) ? env->xregs[0] : env->regs[0];
 
     switch (excp_type) {
     case EXCP_HVC:
@@ -52,27 +50,7 @@ bool arm_is_psci_call(ARMCPU *cpu, int excp_type)
         return false;
     }
 
-    switch (param) {
-    case QEMU_PSCI_0_2_FN_PSCI_VERSION:
-    case QEMU_PSCI_0_2_FN_MIGRATE_INFO_TYPE:
-    case QEMU_PSCI_0_2_FN_AFFINITY_INFO:
-    case QEMU_PSCI_0_2_FN64_AFFINITY_INFO:
-    case QEMU_PSCI_0_2_FN_SYSTEM_RESET:
-    case QEMU_PSCI_0_2_FN_SYSTEM_OFF:
-    case QEMU_PSCI_0_1_FN_CPU_ON:
-    case QEMU_PSCI_0_2_FN_CPU_ON:
-    case QEMU_PSCI_0_2_FN64_CPU_ON:
-    case QEMU_PSCI_0_1_FN_CPU_OFF:
-    case QEMU_PSCI_0_2_FN_CPU_OFF:
-    case QEMU_PSCI_0_1_FN_CPU_SUSPEND:
-    case QEMU_PSCI_0_2_FN_CPU_SUSPEND:
-    case QEMU_PSCI_0_2_FN64_CPU_SUSPEND:
-    case QEMU_PSCI_0_1_FN_MIGRATE:
-    case QEMU_PSCI_0_2_FN_MIGRATE:
-        return true;
-    default:
-        return false;
-    }
+    return true;
 }
 
 void arm_handle_psci_call(ARMCPU *cpu)
@@ -194,10 +172,9 @@ void arm_handle_psci_call(ARMCPU *cpu)
         break;
     case QEMU_PSCI_0_1_FN_MIGRATE:
     case QEMU_PSCI_0_2_FN_MIGRATE:
+    default:
         ret = QEMU_PSCI_RET_NOT_SUPPORTED;
         break;
-    default:
-        g_assert_not_reached();
     }
 
 err:
Signature · 1 line
2.30.1 (Apple Git-130)
Reviewed-by on v11 10/10
#31 lore raw

Re: [PATCH v11 10/10] arm: tcg: Adhere to SMCCC 1.3 section 5.2

Quoted 7 lines · Alexander Graf

The SMCCC 1.3 spec section 5.2 says

The Unknown SMC Function Identifier is a sign-extended value of (-1) that is returned in the R0, W0 or X0 registers. An implementation must return this error code when it receives:

Quoted 8 lines

* An SMC or HVC call with an unknown Function Identifier
* An SMC or HVC call for a removed Function Identifier
* An SMC64/HVC64 call from AArch32 state

To comply with these statements, let's always return -1 when we encounter an unknown HVC or SMC call.

thanks
-- PMM