Skip to contentSkip to conversation

HSM driver for ACRN hypervisor

PATCH Superseded [email protected] 3 participants 20 messages Sep 9, 2020 – Sep 11, 2020 1 message not in the archive

Subsystems (3)

v3 of 9 Superseded Latest is v9 →

lore
Share & tools
lore
#0 lore raw

v3 cover HSM driver for ACRN hypervisor

From: Shuo Liu <[email protected]>

ACRN is a Type 1 reference hypervisor stack, running directly on the bare-metal hardware, and is suitable for a variety of IoT and embedded device solutions.

ACRN implements a hybrid VMM architecture, using a privileged Service VM. The Service VM manages the system resources (CPU, memory, etc.) and I/O devices of User VMs. Multiple User VMs are supported, with each of them running Linux, Android OS or Windows. Both Service VM and User VMs are guest VM.

Below figure shows the architecture.

                Service VM                    User VM
      +----------------------------+  |  +------------------+
      |        +--------------+    |  |  |                  |
      |        |ACRN userspace|    |  |  |                  |
      |        +--------------+    |  |  |                  |
      |-----------------ioctl------|  |  |                  |   ...
      |kernel space   +----------+ |  |  |                  |
      |               |   HSM    | |  |  | Drivers          |
      |               +----------+ |  |  |                  |
      +--------------------|-------+  |  +------------------+
  +---------------------hypercall----------------------------------------+
  |                       ACRN Hypervisor                                |
  +----------------------------------------------------------------------+
  |                          Hardware                                    |
  +----------------------------------------------------------------------+

There is only one Service VM which could run Linux as OS.

In a typical case, the Service VM will be auto started when ACRN Hypervisor is booted. Then the ACRN userspace (an application running in Service VM) could be used to start/stop User VMs by communicating with ACRN Hypervisor Service Module (HSM).

ACRN Hypervisor Service Module (HSM) is a middle layer that allows the ACRN userspace and Service VM OS kernel to communicate with ACRN Hypervisor and manage different User VMs. This middle layer provides the following functionalities,
- Issues hypercalls to the hypervisor to manage User VMs:
* VM/vCPU management
* Memory management
* Device passthrough
* Interrupts injection
- I/O requests handling from User VMs.
- Exports ioctl through HSM char device.
- Exports function calls for other kernel modules

ACRN is focused on embedded system. So it doesn't support some features. E.g.,
- ACRN doesn't support VM migration.
- ACRN doesn't support vCPU migration.

This patch set adds the HSM to the Linux kernel.

The basic ARCN support was merged to upstream already.
https://lore.kernel.org/lkml/[email protected]/

ChangeLog:
v3:
- Used {get|put}_device() helpers on &acrn_dev->this_device
- Moved unused code from front patches to later ones.
- Removed self-defined pr_fmt() and dev_fmt()
- Provided comments for acrn_vm_list_lock.

v2:
- Removed API version related code. (Dave)
- Replaced pr_*() by dev_*(). (Greg)
- Used -ENOTTY as the error code of unsupported ioctl. (Greg)

Shuo Liu (16):
docs: acrn: Introduce ACRN
x86/acrn: Introduce acrn_{setup, remove}_intr_handler() x86/acrn: Introduce hypercall interfaces
virt: acrn: Introduce ACRN HSM basic driver
virt: acrn: Introduce VM management interfaces
virt: acrn: Introduce an ioctl to set vCPU registers state
virt: acrn: Introduce EPT mapping management
virt: acrn: Introduce I/O request management
virt: acrn: Introduce PCI configuration space PIO accesses combiner
virt: acrn: Introduce interfaces for PCI device passthrough
virt: acrn: Introduce interrupt injection interfaces
virt: acrn: Introduce interfaces to query C-states and P-states
allowed by hypervisor
virt: acrn: Introduce I/O ranges operation interfaces
virt: acrn: Introduce ioeventfd
virt: acrn: Introduce irqfd
virt: acrn: Introduce an interface for Service VM to control vCPU

Yin Fengwei (1):
x86/acrn: Introduce an API to check if a VM is privileged

 .../userspace-api/ioctl/ioctl-number.rst      |   1 +
 Documentation/virt/acrn/index.rst             |  11 +
 Documentation/virt/acrn/introduction.rst      |  40 ++
 Documentation/virt/acrn/io-request.rst        |  97 +++
 Documentation/virt/index.rst                  |   1 +
 MAINTAINERS                                   |   9 +
 arch/x86/include/asm/acrn.h                   |  74 ++
 arch/x86/kernel/cpu/acrn.c                    |  38 +-
 drivers/virt/Kconfig                          |   2 +
 drivers/virt/Makefile                         |   1 +
 drivers/virt/acrn/Kconfig                     |  15 +
 drivers/virt/acrn/Makefile                    |   3 +
 drivers/virt/acrn/acrn_drv.h                  | 228 +++++++
 drivers/virt/acrn/hsm.c                       | 425 ++++++++++++
 drivers/virt/acrn/hypercall.h                 | 254 +++++++
 drivers/virt/acrn/ioeventfd.c                 | 273 ++++++++
 drivers/virt/acrn/ioreq.c                     | 642 ++++++++++++++++++
 drivers/virt/acrn/irqfd.c                     | 235 +++++++
 drivers/virt/acrn/mm.c                        | 300 ++++++++
 drivers/virt/acrn/vm.c                        | 123 ++++
 include/uapi/linux/acrn.h                     | 486 +++++++++++++
 21 files changed, 3257 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/virt/acrn/index.rst
 create mode 100644 Documentation/virt/acrn/introduction.rst
 create mode 100644 Documentation/virt/acrn/io-request.rst
 create mode 100644 arch/x86/include/asm/acrn.h
 create mode 100644 drivers/virt/acrn/Kconfig
 create mode 100644 drivers/virt/acrn/Makefile
 create mode 100644 drivers/virt/acrn/acrn_drv.h
 create mode 100644 drivers/virt/acrn/hsm.c
 create mode 100644 drivers/virt/acrn/hypercall.h
 create mode 100644 drivers/virt/acrn/ioeventfd.c
 create mode 100644 drivers/virt/acrn/ioreq.c
 create mode 100644 drivers/virt/acrn/irqfd.c
 create mode 100644 drivers/virt/acrn/mm.c
 create mode 100644 drivers/virt/acrn/vm.c
 create mode 100644 include/uapi/linux/acrn.h

base-commit: 18445bf405cb331117bc98427b1ba6f12418ad17

Signature · 1 line
2.28.0
#1 lore raw

PATCH v3 1/17 docs: acrn: Introduce ACRN

From: Shuo Liu <[email protected]>

Add documentation on the following aspects of ACRN:

1) A brief introduction on the architecture of ACRN.
2) I/O request handling in ACRN.

To learn more about ACRN, please go to ACRN project website
https://projectacrn.org, or the documentation page
https://projectacrn.github.io/.

Cc Dave Hansen <[email protected]>
Cc Sen Christopherson <[email protected]>
Cc Dan Williams <[email protected]>
Cc Fengwei Yin <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 Documentation/virt/acrn/index.rst        | 11 +++
 Documentation/virt/acrn/introduction.rst | 40 ++++++++++
 Documentation/virt/acrn/io-request.rst   | 97 ++++++++++++++++++++++++
 Documentation/virt/index.rst             |  1 +
 MAINTAINERS                              |  7 ++
 5 files changed, 156 insertions(+)
 create mode 100644 Documentation/virt/acrn/index.rst
 create mode 100644 Documentation/virt/acrn/introduction.rst
 create mode 100644 Documentation/virt/acrn/io-request.rst
Documentation/virt/acrn/index.rst new file +11 -0
new file mode 100644
index 000000000000..e3cf99033bdb
@@ -0,0 +1,11 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+===============
+ACRN Hypervisor
+===============
+
+.. toctree::
+   :maxdepth: 1
+
+   introduction
+   io-request
Documentation/virt/acrn/introduction.rst new file +40 -0
new file mode 100644
index 000000000000..6b44924d5c0e
@@ -0,0 +1,40 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+ACRN Hypervisor Introduction
+============================
+
+The ACRN Hypervisor is a Type 1 hypervisor, running directly on the bare-metal
+hardware. It has a privileged management VM, called Service VM, to manage User
+VMs and do I/O emulation.
+
+ACRN userspace is an application running in the Service VM that emulates
+devices for a User VM based on command line configurations. ACRN Hypervisor
+Service Module (HSM) is a kernel module in the Service VM which provides
+hypervisor services to the ACRN userspace.
+
+Below figure shows the architecture.
+
+::
+
+                Service VM                    User VM
+      +----------------------------+  |  +------------------+
+      |        +--------------+    |  |  |                  |
+      |        |ACRN userspace|    |  |  |                  |
+      |        +--------------+    |  |  |                  |
+      |-----------------ioctl------|  |  |                  |   ...
+      |kernel space   +----------+ |  |  |                  |
+      |               |   HSM    | |  |  | Drivers          |
+      |               +----------+ |  |  |                  |
+      +--------------------|-------+  |  +------------------+
+  +---------------------hypercall----------------------------------------+
+  |                         ACRN Hypervisor                              |
+  +----------------------------------------------------------------------+
+  |                          Hardware                                    |
+  +----------------------------------------------------------------------+
+
+ACRN userspace allocates memory for the User VM, configures and initializes the
+devices used by the User VM, loads the virtual bootloader, initializes the
+virtual CPU state and handles I/O request accesses from the User VM. It uses
+ioctls to communicate with the HSM. HSM implements hypervisor services by
+interacting with the ACRN Hypervisor via hypercalls. HSM exports a char device
+interface (/dev/acrn_hsm) to userspace.
Documentation/virt/acrn/io-request.rst new file +97 -0

97 lines not on this page · raw message

Documentation/virt/index.rst +1 -0
index de1ab81df958..c10b519507f5 100644
@@ -11,6 +11,7 @@ Linux Virtualization Support
    uml/user_mode_linux
    paravirt_ops
    guest-halt-polling
+   acrn/index
 
 .. only:: html and subproject
 
MAINTAINERS +7 -0
index deaafb617361..e0fea5e464b4 100644
@@ -436,6 +436,13 @@ S:	Orphan
 F:	drivers/platform/x86/wmi.c
 F:	include/uapi/linux/wmi.h
 
+ACRN HYPERVISOR SERVICE MODULE
+M:	Shuo Liu <[email protected]>
+L:	[email protected]
+S:	Supported
+W:	https://projectacrn.org
+F:	Documentation/virt/acrn/
+
 AD1889 ALSA SOUND DRIVER
 L:	[email protected]
 S:	Maintained
Signature · 1 line
2.28.0
#2 lore raw

PATCH v3 3/17 x86/acrn: Introduce an API to check if a VM is privileged

In reply to [email protected]

From: Yin Fengwei <[email protected]>

ACRN Hypervisor reports hypervisor features via CPUID leaf 0x40000001 which is similar to KVM. A VM can check if it's the privileged VM using the feature bits. The Service VM is the only privileged VM by design.

Cc Dave Hansen <[email protected]>
Cc Sean Christopherson <[email protected]>
Cc Dan Williams <[email protected]>
Cc Fengwei Yin <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 arch/x86/include/asm/acrn.h |  9 +++++++++
 arch/x86/kernel/cpu/acrn.c  | 19 ++++++++++++++++++-
 2 files changed, 27 insertions(+), 1 deletion(-)
arch/x86/include/asm/acrn.h +9 -0
index ff259b69cde7..a2d4aea3a80d 100644
@@ -2,7 +2,16 @@
 #ifndef _ASM_X86_ACRN_H
 #define _ASM_X86_ACRN_H
 
+/*
+ * This CPUID returns feature bitmaps in EAX.
+ * Guest VM uses this to detect the appropriate feature bit.
+ */
+#define	ACRN_CPUID_FEATURES		0x40000001
+/* Bit 0 indicates whether guest VM is privileged */
+#define	ACRN_FEATURE_PRIVILEGED_VM	BIT(0)
+
 void acrn_setup_intr_handler(void (*handler)(void));
 void acrn_remove_intr_handler(void);
+bool acrn_is_privileged_vm(void);
 
 #endif /* _ASM_X86_ACRN_H */
arch/x86/kernel/cpu/acrn.c +18 -1
index bd1d7e759a0f..6f0a00cbbf7e 100644
@@ -21,9 +21,26 @@
 #include <asm/idtentry.h>
 #include <asm/irq_regs.h>
 
+static u32 acrn_cpuid_base(void)
+{
+	static u32 acrn_cpuid_base;
+
+	if (!acrn_cpuid_base && boot_cpu_has(X86_FEATURE_HYPERVISOR))
+		acrn_cpuid_base = hypervisor_cpuid_base("ACRNACRNACRN", 0);
+
+	return acrn_cpuid_base;
+}
+
+bool acrn_is_privileged_vm(void)
+{
+	return cpuid_eax(acrn_cpuid_base() | ACRN_CPUID_FEATURES) &
+			 ACRN_FEATURE_PRIVILEGED_VM;
+}
+EXPORT_SYMBOL_GPL(acrn_is_privileged_vm);
+
 static u32 __init acrn_detect(void)
 {
-	return hypervisor_cpuid_base("ACRNACRNACRN", 0);
+	return acrn_cpuid_base();
 }
 
 static void __init acrn_init_platform(void)
Signature · 1 line
2.28.0
#3 lore raw

PATCH v3 4/17 x86/acrn: Introduce hypercall interfaces

In reply to [email protected]

From: Shuo Liu <[email protected]>

The Service VM communicates with the hypervisor via conventional hypercalls. VMCALL instruction is used to make the hypercalls.

ACRN hypercall ABI:
* Hypercall number is in R8 register.
* Up to 2 parameters are in RDI and RSI registers.
* Return value is in RAX register.

Introduce the ACRN hypercall interfaces. Because GCC doesn't support R8 register as direct register constraints, here are two ways to use R8 in extended asm:
1) use explicit register variable as input
2) use supported constraint as input with a explicit MOV to R8 in beginning of asm

The number of instructions of above two ways are same. Asm code from 1)

  38:   41 b8 00 00 00 80       mov    $0x80000000,%r8d
  3e:   48 89 c7                mov    %rax,%rdi
  41:   0f 01 c1                vmcall

Here, writes to the lower dword (%r8d) clear the upper dword of %r8 when the CPU is in 64-bit mode.

Asm code from 2)

  38:   48 89 c7                mov    %rax,%rdi
  3b:   49 b8 00 00 00 80 00    movabs $0x80000000,%r8
  42:   00 00 00
  45:   0f 01 c1                vmcall

Choose 1) for code simplicity and a little bit of code size optimization.

Cc Dave Hansen <[email protected]>
Cc Sean Christopherson <[email protected]>
Cc Dan Williams <[email protected]>
Cc Fengwei Yin <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 arch/x86/include/asm/acrn.h | 57 +++++++++++++++++++++++++++++++++++++
 1 file changed, 57 insertions(+)
arch/x86/include/asm/acrn.h +57 -0
index a2d4aea3a80d..23a93b87edeb 100644
@@ -14,4 +14,61 @@ void acrn_setup_intr_handler(void (*handler)(void));
 void acrn_remove_intr_handler(void);
 bool acrn_is_privileged_vm(void);
 
+/*
+ * Hypercalls for ACRN
+ *
+ * - VMCALL instruction is used to implement ACRN hypercalls.
+ * - ACRN hypercall ABI:
+ *   - Hypercall number is passed in R8 register.
+ *   - Up to 2 arguments are passed in RDI, RSI.
+ *   - Return value will be placed in RAX.
+ */
+static inline long acrn_hypercall0(unsigned long hcall_id)
+{
+	register long r8 asm("r8");
+	long result;
+
+	/* Nothing can come between the r8 assignment and the asm: */
+	r8 = hcall_id;
+	asm volatile("vmcall\n\t"
+		     : "=a" (result)
+		     : "r" (r8)
+		     : );
+
+	return result;
+}
+
+static inline long acrn_hypercall1(unsigned long hcall_id,
+				   unsigned long param1)
+{
+	register long r8 asm("r8");
+	long result;
+
+	/* Nothing can come between the r8 assignment and the asm: */
+	r8 = hcall_id;
+	asm volatile("vmcall\n\t"
+		     : "=a" (result)
+		     : "r" (r8), "D" (param1)
+		     : );
+
+	return result;
+}
+
+static inline long acrn_hypercall2(unsigned long hcall_id,
+				   unsigned long param1,
+				   unsigned long param2)
+{
+	register long r8 asm("r8");
+	long result;
+
+	/* Nothing can come between the r8 assignment and the asm: */
+	r8 = hcall_id;
+	asm volatile("vmcall\n\t"
+		     : "=a" (result)
+		     : "r" (r8), "D" (param1), "S" (param2)
+		     : );
+
+	return result;
+}
+
 #endif /* _ASM_X86_ACRN_H */
Signature · 1 line
2.28.0
#4 lore raw

PATCH v3 6/17 virt: acrn: Introduce VM management interfaces

In reply to [email protected]

From: Shuo Liu <[email protected]>

The VM management interfaces expose several VM operations to ACRN userspace via ioctls. For example, creating VM, starting VM, destroying VM and so on.

The ACRN Hypervisor needs to exchange data with the ACRN userspace during the VM operations. HSM provides VM operation ioctls to the ACRN userspace and communicates with the ACRN Hypervisor for VM operations via hypercalls.

HSM maintains a list of User VM. Each User VM will be bound to an existing file descriptor of /dev/acrn_hsm. The User VM will be destroyed when the file descriptor is closed.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 .../userspace-api/ioctl/ioctl-number.rst      |  1 +
 MAINTAINERS                                   |  1 +
 drivers/virt/acrn/Makefile                    |  2 +-
 drivers/virt/acrn/acrn_drv.h                  | 22 +++++-
 drivers/virt/acrn/hsm.c                       | 66 ++++++++++++++++
 drivers/virt/acrn/hypercall.h                 | 78 +++++++++++++++++++
 drivers/virt/acrn/vm.c                        | 69 ++++++++++++++++
 include/uapi/linux/acrn.h                     | 56 +++++++++++++
 8 files changed, 293 insertions(+), 2 deletions(-)
 create mode 100644 drivers/virt/acrn/hypercall.h
 create mode 100644 drivers/virt/acrn/vm.c
 create mode 100644 include/uapi/linux/acrn.h
Documentation/userspace-api/ioctl/ioctl-number.rst +1 -0
index 2a198838fca9..ac60efedb104 100644
@@ -319,6 +319,7 @@ Code  Seq#    Include File                                           Comments
 0xA0  all    linux/sdp/sdp.h                                         Industrial Device Project
                                                                      <mailto:[email protected]>
 0xA1  0      linux/vtpm_proxy.h                                      TPM Emulator Proxy Driver
+0xA2  all    uapi/linux/acrn.h                                       ACRN hypervisor
 0xA3  80-8F                                                          Port ACL  in development:
                                                                      <mailto:[email protected]>
 0xA3  90-9F  linux/dtlk.h
MAINTAINERS +1 -0
index 3030d0e93d02..d4c1ef303c2d 100644
@@ -443,6 +443,7 @@ S:	Supported
 W:	https://projectacrn.org
 F:	Documentation/virt/acrn/
 F:	drivers/virt/acrn/
+F:	include/uapi/linux/acrn.h
 
 AD1889 ALSA SOUND DRIVER
 L:	[email protected]
drivers/virt/acrn/Makefile +1 -1
index 6920ed798aaf..cf8b4ed5e74e 100644
@@ -1,3 +1,3 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_ACRN_HSM)	:= acrn.o
-acrn-y := hsm.o
+acrn-y := hsm.o vm.o
drivers/virt/acrn/acrn_drv.h +21 -1
index 29eedd696327..043ae6840995 100644
@@ -3,16 +3,36 @@
 #ifndef __ACRN_HSM_DRV_H
 #define __ACRN_HSM_DRV_H
 
+#include <linux/acrn.h>
+#include <linux/dev_printk.h>
 #include <linux/types.h>
 
+#include "hypercall.h"
+
 #define ACRN_INVALID_VMID (0xffffU)
 
+#define ACRN_VM_FLAG_DESTROYED		0U
+extern struct list_head acrn_vm_list;
+extern rwlock_t acrn_vm_list_lock;
 /**
  * struct acrn_vm - Properties of ACRN User VM.
+ * @dev:	The struct device this VM belongs to
+ * @list:	Entry within global list of all VMs
  * @vmid:	User VM ID
+ * @vcpu_num:	Number of virtual CPUs in the VM
+ * @flags:	Flags (ACRN_VM_FLAG_*) of the VM. This is VM flag management
+ *		in HSM which is different from the &acrn_vm_creation.vm_flag.
  */
 struct acrn_vm {
-	u16	vmid;
+	struct device		*dev;
+	struct list_head	list;
+	u16			vmid;
+	int			vcpu_num;
+	unsigned long		flags;
 };
 
+struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
+			       struct acrn_vm_creation *vm_param);
+int acrn_vm_destroy(struct acrn_vm *vm);
+
 #endif /* __ACRN_HSM_DRV_H */
drivers/virt/acrn/hsm.c +66 -0

92 lines not on this page · raw message

drivers/virt/acrn/hypercall.h new file +78 -0

78 lines not on this page · raw message

drivers/virt/acrn/vm.c new file +69 -0

69 lines not on this page · raw message

include/uapi/linux/acrn.h new file +56 -0

56 lines not on this page · raw message

Signature · 1 line
2.28.0
#5 lore raw

Re: [PATCH v3 06/17] virt: acrn: Introduce VM management interfaces

Quoted 126 lines · On Wed, Sep 09, 2020 at 05:08:25PM +0800, [email protected] wrote:

From: Shuo Liu <[email protected]>

The VM management interfaces expose several VM operations to ACRN userspace via ioctls. For example, creating VM, starting VM, destroying VM and so on.

The ACRN Hypervisor needs to exchange data with the ACRN userspace during the VM operations. HSM provides VM operation ioctls to the ACRN userspace and communicates with the ACRN Hypervisor for VM operations via hypercalls.

HSM maintains a list of User VM. Each User VM will be bound to an existing file descriptor of /dev/acrn_hsm. The User VM will be destroyed when the file descriptor is closed.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 .../userspace-api/ioctl/ioctl-number.rst      |  1 +
 MAINTAINERS                                   |  1 +
 drivers/virt/acrn/Makefile                    |  2 +-
 drivers/virt/acrn/acrn_drv.h                  | 22 +++++-
 drivers/virt/acrn/hsm.c                       | 66 ++++++++++++++++
 drivers/virt/acrn/hypercall.h                 | 78 +++++++++++++++++++
 drivers/virt/acrn/vm.c                        | 69 ++++++++++++++++
 include/uapi/linux/acrn.h                     | 56 +++++++++++++
 8 files changed, 293 insertions(+), 2 deletions(-)
 create mode 100644 drivers/virt/acrn/hypercall.h
 create mode 100644 drivers/virt/acrn/vm.c
 create mode 100644 include/uapi/linux/acrn.h
Documentation/userspace-api/ioctl/ioctl-number.rst +1 -0
index 2a198838fca9..ac60efedb104 100644
@@ -319,6 +319,7 @@ Code  Seq#    Include File                                           Comments
 0xA0  all    linux/sdp/sdp.h                                         Industrial Device Project
                                                                      <mailto:[email protected]>
 0xA1  0      linux/vtpm_proxy.h                                      TPM Emulator Proxy Driver
+0xA2  all    uapi/linux/acrn.h                                       ACRN hypervisor
 0xA3  80-8F                                                          Port ACL  in development:
                                                                      <mailto:[email protected]>
 0xA3  90-9F  linux/dtlk.h
MAINTAINERS +1 -0
index 3030d0e93d02..d4c1ef303c2d 100644
@@ -443,6 +443,7 @@ S:	Supported
 W:	https://projectacrn.org
 F:	Documentation/virt/acrn/
 F:	drivers/virt/acrn/
+F:	include/uapi/linux/acrn.h
 
 AD1889 ALSA SOUND DRIVER
 L:	[email protected]
drivers/virt/acrn/Makefile +1 -1
index 6920ed798aaf..cf8b4ed5e74e 100644
@@ -1,3 +1,3 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_ACRN_HSM)	:= acrn.o
-acrn-y := hsm.o
+acrn-y := hsm.o vm.o
drivers/virt/acrn/acrn_drv.h +21 -1
index 29eedd696327..043ae6840995 100644
@@ -3,16 +3,36 @@
 #ifndef __ACRN_HSM_DRV_H
 #define __ACRN_HSM_DRV_H
 
+#include <linux/acrn.h>
+#include <linux/dev_printk.h>
 #include <linux/types.h>
 
+#include "hypercall.h"
+
 #define ACRN_INVALID_VMID (0xffffU)
 
+#define ACRN_VM_FLAG_DESTROYED		0U
+extern struct list_head acrn_vm_list;
+extern rwlock_t acrn_vm_list_lock;
 /**
  * struct acrn_vm - Properties of ACRN User VM.
+ * @dev:	The struct device this VM belongs to
+ * @list:	Entry within global list of all VMs
  * @vmid:	User VM ID
+ * @vcpu_num:	Number of virtual CPUs in the VM
+ * @flags:	Flags (ACRN_VM_FLAG_*) of the VM. This is VM flag management
+ *		in HSM which is different from the &acrn_vm_creation.vm_flag.
  */
 struct acrn_vm {
-	u16	vmid;
+	struct device		*dev;
+	struct list_head	list;
+	u16			vmid;
+	int			vcpu_num;
+	unsigned long		flags;
 };
 
+struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
+			       struct acrn_vm_creation *vm_param);
+int acrn_vm_destroy(struct acrn_vm *vm);
+
 #endif /* __ACRN_HSM_DRV_H */
drivers/virt/acrn/hsm.c +2 -0
index 28a3052ffa55..bc85a3c14f87 100644
@@ -19,6 +19,8 @@
 
 #include "acrn_drv.h"
 
+static struct miscdevice acrn_dev;
+
 /*
  * When /dev/acrn_hsm is opened, a 'struct acrn_vm' object is created to
  * represent a VM instance and continues to be associated with the opened file

drivers/virt/acrn/hsm.c static int acrn_dev_open(struct inode *inode, struct file *filp) · line 39

Quoted 4 lines
drivers/virt/acrn/hsm.c +1 -0
@@ -34,14 +36,77 @@ static int acrn_dev_open(struct inode *inode, struct file *filp)
 		return -ENOMEM;
 
 	vm->vmid = ACRN_INVALID_VMID;
+	vm->dev = get_device(acrn_dev.this_device);

You are grabbing a reference on a static structure?

Ugh, who reviewed this code before it was sent out???

Think about what you just asked for here, and if that actually makes any sense at all...

Quoted 4 lines
drivers/virt/acrn/hsm.c +0 -0
 	filp->private_data = vm;
 	return 0;
 }
 

drivers/virt/acrn/hsm.c static int acrn_dev_open(struct inode *inode, struct file *filp) · line 53

Quoted 10 lines
drivers/virt/acrn/hsm.c +10 -0
+static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
+			   unsigned long ioctl_param)
+{
+	struct acrn_vm *vm = filp->private_data;
+	struct acrn_vm_creation *vm_param;
+	int ret = 0;
+
+	if (vm->vmid == ACRN_INVALID_VMID && cmd != ACRN_IOCTL_CREATE_VM) {
+		dev_err(vm->dev, "ioctl 0x%x: Invalid VM state!\n", cmd);
+		return -EFAULT;

You did not just cause a memory fault, so please do not return that error to userspace, it will just get confused.

And does this allow userspace to spam the kernel log with errors by calling this ioctl with invalid commands? If so, make this a debugging message please.

drivers/virt/acrn/hsm.c static int acrn_dev_open(struct inode *inode, struct file *filp) · line 65

Quoted 12 lines
drivers/virt/acrn/hsm.c +12 -0
+	}
+
+	switch (cmd) {
+	case ACRN_IOCTL_CREATE_VM:
+		vm_param = memdup_user((void __user *)ioctl_param,
+				       sizeof(struct acrn_vm_creation));
+		if (IS_ERR(vm_param))
+			return PTR_ERR(vm_param);
+
+		vm = acrn_vm_create(vm, vm_param);
+		if (!vm) {
+			ret = -EFAULT;

Again, this is not a fault, fix all of these up please.

And you really are doing the validation of the parameters in the hypervisor itself? You should add a comment here explicitly saying you are trusting that layer to do this, as it will come up in the future once bugs start being reported :)

drivers/virt/acrn/hsm.c static int acrn_dev_open(struct inode *inode, struct file *filp) · line 73

Quoted 8 lines
drivers/virt/acrn/hsm.c +8 -0
+			kfree(vm_param);
+			break;
+		}
+
+		if (copy_to_user((void __user *)ioctl_param, vm_param,
+				 sizeof(struct acrn_vm_creation))) {
+			acrn_vm_destroy(vm);
+			ret = -EFAULT;

That's a correct error :)

Quoted 167 lines
drivers/virt/acrn/hsm.c +33 -0
+		}
+
+		kfree(vm_param);
+		break;
+	case ACRN_IOCTL_START_VM:
+		ret = hcall_start_vm(vm->vmid);
+		if (ret < 0)
+			dev_err(vm->dev, "Failed to start VM %u!\n", vm->vmid);
+		break;
+	case ACRN_IOCTL_PAUSE_VM:
+		ret = hcall_pause_vm(vm->vmid);
+		if (ret < 0)
+			dev_err(vm->dev, "Failed to pause VM %u!\n", vm->vmid);
+		break;
+	case ACRN_IOCTL_RESET_VM:
+		ret = hcall_reset_vm(vm->vmid);
+		if (ret < 0)
+			dev_err(vm->dev, "Failed to restart VM %u!\n", vm->vmid);
+		break;
+	case ACRN_IOCTL_DESTROY_VM:
+		ret = acrn_vm_destroy(vm);
+		break;
+	default:
+		dev_warn(vm->dev, "Unknown IOCTL 0x%x!\n", cmd);
+		ret = -ENOTTY;
+	}
+
+	return ret;
+}
+
 static int acrn_dev_release(struct inode *inode, struct file *filp)
 {
 	struct acrn_vm *vm = filp->private_data;
 
+	acrn_vm_destroy(vm);
+	put_device(vm->dev);
 	kfree(vm);
 	return 0;
 }
@@ -50,6 +115,7 @@ static const struct file_operations acrn_fops = {
 	.owner		= THIS_MODULE,
 	.open		= acrn_dev_open,
 	.release	= acrn_dev_release,
+	.unlocked_ioctl = acrn_dev_ioctl,
 };
 
 static struct miscdevice acrn_dev = {
drivers/virt/acrn/hypercall.h new file +78 -0
new file mode 100644
index 000000000000..426b66cadb1f
@@ -0,0 +1,78 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * ACRN HSM: hypercalls of ACRN Hypervisor
+ */
+#ifndef __ACRN_HSM_HYPERCALL_H
+#define __ACRN_HSM_HYPERCALL_H
+#include <asm/acrn.h>
+
+/*
+ * Hypercall IDs of the ACRN Hypervisor
+ */
+#define _HC_ID(x, y) (((x) << 24) | (y))
+
+#define HC_ID 0x80UL
+
+#define HC_ID_VM_BASE			0x10UL
+#define HC_CREATE_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x00)
+#define HC_DESTROY_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x01)
+#define HC_START_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x02)
+#define HC_PAUSE_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x03)
+#define HC_RESET_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x05)
+
+/**
+ * hcall_create_vm() - Create a User VM
+ * @vminfo:	Service VM GPA of info of User VM creation
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_create_vm(u64 vminfo)
+{
+	return acrn_hypercall1(HC_CREATE_VM, vminfo);
+}
+
+/**
+ * hcall_start_vm() - Start a User VM
+ * @vmid:	User VM ID
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_start_vm(u64 vmid)
+{
+	return acrn_hypercall1(HC_START_VM, vmid);
+}
+
+/**
+ * hcall_pause_vm() - Pause a User VM
+ * @vmid:	User VM ID
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_pause_vm(u64 vmid)
+{
+	return acrn_hypercall1(HC_PAUSE_VM, vmid);
+}
+
+/**
+ * hcall_destroy_vm() - Destroy a User VM
+ * @vmid:	User VM ID
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_destroy_vm(u64 vmid)
+{
+	return acrn_hypercall1(HC_DESTROY_VM, vmid);
+}
+
+/**
+ * hcall_reset_vm() - Reset a User VM
+ * @vmid:	User VM ID
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_reset_vm(u64 vmid)
+{
+	return acrn_hypercall1(HC_RESET_VM, vmid);
+}
+
+#endif /* __ACRN_HSM_HYPERCALL_H */
drivers/virt/acrn/vm.c new file +30 -0
new file mode 100644
index 000000000000..851a9481a78e
@@ -0,0 +1,69 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * ACRN_HSM: Virtual Machine management
+ *
+ * Copyright (C) 2020 Intel Corporation. All rights reserved.
+ *
+ * Authors:
+ *	Jason Chen CJ <[email protected]>
+ *	Yakui Zhao <[email protected]>
+ */
+#include <linux/io.h>
+#include <linux/mm.h>
+#include <linux/slab.h>
+
+#include "acrn_drv.h"
+
+/* List of VMs */
+LIST_HEAD(acrn_vm_list);
+/*
+ * acrn_vm_list is read in a tasklet which dispatch I/O requests and is wrote
+ * in VM creation ioctl. Use the rwlock mechanism to protect it.
+ */
+DEFINE_RWLOCK(acrn_vm_list_lock);
+
+struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
+			       struct acrn_vm_creation *vm_param)
+{
+	int ret;
+
+	ret = hcall_create_vm(virt_to_phys(vm_param));

drivers/virt/acrn/vm.c line 40

Quoted 10 lines
drivers/virt/acrn/vm.c new file +10 -0
+	if (ret < 0 || vm_param->vmid == ACRN_INVALID_VMID) {
+		dev_err(vm->dev, "Failed to create VM! Error: %d\n", ret);
+		return NULL;
+	}
+
+	vm->vmid = vm_param->vmid;
+	vm->vcpu_num = vm_param->vcpu_num;
+
+	write_lock_bh(&acrn_vm_list_lock);
+	list_add(&vm->list, &acrn_vm_list);

Wait, why do you have a global list of devices? Shouldn't that device be tied to the vm structure? Who will be iterating this list that does not have the file handle to start with?

Quoted 8 lines
drivers/virt/acrn/vm.c new file +8 -0
+	write_unlock_bh(&acrn_vm_list_lock);
+
+	dev_dbg(vm->dev, "VM %u created.\n", vm->vmid);
+	return vm;
+}
+
+int acrn_vm_destroy(struct acrn_vm *vm)
+{

drivers/virt/acrn/vm.c line 58

Quoted 10 lines
drivers/virt/acrn/vm.c new file +10 -0
+	int ret;
+
+	if (vm->vmid == ACRN_INVALID_VMID ||
+	    test_and_set_bit(ACRN_VM_FLAG_DESTROYED, &vm->flags))
+		return 0;
+
+	/* Remove from global VM list */
+	write_lock_bh(&acrn_vm_list_lock);
+	list_del_init(&vm->list);
+	write_unlock_bh(&acrn_vm_list_lock);

Again, why this list at all?

thanks,

greg k-h

author
#6 lore raw

Re: [PATCH v3 06/17] virt: acrn: Introduce VM management interfaces

Hi Greg,

Quoted 129 lines · Greg Kroah-Hartman
Quoted 128 lines · On Wed, Sep 09, 2020 at 05:08:25PM +0800, [email protected] wrote:

From: Shuo Liu <[email protected]>

The VM management interfaces expose several VM operations to ACRN userspace via ioctls. For example, creating VM, starting VM, destroying VM and so on.

The ACRN Hypervisor needs to exchange data with the ACRN userspace during the VM operations. HSM provides VM operation ioctls to the ACRN userspace and communicates with the ACRN Hypervisor for VM operations via hypercalls.

HSM maintains a list of User VM. Each User VM will be bound to an existing file descriptor of /dev/acrn_hsm. The User VM will be destroyed when the file descriptor is closed.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 .../userspace-api/ioctl/ioctl-number.rst      |  1 +
 MAINTAINERS                                   |  1 +
 drivers/virt/acrn/Makefile                    |  2 +-
 drivers/virt/acrn/acrn_drv.h                  | 22 +++++-
 drivers/virt/acrn/hsm.c                       | 66 ++++++++++++++++
 drivers/virt/acrn/hypercall.h                 | 78 +++++++++++++++++++
 drivers/virt/acrn/vm.c                        | 69 ++++++++++++++++
 include/uapi/linux/acrn.h                     | 56 +++++++++++++
 8 files changed, 293 insertions(+), 2 deletions(-)
 create mode 100644 drivers/virt/acrn/hypercall.h
 create mode 100644 drivers/virt/acrn/vm.c
 create mode 100644 include/uapi/linux/acrn.h
Documentation/userspace-api/ioctl/ioctl-number.rst +1 -0
index 2a198838fca9..ac60efedb104 100644
@@ -319,6 +319,7 @@ Code  Seq#    Include File                                           Comments
 0xA0  all    linux/sdp/sdp.h                                         Industrial Device Project
                                                                      <mailto:[email protected]>
 0xA1  0      linux/vtpm_proxy.h                                      TPM Emulator Proxy Driver
+0xA2  all    uapi/linux/acrn.h                                       ACRN hypervisor
 0xA3  80-8F                                                          Port ACL  in development:
                                                                      <mailto:[email protected]>
 0xA3  90-9F  linux/dtlk.h
MAINTAINERS +1 -0
index 3030d0e93d02..d4c1ef303c2d 100644
@@ -443,6 +443,7 @@ S:	Supported
 W:	https://projectacrn.org
 F:	Documentation/virt/acrn/
 F:	drivers/virt/acrn/
+F:	include/uapi/linux/acrn.h
 
 AD1889 ALSA SOUND DRIVER
 L:	[email protected]
drivers/virt/acrn/Makefile +1 -1
index 6920ed798aaf..cf8b4ed5e74e 100644
@@ -1,3 +1,3 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_ACRN_HSM)	:= acrn.o
-acrn-y := hsm.o
+acrn-y := hsm.o vm.o
drivers/virt/acrn/acrn_drv.h +21 -1
index 29eedd696327..043ae6840995 100644
@@ -3,16 +3,36 @@
 #ifndef __ACRN_HSM_DRV_H
 #define __ACRN_HSM_DRV_H
 
+#include <linux/acrn.h>
+#include <linux/dev_printk.h>
 #include <linux/types.h>
 
+#include "hypercall.h"
+
 #define ACRN_INVALID_VMID (0xffffU)
 
+#define ACRN_VM_FLAG_DESTROYED		0U
+extern struct list_head acrn_vm_list;
+extern rwlock_t acrn_vm_list_lock;
 /**
  * struct acrn_vm - Properties of ACRN User VM.
+ * @dev:	The struct device this VM belongs to
+ * @list:	Entry within global list of all VMs
  * @vmid:	User VM ID
+ * @vcpu_num:	Number of virtual CPUs in the VM
+ * @flags:	Flags (ACRN_VM_FLAG_*) of the VM. This is VM flag management
+ *		in HSM which is different from the &acrn_vm_creation.vm_flag.
  */
 struct acrn_vm {
-	u16	vmid;
+	struct device		*dev;
+	struct list_head	list;
+	u16			vmid;
+	int			vcpu_num;
+	unsigned long		flags;
 };
 
+struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
+			       struct acrn_vm_creation *vm_param);
+int acrn_vm_destroy(struct acrn_vm *vm);
+
 #endif /* __ACRN_HSM_DRV_H */
drivers/virt/acrn/hsm.c +2 -0
index 28a3052ffa55..bc85a3c14f87 100644
@@ -19,6 +19,8 @@
 
 #include "acrn_drv.h"
 
+static struct miscdevice acrn_dev;
+
 /*
  * When /dev/acrn_hsm is opened, a 'struct acrn_vm' object is created to
  * represent a VM instance and continues to be associated with the opened file
@@ -34,14 +36,77 @@ static int acrn_dev_open(struct inode *inode, struct file *filp)
 		return -ENOMEM;
 
Quoted 4 lines

drivers/virt/acrn/hsm.c static int acrn_dev_open(struct inode *inode, struct file *filp) · line 39

Quoted 2 lines
drivers/virt/acrn/hsm.c +1 -0
 	vm->vmid = ACRN_INVALID_VMID;
+	vm->dev = get_device(acrn_dev.this_device);

You are grabbing a reference on a static structure?

acrn_dev is static, but acrn_dev.this_device is not.

Quoted 2 lines

Ugh, who reviewed this code before it was sent out???

This part was just newly added according to your suggestion.. Please refer to
https://lore.kernel.org/lkml/[email protected]/

Quoted 8 lines

Think about what you just asked for here, and if that actually makes any sense at all...

Quoted 4 lines
drivers/virt/acrn/hsm.c +0 -0
 	filp->private_data = vm;
 	return 0;
 }
 
Quoted 13 lines

drivers/virt/acrn/hsm.c static int acrn_dev_open(struct inode *inode, struct file *filp) · line 53

Quoted 10 lines
drivers/virt/acrn/hsm.c +10 -0
+static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
+			   unsigned long ioctl_param)
+{
+	struct acrn_vm *vm = filp->private_data;
+	struct acrn_vm_creation *vm_param;
+	int ret = 0;
+
+	if (vm->vmid == ACRN_INVALID_VMID && cmd != ACRN_IOCTL_CREATE_VM) {
+		dev_err(vm->dev, "ioctl 0x%x: Invalid VM state!\n", cmd);
+		return -EFAULT;

You did not just cause a memory fault, so please do not return that error to userspace, it will just get confused.

Will change to -EINVAL.

Quoted 4 lines

And does this allow userspace to spam the kernel log with errors by calling this ioctl with invalid commands? If so, make this a debugging message please.

OK. It will be dev_dbg().

Quoted 15 lines

drivers/virt/acrn/hsm.c static int acrn_dev_open(struct inode *inode, struct file *filp) · line 65

Quoted 13 lines
drivers/virt/acrn/hsm.c +12 -0
+	}
+
+	switch (cmd) {
+	case ACRN_IOCTL_CREATE_VM:
+		vm_param = memdup_user((void __user *)ioctl_param,
+				       sizeof(struct acrn_vm_creation));
+		if (IS_ERR(vm_param))
+			return PTR_ERR(vm_param);
+
+		vm = acrn_vm_create(vm, vm_param);
+		if (!vm) {
+			ret = -EFAULT;

Again, this is not a fault, fix all of these up please.

Sure. I will change them to -EINVAL.

Quoted 5 lines

And you really are doing the validation of the parameters in the hypervisor itself? You should add a comment here explicitly saying you are trusting that layer to do this, as it will come up in the future once bugs start being reported :)

Yes, the hypervisor has. I will mention it somewhere. Maybe comments before acrn_dev_ioctl().

Quoted 185 lines

drivers/virt/acrn/hsm.c static int acrn_dev_open(struct inode *inode, struct file *filp) · line 73

Quoted 9 lines
drivers/virt/acrn/hsm.c +8 -0
+			kfree(vm_param);
+			break;
+		}
+
+		if (copy_to_user((void __user *)ioctl_param, vm_param,
+				 sizeof(struct acrn_vm_creation))) {
+			acrn_vm_destroy(vm);
+			ret = -EFAULT;

That's a correct error :)

Quoted 173 lines
drivers/virt/acrn/hsm.c +33 -0
+		}
+
+		kfree(vm_param);
+		break;
+	case ACRN_IOCTL_START_VM:
+		ret = hcall_start_vm(vm->vmid);
+		if (ret < 0)
+			dev_err(vm->dev, "Failed to start VM %u!\n", vm->vmid);
+		break;
+	case ACRN_IOCTL_PAUSE_VM:
+		ret = hcall_pause_vm(vm->vmid);
+		if (ret < 0)
+			dev_err(vm->dev, "Failed to pause VM %u!\n", vm->vmid);
+		break;
+	case ACRN_IOCTL_RESET_VM:
+		ret = hcall_reset_vm(vm->vmid);
+		if (ret < 0)
+			dev_err(vm->dev, "Failed to restart VM %u!\n", vm->vmid);
+		break;
+	case ACRN_IOCTL_DESTROY_VM:
+		ret = acrn_vm_destroy(vm);
+		break;
+	default:
+		dev_warn(vm->dev, "Unknown IOCTL 0x%x!\n", cmd);
+		ret = -ENOTTY;
+	}
+
+	return ret;
+}
+
 static int acrn_dev_release(struct inode *inode, struct file *filp)
 {
 	struct acrn_vm *vm = filp->private_data;
 
+	acrn_vm_destroy(vm);
+	put_device(vm->dev);
 	kfree(vm);
 	return 0;
 }
@@ -50,6 +115,7 @@ static const struct file_operations acrn_fops = {
 	.owner		= THIS_MODULE,
 	.open		= acrn_dev_open,
 	.release	= acrn_dev_release,
+	.unlocked_ioctl = acrn_dev_ioctl,
 };
 
 static struct miscdevice acrn_dev = {
drivers/virt/acrn/hypercall.h new file +78 -0
new file mode 100644
index 000000000000..426b66cadb1f
@@ -0,0 +1,78 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * ACRN HSM: hypercalls of ACRN Hypervisor
+ */
+#ifndef __ACRN_HSM_HYPERCALL_H
+#define __ACRN_HSM_HYPERCALL_H
+#include <asm/acrn.h>
+
+/*
+ * Hypercall IDs of the ACRN Hypervisor
+ */
+#define _HC_ID(x, y) (((x) << 24) | (y))
+
+#define HC_ID 0x80UL
+
+#define HC_ID_VM_BASE			0x10UL
+#define HC_CREATE_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x00)
+#define HC_DESTROY_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x01)
+#define HC_START_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x02)
+#define HC_PAUSE_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x03)
+#define HC_RESET_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x05)
+
+/**
+ * hcall_create_vm() - Create a User VM
+ * @vminfo:	Service VM GPA of info of User VM creation
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_create_vm(u64 vminfo)
+{
+	return acrn_hypercall1(HC_CREATE_VM, vminfo);
+}
+
+/**
+ * hcall_start_vm() - Start a User VM
+ * @vmid:	User VM ID
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_start_vm(u64 vmid)
+{
+	return acrn_hypercall1(HC_START_VM, vmid);
+}
+
+/**
+ * hcall_pause_vm() - Pause a User VM
+ * @vmid:	User VM ID
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_pause_vm(u64 vmid)
+{
+	return acrn_hypercall1(HC_PAUSE_VM, vmid);
+}
+
+/**
+ * hcall_destroy_vm() - Destroy a User VM
+ * @vmid:	User VM ID
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_destroy_vm(u64 vmid)
+{
+	return acrn_hypercall1(HC_DESTROY_VM, vmid);
+}
+
+/**
+ * hcall_reset_vm() - Reset a User VM
+ * @vmid:	User VM ID
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_reset_vm(u64 vmid)
+{
+	return acrn_hypercall1(HC_RESET_VM, vmid);
+}
+
+#endif /* __ACRN_HSM_HYPERCALL_H */
drivers/virt/acrn/vm.c new file +36 -0
new file mode 100644
index 000000000000..851a9481a78e
@@ -0,0 +1,69 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * ACRN_HSM: Virtual Machine management
+ *
+ * Copyright (C) 2020 Intel Corporation. All rights reserved.
+ *
+ * Authors:
+ *	Jason Chen CJ <[email protected]>
+ *	Yakui Zhao <[email protected]>
+ */
+#include <linux/io.h>
+#include <linux/mm.h>
+#include <linux/slab.h>
+
+#include "acrn_drv.h"
+
+/* List of VMs */
+LIST_HEAD(acrn_vm_list);
+/*
+ * acrn_vm_list is read in a tasklet which dispatch I/O requests and is wrote
+ * in VM creation ioctl. Use the rwlock mechanism to protect it.
+ */
+DEFINE_RWLOCK(acrn_vm_list_lock);
+
+struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
+			       struct acrn_vm_creation *vm_param)
+{
+	int ret;
+
+	ret = hcall_create_vm(virt_to_phys(vm_param));
+	if (ret < 0 || vm_param->vmid == ACRN_INVALID_VMID) {
+		dev_err(vm->dev, "Failed to create VM! Error: %d\n", ret);
+		return NULL;
+	}
+
+	vm->vmid = vm_param->vmid;
Quoted 8 lines

drivers/virt/acrn/vm.c line 40

Quoted 4 lines
drivers/virt/acrn/vm.c new file +4 -0
+	vm->vcpu_num = vm_param->vcpu_num;
+
+	write_lock_bh(&acrn_vm_list_lock);
+	list_add(&vm->list, &acrn_vm_list);

Wait, why do you have a global list of devices? Shouldn't that device be tied to the vm structure? Who will be iterating this list that does not have the file handle to start with?

Active VMs in this list will be used by the I/O requests dispatching tasklet ioreq_tasklet, whose callback function is ioreq_tasklet_handler() in patch 0009. ioreq_tasklet_handler() currently handles the notification interrupt from the hypervisor and dispatches I/O requests to each VMs.

So, the VM list is iterated in a tasklet and be protected by a rwlock. The device tied to vm structure is just for log API dev_*() now.

Quoted 15 lines
Quoted 15 lines
drivers/virt/acrn/vm.c new file +12 -0
+	write_unlock_bh(&acrn_vm_list_lock);
+
+	dev_dbg(vm->dev, "VM %u created.\n", vm->vmid);
+	return vm;
+}
+
+int acrn_vm_destroy(struct acrn_vm *vm)
+{
+	int ret;
+
+	if (vm->vmid == ACRN_INVALID_VMID ||
+	    test_and_set_bit(ACRN_VM_FLAG_DESTROYED, &vm->flags))
Quoted 8 lines

drivers/virt/acrn/vm.c line 58

Quoted 6 lines
drivers/virt/acrn/vm.c new file +6 -0
+		return 0;
+
+	/* Remove from global VM list */
+	write_lock_bh(&acrn_vm_list_lock);
+	list_del_init(&vm->list);
+	write_unlock_bh(&acrn_vm_list_lock);

Again, why this list at all?

Please refer to the previous comments.

Thanks
shuo

#7 lore raw

Re: [PATCH v3 06/17] virt: acrn: Introduce VM management interfaces

Quoted 132 lines · Shuo A Liu

Hi Greg,

Quoted 129 lines · Greg Kroah-Hartman
Quoted 128 lines · On Wed, Sep 09, 2020 at 05:08:25PM +0800, [email protected] wrote:

From: Shuo Liu <[email protected]>

The VM management interfaces expose several VM operations to ACRN userspace via ioctls. For example, creating VM, starting VM, destroying VM and so on.

The ACRN Hypervisor needs to exchange data with the ACRN userspace during the VM operations. HSM provides VM operation ioctls to the ACRN userspace and communicates with the ACRN Hypervisor for VM operations via hypercalls.

HSM maintains a list of User VM. Each User VM will be bound to an existing file descriptor of /dev/acrn_hsm. The User VM will be destroyed when the file descriptor is closed.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 .../userspace-api/ioctl/ioctl-number.rst      |  1 +
 MAINTAINERS                                   |  1 +
 drivers/virt/acrn/Makefile                    |  2 +-
 drivers/virt/acrn/acrn_drv.h                  | 22 +++++-
 drivers/virt/acrn/hsm.c                       | 66 ++++++++++++++++
 drivers/virt/acrn/hypercall.h                 | 78 +++++++++++++++++++
 drivers/virt/acrn/vm.c                        | 69 ++++++++++++++++
 include/uapi/linux/acrn.h                     | 56 +++++++++++++
 8 files changed, 293 insertions(+), 2 deletions(-)
 create mode 100644 drivers/virt/acrn/hypercall.h
 create mode 100644 drivers/virt/acrn/vm.c
 create mode 100644 include/uapi/linux/acrn.h
Documentation/userspace-api/ioctl/ioctl-number.rst +1 -0
index 2a198838fca9..ac60efedb104 100644
@@ -319,6 +319,7 @@ Code  Seq#    Include File                                           Comments
 0xA0  all    linux/sdp/sdp.h                                         Industrial Device Project
                                                                      <mailto:[email protected]>
 0xA1  0      linux/vtpm_proxy.h                                      TPM Emulator Proxy Driver
+0xA2  all    uapi/linux/acrn.h                                       ACRN hypervisor
 0xA3  80-8F                                                          Port ACL  in development:
                                                                      <mailto:[email protected]>
 0xA3  90-9F  linux/dtlk.h
MAINTAINERS +1 -0
index 3030d0e93d02..d4c1ef303c2d 100644
@@ -443,6 +443,7 @@ S:	Supported
 W:	https://projectacrn.org
 F:	Documentation/virt/acrn/
 F:	drivers/virt/acrn/
+F:	include/uapi/linux/acrn.h
 
 AD1889 ALSA SOUND DRIVER
 L:	[email protected]
drivers/virt/acrn/Makefile +1 -1
index 6920ed798aaf..cf8b4ed5e74e 100644
@@ -1,3 +1,3 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_ACRN_HSM)	:= acrn.o
-acrn-y := hsm.o
+acrn-y := hsm.o vm.o
drivers/virt/acrn/acrn_drv.h +21 -1
index 29eedd696327..043ae6840995 100644
@@ -3,16 +3,36 @@
 #ifndef __ACRN_HSM_DRV_H
 #define __ACRN_HSM_DRV_H
 
+#include <linux/acrn.h>
+#include <linux/dev_printk.h>
 #include <linux/types.h>
 
+#include "hypercall.h"
+
 #define ACRN_INVALID_VMID (0xffffU)
 
+#define ACRN_VM_FLAG_DESTROYED		0U
+extern struct list_head acrn_vm_list;
+extern rwlock_t acrn_vm_list_lock;
 /**
  * struct acrn_vm - Properties of ACRN User VM.
+ * @dev:	The struct device this VM belongs to
+ * @list:	Entry within global list of all VMs
  * @vmid:	User VM ID
+ * @vcpu_num:	Number of virtual CPUs in the VM
+ * @flags:	Flags (ACRN_VM_FLAG_*) of the VM. This is VM flag management
+ *		in HSM which is different from the &acrn_vm_creation.vm_flag.
  */
 struct acrn_vm {
-	u16	vmid;
+	struct device		*dev;
+	struct list_head	list;
+	u16			vmid;
+	int			vcpu_num;
+	unsigned long		flags;
 };
 
+struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
+			       struct acrn_vm_creation *vm_param);
+int acrn_vm_destroy(struct acrn_vm *vm);
+
 #endif /* __ACRN_HSM_DRV_H */
drivers/virt/acrn/hsm.c +2 -0
index 28a3052ffa55..bc85a3c14f87 100644
@@ -19,6 +19,8 @@
 
 #include "acrn_drv.h"
 
+static struct miscdevice acrn_dev;
+
 /*
  * When /dev/acrn_hsm is opened, a 'struct acrn_vm' object is created to
  * represent a VM instance and continues to be associated with the opened file
@@ -34,14 +36,77 @@ static int acrn_dev_open(struct inode *inode, struct file *filp)
 		return -ENOMEM;
 
Quoted 6 lines
Quoted 4 lines

drivers/virt/acrn/hsm.c static int acrn_dev_open(struct inode *inode, struct file *filp) · line 39

Quoted 2 lines
drivers/virt/acrn/hsm.c +1 -0
 	vm->vmid = ACRN_INVALID_VMID;
+	vm->dev = get_device(acrn_dev.this_device);

You are grabbing a reference on a static structure?

acrn_dev is static, but acrn_dev.this_device is not.

But you don't control that device, the misc device core does.

Quoted 6 lines
Quoted 2 lines

Ugh, who reviewed this code before it was sent out???

This part was just newly added according to your suggestion.. Please refer to
https://lore.kernel.org/lkml/[email protected]/

What you were doing there was wrong, and what you are doing here is just odd.

Step back please, and describe exactly what you are trying to do. And then explain how grabbing a reference to the device reference count for the misc device is going to help solve that issue.

Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
drivers/virt/acrn/hsm.c +7 -0
+	ret = hcall_create_vm(virt_to_phys(vm_param));
+	if (ret < 0 || vm_param->vmid == ACRN_INVALID_VMID) {
+		dev_err(vm->dev, "Failed to create VM! Error: %d\n", ret);
+		return NULL;
+	}
+
+	vm->vmid = vm_param->vmid;
Quoted 8 lines

drivers/virt/acrn/hsm.c static int acrn_dev_open(struct inode *inode, struct file *filp) · line 50

Quoted 4 lines
drivers/virt/acrn/hsm.c +4 -0
+	vm->vcpu_num = vm_param->vcpu_num;
+
+	write_lock_bh(&acrn_vm_list_lock);
+	list_add(&vm->list, &acrn_vm_list);

Wait, why do you have a global list of devices? Shouldn't that device be tied to the vm structure? Who will be iterating this list that does not have the file handle to start with?

Quoted 4 lines

Active VMs in this list will be used by the I/O requests dispatching tasklet ioreq_tasklet, whose callback function is ioreq_tasklet_handler() in patch 0009. ioreq_tasklet_handler() currently handles the notification interrupt from the hypervisor and dispatches I/O requests to each VMs.

So you need to somehow look through the whole list of devices for every I/O request? That feels really really wrong, why don't you have that pointer in the first place?

Again, step back and describe what you need/desire and then think about how to best solve that. Almost always, a list of objects that you have to iterate over all the time is not the way to do it...

Somedays I think we need an "here's how to do the things you really need to do in a driver" chapter in the Linux Device Driver's book...

thanks,

greg k-h

#8 lore raw

Re: [PATCH v3 06/17] virt: acrn: Introduce VM management interfaces

Hi Shuo and Greg,

Quoted 3 lines · Greg Kroah-Hartman
Quoted 2 lines · Shuo A Liu
Quoted 1 line · Greg Kroah-Hartman

On Wed, Sep 09, 2020 at 05:08:25PM +0800, [email protected] wrote:

...

Quoted 55 lines
Quoted 39 lines
Quoted 39 lines
Quoted 39 lines

/**
* struct acrn_vm - Properties of ACRN User VM.
+ * @dev: The struct device this VM belongs to
+ * @list: Entry within global list of all VMs
* @vmid: User VM ID
+ * @vcpu_num: Number of virtual CPUs in the VM
+ * @flags: Flags (ACRN_VM_FLAG_*) of the VM. This is VM flag management
+ * in HSM which is different from the &acrn_vm_creation.vm_flag.
*/
struct acrn_vm {

drivers/virt/acrn/acrn_drv.h +9 -1
-	u16	vmid;
+	struct device		*dev;
+	struct list_head	list;
+	u16			vmid;
+	int			vcpu_num;
+	unsigned long		flags;
 };
 
+struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
+			       struct acrn_vm_creation *vm_param);
+int acrn_vm_destroy(struct acrn_vm *vm);
+
 #endif /* __ACRN_HSM_DRV_H */
drivers/virt/acrn/hsm.c +2 -0
index 28a3052ffa55..bc85a3c14f87 100644
@@ -19,6 +19,8 @@
 
 #include "acrn_drv.h"
 
+static struct miscdevice acrn_dev;
+
 /*
  * When /dev/acrn_hsm is opened, a 'struct acrn_vm' object is created to
  * represent a VM instance and continues to be associated with the opened file
@@ -34,14 +36,77 @@ static int acrn_dev_open(struct inode *inode, struct file *filp)
 		return -ENOMEM;
 
Quoted 6 lines
Quoted 4 lines

drivers/virt/acrn/hsm.c static int acrn_dev_open(struct inode *inode, struct file *filp) · line 39

Quoted 2 lines
drivers/virt/acrn/hsm.c +1 -0
 	vm->vmid = ACRN_INVALID_VMID;
+	vm->dev = get_device(acrn_dev.this_device);

You are grabbing a reference on a static structure?

acrn_dev is static, but acrn_dev.this_device is not.

But you don't control that device, the misc device core does.

Quoted 6 lines
Quoted 2 lines

Ugh, who reviewed this code before it was sent out???

This part was just newly added according to your suggestion.. Please refer to
https://lore.kernel.org/lkml/[email protected]/

Quoted 7 lines

What you were doing there was wrong, and what you are doing here is just odd.

Step back please, and describe exactly what you are trying to do. And then explain how grabbing a reference to the device reference count for the misc device is going to help solve that issue.

From what I understand this current path was entered after Greg's original suggestion from
https://lore.kernel.org/lkml/[email protected]
to replace the pr_xxx() calls with the more informative dev_xxx() ones.

Perhaps all that is needed to follow the original suggestion is to replace the pr_xxx() calls with dev_xxx(acrn_dev.this_device, ...) ?

(An example of this is drivers/input/serio/userio.c and some others)

The device (acrn_dev.this_device) can be trusted to exist for the lifetime of the driver since it is removed on module unload where the misc device is unregistered.

Reinette

#9 lore raw

Re: [PATCH v3 06/17] virt: acrn: Introduce VM management interfaces

In reply to Greg Kroah-Hartman

Hi Greg,

Quoted 40 lines · Greg Kroah-Hartman
Quoted 39 lines · Shuo A Liu
Quoted 38 lines · Greg Kroah-Hartman
Quoted 37 lines · On Wed, Sep 09, 2020 at 05:08:25PM +0800, [email protected] wrote:

From: Shuo Liu <[email protected]>

The VM management interfaces expose several VM operations to ACRN userspace via ioctls. For example, creating VM, starting VM, destroying VM and so on.

The ACRN Hypervisor needs to exchange data with the ACRN userspace during the VM operations. HSM provides VM operation ioctls to the ACRN userspace and communicates with the ACRN Hypervisor for VM operations via hypercalls.

HSM maintains a list of User VM. Each User VM will be bound to an existing file descriptor of /dev/acrn_hsm. The User VM will be destroyed when the file descriptor is closed.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 .../userspace-api/ioctl/ioctl-number.rst      |  1 +
 MAINTAINERS                                   |  1 +
 drivers/virt/acrn/Makefile                    |  2 +-
 drivers/virt/acrn/acrn_drv.h                  | 22 +++++-
 drivers/virt/acrn/hsm.c                       | 66 ++++++++++++++++
 drivers/virt/acrn/hypercall.h                 | 78 +++++++++++++++++++
 drivers/virt/acrn/vm.c                        | 69 ++++++++++++++++
 include/uapi/linux/acrn.h                     | 56 +++++++++++++
 8 files changed, 293 insertions(+), 2 deletions(-)
 create mode 100644 drivers/virt/acrn/hypercall.h
 create mode 100644 drivers/virt/acrn/vm.c
 create mode 100644 include/uapi/linux/acrn.h

[...]

Quoted 21 lines
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
drivers/virt/acrn/vm.c new file +7 -0
+	ret = hcall_create_vm(virt_to_phys(vm_param));
+	if (ret < 0 || vm_param->vmid == ACRN_INVALID_VMID) {
+		dev_err(vm->dev, "Failed to create VM! Error: %d\n", ret);
+		return NULL;
+	}
+
+	vm->vmid = vm_param->vmid;
Quoted 8 lines

drivers/virt/acrn/vm.c line 40

Quoted 4 lines
drivers/virt/acrn/vm.c new file +4 -0
+	vm->vcpu_num = vm_param->vcpu_num;
+
+	write_lock_bh(&acrn_vm_list_lock);
+	list_add(&vm->list, &acrn_vm_list);

Wait, why do you have a global list of devices? Shouldn't that device be tied to the vm structure? Who will be iterating this list that does not have the file handle to start with?

Quoted 4 lines

Active VMs in this list will be used by the I/O requests dispatching tasklet ioreq_tasklet, whose callback function is ioreq_tasklet_handler() in patch 0009. ioreq_tasklet_handler() currently handles the notification interrupt from the hypervisor and dispatches I/O requests to each VMs.

Quoted 7 lines

So you need to somehow look through the whole list of devices for every I/O request? That feels really really wrong, why don't you have that pointer in the first place?

Again, step back and describe what you need/desire and then think about how to best solve that. Almost always, a list of objects that you have to iterate over all the time is not the way to do it...

Each VM has a shared buffer for I/O requests passing with the hypervisor. Currently, the hypervisor doesn't indicate the VMs which has pending I/O requests. So when kernel get the notification interrupt, it search all VMs' shared buffer and dispatch the pending I/O requests.

The current I/O requests dispatching implementation uses one global tasklet (be scheduled in the hypervisor notification interrupt), so it needs to iterate all VMs to do the dispatching.

Each VM has a dedicated hypervisor notification interrupt vector might be suited (a vector can be linked with a VM). The disadvantage is that it might occupy many vectors.

Looking forward to more suggestions. Thanks very much.

Quoted 3 lines

Somedays I think we need an "here's how to do the things you really need to do in a driver" chapter in the Linux Device Driver's book..

That will be great. :)

Thanks
shuo

#10 lore raw

PATCH v3 7/17 virt: acrn: Introduce an ioctl to set vCPU registers state

In reply to [email protected]

From: Shuo Liu <[email protected]>

A virtual CPU of User VM has different context due to the different registers state. ACRN userspace needs to set the virtual CPU registers state (e.g. giving a initial registers state to a virtual BSP of a User VM).

HSM provides an ioctl ACRN_IOCTL_SET_VCPU_REGS to do the virtual CPU registers state setting. The ioctl passes the registers state from ACRN userspace to the hypervisor directly.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 drivers/virt/acrn/hsm.c       | 14 +++++++
 drivers/virt/acrn/hypercall.h | 13 +++++++
 include/uapi/linux/acrn.h     | 71 +++++++++++++++++++++++++++++++++++
 3 files changed, 98 insertions(+)
drivers/virt/acrn/hsm.c +14 -0
index bc85a3c14f87..aa6e9999d147 100644
@@ -9,6 +9,7 @@
  *	Yakui Zhao <[email protected]>
  */
 
+#include <linux/io.h>
 #include <linux/miscdevice.h>
 #include <linux/mm.h>
 #include <linux/module.h>
@@ -46,6 +47,7 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 {
 	struct acrn_vm *vm = filp->private_data;
 	struct acrn_vm_creation *vm_param;
+	struct acrn_vcpu_regs *cpu_regs;
 	int ret = 0;
 
 	if (vm->vmid == ACRN_INVALID_VMID && cmd != ACRN_IOCTL_CREATE_VM) {
@@ -93,6 +95,18 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 	case ACRN_IOCTL_DESTROY_VM:
 		ret = acrn_vm_destroy(vm);
 		break;
+	case ACRN_IOCTL_SET_VCPU_REGS:
+		cpu_regs = memdup_user((void __user *)ioctl_param,
+				       sizeof(struct acrn_vcpu_regs));
+		if (IS_ERR(cpu_regs))
+			return PTR_ERR(cpu_regs);
+
+		ret = hcall_set_vcpu_regs(vm->vmid, virt_to_phys(cpu_regs));
+		if (ret < 0)
+			dev_err(vm->dev, "Failed to set regs state of VM%u!\n",
+				vm->vmid);
+		kfree(cpu_regs);
+		break;
 	default:
 		dev_warn(vm->dev, "Unknown IOCTL 0x%x!\n", cmd);
 		ret = -ENOTTY;
drivers/virt/acrn/hypercall.h +13 -0
index 426b66cadb1f..f29cfae08862 100644
@@ -19,6 +19,7 @@
 #define HC_START_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x02)
 #define HC_PAUSE_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x03)
 #define HC_RESET_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x05)
+#define HC_SET_VCPU_REGS		_HC_ID(HC_ID, HC_ID_VM_BASE + 0x06)
 
 /**
  * hcall_create_vm() - Create a User VM
@@ -75,4 +76,16 @@ static inline long hcall_reset_vm(u64 vmid)
 	return acrn_hypercall1(HC_RESET_VM, vmid);
 }
 
+/**
+ * hcall_set_vcpu_regs() - Set up registers of virtual CPU of a User VM
+ * @vmid:	User VM ID
+ * @regs_state:	Service VM GPA of registers state
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_set_vcpu_regs(u64 vmid, u64 regs_state)
+{
+	return acrn_hypercall2(HC_SET_VCPU_REGS, vmid, regs_state);
+}
+
 #endif /* __ACRN_HSM_HYPERCALL_H */
include/uapi/linux/acrn.h +71 -0

82 lines not on this page · raw message

Signature · 1 line
2.28.0
#11 lore raw

PATCH v3 9/17 virt: acrn: Introduce I/O request management

In reply to [email protected]

From: Shuo Liu <[email protected]>

An I/O request of a User VM, which is constructed by the hypervisor, is distributed by the ACRN Hypervisor Service Module to an I/O client corresponding to the address range of the I/O request.

For each User VM, there is a shared 4-KByte memory region used for I/O requests communication between the hypervisor and Service VM. An I/O request is a 256-byte structure buffer, which is 'struct acrn_io_request', that is filled by an I/O handler of the hypervisor when a trapped I/O access happens in a User VM. ACRN userspace in the Service VM first allocates a 4-KByte page and passes the GPA (Guest Physical Address) of the buffer to the hypervisor. The buffer is used as an array of 16 I/O request slots with each I/O request slot being 256 bytes. This array is indexed by vCPU ID.

An I/O client, which is 'struct acrn_ioreq_client', is responsible for handling User VM I/O requests whose accessed GPA falls in a certain range. Multiple I/O clients can be associated with each User VM. There is a special client associated with each User VM, called the default client, that handles all I/O requests that do not fit into the range of any other I/O clients. The ACRN userspace acts as the default client for each User VM.

The state transitions of a ACRN I/O request are as follows.

FREE -> PENDING -> PROCESSING -> COMPLETE -> FREE -> ...

FREE: this I/O request slot is empty
PENDING: a valid I/O request is pending in this slot
PROCESSING: the I/O request is being processed
COMPLETE: the I/O request has been processed

An I/O request in COMPLETE or FREE state is owned by the hypervisor. HSM and ACRN userspace are in charge of processing the others.

The processing flow of I/O requests are listed as following:

a) The I/O handler of the hypervisor will fill an I/O request with PENDING state when a trapped I/O access happens in a User VM.
b) The hypervisor makes an upcall, which is a notification interrupt, to the Service VM.
c) The upcall handler schedules a tasklet to dispatch I/O requests.
d) The tasklet looks for the PENDING I/O requests, assigns them to different registered clients based on the address of the I/O accesses, updates their state to PROCESSING, and notifies the corresponding client to handle.
e) The notified client handles the assigned I/O requests.
f) The HSM updates I/O requests states to COMPLETE and notifies the hypervisor of the completion via hypercalls.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 drivers/virt/acrn/Makefile    |   2 +-
 drivers/virt/acrn/acrn_drv.h  |  80 ++++++
 drivers/virt/acrn/hsm.c       |  32 ++-
 drivers/virt/acrn/hypercall.h |  28 ++
 drivers/virt/acrn/ioreq.c     | 506 ++++++++++++++++++++++++++++++++++
 drivers/virt/acrn/vm.c        |  10 +
 include/uapi/linux/acrn.h     | 134 +++++++++
 7 files changed, 789 insertions(+), 3 deletions(-)
 create mode 100644 drivers/virt/acrn/ioreq.c
drivers/virt/acrn/Makefile +1 -1
index 38bc44b6edcd..21721cbf6a80 100644
@@ -1,3 +1,3 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_ACRN_HSM)	:= acrn.o
-acrn-y := hsm.o vm.o mm.o
+acrn-y := hsm.o vm.o mm.o ioreq.o
drivers/virt/acrn/acrn_drv.h +80 -0

115 lines not on this page · raw message

drivers/virt/acrn/hsm.c +30 -2
index a0ddb17ea284..b7cae2025851 100644
@@ -48,6 +48,7 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 	struct acrn_vm *vm = filp->private_data;
 	struct acrn_vm_creation *vm_param;
 	struct acrn_vcpu_regs *cpu_regs;
+	struct acrn_ioreq_notify notify;
 	struct acrn_vm_memmap memmap;
 	int ret = 0;
 
@@ -122,6 +123,29 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 
 		ret = acrn_vm_memseg_unmap(vm, &memmap);
 		break;
+	case ACRN_IOCTL_CREATE_IOREQ_CLIENT:
+		if (vm->default_client)
+			return -EEXIST;
+		if (!acrn_ioreq_client_create(vm, NULL, NULL, true, "acrndm"))
+			ret = -EFAULT;
+		break;
+	case ACRN_IOCTL_DESTROY_IOREQ_CLIENT:
+		if (vm->default_client)
+			acrn_ioreq_client_destroy(vm->default_client);
+		break;
+	case ACRN_IOCTL_ATTACH_IOREQ_CLIENT:
+		if (vm->default_client)
+			ret = acrn_ioreq_client_wait(vm->default_client);
+		break;
+	case ACRN_IOCTL_NOTIFY_REQUEST_FINISH:
+		if (copy_from_user(&notify, (void __user *)ioctl_param,
+				   sizeof(struct acrn_ioreq_notify)))
+			return -EFAULT;
+		ret = acrn_ioreq_request_default_complete(vm, notify.vcpu);
+		break;
+	case ACRN_IOCTL_CLEAR_VM_IOREQ:
+		acrn_ioreq_request_clear(vm);
+		break;
 	default:
 		dev_warn(vm->dev, "Unknown IOCTL 0x%x!\n", cmd);
 		ret = -ENOTTY;
@@ -164,14 +188,18 @@ static int __init hsm_init(void)
 		return -EPERM;
 
 	ret = misc_register(&acrn_dev);
-	if (ret)
+	if (ret) {
 		pr_err("Create misc dev failed!\n");
+		return ret;
+	}
 
-	return ret;
+	acrn_ioreq_intr_setup();
+	return 0;
 }
 
 static void __exit hsm_exit(void)
 {
+	acrn_ioreq_intr_remove();
 	misc_deregister(&acrn_dev);
 }
 module_init(hsm_init);
drivers/virt/acrn/hypercall.h +28 -0
index a1a70a071713..5eba29e3ed38 100644
@@ -21,6 +21,10 @@
 #define HC_RESET_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x05)
 #define HC_SET_VCPU_REGS		_HC_ID(HC_ID, HC_ID_VM_BASE + 0x06)
 
+#define HC_ID_IOREQ_BASE		0x30UL
+#define HC_SET_IOREQ_BUFFER		_HC_ID(HC_ID, HC_ID_IOREQ_BASE + 0x00)
+#define HC_NOTIFY_REQUEST_FINISH	_HC_ID(HC_ID, HC_ID_IOREQ_BASE + 0x01)
+
 #define HC_ID_MEM_BASE			0x40UL
 #define HC_VM_SET_MEMORY_REGIONS	_HC_ID(HC_ID, HC_ID_MEM_BASE + 0x02)
 
@@ -91,6 +95,30 @@ static inline long hcall_set_vcpu_regs(u64 vmid, u64 regs_state)
 	return acrn_hypercall2(HC_SET_VCPU_REGS, vmid, regs_state);
 }
 
+/**
+ * hcall_set_ioreq_buffer() - Set up the shared buffer for I/O Requests.
+ * @vmid:	User VM ID
+ * @buffer:	Service VM GPA of the shared buffer
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_set_ioreq_buffer(u64 vmid, u64 buffer)
+{
+	return acrn_hypercall2(HC_SET_IOREQ_BUFFER, vmid, buffer);
+}
+
+/**
+ * hcall_notify_req_finish() - Notify ACRN Hypervisor of I/O request completion.
+ * @vmid:	User VM ID
+ * @vcpu:	The vCPU which initiated the I/O request
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_notify_req_finish(u64 vmid, u64 vcpu)
+{
+	return acrn_hypercall2(HC_NOTIFY_REQUEST_FINISH, vmid, vcpu);
+}
+
 /**
  * hcall_set_memory_regions() - Inform the hypervisor to set up EPT mappings
  * @regions_pa:	Service VM GPA of &struct vm_memory_region_batch
drivers/virt/acrn/ioreq.c new file +506 -0

506 lines not on this page · raw message

drivers/virt/acrn/vm.c +10 -0
index 044c0615bd87..12bd5de0a676 100644
@@ -34,9 +34,17 @@ struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
 	}
 
 	mutex_init(&vm->regions_mapping_lock);
+	INIT_LIST_HEAD(&vm->ioreq_clients);
+	spin_lock_init(&vm->ioreq_clients_lock);
 	vm->vmid = vm_param->vmid;
 	vm->vcpu_num = vm_param->vcpu_num;
 
+	if (acrn_ioreq_init(vm, vm_param->ioreq_buf) < 0) {
+		hcall_destroy_vm(vm_param->vmid);
+		vm->vmid = ACRN_INVALID_VMID;
+		return NULL;
+	}
+
 	write_lock_bh(&acrn_vm_list_lock);
 	list_add(&vm->list, &acrn_vm_list);
 	write_unlock_bh(&acrn_vm_list_lock);
@@ -58,6 +66,8 @@ int acrn_vm_destroy(struct acrn_vm *vm)
 	list_del_init(&vm->list);
 	write_unlock_bh(&acrn_vm_list_lock);
 
+	acrn_ioreq_deinit(vm);
+
 	ret = hcall_destroy_vm(vm->vmid);
 	if (ret < 0) {
 		dev_err(vm->dev, "Failed to destroy VM %u\n", vm->vmid);
include/uapi/linux/acrn.h +134 -0

146 lines not on this page · raw message

Signature · 1 line
2.28.0
#12 lore raw

PATCH v3 10/17 virt: acrn: Introduce PCI configuration space PIO accesses combiner

In reply to [email protected]

From: Shuo Liu <[email protected]>

A User VM can access its virtual PCI configuration spaces via port IO approach, which has two following steps:
1) writes address into port 0xCF8
2) put/get data in/from port 0xCFC

To distribute a complete PCI configuration space access one time, HSM need to combine such two accesses together.

Combine two paired PIO I/O requests into one PCI I/O request and continue the I/O request distribution.

---
 drivers/virt/acrn/acrn_drv.h |  2 +
 drivers/virt/acrn/ioreq.c    | 76 ++++++++++++++++++++++++++++++++++++
 include/uapi/linux/acrn.h    | 15 +++++++
 3 files changed, 93 insertions(+)
drivers/virt/acrn/acrn_drv.h +2 -0
index 9a8e3bb007b1..4f86feaa5e1b 100644
@@ -154,6 +154,7 @@ extern rwlock_t acrn_vm_list_lock;
  * @default_client:		The default I/O request client
  * @ioreq_buf:			I/O request shared buffer
  * @ioreq_page:			The page of the I/O request shared buffer
+ * @pci_conf_addr:		Address of a PCI configuration access emulation
  */
 struct acrn_vm {
 	struct device			*dev;
@@ -169,6 +170,7 @@ struct acrn_vm {
 	struct acrn_ioreq_client	*default_client;
 	struct acrn_io_request_buffer	*ioreq_buf;
 	struct page			*ioreq_page;
+	u32				pci_conf_addr;
 };
 
 struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
drivers/virt/acrn/ioreq.c +76 -0

88 lines not on this page · raw message

include/uapi/linux/acrn.h +15 -0
index 8eb687f1482c..31cf0fd73bcc 100644
@@ -20,6 +20,7 @@
 
 #define ACRN_IOREQ_TYPE_PORTIO		0
 #define ACRN_IOREQ_TYPE_MMIO		1
+#define ACRN_IOREQ_TYPE_PCICFG		2
 
 #define ACRN_IOREQ_DIR_READ		0
 #define ACRN_IOREQ_DIR_WRITE		1
@@ -40,6 +41,18 @@ struct acrn_pio_request {
 	__u32	value;
 } __attribute__((aligned(8)));
 
+/* Need keep same header fields with pio_request */
+struct acrn_pci_request {
+	__u32	direction;
+	__u32	reserved[3];
+	__u64	size;
+	__u32	value;
+	__u32	bus;
+	__u32	dev;
+	__u32	func;
+	__u32	reg;
+} __attribute__((aligned(8)));
+
 /**
  * struct acrn_io_request - 256-byte ACRN I/O request
  * @type:		Type of this request (ACRN_IOREQ_TYPE_*).
@@ -48,6 +61,7 @@ struct acrn_pio_request {
  * @reserved0:		Reserved fields.
  * @reqs:		Union of different types of request. Byte offset: 64.
  * @reqs.pio_request:	PIO request data of the I/O request.
+ * @reqs.pci_request:	PCI configuration space request data of the I/O request.
  * @reqs.mmio_request:	MMIO request data of the I/O request.
  * @reqs.data:		Raw data of the I/O request.
  * @reserved1:		Reserved fields.
@@ -107,6 +121,7 @@ struct acrn_io_request {
 	__u32	reserved0[14];
 	union {
 		struct acrn_pio_request		pio_request;
+		struct acrn_pci_request		pci_request;
 		struct acrn_mmio_request	mmio_request;
 		__u64				data[8];
 	} reqs;
Signature · 1 line
2.28.0
#13 lore raw

PATCH v3 12/17 virt: acrn: Introduce interrupt injection interfaces

In reply to [email protected]

From: Shuo Liu <[email protected]>

ACRN userspace need to inject virtual interrupts into a User VM in devices emulation.

HSM needs provide interfaces to do so.

Introduce following interrupt injection interfaces:

ioctl ACRN_IOCTL_SET_IRQLINE:
Pass data from userspace to the hypervisor, and inform the hypervisor to inject a virtual IOAPIC GSI interrupt to a User VM.

ioctl ACRN_IOCTL_INJECT_MSI:
Pass data struct acrn_msi_entry from userspace to the hypervisor, and inform the hypervisor to inject a virtual MSI to a User VM.

ioctl ACRN_IOCTL_VM_INTR_MONITOR:
Set a 4-Kbyte aligned shared page for statistics information of interrupts of a User VM.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 drivers/virt/acrn/acrn_drv.h  |  4 ++++
 drivers/virt/acrn/hsm.c       | 35 ++++++++++++++++++++++++++++++
 drivers/virt/acrn/hypercall.h | 41 +++++++++++++++++++++++++++++++++++
 drivers/virt/acrn/vm.c        | 35 ++++++++++++++++++++++++++++++
 include/uapi/linux/acrn.h     | 17 +++++++++++++++
 5 files changed, 132 insertions(+)
drivers/virt/acrn/acrn_drv.h +4 -0
index 4f86feaa5e1b..84d9a8f80101 100644
@@ -155,6 +155,7 @@ extern rwlock_t acrn_vm_list_lock;
  * @ioreq_buf:			I/O request shared buffer
  * @ioreq_page:			The page of the I/O request shared buffer
  * @pci_conf_addr:		Address of a PCI configuration access emulation
+ * @monitor_page:		Page of interrupt statistics of User VM
  */
 struct acrn_vm {
 	struct device			*dev;
@@ -171,6 +172,7 @@ struct acrn_vm {
 	struct acrn_io_request_buffer	*ioreq_buf;
 	struct page			*ioreq_page;
 	u32				pci_conf_addr;
+	struct page			*monitor_page;
 };
 
 struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
@@ -197,4 +199,6 @@ struct acrn_ioreq_client *acrn_ioreq_client_create(struct acrn_vm *vm,
 						   const char *name);
 void acrn_ioreq_client_destroy(struct acrn_ioreq_client *client);
 
+int acrn_msi_inject(struct acrn_vm *vm, u64 msi_addr, u64 msi_data);
+
 #endif /* __ACRN_HSM_DRV_H */
drivers/virt/acrn/hsm.c +35 -0
index 6fb8032b007c..d6e1f60126a8 100644
@@ -51,7 +51,9 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 	struct acrn_ioreq_notify notify;
 	struct acrn_ptdev_irq *irq_info;
 	struct acrn_vm_memmap memmap;
+	struct acrn_msi_entry *msi;
 	struct acrn_pcidev *pcidev;
+	struct page *page;
 	int ret = 0;
 
 	if (vm->vmid == ACRN_INVALID_VMID && cmd != ACRN_IOCTL_CREATE_VM) {
@@ -169,6 +171,39 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 			dev_err(vm->dev, "Failed to reset intr for ptdev!\n");
 		kfree(irq_info);
 		break;
+	case ACRN_IOCTL_SET_IRQLINE:
+		ret = hcall_set_irqline(vm->vmid, ioctl_param);
+		if (ret < 0)
+			dev_err(vm->dev, "Failed to set interrupt line!\n");
+		break;
+	case ACRN_IOCTL_INJECT_MSI:
+		msi = memdup_user((void __user *)ioctl_param,
+				  sizeof(struct acrn_msi_entry));
+		if (IS_ERR(msi))
+			return PTR_ERR(msi);
+
+		ret = hcall_inject_msi(vm->vmid, virt_to_phys(msi));
+		if (ret < 0)
+			dev_err(vm->dev, "Failed to inject MSI!\n");
+		kfree(msi);
+		break;
+	case ACRN_IOCTL_VM_INTR_MONITOR:
+		ret = get_user_pages_fast(ioctl_param, 1, FOLL_WRITE, &page);
+		if (unlikely(ret != 1)) {
+			dev_err(vm->dev, "Failed to pin intr hdr buffer!\n");
+			return -ENOMEM;
+		}
+
+		ret = hcall_vm_intr_monitor(vm->vmid, page_to_phys(page));
+		if (ret < 0) {
+			put_page(page);
+			dev_err(vm->dev, "Failed to monitor intr data!\n");
+			return ret;
+		}
+		if (vm->monitor_page)
+			put_page(vm->monitor_page);
+		vm->monitor_page = page;
+		break;
 	case ACRN_IOCTL_CREATE_IOREQ_CLIENT:
 		if (vm->default_client)
 			return -EEXIST;
drivers/virt/acrn/hypercall.h +41 -0

53 lines not on this page · raw message

drivers/virt/acrn/vm.c +35 -0
index 12bd5de0a676..c1fefd92e571 100644
@@ -67,6 +67,10 @@ int acrn_vm_destroy(struct acrn_vm *vm)
 	write_unlock_bh(&acrn_vm_list_lock);
 
 	acrn_ioreq_deinit(vm);
+	if (vm->monitor_page) {
+		put_page(vm->monitor_page);
+		vm->monitor_page = NULL;
+	}
 
 	ret = hcall_destroy_vm(vm->vmid);
 	if (ret < 0) {
@@ -81,3 +85,34 @@ int acrn_vm_destroy(struct acrn_vm *vm)
 	vm->vmid = ACRN_INVALID_VMID;
 	return 0;
 }
+
+/**
+ * acrn_inject_msi() - Inject a MSI interrupt into a User VM
+ * @vm:		User VM
+ * @msi_addr:	The MSI address
+ * @msi_data:	The MSI data
+ *
+ * Return: 0 on success, <0 on error
+ */
+int acrn_msi_inject(struct acrn_vm *vm, u64 msi_addr, u64 msi_data)
+{
+	struct acrn_msi_entry *msi;
+	int ret;
+
+	/* might be used in interrupt context, so use GFP_ATOMIC */
+	msi = kzalloc(sizeof(*msi), GFP_ATOMIC);
+	if (!msi)
+		return -ENOMEM;
+
+	/*
+	 * msi_addr: addr[19:12] with dest vcpu id
+	 * msi_data: data[7:0] with vector
+	 */
+	msi->msi_addr = msi_addr;
+	msi->msi_data = msi_data;
+	ret = hcall_inject_msi(vm->vmid, virt_to_phys(msi));
+	if (ret < 0)
+		dev_err(vm->dev, "Failed to inject MSI to VM %u!\n", vm->vmid);
+	kfree(msi);
+	return ret;
+}
include/uapi/linux/acrn.h +17 -0
index 893389babbcb..7764459e130c 100644
@@ -343,6 +343,16 @@ struct acrn_pcidev {
 	__u32	reserved[6];
 } __attribute__((aligned(8)));
 
+/**
+ * struct acrn_msi_entry - Info for injecting a MSI interrupt to a VM
+ * @msi_addr:	MSI addr[19:12] with dest vCPU ID
+ * @msi_data:	MSI data[7:0] with vector
+ */
+struct acrn_msi_entry {
+	__u64	msi_addr;
+	__u64	msi_data;
+};
+
 /* The ioctl type, documented in ioctl-number.rst */
 #define ACRN_IOCTL_TYPE			0xA2
 
@@ -362,6 +372,13 @@ struct acrn_pcidev {
 #define ACRN_IOCTL_SET_VCPU_REGS	\
 	_IOW(ACRN_IOCTL_TYPE, 0x16, struct acrn_vcpu_regs)
 
+#define ACRN_IOCTL_INJECT_MSI		\
+	_IOW(ACRN_IOCTL_TYPE, 0x23, struct acrn_msi_entry)
+#define ACRN_IOCTL_VM_INTR_MONITOR	\
+	_IOW(ACRN_IOCTL_TYPE, 0x24, unsigned long)
+#define ACRN_IOCTL_SET_IRQLINE		\
+	_IOW(ACRN_IOCTL_TYPE, 0x25, __u64)
+
 #define ACRN_IOCTL_NOTIFY_REQUEST_FINISH \
 	_IOW(ACRN_IOCTL_TYPE, 0x31, struct acrn_ioreq_notify)
 #define ACRN_IOCTL_CREATE_IOREQ_CLIENT	\
Signature · 1 line
2.28.0
#14 lore raw

PATCH v3 13/17 virt: acrn: Introduce interfaces to query C-states and P-states allowed by hypervisor

In reply to [email protected]

From: Shuo Liu <[email protected]>

The C-states and P-states data are used to support CPU power management. The hypervisor controls C-states and P-states for a User VM.

ACRN userspace need to query the data from the hypervisor to build ACPI tables for a User VM.

HSM provides ioctls for ACRN userspace to query C-states and P-states data obtained from the hypervisor.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 drivers/virt/acrn/hsm.c       | 69 +++++++++++++++++++++++++++++++++++
 drivers/virt/acrn/hypercall.h | 12 ++++++
 include/uapi/linux/acrn.h     | 35 ++++++++++++++++++
 3 files changed, 116 insertions(+)
drivers/virt/acrn/hsm.c +69 -0
index d6e1f60126a8..7c0e410cf32d 100644
@@ -42,6 +42,67 @@ static int acrn_dev_open(struct inode *inode, struct file *filp)
 	return 0;
 }
 
+static int pmcmd_ioctl(u64 cmd, void __user *uptr)
+{
+	struct acrn_pstate_data *px_data;
+	struct acrn_cstate_data *cx_data;
+	u64 *pm_info;
+	int ret = 0;
+
+	switch (cmd & PMCMD_TYPE_MASK) {
+	case ACRN_PMCMD_GET_PX_CNT:
+	case ACRN_PMCMD_GET_CX_CNT:
+		pm_info = kmalloc(sizeof(u64), GFP_KERNEL);
+		if (!pm_info)
+			return -ENOMEM;
+
+		ret = hcall_get_cpu_state(cmd, virt_to_phys(pm_info));
+		if (ret < 0) {
+			kfree(pm_info);
+			break;
+		}
+
+		if (copy_to_user(uptr, pm_info, sizeof(u64)))
+			ret = -EFAULT;
+		kfree(pm_info);
+		break;
+	case ACRN_PMCMD_GET_PX_DATA:
+		px_data = kmalloc(sizeof(*px_data), GFP_KERNEL);
+		if (!px_data)
+			return -ENOMEM;
+
+		ret = hcall_get_cpu_state(cmd, virt_to_phys(px_data));
+		if (ret < 0) {
+			kfree(px_data);
+			break;
+		}
+
+		if (copy_to_user(uptr, px_data, sizeof(*px_data)))
+			ret = -EFAULT;
+		kfree(px_data);
+		break;
+	case ACRN_PMCMD_GET_CX_DATA:
+		cx_data = kmalloc(sizeof(*cx_data), GFP_KERNEL);
+		if (!cx_data)
+			return -ENOMEM;
+
+		ret = hcall_get_cpu_state(cmd, virt_to_phys(cx_data));
+		if (ret < 0) {
+			kfree(cx_data);
+			break;
+		}
+
+		if (copy_to_user(uptr, cx_data, sizeof(*cx_data)))
+			ret = -EFAULT;
+		kfree(cx_data);
+		break;
+	default:
+		break;
+	}
+
+	return ret;
+}
+
 static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 			   unsigned long ioctl_param)
 {
@@ -54,6 +115,7 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 	struct acrn_msi_entry *msi;
 	struct acrn_pcidev *pcidev;
 	struct page *page;
+	u64 cstate_cmd;
 	int ret = 0;
 
 	if (vm->vmid == ACRN_INVALID_VMID && cmd != ACRN_IOCTL_CREATE_VM) {
@@ -227,6 +289,13 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 	case ACRN_IOCTL_CLEAR_VM_IOREQ:
 		acrn_ioreq_request_clear(vm);
 		break;
+	case ACRN_IOCTL_PM_GET_CPU_STATE:
+		if (copy_from_user(&cstate_cmd, (void *)ioctl_param,
+				   sizeof(cstate_cmd)))
+			return -EFAULT;
+
+		ret = pmcmd_ioctl(cstate_cmd, (void __user *)ioctl_param);
+		break;
 	default:
 		dev_warn(vm->dev, "Unknown IOCTL 0x%x!\n", cmd);
 		ret = -ENOTTY;
drivers/virt/acrn/hypercall.h +12 -0
index a8813397a3fe..e640632366f0 100644
@@ -39,6 +39,9 @@
 #define HC_ASSIGN_PCIDEV		_HC_ID(HC_ID, HC_ID_PCI_BASE + 0x05)
 #define HC_DEASSIGN_PCIDEV		_HC_ID(HC_ID, HC_ID_PCI_BASE + 0x06)
 
+#define HC_ID_PM_BASE			0x80UL
+#define HC_PM_GET_CPU_STATE		_HC_ID(HC_ID, HC_ID_PM_BASE + 0x00)
+
 /**
  * hcall_create_vm() - Create a User VM
  * @vminfo:	Service VM GPA of info of User VM creation
@@ -225,4 +228,13 @@ static inline long hcall_reset_ptdev_intr(u64 vmid, u64 irq)
 	return acrn_hypercall2(HC_RESET_PTDEV_INTR, vmid, irq);
 }
 
+/*
+ * hcall_get_cpu_state() - Get P-states and C-states info from the hypervisor
+ * @state:	Service VM GPA of buffer of P-states and C-states
+ */
+static inline long hcall_get_cpu_state(u64 cmd, u64 state)
+{
+	return acrn_hypercall2(HC_PM_GET_CPU_STATE, cmd, state);
+}
+
 #endif /* __ACRN_HSM_HYPERCALL_H */
include/uapi/linux/acrn.h +35 -0
index 7764459e130c..9fed7209a8ef 100644
@@ -353,6 +353,38 @@ struct acrn_msi_entry {
 	__u64	msi_data;
 };
 
+struct acrn_acpi_generic_address {
+	__u8	space_id;
+	__u8	bit_width;
+	__u8	bit_offset;
+	__u8	access_size;
+	__u64	address;
+} __attribute__ ((__packed__));
+
+struct acrn_cstate_data {
+	struct acrn_acpi_generic_address	cx_reg;
+	__u8					type;
+	__u32					latency;
+	__u64					power;
+} __attribute__((aligned(8)));
+
+struct acrn_pstate_data {
+	__u64	core_frequency;
+	__u64	power;
+	__u64	transition_latency;
+	__u64	bus_master_latency;
+	__u64	control;
+	__u64	status;
+} __attribute__((aligned(8)));
+
+#define PMCMD_TYPE_MASK		0x000000ff
+enum acrn_pm_cmd_type {
+	ACRN_PMCMD_GET_PX_CNT,
+	ACRN_PMCMD_GET_PX_DATA,
+	ACRN_PMCMD_GET_CX_CNT,
+	ACRN_PMCMD_GET_CX_DATA,
+};
+
 /* The ioctl type, documented in ioctl-number.rst */
 #define ACRN_IOCTL_TYPE			0xA2
 
@@ -404,4 +436,7 @@ struct acrn_msi_entry {
 #define ACRN_IOCTL_DEASSIGN_PCIDEV	\
 	_IOW(ACRN_IOCTL_TYPE, 0x56, struct acrn_pcidev)
 
+#define ACRN_IOCTL_PM_GET_CPU_STATE	\
+	_IOWR(ACRN_IOCTL_TYPE, 0x60, __u64)
+
 #endif /* _UAPI_ACRN_H */
Signature · 1 line
2.28.0
#15 lore raw

PATCH v3 14/17 virt: acrn: Introduce I/O ranges operation interfaces

In reply to [email protected]

From: Shuo Liu <[email protected]>

An I/O request of a User VM, which is constructed by hypervisor, is distributed by the ACRN Hypervisor Service Module to an I/O client corresponding to the address range of the I/O request.

I/O client maintains a list of address ranges. Introduce acrn_ioreq_range_{add,del}() to manage these address ranges.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 drivers/virt/acrn/acrn_drv.h |  4 +++
 drivers/virt/acrn/ioreq.c    | 60 ++++++++++++++++++++++++++++++++++++
 2 files changed, 64 insertions(+)
drivers/virt/acrn/acrn_drv.h +4 -0
index 84d9a8f80101..cde12eaa492e 100644
@@ -198,6 +198,10 @@ struct acrn_ioreq_client *acrn_ioreq_client_create(struct acrn_vm *vm,
 						   void *data, bool is_default,
 						   const char *name);
 void acrn_ioreq_client_destroy(struct acrn_ioreq_client *client);
+int acrn_ioreq_range_add(struct acrn_ioreq_client *client,
+			 u32 type, u64 start, u64 end);
+void acrn_ioreq_range_del(struct acrn_ioreq_client *client,
+			  u32 type, u64 start, u64 end);
 
 int acrn_msi_inject(struct acrn_vm *vm, u64 msi_addr, u64 msi_data);
 
drivers/virt/acrn/ioreq.c +60 -0

66 lines not on this page · raw message

Signature · 1 line
2.28.0
#16 lore raw

PATCH v3 15/17 virt: acrn: Introduce ioeventfd

In reply to [email protected]

From: Shuo Liu <[email protected]>

ioeventfd is a mechanism to register PIO/MMIO regions to trigger an eventfd signal when written to by a User VM. ACRN userspace can register any arbitrary I/O address with a corresponding eventfd and then pass the eventfd to a specific end-point of interest for handling.

Vhost is a kernel-level virtio server which uses eventfd for signalling. To support vhost on ACRN, ioeventfd is introduced in HSM.

A new I/O client dedicated to ioeventfd is associated with a User VM during VM creation. HSM provides ioctls to associate an I/O region with a eventfd. The I/O client signals a eventfd once its corresponding I/O region is matched with an I/O request.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 drivers/virt/acrn/Kconfig     |   1 +
 drivers/virt/acrn/Makefile    |   2 +-
 drivers/virt/acrn/acrn_drv.h  |  10 ++
 drivers/virt/acrn/hsm.c       |   8 +
 drivers/virt/acrn/ioeventfd.c | 273 ++++++++++++++++++++++++++++++++++
 drivers/virt/acrn/vm.c        |   2 +
 include/uapi/linux/acrn.h     |  29 ++++
 7 files changed, 324 insertions(+), 1 deletion(-)
 create mode 100644 drivers/virt/acrn/ioeventfd.c
drivers/virt/acrn/Kconfig +1 -0
index 36c80378c30c..3e1a61c9d8d8 100644
@@ -2,6 +2,7 @@
 config ACRN_HSM
 	tristate "ACRN Hypervisor Service Module"
 	depends on ACRN_GUEST
+	select EVENTFD
 	help
 	  ACRN Hypervisor Service Module (HSM) is a kernel module which
 	  communicates with ACRN userspace through ioctls and talks to
drivers/virt/acrn/Makefile +1 -1
index 21721cbf6a80..755b583b32ca 100644
@@ -1,3 +1,3 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_ACRN_HSM)	:= acrn.o
-acrn-y := hsm.o vm.o mm.o ioreq.o
+acrn-y := hsm.o vm.o mm.o ioreq.o ioeventfd.o
drivers/virt/acrn/acrn_drv.h +10 -0
index cde12eaa492e..4cb8a61d4ccd 100644
@@ -156,6 +156,9 @@ extern rwlock_t acrn_vm_list_lock;
  * @ioreq_page:			The page of the I/O request shared buffer
  * @pci_conf_addr:		Address of a PCI configuration access emulation
  * @monitor_page:		Page of interrupt statistics of User VM
+ * @ioeventfds_lock:		Lock to protect ioeventfds list
+ * @ioeventfds:			List to link all hsm_ioeventfd
+ * @ioeventfd_client:		I/O client for ioeventfds of the VM
  */
 struct acrn_vm {
 	struct device			*dev;
@@ -173,6 +176,9 @@ struct acrn_vm {
 	struct page			*ioreq_page;
 	u32				pci_conf_addr;
 	struct page			*monitor_page;
+	struct mutex			ioeventfds_lock;
+	struct list_head		ioeventfds;
+	struct acrn_ioreq_client	*ioeventfd_client;
 };
 
 struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
@@ -205,4 +211,8 @@ void acrn_ioreq_range_del(struct acrn_ioreq_client *client,
 
 int acrn_msi_inject(struct acrn_vm *vm, u64 msi_addr, u64 msi_data);
 
+int acrn_ioeventfd_init(struct acrn_vm *vm);
+int acrn_ioeventfd_config(struct acrn_vm *vm, struct acrn_ioeventfd *args);
+void acrn_ioeventfd_deinit(struct acrn_vm *vm);
+
 #endif /* __ACRN_HSM_DRV_H */
drivers/virt/acrn/hsm.c +8 -0
index 7c0e410cf32d..d8325f3ef6c0 100644
@@ -111,6 +111,7 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 	struct acrn_vcpu_regs *cpu_regs;
 	struct acrn_ioreq_notify notify;
 	struct acrn_ptdev_irq *irq_info;
+	struct acrn_ioeventfd ioeventfd;
 	struct acrn_vm_memmap memmap;
 	struct acrn_msi_entry *msi;
 	struct acrn_pcidev *pcidev;
@@ -296,6 +297,13 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 
 		ret = pmcmd_ioctl(cstate_cmd, (void __user *)ioctl_param);
 		break;
+	case ACRN_IOCTL_IOEVENTFD:
+		if (copy_from_user(&ioeventfd, (void __user *)ioctl_param,
+				   sizeof(ioeventfd)))
+			return -EFAULT;
+
+		ret = acrn_ioeventfd_config(vm, &ioeventfd);
+		break;
 	default:
 		dev_warn(vm->dev, "Unknown IOCTL 0x%x!\n", cmd);
 		ret = -ENOTTY;
drivers/virt/acrn/ioeventfd.c new file +273 -0

273 lines not on this page · raw message

drivers/virt/acrn/vm.c +2 -0
index c1fefd92e571..6ca7270114a6 100644
@@ -49,6 +49,7 @@ struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
 	list_add(&vm->list, &acrn_vm_list);
 	write_unlock_bh(&acrn_vm_list_lock);
 
+	acrn_ioeventfd_init(vm);
 	dev_dbg(vm->dev, "VM %u created.\n", vm->vmid);
 	return vm;
 }
@@ -66,6 +67,7 @@ int acrn_vm_destroy(struct acrn_vm *vm)
 	list_del_init(&vm->list);
 	write_unlock_bh(&acrn_vm_list_lock);
 
+	acrn_ioeventfd_deinit(vm);
 	acrn_ioreq_deinit(vm);
 	if (vm->monitor_page) {
 		put_page(vm->monitor_page);
include/uapi/linux/acrn.h +29 -0
index 9fed7209a8ef..7a99124c7d4d 100644
@@ -385,6 +385,32 @@ enum acrn_pm_cmd_type {
 	ACRN_PMCMD_GET_CX_DATA,
 };
 
+#define ACRN_IOEVENTFD_FLAG_PIO		0x01
+#define ACRN_IOEVENTFD_FLAG_DATAMATCH	0x02
+#define ACRN_IOEVENTFD_FLAG_DEASSIGN	0x04
+/**
+ * struct acrn_ioeventfd - Data to operate a &struct hsm_ioeventfd
+ * @fd:		The fd of eventfd associated with a hsm_ioeventfd
+ * @flags:	Logical-OR of ACRN_IOEVENTFD_FLAG_*
+ * @addr:	The start address of IO range of ioeventfd
+ * @len:	The length of IO range of ioeventfd
+ * @reserved:	Reserved
+ * @data:	Data for data matching
+ *
+ * Without flag ACRN_IOEVENTFD_FLAG_DEASSIGN, ioctl ACRN_IOCTL_IOEVENTFD
+ * creates a &struct hsm_ioeventfd with properties originated from &struct
+ * acrn_ioeventfd. With flag ACRN_IOEVENTFD_FLAG_DEASSIGN, ioctl
+ * ACRN_IOCTL_IOEVENTFD destroys the &struct hsm_ioeventfd matching the fd.
+ */
+struct acrn_ioeventfd {
+	__u32	fd;
+	__u32	flags;
+	__u64	addr;
+	__u32	len;
+	__u32	reserved;
+	__u64	data;
+};
+
 /* The ioctl type, documented in ioctl-number.rst */
 #define ACRN_IOCTL_TYPE			0xA2
 
@@ -439,4 +465,7 @@ enum acrn_pm_cmd_type {
 #define ACRN_IOCTL_PM_GET_CPU_STATE	\
 	_IOWR(ACRN_IOCTL_TYPE, 0x60, __u64)
 
+#define ACRN_IOCTL_IOEVENTFD		\
+	_IOW(ACRN_IOCTL_TYPE, 0x70, struct acrn_ioeventfd)
+
 #endif /* _UAPI_ACRN_H */
Signature · 1 line
2.28.0
#17 lore raw

PATCH v3 17/17 virt: acrn: Introduce an interface for Service VM to control vCPU

In reply to [email protected]

From: Shuo Liu <[email protected]>

ACRN supports partition mode to achieve real-time requirements. In partition mode, a CPU core can be dedicated to a vCPU of User VM. The local APIC of the dedicated CPU core can be passthrough to the User VM. The Service VM controls the assignment of the CPU cores.

Introduce an interface for the Service VM to remove the control of CPU core from hypervisor perspective so that the CPU core can be a dedicated CPU core of User VM.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 drivers/virt/acrn/hsm.c       | 50 +++++++++++++++++++++++++++++++++++
 drivers/virt/acrn/hypercall.h | 14 ++++++++++
 2 files changed, 64 insertions(+)
drivers/virt/acrn/hsm.c +50 -0
index 5594ef8e4947..72144cb7569f 100644
@@ -9,6 +9,7 @@
  *	Yakui Zhao <[email protected]>
  */
 
+#include <linux/cpu.h>
 #include <linux/io.h>
 #include <linux/miscdevice.h>
 #include <linux/mm.h>
@@ -342,6 +343,47 @@ static struct miscdevice acrn_dev = {
 	.fops	= &acrn_fops,
 };
 
+static ssize_t remove_cpu_store(struct device *dev,
+				struct device_attribute *attr,
+				const char *buf, size_t count)
+{
+	u64 cpu, lapicid;
+	int ret;
+
+	if (kstrtoull(buf, 0, &cpu) < 0)
+		return -EINVAL;
+
+	if (cpu >= num_possible_cpus() || cpu == 0 || !cpu_is_hotpluggable(cpu))
+		return -EINVAL;
+
+	if (cpu_online(cpu))
+		remove_cpu(cpu);
+
+	lapicid = cpu_data(cpu).apicid;
+	dev_dbg(dev, "Try to remove cpu %lld with lapicid %lld\n", cpu, lapicid);
+	ret = hcall_sos_remove_cpu(lapicid);
+	if (ret < 0) {
+		dev_err(dev, "Failed to remove cpu %lld!\n", cpu);
+		goto fail_remove;
+	}
+
+	return count;
+
+fail_remove:
+	add_cpu(cpu);
+	return ret;
+}
+static DEVICE_ATTR_WO(remove_cpu);
+
+static struct attribute *acrn_attrs[] = {
+	&dev_attr_remove_cpu.attr,
+	NULL
+};
+
+static struct attribute_group acrn_attr_group = {
+	.attrs = acrn_attrs,
+};
+
 static int __init hsm_init(void)
 {
 	int ret;
@@ -358,13 +400,21 @@ static int __init hsm_init(void)
 		return ret;
 	}
 
+	ret = sysfs_create_group(&acrn_dev.this_device->kobj, &acrn_attr_group);
+	if (ret) {
+		dev_warn(acrn_dev.this_device, "sysfs create failed\n");
+		misc_deregister(&acrn_dev);
+		return ret;
+	}
 	acrn_ioreq_intr_setup();
+
 	return 0;
 }
 
 static void __exit hsm_exit(void)
 {
 	acrn_ioreq_intr_remove();
+	sysfs_remove_group(&acrn_dev.this_device->kobj, &acrn_attr_group);
 	misc_deregister(&acrn_dev);
 }
 module_init(hsm_init);
drivers/virt/acrn/hypercall.h +14 -0
index e640632366f0..0cfad05bd1a9 100644
@@ -13,6 +13,9 @@
 
 #define HC_ID 0x80UL
 
+#define HC_ID_GEN_BASE			0x0UL
+#define HC_SOS_REMOVE_CPU		_HC_ID(HC_ID, HC_ID_GEN_BASE + 0x01)
+
 #define HC_ID_VM_BASE			0x10UL
 #define HC_CREATE_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x00)
 #define HC_DESTROY_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x01)
@@ -42,6 +45,17 @@
 #define HC_ID_PM_BASE			0x80UL
 #define HC_PM_GET_CPU_STATE		_HC_ID(HC_ID, HC_ID_PM_BASE + 0x00)
 
+/**
+ * hcall_sos_remove_cpu() - Remove a vCPU of Service VM
+ * @cpu: The vCPU to be removed
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_sos_remove_cpu(u64 cpu)
+{
+	return acrn_hypercall1(HC_SOS_REMOVE_CPU, cpu);
+}
+
 /**
  * hcall_create_vm() - Create a User VM
  * @vminfo:	Service VM GPA of info of User VM creation
Signature · 1 line
2.28.0

A message that is not in the archive (sent off-list or not yet received): [email protected]

#19 lore raw

Re: [PATCH v3 02/17] x86/acrn: Introduce acrn_{setup, remove}_intr_handler()

Quoted 62 lines · On Wed, Sep 09, 2020 at 05:08:21PM +0800, [email protected] wrote:

From: Shuo Liu <[email protected]>

The ACRN Hypervisor builds an I/O request when a trapped I/O access happens in User VM. Then, ACRN Hypervisor issues an upcall by sending a notification interrupt to the Service VM. HSM in the Service VM needs to hook the notification interrupt to handle I/O requests.

Notification interrupts from ACRN Hypervisor are already supported and a, currently uninitialized, callback called.

Export two APIs for HSM to setup/remove its callback.

Cc Dave Hansen <[email protected]>
Cc Sean Christopherson <[email protected]>
Cc Dan Williams <[email protected]>
Cc Fengwei Yin <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 arch/x86/include/asm/acrn.h |  8 ++++++++
 arch/x86/kernel/cpu/acrn.c  | 19 +++++++++++++++++++
 2 files changed, 27 insertions(+)
 create mode 100644 arch/x86/include/asm/acrn.h
arch/x86/include/asm/acrn.h new file +8 -0
new file mode 100644
index 000000000000..ff259b69cde7
@@ -0,0 +1,8 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_ACRN_H
+#define _ASM_X86_ACRN_H
+
+void acrn_setup_intr_handler(void (*handler)(void));
+void acrn_remove_intr_handler(void);
+
+#endif /* _ASM_X86_ACRN_H */
arch/x86/kernel/cpu/acrn.c +4 -0
index 0b2c03943ac6..bd1d7e759a0f 100644
@@ -9,7 +9,11 @@
  *
  */
 
+#define pr_fmt(fmt) "acrn: " fmt
+
 #include <linux/interrupt.h>
+
+#include <asm/acrn.h>
 #include <asm/apic.h>
 #include <asm/cpufeatures.h>
 #include <asm/desc.h>

arch/x86/kernel/cpu/acrn.c DEFINE_IDTENTRY_SYSVEC(sysvec_acrn_hv_callback) · line 65

Quoted 7 lines
arch/x86/kernel/cpu/acrn.c +4 -0
@@ -55,6 +59,21 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_acrn_hv_callback)
 	set_irq_regs(old_regs);
 }
 
+void acrn_setup_intr_handler(void (*handler)(void))
+{
+	if (acrn_intr_handler)
+		pr_warn("Overwrite the acrn_intr_handler.\n");

What can someone do with this warning? If it's really an "error", why not prevent this from happening?

Don't scare users with things they can do nothing about.

thanks,

greg k-h

author
#20 lore raw

Re: [PATCH v3 02/17] x86/acrn: Introduce acrn_{setup, remove}_intr_handler()

Hi Greg,

Quoted 66 lines · Greg Kroah-Hartman
Quoted 65 lines · On Wed, Sep 09, 2020 at 05:08:21PM +0800, [email protected] wrote:

From: Shuo Liu <[email protected]>

The ACRN Hypervisor builds an I/O request when a trapped I/O access happens in User VM. Then, ACRN Hypervisor issues an upcall by sending a notification interrupt to the Service VM. HSM in the Service VM needs to hook the notification interrupt to handle I/O requests.

Notification interrupts from ACRN Hypervisor are already supported and a, currently uninitialized, callback called.

Export two APIs for HSM to setup/remove its callback.

Cc Dave Hansen <[email protected]>
Cc Sean Christopherson <[email protected]>
Cc Dan Williams <[email protected]>
Cc Fengwei Yin <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 arch/x86/include/asm/acrn.h |  8 ++++++++
 arch/x86/kernel/cpu/acrn.c  | 19 +++++++++++++++++++
 2 files changed, 27 insertions(+)
 create mode 100644 arch/x86/include/asm/acrn.h
arch/x86/include/asm/acrn.h new file +8 -0
new file mode 100644
index 000000000000..ff259b69cde7
@@ -0,0 +1,8 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_ACRN_H
+#define _ASM_X86_ACRN_H
+
+void acrn_setup_intr_handler(void (*handler)(void));
+void acrn_remove_intr_handler(void);
+
+#endif /* _ASM_X86_ACRN_H */
arch/x86/kernel/cpu/acrn.c +4 -0
index 0b2c03943ac6..bd1d7e759a0f 100644
@@ -9,7 +9,11 @@
  *
  */
 
+#define pr_fmt(fmt) "acrn: " fmt
+
 #include <linux/interrupt.h>
+
+#include <asm/acrn.h>
 #include <asm/apic.h>
 #include <asm/cpufeatures.h>
 #include <asm/desc.h>
@@ -55,6 +59,21 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_acrn_hv_callback)
 	set_irq_regs(old_regs);
 }
 
Quoted 7 lines

arch/x86/kernel/cpu/acrn.c DEFINE_IDTENTRY_SYSVEC(sysvec_acrn_hv_callback) · line 65

Quoted 4 lines
arch/x86/kernel/cpu/acrn.c +4 -0
+void acrn_setup_intr_handler(void (*handler)(void))
+{
+	if (acrn_intr_handler)
+		pr_warn("Overwrite the acrn_intr_handler.\n");

What can someone do with this warning? If it's really an "error", why not prevent this from happening?

It cannot happen if there are not new user of it. The current user is HSM hsm_init()/hsm_exit() in the later patches of this series.

I can remove the warn.

Quoted 2 lines

Don't scare users with things they can do nothing about.

OK. Got it. Thanks.

Thanks
shuo