Skip to contentSkip to conversation

HSM driver for ACRN hypervisor

PATCH Superseded [email protected] 10 participants 58 messages Sep 22, 2020 – Oct 13, 2020 1 message not in the archive

Subsystems (3)

v4 of 9 Superseded Latest is v9 →

lore
Share & tools
lore
#0 lore raw

v4 cover HSM driver for ACRN hypervisor

From: Shuo Liu <[email protected]>

ACRN is a Type 1 reference hypervisor stack, running directly on the bare-metal hardware, and is suitable for a variety of IoT and embedded device solutions.

ACRN implements a hybrid VMM architecture, using a privileged Service VM. The Service VM manages the system resources (CPU, memory, etc.) and I/O devices of User VMs. Multiple User VMs are supported, with each of them running Linux, Android OS or Windows. Both Service VM and User VMs are guest VM.

Below figure shows the architecture.

                Service VM                    User VM
      +----------------------------+  |  +------------------+
      |        +--------------+    |  |  |                  |
      |        |ACRN userspace|    |  |  |                  |
      |        +--------------+    |  |  |                  |
      |-----------------ioctl------|  |  |                  |   ...
      |kernel space   +----------+ |  |  |                  |
      |               |   HSM    | |  |  | Drivers          |
      |               +----------+ |  |  |                  |
      +--------------------|-------+  |  +------------------+
  +---------------------hypercall----------------------------------------+
  |                       ACRN Hypervisor                                |
  +----------------------------------------------------------------------+
  |                          Hardware                                    |
  +----------------------------------------------------------------------+

There is only one Service VM which could run Linux as OS.

In a typical case, the Service VM will be auto started when ACRN Hypervisor is booted. Then the ACRN userspace (an application running in Service VM) could be used to start/stop User VMs by communicating with ACRN Hypervisor Service Module (HSM).

ACRN Hypervisor Service Module (HSM) is a middle layer that allows the ACRN userspace and Service VM OS kernel to communicate with ACRN Hypervisor and manage different User VMs. This middle layer provides the following functionalities,
- Issues hypercalls to the hypervisor to manage User VMs:
* VM/vCPU management
* Memory management
* Device passthrough
* Interrupts injection
- I/O requests handling from User VMs.
- Exports ioctl through HSM char device.
- Exports function calls for other kernel modules

ACRN is focused on embedded system. So it doesn't support some features. E.g.,
- ACRN doesn't support VM migration.
- ACRN doesn't support vCPU migration.

This patch set adds the HSM to the Linux kernel.

The basic ARCN support was merged to upstream already.
https://lore.kernel.org/lkml/[email protected]/

ChangeLog:
v4:
- Used acrn_dev.this_device directly for dev_*() (Reinette)
- Removed the odd usage of {get|put}_device() on &acrn_dev->this_device (Greg)
- Removed unused log code. (Greg)
- Corrected the return error values. (Greg)
- Mentioned that HSM relies hypervisor for sanity check in acrn_dev_ioctl() comments (Greg)

v3:
- Used {get|put}_device() helpers on &acrn_dev->this_device
- Moved unused code from front patches to later ones.
- Removed self-defined pr_fmt() and dev_fmt()
- Provided comments for acrn_vm_list_lock.

v2:
- Removed API version related code. (Dave)
- Replaced pr_*() by dev_*(). (Greg)
- Used -ENOTTY as the error code of unsupported ioctl. (Greg)

Shuo Liu (16):
docs: acrn: Introduce ACRN
x86/acrn: Introduce acrn_{setup, remove}_intr_handler() x86/acrn: Introduce hypercall interfaces
virt: acrn: Introduce ACRN HSM basic driver
virt: acrn: Introduce VM management interfaces
virt: acrn: Introduce an ioctl to set vCPU registers state
virt: acrn: Introduce EPT mapping management
virt: acrn: Introduce I/O request management
virt: acrn: Introduce PCI configuration space PIO accesses combiner
virt: acrn: Introduce interfaces for PCI device passthrough
virt: acrn: Introduce interrupt injection interfaces
virt: acrn: Introduce interfaces to query C-states and P-states
allowed by hypervisor
virt: acrn: Introduce I/O ranges operation interfaces
virt: acrn: Introduce ioeventfd
virt: acrn: Introduce irqfd
virt: acrn: Introduce an interface for Service VM to control vCPU

Yin Fengwei (1):
x86/acrn: Introduce an API to check if a VM is privileged

 .../userspace-api/ioctl/ioctl-number.rst      |   1 +
 Documentation/virt/acrn/index.rst             |  11 +
 Documentation/virt/acrn/introduction.rst      |  40 ++
 Documentation/virt/acrn/io-request.rst        |  97 +++
 Documentation/virt/index.rst                  |   1 +
 MAINTAINERS                                   |   9 +
 arch/x86/include/asm/acrn.h                   |  74 ++
 arch/x86/kernel/cpu/acrn.c                    |  35 +-
 drivers/virt/Kconfig                          |   2 +
 drivers/virt/Makefile                         |   1 +
 drivers/virt/acrn/Kconfig                     |  15 +
 drivers/virt/acrn/Makefile                    |   3 +
 drivers/virt/acrn/acrn_drv.h                  | 229 +++++++
 drivers/virt/acrn/hsm.c                       | 437 ++++++++++++
 drivers/virt/acrn/hypercall.h                 | 254 +++++++
 drivers/virt/acrn/ioeventfd.c                 | 273 ++++++++
 drivers/virt/acrn/ioreq.c                     | 645 ++++++++++++++++++
 drivers/virt/acrn/irqfd.c                     | 235 +++++++
 drivers/virt/acrn/mm.c                        | 305 +++++++++
 drivers/virt/acrn/vm.c                        | 126 ++++
 include/uapi/linux/acrn.h                     | 486 +++++++++++++
 21 files changed, 3278 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/virt/acrn/index.rst
 create mode 100644 Documentation/virt/acrn/introduction.rst
 create mode 100644 Documentation/virt/acrn/io-request.rst
 create mode 100644 arch/x86/include/asm/acrn.h
 create mode 100644 drivers/virt/acrn/Kconfig
 create mode 100644 drivers/virt/acrn/Makefile
 create mode 100644 drivers/virt/acrn/acrn_drv.h
 create mode 100644 drivers/virt/acrn/hsm.c
 create mode 100644 drivers/virt/acrn/hypercall.h
 create mode 100644 drivers/virt/acrn/ioeventfd.c
 create mode 100644 drivers/virt/acrn/ioreq.c
 create mode 100644 drivers/virt/acrn/irqfd.c
 create mode 100644 drivers/virt/acrn/mm.c
 create mode 100644 drivers/virt/acrn/vm.c
 create mode 100644 include/uapi/linux/acrn.h

base-commit: 18445bf405cb331117bc98427b1ba6f12418ad17

Signature · 1 line
2.28.0
#1 lore raw

PATCH v4 1/17 docs: acrn: Introduce ACRN

From: Shuo Liu <[email protected]>

Add documentation on the following aspects of ACRN:

1) A brief introduction on the architecture of ACRN.
2) I/O request handling in ACRN.

To learn more about ACRN, please go to ACRN project website
https://projectacrn.org, or the documentation page
https://projectacrn.github.io/.

Cc Dave Hansen <[email protected]>
Cc Sen Christopherson <[email protected]>
Cc Dan Williams <[email protected]>
Cc Fengwei Yin <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 Documentation/virt/acrn/index.rst        | 11 +++
 Documentation/virt/acrn/introduction.rst | 40 ++++++++++
 Documentation/virt/acrn/io-request.rst   | 97 ++++++++++++++++++++++++
 Documentation/virt/index.rst             |  1 +
 MAINTAINERS                              |  7 ++
 5 files changed, 156 insertions(+)
 create mode 100644 Documentation/virt/acrn/index.rst
 create mode 100644 Documentation/virt/acrn/introduction.rst
 create mode 100644 Documentation/virt/acrn/io-request.rst
Documentation/virt/acrn/index.rst new file +11 -0
new file mode 100644
index 000000000000..e3cf99033bdb
@@ -0,0 +1,11 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+===============
+ACRN Hypervisor
+===============
+
+.. toctree::
+   :maxdepth: 1
+
+   introduction
+   io-request
Documentation/virt/acrn/introduction.rst new file +40 -0
new file mode 100644
index 000000000000..6b44924d5c0e
@@ -0,0 +1,40 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+ACRN Hypervisor Introduction
+============================
+
+The ACRN Hypervisor is a Type 1 hypervisor, running directly on the bare-metal
+hardware. It has a privileged management VM, called Service VM, to manage User
+VMs and do I/O emulation.
+
+ACRN userspace is an application running in the Service VM that emulates
+devices for a User VM based on command line configurations. ACRN Hypervisor
+Service Module (HSM) is a kernel module in the Service VM which provides
+hypervisor services to the ACRN userspace.
+
+Below figure shows the architecture.
+
+::
+
+                Service VM                    User VM
+      +----------------------------+  |  +------------------+
+      |        +--------------+    |  |  |                  |
+      |        |ACRN userspace|    |  |  |                  |
+      |        +--------------+    |  |  |                  |
+      |-----------------ioctl------|  |  |                  |   ...
+      |kernel space   +----------+ |  |  |                  |
+      |               |   HSM    | |  |  | Drivers          |
+      |               +----------+ |  |  |                  |
+      +--------------------|-------+  |  +------------------+
+  +---------------------hypercall----------------------------------------+
+  |                         ACRN Hypervisor                              |
+  +----------------------------------------------------------------------+
+  |                          Hardware                                    |
+  +----------------------------------------------------------------------+
+
+ACRN userspace allocates memory for the User VM, configures and initializes the
+devices used by the User VM, loads the virtual bootloader, initializes the
+virtual CPU state and handles I/O request accesses from the User VM. It uses
+ioctls to communicate with the HSM. HSM implements hypervisor services by
+interacting with the ACRN Hypervisor via hypercalls. HSM exports a char device
+interface (/dev/acrn_hsm) to userspace.
Documentation/virt/acrn/io-request.rst new file +97 -0

97 lines not on this page · raw message

Documentation/virt/index.rst +1 -0
index de1ab81df958..c10b519507f5 100644
@@ -11,6 +11,7 @@ Linux Virtualization Support
    uml/user_mode_linux
    paravirt_ops
    guest-halt-polling
+   acrn/index
 
 .. only:: html and subproject
 
MAINTAINERS +7 -0
index deaafb617361..e0fea5e464b4 100644
@@ -436,6 +436,13 @@ S:	Orphan
 F:	drivers/platform/x86/wmi.c
 F:	include/uapi/linux/wmi.h
 
+ACRN HYPERVISOR SERVICE MODULE
+M:	Shuo Liu <[email protected]>
+L:	[email protected]
+S:	Supported
+W:	https://projectacrn.org
+F:	Documentation/virt/acrn/
+
 AD1889 ALSA SOUND DRIVER
 L:	[email protected]
 S:	Maintained
Signature · 1 line
2.28.0
reviewer
#2 lore raw

Re: [PATCH v4 01/17] docs: acrn: Introduce ACRN

Quoted 54 lines · On 9/22/20 4:42 AM, [email protected] wrote:

From: Shuo Liu <[email protected]>

Add documentation on the following aspects of ACRN:

1) A brief introduction on the architecture of ACRN.
2) I/O request handling in ACRN.

To learn more about ACRN, please go to ACRN project website
https://projectacrn.org, or the documentation page
https://projectacrn.github.io/.

Cc Dave Hansen <[email protected]>
Cc Sen Christopherson <[email protected]>
Cc Dan Williams <[email protected]>
Cc Fengwei Yin <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 Documentation/virt/acrn/index.rst        | 11 +++
 Documentation/virt/acrn/introduction.rst | 40 ++++++++++
 Documentation/virt/acrn/io-request.rst   | 97 ++++++++++++++++++++++++
 Documentation/virt/index.rst             |  1 +
 MAINTAINERS                              |  7 ++
 5 files changed, 156 insertions(+)
 create mode 100644 Documentation/virt/acrn/index.rst
 create mode 100644 Documentation/virt/acrn/introduction.rst
 create mode 100644 Documentation/virt/acrn/io-request.rst
Documentation/virt/acrn/io-request.rst new file +13 -0
new file mode 100644
index 000000000000..019dc5978f7c
@@ -0,0 +1,97 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+I/O request handling
+====================
+
+An I/O request of a User VM, which is constructed by the hypervisor, is
+distributed by the ACRN Hypervisor Service Module to an I/O client
+corresponding to the address range of the I/O request. Details of I/O request
+handling are described in the following sections.
+
+1. I/O request
+--------------
+

...

Quoted 4 lines
Documentation/virt/acrn/io-request.rst new file +4 -0
+
+2. I/O clients
+--------------
+

...

Documentation/virt/acrn/io-request.rst line 70

Quoted 5 lines
Documentation/virt/acrn/io-request.rst new file +5 -0
+
+3. I/O request state transition
+-------------------------------
+
+The state transitions of a ACRN I/O request are as follows.

of an ACRN

Quoted 6 lines
Documentation/virt/acrn/io-request.rst new file +6 -0
+
+::
+
+   FREE -> PENDING -> PROCESSING -> COMPLETE -> FREE -> ...
+
+- FREE: this I/O request slot is empty

Documentation/virt/acrn/io-request.rst line 86

Quoted 10 lines
Documentation/virt/acrn/io-request.rst new file +10 -0
+- PENDING: a valid I/O request is pending in this slot
+- PROCESSING: the I/O request is being processed
+- COMPLETE: the I/O request has been processed
+
+An I/O request in COMPLETE or FREE state is owned by the hypervisor. HSM and
+ACRN userspace are in charge of processing the others.
+
+4. Processing flow of I/O requests
+-------------------------------
+

...

thanks.

Signature · 1 line
~Randy
author
#3 lore raw

Re: [PATCH v4 01/17] docs: acrn: Introduce ACRN

Quoted 65 lines · Randy Dunlap
Quoted 54 lines · On 9/22/20 4:42 AM, [email protected] wrote:

From: Shuo Liu <[email protected]>

Add documentation on the following aspects of ACRN:

1) A brief introduction on the architecture of ACRN.
2) I/O request handling in ACRN.

To learn more about ACRN, please go to ACRN project website
https://projectacrn.org, or the documentation page
https://projectacrn.github.io/.

Cc Dave Hansen <[email protected]>
Cc Sen Christopherson <[email protected]>
Cc Dan Williams <[email protected]>
Cc Fengwei Yin <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 Documentation/virt/acrn/index.rst        | 11 +++
 Documentation/virt/acrn/introduction.rst | 40 ++++++++++
 Documentation/virt/acrn/io-request.rst   | 97 ++++++++++++++++++++++++
 Documentation/virt/index.rst             |  1 +
 MAINTAINERS                              |  7 ++
 5 files changed, 156 insertions(+)
 create mode 100644 Documentation/virt/acrn/index.rst
 create mode 100644 Documentation/virt/acrn/introduction.rst
 create mode 100644 Documentation/virt/acrn/io-request.rst
Documentation/virt/acrn/io-request.rst new file +13 -0
new file mode 100644
index 000000000000..019dc5978f7c
@@ -0,0 +1,97 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+I/O request handling
+====================
+
+An I/O request of a User VM, which is constructed by the hypervisor, is
+distributed by the ACRN Hypervisor Service Module to an I/O client
+corresponding to the address range of the I/O request. Details of I/O request
+handling are described in the following sections.
+
+1. I/O request
+--------------
+

...

Quoted 4 lines
Documentation/virt/acrn/io-request.rst new file +4 -0
+
+2. I/O clients
+--------------
+

...

Quoted 7 lines

Documentation/virt/acrn/io-request.rst line 70

Quoted 5 lines
Documentation/virt/acrn/io-request.rst new file +5 -0
+
+3. I/O request state transition
+-------------------------------
+
+The state transitions of a ACRN I/O request are as follows.

of an ACRN

OK. Thanks for review.

Thanks
shuo

#4 lore raw

PATCH v4 2/17 x86/acrn: Introduce acrn_{setup, remove}_intr_handler()

In reply to [email protected]

From: Shuo Liu <[email protected]>

The ACRN Hypervisor builds an I/O request when a trapped I/O access happens in User VM. Then, ACRN Hypervisor issues an upcall by sending a notification interrupt to the Service VM. HSM in the Service VM needs to hook the notification interrupt to handle I/O requests.

Notification interrupts from ACRN Hypervisor are already supported and a, currently uninitialized, callback called.

Export two APIs for HSM to setup/remove its callback.

Cc Dave Hansen <[email protected]>
Cc Sean Christopherson <[email protected]>
Cc Dan Williams <[email protected]>
Cc Fengwei Yin <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 arch/x86/include/asm/acrn.h |  8 ++++++++
 arch/x86/kernel/cpu/acrn.c  | 16 ++++++++++++++++
 2 files changed, 24 insertions(+)
 create mode 100644 arch/x86/include/asm/acrn.h
arch/x86/include/asm/acrn.h new file +8 -0
new file mode 100644
index 000000000000..ff259b69cde7
@@ -0,0 +1,8 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_ACRN_H
+#define _ASM_X86_ACRN_H
+
+void acrn_setup_intr_handler(void (*handler)(void));
+void acrn_remove_intr_handler(void);
+
+#endif /* _ASM_X86_ACRN_H */
arch/x86/kernel/cpu/acrn.c +16 -0
index 0b2c03943ac6..42e88d01ccf9 100644
@@ -9,7 +9,11 @@
  *
  */
 
+#define pr_fmt(fmt) "acrn: " fmt
+
 #include <linux/interrupt.h>
+
+#include <asm/acrn.h>
 #include <asm/apic.h>
 #include <asm/cpufeatures.h>
 #include <asm/desc.h>
@@ -55,6 +59,18 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_acrn_hv_callback)
 	set_irq_regs(old_regs);
 }
 
+void acrn_setup_intr_handler(void (*handler)(void))
+{
+	acrn_intr_handler = handler;
+}
+EXPORT_SYMBOL_GPL(acrn_setup_intr_handler);
+
+void acrn_remove_intr_handler(void)
+{
+	acrn_intr_handler = NULL;
+}
+EXPORT_SYMBOL_GPL(acrn_remove_intr_handler);
+
 const __initconst struct hypervisor_x86 x86_hyper_acrn = {
 	.name                   = "ACRN",
 	.detect                 = acrn_detect,
Signature · 1 line
2.28.0
#5 lore raw

Re: [PATCH v4 02/17] x86/acrn: Introduce acrn_{setup, remove}_intr_handler()

Quoted 50 lines · On Tue, Sep 22, 2020 at 07:42:56PM +0800, [email protected] wrote:

From: Shuo Liu <[email protected]>

The ACRN Hypervisor builds an I/O request when a trapped I/O access happens in User VM. Then, ACRN Hypervisor issues an upcall by sending a notification interrupt to the Service VM. HSM in the Service VM needs to hook the notification interrupt to handle I/O requests.

Notification interrupts from ACRN Hypervisor are already supported and a, currently uninitialized, callback called.

Export two APIs for HSM to setup/remove its callback.

Cc Dave Hansen <[email protected]>
Cc Sean Christopherson <[email protected]>
Cc Dan Williams <[email protected]>
Cc Fengwei Yin <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 arch/x86/include/asm/acrn.h |  8 ++++++++
 arch/x86/kernel/cpu/acrn.c  | 16 ++++++++++++++++
 2 files changed, 24 insertions(+)
 create mode 100644 arch/x86/include/asm/acrn.h
arch/x86/include/asm/acrn.h new file +8 -0
new file mode 100644
index 000000000000..ff259b69cde7
@@ -0,0 +1,8 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_ACRN_H
+#define _ASM_X86_ACRN_H
+
+void acrn_setup_intr_handler(void (*handler)(void));
+void acrn_remove_intr_handler(void);
+
+#endif /* _ASM_X86_ACRN_H */
arch/x86/kernel/cpu/acrn.c +0 -0
index 0b2c03943ac6..42e88d01ccf9 100644

arch/x86/kernel/cpu/acrn.c line 12

Quoted 4 lines
arch/x86/kernel/cpu/acrn.c +1 -0
@@ -9,7 +9,11 @@
  *
  */
 
+#define pr_fmt(fmt) "acrn: " fmt

Why is this needed, if you are not adding pr_* calls in this patch?

thanks,

greg k-h

author
#6 lore raw

Re: [PATCH v4 02/17] x86/acrn: Introduce acrn_{setup, remove}_intr_handler()

Quoted 54 lines · Greg Kroah-Hartman
Quoted 53 lines · On Tue, Sep 22, 2020 at 07:42:56PM +0800, [email protected] wrote:

From: Shuo Liu <[email protected]>

The ACRN Hypervisor builds an I/O request when a trapped I/O access happens in User VM. Then, ACRN Hypervisor issues an upcall by sending a notification interrupt to the Service VM. HSM in the Service VM needs to hook the notification interrupt to handle I/O requests.

Notification interrupts from ACRN Hypervisor are already supported and a, currently uninitialized, callback called.

Export two APIs for HSM to setup/remove its callback.

Cc Dave Hansen <[email protected]>
Cc Sean Christopherson <[email protected]>
Cc Dan Williams <[email protected]>
Cc Fengwei Yin <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 arch/x86/include/asm/acrn.h |  8 ++++++++
 arch/x86/kernel/cpu/acrn.c  | 16 ++++++++++++++++
 2 files changed, 24 insertions(+)
 create mode 100644 arch/x86/include/asm/acrn.h
arch/x86/include/asm/acrn.h new file +8 -0
new file mode 100644
index 000000000000..ff259b69cde7
@@ -0,0 +1,8 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_ACRN_H
+#define _ASM_X86_ACRN_H
+
+void acrn_setup_intr_handler(void (*handler)(void));
+void acrn_remove_intr_handler(void);
+
+#endif /* _ASM_X86_ACRN_H */
arch/x86/kernel/cpu/acrn.c +0 -0
index 0b2c03943ac6..42e88d01ccf9 100644
@@ -9,7 +9,11 @@
  *
  */
 
Quoted 3 lines

arch/x86/kernel/cpu/acrn.c line 12

Quoted 1 line
arch/x86/kernel/cpu/acrn.c +1 -0
+#define pr_fmt(fmt) "acrn: " fmt

Why is this needed, if you are not adding pr_* calls in this patch?

True. I will remove it. Thanks.

maintainer
#7 lore raw

Re: [PATCH v4 02/17] x86/acrn: Introduce acrn_{setup, remove}_intr_handler()

In reply to [email protected]

arch/x86/kernel/cpu/acrn.c DEFINE_IDTENTRY_SYSVEC(sysvec_acrn_hv_callback) · line 72

Quoted 11 lines · On Tue, Sep 22, 2020 at 07:42:56PM +0800, [email protected] wrote:
arch/x86/kernel/cpu/acrn.c +11 -0
+void acrn_setup_intr_handler(void (*handler)(void))
+{
+	acrn_intr_handler = handler;
+}
+EXPORT_SYMBOL_GPL(acrn_setup_intr_handler);
+
+void acrn_remove_intr_handler(void)
+{
+	acrn_intr_handler = NULL;
+}
+EXPORT_SYMBOL_GPL(acrn_remove_intr_handler);

I don't like this one bit.

Also, what stops the module from doing acrn_remove_intr_handler() while it gets a HYPERVISOR_CALLBACK_VECTOR interrupt and the handler disappearing from under it?

IOW, this should be an atomic notifier instead.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#8 lore raw

Re: [PATCH v4 02/17] x86/acrn: Introduce acrn_{setup, remove}_intr_handler()

Quoted 15 lines · Borislav Petkov

arch/x86/kernel/cpu/acrn.c DEFINE_IDTENTRY_SYSVEC(sysvec_acrn_hv_callback) · line 72

Quoted 11 lines · On Tue, Sep 22, 2020 at 07:42:56PM +0800, [email protected] wrote:
arch/x86/kernel/cpu/acrn.c +11 -0
+void acrn_setup_intr_handler(void (*handler)(void))
+{
+	acrn_intr_handler = handler;
+}
+EXPORT_SYMBOL_GPL(acrn_setup_intr_handler);
+
+void acrn_remove_intr_handler(void)
+{
+	acrn_intr_handler = NULL;
+}
+EXPORT_SYMBOL_GPL(acrn_remove_intr_handler);

I don't like this one bit.

Quoted 5 lines

Also, what stops the module from doing acrn_remove_intr_handler() while it gets a HYPERVISOR_CALLBACK_VECTOR interrupt and the handler disappearing from under it?

IOW, this should be an atomic notifier instead.

That does not prevent that either and notifiers suck. The pointer is fine and if something removes the handler before all of the muck is shutdown then the author can keep the pieces and mop up the remains.

Thanks,

tglx

maintainer
#9 lore raw

Re: [PATCH v4 02/17] x86/acrn: Introduce acrn_{setup, remove}_intr_handler()

Quoted 1 line · Thomas Gleixner

That does not prevent that either and notifiers suck.

Bah, atomic notifiers run functions which cannot block, not what is needed here, right.

Quoted 3 lines

The pointer is fine and if something removes the handler before all of the muck is shutdown then the author can keep the pieces and mop up the remains.

Uhu, so what makes sure that the module is not removed while an IRQ is happening?

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
author
#10 lore raw

Re: [PATCH v4 02/17] x86/acrn: Introduce acrn_{setup, remove}_intr_handler()

Hi Boris,

Quoted 6 lines · Borislav Petkov
Quoted 1 line · Thomas Gleixner

That does not prevent that either and notifiers suck.

Bah, atomic notifiers run functions which cannot block, not what is needed here, right.

Quoted 6 lines
Quoted 3 lines

The pointer is fine and if something removes the handler before all of the muck is shutdown then the author can keep the pieces and mop up the remains.

Uhu, so what makes sure that the module is not removed while an IRQ is happening?

The precondition of the removing of the module is that there is no User VM living (every opening of the dev file will hold a ref count of the module). The interrupt only can occur with active User VMs. So if a notification interrupt is happending, the module cannot be removed as there is still User VM living.

Thanks
shuo

#11 lore raw

PATCH v4 3/17 x86/acrn: Introduce an API to check if a VM is privileged

In reply to [email protected]

From: Yin Fengwei <[email protected]>

ACRN Hypervisor reports hypervisor features via CPUID leaf 0x40000001 which is similar to KVM. A VM can check if it's the privileged VM using the feature bits. The Service VM is the only privileged VM by design.

Cc Dave Hansen <[email protected]>
Cc Sean Christopherson <[email protected]>
Cc Dan Williams <[email protected]>
Cc Fengwei Yin <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 arch/x86/include/asm/acrn.h |  9 +++++++++
 arch/x86/kernel/cpu/acrn.c  | 19 ++++++++++++++++++-
 2 files changed, 27 insertions(+), 1 deletion(-)
arch/x86/include/asm/acrn.h +9 -0
index ff259b69cde7..a2d4aea3a80d 100644
@@ -2,7 +2,16 @@
 #ifndef _ASM_X86_ACRN_H
 #define _ASM_X86_ACRN_H
 
+/*
+ * This CPUID returns feature bitmaps in EAX.
+ * Guest VM uses this to detect the appropriate feature bit.
+ */
+#define	ACRN_CPUID_FEATURES		0x40000001
+/* Bit 0 indicates whether guest VM is privileged */
+#define	ACRN_FEATURE_PRIVILEGED_VM	BIT(0)
+
 void acrn_setup_intr_handler(void (*handler)(void));
 void acrn_remove_intr_handler(void);
+bool acrn_is_privileged_vm(void);
 
 #endif /* _ASM_X86_ACRN_H */
arch/x86/kernel/cpu/acrn.c +18 -1
index 42e88d01ccf9..b04fef8bd50b 100644
@@ -21,9 +21,26 @@
 #include <asm/idtentry.h>
 #include <asm/irq_regs.h>
 
+static u32 acrn_cpuid_base(void)
+{
+	static u32 acrn_cpuid_base;
+
+	if (!acrn_cpuid_base && boot_cpu_has(X86_FEATURE_HYPERVISOR))
+		acrn_cpuid_base = hypervisor_cpuid_base("ACRNACRNACRN", 0);
+
+	return acrn_cpuid_base;
+}
+
+bool acrn_is_privileged_vm(void)
+{
+	return cpuid_eax(acrn_cpuid_base() | ACRN_CPUID_FEATURES) &
+			 ACRN_FEATURE_PRIVILEGED_VM;
+}
+EXPORT_SYMBOL_GPL(acrn_is_privileged_vm);
+
 static u32 __init acrn_detect(void)
 {
-	return hypervisor_cpuid_base("ACRNACRNACRN", 0);
+	return acrn_cpuid_base();
 }
 
 static void __init acrn_init_platform(void)
Signature · 1 line
2.28.0
maintainer
#12 lore raw

Re: [PATCH v4 03/17] x86/acrn: Introduce an API to check if a VM is privileged

arch/x86/kernel/cpu/acrn.c line 36

Quoted 13 lines · On Tue, Sep 22, 2020 at 07:42:57PM +0800, [email protected] wrote:
arch/x86/kernel/cpu/acrn.c +13 -0
+static u32 acrn_cpuid_base(void)
+{
+	static u32 acrn_cpuid_base;
+
+	if (!acrn_cpuid_base && boot_cpu_has(X86_FEATURE_HYPERVISOR))
+		acrn_cpuid_base = hypervisor_cpuid_base("ACRNACRNACRN", 0);
+
+	return acrn_cpuid_base;
+}
+
+bool acrn_is_privileged_vm(void)
+{
+	return cpuid_eax(acrn_cpuid_base() | ACRN_CPUID_FEATURES) &

What's that dance and acrn_cpuid_base static thing needed for? Why not simply:

cpuid_eax(ACRN_CPUID_FEATURES) & ...

?

arch/x86/kernel/cpu/acrn.c line 39

Quoted 3 lines
arch/x86/kernel/cpu/acrn.c +3 -0
+			 ACRN_FEATURE_PRIVILEGED_VM;
+}
+EXPORT_SYMBOL_GPL(acrn_is_privileged_vm);

Also, if you're going to need more of those bit checkers acrn_is_<something> which look at ACRN_CPUID_FEATURES, just stash CPUID_0x40000001_EAX locally and use a

acrn_has(ACRN_FEATURE_PRIVILEGED_VM)

which does the bit testing.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
author
#13 lore raw

Re: [PATCH v4 03/17] x86/acrn: Introduce an API to check if a VM is privileged

Hi Boris,

Quoted 18 lines · Borislav Petkov

arch/x86/kernel/cpu/acrn.c line 36

Quoted 13 lines · On Tue, Sep 22, 2020 at 07:42:57PM +0800, [email protected] wrote:
arch/x86/kernel/cpu/acrn.c +13 -0
+static u32 acrn_cpuid_base(void)
+{
+	static u32 acrn_cpuid_base;
+
+	if (!acrn_cpuid_base && boot_cpu_has(X86_FEATURE_HYPERVISOR))
+		acrn_cpuid_base = hypervisor_cpuid_base("ACRNACRNACRN", 0);
+
+	return acrn_cpuid_base;
+}
+
+bool acrn_is_privileged_vm(void)
+{
+	return cpuid_eax(acrn_cpuid_base() | ACRN_CPUID_FEATURES) &

What's that dance and acrn_cpuid_base static thing needed for? Why not simply:

Quoted 3 lines

cpuid_eax(ACRN_CPUID_FEATURES) & ...

?

hypervisor_cpuid_base() searches reserved hypervisor cpuid region and return the base matched the right signature, the base might vary. So i put it here.

Quoted 9 lines

arch/x86/kernel/cpu/acrn.c line 39

Quoted 4 lines
arch/x86/kernel/cpu/acrn.c +3 -0
+			 ACRN_FEATURE_PRIVILEGED_VM;
+}
+EXPORT_SYMBOL_GPL(acrn_is_privileged_vm);

Also, if you're going to need more of those bit checkers acrn_is_<something> which look at ACRN_CPUID_FEATURES, just stash CPUID_0x40000001_EAX locally and use a

Quoted 3 lines

acrn_has(ACRN_FEATURE_PRIVILEGED_VM)

which does the bit testing.

Thanks. Currently, there is only one feature bit. I will introduce that you suggested with more feature bits to be tested.

Thanks
shuo

#14 lore raw

PATCH v4 4/17 x86/acrn: Introduce hypercall interfaces

In reply to [email protected]

From: Shuo Liu <[email protected]>

The Service VM communicates with the hypervisor via conventional hypercalls. VMCALL instruction is used to make the hypercalls.

ACRN hypercall ABI:
* Hypercall number is in R8 register.
* Up to 2 parameters are in RDI and RSI registers.
* Return value is in RAX register.

Introduce the ACRN hypercall interfaces. Because GCC doesn't support R8 register as direct register constraints, here are two ways to use R8 in extended asm:
1) use explicit register variable as input
2) use supported constraint as input with a explicit MOV to R8 in beginning of asm

The number of instructions of above two ways are same. Asm code from 1)

  38:   41 b8 00 00 00 80       mov    $0x80000000,%r8d
  3e:   48 89 c7                mov    %rax,%rdi
  41:   0f 01 c1                vmcall

Here, writes to the lower dword (%r8d) clear the upper dword of %r8 when the CPU is in 64-bit mode.

Asm code from 2)

  38:   48 89 c7                mov    %rax,%rdi
  3b:   49 b8 00 00 00 80 00    movabs $0x80000000,%r8
  42:   00 00 00
  45:   0f 01 c1                vmcall

Choose 1) for code simplicity and a little bit of code size optimization.

Cc Dave Hansen <[email protected]>
Cc Sean Christopherson <[email protected]>
Cc Dan Williams <[email protected]>
Cc Fengwei Yin <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 arch/x86/include/asm/acrn.h | 57 +++++++++++++++++++++++++++++++++++++
 1 file changed, 57 insertions(+)
arch/x86/include/asm/acrn.h +57 -0
index a2d4aea3a80d..23a93b87edeb 100644
@@ -14,4 +14,61 @@ void acrn_setup_intr_handler(void (*handler)(void));
 void acrn_remove_intr_handler(void);
 bool acrn_is_privileged_vm(void);
 
+/*
+ * Hypercalls for ACRN
+ *
+ * - VMCALL instruction is used to implement ACRN hypercalls.
+ * - ACRN hypercall ABI:
+ *   - Hypercall number is passed in R8 register.
+ *   - Up to 2 arguments are passed in RDI, RSI.
+ *   - Return value will be placed in RAX.
+ */
+static inline long acrn_hypercall0(unsigned long hcall_id)
+{
+	register long r8 asm("r8");
+	long result;
+
+	/* Nothing can come between the r8 assignment and the asm: */
+	r8 = hcall_id;
+	asm volatile("vmcall\n\t"
+		     : "=a" (result)
+		     : "r" (r8)
+		     : );
+
+	return result;
+}
+
+static inline long acrn_hypercall1(unsigned long hcall_id,
+				   unsigned long param1)
+{
+	register long r8 asm("r8");
+	long result;
+
+	/* Nothing can come between the r8 assignment and the asm: */
+	r8 = hcall_id;
+	asm volatile("vmcall\n\t"
+		     : "=a" (result)
+		     : "r" (r8), "D" (param1)
+		     : );
+
+	return result;
+}
+
+static inline long acrn_hypercall2(unsigned long hcall_id,
+				   unsigned long param1,
+				   unsigned long param2)
+{
+	register long r8 asm("r8");
+	long result;
+
+	/* Nothing can come between the r8 assignment and the asm: */
+	r8 = hcall_id;
+	asm volatile("vmcall\n\t"
+		     : "=a" (result)
+		     : "r" (r8), "D" (param1), "S" (param2)
+		     : );
+
+	return result;
+}
+
 #endif /* _ASM_X86_ACRN_H */
Signature · 1 line
2.28.0
#15 lore raw

Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces

Quoted 68 lines · On Tue, Sep 22, 2020 at 07:42:58PM +0800, [email protected] wrote:

From: Shuo Liu <[email protected]>

The Service VM communicates with the hypervisor via conventional hypercalls. VMCALL instruction is used to make the hypercalls.

ACRN hypercall ABI:
* Hypercall number is in R8 register.
* Up to 2 parameters are in RDI and RSI registers.
* Return value is in RAX register.

Introduce the ACRN hypercall interfaces. Because GCC doesn't support R8 register as direct register constraints, here are two ways to use R8 in extended asm:
1) use explicit register variable as input
2) use supported constraint as input with a explicit MOV to R8 in beginning of asm

The number of instructions of above two ways are same. Asm code from 1)

  38:   41 b8 00 00 00 80       mov    $0x80000000,%r8d
  3e:   48 89 c7                mov    %rax,%rdi
  41:   0f 01 c1                vmcall

Here, writes to the lower dword (%r8d) clear the upper dword of %r8 when the CPU is in 64-bit mode.

Asm code from 2)

  38:   48 89 c7                mov    %rax,%rdi
  3b:   49 b8 00 00 00 80 00    movabs $0x80000000,%r8
  42:   00 00 00
  45:   0f 01 c1                vmcall

Choose 1) for code simplicity and a little bit of code size optimization.

Cc Dave Hansen <[email protected]>
Cc Sean Christopherson <[email protected]>
Cc Dan Williams <[email protected]>
Cc Fengwei Yin <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 arch/x86/include/asm/acrn.h | 57 +++++++++++++++++++++++++++++++++++++
 1 file changed, 57 insertions(+)
arch/x86/include/asm/acrn.h +10 -0
index a2d4aea3a80d..23a93b87edeb 100644
@@ -14,4 +14,61 @@ void acrn_setup_intr_handler(void (*handler)(void));
 void acrn_remove_intr_handler(void);
 bool acrn_is_privileged_vm(void);
 
+/*
+ * Hypercalls for ACRN
+ *
+ * - VMCALL instruction is used to implement ACRN hypercalls.
+ * - ACRN hypercall ABI:
+ *   - Hypercall number is passed in R8 register.
+ *   - Up to 2 arguments are passed in RDI, RSI.
+ *   - Return value will be placed in RAX.
+ */
+static inline long acrn_hypercall0(unsigned long hcall_id)

arch/x86/include/asm/acrn.h void acrn_setup_intr_handler(void (*handler)(void)); · line 36

Quoted 10 lines
arch/x86/include/asm/acrn.h +10 -0
+{
+	register long r8 asm("r8");
+	long result;
+
+	/* Nothing can come between the r8 assignment and the asm: */
+	r8 = hcall_id;
+	asm volatile("vmcall\n\t"
+		     : "=a" (result)
+		     : "r" (r8)
+		     : );

What keeps an interrupt from happening between the r8 assignment and the
asm: ?

Is this something that most hypercalls need to handle? I don't see other ones needing this type of thing, is it just because of how these are defined?

confused,

greg k-h

#16 lore raw

Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces

Quoted 83 lines · Greg Kroah-Hartman
Quoted 68 lines · On Tue, Sep 22, 2020 at 07:42:58PM +0800, [email protected] wrote:

From: Shuo Liu <[email protected]>

The Service VM communicates with the hypervisor via conventional hypercalls. VMCALL instruction is used to make the hypercalls.

ACRN hypercall ABI:
* Hypercall number is in R8 register.
* Up to 2 parameters are in RDI and RSI registers.
* Return value is in RAX register.

Introduce the ACRN hypercall interfaces. Because GCC doesn't support R8 register as direct register constraints, here are two ways to use R8 in extended asm:
1) use explicit register variable as input
2) use supported constraint as input with a explicit MOV to R8 in beginning of asm

The number of instructions of above two ways are same. Asm code from 1)

  38:   41 b8 00 00 00 80       mov    $0x80000000,%r8d
  3e:   48 89 c7                mov    %rax,%rdi
  41:   0f 01 c1                vmcall

Here, writes to the lower dword (%r8d) clear the upper dword of %r8 when the CPU is in 64-bit mode.

Asm code from 2)

  38:   48 89 c7                mov    %rax,%rdi
  3b:   49 b8 00 00 00 80 00    movabs $0x80000000,%r8
  42:   00 00 00
  45:   0f 01 c1                vmcall

Choose 1) for code simplicity and a little bit of code size optimization.

Cc Dave Hansen <[email protected]>
Cc Sean Christopherson <[email protected]>
Cc Dan Williams <[email protected]>
Cc Fengwei Yin <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 arch/x86/include/asm/acrn.h | 57 +++++++++++++++++++++++++++++++++++++
 1 file changed, 57 insertions(+)
arch/x86/include/asm/acrn.h +10 -0
index a2d4aea3a80d..23a93b87edeb 100644
@@ -14,4 +14,61 @@ void acrn_setup_intr_handler(void (*handler)(void));
 void acrn_remove_intr_handler(void);
 bool acrn_is_privileged_vm(void);
 
+/*
+ * Hypercalls for ACRN
+ *
+ * - VMCALL instruction is used to implement ACRN hypercalls.
+ * - ACRN hypercall ABI:
+ *   - Hypercall number is passed in R8 register.
+ *   - Up to 2 arguments are passed in RDI, RSI.
+ *   - Return value will be placed in RAX.
+ */
+static inline long acrn_hypercall0(unsigned long hcall_id)

arch/x86/include/asm/acrn.h void acrn_setup_intr_handler(void (*handler)(void)); · line 36

Quoted 10 lines
arch/x86/include/asm/acrn.h +10 -0
+{
+	register long r8 asm("r8");
+	long result;
+
+	/* Nothing can come between the r8 assignment and the asm: */
+	r8 = hcall_id;
+	asm volatile("vmcall\n\t"
+		     : "=a" (result)
+		     : "r" (r8)
+		     : );

What keeps an interrupt from happening between the r8 assignment and the
asm: ?

Quoted 3 lines

Is this something that most hypercalls need to handle? I don't see other ones needing this type of thing, is it just because of how these are defined?

Ah, the changelog above explains this. You should put that in the code itself, as a comment, otherwise we will not know this at all in 5 years, when gcc is changed to allow r8 access :)

thanks,

greg k-h

author
#17 lore raw

Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces

Quoted 75 lines · Greg Kroah-Hartman
Quoted 74 lines · Greg Kroah-Hartman
Quoted 73 lines · On Tue, Sep 22, 2020 at 07:42:58PM +0800, [email protected] wrote:

From: Shuo Liu <[email protected]>

The Service VM communicates with the hypervisor via conventional hypercalls. VMCALL instruction is used to make the hypercalls.

ACRN hypercall ABI:
* Hypercall number is in R8 register.
* Up to 2 parameters are in RDI and RSI registers.
* Return value is in RAX register.

Introduce the ACRN hypercall interfaces. Because GCC doesn't support R8 register as direct register constraints, here are two ways to use R8 in extended asm:
1) use explicit register variable as input
2) use supported constraint as input with a explicit MOV to R8 in beginning of asm

The number of instructions of above two ways are same. Asm code from 1)

  38:   41 b8 00 00 00 80       mov    $0x80000000,%r8d
  3e:   48 89 c7                mov    %rax,%rdi
  41:   0f 01 c1                vmcall

Here, writes to the lower dword (%r8d) clear the upper dword of %r8 when the CPU is in 64-bit mode.

Asm code from 2)

  38:   48 89 c7                mov    %rax,%rdi
  3b:   49 b8 00 00 00 80 00    movabs $0x80000000,%r8
  42:   00 00 00
  45:   0f 01 c1                vmcall

Choose 1) for code simplicity and a little bit of code size optimization.

Cc Dave Hansen <[email protected]>
Cc Sean Christopherson <[email protected]>
Cc Dan Williams <[email protected]>
Cc Fengwei Yin <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 arch/x86/include/asm/acrn.h | 57 +++++++++++++++++++++++++++++++++++++
 1 file changed, 57 insertions(+)
arch/x86/include/asm/acrn.h +15 -0
index a2d4aea3a80d..23a93b87edeb 100644
@@ -14,4 +14,61 @@ void acrn_setup_intr_handler(void (*handler)(void));
 void acrn_remove_intr_handler(void);
 bool acrn_is_privileged_vm(void);
 
+/*
+ * Hypercalls for ACRN
+ *
+ * - VMCALL instruction is used to implement ACRN hypercalls.
+ * - ACRN hypercall ABI:
+ *   - Hypercall number is passed in R8 register.
+ *   - Up to 2 arguments are passed in RDI, RSI.
+ *   - Return value will be placed in RAX.
+ */
+static inline long acrn_hypercall0(unsigned long hcall_id)
+{
+	register long r8 asm("r8");
+	long result;
+
+	/* Nothing can come between the r8 assignment and the asm: */
Quoted 8 lines
Quoted 8 lines

arch/x86/include/asm/acrn.h void acrn_setup_intr_handler(void (*handler)(void)); · line 36

Quoted 5 lines
arch/x86/include/asm/acrn.h +5 -0
+	r8 = hcall_id;
+	asm volatile("vmcall\n\t"
+		     : "=a" (result)
+		     : "r" (r8)
+		     : );

What keeps an interrupt from happening between the r8 assignment and the
asm: ?

Dave gave a good explanation in another email. I will apply his better comment that "No other C code can come between this r8 assignment and the inline asm".

Quoted 8 lines
Quoted 4 lines

Is this something that most hypercalls need to handle? I don't see other ones needing this type of thing, is it just because of how these are defined?

Ah, the changelog above explains this. You should put that in the code itself, as a comment, otherwise we will not know this at all in 5 years, when gcc is changed to allow r8 access :)

OK. I will copy that into code as well.

Thanks
shuo

#18 lore raw

Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces

In reply to Greg Kroah-Hartman
Quoted 5 lines · Greg Kroah-Hartman
Quoted 5 lines
arch/x86/include/asm/acrn.h +5 -0
+static inline long acrn_hypercall0(unsigned long hcall_id)
+{
+	register long r8 asm("r8");
+	long result;
+
Quoted 8 lines

arch/x86/include/asm/acrn.h void acrn_setup_intr_handler(void (*handler)(void)); · line 36

Quoted 6 lines
arch/x86/include/asm/acrn.h +6 -0
+	/* Nothing can come between the r8 assignment and the asm: */
+	r8 = hcall_id;
+	asm volatile("vmcall\n\t"
+		     : "=a" (result)
+		     : "r" (r8)
+		     : );

What keeps an interrupt from happening between the r8 assignment and the
asm: ?

It's probably better phrased something like: "No other C code can come between this r8 assignment and the inline asm". An interrupt would actually be fine in there because interrupts save and restore all register state, including r8.

The problem (mentioned in the changelog) is that gcc does not let you place data directly into r8. But, it does allow you to declare a register variable that you can assign to use r8. There might be a problem if a function calls was in between and clobber the register, thus the "nothing can come between" comment.

The comment is really intended to scare away anyone from adding printk()'s.

More information about these register variables is here:

Quoted 1 line

https://gcc.gnu.org/onlinedocs/gcc/Local-Register-Variables.html#Local-Register-Variables

Any better ideas for comments would be greatly appreciated. It has 4 or 5 copies so I wanted it to be succinct.

#19 lore raw

Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces

Quoted 26 lines · Dave Hansen
Quoted 5 lines · Greg Kroah-Hartman
Quoted 5 lines
arch/x86/include/asm/acrn.h +5 -0
+static inline long acrn_hypercall0(unsigned long hcall_id)
+{
+	register long r8 asm("r8");
+	long result;
+
Quoted 8 lines

arch/x86/include/asm/acrn.h void acrn_setup_intr_handler(void (*handler)(void)); · line 36

Quoted 6 lines
arch/x86/include/asm/acrn.h +6 -0
+	/* Nothing can come between the r8 assignment and the asm: */
+	r8 = hcall_id;
+	asm volatile("vmcall\n\t"
+		     : "=a" (result)
+		     : "r" (r8)
+		     : );

What keeps an interrupt from happening between the r8 assignment and the
asm: ?

It's probably better phrased something like: "No other C code can come between this r8 assignment and the inline asm". An interrupt would actually be fine in there because interrupts save and restore all register state, including r8.

The problem (mentioned in the changelog) is that gcc does not let you place data directly into r8. But, it does allow you to declare a register variable that you can assign to use r8. There might be a problem if a function calls was in between and clobber the register, thus the "nothing can come between" comment.

Quoted 8 lines

The comment is really intended to scare away anyone from adding printk()'s.

More information about these register variables is here:

Quoted 1 line

https://gcc.gnu.org/onlinedocs/gcc/Local-Register-Variables.html#Local-Register-Variables

Any better ideas for comments would be greatly appreciated. It has 4 or 5 copies so I wanted it to be succinct.

This is disguisting.. Segher, does this actually work? Nick, does clang also support this?

#20 lore raw

Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces

Hi!

Quoted 31 lines · Peter Zijlstra
Quoted 30 lines · Dave Hansen
Quoted 5 lines · Greg Kroah-Hartman
Quoted 5 lines
arch/x86/include/asm/acrn.h +5 -0
+static inline long acrn_hypercall0(unsigned long hcall_id)
+{
+	register long r8 asm("r8");
+	long result;
+
Quoted 8 lines

arch/x86/include/asm/acrn.h void acrn_setup_intr_handler(void (*handler)(void)); · line 36

Quoted 6 lines
arch/x86/include/asm/acrn.h +6 -0
+	/* Nothing can come between the r8 assignment and the asm: */
+	r8 = hcall_id;
+	asm volatile("vmcall\n\t"
+		     : "=a" (result)
+		     : "r" (r8)
+		     : );

What keeps an interrupt from happening between the r8 assignment and the
asm: ?

It's probably better phrased something like: "No other C code can come between this r8 assignment and the inline asm". An interrupt would actually be fine in there because interrupts save and restore all register state, including r8.

The problem (mentioned in the changelog) is that gcc does not let you place data directly into r8. But, it does allow you to declare a register variable that you can assign to use r8. There might be a problem if a function calls was in between and clobber the register, thus the "nothing can come between" comment.

The comment is really intended to scare away anyone from adding printk()'s.

More information about these register variables is here:

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

https://gcc.gnu.org/onlinedocs/gcc/Local-Register-Variables.html#Local-Register-Variables

Any better ideas for comments would be greatly appreciated. It has 4 or 5 copies so I wanted it to be succinct.

This is disguisting.. Segher, does this actually work? Nick, does clang also support this?

The C variable "r8" is just a variable like any other; it can live in memory, or in any register, and different in all places, too. It can be moved around too; where "the assignment to it" happens is a philosophical question more than anything (the assignment there can be optimised away completely, for example; it is just a C variable, there is no magic).

Since this variable is a local register asm, on entry to the asm the compiler guarantees that the value lives in the assigned register (the "r8" hardware register in this case). This all works completely fine. This is the only guaranteed behaviour for local register asm (well, together with analogous behaviour for outputs).

If you want to *always* have it live in the hardware reg "r8", you have to use a global register asm, and almost certainly do that for all translation units, and use -ffixed-r8 as well. This of course is extremely costly.

Segher

#21 lore raw

Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces

Quoted 5 lines · Segher Boessenkool

Since this variable is a local register asm, on entry to the asm the compiler guarantees that the value lives in the assigned register (the "r8" hardware register in this case). This all works completely fine. This is the only guaranteed behaviour for local register asm (well, together with analogous behaviour for outputs).

Right, that's what they're trying to achieve. The hypervisor calling convention needs that variable in %r8 (which is somewhat unfortunate).

AFAIK this is the first such use in the kernel, but at least the gcc-4.9 (our oldest supported version) claims to support this.

So now we need to know if clang will actually do this too..

#22 lore raw

Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces

Quoted 9 lines · Peter Zijlstra
Quoted 5 lines · Segher Boessenkool

Since this variable is a local register asm, on entry to the asm the compiler guarantees that the value lives in the assigned register (the "r8" hardware register in this case). This all works completely fine. This is the only guaranteed behaviour for local register asm (well, together with analogous behaviour for outputs).

Quoted 7 lines

Right, that's what they're trying to achieve. The hypervisor calling convention needs that variable in %r8 (which is somewhat unfortunate).

AFAIK this is the first such use in the kernel, but at least the gcc-4.9 (our oldest supported version) claims to support this.

So now we need to know if clang will actually do this too..

Does clang support register local storage? Let's use godbolt.org to find out:
https://godbolt.org/z/YM45W5
Looks like yes. You can even check different GCC versions via the dropdown in the top right.

The -ffixed-* flags are less well supported in Clang; they need to be reimplemented on a per-backend basis. aarch64 is relatively well supported, but other arches not so much IME.

Do we need register local storage here?

static inline long bar(unsigned long hcall_id)
{
long result;
asm volatile("movl %1, %%r8d\n\t"
"vmcall\n\t"

    : "=a" (result)
    : "ir" (hcall_id)
    : );

return result;
}

Signature · 2 lines
Thanks,
~Nick Desaulniers
#23 lore raw

Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces

Quoted 16 lines · Nick Desaulniers
Quoted 15 lines · Peter Zijlstra
Quoted 5 lines · Segher Boessenkool

Since this variable is a local register asm, on entry to the asm the compiler guarantees that the value lives in the assigned register (the "r8" hardware register in this case). This all works completely fine. This is the only guaranteed behaviour for local register asm (well, together with analogous behaviour for outputs).

Right, that's what they're trying to achieve. The hypervisor calling convention needs that variable in %r8 (which is somewhat unfortunate).

AFAIK this is the first such use in the kernel, but at least the gcc-4.9 (our oldest supported version) claims to support this.

Quoted 6 lines
Quoted 1 line

So now we need to know if clang will actually do this too..

Does clang support register local storage? Let's use godbolt.org to find out:
https://godbolt.org/z/YM45W5
Looks like yes. You can even check different GCC versions via the dropdown in the top right.

That only tells me it compiles it, not if that (IMO) weird construct is actually guaranteed to work as expected.

I'd almost dive into the GCC archives to read the back-story to this 'feature', it just seems to weird to me. A well, for another day that.

#24 lore raw

Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces

Quoted 8 lines · Peter Zijlstra
Quoted 2 lines

Looks like yes. You can even check different GCC versions via the dropdown in the top right.

That only tells me it compiles it, not if that (IMO) weird construct is actually guaranteed to work as expected.

I'd almost dive into the GCC archives to read the back-story to this 'feature', it just seems to weird to me. A well, for another day that.

It was documented in 1996 (<https://gcc.gnu.org/g:c1f7febfcb10>), the feature is older than that though.

In 2004 (in <https://gcc.gnu.org/g:805c33df1366>) the documentation for this was made more explicit (it has been rewritten since, but it still says the same thing).

Segher

#25 lore raw

Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces

In reply to Nick Desaulniers
Quoted 9 lines · Nick Desaulniers
Quoted 8 lines · Peter Zijlstra
Quoted 5 lines · Segher Boessenkool

Since this variable is a local register asm, on entry to the asm the compiler guarantees that the value lives in the assigned register (the "r8" hardware register in this case). This all works completely fine. This is the only guaranteed behaviour for local register asm (well, together with analogous behaviour for outputs).

How strict is the guarantee? This is an inline function -- could the compiler decide to reorder some other code in between the r8 assignment and the asm statement when it gets inlined?

Quoted 25 lines
Quoted 8 lines

Right, that's what they're trying to achieve. The hypervisor calling convention needs that variable in %r8 (which is somewhat unfortunate).

AFAIK this is the first such use in the kernel, but at least the gcc-4.9 (our oldest supported version) claims to support this.

So now we need to know if clang will actually do this too..

Does clang support register local storage? Let's use godbolt.org to find out:
https://godbolt.org/z/YM45W5
Looks like yes. You can even check different GCC versions via the dropdown in the top right.

The -ffixed-* flags are less well supported in Clang; they need to be reimplemented on a per-backend basis. aarch64 is relatively well supported, but other arches not so much IME.

Do we need register local storage here?

static inline long bar(unsigned long hcall_id)
{
long result;
asm volatile("movl %1, %%r8d\n\t"
"vmcall\n\t"

Quoted 5 lines
    : "=a" (result)
    : "ir" (hcall_id)
    : );

return result;
}

This seems more robust, though you probably need an r8 clobber in there? Is hcall_id actually just 32 bits or can it be >=2^32?

#26 lore raw

Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces

Quoted 46 lines · Arvind Sankar
Quoted 9 lines · Nick Desaulniers
Quoted 8 lines · Peter Zijlstra
Quoted 5 lines · Segher Boessenkool

Since this variable is a local register asm, on entry to the asm the compiler guarantees that the value lives in the assigned register (the "r8" hardware register in this case). This all works completely fine. This is the only guaranteed behaviour for local register asm (well, together with analogous behaviour for outputs).

How strict is the guarantee? This is an inline function -- could the compiler decide to reorder some other code in between the r8 assignment and the asm statement when it gets inlined?

Quoted 25 lines
Quoted 8 lines

Right, that's what they're trying to achieve. The hypervisor calling convention needs that variable in %r8 (which is somewhat unfortunate).

AFAIK this is the first such use in the kernel, but at least the gcc-4.9 (our oldest supported version) claims to support this.

So now we need to know if clang will actually do this too..

Does clang support register local storage? Let's use godbolt.org to find out:
https://godbolt.org/z/YM45W5
Looks like yes. You can even check different GCC versions via the dropdown in the top right.

The -ffixed-* flags are less well supported in Clang; they need to be reimplemented on a per-backend basis. aarch64 is relatively well supported, but other arches not so much IME.

Do we need register local storage here?

static inline long bar(unsigned long hcall_id)
{
long result;
asm volatile("movl %1, %%r8d\n\t"
"vmcall\n\t"

Quoted 5 lines
    : "=a" (result)
    : "ir" (hcall_id)
    : );

return result;
}

Quoted 2 lines

This seems more robust, though you probably need an r8 clobber in there? Is hcall_id actually just 32 bits or can it be >=2^32?

Also, I think you need memory clobbers for all of these in either case, no?

#27 lore raw

Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces

In reply to Arvind Sankar
Quoted 11 lines · Arvind Sankar
Quoted 9 lines · Nick Desaulniers
Quoted 8 lines · Peter Zijlstra
Quoted 5 lines · Segher Boessenkool

Since this variable is a local register asm, on entry to the asm the compiler guarantees that the value lives in the assigned register (the "r8" hardware register in this case). This all works completely fine. This is the only guaranteed behaviour for local register asm (well, together with analogous behaviour for outputs).

Quoted 3 lines

How strict is the guarantee? This is an inline function -- could the compiler decide to reorder some other code in between the r8 assignment and the asm statement when it gets inlined?

Nope. It will be in r8 on entry to the asm. A guarantee is a guarantee; it is not a "yeah maybe, we'll see".

Quoted 13 lines
Quoted 7 lines

Do we need register local storage here?

static inline long bar(unsigned long hcall_id)
{
long result;
asm volatile("movl %1, %%r8d\n\t"
"vmcall\n\t"

Quoted 5 lines
    : "=a" (result)
    : "ir" (hcall_id)
    : );

return result;
}

Quoted 1 line

This seems more robust, though you probably need an r8 clobber in there?

Oh, x86 has the operand order inverted, so this should work in fact.

Segher

#28 lore raw

Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces

In reply to Nick Desaulniers
Quoted 1 line · Nick Desaulniers

Do we need register local storage here?

Depends what you want. It looks like you do:

Quoted 10 lines

static inline long bar(unsigned long hcall_id)
{
long result;
asm volatile("movl %1, %%r8d\n\t"
"vmcall\n\t"

    : "=a" (result)
    : "ir" (hcall_id)
    : );

return result;
}

"result" as output from the asm is in %rax, and the compiler will shuffle that to wherever it needs it as the function return value. That part will work fine.

But how you are accessing %r8d is not correct, that needs to be a local register asm (or r8 be made a fixed reg, probably not what you want ;-) )

Segher

#29 lore raw

Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces

Quoted 16 lines · Segher Boessenkool
Quoted 1 line · Nick Desaulniers

Do we need register local storage here?

Depends what you want. It looks like you do:

Quoted 10 lines

static inline long bar(unsigned long hcall_id)
{
long result;
asm volatile("movl %1, %%r8d\n\t"
"vmcall\n\t"

    : "=a" (result)
    : "ir" (hcall_id)
    : );

return result;
}

Quoted 7 lines

"result" as output from the asm is in %rax, and the compiler will shuffle that to wherever it needs it as the function return value. That part will work fine.

But how you are accessing %r8d is not correct, that needs to be a local register asm (or r8 be made a fixed reg, probably not what you want ;-) )

Doesn't it just need an "r8" clobber to allow using r8d?

Quoted 2 lines

Segher

#30 lore raw

Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces

Quoted 17 lines · Arvind Sankar
Quoted 16 lines · Segher Boessenkool
Quoted 1 line · Nick Desaulniers

Do we need register local storage here?

Depends what you want. It looks like you do:

Quoted 10 lines

static inline long bar(unsigned long hcall_id)
{
long result;
asm volatile("movl %1, %%r8d\n\t"
"vmcall\n\t"

    : "=a" (result)
    : "ir" (hcall_id)
    : );

return result;
}

Quoted 8 lines
Quoted 6 lines

"result" as output from the asm is in %rax, and the compiler will shuffle that to wherever it needs it as the function return value. That part will work fine.

But how you are accessing %r8d is not correct, that needs to be a local register asm (or r8 be made a fixed reg, probably not what you want ;-) )

Doesn't it just need an "r8" clobber to allow using r8d?

Yes, x86 asm is hard to read, what can I say :-) Sorry about that.

Segher

author
#31 lore raw

Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces

In reply to Nick Desaulniers
Quoted 29 lines · Nick Desaulniers
Quoted 9 lines · Peter Zijlstra
Quoted 5 lines · Segher Boessenkool

Since this variable is a local register asm, on entry to the asm the compiler guarantees that the value lives in the assigned register (the "r8" hardware register in this case). This all works completely fine. This is the only guaranteed behaviour for local register asm (well, together with analogous behaviour for outputs).

Quoted 7 lines

Right, that's what they're trying to achieve. The hypervisor calling convention needs that variable in %r8 (which is somewhat unfortunate).

AFAIK this is the first such use in the kernel, but at least the gcc-4.9 (our oldest supported version) claims to support this.

So now we need to know if clang will actually do this too..

Does clang support register local storage? Let's use godbolt.org to find out:
https://godbolt.org/z/YM45W5
Looks like yes. You can even check different GCC versions via the dropdown in the top right.

The -ffixed-* flags are less well supported in Clang; they need to be reimplemented on a per-backend basis. aarch64 is relatively well supported, but other arches not so much IME.

Do we need register local storage here?

Quoted 10 lines

static inline long bar(unsigned long hcall_id)
{
long result;
asm volatile("movl %1, %%r8d\n\t"
"vmcall\n\t"
: "=a" (result)
: "ir" (hcall_id)
: );
return result;
}

Yeah, this approach is also mentioned in the changelog. I will change to this way to follow your preference. With an addtional "r8" clobber what Arvind mentioned.

Thanks
shuo

#32 lore raw

Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces

Quoted 41 lines · Shuo A Liu
Quoted 29 lines · Nick Desaulniers
Quoted 9 lines · Peter Zijlstra
Quoted 5 lines · Segher Boessenkool

Since this variable is a local register asm, on entry to the asm the compiler guarantees that the value lives in the assigned register (the "r8" hardware register in this case). This all works completely fine. This is the only guaranteed behaviour for local register asm (well, together with analogous behaviour for outputs).

Quoted 7 lines

Right, that's what they're trying to achieve. The hypervisor calling convention needs that variable in %r8 (which is somewhat unfortunate).

AFAIK this is the first such use in the kernel, but at least the gcc-4.9 (our oldest supported version) claims to support this.

So now we need to know if clang will actually do this too..

Does clang support register local storage? Let's use godbolt.org to find out:
https://godbolt.org/z/YM45W5
Looks like yes. You can even check different GCC versions via the dropdown in the top right.

The -ffixed-* flags are less well supported in Clang; they need to be reimplemented on a per-backend basis. aarch64 is relatively well supported, but other arches not so much IME.

Do we need register local storage here?

Quoted 10 lines

static inline long bar(unsigned long hcall_id)
{
long result;
asm volatile("movl %1, %%r8d\n\t"
"vmcall\n\t"
: "=a" (result)
: "ir" (hcall_id)
: );
return result;
}

Quoted 6 lines

Yeah, this approach is also mentioned in the changelog. I will change to this way to follow your preference. With an addtional "r8" clobber what Arvind mentioned.

Thanks
shuo

Btw, I noticed that arch/x86/xen/hypercall.h uses register-local variables already for its hypercalls for quite some time, so this wouldn't be unprecedented. [0]

Do these calls also need a memory clobber? The KVM/xen hypercall functions all have one.

Thanks.

[0] e74359028d548 ("xen64: fix calls into hypercall page")

author
#33 lore raw

Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces

Quoted 49 lines · Arvind Sankar
Quoted 41 lines · Shuo A Liu
Quoted 29 lines · Nick Desaulniers
Quoted 9 lines · Peter Zijlstra
Quoted 5 lines · Segher Boessenkool

Since this variable is a local register asm, on entry to the asm the compiler guarantees that the value lives in the assigned register (the "r8" hardware register in this case). This all works completely fine. This is the only guaranteed behaviour for local register asm (well, together with analogous behaviour for outputs).

Quoted 7 lines

Right, that's what they're trying to achieve. The hypervisor calling convention needs that variable in %r8 (which is somewhat unfortunate).

AFAIK this is the first such use in the kernel, but at least the gcc-4.9 (our oldest supported version) claims to support this.

So now we need to know if clang will actually do this too..

Does clang support register local storage? Let's use godbolt.org to find out:
https://godbolt.org/z/YM45W5
Looks like yes. You can even check different GCC versions via the dropdown in the top right.

The -ffixed-* flags are less well supported in Clang; they need to be reimplemented on a per-backend basis. aarch64 is relatively well supported, but other arches not so much IME.

Do we need register local storage here?

Quoted 10 lines

static inline long bar(unsigned long hcall_id)
{
long result;
asm volatile("movl %1, %%r8d\n\t"
"vmcall\n\t"
: "=a" (result)
: "ir" (hcall_id)
: );
return result;
}

Quoted 6 lines

Yeah, this approach is also mentioned in the changelog. I will change to this way to follow your preference. With an addtional "r8" clobber what Arvind mentioned.

Thanks
shuo

Quoted 6 lines

Btw, I noticed that arch/x86/xen/hypercall.h uses register-local variables already for its hypercalls for quite some time, so this wouldn't be unprecedented. [0]

Do these calls also need a memory clobber? The KVM/xen hypercall functions all have one.

Yes. it's needed. I will add it. Thanks

Quoted 4 lines

Thanks.

[0] e74359028d548 ("xen64: fix calls into hypercall page")

maintainer
#34 lore raw

Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces

In reply to [email protected]
Quoted 9 lines · On Tue, Sep 22, 2020 at 07:42:58PM +0800, [email protected] wrote:

From: Shuo Liu <[email protected]>

The Service VM communicates with the hypervisor via conventional hypercalls. VMCALL instruction is used to make the hypercalls.

ACRN hypercall ABI:
* Hypercall number is in R8 register.
* Up to 2 parameters are in RDI and RSI registers.
* Return value is in RAX register.

I'm assuming this is already cast in stone in the HV and it cannot be changed?

Quoted 15 lines

Introduce the ACRN hypercall interfaces. Because GCC doesn't support R8 register as direct register constraints, here are two ways to use R8 in extended asm:
1) use explicit register variable as input
2) use supported constraint as input with a explicit MOV to R8 in beginning of asm

The number of instructions of above two ways are same. Asm code from 1)

  38:   41 b8 00 00 00 80       mov    $0x80000000,%r8d
  3e:   48 89 c7                mov    %rax,%rdi
  41:   0f 01 c1                vmcall

Here, writes to the lower dword (%r8d) clear the upper dword of %r8 when the CPU is in 64-bit mode.

Quoted 8 lines

Asm code from 2)

  38:   48 89 c7                mov    %rax,%rdi
  3b:   49 b8 00 00 00 80 00    movabs $0x80000000,%r8
  42:   00 00 00
  45:   0f 01 c1                vmcall

Choose 1) for code simplicity and a little bit of code size optimization.

What?

How much "optimization" is this actually? A couple of bytes?

And all that for this

	/* Nothing can come between the r8 assignment and the asm: */

restriction?

If it is only a couple of bytes, just do the explicit MOV to %r8 and f'get about it.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
author
#35 lore raw

Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces

Quoted 6 lines · Borislav Petkov
Quoted 5 lines · On Tue, Sep 22, 2020 at 07:42:58PM +0800, [email protected] wrote:

From: Shuo Liu <[email protected]>

The Service VM communicates with the hypervisor via conventional hypercalls. VMCALL instruction is used to make the hypercalls.

Quoted 7 lines
Quoted 4 lines

ACRN hypercall ABI:
* Hypercall number is in R8 register.
* Up to 2 parameters are in RDI and RSI registers.
* Return value is in RAX register.

I'm assuming this is already cast in stone in the HV and it cannot be changed?

Yes, it is.

Quoted 29 lines
Quoted 16 lines

Introduce the ACRN hypercall interfaces. Because GCC doesn't support R8 register as direct register constraints, here are two ways to use R8 in extended asm:
1) use explicit register variable as input
2) use supported constraint as input with a explicit MOV to R8 in beginning of asm

The number of instructions of above two ways are same. Asm code from 1)

  38:   41 b8 00 00 00 80       mov    $0x80000000,%r8d
  3e:   48 89 c7                mov    %rax,%rdi
  41:   0f 01 c1                vmcall

Here, writes to the lower dword (%r8d) clear the upper dword of %r8 when the CPU is in 64-bit mode.

Quoted 8 lines

Asm code from 2)

  38:   48 89 c7                mov    %rax,%rdi
  3b:   49 b8 00 00 00 80 00    movabs $0x80000000,%r8
  42:   00 00 00
  45:   0f 01 c1                vmcall

Choose 1) for code simplicity and a little bit of code size optimization.

What?

How much "optimization" is this actually? A couple of bytes?

Quoted 8 lines

And all that for this

	/* Nothing can come between the r8 assignment and the asm: */

restriction?

If it is only a couple of bytes, just do the explicit MOV to %r8 and f'get about it.

Yes. Just a couple of bytes. Number of instructions is same. sure, i can change to approach 2)
2) use supported constraint as input with a explicit MOV to R8 in beginning of asm

Thanks
shuo

#36 lore raw

PATCH v4 5/17 virt: acrn: Introduce ACRN HSM basic driver

In reply to [email protected]

From: Shuo Liu <[email protected]>

ACRN Hypervisor Service Module (HSM) is a kernel module in Service VM which communicates with ACRN userspace through ioctls and talks to ACRN Hypervisor through hypercalls.

Add a basic HSM driver which allows Service VM userspace to communicate with ACRN. The following patches will add more ioctls, guest VM memory mapping caching, I/O request processing, ioeventfd and irqfd into this module. HSM exports a char device interface (/dev/acrn_hsm) to userspace.

Cc Dave Hansen <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 MAINTAINERS                  |  1 +
 drivers/virt/Kconfig         |  2 +
 drivers/virt/Makefile        |  1 +
 drivers/virt/acrn/Kconfig    | 14 ++++++
 drivers/virt/acrn/Makefile   |  3 ++
 drivers/virt/acrn/acrn_drv.h | 18 ++++++++
 drivers/virt/acrn/hsm.c      | 87 ++++++++++++++++++++++++++++++++++++
 7 files changed, 126 insertions(+)
 create mode 100644 drivers/virt/acrn/Kconfig
 create mode 100644 drivers/virt/acrn/Makefile
 create mode 100644 drivers/virt/acrn/acrn_drv.h
 create mode 100644 drivers/virt/acrn/hsm.c
MAINTAINERS +1 -0
index e0fea5e464b4..3030d0e93d02 100644
@@ -442,6 +442,7 @@ L:	[email protected]
 S:	Supported
 W:	https://projectacrn.org
 F:	Documentation/virt/acrn/
+F:	drivers/virt/acrn/
 
 AD1889 ALSA SOUND DRIVER
 L:	[email protected]
drivers/virt/Kconfig +2 -0
index cbc1f25c79ab..d9484a2e9b46 100644
@@ -32,4 +32,6 @@ config FSL_HV_MANAGER
 	     partition shuts down.
 
 source "drivers/virt/vboxguest/Kconfig"
+
+source "drivers/virt/acrn/Kconfig"
 endif
drivers/virt/Makefile +1 -0
index fd331247c27a..f0491bbf0d4d 100644
@@ -5,3 +5,4 @@
 
 obj-$(CONFIG_FSL_HV_MANAGER)	+= fsl_hypervisor.o
 obj-y				+= vboxguest/
+obj-$(CONFIG_ACRN_HSM)		+= acrn/
drivers/virt/acrn/Kconfig new file +14 -0
new file mode 100644
index 000000000000..36c80378c30c
@@ -0,0 +1,14 @@
+# SPDX-License-Identifier: GPL-2.0
+config ACRN_HSM
+	tristate "ACRN Hypervisor Service Module"
+	depends on ACRN_GUEST
+	help
+	  ACRN Hypervisor Service Module (HSM) is a kernel module which
+	  communicates with ACRN userspace through ioctls and talks to
+	  the ACRN Hypervisor through hypercalls. HSM will only run in
+	  a privileged management VM, called Service VM, to manage User
+	  VMs and do I/O emulation. Not required for simply running
+	  under ACRN as a User VM.
+
+	  To compile as a module, choose M, the module will be called
+	  acrn. If unsure, say N.
drivers/virt/acrn/Makefile new file +3 -0
new file mode 100644
index 000000000000..6920ed798aaf
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_ACRN_HSM)	:= acrn.o
+acrn-y := hsm.o
drivers/virt/acrn/acrn_drv.h new file +18 -0
new file mode 100644
index 000000000000..29eedd696327
@@ -0,0 +1,18 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#ifndef __ACRN_HSM_DRV_H
+#define __ACRN_HSM_DRV_H
+
+#include <linux/types.h>
+
+#define ACRN_INVALID_VMID (0xffffU)
+
+/**
+ * struct acrn_vm - Properties of ACRN User VM.
+ * @vmid:	User VM ID
+ */
+struct acrn_vm {
+	u16	vmid;
+};
+
+#endif /* __ACRN_HSM_DRV_H */
drivers/virt/acrn/hsm.c new file +87 -0

87 lines not on this page · raw message

Signature · 1 line
2.28.0
#37 lore raw

PATCH v4 8/17 virt: acrn: Introduce EPT mapping management

In reply to [email protected]

From: Shuo Liu <[email protected]>

The HSM provides hypervisor services to the ACRN userspace. While launching a User VM, ACRN userspace needs to allocate memory and request the ACRN Hypervisor to set up the EPT mapping for the VM.

A mapping cache is introduced for accelerating the translation between the Service VM kernel virtual address and User VM physical address.

From the perspective of the hypervisor, the types of GPA of User VM can be listed as following:
1) RAM region, which is used by User VM as system ram.
2) MMIO region, which is recognized by User VM as MMIO. MMIO region is used to be utilized for devices emulation.

Generally, User VM RAM regions mapping is set up before VM started and is released in the User VM destruction. MMIO regions mapping may be set and unset dynamically during User VM running.

To achieve this, ioctls ACRN_IOCTL_SET_MEMSEG and ACRN_IOCTL_UNSET_MEMSEG are introduced in HSM.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 drivers/virt/acrn/Makefile    |   2 +-
 drivers/virt/acrn/acrn_drv.h  |  98 ++++++++++-
 drivers/virt/acrn/hsm.c       |  15 ++
 drivers/virt/acrn/hypercall.h |  14 ++
 drivers/virt/acrn/mm.c        | 305 ++++++++++++++++++++++++++++++++++
 drivers/virt/acrn/vm.c        |   4 +
 include/uapi/linux/acrn.h     |  51 ++++++
 7 files changed, 479 insertions(+), 10 deletions(-)
 create mode 100644 drivers/virt/acrn/mm.c
drivers/virt/acrn/Makefile +1 -1
index cf8b4ed5e74e..38bc44b6edcd 100644
@@ -1,3 +1,3 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_ACRN_HSM)	:= acrn.o
-acrn-y := hsm.o vm.o
+acrn-y := hsm.o vm.o mm.o
drivers/virt/acrn/acrn_drv.h +89 -9

116 lines not on this page · raw message

drivers/virt/acrn/hsm.c +15 -0
index e45f3abbc87f..ea5dfd04163b 100644
@@ -48,6 +48,7 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 	struct acrn_vm *vm = filp->private_data;
 	struct acrn_vm_creation *vm_param;
 	struct acrn_vcpu_regs *cpu_regs;
+	struct acrn_vm_memmap memmap;
 	int ret = 0;
 
 	if (vm->vmid == ACRN_INVALID_VMID && cmd != ACRN_IOCTL_CREATE_VM) {
@@ -112,6 +113,20 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 				vm->vmid);
 		kfree(cpu_regs);
 		break;
+	case ACRN_IOCTL_SET_MEMSEG:
+		if (copy_from_user(&memmap, (void __user *)ioctl_param,
+				   sizeof(memmap)))
+			return -EFAULT;
+
+		ret = acrn_vm_memseg_map(vm, &memmap);
+		break;
+	case ACRN_IOCTL_UNSET_MEMSEG:
+		if (copy_from_user(&memmap, (void __user *)ioctl_param,
+				   sizeof(memmap)))
+			return -EFAULT;
+
+		ret = acrn_vm_memseg_unmap(vm, &memmap);
+		break;
 	default:
 		dev_warn(acrn_dev.this_device, "Unknown IOCTL 0x%x!\n", cmd);
 		ret = -ENOTTY;
drivers/virt/acrn/hypercall.h +14 -0
index f29cfae08862..a1a70a071713 100644
@@ -21,6 +21,9 @@
 #define HC_RESET_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x05)
 #define HC_SET_VCPU_REGS		_HC_ID(HC_ID, HC_ID_VM_BASE + 0x06)
 
+#define HC_ID_MEM_BASE			0x40UL
+#define HC_VM_SET_MEMORY_REGIONS	_HC_ID(HC_ID, HC_ID_MEM_BASE + 0x02)
+
 /**
  * hcall_create_vm() - Create a User VM
  * @vminfo:	Service VM GPA of info of User VM creation
@@ -88,4 +91,15 @@ static inline long hcall_set_vcpu_regs(u64 vmid, u64 regs_state)
 	return acrn_hypercall2(HC_SET_VCPU_REGS, vmid, regs_state);
 }
 
+/**
+ * hcall_set_memory_regions() - Inform the hypervisor to set up EPT mappings
+ * @regions_pa:	Service VM GPA of &struct vm_memory_region_batch
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_set_memory_regions(u64 regions_pa)
+{
+	return acrn_hypercall1(HC_VM_SET_MEMORY_REGIONS, regions_pa);
+}
+
 #endif /* __ACRN_HSM_HYPERCALL_H */
drivers/virt/acrn/mm.c new file +305 -0

305 lines not on this page · raw message

drivers/virt/acrn/vm.c +4 -0
index 920ca48f4847..c088362cc3e3 100644
@@ -34,6 +34,7 @@ struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
 		return NULL;
 	}
 
+	mutex_init(&vm->regions_mapping_lock);
 	vm->vmid = vm_param->vmid;
 	vm->vcpu_num = vm_param->vcpu_num;
 
@@ -65,6 +66,9 @@ int acrn_vm_destroy(struct acrn_vm *vm)
 		clear_bit(ACRN_VM_FLAG_DESTROYED, &vm->flags);
 		return ret;
 	}
+
+	acrn_vm_all_ram_unmap(vm);
+
 	dev_dbg(acrn_dev.this_device, "VM %u destroyed.\n", vm->vmid);
 	vm->vmid = ACRN_INVALID_VMID;
 	return 0;
include/uapi/linux/acrn.h +51 -0

61 lines not on this page · raw message

Signature · 1 line
2.28.0
#38 lore raw

PATCH v4 10/17 virt: acrn: Introduce PCI configuration space PIO accesses combiner

In reply to [email protected]

From: Shuo Liu <[email protected]>

A User VM can access its virtual PCI configuration spaces via port IO approach, which has two following steps:
1) writes address into port 0xCF8
2) put/get data in/from port 0xCFC

To distribute a complete PCI configuration space access one time, HSM need to combine such two accesses together.

Combine two paired PIO I/O requests into one PCI I/O request and continue the I/O request distribution.

---
 drivers/virt/acrn/acrn_drv.h |  2 +
 drivers/virt/acrn/ioreq.c    | 76 ++++++++++++++++++++++++++++++++++++
 include/uapi/linux/acrn.h    | 15 +++++++
 3 files changed, 93 insertions(+)
drivers/virt/acrn/acrn_drv.h +2 -0
index cf9143cf760d..97d2aab8b70a 100644
@@ -156,6 +156,7 @@ extern rwlock_t acrn_vm_list_lock;
  * @default_client:		The default I/O request client
  * @ioreq_buf:			I/O request shared buffer
  * @ioreq_page:			The page of the I/O request shared buffer
+ * @pci_conf_addr:		Address of a PCI configuration access emulation
  */
 struct acrn_vm {
 	struct list_head		list;
@@ -170,6 +171,7 @@ struct acrn_vm {
 	struct acrn_ioreq_client	*default_client;
 	struct acrn_io_request_buffer	*ioreq_buf;
 	struct page			*ioreq_page;
+	u32				pci_conf_addr;
 };
 
 struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
drivers/virt/acrn/ioreq.c +76 -0

88 lines not on this page · raw message

include/uapi/linux/acrn.h +15 -0
index 8eb687f1482c..31cf0fd73bcc 100644
@@ -20,6 +20,7 @@
 
 #define ACRN_IOREQ_TYPE_PORTIO		0
 #define ACRN_IOREQ_TYPE_MMIO		1
+#define ACRN_IOREQ_TYPE_PCICFG		2
 
 #define ACRN_IOREQ_DIR_READ		0
 #define ACRN_IOREQ_DIR_WRITE		1
@@ -40,6 +41,18 @@ struct acrn_pio_request {
 	__u32	value;
 } __attribute__((aligned(8)));
 
+/* Need keep same header fields with pio_request */
+struct acrn_pci_request {
+	__u32	direction;
+	__u32	reserved[3];
+	__u64	size;
+	__u32	value;
+	__u32	bus;
+	__u32	dev;
+	__u32	func;
+	__u32	reg;
+} __attribute__((aligned(8)));
+
 /**
  * struct acrn_io_request - 256-byte ACRN I/O request
  * @type:		Type of this request (ACRN_IOREQ_TYPE_*).
@@ -48,6 +61,7 @@ struct acrn_pio_request {
  * @reserved0:		Reserved fields.
  * @reqs:		Union of different types of request. Byte offset: 64.
  * @reqs.pio_request:	PIO request data of the I/O request.
+ * @reqs.pci_request:	PCI configuration space request data of the I/O request.
  * @reqs.mmio_request:	MMIO request data of the I/O request.
  * @reqs.data:		Raw data of the I/O request.
  * @reserved1:		Reserved fields.
@@ -107,6 +121,7 @@ struct acrn_io_request {
 	__u32	reserved0[14];
 	union {
 		struct acrn_pio_request		pio_request;
+		struct acrn_pci_request		pci_request;
 		struct acrn_mmio_request	mmio_request;
 		__u64				data[8];
 	} reqs;
Signature · 1 line
2.28.0
#39 lore raw

PATCH v4 11/17 virt: acrn: Introduce interfaces for PCI device passthrough

In reply to [email protected]

From: Shuo Liu <[email protected]>

PCI device passthrough enables an OS in a virtual machine to directly access a PCI device in the host. It promises almost the native performance, which is required in performance-critical scenarios of ACRN.

HSM provides the following ioctls:
- Assign - ACRN_IOCTL_ASSIGN_PCIDEV
Pass data struct acrn_pcidev from userspace to the hypervisor, and inform the hypervisor to assign a PCI device to a User VM.

- De-assign - ACRN_IOCTL_DEASSIGN_PCIDEV
Pass data struct acrn_pcidev from userspace to the hypervisor, and inform the hypervisor to de-assign a PCI device from a User VM.

- Set a interrupt of a passthrough device - ACRN_IOCTL_SET_PTDEV_INTR Pass data struct acrn_ptdev_irq from userspace to the hypervisor, and inform the hypervisor to map a INTx interrupt of passthrough device of User VM.

- Reset passthrough device interrupt - ACRN_IOCTL_RESET_PTDEV_INTR Pass data struct acrn_ptdev_irq from userspace to the hypervisor, and inform the hypervisor to unmap a INTx interrupt of passthrough device of User VM.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 drivers/virt/acrn/hsm.c       | 50 +++++++++++++++++++++++++++
 drivers/virt/acrn/hypercall.h | 54 ++++++++++++++++++++++++++++++
 include/uapi/linux/acrn.h     | 63 +++++++++++++++++++++++++++++++++++
 3 files changed, 167 insertions(+)
drivers/virt/acrn/hsm.c +50 -0
index b4b5d99ff6bb..bcb5e273dcc3 100644
@@ -49,7 +49,9 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 	struct acrn_vm_creation *vm_param;
 	struct acrn_vcpu_regs *cpu_regs;
 	struct acrn_ioreq_notify notify;
+	struct acrn_ptdev_irq *irq_info;
 	struct acrn_vm_memmap memmap;
+	struct acrn_pcidev *pcidev;
 	int ret = 0;
 
 	if (vm->vmid == ACRN_INVALID_VMID && cmd != ACRN_IOCTL_CREATE_VM) {
@@ -128,6 +130,54 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 
 		ret = acrn_vm_memseg_unmap(vm, &memmap);
 		break;
+	case ACRN_IOCTL_ASSIGN_PCIDEV:
+		pcidev = memdup_user((void __user *)ioctl_param,
+				     sizeof(struct acrn_pcidev));
+		if (IS_ERR(pcidev))
+			return PTR_ERR(pcidev);
+
+		ret = hcall_assign_pcidev(vm->vmid, virt_to_phys(pcidev));
+		if (ret < 0)
+			dev_err(acrn_dev.this_device,
+				"Failed to assign pci device!\n");
+		kfree(pcidev);
+		break;
+	case ACRN_IOCTL_DEASSIGN_PCIDEV:
+		pcidev = memdup_user((void __user *)ioctl_param,
+				     sizeof(struct acrn_pcidev));
+		if (IS_ERR(pcidev))
+			return PTR_ERR(pcidev);
+
+		ret = hcall_deassign_pcidev(vm->vmid, virt_to_phys(pcidev));
+		if (ret < 0)
+			dev_err(acrn_dev.this_device,
+				"Failed to deassign pci device!\n");
+		kfree(pcidev);
+		break;
+	case ACRN_IOCTL_SET_PTDEV_INTR:
+		irq_info = memdup_user((void __user *)ioctl_param,
+				       sizeof(struct acrn_ptdev_irq));
+		if (IS_ERR(irq_info))
+			return PTR_ERR(irq_info);
+
+		ret = hcall_set_ptdev_intr(vm->vmid, virt_to_phys(irq_info));
+		if (ret < 0)
+			dev_err(acrn_dev.this_device,
+				"Failed to configure intr for ptdev!\n");
+		kfree(irq_info);
+		break;
+	case ACRN_IOCTL_RESET_PTDEV_INTR:
+		irq_info = memdup_user((void __user *)ioctl_param,
+				       sizeof(struct acrn_ptdev_irq));
+		if (IS_ERR(irq_info))
+			return PTR_ERR(irq_info);
+
+		ret = hcall_reset_ptdev_intr(vm->vmid, virt_to_phys(irq_info));
+		if (ret < 0)
+			dev_err(acrn_dev.this_device,
+				"Failed to reset intr for ptdev!\n");
+		kfree(irq_info);
+		break;
 	case ACRN_IOCTL_CREATE_IOREQ_CLIENT:
 		if (vm->default_client)
 			return -EEXIST;
drivers/virt/acrn/hypercall.h +54 -0

64 lines not on this page · raw message

include/uapi/linux/acrn.h +63 -0

73 lines not on this page · raw message

Signature · 1 line
2.28.0
#40 lore raw

PATCH v4 12/17 virt: acrn: Introduce interrupt injection interfaces

In reply to [email protected]

From: Shuo Liu <[email protected]>

ACRN userspace need to inject virtual interrupts into a User VM in devices emulation.

HSM needs provide interfaces to do so.

Introduce following interrupt injection interfaces:

ioctl ACRN_IOCTL_SET_IRQLINE:
Pass data from userspace to the hypervisor, and inform the hypervisor to inject a virtual IOAPIC GSI interrupt to a User VM.

ioctl ACRN_IOCTL_INJECT_MSI:
Pass data struct acrn_msi_entry from userspace to the hypervisor, and inform the hypervisor to inject a virtual MSI to a User VM.

ioctl ACRN_IOCTL_VM_INTR_MONITOR:
Set a 4-Kbyte aligned shared page for statistics information of interrupts of a User VM.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 drivers/virt/acrn/acrn_drv.h  |  4 ++++
 drivers/virt/acrn/hsm.c       | 39 +++++++++++++++++++++++++++++++++
 drivers/virt/acrn/hypercall.h | 41 +++++++++++++++++++++++++++++++++++
 drivers/virt/acrn/vm.c        | 36 ++++++++++++++++++++++++++++++
 include/uapi/linux/acrn.h     | 17 +++++++++++++++
 5 files changed, 137 insertions(+)
drivers/virt/acrn/acrn_drv.h +4 -0
index 97d2aab8b70a..701c83319115 100644
@@ -157,6 +157,7 @@ extern rwlock_t acrn_vm_list_lock;
  * @ioreq_buf:			I/O request shared buffer
  * @ioreq_page:			The page of the I/O request shared buffer
  * @pci_conf_addr:		Address of a PCI configuration access emulation
+ * @monitor_page:		Page of interrupt statistics of User VM
  */
 struct acrn_vm {
 	struct list_head		list;
@@ -172,6 +173,7 @@ struct acrn_vm {
 	struct acrn_io_request_buffer	*ioreq_buf;
 	struct page			*ioreq_page;
 	u32				pci_conf_addr;
+	struct page			*monitor_page;
 };
 
 struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
@@ -198,4 +200,6 @@ struct acrn_ioreq_client *acrn_ioreq_client_create(struct acrn_vm *vm,
 						   const char *name);
 void acrn_ioreq_client_destroy(struct acrn_ioreq_client *client);
 
+int acrn_msi_inject(struct acrn_vm *vm, u64 msi_addr, u64 msi_data);
+
 #endif /* __ACRN_HSM_DRV_H */
drivers/virt/acrn/hsm.c +39 -0
index bcb5e273dcc3..5c82e0491210 100644
@@ -51,7 +51,9 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 	struct acrn_ioreq_notify notify;
 	struct acrn_ptdev_irq *irq_info;
 	struct acrn_vm_memmap memmap;
+	struct acrn_msi_entry *msi;
 	struct acrn_pcidev *pcidev;
+	struct page *page;
 	int ret = 0;
 
 	if (vm->vmid == ACRN_INVALID_VMID && cmd != ACRN_IOCTL_CREATE_VM) {
@@ -178,6 +180,43 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 				"Failed to reset intr for ptdev!\n");
 		kfree(irq_info);
 		break;
+	case ACRN_IOCTL_SET_IRQLINE:
+		ret = hcall_set_irqline(vm->vmid, ioctl_param);
+		if (ret < 0)
+			dev_err(acrn_dev.this_device,
+				"Failed to set interrupt line!\n");
+		break;
+	case ACRN_IOCTL_INJECT_MSI:
+		msi = memdup_user((void __user *)ioctl_param,
+				  sizeof(struct acrn_msi_entry));
+		if (IS_ERR(msi))
+			return PTR_ERR(msi);
+
+		ret = hcall_inject_msi(vm->vmid, virt_to_phys(msi));
+		if (ret < 0)
+			dev_err(acrn_dev.this_device,
+				"Failed to inject MSI!\n");
+		kfree(msi);
+		break;
+	case ACRN_IOCTL_VM_INTR_MONITOR:
+		ret = get_user_pages_fast(ioctl_param, 1, FOLL_WRITE, &page);
+		if (unlikely(ret != 1)) {
+			dev_err(acrn_dev.this_device,
+				"Failed to pin intr hdr buffer!\n");
+			return -EFAULT;
+		}
+
+		ret = hcall_vm_intr_monitor(vm->vmid, page_to_phys(page));
+		if (ret < 0) {
+			put_page(page);
+			dev_err(acrn_dev.this_device,
+				"Failed to monitor intr data!\n");
+			return ret;
+		}
+		if (vm->monitor_page)
+			put_page(vm->monitor_page);
+		vm->monitor_page = page;
+		break;
 	case ACRN_IOCTL_CREATE_IOREQ_CLIENT:
 		if (vm->default_client)
 			return -EEXIST;
drivers/virt/acrn/hypercall.h +41 -0

53 lines not on this page · raw message

drivers/virt/acrn/vm.c +36 -0
index 4ee1a99df4b7..38304aeef181 100644
@@ -68,6 +68,10 @@ int acrn_vm_destroy(struct acrn_vm *vm)
 	write_unlock_bh(&acrn_vm_list_lock);
 
 	acrn_ioreq_deinit(vm);
+	if (vm->monitor_page) {
+		put_page(vm->monitor_page);
+		vm->monitor_page = NULL;
+	}
 
 	ret = hcall_destroy_vm(vm->vmid);
 	if (ret < 0) {
@@ -83,3 +87,35 @@ int acrn_vm_destroy(struct acrn_vm *vm)
 	vm->vmid = ACRN_INVALID_VMID;
 	return 0;
 }
+
+/**
+ * acrn_inject_msi() - Inject a MSI interrupt into a User VM
+ * @vm:		User VM
+ * @msi_addr:	The MSI address
+ * @msi_data:	The MSI data
+ *
+ * Return: 0 on success, <0 on error
+ */
+int acrn_msi_inject(struct acrn_vm *vm, u64 msi_addr, u64 msi_data)
+{
+	struct acrn_msi_entry *msi;
+	int ret;
+
+	/* might be used in interrupt context, so use GFP_ATOMIC */
+	msi = kzalloc(sizeof(*msi), GFP_ATOMIC);
+	if (!msi)
+		return -ENOMEM;
+
+	/*
+	 * msi_addr: addr[19:12] with dest vcpu id
+	 * msi_data: data[7:0] with vector
+	 */
+	msi->msi_addr = msi_addr;
+	msi->msi_data = msi_data;
+	ret = hcall_inject_msi(vm->vmid, virt_to_phys(msi));
+	if (ret < 0)
+		dev_err(acrn_dev.this_device,
+			"Failed to inject MSI to VM %u!\n", vm->vmid);
+	kfree(msi);
+	return ret;
+}
include/uapi/linux/acrn.h +17 -0
index 893389babbcb..7764459e130c 100644
@@ -343,6 +343,16 @@ struct acrn_pcidev {
 	__u32	reserved[6];
 } __attribute__((aligned(8)));
 
+/**
+ * struct acrn_msi_entry - Info for injecting a MSI interrupt to a VM
+ * @msi_addr:	MSI addr[19:12] with dest vCPU ID
+ * @msi_data:	MSI data[7:0] with vector
+ */
+struct acrn_msi_entry {
+	__u64	msi_addr;
+	__u64	msi_data;
+};
+
 /* The ioctl type, documented in ioctl-number.rst */
 #define ACRN_IOCTL_TYPE			0xA2
 
@@ -362,6 +372,13 @@ struct acrn_pcidev {
 #define ACRN_IOCTL_SET_VCPU_REGS	\
 	_IOW(ACRN_IOCTL_TYPE, 0x16, struct acrn_vcpu_regs)
 
+#define ACRN_IOCTL_INJECT_MSI		\
+	_IOW(ACRN_IOCTL_TYPE, 0x23, struct acrn_msi_entry)
+#define ACRN_IOCTL_VM_INTR_MONITOR	\
+	_IOW(ACRN_IOCTL_TYPE, 0x24, unsigned long)
+#define ACRN_IOCTL_SET_IRQLINE		\
+	_IOW(ACRN_IOCTL_TYPE, 0x25, __u64)
+
 #define ACRN_IOCTL_NOTIFY_REQUEST_FINISH \
 	_IOW(ACRN_IOCTL_TYPE, 0x31, struct acrn_ioreq_notify)
 #define ACRN_IOCTL_CREATE_IOREQ_CLIENT	\
Signature · 1 line
2.28.0
#41 lore raw

PATCH v4 14/17 virt: acrn: Introduce I/O ranges operation interfaces

In reply to [email protected]

From: Shuo Liu <[email protected]>

An I/O request of a User VM, which is constructed by hypervisor, is distributed by the ACRN Hypervisor Service Module to an I/O client corresponding to the address range of the I/O request.

I/O client maintains a list of address ranges. Introduce acrn_ioreq_range_{add,del}() to manage these address ranges.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 drivers/virt/acrn/acrn_drv.h |  4 +++
 drivers/virt/acrn/ioreq.c    | 60 ++++++++++++++++++++++++++++++++++++
 2 files changed, 64 insertions(+)
drivers/virt/acrn/acrn_drv.h +4 -0
index 701c83319115..5b824fa1ee57 100644
@@ -199,6 +199,10 @@ struct acrn_ioreq_client *acrn_ioreq_client_create(struct acrn_vm *vm,
 						   void *data, bool is_default,
 						   const char *name);
 void acrn_ioreq_client_destroy(struct acrn_ioreq_client *client);
+int acrn_ioreq_range_add(struct acrn_ioreq_client *client,
+			 u32 type, u64 start, u64 end);
+void acrn_ioreq_range_del(struct acrn_ioreq_client *client,
+			  u32 type, u64 start, u64 end);
 
 int acrn_msi_inject(struct acrn_vm *vm, u64 msi_addr, u64 msi_data);
 
drivers/virt/acrn/ioreq.c +60 -0

66 lines not on this page · raw message

Signature · 1 line
2.28.0
#42 lore raw

PATCH v4 16/17 virt: acrn: Introduce irqfd

In reply to [email protected]

From: Shuo Liu <[email protected]>

irqfd is a mechanism to inject a specific interrupt to a User VM using a decoupled eventfd mechanism.

Vhost is a kernel-level virtio server which uses eventfd for interrupt injection. To support vhost on ACRN, irqfd is introduced in HSM.

HSM provides ioctls to associate a virtual Message Signaled Interrupt (MSI) with an eventfd. The corresponding virtual MSI will be injected into a User VM once the eventfd got signal.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 drivers/virt/acrn/Makefile   |   2 +-
 drivers/virt/acrn/acrn_drv.h |  10 ++
 drivers/virt/acrn/hsm.c      |   7 ++
 drivers/virt/acrn/irqfd.c    | 235 +++++++++++++++++++++++++++++++++++
 drivers/virt/acrn/vm.c       |   3 +
 include/uapi/linux/acrn.h    |  15 +++
 6 files changed, 271 insertions(+), 1 deletion(-)
 create mode 100644 drivers/virt/acrn/irqfd.c
drivers/virt/acrn/Makefile +1 -1
index 755b583b32ca..08ce641dcfa1 100644
@@ -1,3 +1,3 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_ACRN_HSM)	:= acrn.o
-acrn-y := hsm.o vm.o mm.o ioreq.o ioeventfd.o
+acrn-y := hsm.o vm.o mm.o ioreq.o ioeventfd.o irqfd.o
drivers/virt/acrn/acrn_drv.h +10 -0
index c66c620b9f10..8354d0d5881c 100644
@@ -161,6 +161,9 @@ extern rwlock_t acrn_vm_list_lock;
  * @ioeventfds_lock:		Lock to protect ioeventfds list
  * @ioeventfds:			List to link all hsm_ioeventfd
  * @ioeventfd_client:		I/O client for ioeventfds of the VM
+ * @irqfds_lock:		Lock to protect irqfds list
+ * @irqfds:			List to link all hsm_irqfd
+ * @irqfd_wq:			Workqueue for irqfd async shutdown
  */
 struct acrn_vm {
 	struct list_head		list;
@@ -180,6 +183,9 @@ struct acrn_vm {
 	struct mutex			ioeventfds_lock;
 	struct list_head		ioeventfds;
 	struct acrn_ioreq_client	*ioeventfd_client;
+	struct mutex			irqfds_lock;
+	struct list_head		irqfds;
+	struct workqueue_struct		*irqfd_wq;
 };
 
 struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
@@ -216,4 +222,8 @@ int acrn_ioeventfd_init(struct acrn_vm *vm);
 int acrn_ioeventfd_config(struct acrn_vm *vm, struct acrn_ioeventfd *args);
 void acrn_ioeventfd_deinit(struct acrn_vm *vm);
 
+int acrn_irqfd_init(struct acrn_vm *vm);
+int acrn_irqfd_config(struct acrn_vm *vm, struct acrn_irqfd *args);
+void acrn_irqfd_deinit(struct acrn_vm *vm);
+
 #endif /* __ACRN_HSM_DRV_H */
drivers/virt/acrn/hsm.c +7 -0
index 40ec16a21308..aaf4e76d27b4 100644
@@ -115,6 +115,7 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 	struct acrn_vm_memmap memmap;
 	struct acrn_msi_entry *msi;
 	struct acrn_pcidev *pcidev;
+	struct acrn_irqfd irqfd;
 	struct page *page;
 	u64 cstate_cmd;
 	int ret = 0;
@@ -317,6 +318,12 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 
 		ret = acrn_ioeventfd_config(vm, &ioeventfd);
 		break;
+	case ACRN_IOCTL_IRQFD:
+		if (copy_from_user(&irqfd, (void __user *)ioctl_param,
+				   sizeof(irqfd)))
+			return -EFAULT;
+		ret = acrn_irqfd_config(vm, &irqfd);
+		break;
 	default:
 		dev_warn(acrn_dev.this_device, "Unknown IOCTL 0x%x!\n", cmd);
 		ret = -ENOTTY;
drivers/virt/acrn/irqfd.c new file +235 -0

235 lines not on this page · raw message

drivers/virt/acrn/vm.c +3 -0
index 3c671b03b273..7f152a74b591 100644
@@ -51,6 +51,7 @@ struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
 	write_unlock_bh(&acrn_vm_list_lock);
 
 	acrn_ioeventfd_init(vm);
+	acrn_irqfd_init(vm);
 	dev_dbg(acrn_dev.this_device, "VM %u created.\n", vm->vmid);
 	return vm;
 }
@@ -69,7 +70,9 @@ int acrn_vm_destroy(struct acrn_vm *vm)
 	write_unlock_bh(&acrn_vm_list_lock);
 
 	acrn_ioeventfd_deinit(vm);
+	acrn_irqfd_deinit(vm);
 	acrn_ioreq_deinit(vm);
+
 	if (vm->monitor_page) {
 		put_page(vm->monitor_page);
 		vm->monitor_page = NULL;
include/uapi/linux/acrn.h +15 -0
index 7a99124c7d4d..75a687838a43 100644
@@ -411,6 +411,19 @@ struct acrn_ioeventfd {
 	__u64	data;
 };
 
+#define ACRN_IRQFD_FLAG_DEASSIGN	0x01
+/**
+ * struct acrn_irqfd - Data to operate a &struct hsm_irqfd
+ * @fd:		The fd of eventfd associated with a hsm_irqfd
+ * @flags:	Logical-OR of ACRN_IRQFD_FLAG_*
+ * @msi:	Info of MSI associated with the irqfd
+ */
+struct acrn_irqfd {
+	__s32			fd;
+	__u32			flags;
+	struct acrn_msi_entry	msi;
+};
+
 /* The ioctl type, documented in ioctl-number.rst */
 #define ACRN_IOCTL_TYPE			0xA2
 
@@ -467,5 +480,7 @@ struct acrn_ioeventfd {
 
 #define ACRN_IOCTL_IOEVENTFD		\
 	_IOW(ACRN_IOCTL_TYPE, 0x70, struct acrn_ioeventfd)
+#define ACRN_IOCTL_IRQFD		\
+	_IOW(ACRN_IOCTL_TYPE, 0x71, struct acrn_irqfd)
 
 #endif /* _UAPI_ACRN_H */
Signature · 1 line
2.28.0
#43 lore raw

PATCH v4 17/17 virt: acrn: Introduce an interface for Service VM to control vCPU

In reply to [email protected]

From: Shuo Liu <[email protected]>

ACRN supports partition mode to achieve real-time requirements. In partition mode, a CPU core can be dedicated to a vCPU of User VM. The local APIC of the dedicated CPU core can be passthrough to the User VM. The Service VM controls the assignment of the CPU cores.

Introduce an interface for the Service VM to remove the control of CPU core from hypervisor perspective so that the CPU core can be a dedicated CPU core of User VM.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 drivers/virt/acrn/hsm.c       | 50 +++++++++++++++++++++++++++++++++++
 drivers/virt/acrn/hypercall.h | 14 ++++++++++
 2 files changed, 64 insertions(+)
drivers/virt/acrn/hsm.c +50 -0
index aaf4e76d27b4..ef5f77a38d1f 100644
@@ -9,6 +9,7 @@
  *	Yakui Zhao <[email protected]>
  */
 
+#include <linux/cpu.h>
 #include <linux/io.h>
 #include <linux/mm.h>
 #include <linux/module.h>
@@ -354,6 +355,47 @@ struct miscdevice acrn_dev = {
 	.fops	= &acrn_fops,
 };
 
+static ssize_t remove_cpu_store(struct device *dev,
+				struct device_attribute *attr,
+				const char *buf, size_t count)
+{
+	u64 cpu, lapicid;
+	int ret;
+
+	if (kstrtoull(buf, 0, &cpu) < 0)
+		return -EINVAL;
+
+	if (cpu >= num_possible_cpus() || cpu == 0 || !cpu_is_hotpluggable(cpu))
+		return -EINVAL;
+
+	if (cpu_online(cpu))
+		remove_cpu(cpu);
+
+	lapicid = cpu_data(cpu).apicid;
+	dev_dbg(dev, "Try to remove cpu %lld with lapicid %lld\n", cpu, lapicid);
+	ret = hcall_sos_remove_cpu(lapicid);
+	if (ret < 0) {
+		dev_err(dev, "Failed to remove cpu %lld!\n", cpu);
+		goto fail_remove;
+	}
+
+	return count;
+
+fail_remove:
+	add_cpu(cpu);
+	return ret;
+}
+static DEVICE_ATTR_WO(remove_cpu);
+
+static struct attribute *acrn_attrs[] = {
+	&dev_attr_remove_cpu.attr,
+	NULL
+};
+
+static struct attribute_group acrn_attr_group = {
+	.attrs = acrn_attrs,
+};
+
 static int __init hsm_init(void)
 {
 	int ret;
@@ -370,13 +412,21 @@ static int __init hsm_init(void)
 		return ret;
 	}
 
+	ret = sysfs_create_group(&acrn_dev.this_device->kobj, &acrn_attr_group);
+	if (ret) {
+		dev_warn(acrn_dev.this_device, "sysfs create failed\n");
+		misc_deregister(&acrn_dev);
+		return ret;
+	}
 	acrn_ioreq_intr_setup();
+
 	return 0;
 }
 
 static void __exit hsm_exit(void)
 {
 	acrn_ioreq_intr_remove();
+	sysfs_remove_group(&acrn_dev.this_device->kobj, &acrn_attr_group);
 	misc_deregister(&acrn_dev);
 }
 module_init(hsm_init);
drivers/virt/acrn/hypercall.h +14 -0
index e640632366f0..0cfad05bd1a9 100644
@@ -13,6 +13,9 @@
 
 #define HC_ID 0x80UL
 
+#define HC_ID_GEN_BASE			0x0UL
+#define HC_SOS_REMOVE_CPU		_HC_ID(HC_ID, HC_ID_GEN_BASE + 0x01)
+
 #define HC_ID_VM_BASE			0x10UL
 #define HC_CREATE_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x00)
 #define HC_DESTROY_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x01)
@@ -42,6 +45,17 @@
 #define HC_ID_PM_BASE			0x80UL
 #define HC_PM_GET_CPU_STATE		_HC_ID(HC_ID, HC_ID_PM_BASE + 0x00)
 
+/**
+ * hcall_sos_remove_cpu() - Remove a vCPU of Service VM
+ * @cpu: The vCPU to be removed
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_sos_remove_cpu(u64 cpu)
+{
+	return acrn_hypercall1(HC_SOS_REMOVE_CPU, cpu);
+}
+
 /**
  * hcall_create_vm() - Create a User VM
  * @vminfo:	Service VM GPA of info of User VM creation
Signature · 1 line
2.28.0
#44 lore raw

Re: [PATCH v4 17/17] virt: acrn: Introduce an interface for Service VM to control vCPU

Quoted 70 lines · On Tue, Sep 22, 2020 at 07:43:11PM +0800, [email protected] wrote:

From: Shuo Liu <[email protected]>

ACRN supports partition mode to achieve real-time requirements. In partition mode, a CPU core can be dedicated to a vCPU of User VM. The local APIC of the dedicated CPU core can be passthrough to the User VM. The Service VM controls the assignment of the CPU cores.

Introduce an interface for the Service VM to remove the control of CPU core from hypervisor perspective so that the CPU core can be a dedicated CPU core of User VM.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 drivers/virt/acrn/hsm.c       | 50 +++++++++++++++++++++++++++++++++++
 drivers/virt/acrn/hypercall.h | 14 ++++++++++
 2 files changed, 64 insertions(+)
drivers/virt/acrn/hsm.c +31 -0
index aaf4e76d27b4..ef5f77a38d1f 100644
@@ -9,6 +9,7 @@
  *	Yakui Zhao <[email protected]>
  */
 
+#include <linux/cpu.h>
 #include <linux/io.h>
 #include <linux/mm.h>
 #include <linux/module.h>
@@ -354,6 +355,47 @@ struct miscdevice acrn_dev = {
 	.fops	= &acrn_fops,
 };
 
+static ssize_t remove_cpu_store(struct device *dev,
+				struct device_attribute *attr,
+				const char *buf, size_t count)
+{
+	u64 cpu, lapicid;
+	int ret;
+
+	if (kstrtoull(buf, 0, &cpu) < 0)
+		return -EINVAL;
+
+	if (cpu >= num_possible_cpus() || cpu == 0 || !cpu_is_hotpluggable(cpu))
+		return -EINVAL;
+
+	if (cpu_online(cpu))
+		remove_cpu(cpu);
+
+	lapicid = cpu_data(cpu).apicid;
+	dev_dbg(dev, "Try to remove cpu %lld with lapicid %lld\n", cpu, lapicid);
+	ret = hcall_sos_remove_cpu(lapicid);
+	if (ret < 0) {
+		dev_err(dev, "Failed to remove cpu %lld!\n", cpu);
+		goto fail_remove;
+	}
+
+	return count;
+
+fail_remove:
+	add_cpu(cpu);
+	return ret;
+}

drivers/virt/acrn/hsm.c struct miscdevice acrn_dev = { · line 397

Quoted 10 lines
drivers/virt/acrn/hsm.c +10 -0
+static DEVICE_ATTR_WO(remove_cpu);
+
+static struct attribute *acrn_attrs[] = {
+	&dev_attr_remove_cpu.attr,
+	NULL
+};
+
+static struct attribute_group acrn_attr_group = {
+	.attrs = acrn_attrs,
+};

You create a sysfs attribute without any Documentation/ABI/ update as well? That's not good.

And why are you trying to emulate CPU hotplug here and not using the existing CPU hotplug mechanism?

Quoted 5 lines
drivers/virt/acrn/hsm.c +1 -0
+
 static int __init hsm_init(void)
 {
 	int ret;

drivers/virt/acrn/hsm.c static int __init hsm_init(void) · line 420

Quoted 9 lines
drivers/virt/acrn/hsm.c +6 -0
@@ -370,13 +412,21 @@ static int __init hsm_init(void)
 		return ret;
 	}
 
+	ret = sysfs_create_group(&acrn_dev.this_device->kobj, &acrn_attr_group);
+	if (ret) {
+		dev_warn(acrn_dev.this_device, "sysfs create failed\n");
+		misc_deregister(&acrn_dev);
+		return ret;
+	}

You just raced with userspace and lost. If you want to add attribute files to a device, use the default attribute group list, and it will be managed properly for you by the driver core.

Huge hint, if a driver every has to touch a kobject, or call sysfs_*, then it is probably doing something wrong.

greg k-h

author
#45 lore raw

Re: [PATCH v4 17/17] virt: acrn: Introduce an interface for Service VM to control vCPU

Hi Greg,

Quoted 76 lines · Greg Kroah-Hartman
Quoted 75 lines · On Tue, Sep 22, 2020 at 07:43:11PM +0800, [email protected] wrote:

From: Shuo Liu <[email protected]>

ACRN supports partition mode to achieve real-time requirements. In partition mode, a CPU core can be dedicated to a vCPU of User VM. The local APIC of the dedicated CPU core can be passthrough to the User VM. The Service VM controls the assignment of the CPU cores.

Introduce an interface for the Service VM to remove the control of CPU core from hypervisor perspective so that the CPU core can be a dedicated CPU core of User VM.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 drivers/virt/acrn/hsm.c       | 50 +++++++++++++++++++++++++++++++++++
 drivers/virt/acrn/hypercall.h | 14 ++++++++++
 2 files changed, 64 insertions(+)
drivers/virt/acrn/hsm.c +36 -0
index aaf4e76d27b4..ef5f77a38d1f 100644
@@ -9,6 +9,7 @@
  *	Yakui Zhao <[email protected]>
  */
 
+#include <linux/cpu.h>
 #include <linux/io.h>
 #include <linux/mm.h>
 #include <linux/module.h>
@@ -354,6 +355,47 @@ struct miscdevice acrn_dev = {
 	.fops	= &acrn_fops,
 };
 
+static ssize_t remove_cpu_store(struct device *dev,
+				struct device_attribute *attr,
+				const char *buf, size_t count)
+{
+	u64 cpu, lapicid;
+	int ret;
+
+	if (kstrtoull(buf, 0, &cpu) < 0)
+		return -EINVAL;
+
+	if (cpu >= num_possible_cpus() || cpu == 0 || !cpu_is_hotpluggable(cpu))
+		return -EINVAL;
+
+	if (cpu_online(cpu))
+		remove_cpu(cpu);
+
+	lapicid = cpu_data(cpu).apicid;
+	dev_dbg(dev, "Try to remove cpu %lld with lapicid %lld\n", cpu, lapicid);
+	ret = hcall_sos_remove_cpu(lapicid);
+	if (ret < 0) {
+		dev_err(dev, "Failed to remove cpu %lld!\n", cpu);
+		goto fail_remove;
+	}
+
+	return count;
+
+fail_remove:
+	add_cpu(cpu);
+	return ret;
+}
+static DEVICE_ATTR_WO(remove_cpu);
+
+static struct attribute *acrn_attrs[] = {
+	&dev_attr_remove_cpu.attr,
+	NULL
Quoted 8 lines

drivers/virt/acrn/hsm.c struct miscdevice acrn_dev = { · line 397

Quoted 5 lines
drivers/virt/acrn/hsm.c +5 -0
+};
+
+static struct attribute_group acrn_attr_group = {
+	.attrs = acrn_attrs,
+};

You create a sysfs attribute without any Documentation/ABI/ update as well? That's not good.

Sorry, i will add it in the ABI/testing.

Quoted 3 lines

And why are you trying to emulate CPU hotplug here and not using the existing CPU hotplug mechanism?

The interface introduced here includes:
1) The Service VM virtual CPU hotplug
2) hypercall to the hypervisor to remove one virtual CPU from the Service VM
The 1) just do the CPU hotplug with kernel API remove_cpu(), and can be resume back (by CPU online interface) if only 1) is done.
If 2) is done, then the physical CPU will be removed from the Service VM's CPU pool. The ACRN hypervisor supports passthrough a physical CPU to a VM. The precondition is that the physical CPU cannot be occupied by any other VM. This interface intends to do that.

Quoted 20 lines
Quoted 6 lines
drivers/virt/acrn/hsm.c +1 -0
+
 static int __init hsm_init(void)
 {
 	int ret;

drivers/virt/acrn/hsm.c static int __init hsm_init(void) · line 420

Quoted 9 lines
drivers/virt/acrn/hsm.c +6 -0
@@ -370,13 +412,21 @@ static int __init hsm_init(void)
 		return ret;
 	}
 
+	ret = sysfs_create_group(&acrn_dev.this_device->kobj, &acrn_attr_group);
+	if (ret) {
+		dev_warn(acrn_dev.this_device, "sysfs create failed\n");
+		misc_deregister(&acrn_dev);
+		return ret;
+	}

You just raced with userspace and lost. If you want to add attribute files to a device, use the default attribute group list, and it will be managed properly for you by the driver core.

Quoted 2 lines

Huge hint, if a driver every has to touch a kobject, or call sysfs_*, then it is probably doing something wrong.

Do you mean use .groups of struct miscdevice directly ?

If yes, let me follow drivers/char/hw_random/s390-trng.c to do this. BTW, few driver use the .groups directly. :)

Thanks
shuo

#46 lore raw

Re: [PATCH v4 17/17] virt: acrn: Introduce an interface for Service VM to control vCPU

Quoted 11 lines · Shuo A Liu
Quoted 6 lines

You just raced with userspace and lost. If you want to add attribute files to a device, use the default attribute group list, and it will be managed properly for you by the driver core.

Huge hint, if a driver every has to touch a kobject, or call sysfs_*, then it is probably doing something wrong.

Do you mean use .groups of struct miscdevice directly ?

If yes, let me follow drivers/char/hw_random/s390-trng.c to do this. BTW, few driver use the .groups directly. :)

Drivers should almost never be messing with individual sysfs files. And this ability to use .groups is a "new" one, conversions of existing code that do not use them is always welcome.

thanks,

greg k-h

author
#47 lore raw

Re: [PATCH v4 17/17] virt: acrn: Introduce an interface for Service VM to control vCPU

Quoted 8 lines · Greg Kroah-Hartman
Quoted 7 lines · Shuo A Liu
Quoted 6 lines

You just raced with userspace and lost. If you want to add attribute files to a device, use the default attribute group list, and it will be managed properly for you by the driver core.

Huge hint, if a driver every has to touch a kobject, or call sysfs_*, then it is probably doing something wrong.

Quoted 8 lines
Quoted 4 lines

Do you mean use .groups of struct miscdevice directly ?

If yes, let me follow drivers/char/hw_random/s390-trng.c to do this. BTW, few driver use the .groups directly. :)

Drivers should almost never be messing with individual sysfs files. And this ability to use .groups is a "new" one, conversions of existing code that do not use them is always welcome.

OK. Thanks for the explanation. I will follow the 'new' way :)

Thanks
shuo

#48 lore raw
In reply to [email protected]

Ping...

Quoted 140 lines · On 9/22/2020 19:42, [email protected] wrote:

From: Shuo Liu <[email protected]>

ACRN is a Type 1 reference hypervisor stack, running directly on the bare-metal hardware, and is suitable for a variety of IoT and embedded device solutions.

ACRN implements a hybrid VMM architecture, using a privileged Service VM. The Service VM manages the system resources (CPU, memory, etc.) and I/O devices of User VMs. Multiple User VMs are supported, with each of them running Linux, Android OS or Windows. Both Service VM and User VMs are guest VM.

Below figure shows the architecture.

                Service VM                    User VM
      +----------------------------+  |  +------------------+
      |        +--------------+    |  |  |                  |
      |        |ACRN userspace|    |  |  |                  |
      |        +--------------+    |  |  |                  |
      |-----------------ioctl------|  |  |                  |   ...
      |kernel space   +----------+ |  |  |                  |
      |               |   HSM    | |  |  | Drivers          |
      |               +----------+ |  |  |                  |
      +--------------------|-------+  |  +------------------+
  +---------------------hypercall----------------------------------------+
  |                       ACRN Hypervisor                                |
  +----------------------------------------------------------------------+
  |                          Hardware                                    |
  +----------------------------------------------------------------------+

There is only one Service VM which could run Linux as OS.

In a typical case, the Service VM will be auto started when ACRN Hypervisor is booted. Then the ACRN userspace (an application running in Service VM) could be used to start/stop User VMs by communicating with ACRN Hypervisor Service Module (HSM).

ACRN Hypervisor Service Module (HSM) is a middle layer that allows the ACRN userspace and Service VM OS kernel to communicate with ACRN Hypervisor and manage different User VMs. This middle layer provides the following functionalities,
- Issues hypercalls to the hypervisor to manage User VMs:
* VM/vCPU management
* Memory management
* Device passthrough
* Interrupts injection
- I/O requests handling from User VMs.
- Exports ioctl through HSM char device.
- Exports function calls for other kernel modules

ACRN is focused on embedded system. So it doesn't support some features. E.g.,
- ACRN doesn't support VM migration.
- ACRN doesn't support vCPU migration.

This patch set adds the HSM to the Linux kernel.

The basic ARCN support was merged to upstream already.
https://lore.kernel.org/lkml/[email protected]/

ChangeLog:
v4:
- Used acrn_dev.this_device directly for dev_*() (Reinette)
- Removed the odd usage of {get|put}_device() on &acrn_dev->this_device (Greg)
- Removed unused log code. (Greg)
- Corrected the return error values. (Greg)
- Mentioned that HSM relies hypervisor for sanity check in acrn_dev_ioctl() comments (Greg)

v3:
- Used {get|put}_device() helpers on &acrn_dev->this_device
- Moved unused code from front patches to later ones.
- Removed self-defined pr_fmt() and dev_fmt()
- Provided comments for acrn_vm_list_lock.

v2:
- Removed API version related code. (Dave)
- Replaced pr_*() by dev_*(). (Greg)
- Used -ENOTTY as the error code of unsupported ioctl. (Greg)

Shuo Liu (16):
docs: acrn: Introduce ACRN
x86/acrn: Introduce acrn_{setup, remove}_intr_handler() x86/acrn: Introduce hypercall interfaces
virt: acrn: Introduce ACRN HSM basic driver
virt: acrn: Introduce VM management interfaces
virt: acrn: Introduce an ioctl to set vCPU registers state
virt: acrn: Introduce EPT mapping management
virt: acrn: Introduce I/O request management
virt: acrn: Introduce PCI configuration space PIO accesses combiner
virt: acrn: Introduce interfaces for PCI device passthrough
virt: acrn: Introduce interrupt injection interfaces
virt: acrn: Introduce interfaces to query C-states and P-states
allowed by hypervisor
virt: acrn: Introduce I/O ranges operation interfaces
virt: acrn: Introduce ioeventfd
virt: acrn: Introduce irqfd
virt: acrn: Introduce an interface for Service VM to control vCPU

Yin Fengwei (1):
x86/acrn: Introduce an API to check if a VM is privileged

 .../userspace-api/ioctl/ioctl-number.rst      |   1 +
 Documentation/virt/acrn/index.rst             |  11 +
 Documentation/virt/acrn/introduction.rst      |  40 ++
 Documentation/virt/acrn/io-request.rst        |  97 +++
 Documentation/virt/index.rst                  |   1 +
 MAINTAINERS                                   |   9 +
 arch/x86/include/asm/acrn.h                   |  74 ++
 arch/x86/kernel/cpu/acrn.c                    |  35 +-
 drivers/virt/Kconfig                          |   2 +
 drivers/virt/Makefile                         |   1 +
 drivers/virt/acrn/Kconfig                     |  15 +
 drivers/virt/acrn/Makefile                    |   3 +
 drivers/virt/acrn/acrn_drv.h                  | 229 +++++++
 drivers/virt/acrn/hsm.c                       | 437 ++++++++++++
 drivers/virt/acrn/hypercall.h                 | 254 +++++++
 drivers/virt/acrn/ioeventfd.c                 | 273 ++++++++
 drivers/virt/acrn/ioreq.c                     | 645 ++++++++++++++++++
 drivers/virt/acrn/irqfd.c                     | 235 +++++++
 drivers/virt/acrn/mm.c                        | 305 +++++++++
 drivers/virt/acrn/vm.c                        | 126 ++++
 include/uapi/linux/acrn.h                     | 486 +++++++++++++
 21 files changed, 3278 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/virt/acrn/index.rst
 create mode 100644 Documentation/virt/acrn/introduction.rst
 create mode 100644 Documentation/virt/acrn/io-request.rst
 create mode 100644 arch/x86/include/asm/acrn.h
 create mode 100644 drivers/virt/acrn/Kconfig
 create mode 100644 drivers/virt/acrn/Makefile
 create mode 100644 drivers/virt/acrn/acrn_drv.h
 create mode 100644 drivers/virt/acrn/hsm.c
 create mode 100644 drivers/virt/acrn/hypercall.h
 create mode 100644 drivers/virt/acrn/ioeventfd.c
 create mode 100644 drivers/virt/acrn/ioreq.c
 create mode 100644 drivers/virt/acrn/irqfd.c
 create mode 100644 drivers/virt/acrn/mm.c
 create mode 100644 drivers/virt/acrn/vm.c
 create mode 100644 include/uapi/linux/acrn.h

base-commit: 18445bf405cb331117bc98427b1ba6f12418ad17

#49 lore raw
Quoted 1 line · Shuo A

Ping...

It's been less than a week since you sent this. Please relax and if you really need reviews, get them from within Intel, where you can impose a deadline on those developers. Otherwise, your patch is in good company:

	$ mdfrm -c ~/mail/todo/
	993 messages in /home/gregkh/mail/todo/

And will be handled when I get to it.

thanks,

greg "Intel still owes me lots of liquor" k-h

A message that is not in the archive (sent off-list or not yet received): [email protected]

#51 lore raw

Re: [PATCH v4 06/17] virt: acrn: Introduce VM management interfaces

Quoted 76 lines · On Tue, Sep 22, 2020 at 07:43:00PM +0800, [email protected] wrote:

From: Shuo Liu <[email protected]>

The VM management interfaces expose several VM operations to ACRN userspace via ioctls. For example, creating VM, starting VM, destroying VM and so on.

The ACRN Hypervisor needs to exchange data with the ACRN userspace during the VM operations. HSM provides VM operation ioctls to the ACRN userspace and communicates with the ACRN Hypervisor for VM operations via hypercalls.

HSM maintains a list of User VM. Each User VM will be bound to an existing file descriptor of /dev/acrn_hsm. The User VM will be destroyed when the file descriptor is closed.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 .../userspace-api/ioctl/ioctl-number.rst      |  1 +
 MAINTAINERS                                   |  1 +
 drivers/virt/acrn/Makefile                    |  2 +-
 drivers/virt/acrn/acrn_drv.h                  | 23 +++++-
 drivers/virt/acrn/hsm.c                       | 73 ++++++++++++++++-
 drivers/virt/acrn/hypercall.h                 | 78 +++++++++++++++++++
 drivers/virt/acrn/vm.c                        | 71 +++++++++++++++++
 include/uapi/linux/acrn.h                     | 56 +++++++++++++
 8 files changed, 301 insertions(+), 4 deletions(-)
 create mode 100644 drivers/virt/acrn/hypercall.h
 create mode 100644 drivers/virt/acrn/vm.c
 create mode 100644 include/uapi/linux/acrn.h
Documentation/userspace-api/ioctl/ioctl-number.rst +1 -0
index 2a198838fca9..ac60efedb104 100644
@@ -319,6 +319,7 @@ Code  Seq#    Include File                                           Comments
 0xA0  all    linux/sdp/sdp.h                                         Industrial Device Project
                                                                      <mailto:[email protected]>
 0xA1  0      linux/vtpm_proxy.h                                      TPM Emulator Proxy Driver
+0xA2  all    uapi/linux/acrn.h                                       ACRN hypervisor
 0xA3  80-8F                                                          Port ACL  in development:
                                                                      <mailto:[email protected]>
 0xA3  90-9F  linux/dtlk.h
MAINTAINERS +1 -0
index 3030d0e93d02..d4c1ef303c2d 100644
@@ -443,6 +443,7 @@ S:	Supported
 W:	https://projectacrn.org
 F:	Documentation/virt/acrn/
 F:	drivers/virt/acrn/
+F:	include/uapi/linux/acrn.h
 
 AD1889 ALSA SOUND DRIVER
 L:	[email protected]
drivers/virt/acrn/Makefile +1 -1
index 6920ed798aaf..cf8b4ed5e74e 100644
@@ -1,3 +1,3 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_ACRN_HSM)	:= acrn.o
-acrn-y := hsm.o
+acrn-y := hsm.o vm.o
drivers/virt/acrn/acrn_drv.h +0 -0
index 29eedd696327..72d92b60d944 100644
@@ -3,16 +3,37 @@
 #ifndef __ACRN_HSM_DRV_H

drivers/virt/acrn/acrn_drv.h line 13

Quoted 10 lines
drivers/virt/acrn/acrn_drv.h +6 -0
 #define __ACRN_HSM_DRV_H
 
+#include <linux/acrn.h>
+#include <linux/dev_printk.h>
+#include <linux/miscdevice.h>
 #include <linux/types.h>
 
+#include "hypercall.h"
+
+extern struct miscdevice acrn_dev;

Who else needs to get to this structure in your driver?

Quoted 95 lines
drivers/virt/acrn/acrn_drv.h +16 -1
+
 #define ACRN_INVALID_VMID (0xffffU)
 
+#define ACRN_VM_FLAG_DESTROYED		0U
+extern struct list_head acrn_vm_list;
+extern rwlock_t acrn_vm_list_lock;
 /**
  * struct acrn_vm - Properties of ACRN User VM.
+ * @list:	Entry within global list of all VMs
  * @vmid:	User VM ID
+ * @vcpu_num:	Number of virtual CPUs in the VM
+ * @flags:	Flags (ACRN_VM_FLAG_*) of the VM. This is VM flag management
+ *		in HSM which is different from the &acrn_vm_creation.vm_flag.
  */
 struct acrn_vm {
-	u16	vmid;
+	struct list_head	list;
+	u16			vmid;
+	int			vcpu_num;
+	unsigned long		flags;
 };
 
+struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
+			       struct acrn_vm_creation *vm_param);
+int acrn_vm_destroy(struct acrn_vm *vm);
+
 #endif /* __ACRN_HSM_DRV_H */
drivers/virt/acrn/hsm.c +52 -1
index 28a3052ffa55..f3e6467b8723 100644
@@ -9,7 +9,6 @@
  *	Yakui Zhao <[email protected]>
  */
 
-#include <linux/miscdevice.h>
 #include <linux/mm.h>
 #include <linux/module.h>
 #include <linux/slab.h>
@@ -38,10 +37,79 @@ static int acrn_dev_open(struct inode *inode, struct file *filp)
 	return 0;
 }
 
+/*
+ * HSM relies on hypercall layer of the ACRN hypervisor to do the
+ * sanity check against the input parameters.
+ */
+static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
+			   unsigned long ioctl_param)
+{
+	struct acrn_vm *vm = filp->private_data;
+	struct acrn_vm_creation *vm_param;
+	int ret = 0;
+
+	if (vm->vmid == ACRN_INVALID_VMID && cmd != ACRN_IOCTL_CREATE_VM) {
+		dev_dbg(acrn_dev.this_device,
+			"ioctl 0x%x: Invalid VM state!\n", cmd);
+		return -EINVAL;
+	}
+
+	switch (cmd) {
+	case ACRN_IOCTL_CREATE_VM:
+		vm_param = memdup_user((void __user *)ioctl_param,
+				       sizeof(struct acrn_vm_creation));
+		if (IS_ERR(vm_param))
+			return PTR_ERR(vm_param);
+
+		vm = acrn_vm_create(vm, vm_param);
+		if (!vm) {
+			ret = -EINVAL;
+			kfree(vm_param);
+			break;
+		}
+
+		if (copy_to_user((void __user *)ioctl_param, vm_param,
+				 sizeof(struct acrn_vm_creation))) {
+			acrn_vm_destroy(vm);
+			ret = -EFAULT;
+		}
+
+		kfree(vm_param);
+		break;
+	case ACRN_IOCTL_START_VM:
+		ret = hcall_start_vm(vm->vmid);
+		if (ret < 0)
+			dev_err(acrn_dev.this_device,
+				"Failed to start VM %u!\n", vm->vmid);
+		break;
+	case ACRN_IOCTL_PAUSE_VM:
+		ret = hcall_pause_vm(vm->vmid);
+		if (ret < 0)
+			dev_err(acrn_dev.this_device,
+				"Failed to pause VM %u!\n", vm->vmid);
+		break;
+	case ACRN_IOCTL_RESET_VM:

drivers/virt/acrn/hsm.c static int acrn_dev_open(struct inode *inode, struct file *filp) · line 101

Quoted 10 lines
drivers/virt/acrn/hsm.c +10 -0
+		ret = hcall_reset_vm(vm->vmid);
+		if (ret < 0)
+			dev_err(acrn_dev.this_device,
+				"Failed to restart VM %u!\n", vm->vmid);
+		break;
+	case ACRN_IOCTL_DESTROY_VM:
+		ret = acrn_vm_destroy(vm);
+		break;
+	default:
+		dev_warn(acrn_dev.this_device, "Unknown IOCTL 0x%x!\n", cmd);

Do not let userspace spam kernel logs with invalid stuff, that's a sure way to cause a DoS.

thanks,

greg k-h

author
#52 lore raw

Re: [PATCH v4 06/17] virt: acrn: Introduce VM management interfaces

Hi Greg,

Quoted 84 lines · Greg Kroah-Hartman
Quoted 83 lines · On Tue, Sep 22, 2020 at 07:43:00PM +0800, [email protected] wrote:

From: Shuo Liu <[email protected]>

The VM management interfaces expose several VM operations to ACRN userspace via ioctls. For example, creating VM, starting VM, destroying VM and so on.

The ACRN Hypervisor needs to exchange data with the ACRN userspace during the VM operations. HSM provides VM operation ioctls to the ACRN userspace and communicates with the ACRN Hypervisor for VM operations via hypercalls.

HSM maintains a list of User VM. Each User VM will be bound to an existing file descriptor of /dev/acrn_hsm. The User VM will be destroyed when the file descriptor is closed.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 .../userspace-api/ioctl/ioctl-number.rst      |  1 +
 MAINTAINERS                                   |  1 +
 drivers/virt/acrn/Makefile                    |  2 +-
 drivers/virt/acrn/acrn_drv.h                  | 23 +++++-
 drivers/virt/acrn/hsm.c                       | 73 ++++++++++++++++-
 drivers/virt/acrn/hypercall.h                 | 78 +++++++++++++++++++
 drivers/virt/acrn/vm.c                        | 71 +++++++++++++++++
 include/uapi/linux/acrn.h                     | 56 +++++++++++++
 8 files changed, 301 insertions(+), 4 deletions(-)
 create mode 100644 drivers/virt/acrn/hypercall.h
 create mode 100644 drivers/virt/acrn/vm.c
 create mode 100644 include/uapi/linux/acrn.h
Documentation/userspace-api/ioctl/ioctl-number.rst +1 -0
index 2a198838fca9..ac60efedb104 100644
@@ -319,6 +319,7 @@ Code  Seq#    Include File                                           Comments
 0xA0  all    linux/sdp/sdp.h                                         Industrial Device Project
                                                                      <mailto:[email protected]>
 0xA1  0      linux/vtpm_proxy.h                                      TPM Emulator Proxy Driver
+0xA2  all    uapi/linux/acrn.h                                       ACRN hypervisor
 0xA3  80-8F                                                          Port ACL  in development:
                                                                      <mailto:[email protected]>
 0xA3  90-9F  linux/dtlk.h
MAINTAINERS +1 -0
index 3030d0e93d02..d4c1ef303c2d 100644
@@ -443,6 +443,7 @@ S:	Supported
 W:	https://projectacrn.org
 F:	Documentation/virt/acrn/
 F:	drivers/virt/acrn/
+F:	include/uapi/linux/acrn.h
 
 AD1889 ALSA SOUND DRIVER
 L:	[email protected]
drivers/virt/acrn/Makefile +1 -1
index 6920ed798aaf..cf8b4ed5e74e 100644
@@ -1,3 +1,3 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_ACRN_HSM)	:= acrn.o
-acrn-y := hsm.o
+acrn-y := hsm.o vm.o
drivers/virt/acrn/acrn_drv.h +3 -0
index 29eedd696327..72d92b60d944 100644
@@ -3,16 +3,37 @@
 #ifndef __ACRN_HSM_DRV_H
 #define __ACRN_HSM_DRV_H
 
+#include <linux/acrn.h>
+#include <linux/dev_printk.h>
+#include <linux/miscdevice.h>
 #include <linux/types.h>
 
Quoted 5 lines

drivers/virt/acrn/acrn_drv.h line 13

Quoted 3 lines
drivers/virt/acrn/acrn_drv.h +3 -0
+#include "hypercall.h"
+
+extern struct miscdevice acrn_dev;

Who else needs to get to this structure in your driver?

Other files of the driver need to use it for dev_*() log APIs.

Quoted 101 lines
Quoted 101 lines
drivers/virt/acrn/acrn_drv.h +16 -1
+
 #define ACRN_INVALID_VMID (0xffffU)
 
+#define ACRN_VM_FLAG_DESTROYED		0U
+extern struct list_head acrn_vm_list;
+extern rwlock_t acrn_vm_list_lock;
 /**
  * struct acrn_vm - Properties of ACRN User VM.
+ * @list:	Entry within global list of all VMs
  * @vmid:	User VM ID
+ * @vcpu_num:	Number of virtual CPUs in the VM
+ * @flags:	Flags (ACRN_VM_FLAG_*) of the VM. This is VM flag management
+ *		in HSM which is different from the &acrn_vm_creation.vm_flag.
  */
 struct acrn_vm {
-	u16	vmid;
+	struct list_head	list;
+	u16			vmid;
+	int			vcpu_num;
+	unsigned long		flags;
 };
 
+struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
+			       struct acrn_vm_creation *vm_param);
+int acrn_vm_destroy(struct acrn_vm *vm);
+
 #endif /* __ACRN_HSM_DRV_H */
drivers/virt/acrn/hsm.c +57 -1
index 28a3052ffa55..f3e6467b8723 100644
@@ -9,7 +9,6 @@
  *	Yakui Zhao <[email protected]>
  */
 
-#include <linux/miscdevice.h>
 #include <linux/mm.h>
 #include <linux/module.h>
 #include <linux/slab.h>
@@ -38,10 +37,79 @@ static int acrn_dev_open(struct inode *inode, struct file *filp)
 	return 0;
 }
 
+/*
+ * HSM relies on hypercall layer of the ACRN hypervisor to do the
+ * sanity check against the input parameters.
+ */
+static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
+			   unsigned long ioctl_param)
+{
+	struct acrn_vm *vm = filp->private_data;
+	struct acrn_vm_creation *vm_param;
+	int ret = 0;
+
+	if (vm->vmid == ACRN_INVALID_VMID && cmd != ACRN_IOCTL_CREATE_VM) {
+		dev_dbg(acrn_dev.this_device,
+			"ioctl 0x%x: Invalid VM state!\n", cmd);
+		return -EINVAL;
+	}
+
+	switch (cmd) {
+	case ACRN_IOCTL_CREATE_VM:
+		vm_param = memdup_user((void __user *)ioctl_param,
+				       sizeof(struct acrn_vm_creation));
+		if (IS_ERR(vm_param))
+			return PTR_ERR(vm_param);
+
+		vm = acrn_vm_create(vm, vm_param);
+		if (!vm) {
+			ret = -EINVAL;
+			kfree(vm_param);
+			break;
+		}
+
+		if (copy_to_user((void __user *)ioctl_param, vm_param,
+				 sizeof(struct acrn_vm_creation))) {
+			acrn_vm_destroy(vm);
+			ret = -EFAULT;
+		}
+
+		kfree(vm_param);
+		break;
+	case ACRN_IOCTL_START_VM:
+		ret = hcall_start_vm(vm->vmid);
+		if (ret < 0)
+			dev_err(acrn_dev.this_device,
+				"Failed to start VM %u!\n", vm->vmid);
+		break;
+	case ACRN_IOCTL_PAUSE_VM:
+		ret = hcall_pause_vm(vm->vmid);
+		if (ret < 0)
+			dev_err(acrn_dev.this_device,
+				"Failed to pause VM %u!\n", vm->vmid);
+		break;
+	case ACRN_IOCTL_RESET_VM:
+		ret = hcall_reset_vm(vm->vmid);
+		if (ret < 0)
+			dev_err(acrn_dev.this_device,
+				"Failed to restart VM %u!\n", vm->vmid);
+		break;
Quoted 8 lines

drivers/virt/acrn/hsm.c static int acrn_dev_open(struct inode *inode, struct file *filp) · line 101

Quoted 5 lines
drivers/virt/acrn/hsm.c +5 -0
+	case ACRN_IOCTL_DESTROY_VM:
+		ret = acrn_vm_destroy(vm);
+		break;
+	default:
+		dev_warn(acrn_dev.this_device, "Unknown IOCTL 0x%x!\n", cmd);

Do not let userspace spam kernel logs with invalid stuff, that's a sure way to cause a DoS.

OK. Got it. Will be dev_dbg().

Thanks
shuo

#53 lore raw

Re: [PATCH v4 06/17] virt: acrn: Introduce VM management interfaces

In reply to a message not in the archive
Quoted 339 lines · On Tue, Sep 22, 2020 at 07:43:00PM +0800, [email protected] wrote:

From: Shuo Liu <[email protected]>

The VM management interfaces expose several VM operations to ACRN userspace via ioctls. For example, creating VM, starting VM, destroying VM and so on.

The ACRN Hypervisor needs to exchange data with the ACRN userspace during the VM operations. HSM provides VM operation ioctls to the ACRN userspace and communicates with the ACRN Hypervisor for VM operations via hypercalls.

HSM maintains a list of User VM. Each User VM will be bound to an existing file descriptor of /dev/acrn_hsm. The User VM will be destroyed when the file descriptor is closed.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 .../userspace-api/ioctl/ioctl-number.rst      |  1 +
 MAINTAINERS                                   |  1 +
 drivers/virt/acrn/Makefile                    |  2 +-
 drivers/virt/acrn/acrn_drv.h                  | 23 +++++-
 drivers/virt/acrn/hsm.c                       | 73 ++++++++++++++++-
 drivers/virt/acrn/hypercall.h                 | 78 +++++++++++++++++++
 drivers/virt/acrn/vm.c                        | 71 +++++++++++++++++
 include/uapi/linux/acrn.h                     | 56 +++++++++++++
 8 files changed, 301 insertions(+), 4 deletions(-)
 create mode 100644 drivers/virt/acrn/hypercall.h
 create mode 100644 drivers/virt/acrn/vm.c
 create mode 100644 include/uapi/linux/acrn.h
Documentation/userspace-api/ioctl/ioctl-number.rst +1 -0
index 2a198838fca9..ac60efedb104 100644
@@ -319,6 +319,7 @@ Code  Seq#    Include File                                           Comments
 0xA0  all    linux/sdp/sdp.h                                         Industrial Device Project
                                                                      <mailto:[email protected]>
 0xA1  0      linux/vtpm_proxy.h                                      TPM Emulator Proxy Driver
+0xA2  all    uapi/linux/acrn.h                                       ACRN hypervisor
 0xA3  80-8F                                                          Port ACL  in development:
                                                                      <mailto:[email protected]>
 0xA3  90-9F  linux/dtlk.h
MAINTAINERS +1 -0
index 3030d0e93d02..d4c1ef303c2d 100644
@@ -443,6 +443,7 @@ S:	Supported
 W:	https://projectacrn.org
 F:	Documentation/virt/acrn/
 F:	drivers/virt/acrn/
+F:	include/uapi/linux/acrn.h
 
 AD1889 ALSA SOUND DRIVER
 L:	[email protected]
drivers/virt/acrn/Makefile +1 -1
index 6920ed798aaf..cf8b4ed5e74e 100644
@@ -1,3 +1,3 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_ACRN_HSM)	:= acrn.o
-acrn-y := hsm.o
+acrn-y := hsm.o vm.o
drivers/virt/acrn/acrn_drv.h +22 -1
index 29eedd696327..72d92b60d944 100644
@@ -3,16 +3,37 @@
 #ifndef __ACRN_HSM_DRV_H
 #define __ACRN_HSM_DRV_H
 
+#include <linux/acrn.h>
+#include <linux/dev_printk.h>
+#include <linux/miscdevice.h>
 #include <linux/types.h>
 
+#include "hypercall.h"
+
+extern struct miscdevice acrn_dev;
+
 #define ACRN_INVALID_VMID (0xffffU)
 
+#define ACRN_VM_FLAG_DESTROYED		0U
+extern struct list_head acrn_vm_list;
+extern rwlock_t acrn_vm_list_lock;
 /**
  * struct acrn_vm - Properties of ACRN User VM.
+ * @list:	Entry within global list of all VMs
  * @vmid:	User VM ID
+ * @vcpu_num:	Number of virtual CPUs in the VM
+ * @flags:	Flags (ACRN_VM_FLAG_*) of the VM. This is VM flag management
+ *		in HSM which is different from the &acrn_vm_creation.vm_flag.
  */
 struct acrn_vm {
-	u16	vmid;
+	struct list_head	list;
+	u16			vmid;
+	int			vcpu_num;
+	unsigned long		flags;
 };
 
+struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
+			       struct acrn_vm_creation *vm_param);
+int acrn_vm_destroy(struct acrn_vm *vm);
+
 #endif /* __ACRN_HSM_DRV_H */
drivers/virt/acrn/hsm.c +71 -2
index 28a3052ffa55..f3e6467b8723 100644
@@ -9,7 +9,6 @@
  *	Yakui Zhao <[email protected]>
  */
 
-#include <linux/miscdevice.h>
 #include <linux/mm.h>
 #include <linux/module.h>
 #include <linux/slab.h>
@@ -38,10 +37,79 @@ static int acrn_dev_open(struct inode *inode, struct file *filp)
 	return 0;
 }
 
+/*
+ * HSM relies on hypercall layer of the ACRN hypervisor to do the
+ * sanity check against the input parameters.
+ */
+static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
+			   unsigned long ioctl_param)
+{
+	struct acrn_vm *vm = filp->private_data;
+	struct acrn_vm_creation *vm_param;
+	int ret = 0;
+
+	if (vm->vmid == ACRN_INVALID_VMID && cmd != ACRN_IOCTL_CREATE_VM) {
+		dev_dbg(acrn_dev.this_device,
+			"ioctl 0x%x: Invalid VM state!\n", cmd);
+		return -EINVAL;
+	}
+
+	switch (cmd) {
+	case ACRN_IOCTL_CREATE_VM:
+		vm_param = memdup_user((void __user *)ioctl_param,
+				       sizeof(struct acrn_vm_creation));
+		if (IS_ERR(vm_param))
+			return PTR_ERR(vm_param);
+
+		vm = acrn_vm_create(vm, vm_param);
+		if (!vm) {
+			ret = -EINVAL;
+			kfree(vm_param);
+			break;
+		}
+
+		if (copy_to_user((void __user *)ioctl_param, vm_param,
+				 sizeof(struct acrn_vm_creation))) {
+			acrn_vm_destroy(vm);
+			ret = -EFAULT;
+		}
+
+		kfree(vm_param);
+		break;
+	case ACRN_IOCTL_START_VM:
+		ret = hcall_start_vm(vm->vmid);
+		if (ret < 0)
+			dev_err(acrn_dev.this_device,
+				"Failed to start VM %u!\n", vm->vmid);
+		break;
+	case ACRN_IOCTL_PAUSE_VM:
+		ret = hcall_pause_vm(vm->vmid);
+		if (ret < 0)
+			dev_err(acrn_dev.this_device,
+				"Failed to pause VM %u!\n", vm->vmid);
+		break;
+	case ACRN_IOCTL_RESET_VM:
+		ret = hcall_reset_vm(vm->vmid);
+		if (ret < 0)
+			dev_err(acrn_dev.this_device,
+				"Failed to restart VM %u!\n", vm->vmid);
+		break;
+	case ACRN_IOCTL_DESTROY_VM:
+		ret = acrn_vm_destroy(vm);
+		break;
+	default:
+		dev_warn(acrn_dev.this_device, "Unknown IOCTL 0x%x!\n", cmd);
+		ret = -ENOTTY;
+	}
+
+	return ret;
+}
+
 static int acrn_dev_release(struct inode *inode, struct file *filp)
 {
 	struct acrn_vm *vm = filp->private_data;
 
+	acrn_vm_destroy(vm);
 	kfree(vm);
 	return 0;
 }
@@ -50,9 +118,10 @@ static const struct file_operations acrn_fops = {
 	.owner		= THIS_MODULE,
 	.open		= acrn_dev_open,
 	.release	= acrn_dev_release,
+	.unlocked_ioctl = acrn_dev_ioctl,
 };
 
-static struct miscdevice acrn_dev = {
+struct miscdevice acrn_dev = {
 	.minor	= MISC_DYNAMIC_MINOR,
 	.name	= "acrn_hsm",
 	.fops	= &acrn_fops,
drivers/virt/acrn/hypercall.h new file +78 -0
new file mode 100644
index 000000000000..426b66cadb1f
@@ -0,0 +1,78 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * ACRN HSM: hypercalls of ACRN Hypervisor
+ */
+#ifndef __ACRN_HSM_HYPERCALL_H
+#define __ACRN_HSM_HYPERCALL_H
+#include <asm/acrn.h>
+
+/*
+ * Hypercall IDs of the ACRN Hypervisor
+ */
+#define _HC_ID(x, y) (((x) << 24) | (y))
+
+#define HC_ID 0x80UL
+
+#define HC_ID_VM_BASE			0x10UL
+#define HC_CREATE_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x00)
+#define HC_DESTROY_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x01)
+#define HC_START_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x02)
+#define HC_PAUSE_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x03)
+#define HC_RESET_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x05)
+
+/**
+ * hcall_create_vm() - Create a User VM
+ * @vminfo:	Service VM GPA of info of User VM creation
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_create_vm(u64 vminfo)
+{
+	return acrn_hypercall1(HC_CREATE_VM, vminfo);
+}
+
+/**
+ * hcall_start_vm() - Start a User VM
+ * @vmid:	User VM ID
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_start_vm(u64 vmid)
+{
+	return acrn_hypercall1(HC_START_VM, vmid);
+}
+
+/**
+ * hcall_pause_vm() - Pause a User VM
+ * @vmid:	User VM ID
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_pause_vm(u64 vmid)
+{
+	return acrn_hypercall1(HC_PAUSE_VM, vmid);
+}
+
+/**
+ * hcall_destroy_vm() - Destroy a User VM
+ * @vmid:	User VM ID
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_destroy_vm(u64 vmid)
+{
+	return acrn_hypercall1(HC_DESTROY_VM, vmid);
+}
+
+/**
+ * hcall_reset_vm() - Reset a User VM
+ * @vmid:	User VM ID
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_reset_vm(u64 vmid)
+{
+	return acrn_hypercall1(HC_RESET_VM, vmid);
+}
+
+#endif /* __ACRN_HSM_HYPERCALL_H */
drivers/virt/acrn/vm.c new file +32 -0
new file mode 100644
index 000000000000..920ca48f4847
@@ -0,0 +1,71 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * ACRN_HSM: Virtual Machine management
+ *
+ * Copyright (C) 2020 Intel Corporation. All rights reserved.
+ *
+ * Authors:
+ *	Jason Chen CJ <[email protected]>
+ *	Yakui Zhao <[email protected]>
+ */
+#include <linux/io.h>
+#include <linux/mm.h>
+#include <linux/slab.h>
+
+#include "acrn_drv.h"
+
+/* List of VMs */
+LIST_HEAD(acrn_vm_list);
+/*
+ * acrn_vm_list is read in a tasklet which dispatch I/O requests and is wrote
+ * in VM creation ioctl. Use the rwlock mechanism to protect it.
+ */
+DEFINE_RWLOCK(acrn_vm_list_lock);
+
+struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
+			       struct acrn_vm_creation *vm_param)
+{
+	int ret;
+
+	ret = hcall_create_vm(virt_to_phys(vm_param));
+	if (ret < 0 || vm_param->vmid == ACRN_INVALID_VMID) {
+		dev_err(acrn_dev.this_device,

drivers/virt/acrn/vm.c line 42

Quoted 10 lines
drivers/virt/acrn/vm.c new file +10 -0
+			"Failed to create VM! Error: %d\n", ret);
+		return NULL;
+	}
+
+	vm->vmid = vm_param->vmid;
+	vm->vcpu_num = vm_param->vcpu_num;
+
+	write_lock_bh(&acrn_vm_list_lock);
+	list_add(&vm->list, &acrn_vm_list);
+	write_unlock_bh(&acrn_vm_list_lock);

Why are the _bh() variants being used here?

You are only accessing this list from userspace context in this patch.

Heck, you aren't even reading from the list, only writing to it...

greg k-h

author
#54 lore raw

Re: [PATCH v4 06/17] virt: acrn: Introduce VM management interfaces

Hi Greg,

Quoted 353 lines · Greg Kroah-Hartman
Quoted 339 lines · On Tue, Sep 22, 2020 at 07:43:00PM +0800, [email protected] wrote:

From: Shuo Liu <[email protected]>

The VM management interfaces expose several VM operations to ACRN userspace via ioctls. For example, creating VM, starting VM, destroying VM and so on.

The ACRN Hypervisor needs to exchange data with the ACRN userspace during the VM operations. HSM provides VM operation ioctls to the ACRN userspace and communicates with the ACRN Hypervisor for VM operations via hypercalls.

HSM maintains a list of User VM. Each User VM will be bound to an existing file descriptor of /dev/acrn_hsm. The User VM will be destroyed when the file descriptor is closed.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 .../userspace-api/ioctl/ioctl-number.rst      |  1 +
 MAINTAINERS                                   |  1 +
 drivers/virt/acrn/Makefile                    |  2 +-
 drivers/virt/acrn/acrn_drv.h                  | 23 +++++-
 drivers/virt/acrn/hsm.c                       | 73 ++++++++++++++++-
 drivers/virt/acrn/hypercall.h                 | 78 +++++++++++++++++++
 drivers/virt/acrn/vm.c                        | 71 +++++++++++++++++
 include/uapi/linux/acrn.h                     | 56 +++++++++++++
 8 files changed, 301 insertions(+), 4 deletions(-)
 create mode 100644 drivers/virt/acrn/hypercall.h
 create mode 100644 drivers/virt/acrn/vm.c
 create mode 100644 include/uapi/linux/acrn.h
Documentation/userspace-api/ioctl/ioctl-number.rst +1 -0
index 2a198838fca9..ac60efedb104 100644
@@ -319,6 +319,7 @@ Code  Seq#    Include File                                           Comments
 0xA0  all    linux/sdp/sdp.h                                         Industrial Device Project
                                                                      <mailto:[email protected]>
 0xA1  0      linux/vtpm_proxy.h                                      TPM Emulator Proxy Driver
+0xA2  all    uapi/linux/acrn.h                                       ACRN hypervisor
 0xA3  80-8F                                                          Port ACL  in development:
                                                                      <mailto:[email protected]>
 0xA3  90-9F  linux/dtlk.h
MAINTAINERS +1 -0
index 3030d0e93d02..d4c1ef303c2d 100644
@@ -443,6 +443,7 @@ S:	Supported
 W:	https://projectacrn.org
 F:	Documentation/virt/acrn/
 F:	drivers/virt/acrn/
+F:	include/uapi/linux/acrn.h
 
 AD1889 ALSA SOUND DRIVER
 L:	[email protected]
drivers/virt/acrn/Makefile +1 -1
index 6920ed798aaf..cf8b4ed5e74e 100644
@@ -1,3 +1,3 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_ACRN_HSM)	:= acrn.o
-acrn-y := hsm.o
+acrn-y := hsm.o vm.o
drivers/virt/acrn/acrn_drv.h +22 -1
index 29eedd696327..72d92b60d944 100644
@@ -3,16 +3,37 @@
 #ifndef __ACRN_HSM_DRV_H
 #define __ACRN_HSM_DRV_H
 
+#include <linux/acrn.h>
+#include <linux/dev_printk.h>
+#include <linux/miscdevice.h>
 #include <linux/types.h>
 
+#include "hypercall.h"
+
+extern struct miscdevice acrn_dev;
+
 #define ACRN_INVALID_VMID (0xffffU)
 
+#define ACRN_VM_FLAG_DESTROYED		0U
+extern struct list_head acrn_vm_list;
+extern rwlock_t acrn_vm_list_lock;
 /**
  * struct acrn_vm - Properties of ACRN User VM.
+ * @list:	Entry within global list of all VMs
  * @vmid:	User VM ID
+ * @vcpu_num:	Number of virtual CPUs in the VM
+ * @flags:	Flags (ACRN_VM_FLAG_*) of the VM. This is VM flag management
+ *		in HSM which is different from the &acrn_vm_creation.vm_flag.
  */
 struct acrn_vm {
-	u16	vmid;
+	struct list_head	list;
+	u16			vmid;
+	int			vcpu_num;
+	unsigned long		flags;
 };
 
+struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
+			       struct acrn_vm_creation *vm_param);
+int acrn_vm_destroy(struct acrn_vm *vm);
+
 #endif /* __ACRN_HSM_DRV_H */
drivers/virt/acrn/hsm.c +71 -2
index 28a3052ffa55..f3e6467b8723 100644
@@ -9,7 +9,6 @@
  *	Yakui Zhao <[email protected]>
  */
 
-#include <linux/miscdevice.h>
 #include <linux/mm.h>
 #include <linux/module.h>
 #include <linux/slab.h>
@@ -38,10 +37,79 @@ static int acrn_dev_open(struct inode *inode, struct file *filp)
 	return 0;
 }
 
+/*
+ * HSM relies on hypercall layer of the ACRN hypervisor to do the
+ * sanity check against the input parameters.
+ */
+static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
+			   unsigned long ioctl_param)
+{
+	struct acrn_vm *vm = filp->private_data;
+	struct acrn_vm_creation *vm_param;
+	int ret = 0;
+
+	if (vm->vmid == ACRN_INVALID_VMID && cmd != ACRN_IOCTL_CREATE_VM) {
+		dev_dbg(acrn_dev.this_device,
+			"ioctl 0x%x: Invalid VM state!\n", cmd);
+		return -EINVAL;
+	}
+
+	switch (cmd) {
+	case ACRN_IOCTL_CREATE_VM:
+		vm_param = memdup_user((void __user *)ioctl_param,
+				       sizeof(struct acrn_vm_creation));
+		if (IS_ERR(vm_param))
+			return PTR_ERR(vm_param);
+
+		vm = acrn_vm_create(vm, vm_param);
+		if (!vm) {
+			ret = -EINVAL;
+			kfree(vm_param);
+			break;
+		}
+
+		if (copy_to_user((void __user *)ioctl_param, vm_param,
+				 sizeof(struct acrn_vm_creation))) {
+			acrn_vm_destroy(vm);
+			ret = -EFAULT;
+		}
+
+		kfree(vm_param);
+		break;
+	case ACRN_IOCTL_START_VM:
+		ret = hcall_start_vm(vm->vmid);
+		if (ret < 0)
+			dev_err(acrn_dev.this_device,
+				"Failed to start VM %u!\n", vm->vmid);
+		break;
+	case ACRN_IOCTL_PAUSE_VM:
+		ret = hcall_pause_vm(vm->vmid);
+		if (ret < 0)
+			dev_err(acrn_dev.this_device,
+				"Failed to pause VM %u!\n", vm->vmid);
+		break;
+	case ACRN_IOCTL_RESET_VM:
+		ret = hcall_reset_vm(vm->vmid);
+		if (ret < 0)
+			dev_err(acrn_dev.this_device,
+				"Failed to restart VM %u!\n", vm->vmid);
+		break;
+	case ACRN_IOCTL_DESTROY_VM:
+		ret = acrn_vm_destroy(vm);
+		break;
+	default:
+		dev_warn(acrn_dev.this_device, "Unknown IOCTL 0x%x!\n", cmd);
+		ret = -ENOTTY;
+	}
+
+	return ret;
+}
+
 static int acrn_dev_release(struct inode *inode, struct file *filp)
 {
 	struct acrn_vm *vm = filp->private_data;
 
+	acrn_vm_destroy(vm);
 	kfree(vm);
 	return 0;
 }
@@ -50,9 +118,10 @@ static const struct file_operations acrn_fops = {
 	.owner		= THIS_MODULE,
 	.open		= acrn_dev_open,
 	.release	= acrn_dev_release,
+	.unlocked_ioctl = acrn_dev_ioctl,
 };
 
-static struct miscdevice acrn_dev = {
+struct miscdevice acrn_dev = {
 	.minor	= MISC_DYNAMIC_MINOR,
 	.name	= "acrn_hsm",
 	.fops	= &acrn_fops,
drivers/virt/acrn/hypercall.h new file +78 -0
new file mode 100644
index 000000000000..426b66cadb1f
@@ -0,0 +1,78 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * ACRN HSM: hypercalls of ACRN Hypervisor
+ */
+#ifndef __ACRN_HSM_HYPERCALL_H
+#define __ACRN_HSM_HYPERCALL_H
+#include <asm/acrn.h>
+
+/*
+ * Hypercall IDs of the ACRN Hypervisor
+ */
+#define _HC_ID(x, y) (((x) << 24) | (y))
+
+#define HC_ID 0x80UL
+
+#define HC_ID_VM_BASE			0x10UL
+#define HC_CREATE_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x00)
+#define HC_DESTROY_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x01)
+#define HC_START_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x02)
+#define HC_PAUSE_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x03)
+#define HC_RESET_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x05)
+
+/**
+ * hcall_create_vm() - Create a User VM
+ * @vminfo:	Service VM GPA of info of User VM creation
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_create_vm(u64 vminfo)
+{
+	return acrn_hypercall1(HC_CREATE_VM, vminfo);
+}
+
+/**
+ * hcall_start_vm() - Start a User VM
+ * @vmid:	User VM ID
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_start_vm(u64 vmid)
+{
+	return acrn_hypercall1(HC_START_VM, vmid);
+}
+
+/**
+ * hcall_pause_vm() - Pause a User VM
+ * @vmid:	User VM ID
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_pause_vm(u64 vmid)
+{
+	return acrn_hypercall1(HC_PAUSE_VM, vmid);
+}
+
+/**
+ * hcall_destroy_vm() - Destroy a User VM
+ * @vmid:	User VM ID
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_destroy_vm(u64 vmid)
+{
+	return acrn_hypercall1(HC_DESTROY_VM, vmid);
+}
+
+/**
+ * hcall_reset_vm() - Reset a User VM
+ * @vmid:	User VM ID
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_reset_vm(u64 vmid)
+{
+	return acrn_hypercall1(HC_RESET_VM, vmid);
+}
+
+#endif /* __ACRN_HSM_HYPERCALL_H */
drivers/virt/acrn/vm.c new file +32 -0
new file mode 100644
index 000000000000..920ca48f4847
@@ -0,0 +1,71 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * ACRN_HSM: Virtual Machine management
+ *
+ * Copyright (C) 2020 Intel Corporation. All rights reserved.
+ *
+ * Authors:
+ *	Jason Chen CJ <[email protected]>
+ *	Yakui Zhao <[email protected]>
+ */
+#include <linux/io.h>
+#include <linux/mm.h>
+#include <linux/slab.h>
+
+#include "acrn_drv.h"
+
+/* List of VMs */
+LIST_HEAD(acrn_vm_list);
+/*
+ * acrn_vm_list is read in a tasklet which dispatch I/O requests and is wrote
+ * in VM creation ioctl. Use the rwlock mechanism to protect it.
+ */
+DEFINE_RWLOCK(acrn_vm_list_lock);
+
+struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
+			       struct acrn_vm_creation *vm_param)
+{
+	int ret;
+
+	ret = hcall_create_vm(virt_to_phys(vm_param));
+	if (ret < 0 || vm_param->vmid == ACRN_INVALID_VMID) {
+		dev_err(acrn_dev.this_device,

drivers/virt/acrn/vm.c line 42

Quoted 10 lines
drivers/virt/acrn/vm.c new file +10 -0
+			"Failed to create VM! Error: %d\n", ret);
+		return NULL;
+	}
+
+	vm->vmid = vm_param->vmid;
+	vm->vcpu_num = vm_param->vcpu_num;
+
+	write_lock_bh(&acrn_vm_list_lock);
+	list_add(&vm->list, &acrn_vm_list);
+	write_unlock_bh(&acrn_vm_list_lock);

Why are the _bh() variants being used here?

Quoted 3 lines

You are only accessing this list from userspace context in this patch.

Heck, you aren't even reading from the list, only writing to it...

acrn_vm_list is read in a tasklet which dispatch I/O requests and is wrote in VM creation ioctl. Use the rwlock mechanism to protect it. The reading operation is introduced in the following patches of this series. So i keep the lock type at the moment of introduction.

Thanks
shuo

#55 lore raw

Re: [PATCH v4 06/17] virt: acrn: Introduce VM management interfaces

Quoted 6 lines · Shuo A Liu
Quoted 6 lines

Review

Quoted 3 lines
(quoted diff) +3 -0
+	write_lock_bh(&acrn_vm_list_lock);
+	list_add(&vm->list, &acrn_vm_list);
+	write_unlock_bh(&acrn_vm_list_lock);

Why are the _bh() variants being used here?

Quoted 8 lines
Quoted 3 lines

You are only accessing this list from userspace context in this patch.

Heck, you aren't even reading from the list, only writing to it...

acrn_vm_list is read in a tasklet which dispatch I/O requests and is wrote in VM creation ioctl. Use the rwlock mechanism to protect it. The reading operation is introduced in the following patches of this series. So i keep the lock type at the moment of introduction.

Ok, but think about someone trying to review this code. Does this lock actually make sense here? No, it does not. How am I supposed to know to look at future patches to determine that it changes location and usage to require this?

That's just not fair, would you want to review something like this?

And a HUGE meta-comment, again, why am I the only one reviewing this stuff? Why do you have a ton of Intel people on the Cc: yet it is, once again, my job to do this?

If you all are wanting to burn me out, you are doing a good job...

greg k-h

author
#56 lore raw

Re: [PATCH v4 06/17] virt: acrn: Introduce VM management interfaces

Quoted 16 lines · Greg Kroah-Hartman
Quoted 6 lines · Shuo A Liu
Quoted 6 lines

Review

Quoted 3 lines
(quoted diff) +3 -0
+	write_lock_bh(&acrn_vm_list_lock);
+	list_add(&vm->list, &acrn_vm_list);
+	write_unlock_bh(&acrn_vm_list_lock);

Why are the _bh() variants being used here?

Quoted 8 lines
Quoted 3 lines

You are only accessing this list from userspace context in this patch.

Heck, you aren't even reading from the list, only writing to it...

acrn_vm_list is read in a tasklet which dispatch I/O requests and is wrote in VM creation ioctl. Use the rwlock mechanism to protect it. The reading operation is introduced in the following patches of this series. So i keep the lock type at the moment of introduction.

Quoted 4 lines

Ok, but think about someone trying to review this code. Does this lock actually make sense here? No, it does not. How am I supposed to know to look at future patches to determine that it changes location and usage to require this?

OK. May i know how to handle such kind of code submission? Or which way following do you prefer?
1) Use a mutex lock here, then change it to rwlock in a later patch of this series.
2) Add more comments in changelog about the lock. (Now, there is comment around the acrn_vm_list_lock)

Quoted 6 lines

That's just not fair, would you want to review something like this?

And a HUGE meta-comment, again, why am I the only one reviewing this stuff? Why do you have a ton of Intel people on the Cc: yet it is, once again, my job to do this?

The patchset has been reviewed in Intel's internal mailist several rounds and got Reviewed-by: before send out. That's why i Cced many Intel people as well.

This patchset is all about a common driver for the ACRN hypervisor support. I put the code in drivers/virt/ and found you are one of the maintainer of vboxguest driver which is in the same subdirectory. I thought you should be the right person to be Cced when i submitted this series.

Certainly, any comments are welcome. And really appreciate your review and help. I have little experience to submit a new driver to the community, my apologies if thing goes wrong.

Thanks
shuo

#57 lore raw

Re: [PATCH v4 06/17] virt: acrn: Introduce VM management interfaces

Quoted 22 lines · Shuo A Liu
Quoted 16 lines · Greg Kroah-Hartman
Quoted 6 lines · Shuo A Liu
Quoted 6 lines

Review

Quoted 3 lines
(quoted diff) +3 -0
+	write_lock_bh(&acrn_vm_list_lock);
+	list_add(&vm->list, &acrn_vm_list);
+	write_unlock_bh(&acrn_vm_list_lock);

Why are the _bh() variants being used here?

Quoted 8 lines
Quoted 3 lines

You are only accessing this list from userspace context in this patch.

Heck, you aren't even reading from the list, only writing to it...

acrn_vm_list is read in a tasklet which dispatch I/O requests and is wrote in VM creation ioctl. Use the rwlock mechanism to protect it. The reading operation is introduced in the following patches of this series. So i keep the lock type at the moment of introduction.

Quoted 4 lines

Ok, but think about someone trying to review this code. Does this lock actually make sense here? No, it does not. How am I supposed to know to look at future patches to determine that it changes location and usage to require this?

Quoted 4 lines

OK. May i know how to handle such kind of code submission? Or which way following do you prefer?
1) Use a mutex lock here, then change it to rwlock in a later patch of this series.

Wouldn't this make more sense if you had to read these one after another?

Quoted 2 lines

2) Add more comments in changelog about the lock. (Now, there is comment around the acrn_vm_list_lock)

It's hard to verify a comment's statement without digging through other patches in the series, right? You want the reviewer to just trust you? :)

Again, what would _YOU_ want to see if you had to review this?

Quoted 9 lines
Quoted 5 lines

That's just not fair, would you want to review something like this?

And a HUGE meta-comment, again, why am I the only one reviewing this stuff? Why do you have a ton of Intel people on the Cc: yet it is, once again, my job to do this?

The patchset has been reviewed in Intel's internal mailist several rounds and got Reviewed-by: before send out. That's why i Cced many Intel people as well.

Then why didn't any of those intel people on the cc: actually review it after you have sent it out? Why is it only me? Do I need to wait longer for them to get to this? I'll gladly do so next time...

Quoted 5 lines

This patchset is all about a common driver for the ACRN hypervisor support. I put the code in drivers/virt/ and found you are one of the maintainer of vboxguest driver which is in the same subdirectory. I thought you should be the right person to be Cced when i submitted this series.

I am, I'm not complaining about that. I'm complaining that it seems to be _only_ me reviewing this here, and not any of the people you are cc:ing from intel. Most of those people should be giving you this same type of review comments and not forcing an external person to do so, right?

Quoted 3 lines

Certainly, any comments are welcome. And really appreciate your review and help. I have little experience to submit a new driver to the community, my apologies if thing goes wrong.

You didn't do anything wrong, I'm arguing about the larger meta-issue I have right now with Intel and the lack of reviews that seems to happen from other Intel people on their co-workers patches.

Anyway, you are doing fine, it's an iterative process, hopefully you can also review other people's patches in this area that are being posted as well.

thanks,

greg k-h

author
#58 lore raw

Re: [PATCH v4 06/17] virt: acrn: Introduce VM management interfaces

Quoted 23 lines · Greg Kroah-Hartman
Quoted 22 lines · Shuo A Liu
Quoted 20 lines · Greg Kroah-Hartman
Quoted 6 lines · Shuo A Liu
Quoted 6 lines

Review

Quoted 3 lines
(quoted diff) +3 -0
+	write_lock_bh(&acrn_vm_list_lock);
+	list_add(&vm->list, &acrn_vm_list);
+	write_unlock_bh(&acrn_vm_list_lock);

Why are the _bh() variants being used here?

Quoted 8 lines
Quoted 3 lines

You are only accessing this list from userspace context in this patch.

Heck, you aren't even reading from the list, only writing to it...

acrn_vm_list is read in a tasklet which dispatch I/O requests and is wrote in VM creation ioctl. Use the rwlock mechanism to protect it. The reading operation is introduced in the following patches of this series. So i keep the lock type at the moment of introduction.

Ok, but think about someone trying to review this code. Does this lock actually make sense here? No, it does not. How am I supposed to know to look at future patches to determine that it changes location and usage to require this?

Quoted 7 lines
Quoted 4 lines

OK. May i know how to handle such kind of code submission? Or which way following do you prefer?
1) Use a mutex lock here, then change it to rwlock in a later patch of this series.

Wouldn't this make more sense if you had to read these one after another?

OK. I will change to mutex firstly for more readable.

Quoted 39 lines
Quoted 3 lines

2) Add more comments in changelog about the lock. (Now, there is comment around the acrn_vm_list_lock)

It's hard to verify a comment's statement without digging through other patches in the series, right? You want the reviewer to just trust you? :)

Again, what would _YOU_ want to see if you had to review this?

Quoted 9 lines
Quoted 5 lines

That's just not fair, would you want to review something like this?

And a HUGE meta-comment, again, why am I the only one reviewing this stuff? Why do you have a ton of Intel people on the Cc: yet it is, once again, my job to do this?

The patchset has been reviewed in Intel's internal mailist several rounds and got Reviewed-by: before send out. That's why i Cced many Intel people as well.

Then why didn't any of those intel people on the cc: actually review it after you have sent it out? Why is it only me? Do I need to wait longer for them to get to this? I'll gladly do so next time...

Quoted 5 lines

This patchset is all about a common driver for the ACRN hypervisor support. I put the code in drivers/virt/ and found you are one of the maintainer of vboxguest driver which is in the same subdirectory. I thought you should be the right person to be Cced when i submitted this series.

I am, I'm not complaining about that. I'm complaining that it seems to be _only_ me reviewing this here, and not any of the people you are cc:ing from intel. Most of those people should be giving you this same type of review comments and not forcing an external person to do so, right?

Quoted 3 lines

Certainly, any comments are welcome. And really appreciate your review and help. I have little experience to submit a new driver to the community, my apologies if thing goes wrong.

Quoted 7 lines

You didn't do anything wrong, I'm arguing about the larger meta-issue I have right now with Intel and the lack of reviews that seems to happen from other Intel people on their co-workers patches.

Anyway, you are doing fine, it's an iterative process, hopefully you can also review other people's patches in this area that are being posted as well.

Sorry, i have no answer about some of your question above. :( However, i will try my best to help review other people's patches in this area.

Thanks
shuo