Skip to contentSkip to conversation

HSM driver for ACRN hypervisor

PATCH Superseded [email protected] 7 participants 61 messages Oct 19, 2020 – Nov 11, 2020

Subsystems (3)

v5 of 9 Superseded Latest is v9 →

lore
Share & tools
lore
#0 lore raw

v5 cover HSM driver for ACRN hypervisor

From: Shuo Liu <[email protected]>

ACRN is a Type 1 reference hypervisor stack, running directly on the bare-metal hardware, and is suitable for a variety of IoT and embedded device solutions.

ACRN implements a hybrid VMM architecture, using a privileged Service VM. The Service VM manages the system resources (CPU, memory, etc.) and I/O devices of User VMs. Multiple User VMs are supported, with each of them running Linux, Android OS or Windows. Both Service VM and User VMs are guest VM.

Below figure shows the architecture.

                Service VM                    User VM
      +----------------------------+  |  +------------------+
      |        +--------------+    |  |  |                  |
      |        |ACRN userspace|    |  |  |                  |
      |        +--------------+    |  |  |                  |
      |-----------------ioctl------|  |  |                  |   ...
      |kernel space   +----------+ |  |  |                  |
      |               |   HSM    | |  |  | Drivers          |
      |               +----------+ |  |  |                  |
      +--------------------|-------+  |  +------------------+
  +---------------------hypercall----------------------------------------+
  |                       ACRN Hypervisor                                |
  +----------------------------------------------------------------------+
  |                          Hardware                                    |
  +----------------------------------------------------------------------+

There is only one Service VM which could run Linux as OS.

In a typical case, the Service VM will be auto started when ACRN Hypervisor is booted. Then the ACRN userspace (an application running in Service VM) could be used to start/stop User VMs by communicating with ACRN Hypervisor Service Module (HSM).

ACRN Hypervisor Service Module (HSM) is a middle layer that allows the ACRN userspace and Service VM OS kernel to communicate with ACRN Hypervisor and manage different User VMs. This middle layer provides the following functionalities,
- Issues hypercalls to the hypervisor to manage User VMs:
* VM/vCPU management
* Memory management
* Device passthrough
* Interrupts injection
- I/O requests handling from User VMs.
- Exports ioctl through HSM char device.
- Exports function calls for other kernel modules

ACRN is focused on embedded system. So it doesn't support some features. E.g.,
- ACRN doesn't support VM migration.
- ACRN doesn't support vCPU migration.

This patch set adds the HSM to the Linux kernel.

The basic ARCN support was merged to upstream already.
https://lore.kernel.org/lkml/[email protected]/

ChangeLog:
v5:
- Corrected typo in documentation.
- Removed unused pr_fmt().
- Used supported constraint with a explicit MOV to R8 at beginning of ASM for hypercall interface.
- Used dev_dbg() to replace dev_err() in places which might cause a DoS.
- Introduced acrn_vm_list_lock as a mutex for friendly review.
- Changed to use default attribute group list to add attribute files.

v4:
- Used acrn_dev.this_device directly for dev_*() (Reinette)
- Removed the odd usage of {get|put}_device() on &acrn_dev->this_device (Greg)
- Removed unused log code. (Greg)
- Corrected the return error values. (Greg)
- Mentioned that HSM relies hypervisor for sanity check in acrn_dev_ioctl() comments (Greg)

v3:
- Used {get|put}_device() helpers on &acrn_dev->this_device
- Moved unused code from front patches to later ones.
- Removed self-defined pr_fmt() and dev_fmt()
- Provided comments for acrn_vm_list_lock.

v2:
- Removed API version related code. (Dave)
- Replaced pr_*() by dev_*(). (Greg)
- Used -ENOTTY as the error code of unsupported ioctl. (Greg)

Shuo Liu (16):
docs: acrn: Introduce ACRN
x86/acrn: Introduce acrn_{setup, remove}_intr_handler() x86/acrn: Introduce hypercall interfaces
virt: acrn: Introduce ACRN HSM basic driver
virt: acrn: Introduce VM management interfaces
virt: acrn: Introduce an ioctl to set vCPU registers state
virt: acrn: Introduce EPT mapping management
virt: acrn: Introduce I/O request management
virt: acrn: Introduce PCI configuration space PIO accesses combiner
virt: acrn: Introduce interfaces for PCI device passthrough
virt: acrn: Introduce interrupt injection interfaces
virt: acrn: Introduce interfaces to query C-states and P-states
allowed by hypervisor
virt: acrn: Introduce I/O ranges operation interfaces
virt: acrn: Introduce ioeventfd
virt: acrn: Introduce irqfd
virt: acrn: Introduce an interface for Service VM to control vCPU

Yin Fengwei (1):
x86/acrn: Introduce an API to check if a VM is privileged

 .../userspace-api/ioctl/ioctl-number.rst      |   1 +
 Documentation/virt/acrn/index.rst             |  11 +
 Documentation/virt/acrn/introduction.rst      |  40 ++
 Documentation/virt/acrn/io-request.rst        |  97 +++
 Documentation/virt/index.rst                  |   1 +
 MAINTAINERS                                   |   9 +
 arch/x86/include/asm/acrn.h                   |  71 ++
 arch/x86/kernel/cpu/acrn.c                    |  33 +-
 drivers/virt/Kconfig                          |   2 +
 drivers/virt/Makefile                         |   1 +
 drivers/virt/acrn/Kconfig                     |  15 +
 drivers/virt/acrn/Makefile                    |   3 +
 drivers/virt/acrn/acrn_drv.h                  | 229 +++++++
 drivers/virt/acrn/hsm.c                       | 435 ++++++++++++
 drivers/virt/acrn/hypercall.h                 | 254 +++++++
 drivers/virt/acrn/ioeventfd.c                 | 273 ++++++++
 drivers/virt/acrn/ioreq.c                     | 645 ++++++++++++++++++
 drivers/virt/acrn/irqfd.c                     | 235 +++++++
 drivers/virt/acrn/mm.c                        | 305 +++++++++
 drivers/virt/acrn/vm.c                        | 126 ++++
 include/uapi/linux/acrn.h                     | 486 +++++++++++++
 21 files changed, 3271 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/virt/acrn/index.rst
 create mode 100644 Documentation/virt/acrn/introduction.rst
 create mode 100644 Documentation/virt/acrn/io-request.rst
 create mode 100644 arch/x86/include/asm/acrn.h
 create mode 100644 drivers/virt/acrn/Kconfig
 create mode 100644 drivers/virt/acrn/Makefile
 create mode 100644 drivers/virt/acrn/acrn_drv.h
 create mode 100644 drivers/virt/acrn/hsm.c
 create mode 100644 drivers/virt/acrn/hypercall.h
 create mode 100644 drivers/virt/acrn/ioeventfd.c
 create mode 100644 drivers/virt/acrn/ioreq.c
 create mode 100644 drivers/virt/acrn/irqfd.c
 create mode 100644 drivers/virt/acrn/mm.c
 create mode 100644 drivers/virt/acrn/vm.c
 create mode 100644 include/uapi/linux/acrn.h

base-commit: 18445bf405cb331117bc98427b1ba6f12418ad17

Signature · 1 line
2.28.0
#1 lore raw

PATCH v5 1/17 docs: acrn: Introduce ACRN

From: Shuo Liu <[email protected]>

Add documentation on the following aspects of ACRN:

1) A brief introduction on the architecture of ACRN.
2) I/O request handling in ACRN.

To learn more about ACRN, please go to ACRN project website
https://projectacrn.org, or the documentation page
https://projectacrn.github.io/.

Cc Dave Hansen <[email protected]>
Cc Sen Christopherson <[email protected]>
Cc Dan Williams <[email protected]>
Cc Fengwei Yin <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
Cc Randy Dunlap <[email protected]>
---
 Documentation/virt/acrn/index.rst        | 11 +++
 Documentation/virt/acrn/introduction.rst | 40 ++++++++++
 Documentation/virt/acrn/io-request.rst   | 97 ++++++++++++++++++++++++
 Documentation/virt/index.rst             |  1 +
 MAINTAINERS                              |  7 ++
 5 files changed, 156 insertions(+)
 create mode 100644 Documentation/virt/acrn/index.rst
 create mode 100644 Documentation/virt/acrn/introduction.rst
 create mode 100644 Documentation/virt/acrn/io-request.rst
Documentation/virt/acrn/index.rst new file +11 -0
new file mode 100644
index 000000000000..e3cf99033bdb
@@ -0,0 +1,11 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+===============
+ACRN Hypervisor
+===============
+
+.. toctree::
+   :maxdepth: 1
+
+   introduction
+   io-request
Documentation/virt/acrn/introduction.rst new file +40 -0
new file mode 100644
index 000000000000..6b44924d5c0e
@@ -0,0 +1,40 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+ACRN Hypervisor Introduction
+============================
+
+The ACRN Hypervisor is a Type 1 hypervisor, running directly on the bare-metal
+hardware. It has a privileged management VM, called Service VM, to manage User
+VMs and do I/O emulation.
+
+ACRN userspace is an application running in the Service VM that emulates
+devices for a User VM based on command line configurations. ACRN Hypervisor
+Service Module (HSM) is a kernel module in the Service VM which provides
+hypervisor services to the ACRN userspace.
+
+Below figure shows the architecture.
+
+::
+
+                Service VM                    User VM
+      +----------------------------+  |  +------------------+
+      |        +--------------+    |  |  |                  |
+      |        |ACRN userspace|    |  |  |                  |
+      |        +--------------+    |  |  |                  |
+      |-----------------ioctl------|  |  |                  |   ...
+      |kernel space   +----------+ |  |  |                  |
+      |               |   HSM    | |  |  | Drivers          |
+      |               +----------+ |  |  |                  |
+      +--------------------|-------+  |  +------------------+
+  +---------------------hypercall----------------------------------------+
+  |                         ACRN Hypervisor                              |
+  +----------------------------------------------------------------------+
+  |                          Hardware                                    |
+  +----------------------------------------------------------------------+
+
+ACRN userspace allocates memory for the User VM, configures and initializes the
+devices used by the User VM, loads the virtual bootloader, initializes the
+virtual CPU state and handles I/O request accesses from the User VM. It uses
+ioctls to communicate with the HSM. HSM implements hypervisor services by
+interacting with the ACRN Hypervisor via hypercalls. HSM exports a char device
+interface (/dev/acrn_hsm) to userspace.
Documentation/virt/acrn/io-request.rst new file +97 -0

97 lines not on this page · raw message

Documentation/virt/index.rst +1 -0
index de1ab81df958..c10b519507f5 100644
@@ -11,6 +11,7 @@ Linux Virtualization Support
    uml/user_mode_linux
    paravirt_ops
    guest-halt-polling
+   acrn/index
 
 .. only:: html and subproject
 
MAINTAINERS +7 -0
index deaafb617361..e0fea5e464b4 100644
@@ -436,6 +436,13 @@ S:	Orphan
 F:	drivers/platform/x86/wmi.c
 F:	include/uapi/linux/wmi.h
 
+ACRN HYPERVISOR SERVICE MODULE
+M:	Shuo Liu <[email protected]>
+L:	[email protected]
+S:	Supported
+W:	https://projectacrn.org
+F:	Documentation/virt/acrn/
+
 AD1889 ALSA SOUND DRIVER
 L:	[email protected]
 S:	Maintained
Signature · 1 line
2.28.0
#2 lore raw

PATCH v5 2/17 x86/acrn: Introduce acrn_{setup, remove}_intr_handler()

In reply to [email protected]

From: Shuo Liu <[email protected]>

The ACRN Hypervisor builds an I/O request when a trapped I/O access happens in User VM. Then, ACRN Hypervisor issues an upcall by sending a notification interrupt to the Service VM. HSM in the Service VM needs to hook the notification interrupt to handle I/O requests.

Notification interrupts from ACRN Hypervisor are already supported and a, currently uninitialized, callback called.

Export two APIs for HSM to setup/remove its callback.

Cc Dave Hansen <[email protected]>
Cc Sean Christopherson <[email protected]>
Cc Dan Williams <[email protected]>
Cc Fengwei Yin <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 arch/x86/include/asm/acrn.h |  8 ++++++++
 arch/x86/kernel/cpu/acrn.c  | 14 ++++++++++++++
 2 files changed, 22 insertions(+)
 create mode 100644 arch/x86/include/asm/acrn.h
arch/x86/include/asm/acrn.h new file +8 -0
new file mode 100644
index 000000000000..ff259b69cde7
@@ -0,0 +1,8 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_ACRN_H
+#define _ASM_X86_ACRN_H
+
+void acrn_setup_intr_handler(void (*handler)(void));
+void acrn_remove_intr_handler(void);
+
+#endif /* _ASM_X86_ACRN_H */
arch/x86/kernel/cpu/acrn.c +14 -0
index 0b2c03943ac6..e0c181781905 100644
@@ -10,6 +10,8 @@
  */
 
 #include <linux/interrupt.h>
+
+#include <asm/acrn.h>
 #include <asm/apic.h>
 #include <asm/cpufeatures.h>
 #include <asm/desc.h>
@@ -55,6 +57,18 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_acrn_hv_callback)
 	set_irq_regs(old_regs);
 }
 
+void acrn_setup_intr_handler(void (*handler)(void))
+{
+	acrn_intr_handler = handler;
+}
+EXPORT_SYMBOL_GPL(acrn_setup_intr_handler);
+
+void acrn_remove_intr_handler(void)
+{
+	acrn_intr_handler = NULL;
+}
+EXPORT_SYMBOL_GPL(acrn_remove_intr_handler);
+
 const __initconst struct hypervisor_x86 x86_hyper_acrn = {
 	.name                   = "ACRN",
 	.detect                 = acrn_detect,
Signature · 1 line
2.28.0
#3 lore raw

PATCH v5 3/17 x86/acrn: Introduce an API to check if a VM is privileged

In reply to [email protected]

From: Yin Fengwei <[email protected]>

ACRN Hypervisor reports hypervisor features via CPUID leaf 0x40000001 which is similar to KVM. A VM can check if it's the privileged VM using the feature bits. The Service VM is the only privileged VM by design.

Cc Dave Hansen <[email protected]>
Cc Sean Christopherson <[email protected]>
Cc Dan Williams <[email protected]>
Cc Fengwei Yin <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 arch/x86/include/asm/acrn.h |  9 +++++++++
 arch/x86/kernel/cpu/acrn.c  | 19 ++++++++++++++++++-
 2 files changed, 27 insertions(+), 1 deletion(-)
arch/x86/include/asm/acrn.h +9 -0
index ff259b69cde7..a2d4aea3a80d 100644
@@ -2,7 +2,16 @@
 #ifndef _ASM_X86_ACRN_H
 #define _ASM_X86_ACRN_H
 
+/*
+ * This CPUID returns feature bitmaps in EAX.
+ * Guest VM uses this to detect the appropriate feature bit.
+ */
+#define	ACRN_CPUID_FEATURES		0x40000001
+/* Bit 0 indicates whether guest VM is privileged */
+#define	ACRN_FEATURE_PRIVILEGED_VM	BIT(0)
+
 void acrn_setup_intr_handler(void (*handler)(void));
 void acrn_remove_intr_handler(void);
+bool acrn_is_privileged_vm(void);
 
 #endif /* _ASM_X86_ACRN_H */
arch/x86/kernel/cpu/acrn.c +18 -1
index e0c181781905..5528bfc913ea 100644
@@ -19,9 +19,26 @@
 #include <asm/idtentry.h>
 #include <asm/irq_regs.h>
 
+static u32 acrn_cpuid_base(void)
+{
+	static u32 acrn_cpuid_base;
+
+	if (!acrn_cpuid_base && boot_cpu_has(X86_FEATURE_HYPERVISOR))
+		acrn_cpuid_base = hypervisor_cpuid_base("ACRNACRNACRN", 0);
+
+	return acrn_cpuid_base;
+}
+
+bool acrn_is_privileged_vm(void)
+{
+	return cpuid_eax(acrn_cpuid_base() | ACRN_CPUID_FEATURES) &
+			 ACRN_FEATURE_PRIVILEGED_VM;
+}
+EXPORT_SYMBOL_GPL(acrn_is_privileged_vm);
+
 static u32 __init acrn_detect(void)
 {
-	return hypervisor_cpuid_base("ACRNACRNACRN", 0);
+	return acrn_cpuid_base();
 }
 
 static void __init acrn_init_platform(void)
Signature · 1 line
2.28.0
maintainer
#4 lore raw

Re: [PATCH v5 03/17] x86/acrn: Introduce an API to check if a VM is privileged

arch/x86/kernel/cpu/acrn.c line 35

Quoted 4 lines · On Mon, Oct 19, 2020 at 02:17:49PM +0800, [email protected] wrote:
arch/x86/kernel/cpu/acrn.c +4 -0
+bool acrn_is_privileged_vm(void)
+{
+	return cpuid_eax(acrn_cpuid_base() | ACRN_CPUID_FEATURES) &
+			 ACRN_FEATURE_PRIVILEGED_VM;

I asked in the previous review why that acrn_cpuid_base() is used here, you said that the base might vary. Looking at hypervisor_cpuid_base(), it searches in the range [0x40000000, 0x40010000] with an 0x100 offset.

So you're saying that ACRN_CPUID_FEATURES is the first leaf beyond the base. Close?

If so, why isn't the code doing this?

	return cpuid_eax(acrn_cpuid_base() + 1)...

and why doesn't it have a comment above it explaining that the base can change and it needs to be discovered each time?

Quoted 1 line

+EXPORT_SYMBOL_GPL(acrn_is_privileged_vm);

Also, that acrn_is_privileged_vm() silly helper is used only once and I don't like the exported symbols pollution we're doing. So make that function give you the eax of ACRN_CPUID_FEATURES and callers can do their testing themselves.

When it turns out that code patterns get repeated, you can then aggregate stuff into a helper.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
author
#5 lore raw

Re: [PATCH v5 03/17] x86/acrn: Introduce an API to check if a VM is privileged

Hi Boris,

Quoted 10 lines · Borislav Petkov

arch/x86/kernel/cpu/acrn.c line 35

Quoted 4 lines · On Mon, Oct 19, 2020 at 02:17:49PM +0800, [email protected] wrote:
arch/x86/kernel/cpu/acrn.c +4 -0
+bool acrn_is_privileged_vm(void)
+{
+	return cpuid_eax(acrn_cpuid_base() | ACRN_CPUID_FEATURES) &
+			 ACRN_FEATURE_PRIVILEGED_VM;

I asked in the previous review why that acrn_cpuid_base() is used here, you said that the base might vary. Looking at hypervisor_cpuid_base(), it searches in the range [0x40000000, 0x40010000] with an 0x100 offset.

Quoted 2 lines

So you're saying that ACRN_CPUID_FEATURES is the first leaf beyond the base. Close?

Yes.

Quoted 7 lines

If so, why isn't the code doing this?

	return cpuid_eax(acrn_cpuid_base() + 1)...

and why doesn't it have a comment above it explaining that the base can change and it needs to be discovered each time?

The code just followed KVM style (see kvm_arch_para_features()). I can change to use

	cpuid_eax(acrn_cpuid_base() + 1)...

If you prefer to.

hypervisor_cpuid_base() implies the base is variable, no? We use this function to detect the base.

Quoted 7 lines
Quoted 2 lines

+EXPORT_SYMBOL_GPL(acrn_is_privileged_vm);

Also, that acrn_is_privileged_vm() silly helper is used only once and I don't like the exported symbols pollution we're doing. So make that function give you the eax of ACRN_CPUID_FEATURES and callers can do their testing themselves.

OK. Then i will define acrn_cpuid_base() as a static inline function in asm/acrn.h for callers.

Quoted 3 lines

When it turns out that code patterns get repeated, you can then aggregate stuff into a helper.

Got it. Thanks.

Thanks
shuo

maintainer
#6 lore raw

Re: [PATCH v5 03/17] x86/acrn: Introduce an API to check if a VM is privileged

Quoted 1 line · Shuo A Liu

The code just followed KVM style (see kvm_arch_para_features()).

Do you see Documentation/virt/kvm/cpuid.rst?

Now where is yours explaining what your hypervisor is doing?

Quoted 3 lines

I can change to use

	cpuid_eax(acrn_cpuid_base() + 1)...

If you prefer to.

Yes please.

Quoted 2 lines

hypervisor_cpuid_base() implies the base is variable, no? We use this function to detect the base.

Yes, but you need to document all that and make it clear and understandable. If Linux is supposed to run as an acrn guest, that interface better be documented just like KVM does.

Also, if there's a bug in the KVM guest/host interface, we might be able to fix it modulo ABI. Is that possible with acrn?

I'm guessing the answer to that is yes if I'm looking at

https://github.com/projectacrn/acrn-hypervisor

?

Quoted 2 lines

OK. Then i will define acrn_cpuid_base() as a static inline function in asm/acrn.h for callers.

Yah, that function is simple enough.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
author
#7 lore raw

Re: [PATCH v5 03/17] x86/acrn: Introduce an API to check if a VM is privileged

Quoted 4 lines · Borislav Petkov
Quoted 1 line · Shuo A Liu

The code just followed KVM style (see kvm_arch_para_features()).

Do you see Documentation/virt/kvm/cpuid.rst?

OK. It documents the leaf number.

Quoted 2 lines

Now where is yours explaining what your hypervisor is doing?

Currently, it is in arch/x86/include/asm/acrn.h.

Quoted 6 lines
Quoted 4 lines

I can change to use

	cpuid_eax(acrn_cpuid_base() + 1)...

If you prefer to.

Yes please.

Sure.

If the leaf numbers be documented explicitly (like kvm), i think i can use them as eax of cpuid_eax() directly (back to your first comment).
cpuid_eax(ACRN_CPUID_FEATURES)...

If you looking at implementation of acrn-hypervisor, you will found the leaf number is hardcoded in the hypervisor. So, they also can be documented explicitly.

Quoted 7 lines
Quoted 3 lines

hypervisor_cpuid_base() implies the base is variable, no? We use this function to detect the base.

Yes, but you need to document all that and make it clear and understandable. If Linux is supposed to run as an acrn guest, that interface better be documented just like KVM does.

OK. I can add a similar cpuid.rst for acrn.

Quoted 9 lines

Also, if there's a bug in the KVM guest/host interface, we might be able to fix it modulo ABI. Is that possible with acrn?

I'm guessing the answer to that is yes if I'm looking at

https://github.com/projectacrn/acrn-hypervisor

?

Yes. Fix patches are always welcome.

Quoted 5 lines
Quoted 3 lines

OK. Then i will define acrn_cpuid_base() as a static inline function in asm/acrn.h for callers.

Yah, that function is simple enough.

Thanks
shuo

maintainer
#8 lore raw

Re: [PATCH v5 03/17] x86/acrn: Introduce an API to check if a VM is privileged

Quoted 7 lines · Shuo A Liu
Quoted 4 lines · Borislav Petkov
Quoted 1 line · Shuo A Liu

The code just followed KVM style (see kvm_arch_para_features()).

Do you see Documentation/virt/kvm/cpuid.rst?

OK. It documents the leaf number.

It also says

"Note also that old hosts set eax value to 0x0. This should be interpreted as if the value was 0x40000001."

Does this hold true for the acrn HV? The fact that I'm asking about all those things should give you a hint that documenting the API is important.

Quoted 3 lines
Quoted 1 line

Now where is yours explaining what your hypervisor is doing?

Currently, it is in arch/x86/include/asm/acrn.h.

Yeah, you can't expect people to go scrape it from headers though - it should be concentrated in a doc file.

Quoted 2 lines

If the leaf numbers be documented explicitly (like kvm), i think i can use them as eax of cpuid_eax() directly (back to your first comment).

Which means, you don't need to do the logical OR-ing which kvm does because of what I pasted above about eax being 0 on old hosts. Now we're getting somewhere...

Quoted 5 lines

cpuid_eax(ACRN_CPUID_FEATURES)...

If you looking at implementation of acrn-hypervisor, you will found the leaf number is hardcoded in the hypervisor. So, they also can be documented explicitly.

Ok.

Quoted 1 line

OK. I can add a similar cpuid.rst for acrn.

Yes please.

Quoted 1 line

Yes. Fix patches are always welcome.

Ok, good, the thing is open. You could put that in the doc too, along with the link so that people can find it.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
author
#9 lore raw

Re: [PATCH v5 03/17] x86/acrn: Introduce an API to check if a VM is privileged

Quoted 5 lines · Borislav Petkov
Quoted 4 lines · Shuo A Liu
Quoted 3 lines · Borislav Petkov
Quoted 1 line · Shuo A Liu

The code just followed KVM style (see kvm_arch_para_features()).

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Do you see Documentation/virt/kvm/cpuid.rst?

OK. It documents the leaf number.

It also says

"Note also that old hosts set eax value to 0x0. This should be interpreted as if the value was 0x40000001."

Ok.

Quoted 4 lines

Does this hold true for the acrn HV? The fact that I'm asking about all those things should give you a hint that documenting the API is important.

No, acrn HV is always return eax with maximum leaf number.

Let me add the document.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Now where is yours explaining what your hypervisor is doing?

Currently, it is in arch/x86/include/asm/acrn.h.

Yeah, you can't expect people to go scrape it from headers though - it should be concentrated in a doc file.

Quoted 6 lines
Quoted 2 lines

If the leaf numbers be documented explicitly (like kvm), i think i can use them as eax of cpuid_eax() directly (back to your first comment).

Which means, you don't need to do the logical OR-ing which kvm does because of what I pasted above about eax being 0 on old hosts. Now we're getting somewhere...

Yes.

Quoted 9 lines
Quoted 6 lines

cpuid_eax(ACRN_CPUID_FEATURES)...

If you looking at implementation of acrn-hypervisor, you will found the leaf number is hardcoded in the hypervisor. So, they also can be documented explicitly.

Ok.

Quoted 8 lines
Quoted 1 line

OK. I can add a similar cpuid.rst for acrn.

Yes please.

Quoted 1 line

Yes. Fix patches are always welcome.

Ok, good, the thing is open. You could put that in the doc too, along with the link so that people can find it.

OK.

Thanks
shuo

#10 lore raw

PATCH v5 4/17 x86/acrn: Introduce hypercall interfaces

In reply to [email protected]

From: Shuo Liu <[email protected]>

The Service VM communicates with the hypervisor via conventional hypercalls. VMCALL instruction is used to make the hypercalls.

ACRN hypercall ABI:
* Hypercall number is in R8 register.
* Up to 2 parameters are in RDI and RSI registers.
* Return value is in RAX register.

Introduce the ACRN hypercall interfaces. Because GCC doesn't support R8 register as direct register constraints, use supported constraint as input with a explicit MOV to R8 in beginning of asm.

Cc Dave Hansen <[email protected]>
Cc Sean Christopherson <[email protected]>
Cc Dan Williams <[email protected]>
Cc Fengwei Yin <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Arvind Sankar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Nick Desaulniers <[email protected]>
Cc Segher Boessenkool <[email protected]>
---
 arch/x86/include/asm/acrn.h | 54 +++++++++++++++++++++++++++++++++++++
 1 file changed, 54 insertions(+)
arch/x86/include/asm/acrn.h +54 -0
index a2d4aea3a80d..03e420245505 100644
@@ -14,4 +14,58 @@ void acrn_setup_intr_handler(void (*handler)(void));
 void acrn_remove_intr_handler(void);
 bool acrn_is_privileged_vm(void);
 
+/*
+ * Hypercalls for ACRN
+ *
+ * - VMCALL instruction is used to implement ACRN hypercalls.
+ * - ACRN hypercall ABI:
+ *   - Hypercall number is passed in R8 register.
+ *   - Up to 2 arguments are passed in RDI, RSI.
+ *   - Return value will be placed in RAX.
+ *
+ * Because GCC doesn't support R8 register as direct register constraints, use
+ * supported constraint as input with a explicit MOV to R8 in beginning of asm.
+ */
+static inline long acrn_hypercall0(unsigned long hcall_id)
+{
+	long result;
+
+	asm volatile("movl %1, %%r8d\n\t"
+		     "vmcall\n\t"
+		     : "=a" (result)
+		     : "ir" (hcall_id)
+		     : "r8", "memory");
+
+	return result;
+}
+
+static inline long acrn_hypercall1(unsigned long hcall_id,
+				   unsigned long param1)
+{
+	long result;
+
+	asm volatile("movl %1, %%r8d\n\t"
+		     "vmcall\n\t"
+		     : "=a" (result)
+		     : "ir" (hcall_id), "D" (param1)
+		     : "r8", "memory");
+
+	return result;
+}
+
+static inline long acrn_hypercall2(unsigned long hcall_id,
+				   unsigned long param1,
+				   unsigned long param2)
+{
+	long result;
+
+	asm volatile("movl %1, %%r8d\n\t"
+		     "vmcall\n\t"
+		     : "=a" (result)
+		     : "ir" (hcall_id), "D" (param1), "S" (param2)
+		     : "r8", "memory");
+
+	return result;
+}
+
 #endif /* _ASM_X86_ACRN_H */
Signature · 1 line
2.28.0
Reviewed-by on v5 4/17
#11 lore raw

Re: [PATCH v5 04/17] x86/acrn: Introduce hypercall interfaces

Quoted 6 lines · On Sun, Oct 18, 2020 at 11:18 PM <[email protected]> wrote:

From: Shuo Liu <[email protected]>

The Service VM communicates with the hypervisor via conventional hypercalls. VMCALL instruction is used to make the hypercalls.

Quoted 8 lines

ACRN hypercall ABI:
* Hypercall number is in R8 register.
* Up to 2 parameters are in RDI and RSI registers.
* Return value is in RAX register.

Introduce the ACRN hypercall interfaces. Because GCC doesn't support R8 register as direct register constraints, use supported constraint as input with a explicit MOV to R8 in beginning of asm.

Quoted 88 lines
Cc Dave Hansen <[email protected]>
Cc Sean Christopherson <[email protected]>
Cc Dan Williams <[email protected]>
Cc Fengwei Yin <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Arvind Sankar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Nick Desaulniers <[email protected]>
Cc Segher Boessenkool <[email protected]>
---
 arch/x86/include/asm/acrn.h | 54 +++++++++++++++++++++++++++++++++++++
 1 file changed, 54 insertions(+)
arch/x86/include/asm/acrn.h +54 -1
index a2d4aea3a80d..03e420245505 100644
@@ -14,4 +14,58 @@ void acrn_setup_intr_handler(void (*handler)(void));
 void acrn_remove_intr_handler(void);
 bool acrn_is_privileged_vm(void);
 
+/*
+ * Hypercalls for ACRN
+ *
+ * - VMCALL instruction is used to implement ACRN hypercalls.
+ * - ACRN hypercall ABI:
+ *   - Hypercall number is passed in R8 register.
+ *   - Up to 2 arguments are passed in RDI, RSI.
+ *   - Return value will be placed in RAX.
+ *
+ * Because GCC doesn't support R8 register as direct register constraints, use
+ * supported constraint as input with a explicit MOV to R8 in beginning of asm.
+ */
+static inline long acrn_hypercall0(unsigned long hcall_id)
+{
+       long result;
+
+       asm volatile("movl %1, %%r8d\n\t"
+                    "vmcall\n\t"
+                    : "=a" (result)
+                    : "ir" (hcall_id)
+                    : "r8", "memory");
+
+       return result;
+}
+
+static inline long acrn_hypercall1(unsigned long hcall_id,
+                                  unsigned long param1)
+{
+       long result;
+
+       asm volatile("movl %1, %%r8d\n\t"
+                    "vmcall\n\t"
+                    : "=a" (result)
+                    : "ir" (hcall_id), "D" (param1)
+                    : "r8", "memory");
+
+       return result;
+}
+
+static inline long acrn_hypercall2(unsigned long hcall_id,
+                                  unsigned long param1,
+                                  unsigned long param2)
+{
+       long result;
+
+       asm volatile("movl %1, %%r8d\n\t"
+                    "vmcall\n\t"
+                    : "=a" (result)
+                    : "ir" (hcall_id), "D" (param1), "S" (param2)
+                    : "r8", "memory");
+
+       return result;
+}
+
 #endif /* _ASM_X86_ACRN_H */
--

2.28.0

Signature · 2 lines
Thanks,
~Nick Desaulniers
#12 lore raw

Re: [PATCH v5 04/17] x86/acrn: Introduce hypercall interfaces

In reply to [email protected]
Quoted 14 lines · On Mon, Oct 19, 2020 at 02:17:50PM +0800, [email protected] wrote:

From: Shuo Liu <[email protected]>

The Service VM communicates with the hypervisor via conventional hypercalls. VMCALL instruction is used to make the hypercalls.

ACRN hypercall ABI:
* Hypercall number is in R8 register.
* Up to 2 parameters are in RDI and RSI registers.
* Return value is in RAX register.

Introduce the ACRN hypercall interfaces. Because GCC doesn't support R8 register as direct register constraints, use supported constraint as input with a explicit MOV to R8 in beginning of asm.

arch/x86/include/asm/acrn.h void acrn_setup_intr_handler(void (*handler)(void)); · line 36

Quoted 8 lines
arch/x86/include/asm/acrn.h +8 -0
+static inline long acrn_hypercall0(unsigned long hcall_id)
+{
+	long result;
+
+	asm volatile("movl %1, %%r8d\n\t"
+		     "vmcall\n\t"
+		     : "=a" (result)
+		     : "ir" (hcall_id)

Is the hypercall id an unsigned long (64 bits) or an unsigned int (32 bits)? This will generate broken assembly if the "r" option is chosen, eg something like

	movl %rdi, %r8d
Quoted 38 lines
arch/x86/include/asm/acrn.h +34 -0
+		     : "r8", "memory");
+
+	return result;
+}
+
+static inline long acrn_hypercall1(unsigned long hcall_id,
+				   unsigned long param1)
+{
+	long result;
+
+	asm volatile("movl %1, %%r8d\n\t"
+		     "vmcall\n\t"
+		     : "=a" (result)
+		     : "ir" (hcall_id), "D" (param1)
+		     : "r8", "memory");
+
+	return result;
+}
+
+static inline long acrn_hypercall2(unsigned long hcall_id,
+				   unsigned long param1,
+				   unsigned long param2)
+{
+	long result;
+
+	asm volatile("movl %1, %%r8d\n\t"
+		     "vmcall\n\t"
+		     : "=a" (result)
+		     : "ir" (hcall_id), "D" (param1), "S" (param2)
+		     : "r8", "memory");
+
+	return result;
+}
+
 #endif /* _ASM_X86_ACRN_H */
Signature · 1 line
2.28.0
author
#13 lore raw

Re: [PATCH v5 04/17] x86/acrn: Introduce hypercall interfaces

Quoted 27 lines · Arvind Sankar
Quoted 14 lines · On Mon, Oct 19, 2020 at 02:17:50PM +0800, [email protected] wrote:

From: Shuo Liu <[email protected]>

The Service VM communicates with the hypervisor via conventional hypercalls. VMCALL instruction is used to make the hypercalls.

ACRN hypercall ABI:
* Hypercall number is in R8 register.
* Up to 2 parameters are in RDI and RSI registers.
* Return value is in RAX register.

Introduce the ACRN hypercall interfaces. Because GCC doesn't support R8 register as direct register constraints, use supported constraint as input with a explicit MOV to R8 in beginning of asm.

arch/x86/include/asm/acrn.h void acrn_setup_intr_handler(void (*handler)(void)); · line 36

Quoted 8 lines
arch/x86/include/asm/acrn.h +8 -0
+static inline long acrn_hypercall0(unsigned long hcall_id)
+{
+	long result;
+
+	asm volatile("movl %1, %%r8d\n\t"
+		     "vmcall\n\t"
+		     : "=a" (result)
+		     : "ir" (hcall_id)

Is the hypercall id an unsigned long (64 bits) or an unsigned int (32 bits)? This will generate broken assembly if the "r" option is chosen, eg something like

Quoted 1 line
	movl %rdi, %r8d

Yes, it can be an unsigned long. So do MOV explicitly.

	asm volatile("movq %1, %%r8\n\t"
		     "vmcall\n\t"
		     : "=a" (result)
		     : "ir" (hcall_id)

Thanks

#14 lore raw

Re: [PATCH v5 04/17] x86/acrn: Introduce hypercall interfaces

Quoted 30 lines · Shuo A Liu
Quoted 27 lines · Arvind Sankar
Quoted 14 lines · On Mon, Oct 19, 2020 at 02:17:50PM +0800, [email protected] wrote:

From: Shuo Liu <[email protected]>

The Service VM communicates with the hypervisor via conventional hypercalls. VMCALL instruction is used to make the hypercalls.

ACRN hypercall ABI:
* Hypercall number is in R8 register.
* Up to 2 parameters are in RDI and RSI registers.
* Return value is in RAX register.

Introduce the ACRN hypercall interfaces. Because GCC doesn't support R8 register as direct register constraints, use supported constraint as input with a explicit MOV to R8 in beginning of asm.

arch/x86/include/asm/acrn.h void acrn_setup_intr_handler(void (*handler)(void)); · line 36

Quoted 8 lines
arch/x86/include/asm/acrn.h +8 -0
+static inline long acrn_hypercall0(unsigned long hcall_id)
+{
+	long result;
+
+	asm volatile("movl %1, %%r8d\n\t"
+		     "vmcall\n\t"
+		     : "=a" (result)
+		     : "ir" (hcall_id)

Is the hypercall id an unsigned long (64 bits) or an unsigned int (32 bits)? This will generate broken assembly if the "r" option is chosen, eg something like

Quoted 1 line
	movl %rdi, %r8d
Quoted 8 lines

Yes, it can be an unsigned long. So do MOV explicitly.

	asm volatile("movq %1, %%r8\n\t"
		     "vmcall\n\t"
		     : "=a" (result)
		     : "ir" (hcall_id)

Thanks

All the hypercall ID's defined seem to be only 32 bits though?

author
#15 lore raw

Re: [PATCH v5 04/17] x86/acrn: Introduce hypercall interfaces

Quoted 33 lines · Arvind Sankar
Quoted 32 lines · Shuo A Liu
Quoted 27 lines · Arvind Sankar
Quoted 14 lines · On Mon, Oct 19, 2020 at 02:17:50PM +0800, [email protected] wrote:

From: Shuo Liu <[email protected]>

The Service VM communicates with the hypervisor via conventional hypercalls. VMCALL instruction is used to make the hypercalls.

ACRN hypercall ABI:
* Hypercall number is in R8 register.
* Up to 2 parameters are in RDI and RSI registers.
* Return value is in RAX register.

Introduce the ACRN hypercall interfaces. Because GCC doesn't support R8 register as direct register constraints, use supported constraint as input with a explicit MOV to R8 in beginning of asm.

arch/x86/include/asm/acrn.h void acrn_setup_intr_handler(void (*handler)(void)); · line 36

Quoted 8 lines
arch/x86/include/asm/acrn.h +8 -0
+static inline long acrn_hypercall0(unsigned long hcall_id)
+{
+	long result;
+
+	asm volatile("movl %1, %%r8d\n\t"
+		     "vmcall\n\t"
+		     : "=a" (result)
+		     : "ir" (hcall_id)

Is the hypercall id an unsigned long (64 bits) or an unsigned int (32 bits)? This will generate broken assembly if the "r" option is chosen, eg something like

Quoted 1 line
	movl %rdi, %r8d

Yes, it can be an unsigned long. So do MOV explicitly.

Quoted 8 lines
Quoted 6 lines
	asm volatile("movq %1, %%r8\n\t"
		     "vmcall\n\t"
		     : "=a" (result)
		     : "ir" (hcall_id)

Thanks

All the hypercall ID's defined seem to be only 32 bits though?

Yes, they are.
The paramter is unsigned long, use movq to align it.

#16 lore raw

Re: [PATCH v5 04/17] x86/acrn: Introduce hypercall interfaces

Quoted 39 lines · Shuo A Liu
Quoted 38 lines · Arvind Sankar
Quoted 37 lines · Shuo A Liu
Quoted 27 lines · Arvind Sankar
Quoted 14 lines · On Mon, Oct 19, 2020 at 02:17:50PM +0800, [email protected] wrote:

From: Shuo Liu <[email protected]>

The Service VM communicates with the hypervisor via conventional hypercalls. VMCALL instruction is used to make the hypercalls.

ACRN hypercall ABI:
* Hypercall number is in R8 register.
* Up to 2 parameters are in RDI and RSI registers.
* Return value is in RAX register.

Introduce the ACRN hypercall interfaces. Because GCC doesn't support R8 register as direct register constraints, use supported constraint as input with a explicit MOV to R8 in beginning of asm.

arch/x86/include/asm/acrn.h void acrn_setup_intr_handler(void (*handler)(void)); · line 36

Quoted 8 lines
arch/x86/include/asm/acrn.h +8 -0
+static inline long acrn_hypercall0(unsigned long hcall_id)
+{
+	long result;
+
+	asm volatile("movl %1, %%r8d\n\t"
+		     "vmcall\n\t"
+		     : "=a" (result)
+		     : "ir" (hcall_id)

Is the hypercall id an unsigned long (64 bits) or an unsigned int (32 bits)? This will generate broken assembly if the "r" option is chosen, eg something like

Quoted 1 line
	movl %rdi, %r8d

Yes, it can be an unsigned long. So do MOV explicitly.

	asm volatile("movq %1, %%r8\n\t"
		     "vmcall\n\t"
		     : "=a" (result)
		     : "ir" (hcall_id)
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Thanks

All the hypercall ID's defined seem to be only 32 bits though?

Yes, they are.
The paramter is unsigned long, use movq to align it.

I don't understand what you mean by alignment here, but I was asking why hcall_id is unsigned long and not unsigned int (or u32) if you only need 32 bits?

author
#17 lore raw

Re: [PATCH v5 04/17] x86/acrn: Introduce hypercall interfaces

Quoted 42 lines · Arvind Sankar
Quoted 41 lines · Shuo A Liu
Quoted 40 lines · Arvind Sankar
Quoted 38 lines · Shuo A Liu
Quoted 27 lines · Arvind Sankar
Quoted 14 lines · On Mon, Oct 19, 2020 at 02:17:50PM +0800, [email protected] wrote:

From: Shuo Liu <[email protected]>

The Service VM communicates with the hypervisor via conventional hypercalls. VMCALL instruction is used to make the hypercalls.

ACRN hypercall ABI:
* Hypercall number is in R8 register.
* Up to 2 parameters are in RDI and RSI registers.
* Return value is in RAX register.

Introduce the ACRN hypercall interfaces. Because GCC doesn't support R8 register as direct register constraints, use supported constraint as input with a explicit MOV to R8 in beginning of asm.

arch/x86/include/asm/acrn.h void acrn_setup_intr_handler(void (*handler)(void)); · line 36

Quoted 8 lines
arch/x86/include/asm/acrn.h +8 -0
+static inline long acrn_hypercall0(unsigned long hcall_id)
+{
+	long result;
+
+	asm volatile("movl %1, %%r8d\n\t"
+		     "vmcall\n\t"
+		     : "=a" (result)
+		     : "ir" (hcall_id)

Is the hypercall id an unsigned long (64 bits) or an unsigned int (32 bits)? This will generate broken assembly if the "r" option is chosen, eg something like

Quoted 1 line
	movl %rdi, %r8d

Yes, it can be an unsigned long. So do MOV explicitly.

	asm volatile("movq %1, %%r8\n\t"
		     "vmcall\n\t"
		     : "=a" (result)
		     : "ir" (hcall_id)

Thanks

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

All the hypercall ID's defined seem to be only 32 bits though?

Yes, they are.
The paramter is unsigned long, use movq to align it.

I don't understand what you mean by alignment here, but I was asking why hcall_id is unsigned long and not unsigned int (or u32) if you only need 32 bits?

The hypervisor is using R8 as the input of hcall_id. So i just want to keep it consistent.

maintainer
#18 lore raw

Re: [PATCH v5 04/17] x86/acrn: Introduce hypercall interfaces

In reply to [email protected]

arch/x86/include/asm/acrn.h void acrn_setup_intr_handler(void (*handler)(void)); · line 36

Quoted 8 lines · On Mon, Oct 19, 2020 at 02:17:50PM +0800, [email protected] wrote:
arch/x86/include/asm/acrn.h +8 -0
+static inline long acrn_hypercall0(unsigned long hcall_id)
+{
+	long result;
+
+	asm volatile("movl %1, %%r8d\n\t"
+		     "vmcall\n\t"
+		     : "=a" (result)
+		     : "ir" (hcall_id)
			^^

Not "irm"? Or simply "g" in that case?

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#19 lore raw

Re: [PATCH v5 04/17] x86/acrn: Introduce hypercall interfaces

Quoted 11 lines · Borislav Petkov

arch/x86/include/asm/acrn.h void acrn_setup_intr_handler(void (*handler)(void)); · line 36

Quoted 8 lines · On Mon, Oct 19, 2020 at 02:17:50PM +0800, [email protected] wrote:
arch/x86/include/asm/acrn.h +8 -0
+static inline long acrn_hypercall0(unsigned long hcall_id)
+{
+	long result;
+
+	asm volatile("movl %1, %%r8d\n\t"
+		     "vmcall\n\t"
+		     : "=a" (result)
+		     : "ir" (hcall_id)
			^^
Quoted 1 line

Not "irm"? Or simply "g" in that case?

I think that will work for x86_64. But it won't matter much, most of the time you give an immediate number. It is a tiny bit neater of course (if anyone still remembers what "g" is, you cannot use it much these days).

Segher

maintainer
#20 lore raw

Re: [PATCH v5 04/17] x86/acrn: Introduce hypercall interfaces

Quoted 2 lines · Segher Boessenkool

I think that will work for x86_64. But it won't matter much, most of the time you give an immediate number.

Yeah, my question is more along the lines of "is this constraint somehow special, i.e., 'ir' (and not 'm') or can it be whatever, i.e., 'g'"?

Quoted 2 lines

It is a tiny bit neater of course (if anyone still remembers what "g" is, you cannot use it much these days).

Oh I always remember what it is because it is right there in the docs:

"6.47.3.1 Simple Constraints

...

‘g’ Any register, memory or immediate integer operand is allowed,
except for registers that are not general registers."

;-)

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#21 lore raw

Re: [PATCH v5 04/17] x86/acrn: Introduce hypercall interfaces

Hi!

Quoted 6 lines · Borislav Petkov
Quoted 2 lines · Segher Boessenkool

I think that will work for x86_64. But it won't matter much, most of the time you give an immediate number.

Yeah, my question is more along the lines of "is this constraint somehow special, i.e., 'ir' (and not 'm') or can it be whatever, i.e., 'g'"?

movl works for moving anything g->r. This is not true for most insns, and not for most architectures at all (usually loading from memory has a separate mnemonic; moving an immediate number often as well (and it does not usually allow *every* immediate anyway, not even all numbers!)

Quoted 11 lines
Quoted 2 lines

It is a tiny bit neater of course (if anyone still remembers what "g" is, you cannot use it much these days).

Oh I always remember what it is because it is right there in the docs:

"6.47.3.1 Simple Constraints

...

‘g’ Any register, memory or immediate integer operand is allowed,
except for registers that are not general registers."

Most people do not read the documentation, they just copy from (bad) examples (and arguably, any example you do not really understand is a bad example).

(It does not allow *all* memory and *all* constants, btw... And the condition for registers is not really "general register", whatever that means... I hope no one ever told you our documentation does not have white lies!)

Segher

maintainer
#22 lore raw

Re: [PATCH v5 04/17] x86/acrn: Introduce hypercall interfaces

Quoted 1 line · Segher Boessenkool

movl works for moving anything g->r.

Right.

Quoted 4 lines

This is not true for most insns, and not for most architectures at all (usually loading from memory has a separate mnemonic; moving an immediate number often as well (and it does not usually allow *every* immediate anyway, not even all numbers!)

So I've heard from other architectures. Every arch has its own problems. :-)

Quoted 3 lines

Most people do not read the documentation, they just copy from (bad) examples (and arguably, any example you do not really understand is a bad example).

Unfortunately.

Quoted 3 lines

(It does not allow *all* memory and *all* constants, btw... And the condition for registers is not really "general register", whatever that means...

I think that means general purpose registers. I.e., it won't use, say FPU, XMM or whatever special regs.

What does the asm() parsing code in gcc do for "g"? There should be some logic which constraints what register is valid...

Quoted 2 lines

I hope no one ever told you our documentation does not have white lies!)

I have convinced myself of this a couple of times already so I either go ask our gcc friends or go look straight at gcc source. It is useful to know folks which hack on it so that I can ask them stupid questions and not go off into the weeds, trying to figure out what the documentation says.

But hey, if that gets the documentation improved, that's a win-win situation right there. Might even make people copying from bad examples to go look at the docs first...

:-)

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#23 lore raw

Re: [PATCH v5 04/17] x86/acrn: Introduce hypercall interfaces

Quoted 10 lines · Borislav Petkov
Quoted 3 lines · Segher Boessenkool

(It does not allow *all* memory and *all* constants, btw... And the condition for registers is not really "general register", whatever that means...

I think that means general purpose registers. I.e., it won't use, say FPU, XMM or whatever special regs.

What does the asm() parsing code in gcc do for "g"? There should be some logic which constraints what register is valid...

It just asks the general_operand function, which (for registers) accepts the hard registers that are accessible. This does include the float and vector etc. registers, normally.

But you usually have a pseudo-register there (which is always allowed here), unless you used some register asm variable. And pseudos usually are allocated a simple integer register during register allocation, in an asm that is.

Quoted 11 lines
Quoted 2 lines

I hope no one ever told you our documentation does not have white lies!)

I have convinced myself of this a couple of times already so I either go ask our gcc friends or go look straight at gcc source. It is useful to know folks which hack on it so that I can ask them stupid questions and not go off into the weeds, trying to figure out what the documentation says.

But hey, if that gets the documentation improved, that's a win-win situation right there.

Yes :-)

Quoted 2 lines

Might even make people copying from bad examples
to go look at the docs first...

Optimism is cool :-)

Segher

maintainer
#24 lore raw

Re: [PATCH v5 04/17] x86/acrn: Introduce hypercall interfaces

Quoted 6 lines · Segher Boessenkool

It just asks the general_operand function, which (for registers) accepts the hard registers that are accessible. This does include the float and vector etc. registers, normally.

But you usually have a pseudo-register there (which is always allowed here), unless you used some register asm variable.

You mean like this:

---
int main(void)

{
	register float foo asm ("xmm0") = 0.99f;

        asm volatile("movl %0, %%r8d\n\t"
                      "vmcall\n\t"
                      :: "g" (foo));

        return 0;
}

---

That works ok AFAICT:

---

0000000000000000 <main>:

   0:   55                      push   %rbp
   1:   48 89 e5                mov    %rsp,%rbp
   4:   f3 0f 10 05 00 00 00    movss  0x0(%rip),%xmm0        # c <main+0xc>
   b:   00 
   c:   66 0f 7e c0             movd   %xmm0,%eax
  10:   41 89 c0                mov    %eax,%r8d
  13:   0f 01 c1                vmcall 
  16:   b8 00 00 00 00          mov    $0x0,%eax
  1b:   5d                      pop    %rbp
  1c:   c3                      retq

---

gcc smartypants shuffles it through a GPR before sticking it into %r8.

It works too If I use a float immediate:

---
int main(void)

{
        asm volatile("movl %0, %%r8d\n\t"
                      "vmcall\n\t"
                      :: "g" (0.99f));
                      
        return 0;     
}

---

---
0000000000000000 <main>:

   0:   55                      push   %rbp
   1:   48 89 e5                mov    %rsp,%rbp
   4:   41 b8 a4 70 7d 3f       mov    $0x3f7d70a4,%r8d
   a:   0f 01 c1                vmcall 
   d:   b8 00 00 00 00          mov    $0x0,%eax
  12:   5d                      pop    %rbp
  13:   c3                      retq

---

or maybe I'm missing some freaky way to specify the input operand so that I can make it take a float register. But even if I could, it would error out when generating the asm, I presume, due to invalid insn or whatnot.

Quoted 2 lines

And pseudos usually are allocated a simple integer register during register allocation, in an asm that is.

Interesting.

Quoted 4 lines
Quoted 2 lines

Might even make people copying from bad examples
to go look at the docs first...

Optimism is cool :-)

In my experience so far, documenting stuff better might not always bring the expected results but sometimes it does move people in the most unexpected way and miracles happen.

:-)))

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#25 lore raw

Re: [PATCH v5 04/17] x86/acrn: Introduce hypercall interfaces

Quoted 39 lines · Borislav Petkov
Quoted 6 lines · Segher Boessenkool

It just asks the general_operand function, which (for registers) accepts the hard registers that are accessible. This does include the float and vector etc. registers, normally.

But you usually have a pseudo-register there (which is always allowed here), unless you used some register asm variable.

You mean like this:

---
int main(void)

{
	register float foo asm ("xmm0") = 0.99f;

        asm volatile("movl %0, %%r8d\n\t"
                      "vmcall\n\t"
                      :: "g" (foo));

        return 0;
}

---

That works ok AFAICT:

---

0000000000000000 <main>:

   0:   55                      push   %rbp
   1:   48 89 e5                mov    %rsp,%rbp
   4:   f3 0f 10 05 00 00 00    movss  0x0(%rip),%xmm0        # c <main+0xc>
   b:   00 
   c:   66 0f 7e c0             movd   %xmm0,%eax
  10:   41 89 c0                mov    %eax,%r8d
  13:   0f 01 c1                vmcall 
  16:   b8 00 00 00 00          mov    $0x0,%eax
  1b:   5d                      pop    %rbp
  1c:   c3                      retq
Quoted 1 line

---

That is invalid actually: local register asm as input to an inline asm should use *that* register!

This is all correct until LRA ("reload"). Not that "movl %xmm0,$eax" works, but at least it screams its head off, as it should. And then LRA puts it in %eax, a different register than asked for.

Quoted 17 lines

gcc smartypants shuffles it through a GPR before sticking it into %r8.

It works too If I use a float immediate:

---
int main(void)

{
        asm volatile("movl %0, %%r8d\n\t"
                      "vmcall\n\t"
                      :: "g" (0.99f));
                      
        return 0;     
}

---

---
0000000000000000 <main>:

Quoted 8 lines
   0:   55                      push   %rbp
   1:   48 89 e5                mov    %rsp,%rbp
   4:   41 b8 a4 70 7d 3f       mov    $0x3f7d70a4,%r8d
   a:   0f 01 c1                vmcall 
   d:   b8 00 00 00 00          mov    $0x0,%eax
  12:   5d                      pop    %rbp
  13:   c3                      retq

---

(this one is correct code)

Quoted 4 lines

or maybe I'm missing some freaky way to specify the input operand so that I can make it take a float register. But even if I could, it would error out when generating the asm, I presume, due to invalid insn or whatnot.

Yes. But GCC doing what you should have said instead of doing what you said, is not good.

Quoted 8 lines
Quoted 2 lines

And pseudos usually are allocated a simple integer register during register allocation, in an asm that is.

Interesting.

Quoted 3 lines
Quoted 2 lines

Might even make people copying from bad examples
to go look at the docs first...

Quoted 7 lines
Quoted 1 line

Optimism is cool :-)

In my experience so far, documenting stuff better might not always bring the expected results but sometimes it does move people in the most unexpected way and miracles happen.

:-)))

Now if only we had time to document what we wrote! We *do* write docs to go with new code (maybe not enough always), but no one spends a lot of time on documenting the existing compiler, or with a larger view than just a single aspect of the compiler. Alas.

Segher

maintainer
#26 lore raw

Re: [PATCH v5 04/17] x86/acrn: Introduce hypercall interfaces

Quoted 5 lines · Segher Boessenkool

That is invalid actually: local register asm as input to an inline asm should use *that* register!

This is all correct until LRA ("reload"). Not that "movl %xmm0,$eax" works, but at least it screams its head off, as it should.

Screams how?

It builds fine without a single peep with -Wall here.

Btw, that's a MOVD - not a MOVL. MOVD can do xmm -> gpr moves. And singlestepping it with gdb does, well, something, which is clearly wrong but nothing complains:

=> 0x555555555131 <main+12>: movd %xmm0,%eax

and %xmm0 has:

(gdb) p $xmm0
$2 = {v4_float = {0.99000001, 0, 0, 0}, v2_double = {5.2627153433055495e-315, 0}, v16_int8 = {-92, 112, 125, 63,

		  ^^^^^^^^^^

so that is correct.

    0 <repeats 12 times>}, v8_int16 = {28836, 16253, 0, 0, 0, 0, 0, 0}, v4_int32 = {1065185444, 0, 0, 0}, v2_int64 = {
    1065185444, 0}, uint128 = 1065185444}

Then:

movd %xmm0,%eax
rax 0x3f7d70a4

and that's the hex representation of the 0.99 float.

and that same value goes into %r8d:

mov %eax,%r8d
r8 0x3f7d70a4

:-)

Quoted 2 lines

Yes. But GCC doing what you should have said instead of doing what you said, is not good.

Oh well, should I open a low prio bug, would that help?

I probably should test with the latest gcc first, though...

Quoted 4 lines

Now if only we had time to document what we wrote! We *do* write docs to go with new code (maybe not enough always), but no one spends a lot of time on documenting the existing compiler, or with a larger view than just a single aspect of the compiler. Alas.

The problem every big project has.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#27 lore raw

Re: [PATCH v5 04/17] x86/acrn: Introduce hypercall interfaces

Quoted 8 lines · Borislav Petkov
Quoted 5 lines · Segher Boessenkool

That is invalid actually: local register asm as input to an inline asm should use *that* register!

This is all correct until LRA ("reload"). Not that "movl %xmm0,$eax" works, but at least it screams its head off, as it should.

Screams how?

$ cat xmm1.s
        movl %xmm0,%eax
$ x86_64-linux-as xmm1.s -o xmm1.o
xmm1.s: Assembler messages:
xmm1.s:1: Error: unsupported instruction `mov'

(This isn't an existing insn IIUC.)

Quoted 8 lines

It builds fine without a single peep with -Wall here.

Btw, that's a MOVD - not a MOVL. MOVD can do xmm -> gpr moves. And singlestepping it with gdb does, well, something, which is clearly wrong but nothing complains:

=> 0x555555555131 <main+12>: movd %xmm0,%eax

Quoted 7 lines

and %xmm0 has:

(gdb) p $xmm0
$2 = {v4_float = {0.99000001, 0, 0, 0}, v2_double = {5.2627153433055495e-315, 0}, v16_int8 = {-92, 112, 125, 63,

		  ^^^^^^^^^^

so that is correct.

The original code had movl. And movl is needed for GPRs.

Quoted 3 lines

and that same value goes into %r8d:

mov %eax,%r8d

Which violates what is required by register asm :-(

Quoted 4 lines
Quoted 2 lines

Yes. But GCC doing what you should have said instead of doing what you said, is not good.

Oh well, should I open a low prio bug, would that help?

Sure, thanks!

Quoted 1 line

I probably should test with the latest gcc first, though...

Yeah... FWIW, I tested with
x86_64-linux-gcc (GCC) 11.0.0 20201015 (experimental) so I doubt current ToT will have it fixed, but who knows.

Thanks,

Segher

maintainer
#28 lore raw

Re: [PATCH v5 04/17] x86/acrn: Introduce hypercall interfaces

Quoted 9 lines · Segher Boessenkool
Quoted 1 line

Oh well, should I open a low prio bug, would that help?

Sure, thanks!

Quoted 1 line

I probably should test with the latest gcc first, though...

Yeah... FWIW, I tested with
x86_64-linux-gcc (GCC) 11.0.0 20201015 (experimental) so I doubt current ToT will have it fixed, but who knows.

Ok, I'll open it and put you on CC and if they ask me to test latest, I will build gcc from git.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#29 lore raw

PATCH v5 5/17 virt: acrn: Introduce ACRN HSM basic driver

In reply to [email protected]

From: Shuo Liu <[email protected]>

ACRN Hypervisor Service Module (HSM) is a kernel module in Service VM which communicates with ACRN userspace through ioctls and talks to ACRN Hypervisor through hypercalls.

Add a basic HSM driver which allows Service VM userspace to communicate with ACRN. The following patches will add more ioctls, guest VM memory mapping caching, I/O request processing, ioeventfd and irqfd into this module. HSM exports a char device interface (/dev/acrn_hsm) to userspace.

Cc Dave Hansen <[email protected]>
Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 MAINTAINERS                  |  1 +
 drivers/virt/Kconfig         |  2 +
 drivers/virt/Makefile        |  1 +
 drivers/virt/acrn/Kconfig    | 14 ++++++
 drivers/virt/acrn/Makefile   |  3 ++
 drivers/virt/acrn/acrn_drv.h | 18 ++++++++
 drivers/virt/acrn/hsm.c      | 87 ++++++++++++++++++++++++++++++++++++
 7 files changed, 126 insertions(+)
 create mode 100644 drivers/virt/acrn/Kconfig
 create mode 100644 drivers/virt/acrn/Makefile
 create mode 100644 drivers/virt/acrn/acrn_drv.h
 create mode 100644 drivers/virt/acrn/hsm.c
MAINTAINERS +1 -0
index e0fea5e464b4..3030d0e93d02 100644
@@ -442,6 +442,7 @@ L:	[email protected]
 S:	Supported
 W:	https://projectacrn.org
 F:	Documentation/virt/acrn/
+F:	drivers/virt/acrn/
 
 AD1889 ALSA SOUND DRIVER
 L:	[email protected]
drivers/virt/Kconfig +2 -0
index cbc1f25c79ab..d9484a2e9b46 100644
@@ -32,4 +32,6 @@ config FSL_HV_MANAGER
 	     partition shuts down.
 
 source "drivers/virt/vboxguest/Kconfig"
+
+source "drivers/virt/acrn/Kconfig"
 endif
drivers/virt/Makefile +1 -0
index fd331247c27a..f0491bbf0d4d 100644
@@ -5,3 +5,4 @@
 
 obj-$(CONFIG_FSL_HV_MANAGER)	+= fsl_hypervisor.o
 obj-y				+= vboxguest/
+obj-$(CONFIG_ACRN_HSM)		+= acrn/
drivers/virt/acrn/Kconfig new file +14 -0
new file mode 100644
index 000000000000..36c80378c30c
@@ -0,0 +1,14 @@
+# SPDX-License-Identifier: GPL-2.0
+config ACRN_HSM
+	tristate "ACRN Hypervisor Service Module"
+	depends on ACRN_GUEST
+	help
+	  ACRN Hypervisor Service Module (HSM) is a kernel module which
+	  communicates with ACRN userspace through ioctls and talks to
+	  the ACRN Hypervisor through hypercalls. HSM will only run in
+	  a privileged management VM, called Service VM, to manage User
+	  VMs and do I/O emulation. Not required for simply running
+	  under ACRN as a User VM.
+
+	  To compile as a module, choose M, the module will be called
+	  acrn. If unsure, say N.
drivers/virt/acrn/Makefile new file +3 -0
new file mode 100644
index 000000000000..6920ed798aaf
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_ACRN_HSM)	:= acrn.o
+acrn-y := hsm.o
drivers/virt/acrn/acrn_drv.h new file +18 -0
new file mode 100644
index 000000000000..29eedd696327
@@ -0,0 +1,18 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#ifndef __ACRN_HSM_DRV_H
+#define __ACRN_HSM_DRV_H
+
+#include <linux/types.h>
+
+#define ACRN_INVALID_VMID (0xffffU)
+
+/**
+ * struct acrn_vm - Properties of ACRN User VM.
+ * @vmid:	User VM ID
+ */
+struct acrn_vm {
+	u16	vmid;
+};
+
+#endif /* __ACRN_HSM_DRV_H */
drivers/virt/acrn/hsm.c new file +87 -0

87 lines not on this page · raw message

Signature · 1 line
2.28.0
#30 lore raw

PATCH v5 6/17 virt: acrn: Introduce VM management interfaces

In reply to [email protected]

From: Shuo Liu <[email protected]>

The VM management interfaces expose several VM operations to ACRN userspace via ioctls. For example, creating VM, starting VM, destroying VM and so on.

The ACRN Hypervisor needs to exchange data with the ACRN userspace during the VM operations. HSM provides VM operation ioctls to the ACRN userspace and communicates with the ACRN Hypervisor for VM operations via hypercalls.

HSM maintains a list of User VM. Each User VM will be bound to an existing file descriptor of /dev/acrn_hsm. The User VM will be destroyed when the file descriptor is closed.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 .../userspace-api/ioctl/ioctl-number.rst      |  1 +
 MAINTAINERS                                   |  1 +
 drivers/virt/acrn/Makefile                    |  2 +-
 drivers/virt/acrn/acrn_drv.h                  | 21 ++++-
 drivers/virt/acrn/hsm.c                       | 73 ++++++++++++++++-
 drivers/virt/acrn/hypercall.h                 | 78 +++++++++++++++++++
 drivers/virt/acrn/vm.c                        | 68 ++++++++++++++++
 include/uapi/linux/acrn.h                     | 56 +++++++++++++
 8 files changed, 296 insertions(+), 4 deletions(-)
 create mode 100644 drivers/virt/acrn/hypercall.h
 create mode 100644 drivers/virt/acrn/vm.c
 create mode 100644 include/uapi/linux/acrn.h
Documentation/userspace-api/ioctl/ioctl-number.rst +1 -0
index 2a198838fca9..ac60efedb104 100644
@@ -319,6 +319,7 @@ Code  Seq#    Include File                                           Comments
 0xA0  all    linux/sdp/sdp.h                                         Industrial Device Project
                                                                      <mailto:[email protected]>
 0xA1  0      linux/vtpm_proxy.h                                      TPM Emulator Proxy Driver
+0xA2  all    uapi/linux/acrn.h                                       ACRN hypervisor
 0xA3  80-8F                                                          Port ACL  in development:
                                                                      <mailto:[email protected]>
 0xA3  90-9F  linux/dtlk.h
MAINTAINERS +1 -0
index 3030d0e93d02..d4c1ef303c2d 100644
@@ -443,6 +443,7 @@ S:	Supported
 W:	https://projectacrn.org
 F:	Documentation/virt/acrn/
 F:	drivers/virt/acrn/
+F:	include/uapi/linux/acrn.h
 
 AD1889 ALSA SOUND DRIVER
 L:	[email protected]
drivers/virt/acrn/Makefile +1 -1
index 6920ed798aaf..cf8b4ed5e74e 100644
@@ -1,3 +1,3 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_ACRN_HSM)	:= acrn.o
-acrn-y := hsm.o
+acrn-y := hsm.o vm.o
drivers/virt/acrn/acrn_drv.h +20 -1
index 29eedd696327..e5aba86cad8c 100644
@@ -3,16 +3,35 @@
 #ifndef __ACRN_HSM_DRV_H
 #define __ACRN_HSM_DRV_H
 
+#include <linux/acrn.h>
+#include <linux/dev_printk.h>
+#include <linux/miscdevice.h>
 #include <linux/types.h>
 
+#include "hypercall.h"
+
+extern struct miscdevice acrn_dev;
+
 #define ACRN_INVALID_VMID (0xffffU)
 
+#define ACRN_VM_FLAG_DESTROYED		0U
 /**
  * struct acrn_vm - Properties of ACRN User VM.
+ * @list:	Entry within global list of all VMs
  * @vmid:	User VM ID
+ * @vcpu_num:	Number of virtual CPUs in the VM
+ * @flags:	Flags (ACRN_VM_FLAG_*) of the VM. This is VM flag management
+ *		in HSM which is different from the &acrn_vm_creation.vm_flag.
  */
 struct acrn_vm {
-	u16	vmid;
+	struct list_head	list;
+	u16			vmid;
+	int			vcpu_num;
+	unsigned long		flags;
 };
 
+struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
+			       struct acrn_vm_creation *vm_param);
+int acrn_vm_destroy(struct acrn_vm *vm);
+
 #endif /* __ACRN_HSM_DRV_H */
drivers/virt/acrn/hsm.c +71 -2

97 lines not on this page · raw message

drivers/virt/acrn/hypercall.h new file +78 -0

78 lines not on this page · raw message

drivers/virt/acrn/vm.c new file +68 -0

68 lines not on this page · raw message

include/uapi/linux/acrn.h new file +56 -0

56 lines not on this page · raw message

Signature · 1 line
2.28.0
#31 lore raw

Re: [PATCH v5 06/17] virt: acrn: Introduce VM management interfaces

Quoted 18 lines · On Mon, Oct 19, 2020 at 02:17:52PM +0800, [email protected] wrote:
include/uapi/linux/acrn.h new file +15 -0
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Userspace interface for /dev/acrn_hsm - ACRN Hypervisor Service Module
+ *
+ * This file can be used by applications that need to communicate with the HSM
+ * via the ioctl interface.
+ */
+
+#ifndef _UAPI_ACRN_H
+#define _UAPI_ACRN_H
+
+#include <linux/types.h>
+
+/**
+ * struct acrn_vm_creation - Info to create a User VM

include/uapi/linux/acrn.h line 25

Quoted 10 lines
include/uapi/linux/acrn.h new file +10 -0
+ * @vmid:		User VM ID returned from the hypervisor
+ * @reserved0:		Reserved
+ * @vcpu_num:		Number of vCPU in the VM. Return from hypervisor.
+ * @reserved1:		Reserved
+ * @uuid:		UUID of the VM. Pass to hypervisor directly.
+ * @vm_flag:		Flag of the VM creating. Pass to hypervisor directly.
+ * @ioreq_buf:		Service VM GPA of I/O request buffer. Pass to
+ *			hypervisor directly.
+ * @cpu_affinity:	CPU affinity of the VM. Pass to hypervisor directly.
+ * @reserved2:		Reserved

Reserved and must be 0? What are they reserved for?

Same for all of the reserved fields, why?

include/uapi/linux/acrn.h line 32

Quoted 7 lines
include/uapi/linux/acrn.h new file +7 -0
+ */
+struct acrn_vm_creation {
+	__u16	vmid;
+	__u16	reserved0;
+	__u16	vcpu_num;
+	__u16	reserved1;
+	__u8	uuid[16];

We have a userspace-visable uid structure in include/uapi/uuid.h, why not use that?

thanks,

greg k-h

author
#32 lore raw

Re: [PATCH v5 06/17] virt: acrn: Introduce VM management interfaces

Quoted 23 lines · Greg Kroah-Hartman
Quoted 22 lines · On Mon, Oct 19, 2020 at 02:17:52PM +0800, [email protected] wrote:
include/uapi/linux/acrn.h new file +19 -0
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Userspace interface for /dev/acrn_hsm - ACRN Hypervisor Service Module
+ *
+ * This file can be used by applications that need to communicate with the HSM
+ * via the ioctl interface.
+ */
+
+#ifndef _UAPI_ACRN_H
+#define _UAPI_ACRN_H
+
+#include <linux/types.h>
+
+/**
+ * struct acrn_vm_creation - Info to create a User VM
+ * @vmid:		User VM ID returned from the hypervisor
+ * @reserved0:		Reserved
+ * @vcpu_num:		Number of vCPU in the VM. Return from hypervisor.
+ * @reserved1:		Reserved
Quoted 8 lines

include/uapi/linux/acrn.h line 25

Quoted 6 lines
include/uapi/linux/acrn.h new file +6 -0
+ * @uuid:		UUID of the VM. Pass to hypervisor directly.
+ * @vm_flag:		Flag of the VM creating. Pass to hypervisor directly.
+ * @ioreq_buf:		Service VM GPA of I/O request buffer. Pass to
+ *			hypervisor directly.
+ * @cpu_affinity:	CPU affinity of the VM. Pass to hypervisor directly.
+ * @reserved2:		Reserved

Reserved and must be 0?

Not a must.

Quoted 3 lines

What are they reserved for?

Same for all of the reserved fields, why?

Some reserved fields are to map layout in the hypervisor side, others are for future use.

Quoted 11 lines

include/uapi/linux/acrn.h line 32

Quoted 8 lines
include/uapi/linux/acrn.h new file +7 -0
+ */
+struct acrn_vm_creation {
+	__u16	vmid;
+	__u16	reserved0;
+	__u16	vcpu_num;
+	__u16	reserved1;
+	__u8	uuid[16];

We have a userspace-visable uid structure in include/uapi/uuid.h, why not use that?

we just pass the uuid data from user space to hypervisor. So, we can remove a header dependeny with using raw data format.

Thanks
shuo

#33 lore raw

Re: [PATCH v5 06/17] virt: acrn: Introduce VM management interfaces

Quoted 33 lines · Shuo A Liu
Quoted 23 lines · Greg Kroah-Hartman
Quoted 22 lines · On Mon, Oct 19, 2020 at 02:17:52PM +0800, [email protected] wrote:
include/uapi/linux/acrn.h new file +19 -0
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Userspace interface for /dev/acrn_hsm - ACRN Hypervisor Service Module
+ *
+ * This file can be used by applications that need to communicate with the HSM
+ * via the ioctl interface.
+ */
+
+#ifndef _UAPI_ACRN_H
+#define _UAPI_ACRN_H
+
+#include <linux/types.h>
+
+/**
+ * struct acrn_vm_creation - Info to create a User VM
+ * @vmid:		User VM ID returned from the hypervisor
+ * @reserved0:		Reserved
+ * @vcpu_num:		Number of vCPU in the VM. Return from hypervisor.
+ * @reserved1:		Reserved
Quoted 8 lines

include/uapi/linux/acrn.h line 25

Quoted 6 lines
include/uapi/linux/acrn.h new file +6 -0
+ * @uuid:		UUID of the VM. Pass to hypervisor directly.
+ * @vm_flag:		Flag of the VM creating. Pass to hypervisor directly.
+ * @ioreq_buf:		Service VM GPA of I/O request buffer. Pass to
+ *			hypervisor directly.
+ * @cpu_affinity:	CPU affinity of the VM. Pass to hypervisor directly.
+ * @reserved2:		Reserved

Reserved and must be 0?

Quoted 1 line

Not a must.

That's guaranteed to come back and bite you in the end. You all have read the "how to write a good api" document, right?

Quoted 6 lines
Quoted 3 lines

What are they reserved for?

Same for all of the reserved fields, why?

Some reserved fields are to map layout in the hypervisor side, others are for future use.

ioctls should not have these, again, please read the documentation. If you need something new in the future, just make a new ioctl.

Quoted 11 lines
Quoted 10 lines

include/uapi/linux/acrn.h line 32

Quoted 7 lines
include/uapi/linux/acrn.h new file +7 -0
+ */
+struct acrn_vm_creation {
+	__u16	vmid;
+	__u16	reserved0;
+	__u16	vcpu_num;
+	__u16	reserved1;
+	__u8	uuid[16];

We have a userspace-visable uid structure in include/uapi/uuid.h, why not use that?

Quoted 2 lines

we just pass the uuid data from user space to hypervisor. So, we can remove a header dependeny with using raw data format.

I do not understand this, please use the built-in kernel types we have.

thanks,

greg k-h

author
#34 lore raw

Re: [PATCH v5 06/17] virt: acrn: Introduce VM management interfaces

Quoted 34 lines · Greg Kroah-Hartman
Quoted 33 lines · Shuo A Liu
Quoted 31 lines · Greg Kroah-Hartman
Quoted 18 lines · On Mon, Oct 19, 2020 at 02:17:52PM +0800, [email protected] wrote:
include/uapi/linux/acrn.h new file +15 -0
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Userspace interface for /dev/acrn_hsm - ACRN Hypervisor Service Module
+ *
+ * This file can be used by applications that need to communicate with the HSM
+ * via the ioctl interface.
+ */
+
+#ifndef _UAPI_ACRN_H
+#define _UAPI_ACRN_H
+
+#include <linux/types.h>
+
+/**
+ * struct acrn_vm_creation - Info to create a User VM

include/uapi/linux/acrn.h line 25

Quoted 10 lines
include/uapi/linux/acrn.h new file +10 -0
+ * @vmid:		User VM ID returned from the hypervisor
+ * @reserved0:		Reserved
+ * @vcpu_num:		Number of vCPU in the VM. Return from hypervisor.
+ * @reserved1:		Reserved
+ * @uuid:		UUID of the VM. Pass to hypervisor directly.
+ * @vm_flag:		Flag of the VM creating. Pass to hypervisor directly.
+ * @ioreq_buf:		Service VM GPA of I/O request buffer. Pass to
+ *			hypervisor directly.
+ * @cpu_affinity:	CPU affinity of the VM. Pass to hypervisor directly.
+ * @reserved2:		Reserved

Reserved and must be 0?

Quoted 3 lines
Quoted 1 line

Not a must.

That's guaranteed to come back and bite you in the end.

OK. I can fill them with zero before passing them to hypervisor.

Quoted 1 line

You all have read the "how to write a good api" document, right?

Is it Documentation/driver-api/ioctl.rst? Or i missed..

Quoted 10 lines
Quoted 7 lines
Quoted 4 lines

What are they reserved for?

Same for all of the reserved fields, why?

Some reserved fields are to map layout in the hypervisor side, others are for future use.

ioctls should not have these, again, please read the documentation. If you need something new in the future, just make a new ioctl.

OK. I will remove some reserved fields for scalability. Though i can keep some reserved fields for alignment (and to keep same data structure layout with the hypervisor), right? Documentation/driver-api/ioctl.rst says that explicit reserved fields could be used.

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines

include/uapi/linux/acrn.h line 32

Quoted 8 lines
include/uapi/linux/acrn.h new file +7 -0
+ */
+struct acrn_vm_creation {
+	__u16	vmid;
+	__u16	reserved0;
+	__u16	vcpu_num;
+	__u16	reserved1;
+	__u8	uuid[16];

We have a userspace-visable uid structure in include/uapi/uuid.h, why not use that?

Quoted 4 lines
Quoted 2 lines

we just pass the uuid data from user space to hypervisor. So, we can remove a header dependeny with using raw data format.

I do not understand this, please use the built-in kernel types we have.

OK. I will use that.

Thanks
shuo

#35 lore raw

Re: [PATCH v5 06/17] virt: acrn: Introduce VM management interfaces

Quoted 37 lines · Shuo A Liu
Quoted 36 lines · Greg Kroah-Hartman
Quoted 34 lines · Shuo A Liu
Quoted 23 lines · Greg Kroah-Hartman
Quoted 22 lines · On Mon, Oct 19, 2020 at 02:17:52PM +0800, [email protected] wrote:
include/uapi/linux/acrn.h new file +19 -0
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Userspace interface for /dev/acrn_hsm - ACRN Hypervisor Service Module
+ *
+ * This file can be used by applications that need to communicate with the HSM
+ * via the ioctl interface.
+ */
+
+#ifndef _UAPI_ACRN_H
+#define _UAPI_ACRN_H
+
+#include <linux/types.h>
+
+/**
+ * struct acrn_vm_creation - Info to create a User VM
+ * @vmid:		User VM ID returned from the hypervisor
+ * @reserved0:		Reserved
+ * @vcpu_num:		Number of vCPU in the VM. Return from hypervisor.
+ * @reserved1:		Reserved
Quoted 8 lines

include/uapi/linux/acrn.h line 25

Quoted 6 lines
include/uapi/linux/acrn.h new file +6 -0
+ * @uuid:		UUID of the VM. Pass to hypervisor directly.
+ * @vm_flag:		Flag of the VM creating. Pass to hypervisor directly.
+ * @ioreq_buf:		Service VM GPA of I/O request buffer. Pass to
+ *			hypervisor directly.
+ * @cpu_affinity:	CPU affinity of the VM. Pass to hypervisor directly.
+ * @reserved2:		Reserved

Reserved and must be 0?

Not a must.

Quoted 7 lines
Quoted 1 line

That's guaranteed to come back and bite you in the end.

OK. I can fill them with zero before passing them to hypervisor.

Quoted 1 line

You all have read the "how to write a good api" document, right?

Is it Documentation/driver-api/ioctl.rst? Or i missed..

That's one good document, but no, not what I was referring to. I was thinking of Documentation/process/adding-syscalls.rst, which is what you are doing here implicitly with these new ioctls (every ioctl is a brand new syscall.)

Quoted 11 lines
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

What are they reserved for?

Same for all of the reserved fields, why?

Some reserved fields are to map layout in the hypervisor side, others are for future use.

ioctls should not have these, again, please read the documentation. If you need something new in the future, just make a new ioctl.

OK. I will remove some reserved fields for scalability.

"scalability" should have nothing to do with any of this, right? What am I missing?

Quoted 5 lines

Though i can
keep some reserved fields for alignment (and to keep same data structure layout with the hypervisor), right? Documentation/driver-api/ioctl.rst says that explicit reserved fields could be used.

If you need alignment, yes, that is fine, but that's not what you are saying these are for. And if you need alignment, why not move things around so they are properly aligned.

And this structure has nothing to do with the hypervisor structure, that's a internal-kernel structure, not a userspace-visable thing if you are doing things correctly.

As an example of all of this type of review and conversation, please refer to the review of the recent nitro_enclaves code that got merged. All of the discussions there about ioctls are also relevant here.

thanks,

greg k-h

author
#36 lore raw

Re: [PATCH v5 06/17] virt: acrn: Introduce VM management interfaces

Quoted 42 lines · Greg Kroah-Hartman
Quoted 41 lines · Shuo A Liu
Quoted 34 lines · Greg Kroah-Hartman
Quoted 33 lines · Shuo A Liu
Quoted 31 lines · Greg Kroah-Hartman
Quoted 18 lines · On Mon, Oct 19, 2020 at 02:17:52PM +0800, [email protected] wrote:
include/uapi/linux/acrn.h new file +15 -0
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Userspace interface for /dev/acrn_hsm - ACRN Hypervisor Service Module
+ *
+ * This file can be used by applications that need to communicate with the HSM
+ * via the ioctl interface.
+ */
+
+#ifndef _UAPI_ACRN_H
+#define _UAPI_ACRN_H
+
+#include <linux/types.h>
+
+/**
+ * struct acrn_vm_creation - Info to create a User VM

include/uapi/linux/acrn.h line 25

Quoted 10 lines
include/uapi/linux/acrn.h new file +10 -0
+ * @vmid:		User VM ID returned from the hypervisor
+ * @reserved0:		Reserved
+ * @vcpu_num:		Number of vCPU in the VM. Return from hypervisor.
+ * @reserved1:		Reserved
+ * @uuid:		UUID of the VM. Pass to hypervisor directly.
+ * @vm_flag:		Flag of the VM creating. Pass to hypervisor directly.
+ * @ioreq_buf:		Service VM GPA of I/O request buffer. Pass to
+ *			hypervisor directly.
+ * @cpu_affinity:	CPU affinity of the VM. Pass to hypervisor directly.
+ * @reserved2:		Reserved

Reserved and must be 0?

Quoted 3 lines
Quoted 1 line

Not a must.

That's guaranteed to come back and bite you in the end.

OK. I can fill them with zero before passing them to hypervisor.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

You all have read the "how to write a good api" document, right?

Is it Documentation/driver-api/ioctl.rst? Or i missed..

That's one good document, but no, not what I was referring to. I was thinking of Documentation/process/adding-syscalls.rst, which is what you are doing here implicitly with these new ioctls (every ioctl is a brand new syscall.)

I will read it as well. Thanks.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

What are they reserved for?

Same for all of the reserved fields, why?

Some reserved fields are to map layout in the hypervisor side, others are for future use.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

ioctls should not have these, again, please read the documentation. If you need something new in the future, just make a new ioctl.

OK. I will remove some reserved fields for scalability.

"scalability" should have nothing to do with any of this, right? What am I missing?

Sorry, i meant reserved fields for future use.

Quoted 7 lines
Quoted 6 lines

Though i can
keep some reserved fields for alignment (and to keep same data structure layout with the hypervisor), right? Documentation/driver-api/ioctl.rst says that explicit reserved fields could be used.

Quoted 7 lines

If you need alignment, yes, that is fine, but that's not what you are saying these are for. And if you need alignment, why not move things around so they are properly aligned.

And this structure has nothing to do with the hypervisor structure, that's a internal-kernel structure, not a userspace-visable thing if you are doing things correctly.

It's the same structure with the one in hypervisor. HSM driver doesn't maintain the VM much, it just pass the data for VM creation from userspace to hypervisor.

Quoted 4 lines

As an example of all of this type of review and conversation, please refer to the review of the recent nitro_enclaves code that got merged. All of the discussions there about ioctls are also relevant here.

I will. Thanks very much.

Thanks
shuo

#37 lore raw

Re: [PATCH v5 06/17] virt: acrn: Introduce VM management interfaces

Quoted 65 lines · Shuo A Liu
Quoted 42 lines · Greg Kroah-Hartman
Quoted 41 lines · Shuo A Liu
Quoted 34 lines · Greg Kroah-Hartman
Quoted 33 lines · Shuo A Liu
Quoted 31 lines · Greg Kroah-Hartman
Quoted 18 lines · On Mon, Oct 19, 2020 at 02:17:52PM +0800, [email protected] wrote:
include/uapi/linux/acrn.h new file +15 -0
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Userspace interface for /dev/acrn_hsm - ACRN Hypervisor Service Module
+ *
+ * This file can be used by applications that need to communicate with the HSM
+ * via the ioctl interface.
+ */
+
+#ifndef _UAPI_ACRN_H
+#define _UAPI_ACRN_H
+
+#include <linux/types.h>
+
+/**
+ * struct acrn_vm_creation - Info to create a User VM

include/uapi/linux/acrn.h line 25

Quoted 10 lines
include/uapi/linux/acrn.h new file +10 -0
+ * @vmid:		User VM ID returned from the hypervisor
+ * @reserved0:		Reserved
+ * @vcpu_num:		Number of vCPU in the VM. Return from hypervisor.
+ * @reserved1:		Reserved
+ * @uuid:		UUID of the VM. Pass to hypervisor directly.
+ * @vm_flag:		Flag of the VM creating. Pass to hypervisor directly.
+ * @ioreq_buf:		Service VM GPA of I/O request buffer. Pass to
+ *			hypervisor directly.
+ * @cpu_affinity:	CPU affinity of the VM. Pass to hypervisor directly.
+ * @reserved2:		Reserved

Reserved and must be 0?

Quoted 3 lines
Quoted 1 line

Not a must.

That's guaranteed to come back and bite you in the end.

OK. I can fill them with zero before passing them to hypervisor.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

You all have read the "how to write a good api" document, right?

Is it Documentation/driver-api/ioctl.rst? Or i missed..

That's one good document, but no, not what I was referring to. I was thinking of Documentation/process/adding-syscalls.rst, which is what you are doing here implicitly with these new ioctls (every ioctl is a brand new syscall.)

I will read it as well. Thanks.

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 7 lines
Quoted 4 lines

What are they reserved for?

Same for all of the reserved fields, why?

Some reserved fields are to map layout in the hypervisor side, others are for future use.

ioctls should not have these, again, please read the documentation. If you need something new in the future, just make a new ioctl.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

OK. I will remove some reserved fields for scalability.

"scalability" should have nothing to do with any of this, right? What am I missing?

Sorry, i meant reserved fields for future use.

Again, this is not how you do that at all. If you need something "in the future", create it then. What you are doing here ensures that you will never be able to do it then either, so don't even pretend :)

Read the syscall document for why this is the case.

Quoted 10 lines
Quoted 10 lines
Quoted 5 lines

Though i can
keep some reserved fields for alignment (and to keep same data structure layout with the hypervisor), right? Documentation/driver-api/ioctl.rst says that explicit reserved fields could be used.

If you need alignment, yes, that is fine, but that's not what you are saying these are for. And if you need alignment, why not move things around so they are properly aligned.

Quoted 7 lines
Quoted 3 lines

And this structure has nothing to do with the hypervisor structure, that's a internal-kernel structure, not a userspace-visable thing if you are doing things correctly.

It's the same structure with the one in hypervisor. HSM driver doesn't maintain the VM much, it just pass the data for VM creation from userspace to hypervisor.

That sounds ripe for abuse, good luck!

greg k-h

author
#38 lore raw

Re: [PATCH v5 06/17] virt: acrn: Introduce VM management interfaces

Quoted 71 lines · Greg Kroah-Hartman
Quoted 70 lines · Shuo A Liu
Quoted 42 lines · Greg Kroah-Hartman
Quoted 41 lines · Shuo A Liu
Quoted 34 lines · Greg Kroah-Hartman
Quoted 33 lines · Shuo A Liu
Quoted 31 lines · Greg Kroah-Hartman
Quoted 18 lines · On Mon, Oct 19, 2020 at 02:17:52PM +0800, [email protected] wrote:
include/uapi/linux/acrn.h new file +15 -0
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Userspace interface for /dev/acrn_hsm - ACRN Hypervisor Service Module
+ *
+ * This file can be used by applications that need to communicate with the HSM
+ * via the ioctl interface.
+ */
+
+#ifndef _UAPI_ACRN_H
+#define _UAPI_ACRN_H
+
+#include <linux/types.h>
+
+/**
+ * struct acrn_vm_creation - Info to create a User VM

include/uapi/linux/acrn.h line 25

Quoted 10 lines
include/uapi/linux/acrn.h new file +10 -0
+ * @vmid:		User VM ID returned from the hypervisor
+ * @reserved0:		Reserved
+ * @vcpu_num:		Number of vCPU in the VM. Return from hypervisor.
+ * @reserved1:		Reserved
+ * @uuid:		UUID of the VM. Pass to hypervisor directly.
+ * @vm_flag:		Flag of the VM creating. Pass to hypervisor directly.
+ * @ioreq_buf:		Service VM GPA of I/O request buffer. Pass to
+ *			hypervisor directly.
+ * @cpu_affinity:	CPU affinity of the VM. Pass to hypervisor directly.
+ * @reserved2:		Reserved

Reserved and must be 0?

Quoted 3 lines
Quoted 1 line

Not a must.

That's guaranteed to come back and bite you in the end.

OK. I can fill them with zero before passing them to hypervisor.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

You all have read the "how to write a good api" document, right?

Is it Documentation/driver-api/ioctl.rst? Or i missed..

That's one good document, but no, not what I was referring to. I was thinking of Documentation/process/adding-syscalls.rst, which is what you are doing here implicitly with these new ioctls (every ioctl is a brand new syscall.)

I will read it as well. Thanks.

Quoted 15 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

What are they reserved for?

Same for all of the reserved fields, why?

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Some reserved fields are to map layout in the hypervisor side, others are for future use.

ioctls should not have these, again, please read the documentation. If you need something new in the future, just make a new ioctl.

OK. I will remove some reserved fields for scalability.

"scalability" should have nothing to do with any of this, right? What am I missing?

Quoted 7 lines
Quoted 1 line

Sorry, i meant reserved fields for future use.

Again, this is not how you do that at all. If you need something "in the future", create it then. What you are doing here ensures that you will never be able to do it then either, so don't even pretend :)

Read the syscall document for why this is the case.

Alright. I will remove that type of reserved fields. Thank you.

Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
Quoted 6 lines

Though i can
keep some reserved fields for alignment (and to keep same data structure layout with the hypervisor), right? Documentation/driver-api/ioctl.rst says that explicit reserved fields could be used.

If you need alignment, yes, that is fine, but that's not what you are saying these are for. And if you need alignment, why not move things around so they are properly aligned.

And this structure has nothing to do with the hypervisor structure, that's a internal-kernel structure, not a userspace-visable thing if you are doing things correctly.

Quoted 5 lines
Quoted 3 lines

It's the same structure with the one in hypervisor. HSM driver doesn't maintain the VM much, it just pass the data for VM creation from userspace to hypervisor.

That sounds ripe for abuse, good luck!

The hypervisor will do the sanity check. In this case, HSM driver can be kept simple.

Thanks
shuo

#39 lore raw

Re: [PATCH v5 06/17] virt: acrn: Introduce VM management interfaces

Quoted 14 lines · Shuo A Liu
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 5 lines

Though i can
keep some reserved fields for alignment (and to keep same data structure layout with the hypervisor), right? Documentation/driver-api/ioctl.rst says that explicit reserved fields could be used.

If you need alignment, yes, that is fine, but that's not what you are saying these are for. And if you need alignment, why not move things around so they are properly aligned.

And this structure has nothing to do with the hypervisor structure, that's a internal-kernel structure, not a userspace-visable thing if you are doing things correctly.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

It's the same structure with the one in hypervisor. HSM driver doesn't maintain the VM much, it just pass the data for VM creation from userspace to hypervisor.

That sounds ripe for abuse, good luck!

The hypervisor will do the sanity check. In this case, HSM driver can be kept simple.

Hah, good luck with the fuzzers, crashing a hypervisor will be fun for them!

greg k-h

#40 lore raw

PATCH v5 7/17 virt: acrn: Introduce an ioctl to set vCPU registers state

In reply to [email protected]

From: Shuo Liu <[email protected]>

A virtual CPU of User VM has different context due to the different registers state. ACRN userspace needs to set the virtual CPU registers state (e.g. giving a initial registers state to a virtual BSP of a User VM).

HSM provides an ioctl ACRN_IOCTL_SET_VCPU_REGS to do the virtual CPU registers state setting. The ioctl passes the registers state from ACRN userspace to the hypervisor directly.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 drivers/virt/acrn/hsm.c       | 15 ++++++++
 drivers/virt/acrn/hypercall.h | 13 +++++++
 include/uapi/linux/acrn.h     | 71 +++++++++++++++++++++++++++++++++++
 3 files changed, 99 insertions(+)
drivers/virt/acrn/hsm.c +15 -0
index cbda67d4eb89..58ceb02e82db 100644
@@ -9,6 +9,7 @@
  *	Yakui Zhao <[email protected]>
  */
 
+#include <linux/io.h>
 #include <linux/mm.h>
 #include <linux/module.h>
 #include <linux/slab.h>
@@ -46,6 +47,7 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 {
 	struct acrn_vm *vm = filp->private_data;
 	struct acrn_vm_creation *vm_param;
+	struct acrn_vcpu_regs *cpu_regs;
 	int ret = 0;
 
 	if (vm->vmid == ACRN_INVALID_VMID && cmd != ACRN_IOCTL_CREATE_VM) {
@@ -97,6 +99,19 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 	case ACRN_IOCTL_DESTROY_VM:
 		ret = acrn_vm_destroy(vm);
 		break;
+	case ACRN_IOCTL_SET_VCPU_REGS:
+		cpu_regs = memdup_user((void __user *)ioctl_param,
+				       sizeof(struct acrn_vcpu_regs));
+		if (IS_ERR(cpu_regs))
+			return PTR_ERR(cpu_regs);
+
+		ret = hcall_set_vcpu_regs(vm->vmid, virt_to_phys(cpu_regs));
+		if (ret < 0)
+			dev_dbg(acrn_dev.this_device,
+				"Failed to set regs state of VM%u!\n",
+				vm->vmid);
+		kfree(cpu_regs);
+		break;
 	default:
 		dev_dbg(acrn_dev.this_device, "Unknown IOCTL 0x%x!\n", cmd);
 		ret = -ENOTTY;
drivers/virt/acrn/hypercall.h +13 -0
index 426b66cadb1f..f29cfae08862 100644
@@ -19,6 +19,7 @@
 #define HC_START_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x02)
 #define HC_PAUSE_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x03)
 #define HC_RESET_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x05)
+#define HC_SET_VCPU_REGS		_HC_ID(HC_ID, HC_ID_VM_BASE + 0x06)
 
 /**
  * hcall_create_vm() - Create a User VM
@@ -75,4 +76,16 @@ static inline long hcall_reset_vm(u64 vmid)
 	return acrn_hypercall1(HC_RESET_VM, vmid);
 }
 
+/**
+ * hcall_set_vcpu_regs() - Set up registers of virtual CPU of a User VM
+ * @vmid:	User VM ID
+ * @regs_state:	Service VM GPA of registers state
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_set_vcpu_regs(u64 vmid, u64 regs_state)
+{
+	return acrn_hypercall2(HC_SET_VCPU_REGS, vmid, regs_state);
+}
+
 #endif /* __ACRN_HSM_HYPERCALL_H */
include/uapi/linux/acrn.h +71 -0

82 lines not on this page · raw message

Signature · 1 line
2.28.0
#41 lore raw

Re: [PATCH v5 07/17] virt: acrn: Introduce an ioctl to set vCPU registers state

Quoted 46 lines · On Mon, Oct 19, 2020 at 02:17:53PM +0800, [email protected] wrote:

From: Shuo Liu <[email protected]>

A virtual CPU of User VM has different context due to the different registers state. ACRN userspace needs to set the virtual CPU registers state (e.g. giving a initial registers state to a virtual BSP of a User VM).

HSM provides an ioctl ACRN_IOCTL_SET_VCPU_REGS to do the virtual CPU registers state setting. The ioctl passes the registers state from ACRN userspace to the hypervisor directly.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 drivers/virt/acrn/hsm.c       | 15 ++++++++
 drivers/virt/acrn/hypercall.h | 13 +++++++
 include/uapi/linux/acrn.h     | 71 +++++++++++++++++++++++++++++++++++
 3 files changed, 99 insertions(+)
drivers/virt/acrn/hsm.c +2 -0
index cbda67d4eb89..58ceb02e82db 100644
@@ -9,6 +9,7 @@
  *	Yakui Zhao <[email protected]>
  */
 
+#include <linux/io.h>
 #include <linux/mm.h>
 #include <linux/module.h>
 #include <linux/slab.h>
@@ -46,6 +47,7 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 {
 	struct acrn_vm *vm = filp->private_data;
 	struct acrn_vm_creation *vm_param;
+	struct acrn_vcpu_regs *cpu_regs;
 	int ret = 0;
 
 	if (vm->vmid == ACRN_INVALID_VMID && cmd != ACRN_IOCTL_CREATE_VM) {

drivers/virt/acrn/hsm.c static long acrn_dev_ioctl(struct file *filp, unsigned int cmd, · line 108

Quoted 10 lines
drivers/virt/acrn/hsm.c +7 -0
@@ -97,6 +99,19 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 	case ACRN_IOCTL_DESTROY_VM:
 		ret = acrn_vm_destroy(vm);
 		break;
+	case ACRN_IOCTL_SET_VCPU_REGS:
+		cpu_regs = memdup_user((void __user *)ioctl_param,
+				       sizeof(struct acrn_vcpu_regs));
+		if (IS_ERR(cpu_regs))
+			return PTR_ERR(cpu_regs);
+
+		ret = hcall_set_vcpu_regs(vm->vmid, virt_to_phys(cpu_regs));

Why the virt_to_phys() call here? And there really is no validation of any fields?

drivers/virt/acrn/hsm.c static long acrn_dev_ioctl(struct file *filp, unsigned int cmd, · line 110

Quoted 2 lines
drivers/virt/acrn/hsm.c +2 -0
+		if (ret < 0)
+			dev_dbg(acrn_dev.this_device,

Wait, a global, static device? Where did I miss that? That feels odd, why is there just one?

Quoted 65 lines
drivers/virt/acrn/hsm.c +4 -0
+				"Failed to set regs state of VM%u!\n",
+				vm->vmid);
+		kfree(cpu_regs);
+		break;
 	default:
 		dev_dbg(acrn_dev.this_device, "Unknown IOCTL 0x%x!\n", cmd);
 		ret = -ENOTTY;
drivers/virt/acrn/hypercall.h +13 -0
index 426b66cadb1f..f29cfae08862 100644
@@ -19,6 +19,7 @@
 #define HC_START_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x02)
 #define HC_PAUSE_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x03)
 #define HC_RESET_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x05)
+#define HC_SET_VCPU_REGS		_HC_ID(HC_ID, HC_ID_VM_BASE + 0x06)
 
 /**
  * hcall_create_vm() - Create a User VM
@@ -75,4 +76,16 @@ static inline long hcall_reset_vm(u64 vmid)
 	return acrn_hypercall1(HC_RESET_VM, vmid);
 }
 
+/**
+ * hcall_set_vcpu_regs() - Set up registers of virtual CPU of a User VM
+ * @vmid:	User VM ID
+ * @regs_state:	Service VM GPA of registers state
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_set_vcpu_regs(u64 vmid, u64 regs_state)
+{
+	return acrn_hypercall2(HC_SET_VCPU_REGS, vmid, regs_state);
+}
+
 #endif /* __ACRN_HSM_HYPERCALL_H */
include/uapi/linux/acrn.h +21 -0
index 364b1a783074..1d5b82e154fb 100644
@@ -36,6 +36,75 @@ struct acrn_vm_creation {
 	__u8	reserved2[8];
 } __attribute__((aligned(8)));
 
+struct acrn_gp_regs {
+	__u64	rax;
+	__u64	rcx;
+	__u64	rdx;
+	__u64	rbx;
+	__u64	rsp;
+	__u64	rbp;
+	__u64	rsi;
+	__u64	rdi;
+	__u64	r8;
+	__u64	r9;
+	__u64	r10;
+	__u64	r11;
+	__u64	r12;
+	__u64	r13;
+	__u64	r14;
+	__u64	r15;
+};
+
+struct acrn_descriptor_ptr {
+	__u16	limit;

include/uapi/linux/acrn.h struct acrn_vm_creation { · line 69

Quoted 10 lines
include/uapi/linux/acrn.h +10 -0
+	__u64	base;
+	__u16	reserved[3];
+} __attribute__ ((__packed__));
+
+struct acrn_regs {
+	struct acrn_gp_regs		gprs;
+	struct acrn_descriptor_ptr	gdt;
+	struct acrn_descriptor_ptr	idt;
+
+	__u64				rip;

As these are all crossing the user/kernel boundry and then on to somewhere "else", you have to specify the endian of all of these, right?

if not, why not?

Quoted 24 lines
include/uapi/linux/acrn.h +24 -0
+	__u64				cs_base;
+	__u64				cr0;
+	__u64				cr4;
+	__u64				cr3;
+	__u64				ia32_efer;
+	__u64				rflags;
+	__u64				reserved_64[4];
+
+	__u32				cs_ar;
+	__u32				cs_limit;
+	__u32				reserved_32[3];
+
+	__u16				cs_sel;
+	__u16				ss_sel;
+	__u16				ds_sel;
+	__u16				es_sel;
+	__u16				fs_sel;
+	__u16				gs_sel;
+	__u16				ldt_sel;
+	__u16				tr_sel;
+
+	__u16				reserved_16[4];
+};
+

include/uapi/linux/acrn.h struct acrn_vm_creation { · line 103

Quoted 10 lines
include/uapi/linux/acrn.h +10 -0
+/**
+ * struct acrn_vcpu_regs - Info of vCPU registers state
+ * @vcpu_id:	vCPU ID
+ * @reserved0:	Reserved
+ * @vcpu_regs:	vCPU registers state
+ *
+ * This structure will be passed to hypervisor directly.
+ */
+struct acrn_vcpu_regs {
+	__u16			vcpu_id;

Endian?

include/uapi/linux/acrn.h struct acrn_vm_creation { · line 104

Quoted 1 line
include/uapi/linux/acrn.h +1 -0
+	__u16			reserved0[3];

What does the reserved fields do?

Is there a pointer to a public document for all of these structures somewhere?

include/uapi/linux/acrn.h struct acrn_vm_creation { · line 106

Quoted 2 lines
include/uapi/linux/acrn.h +2 -0
+	struct acrn_regs	vcpu_regs;
+} __attribute__((aligned(8)));

What does the alignment do here?

thanks,

greg k-h

author
#42 lore raw

Re: [PATCH v5 07/17] virt: acrn: Introduce an ioctl to set vCPU registers state

Hi Greg,

Quoted 45 lines · Greg Kroah-Hartman
Quoted 44 lines · On Mon, Oct 19, 2020 at 02:17:53PM +0800, [email protected] wrote:

From: Shuo Liu <[email protected]>

A virtual CPU of User VM has different context due to the different registers state. ACRN userspace needs to set the virtual CPU registers state (e.g. giving a initial registers state to a virtual BSP of a User VM).

HSM provides an ioctl ACRN_IOCTL_SET_VCPU_REGS to do the virtual CPU registers state setting. The ioctl passes the registers state from ACRN userspace to the hypervisor directly.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 drivers/virt/acrn/hsm.c       | 15 ++++++++
 drivers/virt/acrn/hypercall.h | 13 +++++++
 include/uapi/linux/acrn.h     | 71 +++++++++++++++++++++++++++++++++++
 3 files changed, 99 insertions(+)
drivers/virt/acrn/hsm.c +2 -0
index cbda67d4eb89..58ceb02e82db 100644
@@ -9,6 +9,7 @@
  *	Yakui Zhao <[email protected]>
  */
 
+#include <linux/io.h>
 #include <linux/mm.h>
 #include <linux/module.h>
 #include <linux/slab.h>
@@ -46,6 +47,7 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 {
 	struct acrn_vm *vm = filp->private_data;
 	struct acrn_vm_creation *vm_param;
+	struct acrn_vcpu_regs *cpu_regs;
 	int ret = 0;
 
Quoted 14 lines

drivers/virt/acrn/hsm.c static long acrn_dev_ioctl(struct file *filp, unsigned int cmd, · line 108

Quoted 12 lines
drivers/virt/acrn/hsm.c +7 -0
 	if (vm->vmid == ACRN_INVALID_VMID && cmd != ACRN_IOCTL_CREATE_VM) {
@@ -97,6 +99,19 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 	case ACRN_IOCTL_DESTROY_VM:
 		ret = acrn_vm_destroy(vm);
 		break;
+	case ACRN_IOCTL_SET_VCPU_REGS:
+		cpu_regs = memdup_user((void __user *)ioctl_param,
+				       sizeof(struct acrn_vcpu_regs));
+		if (IS_ERR(cpu_regs))
+			return PTR_ERR(cpu_regs);
+
+		ret = hcall_set_vcpu_regs(vm->vmid, virt_to_phys(cpu_regs));

Why the virt_to_phys() call here?

The hypervisor need a Guest Physical Address to access the data, so virt_to_phys() is used here.

Quoted 2 lines

And there really is no validation of
any fields?

Yes. Because HSM driver has little knowledge to do the validation.

Quoted 6 lines

drivers/virt/acrn/hsm.c static long acrn_dev_ioctl(struct file *filp, unsigned int cmd, · line 110

Quoted 3 lines
drivers/virt/acrn/hsm.c +2 -0
+		if (ret < 0)
+			dev_dbg(acrn_dev.this_device,

Wait, a global, static device? Where did I miss that? That feels odd, why is there just one?

The device is just for dev_*() usage. The driver links all active VMs in a global list acrn_vm_list. And also the notification interrupt (hypervisor to Service VM) is global. So, there is only one device.

Quoted 82 lines
Quoted 68 lines
drivers/virt/acrn/hsm.c +4 -0
+				"Failed to set regs state of VM%u!\n",
+				vm->vmid);
+		kfree(cpu_regs);
+		break;
 	default:
 		dev_dbg(acrn_dev.this_device, "Unknown IOCTL 0x%x!\n", cmd);
 		ret = -ENOTTY;
drivers/virt/acrn/hypercall.h +13 -0
index 426b66cadb1f..f29cfae08862 100644
@@ -19,6 +19,7 @@
 #define HC_START_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x02)
 #define HC_PAUSE_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x03)
 #define HC_RESET_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x05)
+#define HC_SET_VCPU_REGS		_HC_ID(HC_ID, HC_ID_VM_BASE + 0x06)
 
 /**
  * hcall_create_vm() - Create a User VM
@@ -75,4 +76,16 @@ static inline long hcall_reset_vm(u64 vmid)
 	return acrn_hypercall1(HC_RESET_VM, vmid);
 }
 
+/**
+ * hcall_set_vcpu_regs() - Set up registers of virtual CPU of a User VM
+ * @vmid:	User VM ID
+ * @regs_state:	Service VM GPA of registers state
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_set_vcpu_regs(u64 vmid, u64 regs_state)
+{
+	return acrn_hypercall2(HC_SET_VCPU_REGS, vmid, regs_state);
+}
+
 #endif /* __ACRN_HSM_HYPERCALL_H */
include/uapi/linux/acrn.h +21 -0
index 364b1a783074..1d5b82e154fb 100644
@@ -36,6 +36,75 @@ struct acrn_vm_creation {
 	__u8	reserved2[8];
 } __attribute__((aligned(8)));
 
+struct acrn_gp_regs {
+	__u64	rax;
+	__u64	rcx;
+	__u64	rdx;
+	__u64	rbx;
+	__u64	rsp;
+	__u64	rbp;
+	__u64	rsi;
+	__u64	rdi;
+	__u64	r8;
+	__u64	r9;
+	__u64	r10;
+	__u64	r11;
+	__u64	r12;
+	__u64	r13;
+	__u64	r14;
+	__u64	r15;
+};
+
+struct acrn_descriptor_ptr {
+	__u16	limit;

include/uapi/linux/acrn.h struct acrn_vm_creation { · line 69

Quoted 10 lines
include/uapi/linux/acrn.h +10 -0
+	__u64	base;
+	__u16	reserved[3];
+} __attribute__ ((__packed__));
+
+struct acrn_regs {
+	struct acrn_gp_regs		gprs;
+	struct acrn_descriptor_ptr	gdt;
+	struct acrn_descriptor_ptr	idt;
+
+	__u64				rip;

As these are all crossing the user/kernel boundry and then on to somewhere "else", you have to specify the endian of all of these, right?

Quoted 1 line

if not, why not?

The hypervisor and the driver only support X86_64 platform for now. So, the endian should be certain.

Quoted 40 lines
Quoted 27 lines
include/uapi/linux/acrn.h +24 -0
+	__u64				cs_base;
+	__u64				cr0;
+	__u64				cr4;
+	__u64				cr3;
+	__u64				ia32_efer;
+	__u64				rflags;
+	__u64				reserved_64[4];
+
+	__u32				cs_ar;
+	__u32				cs_limit;
+	__u32				reserved_32[3];
+
+	__u16				cs_sel;
+	__u16				ss_sel;
+	__u16				ds_sel;
+	__u16				es_sel;
+	__u16				fs_sel;
+	__u16				gs_sel;
+	__u16				ldt_sel;
+	__u16				tr_sel;
+
+	__u16				reserved_16[4];
+};
+

include/uapi/linux/acrn.h struct acrn_vm_creation { · line 103

Quoted 10 lines
include/uapi/linux/acrn.h +10 -0
+/**
+ * struct acrn_vcpu_regs - Info of vCPU registers state
+ * @vcpu_id:	vCPU ID
+ * @reserved0:	Reserved
+ * @vcpu_regs:	vCPU registers state
+ *
+ * This structure will be passed to hypervisor directly.
+ */
+struct acrn_vcpu_regs {
+	__u16			vcpu_id;

Endian?

Quoted 3 lines

include/uapi/linux/acrn.h struct acrn_vm_creation { · line 104

Quoted 1 line
include/uapi/linux/acrn.h +1 -0
+	__u16			reserved0[3];

What does the reserved fields do?

To keep same layout with the hypervisor. Because the structure will be passed to hypervisor directly.

Quoted 3 lines

Is there a pointer to a public document for all of these structures somewhere?

Unfortunately, no. I have added some documents for some strutures in the code via kernel-doc format.

Quoted 5 lines

include/uapi/linux/acrn.h struct acrn_vm_creation { · line 106

Quoted 3 lines
include/uapi/linux/acrn.h +2 -0
+	struct acrn_regs	vcpu_regs;
+} __attribute__((aligned(8)));

What does the alignment do here?

The hypervisor wants to access aligned data block to improve the efficiency. Currently, the hypervisor only runs on x86_64 platform.

Thanks
shuo

#43 lore raw

Re: [PATCH v5 07/17] virt: acrn: Introduce an ioctl to set vCPU registers state

Quoted 4 lines · Shuo A Liu
Quoted 2 lines

And there really is no validation of
any fields?

Yes. Because HSM driver has little knowledge to do the validation.

What is "HSM driver"? And you all are ready for fuzzers to break this into small pieces, right? No validation of any input parameters feels really really wrong. Best of luck!

Quoted 10 lines
Quoted 10 lines

include/uapi/linux/acrn.h struct acrn_vm_creation { · line 69

Quoted 6 lines
include/uapi/linux/acrn.h +6 -0
+struct acrn_regs {
+	struct acrn_gp_regs		gprs;
+	struct acrn_descriptor_ptr	gdt;
+	struct acrn_descriptor_ptr	idt;
+
+	__u64				rip;

As these are all crossing the user/kernel boundry and then on to somewhere "else", you have to specify the endian of all of these, right?

Quoted 4 lines
Quoted 1 line

if not, why not?

The hypervisor and the driver only support X86_64 platform for now. So, the endian should be certain.

Then specify it please.

Quoted 8 lines
Quoted 3 lines

include/uapi/linux/acrn.h struct acrn_vm_creation { · line 104

Quoted 1 line
include/uapi/linux/acrn.h +1 -0
+	__u16			reserved0[3];

What does the reserved fields do?

To keep same layout with the hypervisor. Because the structure will be passed to hypervisor directly.

Quoted 1 line
Quoted 5 lines
Quoted 2 lines

Is there a pointer to a public document for all of these structures somewhere?

Unfortunately, no. I have added some documents for some strutures in the code via kernel-doc format.

Is this not the hypervisor that this code is for:
https://projectacrn.org/
?

If not, what is this thing?

If so, how is there not documentation for it?

Quoted 7 lines
Quoted 4 lines

include/uapi/linux/acrn.h struct acrn_vm_creation { · line 106

Quoted 2 lines
include/uapi/linux/acrn.h +2 -0
+	struct acrn_regs	vcpu_regs;
+} __attribute__((aligned(8)));

What does the alignment do here?

The hypervisor wants to access aligned data block to improve the efficiency. Currently, the hypervisor only runs on x86_64 platform.

That's nice, but what do you think that adding this attribute to a structure provides you? Have you tested this really is doing what you think it is doing?

thanks,

greg k-h

author
#44 lore raw

Re: [PATCH v5 07/17] virt: acrn: Introduce an ioctl to set vCPU registers state

Quoted 9 lines · Greg Kroah-Hartman
Quoted 4 lines · Shuo A Liu
Quoted 2 lines

And there really is no validation of
any fields?

Yes. Because HSM driver has little knowledge to do the validation.

What is "HSM driver"? And you all are ready for fuzzers to break this into small pieces, right? No validation of any input parameters feels really really wrong. Best of luck!

This driver is HSM (Hypervisor Service Module) driver.
Take this hypercall for example. The register values are set by user, we can do nothing to verify them. If the values are wrong, the VM will crash and the hypervisor will handle that.

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

include/uapi/linux/acrn.h struct acrn_vm_creation { · line 69

Quoted 7 lines
include/uapi/linux/acrn.h +6 -0
+struct acrn_regs {
+	struct acrn_gp_regs		gprs;
+	struct acrn_descriptor_ptr	gdt;
+	struct acrn_descriptor_ptr	idt;
+
+	__u64				rip;

As these are all crossing the user/kernel boundry and then on to somewhere "else", you have to specify the endian of all of these, right?

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

if not, why not?

The hypervisor and the driver only support X86_64 platform for now. So, the endian should be certain.

Then specify it please.

Ok. Will be __le64. I will also fix other instances in this file.

Quoted 15 lines
Quoted 9 lines
Quoted 4 lines

include/uapi/linux/acrn.h struct acrn_vm_creation { · line 104

Quoted 2 lines
include/uapi/linux/acrn.h +1 -0
+	__u16			reserved0[3];

What does the reserved fields do?

To keep same layout with the hypervisor. Because the structure will be passed to hypervisor directly.

Quoted 1 line
Quoted 5 lines
Quoted 2 lines

Is there a pointer to a public document for all of these structures somewhere?

Unfortunately, no. I have added some documents for some strutures in the code via kernel-doc format.

Quoted 7 lines

Is this not the hypervisor that this code is for:
https://projectacrn.org/
?

If not, what is this thing?

If so, how is there not documentation for it?

Oh, yes. We have the structures documentation on the website. Pls refer
https://projectacrn.github.io/latest/api/hypercall_api.html#
I meant that some fields of structures might be lack of explanation.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

include/uapi/linux/acrn.h struct acrn_vm_creation { · line 106

Quoted 3 lines
include/uapi/linux/acrn.h +2 -0
+	struct acrn_regs	vcpu_regs;
+} __attribute__((aligned(8)));

What does the alignment do here?

Quoted 6 lines
Quoted 2 lines

The hypervisor wants to access aligned data block to improve the efficiency. Currently, the hypervisor only runs on x86_64 platform.

That's nice, but what do you think that adding this attribute to a structure provides you? Have you tested this really is doing what you think it is doing?

It could make the compiler put the data structure at aligned address. In the kernel the structures are almost from kmalloc, so the attribute might be not meaningful. But in the hypervisor, there are many such data structures in stack or in static pool, this attribute can make sure the data structures are located at the aligned address.

Thanks
shuo

#45 lore raw

Re: [PATCH v5 07/17] virt: acrn: Introduce an ioctl to set vCPU registers state

Quoted 7 lines · Shuo A Liu
Quoted 6 lines · Greg Kroah-Hartman
Quoted 4 lines · Shuo A Liu
Quoted 2 lines

And there really is no validation of
any fields?

Yes. Because HSM driver has little knowledge to do the validation.

Quoted 8 lines
Quoted 3 lines

What is "HSM driver"? And you all are ready for fuzzers to break this into small pieces, right? No validation of any input parameters feels really really wrong. Best of luck!

This driver is HSM (Hypervisor Service Module) driver.
Take this hypercall for example. The register values are set by user, we can do nothing to verify them. If the values are wrong, the VM will crash and the hypervisor will handle that.

Ah, nice, so you are creating a situation where any user can crash the VM, what can go wrong :(

Quoted 11 lines
Quoted 11 lines
Quoted 6 lines
Quoted 3 lines

Is there a pointer to a public document for all of these structures somewhere?

Unfortunately, no. I have added some documents for some strutures in the code via kernel-doc format.

Is this not the hypervisor that this code is for:
https://projectacrn.org/
?

Quoted 7 lines
Quoted 3 lines

If not, what is this thing?

If so, how is there not documentation for it?

Oh, yes. We have the structures documentation on the website. Pls refer
https://projectacrn.github.io/latest/api/hypercall_api.html#
I meant that some fields of structures might be lack of explanation.

Please work on the documentation of the fields, otherwise it doesn't really make much sense what is happening here, right?

Along these lines, where is the userspace code that makes all of these ioctls? Surely the fields must be documented there, right?

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

include/uapi/linux/acrn.h struct acrn_vm_creation { · line 106

Quoted 2 lines
include/uapi/linux/acrn.h +2 -0
+	struct acrn_regs	vcpu_regs;
+} __attribute__((aligned(8)));

What does the alignment do here?

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

The hypervisor wants to access aligned data block to improve the efficiency. Currently, the hypervisor only runs on x86_64 platform.

That's nice, but what do you think that adding this attribute to a structure provides you? Have you tested this really is doing what you think it is doing?

It could make the compiler put the data structure at aligned address.

Are you sure this is the correct way to do that?

Quoted 4 lines

In the kernel the structures are almost from kmalloc, so the attribute might be not meaningful. But in the hypervisor, there are many such data structures in stack or in static pool, this attribute can make sure the data structures are located at the aligned address.

This is kernel data, not hypervisor data, in kernel memory. If you require the hypervisor to get the structures at a specific alignment in memory, you better check that in the kernel code, otherwise how can you ensure it?

thanks,

greg k-h

author
#46 lore raw

Re: [PATCH v5 07/17] virt: acrn: Introduce an ioctl to set vCPU registers state

Quoted 12 lines · Greg Kroah-Hartman
Quoted 11 lines · Shuo A Liu
Quoted 9 lines · Greg Kroah-Hartman
Quoted 4 lines · Shuo A Liu
Quoted 2 lines

And there really is no validation of
any fields?

Yes. Because HSM driver has little knowledge to do the validation.

What is "HSM driver"? And you all are ready for fuzzers to break this into small pieces, right? No validation of any input parameters feels really really wrong. Best of luck!

Quoted 7 lines
Quoted 4 lines

This driver is HSM (Hypervisor Service Module) driver.
Take this hypercall for example. The register values are set by user, we can do nothing to verify them. If the values are wrong, the VM will crash and the hypervisor will handle that.

Ah, nice, so you are creating a situation where any user can crash the VM, what can go wrong :(

Not any user, only the one who create the VM. Others cannot access the same VM. Let me list a piece of pesudo code of the device usage:

fd = open the /dev/acrn_hsm, HSM driver binds a VM with the fd ioctl (fd, CREATE_VM, SET_VCPU_REGS, START_VM ...) close fd

The one who create the VM should have full control of the VM.

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 7 lines
Quoted 4 lines

Is there a pointer to a public document for all of these structures somewhere?

Unfortunately, no. I have added some documents for some strutures in the code via kernel-doc format.

Is this not the hypervisor that this code is for:
https://projectacrn.org/
?

If not, what is this thing?

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

If so, how is there not documentation for it?

Oh, yes. We have the structures documentation on the website. Pls refer
https://projectacrn.github.io/latest/api/hypercall_api.html#
I meant that some fields of structures might be lack of explanation.

Please work on the documentation of the fields, otherwise it doesn't really make much sense what is happening here, right?

Right. We will try to enrich them.

Quoted 3 lines

Along these lines, where is the userspace code that makes all of these ioctls? Surely the fields must be documented there, right?

A userspace tool uses the ioctls, you can find the source from:
https://github.com/projectacrn/acrn-hypervisor/blob/master/devicemodel/core/vmmapi.c
There is a bit difference with the version i posted as i did some polish for upstream.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines

include/uapi/linux/acrn.h struct acrn_vm_creation { · line 106

Quoted 3 lines
include/uapi/linux/acrn.h +2 -0
+	struct acrn_regs	vcpu_regs;
+} __attribute__((aligned(8)));

What does the alignment do here?

The hypervisor wants to access aligned data block to improve the efficiency. Currently, the hypervisor only runs on x86_64 platform.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

That's nice, but what do you think that adding this attribute to a structure provides you? Have you tested this really is doing what you think it is doing?

It could make the compiler put the data structure at aligned address.

Are you sure this is the correct way to do that?

I tried the attribute it works.

test.c:

struct foo_256_aligned {
	int a;
	unsigned char b;
} __attribute__((aligned(256)));
struct foo {
	int a;
	unsigned char b;
};
int main(int argc, char** argv) {
	struct foo_256_aligned a;
	struct foo b;
	printf("%p %p\n", &a, &b);
}
# gcc -o test test.c && ./test
0x7ffe2af04b00 0x7ffe2af04af8
Quoted 10 lines
Quoted 5 lines

In the kernel the structures are almost from kmalloc, so the attribute might be not meaningful. But in the hypervisor, there are many such data structures in stack or in static pool, this attribute can make sure the data structures are located at the aligned address.

This is kernel data, not hypervisor data, in kernel memory. If you require the hypervisor to get the structures at a specific alignment in memory, you better check that in the kernel code, otherwise how can you ensure it?

Oh, this is a public header file which will be used by the hypervisor as well. For kernel data from kmalloc, it should be also ok even if it isn't 8 bytes aligned. The hypervisor will copy them if need keep persistent.

Thanks
shuo

#47 lore raw

Re: [PATCH v5 07/17] virt: acrn: Introduce an ioctl to set vCPU registers state

Quoted 21 lines · Shuo A Liu
Quoted 12 lines · Greg Kroah-Hartman
Quoted 11 lines · Shuo A Liu
Quoted 9 lines · Greg Kroah-Hartman
Quoted 4 lines · Shuo A Liu
Quoted 2 lines

And there really is no validation of
any fields?

Yes. Because HSM driver has little knowledge to do the validation.

What is "HSM driver"? And you all are ready for fuzzers to break this into small pieces, right? No validation of any input parameters feels really really wrong. Best of luck!

Quoted 7 lines
Quoted 4 lines

This driver is HSM (Hypervisor Service Module) driver.
Take this hypercall for example. The register values are set by user, we can do nothing to verify them. If the values are wrong, the VM will crash and the hypervisor will handle that.

Ah, nice, so you are creating a situation where any user can crash the VM, what can go wrong :(

Quoted 8 lines

Not any user, only the one who create the VM. Others cannot access the same VM. Let me list a piece of pesudo code of the device usage:

fd = open the /dev/acrn_hsm, HSM driver binds a VM with the fd ioctl (fd, CREATE_VM, SET_VCPU_REGS, START_VM ...) close fd

The one who create the VM should have full control of the VM.

Including crashing it and potentially corrupting the hypervisor? Great, remind me to never use this code :)

Quoted 25 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 6 lines
Quoted 3 lines

Is there a pointer to a public document for all of these structures somewhere?

Unfortunately, no. I have added some documents for some strutures in the code via kernel-doc format.

Is this not the hypervisor that this code is for:
https://projectacrn.org/
?

If not, what is this thing?

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

If so, how is there not documentation for it?

Oh, yes. We have the structures documentation on the website. Pls refer
https://projectacrn.github.io/latest/api/hypercall_api.html#
I meant that some fields of structures might be lack of explanation.

Please work on the documentation of the fields, otherwise it doesn't really make much sense what is happening here, right?

Right. We will try to enrich them.

Quoted 1 line
Quoted 7 lines
Quoted 2 lines

Along these lines, where is the userspace code that makes all of these ioctls? Surely the fields must be documented there, right?

A userspace tool uses the ioctls, you can find the source from:
https://github.com/projectacrn/acrn-hypervisor/blob/master/devicemodel/core/vmmapi.c
There is a bit difference with the version i posted as i did some polish for upstream.

I do not understand, this isn't your version? Where is the "correct" implementation? Shouldn't that be part of the patch series documentation? Or was it and I missed it as this thread is so long?

Quoted 35 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+	struct acrn_regs	vcpu_regs;
+} __attribute__((aligned(8)));

What does the alignment do here?

The hypervisor wants to access aligned data block to improve the efficiency. Currently, the hypervisor only runs on x86_64 platform.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

That's nice, but what do you think that adding this attribute to a structure provides you? Have you tested this really is doing what you think it is doing?

It could make the compiler put the data structure at aligned address.

Are you sure this is the correct way to do that?

I tried the attribute it works.

test.c:

struct foo_256_aligned {
	int a;
	unsigned char b;
} __attribute__((aligned(256)));
struct foo {
	int a;
	unsigned char b;
};
int main(int argc, char** argv) {
	struct foo_256_aligned a;
	struct foo b;
	printf("%p %p\n", &a, &b);
}
Quoted 2 lines
# gcc -o test test.c && ./test
0x7ffe2af04b00 0x7ffe2af04af8

But you will not have these data structures defined on the stack like this in the kernel source.

You will be assigning the structure to some chunk of memory you dynamically allocate, right? That's my main point here, if this is something that you HAVE to have right, then you HAVE to test and verify it is true when the structure is sent to you.

To just assume that a compiler marking will somehow enforce this alignment on random chunks of allocated memory is very odd.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

In the kernel the structures are almost from kmalloc, so the attribute might be not meaningful. But in the hypervisor, there are many such data structures in stack or in static pool, this attribute can make sure the data structures are located at the aligned address.

Quoted 7 lines
Quoted 4 lines

This is kernel data, not hypervisor data, in kernel memory. If you require the hypervisor to get the structures at a specific alignment in memory, you better check that in the kernel code, otherwise how can you ensure it?

Oh, this is a public header file which will be used by the hypervisor as well.

Then you HAVE to use the proper user/kernel structures for it.

Quoted 3 lines

For kernel data from kmalloc, it should be also ok even if it isn't 8 bytes aligned. The hypervisor will copy them if need keep persistent.

See, your compiler marking didn't work :)

greg k-h

author
#48 lore raw

Re: [PATCH v5 07/17] virt: acrn: Introduce an ioctl to set vCPU registers state

Quoted 25 lines · Greg Kroah-Hartman
Quoted 24 lines · Shuo A Liu
Quoted 12 lines · Greg Kroah-Hartman
Quoted 11 lines · Shuo A Liu
Quoted 9 lines · Greg Kroah-Hartman
Quoted 4 lines · Shuo A Liu
Quoted 2 lines

And there really is no validation of
any fields?

Yes. Because HSM driver has little knowledge to do the validation.

What is "HSM driver"? And you all are ready for fuzzers to break this into small pieces, right? No validation of any input parameters feels really really wrong. Best of luck!

Quoted 7 lines
Quoted 4 lines

This driver is HSM (Hypervisor Service Module) driver.
Take this hypercall for example. The register values are set by user, we can do nothing to verify them. If the values are wrong, the VM will crash and the hypervisor will handle that.

Ah, nice, so you are creating a situation where any user can crash the VM, what can go wrong :(

Not any user, only the one who create the VM. Others cannot access the same VM. Let me list a piece of pesudo code of the device usage:

Quoted 8 lines
Quoted 5 lines

fd = open the /dev/acrn_hsm, HSM driver binds a VM with the fd ioctl (fd, CREATE_VM, SET_VCPU_REGS, START_VM ...) close fd

The one who create the VM should have full control of the VM.

Including crashing it and potentially corrupting the hypervisor? Great, remind me to never use this code :)

The crashing here is within guest VM itself. It should not affect the hypervisor, else bug appears. The hypervisor should handle such case correctly (e.g. add more validataion in hypervisor) :)

Quoted 29 lines
Quoted 29 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 7 lines
Quoted 4 lines

Is there a pointer to a public document for all of these structures somewhere?

Unfortunately, no. I have added some documents for some strutures in the code via kernel-doc format.

Is this not the hypervisor that this code is for:
https://projectacrn.org/
?

If not, what is this thing?

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

If so, how is there not documentation for it?

Oh, yes. We have the structures documentation on the website. Pls refer
https://projectacrn.github.io/latest/api/hypercall_api.html#
I meant that some fields of structures might be lack of explanation.

Please work on the documentation of the fields, otherwise it doesn't really make much sense what is happening here, right?

Right. We will try to enrich them.

Quoted 3 lines

Along these lines, where is the userspace code that makes all of these ioctls? Surely the fields must be documented there, right?

Quoted 8 lines
Quoted 4 lines

A userspace tool uses the ioctls, you can find the source from:
https://github.com/projectacrn/acrn-hypervisor/blob/master/devicemodel/core/vmmapi.c
There is a bit difference with the version i posted as i did some polish for upstream.

I do not understand, this isn't your version? Where is the "correct" implementation? Shouldn't that be part of the patch series documentation? Or was it and I missed it as this thread is so long?

You didn't miss that. The link i pasted above is from a userspace application (named devicemodel in ACRN project) that using this driver on ACRN hypervisor (like QEMU on KVM). I didn't put it in the patch, because it's a huge program. There are some difference of the ioctl definition within this patchset (changing of name and definition), because we are using that currently for daily development. When the upstreaming patch got merged, they will be updated.

So, it seems you wanted a brief example code of driver usage in documentation, am i right? If yes, i can try to make one. But it might only show a brief flow of VM creating, memory setup, vcpu setup, VM starting, maybe IO handling, VM destroy, ..etc. Is that ok?

Quoted 36 lines
Quoted 36 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines

Review

Quoted 3 lines
(quoted diff) +2 -0
+	struct acrn_regs	vcpu_regs;
+} __attribute__((aligned(8)));

What does the alignment do here?

The hypervisor wants to access aligned data block to improve the efficiency. Currently, the hypervisor only runs on x86_64 platform.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

That's nice, but what do you think that adding this attribute to a structure provides you? Have you tested this really is doing what you think it is doing?

It could make the compiler put the data structure at aligned address.

Are you sure this is the correct way to do that?

I tried the attribute it works.

test.c:

struct foo_256_aligned {
	int a;
	unsigned char b;
} __attribute__((aligned(256)));
struct foo {
	int a;
	unsigned char b;
};
int main(int argc, char** argv) {
	struct foo_256_aligned a;
	struct foo b;
	printf("%p %p\n", &a, &b);
}
Quoted 8 lines
Quoted 2 lines
# gcc -o test test.c && ./test
0x7ffe2af04b00 0x7ffe2af04af8

But you will not have these data structures defined on the stack like this in the kernel source.

You will be assigning the structure to some chunk of memory you dynamically allocate, right? That's my main point here, if this is

Understand.

Quoted 16 lines

something that you HAVE to have right, then you HAVE to test and verify it is true when the structure is sent to you.

To just assume that a compiler marking will somehow enforce this alignment on random chunks of allocated memory is very odd.

Quoted 10 lines
Quoted 9 lines
Quoted 4 lines

In the kernel the structures are almost from kmalloc, so the attribute might be not meaningful. But in the hypervisor, there are many such data structures in stack or in static pool, this attribute can make sure the data structures are located at the aligned address.

This is kernel data, not hypervisor data, in kernel memory. If you require the hypervisor to get the structures at a specific alignment in memory, you better check that in the kernel code, otherwise how can you ensure it?

Quoted 4 lines
Quoted 2 lines

Oh, this is a public header file which will be used by the hypervisor as well.

Then you HAVE to use the proper user/kernel structures for it.

Sorry, i didn't follow here. The structures are used in

* devicemodel (userspace application, like QEMU in KVM)
* HSM (this driver, pass the data from devicemodel to hypervisor)
* hypervisor

They all should be compiled with the same public header from kernel, is my understanding right? If hypervisor source wants to use the compiler marking, need i only duplicate one header with the modification for that?

Thanks
shuo

#49 lore raw

PATCH v5 8/17 virt: acrn: Introduce EPT mapping management

In reply to [email protected]

From: Shuo Liu <[email protected]>

The HSM provides hypervisor services to the ACRN userspace. While launching a User VM, ACRN userspace needs to allocate memory and request the ACRN Hypervisor to set up the EPT mapping for the VM.

A mapping cache is introduced for accelerating the translation between the Service VM kernel virtual address and User VM physical address.

From the perspective of the hypervisor, the types of GPA of User VM can be listed as following:
1) RAM region, which is used by User VM as system ram.
2) MMIO region, which is recognized by User VM as MMIO. MMIO region is used to be utilized for devices emulation.

Generally, User VM RAM regions mapping is set up before VM started and is released in the User VM destruction. MMIO regions mapping may be set and unset dynamically during User VM running.

To achieve this, ioctls ACRN_IOCTL_SET_MEMSEG and ACRN_IOCTL_UNSET_MEMSEG are introduced in HSM.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 drivers/virt/acrn/Makefile    |   2 +-
 drivers/virt/acrn/acrn_drv.h  |  98 ++++++++++-
 drivers/virt/acrn/hsm.c       |  15 ++
 drivers/virt/acrn/hypercall.h |  14 ++
 drivers/virt/acrn/mm.c        | 305 ++++++++++++++++++++++++++++++++++
 drivers/virt/acrn/vm.c        |   4 +
 include/uapi/linux/acrn.h     |  51 ++++++
 7 files changed, 479 insertions(+), 10 deletions(-)
 create mode 100644 drivers/virt/acrn/mm.c
drivers/virt/acrn/Makefile +1 -1
index cf8b4ed5e74e..38bc44b6edcd 100644
@@ -1,3 +1,3 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_ACRN_HSM)	:= acrn.o
-acrn-y := hsm.o vm.o
+acrn-y := hsm.o vm.o mm.o
drivers/virt/acrn/acrn_drv.h +89 -9

115 lines not on this page · raw message

drivers/virt/acrn/hsm.c +15 -0
index 58ceb02e82db..016a6f2a4374 100644
@@ -48,6 +48,7 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 	struct acrn_vm *vm = filp->private_data;
 	struct acrn_vm_creation *vm_param;
 	struct acrn_vcpu_regs *cpu_regs;
+	struct acrn_vm_memmap memmap;
 	int ret = 0;
 
 	if (vm->vmid == ACRN_INVALID_VMID && cmd != ACRN_IOCTL_CREATE_VM) {
@@ -112,6 +113,20 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 				vm->vmid);
 		kfree(cpu_regs);
 		break;
+	case ACRN_IOCTL_SET_MEMSEG:
+		if (copy_from_user(&memmap, (void __user *)ioctl_param,
+				   sizeof(memmap)))
+			return -EFAULT;
+
+		ret = acrn_vm_memseg_map(vm, &memmap);
+		break;
+	case ACRN_IOCTL_UNSET_MEMSEG:
+		if (copy_from_user(&memmap, (void __user *)ioctl_param,
+				   sizeof(memmap)))
+			return -EFAULT;
+
+		ret = acrn_vm_memseg_unmap(vm, &memmap);
+		break;
 	default:
 		dev_dbg(acrn_dev.this_device, "Unknown IOCTL 0x%x!\n", cmd);
 		ret = -ENOTTY;
drivers/virt/acrn/hypercall.h +14 -0
index f29cfae08862..a1a70a071713 100644
@@ -21,6 +21,9 @@
 #define HC_RESET_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x05)
 #define HC_SET_VCPU_REGS		_HC_ID(HC_ID, HC_ID_VM_BASE + 0x06)
 
+#define HC_ID_MEM_BASE			0x40UL
+#define HC_VM_SET_MEMORY_REGIONS	_HC_ID(HC_ID, HC_ID_MEM_BASE + 0x02)
+
 /**
  * hcall_create_vm() - Create a User VM
  * @vminfo:	Service VM GPA of info of User VM creation
@@ -88,4 +91,15 @@ static inline long hcall_set_vcpu_regs(u64 vmid, u64 regs_state)
 	return acrn_hypercall2(HC_SET_VCPU_REGS, vmid, regs_state);
 }
 
+/**
+ * hcall_set_memory_regions() - Inform the hypervisor to set up EPT mappings
+ * @regions_pa:	Service VM GPA of &struct vm_memory_region_batch
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_set_memory_regions(u64 regions_pa)
+{
+	return acrn_hypercall1(HC_VM_SET_MEMORY_REGIONS, regions_pa);
+}
+
 #endif /* __ACRN_HSM_HYPERCALL_H */
drivers/virt/acrn/mm.c new file +305 -0

305 lines not on this page · raw message

drivers/virt/acrn/vm.c +4 -0
index 3f667ac8ac1e..ff5df7acb551 100644
@@ -31,6 +31,7 @@ struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
 		return NULL;
 	}
 
+	mutex_init(&vm->regions_mapping_lock);
 	vm->vmid = vm_param->vmid;
 	vm->vcpu_num = vm_param->vcpu_num;
 
@@ -62,6 +63,9 @@ int acrn_vm_destroy(struct acrn_vm *vm)
 		clear_bit(ACRN_VM_FLAG_DESTROYED, &vm->flags);
 		return ret;
 	}
+
+	acrn_vm_all_ram_unmap(vm);
+
 	dev_dbg(acrn_dev.this_device, "VM %u destroyed.\n", vm->vmid);
 	vm->vmid = ACRN_INVALID_VMID;
 	return 0;
include/uapi/linux/acrn.h +51 -0

61 lines not on this page · raw message

Signature · 1 line
2.28.0
#50 lore raw

PATCH v5 9/17 virt: acrn: Introduce I/O request management

In reply to [email protected]

From: Shuo Liu <[email protected]>

An I/O request of a User VM, which is constructed by the hypervisor, is distributed by the ACRN Hypervisor Service Module to an I/O client corresponding to the address range of the I/O request.

For each User VM, there is a shared 4-KByte memory region used for I/O requests communication between the hypervisor and Service VM. An I/O request is a 256-byte structure buffer, which is 'struct acrn_io_request', that is filled by an I/O handler of the hypervisor when a trapped I/O access happens in a User VM. ACRN userspace in the Service VM first allocates a 4-KByte page and passes the GPA (Guest Physical Address) of the buffer to the hypervisor. The buffer is used as an array of 16 I/O request slots with each I/O request slot being 256 bytes. This array is indexed by vCPU ID.

An I/O client, which is 'struct acrn_ioreq_client', is responsible for handling User VM I/O requests whose accessed GPA falls in a certain range. Multiple I/O clients can be associated with each User VM. There is a special client associated with each User VM, called the default client, that handles all I/O requests that do not fit into the range of any other I/O clients. The ACRN userspace acts as the default client for each User VM.

The state transitions of a ACRN I/O request are as follows.

FREE -> PENDING -> PROCESSING -> COMPLETE -> FREE -> ...

FREE: this I/O request slot is empty
PENDING: a valid I/O request is pending in this slot
PROCESSING: the I/O request is being processed
COMPLETE: the I/O request has been processed

An I/O request in COMPLETE or FREE state is owned by the hypervisor. HSM and ACRN userspace are in charge of processing the others.

The processing flow of I/O requests are listed as following:

a) The I/O handler of the hypervisor will fill an I/O request with PENDING state when a trapped I/O access happens in a User VM.
b) The hypervisor makes an upcall, which is a notification interrupt, to the Service VM.
c) The upcall handler schedules a tasklet to dispatch I/O requests.
d) The tasklet looks for the PENDING I/O requests, assigns them to different registered clients based on the address of the I/O accesses, updates their state to PROCESSING, and notifies the corresponding client to handle.
e) The notified client handles the assigned I/O requests.
f) The HSM updates I/O requests states to COMPLETE and notifies the hypervisor of the completion via hypercalls.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 drivers/virt/acrn/Makefile    |   2 +-
 drivers/virt/acrn/acrn_drv.h  |  82 ++++++
 drivers/virt/acrn/hsm.c       |  32 ++-
 drivers/virt/acrn/hypercall.h |  28 ++
 drivers/virt/acrn/ioreq.c     | 509 ++++++++++++++++++++++++++++++++++
 drivers/virt/acrn/vm.c        |  27 +-
 include/uapi/linux/acrn.h     | 134 +++++++++
 7 files changed, 804 insertions(+), 10 deletions(-)
 create mode 100644 drivers/virt/acrn/ioreq.c
drivers/virt/acrn/Makefile +1 -1
index 38bc44b6edcd..21721cbf6a80 100644
@@ -1,3 +1,3 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_ACRN_HSM)	:= acrn.o
-acrn-y := hsm.o vm.o mm.o
+acrn-y := hsm.o vm.o mm.o ioreq.o
drivers/virt/acrn/acrn_drv.h +82 -0

117 lines not on this page · raw message

drivers/virt/acrn/hsm.c +30 -2
index 016a6f2a4374..15cfb4fb5f1b 100644
@@ -48,6 +48,7 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 	struct acrn_vm *vm = filp->private_data;
 	struct acrn_vm_creation *vm_param;
 	struct acrn_vcpu_regs *cpu_regs;
+	struct acrn_ioreq_notify notify;
 	struct acrn_vm_memmap memmap;
 	int ret = 0;
 
@@ -127,6 +128,29 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 
 		ret = acrn_vm_memseg_unmap(vm, &memmap);
 		break;
+	case ACRN_IOCTL_CREATE_IOREQ_CLIENT:
+		if (vm->default_client)
+			return -EEXIST;
+		if (!acrn_ioreq_client_create(vm, NULL, NULL, true, "acrndm"))
+			ret = -EINVAL;
+		break;
+	case ACRN_IOCTL_DESTROY_IOREQ_CLIENT:
+		if (vm->default_client)
+			acrn_ioreq_client_destroy(vm->default_client);
+		break;
+	case ACRN_IOCTL_ATTACH_IOREQ_CLIENT:
+		if (vm->default_client)
+			ret = acrn_ioreq_client_wait(vm->default_client);
+		break;
+	case ACRN_IOCTL_NOTIFY_REQUEST_FINISH:
+		if (copy_from_user(&notify, (void __user *)ioctl_param,
+				   sizeof(struct acrn_ioreq_notify)))
+			return -EFAULT;
+		ret = acrn_ioreq_request_default_complete(vm, notify.vcpu);
+		break;
+	case ACRN_IOCTL_CLEAR_VM_IOREQ:
+		acrn_ioreq_request_clear(vm);
+		break;
 	default:
 		dev_dbg(acrn_dev.this_device, "Unknown IOCTL 0x%x!\n", cmd);
 		ret = -ENOTTY;
@@ -168,14 +192,18 @@ static int __init hsm_init(void)
 		return -EPERM;
 
 	ret = misc_register(&acrn_dev);
-	if (ret)
+	if (ret) {
 		pr_err("Create misc dev failed!\n");
+		return ret;
+	}
 
-	return ret;
+	acrn_ioreq_intr_setup();
+	return 0;
 }
 
 static void __exit hsm_exit(void)
 {
+	acrn_ioreq_intr_remove();
 	misc_deregister(&acrn_dev);
 }
 module_init(hsm_init);
drivers/virt/acrn/hypercall.h +28 -0
index a1a70a071713..5eba29e3ed38 100644
@@ -21,6 +21,10 @@
 #define HC_RESET_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x05)
 #define HC_SET_VCPU_REGS		_HC_ID(HC_ID, HC_ID_VM_BASE + 0x06)
 
+#define HC_ID_IOREQ_BASE		0x30UL
+#define HC_SET_IOREQ_BUFFER		_HC_ID(HC_ID, HC_ID_IOREQ_BASE + 0x00)
+#define HC_NOTIFY_REQUEST_FINISH	_HC_ID(HC_ID, HC_ID_IOREQ_BASE + 0x01)
+
 #define HC_ID_MEM_BASE			0x40UL
 #define HC_VM_SET_MEMORY_REGIONS	_HC_ID(HC_ID, HC_ID_MEM_BASE + 0x02)
 
@@ -91,6 +95,30 @@ static inline long hcall_set_vcpu_regs(u64 vmid, u64 regs_state)
 	return acrn_hypercall2(HC_SET_VCPU_REGS, vmid, regs_state);
 }
 
+/**
+ * hcall_set_ioreq_buffer() - Set up the shared buffer for I/O Requests.
+ * @vmid:	User VM ID
+ * @buffer:	Service VM GPA of the shared buffer
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_set_ioreq_buffer(u64 vmid, u64 buffer)
+{
+	return acrn_hypercall2(HC_SET_IOREQ_BUFFER, vmid, buffer);
+}
+
+/**
+ * hcall_notify_req_finish() - Notify ACRN Hypervisor of I/O request completion.
+ * @vmid:	User VM ID
+ * @vcpu:	The vCPU which initiated the I/O request
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_notify_req_finish(u64 vmid, u64 vcpu)
+{
+	return acrn_hypercall2(HC_NOTIFY_REQUEST_FINISH, vmid, vcpu);
+}
+
 /**
  * hcall_set_memory_regions() - Inform the hypervisor to set up EPT mappings
  * @regions_pa:	Service VM GPA of &struct vm_memory_region_batch
drivers/virt/acrn/ioreq.c new file +509 -0

509 lines not on this page · raw message

drivers/virt/acrn/vm.c +20 -7
index ff5df7acb551..4ee1a99df4b7 100644
@@ -15,9 +15,12 @@
 #include "acrn_drv.h"
 
 /* List of VMs */
-static LIST_HEAD(acrn_vm_list);
-/* To protect acrn_vm_list */
-static DEFINE_MUTEX(acrn_vm_list_lock);
+LIST_HEAD(acrn_vm_list);
+/*
+ * acrn_vm_list is read in a tasklet which dispatch I/O requests and is wrote
+ * in VM creation ioctl. Use the rwlock mechanism to protect it.
+ */
+DEFINE_RWLOCK(acrn_vm_list_lock);
 
 struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
 			       struct acrn_vm_creation *vm_param)
@@ -32,12 +35,20 @@ struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
 	}
 
 	mutex_init(&vm->regions_mapping_lock);
+	INIT_LIST_HEAD(&vm->ioreq_clients);
+	spin_lock_init(&vm->ioreq_clients_lock);
 	vm->vmid = vm_param->vmid;
 	vm->vcpu_num = vm_param->vcpu_num;
 
-	mutex_lock(&acrn_vm_list_lock);
+	if (acrn_ioreq_init(vm, vm_param->ioreq_buf) < 0) {
+		hcall_destroy_vm(vm_param->vmid);
+		vm->vmid = ACRN_INVALID_VMID;
+		return NULL;
+	}
+
+	write_lock_bh(&acrn_vm_list_lock);
 	list_add(&vm->list, &acrn_vm_list);
-	mutex_unlock(&acrn_vm_list_lock);
+	write_unlock_bh(&acrn_vm_list_lock);
 
 	dev_dbg(acrn_dev.this_device, "VM %u created.\n", vm->vmid);
 	return vm;
@@ -52,9 +63,11 @@ int acrn_vm_destroy(struct acrn_vm *vm)
 		return 0;
 
 	/* Remove from global VM list */
-	mutex_lock(&acrn_vm_list_lock);
+	write_lock_bh(&acrn_vm_list_lock);
 	list_del_init(&vm->list);
-	mutex_unlock(&acrn_vm_list_lock);
+	write_unlock_bh(&acrn_vm_list_lock);
+
+	acrn_ioreq_deinit(vm);
 
 	ret = hcall_destroy_vm(vm->vmid);
 	if (ret < 0) {
include/uapi/linux/acrn.h +134 -0

146 lines not on this page · raw message

Signature · 1 line
2.28.0
#51 lore raw

PATCH v5 10/17 virt: acrn: Introduce PCI configuration space PIO accesses combiner

In reply to [email protected]

From: Shuo Liu <[email protected]>

A User VM can access its virtual PCI configuration spaces via port IO approach, which has two following steps:
1) writes address into port 0xCF8
2) put/get data in/from port 0xCFC

To distribute a complete PCI configuration space access one time, HSM need to combine such two accesses together.

Combine two paired PIO I/O requests into one PCI I/O request and continue the I/O request distribution.

---
 drivers/virt/acrn/acrn_drv.h |  2 +
 drivers/virt/acrn/ioreq.c    | 76 ++++++++++++++++++++++++++++++++++++
 include/uapi/linux/acrn.h    | 15 +++++++
 3 files changed, 93 insertions(+)
drivers/virt/acrn/acrn_drv.h +2 -0
index cf9143cf760d..97d2aab8b70a 100644
@@ -156,6 +156,7 @@ extern rwlock_t acrn_vm_list_lock;
  * @default_client:		The default I/O request client
  * @ioreq_buf:			I/O request shared buffer
  * @ioreq_page:			The page of the I/O request shared buffer
+ * @pci_conf_addr:		Address of a PCI configuration access emulation
  */
 struct acrn_vm {
 	struct list_head		list;
@@ -170,6 +171,7 @@ struct acrn_vm {
 	struct acrn_ioreq_client	*default_client;
 	struct acrn_io_request_buffer	*ioreq_buf;
 	struct page			*ioreq_page;
+	u32				pci_conf_addr;
 };
 
 struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
drivers/virt/acrn/ioreq.c +76 -0

88 lines not on this page · raw message

include/uapi/linux/acrn.h +15 -0
index 8eb687f1482c..31cf0fd73bcc 100644
@@ -20,6 +20,7 @@
 
 #define ACRN_IOREQ_TYPE_PORTIO		0
 #define ACRN_IOREQ_TYPE_MMIO		1
+#define ACRN_IOREQ_TYPE_PCICFG		2
 
 #define ACRN_IOREQ_DIR_READ		0
 #define ACRN_IOREQ_DIR_WRITE		1
@@ -40,6 +41,18 @@ struct acrn_pio_request {
 	__u32	value;
 } __attribute__((aligned(8)));
 
+/* Need keep same header fields with pio_request */
+struct acrn_pci_request {
+	__u32	direction;
+	__u32	reserved[3];
+	__u64	size;
+	__u32	value;
+	__u32	bus;
+	__u32	dev;
+	__u32	func;
+	__u32	reg;
+} __attribute__((aligned(8)));
+
 /**
  * struct acrn_io_request - 256-byte ACRN I/O request
  * @type:		Type of this request (ACRN_IOREQ_TYPE_*).
@@ -48,6 +61,7 @@ struct acrn_pio_request {
  * @reserved0:		Reserved fields.
  * @reqs:		Union of different types of request. Byte offset: 64.
  * @reqs.pio_request:	PIO request data of the I/O request.
+ * @reqs.pci_request:	PCI configuration space request data of the I/O request.
  * @reqs.mmio_request:	MMIO request data of the I/O request.
  * @reqs.data:		Raw data of the I/O request.
  * @reserved1:		Reserved fields.
@@ -107,6 +121,7 @@ struct acrn_io_request {
 	__u32	reserved0[14];
 	union {
 		struct acrn_pio_request		pio_request;
+		struct acrn_pci_request		pci_request;
 		struct acrn_mmio_request	mmio_request;
 		__u64				data[8];
 	} reqs;
Signature · 1 line
2.28.0
#52 lore raw

PATCH v5 11/17 virt: acrn: Introduce interfaces for PCI device passthrough

In reply to [email protected]

From: Shuo Liu <[email protected]>

PCI device passthrough enables an OS in a virtual machine to directly access a PCI device in the host. It promises almost the native performance, which is required in performance-critical scenarios of ACRN.

HSM provides the following ioctls:
- Assign - ACRN_IOCTL_ASSIGN_PCIDEV
Pass data struct acrn_pcidev from userspace to the hypervisor, and inform the hypervisor to assign a PCI device to a User VM.

- De-assign - ACRN_IOCTL_DEASSIGN_PCIDEV
Pass data struct acrn_pcidev from userspace to the hypervisor, and inform the hypervisor to de-assign a PCI device from a User VM.

- Set a interrupt of a passthrough device - ACRN_IOCTL_SET_PTDEV_INTR Pass data struct acrn_ptdev_irq from userspace to the hypervisor, and inform the hypervisor to map a INTx interrupt of passthrough device of User VM.

- Reset passthrough device interrupt - ACRN_IOCTL_RESET_PTDEV_INTR Pass data struct acrn_ptdev_irq from userspace to the hypervisor, and inform the hypervisor to unmap a INTx interrupt of passthrough device of User VM.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 drivers/virt/acrn/hsm.c       | 50 +++++++++++++++++++++++++++
 drivers/virt/acrn/hypercall.h | 54 ++++++++++++++++++++++++++++++
 include/uapi/linux/acrn.h     | 63 +++++++++++++++++++++++++++++++++++
 3 files changed, 167 insertions(+)
drivers/virt/acrn/hsm.c +50 -0
index 15cfb4fb5f1b..e72837adba21 100644
@@ -49,7 +49,9 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 	struct acrn_vm_creation *vm_param;
 	struct acrn_vcpu_regs *cpu_regs;
 	struct acrn_ioreq_notify notify;
+	struct acrn_ptdev_irq *irq_info;
 	struct acrn_vm_memmap memmap;
+	struct acrn_pcidev *pcidev;
 	int ret = 0;
 
 	if (vm->vmid == ACRN_INVALID_VMID && cmd != ACRN_IOCTL_CREATE_VM) {
@@ -128,6 +130,54 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 
 		ret = acrn_vm_memseg_unmap(vm, &memmap);
 		break;
+	case ACRN_IOCTL_ASSIGN_PCIDEV:
+		pcidev = memdup_user((void __user *)ioctl_param,
+				     sizeof(struct acrn_pcidev));
+		if (IS_ERR(pcidev))
+			return PTR_ERR(pcidev);
+
+		ret = hcall_assign_pcidev(vm->vmid, virt_to_phys(pcidev));
+		if (ret < 0)
+			dev_dbg(acrn_dev.this_device,
+				"Failed to assign pci device!\n");
+		kfree(pcidev);
+		break;
+	case ACRN_IOCTL_DEASSIGN_PCIDEV:
+		pcidev = memdup_user((void __user *)ioctl_param,
+				     sizeof(struct acrn_pcidev));
+		if (IS_ERR(pcidev))
+			return PTR_ERR(pcidev);
+
+		ret = hcall_deassign_pcidev(vm->vmid, virt_to_phys(pcidev));
+		if (ret < 0)
+			dev_dbg(acrn_dev.this_device,
+				"Failed to deassign pci device!\n");
+		kfree(pcidev);
+		break;
+	case ACRN_IOCTL_SET_PTDEV_INTR:
+		irq_info = memdup_user((void __user *)ioctl_param,
+				       sizeof(struct acrn_ptdev_irq));
+		if (IS_ERR(irq_info))
+			return PTR_ERR(irq_info);
+
+		ret = hcall_set_ptdev_intr(vm->vmid, virt_to_phys(irq_info));
+		if (ret < 0)
+			dev_dbg(acrn_dev.this_device,
+				"Failed to configure intr for ptdev!\n");
+		kfree(irq_info);
+		break;
+	case ACRN_IOCTL_RESET_PTDEV_INTR:
+		irq_info = memdup_user((void __user *)ioctl_param,
+				       sizeof(struct acrn_ptdev_irq));
+		if (IS_ERR(irq_info))
+			return PTR_ERR(irq_info);
+
+		ret = hcall_reset_ptdev_intr(vm->vmid, virt_to_phys(irq_info));
+		if (ret < 0)
+			dev_dbg(acrn_dev.this_device,
+				"Failed to reset intr for ptdev!\n");
+		kfree(irq_info);
+		break;
 	case ACRN_IOCTL_CREATE_IOREQ_CLIENT:
 		if (vm->default_client)
 			return -EEXIST;
drivers/virt/acrn/hypercall.h +54 -0

64 lines not on this page · raw message

include/uapi/linux/acrn.h +63 -0

73 lines not on this page · raw message

Signature · 1 line
2.28.0
#53 lore raw

PATCH v5 12/17 virt: acrn: Introduce interrupt injection interfaces

In reply to [email protected]

From: Shuo Liu <[email protected]>

ACRN userspace need to inject virtual interrupts into a User VM in devices emulation.

HSM needs provide interfaces to do so.

Introduce following interrupt injection interfaces:

ioctl ACRN_IOCTL_SET_IRQLINE:
Pass data from userspace to the hypervisor, and inform the hypervisor to inject a virtual IOAPIC GSI interrupt to a User VM.

ioctl ACRN_IOCTL_INJECT_MSI:
Pass data struct acrn_msi_entry from userspace to the hypervisor, and inform the hypervisor to inject a virtual MSI to a User VM.

ioctl ACRN_IOCTL_VM_INTR_MONITOR:
Set a 4-Kbyte aligned shared page for statistics information of interrupts of a User VM.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 drivers/virt/acrn/acrn_drv.h  |  4 ++++
 drivers/virt/acrn/hsm.c       | 39 +++++++++++++++++++++++++++++++++
 drivers/virt/acrn/hypercall.h | 41 +++++++++++++++++++++++++++++++++++
 drivers/virt/acrn/vm.c        | 36 ++++++++++++++++++++++++++++++
 include/uapi/linux/acrn.h     | 17 +++++++++++++++
 5 files changed, 137 insertions(+)
drivers/virt/acrn/acrn_drv.h +4 -0
index 97d2aab8b70a..701c83319115 100644
@@ -157,6 +157,7 @@ extern rwlock_t acrn_vm_list_lock;
  * @ioreq_buf:			I/O request shared buffer
  * @ioreq_page:			The page of the I/O request shared buffer
  * @pci_conf_addr:		Address of a PCI configuration access emulation
+ * @monitor_page:		Page of interrupt statistics of User VM
  */
 struct acrn_vm {
 	struct list_head		list;
@@ -172,6 +173,7 @@ struct acrn_vm {
 	struct acrn_io_request_buffer	*ioreq_buf;
 	struct page			*ioreq_page;
 	u32				pci_conf_addr;
+	struct page			*monitor_page;
 };
 
 struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
@@ -198,4 +200,6 @@ struct acrn_ioreq_client *acrn_ioreq_client_create(struct acrn_vm *vm,
 						   const char *name);
 void acrn_ioreq_client_destroy(struct acrn_ioreq_client *client);
 
+int acrn_msi_inject(struct acrn_vm *vm, u64 msi_addr, u64 msi_data);
+
 #endif /* __ACRN_HSM_DRV_H */
drivers/virt/acrn/hsm.c +39 -0
index e72837adba21..5c38aa841b63 100644
@@ -51,7 +51,9 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 	struct acrn_ioreq_notify notify;
 	struct acrn_ptdev_irq *irq_info;
 	struct acrn_vm_memmap memmap;
+	struct acrn_msi_entry *msi;
 	struct acrn_pcidev *pcidev;
+	struct page *page;
 	int ret = 0;
 
 	if (vm->vmid == ACRN_INVALID_VMID && cmd != ACRN_IOCTL_CREATE_VM) {
@@ -178,6 +180,43 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 				"Failed to reset intr for ptdev!\n");
 		kfree(irq_info);
 		break;
+	case ACRN_IOCTL_SET_IRQLINE:
+		ret = hcall_set_irqline(vm->vmid, ioctl_param);
+		if (ret < 0)
+			dev_dbg(acrn_dev.this_device,
+				"Failed to set interrupt line!\n");
+		break;
+	case ACRN_IOCTL_INJECT_MSI:
+		msi = memdup_user((void __user *)ioctl_param,
+				  sizeof(struct acrn_msi_entry));
+		if (IS_ERR(msi))
+			return PTR_ERR(msi);
+
+		ret = hcall_inject_msi(vm->vmid, virt_to_phys(msi));
+		if (ret < 0)
+			dev_dbg(acrn_dev.this_device,
+				"Failed to inject MSI!\n");
+		kfree(msi);
+		break;
+	case ACRN_IOCTL_VM_INTR_MONITOR:
+		ret = get_user_pages_fast(ioctl_param, 1, FOLL_WRITE, &page);
+		if (unlikely(ret != 1)) {
+			dev_dbg(acrn_dev.this_device,
+				"Failed to pin intr hdr buffer!\n");
+			return -EFAULT;
+		}
+
+		ret = hcall_vm_intr_monitor(vm->vmid, page_to_phys(page));
+		if (ret < 0) {
+			put_page(page);
+			dev_dbg(acrn_dev.this_device,
+				"Failed to monitor intr data!\n");
+			return ret;
+		}
+		if (vm->monitor_page)
+			put_page(vm->monitor_page);
+		vm->monitor_page = page;
+		break;
 	case ACRN_IOCTL_CREATE_IOREQ_CLIENT:
 		if (vm->default_client)
 			return -EEXIST;
drivers/virt/acrn/hypercall.h +41 -0

53 lines not on this page · raw message

drivers/virt/acrn/vm.c +36 -0
index 4ee1a99df4b7..38304aeef181 100644
@@ -68,6 +68,10 @@ int acrn_vm_destroy(struct acrn_vm *vm)
 	write_unlock_bh(&acrn_vm_list_lock);
 
 	acrn_ioreq_deinit(vm);
+	if (vm->monitor_page) {
+		put_page(vm->monitor_page);
+		vm->monitor_page = NULL;
+	}
 
 	ret = hcall_destroy_vm(vm->vmid);
 	if (ret < 0) {
@@ -83,3 +87,35 @@ int acrn_vm_destroy(struct acrn_vm *vm)
 	vm->vmid = ACRN_INVALID_VMID;
 	return 0;
 }
+
+/**
+ * acrn_inject_msi() - Inject a MSI interrupt into a User VM
+ * @vm:		User VM
+ * @msi_addr:	The MSI address
+ * @msi_data:	The MSI data
+ *
+ * Return: 0 on success, <0 on error
+ */
+int acrn_msi_inject(struct acrn_vm *vm, u64 msi_addr, u64 msi_data)
+{
+	struct acrn_msi_entry *msi;
+	int ret;
+
+	/* might be used in interrupt context, so use GFP_ATOMIC */
+	msi = kzalloc(sizeof(*msi), GFP_ATOMIC);
+	if (!msi)
+		return -ENOMEM;
+
+	/*
+	 * msi_addr: addr[19:12] with dest vcpu id
+	 * msi_data: data[7:0] with vector
+	 */
+	msi->msi_addr = msi_addr;
+	msi->msi_data = msi_data;
+	ret = hcall_inject_msi(vm->vmid, virt_to_phys(msi));
+	if (ret < 0)
+		dev_err(acrn_dev.this_device,
+			"Failed to inject MSI to VM %u!\n", vm->vmid);
+	kfree(msi);
+	return ret;
+}
include/uapi/linux/acrn.h +17 -0
index 893389babbcb..7764459e130c 100644
@@ -343,6 +343,16 @@ struct acrn_pcidev {
 	__u32	reserved[6];
 } __attribute__((aligned(8)));
 
+/**
+ * struct acrn_msi_entry - Info for injecting a MSI interrupt to a VM
+ * @msi_addr:	MSI addr[19:12] with dest vCPU ID
+ * @msi_data:	MSI data[7:0] with vector
+ */
+struct acrn_msi_entry {
+	__u64	msi_addr;
+	__u64	msi_data;
+};
+
 /* The ioctl type, documented in ioctl-number.rst */
 #define ACRN_IOCTL_TYPE			0xA2
 
@@ -362,6 +372,13 @@ struct acrn_pcidev {
 #define ACRN_IOCTL_SET_VCPU_REGS	\
 	_IOW(ACRN_IOCTL_TYPE, 0x16, struct acrn_vcpu_regs)
 
+#define ACRN_IOCTL_INJECT_MSI		\
+	_IOW(ACRN_IOCTL_TYPE, 0x23, struct acrn_msi_entry)
+#define ACRN_IOCTL_VM_INTR_MONITOR	\
+	_IOW(ACRN_IOCTL_TYPE, 0x24, unsigned long)
+#define ACRN_IOCTL_SET_IRQLINE		\
+	_IOW(ACRN_IOCTL_TYPE, 0x25, __u64)
+
 #define ACRN_IOCTL_NOTIFY_REQUEST_FINISH \
 	_IOW(ACRN_IOCTL_TYPE, 0x31, struct acrn_ioreq_notify)
 #define ACRN_IOCTL_CREATE_IOREQ_CLIENT	\
Signature · 1 line
2.28.0
#54 lore raw

PATCH v5 13/17 virt: acrn: Introduce interfaces to query C-states and P-states allowed by hypervisor

In reply to [email protected]

From: Shuo Liu <[email protected]>

The C-states and P-states data are used to support CPU power management. The hypervisor controls C-states and P-states for a User VM.

ACRN userspace need to query the data from the hypervisor to build ACPI tables for a User VM.

HSM provides ioctls for ACRN userspace to query C-states and P-states data obtained from the hypervisor.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 drivers/virt/acrn/hsm.c       | 69 +++++++++++++++++++++++++++++++++++
 drivers/virt/acrn/hypercall.h | 12 ++++++
 include/uapi/linux/acrn.h     | 35 ++++++++++++++++++
 3 files changed, 116 insertions(+)
drivers/virt/acrn/hsm.c +69 -0
index 5c38aa841b63..9565b4d64ab7 100644
@@ -38,6 +38,67 @@ static int acrn_dev_open(struct inode *inode, struct file *filp)
 	return 0;
 }
 
+static int pmcmd_ioctl(u64 cmd, void __user *uptr)
+{
+	struct acrn_pstate_data *px_data;
+	struct acrn_cstate_data *cx_data;
+	u64 *pm_info;
+	int ret = 0;
+
+	switch (cmd & PMCMD_TYPE_MASK) {
+	case ACRN_PMCMD_GET_PX_CNT:
+	case ACRN_PMCMD_GET_CX_CNT:
+		pm_info = kmalloc(sizeof(u64), GFP_KERNEL);
+		if (!pm_info)
+			return -ENOMEM;
+
+		ret = hcall_get_cpu_state(cmd, virt_to_phys(pm_info));
+		if (ret < 0) {
+			kfree(pm_info);
+			break;
+		}
+
+		if (copy_to_user(uptr, pm_info, sizeof(u64)))
+			ret = -EFAULT;
+		kfree(pm_info);
+		break;
+	case ACRN_PMCMD_GET_PX_DATA:
+		px_data = kmalloc(sizeof(*px_data), GFP_KERNEL);
+		if (!px_data)
+			return -ENOMEM;
+
+		ret = hcall_get_cpu_state(cmd, virt_to_phys(px_data));
+		if (ret < 0) {
+			kfree(px_data);
+			break;
+		}
+
+		if (copy_to_user(uptr, px_data, sizeof(*px_data)))
+			ret = -EFAULT;
+		kfree(px_data);
+		break;
+	case ACRN_PMCMD_GET_CX_DATA:
+		cx_data = kmalloc(sizeof(*cx_data), GFP_KERNEL);
+		if (!cx_data)
+			return -ENOMEM;
+
+		ret = hcall_get_cpu_state(cmd, virt_to_phys(cx_data));
+		if (ret < 0) {
+			kfree(cx_data);
+			break;
+		}
+
+		if (copy_to_user(uptr, cx_data, sizeof(*cx_data)))
+			ret = -EFAULT;
+		kfree(cx_data);
+		break;
+	default:
+		break;
+	}
+
+	return ret;
+}
+
 /*
  * HSM relies on hypercall layer of the ACRN hypervisor to do the
  * sanity check against the input parameters.
@@ -54,6 +115,7 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 	struct acrn_msi_entry *msi;
 	struct acrn_pcidev *pcidev;
 	struct page *page;
+	u64 cstate_cmd;
 	int ret = 0;
 
 	if (vm->vmid == ACRN_INVALID_VMID && cmd != ACRN_IOCTL_CREATE_VM) {
@@ -240,6 +302,13 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 	case ACRN_IOCTL_CLEAR_VM_IOREQ:
 		acrn_ioreq_request_clear(vm);
 		break;
+	case ACRN_IOCTL_PM_GET_CPU_STATE:
+		if (copy_from_user(&cstate_cmd, (void *)ioctl_param,
+				   sizeof(cstate_cmd)))
+			return -EFAULT;
+
+		ret = pmcmd_ioctl(cstate_cmd, (void __user *)ioctl_param);
+		break;
 	default:
 		dev_dbg(acrn_dev.this_device, "Unknown IOCTL 0x%x!\n", cmd);
 		ret = -ENOTTY;
drivers/virt/acrn/hypercall.h +12 -0
index a8813397a3fe..e640632366f0 100644
@@ -39,6 +39,9 @@
 #define HC_ASSIGN_PCIDEV		_HC_ID(HC_ID, HC_ID_PCI_BASE + 0x05)
 #define HC_DEASSIGN_PCIDEV		_HC_ID(HC_ID, HC_ID_PCI_BASE + 0x06)
 
+#define HC_ID_PM_BASE			0x80UL
+#define HC_PM_GET_CPU_STATE		_HC_ID(HC_ID, HC_ID_PM_BASE + 0x00)
+
 /**
  * hcall_create_vm() - Create a User VM
  * @vminfo:	Service VM GPA of info of User VM creation
@@ -225,4 +228,13 @@ static inline long hcall_reset_ptdev_intr(u64 vmid, u64 irq)
 	return acrn_hypercall2(HC_RESET_PTDEV_INTR, vmid, irq);
 }
 
+/*
+ * hcall_get_cpu_state() - Get P-states and C-states info from the hypervisor
+ * @state:	Service VM GPA of buffer of P-states and C-states
+ */
+static inline long hcall_get_cpu_state(u64 cmd, u64 state)
+{
+	return acrn_hypercall2(HC_PM_GET_CPU_STATE, cmd, state);
+}
+
 #endif /* __ACRN_HSM_HYPERCALL_H */
include/uapi/linux/acrn.h +35 -0
index 7764459e130c..9fed7209a8ef 100644
@@ -353,6 +353,38 @@ struct acrn_msi_entry {
 	__u64	msi_data;
 };
 
+struct acrn_acpi_generic_address {
+	__u8	space_id;
+	__u8	bit_width;
+	__u8	bit_offset;
+	__u8	access_size;
+	__u64	address;
+} __attribute__ ((__packed__));
+
+struct acrn_cstate_data {
+	struct acrn_acpi_generic_address	cx_reg;
+	__u8					type;
+	__u32					latency;
+	__u64					power;
+} __attribute__((aligned(8)));
+
+struct acrn_pstate_data {
+	__u64	core_frequency;
+	__u64	power;
+	__u64	transition_latency;
+	__u64	bus_master_latency;
+	__u64	control;
+	__u64	status;
+} __attribute__((aligned(8)));
+
+#define PMCMD_TYPE_MASK		0x000000ff
+enum acrn_pm_cmd_type {
+	ACRN_PMCMD_GET_PX_CNT,
+	ACRN_PMCMD_GET_PX_DATA,
+	ACRN_PMCMD_GET_CX_CNT,
+	ACRN_PMCMD_GET_CX_DATA,
+};
+
 /* The ioctl type, documented in ioctl-number.rst */
 #define ACRN_IOCTL_TYPE			0xA2
 
@@ -404,4 +436,7 @@ struct acrn_msi_entry {
 #define ACRN_IOCTL_DEASSIGN_PCIDEV	\
 	_IOW(ACRN_IOCTL_TYPE, 0x56, struct acrn_pcidev)
 
+#define ACRN_IOCTL_PM_GET_CPU_STATE	\
+	_IOWR(ACRN_IOCTL_TYPE, 0x60, __u64)
+
 #endif /* _UAPI_ACRN_H */
Signature · 1 line
2.28.0
#55 lore raw

PATCH v5 14/17 virt: acrn: Introduce I/O ranges operation interfaces

In reply to [email protected]

From: Shuo Liu <[email protected]>

An I/O request of a User VM, which is constructed by hypervisor, is distributed by the ACRN Hypervisor Service Module to an I/O client corresponding to the address range of the I/O request.

I/O client maintains a list of address ranges. Introduce acrn_ioreq_range_{add,del}() to manage these address ranges.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 drivers/virt/acrn/acrn_drv.h |  4 +++
 drivers/virt/acrn/ioreq.c    | 60 ++++++++++++++++++++++++++++++++++++
 2 files changed, 64 insertions(+)
drivers/virt/acrn/acrn_drv.h +4 -0
index 701c83319115..5b824fa1ee57 100644
@@ -199,6 +199,10 @@ struct acrn_ioreq_client *acrn_ioreq_client_create(struct acrn_vm *vm,
 						   void *data, bool is_default,
 						   const char *name);
 void acrn_ioreq_client_destroy(struct acrn_ioreq_client *client);
+int acrn_ioreq_range_add(struct acrn_ioreq_client *client,
+			 u32 type, u64 start, u64 end);
+void acrn_ioreq_range_del(struct acrn_ioreq_client *client,
+			  u32 type, u64 start, u64 end);
 
 int acrn_msi_inject(struct acrn_vm *vm, u64 msi_addr, u64 msi_data);
 
drivers/virt/acrn/ioreq.c +60 -0

66 lines not on this page · raw message

Signature · 1 line
2.28.0
#56 lore raw

PATCH v5 15/17 virt: acrn: Introduce ioeventfd

In reply to [email protected]

From: Shuo Liu <[email protected]>

ioeventfd is a mechanism to register PIO/MMIO regions to trigger an eventfd signal when written to by a User VM. ACRN userspace can register any arbitrary I/O address with a corresponding eventfd and then pass the eventfd to a specific end-point of interest for handling.

Vhost is a kernel-level virtio server which uses eventfd for signalling. To support vhost on ACRN, ioeventfd is introduced in HSM.

A new I/O client dedicated to ioeventfd is associated with a User VM during VM creation. HSM provides ioctls to associate an I/O region with a eventfd. The I/O client signals a eventfd once its corresponding I/O region is matched with an I/O request.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 drivers/virt/acrn/Kconfig     |   1 +
 drivers/virt/acrn/Makefile    |   2 +-
 drivers/virt/acrn/acrn_drv.h  |  10 ++
 drivers/virt/acrn/hsm.c       |   8 +
 drivers/virt/acrn/ioeventfd.c | 273 ++++++++++++++++++++++++++++++++++
 drivers/virt/acrn/vm.c        |   2 +
 include/uapi/linux/acrn.h     |  29 ++++
 7 files changed, 324 insertions(+), 1 deletion(-)
 create mode 100644 drivers/virt/acrn/ioeventfd.c
drivers/virt/acrn/Kconfig +1 -0
index 36c80378c30c..3e1a61c9d8d8 100644
@@ -2,6 +2,7 @@
 config ACRN_HSM
 	tristate "ACRN Hypervisor Service Module"
 	depends on ACRN_GUEST
+	select EVENTFD
 	help
 	  ACRN Hypervisor Service Module (HSM) is a kernel module which
 	  communicates with ACRN userspace through ioctls and talks to
drivers/virt/acrn/Makefile +1 -1
index 21721cbf6a80..755b583b32ca 100644
@@ -1,3 +1,3 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_ACRN_HSM)	:= acrn.o
-acrn-y := hsm.o vm.o mm.o ioreq.o
+acrn-y := hsm.o vm.o mm.o ioreq.o ioeventfd.o
drivers/virt/acrn/acrn_drv.h +10 -0
index 5b824fa1ee57..c66c620b9f10 100644
@@ -158,6 +158,9 @@ extern rwlock_t acrn_vm_list_lock;
  * @ioreq_page:			The page of the I/O request shared buffer
  * @pci_conf_addr:		Address of a PCI configuration access emulation
  * @monitor_page:		Page of interrupt statistics of User VM
+ * @ioeventfds_lock:		Lock to protect ioeventfds list
+ * @ioeventfds:			List to link all hsm_ioeventfd
+ * @ioeventfd_client:		I/O client for ioeventfds of the VM
  */
 struct acrn_vm {
 	struct list_head		list;
@@ -174,6 +177,9 @@ struct acrn_vm {
 	struct page			*ioreq_page;
 	u32				pci_conf_addr;
 	struct page			*monitor_page;
+	struct mutex			ioeventfds_lock;
+	struct list_head		ioeventfds;
+	struct acrn_ioreq_client	*ioeventfd_client;
 };
 
 struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
@@ -206,4 +212,8 @@ void acrn_ioreq_range_del(struct acrn_ioreq_client *client,
 
 int acrn_msi_inject(struct acrn_vm *vm, u64 msi_addr, u64 msi_data);
 
+int acrn_ioeventfd_init(struct acrn_vm *vm);
+int acrn_ioeventfd_config(struct acrn_vm *vm, struct acrn_ioeventfd *args);
+void acrn_ioeventfd_deinit(struct acrn_vm *vm);
+
 #endif /* __ACRN_HSM_DRV_H */
drivers/virt/acrn/hsm.c +8 -0
index 9565b4d64ab7..16f0280148f3 100644
@@ -111,6 +111,7 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 	struct acrn_vcpu_regs *cpu_regs;
 	struct acrn_ioreq_notify notify;
 	struct acrn_ptdev_irq *irq_info;
+	struct acrn_ioeventfd ioeventfd;
 	struct acrn_vm_memmap memmap;
 	struct acrn_msi_entry *msi;
 	struct acrn_pcidev *pcidev;
@@ -309,6 +310,13 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 
 		ret = pmcmd_ioctl(cstate_cmd, (void __user *)ioctl_param);
 		break;
+	case ACRN_IOCTL_IOEVENTFD:
+		if (copy_from_user(&ioeventfd, (void __user *)ioctl_param,
+				   sizeof(ioeventfd)))
+			return -EFAULT;
+
+		ret = acrn_ioeventfd_config(vm, &ioeventfd);
+		break;
 	default:
 		dev_dbg(acrn_dev.this_device, "Unknown IOCTL 0x%x!\n", cmd);
 		ret = -ENOTTY;
drivers/virt/acrn/ioeventfd.c new file +273 -0

273 lines not on this page · raw message

drivers/virt/acrn/vm.c +2 -0
index 38304aeef181..3c671b03b273 100644
@@ -50,6 +50,7 @@ struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
 	list_add(&vm->list, &acrn_vm_list);
 	write_unlock_bh(&acrn_vm_list_lock);
 
+	acrn_ioeventfd_init(vm);
 	dev_dbg(acrn_dev.this_device, "VM %u created.\n", vm->vmid);
 	return vm;
 }
@@ -67,6 +68,7 @@ int acrn_vm_destroy(struct acrn_vm *vm)
 	list_del_init(&vm->list);
 	write_unlock_bh(&acrn_vm_list_lock);
 
+	acrn_ioeventfd_deinit(vm);
 	acrn_ioreq_deinit(vm);
 	if (vm->monitor_page) {
 		put_page(vm->monitor_page);
include/uapi/linux/acrn.h +29 -0
index 9fed7209a8ef..7a99124c7d4d 100644
@@ -385,6 +385,32 @@ enum acrn_pm_cmd_type {
 	ACRN_PMCMD_GET_CX_DATA,
 };
 
+#define ACRN_IOEVENTFD_FLAG_PIO		0x01
+#define ACRN_IOEVENTFD_FLAG_DATAMATCH	0x02
+#define ACRN_IOEVENTFD_FLAG_DEASSIGN	0x04
+/**
+ * struct acrn_ioeventfd - Data to operate a &struct hsm_ioeventfd
+ * @fd:		The fd of eventfd associated with a hsm_ioeventfd
+ * @flags:	Logical-OR of ACRN_IOEVENTFD_FLAG_*
+ * @addr:	The start address of IO range of ioeventfd
+ * @len:	The length of IO range of ioeventfd
+ * @reserved:	Reserved
+ * @data:	Data for data matching
+ *
+ * Without flag ACRN_IOEVENTFD_FLAG_DEASSIGN, ioctl ACRN_IOCTL_IOEVENTFD
+ * creates a &struct hsm_ioeventfd with properties originated from &struct
+ * acrn_ioeventfd. With flag ACRN_IOEVENTFD_FLAG_DEASSIGN, ioctl
+ * ACRN_IOCTL_IOEVENTFD destroys the &struct hsm_ioeventfd matching the fd.
+ */
+struct acrn_ioeventfd {
+	__u32	fd;
+	__u32	flags;
+	__u64	addr;
+	__u32	len;
+	__u32	reserved;
+	__u64	data;
+};
+
 /* The ioctl type, documented in ioctl-number.rst */
 #define ACRN_IOCTL_TYPE			0xA2
 
@@ -439,4 +465,7 @@ enum acrn_pm_cmd_type {
 #define ACRN_IOCTL_PM_GET_CPU_STATE	\
 	_IOWR(ACRN_IOCTL_TYPE, 0x60, __u64)
 
+#define ACRN_IOCTL_IOEVENTFD		\
+	_IOW(ACRN_IOCTL_TYPE, 0x70, struct acrn_ioeventfd)
+
 #endif /* _UAPI_ACRN_H */
Signature · 1 line
2.28.0
#57 lore raw

PATCH v5 16/17 virt: acrn: Introduce irqfd

In reply to [email protected]

From: Shuo Liu <[email protected]>

irqfd is a mechanism to inject a specific interrupt to a User VM using a decoupled eventfd mechanism.

Vhost is a kernel-level virtio server which uses eventfd for interrupt injection. To support vhost on ACRN, irqfd is introduced in HSM.

HSM provides ioctls to associate a virtual Message Signaled Interrupt (MSI) with an eventfd. The corresponding virtual MSI will be injected into a User VM once the eventfd got signal.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 drivers/virt/acrn/Makefile   |   2 +-
 drivers/virt/acrn/acrn_drv.h |  10 ++
 drivers/virt/acrn/hsm.c      |   7 ++
 drivers/virt/acrn/irqfd.c    | 235 +++++++++++++++++++++++++++++++++++
 drivers/virt/acrn/vm.c       |   3 +
 include/uapi/linux/acrn.h    |  15 +++
 6 files changed, 271 insertions(+), 1 deletion(-)
 create mode 100644 drivers/virt/acrn/irqfd.c
drivers/virt/acrn/Makefile +1 -1
index 755b583b32ca..08ce641dcfa1 100644
@@ -1,3 +1,3 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_ACRN_HSM)	:= acrn.o
-acrn-y := hsm.o vm.o mm.o ioreq.o ioeventfd.o
+acrn-y := hsm.o vm.o mm.o ioreq.o ioeventfd.o irqfd.o
drivers/virt/acrn/acrn_drv.h +10 -0
index c66c620b9f10..8354d0d5881c 100644
@@ -161,6 +161,9 @@ extern rwlock_t acrn_vm_list_lock;
  * @ioeventfds_lock:		Lock to protect ioeventfds list
  * @ioeventfds:			List to link all hsm_ioeventfd
  * @ioeventfd_client:		I/O client for ioeventfds of the VM
+ * @irqfds_lock:		Lock to protect irqfds list
+ * @irqfds:			List to link all hsm_irqfd
+ * @irqfd_wq:			Workqueue for irqfd async shutdown
  */
 struct acrn_vm {
 	struct list_head		list;
@@ -180,6 +183,9 @@ struct acrn_vm {
 	struct mutex			ioeventfds_lock;
 	struct list_head		ioeventfds;
 	struct acrn_ioreq_client	*ioeventfd_client;
+	struct mutex			irqfds_lock;
+	struct list_head		irqfds;
+	struct workqueue_struct		*irqfd_wq;
 };
 
 struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
@@ -216,4 +222,8 @@ int acrn_ioeventfd_init(struct acrn_vm *vm);
 int acrn_ioeventfd_config(struct acrn_vm *vm, struct acrn_ioeventfd *args);
 void acrn_ioeventfd_deinit(struct acrn_vm *vm);
 
+int acrn_irqfd_init(struct acrn_vm *vm);
+int acrn_irqfd_config(struct acrn_vm *vm, struct acrn_irqfd *args);
+void acrn_irqfd_deinit(struct acrn_vm *vm);
+
 #endif /* __ACRN_HSM_DRV_H */
drivers/virt/acrn/hsm.c +7 -0
index 16f0280148f3..c7290e177b1e 100644
@@ -115,6 +115,7 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 	struct acrn_vm_memmap memmap;
 	struct acrn_msi_entry *msi;
 	struct acrn_pcidev *pcidev;
+	struct acrn_irqfd irqfd;
 	struct page *page;
 	u64 cstate_cmd;
 	int ret = 0;
@@ -317,6 +318,12 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,
 
 		ret = acrn_ioeventfd_config(vm, &ioeventfd);
 		break;
+	case ACRN_IOCTL_IRQFD:
+		if (copy_from_user(&irqfd, (void __user *)ioctl_param,
+				   sizeof(irqfd)))
+			return -EFAULT;
+		ret = acrn_irqfd_config(vm, &irqfd);
+		break;
 	default:
 		dev_dbg(acrn_dev.this_device, "Unknown IOCTL 0x%x!\n", cmd);
 		ret = -ENOTTY;
drivers/virt/acrn/irqfd.c new file +235 -0

235 lines not on this page · raw message

drivers/virt/acrn/vm.c +3 -0
index 3c671b03b273..7f152a74b591 100644
@@ -51,6 +51,7 @@ struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
 	write_unlock_bh(&acrn_vm_list_lock);
 
 	acrn_ioeventfd_init(vm);
+	acrn_irqfd_init(vm);
 	dev_dbg(acrn_dev.this_device, "VM %u created.\n", vm->vmid);
 	return vm;
 }
@@ -69,7 +70,9 @@ int acrn_vm_destroy(struct acrn_vm *vm)
 	write_unlock_bh(&acrn_vm_list_lock);
 
 	acrn_ioeventfd_deinit(vm);
+	acrn_irqfd_deinit(vm);
 	acrn_ioreq_deinit(vm);
+
 	if (vm->monitor_page) {
 		put_page(vm->monitor_page);
 		vm->monitor_page = NULL;
include/uapi/linux/acrn.h +15 -0
index 7a99124c7d4d..75a687838a43 100644
@@ -411,6 +411,19 @@ struct acrn_ioeventfd {
 	__u64	data;
 };
 
+#define ACRN_IRQFD_FLAG_DEASSIGN	0x01
+/**
+ * struct acrn_irqfd - Data to operate a &struct hsm_irqfd
+ * @fd:		The fd of eventfd associated with a hsm_irqfd
+ * @flags:	Logical-OR of ACRN_IRQFD_FLAG_*
+ * @msi:	Info of MSI associated with the irqfd
+ */
+struct acrn_irqfd {
+	__s32			fd;
+	__u32			flags;
+	struct acrn_msi_entry	msi;
+};
+
 /* The ioctl type, documented in ioctl-number.rst */
 #define ACRN_IOCTL_TYPE			0xA2
 
@@ -467,5 +480,7 @@ struct acrn_ioeventfd {
 
 #define ACRN_IOCTL_IOEVENTFD		\
 	_IOW(ACRN_IOCTL_TYPE, 0x70, struct acrn_ioeventfd)
+#define ACRN_IOCTL_IRQFD		\
+	_IOW(ACRN_IOCTL_TYPE, 0x71, struct acrn_irqfd)
 
 #endif /* _UAPI_ACRN_H */
Signature · 1 line
2.28.0
#58 lore raw

PATCH v5 17/17 virt: acrn: Introduce an interface for Service VM to control vCPU

In reply to [email protected]

From: Shuo Liu <[email protected]>

ACRN supports partition mode to achieve real-time requirements. In partition mode, a CPU core can be dedicated to a vCPU of User VM. The local APIC of the dedicated CPU core can be passthrough to the User VM. The Service VM controls the assignment of the CPU cores.

Introduce an interface for the Service VM to remove the control of CPU core from hypervisor perspective so that the CPU core can be a dedicated CPU core of User VM.

Cc Zhi Wang <[email protected]>
Cc Zhenyu Wang <[email protected]>
Cc Yu Wang <[email protected]>
Cc Reinette Chatre <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
---
 drivers/virt/acrn/hsm.c       | 48 +++++++++++++++++++++++++++++++++++
 drivers/virt/acrn/hypercall.h | 14 ++++++++++
 2 files changed, 62 insertions(+)
drivers/virt/acrn/hsm.c +48 -0
index c7290e177b1e..8c3799cc0313 100644
@@ -9,6 +9,7 @@
  *	Yakui Zhao <[email protected]>
  */
 
+#include <linux/cpu.h>
 #include <linux/io.h>
 #include <linux/mm.h>
 #include <linux/module.h>
@@ -341,6 +342,52 @@ static int acrn_dev_release(struct inode *inode, struct file *filp)
 	return 0;
 }
 
+static ssize_t remove_cpu_store(struct device *dev,
+				struct device_attribute *attr,
+				const char *buf, size_t count)
+{
+	u64 cpu, lapicid;
+	int ret;
+
+	if (kstrtoull(buf, 0, &cpu) < 0)
+		return -EINVAL;
+
+	if (cpu >= num_possible_cpus() || cpu == 0 || !cpu_is_hotpluggable(cpu))
+		return -EINVAL;
+
+	if (cpu_online(cpu))
+		remove_cpu(cpu);
+
+	lapicid = cpu_data(cpu).apicid;
+	dev_dbg(dev, "Try to remove cpu %lld with lapicid %lld\n", cpu, lapicid);
+	ret = hcall_sos_remove_cpu(lapicid);
+	if (ret < 0) {
+		dev_err(dev, "Failed to remove cpu %lld!\n", cpu);
+		goto fail_remove;
+	}
+
+	return count;
+
+fail_remove:
+	add_cpu(cpu);
+	return ret;
+}
+static DEVICE_ATTR_WO(remove_cpu);
+
+static struct attribute *acrn_attrs[] = {
+	&dev_attr_remove_cpu.attr,
+	NULL
+};
+
+static struct attribute_group acrn_attr_group = {
+	.attrs = acrn_attrs,
+};
+
+static const struct attribute_group *acrn_attr_groups[] = {
+	&acrn_attr_group,
+	NULL
+};
+
 static const struct file_operations acrn_fops = {
 	.owner		= THIS_MODULE,
 	.open		= acrn_dev_open,
@@ -352,6 +399,7 @@ struct miscdevice acrn_dev = {
 	.minor	= MISC_DYNAMIC_MINOR,
 	.name	= "acrn_hsm",
 	.fops	= &acrn_fops,
+	.groups	= acrn_attr_groups,
 };
 
 static int __init hsm_init(void)
drivers/virt/acrn/hypercall.h +14 -0
index e640632366f0..0cfad05bd1a9 100644
@@ -13,6 +13,9 @@
 
 #define HC_ID 0x80UL
 
+#define HC_ID_GEN_BASE			0x0UL
+#define HC_SOS_REMOVE_CPU		_HC_ID(HC_ID, HC_ID_GEN_BASE + 0x01)
+
 #define HC_ID_VM_BASE			0x10UL
 #define HC_CREATE_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x00)
 #define HC_DESTROY_VM			_HC_ID(HC_ID, HC_ID_VM_BASE + 0x01)
@@ -42,6 +45,17 @@
 #define HC_ID_PM_BASE			0x80UL
 #define HC_PM_GET_CPU_STATE		_HC_ID(HC_ID, HC_ID_PM_BASE + 0x00)
 
+/**
+ * hcall_sos_remove_cpu() - Remove a vCPU of Service VM
+ * @cpu: The vCPU to be removed
+ *
+ * Return: 0 on success, <0 on failure
+ */
+static inline long hcall_sos_remove_cpu(u64 cpu)
+{
+	return acrn_hypercall1(HC_SOS_REMOVE_CPU, cpu);
+}
+
 /**
  * hcall_create_vm() - Create a User VM
  * @vminfo:	Service VM GPA of info of User VM creation
Signature · 1 line
2.28.0
author
#60 lore raw

Hi Dave,

Quoted 5 lines · Dave Hansen

How widely is ACRN used? It is some little Intel toy hypervisor, or is it already seeing broad use in the world? This, for instance, seems to have a backport:

https://github.com/teslamotors/linux/tree/intel-4.14

So far as i know, ACRN is not widely used now. But it indeed has a commercial IVI (In-Vehicle Infotainment) product "Chery EXEED" [1]. Today ACRN also has several design wins in industrial domain. For tree of teslamotors, we were unaware. I think teslamotors' tree was derived from [2].

Thanks
shuo

[1] https://projectacrn.org/acrn-project-chery-exeed-launch/
[2] https://github.com/projectacrn/acrn-kernel