HSM driver for ACRN hypervisor
Subsystems (3)
v4 cover HSM driver for ACRN hypervisor
From: Shuo Liu <[email protected]>
ACRN is a Type 1 reference hypervisor stack, running directly on the bare-metal hardware, and is suitable for a variety of IoT and embedded device solutions.
ACRN implements a hybrid VMM architecture, using a privileged Service VM. The Service VM manages the system resources (CPU, memory, etc.) and I/O devices of User VMs. Multiple User VMs are supported, with each of them running Linux, Android OS or Windows. Both Service VM and User VMs are guest VM.
Below figure shows the architecture.
Service VM User VM
+----------------------------+ | +------------------+
| +--------------+ | | | |
| |ACRN userspace| | | | |
| +--------------+ | | | |
|-----------------ioctl------| | | | ...
|kernel space +----------+ | | | |
| | HSM | | | | Drivers |
| +----------+ | | | |
+--------------------|-------+ | +------------------+
+---------------------hypercall----------------------------------------+
| ACRN Hypervisor |
+----------------------------------------------------------------------+
| Hardware |
+----------------------------------------------------------------------+There is only one Service VM which could run Linux as OS.
In a typical case, the Service VM will be auto started when ACRN Hypervisor is booted. Then the ACRN userspace (an application running in Service VM) could be used to start/stop User VMs by communicating with ACRN Hypervisor Service Module (HSM).
ACRN Hypervisor Service Module (HSM) is a middle layer that allows the ACRN userspace and Service VM OS kernel to communicate with ACRN Hypervisor and manage different User VMs. This middle layer provides the following functionalities,
- Issues hypercalls to the hypervisor to manage User VMs:
* VM/vCPU management
* Memory management
* Device passthrough
* Interrupts injection
- I/O requests handling from User VMs.
- Exports ioctl through HSM char device.
- Exports function calls for other kernel modules
ACRN is focused on embedded system. So it doesn't support some features. E.g.,
- ACRN doesn't support VM migration.
- ACRN doesn't support vCPU migration.
This patch set adds the HSM to the Linux kernel.
The basic ARCN support was merged to upstream already.
https://lore.kernel.org/lkml/[email protected]/
ChangeLog:
v4:
- Used acrn_dev.this_device directly for dev_*() (Reinette)
- Removed the odd usage of {get|put}_device() on &acrn_dev->this_device (Greg)
- Removed unused log code. (Greg)
- Corrected the return error values. (Greg)
- Mentioned that HSM relies hypervisor for sanity check in acrn_dev_ioctl() comments (Greg)
v3:
- Used {get|put}_device() helpers on &acrn_dev->this_device
- Moved unused code from front patches to later ones.
- Removed self-defined pr_fmt() and dev_fmt()
- Provided comments for acrn_vm_list_lock.
v2:
- Removed API version related code. (Dave)
- Replaced pr_*() by dev_*(). (Greg)
- Used -ENOTTY as the error code of unsupported ioctl. (Greg)
Shuo Liu (16):
docs: acrn: Introduce ACRN
x86/acrn: Introduce acrn_{setup, remove}_intr_handler() x86/acrn: Introduce hypercall interfaces
virt: acrn: Introduce ACRN HSM basic driver
virt: acrn: Introduce VM management interfaces
virt: acrn: Introduce an ioctl to set vCPU registers state
virt: acrn: Introduce EPT mapping management
virt: acrn: Introduce I/O request management
virt: acrn: Introduce PCI configuration space PIO accesses combiner
virt: acrn: Introduce interfaces for PCI device passthrough
virt: acrn: Introduce interrupt injection interfaces
virt: acrn: Introduce interfaces to query C-states and P-states
allowed by hypervisor
virt: acrn: Introduce I/O ranges operation interfaces
virt: acrn: Introduce ioeventfd
virt: acrn: Introduce irqfd
virt: acrn: Introduce an interface for Service VM to control vCPU
Yin Fengwei (1):
x86/acrn: Introduce an API to check if a VM is privileged
.../userspace-api/ioctl/ioctl-number.rst | 1 + Documentation/virt/acrn/index.rst | 11 + Documentation/virt/acrn/introduction.rst | 40 ++ Documentation/virt/acrn/io-request.rst | 97 +++ Documentation/virt/index.rst | 1 + MAINTAINERS | 9 + arch/x86/include/asm/acrn.h | 74 ++ arch/x86/kernel/cpu/acrn.c | 35 +- drivers/virt/Kconfig | 2 + drivers/virt/Makefile | 1 + drivers/virt/acrn/Kconfig | 15 + drivers/virt/acrn/Makefile | 3 + drivers/virt/acrn/acrn_drv.h | 229 +++++++ drivers/virt/acrn/hsm.c | 437 ++++++++++++ drivers/virt/acrn/hypercall.h | 254 +++++++ drivers/virt/acrn/ioeventfd.c | 273 ++++++++ drivers/virt/acrn/ioreq.c | 645 ++++++++++++++++++ drivers/virt/acrn/irqfd.c | 235 +++++++ drivers/virt/acrn/mm.c | 305 +++++++++ drivers/virt/acrn/vm.c | 126 ++++ include/uapi/linux/acrn.h | 486 +++++++++++++ 21 files changed, 3278 insertions(+), 1 deletion(-) create mode 100644 Documentation/virt/acrn/index.rst create mode 100644 Documentation/virt/acrn/introduction.rst create mode 100644 Documentation/virt/acrn/io-request.rst create mode 100644 arch/x86/include/asm/acrn.h create mode 100644 drivers/virt/acrn/Kconfig create mode 100644 drivers/virt/acrn/Makefile create mode 100644 drivers/virt/acrn/acrn_drv.h create mode 100644 drivers/virt/acrn/hsm.c create mode 100644 drivers/virt/acrn/hypercall.h create mode 100644 drivers/virt/acrn/ioeventfd.c create mode 100644 drivers/virt/acrn/ioreq.c create mode 100644 drivers/virt/acrn/irqfd.c create mode 100644 drivers/virt/acrn/mm.c create mode 100644 drivers/virt/acrn/vm.c create mode 100644 include/uapi/linux/acrn.h
base-commit: 18445bf405cb331117bc98427b1ba6f12418ad17
Signature · 1 line
2.28.0
PATCH v4 1/17 docs: acrn: Introduce ACRN
From: Shuo Liu <[email protected]>
Add documentation on the following aspects of ACRN:
1) A brief introduction on the architecture of ACRN.
2) I/O request handling in ACRN.
To learn more about ACRN, please go to ACRN project website
https://projectacrn.org, or the documentation page
https://projectacrn.github.io/.
--- Documentation/virt/acrn/index.rst | 11 +++ Documentation/virt/acrn/introduction.rst | 40 ++++++++++ Documentation/virt/acrn/io-request.rst | 97 ++++++++++++++++++++++++ Documentation/virt/index.rst | 1 + MAINTAINERS | 7 ++ 5 files changed, 156 insertions(+) create mode 100644 Documentation/virt/acrn/index.rst create mode 100644 Documentation/virt/acrn/introduction.rst create mode 100644 Documentation/virt/acrn/io-request.rst
Documentation/virt/acrn/index.rst new file +11 -0
@@ -0,0 +1,11 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=============== +ACRN Hypervisor +=============== + +.. toctree:: + :maxdepth: 1 + + introduction + io-request
Documentation/virt/acrn/introduction.rst new file +40 -0
@@ -0,0 +1,40 @@ +.. SPDX-License-Identifier: GPL-2.0 + +ACRN Hypervisor Introduction +============================ + +The ACRN Hypervisor is a Type 1 hypervisor, running directly on the bare-metal +hardware. It has a privileged management VM, called Service VM, to manage User +VMs and do I/O emulation. + +ACRN userspace is an application running in the Service VM that emulates +devices for a User VM based on command line configurations. ACRN Hypervisor +Service Module (HSM) is a kernel module in the Service VM which provides +hypervisor services to the ACRN userspace. + +Below figure shows the architecture. + +:: + + Service VM User VM + +----------------------------+ | +------------------+ + | +--------------+ | | | | + | |ACRN userspace| | | | | + | +--------------+ | | | | + |-----------------ioctl------| | | | ... + |kernel space +----------+ | | | | + | | HSM | | | | Drivers | + | +----------+ | | | | + +--------------------|-------+ | +------------------+ + +---------------------hypercall----------------------------------------+ + | ACRN Hypervisor | + +----------------------------------------------------------------------+ + | Hardware | + +----------------------------------------------------------------------+ + +ACRN userspace allocates memory for the User VM, configures and initializes the +devices used by the User VM, loads the virtual bootloader, initializes the +virtual CPU state and handles I/O request accesses from the User VM. It uses +ioctls to communicate with the HSM. HSM implements hypervisor services by +interacting with the ACRN Hypervisor via hypercalls. HSM exports a char device +interface (/dev/acrn_hsm) to userspace.
Documentation/virt/acrn/io-request.rst new file +97 -0
97 lines not on this page · raw message
Documentation/virt/index.rst +1 -0
@@ -11,6 +11,7 @@ Linux Virtualization Support uml/user_mode_linux paravirt_ops guest-halt-polling + acrn/index .. only:: html and subproject
MAINTAINERS +7 -0
@@ -436,6 +436,13 @@ S: Orphan F: drivers/platform/x86/wmi.c F: include/uapi/linux/wmi.h +ACRN HYPERVISOR SERVICE MODULE +M: Shuo Liu <[email protected]> +L: [email protected] +S: Supported +W: https://projectacrn.org +F: Documentation/virt/acrn/ + AD1889 ALSA SOUND DRIVER L: [email protected] S: Maintained
Signature · 1 line
2.28.0
Randy Dunlap
reviewerRe: [PATCH v4 01/17] docs: acrn: Introduce ACRN
Quoted 54 lines · On 9/22/20 4:42 AM, [email protected] wrote:
From: Shuo Liu <[email protected]>
Add documentation on the following aspects of ACRN:
1) A brief introduction on the architecture of ACRN.
2) I/O request handling in ACRN.To learn more about ACRN, please go to ACRN project website
https://projectacrn.org, or the documentation page
https://projectacrn.github.io/.Signed-off-by Shuo Liu <[email protected]>Reviewed-by Zhi Wang <[email protected]>Reviewed-by Reinette Chatre <[email protected]>Cc Dave Hansen <[email protected]>Cc Sen Christopherson <[email protected]>Cc Dan Williams <[email protected]>Cc Fengwei Yin <[email protected]>Cc Zhi Wang <[email protected]>Cc Zhenyu Wang <[email protected]>Cc Yu Wang <[email protected]>Cc Reinette Chatre <[email protected]>Cc Greg Kroah-Hartman <[email protected]>--- Documentation/virt/acrn/index.rst | 11 +++ Documentation/virt/acrn/introduction.rst | 40 ++++++++++ Documentation/virt/acrn/io-request.rst | 97 ++++++++++++++++++++++++ Documentation/virt/index.rst | 1 + MAINTAINERS | 7 ++ 5 files changed, 156 insertions(+) create mode 100644 Documentation/virt/acrn/index.rst create mode 100644 Documentation/virt/acrn/introduction.rst create mode 100644 Documentation/virt/acrn/io-request.rstDocumentation/virt/acrn/io-request.rst new file +13
-0@@ -0,0 +1,97 @@ +.. SPDX-License-Identifier: GPL-2.0 + +I/O request handling +==================== + +An I/O request of a User VM, which is constructed by the hypervisor, is +distributed by the ACRN Hypervisor Service Module to an I/O client +corresponding to the address range of the I/O request. Details of I/O request +handling are described in the following sections. + +1. I/O request +-------------- +
...
Quoted 4 lines
Documentation/virt/acrn/io-request.rst new file +4
-0+ +2. I/O clients +-------------- +
...
Documentation/virt/acrn/io-request.rst line 70
Quoted 5 lines
Documentation/virt/acrn/io-request.rst new file +5
-0+ +3. I/O request state transition +------------------------------- + +The state transitions of a ACRN I/O request are as follows.
of an ACRN
Quoted 6 lines
Documentation/virt/acrn/io-request.rst new file +6
-0+ +:: + + FREE -> PENDING -> PROCESSING -> COMPLETE -> FREE -> ... + +- FREE: this I/O request slot is empty
Documentation/virt/acrn/io-request.rst line 86
Quoted 10 lines
Documentation/virt/acrn/io-request.rst new file +10
-0+- PENDING: a valid I/O request is pending in this slot +- PROCESSING: the I/O request is being processed +- COMPLETE: the I/O request has been processed + +An I/O request in COMPLETE or FREE state is owned by the hypervisor. HSM and +ACRN userspace are in charge of processing the others. + +4. Processing flow of I/O requests +------------------------------- +
...
thanks.
Signature · 1 line
~Randy
Re: [PATCH v4 01/17] docs: acrn: Introduce ACRN
Quoted 65 lines · Randy Dunlap
Quoted 54 lines · On 9/22/20 4:42 AM, [email protected] wrote:
From: Shuo Liu <[email protected]>
Add documentation on the following aspects of ACRN:
1) A brief introduction on the architecture of ACRN.
2) I/O request handling in ACRN.To learn more about ACRN, please go to ACRN project website
https://projectacrn.org, or the documentation page
https://projectacrn.github.io/.Signed-off-by Shuo Liu <[email protected]>Reviewed-by Zhi Wang <[email protected]>Reviewed-by Reinette Chatre <[email protected]>Cc Dave Hansen <[email protected]>Cc Sen Christopherson <[email protected]>Cc Dan Williams <[email protected]>Cc Fengwei Yin <[email protected]>Cc Zhi Wang <[email protected]>Cc Zhenyu Wang <[email protected]>Cc Yu Wang <[email protected]>Cc Reinette Chatre <[email protected]>Cc Greg Kroah-Hartman <[email protected]>--- Documentation/virt/acrn/index.rst | 11 +++ Documentation/virt/acrn/introduction.rst | 40 ++++++++++ Documentation/virt/acrn/io-request.rst | 97 ++++++++++++++++++++++++ Documentation/virt/index.rst | 1 + MAINTAINERS | 7 ++ 5 files changed, 156 insertions(+) create mode 100644 Documentation/virt/acrn/index.rst create mode 100644 Documentation/virt/acrn/introduction.rst create mode 100644 Documentation/virt/acrn/io-request.rstDocumentation/virt/acrn/io-request.rst new file +13
-0@@ -0,0 +1,97 @@ +.. SPDX-License-Identifier: GPL-2.0 + +I/O request handling +==================== + +An I/O request of a User VM, which is constructed by the hypervisor, is +distributed by the ACRN Hypervisor Service Module to an I/O client +corresponding to the address range of the I/O request. Details of I/O request +handling are described in the following sections. + +1. I/O request +-------------- +...
Quoted 4 lines
Documentation/virt/acrn/io-request.rst new file +4
-0+ +2. I/O clients +-------------- +...
Quoted 7 lines
Documentation/virt/acrn/io-request.rst line 70
Quoted 5 lines
Documentation/virt/acrn/io-request.rst new file +5
-0+ +3. I/O request state transition +------------------------------- + +The state transitions of a ACRN I/O request are as follows.of an ACRN
OK. Thanks for review.
Thanks
shuo
PATCH v4 2/17 x86/acrn: Introduce acrn_{setup, remove}_intr_handler()
In reply to [email protected]From: Shuo Liu <[email protected]>
The ACRN Hypervisor builds an I/O request when a trapped I/O access happens in User VM. Then, ACRN Hypervisor issues an upcall by sending a notification interrupt to the Service VM. HSM in the Service VM needs to hook the notification interrupt to handle I/O requests.
Notification interrupts from ACRN Hypervisor are already supported and a, currently uninitialized, callback called.
Export two APIs for HSM to setup/remove its callback.
--- arch/x86/include/asm/acrn.h | 8 ++++++++ arch/x86/kernel/cpu/acrn.c | 16 ++++++++++++++++ 2 files changed, 24 insertions(+) create mode 100644 arch/x86/include/asm/acrn.h
arch/x86/include/asm/acrn.h new file +8 -0
@@ -0,0 +1,8 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_ACRN_H +#define _ASM_X86_ACRN_H + +void acrn_setup_intr_handler (void (*handler)(void )); +void acrn_remove_intr_handler (void ); + +#endif /* _ASM_X86_ACRN_H */
arch/x86/kernel/cpu/acrn.c +16 -0
@@ -9,7 +9,11 @@* */ +#define pr_fmt (fmt) "acrn: " fmt +#include <linux/interrupt.h> + +#include <asm/acrn.h> #include <asm/apic.h> #include <asm/cpufeatures.h> #include <asm/desc.h>
@@ -55,6 +59,18 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_acrn_hv_callback)set_irq_regs (old_regs); } +void acrn_setup_intr_handler (void (*handler)(void )) +{ + acrn_intr_handler = handler; +} +EXPORT_SYMBOL_GPL (acrn_setup_intr_handler); + +void acrn_remove_intr_handler (void ) +{ + acrn_intr_handler =NULL ; +} +EXPORT_SYMBOL_GPL (acrn_remove_intr_handler); +const __initconst struct hypervisor_x86 x86_hyper_acrn = { .name ="ACRN" , .detect = acrn_detect,
Signature · 1 line
2.28.0
Re: [PATCH v4 02/17] x86/acrn: Introduce acrn_{setup, remove}_intr_handler()
Quoted 50 lines · On Tue, Sep 22, 2020 at 07:42:56PM +0800, [email protected] wrote:
From: Shuo Liu <[email protected]>
The ACRN Hypervisor builds an I/O request when a trapped I/O access happens in User VM. Then, ACRN Hypervisor issues an upcall by sending a notification interrupt to the Service VM. HSM in the Service VM needs to hook the notification interrupt to handle I/O requests.
Notification interrupts from ACRN Hypervisor are already supported and a, currently uninitialized, callback called.
Export two APIs for HSM to setup/remove its callback.
Originally-by Yakui Zhao <[email protected]>Signed-off-by Shuo Liu <[email protected]>Reviewed-by Zhi Wang <[email protected]>Reviewed-by Reinette Chatre <[email protected]>Cc Dave Hansen <[email protected]>Cc Sean Christopherson <[email protected]>Cc Dan Williams <[email protected]>Cc Fengwei Yin <[email protected]>Cc Zhi Wang <[email protected]>Cc Zhenyu Wang <[email protected]>Cc Yu Wang <[email protected]>Cc Reinette Chatre <[email protected]>Cc Greg Kroah-Hartman <[email protected]>--- arch/x86/include/asm/acrn.h | 8 ++++++++ arch/x86/kernel/cpu/acrn.c | 16 ++++++++++++++++ 2 files changed, 24 insertions(+) create mode 100644 arch/x86/include/asm/acrn.harch/x86/include/asm/acrn.h new file +8
-0@@ -0,0 +1,8 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_ACRN_H +#define _ASM_X86_ACRN_H + +void acrn_setup_intr_handler (void (*handler)(void )); +void acrn_remove_intr_handler (void ); + +#endif /* _ASM_X86_ACRN_H */ arch/x86/kernel/cpu/acrn.c +0
-0
arch/x86/kernel/cpu/acrn.c line 12
Quoted 4 lines
arch/x86/kernel/cpu/acrn.c +1
-0@@ -9,7 +9,11 @@* */ +#define pr_fmt (fmt) "acrn: " fmt
Why is this needed, if you are not adding pr_* calls in this patch?
thanks,
greg k-h
Re: [PATCH v4 02/17] x86/acrn: Introduce acrn_{setup, remove}_intr_handler()
Quoted 54 lines · Greg Kroah-Hartman
Quoted 53 lines · On Tue, Sep 22, 2020 at 07:42:56PM +0800, [email protected] wrote:
From: Shuo Liu <[email protected]>
The ACRN Hypervisor builds an I/O request when a trapped I/O access happens in User VM. Then, ACRN Hypervisor issues an upcall by sending a notification interrupt to the Service VM. HSM in the Service VM needs to hook the notification interrupt to handle I/O requests.
Notification interrupts from ACRN Hypervisor are already supported and a, currently uninitialized, callback called.
Export two APIs for HSM to setup/remove its callback.
Originally-by Yakui Zhao <[email protected]>Signed-off-by Shuo Liu <[email protected]>Reviewed-by Zhi Wang <[email protected]>Reviewed-by Reinette Chatre <[email protected]>Cc Dave Hansen <[email protected]>Cc Sean Christopherson <[email protected]>Cc Dan Williams <[email protected]>Cc Fengwei Yin <[email protected]>Cc Zhi Wang <[email protected]>Cc Zhenyu Wang <[email protected]>Cc Yu Wang <[email protected]>Cc Reinette Chatre <[email protected]>Cc Greg Kroah-Hartman <[email protected]>--- arch/x86/include/asm/acrn.h | 8 ++++++++ arch/x86/kernel/cpu/acrn.c | 16 ++++++++++++++++ 2 files changed, 24 insertions(+) create mode 100644 arch/x86/include/asm/acrn.harch/x86/include/asm/acrn.h new file +8
-0@@ -0,0 +1,8 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_ACRN_H +#define _ASM_X86_ACRN_H + +void acrn_setup_intr_handler (void (*handler)(void )); +void acrn_remove_intr_handler (void ); + +#endif /* _ASM_X86_ACRN_H */ arch/x86/kernel/cpu/acrn.c +0
-0@@ -9,7 +9,11 @@* */
Quoted 3 lines
arch/x86/kernel/cpu/acrn.c line 12
Quoted 1 line
arch/x86/kernel/cpu/acrn.c +1
-0+#define pr_fmt (fmt) "acrn: " fmtWhy is this needed, if you are not adding pr_* calls in this patch?
True. I will remove it. Thanks.
Borislav Petkov
maintainerRe: [PATCH v4 02/17] x86/acrn: Introduce acrn_{setup, remove}_intr_handler()
In reply to [email protected]arch/x86/kernel/cpu/acrn.c DEFINE_IDTENTRY_SYSVEC(sysvec_acrn_hv_callback) · line 72
Quoted 11 lines · On Tue, Sep 22, 2020 at 07:42:56PM +0800, [email protected] wrote:
arch/x86/kernel/cpu/acrn.c +11
-0+void acrn_setup_intr_handler (void (*handler)(void )) +{ + acrn_intr_handler = handler; +} +EXPORT_SYMBOL_GPL (acrn_setup_intr_handler); + +void acrn_remove_intr_handler (void ) +{ + acrn_intr_handler =NULL ; +} +EXPORT_SYMBOL_GPL (acrn_remove_intr_handler);
I don't like this one bit.
Also, what stops the module from doing acrn_remove_intr_handler() while it gets a HYPERVISOR_CALLBACK_VECTOR interrupt and the handler disappearing from under it?
IOW, this should be an atomic notifier instead.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v4 02/17] x86/acrn: Introduce acrn_{setup, remove}_intr_handler()
Quoted 15 lines · Borislav Petkov
arch/x86/kernel/cpu/acrn.c DEFINE_IDTENTRY_SYSVEC(sysvec_acrn_hv_callback) · line 72
Quoted 11 lines · On Tue, Sep 22, 2020 at 07:42:56PM +0800, [email protected] wrote:
arch/x86/kernel/cpu/acrn.c +11
-0+void acrn_setup_intr_handler (void (*handler)(void )) +{ + acrn_intr_handler = handler; +} +EXPORT_SYMBOL_GPL (acrn_setup_intr_handler); + +void acrn_remove_intr_handler (void ) +{ + acrn_intr_handler =NULL ; +} +EXPORT_SYMBOL_GPL (acrn_remove_intr_handler);I don't like this one bit.
Quoted 5 lines
Also, what stops the module from doing
acrn_remove_intr_handler()while it gets aHYPERVISOR_CALLBACK_VECTORinterrupt and the handler disappearing from under it?IOW, this should be an atomic notifier instead.
That does not prevent that either and notifiers suck. The pointer is fine and if something removes the handler before all of the muck is shutdown then the author can keep the pieces and mop up the remains.
Thanks,
tglx
Borislav Petkov
maintainerRe: [PATCH v4 02/17] x86/acrn: Introduce acrn_{setup, remove}_intr_handler()
Quoted 1 line · Thomas Gleixner
That does not prevent that either and notifiers suck.
Bah, atomic notifiers run functions which cannot block, not what is needed here, right.
Quoted 3 lines
The pointer is fine and if something removes the handler before all of the muck is shutdown then the author can keep the pieces and mop up the remains.
Uhu, so what makes sure that the module is not removed while an IRQ is happening?
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v4 02/17] x86/acrn: Introduce acrn_{setup, remove}_intr_handler()
Hi Boris,
Quoted 6 lines · Borislav Petkov
Quoted 1 line · Thomas Gleixner
That does not prevent that either and notifiers suck.
Bah, atomic notifiers run functions which cannot block, not what is needed here, right.
Quoted 6 lines
Quoted 3 lines
The pointer is fine and if something removes the handler before all of the muck is shutdown then the author can keep the pieces and mop up the remains.
Uhu, so what makes sure that the module is not removed while an IRQ is happening?
The precondition of the removing of the module is that there is no User VM living (every opening of the dev file will hold a ref count of the module). The interrupt only can occur with active User VMs. So if a notification interrupt is happending, the module cannot be removed as there is still User VM living.
Thanks
shuo
PATCH v4 3/17 x86/acrn: Introduce an API to check if a VM is privileged
In reply to [email protected]From: Yin Fengwei <[email protected]>
ACRN Hypervisor reports hypervisor features via CPUID leaf 0x40000001 which is similar to KVM. A VM can check if it's the privileged VM using the feature bits. The Service VM is the only privileged VM by design.
--- arch/x86/include/asm/acrn.h | 9 +++++++++ arch/x86/kernel/cpu/acrn.c | 19 ++++++++++++++++++- 2 files changed, 27 insertions(+), 1 deletion(-)
arch/x86/include/asm/acrn.h +9 -0
@@ -2,7 +2,16 @@#ifndef _ASM_X86_ACRN_H#define _ASM_X86_ACRN_H +/* +* This CPUID returns feature bitmaps in EAX. +* Guest VM uses this to detect the appropriate feature bit. +*/ +#define ACRN_CPUID_FEATURES 0x40000001 +/* Bit 0 indicates whether guest VM is privileged */ +#define ACRN_FEATURE_PRIVILEGED_VM BIT(0) +void acrn_setup_intr_handler (void (*handler)(void ));void acrn_remove_intr_handler (void ); +bool acrn_is_privileged_vm (void );#endif /* _ASM_X86_ACRN_H */
arch/x86/kernel/cpu/acrn.c +18 -1
@@ -21,9 +21,26 @@#include <asm/idtentry.h> #include <asm/irq_regs.h> +static u32 acrn_cpuid_base (void ) +{ +static u32 acrn_cpuid_base; + +if (!acrn_cpuid_base &&boot_cpu_has (X86_FEATURE_HYPERVISOR )) + acrn_cpuid_base =hypervisor_cpuid_base ("ACRNACRNACRN" ,0 ); + +return acrn_cpuid_base; +} + +bool acrn_is_privileged_vm (void ) +{ +return cpuid_eax (acrn_cpuid_base () |ACRN_CPUID_FEATURES ) & +ACRN_FEATURE_PRIVILEGED_VM ; +} +EXPORT_SYMBOL_GPL (acrn_is_privileged_vm); +static u32 __init acrn_detect (void ) { -return hypervisor_cpuid_base ("ACRNACRNACRN" ,0 ); +return acrn_cpuid_base (); }static void __init acrn_init_platform (void )
Signature · 1 line
2.28.0
Borislav Petkov
maintainerRe: [PATCH v4 03/17] x86/acrn: Introduce an API to check if a VM is privileged
arch/x86/kernel/cpu/acrn.c line 36
Quoted 13 lines · On Tue, Sep 22, 2020 at 07:42:57PM +0800, [email protected] wrote:
arch/x86/kernel/cpu/acrn.c +13
-0+static u32 acrn_cpuid_base (void ) +{ +static u32 acrn_cpuid_base; + +if (!acrn_cpuid_base &&boot_cpu_has (X86_FEATURE_HYPERVISOR )) + acrn_cpuid_base =hypervisor_cpuid_base ("ACRNACRNACRN" ,0 ); + +return acrn_cpuid_base; +} + +bool acrn_is_privileged_vm (void ) +{ + return cpuid_eax(acrn_cpuid_base() | ACRN_CPUID_FEATURES) &
What's that dance and acrn_cpuid_base static thing needed for? Why not simply:
cpuid_eax(ACRN_CPUID_FEATURES) & ...
?
arch/x86/kernel/cpu/acrn.c line 39
Quoted 3 lines
arch/x86/kernel/cpu/acrn.c +3
-0+ACRN_FEATURE_PRIVILEGED_VM ; +} +EXPORT_SYMBOL_GPL (acrn_is_privileged_vm);
Also, if you're going to need more of those bit checkers acrn_is_<something> which look at ACRN_CPUID_FEATURES, just stash CPUID_0x40000001_EAX locally and use a
acrn_has(ACRN_FEATURE_PRIVILEGED_VM)
which does the bit testing.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v4 03/17] x86/acrn: Introduce an API to check if a VM is privileged
Hi Boris,
Quoted 18 lines · Borislav Petkov
arch/x86/kernel/cpu/acrn.c line 36
Quoted 13 lines · On Tue, Sep 22, 2020 at 07:42:57PM +0800, [email protected] wrote:
arch/x86/kernel/cpu/acrn.c +13
-0+static u32 acrn_cpuid_base (void ) +{ +static u32 acrn_cpuid_base; + +if (!acrn_cpuid_base &&boot_cpu_has (X86_FEATURE_HYPERVISOR )) + acrn_cpuid_base =hypervisor_cpuid_base ("ACRNACRNACRN" ,0 ); + +return acrn_cpuid_base; +} + +bool acrn_is_privileged_vm (void ) +{ + return cpuid_eax(acrn_cpuid_base() | ACRN_CPUID_FEATURES) &What's that dance and
acrn_cpuid_basestatic thing needed for? Why not simply:
Quoted 3 lines
cpuid_eax(ACRN_CPUID_FEATURES)& ...?
hypervisor_cpuid_base() searches reserved hypervisor cpuid region and return the base matched the right signature, the base might vary. So i put it here.
Quoted 9 lines
arch/x86/kernel/cpu/acrn.c line 39
Quoted 4 lines
arch/x86/kernel/cpu/acrn.c +3
-0+ACRN_FEATURE_PRIVILEGED_VM ; +} +EXPORT_SYMBOL_GPL (acrn_is_privileged_vm);Also, if you're going to need more of those bit checkers acrn_is_<something> which look at
ACRN_CPUID_FEATURES, just stashCPUID_0x40000001_EAXlocally and use a
Quoted 3 lines
acrn_has(ACRN_FEATURE_PRIVILEGED_VM)which does the bit testing.
Thanks. Currently, there is only one feature bit. I will introduce that you suggested with more feature bits to be tested.
Thanks
shuo
PATCH v4 4/17 x86/acrn: Introduce hypercall interfaces
In reply to [email protected]From: Shuo Liu <[email protected]>
The Service VM communicates with the hypervisor via conventional hypercalls. VMCALL instruction is used to make the hypercalls.
ACRN hypercall ABI:
* Hypercall number is in R8 register.
* Up to 2 parameters are in RDI and RSI registers.
* Return value is in RAX register.
Introduce the ACRN hypercall interfaces. Because GCC doesn't support R8 register as direct register constraints, here are two ways to use R8 in extended asm:
1) use explicit register variable as input
2) use supported constraint as input with a explicit MOV to R8 in beginning of asm
The number of instructions of above two ways are same. Asm code from 1)
38: 41 b8 00 00 00 80 mov $0x80000000,%r8d 3e: 48 89 c7 mov %rax,%rdi 41: 0f 01 c1 vmcall
Here, writes to the lower dword (%r8d) clear the upper dword of %r8 when the CPU is in 64-bit mode.
Asm code from 2)
38: 48 89 c7 mov %rax,%rdi 3b: 49 b8 00 00 00 80 00 movabs $0x80000000,%r8 42: 00 00 00 45: 0f 01 c1 vmcall
Choose 1) for code simplicity and a little bit of code size optimization.
--- arch/x86/include/asm/acrn.h | 57 +++++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+)
arch/x86/include/asm/acrn.h +57 -0
@@ -14,4 +14,61 @@ void acrn_setup_intr_handler(void (*handler)(void));void acrn_remove_intr_handler (void );bool acrn_is_privileged_vm (void ); +/* +* Hypercalls for ACRN +* +* - VMCALL instruction is used to implement ACRN hypercalls. +* - ACRN hypercall ABI: +* - Hypercall number is passed in R8 register. +* - Up to 2 arguments are passed in RDI, RSI. +* - Return value will be placed in RAX. +*/ +static inline long acrn_hypercall0 (unsigned long hcall_id) +{ + registerlong r8 asm("r8" ); +long result; + +/* Nothing can come between the r8 assignment and the asm: */ + r8 = hcall_id; + asmvolatile ("vmcall\n\t" + :"=a" (result) + :"r" (r8) + : ); + +return result; +} + +static inline long acrn_hypercall1 (unsigned long hcall_id, +unsigned long param1) +{ + registerlong r8 asm("r8" ); +long result; + +/* Nothing can come between the r8 assignment and the asm: */ + r8 = hcall_id; + asmvolatile ("vmcall\n\t" + :"=a" (result) + :"r" (r8),"D" (param1) + : ); + +return result; +} + +static inline long acrn_hypercall2 (unsigned long hcall_id, +unsigned long param1, +unsigned long param2) +{ + registerlong r8 asm("r8" ); +long result; + +/* Nothing can come between the r8 assignment and the asm: */ + r8 = hcall_id; + asmvolatile ("vmcall\n\t" + :"=a" (result) + :"r" (r8),"D" (param1),"S" (param2) + : ); + +return result; +} +#endif /* _ASM_X86_ACRN_H */
Signature · 1 line
2.28.0
Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces
Quoted 68 lines · On Tue, Sep 22, 2020 at 07:42:58PM +0800, [email protected] wrote:
From: Shuo Liu <[email protected]>
The Service VM communicates with the hypervisor via conventional hypercalls. VMCALL instruction is used to make the hypercalls.
ACRN hypercall ABI:
* Hypercall number is in R8 register.
* Up to 2 parameters are in RDI and RSI registers.
* Return value is in RAX register.Introduce the ACRN hypercall interfaces. Because GCC doesn't support R8 register as direct register constraints, here are two ways to use R8 in extended asm:
1) use explicit register variable as input
2) use supported constraint as input with a explicit MOV to R8 in beginning of asmThe number of instructions of above two ways are same. Asm code from 1)
38: 41 b8 00 00 00 80 mov $0x80000000,%r8d 3e: 48 89 c7 mov %rax,%rdi 41: 0f 01 c1 vmcallHere, writes to the lower dword (%r8d) clear the upper dword of %r8 when the CPU is in 64-bit mode.
Asm code from 2)
38: 48 89 c7 mov %rax,%rdi 3b: 49 b8 00 00 00 80 00 movabs $0x80000000,%r8 42: 00 00 00 45: 0f 01 c1 vmcallChoose 1) for code simplicity and a little bit of code size optimization.
Originally-by Yakui Zhao <[email protected]>Signed-off-by Shuo Liu <[email protected]>Reviewed-by Reinette Chatre <[email protected]>Cc Dave Hansen <[email protected]>Cc Sean Christopherson <[email protected]>Cc Dan Williams <[email protected]>Cc Fengwei Yin <[email protected]>Cc Zhi Wang <[email protected]>Cc Zhenyu Wang <[email protected]>Cc Yu Wang <[email protected]>Cc Reinette Chatre <[email protected]>Cc Greg Kroah-Hartman <[email protected]>--- arch/x86/include/asm/acrn.h | 57 +++++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+)arch/x86/include/asm/acrn.h +10
-0@@ -14,4 +14,61 @@ void acrn_setup_intr_handler(void (*handler)(void));void acrn_remove_intr_handler (void );bool acrn_is_privileged_vm (void ); +/* +* Hypercalls for ACRN +* +* - VMCALL instruction is used to implement ACRN hypercalls. +* - ACRN hypercall ABI: +* - Hypercall number is passed in R8 register. +* - Up to 2 arguments are passed in RDI, RSI. +* - Return value will be placed in RAX. +*/ +static inline long acrn_hypercall0 (unsigned long hcall_id)
arch/x86/include/asm/acrn.h void acrn_setup_intr_handler(void (*handler)(void)); · line 36
Quoted 10 lines
arch/x86/include/asm/acrn.h +10
-0+{ + registerlong r8 asm("r8" ); +long result; + +/* Nothing can come between the r8 assignment and the asm: */ + r8 = hcall_id; + asmvolatile ("vmcall\n\t" + :"=a" (result) + :"r" (r8) + : );
What keeps an interrupt from happening between the r8 assignment and the
asm: ?
Is this something that most hypercalls need to handle? I don't see other ones needing this type of thing, is it just because of how these are defined?
confused,
greg k-h
Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces
Quoted 83 lines · Greg Kroah-Hartman
Quoted 68 lines · On Tue, Sep 22, 2020 at 07:42:58PM +0800, [email protected] wrote:
From: Shuo Liu <[email protected]>
The Service VM communicates with the hypervisor via conventional hypercalls. VMCALL instruction is used to make the hypercalls.
ACRN hypercall ABI:
* Hypercall number is in R8 register.
* Up to 2 parameters are in RDI and RSI registers.
* Return value is in RAX register.Introduce the ACRN hypercall interfaces. Because GCC doesn't support R8 register as direct register constraints, here are two ways to use R8 in extended asm:
1) use explicit register variable as input
2) use supported constraint as input with a explicit MOV to R8 in beginning of asmThe number of instructions of above two ways are same. Asm code from 1)
38: 41 b8 00 00 00 80 mov $0x80000000,%r8d 3e: 48 89 c7 mov %rax,%rdi 41: 0f 01 c1 vmcallHere, writes to the lower dword (%r8d) clear the upper dword of %r8 when the CPU is in 64-bit mode.
Asm code from 2)
38: 48 89 c7 mov %rax,%rdi 3b: 49 b8 00 00 00 80 00 movabs $0x80000000,%r8 42: 00 00 00 45: 0f 01 c1 vmcallChoose 1) for code simplicity and a little bit of code size optimization.
Originally-by Yakui Zhao <[email protected]>Signed-off-by Shuo Liu <[email protected]>Reviewed-by Reinette Chatre <[email protected]>Cc Dave Hansen <[email protected]>Cc Sean Christopherson <[email protected]>Cc Dan Williams <[email protected]>Cc Fengwei Yin <[email protected]>Cc Zhi Wang <[email protected]>Cc Zhenyu Wang <[email protected]>Cc Yu Wang <[email protected]>Cc Reinette Chatre <[email protected]>Cc Greg Kroah-Hartman <[email protected]>--- arch/x86/include/asm/acrn.h | 57 +++++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+)arch/x86/include/asm/acrn.h +10
-0@@ -14,4 +14,61 @@ void acrn_setup_intr_handler(void (*handler)(void));void acrn_remove_intr_handler (void );bool acrn_is_privileged_vm (void ); +/* +* Hypercalls for ACRN +* +* - VMCALL instruction is used to implement ACRN hypercalls. +* - ACRN hypercall ABI: +* - Hypercall number is passed in R8 register. +* - Up to 2 arguments are passed in RDI, RSI. +* - Return value will be placed in RAX. +*/ +static inline long acrn_hypercall0 (unsigned long hcall_id)arch/x86/include/asm/acrn.h void acrn_setup_intr_handler(void (*handler)(void)); · line 36
Quoted 10 lines
arch/x86/include/asm/acrn.h +10
-0+{ + registerlong r8 asm("r8" ); +long result; + +/* Nothing can come between the r8 assignment and the asm: */ + r8 = hcall_id; + asmvolatile ("vmcall\n\t" + :"=a" (result) + :"r" (r8) + : );What keeps an interrupt from happening between the r8 assignment and the
asm: ?
Quoted 3 lines
Is this something that most hypercalls need to handle? I don't see other ones needing this type of thing, is it just because of how these are defined?
Ah, the changelog above explains this. You should put that in the code itself, as a comment, otherwise we will not know this at all in 5 years, when gcc is changed to allow r8 access :)
thanks,
greg k-h
Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces
Quoted 75 lines · Greg Kroah-Hartman
Quoted 74 lines · Greg Kroah-Hartman
Quoted 73 lines · On Tue, Sep 22, 2020 at 07:42:58PM +0800, [email protected] wrote:
From: Shuo Liu <[email protected]>
The Service VM communicates with the hypervisor via conventional hypercalls. VMCALL instruction is used to make the hypercalls.
ACRN hypercall ABI:
* Hypercall number is in R8 register.
* Up to 2 parameters are in RDI and RSI registers.
* Return value is in RAX register.Introduce the ACRN hypercall interfaces. Because GCC doesn't support R8 register as direct register constraints, here are two ways to use R8 in extended asm:
1) use explicit register variable as input
2) use supported constraint as input with a explicit MOV to R8 in beginning of asmThe number of instructions of above two ways are same. Asm code from 1)
38: 41 b8 00 00 00 80 mov $0x80000000,%r8d 3e: 48 89 c7 mov %rax,%rdi 41: 0f 01 c1 vmcallHere, writes to the lower dword (%r8d) clear the upper dword of %r8 when the CPU is in 64-bit mode.
Asm code from 2)
38: 48 89 c7 mov %rax,%rdi 3b: 49 b8 00 00 00 80 00 movabs $0x80000000,%r8 42: 00 00 00 45: 0f 01 c1 vmcallChoose 1) for code simplicity and a little bit of code size optimization.
Originally-by Yakui Zhao <[email protected]>Signed-off-by Shuo Liu <[email protected]>Reviewed-by Reinette Chatre <[email protected]>Cc Dave Hansen <[email protected]>Cc Sean Christopherson <[email protected]>Cc Dan Williams <[email protected]>Cc Fengwei Yin <[email protected]>Cc Zhi Wang <[email protected]>Cc Zhenyu Wang <[email protected]>Cc Yu Wang <[email protected]>Cc Reinette Chatre <[email protected]>Cc Greg Kroah-Hartman <[email protected]>--- arch/x86/include/asm/acrn.h | 57 +++++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+)arch/x86/include/asm/acrn.h +15
-0@@ -14,4 +14,61 @@ void acrn_setup_intr_handler(void (*handler)(void));void acrn_remove_intr_handler (void );bool acrn_is_privileged_vm (void ); +/* +* Hypercalls for ACRN +* +* - VMCALL instruction is used to implement ACRN hypercalls. +* - ACRN hypercall ABI: +* - Hypercall number is passed in R8 register. +* - Up to 2 arguments are passed in RDI, RSI. +* - Return value will be placed in RAX. +*/ +static inline long acrn_hypercall0 (unsigned long hcall_id) +{ + registerlong r8 asm("r8" ); +long result; + +/* Nothing can come between the r8 assignment and the asm: */
Quoted 8 lines
Quoted 8 lines
arch/x86/include/asm/acrn.h void acrn_setup_intr_handler(void (*handler)(void)); · line 36
Quoted 5 lines
arch/x86/include/asm/acrn.h +5
-0+ r8 = hcall_id; + asmvolatile ("vmcall\n\t" + :"=a" (result) + :"r" (r8) + : );What keeps an interrupt from happening between the r8 assignment and the
asm: ?
Dave gave a good explanation in another email. I will apply his better comment that "No other C code can come between this r8 assignment and the inline asm".
Quoted 8 lines
Quoted 4 lines
Is this something that most hypercalls need to handle? I don't see other ones needing this type of thing, is it just because of how these are defined?
Ah, the changelog above explains this. You should put that in the code itself, as a comment, otherwise we will not know this at all in 5 years, when gcc is changed to allow r8 access :)
OK. I will copy that into code as well.
Thanks
shuo
Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces
In reply to Greg Kroah-HartmanQuoted 5 lines · Greg Kroah-Hartman
Quoted 5 lines
arch/x86/include/asm/acrn.h +5
-0+static inline long acrn_hypercall0 (unsigned long hcall_id) +{ + registerlong r8 asm("r8" ); +long result; +
Quoted 8 lines
arch/x86/include/asm/acrn.h void acrn_setup_intr_handler(void (*handler)(void)); · line 36
Quoted 6 lines
arch/x86/include/asm/acrn.h +6
-0+/* Nothing can come between the r8 assignment and the asm: */ + r8 = hcall_id; + asmvolatile ("vmcall\n\t" + :"=a" (result) + :"r" (r8) + : );What keeps an interrupt from happening between the r8 assignment and the
asm: ?
It's probably better phrased something like: "No other C code can come between this r8 assignment and the inline asm". An interrupt would actually be fine in there because interrupts save and restore all register state, including r8.
The problem (mentioned in the changelog) is that gcc does not let you place data directly into r8. But, it does allow you to declare a register variable that you can assign to use r8. There might be a problem if a function calls was in between and clobber the register, thus the "nothing can come between" comment.
The comment is really intended to scare away anyone from adding printk()'s.
More information about these register variables is here:
Quoted 1 line
https://gcc.gnu.org/onlinedocs/gcc/Local-Register-Variables.html#Local-Register-Variables
Any better ideas for comments would be greatly appreciated. It has 4 or 5 copies so I wanted it to be succinct.
Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces
Quoted 26 lines · Dave Hansen
Quoted 5 lines · Greg Kroah-Hartman
Quoted 5 lines
arch/x86/include/asm/acrn.h +5
-0+static inline long acrn_hypercall0 (unsigned long hcall_id) +{ + registerlong r8 asm("r8" ); +long result; +Quoted 8 lines
arch/x86/include/asm/acrn.h void acrn_setup_intr_handler(void (*handler)(void)); · line 36
Quoted 6 lines
arch/x86/include/asm/acrn.h +6
-0+/* Nothing can come between the r8 assignment and the asm: */ + r8 = hcall_id; + asmvolatile ("vmcall\n\t" + :"=a" (result) + :"r" (r8) + : );What keeps an interrupt from happening between the r8 assignment and the
asm: ?It's probably better phrased something like: "No other C code can come between this r8 assignment and the inline asm". An interrupt would actually be fine in there because interrupts save and restore all register state, including r8.
The problem (mentioned in the changelog) is that gcc does not let you place data directly into r8. But, it does allow you to declare a register variable that you can assign to use r8. There might be a problem if a function calls was in between and clobber the register, thus the "nothing can come between" comment.
Quoted 8 lines
The comment is really intended to scare away anyone from adding printk()'s.
More information about these register variables is here:
Quoted 1 line
https://gcc.gnu.org/onlinedocs/gcc/Local-Register-Variables.html#Local-Register-Variables
Any better ideas for comments would be greatly appreciated. It has 4 or 5 copies so I wanted it to be succinct.
This is disguisting.. Segher, does this actually work? Nick, does clang also support this?
Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces
Hi!
Quoted 31 lines · Peter Zijlstra
Quoted 30 lines · Dave Hansen
Quoted 5 lines · Greg Kroah-Hartman
Quoted 5 lines
arch/x86/include/asm/acrn.h +5
-0+static inline long acrn_hypercall0 (unsigned long hcall_id) +{ + registerlong r8 asm("r8" ); +long result; +Quoted 8 lines
arch/x86/include/asm/acrn.h void acrn_setup_intr_handler(void (*handler)(void)); · line 36
Quoted 6 lines
arch/x86/include/asm/acrn.h +6
-0+/* Nothing can come between the r8 assignment and the asm: */ + r8 = hcall_id; + asmvolatile ("vmcall\n\t" + :"=a" (result) + :"r" (r8) + : );What keeps an interrupt from happening between the r8 assignment and the
asm: ?It's probably better phrased something like: "No other C code can come between this r8 assignment and the inline asm". An interrupt would actually be fine in there because interrupts save and restore all register state, including r8.
The problem (mentioned in the changelog) is that gcc does not let you place data directly into r8. But, it does allow you to declare a register variable that you can assign to use r8. There might be a problem if a function calls was in between and clobber the register, thus the "nothing can come between" comment.
The comment is really intended to scare away anyone from adding printk()'s.
More information about these register variables is here:
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
https://gcc.gnu.org/onlinedocs/gcc/Local-Register-Variables.html#Local-Register-Variables
Any better ideas for comments would be greatly appreciated. It has 4 or 5 copies so I wanted it to be succinct.
This is disguisting.. Segher, does this actually work? Nick, does clang also support this?
The C variable "r8" is just a variable like any other; it can live in memory, or in any register, and different in all places, too. It can be moved around too; where "the assignment to it" happens is a philosophical question more than anything (the assignment there can be optimised away completely, for example; it is just a C variable, there is no magic).
Since this variable is a local register asm, on entry to the asm the compiler guarantees that the value lives in the assigned register (the "r8" hardware register in this case). This all works completely fine. This is the only guaranteed behaviour for local register asm (well, together with analogous behaviour for outputs).
If you want to *always* have it live in the hardware reg "r8", you have to use a global register asm, and almost certainly do that for all translation units, and use -ffixed-r8 as well. This of course is extremely costly.
Segher
Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces
Quoted 5 lines · Segher Boessenkool
Since this variable is a local register asm, on entry to the asm the compiler guarantees that the value lives in the assigned register (the "r8" hardware register in this case). This all works completely fine. This is the only guaranteed behaviour for local register asm (well, together with analogous behaviour for outputs).
Right, that's what they're trying to achieve. The hypervisor calling convention needs that variable in %r8 (which is somewhat unfortunate).
AFAIK this is the first such use in the kernel, but at least the gcc-4.9 (our oldest supported version) claims to support this.
So now we need to know if clang will actually do this too..
Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces
Quoted 9 lines · Peter Zijlstra
Quoted 5 lines · Segher Boessenkool
Since this variable is a local register asm, on entry to the asm the compiler guarantees that the value lives in the assigned register (the "r8" hardware register in this case). This all works completely fine. This is the only guaranteed behaviour for local register asm (well, together with analogous behaviour for outputs).
Quoted 7 lines
Right, that's what they're trying to achieve. The hypervisor calling convention needs that variable in %r8 (which is somewhat unfortunate).
AFAIK this is the first such use in the kernel, but at least the gcc-4.9 (our oldest supported version) claims to support this.
So now we need to know if clang will actually do this too..
Does clang support register local storage? Let's use godbolt.org to find out:
https://godbolt.org/z/YM45W5
Looks like yes. You can even check different GCC versions via the dropdown in the top right.
The -ffixed-* flags are less well supported in Clang; they need to be reimplemented on a per-backend basis. aarch64 is relatively well supported, but other arches not so much IME.
Do we need register local storage here?
static inline long bar(unsigned long hcall_id)
{
long result;
asm volatile("movl %1, %%r8d\n\t"
"vmcall\n\t"
: "=a" (result)
: "ir" (hcall_id)
: ); return result;
}
Signature · 2 lines
Thanks, ~Nick Desaulniers
Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces
Quoted 16 lines · Nick Desaulniers
Quoted 15 lines · Peter Zijlstra
Quoted 5 lines · Segher Boessenkool
Since this variable is a local register asm, on entry to the asm the compiler guarantees that the value lives in the assigned register (the "r8" hardware register in this case). This all works completely fine. This is the only guaranteed behaviour for local register asm (well, together with analogous behaviour for outputs).
Right, that's what they're trying to achieve. The hypervisor calling convention needs that variable in %r8 (which is somewhat unfortunate).
AFAIK this is the first such use in the kernel, but at least the gcc-4.9 (our oldest supported version) claims to support this.
Quoted 6 lines
Quoted 1 line
So now we need to know if clang will actually do this too..
Does clang support register local storage? Let's use godbolt.org to find out:
https://godbolt.org/z/YM45W5
Looks like yes. You can even check different GCC versions via the dropdown in the top right.
That only tells me it compiles it, not if that (IMO) weird construct is actually guaranteed to work as expected.
I'd almost dive into the GCC archives to read the back-story to this 'feature', it just seems to weird to me. A well, for another day that.
Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces
Quoted 8 lines · Peter Zijlstra
Quoted 2 lines
Looks like yes. You can even check different GCC versions via the dropdown in the top right.
That only tells me it compiles it, not if that (IMO) weird construct is actually guaranteed to work as expected.
I'd almost dive into the GCC archives to read the back-story to this 'feature', it just seems to weird to me. A well, for another day that.
It was documented in 1996 (<https://gcc.gnu.org/g:c1f7febfcb10>), the feature is older than that though.
In 2004 (in <https://gcc.gnu.org/g:805c33df1366>) the documentation for this was made more explicit (it has been rewritten since, but it still says the same thing).
Segher
Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces
In reply to Nick DesaulniersQuoted 9 lines · Nick Desaulniers
Quoted 8 lines · Peter Zijlstra
Quoted 5 lines · Segher Boessenkool
Since this variable is a local register asm, on entry to the asm the compiler guarantees that the value lives in the assigned register (the "r8" hardware register in this case). This all works completely fine. This is the only guaranteed behaviour for local register asm (well, together with analogous behaviour for outputs).
How strict is the guarantee? This is an inline function -- could the compiler decide to reorder some other code in between the r8 assignment and the asm statement when it gets inlined?
Quoted 25 lines
Quoted 8 lines
Right, that's what they're trying to achieve. The hypervisor calling convention needs that variable in %r8 (which is somewhat unfortunate).
AFAIK this is the first such use in the kernel, but at least the gcc-4.9 (our oldest supported version) claims to support this.
So now we need to know if clang will actually do this too..
Does clang support register local storage? Let's use godbolt.org to find out:
https://godbolt.org/z/YM45W5
Looks like yes. You can even check different GCC versions via the dropdown in the top right.The -ffixed-* flags are less well supported in Clang; they need to be reimplemented on a per-backend basis. aarch64 is relatively well supported, but other arches not so much IME.
Do we need register local storage here?
static inline long
bar(unsigned long hcall_id)
{
long result;
asm volatile("movl %1, %%r8d\n\t"
"vmcall\n\t"
Quoted 5 lines
: "=a" (result) : "ir" (hcall_id) : );return result;
}
This seems more robust, though you probably need an r8 clobber in there? Is hcall_id actually just 32 bits or can it be >=2^32?
Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces
Quoted 46 lines · Arvind Sankar
Quoted 9 lines · Nick Desaulniers
Quoted 8 lines · Peter Zijlstra
Quoted 5 lines · Segher Boessenkool
Since this variable is a local register asm, on entry to the asm the compiler guarantees that the value lives in the assigned register (the "r8" hardware register in this case). This all works completely fine. This is the only guaranteed behaviour for local register asm (well, together with analogous behaviour for outputs).
How strict is the guarantee? This is an inline function -- could the compiler decide to reorder some other code in between the r8 assignment and the asm statement when it gets inlined?
Quoted 25 lines
Quoted 8 lines
Right, that's what they're trying to achieve. The hypervisor calling convention needs that variable in %r8 (which is somewhat unfortunate).
AFAIK this is the first such use in the kernel, but at least the gcc-4.9 (our oldest supported version) claims to support this.
So now we need to know if clang will actually do this too..
Does clang support register local storage? Let's use godbolt.org to find out:
https://godbolt.org/z/YM45W5
Looks like yes. You can even check different GCC versions via the dropdown in the top right.The -ffixed-* flags are less well supported in Clang; they need to be reimplemented on a per-backend basis. aarch64 is relatively well supported, but other arches not so much IME.
Do we need register local storage here?
static inline long
bar(unsigned long hcall_id)
{
long result;
asm volatile("movl %1, %%r8d\n\t"
"vmcall\n\t"Quoted 5 lines
: "=a" (result) : "ir" (hcall_id) : );return result;
}
Quoted 2 lines
This seems more robust, though you probably need an r8 clobber in there? Is
hcall_idactually just 32 bits or can it be >=2^32?
Also, I think you need memory clobbers for all of these in either case, no?
Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces
In reply to Arvind SankarQuoted 11 lines · Arvind Sankar
Quoted 9 lines · Nick Desaulniers
Quoted 8 lines · Peter Zijlstra
Quoted 5 lines · Segher Boessenkool
Since this variable is a local register asm, on entry to the asm the compiler guarantees that the value lives in the assigned register (the "r8" hardware register in this case). This all works completely fine. This is the only guaranteed behaviour for local register asm (well, together with analogous behaviour for outputs).
Quoted 3 lines
How strict is the guarantee? This is an inline function -- could the compiler decide to reorder some other code in between the r8 assignment and the asm statement when it gets inlined?
Nope. It will be in r8 on entry to the asm. A guarantee is a guarantee; it is not a "yeah maybe, we'll see".
Quoted 13 lines
Quoted 7 lines
Do we need register local storage here?
static inline long
bar(unsigned long hcall_id)
{
long result;
asm volatile("movl %1, %%r8d\n\t"
"vmcall\n\t"Quoted 5 lines
: "=a" (result) : "ir" (hcall_id) : );return result;
}
Quoted 1 line
This seems more robust, though you probably need an r8 clobber in there?
Oh, x86 has the operand order inverted, so this should work in fact.
Segher
Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces
In reply to Nick DesaulniersQuoted 1 line · Nick Desaulniers
Do we need register local storage here?
Depends what you want. It looks like you do:
Quoted 10 lines
static inline long
bar(unsigned long hcall_id)
{
long result;
asm volatile("movl %1, %%r8d\n\t"
"vmcall\n\t": "=a" (result) : "ir" (hcall_id) : );return result;
}
"result" as output from the asm is in %rax, and the compiler will shuffle that to wherever it needs it as the function return value. That part will work fine.
But how you are accessing %r8d is not correct, that needs to be a local register asm (or r8 be made a fixed reg, probably not what you want ;-) )
Segher
Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces
Quoted 16 lines · Segher Boessenkool
Quoted 1 line · Nick Desaulniers
Do we need register local storage here?
Depends what you want. It looks like you do:
Quoted 10 lines
static inline long
bar(unsigned long hcall_id)
{
long result;
asm volatile("movl %1, %%r8d\n\t"
"vmcall\n\t": "=a" (result) : "ir" (hcall_id) : );return result;
}
Quoted 7 lines
"result" as output from the asm is in %rax, and the compiler will shuffle that to wherever it needs it as the function return value. That part will work fine.
But how you are accessing %r8d is not correct, that needs to be a local register asm (or r8 be made a fixed reg, probably not what you want ;-) )
Doesn't it just need an "r8" clobber to allow using r8d?
Quoted 2 lines
Segher
Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces
Quoted 17 lines · Arvind Sankar
Quoted 16 lines · Segher Boessenkool
Quoted 1 line · Nick Desaulniers
Do we need register local storage here?
Depends what you want. It looks like you do:
Quoted 10 lines
static inline long
bar(unsigned long hcall_id)
{
long result;
asm volatile("movl %1, %%r8d\n\t"
"vmcall\n\t": "=a" (result) : "ir" (hcall_id) : );return result;
}
Quoted 8 lines
Quoted 6 lines
"result" as output from the asm is in %rax, and the compiler will shuffle that to wherever it needs it as the function return value. That part will work fine.
But how you are accessing %r8d is not correct, that needs to be a local register asm (or r8 be made a fixed reg, probably not what you want ;-) )
Doesn't it just need an "r8" clobber to allow using r8d?
Yes, x86 asm is hard to read, what can I say :-) Sorry about that.
Segher
Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces
In reply to Nick DesaulniersQuoted 29 lines · Nick Desaulniers
Quoted 9 lines · Peter Zijlstra
Quoted 5 lines · Segher Boessenkool
Since this variable is a local register asm, on entry to the asm the compiler guarantees that the value lives in the assigned register (the "r8" hardware register in this case). This all works completely fine. This is the only guaranteed behaviour for local register asm (well, together with analogous behaviour for outputs).
Quoted 7 lines
Right, that's what they're trying to achieve. The hypervisor calling convention needs that variable in %r8 (which is somewhat unfortunate).
AFAIK this is the first such use in the kernel, but at least the gcc-4.9 (our oldest supported version) claims to support this.
So now we need to know if clang will actually do this too..
Does clang support register local storage? Let's use godbolt.org to find out:
https://godbolt.org/z/YM45W5
Looks like yes. You can even check different GCC versions via the dropdown in the top right.The -ffixed-* flags are less well supported in Clang; they need to be reimplemented on a per-backend basis. aarch64 is relatively well supported, but other arches not so much IME.
Do we need register local storage here?
Quoted 10 lines
static inline long
bar(unsigned long hcall_id)
{
long result;
asm volatile("movl %1, %%r8d\n\t"
"vmcall\n\t"
: "=a" (result)
: "ir" (hcall_id)
: );
return result;
}
Yeah, this approach is also mentioned in the changelog. I will change to this way to follow your preference. With an addtional "r8" clobber what Arvind mentioned.
Thanks
shuo
Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces
Quoted 41 lines · Shuo A Liu
Quoted 29 lines · Nick Desaulniers
Quoted 9 lines · Peter Zijlstra
Quoted 5 lines · Segher Boessenkool
Since this variable is a local register asm, on entry to the asm the compiler guarantees that the value lives in the assigned register (the "r8" hardware register in this case). This all works completely fine. This is the only guaranteed behaviour for local register asm (well, together with analogous behaviour for outputs).
Quoted 7 lines
Right, that's what they're trying to achieve. The hypervisor calling convention needs that variable in %r8 (which is somewhat unfortunate).
AFAIK this is the first such use in the kernel, but at least the gcc-4.9 (our oldest supported version) claims to support this.
So now we need to know if clang will actually do this too..
Does clang support register local storage? Let's use godbolt.org to find out:
https://godbolt.org/z/YM45W5
Looks like yes. You can even check different GCC versions via the dropdown in the top right.The -ffixed-* flags are less well supported in Clang; they need to be reimplemented on a per-backend basis. aarch64 is relatively well supported, but other arches not so much IME.
Do we need register local storage here?
Quoted 10 lines
static inline long
bar(unsigned long hcall_id)
{
long result;
asm volatile("movl %1, %%r8d\n\t"
"vmcall\n\t"
: "=a" (result)
: "ir" (hcall_id)
: );
return result;
}
Quoted 6 lines
Yeah, this approach is also mentioned in the changelog. I will change to this way to follow your preference. With an addtional "r8" clobber what Arvind mentioned.
Thanks
shuo
Btw, I noticed that arch/x86/xen/hypercall.h uses register-local variables already for its hypercalls for quite some time, so this wouldn't be unprecedented. [0]
Do these calls also need a memory clobber? The KVM/xen hypercall functions all have one.
Thanks.
[0] e74359028d548 ("xen64: fix calls into hypercall page")
Re: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces
Quoted 49 lines · Arvind Sankar
Quoted 41 lines · Shuo A Liu
Quoted 29 lines · Nick Desaulniers
Quoted 9 lines · Peter Zijlstra
Quoted 5 lines · Segher Boessenkool
Since this variable is a local register asm, on entry to the asm the compiler guarantees that the value lives in the assigned register (the "r8" hardware register in this case). This all works completely fine. This is the only guaranteed behaviour for local register asm (well, together with analogous behaviour for outputs).
Quoted 7 lines
Right, that's what they're trying to achieve. The hypervisor calling convention needs that variable in %r8 (which is somewhat unfortunate).
AFAIK this is the first such use in the kernel, but at least the gcc-4.9 (our oldest supported version) claims to support this.
So now we need to know if clang will actually do this too..
Does clang support register local storage? Let's use godbolt.org to find out:
https://godbolt.org/z/YM45W5
Looks like yes. You can even check different GCC versions via the dropdown in the top right.The -ffixed-* flags are less well supported in Clang; they need to be reimplemented on a per-backend basis. aarch64 is relatively well supported, but other arches not so much IME.
Do we need register local storage here?
Quoted 10 lines
static inline long
bar(unsigned long hcall_id)
{
long result;
asm volatile("movl %1, %%r8d\n\t"
"vmcall\n\t"
: "=a" (result)
: "ir" (hcall_id)
: );
return result;
}Quoted 6 lines
Yeah, this approach is also mentioned in the changelog. I will change to this way to follow your preference. With an addtional "r8" clobber what Arvind mentioned.
Thanks
shuo
Quoted 6 lines
Btw, I noticed that
arch/x86/xen/hypercall.huses register-local variables already for its hypercalls for quite some time, so this wouldn't be unprecedented. [0]Do these calls also need a memory clobber? The KVM/xen hypercall functions all have one.
Yes. it's needed. I will add it. Thanks
Quoted 4 lines
Thanks.
[0]
e74359028d548("xen64: fix calls into hypercall page")
Borislav Petkov
maintainerRe: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces
In reply to [email protected]Quoted 9 lines · On Tue, Sep 22, 2020 at 07:42:58PM +0800, [email protected] wrote:
From: Shuo Liu <[email protected]>
The Service VM communicates with the hypervisor via conventional hypercalls. VMCALL instruction is used to make the hypercalls.
ACRN hypercall ABI:
* Hypercall number is in R8 register.
* Up to 2 parameters are in RDI and RSI registers.
* Return value is in RAX register.
I'm assuming this is already cast in stone in the HV and it cannot be changed?
Quoted 15 lines
Introduce the ACRN hypercall interfaces. Because GCC doesn't support R8 register as direct register constraints, here are two ways to use R8 in extended asm:
1) use explicit register variable as input
2) use supported constraint as input with a explicit MOV to R8 in beginning of asmThe number of instructions of above two ways are same. Asm code from 1)
38: 41 b8 00 00 00 80 mov $0x80000000,%r8d 3e: 48 89 c7 mov %rax,%rdi 41: 0f 01 c1 vmcallHere, writes to the lower dword (%r8d) clear the upper dword of %r8 when the CPU is in 64-bit mode.
Quoted 8 lines
Asm code from 2)
38: 48 89 c7 mov %rax,%rdi 3b: 49 b8 00 00 00 80 00 movabs $0x80000000,%r8 42: 00 00 00 45: 0f 01 c1 vmcallChoose 1) for code simplicity and a little bit of code size optimization.
What?
How much "optimization" is this actually? A couple of bytes?
And all that for this
/* Nothing can come between the r8 assignment and the asm: */
restriction?
If it is only a couple of bytes, just do the explicit MOV to %r8 and f'get about it.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v4 04/17] x86/acrn: Introduce hypercall interfaces
Quoted 6 lines · Borislav Petkov
Quoted 5 lines · On Tue, Sep 22, 2020 at 07:42:58PM +0800, [email protected] wrote:
From: Shuo Liu <[email protected]>
The Service VM communicates with the hypervisor via conventional hypercalls. VMCALL instruction is used to make the hypercalls.
Quoted 7 lines
Quoted 4 lines
ACRN hypercall ABI:
* Hypercall number is in R8 register.
* Up to 2 parameters are in RDI and RSI registers.
* Return value is in RAX register.I'm assuming this is already cast in stone in the HV and it cannot be changed?
Yes, it is.
Quoted 29 lines
Quoted 16 lines
Introduce the ACRN hypercall interfaces. Because GCC doesn't support R8 register as direct register constraints, here are two ways to use R8 in extended asm:
1) use explicit register variable as input
2) use supported constraint as input with a explicit MOV to R8 in beginning of asmThe number of instructions of above two ways are same. Asm code from 1)
38: 41 b8 00 00 00 80 mov $0x80000000,%r8d 3e: 48 89 c7 mov %rax,%rdi 41: 0f 01 c1 vmcallHere, writes to the lower dword (%r8d) clear the upper dword of %r8 when the CPU is in 64-bit mode.
Quoted 8 lines
Asm code from 2)
38: 48 89 c7 mov %rax,%rdi 3b: 49 b8 00 00 00 80 00 movabs $0x80000000,%r8 42: 00 00 00 45: 0f 01 c1 vmcallChoose 1) for code simplicity and a little bit of code size optimization.
What?
How much "optimization" is this actually? A couple of bytes?
Quoted 8 lines
And all that for this
/* Nothing can come between the r8 assignment and the asm: */restriction?
If it is only a couple of bytes, just do the explicit MOV to %r8 and f'get about it.
Yes. Just a couple of bytes. Number of instructions is same. sure, i can change to approach 2)
2) use supported constraint as input with a explicit MOV to R8 in beginning of asm
Thanks
shuo
PATCH v4 5/17 virt: acrn: Introduce ACRN HSM basic driver
In reply to [email protected]From: Shuo Liu <[email protected]>
ACRN Hypervisor Service Module (HSM) is a kernel module in Service VM which communicates with ACRN userspace through ioctls and talks to ACRN Hypervisor through hypercalls.
Add a basic HSM driver which allows Service VM userspace to communicate with ACRN. The following patches will add more ioctls, guest VM memory mapping caching, I/O request processing, ioeventfd and irqfd into this module. HSM exports a char device interface (/dev/acrn_hsm) to userspace.
--- MAINTAINERS | 1 + drivers/virt/Kconfig | 2 + drivers/virt/Makefile | 1 + drivers/virt/acrn/Kconfig | 14 ++++++ drivers/virt/acrn/Makefile | 3 ++ drivers/virt/acrn/acrn_drv.h | 18 ++++++++ drivers/virt/acrn/hsm.c | 87 ++++++++++++++++++++++++++++++++++++ 7 files changed, 126 insertions(+) create mode 100644 drivers/virt/acrn/Kconfig create mode 100644 drivers/virt/acrn/Makefile create mode 100644 drivers/virt/acrn/acrn_drv.h create mode 100644 drivers/virt/acrn/hsm.c
MAINTAINERS +1 -0
@@ -442,6 +442,7 @@ L: [email protected] S: Supported W: https://projectacrn.org F: Documentation/virt/acrn/ +F: drivers/virt/acrn/ AD1889 ALSA SOUND DRIVER L: [email protected]
drivers/virt/Kconfig +2 -0
@@ -32,4 +32,6 @@ config FSL_HV_MANAGER partition shuts down. source "drivers/virt/vboxguest/Kconfig" + +source "drivers/virt/acrn/Kconfig" endif
drivers/virt/Makefile +1 -0
@@ -5,3 +5,4 @@ obj-$(CONFIG_FSL_HV_MANAGER) += fsl_hypervisor.o obj-y += vboxguest/ +obj-$(CONFIG_ACRN_HSM) += acrn/
drivers/virt/acrn/Kconfig new file +14 -0
@@ -0,0 +1,14 @@ +# SPDX-License-Identifier: GPL-2.0 +config ACRN_HSM + tristate "ACRN Hypervisor Service Module" + depends on ACRN_GUEST + help + ACRN Hypervisor Service Module (HSM) is a kernel module which + communicates with ACRN userspace through ioctls and talks to + the ACRN Hypervisor through hypercalls. HSM will only run in + a privileged management VM, called Service VM, to manage User + VMs and do I/O emulation. Not required for simply running + under ACRN as a User VM. + + To compile as a module, choose M, the module will be called + acrn. If unsure, say N.
drivers/virt/acrn/Makefile new file +3 -0
@@ -0,0 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0 +obj-$(CONFIG_ACRN_HSM) := acrn.o +acrn-y := hsm.o
drivers/virt/acrn/acrn_drv.h new file +18 -0
@@ -0,0 +1,18 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef __ACRN_HSM_DRV_H +#define __ACRN_HSM_DRV_H + +#include <linux/types.h> + +#define ACRN_INVALID_VMID (0xffffU) + +/** +* struct acrn_vm - Properties of ACRN User VM. +* @vmid: User VM ID +*/ +struct acrn_vm { +u16 vmid; +}; + +#endif /* __ACRN_HSM_DRV_H */
drivers/virt/acrn/hsm.c new file +87 -0
87 lines not on this page · raw message
Signature · 1 line
2.28.0
PATCH v4 8/17 virt: acrn: Introduce EPT mapping management
In reply to [email protected]From: Shuo Liu <[email protected]>
The HSM provides hypervisor services to the ACRN userspace. While launching a User VM, ACRN userspace needs to allocate memory and request the ACRN Hypervisor to set up the EPT mapping for the VM.
A mapping cache is introduced for accelerating the translation between the Service VM kernel virtual address and User VM physical address.
From the perspective of the hypervisor, the types of GPA of User VM can be listed as following:
1) RAM region, which is used by User VM as system ram.
2) MMIO region, which is recognized by User VM as MMIO. MMIO region is used to be utilized for devices emulation.
Generally, User VM RAM regions mapping is set up before VM started and is released in the User VM destruction. MMIO regions mapping may be set and unset dynamically during User VM running.
To achieve this, ioctls ACRN_IOCTL_SET_MEMSEG and ACRN_IOCTL_UNSET_MEMSEG are introduced in HSM.
--- drivers/virt/acrn/Makefile | 2 +- drivers/virt/acrn/acrn_drv.h | 98 ++++++++++- drivers/virt/acrn/hsm.c | 15 ++ drivers/virt/acrn/hypercall.h | 14 ++ drivers/virt/acrn/mm.c | 305 ++++++++++++++++++++++++++++++++++ drivers/virt/acrn/vm.c | 4 + include/uapi/linux/acrn.h | 51 ++++++ 7 files changed, 479 insertions(+), 10 deletions(-) create mode 100644 drivers/virt/acrn/mm.c
drivers/virt/acrn/Makefile +1 -1
@@ -1,3 +1,3 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_ACRN_HSM) := acrn.o -acrn-y := hsm.o vm.o +acrn-y := hsm.o vm.o mm.o
drivers/virt/acrn/acrn_drv.h +89 -9
116 lines not on this page · raw message
drivers/virt/acrn/hsm.c +15 -0
@@ -48,6 +48,7 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,struct acrn_vm *vm = filp->private_data;struct acrn_vm_creation *vm_param;struct acrn_vcpu_regs *cpu_regs; +struct acrn_vm_memmap memmap;int ret =0 ;if (vm->vmid ==ACRN_INVALID_VMID && cmd !=ACRN_IOCTL_CREATE_VM ) {
@@ -112,6 +113,20 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd, vm->vmid);kfree (cpu_regs);break ; +case ACRN_IOCTL_SET_MEMSEG : +if (copy_from_user (&memmap, (void __user *)ioctl_param, +sizeof (memmap))) +return -EFAULT ; + + ret =acrn_vm_memseg_map (vm, &memmap); +break ; +case ACRN_IOCTL_UNSET_MEMSEG : +if (copy_from_user (&memmap, (void __user *)ioctl_param, +sizeof (memmap))) +return -EFAULT ; + + ret =acrn_vm_memseg_unmap (vm, &memmap); +break ;default :dev_warn (acrn_dev.this_device,"Unknown IOCTL 0x%x!\n" , cmd); ret = -ENOTTY ;
drivers/virt/acrn/hypercall.h +14 -0
@@ -21,6 +21,9 @@#define HC_RESET_VM _HC_ID(HC_ID, HC_ID_VM_BASE + 0x05)#define HC_SET_VCPU_REGS _HC_ID(HC_ID, HC_ID_VM_BASE + 0x06) +#define HC_ID_MEM_BASE 0x40UL +#define HC_VM_SET_MEMORY_REGIONS _HC_ID(HC_ID, HC_ID_MEM_BASE + 0x02) +/** * hcall_create_vm() - Create a User VM * @vminfo: Service VM GPA of info of User VM creation
@@ -88,4 +91,15 @@ static inline long hcall_set_vcpu_regs(u64 vmid, u64 regs_state) return acrn_hypercall2(HC_SET_VCPU_REGS, vmid, regs_state); } +/** +* hcall_set_memory_regions() - Inform the hypervisor to set up EPT mappings +* @regions_pa: Service VM GPA of &struct vm_memory_region_batch +* +* Return: 0 on success, <0 on failure +*/ +static inline long hcall_set_memory_regions (u64 regions_pa) +{ +return acrn_hypercall1 (HC_VM_SET_MEMORY_REGIONS , regions_pa); +} +#endif /* __ACRN_HSM_HYPERCALL_H */
drivers/virt/acrn/mm.c new file +305 -0
305 lines not on this page · raw message
drivers/virt/acrn/vm.c +4 -0
@@ -34,6 +34,7 @@ struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,return NULL ; } +mutex_init (&vm->regions_mapping_lock); vm->vmid = vm_param->vmid; vm->vcpu_num = vm_param->vcpu_num;
@@ -65,6 +66,9 @@ int acrn_vm_destroy(struct acrn_vm *vm)clear_bit (ACRN_VM_FLAG_DESTROYED , &vm->flags);return ret; } + +acrn_vm_all_ram_unmap (vm); +dev_dbg (acrn_dev.this_device,"VM %u destroyed.\n" , vm->vmid); vm->vmid =ACRN_INVALID_VMID ;return 0 ;
include/uapi/linux/acrn.h +51 -0
61 lines not on this page · raw message
Signature · 1 line
2.28.0
PATCH v4 10/17 virt: acrn: Introduce PCI configuration space PIO accesses combiner
In reply to [email protected]From: Shuo Liu <[email protected]>
A User VM can access its virtual PCI configuration spaces via port IO approach, which has two following steps:
1) writes address into port 0xCF8
2) put/get data in/from port 0xCFC
To distribute a complete PCI configuration space access one time, HSM need to combine such two accesses together.
Combine two paired PIO I/O requests into one PCI I/O request and continue the I/O request distribution.
--- drivers/virt/acrn/acrn_drv.h | 2 + drivers/virt/acrn/ioreq.c | 76 ++++++++++++++++++++++++++++++++++++ include/uapi/linux/acrn.h | 15 +++++++ 3 files changed, 93 insertions(+)
drivers/virt/acrn/acrn_drv.h +2 -0
@@ -156,6 +156,7 @@ extern rwlock_t acrn_vm_list_lock;* @default_client: The default I/O request client * @ioreq_buf: I/O request shared buffer * @ioreq_page: The page of the I/O request shared buffer +* @pci_conf_addr: Address of a PCI configuration access emulation */ struct acrn_vm {struct list_head list;
@@ -170,6 +171,7 @@ struct acrn_vm {struct acrn_ioreq_client *default_client;struct acrn_io_request_buffer *ioreq_buf;struct page *ioreq_page; +u32 pci_conf_addr; }; struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
drivers/virt/acrn/ioreq.c +76 -0
88 lines not on this page · raw message
include/uapi/linux/acrn.h +15 -0
@@ -20,6 +20,7 @@#define ACRN_IOREQ_TYPE_PORTIO 0#define ACRN_IOREQ_TYPE_MMIO 1 +#define ACRN_IOREQ_TYPE_PCICFG 2#define ACRN_IOREQ_DIR_READ 0#define ACRN_IOREQ_DIR_WRITE 1
@@ -40,6 +41,18 @@ struct acrn_pio_request {__u32 value; } __attribute__((aligned(8))); +/* Need keep same header fields with pio_request */ +struct acrn_pci_request { +__u32 direction; +__u32 reserved[3 ]; +__u64 size; +__u32 value; +__u32 bus; +__u32 dev; +__u32 func; +__u32 reg; +} __attribute__((aligned (8 ))); +/** * struct acrn_io_request - 256-byte ACRN I/O request * @type: Type of this request (ACRN_IOREQ_TYPE_*).
@@ -48,6 +61,7 @@ struct acrn_pio_request {* @reserved0: Reserved fields. * @reqs: Union of different types of request. Byte offset: 64. * @reqs.pio_request: PIO request data of the I/O request. +* @reqs.pci_request: PCI configuration space request data of the I/O request. * @reqs.mmio_request: MMIO request data of the I/O request. * @reqs.data: Raw data of the I/O request. * @reserved1: Reserved fields.
@@ -107,6 +121,7 @@ struct acrn_io_request {__u32 reserved0[14 ];union {struct acrn_pio_request pio_request; +struct acrn_pci_request pci_request;struct acrn_mmio_request mmio_request;__u64 data[8 ]; } reqs;
Signature · 1 line
2.28.0
PATCH v4 11/17 virt: acrn: Introduce interfaces for PCI device passthrough
In reply to [email protected]From: Shuo Liu <[email protected]>
PCI device passthrough enables an OS in a virtual machine to directly access a PCI device in the host. It promises almost the native performance, which is required in performance-critical scenarios of ACRN.
HSM provides the following ioctls:
- Assign - ACRN_IOCTL_ASSIGN_PCIDEV
Pass data struct acrn_pcidev from userspace to the hypervisor, and inform the hypervisor to assign a PCI device to a User VM.
- De-assign - ACRN_IOCTL_DEASSIGN_PCIDEV
Pass data struct acrn_pcidev from userspace to the hypervisor, and inform the hypervisor to de-assign a PCI device from a User VM.
- Set a interrupt of a passthrough device - ACRN_IOCTL_SET_PTDEV_INTR Pass data struct acrn_ptdev_irq from userspace to the hypervisor, and inform the hypervisor to map a INTx interrupt of passthrough device of User VM.
- Reset passthrough device interrupt - ACRN_IOCTL_RESET_PTDEV_INTR Pass data struct acrn_ptdev_irq from userspace to the hypervisor, and inform the hypervisor to unmap a INTx interrupt of passthrough device of User VM.
--- drivers/virt/acrn/hsm.c | 50 +++++++++++++++++++++++++++ drivers/virt/acrn/hypercall.h | 54 ++++++++++++++++++++++++++++++ include/uapi/linux/acrn.h | 63 +++++++++++++++++++++++++++++++++++ 3 files changed, 167 insertions(+)
drivers/virt/acrn/hsm.c +50 -0
@@ -49,7 +49,9 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,struct acrn_vm_creation *vm_param;struct acrn_vcpu_regs *cpu_regs;struct acrn_ioreq_notify notify; +struct acrn_ptdev_irq *irq_info;struct acrn_vm_memmap memmap; +struct acrn_pcidev *pcidev;int ret =0 ;if (vm->vmid ==ACRN_INVALID_VMID && cmd !=ACRN_IOCTL_CREATE_VM ) {
@@ -128,6 +130,54 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd, ret =acrn_vm_memseg_unmap (vm, &memmap);break ; +case ACRN_IOCTL_ASSIGN_PCIDEV : + pcidev =memdup_user ((void __user *)ioctl_param, +sizeof (struct acrn_pcidev )); +if (IS_ERR (pcidev)) +return PTR_ERR (pcidev); + + ret =hcall_assign_pcidev (vm->vmid,virt_to_phys (pcidev)); +if (ret <0 ) +dev_err (acrn_dev.this_device, +"Failed to assign pci device!\n" ); +kfree (pcidev); +break ; +case ACRN_IOCTL_DEASSIGN_PCIDEV : + pcidev =memdup_user ((void __user *)ioctl_param, +sizeof (struct acrn_pcidev )); +if (IS_ERR (pcidev)) +return PTR_ERR (pcidev); + + ret =hcall_deassign_pcidev (vm->vmid,virt_to_phys (pcidev)); +if (ret <0 ) +dev_err (acrn_dev.this_device, +"Failed to deassign pci device!\n" ); +kfree (pcidev); +break ; +case ACRN_IOCTL_SET_PTDEV_INTR : + irq_info =memdup_user ((void __user *)ioctl_param, +sizeof (struct acrn_ptdev_irq )); +if (IS_ERR (irq_info)) +return PTR_ERR (irq_info); + + ret =hcall_set_ptdev_intr (vm->vmid,virt_to_phys (irq_info)); +if (ret <0 ) +dev_err (acrn_dev.this_device, +"Failed to configure intr for ptdev!\n" ); +kfree (irq_info); +break ; +case ACRN_IOCTL_RESET_PTDEV_INTR : + irq_info =memdup_user ((void __user *)ioctl_param, +sizeof (struct acrn_ptdev_irq )); +if (IS_ERR (irq_info)) +return PTR_ERR (irq_info); + + ret =hcall_reset_ptdev_intr (vm->vmid,virt_to_phys (irq_info)); +if (ret <0 ) +dev_err (acrn_dev.this_device, +"Failed to reset intr for ptdev!\n" ); +kfree (irq_info); +break ;case ACRN_IOCTL_CREATE_IOREQ_CLIENT :if (vm->default_client)return -EEXIST ;
drivers/virt/acrn/hypercall.h +54 -0
64 lines not on this page · raw message
include/uapi/linux/acrn.h +63 -0
73 lines not on this page · raw message
Signature · 1 line
2.28.0
PATCH v4 12/17 virt: acrn: Introduce interrupt injection interfaces
In reply to [email protected]From: Shuo Liu <[email protected]>
ACRN userspace need to inject virtual interrupts into a User VM in devices emulation.
HSM needs provide interfaces to do so.
Introduce following interrupt injection interfaces:
ioctl ACRN_IOCTL_SET_IRQLINE:
Pass data from userspace to the hypervisor, and inform the hypervisor to inject a virtual IOAPIC GSI interrupt to a User VM.
ioctl ACRN_IOCTL_INJECT_MSI:
Pass data struct acrn_msi_entry from userspace to the hypervisor, and inform the hypervisor to inject a virtual MSI to a User VM.
ioctl ACRN_IOCTL_VM_INTR_MONITOR:
Set a 4-Kbyte aligned shared page for statistics information of interrupts of a User VM.
--- drivers/virt/acrn/acrn_drv.h | 4 ++++ drivers/virt/acrn/hsm.c | 39 +++++++++++++++++++++++++++++++++ drivers/virt/acrn/hypercall.h | 41 +++++++++++++++++++++++++++++++++++ drivers/virt/acrn/vm.c | 36 ++++++++++++++++++++++++++++++ include/uapi/linux/acrn.h | 17 +++++++++++++++ 5 files changed, 137 insertions(+)
drivers/virt/acrn/acrn_drv.h +4 -0
@@ -157,6 +157,7 @@ extern rwlock_t acrn_vm_list_lock;* @ioreq_buf: I/O request shared buffer * @ioreq_page: The page of the I/O request shared buffer * @pci_conf_addr: Address of a PCI configuration access emulation +* @monitor_page: Page of interrupt statistics of User VM */ struct acrn_vm {struct list_head list;
@@ -172,6 +173,7 @@ struct acrn_vm {struct acrn_io_request_buffer *ioreq_buf;struct page *ioreq_page;u32 pci_conf_addr; +struct page *monitor_page; }; struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
@@ -198,4 +200,6 @@ struct acrn_ioreq_client *acrn_ioreq_client_create(struct acrn_vm *vm, const char *name);void acrn_ioreq_client_destroy (struct acrn_ioreq_client *client); +int acrn_msi_inject (struct acrn_vm *vm,u64 msi_addr,u64 msi_data); +#endif /* __ACRN_HSM_DRV_H */
drivers/virt/acrn/hsm.c +39 -0
@@ -51,7 +51,9 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,struct acrn_ioreq_notify notify;struct acrn_ptdev_irq *irq_info;struct acrn_vm_memmap memmap; +struct acrn_msi_entry *msi;struct acrn_pcidev *pcidev; +struct page *page;int ret =0 ;if (vm->vmid ==ACRN_INVALID_VMID && cmd !=ACRN_IOCTL_CREATE_VM ) {
@@ -178,6 +180,43 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd, "Failed to reset intr for ptdev!\n");kfree (irq_info);break ; +case ACRN_IOCTL_SET_IRQLINE : + ret =hcall_set_irqline (vm->vmid, ioctl_param); +if (ret <0 ) +dev_err (acrn_dev.this_device, +"Failed to set interrupt line!\n" ); +break ; +case ACRN_IOCTL_INJECT_MSI : + msi =memdup_user ((void __user *)ioctl_param, +sizeof (struct acrn_msi_entry )); +if (IS_ERR (msi)) +return PTR_ERR (msi); + + ret =hcall_inject_msi (vm->vmid,virt_to_phys (msi)); +if (ret <0 ) +dev_err (acrn_dev.this_device, +"Failed to inject MSI!\n" ); +kfree (msi); +break ; +case ACRN_IOCTL_VM_INTR_MONITOR : + ret =get_user_pages_fast (ioctl_param,1 ,FOLL_WRITE , &page); +if (unlikely (ret !=1 )) { +dev_err (acrn_dev.this_device, +"Failed to pin intr hdr buffer!\n" ); +return -EFAULT ; + } + + ret =hcall_vm_intr_monitor (vm->vmid,page_to_phys (page)); +if (ret <0 ) { +put_page (page); +dev_err (acrn_dev.this_device, +"Failed to monitor intr data!\n" ); +return ret; + } +if (vm->monitor_page) +put_page (vm->monitor_page); + vm->monitor_page = page; +break ;case ACRN_IOCTL_CREATE_IOREQ_CLIENT :if (vm->default_client)return -EEXIST ;
drivers/virt/acrn/hypercall.h +41 -0
53 lines not on this page · raw message
drivers/virt/acrn/vm.c +36 -0
@@ -68,6 +68,10 @@ int acrn_vm_destroy(struct acrn_vm *vm)write_unlock_bh (&acrn_vm_list_lock);acrn_ioreq_deinit (vm); +if (vm->monitor_page) { +put_page (vm->monitor_page); + vm->monitor_page =NULL ; + } ret =hcall_destroy_vm (vm->vmid);if (ret <0 ) {
@@ -83,3 +87,35 @@ int acrn_vm_destroy(struct acrn_vm *vm) vm->vmid =ACRN_INVALID_VMID ;return 0 ; } + +/** +* acrn_inject_msi() - Inject a MSI interrupt into a User VM +* @vm: User VM +* @msi_addr: The MSI address +* @msi_data: The MSI data +* +* Return: 0 on success, <0 on error +*/ +int acrn_msi_inject (struct acrn_vm *vm,u64 msi_addr,u64 msi_data) +{ +struct acrn_msi_entry *msi; +int ret; + +/* might be used in interrupt context, so use GFP_ATOMIC */ + msi =kzalloc (sizeof (*msi),GFP_ATOMIC ); +if (!msi) +return -ENOMEM ; + +/* +* msi_addr: addr[19:12] with dest vcpu id +* msi_data: data[7:0] with vector +*/ + msi->msi_addr = msi_addr; + msi->msi_data = msi_data; + ret =hcall_inject_msi (vm->vmid,virt_to_phys (msi)); +if (ret <0 ) +dev_err (acrn_dev.this_device, +"Failed to inject MSI to VM %u!\n" , vm->vmid); +kfree (msi); +return ret; +}
include/uapi/linux/acrn.h +17 -0
@@ -343,6 +343,16 @@ struct acrn_pcidev {__u32 reserved[6 ]; } __attribute__((aligned(8))); +/** +* struct acrn_msi_entry - Info for injecting a MSI interrupt to a VM +* @msi_addr: MSI addr[19:12] with dest vCPU ID +* @msi_data: MSI data[7:0] with vector +*/ +struct acrn_msi_entry { +__u64 msi_addr; +__u64 msi_data; +}; +/* The ioctl type, documented in ioctl-number.rst */ #define ACRN_IOCTL_TYPE 0xA2
@@ -362,6 +372,13 @@ struct acrn_pcidev {#define ACRN_IOCTL_SET_VCPU_REGS \ _IOW(ACRN_IOCTL_TYPE, 0x16, struct acrn_vcpu_regs) +#define ACRN_IOCTL_INJECT_MSI \ + _IOW(ACRN_IOCTL_TYPE, 0x23, struct acrn_msi_entry) +#define ACRN_IOCTL_VM_INTR_MONITOR \ + _IOW(ACRN_IOCTL_TYPE, 0x24, unsigned long) +#define ACRN_IOCTL_SET_IRQLINE \ + _IOW(ACRN_IOCTL_TYPE, 0x25, __u64) +#define ACRN_IOCTL_NOTIFY_REQUEST_FINISH \ _IOW(ACRN_IOCTL_TYPE, 0x31, struct acrn_ioreq_notify)#define ACRN_IOCTL_CREATE_IOREQ_CLIENT \
Signature · 1 line
2.28.0
PATCH v4 14/17 virt: acrn: Introduce I/O ranges operation interfaces
In reply to [email protected]From: Shuo Liu <[email protected]>
An I/O request of a User VM, which is constructed by hypervisor, is distributed by the ACRN Hypervisor Service Module to an I/O client corresponding to the address range of the I/O request.
I/O client maintains a list of address ranges. Introduce acrn_ioreq_range_{add,del}() to manage these address ranges.
--- drivers/virt/acrn/acrn_drv.h | 4 +++ drivers/virt/acrn/ioreq.c | 60 ++++++++++++++++++++++++++++++++++++ 2 files changed, 64 insertions(+)
drivers/virt/acrn/acrn_drv.h +4 -0
@@ -199,6 +199,10 @@ struct acrn_ioreq_client *acrn_ioreq_client_create(struct acrn_vm *vm, void *data, bool is_default, const char *name);void acrn_ioreq_client_destroy (struct acrn_ioreq_client *client); +int acrn_ioreq_range_add (struct acrn_ioreq_client *client, +u32 type,u64 start,u64 end); +void acrn_ioreq_range_del (struct acrn_ioreq_client *client, +u32 type,u64 start,u64 end);int acrn_msi_inject (struct acrn_vm *vm,u64 msi_addr,u64 msi_data);
drivers/virt/acrn/ioreq.c +60 -0
66 lines not on this page · raw message
Signature · 1 line
2.28.0
PATCH v4 16/17 virt: acrn: Introduce irqfd
In reply to [email protected]From: Shuo Liu <[email protected]>
irqfd is a mechanism to inject a specific interrupt to a User VM using a decoupled eventfd mechanism.
Vhost is a kernel-level virtio server which uses eventfd for interrupt injection. To support vhost on ACRN, irqfd is introduced in HSM.
HSM provides ioctls to associate a virtual Message Signaled Interrupt (MSI) with an eventfd. The corresponding virtual MSI will be injected into a User VM once the eventfd got signal.
--- drivers/virt/acrn/Makefile | 2 +- drivers/virt/acrn/acrn_drv.h | 10 ++ drivers/virt/acrn/hsm.c | 7 ++ drivers/virt/acrn/irqfd.c | 235 +++++++++++++++++++++++++++++++++++ drivers/virt/acrn/vm.c | 3 + include/uapi/linux/acrn.h | 15 +++ 6 files changed, 271 insertions(+), 1 deletion(-) create mode 100644 drivers/virt/acrn/irqfd.c
drivers/virt/acrn/Makefile +1 -1
@@ -1,3 +1,3 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_ACRN_HSM) := acrn.o -acrn-y := hsm.o vm.o mm.o ioreq.o ioeventfd.o +acrn-y := hsm.o vm.o mm.o ioreq.o ioeventfd.o irqfd.o
drivers/virt/acrn/acrn_drv.h +10 -0
@@ -161,6 +161,9 @@ extern rwlock_t acrn_vm_list_lock;* @ioeventfds_lock: Lock to protect ioeventfds list * @ioeventfds: List to link all hsm_ioeventfd * @ioeventfd_client: I/O client for ioeventfds of the VM +* @irqfds_lock: Lock to protect irqfds list +* @irqfds: List to link all hsm_irqfd +* @irqfd_wq: Workqueue for irqfd async shutdown */ struct acrn_vm {struct list_head list;
@@ -180,6 +183,9 @@ struct acrn_vm {struct mutex ioeventfds_lock;struct list_head ioeventfds;struct acrn_ioreq_client *ioeventfd_client; +struct mutex irqfds_lock; +struct list_head irqfds; +struct workqueue_struct *irqfd_wq; }; struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,
@@ -216,4 +222,8 @@ int acrn_ioeventfd_init(struct acrn_vm *vm);int acrn_ioeventfd_config (struct acrn_vm *vm,struct acrn_ioeventfd *args);void acrn_ioeventfd_deinit (struct acrn_vm *vm); +int acrn_irqfd_init (struct acrn_vm *vm); +int acrn_irqfd_config (struct acrn_vm *vm,struct acrn_irqfd *args); +void acrn_irqfd_deinit (struct acrn_vm *vm); +#endif /* __ACRN_HSM_DRV_H */
drivers/virt/acrn/hsm.c +7 -0
@@ -115,6 +115,7 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd,struct acrn_vm_memmap memmap;struct acrn_msi_entry *msi;struct acrn_pcidev *pcidev; +struct acrn_irqfd irqfd;struct page *page;u64 cstate_cmd;int ret =0 ;
@@ -317,6 +318,12 @@ static long acrn_dev_ioctl(struct file *filp, unsigned int cmd, ret =acrn_ioeventfd_config (vm, &ioeventfd);break ; +case ACRN_IOCTL_IRQFD : +if (copy_from_user (&irqfd, (void __user *)ioctl_param, +sizeof (irqfd))) +return -EFAULT ; + ret =acrn_irqfd_config (vm, &irqfd); +break ;default :dev_warn (acrn_dev.this_device,"Unknown IOCTL 0x%x!\n" , cmd); ret = -ENOTTY ;
drivers/virt/acrn/irqfd.c new file +235 -0
235 lines not on this page · raw message
drivers/virt/acrn/vm.c +3 -0
@@ -51,6 +51,7 @@ struct acrn_vm *acrn_vm_create(struct acrn_vm *vm,write_unlock_bh (&acrn_vm_list_lock);acrn_ioeventfd_init (vm); +acrn_irqfd_init (vm);dev_dbg (acrn_dev.this_device,"VM %u created.\n" , vm->vmid);return vm; }
@@ -69,7 +70,9 @@ int acrn_vm_destroy(struct acrn_vm *vm)write_unlock_bh (&acrn_vm_list_lock);acrn_ioeventfd_deinit (vm); +acrn_irqfd_deinit (vm);acrn_ioreq_deinit (vm); +if (vm->monitor_page) {put_page (vm->monitor_page); vm->monitor_page =NULL ;
include/uapi/linux/acrn.h +15 -0
@@ -411,6 +411,19 @@ struct acrn_ioeventfd {__u64 data; }; +#define ACRN_IRQFD_FLAG_DEASSIGN 0x01 +/** +* struct acrn_irqfd - Data to operate a &struct hsm_irqfd +* @fd: The fd of eventfd associated with a hsm_irqfd +* @flags: Logical-OR of ACRN_IRQFD_FLAG_* +* @msi: Info of MSI associated with the irqfd +*/ +struct acrn_irqfd { +__s32 fd; +__u32 flags; +struct acrn_msi_entry msi; +}; +/* The ioctl type, documented in ioctl-number.rst */ #define ACRN_IOCTL_TYPE 0xA2
@@ -467,5 +480,7 @@ struct acrn_ioeventfd {#define ACRN_IOCTL_IOEVENTFD \ _IOW(ACRN_IOCTL_TYPE, 0x70, struct acrn_ioeventfd) +#define ACRN_IOCTL_IRQFD \ + _IOW(ACRN_IOCTL_TYPE, 0x71, struct acrn_irqfd)#endif /* _UAPI_ACRN_H */
Signature · 1 line
2.28.0
PATCH v4 17/17 virt: acrn: Introduce an interface for Service VM to control vCPU
In reply to [email protected]From: Shuo Liu <[email protected]>
ACRN supports partition mode to achieve real-time requirements. In partition mode, a CPU core can be dedicated to a vCPU of User VM. The local APIC of the dedicated CPU core can be passthrough to the User VM. The Service VM controls the assignment of the CPU cores.
Introduce an interface for the Service VM to remove the control of CPU core from hypervisor perspective so that the CPU core can be a dedicated CPU core of User VM.
--- drivers/virt/acrn/hsm.c | 50 +++++++++++++++++++++++++++++++++++ drivers/virt/acrn/hypercall.h | 14 ++++++++++ 2 files changed, 64 insertions(+)
drivers/virt/acrn/hsm.c +50 -0
@@ -9,6 +9,7 @@* Yakui Zhao <[email protected]> */ +#include <linux/cpu.h> #include <linux/io.h> #include <linux/mm.h> #include <linux/module.h>
@@ -354,6 +355,47 @@ struct miscdevice acrn_dev = { .fops = &acrn_fops, }; +static ssize_t remove_cpu_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t count) +{ +u64 cpu, lapicid; +int ret; + +if (kstrtoull (buf,0 , &cpu) <0 ) +return -EINVAL ; + +if (cpu >=num_possible_cpus () || cpu ==0 || !cpu_is_hotpluggable (cpu)) +return -EINVAL ; + +if (cpu_online (cpu)) +remove_cpu (cpu); + + lapicid =cpu_data (cpu).apicid; +dev_dbg (dev,"Try to remove cpu %lld with lapicid %lld\n" , cpu, lapicid); + ret =hcall_sos_remove_cpu (lapicid); +if (ret <0 ) { +dev_err (dev,"Failed to remove cpu %lld!\n" , cpu); + goto fail_remove; + } + +return count; + +fail_remove: +add_cpu (cpu); +return ret; +} +static DEVICE_ATTR_WO (remove_cpu); + +static struct attribute *acrn_attrs[] = { + &dev_attr_remove_cpu.attr, +NULL +}; + +static struct attribute_group acrn_attr_group = { + .attrs = acrn_attrs, +}; +static int __init hsm_init (void ) {int ret;
@@ -370,13 +412,21 @@ static int __init hsm_init(void)return ret; } + ret =sysfs_create_group (&acrn_dev.this_device->kobj, &acrn_attr_group); +if (ret) { +dev_warn (acrn_dev.this_device,"sysfs create failed\n" ); +misc_deregister (&acrn_dev); +return ret; + }acrn_ioreq_intr_setup (); +return 0 ; } static void __exit hsm_exit(void) {acrn_ioreq_intr_remove (); +sysfs_remove_group (&acrn_dev.this_device->kobj, &acrn_attr_group);misc_deregister (&acrn_dev); }module_init (hsm_init);
drivers/virt/acrn/hypercall.h +14 -0
@@ -13,6 +13,9 @@#define HC_ID 0x80UL +#define HC_ID_GEN_BASE 0x0UL +#define HC_SOS_REMOVE_CPU _HC_ID(HC_ID, HC_ID_GEN_BASE + 0x01) +#define HC_ID_VM_BASE 0x10UL#define HC_CREATE_VM _HC_ID(HC_ID, HC_ID_VM_BASE + 0x00)#define HC_DESTROY_VM _HC_ID(HC_ID, HC_ID_VM_BASE + 0x01)
@@ -42,6 +45,17 @@#define HC_ID_PM_BASE 0x80UL#define HC_PM_GET_CPU_STATE _HC_ID(HC_ID, HC_ID_PM_BASE + 0x00) +/** +* hcall_sos_remove_cpu() - Remove a vCPU of Service VM +* @cpu: The vCPU to be removed +* +* Return: 0 on success, <0 on failure +*/ +static inline long hcall_sos_remove_cpu (u64 cpu) +{ +return acrn_hypercall1 (HC_SOS_REMOVE_CPU , cpu); +} +/** * hcall_create_vm() - Create a User VM * @vminfo: Service VM GPA of info of User VM creation
Signature · 1 line
2.28.0
Re: [PATCH v4 17/17] virt: acrn: Introduce an interface for Service VM to control vCPU
Quoted 70 lines · On Tue, Sep 22, 2020 at 07:43:11PM +0800, [email protected] wrote:
From: Shuo Liu <[email protected]>
ACRN supports partition mode to achieve real-time requirements. In partition mode, a CPU core can be dedicated to a vCPU of User VM. The local APIC of the dedicated CPU core can be passthrough to the User VM. The Service VM controls the assignment of the CPU cores.
Introduce an interface for the Service VM to remove the control of CPU core from hypervisor perspective so that the CPU core can be a dedicated CPU core of User VM.
Signed-off-by Shuo Liu <[email protected]>Reviewed-by Zhi Wang <[email protected]>Reviewed-by Reinette Chatre <[email protected]>Cc Zhi Wang <[email protected]>Cc Zhenyu Wang <[email protected]>Cc Yu Wang <[email protected]>Cc Reinette Chatre <[email protected]>Cc Greg Kroah-Hartman <[email protected]>--- drivers/virt/acrn/hsm.c | 50 +++++++++++++++++++++++++++++++++++ drivers/virt/acrn/hypercall.h | 14 ++++++++++ 2 files changed, 64 insertions(+)drivers/virt/acrn/hsm.c +31
-0@@ -9,6 +9,7 @@* Yakui Zhao <[email protected]> */ +#include <linux/cpu.h> #include <linux/io.h> #include <linux/mm.h> #include <linux/module.h> @@ -354,6 +355,47 @@ struct miscdevice acrn_dev = { .fops = &acrn_fops, }; +static ssize_t remove_cpu_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t count) +{ +u64 cpu, lapicid; +int ret; + +if (kstrtoull (buf,0 , &cpu) <0 ) +return -EINVAL ; + +if (cpu >=num_possible_cpus () || cpu ==0 || !cpu_is_hotpluggable (cpu)) +return -EINVAL ; + +if (cpu_online (cpu)) +remove_cpu (cpu); + + lapicid =cpu_data (cpu).apicid; +dev_dbg (dev,"Try to remove cpu %lld with lapicid %lld\n" , cpu, lapicid); + ret =hcall_sos_remove_cpu (lapicid); +if (ret <0 ) { +dev_err (dev,"Failed to remove cpu %lld!\n" , cpu); + goto fail_remove; + } + +return count; + +fail_remove: +add_cpu (cpu); +return ret; +}
drivers/virt/acrn/hsm.c struct miscdevice acrn_dev = { · line 397
Quoted 10 lines
drivers/virt/acrn/hsm.c +10
-0+static DEVICE_ATTR_WO (remove_cpu); + +static struct attribute *acrn_attrs[] = { + &dev_attr_remove_cpu.attr, +NULL +}; + +static struct attribute_group acrn_attr_group = { + .attrs = acrn_attrs, +};
You create a sysfs attribute without any Documentation/ABI/ update as well? That's not good.
And why are you trying to emulate CPU hotplug here and not using the existing CPU hotplug mechanism?
Quoted 5 lines
drivers/virt/acrn/hsm.c +1
-0+static int __init hsm_init (void ) {int ret;
drivers/virt/acrn/hsm.c static int __init hsm_init(void) · line 420
Quoted 9 lines
drivers/virt/acrn/hsm.c +6
-0@@ -370,13 +412,21 @@ static int __init hsm_init(void)return ret; } + ret =sysfs_create_group (&acrn_dev.this_device->kobj, &acrn_attr_group); +if (ret) { +dev_warn (acrn_dev.this_device,"sysfs create failed\n" ); +misc_deregister (&acrn_dev); +return ret; + }
You just raced with userspace and lost. If you want to add attribute files to a device, use the default attribute group list, and it will be managed properly for you by the driver core.
Huge hint, if a driver every has to touch a kobject, or call sysfs_*, then it is probably doing something wrong.
greg k-h
Re: [PATCH v4 17/17] virt: acrn: Introduce an interface for Service VM to control vCPU
Hi Greg,
Quoted 76 lines · Greg Kroah-Hartman
Quoted 75 lines · On Tue, Sep 22, 2020 at 07:43:11PM +0800, [email protected] wrote:
From: Shuo Liu <[email protected]>
ACRN supports partition mode to achieve real-time requirements. In partition mode, a CPU core can be dedicated to a vCPU of User VM. The local APIC of the dedicated CPU core can be passthrough to the User VM. The Service VM controls the assignment of the CPU cores.
Introduce an interface for the Service VM to remove the control of CPU core from hypervisor perspective so that the CPU core can be a dedicated CPU core of User VM.
Signed-off-by Shuo Liu <[email protected]>Reviewed-by Zhi Wang <[email protected]>Reviewed-by Reinette Chatre <[email protected]>Cc Zhi Wang <[email protected]>Cc Zhenyu Wang <[email protected]>Cc Yu Wang <[email protected]>Cc Reinette Chatre <[email protected]>Cc Greg Kroah-Hartman <[email protected]>--- drivers/virt/acrn/hsm.c | 50 +++++++++++++++++++++++++++++++++++ drivers/virt/acrn/hypercall.h | 14 ++++++++++ 2 files changed, 64 insertions(+)drivers/virt/acrn/hsm.c +36
-0@@ -9,6 +9,7 @@* Yakui Zhao <[email protected]> */ +#include <linux/cpu.h> #include <linux/io.h> #include <linux/mm.h> #include <linux/module.h> @@ -354,6 +355,47 @@ struct miscdevice acrn_dev = { .fops = &acrn_fops, }; +static ssize_t remove_cpu_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t count) +{ +u64 cpu, lapicid; +int ret; + +if (kstrtoull (buf,0 , &cpu) <0 ) +return -EINVAL ; + +if (cpu >=num_possible_cpus () || cpu ==0 || !cpu_is_hotpluggable (cpu)) +return -EINVAL ; + +if (cpu_online (cpu)) +remove_cpu (cpu); + + lapicid =cpu_data (cpu).apicid; +dev_dbg (dev,"Try to remove cpu %lld with lapicid %lld\n" , cpu, lapicid); + ret =hcall_sos_remove_cpu (lapicid); +if (ret <0 ) { +dev_err (dev,"Failed to remove cpu %lld!\n" , cpu); + goto fail_remove; + } + +return count; + +fail_remove: +add_cpu (cpu); +return ret; +} +static DEVICE_ATTR_WO (remove_cpu); + +static struct attribute *acrn_attrs[] = { + &dev_attr_remove_cpu.attr, +NULL
Quoted 8 lines
drivers/virt/acrn/hsm.c struct miscdevice acrn_dev = { · line 397
Quoted 5 lines
drivers/virt/acrn/hsm.c +5
-0+}; + +static struct attribute_group acrn_attr_group = { + .attrs = acrn_attrs, +};You create a sysfs attribute without any
Documentation/ABI/update as well? That's not good.
Sorry, i will add it in the ABI/testing.
Quoted 3 lines
And why are you trying to emulate CPU hotplug here and not using the existing CPU hotplug mechanism?
The interface introduced here includes:
1) The Service VM virtual CPU hotplug
2) hypercall to the hypervisor to remove one virtual CPU from the Service VM
The 1) just do the CPU hotplug with kernel API remove_cpu(), and can be resume back (by CPU online interface) if only 1) is done.
If 2) is done, then the physical CPU will be removed from the Service VM's CPU pool. The ACRN hypervisor supports passthrough a physical CPU to a VM. The precondition is that the physical CPU cannot be occupied by any other VM. This interface intends to do that.
Quoted 20 lines
Quoted 6 lines
drivers/virt/acrn/hsm.c +1
-0+static int __init hsm_init (void ) {int ret;drivers/virt/acrn/hsm.c static int __init hsm_init(void) · line 420
Quoted 9 lines
drivers/virt/acrn/hsm.c +6
-0@@ -370,13 +412,21 @@ static int __init hsm_init(void)return ret; } + ret =sysfs_create_group (&acrn_dev.this_device->kobj, &acrn_attr_group); +if (ret) { +dev_warn (acrn_dev.this_device,"sysfs create failed\n" ); +misc_deregister (&acrn_dev); +return ret; + }You just raced with userspace and lost. If you want to add attribute files to a device, use the default attribute group list, and it will be managed properly for you by the driver core.
Quoted 2 lines
Huge hint, if a driver every has to touch a kobject, or call sysfs_*, then it is probably doing something wrong.
Do you mean use .groups of struct miscdevice directly ?
If yes, let me follow drivers/char/hw_random/s390-trng.c to do this. BTW, few driver use the .groups directly. :)
Thanks
shuo
Re: [PATCH v4 17/17] virt: acrn: Introduce an interface for Service VM to control vCPU
Quoted 11 lines · Shuo A Liu
Quoted 6 lines
You just raced with userspace and lost. If you want to add attribute files to a device, use the default attribute group list, and it will be managed properly for you by the driver core.
Huge hint, if a driver every has to touch a kobject, or call sysfs_*, then it is probably doing something wrong.
Do you mean use .groups of struct miscdevice directly ?
If yes, let me follow
drivers/char/hw_random/s390-trng.cto do this. BTW, few driver use the .groups directly. :)
Drivers should almost never be messing with individual sysfs files. And this ability to use .groups is a "new" one, conversions of existing code that do not use them is always welcome.
thanks,
greg k-h
Re: [PATCH v4 17/17] virt: acrn: Introduce an interface for Service VM to control vCPU
Quoted 8 lines · Greg Kroah-Hartman
Quoted 7 lines · Shuo A Liu
Quoted 6 lines
You just raced with userspace and lost. If you want to add attribute files to a device, use the default attribute group list, and it will be managed properly for you by the driver core.
Huge hint, if a driver every has to touch a kobject, or call sysfs_*, then it is probably doing something wrong.
Quoted 8 lines
Quoted 4 lines
Do you mean use .groups of struct miscdevice directly ?
If yes, let me follow
drivers/char/hw_random/s390-trng.cto do this. BTW, few driver use the .groups directly. :)Drivers should almost never be messing with individual sysfs files. And this ability to use .groups is a "new" one, conversions of existing code that do not use them is always welcome.
OK. Thanks for the explanation. I will follow the 'new' way :)
Thanks
shuo
Ping...
Quoted 140 lines · On 9/22/2020 19:42, [email protected] wrote:
From: Shuo Liu <[email protected]>
ACRN is a Type 1 reference hypervisor stack, running directly on the bare-metal hardware, and is suitable for a variety of IoT and embedded device solutions.
ACRN implements a hybrid VMM architecture, using a privileged Service VM. The Service VM manages the system resources (CPU, memory, etc.) and I/O devices of User VMs. Multiple User VMs are supported, with each of them running Linux, Android OS or Windows. Both Service VM and User VMs are guest VM.
Below figure shows the architecture.
Service VM User VM +----------------------------+ | +------------------+ | +--------------+ | | | | | |ACRN userspace| | | | | | +--------------+ | | | | |-----------------ioctl------| | | | ... |kernel space +----------+ | | | | | | HSM | | | | Drivers | | +----------+ | | | | +--------------------|-------+ | +------------------+ +---------------------hypercall----------------------------------------+ | ACRN Hypervisor | +----------------------------------------------------------------------+ | Hardware | +----------------------------------------------------------------------+There is only one Service VM which could run Linux as OS.
In a typical case, the Service VM will be auto started when ACRN Hypervisor is booted. Then the ACRN userspace (an application running in Service VM) could be used to start/stop User VMs by communicating with ACRN Hypervisor Service Module (HSM).
ACRN Hypervisor Service Module (HSM) is a middle layer that allows the ACRN userspace and Service VM OS kernel to communicate with ACRN Hypervisor and manage different User VMs. This middle layer provides the following functionalities,
- Issues hypercalls to the hypervisor to manage User VMs:
* VM/vCPU management
* Memory management
* Device passthrough
* Interrupts injection
- I/O requests handling from User VMs.
- Exports ioctl through HSM char device.
- Exports function calls for other kernel modulesACRN is focused on embedded system. So it doesn't support some features. E.g.,
- ACRN doesn't support VM migration.
- ACRN doesn't support vCPU migration.This patch set adds the HSM to the Linux kernel.
The basic ARCN support was merged to upstream already.
https://lore.kernel.org/lkml/[email protected]/ChangeLog:
v4:
- Used acrn_dev.this_device directly for dev_*() (Reinette)
- Removed the odd usage of {get|put}_device() on&acrn_dev->this_device(Greg)
- Removed unused log code. (Greg)
- Corrected the return error values. (Greg)
- Mentioned that HSM relies hypervisor for sanity check inacrn_dev_ioctl()comments (Greg)v3:
- Used {get|put}_device() helpers on&acrn_dev->this_device
- Moved unused code from front patches to later ones.
- Removed self-definedpr_fmt()anddev_fmt()
- Provided comments foracrn_vm_list_lock.v2:
- Removed API version related code. (Dave)
- Replaced pr_*() by dev_*(). (Greg)
- Used-ENOTTYas the error code of unsupported ioctl. (Greg)Shuo Liu (16):
docs: acrn: Introduce ACRN
x86/acrn: Introduce acrn_{setup, remove}_intr_handler() x86/acrn: Introduce hypercall interfaces
virt: acrn: Introduce ACRN HSM basic driver
virt: acrn: Introduce VM management interfaces
virt: acrn: Introduce an ioctl to set vCPU registers state
virt: acrn: Introduce EPT mapping management
virt: acrn: Introduce I/O request management
virt: acrn: Introduce PCI configuration space PIO accesses combiner
virt: acrn: Introduce interfaces for PCI device passthrough
virt: acrn: Introduce interrupt injection interfaces
virt: acrn: Introduce interfaces to query C-states and P-states
allowed by hypervisor
virt: acrn: Introduce I/O ranges operation interfaces
virt: acrn: Introduce ioeventfd
virt: acrn: Introduce irqfd
virt: acrn: Introduce an interface for Service VM to control vCPUYin Fengwei (1):
x86/acrn: Introduce an API to check if a VM is privileged.../userspace-api/ioctl/ioctl-number.rst | 1 + Documentation/virt/acrn/index.rst | 11 + Documentation/virt/acrn/introduction.rst | 40 ++ Documentation/virt/acrn/io-request.rst | 97 +++ Documentation/virt/index.rst | 1 + MAINTAINERS | 9 + arch/x86/include/asm/acrn.h | 74 ++ arch/x86/kernel/cpu/acrn.c | 35 +- drivers/virt/Kconfig | 2 + drivers/virt/Makefile | 1 + drivers/virt/acrn/Kconfig | 15 + drivers/virt/acrn/Makefile | 3 + drivers/virt/acrn/acrn_drv.h | 229 +++++++ drivers/virt/acrn/hsm.c | 437 ++++++++++++ drivers/virt/acrn/hypercall.h | 254 +++++++ drivers/virt/acrn/ioeventfd.c | 273 ++++++++ drivers/virt/acrn/ioreq.c | 645 ++++++++++++++++++ drivers/virt/acrn/irqfd.c | 235 +++++++ drivers/virt/acrn/mm.c | 305 +++++++++ drivers/virt/acrn/vm.c | 126 ++++ include/uapi/linux/acrn.h | 486 +++++++++++++ 21 files changed, 3278 insertions(+), 1 deletion(-) create mode 100644 Documentation/virt/acrn/index.rst create mode 100644 Documentation/virt/acrn/introduction.rst create mode 100644 Documentation/virt/acrn/io-request.rst create mode 100644 arch/x86/include/asm/acrn.h create mode 100644 drivers/virt/acrn/Kconfig create mode 100644 drivers/virt/acrn/Makefile create mode 100644 drivers/virt/acrn/acrn_drv.h create mode 100644 drivers/virt/acrn/hsm.c create mode 100644 drivers/virt/acrn/hypercall.h create mode 100644 drivers/virt/acrn/ioeventfd.c create mode 100644 drivers/virt/acrn/ioreq.c create mode 100644 drivers/virt/acrn/irqfd.c create mode 100644 drivers/virt/acrn/mm.c create mode 100644 drivers/virt/acrn/vm.c create mode 100644 include/uapi/linux/acrn.hbase-commit:
18445bf405cb331117bc98427b1ba6f12418ad17
Quoted 1 line · Shuo A
Ping...
It's been less than a week since you sent this. Please relax and if you really need reviews, get them from within Intel, where you can impose a deadline on those developers. Otherwise, your patch is in good company:
$ mdfrm -c ~/mail/todo/ 993 messages in /home/gregkh/mail/todo/
And will be handled when I get to it.
thanks,
greg "Intel still owes me lots of liquor" k-h
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH v4 06/17] virt: acrn: Introduce VM management interfaces
Quoted 76 lines · On Tue, Sep 22, 2020 at 07:43:00PM +0800, [email protected] wrote:
From: Shuo Liu <[email protected]>
The VM management interfaces expose several VM operations to ACRN userspace via ioctls. For example, creating VM, starting VM, destroying VM and so on.
The ACRN Hypervisor needs to exchange data with the ACRN userspace during the VM operations. HSM provides VM operation ioctls to the ACRN userspace and communicates with the ACRN Hypervisor for VM operations via hypercalls.
HSM maintains a list of User VM. Each User VM will be bound to an existing file descriptor of
/dev/acrn_hsm. The User VM will be destroyed when the file descriptor is closed.Signed-off-by Shuo Liu <[email protected]>Reviewed-by Zhi Wang <[email protected]>Reviewed-by Reinette Chatre <[email protected]>Cc Zhi Wang <[email protected]>Cc Zhenyu Wang <[email protected]>Cc Yu Wang <[email protected]>Cc Reinette Chatre <[email protected]>Cc Greg Kroah-Hartman <[email protected]>--- .../userspace-api/ioctl/ioctl-number.rst | 1 + MAINTAINERS | 1 + drivers/virt/acrn/Makefile | 2 +- drivers/virt/acrn/acrn_drv.h | 23 +++++- drivers/virt/acrn/hsm.c | 73 ++++++++++++++++- drivers/virt/acrn/hypercall.h | 78 +++++++++++++++++++ drivers/virt/acrn/vm.c | 71 +++++++++++++++++ include/uapi/linux/acrn.h | 56 +++++++++++++ 8 files changed, 301 insertions(+), 4 deletions(-) create mode 100644 drivers/virt/acrn/hypercall.h create mode 100644 drivers/virt/acrn/vm.c create mode 100644 include/uapi/linux/acrn.hDocumentation/userspace-api/ioctl/ioctl-number.rst +1
-0@@ -319,6 +319,7 @@ Code Seq# Include File Comments 0xA0 all linux/sdp/sdp.h Industrial Device Project <mailto:[email protected]> 0xA1 0 linux/vtpm_proxy.h TPM Emulator Proxy Driver +0xA2 all uapi/linux/acrn.h ACRN hypervisor 0xA3 80-8F Port ACL in development: <mailto:[email protected]> 0xA3 90-9F linux/dtlk.hMAINTAINERS +1
-0@@ -443,6 +443,7 @@ S: Supported W: https://projectacrn.org F: Documentation/virt/acrn/ F: drivers/virt/acrn/ +F: include/uapi/linux/acrn.h AD1889 ALSA SOUND DRIVER L: [email protected]drivers/virt/acrn/Makefile +1
-1@@ -1,3 +1,3 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_ACRN_HSM) := acrn.o -acrn-y := hsm.o +acrn-y := hsm.o vm.odrivers/virt/acrn/acrn_drv.h +0
-0@@ -3,16 +3,37 @@#ifndef __ACRN_HSM_DRV_H
drivers/virt/acrn/acrn_drv.h line 13
Quoted 10 lines
drivers/virt/acrn/acrn_drv.h +6
-0#define __ACRN_HSM_DRV_H +#include <linux/acrn.h> +#include <linux/dev_printk.h> +#include <linux/miscdevice.h> #include <linux/types.h> +#include "hypercall.h" + +extern struct miscdevice acrn_dev;
Who else needs to get to this structure in your driver?
Quoted 95 lines
drivers/virt/acrn/acrn_drv.h +16
-1+#define ACRN_INVALID_VMID (0xffffU) +#define ACRN_VM_FLAG_DESTROYED 0U +extern struct list_head acrn_vm_list; +extern rwlock_t acrn_vm_list_lock;/** * struct acrn_vm - Properties of ACRN User VM. +* @list: Entry within global list of all VMs * @vmid: User VM ID +* @vcpu_num: Number of virtual CPUs in the VM +* @flags: Flags (ACRN_VM_FLAG_*) of the VM. This is VM flag management +* in HSM which is different from the &acrn_vm_creation.vm_flag. */ struct acrn_vm { -u16 vmid; +struct list_head list; +u16 vmid; +int vcpu_num; +unsigned long flags; }; +struct acrn_vm *acrn_vm_create (struct acrn_vm *vm, +struct acrn_vm_creation *vm_param); +int acrn_vm_destroy (struct acrn_vm *vm); +#endif /* __ACRN_HSM_DRV_H */ drivers/virt/acrn/hsm.c +52
-1@@ -9,7 +9,6 @@* Yakui Zhao <[email protected]> */ -#include <linux/miscdevice.h> #include <linux/mm.h> #include <linux/module.h> #include <linux/slab.h> @@ -38,10 +37,79 @@ static int acrn_dev_open(struct inode *inode, struct file *filp)return 0 ; } +/* +* HSM relies on hypercall layer of the ACRN hypervisor to do the +* sanity check against the input parameters. +*/ +static long acrn_dev_ioctl (struct file *filp,unsigned int cmd, +unsigned long ioctl_param) +{ +struct acrn_vm *vm = filp->private_data; +struct acrn_vm_creation *vm_param; +int ret =0 ; + +if (vm->vmid ==ACRN_INVALID_VMID && cmd !=ACRN_IOCTL_CREATE_VM ) { +dev_dbg (acrn_dev.this_device, +"ioctl 0x%x: Invalid VM state!\n" , cmd); +return -EINVAL ; + } + +switch (cmd) { +case ACRN_IOCTL_CREATE_VM : + vm_param =memdup_user ((void __user *)ioctl_param, +sizeof (struct acrn_vm_creation )); +if (IS_ERR (vm_param)) +return PTR_ERR (vm_param); + + vm =acrn_vm_create (vm, vm_param); +if (!vm) { + ret = -EINVAL ; +kfree (vm_param); +break ; + } + +if (copy_to_user ((void __user *)ioctl_param, vm_param, +sizeof (struct acrn_vm_creation ))) { +acrn_vm_destroy (vm); + ret = -EFAULT ; + } + +kfree (vm_param); +break ; +case ACRN_IOCTL_START_VM : + ret =hcall_start_vm (vm->vmid); +if (ret <0 ) +dev_err (acrn_dev.this_device, +"Failed to start VM %u!\n" , vm->vmid); +break ; +case ACRN_IOCTL_PAUSE_VM : + ret =hcall_pause_vm (vm->vmid); +if (ret <0 ) +dev_err (acrn_dev.this_device, +"Failed to pause VM %u!\n" , vm->vmid); +break ; +case ACRN_IOCTL_RESET_VM :
drivers/virt/acrn/hsm.c static int acrn_dev_open(struct inode *inode, struct file *filp) · line 101
Quoted 10 lines
drivers/virt/acrn/hsm.c +10
-0+ ret =hcall_reset_vm (vm->vmid); +if (ret <0 ) +dev_err (acrn_dev.this_device, +"Failed to restart VM %u!\n" , vm->vmid); +break ; +case ACRN_IOCTL_DESTROY_VM : + ret =acrn_vm_destroy (vm); +break ; +default : +dev_warn (acrn_dev.this_device,"Unknown IOCTL 0x%x!\n" , cmd);
Do not let userspace spam kernel logs with invalid stuff, that's a sure way to cause a DoS.
thanks,
greg k-h
Re: [PATCH v4 06/17] virt: acrn: Introduce VM management interfaces
Hi Greg,
Quoted 84 lines · Greg Kroah-Hartman
Quoted 83 lines · On Tue, Sep 22, 2020 at 07:43:00PM +0800, [email protected] wrote:
From: Shuo Liu <[email protected]>
The VM management interfaces expose several VM operations to ACRN userspace via ioctls. For example, creating VM, starting VM, destroying VM and so on.
The ACRN Hypervisor needs to exchange data with the ACRN userspace during the VM operations. HSM provides VM operation ioctls to the ACRN userspace and communicates with the ACRN Hypervisor for VM operations via hypercalls.
HSM maintains a list of User VM. Each User VM will be bound to an existing file descriptor of
/dev/acrn_hsm. The User VM will be destroyed when the file descriptor is closed.Signed-off-by Shuo Liu <[email protected]>Reviewed-by Zhi Wang <[email protected]>Reviewed-by Reinette Chatre <[email protected]>Cc Zhi Wang <[email protected]>Cc Zhenyu Wang <[email protected]>Cc Yu Wang <[email protected]>Cc Reinette Chatre <[email protected]>Cc Greg Kroah-Hartman <[email protected]>--- .../userspace-api/ioctl/ioctl-number.rst | 1 + MAINTAINERS | 1 + drivers/virt/acrn/Makefile | 2 +- drivers/virt/acrn/acrn_drv.h | 23 +++++- drivers/virt/acrn/hsm.c | 73 ++++++++++++++++- drivers/virt/acrn/hypercall.h | 78 +++++++++++++++++++ drivers/virt/acrn/vm.c | 71 +++++++++++++++++ include/uapi/linux/acrn.h | 56 +++++++++++++ 8 files changed, 301 insertions(+), 4 deletions(-) create mode 100644 drivers/virt/acrn/hypercall.h create mode 100644 drivers/virt/acrn/vm.c create mode 100644 include/uapi/linux/acrn.hDocumentation/userspace-api/ioctl/ioctl-number.rst +1
-0@@ -319,6 +319,7 @@ Code Seq# Include File Comments 0xA0 all linux/sdp/sdp.h Industrial Device Project <mailto:[email protected]> 0xA1 0 linux/vtpm_proxy.h TPM Emulator Proxy Driver +0xA2 all uapi/linux/acrn.h ACRN hypervisor 0xA3 80-8F Port ACL in development: <mailto:[email protected]> 0xA3 90-9F linux/dtlk.hMAINTAINERS +1
-0@@ -443,6 +443,7 @@ S: Supported W: https://projectacrn.org F: Documentation/virt/acrn/ F: drivers/virt/acrn/ +F: include/uapi/linux/acrn.h AD1889 ALSA SOUND DRIVER L: [email protected]drivers/virt/acrn/Makefile +1
-1@@ -1,3 +1,3 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_ACRN_HSM) := acrn.o -acrn-y := hsm.o +acrn-y := hsm.o vm.odrivers/virt/acrn/acrn_drv.h +3
-0@@ -3,16 +3,37 @@#ifndef __ACRN_HSM_DRV_H#define __ACRN_HSM_DRV_H +#include <linux/acrn.h> +#include <linux/dev_printk.h> +#include <linux/miscdevice.h> #include <linux/types.h>
Quoted 5 lines
drivers/virt/acrn/acrn_drv.h line 13
Quoted 3 lines
drivers/virt/acrn/acrn_drv.h +3
-0+#include "hypercall.h" + +extern struct miscdevice acrn_dev;Who else needs to get to this structure in your driver?
Other files of the driver need to use it for dev_*() log APIs.
Quoted 101 lines
Quoted 101 lines
drivers/virt/acrn/acrn_drv.h +16
-1+#define ACRN_INVALID_VMID (0xffffU) +#define ACRN_VM_FLAG_DESTROYED 0U +extern struct list_head acrn_vm_list; +extern rwlock_t acrn_vm_list_lock;/** * struct acrn_vm - Properties of ACRN User VM. +* @list: Entry within global list of all VMs * @vmid: User VM ID +* @vcpu_num: Number of virtual CPUs in the VM +* @flags: Flags (ACRN_VM_FLAG_*) of the VM. This is VM flag management +* in HSM which is different from the &acrn_vm_creation.vm_flag. */ struct acrn_vm { -u16 vmid; +struct list_head list; +u16 vmid; +int vcpu_num; +unsigned long flags; }; +struct acrn_vm *acrn_vm_create (struct acrn_vm *vm, +struct acrn_vm_creation *vm_param); +int acrn_vm_destroy (struct acrn_vm *vm); +#endif /* __ACRN_HSM_DRV_H */ drivers/virt/acrn/hsm.c +57
-1@@ -9,7 +9,6 @@* Yakui Zhao <[email protected]> */ -#include <linux/miscdevice.h> #include <linux/mm.h> #include <linux/module.h> #include <linux/slab.h> @@ -38,10 +37,79 @@ static int acrn_dev_open(struct inode *inode, struct file *filp)return 0 ; } +/* +* HSM relies on hypercall layer of the ACRN hypervisor to do the +* sanity check against the input parameters. +*/ +static long acrn_dev_ioctl (struct file *filp,unsigned int cmd, +unsigned long ioctl_param) +{ +struct acrn_vm *vm = filp->private_data; +struct acrn_vm_creation *vm_param; +int ret =0 ; + +if (vm->vmid ==ACRN_INVALID_VMID && cmd !=ACRN_IOCTL_CREATE_VM ) { +dev_dbg (acrn_dev.this_device, +"ioctl 0x%x: Invalid VM state!\n" , cmd); +return -EINVAL ; + } + +switch (cmd) { +case ACRN_IOCTL_CREATE_VM : + vm_param =memdup_user ((void __user *)ioctl_param, +sizeof (struct acrn_vm_creation )); +if (IS_ERR (vm_param)) +return PTR_ERR (vm_param); + + vm =acrn_vm_create (vm, vm_param); +if (!vm) { + ret = -EINVAL ; +kfree (vm_param); +break ; + } + +if (copy_to_user ((void __user *)ioctl_param, vm_param, +sizeof (struct acrn_vm_creation ))) { +acrn_vm_destroy (vm); + ret = -EFAULT ; + } + +kfree (vm_param); +break ; +case ACRN_IOCTL_START_VM : + ret =hcall_start_vm (vm->vmid); +if (ret <0 ) +dev_err (acrn_dev.this_device, +"Failed to start VM %u!\n" , vm->vmid); +break ; +case ACRN_IOCTL_PAUSE_VM : + ret =hcall_pause_vm (vm->vmid); +if (ret <0 ) +dev_err (acrn_dev.this_device, +"Failed to pause VM %u!\n" , vm->vmid); +break ; +case ACRN_IOCTL_RESET_VM : + ret =hcall_reset_vm (vm->vmid); +if (ret <0 ) +dev_err (acrn_dev.this_device, +"Failed to restart VM %u!\n" , vm->vmid); +break ;
Quoted 8 lines
drivers/virt/acrn/hsm.c static int acrn_dev_open(struct inode *inode, struct file *filp) · line 101
Quoted 5 lines
drivers/virt/acrn/hsm.c +5
-0+case ACRN_IOCTL_DESTROY_VM : + ret =acrn_vm_destroy (vm); +break ; +default : +dev_warn (acrn_dev.this_device,"Unknown IOCTL 0x%x!\n" , cmd);Do not let userspace spam kernel logs with invalid stuff, that's a sure way to cause a DoS.
OK. Got it. Will be dev_dbg().
Thanks
shuo
Re: [PATCH v4 06/17] virt: acrn: Introduce VM management interfaces
In reply to a message not in the archiveQuoted 339 lines · On Tue, Sep 22, 2020 at 07:43:00PM +0800, [email protected] wrote:
From: Shuo Liu <[email protected]>
The VM management interfaces expose several VM operations to ACRN userspace via ioctls. For example, creating VM, starting VM, destroying VM and so on.
The ACRN Hypervisor needs to exchange data with the ACRN userspace during the VM operations. HSM provides VM operation ioctls to the ACRN userspace and communicates with the ACRN Hypervisor for VM operations via hypercalls.
HSM maintains a list of User VM. Each User VM will be bound to an existing file descriptor of
/dev/acrn_hsm. The User VM will be destroyed when the file descriptor is closed.Signed-off-by Shuo Liu <[email protected]>Reviewed-by Zhi Wang <[email protected]>Reviewed-by Reinette Chatre <[email protected]>Cc Zhi Wang <[email protected]>Cc Zhenyu Wang <[email protected]>Cc Yu Wang <[email protected]>Cc Reinette Chatre <[email protected]>Cc Greg Kroah-Hartman <[email protected]>--- .../userspace-api/ioctl/ioctl-number.rst | 1 + MAINTAINERS | 1 + drivers/virt/acrn/Makefile | 2 +- drivers/virt/acrn/acrn_drv.h | 23 +++++- drivers/virt/acrn/hsm.c | 73 ++++++++++++++++- drivers/virt/acrn/hypercall.h | 78 +++++++++++++++++++ drivers/virt/acrn/vm.c | 71 +++++++++++++++++ include/uapi/linux/acrn.h | 56 +++++++++++++ 8 files changed, 301 insertions(+), 4 deletions(-) create mode 100644 drivers/virt/acrn/hypercall.h create mode 100644 drivers/virt/acrn/vm.c create mode 100644 include/uapi/linux/acrn.hDocumentation/userspace-api/ioctl/ioctl-number.rst +1
-0@@ -319,6 +319,7 @@ Code Seq# Include File Comments 0xA0 all linux/sdp/sdp.h Industrial Device Project <mailto:[email protected]> 0xA1 0 linux/vtpm_proxy.h TPM Emulator Proxy Driver +0xA2 all uapi/linux/acrn.h ACRN hypervisor 0xA3 80-8F Port ACL in development: <mailto:[email protected]> 0xA3 90-9F linux/dtlk.hMAINTAINERS +1
-0@@ -443,6 +443,7 @@ S: Supported W: https://projectacrn.org F: Documentation/virt/acrn/ F: drivers/virt/acrn/ +F: include/uapi/linux/acrn.h AD1889 ALSA SOUND DRIVER L: [email protected]drivers/virt/acrn/Makefile +1
-1@@ -1,3 +1,3 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_ACRN_HSM) := acrn.o -acrn-y := hsm.o +acrn-y := hsm.o vm.odrivers/virt/acrn/acrn_drv.h +22
-1@@ -3,16 +3,37 @@#ifndef __ACRN_HSM_DRV_H#define __ACRN_HSM_DRV_H +#include <linux/acrn.h> +#include <linux/dev_printk.h> +#include <linux/miscdevice.h> #include <linux/types.h> +#include "hypercall.h" + +extern struct miscdevice acrn_dev; +#define ACRN_INVALID_VMID (0xffffU) +#define ACRN_VM_FLAG_DESTROYED 0U +extern struct list_head acrn_vm_list; +extern rwlock_t acrn_vm_list_lock;/** * struct acrn_vm - Properties of ACRN User VM. +* @list: Entry within global list of all VMs * @vmid: User VM ID +* @vcpu_num: Number of virtual CPUs in the VM +* @flags: Flags (ACRN_VM_FLAG_*) of the VM. This is VM flag management +* in HSM which is different from the &acrn_vm_creation.vm_flag. */ struct acrn_vm { -u16 vmid; +struct list_head list; +u16 vmid; +int vcpu_num; +unsigned long flags; }; +struct acrn_vm *acrn_vm_create (struct acrn_vm *vm, +struct acrn_vm_creation *vm_param); +int acrn_vm_destroy (struct acrn_vm *vm); +#endif /* __ACRN_HSM_DRV_H */ drivers/virt/acrn/hsm.c +71
-2@@ -9,7 +9,6 @@* Yakui Zhao <[email protected]> */ -#include <linux/miscdevice.h> #include <linux/mm.h> #include <linux/module.h> #include <linux/slab.h> @@ -38,10 +37,79 @@ static int acrn_dev_open(struct inode *inode, struct file *filp)return 0 ; } +/* +* HSM relies on hypercall layer of the ACRN hypervisor to do the +* sanity check against the input parameters. +*/ +static long acrn_dev_ioctl (struct file *filp,unsigned int cmd, +unsigned long ioctl_param) +{ +struct acrn_vm *vm = filp->private_data; +struct acrn_vm_creation *vm_param; +int ret =0 ; + +if (vm->vmid ==ACRN_INVALID_VMID && cmd !=ACRN_IOCTL_CREATE_VM ) { +dev_dbg (acrn_dev.this_device, +"ioctl 0x%x: Invalid VM state!\n" , cmd); +return -EINVAL ; + } + +switch (cmd) { +case ACRN_IOCTL_CREATE_VM : + vm_param =memdup_user ((void __user *)ioctl_param, +sizeof (struct acrn_vm_creation )); +if (IS_ERR (vm_param)) +return PTR_ERR (vm_param); + + vm =acrn_vm_create (vm, vm_param); +if (!vm) { + ret = -EINVAL ; +kfree (vm_param); +break ; + } + +if (copy_to_user ((void __user *)ioctl_param, vm_param, +sizeof (struct acrn_vm_creation ))) { +acrn_vm_destroy (vm); + ret = -EFAULT ; + } + +kfree (vm_param); +break ; +case ACRN_IOCTL_START_VM : + ret =hcall_start_vm (vm->vmid); +if (ret <0 ) +dev_err (acrn_dev.this_device, +"Failed to start VM %u!\n" , vm->vmid); +break ; +case ACRN_IOCTL_PAUSE_VM : + ret =hcall_pause_vm (vm->vmid); +if (ret <0 ) +dev_err (acrn_dev.this_device, +"Failed to pause VM %u!\n" , vm->vmid); +break ; +case ACRN_IOCTL_RESET_VM : + ret =hcall_reset_vm (vm->vmid); +if (ret <0 ) +dev_err (acrn_dev.this_device, +"Failed to restart VM %u!\n" , vm->vmid); +break ; +case ACRN_IOCTL_DESTROY_VM : + ret =acrn_vm_destroy (vm); +break ; +default : +dev_warn (acrn_dev.this_device,"Unknown IOCTL 0x%x!\n" , cmd); + ret = -ENOTTY ; + } + +return ret; +} +static int acrn_dev_release (struct inode *inode,struct file *filp) {struct acrn_vm *vm = filp->private_data; +acrn_vm_destroy (vm);kfree (vm);return 0 ; }@@ -50,9 +118,10 @@ static const struct file_operations acrn_fops = { .owner = THIS_MODULE, .open = acrn_dev_open, .release = acrn_dev_release, + .unlocked_ioctl = acrn_dev_ioctl, }; -static struct miscdevice acrn_dev = { +struct miscdevice acrn_dev = { .minor =MISC_DYNAMIC_MINOR , .name ="acrn_hsm" , .fops = &acrn_fops,drivers/virt/acrn/hypercall.h new file +78
-0@@ -0,0 +1,78 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* ACRN HSM: hypercalls of ACRN Hypervisor +*/ +#ifndef __ACRN_HSM_HYPERCALL_H +#define __ACRN_HSM_HYPERCALL_H +#include <asm/acrn.h> + +/* +* Hypercall IDs of the ACRN Hypervisor +*/ +#define _HC_ID (x, y) (((x) << 24) | (y)) + +#define HC_ID 0x80UL + +#define HC_ID_VM_BASE 0x10UL +#define HC_CREATE_VM _HC_ID(HC_ID, HC_ID_VM_BASE + 0x00) +#define HC_DESTROY_VM _HC_ID(HC_ID, HC_ID_VM_BASE + 0x01) +#define HC_START_VM _HC_ID(HC_ID, HC_ID_VM_BASE + 0x02) +#define HC_PAUSE_VM _HC_ID(HC_ID, HC_ID_VM_BASE + 0x03) +#define HC_RESET_VM _HC_ID(HC_ID, HC_ID_VM_BASE + 0x05) + +/** +* hcall_create_vm() - Create a User VM +* @vminfo: Service VM GPA of info of User VM creation +* +* Return: 0 on success, <0 on failure +*/ +static inline long hcall_create_vm (u64 vminfo) +{ +return acrn_hypercall1 (HC_CREATE_VM , vminfo); +} + +/** +* hcall_start_vm() - Start a User VM +* @vmid: User VM ID +* +* Return: 0 on success, <0 on failure +*/ +static inline long hcall_start_vm (u64 vmid) +{ +return acrn_hypercall1 (HC_START_VM , vmid); +} + +/** +* hcall_pause_vm() - Pause a User VM +* @vmid: User VM ID +* +* Return: 0 on success, <0 on failure +*/ +static inline long hcall_pause_vm (u64 vmid) +{ +return acrn_hypercall1 (HC_PAUSE_VM , vmid); +} + +/** +* hcall_destroy_vm() - Destroy a User VM +* @vmid: User VM ID +* +* Return: 0 on success, <0 on failure +*/ +static inline long hcall_destroy_vm (u64 vmid) +{ +return acrn_hypercall1 (HC_DESTROY_VM , vmid); +} + +/** +* hcall_reset_vm() - Reset a User VM +* @vmid: User VM ID +* +* Return: 0 on success, <0 on failure +*/ +static inline long hcall_reset_vm (u64 vmid) +{ +return acrn_hypercall1 (HC_RESET_VM , vmid); +} + +#endif /* __ACRN_HSM_HYPERCALL_H */ drivers/virt/acrn/vm.c new file +32
-0@@ -0,0 +1,71 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* ACRN_HSM: Virtual Machine management +* +* Copyright (C) 2020 Intel Corporation. All rights reserved. +* +* Authors: +* Jason Chen CJ <[email protected]> +* Yakui Zhao <[email protected]> +*/ +#include <linux/io.h> +#include <linux/mm.h> +#include <linux/slab.h> + +#include "acrn_drv.h" + +/* List of VMs */ +LIST_HEAD (acrn_vm_list); +/* +* acrn_vm_list is read in a tasklet which dispatch I/O requests and is wrote +* in VM creation ioctl. Use the rwlock mechanism to protect it. +*/ +DEFINE_RWLOCK (acrn_vm_list_lock); + +struct acrn_vm *acrn_vm_create (struct acrn_vm *vm, +struct acrn_vm_creation *vm_param) +{ +int ret; + + ret =hcall_create_vm (virt_to_phys (vm_param)); +if (ret <0 || vm_param->vmid ==ACRN_INVALID_VMID ) { + dev_err(acrn_dev.this_device,
drivers/virt/acrn/vm.c line 42
Quoted 10 lines
drivers/virt/acrn/vm.c new file +10
-0+ "Failed to create VM! Error: %d\n", ret); +return NULL ; + } + + vm->vmid = vm_param->vmid; + vm->vcpu_num = vm_param->vcpu_num; + +write_lock_bh (&acrn_vm_list_lock); +list_add (&vm->list, &acrn_vm_list); +write_unlock_bh (&acrn_vm_list_lock);
Why are the _bh() variants being used here?
You are only accessing this list from userspace context in this patch.
Heck, you aren't even reading from the list, only writing to it...
greg k-h
Re: [PATCH v4 06/17] virt: acrn: Introduce VM management interfaces
Hi Greg,
Quoted 353 lines · Greg Kroah-Hartman
Quoted 339 lines · On Tue, Sep 22, 2020 at 07:43:00PM +0800, [email protected] wrote:
From: Shuo Liu <[email protected]>
The VM management interfaces expose several VM operations to ACRN userspace via ioctls. For example, creating VM, starting VM, destroying VM and so on.
The ACRN Hypervisor needs to exchange data with the ACRN userspace during the VM operations. HSM provides VM operation ioctls to the ACRN userspace and communicates with the ACRN Hypervisor for VM operations via hypercalls.
HSM maintains a list of User VM. Each User VM will be bound to an existing file descriptor of
/dev/acrn_hsm. The User VM will be destroyed when the file descriptor is closed.Signed-off-by Shuo Liu <[email protected]>Reviewed-by Zhi Wang <[email protected]>Reviewed-by Reinette Chatre <[email protected]>Cc Zhi Wang <[email protected]>Cc Zhenyu Wang <[email protected]>Cc Yu Wang <[email protected]>Cc Reinette Chatre <[email protected]>Cc Greg Kroah-Hartman <[email protected]>--- .../userspace-api/ioctl/ioctl-number.rst | 1 + MAINTAINERS | 1 + drivers/virt/acrn/Makefile | 2 +- drivers/virt/acrn/acrn_drv.h | 23 +++++- drivers/virt/acrn/hsm.c | 73 ++++++++++++++++- drivers/virt/acrn/hypercall.h | 78 +++++++++++++++++++ drivers/virt/acrn/vm.c | 71 +++++++++++++++++ include/uapi/linux/acrn.h | 56 +++++++++++++ 8 files changed, 301 insertions(+), 4 deletions(-) create mode 100644 drivers/virt/acrn/hypercall.h create mode 100644 drivers/virt/acrn/vm.c create mode 100644 include/uapi/linux/acrn.hDocumentation/userspace-api/ioctl/ioctl-number.rst +1
-0@@ -319,6 +319,7 @@ Code Seq# Include File Comments 0xA0 all linux/sdp/sdp.h Industrial Device Project <mailto:[email protected]> 0xA1 0 linux/vtpm_proxy.h TPM Emulator Proxy Driver +0xA2 all uapi/linux/acrn.h ACRN hypervisor 0xA3 80-8F Port ACL in development: <mailto:[email protected]> 0xA3 90-9F linux/dtlk.hMAINTAINERS +1
-0@@ -443,6 +443,7 @@ S: Supported W: https://projectacrn.org F: Documentation/virt/acrn/ F: drivers/virt/acrn/ +F: include/uapi/linux/acrn.h AD1889 ALSA SOUND DRIVER L: [email protected]drivers/virt/acrn/Makefile +1
-1@@ -1,3 +1,3 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_ACRN_HSM) := acrn.o -acrn-y := hsm.o +acrn-y := hsm.o vm.odrivers/virt/acrn/acrn_drv.h +22
-1@@ -3,16 +3,37 @@#ifndef __ACRN_HSM_DRV_H#define __ACRN_HSM_DRV_H +#include <linux/acrn.h> +#include <linux/dev_printk.h> +#include <linux/miscdevice.h> #include <linux/types.h> +#include "hypercall.h" + +extern struct miscdevice acrn_dev; +#define ACRN_INVALID_VMID (0xffffU) +#define ACRN_VM_FLAG_DESTROYED 0U +extern struct list_head acrn_vm_list; +extern rwlock_t acrn_vm_list_lock;/** * struct acrn_vm - Properties of ACRN User VM. +* @list: Entry within global list of all VMs * @vmid: User VM ID +* @vcpu_num: Number of virtual CPUs in the VM +* @flags: Flags (ACRN_VM_FLAG_*) of the VM. This is VM flag management +* in HSM which is different from the &acrn_vm_creation.vm_flag. */ struct acrn_vm { -u16 vmid; +struct list_head list; +u16 vmid; +int vcpu_num; +unsigned long flags; }; +struct acrn_vm *acrn_vm_create (struct acrn_vm *vm, +struct acrn_vm_creation *vm_param); +int acrn_vm_destroy (struct acrn_vm *vm); +#endif /* __ACRN_HSM_DRV_H */ drivers/virt/acrn/hsm.c +71
-2@@ -9,7 +9,6 @@* Yakui Zhao <[email protected]> */ -#include <linux/miscdevice.h> #include <linux/mm.h> #include <linux/module.h> #include <linux/slab.h> @@ -38,10 +37,79 @@ static int acrn_dev_open(struct inode *inode, struct file *filp)return 0 ; } +/* +* HSM relies on hypercall layer of the ACRN hypervisor to do the +* sanity check against the input parameters. +*/ +static long acrn_dev_ioctl (struct file *filp,unsigned int cmd, +unsigned long ioctl_param) +{ +struct acrn_vm *vm = filp->private_data; +struct acrn_vm_creation *vm_param; +int ret =0 ; + +if (vm->vmid ==ACRN_INVALID_VMID && cmd !=ACRN_IOCTL_CREATE_VM ) { +dev_dbg (acrn_dev.this_device, +"ioctl 0x%x: Invalid VM state!\n" , cmd); +return -EINVAL ; + } + +switch (cmd) { +case ACRN_IOCTL_CREATE_VM : + vm_param =memdup_user ((void __user *)ioctl_param, +sizeof (struct acrn_vm_creation )); +if (IS_ERR (vm_param)) +return PTR_ERR (vm_param); + + vm =acrn_vm_create (vm, vm_param); +if (!vm) { + ret = -EINVAL ; +kfree (vm_param); +break ; + } + +if (copy_to_user ((void __user *)ioctl_param, vm_param, +sizeof (struct acrn_vm_creation ))) { +acrn_vm_destroy (vm); + ret = -EFAULT ; + } + +kfree (vm_param); +break ; +case ACRN_IOCTL_START_VM : + ret =hcall_start_vm (vm->vmid); +if (ret <0 ) +dev_err (acrn_dev.this_device, +"Failed to start VM %u!\n" , vm->vmid); +break ; +case ACRN_IOCTL_PAUSE_VM : + ret =hcall_pause_vm (vm->vmid); +if (ret <0 ) +dev_err (acrn_dev.this_device, +"Failed to pause VM %u!\n" , vm->vmid); +break ; +case ACRN_IOCTL_RESET_VM : + ret =hcall_reset_vm (vm->vmid); +if (ret <0 ) +dev_err (acrn_dev.this_device, +"Failed to restart VM %u!\n" , vm->vmid); +break ; +case ACRN_IOCTL_DESTROY_VM : + ret =acrn_vm_destroy (vm); +break ; +default : +dev_warn (acrn_dev.this_device,"Unknown IOCTL 0x%x!\n" , cmd); + ret = -ENOTTY ; + } + +return ret; +} +static int acrn_dev_release (struct inode *inode,struct file *filp) {struct acrn_vm *vm = filp->private_data; +acrn_vm_destroy (vm);kfree (vm);return 0 ; }@@ -50,9 +118,10 @@ static const struct file_operations acrn_fops = { .owner = THIS_MODULE, .open = acrn_dev_open, .release = acrn_dev_release, + .unlocked_ioctl = acrn_dev_ioctl, }; -static struct miscdevice acrn_dev = { +struct miscdevice acrn_dev = { .minor =MISC_DYNAMIC_MINOR , .name ="acrn_hsm" , .fops = &acrn_fops,drivers/virt/acrn/hypercall.h new file +78
-0@@ -0,0 +1,78 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* ACRN HSM: hypercalls of ACRN Hypervisor +*/ +#ifndef __ACRN_HSM_HYPERCALL_H +#define __ACRN_HSM_HYPERCALL_H +#include <asm/acrn.h> + +/* +* Hypercall IDs of the ACRN Hypervisor +*/ +#define _HC_ID (x, y) (((x) << 24) | (y)) + +#define HC_ID 0x80UL + +#define HC_ID_VM_BASE 0x10UL +#define HC_CREATE_VM _HC_ID(HC_ID, HC_ID_VM_BASE + 0x00) +#define HC_DESTROY_VM _HC_ID(HC_ID, HC_ID_VM_BASE + 0x01) +#define HC_START_VM _HC_ID(HC_ID, HC_ID_VM_BASE + 0x02) +#define HC_PAUSE_VM _HC_ID(HC_ID, HC_ID_VM_BASE + 0x03) +#define HC_RESET_VM _HC_ID(HC_ID, HC_ID_VM_BASE + 0x05) + +/** +* hcall_create_vm() - Create a User VM +* @vminfo: Service VM GPA of info of User VM creation +* +* Return: 0 on success, <0 on failure +*/ +static inline long hcall_create_vm (u64 vminfo) +{ +return acrn_hypercall1 (HC_CREATE_VM , vminfo); +} + +/** +* hcall_start_vm() - Start a User VM +* @vmid: User VM ID +* +* Return: 0 on success, <0 on failure +*/ +static inline long hcall_start_vm (u64 vmid) +{ +return acrn_hypercall1 (HC_START_VM , vmid); +} + +/** +* hcall_pause_vm() - Pause a User VM +* @vmid: User VM ID +* +* Return: 0 on success, <0 on failure +*/ +static inline long hcall_pause_vm (u64 vmid) +{ +return acrn_hypercall1 (HC_PAUSE_VM , vmid); +} + +/** +* hcall_destroy_vm() - Destroy a User VM +* @vmid: User VM ID +* +* Return: 0 on success, <0 on failure +*/ +static inline long hcall_destroy_vm (u64 vmid) +{ +return acrn_hypercall1 (HC_DESTROY_VM , vmid); +} + +/** +* hcall_reset_vm() - Reset a User VM +* @vmid: User VM ID +* +* Return: 0 on success, <0 on failure +*/ +static inline long hcall_reset_vm (u64 vmid) +{ +return acrn_hypercall1 (HC_RESET_VM , vmid); +} + +#endif /* __ACRN_HSM_HYPERCALL_H */ drivers/virt/acrn/vm.c new file +32
-0@@ -0,0 +1,71 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* ACRN_HSM: Virtual Machine management +* +* Copyright (C) 2020 Intel Corporation. All rights reserved. +* +* Authors: +* Jason Chen CJ <[email protected]> +* Yakui Zhao <[email protected]> +*/ +#include <linux/io.h> +#include <linux/mm.h> +#include <linux/slab.h> + +#include "acrn_drv.h" + +/* List of VMs */ +LIST_HEAD (acrn_vm_list); +/* +* acrn_vm_list is read in a tasklet which dispatch I/O requests and is wrote +* in VM creation ioctl. Use the rwlock mechanism to protect it. +*/ +DEFINE_RWLOCK (acrn_vm_list_lock); + +struct acrn_vm *acrn_vm_create (struct acrn_vm *vm, +struct acrn_vm_creation *vm_param) +{ +int ret; + + ret =hcall_create_vm (virt_to_phys (vm_param)); +if (ret <0 || vm_param->vmid ==ACRN_INVALID_VMID ) { + dev_err(acrn_dev.this_device,drivers/virt/acrn/vm.c line 42
Quoted 10 lines
drivers/virt/acrn/vm.c new file +10
-0+ "Failed to create VM! Error: %d\n", ret); +return NULL ; + } + + vm->vmid = vm_param->vmid; + vm->vcpu_num = vm_param->vcpu_num; + +write_lock_bh (&acrn_vm_list_lock); +list_add (&vm->list, &acrn_vm_list); +write_unlock_bh (&acrn_vm_list_lock);Why are the
_bh()variants being used here?
Quoted 3 lines
You are only accessing this list from userspace context in this patch.
Heck, you aren't even reading from the list, only writing to it...
acrn_vm_list is read in a tasklet which dispatch I/O requests and is wrote in VM creation ioctl. Use the rwlock mechanism to protect it. The reading operation is introduced in the following patches of this series. So i keep the lock type at the moment of introduction.
Thanks
shuo
Re: [PATCH v4 06/17] virt: acrn: Introduce VM management interfaces
Quoted 6 lines · Shuo A Liu
Quoted 6 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ write_lock_bh(&acrn_vm_list_lock); + list_add(&vm->list, &acrn_vm_list); + write_unlock_bh(&acrn_vm_list_lock);Why are the
_bh()variants being used here?
Quoted 8 lines
Quoted 3 lines
You are only accessing this list from userspace context in this patch.
Heck, you aren't even reading from the list, only writing to it...
acrn_vm_listis read in a tasklet which dispatch I/O requests and is wrote in VM creation ioctl. Use the rwlock mechanism to protect it. The reading operation is introduced in the following patches of this series. So i keep the lock type at the moment of introduction.
Ok, but think about someone trying to review this code. Does this lock actually make sense here? No, it does not. How am I supposed to know to look at future patches to determine that it changes location and usage to require this?
That's just not fair, would you want to review something like this?
And a HUGE meta-comment, again, why am I the only one reviewing this stuff? Why do you have a ton of Intel people on the Cc: yet it is, once again, my job to do this?
If you all are wanting to burn me out, you are doing a good job...
greg k-h
Re: [PATCH v4 06/17] virt: acrn: Introduce VM management interfaces
Quoted 16 lines · Greg Kroah-Hartman
Quoted 6 lines · Shuo A Liu
Quoted 6 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ write_lock_bh(&acrn_vm_list_lock); + list_add(&vm->list, &acrn_vm_list); + write_unlock_bh(&acrn_vm_list_lock);Why are the
_bh()variants being used here?Quoted 8 lines
Quoted 3 lines
You are only accessing this list from userspace context in this patch.
Heck, you aren't even reading from the list, only writing to it...
acrn_vm_listis read in a tasklet which dispatch I/O requests and is wrote in VM creation ioctl. Use the rwlock mechanism to protect it. The reading operation is introduced in the following patches of this series. So i keep the lock type at the moment of introduction.
Quoted 4 lines
Ok, but think about someone trying to review this code. Does this lock actually make sense here? No, it does not. How am I supposed to know to look at future patches to determine that it changes location and usage to require this?
OK. May i know how to handle such kind of code submission? Or which way following do you prefer?
1) Use a mutex lock here, then change it to rwlock in a later patch of this series.
2) Add more comments in changelog about the lock. (Now, there is comment around the acrn_vm_list_lock)
Quoted 6 lines
That's just not fair, would you want to review something like this?
And a HUGE meta-comment, again, why am I the only one reviewing this stuff? Why do you have a ton of Intel people on the Cc: yet it is, once again, my job to do this?
The patchset has been reviewed in Intel's internal mailist several rounds and got Reviewed-by: before send out. That's why i Cced many Intel people as well.
This patchset is all about a common driver for the ACRN hypervisor support. I put the code in drivers/virt/ and found you are one of the maintainer of vboxguest driver which is in the same subdirectory. I thought you should be the right person to be Cced when i submitted this series.
Certainly, any comments are welcome. And really appreciate your review and help. I have little experience to submit a new driver to the community, my apologies if thing goes wrong.
Thanks
shuo
Re: [PATCH v4 06/17] virt: acrn: Introduce VM management interfaces
Quoted 22 lines · Shuo A Liu
Quoted 16 lines · Greg Kroah-Hartman
Quoted 6 lines · Shuo A Liu
Quoted 6 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ write_lock_bh(&acrn_vm_list_lock); + list_add(&vm->list, &acrn_vm_list); + write_unlock_bh(&acrn_vm_list_lock);Why are the
_bh()variants being used here?Quoted 8 lines
Quoted 3 lines
You are only accessing this list from userspace context in this patch.
Heck, you aren't even reading from the list, only writing to it...
acrn_vm_listis read in a tasklet which dispatch I/O requests and is wrote in VM creation ioctl. Use the rwlock mechanism to protect it. The reading operation is introduced in the following patches of this series. So i keep the lock type at the moment of introduction.Quoted 4 lines
Ok, but think about someone trying to review this code. Does this lock actually make sense here? No, it does not. How am I supposed to know to look at future patches to determine that it changes location and usage to require this?
Quoted 4 lines
OK. May i know how to handle such kind of code submission? Or which way following do you prefer?
1) Use a mutex lock here, then change it to rwlock in a later patch of this series.
Wouldn't this make more sense if you had to read these one after another?
Quoted 2 lines
2) Add more comments in changelog about the lock. (Now, there is comment around the
acrn_vm_list_lock)
It's hard to verify a comment's statement without digging through other patches in the series, right? You want the reviewer to just trust you? :)
Again, what would _YOU_ want to see if you had to review this?
Quoted 9 lines
Quoted 5 lines
That's just not fair, would you want to review something like this?
And a HUGE meta-comment, again, why am I the only one reviewing this stuff? Why do you have a ton of Intel people on the Cc: yet it is, once again, my job to do this?
The patchset has been reviewed in Intel's internal mailist several rounds and got Reviewed-by: before send out. That's why i Cced many Intel people as well.
Then why didn't any of those intel people on the cc: actually review it after you have sent it out? Why is it only me? Do I need to wait longer for them to get to this? I'll gladly do so next time...
Quoted 5 lines
This patchset is all about a common driver for the ACRN hypervisor support. I put the code in
drivers/virt/and found you are one of the maintainer of vboxguest driver which is in the same subdirectory. I thought you should be the right person to be Cced when i submitted this series.
I am, I'm not complaining about that. I'm complaining that it seems to be _only_ me reviewing this here, and not any of the people you are cc:ing from intel. Most of those people should be giving you this same type of review comments and not forcing an external person to do so, right?
Quoted 3 lines
Certainly, any comments are welcome. And really appreciate your review and help. I have little experience to submit a new driver to the community, my apologies if thing goes wrong.
You didn't do anything wrong, I'm arguing about the larger meta-issue I have right now with Intel and the lack of reviews that seems to happen from other Intel people on their co-workers patches.
Anyway, you are doing fine, it's an iterative process, hopefully you can also review other people's patches in this area that are being posted as well.
thanks,
greg k-h
Re: [PATCH v4 06/17] virt: acrn: Introduce VM management interfaces
Quoted 23 lines · Greg Kroah-Hartman
Quoted 22 lines · Shuo A Liu
Quoted 20 lines · Greg Kroah-Hartman
Quoted 6 lines · Shuo A Liu
Quoted 6 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ write_lock_bh(&acrn_vm_list_lock); + list_add(&vm->list, &acrn_vm_list); + write_unlock_bh(&acrn_vm_list_lock);Why are the
_bh()variants being used here?Quoted 8 lines
Quoted 3 lines
You are only accessing this list from userspace context in this patch.
Heck, you aren't even reading from the list, only writing to it...
acrn_vm_listis read in a tasklet which dispatch I/O requests and is wrote in VM creation ioctl. Use the rwlock mechanism to protect it. The reading operation is introduced in the following patches of this series. So i keep the lock type at the moment of introduction.Ok, but think about someone trying to review this code. Does this lock actually make sense here? No, it does not. How am I supposed to know to look at future patches to determine that it changes location and usage to require this?
Quoted 7 lines
Quoted 4 lines
OK. May i know how to handle such kind of code submission? Or which way following do you prefer?
1) Use a mutex lock here, then change it to rwlock in a later patch of this series.Wouldn't this make more sense if you had to read these one after another?
OK. I will change to mutex firstly for more readable.
Quoted 39 lines
Quoted 3 lines
2) Add more comments in changelog about the lock. (Now, there is comment around the
acrn_vm_list_lock)It's hard to verify a comment's statement without digging through other patches in the series, right? You want the reviewer to just trust you? :)
Again, what would _YOU_ want to see if you had to review this?
Quoted 9 lines
Quoted 5 lines
That's just not fair, would you want to review something like this?
And a HUGE meta-comment, again, why am I the only one reviewing this stuff? Why do you have a ton of Intel people on the Cc: yet it is, once again, my job to do this?
The patchset has been reviewed in Intel's internal mailist several rounds and got Reviewed-by: before send out. That's why i Cced many Intel people as well.
Then why didn't any of those intel people on the cc: actually review it after you have sent it out? Why is it only me? Do I need to wait longer for them to get to this? I'll gladly do so next time...
Quoted 5 lines
This patchset is all about a common driver for the ACRN hypervisor support. I put the code in
drivers/virt/and found you are one of the maintainer of vboxguest driver which is in the same subdirectory. I thought you should be the right person to be Cced when i submitted this series.I am, I'm not complaining about that. I'm complaining that it seems to be _only_ me reviewing this here, and not any of the people you are cc:ing from intel. Most of those people should be giving you this same type of review comments and not forcing an external person to do so, right?
Quoted 3 lines
Certainly, any comments are welcome. And really appreciate your review and help. I have little experience to submit a new driver to the community, my apologies if thing goes wrong.
Quoted 7 lines
You didn't do anything wrong, I'm arguing about the larger meta-issue I have right now with Intel and the lack of reviews that seems to happen from other Intel people on their co-workers patches.
Anyway, you are doing fine, it's an iterative process, hopefully you can also review other people's patches in this area that are being posted as well.
Sorry, i have no answer about some of your question above. :( However, i will try my best to help review other people's patches in this area.
Thanks
shuo