X86-64 Syscall API/ABI Document
Hi,
as promised here is an initial draft of the Syscall document I worked on. It is not ready yet, lacks some details and is definitely subject to change.
But I think the general ideas are in there already and it is a good enough base to define TDX and SNP specific VM management APIs on top of it.
So please review the document and provide your feedback. If time allows we can have some initial discussion around it in todays meeting. A more general discussion has to wait for the next meeting, when everyone had time to review it.
Regards,
Joerg
Hi Joerg,
Thank you for sharing the design document! It has greatly enhanced our understanding of the user mode design philosophy. Here are some questions and comments from our side:
THREAD_CREATE:
a) From the input parameter, seems it only supports creating thread on the current CPU. Is there a plan to allow creating thread on a remote CPU? Or we should avoid this case?
b) Is there any plan to add CPU affinity to threads?
BATCH:
a) Is the user_data in SysCallIn an identifier, to pair the SysCallOut and SysCallIn which have the same value for the user_data? Would like to better understand the usage of the user_data.
OPEN_PHYSMEM:
a) Suppose the SVMS kernel memory range is excluded, right?
b) Is there any mechanism to let the user-space process know the valid memory range represented by the object handle? E.g., the end of the memory addresses
CAPABILITIES:
a) For SEV, VMPL0 is svsm and VM starts from VMPL1, and for TDX, vm-id 0 is svsm and VM starts from vm-id 1. From this point of view, looks like SEV and TDX are similar. So maybe can use a unified VM index bitmap format for them?
VM_OPEN:
a) If one VM index is opened by a user-space process, the same index cannot be opened again by another user-space process which guarantees that only one process can operate with this VM object handle. But the all threads created by this user-space process can operate with this VM object handle. Is this a correct understanding?
VM_CAPABILITIES:
a) Should report the VM memory range? Or always assume VM can see the entire range supported by the ObjHandle created by OPEN_PHYSMEM?
MMIO/IOIO_EVENT:
a) Is there any way to know the length of the mappable area provided by the evt_obj? Or the length is some fixed size defined by the kernel mode?
b) Would like to have a better understanding of the usage. We assume that, the MMIO/IOIO range set by the syscalls are monitored by some device model thread in the user space. When VM occurs vmexit due to MMIO/IOIO accessing, the kernel vcpu thread decodes the MMIO/IOIO address and if the address is located in the range set by these syscalls, the kernel vcpu thread wakes up the monitoring device model thread and then waits. The device model thread in the user mode will get the address/data via the mappable area and do emulation. After the emulation is completed, the device model thread wakes up the kernel vcpu thread. If the decoding part is not in the kernel-space but in the user-space, then the vcpu should back to the user mode to decode instruction and wait to be waked up by the device model thread. Is this the expected flow for handling MMIO/IOIO in the user mode?
SET_MEM_STATE:
a) Any consideration for using paddr + page_size instead of start_paddr + end_paddr as the input parameters? Using start_paddr + end_paddr may be more efficient to set state for a memory region.
VCPU_CREATE:
a) Regarding "The VCPU can be run with the WAIT_FOR_EVENT() system call", the expectation is that the WAIT_FOR_EVENT(vcpu_obj) will perform vmenter in the kernel mode, is this understanding correct?
GET/SET_STATE:
a) The type struct VmsaGpa is defined as AMD only. Does this mean the user mode should have knowledge about if the platform is SEV or TDX? Or there will be a per-architecture user mode VMM executable binary?
b) For TDX, all the VCPU states are stored in the VMCS. Probably need to introduce a new type for TDX to set/get vcpu state:
Type Data Structure Description
--------------------------------------------------------------------------------------------------------
VMCS Struct VmcsField The VMCS field encoding and the
corresponding data (Intel Only)
Struct VmcsField {
encoding: u32, // VMCS field encoding
data: u64, // VMCS field data
}
GET/SET_STATE syscall can be batched so that can set/get multiple VCPU states by one time.
c) The vLAPIC is emulated in the kernel mode. In this case, is it necessary to allow user mode get the vLAPIC state? The potential user case is for the user mode to emulate vCPUID 01 EBX which requires the vLAPIC ID.
Possible new syscalls for Class0: to help the user mode to execute some privilege instructions:
a) The user mode may need to read/write L1's MSR to emulate the vMSR for VM, but rdmsr/wrmsr instructions are not allowed in the user mode for both SEV and TDX. The RDMSR/WRMSR syscalls can help the user mode to achieve this:
ACCESS_MSR
Read data from a MSR, or write data to a MSR
System Call number: 11
Parameters
-------------------------------------------------------------------------------------------------
Parameter Type Description
-------------------------------------------------------------------------------------------------
1. MSR index u32 The MSR to access
2. read bool Indicate if this is a MSR read or write.
2. data VirtAddr The virtual address to store the data
reading from MSR, or the data written to the MSR.
Returns 0 on success, error code on failure
b) The user mode may need to access some MMIO or IO port emulated by L0 to emulate certain MMIO/IOIO event for VM. The accessing in user mode can be done via MOV/IOIO instructions, but it can trigger #VC/#VE from the user mode and requires instruction decoding/emulating. To simplify, the MMIO/IOIO accessing can be done via the enlighten way (VMGEXIT for SEV and TDCALL for TDX). But for TDX, the TDCALL is privileged command which is not allowed in the user mode. So it has to be done in the kernel mode. If using the enlighten way for the user mode to access MMIO/IOIO is preferred, then new syscalls are necessary for TDX. Shall we? ACCESS_IO
Read data from an IO port, or write data to an IO port.
System Call number: 12
Parameters
-------------------------------------------------------------------------------------------------
Parameter Type Description
-------------------------------------------------------------------------------------------------
1. port u16 The IO port to access.
2. size u8 The IO port size. Only 1(u8)/2(u16)/4(u32)
are supported.
3. read bool Indicate if this is an IO read or write.
4. data VirtAddr The virtual address to store the data
reading from the IO port, or the data written to the IO port.
Returns 0 on success, error code on failure
ACCESS_MMIO
Read a MMIO, or write data to a MMIO.
System Call number: 13
Parameters
-------------------------------------------------------------------------------------------------
Parameter Type Description
-------------------------------------------------------------------------------------------------
1. paddr u64 The MMIO address to access.
2. size u8 The MMIO size. Only 1(u8)/2(u16)/4(u32)/
8(u64) are supported.
3. read bool Indicate if this is a MMIO read or write.
4. data VirtAddr The virtual address to store the data
reading from the MMIO, or the data written to the MMIO
Returns 0 on success, error code on failure
Questions for ObjHandle:
a) Although the ObjHandle seems a common type, a given ObjHandle should only be used as an input for specific syscalls. For example, the ObjHandle returned by OPEN cannot be used as input for the VM_CAPABILITIES. Is this a correct understanding?
b) If a) is true, sounds like for a given ObjHandle, the user should know how the ObjHandle is created, otherwise the it is hard to distinguish which syscalls can use this ObjHandle as an input?
Thanks
Chuanxiao
Quoted 21 lines
-----Original Message-----
From: Svsm-devel <[email protected]> On Behalf Of J?rg R?del
Sent: Wednesday, June 12, 2024 8:24 PM
To: [email protected]
Subject: [svsm-devel] X86-64 Syscall API/ABI DocumentHi,
as promised here is an initial draft of the Syscall document I worked on. It is not ready yet, lacks some details and is definitely subject to change.
But I think the general ideas are in there already and it is a good enough base to define TDX and SNP specific VM management APIs on top of it.
So please review the document and provide your feedback. If time allows we can have some initial discussion around it in todays meeting. A more general discussion has to wait for the next meeting, when everyone had time to review it.
Regards,
Joerg
Hi Chuanxiao,
Thanks for your feedback and questions, please see my comments inline.
Quoted 7 lines · Chuanxiao
Thank you for sharing the design document! It has greatly enhanced our understanding of the user mode design philosophy. Here are some questions and comments from our side:
THREAD_CREATE:
a) From the input parameter, seems it only supports creating thread on the current CPU. Is there a plan to allow creating thread on a remote CPU? Or we should avoid this case?
b) Is there any plan to add CPU affinity to threads?
There will be no direct concept of CPU affinity for threads. Affinity is implemented via event sources. Each event source is either global or bound to a specific CPU. If a thread waits on a CPU-bound event source and is woken up, it will get to run on the CPU the event is bound to.
So, for example, if user-space aquired an object representing a specific VCPU and waits for events on that object, the thread will run on the CPU the VCPU object is bound to, which is the same CPU as where the event happened.
Quoted 4 lines
BATCH:
a) Is theuser_datain SysCallIn an identifier, to pair the SysCallOut and SysCallIn which have the same value for theuser_data? Would like to better understand the usage of theuser_data.
User-data is a just a piece of data forwared from SysCallIn to SysCallOut to help user-space to associate the request with its internal state. The data is not interpreted by the COCONUT kernel in any way.
Quoted 2 lines
OPEN_PHYSMEM:
a) Suppose the SVMS kernel memory range is excluded, right?
Yes, it definitly is.
Quoted 3 lines
b) Is there any mechanism to let the user-space process know the valid memory range represented by the object handle? E.g., the end of the memory addresses
No, not yet, we need to add a way to inform user-space about the valid memory ranges.
In general there also needs to be a limit on how much user-space can map, as we need to avoid that user-space just maps everything forever.
There is also a requirement that the COCONUT kernel can not allow shared/private state changes of pages which are currently mapped to user-space, so any physmem mapping needs to be temporary.
Quoted 5 lines
CAPABILITIES:
a) For SEV, VMPL0 is svsm and VM starts from VMPL1, and for TDX, vm-id 0 is svsm and VM starts from vm-id 1. From this point of view, looks like SEV and TDX are similar. So maybe can use a unified VM index bitmap format for them?
Yes, that is the idea. I want to keep the interface as common as possible between TDX and SEV.
Quoted 6 lines
VM_OPEN:
a) If one VM index is opened by a user-space process, the same index cannot be opened again by another user-space process which guarantees that only one process can operate with this VM object handle. But the all threads created by this user-space process can operate with this VM object handle. Is this a correct understanding?
Yes, this is correct. Though what is missing so far is the ability to pass object handles to newly created processes. This is important for device emulations like the TPM or serial port. The MMIO or IOIO object handles have to be created from the VM object handle, but the handling of actual MMIO and IOIO events needs to happen in separate processes. So there needs to be a way for forward handles on EXEC.
Quoted 3 lines
VM_CAPABILITIES:
a) Should report the VM memory range? Or always assume VM can see the entire range supported by the ObjHandle created byOPEN_PHYSMEM?
This depends on how the memory ranges given to specific VMs are going to be defined. It should probably be up to user-space to assign memory to a given VM. I need to think more about that.
Quoted 4 lines
MMIO/IOIO_EVENT:
a) Is there any way to know the length of the mappable area provided by theevt_obj? Or the length is some fixed size defined by the kernel mode?
I consider these architecturally defined, the MMIO area range is all of physical memory, the IOIO range is the whole IO port range.
Quoted 5 lines
b) Would like to have a better understanding of the usage. We assume that, the MMIO/IOIO range set by the syscalls are monitored by some device model thread in the user space. When VM occurs vmexit due to MMIO/IOIO accessing, the kernel vcpu thread decodes the MMIO/IOIO
address and if the address is located in the range set by
Quoted 8 lines
these syscalls, the kernel vcpu thread wakes up the monitoring device model thread and then waits. The device model thread in the user mode will get the address/data via the mappable area and do emulation. After the emulation is completed, the device model thread wakes up the kernel vcpu thread. If the decoding part is not in the kernel-space but in the user-space, then the vcpu should back to the user mode to decode instruction and wait to be waked up by the device model thread. Is this the expected flow for handling MMIO/IOIO in the user mode?
As I have written above, the device models should run in a separate process. When the COOCNUT kernel is waked up with an MMIO or IOIO event from a guest is decodes the instruction and based on the address which is accessed it will wake up the corresponding device emulation process.
That process then handles the request and updates device and CPU state. Upon return to kernel mode COCONUT will directly go back into executing the guest OS.
I think this can be done without or with very minimal involvement of the main VM management process.
Quoted 4 lines
SET_MEM_STATE:
a) Any consideration for using paddr +page_sizeinstead ofstart_paddr+end_paddras the input parameters? Usingstart_paddr+end_paddrmay be more efficient to set state for a memory region.
The SET_MEM_STATE input is defined to also suit AMDs use-case. on AMD the PVALIDATE instruction needs a page-size specified, so there needs to be a way to pass that information in. This is difficult to do with a range-based interface.
Also, errors are harder to propagate on a range-based input, as validation can fail in the middle of a range and user-space needs a consistent picture of what was validated.
For validating a whole range the BATCH system call can be used to send as many SET_MEM_STATE requests as needed in one invocation.
Quoted 4 lines
VCPU_CREATE:
a) Regarding "The VCPU can be run with theWAIT_FOR_EVENT()system call", the expectation is that theWAIT_FOR_EVENT(vcpu_obj)will perform vmenter in the kernel mode, is this understanding correct?
Yes, this is correct. I had a RUN system call specified initially, but found it redundant with the WAIT_FOR_EVENT call in this case.
Quoted 4 lines
GET/SET_STATE:
a) The type struct VmsaGpa is defined as AMD only. Does this mean the user mode should have knowledge about if the platform is SEV or TDX? Or there will be a per-architecture user mode VMM executable binary?
My hope is that we can one binary which is mostly common between TDX and SEV-SNP. The binary has to be aware of the platform as there will always be platform-specific handling required to some degree, but a lot of the main handling infrastructure can be common, I think.
Quoted 12 lines
b) For TDX, all the VCPU states are stored in the VMCS. Probably need to introduce a new type for TDX to set/get vcpu state:
Type Data Structure Description
--------------------------------------------------------------------------------------------------------
VMCS Struct VmcsField The VMCS field encoding and the
corresponding data (Intel Only)
Struct VmcsField {
encoding: u32, // VMCS field encoding
data: u64, // VMCS field data
}
GET/SET_STATE syscall can be batched so that can set/get multiple VCPU states by one time.
This depends, a lot of state in the VMCS also exists in the VMSA, for those there will be common state definitions, e.g. for:
* GPRs
* FPU/XMM/YMM state
* Control registers
* Architectural MSRs (if needed)
* APIC and IRQ injection state
There are probably some VMCS specific fields which are not covered by common state abstractions, for those TDX specific state calls are fine.
Quoted 4 lines
c) The vLAPIC is emulated in the kernel mode. In this case, is it necessary to allow user mode get the vLAPIC state? The potential user case is for the user mode to emulate vCPUID 01 EBX which requires the vLAPIC ID.
It is at least a way needed for device emulation processes to inject an IRQ into the guest OS. Full APIC access might not be needed, the vLAPIC ID can also be propagated via a VCPU GET_STATE call or something similar.
Quoted 5 lines
Possible new syscalls for Class0: to help the user mode to execute some privilege instructions:
a) The user mode may need to read/write L1's MSR to emulate the vMSR
for VM, but rdmsr/wrmsr instructions are not allowed in the user mode
for both SEV and TDX. The RDMSR/WRMSR syscalls can help the user mode
to achieve this:
Can you elaborate a bit on that? Whic L1 MSRs need to be accessible and for what reason? In general I do not think it is a good idea to allow user-space accessing arbitratry MSRs.
Quoted 10 lines
b) The user mode may need to access some MMIO or IO port emulated by L0 to emulate certain MMIO/IOIO event for VM. The accessing in user mode can be done via MOV/IOIO instructions, but it can trigger #VC/#VE from the user mode and requires instruction decoding/emulating. To simplify, the MMIO/IOIO accessing can be done via the enlighten way (VMGEXIT for SEV and TDCALL for TDX). But for TDX, the TDCALL is privileged command which is not allowed in the user mode. So it has to be done in the kernel mode. If using the enlighten way for the user mode to access MMIO/IOIO is preferred, then new syscalls are necessary for TDX. Shall we?
Yes, you are right, a way for user-space to forward request to the host is currently completely missing.
I wonder what the best way here is, but routing these through user-space if maybe not the best approach, or do you see cases where user-space won't forward request unmodified?
Other options would be:
* Let the COCONUT kernel forward MMIO and IOIO requests for which it has no handler to the host and use the result for emulation
* Re-inject these events as #VC/#VE execeptions into the guest OS and let it deal with it.
The right approach probably also depends on wether the OS runs in englightened or paravisor mode.
Quoted 5 lines
Questions for ObjHandle:
a) Although the ObjHandle seems a common type, a given ObjHandle should only be used as an input for specific syscalls. For example, the ObjHandle returned by OPEN cannot be used as input for theVM_CAPABILITIES. Is this a correct understanding?
Yes, that is correct. The system call design follows an "everything is an object" model with the system call classes being traits for these objects. Every object only implements a subset of system call classes.
Quoted 3 lines
b) If a) is true, sounds like for a given ObjHandle, the user should know how the ObjHandle is created, otherwise the it is hard to distinguish which syscalls can use this ObjHandle as an input?
Yes, user-space knows what type an object handle is of by the way it created it. VM objects cans only be created by VM_OPEN, VCPU objects only by VCPU_CREATE, and so on.
(VCPU_CREATE should be renamed to VCPU_OPEN, as VCPUs conceptually
already exist, I will update the document).
Regards,
Joerg
Signature · 11 lines
Jörg Rödel [email protected] SUSE Software Solutions Germany GmbH Frankenstraße 146 90461 Nürnberg Germany https://www.suse.com/ Geschäftsführer: Ivo Totev, Andrew McDonald, Werner Knoblich (HRB 36809, AG Nürnberg)
Quoted 22 lines
-----Original Message-----
From: Jörg Rödel <[email protected]>
Sent: Friday, June 21, 2024 6:09 PM
To: Dong, Chuanxiao <[email protected]>
Cc: Jörg Rödel <[email protected]>; [email protected]
Subject: Re: [svsm-devel] X86-64 Syscall API/ABI DocumentHi Chuanxiao,
Thanks for your feedback and questions, please see my comments inline.
Quoted 1 line · Chuanxiao
Thank you for sharing the design document! It has greatly enhanced our understanding of the user
mode design philosophy. Here are some questions and comments from our side:
Quoted 6 lines
THREAD_CREATE:
a) From the input parameter, seems it only supports creating thread on the current CPU. Is there a plan to allow creating thread on a remote CPU? Or we should avoid this case?
b) Is there any plan to add CPU affinity to threads?
Quoted 7 lines
There will be no direct concept of CPU affinity for threads. Affinity is implemented via event sources. Each event source is either global or bound to a specific CPU. If a thread waits on a CPU-bound event source and is woken up, it will get to run on the CPU the event is bound to.
So, for example, if user-space aquired an object representing a specific VCPU and waits for events on that object, the thread will run on the CPU the VCPU object is bound to, which is the same CPU as where the event happened.
I see. So still taking vcpu as an example, user space process first use THREAD_CREATE to create a vcpu thread but this vcpu thread runs on the current CPU. The vcpu thread uses VCPU_CREATE syscall to get a vcpu_obj and uses WAIT_FOR_EVENT(vcpu_obj) syscall to schedule it to the CPU where the vcpu_obj is bound to, and then performs the VM enter on the right CPU.
Quoted 9 lines
Quoted 6 lines
BATCH:
a) Is theuser_datain SysCallIn an identifier, to pair the SysCallOut and SysCallIn which have the same value for theuser_data? Would like to better understand the usage of theuser_data.User-data is a just a piece of data forwared from SysCallIn to SysCallOut to help user-space to associate the request with its internal state. The data is not interpreted by the COCONUT kernel in any way.
When the BATCH syscall returns, if the user_data in the SysCallIn is the same with the user_data in the corresponding SysCallOut, means kernel has forwarded the user_data to SysCallOut which represents this sub syscall is completed. Is this understanding correct?
Quoted 10 lines
Quoted 3 lines
OPEN_PHYSMEM:
a) Suppose the SVMS kernel memory range is excluded, right?Yes, it definitly is.
Quoted 3 lines
b) Is there any mechanism to let the user-space process know the valid memory range represented by the object handle? E.g., the end of the memory addresses
Quoted 7 lines
No, not yet, we need to add a way to inform user-space about the valid memory ranges.
In general there also needs to be a limit on how much user-space can map, as we need to avoid that user-space just maps everything forever.
There is also a requirement that the COCONUT kernel can not allow shared/private state changes of pages which are currently mapped to user-space, so any physmem mapping needs to be temporary.
Makes sense. So the SET_MEM_STATE can fail to change page shared/private state due to the page is MMAP, until such page is MUNMAP.
Quoted 16 lines
Quoted 6 lines
CAPABILITIES:
a) For SEV, VMPL0 is svsm and VM starts from VMPL1, and for TDX, vm-id 0 is svsm and VM starts from vm-id 1. From this point of view, looks like SEV and TDX are similar. So maybe can use a unified VM index bitmap format for them?Yes, that is the idea. I want to keep the interface as common as possible between TDX and SEV.
Quoted 6 lines
VM_OPEN:
a) If one VM index is opened by a user-space process, the same index cannot be opened again by another user-space process which guarantees that only one process can operate with this VM object handle. But the all threads created by this user-space process can operate with this VM object handle. Is this a correct understanding?
Quoted 4 lines
Yes, this is correct. Though what is missing so far is the ability to pass object handles to newly created processes. This is important for device emulations like the TPM or serial port. The MMIO or IOIO object handles have to be created from the VM object handle, but the handling of actual MMIO and IOIO events needs to happen in separate processes. So there needs to be a way for forward handles on EXEC.
Yes. As this is related a process to use object handles created by another process, it appears that there is no policy restricting syscall operations on the object handle exclusively to the process that created it, and the object handle is unique globally, is this correct?
Quoted 8 lines
Quoted 4 lines
VM_CAPABILITIES:
a) Should report the VM memory range? Or always assume VM can see the entire range supported by the ObjHandle created byOPEN_PHYSMEM?This depends on how the memory ranges given to specific VMs are going to be defined. It should probably be up to user-space to assign memory to a given VM. I need to think more about that.
Quoted 7 lines
Quoted 4 lines
MMIO/IOIO_EVENT:
a) Is there any way to know the length of the mappable area provided by theevt_obj? Or the length is some fixed size defined by the kernel mode?I consider these architecturally defined, the MMIO area range is all of physical memory, the IOIO range is the whole IO port range.
Does this mean, for MMIO_EVENT, it can take any valid range as input parameters, no matter it is MMIO or RAM, and for IOIO_EVENT, it can take any valid IO port range as input parameters?
Actually I also want to understand the mappable area provided by an evt_obj. As it is mappable area, suppose it can be MMAP(). So wondering how this mappable area be used?
Quoted 15 lines
Quoted 14 lines
b) Would like to have a better understanding of the usage. We assume that, the MMIO/IOIO range set by the syscalls are monitored by some device model thread in the user space. When VM occurs vmexit due to MMIO/IOIO accessing, the kernel vcpu thread decodes the MMIO/IOIO
address and if the address is located in the range set by
these syscalls, the kernel vcpu thread wakes up the monitoring device model thread and then waits. The device model thread in the user mode will get the address/data via the mappable area and do emulation. After the emulation is completed, the device model thread wakes up the kernel vcpu thread. If the decoding part is not in the kernel-space but in the user-space, then the vcpu should back to the user mode to decode instruction and wait to be waked up by the device model thread. Is this the expected flow for handling MMIO/IOIO in the user mode?
Quoted 3 lines
As I have written above, the device models should run in a separate process. When the COOCNUT kernel is waked up with an MMIO or IOIO event from a guest is decodes the instruction and based on the address which is accessed it will wake up the corresponding device emulation process.
So MMIO_EVENT/IOIO_EVENT syscalls tell the kernel that, the range specified in the syscalls is emulated by some user-space process. If an MMIO/IOIO address is located in this range, the corresponding user-space process should be waked up, right?
BTW, what we though is when MMIO/IOIO event happens, vcpu thread exits from the guest mode and decodes the instruction to get the MMIO address and data (if it is an MMIO write), then waked up the device model process. I think this matches with what you described here. Please let me know if it is not.
Quoted 2 lines
That process then handles the request and updates device and CPU state.
How the device model process to get the detailed request? I guess it is via the mappable area provided by the evt_obj, but not sure so would like to raise the above questions about the mappable area.
Any consideration for updating CPU state in the device model process? Taking the MMIO reading as an example, the reading data may be written back to the vCPU register, or VM's memory (depending on the instruction). My understanding is that this is part of the instruction decoding. The device model process can emulate the MMIO/IOIO request and provide the emulated data to the process which did the instruction decoding, and that process can write the data back via the instruction decoder.
Quoted 4 lines
Upon return to kernel mode COCONUT will directly go back into executing the guest OS.
I think this can be done without or with very minimal involvement of the main VM management process.
Does the "main VM management process" represent the vCPU threads?
Quoted 10 lines
Quoted 5 lines
SET_MEM_STATE:
a) Any consideration for using paddr +page_sizeinstead ofstart_paddr+end_paddras the input parameters? Usingstart_paddr+end_paddrmay be more efficient to set state for a memory region.The
SET_MEM_STATEinput is defined to also suit AMDs use-case. on AMD the PVALIDATE instruction needs a page-size specified, so there needs to be a way to pass that information in. This is difficult to do with a range-based interface.
Quoted 5 lines
Also, errors are harder to propagate on a range-based input, as validation can fail in the middle of a range and user-space needs a consistent picture of what was validated.
For validating a whole range the BATCH system call can be used to send as many
SET_MEM_STATErequests as needed in one invocation.
Got it. Then it makes more sense for using paddr + page_size.
Quoted 9 lines
Quoted 5 lines
VCPU_CREATE:
a) Regarding "The VCPU can be run with theWAIT_FOR_EVENT()system call", the expectation is that theWAIT_FOR_EVENT(vcpu_obj)will perform vmenter in the kernel mode, is this understanding correct?Yes, this is correct. I had a RUN system call specified initially, but found it redundant with the
WAIT_FOR_EVENTcall in this case.
Quoted 8 lines
Quoted 4 lines
GET/SET_STATE:
a) The type struct VmsaGpa is defined as AMD only. Does this mean the user mode should have knowledge about if the platform is SEV or TDX? Or there will be a per-architecture user mode VMM executable binary?My hope is that we can one binary which is mostly common between TDX and SEV-SNP. The binary has to be aware of the platform as there will always be platform-specific handling required to some degree, but a lot of the main handling infrastructure can be common, I think.
Additional question is how to make the binary aware of the platform? Though some compiling option or some syscall?
Quoted 17 lines
Quoted 4 lines
b) For TDX, all the VCPU states are stored in the VMCS. Probably need to introduce a new type for TDX to set/get vcpu state:
Quoted 9 lines
Type Data Structure Description
--------------------------------------------------------------------------------------------------------
VMCS Struct VmcsField The VMCS field encoding and the
corresponding data (Intel Only)
Struct VmcsField {
encoding: u32, // VMCS field encoding
data: u64, // VMCS field data
}
GET/SET_STATE syscall can be batched so that can set/get multiple VCPU states by one time.This depends, a lot of state in the VMCS also exists in the VMSA, for those there will be common state definitions, e.g. for:
Quoted 8 lines
* GPRs
* FPU/XMM/YMM state
* Control registers
* Architectural MSRs (if needed)
* APIC and IRQ injection stateThere are probably some VMCS specific fields which are not covered by common state abstractions, for those TDX specific state calls are fine.
I suppose when GET/SET_STATE(VMSA_GPA), the kernel mode will return all the CPU states defined in the VMSA page by coping data from one page to another, is this correct? For TDX, this is more expensive because reading one VMCS field will require sending one tdcall to TDX module. For example, if user mode only wants to know CR0, it is better for TDX to just send one tdcall to read the CR0 field instead of sending many tdcalls to read all the CPU state from VMCS field to construct the VMSA page.
Quoted 8 lines
Quoted 6 lines
c) The vLAPIC is emulated in the kernel mode. In this case, is it necessary to allow user mode get the vLAPIC state? The potential user case is for the user mode to emulate vCPUID 01 EBX which requires the vLAPIC ID.
It is at least a way needed for device emulation processes to inject an IRQ into the guest OS.
Indeed, we can think more about the way for the device to process to inject an IRQ. SET_STATE might be an option.
Quoted 2 lines
Full APIC access might not be needed, the vLAPIC ID can also be propagated via a VCPU
GET_STATEcall or something similar.
Agree.
Quoted 9 lines
Quoted 6 lines
Possible new syscalls for Class0: to help the user mode to execute some privilege instructions:
a) The user mode may need to read/write L1's MSR to emulate the vMSR
for VM, but rdmsr/wrmsr instructions are not allowed in the user mode
for both SEV and TDX. The RDMSR/WRMSR syscalls can help the user mode
to achieve this:Can you elaborate a bit on that? Whic L1 MSRs need to be accessible and for what reason? In general I do not think it is a good idea to allow user-space accessing arbitratry MSRs.
Sure. As some of the L1's MSR are not used by svsm, we "passthrough" such MSR to the VM to simplify the MSR emulation. When VM accesses such MSR, the vmexit is handled by reading from/writing to the corresponding L1's MSR. Here is the list of "passthrough" MSR we made for TDP guest: https://github.com/intel-staging/td-partitioning-svsm/blob/svsm-tdp/kernel/src/tdx/vmsr.rs#L595
Quoted 12 lines
Quoted 11 lines
b) The user mode may need to access some MMIO or IO port emulated by L0 to emulate certain MMIO/IOIO event for VM. The accessing in user mode can be done via MOV/IOIO instructions, but it can trigger #VC/#VE from the user mode and requires instruction decoding/emulating. To simplify, the MMIO/IOIO accessing can be done via the enlighten way (VMGEXIT for SEV and TDCALL for TDX). But for TDX, the TDCALL is privileged command which is not allowed in the user mode. So it has to be done in the kernel mode. If using the enlighten way for the user mode to access MMIO/IOIO is preferred, then new syscalls are necessary for TDX. Shall we?
Quoted 4 lines
Yes, you are right, a way for user-space to forward request to the host is currently completely missing.
I wonder what the best way here is, but routing these through user-space if maybe not the best approach, or do you see cases where user-space won't forward request unmodified?
I don't see the need for routing such MMIO/IOIO requests from the kernel-space to the user-space, if we allow kernel-space to directly forward these MMIO/IOIO requests to host.
Quoted 10 lines
Other options would be:
* Let the COCONUT kernel forward MMIO and IOIO requests for which it has no handler to the host and use the result for emulation
* Re-inject these events as #VC/#VE execeptions into the guest OS and let it deal with it.The right approach probably also depends on wether the OS runs in englightened or paravisor mode.
For the either enlightened or paravisor mode, is the 1st option workable for both? If so, looks like the 1st option is better as it is more efficient.
Quoted 14 lines
Quoted 6 lines
Questions for ObjHandle:
a) Although the ObjHandle seems a common type, a given ObjHandle should only be used as an input for specific syscalls. For example, the ObjHandle returned by OPEN cannot be used as input for theVM_CAPABILITIES. Is this a correct understanding?Yes, that is correct. The system call design follows an "everything is an object" model with the system call classes being traits for these objects. Every object only implements a subset of system call classes.
Quoted 3 lines
b) If a) is true, sounds like for a given ObjHandle, the user should know how the ObjHandle is created, otherwise the it is hard to distinguish which syscalls can use this ObjHandle as an input?
Quoted 5 lines
Yes, user-space knows what type an object handle is of by the way it created it. VM objects cans only be created by
VM_OPEN, VCPU objects only byVCPU_CREATE, and so on.(
VCPU_CREATEshould be renamed toVCPU_OPEN, as VCPUs conceptually already exist, I will update the document).
Got it. This renaming sounds good to me.
Thanks
Chuanxiao
Quoted 16 lines
Regards,
Joerg
--
Jörg Rödel
[email protected]SUSE Software Solutions Germany GmbH
Frankenstraße 146
90461 Nürnberg
Germany
https://www.suse.com/Geschäftsführer: Ivo Totev, Andrew McDonald, Werner Knoblich (HRB 36809, AG Nürnberg)
Hi Chuanxiao,
Quoted 6 lines · Chuanxiao
I see. So still taking vcpu as an example, user space process first use
THREAD_CREATEto create a vcpu thread but this vcpu thread runs on the current CPU. The vcpu thread usesVCPU_CREATEsyscall to get avcpu_objand usesWAIT_FOR_EVENT(vcpu_obj)syscall to schedule it to the CPU where thevcpu_objis bound to, and then performs the VM enter on the right CPU.
Right, this is how thread-cpu affinity is designed to work.
Quoted 4 lines
When the BATCH syscall returns, if the
user_datain the SysCallIn is the same with theuser_datain the corresponding SysCallOut, means kernel has forwarded theuser_datato SysCallOut which represents this sub syscall is completed. Is this understanding correct?
Yes, but note that no syscall can be considered complete until the actual BATCH syscall returns.
Quoted 3 lines
Makes sense. So the
SET_MEM_STATEcan fail to change page shared/private state due to the page is MMAP, until such page is MUNMAP.
Correct, the COCONUT kernel needs to track which PHYSMEM pages are mapped and fail PSC on them.
Quoted 5 lines
Yes. As this is related a process to use object handles created by another process, it appears that there is no policy restricting syscall operations on the object handle exclusively to the process that created it, and the object handle is unique globally, is this correct?
An object handle itself is not global, but local to the process acquiring it. It will proably be just an index into an in-kernel array.
What an object handle points to is usually global, but depending on what it points to there can be multiple handles to the same object (e.g. files) or just one (e.g. VM, VCPUs).
Passing object handles to new processes will be semantically equivalent to closing the handle in the originating process and open it again in the new process.
There must be direct way via the EXEC syscall for this as the new process can not open some handles directly. E.g. a handle for an mmio range needs a vm handle to be created, but there can only be one handle per vm at any time.
Quoted 3 lines
Does this mean, for
MMIO_EVENT, it can take any valid range as input parameters, no matter it is MMIO or RAM, and forIOIO_EVENT, it can take any valid IO port range as input parameters?
I don't see why not, if it is a RAM range there will be no events reported, as the hardware will not generate them. Also, any limitation here would need to trust information from the HV on where the MMIO regions actually are.
Quoted 3 lines
Actually I also want to understand the mappable area provided by an
evt_obj. As it is mappable area, suppose it can beMMAP(). So wondering how this mappable area be used?
The idea is to use it for sharing information related to the event. E.g. the for a VCPU obj the area will contain information about the last exit reason. For MMIO it will contain information about the actual access which triggered the event.
Quoted 4 lines
So MMIO_EVENT/IOIO_EVENT syscalls tell the kernel that, the range specified in the syscalls is emulated by some user-space process. If an MMIO/IOIO address is located in this range, the corresponding user-space process should be waked up, right?
Correct.
Quoted 5 lines
BTW, what we though is when MMIO/IOIO event happens, vcpu thread exits from the guest mode and decodes the instruction to get the MMIO address and data (if it is an MMIO write), then waked up the device model process. I think this matches with what you described here. Please let me know if it is not.
Yes, that is the envisioned flow. After an exit from the guest the COCONUT kernel runs and decodes the event. In case of an MMIO or IOIO event it will also decode the instruction and look up the event which needs to be triggered. Once triggered, the user-space process running the device model wakes up and handles the event.
Quoted 3 lines
How the device model process to get the detailed request? I guess it is via the mappable area provided by the
evt_obj, but not sure so would like to raise the above questions about the mappable area.
Yes, information about the actual event will be stored in the mmap'ed area of the event object.
Quoted 8 lines
Any consideration for updating CPU state in the device model process? Taking the MMIO reading as an example, the reading data may be written back to the vCPU register, or VM's memory (depending on the instruction). My understanding is that this is part of
the instruction decoding. The device model process can emulate the MMIO/IOIO request and provide the emulated data to the process which did the instruction decoding, and that process can write the data back via the instruction decoder.
Correct, the plan is that the COCONUT kernel does the state updates for MMIO and IOIO events.
Quoted 1 line
Does the "main VM management process" represent the vCPU threads?
The main VM management process drives a VM and starts separate threads for each VCPU. All of these will be in one process within the same address space.
Quoted 2 lines
Additional question is how to make the binary aware of the platform? Though some compiling option or some syscall?
There will be initial support for writing binaries for the COCONUT platform. This will include a basic library around the system calls, platform setup code, memory allocator and so on.
All of this will hopefully flow into making COCONUT a Rust platform target, so that binaries can be written using Rust standard library support. But in the beginning there will only be a special library crate to build binaries against.
Quoted 8 lines
I suppose when GET/SET_STATE(VMSA_GPA), the kernel mode will return all the CPU states defined in the VMSA page by coping data from one page to another, is this correct? For TDX, this is more expensive because reading one VMCS field will require sending one tdcall to TDX module. For example, if user mode only wants to know CR0, it is better for TDX to just send one tdcall to read the CR0 field instead of sending many tdcalls to read all the CPU state from VMCS field to construct the VMSA page.
Okay, it might make sense to have a more fine-grained state access. That should not be a problem, on the AMD platform these calls can be batched and still update as many state as desired with one user/kernel transition.
Quoted 2 lines
Indeed, we can think more about the way for the device to process to inject an IRQ.
SET_STATEmight be an option.
SET_STATE is bound to the VCPU object, which is not accessible in the device emulation process. From the model that evolves here an IRQ event object which can be triggered from a separate process would make more sense.
Quoted 6 lines
Sure. As some of the L1's MSR are not used by svsm, we "passthrough" such MSR to the VM to simplify the MSR emulation. When VM accesses such MSR, the vmexit is handled by reading from/writing to the corresponding L1's MSR.
Here is the list of "passthrough" MSR we made for TDP guest:
https://github.com/intel-staging/td-partitioning-svsm/blob/svsm-tdp/kernel/src/tdx/vmsr.rs#L595
Okay, whether these are switched on L1/L2 transitions should be transparent to user-space. User-space can change them via the SET_STATE call and the kernel handles the details.
Quoted 9 lines
Quoted 9 lines
Other options would be:
* Let the COCONUT kernel forward MMIO and IOIO requests for which it has no handler to the host and use the result for emulation
* Re-inject these events as #VC/#VE execeptions into the guest OS and let it deal with it.
Quoted 3 lines
Quoted 1 line
The right approach probably also depends on wether the OS runs in englightened or paravisor mode.
For the either enlightened or paravisor mode, is the 1st option workable for both? If so, looks like the 1st option is better as it is more efficient.
Yes, the first option works for both and is also the most efficient.
Regards,
Joerg
Hi Joerg,
Quoted 20 lines
-----Original Message-----
From: Jörg Rödel <[email protected]>
Sent: Tuesday, June 25, 2024 9:24 PM
To: Dong, Chuanxiao <[email protected]>
Cc: Rodel, Jorg <[email protected]>; [email protected]
Subject: Re: [svsm-devel] X86-64 Syscall API/ABI DocumentHi Chuanxiao,
Quoted 6 lines · Chuanxiao
I see. So still taking vcpu as an example, user space process first use
THREAD_CREATEto create a vcpu thread but this vcpu thread runs on the current CPU. The vcpu thread usesVCPU_CREATEsyscall to get avcpu_objand usesWAIT_FOR_EVENT(vcpu_obj)syscall to schedule it to the CPU where thevcpu_objis bound to, and then performs the VM enter on the right CPU.Right, this is how thread-cpu affinity is designed to work.
Quoted 6 lines
Quoted 4 lines
When the BATCH syscall returns, if the
user_datain the SysCallIn is the same with theuser_datain the corresponding SysCallOut, means kernel has forwarded theuser_datato SysCallOut which represents this sub syscall is completed. Is this understanding correct?Yes, but note that no syscall can be considered complete until the actual BATCH syscall returns.
Sure, definitely.
Quoted 20 lines
Quoted 4 lines
Makes sense. So the
SET_MEM_STATEcan fail to change page shared/private state due to the page is MMAP, until such page is MUNMAP.Correct, the COCONUT kernel needs to track which PHYSMEM pages are mapped and fail PSC on them.
Quoted 5 lines
Yes. As this is related a process to use object handles created by another process, it appears that there is no policy restricting syscall operations on the object handle exclusively to the process that created it, and the object handle is unique globally, is this correct?
An object handle itself is not global, but local to the process acquiring it. It will proably be just an index into an in-kernel array.
What an object handle points to is usually global, but depending on what it points to there can be multiple handles to the same object (e.g.
files) or just one (e.g. VM, VCPUs).
Quoted 6 lines
Passing object handles to new processes will be semantically equivalent to closing the handle in the originating process and open it again in the new process.
There must be direct way via the EXEC syscall for this as the new process can not open some handles directly. E.g. a handle for an mmio range needs a vm handle to be created, but there can only be one handle per vm at any time.
Does this mean, although the vm_obj for a specific vm-index cannot be opened directly via VM_OPEN() again if such vm_obj is already opened, there can be multiple vm_obj copies (no via VM_OPEN) at the same so that processes or threads can use the same vm_obj at the same time?
Probably also needs to consider for the THREAD_CREATE syscall? One possible scenario might be, the main VM process gets the vm_obj via VM_OPEN() and creates multiple vcpu threads via THREAD_CREATE(). Then when a vcpu thread gets the corresponding vcpu_obj via VCPU_CREATE(), the vm_obj will be needed by this vcpu thread. Another possible example might be, if the private/share memory conversion events from the VM is handled by the vcpu thread, then SET_MEM_STATE() might be used by the vcpu thread, which also needs the vm_obj.
Quoted 8 lines
Quoted 4 lines
Does this mean, for
MMIO_EVENT, it can take any valid range as input parameters, no matter it is MMIO or RAM, and forIOIO_EVENT, it can take any valid IO port range as input parameters?I don't see why not, if it is a RAM range there will be no events reported, as the hardware will not generate them. Also, any limitation here would need to trust information from the HV on where the MMIO regions actually are.
When the guest uses "movs" instruction to access a MMIO, two addresses can be decoded (one is MMIO address and the other is RAM address) from the instruction. In this case, if this RAM address is also monitored by the user-space via MMIO_EVENT(), I guess kernel can report a RAM event if it wants, but we can implement to let kernel ignore the monitoring for the RAM address.
Quoted 8 lines
Quoted 4 lines
Actually I also want to understand the mappable area provided by an
evt_obj. As it is mappable area, suppose it can beMMAP(). So wondering how this mappable area be used?The idea is to use it for sharing information related to the event. E.g.
the for a VCPU obj the area will contain information about the last exit reason. For MMIO it will contain information about the actual access which triggered the event.
Got it. So seems like the mappable area of a vcpu or MMIO/IOIO object should have some pre-defined format, so that the user-space can understand the mapping length and how to get the data from it.
Quoted 14 lines
Quoted 5 lines
So MMIO_EVENT/IOIO_EVENT syscalls tell the kernel that, the range specified in the syscalls is emulated by some user-space process. If an MMIO/IOIO address is located in this range, the corresponding user-space process should be waked up, right?
Correct.
Quoted 5 lines
BTW, what we though is when MMIO/IOIO event happens, vcpu thread exits from the guest mode and decodes the instruction to get the MMIO address and data (if it is an MMIO write), then waked up the device model process. I think this matches with what you described here. Please let me know if it is not.
Quoted 4 lines
Yes, that is the envisioned flow. After an exit from the guest the COCONUT kernel runs and decodes the event. In case of an MMIO or IOIO event it will also decode the instruction and look up the event which needs to be triggered. Once triggered, the user-space process running the device model wakes up and handles the event.
Then once the COCONUT kernel triggers event and wakes up (equivalent calling TRIGGER_EDGE(mmio/ioio_obj) but in kernel-space) the user-space device model process which has called WAIT_FOR_EVENT(mmio/ioio_obj), it starts to wait for the MMIO/IOIO event in the kernel-space (equivalent calling WAIT_FOR_EVENT (mmio/ioio_obj) but in kernel-space). After the user-space device model process completes the event handling, it will wake up the waiting COCONUT kernel thread via the TRIGGER_EDGE(mmio/ioio_obj), is this correct?
Quoted 6 lines
Quoted 4 lines
How the device model process to get the detailed request? I guess it is via the mappable area provided by the
evt_obj, but not sure so would like to raise the above questions about the mappable area.Yes, information about the actual event will be stored in the mmap'ed area of the event object.
Got it. Seems like for a MMIO/IOIO read event, after the device mode process completes the event handling, the read value will also be stored in the mapped area of the event object, for the COCONUT kernel to do the state updates.
Quoted 11 lines
Quoted 9 lines
Any consideration for updating CPU state in the device model process? Taking the MMIO reading as an example, the reading data may be written back to the vCPU register, or VM's memory (depending on the instruction). My understanding is that this is part of the instruction decoding. The device model process can emulate the MMIO/IOIO request and provide the emulated data to the process which did the instruction decoding, and that process can write the data back via the instruction decoder.
Correct, the plan is that the COCONUT kernel does the state updates for MMIO and IOIO events.
Cool. That is also my expectation.
Quoted 9 lines
Quoted 2 lines
Does the "main VM management process" represent the vCPU threads?
The main VM management process drives a VM and starts separate threads for each VCPU. All of these will be in one process within the same address space.
Quoted 2 lines
Additional question is how to make the binary aware of the platform? Though some compiling option or some syscall?
Quoted 6 lines
There will be initial support for writing binaries for the COCONUT platform. This will include a basic library around the system calls, platform setup code, memory allocator and so on.
All of this will hopefully flow into making COCONUT a Rust platform target, so that binaries can be written using Rust standard library support. But in the beginning there will only be a special library crate to build binaries against.
Understand. This should be the item "[StdRust] COCONUT-SVSM as Rust Tier3/2 Target" in the development plan.
As "the binary has to be aware of the platform as there will always be platform-specific handling required to some degree", I though this requires the binary being able to distinguish SEV, TDX and other possible architectures, so wondering what might be the preferred way to do so.
Quoted 10 lines
Quoted 9 lines
I suppose when GET/SET_STATE(VMSA_GPA), the kernel mode will return all the CPU states defined in the VMSA page by coping data from one page to another, is this correct? For TDX, this is more expensive because reading one VMCS field will require sending one tdcall to TDX module. For example, if user mode only wants to know CR0, it is better for TDX to just send one tdcall to read the CR0 field instead of sending many tdcalls to read all the CPU state from VMCS field to construct the VMSA page.
Quoted 3 lines
Okay, it might make sense to have a more fine-grained state access. That should not be a problem, on the AMD platform these calls can be batched and still update as many state as desired with one user/kernel transition.
Yes, sure. Let's have a more fine-grained state access.
Quoted 7 lines
Quoted 3 lines
Indeed, we can think more about the way for the device to process to inject an IRQ.
SET_STATEmight be an option.
SET_STATEis bound to the VCPU object, which is not accessible in the device emulation process. From the model that evolves here an IRQ event object which can be triggered from a separate process would make more sense.
I see. Ultimately, the IRQ event will be synced to virtual LAPIC in the kernel, but I am currently not clear about how this is achieved from the device model process. If you have more detailed ideas which is ready to share, I would be interested in learning about it.
Quoted 11 lines
Quoted 8 lines
Sure. As some of the L1's MSR are not used by svsm, we "passthrough" such MSR to the VM to simplify the MSR emulation. When VM accesses such MSR, the vmexit is handled by reading from/writing to the corresponding L1's MSR.
Here is the list of "passthrough" MSR we made for TDP guest:
https://github.com/intel-staging/td-partitioning-svsm/blob/svsm-tdp/ke
rnel/src/tdx/vmsr.rs#L595Okay, whether these are switched on L1/L2 transitions should be transparent to user-space. User-space can change them via the
SET_STATEcall and the kernel handles the details.
Got it. BTW, do you think if should have a way to tell whether a guest's MSR is emulated by the COCONUT kernel or by the user-space process?
As the virtual LAPIC is implemented in the kernel, I believe the x2apic MSRs will be emulated directly in the COCONUT kernel without returning back to the user-space. If we want the some MSRs to be emulated by the user-space, probably needs to tell kernel about this so that, the COCONUT kernel can return back to the user space in case of the vmexit occurs due to these user-space monitored MSR?
Thanks
Chuanxiao
Quoted 21 lines
Quoted 11 lines
Quoted 11 lines
Other options would be:
* Let the COCONUT kernel forward MMIO and IOIO requests for which it has no handler to the host and use the result for emulation
* Re-inject these events as #VC/#VE execeptions into the guest OS and let it deal with it.The right approach probably also depends on wether the OS runs in englightened or paravisor
mode.
Quoted 2 lines
For the either enlightened or paravisor mode, is the 1st option workable for both? If so, looks like the
1st option is better as it is more efficient.
Yes, the first option works for both and is also the most efficient.
Regards,
Joerg