KVM: optee: Introduce OP-TEE Mediator for exposing secure world to KVM guests
cover KVM: optee: Introduce OP-TEE Mediator for exposing secure world to KVM guests
A KVM guest running on an arm64 machine will not be able to interact with a trusted execution environment (which supports non-secure guests) like OP-TEE in the secure world. This is because, instructions provided by the architecture (such as, SMC) which switch control to the firmware, are trapped in EL2 when the guest is executes them.
This series adds a feature into the kernel called the TEE mediator abstraction layer, which lets a guest interact with the secure world. Additionally, a OP-TEE specific mediator is also implemented, which hooks itself to the TEE mediator layer and intercepts guest SMCs targetted at OP-TEE.
Overview
=========
Essentially, if the kernel wants to interact with OP-TEE, it makes an "smc - secure monitor call instruction", after loading in arguments into CPU registers. What these arguments consists of and how both these entities communicate can vary. If a guest wants to establish a connection with the secure world, its not possible. This is because of the fact that "smc" by the guest are trapped by the hypervisor in EL2. This is done by setting the HCR_EL2.TSC bit before entering the guest.
Hence, this feature which I we may call TEE mediator, acts as an intermediary between the guest and OP-TEE. Instead of denying the guest SMC and jumping back into the guest, the mediator forwards the request to OP-TEE.
OP-TEE supports virtualization in the normal world and expects 6 things from the NS-hypervisor:
1. Notify OP-TEE when a VM is created.
2. Notify OP-TEE when a VM is destroyed.
3. Any SMC to OP-TEE has to contain the VMID in x7. If its the hypervisor sending, then VMID is 0.
4. Hypervisor has to perform IPA->PA translations of the memory addresses sent by guest.
5. Memory shared by the VM to OP-TEE has to remain pinned.
6. The hypervisor has to follow the OP-TEE protocol, so the guest thinks it is directly speaking to OP-TEE.
Its important to note that, if OP-TEE is built with NS-virtualization support, it can only function if there is a hypervisor with a mediator in normal world.
This implementation has been heavily inspired by Xen's OP-TEE mediator.
Design
======
The unique design of KVM makes it quite challenging to implement such a mediator. OP-TEE is not aware of the host-guest paradigm. Hence, the mediator treats the host as a VM with VMID 1. The guests are assigned VMIDs starting from 2 (note, these are not the VMIDs tagged in TLB, rather we implement our own simple indexing mechanism).
When the host's OP-TEE driver is initialised or released, OP-TEE is notified about VM 1 being created/destroyed.
When a VMM (such as, QEMU) created a guest through KVM ioctls, a call to the TEE mediator layer is made, which in-turn calls OP-TEE mediator which eventually assigns a VM context, VMID, etc. and notifies OP-TEE about guest creation. The opposite happens on guest destruction.
When the guest makes an SMC targetting OP-TEE, it is trapped by the hypervisor and the register state (kvm_vcpu) is sent to the OP-TEE mediator through the TEE layer. Here there are two possibilities.
The guest may make an SMC with arguments which are simple numeric values, exchanging UUID, version information, etc. In this case, the mediator has much less work. It has to attach VMID into X7 and pass the register state to OP-TEE.
But, when guest passes memory addresses as arguments, the mediator has to translate these into physical addresses from intermediate physical addresses (IPA). According to the OP-TEE protocol (as documented in optee_smc.h and optee_msg.h), the guest OP-TEE driver would share a buffer filled with pointers, which the mediator translates.
The OP-TEE mediator also keeps track of active calls between each guest and OP-TEE, and pins pages which are already shared. This is to avoid swapping of shared pages by the host under memory pressure. These pages are unpinned as soon as guest's transaction completes with OP-TEE.
Testing
=======
The feature has been tested on QEMU virt platform using "xtest" as the test suite. As of now, all of 35000+ tests pass. The mediator has also been stressed under memory pressure and all tests pass too. Any suggestions on further testing the feature are welcome.
Call for review
===============
Any insights/suggestions regarding the implementation are appreciated.
Yuvraj Sakshith (7):
firmware: smccc: Add macros for Trusted OS/App owner check on SMC
value
tee: Add TEE Mediator module which aims to expose TEE to a KVM guest.
KVM: Notify TEE Mediator when KVM creates and destroys guests
KVM: arm64: Forward guest CPU state to TEE mediator on SMC trap
tee: optee: Add OPTEE_SMC_VM_CREATED and OPTEE_SMC_VM_DESTROYED
tee: optee: Add OP-TEE Mediator
tee: optee: Notify TEE Mediator on OP-TEE driver initialization and
release
arch/arm64/kvm/hypercalls.c | 15 +- drivers/tee/Kconfig | 5 + drivers/tee/Makefile | 1 + drivers/tee/optee/Kconfig | 7 + drivers/tee/optee/Makefile | 1 + drivers/tee/optee/core.c | 13 +- drivers/tee/optee/optee_mediator.c | 1319 ++++++++++++++++++++++++++++ drivers/tee/optee/optee_mediator.h | 103 +++ drivers/tee/optee/optee_smc.h | 53 ++ drivers/tee/optee/smc_abi.c | 6 + drivers/tee/tee_mediator.c | 145 +++ include/linux/arm-smccc.h | 8 + include/linux/tee_mediator.h | 39 + virt/kvm/kvm_main.c | 11 +- 14 files changed, 1721 insertions(+), 5 deletions(-) create mode 100644 drivers/tee/optee/optee_mediator.c create mode 100644 drivers/tee/optee/optee_mediator.h create mode 100644 drivers/tee/tee_mediator.c create mode 100644 include/linux/tee_mediator.h
Signature · 1 line
2.43.0
RFC PATCH 1/7 firmware: smccc: Add macros for Trusted OS/App owner check on SMC value
This patch adds ARM_SMCCC_IS_OWNER_TRUSTED_APP() and ARM_SMCCC_IS_OWNER_TRUSTED_OS() macros. These can be used to identify if the SMC is targetted at a Trusted OS/App in the secure world.
--- include/linux/arm-smccc.h | 8 ++++++++ 1 file changed, 8 insertions(+)
include/linux/arm-smccc.h +8 -0
@@ -56,6 +56,14 @@#define ARM_SMCCC_OWNER_TRUSTED_OS 50#define ARM_SMCCC_OWNER_TRUSTED_OS_END 63 +#define ARM_SMCCC_IS_OWNER_TRUSTED_APP (smc_val) \ + ((ARM_SMCCC_OWNER_NUM(smc_val) >= ARM_SMCCC_OWNER_TRUSTED_APP) && \ + (ARM_SMCCC_OWNER_NUM(smc_val) <= ARM_SMCCC_OWNER_TRUSTED_APP_END)) + +#define ARM_SMCCC_IS_OWNER_TRUSTED_OS (smc_val) \ + ((ARM_SMCCC_OWNER_NUM(smc_val) >= ARM_SMCCC_OWNER_TRUSTED_OS) && \ + (ARM_SMCCC_OWNER_NUM(smc_val) <= ARM_SMCCC_OWNER_TRUSTED_OS_END)) +#define ARM_SMCCC_FUNC_QUERY_CALL_UID 0xff01#define ARM_SMCCC_QUIRK_NONE 0
Signature · 1 line
2.43.0
RFC PATCH 2/7 tee: Add TEE Mediator module which aims to expose TEE to a KVM guest.
In reply to Yuvraj SakshithThe TEE Mediator module is an upper abstraction layer which lets KVM guests interact with a trusted execution environment.
TEE specific subsystems (such as OP-TEE, for example) can register a set of handlers through tee_mediator_register_ops() with the TEE Mediator, which will be called by the kernel when required.
Given this module, architecture specific TEE drivers can implement handler functions to work with these events if necessary. In most implementations, a special instruction (such as SMC, in arm64) switches control leading to the TEE. These instructions are usually trapped by the hypervisor when executed by the guest.
This module allows making use of these trapped instructions and mediating the request between guest and TEE.
--- drivers/tee/Kconfig | 5 ++ drivers/tee/Makefile | 1 + drivers/tee/tee_mediator.c | 145 +++++++++++++++++++++++++++++++++++ include/linux/tee_mediator.h | 39 ++++++++++ 4 files changed, 190 insertions(+) create mode 100644 drivers/tee/tee_mediator.c create mode 100644 include/linux/tee_mediator.h
drivers/tee/Kconfig +5 -0
@@ -11,6 +11,11 @@ menuconfig TEE This implements a generic interface towards a Trusted Execution Environment (TEE). +config TEE_MEDIATOR + bool "Trusted Execution Environment Mediator support" + depends on KVM + help + Provides an abstraction layer for TEE drivers to mediate KVM guest requests to the TEE. if TEE source "drivers/tee/optee/Kconfig"
drivers/tee/Makefile +1 -0
@@ -1,5 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_TEE) += tee.o +obj-$(CONFIG_TEE_MEDIATOR) += tee_mediator.o tee-objs += tee_core.o tee-objs += tee_shm.o tee-objs += tee_shm_pool.o
drivers/tee/tee_mediator.c new file +145 -0
145 lines not on this page · raw message
include/linux/tee_mediator.h new file +39 -0
@@ -0,0 +1,39 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* TEE Mediator for the Linux Kernel +* +* This module enables a KVM guest to interact with a +* Trusted Execution Environment in the secure processing +* state provided by the architecture. +* +* Author: +* Yuvraj Sakshith <[email protected]> +*/ + +#ifndef __TEE_MEDIATOR_H +#define __TEE_MEDIATOR_H + +#include <linux/kvm_host.h> + +struct tee_mediator_ops { +int (*create_host)(void ); +int (*destroy_host)(void ); +int (*create_vm)(struct kvm *kvm); +int (*destroy_vm)(struct kvm *kvm); +void (*forward_request)(struct kvm_vcpu *vcpu); +int (*is_active)(void ); +}; + +struct tee_mediator { +struct tee_mediator_ops *ops; +}; + +int tee_mediator_create_host (void ); +int tee_mediator_destroy_host (void ); +int tee_mediator_create_vm (struct kvm *kvm); +int tee_mediator_destroy_vm (struct kvm *kvm); +void tee_mediator_forward_request (struct kvm_vcpu *vcpu); +int tee_mediator_is_active (void ); +int tee_mediator_register_ops (struct tee_mediator_ops *ops); + +#endif
Signature · 1 line
2.43.0
RFC PATCH 3/7 KVM: Notify TEE Mediator when KVM creates and destroys guests
In reply to Yuvraj SakshithTEEs supporting virtualization in the rich execution environment would want to know about guest creation and destruction by the hypervisor.
This change notifies the TEE mediator of these events (if its active).
--- virt/kvm/kvm_main.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
virt/kvm/kvm_main.c +9 -2
@@ -49,6 +49,7 @@#include <linux/lockdep.h> #include <linux/kthread.h> #include <linux/suspend.h> +#include <linux/tee_mediator.h> #include <asm/processor.h> #include <asm/ioctl.h>
@@ -1250,7 +1251,10 @@ static void kvm_destroy_vm(struct kvm *kvm) {int i;struct mm_struct *mm = kvm->mm; - +#ifdef CONFIG_TEE_MEDIATOR +if (tee_mediator_is_active ()) + (void )tee_mediator_destroy_vm (kvm); +#endif kvm_destroy_pm_notifier (kvm);kvm_uevent_notify_change (KVM_EVENT_DESTROY_VM , kvm);kvm_destroy_vm_debugfs (kvm);
@@ -5407,7 +5411,10 @@ static int kvm_dev_ioctl_create_vm(unsigned long type)* care of doing kvm_put_kvm(kvm). */ kvm_uevent_notify_change (KVM_EVENT_CREATE_VM , kvm); - +#ifdef CONFIG_TEE_MEDIATOR +if (tee_mediator_is_active ()) + (void )tee_mediator_create_vm (kvm); +#endif fd_install (fd, file);return fd;
Signature · 1 line
2.43.0
RFC PATCH 4/7 KVM: arm64: Forward guest CPU state to TEE mediator on SMC trap
In reply to Yuvraj SakshithWhen guest makes an SMC, the call is denied by the hypervisor and not handled (ignored). In the presence of the TEE Mediator module, the SMC from guest is forwarded with it's vCPU register state through tee_mediator_forward_request().
--- arch/arm64/kvm/hypercalls.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-)
arch/arm64/kvm/hypercalls.c +13 -2
@@ -3,6 +3,7 @@#include <linux/arm-smccc.h> #include <linux/kvm_host.h> +#include <linux/tee_mediator.h> #include <asm/kvm_emulate.h>
@@ -90,7 +91,10 @@ static bool kvm_smccc_default_allowed(u32 func_id)*/ if (func_id >=KVM_PSCI_FN (0 ) && func_id <=KVM_PSCI_FN (3 ))return true; - +#ifdef CONFIG_TEE_MEDIATOR +if (ARM_SMCCC_IS_OWNER_TRUSTED_APP (func_id) ||ARM_SMCCC_IS_OWNER_TRUSTED_OS (func_id)) +return true; +#endif return false; } }
@@ -284,7 +288,14 @@ int kvm_smccc_call_handler(struct kvm_vcpu *vcpu)WARN_RATELIMIT (1 ,"Unhandled SMCCC filter action: %d\n" , action); goto out; } - +#ifdef CONFIG_TEE_MEDIATOR +if (ARM_SMCCC_IS_OWNER_TRUSTED_APP (func_id) ||ARM_SMCCC_IS_OWNER_TRUSTED_OS (func_id)) { +if (tee_mediator_is_active ()) { +tee_mediator_forward_request (vcpu); +return 1 ; + } + } +#endif switch (func_id) {case ARM_SMCCC_VERSION_FUNC_ID : val[0 ] =ARM_SMCCC_VERSION_1_1 ;
Signature · 1 line
2.43.0
RFC PATCH 5/7 tee: optee: Add OPTEE_SMC_VM_CREATED and OPTEE_SMC_VM_DESTROYED
In reply to Yuvraj SakshithOP-TEE when compiled with NS-Virtualization support expects NS-Hypervisor to notify events such as guest creation and destruction through SMCs.
This change adds two macros OPTEE_SMC_VM_CREATED and OPTEE_SMC_VM_DESTROYED.
--- drivers/tee/optee/optee_smc.h | 53 +++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+)
drivers/tee/optee/optee_smc.h +53 -0
@@ -452,6 +452,59 @@ struct optee_smc_disable_shm_cache_result {/* See OPTEE_SMC_CALL_WITH_REGD_ARG above */ #define OPTEE_SMC_FUNCID_CALL_WITH_REGD_ARG 19 +/* +* Inform OP-TEE about a new virtual machine +* +* Hypervisor issues this call during virtual machine (guest) creation. +* OP-TEE records client id of new virtual machine and prepares +* to receive requests from it. This call is available only if OP-TEE +* was built with virtualization support. +* +* Call requests usage: +* a0 SMC Function ID, OPTEE_SMC_VM_CREATED +* a1 Hypervisor Client ID of newly created virtual machine +* a2-6 Not used +* a7 Hypervisor Client ID register. Must be 0, because only hypervisor +* can issue this call +* +* Normal return register usage: +* a0 OPTEE_SMC_RETURN_OK +* a1-7 Preserved +* +* Error return: +* a0 OPTEE_SMC_RETURN_ENOTAVAIL OP-TEE have no resources for +* another VM +* a1-7 Preserved +* +*/ +#define OPTEE_SMC_FUNCID_VM_CREATED 13 +#define OPTEE_SMC_VM_CREATED \ + OPTEE_SMC_FAST_CALL_VAL(OPTEE_SMC_FUNCID_VM_CREATED) + +/* +* Inform OP-TEE about shutdown of a virtual machine +* +* Hypervisor issues this call during virtual machine (guest) destruction. +* OP-TEE will clean up all resources associated with this VM. This call is +* available only if OP-TEE was built with virtualization support. +* +* Call requests usage: +* a0 SMC Function ID, OPTEE_SMC_VM_DESTROYED +* a1 Hypervisor Client ID of virtual machine being shut down +* a2-6 Not used +* a7 Hypervisor Client ID register. Must be 0, because only hypervisor +* can issue this call +* +* Normal return register usage: +* a0 OPTEE_SMC_RETURN_OK +* a1-7 Preserved +* +*/ + +#define OPTEE_SMC_FUNCID_VM_DESTROYED 14 +#define OPTEE_SMC_VM_DESTROYED \ + OPTEE_SMC_FAST_CALL_VAL(OPTEE_SMC_FUNCID_VM_DESTROYED) +/* * Resume from RPC (for example after processing a foreign interrupt) *
Signature · 1 line
2.43.0
RFC PATCH 6/7 tee: optee: Add OP-TEE Mediator
In reply to Yuvraj SakshithThe OP-TEE mediator is a software entity resposible for bridging the gap between a KVM guest and OP-TEE in the secure world.
The guest's OP-TEE driver issues an SMC instruction after populating its CPU registers with appropriate arguments. This SMC is trapped by the hypervisor and control is passed to the OP-TEE mediator with the vCPU state.
The mediator is resposible for manipulating the vCPU state accordingly and keeping track of active transactions between the guest and the TEE.
This implementation adds event handlers that gets hooked to the TEE Mediator layer and are called when events such as guest creation/destruction, mediator status check and guest SMC trap happen.
Important routines implemented:
- optee_mediator_{create|destroy}_vm(): Sends an SMC to OP-TEE notifying KVM guest creation or destruction.
- optee_mediator_{create|destroy}_host(): Sends an SMC to OP-TEE notifying host OP-TEE driver initalization/release (OP-TEE treats all NS-EL1 entities as a guest, and is not aware of host privilege).
- optee_mediator_forward_smc(): Changes vCPU register state as required by OP-TEE and keeps track of standard calls and memory shared by guest.
The OP-TEE mediator is implemented in such a way that, the guest/VMM can remain unmodified. The guest interacts with OP-TEE just as it would if it was running natively.
--- drivers/tee/optee/Kconfig | 7 + drivers/tee/optee/Makefile | 1 + drivers/tee/optee/optee_mediator.c | 1319 ++++++++++++++++++++++++++++ drivers/tee/optee/optee_mediator.h | 103 +++ 4 files changed, 1430 insertions(+) create mode 100644 drivers/tee/optee/optee_mediator.c create mode 100644 drivers/tee/optee/optee_mediator.h
drivers/tee/optee/Kconfig +7 -0
@@ -25,3 +25,10 @@ config OPTEE_INSECURE_LOAD_IMAGE Additional documentation on kernel security risks are at Documentation/tee/op-tee.rst. + +config OPTEE_MEDIATOR + bool "OP-TEE Mediator support" + depends on TEE_MEDIATOR && OPTEE && ARM64 && KVM + help + This enables a KVM guest equipped with an OP-TEE driver to interact with OP-TEE + in the secure world.
drivers/tee/optee/Makefile +1 -0
@@ -1,4 +1,5 @@ # SPDX-License-Identifier: GPL-2.0 +obj-$(CONFIG_OPTEE_MEDIATOR) += optee_mediator.o obj-$(CONFIG_OPTEE) += optee.o optee-objs += core.o optee-objs += call.o
drivers/tee/optee/optee_mediator.c new file +1319 -0
1,319 lines not on this page · raw message
drivers/tee/optee/optee_mediator.h new file +103 -0
103 lines not on this page · raw message
Signature · 1 line
2.43.0
RFC PATCH 7/7 tee: optee: Notify TEE Mediator on OP-TEE driver initialization and release
In reply to Yuvraj SakshithWhen host initializes or releases its OP-TEE driver through optee_core_init()/optee_core_exit(), notify OP-TEE in the secure world about this change.
If OP-TEE is built with NS-Virtualization support, it will treat SMCs coming from the host as if it were coming from a VM (as OP-TEE does not understand the KVM paradigm).
Hence, OPTEE_SMC_VM_CREATED and OPTEE_SMC_VM_DESTROYED SMCs have to be made for its internal book-keeping.
--- drivers/tee/optee/core.c | 13 ++++++++++++- drivers/tee/optee/smc_abi.c | 6 ++++++ 2 files changed, 18 insertions(+), 1 deletion(-)
drivers/tee/optee/core.c +12 -1
@@ -14,6 +14,7 @@#include <linux/slab.h> #include <linux/string.h> #include <linux/tee_core.h> +#include <linux/tee_mediator.h> #include <linux/types.h> #include "optee_private.h"
@@ -195,7 +196,13 @@ static bool intf_is_regged;static int __init optee_core_init (void ) {int rc; - +#ifdef CONFIG_TEE_MEDIATOR +if (tee_mediator_is_active ()) { + rc =tee_mediator_create_host (); +if (rc <0 ) +return rc; + } +#endif /* * The kernel may have crashed@the same time that all available * secure world threads were suspended and we cannot reschedule the
@@ -240,6 +247,10 @@ static void __exit optee_core_exit(void)optee_smc_abi_unregister ();if (!ffa_abi_rc)optee_ffa_abi_unregister (); +#ifdef CONFIG_TEE_MEDIATOR +if (tee_mediator_is_active ()) +tee_mediator_destroy_host (); +#endif }module_exit (optee_core_exit);
drivers/tee/optee/smc_abi.c +6 -0
@@ -25,8 +25,10 @@#include <linux/slab.h> #include <linux/string.h> #include <linux/tee_core.h> +#include <linux/tee_mediator.h> #include <linux/types.h> #include <linux/workqueue.h> +#include "optee_mediator.h" #include "optee_private.h" #include "optee_smc.h" #include "optee_rpc_cmd.h"
@@ -1396,6 +1398,10 @@ static void optee_smccc_smc(unsigned long a0, unsigned long a1, unsigned long a6, unsigned long a7, struct arm_smccc_res *res) { +#ifdef CONFIG_TEE_MEDIATOR + if (tee_mediator_is_active()) + a7 =OPTEE_HOST_VMID ; +#endif arm_smccc_smc (a0, a1, a2, a3, a4, a5, a6, a7, res); }
Signature · 1 line
2.43.0
Marc Zyngier
maintainerQuoted 32 lines · Yuvraj Sakshith
A KVM guest running on an arm64 machine will not be able to interact with a trusted execution environment (which supports non-secure guests) like OP-TEE in the secure world. This is because, instructions provided by the architecture (such as, SMC) which switch control to the firmware, are trapped in EL2 when the guest is executes them.
This series adds a feature into the kernel called the TEE mediator abstraction layer, which lets a guest interact with the secure world. Additionally, a OP-TEE specific mediator is also implemented, which hooks itself to the TEE mediator layer and intercepts guest SMCs targetted at OP-TEE.
Overview
=========Essentially, if the kernel wants to interact with OP-TEE, it makes an "smc - secure monitor call instruction", after loading in arguments into CPU registers. What these arguments consists of and how both these entities communicate can vary. If a guest wants to establish a connection with the secure world, its not possible. This is because of the fact that "smc" by the guest are trapped by the hypervisor in EL2. This is done by setting the HCR_EL2.TSC bit before entering the guest.
Hence, this feature which I we may call TEE mediator, acts as an intermediary between the guest and OP-TEE. Instead of denying the guest SMC and jumping back into the guest, the mediator forwards the request to OP-TEE.
OP-TEE supports virtualization in the normal world and expects 6 things from the NS-hypervisor:
1. Notify OP-TEE when a VM is created.
2. Notify OP-TEE when a VM is destroyed.
3. Any SMC to OP-TEE has to contain the VMID in x7. If its the hypervisor sending, then VMID is 0.
4. Hypervisor has to performIPA->PAtranslations of the memory addresses sent by guest.
5. Memory shared by the VM to OP-TEE has to remain pinned.
6. The hypervisor has to follow the OP-TEE protocol, so the guest thinks it is directly speaking to OP-TEE.
Quoted 5 lines
Its important to note that, if OP-TEE is built with NS-virtualization support, it can only function if there is a hypervisor with a mediator in normal world.
This implementation has been heavily inspired by Xen's OP-TEE mediator.
[...]
And I think this inspiration is the source of most of the problems in this series.
Routing Secure Calls from the guest to whatever is on the secure side should not be the kernel's job at all. It should be the VMM's job. All you need to do is to route the SMCs from the guest to userspace, and we already have all the required infrastructure for that.
It is the VMM that should:
- signal the TEE of VM creation/teardown
- translate between IPAs and host VAs without involving KVM
- let the host TEE driver translate between VAs and PAs and deal with the pinning as required, just like it would do for any userspace (without ever using the KVM memslot interface)
- proxy requests from the guest to the TEE
- in general, bear the complexity of anything related to the TEE
In short, the VMM is just another piece of userspace using the TEE to do whatever it wants. The TEE driver on the host must obviously know about VMs, but that's about it.
Crucially, KVM should:
- be completely TEE agnostic and never call into something that is TEE-specific
- allow a TEE implementation entirely in userspace, specially for the machines that do not have EL3
As it stands, your design looks completely upside-down. Most of this code should be userspace code and live in (or close to) the VMM, with the host kernel only providing the basic primitives, most of which should already be there.
Thanks,
M.
Signature · 1 line
Jazz isn't dead. It just smells funny.
Quoted 42 lines · Marc Zyngier
Quoted 37 lines · Yuvraj Sakshith
A KVM guest running on an arm64 machine will not be able to interact with a trusted execution environment (which supports non-secure guests) like OP-TEE in the secure world. This is because, instructions provided by the architecture (such as, SMC) which switch control to the firmware, are trapped in EL2 when the guest is executes them.
This series adds a feature into the kernel called the TEE mediator abstraction layer, which lets a guest interact with the secure world. Additionally, a OP-TEE specific mediator is also implemented, which hooks itself to the TEE mediator layer and intercepts guest SMCs targetted at OP-TEE.
Overview
=========Essentially, if the kernel wants to interact with OP-TEE, it makes an "smc - secure monitor call instruction", after loading in arguments into CPU registers. What these arguments consists of and how both these entities communicate can vary. If a guest wants to establish a connection with the secure world, its not possible. This is because of the fact that "smc" by the guest are trapped by the hypervisor in EL2. This is done by setting the HCR_EL2.TSC bit before entering the guest.
Hence, this feature which I we may call TEE mediator, acts as an intermediary between the guest and OP-TEE. Instead of denying the guest SMC and jumping back into the guest, the mediator forwards the request to OP-TEE.
OP-TEE supports virtualization in the normal world and expects 6 things from the NS-hypervisor:
1. Notify OP-TEE when a VM is created.
2. Notify OP-TEE when a VM is destroyed.
3. Any SMC to OP-TEE has to contain the VMID in x7. If its the hypervisor sending, then VMID is 0.
4. Hypervisor has to performIPA->PAtranslations of the memory addresses sent by guest.
5. Memory shared by the VM to OP-TEE has to remain pinned.
6. The hypervisor has to follow the OP-TEE protocol, so the guest thinks it is directly speaking to OP-TEE.Its important to note that, if OP-TEE is built with NS-virtualization support, it can only function if there is a hypervisor with a mediator in normal world.
This implementation has been heavily inspired by Xen's OP-TEE mediator.
[...]
Quoted 8 lines
And I think this inspiration is the source of most of the problems in this series.
Routing Secure Calls from the guest to whatever is on the secure side should not be the kernel's job at all. It should be the VMM's job. All you need to do is to route the SMCs from the guest to userspace, and we already have all the required infrastructure for that.
Yes, this was an argument at the time of designing this solution.
Quoted 6 lines
It is the VMM that should:
- signal the TEE of VM creation/teardown
- translate between IPAs and host VAs without involving KVM
Quoted 8 lines
- let the host TEE driver translate between VAs and PAs and deal with the pinning as required, just like it would do for any userspace (without ever using the KVM memslot interface)
- proxy requests from the guest to the TEE
- in general, bear the complexity of anything related to the TEE
Major reason why I went with placing the implementation inside the kernel is,
- OP-TEE userspace lib (client) does not support sending SMCs for VM events and needs modification.
- QEMU (or every other VMM) will have to be modified.
- OP-TEE driver is anyways in the kernel. A mediator will just be an addition and not a completely new entity.
- (Potential) issues if we would want to mediate requests from VM which has private mem.
- Heavy VM exits if guest makes frequent TOS calls.
Hence, the thought of making changes to too many entities (libteec, VMM, etc.) was a strong reason, although arguable.
Quoted 4 lines
In short, the VMM is just another piece of userspace using the TEE to do whatever it wants. The TEE driver on the host must obviously know about VMs, but that's about it.
Quoted 8 lines
Crucially, KVM should:
- be completely TEE agnostic and never call into something that is TEE-specific
- allow a TEE implementation entirely in userspace, specially for the machines that do not have EL3
Yes, you're right. Although I believe there still are some changes that need to be made to KVM for facilitating this. For example, kvm_smccc_get_action() would deny TOS call.
So, having an implementation completely in VMM without any change in KVM might be challenging, any potential solutions are welcome.
Quoted 11 lines
As it stands, your design looks completely upside-down. Most of this code should be userspace code and live in (or close to) the VMM, with the host kernel only providing the basic primitives, most of which should already be there.
Thanks,
M.Signature · 1 line
Jazz isn't dead. It just smells funny.
Marc Zyngier
maintainerQuoted 5 lines · Yuvraj Sakshith
Quoted 3 lines · Marc Zyngier
Quoted 1 line · Yuvraj Sakshith
[...]
Quoted 27 lines
Quoted 5 lines
Quoted 2 lines
This implementation has been heavily inspired by Xen's OP-TEE mediator.
[...]
Quoted 8 lines
And I think this inspiration is the source of most of the problems in this series.
Routing Secure Calls from the guest to whatever is on the secure side should not be the kernel's job at all. It should be the VMM's job. All you need to do is to route the SMCs from the guest to userspace, and we already have all the required infrastructure for that.
Yes, this was an argument at the time of designing this solution.
Quoted 12 lines
It is the VMM that should:
- signal the TEE of VM creation/teardown
- translate between IPAs and host VAs without involving KVM
- let the host TEE driver translate between VAs and PAs and deal with the pinning as required, just like it would do for any userspace (without ever using the KVM memslot interface)
- proxy requests from the guest to the TEE
Quoted 7 lines
Quoted 2 lines
- in general, bear the complexity of anything related to the TEE
Major reason why I went with placing the implementation inside the kernel is,
- OP-TEE userspace lib (client) does not support sending SMCs for VM events and needs modification.
- QEMU (or every other VMM) will have to be modified.
Sure. And what? New feature, new API, new code. And what will happen once someone wants to use something other than OP-TEE? Or one of the many forks of OP-TEE that have a completely different ABI (cue the Android forks -- yes, plural)?
Quoted 2 lines
- OP-TEE driver is anyways in the kernel. A mediator will just be an addition and not a completely new entity.
Of course not. The TEE can be anywhere I want. On another machine if I decide so. Just because OP-TEE has a very simplistic model doesn't mean we have to be constrained by it.
Quoted 2 lines
- (Potential) issues if we would want to mediate requests from VM which has private mem.
Private memory means that not even the host has access to it, as it is the case with pKVM. How would that be an issue?
Quoted 1 line
- Heavy VM exits if guest makes frequent TOS calls.
Sorry, I have to completely dismiss the argument here. I'm not even remotely considering performance for something that is essentially a full context switch of the whole machine. By definition, calling into EL3, and then S-EL1/S-EL2 is going to be as fast as a dying snail, and an additional exit to userspace will hardly register for anything other than a pointless latency benchmark.
Quoted 3 lines
Hence, the thought of making changes to too many entities (libteec, VMM, etc.) was a strong reason, although arguable.
It is a *terrible* reason. By this reasoning, we would have subsumed the whole VMM into the kernel (just like Xen), because "we don't want to change userspace".
Furthermore, you are not even considering basic things such as permissions. Your approach completely circumvents any form of access control, meaning that if any user that can create a VM can talk to the TEE, even if they don't have access to the TEE driver.
Yes, you could replicate access permission, SE-Linux, seccomp (and the rest of the security theater) at the KVM/TEE boundary, making the whole thing even more of a twisted mess.
Or you could simply do the right thing and let the kernel do its job the way it was intended by using the syscall interface from userspace.
Quoted 10 lines
Quoted 10 lines
In short, the VMM is just another piece of userspace using the TEE to do whatever it wants. The TEE driver on the host must obviously know about VMs, but that's about it.
Crucially, KVM should:
- be completely TEE agnostic and never call into something that is TEE-specific
Quoted 7 lines
Quoted 3 lines
- allow a TEE implementation entirely in userspace, specially for the machines that do not have EL3
Yes, you're right. Although I believe there still are some changes that need to be made to KVM for facilitating this. For example,
kvm_smccc_get_action()would deny TOS call.
If something is missing in KVM to allow routing of SMCs to userspace, I'm more than happy to entertain the change.
Quoted 2 lines
So, having an implementation completely in VMM without any change in KVM might be challenging, any potential solutions are welcome.
I've said what I have to say already, and pointed you in a direction that I see as both correct and maintainable.
Thanks,
M.
Signature · 1 line
Jazz isn't dead. It just smells funny.
Quoted 45 lines · Marc Zyngier
Quoted 5 lines · Yuvraj Sakshith
Quoted 3 lines · Marc Zyngier
Quoted 1 line · Yuvraj Sakshith
[...]
Quoted 27 lines
Quoted 5 lines
Quoted 2 lines
This implementation has been heavily inspired by Xen's OP-TEE mediator.
[...]
Quoted 8 lines
And I think this inspiration is the source of most of the problems in this series.
Routing Secure Calls from the guest to whatever is on the secure side should not be the kernel's job at all. It should be the VMM's job. All you need to do is to route the SMCs from the guest to userspace, and we already have all the required infrastructure for that.
Yes, this was an argument at the time of designing this solution.
Quoted 12 lines
It is the VMM that should:
- signal the TEE of VM creation/teardown
- translate between IPAs and host VAs without involving KVM
- let the host TEE driver translate between VAs and PAs and deal with the pinning as required, just like it would do for any userspace (without ever using the KVM memslot interface)
- proxy requests from the guest to the TEE
Quoted 7 lines
Quoted 2 lines
- in general, bear the complexity of anything related to the TEE
Major reason why I went with placing the implementation inside the kernel is,
- OP-TEE userspace lib (client) does not support sending SMCs for VM events and needs modification.
- QEMU (or every other VMM) will have to be modified.
Quoted 4 lines
Sure. And what? New feature, new API, new code. And what will happen once someone wants to use something other than OP-TEE? Or one of the many forks of OP-TEE that have a completely different ABI (cue the Android forks -- yes, plural)?
If something other than OP-TEE has to be supported, a specific mediator (such as drivers/tee/optee/optee_mediator.c) has to be constructed with handlers hooked via tee_mediator_register_ops().
But yes, the ABI might change and the implementor has the freedom to mediate it as required.
Quoted 7 lines
Quoted 2 lines
- OP-TEE driver is anyways in the kernel. A mediator will just be an addition and not a completely new entity.
Of course not. The TEE can be anywhere I want. On another machine if I decide so. Just because OP-TEE has a very simplistic model doesn't mean we have to be constrained by it.
Quoted 6 lines
Quoted 2 lines
- (Potential) issues if we would want to mediate requests from VM which has private mem.
Private memory means that not even the host has access to it, as it is the case with pKVM. How would that be an issue?
Guest shares memory to OP-TEE through a buffer filled with pointers, which the mediator has to read for IPA->PA translations of all these pointers. VMM wont be able to read these if memory is private.
But, this is a "potential" solution and if at all the mediator is moved to VMM, this is completely ruled out.
Quoted 9 lines
Quoted 1 line
- Heavy VM exits if guest makes frequent TOS calls.
Sorry, I have to completely dismiss the argument here. I'm not even remotely considering performance for something that is essentially a full context switch of the whole machine. By definition, calling into EL3, and then S-EL1/S-EL2 is going to be as fast as a dying snail, and an additional exit to userspace will hardly register for anything other than a pointless latency benchmark.
Okay, makes sense.
Quoted 4 lines
Quoted 3 lines
Hence, the thought of making changes to too many entities (libteec, VMM, etc.) was a strong reason, although arguable.
Quoted 8 lines
It is a *terrible* reason. By this reasoning, we would have subsumed the whole VMM into the kernel (just like Xen), because "we don't want to change userspace".
Furthermore, you are not even considering basic things such as permissions. Your approach completely circumvents any form of access control, meaning that if any user that can create a VM can talk to the TEE, even if they don't have access to the TEE driver.
Well, this is a good point. OP-TEE built for NS-Virt supports handles calls from different VMs under different MMU partitions (will need to go off track to explain this). But, each VM's state and data remains isolated internally in S-EL1.
Quoted 21 lines
Yes, you could replicate access permission, SE-Linux, seccomp (and the rest of the security theater) at the KVM/TEE boundary, making the whole thing even more of a twisted mess.
Or you could simply do the right thing and let the kernel do its job the way it was intended by using the syscall interface from userspace.
Quoted 14 lines
Quoted 13 lines
In short, the VMM is just another piece of userspace using the TEE to do whatever it wants. The TEE driver on the host must obviously know about VMs, but that's about it.
Crucially, KVM should:
- be completely TEE agnostic and never call into something that is TEE-specific
- allow a TEE implementation entirely in userspace, specially for the machines that do not have EL3
Quoted 6 lines
Quoted 3 lines
Yes, you're right. Although I believe there still are some changes that need to be made to KVM for facilitating this. For example,
kvm_smccc_get_action()would deny TOS call.If something is missing in KVM to allow routing of SMCs to userspace, I'm more than happy to entertain the change.
Okay.
Quoted 6 lines
Quoted 2 lines
So, having an implementation completely in VMM without any change in KVM might be challenging, any potential solutions are welcome.
I've said what I have to say already, and pointed you in a direction that I see as both correct and maintainable.
Yes, I get your point on placing mediator in VMM. And now that I think of it, I believe I can make an improvement.
But yes, since too many entities are involved, the design of this solution has been a nightmare. Good to have been pushed this way.
Quoted 6 lines
Thanks,
M.Signature · 1 line
Jazz isn't dead. It just smells funny.
Thanks,
Yuvraj Sakshith