Skip to contentSkip to conversation

KVM: optee: Introduce OP-TEE Mediator for exposing secure world to KVM guests

RFC PATCH In review Yuvraj Sakshith 2 participants 12 messages Apr 1, 2025 – Apr 2, 2025

Subsystems (6)
KVM arm64 KVM arm64 TEE +2 more subsystems
lore
Share & tools
lore
#0 lore raw

cover KVM: optee: Introduce OP-TEE Mediator for exposing secure world to KVM guests

A KVM guest running on an arm64 machine will not be able to interact with a trusted execution environment (which supports non-secure guests) like OP-TEE in the secure world. This is because, instructions provided by the architecture (such as, SMC) which switch control to the firmware, are trapped in EL2 when the guest is executes them.

This series adds a feature into the kernel called the TEE mediator abstraction layer, which lets a guest interact with the secure world. Additionally, a OP-TEE specific mediator is also implemented, which hooks itself to the TEE mediator layer and intercepts guest SMCs targetted at OP-TEE.

Overview
=========

Essentially, if the kernel wants to interact with OP-TEE, it makes an "smc - secure monitor call instruction", after loading in arguments into CPU registers. What these arguments consists of and how both these entities communicate can vary. If a guest wants to establish a connection with the secure world, its not possible. This is because of the fact that "smc" by the guest are trapped by the hypervisor in EL2. This is done by setting the HCR_EL2.TSC bit before entering the guest.

Hence, this feature which I we may call TEE mediator, acts as an intermediary between the guest and OP-TEE. Instead of denying the guest SMC and jumping back into the guest, the mediator forwards the request to OP-TEE.

OP-TEE supports virtualization in the normal world and expects 6 things from the NS-hypervisor:

1. Notify OP-TEE when a VM is created.
2. Notify OP-TEE when a VM is destroyed.
3. Any SMC to OP-TEE has to contain the VMID in x7. If its the hypervisor sending, then VMID is 0.
4. Hypervisor has to perform IPA->PA translations of the memory addresses sent by guest.
5. Memory shared by the VM to OP-TEE has to remain pinned.
6. The hypervisor has to follow the OP-TEE protocol, so the guest thinks it is directly speaking to OP-TEE.

Its important to note that, if OP-TEE is built with NS-virtualization support, it can only function if there is a hypervisor with a mediator in normal world.

This implementation has been heavily inspired by Xen's OP-TEE mediator.

Design
======

The unique design of KVM makes it quite challenging to implement such a mediator. OP-TEE is not aware of the host-guest paradigm. Hence, the mediator treats the host as a VM with VMID 1. The guests are assigned VMIDs starting from 2 (note, these are not the VMIDs tagged in TLB, rather we implement our own simple indexing mechanism).

When the host's OP-TEE driver is initialised or released, OP-TEE is notified about VM 1 being created/destroyed.

When a VMM (such as, QEMU) created a guest through KVM ioctls, a call to the TEE mediator layer is made, which in-turn calls OP-TEE mediator which eventually assigns a VM context, VMID, etc. and notifies OP-TEE about guest creation. The opposite happens on guest destruction.

When the guest makes an SMC targetting OP-TEE, it is trapped by the hypervisor and the register state (kvm_vcpu) is sent to the OP-TEE mediator through the TEE layer. Here there are two possibilities.

The guest may make an SMC with arguments which are simple numeric values, exchanging UUID, version information, etc. In this case, the mediator has much less work. It has to attach VMID into X7 and pass the register state to OP-TEE.

But, when guest passes memory addresses as arguments, the mediator has to translate these into physical addresses from intermediate physical addresses (IPA). According to the OP-TEE protocol (as documented in optee_smc.h and optee_msg.h), the guest OP-TEE driver would share a buffer filled with pointers, which the mediator translates.

The OP-TEE mediator also keeps track of active calls between each guest and OP-TEE, and pins pages which are already shared. This is to avoid swapping of shared pages by the host under memory pressure. These pages are unpinned as soon as guest's transaction completes with OP-TEE.

Testing
=======

The feature has been tested on QEMU virt platform using "xtest" as the test suite. As of now, all of 35000+ tests pass. The mediator has also been stressed under memory pressure and all tests pass too. Any suggestions on further testing the feature are welcome.

Call for review
===============
Any insights/suggestions regarding the implementation are appreciated.

Yuvraj Sakshith (7):
firmware: smccc: Add macros for Trusted OS/App owner check on SMC
value
tee: Add TEE Mediator module which aims to expose TEE to a KVM guest.
KVM: Notify TEE Mediator when KVM creates and destroys guests
KVM: arm64: Forward guest CPU state to TEE mediator on SMC trap
tee: optee: Add OPTEE_SMC_VM_CREATED and OPTEE_SMC_VM_DESTROYED
tee: optee: Add OP-TEE Mediator
tee: optee: Notify TEE Mediator on OP-TEE driver initialization and
release

 arch/arm64/kvm/hypercalls.c        |   15 +-
 drivers/tee/Kconfig                |    5 +
 drivers/tee/Makefile               |    1 +
 drivers/tee/optee/Kconfig          |    7 +
 drivers/tee/optee/Makefile         |    1 +
 drivers/tee/optee/core.c           |   13 +-
 drivers/tee/optee/optee_mediator.c | 1319 ++++++++++++++++++++++++++++
 drivers/tee/optee/optee_mediator.h |  103 +++
 drivers/tee/optee/optee_smc.h      |   53 ++
 drivers/tee/optee/smc_abi.c        |    6 +
 drivers/tee/tee_mediator.c         |  145 +++
 include/linux/arm-smccc.h          |    8 +
 include/linux/tee_mediator.h       |   39 +
 virt/kvm/kvm_main.c                |   11 +-
 14 files changed, 1721 insertions(+), 5 deletions(-)
 create mode 100644 drivers/tee/optee/optee_mediator.c
 create mode 100644 drivers/tee/optee/optee_mediator.h
 create mode 100644 drivers/tee/tee_mediator.c
 create mode 100644 include/linux/tee_mediator.h
Signature · 1 line
2.43.0
#1 lore raw

RFC PATCH 1/7 firmware: smccc: Add macros for Trusted OS/App owner check on SMC value

This patch adds ARM_SMCCC_IS_OWNER_TRUSTED_APP() and ARM_SMCCC_IS_OWNER_TRUSTED_OS() macros. These can be used to identify if the SMC is targetted at a Trusted OS/App in the secure world.

---
 include/linux/arm-smccc.h | 8 ++++++++
 1 file changed, 8 insertions(+)
include/linux/arm-smccc.h +8 -0
index f19be5754090..da2b4565d5b3 100644
@@ -56,6 +56,14 @@
 #define ARM_SMCCC_OWNER_TRUSTED_OS	50
 #define ARM_SMCCC_OWNER_TRUSTED_OS_END	63
 
+#define ARM_SMCCC_IS_OWNER_TRUSTED_APP(smc_val) \
+		((ARM_SMCCC_OWNER_NUM(smc_val) >= ARM_SMCCC_OWNER_TRUSTED_APP) && \
+			(ARM_SMCCC_OWNER_NUM(smc_val) <= ARM_SMCCC_OWNER_TRUSTED_APP_END))
+
+#define ARM_SMCCC_IS_OWNER_TRUSTED_OS(smc_val)  \
+		((ARM_SMCCC_OWNER_NUM(smc_val) >= ARM_SMCCC_OWNER_TRUSTED_OS) && \
+			(ARM_SMCCC_OWNER_NUM(smc_val) <= ARM_SMCCC_OWNER_TRUSTED_OS_END))
+
 #define ARM_SMCCC_FUNC_QUERY_CALL_UID  0xff01
 
 #define ARM_SMCCC_QUIRK_NONE		0
Signature · 1 line
2.43.0
#2 lore raw

RFC PATCH 2/7 tee: Add TEE Mediator module which aims to expose TEE to a KVM guest.

In reply to Yuvraj Sakshith

The TEE Mediator module is an upper abstraction layer which lets KVM guests interact with a trusted execution environment.

TEE specific subsystems (such as OP-TEE, for example) can register a set of handlers through tee_mediator_register_ops() with the TEE Mediator, which will be called by the kernel when required.

Given this module, architecture specific TEE drivers can implement handler functions to work with these events if necessary. In most implementations, a special instruction (such as SMC, in arm64) switches control leading to the TEE. These instructions are usually trapped by the hypervisor when executed by the guest.

This module allows making use of these trapped instructions and mediating the request between guest and TEE.

---
 drivers/tee/Kconfig          |   5 ++
 drivers/tee/Makefile         |   1 +
 drivers/tee/tee_mediator.c   | 145 +++++++++++++++++++++++++++++++++++
 include/linux/tee_mediator.h |  39 ++++++++++
 4 files changed, 190 insertions(+)
 create mode 100644 drivers/tee/tee_mediator.c
 create mode 100644 include/linux/tee_mediator.h
drivers/tee/Kconfig +5 -0
index 61b507c18780..dc446c9746ee 100644
@@ -11,6 +11,11 @@ menuconfig TEE
 	  This implements a generic interface towards a Trusted Execution
 	  Environment (TEE).
 
+config TEE_MEDIATOR
+	bool "Trusted Execution Environment Mediator support"
+	depends on KVM
+	help
+	  Provides an abstraction layer for TEE drivers to mediate KVM guest requests to the TEE.
 if TEE
 
 source "drivers/tee/optee/Kconfig"
drivers/tee/Makefile +1 -0
index 5488cba30bd2..46c44e59dd0b 100644
@@ -1,5 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_TEE) += tee.o
+obj-$(CONFIG_TEE_MEDIATOR) += tee_mediator.o
 tee-objs += tee_core.o
 tee-objs += tee_shm.o
 tee-objs += tee_shm_pool.o
drivers/tee/tee_mediator.c new file +145 -0

145 lines not on this page · raw message

include/linux/tee_mediator.h new file +39 -0
new file mode 100644
index 000000000000..4a971de158ec
@@ -0,0 +1,39 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * TEE Mediator for the Linux Kernel
+ *
+ * This module enables a KVM guest to interact with a
+ * Trusted Execution Environment in the secure processing
+ * state provided by the architecture.
+ *
+ * Author:
+ *   Yuvraj Sakshith <[email protected]>
+ */
+
+#ifndef __TEE_MEDIATOR_H
+#define __TEE_MEDIATOR_H
+
+#include <linux/kvm_host.h>
+
+struct tee_mediator_ops {
+	int (*create_host)(void);
+	int (*destroy_host)(void);
+	int (*create_vm)(struct kvm *kvm);
+	int (*destroy_vm)(struct kvm *kvm);
+	void (*forward_request)(struct kvm_vcpu *vcpu);
+	int (*is_active)(void);
+};
+
+struct tee_mediator {
+	struct tee_mediator_ops *ops;
+};
+
+int tee_mediator_create_host(void);
+int tee_mediator_destroy_host(void);
+int tee_mediator_create_vm(struct kvm *kvm);
+int tee_mediator_destroy_vm(struct kvm *kvm);
+void tee_mediator_forward_request(struct kvm_vcpu *vcpu);
+int tee_mediator_is_active(void);
+int tee_mediator_register_ops(struct tee_mediator_ops *ops);
+
+#endif
Signature · 1 line
2.43.0
#3 lore raw

RFC PATCH 3/7 KVM: Notify TEE Mediator when KVM creates and destroys guests

In reply to Yuvraj Sakshith

TEEs supporting virtualization in the rich execution environment would want to know about guest creation and destruction by the hypervisor.

This change notifies the TEE mediator of these events (if its active).

---
 virt/kvm/kvm_main.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
virt/kvm/kvm_main.c +9 -2
index ba0327e2d0d3..65f1f5075fdd 100644
@@ -49,6 +49,7 @@
 #include <linux/lockdep.h>
 #include <linux/kthread.h>
 #include <linux/suspend.h>
+#include <linux/tee_mediator.h>
 
 #include <asm/processor.h>
 #include <asm/ioctl.h>
@@ -1250,7 +1251,10 @@ static void kvm_destroy_vm(struct kvm *kvm)
 {
 	int i;
 	struct mm_struct *mm = kvm->mm;
-
+#ifdef CONFIG_TEE_MEDIATOR
+	if (tee_mediator_is_active())
+		(void) tee_mediator_destroy_vm(kvm);
+#endif
 	kvm_destroy_pm_notifier(kvm);
 	kvm_uevent_notify_change(KVM_EVENT_DESTROY_VM, kvm);
 	kvm_destroy_vm_debugfs(kvm);
@@ -5407,7 +5411,10 @@ static int kvm_dev_ioctl_create_vm(unsigned long type)
 	 * care of doing kvm_put_kvm(kvm).
 	 */
 	kvm_uevent_notify_change(KVM_EVENT_CREATE_VM, kvm);
-
+#ifdef CONFIG_TEE_MEDIATOR
+	if (tee_mediator_is_active())
+		(void) tee_mediator_create_vm(kvm);
+#endif
 	fd_install(fd, file);
 	return fd;
 
Signature · 1 line
2.43.0
#4 lore raw

RFC PATCH 4/7 KVM: arm64: Forward guest CPU state to TEE mediator on SMC trap

In reply to Yuvraj Sakshith

When guest makes an SMC, the call is denied by the hypervisor and not handled (ignored). In the presence of the TEE Mediator module, the SMC from guest is forwarded with it's vCPU register state through tee_mediator_forward_request().

---
 arch/arm64/kvm/hypercalls.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
arch/arm64/kvm/hypercalls.c +13 -2
index 569941eeb3fe..cb34bb87188c 100644
@@ -3,6 +3,7 @@
 
 #include <linux/arm-smccc.h>
 #include <linux/kvm_host.h>
+#include <linux/tee_mediator.h>
 
 #include <asm/kvm_emulate.h>
 
@@ -90,7 +91,10 @@ static bool kvm_smccc_default_allowed(u32 func_id)
 		 */
 		if (func_id >= KVM_PSCI_FN(0) && func_id <= KVM_PSCI_FN(3))
 			return true;
-
+#ifdef CONFIG_TEE_MEDIATOR
+		if (ARM_SMCCC_IS_OWNER_TRUSTED_APP(func_id) || ARM_SMCCC_IS_OWNER_TRUSTED_OS(func_id))
+			return true;
+#endif
 		return false;
 	}
 }
@@ -284,7 +288,14 @@ int kvm_smccc_call_handler(struct kvm_vcpu *vcpu)
 		WARN_RATELIMIT(1, "Unhandled SMCCC filter action: %d\n", action);
 		goto out;
 	}
-
+#ifdef CONFIG_TEE_MEDIATOR
+	if (ARM_SMCCC_IS_OWNER_TRUSTED_APP(func_id) || ARM_SMCCC_IS_OWNER_TRUSTED_OS(func_id)) {
+		if (tee_mediator_is_active()) {
+			tee_mediator_forward_request(vcpu);
+			return 1;
+		}
+	}
+#endif
 	switch (func_id) {
 	case ARM_SMCCC_VERSION_FUNC_ID:
 		val[0] = ARM_SMCCC_VERSION_1_1;
Signature · 1 line
2.43.0
#5 lore raw

RFC PATCH 5/7 tee: optee: Add OPTEE_SMC_VM_CREATED and OPTEE_SMC_VM_DESTROYED

In reply to Yuvraj Sakshith

OP-TEE when compiled with NS-Virtualization support expects NS-Hypervisor to notify events such as guest creation and destruction through SMCs.

This change adds two macros OPTEE_SMC_VM_CREATED and OPTEE_SMC_VM_DESTROYED.

---
 drivers/tee/optee/optee_smc.h | 53 +++++++++++++++++++++++++++++++++++
 1 file changed, 53 insertions(+)
drivers/tee/optee/optee_smc.h +53 -0
index 879426300821..988539b2407b 100644
@@ -452,6 +452,59 @@ struct optee_smc_disable_shm_cache_result {
 /* See OPTEE_SMC_CALL_WITH_REGD_ARG above */
 #define OPTEE_SMC_FUNCID_CALL_WITH_REGD_ARG	19
 
+/*
+ * Inform OP-TEE about a new virtual machine
+ *
+ * Hypervisor issues this call during virtual machine (guest) creation.
+ * OP-TEE records client id of new virtual machine and prepares
+ * to receive requests from it. This call is available only if OP-TEE
+ * was built with virtualization support.
+ *
+ * Call requests usage:
+ * a0	SMC Function ID, OPTEE_SMC_VM_CREATED
+ * a1	Hypervisor Client ID of newly created virtual machine
+ * a2-6 Not used
+ * a7	Hypervisor Client ID register. Must be 0, because only hypervisor
+ *      can issue this call
+ *
+ * Normal return register usage:
+ * a0	OPTEE_SMC_RETURN_OK
+ * a1-7	Preserved
+ *
+ * Error return:
+ * a0	OPTEE_SMC_RETURN_ENOTAVAIL	OP-TEE have no resources for
+ *					another VM
+ * a1-7	Preserved
+ *
+ */
+#define OPTEE_SMC_FUNCID_VM_CREATED		13
+#define OPTEE_SMC_VM_CREATED \
+	OPTEE_SMC_FAST_CALL_VAL(OPTEE_SMC_FUNCID_VM_CREATED)
+
+/*
+ * Inform OP-TEE about shutdown of a virtual machine
+ *
+ * Hypervisor issues this call during virtual machine (guest) destruction.
+ * OP-TEE will clean up all resources associated with this VM. This call is
+ * available only if OP-TEE was built with virtualization support.
+ *
+ * Call requests usage:
+ * a0	SMC Function ID, OPTEE_SMC_VM_DESTROYED
+ * a1	Hypervisor Client ID of virtual machine being shut down
+ * a2-6 Not used
+ * a7	Hypervisor Client ID register. Must be 0, because only hypervisor
+ *      can issue this call
+ *
+ * Normal return register usage:
+ * a0	OPTEE_SMC_RETURN_OK
+ * a1-7	Preserved
+ *
+ */
+
+#define OPTEE_SMC_FUNCID_VM_DESTROYED	14
+#define OPTEE_SMC_VM_DESTROYED \
+	OPTEE_SMC_FAST_CALL_VAL(OPTEE_SMC_FUNCID_VM_DESTROYED)
+
 /*
  * Resume from RPC (for example after processing a foreign interrupt)
  *
Signature · 1 line
2.43.0
#6 lore raw

RFC PATCH 6/7 tee: optee: Add OP-TEE Mediator

In reply to Yuvraj Sakshith

The OP-TEE mediator is a software entity resposible for bridging the gap between a KVM guest and OP-TEE in the secure world.

The guest's OP-TEE driver issues an SMC instruction after populating its CPU registers with appropriate arguments. This SMC is trapped by the hypervisor and control is passed to the OP-TEE mediator with the vCPU state.

The mediator is resposible for manipulating the vCPU state accordingly and keeping track of active transactions between the guest and the TEE.

This implementation adds event handlers that gets hooked to the TEE Mediator layer and are called when events such as guest creation/destruction, mediator status check and guest SMC trap happen.

Important routines implemented:

- optee_mediator_{create|destroy}_vm(): Sends an SMC to OP-TEE notifying KVM guest creation or destruction.

- optee_mediator_{create|destroy}_host(): Sends an SMC to OP-TEE notifying host OP-TEE driver initalization/release (OP-TEE treats all NS-EL1 entities as a guest, and is not aware of host privilege).

- optee_mediator_forward_smc(): Changes vCPU register state as required by OP-TEE and keeps track of standard calls and memory shared by guest.

The OP-TEE mediator is implemented in such a way that, the guest/VMM can remain unmodified. The guest interacts with OP-TEE just as it would if it was running natively.

---
 drivers/tee/optee/Kconfig          |    7 +
 drivers/tee/optee/Makefile         |    1 +
 drivers/tee/optee/optee_mediator.c | 1319 ++++++++++++++++++++++++++++
 drivers/tee/optee/optee_mediator.h |  103 +++
 4 files changed, 1430 insertions(+)
 create mode 100644 drivers/tee/optee/optee_mediator.c
 create mode 100644 drivers/tee/optee/optee_mediator.h
drivers/tee/optee/Kconfig +7 -0
index 7bb7990d0b07..ef41d6d1793e 100644
@@ -25,3 +25,10 @@ config OPTEE_INSECURE_LOAD_IMAGE
 
 	  Additional documentation on kernel security risks are at
 	  Documentation/tee/op-tee.rst.
+
+config OPTEE_MEDIATOR
+	bool "OP-TEE Mediator support"
+	depends on TEE_MEDIATOR && OPTEE && ARM64 && KVM
+	help
+	  This enables a KVM guest equipped with an OP-TEE driver to interact with OP-TEE
+	  in the secure world.
drivers/tee/optee/Makefile +1 -0
index a6eff388d300..4a777940e0df 100644
@@ -1,4 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_OPTEE_MEDIATOR) += optee_mediator.o
 obj-$(CONFIG_OPTEE) += optee.o
 optee-objs += core.o
 optee-objs += call.o
drivers/tee/optee/optee_mediator.c new file +1319 -0

1,319 lines not on this page · raw message

drivers/tee/optee/optee_mediator.h new file +103 -0

103 lines not on this page · raw message

Signature · 1 line
2.43.0
#7 lore raw

RFC PATCH 7/7 tee: optee: Notify TEE Mediator on OP-TEE driver initialization and release

In reply to Yuvraj Sakshith

When host initializes or releases its OP-TEE driver through optee_core_init()/optee_core_exit(), notify OP-TEE in the secure world about this change.

If OP-TEE is built with NS-Virtualization support, it will treat SMCs coming from the host as if it were coming from a VM (as OP-TEE does not understand the KVM paradigm).

Hence, OPTEE_SMC_VM_CREATED and OPTEE_SMC_VM_DESTROYED SMCs have to be made for its internal book-keeping.

---
 drivers/tee/optee/core.c    | 13 ++++++++++++-
 drivers/tee/optee/smc_abi.c |  6 ++++++
 2 files changed, 18 insertions(+), 1 deletion(-)
drivers/tee/optee/core.c +12 -1
index c75fddc83576..5f2ab0ee0893 100644
@@ -14,6 +14,7 @@
 #include <linux/slab.h>
 #include <linux/string.h>
 #include <linux/tee_core.h>
+#include <linux/tee_mediator.h>
 #include <linux/types.h>
 #include "optee_private.h"
 
@@ -195,7 +196,13 @@ static bool intf_is_regged;
 static int __init optee_core_init(void)
 {
 	int rc;
-
+#ifdef CONFIG_TEE_MEDIATOR
+	if (tee_mediator_is_active()) {
+		rc = tee_mediator_create_host();
+		if (rc < 0)
+			return rc;
+	}
+#endif
 	/*
 	 * The kernel may have crashed@the same time that all available
 	 * secure world threads were suspended and we cannot reschedule the
@@ -240,6 +247,10 @@ static void __exit optee_core_exit(void)
 		optee_smc_abi_unregister();
 	if (!ffa_abi_rc)
 		optee_ffa_abi_unregister();
+#ifdef CONFIG_TEE_MEDIATOR
+	if (tee_mediator_is_active())
+		tee_mediator_destroy_host();
+#endif
 }
 module_exit(optee_core_exit);
 
drivers/tee/optee/smc_abi.c +6 -0
index f0c3ac1103bb..a930ca8cde23 100644
@@ -25,8 +25,10 @@
 #include <linux/slab.h>
 #include <linux/string.h>
 #include <linux/tee_core.h>
+#include <linux/tee_mediator.h>
 #include <linux/types.h>
 #include <linux/workqueue.h>
+#include "optee_mediator.h"
 #include "optee_private.h"
 #include "optee_smc.h"
 #include "optee_rpc_cmd.h"
@@ -1396,6 +1398,10 @@ static void optee_smccc_smc(unsigned long a0, unsigned long a1,
 			    unsigned long a6, unsigned long a7,
 			    struct arm_smccc_res *res)
 {
+#ifdef CONFIG_TEE_MEDIATOR
+	if (tee_mediator_is_active())
+		a7 = OPTEE_HOST_VMID;
+#endif
 	arm_smccc_smc(a0, a1, a2, a3, a4, a5, a6, a7, res);
 }
 
Signature · 1 line
2.43.0
maintainer
#8 lore raw
In reply to Yuvraj Sakshith
Quoted 32 lines · Yuvraj Sakshith

A KVM guest running on an arm64 machine will not be able to interact with a trusted execution environment (which supports non-secure guests) like OP-TEE in the secure world. This is because, instructions provided by the architecture (such as, SMC) which switch control to the firmware, are trapped in EL2 when the guest is executes them.

This series adds a feature into the kernel called the TEE mediator abstraction layer, which lets a guest interact with the secure world. Additionally, a OP-TEE specific mediator is also implemented, which hooks itself to the TEE mediator layer and intercepts guest SMCs targetted at OP-TEE.

Overview
=========

Essentially, if the kernel wants to interact with OP-TEE, it makes an "smc - secure monitor call instruction", after loading in arguments into CPU registers. What these arguments consists of and how both these entities communicate can vary. If a guest wants to establish a connection with the secure world, its not possible. This is because of the fact that "smc" by the guest are trapped by the hypervisor in EL2. This is done by setting the HCR_EL2.TSC bit before entering the guest.

Hence, this feature which I we may call TEE mediator, acts as an intermediary between the guest and OP-TEE. Instead of denying the guest SMC and jumping back into the guest, the mediator forwards the request to OP-TEE.

OP-TEE supports virtualization in the normal world and expects 6 things from the NS-hypervisor:

1. Notify OP-TEE when a VM is created.
2. Notify OP-TEE when a VM is destroyed.
3. Any SMC to OP-TEE has to contain the VMID in x7. If its the hypervisor sending, then VMID is 0.
4. Hypervisor has to perform IPA->PA translations of the memory addresses sent by guest.
5. Memory shared by the VM to OP-TEE has to remain pinned.
6. The hypervisor has to follow the OP-TEE protocol, so the guest thinks it is directly speaking to OP-TEE.

Quoted 5 lines

Its important to note that, if OP-TEE is built with NS-virtualization support, it can only function if there is a hypervisor with a mediator in normal world.

This implementation has been heavily inspired by Xen's OP-TEE mediator.

[...]

And I think this inspiration is the source of most of the problems in this series.

Routing Secure Calls from the guest to whatever is on the secure side should not be the kernel's job at all. It should be the VMM's job. All you need to do is to route the SMCs from the guest to userspace, and we already have all the required infrastructure for that.

It is the VMM that should:

- signal the TEE of VM creation/teardown

- translate between IPAs and host VAs without involving KVM

- let the host TEE driver translate between VAs and PAs and deal with the pinning as required, just like it would do for any userspace (without ever using the KVM memslot interface)

- proxy requests from the guest to the TEE

- in general, bear the complexity of anything related to the TEE

In short, the VMM is just another piece of userspace using the TEE to do whatever it wants. The TEE driver on the host must obviously know about VMs, but that's about it.

Crucially, KVM should:

- be completely TEE agnostic and never call into something that is TEE-specific

- allow a TEE implementation entirely in userspace, specially for the machines that do not have EL3

As it stands, your design looks completely upside-down. Most of this code should be userspace code and live in (or close to) the VMM, with the host kernel only providing the basic primitives, most of which should already be there.

Thanks,

	M.
Signature · 1 line
Jazz isn't dead. It just smells funny.
#9 lore raw
Quoted 42 lines · Marc Zyngier
Quoted 37 lines · Yuvraj Sakshith

A KVM guest running on an arm64 machine will not be able to interact with a trusted execution environment (which supports non-secure guests) like OP-TEE in the secure world. This is because, instructions provided by the architecture (such as, SMC) which switch control to the firmware, are trapped in EL2 when the guest is executes them.

This series adds a feature into the kernel called the TEE mediator abstraction layer, which lets a guest interact with the secure world. Additionally, a OP-TEE specific mediator is also implemented, which hooks itself to the TEE mediator layer and intercepts guest SMCs targetted at OP-TEE.

Overview
=========

Essentially, if the kernel wants to interact with OP-TEE, it makes an "smc - secure monitor call instruction", after loading in arguments into CPU registers. What these arguments consists of and how both these entities communicate can vary. If a guest wants to establish a connection with the secure world, its not possible. This is because of the fact that "smc" by the guest are trapped by the hypervisor in EL2. This is done by setting the HCR_EL2.TSC bit before entering the guest.

Hence, this feature which I we may call TEE mediator, acts as an intermediary between the guest and OP-TEE. Instead of denying the guest SMC and jumping back into the guest, the mediator forwards the request to OP-TEE.

OP-TEE supports virtualization in the normal world and expects 6 things from the NS-hypervisor:

1. Notify OP-TEE when a VM is created.
2. Notify OP-TEE when a VM is destroyed.
3. Any SMC to OP-TEE has to contain the VMID in x7. If its the hypervisor sending, then VMID is 0.
4. Hypervisor has to perform IPA->PA translations of the memory addresses sent by guest.
5. Memory shared by the VM to OP-TEE has to remain pinned.
6. The hypervisor has to follow the OP-TEE protocol, so the guest thinks it is directly speaking to OP-TEE.

Its important to note that, if OP-TEE is built with NS-virtualization support, it can only function if there is a hypervisor with a mediator in normal world.

This implementation has been heavily inspired by Xen's OP-TEE mediator.

[...]

Quoted 8 lines

And I think this inspiration is the source of most of the problems in this series.

Routing Secure Calls from the guest to whatever is on the secure side should not be the kernel's job at all. It should be the VMM's job. All you need to do is to route the SMCs from the guest to userspace, and we already have all the required infrastructure for that.

Yes, this was an argument at the time of designing this solution.

Quoted 6 lines

It is the VMM that should:

- signal the TEE of VM creation/teardown

- translate between IPAs and host VAs without involving KVM

Quoted 8 lines

- let the host TEE driver translate between VAs and PAs and deal with the pinning as required, just like it would do for any userspace (without ever using the KVM memslot interface)

- proxy requests from the guest to the TEE

- in general, bear the complexity of anything related to the TEE

Major reason why I went with placing the implementation inside the kernel is,
- OP-TEE userspace lib (client) does not support sending SMCs for VM events and needs modification.
- QEMU (or every other VMM) will have to be modified.
- OP-TEE driver is anyways in the kernel. A mediator will just be an addition and not a completely new entity.
- (Potential) issues if we would want to mediate requests from VM which has private mem.
- Heavy VM exits if guest makes frequent TOS calls.

Hence, the thought of making changes to too many entities (libteec, VMM, etc.) was a strong reason, although arguable.

Quoted 4 lines

In short, the VMM is just another piece of userspace using the TEE to do whatever it wants. The TEE driver on the host must obviously know about VMs, but that's about it.

Quoted 8 lines

Crucially, KVM should:

- be completely TEE agnostic and never call into something that is TEE-specific

- allow a TEE implementation entirely in userspace, specially for the machines that do not have EL3

Yes, you're right. Although I believe there still are some changes that need to be made to KVM for facilitating this. For example, kvm_smccc_get_action() would deny TOS call.

So, having an implementation completely in VMM without any change in KVM might be challenging, any potential solutions are welcome.

Quoted 11 lines

As it stands, your design looks completely upside-down. Most of this code should be userspace code and live in (or close to) the VMM, with the host kernel only providing the basic primitives, most of which should already be there.

Thanks,

	M.
Signature · 1 line
Jazz isn't dead. It just smells funny.
maintainer
#10 lore raw
Quoted 5 lines · Yuvraj Sakshith
Quoted 3 lines · Marc Zyngier
Quoted 1 line · Yuvraj Sakshith

[...]

Quoted 27 lines
Quoted 5 lines
Quoted 2 lines

This implementation has been heavily inspired by Xen's OP-TEE mediator.

[...]

Quoted 8 lines

And I think this inspiration is the source of most of the problems in this series.

Routing Secure Calls from the guest to whatever is on the secure side should not be the kernel's job at all. It should be the VMM's job. All you need to do is to route the SMCs from the guest to userspace, and we already have all the required infrastructure for that.

Yes, this was an argument at the time of designing this solution.

Quoted 12 lines

It is the VMM that should:

- signal the TEE of VM creation/teardown

- translate between IPAs and host VAs without involving KVM

- let the host TEE driver translate between VAs and PAs and deal with the pinning as required, just like it would do for any userspace (without ever using the KVM memslot interface)

- proxy requests from the guest to the TEE

Quoted 7 lines
Quoted 2 lines

- in general, bear the complexity of anything related to the TEE

Major reason why I went with placing the implementation inside the kernel is,
- OP-TEE userspace lib (client) does not support sending SMCs for VM events and needs modification.
- QEMU (or every other VMM) will have to be modified.

Sure. And what? New feature, new API, new code. And what will happen once someone wants to use something other than OP-TEE? Or one of the many forks of OP-TEE that have a completely different ABI (cue the Android forks -- yes, plural)?

Quoted 2 lines

- OP-TEE driver is anyways in the kernel. A mediator will just be an addition and not a completely new entity.

Of course not. The TEE can be anywhere I want. On another machine if I decide so. Just because OP-TEE has a very simplistic model doesn't mean we have to be constrained by it.

Quoted 2 lines

- (Potential) issues if we would want to mediate requests from VM which has private mem.

Private memory means that not even the host has access to it, as it is the case with pKVM. How would that be an issue?

Quoted 1 line

- Heavy VM exits if guest makes frequent TOS calls.

Sorry, I have to completely dismiss the argument here. I'm not even remotely considering performance for something that is essentially a full context switch of the whole machine. By definition, calling into EL3, and then S-EL1/S-EL2 is going to be as fast as a dying snail, and an additional exit to userspace will hardly register for anything other than a pointless latency benchmark.

Quoted 3 lines

Hence, the thought of making changes to too many entities (libteec, VMM, etc.) was a strong reason, although arguable.

It is a *terrible* reason. By this reasoning, we would have subsumed the whole VMM into the kernel (just like Xen), because "we don't want to change userspace".

Furthermore, you are not even considering basic things such as permissions. Your approach completely circumvents any form of access control, meaning that if any user that can create a VM can talk to the TEE, even if they don't have access to the TEE driver.

Yes, you could replicate access permission, SE-Linux, seccomp (and the rest of the security theater) at the KVM/TEE boundary, making the whole thing even more of a twisted mess.

Or you could simply do the right thing and let the kernel do its job the way it was intended by using the syscall interface from userspace.

Quoted 10 lines
Quoted 10 lines

In short, the VMM is just another piece of userspace using the TEE to do whatever it wants. The TEE driver on the host must obviously know about VMs, but that's about it.

Crucially, KVM should:

- be completely TEE agnostic and never call into something that is TEE-specific

Quoted 7 lines
Quoted 3 lines

- allow a TEE implementation entirely in userspace, specially for the machines that do not have EL3

Yes, you're right. Although I believe there still are some changes that need to be made to KVM for facilitating this. For example, kvm_smccc_get_action() would deny TOS call.

If something is missing in KVM to allow routing of SMCs to userspace, I'm more than happy to entertain the change.

Quoted 2 lines

So, having an implementation completely in VMM without any change in KVM might be challenging, any potential solutions are welcome.

I've said what I have to say already, and pointed you in a direction that I see as both correct and maintainable.

Thanks,

	M.
Signature · 1 line
Jazz isn't dead. It just smells funny.
#11 lore raw
Quoted 45 lines · Marc Zyngier
Quoted 5 lines · Yuvraj Sakshith
Quoted 3 lines · Marc Zyngier
Quoted 1 line · Yuvraj Sakshith

[...]

Quoted 27 lines
Quoted 5 lines
Quoted 2 lines

This implementation has been heavily inspired by Xen's OP-TEE mediator.

[...]

Quoted 8 lines

And I think this inspiration is the source of most of the problems in this series.

Routing Secure Calls from the guest to whatever is on the secure side should not be the kernel's job at all. It should be the VMM's job. All you need to do is to route the SMCs from the guest to userspace, and we already have all the required infrastructure for that.

Yes, this was an argument at the time of designing this solution.

Quoted 12 lines

It is the VMM that should:

- signal the TEE of VM creation/teardown

- translate between IPAs and host VAs without involving KVM

- let the host TEE driver translate between VAs and PAs and deal with the pinning as required, just like it would do for any userspace (without ever using the KVM memslot interface)

- proxy requests from the guest to the TEE

Quoted 7 lines
Quoted 2 lines

- in general, bear the complexity of anything related to the TEE

Major reason why I went with placing the implementation inside the kernel is,
- OP-TEE userspace lib (client) does not support sending SMCs for VM events and needs modification.
- QEMU (or every other VMM) will have to be modified.

Quoted 4 lines

Sure. And what? New feature, new API, new code. And what will happen once someone wants to use something other than OP-TEE? Or one of the many forks of OP-TEE that have a completely different ABI (cue the Android forks -- yes, plural)?

If something other than OP-TEE has to be supported, a specific mediator (such as drivers/tee/optee/optee_mediator.c) has to be constructed with handlers hooked via tee_mediator_register_ops().

But yes, the ABI might change and the implementor has the freedom to mediate it as required.

Quoted 7 lines
Quoted 2 lines

- OP-TEE driver is anyways in the kernel. A mediator will just be an addition and not a completely new entity.

Of course not. The TEE can be anywhere I want. On another machine if I decide so. Just because OP-TEE has a very simplistic model doesn't mean we have to be constrained by it.

Quoted 6 lines
Quoted 2 lines

- (Potential) issues if we would want to mediate requests from VM which has private mem.

Private memory means that not even the host has access to it, as it is the case with pKVM. How would that be an issue?

Guest shares memory to OP-TEE through a buffer filled with pointers, which the mediator has to read for IPA->PA translations of all these pointers. VMM wont be able to read these if memory is private.

But, this is a "potential" solution and if at all the mediator is moved to VMM, this is completely ruled out.

Quoted 9 lines
Quoted 1 line

- Heavy VM exits if guest makes frequent TOS calls.

Sorry, I have to completely dismiss the argument here. I'm not even remotely considering performance for something that is essentially a full context switch of the whole machine. By definition, calling into EL3, and then S-EL1/S-EL2 is going to be as fast as a dying snail, and an additional exit to userspace will hardly register for anything other than a pointless latency benchmark.

Okay, makes sense.

Quoted 4 lines
Quoted 3 lines

Hence, the thought of making changes to too many entities (libteec, VMM, etc.) was a strong reason, although arguable.

Quoted 8 lines

It is a *terrible* reason. By this reasoning, we would have subsumed the whole VMM into the kernel (just like Xen), because "we don't want to change userspace".

Furthermore, you are not even considering basic things such as permissions. Your approach completely circumvents any form of access control, meaning that if any user that can create a VM can talk to the TEE, even if they don't have access to the TEE driver.

Well, this is a good point. OP-TEE built for NS-Virt supports handles calls from different VMs under different MMU partitions (will need to go off track to explain this). But, each VM's state and data remains isolated internally in S-EL1.

Quoted 21 lines

Yes, you could replicate access permission, SE-Linux, seccomp (and the rest of the security theater) at the KVM/TEE boundary, making the whole thing even more of a twisted mess.

Or you could simply do the right thing and let the kernel do its job the way it was intended by using the syscall interface from userspace.

Quoted 14 lines
Quoted 13 lines

In short, the VMM is just another piece of userspace using the TEE to do whatever it wants. The TEE driver on the host must obviously know about VMs, but that's about it.

Crucially, KVM should:

- be completely TEE agnostic and never call into something that is TEE-specific

- allow a TEE implementation entirely in userspace, specially for the machines that do not have EL3

Quoted 6 lines
Quoted 3 lines

Yes, you're right. Although I believe there still are some changes that need to be made to KVM for facilitating this. For example, kvm_smccc_get_action() would deny TOS call.

If something is missing in KVM to allow routing of SMCs to userspace, I'm more than happy to entertain the change.

Okay.

Quoted 6 lines
Quoted 2 lines

So, having an implementation completely in VMM without any change in KVM might be challenging, any potential solutions are welcome.

I've said what I have to say already, and pointed you in a direction that I see as both correct and maintainable.

Yes, I get your point on placing mediator in VMM. And now that I think of it, I believe I can make an improvement.

But yes, since too many entities are involved, the design of this solution has been a nightmare. Good to have been pushed this way.

Quoted 6 lines

Thanks,

	M.
Signature · 1 line
Jazz isn't dead. It just smells funny.

Thanks,
Yuvraj Sakshith